fast_curl

Ultra-fast parallel HTTP client for Ruby. C extension built on libcurl curl_multi API.

• Parallel requests via curl_multi — no threads, no fibers needed
• GVL release — rb_thread_call_without_gvl during I/O, other Ruby threads keep running
• Fiber scheduler compatible — works inside Async do ... end without blocking other fibers
• Three modes: execute (all), first_execute (first N), stream_execute (yield as ready)
• Zero dependencies — only libcurl (available everywhere)

Requirements: Ruby >= 3.1, libcurl

Why Ruby 3.1? The C extension uses rb_fiber_scheduler_current, rb_fiber_scheduler_block and rb_fiber_scheduler_unblock to properly yield control to the Fiber Scheduler during I/O. These APIs are stable starting from Ruby 3.1. Without them, there is no correct way for a C extension to cooperate with the scheduler — earlier approaches (rb_thread_schedule) hold the GVL and block other fibers.

gem 'fast_curl'

Requires libcurl development headers:

# macOS
brew install curl

# Ubuntu/Debian
apt-get install libcurl4-openssl-dev

# Alpine
apk add curl-dev

results = FastCurl.get([
  { url: "https://api.example.com/users" },
  { url: "https://api.example.com/posts" }
], connections: 20, timeout: 30)

results.each do |index, response|
  puts "#{index}: #{response[:status]} — #{response[:body]}"
end

FastCurl.post([
  {
    url: "https://api.example.com/users",
    headers: { "Authorization" => "Bearer token" },
    body: { name: "John" }
  }
])

result = FastCurl.first_get([
  { url: "https://mirror1.example.com/file" },
  { url: "https://mirror2.example.com/file" },
  { url: "https://mirror3.example.com/file" }
], count: 1)

FastCurl.stream_get(urls, connections: 50) do |index, response|
  puts "Got response #{index}: #{response[:status]}"
end

# Automatic retry on network errors (timeout, connection issues)
results = FastCurl.get([
  { url: "https://unreliable-api.com/data" }
], retries: 3, retry_delay: 1000)  # 3 retries with 1s delay

# Retry on specific HTTP status codes
results = FastCurl.get([
  { url: "https://api.example.com/data" }
], retries: 2, retry_codes: [500, 502, 503], retry_delay: 500)

# Disable retries (default is 1 retry)
results = FastCurl.get(urls, retries: 0)

require "async"

Async do
  # fast_curl detects the fiber scheduler and yields
  # to other fibers during I/O instead of blocking
  results = FastCurl.get(urls, connections: 20)
end

[index, {
  status: 200,             # HTTP status code (0 on error)
  headers: { "Key" => "Value" },
  body: "response body"
}]

Benchmarks against httpbin.org, 5 iterations with 1 warmup, median times. Run yourself: bundle exec ruby benchmark/local_bench.rb.

Each request hits /delay/1 (server-side 1-second delay), so sequential baseline grows linearly while parallel clients stay near ~1s plus network overhead.

Ruby heap delta stays near zero across all scenarios for fast_curl — most allocation happens in C, not on the Ruby heap.

Bounded by timeout=2s rather than by the slow request.

• Net::HTTP sequential is the proof-of-parallelism baseline — it confirms fast_curl and Async are actually running concurrently, not that they "beat" a different library. 4×1s sequentially = 4s, parallel = ~1s + overhead.
• Variance is high against remote endpoints (httpbin.org). For stable numbers, use --local which spawns a WEBrick server on 127.0.0.1.
• fast_curl (thread) vs (fiber): same underlying C code, different scheduling. "thread" is the default; "fiber" kicks in automatically when called inside Async do ... end.

MIT