em-http-fetcher¶ ↑

HTTP fetch client based on ruby EventMachne and EM-HTTP-Request that has configureable concurrency regardless of EM’s thread pool.

EM.run do
  trap(:INT) { EM.stop }
  fetcher = EM::HttpFetcher.new
  fetcher.callback do |req| # req is HttpRequest instance
    # Here is global callback block for all request
    p "Fetch success! #{req.last_effective_url} (#{req.response.size} bytes)"
  end

  %w(
    http://www.google.com/
    http://heroku.com/
    http://sourceforge.net/
    http://github.com/
  ).each do |url|
    fetcher.request url
  end

  req = fetcher.request 'http://www.ruby-lang.org/'
  req.callback do
    # Here is appendix callback block for this request.
    # Global callback block will also be called.
    puts "Hello Ruby!"
  end
end

You can install with gem.

$ gem install em-http-fetcher

However em-http-request (=< 1.0.3) has a redirection issue. If you need to handle redirection properly, try following workaround.

gem install bundler

And create Gemfile to fetch develop version of em-http-request.

source "http://rubygems.org"
gem 'em-http-request', :git => 'git://github.com/igrigorik/em-http-request.git'

Then run bundle to install gems.

$ bundle

Finally run your script with “bundle exec”.

$ bundle exec YOUR_SCRIPT

:concurrency

Concurrency for all request.

:host_concurrency

Concurrency per host.

:host_request_wait

Wait specified seconds after request on each request thread.

(all other keys)

Pass through for HttpRequest.new

:uri

Target URI (String or URI object)

:method

Request method (get/head/put…) (default=:get)

(all other keys)

Pass through for HttpRequest#(get/head/put…)

If first argument is not a hash, it will be treated as :uri.

• :host_concurrency is checked only for initial URI. When request is redirected to another host, number of parallel requests for one host may be over host_concurrency.

• Redirections will not work until issue #230 of em-http-request is resolved; github.com/igrigorik/em-http-request/pull/230

Same as Ruby 2.0 (2-clause BSDL or Ruby original license)

EventMachine

rubyeventmachine.com/

EM-HTTP-Request

github.com/igrigorik/em-http-request