Project: shotgunner - The Ruby Toolbox

Shotgunner

Problem:

** If there is a necessity to fetch data from external APIs consequently or perform time cost operation.

Solution:

Use 'pmap' method instead 'map' and get fantastic performance boost just out of the box!
Or use it this with native run method:
1. Define options for parallel call
2. Yield api caller into the 'run' method.
3. Use assembled result's array that keeps original order

Notes:

Calls performed based on thread pooling.
Since 0.1.5 result data keeps original order given from the tasks array.
Since 1.0.1 presented traditional map approach which utilize same library. pmap method appears in the array object.
Use it carefully! It is mainly optimized for async task execution. It doesn't optimize hardware resources!
Before usage it is strongly recommended to "benchmark" result. See example below.

Install

gem install shotgunner

Rails

gem 'shotgunner', require: %w[shotgunner/parallel array]

Examples

class Service
  include Shotgunner::Parallel

  class << self
    def some_cool_logic
      urls_to_be_fetched =
        [URI('http://example.com/index.html?page=1'),
         URI('http://example.com/index.html?page=2'),
         URI('http://example.com/index.html?page=3')]

      ## pmap instead map
      data = urls_to_be_fetched.pmap do |uri|
        JSON.parse(Net::HTTP.get(uri))
      end
      
      ## native approach
      data = run tasks: urls_to_be_fetched, threads: 6 do |uri|
        Net::HTTP.get(uri)
      end
      
      data.map { |j| JSON.parse(j) }
    end
  end
end

# execution time test pmap
pry(main)> start_time = Time.now
pry(main)> [1,2,3,4].pmap{|i| sleep(1); i}
pry(main)> pp Time.now - start_time

=> 1.002188

# execution time test map
pry(main)> start_time = Time.now
pry(main)> [1,2,3,4].map{|i| sleep(1); i}
pry(main)> pp Time.now - start_time

=> 4.018127

# benchmark of fetching external url
pry(main)> Benchmark.bm do |x|
pry(main)*   x.report("pmap:") { 100.times.map{|i| URI("https://jsonplaceholder.typicode.com/posts/#{i}")}.pmap{|uri| JSON.parse(Net::HTTP.get(uri)) } }
pry(main)*   x.report("map:") { 100.times.map{|i| URI("https://jsonplaceholder.typicode.com/posts/#{i}")}.map{|uri| JSON.parse(Net::HTTP.get(uri)) } }
pry(main)* end

       user     system      total        real
pmap:  0.445716   0.065193   0.510909 (  4.736261)
map:  0.439792   0.074125   0.513917 ( 19.288908)