farm.rb

Distribute work across CPUs by measuring what the machine affords, rather than by the caller naming a concurrency primitive.

The caller knows the shape of its work — map these rows, do this to each item — and nothing about ractors, threads, processes, or how many of them this machine can usefully hold. Which substrate to run upon and how wide to run it follow from measurement: the cores there are, the memory there is, how busy the machine already is, and what things of this kind have cost to run before. Naming a primitive in the caller fixes all of that at write time, upon whoever's laptop happened to be in front of the author.

This is a walking skeleton, 0.0.0, and most of that deciding is not here yet. What is here: the API that hides it, two executors which run it, the probe that will inform it, and the fall-back that makes a wrong guess safe — Farm::RactorExecutor runs the work across a pool of ractors as wide as the machine has performance cores, and anything which cannot cross the boundary, or that machines it, is retried serially rather than surfaced. The caller gets the right answer slowly instead of an exception quickly, and the library gets the observation.

Work travels as a name rather than a block: Farm.map(rows, Revenue) names the receiver and names call, or another method, upon it, so the work itself can cross to a worker. A block cannot be named over the boundary and blocks are run serially, faithfully and without pretence of parallelism.

Add this line to your application's Gemfile:

gem 'farm.rb'

And then execute:

$ bundle install

Or install it yourself as:

$ gem install farm.rb

require 'farm.rb'

module Revenue
  def self.call(row)
    row.quantity * row.price
  end

  def self.net(row)
    call(row) - row.discount
  end
end

Farm.map(rows, Revenue)
# => the map of Revenue.call over rows, spread across the performance cores

Farm.map(rows, Revenue, :net)
# => the same, naming the method

Farm.each(rows, Revenue)
# => run for effect; returns rows

Farm.map(rows){|row| row.upcase}
# => blocks cannot cross to a worker and are run serially

Whatever the executor cannot run it falls back from: work touching globals or shared state raises inside a worker, is tagged by the pool, and the whole run is retried serially in the caller's own process, where such work belongs. Farm::Probe exposes the measurements the deciding will rest upon — Farm::Probe.performance_cores, logical_cores, total_memory, resident_memory, load_averages — each asked of the platform and nil where the platform does not answer.

Ruby 4.0 or later, the Ractor API having changed at 4.0: Ractor::Port is the queue and #value answers where #take used to. Ruby still calls the API experimental and says so once per process upon the first worker spawned; the warning is Ruby's and is left alone.

1. Fork it ( https://github.com/thoran/farm/fork )
2. Create your feature branch (git checkout -b my-new-feature)
3. Commit your changes (git commit -am 'Add some feature')
4. Push to the branch (git push origin my-new-feature)
5. Create a new pull request

MIT