resque-fair-share

Requires Resque >= 2.0

Partition-level queue fairness for Resque.

A Resque plugin that implements weighted fair-share scheduling at the worker level. Instead of FIFO processing, workers rotate through partitions (accounts, tenants, teams, etc.) proportionally so that high-volume partitions don't starve smaller ones (inspired by Weighted Fair Queuing).

A single large account enqueues thousands of jobs and occupies the queue for hours while smaller accounts wait. Standard FIFO processing means whoever enqueues first gets processed first, regardless of how many jobs they already have in flight.

• On enqueue: routes the job into a per-partition sub-queue and registers the partition in a priority ZSET (scored by in-flight count). Also increments a pending counter in Redis.
• On dequeue (worker-side): a Lua script runs atomically that:
  1. Queries the ZSET for the lowest-scored eligible partition via ZRANGEBYSCORE ... LIMIT 0 1 (O(log k))
  2. LPOPs from that partition's sub-queue (O(1))
  3. Removes the item from the main Resque queue for consistency
• On perform: transitions the job from pending to in-flight, and back down on completion or failure
• Hard cap: configurable max_in_flight (default: 50 concurrent jobs per partition across all workers). Partitions at capacity are skipped entirely during dequeue.
• Observability: optional StatsD gauges of per-partition queue depth

The queue always drains at full speed. No jobs are artificially delayed or moved to holding queues.

Unlike a scan-window approach, the sub-queue strategy always sees all active partitions regardless of how many jobs one tenant has enqueued. A single tenant flooding the queue cannot starve others.

resque-restriction pauses excess jobs into a holding queue and releases them on a timer. Fair-share instead interleaves partitions dynamically. High-volume partitions still process, just not at the expense of everyone else.

Add to your Gemfile:

gem 'resque-fair-share'

Resque::FairShare.configure do |config|
  config.dequeue_strategy = :sub_queues  # :sub_queues (default) or :legacy_scan
  config.max_in_flight = 50              # hard cap per partition across all workers
  config.partition_key = :account_id     # default key to look up in job args
  config.statsd_client = MyApp.statsd    # optional, must respond to #gauge
  config.scan_size = 100                 # legacy_scan only: queue items to examine
end

The simplest case. Your job receives a hash containing the partition identifier:

class ProcessRecordsJob
  extend Resque::Plugins::FairShare

  @queue = :default
  fair_share_on :account_id

  def self.perform(params)
    account_id = params['account_id']
    # ...
  end
end

Resque.enqueue(ProcessRecordsJob, account_id: 42, record_ids: [1, 2, 3])

When the partition value isn't in a hash, use a block to extract it:

class SyncTenantJob
  extend Resque::Plugins::FairShare

  @queue = :sync
  fair_share_on { |args| args[0] }

  def self.perform(tenant_id, options)
    # ...
  end
end

Resque.enqueue(SyncTenantJob, 'tenant_123', full: true)

Note: with the default sub_queues strategy, the block form works fully for both enqueue and dequeue (partition routing is resolved in Ruby at enqueue time). With legacy_scan, the Lua script can only do hash-key lookup, so ensure the partition value is also present as a key in a hash argument.

If you don't call fair_share_on, the plugin uses config.partition_key (default: :account_id) and searches all hash arguments for that key.

class SimpleJob
  extend Resque::Plugins::FairShare

  @queue = :default

  def self.perform(params)
    # ...
  end
end

Resque::FairShare.configure { |c| c.partition_key = :org_id }
Resque.enqueue(SimpleJob, org_id: 7, data: 'hello')

The plugin maintains these keys in Redis (under the resque: namespace):

When config.statsd_client is set, the plugin emits gauges after each counter mutation:

• resque.fair_share.{queue}.{partition_value}.pending
• resque.fair_share.{queue}.{partition_value}.in_flight

Any object responding to #gauge(name, value) works (Datadog's dogstatsd-ruby, statsd-instrument, etc.).

The sub-queue dequeue runs atomically inside Redis. Partition selection is O(log k) via a ZSET scored by in-flight count. The LPOP from the chosen sub-queue is O(1). This scales to high-cardinality partition keys (100k+ unique tenants) without degradation.

The overhead grows with queue depth because of the LREM on the main queue (kept for resque-web compatibility). In absolute terms, even the worst case (100k items) stays under 1ms per dequeue.

When one tenant dominates, the ZSET picks the correct (underserved) partition in O(log k) with no scan. The plain Resque LPOP picks the dominant tenant's job every time (fast but unfair).

The fair-share enqueue does 3 Redis operations per job (RPUSH to main queue, RPUSH to sub-queue, ZADD NX to ZSET) plus an INCR for the pending counter. The constant ~0.4ms additional cost is independent of queue depth or partition cardinality.

Memory overhead is ~2x from dual-writing job payloads to both the main queue and sub-queues. The ZSET and counter keys add negligible overhead. At 10k jobs with 1000 tenants, the total fair-share footprint is ~1.2 MB.

bundle exec rspec --tag benchmark

If you have existing jobs in the queue that were enqueued before the sub-queue strategy was enabled, run the migration to backfill sub-queues:

Resque::FairShare::Migration.perform('default')

This reads the main queue in batches, extracts partition values, and populates the sub-queues. It's safe to run while workers are active (new jobs are dual-written by the hooks). You can also roll back to the legacy strategy at any time:

Resque::FairShare.configure { |c| c.dequeue_strategy = :legacy_scan }

docker compose up -d   # starts Redis on port 6380
bundle install
bundle exec rspec      # runs unit + integration tests
bundle exec rubocop    # linting

Or point to an existing Redis:

REDIS_URL=redis://localhost:$PORT/15 bundle exec rspec

MIT