💎 Resque::UniqueAtRuntime

if ci_badges.map(&:color).detect { it != "green"} ☝️ let me know on Discord or RubyForum, as I may have missed the notification.

if ci_badges.map(&:color).all? { it == "green"} 👇️ send money so I can do more of this. FLOSS maintenance is now my full-time job.

I've summarized my thoughts in this blog post.

resque-unique_at_runtime prevents more than one worker from performing the same class of work at once. A job including Resque::Plugins::UniqueAtRuntime obtains a Redis lock before perform; when the lock is held, the job is re-enqueued after a short delay.

By default the uniqueness key is the Resque queue. Override unique_at_runtime_redis_key(*args) when uniqueness should be per account, tenant, or resource instead of per queue.

Tokens to Remember
Works with JRuby
Works with Truffle Ruby
Works with MRI Ruby 4
Works with MRI Ruby 3
Works with MRI Ruby 2
Support & Community
Source
Documentation
Compliance
Style
Maintainer 🎖️
... 💖	🧊 🐙 🛖 🧪

Compatible with MRI Ruby 2.3.0+, and concordant releases of JRuby, and TruffleRuby. CI workflows and Appraisals are generated for MRI Ruby 2.4+. This test floor is configured by ruby.test_minimum in .kettle-jem.yml and may be higher than the gem's runtime compatibility floor when legacy Rubies are not practical for the current toolchain.

The amazing test matrix is powered by the kettle-dev stack.

Gem	Source	Role	Daily download rank
appraisal2	GitHub	multi-dependency Appraisal matrix generation
appraisal2-rubocop	GitHub	RuboCop Appraisal generator integration
kettle-dev	GitHub	development, release, and CI workflow tooling
kettle-jem	GitHub	Appraisals & CI workflow templates
kettle-soup-cover	GitHub	SimpleCov coverage policy and reporting
kettle-test	GitHub	standard test runner and coverage harness
rubocop-lts	GitHub	Ruby-version-aware linting
turbo_tests2	GitHub	parallel test execution

Available as part of the Tidelift Subscription.

The maintainers of this and thousands of other packages are working with Tidelift to deliver commercial support and maintenance for the open source packages you use to build your applications. Save time, reduce risk, and improve code health, while paying the maintainers of the exact packages you use.

• 💡Subscribe for support guarantees covering all your FLOSS dependencies
• 💡Tidelift is part of Sonar
• 💡Tidelift pays maintainers to maintain the software you depend on!
  📊@Pointy Haired Boss: An enterprise support subscription is "never gonna let you down", and supports open source maintainers

Alternatively:

Install the gem and add to the application's Gemfile by executing:

bundle add resque-unique_at_runtime

If bundler is not being used to manage dependencies, install the gem by executing:

gem install resque-unique_at_runtime

Configure application defaults before workers start:

Resque::UniqueAtRuntime.configure do |config|
  config.lock_timeout = 5 * 24 * 60 * 60
  config.requeue_interval = 1
  config.unique_at_runtime_key_base = "r-uar"
end

lock_timeout is the recovery timeout for a stale lock, so it must exceed the longest expected job duration. requeue_interval is the wait before a contested job is re-enqueued. Jobs may override these values with @runtime_lock_timeout, @runtime_requeue_interval, and @unique_at_runtime_key_base.

Keep the key base global for the application. Set RESQUE_DEBUG=true (or a value containing runtime) to enable plugin debug logging.

resque-unique_at_runtime utilizes 3 class instance variables that can be set in your Jobs, in addition to the standard @queue. Here they are, with their default values:

@runtime_lock_timeout = 60 * 60 * 24 * 5
@runtime_requeue_interval = 1
@unique_at_runtime_key_base = "r-uar".freeze

The last one, in normal circumstances, shouldn't be set as different per class, or uniqueness cleanup becomes more difficult.

It should be set only once, globally:

Resque::UniqueAtRuntime.configuration.unique_at_runtime_key_base = "my-custom"

require 'resque-unique_at_runtime'

class StrictlySerialJob
  include Resque::Plugins::UniqueAtRuntime

  @queue = :serial_work

  def self.perform
    # only one at a time in this block, no parallelism allowed for this
    # particular queue
  end
end

Let's say you want the serial constraint to apply at a more granular level. Instead of applying at the queue level, you can overwrite the .redis_key method.

require 'resque-unique_at_runtime'

class StrictlySerialJob
  include Resque::Plugins::UniqueAtRuntime

  @queue = :serial_work

  # Returns a string that will be used as the redis key
  # NOTE: it is recommended to prefix your string with the 'unique_at_runtime:' to
  # namespace your key!
  def self.unique_at_runtime_redis_key(account_id, *args)
    "unique_at_runtime:strictly_serial_job:#{account_id}"
  end

  # Overwrite reenqueue to lpush instead of default rpush.  This attempts to
  # preserve job ordering but job order is *NOT* guaranteed and also not
  # likely. See the comment on SHA: e9912fb2 for why.
  def self.reenqueue(*args)
    Resque.redis.lpush("queue:#{Resque.queue_from_class(self)}", Resque.encode(class: self, args: args))
  end

  def self.perform(account_id, *args)
    # only one at a time in this block, no parallelism allowed for this
    # particular unique_at_runtime_redis_key
  end
end

NOTE: Without careful consideration of your problem domain, worker starvation and/or unfairness is possible for jobs in this example. Imagine a scenario where you have three jobs in the queue with two resque workers:

+---------------------------------------------------+
| :serial_work                                      |
|---------------------------------------------------|
|             |             |             |         |
| unique_at_runtime_redis_key:  | unique_at_runtime_redis_key:  | unique_at_runtime_redis_key:  | ...     |
|    A        |    A        |    B        |         |
|             |             |             |         |
| job 1       | job 2       | job 3       |         |
+---------------------------------------------------+
                                  ^
                                  |
  Possible starvation +-----------+
  for this job and
  subsequent ones

When the first worker grabs job 1, it'll acquire the mutex for processing redis_key A. The second worker tries to grab the next job off the queue but is unable to acquire the mutex for redis_key A so it places job 2 back at the head of the :serial_work queue. Until worker 1 completes job 1 and releases the mutex for redis_key A, no work will be done in this queue.

This issue may be avoided by employing dynamic queues, http://blog.kabisa.nl/2010/03/16/dynamic-queue-assignment-for-resque-jobs/, where the queue is a one to one mapping to the redis_key.

The secret to preserving job order semantics is to remove critical data from the resque job and store data in a separate redis list. Part of a running job's responsibility will be to grab data off of the separate redis list needed for it to complete its job.

+---------------------------------------------------+
| :serial_work for jobs associated with key A       |
|---------------------------------------------------|
|   data x    |   data y    |   data z    | ...     |
+---------------------------------------------------+

+---------------------------------------------------+
| :serial_work for jobs associated with key B       |
|---------------------------------------------------|
|   data m    |   data n    |   data o    | ...     |
+---------------------------------------------------+

+---------------------------------------------------+
| :serial_work                                      |
|---------------------------------------------------|
|             |             |             |         |
| unique_at_runtime_redis_key:  | unique_at_runtime_redis_key:  | unique_at_runtime_redis_key:  | ...     |
|    A        |    A        |    B        |         |
|             |             |             |         |
| job 1       | job 2       | job 3       |         |
+---------------------------------------------------+

It now doesn't matter whether job 1 and job 2 are re-ordered as whichever goes first will perform an atomic pop on the redis list that contains the data needed for its job (data x, data y, data z).

The behavior when multiple jobs exist in a queue protected by resque-unique_at_runtime is for one job to be worked, while the other is continuously dequeued and requeued until the first job is finished. This can result in that worker process pegging a CPU/core on a worker server. To guard against this, the default behavior is to sleep for 1 second before the requeue, which will allow the cpu to perform other work.

This can be customized using a @runtime_requeue_interval class instance variable in your job like so:

require 'resque-unique_at_runtime'

class StrictlySerialJob
  include Resque::Plugins::UniqueAtRuntime

  @queue = :serial_work
  @runtime_requeue_interval = 5         # sleep for 5 seconds before requeueing

  def self.perform
    # some implementation
  end
end

resque-unique_at_runtime extends Resque with a new method running?.

It can tell you if a job with a specific signature is running, based on the presence of the runtime uniqueness key.

Caveat: It can be difficult to get the signature right, especially if you have tools that enhance job signatures with options.

While resque tools are free software and will always be, the project would benefit immensely from some funding. Raising a monthly budget of... "dollars" would make the project more sustainable.

We welcome both individual and corporate sponsors! We also offer a wide array of funding channels to account for your preferences. Currently, Open Collective is our preferred funding platform.

If you're working in a company that's making significant use of resque tools we'd appreciate it if you suggest to your company to become a resque sponsor.

You can support the development of resque tools via GitHub Sponsors, Liberapay, PayPal, Open Collective and Tidelift.

Support us with a monthly donation and help us continue our activities. [Become a backer]

NOTE: kettle-readme-backers updates this list every day, automatically.

No backers yet. Be the first!

Become a sponsor and get your logo on our README on GitHub with a link to your site. [Become a sponsor]

NOTE: kettle-readme-backers updates this list every day, automatically.

No sponsors yet. Be the first!

I’m driven by a passion to foster a thriving open-source community – a space where people can tackle complex problems, no matter how small. Revitalizing libraries that have fallen into disrepair, and building new libraries focused on solving real-world challenges, are my passions. I was recently affected by layoffs, and the tech jobs market is unwelcoming. I’m reaching out here because your support would significantly aid my efforts to provide for my family, and my farm (11 🐔 chickens, 2 🐶 dogs, 3 🐰 rabbits, 8 🐈‍ cats).

If you work at a company that uses my work, please encourage them to support me as a corporate sponsor. My work on gems you use might show up in bundle fund.

I’m developing a new library, floss_funding, designed to empower open-source developers like myself to get paid for the work we do, in a sustainable way. Please give it a look.

Floss-Funding.dev: 👉️ No network calls. 👉️ No tracking. 👉️ No oversight. 👉️ Minimal crypto hashing. 💡 Easily disabled nags

See SECURITY.md.

If you need some ideas of where to help, you could work on adding more code coverage, or if it is already 💯 (see below) check issues or PRs, or use the gem and think about how it could be better.

We so if you make changes, remember to update it.

See CONTRIBUTING.md for more detailed instructions.

See CONTRIBUTING.md.

Everyone interacting with this project's codebases, issue trackers, chat rooms and mailing lists agrees to follow the .

Made with contributors-img.

Also see GitLab Contributors: https://gitlab.com/resque/resque-unique_at_runtime/-/graphs/main

This library follows for its public API where practical. For most applications, prefer the Pessimistic Version Constraint with two digits of precision.

For example:

spec.add_dependency("resque-unique_at_runtime", "~> 4.0")

Dropping support for a platform can be a breaking change for affected users. If a release changes supported platforms, it should be called out clearly in the changelog and versioned with that impact in mind.

To get a better understanding of how SemVer is intended to work over a project's lifetime, read this article from the creator of SemVer:

• "Major Version Numbers are Not Sacred"

See CHANGELOG.md for a list of releases.

The gem is available as open source under the terms of the MIT .

See LICENSE.md for the official copyright notice.

• Copyright (c) 2012-2014 Jonathan R. Wallace
• Copyright (c) 2013 Tatsuhiko Miyagawa
• Copyright (c) 2014 Lance Woodson
• Copyright (c) 2017-2018, 2026 Peter H. Boling
• Copyright (c) 2020 Daniel Cunha
• Copyright (c) 2024 jiikko

Maintainers have teeth and need to pay their dentists. After getting laid off in an RIF in March, and encountering difficulty finding a new one, I began spending most of my time building open source tools. I'm hoping to be able to pay for my kids' health insurance this month, so if you value the work I am doing, I need your support. Please consider sponsoring me or the project.

To join the community or get help, use the RubyForum or Discord.

To say "thanks!" ☝️ Join the community or 👇️ send money.

💌 💌 💌

Many parts of this project are actively managed by a kettle-jem smart template utilizing StructuredMerge.org merge contracts.

Thanks for RTFM. ☺️