The project is in a healthy, maintained state
Synchronize execution across numerous instances.
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
2015
2016
2017
2018
2019
2020
2021
2022
2023
2024
2025
 Dependencies

Runtime

~> 5.3.0
 Project Readme

Build Status RSpec Status CodeQL Status Rubocop Status Benchmark Status

Redis Single File - Distributed Execution Synchronization

Redis single file is a queue-based implementation of a remote/shared semaphore for distributed execution synchronization. A distributed semaphore may be useful for synchronizing execution across numerous instances or between the application and background job workers.

Installation

Add this line to your application's Gemfile:

gem 'redis-single-file'

And then execute:

$ bundle

Or install it yourself as:

$ gem install redis-single-file

Configuration

Configure redis single file via its configuration object.

RedisSingleFile.configuration do |config|
  # config.host = 'localhost'
  # config.port = '6379'
  # config.name = 'default'
  # config.expire_in = 300
end

Usage Examples

Default lock name and infinite blocking

  semaphore = RedisSingleFile.new
  semaphore.synchronize do
    # synchronized logic defined here...
  end

Named locks can provide exclusive synchronization

   semaphore = RedisSingleFile.new(name: :user_cache_update)
   semaphore.synchronize do
      # synchronized logic defined here...
   end

Prevent deadlocks by providing a timeout

   semaphore = RedisSingleFile.new(name: :s3_file_upload)
   semaphore.synchronize(timeout: 15) do
      # synchronized logic defined here...
   end

Use your own redis client instance

   redis = Redis.new(...)
   semaphore = RedisSingleFile.new(redis:)
   semaphore.synchronize do
      # synchronized logic defined here...
   end

Documentation

Distributed Queue Design

The redis blpop command will attempt to pop (delete and return) a value from a queue but will block when no values are present in the queue. A timeout can be provided to prevent deadlock situations.

To unblock (unlock) an instance, add/push an item to the queue. This is done one at a time to controll the serialization of the distrubuted execution. Redis selects the instance waiting the longest each time a new token is added.

Auto Expiration

All redis keys are expired and automatically removed after a certain period but will be recreated again on the next use. Each new client should face one of two scenarios when entering synchronization.

  1. The mutex key is not set causing the client to create the keys and prime the queue with its first token unlocking it for the first execution.

  2. The mutex key is already set so the client will skip the priming and enter directly into the queue where it should immediately find a token left by the last client upon completion or block waiting for the current client to finish execution.

Considerations over redlock approach

Redlock is the current standard and the official approach suggested by redis themselves but the design does have some complexities/drawbacks that some may wish to avoid. The following is a list of pros and cons of redis single file over redlock.

Pro: Multi-master redis node configuration not required
The redlock design requires a multi-master redis node setup where each node is completely independent of the others (no replication). This would be uncommon in most standard application deployment environments so a seperate redis setup would be required just for the distributed lock management.

Redis single file will work with your existing redis configuration so no need to maintain a seperate redis setup for the application of distributed semaphores.
Pro: No polling or waiting logic needed as redis does all the blocking
The redlock design requires the client to enter into a polling loop checking for the ability to execute its logic repeatedly. This approach is less efficient and requires quite a bit more logic to accomplish also making it more prone to error.

Redis single file pushes much of this responsibility off to redis itself with the use of the blpop command. Redis will block on that call when no item is present in the queue and will allocate tokens to competing clients waiting their turn on a `first-come, first-served basis`.
Pro: Replication lag is not a concern with blpop
The redlock design requires a multi-master setup given it utilizes read operations that could be delegated to a read replica in a standard clustered redis deployement. Redis replication is handled in an async manner so replication lag can hinder distributed synchronization when using read operations against a cluster utlizing replication.

Redis single file is not susceptible to this limitation given that blpop is a write operation meaning it will always be handled by the master node eliminating concerns over replication lag.
Con: Redis cluster failover could disrupt currently queued clients
Redis single file does attempt to recognize a connection failure and proceeds in rejoining the queue when detected but there is still a small chance that a cluster failover could cause already queued clients to have issues.

Redlock is not susceptible to this given the use of the multi-master deployment and absence of read-replicas so cluster failover (and recovery) is not a concern.

Run Tests

$ bundle exec rspec
Finished in 0.00818 seconds (files took 0.09999 seconds to load)
22 examples, 0 failures

Benchmark

$ bundle exec ruby benchmark.rb
ruby 3.2.0 (2022-12-25 revision a528908271) [arm64-darwin22]
Warming up --------------------------------------
         synchronize   434.000 i/100ms
        synchronize!   434.000 i/100ms
      threaded (10x)    29.000 i/100ms
        forked (10x)     8.000 i/100ms
Calculating -------------------------------------
         synchronize      4.329k (± 1.9%) i/s  (230.98 μs/i) -     21.700k in   5.014460s
        synchronize!      4.352k (± 0.3%) i/s  (229.79 μs/i) -     22.134k in   5.086272s
      threaded (10x)    249.794 (±28.4%) i/s    (4.00 ms/i) -      1.073k in   5.058461s
        forked (10x)     56.588 (± 3.5%) i/s   (17.67 ms/i) -    288.000 in   5.097885s

Comparison:
        synchronize!:     4351.8 i/s
         synchronize:     4329.4 i/s - same-ish: difference falls within error
      threaded (10x):      249.8 i/s - 17.42x  slower
        forked (10x):       56.6 i/s - 76.90x  slower

Disclaimer

Warning

Make sure you understand the limitations and reliability inherent in this implementation prior to using it in a production environment. No guarantees are made. Use at your own risk!

Inspiration

Inspiration for this gem was taken from a number of existing projects. It would be beneficial for anyone interested to take a look at all 3.

  1. Redlock
  2. redis-semaphore
  3. redis-mutex

Contributing

  1. Fork it
  2. Create your feature branch (git checkout -b my-new-feature)
  3. Commit your changes (git commit -am 'Add some feature')
  4. Push to the branch (git push origin my-new-feature)
  5. Create a new Pull Request