Grape::Idempotency 🍇🔁

Gem for supporting idempotency in your Grape APIs.

Implementation based on the Active Internet-Draft draft-ietf-httpapi-idempotency-key-header-04

Topics covered in this README:

• Installation
• Compatibility
• Basic Usage
• How it works
  • Making idempotency key header mandatory
  • Redis Storage Connectivity Issue
  • Stored data
• Configuration
• Changelog
• Contributing

Ruby 2.6 or newer is required. Grape 1 or newer is required.

grape-idempotency is available as a gem, to install it run:

gem install grape-idempotency

• Ruby 2.6 or newer.
• Grape 1.x and 2.x or newer are supported from a single codebase.
• Grape 1.0 - 1.4 are allowed by the gemspec but are not tested.
• Grape 1.x does not work with Rack 3, so if you are using Grape 1.x add gem 'rack', '< 3' to your Gemfile.
• Grape 1.5 does not load with ActiveSupport 7.1.x (7.0 and 7.2+ work). On Ruby 2.7 / 3.0, where 7.1 is the newest installable ActiveSupport, add gem 'activesupport', '< 7.1' to your Gemfile.

The tested Ruby and Grape combinations live in the CI workflow: pushes and pull requests run the combinations where Ruby or Grape behaviour changes, and a weekly (or manually triggered) run tests every Grape minor line from 1.5 onwards against every Ruby version it supports, plus the latest Grape release on the newest Ruby. To run the specs against a specific Grape line locally, set GRAPE_VERSION (this rewrites your local Gemfile.lock, do not commit it):

GRAPE_VERSION="~> 1.8.0" bundle update grape && bundle exec rspec

Configure the Grape::Idempotency class with a Redis instance.

require 'redis'
require 'grape-idempotency'

redis = Redis.new(host: 'localhost', port: 6379)
Grape::Idempotency.configure do |c|
  c.storage = redis
end

Now you can wrap your code inside the idempotent method:

require 'grape'
require 'grape-idempotency'

class API < Grape::API
    post '/payments' do
      idempotent do
        status 201
        Payment.create!({
          amount: params[:amount]
        })
      end
    end
  end
end

That's all! 🚀

Once you've set up the gem and enclosed your endpoint code within the idempotent method, your endpoint will exhibit idempotent behavior, but this will only occur if the consumer of the endpoint includes an idempotency key in their request. (If you want to make the idempotency key header mandatory for your endpoint, check How to make idempotency key header mandatory section)

This key allows your consumer to make the same request again in case of a connection error, without the risk of creating a duplicate object or executing the update twice.

To execute an idempotent request, simply request your user to include an extra Idempotency-Key: <key> header as part of his request.

This gem operates by storing the initial request's status code, response headers and response body, regardless of whether the request succeeded or failed, using a specific idempotency key. Subsequent requests with the same key will consistently yield the same result, even if there were 500 errors.

The stored response is the final one built by Grape (the formatted body, also for responses generated by error! or by rescue_from handlers), and it is returned byte for byte. When a response is replayed, your endpoint is not run, including any code placed after the idempotent block.

Some response headers are never stored nor replayed: Set-Cookie, hop-by-hop headers (e.g. Connection or Transfer-Encoding), Content-Length, Date, Age, Via, Server-Timing, X-Runtime, Original-Request and the configured idempotency key and request id headers. You can exclude more headers with the replay_headers_denylist configuration option.

Responses whose body is a stream or a file (e.g. using stream or sendfile) are not stored: the idempotency key is released and a warning is logged, so those endpoints are not idempotent.

Keys are automatically removed from the system once they are older than the configured expires_in (60 hours by default), and a new request is generated when a key is reused after the original has been removed. The idempotency layer compares incoming parameters to those of the original request and returns a 422 - Unprocessable Entity status code if they don't match, preventing accidental misuse. If a request is received while another one with the same idempotency key is still being processed the idempotency layer returns a 409 - Conflict status

Results are only saved if an API endpoint begins its execution. If incoming parameters fail validation or if the request conflicts with another one executing concurrently, no idempotent result is stored because no API endpoint has initiated execution. In such cases, retrying these requests is safe.

Additionally, this gem automatically appends the Original-Request header and the Idempotency-Key header to your API's response, enabling you to trace back to the initial request that generated that specific response.

For some endpoints, you want to enforce your consumers to provide idempotency key. So, when wrapping the code inside the idempotent method, you can mark it as required:

require 'grape'
require 'grape-idempotency'

class API < Grape::API
    post '/payments' do
      idempotent(required: true) do
        status 201
        Payment.create!({
          amount: params[:amount]
        })
      end
    end
  end
end

If the Idempotency-Key request header is missing for a idempotent operation requiring this header, the gem will reply with an HTTP 400 status code with the following body:

{
  "title": "Idempotency-Key is missing",
  "detail": "This operation is idempotent and it requires correct usage of Idempotency Key.",
}

If you want to change the error message returned in this scenario, check How to configure idempotency key missing error message section.

By default, Redis exceptions are not handled by the grape-idempotency gem.

Therefore, if an exception arises while attempting to read, write or delete data from the Redis storage, the gem will re-raise the identical exception to your application. Thus, you will be responsible for handling it within your own code, such as:

require 'grape'
require 'grape-idempotency'

class API < Grape::API
    post '/payments' do
      begin
        idempotent do
          status 201
          Payment.create!({
            amount: params[:amount]
          })
        end
      rescue Redis::BaseError => e
        error!("Redis error! Idempotency is very important here and we cannot continue.", 500)
      end
    end
  end
end

If you want to avoid this functionality, and you want the gem handles the potential Redis exceptions, you have the option to configure the gem for handling these Redis exceptions. Please refer to the manage_redis_exceptions configuration property.

🚨 WARNING: If a Redis exception appears AFTER performing the wrapped code, nothing will be re-raised. The process will continue working and the response will be returned to the consumer of your API. However, a 409 Conflict response can be returned to your consumer if it retried the same call with the same idempotency key. This is because the gem was not able to associate the response of the original request to the original idempotency key because those connectivity issues.

For each request with an idempotency key, the gem stores in Redis, under the grape:idempotency:<idempotency-key> key:

• The request path.
• A SHA-256 digest of the request params, used to detect when an idempotency key is reused with different params. The params themselves are NOT stored.
• The response status, headers (except the ones listed in How it works) and body, which are returned again when the request is retried.
• The original request id.

Stored requests are removed after expires_in seconds (60 hours by default).

Since 1.2.0 the request params are not stored. While deploying 1.2.0, instances still running an older version of the gem return the conflict_error_response for retries of requests stored by 1.2.0, as they compare the params of the request with the stored ones.

The response is stored as returned by your endpoint, so it may include sensitive data. Configure an encryption_key or an encryptor to encrypt everything the gem stores. Until you do, the gem logs a warning the first time an idempotent endpoint is performed.

The gem never logs idempotency keys, request params or response bodies.

In addition to the storage aspect, you have the option to supply additional configuration details to tailor the gem to the specific requirements of your project.

As we have mentioned in the How it works section, keys are automatically removed from the system once they are older than expires_in (60 hours by default). However, this timeframe may not be suitable for your project. To accommodate your specific needs, you can adjust this duration by using the expires_in parameter for configuration:

Grape::Idempotency.configure do |c|
  c.storage = @storage
  c.expires_in = 1800
end

So with the cofiguration above, the keys will expire in 30 minutes.

As outlined in the How it works section, in order to perform an idempotent request, you need to instruct your users to include an additional Idempotency-Key: <key> header with their request. However, if this header format doesn't align with your project requirements, you have the flexibility to configure the specific header that the gem will examine to determine idempotent behavior:

Grape::Idempotency.configure do |c|
  c.storage = @storage
  c.idempotency_key_header = "x-custom-idempotency-key"
end

Given the previous configuration, the gem will examine the X-Custom-Idempotency-Key: <key> for determine the idempotent behavior.

By default, this gem stores a random hex value as identifier when storing the original request and returns it in all the subsequent requests that use the same idempotency-key as Original-Request header in the response.

This value can be also provided by your consumer using the X-Request-Id: <request-id> header when performing the request to your API.

However, if you prefer to use a different format for getting the request identifier, you can configure the header to check using the request_id_header parameter:

Grape::Idempotency.configure do |c|
  c.storage = @storage
  c.request_id_header = "x-trace-id"
end

In the case above, you request your consumers to use the X-Trace-Id: <trace-id> header when requesting your API.

By default, the logger used by the gem is configured like Logger.new(STDOUT) and INFO level. As this gem does not log any message with INFO level, only WARN and ERROR messages will be logged.

If you want to provide your own logger, you want to change the level to DEBUG or you want to provide your own prefix, you can configure the gem like:

Grape::Idempotency.configure do |c|
  c.storage = @storage
  c.logger = Infrastructure::MyLogger.new
  c.logger_level = :debug
  c.logger_prefix = '[my-own-prefix]'
end

An example of the logged information when changing the level of the log to DEBUG and customizing the logger_prefix:

I, [2023-11-23T22:41:39.148163 #1]  DEBUG -- : [my-own-prefix] Performing endpoint "/payments" with idempotency.
I, [2023-11-23T22:41:39.148176 #1]  DEBUG -- : [my-own-prefix] Idempotency key is NOT mandatory for this endpoint.
I, [2023-11-23T22:41:39.148192 #1]  DEBUG -- : [my-own-prefix] Idempotency key received in request header "x-custom-idempotency-key".
I, [2023-11-23T22:41:39.148210 #1]  DEBUG -- : [my-own-prefix] Previous request information has NOT been found for the provided idempotency key.
I, [2023-11-23T22:41:39.148248 #1]  DEBUG -- : [my-own-prefix] Request stored as processing.
I, [2023-11-23T22:41:39.148261 #1]  DEBUG -- : [my-own-prefix] Performing the provided block.
I, [2023-11-23T22:41:39.148268 #1]  DEBUG -- : [my-own-prefix] Block has been performed.
I, [2023-11-23T22:41:39.148287 #1]  DEBUG -- : [my-own-prefix] Storing response.
I, [2023-11-23T22:41:39.148317 #1]  DEBUG -- : [my-own-prefix] Response stored.
I, [2023-11-23T22:41:39.148473 #1]  DEBUG -- : [my-own-prefix] Performing endpoint "/payments" with idempotency.
I, [2023-11-23T22:41:39.148486 #1]  DEBUG -- : [my-own-prefix] Idempotency key is NOT mandatory for this endpoint.
I, [2023-11-23T22:41:39.148502 #1]  DEBUG -- : [my-own-prefix] Idempotency key received in request header "x-custom-idempotency-key".
I, [2023-11-23T22:41:39.148523 #1]  DEBUG -- : [my-own-prefix] Request has been found for the provided idempotency key.
I, [2023-11-23T22:41:39.148537 #1]  DEBUG -- : [my-own-prefix] Returning the response from the original request.

By default, the grape-idempotency gem is configured to re-raise Redis exceptions.

If you want to delegate the Redis exception management into the gem, you can configure it using the manage_redis_exceptions configuration property.

Grape::Idempotency.configure do |c|
  c.storage = @storage
  c.manage_redis_exceptions = true
end

However, this approach carries a certain level of risk. In the case that Redis experiences an outage, the idempotent functionality will be lost, the endpoint will behave as no idempotent, and this issue may go unnoticed.

By default, all the response headers are stored and replayed except the ones listed in the How it works section. If your API returns other headers that must not be replayed, you can add them using the replay_headers_denylist configuration option. Header names are case-insensitive, and the headers excluded by default are always excluded:

Grape::Idempotency.configure do |c|
  c.storage = @storage
  c.replay_headers_denylist = ["X-Session-Token", "X-Rate-Limit-Remaining"]
end

Encrypts everything the gem stores in Redis (see Stored data) using ActiveSupport::MessageEncryptor with aes-256-gcm. The encryption key is derived from the provided value, so use a long random secret and share the same value across all the instances of your API, otherwise a retried request reaching another instance would not be able to read the stored response.

Grape::Idempotency.configure do |c|
  c.storage = @storage
  c.encryption_key = ENV.fetch("IDEMPOTENCY_ENCRYPTION_KEY") # e.g. generated with `SecureRandom.hex(32)`
end

Requests stored before configuring the encryption are still read, so you can enable it without affecting the retries in progress. While deploying it, instances running a version older than 1.2.0 are not able to read the encrypted requests, so upgrade the gem before configuring the encryption.

If a stored request cannot be decrypted (e.g. after changing the key, or after removing the encryption), the gem returns the conflict_error_response instead of performing the request again. Those requests are removed once expires_in has elapsed.

If you want to use your own encryption (e.g. a KMS, or your existing ActiveSupport::MessageEncryptor), provide an object responding to encrypt(string) and decrypt(string), both returning a String. decrypt must raise an error (or return nil) when the value cannot be decrypted.

class IdempotencyEncryptor
  def initialize(message_encryptor)
    @message_encryptor = message_encryptor
  end

  def encrypt(value)
    @message_encryptor.encrypt_and_sign(value)
  end

  def decrypt(value)
    @message_encryptor.decrypt_and_verify(value)
  end
end

Grape::Idempotency.configure do |c|
  c.storage = @storage
  c.encryptor = IdempotencyEncryptor.new(my_message_encryptor)
end

Configure either an encryption_key or an encryptor, not both.

When providing a Idempotency-Key: <key> header, this gem compares incoming parameters to those of the original request (if exists) and returns a 409 - Conflict status code if they don't match, preventing accidental misuse. The response body returned by the gem looks like:

{

  "title": "Idempotency-Key is already used",
  "detail": "This operation is idempotent and it requires correct usage of Idempotency Key. Idempotency Key MUST not be reused across different payloads of this operation."
}

You have the option to specify the desired response body to be returned to your users when this error occurs. This allows you to align the error format with the one used in your application.

Grape::Idempotency.configure do |c|
  c.storage = @storage
  c.conflict_error_response = {
    "type": "about:blank",
    "status": 409,
    "title": "Conflict",
    "detail": "You are using the same idempotent key for two different requests"
}
end

In the configuration above, the error is following the RFC-7807 format.

When a request with a Idempotency-Key: <key> header is performed while a previous one still on going with the same idempotency value, this gem returns a 409 - Conflict status. The response body returned by the gem looks like:

{

  "title": "A request is outstanding for this Idempotency-Key",
  "detail": "A request with the same idempotent key for the same operation is being processed or is outstanding."
}

You have the option to specify the desired response body to be returned to your users when this error occurs. This allows you to align the error format with the one used in your application.

Grape::Idempotency.configure do |c|
  c.storage = @storage
  c.processing_response = {
    "type": "about:blank",
    "status": 409,
    "title": "A request is still being processed",
    "detail": "A request with the same idempotent key is being procesed"
}
end

In the configuration above, the error is following the RFC-7807 format.

If the Idempotency-Key request header is missing for a idempotent operation requiring this header, the gem will reply with an HTTP 400 status code with the following body:

{
  "title": "Idempotency-Key is missing",
  "detail": "This operation is idempotent and it requires correct usage of Idempotency Key.",
}

You have the option to specify the desired response body to be returned to your users when this error occurs. This allows you to align the error format with the one used in your application.

Grape::Idempotency.configure do |c|
  c.storage = @storage
  c.mandatory_header_response = {
    "type": "about:blank",
    "status": 400,
    "title": "Idempotency-Key is missing",
    "detail": "Please, provide a valid idempotent key in the headers for performing this operation"
}
end

In the configuration above, the error is following the RFC-7807 format.

If you're interested in seeing the changes and bug fixes between each version of grape-idempotency, read the Changelog.

We welcome and appreciate contributions from the open-source community. Before you get started, please take a moment to review the guidelines below.

1. Fork the repository.
2. Clone the repository to your local machine.
3. Create a new branch for your contribution.
4. Make your changes and ensure they meet project standards.
5. Commit your changes with clear messages.
6. Push your branch to your GitHub repository.
7. Open a pull request in our repository.
8. Participate in code review and address feedback.
9. Once approved, your changes will be merged.

This project is dockerized, so be sure you have docker installed in your machine.

Once you clone the repository, you can use the Make commands to build the project.

make build

You can pass the tests running:

make test

Open issues on the GitHub issue tracker with clear information.

• Juan Carlos García - Creator - https://github.com/jcagarcia
• Carlos Cabanero - Contributor - https://github.com/Flip120
• Salvador Revert Pedron - Contributor - https://github.com/salrepe