Lumberjack Sidekiq

This gem provides an enhanced logging setup for Sidekiq using the lumberjack structured logging framework. It replaces Sidekiq's default job logging behavior with one that provides rich structured logging with automatic tagging, timing information, and context propagation.

Key Features:

• Structured Job Logging: Automatically adds structured attributes for job metadata (class, job ID, queue, duration, etc.)
• Context Propagation: Pass log attributes from client to server to maintain request context across job execution
• Flexible Configuration: Control logging behavior per job with options for log levels, argument filtering, and custom attributes
• Performance Tracking: Automatic timing of job execution and queue wait times

The Lumberjack::Sidekiq::JobLogger provides structured logging for Sidekiq jobs with automatic tagging and timing information.

To use it, configure Sidekiq to use the Lumberjack job logger:

require 'lumberjack_sidekiq'

# First you'll need a Lumberjack logger instance
logger = Lumberjack::Logger.new(STDOUT)

# Configure Sidekiq to use Lumberjack
Sidekiq.configure_server do |config|
  config.logger = logger
  config[:job_logger] = Lumberjack::Sidekiq::JobLogger
end

The job logger automatically adds structured attributes to your log entries:

• class - The worker class name
• jid - The job ID
• bid - The batch ID (if using Sidekiq batch)
• queue - The queue name
• duration - Job execution time in seconds
• enqueued_ms - Time the job was queued before execution
• retry_count - Number of retries (if > 0)
• attributes - Any custom Sidekiq attributes

You can add an optional prefix to all attributes:

Sidekiq.configure_server do |config|
  config[:log_attribute_prefix] = "sidekiq."
end

The Lumberjack::Sidekiq::AttributePassthroughMiddleware allows you to pass log attributes from the client (where jobs are enqueued) to the server (where jobs are executed). This is useful for maintaining context like user IDs or request IDs across the job execution.

Configure the middleware on the client side:

Sidekiq.configure_client do |config|
  config.client_middleware do |chain|
    # Pass through :user_id and :request_id attributes to the job logger
    chain.add(Lumberjack::Sidekiq::AttributePassthroughMiddleware, :user_id, :request_id)
  end
end

Now when you enqueue a job with those attributes in the current logging context, they will be propagated to the logs when the job runs.

logger.tag(user_id: 123, request_id: "abc-def") do
  MyWorker.perform_async(params)
end

You can add additional metadata to your job logs by adding your own server middleware. Job logging sets up an attribute context so any attributes you add in your middleware will be included in the job log when it finishes.

Attributes added before the yield in your middleware will be included in all logs for the job processing. Attributes added after the yield will only be included in the final job lifecycle event log.

class MyLogTaggingMiddleware
  include Sidekiq::ServerMiddleware

  def call(worker, job, queue)
    # Add tag_1 to all logs for this job.
    Sidekiq.logger.tag(tag_1: job["value_1"]) if Sidekiq.logger.is_a?(Lumberjack::Logger)

    yield

    # Add tag_2 to the final job log only.
    Sidekiq.logger.tag(tag_2: job["value_2"]) if Sidekiq.logger.is_a?(Lumberjack::Logger)
  end
end

Sidekiq.configure_server do |config|
  config.server_middleware do |chain|
    chain.add MyLogTaggingMiddleware
  end
end

You can control logging behavior on a per-job basis by setting logging options:

class MyWorker
  include Sidekiq::Worker

  sidekiq_options logging: {
    level: "warn",           # Set log level for this job
    skip: false,             # Skip logging lifecycle events for this job
    skip_start: true,        # Skip the "Start job" lifecycle log message
    args: ["param1"],        # Only log specific arguments by name; can specify false to omit all args
    attributes: {custom: "value"}  # Add custom attributes to job logs
  }

  def perform(param1, param2)
    # Your job logic here
  end
end

You can globally disable logging job start events by setting :skip_start_job_logging to true in the Sidekiq configuration.

Sidekiq.configure_server do |config|
  config[:skip_start_job_logging] = true
end

You can add a prefix to all automatically generated log attributes by setting :log_attribute_prefix.

Sidekiq.configure_server do |config|
  config[:log_attribute_prefix] = "sidekiq."
end

You can disable logging the enqueued time by setting :skip_enqueued_time_logging to true.

Sidekiq.configure_server do |config|
  config[:skip_enqueued_time_logging] = true
end

You can disable logging any job arguments by setting :skip_logging_job_arguments to true.

Sidekiq.configure_server do |config|
  config[:skip_logging_job_arguments] = true
end

You can customize the message format by implementing your own Lumberjack::Sidekiq::MessageFormatter and setting it in the configuration. You can use this if your existing log processing pipeline is expecting specific message formats.

Sidekiq.configure_server do |config|
  config[:job_logger_message_formatter] = MyCustomMessageFormatter.new(config)
end

Add this line to your application's Gemfile:

gem "lumberjack_sidekiq"

And then execute:

$ bundle install

Or install it yourself as:

$ gem install lumberjack_sidekiq

Open a pull request on GitHub.

Please use the standardrb syntax and lint your code with standardrb --fix before submitting.

The gem is available as open source under the terms of the MIT License.