Trifle::Traces

Structured execution tracing for Ruby. Track timestamped outputs from background jobs, API integrations, and anything else that runs in a black box. Know exactly what happened, when, and in what order.

Part of the Trifle ecosystem.

gem 'trifle-traces'

Trifle::Traces.tracer = Trifle::Traces::Tracer::Hash.new(key: 'jobs/sync')

Trifle::Traces.trace('Starting sync')
result = Trifle::Traces.trace('Fetching records from API') { api.fetch_all }
Trifle::Traces.trace('Sync complete')

Trifle::Traces.tracer.wrapup

Every message becomes a structured timeline entry on the tracer:

Trifle::Traces.tracer.data
#=> [
#     {at: 1739700000, message: 'Starting sync', state: :success, type: :text, level: 0},
#     {at: 1739700001, message: 'Fetching records from API', state: :success, type: :text, level: 0},
#     {at: 1739700003, message: 'Sync complete', state: :success, type: :text, level: 0}
#   ]

Ideal for debugging those background-job-that-talks-to-API-and-works-every-time-when-you-run-it-manually-but-never-in-production type of jobs.

Traces persist through two pluggable drivers: an index driver (searchable trace metadata) and a data driver (the payload). Configure both and the gem handles the whole lifecycle — no hand-written persistence callbacks.

Trifle::Traces.configure do |config|
  config.index_driver = Trifle::Traces::Driver::Index::Mongo.new(
    mongo_client, collection_name: 'trifle_traces'
  )
  config.data_driver = Trifle::Traces::Driver::Data::S3.new(
    client: s3_client, buckets: %w[traces-a traces-b], gzip: true
  )

  config.context   = ->(tracer) { { tenant_id: tracer.meta&.first } } # extra index fields
  config.retention = ->(tracer) { 3 }                                 # days, value or callable
end

• Index drivers: Mongo (production-proven at 100M+ traces/day), Memory, Null. ClickHouse/OpenSearch/Postgres planned — see lib/trifle/traces/driver/README.md to write your own.
• Data drivers: S3 (any S3-compatible storage, multi-bucket sharding, gzip), File, Memory, Null.
• Retention: carried on every record (retention days + expires_at). Mongo expires metadata via TTL index; S3 expires payloads via one lifecycle rule per retention class (Driver::Data::S3.setup! creates them).
• Without configured drivers nothing persists — data stays on the tracer for your callbacks, as in 1.x.

Every liftoff/bump/wrapup writes to the index in :live mode (default), giving you live progress. For fast, line-heavy, high-volume jobs use :deferred — zero I/O until wrapup, then exactly one index write and one payload part per trace:

class Commodity::PollJob
  include Sidekiq::Job
  sidekiq_options tracer_key: 'commodity/poll', tracer_mode: :deferred
end

record = Trifle::Traces.find(reference)                    #=> TraceRecord
result = Trifle::Traces.search(segment: 'jobs/sync', tags: ['tenant:42'],
                               state: :error, limit: 50, cursor: nil)
entries = Trifle::Traces.payload(record)                   # all parts, in order

Search is intentionally narrow — key-path segment, tags, state, newest-first with cursor pagination. That is what stays fast at hundreds of millions of traces.

Callbacks still fire on :liftoff, :bump and :wrapup — use them for side effects like emitting metrics (persistence no longer belongs here):

Trifle::Traces.configure do |config|
  config.on(:wrapup) do |tracer|
    tracer.keys.each do |key|
      Trifle::Stats.track(key: "traces::#{key}", at: Time.now, values: { count: 1 })
    end
  end
end

• The return value of :liftoff callbacks no longer becomes tracer.reference — the index driver generates references. Move persistence out of callbacks into drivers (or implement a custom driver; see lib/trifle/traces/driver/README.md).
• Persistence errors are no longer silent: liftoff/wrapup failures raise, bump failures re-queue the data and retry on the next flush. Override config.error_handler to customize.

• Simple tracing. Collect messages and return values from code execution.
• Driver-based persistence. Mongo + S3/File out of the box; contracts for custom backends.
• Deferred mode. One write per trace for high-volume jobs (100M/day proven).
• State management. Automatic success/error state tracking.
• Callbacks. Hook into trace events for custom processing.
• Middleware integration. Built-in support for Rack, Rails, and Sidekiq.
• Thread-safe. Safe for concurrent execution.
• Zero runtime dependencies. Database/storage clients are injected.

Trifle::Traces provides middleware for popular frameworks:

• Rack. HTTP request tracing.
• Rails. Controller and view tracing.
• Sidekiq. Background job tracing (sidekiq_options tracer_key:, tracer_mode:).

Full guides and API reference at docs.trifle.io/trifle-traces

Bug reports and pull requests are welcome on GitHub at https://github.com/trifle-io/trifle-traces.

The gem is available as open source under the terms of the MIT License.