LlmLogs

Mountable Rails engine for LLM call tracing and prompt management. Auto-instruments ruby_llm to capture every LLM call as a span within a trace, and provides a web UI for browsing logs and editing prompts.

Installation

Add to your Gemfile:

gem "llm_logs"

Run the install generator:

bin/rails generate llm_logs:install
bin/rails db:migrate

Or manually:

# config/routes.rb
mount LlmLogs::Engine, at: "/llm_logs"

# config/initializers/llm_logs.rb
LlmLogs.setup do |config|
  config.enabled = true
  config.auto_instrument = true
  config.prompts_source_path = Rails.root.join("db/data/prompts")
end

Tracing

Wrap operations in a trace block. All ruby_llm calls inside become child spans automatically.

LlmLogs.trace("strategy_analysis", metadata: { strategy_id: 42 }) do
  chat = RubyLLM.chat(model: "anthropic/claude-sonnet-4")
  chat.ask("Analyze this strategy...")
end

Nested traces are supported:

LlmLogs.trace("full_pipeline") do
  chat.ask("Step 1...")

  LlmLogs.trace("risk_assessment") do
    chat.ask("What are the risks?")
  end
end

LLM calls made outside an explicit trace are auto-wrapped in one.

What Gets Captured

Each span records:

Model and provider
Input messages and output content
Input, output, and cached token counts
Duration in milliseconds
Error messages (on failure)
Custom metadata

Tool calls are captured as child spans with tool name and arguments.

Prompt Management

Create prompts with Mustache templates and auto-versioning.

Create a Prompt

prompt = LlmLogs::Prompt.create!(
  slug: "strategy-analysis",
  name: "Strategy Analysis"
)

prompt.update_content!(
  messages: [
    { "role" => "system", "content" => "You analyze trading strategies for {{app_name}}." },
    { "role" => "user", "content" => "Analyze {{strategy_name}} on the {{timeframe}} timeframe." }
  ],
  model: "claude-sonnet-4",
  model_params: { "temperature" => 0.3, "max_tokens" => 2048 }
)

Load and Render

prompt = LlmLogs::Prompt.load("strategy-analysis")
params = prompt.build(
  app_name: "Tradebot",
  strategy_name: "Momentum Alpha",
  timeframe: "4h"
)
# => { model: "claude-sonnet-4", messages: [...], temperature: 0.3, max_tokens: 2048 }

Versioning

Every save creates a new version. Previous versions are never modified.

prompt.update_content!(
  messages: [{ "role" => "user", "content" => "Updated prompt" }],
  changelog: "Simplified the prompt"
)

prompt.current_version       # latest
prompt.version(1)            # specific version
prompt.rollback_to!(1)       # creates new version from v1 content

Sync Prompts From Files

Store prompts as Markdown files and sync them into LlmLogs::Prompt records with the rake task.

# config/initializers/llm_logs.rb
LlmLogs.setup do |config|
  config.prompts_source_path = Rails.root.join("db/data/prompts")
  config.prompt_subfolders = %w[skills fragments templates]
end

bin/rails llm_logs:prompts:sync

The syncer reads *.md files from each configured subfolder. The subfolder name is added as a prompt tag automatically.

db/data/prompts/
  skills/
    backtest-evaluation.md
  fragments/
    provider-notes.md
  templates/
    trading-memo.md

Single-message prompts use the Markdown body as the system message:

---
slug: backtest-evaluation
name: Backtest Evaluation
description: How to evaluate backtests
tags: [evaluation]
model: anthropic/claude-sonnet-4
model_params:
  temperature: 0.3
---
Body content here.

Multi-message prompts can reference sibling body files:

---
slug: trading-memo
name: Trading Memo
model: deepseek/deepseek-v3.2
messages:
  - role: system
    body_file: trading_memo_system.md
  - role: user
    body_file: trading_memo_user.md
---

Running the task creates missing prompts, updates metadata, and creates a new prompt version only when messages, model, or model parameters changed.

Batches

Send latency-insensitive requests through a provider's Batch API for roughly half the cost. LlmLogs persists each request, groups pending requests into a provider batch, reconciles results, and records a trace per request — so batched work shows up in the dashboard alongside synchronous calls.

Two batch backends are supported, selected per model:

OpenAI Responses Batch API via ruby_llm-responses_api — the default for OpenAI models.
AWS Bedrock Batch API (CreateModelInvocationJob) for Anthropic Claude models.

Add the OpenAI provider to your app's Gemfile:

gem "ruby_llm-responses_api"

Enqueue a Request

Requests are persisted immediately and grouped by purpose + model when submitted:

LlmLogs::Batch.enqueue(
  purpose: "chat_summary",
  model: "gpt-4.1-mini",
  instructions: "Summarize the conversation in two sentences.",
  input: conversation_text,
  schema: SummarySchema,          # optional RubyLLM::Schema for structured output
  routing: { conversation_id: 42 }, # your keys, echoed into the trace metadata
  temperature: 0.2                  # optional
)

routing is arbitrary metadata you control. It rides along with the request and is copied onto the recorded trace, so you can trace a result back to your own records.

Handle Results

# config/initializers/llm_logs.rb
LlmLogs.register_batch_handler("chat_summary", ChatSummaryHandler.new)

class ChatSummaryHandler
  # Called once a request succeeds. `message` is the RubyLLM::Message.
  def call(request, message)
    Conversation.find(request.routing["conversation_id"])
      .update!(summary: message.content)
  end

  # Called when a request fails or its batch expires.
  def on_failure(request, error)
    Rails.logger.warn("[chat_summary] #{request.custom_id} failed: #{error}")
  end
end

A request is marked succeeded only after its handler completes; a handler that raises leaves the request failed with the error visible in the dashboard, so a result is never silently lost.

Submit and Reconcile

Two background jobs drive the lifecycle — schedule them on your own cadence (e.g. via cron, solid_queue recurring tasks, or sidekiq-cron):

# Group this purpose's pending requests into provider batches and submit them.
LlmLogs::Batch::FlushJob.perform_later("chat_summary")

# Reconcile every in-flight batch: fetch results, run handlers, recover stale claims.
LlmLogs::Batch::PollJob.perform_later

FlushJob claims pending rows with FOR UPDATE SKIP LOCKED, so concurrent runs never double-submit. PollJob reconciles all unfinished batches and recovers requests stranded by an interrupted submission. Both are idempotent at the request level — already-resolved requests are skipped on re-run.

AWS Bedrock batches

Anthropic Claude models can batch through the AWS Bedrock Batch API. Bedrock batching is file-based: LlmLogs writes a JSONL manifest to S3, starts a CreateModelInvocationJob, and reads the results back from S3. The enqueue/handler/flush/reconcile flow above is identical — only the backend differs.

Add the AWS SDKs to your app's Gemfile:

gem "aws-sdk-bedrock"
gem "aws-sdk-s3"

Configure the Bedrock backend and register its adapter, pointing it at an S3 bucket and an IAM role Bedrock can assume:

# config/initializers/llm_logs.rb
LlmLogs.configuration.bedrock_batch = LlmLogs::Configuration::BedrockBatch.new(
  role_arn:      "arn:aws:iam::<account>:role/<bedrock-batch-role>", # role Bedrock assumes to read/write S3
  s3_bucket:     "my-bedrock-batch-bucket",                          # must be in the model's region
  s3_prefix:     "llm-batch",
  min_records:   100,                                                # Bedrock's minimum records per job
  model_matcher: /\Aanthropic\./,                                    # model ids that route to Bedrock
  region:        "us-east-1"
)
LlmLogs.register_batch_adapter(:bedrock, LlmLogs::Batch::Adapters::Bedrock.new)

Provider selection is per model: LlmLogs::Batch.batch_provider_for(model) returns :bedrock when the adapter is registered and model_matcher matches, otherwise the OpenAI backend when the model resolves there, otherwise nil (not batchable — run it synchronously). Bedrock enforces a minimum records per job, so check LlmLogs::Batch.min_records_for(model) and fall back to a synchronous call when a batch would be under the floor.

The adapter builds its AWS clients from the ambient credential chain by default; inject your own to authenticate explicitly:

LlmLogs::Batch::Adapters::Bedrock.new(
  s3:      Aws::S3::Client.new(region: "us-east-1", credentials: creds),
  bedrock: Aws::Bedrock::Client.new(region: "us-east-1", credentials: creds)
)

Prerequisites: an S3 bucket in the model's region, and an IAM service role trusting bedrock.amazonaws.com with s3:GetObject/s3:PutObject on the bucket prefix. The principal that calls the API needs bedrock:CreateModelInvocationJob, bedrock:GetModelInvocationJob, and iam:PassRole on that role.

Web UI

Browse traces and manage prompts at /llm_logs.

Traces — list with filtering by status, drill into hierarchical span trees with collapsible input/output.

Prompts — CRUD with Mustache template editor, model configuration, and version history.

Batches — list batches with status and request counts, drill into per-request results, tokens, routing metadata, and linked traces.

Configuration

LlmLogs.setup do |config|
  config.enabled = true                                      # master switch for logging
  config.auto_instrument = true                              # auto-prepend on RubyLLM::Chat
  config.retention_days = 30                                 # for future cleanup job
  config.prompts_source_path = Rails.root.join("db/data/prompts")
  config.prompt_subfolders = %w[skills fragments templates]
  config.batch_enabled = true                                # enable the batch API integration
  config.batch_provider = :openai_responses                  # default (OpenAI) backend; Bedrock is registered separately (see Batches)
  config.page_size = 50                                      # rows per page on all index pages
end

Requirements

Rails 8.0+
Ruby 3.3+
PostgreSQL

License

MIT

llm_logs

Development

Runtime