Tracebook

Cost tracking and review dashboard for RubyLLM conversations.

Tracebook is a Rails engine that sits on top of RubyLLM's acts_as_chat and acts_as_message models. It adds per-message cost calculation, chat-level review workflows, and a Hotwire-powered dashboard — without duplicating any conversation data.

• Cost tracking: Per-message cost calculation based on configurable pricing rules
• Review workflow: Approve or flag entire chat conversations with comments
• Dashboard: Browse chats, view conversation threads, see cost/token summaries
• RubyLLM native: Reads directly from your Chat and Message models — no data duplication

• Ruby 3.4+
• Rails 8.1+
• RubyLLM with acts_as_chat / acts_as_message models

bundle add tracebook
bin/rails generate tracebook:install
bin/rails db:migrate

Mount the engine in config/routes.rb:

mount Tracebook::Engine => "/tracebook"

Seed pricing rules for common providers:

bin/rails tracebook:seed_pricing

# config/initializers/tracebook.rb
Tracebook.configure do |config|
  # Class names for your RubyLLM models
  config.chat_class = "Chat"       # default
  config.message_class = "Message" # default

  # Currency for cost calculations
  config.default_currency = "USD"  # default

  # How to display the user in the dashboard
  config.actor_display = ->(actor) { actor.try(:name) }

  # Items per page
  config.per_page = 25             # default
end

Tracebook includes an opt-in PII redaction pipeline for unstructured natural language in LLM conversations. Nothing is redacted unless explicitly configured.

Tracebook.configure do |config|
  # Enable individual patterns
  config.redact :email, :phone, :ssn, :credit_card

  # Or enable a whole group
  config.redact :pii, :api_keys
end

Tracebook.configure do |config|
  config.redact_pattern(
    /policy[:\s]*\d{10}/i,
    "[POLICY_NUMBER]",
    name: "policy_number"
  )
end

Provide any callable (proc, lambda, or object responding to call):

Tracebook.configure do |config|
  config.custom_redactors << ->(text) {
    text.gsub(/MRN-\d{8}/, "[MEDICAL_RECORD]")
  }
end

# Redact text directly
Tracebook.redact("Email user@test.com or call 555-123-4567")
# => "Email [EMAIL] or call [PHONE]"

# Use in your application before saving messages
content = Tracebook.redact(user_input)
chat.ask(content)

For context-sensitive PII that regex can't catch (e.g. "my social is seven eight two three three three two"), a future version will support LLM-based redaction using a local model (e.g., Ollama) to detect PII in natural language before persistence.

Tracebook adds four tables — all prefixed with tracebook_ to avoid collisions:

Your Chat and Message tables are untouched.

After an LLM response, call Tracebook.calculate_cost! to record the cost:

Tracebook.calculate_cost!(
  message,
  provider: "openai",
  model: "gpt-4o",
  latency_ms: elapsed_ms
)

This looks up the matching pricing rule, calculates input/output costs, and creates a tracebook_message_costs record joined to the message.

In a typical RubyLLM app, hook into the chat response flow:

class ChatResponseJob < ApplicationJob
  def perform(chat_id, content)
    chat = Chat.find(chat_id)

    chat.ask(content) do |chunk|
      # stream chunks...
    end

    # After response, calculate cost for the last assistant message
    message = chat.messages.where(role: "assistant").last
    model = chat.model

    Tracebook.calculate_cost!(
      message,
      provider: model.provider,
      model: model.model_id
    )
  end
end

Tracebook calculates costs using PricingRule records. Seed defaults for common providers:

bin/rails tracebook:seed_pricing

This creates rules for OpenAI, Anthropic, Gemini, and Ollama models.

Tracebook::PricingRule.create!(
  provider: "xai",
  model_glob: "grok-4-1-fast*",
  input_cents_per_unit: 20,   # per 1k tokens
  output_cents_per_unit: 50,
  effective_from: Date.new(2025, 7, 1),
  currency: "USD"
)

• gpt-4o — exact match
• gpt-4o* — matches gpt-4o, gpt-4o-mini, gpt-4o-2024-08-06
• claude-3-5-* — matches all Claude 3.5 models
• * — fallback for any model

When multiple rules match, Tracebook prefers the most specific glob (most literal characters), then the most recent effective_from date.

Reviews happen at the chat level, not per-message. In the dashboard:

1. Open a chat to see the full conversation thread
2. Click Approve or Flag
3. Add comments for context

Programmatic access:

chat = Chat.find(id)
review = Tracebook::ChatReview.for_chat(chat)

review.update!(
  review_state: :approved,
  reviewed_by: "admin@example.com"
)

review.comments.create!(author: "admin", body: "Looks good")

The dashboard is available at /tracebook/chats (or wherever you mount the engine).

• All chats with actor, model, message count, token usage, cost, review state
• KPIs: total chats, messages, cost

• Full conversation thread (user and assistant messages)
• Per-message token counts and costs
• Review controls (approve/flag/reset)
• Comment thread

By default, actors are shown as Name or ClassName#id. Customize with:

config.actor_display = ->(actor) {
  case actor
  when User then actor.email
  else "#{actor.class}##{actor.id}"
  end
}

The engine inherits from ActionController::Base. Restrict access with route constraints:

# HTTP Basic Auth
mount Tracebook::Engine => "/tracebook",
  constraints: ->(req) {
    Rack::Auth::Basic::Request.new(req.env).provided? &&
    Rack::Auth::Basic::Request.new(req.env).credentials == ["admin", ENV["TRACEBOOK_PASSWORD"]]
  }

# Devise
authenticate :user, ->(u) { u.admin? } do
  mount Tracebook::Engine => "/tracebook"
end

# Run tests
bin/rails test

# Seed pricing in development
bin/rails tracebook:seed_pricing

setup { Tracebook.reset_configuration! }
teardown { Tracebook.reset_configuration! }

MIT License. See MIT-LICENSE.