RubyLLM::Monitoring

Monitor your LLM usage within your Rails application.

Installation

Note

This engine relies on RubyLLM. Make sure you have it installed and configured.

Add this line to your application's Gemfile:

gem "ruby_llm-monitoring"

And then execute:

$ bundle

To copy and migrate RubyLLM::Monitoring's migrations, run:

$ rails ruby_llm_monitoring:install:migrations db:migrate

And then mount the engine in your config/routes.rb:

Rails.application.routes.draw do
  # ...

  mount RubyLLM::Monitoring::Engine, at: "/monitoring"
end

Now you should be able to browse to /monitoring and monitor your LLM usage.

Authentication and authorization

RubyLLM::Monitoring leaves authentication and authorization to the user. If no authentication is enforced, /monitoring will be available to everyone.

To enforce authentication, you can use route constraints, or set up a HTTP Basic auth middleware.

For example, if you're using devise, you can do this:

# config/routes.rb
authenticate :user do
  mount RubyLLM::Monitoring::Engine, at: "/monitoring"
end

See more examples here.

However, if you're using Rails' default authentication generator, or an authentication solution that doesn't provide constraints, you need to roll out your own solution:

# config/routes.rb
constraints ->(request) { Constraints::Auth.authenticated?(request) } do
  mount RubyLLM::Monitoring::Engine, at: "/monitoring"
end

# lib/constraints/auth.rb
class Constraints::Auth
  def self.authenticated?(request)
    cookies = ActionDispatch::Cookies::CookieJar.build(request, request.cookies)

    Session.find_by id: cookies.signed[:session_id]
  end
end

You can also set up a HTTP Basic auth middleware in the engine:

# config/initializers/ruby_llm-monitoring.rb
RubyLLM::Monitoring::Engine.middleware.use(Rack::Auth::Basic) do |username, password|
  ActiveSupport::SecurityUtils.secure_compare(Rails.application.credentials.ruby_llm_monitoring_username, username) &
    ActiveSupport::SecurityUtils.secure_compare(Rails.application.credentials.ruby_llm_monitoring_password, password)
end

Metrics

The dashboard displays four metrics by default: Throughput, Cost, Response Time, and Error Rate. You can customize which metrics are shown or add your own custom metrics.

Configuration

In config/initializers/ruby_llm_monitoring.rb, you can configure which metrics are displayed:

RubyLLM::Monitoring.metrics = [
  RubyLLM::Monitoring::Metrics::Throughput,
  RubyLLM::Monitoring::Metrics::Cost,
  RubyLLM::Monitoring::Metrics::ResponseTime,
  RubyLLM::Monitoring::Metrics::ErrorCount
]

To remove a metric, simply omit it from the array:

RubyLLM::Monitoring.metrics = [
  RubyLLM::Monitoring::Metrics::Throughput,
  RubyLLM::Monitoring::Metrics::Cost,
  RubyLLM::Monitoring::Metrics::ResponseTime
]

Custom metrics

Create custom metrics by inheriting from RubyLLM::Monitoring::Metrics::Base:

class CostByFeature < RubyLLM::Monitoring::Metrics::Base
  title "Cost by Feature"
  unit "money"

  private

  def metric_data
    # Extract metadata from JSON payload and group by feature
    scope.group("json_extract(payload, '$.metadata.feature')").sum(:cost)
  end

  def build_series(aggregated_data)
    aggregated_data
      .group_by { |(_, feature), _| [feature || "unknown"] }
      .transform_values { |entries|
        entries.map { |(timestamp, _), value|
          [timestamp.to_i * 1000, value || default_value]
        }
      }
      .map { |keys, data| { name: keys.first, data: data } }
  end
end

The scope is an ActiveRecord relation of Event records grouped by time bucket. Your metric_data method should return aggregated data that will be displayed as a time series chart.

Note: JSON extraction syntax varies by database:

SQLite: json_extract(payload, '$.metadata.feature')
PostgreSQL: payload->'metadata'->>'feature'
MySQL: payload->>'$.metadata.feature'

This example assumes you're setting metadata using RubyLLM::Instrumentation.with():

RubyLLM::Instrumentation.with(feature: "chat_assistant") do
  RubyLLM.chat.ask("Hello")
end

Then add your custom metric to the configuration:

RubyLLM::Monitoring.metrics = [
  RubyLLM::Monitoring::Metrics::Throughput,
  RubyLLM::Monitoring::Metrics::Cost,
  CostByFeature
]

Alerts

RubyLLM::Monitoring can send alerts when certain conditions are met. Useful for monitoring cost, errors, etc.

Configuration

In config/initializers/ruby_llm_monitoring.rb, you can set the notification channels and alert rules:

RubyLLM::Monitoring.channels = {
  email: { to: "team@example.com" },
  slack: { webhook_url: ENV["SLACK_WEBHOOK_URL"] },
}

# Default cooldown between repeated alerts (optional, defaults to 5 minutes)
RubyLLM::Monitoring.alert_cooldown = 15.minutes

RubyLLM::Monitoring.alert_rules += [{
  time_range: -> { 1.hour.ago.. },
  rule: ->(events) { events.where.not(exception_class: nil).count > 10 },
  channels: [:slack],
  message: { text: "More than 10 errors in the last hour" }
}, {
  time_range: -> { Time.current.at_beginning_of_month.. },
  rule: ->(events) { events.sum(:cost) >= 500 },
  channels: [:email, :slack],
  message: { text: "More than $500 spent this month" }
}]

Rule options

Option	Required	Description
`time_range`	Yes	Lambda returning a range for filtering events (e.g., `-> { 1.hour.ago.. }`)
`rule`	Yes	Lambda receiving events scope, returns true to trigger alert
`channels`	Yes	Array of channel names to notify
`message`	Yes	Hash with `:text` key for the alert message
`cooldown`	No	Override default cooldown for this rule

Built-in channels

Slack

RubyLLM::Monitoring.channels = {
  slack: {
    webhook_url: ENV["SLACK_WEBHOOK_URL"]
  }
}

Email

RubyLLM::Monitoring.channels = {
  email: {
    to: "team@example.com",
    from: "alerts@example.com",  # optional
    subject: "LLM Alert"         # optional
  }
}

Custom channels

class PagerDutyChannel < RubyLLM::Monitoring::Channels::Base
  def self.deliver(message, config)
    # Your implementation
    # message[:text] contains the alert text
    # config contains channel configuration
  end
end

RubyLLM::Monitoring.channel_registry.register(:pagerduty, PagerDutyChannel)

RubyLLM::Monitoring.channels = {
  pagerduty: { api_key: ENV["PAGERDUTY_API_KEY"] }
}

Contributing

You can open an issue or a PR in GitHub.

License

The gem is available as open source under the terms of the MIT License.