ask-monitoring

LLM monitoring dashboard for Rails. Tracks cost, throughput, error rates, and response times for all LLM calls in your application. Real-time updates via Hotwire Turbo: metrics auto-refresh every 30 seconds. Slack and email alerts.

Works with any LLM provider via ask-instrumentation events.

gem "ask-monitoring"

bundle install
rails generate ask:monitoring:install
rails db:migrate

Then visit /ask/monitoring.

The dashboard uses Hotwire Turbo to auto-refresh every 30 seconds, with no Redis, no ActionCable, and no JavaScript framework needed. Turbo Frames intercept the refresh and morph in updated metrics without a full page reload.

Filter by time range (1h, 24h, 7d, 30d), provider, or model.

Ask::Monitoring::Cost.for("openai/gpt-4", tokens: { input: 100, output: 50 })
# => 0.006 (USD)

Built-in pricing for 22 models across OpenAI, Anthropic, Google, Mistral, Cohere, and Bedrock.

Ask::Monitoring::Cost.register("my-provider/my-model", input: 0.001, output: 0.002)

Ask::Monitoring.configure do |config|
  config.alert_rules << {
    name: "High error rate",
    condition: ->(metrics) { metrics[:error_rate] > 0.05 },
    channels: [:slack]
  }
end

Alert channels:

Ask::Monitoring::Channels::Slack.new(webhook_url: ENV["SLACK_WEBHOOK_URL"]).deliver(alert)
Ask::Monitoring::Channels::Email.new(from: "ops@example.com", to: "team@example.com").deliver(alert)

config.alert_cooldown (default 300 seconds) sets the minimum time between repeated alerts from the same rule; each rule can override it with a cooldown: option.

The full ask-rb documentation lives at https://ask-rb.github.io/ask-docs. Production monitoring covers ask-monitoring in depth. API reference: https://ask-rb.github.io/ask-docs/reference/api.

git clone https://github.com/ask-rb/ask-monitoring.git
cd ask-monitoring
bundle install
bundle exec rake test

MIT