ask-monitoring

LLM monitoring dashboard for Rails. Tracks cost, throughput, error rates, and response times for all LLM calls in your application. Real-time updates via Hotwire Turbo — metrics auto-refresh every 30 seconds. Slack alerts.

Works with any LLM provider via ask-instrumentation events.

gem "ask-monitoring"

bundle install
rails generate ask:monitoring:install
rails db:migrate

Then visit /ask/monitoring.

The dashboard uses Hotwire Turbo to auto-refresh every 30 seconds — no Redis, no ActionCable, no JavaScript framework needed. Turbo Frames intercept the refresh and smoothly morph in updated metrics without a full page reload.

Filter by time range (1h, 24h, 7d, 30d), provider, or model.

Ask::Monitoring::Cost.for("openai/gpt-4", tokens: { input: 100, output: 50 })
# => 0.006 (USD)

Built-in pricing for 22+ models across OpenAI, Anthropic, Google, Mistral, Cohere, and Bedrock.

Ask::Monitoring::Cost.register("my-provider/my-model", input: 0.001, output: 0.002)

Ask::Monitoring.configure do |config|
  config.alert_rules << {
    name: "High error rate",
    condition: ->(metrics) { metrics[:error_rate] > 0.05 },
    channels: [:slack]
  }
end

Slack alerts use Incoming Webhooks:

Ask::Monitoring::Channels::Slack.new(
  webhook_url: ENV["SLACK_WEBHOOK_URL"]
).deliver(alert)

git clone https://github.com/ask-rb/ask-monitoring.git
cd ask-monitoring
bundle install
bundle exec rake test

MIT