Langsmith Rails

LangsmithrbRails provides seamless integration with LangSmith for your Rails applications. LangSmith is a platform for debugging, testing, evaluating, and monitoring LLM applications.

This gem makes it easy to add LangSmith tracing to your Rails application, allowing you to monitor and debug your LLM operations. It provides a comprehensive set of features including request tracing, PII redaction, local buffering, evaluation frameworks, and more.

• 🔍 Request Tracing: Automatically trace HTTP requests with middleware
• 🧩 Service & Job Tracing: Trace your services and background jobs
• 🔒 PII Redaction: Automatically redact sensitive information
• 💾 Local Buffering: Store traces locally and send them in batches
• 📊 Evaluations: Evaluate LLM responses with customizable criteria
• 🔄 CI Integration: Run evaluations in your CI pipeline
• 🚀 Demo Application: Get started quickly with a sample application

Add this line to your application's Gemfile:

gem 'langsmithrb_rails'

And then execute:

$ bundle install

Or install it yourself as:

$ gem install langsmithrb_rails

After installing the gem, run the install generator to set up LangSmith in your Rails application:

$ rails generate langsmithrb_rails:install

This will:

1. Create a LangSmith initializer at config/initializers/langsmith.rb
2. Generate a YAML configuration file at config/langsmith.yml
3. Display post-install instructions

LangSmith is configured through environment variables and a YAML configuration file:

# Required
LANGSMITH_API_KEY=your_api_key

# Optional
LANGSMITH_PROJECT=your_project_name
LANGSMITH_SAMPLING=1.0  # Sampling rate (0.0 to 1.0)
LANGSMITH_ENABLED=true  # Enable/disable tracing

You can get your API key from LangSmith.

The YAML configuration file (config/langsmith.yml) allows for environment-specific settings:

default: &default
  api_key: <%= ENV.fetch("LANGSMITH_API_KEY", nil) %>
  project_name: <%= ENV.fetch("LANGSMITH_PROJECT", nil) %>
  sampling_rate: <%= ENV.fetch("LANGSMITH_SAMPLING", 1.0).to_f %>
  enabled: <%= ENV.fetch("LANGSMITH_ENABLED", "true") == "true" %>
  redact_pii: true
  timeout: 5

development:
  <<: *default

test:
  <<: *default

production:
  <<: *default
  sampling_rate: <%= ENV.fetch("LANGSMITH_SAMPLING", 0.1).to_f %>

You can also configure LangSmith programmatically in your initializer:

LangsmithrbRails.configure do |config|
  config.enabled = true # Enable LangSmith tracing
  config.api_key = "your_api_key" # Your LangSmith API key
  config.project_name = "your_project" # Optional project name
  config.sampling_rate = 0.1 # Sample 10% of traces
  config.redact_pii = true # Redact PII from traces
end

LangsmithrbRails provides several generators to help you set up different features:

$ rails generate langsmithrb_rails:install

Sets up the basic configuration for LangSmith.

$ rails generate langsmithrb_rails:tracing

Adds middleware and concerns for request-level tracing:

• Creates a middleware for tracing HTTP requests
• Adds a concern for tracing services
• Adds a concern for tracing background jobs
• Updates your application configuration

$ rails generate langsmithrb_rails:buffer

Sets up local buffering for traces:

• Creates a migration for the buffer table
• Adds a model for the buffer
• Adds a job for flushing the buffer
• Adds rake tasks for manual flushing

$ rails generate langsmithrb_rails:evals

Adds an evaluation framework for LLM responses:

• Creates sample datasets for evaluation
• Adds evaluation checks (correctness, LLM-graded)
• Adds evaluation targets (HTTP, Ruby)
• Adds rake tasks for running evaluations

$ rails generate langsmithrb_rails:ci

Sets up CI integration for LangSmith evaluations:

• Creates a GitHub Actions workflow
• Adds a script for generating evaluation summaries
• Configures PR comments with evaluation results

$ rails generate langsmithrb_rails:privacy

Adds privacy features for LangSmith traces:

• Creates a custom redactor for PII
• Adds a privacy initializer
• Generates a privacy configuration file

$ rails generate langsmithrb_rails:demo

Adds a demo application with LangSmith tracing:

• Creates a chat interface with LLM integration
• Adds a service for interacting with LLMs
• Configures tracing for all LLM operations

Once configured, you can trace your code using the LangsmithTraced concern:

class MyService
  include LangsmithTraced
  
  def process(input)
    langsmith_trace("my_operation", inputs: { input: input }) do |run|
      # Your code here
      result = do_something(input)
      
      # Record the output
      run.outputs = { result: result }
      
      result
    end
  end
end

For background jobs, use the LangsmithTracedJob concern:

class MyJob < ApplicationJob
  include LangsmithTracedJob
  
  def perform(args)
    # Job is automatically traced
    # ...
  end
end

If you've set up buffering, traces will be stored locally and sent in batches. You can manually flush the buffer:

$ rails langsmith:flush

To run an evaluation:

$ rails langsmith:eval[sample,http,my_experiment]

Where:

• sample is the dataset name
• http is the target name
• my_experiment is the experiment name

To compare two experiments:

$ rails langsmith:compare[exp_a,exp_b]

After checking out the repo, run bin/setup to install dependencies. Then, run rake spec to run the tests. You can also run bin/console for an interactive prompt that will allow you to experiment.

To install this gem onto your local machine, run bundle exec rake install.

Bug reports and pull requests are welcome on GitHub at https://github.com/protocolgrid/langsmithrb_rails.

The gem is available as open source under the terms of the MIT License.