Project

rspec-llm

0.0
The project is in a healthy, maintained state
rspec-llmwashu/rspec-llmHomepageDocumentationSource CodeBug TrackerWiki
rspec-llm adds first-class RSpec support for testing Large Language Model interactions. Ships LLM-as-judge matchers, JSON Schema validation, semantic similarity matchers, a programmable fake adapter, and a thin DSL for batch evaluations. Works with the ruby_llm and langchainrb gems out of the box.
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
2015
2016
2017
2018
2019
2020
2021
2022
2023
2024
2025
2026
 Popularity
Downloads
512
Stars
0
Forks
0
Watchers
0
 Releases
Current version
0.2.0
Total releases
2
First release
2026-06-04
Latest release
2026-06-05
 Issues
Open Issues
0
Closed Issues
1
Total Issues
1
Issue Closure Rate
100%
 Development
Primary Language
Ruby
Licenses
MIT
Average date of last 50 commits
2026-06-05
Reverse Dependencies
0

 Dependencies

Runtime

json-schema
~> 4.0
rspec-core
~> 3.0
rspec-expectations
~> 3.0
 Project Readme

rspec-llm

CI Coverage Gem Version Gem Downloads

RSpec matchers, helpers, and a thin DSL for testing LLM-backed code in Ruby.

rspec-llm provides:

  • Custom matchers for asserting on LLM outputs:
    • pass_llm_judge("criterion") — LLM-as-judge boolean evaluation
    • match_llm_intent("the response is a polite apology") — judge framed as intent matching
    • match_json_schema(schema) — JSON Schema validation
    • be_semantically_similar_to("reference text") — cosine similarity over embeddings
  • Adapters for the ruby_llm and langchainrb gems, plus a programmable in-memory fake for hermetic unit specs.
  • A describe_llm / evaluate DSL for running batches of prompt → expectation pairs.
  • VCR-friendly: deterministic playback is delegated to your existing VCR config (see VCR pattern).

Installation

bundle add rspec-llm

You only need to install one of ruby_llm or langchainrb (or both) depending on which adapter you intend to use. rspec-llm doesn't pull either in as a hard dependency.

bundle add ruby_llm        # if you use ruby_llm
bundle add langchainrb     # if you use langchainrb

Configuration

Configure once in spec/spec_helper.rb:

require "rspec/llm"
require "ruby_llm"

RSpec::LLM.configure do |c|
  c.client   = RubyLLM.chat(model: "claude-sonnet-4-6")
  c.judge    = RubyLLM.chat(model: "claude-haiku-4-5")
  c.embedder = ->(text) { RubyLLM.embed(text).vectors }
  c.similarity_threshold = 0.8
end

With langchainrb it looks like:

require "rspec/llm"
require "langchain"

llm = Langchain::LLM::OpenAI.new(api_key: ENV.fetch("OPENAI_API_KEY"))

RSpec::LLM.configure do |c|
  c.client   = llm
  c.judge    = llm
  c.embedder = ->(text) { llm.embed(text: text).embedding }
end

Matchers

RSpec.describe "Summarizer" do
  let(:response) { MyApp.summarize(article) }

  it { expect(response).to pass_llm_judge("contains a single-sentence summary") }
  it { expect(response).to match_llm_intent("a summary of the article") }
  it { expect(response).to match_json_schema(MyApp::SUMMARY_SCHEMA) }
  it { expect(response).to be_semantically_similar_to(expected_gist).within(0.85) }
end

pass_llm_judge

Sends the response and a criterion to the configured judge model. When the ruby_llm gem is loaded, the judge uses a RubyLLM::Schema-backed structured contract that forces the model to return { passed: boolean, reason: string } — eliminating brittle first-token YES/NO parsing and always surfacing a rich explanation in the failure message. For all other adapters the matcher falls back to parsing the first YES/NO token transparently.

expect(reply).to pass_llm_judge("is polite and apologetic")

To use a different judge for a single matcher, chain .using(some_client).

match_llm_intent

Same machinery as pass_llm_judge, framed as intent matching — useful when the "criterion" is naturally a description of what the response should say.

expect(reply).to match_llm_intent("a refund confirmation for order #12345")

match_json_schema

Parses the actual value as JSON (or accepts a Hash/Array directly) and validates against a schema via the json-schema gem. The schema argument can be:

A raw JSON Schema hash (original behaviour — fully backward-compatible):

schema = {
  "type" => "object",
  "required" => ["summary"],
  "properties" => { "summary" => { "type" => "string" } }
}
expect(response).to match_json_schema(schema)

A Ruby class — Data.define, Struct, or any PORO with attr_accessor. The matcher introspects the class and derives the required fields automatically:

# Data.define (Ruby >= 3.2)
UserProfile = Data.define(:full_name, :verified_email)
expect(response).to match_json_schema(UserProfile)

# Struct
Point = Struct.new(:x, :y)
expect(response).to match_json_schema(Point)

# PORO
class OrderSummary
  attr_accessor :order_id, :total, :status
end
expect(response).to match_json_schema(OrderSummary)

be_semantically_similar_to

Embeds both sides via the configured embedder, computes cosine similarity, and compares to the threshold. Override the threshold per-matcher with .within(0.9).

expect(response).to be_semantically_similar_to("the cat sat on the mat").within(0.9)

Fake adapter

For fast, hermetic unit tests, stub the client with the built-in fake:

RSpec.describe "Greeter" do
  it "returns the canned greeting" do
    stub_llm do |fake|
      fake.respond_to("Say hi to Alice").with("Hi, Alice!")
      fake.respond_to_pattern(/^Say hi to/).with { |prompt| "Hi! (from #{prompt})" }
      fake.default("…")
    end

    expect(MyApp.greet("Alice")).to eq("Hi, Alice!")
  end
end

Use stub_llm_judge to stub the judge model separately — handy when testing your own code that wraps pass_llm_judge. Stub with a JSON string to exercise the structured-output path, or with a YES/NO string to exercise the text-parsing fallback:

# Structured output (recommended — matches real ruby_llm behaviour)
stub_llm_judge do |fake|
  fake.default('{"passed":true,"reason":"Looks good to me."}')
end

# Legacy text format (still works for backward compatibility)
stub_llm_judge do |fake|
  fake.default("YES\nLooks good to me.")
end

DSL

describe_llm is a thin wrapper around RSpec.describe that adds an evaluate group helper:

RSpec.describe_llm "Summarizer evals" do
  evaluate "single sentence",
    prompt: "Summarize: #{ARTICLE}",
    expect: [
      pass_llm_judge("is one sentence"),
      match_llm_intent("a summary of the article")
    ]
end

Each evaluate call defines one RSpec example: it calls the configured client with the prompt and applies every matcher in expect:.

VCR pattern

For integration tests against real APIs, record once and replay deterministically using VCR + WebMock:

# spec/spec_helper.rb
require "vcr"

VCR.configure do |c|
  c.cassette_library_dir = "spec/cassettes"
  c.hook_into :webmock
  c.filter_sensitive_data("<OPENAI_API_KEY>") { ENV["OPENAI_API_KEY"] }
  c.configure_rspec_metadata!
end

RSpec.describe "Summarizer", vcr: true do
  it "summarizes" do
    expect(MyApp.summarize(article)).to pass_llm_judge("is concise")
  end
end

Development

bin/setup            # install deps
bundle exec rspec    # run tests
bundle exec rubocop  # lint
bin/console          # interactive prompt

Contributing

Bug reports and pull requests welcome on GitHub at https://github.com/washu/rspec-llm.

License

MIT — see LICENSE.txt.