branchout
branchout runs a Ruby method on many inputs, shows the branch arms that no input took, and writes RSpec or minitest examples that pin what the method does now.
Quick start in your Rails app
These steps add branchout to a Rails app that tests with RSpec and factories, and explore one model method inside the suite, with the app booted and the test database.
Pick a first target: a model method with two to five if, unless, or case arms that returns a value, on a model that has a factory. A method that a recent commit changed is a good start. git log -5 --oneline -- app/models lists those commits.
- Add the gem to the test group of your Gemfile, and install it:
group :test do
gem "branchout", require: false
endbundle install- Write one spec file that explores the method. Replace
Loan,approve, the factory call, the argument, and the two file names with yours. The receiver and each argument are Ruby source in a String:
# spec/models/loan_explore_spec.rb
require "branchout/coverage" if ENV["BRANCHOUT"]
require "rails_helper"
require "branchout"
RSpec.describe Loan, type: :model do
it "explores approve", if: ENV["BRANCHOUT"] do
clerk = {ruby: "{name: 'Grace', senior: false}"}
puts Branchout.explore("Loan#approve", within: self, emit_tests: "spec/models/loan_approve_spec.rb", inputs: [
{receiver: {ruby: "create(:loan)"}, args: [clerk]}
])
end
endThe first line starts coverage before rails_helper boots the app, so that branchout can count the arms of the model's file. A helper that starts SimpleCov can stay as it is: branchout takes over coverage for this run. Without BRANCHOUT the example is skipped, so the file can stay in the suite. Each input runs in a savepoint that is rolled back, so no input sees the rows of another. A bare create or build needs FactoryBot::Syntax::Methods included in the suite's configuration; without it, write FactoryBot.create(:loan). An input whose own Ruby raises, before the method runs, stops explore with a message that names the input, so its error is never pinned as the method's.
- Run it:
BRANCHOUT=1 bundle exec rspec spec/models/loan_explore_spec.rbWhat you will see
This output is from a small sample app, where the method is:
def approve(officer)
return :already if state == "approved"
return :over_limit if borrower.approved_cents + amount_cents > borrower.limit_cents
return :needs_senior if amount_cents > 10_000 && !officer.fetch(:senior)
update!(state: "approved", approved_by: officer.fetch(:name))
:approved
endbranchout explore app/models/loan.rb Loan#approve
covered: 3 of 6 arms by 1 input
raised: none
tests: spec/models/loan_approve_spec.rb
FRONTIER (add an input like the nearest above each arm, changed as the arm says)
nearest: {"args":[{"ruby":"{name: 'Grace', senior: false}"}],"receiver":{"ruby":"create(:loan)"}}
app/models/loan.rb:7:4 makes `state == "approved"` true
app/models/loan.rb:8:4 makes `borrower.approved_cents + amount_cents > borrower.limit_cents` true
app/models/loan.rb:9:4 makes `amount_cents > 10_000 && !officer.fetch(:senior)` true
RETURNED (coverage does not check a value; compare each with what the method should return)
returns :approved for inputs[0]
OPERATORS (coverage does not see which side ran; vary each left side)
app/models/loan.rb:9:50 && in `amount_cents > 10_000 && !officer.fetch(:senior)`
BOUNDARIES (each comparison in the counted code, listed whether or not an input made both sides equal; coverage does not see which did)
app/models/loan.rb:8:65 > in `borrower.approved_cents + amount_cents > borrower.limit_cents`
app/models/loan.rb:9:41 > in `amount_cents > 10_000`
Pass verbose: true for every covered arm and the run settings.
.
Finished in 0.28375 seconds (files took 2.14 seconds to load)
1 example, 0 failures
The one input took 3 of the 6 arms. Each line under FRONTIER is an arm that no input took, at its file, line, and column, and says what an input must do to take it. nearest is the input that came closest, so copy it and change it as the line says. OPERATORS names an && whose sides coverage does not count, so give inputs that make its left side true and false. BOUNDARIES names each <, <=, >, and >=: every arm can be taken while no input makes the two sides equal, the one value that tells > from >=, so give an input that does. The list comes from the source, so a comparison stays on it after an input made its sides equal. The run also wrote spec/models/loan_approve_spec.rb, which pins what each input returned.
Next: close the frontier
Add one input for each FRONTIER entry, and run step 3 again. In the sample app a borrower's limit is 20,000 cents:
# spec/models/loan_explore_spec.rb
require "branchout/coverage" if ENV["BRANCHOUT"]
require "rails_helper"
require "branchout"
RSpec.describe Loan, type: :model do
it "explores approve", if: ENV["BRANCHOUT"] do
clerk = {ruby: "{name: 'Grace', senior: false}"}
puts Branchout.explore("Loan#approve", within: self, emit_tests: "spec/models/loan_approve_spec.rb", inputs: [
{receiver: {ruby: "create(:loan)"}, args: [clerk]},
{receiver: {ruby: "create(:loan, :approved)"}, args: [clerk]},
{receiver: {ruby: "create(:loan, amount_cents: 25_000)"}, args: [clerk]},
{receiver: {ruby: "create(:loan, amount_cents: 15_000)"}, args: [clerk]}
])
end
endBRANCHOUT=1 bundle exec rspec spec/models/loan_explore_spec.rbbranchout explore app/models/loan.rb Loan#approve
covered: 6 of 6 arms by 4 inputs
raised: none
tests: spec/models/loan_approve_spec.rb
FRONTIER
(none: every arm ran, which does not mean each value is right; check RETURNED)
RETURNED (coverage does not check a value; compare each with what the method should return)
returns :approved for inputs[0]
returns :already for inputs[1]
returns :needs_senior for inputs[3]
returns :over_limit for inputs[2]
OPERATORS (coverage does not see which side ran; vary each left side)
app/models/loan.rb:9:50 && in `amount_cents > 10_000 && !officer.fetch(:senior)`
BOUNDARIES (each comparison in the counted code, listed whether or not an input made both sides equal; coverage does not see which did)
app/models/loan.rb:8:65 > in `borrower.approved_cents + amount_cents > borrower.limit_cents`
app/models/loan.rb:9:41 > in `amount_cents > 10_000`
Pass verbose: true for every covered arm and the run settings.
.
Finished in 0.35302 seconds (files took 0.98271 seconds to load)
1 example, 0 failures
Every arm is covered. That says only that each arm ran, so read each RETURNED line against what the method should do: here each loan gets the answer its state and amount call for. The emitted spec runs as any other spec, without BRANCHOUT:
bundle exec rspec spec/models/loan_approve_spec.rb --format documentationLoan
#approve
with create(:loan)
returns :approved
with create(:loan, :approved)
returns :already
with create(:loan, amount_cents: 15_000)
returns :needs_senior
with create(:loan, amount_cents: 25_000)
returns :over_limit
Finished in 0.04388 seconds (files took 0.93715 seconds to load)
4 examples, 0 failures
Commit the emitted spec: it fails when what the method returns changes. Keep the explore spec for the next round, or delete it. A coding agent runs the same loop: it reads FRONTIER, appends an input, and reruns. skills/branchout/SKILL.md is that procedure, and Inside a test suite has every option of Branchout.explore.
Try it without an app
The CLI explores a method that loads without an app, such as a module under lib/, from its file.
Install the gem, and RSpec to run what it writes:
gem install branchout rspecIn an empty folder, write a method with a few branches:
cat > shipping.rb <<'RUBY'
module Shipping
def self.fee(weight, express: false)
raise ArgumentError, "weight must be positive" unless weight.positive?
base = weight <= 2 ? 5 : 5 + (weight - 2) * 2
express ? base * 2 : base
end
end
RUBYExplore it and write the specs:
branchout explore shipping.rb Shipping.fee --emit-tests shipping_spec.rbbranchout explore shipping.rb Shipping.fee
covered: 5 of 6 arms by 13 inputs
raised: 3 inputs
tests: shipping_spec.rb
FRONTIER (add an input like the nearest above each arm, changed as the arm says)
nearest: [1]
shipping.rb:6:14 makes `express` true
RETURNED (coverage does not check a value; compare each with what the method should return)
returns 5 for [1]
returns 7 for [3]
EXCEPTIONS
ArgumentError at shipping.rb:3 raised by 3 inputs, such as [-1]: weight must be positive
BOUNDARIES (each comparison in the counted code, listed whether or not an input made both sides equal; coverage does not see which did)
shipping.rb:5:18 <= in `weight <= 2`
Pass --verbose for every covered arm and the run settings.
branchout ran 13 inputs that it made from the method's own literals and from a seeded sample. They took 5 of the 6 arms. FRONTIER is the arm that no input took: express was never true, so add an input like nearest that makes it true. RETURNED shows what each input returned, grouped by value: coverage counts the arms an input took and never checks a value, so read each line against what the method should return. A value longer than a line shows the Hash leaves that tell it from the other values first, then the rest of its top-level scalars that fit (returns a Hash with due_cents: 900, discount: 10, discount_cents: 100, ...). An Array, a Struct, or a Data shows its size or class, then the leaves that tell it from the other values of that size or class (returns an Array of 2 with [1].kind: "skips-callbacks", [0].line: 2, ...), and an input from the inputs file longer than a line shows its place there (inputs[3]). --json gives each whole value as a Ruby literal (value) and each whole input. A path under the temporary or the home directory shows as <tmp>/ or <home>/ with .../ for its directories ("No hit under <tmp>/.../app."), in value too, so the report is the same on every machine. EXCEPTIONS gives one line for each error and the place that raised it, with how many inputs raised it and one of them. --verbose also prints the input that took each covered arm.
The emitted spec pins each return value and each error:
cat shipping_spec.rb# frozen_string_literal: true
# Generated by branchout 0.2.0. Characterization specs for discovered inputs.
# target: shipping.rb
# entry: Shipping.fee
require_relative "./shipping.rb"
RSpec.describe Shipping do
describe ".fee" do
it "raises ArgumentError when weight is -1" do
# covers:
# line 3: unless weight.positive? -> then
expect { described_class.fee(-1) }.to raise_error(ArgumentError, "weight must be positive")
end
it "raises ArgumentError when weight is -2" do
# covers:
# line 3: unless weight.positive? -> then
expect { described_class.fee(-2) }.to raise_error(ArgumentError, "weight must be positive")
end
it "raises ArgumentError when weight is 0" do
# covers:
# line 3: unless weight.positive? -> then
expect { described_class.fee(0) }.to raise_error(ArgumentError, "weight must be positive")
end
it "returns 5 when weight is 1" do
# covers:
# line 3: unless weight.positive? -> else
# line 5: weight <= 2 ? 5 : 5 + (weight - 2) * 2 -> then
# line 6: express ? base * 2 : base -> else
expect(described_class.fee(1)).to eq(5)
end
it "returns 7 when weight is 3" do
# covers:
# line 3: unless weight.positive? -> else
# line 5: weight <= 2 ? 5 : 5 + (weight - 2) * 2 -> else
# line 6: express ? base * 2 : base -> else
expect(described_class.fee(3)).to eq(7)
end
end
endrspec shipping_spec.rb.....
Finished in 0.00334 seconds (files took 0.07131 seconds to load)
5 examples, 0 failures
Next, close the frontier. Write an input that takes the missing arm into an inputs file, and run the same command with --inputs:
cat > inputs.json <<'JSON'
{"inputs": [{"args": [3], "kwargs": {"express": true}}]}
JSON
branchout explore shipping.rb Shipping.fee --inputs inputs.json --emit-tests shipping_spec.rbbranchout explore shipping.rb Shipping.fee
covered: 6 of 6 arms by 14 inputs
raised: 3 inputs
tests: shipping_spec.rb
FRONTIER
(none: every arm ran, which does not mean each value is right; check RETURNED)
RETURNED (coverage does not check a value; compare each with what the method should return)
returns 5 for [1]
returns 14 for {"args":[3],"kwargs":{"express":true}}
EXCEPTIONS
ArgumentError at shipping.rb:3 raised by 3 inputs, such as [-1]: weight must be positive
BOUNDARIES (each comparison in the counted code, listed whether or not an input made both sides equal; coverage does not see which did)
shipping.rb:5:18 <= in `weight <= 2`
Pass --verbose for every covered arm and the run settings.
rspec shipping_spec.rb --format documentationShipping
.fee
raises ArgumentError when weight is -1
raises ArgumentError when weight is -2
raises ArgumentError when weight is 0
returns 5 when weight is 1
returns 14 when weight is 3 and express is true
Finished in 0.00228 seconds (files took 0.07051 seconds to load)
5 examples, 0 failures
Every arm is covered now. The emitted spec keeps each input of the inputs file, and one input for each other arm and each error. Keep the spec beside the code: it fails when the method's behavior changes. A coding agent runs the same loop: it reads FRONTIER, appends an input, and reruns. skills/branchout/SKILL.md is that procedure.
CLI reference
A coding agent adding tests can see that a branch was missed and still not know which comparison failed, or which input came closest. branchout prints that frontier: the condition, the direction that selects the arm, and the nearest input that reached the decision. The agent appends an input and reruns the same command. branchout does not edit the inputs file. When the search stops, it writes a minitest or RSpec example that pins each discovered return value or exception.
The method follows concolic testing (DART, CUTE) and LLM-guided concolic work such as ConcoLLMic. branchout is inspired by concolic testing but does no symbolic solving. The agent proposes inputs from the branch frontier.
From a checkout of this repository, ruby bin/branchout runs the same command as branchout. The seed file holds one input, [0]. --verbose prints the whole report: the run settings of the strategies that ran and the counts, then COVERED (the input that took each arm), FRONTIER (each arm no input took, with path, the arms nearest took), RETURNED (each value and the inputs that returned it), EXCEPTIONS (one record for each input that raised), AFTER when an input has an after (each value it read and the inputs that read it), NOT COUNTED, OPERATORS, and BOUNDARIES. Without it, the report keeps the counts, the FRONTIER arms that an input reached, one line each under the nearest input to change, one line for each decision that no input reached, one line for each returned value, and one line for each error and the place that raised it, as in Try it without an app. --json holds every field of the whole report.
ruby bin/branchout explore examples/magic_value.rb MagicValue.classify \
--inputs examples/inputs/magic_value.seed.json \
--strategy inputs \
--random 0 \
--seed 1 \
--verbosebranchout explore
target: examples/magic_value.rb
entry: MagicValue.classify
strategies: inputs
timeout: 1
inputs_ran: 1
executions: 1
arms_covered: 1
arms_total: 2
exceptions: 0
callees_not_counted: 0
operators_not_counted: 0
boundaries_not_counted: 0
COVERED
id: if@5:4/else@8:6
loc: examples/magic_value.rb:8:6
cond: x == 114514
sense: cond-false
input: [0]
FRONTIER
id: if@5:4/then@6:6
loc: examples/magic_value.rb:6:6
cond: x == 114514
sense: cond-true
parent: reached
nearest: [0]
path: if@5:4/else@8:6
RETURNED
returns: :ordinary
input: [0]
EXCEPTIONS
(none)
NOT COUNTED
(none)
OPERATORS
(none)
BOUNDARIES
(none)
sense: cond-true on x == 114514 is the arm [0] did not take. Append [114514] and rerun the same command. examples/inputs/magic_value.json is that file. It reports arms_covered: 2 and FRONTIER (none).
ENTRY is Const.method or Const#method. A method under module_function or in a module that calls extend self is Const.method. For Const#method, each input gets its own receiver from Const.new. If initialize takes arguments, put them in new on each input, in the same shape as an input: {"args": [5], "new": [10]} calls Const.new(10).method(5). A new of keywords only may leave out args: {"args": [5], "new": {"kwargs": {"main": 1}}} calls Const.new(main: 1).method(5). When the state the method reads comes from calls after new, list them in setup, in order: {"args": [], "setup": [{"method": "add", "args": ["apple", 3]}]} calls add("apple", 3) on the new receiver and then the method. A step takes method, args, kwargs, and block ({"method": "start", "args": ["r1"], "block": "{ hold.pause(:read); :end }"}), and with setup steps the receiver takes the name the emitted test gives it, so a step's arguments and block can read it. Each step takes method, args, and kwargs. Coverage counts only the call to the method, not new or setup. When the target file defines initialize, mutate and random fill new from its parameters too. --entry ENTRY may replace the positional entry.
While searching, pass --strategy inputs --random 0 and keep --seed fixed, so a new covered arm comes from an input you added. The search procedure is in skills/branchout/SKILL.md.
| Flag | Default | Meaning |
|---|---|---|
--inputs PATH |
none | JSON file of inputs |
--require PATH |
none | Load PATH before the target, for a target that needs the rest of its library. Repeatable. PATH's directory joins the end of the load path, after the gems, so lib/name.rb can require "name/part", and config/puma.rb beside config/environment.rb does not hide the puma gem. A PATH that names no file under the working directory is a feature, as ruby -r reads it: --require shop -I lib loads lib/shop.rb, and a name that no -I directory holds is looked up on the load path. Emitted tests require PATH too |
-I DIR |
none | Put DIR on the load path while exploring, as ruby -I does. Repeatable. With --require-by-name, a file under DIR is required by its name from DIR, such as a Rails model under app/models
|
--seed N |
0 |
RNG seed |
--random N |
16 |
Number of random samples |
--mutate-cap N |
256 |
Literal-mutation budget |
--strategy LIST |
inputs,mutate,random |
Comma-separated strategies, in the order below |
--timeout SEC |
1 |
Per-input timeout. Must be > 0 |
--verbose |
off | Print the whole text report: the run settings, every covered arm, and each frontier arm's path. explore only. Refused with --json, which always holds all of it |
--json |
off | Print JSON on stdout instead of the text report |
--json-out PATH |
Write the JSON report | |
--md-out PATH |
Write the text report that --verbose prints, as Markdown |
|
--emit-tests PATH |
Write characterization tests. explore only |
|
--format NAME |
minitest |
minitest or rspec. Requires --emit-tests. If omitted, a path ending in _spec.rb selects RSpec |
--pin-keys LIST |
Comma list of keys. A returned Hash pins only these keys (slice in minitest, include in RSpec), so a key added later fails no test. Requires --emit-tests
|
|
--require-by-name |
off | Emitted tests require the --require files and the target through the load path, named from the first -I directory or --require file's directory that holds each (require "shop/cart"). With neither, named from the nearest lib directory above the target, or the target's own directory. A file that no such directory holds is required by path, with a comment above it and a line on stderr that say so. A # Load path: -I lib comment above the requires names each directory the test needs on the load path; an RSpec file leaves out spec and lib, which rspec adds itself. Requires --emit-tests. A test emitted under a .jitcatch directory gets this without the flag: jitcatch copies it into each checkout, and its rerun line loads it where it was written, so a require_relative would load another file from each place |
--entry ENTRY |
the positional argument |
Const.method or Const#method
|
| Strategy | What it runs |
|---|---|
inputs |
Entries in the inputs file. Duplicate canonical JSON is dropped |
mutate |
Each source literal, tried in each argument slot. Integers also include 0, 1, -1, and literal±1. Floats also include 0.0, 1.0, -1.0, and ±1. Strings also include "", "a", the literal without its last character, and the literal plus "1". When the first two positional parameters are numeric, those slots also get a cartesian product, cut by --mutate-cap. String literals are not concatenated. The literals are those of the entry and of each method it calls in the same file, and a constant the file sets to a literal (HOLD = "sanctions") counts as that literal. A slot's type comes from the same methods: an argument passed to a helper takes the type the helper's use of its parameter gives (held?(reason) with def held?(text) = text == HOLD makes reason a String), and doc["key"] or doc.dig(:key) makes doc a Hash |
random |
Seeded Park-Miller samples. Integers are -3..12. Strings have length 0..3 over abc01.-
|
ruby bin/branchout baselines examples/magic_value.rb MagicValue.classify \
--inputs examples/inputs/magic_value.json \
--seed 1 --random 40baselines runs random, mutate, and agent (the inputs file only) as separate rows. The random and mutate rows do not run the inputs file. On this target, random executes 15 inputs and covers 1 of 2 arms, mutate executes 6 and covers both, and agent executes 2 and covers both.
An inputs file starts as one seed:
{
"inputs": [
{"args": [0]}
]
}An entry may be a bare array. Keywords go in kwargs, constructor arguments go in new, and calls that build the receiver's state go in setup. A receiver that new cannot build, such as a saved record, is Ruby in receiver: {"receiver": {"ruby": "Factories.loan(3)"}, "args": [{"ruby": "loan.amount"}]}. The receiver takes the name the emitted test gives it, the class in snake case, so the arguments, setup, and block can read from it. Ruby in before runs before the receiver and the arguments, in the input's own savepoint, and makes what the method reads, such as the rows a class method looks up: {"before": {"ruby": "FactoryBot.create(:part, sku: \"A\")"}, "args": ["A"]}. RSpec writes it as the before of the input's context, and minitest as the line before the call, each line of it indented except the body and the terminator of a << or <<- heredoc, which keep their columns so that the heredoc ends and its text stays the same. A before of several lines is named by the lines that differ from its siblings', a call by its keywords that differ (after FactoryBot.create(:membership) with enabled: false). A Ruby argument that runs a block through tap or then is named by its block when its siblings differ there (when fact.tap { Parts.rows.replace(["A"]) }), and so is a before that does (after FactoryBot.create(:part).tap { |p| p.update!(size: 1) }). A before named by a call whose label would pass 80 characters keeps the fewest of its keywords and its block that tell it from each sibling (after FactoryBot.create(:part) with code: "B"). A block over 30 characters in a text that would still pass 80 is cut where it parts from its siblings' blocks (when fact.tap { ...long piece', 2]) }). An entry with new, receiver, setup, kwargs, block, before, or after may leave out args for a call with no positional argument: {"receiver": {"ruby": "FactoryBot.build(:user)"}}. If type inference is wrong, set "types" to one of int, float, string, array, hash, or bool per positional parameter. A default value of [], {}, true, or false gives its type, and bool makes mutation and random sampling try both false and true. Other top-level keys, such as meta, are ignored. A note on an entry keeps the reason for that input beside it and takes no part in the call; it names the input: RETURNED shows a long input by its place and its note (inputs[9] (one save)), --json keeps it on the input, a minitest name is made from its words (test_one_save_returns_array_of_1), and an RSpec example says it after the outcome (returns 1 (one save)). A note with no Latin letter or digit gives a minitest name no words, so that input is named by its values. An entry's pin_keys ({"args": [3], "pin_keys": ["tax"]}) replaces --pin-keys for that input's test, and "pin_keys": [] pins its Hash whole. A value JSON cannot write, such as a record or an exception, is Ruby in {"ruby": "..."}, often a factory in a --require file: {"args": [{"ruby": "Factories.loan(3)"}]}. Each call evaluates it again at the top level, and the emitted tests write it as given (Lending.approve(Factories.loan(3))). A block the method takes goes in block, with braces: {"args": [5], "block": "{ |amount| amount > 2 }"}. A Hash argument whose one key is "ruby" is written as Ruby too: {"ruby": "{\"ruby\" => \"3.4\"}"}. A String that names a path under the emitted test's project (the nearest directory above the test with a Gemfile, a gemspec, or .git) is written from the test's own directory, File.expand_path("../fixtures/files/rates.csv", __dir__), so the test passes in any checkout of the project, a jitcatch copy too. Its name and description say the path from the project root (spec/fixtures/files/rates.csv). A path under the temporary or the home directory and outside the project stays as written, and stderr names it: one that exists, since the test passes only on this machine, and one that does not, since the test still holds this machine's directories. Put such a path under the project, for example in spec/fixtures/files. A returned value or an error message that holds a path under the project names the project root from the test's own directory too ("No such file or directory @ rb_sysopen - #{File.expand_path("..", __dir__)}/spec/fixtures/none.csv"). One that holds a path of this machine outside the project stays as written, and stderr names it. A test name or description says a path under the temporary or the home directory by its file alone, so it is the same on every machine. Append entries. Do not delete or reorder them.
For a method that changes state, the value is half of what it does. after reads the other half: one Ruby expression that runs once the call returned or raised, in the input's scope, so it reads the receiver by the name the emitted test gives it. It goes on an entry, or at the top of the file for every input, where an entry's own after replaces it and the inputs that mutate and random make take it too. Inside a suite it runs before the input's savepoint is rolled back. In a checkout of quint_driver, this file explores QuintDriver::Hold#start, which runs a block until the block pauses and holds its name until then:
{
"after": {"ruby": "hold.held"},
"inputs": [
{"note": "pauses", "args": ["r1"], "block": "{ hold.pause(:read); :written }"},
{"note": "ends without a pause", "args": ["r1"], "block": "{ :written }"},
{"note": "raises before a pause", "args": ["r1"], "block": "{ raise \"boom\" }"},
{"note": "name already held", "args": ["r1"], "block": "{ :again }",
"setup": [{"method": "start", "args": ["r1"], "block": "{ hold.pause(:read) }"}]}
]
}branchout explore lib/quint_driver/hold.rb 'QuintDriver::Hold#start' --inputs hold.json \
--strategy inputs --random 0 --seed 1 --emit-tests test/hold_start_test.rbbranchout explore lib/quint_driver/hold.rb QuintDriver::Hold#start
covered: 5 of 6 arms by 4 inputs
raised: 2 inputs
tests: test/hold_start_test.rb
FRONTIER (add an input like the nearest above each arm, changed as the arm says)
nearest: {"after":{"ruby":"hold.held"},"args":["r1"],"block":"{ :written }"}
lib/quint_driver/hold.rb:71:10 makes `%i[paused done].include?(kind) && answer.is_a?(Array)` false
RETURNED (coverage does not check a value; compare each with what the method should return)
returns :written for inputs[1] (ends without a pause)
returns :read for inputs[0] (pauses)
EXCEPTIONS
ArgumentError at lib/quint_driver/hold.rb:32 raised by {"after":{"ruby":"hold.held"},"args":["r1"],"block":"{ :again }","setup":[{"args":["r1"],"block":"{ hold.pause(:read) }","method":"start"}]}: quint: "r1" is already held; finish it first
RuntimeError at lib/quint_driver/hold.rb:34 raised by {"after":{"ruby":"hold.held"},"args":["r1"],"block":"{ raise \"boom\" }"}: boom
AFTER (what each after read once its call returned or raised; compare each with the state the call should leave)
hold.held is ["r1"] for inputs[3] (name already held), inputs[0] (pauses)
hold.held is [] for inputs[1] (ends without a pause), inputs[2] (raises before a pause)
OPERATORS (coverage does not see which side ran; vary each left side)
lib/quint_driver/hold.rb:70:46 && in `%i[paused done].include?(kind) && answer.is_a?(Array)`
Pass --verbose for every covered arm and the run settings.
The emitted test pins what the after read as it pins a value, after the value or the error. For a call that raised, the receiver is built before assert_raises, so the after can read it:
def test_raises_before_a_pause_raises_runtime_error
# covers:
# line 32: if held?(name) -> else
hold = QuintDriver::Hold.new
error = assert_raises(RuntimeError) do
hold.start("r1") { raise "boom" }
end
assert_equal "boom", error.message
assert_equal [], hold.held
endRSpec writes expect(hold.held).to eq([]) after expect { hold.start("r1") { raise "boom" } }.to raise_error(RuntimeError, "boom"), on the let the call ran on. Without the rescue in Hold#run that frees the name, start still raises RuntimeError with boom, and only the after fails: Expected: [], Actual: ["r1"]. A value an after reads anew on each run is not pinned, a Hash keeps the keys that were the same on both runs, and stderr names each, as for a returned value. A path in it is written as a returned path is. Read several values as one Array or Hash ([cart.items, cart.total]); a modifier if or two statements are refused. An after that raises stops explore and names the input. When new or a setup call raised, no call left a state, so AFTER says read nothing for that input and its test pins the error as before. --json lists each value under after, with after, is, value, and inputs.
ruby bin/branchout explore examples/hidden_bug.rb HiddenBug.scale \
--inputs examples/inputs/hidden_bug.json \
--strategy inputs --random 0 --seed 1 \
--emit-tests spec/generated/hidden_bug_spec.rb \
--format rspecThat run covers 4 of 4 arms and records ZeroDivisionError (divided by 0) for [24,12]. The emitted file pins each return value, or the exception class and message, a message over several lines as it was raised. Each test starts with a # covers: list of the arms its input took, one per line, as the condition's line, the condition as written, and the arm (line 20: unless rate.is_a?(Numeric) -> else, line 9: elsif tag.start_with?("beta") -> then). A ternary is written whole (line 10: label ? :noted : :plain -> then), and a case arm sits on its own when, in, or else line, which names it (line 6: when 1, 2 then :low). A NameError or NoMethodError is pinned by its class and missing name (error.name), because Ruby 3.4 reworded their messages. A message that quotes a backtrace frame (lib/x.rb:3:in 'Box.run') is matched by a Regexp that accepts the frame's label in either form, because Ruby 3.4 changed labels from `run' to 'Box.run'. So is a returned String that quotes one, such as a raised in: line. An Errno message that names a file descriptor (Is a directory @ io_fillbuf - fd:5 ...) is matched by a Regexp that accepts any descriptor, because its number depends on what the process opened before. A Set is pinned by its members (Set[1, 2]), and a Date by its value (Date.new(2026, 4, 1)). A Struct or Data whose members all have a literal is pinned by its class and members (eq(Rule.new(action: "borrow", on: Date.new(2026, 4, 1)))), since == compares the members; one whose own initialize makes another value from those members is not. A return value with no Ruby literal, such as an Object, a Time, or a record, is pinned by its class only. In RSpec a Struct or Data that holds such a value is pinned by its class and members, and an Array or Hash that holds one item by item (match([a_kind_of(Rule).and(have_attributes(action: "borrow", at: a_kind_of(Time)))])); minitest checks the same parts: each part with no literal by its class (assert_kind_of Time, value[0].at), a Struct or Data by its class and members, and an Array or Hash by comparing it with a literal that reads those parts back from the value (assert_equal([value[0]], value)), so a missing key or item fails too. A Hash with a key that has no literal is pinned by its class in both formats. An expected Hash keeps the value's own key order, so a failure diff lines up key by key. A private or protected entry, and a private new, are called through send, as explore called them. A String argument is a plain literal, which the file's frozen_string_literal freezes, and explore passes it frozen as well. An input that raises FrozenError with frozen Strings, and not with unfrozen ones, keeps the unfrozen run: its test writes each String as +"..." under a comment that says so. That run came second, so it saw what the first run changed before it raised: a savepoint undoes rows, but no run undoes a global, a class variable, or a file. The comment says this too. Run the emitted test on its own to check that value. A class or module that private_constant hides is reached through const_get (Shop.const_get(:Ledger)), and the RSpec describe names it as a string. A minitest test is named after its input and outcome, such as test_24_12_raises_zero_division_error. Inputs whose names would be the same are named by the values that differ between them, such as test_status_timeout_and_error_class_timeout_returns_ok, so adding an input can rename those. Strings that share their start are named from the words where they part. Ruby source that shares its start, such as two factory calls in receiver, keeps the word before they part (test_limit_100_returns_ok). A name is cut between words, a blank string shows as blank, an empty Array or Hash argument as empty_array or empty_hash, and a Hash argument is named by its values. A collection in a Hash argument is named by its key and size (groups_3, tags_empty), and an Array of collections by its size (array_of_2). An Array that holds scalars and collections says array, then each scalar and the keys of each Hash (returns [:ok, a Hash of id, name], returns_array_ok_hash_id_and_name), so ["Done", []] is not named as the String "Done" (returns_array_done_empty_array). A number of more than 15 digits is named by its first three digits and its power of ten (123e46). A key that ends another leaf's path too keeps the keys above it (items_id beside id). A Hash outcome is named by the fewest leaves, nested ones too, that tell its siblings apart (returns_due_cents_0_and_discount_100), the same leaves in both formats, or by its keys when it has no sibling (returns_hash_lines_and_discount_cents). A Hash or an Array that holds a part with no literal, as one that --pin-keys or a varying id leaves may, is named in both formats by its parts that have one: the leaves that tell its siblings apart (returns_due_cents_100, RSpec returns due_cents: 100), else its keys (returns_hash_name_and_start_at, RSpec returns a Hash of name, start_at, end_at) or its size (returns_array_of_2, RSpec returns an Array of 2). Each key after the first comes after and, and a cut never splits a key. An Array of scalars says array ([nil] is returns_array_nil), and a String or a Symbol that reads as nil, true, or false says text (returns_text_nil). A Hash key that reads so says key (returns_key_nil_0 for {"NIL" => 0}). A test emitted outside the target's directory tree requires the target by absolute path, unless --require-by-name is given and an -I or --require directory holds it. When a test requires a file by its path from out of its own project (the nearest directory with a Gemfile, a gemspec, or .git), by absolute path or by a climb into another project, stderr warns that the test runs only where that file sits on this machine, and names the fix. A target that a --require file already loads gets no require of its own.
An RSpec file describes the constant, then .method or #method, and calls the class as described_class. It pins true and false with be(true) and be(false), and nil with be_nil, and a Hash literal in either format has a space inside its braces ({ total: 2 }), as RuboCop and rubocop-rspec expect by default, so a file emitted again needs no autocorrect. The receiver is a let, each setup call a before, and inputs that start with the same receiver, calls, or argument values share a context. A receiver whose new differs from a sibling's only at some keywords is named by those keywords (context "with start_at: Date.new(2026, 5, 1)"), and new takes its keywords in the input's order. An example says its outcome first, then the argument values of its own, so rspec --format documentation reads like a spec. A value inside an argument is named by its path, and an Array of scalars as a whole (items is [1, 2]). A path writes a String key quoted and a Symbol key bare, so a String-keyed Hash such as Rails params does not read as a Symbol lookup: the JSON input [{"sku": "B"}] reads items[0]["sku"] is "B", and the Ruby literal {"ruby": "[{sku: \"B\"}]"} reads items[0].sku is "B". Ruby in {"ruby": ...} that is a plain literal is named by its value in this way, in both formats. A Ruby call with keyword arguments, such as FactoryBot.build(:fact, a: "X", b: "Y"), is named by its keywords as a Hash is (when b is "Y"), so factory calls that share a long start read apart; the call before its keywords is named only when it differs. A Hash outcome writes each key as its literal does (returns "state" => "open", returns due_cents: 0), and a nested leaf under a String key as the literal that holds it, with the leaves under one key in one literal (returns "counts" => {"discarded" => 1}, returns "tests" => [{"file" => "a_spec.rb", "disposition" => "flaky"}]). A context says only the values that tell it from its siblings. No text is longer than 80 characters: a longer example says only the values that tell it apart, by their deepest key (when status is "timeout"), or by the keys above it too when another leaf ends in the same key (when result.status is "timeout"). A String key is named with the argument and the literal that holds the leaf (when doc holds "status" => "timeout", when doc holds "result" => {"status" => "timeout"}), since doc["status"] would name another leaf, and a long String is cut where it parts from its siblings, and ends before a word the cut would split. An example still over 80 characters cuts its outcome, not those values (returns ["require \"shop\"",... when target_path is "app/cart.rb"), and an example that raises keeps the error's class whole and cuts the values after it. An outcome over 40 characters that is a String or an Array of scalars shows its start, then is cut (returns ["require_relative \"../lib/shop/cart...). A text with a double quote is written in single quotes, so it needs no escape, even when a String value in it shows its own backslashes ('returns 1 when x is "say \"hi\""').
Arithmetic
.classify
returns :miss when x is 0, y is 0 and z is 0
returns :miss when x is 114514, y is 0 and z is 0
when x is 138
returns :x_only when y is 0 and z is 0
when y is 143
returns :need_z when z is 0
returns :hit when z is 7
A method that takes no argument in any input becomes the subject. An input that raised keeps its setup calls inside the expect block, so a raise from setup is still pinned. A minitest path uses --format minitest and runs with ruby. An RSpec file needs the rspec gem.
Inside a test suite
Branchout.explore runs from inside a test and returns the report the CLI prints. Quick start in your Rails app shows the Gemfile line and one explore spec. The spec's first line, require "branchout/coverage" if ENV["BRANCHOUT"], sets up Coverage before the helper boots the app, so the models load after it. In minitest it goes before require "test_helper". In a suite without Rails it works the same way.
Run the explore file alone: in a run of the whole suite, another spec file can load the model first. When .rspec requires rails_helper, the app boots before the spec's first line, so a model that an initializer reads, or every model when the test environment eager-loads (config.eager_load, which Rails sets from ENV["CI"]), loads too early, and explore refuses it. Then start Coverage before rspec loads anything: BRANCHOUT=1 RUBYOPT="-rbundler/setup -rbranchout/coverage" bundle exec rspec spec/models/loan_explore_spec.rb. bundle exec reads RUBYOPT before Bundler sets the load path, so -rbundler/setup comes first: without it, -rbranchout/coverage loads an installed branchout instead of the one the Gemfile names, or raises LoadError when the Gemfile pins branchout with path: or git:. rspec -r branchout/coverage does not help, because rspec requires it after the files .rspec names. The refusal names the files rspec required first and that command. When .rspec requires a helper that loads no model before the one that does, such as spec_helper before rails_helper, the refusal also names rspec --options /dev/null -r spec_helper <the explore file>, which leaves out the helpers that loaded the model and the other options of .rspec. Set up before Rails, Coverage measures every gem too. In the sample app of the Quick start, the explore run took about 0.3 seconds longer to boot and about 12 ms longer for each input than with the line in spec/rails_helper.rb, right after the line that loads config/environment. That placement is the faster one when nothing loads a model during boot: the gems are loaded by then, and the models load later, on first use.
The inputs are the entries of an inputs file, or the file's path. Their Ruby runs in the test (within: self), so it reaches the suite's factories, fixtures, and helpers. When ActiveRecord is loaded, each input runs in a savepoint that is rolled back: no input sees the rows another wrote, and the test's rows stay as they were. around: replaces that, as a lambda that takes the input as a block. An input that raises FrozenError with frozen Strings runs again with unfrozen ones only when it ran in Branchout::ROLL_BACK, the default or inside an around: that calls it. With any other around:, such as Branchout::RUN_ONLY, the FrozenError is pinned, because branchout cannot tell whether that around undid the first run's rows. Each input that ran in Branchout::ROLL_BACK runs once more, as each input of the CLI does, so a value made anew on each run, such as a generated UUID id, a factory sequence, or a SecureRandom token, is not pinned: a returned Hash pins only the keys whose values were the same both times (include in RSpec, slice in minitest, and a value with no literal, such as a Time or a record, by its class), a value that differed as a whole gets no test (a file left with none skips one example, pins no input, that says so), an error whose message differed is pinned by its class, and stderr names each such input. The report lists under NOT PINNED each covered arm that only such inputs took, so no emitted test checks it. With --emit-tests (emit_tests:), the report lists under PINNED BY CLASS each part of a returned value that the emitted tests check only by its class, such as a Time or a record, with the first three inputs that returned it (value[:start_at] is a Time: ["A",{"ruby":"Time.at(0)"}]), so you can assert what matters in it by hand. The JSON report (json: true) holds the same parts as pinned_by_class, with or without emit_tests:, as each part's path, class, and every input that returned it. The strategy is inputs only unless strategies: says otherwise, and timeout: is 1 second unless set (nil turns it off). target:, emit_tests:, format:, json:, verbose:, pin_keys:, requires:, seed:, random:, and mutate_cap: mean what the CLI flags mean. In minitest the same call goes in a test of an ActiveSupport::TestCase, run with BRANCHOUT=1 bin/rails test.
The emitted test requires the helper the suite loaded (rails_helper, else test_helper, else spec_helper) by name. A suite that loaded none of them names its helper in requires:, or emit_tests: is refused. requires: takes one path as a String, or a list of paths. A requires: entry that names no file is found on the load path, and one found nowhere stops with requires: "spec/helper" names no file under the working directory or on the load path. It does not require the model, which the app loads. A minitest test inherits from the exploring test's base class, such as ActiveSupport::TestCase. An RSpec file gives its describe the exploring example group's type, such as type: :model. rspec puts the project's spec and lib on the load path itself, so the # Load path: comment of an RSpec file does not name them. Run it as any other test, without BRANCHOUT.
Coverage counts a file only when the file loads after branchout/coverage. A target that loaded earlier is refused, not loaded again, because loading a model again declares its validations a second time. A suite that starts SimpleCov in its helper, before or after branchout/coverage, needs no change: branchout measures the run, and SimpleCov reports 0% for it, because explore clears the counts after each input. An input's Ruby must not read a local of the exploring example, and an input that reads a let or subject is refused: the emitted file has neither, and a let read first inside an input would be made in its rolled-back savepoint and kept for the next input.
The CLI can load the app with RAILS_ENV=test and --require config/environment.rb, but then each input's rows stay in the test database, later inputs see them, and the test's helpers are not there.
How it builds the frontier
branchout loads the --require files and then the target after Coverage.setup(lines: true, branches: true) and discards the counts from loading. A target that a --require file already loaded, with require or with Kernel#load as Rails loads an initializer, is not loaded again. For each input it resumes coverage, calls the method, suspends coverage, and reads Coverage.result. It keeps the entry method and same-file callees reached by a bare call or self.. A class method that builds its own instance, as in def self.apply(doc) = new(doc).apply, is followed into initialize and the instance method it calls, also through tap(&:parse) or then(&:parse).
A method in another file is not counted, so its arms are not in the total, the M of covered: N of M arms (arms_total in --verbose and --json). When a check moves from a subclass's method into a parent class's helper, the total gets smaller while the logic stays the same. NOT COUNTED names each such callee: a method that the counted code calls by a bare call or self. and that Ruby finds in another file, such as a parent class or a mixin, and a method it calls on an object it builds with Const.new, kept in a local (journal = Ledger.new) or in an instance variable that a method of the class sets (@ledger = Ledger.new in initialize). Each entry has callee (Register#required_sku), loc (the file and line of its def), and arms (the branch arms in that def, or unknown when its file loaded before coverage started or the method has no def, as with define_method). Explore the callee as its own entry to cover those arms. Methods of an installed gem or of Ruby's own library are not listed, because an app cannot move a check into one: a Rails model calls dozens of them, such as an association reader or update!. Methods written in C, Ruby's own prelude methods, attribute readers and writers, methods with no arms, and calls on an object whose class the code does not show are not listed.
Branch coverage counts no arm for ||, &&, ||=, or &&=, so it cannot say whether their right side ran. A method that branches only through them, as user&.admin? || params[:force] does past its &., would report fewer arms than it has, or covered: 0 of 0 arms, which reads as fully covered. OPERATORS lists each one in the counted code, and operators_not_counted counts them: loc (the operator's line and column), op, and text (the whole expression, cut at 160 characters). Give inputs that make each left side truthy and falsy. An @ivar ||= or @ivar &&= on an instance variable the file sets nowhere else is left out: it memoizes across calls on one object, and each input builds its own receiver, so no input can vary its left side. When the entry has no arm but has such an operator, stderr says so. When its only branch is such a memo, stderr names the memo and says to explore the methods its right side calls and to test the memo by hand, with two calls on one object. A memo on defined?(@ivar) (return @value if defined?(@value)) has arms that coverage counts, and an input with a setup call of the method covers the cached arm, but that arm returns what the first call stored, so the pinned values read the same without the memo; stderr names each such memo and says to test it the same way. Coverage sees no side of a <, <=, >, or >= either, so every arm can be taken with no input that makes the two sides equal, and only such an input catches > changed to >=. BOUNDARIES lists each comparison in the counted code, in the shape of OPERATORS, and boundaries_not_counted counts them; give an input that makes its sides equal. The list comes from the source, so it does not change when an input does that. &. has arms of its own (receiver-present, receiver-nil).
An arm id is kind@line:column/arm@line:column, from the Coverage branch key. Columns are 0-based. cond is the Prism predicate. For a when or in arm it is that line. If the locations do not match, the source line is used.
The frontier is every arm whose count is 0. parent: reached means some input executed that decision. nearest is the one that reached it with the fewest taken arms, then the smallest weight (absolute numeric size, plus string and container length), then the smallest canonical JSON. path is the sorted set of arms that input took. Coverage does not record order. A loop count greater than 1 is written as *count.
sense is the direction that selects the arm: cond-true (if then, unless else), cond-false (if else, unless then), arm-matched (that when or in), arm-else, body-entered (while or until body), receiver-present, or receiver-nil (&.).
Exceptions are listed apart from arm coverage. A covered arm can still raise. The same inputs, flags, and CRuby produce the same stdout. An input equal to an earlier one runs once, and stderr says how many repeated. The report has no timestamps. Paths are relative to the working directory.
Install
Ruby 3.3 or newer. Exploring uses Coverage, Prism, and json from the standard library. gem install branchout installs the branchout command. In a project with a Gemfile, add gem "branchout", group: :test, require: false instead and run bundle exec branchout. A checkout of this repository works too: run ruby bin/branchout, or put bin on PATH. Emitted minitest files run under ruby. Emitted RSpec files need the rspec gem. Outside a bundle, bin/branchout activates the newest installed prism that every installed gem accepts, so a target that requires minitest 6 (prism ~> 1.5) still loads when a newer prism is installed.
Install the branchout skill with the skills CLI (Node.js 22.20 or later). This repository is private, so these commands need GitHub access.
npx skills add meganemura/branchoutnpx skills add meganemura/branchout --skill branchoutnpx skills add meganemura/branchout -gnpx skills add meganemura/branchout -a cursor--skill selects this skill, -g installs it for every project, and -a targets one agent.
To copy or symlink it instead:
mkdir -p .agents/skills
ln -s /path/to/branchout/skills/branchout .agents/skills/branchout.claude/skills/ and .cursor/skills/ work the same way.
Limits
- The target runs. Its stdout and stderr, at load time and in each call, are discarded. Files, globals, and the network are not sandboxed.
- An input that raised
FrozenErrorand ran again with unfrozen Strings ran in the same process as its first run, after it. A global, a class variable, or a file that the first run changed is not undone. - The target loads into branchout's own process. A target or
--requirefile that loads another checkout of branchout is refused, because it would replace the tool's classes. To explore branchout's own code, run that checkout'sbin/branchout. When an argument names one of its own files, it sets coverage up before it loads itself, so that coverage measures the file without loading it again. A file that loaded before coverage was set up, such as one of Ruby's own library, is refused: loading it again would define its classes a second time, and an object built from the first ones would no longer be one of them. - The entry must be defined in the target file, under its constant or under a superclass declared in the same file. An inherited entry follows the subclass's overrides, as Ruby does.
- A method that depends on time, randomness, or I/O will not produce a stable report.
- A path is a set of arms, not an order.
&&,||,||=, and&&=are not arms;OPERATORSlists them.whileanduntilrecord a body arm; a condition that ran but did not enter the body is recovered from the line count. - A modifier
unlesscan putthenandelseat the same location. The arm name in the id distinguishes them. - Only a bare call, a
self.call, and a call on a barenew(...)in the same file are followed. An Enumerable method called that way, such asselect(&:ok?)in a class that includesEnumerable, reaches theeachthe file defines. A call on another receiver, such asOther.new.check(x), is not. When the entry reaches no arm,explorewarns on stderr; target the method that branches instead.--emit-testsstill pins each input's return value or error. -
Timeout.timeoutdoes not stop every block. If the timeout backtrace has no target line,locis the method definition line. - An unsatisfiable condition stays on the frontier. In
examples/arithmetic.rb,x == 114514 && x == 0is one. - The random and mutation domains above are narrow on purpose.
- Coverage is process-global. Run the CLI in its own process.
Development
These commands are for a source checkout.
bundle install
ruby test/run_all.rb
bundle exec rspec spec/generatedtest/run_all.rb loads test/test_*.rb and test/generated/*_test.rb. The property tests use hegeltest, which needs Ruby 3.4 or newer. On Ruby 3.3, run BUNDLE_WITHOUT=property bundle install, and they report one skip. spec/generated/ is the emitted RSpec for examples/. Coverage is process-global, so each CLI test runs branchout in a process of its own, forked from a preloaded server where fork exists.
The gem ships RBS signatures in sig/. They are written from the type comments in lib/:
bundle exec rbs-inline --output=sig/generated lib
bundle exec steep checkLicense
MIT