Evilution — Mutation Testing for Ruby
Purpose: Validate test suite quality by injecting small code changes (mutations) and checking whether tests detect them. Surviving mutations indicate gaps in test coverage.
- License: MIT (free, no commercial restrictions)
- Language: Ruby >= 3.3
- Parser: Prism (Ruby's official AST parser, ships with Ruby 3.3+)
- Test frameworks: RSpec, Minitest, and Test::Unit
Installation
Add to Gemfile:
gem "evilution", group: :testThen:
bundle installOr standalone:
gem install evilutionRequires prism >= 1.5, < 2. Older Rails apps (e.g. Rails 7.1 pins prism 0.19) must upgrade prism — the gemspec constraint forces bundler to resolve a compatible 1.x version. If your app pins prism 2.x, bundler will reject the install until evilution widens its upper bound.
Installing on Rails 7.1 + Ruby 3.3
Two Bundler conflicts hit fresh installs on this stack:
-
cgiactivation conflict. Rails 7.1'sGemfile.lockpinscgi 0.5.0. Ruby 3.3.x shipscgi 0.5.1as a default gem. Loading evilution via Bundler aborts with:Gem::LoadError: can't activate cgi-0.5.1, already activated cgi-0.5.0. Make sure all dependencies are added to Gemfile. -
prismpin. Same lockfile pinsprism 0.19, which lacks theIfNode#subsequentaccessor evilution uses (older Prism releases exposed it asconsequent). Symptom:NoMethodError: undefined method 'subsequent' for an instance of Prism::IfNode.
Both resolve cleanly with a sidecar Gemfile.local that re-evaluates the project Gemfile and adds evilution + prism on top — no edits to the main Gemfile.lock:
# Gemfile.local
eval_gemfile("Gemfile")
group :test, :development do
gem "evilution"
gem "prism", "~> 1.5"
endThen invoke evilution against that Gemfile:
BUNDLE_GEMFILE=Gemfile.local bundle install
BUNDLE_GEMFILE=Gemfile.local bundle exec evilution run lib/foo.rbThe first command writes a sibling Gemfile.local.lock. Decide whether to commit or .gitignore it the same way you would for any developer-only Gemfile — typically gitignored when only one or two engineers run mutation testing locally, committed when CI also runs evilution against the sidecar Gemfile.
The evilution gemspec already declares prism >= 1.5, < 2, so adding the gem "prism" line above is only necessary on stacks that also pin prism in Gemfile.lock.
Migrating from mutant
Coming from the commercially-licensed mutant gem? See
docs/migration-from-mutant.md for a command/config
mapping, output-terminology differences, and how to diff the two tools' results with
evilution compare (it ingests mutant session JSON directly).
Command Reference
evilution [command] [options] [files...]
The shorter alias evil ships alongside evilution and accepts identical arguments (handy with alias be='bundle exec' → be evil run ...).
Every command, subcommand, and flag listed in this section is part of evilution's public CLI contract; see docs/versioning.md for stability and deprecation rules.
Commands
| Command | Description | Default |
|---|---|---|
run (alias mutate) |
Execute mutation testing against files | Yes |
init |
Generate .evilution.yml config file |
|
version |
Print version string | |
subjects [files] |
List mutation subjects with locations and counts | |
tests list [files] |
List the spec files run would use for each source (layout, spec_mappings, spec_pattern) |
|
session list |
List saved session results | |
session show FILE |
Display detailed session results | |
session diff A B |
Compare two sessions (fixed/new/persistent) | |
session gc --older-than D |
Garbage-collect sessions older than D (e.g. 30d) | |
util mutation |
Preview mutations for a file or inline code | |
environment show |
Display runtime environment and settings | |
compare --against A --current B |
Compare two saved session JSON files into fixed / new / persistent / flaky / reintroduced buckets |
Options (for run command)
| Flag | Type | Default | Description |
|---|---|---|---|
-t, --timeout N
|
Integer | (adaptive) | Per-mutation timeout in seconds, the same for every mutation. Without it the limit adapts to the baseline, see Per-mutation timeout. |
--timeout-factor K |
Float | 10 | Without --timeout: limit each mutation to K × the baseline time of its spec files. |
--timeout-floor N |
Float | 10 | Without --timeout: the shortest such limit, in seconds. |
--timeout-ceiling N |
Float | 60 | Without --timeout: the longest such limit, in seconds. |
-f, --format FORMAT
|
String | text |
Output format: text, json, or html. |
--target EXPR |
String | (none) | Only mutate matching subjects. Supports method name (Foo::Bar#calculate), class (Foo, which also matches a value-object constant named Foo), namespace wildcards (Foo::Bar*), method-type selectors (Foo#, Foo.), descendants (descendants:Foo), and source globs (source:lib/**/*.rb). |
--since REF |
String | (none) | Only mutate the lines changed since git REF (a branch, tag or commit), diffed from its merge base with HEAD, uncommitted and untracked changes included. Without file arguments: every changed .rb file under lib/ and app/. With them: only those that changed, line ranges narrowed to their changed lines. Also accepted by subjects. See Mutating only what changed. |
--output FILE |
String | (stdout) | Write the report to FILE instead of stdout. Useful when a preloaded spec helper writes to stdout on exit. |
--min-score FLOAT |
Float | 0.0 | Minimum mutation score (0.0–1.0) to pass. |
--spec FILES |
Array | (none) | Spec files to run (comma-separated). Defaults to auto-detection via SpecResolver, which also resolves non-mirrored (spec/unit, test/unit), dir-grouped (test/unit/<class>/*_test.rb), and flat test_-prefixed (test/test_connection_pool_timed_stack.rb) layouts. |
--spec-dir DIR |
String | (none) | Include all *_spec.rb files in DIR recursively. Composable with --spec. |
--spec-pattern GLOB |
String | (none) | Restrict resolved spec candidates to files matching GLOB (e.g. spec/models/**/*_spec.rb). |
--no-example-targeting |
Boolean | (enabled) | Disable per-mutation example targeting (always run every example in the resolved spec file). Example targeting scans each example body for symbols from the mutated method and runs only the matching subset. |
--example-targeting MODE |
String | lexical |
How targeting picks examples: lexical (name-grep example bodies for the mutated method/class), coverage (run only the examples that actually execute the mutated line, from a cached line-coverage map), or full_file (run all resolved examples). |
--example-targeting-fallback MODE |
String | full_file |
Behavior when no example matches: full_file (run the whole spec file) or unresolved (skip the mutation as :unresolved). |
--[no-]skip-uncovered |
Boolean | (enabled) | With --example-targeting coverage: report a mutation whose changed lines no example executes as uncovered, without running it. See Uncovered mutations. |
--[no-]probe-first |
Boolean | (disabled) | Experimental. Run each method's body-to-raise mutant first; when it survives, report the method's other mutants as survivors implied by it, without running them. See Probe-first. |
-j, --jobs N
|
Integer | 1 | Number of parallel workers. Uses demand-driven work distribution with pipe-based IPC. |
--no-baseline |
Boolean | (enabled) | Skip baseline test suite check. By default, a baseline run detects pre-existing failures, so that a mutation whose tests fail only on those is recorded neutral rather than killed. |
--[no-]canary |
Boolean | (enabled) | Run a proof-of-life synthetic mutation at session start; abort the run if the pipeline misreports it. Catches misconfigured isolation, broken autoload, and reporter-plugin eviction before any real score is produced. Pass --no-canary to skip (e.g. CI speed, or when the canary itself is the thing under test). |
--[no-]stop-on-kill |
Boolean | (enabled) | Stop each mutation's test run at the first failure that decides the verdict. Disable to run every selected example. |
--[no-]killer-first |
Boolean | (disabled) | RSpec, with stop-on-kill: run first the examples that killed nearby mutations, then the cheapest. Only for suites whose examples pass in any order. See Killer-first order. |
--fail-fast [N] |
Integer | (none) | Stop after N surviving mutants (default 1 if no value given). |
-v, --verbose
|
Boolean | false | Verbose output with RSS memory and GC stats per phase and per mutation; also prints error class, message, and first 5 backtrace lines for errored mutations. |
--suggest-tests |
Boolean | false | Generate concrete test code in suggestions (RSpec or Minitest, based on --integration; falls back to generic phrasing for test-unit). |
-q, --quiet
|
Boolean | false | Suppress output. |
--stdin |
Boolean | false | Read target file paths from stdin (one per line). |
--integration NAME |
String | rspec |
Test framework integration: rspec, minitest, or test-unit. See docs/integrations.md. |
--[no-]incremental |
Boolean | false | Cache each mutation's verdict (killed, survived, timeout) and skip it on re-runs until its subject, its spec files or the run's settings change — see Incremental cache. Pass --no-incremental to override incremental: true from the config file for one invocation (e.g. cold-cache debugging). Last flag wins when both are given. |
--save-session |
Boolean | false | Persist results as timestamped JSON under .evilution/results/. |
--no-progress |
Boolean | (enabled) | Disable the TTY progress bar. |
--quiet-children |
Boolean | false | Redirect each forked worker's stdout/stderr to per-pid files under tmp/evilution_children/<pid>.{out,err} so noisy app initializers (Datadog, Bullet, etc.) don't merge with parent output. Trade-off: live worker errors only appear in the side files, not the terminal — tail -f tmp/evilution_children/*.err to watch them. |
--quiet-children-dir DIR |
String | tmp/evilution_children |
Override the directory used by --quiet-children. |
--isolation MODE |
String | auto |
Isolation strategy: auto, fork, or in_process. auto selects fork for Rails projects and packaged gems (*.gemspec), in_process otherwise. Under in_process a mutation that takes the Ruby process down ends the run (what in_process cannot contain). See docs/isolation.md. |
--preload FILE |
String | (auto) | File to require in parent before forking workers. Auto-detect chain for Rails projects: spec/rails_helper.rb → spec/spec_helper.rb → test/test_helper.rb. For non-Rails gems: spec/spec_helper.rb → test/test_helper.rb → test/helper.rb, falling back to the gem entry lib/<gem>.rb (with a warning naming the searched paths). Pass --no-preload to opt out. |
--no-preload |
Boolean | (enabled) | Disable parent-process preload. |
--warmup NAME |
String | none |
After preload, warm lazily-initialised framework state once in the parent: none or rails. See Warming up Rails before forking. |
--[no-]copy-worker-databases |
Boolean | (enabled) | With --jobs above 1 and a preloaded Rails app, give each worker its own copy of a SQLite test database the workers would otherwise share. See Per-worker database copies. |
--skip-heredoc-literals |
Boolean | false | Skip all string literal mutations inside heredocs. |
--skip-sites PRESETS |
List | (none) | Leave low-value code unmutated, comma-separated: logging, debug_output, error_messages, requires. --no-skip-sites overrides skip_sites from the config file. See Skipping Low-Value Code. |
--show-disabled |
Boolean | false | Report mutations skipped by # evilution:disable comments. |
--fallback-full-suite |
Boolean | false | When no matching spec/test resolves for a mutation, run the whole test suite instead of marking it :unresolved and skipping. |
--related-specs-heuristic |
Boolean | false | When a mutation removes an includes(...) call, also run matching specs from spec/{requests,integration,features,system} (Rails-style domain match on the source file's basename). Trades extra spec runs for higher kill rate on ORM mutations. |
--baseline-session PATH |
String | (none) | Saved session file for HTML report comparison. |
-e CODE, --eval CODE
|
String | (none) | Inline Ruby code for util mutation command. |
--profile NAME |
String | default |
Operator profile: default or strict. strict adds aggressive mutators (e.g. replaces x.predicate? with nil, or appends rescue nil to a raising call) intended for pre-merge audits. |
--strict |
Boolean | false | Shortcut for --profile=strict. |
Options (for session subcommands)
| Flag | Type | Default | Description |
|---|---|---|---|
--results-dir DIR |
String | .evilution/results |
Directory containing session result JSON files. Honored by session list and session gc. |
--limit N |
Integer | (none) | (session list) Show only the N most recent sessions. |
--since DATE |
String | (none) | (session list) Show only sessions created on or after DATE (YYYY-MM-DD). For run and subjects, --since takes a git ref instead. |
--older-than D |
String | (required) | (session gc) Delete sessions older than D (e.g. 30d, 24h, 1w). |
Options (for compare command)
| Flag | Type | Default | Description |
|---|---|---|---|
--against PATH |
String | (none) | Prior (older) session JSON to diff against. Positional first argument is accepted as a fallback. |
--current PATH |
String | (none) | Current (newer) session JSON. Positional second argument is accepted as a fallback. |
Operator Profiles
Two profiles ship out of the box:
-
default— the 155 stable operators registered inMutator::Registry.default. Suitable for everyday CI runs; balances coverage signal against survivor noise. -
strict— adds extra aggressive mutators on top ofdefault:-
PredicateToNilreplaces everyx.predicate?call withnilto surface tests that only assert truthiness rather than exact return values. -
ExceptionSwallowappendsrescue nilto a statement that raises by convention — a bang method,fetchwithout a default,Integer/Float/Rational— to surface tests that never make it fail and check the error comes out (record.save!->record.save! rescue nil). It skips Ruby core in-place bangs (uniq!,sort_by!, …),exit!, statements already under a rescue, andraise; project bangs that mutate rather than raise will still show up as survivors. -
StatementReorderswaps two adjacent commands (charge(card); send_receipt(user)->send_receipt(user); charge(card)) to surface side effects whose order is never asserted. It only swaps statements that both act — call a method, append, write an index, yield, call super — and leaves alone pairs that share a variable, contain control flow or a heredoc, involve a local or instance variable assignment, write different literal keys of the same receiver, or move the last statement of a body. Expect survivors where the order only shows up in help text or in the order a list is built. -
SafeNavigationInsertionreplaces a plain call with safe navigation (user.name->user&.name) to surface calls no test reaches with a nil receiver. With a nil receiver the mutant skips the call and givesnil, where the plain call raisesNoMethodErroror runs a methodnilhas (nil.to_a). It touches nearly every call with a receiver and survives wherever that receiver is never nil. It skipsself, constants and literal receivers other thannil, and::calls. The opposite direction,safe_navigation_removal, is in the default profile.
Use for pre-merge audits where you want maximum sensitivity at the cost of more survivors.
-
Set via --profile=strict, the --strict shortcut, or profile: strict in .evilution.yml.
Exit Codes
| Code | Meaning | Agent action |
|---|---|---|
| 0 | Mutation score meets or exceeds --min-score, or a --since run found nothing to mutate |
Success. No action needed. |
| 1 | Mutation score below --min-score, or a target file resolved to no spec |
Parse output, fix surviving mutants. |
| 2 | Tool error (bad config, parse failure, a run under --isolation in_process that a mutation took down, etc.) |
Check stderr, fix invocation. |
min_score defaults to 0.0, so no score gate is armed unless you set one. The Result: line says so rather than implying a threshold nobody configured:
$ evilution run lib/half_tested.rb # no gate
Result: 66.67% (no minimum score set) # exit 0
$ evilution run lib/half_tested.rb --min-score 0.8
Result: FAIL (score 66.67% < 80.00%) # exit 1
A run under --isolation in_process can end without a report: a mutation that takes the Ruby process down takes the run with it. It ends with 2, and a line on stderr names the mutation and points at --isolation fork. See What in_process cannot contain.
Evilution owns the exit status: --preload loads the project's own spec helper into the parent process, and an at-exit hook it installs (SimpleCov calls exit with its own status when coverage is below the minimum) would otherwise replace the status evilution computed (GH #1608).
The printed verdict and the exit code always use the same threshold. Previously the line was printed against a hard-coded 80% that the exit code did not share, so a failing-looking run still exited 0 (GH #1604).
Mutating only what changed (--since)
--since REF limits a run to the lines changed since a git ref, which is what a pull request needs checked:
bundle exec evilution run --since origin/main # every changed .rb file under lib/ and app/
bundle exec evilution run lib/billing.rb --since origin/main # only this file, if it changed
bundle exec evilution subjects --since origin/main # preview the subjects and mutation countsWhat counts as changed:
-
The branch's own changes. The diff runs from the merge base of
REFandHEAD(whatgit diff REF...HEADcompares), so commitsREFgained after the branch left it do not count, to the working tree, so uncommitted changes, staged or not, do. - Added and modified lines. A mutation is run when any line it changes is one of them; a subject (method, constant) is mutated when any of its lines is. Lines that were only deleted leave no line behind: the lines either side of the gap count instead, so a method whose only change is a removed line is still mutated there.
-
New files count whole, untracked ones included (those
.gitignoreleaves out do not). Renamed files count only their edited lines, under the new name. Deleted files, binary files and files outside the current directory do not count. -
Which files. Without file arguments, the changed
.rbfiles underlib/andapp/, as a run without arguments takes. File arguments narrow that: a file that did not change drops out, andlib/foo.rb:15-30keeps only the changed lines from 15 to 30. A file argument must be an existing file: a missing path or a directory is an error (exit 2), as without--since.--target source:GLOBcannot be combined with--since(pass the files instead). -
A method
--targetis looked for in the changed files only. When its file changed but the method did not, there is nothing to mutate and the run passes. When its file did not change, the target matches nothing and the run fails with exit 2 (no subject matched '...' in the files changed since REF), as a misspelled target does: evilution cannot tell the two apart without reading the unchanged files. A CI job that pins a--targetshould run only when the target's file changed. -
Nothing to mutate passes. A run whose changed lines hold nothing to mutate (a docs-only pull request) reports zero mutations and exits 0, whatever
--min-scoresays.
REF must be known to the clone, and the history back to the merge base must be there: a shallow CI checkout fails with a message saying so. In GitHub Actions:
- uses: actions/checkout@v4
with:
fetch-depth: 0 # the merge base with the target branch must be fetched
- run: bundle exec evilution run --since origin/${{ github.base_ref }} --format json --min-score 0.8 --output evilution.json--since is not a config file key: a ref in .evilution.yml would scope every run, explicit file lists included. The MCP evilution-mutate tool takes the same as its since parameter.
Configuration
Generate default config: bundle exec evilution init
Creates .evilution.yml:
schema_version: 1 # opts into strict validation (rejects unknown keys, refuses future versions)
# timeout: 30 # seconds per mutation; unset = adaptive (see Per-mutation timeout)
# timeout_factor: 10 # adaptive limit = factor x baseline time of the mutation's spec files...
# timeout_floor: 10 # ...but at least this many seconds...
# timeout_ceiling: 60 # ...and at most this many
# format: text # text | json | html
# min_score: 0.0 # 0.0–1.0
# integration: rspec # test framework: rspec, minitest, test_unit
# suggest_tests: false # concrete test code in suggestions (matches integration)
# save_session: false # persist results under .evilution/results/
# isolation: auto # auto | fork | in_process (auto selects fork for Rails + gems, in_process otherwise; in_process cannot contain a mutation that takes the Ruby process down, see docs/isolation.md)
# canary: true # proof-of-life synthetic mutation at session start (false to skip)
# stop_on_kill: true # stop each mutation's test run at the first deciding failure (false runs every selected example)
# killer_first: false # RSpec + stop_on_kill: run likely killers and cheap examples first (only for order-independent suites)
# skip_uncovered: true # with coverage targeting, report mutations no example executes as uncovered without running them
# probe_first: false # experimental: when a method's body-to-raise mutant survives, report its other mutants as implied survivors without running them
# preload: null # path to preload before forking; false to disable; auto-detects for Rails + gems
# warmup: none # rails: warm I18n, routes, assets and templates once after preload (fork isolation)
# copy_worker_databases: true # with jobs > 1 and a preloaded Rails app, give each worker its own copy of a shared SQLite test database
# skip_heredoc_literals: false # skip string literal mutations inside heredocs (recommended for Rails: heredoc SQL/templates rarely have test coverage)
# show_disabled: false # report mutations skipped by disable comments
# baseline_session: null # path to session file for HTML comparison
# ignore_patterns: [] # AST patterns to exclude (see docs/ast_pattern_syntax.md)
# skip_sites: [] # presets of low-value code to leave unmutated: logging, debug_output, error_messages, requires
# progress: true # TTY progress bar
# example_targeting: true # per-mutation example targeting via body-token scan
# example_targeting_fallback: full_file # full_file | unresolved
# spec_pattern: null # restrict resolved spec candidates to files matching GLOBPrecedence: CLI flags override .evilution.yml values.
Schema versioning
.evilution.yml may declare an integer schema_version (currently 1). Behavior:
-
When declared — strict mode. Unknown top-level keys raise
Evilution::ConfigError. Aschema_versiongreater than the installed gem supports is rejected so an old gem cannot silently misread a newer config. - When omitted — legacy lenient mode. Unknown keys are ignored.
Compatibility policy for the 1.x gem line:
- New configuration keys are added in MINOR releases (additive only). Each new key takes a default that preserves prior behavior.
- Existing keys are not removed, renamed, or have their semantics changed in any
1.xrelease. Deprecated keys keep working through the entire1.xline; see docs/versioning.md. -
schema_versionis bumped only on incompatible changes — i.e. only at the next MAJOR release.schema_version: 2will ship withevilution 2.0.
A JSON Schema covering every supported key lives at schema/evilution.config.schema.json. Point editor / IDE YAML extensions at it for autocomplete and inline validation (e.g. VS Code yaml.schemas, JetBrains "Custom JSON Schema").
Configuration reference
All keys recognised under schema_version: 1:
| Key | Type | Default | Description |
|---|---|---|---|
schema_version |
Integer | 1 |
Config schema version. Declaring it enables strict validation; omit for lenient mode. |
timeout |
Integer | (adaptive) | Per-mutation timeout in seconds, the same for every mutation. Unset, the limit adapts to the baseline (see Per-mutation timeout). |
timeout_factor |
Number | 10 |
Adaptive limit: this many times the baseline time of the mutation's spec files. Ignored when timeout is set. |
timeout_floor |
Number | 10 |
Shortest adaptive limit, in seconds. Must not exceed timeout_ceiling. |
timeout_ceiling |
Number | 60 |
Longest adaptive limit, in seconds. |
format |
String | text |
Output format: text, json, html. |
target |
String / null | null |
Filter expression: method (Foo#bar), class (Foo), namespace (Foo*), descendants (descendants:Foo), source glob (source:**/*.rb). |
output |
String | (stdout) | Write the report to this file instead of stdout. |
min_score |
Float | 0.0 |
Minimum mutation score (0.0–1.0) for exit code 0. |
integration |
String | rspec |
Test framework: rspec, minitest, or test_unit. |
verbose |
Boolean | false |
Verbose output (RSS/GC stats per phase, error details for errored mutations). |
quiet |
Boolean | false |
Suppress output. |
jobs |
Integer | 1 |
Number of parallel workers. |
fail_fast |
Integer / null | null |
Stop after N surviving mutants. null = disabled. |
baseline |
Boolean | true |
Run baseline test suite to detect pre-existing failures (a mutation failing only on those is :neutral). |
stop_on_kill |
Boolean | true |
Stop each mutation's test run at the first failure that decides the verdict. Set false to run every selected example. |
killer_first |
Boolean | false |
With RSpec and stop_on_kill, run first the examples that killed nearby mutations, then the cheapest. Only for suites whose examples pass in any order; see Killer-first order. |
isolation |
String | auto |
Isolation strategy: auto, fork, in_process. auto selects fork for Rails projects and packaged gems (*.gemspec), in_process otherwise. Under in_process a mutation that takes the Ruby process down ends the run: see What in_process cannot contain. |
incremental |
Boolean | false |
Cache verdicts across runs; see Incremental cache. |
suggest_tests |
Boolean | false |
Generate concrete test code in survivor suggestions (matches integration). |
progress |
Boolean | true |
TTY progress bar. |
save_session |
Boolean | false |
Save session JSON under .evilution/results/. |
line_ranges |
Hash | {} |
Per-file line-range constraints. Typically set via CLI; rare in YAML. |
spec_files |
Array<String> | [] |
Explicit spec files to run. Bypasses auto-detection when non-empty. |
ignore_patterns |
Array<String> | [] |
AST patterns to skip during mutation generation. See docs/ast_pattern_syntax.md. |
skip_sites |
Array<String> | [] |
Presets of low-value code to leave unmutated: logging, debug_output, error_messages, requires. See Skipping Low-Value Code. |
show_disabled |
Boolean | false |
Report mutations skipped by # evilution:disable comments. |
baseline_session |
String / null | null |
Saved session file path for HTML report comparison. |
skip_heredoc_literals |
Boolean | false |
Skip string literal mutations inside heredocs. |
related_specs_heuristic |
Boolean | false |
Append related request/integration/feature/system specs for includes(...) mutations. |
fallback_to_full_suite |
Boolean | false |
When no matching spec resolves, run the entire suite instead of marking the mutation :unresolved. |
preload |
String / Boolean / null | null |
File to preload in parent before forking. false to disable. null to auto-detect for Rails projects and packaged gems. |
warmup |
String | none |
rails warms lazily-initialised Rails state once after preload so forks do not each pay it. none / false to skip. |
copy_worker_databases |
Boolean | true |
With jobs above 1 and a preloaded Rails app, give each worker its own copy of a SQLite test database the workers would otherwise share. See Per-worker database copies. |
spec_mappings |
Hash<String, String/Array> | {} |
Custom mapping from source path to spec path(s). |
spec_pattern |
String / null | null |
Glob restricting resolved spec candidates. |
example_targeting |
Boolean | true |
Per-mutation example-level targeting. |
example_targeting_strategy |
String | lexical |
How targeting picks examples: lexical (name-grep) or coverage (examples that execute the mutated line, from a cached coverage map). |
example_targeting_fallback |
String | full_file |
When targeting finds no example: full_file or unresolved. |
example_targeting_cache |
Hash | { max_files: 50, max_blocks: 10000 } |
LRU cache bounds for the example-targeting AST parser. |
skip_uncovered |
Boolean | true |
With example_targeting_strategy: coverage, report a mutation no example executes as uncovered without running it. No effect otherwise. |
probe_first |
Boolean | false |
Experimental. Run each method's method_body_to_raise mutant first; when it survives, report the method's other mutants as survivors implied by it, without running them. See Probe-first. |
quiet_children |
Boolean | false |
Redirect each worker's stdout/stderr to per-pid files under quiet_children_dir. |
quiet_children_dir |
String | tmp/evilution_children |
Directory for --quiet-children per-pid log files. |
profile |
String | default |
Operator profile: default or strict. |
hooks |
Hash<String, String> | {} |
Lifecycle hooks: event name → path to a Ruby file returning a Proc. |
Stopping at the first kill (stop_on_kill)
By default each mutation's test run stops at the first failing example that decides the verdict, so the remaining selected examples are not run. Failures that were already failing in the baseline run do not stop it (the mutation is still recorded neutral if only those fail), and neither do infrastructure crashes (the mutation is still recorded neutral if only those occur). A mutant whose first failure is followed by a hang or a dead test process is now reported killed; a --no-stop-on-kill run may report the same mutant as timeout or error, so its score can be slightly lower. Pass --no-stop-on-kill (or stop_on_kill: false) to run every selected example. Minitest classes that call parallelize_me! are not stopped early: their tests all run, as they would with --no-stop-on-kill.
Killer-first order (killer_first)
With RSpec (3.13 or later), --killer-first (or killer_first: true) lets a run that stops at its first kill choose the order of its examples, since everything before the kill is its cost: first the examples that killed other mutations on the same line, then elsewhere in the same subject, earlier in this run (the latest killer first); then the rest, cheapest first by the time each took in the baseline; examples with no baseline time last. Examples stay inside their example group, and a group's before(:context) hooks still run once around its examples; a group goes first when it holds a killer, otherwise by the time its timed examples take in all (a group with no timed example goes last). Only the order changes, never which examples run. A survivor runs every selected example either way, and its confirmation run against the whole file keeps the defined order, as does the command reported in test_command. It has no effect with --no-stop-on-kill. In a parallel run, the mutations already handed to a worker when a kill comes back go without that killer. Minitest and test-unit runs keep their own order.
It is off by default because it is safe only for a suite whose examples pass in any order. An example that relies on one defined before it (a global, a class variable or a record that an earlier example sets up) fails when it runs first, whatever the mutant, so the mutation is reported killed even though a run in the defined order, including the reported test_command, would let it survive. Nothing in the report shows this, and it only raises the score. If your suite runs in random order (config.order = :random) and passes, it is a fair candidate. A run that times out in the killer-first order is run again in the defined order, which decides the verdict: a mutant that hangs the cheapest example would otherwise be reported timeout where the defined order reports killed. Such a mutation can take up to twice its time limit, and a parallel run gives each mutation twice the usual time before it gives a worker up. Under an adaptive limit this happens under each limit: a reordered run that uses up the adaptive limit runs again in the defined order under it, and only when that times out too does the mutation run again under the fixed limit, reordered and then, if it times out once more, in the defined order, which decides.
The JSON report lists, per mutation, the example that killed it (killing_test) and how many examples ran (examples_run), whatever the order.
Uncovered mutations
With coverage targeting (--example-targeting coverage), evilution records which lines each example of the resolved spec files executes, and which executable lines nothing executed at all. A mutation whose changed lines nothing executes could only survive: no test reaches the code. Running it would cost the most of any verdict — the whole resolved spec file, then the survivor confirmation run — and say nothing new. Such a mutation is reported uncovered instead, without a test run.
Score: an uncovered mutation counts as a survivor. It stays in the score's denominator, so the score, --min-score / min_score and the exit status are the same as running it would have given; the report lists it in its own uncovered bucket (text, JSON, HTML, MCP and saved sessions) rather than with the survivors that ran, and --fail-fast counts it as a survivor.
A mutation is reported uncovered only when the coverage map can vouch for it, and runs as before otherwise:
- Only the lines the mutation changes count. A line that runs only while files load — a class body, a constant,
attr_reader, adefline — counts as executed, so a mutation of it runs. A mutation that replaces a method body is reported at thedefline but changes only the body, so it is uncovered when no example calls the method. - At least one changed line is executable and never ran; none of them ran.
- The map was built from every spec file the mutation would run. A mutation that runs a spec file the build did not (the related-specs heuristic, the full-suite fallback) runs.
- Every example of those spec files ran in the build. A spec file that failed to load, or a build cut short (
fail_fastin.rspec, an interrupt), makes the map vouch for no line. - The file was loaded while lines were counted. When coverage targeting is on and the tests run in forked processes (
isolation: fork, the default for Rails apps and gems), evilution counts from before thepreloadruns, so a preloaded file is covered too; a file loaded while counting was off is never vouched for. Each forked test process stops that counting before its tests run, so a spec helper that starts line coverage itself (SimpleCov) works as it does without coverage targeting. Underin_processisolation the preload is not counted. - No example forked a process. A forked process counts its lines apart, where the build cannot see them, while a mutation applied before the fork reaches it; when any example forks, the map vouches for no line.
- The mutation neither adds nor removes a local variable assignment. Ruby settles while parsing whether a name is a local, so such a change reaches lines that do run (a later read of the name becomes a method call), even when its own line never runs.
The coverage map is cached under .evilution/coverage/ and rebuilt when any file it depends on changes: a target file, a resolved spec file, one of the helpers every spec file loads (.rspec, spec/spec_helper.rb, spec/support/**/*.rb, ... — the same list as the incremental cache), any file under the directory the spec files sit in (spec/, test/: shared examples, factories, fixture data), and any file of the project the build loaded (another file under lib/ that calls the code, for instance). It is RSpec only, like coverage targeting. Pass --no-skip-uncovered (or skip_uncovered: false) to run these mutations anyway.
summary.uncovered counts these mutations; it is unrelated to summary.uncovered_code, which lists targeted lines that hold no subject, so no mutation was generated for them.
Probe-first (experimental)
--probe-first (or probe_first: true) runs one sentinel mutant per method first: the method's whole body replaced by a bare raise (method_body_to_raise). When even that survives, no selected test calls the method on a path it checks, and the method's other mutants are reported survived without running them; their JSON entries carry implied_by: "method_body_to_raise". A sentinel reported uncovered counts the same. A method whose sentinel was killed, timed out or errored, and a subject that has no sentinel (an empty body, a body that already only raises, a constant or a declaration), runs all its mutants as before. So does a method with a rescue or ensure clause of its own (def safe_div(a, b); a / b; rescue StandardError; -1; end): its body-to-raise mutant keeps the clause, which can catch the raise or return over it, so that mutant surviving says nothing about the tests and is not used as a sentinel.
The run goes in two batches: all sentinels, then the rest, so under --jobs the second batch waits for the slowest sentinel, and each batch has its own worker pool: the run's timings.workers lists the workers of both (up to twice --jobs), the first pool's with a few sentinels each. An implied survivor is scored like any survivor, gets no survivor confirmation run and is not cached; the text report marks it (implied by method_body_to_raise, not run) and counts the implied survivors next to survived. It can be wrong: a test that swallows the sentinel's exception (a rescue around the call, a matcher that expects some error) lets the sentinel survive while another mutant of the method is killed. It is off by default; see docs/probe_first_spike.md for what it saves and how often it is wrong on real projects.
Disable Comments
Suppress mutations on specific code with inline comments:
# Disable a single line
log(message) # evilution:disable
# Disable an entire method (place comment immediately before def)
# evilution:disable
def infrastructure_method
# no mutations generated for this method body
end
# Disable a region
# evilution:disable
setup_logging
configure_metrics
# evilution:enableUse --show-disabled to see which mutations were skipped.
Skipping Low-Value Code
Some mutations cost a full test run and rarely say anything: a changed log message, a reworded error message, a dropped require that the preloaded process has already run. skip_sites (or --skip-sites) names presets of such code to leave unmutated:
skip_sites: [logging, error_messages]Each preset is a structural rule over the parsed source, like ignore_patterns, but a preset only skips what changes what the code reports. The raise, the class raised, the condition guarding a raise or a log line, code that may change state, a call whose value is used, and any mutation that puts a raise in are always mutated.
| Preset | Sites | Skipped inside a site |
|---|---|---|
logging |
A log level (debug info warn error fatal unknown add log) sent to a receiver named logger or LOGGER (logger, @logger, Rails.logger, config.logger, App::LOGGER); warn / deprecation_warning sent to a deprecator (ActiveSupport::Deprecation, ActiveSupport.deprecator, Rails.deprecator, deprecator); warn(..., category: :deprecated). Not logger.debug? or logger.level = .... |
Literal text of the message; anything inside an argument or block that only reads. The call itself (deleting the line) and its receiver, when every argument only reads and the call's value goes unused: a statement followed by another one, not the last statement of a method or block (whose value it returns), nor an operand or an assigned value (logger.info(...) && save). |
debug_output |
puts, print, printf, warn sent to no receiver or to Kernel; puts / print / printf / write sent to $stdout, $stderr, STDOUT, STDERR; p / pp and a stream's write only where their value goes unused, since they return what they print or the bytes written (p x as the last statement returns x). |
As for logging. Kept apart from logging because console output is a CLI's actual job. |
error_messages |
raise / fail sent to no receiver or to Kernel. The message is the second positional argument (raise Error, "msg"), the only one when it is a string (raise "msg"), or the string arguments of an error built in place on a constant (raise Error.new("msg"), .exception). Keywords (cause:), the backtrace argument and any argument after a splat are not messages. |
Literal text of the message; anything inside a message that only reads. Never the raise, its class, or a non-string argument of Error.new(...), which the error may keep as data. |
requires |
require / require_relative sent to no receiver or to Kernel that run while the file loads: at the top of the file or in a class or module body. Not inside a method, block or lambda (lazy loading), nor a require whose value is used (HAS_OJ = require "oj"). |
Everything, the call included. Evilution mutates load-time code only in constant assignments and class-body declarations (include, private, alias), none of which is a plain require line, so today this preset skips nothing; it guards such lines should a subject ever reach them. |
"Only reads" means: literals, variable and constant reads, interpolation, && / || / ternaries, [], operators, format, sprintf, join, to_s, and, without arguments or a block, predicates (empty?) and a fixed list of common readers (class, name, message, inspect, id, size, length, first, last, to_s, to_i, to_sym, path, ...). Any other call, even one without arguments, may change state (queue.pop, io.read, enum.next), so a message such as "left: #{queue.pop}" keeps the call mutated (only its literal text is skipped), and a log line that calls one is never deleted. Mutations whose node is the whole raise call (removing or nil-ing its message argument) are not skipped, nor is replacing the message of raise "msg" as a whole: raise nil raises TypeError instead of RuntimeError.
A skipped mutation is not run and not scored. It is counted in summary.skipped, and per preset in summary.skipped_sites (JSON) and on a Skipped sites: line in the text report. A mutation an ignore_patterns pattern also matches is counted under the pattern. skip_sites changes no verdict, so it does not invalidate the incremental cache.
Default: none. Under the 1.x compatibility policy a new key's default preserves prior behaviour, and turning a preset on removes mutants, and with them points from the score, that existing users see today. Measured over the whole lib/ of dry-monads, httparty, grape and i18n and Redmine's app/ and lib/redmine, all four presets together skip 0.3-0.7% of the mutants (in Redmine mostly logging, elsewhere mostly error_messages), and some of the skipped error-message mutants are killed by specs that assert the message, so the speed-up a default would buy is small next to a changed mutant set and score. Turn on the presets that fit your code base.
JSON Output Schema
Use --format json for machine-readable output. The same shape is used for both stdout reports (--format json) and saved session files (--save-session → .evilution/results/*.json); session files add a small set of extra top-level fields described under Session JSON files below.
Schema:
{
"schema_version": "integer — schema version of this JSON document (current: 1)",
"version": "string — gem version",
"timestamp": "string — ISO 8601 timestamp of the report",
"summary": {
"total": "integer — total mutations generated",
"killed": "integer — mutations detected by tests (test failed = good)",
"survived": "integer — mutations NOT detected (test passed = gap in coverage)",
"timed_out": "integer — mutations that exceeded timeout",
"errors": "integer — mutations that caused unexpected errors",
"neutral": "integer — mutations with no verdict: their tests failed only on examples that were already failing in the baseline, or the test process crashed on infrastructure",
"equivalent": "integer — mutations proven to have identical behavior to the original",
"unresolved": "integer — mutations where no spec file resolved (coverage gap, not a failure)",
"unresolved_target_files": "array of strings (optional) — target files that resolved to no spec at all; present only when non-empty, and the run fails when it is",
"infra_retried": "integer (optional) — mutations a parallel pass could not judge because the test process crashed on infrastructure, re-run serially afterwards; present only when non-zero",
"uncovered_code": "array (optional) — targeted lines that hold code outside every subject, so no mutation was generated for them: [{ file, lines: ['2-4', '9'] }]; checked for a line range as given, and for a whole file only when it has no subject at all; present only when non-empty",
"baseline_neutralized": "integer (optional) — kills not counted because only examples already failing in the baseline failed; those mutations are recorded neutral; present only when non-zero",
"baseline_failures": "array (optional) — one entry per spec file that was red in the baseline: { spec_file, error: string|null — what stopped the file outside any example (load error, runner exception, timeout, a baseline process that died), failing_examples: integer — every failing example, examples: [{ id, description, message }] — the first few of them }; present only when non-empty",
"unparseable": "integer — mutations whose mutated source did not parse (short-circuited, never executed)",
"uncovered": "integer — mutations whose changed lines no example executes, reported without a test run (coverage targeting only); scored as survivors",
"score": "float — killed / (total - errors - neutral - equivalent - unresolved - unparseable), range 0.0-1.0, rounded to 4 decimals; uncovered mutations stay in the denominator, as survivors",
"duration": "float — total wall-clock seconds of the run, from subject discovery through the audit of target files for unresolved specs (the phases in timings.phases add up to it), rounded to 4 decimals",
"peak_memory_mb": "float (optional) — peak RSS across all mutation child processes, in MB",
"skipped": "integer (optional) — mutations left out during generation (ignore_patterns, skip_sites, # evilution:disable, Sorbet sigs), never run or scored; present only when non-zero",
"skipped_sites": "object (optional) — the share of each skip_sites preset in skipped, e.g. { logging: 3, error_messages: 12 }; present only when a preset skipped something"
},
"survived": [
{
"operator": "string — mutation operator name (see Operators table)",
"file": "string — relative path to mutated file",
"line": "integer — line number of the mutation",
"status": "string — result status: 'survived', 'killed', 'timeout', 'error', 'neutral', 'equivalent', 'unresolved', 'unparseable', or 'uncovered'",
"duration": "float — seconds this mutation took, rounded to 4 decimals",
"diff": "string — legacy +/- diff snippet",
"unified_diff": "string (optional, survived and uncovered only) — git-style unified diff with `--- a/file`, `+++ b/file`, `@@` hunk header and sdiff body; omitted when source slices are unavailable",
"suggestion": "string — actionable hint for surviving mutants (survived and uncovered only)",
"test_command": "string (optional) — the command that reproduces the run behind this verdict, shell-escaped so it can be pasted as is (a targeted RSpec run names example ids such as `spec/x_spec.rb[1:2]`); for a survivor confirmed against its whole spec files, that confirming run",
"killing_test": "string (optional, RSpec only) — the example whose failure killed the mutation, as `/absolute/path/spec/file_spec.rb[1:2]` (spec file and RSpec scoped id); absent when no example decided the kill",
"examples_run": "integer (optional, RSpec only) — how many examples ran against the mutation, the survivor confirmation run included; absent for results that did not run (cached, uncovered, unparseable)",
"implied_by": "string (optional, survived only, --probe-first) — the sentinel operator ('method_body_to_raise') whose survival implied this verdict; the mutation itself did not run",
"time_limit": "float (optional, timeout only) — the seconds this mutation's test run was allowed, rounded to 4 decimals; for a timeout confirmed by a second run (see Per-mutation timeout) the limit of that second run; for a timeout read from the --incremental cache, the limit it was recorded under",
"neutral_reason": "object (optional, neutral only) — { kind: 'baseline_failure' | 'infra_error', detail: string|null — the failing spec file or the crash class; null when the run was given explicit --spec files and no single spec can be named }",
"timings": "object (optional) — where this mutation's duration went, in seconds rounded to 4 decimals; only the phases that happened are listed: { apply — parsing and evaluating the mutated source, test_run — the first test run, confirmation — the re-run of a survivor against its whole spec files (RSpec, targeted runs only), queue_wait — parallel runs only: from the dispatcher sending it to a worker to that worker starting it; a worker is sent its next mutation as soon as it finishes one, so this is the cost of the hand-over, and for a worker's first mutation also of the worker starting up, adaptive_timeout — a first run that used up an adaptive limit shorter than the fixed one, after which the mutation ran again under the fixed limit and the other phases describe that second run (see Per-mutation timeout) }; what duration holds beyond these is the isolation around the run (fork, sandbox, reap); a timed-out mutation reports nothing itself, so its whole duration is counted as test_run; absent for results read from the --incremental cache"
}
],
"timings": {
"phases": "object — wall-clock seconds per phase of the run, in the order they ran: subject_discovery, preload, canary, baseline, planning (mutation generation and equivalence detection), execution (every mutation, including the serial re-run of infrastructure crashes and building the coverage map when coverage targeting is on), finalize; they add up to about summary.duration",
"baseline": "array — one entry per spec file the baseline ran, in order: { spec_file, duration }; empty when the baseline did not run",
"mutations": "object — each per-mutation timing phase summed over all mutations: { apply, test_run, confirmation, queue_wait, adaptive_timeout }; process seconds, so in a parallel run they add up across workers and can exceed the execution phase",
"workers": "array — one entry per worker process of a parallel run: { pid, items, busy, wall, utilization — busy / wall }; workers live for the whole run, so there are --jobs entries (fewer when there are fewer mutations to run; with --probe-first the sentinels and the other mutations run in two pools one after the other, so up to twice as many) plus one for each worker that replaced one killed after a timeout, a crash or an unreadable reply; under --isolation in_process each mutation gets a fresh worker, so there is one entry per mutation run; busy and wall are 0.0 for a worker that died before reporting them; empty for --jobs 1"
},
"subjects": [
{
"name": "string — subject name (e.g. 'Foo#bar')",
"file": "string — relative path to source file",
"total": "integer — mutations generated for this subject",
"killed": "integer — mutations detected",
"verified": "integer — mutations that got a verdict (killed + survived + timed out + uncovered)",
"survived": "integer — mutations that went undetected",
"uncovered": "integer — mutations no example executes, reported without a test run; counted in verified",
"score": "float — killed / verified, 0.0 when nothing was verified, rounded to 4 decimals",
"reached": "boolean — false when no mutation of this subject got a verdict at all"
}
],
"coverage_gaps": [
{
"file": "string — relative path to source file",
"subject": "string — method name (e.g. 'Foo#bar')",
"line": "integer — line number",
"operators": ["string — operator names involved"],
"count": "integer — number of survived and uncovered mutations in this gap",
"mutations": ["... same shape as survived entries; `status` tells a survivor from an uncovered mutation ..."]
}
],
"killed": ["... same shape as survived entries ..."],
"neutral": ["... same shape as survived entries ..."],
"equivalent": ["... same shape as survived entries ..."],
"unresolved": ["... same shape as survived entries — coverage gap: no spec file resolved for these mutations"],
"unparseable": ["... same shape as survived entries — mutated source failed to parse and was never executed"],
"uncovered": ["... same shape as survived entries — no example executes the lines the mutation changes; never executed, scored as survived"],
"timed_out": ["... same shape as survived entries ..."],
"errors": [
{
"... same shape as survived entries, plus: ...": "",
"error_message": "string (optional) — error message from the failing mutation",
"error_class": "string (optional) — exception class name (e.g. 'SyntaxError', 'NoMethodError')",
"error_backtrace": ["string (optional) — first 5 backtrace lines from the exception"]
}
]
}Key metric: summary.score — the mutation score. Higher is better. 1.0 means all mutations were caught.
Session JSON files
Sessions saved by --save-session (under .evilution/results/*.json) and consumed by evilution session show, evilution session diff, evilution compare, and the HTML reporter share the schema above with these additions:
| Field | Type | Description |
|---|---|---|
git |
Object |
{ "sha": "<full SHA or null>", "branch": "<branch name or null>" } captured at run time. |
killed_count |
Integer | Top-level convenience counter; mirrors summary.killed. |
timed_out_count |
Integer | Mirrors summary.timed_out. |
error_count |
Integer | Mirrors summary.errors. |
neutral_count |
Integer | Mirrors summary.neutral. |
equivalent_count |
Integer | Mirrors summary.equivalent. |
uncovered_count |
Integer | Mirrors summary.uncovered. |
skipped_count |
Integer | Mutations left out during generation: ignore_patterns, skip_sites, # evilution:disable, Sorbet sigs (mirrors summary.skipped, omitted from summary unless positive; summary.skipped_sites carries the per-preset share). |
Saved sessions also omit the per-status arrays (killed, neutral, equivalent, unresolved, unparseable, timed_out, errors) — only survived, uncovered and coverage_gaps are persisted. evilution session diff treats an uncovered mutation as a survivor, so one that survived before is not reported fixed. The score, totals, and timestamps are stable for diff/compare consumers.
stdout in JSON mode
With --format json, stdout carries the JSON document and nothing else. Each mutation's test run writes to buffers evilution owns, whatever the isolation mode: under in_process the framework's configuration can outlive a single run — --preload builds it before isolation swaps $stdout — so the run claims RSpec's output and error streams outright rather than relying on RSpec to redirect them (GH #1627). Once the document is written, stdout is pointed at stderr, so anything a preloaded spec helper prints on the way out — SimpleCov's coverage report, for example — lands on stderr instead of after the document where it would leave JSON.parse with nothing to work with. --output FILE writes the document to a file and leaves stdout alone entirely.
Schema versioning
Every session and stdout JSON document carries a top-level schema_version integer (currently 1). On read:
-
schema_versionmatches what this gem supports — proceed normally. -
schema_versionis omitted — treated as version1(the JSON shape that defined version 1). Sessions written before this field existed continue to load. -
schema_versionis greater than what this gem supports —Evilution::Session::Store#load,evilution compare,evilution session show,evilution session diff, and the HTML reporter raiseEvilution::Errorwith the offending file path and a "Upgrade the evilution gem" message. We refuse to silently misread a newer document.
Compatibility policy for the 1.x gem line:
- New top-level fields are added in MINOR releases (additive only). Consumers that ignore unknown fields keep working without changes.
- Existing fields are not removed, renamed, or have their semantics changed in any
1.xrelease. -
schema_versionis bumped only on incompatible changes — i.e. only at the next MAJOR release.schema_version: 2will ship withevilution 2.0. See docs/versioning.md for the umbrella SemVer policy.
Mutation Statuses
| Status | Meaning | Counted in score? |
|---|---|---|
killed |
A test failed when the mutation was applied — test suite caught it. Also a mutation that made its own file raise while loading, before any test ran, when the original file still loads (error_message starts with load failed:). Also a mutation whose test process died on an exception no handler saw, after the mutated source loaded and while the tests ran, as a crashing test would: under fork isolation error_message starts with child died and names the exception, under in-process isolation it starts with test run ended by
|
numerator + denominator |
survived |
No test failed — gap in coverage | denominator only |
timeout |
Test run exceeded its time limit (--timeout, or the adaptive limit) — treated like survived for scoring |
denominator only |
error |
Mutation caused an unexpected error (syntax error, boot failure, a load failure the original file shares, etc.). Also a test process that died on an unhandled exception before the tests ran: under fork isolation error_message starts with child died, says before the tests ran and error_class is the exception's own class. Also a result that never arrived or could not be read, which is no verdict on the mutation (see Diagnosing errored mutations) |
excluded from denominator |
neutral |
No verdict: the tests failed only on examples that were already failing in the baseline, or the test process crashed on infrastructure (DB lock, statement timeout) rather than on the mutation. Every neutral records which of the two, and the report groups them by it | excluded |
equivalent |
Mutation is provably identical to the original (e.g. no-op replacement) | excluded |
unresolved |
No spec file resolved for the mutated source — coverage gap, not a failure. Use --fallback-full-suite to run the full suite instead. |
excluded |
unparseable |
Mutated source failed to parse (e.g. dangling heredoc opener after method_body_replacement). Short-circuited — never executed. |
excluded |
uncovered |
No example executes the lines the mutation changes (coverage targeting only, see Uncovered mutations). Never executed — it could only have survived. | denominator only, as survived |
Unresolved mutations indicate a missing test mapping — the file has no corresponding test file that the resolver could find (for example, an RSpec _spec.rb file or a Minitest _test.rb file, depending on configuration). The resolver searches the lib/-mirrored path, common non-mirrored buckets (spec/unit, spec/lib, test/unit, test/lib), and the flat test_-prefixed Minitest/Test::Unit convention (test/test_connection_pool_timed_stack.rb), so a high unresolved rate usually means a genuinely missing or unconventionally-placed test; a run that leaves many mutations unresolved prints an unresolved-rate warning with a best-guess spec path per source file. They are reported separately so you can act on them (add a test, adjust test naming, pass --spec, or opt in to the full-suite fallback) without inflating the error count.
A target file that resolves to no test at all is a stronger condition than an individual unresolved mutation, and is reported on its own terms: evilution names the file and fails the run, whatever the mutations it did measure scored. Without this, adding one well-tested file to the command dilutes the unresolved rate and the untested file disappears behind a PASS (GH #1603):
$ evilution run app/services/untested.rb app/services/well_tested.rb
Mutations: 98 total, 82 killed, 0 survived, 0 timed out, 2 neutral, 14 unresolved
Score: 100.00% (82/82)
! 1 of 2 target files has no resolvable spec — it was never tested:
app/services/untested.rb
Result: FAIL (1 target file has no resolvable spec)
$ echo $?
1
The check covers files evilution found something to mutate in; a file it produced no mutations for (a constants-only file, say) is not reported. --fallback-full-suite runs such a file against the whole suite instead, so nothing goes untested and nothing is reported. The file list is also in JSON output under summary.unresolved_target_files.
Survivor Confirmation
Per-mutation targeting runs a subset of a spec file's examples, chosen by matching the enclosing method's name against example bodies. That match keys on identifier text, so it can miss the example that would have caught a mutation — and the mutation is then reported as a survivor the suite actually covers. A survivor nobody can reproduce is worse than a missed kill: it sends the reader to write a test that is already there (GH #1624).
Before any survivor is reported, it is therefore re-run against the whole resolved spec file, and that run is the one reported. Only survivors pay for the extra run, and only where the targeted subset was narrower than the file; a mutation the targeted examples already killed is never re-run.
On evilution's own lib/evilution/reporter/json/subjects.rb this moved the reported score from 74.19% with 8 survivors to 93.55% with 2 — the six that disappeared were killed by an example in the same file all along, and the run now agrees with --no-example-targeting instead of contradicting it.
Per-mutation timeout
A mutation that makes the code loop runs until its time limit, so the limit is most of what such a mutant costs. A fixed limit is either far too long for a spec file that passes in a fraction of a second or too short for a slow one.
So when no timeout is given, each mutation's limit follows the time its own spec files took in the baseline:
limit = clamp(timeout_factor × baseline seconds of the mutation's spec files, timeout_floor, timeout_ceiling)
With the defaults (factor 10, floor 10s, ceiling 60s) a mutant on a spec file that passes in 0.3s is first given 10s instead of 30s, and one whose spec files take 4s gets 40s. The baseline runs each spec file in a forked process of its own, so its time holds the fork and loading the file; the factor leaves room for the re-run of a survivor against its whole spec files (Survivor Confirmation) and for workers competing for the CPU under --jobs, and the floor covers spec files so fast that scheduling noise outweighs them.
A mutant that slows the code down can still fail its test well after its baseline suggests, and a timeout reported there would hide a kill. So a mutation that uses up a limit shorter than the fixed 30s is run again under 30s, and the verdict of that second run is the one reported: a mutation is only reported as a timeout where the fixed limit would have reported one too. The first run is part of what the mutation cost: it is in its duration and in the JSON report's timings as adaptive_timeout. A mutant that loops therefore costs the shorter limit plus 30s; the time saved is on kills and survivors, and on spec files slow enough to get a longer limit. Raise timeout_floor if many mutations are run twice.
A mutation keeps the fixed 30s limit when the baseline cannot vouch for its spec files: no baseline ran (--no-baseline), no spec file resolves for its source (it falls back to the whole suite, or is reported unresolved), one of its spec files failed in the baseline, or related_specs_heuristic adds spec files the baseline never ran. Every mutation keeps it when a worker_process_start, mutation_insert_pre or mutation_insert_post hook is configured: those run inside the time a mutation is allowed, and the baseline runs none of them.
--timeout N (or timeout: in the config file) gives every mutation the same N seconds, as before; pass --timeout 30 for the previous behaviour. The limit a timed-out mutation ran under is in the JSON report as time_limit. Under --jobs the pool gives up on a worker that has not answered within twice the longest time a mutation of the run may take, both of its runs together.
With --incremental, a cached timeout is reused only while the mutation would get no more time than the limit it was recorded under; raising --timeout, timeout_floor or the factor runs it again. Timeouts cached before the limit was recorded are run again once.
Incremental cache
--incremental (or incremental: true) keeps each mutation's verdict — killed, survived or timeout — in tmp/evilution_cache/ and reuses it on the next run instead of running the mutation again. A reused verdict is reported and scored like a fresh one; it carries no timings and no memory figures, since nothing ran. A cached timeout is reused only while the mutation would get no more time than the limit it was recorded under (see Per-mutation timeout).
Verdicts are kept per subject (a method, a constant, a declaration). A cached verdict is reused only while all of these are unchanged:
- the subject's own source;
- the rest of its file outside every method definition: class and module lines,
includes,attr_*s, other class-body code, every constant of the file, and every class-body declaration (callbacks, validations, scopes, AASM machines), since these decide whether and when the file's methods run; - the contents of the spec files that resolve for the file (
--spec,spec_mappings, or the resolver), and of the files the related-specs heuristic adds for the mutation when it is on; - the conventional helpers every spec file loads:
.rspec,spec/spec_helper.rb,spec/rails_helper.rb,spec/support/**/*.rb,test/test_helper.rb,test/support/**/*.rb, thepreloadfile, andGemfile.lock/gems.locked; - the evilution and Ruby versions;
- the settings that decide how a mutation is run and judged:
integration,isolation,profile,timeoutand the adaptive timeout'stimeout_factor/timeout_floor/timeout_ceiling,baseline, the example-targeting settings,related_specs_heuristic,fallback_to_full_suite,preload,stop_on_kill,killer_first,warmup, whether uncovered mutations are skipped (skip_uncoveredwith coverage targeting),probe_firstandcopy_worker_databases.
So editing one method re-runs that method's mutations only, and a method that only moved because a method above it grew or shrank keeps its verdicts. Editing a constant, a callback, validation or scope declaration, any other class-body line (comments between methods included) or a spec file re-runs every subject of the file. A method defined twice in one file (say, in both branches of an if) keeps a verdict per definition.
What the cache does not see: other methods the subject calls (in the same file or another), other source files, and files a spec loads beyond the helpers above (factories, fixtures, shared examples outside spec/support), as well as hooks. A change there keeps the cached verdicts, so a survivor can stay a survivor after a test it depends on indirectly was fixed, or the other way around. Run without --incremental (or delete tmp/evilution_cache/) for a result that owes nothing to earlier runs, e.g. in CI.
Not cached: uncovered mutations, which are worked out again from the coverage map on every run (it has a cache of its own); survivors implied by a --probe-first sentinel; mutations whose source has no spec file that resolves (with or without --fallback-full-suite), errors, neutrals, kills from an infrastructure crash, and kills in which only examples already failing in the baseline failed.
The cache holds one JSON file per source file under tmp/evilution_cache/v2/, read once per run and written once per source file, so its cost grows with the number of mutations, not with their square. A file keeps one version of each subject: writing it drops the verdicts of a subject's earlier source and of subjects the file no longer has. Cache files of the earlier format (directly in tmp/evilution_cache/, keyed by the whole file's content) are not read, and are deleted the first time the new cache is written.
Neutral Mutations
Neutral means the run has no verdict on a mutation. It covers two unrelated situations that want opposite responses, and each neutral records which; the report groups by it, naming the spec or the error class (GH #1606):
- The tests failed, but only on examples that were already failing before any mutation ran. Such an example fails whatever the mutation is, so its failure is not a kill. Nothing that was passing caught the mutation either, but the red example might have, once fixed.
- The test process died on infrastructure rather than on the mutation.
Score: 100.00% (10/10 verified of 17 mutations, 7 neutral)
! 7 kills not counted for spec/tally_spec.rb: only examples already failing in the baseline failed, so those mutations have no verdict.
./spec/tally_spec.rb[1:3] Tally adds -- NameError: undefined local variable or method 'tally'
Neutral mutations (7, not verified):
baseline already failing (spec/tally_spec.rb):
arithmetic_replacement: lib/tally.rb:9
integer_literal: lib/tally.rb:9
The score line names the remainder whenever the run left mutations out of the denominator, because full marks over a fraction of a run otherwise reads as a verdict on all of it. A clean run still prints the plain Score: 100.00% (17/17). The line under it says how many kills were not counted, for which spec file, and why the baseline was red for it — the failing examples with their first error line, or the error that stopped the file before any example ran. The same detail is printed to stderr once per red spec file as soon as the baseline finishes, and is in JSON output as summary.baseline_neutralized and summary.baseline_failures.
What a red baseline does not do:
- It does not touch a kill in which something that was passing failed too. That is a kill like any other.
- It does not touch survivors. A mutation whose tests passed is a survivor whatever the baseline did — including when the baseline's failure did not happen again in the mutation run (a flaky example, a failure of the baseline process itself). Earlier versions recorded every survivor covered by a red spec file as neutral, which hid exactly those gaps.
-
It does not second-guess a baseline failure with no failing example to name — a file that did not load, a timeout, a baseline process that died. A mutation run that hits the same problem reports
errorortimeoutitself.
Recognising an already-failing example needs the test framework to name the examples that failed, the same way in the baseline and in a mutation run. RSpec examples are named by spec file and position, Minitest tests as Class#test, Test::Unit tests as test(Class). Two Minitest test classes without a name that share a test method name cannot be told apart.
JSON output carries neutral_reason as { kind, detail } on neutral entries that have one; detail is null where no single spec can be named (an explicit --spec run), and the field is absent on a result recorded without a reason, which the text report shows as reason not recorded.
Per-Subject Scores
The run's score is computed per file, and test selection resolves per file too, so a well-tested file that gains new untested methods still reports 100% — the number only ever describes what the resolved spec reaches (GH #1605). Every report therefore breaks the run down per subject, and the text report names the subjects the file-level score does not speak for:
Mutations: 33 total, 7 killed, 0 survived, 0 timed out, 26 unresolved
Score: 100.00% (7/7)
Subjects needing attention (2 subjects in 1 file):
lib/helper.rb
#summary_for 0.00% (0/17) nothing reached this subject
#total_for 0.00% (0/9) nothing reached this subject
A subject is listed when something survived, or when nothing reached it at all — zero verdicts, every mutation unresolved or neutral. Fully-killed subjects are not listed, so the section stays actionable. JSON output carries every subject under subjects, whether or not it needs attention, so a CI step can assert on reached or on a per-subject score.
Mutation Operators (155 total)
Each operator name is stable and appears in JSON output under survived[].operator.
| Operator | What it does | Example |
|---|---|---|
arithmetic_replacement |
Swap arithmetic operators |
a + b -> a - b
|
comparison_replacement |
Swap comparison operators |
a >= b -> a > b
|
boolean_operator_replacement |
Swap && / ||
|
a && b -> a || b
|
short_circuit_to_eager |
Evaluate both operands of && / || / and / or (an operand that would regroup is parenthesized; an operand that only jumps, such as return, is skipped) |
a && b -> a & b, a || b -> a | b
|
boolean_operand_promotion |
Drop one side of && / ||
|
a && b -> a, b
|
boolean_expression_to_nil |
Replace a whole compound boolean with nil
|
a && b -> nil
|
boolean_literal_replacement |
Flip boolean literals |
true -> false
|
nil_replacement |
Replace nil with true, false, 0, ""
|
nil -> true
|
integer_literal |
Boundary-value integer mutations, plus one sentinel past the next integer width (int8, uint8, int16, uint16, int32, uint32, int64) above the literal's magnitude |
n -> 0, 1, n+1, n-1; 42 -> 167
|
float_literal |
Boundary-value float mutations, plus NaN and both infinities (not inside a pattern, where they do not parse) |
f -> 0.0, 1.0, Float::NAN, Float::INFINITY, -Float::INFINITY
|
complex_literal |
Replace a complex literal with 0i, 1i, its neighbours along the imaginary axis and nil (a rational part such as 3ri gets no neighbours) |
5i -> 0i, 1i, 6i, 4i, nil
|
rational_literal |
Replace a rational literal with 0r, 1r, its two neighbours and nil (the rational part of a complex literal such as 3ri is left to complex_literal) |
1.5r -> 0r, 1r, 2.5r, 0.5r, nil
|
string_literal |
Empty the string or replace it with nil; an interpolated string is also replaced as a whole; a backtick or %x command is replaced with nil only, its text never rewritten |
"str" -> "", nil; "a #{x}" -> "", nil; `ls` -> nil
|
array_literal |
Empty the array, replace it with nil, delete each element in turn (two or more elements; a heredoc element is kept), and replace a single-element array with its element (not a splat, bare keywords, a %w / %i word or a heredoc) |
[a, b] -> [], nil, [b], [a]; [x] -> x
|
hash_literal |
Empty the hash, replace it with nil, delete each key/value pair in turn (two or more elements; a **splat and a pair holding a heredoc are kept), and rename each label key in turn (a key written with => is left to symbol_literal / string_literal) |
{a: 1, b: 2} -> {}, nil, {b: 2}, {a: 1}, {__evilution_mutated__: 1, b: 2}
|
symbol_literal |
Replace with sentinel symbol or nil; an interpolated symbol is replaced as a whole with the empty symbol or nil
|
:foo -> :__evilution_mutated__, nil; :"a_#{x}" -> :"", nil
|
conditional_negation |
Replace condition with true/false
|
if cond -> if true
|
conditional_branch |
Remove if/unless/else branch | Deletes branch body |
if_branch_swap |
Replace the if-branch with the else body, drop the else |
if c; x; else; y; end -> if c; y; end
|
conditional_flip |
Flip if to unless and vice versa |
if cond -> unless cond
|
statement_deletion |
Remove statements from method bodies | Deletes a statement |
method_body_replacement |
Replace entire method body | Method body -> nil, self, super
|
negation_insertion |
Negate predicate methods |
x.empty? -> !x.empty?
|
return_value_removal |
Strip return values |
return x -> return
|
return_keyword_removal |
Drop the return keyword and keep its value, so control flow continues past a guard clause or early return (several values become an array; skips returns in tail position, where the value is the result anyway, and bare return) |
return :neg if x.negative? -> :neg if x.negative?
|
collection_replacement |
Swap collection methods |
map -> each, select <-> reject
|
collection_return |
Replace collection return values |
return [1] -> return []
|
scalar_return |
Replace scalar return values |
return 42 -> return 0
|
method_call_removal |
Remove method calls, keep receiver |
obj.foo(x) -> obj
|
argument_removal |
Remove individual arguments |
foo(a, b) -> foo(b)
|
argument_nil_substitution |
Replace arguments with nil
|
foo(a, b) -> foo(nil, b)
|
call_to_nil |
Replace a method call with nil (skips void statements, receivers of another call, and attribute or index writes) |
user.name -> nil
|
constant_read_to_nil |
Replace a constant reference with nil, a path as a whole (skips call receivers, void statements and rescue exception classes) |
MAX -> nil, Config::MAX -> nil
|
variable_read_to_nil |
Replace a read of a local, instance, class or global variable, or self, with nil (skips call receivers, void statements, if / unless / while / until conditions, left operands of && / ||, block-pass and double-splat arguments) |
total = count -> total = nil, build(name: @name) -> build(name: nil)
|
constant_namespace_strip |
Strip the namespace off a constant path, at each level of a longer one (skips top-level ::A paths and operator-write targets such as `A::B |
|
constant_write_to_nil |
Assign nil to a constant instead of its value. A constant assignment outside a method is its own subject, named after the constant, and the literal operators mutate its value too; code that read the constant while the file loaded keeps the old value (skips a constant already nil; Data.define / Struct.new assignments are left to data_struct_member) |
LIMIT = 10 -> LIMIT = nil
|
safe_navigation_removal |
Replace &. with a plain call (skips self and literal receivers) |
user&.name -> user.name
|
attribute_write_to_read |
Replace an attribute or index write with the matching read (skips writes statement_deletion already removes) |
a.foo = b -> a.foo, a[i] = b -> a[i]
|
argument_propagation |
Replace a call with its only positional argument (skips operator methods, attribute writes and void statements) |
normalize(value) -> value
|
argument_list_removal |
Drop a call's whole argument list, keeping any block (skips operator methods and attribute writes) |
foo(a, b) -> foo, foo(a, &b) -> foo(&b)
|
symbol_to_proc_replacement |
Apply the send_mutation and collection_replacement selector tables to a symbol block-pass (skips alias swaps) |
map(&:to_s) -> map(&:to_i)
|
dynamic_dispatch_resolution |
Resolve send / __send__ / public_send with a literal symbol into a direct call, only where visibility checks can differ |
obj.send(:m, a) -> obj.m(a)
|
keyword_argument_removal |
Drop one key: value argument at a call site (keeps **opts; skips a lone keyword) |
f(a: 1, b: 2) -> f(b: 2)
|
double_negation_removal |
Replace a double negation with its operand |
!!x -> x
|
regexp_anchor_to_predicate |
Rewrite a match against an anchored literal regexp as a prefix / suffix predicate (skips match? with \A / \z, an exact equivalent) |
x =~ /^foo/ -> x.start_with?("foo")
|
reduce_to_sum |
Rewrite a + reduction (reduce / inject with :+ or &:+) into sum
|
xs.reduce(:+) -> xs.sum, xs.inject(0, :+) -> xs.sum(0)
|
array_coercion_to_literal |
Replace an Array() coercion with an array literal |
Array(x) -> [x]
|
to_i_to_integer |
Replace lenient to_i with strict Integer() (skips numeric literal receivers) |
x.to_i -> Integer(x), x.to_i(16) -> Integer(x, 16)
|
coercion_emptying |
Replace a conversion with an empty value of its type (to_a / to_ary, to_h / to_hash, to_s / to_str) |
x.to_s -> "", x.to_h -> {}
|
const_get_to_constant_path |
Resolve const_get with a literal symbol into a constant path |
X.const_get(:Y) -> X::Y
|
proc_to_lambda |
Turn proc { } / Proc.new { } into lambda { } (arity checks, local return) |
proc { |x| x } -> lambda { |x| x }
|
dig_to_fetch_chain |
Make the first step of a multi-key dig strict |
x.dig(a, b) -> x.fetch(a).dig(b)
|
inequality_to_negated_identity |
Rewrite != with the stricter eql? / equal? (skips nil, boolean and symbol literals) |
a != b -> !a.eql?(b), !a.equal?(b)
|
binary_operand_promotion |
Replace an arithmetic or bitwise expression with one of its operands (skips integer identities and void statements) |
a + b -> a, b
|
receiver_constructor_swap |
Swap Date / DateTime / Time.parse for a stricter sibling constructor on the same receiver |
Date.parse(s) -> Date.iso8601(s), Date.strptime(s), ... |
block_body_to_nil |
Replace a block body with nil, keeping the iteration (skips loop and endless cycle / cycle(nil), which would hang) |
xs.each { |x| log(x) } -> xs.each { |x| nil }
|
block_body_to_raise |
Replace a block body with raise to prove the block is invoked (skips bodies with a rescue clause, which would swallow it) |
xs.each { |x| log(x) } -> xs.each { |x| raise }
|
block_body_promotion |
Replace a call with its parameter-less block body, run once (skips blocks with parameters or a rescue / ensure clause) |
Base.transaction { save! } -> save!, 3.times { poll } -> poll
|
data_struct_member |
Drop a member, or swap adjacent members, of a Data.define / Struct.new definition (a definition outside any method — Point = Data.define(...), class Coord < Struct.new(...) — is its own subject, named after the constant; skips single-member and splatted lists) |
Data.define(:a, :b) -> Data.define(:b), Data.define(:b, :a)
|
pin_operator_removal |
Drop the pin of a pattern variable so it captures instead of comparing (local variables only; skips pins inside alternative patterns, where a capture is not allowed) |
in ^expected -> in expected
|
rightward_assignment |
Turn a rightward pattern match into a pattern predicate, so a mismatch returns false instead of raising (skips a bare capture, which cannot fail) |
value => [a, b] -> value in [a, b]
|
numbered_parameter_swap |
Swap neighbouring numbered block parameters that the body reads (skips blocks reading a single one; it has nothing to swap with) |
pairs.map { _1 - _2 } -> pairs.map { _2 - _1 }
|
forwarded_argument_drop |
Stop forwarding one part of the arguments a method passes through unnamed; a ... signature is spelled out so a single part can be left out (skips a lone anonymous argument and named splats) |
g(*, **, &) -> g(**, &), def f(...) = g(...) -> def f(*, **, &) = g(*, **)
|
pattern_wildcard_widening |
Widen a pattern so it accepts shapes it rejects: append a rest to an array pattern, wildcard or drop a hash-pattern pair, remove **nil (skips pairs that bind a variable and patterns that already have a rest) |
in [a, b] -> in [a, b, *], in { age: Integer } -> in { age: _ }
|
no_matching_pattern_else |
Add an empty else to a case/in that has none, so an unmatched value yields nil instead of raising (skips a case/in with a bare capture clause, which matches everything) |
case v; in Integer then 1; end -> case v; in Integer then 1; else; end
|
argument_order_permutation |
Swap neighbouring positional arguments of a call, super or yield (keeps keywords, block and splats in place; skips identical neighbours, raise / fail, format / sprintf / printf, index writes, start_with? / end_with? and Set[...]). Calls that accept arguments in any order, such as OptionParser#on, can be silenced with ignore_patterns, e.g. call{name=on, receiver=local_variable_read{name=opts}}; the pattern ignores every mutation inside those calls, their handler blocks included. To keep a handler's logic mutated, write its body on its own lines (do |n| … end) and end the opts.on line with # evilution:disable instead |
compute(a, b) -> compute(b, a)
|
keyword_value_swap |
Swap the values of neighbouring keyword arguments, keeping the keys (keeps positional arguments, **splat and block in place; skips identical values and braced hash arguments; spells out shorthand keys) |
compute(x: a, y: b) -> compute(x: b, y: a)
|
comparison_operand_swap |
Swap the operands of a spaceship comparison, reversing a sort block or a custom ordering (skips identical operands and safe navigation; between? / clamp bounds are swapped by argument_order_permutation) |
x.age <=> y.age -> y.age <=> x.age
|
integer_division_to_fdiv |
Turn a division into fdiv, so integer division keeps its remainder (groups a compound dividend; skips float literals, /= and the explicit ./(...) form) |
a / b -> a.fdiv(b)
|
round_half_mode_swap |
Swap the tie-breaking mode of round for the other two, so a value sitting exactly on a tie rounds differently (literal :up / :even / :down only; skips a mode held in a variable; symbol_literal leaves these modes alone) |
amount.round(2, half: :even) -> half: :up, half: :down
|
off_by_one_boundary |
Lower a count held in a variable by one: the receiver of times, the argument of upto / downto / take / first / last / drop / each_slice / each_cons (skips literal counts, which integer_literal shifts, and safe navigation) |
n.times -> (n - 1).times, items.first(n) -> items.first(n - 1)
|
format_specifier_swap |
Loosen one conversion specifier of a literal format string in format / sprintf / printf / String#%: drop flags, width and precision, or render a non-integer number as %s (skips %%, %{name}, * widths, %c, %p and interpolated or heredoc strings) |
format("%05d", n) -> format("%d", n), "%.2f" -> "%f", "%s"
|
alias_removal |
Drop an alias or alias_method declaration from a class, module or class << self body, attributed to the first method of that scope (skips aliases inside methods, alias_method sent to another receiver and global-variable aliases) |
alias length size -> (removed)
|
freeze_removal |
Drop .freeze from a constant defined in a class or module body, attributed to the first method of that scope, or to the constant itself when the scope has no method of its own (skips values that are frozen anyway: symbols, numbers, nil, booleans, ranges, regexps, and string literals under # frozen_string_literal: true; a freeze inside a method is left to method_call_removal) |
LIST = %w[a b].freeze -> LIST = %w[a b]
|
visibility_removal |
Drop a private, protected or module_function declaration from a class, module or class << self body, attributed to the first method of that scope; a definition it wraps is kept (skips public, private_constant, private_class_method, and declarations guarded by a modifier) |
private -> (removed), private def a; end -> def a; end
|
attr_accessor_swap |
Widen an attr_reader or attr_writer to attr_accessor, and remove an attr_reader / attr_writer / attr_accessor declaration, in a class, module or class << self body, attributed to the first method of that scope (skips a declaration wrapped in private / protected, or guarded by a modifier) |
attr_reader :name -> attr_accessor :name, (removed)
|
sleep_to_zero |
Take the wait out of a sleep / Kernel.sleep whose duration is a variable or an expression (a literal duration is left to integer_literal / float_literal, which already emit sleep 0; skips a sleep sent to another receiver) |
sleep delay -> sleep 0
|
regexp_character_type_complement |
Flip a regexp character type or word boundary to its complement, one at a time (leaves \X, \R and Unicode properties alone) |
/\d+/ -> /\D+/, /\bword/ -> /\Bword/
|
regexp_anchor_promotion |
Promote a regexp line anchor to a string anchor, so the pattern stops matching around newlines (^ -> \A, $ and \Z -> \z) |
/^admin$/ -> /\Aadmin$/, /^admin\z/
|
regexp_alternation_branch_deletion |
Delete one branch of a regexp alternation at a time (skips alternations of more than ten branches, which are lookup tables, and deletions that would break a reference inside the pattern) |
/cat|dog/ -> /dog/, /cat/
|
regexp_quantifier_minimum_swap |
Swap a regexp quantifier between zero-or-more and one-or-more, keeping it lazy or possessive, so only the empty case changes |
/\A\d*\z/ -> /\A\d+\z/, /a+?/ -> /a*?/
|
regexp_capture_to_passive |
Turn a regexp capture group into a passive group, so $1, m[1] or \1 in a replacement loses its value (skips patterns passed to match? / !~, patterns with named groups, and patterns with numbered references) |
/id=(\d+)/ -> /id=(?:\d+)/
|
regexp_named_group_rename |
Rename a named regexp group, with its references inside the pattern, so m[:name] and the local bound by /(?<name>…)/ =~ s no longer find it (skips patterns passed to match? / !~) |
/(?<user>\w+)@/ -> /(?<_user>\w+)@/
|
regexp_option_removal |
Drop the i or m option of a regexp literal where it changes something: i with a cased letter in the pattern, m with a . outside a character class |
/admin/i -> /admin/, /a.b/m -> /a.b/
|
keyword_argument |
Remove keyword defaults/params |
def foo(bar: 42) -> def foo(bar:)
|
multiple_assignment |
Remove targets or swap order |
a, b = 1, 2 -> b, a = 1, 2
|
block_removal |
Remove blocks from method calls |
items.map { |x| x * 2 } -> items.map
|
block_pass_removal |
Remove block arguments passed with &
|
items.map(&:to_s) -> items.map
|
range_replacement |
Swap inclusive/exclusive ranges |
1..10 -> 1...10
|
regexp_mutation |
Replace regexp with always/never matching |
/pat/ -> /a\A/
|
regex_simplification |
Simplify regex quantifiers, anchors, ranges |
/\d+/ -> /\d/, /[a-z]/ -> /[az]/
|
receiver_replacement |
Drop explicit self receiver |
self.foo -> foo
|
send_mutation |
Swap semantically related methods |
detect -> find, map -> flat_map, round -> floor / ceil
|
compound_assignment |
Swap compound assignment operators, or replace the whole assignment with nil; on variables, attributes (obj.count += 1, obj.name ||= x) and indexes (counts[key] += 1, cache[key] ||= x) and, for operator writes, on constants (TOTAL += 1, a subject of its own like a constant assignment) |
+= -> -=, &&= -> ||=
|
operator_write_receiver_promotion |
Retarget an operator write on an attribute at the instance variable it goes through (skips other receivers, logical writes and index writes) |
@account.balance += x -> @account += x
|
memoization_break |
Recompute a memoized instance variable on every call (instance variables only; a local, @@x, $x or @cache[key] ||= is left alone) |
@x ||= expr -> @x = expr
|
local_variable_assignment |
Replace variable assignment with nil
|
x = expr -> x = nil
|
instance_variable_write |
Replace ivar assignment with nil
|
@x = expr -> @x = nil
|
class_variable_write |
Replace cvar assignment with nil
|
@@x = expr -> @@x = nil
|
global_variable_write |
Replace gvar assignment with nil
|
$x = expr -> $x = nil
|
mixin_removal |
Remove include/extend/prepend |
include Foo -> removed |
superclass_removal |
Remove class inheritance |
class Foo < Bar -> class Foo
|
rescue_removal |
Remove rescue clauses | Deletes rescue block |
rescue_body_replacement |
Replace rescue body with nil
|
Rescue body -> nil
|
rescue_handler_promotion |
Run a rescue handler instead of the code it protects, so the happy path never runs: each handler of a begin / method / block rescue in turn, and the fallback of a rescue modifier (drops else, keeps ensure; skips handlers that read the exception, re-raise or retry) |
begin; fetch(id); rescue NotFound; default; end -> begin; default; end, fetch(id) rescue default -> default
|
rescue_handler_concatenation |
Run a rescue handler after the protected code as well, keeping the rescue, so a successful run also does what the handler does (skips empty and value-only handlers, handlers that read the exception, re-raise or retry, and rescue modifiers) |
begin; fetch(id); rescue NotFound; rollback; end -> begin; fetch(id); rollback; rescue NotFound; rollback; end
|
rescue_else_concatenation |
Move the else body of a rescue into the protected code, so an error it raises is now rescued (only under a bare rescue or one naming StandardError / Exception; keeps the rescue and ensure) |
begin; a; rescue; b; else; c; end -> begin; a; c; rescue; b; end
|
rescue_class_widening |
Widen what a rescue clause catches, so a sibling error that should escape is handled too: the listed classes become StandardError, then Exception; a bare rescue, StandardError, or a core class outside it (Interrupt, LoadError, ...) becomes Exception only (skips a clause already naming Exception, and rescue modifiers) |
rescue KeyError => e -> rescue StandardError => e, rescue Exception => e
|
inline_rescue |
Remove inline rescue fallback |
expr rescue val -> expr
|
ensure_removal |
Remove ensure blocks | Deletes ensure block |
break_statement |
Remove break statements |
break -> removed |
next_statement |
Remove next statements |
next -> removed |
redo_statement |
Remove redo statements |
redo -> removed |
bang_method |
Swap bang with non-bang methods |
sort! -> sort
|
bitwise_replacement |
Swap bitwise operators |
a & b -> a | b
|
bitwise_complement |
Remove or swap ~
|
~x -> x, ~x -> -x
|
zsuper_removal |
Replace implicit super with nil
|
super -> nil
|
explicit_super_mutation |
Mutate explicit super arguments |
super(a, b) -> super
|
index_to_at |
Replace [] with .at() for arrays |
arr[0] -> arr.at(0)
|
index_to_fetch |
Replace [] with .fetch()
|
h[k] -> h.fetch(k)
|
index_to_dig |
Replace [] chains with .dig()
|
h[a][b] -> h.dig(a, b)
|
index_receiver_to_self |
Send an index read to self instead of its receiver (skips index writes and reads already on self) |
h[k] -> self[k]
|
index_to_key_predicate |
Replace an index read with a key? check (skips integer and range indexes, reads with several arguments, void statements and index writes) |
h[k] -> h.key?(k)
|
index_range_to_drop |
Replace a to-the-end range index with drop (also the endless a[n..] and a[n...]; skips a literal start of zero or below, ranges stopping short of the last element, void statements and index writes) |
a[n..-1] -> a.drop(n)
|
index_write_to_value |
Replace an index write with the value it assigns, dropping the store; the assignment syntax still evaluates to the same value, the a.[]=(k, v) / a&.[]=(k, v) spellings may not (skips void statements, operator writes and multiple-assignment targets) |
h[k] = v -> v
|
index_assignment_removal |
Remove []= assignments |
h[k] = v -> removed |
pattern_matching_guard |
Remove/negate pattern guards |
in x if cond -> in x
|
pattern_matching_alternative |
Remove/reorder alternatives |
pat1 | pat2 -> pat1
|
pattern_matching_array |
Remove/wildcard array elements |
[a, b] -> [a, _]
|
yield_statement |
Remove yield or its arguments |
yield(x) -> yield
|
splat_operator |
Remove splat/double-splat (skips ** in a hash literal, ** after a keyword argument or another **, and the rest of a pattern) |
foo(*args) -> foo(args)
|
defined_check |
Replace defined? with true
|
defined?(x) -> true
|
regex_capture |
Swap or nil-ify capture refs |
$1 -> $2, $1 -> nil
|
loop_flip |
Swap while/until loops |
while cond -> until cond
|
loop_body_to_raise |
Replace a loop body with raise
|
while c; body; end -> while c; raise; end
|
method_body_to_raise |
Replace a whole method body with raise
|
def foo; body; end -> def foo; raise; end
|
method_body_to_super |
Replace a method body with bare super where a super target exists |
def foo; body; end -> def foo; super; end
|
typed_default_return |
Replace a single-expression body with the empty value of its inferred type |
def names(u); u.map(&:name); end -> def names(u); []; end
|
block_parameter_drop |
Drop a block's single parameter | `users.each { |
optional_parameter_to_required |
Drop an optional positional parameter's default; of several, only the ones Ruby lets become required: the first, and the last unless a *rest parameter follows |
def f(a = 1) -> def f(a)
|
optional_default_injection |
Overwrite an optional parameter with its own default at the top of the body |
def f(a = 1); body; end -> def f(a = 1); a = 1; body; end
|
block_destructuring_expansion |
Flatten a destructuring group in a block's parameters | `pairs.each_with_index { |
forwarding_super_to_explicit |
Give a forwarding super an empty argument list |
def f(a); super; end -> def f(a); super(); end
|
string_interpolation |
Replace interpolation content with nil |
"hello #{name}" -> "hello #{nil}"
|
retry_removal |
Remove retry statements |
retry -> nil
|
case_when |
Remove/replace case/when branches | Remove when branch, drop one condition from when a, b, body -> nil, empty body -> raise, remove else
|
case_in |
Drop one in clause from a case/in, or its else
|
case x; in Integer; …; in String; …; end -> drops an arm; else -> NoMatchingPatternError
|
pattern_predicate |
One-line pattern match -> false
|
x in Integer -> false
|
predicate_replacement |
Replace predicate calls with booleans |
x.empty? -> true, x.empty? -> false
|
equality_to_identity |
Replace equality with identity check |
a == b -> a.equal?(b)
|
lambda_body |
Replace lambda body with nil |
-> { expr } -> -> { nil }
|
begin_unwrap |
Remove begin/end wrapper |
begin; expr; end -> expr
|
block_param_removal |
Remove explicit block parameter |
def foo(&block) -> def foo
|
last_expression_removal |
Strip trailing literal return from method body |
def foo?; warn; true; end -> def foo?; warn; end
|
argument_method_call_replacement |
Replace a method-call argument with its receiver |
fn(x.attr) -> fn(x)
|
MCP Server (AI Agent Integration)
Evilution includes a built-in Model Context Protocol server for direct tool invocation by AI agents (Claude Code, VS Code Copilot, etc.).
Setup
Create a .mcp.json file in your project root:
{
"mcpServers": {
"evilution": {
"type": "stdio",
"command": "bundle",
"args": ["exec", "evilution", "mcp"],
"env": {}
}
}
}Run the server through bundle exec so the project's Gemfile.lock decides which versions of evilution, your test framework and their shared dependencies get loaded. Started outside Bundler, RubyGems picks the newest installed copy of each gem, which your test framework may refuse — the run then stops at the proof-of-life canary with a gem activation conflict. Projects without a Gemfile can use "command": "evilution" and "args": ["mcp"].
The server exposes the following tools:
| Tool | Description |
|---|---|
evilution-mutate |
Run mutation testing on target files with structured JSON results |
evilution-session |
Inspect mutation testing history — action: list browses saved sessions, action: show displays one, action: diff compares two (fixed/new/persistent survivors, score delta) |
evilution-info |
Discovery before mutation — action: subjects lists the mutation subjects (methods, and class-body subjects such as constant assignments, scopes and callback declarations) with mutation counts, action: tests resolves which specs cover given sources, action: environment dumps the effective config, action: statuses returns the mutation-result status glossary, action: feedback returns the public Discussions URL plus consent + privacy guidance for posting feedback |
Verbosity Control
The evilution-mutate tool accepts a verbosity parameter to control response size:
| Level | Default | What's included |
|---|---|---|
summary |
Yes |
summary + survived + uncovered + timed_out + errors + unresolved (entries other than survived and uncovered shed diff and error_backtrace; the run's timings block and every entry's timings are dropped, to bound payload size) |
full |
All entries (killed/neutral/equivalent diffs stripped), including timings
|
|
minimal |
summary + survived + uncovered (plus a trimmed sample of up to 3 errored entries when errors > 0) |
Use minimal when context window budget is tight and you only need to see what survived (an uncovered mutation is a survivor no test executes, see Uncovered mutations). The trimmed errors sample (each entry: error_message, error_class, location, plus the first 5 backtrace lines) is added so a partly-broken run is still self-diagnosable without escalating verbosity. Use full when you need to inspect killed/neutral/equivalent entries for debugging.
What survives trimming matters when you are deciding whether to trust a score:
-
neutralentries — and with them eachneutral_reason— are dropped atsummaryandminimal. Usefullto see why mutations were neutralised. -
subjectsis kept atfullandsummary, and dropped atminimal, which keeps onlysummary,survivedanduncovered. - Everything inside
summarysurvives at every level, includingunresolved_target_files,uncovered_code,infra_retried,baseline_neutralized,baseline_failuresand theneutralcount — so even aminimalresponse still says whether a target file went untested and how much of the run the score covers.
Enriched Survived Entries
Unlike evilution --format json, every survived and uncovered entry returned by evilution-mutate carries extra fields so the agent can act without a second round-trip:
| Field | What it gives you |
|---|---|
subject |
Class#method for the mutated subject — points at the exact method to test |
spec_file |
Resolved spec/test path (when one exists) — e.g. an RSpec spec file or Minitest test file, so you can drop new tests straight into it |
next_step |
Concrete natural-language hint — "add a test in X that fails against this mutation at Y:line" |
These fields are added in addition to the existing operator, file, line, diff, unified_diff, suggestion, and test_command so agents can triage survivors in one pass.
Concrete Test Suggestions
The evilution-mutate tool accepts a suggest_tests boolean parameter (default: false). When enabled, survived mutation suggestions contain concrete test code that an agent can drop into a test file, instead of static description text. It currently generates RSpec-style suggestions (it/expect blocks).
Pass suggest_tests: true in the evilution-mutate call to activate this mode. The CLI also supports --suggest-tests; when using the CLI, generated suggestions match the --integration setting (RSpec it/expect blocks or Minitest def test_/assert_equal methods). Concrete templates for test-unit are not yet implemented — test-unit runs fall back to generic suggestion text.
Project Config File
evilution-mutate and evilution-info load .evilution.yml (or config/evilution.yml) by default, matching evilution CLI behavior — so timeout, jobs, integration, target, ignore_patterns, and other project settings carry over without the agent having to re-pass them on every call. Explicit tool parameters still win over file settings.
Pass skip_config: true to ignore the project config file. This skips loading .evilution.yml / config/evilution.yml, but MCP-specific overrides (JSON output, quiet mode, preload disabled) and explicit tool parameters still apply.
Iterative Workflow Parameters
evilution-mutate exposes the full set of CLI knobs agents need for iterative TDD:
| Parameter | Purpose |
|---|---|
incremental |
Cache verdicts (killed, survived, timeout) across runs — set true when iterating on the same files; editing one method re-runs only that method's mutations |
integration |
rspec, minitest, or test_unit
|
isolation |
auto, fork, or in_process
|
baseline |
false to skip the baseline suite check when you already know it's green |
save_session |
Persist results to .evilution/results/ for inspection via evilution-session
|
stop_on_kill |
false to run every selected example instead of stopping at the first deciding failure (default true) |
skip_sites |
Presets of low-value code to leave unmutated: logging, debug_output, error_messages, requires (see Skipping Low-Value Code); skips are counted per preset in summary.skipped_sites
|
since |
A git ref: mutate only the lines changed since it, as --since REF does (see Mutating only what changed) |
Note:
.mcp.jsonis gitignored by default since it is a local editor/agent configuration file.
After upgrading the gem: restart the MCP server
The MCP server is a long-lived stdio process spawned by the agent host. bundle update evilution swaps the gem on disk but the running process keeps the old code in memory — symptom is opaque "Internal error" responses to flags or shapes the old build doesn't recognize. Restart the server (reload the workspace in Claude Code / Copilot / etc.) so the new gem loads.
evilution version prints the gem version and the bundled mcp gem version on separate lines — run it in the same bundle the MCP server uses to confirm what's loaded.
Feedback channel
When evilution causes friction (errors, usage problems, missing capabilities you wish were there), the MCP responses include a feedback_url plus feedback_hint. The evilution-info tool also exposes action=feedback, which returns the channel URL and posting guidance on demand.
Agents must never post on the user's behalf without explicit user permission. Show the user exactly what you would post, get explicit approval, then post. Never include secrets, environment variables, the project name, file paths, source code, or class/method names from user code — the feedback channel is public.
Discussion URL: https://github.com/marinazzio/evilution/discussions
Contract stability
The three MCP tools (evilution-mutate, evilution-session, evilution-info) form evilution's public contract for AI agents. From 1.0.0 onwards the following are governed by the gem's SemVer policy:
-
Tool names:
evilution-mutate,evilution-session,evilution-info. -
Input schemas: every parameter listed in each tool's
input_schema(name, type, enum values,required). -
Action enumerations: the action enum on
evilution-session(list,show,diff) andevilution-info(subjects,tests,environment,statuses,feedback). - Output payload top-level shape: the keys and value types documented per action below.
-
Error envelope: an error response is a single text content with the response's
errorflag set to true. The body is a JSON object with at minimum anerrorkey shaped{ "type": <string>, "message": <string> }; tools may add additional top-level keys (e.g.evilution-mutateincludesfeedback_urlandfeedback_hintto point agents at the public Discussions channel). Consumers must read theerror.typediscriminator and ignore unknown extras. The error type strings — currentlyconfig_error,parse_error,not_found, andruntime_error— are part of the contract; new types may be added in MINOR releases (additive).
Output schema_version
Successful MCP responses carry a top-level schema_version integer. Two distinct version spaces are in play; both happen to be 1 today and either may be bumped independently at the next MAJOR release:
-
MCP contract
schema_version—Evilution::MCP::CONTRACT_VERSION(currently1). Stamped on envelopes that exist solely to wrap MCP tool output:evilution-infoaction responses,evilution-session list,evilution-session diff. Bumped only when the MCP envelope shape itself changes incompatibly. -
Session JSON
schema_version—Evilution::Session::Schema::CURRENT_VERSION(currently1). Embedded inside payloads whose shape is also written to disk and consumed elsewhere:evilution-mutate(returns a mutation report) andevilution-session show(returns a session JSON document). Bumped only when the report/session shape itself changes incompatibly.
Per-tool placement:
| Tool / action |
schema_version source |
Location in payload |
|---|---|---|
evilution-mutate |
Session::Schema::CURRENT_VERSION |
Top-level of the mutation report JSON (same shape as --save-session output). |
evilution-session list
|
MCP::CONTRACT_VERSION |
Top-level of envelope: { "schema_version": 1, "sessions": [...] }. |
evilution-session show
|
Session::Schema::CURRENT_VERSION |
Inside the returned session JSON document. |
evilution-session diff
|
MCP::CONTRACT_VERSION |
Top-level alongside summary / fixed / new_survivors / persistent. |
evilution-info (all actions) |
MCP::CONTRACT_VERSION |
Top-level of every successful response, injected by the action response formatter. |
Per-tool output shapes
-
evilution-mutate— full schema in the JSON Output Schema section. MCP-specific additions to eachsurvivedentry are documented in Enriched Survived Entries. -
evilution-sessionlist—{ "schema_version": Integer, "sessions": Array<{ file, timestamp, total, killed, survived, score, duration }> }. Sessions are reverse-chronological; the array is filtered bylimitwhen provided. -
evilution-sessionshow— the parsed session JSON document, exactly as written under.evilution/results/*.json. Field reference: see Session JSON files. -
evilution-sessiondiff—{ "schema_version": Integer, "summary": { base_score, head_score, score_delta, base_survived, head_survived, base_total, head_total, base_killed, head_killed }, "fixed": Array, "new_survivors": Array, "persistent": Array }. The mutation arrays carry the same per-mutation fields the sessionsurvivedlist uses (operator,file,line,subject,diff). -
evilution-infosubjects—{ "schema_version": Integer, "subjects": Array<{ name, file, line, mutations }>, "total_subjects": Integer, "total_mutations": Integer }. Discovery only: this lists what can be mutated. Thesubjectsarray in a run's report is a different shape, carrying what each subject scored. -
evilution-infotests—{ "schema_version": Integer, "specs": Array<{ source, spec }>, "unresolved": Array<String>, "total_sources": Integer, "total_specs": Integer }. -
evilution-infoenvironment—{ "schema_version": Integer, "version": String, "ruby": String, "config_file": String|null, "settings": Object }.settingsholds the effective value of each run setting under its config key, plusadaptive_timeout: when it istrue,timeoutis only the fallback and each mutation's limit follows the baseline (timeout_factor,timeout_floor,timeout_ceiling). -
evilution-infostatuses—{ "schema_version": Integer, "statuses": Array<{ status, meaning, counted_in_score }> }. -
evilution-infofeedback—{ "schema_version": Integer, "discussion_url": String, "consent": String, "privacy": String }.
Deprecation cycle
When a parameter, action, or output field on the public MCP contract is deprecated:
- The deprecation is announced in the CHANGELOG and the tool's
descriptiontext gains a deprecation note. - The deprecated form remains functional for the entire
1.xline — a deprecation introduced in1.Xcontinues to work in every subsequent1.X+Nrelease. - The earliest release that may remove the deprecated form is
2.0. Removals are listed in the major-release migration guide. - Adding new parameters, new actions, or new top-level output fields is additive and ships in MINOR releases (existing consumers continue to work).
Not covered by the contract
- The exact wording of error
messagestrings (thetypeis contract; the message is a hint that may be reworded). - The exact ordering of arrays whose contract calls only for membership (e.g.
unresolvedsource files). - Progress-stream notification payloads sent through
server_context.notifications— these are best-effort UI signals, not a stable API. - Performance characteristics (request latency, memory, parallelism) — improvements ship in any release; regressions are bugs but not contract violations.
- Default values that are part of the gem's user-facing semantics (timeout, integration default, etc.) — these follow the umbrella SemVer policy and may be tuned.
Recommended Workflows for AI Agents
1. Full project scan
bundle exec evilution run --target 'source:lib/**/*.rb' --format json --min-score 0.8Parse JSON output. Exit code 0 = pass, 1 = fail — either the score is below --min-score, or a target file resolved to no spec and was never tested (summary.unresolved_target_files names them). Without --min-score no score gate is armed; the run still fails on an untested target file.
2. PR / changed-lines scan (fast feedback)
bundle exec evilution run lib/foo.rb:15-30 lib/bar.rb:5-20 --format json --min-score 0.9Target the exact lines you changed for fast, focused mutation testing. See line-range syntax below.
Or let git find the lines: --since REF mutates only what changed since REF (see Mutating only what changed):
bundle exec evilution run --since origin/main --format json --min-score 0.93. Line-range targeted scan (fastest)
bundle exec evilution run lib/foo.rb:15-30 --format jsonTarget exact lines you changed. Supports multiple syntaxes:
evilution run lib/foo.rb:15-30 # lines 15 through 30
evilution run lib/foo.rb:15 # single line 15
evilution run lib/foo.rb:15- # from line 15 to end of file
evilution run lib/foo.rb # whole file (existing behavior)Methods whose body overlaps the requested range are included. Mix targeted and whole-file arguments freely:
evilution run lib/foo.rb:15-30 lib/bar.rb --format jsonA range is matched against subjects, and not every line belongs to one. Methods are subjects, and so are scopes, AASM guards and callbacks, callback and validation declarations given options or an inline block or lambda (validates :title, presence: true, validate :limit, if: -> { paid? }), Data.define / Struct.new definitions and constant assignments (see step 2 under Internals). Other class-body code — an association, a declaration naming only a method (validate :within_window), a DSL call evilution does not know — is not. When the range holds such code, the run says so on stderr, in the text report and in summary.uncovered_code:
[evilution] app/models/order.rb:12-14 holds code outside every subject; no mutations target those lines. Of the DSL calls in a class body, only scopes, aasm machines, and callback or validation declarations with options or an inline block or lambda are mutated.
Read 0 mutations next to that message as "not measured", not as "nothing to test".
Callback and validation declarations with options add to every model, controller and mailer that has them: about 10–20 mutations for each validates line. Some of those make the declaration invalid (length: {}, inclusion: { in: nil }, validates nil, ...), so the class raises while it loads, and they count as killed like any other mutation that stops the file loading. Each such mutation also runs the declaration once more as written, so option values computed in the class body (in: Status.pluck(:name)) are computed again, and a third-party validates_* macro that keeps its own list of attributes may list it once more. To leave a declaration out, put # evilution:disable on its line or around it (see Disable Comments). Declarations of one class that read the same — the conditions and the options of one declaration, or two validates :email lines — share one subject name, so subjects[] reports them together under one score.
4. Method-name targeted scan
bundle exec evilution run lib/foo.rb --target Foo::Bar#calculate --format jsonTarget a specific method by its fully-qualified name. Useful when you want to focus on a single method without knowing its exact line numbers.
5. Single-file targeted scan
bundle exec evilution run lib/specific_file.rb --format jsonUse when you know which file was modified and want to verify its test coverage.
5a. Multi-file batch scan
bundle exec evilution run lib/models/user.rb lib/models/account.rb lib/models/order.rbPass multiple file paths on a single invocation to amortise startup cost. The framework (Rails, Sorbet, etc.) and the preload chain (spec/rails_helper.rb → spec/spec_helper.rb → test/test_helper.rb) load once in the parent process. When --isolation=fork is selected (the default --isolation=auto resolves to fork on Rails projects and packaged gems), every subsequent mutation across all files forks from that warmed parent — materially faster than scripting a for f in ...; do bundle exec evilution run "$f"; done loop, which pays the bootstrap per file. With --isolation=in_process (default for non-Rails, non-gem projects under auto), there is no per-mutation fork, but the parent-process boot still runs once instead of N times. Per-file paths and line numbers are preserved in the report (survived[].file, HTML grouping by source file).
What to read before acting on a score
A score describes only the mutations that got a verdict. Five fields say what it leaves out, and each points at a different action:
| Field | Meaning | What to do |
|---|---|---|
summary.unresolved_target_files |
A file you named resolved to no spec and was never tested; the run fails on this alone | Write a spec, pass --spec, or map it in spec_mappings — do not trust the score until this is empty |
subjects[].reached == false |
Mutations were generated for that method but none got a verdict | The method is untested even where its file scores well; start here rather than with survived[]
|
summary.uncovered_code |
Lines you targeted hold code outside every subject — class-body code such as DSL calls evilution does not treat as a subject — so no mutation was generated for them; the same is printed to stderr and listed in the text report | A run that reports 0 mutations for those lines has not shown they are tested. Target the methods that code calls, or cover it with ordinary tests |
neutral[].neutral_reason.kind == "baseline_failure" |
The mutation's tests failed only on examples that were already failing before any mutation ran; detail names the red spec file |
Read summary.baseline_failures for why it was red, and fix that first — these mutations have no verdict until it is green |
neutral[].neutral_reason.kind == "infra_error" |
The test process crashed on infrastructure (DB lock, timeout); detail names the class |
Not a coverage gap. Give parallel workers their own database, or run -j 1
|
summary.infra_retried reports how many mutations had to be re-run serially because of the last case; a large number means the parallel run was fighting shared infrastructure rather than measuring your suite.
6. Fixing surviving mutants
Every entry in survived[] has already been re-run against the whole resolved spec file, so it is a gap the suite genuinely does not cover rather than an artefact of per-mutation example targeting.
For each entry in survived[]:
- Read
fileatlineto understand the code context - Read
operatorto understand what was changed - Read
suggestionfor a hint on what test to write (use--suggest-testsfor concrete test code) - Write a test that would fail if the mutation were applied
- Re-run evilution on just that file to verify the mutant is now killed
A test exists but the mutant still survives: check the matcher
Inside have_attributes, match and nested values of include(key: [...]), RSpec compares an expected array against an Enumerable actual (other than a Hash or Struct) through its to_a. ActiveModel::Errors#to_a is full_messages, so have_attributes(errors: ["Name can't be blank"]) passes for any errors object with that message, not just the one the code should return. A mutant that hands back a different errors object (another record's, or one built with a message instead of an error kind) survives that test.
When the errors object itself matters, assert on the error kinds, or on identity:
# Before: passes for any errors object whose full messages match
expect(result).to have_attributes(errors: ["Name can't be blank"])
# After: the error kind, not just its rendered message
expect(result.errors.details).to eq(name: [{ error: :blank }])
expect(result.errors).to be_of_kind(:name, :blank)
# or, when it must be the record's own errors object
expect(result.errors).to be(participation.errors)details and of_kind? still accept another record's errors with the same kinds; only the identity check (be) rules that out. The same to_a comparison applies to other Enumerable values (except Hash and Struct) checked against an array inside these matchers.
7. Diagnosing errored mutations
Entries in the JSON errors[] array represent mutations that raised an exception (syntax error, load failure, or runtime crash) rather than producing a test outcome. A mutation that makes its own file raise while loading is not among them when the original file still loads right after it: the mutation is what broke the load, so it is reported killed with an error_message that starts with load failed:. A load failure stays an error when the original fails to load as well, since that says nothing about the mutation. Each entry includes error_class, error_message, and the first 5 error_backtrace lines. Use these fields to decide whether the error is a bug in the mutation operator (file an issue), a load-time problem in the mutated source (often NoMethodError: super called outside of method or constant-redefinition issues), or a genuine crash that the original tests should have caught. Run with --verbose to stream the same error details to stderr during the run.
A test process that dies on an exception no handler saw (a stack overflow that does not unwind, for one) reports the exception before it goes. What the mutation gets depends on when it died:
error_message starts with |
Isolation | Status | Meaning |
|---|---|---|---|
child died, then while running the tests
|
fork | killed |
The mutated source loaded and the tests were running: what a crashing test does. |
test run ended by |
in-process | killed |
The same, in the process of the run. |
child died, then before the tests ran
|
fork | error |
Nothing was tested yet. error_class is the exception's class. |
Three messages are not about the mutation at all. No verdict was reached, and the entry says only why:
error_message |
Meaning | What to do |
|---|---|---|
child exited 0; its result was unreadable (...) (the status varies) |
Fork isolation. The child finished, but what arrived on its result pipe does not decode to a result; the parentheses say what was found. Every process the child forks inherits that pipe, so code under test that writes to a descriptor it did not open can put bytes on it. | Re-run that one mutation (line-range targeting). If it repeats, look for code that writes to inherited descriptors. |
worker's result could not be read; the worker was replaced |
Worker pool (--jobs above 1). The same thing on the pipe between the run and a worker: the worker was killed and replaced, and the run went on. |
Re-run that one mutation, with --jobs 1 to take the pool out of the picture. |
child died on SIGABRT without writing a result |
Fork isolation. The Ruby VM aborted (a [BUG] report, typically), so nothing could be reported from inside the child. Other signals read the same way: a segfault is SIGSEGV, a child the kernel killed for memory is SIGKILL. |
Re-run that one mutation. If it repeats, the mutant crashes the interpreter: that is Ruby's to fix, and the mutation can be disabled with a comment. |
The messages in the first column of this last table are examples: the exit status, the signal and the text in parentheses vary. Search errors[] for was unreadable, could not be read and without writing a result.
Under --isolation in_process there is no child to lose: a mutation that takes the Ruby process down takes the run with it, and no report is written. See Exit Codes.
Unresolved mutations — behaviour-named spec layouts
SpecResolver auto-detects mirrored (spec/foo_spec.rb), non-mirrored (spec/unit, spec/lib, test/unit), and dir-grouped (test/unit/<class>/*_test.rb) layouts. It deliberately does not guess for behaviour-named layouts — where specs are named by behaviour rather than by lib path, so no path or class name ties a source file to a spec. The canonical example is aasm: lib/aasm/base.rb is the central DSL class, but it has no base_spec.rb; its behaviour is exercised through the public API by spec/unit/{event,callbacks,guard,api}_spec.rb, none of which name AASM::Base. Every mutation in such a file resolves to no spec and is reported :unresolved (a coverage gap, not a failure — errors=0).
This is intentional: a fuzzy content/grep match would pick a wrong-but-plausible spec subset and report a misleading score. :unresolved plus the warning is the honest signal. To get a real score, opt into one of the two explicit recovery paths the warning names:
# run a directory of specs you know covers the file (--spec-dir globs it)
bundle exec evilution mutate lib/aasm/base.rb --spec-dir spec/unit
# or name explicit files as a single comma-separated --spec value (no shell glob)
bundle exec evilution mutate lib/aasm/base.rb --spec spec/unit/event_spec.rb,spec/unit/callbacks_spec.rb
# or run the whole suite per mutation (slower, but correct-by-superset)
bundle exec evilution mutate lib/aasm/base.rb --fallback-full-suiteTo make the mapping permanent, declare it once in .evilution.yml under spec_mappings; the run, its baseline and tests list all honour it. evilution tests list <file> prints exactly the spec files a run would use, so it is the quick way to check a mapping before spending a run on it:
spec_mappings:
lib/aasm/base.rb:
- spec/unit/event_spec.rb
- spec/unit/callbacks_spec.rbWithout --fallback-full-suite an unresolved source is also left out of the baseline, which does not run the whole test directory for it.
Long Minitest fork runs — not a hang
Minitest projects under --isolation=fork re-bootstrap the test environment (test_helper.rb, plugins, runnable state) once per mutation. On constant-heavy files (e.g. Shopify/liquid's lib/liquid/lexer.rb, ~270 mutations) the wall-clock cost is dominated by that per-fork bootstrap and any mutations that hit a --timeout rather than killing the test fast. A single-worker run (-j 1) on a few hundred mutations can take 4+ minutes; combined with --no-progress and a non-TTY stderr (CI, redirected logs) the run looks silent the entire time.
Recommended invocation for Minitest fork canaries:
RUBYOPT="-Itest" bundle exec evilution mutate lib/<file>.rb \
-j 4 -t 10 \
--integration=minitest --isolation=fork \
--spec test/<dir>/<file>_test.rb-j 4 parallelises across workers, -t 10 caps any mutation that pathologically loops at 10 s. On a Rails app, add --warmup rails so each fork does not re-initialise I18n, routes and templates on its first request (see Warming up Rails before forking). Expect the run to print progress only when stderr is a TTY (use bundle exec evilution mutate ... 2>&1 | tee log to get progress while still saving output). The historical "Minitest fork hangs on liquid" report (GH #1211) turned out to be a slow run + silent UX, not an actual deadlock — the worker logs show steady forward progress when captured via --quiet-children --quiet-children-dir DIR.
8. CI gate
bundle exec evilution run --target 'source:lib/**/*.rb' --format json --min-score 0.8 --quiet
# Exit code 0 = pass, 1 = fail, 2 = errorExit 1 covers two conditions: the score missed --min-score, and a target file that resolved to no spec. The second fails the run whether or not a score gate is set, so a CI step that names files explicitly cannot silently stop testing one of them.
Note: --quiet suppresses all stdout output (including JSON). Use it in CI only when you care about the exit code and do not need JSON output. --output FILE is the alternative when a preloaded spec helper (SimpleCov, for example) writes to stdout on exit.
9. Regression tracking across runs (compare)
bundle exec evilution run --target 'source:lib/**/*.rb' --format json --save-session
# later, after edits:
bundle exec evilution run --target 'source:lib/**/*.rb' --format json --save-session
bundle exec evilution compare \
--against .evilution/results/<earlier>.json \
--current .evilution/results/<later>.json \
--format jsonOutput buckets:
| Bucket | Meaning |
|---|---|
fixed |
Survived previously, killed now — the new test actually landed |
new |
Did not exist previously, surviving now — a fresh gap |
persistent |
Survived in both runs — carry-over debt |
reintroduced |
Killed previously, survived now — regression |
flaky |
Status flipped and back — unstable test |
Use in CI to gate merges on reintroduced being empty, or to surface new survivors for reviewer attention without failing the build on persistent debt.
Warming up Rails before forking
Under isolation: fork the parent preloads once and every mutation runs in a fresh fork, which inherits only what the parent had already initialised. Rails initialises a lot lazily on a process's first request: loading locale files, building route helpers, resolving asset paths, compiling templates. Without a warm-up, every forked mutation whose tests make a request pays that cold start again, often several hundred milliseconds each.
warmup: rails (or --warmup rails) does that initialisation once in the parent, right after the preload:
I18n.eager_load!- the route set's
url_helpers(andeager_load!for lazily drawn routes on Rails 8) - an asset path lookup through the controller helpers
- template precompilation, only if the
actionview_precompilergem is installed
Each step is skipped if its library is not loaded, and skipped with a one-line note if it raises, so an app that does not fit a step still runs normally.
Do not warm up by sending a real request from your preload (Rails.application.call(...)). Controller callbacks change process-global state that every fork then inherits: I18n.locale, gem request stores such as PaperTrail's, a session row in the database. Tests that depend on that state then fail in every fork, and evilution reports their mutations as killed. The steps above make no request and leave that state untouched.
Parallel Runs with SQLite
Running with -j N forks worker processes. If your Rails app uses SQLite and every worker opens the same db/test.sqlite3 file, concurrent writers collide on the database-level lock (ActiveRecord::StatementTimeout, SQLite3::BusyException, slow runs), and, worse, one worker's fixture loads and writes land in the middle of another worker's tests. Verdicts then depend on which mutations happen to run at the same time: on a Redmine model, a shared database turned 47 of 500 survivors into kills at -j 4 that a -j 1 run does not reproduce.
Per-worker database copies
By default each worker gets a database of its own, with no setup. When the app is preloaded in the parent (the default for a Rails app under fork isolation, see --preload) and --jobs is above 1, a worker that would share the preloaded SQLite test database makes a copy of it when it starts and reconnects ActiveRecord::Base to the copy. The copy goes through SQLite's backup API, so it holds what the write-ahead log holds and is no larger than the data; it is taken after the preload and the baseline, from a database no worker writes to, so every worker starts from the same schema and data. A worker started in place of a dead or recycled one makes a fresh copy. The ActiveRecord::Base.configurations entry is pointed at the copy too, so establish_connection(:test) in a test stays on it.
The copies live in a directory under the system temp dir (evilution-worker-db-<pid>-<random>/worker-<worker pid>/) and are deleted when the parallel pass ends, also when it ends on an error or Ctrl-C. A worker's copy is deleted as soon as the worker is gone and before its replacement starts, so the disk never holds more than one copy per job; only a run killed with SIGKILL leaves the directory behind. The database in database.yml is not written by the workers.
A worker copies only when all of these hold:
- the app was loaded by the preload, so the worker inherits an
ActiveRecord::Baseconnection; - that connection still matches
database.yml(a connection your test helper set up by other means is left alone); - the adapter is SQLite and the database is a file path, not
:memory:(never shared anyway) or afile:URI; - the worker would otherwise use the preloaded database:
database.ymlnames one database for all workers, or the worker's ownTEST_ENV_NUMBERdatabase (below) is not prepared.
Not covered yet: a run with --no-preload (the app boots in each mutation's process and reads database.yml there, so use TEST_ENV_NUMBER below), connections other than ActiveRecord::Base's (other databases set up with connects_to, replicas keep using the shared databases, and each worker says so in a warning), and PostgreSQL or MySQL, where a per-worker database would come from a template database; for those, set up per-worker databases with TEST_ENV_NUMBER. If a copy fails (a full disk, a locked database), the worker warns and shares the preloaded database as before.
Turn it off with --no-copy-worker-databases or copy_worker_databases: false. The setting is part of the incremental cache fingerprint, since it can change verdicts.
Infrastructure crashes
Contention that still happens (copies off, or a run the copies do not cover) is handled after the fact. Evilution classifies these crashes as :neutral (see GH #814) so the mutation score is not polluted, and because the contention only exists while several workers are running, those mutations are re-run one at a time once the pool is done, and the verdict from the quiet re-run is the one reported. Without it the neutral bucket moved with -j on identical input — the same files scoring 24 killed / 0 neutral at -j 1 and 6 killed / 18 neutral at -j 4 (GH #1607). The run says how much it had to redo:
! 18 mutations hit infrastructure errors under parallel workers; re-ran them serially.
The count is in JSON output as summary.infra_retried. Such a crash is also never written to the --incremental cache: the cache keeps no error class, so a cached :killed would be indistinguishable from a real one on the next run and would short-circuit both the demotion and the retry. A neutral from a failing baseline is never re-run — that is a real statement about the spec, not a missed verdict. The retry costs wall-clock time in proportion to the contention, and it does not undo a wrong kill: a test that failed on another worker's data rather than on a lock is not an infrastructure crash. Give each worker its own database instead.
Your own per-worker databases (TEST_ENV_NUMBER)
Evilution follows the parallel_tests convention: each worker receives a TEST_ENV_NUMBER environment variable ("" for worker 1, "2" for worker 2, "3" for worker 3, …). Interpolate it into config/database.yml so each worker gets its own database:
test:
adapter: sqlite3
database: db/test<%= ENV['TEST_ENV_NUMBER'] %>.sqlite3
pool: 5
timeout: 5000Each worker needs its own prepared database (db/test.sqlite3, db/test2.sqlite3, …, one per job). Create and load them before running Evilution:
bin/rails db:test:prepare
TEST_ENV_NUMBER=2 bin/rails db:test:prepare
TEST_ENV_NUMBER=3 bin/rails db:test:prepare # … up to the number of jobsWithout --preload, a test helper that loads the schema when it boots (for example ActiveRecord::Migration.maintain_test_schema! in rails_helper.rb) can do this instead, because it runs inside each worker.
Per-worker databases also work with --preload, which boots the app (and resolves database.yml) once in the parent before the workers fork: each worker re-renders database.yml with its own TEST_ENV_NUMBER and, when the result differs, re-establishes ActiveRecord::Base's connection to it. Only the primary connection is switched, and only when it still matches database.yml. Under --preload the preload's own schema setup runs once, in the parent, against the parent's database only, so prepare the worker databases with the commands above. A worker whose database is missing or lacks a table or migration of the parent's database goes back to the parent's database and prints which database to prepare; with per-worker copies on, it then uses a copy of the parent's SQLite database, otherwise its mutations share the parent's database as they would without the switch. Worker 1's TEST_ENV_NUMBER is empty, so its database is the parent's; with copies on it uses a copy of it, like any worker left on the parent's database.
When Evilution detects a parallel run against a SQLite test database whose path does not depend on TEST_ENV_NUMBER and the per-worker copies do not cover it (they are off, the app was not preloaded, or the preloaded connection is not one they copy: a file: URI, or a connection that no longer matches database.yml), it prints a one-time startup warning pointing to this section. An in-memory database (:memory:) is never shared between workers and does not trigger it.
Development
Architecture
New to the codebase? docs/architecture.md is the contributor guide: the end-to-end mutation flow (source → result), a module map, and step-by-step recipes for adding a mutator, a reporter, or a test integration.
Memory leak check
Run before releasing to verify no memory regressions:
bundle exec rake memory:checkTests 4 paths (InProcess isolation, Fork isolation, mutation generation + stripping, parallel pool) by running repeated iterations and asserting RSS stays flat. Configurable via environment variables:
-
MEMORY_CHECK_ITERATIONS— number of iterations per check (default: 50) -
MEMORY_CHECK_MAX_GROWTH_KB— maximum allowed RSS growth in KB (default: 10240 = 10 MB)
Internals (for context, not for direct use)
For the full contributor architecture — module map, data flow, and extension points — see docs/architecture.md.
- Parse — Prism parses Ruby files into ASTs with exact byte offsets
-
Extract — Methods are identified as mutation subjects, and so are value-object definitions outside any method (
Point = Data.define(:x, :y),class Coord < Struct.new(:lat, :lng)), which only the operators that apply to them mutate, every other constant assignment outside a method (LIMIT = 10,LIMIT += 1), named after the constant (Config::LIMIT);constant_write_to_nilmutates a plain assignment,compound_assignmentan operator write, and the literal operators the value of either, and ActiveRecord scopes declared with a literal body in a class body (scope :recent, -> { where(recent: true) }), named after the class method they define (Order.recent) and mutated like a method body — also when declared in a concern'sincluded do ... endorprepended do ... endblock, where they are named after the concern (Publishable.published). A method defined in a concern'sincluded do ... endorprepended do ... endblock is mutated on the classes that took the concern before the run as well as on those that take it later; likewise the guards and callbacks written out as lambdas or blocks inside an AASMeventorstatedeclaration (event :ship, guard: -> { address? }), named after the method that declaration defines (Order#ship,Order#paid?), or after the concern when the machine is declared in itsincluded do ... endorprepended do ... endblock (Shippable#ship); and the conditions and bodies written out in callback and validation declarations of a class body or of a concern'sincluded do ... endorprepended do ... endblock (validate :credit_limit, if: -> { paid? },before_save { ... }), named the way the declaration reads (Order.validate(:credit_limit),Order.before_save,Publishable.validates(:title)); and the arguments of such a declaration given options (validates :for_lift, inclusion: { in: [true] }, if: :lift_round?), under the same name — its literals, option values and conditions are mutated and options dropped one at a time. Such a mutation is applied by running the declaration again: what it registered when the file loaded, found by running the declaration as written once more, gives its place to what the mutated declaration registers, or to nothing when it registers none (presence: false) -
Filter — Disable comments, Sorbet
sigblocks, and AST ignore patterns exclude mutations before execution -
Mutate — 155 operators produce text replacements at precise byte offsets (source-level surgery, no AST unparsing); heredoc literal text is skipped by default. Identical byte-mutations from different operators are deduplicated by
(file_path, mutated_source)so the count is not inflated by overlap -
Isolate — Mutations are applied to temporary file copies (never modifying originals); load-path redirection ensures
requireresolves the mutated copy. Default isolation is in-process for plain Ruby projects (no gemspec) and fork for Rails projects and packaged gems (auto-detected);--isolation forkforces forked child processes. Both sequential and parallel (--jobs N) modes respect the configured isolation strategy - Test — The configured test framework (RSpec, Minitest, or Test::Unit) executes against the mutated source
- Collect — Source strings and AST nodes are released after use to minimize memory retention
- Report — Results aggregated into text, JSON, or HTML, including efficiency metrics and peak memory usage
Versioning
evilution follows Semantic Versioning. The full policy — what counts as the public contract, what triggers a major bump, how deprecations work — is documented in docs/versioning.md.
There is no public Ruby API: the entire Evilution:: namespace is internal and may change in any release. Drive evilution through the CLI, .evilution.yml, the MCP tools, or hooks. See docs/public_api.md.