Project

evilution

0.01
The project is in a healthy, maintained state
Evilution is a mutation testing tool for Ruby. It validates test suite quality by making small code changes and checking if tests catch them. AI-agent-first design with JSON output, diff-based targeting, and coverage-based filtering.
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
2015
2016
2017
2018
2019
2020
2021
2022
2023
2024
2025
2026
 Dependencies

Runtime

>= 0.8, < 2
>= 1.5, < 2
>= 1.5, < 3
>= 2.9, < 3
 Project Readme

Evilution mascot

Gem Version

Evilution — Mutation Testing for Ruby

Purpose: Validate test suite quality by injecting small code changes (mutations) and checking whether tests detect them. Surviving mutations indicate gaps in test coverage.

  • License: MIT (free, no commercial restrictions)
  • Language: Ruby >= 3.3
  • Parser: Prism (Ruby's official AST parser, ships with Ruby 3.3+)
  • Test frameworks: RSpec, Minitest, and Test::Unit

Installation

Add to Gemfile:

gem "evilution", group: :test

Then:

bundle install

Or standalone:

gem install evilution

Requires prism >= 1.5, < 2. Older Rails apps (e.g. Rails 7.1 pins prism 0.19) must upgrade prism — the gemspec constraint forces bundler to resolve a compatible 1.x version. If your app pins prism 2.x, bundler will reject the install until evilution widens its upper bound.

Installing on Rails 7.1 + Ruby 3.3

Two Bundler conflicts hit fresh installs on this stack:

  1. cgi activation conflict. Rails 7.1's Gemfile.lock pins cgi 0.5.0. Ruby 3.3.x ships cgi 0.5.1 as a default gem. Loading evilution via Bundler aborts with:

    Gem::LoadError: can't activate cgi-0.5.1, already activated cgi-0.5.0.
    Make sure all dependencies are added to Gemfile.
    
  2. prism pin. Same lockfile pins prism 0.19, which lacks the IfNode#subsequent accessor evilution uses (older Prism releases exposed it as consequent). Symptom: NoMethodError: undefined method 'subsequent' for an instance of Prism::IfNode.

Both resolve cleanly with a sidecar Gemfile.local that re-evaluates the project Gemfile and adds evilution + prism on top — no edits to the main Gemfile.lock:

# Gemfile.local
eval_gemfile("Gemfile")

group :test, :development do
  gem "evilution"
  gem "prism", "~> 1.5"
end

Then invoke evilution against that Gemfile:

BUNDLE_GEMFILE=Gemfile.local bundle install
BUNDLE_GEMFILE=Gemfile.local bundle exec evilution run lib/foo.rb

The first command writes a sibling Gemfile.local.lock. Decide whether to commit or .gitignore it the same way you would for any developer-only Gemfile — typically gitignored when only one or two engineers run mutation testing locally, committed when CI also runs evilution against the sidecar Gemfile.

The evilution gemspec already declares prism >= 1.5, < 2, so adding the gem "prism" line above is only necessary on stacks that also pin prism in Gemfile.lock.

Migrating from mutant

Coming from the commercially-licensed mutant gem? See docs/migration-from-mutant.md for a command/config mapping, output-terminology differences, and how to diff the two tools' results with evilution compare (it ingests mutant session JSON directly).

Command Reference

evilution [command] [options] [files...]

The shorter alias evil ships alongside evilution and accepts identical arguments (handy with alias be='bundle exec' → be evil run ...).

Every command, subcommand, and flag listed in this section is part of evilution's public CLI contract; see docs/versioning.md for stability and deprecation rules.

Commands

Command Description Default
run (alias mutate) Execute mutation testing against files Yes
init Generate .evilution.yml config file
version Print version string
subjects [files] List mutation subjects with locations and counts
tests list [files] List the spec files run would use for each source (layout, spec_mappings, spec_pattern)
session list List saved session results
session show FILE Display detailed session results
session diff A B Compare two sessions (fixed/new/persistent)
session gc --older-than D Garbage-collect sessions older than D (e.g. 30d)
util mutation Preview mutations for a file or inline code
environment show Display runtime environment and settings
compare --against A --current B Compare two saved session JSON files into fixed / new / persistent / flaky / reintroduced buckets

Options (for run command)

Flag Type Default Description
-t, --timeout N Integer (adaptive) Per-mutation timeout in seconds, the same for every mutation. Without it the limit adapts to the baseline, see Per-mutation timeout.
--timeout-factor K Float 10 Without --timeout: limit each mutation to K × the baseline time of its spec files.
--timeout-floor N Float 10 Without --timeout: the shortest such limit, in seconds.
--timeout-ceiling N Float 60 Without --timeout: the longest such limit, in seconds.
-f, --format FORMAT String text Output format: text, json, or html.
--target EXPR String (none) Only mutate matching subjects. Supports method name (Foo::Bar#calculate), class (Foo, which also matches a value-object constant named Foo), namespace wildcards (Foo::Bar*), method-type selectors (Foo#, Foo.), descendants (descendants:Foo), and source globs (source:lib/**/*.rb).
--since REF String (none) Only mutate the lines changed since git REF (a branch, tag or commit), diffed from its merge base with HEAD, uncommitted and untracked changes included. Without file arguments: every changed .rb file under lib/ and app/. With them: only those that changed, line ranges narrowed to their changed lines. Also accepted by subjects. See Mutating only what changed.
--output FILE String (stdout) Write the report to FILE instead of stdout. Useful when a preloaded spec helper writes to stdout on exit.
--min-score FLOAT Float 0.0 Minimum mutation score (0.0–1.0) to pass.
--spec FILES Array (none) Spec files to run (comma-separated). Defaults to auto-detection via SpecResolver, which also resolves non-mirrored (spec/unit, test/unit), dir-grouped (test/unit/<class>/*_test.rb), and flat test_-prefixed (test/test_connection_pool_timed_stack.rb) layouts.
--spec-dir DIR String (none) Include all *_spec.rb files in DIR recursively. Composable with --spec.
--spec-pattern GLOB String (none) Restrict resolved spec candidates to files matching GLOB (e.g. spec/models/**/*_spec.rb).
--no-example-targeting Boolean (enabled) Disable per-mutation example targeting (always run every example in the resolved spec file). Example targeting scans each example body for symbols from the mutated method and runs only the matching subset.
--example-targeting MODE String lexical How targeting picks examples: lexical (name-grep example bodies for the mutated method/class), coverage (run only the examples that actually execute the mutated line, from a cached line-coverage map), or full_file (run all resolved examples).
--example-targeting-fallback MODE String full_file Behavior when no example matches: full_file (run the whole spec file) or unresolved (skip the mutation as :unresolved).
--[no-]skip-uncovered Boolean (enabled) With --example-targeting coverage: report a mutation whose changed lines no example executes as uncovered, without running it. See Uncovered mutations.
--[no-]probe-first Boolean (disabled) Experimental. Run each method's body-to-raise mutant first; when it survives, report the method's other mutants as survivors implied by it, without running them. See Probe-first.
-j, --jobs N Integer 1 Number of parallel workers. Uses demand-driven work distribution with pipe-based IPC.
--no-baseline Boolean (enabled) Skip baseline test suite check. By default, a baseline run detects pre-existing failures, so that a mutation whose tests fail only on those is recorded neutral rather than killed.
--[no-]canary Boolean (enabled) Run a proof-of-life synthetic mutation at session start; abort the run if the pipeline misreports it. Catches misconfigured isolation, broken autoload, and reporter-plugin eviction before any real score is produced. Pass --no-canary to skip (e.g. CI speed, or when the canary itself is the thing under test).
--[no-]stop-on-kill Boolean (enabled) Stop each mutation's test run at the first failure that decides the verdict. Disable to run every selected example.
--[no-]killer-first Boolean (disabled) RSpec, with stop-on-kill: run first the examples that killed nearby mutations, then the cheapest. Only for suites whose examples pass in any order. See Killer-first order.
--fail-fast [N] Integer (none) Stop after N surviving mutants (default 1 if no value given).
-v, --verbose Boolean false Verbose output with RSS memory and GC stats per phase and per mutation; also prints error class, message, and first 5 backtrace lines for errored mutations.
--suggest-tests Boolean false Generate concrete test code in suggestions (RSpec or Minitest, based on --integration; falls back to generic phrasing for test-unit).
-q, --quiet Boolean false Suppress output.
--stdin Boolean false Read target file paths from stdin (one per line).
--integration NAME String rspec Test framework integration: rspec, minitest, or test-unit. See docs/integrations.md.
--[no-]incremental Boolean false Cache each mutation's verdict (killed, survived, timeout) and skip it on re-runs until its subject, its spec files or the run's settings change — see Incremental cache. Pass --no-incremental to override incremental: true from the config file for one invocation (e.g. cold-cache debugging). Last flag wins when both are given.
--save-session Boolean false Persist results as timestamped JSON under .evilution/results/.
--no-progress Boolean (enabled) Disable the TTY progress bar.
--quiet-children Boolean false Redirect each forked worker's stdout/stderr to per-pid files under tmp/evilution_children/<pid>.{out,err} so noisy app initializers (Datadog, Bullet, etc.) don't merge with parent output. Trade-off: live worker errors only appear in the side files, not the terminal — tail -f tmp/evilution_children/*.err to watch them.
--quiet-children-dir DIR String tmp/evilution_children Override the directory used by --quiet-children.
--isolation MODE String auto Isolation strategy: auto, fork, or in_process. auto selects fork for Rails projects and packaged gems (*.gemspec), in_process otherwise. Under in_process a mutation that takes the Ruby process down ends the run (what in_process cannot contain). See docs/isolation.md.
--preload FILE String (auto) File to require in parent before forking workers. Auto-detect chain for Rails projects: spec/rails_helper.rb → spec/spec_helper.rb → test/test_helper.rb. For non-Rails gems: spec/spec_helper.rb → test/test_helper.rb → test/helper.rb, falling back to the gem entry lib/<gem>.rb (with a warning naming the searched paths). Pass --no-preload to opt out.
--no-preload Boolean (enabled) Disable parent-process preload.
--warmup NAME String none After preload, warm lazily-initialised framework state once in the parent: none or rails. See Warming up Rails before forking.
--[no-]copy-worker-databases Boolean (enabled) With --jobs above 1 and a preloaded Rails app, give each worker its own copy of a SQLite test database the workers would otherwise share. See Per-worker database copies.
--skip-heredoc-literals Boolean false Skip all string literal mutations inside heredocs.
--skip-sites PRESETS List (none) Leave low-value code unmutated, comma-separated: logging, debug_output, error_messages, requires. --no-skip-sites overrides skip_sites from the config file. See Skipping Low-Value Code.
--show-disabled Boolean false Report mutations skipped by # evilution:disable comments.
--fallback-full-suite Boolean false When no matching spec/test resolves for a mutation, run the whole test suite instead of marking it :unresolved and skipping.
--related-specs-heuristic Boolean false When a mutation removes an includes(...) call, also run matching specs from spec/{requests,integration,features,system} (Rails-style domain match on the source file's basename). Trades extra spec runs for higher kill rate on ORM mutations.
--baseline-session PATH String (none) Saved session file for HTML report comparison.
-e CODE, --eval CODE String (none) Inline Ruby code for util mutation command.
--profile NAME String default Operator profile: default or strict. strict adds aggressive mutators (e.g. replaces x.predicate? with nil, or appends rescue nil to a raising call) intended for pre-merge audits.
--strict Boolean false Shortcut for --profile=strict.

Options (for session subcommands)

Flag Type Default Description
--results-dir DIR String .evilution/results Directory containing session result JSON files. Honored by session list and session gc.
--limit N Integer (none) (session list) Show only the N most recent sessions.
--since DATE String (none) (session list) Show only sessions created on or after DATE (YYYY-MM-DD). For run and subjects, --since takes a git ref instead.
--older-than D String (required) (session gc) Delete sessions older than D (e.g. 30d, 24h, 1w).

Options (for compare command)

Flag Type Default Description
--against PATH String (none) Prior (older) session JSON to diff against. Positional first argument is accepted as a fallback.
--current PATH String (none) Current (newer) session JSON. Positional second argument is accepted as a fallback.

Operator Profiles

Two profiles ship out of the box:

  • default — the 155 stable operators registered in Mutator::Registry.default. Suitable for everyday CI runs; balances coverage signal against survivor noise.

  • strict — adds extra aggressive mutators on top of default:

    • PredicateToNil replaces every x.predicate? call with nil to surface tests that only assert truthiness rather than exact return values.

    • ExceptionSwallow appends rescue nil to a statement that raises by convention — a bang method, fetch without a default, Integer / Float / Rational — to surface tests that never make it fail and check the error comes out (record.save! -> record.save! rescue nil). It skips Ruby core in-place bangs (uniq!, sort_by!, …), exit!, statements already under a rescue, and raise; project bangs that mutate rather than raise will still show up as survivors.

    • StatementReorder swaps two adjacent commands (charge(card); send_receipt(user) -> send_receipt(user); charge(card)) to surface side effects whose order is never asserted. It only swaps statements that both act — call a method, append, write an index, yield, call super — and leaves alone pairs that share a variable, contain control flow or a heredoc, involve a local or instance variable assignment, write different literal keys of the same receiver, or move the last statement of a body. Expect survivors where the order only shows up in help text or in the order a list is built.

    • SafeNavigationInsertion replaces a plain call with safe navigation (user.name -> user&.name) to surface calls no test reaches with a nil receiver. With a nil receiver the mutant skips the call and gives nil, where the plain call raises NoMethodError or runs a method nil has (nil.to_a). It touches nearly every call with a receiver and survives wherever that receiver is never nil. It skips self, constants and literal receivers other than nil, and :: calls. The opposite direction, safe_navigation_removal, is in the default profile.

    Use for pre-merge audits where you want maximum sensitivity at the cost of more survivors.

Set via --profile=strict, the --strict shortcut, or profile: strict in .evilution.yml.

Exit Codes

Code Meaning Agent action
0 Mutation score meets or exceeds --min-score, or a --since run found nothing to mutate Success. No action needed.
1 Mutation score below --min-score, or a target file resolved to no spec Parse output, fix surviving mutants.
2 Tool error (bad config, parse failure, a run under --isolation in_process that a mutation took down, etc.) Check stderr, fix invocation.

min_score defaults to 0.0, so no score gate is armed unless you set one. The Result: line says so rather than implying a threshold nobody configured:

$ evilution run lib/half_tested.rb              # no gate
Result: 66.67% (no minimum score set)           # exit 0

$ evilution run lib/half_tested.rb --min-score 0.8
Result: FAIL (score 66.67% < 80.00%)            # exit 1

A run under --isolation in_process can end without a report: a mutation that takes the Ruby process down takes the run with it. It ends with 2, and a line on stderr names the mutation and points at --isolation fork. See What in_process cannot contain.

Evilution owns the exit status: --preload loads the project's own spec helper into the parent process, and an at-exit hook it installs (SimpleCov calls exit with its own status when coverage is below the minimum) would otherwise replace the status evilution computed (GH #1608).

The printed verdict and the exit code always use the same threshold. Previously the line was printed against a hard-coded 80% that the exit code did not share, so a failing-looking run still exited 0 (GH #1604).

Mutating only what changed (--since)

--since REF limits a run to the lines changed since a git ref, which is what a pull request needs checked:

bundle exec evilution run --since origin/main                 # every changed .rb file under lib/ and app/
bundle exec evilution run lib/billing.rb --since origin/main  # only this file, if it changed
bundle exec evilution subjects --since origin/main            # preview the subjects and mutation counts

What counts as changed:

  • The branch's own changes. The diff runs from the merge base of REF and HEAD (what git diff REF...HEAD compares), so commits REF gained after the branch left it do not count, to the working tree, so uncommitted changes, staged or not, do.
  • Added and modified lines. A mutation is run when any line it changes is one of them; a subject (method, constant) is mutated when any of its lines is. Lines that were only deleted leave no line behind: the lines either side of the gap count instead, so a method whose only change is a removed line is still mutated there.
  • New files count whole, untracked ones included (those .gitignore leaves out do not). Renamed files count only their edited lines, under the new name. Deleted files, binary files and files outside the current directory do not count.
  • Which files. Without file arguments, the changed .rb files under lib/ and app/, as a run without arguments takes. File arguments narrow that: a file that did not change drops out, and lib/foo.rb:15-30 keeps only the changed lines from 15 to 30. A file argument must be an existing file: a missing path or a directory is an error (exit 2), as without --since. --target source:GLOB cannot be combined with --since (pass the files instead).
  • A method --target is looked for in the changed files only. When its file changed but the method did not, there is nothing to mutate and the run passes. When its file did not change, the target matches nothing and the run fails with exit 2 (no subject matched '...' in the files changed since REF), as a misspelled target does: evilution cannot tell the two apart without reading the unchanged files. A CI job that pins a --target should run only when the target's file changed.
  • Nothing to mutate passes. A run whose changed lines hold nothing to mutate (a docs-only pull request) reports zero mutations and exits 0, whatever --min-score says.

REF must be known to the clone, and the history back to the merge base must be there: a shallow CI checkout fails with a message saying so. In GitHub Actions:

- uses: actions/checkout@v4
  with:
    fetch-depth: 0          # the merge base with the target branch must be fetched
- run: bundle exec evilution run --since origin/${{ github.base_ref }} --format json --min-score 0.8 --output evilution.json

--since is not a config file key: a ref in .evilution.yml would scope every run, explicit file lists included. The MCP evilution-mutate tool takes the same as its since parameter.

Configuration

Generate default config: bundle exec evilution init

Creates .evilution.yml:

schema_version: 1            # opts into strict validation (rejects unknown keys, refuses future versions)
# timeout: 30              # seconds per mutation; unset = adaptive (see Per-mutation timeout)
# timeout_factor: 10       # adaptive limit = factor x baseline time of the mutation's spec files...
# timeout_floor: 10        # ...but at least this many seconds...
# timeout_ceiling: 60      # ...and at most this many
# format: text             # text | json | html
# min_score: 0.0           # 0.0–1.0
# integration: rspec       # test framework: rspec, minitest, test_unit
# suggest_tests: false     # concrete test code in suggestions (matches integration)
# save_session: false      # persist results under .evilution/results/
# isolation: auto          # auto | fork | in_process (auto selects fork for Rails + gems, in_process otherwise; in_process cannot contain a mutation that takes the Ruby process down, see docs/isolation.md)
# canary: true             # proof-of-life synthetic mutation at session start (false to skip)
# stop_on_kill: true       # stop each mutation's test run at the first deciding failure (false runs every selected example)
# killer_first: false      # RSpec + stop_on_kill: run likely killers and cheap examples first (only for order-independent suites)
# skip_uncovered: true     # with coverage targeting, report mutations no example executes as uncovered without running them
# probe_first: false       # experimental: when a method's body-to-raise mutant survives, report its other mutants as implied survivors without running them
# preload: null            # path to preload before forking; false to disable; auto-detects for Rails + gems
# warmup: none            # rails: warm I18n, routes, assets and templates once after preload (fork isolation)
# copy_worker_databases: true  # with jobs > 1 and a preloaded Rails app, give each worker its own copy of a shared SQLite test database
# skip_heredoc_literals: false  # skip string literal mutations inside heredocs (recommended for Rails: heredoc SQL/templates rarely have test coverage)
# show_disabled: false     # report mutations skipped by disable comments
# baseline_session: null   # path to session file for HTML comparison
# ignore_patterns: []      # AST patterns to exclude (see docs/ast_pattern_syntax.md)
# skip_sites: []           # presets of low-value code to leave unmutated: logging, debug_output, error_messages, requires
# progress: true           # TTY progress bar
# example_targeting: true  # per-mutation example targeting via body-token scan
# example_targeting_fallback: full_file  # full_file | unresolved
# spec_pattern: null       # restrict resolved spec candidates to files matching GLOB

Precedence: CLI flags override .evilution.yml values.

Schema versioning

.evilution.yml may declare an integer schema_version (currently 1). Behavior:

  • When declared — strict mode. Unknown top-level keys raise Evilution::ConfigError. A schema_version greater than the installed gem supports is rejected so an old gem cannot silently misread a newer config.
  • When omitted — legacy lenient mode. Unknown keys are ignored.

Compatibility policy for the 1.x gem line:

  • New configuration keys are added in MINOR releases (additive only). Each new key takes a default that preserves prior behavior.
  • Existing keys are not removed, renamed, or have their semantics changed in any 1.x release. Deprecated keys keep working through the entire 1.x line; see docs/versioning.md.
  • schema_version is bumped only on incompatible changes — i.e. only at the next MAJOR release. schema_version: 2 will ship with evilution 2.0.

A JSON Schema covering every supported key lives at schema/evilution.config.schema.json. Point editor / IDE YAML extensions at it for autocomplete and inline validation (e.g. VS Code yaml.schemas, JetBrains "Custom JSON Schema").

Configuration reference

All keys recognised under schema_version: 1:

Key Type Default Description
schema_version Integer 1 Config schema version. Declaring it enables strict validation; omit for lenient mode.
timeout Integer (adaptive) Per-mutation timeout in seconds, the same for every mutation. Unset, the limit adapts to the baseline (see Per-mutation timeout).
timeout_factor Number 10 Adaptive limit: this many times the baseline time of the mutation's spec files. Ignored when timeout is set.
timeout_floor Number 10 Shortest adaptive limit, in seconds. Must not exceed timeout_ceiling.
timeout_ceiling Number 60 Longest adaptive limit, in seconds.
format String text Output format: text, json, html.
target String / null null Filter expression: method (Foo#bar), class (Foo), namespace (Foo*), descendants (descendants:Foo), source glob (source:**/*.rb).
output String (stdout) Write the report to this file instead of stdout.
min_score Float 0.0 Minimum mutation score (0.0–1.0) for exit code 0.
integration String rspec Test framework: rspec, minitest, or test_unit.
verbose Boolean false Verbose output (RSS/GC stats per phase, error details for errored mutations).
quiet Boolean false Suppress output.
jobs Integer 1 Number of parallel workers.
fail_fast Integer / null null Stop after N surviving mutants. null = disabled.
baseline Boolean true Run baseline test suite to detect pre-existing failures (a mutation failing only on those is :neutral).
stop_on_kill Boolean true Stop each mutation's test run at the first failure that decides the verdict. Set false to run every selected example.
killer_first Boolean false With RSpec and stop_on_kill, run first the examples that killed nearby mutations, then the cheapest. Only for suites whose examples pass in any order; see Killer-first order.
isolation String auto Isolation strategy: auto, fork, in_process. auto selects fork for Rails projects and packaged gems (*.gemspec), in_process otherwise. Under in_process a mutation that takes the Ruby process down ends the run: see What in_process cannot contain.
incremental Boolean false Cache verdicts across runs; see Incremental cache.
suggest_tests Boolean false Generate concrete test code in survivor suggestions (matches integration).
progress Boolean true TTY progress bar.
save_session Boolean false Save session JSON under .evilution/results/.
line_ranges Hash {} Per-file line-range constraints. Typically set via CLI; rare in YAML.
spec_files Array<String> [] Explicit spec files to run. Bypasses auto-detection when non-empty.
ignore_patterns Array<String> [] AST patterns to skip during mutation generation. See docs/ast_pattern_syntax.md.
skip_sites Array<String> [] Presets of low-value code to leave unmutated: logging, debug_output, error_messages, requires. See Skipping Low-Value Code.
show_disabled Boolean false Report mutations skipped by # evilution:disable comments.
baseline_session String / null null Saved session file path for HTML report comparison.
skip_heredoc_literals Boolean false Skip string literal mutations inside heredocs.
related_specs_heuristic Boolean false Append related request/integration/feature/system specs for includes(...) mutations.
fallback_to_full_suite Boolean false When no matching spec resolves, run the entire suite instead of marking the mutation :unresolved.
preload String / Boolean / null null File to preload in parent before forking. false to disable. null to auto-detect for Rails projects and packaged gems.
warmup String none rails warms lazily-initialised Rails state once after preload so forks do not each pay it. none / false to skip.
copy_worker_databases Boolean true With jobs above 1 and a preloaded Rails app, give each worker its own copy of a SQLite test database the workers would otherwise share. See Per-worker database copies.
spec_mappings Hash<String, String/Array> {} Custom mapping from source path to spec path(s).
spec_pattern String / null null Glob restricting resolved spec candidates.
example_targeting Boolean true Per-mutation example-level targeting.
example_targeting_strategy String lexical How targeting picks examples: lexical (name-grep) or coverage (examples that execute the mutated line, from a cached coverage map).
example_targeting_fallback String full_file When targeting finds no example: full_file or unresolved.
example_targeting_cache Hash { max_files: 50, max_blocks: 10000 } LRU cache bounds for the example-targeting AST parser.
skip_uncovered Boolean true With example_targeting_strategy: coverage, report a mutation no example executes as uncovered without running it. No effect otherwise.
probe_first Boolean false Experimental. Run each method's method_body_to_raise mutant first; when it survives, report the method's other mutants as survivors implied by it, without running them. See Probe-first.
quiet_children Boolean false Redirect each worker's stdout/stderr to per-pid files under quiet_children_dir.
quiet_children_dir String tmp/evilution_children Directory for --quiet-children per-pid log files.
profile String default Operator profile: default or strict.
hooks Hash<String, String> {} Lifecycle hooks: event name → path to a Ruby file returning a Proc.

Stopping at the first kill (stop_on_kill)

By default each mutation's test run stops at the first failing example that decides the verdict, so the remaining selected examples are not run. Failures that were already failing in the baseline run do not stop it (the mutation is still recorded neutral if only those fail), and neither do infrastructure crashes (the mutation is still recorded neutral if only those occur). A mutant whose first failure is followed by a hang or a dead test process is now reported killed; a --no-stop-on-kill run may report the same mutant as timeout or error, so its score can be slightly lower. Pass --no-stop-on-kill (or stop_on_kill: false) to run every selected example. Minitest classes that call parallelize_me! are not stopped early: their tests all run, as they would with --no-stop-on-kill.

Killer-first order (killer_first)

With RSpec (3.13 or later), --killer-first (or killer_first: true) lets a run that stops at its first kill choose the order of its examples, since everything before the kill is its cost: first the examples that killed other mutations on the same line, then elsewhere in the same subject, earlier in this run (the latest killer first); then the rest, cheapest first by the time each took in the baseline; examples with no baseline time last. Examples stay inside their example group, and a group's before(:context) hooks still run once around its examples; a group goes first when it holds a killer, otherwise by the time its timed examples take in all (a group with no timed example goes last). Only the order changes, never which examples run. A survivor runs every selected example either way, and its confirmation run against the whole file keeps the defined order, as does the command reported in test_command. It has no effect with --no-stop-on-kill. In a parallel run, the mutations already handed to a worker when a kill comes back go without that killer. Minitest and test-unit runs keep their own order.

It is off by default because it is safe only for a suite whose examples pass in any order. An example that relies on one defined before it (a global, a class variable or a record that an earlier example sets up) fails when it runs first, whatever the mutant, so the mutation is reported killed even though a run in the defined order, including the reported test_command, would let it survive. Nothing in the report shows this, and it only raises the score. If your suite runs in random order (config.order = :random) and passes, it is a fair candidate. A run that times out in the killer-first order is run again in the defined order, which decides the verdict: a mutant that hangs the cheapest example would otherwise be reported timeout where the defined order reports killed. Such a mutation can take up to twice its time limit, and a parallel run gives each mutation twice the usual time before it gives a worker up. Under an adaptive limit this happens under each limit: a reordered run that uses up the adaptive limit runs again in the defined order under it, and only when that times out too does the mutation run again under the fixed limit, reordered and then, if it times out once more, in the defined order, which decides.

The JSON report lists, per mutation, the example that killed it (killing_test) and how many examples ran (examples_run), whatever the order.

Uncovered mutations

With coverage targeting (--example-targeting coverage), evilution records which lines each example of the resolved spec files executes, and which executable lines nothing executed at all. A mutation whose changed lines nothing executes could only survive: no test reaches the code. Running it would cost the most of any verdict — the whole resolved spec file, then the survivor confirmation run — and say nothing new. Such a mutation is reported uncovered instead, without a test run.

Score: an uncovered mutation counts as a survivor. It stays in the score's denominator, so the score, --min-score / min_score and the exit status are the same as running it would have given; the report lists it in its own uncovered bucket (text, JSON, HTML, MCP and saved sessions) rather than with the survivors that ran, and --fail-fast counts it as a survivor.

A mutation is reported uncovered only when the coverage map can vouch for it, and runs as before otherwise:

  • Only the lines the mutation changes count. A line that runs only while files load — a class body, a constant, attr_reader, a def line — counts as executed, so a mutation of it runs. A mutation that replaces a method body is reported at the def line but changes only the body, so it is uncovered when no example calls the method.
  • At least one changed line is executable and never ran; none of them ran.
  • The map was built from every spec file the mutation would run. A mutation that runs a spec file the build did not (the related-specs heuristic, the full-suite fallback) runs.
  • Every example of those spec files ran in the build. A spec file that failed to load, or a build cut short (fail_fast in .rspec, an interrupt), makes the map vouch for no line.
  • The file was loaded while lines were counted. When coverage targeting is on and the tests run in forked processes (isolation: fork, the default for Rails apps and gems), evilution counts from before the preload runs, so a preloaded file is covered too; a file loaded while counting was off is never vouched for. Each forked test process stops that counting before its tests run, so a spec helper that starts line coverage itself (SimpleCov) works as it does without coverage targeting. Under in_process isolation the preload is not counted.
  • No example forked a process. A forked process counts its lines apart, where the build cannot see them, while a mutation applied before the fork reaches it; when any example forks, the map vouches for no line.
  • The mutation neither adds nor removes a local variable assignment. Ruby settles while parsing whether a name is a local, so such a change reaches lines that do run (a later read of the name becomes a method call), even when its own line never runs.

The coverage map is cached under .evilution/coverage/ and rebuilt when any file it depends on changes: a target file, a resolved spec file, one of the helpers every spec file loads (.rspec, spec/spec_helper.rb, spec/support/**/*.rb, ... — the same list as the incremental cache), any file under the directory the spec files sit in (spec/, test/: shared examples, factories, fixture data), and any file of the project the build loaded (another file under lib/ that calls the code, for instance). It is RSpec only, like coverage targeting. Pass --no-skip-uncovered (or skip_uncovered: false) to run these mutations anyway.

summary.uncovered counts these mutations; it is unrelated to summary.uncovered_code, which lists targeted lines that hold no subject, so no mutation was generated for them.

Probe-first (experimental)

--probe-first (or probe_first: true) runs one sentinel mutant per method first: the method's whole body replaced by a bare raise (method_body_to_raise). When even that survives, no selected test calls the method on a path it checks, and the method's other mutants are reported survived without running them; their JSON entries carry implied_by: "method_body_to_raise". A sentinel reported uncovered counts the same. A method whose sentinel was killed, timed out or errored, and a subject that has no sentinel (an empty body, a body that already only raises, a constant or a declaration), runs all its mutants as before. So does a method with a rescue or ensure clause of its own (def safe_div(a, b); a / b; rescue StandardError; -1; end): its body-to-raise mutant keeps the clause, which can catch the raise or return over it, so that mutant surviving says nothing about the tests and is not used as a sentinel.

The run goes in two batches: all sentinels, then the rest, so under --jobs the second batch waits for the slowest sentinel, and each batch has its own worker pool: the run's timings.workers lists the workers of both (up to twice --jobs), the first pool's with a few sentinels each. An implied survivor is scored like any survivor, gets no survivor confirmation run and is not cached; the text report marks it (implied by method_body_to_raise, not run) and counts the implied survivors next to survived. It can be wrong: a test that swallows the sentinel's exception (a rescue around the call, a matcher that expects some error) lets the sentinel survive while another mutant of the method is killed. It is off by default; see docs/probe_first_spike.md for what it saves and how often it is wrong on real projects.

Disable Comments

Suppress mutations on specific code with inline comments:

# Disable a single line
log(message) # evilution:disable

# Disable an entire method (place comment immediately before def)
# evilution:disable
def infrastructure_method
  # no mutations generated for this method body
end

# Disable a region
# evilution:disable
setup_logging
configure_metrics
# evilution:enable

Use --show-disabled to see which mutations were skipped.

Skipping Low-Value Code

Some mutations cost a full test run and rarely say anything: a changed log message, a reworded error message, a dropped require that the preloaded process has already run. skip_sites (or --skip-sites) names presets of such code to leave unmutated:

skip_sites: [logging, error_messages]

Each preset is a structural rule over the parsed source, like ignore_patterns, but a preset only skips what changes what the code reports. The raise, the class raised, the condition guarding a raise or a log line, code that may change state, a call whose value is used, and any mutation that puts a raise in are always mutated.

Preset Sites Skipped inside a site
logging A log level (debug info warn error fatal unknown add log) sent to a receiver named logger or LOGGER (logger, @logger, Rails.logger, config.logger, App::LOGGER); warn / deprecation_warning sent to a deprecator (ActiveSupport::Deprecation, ActiveSupport.deprecator, Rails.deprecator, deprecator); warn(..., category: :deprecated). Not logger.debug? or logger.level = .... Literal text of the message; anything inside an argument or block that only reads. The call itself (deleting the line) and its receiver, when every argument only reads and the call's value goes unused: a statement followed by another one, not the last statement of a method or block (whose value it returns), nor an operand or an assigned value (logger.info(...) && save).
debug_output puts, print, printf, warn sent to no receiver or to Kernel; puts / print / printf / write sent to $stdout, $stderr, STDOUT, STDERR; p / pp and a stream's write only where their value goes unused, since they return what they print or the bytes written (p x as the last statement returns x). As for logging. Kept apart from logging because console output is a CLI's actual job.
error_messages raise / fail sent to no receiver or to Kernel. The message is the second positional argument (raise Error, "msg"), the only one when it is a string (raise "msg"), or the string arguments of an error built in place on a constant (raise Error.new("msg"), .exception). Keywords (cause:), the backtrace argument and any argument after a splat are not messages. Literal text of the message; anything inside a message that only reads. Never the raise, its class, or a non-string argument of Error.new(...), which the error may keep as data.
requires require / require_relative sent to no receiver or to Kernel that run while the file loads: at the top of the file or in a class or module body. Not inside a method, block or lambda (lazy loading), nor a require whose value is used (HAS_OJ = require "oj"). Everything, the call included. Evilution mutates load-time code only in constant assignments and class-body declarations (include, private, alias), none of which is a plain require line, so today this preset skips nothing; it guards such lines should a subject ever reach them.

"Only reads" means: literals, variable and constant reads, interpolation, && / || / ternaries, [], operators, format, sprintf, join, to_s, and, without arguments or a block, predicates (empty?) and a fixed list of common readers (class, name, message, inspect, id, size, length, first, last, to_s, to_i, to_sym, path, ...). Any other call, even one without arguments, may change state (queue.pop, io.read, enum.next), so a message such as "left: #{queue.pop}" keeps the call mutated (only its literal text is skipped), and a log line that calls one is never deleted. Mutations whose node is the whole raise call (removing or nil-ing its message argument) are not skipped, nor is replacing the message of raise "msg" as a whole: raise nil raises TypeError instead of RuntimeError.

A skipped mutation is not run and not scored. It is counted in summary.skipped, and per preset in summary.skipped_sites (JSON) and on a Skipped sites: line in the text report. A mutation an ignore_patterns pattern also matches is counted under the pattern. skip_sites changes no verdict, so it does not invalidate the incremental cache.

Default: none. Under the 1.x compatibility policy a new key's default preserves prior behaviour, and turning a preset on removes mutants, and with them points from the score, that existing users see today. Measured over the whole lib/ of dry-monads, httparty, grape and i18n and Redmine's app/ and lib/redmine, all four presets together skip 0.3-0.7% of the mutants (in Redmine mostly logging, elsewhere mostly error_messages), and some of the skipped error-message mutants are killed by specs that assert the message, so the speed-up a default would buy is small next to a changed mutant set and score. Turn on the presets that fit your code base.

JSON Output Schema

Use --format json for machine-readable output. The same shape is used for both stdout reports (--format json) and saved session files (--save-session → .evilution/results/*.json); session files add a small set of extra top-level fields described under Session JSON files below.

Schema:

{
  "schema_version": "integer — schema version of this JSON document (current: 1)",
  "version": "string   — gem version",
  "timestamp": "string — ISO 8601 timestamp of the report",
  "summary": {
    "total": "integer    — total mutations generated",
    "killed": "integer   — mutations detected by tests (test failed = good)",
    "survived": "integer — mutations NOT detected (test passed = gap in coverage)",
    "timed_out": "integer — mutations that exceeded timeout",
    "errors": "integer   — mutations that caused unexpected errors",
    "neutral": "integer  — mutations with no verdict: their tests failed only on examples that were already failing in the baseline, or the test process crashed on infrastructure",
    "equivalent": "integer — mutations proven to have identical behavior to the original",
    "unresolved": "integer — mutations where no spec file resolved (coverage gap, not a failure)",
    "unresolved_target_files": "array of strings (optional) — target files that resolved to no spec at all; present only when non-empty, and the run fails when it is",
    "infra_retried": "integer (optional) — mutations a parallel pass could not judge because the test process crashed on infrastructure, re-run serially afterwards; present only when non-zero",
    "uncovered_code": "array (optional) — targeted lines that hold code outside every subject, so no mutation was generated for them: [{ file, lines: ['2-4', '9'] }]; checked for a line range as given, and for a whole file only when it has no subject at all; present only when non-empty",
    "baseline_neutralized": "integer (optional) — kills not counted because only examples already failing in the baseline failed; those mutations are recorded neutral; present only when non-zero",
    "baseline_failures": "array (optional) — one entry per spec file that was red in the baseline: { spec_file, error: string|null — what stopped the file outside any example (load error, runner exception, timeout, a baseline process that died), failing_examples: integer — every failing example, examples: [{ id, description, message }] — the first few of them }; present only when non-empty",
    "unparseable": "integer — mutations whose mutated source did not parse (short-circuited, never executed)",
    "uncovered": "integer  — mutations whose changed lines no example executes, reported without a test run (coverage targeting only); scored as survivors",
    "score": "float      — killed / (total - errors - neutral - equivalent - unresolved - unparseable), range 0.0-1.0, rounded to 4 decimals; uncovered mutations stay in the denominator, as survivors",
    "duration": "float   — total wall-clock seconds of the run, from subject discovery through the audit of target files for unresolved specs (the phases in timings.phases add up to it), rounded to 4 decimals",
    "peak_memory_mb": "float (optional) — peak RSS across all mutation child processes, in MB",
    "skipped": "integer (optional) — mutations left out during generation (ignore_patterns, skip_sites, # evilution:disable, Sorbet sigs), never run or scored; present only when non-zero",
    "skipped_sites": "object (optional) — the share of each skip_sites preset in skipped, e.g. { logging: 3, error_messages: 12 }; present only when a preset skipped something"
  },
  "survived": [
    {
      "operator": "string — mutation operator name (see Operators table)",
      "file": "string    — relative path to mutated file",
      "line": "integer   — line number of the mutation",
      "status": "string  — result status: 'survived', 'killed', 'timeout', 'error', 'neutral', 'equivalent', 'unresolved', 'unparseable', or 'uncovered'",
      "duration": "float — seconds this mutation took, rounded to 4 decimals",
      "diff": "string    — legacy +/- diff snippet",
      "unified_diff": "string (optional, survived and uncovered only) — git-style unified diff with `--- a/file`, `+++ b/file`, `@@` hunk header and sdiff body; omitted when source slices are unavailable",
      "suggestion": "string — actionable hint for surviving mutants (survived and uncovered only)",
      "test_command": "string (optional) — the command that reproduces the run behind this verdict, shell-escaped so it can be pasted as is (a targeted RSpec run names example ids such as `spec/x_spec.rb[1:2]`); for a survivor confirmed against its whole spec files, that confirming run",
      "killing_test": "string (optional, RSpec only) — the example whose failure killed the mutation, as `/absolute/path/spec/file_spec.rb[1:2]` (spec file and RSpec scoped id); absent when no example decided the kill",
      "examples_run": "integer (optional, RSpec only) — how many examples ran against the mutation, the survivor confirmation run included; absent for results that did not run (cached, uncovered, unparseable)",
      "implied_by": "string (optional, survived only, --probe-first) — the sentinel operator ('method_body_to_raise') whose survival implied this verdict; the mutation itself did not run",
      "time_limit": "float (optional, timeout only) — the seconds this mutation's test run was allowed, rounded to 4 decimals; for a timeout confirmed by a second run (see Per-mutation timeout) the limit of that second run; for a timeout read from the --incremental cache, the limit it was recorded under",
      "neutral_reason": "object (optional, neutral only) — { kind: 'baseline_failure' | 'infra_error', detail: string|null — the failing spec file or the crash class; null when the run was given explicit --spec files and no single spec can be named }",
      "timings": "object (optional) — where this mutation's duration went, in seconds rounded to 4 decimals; only the phases that happened are listed: { apply — parsing and evaluating the mutated source, test_run — the first test run, confirmation — the re-run of a survivor against its whole spec files (RSpec, targeted runs only), queue_wait — parallel runs only: from the dispatcher sending it to a worker to that worker starting it; a worker is sent its next mutation as soon as it finishes one, so this is the cost of the hand-over, and for a worker's first mutation also of the worker starting up, adaptive_timeout — a first run that used up an adaptive limit shorter than the fixed one, after which the mutation ran again under the fixed limit and the other phases describe that second run (see Per-mutation timeout) }; what duration holds beyond these is the isolation around the run (fork, sandbox, reap); a timed-out mutation reports nothing itself, so its whole duration is counted as test_run; absent for results read from the --incremental cache"
    }
  ],
  "timings": {
    "phases": "object — wall-clock seconds per phase of the run, in the order they ran: subject_discovery, preload, canary, baseline, planning (mutation generation and equivalence detection), execution (every mutation, including the serial re-run of infrastructure crashes and building the coverage map when coverage targeting is on), finalize; they add up to about summary.duration",
    "baseline": "array — one entry per spec file the baseline ran, in order: { spec_file, duration }; empty when the baseline did not run",
    "mutations": "object — each per-mutation timing phase summed over all mutations: { apply, test_run, confirmation, queue_wait, adaptive_timeout }; process seconds, so in a parallel run they add up across workers and can exceed the execution phase",
    "workers": "array — one entry per worker process of a parallel run: { pid, items, busy, wall, utilization — busy / wall }; workers live for the whole run, so there are --jobs entries (fewer when there are fewer mutations to run; with --probe-first the sentinels and the other mutations run in two pools one after the other, so up to twice as many) plus one for each worker that replaced one killed after a timeout, a crash or an unreadable reply; under --isolation in_process each mutation gets a fresh worker, so there is one entry per mutation run; busy and wall are 0.0 for a worker that died before reporting them; empty for --jobs 1"
  },
  "subjects": [
    {
      "name": "string      — subject name (e.g. 'Foo#bar')",
      "file": "string      — relative path to source file",
      "total": "integer    — mutations generated for this subject",
      "killed": "integer   — mutations detected",
      "verified": "integer — mutations that got a verdict (killed + survived + timed out + uncovered)",
      "survived": "integer — mutations that went undetected",
      "uncovered": "integer — mutations no example executes, reported without a test run; counted in verified",
      "score": "float      — killed / verified, 0.0 when nothing was verified, rounded to 4 decimals",
      "reached": "boolean  — false when no mutation of this subject got a verdict at all"
    }
  ],
  "coverage_gaps": [
    {
      "file": "string       — relative path to source file",
      "subject": "string    — method name (e.g. 'Foo#bar')",
      "line": "integer      — line number",
      "operators": ["string — operator names involved"],
      "count": "integer     — number of survived and uncovered mutations in this gap",
      "mutations": ["... same shape as survived entries; `status` tells a survivor from an uncovered mutation ..."]
    }
  ],
  "killed": ["... same shape as survived entries ..."],
  "neutral": ["... same shape as survived entries ..."],
  "equivalent": ["... same shape as survived entries ..."],
  "unresolved": ["... same shape as survived entries — coverage gap: no spec file resolved for these mutations"],
  "unparseable": ["... same shape as survived entries — mutated source failed to parse and was never executed"],
  "uncovered": ["... same shape as survived entries — no example executes the lines the mutation changes; never executed, scored as survived"],
  "timed_out": ["... same shape as survived entries ..."],
  "errors": [
    {
      "... same shape as survived entries, plus: ...": "",
      "error_message": "string (optional) — error message from the failing mutation",
      "error_class":   "string (optional) — exception class name (e.g. 'SyntaxError', 'NoMethodError')",
      "error_backtrace": ["string (optional) — first 5 backtrace lines from the exception"]
    }
  ]
}

Key metric: summary.score — the mutation score. Higher is better. 1.0 means all mutations were caught.

Session JSON files

Sessions saved by --save-session (under .evilution/results/*.json) and consumed by evilution session show, evilution session diff, evilution compare, and the HTML reporter share the schema above with these additions:

Field Type Description
git Object { "sha": "<full SHA or null>", "branch": "<branch name or null>" } captured at run time.
killed_count Integer Top-level convenience counter; mirrors summary.killed.
timed_out_count Integer Mirrors summary.timed_out.
error_count Integer Mirrors summary.errors.
neutral_count Integer Mirrors summary.neutral.
equivalent_count Integer Mirrors summary.equivalent.
uncovered_count Integer Mirrors summary.uncovered.
skipped_count Integer Mutations left out during generation: ignore_patterns, skip_sites, # evilution:disable, Sorbet sigs (mirrors summary.skipped, omitted from summary unless positive; summary.skipped_sites carries the per-preset share).

Saved sessions also omit the per-status arrays (killed, neutral, equivalent, unresolved, unparseable, timed_out, errors) — only survived, uncovered and coverage_gaps are persisted. evilution session diff treats an uncovered mutation as a survivor, so one that survived before is not reported fixed. The score, totals, and timestamps are stable for diff/compare consumers.

stdout in JSON mode

With --format json, stdout carries the JSON document and nothing else. Each mutation's test run writes to buffers evilution owns, whatever the isolation mode: under in_process the framework's configuration can outlive a single run — --preload builds it before isolation swaps $stdout — so the run claims RSpec's output and error streams outright rather than relying on RSpec to redirect them (GH #1627). Once the document is written, stdout is pointed at stderr, so anything a preloaded spec helper prints on the way out — SimpleCov's coverage report, for example — lands on stderr instead of after the document where it would leave JSON.parse with nothing to work with. --output FILE writes the document to a file and leaves stdout alone entirely.

Schema versioning

Every session and stdout JSON document carries a top-level schema_version integer (currently 1). On read:

  • schema_version matches what this gem supports — proceed normally.
  • schema_version is omitted — treated as version 1 (the JSON shape that defined version 1). Sessions written before this field existed continue to load.
  • schema_version is greater than what this gem supports — Evilution::Session::Store#load, evilution compare, evilution session show, evilution session diff, and the HTML reporter raise Evilution::Error with the offending file path and a "Upgrade the evilution gem" message. We refuse to silently misread a newer document.

Compatibility policy for the 1.x gem line:

  • New top-level fields are added in MINOR releases (additive only). Consumers that ignore unknown fields keep working without changes.
  • Existing fields are not removed, renamed, or have their semantics changed in any 1.x release.
  • schema_version is bumped only on incompatible changes — i.e. only at the next MAJOR release. schema_version: 2 will ship with evilution 2.0. See docs/versioning.md for the umbrella SemVer policy.

Mutation Statuses

Status Meaning Counted in score?
killed A test failed when the mutation was applied — test suite caught it. Also a mutation that made its own file raise while loading, before any test ran, when the original file still loads (error_message starts with load failed:). Also a mutation whose test process died on an exception no handler saw, after the mutated source loaded and while the tests ran, as a crashing test would: under fork isolation error_message starts with child died and names the exception, under in-process isolation it starts with test run ended by numerator + denominator
survived No test failed — gap in coverage denominator only
timeout Test run exceeded its time limit (--timeout, or the adaptive limit) — treated like survived for scoring denominator only
error Mutation caused an unexpected error (syntax error, boot failure, a load failure the original file shares, etc.). Also a test process that died on an unhandled exception before the tests ran: under fork isolation error_message starts with child died, says before the tests ran and error_class is the exception's own class. Also a result that never arrived or could not be read, which is no verdict on the mutation (see Diagnosing errored mutations) excluded from denominator
neutral No verdict: the tests failed only on examples that were already failing in the baseline, or the test process crashed on infrastructure (DB lock, statement timeout) rather than on the mutation. Every neutral records which of the two, and the report groups them by it excluded
equivalent Mutation is provably identical to the original (e.g. no-op replacement) excluded
unresolved No spec file resolved for the mutated source — coverage gap, not a failure. Use --fallback-full-suite to run the full suite instead. excluded
unparseable Mutated source failed to parse (e.g. dangling heredoc opener after method_body_replacement). Short-circuited — never executed. excluded
uncovered No example executes the lines the mutation changes (coverage targeting only, see Uncovered mutations). Never executed — it could only have survived. denominator only, as survived

Unresolved mutations indicate a missing test mapping — the file has no corresponding test file that the resolver could find (for example, an RSpec _spec.rb file or a Minitest _test.rb file, depending on configuration). The resolver searches the lib/-mirrored path, common non-mirrored buckets (spec/unit, spec/lib, test/unit, test/lib), and the flat test_-prefixed Minitest/Test::Unit convention (test/test_connection_pool_timed_stack.rb), so a high unresolved rate usually means a genuinely missing or unconventionally-placed test; a run that leaves many mutations unresolved prints an unresolved-rate warning with a best-guess spec path per source file. They are reported separately so you can act on them (add a test, adjust test naming, pass --spec, or opt in to the full-suite fallback) without inflating the error count.

A target file that resolves to no test at all is a stronger condition than an individual unresolved mutation, and is reported on its own terms: evilution names the file and fails the run, whatever the mutations it did measure scored. Without this, adding one well-tested file to the command dilutes the unresolved rate and the untested file disappears behind a PASS (GH #1603):

$ evilution run app/services/untested.rb app/services/well_tested.rb
Mutations: 98 total, 82 killed, 0 survived, 0 timed out, 2 neutral, 14 unresolved
Score: 100.00% (82/82)
! 1 of 2 target files has no resolvable spec — it was never tested:
    app/services/untested.rb
Result: FAIL (1 target file has no resolvable spec)
$ echo $?
1

The check covers files evilution found something to mutate in; a file it produced no mutations for (a constants-only file, say) is not reported. --fallback-full-suite runs such a file against the whole suite instead, so nothing goes untested and nothing is reported. The file list is also in JSON output under summary.unresolved_target_files.

Survivor Confirmation

Per-mutation targeting runs a subset of a spec file's examples, chosen by matching the enclosing method's name against example bodies. That match keys on identifier text, so it can miss the example that would have caught a mutation — and the mutation is then reported as a survivor the suite actually covers. A survivor nobody can reproduce is worse than a missed kill: it sends the reader to write a test that is already there (GH #1624).

Before any survivor is reported, it is therefore re-run against the whole resolved spec file, and that run is the one reported. Only survivors pay for the extra run, and only where the targeted subset was narrower than the file; a mutation the targeted examples already killed is never re-run.

On evilution's own lib/evilution/reporter/json/subjects.rb this moved the reported score from 74.19% with 8 survivors to 93.55% with 2 — the six that disappeared were killed by an example in the same file all along, and the run now agrees with --no-example-targeting instead of contradicting it.

Per-mutation timeout

A mutation that makes the code loop runs until its time limit, so the limit is most of what such a mutant costs. A fixed limit is either far too long for a spec file that passes in a fraction of a second or too short for a slow one.

So when no timeout is given, each mutation's limit follows the time its own spec files took in the baseline:

limit = clamp(timeout_factor × baseline seconds of the mutation's spec files, timeout_floor, timeout_ceiling)

With the defaults (factor 10, floor 10s, ceiling 60s) a mutant on a spec file that passes in 0.3s is first given 10s instead of 30s, and one whose spec files take 4s gets 40s. The baseline runs each spec file in a forked process of its own, so its time holds the fork and loading the file; the factor leaves room for the re-run of a survivor against its whole spec files (Survivor Confirmation) and for workers competing for the CPU under --jobs, and the floor covers spec files so fast that scheduling noise outweighs them.

A mutant that slows the code down can still fail its test well after its baseline suggests, and a timeout reported there would hide a kill. So a mutation that uses up a limit shorter than the fixed 30s is run again under 30s, and the verdict of that second run is the one reported: a mutation is only reported as a timeout where the fixed limit would have reported one too. The first run is part of what the mutation cost: it is in its duration and in the JSON report's timings as adaptive_timeout. A mutant that loops therefore costs the shorter limit plus 30s; the time saved is on kills and survivors, and on spec files slow enough to get a longer limit. Raise timeout_floor if many mutations are run twice.

A mutation keeps the fixed 30s limit when the baseline cannot vouch for its spec files: no baseline ran (--no-baseline), no spec file resolves for its source (it falls back to the whole suite, or is reported unresolved), one of its spec files failed in the baseline, or related_specs_heuristic adds spec files the baseline never ran. Every mutation keeps it when a worker_process_start, mutation_insert_pre or mutation_insert_post hook is configured: those run inside the time a mutation is allowed, and the baseline runs none of them.

--timeout N (or timeout: in the config file) gives every mutation the same N seconds, as before; pass --timeout 30 for the previous behaviour. The limit a timed-out mutation ran under is in the JSON report as time_limit. Under --jobs the pool gives up on a worker that has not answered within twice the longest time a mutation of the run may take, both of its runs together.

With --incremental, a cached timeout is reused only while the mutation would get no more time than the limit it was recorded under; raising --timeout, timeout_floor or the factor runs it again. Timeouts cached before the limit was recorded are run again once.

Incremental cache

--incremental (or incremental: true) keeps each mutation's verdict — killed, survived or timeout — in tmp/evilution_cache/ and reuses it on the next run instead of running the mutation again. A reused verdict is reported and scored like a fresh one; it carries no timings and no memory figures, since nothing ran. A cached timeout is reused only while the mutation would get no more time than the limit it was recorded under (see Per-mutation timeout).

Verdicts are kept per subject (a method, a constant, a declaration). A cached verdict is reused only while all of these are unchanged:

  • the subject's own source;
  • the rest of its file outside every method definition: class and module lines, includes, attr_*s, other class-body code, every constant of the file, and every class-body declaration (callbacks, validations, scopes, AASM machines), since these decide whether and when the file's methods run;
  • the contents of the spec files that resolve for the file (--spec, spec_mappings, or the resolver), and of the files the related-specs heuristic adds for the mutation when it is on;
  • the conventional helpers every spec file loads: .rspec, spec/spec_helper.rb, spec/rails_helper.rb, spec/support/**/*.rb, test/test_helper.rb, test/support/**/*.rb, the preload file, and Gemfile.lock / gems.locked;
  • the evilution and Ruby versions;
  • the settings that decide how a mutation is run and judged: integration, isolation, profile, timeout and the adaptive timeout's timeout_factor / timeout_floor / timeout_ceiling, baseline, the example-targeting settings, related_specs_heuristic, fallback_to_full_suite, preload, stop_on_kill, killer_first, warmup, whether uncovered mutations are skipped (skip_uncovered with coverage targeting), probe_first and copy_worker_databases.

So editing one method re-runs that method's mutations only, and a method that only moved because a method above it grew or shrank keeps its verdicts. Editing a constant, a callback, validation or scope declaration, any other class-body line (comments between methods included) or a spec file re-runs every subject of the file. A method defined twice in one file (say, in both branches of an if) keeps a verdict per definition.

What the cache does not see: other methods the subject calls (in the same file or another), other source files, and files a spec loads beyond the helpers above (factories, fixtures, shared examples outside spec/support), as well as hooks. A change there keeps the cached verdicts, so a survivor can stay a survivor after a test it depends on indirectly was fixed, or the other way around. Run without --incremental (or delete tmp/evilution_cache/) for a result that owes nothing to earlier runs, e.g. in CI.

Not cached: uncovered mutations, which are worked out again from the coverage map on every run (it has a cache of its own); survivors implied by a --probe-first sentinel; mutations whose source has no spec file that resolves (with or without --fallback-full-suite), errors, neutrals, kills from an infrastructure crash, and kills in which only examples already failing in the baseline failed.

The cache holds one JSON file per source file under tmp/evilution_cache/v2/, read once per run and written once per source file, so its cost grows with the number of mutations, not with their square. A file keeps one version of each subject: writing it drops the verdicts of a subject's earlier source and of subjects the file no longer has. Cache files of the earlier format (directly in tmp/evilution_cache/, keyed by the whole file's content) are not read, and are deleted the first time the new cache is written.

Neutral Mutations

Neutral means the run has no verdict on a mutation. It covers two unrelated situations that want opposite responses, and each neutral records which; the report groups by it, naming the spec or the error class (GH #1606):

  • The tests failed, but only on examples that were already failing before any mutation ran. Such an example fails whatever the mutation is, so its failure is not a kill. Nothing that was passing caught the mutation either, but the red example might have, once fixed.
  • The test process died on infrastructure rather than on the mutation.
Score: 100.00% (10/10 verified of 17 mutations, 7 neutral)
! 7 kills not counted for spec/tally_spec.rb: only examples already failing in the baseline failed, so those mutations have no verdict.
    ./spec/tally_spec.rb[1:3] Tally adds -- NameError: undefined local variable or method 'tally'

Neutral mutations (7, not verified):
  baseline already failing (spec/tally_spec.rb):
    arithmetic_replacement: lib/tally.rb:9
    integer_literal: lib/tally.rb:9

The score line names the remainder whenever the run left mutations out of the denominator, because full marks over a fraction of a run otherwise reads as a verdict on all of it. A clean run still prints the plain Score: 100.00% (17/17). The line under it says how many kills were not counted, for which spec file, and why the baseline was red for it — the failing examples with their first error line, or the error that stopped the file before any example ran. The same detail is printed to stderr once per red spec file as soon as the baseline finishes, and is in JSON output as summary.baseline_neutralized and summary.baseline_failures.

What a red baseline does not do:

  • It does not touch a kill in which something that was passing failed too. That is a kill like any other.
  • It does not touch survivors. A mutation whose tests passed is a survivor whatever the baseline did — including when the baseline's failure did not happen again in the mutation run (a flaky example, a failure of the baseline process itself). Earlier versions recorded every survivor covered by a red spec file as neutral, which hid exactly those gaps.
  • It does not second-guess a baseline failure with no failing example to name — a file that did not load, a timeout, a baseline process that died. A mutation run that hits the same problem reports error or timeout itself.

Recognising an already-failing example needs the test framework to name the examples that failed, the same way in the baseline and in a mutation run. RSpec examples are named by spec file and position, Minitest tests as Class#test, Test::Unit tests as test(Class). Two Minitest test classes without a name that share a test method name cannot be told apart.

JSON output carries neutral_reason as { kind, detail } on neutral entries that have one; detail is null where no single spec can be named (an explicit --spec run), and the field is absent on a result recorded without a reason, which the text report shows as reason not recorded.

Per-Subject Scores

The run's score is computed per file, and test selection resolves per file too, so a well-tested file that gains new untested methods still reports 100% — the number only ever describes what the resolved spec reaches (GH #1605). Every report therefore breaks the run down per subject, and the text report names the subjects the file-level score does not speak for:

Mutations: 33 total, 7 killed, 0 survived, 0 timed out, 26 unresolved
Score: 100.00% (7/7)

Subjects needing attention (2 subjects in 1 file):
  lib/helper.rb
    #summary_for  0.00%  (0/17)  nothing reached this subject
    #total_for  0.00%  (0/9)  nothing reached this subject

A subject is listed when something survived, or when nothing reached it at all — zero verdicts, every mutation unresolved or neutral. Fully-killed subjects are not listed, so the section stays actionable. JSON output carries every subject under subjects, whether or not it needs attention, so a CI step can assert on reached or on a per-subject score.

Mutation Operators (155 total)

Each operator name is stable and appears in JSON output under survived[].operator.

Operator What it does Example
arithmetic_replacement Swap arithmetic operators a + b -> a - b
comparison_replacement Swap comparison operators a >= b -> a > b
boolean_operator_replacement Swap && / || a && b -> a || b
short_circuit_to_eager Evaluate both operands of && / || / and / or (an operand that would regroup is parenthesized; an operand that only jumps, such as return, is skipped) a && b -> a & b, a || b -> a | b
boolean_operand_promotion Drop one side of && / || a && b -> a, b
boolean_expression_to_nil Replace a whole compound boolean with nil a && b -> nil
boolean_literal_replacement Flip boolean literals true -> false
nil_replacement Replace nil with true, false, 0, "" nil -> true
integer_literal Boundary-value integer mutations, plus one sentinel past the next integer width (int8, uint8, int16, uint16, int32, uint32, int64) above the literal's magnitude n -> 0, 1, n+1, n-1; 42 -> 167
float_literal Boundary-value float mutations, plus NaN and both infinities (not inside a pattern, where they do not parse) f -> 0.0, 1.0, Float::NAN, Float::INFINITY, -Float::INFINITY
complex_literal Replace a complex literal with 0i, 1i, its neighbours along the imaginary axis and nil (a rational part such as 3ri gets no neighbours) 5i -> 0i, 1i, 6i, 4i, nil
rational_literal Replace a rational literal with 0r, 1r, its two neighbours and nil (the rational part of a complex literal such as 3ri is left to complex_literal) 1.5r -> 0r, 1r, 2.5r, 0.5r, nil
string_literal Empty the string or replace it with nil; an interpolated string is also replaced as a whole; a backtick or %x command is replaced with nil only, its text never rewritten "str" -> "", nil; "a #{x}" -> "", nil; `ls` -> nil
array_literal Empty the array, replace it with nil, delete each element in turn (two or more elements; a heredoc element is kept), and replace a single-element array with its element (not a splat, bare keywords, a %w / %i word or a heredoc) [a, b] -> [], nil, [b], [a]; [x] -> x
hash_literal Empty the hash, replace it with nil, delete each key/value pair in turn (two or more elements; a **splat and a pair holding a heredoc are kept), and rename each label key in turn (a key written with => is left to symbol_literal / string_literal) {a: 1, b: 2} -> {}, nil, {b: 2}, {a: 1}, {__evilution_mutated__: 1, b: 2}
symbol_literal Replace with sentinel symbol or nil; an interpolated symbol is replaced as a whole with the empty symbol or nil :foo -> :__evilution_mutated__, nil; :"a_#{x}" -> :"", nil
conditional_negation Replace condition with true/false if cond -> if true
conditional_branch Remove if/unless/else branch Deletes branch body
if_branch_swap Replace the if-branch with the else body, drop the else if c; x; else; y; end -> if c; y; end
conditional_flip Flip if to unless and vice versa if cond -> unless cond
statement_deletion Remove statements from method bodies Deletes a statement
method_body_replacement Replace entire method body Method body -> nil, self, super
negation_insertion Negate predicate methods x.empty? -> !x.empty?
return_value_removal Strip return values return x -> return
return_keyword_removal Drop the return keyword and keep its value, so control flow continues past a guard clause or early return (several values become an array; skips returns in tail position, where the value is the result anyway, and bare return) return :neg if x.negative? -> :neg if x.negative?
collection_replacement Swap collection methods map -> each, select <-> reject
collection_return Replace collection return values return [1] -> return []
scalar_return Replace scalar return values return 42 -> return 0
method_call_removal Remove method calls, keep receiver obj.foo(x) -> obj
argument_removal Remove individual arguments foo(a, b) -> foo(b)
argument_nil_substitution Replace arguments with nil foo(a, b) -> foo(nil, b)
call_to_nil Replace a method call with nil (skips void statements, receivers of another call, and attribute or index writes) user.name -> nil
constant_read_to_nil Replace a constant reference with nil, a path as a whole (skips call receivers, void statements and rescue exception classes) MAX -> nil, Config::MAX -> nil
variable_read_to_nil Replace a read of a local, instance, class or global variable, or self, with nil (skips call receivers, void statements, if / unless / while / until conditions, left operands of && / ||, block-pass and double-splat arguments) total = count -> total = nil, build(name: @name) -> build(name: nil)
constant_namespace_strip Strip the namespace off a constant path, at each level of a longer one (skips top-level ::A paths and operator-write targets such as `A::B
constant_write_to_nil Assign nil to a constant instead of its value. A constant assignment outside a method is its own subject, named after the constant, and the literal operators mutate its value too; code that read the constant while the file loaded keeps the old value (skips a constant already nil; Data.define / Struct.new assignments are left to data_struct_member) LIMIT = 10 -> LIMIT = nil
safe_navigation_removal Replace &. with a plain call (skips self and literal receivers) user&.name -> user.name
attribute_write_to_read Replace an attribute or index write with the matching read (skips writes statement_deletion already removes) a.foo = b -> a.foo, a[i] = b -> a[i]
argument_propagation Replace a call with its only positional argument (skips operator methods, attribute writes and void statements) normalize(value) -> value
argument_list_removal Drop a call's whole argument list, keeping any block (skips operator methods and attribute writes) foo(a, b) -> foo, foo(a, &b) -> foo(&b)
symbol_to_proc_replacement Apply the send_mutation and collection_replacement selector tables to a symbol block-pass (skips alias swaps) map(&:to_s) -> map(&:to_i)
dynamic_dispatch_resolution Resolve send / __send__ / public_send with a literal symbol into a direct call, only where visibility checks can differ obj.send(:m, a) -> obj.m(a)
keyword_argument_removal Drop one key: value argument at a call site (keeps **opts; skips a lone keyword) f(a: 1, b: 2) -> f(b: 2)
double_negation_removal Replace a double negation with its operand !!x -> x
regexp_anchor_to_predicate Rewrite a match against an anchored literal regexp as a prefix / suffix predicate (skips match? with \A / \z, an exact equivalent) x =~ /^foo/ -> x.start_with?("foo")
reduce_to_sum Rewrite a + reduction (reduce / inject with :+ or &:+) into sum xs.reduce(:+) -> xs.sum, xs.inject(0, :+) -> xs.sum(0)
array_coercion_to_literal Replace an Array() coercion with an array literal Array(x) -> [x]
to_i_to_integer Replace lenient to_i with strict Integer() (skips numeric literal receivers) x.to_i -> Integer(x), x.to_i(16) -> Integer(x, 16)
coercion_emptying Replace a conversion with an empty value of its type (to_a / to_ary, to_h / to_hash, to_s / to_str) x.to_s -> "", x.to_h -> {}
const_get_to_constant_path Resolve const_get with a literal symbol into a constant path X.const_get(:Y) -> X::Y
proc_to_lambda Turn proc { } / Proc.new { } into lambda { } (arity checks, local return) proc { |x| x } -> lambda { |x| x }
dig_to_fetch_chain Make the first step of a multi-key dig strict x.dig(a, b) -> x.fetch(a).dig(b)
inequality_to_negated_identity Rewrite != with the stricter eql? / equal? (skips nil, boolean and symbol literals) a != b -> !a.eql?(b), !a.equal?(b)
binary_operand_promotion Replace an arithmetic or bitwise expression with one of its operands (skips integer identities and void statements) a + b -> a, b
receiver_constructor_swap Swap Date / DateTime / Time.parse for a stricter sibling constructor on the same receiver Date.parse(s) -> Date.iso8601(s), Date.strptime(s), ...
block_body_to_nil Replace a block body with nil, keeping the iteration (skips loop and endless cycle / cycle(nil), which would hang) xs.each { |x| log(x) } -> xs.each { |x| nil }
block_body_to_raise Replace a block body with raise to prove the block is invoked (skips bodies with a rescue clause, which would swallow it) xs.each { |x| log(x) } -> xs.each { |x| raise }
block_body_promotion Replace a call with its parameter-less block body, run once (skips blocks with parameters or a rescue / ensure clause) Base.transaction { save! } -> save!, 3.times { poll } -> poll
data_struct_member Drop a member, or swap adjacent members, of a Data.define / Struct.new definition (a definition outside any method — Point = Data.define(...), class Coord < Struct.new(...) — is its own subject, named after the constant; skips single-member and splatted lists) Data.define(:a, :b) -> Data.define(:b), Data.define(:b, :a)
pin_operator_removal Drop the pin of a pattern variable so it captures instead of comparing (local variables only; skips pins inside alternative patterns, where a capture is not allowed) in ^expected -> in expected
rightward_assignment Turn a rightward pattern match into a pattern predicate, so a mismatch returns false instead of raising (skips a bare capture, which cannot fail) value => [a, b] -> value in [a, b]
numbered_parameter_swap Swap neighbouring numbered block parameters that the body reads (skips blocks reading a single one; it has nothing to swap with) pairs.map { _1 - _2 } -> pairs.map { _2 - _1 }
forwarded_argument_drop Stop forwarding one part of the arguments a method passes through unnamed; a ... signature is spelled out so a single part can be left out (skips a lone anonymous argument and named splats) g(*, **, &) -> g(**, &), def f(...) = g(...) -> def f(*, **, &) = g(*, **)
pattern_wildcard_widening Widen a pattern so it accepts shapes it rejects: append a rest to an array pattern, wildcard or drop a hash-pattern pair, remove **nil (skips pairs that bind a variable and patterns that already have a rest) in [a, b] -> in [a, b, *], in { age: Integer } -> in { age: _ }
no_matching_pattern_else Add an empty else to a case/in that has none, so an unmatched value yields nil instead of raising (skips a case/in with a bare capture clause, which matches everything) case v; in Integer then 1; end -> case v; in Integer then 1; else; end
argument_order_permutation Swap neighbouring positional arguments of a call, super or yield (keeps keywords, block and splats in place; skips identical neighbours, raise / fail, format / sprintf / printf, index writes, start_with? / end_with? and Set[...]). Calls that accept arguments in any order, such as OptionParser#on, can be silenced with ignore_patterns, e.g. call{name=on, receiver=local_variable_read{name=opts}}; the pattern ignores every mutation inside those calls, their handler blocks included. To keep a handler's logic mutated, write its body on its own lines (do |n| … end) and end the opts.on line with # evilution:disable instead compute(a, b) -> compute(b, a)
keyword_value_swap Swap the values of neighbouring keyword arguments, keeping the keys (keeps positional arguments, **splat and block in place; skips identical values and braced hash arguments; spells out shorthand keys) compute(x: a, y: b) -> compute(x: b, y: a)
comparison_operand_swap Swap the operands of a spaceship comparison, reversing a sort block or a custom ordering (skips identical operands and safe navigation; between? / clamp bounds are swapped by argument_order_permutation) x.age <=> y.age -> y.age <=> x.age
integer_division_to_fdiv Turn a division into fdiv, so integer division keeps its remainder (groups a compound dividend; skips float literals, /= and the explicit ./(...) form) a / b -> a.fdiv(b)
round_half_mode_swap Swap the tie-breaking mode of round for the other two, so a value sitting exactly on a tie rounds differently (literal :up / :even / :down only; skips a mode held in a variable; symbol_literal leaves these modes alone) amount.round(2, half: :even) -> half: :up, half: :down
off_by_one_boundary Lower a count held in a variable by one: the receiver of times, the argument of upto / downto / take / first / last / drop / each_slice / each_cons (skips literal counts, which integer_literal shifts, and safe navigation) n.times -> (n - 1).times, items.first(n) -> items.first(n - 1)
format_specifier_swap Loosen one conversion specifier of a literal format string in format / sprintf / printf / String#%: drop flags, width and precision, or render a non-integer number as %s (skips %%, %{name}, * widths, %c, %p and interpolated or heredoc strings) format("%05d", n) -> format("%d", n), "%.2f" -> "%f", "%s"
alias_removal Drop an alias or alias_method declaration from a class, module or class << self body, attributed to the first method of that scope (skips aliases inside methods, alias_method sent to another receiver and global-variable aliases) alias length size -> (removed)
freeze_removal Drop .freeze from a constant defined in a class or module body, attributed to the first method of that scope, or to the constant itself when the scope has no method of its own (skips values that are frozen anyway: symbols, numbers, nil, booleans, ranges, regexps, and string literals under # frozen_string_literal: true; a freeze inside a method is left to method_call_removal) LIST = %w[a b].freeze -> LIST = %w[a b]
visibility_removal Drop a private, protected or module_function declaration from a class, module or class << self body, attributed to the first method of that scope; a definition it wraps is kept (skips public, private_constant, private_class_method, and declarations guarded by a modifier) private -> (removed), private def a; end -> def a; end
attr_accessor_swap Widen an attr_reader or attr_writer to attr_accessor, and remove an attr_reader / attr_writer / attr_accessor declaration, in a class, module or class << self body, attributed to the first method of that scope (skips a declaration wrapped in private / protected, or guarded by a modifier) attr_reader :name -> attr_accessor :name, (removed)
sleep_to_zero Take the wait out of a sleep / Kernel.sleep whose duration is a variable or an expression (a literal duration is left to integer_literal / float_literal, which already emit sleep 0; skips a sleep sent to another receiver) sleep delay -> sleep 0
regexp_character_type_complement Flip a regexp character type or word boundary to its complement, one at a time (leaves \X, \R and Unicode properties alone) /\d+/ -> /\D+/, /\bword/ -> /\Bword/
regexp_anchor_promotion Promote a regexp line anchor to a string anchor, so the pattern stops matching around newlines (^ -> \A, $ and \Z -> \z) /^admin$/ -> /\Aadmin$/, /^admin\z/
regexp_alternation_branch_deletion Delete one branch of a regexp alternation at a time (skips alternations of more than ten branches, which are lookup tables, and deletions that would break a reference inside the pattern) /cat|dog/ -> /dog/, /cat/
regexp_quantifier_minimum_swap Swap a regexp quantifier between zero-or-more and one-or-more, keeping it lazy or possessive, so only the empty case changes /\A\d*\z/ -> /\A\d+\z/, /a+?/ -> /a*?/
regexp_capture_to_passive Turn a regexp capture group into a passive group, so $1, m[1] or \1 in a replacement loses its value (skips patterns passed to match? / !~, patterns with named groups, and patterns with numbered references) /id=(\d+)/ -> /id=(?:\d+)/
regexp_named_group_rename Rename a named regexp group, with its references inside the pattern, so m[:name] and the local bound by /(?<name>…)/ =~ s no longer find it (skips patterns passed to match? / !~) /(?<user>\w+)@/ -> /(?<_user>\w+)@/
regexp_option_removal Drop the i or m option of a regexp literal where it changes something: i with a cased letter in the pattern, m with a . outside a character class /admin/i -> /admin/, /a.b/m -> /a.b/
keyword_argument Remove keyword defaults/params def foo(bar: 42) -> def foo(bar:)
multiple_assignment Remove targets or swap order a, b = 1, 2 -> b, a = 1, 2
block_removal Remove blocks from method calls items.map { |x| x * 2 } -> items.map
block_pass_removal Remove block arguments passed with & items.map(&:to_s) -> items.map
range_replacement Swap inclusive/exclusive ranges 1..10 -> 1...10
regexp_mutation Replace regexp with always/never matching /pat/ -> /a\A/
regex_simplification Simplify regex quantifiers, anchors, ranges /\d+/ -> /\d/, /[a-z]/ -> /[az]/
receiver_replacement Drop explicit self receiver self.foo -> foo
send_mutation Swap semantically related methods detect -> find, map -> flat_map, round -> floor / ceil
compound_assignment Swap compound assignment operators, or replace the whole assignment with nil; on variables, attributes (obj.count += 1, obj.name ||= x) and indexes (counts[key] += 1, cache[key] ||= x) and, for operator writes, on constants (TOTAL += 1, a subject of its own like a constant assignment) += -> -=, &&= -> ||=
operator_write_receiver_promotion Retarget an operator write on an attribute at the instance variable it goes through (skips other receivers, logical writes and index writes) @account.balance += x -> @account += x
memoization_break Recompute a memoized instance variable on every call (instance variables only; a local, @@x, $x or @cache[key] ||= is left alone) @x ||= expr -> @x = expr
local_variable_assignment Replace variable assignment with nil x = expr -> x = nil
instance_variable_write Replace ivar assignment with nil @x = expr -> @x = nil
class_variable_write Replace cvar assignment with nil @@x = expr -> @@x = nil
global_variable_write Replace gvar assignment with nil $x = expr -> $x = nil
mixin_removal Remove include/extend/prepend include Foo -> removed
superclass_removal Remove class inheritance class Foo < Bar -> class Foo
rescue_removal Remove rescue clauses Deletes rescue block
rescue_body_replacement Replace rescue body with nil Rescue body -> nil
rescue_handler_promotion Run a rescue handler instead of the code it protects, so the happy path never runs: each handler of a begin / method / block rescue in turn, and the fallback of a rescue modifier (drops else, keeps ensure; skips handlers that read the exception, re-raise or retry) begin; fetch(id); rescue NotFound; default; end -> begin; default; end, fetch(id) rescue default -> default
rescue_handler_concatenation Run a rescue handler after the protected code as well, keeping the rescue, so a successful run also does what the handler does (skips empty and value-only handlers, handlers that read the exception, re-raise or retry, and rescue modifiers) begin; fetch(id); rescue NotFound; rollback; end -> begin; fetch(id); rollback; rescue NotFound; rollback; end
rescue_else_concatenation Move the else body of a rescue into the protected code, so an error it raises is now rescued (only under a bare rescue or one naming StandardError / Exception; keeps the rescue and ensure) begin; a; rescue; b; else; c; end -> begin; a; c; rescue; b; end
rescue_class_widening Widen what a rescue clause catches, so a sibling error that should escape is handled too: the listed classes become StandardError, then Exception; a bare rescue, StandardError, or a core class outside it (Interrupt, LoadError, ...) becomes Exception only (skips a clause already naming Exception, and rescue modifiers) rescue KeyError => e -> rescue StandardError => e, rescue Exception => e
inline_rescue Remove inline rescue fallback expr rescue val -> expr
ensure_removal Remove ensure blocks Deletes ensure block
break_statement Remove break statements break -> removed
next_statement Remove next statements next -> removed
redo_statement Remove redo statements redo -> removed
bang_method Swap bang with non-bang methods sort! -> sort
bitwise_replacement Swap bitwise operators a & b -> a | b
bitwise_complement Remove or swap ~ ~x -> x, ~x -> -x
zsuper_removal Replace implicit super with nil super -> nil
explicit_super_mutation Mutate explicit super arguments super(a, b) -> super
index_to_at Replace [] with .at() for arrays arr[0] -> arr.at(0)
index_to_fetch Replace [] with .fetch() h[k] -> h.fetch(k)
index_to_dig Replace [] chains with .dig() h[a][b] -> h.dig(a, b)
index_receiver_to_self Send an index read to self instead of its receiver (skips index writes and reads already on self) h[k] -> self[k]
index_to_key_predicate Replace an index read with a key? check (skips integer and range indexes, reads with several arguments, void statements and index writes) h[k] -> h.key?(k)
index_range_to_drop Replace a to-the-end range index with drop (also the endless a[n..] and a[n...]; skips a literal start of zero or below, ranges stopping short of the last element, void statements and index writes) a[n..-1] -> a.drop(n)
index_write_to_value Replace an index write with the value it assigns, dropping the store; the assignment syntax still evaluates to the same value, the a.[]=(k, v) / a&.[]=(k, v) spellings may not (skips void statements, operator writes and multiple-assignment targets) h[k] = v -> v
index_assignment_removal Remove []= assignments h[k] = v -> removed
pattern_matching_guard Remove/negate pattern guards in x if cond -> in x
pattern_matching_alternative Remove/reorder alternatives pat1 | pat2 -> pat1
pattern_matching_array Remove/wildcard array elements [a, b] -> [a, _]
yield_statement Remove yield or its arguments yield(x) -> yield
splat_operator Remove splat/double-splat (skips ** in a hash literal, ** after a keyword argument or another **, and the rest of a pattern) foo(*args) -> foo(args)
defined_check Replace defined? with true defined?(x) -> true
regex_capture Swap or nil-ify capture refs $1 -> $2, $1 -> nil
loop_flip Swap while/until loops while cond -> until cond
loop_body_to_raise Replace a loop body with raise while c; body; end -> while c; raise; end
method_body_to_raise Replace a whole method body with raise def foo; body; end -> def foo; raise; end
method_body_to_super Replace a method body with bare super where a super target exists def foo; body; end -> def foo; super; end
typed_default_return Replace a single-expression body with the empty value of its inferred type def names(u); u.map(&:name); end -> def names(u); []; end
block_parameter_drop Drop a block's single parameter `users.each {
optional_parameter_to_required Drop an optional positional parameter's default; of several, only the ones Ruby lets become required: the first, and the last unless a *rest parameter follows def f(a = 1) -> def f(a)
optional_default_injection Overwrite an optional parameter with its own default at the top of the body def f(a = 1); body; end -> def f(a = 1); a = 1; body; end
block_destructuring_expansion Flatten a destructuring group in a block's parameters `pairs.each_with_index {
forwarding_super_to_explicit Give a forwarding super an empty argument list def f(a); super; end -> def f(a); super(); end
string_interpolation Replace interpolation content with nil "hello #{name}" -> "hello #{nil}"
retry_removal Remove retry statements retry -> nil
case_when Remove/replace case/when branches Remove when branch, drop one condition from when a, b, body -> nil, empty body -> raise, remove else
case_in Drop one in clause from a case/in, or its else case x; in Integer; …; in String; …; end -> drops an arm; else -> NoMatchingPatternError
pattern_predicate One-line pattern match -> false x in Integer -> false
predicate_replacement Replace predicate calls with booleans x.empty? -> true, x.empty? -> false
equality_to_identity Replace equality with identity check a == b -> a.equal?(b)
lambda_body Replace lambda body with nil -> { expr } -> -> { nil }
begin_unwrap Remove begin/end wrapper begin; expr; end -> expr
block_param_removal Remove explicit block parameter def foo(&block) -> def foo
last_expression_removal Strip trailing literal return from method body def foo?; warn; true; end -> def foo?; warn; end
argument_method_call_replacement Replace a method-call argument with its receiver fn(x.attr) -> fn(x)

MCP Server (AI Agent Integration)

Evilution includes a built-in Model Context Protocol server for direct tool invocation by AI agents (Claude Code, VS Code Copilot, etc.).

Setup

Create a .mcp.json file in your project root:

{
  "mcpServers": {
    "evilution": {
      "type": "stdio",
      "command": "bundle",
      "args": ["exec", "evilution", "mcp"],
      "env": {}
    }
  }
}

Run the server through bundle exec so the project's Gemfile.lock decides which versions of evilution, your test framework and their shared dependencies get loaded. Started outside Bundler, RubyGems picks the newest installed copy of each gem, which your test framework may refuse — the run then stops at the proof-of-life canary with a gem activation conflict. Projects without a Gemfile can use "command": "evilution" and "args": ["mcp"].

The server exposes the following tools:

Tool Description
evilution-mutate Run mutation testing on target files with structured JSON results
evilution-session Inspect mutation testing history — action: list browses saved sessions, action: show displays one, action: diff compares two (fixed/new/persistent survivors, score delta)
evilution-info Discovery before mutation — action: subjects lists the mutation subjects (methods, and class-body subjects such as constant assignments, scopes and callback declarations) with mutation counts, action: tests resolves which specs cover given sources, action: environment dumps the effective config, action: statuses returns the mutation-result status glossary, action: feedback returns the public Discussions URL plus consent + privacy guidance for posting feedback

Verbosity Control

The evilution-mutate tool accepts a verbosity parameter to control response size:

Level Default What's included
summary Yes summary + survived + uncovered + timed_out + errors + unresolved (entries other than survived and uncovered shed diff and error_backtrace; the run's timings block and every entry's timings are dropped, to bound payload size)
full All entries (killed/neutral/equivalent diffs stripped), including timings
minimal summary + survived + uncovered (plus a trimmed sample of up to 3 errored entries when errors > 0)

Use minimal when context window budget is tight and you only need to see what survived (an uncovered mutation is a survivor no test executes, see Uncovered mutations). The trimmed errors sample (each entry: error_message, error_class, location, plus the first 5 backtrace lines) is added so a partly-broken run is still self-diagnosable without escalating verbosity. Use full when you need to inspect killed/neutral/equivalent entries for debugging.

What survives trimming matters when you are deciding whether to trust a score:

  • neutral entries — and with them each neutral_reason — are dropped at summary and minimal. Use full to see why mutations were neutralised.
  • subjects is kept at full and summary, and dropped at minimal, which keeps only summary, survived and uncovered.
  • Everything inside summary survives at every level, including unresolved_target_files, uncovered_code, infra_retried, baseline_neutralized, baseline_failures and the neutral count — so even a minimal response still says whether a target file went untested and how much of the run the score covers.

Enriched Survived Entries

Unlike evilution --format json, every survived and uncovered entry returned by evilution-mutate carries extra fields so the agent can act without a second round-trip:

Field What it gives you
subject Class#method for the mutated subject — points at the exact method to test
spec_file Resolved spec/test path (when one exists) — e.g. an RSpec spec file or Minitest test file, so you can drop new tests straight into it
next_step Concrete natural-language hint — "add a test in X that fails against this mutation at Y:line"

These fields are added in addition to the existing operator, file, line, diff, unified_diff, suggestion, and test_command so agents can triage survivors in one pass.

Concrete Test Suggestions

The evilution-mutate tool accepts a suggest_tests boolean parameter (default: false). When enabled, survived mutation suggestions contain concrete test code that an agent can drop into a test file, instead of static description text. It currently generates RSpec-style suggestions (it/expect blocks).

Pass suggest_tests: true in the evilution-mutate call to activate this mode. The CLI also supports --suggest-tests; when using the CLI, generated suggestions match the --integration setting (RSpec it/expect blocks or Minitest def test_/assert_equal methods). Concrete templates for test-unit are not yet implemented — test-unit runs fall back to generic suggestion text.

Project Config File

evilution-mutate and evilution-info load .evilution.yml (or config/evilution.yml) by default, matching evilution CLI behavior — so timeout, jobs, integration, target, ignore_patterns, and other project settings carry over without the agent having to re-pass them on every call. Explicit tool parameters still win over file settings.

Pass skip_config: true to ignore the project config file. This skips loading .evilution.yml / config/evilution.yml, but MCP-specific overrides (JSON output, quiet mode, preload disabled) and explicit tool parameters still apply.

Iterative Workflow Parameters

evilution-mutate exposes the full set of CLI knobs agents need for iterative TDD:

Parameter Purpose
incremental Cache verdicts (killed, survived, timeout) across runs — set true when iterating on the same files; editing one method re-runs only that method's mutations
integration rspec, minitest, or test_unit
isolation auto, fork, or in_process
baseline false to skip the baseline suite check when you already know it's green
save_session Persist results to .evilution/results/ for inspection via evilution-session
stop_on_kill false to run every selected example instead of stopping at the first deciding failure (default true)
skip_sites Presets of low-value code to leave unmutated: logging, debug_output, error_messages, requires (see Skipping Low-Value Code); skips are counted per preset in summary.skipped_sites
since A git ref: mutate only the lines changed since it, as --since REF does (see Mutating only what changed)

Note: .mcp.json is gitignored by default since it is a local editor/agent configuration file.

After upgrading the gem: restart the MCP server

The MCP server is a long-lived stdio process spawned by the agent host. bundle update evilution swaps the gem on disk but the running process keeps the old code in memory — symptom is opaque "Internal error" responses to flags or shapes the old build doesn't recognize. Restart the server (reload the workspace in Claude Code / Copilot / etc.) so the new gem loads.

evilution version prints the gem version and the bundled mcp gem version on separate lines — run it in the same bundle the MCP server uses to confirm what's loaded.

Feedback channel

When evilution causes friction (errors, usage problems, missing capabilities you wish were there), the MCP responses include a feedback_url plus feedback_hint. The evilution-info tool also exposes action=feedback, which returns the channel URL and posting guidance on demand.

Agents must never post on the user's behalf without explicit user permission. Show the user exactly what you would post, get explicit approval, then post. Never include secrets, environment variables, the project name, file paths, source code, or class/method names from user code — the feedback channel is public.

Discussion URL: https://github.com/marinazzio/evilution/discussions

Contract stability

The three MCP tools (evilution-mutate, evilution-session, evilution-info) form evilution's public contract for AI agents. From 1.0.0 onwards the following are governed by the gem's SemVer policy:

  • Tool names: evilution-mutate, evilution-session, evilution-info.
  • Input schemas: every parameter listed in each tool's input_schema (name, type, enum values, required).
  • Action enumerations: the action enum on evilution-session (list, show, diff) and evilution-info (subjects, tests, environment, statuses, feedback).
  • Output payload top-level shape: the keys and value types documented per action below.
  • Error envelope: an error response is a single text content with the response's error flag set to true. The body is a JSON object with at minimum an error key shaped { "type": <string>, "message": <string> }; tools may add additional top-level keys (e.g. evilution-mutate includes feedback_url and feedback_hint to point agents at the public Discussions channel). Consumers must read the error.type discriminator and ignore unknown extras. The error type strings — currently config_error, parse_error, not_found, and runtime_error — are part of the contract; new types may be added in MINOR releases (additive).

Output schema_version

Successful MCP responses carry a top-level schema_version integer. Two distinct version spaces are in play; both happen to be 1 today and either may be bumped independently at the next MAJOR release:

  • MCP contract schema_version — Evilution::MCP::CONTRACT_VERSION (currently 1). Stamped on envelopes that exist solely to wrap MCP tool output: evilution-info action responses, evilution-session list, evilution-session diff. Bumped only when the MCP envelope shape itself changes incompatibly.
  • Session JSON schema_version — Evilution::Session::Schema::CURRENT_VERSION (currently 1). Embedded inside payloads whose shape is also written to disk and consumed elsewhere: evilution-mutate (returns a mutation report) and evilution-session show (returns a session JSON document). Bumped only when the report/session shape itself changes incompatibly.

Per-tool placement:

Tool / action schema_version source Location in payload
evilution-mutate Session::Schema::CURRENT_VERSION Top-level of the mutation report JSON (same shape as --save-session output).
evilution-session list MCP::CONTRACT_VERSION Top-level of envelope: { "schema_version": 1, "sessions": [...] }.
evilution-session show Session::Schema::CURRENT_VERSION Inside the returned session JSON document.
evilution-session diff MCP::CONTRACT_VERSION Top-level alongside summary / fixed / new_survivors / persistent.
evilution-info (all actions) MCP::CONTRACT_VERSION Top-level of every successful response, injected by the action response formatter.

Per-tool output shapes

  • evilution-mutate — full schema in the JSON Output Schema section. MCP-specific additions to each survived entry are documented in Enriched Survived Entries.
  • evilution-session list — { "schema_version": Integer, "sessions": Array<{ file, timestamp, total, killed, survived, score, duration }> }. Sessions are reverse-chronological; the array is filtered by limit when provided.
  • evilution-session show — the parsed session JSON document, exactly as written under .evilution/results/*.json. Field reference: see Session JSON files.
  • evilution-session diff — { "schema_version": Integer, "summary": { base_score, head_score, score_delta, base_survived, head_survived, base_total, head_total, base_killed, head_killed }, "fixed": Array, "new_survivors": Array, "persistent": Array }. The mutation arrays carry the same per-mutation fields the session survived list uses (operator, file, line, subject, diff).
  • evilution-info subjects — { "schema_version": Integer, "subjects": Array<{ name, file, line, mutations }>, "total_subjects": Integer, "total_mutations": Integer }. Discovery only: this lists what can be mutated. The subjects array in a run's report is a different shape, carrying what each subject scored.
  • evilution-info tests — { "schema_version": Integer, "specs": Array<{ source, spec }>, "unresolved": Array<String>, "total_sources": Integer, "total_specs": Integer }.
  • evilution-info environment — { "schema_version": Integer, "version": String, "ruby": String, "config_file": String|null, "settings": Object }. settings holds the effective value of each run setting under its config key, plus adaptive_timeout: when it is true, timeout is only the fallback and each mutation's limit follows the baseline (timeout_factor, timeout_floor, timeout_ceiling).
  • evilution-info statuses — { "schema_version": Integer, "statuses": Array<{ status, meaning, counted_in_score }> }.
  • evilution-info feedback — { "schema_version": Integer, "discussion_url": String, "consent": String, "privacy": String }.

Deprecation cycle

When a parameter, action, or output field on the public MCP contract is deprecated:

  1. The deprecation is announced in the CHANGELOG and the tool's description text gains a deprecation note.
  2. The deprecated form remains functional for the entire 1.x line — a deprecation introduced in 1.X continues to work in every subsequent 1.X+N release.
  3. The earliest release that may remove the deprecated form is 2.0. Removals are listed in the major-release migration guide.
  4. Adding new parameters, new actions, or new top-level output fields is additive and ships in MINOR releases (existing consumers continue to work).

Not covered by the contract

  • The exact wording of error message strings (the type is contract; the message is a hint that may be reworded).
  • The exact ordering of arrays whose contract calls only for membership (e.g. unresolved source files).
  • Progress-stream notification payloads sent through server_context.notifications — these are best-effort UI signals, not a stable API.
  • Performance characteristics (request latency, memory, parallelism) — improvements ship in any release; regressions are bugs but not contract violations.
  • Default values that are part of the gem's user-facing semantics (timeout, integration default, etc.) — these follow the umbrella SemVer policy and may be tuned.

Recommended Workflows for AI Agents

1. Full project scan

bundle exec evilution run --target 'source:lib/**/*.rb' --format json --min-score 0.8

Parse JSON output. Exit code 0 = pass, 1 = fail — either the score is below --min-score, or a target file resolved to no spec and was never tested (summary.unresolved_target_files names them). Without --min-score no score gate is armed; the run still fails on an untested target file.

2. PR / changed-lines scan (fast feedback)

bundle exec evilution run lib/foo.rb:15-30 lib/bar.rb:5-20 --format json --min-score 0.9

Target the exact lines you changed for fast, focused mutation testing. See line-range syntax below.

Or let git find the lines: --since REF mutates only what changed since REF (see Mutating only what changed):

bundle exec evilution run --since origin/main --format json --min-score 0.9

3. Line-range targeted scan (fastest)

bundle exec evilution run lib/foo.rb:15-30 --format json

Target exact lines you changed. Supports multiple syntaxes:

evilution run lib/foo.rb:15-30    # lines 15 through 30
evilution run lib/foo.rb:15       # single line 15
evilution run lib/foo.rb:15-      # from line 15 to end of file
evilution run lib/foo.rb          # whole file (existing behavior)

Methods whose body overlaps the requested range are included. Mix targeted and whole-file arguments freely:

evilution run lib/foo.rb:15-30 lib/bar.rb --format json

A range is matched against subjects, and not every line belongs to one. Methods are subjects, and so are scopes, AASM guards and callbacks, callback and validation declarations given options or an inline block or lambda (validates :title, presence: true, validate :limit, if: -> { paid? }), Data.define / Struct.new definitions and constant assignments (see step 2 under Internals). Other class-body code — an association, a declaration naming only a method (validate :within_window), a DSL call evilution does not know — is not. When the range holds such code, the run says so on stderr, in the text report and in summary.uncovered_code:

[evilution] app/models/order.rb:12-14 holds code outside every subject; no mutations target those lines. Of the DSL calls in a class body, only scopes, aasm machines, and callback or validation declarations with options or an inline block or lambda are mutated.

Read 0 mutations next to that message as "not measured", not as "nothing to test".

Callback and validation declarations with options add to every model, controller and mailer that has them: about 10–20 mutations for each validates line. Some of those make the declaration invalid (length: {}, inclusion: { in: nil }, validates nil, ...), so the class raises while it loads, and they count as killed like any other mutation that stops the file loading. Each such mutation also runs the declaration once more as written, so option values computed in the class body (in: Status.pluck(:name)) are computed again, and a third-party validates_* macro that keeps its own list of attributes may list it once more. To leave a declaration out, put # evilution:disable on its line or around it (see Disable Comments). Declarations of one class that read the same — the conditions and the options of one declaration, or two validates :email lines — share one subject name, so subjects[] reports them together under one score.

4. Method-name targeted scan

bundle exec evilution run lib/foo.rb --target Foo::Bar#calculate --format json

Target a specific method by its fully-qualified name. Useful when you want to focus on a single method without knowing its exact line numbers.

5. Single-file targeted scan

bundle exec evilution run lib/specific_file.rb --format json

Use when you know which file was modified and want to verify its test coverage.

5a. Multi-file batch scan

bundle exec evilution run lib/models/user.rb lib/models/account.rb lib/models/order.rb

Pass multiple file paths on a single invocation to amortise startup cost. The framework (Rails, Sorbet, etc.) and the preload chain (spec/rails_helper.rb → spec/spec_helper.rb → test/test_helper.rb) load once in the parent process. When --isolation=fork is selected (the default --isolation=auto resolves to fork on Rails projects and packaged gems), every subsequent mutation across all files forks from that warmed parent — materially faster than scripting a for f in ...; do bundle exec evilution run "$f"; done loop, which pays the bootstrap per file. With --isolation=in_process (default for non-Rails, non-gem projects under auto), there is no per-mutation fork, but the parent-process boot still runs once instead of N times. Per-file paths and line numbers are preserved in the report (survived[].file, HTML grouping by source file).

What to read before acting on a score

A score describes only the mutations that got a verdict. Five fields say what it leaves out, and each points at a different action:

Field Meaning What to do
summary.unresolved_target_files A file you named resolved to no spec and was never tested; the run fails on this alone Write a spec, pass --spec, or map it in spec_mappings — do not trust the score until this is empty
subjects[].reached == false Mutations were generated for that method but none got a verdict The method is untested even where its file scores well; start here rather than with survived[]
summary.uncovered_code Lines you targeted hold code outside every subject — class-body code such as DSL calls evilution does not treat as a subject — so no mutation was generated for them; the same is printed to stderr and listed in the text report A run that reports 0 mutations for those lines has not shown they are tested. Target the methods that code calls, or cover it with ordinary tests
neutral[].neutral_reason.kind == "baseline_failure" The mutation's tests failed only on examples that were already failing before any mutation ran; detail names the red spec file Read summary.baseline_failures for why it was red, and fix that first — these mutations have no verdict until it is green
neutral[].neutral_reason.kind == "infra_error" The test process crashed on infrastructure (DB lock, timeout); detail names the class Not a coverage gap. Give parallel workers their own database, or run -j 1

summary.infra_retried reports how many mutations had to be re-run serially because of the last case; a large number means the parallel run was fighting shared infrastructure rather than measuring your suite.

6. Fixing surviving mutants

Every entry in survived[] has already been re-run against the whole resolved spec file, so it is a gap the suite genuinely does not cover rather than an artefact of per-mutation example targeting.

For each entry in survived[]:

  1. Read file at line to understand the code context
  2. Read operator to understand what was changed
  3. Read suggestion for a hint on what test to write (use --suggest-tests for concrete test code)
  4. Write a test that would fail if the mutation were applied
  5. Re-run evilution on just that file to verify the mutant is now killed

A test exists but the mutant still survives: check the matcher

Inside have_attributes, match and nested values of include(key: [...]), RSpec compares an expected array against an Enumerable actual (other than a Hash or Struct) through its to_a. ActiveModel::Errors#to_a is full_messages, so have_attributes(errors: ["Name can't be blank"]) passes for any errors object with that message, not just the one the code should return. A mutant that hands back a different errors object (another record's, or one built with a message instead of an error kind) survives that test.

When the errors object itself matters, assert on the error kinds, or on identity:

# Before: passes for any errors object whose full messages match
expect(result).to have_attributes(errors: ["Name can't be blank"])

# After: the error kind, not just its rendered message
expect(result.errors.details).to eq(name: [{ error: :blank }])
expect(result.errors).to be_of_kind(:name, :blank)
# or, when it must be the record's own errors object
expect(result.errors).to be(participation.errors)

details and of_kind? still accept another record's errors with the same kinds; only the identity check (be) rules that out. The same to_a comparison applies to other Enumerable values (except Hash and Struct) checked against an array inside these matchers.

7. Diagnosing errored mutations

Entries in the JSON errors[] array represent mutations that raised an exception (syntax error, load failure, or runtime crash) rather than producing a test outcome. A mutation that makes its own file raise while loading is not among them when the original file still loads right after it: the mutation is what broke the load, so it is reported killed with an error_message that starts with load failed:. A load failure stays an error when the original fails to load as well, since that says nothing about the mutation. Each entry includes error_class, error_message, and the first 5 error_backtrace lines. Use these fields to decide whether the error is a bug in the mutation operator (file an issue), a load-time problem in the mutated source (often NoMethodError: super called outside of method or constant-redefinition issues), or a genuine crash that the original tests should have caught. Run with --verbose to stream the same error details to stderr during the run.

A test process that dies on an exception no handler saw (a stack overflow that does not unwind, for one) reports the exception before it goes. What the mutation gets depends on when it died:

error_message starts with Isolation Status Meaning
child died, then while running the tests fork killed The mutated source loaded and the tests were running: what a crashing test does.
test run ended by in-process killed The same, in the process of the run.
child died, then before the tests ran fork error Nothing was tested yet. error_class is the exception's class.

Three messages are not about the mutation at all. No verdict was reached, and the entry says only why:

error_message Meaning What to do
child exited 0; its result was unreadable (...) (the status varies) Fork isolation. The child finished, but what arrived on its result pipe does not decode to a result; the parentheses say what was found. Every process the child forks inherits that pipe, so code under test that writes to a descriptor it did not open can put bytes on it. Re-run that one mutation (line-range targeting). If it repeats, look for code that writes to inherited descriptors.
worker's result could not be read; the worker was replaced Worker pool (--jobs above 1). The same thing on the pipe between the run and a worker: the worker was killed and replaced, and the run went on. Re-run that one mutation, with --jobs 1 to take the pool out of the picture.
child died on SIGABRT without writing a result Fork isolation. The Ruby VM aborted (a [BUG] report, typically), so nothing could be reported from inside the child. Other signals read the same way: a segfault is SIGSEGV, a child the kernel killed for memory is SIGKILL. Re-run that one mutation. If it repeats, the mutant crashes the interpreter: that is Ruby's to fix, and the mutation can be disabled with a comment.

The messages in the first column of this last table are examples: the exit status, the signal and the text in parentheses vary. Search errors[] for was unreadable, could not be read and without writing a result.

Under --isolation in_process there is no child to lose: a mutation that takes the Ruby process down takes the run with it, and no report is written. See Exit Codes.

Unresolved mutations — behaviour-named spec layouts

SpecResolver auto-detects mirrored (spec/foo_spec.rb), non-mirrored (spec/unit, spec/lib, test/unit), and dir-grouped (test/unit/<class>/*_test.rb) layouts. It deliberately does not guess for behaviour-named layouts — where specs are named by behaviour rather than by lib path, so no path or class name ties a source file to a spec. The canonical example is aasm: lib/aasm/base.rb is the central DSL class, but it has no base_spec.rb; its behaviour is exercised through the public API by spec/unit/{event,callbacks,guard,api}_spec.rb, none of which name AASM::Base. Every mutation in such a file resolves to no spec and is reported :unresolved (a coverage gap, not a failure — errors=0).

This is intentional: a fuzzy content/grep match would pick a wrong-but-plausible spec subset and report a misleading score. :unresolved plus the warning is the honest signal. To get a real score, opt into one of the two explicit recovery paths the warning names:

# run a directory of specs you know covers the file (--spec-dir globs it)
bundle exec evilution mutate lib/aasm/base.rb --spec-dir spec/unit
# or name explicit files as a single comma-separated --spec value (no shell glob)
bundle exec evilution mutate lib/aasm/base.rb --spec spec/unit/event_spec.rb,spec/unit/callbacks_spec.rb

# or run the whole suite per mutation (slower, but correct-by-superset)
bundle exec evilution mutate lib/aasm/base.rb --fallback-full-suite

To make the mapping permanent, declare it once in .evilution.yml under spec_mappings; the run, its baseline and tests list all honour it. evilution tests list <file> prints exactly the spec files a run would use, so it is the quick way to check a mapping before spending a run on it:

spec_mappings:
  lib/aasm/base.rb:
    - spec/unit/event_spec.rb
    - spec/unit/callbacks_spec.rb

Without --fallback-full-suite an unresolved source is also left out of the baseline, which does not run the whole test directory for it.

Long Minitest fork runs — not a hang

Minitest projects under --isolation=fork re-bootstrap the test environment (test_helper.rb, plugins, runnable state) once per mutation. On constant-heavy files (e.g. Shopify/liquid's lib/liquid/lexer.rb, ~270 mutations) the wall-clock cost is dominated by that per-fork bootstrap and any mutations that hit a --timeout rather than killing the test fast. A single-worker run (-j 1) on a few hundred mutations can take 4+ minutes; combined with --no-progress and a non-TTY stderr (CI, redirected logs) the run looks silent the entire time.

Recommended invocation for Minitest fork canaries:

RUBYOPT="-Itest" bundle exec evilution mutate lib/<file>.rb \
  -j 4 -t 10 \
  --integration=minitest --isolation=fork \
  --spec test/<dir>/<file>_test.rb

-j 4 parallelises across workers, -t 10 caps any mutation that pathologically loops at 10 s. On a Rails app, add --warmup rails so each fork does not re-initialise I18n, routes and templates on its first request (see Warming up Rails before forking). Expect the run to print progress only when stderr is a TTY (use bundle exec evilution mutate ... 2>&1 | tee log to get progress while still saving output). The historical "Minitest fork hangs on liquid" report (GH #1211) turned out to be a slow run + silent UX, not an actual deadlock — the worker logs show steady forward progress when captured via --quiet-children --quiet-children-dir DIR.

8. CI gate

bundle exec evilution run --target 'source:lib/**/*.rb' --format json --min-score 0.8 --quiet
# Exit code 0 = pass, 1 = fail, 2 = error

Exit 1 covers two conditions: the score missed --min-score, and a target file that resolved to no spec. The second fails the run whether or not a score gate is set, so a CI step that names files explicitly cannot silently stop testing one of them.

Note: --quiet suppresses all stdout output (including JSON). Use it in CI only when you care about the exit code and do not need JSON output. --output FILE is the alternative when a preloaded spec helper (SimpleCov, for example) writes to stdout on exit.

9. Regression tracking across runs (compare)

bundle exec evilution run --target 'source:lib/**/*.rb' --format json --save-session
# later, after edits:
bundle exec evilution run --target 'source:lib/**/*.rb' --format json --save-session

bundle exec evilution compare \
  --against .evilution/results/<earlier>.json \
  --current  .evilution/results/<later>.json \
  --format json

Output buckets:

Bucket Meaning
fixed Survived previously, killed now — the new test actually landed
new Did not exist previously, surviving now — a fresh gap
persistent Survived in both runs — carry-over debt
reintroduced Killed previously, survived now — regression
flaky Status flipped and back — unstable test

Use in CI to gate merges on reintroduced being empty, or to surface new survivors for reviewer attention without failing the build on persistent debt.

Warming up Rails before forking

Under isolation: fork the parent preloads once and every mutation runs in a fresh fork, which inherits only what the parent had already initialised. Rails initialises a lot lazily on a process's first request: loading locale files, building route helpers, resolving asset paths, compiling templates. Without a warm-up, every forked mutation whose tests make a request pays that cold start again, often several hundred milliseconds each.

warmup: rails (or --warmup rails) does that initialisation once in the parent, right after the preload:

  • I18n.eager_load!
  • the route set's url_helpers (and eager_load! for lazily drawn routes on Rails 8)
  • an asset path lookup through the controller helpers
  • template precompilation, only if the actionview_precompiler gem is installed

Each step is skipped if its library is not loaded, and skipped with a one-line note if it raises, so an app that does not fit a step still runs normally.

Do not warm up by sending a real request from your preload (Rails.application.call(...)). Controller callbacks change process-global state that every fork then inherits: I18n.locale, gem request stores such as PaperTrail's, a session row in the database. Tests that depend on that state then fail in every fork, and evilution reports their mutations as killed. The steps above make no request and leave that state untouched.

Parallel Runs with SQLite

Running with -j N forks worker processes. If your Rails app uses SQLite and every worker opens the same db/test.sqlite3 file, concurrent writers collide on the database-level lock (ActiveRecord::StatementTimeout, SQLite3::BusyException, slow runs), and, worse, one worker's fixture loads and writes land in the middle of another worker's tests. Verdicts then depend on which mutations happen to run at the same time: on a Redmine model, a shared database turned 47 of 500 survivors into kills at -j 4 that a -j 1 run does not reproduce.

Per-worker database copies

By default each worker gets a database of its own, with no setup. When the app is preloaded in the parent (the default for a Rails app under fork isolation, see --preload) and --jobs is above 1, a worker that would share the preloaded SQLite test database makes a copy of it when it starts and reconnects ActiveRecord::Base to the copy. The copy goes through SQLite's backup API, so it holds what the write-ahead log holds and is no larger than the data; it is taken after the preload and the baseline, from a database no worker writes to, so every worker starts from the same schema and data. A worker started in place of a dead or recycled one makes a fresh copy. The ActiveRecord::Base.configurations entry is pointed at the copy too, so establish_connection(:test) in a test stays on it.

The copies live in a directory under the system temp dir (evilution-worker-db-<pid>-<random>/worker-<worker pid>/) and are deleted when the parallel pass ends, also when it ends on an error or Ctrl-C. A worker's copy is deleted as soon as the worker is gone and before its replacement starts, so the disk never holds more than one copy per job; only a run killed with SIGKILL leaves the directory behind. The database in database.yml is not written by the workers.

A worker copies only when all of these hold:

  • the app was loaded by the preload, so the worker inherits an ActiveRecord::Base connection;
  • that connection still matches database.yml (a connection your test helper set up by other means is left alone);
  • the adapter is SQLite and the database is a file path, not :memory: (never shared anyway) or a file: URI;
  • the worker would otherwise use the preloaded database: database.yml names one database for all workers, or the worker's own TEST_ENV_NUMBER database (below) is not prepared.

Not covered yet: a run with --no-preload (the app boots in each mutation's process and reads database.yml there, so use TEST_ENV_NUMBER below), connections other than ActiveRecord::Base's (other databases set up with connects_to, replicas keep using the shared databases, and each worker says so in a warning), and PostgreSQL or MySQL, where a per-worker database would come from a template database; for those, set up per-worker databases with TEST_ENV_NUMBER. If a copy fails (a full disk, a locked database), the worker warns and shares the preloaded database as before.

Turn it off with --no-copy-worker-databases or copy_worker_databases: false. The setting is part of the incremental cache fingerprint, since it can change verdicts.

Infrastructure crashes

Contention that still happens (copies off, or a run the copies do not cover) is handled after the fact. Evilution classifies these crashes as :neutral (see GH #814) so the mutation score is not polluted, and because the contention only exists while several workers are running, those mutations are re-run one at a time once the pool is done, and the verdict from the quiet re-run is the one reported. Without it the neutral bucket moved with -j on identical input — the same files scoring 24 killed / 0 neutral at -j 1 and 6 killed / 18 neutral at -j 4 (GH #1607). The run says how much it had to redo:

! 18 mutations hit infrastructure errors under parallel workers; re-ran them serially.

The count is in JSON output as summary.infra_retried. Such a crash is also never written to the --incremental cache: the cache keeps no error class, so a cached :killed would be indistinguishable from a real one on the next run and would short-circuit both the demotion and the retry. A neutral from a failing baseline is never re-run — that is a real statement about the spec, not a missed verdict. The retry costs wall-clock time in proportion to the contention, and it does not undo a wrong kill: a test that failed on another worker's data rather than on a lock is not an infrastructure crash. Give each worker its own database instead.

Your own per-worker databases (TEST_ENV_NUMBER)

Evilution follows the parallel_tests convention: each worker receives a TEST_ENV_NUMBER environment variable ("" for worker 1, "2" for worker 2, "3" for worker 3, …). Interpolate it into config/database.yml so each worker gets its own database:

test:
  adapter: sqlite3
  database: db/test<%= ENV['TEST_ENV_NUMBER'] %>.sqlite3
  pool: 5
  timeout: 5000

Each worker needs its own prepared database (db/test.sqlite3, db/test2.sqlite3, …, one per job). Create and load them before running Evilution:

bin/rails db:test:prepare
TEST_ENV_NUMBER=2 bin/rails db:test:prepare
TEST_ENV_NUMBER=3 bin/rails db:test:prepare   # … up to the number of jobs

Without --preload, a test helper that loads the schema when it boots (for example ActiveRecord::Migration.maintain_test_schema! in rails_helper.rb) can do this instead, because it runs inside each worker.

Per-worker databases also work with --preload, which boots the app (and resolves database.yml) once in the parent before the workers fork: each worker re-renders database.yml with its own TEST_ENV_NUMBER and, when the result differs, re-establishes ActiveRecord::Base's connection to it. Only the primary connection is switched, and only when it still matches database.yml. Under --preload the preload's own schema setup runs once, in the parent, against the parent's database only, so prepare the worker databases with the commands above. A worker whose database is missing or lacks a table or migration of the parent's database goes back to the parent's database and prints which database to prepare; with per-worker copies on, it then uses a copy of the parent's SQLite database, otherwise its mutations share the parent's database as they would without the switch. Worker 1's TEST_ENV_NUMBER is empty, so its database is the parent's; with copies on it uses a copy of it, like any worker left on the parent's database.

When Evilution detects a parallel run against a SQLite test database whose path does not depend on TEST_ENV_NUMBER and the per-worker copies do not cover it (they are off, the app was not preloaded, or the preloaded connection is not one they copy: a file: URI, or a connection that no longer matches database.yml), it prints a one-time startup warning pointing to this section. An in-memory database (:memory:) is never shared between workers and does not trigger it.

Development

Architecture

New to the codebase? docs/architecture.md is the contributor guide: the end-to-end mutation flow (source → result), a module map, and step-by-step recipes for adding a mutator, a reporter, or a test integration.

Memory leak check

Run before releasing to verify no memory regressions:

bundle exec rake memory:check

Tests 4 paths (InProcess isolation, Fork isolation, mutation generation + stripping, parallel pool) by running repeated iterations and asserting RSS stays flat. Configurable via environment variables:

  • MEMORY_CHECK_ITERATIONS — number of iterations per check (default: 50)
  • MEMORY_CHECK_MAX_GROWTH_KB — maximum allowed RSS growth in KB (default: 10240 = 10 MB)

Internals (for context, not for direct use)

For the full contributor architecture — module map, data flow, and extension points — see docs/architecture.md.

  1. Parse — Prism parses Ruby files into ASTs with exact byte offsets
  2. Extract — Methods are identified as mutation subjects, and so are value-object definitions outside any method (Point = Data.define(:x, :y), class Coord < Struct.new(:lat, :lng)), which only the operators that apply to them mutate, every other constant assignment outside a method (LIMIT = 10, LIMIT += 1), named after the constant (Config::LIMIT); constant_write_to_nil mutates a plain assignment, compound_assignment an operator write, and the literal operators the value of either, and ActiveRecord scopes declared with a literal body in a class body (scope :recent, -> { where(recent: true) }), named after the class method they define (Order.recent) and mutated like a method body — also when declared in a concern's included do ... end or prepended do ... end block, where they are named after the concern (Publishable.published). A method defined in a concern's included do ... end or prepended do ... end block is mutated on the classes that took the concern before the run as well as on those that take it later; likewise the guards and callbacks written out as lambdas or blocks inside an AASM event or state declaration (event :ship, guard: -> { address? }), named after the method that declaration defines (Order#ship, Order#paid?), or after the concern when the machine is declared in its included do ... end or prepended do ... end block (Shippable#ship); and the conditions and bodies written out in callback and validation declarations of a class body or of a concern's included do ... end or prepended do ... end block (validate :credit_limit, if: -> { paid? }, before_save { ... }), named the way the declaration reads (Order.validate(:credit_limit), Order.before_save, Publishable.validates(:title)); and the arguments of such a declaration given options (validates :for_lift, inclusion: { in: [true] }, if: :lift_round?), under the same name — its literals, option values and conditions are mutated and options dropped one at a time. Such a mutation is applied by running the declaration again: what it registered when the file loaded, found by running the declaration as written once more, gives its place to what the mutated declaration registers, or to nothing when it registers none (presence: false)
  3. Filter — Disable comments, Sorbet sig blocks, and AST ignore patterns exclude mutations before execution
  4. Mutate — 155 operators produce text replacements at precise byte offsets (source-level surgery, no AST unparsing); heredoc literal text is skipped by default. Identical byte-mutations from different operators are deduplicated by (file_path, mutated_source) so the count is not inflated by overlap
  5. Isolate — Mutations are applied to temporary file copies (never modifying originals); load-path redirection ensures require resolves the mutated copy. Default isolation is in-process for plain Ruby projects (no gemspec) and fork for Rails projects and packaged gems (auto-detected); --isolation fork forces forked child processes. Both sequential and parallel (--jobs N) modes respect the configured isolation strategy
  6. Test — The configured test framework (RSpec, Minitest, or Test::Unit) executes against the mutated source
  7. Collect — Source strings and AST nodes are released after use to minimize memory retention
  8. Report — Results aggregated into text, JSON, or HTML, including efficiency metrics and peak memory usage

Versioning

evilution follows Semantic Versioning. The full policy — what counts as the public contract, what triggers a major bump, how deprecations work — is documented in docs/versioning.md.

There is no public Ruby API: the entire Evilution:: namespace is internal and may change in any release. Drive evilution through the CLI, .evilution.yml, the MCP tools, or hooks. See docs/public_api.md.

Repository

https://github.com/marinazzio/evilution