Smackdown lets you compare changes introduced by a particular git repo's branch, and compares that against a simplecov-format test coverage report in order to report on changed or added lines that are not covered by any tests.

The main intended use case for this is as a test coverage "linter" for pull requests, to make sure that new or changed code is supported by tests.

Say we're working on a feature branch called my_feature, that we originally branched off of master. In order to check that all our new and changed code is covered by tests, do the following:

• If you don't already have simplecov set up, check out the README to get started: https://github.com/colszowka/simplecov.

• Install the simplecov-json gem and configure simplecov to output JSON as well:

Gemfile

gem 'simplecov-json', require: false
gem 'smackdown'

Rakefile

require 'smackdown'

test_helper.rb (or wherever you are running SimpleCov.start)

require 'simplecov'
require 'simplecov-json'

SimpleCov.start do
  SimpleCov.formatters = [
    SimpleCov::Formatter::HTMLFormatter,
    SimpleCov::Formatter::JSONFormatter
  ]
end

• Run your tests to produce the coverage report (coverage/coverage.json).

bundle exec rake test

• Run smackdown.

bundle exec rake smackdown

=> All new and modified code is covered!

• If your branch doesn't have tests for all new and modified code you've committed (compared to master), you'll instead see a report resembling this:

lib/class_with_uncovered_lines.rb
18:     puts "This method was added but no test is written to cover it!"
19:     @some_var = 5

-----------------------------------------
lib/new_class_with_complete_coverage.rb
100% covered!

-----------------------------------------
lib/new_class_with_no_coverage.rb
No coverage for this file!

-----------------------------------------

• After you add new tests, remember to re-run your test suite to generate a new coverage report for smackdown to use.

lib/tasks/my_custom_smackdown_task.rake

desc 'My custom smackdown task'
task :smackdown_custom do
  reporter = Smackdown::CoverageDiffReporter.new(
    "path/to/repo",
    coverage_report_path: "/path/to/repo/coverage/coverage.json",  # This can be a filepath or even a url
    report_path_prefix: "/path/to/repo",  # A string to prefix to files' relative paths in order to make them match the paths
                                          # recorded in the coverage report
    head: "my_feature_for_next_release",
    merge_base: "release_candidate_2016_08_01",
    filters: Smackdown::CoverageDiffReporter.DEFAULT_FILTERS + %w(one_time_scripts/)
  )
  reporter.run

  if reporter.completely_covered?
    puts "All new and modified code is covered!"
  else
    reporter.report do |file_coverage_diff|
    puts file_coverage_diff.relative_path

    if file_coverage_diff.coverage_available?
      if file_coverage_diff.covered?
        puts "100\% covered!"
      else
        file_coverage_diff.uncovered_lines.each do |line|
          puts "#{line.line_num}: #{line.line_content}"
        end
      end
    else
      puts "No coverage for this file!"
    end

    puts "\n-----------------------------------------\n"
  end

  exit(reporter.completely_covered?)
end

NOTES:

• This is example usage and what you do in the block passed to the CoverageDiffReporter#report method is up to you.
• Many of the options passed to CoverageDiffReporter#new are optional and default to common values. See lib/coverage_diff_reporter.rb for more information.
• Instead of passing the path to the coverage report in the coverage_report_path parameter, you can instead pass a json string in as coverage_json: my_json_string

Clone the repo, then:

cd smackdown
bundle install
git submodule init
git submodule update --remote
ruby test/script/init.rb  # This will run the tests in the smackdown_test_repo subproject on the my_branch 
                          # and full_smackdown_coverage branches and produce test coverage reports for each
bundle exec rake test

Open coverage/index.html and verify that you have 100% test coverage!

If you create a branch (i.e. in order to contribute to the gem), you can even use smackdown on the smackdown repo itself in order to verify that your branch's changes are all tested:

bundle exec rake smackdown  # dogfooding!

The test setup here is a little unusual due to the fact that this gem's functionality fundamentally involves comparing different revisions in a git repo.

We test this by making use of an ACTUAL git repo which exists solely for the purpose of testing the smackdown gem, and which is included here as a submodule (hence the submodule init and update steps): https://github.com/robesris/smackdown_test_repo

The smackdown_test_repo contains three branches as of this writing: master, my_branch, and full_smackdown_coverage. The master branch includes some sample "app code" that simply defines a few classes with arbitrary methods, as well as a tiny test suite. The test suite tests one of the methods in the class_with_uncovered_code_added.rb file (covered_method) but not the other (existing_uncovered_method).

Now, my_branch represents code changes from master, and adds a new method (really just uncomments the uncovered_method method). However, it does not add any tests to cover this new method. It also adds a new file with comprehensive tests (new_class_with_complete_coverage.rb), and omits tests entirely for another new file (new_class_with_no_coverage.rb)

Finally, the full_smackdown_coverage branch adds tests for all new and changed code that remains untested in my_branch, for the purpose of testing a 100% coverage scenario.

Running ruby test/script/init.rb as described above runs the test suite on the full_smackdown_coverage and my_branch branches, and creates a coverage report for each in the submodule root as coverage/full_smackdown_coverage.json and coverage/coverage.json, respectively.

What we want to see, in order to verify that smackdown is functioning correctly, is as follows:

• The new untested method (uncovered_method) is identified by smackdown as newly-introduced but uncovered code.
• The untested method that already existed on master (existing_uncovered_method) is ignored, as it was not introduced by the code changes made in my_branch.
• The file with no coverage at all (new_class_with_no_coverage) is reported as such.
• The full_smackdown_coverage branch is reported as having no changed or added code that is untested.

The tests in the main smackdown repo are geared towards testing for these characteristics.

Thanks to @ivantsepp (and his lintrunner gem: https://github.com/ivantsepp/lintrunner) for help and inspiration getting started with this project!