rspec-conductor

There is a common issue when running parallel spec runners with parallel-tests: since you have to decide on the list of spec files for each runner before the run starts, you don't have good control over how well the load is distributed. What ends up happening is one runner finishes after 3 minutes, another after 7 minutes, not utilizing the CPU effectively.

rspec-conductor uses a different approach, it spawns a bunch of workers, then gives each of them one spec file to run. As soon as a worker finishes, it gives them another spec file, etc.

User experience was designed to serve as a simple, almost drop-in, replacement for the parallel_tests gem.

2x sped-up recording of what it looks like in a real project.

Add to your Gemfile:

gem 'rspec-conductor'

rspec-conductor <OPTIONS> -- <RSPEC_OPTIONS> <SPEC_PATHS>
rspec-conductor --workers 10 -- --tag '~@flaky' spec
# shorthand for setting the paths when there are no rspec options is also supported
rspec-conductor --workers 10 spec

--verbose flag is especially useful for troubleshooting.

Server process preloads the rails_helper, prepares a list of files to work, then spawns the workers, each with ENV['TEST_ENV_NUMBER'] = <worker_number> (same as parallel-tests). The two communicate over a standard unix socket. Message format is basically a tuple of (size, json_payload). It should also be possible to run this process over the network, but I haven't found a solid usecase for this yet.

rspec-conductor comes with 3 formatters out of the box:

• --formatter plain displays a bunch of dots, like rspec --format progress
• --formatter ci displays current progress every 10 seconds, useful in CI environments, especially if the output isn't flushed automatically for whatever reason (jenkins does that), looks like this:

--------------------------------
Current status [17:37:11]:
Processed: 24 / 159 (15%)
322 passed, 0 failed, 0 pending
--------------------------------

• --formatter fancy is the formatter shown in the gif above, it displays a list of workers, a row of dots and the last error message (if any). Useful for local development.

When launched without an explicitly specified formatter option, rspec-conductor defaults to either plain or fancy depending on the terminal parameters.

If you want ten workers, you're going to need ten databases. Something like this in your database.yml file:

test:
  primary:
    <<: *default
    database: database_test<%= ENV['TEST_ENV_NUMBER'] %>

That means worker number 3 is going to use database database_test3, worker number 4 database_test4 and so on. Worker number 1 is special: with --first-is-1 flag on it uses TEST_ENV_NUMBER="1", but without it, it uses TEST_ENV_NUMBER="" (empty string).

In order to bootstrap the test databases, there is a rake task:

# Recreate and seed test databases with TEST_ENV_NUMBER 1 to 10
rails rspec_conductor:setup[10]

# If you like the first-is-1 mode, keeping your parallel test envs separate from your regular env:
RSPEC_CONDUCTOR_FIRST_IS_1=1 rails rspec_conductor:setup[10]

You can also set the env variable RSPEC_CONDUCTOR_DEFAULT_WORKER_COUNT to change the default worker count to avoid typing the quotes for the rake task arguments in zsh.

export RSPEC_CONDUCTOR_DEFAULT_WORKER_COUNT=10
rails rspec_conductor:setup # assumes [10]

This library is very minimalistic by design. While there is a dependency on rspec-core, obviously, but there are no dependencies beyond that. Even the TUI stuff is handled internally.

I will also try my best to keep supporting as many rubies / rspec versions as I can, which is, currently, ruby 2.6.0+ (released in 2018) and rspec 3.8.0+ (also released in 2018), although that is more of a soft promise than a blood vow.

• +[__NSCFConstantString initialize] may have been in progress in another thread when fork() was called. on M-based Mac machines
  • This is a common issue with ruby code, compiled libraries and forking. Set OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES environment variable to work around this
• Something gets loaded that shouldn't get loaded, or in a different order
  • There are two simple ways to hook into preloads, exposed as CLI flags, --prefork-require (defaults to config/application.rb) and --postfork-require (defaults to either rails_helper.rb or spec_helper.rb, whichever is present on your machine). You can set any of those to whatever you need and control the load order
• --require options are ignored in command-line arguments or my .rspec
  • rspec-conductor uses the full rspec configuration machinery to parse all the params. That includes .rspec, which, by default, includes --require spec_helper or --require rails_helper in most setups. If you need tighter control over what you load, use either --prefork-require or --postfork-require, these are actual ruby files, you can put any code or any require you need there.

• Why not preload the whole rails environment before spawning the workers instead of just rails_helper?

Short answer: it's unsafe. Any file descriptors, such as db connections, redis connections and even libcurl environment (which we use for elasticsearch), are shared between all the child processes, leading to hard to debug bugs.

• Why not use any of the existing libraries? (see Prior Art section)

test-queue forks after loading the whole environment rather than just the rails_helper (see above). ci-queue is deprecated for rspec. rspecq I couldn't get working and also I didn't like the design.

• parallel-tests
• flatware

• test-queue
• rspecq
• ci-queue