Buildkite Collectors for Ruby

DEPRECATION NOTICE Versions prior to 2.1.x are unsupported and will not work after mid-2023. Please upgrade to the latest version.

Official Buildkite Test Engine collectors for Ruby test frameworks ✨

⚒ Supported test frameworks: RSpec, Minitest, Cucumber, and more coming soon.

📦 Supported CI systems: Buildkite, GitHub Actions, CircleCI, Codeship, and others via the BUILDKITE_ANALYTICS_* environment variables.

Create a test suite, and copy the API token that it gives you.

Add the buildkite-test_collector gem:

gem install buildkite-test_collector

Or add this to your Gemfile’s test group:

group :test do
  gem 'buildkite-test_collector'
end

Add the following code to your RSpec setup file:

# spec/spec_helper.rb
require 'buildkite/test_collector'
Buildkite::TestCollector.configure(hook: :rspec)

Run your tests locally:

BUILDKITE_ANALYTICS_TOKEN=xyz rspec

Add the following code to your Minitest setup file:

# test/test_helper.rb
require 'buildkite/test_collector'
Buildkite::TestCollector.configure(hook: :minitest)

Run your tests locally:

BUILDKITE_ANALYTICS_TOKEN=xyz rake

Add the following code to your Cucumber setup file:

# features/support/env.rb
require 'buildkite/test_collector'
Buildkite::TestCollector.configure(hook: :cucumber)

Run your tests locally:

BUILDKITE_ANALYTICS_TOKEN=xyz cucumber

Add the BUILDKITE_ANALYTICS_TOKEN secret to your CI, push your changes to a branch, and open a pull request 🎉

RSpec suites can also send an OpenTelemetry trace per test execution to Buildkite, showing what each test did and where it spent its time. Each trace is rooted in a test.execution span naming the test, its file, and whether it passed.

This is still under development and everything here may change. It is off by default, so opt in when you configure the collector:

Buildkite::TestCollector.configure(hook: :rspec, otel_enabled: true)

If your suite already runs OpenTelemetry, we use your existing setup and your instrumentation as it is. If it doesn't, we set one up and install the instrumentation this gem bundles, so you get spans for the databases, caches, HTTP clients, and background jobs your tests actually touch.

Export needs Ruby 3.3 or newer, which is what the OpenTelemetry gems require. On older Rubies the option is accepted and does nothing.

Spans need BUILDKITE_ANALYTICS_TOKEN to be an agent OIDC token with the write_uploads scope, from buildkite-agent oidc request-token. A suite API token still uploads executions, but its spans are rejected.

Export failures never fail a test or block the normal Test Engine upload. See the OpenTelemetry guide for what you get and how it fits around an existing OpenTelemetry setup.

For more use cases such as custom tags, annotations, and span tracking, please visit our official Ruby collector documentation for details.

After cloning the repository, install the dependencies:

bundle

And run the tests:

bundle exec rspec

Useful resources for developing collectors include the Buildkite Test Engine docs.

See DESIGN.md for an overview of the design of this gem.

Bug reports and pull requests are welcome on GitHub at https://github.com/buildkite/test-collector-ruby

1. Bump the version in version.rb and run bundle to update the Gemfile.lock.
2. Update the CHANGELOG.md with your new version and a description of your changes.
3. Once your PR is merged to main git tag the merge commit and push:

git tag vX.X.X
git push origin vX.X.X

4. Visit the release pipeline to unblock it and confirm the new version is pushed to rubygems.org
5. Create a new release in github.

The gem is available as open source under the terms of the MIT License.