Checkset

A browser-based verification suite built on Playwright. Run checks against any URL — local dev, staging, or production — with automatic screenshots, structured JSON results, and a prep system for state management.

Inspired by Basecamp's Upright Playwright probes, but focused on on-demand verification rather than synthetic monitoring.

Installation

Add to your Gemfile:

gem "checkset"

Then install Playwright:

npm install playwright
npx playwright install chromium

Quick Start

Scaffold a new project:

bundle exec checkset init

This creates a starter checkset.yml and checks/homepage_loads.rb to get you going.

Create a check in checks/:

# checks/homepage.rb
class Checks::Homepage < Checkset::Check
  description "Verifies the homepage loads correctly"

  def call
    visit "/"

    verify "page has title" do
      page.title.include?("My App")
    end

    verify "has main heading" do
      page.get_by_role("heading", name: "Welcome").visible?
    end
  end
end

Run it:

bundle exec checkset --target https://staging.myapp.com

The DSL

Checks are Ruby classes that subclass Checkset::Check. Inside call, you use two primitives:

`verify` — assertions that continue on failure

verify "cart has items" do
  page.get_by_test_id("cart-count").text_content.to_i > 0
end

The block returns truthy (pass) or falsy (fail). On failure, the check keeps running to collect all failures in one run.

`step` — actions that halt on failure

step "add product to cart" do
  page.get_by_role("button", name: "Add to Cart").first.click
end

Steps perform navigation or interaction. On failure, the check stops because subsequent steps depend on the state this one creates.

`visit` — navigate relative to the target URL

visit "/products"  # goes to https://staging.myapp.com/products

Full example

class Checks::UserCanCheckout < Checkset::Check
  prep :sign_in_as_customer
  description "Verifies the full checkout flow"
  tags :checkout, :critical

  def call
    visit "/products"

    step "add product to cart" do
      page.get_by_role("button", name: "Add to Cart").first.click
    end

    verify "cart has items" do
      page.get_by_test_id("cart-count").text_content.to_i > 0
    end

    step "go to checkout" do
      page.get_by_role("link", name: "Cart").click
      page.get_by_role("button", name: "Checkout").click
    end

    verify "reached checkout page" do
      page.url.include?("/checkout")
    end
  end
end

Preps

Preps handle state setup before a check runs — signing in, seeding data, etc. They run in the same browser context as the check, so session state carries over.

# preps/sign_in_as_admin.rb
class Preps::SignInAsAdmin < Checkset::Prep
  def satisfy(page)
    page.goto("#{target_url}/login")
    page.get_by_label("Email").fill(credentials[:admin_email])
    page.get_by_label("Password").fill(credentials[:admin_password])
    page.get_by_role("button", name: "Sign in").click
    page.wait_for_url("**/dashboard")
  end

  # Optional: skip sign-in if already authenticated
  def satisfied?(page)
    page.goto("#{target_url}/dashboard")
    !page.url.include?("/login")
  rescue
    false
  end
end

Declare preps on your check:

class Checks::AdminDashboard < Checkset::Check
  prep :sign_in_as_admin

  def call
    visit "/admin/dashboard"
    verify("dashboard loads") { page.get_by_role("heading", name: "Dashboard").visible? }
  end
end

Multiple preps run in declaration order:

class Checks::UserCanCheckout < Checkset::Check
  prep :sign_in_as_customer
  prep :test_customer_account
  # ...
end

Suites

When you need to run checks against multiple domains, create a checkset.yml to map suite names to target URLs. Checks are auto-discovered by folder — no need to list them manually.

Folder structure

checks/
├── app/
│   ├── user_can_sign_in.rb    → belongs to "app" suite
│   └── user_can_checkout.rb   → belongs to "app" suite
├── admin/
│   └── admin_dashboard.rb     → belongs to "admin" suite
└── homepage.rb                → top-level = runs in EVERY suite

`checkset.yml`

# Default domain for %{domain} interpolation
base_domain: afomera.dev

# Optional: override the checks directory (default: checks)
# checks_dir: path/to/checks

suites:
  app:
    target: https://app.%{domain}
  admin:
    target: https://admin.%{domain}

Targets support two kinds of interpolation:

%{domain} — replaced with base_domain from the yml (or --domain CLI flag)
${ENV_VAR} — replaced with the environment variable's value

suites:
  app:
    target: https://app.%{domain}         # uses base_domain or --domain
  monitoring:
    target: https://${MONITORING_HOST}     # uses env var
  marketing:
    target: https://marketing.example.com  # hardcoded, no interpolation

Running suites

# Run all suites (uses base_domain from yml)
bundle exec checkset

# Override the domain for staging/preview environments
bundle exec checkset --domain staging.afomera.dev

# Run just one suite
bundle exec checkset --suite app

# ENV vars work too
MONITORING_HOST=status.example.com bundle exec checkset

# Override with --target (ignores yml, loads all checks)
bundle exec checkset --target https://staging.myapp.com

Each suite runs checks from its subfolder (checks/app/**/*.rb) plus any top-level checks (checks/*.rb). All suites run concurrently, sharing a single browser for efficiency.

CLI

# Basic usage
bundle exec checkset --target https://staging.myapp.com

# Run a specific check
bundle exec checkset --target https://staging.myapp.com Checks::UserCanCheckout

# Debug mode — headed browser with slow motion
bundle exec checkset --target http://localhost:3000 --headed --slow-mo 500

# Run checks in parallel
bundle exec checkset --target https://staging.myapp.com --parallel 4

# Run only checks tagged :critical
bundle exec checkset --target https://staging.myapp.com --tag critical

# Retry failed checks up to 2 times (useful for flaky browser checks)
bundle exec checkset --target https://staging.myapp.com --retries 2

# All options
bundle exec checkset --help

Options

Flag	Description	Default
`--target URL`	Target URL to verify against	required (unless `checkset.yml` exists)
`--suite NAME`	Run only the named suite from `checkset.yml`	all suites
`--config FILE`	Path to config file	`checkset.yml`
`--domain DOMAIN`	Base domain for `%{domain}` in suite targets	`base_domain` from yml
`--tag TAG`	Only run checks tagged with TAG	all checks
`--retries N`	Retry failed checks N times	`0`
`--clean`	Remove all screenshots, traces, and results
`--headed`	Run browser in headed mode	headless
`--browser TYPE`	`chromium`, `firefox`, or `webkit`	`chromium`
`--parallel N`	Run N checks concurrently	`1`
`--slow-mo MS`	Slow down actions by N ms	off
`--timeout MS`	Default timeout in ms	`10000`
`--screenshots-dir DIR`	Screenshot output directory	`tmp/checkset/screenshots`
`--checks-dir DIR`	Directory containing checks	`checks`
`--preps-dir DIR`	Directory containing preps	`preps`
`--playwright-server URL`	WebSocket URL for remote Playwright	local

Output

Terminal

Single target:

Checkset — https://staging.myapp.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

   PASS   UserCanSignIn .................. 1.2s
   PASS   UserCanCheckout ................ 3.4s
   FAIL   AdminCanViewDashboard .......... 2.1s
          ✗ verify "dashboard loads" — timed out waiting for selector
          screenshot: tmp/checkset/screenshots/admin_can_view_dashboard/FAIL_...png
          trace: tmp/checkset/traces/admin_can_view_dashboard_...zip

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
2 passed, 1 failed (6.7s)
Results: tmp/checkset/results/results_20260218_103000.json

Suite mode (suites run concurrently, results printed per suite):

Checkset — app → https://app.afomera.dev
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

   PASS   PageLoads ........................................ 0.8s
   PASS   UserCanSignIn .................................... 1.2s

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
2 passed (2.0s)
Results: tmp/checkset/results/results_app_20260218_103000.json


Checkset — admin → https://admin.afomera.dev
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

   PASS   PageLoads ........................................ 0.7s
   PASS   AdminDashboard ................................... 1.5s

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
2 passed (2.2s)
Results: tmp/checkset/results/results_admin_20260218_103000.json


━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
All suites: 4 checks across 2 suites — 4 passed, 0 failed, 0 skipped (2.5s)

JSON

Every run writes structured results to tmp/checkset/results/. In suite mode, filenames include the suite name (e.g. results_app_20260218_103000.json).

{
  "run_at": "2026-02-18T10:30:00-07:00",
  "target_url": "https://staging.myapp.com",
  "duration": 6.7,
  "summary": { "total": 3, "passed": 2, "failed": 1, "skipped": 0 },
  "checks": [
    {
      "check": "Checks::UserCanCheckout",
      "status": "passed",
      "duration": 3.4,
      "steps": [
        { "name": "cart has items", "type": "verify", "status": "passed", "duration": 0.5, "screenshot_path": "..." }
      ]
    }
  ]
}

Screenshots & Traces

Screenshots are captured on every verify and step (pass or fail)
Full-page failure screenshots are taken when a verify/step fails
Playwright traces (.zip) are saved only on failure — open with npx playwright show-trace trace.zip

Configuration

# checkset.rb or anywhere before running
Checkset.configure do |c|
  c.target_url = "https://staging.myapp.com"
  c.headless = true
  c.browser_type = :chromium
  c.default_timeout = 10_000
  c.viewport_size = { width: 1280, height: 720 }
  c.credentials_provider = :env  # or :rails_credentials
end

Credentials

By default, credentials come from environment variables:

credentials[:admin_email]  # reads ENV["ADMIN_EMAIL"]

With Rails, you can use encrypted credentials:

Checkset.configure { |c| c.credentials_provider = :rails_credentials }
# reads Rails.application.credentials.dig(:checkset, :admin_email)

Try the Examples

The examples/ directory includes sample checks organized into suites. To try them out:

# Run the built-in smoke test against example.com
bundle exec checkset --target https://example.com

# Run example checks against a single target
bundle exec checkset --target https://hey.com --checks-dir examples/checks

# Run all example suites using the example checkset.yml
bundle exec checkset --config examples/checkset.yml

# Run just the "hey" suite from the examples
bundle exec checkset --config examples/checkset.yml --suite hey

# Run in headed mode to watch the browser
bundle exec checkset --config examples/checkset.yml --suite basecamp --headed

# Run all suites with checks running in parallel within each suite
bundle exec checkset --config examples/checkset.yml --parallel 3

# Run a specific check by name
bundle exec checkset --target https://hey.com --checks-dir examples/checks HeyHomepage

# Clean up all output (screenshots, traces, results)
bundle exec checkset --clean

Exit Codes

0 — all checks passed
1 — one or more checks failed

License

MIT

checkset

Runtime