Cogworker
Redis-backed background job processing for Ruby. It includes a worker DSL, retries with backoff, delayed and periodic (cron) jobs, unique jobs, job status and run history, a multi-process supervisor with phased restarts, a Web UI, and a Prometheus exporter.
- Ruby ≥ 3.1, Redis
- The Web UI works fully offline. htmx, its stylesheet, fonts, icons, AG Grid and Chart.js all ship inside the gem, with no CDN.
- MIT license
Installation
# Gemfile
gem 'cogworker'bundle installRequires a standalone Redis (or a primary/replica setup) — Redis Cluster isn't supported: several Lua scripts touch keys they don't declare up front.
Quick start
1. Define a job
class GreetingJob
include Cogworker::Worker # or Cogworker::Job, the same module
def perform(name)
Cogworker.logger.info { "Hello, #{name}!" }
end
endArguments are serialized as JSON, so pass only simple values: strings,
numbers, true/false/nil, arrays and hashes. Hash keys arrive in
perform as strings, so don't use keyword arguments (**opts) in
perform.
2. Enqueue it
jid = GreetingJob.perform_async('world') # now
GreetingJob.perform_in(300, 'world') # in 5 minutes
GreetingJob.perform_at(Time.now + 3600, 'world') # at a specific time
# Low-level, without a job class method:
Cogworker::Client.push('class' => 'GreetingJob', 'args' => ['world'], 'queue' => 'default')3. Write an init file
# config/cogworker.rb
require 'cogworker'
require_relative '../app/jobs/greeting_job'
Cogworker.configure_server do |config|
config.redis = { url: ENV.fetch('REDIS_URL', 'redis://localhost:6379/0') }
end
Cogworker.configure_client do |config|
config.redis = { url: ENV.fetch('REDIS_URL', 'redis://localhost:6379/0') }
endThe configure_server block runs only in a worker process (one started via
exe/cogworker/exe/cogworkerswarm). The configure_client block runs only
in every other process (web app, console). This lets one file be loaded in
both places.
Redis options are url:, or host:/port:/password:/db:. On the
server, the connection pool size is concurrency + 5.
Load your job classes up front, in the init file, rather than leaving them to
an autoloader. A worker resolves a job's class on whichever of its threads
picks the job up, so with an autoloader that isn't thread-safe, two threads
can race to load the same class and one of them sees it half-defined
(undefined method 'perform'). With Rails, keep config.eager_load = true
in the environment the worker runs in; with Zeitwerk on its own, call
loader.eager_load after loader.setup. Classes named in
config.periodic are resolved on the main thread at boot either way, and one
that can't be found is logged as a warning.
4. Start a worker
bundle exec cogworker -r ./config/cogworker.rb -C ./config/cogworker.ymlProcess configuration
CLI flags
| Flag | Purpose |
|---|---|
-r, --require PATH |
the app's init file (or a directory, in which case PATH/config/environment is required) |
-C, --config PATH |
YAML config file (see below) |
-c, --concurrency N |
number of worker threads (default 10) |
-q, --queue NAME[,WEIGHT] |
a queue with an optional weight; repeatable; overrides :queues: from the config file |
-L, --logfile PATH |
write the log to a file |
-e, --environment ENV |
environment (default APP_ENV/RAILS_ENV/RACK_ENV/development); exported as APP_ENV, RAILS_ENV and RACK_ENV before the config file and app code are loaded |
YAML config
The file is rendered through ERB before it is parsed:
---
:concurrency: <%= ENV["CONCURRENCY"] || 5 %>
:queues:
- default
- default # repeating a queue sets its weight: default is polled twice as often as low
- low
:fetch: reliable # or basic — see "Fetch modes" belowCLI flags take precedence over the file.
Signals
| Signal | Effect |
|---|---|
TSTP |
quiet: stop fetching new jobs; jobs already running are finished |
TERM / INT
|
graceful stop: wait for running jobs (up to 25 s), put any still unfinished back on their queues, then exit |
Fetch modes
config.fetch (or :fetch: in the YAML file) picks how workers take jobs
off their queues:
-
:reliable(default, needs Redis >= 6.2): a job is moved atomically onto the process's own in-progress list and stays there until it has finished. If the process dies mid-job (OOM,SIGKILL, a lost host), the job is put back on its queue by another live process once the dead one has goneconfig.orphan_thresholdwithout a heartbeat (default 5 minutes, then up to a minute until the next check). The threshold is deliberately well above the heartbeat's own 60 s expiry: a process that is alive but couldn't reach Redis for a while (an outage, a failover) mustn't have its running jobs started a second time elsewhere. Lower it if crashed processes' jobs must come back sooner; raise it if Redis blips longer than that are expected. A worker with a single queue waits on Redis for jobs (BLMOVE), picking a new one up immediately. With several (weighted) queues they are polled instead: while they stay empty, each worker thread waits 0.25 s, then 0.5 s, … up toconfig.fetch_idle_max_interval(default 1 s) between polls, and goes back to 0.25 s as soon as it finds a job. That cap is the longest an idle worker takes to notice a new job; lower it (e.g.0.25) if that matters more than how often idle workers poll Redis. -
:basic: a single blockingBRPOPacross all queues, as in earlier versions. A job popped by a process that then dies is lost.
On Redis older than 6.2, :reliable falls back to :basic with a warning —
checked when workers start, and again on the first fetch if the server turns
out not to support it after all. A worker doesn't start taking jobs until it
has registered itself in Redis, so with Redis unreachable at boot it waits
(and still stops on TERM).
Either way delivery is at least once: a job that was partly done when its
process died or was stopped runs again from the start, so make jobs safe to
repeat. While the worker process stays up, a job is not re-run because of a
Redis error after it ran: once it has run it is acknowledged (and, if it
failed, recorded in Retry/Dead). If recording the failure itself fails, the
job goes to Retry after a short delay — or straight to Dead if it had no
retries left (retry: false/0, or its last one used up), and to Dead too
after 3 such interruptions. If even that, or the acknowledgement, can't be
written, the process remembers the job and keeps trying to settle it,
without running it again. That memory doesn't survive the process: if it
dies (or is stopped) while Redis is still failing, such a job can run once
more after recovery.
A job whose process dies while running it more than
config.max_orphanings times (default 3; one that runs the process out of
memory, say) goes to Dead with Cogworker::ProcessDied instead of being
recovered again — whatever its retry setting. A restart or deploy stopping
it cleanly doesn't count towards that, and the count resets once the job
gets to finish. It can't single out the culprit, though: every job running
in a process that crashed is counted, so a job that just happened to share a
process with a memory hog several times can end up in Dead too — raise the
limit if that's plausible for you.
With redis-rb 4.x, a connection that drops in the middle of a blocking fetch is retried by the client itself; if Redis had already handed out a job on the dropped connection, that job waits on the process's in-progress list until it's noticed (about two minutes) rather than running at once. It isn't lost.
Damaged keys
Every worker checks this gem's Redis keys once a minute (and at boot). A
key holding the wrong type of value — which only a manual write or data
corruption produces, but which breaks every command on it — is renamed to
cogworker:quarantine:<key>:<time>:<random> (kept 30 days, at most the 5
newest copies per key) and logged as an error,
and everything carries on with a fresh key. In between checks, work around
such a key keeps going: a job that can't be put back on a broken queue
waits behind the others rather than blocking them, a retry/schedule entry
that can't be moved is postponed a minute, and a failed job that can't be
filed when its process shuts down is handed over to
cogworker:unsettled for another process to file — not run again. (One
corner isn't covered: a job that hit config.max_orphanings while the list
used to park such jobs was itself damaged goes into Dead raw first and is
completed with its error right after — retried until that succeeds, but
only by the process that put it there: if that process dies first, the raw
entry stays as it is.)
A lock release (until_executed, unique jobs) that fails because Redis is
briefly away when a job finishes is retried on every heartbeat (5 s) until
it goes through, rather than leaving the lock to expire on its own — kept in
the process's memory, so if the process stops first, the lock expires on its
TTL instead.
Multiple processes (swarm)
COGWORKER_COUNT=4 PHASED_RESTART=true \
bundle exec cogworkerswarm -r ./config/cogworker.rb -C ./config/cogworker.ymlcogworkerswarm accepts the same flags as cogworker. It forks
COGWORKER_COUNT child processes and restarts any that die. A child that
keeps crashing within 30 s of starting is restarted with an exponential
backoff (1 s, 2 s, 4 s, … up to 60 s) instead of in a tight loop. With
PHASED_RESTART=true, sending USR2 to the parent restarts the children one
at a time, so processing never stops. TERM/INT are forwarded to all
children.
Job options
class ReportJob
include Cogworker::Worker
cogworker_options queue: 'low', retry: 5
end| Option | Meaning |
|---|---|
queue: |
queue name (default default) |
retry: |
true or unset: 25 attempts; an integer: that many retries; false or 0: straight to Dead |
unique: :until_executed |
don't enqueue a duplicate (same class + queue + args) until the previous one has finished |
Subclasses inherit options. Any other keys (for example
lock_run: :while_executing) are copied unchanged into the job hash, where
your own middleware can read them.
Job arguments
Arguments are stored as JSON and are neither validated nor coerced on the
way in, so perform receives whatever survives a JSON round trip: Symbols
come back as Strings, Hash keys as Strings, and objects like Time as their
to_s. Pass plain JSON types (String, Integer, Float, true/false/nil,
Array, Hash with String keys) to get back exactly what you pushed.
Cogworker.strict_args! is accepted but does nothing: since there's no
coercion, there's nothing to make stricter.
Testing.fake!/inline! skip that round trip, so a test can pass while a
real worker would receive Strings instead of Symbols.
Retries and Dead
A failed job moves to the Retry set with a growing delay
(count**4 + 15 + jitter seconds, count being the retry about to happen,
1 for the first). retry: sets the budget: true or unset gives 25
retries, an integer that many, false or 0 none. When it runs out of
retries, it moves to Dead. You can put it back on its queue by hand from the
Web UI.
Custom delay: cogworker_retry_in
class SendEventJob
include Cogworker::Worker
cogworker_options retry: 10
cogworker_retry_in do |count, exception, job|
next :kill if exception.is_a?(PermanentError)
60 * (2**count)
end
endcount is the number of retries made before this failure (0 on the first
one), exception what the attempt raised, job a copy of the job hash (the
block may take fewer parameters). It returns:
| Value | Effect |
|---|---|
| Integer / Float ≥ 0 | retry after that many seconds, no jitter added; 0 means on the next poll |
nil |
the default delay |
:kill |
straight to Dead, retries left or not; the death hooks run |
:discard |
dropped: neither Retry nor Dead, no death hooks; counted as failed, its unique locks released |
| anything else | the default delay, with a warning in the log |
If the block raises, the default delay is used and the error is logged. It
isn't called for a failure that is the last one anyway (no retry left, or
retry: false), nor for a job whose class can't be loaded. It's asked as
soon as perform raises, before any server middleware sees the error — so
it also runs when a middleware then swallows that error; keep it free of
side effects. A delay is never capped (one over a year is logged as a
likely mistake), and the job's until_executed lock is extended by it, so
the lock never runs out before the retry — however long that is. If
recording the failure itself fails (Redis trouble), the job is retried
after a fixed 30 s × n instead, whatever the hook said. The hook is
inherited, so a base class of your own (class ApplicationJob) is the place
for an app-wide policy. The Web UI's retry timeline shows :kill and
:discard as such.
Death hooks: cogworker_retries_exhausted and death_handlers
class SendEventJob
include Cogworker::Worker
cogworker_retries_exhausted do |job, exception|
Metrics.increment('events.lost')
end
end
Cogworker.configure_server do |config|
config.death_handlers << ->(job, exception) { ErrorTracker.notify(exception, job) }
endWhen a job lands in Dead for good, its class's cogworker_retries_exhausted
runs first, then every death_handlers entry in order. job is the entry
as stored in Dead (with error_class, error_message, retry_count,
failed_at); each hook gets its own copy, so changing it changes nothing.
| How the job died | Class hook | death_handlers |
|---|---|---|
out of retries, or retry: false / 0
|
yes | yes |
cogworker_retry_in returned :kill
|
yes | yes |
its process kept dying while running it (config.max_orphanings) |
yes, with a Cogworker::JobOrphanedError
|
yes |
| its failure was filed later, by a reconcile pass | yes, with a Cogworker::JobFailedError carrying the original message (the original class is in job['error_class']) |
yes |
| its class can't be loaded (renamed, removed) | no | yes |
cogworker_retry_in returned :discard
|
no | no |
| an unparseable payload | no | no |
| deleted or retried from the Web UI | no | no |
The hooks run only after the job is safely in Dead, at most once per death: on the reconcile paths, only the process that actually filed it runs them. They run synchronously, on the processor thread (once the job is acknowledged, before the thread takes the next one) or on the reconcile thread, with no timeout of their own — keep them quick. One that raises is logged and doesn't stop the others, nor the job's ack or lock release.
A hook can be skipped or cut short, never repeated: a process that crashes
right after the write never runs it, and on shutdown a hook still running
when the drain timeout (25 s) is up may be cut off as the process exits.
The job itself is safe either way — already in Dead and acknowledged, so it
won't run again. Testing.inline! raises the job's error to the caller as before
and runs no hooks; test a hook directly through
SomeJob.cogworker_retry_in_block / SomeJob.cogworker_retries_exhausted_block.
Unique jobs
With unique: :until_executed, calling perform_async again with the same
arguments returns nil instead of a jid while the original job is still
enqueued or running. The lock is released when the job succeeds or fails for
the last time. In case a process crashes, the lock also has a safety TTL,
config.unique_lock_ttl (default 24 h), counted from when the job is due (so
a perform_in further out than that keeps its lock) and extended by the
backoff each time an attempt fails and will be retried.
Middleware
There are two chains: the client chain runs when a job is enqueued, and the
server chain runs around perform.
class TimingMiddleware
# server: call(worker, job, queue)
def call(_worker, job, queue)
start = Time.now
yield
Cogworker.logger.info { "#{job['class']} on #{queue}: #{Time.now - start}s" }
end
end
class TagMiddleware
# client: call(worker_class, job, queue, redis_pool)
def call(_worker_class, job, _queue, _redis_pool)
job['tags'] = ['api']
yield
end
end
Cogworker.configure_server do |config|
config.server_middleware { |chain| chain.add(TimingMiddleware) }
end
Cogworker.configure_client do |config|
config.client_middleware { |chain| chain.add(TagMiddleware) }
endAnything a server middleware raises counts as the job failing — including
an error from its own bookkeeping after yield (writing to Redis during a
failover, say), which would retry a job that actually ran fine. The
built-in ones make every write around yield best-effort; do the same in
yours: rescue (and log) errors from your own writes, and only let the job's
own exception through.
chain.add(Klass, *args) passes args to the constructor. A new middleware
instance is created for every call, so keep any state shared between jobs at
the class level.
Periodic jobs (cron)
Cogworker.configure_server do |config|
config.periodic do |mgr|
mgr.register '0 * * * *', 'HourlyCleanupJob'
mgr.register '*/5 * * * *', 'DailyReportJob', retry: 0, unique: :until_executed,
args: [{ section: 'digest' }]
end
endEach schedule slot runs exactly once, no matter how many processes are
running. (Two narrow exceptions: a push that times out after Redis already
applied it can't be told apart from one that never arrived, so that slot may
run twice; and if a push fails while Redis is unreachable, undoing that
slot's claim is retried from memory — lost if the process restarts first,
in which case the slot is skipped and, for an until_executed entry, the
entry stays locked until that lock's TTL, config.unique_lock_ttl.) An
entry that keeps failing (its push, say) is retried with a growing pause,
5 s up to 5 minutes, instead of on every 5 s tick — except while Redis
itself is unreachable or failing over, which is retried every 5 s and
resumes as soon as Redis is back. (Redis refusing writes for lack of memory,
OOM, counts as the entry failing: it backs off like any other error.)
Right after Redis comes back from an outage longer than a running
until_executed job's lock, there's a window of up to 5 s before its
process re-takes that lock, in which the next slot could start alongside
it. Conversely, a slot found blocked by the previous run's lock is retried
for 15 s before it counts as skipped — long enough for a release that the
outage delayed to go through. Processes claim a slot
atomically in Redis, with no leader. With
unique: :until_executed, a new slot is skipped while the previous run is
still in progress. If the process running it dies (OOM, SIGKILL), the
interrupted run is recovered and put back on its queue — still holding the
entry's lock — once config.orphan_threshold has passed (5 minutes by
default); the lock itself lapses on its own config.orphan_threshold plus
about two minutes after the process's last heartbeat (7 minutes by default).
On first start against an empty Redis, each entry's most recent due slot
fires right away. To turn that off, set config.periodic_catch_up = false.
Entries are only ever added to the published schedule (what the Schedules
tab shows), so one removed from your code stays listed there. Pass
replace: true to clear what was registered before and replace the
published schedule with exactly this set on every boot:
config.periodic(replace: true, &PERIODIC_JOBS)Don't use it if different processes register different schedules against the same Redis — each would erase the others' entries from the tab.
Job status
Cogworker.configure_server do |config|
Cogworker::Status.configure_server_middleware(config, expiration: 1800)
end
Cogworker.configure_client do |config|
Cogworker::Status.configure_client_middleware(config, expiration: 1800)
end
class ImportJob
include Cogworker::Worker
include Cogworker::Status::Worker
def perform(file)
at(50, 'half way') # progress percentage + message
store('rows' => 1000) # arbitrary fields
end
end
Cogworker::Status.status(jid) # => :queued | :working | :retrying | :complete | :failed (nil if unknown)
Cogworker::Status.get(jid) # => the full hash: status, pct, message, ...Status is stored in Redis with a TTL of expiration seconds.
Run history
Status holds only the current state. History records every finished run: its arguments, its duration and, on failure, the error class, message and backtrace. You can browse it on the Web UI's History tab.
Cogworker.configure_server do |config|
Cogworker::History.configure_server_middleware(
config,
retention_days: 30, # the main trim: by age
max_entries: 50_000, # safety ceiling by count
daily_stats_retention_days: 400 # counters behind the "Runs per day" chart
)
endWeb UI
# config.ru
require_relative 'config/cogworker'
Cogworker::Web.time_format = '%d.%m.%Y %H:%M:%S' # timestamp format (strftime)
Cogworker::Web.live_update_interval = 5 # auto-refresh interval, seconds
Cogworker::Web.history_per_page = 50 # rows per History page
map '/cogworker' do
run Cogworker::Web
endTabs:
- Overview: counters, latency per queue, the queue list (browse, delete, pause, retry all), a 24-hour throughput chart, a "Runs per day" chart, and Redis info. A second "Queue first" layout shows one queue in detail.
- Jobs: every job in one table (enqueued, running, scheduled, retrying, dead), with a status filter, search, and a detail panel (args, attempt history, last error, retry/reschedule/delete).
- Schedules: periodic jobs with their last and next run, plus "Run now" and "Disable"/"Enable" buttons.
- Workers: live processes (queues, load, memory, heartbeat, quiet/resume/stop) and the jobs running right now.
- History: the run log, with server-side sorting, filtering and paging, and backtraces for failed runs.
The header also has these controls:
- Live toggles auto-refresh on every tab at once.
- Pause intake / Resume intake quiet or resume every worker process at once.
- A width toggle switches between fixed-width and full-width layout.
Timestamps are shown in the browser's time zone. Every button also works without JavaScript.
Authentication and custom middleware
Cogworker::Web.use(MyAuthMiddleware) # wraps the whole Web UI appUnsafe methods (POST etc.) are accepted only with
Sec-Fetch-Site: same-origin. To allow an exception (for example, an SSO
callback), override Cogworker::Web.safe_request?(env).
The Web UI's session cookie (used only by middleware you add with .use) is
skipped automatically when the app mounting the Web UI already has a session
(the middleware then reads and writes the app's own session), and can be
turned off with Cogworker::Web.builtin_session = false. It is signed with a
random per-process secret by default. When running several
web processes (e.g. Puma workers), set the same secret for all of them —
Cogworker::Web.session_secret = '...' (at least 64 characters) or the
COGWORKER_SESSION_SECRET environment variable.
Extensions
module MyTab
def self.registered(app)
app.get('/my_tab') { ... }
end
end
Cogworker::Web.register(MyTab, tab: 'My tab', index: 'my_tab')Prometheus
The Web UI serves GET <mount>/metrics in Prometheus text format. The
metrics are:
cogworker_processed_totalcogworker_failed_totalcogworker_retry_sizecogworker_scheduled_sizecogworker_dead_sizecogworker_queue_size{queue=…}cogworker_queue_latency_seconds{queue=…}cogworker_busy_workers
To turn the endpoint off, set Cogworker::Web.prometheus_exporter_enabled = false.
Introspection API
stats = Cogworker::Stats.new
stats.processed; stats.failed; stats.enqueued
stats.retry_size; stats.scheduled_size; stats.dead_size
q = Cogworker::Queue.new('default')
q.size; q.latency # latency: age of the oldest job, in seconds
q.each { |job| job.klass; job.args; job.jid }
q.pause!; q.resume!; q.clear
Cogworker::ProcessSet.new.each do |process|
process['identity']; process['busy']
process.quiet! # remotely: stop fetching new jobs
process.resume! # remotely: start fetching again
process.stop! # remotely: finish running jobs and exit
end
Cogworker::WorkSet.new.each { |identity, thread_id, work| work.job } # jobs running right nowTesting
require 'cogworker'
Cogworker::Testing.fake! # jobs are collected in memory; Redis isn't touched
GreetingJob.perform_async('bob')
expect(GreetingJob.jobs.size).to eq(1)
expect(GreetingJob.jobs.first['args']).to eq(['bob'])
GreetingJob.perform_one # run the oldest recorded job (raises Testing::EmptyQueueError if none)
GreetingJob.drain # run them all, including jobs they enqueue
Cogworker::Testing.drain_all # the same, for every job class
GreetingJob.clear # or Cogworker::Testing.clear_jobs!
Cogworker::Testing.inline! do
GreetingJob.perform_async('bob') # runs synchronously through the server middleware chain
end
Cogworker::Testing.disable! # back to real Redis (the default mode)All three methods accept a block: the mode applies only inside the block,
then the previous mode comes back. In inline! mode, an exception from
perform is raised to the caller, and the perform_in delay is ignored.
Example app
examples/ contains a working app: jobs, an init file, cogworker.yml,
and a config.ru that mounts the Web UI. See
examples/README.md for how to run it.
Upgrading
-
From 0.3.0, with a rolling upgrade: until every process runs a newer
version, 0.3.0 processes still treat another process as dead as soon as
its 60 s heartbeat key expires (newer ones wait for
config.orphan_threshold), so a live process that couldn't reach Redis for a minute can have its running jobs started a second time. Upgrade the whole fleet together, or accept that window. -
until_executedperiodic jobs already enqueued by 0.3.0 don't carry the marker that lets a recovery re-take their running lock: if such a job is recovered after a crash, the entry's next slot can run alongside it. -
Status.status(jid)returns a Symbol since 0.3.0 (it was a String).
Development
bundle install
bundle exec rspec # needs a local Redis at redis://localhost:6379/15 (the db is flushed!)
bundle exec rubocop # must pass — CI runs it too.rubocop.yml sets the limits for new code; files that already exceed
them are listed in .rubocop_todo.yml. Fix and remove entries there rather
than adding new ones.
To use a different test Redis, set COGWORKER_TEST_REDIS_URL. The browser
tests (spec/cogworker/web_system_spec.rb) need Chrome or Chromium
installed.
CI runs the suite on Ruby 3.1–4.0 against both redis-rb 4.8 and 5.x, via
gemfiles/redis_4.gemfile / gemfiles/redis_5.gemfile. To reproduce one
combination locally:
BUNDLE_GEMFILE=gemfiles/redis_4.gemfile bundle install
BUNDLE_GEMFILE=gemfiles/redis_4.gemfile bundle exec rspecCOGWORKER_RELOAD=true hot-reloads the code (via Zeitwerk) on every Web UI
request. Use it only in development, and only for the web process:
COGWORKER_RELOAD=true bundle exec rackup ./examples/config.ru -p 9394License
MIT, see LICENSE.txt.