SolidLoop

Infrastructure layer for running autonomous AI agents as long-lived Rails background processes. Built on ActiveJob + PostgreSQL with a Rack-style middleware architecture and Model Context Protocol (MCP) tool support.

Each agent turn (LLM call or tool execution) is a discrete background job. State lives in the database — not in memory. Crash the process, restart it, the agent continues from where it stopped.

Add to your Gemfile:

gem "solid_loop", "~> 0.0.5"

Or track the repository directly, pinned to a release tag:

gem "solid_loop", github: "ruslan/solid_loop", tag: "v0.0.5"

Then run:

bundle install
bin/rails solid_loop:install:migrations
bin/rails db:migrate

Requires Rails >= 7.1, PostgreSQL, faraday, and event_stream_parser.

SolidLoop requires a same-database transactional ActiveJob backend — Solid Queue or GoodJob running in the application database. SolidLoop enqueues each successor job inside the transaction that advances loop state, so state advancement and job dispatch commit atomically: a crash leaves neither a state change without its job nor a job pointing at rows that never committed. (SolidLoop::ApplicationJob sets enqueue_after_transaction_commit = false so the enqueue actually participates in the transaction.)

Non-transactional or external backends (Sidekiq/Resque on Redis, SQS, …) do not get this guarantee: the enqueue escapes the transaction, reintroducing a crash window between state commit and dispatch — and jobs may even run before the enclosing transaction commits. They would need a transactional-outbox layer, which SolidLoop does not provide.

Define an agent class inheriting from SolidLoop::Base:

class MyAgent < SolidLoop::Base
  def system_prompt
    "You are a helpful assistant."
  end

  def llm_provider
    {
      # Origin ONLY — the dialect appends its own path (`/v1/chat/completions`
      # for open_ai). A base_url ending in /v1 would produce /v1/v1/... .
      base_url: "https://api.openai.com",
      api_token: ENV["OPENAI_API_KEY"],
      model: "gpt-4o",         # used in the JSON payload (all dialects)
      # llm_model_name: "...", # used for URL construction in the Gemini dialect
      read_timeout: 120
    }
  end
end

Every tool in SolidLoop is an MCP tool. MCP is the interface; where and how a tool server runs is a transport concern. An agent declares its tool servers via mcps — name: is required in every entry:

class MyAgent < SolidLoop::Base
  def mcps
    [
      # 1. Remote MCP server over Streamable HTTP
      { name: :main,
        url: ENV["MCP_URL"],
        api_token: ENV["MCP_TOKEN"],
        tools: %w[list_files read_file],       # optional whitelist
        required_tools: %w[list_files] },       # optional fail-fast validation

      # 2. In-process toolset — the class below IS the MCP server, no HTTP
      { name: :search, toolset: SearchTools },

      # 3. Child-process MCP server over stdio (spawn-per-call, stateless only)
      { name: :fs,
        transport: SolidLoop::Mcp::StdioTransport.new(
          command: ["npx", "-y", "@modelcontextprotocol/server-filesystem", "/data"],
          timeout: 30
        ) }
    ]
  end
end

A toolset groups the tools of one domain into a logical MCP server that runs in-process, on the caller's thread, with full access to your app:

class SearchTools < SolidLoop::Mcp::Toolset
  server_name "search"

  tool "web_search",
       description: "Search the web and return the top results",
       input_schema: {
         type: "object",
         properties: { query: { type: "string" } },
         required: %w[query]
       } do |args, ctx|
    Search::Web.call(loop: ctx.loop, **args.symbolize_keys)
  end

  tool "fetch_url",
       description: "Fetch a URL and return a text summary",
       input_schema: {
         type: "object",
         properties: { url: { type: "string" } },
         required: %w[url]
       } do |args, ctx|
    Search::Fetch.call(loop: ctx.loop, **args.symbolize_keys).to_json
  end
end

The block receives string-keyed args (exactly as they would arrive over HTTP) and a CallContext (ctx.agent, ctx.loop). A String return value becomes text content; a Hash is also exposed as structuredContent; raising produces an isError tool result the model can self-correct on.

Tool names must be loop-unique across servers — add prefix: "search" to an entry to advertise its tools as search__*; collisions without a prefix fail loudly at session initialization. Details — transports, custom adapters, the conformance suite, session_recovery: — in docs/guides/mcp_transports.md; the decision and the migration guide from 0.0.2 native tools in docs/decisions/mcp-only-tooling.md.

Define a toolset once — the in-process agent consumes it locally, and external MCP clients (Claude Code, other apps) consume the same tools remotely, with shared schemas, wire logs and admin. Mount each toolset you want to expose:

# config/routes.rb
mount SolidLoop::Mcp.server(SearchTools, auth: McpAuth.new) => "/mcp/search"

An unmounted toolset is not exposed. The endpoint is fail-closed: auth: is required and is any object responding to call(token, request) -> principal | nil — return the caller's identity (an API-client record, a user, a symbol) or nil for a 401. Inside tool handlers external calls arrive with ctx.principal set and ctx.agent / ctx.loop nil. Inbound sessions and per-request wire logs appear in the admin UI under Inbound MCP. Details in docs/guides/mcp_transports.md; the design in docs/decisions/mcp-server.md.

Override configure_llm_middlewares or configure_tool_middlewares to inject middleware into the pipeline for a specific agent only:

class MyAgent < SolidLoop::Base
  # Runs before each LlmCompletionJob — insert/remove LLM pipeline layers here.
  def configure_llm_middlewares(builder)
    builder.insert_before(
      SolidLoop::Middlewares::ResponseParsing,
      SolidLoop::Middlewares::ToolCallXmlParser  # enable XML tool-call parsing for this agent
    )
  end

  # Runs before each ToolExecutionJob — insert/remove tool pipeline layers here.
  def configure_tool_middlewares(builder)
    builder.insert_after(SolidLoop::ToolMiddlewares::ToolExecution, MyAuditMiddleware)
  end
end

Global pipeline changes (applying to all agents) go in an initializer instead — see docs/guides/llm_middlewares.md.

Start a loop (creates DB records and enqueues LlmCompletionJob):

loop_record = MyAgent.start!(
  subject: current_user,
  message: "Summarize this week's engineering updates."
)

Pause and resume:

agent = MyAgent.new(loop_record)
agent.pause!
agent.resume!

resume! re-enters init, paused, failed and completed loops, returning false for anything else. Resuming a completed loop is deliberate — continuing a finished conversation is a normal thing to do — but it is the wrong default wherever a loop has already reported a final result, because the resumed turn appends to it. Narrow it with from::

agent.resume!(from: %i[paused failed])   # restart what stalled, leave finished runs alone

Passing a status that can never be resumed (running, queued) raises ArgumentError rather than silently never matching. The full set is SolidLoop::Loop::RESUMABLE_STATUSES.

Loop status lifecycle: init → queued → running → completed / failed / paused.

Some workloads use tools as a typed output format rather than as a conversation. A batch job hands the model twenty skeleton records and it answers with one turn containing twenty atomic tool calls that write them; a labelling pass emits one call per item; a codemod emits one call per file.

When every call in such a turn succeeds, the model has nothing left to say — but the loop schedules another turn anyway, and that turn is a paid request whose entire content is the word "done". Over a run of a few hundred batches that adds up to a few hundred requests bought for nothing.

A loop is normally finished by the model, by returning a turn with no tool calls. An agent that already knows the work is done can finish its own loop instead, with a middleware and no further request:

class CompleteOnFullySuccessfulToolTurn
  def initialize(app)
    @app = app
  end

  def call(env)
    last  = env.loop.messages.where(role: "assistant").order(id: :desc).first
    calls = last&.tool_calls.to_a

    if calls.any? && calls.all?(&:is_success)
      env.loop.transition_status(
        from: :running,
        to: :completed,
        expected_execution_token: env.execution_token,
        execution_token: nil,
        lease_expires_at: nil
      )
      return
    end

    @app.call(env)
  end
end

class BatchSeedAgent < SolidLoop::Base
  def configure_llm_middlewares(builder)
    builder.insert_after(
      SolidLoop::Middlewares::AgentInitialization,
      CompleteOnFullySuccessfulToolTurn
    )
  end
end

Three details are load-bearing:

• Insert after AgentInitialization, before NetworkCalling. The assistant shell message is created in NetworkCalling; a middleware placed after it would strand that shell in processing with nothing left to finish it.
• Return without calling @app.call(env). Returning is what skips the request. The status transition alone would not.
• Keep expected_execution_token:. It is not decoration: without it a worker whose generation was already rotated away could complete a loop a newer generation now owns.

The result is a terminal state indistinguishable from a model-driven completion — same completed status, execution_token and lease_expires_at cleared, blank error_message, no orphaned shell, nothing for the reaper — reached one request sooner. step_count counts only the turns that actually happened. This is covered by spec/integration/host_driven_completion_spec.rb.

Two things worth keeping:

Do not stop the loop by raising instead. Raising LimitExceededError (or setting max_steps 1) also ends the turn, but lands the loop in paused — which resume!(from: %i[paused failed]), the documented "restart what stalled" sweep, is supposed to pick up. Every finished batch would then re-enter and spend the request the recipe existed to save. completed is not in that sweep.

Test is_success, not merely "did every call answer". An MCP error result is persisted with is_success: false and remains an ordinary tool response precisely so the model can see it and correct itself. Ending the loop on "all calls resolved" would cut that short.

SolidLoop ships with a mountable admin UI for inspecting loops, messages, tool calls, MCP sessions/tools, events, and a lightweight dashboard.

Mount the engine in your app routes:

# config/routes.rb
mount SolidLoop::Engine => "/solid_loop"

The admin UI is intentionally unauthenticated inside the gem. Protect the mounted path in the host application the same way you would protect Sidekiq Web or GoodJob admin.

Features include:

• loop/message/tool/event browsing
• loop control actions: observe, freeze, unfreeze, force stop
• live loop message updates with Turbo when observe mode is enabled

The "observe" mode broadcasts new messages over Turbo Streams, so it depends on turbo-rails being present in the host application. turbo-rails is not a hard dependency of the gem — everything else works without it.

If a loop has observe mode enabled but turbo-rails is not available, SolidLoop skips the live broadcast and logs a warning (Rails.logger.warn) instead of raising. To use live observation, add gem "turbo-rails" to your host app.

Each agent controls streaming via the streaming? method. This is a conscious architectural decision, not a default setting.

class MyAgent
  def streaming?
    true  # or false
  end
end

Rule of thumb: if a human is watching a screen, use streaming. If it's a background job, don't.

With a slow CPU/GPU model, streaming is almost always the right call even for background agents:

• read_timeout is counted between chunks, not over the full response. Each token resets the timer, so a 120s timeout is safe even for a 30-minute generation.
• Without streaming, read_timeout must cover the entire generation — potentially hours. Any network hiccup silently kills the job.

The one exception: if the model runs in a completely isolated environment with no network risk and no UI, non-streaming with a generous timeout is simpler.

SolidLoop's built-in streaming parser (SseStreamAggregator) handles OpenAI-compatible SSE format. Anthropic and Gemini use different wire formats — their native streaming will not parse correctly with the anthropic or gemini dialects today.

Solution: use an LLM proxy that presents any provider as an OpenAI-compatible endpoint, then use the open_ai dialect with streaming: true:

# Anthropic Claude via LiteLLM proxy — full streaming support
class MyAgent < SolidLoop::Base
  def llm_dialect_name = :open_ai   # proxy speaks OpenAI SSE

  def streaming? = true

  def llm_provider
    {
      base_url:  "http://localhost:4000",   # LiteLLM proxy
      api_token: ENV["LITELLM_API_KEY"],
      model:     "anthropic/claude-opus-4-5"
    }
  end
end

configure_llm_middlewares / configure_tool_middlewares receive a Pipeline::Builder. Positional edits (insert_before, insert_after, replace) raise Pipeline::Builder::UnknownMiddleware if the named middleware is not in the stack — placement is the point, so a missed target is an error rather than a guess. delete is lenient and no-ops when the target is absent.

def configure_tool_middlewares(builder)
  builder.insert_after(SolidLoop::ToolMiddlewares::ToolExecution, MyMiddleware)
end

Order matters: anything inserted before AgentInitialization runs against an unpopulated context. The engine rebuilds the stacks on every reload, so a host to_prepare block that customizes them re-applies onto a fresh stack rather than accumulating duplicates.

SolidLoop builds a deliberately small payload — model, messages, stream, and tools. Anything else the provider accepts comes from the agent's llm_params, deep-merged over that payload as the last step before the dialect renders it:

class MyAgent < SolidLoop::Base
  def llm_params
    { reasoning_effort: "low", temperature: 0.2, max_tokens: 8192 }
  end
end

Two knobs have first-class hooks because they need per-dialect translation: reasoning_effort and max_tokens.

class MyAgent < SolidLoop::Base
  def reasoning_effort = "low"   # nil (default) sends nothing
end

Accepted levels differ by provider, so SolidLoop passes the value through rather than validating it. Note that llm_params cannot substitute for this hook on the anthropic and gemini dialects: both rebuild the payload in render_payload and drop keys they do not recognize.

max_tokens has its own hook (SolidLoop::Base#max_tokens) and defaults to nil — no cap is sent and the provider applies its own default. Override it to opt in, or set it through llm_params like any other key.

Because the merge happens last, llm_params can also override a key the gem set. Under the open_ai dialect the keys reach the wire verbatim; the anthropic and gemini dialects re-render the payload, so only the keys those dialects read survive.

The native anthropic and gemini dialects are still useful for non-streaming production use (background agents, batch jobs) where you control the provider directly without an extra hop.

Most providers treat reasoning as output-only: they hand back a <think> block or a reasoning_content field, and the next request has to decide what to do with it. Drop it and the model forgets its own plan; send it back the wrong way and you pay for it on every turn from then on.

SolidLoop::Base#reasoning_strategies picks the shape. It defaults to nil, which means "let the dialect choose":

class MyAgent < SolidLoop::Base
  def reasoning_strategies
    [ "reasoning_content" ]        # an explicit field
    # [:xml]                       # weld into content — works anywhere
    # [:xml!]                      # weld even alongside a native strategy
    # [:gemini_signed]             # signed thought part
    # [:anthropic_signed]          # signed thinking block
    # []                           # send no reasoning back at all
  end
end

:xml is not free. It welds every past thought into the assistant message body, so it is billed as prompt tokens on every subsequent turn and compounds over the run. Measured on vLLM's /tokenize against the Qwen3 chat template, the same six-message conversation cost 1438 prompt tokens welded and 108 in reasoning_content — and 108 with the reasoning omitted entirely, because that template drops the field from history exactly as the model's own guidance asks ("historical model output should only include the final output part").

Note what that equality means: the two options are not the same request with a different encoding. Welding puts the old thoughts in front of the model as ordinary text; the field, on this template, does not. So :xml buys re-showing the model its own past reasoning, and on Qwen3 it costs ~13x the tokens for something the model's authors advise against — and a long loop can end up spending a large fraction of its context window on it. Prefer the field where the provider has one; keep :xml where the provider strips it, or where you have measured that the model does better for having its old thoughts back.

The signed strategies need their signature. Anthropic and Gemini both issue one alongside the reasoning and verify it when the reasoning comes back; the dialects capture it into message.metadata["thought_signature"]. A message without one — anything written before 0.0.5, or by a model that returns no signature — sends no reasoning at all rather than a shape the provider would reject. Pair with :xml ([:anthropic_signed, :xml]) if you would rather weld the unsigned ones into the body than lose them: the :xml fallback fires exactly when the signed strategy could not.

Both signed shapes are built from the providers' published documentation and covered by unit tests, but neither has been exercised against a live key — see docs/validation.md. On anthropic or gemini, set reasoning_strategies to [:xml] to keep the pre-0.0.5 behaviour while you evaluate.

Do not trim reasoning turn by turn. It is the obvious idea and it is a trap: a sliding retained window rewrites the prompt in the middle every turn, so the prefix cache diverges at the message that just aged out and everything after it is prefilled again. Measured on llama.cpp (Qwen3.8-27B Q6_K, prefix caching on) over an 8-step agent chain — keeping the newest 2 turns' reasoning cost 1847 ms of prefill per turn against 910 ms for the untrimmed history it was meant to be cheaper than, and 500 ms for sending none. Trimming is not the problem; sliding is. A stable rule (always all, or always none) keeps the prompt monotonically extending and the cache intact.

So there are two honest positions: carry the reasoning and let the cache pay for it, or send none. When a run genuinely outgrows its window, the answer is compaction — summarise the old history once, keep a recent tail, resume extending — not a per-turn trim. codex and opencode both do exactly this, and SolidLoop does not have it yet.

Whatever the strategy, the cost is recorded. Each assistant message's metadata carries a reasoning_carried entry describing what that request's prompt spent on earlier reasoning — messages, chars, inline_chars (the part welded into content) and tokens_est.

read_timeout controls how long to wait for data from the LLM before aborting. It is set per-agent via llm_provider and defaults to 10 minutes.

def llm_provider
  {
    base_url: "...",
    api_token: "...",
    model: "...",
    read_timeout: 300  # seconds
  }
end

How it works: the timeout counts silence between consecutive reads from the socket — not total request duration. During streaming, each incoming token resets the timer. The dangerous window is prefill (before the first token), when the model is silent.

Rule of thumb: with streaming enabled, set the timeout to cover your worst-case prefill time. Without streaming, it must cover the entire generation — for a large local model producing thousands of tokens on CPU, that can be hours.

Every LLM turn is claimed with an execution token (the generation fence) and a lease (lease_expires_at). The lease is derived per claim from the agent's read_timeout plus a margin. When a worker dies mid-turn, the lease expires and a periodic reaper heals the loop.

read_timeout is an inactivity timeout — the silence between two socket reads — not a bound on total request duration (see Read Timeout above). A healthy model can stream for tens of minutes with every inter-chunk gap under read_timeout, and pre-call MCP session init is likewise unbounded by it. So a static read_timeout + margin lease would expire on a live, healthy worker mid-turn and be falsely reclaimed.

To prevent that, a lightweight heartbeat renews the lease for the entire claimed LLM job — MCP init + the whole stream (including a silent prefill) + finalization — on a dedicated DB connection, at ~lease/4 cadence. Each renew is conditional on the loop still being running with the same execution token, so it can never renew a lease it no longer owns; if it can't (the generation rotated away or the reaper reclaimed the turn), the worker raises SolidLoop::LostLease before writing anything and yields the turn. Correctness ultimately rests on the commit-time fence check under the row lock; the heartbeat only keeps a healthy turn from being falsely reclaimed.

Each unresolved tool call carries its own per-tool lease (lease_token / leased_until) so the reaper can reclaim one dead parallel tool without fencing its healthy siblings (parallel tools are the shipped default). Unlike the LLM path there is no tool heartbeat — a tool invocation is a single bounded client call, so the per-claim lease derivation suffices:

• The tool lease is derived as max(default_tool_timeout, slowest resolved HTTP MCP client timeout) + lease_margin. An HTTP MCP client's timeout (mcp_config[:timeout] || 60, operator-settable) is folded in, so a healthy long-running HTTP tool is never reclaimed mid-call.
• Residual — in-process / custom-transport tools: these expose no client timeout to derive from, so default_tool_timeout is their ceiling. An in-process tool that legitimately runs longer than default_tool_timeout MAY be reclaimed → a duplicate invocation. The both-token fence keeps SolidLoop's own record clean — exactly one canonical result is committed regardless — but the tool's external effect is at-least-once: the second invocation's side effect DOES run and will repeat unless the handler atomically deduplicates on the stable idempotency key (solid_loop:tool_call:<id>, delivered in-process via CallContext#idempotency_key and on the wire as MCP _meta). So set default_tool_timeout above your slowest in-process tool to avoid the duplicate entirely, and make irreversible handlers dedupe on that key.

Install the reaper recurrence with whatever scheduler you already run — the gem does not own a scheduler. SolidLoop::ReaperJob is a plain entry point for SolidLoop.reap!:

# good_job (config/application.rb or an initializer)
config.good_job.enable_cron = true
config.good_job.cron = {
  solid_loop_reaper: { cron: "* * * * *", class: "SolidLoop::ReaperJob" }
}

SolidLoop.reap! is idempotent and safe to run concurrently (each case re-checks its condition under a row lock, so two racing reapers resolve to one). It re-enqueues dead LLM turns (expired lease), reclaims stalled tool calls, repairs dropped enqueues, re-enqueues loops stuck queued past a threshold, and sweeps orphaned processing shells off paused/terminal loops. SolidLoop.last_reaped_at exposes the timestamp of the last successful reap as a liveness health signal (process-local — see the known limitations below).

SolidLoop routes its jobs onto three fixed queues. A host that runs a worker with an explicit queue allowlist (rather than "all queues") MUST consume every one of them, or work silently stalls:

The ReaperJob queue name (solid_loop_maintenance) is a permanent, stable contract. For example, a GoodJob worker that allowlists queues must include all three:

# config/environments/production.rb (GoodJob example)
config.good_job.queues = "solid_loop_llm,solid_loop_tool,solid_loop_maintenance"
# (or simply "*" to consume every queue)

Tunables (all optional):

SolidLoop.configure do |c|
  c.config.lease_margin          = 60   # seconds added on top of read_timeout / tool timeout when deriving a lease (must be > 0)
  c.config.queued_reap_threshold = 300  # a loop queued longer than this is re-enqueued (lost queue row)
  c.config.default_read_timeout  = 600  # fallback read_timeout when an agent's llm_provider omits one
  c.config.default_tool_timeout  = 600  # ceiling for an in-process/custom tool's lease (raise above your slowest such tool)
  c.config.lease_leak_grace      = 300  # grace (s) added on top of an agent's max_duration to form the renewer's leak ceiling — a leak-safety backstop; must be > 0
  c.config.default_max_duration  = 7200 # fallback max_duration (s) sizing the leak ceiling when a registration omits one (mirrors SolidLoop::Base#max_duration)

  # Transient LLM failures (see below)
  c.config.llm_retries           = 2    # re-sends on top of the first attempt; 0 disables
  c.config.llm_retry_base_delay  = 1.0  # backoff base (s); delay for attempt N is base * 2**(N-1)
  c.config.llm_retry_max_delay   = 60.0 # ceiling on any single wait, `Retry-After` included
  c.config.llm_retry_statuses    = [408, 409, 429, 500, 502, 503, 504]

  # Retention (see below) — nil means keep forever, which is the default
  c.config.event_retention               = nil # e.g. 90.days
  c.config.mcp_inbound_session_retention = nil # e.g. 30.days
  c.config.janitor_batch_size            = 1000
end

A 429 or 5xx from a hosted provider is routine, and providers often say exactly how long to wait. The adapter re-sends such a response up to llm_retries times before the turn is allowed to fail, honouring the response's Retry-After (both delta-seconds and HTTP-date forms) over its own exponential backoff, capped by llm_retry_max_delay. Deterministic statuses — 400/401/403/404/422 — are never retried; repeating them only burns quota.

Two boundaries are deliberate:

• A read timeout is not retried. By the time it fires the full read_timeout has already elapsed and the request may well have been served; repeating it would multiply the turn's wall clock. Only a response with a retryable status, or a connection that never established (Faraday::ConnectionFailed), is re-sent.
• A stream that has already emitted is not retried. Once a chunk reaches the caller the assistant message shell holds partial content, and re-sending would concatenate two generations. The latch is set at the point content is handed over, not merely when bytes arrive — so a streamed error body stays retryable.

Backoff waits are cancellation-aware: a pause or stop issued during the wait is honoured within a second rather than after the full delay. Retries are recorded in the turn's wire log; ttft and duration describe the attempt that produced the answer.

solid_loop_events (one row per LLM turn, tool call and inbound MCP request) and solid_loop_mcp_inbound_sessions grow without bound. Retention is off by default — silently deleting a host's audit trail on upgrade would be the wrong default — and is opted into per table:

SolidLoop.configure do |c|
  c.config.event_retention               = 90.days
  c.config.mcp_inbound_session_retention = 30.days
end

Install the recurrence like the reaper's; unlike the reaper this is a daily concern, not a per-minute one:

config.good_job.cron = {
  solid_loop_reaper:  { cron: "* * * * *",  class: "SolidLoop::ReaperJob" },
  solid_loop_janitor: { cron: "17 4 * * *", class: "SolidLoop::JanitorJob" }
}

SolidLoop.prune! deletes in batches of janitor_batch_size so a sweep never takes one long lock on a table being written to concurrently. It is idempotent and safe to run concurrently. Sessions are pruned by creation age rather than last use, so a client that never terminates and never returns cannot pin its row forever; a doomed session's events are removed with it even when event_retention is unset, so no event is left pointing at a deleted session. SolidLoop.last_pruned_at is the liveness signal, mirroring last_reaped_at (process-local).

The first sweep after enabling retention on an existing install may have years of backlog to clear and can run long — either let the recurring job chip away at it, or do the initial bulk delete out of band.

The invariant lease_margin > 0 guarantees lease > read_timeout (LLM) and lease > tool timeout (tools) per claim; the heartbeat then keeps a healthy LLM turn's lease alive for its full duration. The default_read_timeout knob is the single coherent source for BOTH the derived LLM lease and the HTTP client's fallback read timeout, so the two can never diverge.

The heartbeat is a single process-wide background service (SolidLoop::LeaseRenewer), not one thread per turn. Each in-flight LlmCompletionJob registers its (loop_id, execution_token, lease_duration) on start and deregisters when the turn ends; the renewer batch-renews every registered lease each tick from one long-lived DB connection. This keeps renewal's connection demand O(1) — a single checkout for all in-flight turns — regardless of worker concurrency. (A per-turn renewer thread each checking out its own connection would livelock at full pool occupancy: every heartbeat would block waiting for a connection and treat the exhaustion as a lost lease.)

Because the renewer holds one connection for the process lifetime, size your DB pool as worker_concurrency + 1 so a full set of busy workers plus the renewer never contends. The renewer draws from the same pool as the workers (so it shares their exact view of the data).

Leak safety does not depend on the job's ensure ever running: each registration carries a hard ceiling tied to the owning agent's own legal turn bound — max_duration + lease_leak_grace (default 2h + 5min). Once a registration exceeds it — only possible if a turn somehow never deregistered — the renewer drops it and stops renewing, so its lease lapses and the reaper reclaims the loop. Crucially, a legitimate long turn is never dropped: the agent already caps a legal turn at max_duration (its own contract), so any turn within that bound finishes before the ceiling. (Tying the ceiling to lease_duration × K instead would be wrong — a healthy turn stays registered for its whole life, so any legitimate stream longer than K lease-widths, e.g. a small read_timeout with steady chunks, would be dropped and reaper-reclaimed mid-flight, and no long turn could ever complete.)

The durability guarantees above are the shipped contract; the following are accepted, documented boundaries for the 0.0.x line (recovery is at-least-once, and canonical DB effects are deduped by the executed_at guard + stable ToolCall key):

• (a) Continuation failure after a tool checkpoint surfaces late. A continuation failure immediately after a tool checkpoint may surface only after the reaper redelivers the checkpointed call — the tool body is not repeated; only the dropped continuation is retried.
• (b) Slow in-process/custom tools may duplicate their external effect. An in-process or custom-transport tool that runs longer than default_tool_timeout may be reclaimed and re-invoked, so its external side effect can run more than once (the canonical DB result is still exactly one). Mitigated by the stable solid_loop:tool_call:<id> idempotency key; raise default_tool_timeout above your slowest such tool.
• (c) Tool-lease derivation bounds configured HTTP MCP calls only. Cumulative setup/recovery time and opaque custom transports can exceed the derived tool lease and be redelivered under the at-least-once contract.
• (d) Reaper case-2 continuation can enqueue duplicates. A dropped-continuation re-enqueue may deliver a duplicate ToolExecutionJob; canonical effects are deduped (idempotent response_message check), so it is harmless.
• (e) SolidLoop.last_reaped_at is process-local, not fleet-wide. It reflects only the reap runs of the current process. For fleet liveness, alert on your scheduler/backend metrics (e.g. GoodJob cron execution), not this value.
• (f) Mixed old/new workers across the lease upgrade are unsupported. Drain jobs or deploy atomically — do not run pre-lease and post-lease workers against the same database simultaneously.
• (g) Lease timestamps are Rails UTC :datetime. Do not reinterpret lease_expires_at / leased_until as local time.
• (h) Reaper scans/indexes are adequate for 0.0.x, not scale-tuned. The partial indexes and per-loop scans are correct and cheap at current scale; large-fleet tuning is an additive later change.
• (i) Middleware that REPLACES a canonical persistence finalizer forfeits the guarantees. Custom middleware that replaces (rather than wraps) a canonical finalizer (ResponseParsing / ToolExecution / ResponseCreation) must reproduce the heartbeat / both-token / row-lock fences itself, or it loses the durability guarantees.
• (j) A hand-rolled paused→running resume MUST revoke the gated tool_call's lease before re-enqueuing. If your host implements its own resume that re-enqueues a ToolExecutionJob for an unexecuted tool_call, it must FIRST NULL that call's lease_token/leased_until (as well as rotate a fresh execution_token) — otherwise the re-enqueued job's claim CAS matches zero rows and no-ops silently, stranding the loop until the stale lease's TTL expires and the reaper reclaims it. SolidLoop::Base#resume! does exactly this (revoke-then-enqueue in one locked txn) and is the pattern to mirror. The claim no-op now logs a ToolExecutionJob: tool lease claim lost … warning so the strand is diagnosable.

Note (streaming UI): mid-stream partials are stored on a hidden message row until the turn commits, so a reclaimed/cancelled/paused partial never re-enters history. pause / force-stop / freeze (and a reaper backstop) mark any orphaned processing shell failed+hidden. To render live streaming in a host view, include hidden processing assistant shells in your live query (they are still broadcast).

The test suite uses RSpec, WebMock, and FactoryBot against a PostgreSQL dummy app. Tests run against Rails 7.1, 7.2, and 8.1 via appraisals.

bundle install
bundle exec appraisal install

Create the test database (uses DB_HOST, DB_USERNAME, DB_PASSWORD env vars, defaulting to localhost / postgres / ""):

bundle exec rails db:create db:migrate RAILS_ENV=test

# Against the default Gemfile (Rails 8.1)
bundle exec rspec

# Against a specific Rails version
bundle exec appraisal rails-7-1 rspec
bundle exec appraisal rails-7-2 rspec
bundle exec appraisal rails-8-1 rspec

# All versions
bundle exec appraisal rspec

StreamingLlmEmulator — stubs the LLM HTTP endpoint (OpenAI-compatible). Matches requests by the last message's content and returns pre-configured streaming (SSE) or non-streaming responses. Records all requests for assertion.

let(:emulator) { StreamingLlmEmulator.new("http://llm.local", "my-token") }

before { emulator.set_answer(/hello/i, { role: "assistant", content: "Hi!" }) }

McpEmulator — stubs an MCP server. Handles initialize / tools/list / tools/call JSON-RPC methods with session validation.

let(:mcp) { McpEmulator.new("http://mcp.local") }

before { mcp.set_tool_result("my_tool", "tool output") }

TestAgent — a minimal SolidLoop::Base subclass defined in the dummy app. Configure it via loop_record.state:

loop_record = TestAgent.start!(
  subject: nil,
  message: "Hello",
  base_url:  "http://llm.local",
  api_token: "my-token",
  model:     "test-model",
  streaming: false
)

The suite stubs every provider. For the other half of the picture — the models, inference servers and hardware SolidLoop has been exercised on before a release, and the paths that are still untested against a live key — see docs/validation.md.

The gem is available as open source under the terms of the MIT License.