ActiveHarness

⚠️ Work in progress. The API is under active development and may change between versions without notice.

📛 Naming note. As of v0.3.0, ActiveHarness::Agent was renamed to ActiveHarness::Request — it's a single resilient LLM call (model chain, fallback, retry, hooks), not an autonomous entity, and Request describes that more accurately. The Agent name now belongs to a separate abstraction — a loop that uses a model to decide and take actions through safe Tools (see docs/AGENT.md and docs/TOOLS.md); it is a first iteration.

Running a single LLM call is easy. Running a reliable, observable, cost-controlled AI system is not.

ActiveHarness is a Ruby framework for building production-grade LLM pipelines — with deep observability, consensus-based decisions, automatic fallbacks, and real-time cost and timing control. Made for Rails, works in plain Ruby too.

ActiveHarness gem gives you the scaffolding to build multi-step pipelines where every request is under full control: its inputs are directed, its outputs are observed, its errors are retried, and its cost is tracked. You define the logic; ActiveHarness handles the infrastructure.

A harness in software is scaffolding that keeps a component under control — directing its inputs, observing its outputs, and enforcing rules around it. ActiveHarness does exactly that for LLM requests.

Build multi-step, trackable, cost-effective, and reliable AI flows with a clean, Rails-native DSL.

Group related steps into reusable sub-pipelines, and compose complex workflows from smaller ones. Each pipeline is just another step, with its own stop conditions, context forwarding, and execution time tracking.

Orchestrate deterministic and AI steps together: plain Ruby does what code does best, a model does what only a model can, and the pipeline passes the result from one to the other.

The picture below is the OpenAI Pricing pipeline from the playground. It has two steps:

1. Fetch the page — deterministic. A Playwright scraper downloads the live OpenAI API pricing page and returns its text. No model, no tokens, the same result every time.
2. Extract the prices — LLM. A request reads that text and turns it into structured JSON: model names and token prices.

The picture shows both steps as they run, and the final result: the table of prices extracted from the page.

With ActiveHarness you can track time, tokens, and dollars for every request call, pipeline step, and tribunal.

Cost in Application	Provider's Cost

A Tool is a controlled action a model may ask for: typed parameters, ordered steps that validate and transform the input, default deny. A refusal or a failure goes back to the model as a safe message instead of crashing anything. Tools are plain classes — an Agent is only one of the ways to use them.

# app/ai/tools/read_file_tool.rb
class ReadFileTool < ActiveHarness::Tool
  include ActiveHarness::Tool::PathPolicy

  tool_scope :os          # full name: os::read_file
  tool_name  :read_file

  description   "Read the first lines of a text file inside the workspace"
  param :path,  "Absolute path of the file"
  param :lines, "How many lines to read", type: :integer, required: false

  ALLOWED_PATHS = [Rails.root.join("storage").to_s].freeze

  # Steps run in order; `deny!` refuses the call, `transform: true` rewrites the input.
  step :resolve_path, transform: true do |input|
    input.merge(path: safe_path!(input[:path]))
  end

  def perform(input)
    File.foreach(input[:path]).first((input[:lines] || 50).to_i).join
  end
end

See docs/TOOLS.md.

An Agent runs the loop: ask the model, run the tools it calls, check the goal, stop at the limits. It owns the loop and the limits; the tools it may use are listed explicitly (anything else is refused).

A goal check — decides whether the final answer is good enough:

# app/ai/goals/answer_present_goal.rb
class AnswerPresentGoal < ActiveHarness::GoalCheck
  def call(candidate:, goal:, state:)
    ok = !candidate.to_s.strip.empty?
    ActiveHarness::GoalCheckResult.new(status: ok ? :achieved : :continue, feedback: "Give a non-empty answer")
  end
end

An agent — the loop that ties a request, tools and a goal together:

# app/ai/agents/files_agent.rb
class FilesAgent < ActiveHarness::Agent
  max_turns 8

  request FilesRequest          # owns the model chain, retries and fallback
  tools   LsTool, ReadFileTool  # the only tools the model may call
  goal    AnswerPresentGoal
end

Run it:

agent = FilesAgent.call(input: "What is in the storage directory?")
agent.output       # final answer
agent.stop_reason  # :goal_reached, :max_turns, ...
puts agent.trace

rails g active_harness:install creates a working FilesAgent with its tools, goal check and tracing concerns; rails g active_harness:agent, :tool and :goal add more. Tool calling works with OpenAI-compatible providers and Anthropic. First iteration — see docs/AGENT.md and the overview.

Use Tribunals to run multiple requests in parallel and make Verdicts based on their agreement — improving reliability and reducing biases and hallucinations.

Use power of event hooks to log and trace every step of your AI flows, from individual request calls to multi-step pipelines and parallel tribunals.

Event Tracing Architecture	Grafana Dashboard

Backend Agnostic — Built on OpenTelemetry, ready for any collector (Jaeger, Datadog, Honeycomb, or custom).

Store conversation history in JSON, SQLite and PostgreSQL. Inject memory into prompts to make requests that remember past interactions.

Rails App	Console

Not every request needs a chat model. Use provider: :vercel to reach Jev, TypeSafe AI's "System One" evaluation model, via Vercel AI Gateway — ask typed score/choice/noul questions in one call and get back typed, probabilistic answers instead of free text. A natural fit for classification, routing, and content moderation.

See docs/JEV.md for the full request/response schema and use-case examples (ticket triage, refund detection, tool-call risk gating, content moderation).

File structure for Ruby and Ruby on Rails applications:

Place all of your AI-related code in app/ai to keep it organized and separate from your core application logic. You can further organize it into subdirectories for prompts, requests, tribunals, pipelines, memory, agents, tools, goals, and tracing concerns.

app/
├── models/
├── controllers/
├── views/
└── ai/
    ├── prompts/      # system prompt classes
    ├── requests/       # request classes
    ├── tribunals/    # parallel verdict panels
    ├── pipelines/    # multi-step pipelines
    ├── memory/       # custom memory classes
    ├── agents/       # tool-calling agents
    ├── tools/        # tools agents can call
    ├── goals/        # goal checks (ActiveHarness::GoalCheck)
    └── concerns/     # shared tracing (request/tribunal/pipeline/agent/tool)

• Structure
• Prompt
• Request
• Pipeline
• Tribunal
• Memory
• Tool
• Agent
• Agent loop
• Hooks, streaming, tracing

• How Prompts Work
• Minimal Prompt
• Multi-line Prompt
• Dynamic Content with @context
• Tuning with @params
• Using @input in the Prompt
• JSON Output Prompt
• Using @memory for Conversation History
• Respecting @context_window
• Generator

• How to Create Your First Request in 5 Minutes
• How to Provide Fallbacks
• Model Options
• Modifying the Model Chain at Runtime
• How to Track Retries and Failures
• How to Use with RubyLLM
• JSON Output and Parsing
• Lifecycle Events
• Custom Providers
• Streaming in the Console
• Streaming in a Rails App
• Image Generation
• Audio Transcription
• Jev / TypeSafe AI Evaluation (Vercel AI Gateway)

• Overview of all concepts
• Idea and DSL
• Data contracts (Goal, GoalCheck, AgentState)
• The standard loop
• Hooks, events, tracing
• Completion and failure

• Design (steps, deny, hooks, schema)
• Examples
• Security checklist

• How to Create Your First Pipeline in 5 Minutes
• Step Types
• Stop Conditions
• Tribunal Step
• Context: Accessing Previous Step Results
• Lifecycle Events
• Event Streams
• Full Example

• How It Works
• Minimal Example
• Stopping the Outer Pipeline from the Inside
• transform — Why It Is Required
• Accessing Inner Results from the Outer Context
• Event Streams
• Multiple Levels of Nesting
• Reference

• How to Create Your First Tribunal in 5 Minutes
• Tribunal from Different Requests
• Verdict Strategies
• Custom Verdict Logic
• Tolerating Partial Failures
• Same Request, Different Models
• Runtime Model Prepend per Request
• Direct Usage
• Lifecycle Events

• How Memory Works
• JsonFile Memory
• Custom Memory Class
• Managing Memory via Request Callbacks
• Injection Patterns
• Filtering History with to_messages
• Memory API Reference
• Memory with namespace:
• Sharing Memory Across Pipeline Steps
• PostgreSQL Backend
• SQLite Backend

• Installation
• Rails Setup
• Plain Ruby Setup
• Configuration
• Providers Reference
• Global Settings
• Generators

• How It Works
• Simple Logging with Hooks
• OpenTelemetry Setup
• RequestTracing Concern
• TribunalTracing Concern
• PipelineTracing Concern
• Span Hierarchy
• Connecting to a Backend

MIT © the-teacher