lex-confabulation

A LegionIO cognitive architecture extension for confabulation detection and confidence calibration. Confabulation is the cognitive phenomenon of producing plausible but false narratives to fill gaps — occurring when an agent's confidence in a claim exceeds the evidence supporting it. This extension tracks cognitive claims, monitors the confidence-evidence gap, and surfaces high-risk assertions before they influence downstream decisions.

What It Does

Manages a registry of claims — assertions made by the agent with an associated confidence level and an evidence strength rating. The gap between these two values is the confabulation risk.

Register claims as the agent makes assertions, capturing confidence and evidence strength at the time
Verify claims when supporting evidence is confirmed
Flag confabulation when a claim is later found to be false or unsupported
Monitor calibration — the average alignment between confidence and evidence across all claims
Surface high-risk claims — those where confidence significantly exceeds evidence (risk >= 0.6)

A periodic Decay actor fires confabulation_report every 300 seconds for ongoing monitoring.

Usage

require 'lex-confabulation'

client = Legion::Extensions::Confabulation::Client.new

# Register a claim made by the agent
claim = client.register_claim(
  content:           'The database migration completed successfully',
  claim_type:        :factual,
  confidence:        0.9,
  evidence_strength: 0.4
)
# => { id: "uuid...", content: "...", claim_type: :factual,
#      confidence: 0.9, evidence_strength: 0.4,
#      confabulation_risk: 0.5, risk_label: :moderate,
#      verified: false, confabulated: false, created_at: "..." }

claim_id = claim[:id]

# Verify a claim when evidence is confirmed
client.verify_claim(claim_id: claim_id)
# => { found: true, claim_id: "uuid...", verified: true }

# Flag a claim as confabulation when found to be false
client.flag_confabulation(claim_id: claim_id)
# => { found: true, claim_id: "uuid...", confabulated: true }

# Retrieve all claims with high confabulation risk (risk >= 0.6)
client.high_risk_claims
# => { claims: [...], count: 1 }

# Full report across all claims
client.confabulation_report
# => { total_claims: 5, high_risk_claims: 1, verified_claims: 2,
#      confabulated_claims: 1, confabulation_rate: 0.2,
#      average_calibration: 0.75, overall_risk: 0.2, risk_label: :low }

# Engine summary
client.confabulation_status
# => { engine: { claim_count: 5, confabulation_rate: 0.2, average_calibration: 0.75 } }

Development

bundle install
bundle exec rspec
bundle exec rubocop

License

MIT