Project

lex-validator

0.0
No release in over 3 years
lex-validatorlegionio/lex-validatorHomepageDocumentationSource CodeBug TrackerWiki
Fleet pipeline validation: tests, lint, security scan, adversarial LLM review
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
2015
2016
2017
2018
2019
2020
2021
2022
2023
2024
2025
2026
 Popularity
Downloads
76
Stars
0
Forks
0
Watchers
0
 Releases
Current version
0.1.0
Total releases
1
First release
1980-01-02
Latest release
1980-01-02
 Development
Primary Language
Ruby
Licenses
MIT
Average date of last 50 commits
2026-04-14
Reverse Dependencies
0

 Dependencies

Development

rspec
~> 3.13
rspec_junit_formatter
>= 0
rubocop
~> 1.75
rubocop-rspec
>= 0
simplecov
>= 0

Runtime

concurrent-ruby
>= 1.1
legion-cache
>= 1.3.11
legion-crypt
>= 1.4.9
legion-data
>= 1.4.17
legion-json
>= 1.2.1
legion-llm
>= 0.1
legion-logging
>= 1.3.2
legion-settings
>= 1.3.14
legion-transport
>= 1.3.9
 Project Readme

lex-validator

Fleet pipeline validation for LegionIO. Receives implemented work items and runs them through a four-stage gauntlet — tests, lint, security scan, and adversarial multi-model LLM review — before rendering a verdict.

Overview

lex-validator is the fourth and final stage of the LegionIO Fleet Pipeline:

lex-assessor → lex-planner → lex-developer → lex-validator → ship
                                                    ↓
                                          (rejected) → lex-developer (feedback loop)

When a work item arrives, the validator runs each check in sequence. All checks must pass. Any failure produces a rejected verdict with structured feedback routed back to the developer for the next iteration. A passing item advances to the ship stage.

Validation Pipeline

Step Check Early Exit
1 Tests — bundle exec rspec via lex-exec No
2 Lint — bundle exec rubocop via lex-exec No
3 Security scan — per-file pattern analysis No
4 PR comment check — filter out unresolved external review comments Yes — on unresolved human feedback
5 Stale diff guard — verify PR head SHA matches developer push Yes — on SHA mismatch
6 Adversarial LLM review — k independent reviewers, unanimity required No
7 Quality gate — weighted scoring, threshold ≥ 0.8 No

Quality Gate

Scores are composed from the check results and weighted:

Dimension Weight Signal
Completeness 35% Test pass rate
Correctness 35% Lint clean + security clean
Quality 20% LLM review verdict
Security 10% Security scan clean

Weights and threshold are configurable via Legion::Settings:

fleet:
  validation:
    quality_gate_threshold: 0.8
    quality_weights:
      completeness: 0.35
      correctness: 0.35
      quality: 0.20
      security: 0.10

Adversarial Review

lex-validator runs k independent LLM reviewers in parallel (configurable per work item, default from implementation.validators). Unanimity is required — a single dissenting vote rejects the work item. Errors and timeouts also reject (fail closed).

Each reviewer receives a different intent.capability level (basic, moderate, reasoning) to encourage model diversity across the review pool (MAKER approach). Upstream models from the pipeline trace are excluded to prevent same-model bias.

fleet:
  llm:
    validator_timeout_seconds: 120

Installation

gem 'lex-validator'

Usage

require 'legion/extensions/validator'

result = Legion::Extensions::Validator::Runners::Validator.validate(
  work_item: {
    work_item_id: 'uuid-001',
    title: 'Fix sandbox timeout on macOS',
    repo: { owner: 'LegionIO', name: 'lex-exec', language: 'ruby' },
    config: {
      estimated_difficulty: 0.5,
      implementation: { validators: 3 },
      validation: {
        enabled: true,
        run_tests: true,
        run_lint: true,
        security_scan: true,
        adversarial_review: true
      }
    },
    pipeline: {
      stage: 'implemented',
      trace: [],
      attempt: 0,
      feedback_history: [],
      changes: ['lib/sandbox.rb'],
      pr_number: 42,
      branch_name: 'fleet/fix-lex-exec-42'
    }
  }
)

result[:success]                                    # => true
result[:work_item][:pipeline][:stage]               # => 'validated'
result[:work_item][:pipeline][:review_result]       # => { verdict: 'approved', score: { aggregate: 0.95, ... }, ... }
result[:work_item][:pipeline][:trace].last[:stage]  # => 'validator'

Verdict shape

{
  verdict:        'approved',   # or 'rejected'
  score: {
    completeness: 1.0,
    correctness:  1.0,
    quality:      1.0,
    security:     1.0,
    aggregate:    1.0
  },
  issues:          [],          # strings describing failures
  merged_feedback: []           # deduplicated feedback for developer
}

On rejection, pipeline.feedback_history is appended with the round's issues, enabling the developer stage to incorporate targeted feedback on the next attempt.

Disabling checks

Any validation check can be disabled individually:

validation: {
  enabled: true,
  run_tests: false,        # skip rspec
  run_lint: false,         # skip rubocop
  security_scan: false,    # skip security scan
  adversarial_review: false  # skip LLM review (quality score defaults to 1.0)
}

Set enabled: false to skip validation entirely and return an immediate approved verdict.

Transport

  • Exchange: lex.validator (topic, durable)
  • Queue: lex.validator.runners.validator (durable, routing key lex.validator.runners.validator.#)

Development

bundle install
bundle exec rspec        # 48 examples
bundle exec rubocop      # 0 offenses

License

MIT