lex-llm

Base provider framework for all LegionIO LLM provider extensions.

lex-llm is a standard Legion extension gem that provides provider-neutral primitives for LLM integration. It does not include concrete provider implementations -- those live in lex-llm-* gems (e.g. lex-llm-ollama, lex-llm-openai, lex-llm-bedrock). The routing unit is a model offering, not a provider, enabling Legion to reason about any combination of local instances, remote servers, cloud providers, and fleet workers.

gem 'lex-llm'

Provider extensions should declare lex-llm as a gemspec dependency:

spec.add_dependency 'lex-llm', '>= 0.4.3'

For local development across LegionIO repos, prefer a local path override in the app or test Gemfile, not a permanent git dependency in the gemspec.

Load the extension through the Legion namespace:

require 'legion/extensions/llm'

All classes live under Legion::Extensions::Llm. Provider gems must use nested Legion extension namespaces so LegionIO autoloading finds them consistently:

require 'legion/extensions/llm'

module Legion
  module Extensions
    module Llm
      module Ollama
        def self.default_settings
          Legion::Extensions::Llm.provider_settings(
            family: :ollama,
            instance: { base_url: 'http://localhost:11434' }
          )
        end
      end
    end
  end
end

A model offering describes one concrete model made available by one provider instance. It is the base unit for routing, filtering, fleet lane creation, health, policy, and cost decisions.

offering = Legion::Extensions::Llm::Routing::ModelOffering.new(
  offering_id: 'ollama:macbook_m4_max:inference:qwen3-6-27b-q4-k-m',
  provider_family: :ollama,
  provider_instance: :macbook_m4_max,
  transport: :local,
  tier: :local,
  model: 'qwen3.6:27b-q4_K_M',
  canonical_model_alias: 'qwen3.6:27b-q4_K_M',
  model_family: :qwen,
  usage_type: :inference,
  capabilities: %i[chat tools vision thinking],
  limits: {
    context_window: 32_768,
    max_output_tokens: 8_192
  },
  health: {
    ready: true,
    latency_ms: 180
  },
  policy_tags: %i[internal_only phi_allowed],
  routing_metadata: {
    region: :local,
    accelerator: :metal
  },
  metadata: {
    enabled: true,
    eligibility: {
      ac_power: true
    }
  }
)

offering.eligible_for?(
  usage_type: :inference,
  required_capabilities: %i[tools],
  min_context_window: 16_000,
  policy_tags: %i[internal_only]
)
# => true

Common offering fields:

• offering_id: stable identifier; generated from provider, instance, usage type, and canonical alias when omitted
• provider_family: :ollama, :vllm, :bedrock, :anthropic, :openai, etc.
• provider_instance: concrete provider instance, account, node, region, or local runtime
• instance_id: compatibility alias for provider_instance
• model_family: provider-neutral family such as :openai, :anthropic, :qwen, :llama
• transport: :local, :http, :rabbitmq, :sdk
• tier: :local, :private, :fleet, :cloud, :frontier
• model: provider model name or normalized alias
• canonical_model_alias: provider-neutral alias for routers and fleet lanes
• usage_type: :inference or :embedding
• capabilities: :chat, :tools, :json_schema, :vision, :thinking, :embedding, :function_calling
• limits: context window, output token limits, rate limits, concurrency
• health: readiness, latency, recent failures
• policy_tags: :internal_only, :phi_allowed, :hipaa
• routing_metadata: scheduling metadata for routers
• metadata: extension metadata; sensitive values excluded from fleet fingerprints

Legion::Extensions::Llm::Aliases.canonical_model_alias(model, provider) normalizes aliases from aliases.json.

Legion::Extensions::Llm::Routing::OfferingRegistry is an in-memory index.

registry = Legion::Extensions::Llm::Routing::OfferingRegistry.new
registry.register(offering)

registry.find(offering.offering_id)
registry.find_by_model_alias('qwen3.6:27b-q4_K_M')
registry.filter(
  provider_family: :ollama,
  provider_instance: :macbook_m4_max,
  model_family: :qwen,
  capability: :tools
)

Fleet routing uses shared work lanes derived from offerings. A lane describes the work, not the worker:

offering.lane_key
# => "llm.fleet.inference.qwen3-6-27b-q4-k-m.ctx32768"

Embedding lanes omit context size:

Legion::Extensions::Llm::Routing::ModelOffering.new(
  provider_family: :ollama,
  instance_id: :gpu_embed_01,
  transport: :rabbitmq,
  model: 'nomic-embed-text',
  usage_type: :embedding,
  capabilities: %i[embedding]
).lane_key
# => "llm.fleet.embed.nomic-embed-text"

Any eligible worker can bind to the same lane: local MacBooks, GPU servers, vLLM workers, Ollama workers, or cloud-side LegionIO workers near Bedrock/Vertex/Azure.

Fleet communication uses protocol v2 envelopes with strict validation:

• FleetRequest: encrypted request envelope with operation, request_id, correlation_id, idempotency_key, message_context, and signed JWT replay token
• FleetResponse: encrypted response envelope with provider output
• FleetError: encrypted error envelope with typed error metadata

When fleet.compliance.encrypt_fleet is true (default), all envelopes are encrypted via Legion::Crypt. JWT replay tokens validate the issuer claim and use hash-based claim validation (no raw PHI in base64 payloads).

Fleet::ProviderResponder handles the responder side: token validation, idempotency, provider dispatch, response publishing. Fleet::WorkerExecution handles the worker side: lane binding, message consumption, backpressure.

Default fleet settings via Legion::Extensions::Llm.default_settings -- fleet and endpoint modes are disabled by default:

{
  fleet: {
    enabled: false,
    scheduler: :basic_get,
    consumer_priority: 0,
    queue_expires_ms: 60_000,
    message_ttl_ms: 120_000,
    queue_max_length: 100,
    delivery_limit: 3,
    consumer_ack_timeout_ms: 300_000,
    endpoint: {
      enabled: false,
      empty_lane_backoff_ms: 250,
      idle_backoff_ms: 1_000,
      max_consecutive_pulls_per_lane: 0,
      accept_when: []
    }
  }
}

Legion::Extensions::Llm::Routing::RegistryEvent builds envelopes for llm.registry publishing.

event = Legion::Extensions::Llm::Routing::RegistryEvent.available(
  offering,
  runtime: { host_id: 'macbook-m4-max', process: { pid: 12_345 } },
  capacity: { concurrency: 4, queued: 0 },
  health: { ready: true, latency_ms: 180 },
  lane: offering.lane_key,
  metadata: { observed_by: :lex_llm_ollama }
)

event.to_h
# => { event_id: "...", event_type: :offering_available, offering: { ... }, ... }

Supported types: :offering_available, :offering_unavailable, :offering_degraded, :offering_heartbeat. Sensitive keys (credentials, tokens, secrets, URLs, prompts) are rejected during sanitization.

Publishing is handled by RegistryPublisher (parameterized by provider_family) through the llm.registry exchange.

CredentialSources provides read-only credential discovery:

Legion::Extensions::Llm::CredentialSources.discover_credentials(
  family: :openai,
  setting_key: 'OPENAI_API_KEY'
)

Probes env vars, ~/.claude/settings.json, ~/.codex/auth.json, Legion::Settings, and optional socket/HTTP endpoints. Credentials are deduplicated via credential_fingerprint (first 8 chars of SHA-256). Probing is gated behind extensions.llm.security.credential_source_probing.

Each source gets a provenance tag: CredentialSources.source_tag(type, location, key).

AutoRegistration mixin enables providers to self-discover instances and register offerings into Call::Registry:

class MyProvider < Legion::Extensions::Llm::Provider
  extend Legion::Extensions::Llm::AutoRegistration
end

MyProvider.rediscover!  # Re-probe all instances

Discovers instances from settings, builds model offerings via discover_offerings, and registers them. Passes tier and capabilities metadata to the registry.

Streaming provides the streaming framework for OpenAI-compatible SSE responses:

• Faraday middleware handles chunk parsing, thinking extraction, and error handling
• StreamAccumulator accumulates deltas into complete messages with tool-call assembly
• Retries on HTTP 500 with partial body preservation
• Handles both Net::HTTP and Typhoeus adapters (Typhoeus chunks arrive with nil/0 status during streaming)
• Provider thinking (</think> tags, reasoning_content) is stripped from caller-visible content

provider.stream_chat(messages:, model:, tools: []) do |chunk|
  # chunk is a Chunk or StreamChunk with content_delta, reasoning_delta, tool_call_delta
end

Legion::Extensions::Llm::Schema bridges ruby_llm-schema for JSON schema tool definitions. Tools are defined as:

Legion::Extensions::Llm::Tool.new(
  name: 'search',
  description: 'Search the knowledge base',
  parameters: {
    type: 'object',
    properties: {
      query: { type: 'string', description: 'Search query' }
    },
    required: %w[query]
  }
)

All provider responses should normalize through the shared response objects:

• Responses::ChatResponse -- chat completions with message, usage, thinking, finish_reason
• Responses::EmbeddingResponse -- vectors, usage, model
• Responses::StreamChunk -- streaming deltas
• Responses::ThinkingExtractor -- extracts thinking from multiple formats (reasoning_content, </think> tags, untagged preambles)

Provider-specific thinking is always separated from caller-visible content.

A provider gem uses lex-llm for shared behavior and implements only provider-specific transport, authentication, model discovery, and translation.

At minimum, a provider extension defines:

• Legion::Extensions::Llm::<Provider> namespace
• Provider default settings
• Model discovery or static model offering registry
• Provider request/response translation
• Health and readiness checks

Canonical provider calls (all keyword-based):

provider.chat(messages:, model:, tools: [], temperature: nil, params: {}, headers: {}, schema: nil, thinking: nil)
provider.stream_chat(messages:, model:, tools: [], temperature: nil, params: {}, headers: {}, schema: nil, thinking: nil) { |chunk| ... }
provider.embed(text:, model:, dimensions: nil, params: {}, headers: {})
provider.image(prompt:, model:, size:, with: nil, mask: nil, params: {})
provider.count_tokens(messages:, model:, params: {})
provider.health(live: false)
provider.discover_offerings(live: false, **filters)

Inherited from Provider:

• #readiness(live: false) -- configured state, locality, base URL, non-live health metadata
• #model_detail(model_name) -- cache-backed lookup (24h TTL; nil results not cached)
• #model_allowed?(model_name) -- whitelist/blacklist check
• #discover_offerings(live: false) -- cached live discovery when live: false, probes endpoints when true
• #offering_transport / #offering_tier -- instance methods with class-level default_transport/default_tier overrides
• #runtime_provider_setting(key) -- fallback to Legion::Settings for model whitelist/blacklist

Inherited from Provider::OpenAICompatible:

• Full OpenAI-compatible API translation
• Model list parsing with capability/modality normalization
• Streaming with thinking extraction
• Embedding, image, transcription, moderation support
• fetch_model_detail override hook for live API model metadata

Provider settings are built with Legion::Extensions::Llm.provider_settings:

Legion::Extensions::Llm.provider_settings(
  family: :ollama,
  instance: {
    base_url: 'http://localhost:11434',
    fleet: { enabled: true, consumer_priority: 10 }
  }
)

ProviderSettings.infer_tier_from_endpoint(url) returns :local for localhost/loopback, :direct for all other hosts.

Key settings paths:

• extensions.llm.fleet -- fleet participation and behavior
• extensions.llm.fleet.endpoint -- endpoint-style worker configuration
• extensions.llm.fleet.compliance.encrypt_fleet -- encrypt fleet envelopes (default true)
• extensions.llm.fleet.auth.verify_issuer -- validate JWT issuer (default true)
• extensions.llm.security.credential_source_probing -- gate credential probing (default true)
• extensions.llm.model_whitelist / model_blacklist -- provider-level model filters
• extensions.llm.<family>.instance.<name>.model_whitelist -- per-instance override

Runtime gem dependencies: legion-json, legion-settings, legion-logging, legion-cache, faraday, faraday-typhoeus, ruby_llm-schema.

Install dependencies:

bundle install

Run the full test suite:

bundle exec rspec

Run lint and auto-correct:

bundle exec rubocop -A

Gemfile.lock is intentionally not committed for this repo.

• Do NOT mock Legion::Settings, Legion::Logging, Legion::JSON, or Legion::Cache -- require the real gems
• Legion::Cache.setup activates the Memory adapter in test (no Redis needed)
• Faraday::ConnectionFailed is rescued in discover_offerings with a concise log
• bundle exec rspec && bundle exec rubocop -A is the gate before committing

• Provider includes Legion::Cache::Helper -- use cache_get/cache_set directly
• model_detail(model_name) -- cache-backed lookup (cache_get -> fetch_model_detail -> cache_set if non-nil)
• fetch_model_detail -- override in subclass for live API calls; return { context_window: N } or nil
• model_detail_cache_key includes credential fingerprint for non-local providers
• model_whitelist/model_blacklist -- checks instance config first, then provider settings
• discover_offerings filters via model_allowed? and rescues Faraday::ConnectionFailed
• Faraday response logger: errors: false -- never dump raw stacktraces from HTTP failures
• CredentialSources.source_tag(type, location, key) -- provenance tag for discovered credentials
• CredentialSources.credential_fingerprint(value) -- first 8 chars of SHA-256

lex-llm began as a LegionIO fork of RubyLLM. RubyLLM remains credited under the MIT license in LICENSE.