ruby_llm-bedrock_invoke

A RubyLLM provider for Anthropic Claude via the AWS Bedrock InvokeModel API, registered as :bedrock_invoke. Works with released RubyLLM 1.13+ and the unreleased 2.0 architecture from one gem.

RubyLLM's built-in :bedrock provider speaks AWS's unified Converse API. Converse is great for portability, but it is a translation layer — and Anthropic features that don't fit its schema simply don't exist there. As of mid-2026, if you need Claude-specific capabilities and your data must stay inside the AWS boundary, the feature matrix looks like this:

InvokeModel is the only surface with the full set. It takes the native Anthropic Messages body verbatim (plus anthropic_version), so this gem inherits RubyLLM's entire Anthropic wire format and swaps only the transport: SigV4 signing against bedrock-runtime and AWS event-stream decoding.

Agents with large tool catalogs (many MCP servers, dozens of tools) face an ugly trade-off:

• Load every tool eagerly → tool schemas consume thousands of prompt tokens on every request.
• Load tools dynamically (intent routing, per-request tool selection) → the tools array is part of the prompt-cache prefix, so any change to it invalidates the cache for the whole conversation.

Anthropic's tool search tool is the first-party fix: mark long-tail tools defer_loading: true and they are excluded from the context and the cache prefix until the model discovers them via a server-side search tool. Discovery happens inline in the message stream — the prefix never changes, so the cache survives.

This is measurable. bin/cache_smoke runs both strategies live against Bedrock with a ~9k-token cached system prefix:

RUN A — deferred tools + tool search        RUN B — inject tool def mid-conversation
  response              cache READ  WRITE     response              cache READ  WRITE
  R1 search + call            9064   9064     R1 tool call                   0   8851
  R2 final answer             9064      0     R2 final answer             8851      0
  R3 tool call (turn 2)       9064      0     R3 tool call (def added!)      0   8936  <- cache busted
  R4 final answer             9064      0     R4 final answer             8936      0

Run A discovers and calls a hidden tool with the cache intact on every subsequent request. Run B "loads" the same tool by adding its definition — and pays a full cache re-write. That contrast is the reason this gem targets InvokeModel: on Bedrock, tool search exists nowhere else.

gem 'ruby_llm-bedrock_invoke'

Dependencies: ruby_llm (>= 1.13, < 3), aws-sdk-core (credential chain + SigV4), aws-eventstream (streaming).

require 'ruby_llm-bedrock_invoke'

RubyLLM.configure do |config|
  config.bedrock_invoke_region = 'us-east-1' # or rely on AWS_REGION
  # No credential config needed: the standard AWS chain is used automatically.
end

chat = RubyLLM.chat(
  model: 'global.anthropic.claude-haiku-4-5-20251001-v1:0',
  provider: :bedrock_invoke,
  assume_model_exists: true # Bedrock ids/ARNs aren't in RubyLLM's registry
)

chat.ask 'Hello from inside the AWS boundary!'

Everything RubyLLM documents for chat works unchanged: streaming (chat.ask('...') { |chunk| print chunk.content }), tools, with_temperature, with_params.

Credentials resolve exactly like any AWS SDK client — env vars, shared config/SSO, ECS/EKS Pod Identity, IRSA web-identity tokens, IMDS — resolved once per process and cached (the credentials object self-refreshes). A token projected into a pod just works with zero RubyLLM configuration. This is deliberately different from core's :bedrock provider, which requires static keys or a hand-built credential provider object. To override: config.bedrock_invoke_credential_provider = anything_responding_to_credentials.

Mark long-tail tools deferred; keep your 3–5 hottest tools eager. When at least one tool is deferred, the provider automatically flags those definitions defer_loading: true, injects the tool_search_tool_regex server tool, and opts into the tool-search-tool-2025-10-19 beta (in the request body — Bedrock has no beta HTTP header).

class DatabaseQuery < RubyLLM::Tool
  include RubyLLM::BedrockInvoke::Deferred   # class-level: always deferred
  description 'Run a read-only SQL query against the warehouse'
  # ...
end

chat.with_tools(HotTool, DatabaseQuery)

For tool instances you don't control (e.g. built by ruby_llm-mcp), defer per-instance:

chat.with_tools(hot_tool, *mcp_tools.map { |t| RubyLLM::BedrockInvoke.defer(t) })

You must tell the model that hidden tools exist. In live testing, models look at their visible tool list, conclude the capability is missing, and never search. One paragraph in your (cached) system prompt fixes it:

chat.with_instructions <<~PROMPT
  Your visible tool list is only a subset of the tools available. Before saying
  a capability is missing, use tool_search_tool_regex to search the full catalog
  by keyword. Discovered tools can then be called normally.
PROMPT

1. The model calls the search tool server-side (server_tool_use block) with a regex.
2. The API returns a tool_search_tool_result containing tool_reference blocks; matching deferred tools are expanded into context inline in the message stream — the cached prefix is untouched.
3. The model calls the discovered tool like any other; RubyLLM's tool loop executes it normally.

Responses that used tool search carry the full server content blocks, which this gem replays verbatim on subsequent requests (required by the API). Access them on either RubyLLM generation with:

blocks = RubyLLM::BedrockInvoke.raw_blocks(message) # nil for ordinary messages

If you persist conversations (Rails acts_as_chat), these blocks are process-local — persist and restore them yourself if tool-search turns must survive a reload.

Caching is GA and on by default on InvokeModel — you only place breakpoints. The prefix is cached in order tools → system → messages, so one cache_control breakpoint at the end of your system prompt covers the tool schemas and instructions:

chat = RubyLLM.chat(model: MODEL, provider: :bedrock_invoke, assume_model_exists: true)
chat.with_params(system: [                       # with_provider_options on RubyLLM 2.0
  { type: 'text', text: BIG_SYSTEM_PROMPT, cache_control: { type: 'ephemeral' } }
])

Note that with_params(system:) replaces the rendered system prompt: if you also use with_instructions (e.g. for the tool-search guidance above), fold that text into the blocks array rather than setting both. On the unreleased RubyLLM 2.0 you can use the native chat.with_caching API instead.

Verify it's working by reading usage off the response:

response = chat.ask 'First question'
response.cache_creation_tokens  # > 0: the prefix was written    (2.0: response.tokens.cache_creation)
response.cached_tokens          # subsequent asks: > 0 = cache hit (2.0: response.tokens.cached)

Why this matters with deferred tools: deferred definitions are excluded from the cached prefix (a deferred tool cannot even carry cache_control — the API 400s), so the catalog can grow or shrink between conversations and discovery can happen within one, all without invalidating the cache. Mind the per-model minimum cacheable size (4,096 tokens for Haiku 4.5-class models) — below it you'll see neither reads nor writes.

# Structured output — GA on InvokeModel, no beta flag:
chat.with_schema(
  type: 'object',
  properties: { answer: { type: 'string' } },
  required: ['answer'],
  additionalProperties: false
).ask('...')

# Streaming — InvokeModelWithResponseStream (AWS event-stream framing, decoded for you):
chat.ask('Tell me a story') { |chunk| print chunk.content }

# Extended thinking — budget-based only (Bedrock has no adaptive/effort API):
chat.with_thinking(budget: 8_000).ask('Think hard about this')

• Model coverage: Claude Sonnet 5 is not served through the InvokeModel surface; Fable 5, Opus 4.8/4.7, Haiku 4.5 and earlier are. Model IDs are not registry-validated, so ARNs, inference profiles, and global./regional prefixes all work — but resolved models carry generic capability metadata: no pricing, and max_tokens defaults to 4096 (raise it with chat.with_params(max_tokens: 32_000)).
• Tool search on Bedrock is a beta and only offers the regex variant (tool_search_tool_regex); no BM25 variant, unlike the first-party API.
• Betas ride in the body: Bedrock InvokeModel has no anthropic-beta HTTP header. Opt into additional betas via chat.with_params(anthropic_beta: [...]) — the tool-search flag is re-applied automatically if your params replace the array.
• Keep deferred tools attached for the lifetime of a conversation that used tool search: past turns replay tool_reference blocks that must still resolve to sent tool definitions.
• On a retried stream (throttling mid-stream), your streaming block may see chunks from the abandoned attempt; the final returned message contains only the successful attempt.
• Embeddings, image generation, Batches, and the Files API are out of scope (use the stock :bedrock provider or first-party Anthropic).

bundle exec rspec                                                # newest 1.x
BUNDLE_GEMFILE=gemfiles/ruby_llm_1_13.gemfile bundle exec rspec  # oldest supported 1.x
BUNDLE_GEMFILE=gemfiles/ruby_llm_2_0.gemfile bundle exec rspec   # 2.0: upstream main (needs Ruby >= 3.4;
                                                                 #      RUBY_LLM_PATH=... for a local checkout)

bundle exec bin/smoke        # live AWS: completion, streaming, schema, tool search
bundle exec bin/cache_smoke  # live AWS: the cache-preservation proof shown above

Upstream RubyLLM removed its original InvokeModel chat path in 1.12.0 in favor of Converse, and the maintainer closed the community tool-search PR (crmne/ruby_llm#745) asking for a provider-agnostic design first — explicitly noting that "Bedrock exposes Claude's version through InvokeModel, though not Converse." The open feature request is crmne/ruby_llm#748. This gem is the working Bedrock-side reference implementation for that design conversation: #745's defer:/deferred user-facing API plus this provider underneath would cover both the first-party and Bedrock dialects.

MIT