fast-protowire

The Protocol Buffers wire format for Ruby, with nothing on top of it: declare a message's fields, get encode and decode for exactly those bytes.

It is not a replacement for google-protobuf. There are no descriptors, no reflection, no JSON mapping and no generated code. It exists for libraries that emit or read a fixed, known schema and want the memory cost of doing so to be roughly the size of the encoded output, rather than a native message object and arena for every field, as google-protobuf allocates.

gem "fast-protowire"

require "fast/protowire" has no dependencies.

Declarations mirror the .proto text. Run this with bundle exec ruby from a project that has the gem:

require "fast/protowire"

class LabelPair < Fast::Protowire::Message
  field :name, :string, 1
  field :value, :string, 2
end

class Counter < Fast::Protowire::Message
  field :value, :double, 1
end

class Metric < Fast::Protowire::Message
  repeated :label, LabelPair, 1
  field :counter, Counter, 3
end

metric = Metric.new(label: [{ name: "method", value: "GET" }], counter: { value: 12.0 })
bytes = metric.encode
Metric.decode(bytes) == metric # => true

Fields encode in field-number order, unknown fields survive a decode/encode round trip, and the output is byte-identical to what protoc-generated code and google-protobuf produce for the same values. The tutorial walks through a full schema and proves that.

Measured against google-protobuf on the Prometheus client model, a family of 36,000 metrics with twelve labels each (Ruby 4.0.7; conditions and every table on the benchmarks page):

• Encoding allocates one object, the output, whatever the message's size or depth; 0.1.0 allocated 647,000 for this family. Decoding allocates only the messages, containers and Strings it returns, 5x fewer than 0.1.0 and 1.3x faster.
• Built a message at a time, the way an exposition builds series, google-protobuf leaves 504,001 native arenas and 225 MiB behind for an 11.76 MB body and spends 1.77 s of every ten builds in GC; fast-protowire leaves 16.5 MiB, no arenas, and 0.27 s.
• google-protobuf is native, and 9 to 25x faster per operation on an existing tree or one nested Hash. This gem trades that speed for memory that is roughly the size of the output; fast-prometheus's scrape path goes further and writes series with Wire directly, with no message per series at all.

• Tutorial: declare, encode and decode a message — declare a slice of the Prometheus client model, encode, decode, and check the bytes against google-protobuf.

• How to declare a schema from a .proto file — the keyword-by-keyword translation, including proto2, oneofs, maps, packing and forward references.
• How to stream a large repeated field — append thousands of entries to a message without holding them all at once.
• How to use declared messages with protocol-grpc — request and response classes for protocol-grpc and async-grpc stubs.
• How to verify parity against google-protobuf — add a case to the parity suite, or run the same check in your project.

• Reference: Message — the declaration DSL, the instance API, and the errors.
• Reference: field types — every scalar type with its wire type, Ruby value, range, presence and packing rules; enums.
• Reference: Wire and Reader — the primitives for writing and reading the wire format directly.

• Design: the wire format and nothing else — what google-protobuf costs per message, what this gem does instead, and what it leaves out.
• How encoding works — compiled encoders, buffers, field order, presence, and decoding.
• Benchmarks — encode, build and decode against google-protobuf, what changed since 0.1.0, and how to reproduce them.

bundle exec sus
bundle exec rubocop
BENCH_QUICK=1 bundle exec ruby benchmark/messages.rb   # encode, build and decode against google-protobuf