Project

pbwire

0.0
The project is in a healthy, maintained state
A FileDescriptorSet becomes the schema at runtime; messages go to/from wire bytes with no generated code, through a C extension over the pbwire core.
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
2015
2016
2017
2018
2019
2020
2021
2022
2023
2024
2025
2026
 Dependencies
 Project Readme

CI License Language Dependencies

Release v0.1.11 — every binding ships this one version; VERSION is the source of truth, scripts/sync-versions.sh propagates it.

pbwire — a dynamic Protobuf runtime for C, no code generation

pbwire is a dependency-free Protobuf runtime written in C. It loads a schema at runtime — a hand-built descriptor tree or a standard FileDescriptorSet — then builds, reads, encodes and decodes messages dynamically through reflection. No protoc step, no generated message code, no libprotobuf in the process. It speaks proto2, proto3 and edition-2023 descriptor sets, the JSON mapping and the text format, and drives fifteen host-language bindings from that one core.

The point is not to re-implement Protobuf encoding. It is to let every language with a C ABI share one dynamic Protobuf schema and one message-processing core instead of re-implementing Protobuf per host language.

uint8_t *buf;
size_t len;
pbw_encode(msg, &buf, &len);   /* 0a 03 61 62 63 10 ff ff ff ff ff ff ff ff ff 01 */
pbw_free(buf);

Documentation

Document Contents
C API guide the tutorial, hand-built schema, FileDescriptorSet load, pbw_ctx loop, ownership rules, streaming, unknown fields, JSON/text, ABI
Design and architecture how it works, dynamic message semantics, the two allocation strategies, scope, limits, use cases, repository layout
Performance benchmarks the performance document: the four standard tiers, every binding against its official runtime, and the cross-binding comparison
Testing and conformance the self-check, pinned reference bytes, difftest, the official conformance suite
FAQ syntax support, thread safety, unknown fields, untrusted input, .proto at runtime
Protobuf pain points which official-library complaints pbwire solves, eases or leaves alone — the searchable index
Bindings every host binding, where its schema comes from, and the per-language pain point each one removes
include/pbwire.h the whole public API with per-function contracts
AGENTS.md · CONTRIBUTING.md · llms.txt working on this repo, commit rules, a one-page factual summary

Why use pbwire

  • Schema at runtime. Load a FileDescriptorSet produced by protoc, buf or pbwire-protoc, and handle message types you did not compile in.
  • No runtime dependencies. C99, one public header, ~13k lines of C (about 7.5k of it the binary codec, the rest the JSON mapping and text format). make and make test need nothing but a compiler — no libprotobuf, no protoc, no JSON/C++ runtime underneath.
  • Dynamic messages and reflection. Create messages from a descriptor, get/set fields by number or name, walk nested structures, oneofs and maps.
  • Byte-identical to the reference implementation. Encodings match protoc 36.2 for every scalar type, packed/unpacked repetition and nested messages; pinned bytes in the test suite keep parity from regressing silently.
  • Built for embedding. Stable C ABI, no globals, caller-confined schema/message objects, arena-backed pbw_ctx for per-thread hot loops, wire input validated at the boundary.
  • One core, twenty-three host languages. C, C++, Python, Ruby, Java, Kotlin, Node, Objective-C, PHP, Dart, C#, Rust, Go, Erlang, Lua, Swift, Scala, Elixir, Zig, F#, Clojure, Julia and R all speak the same wire format, each with a measured answer to its runtime's own pain point — see what each binding solves.

At a glance

Language C99 (C++ consumers include the same header)
Wire format Google Protobuf binary, proto2 + proto3 (edition-2023 descriptor sets load too)
Schema input hand-built descriptors, or a standard FileDescriptorSet
Message handling dynamic value tree with reflection; no generated code
Runtime dependencies none (optional importer pbwire-protoc needs libprotobuf, C++17)
Not implemented RPC, MessageSet decoding into extensions
Memory model one allocation per decoded tree, or one reusable arena pbw_ctx per thread
Limits nesting depth 256, field numbers up to 2^29-1
Platforms Linux, macOS, anywhere with a C99 compiler
License Apache-2.0

Requirements and build

A C99 compiler for the core. CMake plus a C++17 compiler and libprotobuf only for pbwire-protoc.

make          # build build/libpbwire.a
make test     # self-checks: core, threads, descriptor set, C structs, examples round trip
make shared   # libpbwire.so.1 / libpbwire.1.dylib, for FFI hosts that dlopen it
make format   # clang-format (>= 19); make format-check
make clean

CMake is the packaging path and the only build that knows about pbwire-protoc:

cmake -B build-cmake                        # core only, no protobuf needed
cmake -B build-cmake -DPBW_BUILD_PROTOC=ON   # core + .proto importer (needs libprotobuf)
cmake --build build-cmake
(cd build-cmake && ctest)                    # same checks as `make test`, plus test_importer

# installed targets: pbwire::pbwire and pbwire::protoc
cmake --install build-cmake --prefix /usr/local

# then, in a consumer:
# find_package(pbwire REQUIRED)
# target_link_libraries(app PRIVATE pbwire::pbwire)

Everything else is optional and outside the commit gate: make difftest, make conformance and the per-binding *-test / *-bench targets. See docs/testing.md for what they verify and when they are worth running; binding targets are listed in each binding's README and in BENCHMARKS.md.

Quick start

#include "pbwire.h"

pbw_schema *schema = pbw_schema_new();
pbw_descriptor *person = pbw_schema_add_message(schema, "Person");
pbw_message_add_field(person, "name", 1, PBW_TYPE_STRING, PBW_LABEL_OPTIONAL);
pbw_message_add_field(person, "id", 2, PBW_TYPE_INT32, PBW_LABEL_OPTIONAL);
pbw_schema_link(schema);

pbw_message *msg = pbw_message_new(person);
pbw_message_set(msg, 1, pbw_value_new_string("abc"));
pbw_message_set(msg, 2, pbw_value_new_int(-1));

uint8_t *buf;
size_t len;
pbw_encode(msg, &buf, &len);   /* 0a 03 61 62 63 10 ff ff ff ff ff ff ff ff ff 01 */

pbw_message *decoded;
pbw_decode(person, buf, len, &decoded);
const char *name = pbw_value_get_string(pbw_message_get(decoded, 1));

pbw_free(buf);
pbw_message_free(decoded);
pbw_message_free(msg);
pbw_schema_free(schema);

Real programs rarely build a schema by hand — they load a FileDescriptorSet at runtime and process types they were never compiled against. The walk-through for that, the arena-backed hot loop, streaming framing and the JSON/text entry points is docs/guide.md; a runnable tutorial schema (proto2 address book, loaded from addressbook.desc at runtime) is in examples/ and runs with make examples.

Performance

make bench verifies byte-for-byte equality with libprotobuf before timing anything, so a speed number can never come from a wrong encoder. On an Apple M4 Pro with -O2 and protoc 36.2, the in-process dynamic core with a pbw_ctx runs 1.2-1.4x the throughput of the generated C++ codec on encode and 1.5-1.7x on decode, on every tier (132 B to 86 KB), and 8-15x the speed of C++ DynamicMessage on encode and 1.7-3.2x on decode. Generated code only wins where a dynamic codec meets generated straight-line writes on a tiny message (Java/Kotlin, 136 B, 0.82x/0.80x); the arena API wins every tier, and the plain API now beats generated code on every tier too.

The per-language tables — the same four payload tiers for every binding against its official runtime (Python, Java, Kotlin, Node, Ruby, Objective-C, PHP, Dart, Rust, C#, Erlang, Swift, Scala, Elixir, F#, Clojure, Julia), plus the fastest third-party runtime wherever one meaningfully exists (Node, Rust, C#, PHP), the C struct codec, 1-vs-8-thread scaling, and the reason behind each gap — live in BENCHMARKS.md. The one-row summary is here; the detail is there.

Language How it talks to the core Best encode / decode vs official Baseline
C static link, struct codec 1.30x (xlarge) / 2.31x (xlarge) protoc C++
C++ same C runtime, same header 1.30x (xlarge) / 2.31x (xlarge)¹ protoc C++
Python CPython extension 6.91x (xlarge) / 1.70x (xlarge) google.protobuf (upb)
Ruby C extension, linked 1.37x (small) / 1.92x (medium) google-protobuf gem
Java pure-Java codec, no native code 1.25x (medium) / 2.36x (xlarge) protobuf-java
Kotlin Java runtime + DSL 1.26x (medium) / 2.07x (xlarge) protobuf-kotlin
Node N-API addon 3.15x (large) / 3.98x (xlarge) vs protobufjs jspb + protobufjs
Objective-C ARC wrapper 7.00x (medium) / 3.24x (medium) protobuf-objc
PHP C extension 1.73x (small) / 1.98x (small) vs C ext (342x / 130x vs pure PHP) protobuf-php
Dart dart:ffi 15.4x (xlarge) / 4.42x (large) protobuf-dart
Rust static link 1.27x (medium) / 1.87x (xlarge) vs prost rust-protobuf 3.x + prost
C# P/Invoke 3.31x (medium) / 2.46x (medium) Google.Protobuf + protobuf-net
Go cgo dynamic (hot-swappable registry) 2.68x (xlarge) / 2.23x (xlarge) google.golang.org/protobuf
Erlang NIF 3.92x (xlarge) / 2.42x (small) gpb
Lua C module, tables + builder 8.06x (xlarge, builder) / 1.20x (large) lua-protobuf
Swift SwiftPM, static link 5.45x (xlarge) / 4.41x (medium) swift-protobuf
Scala Java runtime + DSL 1.62x (small) / 1.46x (xlarge) ScalaPB
Elixir NIF (shared core) 3.26x (small) / 1.71x (large) elixir-protobuf
Zig static link 1.87x (xlarge) / 45.2x (small) zig-protobuf
F# C# runtime + facade 3.35x (medium) / 2.46x (medium) Google.Protobuf
Clojure Java runtime + interop 9.92x (small) / 2.16x (xlarge) protobuf-java
Julia dlopen + ccall 123.9x (xlarge) / 1.24x (medium) ProtoBuf.jl
R C extension, named lists 59.7x (xlarge) / 45.3x (xlarge) RProtoBuf

¹ C++ has no separate bench; same engine as the C struct codec. Best columns take the best ratio across the four tiers for encode/decode separately, tier in parentheses (Apple M4 Pro, protoc 36.2); tiers and absolute MB/s are in BENCHMARKS.md.

Comparison with generated code, DynamicMessage and upb

Schema at runtime Generated code Runtime dependency Dynamic messages + reflection
pbwire yes none none (C99) yes
protoc generated C++ no yes libprotobuf via DynamicMessage, slower
DynamicMessage (C++) yes no libprotobuf yes
upb yes optional bundled C runtime yes
nanopb no yes none (embedded C) no

Pick pbwire when the schema is only known at runtime, when one C core serves several host languages, or when libprotobuf must not be in the process. Pick generated code only when the schema never changes and you accept a codegen step: even then generated straight-line writes win only the smallest encode cells against dynamic dispatch — the Java and Kotlin bindings at 136 B (0.82x and 0.80x) — while the in-process pbw_ctx wins every tier of the core benchmark and the C struct codec wins every tier of its own.

Contributing

See CONTRIBUTING.md. Enable hooks once per clone:

./scripts/setup-git-hooks.sh

License

Apache-2.0.