Release
v0.1.11— every binding ships this one version;VERSIONis the source of truth,scripts/sync-versions.shpropagates it.
pbwire — a dynamic Protobuf runtime for C, no code generation
pbwire is a dependency-free Protobuf runtime written in C. It loads a schema
at runtime — a hand-built descriptor tree or a standard FileDescriptorSet —
then builds, reads, encodes and decodes messages dynamically through reflection.
No protoc step, no generated message code, no libprotobuf in the process. It
speaks proto2, proto3 and edition-2023 descriptor sets, the JSON mapping and the
text format, and drives fifteen host-language bindings from that one core.
The point is not to re-implement Protobuf encoding. It is to let every language with a C ABI share one dynamic Protobuf schema and one message-processing core instead of re-implementing Protobuf per host language.
uint8_t *buf;
size_t len;
pbw_encode(msg, &buf, &len); /* 0a 03 61 62 63 10 ff ff ff ff ff ff ff ff ff 01 */
pbw_free(buf);Documentation
| Document | Contents |
|---|---|
| C API guide | the tutorial, hand-built schema, FileDescriptorSet load, pbw_ctx loop, ownership rules, streaming, unknown fields, JSON/text, ABI |
| Design and architecture | how it works, dynamic message semantics, the two allocation strategies, scope, limits, use cases, repository layout |
| Performance benchmarks | the performance document: the four standard tiers, every binding against its official runtime, and the cross-binding comparison |
| Testing and conformance | the self-check, pinned reference bytes, difftest, the official conformance suite |
| FAQ | syntax support, thread safety, unknown fields, untrusted input, .proto at runtime |
| Protobuf pain points | which official-library complaints pbwire solves, eases or leaves alone — the searchable index |
| Bindings | every host binding, where its schema comes from, and the per-language pain point each one removes |
| include/pbwire.h | the whole public API with per-function contracts |
| AGENTS.md · CONTRIBUTING.md · llms.txt | working on this repo, commit rules, a one-page factual summary |
Why use pbwire
-
Schema at runtime. Load a
FileDescriptorSetproduced byprotoc,buforpbwire-protoc, and handle message types you did not compile in. -
No runtime dependencies. C99, one public header, ~13k lines of C (about
7.5k of it the binary codec, the rest the JSON mapping and text format).
makeandmake testneed nothing but a compiler — no libprotobuf, noprotoc, no JSON/C++ runtime underneath. - Dynamic messages and reflection. Create messages from a descriptor, get/set fields by number or name, walk nested structures, oneofs and maps.
-
Byte-identical to the reference implementation. Encodings match
protoc 36.2for every scalar type, packed/unpacked repetition and nested messages; pinned bytes in the test suite keep parity from regressing silently. -
Built for embedding. Stable C ABI, no globals, caller-confined
schema/message objects, arena-backed
pbw_ctxfor per-thread hot loops, wire input validated at the boundary. - One core, twenty-three host languages. C, C++, Python, Ruby, Java, Kotlin, Node, Objective-C, PHP, Dart, C#, Rust, Go, Erlang, Lua, Swift, Scala, Elixir, Zig, F#, Clojure, Julia and R all speak the same wire format, each with a measured answer to its runtime's own pain point — see what each binding solves.
At a glance
| Language | C99 (C++ consumers include the same header) |
| Wire format | Google Protobuf binary, proto2 + proto3 (edition-2023 descriptor sets load too) |
| Schema input | hand-built descriptors, or a standard FileDescriptorSet
|
| Message handling | dynamic value tree with reflection; no generated code |
| Runtime dependencies | none (optional importer pbwire-protoc needs libprotobuf, C++17) |
| Not implemented | RPC, MessageSet decoding into extensions |
| Memory model | one allocation per decoded tree, or one reusable arena pbw_ctx per thread |
| Limits | nesting depth 256, field numbers up to 2^29-1 |
| Platforms | Linux, macOS, anywhere with a C99 compiler |
| License | Apache-2.0 |
Requirements and build
A C99 compiler for the core. CMake plus a C++17 compiler and libprotobuf only
for pbwire-protoc.
make # build build/libpbwire.a
make test # self-checks: core, threads, descriptor set, C structs, examples round trip
make shared # libpbwire.so.1 / libpbwire.1.dylib, for FFI hosts that dlopen it
make format # clang-format (>= 19); make format-check
make cleanCMake is the packaging path and the only build that knows about
pbwire-protoc:
cmake -B build-cmake # core only, no protobuf needed
cmake -B build-cmake -DPBW_BUILD_PROTOC=ON # core + .proto importer (needs libprotobuf)
cmake --build build-cmake
(cd build-cmake && ctest) # same checks as `make test`, plus test_importer
# installed targets: pbwire::pbwire and pbwire::protoc
cmake --install build-cmake --prefix /usr/local
# then, in a consumer:
# find_package(pbwire REQUIRED)
# target_link_libraries(app PRIVATE pbwire::pbwire)Everything else is optional and outside the commit gate: make difftest,
make conformance and the per-binding *-test / *-bench targets. See
docs/testing.md for what they verify and when they are worth
running; binding targets are listed in each binding's README and in
BENCHMARKS.md.
Quick start
#include "pbwire.h"
pbw_schema *schema = pbw_schema_new();
pbw_descriptor *person = pbw_schema_add_message(schema, "Person");
pbw_message_add_field(person, "name", 1, PBW_TYPE_STRING, PBW_LABEL_OPTIONAL);
pbw_message_add_field(person, "id", 2, PBW_TYPE_INT32, PBW_LABEL_OPTIONAL);
pbw_schema_link(schema);
pbw_message *msg = pbw_message_new(person);
pbw_message_set(msg, 1, pbw_value_new_string("abc"));
pbw_message_set(msg, 2, pbw_value_new_int(-1));
uint8_t *buf;
size_t len;
pbw_encode(msg, &buf, &len); /* 0a 03 61 62 63 10 ff ff ff ff ff ff ff ff ff 01 */
pbw_message *decoded;
pbw_decode(person, buf, len, &decoded);
const char *name = pbw_value_get_string(pbw_message_get(decoded, 1));
pbw_free(buf);
pbw_message_free(decoded);
pbw_message_free(msg);
pbw_schema_free(schema);Real programs rarely build a schema by hand — they load a FileDescriptorSet
at runtime and process types they were never compiled against. The walk-through
for that, the arena-backed hot loop, streaming framing and the JSON/text entry
points is docs/guide.md; a runnable tutorial schema (proto2
address book, loaded from addressbook.desc at runtime) is in
examples/ and runs with make examples.
Performance
make bench verifies byte-for-byte equality with libprotobuf before timing
anything, so a speed number can never come from a wrong encoder. On an Apple
M4 Pro with -O2 and protoc 36.2, the in-process dynamic core with a
pbw_ctx runs 1.2-1.4x the throughput of the generated C++ codec on encode
and 1.5-1.7x on decode, on every tier
(132 B to 86 KB), and 8-15x the speed of C++
DynamicMessage on encode and 1.7-3.2x on decode. Generated code only wins where
a dynamic codec meets generated straight-line writes on a tiny message
(Java/Kotlin, 136 B, 0.82x/0.80x); the arena API wins every tier, and the plain
API now beats generated code on every tier too.
The per-language tables — the same four payload tiers for every binding against its official runtime (Python, Java, Kotlin, Node, Ruby, Objective-C, PHP, Dart, Rust, C#, Erlang, Swift, Scala, Elixir, F#, Clojure, Julia), plus the fastest third-party runtime wherever one meaningfully exists (Node, Rust, C#, PHP), the C struct codec, 1-vs-8-thread scaling, and the reason behind each gap — live in BENCHMARKS.md. The one-row summary is here; the detail is there.
| Language | How it talks to the core | Best encode / decode vs official | Baseline |
|---|---|---|---|
| C | static link, struct codec | 1.30x (xlarge) / 2.31x (xlarge) | protoc C++ |
| C++ | same C runtime, same header | 1.30x (xlarge) / 2.31x (xlarge)¹ | protoc C++ |
| Python | CPython extension | 6.91x (xlarge) / 1.70x (xlarge) | google.protobuf (upb) |
| Ruby | C extension, linked | 1.37x (small) / 1.92x (medium) | google-protobuf gem |
| Java | pure-Java codec, no native code | 1.25x (medium) / 2.36x (xlarge) | protobuf-java |
| Kotlin | Java runtime + DSL | 1.26x (medium) / 2.07x (xlarge) | protobuf-kotlin |
| Node | N-API addon | 3.15x (large) / 3.98x (xlarge) vs protobufjs | jspb + protobufjs |
| Objective-C | ARC wrapper | 7.00x (medium) / 3.24x (medium) | protobuf-objc |
| PHP | C extension | 1.73x (small) / 1.98x (small) vs C ext (342x / 130x vs pure PHP) | protobuf-php |
| Dart | dart:ffi |
15.4x (xlarge) / 4.42x (large) | protobuf-dart |
| Rust | static link | 1.27x (medium) / 1.87x (xlarge) vs prost | rust-protobuf 3.x + prost |
| C# | P/Invoke | 3.31x (medium) / 2.46x (medium) | Google.Protobuf + protobuf-net |
| Go | cgo dynamic (hot-swappable registry) | 2.68x (xlarge) / 2.23x (xlarge) | google.golang.org/protobuf |
| Erlang | NIF | 3.92x (xlarge) / 2.42x (small) | gpb |
| Lua | C module, tables + builder | 8.06x (xlarge, builder) / 1.20x (large) | lua-protobuf |
| Swift | SwiftPM, static link | 5.45x (xlarge) / 4.41x (medium) | swift-protobuf |
| Scala | Java runtime + DSL | 1.62x (small) / 1.46x (xlarge) | ScalaPB |
| Elixir | NIF (shared core) | 3.26x (small) / 1.71x (large) | elixir-protobuf |
| Zig | static link | 1.87x (xlarge) / 45.2x (small) | zig-protobuf |
| F# | C# runtime + facade | 3.35x (medium) / 2.46x (medium) | Google.Protobuf |
| Clojure | Java runtime + interop | 9.92x (small) / 2.16x (xlarge) | protobuf-java |
| Julia |
dlopen + ccall
|
123.9x (xlarge) / 1.24x (medium) | ProtoBuf.jl |
| R | C extension, named lists | 59.7x (xlarge) / 45.3x (xlarge) | RProtoBuf |
¹ C++ has no separate bench; same engine as the C struct codec. Best columns take the best ratio across the four tiers for encode/decode separately, tier in parentheses (Apple M4 Pro, protoc 36.2); tiers and absolute MB/s are in BENCHMARKS.md.
Comparison with generated code, DynamicMessage and upb
| Schema at runtime | Generated code | Runtime dependency | Dynamic messages + reflection | |
|---|---|---|---|---|
| pbwire | yes | none | none (C99) | yes |
protoc generated C++ |
no | yes | libprotobuf | via DynamicMessage, slower |
DynamicMessage (C++) |
yes | no | libprotobuf | yes |
| upb | yes | optional | bundled C runtime | yes |
| nanopb | no | yes | none (embedded C) | no |
Pick pbwire when the schema is only known at runtime, when one C core serves
several host languages, or when libprotobuf must not be in the process. Pick
generated code only when the schema never changes and you accept a codegen
step: even then generated straight-line writes win only the smallest encode
cells against dynamic dispatch — the Java and Kotlin bindings at 136 B (0.82x
and 0.80x) — while the in-process pbw_ctx wins every tier of the core
benchmark and the C struct codec wins every tier of its own.
Contributing
See CONTRIBUTING.md. Enable hooks once per clone:
./scripts/setup-git-hooks.sh