serialize.rs

A simple bitpacking serializer for Rust.

This is a port of the C++ serialize library and is bit-for-bit wire compatible with it. It carries serialize format 2.0: the wire carries only integers and bytes, and every value has exactly one encoding (STANDARD.md). The golden wire format test pins the 106 exact bytes of STANDARD.md's "The whole message" — byte for byte the vector golden_wire_bytes pins in the C++ test suite — and on every push and pull request CI builds the real C++ library at the release pinned in .github/workflows/ci.yml and runs both implementations head to head over the library's own golden serialize function: they write byte-identical data and each decodes the other's output. That machine check covers all 106 of those bytes, and a second, harness-defined packet beside them covering what the golden data does not: degenerate ranges (min == max, zero bits on the wire, 32 and 64 bit, mid-sequence) and compressed floats that land between quanta — STANDARD.md's discriminating vector, which fails the byte comparison under FMA contraction or double widening where an on-quantum value would pass silently. The pin never goes backwards past v2.0.0, the first C++ release carrying format 2.0: its golden message ends with the end, and it reads each packet as exactly the bytes its writer reported.

Nothing here exercises the C, C# and Go ports, so this repo proves no leg of the family beyond the C++ one. What ties the family together is the C++ library as the common reference: each port runs its own head-to-head check against it in its own CI, all four ports vendor a byte-identical copy of STANDARD.md and of the shared conformance/ corpus — the upstream specification and its vectors, with a CI job in each repo that fails on drift. This port runs every vector of that corpus, the golden message included, and refuses each refused vector for the reason it states. Compatibility across the family rests on that shared reference and that shared specification, not on any Rust-to-Go or Rust-to-C# test.

Zero dependencies, no unsafe code, BSD 3-Clause. Install with cargo add serialize-official (the package name on crates.io; in code it is simply use serialize::*). That resolves to the version the registry holds, which is the one a consumer gets, and every release is listed on the releases page: a release states a format version, and two endpoints interoperate only when they carry the same one. To build against a tree the registry does not hold yet, take it from git and pin a release tag rather than tracking main:

[dependencies]
serialize-official = { git = "https://github.com/mas-bandwidth/serialize.rs", tag = "v3.0.0" }

Two CI jobs hold the registry and this repository to each other, so no version number written here has to. The consumer job installs the crate from crates.io at the version the index advertises, into a fresh project outside this repository, and encodes and decodes a message through it; it runs nightly as well as on every push, because the registry can move without a commit here. The published version job compares this repository's manifest against the index, reporting the drift on main and on pull requests and gating on a release tag.

Values are packed with exactly the number of bits they need: a bool takes 1 bit, an integer in [0,31] takes 5 bits. Write one serialize function and it handles write, read and measure — the stream type is a generic parameter, so the branches are resolved at compile time, exactly like the C++ library's templated serialize methods:

use serialize::{Stream, WriteStream, ReadStream, Result};

struct Packet {
    position: i32,
    health: i32,
    alive: bool,
}

impl Packet {
    fn serialize<S: Stream>(&mut self, stream: &mut S) -> Result<(), S::Error> {
        stream.serialize_int(&mut self.position, -1000, 1000)?;
        stream.serialize_int(&mut self.health, 0, 100)?;
        stream.serialize_bool(&mut self.alive)?;
        Ok(())
    }
}

A packet reader reads the end after the packet's top-level object, on a stream holding exactly the bytes the writer reported. STANDARD.md requires it: the end refuses a whole byte left over (trailing_data) and a set bit in the final byte's padding (nonzero_padding), so one value has one byte sequence. On write and measure the end emits nothing:

let mut stream = ReadStream::new(&buffer, bytes_written);
packet.serialize(&mut stream)?;
stream.serialize_end()?;     // the packet ends exactly here

The stream picks the error type: S::Error is Error for ReadStream and Infallible for WriteStream and MeasureStream, so the write and measure instantiations of that one function are statically incapable of failing — let Ok(()) = packet.serialize(&mut writer); is irrefutable, and ? compiles to nothing.

See examples/packet.rs for a fuller example with nested objects, variable length arrays and measuring.

Fixed point values serialize exactly — unlike compressed floats there is no quantization step — and 128 bit integers are first class, using Rust's native i128/u128:

use serialize::{Stream, Result};

struct Player {
    position_x: i64,     // Q48.16 fixed point, in ±8192 whole units
    position_y: i64,
    position_z: i64,
    entity_id: u128,     // 128 bit globally unique id
    sector_offset: i128, // ranged 128 bit integer
}

impl Player {
    fn serialize<S: Stream>(&mut self, stream: &mut S) -> Result<(), S::Error> {
        stream.serialize_fixed(&mut self.position_x, 48, 16, -8192, 8192)?;
        stream.serialize_fixed(&mut self.position_y, 48, 16, -8192, 8192)?;
        stream.serialize_fixed(&mut self.position_z, 48, 16, -8192, 8192)?;
        stream.serialize_u128(&mut self.entity_id)?;
        stream.serialize_int128(&mut self.sector_offset, -(1 << 70), 1 << 70)?;
        Ok(())
    }
}

serialize_u128 is a raw 128 bit field and always costs 128 bits. serialize_int128 is the ranged form: it costs only the bits its range needs, and where that range fits 64 bits the bytes are identical to serialize_int64. Both are byte compatible with the C++ library's serialize_uint128/serialize_int128, whether the C++ side uses native __int128 or its emulated pair.

If this library helps you, please support it

Reading untrusted data

The read path is the trust boundary. Every read is bounds checked and range validated at runtime and fails with an Error instead of panicking — malicious packet data never panics. The ? operator aborts the entire serialize function on the first error, so a value that controls a loop (a count, a length) is always validated before it drives anything. This is the Rust rendering of the C++ library's early-return serialize macros and the Go port's sticky errors, and it is the reason serialize methods take &mut values and return Result.

Every refusal names the rule that refused it, from STANDARD.md's closed list, as an Error variant whose name() is the family-wide spelling: past_end, value_out_of_range, nonzero_padding, trailing_data, non_finite_float, invalid_utf8, embedded_nul and non_canonical. ReadStream::failure() reports the first refusal's reason; the later refusals of a failed stream are its echoes.

Three obligations from STANDARD.md's Reader Obligations hold in every build:

• A refused read leaves its scalar destination unwritten. The value the caller passed in is exactly what it holds afterwards, so a caller that trusts the destination over the return code never proceeds on a value the stream did not carry. The two operations that fill a caller-owned buffer — serialize_bytes and serialize_string — leave that buffer's contents unspecified after a refusal, and a composite read may leave earlier members written, because it is a sequence of primitive reads and each one carries the rule alone.
• Failure is terminal. The first refused read latches the ReadStream: every later read on it fails with the same reason, consuming no bits and writing no destination, and ReadStream::failure() reports the first reason. Nothing after a failing operation has a defined position, so the stream enforces that rather than the caller's discipline. The latch clears only by re-initialization, which here is constructing a new stream. Custom serialize functions reject a decoded value with stream.refuse(error), which latches; Stream::fail is the error constructor it is built from and does not.
• serialize_int_relative carries the non-negative int32 domain, 0 to i32::MAX. Both previous and current lie in it, and the sequence is strictly increasing — a caller with a wrapping counter unwraps it before serializing. Every tier reconstructs current in a width that cannot wrap and then refuses a result outside the domain or not above previous, the absolute tier's 32 raw bits included: they are unsigned, so a top bit set is 2^31 or above, not a negative sequence number. The absolute tier is canonical too: a current no more than 69914 above previous has a bounded encoding, so carried in the absolute tier it is refused as non_canonical. previous is the caller's own state and never arrives off the wire, so a previous outside the domain is a debug assertion.

The write path is trusted and, as of 2.0, infallible — the writer-trusted contract the whole family shares, matching the C++ library's checkless writer. WriteStream and MeasureStream set S::Error = Infallible, so no write or measure can return an error and no error control flow survives in the compiled write path. Invalid arguments and out of range values are debug assertions, compiled out in release, where correctness is the caller's responsibility — size buffers conservatively or pre-measure with MeasureStream (its estimate is guaranteed conservative). A violated write contract in release produces a malformed stream that checked readers reject; it is never memory unsafety, and writing past the end of a buffer still panics via the slice bounds check rather than being undefined behavior. Readers keep every packet data check: fully fallible, in every build.

API misuse (bits out of [1,32]/[1,64], min > max, a buffer_size below 2, a write buffer not a multiple of 8 bytes) is a debug assertion on every stream, read and write alike, compiled out in release exactly like the C++ library's serialize_assert — the family standard is minimal runtime checking in release, and the hard checks release keeps are the read path's buffer-end and content refusals, which validate packet data, never arguments. A degenerate range where min == max is not misuse — the format defines it as costing zero bits, and the ranged integer methods accept it and recover the value from the range alone. serialize_f32 and serialize_f64 carry finite values only: a reader refuses NaN and both infinities as non_finite_float, testing the raw bits, and a writer debug-asserts on them; negative zero and the subnormals travel bit exact. Non-finite values through serialize_compressed_float are non-conforming and debug-assert on write, as does a compressed float declaration whose max - min, quantum count or top decode (min plus max - min) is not finite; a reader refuses a non-finite decode as non_finite_float.

Buffer contracts

• Write buffers must be a multiple of 8 bytes. The writer flushes 64 bit words to memory (half as many flushes as a 32 bit design). Bytes past the written data are only ever written as zeros.
• Give the reader slack for full speed. The reader loads 64 bit windows at byte granularity. ReadStream::new(buffer, bytes) takes the full buffer plus the packet length: when the buffer extends at least 7 bytes past the packet data, every load stays on the branchless fast path (the same trick the Go port plays with slice capacity). Without slack, loads near the end fall back to a guarded copy — correct, just slower.

Differences from the C++ library

• Errors instead of return false — on the read path: serialize functions return Result and propagate with ?. No macros needed. On the write and measure paths the error type is uninhabited (Infallible), which is Rust for the C++ writer's contract: it cannot fail, and the compiler knows it.
• serialize_string operates on String and refuses a NUL byte (embedded_nul, checked over the whole payload first) and invalid UTF-8 (invalid_utf8) on read, accepting every well-formed payload byte for byte. Format 2.0 removed the wide string: text travels as UTF-8 only, and a Rust String already is.
• The stream context is &dyn Any instead of void*. There is no allocator pointer — Rust serialize functions can carry whatever state they need.
• 128 bit values use Rust's native i128/u128 on every platform; there is no equivalent of the C++ library's emulated 128 bit pair, and none is needed. The wire is identical.
• serialize_fixed takes its Q format and bounds as runtime arguments checked by debug assertions (the storage type is generic over FixedPointStorage, so stream.serialize_fixed(&mut value, 48, 16, min, max) mirrors the C++ call shape); the C++ library checks the same constraints at compile time with static_asserts. The wire is identical, and the C++ contract notes about 128 bit division do not apply: the codec never divides in either implementation.
• Buffer sizes and bit counts are u64 internally, matching the C++ library's 64 bit bookkeeping (buffers past 256 MB round trip; the test suite proves it).

Benchmarking for the serialize family lives in mas-bandwidth/schema's data-driven bench, which measures the generated codecs across every language on one corpus.

Tests

cargo test                                   # the C++ suite, ported, plus differential tests
cargo test --release -- --include-ignored    # includes the 320 MB large buffer test
cargo clippy --all-targets -- -D warnings    # pedantic, configured via [lints] in Cargo.toml
cargo fmt --check
cargo +nightly miri test                     # the whole suite under Miri
cargo +nightly fuzz run hostile_read         # libFuzzer (also: round_trip)

The test suite mirrors serialize.h test-for-test, including the adversarial cases (out-of-range values smuggled into bit headroom, full-range integers, NaN handling, int_relative reconstructions that leave the domain) and the golden wire format test. tests/conformance.rs runs every vector in the vendored conformance/ corpus through the reader — the shared instrument, not expectations regenerated from this port — tests/terminal.rs pins the terminal-failure latch across six failure shapes, tests/differential.rs adds a deterministic differential write→read round trip and a hostile read pass, tests/degenerate.rs pins the zero-bit min == max range across all three streams, and fuzz/ carries the same two passes as real libFuzzer targets, mirroring the C++ library's fuzz harness.

CI runs the test matrix on Linux/macOS/Windows (debug and release), pedantic clippy, rustfmt, rustdoc, an MSRV (1.85) check, the whole suite under Miri, 60 seconds of each fuzz target, a zero-dependency guard, big-endian s390x and 32 bit i686 runs under qemu, a wasm32 build check, a drift check of STANDARD.md and conformance/ against upstream, and cargo semver-checks on pull requests.

Unsafe code is forbidden crate-wide and enforced by the compiler: the crate declares unsafe_code = "forbid" under [lints.rust] in Cargo.toml, which applies the same forbid(unsafe_code) lint level as the source attribute would, to every target in the crate.

License

BSD 3-Clause — permissive; keep the copyright notice clause, described under Crediting below.

The whole serialize family is BSD 3-Clause under the same copyright holder, so a project that links more than one of the ports has one licence to read, not several.

Crediting

Licensed BSD 3-Clause, which asks only that you keep the copyright notice. Credit is not required — but if you would like to give it:

serialize.rs by Glenn Fiedler and Rowan Claude

Fair credit keeps open source honest.