std.encoding

import std.encoding provides hex, base32, base64, PEM and custom codecs and fixed-width integer and IEEE 754 float packing over std.bytes buffers, with one error type, EncodingError.

Byte order, range checks, growth and byte manipulation are checked Plenty code. Four std-only compiler intrinsics reinterpret same-width integer/float bits with a native bitcast; they add no runtime library or dependency. Bytes are accessed through validated buffer operations. Availability matches integer packing on the supported targets; the float tests are qualified on x86-64 Linux GNU.

These are binary encodings. They do not normalize, transcode, or validate text; turning decoded bytes into a str is the separate, strict Bytes.to_text conversion.

import std.bytes
import std.encoding
from std.encoding import EncodingError, Endian

def main() -> Result[(), AllocError | IoError | std.bytes.BytesError | EncodingError]:
    data = std.bytes.from_text("hi!")?
    print(std.encoding.hex_text(&data)?)?
    print(std.encoding.base64_text(&data)?)?
    print(std.encoding.base64_from_text("aGkh")?.to_text()?)?
    mut header = std.bytes.with_capacity(4)?
    std.encoding.append_u32(&mut header, 258u32, Endian.Big)?
    print(std.encoding.hex_text(&header)?)?
    print(std.encoding.unpack_u32(&header, 0, Endian.Big))?
    Ok(())

686921
aGkh
hi!
00000102
258

Codecs

source is a byte view &[u8]: pass &buffer for a whole Bytes, &buffer[start:stop] for part of it, or a b"..." literal.

Each result is a new buffer whose capacity equals its length. Encoders produce ASCII only.

For base32/base32hex, MIME line wrapping, PEM, and reusable custom alphabets, see additional codecs. They use the same views, result types, and byte indexes as the functions here.

Accepted forms

The codecs in this table accept one form. They never skip, trim, or repair input: whitespace, line breaks, and any other byte outside the alphabet are errors.

Base64 text must be canonical: bits of the last symbol that encode no byte must be zero, so each byte sequence has exactly one accepted text per codec. The empty range encodes and decodes to an empty buffer.

Decoder checks, in order:

1. The range, then the length.
2. For base64, up to two trailing = are set aside as padding.
3. Bytes are examined in order. The first one outside the alphabet is InvalidPadding if it is = and InvalidByte otherwise.
4. The trailing bits of a final partial group.

Integer packing

value has the type in the function name; offset is i64; order is Endian.Little or Endian.Big. There is no default or native order.

Endian.Big puts the most significant byte at the lowest offset. Signed values use two’s complement. pack_ and unpack_ require 0 <= offset <= len - width; a short buffer or invalid offset traps before anything is read or written. Bytes are accessed one at a time, so no offset has an alignment requirement. A single byte needs no order: use Bytes.get, set, and push.

Floating-point packing

The value type is the width in the name: f32 is IEEE 754 binary32 and f64 is binary64. Endianness is explicit, with no native-order default, alignment, padding, or object-layout serialization. Sources use the same byte views as integer unpacking: a whole buffer, a borrowed range, or a byte literal.

These operations reinterpret bits without numeric conversion, promotion, arithmetic, or text formatting. They preserve every representation, including positive and negative zero, infinities, subnormals, and quiet/signaling NaNs with their signs and payloads. Every input bit pattern is accepted. Subsequent arithmetic on a decoded float can change a NaN representation and lies outside this codec’s guarantee. Compare re-encoded bytes when testing representations; float equality cannot distinguish signed zeros or verify NaNs.

The integer bounds and failure guarantees apply unchanged. pack_ and unpack_ require 0 <= offset <= len - width, including at unaligned offsets inside a larger record. Negative, excessively large, and truncated ranges trap before reading or modifying bytes. The messages are pack: offset outside target buffer and unpack: offset outside source buffer. Packing leaves length, capacity, and surrounding bytes unchanged. Appending keeps the existing prefix; when growth fails, contents and length remain unchanged and the error is Allocation(cause). Range checks avoid overflowing offset arithmetic.

Errors

pub enum EncodingError:
    InvalidLength
    InvalidByte(i64)
    InvalidPadding(i64)
    NonCanonical(i64)
    InvalidAlphabet
    InvalidPem(i64)
    PemLabel
    Allocation(AllocError)

An index is a position in the source view; for the _from_text forms it is a UTF-8 byte offset into the text. Every fallible operation returns Result[T, EncodingError]. A failed operation leaves its target unchanged and releases anything it allocated. Wrap a result with std.error.context to name the operation.

Hex, base64, MIME and PEM expansion sizes are checked before allocating output. They return Allocation(CapacityOverflow) rather than trapping on integer overflow. Bounded decoder cursors, fixed-width packing and alphabet arithmetic use explicit unchecked: regions whose source comments state their bounds. PEM decoding records the validated body endpoint instead of reconstructing it with a potentially overflowing size formula. Invalid input keeps its existing codec error and source offset; callers’ arithmetic modes do not change these contracts.

Ownership and threads

Sources are shared loans and targets exclusive loans, so the view rules apply: a buffer cannot be packed into while a view of it is live, and cannot be encoded into itself. The module holds no global state. Its functions pass the ordinary worker-effect checks and may run in any thread.

Costs

This table covers the original hex/base64 codecs and integer/float packing. See additional codec costs for custom table construction and MIME’s temporary buffers.

Output sizes for n input bytes: hex 2n; base64 4 * ceil(n / 3); base64url ceil(4n / 3). A decoder allocates its output before it examines the alphabet, so rejected input pays for that buffer, which is released before returning.

Every byte goes through a checked single-element Bytes call. Measured with a release compiler on x86-64 Linux, pinned to one core, on an 8,388,608-byte input:

That is a few megabytes per second. Bulk native primitives would be faster and are not implemented.

Limits

• No streaming or incremental codecs: input and output are whole buffers.
• No arbitrary line widths, automatic case folding, or permissive recovery from damaged encodings.
• No variable-length integers, 128-bit integers, or additional float formats.
• No record-layout description: pack fields one call at a time.

Acknowledgements

• RFC 4648 specifies the base16, base32 and base64 formats implemented by the codecs.
• RFC 7468 specifies the textual encapsulation and label syntax used by PEM encoding.
• IEEE 754 defines the binary32 and binary64 representations used by float packing; no standard text or external implementation is shipped.