std.compress

import std.compress compresses and decompresses DEFLATE (RFC 1951), zlib (RFC 1950), gzip (RFC 1952) and Brotli (RFC 7932) data, whole std.bytes buffers at once or as std.io streams. It has one error type, CompressError.

DEFLATE codecs and CRC-32 use zlib-rs; Brotli uses the reference codec with fallible allocation callbacks, bundled into the runtime. The Plenty module owns buffering, the output limit, and the policy for input that follows a stream. The module is available on the supported x86-64 Linux GNU target.

import std.bytes
import std.compress
from std.compress import Format

def main() -> Result[(), IoError | AllocError | std.bytes.BytesError | std.compress.CompressError]:
    mut log = std.bytes.empty()?
    for _ in range(50):
        log.extend(&std.bytes.from_text("GET /index.html 200\n")?)?
    packed = std.compress.compress(Format.Gzip, &log, 6)?
    print(log.len(), packed.len() < log.len())?
    restored = std.compress.decompress(Format.Gzip, &packed, 4096)?
    print(restored == log)?
    Ok(())

1000 True
True

Functions

Format is Deflate (raw, no header or checksum), Zlib (Adler-32), Gzip (CRC-32 and length), or Brotli (no checksum). For DEFLATE formats, level runs from 0, which stores the data uncompressed in blocks, to 9, which is smallest and slowest; 6 is a usual choice. Brotli takes qualities 0–11; 5 is a useful starting point, while 10–11 spend substantially more time and memory. A level outside its format’s range is OutOfRange.

crc32 is the gzip and zip checksum; start a fresh sum with seed 0. The check value of the nine bytes 123456789 is 3421780262. seed must lie in 0 to 2³²−1. CRC-32 retains its native 32-bit polynomial checksum semantics; checked integer defaults do not change the checksum algorithm.

Input policy

A decoder accepts exactly one stream, with these rules:

• Truncation. Input that ends inside a stream is Truncated.
• Concatenated gzip members. When a gzip member ends and the next byte is 31 (the first byte of the gzip magic), the next member is decoded and its output appended, as gzip -d does for cat a.gz b.gz. Each member’s CRC and length are checked.
• Trailing data. Any other byte after the end of a stream is TrailingData, including the zero padding some tools append to gzip files. Raw DEFLATE, zlib and Brotli streams never continue.
• Integrity. A gzip CRC-32 or length mismatch, a zlib Adler-32 mismatch, and a gzip header CRC mismatch are Checksum. Raw DEFLATE and Brotli have no checksum. Any other malformed input, a zlib stream that needs a preset dictionary, or a stream of another format is Corrupt.
• Output limit. Producing more than limit bytes in total, counted across gzip members, is LimitExceeded. Exactly limit bytes succeeds. Choose the limit from what the program can hold, not from the input size: a few kilobytes of DEFLATE can expand by a factor of 1000. Decoded progress stays exact: exceeding the i64 representation also reports LimitExceeded, rather than wrapping or saturating the counter.

Gzip header fields (name, comment, modification time, extra data) are validated and discarded; the encoder writes a minimal header with no name and time 0, so equal input gives equal output.

Compressor

Compressor[W] has the Writer methods, so write_all, copy_all, and anything else generic over Writer accept it. inner is public.

Output reaches inner in order, on later write calls and in flush and finish. A write first delivers pending output from the previous one; if that fails, the error is returned and none of source is consumed. A failure keeps the undelivered bytes, so calling the method again resumes where it stopped. Interrupted inner writes are retried; a zero-byte inner write is WriteZero.

Call finish before using the result. Dropping a compressor without it leaves inner holding an incomplete stream, which a decoder reports as Truncated. After finish, write is StreamError.Io(IoError.Closed) and finish and flush only deliver what is left. Each flush ends the current block, which costs some compression ratio; flush at message boundaries, not after every write.

Decompressor

Decompressor[R] has the Reader method read(buffer, start, stop), which returns at least one decoded byte, or 0 once the stream and the input have both ended. inner is public.

An error from inner is returned unchanged and leaves the reader usable, so Interrupted and other transient failures may be retried. A decoding failure ends the stream: read returns it through the Reader protocol, and every later call repeats it. failure gives the precise cause, which stream_error maps as follows:

gzip and zlib store their checksum after the data, so a reader may already have returned bytes when it finds a Checksum failure. Treat decoded output as unverified until read has returned 0. Bytes decoded in the step that fails are discarded. decompress returns nothing on any error.

Inflater

Inflater is the decoder inside Decompressor, for a reader that owns its input and cannot hand over an inner stream, as std.archive does for ZIP entries. It applies the same input policy and limit.

Empty the output with clear before each step. After consuming the input, call step(b"") with an empty view until complete() or a step produces no output: Brotli can have pending decoded bytes after consuming its final input byte. A step that returns 0 with no output needs more input unless complete. Discard the Inflater after any error; do not rely on retries of codec errors.

Errors

pub enum CompressError:
    OutOfRange
    Corrupt
    Checksum
    Truncated
    TrailingData
    LimitExceeded
    Allocation(AllocError)

Ownership and threads

A Compressor or Decompressor owns its inner stream, its codec state, and its buffers; dropping it releases them all, finished or not, and does not flush or close inner. Neither can be copied. The module has no global state, passes the ordinary worker-effect checks, and may be used from any thread; moving a stream to a worker moves its ownership.

The codec state is held in a Bytes whose contents are the pending output. The __compress_* intrinsics that operate on it are reserved for the standard library. Given a buffer not made by std.compress, they stop the program with operand is not a std.compress state. Both native engines report a successful consumed count within the offered input view and keep pending output within the reserved capacity. The Plenty adapters rely on these bounds for their buffer cursors; these native wrapper contracts are unchanged by checked arithmetic defaults. Inner stream progress is still validated separately: InvalidProgress does not advance the cursor or discard the pending tail.

Costs

For the DEFLATE formats, memory is fixed when a stream is made and does not grow with the data. Brotli’s different allocation contract is described separately:

Inputs are read directly from the caller’s buffer; output is copied once, from the codec’s buffer to inner or the caller’s buffer. Allocation failures are reported as Allocation; dropping a failed or unfinished codec releases its state and buffers.

Supply

The runtime builds zlib-rs from an unmodified copy under plenty-runtime/vendor, so building the compiler needs no network and an installed compiler needs neither the sources nor a Rust toolchain:

zlib-rs has no dependencies. It is a Rust port of zlib with zlib-ng’s optimisations, and reports allocation failure as an error rather than aborting, which Allocation relies on. The runtime calls its zlib-style interface (deflate::init, inflate::inflate, and so on), enabled by the crate’s __internal-api feature.

The licence file is kept with the sources. The version is pinned in Cargo.lock, and a test fails if the two disagree. Updating it is the runtime maintainers’ job, with each release that fixes a defect: replace the directory with the new release, run cargo update -p zlib-rs --precise VERSION, and rerun tests/test_std_compress.rs, whose fixtures come from CPython’s zlib and gzip and from the system gzip tool. The __internal-api interface is not covered by semver, so read the release’s changes to deflate.rs and inflate.rs.

Executables are linked with --gc-sections, and the codecs are reached only through their own runtime entry symbols, so a program that does not use std.compress contains none of them.

Limits

• No bzip2, xz, zstd, or lz4. ZIP and TAR are in std.archive.
• No preset dictionaries, compression strategies, or window and memory settings; DEFLATE encoders use a 32 KiB window and zlib’s default memory level, and Brotli uses the standard 4 MiB encoder window.
• No access to gzip header fields.
• Trailing padding after gzip members is an error, not ignored.
• No seeking within compressed data.
• One target, x86-64 Linux GNU.

Acknowledgements

• Wraps zlib-rs 0.6.8, vendored in the runtime (Zlib).