std.unicode

Normalization and a deliberately small classification surface, pinned to Unicode 17.0.0 independently of the host Rust compiler. version() returns the string "17.0.0".

import std.unicode
from std.text import TextError

def main() -> Result[(), AllocError | IoError | TextError]:
    decomposed = std.unicode.nfd("café", 64)?
    print(len("café"), len(decomposed), std.unicode.nfc(&decomposed, 64)? == "café")?
    accent = decomposed[4]
    print(std.unicode.is_mark(&accent), std.unicode.combining_class(&accent))?
    print(std.unicode.nfkc("file ①", 64)?)?
    Ok(())

4 5 True
Option[bool].Some(True) Option[u32].Some(230)
file 1

Normalization returns Result[str, std.text.TextError]. It borrows input and returns independent text. The explicit scalar limit bounds the input, fully decomposed intermediate, and result. Exceeding any gives LimitExceeded. The limit is NonNegative[i64], and a limit above i64.MAX // 4 traps. An empty string fits a zero limit. Output UTF-8 is consequently at most four times the limit. Normalization’s explicit arithmetic opt-outs rely on that validated ceiling, successfully appended scalar counts, and positive insertion-sort indices. They do not bypass scalar limits or allocation failures, and normalization adds no arithmetic error variants to TextError. The limit counts Unicode scalars, not bytes or grapheme clusters, and canonical composition does not let an oversized decomposition bypass it.

Classification allocates nothing and returns Nothing for zero or multiple scalars. Mark and nonzero combining class are distinct properties. This module does not offer locale collation, case folding, or general categories beyond Mark. Grapheme, word and sentence segmentation and display width are in the official unicode_text package, described with text segmentation and width. Existing string casing and ASCII classification remain the text builtins. Compatibility normalization can erase meaningful presentation distinctions; callers choose it explicitly. No implicit normalization occurs in equality, hashing, paths, decoding, or text builders.

Stable canonical ordering and blocked composition follow UAX #15, including Hangul and composition exclusions. Checked Plenty performs the algorithms; allocation-free scalar lookups reuse bundled unicode-normalization 0.1.25 tables and primitives. Canonical ordering uses stable insertion: O(n²) worst-case time for adversarial descending combining runs, O(n) otherwise, where n is decomposed scalar count. Storage is O(n), with two scalar lists and bounded UTF-8 construction; allocation failure is recoverable. Choose a suitably small limit for untrusted input. No hidden global state or thread affinity is introduced; ordinary ownership and worker-effect checking still apply.

The complete Unicode 17 NormalizationTest.txt fixture checks all twenty normalization equalities for each row through one compiled Plenty harness. Unicode upgrades are deliberate source changes: replace the pinned upstream sources, update the version function, fixture and licenses together, and rerun the corpus, allocation tests, tutorial tests and packaging tests. Do not derive table versions from the host toolchain. Vendoring preserves the dependency-free runtime build and avoids the upstream iterator’s infallible scratch allocation. Provenance and exact update steps are recorded beside the vendored sources.

The native tables add no external libraries or build-time network access and ship inside both runtime archives. Supported target: x86_64 Linux GNU. See the lesson and std.text for bounded output construction and error variants.

Acknowledgements

• Normalization uses tables and lookups from unicode-normalization 0.1.25, vendored in the runtime (MIT OR Apache-2.0), with data from the Unicode Character Database 17.0.0 (Unicode-3.0).