Introduction
Plenty is a general-purpose programming language. Plenty is statically-typed and compiles ahead of time to native code. Plenty is most heavily influenced by Python, Rust, and Zig.
Why the name “Plenty”?
I wanted a language without trade-offs: simple syntax that is easy to learn, easy to read and write, but also produces fast code, and is safe and reliable, and with a huge standard library to reduce the need for third-party dependencies. And more! I wanted everything that is good about each programming language. The programmer’s version of “a land of plenty” where you can have it all.
Goals
There are multiple goals for Plenty:
- Convenience and safety: Could we merge the best parts of Python and Rust? This is Plenty.
- Explicit ownership: Ownership and borrowing are explicit, with compile-time checks.
- Sum types: Enums are tagged unions and are used to represent sum types. Very similar to Rust.
- Fast compilation: The compiler is designed to compile quickly, even for large codebases.
- Efficient: An optimizing compiler generating machine code (Cranelift). SIMD support. Parallel tools like Rust’s
rayon. Exploit hardware features. - No garbage collector: Deterministic memory usage patterns.
- No dynamic behaviour: Figure out as much as possible at compile time.
- Fallible memory allocation: All operations that allocate memory can report allocation failure without exceptions.
- Simple FFI interop with C: In both directions. There is an interface file format to specify lifetime information for resources from a C library so that Plenty can manage them safely.
- No inheritance: Classes and enums have fixed declarations rather than dynamic attributes or inheritance.
- Generics and Protocols: A native version of Python’s
typing.Protocol. Something like Go interfaces, or a simpler version of Rust’s traits. - Batteries included: A standard library so stuffed to the gills with awesome that many programs can be written using the standard library alone.
- Simple package management: No central package repo. Use golang-inspired git links for dependencies.
The primary influences on Plenty are:
- Python, for syntax
- Rust, for ownership and borrowing, and getting the details right
- Zig, for emphasizing fallible memory allocation
Getting Started
This book provides:
- Learn Plenty teaches the language with runnable examples. Start here.
- Reference defines language rules, built-in facilities, and compiler/runtime architecture in separate areas.
Tested examples
cargo test --test integration test_tutorial:: compiles and runs every
example in Learn Plenty. It compares the output with the expected output on
the page. An example that must fail must give the expected diagnostic.
Build this book
mdbook serve book --open
Learning Plenty
Plenty combines Python-shaped syntax with explicit types, ownership, and native compilation. The examples are self-contained and teach implemented features.
Start with Setting up to install a release and put plenty
on your PATH. Then follow the ten core parts to build a small command-line tool:
- Your first program: compile, run, and check.
- Values, functions, and control flow: name values, choose types, define functions, and repeat work.
- Absence and failure: handle missing values and errors
with
Option,Result,match, and?. - Collections, ownership, and borrowing: keep groups of values, transfer ownership, copy, and borrow.
- Iteration and transformation: visit values, transform them, and produce them incrementally.
- Defining your own data: write records, methods, constructors, and enums.
- Heterogeneous collections: combine different kinds of values with enums and anonymous sums.
- Special-purpose data structures: choose columnar storage for calculations over selected fields.
- Organizing a program: split a program into modules and choose its public interfaces.
- Useful programs and resources: use command-line arguments, text, standard streams, and files in a complete application.
Then choose an optional path when your program needs it:
- Generic code and protocols reuse algorithms across different types.
- Functions as values introduce callbacks and captured state, then stored and consuming callbacks.
- Concurrent and parallel programs coordinate workers, communicate, stop work, and bound waiting.
- Working with C libraries call foreign code, own foreign resources, and export library interfaces. This path does not require concurrency.
The Reference describes detailed contracts and limits. Use the supplementary examples to look up collection methods, text operations, and advanced file usage.
Setting up
Install Plenty once, then use the plenty command throughout this tutorial.
You do not need Rust, Cargo, or a copy of the compiler’s source repository.
Building from a source checkout instead? The local build recipes
provide just build for the release compiler, both optional engines, and their
licenses/metadata. Put that checkout’s target/release/ on PATH. These builds
require Rust and native build tools and target your local system, unlike the
portable binary download described here.
Check your system
Binary releases support x86-64 and ARM64 Linux with glibc 2.28 or later.
This page uses the x86-64 archive names; on an ARM64 machine (uname -m
prints aarch64), replace x86_64-unknown-linux-gnu with
aarch64-unknown-linux-gnu in every command. Musl-based distributions such as
Alpine are unsupported. Releases also include
experimental macOS and Windows downloads,
whose READMEs replace the Linux steps below. See the
installation requirements
for supported systems and troubleshooting.
Plenty also needs a C linker and development libraries to run or compile a program. Install the packages for your distribution before continuing:
# Debian or Ubuntu
sudo apt-get update && sudo apt-get install gcc libc6-dev
# Fedora, RHEL, AlmaLinux, or Rocky Linux
sudo dnf install gcc tar gzip
# Arch Linux
sudo pacman -Syu --needed gcc
Download and extract Plenty
Open Plenty’s GitHub Releases and choose a published release. Under Assets, download both:
plenty-VERSION-x86_64-unknown-linux-gnu.tar.gzplenty-VERSION-x86_64-unknown-linux-gnu.tar.gz.sha256
Choose the binary archive, rather than GitHub’s Source code (zip) download.
Plenty currently ships as a compressed tar archive, so extract it with tar.
In every command below, replace VERSION with the version you downloaded;
keep the same version in the archive name and directory name.
Assuming both files are in ~/Downloads, verify and extract them:
cd "$HOME/Downloads"
sha256sum -c plenty-VERSION-x86_64-unknown-linux-gnu.tar.gz.sha256
mkdir -p "$HOME/.local/opt"
tar -xzf plenty-VERSION-x86_64-unknown-linux-gnu.tar.gz -C "$HOME/.local/opt"
Check that the checksum command reports OK before extracting. The resulting
directory contains bin/plenty and bin/plentyc. Keep the extracted directory;
the shell setup below points to its bin directory.
Add Plenty to your PATH
PATH is the list of directories your shell searches for commands. Choose the
instructions for your shell. These paths are quoted so they also work when
your home directory contains spaces.
Bash
Add this line to ~/.bashrc, replacing VERSION:
export PATH="$HOME/.local/opt/plenty-VERSION-x86_64-unknown-linux-gnu/bin:$PATH"
Run that same line in your current terminal to use Plenty immediately. New
interactive terminals read .bashrc. If you use a Bash login shell, ensure
your login startup file also sources .bashrc; see
Bash startup files.
Zsh
Add this line to ~/.zshrc, replacing VERSION:
export PATH="$HOME/.local/opt/plenty-VERSION-x86_64-unknown-linux-gnu/bin:$PATH"
Run that same line in your current terminal. New interactive Zsh terminals
read .zshrc. If you configured ZDOTDIR, use $ZDOTDIR/.zshrc instead;
see Zsh startup files.
Fish
Run this command once in Fish, replacing VERSION:
fish_add_path --universal --prepend --move "$HOME/.local/opt/plenty-VERSION-x86_64-unknown-linux-gnu/bin"
fish_add_path
updates the current shell and remembers the directory for future Fish sessions.
The directory must already exist, so extract the archive first. There is no
need to edit config.fish for this universal setting.
Check the installation
In any of these shells, run:
command -v plenty
plenty --version
cc --version
The first command should name plenty inside your extracted archive; the second
prints the installed version and target. If plenty is not found, check that
the PATH entry matches the actual extracted directory. If another installation
appears, put the new directory first on PATH. If cc is missing, install the
linking packages above.
You are ready for Your first program. Save your programs in any working directory you choose. When installing a newer release, update the PATH entry to its new directory; see upgrading and switching versions.
Your first program
After setting up Plenty, run a complete program, then check and compile changes.
Run a program
Save this as hello.plenty:
def main() -> Result[(), IoError | OverflowError[i64]]:
print("Hello, Plenty!")?
print(6 * 7)?
Ok(())
Hello, Plenty!
42
In the directory where you saved the file, run:
plenty hello.plenty
A run compiles your file to a native executable and then runs it. If your shell
cannot find plenty, follow Setting up first.
Plenty caches compiled code automatically. On supported Linux toolchains, unchanged runs also reuse the linked executable, so they do not link again. See the compile cache for invalidation and toolchain limits. The program still runs each time.
On Unix, you can put #!/usr/bin/env plenty before the first line, then run:
chmod +x hello.plenty
./hello.plenty
This uses the same automatic caches as plenty hello.plenty.
Execution begins at main. Indentation groups its statements; use four spaces.
print prints one value followed by a newline. Put executable statements inside
functions.
Printing can fail. The ? after each call continues when printing succeeds and
returns an output error when it fails. Integer multiplication can overflow too,
so the signature includes OverflowError[i64]. Its operands fit here, but the
function still declares the arithmetic error contract. Ok(()) reports that the
program finished successfully. Keep this pattern for printing;
Absence and failure explains how to handle errors.
Each example is a complete program. Run it in a new file, along with any labelled companion modules shown on its page.
Change and check a program
Change the greeting or calculation, then run the file again. Each run starts a fresh process.
Use plenty --check hello.plenty to check syntax and types without
running the program. A check needs no linker. A run or native compilation needs
a cc-compatible linker driver installed.
To keep an executable:
plenty --compile hello.plenty -o /tmp/hello-plenty
/tmp/hello-plenty
The executable can run without Plenty installed, on a compatible host system. To ship an application directory instead:
plenty --compile hello.plenty --dist dist/hello
./dist/hello/hello
Choose a destination that does not already exist. This greeting needs only the executable. Applications reaching optional native engines also get the exact libraries and packaged notices they need, without development-machine lookup paths. This is not a static or cross-platform build; see application distributions for engine setup and platform requirements.
Parse and type errors prevent execution. A runtime error can happen after
earlier effects, such as printing. Use --check when you want feedback
without those effects, or print(value)? inside main to inspect a value.
Values, functions, and control flow
Name values, write typed functions, and choose or repeat work:
- Name values
- Choose numeric types
- Calculate with numbers Floating-point calculations extend these rules.
- Write text
- Build text with f-strings
- Define functions
- Complete work without returning data
- Choose between branches
- Return early
- Keep names in scope
- Repeat with a range
- Repeat while a condition holds
- Give types descriptive names
Next: Absence and failure.
Name values
Use = to create a binding. Plenty infers the type of a local value, so you
usually do not need an annotation:
def main() -> Result[(), IoError | OverflowError[i64]]:
# Comments begin with a hash.
price = 20
quantity = 3
total = price * quantity
print(total)?
Ok(())
60
To see inferred types without writing annotations, enable your editor’s
inlay hints.
For example, the Plenty language server can show : i64 after price,
quantity, and total. These labels are only displayed in the editor; they
are not added to your source file. Hover also shows a binding’s type.
Bindings are immutable by default:
def main() -> ():
score = 10
score = 11
`score` is immutable; declare it with `mut`
Use mut when a value needs to change. Write it once, at the declaration:
def main() -> Result[(), IoError | OverflowError[i64]]:
mut score: i64 = 10
score += 5
print(score)?
Ok(())
15
A mutable binding keeps its original type. score: i64 states the type
explicitly; without the annotation, this integer literal also infers i64.
score += 5 adds to the current score. Numeric values also support -=, *=,
/=, //=, and %=. Use //= for integer division and /= for float division;
%= requires integers. These updates require an existing writable destination
and keep its type. Strings and collections use their explicit operations.
Choose the width of an integer
The built-in integer names specify their sizes:
| Types | Meaning |
|---|---|
i8, i16, i32, i64 | Signed integers of 8, 16, 32, or 64 bits |
u8, u16, u32, u64 | Unsigned integers of those widths |
For example, i8 can hold -128 through 127, and u8 can hold 0 through 255.
A whole-number literal without a suffix, such as 42, defaults to i64 when
its type is unconstrained. An annotation, parameter, return type, or arithmetic
operand can supply the type instead.
Append a built-in integer type to choose another width:
def main() -> Result[(), IoError]:
small: u8 = 200u8
offset: i32 = -12i32
large: u64 = 1_000_000u64
print(small)?
print(offset)?
print(large)?
Ok(())
200
-12
1000000
An annotation guides an unsuffixed literal, so small: u8 = 200 works.
It does not convert an already typed initializer:
convert the value explicitly for that.
Underscores separate digits for readability, as in 1_000_000u64 above.
Integers may also be written in hexadecimal, octal, or binary, with the same
suffixes and range checks:
def main() -> Result[(), IoError]:
mask: u8 = 0b1111_0000
print(0xFF, 0o755, mask, 0x1F600u32)?
Ok(())
255 493 240 128512
Calculate with numbers
Arithmetic requires matching types. Integer failures are recoverable by default:
list their error types in the function’s Result return, and use Ok(...)
for success. Operators themselves still produce ordinary numbers.
def main() -> Result[(), IoError]:
small = 7u8
print(small + 1i64)?
Ok(())
cannot combine u8 with i64; convert with `i64(small)`
Choose a matching literal or explicitly convert a value:
def main() -> Result[(), IoError | OverflowError[u8] | OverflowError[i64]]:
small = 7u8
print(small + 1u8)?
print(i64(small) + 1)?
Ok(())
8
8
i64(small) is safe because every u8 value fits in an i64. When the target
cannot hold every value of the source, use try_from. It checks the range and
returns a Result: Ok with the converted value, or
Err(RangeError.OutOfRange) when the value does not fit. RangeError has only
this one variant; parsing text uses the separate ParseError type.
def main() -> Result[(), RangeError | IoError | OverflowError[u8]]:
wide = 200
small = u8.try_from(wide)?
print(small + 1u8)?
Ok(())
201
? takes the converted value or returns its error to the caller. The
RangeError | IoError error type preserves either a conversion error or a
printing error. Absence and failure explains propagation
and how to recover from an error.
A cast to an integer type requires the value to fit. A literal that does not fit is rejected when the program is compiled:
def main() -> ():
small = u8(300)
300 is out of range for u8; u8.wrapping_from keeps the low bits
A value known only when the program runs is checked then. With wide = 300,
u8(wide) stops the program with error: integer cast out of range.
Casts retain this trapping behavior; arithmetic errors use typed returns. The same applies to a negative value cast to an
unsigned type, such as u64(-1), and to an unsigned value above the signed
maximum, such as i8(200u8). Cast when the value is known to fit, and use
try_from when it might not.
When discarding the high bits is the intent, say so with wrapping_from, which
never fails:
def main() -> Result[(), IoError]:
wide = 300
negative = -1
print(u8.wrapping_from(wide))?
print(u64.wrapping_from(negative))?
print(i8.wrapping_from(200u8))?
Ok(())
44
18446744073709551615
-56
Integer overflow returns OverflowError[T], where T is the operand type.
An unsigned type has no negative values, so unary minus requires a signed
integer or a float:
def main() -> Result[(), IoError]:
small = 7u8
print(-small)?
Ok(())
unary minus is not defined for unsigned type u8
Convert first: -i16(small) is -7.
The same rule applies to literals, including zero:
def main() -> ():
zero: u8 = -0
unary minus is not defined for unsigned type u8
+, -, and * have their usual arithmetic precedence. // divides integers
and rounds down, including for negative values:
def main() -> Result[(), IoError | OverflowError[i64] | ZeroDivisionError]:
print(1 + 2 * 3)?
print((1 + 2) * 3)?
print(-7 // 3)?
Ok(())
7
9
-3
Integer division by zero returns ZeroDivisionError. Signed division also
requires OverflowError[T] for T.MIN // -1. The contract follows the
operators and operand types, even with a known nonzero divisor such as 3.
Use / for floating-point division.
Integers also support bitwise operations and shifts. A shift count must be nonnegative and smaller than the integer width. Signed right shifts preserve the sign; unsigned right shifts fill with zero. Left shifts discard high bits.
def main() -> Result[(), IoError]:
bits = 5u8
print(bits & 3)?
print(bits | 2)?
print(bits ^ 3)?
print(~bits)?
print(bits << 1)?
print(bits >> 1)?
print(255u8.wrapping_add(1))?
print(0u8.wrapping_sub(1))?
print(128u8.wrapping_mul(2))?
Ok(())
1
7
6
250
10
2
0
255
0
Use wrapping_add, wrapping_sub, or wrapping_mul when a calculation
deliberately wraps at the integer width, as in a hash or a random-number
generator. Ordinary arithmetic returns a typed overflow error. Parenthesize mixed
bitwise expressions when that makes the intended grouping easier to read.
rotate_left and rotate_right move bits around the ends of an integer
instead of discarding them. The count has the integer’s own type. No count
loses bits, so rotation never traps: a count of the full width leaves the value
unchanged.
def main() -> Result[(), IoError]:
bits = 129u8
print(bits.rotate_left(1))?
print(bits.rotate_right(1))?
print(bits.rotate_left(8))?
print(bits << 1)?
Ok(())
3
192
129
2
pow raises a number to a power, and each integer type names its limits as
MIN and MAX. As with the operators, the number and its exponent must have
one type, and the result has that type:
def main() -> Result[(), IoError | OverflowError[i64] | OverflowError[u8]]:
change = -7
small = 200u8
print(change.pow(2))?
print((2).pow(10))?
print(u8.MAX - small)?
print(i64.MIN)?
Ok(())
49
1024
55
-9223372036854775808
There is no ** operator. An integer literal receiver needs parentheses, as in
(2).pow(10).
A result that does not fit returns the same typed overflow error as the
operators. An integer power with a negative exponent still stops with
negative integer exponent; convert to a float for that.
Recover from overflow
Declare the errors a calculation can produce and propagate them to its caller.
An arithmetic failure returns Err immediately; otherwise explicitly return
Ok with the successful value.
def average(total: u8, count: u8) -> Result[u8, OverflowError[u8] | ZeroDivisionError]:
rounded = total + count // 2
Ok(rounded // count)
def main() -> Result[(), OverflowError[u8] | ZeroDivisionError | IoError]:
print(average(200, 4)?)?
print(average(250, 20).is_err())?
print(average(250, 0).is_err())?
Ok(())
50
True
True
The error type lists what the function’s operators can produce:
OverflowError[u8] for + on u8, and ZeroDivisionError for //.
Unsigned // cannot overflow, so it adds only ZeroDivisionError. ? passes
either error to the caller, as with try_from. An OverflowError holds the
wrapped result in its wrapped field: 250 + 10 on a u8 wraps to 4.
Keep several error types shows how to
handle each error instead of passing it on.
For local recovery, use checked:. Its final expression becomes the
successful value of the block’s Result; no explicit Ok is needed there:
def increment_or_zero(value: i64) -> i64:
result = checked:
value + 1
match result:
case Ok(incremented):
incremented
case Err(_):
0
def main() -> Result[(), IoError]:
print(increment_or_zero(4), increment_or_zero(i64.MAX))?
Ok(())
5 0
Every named function and closure keeps its own declared error contract,
including when defined inside checked:. Calls returning Result still
need explicit handling.
Use unchecked: only where you can show the arithmetic cannot fail. It
yields an ordinary value, not a Result. A failure inside it terminates the
process by default, but --unchecked assume builds omit that check, so do not
use the suite to reject bad input. In assume builds, list, slice-view and
Bytes indexing inside the suite also requires nonnegative, in-range indices;
normal negative indexing is unavailable there. Keep these suites small and
justify every indexed access as well as the arithmetic. Write unchecked trap:
when termination is the intent. See
arithmetic modes.
Integer std.math helpers return ordinary
overflow results; their float variants remain infallible. Remaining std
implementations and parallel reductions still need the coordinated adoption
described in B14. Calls never inherit a caller’s mode.
Validate a divisor once
When a divisor comes from input, validate it with NonZero.try_from and pass
the resulting value to repeated divisions. A plain integer accepts that divisor
directly; the compiler omits its zero check.
def per_group(total: i64, groups: NonZero[i64]) -> Result[i64, OverflowError[i64]]:
Ok(total // groups)
def main() -> Result[(), IoError | RangeError | OverflowError[i64]]:
count = 3
groups = NonZero.try_from(count)?
print(per_group(14, groups)?, 14 % groups)?
Ok(())
4 2
The quotient and remainder are ordinary integers and can be zero. Signed
i64.MIN // -1 can still overflow, so the function’s error type includes
OverflowError[i64]. A locally handled checked block
can instead keep this error out of the function signature.
See nonzero integers
for construction and arithmetic rules.
Work with floating-point numbers
A decimal or exponent literal defaults to f64. Use an f32 suffix when you
want 32-bit arithmetic. Both operands must have the same type; casts are explicit:
def main() -> Result[(), IoError]:
distance: f64 = 7.5
time: f64 = 2.0
print(distance / time)?
print(1.25f32 + 0.5f32)?
print(2.5e-2)?
print(f64(3) / 2.0)?
print(i32(-2.75))?
Ok(())
3.75
1.75
0.025
1.5
-2
Float-to-integer casts discard the fractional part toward zero. The result must
fit the integer type: u8(255.9) is 255, while a 300.7, an infinity, or NaN
held in a variable stops the program with error: integer cast out of range.
Integer-to-float casts and f64 to f32 casts may round. There is no implicit widening or narrowing:
def main() -> ():
small: f32 = 1.5f64
expected f32, got f64
Write 1.5f32 or f32(1.5) instead. Floats follow IEEE arithmetic: division by
zero can produce infinity or NaN, and arithmetic overflow can produce infinity.
NaN is unequal to everything, including itself; ordered comparisons with it are
false.
def main() -> Result[(), IoError]:
zero = 0.0
print(1.0 / zero)?
unknown = zero / zero
print(unknown == unknown)?
print(unknown != unknown)?
Ok(())
inf
False
True
When a zero divisor is an error rather than a source of infinity,
x.div_checked(y) returns an Err holding a ZeroDivisionError for a zero y and
Ok(x / y) otherwise; see
recover from overflow.
Floats support +, -, *, /, unary signs, and comparisons.
// and % currently require integers. Floats cannot be dictionary keys or set elements. Float literals that overflow their declared width are compile errors.
Math functions are methods on a float, and each float type provides the
constants PI and E:
def main() -> Result[(), IoError]:
radius = 2.0
print(f64.PI * radius.pow(2.0))?
print((9.0).sqrt())?
print(f64.E.log())?
print((0.0).cos())?
print((2.5).floor())?
print((2.5).ceil())?
print((7.5).fmod(2.0))?
Ok(())
12.566370614359172
3.0
1.0
1.0
2.0
3.0
1.5
log is the natural logarithm, sin and cos take radians, and exp raises
e to a power. floor and ceil return floats; cast the result for an integer,
as in i64(x.floor()). fmod is the float remainder. These methods never stop
the program: (-1.0).sqrt() is NaN. An integer needs a cast first, as in
f64(n).sqrt().
Write text
The type str represents text. Single and double quotes both make strings;
triple quotes allow physical newlines. Escapes include \n, \r, \t,
escaped quotes, \\, \0 for an embedded NUL, and \u{H...} for any
character by its hexadecimal code point, such as "e\u{301}" for an e
followed by a combining accent. See escapes.
def main() -> Result[(), IoError]:
name = 'Plenty'
print(name)?
print(contains(name, "lent"))?
print(len("hé"))?
print("hé".byte_len())?
Ok(())
Plenty
True
2
3
print displays text without quotes. contains(text, part) tests for a
substring. len counts Unicode scalar values, so the accented character above
counts once; this is not a count of visible grapheme clusters. byte_len()
gives the size of the UTF-8 encoding instead, which is what a file, a buffer,
or a size limit measures.
Strings are immutable. A mut binding allows replacing a string, rather than
editing its bytes.
Compare strings in Unicode scalar order, inspect ASCII characters, or convert a single character to its code point:
def main() -> Result[(), AllocError | IoError | ParseError]:
print("apple" < "pear")?
print("abc".isalpha())?
print("123".isdigit())?
print(ord("é")?)?
print(chr(233)?)?
print("Hello".lower()?)?
print("Straße".upper()?)?
print(" one\t two\n".split()?)?
Ok(())
True
True
True
233
é
hello
STRASSE
["one", "two"]
ord requires exactly one Unicode scalar; chr takes a u32 code point and
rejects surrogates and values outside Unicode. Both return Result and allocate
nothing. Case conversion and splitting return fallible, independently owned
results. The ASCII predicates return false for empty or non-ASCII text.
Build text with f-strings
Put an f before a string to fill in values. Each {field} is replaced by
its value. An f-string builds a new str, which can fail to allocate, so it
returns Result[str, AllocError] like a list literal does: write ? after it.
def main() -> Result[(), AllocError | IoError | OverflowError[i64]]:
name = "Ada"
year = 1843
scores = [90, 85]?
print(f"{name} published in {year}."?)?
print(f"first score: {scores[0]}, count: {len(scores)}"?)?
print(f"all scores: {scores}, {{braces}}"?)?
print(f"total: {scores[0] + scores[1]}, passed: {scores[1] > 80}"?)?
print(f"{name} was {'young' if 1843 - 1815 < 30 else 'older'}"?)?
Ok(())
Ada published in 1843.
first score: 90, count: 2
all scores: [90, 85], {braces}
total: 175, passed: True
Ada was young
A field shows its value as print would. Double the braces for literal ones.
A field can be any expression. A ? inside a field returns from your
function, as ? does anywhere else; the ? after the f-string handles only
failure to build the text.
After a colon, a format spec chooses width, alignment, fill, sign, digit grouping, precision, and a type, as in Python:
def main() -> Result[(), AllocError | IoError]:
item = "tea"
price = 3.14159
count = 1234567
print(f"[{item:<6}] [{item:>6}] [{item:*^7}]"?)?
print(f"{price:.2f} {price:8.3f} {price:+.1e} {0.25:.0%}"?)?
print(f"{count:,} {count:_} {255:#x} {255:08b} {-42:05}"?)?
Ok(())
[tea ] [ tea] [**tea**]
3.14 3.142 +3.1e+00 25%
1,234,567 1_234_567 0xff 11111111 -0042
A spec can take its width or precision from a nested field, and its fill or
alignment from a nested str field. Such a value can be invalid, such as a
negative width, so the f-string’s error becomes AllocError | FormatError;
FormatError needs no import:
def main() -> Result[(), AllocError | IoError | FormatError]:
width = 8
digits = 2
align = "^"
print(f"[{3.14159:{width}.{digits}f}] [{'tea':*{align}{width}}]"?)?
Ok(())
[ 3.14] [**tea***]
!r shows the value’s representation, with quotes around text, and = shows
the field’s source before its value, which helps when debugging:
def main() -> Result[(), AllocError | IoError]:
name = "Ada"
year = 1843
print(f"{name!r} {name=} {year=}"?)?
Ok(())
"Ada" name="Ada" year=1843
!a is like !r but spells every non-ASCII character as an escape, so
f"{'café'!a}" gives "caf\u{e9}", which you can paste back into source as
the string literal "caf\u{e9}".
The compiler checks every spec against the field’s type. A precision does not apply to an integer:
def main() -> Result[(), AllocError | IoError]:
count = 3
print(f"{count:.2f}"?)?
Ok(())
format spec `.2f` does not apply to i64: integers take no precision; convert with f64(...) to format a float
The reference lists every option and conversion.
An f-string suits a fixed number of pieces of any type. A field whose type has
its own formatting methods also adds
FormatError, since that method can fail. Combine two strings with
a.concat(b)? and any number of strings from a list, set, or generator with
separator.join(pieces)?.
Define functions with clear interfaces
Every parameter and return type must be declared:
def double(value: i64) -> Result[i64, OverflowError[i64]]:
"""Return twice the supplied value."""
Ok(value * 2)
def main() -> Result[(), IoError | OverflowError[i64]]:
print(double(21)?)?
Ok(())
42
value: i64 declares the parameter type. The return type
Result[i64, OverflowError[i64]] describes either a doubled integer or overflow.
Ok(value * 2) supplies success; the multiplication returns an error from
double if it cannot fit. You can also write return Ok(value * 2).
The caller uses double(21)? to obtain the successful integer or pass the error
onward. Parameters are immutable. See recoverable errors
for handling either outcome explicitly.
Indent statements in the same block equally, using spaces. Tabs are rejected.
A string at the beginning of a function is its documentation. Triple quotes
allow a docstring to span lines. When a function should immediately return a
string, write return "text" so it cannot be mistaken for documentation.
Functions are known throughout the file, so one function may call a function declared later. Name arguments when that makes their meaning clearer:
def rectangle(width: i64, height: i64 = 2) -> Result[i64, OverflowError[i64]]:
Ok(width * height)
def main() -> Result[(), IoError | OverflowError[i64]]:
print(rectangle(height=3, width=4)?)?
print(rectangle(4)?)?
Ok(())
12
8
Positional arguments come first. After the first named argument, all remaining arguments must be named; their order need not match the declaration. Supplied expressions run in the order you write them. Required parameters come before parameters with defaults.
Defaults run afresh for each omitted argument. Mutable defaults never share storage between calls:
def count(values: &mut list[i64] = []?) -> Result[i64, AllocError]:
values.append(7)?
Ok(len(values))
def main() -> Result[(), AllocError | IoError]:
print(count()?)?
print(count()?)?
mut saved = [1, 2]?
print(count(values=&mut saved)?)?
print(len(saved))?
Ok(())
1
1
3
3
An omitted borrowed argument gets temporary storage that lasts for the call.
Pass your own buffer to keep its changes. Defaults run after supplied arguments,
in declaration order; they cannot refer to another parameter. Allocation still
needs ? and a compatible Result return type. If a default fails, that call
returns the error without entering the body. Callable values and builtins keep
positional arguments; splat unpacking is not supported.
Complete work without returning data
A function can complete without producing data. Its return type is the unit
type, (). This is different from a missing value.
def show_if(enabled: bool, value: i64) -> Result[(), IoError]:
if not enabled:
return Ok(())
print(value)?
Ok(())
def main() -> Result[(), IoError]:
show_if(False, 10)?
show_if(True, 42)?
Ok(())
42
Result[(), IoError] combines unit success with a possible output error.
print(value)? therefore completes without supplying data.
In a function declared -> (), a bare return returns unit. Use pass
for an intentionally empty block. Unit parameters and standalone unit bindings
are not supported. Use Option for absence; there is no None value.
Make decisions that produce values
A condition must have type bool. Comparisons such as ==, !=, <, <=,
>, and >= produce booleans. There is no implicit conversion of zero, an
empty string, or another value to False.
An if can supply a function’s result:
def maximum(first: i64, second: i64) -> i64:
if first > second:
first
else:
second
def main() -> Result[(), IoError]:
print(maximum(7, 12))?
Ok(())
12
Both continuing branches need the same type. Use elif for additional cases.
For a short choice, use a conditional expression:
def main() -> Result[(), IoError]:
age = 20
category = "adult" if age >= 18 else "child"
print(category)?
Ok(())
adult
and and or short-circuit: the right side is evaluated only when needed.
not negates a boolean. All three require boolean operands:
def main() -> Result[(), IoError | ZeroDivisionError | OverflowError[i64]]:
divisor = 0
safe = divisor != 0 and 10 // divisor > 1
print(safe)?
print(not safe)?
Ok(())
False
True
The division is skipped here. Chained comparisons such as 0 < x < 10 are
not supported yet; write 0 < x and x < 10.
Return early and continue on the other path
A guard clause handles a special case before the main computation:
def clamp_low(value: i64, minimum: i64) -> i64:
if value < minimum:
return minimum
value
def main() -> Result[(), IoError]:
print(clamp_low(3, 10))?
print(clamp_low(12, 10))?
Ok(())
10
12
return exits the entire function, even when it appears in nested branches.
Every return must match the function’s declared result. Code after a guaranteed
exit is rejected. Every path that reaches the end must also supply the declared
result; handling just one branch is not enough:
def incomplete(flag: bool) -> i64:
if flag:
return 42
def main() -> ():
pass
expected i64, got ()
Keep branch-local names local
Assignments to an existing mutable binding survive a branch. New names declared inside a branch belong to that branch:
def main() -> Result[(), IoError | OverflowError[i64]]:
mut price = 100
discounted = True
if discounted:
discount = 20
price = price - discount
print(price)?
Ok(())
80
Declare a mutable binding before the branch when subsequent code needs the updated value. For a new selected value, bind an inline conditional when each branch is one expression. A block conditional also supports branch-local work:
def main() -> Result[(), IoError | OverflowError[i64]]:
discounted = True
price = if discounted:
discount = 20
100 - discount
else:
100
print(price)?
Ok(())
80
Only price survives the block; discount still belongs to its branch. This
keeps price immutable when it is selected once. The block must be the whole
right-hand side; see conditional values.
Repeat with a range
A for loop repeats its body for each integer in a range:
def main() -> Result[(), IoError | OverflowError[i64]]:
mut total = 0
for n in range(1, 5):
total = total + n
print(total)?
for n in range(5, 0, -2):
print(n)?
Ok(())
10
5
3
1
range(stop) starts at zero. range(start, stop) selects the start.
The stop is always excluded: range(1, 5) yields 1, 2, 3, and 4.
A third argument selects the step. A negative step counts down. A zero step
would never reach the stop, so the step has type
NonZero[i64]:
an integer that cannot be zero. A literal step converts, and range(5, 0, 0)
is a compile-time error. Convert a step computed while the program runs with
NonZero.try_from, which works like
u8.try_from and returns
Err(RangeError.OutOfRange) for zero:
def main() -> Result[(), RangeError | IoError]:
stride = 3
step = NonZero.try_from(stride)?
for n in range(0, 10, step):
print(n)?
Ok(())
0
3
6
9
Passing stride directly is a compile-time error whose message shows this
conversion. step.get() reads the integer back.
The loop evaluates the range once. Each iteration binds n to the next
integer. Names declared in the body stay there; updates to an enclosing
mut binding persist. A loop completes with unit.
Creating and iterating a range do not allocate a list.
Ranges default to i64. Write range[u8](8) to choose another integer type.
Start and stop must fit that type, even though stop is excluded.
The step stays a signed NonZero[i64], so an unsigned range can count down.
A configured count that must be greater than zero can use
Positive[i64].
Construct positive_count = Positive.try_from(count)? once;
NonZero.from(positive_count) then
uses that positive domain value as a range step without another check.
NonNegative[i64] similarly states that a value may be zero but not negative.
Some standard-library parameters also have a documented ceiling. A direct call with a literal above that ceiling is a compile error; computed values still need to satisfy it when the operation runs. A positive domain checks the lower bound, not every operation’s ceiling. See invalid std arguments.
import std.sketch
def main() -> ():
drop(std.sketch.quantiles(1048577))
above its fixed ceiling of 1048576
Repeat until a condition changes
Use while when a condition determines how long to repeat. The condition must
be bool, and it is checked before every iteration, including the first:
def main() -> Result[(), IoError | OverflowError[i64]]:
mut remaining = 3
while remaining > 0:
print(remaining)?
remaining = remaining - 1
print("go")?
Ok(())
3
2
1
go
continue skips the rest of the current iteration. break exits the loop:
def main() -> Result[(), IoError | OverflowError[i64] | ZeroDivisionError]:
mut n = 0
while True:
n = n + 1
if n % 2 == 0:
continue
if n > 5:
break
print(n)?
Ok(())
1
3
5
Update the condition’s inputs before a continue when needed; otherwise a
while loop can repeat forever. In a for loop, continue automatically
advances to the next element:
def main() -> Result[(), IoError]:
for n in range(6):
if n == 1:
continue
if n == 4:
break
print(n)?
Ok(())
0
2
3
break and continue affect only the innermost loop. Use return to leave
the function. break cannot carry a value, and loop else clauses are not
supported.
Even after while True, a function returning data still needs a result after
the loop.
Statements directly after an unconditional exit are rejected:
def main() -> Result[(), IoError]:
while True:
break
print("unreachable")?
Ok(())
unreachable statement after a control-flow exit
Give types names of your own
There is no built-in int. If you want that name, choose its meaning explicitly:
type int = i32
def increment(value: int) -> Result[int, OverflowError[int]]:
Ok(value + int(1))
def main() -> Result[(), IoError | OverflowError[i32]]:
answer: int = increment(int(41))?
print(answer)?
Ok(())
42
type int = i32 makes int another name for i32, including in casts.
Aliases do not change unconstrained literal defaults, but an annotation such
as answer: int = 41 selects the literal’s width. Aliases are not literal
suffixes: write 41i32, not 41int.
Aliases can describe a value’s purpose and refer to other aliases:
type Count = u32
type ItemCount = Count
def add_one(count: ItemCount) -> Result[ItemCount, OverflowError[ItemCount]]:
Ok(count + 1u32)
def main() -> Result[(), IoError | OverflowError[u32]]:
print(add_one(41u32)?)?
Ok(())
42
Count, ItemCount, and u32 are interchangeable. An alias does not create
a distinct type or enforce units of measurement.
Declare aliases at module scope; they are visible throughout the file. Alias chains must reach a concrete type: cycles and unknown targets are errors. An alias cannot redefine a built-in name, another alias, or a function name.
Absence and failure
A calculation may have no answer, or it may fail with an explanation. Make those possibilities part of its type, and let the caller choose what to do.
- Represent absence
- Represent a recoverable error
- Supply a default or a typed error
- Give constructors enough type information
- Propagate with
? - Choose which error details to keep
- Keep several error types
- Build text with recoverable allocation
Represent absence
Use Option[T] when a function may have no value to return. It has two
variants: Some(value) holds a value of type T; Nothing has no payload.
def positive(n: i64) -> Option[i64]:
Some(n) if n > 0 else Nothing
def show(n: i64) -> Result[(), IoError]:
match positive(n):
case Some(value):
print(value)?
case Nothing:
print("not found")?
Ok(())
def main() -> Result[(), IoError]:
show(-1)?
show(42)?
Ok(())
not found
42
In case Some(value), the pattern binds the payload to value within that
branch. A match must cover both Some and Nothing.
Some and Nothing are built in and need no type prefix. Nothing
belongs to one concrete option type rather than acting as a universal null.
A present zero, False, or empty string is still Some(value).
When only presence matters, ask with is_some() or is_none() instead of
writing a match. Both borrow the option, so it is still usable afterwards.
def main() -> Result[(), IoError]:
found: Option[str] = Some("key")
missing: Option[str] = Nothing
print(found.is_some(), found.is_none())?
print(missing.is_some(), missing.is_none())?
print(found.unwrap_or("default"))?
Ok(())
True False
False True
key
Represent a recoverable error
Use Result[T, E] when failure has an explanation. Ok(value) carries
success data of type T; Err(error) carries a cause of type E.
def divide(left: i64, right: i64) -> Result[i64, str]:
outcome = checked:
left // right
match outcome:
case Ok(value):
Ok(value)
case Err(error):
match error:
case ZeroDivisionError():
Err("division by zero")
case OverflowError(_):
Err("division overflow")
def show(left: i64, right: i64) -> Result[(), IoError]:
match divide(left, right):
case Ok(value):
print(value)?
case Err(message):
print(message)?
Ok(())
def main() -> Result[(), IoError]:
show(8, 2)?
show(8, 0)?
Ok(())
4
division by zero
The local checked: block
collects division failures. Matching the error distinguishes division by zero
from signed overflow (i64.MIN // -1), then gives each a message. This small
example deliberately uses strings as its error API; applications can preserve
those distinct typed errors instead.
The caller can recover and continue, as show does here. A result does not
implicitly unwrap itself or raise an exception.
When success has no data, use unit as the success type:
def validate(name: str) -> Result[(), str]:
if len(name) == 0:
return Err("name is empty")
Ok(())
def show(name: str) -> Result[(), IoError]:
match validate(name):
case Ok(_):
print("valid")?
case Err(message):
print(message)?
Ok(())
def main() -> Result[(), IoError]:
show("Plenty")?
show("")?
Ok(())
valid
name is empty
The pattern _ ignores a payload you do not need. Ok(()) represents success
without data; Nothing represents absence.
is_ok() and is_err() answer whether a result succeeded without looking at
its payload. They borrow the result, so it can still be matched or returned.
def divide(left: i64, right: i64) -> Result[i64, str]:
outcome = checked:
left // right
match outcome:
case Ok(value):
Ok(value)
case Err(error):
match error:
case ZeroDivisionError():
Err("division by zero")
case OverflowError(_):
Err("division overflow")
def main() -> Result[(), IoError]:
outcome = divide(8, 0)
print(outcome.is_ok(), outcome.is_err())?
print(divide(8, 2).is_ok())?
match outcome:
case Ok(value):
print(value)?
case Err(message):
print(message)?
Ok(())
False True
True
division by zero
Supply a default or a typed error
option.unwrap_or(default) returns a present payload or the supplied default.
dictionary.get(key, default) does the same for a dictionary lookup, while
dictionary.get(key) still returns an Option. Zero, False, and an empty
string count as present values.
def main() -> Result[(), AllocError | IoError]:
numbers = {"answer": 42}?
print(numbers.get("answer", 0))?
print(numbers.get("missing", 0))?
missing: Option[i64] = Nothing
print(missing.unwrap_or(9))?
Ok(())
42
0
9
option.ok_or(error) converts absence into a typed Result. Choose an error
that explains the missing value, then use ? inside a function returning a
compatible Result.
enum Missing:
Answer
def answer(numbers: &dict[str, i64]) -> Result[i64, Missing]:
value = numbers.get("answer").ok_or(Missing.Answer)?
Ok(value)
def main() -> Result[(), AllocError | IoError]:
numbers = {"answer": 42}?
match answer(&numbers):
case Ok(value):
print(value)?
case Err(_):
print("missing answer")?
Ok(())
42
These helpers add no allocation. They evaluate their arguments eagerly, even
when a value is present. The Option methods consume owned payloads and drop an
unused default or error. Dictionary lookup supports copyable values; use
pop to remove and take an owned value. Use match when the fallback should
run only on absence, or choose a lazy helper as described below. Bare Nothing may need an Option annotation or a typed
result context to determine its payload type.
Compute a default only when needed
Both Option and Result provide unwrap_or_else: it calls a fallback only
for Nothing or Err. A Result fallback receives the error. Option also has
ok_or_else for constructing an absence error lazily. These take function
values; functions as values
and closures explain
callbacks in more detail.
A fallback can itself fail. In that case, map_or_else lets both branches
return a Result, and ? handles that Result after selection:
def default_port() -> Result[i64, ParseError]:
i64.parse("8080")
def port(configured: Option[i64]) -> Result[i64, ParseError]:
Ok(configured.map_or_else(default_port, Ok)?)
def main() -> Result[(), ParseError | IoError]:
print(port(Some(9000))?, port(Nothing)?)?
Ok(())
9000 8080
default_port runs only for Nothing. Writing
configured.unwrap_or(default_port()?) would run it even for Some(9000).
A match is equally suitable when either branch needs several statements.
Result.unwrap_or(default) also exists for deliberate recovery with a cheap,
already available value. See defaults and recovery
for ownership, evaluation order, and the Result callback’s error argument.
Give constructors enough type information
A binding annotation, parameter, or return signature supplies missing type
information. Some(42) infers Option[i64] from its payload. Nothing needs
an option type from context; Ok and Err each need the other variant’s type:
def main() -> Result[(), IoError]:
found = Some(42)
missing: Option[i64] = Nothing
success: Result[i64, str] = Ok(42)
failure: Result[i64, str] = Err("not ready")
print(found == Some(42))?
print(missing == Option[i64].Nothing)?
match failure:
case Ok(value):
print(value)?
case Err(message):
print(message)?
Ok(())
True
True
not ready
Qualify a constructor, such as Option[i64].Nothing, to supply its type
explicitly. Otherwise, insufficient context is an error:
def main() -> ():
answer = Ok(42)
cannot infer `Ok`
Pass a failure back with ?
Put ? after a Result expression to extract its Ok payload or return its
Err immediately. The example’s main preserves allocation and printing errors
in AllocError | IoError; checked_name keeps its concrete str error.
def validate(name: str) -> Result[(), str]:
if len(name) == 0:
return Err("name is empty")
Ok(())
def checked_name(name: str) -> Result[str, str]:
validate(name)?
Ok(name)
def main() -> Result[(), AllocError | IoError]:
print(str.repr(checked_name("Plenty"))?)?
print(str.repr(checked_name(""))?)?
Ok(())
Result[str, str].Ok("Plenty")
Result[str, str].Err("name is empty")
validate(name)? stops checked_name on failure, skipping Ok(name). Error
propagation cleans up local values and temporaries, just as return does.
str.repr formats a result wrapper so either outcome can be printed.
This example displays both results for inspection; it does not propagate the
validation error out of main. In an application where validation must succeed,
use ? on checked_name and include str in the enclosing function’s error type.
Ordinary print requires a handled result:
print(checked_name("Plenty")?)? prints only the successful name.
An expression statement cannot silently discard a result:
def main() -> Result[(), AllocError | IoError]:
print("hello")
Ok(())
cannot implicitly discard a Result
Propagate with ?, match to recover, or use drop(result) to intentionally
ignore either outcome. The discard check does not cover unused bindings or
Option values.
? works with Option too: it extracts Some or immediately returns Nothing.
def positive(n: i64) -> Option[i64]:
Some(n) if n > 0 else Nothing
def same_positive(n: i64) -> Option[i64]:
Some(positive(n)?)
def main() -> Result[(), IoError]:
print(same_positive(42))?
print(same_positive(-1))?
Ok(())
Option[i64].Some(42)
Option[i64].Nothing
This example passes the positive value through unchanged to isolate optional
propagation. Arithmetic that might overflow needs a Result contract or local
checked recovery;
Option does not silently convert arithmetic failure to absence.
The enclosing function must return the same family: Result for a Result
operand or Option for an Option operand. Success payload types can differ,
but a Result error must either match the declared error type or fit within
its anonymous sum.
For example, Result[(), AllocError | IoError] lets the same function propagate
list allocation failures and printing failures with ?. The sum retains the
original error and adds no allocation. Use match when you want to recover or
translate an error rather than pass it back.
? is not supported inside generators.
def read_number() -> Result[i64, str]:
Err("not a number")
def checked() -> Result[i64, i64]:
Ok(read_number()?)
def main() -> ():
print(checked()).unwrap()
`?` operand error str cannot propagate as i64; the function's error type must include it
Option and Result wrappers do not allocate. Their payloads can:
Some([1, 2]?) allocates the list, with no additional allocation for Some.
Choose which error details to keep
Keep a concrete error type when the caller needs the cause. For example,
IoError preserves an input/output cause, while str can carry a message.
Pass an error back unchanged when the caller can use it directly. When several operations have different errors, the next lesson shows how to keep them together without losing their details.
Sometimes an API needs to add meaning to a lower-level cause. Here,
ConfigError.InvalidCount identifies what was being parsed and retains the
original ParseError. A Result’s map_error method changes just the error
type, forwarding success unchanged:
enum ConfigError:
InvalidCount(ParseError)
def read_count(text: &str) -> Result[i64, ConfigError]:
i64.parse(text).map_error(ConfigError.InvalidCount)
def main() -> Result[(), IoError]:
match read_count("many"):
case Ok(count):
print(count)?
case Err(ConfigError.InvalidCount(cause)):
print(cause == ParseError.Invalid)?
Ok(())
True
ConfigError.InvalidCount written without arguments is a function from
ParseError to ConfigError. map_error calls it once with the original
error, only when parsing fails. On success, the parsed count passes through
unchanged. The conversion policy is explicit, and mapping adds no allocation.
read_count returns that result directly because it has no more work to do.
Use ? when further work depends on the successful value. See the mapping reference
for callback evaluation and ownership details.
When the new error needs more than the cause, such as a second payload or captured context, pass an anonymous function that builds it instead.
The example’s main deliberately handles a failed parse so you can see that the
cause survived. In an application that cannot continue, propagate the error
instead. Returning an error from main reports it on standard error and exits
with status one; printing an error and returning Ok(()) reports success to the
program’s caller. The command-line application
puts this policy into practice.
Avoid replacing useful causes with builtin Failure: propagation into
Result[T, Failure] discards the original error and keeps only
Failure.Unspecified. It is a last resort, not a way to combine errors.
See the reference for its exact
erasure rules.
.unwrap() explicitly extracts a successful payload or terminates on
Err or Nothing, without normal scope cleanup:
def main() -> Result[(), IoError]:
found = Some(42)
print(found.unwrap())?
Ok(())
42
Choose .unwrap() only when termination is your intended policy.
Keep several error types
A sum keeps one complete value from a finite set of types. Use | in the
error type when a function combines operations with different errors:
type SmallNumberError = ParseError | RangeError
def parse_small(text: str) -> Result[u8, SmallNumberError]:
number = i64.parse(text)?
u8.try_from(number)
def main() -> Result[(), IoError]:
match parse_small("300"):
case Ok(number):
print(number)?
case Err(ParseError.Invalid):
print("not an integer")?
case Err(ParseError.OutOfRange):
print("outside i64 range")?
case Err(RangeError.OutOfRange):
print("outside u8 range")?
Ok(())
outside u8 range
? forwards the original parse error. The direct return widens the conversion
result to the declared set. Neither step creates a heap wrapper or loses the
payload. Each variant pattern selects its enum member and then its variant,
so the errors can be handled directly inside Err. Every variant of every
member needs coverage unless you use _.
Use case name: Type: when you need one whole member and its value. The
matching reference describes
how whole-member patterns and variant patterns combine.
Use a module-level type alias to
name a recurring group: SmallNumberError means exactly ParseError | RangeError. It introduces no wrapper or allocation, so ? and exhaustive
matching still work on the original members. An alias also composes with
other errors: SmallNumberError | IoError is the same sum as ParseError | RangeError | IoError.
Name the errors the function actually propagates. An operation may already wrap a lower-level error in its own enum; that does not require adding the lower-level type separately. A new wrapper enum is useful when an API wants to hide or reorganize its causes, but naming a group needs only an alias.
A checked: block
returns one of these sums. Its errors are classes, values with named fields,
and a class pattern matches one and binds its fields in order:
def report(total: u8, count: u8) -> Result[(), IoError]:
result = checked:
rounded = total + count // 2
rounded // count
match result:
case Ok(mean):
print(mean)?
case Err(OverflowError(wrapped)):
print("overflow, wrapped to", wrapped)?
case Err(ZeroDivisionError()):
print("no values")?
Ok(())
def main() -> Result[(), IoError]:
report(200, 4)?
report(250, 20)?
report(250, 0)?
Ok(())
50
overflow, wrapped to 4
no values
OverflowError(wrapped) selects the overflow member and binds its one field.
ZeroDivisionError() has no fields, so its parentheses are empty. Write _
for a field you do not need.
Classes explains fields, and
class patterns
gives the complete rules.
Sums also work outside errors:
def describe(value: i64 | str) -> Result[(), IoError]:
match value:
case number: i64:
print(number)?
case text: str:
print(text)?
Ok(())
def main() -> Result[(), AllocError | IoError]:
describe(42)?
describe("ready")?
Ok(())
42
ready
The parameter supplies the destination type, so either member can be passed
directly. str | i64 means the same type as i64 | str; duplicate members
collapse. To inspect without moving an owned payload, match &value; use
&mut value to change the active payload through a mutable borrow.
Heterogeneous collections uses sums and enum wrappers to store different kinds of values in lists and dictionaries.
Keep stream errors and progress
Use std.io.typed when a transfer needs the original cause and the number of
bytes delivered before failure. Its copy error distinguishes reading from
writing. Existing std.io streams can enter this API through bridges:
import std.http.typed
import std.bytes
import std.io
import std.io.typed
from std.http.typed import fixed_body
from std.io.typed import CopyFailure, copy_all
def main() -> Result[(), AllocError | IoError | std.http.typed.FramingError[std.io.StreamError] | std.bytes.BytesError]:
mut bytes = std.bytes.with_capacity(3)?
bytes.resize(3, 65u8)?
mut scratch = std.bytes.with_capacity(2)?
scratch.resize(2, 0u8)?
mut body = fixed_body(std.io.typed.reader(std.io.cursor(bytes)), 3)?
mut output = std.io.typed.writer(std.io.cursor(std.bytes.with_capacity(3)?))
match copy_all(&mut body, &mut output, &mut scratch):
case Ok(count):
print(count)?
case Err(CopyFailure.Read(count, cause)):
print("read failed after")?
print(count)?
drop(cause)
case Err(CopyFailure.Write(count, cause)):
print("write failed after")?
print(count)?
drop(cause)
cursor = std.io.typed.unwrap_writer(output)
print(cursor.position())?
Ok(())
3
3
The fixed body assumes an already validated Content-Length; it does not parse
HTTP. close_body instead finishes at EOF and requires a connection that will
never be reused. The copy helper retries explicit interruption and preserves
other transport causes. A write failure can leave unwritten data from the last
chunk in scratch; the delivered count alone is not a resume token.
For an HTTP message, use typed head parsing followed by request_body or
response_body. They check framing fields and select the body decoder with
explicit limits. This in-memory response uses chunked framing. Keep the entire
buffered reader when handing it to the body, so prefetched bytes are preserved:
import std.bytes
import std.http
import std.io
import std.io.typed
from std.http.typed import FramingError, read_response, response_body
from std.io.typed import CopyFailure, Partial, ReadFailure, copy_all
type TransportCause = ReadFailure[std.io.StreamError]
type HttpCause = FramingError[TransportCause]
def main() -> Result[(), AllocError | IoError | std.bytes.BytesError | std.io.StreamError | std.http.HttpError | Partial[HttpCause] | HttpCause | CopyFailure[HttpCause, std.io.StreamError]]:
wire = std.bytes.from_text("HTTP/1.1 200 OK\r\nTransfer-Encoding: chunked\r\n\r\n3\r\nABC\r\n0\r\nX-Tag: sample\r\n\r\n")?
limits = std.http.Limits(64, 128, 4, 1024)
mut conn = std.io.typed.buffered_reader(std.io.typed.reader(std.io.cursor(wire)), 128)?
head = read_response(&mut conn, &limits)?.ok_or(std.http.HttpError.Truncated)?
# Our GET request allows keep-alive; the response must also allow it.
mut body = response_body(conn, &head, "GET", True, &limits)?
mut output = std.io.typed.writer(std.io.cursor(std.bytes.with_capacity(3)?))
mut scratch = std.bytes.with_capacity(2)?
scratch.resize(2, 0u8)?
print(copy_all(&mut body, &mut output, &mut scratch)?)?
print(body.is_done(), body.trailers()[0].value)?
# finish checks completion and persistence before handing back ownership.
recovered = body.finish()?
drop(recovered)
Ok(())
3
True sample
Only decoded payload contributes to the copy count. The decoder keeps partial
framing state across interruption and stops after the trailers; it does not
parse an HTTP head. The parsed-head constructor validates framing; finish
checks whether this completed message permits recovering its connection.
Dropping an unfinished or failed body drops its connection. Close-delimited,
closing, upgraded and successful CONNECT connections cannot be recovered by
this API. The reference describes the
remaining caller responsibilities and the lower-level chunked_body adapter.
For real requests, use std.http.client,
which uses these typed paths and also handles deadlines, connection ownership
and reuse. typed_response(response) adapts its response to the typed reader
protocol while preserving ClientError; unwrap it before client.finish.
response.read_all_progress is convenient when collecting a body and retaining
both its original error and the number of bytes appended before failure.
See the typed streams reference for nested error payloads and the framing reference for terminal-state rules.
Build text with recoverable allocation
Concatenating strings can need new storage, so + returns
Result[str, AllocError]. Handle that result before using the new text:
def main() -> Result[(), AllocError | IoError]:
message = ('Hello, ' + "Plenty!")?
print(message)?
print(contains(message, "Plenty"))?
Ok(())
Hello, Plenty!
True
Building the string can fail with AllocError; printing can fail with
IoError. The AllocError | IoError sum lets this function propagate either
cause while keeping its original type and details.
Use str.from(value) to convert a scalar to text, or str.repr(value)
for a representation such as quoted text or a displayed result wrapper.
Both return allocation results; see the text conversion lessons.
A helper can combine construction and printing behind one result:
def greet(name: str) -> Result[(), AllocError | IoError]:
print(("Hello, " + name)?)?
Ok(())
def greet_if(enabled: bool, name: str) -> Result[(), AllocError | IoError]:
if not enabled:
return Ok(())
greet(name)
def main() -> Result[(), AllocError | IoError]:
greet_if(False, "Ada")?
greet_if(True, "Ada")?
Ok(())
Hello, Ada
String-building operations return allocation results even when a particular
short string fits inline. A slice such as
name[1:]? is another: it copies part of the text.
Collections, ownership, and borrowing
Store values in collections and choose when to transfer, copy, or borrow them.
- Build and update a list
- Transfer or copy ownership
- Take part of a list or text
- Borrow a collection
- Borrow an element
- Handle allocation failures
- Associate keys with values Optional lookup handles missing entries explicitly.
- Keep distinct values in a set
- Remove values and take ownership
- Group different types in tuples
- Sort and reorder a list
- Clean up owned values
- Build singly and doubly linked lists
The collection examples provide additional operations.
Build and update a list
A list contains values of one type. Use list[T] in signatures and annotations.
An empty list needs an annotation or a typed constructor such as list[i64]().
def main() -> Result[(), AllocError | IoError]:
mut original: list[i64] = [10, 20]?
mut changed = copy(original)?
changed.append(30)?
changed[0] = 99
print(original)?
print(changed)?
print(changed[-1])?
print(len(changed))?
Ok(())
[10, 20]
[99, 20, 30]
30
3
copy(original)? makes an independent list. Updates happen in place and
require a mut owner. append grows the list; indexed assignment replaces
an existing element. Negative indices count from the end, and an invalid index
is a runtime error.
List construction, copying, and growth return allocation
results, handled here with ?.
Collections can nest. Update an inner list directly through its indices:
def main() -> Result[(), AllocError | IoError | OverflowError[i64]]:
mut rows = [[1, 2]?, [3]?]?
rows[0][-1] += 7
print(rows)?
Ok(())
[[1, 9], [3]]
Nested assignment does not copy or allocate the containing lists.
Numeric elements support augmented assignment such as += too. Each target
index runs once before the right side, which may read the list but cannot change
or resize it during the update. A failed right-side result leaves the element
unchanged.
Indices can be expressions, including elements of another list:
def main() -> Result[(), AllocError | IoError]:
mut values = [30, 10, 20]?
order = [1, 2, 0]?
print(values[order[0]])?
values[order[2]] = 31
print(values)?
Ok(())
10
[31, 10, 20]
Transfer or copy ownership
A list has one owner responsible for its mutable contents. Assigning it to another binding transfers that responsibility:
def main() -> Result[(), AllocError | IoError]:
original = [10, 20]?
mut changed = original
changed.append(30)?
print(changed)?
Ok(())
[10, 20, 30]
The assignment moves the list. The old binding cannot be used afterward:
def main() -> Result[(), AllocError | IoError]:
original = [10, 20]?
changed = original
print(original)?
Ok(())
use of moved binding `original`
Passing a collection to an owned parameter or returning it also transfers ownership. A move transfers the existing storage rather than duplicating it.
Use copy(value)? when you need independent contents. Copying observes the
source and returns Result[T, AllocError], so the source remains usable
whether copying succeeds or fails. Nested mutable contents are copied too;
immutable strings may share their existing storage.
Numbers and booleans copy on assignment. Strings remain usable after assignment because their storage is immutable. Tuples containing only these values copy; tuples containing collections move.
A value that copies needs no copy(...)?: write name = original for a
string. plenty --check warns
about copy(x)? when the type of x already copies. Use copy for values
that move, such as lists, when you need two independent owners.
A mutable owner can receive a new value after moving its previous one.
The compiler checks all continuing paths: a move on one branch can cause a
use of possibly moved binding diagnostic later.
Take part of a list or text
A slice copies part of a list or string into a new value. Write the start and
stop positions inside the brackets, separated by :. The start is included and
the stop is excluded:
def main() -> Result[(), AllocError | IoError]:
scores = [70, 85, 90, 65, 88]?
print(scores[1:3]?)?
print(scores[:2]?)?
print(scores[3:]?)?
print(scores[-2:]?)?
print(scores[::-1]?)?
print(scores)?
Ok(())
[85, 90]
[70, 85]
[65, 88]
[65, 88]
[88, 65, 90, 85, 70]
[70, 85, 90, 65, 88]
An omitted start or stop runs to that end of the list. A negative position
counts from the end, so scores[-2:] takes the last two elements. A third
number is the step: [::2] takes every second element and [::-1] walks
backwards. The source list is unchanged.
A slice allocates its new list, so it returns a Result like a
list literal and needs ?:
def main() -> Result[(), AllocError | IoError]:
scores = [70, 85, 90]?
top: list[i64] = scores[:2]
Ok(())
slice returns Result[list[i64], AllocError]; use `?` to propagate allocation failure
Positions outside the list do not stop the program. They clamp to its ends, as
in Python, so scores[3:100] and scores[100:] are fine; the second is empty.
Slices are copies
A slice is an explicit copy. Each selected element is copied, including nested lists, so changing the source afterwards does not change the slice:
def main() -> Result[(), AllocError | IoError]:
mut rows = [[1, 2]?, [3]?]?
first = rows[:1]?
rows[0].append(99)?
print(first)?
print(rows)?
Ok(())
[[1, 2]]
[[1, 2, 99], [3]]
Copying costs time and memory in proportion to the selected elements. To read
elements without copying them, borrow them as a slice view, &scores[1:], which
the element borrowing lesson
introduces. A loop over their positions also avoids the copy:
def main() -> Result[(), AllocError | IoError | OverflowError[i64]]:
scores = [70, 85, 90, 65, 88]?
mut total = 0
for i in range(1, len(scores)):
total += scores[i]
print(total)?
Ok(())
328
Text
String slices count Unicode characters, the same positions as len and
indexing, so a slice never splits a character:
def main() -> Result[(), AllocError | IoError]:
word = "café!"
print(word[:4]?)?
print(word[-1:]?)?
print(word[::-1]?)?
Ok(())
café
!
!éfac
To read part of a string without copying it, borrow it as &word[:4], which
the element borrowing lesson
introduces.
The step must not be zero. A literal 0 is a compile error, and a computed step
is converted with NonZero.try_from(step)?, which reports zero as an error. The
slice reference covers steps,
tuples, bytes, columnar lists, and views.
Borrow instead of transferring ownership
A shared reference, &T, permits observation. An exclusive reference, &mut T,
permits mutation. The owner remains responsible for cleanup. Function signatures
state which access is needed:
def first(values: &list[i64]) -> i64:
values[0]
def add(values: &mut list[i64], value: i64) -> Result[(), AllocError]:
values.append(value)?
Ok(())
def main() -> Result[(), AllocError | IoError]:
mut numbers = [1, 2]?
print(first(numbers))?
add(&mut numbers, 3)?
print(numbers)?
Ok(())
1
[1, 2, 3]
Collection operations such as append, indexing, and len work directly
through references.
The parameter determines borrowing: first(numbers) implicitly lends numbers
because first expects &list[i64]. Writing first(&numbers) also works.
Use explicit &mut for mutable arguments, as in add(&mut numbers, 3).
An owned parameter still takes ownership.
References can also be local bindings. &mut permits changing the target,
not reassigning the reference binding. Use *reference to read or replace a
scalar target:
def main() -> Result[(), IoError | OverflowError[i64]]:
mut score = 10
reference = &mut score
*reference = *reference + 5
print(*reference)?
score = 20
print(score)?
Ok(())
15
20
The exclusive borrow ends after the last use of reference, so assigning to
score afterward is allowed. A reference that will be used later keeps its loan
live, including across branches and loop iterations:
def main() -> Result[(), AllocError | IoError]:
mut numbers = [1, 2]?
view = &numbers
numbers.append(3)?
print(view)?
Ok(())
conflicting borrow: cannot modify or exclusively borrow `numbers` while it is borrowed
The diagnostic notes identify the borrow’s creation and the later use that keeps it live.
An exclusive reference can temporarily lend shared or exclusive access. Conflicting access through it is suspended until the child borrow’s last use. Passing an existing reference to a reference parameter reborrows it; the owner is not transferred.
When a conflict goes through a reference, the diagnostic names both the owner and the reference written at the failing use:
def main() -> Result[(), AllocError | IoError]:
mut numbers = [1, 2]?
reference = &mut numbers
view = &reference
reference.append(3)?
print(view)?
Ok(())
conflicting borrow: cannot modify or exclusively borrow `numbers` (through `reference`) while it is borrowed
Borrowing collection elements
Borrow an element when you want to inspect or change it without moving its owner:
def main() -> Result[(), AllocError | IoError]:
mut counts = [1, 5]?
value = &mut counts[0]
*value = 2
print(counts)?
Ok(())
[2, 5]
The loan ends after its last use. While an element reference is live, the
collection cannot grow, remove entries, or move. An out-of-range index traps.
Dictionaries lend a value through
get_mut, which returns an Option instead of trapping on
a missing key.
Two list elements written as different number literals are different places, so both can be borrowed at once:
def main() -> Result[(), AllocError | IoError | OverflowError[i64]]:
mut scores = [10, 20, 30]?
first = &mut scores[0]
last = &mut scores[2]
*first = *first + *last
*last = 0
print(scores)?
Ok(())
[40, 20, 0]
The compiler proves only this literal case. Any other index, such as i or
-1, may name the same element, so its loan covers the whole list:
def main() -> Result[(), AllocError | IoError | OverflowError[i64]]:
mut scores = [10, 20, 30]?
i = 2
first = &mut scores[0]
last = &mut scores[i]
*first = *first + *last
Ok(())
while `scores[0]` is exclusively borrowed
Dictionary entries are never separated this way. To exchange two list elements
at computed indices, swap them instead.
Borrowed loops always give references, including when the elements are numbers.
Printing and comparison inspect those references directly. Use * when a
function takes a value or a new collection needs the element itself:
def count_eq[T](values: &list[T], target: &T) -> Result[i64, OverflowError[i64]]:
mut count = 0
for value in values:
if value == target:
count = count + 1
Ok(count)
def main() -> Result[(), AllocError | IoError | OverflowError[i64]]:
values = [1, 2, 1]?
target = 1
print(count_eq(&values, &target)?)?
for value in &values:
print(*value)?
Ok(())
2
1
2
1
The same body works for a list[str]. Shared dictionary items() loops likewise
give &K keys and &V values; mutable views give &mut V values. A borrowed
collection stays protected against changes throughout its loop.
Borrow part of a list
&values[a:b] borrows a run of elements in place. This slice view copies
nothing, so unlike a copying slice it needs no ?. A function
takes a view as &[T], or as &mut [T] to change the elements:
def total(values: &[i64]) -> Result[i64, OverflowError[i64]]:
mut sum = 0
for value in values:
sum += *value
Ok(sum)
def reset(values: &mut [i64]) -> ():
for value in values:
*value = 0
def main() -> Result[(), AllocError | IoError | OverflowError[i64]]:
mut scores = [70, 85, 90, 65, 88]?
print(total(&scores)?, total(&scores[1:])?)?
reset(&mut scores[:2])
print(scores)?
Ok(())
398 328
[0, 0, 90, 65, 88]
The positions clamp as a copying slice’s do. A view supports len, indexing,
narrower views, loops, and == with another view, and prints as a list of its
elements. An exclusive view also takes indexed assignment and
swap. A &[T] parameter also takes a whole list as &scores, which is
a view that copies nothing. A view borrows its
run of the list, and the list cannot grow while the view is live. Two exclusive
views of one list can be live together when one stops where the other starts:
def main() -> Result[(), AllocError | IoError | ZeroDivisionError | OverflowError[i64]]:
mut scores = [70, 85, 90, 65, 88]?
half = len(scores) // 2
low = &mut scores[:half]
high = &mut scores[half:]
low[0] = high[0]
print(scores)?
Ok(())
[90, 85, 90, 65, 88]
Views whose ranges may overlap, such as &mut scores[:3] and &scores[2:],
conflict. Byte buffers have views too. A view cannot
be returned or stored yet; the
slice reference
lists its operations and limits.
Borrow part of a string
&text[a:b] borrows part of a string the same way. Its type is &substr, and
its positions count characters, as a copying string slice’s
do. Loop over it, compare it with a string, or search it without copying:
def count(text: &substr, letter: &str) -> Result[i64, OverflowError[i64]]:
mut n = 0
for c in text:
if c == letter:
n += 1
Ok(n)
def main() -> Result[(), AllocError | IoError | OverflowError[i64]]:
line = "name: Ada Lovelace"
value = &line[6:]
print(value, len(value))?
print(&line[:4] == "name", "Love" in value, count(&line, "a")?)?
first = value[:3]?
print(first)?
Ok(())
Ada Lovelace 12
True True 3
Ada
A function takes a substring as &substr, and a whole string as &line or
a literal. Copy the text with value[a:b]? when you need an owned str,
for example to call a str method or keep it after the string changes. The
substring view reference
lists every operation.
Handle allocation failures
Collection construction returns Result[collection, AllocError]. Use ? to
propagate a failure, or match to recover:
def answers() -> Result[list[i64], AllocError]:
mut values = list[i64].with_capacity(2)?
values.append(21)?
values.append(42)?
Ok(values)
def main() -> Result[(), IoError]:
match answers():
case Ok(values):
print(values)?
case Err(AllocError.OutOfMemory):
print("not enough memory")?
case Err(AllocError.CapacityOverflow):
print("requested capacity is too large")?
Ok(())
[21, 42]
list[i64].with_capacity(2)? constructs an empty list with room for at least
two elements. Appending those elements needs no further list storage allocation.
Use list[i64]() when you do not know the capacity ahead of time.
dict[str, i64]() and set[str]()
construct the other collection kinds. Displays such as [21, 42] return the
same kind of allocation result; extract their collection with ? too.
List displays reserve their full element count automatically. Both mutable and
immutable list bindings benefit. Element expressions may still allocate.
For an existing mutable collection, values.reserve(n)? reserves room for
n additional elements beyond its current length. Existing contents remain
unchanged if reservation fails.
Growth also returns a result: append for lists, insert for dictionaries,
and add for sets. A failed insertion keeps existing contents but consumes
its arguments, cleaning up any owned argument rather than returning it.
AllocError has two payload-free variants. OutOfMemory means the
allocator rejected a request. CapacityOverflow means its size cannot be
represented. Neither variant needs allocation.
Use capacity and reservation to obtain storage before inserting values you want to keep if reservation fails.
Associate keys with values
A dict[K, V] maps keys of one type to values of another. Keys may be integers,
booleans, strings, bytes, or tuples of supported keys; values can include collections.
Named enums can also be keys when their payloads are supported key types; see
heterogeneous dictionaries.
The keys() and values() loops below borrow their elements: *name reads
the value behind a reference. The borrowing lesson
explains references in detail.
def main() -> Result[(), AllocError | IoError]:
mut scores: dict[str, i64] = {"Ada": 10, "Grace": 20}?
scores.insert("Ada", 12)?
scores.insert("Lin", 30)?
print(scores["Ada"])?
print("Grace" in scores)?
for name in scores.keys():
print(*name)?
for score in scores.values():
print(*score)?
Ok(())
12
True
Ada
Grace
Lin
12
20
30
insert adds a key or replaces the value of an existing one, which keeps its
insertion position. Adding a key can allocate, so insert returns an
allocation result. Dictionaries have no indexed assignment: scores["Ada"] = 12
is a compile error. Reading scores["Ada"] traps when the key is missing.
To change a value in place, get_mut returns Option[&mut V]: a mutable
reference when the key is present, or Nothing:
def main() -> Result[(), AllocError | IoError]:
mut groups = {"first": [10, 20]?}?
match groups.get_mut("first"):
case Some(group):
group[0] = 11
group.append(30)?
case Nothing:
pass
print(groups)?
Ok(())
{"first": [11, 20, 30]}
get_mut allocates nothing. The reference borrows the whole dictionary
exclusively, so the dictionary cannot be read or changed while it is live.
To change every value, loop over (&mut groups).values() instead of calling
get_mut per key; dictionary key/value loops
show the mutable form.
To count or group, a missing key needs a starting value. get_or_insert(key, default) inserts the default when the key is absent, then returns a mutable
reference to the stored value. It finds the key with one lookup. Inserting can
allocate, so it returns a Result; ? propagates the AllocError:
def main() -> Result[(), AllocError | IoError | OverflowError[i64]]:
mut counts: dict[str, i64] = {}?
for word in ["red", "blue", "red"]?:
*counts.get_or_insert(word, 0)? += 1
print(counts)?
Ok(())
{"red": 2, "blue": 1}
The default is built before the lookup, even when the key is present. A
default like 0 costs nothing; one that allocates, such as []?, allocates on
every call, so use get_mut when the key is usually present.
keys() and values() borrow elements in insertion order without allocating.
The references protect the dictionary from mutation while they are used.
Keep a key view in a local when checking several properties:
def main() -> Result[(), AllocError | IoError]:
scores = {"Ada": 12, "Grace": 20}?
names = scores.keys()
expected = {"Ada", "Grace", "Lin"}?
print(len(names), "Ada" in names, names.issubset(&expected))?
print(names.union(&expected)? == expected)?
Ok(())
2 True True
True
Read-only checks return plain values. Set algebra and names.to_set() allocate
an owned set and return an allocation result. The view stays local: use
scores.keys_snapshot()? or scores.values_snapshot()? when an owned ordered
list must outlive the borrow. For mutable payloads,
copy(groups)?.values_snapshot()? requests independent copies explicitly.
into_keys() and into_values() consume a dictionary into an ordered list.
See the full view contract
for restrictions and cost details. {} is an empty dictionary; select its types
with an annotation or a constructor such as dict[str, i64]()?.
Reuse a borrowed view
Bind a shared view when several observations use the same dictionary. The view keeps the dictionary borrowed until its last use; it creates no snapshot.
def main() -> Result[(), AllocError | IoError]:
scores = {"Ada": 10, "Grace": 20}?
values = scores.values()
entries = scores.items()
print(len(values), len(entries))?
for score in values:
print(*score)?
for name, score in entries:
print(*name, *score)?
print(iter(values).count())?
Ok(())
2 2
10
20
Ada 10
Grace 20
2
Use iter(view) for transformations and terminals. Item-view callbacks take
a key and a value parameter, just like direct items() pipelines.
Look up a value that may be absent
Missing keys in scores[key] are runtime errors. Use scores.get(key) when
absence is expected. It returns an Option: Some(value) if found, or Nothing
if absent. Match the two possibilities explicitly:
def main() -> Result[(), AllocError | IoError]:
scores = {"Ada": 12, "Grace": 0}?
match scores.get("Ada"):
case Some(score):
print(score)?
case Nothing:
print("unknown player")?
print(scores.get("Grace"))?
print(scores.get("Lin"))?
print(scores["Ada"])?
Ok(())
12
Option[i64].Some(0)
Option[i64].Nothing
12
A stored zero, False, empty string, or Nothing counts as present; looking
up a stored Nothing returns Some(Nothing). Use the
option match to choose a fallback; get has no
default argument.
get leaves the dictionary and key available and does not allocate. It
supports numbers, booleans, strings, and enums of such values. It cannot return
mutable collections or classes: use pop to take ownership or
&dictionary[key] to borrow an existing value. Dictionaries have no optional
borrowed lookup yet; your own functions can
return a borrow that may be absent.
Read a list element that might be missing
items.get(index) returns Some(value) for an existing element or Nothing
for an out-of-range index. It takes one i64 index, including negative indices:
def main() -> Result[(), AllocError | IoError]:
names = ["Ada", "Bea"]?
print(names.get(-1))?
match names.get(2):
case Some(name):
print(name)?
case Nothing:
print("no name at that position")?
print(names)?
Ok(())
Option[str].Some("Bea")
no name at that position
["Ada", "Bea"]
List get has the same element restrictions as dictionary get. For owned
elements, use pop to transfer ownership, copy(items[index])? to duplicate
one, or &items[index] to borrow it. Ordinary indexing still traps when the
index is out of range. The report gives the source position, the index, and the
length: error: main.plenty:3:11: index out of bounds: index 5, length 3.
Both forms of get leave their source unchanged. Numbers, booleans, and
enums copy; strings share immutable storage. The returned value remains valid
if the source is updated or dropped.
Keep distinct values in a set
A set[T] stores each distinct value once. Its elements can be integers,
booleans, or strings:
def main() -> Result[(), AllocError | IoError]:
mut names: set[str] = set()?
names.add("Ada")?
names.add("Ada")?
print(len(names))?
print("Ada" in names)?
names.add("Grace")?
print(names.discard("Ada"))?
print(names.discard("Ada"))?
print(len(names))?
Ok(())
1
True
True
False
1
add needs mutable access and returns an allocation result.
discard removes a member without allocating and returns whether it was
present.
Use set[str]()? for a typed empty set, or provide an annotation as above.
A display such as {"Ada", "Grace"}? builds a set directly. {} is an
empty dictionary. Sets have no promised iteration order.
Dictionary and set equality compare contents independently of order.
Remove an entry and take its value
pop(key) removes a dictionary entry and returns Some(value), or Nothing
when the key is absent. It requires a mutable dictionary and transfers ownership
of the stored value, including nested collections:
def main() -> Result[(), AllocError | IoError]:
mut groups = {"ready": [1, 2]?, "waiting": [3]?}?
match groups.pop("ready"):
case Some(items):
mut work = items
work.append(4)?
print(work)?
case Nothing:
print("no work")?
print(groups)?
print(groups.pop("missing"))?
Ok(())
[1, 2, 4]
{"waiting": [3]}
Option[list[i64]].Nothing
The returned owner remains valid if the dictionary is dropped. Discarding a
successful pop result also cleans up its removed value.
Removal preserves the order of remaining entries without allocating. Reinserting
a removed key puts it at the end. pop takes one key and has no default argument.
Remove list elements
Lists also have pop. With no argument it removes the last element; an i64
index selects another position. Negative indices count from the end:
def main() -> Result[(), AllocError | IoError]:
mut tasks = [[1]?, [2]?, [3]?]?
match tasks.pop(1):
case Some(task):
mut work = task
work.append(4)?
print(work)?
case Nothing:
print("no task")?
print(tasks)?
print(tasks.pop())?
print(tasks.pop(-1))?
print(tasks.pop())?
Ok(())
[2, 4]
[[1], [3]]
Option[list[i64]].Some([3])
Option[list[i64]].Some([1])
Option[list[i64]].Nothing
List pop needs mutable access. Empty lists and out-of-range indices return
Nothing. Removal retains capacity, preserves element order, and does not
allocate. Removing the last element is constant time; removing an earlier one
shifts subsequent elements.
Returning and unpacking tuples
A tuple groups values with different types. Write (value,) for one component;
parentheses without a comma simply group an expression. Function signatures may
spell a tuple as (i64, str) or tuple[i64, str]. Unpacking transfers owned
components, and _ discards a component. Indices must be integer literals; as
with lists, a negative index counts from the end, so pair[-1] is the last
component.
def measurement() -> (i64, str):
(42, "cm")
def main() -> Result[(), IoError]:
value, unit = measurement()
print(value)?
print(unit)?
print((3, "three"))?
Ok(())
42
cm
(3, "three")
Creating (a, b) does not allocate a tuple, so it needs no ?. Operations that
produce its components, such as the list construction below, can still fail.
Nested tuples can be unpacked in one statement. Match each tuple’s shape,
using _ for a component you do not need:
def main() -> Result[(), IoError | OverflowError[i64]]:
((x, y), label) = ((3, 4), "point")
print(label, x + y)?
Ok(())
point 7
Tuples of numbers, strings, or nested tuples of those can be compared. The first
differing component decides the result, just as in sort(). Comparisons leave
both values available:
def main() -> Result[(), IoError]:
first = (1, "apple")
second = (1, "pear")
print(first < second, first <= first)?
print(first, second)?
Ok(())
True True
(1, "apple") (1, "pear")
Tuple float components follow sorting’s total order: -0.0 precedes 0.0, and
NaNs come last. This differs from scalar float comparison; tuple equality still
uses the components’ ordinary equality.
Distinct tuple components can be borrowed and read independently. The borrow below protects component 1 while component 0 remains readable:
def main() -> Result[(), AllocError | IoError]:
mut pair = (1, [2]?)
items = &mut pair[1]
first = pair[0]
print(first)?
print(pair[0])?
items.append(3)?
print(pair)?
Ok(())
1
1
(1, [2, 3])
Copyable components can be changed through mutable references too. A saved copy has its own inline storage:
def main() -> Result[(), IoError]:
mut pair = (1, 2)
saved = pair
first = &mut pair[0]
*first = 5
print(pair)?
print(saved)?
Ok(())
(5, 2)
(1, 2)
Use a reference for the write; direct pair[0] = 5 is not supported.
Reading an exclusively borrowed component is rejected:
def main() -> Result[(), AllocError | IoError]:
mut pair = (1, [2]?)
items = &mut pair[1]
print(pair[1])?
items.append(3)?
Ok(())
conflicting borrow: cannot read or borrow `pair[1]` while it is exclusively borrowed
Print, unpack, and use tuple keys
Print several values with spaces between them. Unpack an owned list when its
length is known; the list must contain exactly as many elements as targets.
The statement moves its elements, and _ discards an element.
def main() -> Result[(), AllocError | IoError | OverflowError[i64]]:
first, second = "north east".split(" ")?
print(first, second, 2)?
mut visits = dict[(i64, i64), i64].new()?
cell = (3, 4)
*visits.get_or_insert(cell, 0)? += 1
*visits.get_or_insert(cell, 0)? += 1
print(visits.get(cell, 0))?
visited = {(3, 4), (3, 4), (4, 3)}?
print(len(visited), cell in visited)?
Ok(())
north east 2
2
2 True
Tuple keys contain integers, booleans, strings, or further tuples of those
types. Values compare by their contents; a grid coordinate needs no integer
encoding or string allocation. get_or_insert starts a missing count at 0
and returns a reference to the stored count; see
dictionaries.
Consume dictionary values directly when the dictionary is finished:
def main() -> Result[(), AllocError | IoError]:
groups = {"first": [1, 2]?, "second": [3]?}?
for group in groups.into_values()?:
print(group)?
Ok(())
[1, 2]
[3]
into_values() consumes the dictionary and transfers its owned values into
an ordered list. Use copy(groups)? first if the original is still needed.
Keep a queue with two ends
Use std.deque.Deque for queues and breadth-first traversal. Removing the
first element does not shift the remaining elements.
from std.deque import Deque
def main() -> Result[(), AllocError | IoError]:
mut queue = Deque[(i64, i64)]()?
queue.append((0, 0))?
queue.append((0, 1))?
queue.appendleft((-1, 0))?
print(len(queue))?
while not queue.is_empty():
match queue.popleft():
case Some(value):
print(value)?
case Nothing:
break
print(queue.pop())?
Ok(())
3
(-1, 0)
(0, 0)
(0, 1)
Option[tuple[i64, i64]].Nothing
len(queue) borrows the queue and calls its shared len method, so the
queue remains available for removal. The same spelling works for built-in
collections and standard containers with a shared len(self) -> i64 method.
Construction and appending return Result with AllocError. End removals
return Option and transfer ownership. Appending takes ownership even on
failure, while the existing queue remains intact. Growth is occasional;
appending within capacity and removing an end allocate nothing. clear()
drops all remaining values and retains the buffer for reuse.
See the module reference for exact operations and costs. A deque is owned; it has no borrowed iterator or indexed API yet.
Sort and reorder a list
sort() puts a list in ascending order. sort(True) gives descending order.
Both change the list in place, need a mut owner, and allocate nothing, so
neither takes ?.
def main() -> Result[(), AllocError | IoError]:
mut scores = [72, 95, 58, 81]?
scores.sort()
print(scores)?
scores.sort(True)
print(scores)?
mut names = ["pear", "fig", "apple"]?
names.sort()
print(names)?
Ok(())
[58, 72, 81, 95]
[95, 81, 72, 58]
["apple", "fig", "pear"]
sort orders integers, floats, strings, and tuples of those. Tuples compare
their first components, then the next where the first are equal. To sort
records by a key, put the key first:
def main() -> Result[(), AllocError | IoError]:
mut ranked = [(72, "Ada"), (95, "Bea"), (72, "Abe")]?
ranked.sort()
print(ranked)?
print(ranked[-1][1])?
Ok(())
[(72, "Abe"), (72, "Ada"), (95, "Bea")]
Bea
Other element types need a key function; natural sorting rejects them:
def main() -> Result[(), AllocError | IoError]:
mut flags = [True, False]?
flags.sort()
Ok(())
sort does not order bool elements; it orders integers, floats, str, and tuples of those
Exchange two elements
swap(i, j) exchanges two elements of any type. Negative indices count from
the end, and a missing index is a runtime error.
def main() -> Result[(), AllocError | IoError]:
mut rows = [[0, 1]?, [2, 3]?, [4, 5]?]?
rows.swap(0, -1)
print(rows)?
Ok(())
[[4, 5], [2, 3], [0, 1]]
The rows are exchanged where they are. Nothing is copied.
An exclusive slice view has the same
swap, so a function can reorder part of a list. Its indices count within the
view:
def ends(values: &mut [i64]) -> ():
values.swap(0, -1)
def main() -> Result[(), AllocError | IoError]:
mut scores = [70, 85, 90, 65, 88]?
ends(&mut scores[1:4])
print(scores)?
Ok(())
[70, 65, 90, 85, 88]
Assign several targets at once
An assignment can name several targets, separated by commas. The whole right side is evaluated first, and then each value is assigned to its target from left to right. That makes an exchange a single statement:
def main() -> Result[(), AllocError | IoError]:
mut values = [10, 20, 30]?
values[0], values[2] = values[2], values[0]
print(values)?
mut low = 9
mut high = 4
low, high = high, low
print((low, high))?
Ok(())
[30, 20, 10]
(4, 9)
The right side is a tuple written without its parentheses. Reading
values[2] copies an integer. A nested list is owned by its row list and cannot
be moved out by reading it, so this form does not exchange owned elements:
def main() -> Result[(), AllocError | IoError]:
mut rows = [[0, 1]?, [2, 3]?]?
rows[0], rows[1] = rows[1], rows[0]
Ok(())
cannot move out of indexed storage; borrow it with &collection[index], take it with replace(collection[index], value) or collection.pop(index), or copy it with copy(collection[index])?
Use rows.swap(0, 1) instead.
Sort by a key or create a sorted copy
sort_by_key reads each element through a shared reference. Its key can be a
number, string, or tuple of those, so class and collection elements can be
ordered too. An optional positional True requests descending order.
class Point:
x: i64
y: i64
def horizontal(point: &Point) -> i64:
point.x
def main() -> Result[(), AllocError | IoError | OverflowError[i64]]:
mut points = [Point(3, 1), Point(1, 2), Point(2, 4)]?
points.sort_by_key(horizontal)
for point in &points:
print(point.x)?
points.sort_by_key(horizontal, True)
print(points[0].x)?
points.sort_by_key(def(point: &Point) -> Result[i64, OverflowError[i64]]: Ok(-point.y))?
print(points[0].x)?
Ok(())
1
2
3
3
2
The operation allocates no key buffer. It calls the key repeatedly, so use a consistent key without side effects. Equal keys may change relative order; include a tie breaker in a tuple key when the order matters. The key can also be an anonymous function written in place, as the last call shows, or a reusable shared-call closure.
sorted creates a copy in natural order, leaving its source available. Copying
is fallible and therefore needs ?:
def main() -> Result[(), AllocError | IoError]:
scores = [72, 95, 58]?
print(sorted(&scores)?)?
print(scores)?
print(sorted(&scores, True)?)?
Ok(())
[58, 72, 95]
[72, 95, 58]
[95, 72, 58]
For a copy ordered by a key, explicitly copy into a mut binding, then call
sort_by_key. Keys may return Result; handle the sorting call with ? and include its error
in the enclosing signature. On failure, elements may already be reordered.
One-shot or mutable-call closures are not supported.
Clean up owned values
Owned values clean up when their scope exits, including through return,
break, continue, and error propagation. A move transfers this
responsibility. Borrowing lends access while leaving cleanup with the owner.
Use drop(value) to release an owner early:
def main() -> Result[(), AllocError | IoError]:
mut numbers = [1, 2, 3]?
drop(numbers)
numbers = [4]?
print(numbers)?
Ok(())
[4]
The old contents are released and the consumed binding is unavailable until reinitialized. Borrow checking prevents dropping an owner while a reference still needs it. A borrow ending at its last use does not destroy the owner.
Dropping a string does not invalidate other strings sharing its immutable storage. Fatal runtime traps terminate without running scope cleanup.
Build singly and doubly linked lists
A linked list connects nodes rather than treating adjacent storage positions as
adjacent elements. It is a useful way to see ownership and borrowing at work.
For an ordinary stack, prefer list[T]; for a queue with two ends, prefer
std.deque. Separate heap allocations and pointer chasing often
make linked nodes more expensive than contiguous storage.
These examples preview classes: a
class groups typed fields, and its generated constructor takes those fields
in declaration order. node.value accesses a field. The pub declarations
make the teaching modules and their fields accessible to the example programs;
they are not encapsulated collection APIs.
A singly linked list owns its successors
Each node owns one value and optionally a boxed successor. Box(value) moves
its argument into a heap allocation and returns a Result; Nothing ends the
chain. The box gives this recursive type a finite size.
pub class Node:
pub value: i64
pub next: Option[Box[Node]]
pub class Linked:
pub head: Option[Box[Node]]
pub def push(xs: &mut Linked, value: i64) -> Result[(), AllocError]:
mut node = Box(Node(value, Nothing))?
node.next = replace(xs.head, Nothing)
xs.head = Some(node)
Ok(())
pub def pop(xs: &mut Linked) -> Option[i64]:
match replace(xs.head, Nothing):
case Some(node):
mut removed: Node = node
xs.head = replace(removed.next, Nothing)
Some(removed.value)
case Nothing:
Nothing
pub def bump(xs: &mut Linked) -> Result[(), OverflowError[i64]]:
match &mut xs.head:
case Some(head):
mut cursor: &mut Node = head
while True:
cursor.value += 1
match &mut cursor.next:
case Some(next):
cursor = next
case Nothing:
break
case Nothing:
pass
Ok(())
replace(place, replacement) installs a new value and returns the old one.
It lets push take the head without leaving the borrowed list uninitialized.
Allocate the detached node before taking that head: if allocation fails,
the caller still owns the original chain. Each successful push allocates one
box; popping and walking allocate nothing.
In pop, the annotation removed: Node extracts the contents of the owned box
and frees its allocation. Taking removed.next transfers the remaining chain
back to the list before the removed node is destroyed.
In bump, matching &mut borrows rather than consumes each link. A borrowed
box converts to a reference to its contents, so cursor can advance from one
node to its successor without copying or recursion. Incrementing a value can
overflow, so bump returns a Result. Earlier nodes remain changed if a later
node fails; this traversal is not transactional. This is a preview of
borrowed matching.
from singly import Linked, push, pop, bump
def main() -> Result[(), AllocError | IoError | OverflowError[i64]]:
mut xs = Linked(Nothing)
for value in range(1, 4):
push(&mut xs, value)?
bump(&mut xs)?
print(pop(&mut xs))?
print(pop(&mut xs))?
print(pop(&mut xs))?
print(pop(&mut xs))?
Ok(())
Option[i64].Some(4)
Option[i64].Some(3)
Option[i64].Some(2)
Option[i64].Nothing
Push and pop operate at the head, so this example behaves as a stack. Each operation takes constant time; walking takes time proportional to the number of nodes. Dropping a remaining chain uses bounded native stack space and allocates nothing. Automatic printing, equality, and copying of recursive owners are unsupported; inspect values by traversing the chain instead.
A doubly linked list uses IDs for both directions
Adding an owning prev: Box[Node] would make neighbours own each other. Plenty
does not support ownership cycles or references stored in fields. Instead,
let a built-in list own every node, and use integer indices as node IDs:
pub class Node:
pub value: i64
pub prev: Option[i64]
pub next: Option[i64]
pub class Linked:
pub nodes: list[Node]
pub head: Option[i64]
pub tail: Option[i64]
pub def push_back(xs: &mut Linked, value: i64) -> Result[(), AllocError]:
id = len(xs.nodes)
xs.nodes.append(Node(value, xs.tail, Nothing))?
match xs.tail:
case Some(old_tail):
xs.nodes[old_tail].next = Some(id)
case Nothing:
xs.head = Some(id)
xs.tail = Some(id)
Ok(())
The empty list has no head or tail. Appending creates a node whose prev is
the old tail, connects that tail’s next, and then updates the tail ID. The
first node also becomes the head. Append happens before existing links change,
so allocation failure leaves them intact.
IDs borrow nothing. Updating neighbouring nodes in separate statements avoids holding simultaneous mutable references into the backing list. Growing that list may move nodes in memory, but their indices remain valid.
from doubly import Linked, push_back
def main() -> Result[(), AllocError | IoError]:
mut xs = Linked([]?, Nothing, Nothing)
push_back(&mut xs, 10)?
push_back(&mut xs, 20)?
push_back(&mut xs, 30)?
mut cursor = xs.head
while True:
match cursor:
case Some(id):
print(xs.nodes[id].value)?
cursor = xs.nodes[id].next
case Nothing:
break
cursor = xs.tail
while True:
match cursor:
case Some(id):
print(xs.nodes[id].value)?
cursor = xs.nodes[id].prev
case Nothing:
break
Ok(())
10
20
30
30
20
10
Forward traversal follows next; backward traversal follows prev. This is a
doubly linked list logically, even though links are indices rather than machine
pointers. Appending is amortized constant time because the backing list can
need to grow; each traversal is linear.
Keep the teaching limits explicit
This example supports append and traversal, not deletion or reusable public handles. It relies on valid IDs and consistent links; public fields let callers break those invariants. Invalid indices trap, and a cycle in the links could make traversal loop forever.
Do not remove a backing-list entry with pop(id): shifting later entries would
invalidate their IDs. To support deletion, use vacant slots, for example a
list[Option[Node]], and reconnect the removed node’s neighbours. If slots are
reused and callers retain IDs, add a generation check to reject stale handles;
handles crossing between collections also need an identity policy. This is more
than adding a prev field to an owned chain.
See representation choices for trade-offs, recursive storage for ownership limits, and parent links with arena IDs for another use of index-based links. The balanced-tree examples apply the same representation to B-tree and red-black insertion, deletion, and sorted traversal. The trie example uses parent and sibling IDs for iterative prefix retrieval and pruning.
Iteration and transformation
Transform collections with loops and comprehensions. Use generators to produce values incrementally, and collect them when you need a stored result.
Count positions and pair inputs with enumerate and
zip directly in loops and comprehensions.
Traverse backward with reversed.
Chain iteration steps with iter to count, test, skip, pair,
and collect items without writing the loop.
Iterate over collections and text
for visits list elements, dictionary keys, set elements, range integers, or
characters of a string. A character is a one-character str.
def main() -> Result[(), AllocError | IoError | OverflowError[i64]]:
mut total = 0
for n in range(1, 5):
total = total + n
print(total)?
scores = {"Ada": 10, "Grace": 20}?
for name in &scores:
print(name)?
print(scores[name])?
print(list(range(5, 0, -2))?)?
for character in "hé":
print(character)?
Ok(())
10
Ada
10
Grace
20
[5, 3, 1]
h
é
range(stop) starts at zero; the stop is excluded. A step may be negative but
never zero: it is a
NonZero[i64]. Ranges can be reused and do not allocate; list(range(5))? allocates
the list. String iteration yields characters without allocating.
Ranges default to i64. Use range[u8](8) or a typed bound to choose another
element type. Start and stop must fit it; the step remains a signed NonZero[i64].
The iterable is evaluated once. Iterating an owned collection transfers it into
the loop; use for item in &values to preserve the owner. Borrowed iteration
prevents conflicting mutation; owned list elements become shared references.
Use &mut values for mutable element references. To iterate a snapshot while
mutating the original, use for item in copy(values)?.
Dictionary key/value loops
items() borrows a dictionary in insertion order. It works directly in loops
and comprehensions, without allocating item tuples or a snapshot. Keys and
values are shared references. Borrow the dictionary exclusively,
(&mut counts).items(), to update every value in place; keys stay shared.
def main() -> Result[(), AllocError | IoError | OverflowError[i64]]:
mut counts = {"apple": 2, "pear": 3}?
for fruit, count in (&mut counts).items():
*count += 1
for fruit, count in counts.items():
print(fruit)?
print(count)?
for a, b in [(1, 2), (3, 4)]?:
print(a + b)?
Ok(())
apple
3
pear
4
3
7
(&mut counts).values() lends only the values. Neither loop looks a key up
or copies a value, so they suit values that cannot be copied. The dictionary
cannot grow, shrink, or be read through another path during a mutable loop.
A shared items() or values() view can be bound to a name and reused; a
mutable one is used directly as the loop source. See
updating every value
for the full contract.
Count positions and pair inputs
Use enumerate(iterable) to get an i64 index starting at zero beside each
element. Supply a second argument to choose the first index. Use
zip(left, right) to pair elements until the shorter input ends.
def main() -> Result[(), AllocError | IoError | OverflowError[i64]]:
for index, letter in enumerate("ab", 10):
print((index, letter))?
for number, letter in zip(range(3), "xy"):
print((number, letter))?
print([index + value for index, value in enumerate(range(3))]?)?
Ok(())
(10, "a")
(11, "b")
(0, "x")
(1, "y")
[0, 2, 4]
An owned list transfers its elements. Borrow a list to preserve it; its elements retain their reference types. Borrow mutably to edit them.
def main() -> Result[(), AllocError | IoError | OverflowError[i64]]:
mut values = [10, 20]?
for index, value in enumerate(&mut values):
*value += index
for value, offset in zip(&values, range(2)):
print(*value + offset)?
print(values)?
Ok(())
10
22
[10, 21]
Both helpers work directly in loop and comprehension sources. They add no
allocation and evaluate their input expressions once, from left to right.
They do not yet produce a value you can store or pass to next. Borrowed
inputs require two unpacked bindings; references cannot be placed into an
item tuple. Nested helpers are supported when their inner items are owned
values. To pair a dictionary’s keys, pass the dictionary itself; items()
remains a separate loop form.
zip tests the left input before the right. A left generator may advance
once more when the right input ends; that unmatched item is dropped. Early
exit drops remaining owned inputs. An enumeration index overflows only when
another element requires an index beyond i64.MAX.
Traverse backward
Use reversed directly in a loop or comprehension to visit a list, text,
or range backward. Borrow a list to keep it afterward. Text yields Unicode
scalars, so a multibyte character stays intact.
def main() -> Result[(), AllocError | IoError | ZeroDivisionError]:
values = [10, 20, 30]?
for index, value in enumerate(reversed(&values)):
print((index, *value))?
for letter in reversed("aé🙂"):
print(letter)?
print([x for x in reversed(range(5)) if x % 2 == 0]?)?
print(values)?
Ok(())
(0, 30)
(1, 20)
(2, 10)
🙂
é
a
[4, 2, 0]
[10, 20, 30]
reversed adds no allocation and does not change a list’s order. Use
reversed(&mut values) to edit its elements in reverse order. It currently
accepts only lists, text, and ranges as loop/comprehension sources and
pipeline sources; it cannot be stored or passed to next.
Borrowing and unpacking while iterating
Borrow a list when a helper needs to inspect or mutate it without taking ownership:
def total(values: &list[i64]) -> Result[i64, OverflowError[i64]]:
mut result = 0
for value in values:
result = result + value
Ok(result)
def add(values: &mut list[i64], value: i64) -> Result[(), AllocError]:
values.append(value)?
Ok(())
def main() -> Result[(), AllocError | IoError | OverflowError[i64]]:
mut numbers = [1, 2]?
print(total(numbers)?)?
add(&mut numbers, 3)?
print(numbers)?
Ok(())
3
[1, 2, 3]
Unpack a tuple in a loop
def measurement() -> (i64, str):
(42, "cm")
def main() -> Result[(), AllocError | IoError]:
value, unit = measurement()
print(value)?
print(unit)?
for number, word in [(1, "one"), (2, "two")]?:
print((number, word))?
print((3, "three"))?
Ok(())
42
cm
(1, "one")
(2, "two")
(3, "three")
Unpacking a borrowed list keeps the list and borrows each component of the
current tuple in place: in for number, word in &pairs, number is a &i64
and word a &str. Over &mut pairs, the names are &mut references, so
the loop can change the tuples:
def main() -> Result[(), AllocError | IoError | OverflowError[i64]]:
mut pairs = [(1, "one"), (2, "two")]?
for number, _ in &mut pairs:
*number *= 10
for number, word in &pairs:
print(*number, word)?
print(pairs)?
Ok(())
10 one
20 two
[(10, "one"), (20, "two")]
The names borrow different components, so both can be passed as &mut
arguments to one call:
def swap(a: &mut i64, b: &mut i64) -> ():
t = *a
*a = *b
*b = t
def main() -> Result[(), AllocError | IoError]:
mut pairs = [(1, 2), (3, 4)]?
for low, high in &mut pairs:
swap(low, high)
print(pairs)?
Ok(())
[(2, 1), (4, 3)]
Tuple targets can nest. Each name still borrows its own component when the source is borrowed:
def main() -> Result[(), AllocError | IoError | OverflowError[i64]]:
mut rows = [((1, 2), "point")]?
x = 99
for (x, y), _ in &mut rows:
*x += 10
*y += 20
print(rows)?
print(x)?
Ok(())
[((11, 22), "point")]
99
The loop’s x is a new binding to the borrowed component. The outer x
remains available after the loop.
Borrow elements while iterating
An exclusive loop lends each element in turn for mutation:
def main() -> Result[(), AllocError | IoError | OverflowError[i64]]:
mut numbers = [10, 20]?
for number in &mut numbers:
*number = *number + 2
print(numbers)?
mut groups = [[1]?, [2, 3]?]?
for group in &mut groups:
group.append(4)?
print(groups)?
Ok(())
[12, 22]
[[1, 4], [2, 3, 4]]
Use &items when inspection is enough. Copyable elements such as integers and
strings are yielded as values; owned elements such as inner lists are shared
references. Use &mut items for mutable references, including scalar elements.
The outer list cannot grow, shrink, or move during either loop. Afterward, the
owner is available again.
Build collections with comprehensions
A comprehension produces a new collection from an iterable and optional filters. The element expression runs only after the filters pass.
def main() -> Result[(), AllocError | IoError | ZeroDivisionError | OverflowError[i64]]:
squares = [n * n for n in range(8) if n % 2 == 0]?
print(squares)?
print({n: n * n for n in [2, 3]?}?)?
print(len({n // 2 for n in range(8)}?))?
print([x * 10 + y for x in range(3) for y in range(x)]?)?
Ok(())
[0, 4, 16, 36]
{2: 4, 3: 9}
4
[10, 20, 21]
Multiple clauses nest from left to right. Later iterables can use earlier
variables, and you can use multiple if filters. Variables remain local to
the comprehension. Parenthesize a conditional expression used as an iterable
or filter. Dictionary comprehensions evaluate each key before its value.
% is modulo: the result follows the divisor’s sign, so -7 % 3 is 2.
Both operands must have the same integer type.
Every element must have the same type; there is no implicit numeric widening:
def main() -> Result[(), AllocError | IoError]:
values = [1, True]?
Ok(())
expected list[i64]
A comprehension over a range or list with no if clause reserves its full
length once; with a filter or another source it grows the output in place. The
? handles allocation failure. A comprehension with one for clause compiles
to the same loop as the equivalent pipeline.
Collect a snapshot with unpacking
Unpacking works in comprehensions too. This one collects dictionary items into a list of tuples:
def main() -> Result[(), AllocError | IoError | OverflowError[i64]]:
mut counts = {"apple": 2, "pear": 3}?
for fruit, count in (&mut counts).items():
*count = *count + 1
print([(*fruit, *count) for fruit, count in counts.items()]?)?
print([a + b for a, b in [(1, 2), (3, 4)]?]?)?
Ok(())
[("apple", 3), ("pear", 4)]
[3, 7]
For dictionaries containing owned values, use copy(value)? when the snapshot
needs independent owners.
Shared item views yield references for every key and value, so the * above
reads each copyable value before storing it in the snapshot tuple.
Text comprehensions collect Unicode scalars:
def main() -> Result[(), AllocError | IoError]:
print([character for character in "hé"]?)?
Ok(())
["h", "é"]
Produce values incrementally
A generator function declares Generator[T] and uses yield statements:
def countdown(start: i64) -> Generator[i64]:
mut remaining = start
while remaining > 0:
yield remaining
# The loop proves 1 <= remaining <= i64.MAX.
unchecked:
remaining -= 1
def main() -> Result[(), IoError]:
numbers = countdown(3)
print("created")?
for number in numbers:
print(number)?
Ok(())
created
3
2
1
Calling a generator evaluates its arguments immediately but starts its body
only when iteration requests a value. Each yield pauses the body, retaining
its locals until the next request. A bare return or the end of the body
finishes it; a generator cannot return a value.
Creating the generator needs no allocation or ?. Allocating arguments and
operations in its body still need their usual error handling. A generator cannot
propagate arithmetic errors to its caller. The countdown uses one small
unchecked: suite: the loop proves that subtracting one stays in range. For
unbounded calculations, yield Result items and let the consumer handle them,
as in lazy expressions.
Comprehensions and collection constructors also consume generators:
def numbers(limit: i64) -> Generator[i64]:
for n in range(limit):
yield n
def main() -> Result[(), AllocError | IoError | ZeroDivisionError | OverflowError[i64]]:
print([n * n for n in numbers(6) if n % 2 == 1]?)?
print(list(numbers(3))?)?
Ok(())
[1, 9, 25]
[0, 1, 2]
Call next(&mut messages) to advance a mutable generator without consuming it.
The shorthand next(messages) also works. It returns Some(value) or Nothing;
an exhausted generator keeps returning Nothing:
def once() -> Generator[str]:
yield "hello"
def main() -> Result[(), IoError]:
mut messages = once()
print(next(&mut messages))?
print(next(&mut messages))?
print(next(&mut messages))?
Ok(())
Option[str].Some("hello")
Option[str].Nothing
Option[str].Nothing
Breaking out of consuming iteration drops the generator without resuming its
body. Yielding an owned value transfers ownership; to keep it, handle
copy(value)’s result and yield the copy.
Generators cannot yield other generators, references, or closures. For a lazy
expression producer, use a generator expression.
yield from, send, and async operations are unsupported.
Pass a lazy expression to a consumer
A generator expression evaluates one element at a time, when its consumer asks for the next one:
def square(n: i64) -> Result[i64, OverflowError[i64]]:
Ok(n * n)
def main() -> Result[(), AllocError | IoError | OverflowError[i64]]:
squares = (square(n) for n in range(4) if n != 2)
mut values: list[i64] = []?
for outcome in squares:
values.append(outcome?)?
print(values)?
words = ["tea", "", "café"]?
print(", ".join(w for w in words if w != "")?)?
Ok(())
[0, 1, 9]
tea, café
The square helper returns a Result because arbitrary i64 squares can
overflow. The generator yields those results without unwrapping them; the loop
uses ? in main before appending each value. A caller’s arithmetic mode never
changes a producer’s declared contract.
separator.join accepts any iterable of str, so the selected words are
joined without first collecting them into a list. A generator cannot be
measured in advance, so join grows a scratch buffer as pieces arrive; a list,
set, or dictionary is measured first and needs only the result’s allocation.
A consumer that stops early, such as the pipeline method
any, leaves the remaining elements unevaluated.
The outer source evaluates when the generator is created. Filters, elements,
and inner for sources evaluate on demand. The generator owns its sources
and copies or moves any enclosing local values it uses; copied captures are
snapshots. Borrowed inputs are unsupported because a suspended frame cannot
retain a reference into its caller. Use copy(values)? to preserve an owned list,
or consume a borrowing pipeline locally instead.
Parentheses can be omitted for a sole argument. With additional arguments,
write consume((n for n in source), other). Generator expressions follow the
same ownership, cleanup, and yield restrictions as generator functions;
they require no intermediate list or heap-allocated frame. A fallible operation
in their body must be handled explicitly, since ? cannot propagate out of
a suspended generator.
Transfer a generator’s progress
A generator owns a position in an advancing computation. Assignment transfers that owner instead of copying the position:
def once() -> Generator[i64]:
yield 42
def main() -> Result[(), AllocError | IoError]:
first = once()
second = first
print(list(second)?)?
Ok(())
[42]
After a move, the old binding cannot be used:
def once() -> Generator[i64]:
yield 42
def main() -> Result[(), AllocError | IoError]:
first = once()
second = first
list(first)?
pass
Ok(())
use of moved binding `first`
Owned arguments, returns, and for iteration also move generators.
next(&mut it) borrows a mutable generator without consuming its owner;
next(it) remains a shorthand. A mutable binding can be reinitialized after
moving its previous value.
The compiler checks all possible continuing paths. A move on just one branch
makes later reuse unsafe, reported as use of possibly moved binding. Within a
loop, an outer generator must be reinitialized before any path repeats the loop.
Generators cannot be stored in collections or user-defined enum payloads yet.
See Borrow a collection for the borrowing rules.
Drop without resuming
Use drop(value) to release an owner earlier. The consumed binding becomes
unavailable; a mutable binding may then receive another value:
def pending() -> Generator[i64]:
yield 1
def main() -> Result[(), AllocError | IoError]:
mut numbers = [1, 2, 3]?
drop(numbers)
numbers = [4]?
print(numbers)?
task = pending()
drop(task)
print("done")?
Ok(())
[4]
done
Dropping a generator cleans its captured values without resuming its body. Borrow checking prevents dropping an owner while a reference still needs it. A borrow can end at its last use; that does not implicitly destroy its owner.
Passing and returning generators
Use Generator[T] to accept or return a generator:
def once(n: i64) -> Generator[i64]:
yield n
def pair() -> Generator[i64]:
yield 10
yield 20
def source(n: i64) -> Generator[i64]:
once(n)
def total(values: Generator[i64]) -> Result[i64, OverflowError[i64]]:
mut result = 0
for value in values:
result = result + value
Ok(result)
def main() -> Result[(), IoError | OverflowError[i64]]:
print(total(source(7))?)?
print(total(pair())?)?
Ok(())
7
30
total accepts either producer. Passing a generator transfers ownership; use
&mut Generator[T] when a helper should advance an existing owner with next.
Once you know closures, Transform a stream of values
shows how to filter, map and reduce a generator without writing the loop.
A factory must return the same producer on every path. Two generators yielding the same type are not interchangeable return values:
def one() -> Generator[i64]:
yield 1
def two() -> Generator[i64]:
yield 2
yield 3
def choose(flag: bool) -> Generator[i64]:
if flag:
return one()
two()
def main() -> ():
drop(choose(True))
expected Generator[i64] from `one`, got Generator[i64] from `two`
Put such a branch inside one generator body instead. A generator may contain another generator, but recursive generator state is rejected: its inline storage would have no finite size.
Creating generators
An ordinary function can create and return a generator:
def numbers() -> Generator[i64]:
yield 10
yield 20
def source() -> Generator[i64]:
numbers()
def total() -> Result[i64, OverflowError[i64]]:
values = source()
mut result = 0
for n in values:
result = result + n
Ok(result)
def main() -> Result[(), IoError | OverflowError[i64]]:
print(total()?)?
Ok(())
30
A factory that can fail can return
Result[Generator[T], E]. Write Ok(numbers()) for success and use ? at the
factory’s call site. Wrapping a generator adds no allocation; dropping the
wrapper releases its captures without resuming the body.
Collect results into owned collections
Return a comprehension’s Result directly when the caller should handle
allocation failure:
def squares() -> Result[list[i64], AllocError | ZeroDivisionError | OverflowError[i64]]:
[n * n for n in range(6) if n % 2 == 0]
def main() -> Result[(), AllocError | IoError | AllocError | OverflowError[i64] | ZeroDivisionError]:
print(squares()?)?
Ok(())
[0, 4, 16]
Collect an owned iterator with recoverable list growth:
def numbers() -> Generator[i64]:
yield 3
yield 6
def collect() -> Result[list[i64], AllocError]:
source = numbers()
list[i64].from(source)
def main() -> Result[(), AllocError | IoError]:
print(collect()?)?
Ok(())
[3, 6]
from consumes the source. If allocation fails, it drops the partial list and
remaining iterator. Neither from nor comprehensions undo earlier effects or
handle failures inside the producer’s body.
Sets provide the same constructor, removing duplicates in first-seen order:
def main() -> Result[(), AllocError | IoError]:
print(set[i64].from([3, 1, 3, 2]?)?)?
Ok(())
{3, 1, 2}
Dictionary sources iterate over keys. from does not support borrowed sources
or string iteration.
Chain iteration steps with iter
iter(source) starts a pipeline: a chain of iteration steps that runs only
when a loop or a final method such as count() consumes it. No intermediate
collection is built.
def main() -> Result[(), AllocError | IoError]:
rolls = [3, 6, 2, 6, 6, 1]?
print(iter(&rolls).filter_value(6).count())?
for turn, roll in iter(&rolls).enumerate(1).skip(2).take(3):
print(turn, *roll)?
print(len(rolls))?
Ok(())
3
3 2
4 6
5 6
6
filter_value(6) keeps the items equal to 6, and count() counts them
without building a list. enumerate(1) numbers the items from one, skip(2)
discards the first two, and take(3) stops after three more. Steps apply in
the order written: enumerate comes first, so the turn numbers count every
roll, including skipped ones.
The source follows the ordinary iteration rules. Borrowing rolls keeps the
list, and each item is a reference, so the loop reads *roll.
filter_value still takes a plain i64: a borrowed item is compared by value.
It also accepts a borrowed value, such as filter_value(&wanted), so a string
can be matched without copying it; wanted then stays borrowed until the
pipeline ends.
A pipeline is single-use. Consuming it again, or using an owned source after the pipeline has taken it, is an ownership error.
Add up and find extremes
sum(), min(), and max() are final methods that need no function either:
def main() -> Result[(), AllocError | IoError | OverflowError[i64]]:
rolls = [3, 6, 2, 6, 6, 1]?
print(iter(&rolls).sum()?, iter(&rolls).min(), iter(&rolls).skip(4).max())?
none: list[i64] = []?
print(iter(&none).sum()?, iter(&none).max().unwrap_or(0))?
Ok(())
24 Option[i64].Some(1) Option[i64].Some(6)
0 0
A borrowed item is read by value, so the results are i64, not references.
sum() gives zero for no items. Integer overflow returns OverflowError[i64],
handled with ? here; floating-point sums retain IEEE behavior.
min() and max() return an Option, Nothing for no items. They
accept numbers and str and keep the first of equal items. Floats follow the
IEEE 754 rules for NaN and signed zero, as std.math.min_all and max_all do;
the pipeline reference
gives them.
Test boolean items
any() reports whether at least one item is True, and all() whether every
item is. Both stop as soon as they know the answer:
def positive(values: list[i64]) -> Generator[bool]:
for value in values:
yield value > 0
def main() -> Result[(), AllocError | IoError]:
present = [True, False, True]?
print(iter(&present).any(), iter(reversed(&present)).all())?
print(positive([2, -1, 3]?).any(), positive([2, -1, 3]?).all())?
none: list[bool] = []?
print(iter(&none).any(), iter(&none).all())?
Ok(())
True False
True False
False True
The generator computes each comparison only when advanced, so all() stops at
-1 and never tests 3; its frame is cleaned up when the method returns.
For no items, any() gives False and all() gives True.
Without an argument, these methods need bool items: numbers and text have no
implicit truth value. To test other items, pass a predicate function; that
form is taught in
Reduce a borrowed collection with a pipeline.
Pair, join, and collect
zip walks two sources together, and chain continues with a second source of
the same item type. collect() builds the collection named by the binding’s
annotation and returns a Result for allocation failure:
def main() -> Result[(), AllocError | IoError]:
names = ["Ada", "Bob", "Cal"]?
present = [True, False, True]?
for name, here in iter(&names).zip(&present):
print(name, *here)?
print(iter(&present).filter_value().count())?
morning = [3, 1]?
evening = [4, 3]?
readings: list[i64] = iter(morning).chain(evening).collect()?
print(readings)?
Ok(())
Ada True
Bob False
Cal True
2
[3, 1, 4, 3]
Without an argument, filter_value() keeps True items. The two owned lists
move into the chain, so their numbers move into readings without copying.
The value must have the item’s type. There is no implicit conversion, and the
no-argument form works only for bool items:
def main() -> Result[(), AllocError | IoError]:
rolls = [3, 6]?
print(iter(&rolls).filter_value("6").count())?
Ok(())
filter_value compares &i64 items with a str value; it requires i64 or &i64
Take one item, then continue
next takes one item from a pipeline, as it does from a generator. The
pipeline keeps its place, so a loop or final method afterwards continues with
the remaining items:
def main() -> Result[(), AllocError | IoError]:
lines = ["name,score", "ada,3", "bob,5"]?
mut rows = iter(lines)
match next(rows):
case Some(header):
print("columns:", header)?
case Nothing:
print("no header")?
for number, row in rows.enumerate(1):
print(number, row)?
Ok(())
columns: name,score
1 ada,3
2 bob,5
The binding needs mut because next advances it. next returns an owned
Option, so it needs owned items: here iter(lines) moves the list into the
pipeline. For a borrowed source, first map each item to an owned value.
Once next returns Nothing, the pipeline stays empty.
Lend a pipeline to a function
A function can take a pipeline as &mut Iter[T] and consume part of it. The
caller lends it with &mut and continues after the items the function took:
def print_header(rows: &mut Iter[str]) -> Result[(), AllocError | IoError]:
match next(rows):
case Some(header):
print("columns:", header)?
case Nothing:
print("no header")?
Ok(())
def main() -> Result[(), AllocError | IoError]:
lines = ["name,score", "", "ada,3", "bob,5"]?
mut rows = iter(lines).filter(def(line: &str) -> bool: line != "")
print_header(&mut rows)?
print(rows.count(), "rows")?
Ok(())
columns: name,score
2 rows
The function can also loop over rows and break or return early; the rest
stays with the caller. It cannot keep or return the pipeline.
A pipeline over a borrowed list yields references, written &mut Iter[&str]
here. The function can use each item while it runs, but cannot return one,
because the caller lent the list only for the call:
def greet_first(names: &mut Iter[&str]) -> Result[(), IoError]:
for name in names:
print("hello", name)?
break
Ok(())
def main() -> Result[(), AllocError | IoError]:
names = ["ada", "bob", "ada", "cal"]?
mut adas = iter(&names).filter_value("ada")
greet_first(&mut adas)?
print(adas.count(), "more")?
print(len(names))?
Ok(())
hello ada
1 more
4
A lent pipeline whose callbacks borrow local values needs those closures written inside the adapter calls; see Lend a pipeline whose callbacks borrow.
Generators have the same methods without iter:
def countdown(start: i64) -> Generator[i64]:
mut remaining = start
while remaining > 0:
yield remaining
# The loop proves 1 <= remaining <= i64.MAX.
unchecked:
remaining -= 1
def main() -> Result[(), AllocError | IoError]:
last: list[i64] = countdown(10).skip(7).collect()?
print(last)?
Ok(())
[3, 2, 1]
skip still advances the generator past the first seven values; it only
discards them.
Prefer a pipeline when a chain of these steps states the task directly, and a
plain loop or comprehension when the work needs its own expression or several
statements. Most other pipeline steps, such as map, filter, and fold,
take a function. They are taught after
functions as values, starting with
Reduce a borrowed collection with a pipeline.
The pipeline reference
lists every step and its cost.
Defining your own data
Use classes for records with fields and methods, and enums for alternatives. Their fields follow the same ownership and cleanup rules as collections.
Heterogeneous collections combines classes and enums in lists and dictionary values, and explains the restrictions on sets and dictionary keys. Special-purpose data structures introduces columnar lists for scanning selected primitive fields.
Group fields and methods in a class
A class is a record with typed fields. Without __init__, its constructor takes
the fields in declaration order:
class Point:
x: i64
y: i64
def squared_length(self) -> Result[i64, OverflowError[i64]]:
Ok(self.x * self.x + self.y * self.y)
def shift(self: &mut Point, amount: i64) -> Result[(), OverflowError[i64]]:
x = self.x + amount
y = self.y + amount
self.x = x
self.y = y
Ok(())
def main() -> Result[(), AllocError | IoError | OverflowError[i64]]:
mut point = Point(y=4, x=3)
print(point.squared_length()?)?
point.shift(amount=1)?
mut changed = copy(point)?
changed.x = 20
print(point)?
print(changed)?
Ok(())
25
Point(x=4, y=5)
Point(x=20, y=5)
Ordinary methods borrow self read-only by default: self is shorthand for
self: &Point here. A method that changes fields declares self: &mut Point.
Calling it requires a mut binding or an exclusive reference. shift calculates
both coordinates before changing either, so overflow leaves the point unchanged.
Arithmetic failure does not undo earlier assignments automatically.
An explicit self: Point instead takes ownership; see
consuming methods for state transitions and finalization.
Methods can share names with integer built-ins. A class receiver selects its own method, including its parameter and return types:
class Counter:
total: i64
def wrapping_add(self: &Counter, other: i64) -> Result[i64, OverflowError[i64]]:
Ok(self.total + other)
def main() -> Result[(), IoError | OverflowError[i64]]:
counter = Counter(1)
print(counter.wrapping_add(2)?)?
Ok(())
3
This method uses ordinary checked addition; its name does not change that.
If you forget the receiver annotation, the error names the required signature:
class Counter:
value: i64
def increment(self) -> Result[(), OverflowError[i64]]:
self.value = self.value + 1
Ok(())
def main() -> ():
pass
declare `self: &mut Counter`
Change the parameter to self: &mut Counter, as in shift above.
The lifecycle methods __init__ and __del__ give bare self exclusive
access automatically.
Instances store their fields inline: Point(3, 4) needs no allocation or ?.
Copying list fields can still fail to allocate.
By default, class instances move even when their fields are all integers.
other = point transfers ownership; copy(point) requests an independent
instance. Integer and string fields can be read directly. Owned fields such as
lists or other classes must be borrowed, observed, or explicitly copied.
Equality compares fields; printing shows the class name and fields.
Copy small value types
Write @copy on the line before class when instances are plain values, like
integers. Assignment, argument passing, returning, and storing in a collection
then copy the instance and leave the original usable:
@copy
class Money:
cents: i64
currency: str
def add(a: Money, b: Money) -> Result[Money, OverflowError[i64]]:
Ok(Money(a.cents + b.cents, a.currency))
def main() -> Result[(), IoError | OverflowError[i64]]:
price = Money(250, "EUR")
tax = Money(50, "EUR")
paid = add(price, tax)?
mut discounted = price
discounted.cents = 200
print(price, paid, discounted)?
Ok(())
Money(cents=250, currency="EUR") Money(cents=300, currency="EUR") Money(cents=200, currency="EUR")
Every field must itself copy: a scalar, str, another @copy class, or a tuple
or enum of those. Copying duplicates the inline fields and never allocates, so
it needs no ?. A field that owns storage is rejected:
@copy
class Basket:
items: list[str]
def main() -> ():
pass
every field of a `@copy` class must copy
A @copy class also cannot define __del__. A
generic @copy class copies when its type
arguments copy, so Pair[i64] copies and Pair[list[i64]] moves. Keep the
default for instances that own resources or should be used only once; see
copy classes.
Hide secrets when printing
Printing a class shows every field, so a password or key reaches any log that
prints its holder. Write @redacted before class or enum to replace the
contents with (@redacted) wherever the value is printed or formatted, however
deeply it is nested:
@redacted
class Password:
text: str
class Login:
user: str
password: Password
def main() -> Result[(), AllocError | IoError]:
login = Login("ada", Password("hunter2"))
print(login)?
print(str.repr(login.password)?)?
print(login.password.text == "hunter2")?
Ok(())
Login(user="ada", password=Password(@redacted))
Password(@redacted)
True
The fields stay readable in code, so redaction prevents accidental printing rather than enforcing secrecy; see redacted types.
Limits
There is no inheritance, dynamic attribute creation, or class-variable syntax.
Fields currently have no default values. Constructors and methods accept named
arguments; an explicit __init__ can give its parameters defaults.
Give arithmetic meaning to your types
Use arithmetic operators when their meaning is natural for your type. A method
named __add__ defines +, and __neg__ defines unary -. These are statically
compiled calls, so using the operator does not introduce runtime method lookup.
The distance wrappers here demonstrate hook and forwarding mechanics. For physical calculations, use quantities: they also check the dimensions of products and quotients, which these wrappers do not.
class Distance:
metres: i64
def __add__(self, other: &Distance) -> Result[Distance, OverflowError[i64]]:
Ok(Distance(self.metres + other.metres))
def __neg__(self) -> Result[Distance, OverflowError[i64]]:
Ok(Distance(-self.metres))
def __mul__(self, scale: i64) -> Result[Distance, OverflowError[i64]]:
Ok(Distance(self.metres * scale))
def __rmul__(self, scale: i64) -> Result[Distance, OverflowError[i64]]:
self * scale
def add(self: &mut Distance, other: &Distance) -> Result[(), OverflowError[i64]]:
self.metres += other.metres
Ok(())
def main() -> Result[(), IoError | OverflowError[i64]]:
mut walk = Distance(10)
extra = Distance(5)
print((walk + extra)?, (-extra)?, (3 * extra)?)?
walk.add(&extra)?
print(walk, extra)?
Ok(())
Distance(metres=15) Distance(metres=-5) Distance(metres=15)
Distance(metres=15) Distance(metres=5)
The ordinary hooks borrow their operands, so extra stays usable. The reflected
hook __rmul__ handles a distance on the right of 3 * extra. The named add
method takes exclusive access and updates walk in place. Its integer addition
can fail, so it returns a Result; the binding must be mutable.
Integer hooks declare overflow just like ordinary functions. A hook can also report a domain error; here the extra check prevents negative combined stock:
class Stock:
amount: i64
def __add__(self, other: &Stock) -> Result[Stock, str | OverflowError[i64]]:
total = self.amount + other.amount
if total < 0:
return Err("negative stock")
Ok(Stock(total))
def main() -> Result[(), IoError | str | OverflowError[i64]]:
a = Stock(5)
b = Stock(3)
combined = (a + b)?
print(combined.amount)?
Ok(())
8
Parenthesize the operation before ?. Allocation errors follow the same pattern.
For a fallible update, write value = (value + other)? or call a named method;
+= requires a unit-returning in-place hook. For large data, a named method that
writes into reusable output storage can avoid intermediate allocations.
A class that only gives a distinct name to one value can forward its field’s
operators instead of defining a hook for each. Annotate it @newtype:
@newtype
class Metres:
value: i64
@newtype
class Seconds:
value: i64
def main() -> Result[(), IoError | OverflowError[i64]]:
run = Metres(400)
mut total = run + run
total += Metres(100)
print(total, total > run)?
lap = Seconds(70)
print(lap * Seconds(2))?
Ok(())
Metres(value=900) True
Seconds(value=140)
run + run adds the two value fields and wraps the sum back in Metres, and
comparisons such as > return bool. Typing stays nominal: run + lap and
run + 100 are compile errors, which keeps metres and seconds apart. An
explicit hook such as __add__ takes over its own operator. @newtype does
not make the class copy; add @copy for that.
Equality still compares fields automatically. Python special methods that
Plenty never calls, such as __eq__, __hash__, __str__ and __len__, are
compile errors rather than silently ignored; see
equality, ordering and hashing
and reserved method names.
For supported hooks, exact signatures, and resolution rules, see
user-defined arithmetic operators.
Return a reference to a field
A function that returns a reference borrows it from its first reference
parameter; for a method, that is self. The returned value must borrow that
parameter, never a later one or a local owner. The caller keeps the original
value alive and borrowed until the result’s last use.
class Pair:
left: i64
right: i64
def left_ref(self: &mut Pair) -> &mut i64:
&mut self.left
def main() -> Result[(), IoError]:
mut pair = Pair(1, 2)
left = pair.left_ref()
*left = 8
print(pair.left)?
Ok(())
8
A mutable returned reference requires the first reference parameter to be
&mut. A temporary receiver cannot supply a reference that outlives it.
Later reference parameters only lend their values for the call. A lookup can take its key by reference and still return a borrow of the collection:
def score(scores: &dict[str, i64], name: &str) -> &i64:
&scores[*name]
def main() -> Result[(), AllocError | IoError]:
scores = {"ada": 3, "bob": 5}?
mut name = "bob"
points = score(&scores, &name)
name = "eve"
print(*points)?
print(name)?
Ok(())
5
eve
The key’s loan ends when score returns, so name can change while points
is live. To return a borrow of name, it would have to be the first parameter.
Return a borrow that may be absent
A search may find nothing. Return Option[&T], or Option[&mut T] to let the
caller change what was found. Some must borrow the first reference parameter;
Nothing borrows nothing:
class Item:
name: str
count: i64
def find(items: &mut list[Item], name: &str) -> Option[&mut Item]:
for item in items:
if item.name == name:
return Some(item)
Nothing
def add(items: &mut list[Item], name: str) -> Result[(), AllocError | OverflowError[i64]]:
match find(items, &name):
case Some(item):
item.count += 1
case Nothing:
items.append(Item(name, 1))?
Ok(())
def main() -> Result[(), AllocError | IoError | AllocError | OverflowError[i64]]:
mut items: list[Item] = []?
add(&mut items, "ada")?
add(&mut items, "bob")?
add(&mut items, "ada")?
for item in items:
print(item.name)?
print(item.count)?
Ok(())
ada
2
bob
1
Match the result directly. In the Some arm, item borrows items, so the
list cannot change there. The Nothing arm holds no borrow, so it can append.
Result[&T, E] works the same way with Ok; its error cannot hold a reference.
? and unwrap() also produce the reference.
The result can be held in a local until it is matched, but it cannot be stored
in a list, a field, or a mut binding: match it and store an owned copy or an
index. The reference
lists every rule.
Shared arguments can borrow temporaries, as in size("ada") for a size(s: &str)
function. The temporary lives through the expression. A result borrowing it may
be used immediately, but cannot be stored or returned. Mutable temporaries are
unsupported.
An exclusive method can use a completed read of its receiver as an argument. This also works through an existing exclusive reference:
import std.bytes
from std.bytes import Bytes
def clear(buffer: &mut Bytes) -> ():
buffer.fill(0, buffer.len(), 0u8)
def main() -> Result[(), AllocError | IoError]:
mut buffer = std.bytes.from_list([1u8, 2u8]?)?
clear(&mut buffer)
print(buffer.to_list()?)?
Ok(())
[0, 0]
The receiver is reserved first, arguments run left to right, and exclusive access begins when the method starts. Arguments may read the receiver, but cannot mutate or move it, or pass a reference into it that remains live during the call. For nested mutations, compute the inner result in a separate local before calling the outer method.
Control construction with init
Define __init__ to calculate or check fields during construction. It replaces
the generated constructor; bare self is an exclusive reference:
class Span:
start: i64
end: i64
def __init__(self, start: i64, length: i64) -> Result[(), OverflowError[i64]]:
self.start = start
self.end = start + length
Ok(())
def length(self) -> Result[i64, OverflowError[i64]]:
Ok(self.end - self.start)
def main() -> Result[(), IoError | OverflowError[i64]]:
span = Span(10, 5)?
print(span)?
print(span.length()?)?
Ok(())
Span(start=10, end=15)
5
The initializer calculates the endpoint from a start and length. Overflow returns
its declared OverflowError[i64], so Span(10, 5) returns
Result[Span, OverflowError[i64]]; ? propagates failure through main.
An initializer returning () instead gives an infallible constructor. Construction
itself allocates nothing and adds no allocation error; see
constructor contracts.
Every field must be initialized on every path out of the constructor. You may
read a field already initialized, but cannot pass the whole unfinished self to
another function or method. A loop might never execute, so initializing a field
only inside a loop is insufficient.
class Pair:
first: i64
second: i64
def __init__(self, first: i64) -> ():
self.first = first
def main() -> ():
pass
fields not initialized: second
Initialize fallible fields
An initializer can return Result[(), E] when setting up its fields may fail.
The constructor then returns Result[Class, E]. Here, reserving list capacity
can fail to allocate:
class Buffer:
values: list[i64]
def __init__(self: &mut Buffer, size: i64) -> Result[(), AllocError]:
self.values = list[i64].with_capacity(size)?
Ok(())
def main() -> Result[(), AllocError | IoError]:
print(Buffer(8)?)?
Ok(())
Buffer(values=[])
If initialization fails, the fields already initialized are dropped, but the
incomplete instance’s __del__ is skipped. Successful initialization must fill
every field and return Ok(()).
Borrow distinct fields
Borrow a field directly with & or &mut:
class Basket:
count: i64
items: list[str]
def main() -> Result[(), AllocError | IoError]:
mut basket = Basket(0, []?)
count = &mut basket.count
*count = 2
basket.items.append("apple")?
basket.items.append("pear")?
print(basket)?
Ok(())
Basket(count=2, items=["apple", "pear"])
Distinct fields can be borrowed or changed independently. Borrowing a whole
record overlaps all its fields.
Collection element references such as &basket.items[0] protect the collection
against changes that could invalidate the element’s address.
You can replace a class-valued field by assignment, but cannot replace a whole
class through *reference = new_instance yet.
class Position:
x: i64
y: i64
def main() -> Result[(), IoError]:
mut position = Position(1, 2)
x = &mut position.x
y = &mut position.y
*x = 10
*y = 20
print(*x)?
print(*y)?
Ok(())
10
20
Borrowing a field through a getter
A simple getter that directly returns a field preserves the field’s identity for the borrow checker. After the call, unrelated fields remain available:
class Pair:
x: i64
y: i64
def x_ref(self: &mut Pair) -> &mut i64:
&mut self.x
def main() -> Result[(), IoError]:
mut pair = Pair(1, 2)
x = pair.x_ref()
pair.y = 7
*x = 9
print(pair)?
Ok(())
Pair(x=9, y=7)
Getters that choose between fields borrow the whole argument. Indexed references also borrow the whole collection. The call still requires the access declared by the receiver type.
Describe alternatives with enums
An enum defines alternatives, each of which may carry typed data. Match its variants to extract that data:
enum Reading:
Missing
Value(i64)
Invalid(str)
def describe(reading: Reading) -> str:
match reading:
case Reading.Missing:
"no reading"
case Reading.Value(number):
"positive" if number > 0 else "nonpositive"
case Reading.Invalid(reason):
reason
def main() -> Result[(), IoError]:
print(describe(Reading.Missing))?
print(describe(Reading.Value(42)))?
print(describe(Reading.Invalid("sensor offline")))?
Ok(())
no reading
positive
sensor offline
Enum payloads are stored inline: Reading.Value(42) needs no allocation or ?.
Variants without data omit parentheses. Payloads may have several positions,
such as Pair(i64, str). Case bindings are immutable and local to that case.
Use _ to ignore a payload position or a final case _: to cover remaining
variants; case _ as name: also binds the remaining value. Duplicate or missing cases are errors:
enum Switch:
On
Off
def main() -> Result[(), IoError]:
match Switch.On:
case Switch.On:
print("on")?
Ok(())
non-exhaustive match; missing Switch.Off
A final match produces the function’s result; cases can also return early or break/continue a loop. A match can also be the value of a binding. Separately declared enums remain distinct types, even with identical variants.
A payload position can hold another pattern instead of a name, so one case
reaches into an enum stored inside an enum. | joins patterns that share an
arm; every alternative must bind the same names:
enum Shape:
Dot
Circle(i64)
Rect(i64, i64)
enum Slot:
Empty
Full(Shape)
def width(slot: Slot) -> i64:
match slot:
case Slot.Full(Shape.Circle(size) | Shape.Rect(size, _)):
size
case Slot.Full(Shape.Dot) | Slot.Empty:
0
def main() -> Result[(), IoError]:
print(width(Slot.Full(Shape.Rect(3, 4))))?
print(width(Slot.Full(Shape.Circle(5))))?
print(width(Slot.Empty))?
Ok(())
3
5
0
Coverage is checked through the nested patterns: leaving out
Slot.Full(Shape.Dot) is an error that names it.
Payloads can contain strings, collections, and other enums. Matching an enum
with mutable payloads consumes it and transfers those payloads; use
match copy(value)? to preserve the owner.
An enum can keep behavior beside its variants. A method’s bare self is a
shared borrow, so matching it borrows the payload. The same method can satisfy
a structural protocol:
protocol Numbered:
def number(self) -> i64:
pass
enum Reading:
Missing
Value(i64)
def number(self) -> i64:
match self:
case Reading.Missing:
0
case Reading.Value(value):
*value
def number[T: Numbered](value: &T) -> i64:
value.number()
def main() -> Result[(), IoError]:
reading = Reading.Value(42)
print(number(&reading))?
print(reading.number())?
Ok(())
42
42
Use pub def when callers in other modules need the method, and
self: &mut Reading when it changes a payload. Variant and method names must
be distinct. The formatting lesson uses an enum
method to produce a message while preserving the error data.
Match combinations with tuple patterns
Match a tuple when the answer depends on several values together. Each component takes a pattern, and the cases cover combinations of components:
def combine(left: Option[i64], right: Option[i64]) -> Result[i64, OverflowError[i64]]:
match (left, right):
case (Some(a), Some(b)):
Ok(a + b)
case (Some(a), Nothing):
Ok(a)
case (Nothing, Some(b)):
Ok(b)
case (Nothing, Nothing):
Ok(0)
def main() -> Result[(), IoError | OverflowError[i64]]:
print(combine(Some(3), Some(4))?)?
print(combine(Nothing, Some(5))?)?
Ok(())
7
5
Leaving out (Nothing, Nothing) is an error: both inputs can be absent.
Tuple patterns also nest inside enum payloads, such as Some((value, offset)).
Write (pattern,) for a singleton tuple; (pattern) groups a pattern.
Matching an owned tuple transfers owned components into the case’s bindings
and destroys ignored components. Match &value or &mut value to borrow its
components, as in borrowed matching. The
matching reference
describes coverage and pattern limits.
Match literal values
A pattern can be an integer, string, or boolean literal. It matches a value
equal to it, so match also selects on an integer, a str, or a bool
directly, and on such a value inside an enum payload:
enum Command:
Move(i64)
Say(str)
def opcode(name: str) -> i64:
match name:
case "cpy":
1
case "inc" | "dec":
2
case _:
0
def describe(command: Command) -> str:
match command:
case Command.Move(0):
"stay"
case Command.Move(_):
"move"
case Command.Say("" | "..."):
"silence"
case Command.Say(words):
words
def main() -> Result[(), IoError]:
print(opcode("cpy"))?
print(opcode("dec"))?
print(opcode("jmp"))?
print(describe(Command.Move(0)))?
print(describe(Command.Move(3)))?
print(describe(Command.Say("...")))?
print(describe(Command.Say("hello")))?
Ok(())
1
2
0
stay
move
silence
hello
Cases are tried in order. Literals cannot name every integer or string, so
such a match ends with case _:, and a payload position needs _ or a name
after its literals:
def main() -> Result[(), IoError]:
count = 3
match count:
case 1:
print("one")?
Ok(())
non-exhaustive match; cases cannot name every i64 value, so end with `case _:`
True and False together cover a bool, so no case _: follows them:
def label(enabled: bool) -> str:
match enabled:
case True:
"on"
case False:
"off"
def main() -> Result[(), IoError]:
print(label(True))?
print(label(3 > 4))?
Ok(())
on
off
An integer literal takes the type of the value it matches and must fit it:
case 300: is an error for a u8. A float is not a pattern; compare floats
with if. A bare name still binds only at a payload position, so the catch-all for
the whole value is case _:, or case _ as name: to bind
it.
Each literal case costs one comparison, tried from the top, exactly like an
if/elif chain. Put the common cases first in a long match.
Bind the value a match leaves over
case _: covers the remaining values but names nothing. To handle a few
variants and pass every other value on, end with case _ as name:, which binds
the whole matched value:
enum Event:
Tick
Key(str)
Resize(i64, i64)
def describe(event: Event) -> str:
match event:
case Event.Tick:
"tick"
case Event.Key(name):
name
case Event.Resize(_, _):
"resize"
def without_ticks(event: Event) -> str:
match event:
case Event.Tick:
"ignored"
case _ as other:
describe(other)
def main() -> Result[(), IoError]:
print(without_ticks(Event.Tick))?
print(without_ticks(Event.Key("enter")))?
print(without_ticks(Event.Resize(80, 24)))?
Ok(())
ignored
enter
resize
without_ticks keeps working when Event gains a variant; only describe
must name it.
The binding follows the matched value. Matching an owned value moves it into
the name; matching &value or &mut value binds a shared or exclusive
reference to it:
def bump(count: &mut Option[i64]) -> Result[(), OverflowError[i64]]:
match count:
case Some(number):
*number = *number + 1
case _ as empty:
*empty = Some(1)
Ok(())
def main() -> Result[(), IoError | OverflowError[i64]]:
mut count: Option[i64] = Nothing
bump(&mut count)?
bump(&mut count)?
print(count)?
Ok(())
Option[i64].Some(2)
_ as name matches every value, so it is the last case and stands alone: a
case after it is unreachable, and it cannot be a | alternative. The _ as is
required. A bare name is rejected, because it is usually a variant written
without its enum:
enum Switch:
On
Off
def main() -> ():
match Switch.On:
case On:
pass
case _:
pass
a case cannot bind the whole value to the bare name `On`; bind it with `_ as On`, match any value with `_`, or qualify a variant with its enum type
Inside a payload a name alone binds, as in Event.Key(name); _ as name is
only for the whole value.
Bind the value a match selects
A match at the end of a function supplies the function’s result. To use the
selected value in the middle of a function, write the match as the value of a
binding. The final expression of the selected case becomes the value:
enum Shape:
Dot
Square(i64)
Rect(i64, i64)
def paint_cost(shape: Shape) -> Result[i64, OverflowError[i64]]:
area = match shape:
case Shape.Dot:
return Ok(1)
case Shape.Square(side):
side * side
case Shape.Rect(width, height):
width * height
Ok(area * 10)
def main() -> Result[(), IoError | OverflowError[i64]]:
print(paint_cost(Shape.Dot)?)?
print(paint_cost(Shape.Square(3))?)?
print(paint_cost(Shape.Rect(2, 5))?)?
Ok(())
1
90
100
The cases that continue must produce one type. A case may instead leave with
return, break, continue, or ?, as Shape.Dot does; it then supplies no
value and nothing is bound.
The same block is the value of an assignment to a mut local, a field, or a
collection element, and of return. An annotation gives every case its
expected type, which Nothing needs:
def half(number: i64) -> Option[i64]:
two: Positive[i64] = 2
return match number % two:
case 0:
Some(number // two)
case _:
Nothing
def main() -> Result[(), AllocError | IoError]:
mut halves = [0, 0]?
for index in range(2):
found: Option[i64] = match index:
case 0:
half(8)
case _:
Nothing
halves[index] = match found:
case Some(value):
value
case Nothing:
-1
print(halves)?
print(half(3))?
Ok(())
[4, -1]
Option[i64].Nothing
return match returns the final expression of the selected case, so nothing
after the block runs.
The block must be the whole right-hand side. Inside a larger expression it is rejected; bind it first:
def main() -> Result[(), IoError]:
number = 3
doubled = 2 * match number:
case 3:
30
case _:
0
print(doubled)?
Ok(())
a `match` block is a value only as the whole right side of a binding, a single-target assignment, or `return`; bind its value to a name first
Cases that disagree are rejected at the case that differs:
def main() -> Result[(), IoError]:
number = 3
label = match number:
case 3:
"three"
case _:
0
print(label)?
Ok(())
expected str, got i64
The reference lists the remaining rules, including why the value cannot be a reference.
Creating enum values
Payload arguments move into the enum. Constructing it adds no allocation:
enum Message:
Empty
Text(str)
def main() -> Result[(), IoError]:
print(Message.Text("hello"))?
print(Message.Empty)?
Ok(())
Message.Text("hello")
Message.Empty
Allocations inside argument expressions, such as building a string, still
return their own Result.
Match borrowed values
Use match &value to inspect an enum while retaining its owner. Payload
bindings are shared references. With match &mut value, they are exclusive
references and can change the original payload.
An existing enum reference can be matched directly. A function can also return a payload reference when it comes from the function’s first reference parameter:
def number(value: &Result[i64, i64]) -> &i64:
match value:
case Ok(payload):
payload
case Err(payload):
payload
def main() -> Result[(), IoError]:
mut outcome: Result[i64, i64] = Err(7)
selected = number(&outcome)
print(*selected)?
outcome = Ok(9)
print(number(&outcome))?
Ok(())
7
9
Replacing outcome is allowed after selected’s last use; doing it before that
use is rejected. References cannot outlive their owner.
Assign through a mutable payload reference to change the original slot, keeping its enclosing variants:
def main() -> Result[(), IoError | OverflowError[i64]]:
mut nested: Option[Result[i64, str]] = Some(Ok(4))
match &mut nested:
case Some(Ok(number)):
*number = *number + 1
case Some(Err(_)) | Nothing:
pass
print(nested)?
Ok(())
Option[Result[i64, str]].Some(Result[i64, str].Ok(5))
match &mut works for every enum, whatever its payloads contain. A copy made
before the match keeps its own payload:
enum Shape:
Circle(i64)
Square(i64)
def main() -> Result[(), IoError | OverflowError[i64]]:
mut shape = Shape.Circle(3)
before = shape
match &mut shape:
case Shape.Circle(radius):
*radius = *radius + 1
case Shape.Square(side):
*side = *side * 2
print(shape)?
print(before)?
Ok(())
Shape.Circle(4)
Shape.Circle(3)
_ ignores a payload without moving it. Borrowed matching uses the same
exhaustiveness checks and qualified variant names as owned matching.
Clean up owned instances
Use __del__ for cleanup that belongs to an owned instance. It runs automatically
when the instance leaves scope, is replaced, or is explicitly dropped. Its bare
self is an exclusive reference, like in __init__:
class Resource:
name: str
def __del__(self) -> ():
drop(print("release", self.name))
class Pair:
first: Resource
second: Resource
def __del__(self) -> ():
drop(print("release pair"))
def work() -> Result[(), IoError]:
pair = Pair(Resource("first"), Resource("second"))
spare = Resource("spare")
print("working")?
Ok(())
def main() -> Result[(), IoError]:
work()?
print("done")?
Ok(())
working
release spare
release pair
release first
release second
done
Locals clean up in reverse declaration order. A class’s __del__ runs first;
its fields then clean up automatically in declaration order. Moving an instance
transfers cleanup responsibility without running the destructor.
Each field finishes cleaning up its children before the next field begins:
class Leaf:
name: str
def __del__(self) -> ():
drop(print(self.name))
class Pair:
first: list[Leaf]
second: Leaf
def main() -> Result[(), AllocError]:
pairs = [Pair([Leaf("a")]?, Leaf("b"))]?
drop(pairs)
Ok(())
a
b
Lifecycle methods are not called directly. A class with custom cleanup, or one
containing a value with custom cleanup, cannot be copied yet. A destructor cannot
yield or report a recoverable error; use an explicit closing method returning
Result when failures matter. Fatal traps do not run cleanup.
The lifecycle trace uses drop(print(...)) deliberately: cleanup cannot return
an error, and a failed diagnostic write must not interrupt cleanup. For output
that must succeed, provide an explicit fallible operation and propagate its
result before leaving the scope.
Consume a value to change its state
Use self: Class when an operation finishes an object or replaces it with a
different state. The method receives ownership, so the caller cannot use the
old class instance again. Bare self still borrows; self: &mut Class mutates
an instance that remains available to the caller.
Here a draft must be validated before it can be finished. Separate classes make
invalid operation ordering a type error. Keeping Ready’s field and constructor
private prevents callers from skipping validation:
pub class Draft:
pub text: str
pub def validate(self: Draft) -> Result[Ready, Draft]:
if self.text == "":
return Err(self)
Ok(Ready(self.text))
pub class Ready:
text: str
pub def finish(self: Ready) -> str:
self.text
import workflow
def main() -> Result[(), IoError]:
draft = workflow.Draft("report")
ready = draft.validate().unwrap()
print(ready.finish())?
# An error can explicitly return the draft for correction and retry.
match workflow.Draft("").validate():
case Ok(ready):
print(ready.finish())?
case Err(retry):
mut corrected = retry
corrected.text = "corrected"
print(corrected.validate().unwrap().finish())?
Ok(())
report
corrected
The unwrap calls assert that the nonempty strings in this example pass
validation. For input that might fail, recover the owner using match, as the
empty-draft example does. A fallible method does not automatically restore the
receiver after failure. The same rule applies to a simple finalization method:
class Message:
text: str
def finish(self: Message) -> str:
self.text
def main() -> ():
message = Message("report")
text = message.finish()
print(message.text).unwrap()
drop(text)
use of moved binding `message`
Ownership moves into the method before its other arguments are evaluated.
Read any needed fields into locals before making the consuming call. If a later
argument returns early with ?, the pending receiver is cleaned up. Inside the
method, owned self is an immutable binding; move it into a mut local if you
need to change fields.
This example reads a copyable string field. A transition can also move an owned field, such as a list, out of the consumed receiver into the next state:
class Pending:
id: i64
items: list[str]
def ship(self: Pending) -> Shipped:
Shipped(self.id, self.items)
class Shipped:
id: i64
items: list[str]
def main() -> Result[(), AllocError | IoError]:
order = Pending(7, ["book", "pen"]?)
shipped = order.ship()
print(shipped.id, len(shipped.items))?
Ok(())
7 2
self.items transfers the list without copying it, and the rest of self is
cleaned up when the method returns. After a field moves out, the object cannot
be used as a whole until that field is assigned again:
class Pending:
items: list[str]
def ship(self: Pending) -> Pending:
items = self.items
drop(items)
self
def main() -> ():
pass
use of partly moved binding `self`: `self.items` is moved; assign `self.items` before using `self` as a whole
Fields cannot move out through a reference, or out of a class that defines
__del__, because its hook receives the whole object. There, keep the field
initialized with Option plus
replace.
See moving fields
for the full rules.
Consuming a value does not require callers to finish it: they can still drop it. Normal cleanup runs exactly once unless the method transfers ownership onward. Enums also accept by-value receivers, but a copyable enum is copied according to ordinary argument rules. Use affine state types when the old state must cease to be usable. See method contracts.
Borrow records in a collection
For a list of records, &items yields shared references and &mut items yields
mutable references. The list cannot grow, shrink, or move during either loop:
class Score:
value: i64
def main() -> Result[(), AllocError | IoError | OverflowError[i64]]:
mut scores = [Score(2), Score(5)]?
for score in &mut scores:
score.value = score.value + 1
print([score.value for score in &scores]?)?
mut numbers = [10, 20]?
for number in &mut numbers:
*number = *number + 2
print(numbers)?
Ok(())
[3, 6]
[12, 22]
For large primitive-field records scanned a few fields at a time, consider columnar storage. Ordinary lists retain the full record-borrowing interface used here.
Heterogeneous collections
A heterogeneous collection holds values of different kinds. Plenty still gives it one element type: an enum or an anonymous sum describes the alternatives. The possible types are known at compile time.
Use the classes and enums you have defined to choose a representation:
| Collection | Representing different kinds of data |
|---|---|
| Lists | Use an enum wrapper for a named interface, or `list[A |
| Sets | Use an enum or anonymous sum whose alternatives are supported keys. |
| Dictionaries | Use enums or anonymous sums for mixed keys, values, or both. |
An anonymous sum, written A | B, holds one value of either type. It is not
a set collection: set[T] holds distinct values. An enum gives its alternatives
names and can define methods; an anonymous sum needs a match to use each
member’s own interface.
These techniques do not create an open-ended collection accepting any type.
Adding a new kind means extending the enum or sum and handling it wherever
matching must be exhaustive. Protocols constrain
generic calls; they are not runtime element types such as list[MyProtocol].
Lists of different kinds of values
A list has one element type, but that type can describe alternatives. Use a wrapper enum when the alternatives form a concept in your program and share operations. Use an anonymous sum when a local choice of whole types is enough.
Give the alternatives a shared interface
A message and a status can both appear in a feed. FeedItem names that concept
and keeps the dispatch in its label method:
class Message:
text: str
def label(self) -> str:
self.text
enum Status:
Ready
Busy
def label(self) -> str:
match self:
case Status.Ready:
"ready"
case Status.Busy:
"busy"
enum FeedItem:
Message(Message)
Status(Status)
def label(self) -> str:
match self:
case FeedItem.Message(message):
message.label()
case FeedItem.Status(status):
status.label()
def main() -> Result[(), AllocError | IoError]:
mut items: list[FeedItem] = []?
items.append(FeedItem.Message(Message("hello")))?
items.append(FeedItem.Status(Status.Ready))?
for item in &items:
print(item.label())?
Ok(())
hello
ready
The wrapper can carry class instances, enum values, or other owned data.
Constructing a variant moves an affine payload into it; appending moves that
wrapper into the list. The loop borrows the list, and matching self borrows
the active payload, so neither operation needs a copy.
Each kind has its own implementation of label, but the wrapper’s method is
what lets the loop call one interface. No protocol declaration is necessary.
Adding a variant makes the compiler check that the wrapper’s match handles it.
Use an anonymous sum for a local choice
If you do not need a named wrapper or its methods, declare the alternatives in
the list’s element type. The annotation lets append accept either member
directly:
class Message:
text: str
enum Status:
Ready
Busy
def main() -> Result[(), AllocError | IoError]:
mut items: list[Message | Status] = []?
items.append(Message("hello"))?
items.append(Status.Ready)?
for item in &items:
match item:
case message: Message:
print(message.text)?
case status: Status:
match status:
case Status.Ready:
print("ready")?
case Status.Busy:
print("busy")?
Ok(())
hello
ready
case message: Message: selects a whole member type. Because item is a
borrow, its matched payload is borrowed too. The Status member contains any
of that enum’s variants, so its own match handles both.
The sum does not automatically forward methods, even if every member has a method with the same signature. Match to select a member, or choose the wrapper pattern when several callers need the same operation.
Both forms store a tag and payload inline; wrapping adds no heap allocation. List creation and growth still return allocation results, as does constructing any payload that itself needs storage. See anonymous sums and enums for the full contracts.
For implementations supplied independently of a fixed list of alternatives, see protocol objects. Each object uses an explicit heap allocation and exposes the methods of one protocol.
Sets and different kinds of values
Sets accept named enums when every payload is a supported key type: integers,
bool, str, bytes, or tuples, classes, enums and anonymous sums of those types. The variant
is part of the identity, so different variants with equal payloads remain distinct.
enum Selection:
Account(i64)
Group(str)
All
def main() -> Result[(), AllocError | IoError]:
mut selected: set[Selection] = set()?
selected.add(Selection.Account(7))?
selected.add(Selection.Group("7"))?
selected.add(Selection.Account(7))?
print(len(selected))?
print(Selection.Account(7) in selected)?
Ok(())
2
True
This is one element type with several alternatives. Enums carrying collections, floats or boxes cannot be set elements. Anonymous sums qualify when every member is a supported key type.
Class payloads
A wrapper can also carry classes whose fields are supported keys. Ordinary classes move into the set; use a borrowed probe to keep a lookup value:
class Account:
region: str
number: i64
enum Selection:
Account(Account)
Group(str)
def main() -> Result[(), AllocError | IoError]:
selected = {Selection.Account(Account("eu", 7)), Selection.Group("editors") }?
probe = Selection.Account(Account("eu", 7))
print(&probe in selected)?
for item in &selected:
match item:
case Selection.Account(account):
print(account.region, account.number)?
case Selection.Group(group):
print(group)?
Ok(())
True
eu 7
editors
The lookup compares fields rather than object addresses. Borrowed iteration keeps the stored keys immutable. For keys without custom cleanup, set algebra creates independent key copies; its result can outlive either input. See sets.
Keep each kind in its own set
Suppose a selection contains numeric account IDs and textual group names. Separate sets preserve both kinds without converting IDs into strings:
def main() -> Result[(), AllocError | IoError]:
mut account_ids: set[i64] = set()?
mut group_names: set[str] = set()?
account_ids.add(7)?
account_ids.add(7)?
group_names.add("editors")?
group_names.add("7")?
print(len(account_ids), len(group_names))?
print(7 in account_ids, "7" in group_names)?
Ok(())
1 2
True True
The integer 7 and the name "7" remain distinct identities. Each set removes
its own duplicates and supports the ordinary
set operations.
Group the two sets in a class if they need to travel together.
Use an anonymous sum
When the types themselves express the alternatives, no wrapper is needed:
def main() -> Result[(), AllocError | IoError]:
mut selected: set[i64 | str] = {7, "editors"}?
selected.add("7")?
selected.add(7)?
print(len(selected), 7 in selected, "7" in selected)?
Ok(())
3 True True
The member type is part of each key’s identity. Every member must support
structural hashing; set[i64 | f64] is rejected because floats are not keys.
Use a named enum when alternatives need different meanings despite sharing a
payload type. Anonymous sums use hash sets, including sums of finite enums.
Small enums without payloads
Use ordinary sets for flags or selections too. An enum with no payloads and at most 256 variants automatically gets compact bitset storage. The compiler selects a 64-, 128-, or 256-bit mask from the number of possible values:
enum Permission:
Read
Write
Execute
def main() -> Result[(), AllocError | IoError]:
mut granted = {Permission.Read}?
granted.add(Permission.Write)?
required = {Permission.Read, Permission.Write}?
print(required.issubset(&granted))?
granted.discard(Permission.Write)
print(Permission.Write in granted)?
Ok(())
True
False
Keep the usual ? handling: constructing a set still allocates its owner.
Adding members to this representation needs no further allocation. The same
programming style works for larger enums and other payload types, which use
hash sets. See small enum bitsets.
Small finite payloads
Payloads can use bitsets too when the complete type has at most 256 possible
values. Booleans and nested enums or tuples of these types qualify.
Here State has three values: Idle, Enabled(False), and Enabled(True).
enum State:
Idle
Enabled(bool)
def main() -> Result[(), AllocError | IoError]:
mut allowed = {State.Idle, State.Enabled(False)}?
allowed.add(State.Enabled(True))?
print(len(allowed), State.Enabled(True) in allowed)?
Ok(())
3 True
The limit counts all possible payload combinations, regardless of how many members the set currently contains. For example, eight boolean fields have 256 combinations; adding a separate payload-free variant makes 257 and selects a hash set. Integer and string payloads use hash sets. No special container or manual integer encoding is needed.
Keys with custom cleanup
A key may define __del__, or contain a value that does. Its fields must still
be supported key types. The collection owns the inserted key and runs cleanup
when that owner is removed or destroyed. Borrow keys for lookup and inspection;
transfer them when a second owner cannot be created.
This example logs cleanup to make ownership visible. It unwraps diagnostic printing inside the destructor because cleanup cannot return an error:
class Ticket:
id: i64
def __del__(self) -> ():
print("released", self.id).unwrap()
def main() -> Result[(), AllocError | IoError]:
mut selected = {Ticket(7)}?
additional = {Ticket(8)}?
selected.update(additional)?
print(len(selected))?
for ticket in &selected:
print(ticket.id)?
# Transfer the owners out of the set before destroying them.
for ticket in selected:
drop(ticket)
Ok(())
2
7
8
released 7
released 8
Iteration order is unspecified. update consumes the source set; an equal
incoming key is destroyed while the stored owner is retained. Use
intersection_update and difference_update to filter in place, and relationship
checks such as issubset to compare without copying.
union, intersection, difference, symmetric_difference, copy(selected)
and dictionary keys().to_set() require independent key copies, so they reject
custom cleanup, including cleanup nested inside an enum or anonymous sum.
A borrowed keys() view still works. To keep dictionary keys after consuming
the dictionary, use into_keys(); see dictionary snapshots and views.
Dictionaries with different kinds of values
A dictionary has one key type and one value type. The value type can be an enum
wrapper or an anonymous sum, so a string key can select any of several kinds of
stored data. Keys retain the narrower key-type contract:
integers, bool, str, bytes, or tuples, classes, enums and anonymous sums
recursively containing those types. Keys with custom cleanup can be borrowed through keys() or
transferred into an owned list with into_keys(); they cannot be copied with
keys().to_set(). See cleanup-bearing keys.
Give mixed values a named wrapper
An enum makes the kinds of settings explicit. Here each value carries either a retry policy class or a mode enum:
class RetryPolicy:
attempts: i64
enum Mode:
Fast
Careful
enum Setting:
Retries(RetryPolicy)
Mode(Mode)
def main() -> Result[(), AllocError | IoError]:
mut settings: dict[str, Setting] = {}?
settings.insert("network", Setting.Retries(RetryPolicy(3)))?
settings.insert("processing", Setting.Mode(Mode.Careful))?
for name, setting in settings.items():
match setting:
case Setting.Retries(policy):
print(*name, policy.attempts)?
case Setting.Mode(mode):
print(*name, *mode)?
Ok(())
network 3
processing Mode.Careful
insert adds or replaces an entry and handles possible allocation with ?.
The dictionary owns each wrapper and its payload. items() borrows them in
insertion order, so inspecting a class payload does not copy or remove it.
A wrapper can also provide shared methods, as in the list example.
For a lookup that may miss, get requires a copyable value: this Setting
contains an affine class, so get cannot return it. Use get_mut to borrow an
optional value for an update, or pop to remove and take ownership of it.
See dictionary lookup and removal.
Use a sum for simple values
An anonymous sum avoids declaring variants when the whole types express the choice. The annotation supplies the expected type for both a literal and later insertions:
def main() -> Result[(), AllocError | IoError]:
mut settings: dict[str, i64 | str] = {"retries": 3}?
settings.insert("mode", "careful")?
for name, value in settings.items():
match value:
case count: i64:
print(*name, *count)?
case text: str:
print(*name, *text)?
Ok(())
retries 3
mode careful
The sum admits only the listed types. If two alternatives have the same payload
type but different meanings, such as seconds and attempts both stored as i64,
use separate named enum variants instead: i64 | i64 is just i64.
Different kinds of keys
An anonymous sum can combine supported key types without merging their identities:
def main() -> Result[(), AllocError | IoError]:
labels: dict[i64 | str, str] = {7: "numeric account", "7": "named account"}?
print(labels[7], labels["7"])?
Ok(())
numeric account named account
A named enum can represent these distinct key domains while preserving their identity:
enum Key:
Id(i64)
Name(str)
def main() -> Result[(), AllocError | IoError]:
labels: dict[Key, str] = {
Key.Id(7): "numeric account",
Key.Name("7"): "named account",
}?
print(labels[Key.Id(7)], labels[Key.Name("7")])?
Ok(())
numeric account named account
Every enum payload must itself be a supported key type. You can also keep separate dictionaries when numeric IDs and textual names are different lookup domains:
def main() -> Result[(), AllocError | IoError]:
by_id: dict[i64, str] = {7: "numeric account"}?
by_name: dict[str, str] = {"7": "named account"}?
print(by_id.get(7).unwrap(), by_name.get("7").unwrap())?
Ok(())
numeric account named account
These lookups unwrap because the example has just inserted both keys. For
external input, match the Option or supply an intentional default instead.
Do not merge distinct identities by converting every key to a string.
Different kinds on both sides
Keys and values can independently use sums. Only the key alternatives must satisfy the key-type contract:
def main() -> Result[(), AllocError | IoError]:
mut settings: dict[i64 | str, i64 | str] = {7: "paused", "retries": 3}?
settings.insert("mode", "careful")?
print(settings[7], settings["retries"], settings["mode"])?
Ok(())
"paused" 3 "careful"
The annotations provide the expected types for insertion and lookup. The
integer key 7 remains distinct from the string key "7". Use a named enum
when domain-specific variant names make the meaning clearer.
For implementations supplied independently of a fixed list of alternatives, see protocol objects. Each object uses an explicit heap allocation and exposes the methods of one protocol.
Special-purpose data structures
Choose a data structure to suit the operations your program performs most often. This section introduces collections with specialized storage and access patterns, along with the tradeoffs that make them useful.
Start with Columnar lists: keep an ordinary class-shaped interface while storing each primitive field in its own contiguous column.
For analytical queries and Parquet datasets, use Dataframes: explicit expression plans executed by an optional Polars engine. Columnar lists remain the lightweight mutable collection.
Query dataframes with Polars
Use std.dataframe when your task is an analytical query over columns or
Parquet files. Plenty describes the query; a separately compiled Polars
engine optimizes and executes it. This exposes native columnar processing,
parallelism and SIMD without asking you to manage vector instructions.
Linux binary releases include the engine
beside the compiler; no local Polars build is needed. When distributing a compiled application, follow
shipping dataframe applications
to include its shared library and notices.
These examples require it when compiled and run.
clist[T] remains useful for lightweight mutable records, a compile-time
class schema and ordinary Plenty computations. A dataframe has a runtime
schema and native query semantics. It is a different tool for larger
analytical operations, not the default replacement for a collection.
Build a query
Copy numeric lists into named columns, create a frame, and build expressions.
An expression is a native query description, not a Plenty function called
once per row. ? handles query errors, separately from null values.
Borrowed parameters accept values directly, so explicit & is optional here.
import std.dataframe as df
def main() -> Result[(), df.DataFrameError | AllocError | IoError]:
readings = [18.0, 23.0, 29.0]?
columns = [df.column_f64("temperature", readings)?]?
data = df.frame(columns)?
temperature = df.col("temperature")?
warm = temperature.gt(df.lit(20.0)?)?
aggregates = [temperature.mean()?.alias("average")?]?
result = data.lazy()?.filter(warm)?.select(aggregates)?.collect()?
print(result.get_f64("average", 0)?)?
Ok(())
Option[f64].Some(26.0)
Column construction copies values in bulk. Plans borrow and share native
buffers; chaining operations does not copy the whole dataset at every step.
collect() executes the plan. A getter returns Result[Option[T], ...]:
Nothing means a null cell, while Err means the operation failed.
Bulk values_f64 and values_i64 return ordinary Plenty lists, but reject
columns containing nulls. Their _nullable variants return Option elements.
Strings and booleans have matching typed APIs; see
Types, nulls and schemas.
Derive columns and group
Select explicit expressions and names. Group order is unspecified, so this example prints the number of groups rather than assuming which comes first.
import std.dataframe as df
def main() -> Result[(), df.DataFrameError | AllocError | IoError]:
stations = [1, 2, 1]?
readings = [10.0, 20.0, 30.0]?
columns = [df.column_i64("station", stations)?, df.column_f64("reading", readings)?]?
data = df.frame(columns)?
derived = [df.col("reading")?.mul(df.lit(2.0)?)?.alias("adjusted")?]?
keys = [df.col("station")?]?
totals = [df.col("adjusted")?.sum()?.alias("total")?]?
result = data.lazy()?.with_columns(derived)?.group_by(keys, totals)?.collect()?
print(result.height()?)?
data.write_parquet("readings.parquet")?
restored = df.scan_parquet("readings.parquet")?.collect()?
print(restored.values_f64("reading")?)?
Ok(())
2
[10.0, 20.0, 30.0]
For an existing Parquet file, start with scan_parquet, add filters and
projections, then collect. Polars can push those operations into the scan.
The write above creates or overwrites its file.
Native query semantics differ from Plenty scalar arithmetic: nulls and NaN are distinct, NaN ordering differs, integer overflow is not checked like Plenty arithmetic, and parallel floating-point reductions can change rounding. Native allocation failure can abort the process. Read the reference contracts before using these operations where those distinctions matter.
Continue with Join and rank results to combine datasets, order results inside Polars and select a row window. For cleaning text, conditional values and per-group comparisons, see Clean and compare column values. Query CSV and write Parquet adds file-backed pipelines and explains when to collect or write directly.
Types, nulls and schemas
A missing reading differs from a reading of zero. Use Option elements
to preserve that distinction, and keep names and flags in their native
string and boolean columns.
This example keeps enabled sensors and reads their nullable values back. The expressions run in Polars; ordinary Plenty code handles the results. It needs the dataframe engine, which Linux releases include.
import std.dataframe as df
def main() -> Result[(), df.DataFrameError | AllocError | IoError]:
names = ["north", "café", "南"]?
enabled = [True, False, True]?
readings: list[Option[f64]] = [Some(18.5), Some(22.0), Nothing]?
data = df.frame([
df.column_str("sensor", names)?,
df.column_bool("enabled", enabled)?,
df.column_f64_nullable("reading", readings)?
]?)?
result = data.lazy()?.filter(df.col("enabled")?)?.collect()?
print(result.values_str("sensor")?)?
print(result.values_f64_nullable("reading")?)?
for field in result.schema()?:
print(field.name, field.dtype)?
Ok(())
["north", "南"]
[Option[f64].Some(18.5), Option[f64].Nothing]
sensor std.dataframe.DType.String
enabled std.dataframe.DType.Bool
reading std.dataframe.DType.F64
Nothing is missing data; Some(""), Some(False), zero and NaN are
present values. Use values_TYPE_nullable for nullable results.
The plain values_TYPE methods reject nulls rather than dropping rows.
Scalar get_TYPE(name, row) returns an Option inside a Result:
absence and query failure are different outcomes.
For an unfamiliar file, inspect df.scan_parquet(path)?.schema()? before
choosing columns or collecting. Its ordered fields describe the native
runtime schema. DType.Other(description) reports a native type that
Plenty cannot yet extract, such as an unsupported nested value.
Inspection reads metadata as needed; collection can still encounter errors.
Keep numeric widths and unsigned identifiers
Every Plenty numeric type has matching column constructors and getters.
Choose the type your data needs: a u64 identifier can exceed i64.MAX,
while f32 readings use less native column memory than f64 readings.
Neither needs a conversion through the default numeric types.
import std.dataframe as df
def main() -> Result[(), df.DataFrameError | AllocError | IoError]:
data = df.frame([
df.column_u64("device", [u64.MAX, 42u64]?)?,
df.column_f32("reading", [18.5f32, 22.0f32]?)?
]?)?
selected = data.lazy()?.filter(df.col("device")?.eq(df.lit_u64(u64.MAX)?)?)?.collect()?
print(selected.values_u64("device")?)?
print(selected.values_f32("reading")?)?
Ok(())
[18446744073709551615]
[18.5]
Use column_TYPE_nullable and values_TYPE_nullable when values may be
missing. Extraction checks the exact dtype; it does not widen integers
or floats implicitly. An expression or aggregation can change the result’s
dtype, so inspect its schema or make a deliberate cast before extracting.
Lists are copied across the native boundary. Keep filtering, grouping and sorting inside the query, then extract only the columns and rows your Plenty code needs. See the type contracts for Unicode, ownership and conversion details, then continue with Join and rank results or Clean and compare column values. For logical calendar data, see Dates and daily reports.
Store binary payloads
Use binary columns for file contents, encoded messages, or other bytes that
are not necessarily text. Values use owned Bytes
buffers; there is no UTF-8 conversion at the dataframe boundary.
This lesson requires the optional engine.
Keep selection inside Polars, then extract only the payloads your application needs. Here, null means missing data and an empty buffer is a present payload:
import std.dataframe as df
import std.bytes as buffers
from std.bytes import Bytes, BytesError
def main() -> Result[(), df.DataFrameError | BytesError | AllocError | IoError]:
payloads: list[Option[Bytes]] = [
Some(buffers.from_bytes(b"\xff\0")?),
Nothing,
Some(buffers.empty()?)
]?
data = df.frame([df.column_binary_nullable("payload", payloads)?]?)?
matches = data.lazy()?.filter(
df.col("payload")?.eq(df.lit_binary(b"\xff\0")?)?
)?.collect()?
print(matches.height()?)?
for payload in matches.values_binary("payload")?:
print(payload.len(), payload[0])?
print(data.get_binary("payload", 1)?.is_none())?
print(data.get_binary("payload", 2)?.unwrap().len())?
Ok(())
1
2 255
True
0
Use values_binary_nullable when the selected result may contain nulls.
values_binary rejects nulls; it never silently drops rows. Both return new
buffers that can outlive the dataframe and be changed independently.
write_parquet and scan_parquet preserve binary values and nulls. CSV does
not support binary columns. Converting a binary expression to DType.String
is explicit and checks UTF-8; it is unnecessary for writing Parquet.
Binary literals accept a byte literal or borrowed view directly, including
&buffer[a:b]. Column construction packs values into a temporary Bytes
buffer and copies them into Polars; extraction returns independent buffers.
Prefer native filtering/projection before extraction, especially for large
payloads. See binary transfer details.
Lists and records inside columns
A row can contain several values or a record of related fields. Keep these values in native columns when queries will process them. Nested columns compose with the same expressions and Parquet I/O as scalar columns.
Store a list per row
Build one flat child column and supply the boundaries between rows. In this
example the rows contain [8, 5], [], and [3].
import std.dataframe as df
def main() -> Result[(), df.DataFrameError | AllocError | IoError]:
values = df.column_i64("item", [8, 5, 3]?)?
scores = df.column_list("scores", values, [0, 2, 2, 3]?)?
report = scores.to_frame()?.lazy()?.select([
df.col("scores")?.list_len()?.alias("count")?,
df.col("scores")?.list_get(-1)?.alias("last")?
]?)?.collect()?
print(report.values_i64("count")?)?
print(report.values_i64_nullable("last")?)?
Ok(())
[2, 0, 1]
[Option[i64].Some(5), Option[i64].Nothing, Option[i64].Some(3)]
list_get returns null for an out-of-bounds index. Use nullable extraction when
an empty or null list can reach the query. A null child element also remains
null; it is not replaced with zero.
column_list_nullable additionally accepts a boolean per row. False marks a
null list; its offsets must describe an empty range. True with an empty range
is an empty list. These states remain distinct through Parquet.
Group fields into records
A child dataframe supplies the fields of a struct column. Select a field with
struct_field; it stays a native expression.
import std.dataframe as df
def main() -> Result[(), df.DataFrameError | AllocError | IoError]:
fields = df.frame([
df.column_str("name", ["Ada", "Lin"]?)?,
df.column_i64("score", [8, 5]?)?
]?)?
people = df.column_struct("person", fields)?.to_frame()?
result = people.lazy()?.select([
df.col("person")?.struct_field("name")?.alias("name")?
]?)?.collect()?
print(result.values_str("name")?)?
for node in people.schema_tree()?:
print(node.parent, node.name, node.dtype)?
Ok(())
["Ada", "Lin"]
-1 person std.dataframe.DType.Struct
0 name std.dataframe.DType.String
0 score std.dataframe.DType.I64
The schema tree uses parent indexes: -1 is a top-level column, and both fields
above belong to node 0. Lists introduce an item child, which may itself be
a struct or another list.
When Plenty needs the values, frame.column(name)?.list_parts() returns native
child values plus owned offsets and parent validity. Its child column is named
item: use parts.values.to_frame()?.values_i64_nullable("item") for nullable
integer elements. struct_parts() similarly returns a child dataframe and the
parent mask. Prefer native expressions for filtering or aggregation; extracting
parts is for application code that actually needs them.
Read the nested-column contracts for ownership, null semantics, reconstruction and Polars’ Parquet map representation.
Join and rank dataframe results
An order table and a price table usually have different shapes. Join them by
product, derive revenue, then group and rank the results. These operations stay
inside the optional Polars engine until collect().
Methods borrow their inputs automatically, including temporary expressions. The example uses named intermediate plans to keep the steps readable; each step still belongs to one lazy query.
import std.dataframe as df
def main() -> Result[(), df.DataFrameError | AllocError | IoError]:
orders = df.frame([df.column_i64("product", [1, 2, 1, 3]?)?, df.column_i64("units", [2, 1, 3, 5]?)?]?)?
prices = df.frame([df.column_i64("product", [1, 2]?)?, df.column_f64("price", [10.0, 25.0]?)?]?)?
keys = [df.col("product")?]?
joined = orders.lazy()?.join(prices.lazy()?, keys, keys, how=df.JoinKind.Left)?
revenue = df.col("price")?.fill_null(df.lit(0.0)?)?.mul(df.col("units")?)?.alias("revenue")?
enriched = joined.with_columns([revenue]?)?
totals = enriched.group_by(keys, [df.col("revenue")?.sum()?.alias("total")?]?)?
ranking = [df.SortKey(df.col("total")?, descending=True), df.SortKey(df.col("product")?)]?
best = totals.sort(ranking)?.limit(2)?.collect()?
print(best.values_i64("product")?)?
print(best.values_f64("total")?)?
Ok(())
[1, 2]
[50.0, 25.0]
The left join keeps product 3 even though it has no price. This example
deliberately treats a missing price as zero with fill_null; a real
application may instead reject or separately report unpriced orders.
The default inner join would drop unmatched orders.
Duplicate keys produce every matching pair. The price table here has one
row per product; duplicate price rows would multiply revenue contributions.
The join does not validate uniqueness. Null keys do not match unless you
pass nulls_equal=True.
Both key columns survive the join. Since both are named product, the
right one becomes product_right; use suffix="_price" to choose another
suffix. A suffix collision is an error, never silent replacement.
The ranking sorts totals descending, then product ascending to break ties.
Nulls go last by default; each SortKey can specify nulls_last=False.
Sorting is stable for equal keys, but a join or group has no promised input
order, so use a meaningful tie-breaking key when you need reproducible output.
For pagination, replace limit(2) with slice(2, 2) to skip two rows and
take up to two more. Dataframe slicing takes an offset and count, not
start/stop endpoints. Negative offsets count from the end; invalid negative
lengths return an error. See the query contracts.
Continue with Membership and full joins to filter by another dataset or reconcile unmatched rows from both sides.
Membership and full joins
Use a semi join to keep rows whose keys appear in another dataset, or an anti join to find keys that are absent. Both return only the left columns. They are useful for checking allowed IDs, finding missing records and filtering against a lookup table without multiplying rows.
These examples require the optional engine.
Keep matching rows or find missing ones
import std.dataframe as df
def main() -> Result[(), df.DataFrameError | AllocError | IoError]:
orders = df.frame([
df.column_i64("id", [1, 2, 2, 3]?)?,
df.column_str("order", ["one", "two-a", "two-b", "three"]?)?
]?)?
allowed = df.frame([df.column_i64("id", [2, 2]?)?]?)?
keys = [df.col("id")?]?
order = [df.SortKey(df.col("order")?)]?
matching = orders.lazy()?.join(
allowed.lazy()?, keys, keys, how=df.JoinKind.Semi
)?.sort(order)?.collect()?
missing = orders.lazy()?.join(
allowed.lazy()?, keys, keys, how=df.JoinKind.Anti
)?.sort(order)?.collect()?
print(matching.values_str("order")?)?
print(missing.values_str("order")?)?
Ok(())
["two-a", "two-b"]
["one", "three"]
The duplicate allowed ID does not duplicate matching orders. Each left row is kept at most once; duplicate left rows would still be separate rows. No right-side columns or suffixes appear in the result.
Null keys do not match by default, so a null left key goes into the anti
result. Set nulls_equal=True only when null should be an ordinary matching
key. Sorting above makes presentation deterministic.
Reconcile both sides with a full join
A full join also keeps rows found only on the right. Both key columns remain, so explicitly derive a common display key if that is what your report needs.
import std.dataframe as df
def main() -> Result[(), df.DataFrameError | AllocError | IoError]:
before = df.frame([
df.column_i64("id", [1, 2]?)?,
df.column_i64("price", [10, 20]?)?
]?)?
after = df.frame([
df.column_i64("id", [2, 3]?)?,
df.column_i64("price", [25, 30]?)?
]?)?
keys = [df.col("id")?]?
joined = before.lazy()?.join(after.lazy()?, keys, keys, how=df.JoinKind.Full)?
display_id = df.col("id")?.fill_null(df.col("id_right")?)?.alias("item")?
result = joined.with_columns([display_id]?)?.sort([
df.SortKey(df.col("item")?)
]?)?.collect()?
print(result.values_i64("item")?)?
print(result.values_i64_nullable("price")?)?
print(result.values_i64_nullable("price_right")?)?
Ok(())
[1, 2, 3]
[Option[i64].Some(10), Option[i64].Some(20), Option[i64].Nothing]
[Option[i64].Nothing, Option[i64].Some(25), Option[i64].Some(30)]
Full joins preserve duplicate matching pairs, just like inner and left joins. They do not verify key uniqueness. The nulls above describe missing sides; they are not query failures. See the join contracts for composite keys, null matching, name collisions and ordering.
Continue with Match nearby observations when a lookup should use the most recent or nearest key rather than an identical key.
Match nearby observations
An equality join requires identical keys. An as-of join instead finds the most recent, next or nearest observation. This is useful for attaching a calibration value to sensor readings or a price to an event.
Use integer keys with a unit your application defines. Here, keys represent
elapsed seconds. They are ordinary i64 values, not a date/time type.
These examples require the optional dataframe engine.
As-of validation materializes both inputs when the query executes, including
with collect_streaming; filter and select the needed columns beforehand
when working with larger data.
Use the most recent calibration
The source below is already ordered. For unsorted data, sort both query inputs by their key before joining. The engine checks ascending order rather than silently producing a misleading match.
import std.dataframe as df
def main() -> Result[(), df.DataFrameError | AllocError | IoError]:
readings = df.frame([
df.column_i64("second", [2, 6, 12]?)?,
df.column_f64("reading", [10.0, 20.0, 30.0]?)?
]?)?
calibration = df.frame([
df.column_i64("second", [1, 5]?)?,
df.column_f64("offset", [0.5, 1.5]?)?
]?)?
joined = readings.lazy()?.join_asof(
calibration.lazy()?, "second", "second", tolerance=Some(3)
)?
corrected = df.col("reading")?.add(df.col("offset")?)?.alias("corrected")?
result = joined.with_columns([corrected]?)?.collect()?
print(result.values_f64_nullable("corrected")?)?
Ok(())
[Option[f64].Some(10.5), Option[f64].Some(21.5), Option[f64].Nothing]
Backward matching is the default: calibration at second 5 can apply to a
reading at second 6, but cannot apply to a reading at second 2. The final
reading has no calibration within three seconds; its null offset propagates
through addition. This is missing data, not a query failure. Both original
key columns remain available as second and second_right for inspection.
Choose the nearest observation
Nearest matching can look into the future. Choose it deliberately when that is meaningful for your data; do not use it for a causal lookup by accident.
import std.dataframe as df
def main() -> Result[(), df.DataFrameError | AllocError | IoError]:
events = df.frame([df.column_i64("second", [2, 4]?)?]?)?
samples = df.frame([
df.column_i64("sample_second", [1, 3, 5]?)?,
df.column_str("label", ["early", "middle", "late"]?)?
]?)?
result = events.lazy()?.join_asof(
samples.lazy()?, "second", "sample_second",
strategy=df.AsOfStrategy.Nearest, tolerance=Some(1)
)?.collect()?
print(result.values_str("label")?)?
Ok(())
["middle", "late"]
Each event is equally far from two samples; nearest matching selects the
greater key. AsOfStrategy.Forward always chooses the next qualifying key.
For backward/forward matching, set allow_exact_matches=False when a strictly
earlier/later sample is required. See as-of join contracts
for duplicate keys, nulls, limits and failure behavior.
Dates and daily reports
Parse a timestamp column once, then keep calendar calculations inside the query. The example below groups events by their UTC calendar date. This is different from splitting timestamp strings or assuming every input has an identical textual layout.
These examples need the dataframe engine,
which Linux releases include. utc=True declares that the input text already
represents UTC; it does not convert another timezone.
import std.dataframe as df
def main() -> Result[(), df.DataFrameError | AllocError | IoError]:
events = df.frame([
df.column_str("text", ["2024-02-28 09:00:00", "2024-02-29 12:30:00", "2024-02-29 14:00:00"]?)?,
df.column_i64("amount", [10, 20, 5]?)?
]?)?
stamp = df.col("text")?.str_to_datetime(
"%Y-%m-%d %H:%M:%S", df.TimeUnit.Microseconds, utc=True
)?.alias("stamp")?
daily = events.lazy()?.with_columns([stamp]?)?.group_by(
[df.col("stamp")?.dt_date()?.alias("day")?]?,
[df.col("amount")?.sum()?.alias("total")?]?
)?.sort([df.SortKey(df.col("day")?)]?)?.select([
df.col("day")?.dt_format("%Y-%m-%d")?.alias("day")?,
df.col("total")?
]?)?.collect()?
print(daily.values_str("day")?)?
print(daily.values_i64("total")?)?
Ok(())
["2024-02-28", "2024-02-29"]
[10, 25]
dt_date() retains a logical Date while grouping and sorting; formatting
happens only for presentation. The chosen unit controls stored precision.
Microseconds keep six fractional decimal places; milliseconds keep three
and nanoseconds keep nine. Extra fractional precision is discarded toward
the earlier instant, so select the unit intentionally.
Keep invalid input visible as missing data
Parsing is strict by default. Use strict=False when invalid input should
become null and your application has a policy for missing data.
import std.dataframe as df
def main() -> Result[(), df.DataFrameError | AllocError | IoError]:
source = df.frame([
df.column_str("text", ["1970-01-01", "1969-12-31", "2001-02-29"]?)?
]?)?
dates = source.lazy()?.select([
df.col("text")?.str_to_date(strict=False)?.alias("day")?
]?)?.collect()?
print(dates.values_date_days_nullable("day")?)?
Ok(())
[Option[i32].Some(0), Option[i32].Some(-1), Option[i32].Nothing]
The raw boundary representation is days since 1970-01-01; datetimes use integer ticks with an explicit unit. In most reports, prefer calendar expressions and formatting to manipulating these representations yourself. Getters check the logical dtype and, for datetimes, the requested unit and UTC metadata. They do not silently rescale timestamps.
Use Parquet when temporal metadata should survive file round-trips. For CSV, choose an explicit text format and parse it again on input. Named timezones, offset parsing and leap seconds are outside this foundation. See the temporal contracts for nulls, calendar bounds, overflow and conversion costs.
Clean and compare column values
Keep cleaning and derived calculations inside a query so Polars can optimize them with the rest of the plan. These examples require the optional engine.
Parse values and label missing data
The explicit non-strict cast below treats invalid numeric text as missing.
That is a deliberate data-cleaning policy; omit strict=False when bad
input should fail the query instead.
import std.dataframe as df
def main() -> Result[(), df.DataFrameError | AllocError | IoError]:
data = df.frame([
df.column_str("team", ["North", "South", "NORTH"]?)?,
df.column_str("amount", ["12", "oops", "8"]?)?
]?)?
cleaned = data.lazy()?.with_columns([
df.col("team")?.str_lower()?,
df.col("amount")?.cast(df.DType.I64, strict=False)?
]?)?
state = df.when(
df.col("amount")?.is_null()?,
df.lit_str("missing")?,
df.lit_str("ready")?
)?.alias("state")?
result = cleaned.with_columns([state]?)?.collect()?
print(result.values_str("team")?)?
print(result.values_i64_nullable("amount")?)?
print(result.values_str("state")?)?
Ok(())
["north", "south", "north"]
[Option[i64].Some(12), Option[i64].Nothing, Option[i64].Some(8)]
["ready", "missing", "ready"]
The second with_columns sees the parsed amount. Expressions within a
single with_columns all see its input schema, so separate dependent steps.
when is a native conditional expression, not a short-circuiting Plenty
if: both branches must be valid. It cannot guard a failing strict cast.
For literal matching, write
df.col("team")?.str_contains(df.lit_str("north")?)?.
The pattern is text, not a regular expression. A column expression can
also supply a different pattern per row.
Compare each row with its group
A grouped aggregate produces one row per group. A window calculation instead places the group’s result alongside each original row.
import std.dataframe as df
def main() -> Result[(), df.DataFrameError | AllocError | IoError]:
data = df.frame([
df.column_str("team", ["north", "south", "north", "north"]?)?,
df.column_i64("units", [2, 4, 3, 2]?)?
]?)?
teams = [df.col("team")?]?
units = df.col("units")?
result = data.lazy()?.with_columns([
units.sum()?.over(teams)?.alias("team_total")?,
units.n_unique()?.over(teams)?.alias("distinct_units")?
]?)?.collect()?
print(result.values_i64("team_total")?)?
print(result.values_i64("distinct_units")?)?
Ok(())
[7, 4, 7, 7]
[2, 1, 2, 2]
over uses the entire partition, not a running window. It maps results
back to the original rows and does not sort them. Distinct counts include
null as one distinct value, whereas count() counts only present values.
Read the expression contracts
for casting, Unicode and null semantics.
Query CSV and write Parquet
Use a lazy CSV scan when Polars should filter, derive or aggregate file-backed data. A direct Parquet sink writes the query’s output without first returning a complete dataframe to Plenty.
This example creates a small input file so it can run by itself. With real
data, begin at scan_csv. The examples need the
dataframe engine; run them in an
empty directory: the new output methods refuse to overwrite existing files.
import std.dataframe as df
def main() -> Result[(), df.DataFrameError | AllocError | IoError]:
readings = df.frame([
df.column_str("station", ["001", "002", "001"]?)?,
df.column_f64("reading", [10.0, 20.0, 30.0]?)?
]?)?
readings.write_csv("readings.csv")?
query = df.scan_csv("readings.csv", schema_overrides=Some([
df.Field("station", df.DType.String),
df.Field("reading", df.DType.F64)
]?))?
warm = query.filter(df.col("reading")?.gt(df.lit(15.0)?)?)?
warm.sink_parquet("warm.parquet")?
result = df.scan_parquet("warm.parquet")?.collect_streaming()?
print(result.values_str("station")?)?
print(result.values_f64("reading")?)?
Ok(())
["002", "001"]
[20.0, 30.0]
The String override keeps leading zeros in station identifiers. CSV contains
text rather than a typed schema. For unfamiliar files, inspect
query.schema()?. With infer_rows=0, unspecified columns stay strings,
so you can validate them and choose explicit casts. The default samples
100 rows; infer_rows=-1 scans the full file for inference.
Missing values and text
A quoted empty string and an unquoted empty field have different meanings. The default writer/reader preserves that distinction. When copying a frame through CSV, pass its schema to preserve column types too.
import std.dataframe as df
def main() -> Result[(), df.DataFrameError | AllocError | IoError]:
notes: list[Option[str]] = [Some(""), Nothing, Some("雪, café")]?
data = df.frame([df.column_str_nullable("note", notes)?]?)?
data.write_csv("notes.csv", separator=";")?
restored = df.scan_csv(
"notes.csv",
separator=";",
schema_overrides=Some(data.schema()?)
)?.collect_streaming()?
print(restored.values_str_nullable("note")?)?
Ok(())
[Option[str].Some(""), Option[str].Nothing, Option[str].Some("雪, café")]
A custom null token is available through write_csv(null_value="NA")
and scan_csv(null_value=Some("NA")). Use it only when that token cannot
be a real value: even a quoted "NA" is treated as null on reading.
Streaming execution does not imply a memory bound. collect_streaming
retains all final rows, and sorts, groups, joins or windows may buffer data.
Prefer sink_parquet when the desired result is a file rather than a Plenty
dataframe. A failed write can leave a partial new file; existing files are
never overwritten by these two new output methods.
See the I/O contracts for options,
error timing and execution limits.
Continue with Application: a daily sales report to combine file scans, cleaning, dates, joins and exports in a complete program.
Application: a daily sales report
The repository’s examples/sales_report/ combines dataframe operations into
a command-line application. It reads order lines, joins a product catalogue,
separates bad records, and exports daily sales totals. The directory includes
small input files, reusable query functions, a CLI and independent output checks.
Start with the engine installation. From a source checkout with the compiler and matching engine installed:
target/release/plenty --compile examples/sales_report/main.plenty -o /tmp/sales-report
target/release/plenty --compile examples/sales_report/convert.plenty -o /tmp/sales-convert
RUN=$(mktemp -d)
mkdir "$RUN/input" "$RUN/output"
/tmp/sales-convert ingest examples/sales_report/data/orders-01.csv "$RUN/input/part-01.parquet"
/tmp/sales-convert ingest examples/sales_report/data/orders-02.csv "$RUN/input/part-02.parquet"
/tmp/sales-report parquet "$RUN/input/part-*.parquet" examples/sales_report/data/products.csv "$RUN/output"
cat "$RUN/output/daily.csv"
The application prints 12 input lines, 5 accepted lines, 7 rejected lines and 4 daily groups. The resulting CSV is:
day,category,lines,total_cents
2024-02-28,books,1,1200
2024-02-29,books,2,1000
2024-02-29,games,1,2500
2024-03-01,games,1,0
It also writes daily.parquet, retaining logical Date values, and
rejected.csv, retaining the original fields with rejection reasons. For a
single CSV input, use sales-report csv ORDERS.csv PRODUCTS.csv OUTPUT_DIR.
Only Parquet supports the quoted glob. The converter keeps the raw String
columns; it does not pre-clean or calculate the report.
Follow the query through the application
Read report.plenty in this order:
scan_ordersselects four required String columns. CSV inference is disabled so product IDs such as001keep their leading zeros. Parquet input uses that same raw ingestion schema.cataloguechecks nonempty IDs/categories and unique product IDs before joining. Duplicate reference keys would otherwise multiply sales totals.classifyparses UTC timestamps and integer cents non-strictly, then performs a left join. A native conditional expression assigns each row an acceptance or rejection reason. Missing prices or dates are not silently replaced with useful-looking values.dailyfilters accepted rows, groups by calendar date and lowercased category, counts lines and sums cents. Sorting gives stable report order.runexports the report and rejected records, then prints reconciliation counts. It propagates structural/query/I/O failures through its fallible entrypoint, giving a nonzero exit status.
These functions build native expressions rather than invoking Plenty once per row. Named intermediate plans make the cleaning policy readable without copying the whole dataset at each step. The classified rows are collected once so both outputs describe the same input; that is an explicit memory cost, not a promise that streaming execution bounds memory.
Amounts are already line totals, expressed as whole cents. Zero is valid; negative amounts, missing amounts and amounts above one billion cents are rejected. Together with a 100-million-line cap before aggregation, this keeps native i64 sums in range. The row cap is checked after collection and cannot protect against running out of memory while reading. Repeated order IDs are separate lines, not automatically deduplicated transactions. This is a batch reporting example, not an accounting system.
Check and ship it
Run the independent checks with the same engine installation:
python3 -B examples/sales_report/check.py --plenty target/release/plenty -v
The checks compile the actual application, compare CSV and Parquet results
with Python stdlib calculations, and exercise invalid rows, duplicate keys,
empty inputs and failures. Add --benchmark-rows 100000 to measure a larger
generated workload, including all output writes. Python is a development
dependency of these checks, not a runtime dependency of the report.
Use a new output directory for each run. Files are not overwritten, but the three writes are not one transaction: a failure may leave partial output. Publish the directory only after the process succeeds. Rejected rows are an intentional successful outcome; inspect the reconciliation counts and file.
The application README describes the exact input schema, rejection precedence, resource costs and measurement procedure. Follow shipping dataframe applications to distribute the executable with its matching engine and license notices.
Columnar lists
clist[T] makes data-oriented performance accessible through ordinary Plenty
code. Declare a class normally, then choose clist[YourClass] when a large
collection of its records is usually processed one or two fields at a time.
You still append ordinary objects and refer to their named fields. The
collection stores each field in its own packed, contiguous column.
For example, averaging temperatures does not need the station identifier from every reading. Keeping temperatures together lets a scan bring more useful values into the CPU cache with each memory access. Column pipelines make this kind of bulk calculation natural: select a field, then use the familiar fluent operations. The storage choice belongs to this collection; it does not change how the class is stored everywhere else.
This is useful as a representation as well as a container: it makes efficient execution easier to express for the programmer and easier to recognize for the compiler. Start with the access pattern you need, rather than expecting every operation on a columnar list to become faster. Gathering a complete row requires reading separate columns. The current implementation accepts only primitive fields; use ordinary lists for strings, nested objects, or borrowed references to whole rows.
For analytical expression plans, grouping and Parquet datasets, consider
dataframes. Their optional Polars engine has runtime
schemas and native query semantics; clist keeps a lightweight mutable
collection and Plenty’s own arithmetic and type checking.
- Build and scan columnar records: reserve capacity, append records, scan a column, and update or copy a row.
- Count matching integer values: express a filter/count calculation that the compiler can accelerate with SIMD.
- Remove rows and reuse storage: take individual records or clear a batch while retaining capacity.
Build and scan columnar records
Declare a class normally, then create a clist to store its fields in separate
columns. This example reserves space for three readings, averages their
temperatures, and updates and copies a row:
class Reading:
station: i64
temperature: f64
def add(total: f64, temperature: f64) -> f64:
total + temperature
def main() -> Result[(), AllocError | AppendError[Reading] | IoError]:
mut readings = clist[Reading].with_capacity(3)?
readings.append(Reading(10, 18.0))?
readings.append(Reading(20, 21.0))?
readings.append(Reading(30, 24.0))?
total = readings.column(Reading.temperature).fold(0.0, add)
print(total / f64(len(readings)))?
readings[0].temperature = 19.0
match readings.copy_row(0):
case Some(reading):
print(reading.station, reading.temperature)?
case Nothing:
()
Ok(())
21.0
10 19.0
Reading.temperature selects that field’s column. The pipeline borrows the
collection and yields scalar values, so add takes f64 directly. Reserving
capacity makes the three appends allocation-free after construction. Errors
from append retain the submitted reading; this example propagates that error
so callers can decide whether to retry.
A column pipeline is consumed once. While it remains live, the collection
cannot be changed or moved. Create a new column pipeline for another pass.
copy_row gathers a row’s fields from the separate columns into an independent
ordinary object. It allocates nothing, but gathering all the fields can cost
more than accessing a record in an ordinary list. Changing the copy does not
change the stored row. Use a column for a selected-field calculation and a row
copy when you actually need the complete object.
The primitive-only restriction keeps this first version’s ownership and packed storage model simple and sound. Prefer an ordinary list for strings, nested objects, row references, or workloads that usually access whole objects. Indexed field access and row copying compile to direct scalar memory operations, with bounds checks. This avoids dispatching through a generic runtime helper for each field, but does not make every access pattern faster. Use column pipelines for repeated scans when the calculation needs selected fields.
Next, count matching integer values with an automatically optimized column pipeline. See the columnar-list reference for the allocation, borrowing and representation contracts.
Count matching integer values
Write the ordinary calculation you need. For a simple comparison over an i32
or u32 column, the default optimized profile can count matches with SIMD: the CPU compares four
values together. You do not choose a vector width or handle leftover rows.
class Sample:
score: i32
def main() -> Result[(), AppendError[Sample] | IoError]:
mut samples = clist[Sample]()
for score in range(9):
samples.append(Sample(i32(score)))?
matches = samples.column(Sample.score).filter(def(x: &i32) -> bool: *x > 5i32).count()
print(matches)?
Ok(())
3
Use runtime bounds
Bounds do not need to be constants. Capture their scalar values to count a range:
class Sample:
score: i32
def count_between(samples: &clist[Sample], low: i32, high: i32) -> i64:
samples.column(Sample.score).filter(def [low, high](x: &i32) -> bool: *x >= low and *x < high).count()
def main() -> Result[(), AppendError[Sample] | IoError]:
mut samples = clist[Sample]()
for score in range(9):
samples.append(Sample(i32(score)))?
print(count_between(&samples, 2i32, 7i32))?
Ok(())
5
[low, high] captures copies of the bounds. They can come from function
arguments or parsed input. The closure keeps those values even if an original
binding later changes. Write two comparisons with and; Plenty does not use
chained comparison syntax.
This is the same program in normal and optimized builds. The SIMD path recognizes
one comparison, or two comparisons joined by and, against constants or up to
two immutable scalar captures of the column’s type. Borrowed captures, arithmetic
inside the predicate, or, and extra adapters keep scalar execution.
Other calculations, including the floating-point average in
the previous lesson, keep ordinary scalar execution.
All scans still run on one thread. See the
columnar-list reference
for the precise optimization limits, allocation, borrowing and representation.
Remove rows and reuse storage
Use remove(index) when you need to take one complete record out of a columnar
list. It returns Some(record) or Nothing for an out-of-bounds index. Remaining
records keep their order, and negative indices count from the end.
class Reading:
station: i64
temperature: f64
def main() -> Result[(), AllocError | AppendError[Reading] | IoError]:
mut readings = clist[Reading].with_capacity(3)?
readings.append(Reading(10, 18.0))?
readings.append(Reading(20, 21.0))?
readings.append(Reading(30, 24.0))?
match readings.remove(1):
case Some(reading):
print("removed", reading.station, reading.temperature)?
case Nothing:
()
for station in readings.column(Reading.station):
print(station)?
capacity = readings.capacity()
readings.clear()
print(len(readings), readings.capacity() == capacity)?
readings.append(Reading(40, 19.0))?
print(readings[0].station)?
Ok(())
removed 20 21.0
10
30
0 True
40
Removal assembles an independent ordinary object, just like copy_row, then
shifts the later values left in every column. It allocates nothing, but removing
near the front of a large collection moves much more data than removing the
last row. Repeated front removals are a poor way to filter a whole dataset.
clear() empties all columns in constant time and retains their allocation.
This is useful when processing successive batches: clear the old batch and
append the next one into the same storage. Removal also retains capacity.
Appends within that capacity do not allocate, although their return type still
requires error handling. Use drop(readings) when you want to release the
storage instead of reusing it.
Both methods require mutable access. A live column pipeline borrows the entire collection, so finish its traversal before removing rows or clearing the owner. See the reference for the complete contract and costs.
Organizing a program
Imports divide a program into source modules. pub controls which declarations
other modules may use.
Split a program into modules
An import names a source file without executing it. Modules contain declarations
and imports; execution starts at the application’s main. Names are private to
their module unless marked pub.
Save this companion file as geometry.plenty:
pub class Point:
pub x: i64
pub y: i64
pub def squared_length(self) -> Result[i64, OverflowError[i64]]:
Ok(self.x * self.x + self.y * self.y)
pub def make_point() -> Point:
Point(3, 4)
Save the application beside it, for example as main.plenty:
import geometry
from geometry import Point as Position
def main() -> Result[(), IoError | OverflowError[i64]]:
point: Position = geometry.make_point()
print(point.squared_length()?)?
print(point.x)?
Ok(())
25
3
import geometry makes declarations reachable through geometry.Name.
from geometry import Point as Position binds just that type under a local name.
You can also write import geometry as geo, then use geo.Point. Aliases refer
to the same declaration. Two modules may each define Point; those are distinct
types.
Imports are absolute. import tools.geometry reads tools/geometry.plenty
under the source root. Directories provide namespaces and need no __init__
file. The root defaults to the entry file’s directory. If the entry file is
deeper in the tree, select the root explicitly:
plenty --module-root src src/app/main.plenty
plenty --module-root src --compile src/app/main.plenty -o app
plenty --module-root src --check-module src/tools/geometry.plenty
--check checks a complete application and requires main. --check-module
checks a library and its imports without that requirement.
Choose a public API
Mark each public field and method explicitly. Without __init__, the generated
field constructor is public only when the class and every field are public. To keep
fields private while allowing construction, declare pub def __init__:
Save this module as counter.plenty:
pub class Counter:
value: i64
pub def __init__(self, start: i64) -> ():
self.value = start
pub def increment(self: &mut Counter) -> Result[(), OverflowError[i64]]:
self.value = self.value + 1
Ok(())
pub def read(self) -> i64:
self.value
def __del__(self) -> ():
drop(print("counter closed"))
Application:
from counter import Counter
def main() -> Result[(), IoError | OverflowError[i64]]:
mut count = Counter(41)
count.increment()?
print(count.read())?
Ok(())
42
counter closed
The private destructor still runs automatically. Private fields and methods are
accessible throughout their module, but not from another module. Reads, writes,
and borrows all enforce this. Save the next module as secret.plenty:
pub class Secret:
value: i64
pub def __init__(self, value: i64) -> ():
self.value = value
from secret import Secret
def main() -> Result[(), IoError]:
item = Secret(42)
print(item.value)?
Ok(())
secret.Secret.value` is private
pub also applies to functions, enums, and type aliases. A public enum exposes
all its variants. Public signatures cannot mention private classes or enums,
even through aliases or containers. Ordinary imports are private bindings;
importing a name does not re-export it. Relative imports, wildcards, circular
imports, and pub import are not supported yet.
Private fields are still included in automatic printing and equality.
The lifecycle trace uses drop(print(...)) deliberately: cleanup cannot return
an error, and a failed diagnostic write must not interrupt cleanup. For output
that must succeed, provide an explicit fallible operation and propagate its
result before leaving the scope.
Use the standard math library
The compiler includes std.math. Import it like any other module; no companion
source file or download is needed. Its floating-point API accepts f64.
import std.math as math
from std.math import pi
const half_turn: f64 = f64.PI
def main() -> Result[(), AllocError | IoError | math.MathError | OverflowError[i64]]:
print(math.sqrt(81.0)?)?
print(math.factorial(5)?)?
print(math.comb(5, 2)?)?
print(math.isclose(math.radians(180.0), pi, 1e-9, 0.0)?)?
print(pi == half_turn)?
print(math.sqrt(-1.0) == Result[f64, math.MathError].Err(math.MathError.Domain))?
values = [2, 3, 4]?
print(math.sum(&values, 0)?)?
print(math.prod(&values, 1)?)?
match math.min_all(&values):
case Some(value):
print(value)?
case Nothing:
print("empty input")?
match math.max_all(&values):
case Some(value):
print(value)?
case Nothing:
print("empty input")?
print(math.round(2.5)?)?
print(math.round(3.5)?)?
quotient, remainder = math.divmod(-7, 3)?
print(quotient)?
print(remainder)?
interval = range(1, 5)
print(math.sum_iter(&interval, 0)?)?
Ok(())
9.0
120
10
True
True
True
9
24
2
4
2
4
-3
2
10
? propagates mathematical errors just like I/O errors. Match on a result when
you want to recover from math.MathError.Domain or math.MathError.Overflow.
Integer combinatorics return u64, reporting overflow instead of wrapping.
Integer sum, sum_iter, and prod return OverflowError[T], where T is
the integer type; their calls need ? too. Floating-point variants such as
sum_float keep IEEE behavior.
floor, ceil, trunc, and round return checked i64 results. round
chooses the even integer at halfway values. divmod returns an integer floor
quotient and remainder with the divisor’s sign; use divmod_float for floats.
Division by zero is a recoverable Domain error.
min_all and max_all borrow a list and return Nothing for an empty list.
They retain the first value on ties. Floats follow the IEEE 754 standard’s
minimum and maximum: -0.0 counts as less than 0.0, and any NaN makes the
result Some(NaN), regardless of its position. The two-value min and max
follow the same rule. sum_iter and prod_iter also borrow
ranges, sets, and dictionary keys; their explicit start supplies the numeric
type and the result for an empty input. The original collections remain usable.
isclose takes both tolerances explicitly. Module constants are math.pi,
math.e, math.tau, math.inf, and math.nan, all f64. Import one as a
value, as the example does with pi. The type constants f64.PI, f64.E,
f64.TAU, f64.INF, and f64.NAN also remain available. Low-level methods such as
x.sqrt() follow IEEE arithmetic and may return NaN; the imported functions
provide recoverable domain errors.
Declare your own immutable constants with const name: T = value, as with
half_turn. The compiler evaluates the initializer, which can combine literals,
type constants, and other constants with operators, as in
const full_turn: f64 = 2.0 * half_turn. Use pub const to export one from a
module. See constants
for the accepted expressions, static assertions, and file contents included as
constants.
See the math reference for available functions and their fixed-width and numerical limits. The standard library reference lists every module.
Work with explicit SIMD lanes
Prefer ordinary pipelines for repeated column scans: the
columnar-list lessons show how
Plenty can accelerate them automatically. Use std.simd when the algorithm
itself is clearer as a fixed set of lanes.
This example compares four measurements with one threshold, then selects the
threshold for lanes below it. The mask is a vector of logical booleans; count
counts its true lanes, and select chooses between already-computed values.
import std.simd as simd
def main() -> Result[(), IoError]:
readings = simd.f32x4.Lanes(18.0f32, 21.0f32, 16.0f32, 24.0f32)
floor = simd.splat_f32x4(20.0f32)
below = readings.simd_lt(floor)
adjusted = below.select(floor, readings)
print(below.count(), adjusted.to_tuple())?
print(readings.to_tuple())?
Ok(())
2 (20.0, 21.0, 20.0, 24.0)
(18.0, 21.0, 16.0, 24.0)
Vectors are copyable and allocate nothing. Operations borrow their inputs and
produce new values, so the original readings remain unchanged. i32x4, u32x4
and f64x2 provide the same pattern with other lane types. Mask shapes must
match the vectors they select.
Use add, sub and mul for ordinary arithmetic. Integer versions return a
Result with a lane-typed overflow error; propagate it with ? or handle it
with match. Integer wrapping_add, wrapping_sub and wrapping_mul explicitly
choose modular arithmetic. Floating sum_ordered(initial) preserves scalar
addition order. Selection is eager: compute and handle both inputs before
selecting between them.
The module also provides checked list loads and stores; they require a complete vector to fit and never allocate. Handle any leftover elements with a scalar loop. Ordinary lists and these initial vector values use tagged scalar storage, so explicit lanes do not promise packed storage or faster code. The default optimized profile enables supported native SIMD operations on x86-64. AArch64 currently uses the scalar implementation, which preserves the same results. See the reference for the exact operations, bounds contracts and current limits.
Measure time and wait
Import std.time for spans, clock readings, and sleep. A Duration is a
nonnegative span; an Instant is a reading of a clock that never goes
backwards.
from std.time import monotonic_now, sleep, milliseconds, seconds
def main() -> Result[(), AllocError | IoError | RangeError]:
pause = milliseconds(5)
start = monotonic_now()
sleep(&pause)
elapsed = monotonic_now().duration_since(&start)?
print(elapsed.as_milliseconds() >= pause.as_milliseconds())?
minutes = seconds(60).checked_mul(90)?
print(minutes.as_seconds())?
print(seconds(1) == milliseconds(1000))?
Ok(())
True
5400
True
sleep blocks for at least the span; a busy machine can make it longer, so the
example prints a comparison, not a measurement. Unit constructors take u32
counts and never fail. Arithmetic that could leave the representable range
returns Result with RangeError, including a subtraction that would be
negative:
from std.time import milliseconds, seconds
def main() -> Result[(), AllocError | IoError | RangeError]:
short = milliseconds(250)
long = seconds(1)
match short.checked_sub(&long):
case Ok(span):
print(span.as_milliseconds())?
case Err(RangeError.OutOfRange):
print("a span cannot be negative")?
print(short.saturating_sub(&long).is_zero())?
print(long.checked_sub(&short)?.as_milliseconds())?
Ok(())
a span cannot be negative
True
750
Time values are class instances, so they move. Pass them by reference, as these calls do, and they stay usable.
Wall-clock time is a different type
wall_now() returns a Timestamp: seconds and nanoseconds since the Unix
epoch. The system clock can be set backwards, so the difference between two
timestamps is fallible. This example builds two readings by hand to show a
clock that was stepped back by two seconds:
from std.time import Timestamp, wall_now
def main() -> Result[(), AllocError | IoError | RangeError]:
first = Timestamp(1700000010, 0)
second = Timestamp(1700000008, 500000000)
match second.duration_since(&first):
case Ok(span):
print(span.as_milliseconds())?
case Err(RangeError.OutOfRange):
print("the clock moved backwards")?
print(first.duration_since(&second)?.as_milliseconds())?
print(wall_now().seconds > first.seconds)?
Ok(())
the clock moved backwards
1500
True
Use instants to measure elapsed time and timestamps to record when something happened. The compiler rejects mixing them:
from std.time import monotonic_now, wall_now
def main() -> Result[(), AllocError | IoError]:
started = monotonic_now()
print(wall_now().duration_since(&started)?.as_seconds())?
Ok(())
expected &std.time.Timestamp, got &std.time.Instant
The module has no calendar, time-zone, or formatting support. See the time reference for every operation, and Share one deadline across waits for timeouts.
Take the most urgent item first
Import Heap from std.collections for a priority queue. pop always removes
the least element, however the elements were pushed.
from std.collections import Heap
def main() -> Result[(), AllocError | IoError]:
mut times = Heap[i64]()?
for time in [40, 10, 30, 20]?:
times.push(time)?
print(times.peek(), times.len())?
while not times.is_empty():
match times.pop():
case Some(value):
print(value)?
case Nothing:
break
print(times.pop())?
Ok(())
Option[i64].Some(10) 4
10
20
30
40
Option[i64].Nothing
Construction and push return Result with AllocError. pop returns
Option and transfers the element. peek returns a copy of the first element
without removing it, or Nothing when the heap is empty.
Heap orders integers, floats, and strings with <. For any other element,
HeapBy takes a function that says which of two elements leaves first:
from std.collections import HeapBy
class Job:
priority: i64
name: str
def urgent(a: &Job, b: &Job) -> bool:
a.priority > b.priority
def main() -> Result[(), AllocError | IoError]:
mut jobs = HeapBy(urgent)?
jobs.push(Job(1, "sweep"))?
jobs.push(Job(9, "deploy"))?
jobs.push(Job(5, "review"))?
match jobs.peek_ref():
case Some(job):
print("next:", job.name)?
case Nothing:
pass
while not jobs.is_empty():
match jobs.pop():
case Some(job):
print(job.priority, job.name)?
case Nothing:
break
Ok(())
next: deploy
9 deploy
5 review
1 sweep
The heap owns its jobs; pop moves one out without copying it. A Job cannot
be copied, so peek is rejected for this heap; peek_ref borrows the first job
instead, as in returning a borrow that may be absent.
The heap cannot change while that borrow is in use. Elements that
compare equal leave in no promised order. See
the module reference for costs and limits.
Find a route through a graph
std.graph stores nodes and the edges between them. UnGraph edges join two
nodes both ways; DiGraph edges run from a source to a target. Each node and
edge owns a value, and adding one returns a handle that names it.
from std.graph import UnGraph, GraphError
def minutes(walk: &i64) -> f64:
f64(*walk)
def main() -> Result[(), AllocError | IoError | GraphError]:
mut town = UnGraph[str, i64]()?
home = town.add_node("home")?
shop = town.add_node("shop")?
park = town.add_node("park")?
school = town.add_node("school")?
town.add_edge(home, shop, 10)?
town.add_edge(home, park, 4)?
town.add_edge(park, shop, 3)?
town.add_edge(shop, school, 6)?
paths = town.dijkstra(home, &minutes)?
for stop in paths.path(school)?:
print(*town.node(stop))?
print(paths.distance(school))?
Ok(())
home
park
shop
school
Option[f64].Some(13.0)
The graph does not know what edge values mean, so dijkstra takes a function
that turns one into a non-negative weight. The result answers distance and
path queries for every node reachable from the start.
Handles stay valid after removals: removing park leaves shop and school
naming the same places, and the path now takes the direct road.
from std.graph import UnGraph, GraphError
def minutes(walk: &i64) -> f64:
f64(*walk)
def main() -> Result[(), AllocError | IoError | GraphError]:
mut town = UnGraph[str, i64]()?
home = town.add_node("home")?
shop = town.add_node("shop")?
park = town.add_node("park")?
town.add_edge(home, shop, 10)?
town.add_edge(home, park, 4)?
town.add_edge(park, shop, 3)?
print(town.remove_node(park))?
print(town.contains_node(park), town.edge_count())?
print(town.get_node_ref(park), town.get_node_ref(shop))?
print(town.dijkstra(home, &minutes)?.distance(shop))?
Ok(())
Option[str].Some("park")
False 1
Option[&str].Nothing Option[&str].Some("shop")
Option[f64].Some(10.0)
Using a removed handle with node or add_edge traps. When a handle may be
stale, get_node_ref and get_edge_ref return Nothing instead of trapping,
and get_node_mut and get_edge_mut do the same for changes. See
the module reference for traversal,
topological order, components, spanning trees and costs.
Encode bytes and pack numbers
Binary formats store integers at fixed widths in a stated byte order, and text
channels carry binary data as hex or base64. std.encoding does both over
std.bytes buffers.
import std.bytes
import std.encoding
from std.encoding import Endian
def main() -> Result[(), AllocError | IoError | std.encoding.EncodingError | std.bytes.BytesError]:
mut frame = std.bytes.empty()?
std.encoding.append_u16(&mut frame, 513u16, Endian.Big)?
std.encoding.append_u32(&mut frame, 7u32, Endian.Little)?
frame.extend(&std.bytes.from_text("hi")?)?
print(std.encoding.hex_text(&frame)?)?
print(std.encoding.base64_text(&frame)?)?
print(std.encoding.base64url_text(&frame[:7])?)?
print(std.encoding.unpack_u16(&frame, 0, Endian.Big), std.encoding.unpack_u16(&frame, 0, Endian.Little))?
std.encoding.pack_u32(&mut frame, 2, 8u32, Endian.Little)
print(std.encoding.unpack_u32(&frame, 2, Endian.Little), frame.len())?
Ok(())
0201070000006869
AgEHAAAAaGk=
AgEHAAAAaA
513 258
8 8
Every integer function names its width and takes the byte order as an argument;
there is no default order. Endian.Big stores the most significant byte first.
The same two bytes read as 513 or 258 depending on the order you state.
append_ grows the buffer by the integer’s width. pack_ overwrites bytes that
already exist and never grows it. unpack_ reads at an offset.
Mixed integer and float fields
Use f32 or f64 when the format specifies IEEE 754 binary32 or binary64.
The same append, overwrite, and offset rules apply. A numeric cast to an
integer changes the value; these codecs preserve its float representation:
import std.bytes
import std.encoding
from std.encoding import Endian
def main() -> Result[(), AllocError | IoError | std.encoding.EncodingError]:
mut record = std.bytes.with_capacity(14)?
std.encoding.append_u16(&mut record, 1u16, Endian.Big)?
std.encoding.append_f32(&mut record, 1.5f32, Endian.Big)?
std.encoding.append_f64(&mut record, -2.5, Endian.Little)?
print(std.encoding.hex_text(&record)?)?
print(std.encoding.unpack_u16(&record, 0, Endian.Big))?
print(std.encoding.unpack_f32(&record, 2, Endian.Big))?
print(std.encoding.unpack_f64(&record, 6, Endian.Little))?
Ok(())
00013fc0000000000000000004c0
1
1.5
-2.5
Widths count bytes, so the fields begin at offsets 0, 2, and 6.
Floats need no alignment padding. The float packing reference
explains signed zero, infinities, NaNs and exact bit preservation.
The codecs take a byte view: &buffer for a whole buffer, &buffer[start:stop]
for part of one, or a b"..." literal. The _text forms return a str; hex_encode, base64_encode,
and base64url_encode return the encoded bytes as a new buffer. base64 uses
+ and / and always pads with =. base64url uses - and _ and never
pads.
Ordinary decoding accepts exactly one form and reports what is wrong with anything else. Nothing is skipped or repaired, including spaces and line breaks:
import std.bytes
import std.encoding
from std.encoding import Endian
def main() -> Result[(), AllocError | IoError | std.bytes.BytesError | std.encoding.EncodingError]:
print(std.encoding.base64_from_text("aGk=")?.to_text()?)?
print(str.repr(std.encoding.hex_from_text("abc"))?)?
print(str.repr(std.encoding.hex_from_text("0g"))?)?
print(str.repr(std.encoding.base64_from_text("aGk"))?)?
print(str.repr(std.encoding.base64_from_text("aG k"))?)?
print(str.repr(std.encoding.base64_from_text("aGl="))?)?
print(str.repr(std.encoding.base64url_from_text("aGk="))?)?
Ok(())
hi
Result[std.bytes.Bytes, std.encoding.EncodingError].Err(std.encoding.EncodingError.InvalidLength)
Result[std.bytes.Bytes, std.encoding.EncodingError].Err(std.encoding.EncodingError.InvalidByte(1))
Result[std.bytes.Bytes, std.encoding.EncodingError].Err(std.encoding.EncodingError.InvalidLength)
Result[std.bytes.Bytes, std.encoding.EncodingError].Err(std.encoding.EncodingError.InvalidByte(2))
Result[std.bytes.Bytes, std.encoding.EncodingError].Err(std.encoding.EncodingError.NonCanonical(2))
Result[std.bytes.Bytes, std.encoding.EncodingError].Err(std.encoding.EncodingError.InvalidPadding(3))
The number in an error is the index of the offending byte. aGl= is
NonCanonical because its last symbol carries bits that belong to no byte:
aGk= is the only accepted text for hi. Reading four bytes from a
three-byte buffer traps. Validate untrusted record lengths before unpacking;
Cursor field reads return Truncated
when input can legitimately end before a field is complete.
Decoded bytes are not text. Call to_text() on the result when you expect
UTF-8; it validates and can fail on its own.
Base32 and protocol-specific forms
Choose the encoding the receiving protocol expects. Base32 uses uppercase
letters and digits 2 through 7, with padding. For a protocol requiring a
different alphabet or no padding, build a validated encoding once and reuse it:
import std.bytes
import std.encoding
def main() -> Result[(), IoError | AllocError | std.bytes.BytesError | std.encoding.EncodingError]:
data = std.bytes.from_text("hello")?
print(std.encoding.base32_text(&data)?)?
lower = std.encoding.custom("abcdefghijklmnopqrstuvwxyz234567", Nothing)?
text = lower.text(&data)?
print(text, lower.from_text(&text)?.to_text()?)?
print(std.encoding.base64mime_from_text("aG Vs\r\nbG8=")?.to_text()?)?
Ok(())
NBSWY3DP
nbswy3dp hello
hello
base32hex offers the alternate alphabet 0–9, A–V. For email body
transfer, base64mime_text wraps padded base64 at 76 columns using CRLF between
lines. Only the MIME decoder skips ASCII space, tab, CR and LF; other invalid
characters, incorrect padding and nonzero trailing bits still fail. It does
not parse MIME headers or append a final CRLF. Ordinary base64_from_text
continues to reject all whitespace.
See the encoding reference for every operation, its errors, and its costs.
Take a URL apart and build one
A URL is several components joined by delimiters, and each component has its
own rules about which characters may appear. std.url splits a URL into those
components and puts data into one of them safely.
import std.url
def main() -> Result[(), AllocError | IoError | std.url.UrlError]:
url = std.url.parse("https://example.com:8443/docs/a%20b?tag=x%26y&page=2#top", 2048)?
print(url.scheme, url.path)?
print(str.repr(&url.host)?, str.repr(&url.port)?)?
print(str.repr(&url.userinfo)?, str.repr(&url.fragment)?)?
match &url.query:
case Some(query):
for pair in &std.url.query_pairs(query)?:
print(pair.key, "=", pair.value)?
case Nothing:
pass
print(std.url.decode(&url.path)?)?
print(url.to_string()?)?
Ok(())
https /docs/a%20b
Option[str].Some("example.com") Option[i64].Some(8443)
Option[str].Nothing Option[str].Some("top")
tag = x&y
page = 2
/docs/a b
https://example.com:8443/docs/a%20b?tag=x%26y&page=2#top
parse takes the text and the longest length in bytes you are willing to
accept. Components come back exactly as written, still percent-encoded:
nothing is lowercased, decoded, or tidied, so format returns the original
text. An optional component is Nothing when its delimiter is absent.
Decode a component only after the URL has been split. %26 in the query is
data, and query_pairs splits at the real & before it decodes, which is why
tag holds x&y.
Put data into a URL
To build a URL, encode each piece for the component it is going into, then fill
in a Url:
import std.url
from std.url import Component, Pair, Url
def main() -> Result[(), AllocError | IoError | std.url.UrlError]:
path = "/files/".concat(&std.url.encode("a/b report.txt", Component.Segment)?)?
mut pairs = list[Pair]()?
pairs.append(Pair("q", "fish & chips"))?
pairs.append(Pair("lang", "en"))?
query = std.url.query_text(&pairs)?
url = Url("https", Nothing, Some("example.com"), Nothing, path, Some(query), Nothing)
print(url.to_string()?)?
print(std.url.encode("a/b", Component.Path)?, std.url.encode("a/b", Component.Segment)?)?
Ok(())
https://example.com/files/a%2Fb%20report.txt?q=fish%20%26%20chips&lang=en
a/b a%2Fb
The Component says where the data is going. As a Segment, the / in a file
name is data and is escaped; as a Path, it is a separator and is kept.
query_text encodes every key and value as a QueryPart, which escapes & and
=.
What is rejected
std.url follows RFC 3986 strictly and never repairs input:
import std.url
from std.url import Url
def main() -> Result[(), AllocError | IoError]:
print(str.repr(std.url.parse("/docs/page", 2048))?)?
print(str.repr(std.url.parse("http://example.com/a b", 2048))?)?
print(str.repr(std.url.parse("http://example.com/100%", 2048))?)?
print(str.repr(std.url.parse("http://good.example@evil.example@host/", 2048))?)?
print(str.repr(std.url.parse("http://example.com:99999/", 2048))?)?
print(str.repr(std.url.parse("http://example.com/", 10))?)?
forged = Url("https", Nothing, Some("example.com/x"), Nothing, "", Nothing, Nothing)
print(str.repr(forged.to_string())?)?
Ok(())
Result[std.url.Url, std.url.UrlError].Err(std.url.UrlError.MissingScheme)
Result[std.url.Url, std.url.UrlError].Err(std.url.UrlError.InvalidByte(20))
Result[std.url.Url, std.url.UrlError].Err(std.url.UrlError.InvalidEscape(22))
Result[std.url.Url, std.url.UrlError].Err(std.url.UrlError.InvalidByte(32))
Result[std.url.Url, std.url.UrlError].Err(std.url.UrlError.InvalidPort(19))
Result[std.url.Url, std.url.UrlError].Err(std.url.UrlError.LimitExceeded)
Result[str, std.url.UrlError].Err(std.url.UrlError.InvalidByte(19))
The number is the byte offset of the problem. A URL must start with a scheme;
relative references are not parsed. A second @ is an error rather than a
guess about which host was meant. format checks a Url you built yourself
with the same rules, so a host containing / cannot turn into a different URL.
Non-ASCII text must be percent-encoded, which encode does for you. A Unicode
host name must already be in its xn-- form.
See the URL reference for the exact syntax, every operation, its errors, and its costs.
Draw random values and make secrets
Plenty has two sources of random values, and they are not interchangeable.
std.random gives repeatable sequences for simulations, games, and tests.
std.secrets gives unpredictable bytes from the operating system for tokens
and keys.
For record and request identifiers, see
std.uuid, which builds standard UUIDs from OS entropy.
Seeded generators
A generator is a value you create from a seed and own. Nothing is seeded for you and there is no shared generator behind the scenes, so the same seed always produces the same run:
import std.random
def main() -> Result[(), AllocError | IoError | std.random.RandomError]:
mut rng = std.random.seeded(1234567u64)
print(rng.next_u64())?
print(rng.between(1, 7)?, rng.between(1, 7)?, rng.between(1, 7)?)?
print(rng.next_bool(), rng.next_f64() < 1.0)?
mut cards = ["ace", "king", "queen", "jack", "ten"]?
rng.shuffle(&mut cards)
print(cards)?
print(rng.sample_indices(10, 3)?)?
print(cards[rng.index(len(cards))?])?
Ok(())
3504822795582309479
5 2 1
True True
["ace", "king", "ten", "queen", "jack"]
[3, 1, 9]
king
Drawing changes the generator, so it must be a mut binding. Ranges are
half-open: between(1, 7) is a die roll from 1 to 6. Every bounded draw is
unbiased. shuffle reorders a list in place without copying its elements.
sample_indices(10, 3) picks three different positions out of ten; index your
collection with them. index(count) picks one.
Because the sequence depends only on the seed, a failing simulation can be
replayed exactly. state() and from_state checkpoint a generator part-way
through:
import std.random
def main() -> Result[(), AllocError | IoError | std.random.RandomError]:
mut first = std.random.seeded(42u64)
mut second = std.random.seeded(42u64)
print(first.next_u64() == second.next_u64())?
(a, b, c, d) = first.state()
mut resumed = std.random.from_state(a, b, c, d)?
print(first.between(0, 100)? == resumed.between(0, 100)?)?
print(str.repr(first.between(5, 5))?)?
print(str.repr(first.below(0u64))?)?
Ok(())
True
True
Result[i64, std.random.RandomError].Err(std.random.RandomError.EmptyRange)
Result[u64, std.random.RandomError].Err(std.random.RandomError.EmptyRange)
An empty range is an error, never a clamped or default value, because it can
come from data such as an empty list. A rejected request does not advance the
generator. An argument no call could accept, such as
sample_indices(3, 4) asking for four of three positions, is a bug in the
caller: it stops the program with a
trap report.
The sequence for a seed is part of the language’s contract and does not change between Plenty versions. It is also predictable by anyone who sees a few outputs, so never use a seeded generator for a password, token, or key.
Secrets
std.secrets asks the operating system for every byte. Its results differ on
every run, so this program prints sizes rather than values:
import std.bytes
import std.secrets
def main() -> Result[(), AllocError | IoError | std.bytes.BytesError | std.secrets.SecretsError]:
token = std.secrets.token_urlsafe(32)?
print(len(token))?
print(len(std.secrets.token_hex(16)?))?
key = std.secrets.token_bytes(32)?
print(key.len())?
print(token != std.secrets.token_urlsafe(32)?)?
print(std.secrets.below(6u64)? < 6u64)?
mut nonce = std.bytes.empty()?
nonce.resize(12, 0u8)?
std.secrets.fill(&mut nonce, 0, 12)?
print(nonce.len())?
print(str.repr(std.secrets.token_hex(-1))?)?
Ok(())
43
32
32
True
True
12
Result[str, std.secrets.SecretsError].Err(std.secrets.SecretsError.Allocation(AllocError.CapacityOverflow))
The argument is a number of secret bytes, not characters: 32 bytes become 43
base64url characters or 64 hex digits. token_urlsafe is safe to put in a URL
or a file name. below is an unbiased integer, for example to pick a winner.
A negative count is a size that cannot be allocated, so it returns
Allocation(AllocError.CapacityOverflow).
If the system cannot supply entropy, these functions return
SecretsError.Entropy. They never fall back to a weaker source, and a failed
fill leaves zeros, not a partly random buffer.
To start a simulation from an unpredictable seed, pass
std.secrets.random_u64()? to std.random.seeded and print the seed so the run
can be repeated.
For normal, exponential or floating uniform samples, pass that generator to
a std.stat distribution. Each distribution documents its
draw count, and failed sampling leaves the generator unchanged.
See the random reference for the algorithm and its stability contract, and the secrets reference for the failure policy and costs.
Summarize observations and model uncertainty
Use std.stat for small numerical datasets and probability calculations.
Choose a population statistic when the observations are the complete group
you want to describe; choose a sample statistic when estimating the spread
of a larger population from a sample.
import std.stat as stat
def main() -> Result[(), AllocError | IoError | stat.StatError]:
measurements = [4.0, 1.0, 3.0, 2.0]?
print(stat.mean(&measurements)?, stat.median(&measurements)?)?
print(stat.quantiles(&measurements)?)?
print(measurements)?
Ok(())
2.5 2.5
[1.75, 2.5, 3.25]
[4.0, 1.0, 3.0, 2.0]
Median and quantiles copy and sort the input; mean and standard deviation do not allocate. Quantiles interpolate between observed values with the smallest and largest at probabilities zero and one. Inputs must be finite, and sample variance needs at least two observations.
Distributions model probabilities. A uniform distribution over a two-to-six second delay has accumulated one quarter of its probability by three seconds:
import std.stat as stat
import std.random as random
def main() -> Result[(), IoError | stat.StatError]:
delays = stat.uniform(2.0, 6.0)?
print(delays.cdf(3.0)?, delays.inverse_cdf(0.25)?)?
mut first = random.seeded(42u64)
mut repeated = random.seeded(42u64)
print(delays.sample(&mut first) == delays.sample(&mut repeated))?
bell = stat.normal()?
print(bell.cdf(0.0)?)?
Ok(())
0.25 3.0
True
0.5
normal(mean, sigma) and exponential(rate) also provide density, CDF,
inverse CDF and sampling. Their samples can overflow and therefore return
Result; failure leaves the supplied generator unchanged. Seed explicitly
to reproduce a simulation, and use std.secrets for security-sensitive
randomness. Normal/exponential calculations use platform math, so their last
bits may differ between math-library versions.
The reference describes numeric limits, errors, costs and sampling’s fixed draw counts.
Identify records with UUIDs
Use std.uuid when records need identifiers that can be created independently,
without asking a central counter. Choose v4() for a random identifier, or
v7() when placing recently created records close together in a sorted index
is useful. Both use operating-system entropy and report failure through
Result.
import std.uuid as uuid
from std.secrets import SecretsError
def main() -> Result[(), IoError | SecretsError | RangeError]:
request_id = uuid.v4()?
record_id = uuid.v7()?
print(request_id.version(), record_id.version())?
Ok(())
Option[u8].Some(4) Option[u8].Some(7)
The identifiers change each run, so this example prints only their versions.
v7 embeds a wall-clock timestamp in milliseconds. It does not promise
creation order within a millisecond, or when the clock moves backwards. Use
std.secrets tokens rather than UUIDs for passwords and access credentials.
Read, store and compare
Keep identifiers as Uuid values while working with them. Convert to canonical
text for display, or to 16 bytes for binary storage:
import std.uuid as uuid
from std.bytes import BytesError
def main() -> Result[(), IoError | ParseError | BytesError]:
id = uuid.parse("017F22E2-79B0-7CC3-98C4-DC0C0C07398F")?
print(id.to_string()?)?
raw = id.to_bytes()?
restored = uuid.from_bytes(&raw)?
print(raw.len(), restored == id)?
print(id.unix_milliseconds())?
Ok(())
017f22e2-79b0-7cc3-98c4-dc0c0c07398f
16 True
Option[u64].Some(1645557742000)
Parsing accepts either case but requires the hyphens exactly as shown. It
rejects whitespace, braces and malformed digits. Formatting always produces
lowercase text. The byte representation is big-endian, matching the text’s
left-to-right byte order. Use to_string() explicitly: ordinary class printing
is intended for inspecting fields, not interchange.
For a sorted list, borrow each UUID and use its pair of unsigned integer halves as the key. This avoids allocating a string per comparison:
import std.uuid as uuid
from std.bytes import BytesError
def uuid_key(id: &uuid.Uuid) -> (u64, u64):
id.as_u64_pair()
def main() -> Result[(), AllocError | IoError | ParseError | BytesError]:
mut records = [uuid.parse("017f22e2-79b1-7000-8000-000000000000")?,
uuid.parse("017f22e2-79b0-7cc3-98c4-dc0c0c07398f")?]?
records.sort_by_key(uuid_key)
for id in records:
print(id.to_string()?)?
Ok(())
017f22e2-79b0-7cc3-98c4-dc0c0c07398f
017f22e2-79b1-7000-8000-000000000000
UUIDs follow ordinary class ownership rules. Their comparison, parsing and numeric sorting keys need no allocation; creating output text or byte buffers does. See the reference for allocation-free writing into existing buffers and deterministic generator testing.
Summarizing a large stream
Use std.sketch when keeping every value costs too much memory and an approximate
answer is useful. Keep exact data when correctness requires exact membership or
counts. A sketch’s memory budget and accuracy are part of the choice.
This example counts distinct numeric customer identities, samples latency quantiles, and tracks frequent customer identities without retaining the stream:
import std.sketch as sketch
from std.sketch import SketchError
from std.bytes import BytesError
def main() -> Result[(), AllocError | IoError | SketchError | BytesError | ZeroDivisionError]:
mut customers = sketch.hyperloglog(10)?
mut latency = sketch.quantiles(128, seed=42u64)?
mut frequent = sketch.frequent_items(16)?
for index in range(1000):
customer = u64(index % 100)
customers.add(customer)
latency.add(f64(index % 50))?
frequent.add(customer)?
print(latency.count(), latency.retained())?
print(latency.quantile(0.0)?, latency.quantile(1.0)?)?
print(customers.estimate() > 80.0 and customers.estimate() < 120.0)?
Ok(())
1000 128
0.0 49.0
True
The quantile sketch retains a uniform sample and exact extrema. Its interior
quantiles are sample estimates, not guaranteed rank bounds. Frequent-item queries
return lower/upper count bounds, and candidates() returns possible heavy
hitters. Increase capacity for better accuracy. See the
reference for the different guarantees.
Skipping unnecessary lookups
A static xor filter can cheaply rule out keys before an expensive exact lookup. A positive answer still needs that lookup:
import std.sketch.xor as xor
from std.sketch import SketchError
from std.bytes import BytesError
def main() -> Result[(), AllocError | IoError | SketchError | BytesError]:
keys = [10u64, 20u64, 30u64]?
filter = xor.build(&keys, max_keys=100)?
print(filter.len(), filter.contains(20u64))?
Ok(())
3 True
For a changing set, std.sketch.cuckoo supports bounded growth and deletion.
Insertion returns a layer receipt. Keep it with the real record; deletion must
use that receipt and a known live occurrence, never just a positive membership
test:
import std.sketch.cuckoo as cuckoo
from std.sketch import SketchError
from std.bytes import BytesError
def main() -> Result[(), AllocError | IoError | SketchError | BytesError]:
mut filter = cuckoo.cuckoo(initial_buckets=16, max_buckets=112)?
layer = filter.insert(42u64)?
print(filter.contains(42u64))?
print(filter.remove(42u64, layer)?, filter.len())?
Ok(())
True
True 0
All three sketches and both filters provide versioned to_bytes() checkpoints
and bounded readers. Save the application’s key-hashing convention and cuckoo
receipts too. See xor filters and
cuckoo filters before choosing one.
Draw a chart
std.plot turns data into SVG without an external
plotting program. Its names follow ggplot2: a plot combines a table, mappings
from column names to visual properties, and a list of layers and styling choices.
This example renders into a memory stream so the lesson creates no files:
import std.plot as gg
import std.io
import std.bytes
from std.plot import Column, PlotError
from std.bytes import BytesError
def main() -> Result[(), AllocError | IoError | PlotError | BytesError]:
data = gg.table([
("hour", Column.Num([9.0, 12.0, 15.0]?)),
("temperature", Column.Num([16.0, 22.0, 20.0]?)),
]?)?
chart = gg.ggplot(data, gg.aes("hour", "temperature"), [
gg.geom_line(colour="navy"),
gg.geom_point(colour="navy", size=2),
gg.labs(title="Afternoon temperatures", x="Hour", y="Degrees C"),
gg.theme_minimal(),
]?)?
mut output = std.io.cursor(std.bytes.empty()?)
notes = chart.write_svg(&mut output)?
print(notes)?
print(output.data.to_text()?.startswith("<svg"))?
Ok(())
[]
True
For a file, replace the cursor and write_svg with
notes = chart.save("temperatures.svg")?. Open the file in a browser or include
it in a document. Saving borrows the chart, so you can save it again at another
size. The default is 6 by 4 inches; named width, height, and units control
dimensions. Only SVG is supported currently.
The distinction between mapping and setting matters. aes(colour="kind")
looks up a column and creates a colour scale and legend. geom_point(colour="navy")
uses one constant colour. Unknown column names fail when the plot is constructed.
Plot your own records
Typed accessors turn lists of ordinary classes into columns. Compute derived
values in these accessors, where Plenty can type-check them; aes does not
evaluate strings as expressions.
import std.plot as gg
import std.io
import std.bytes
from std.plot import PlotError
from std.bytes import BytesError
class Sale:
region: str
amount: f64
def main() -> Result[(), AllocError | IoError | PlotError | BytesError]:
sales = [Sale("North", 12.0), Sale("South", 18.0), Sale("North", 5.0)]?
data = gg.rows(&sales, [
gg.text("region", def(s: &Sale) -> str: s.region)?,
gg.num("amount", def(s: &Sale) -> f64: s.amount)?,
]?)?
base = gg.ggplot(data, gg.aes("region", "amount", fill="region"), [
gg.geom_col(),
gg.scale_fill_manual({"North": "navy", "South": "#D55E00"}?),
gg.theme_minimal(),
]?)?
labelled = base.plus([gg.labs(title="Sales by region", y="Sales")]?)?
mut output = std.io.cursor(std.bytes.empty()?)
drop(labelled.write_svg(&mut output)?)
print(len(sales), output.data.to_text()?.startswith("<svg"))?
Ok(())
3 True
geom_col uses supplied heights and stacks rows at the same x by default. Use
geom_bar with no y mapping to count rows instead, or geom_histogram(bins=20)
with a continuous x mapping to count numeric intervals. For grouped bars,
position=gg.position_dodge() places groups side by side.
The named palette keeps each region’s colour stable even if the input order
changes. Include every observed region; extra names are allowed. For a numeric
colour mapping, use gg.scale_colour_gradient(low="#132B43", high="#56B1F7")
to choose endpoints instead. The legend follows the chosen colours. See
manual palettes and gradients
for interpolation and validation rules.
plus leaves the original plot unchanged and shares its immutable data. rows
copies selected values, so the original records also remain usable. For columns
you already own, table moves the lists without copying them.
Missing and non-finite mapped values are omitted, with a note returned by the
renderer. Check these notes when interpreting a chart. The final example
discards them deliberately because its small, fixed dataset contains no missing
values. File/stream errors and allocation failures still propagate through ?.
See the reference for supported layers, layout
limits, and differences from the larger plotting proposal.
Combine plot layers and control axes
A summary often has different columns and fewer rows than its observations.
Give that layer its own table and mapping. Use inherit_aes=False when it
should not inherit grouping or other aesthetics from the observations.
import std.plot as gg
import std.io
import std.bytes
from std.plot import Column, PlotError
from std.bytes import BytesError
def main() -> Result[(), AllocError | IoError | PlotError | BytesError]:
observations = gg.table([
("dose", Column.Num([1.0, 10.0, 100.0]?)),
("response", Column.Num([2.0, 4.0, 8.0]?)),
("site", Column.Text(["North", "South", "North"]?)),
]?)?
reference = gg.table([
("dose", Column.Num([1.0, 100.0]?)),
("expected", Column.Num([2.0, 7.0]?)),
]?)?
chart = gg.ggplot(observations, gg.aes("dose", "response", colour="site"), [
gg.geom_point(size=2),
gg.geom_line(data=reference, mapping=gg.aes("dose", "expected"),
inherit_aes=False, colour="navy"),
gg.scale_x_log10(breaks=Some([1.0, 10.0, 100.0]?),
labels=Some(["1 mg", "10 mg", "100 mg"]?)),
gg.labs(title="Observed and expected response"),
gg.theme_minimal(),
]?)?
mut output = std.io.cursor(std.bytes.empty()?)
notes = chart.write_svg(&mut output)?
print(notes, output.data.to_text()?.startswith("<svg"))?
Ok(())
[] True
The line inherits no site grouping, so it connects the two reference points. Both layers train the same axes. The logarithmic x scale gives equal horizontal spacing to each tenfold increase, while breaks and labels use original doses. The explicit labels include units without changing the data.
To override only one mapping while keeping the others, use a layer argument
such as mapping=gg.aes(y="expected") with the default inherit_aes=True.
Every inherited name must exist in that layer’s table.
For a categorical axis, gg.scale_x_discrete(limits=Some(["South", "North"]?))
sets the display order and excludes any unlisted categories. Its labels
argument can supply display names in the same order. It requires a discrete
x mapping; the dose example uses a continuous one.
Use coord_cartesian(xlim=(1.0, 10.0)) to zoom while retaining all rows for
statistics. A scale’s limits instead removes values before computation, which
can change histogram counts. Values outside a transformation’s domain are
also omitted and reported in the renderer’s notes. Check these notes when
working with real input.
Long titles and labels wrap automatically. The renderer reserves space for
their text boxes and checks for overlapping ticks before writing SVG. If it
reports that the chart cannot fit, increase width or height, shorten the
labels, or provide fewer breaks. For example, a presentation chart can use
chart.write_svg(&mut output, width=10.0, height=6.0)?. Labels are never
silently dropped to make the chart fit.
See position scales for the full contracts, including square-root and reversed axes, and draw a chart for file output and record accessors.
Compare panels and show ranges
Use facets to compare the same measurements across groups. Add error bars or a ribbon when your data includes lower and upper bounds. The plot draws these supplied values; calculating a confidence interval or another statistical summary is a separate step.
This example uses illustrative measurement bounds, not fitted confidence intervals:
import std.plot as gg
import std.io
import std.bytes
from std.plot import Column, PlotError
from std.bytes import BytesError
def main() -> Result[(), AllocError | IoError | PlotError | BytesError]:
data = gg.table([
("day", Column.Num([1.0, 2.0, 3.0, 1.0, 2.0, 3.0]?)),
("value", Column.Num([2.0, 4.0, 3.0, 3.0, 5.0, 4.0]?)),
("lower", Column.Num([1.0, 2.0, 1.5, 2.0, 3.0, 2.5]?)),
("upper", Column.Num([3.0, 6.0, 4.5, 4.0, 7.0, 5.5]?)),
("site", Column.Text(["North", "North", "North",
"South", "South", "South"]?)),
]?)?
chart = gg.ggplot(data, gg.aes("day", "value", ymin="lower", ymax="upper"), [
gg.geom_ribbon(fill="blue", alpha=0.2),
gg.geom_errorbar(colour="navy", width=0.15),
gg.geom_line(colour="navy"),
gg.geom_point(colour="navy"),
gg.facet_wrap("site", ncol=2),
gg.labs(title="Measurement ranges by site", x="Day", y="Value"),
gg.theme_minimal(),
]?)?
mut output = std.io.cursor(std.bytes.empty()?)
notes = chart.write_svg(&mut output, width=10.0, height=4.0)?
print(notes, output.data.to_text()?.startswith("<svg"))?
Ok(())
[] True
Fixed axes let you compare heights directly between sites. Choose
scales=gg.Scales.FreeY when each panel needs its own vertical range, but then
compare the axis values rather than the visual heights. With two grouping
variables, use gg.facet_grid(rows="region", cols="period").
A reference layer whose table omits site appears in every panel. Give it its
own mapping with inherit_aes=False when it should not inherit aesthetics
whose columns are absent; see combine plot layers.
Histogram counts and bar stacks remain independent within each panel.
Both interval endpoints contribute to the axes. Missing bounds omit an error
bar and interrupt a ribbon at that x; inspect the returned notes with real data.
geom_linerange draws the same interval without caps. To display horizontal
intervals, add coord_flip.
The larger output size gives two panels room for their labels. All panels are checked before writing SVG, so a chart that cannot fit reports an error instead of hiding labels. See facets and uncertainty and intervals for the full contracts and current limits.
Read and write JSON
std.json decodes and encodes JSON under explicit limits. It offers two levels:
a value tree for documents that fit in memory, and events for documents that
should not be held whole.
Parse a document into a tree
import std.json
from std.json import Json
def main() -> Result[(), AllocError | IoError | std.json.JsonError]:
bounds = std.json.limits()
value = std.json.parse_text("{\"name\": \"Ada\", \"tags\": [\"x\", \"y\"], \"age\": 36, \"ratio\": 0.5}", &bounds)?
match &value:
case Json.Object(members):
for member in members:
match &member.value:
case Json.Text(text):
print(member.key, "is the text", text)?
case Json.Int(number):
print(member.key, "is the integer", *number)?
case Json.Float(number):
print(member.key, "is the float", *number)?
case Json.Array(items):
print(member.key, "has", len(items), "items")?
case _:
print(member.key, "is something else")?
print(std.json.find(members, "age"), std.json.find(members, "email"))?
case _:
print("not an object")?
print(std.json.encode_text(&value, &bounds)?)?
Ok(())
name is the text Ada
tags has 2 items
age is the integer 36
ratio is the float 0.5
Option[i64].Some(2) Option[i64].Nothing
{"name":"Ada","tags":["x","y"],"age":36,"ratio":0.5}
parse_text returns a Json value, an enum with one variant per JSON type.
match is how you find out which one you have. A number written without a
fraction or exponent is Json.Int; any other number is Json.Float.
An object is a list[Member] in source order, where each member has a key and
a value. find returns the index of the first member with a key. Output is
compact: no spaces or line breaks, and members stay in the order of the list.
Every call takes Limits. limits() gives the defaults: 64 levels of nesting,
64 MiB of input, 64 MiB of output, and duplicate keys rejected.
Errors say where
A JsonError carries a fault, the byte offset, and a line and column
that both start at 1.
import std.json
from std.json import Limits
def attempt(text: &str, bounds: &Limits) -> Result[(), AllocError | IoError | std.json.JsonError]:
match std.json.parse_text(text, bounds):
case Ok(value):
print("ok", std.json.encode_text(&value, bounds)?)?
case Err(error):
print(error.fault, "at byte", error.offset, "line", error.line, "column", error.column)?
Ok(())
def main() -> Result[(), AllocError | IoError | std.json.JsonError]:
mut bounds = std.json.limits()
attempt("[1, 2,\n oops]", &bounds)?
attempt("[1, 2", &bounds)?
attempt("{\"a\": 1, \"a\": 2}", &bounds)?
attempt("[9223372036854775808]", &bounds)?
attempt("[1] [2]", &bounds)?
bounds.max_depth = 2
attempt("[[1]]", &bounds)?
attempt("[[[1]]]", &bounds)?
Ok(())
std.json.JsonFault.UnexpectedByte at byte 9 line 2 column 3
std.json.JsonFault.UnexpectedEnd at byte 5 line 1 column 6
std.json.JsonFault.DuplicateKey at byte 9 line 1 column 10
std.json.JsonFault.NumberOutOfRange at byte 1 line 1 column 2
std.json.JsonFault.TrailingData at byte 4 line 1 column 5
ok [[1]]
std.json.JsonFault.DepthExceeded at byte 2 line 1 column 3
The decoder accepts standard JSON and nothing else: no comments, no trailing
commas, one document per input. An integer that does not fit i64 is an error
in the tree, never a rounded float. max_depth counts open arrays and objects,
so the limit of 2 accepts [[1]] and stops at the third [.
Stream events instead of building a tree
A Decoder reports one Event at a time, and an Encoder accepts one call per
event. Neither holds the document, so memory stays bounded however long the
input is.
import std.bytes
import std.io
import std.json
from std.json import Event
def main() -> Result[(), IoError | AllocError | ParseError | std.json.JsonError | std.bytes.BytesError | OverflowError[i64]]:
bounds = std.json.limits()
mut reader = std.io.cursor(std.bytes.from_text("[{\"id\": 7, \"ok\": true}, {\"id\": 18446744073709551615}]")?)
mut decoder = std.json.decoder(&bounds)?
mut ids = 0
mut expecting_id = False
while True:
event = std.json.read_event(&mut decoder, &mut reader)?
match &event:
case Event.End:
break
case Event.Key(name):
expecting_id = name == "id"
case Event.Number(text):
if expecting_id:
ids += 1
print("id", u64.parse(text)?)?
case _:
pass
print(ids, "ids in", decoder.offset(), "bytes")?
mut encoder = std.json.encoder(&bounds)?
encoder.start_object()?
encoder.key("ids")?
encoder.integer(ids)?
encoder.key("note")?
encoder.text("line one\nline two")?
encoder.end_object()?
mut out = std.io.cursor(std.bytes.empty()?)
std.json.write_pending(&mut encoder, &mut out)?
print(out.data.to_text()?)?
match encoder.null():
case Ok(_):
print("written")?
case Err(error):
print(error.fault)?
Ok(())
id 7
id 18446744073709551615
2 ids in 53 bytes
{"ids":2,"note":"line one\nline two"}
std.json.JsonFault.InvalidState
read_event pulls bytes from any std.io reader as the decoder needs them and
returns Event.End after the document. Event.Number holds the number’s text
exactly as written, so the program chooses the type: here u64, which holds a
value the tree’s i64 cannot.
The encoder checks call order. A key outside an object, a value where a key
belongs, or a second top-level value is InvalidState, so an encoder cannot
produce malformed JSON. write_pending hands the bytes produced so far to a
writer; call it between events to keep the buffer small.
The std.json reference lists every fault,
limit, and cost.
Rendering text templates
Use std.template when the layout of a page or message is clearer as text than
as a sequence of string concatenations. Register a layout once and pass values
for each render. Templates cannot call your program’s functions or read files.
import std.template
from std.template import TemplateError
from std.json import Json, Member
def main() -> Result[(), TemplateError | AllocError | IoError]:
bounds = std.template.limits()
mut pages = std.template.engine(&bounds)?
pages.add("layout", "<h1>{% block title %}Report{% endblock %}</h1>{% block body %}{% endblock %}")?
pages.add("report", "{% extends 'layout' %}{% block title %}{{ title }}{% endblock %}{% block body %}<ul>{% for name in names %}<li>{{ name }}</li>{% else %}<li>No entries</li>{% endfor %}</ul>{% endblock %}")?
values = Json.Object([
Member("title", Json.Text("Team & friends")),
Member("names", Json.Array([Json.Text("Ada"), Json.Text("Grace")]?)),
]?)
print(pages.render("report", &values)?)?
Ok(())
<h1>Team & friends</h1><ul><li>Ada</li><li>Grace</li></ul>
The context uses the JSON value model explicitly; no
class-to-template conversion happens automatically. Fields such as
{{ person.name }} look up object members. A missing field returns an error
rather than silently producing an empty string. An empty array takes the loop’s
else branch, which is useful for empty reports.
Interpolated text is HTML-escaped by default. Keep user text in HTML text or
quoted ordinary attributes. HTML escaping does not validate links or JavaScript;
do not place user values in scripts, styles or tag names. For a plain-text
message, call pages.render("report", &values, html=False) with a plain-text
template. The safe filter deliberately disables escaping for trusted markup.
Set limits for your application before constructing the engine: for example,
lower max_output for a short notification. Registration and rendering are
fallible, and the example propagates those failures to the caller. Keep the
engine and render it again with different values to reuse its parsed templates.
For a small fragment assembled in your program, use std.template.escape
rather than a chain of string replacements. Choose HtmlContext.Text for text
children or the quote-specific attribute mode for each attribute value. For
example, quotes should stay literal in text but must be escaped inside a
double-quoted attribute:
from std.template import escape, HtmlContext, TemplateError
def main() -> Result[(), TemplateError | AllocError | IoError]:
label = "Say \"hello\" & smile"
attribute = escape(&label, HtmlContext.DoubleQuotedAttribute, max_bytes=256)?
child = escape(&label, HtmlContext.Text, max_bytes=256)?
print(f"<button title=\"{attribute}\">{child}</button>"?)?
Ok(())
<button title="Say "hello" & smile">Say "hello" & smile</button>
max_bytes includes the expanded entities. The helper returns ordinary text:
passing its output into a template would escape it again. Keep raw values in
Json.Text when using Engine.render, and use these helpers once per original
value when assembling HTML directly. The context is your choice; the library
does not detect HTML positions or validate URLs, scripts, styles or names.
See the std.template reference for the
supported syntax, filters, inheritance rules and precise limits.
Solve systems and factor matrices
std.linalg gives you dense matrices and column vectors of f64 values. It
uses faer’s numerical algorithms through an optional native engine. Install
the linear algebra engine before running
these examples. Matrix construction and element access work without it.
Solve a linear system
Pass matrix entries in row order. Use solve to find x in A x = b:
import std.linalg as la
def main() -> Result[(), AllocError | IoError | la.LinalgError]:
a = la.matrix(2, 2, &[3.0, 1.0, 1.0, 2.0]?)?
b = la.vector(&[9.0, 8.0]?)?
x = a.solve(&b)?
print(x.get(0, 0)?, x.get(1, 0)?)?
predicted = (a * x)?
residual = (predicted - b)?
print(residual.to_list()?)?
Ok(())
2.0 3.0
[0.0, 0.0]
Both inputs are preserved. The result is a two-row, one-column matrix. For
several right-hand sides with the same a, compute factor = a.lu()? once,
then call factor.solve(&b)? for each. This reuses the expensive factorization.
a * x is a matrix product, so it also applies a matrix to a column vector.
Addition and subtraction combine corresponding entries of equal-shaped matrices.
The parentheses in (predicted - b)? propagate the operation’s error. A small
residual is useful evidence about a solve; for general floating-point inputs,
compare it with a tolerance appropriate to your data.
Scale with (2.0 * x)? or x.scale(2.0)?, and negate with (-x)?.
Scalar multiplication currently puts the scalar on the left: x * 2.0 is
unsupported. These operations preserve their inputs and allocate new results.
There is no automatic fusion of a chain or fallible +=; use an explicit
assignment such as x = (x + adjustment)? with a mutable binding.
Inspect a low-rank approximation
SVD separates a matrix into two orthogonal factors and a list of singular values. Small singular values indicate directions that contribute little:
import std.linalg as la
def main() -> Result[(), AllocError | IoError | la.LinalgError]:
a = la.diagonal(&[4.0, 2.0, 0.01]?)?
(u, values, v) = a.svd()?
print(values)?
print(u.rows(), u.cols(), v.rows(), v.cols())?
Ok(())
[4.0, 2.0, 0.01]
3 3 3 3
To reconstruct, create d = la.diagonal(&values)? and vt = v.transpose()?,
then compute scaled = (u * d)? and reconstructed = (scaled * vt)?.
For rectangular inputs the factors have only min(rows, cols) columns. qr,
cholesky, eigh, and eig provide other decompositions; their contracts are
in the linear algebra reference.
All matrix-building operations are fallible. Check shapes and handle errors
normally with ?. Floating-point calculations are approximate: compare residuals
with a tolerance appropriate to your data, not exact factor entries. A successful
solve does not guarantee accuracy for a nearly singular matrix.
Hash and sign bytes
std.crypto computes BLAKE3, SHA-256 and SHA-512 digests and keyed tags over
std.bytes buffers, and compares secrets in
constant time. For AEAD encryption and public-key signatures, see
Encrypt and authenticate bytes.
Digests
A digest is a fixed-size fingerprint of some bytes. The one-shot functions
take a whole buffer; a Digest takes input in pieces, which gives the same
result as hashing the pieces joined together:
import std.bytes
import std.crypto
import std.encoding
from std.crypto import Algorithm
type CryptoBufferError = std.encoding.EncodingError | std.crypto.CryptoError | std.bytes.BytesError
def main() -> Result[(), AllocError | IoError | CryptoBufferError]:
data = std.bytes.from_text("abc")?
digest = std.crypto.sha256(&data)?
print(digest.len())?
print(std.encoding.hex_text(&digest)?)?
mut hasher = std.crypto.digest(Algorithm.Sha256)?
hasher.update_text("a")?
hasher.update_text("bc")?
print(std.crypto.equal(&hasher.finish()?, &digest))?
print(str.repr(hasher.update_text("more"))?)?
Ok(())
32
ba7816bf8f01cfea414140de5dae2223b00361a396177a9cb410ff61f20015ad
True
Result[(), std.crypto.CryptoError].Err(std.crypto.CryptoError.Finished)
finish ends a digest. Feeding it afterwards is the error Finished, not a
new hash; make a new Digest instead. update(data, start, stop) takes a
range of a buffer, so a large input can be hashed while it is read, without
first collecting it in memory.
BLAKE3
For BLAKE3, choose Algorithm.Blake3 or the one-shot blake3 function:
import std.bytes
import std.crypto
from std.crypto import Algorithm
def main() -> Result[(), IoError | AllocError | std.bytes.BytesError | std.crypto.CryptoError]:
data = std.bytes.from_text("hello world")?
expected = std.crypto.blake3(&data)?
mut hasher = std.crypto.digest(Algorithm.Blake3)?
hasher.update_text("hello ")?
hasher.update_text("world")?
print(expected.len(), std.crypto.equal(&hasher.finish()?, &expected))?
Ok(())
32 True
BLAKE3 also has a native keyed hash with exactly 32 key bytes and a key
derivation mode. Use blake3_keyed_hasher(key) for a Mac with the same
update and verify methods used for HMAC below. It is a distinct algorithm:
hmac(Algorithm.Blake3, key) reports UnsupportedAlgorithm. See the
BLAKE3 reference for key derivation
contexts, allocation costs and limits.
Signing with HMAC
A digest proves nothing about who made it: anyone can hash a message. An HMAC tag mixes in a secret key, so only a holder of the key can produce a tag that verifies:
import std.bytes
import std.crypto
import std.encoding
from std.bytes import Bytes
from std.crypto import Algorithm
def sign(key: &Bytes, message: &str) -> Result[Bytes, AllocError | IoError | std.crypto.CryptoError]:
mut mac = std.crypto.hmac(Algorithm.Sha256, key)?
mac.update_text(message)?
Ok(mac.finish()?)
def check(key: &Bytes, message: &str, tag: &Bytes) -> Result[bool, AllocError | IoError | std.crypto.CryptoError]:
mut mac = std.crypto.hmac(Algorithm.Sha256, key)?
mac.update_text(message)?
Ok(mac.verify(tag)?)
type CryptoBufferError = std.encoding.EncodingError | std.crypto.CryptoError | std.bytes.BytesError
def main() -> Result[(), AllocError | IoError | CryptoBufferError]:
key = std.bytes.from_text("server key")?
tag = sign(&key, "user=ada")?
print(std.encoding.hex_text(&tag[:8])?)?
print(check(&key, "user=ada", &tag)?)?
print(check(&key, "user=bob", &tag)?)?
other = std.bytes.from_text("another key")?
print(check(&other, "user=ada", &tag)?)?
Ok(())
8f60a639ef523af7
True
False
False
Check a tag with verify, never by finishing and comparing with ==. ==
stops at the first differing byte, so how long it takes reveals how much of a
forged tag was right. verify and std.crypto.equal take the same time
wherever the bytes differ.
A real key should be secret and random: make one with
std.secrets.token_bytes(32) from Randomness and secrets
rather than from text.
See the crypto reference for the guarantees, costs, and what the module does not provide.
Encrypt and authenticate bytes
std.crypto supplies authenticated
encryption and public-key operations over Bytes. Use a defined protocol
for exchanged messages: encryption alone does not manage identities,
replays, message framing, or keys.
Encrypt one message
AEAD encryption protects both the message and associated data such as a protocol header. The associated data stays readable, but changing it makes verification fail. A recipient must have the same secret key, nonce, and associated data.
This example creates a fresh key for one message. When reusing a key, the application must ensure it never repeats a nonce, including across restarts. The API does not track that for you.
import std.bytes
import std.crypto
import std.secrets
from std.crypto import AeadAlgorithm
type BufferError = std.bytes.BytesError | std.crypto.AdvancedError | std.secrets.SecretsError
def main() -> Result[(), IoError | AllocError | BufferError]:
key = std.secrets.token_bytes(32)?
nonce = std.secrets.token_bytes(12)?
header = std.bytes.from_text("example-v1")?
mut message = std.bytes.from_text("meet at noon")?
std.crypto.seal_in_place(AeadAlgorithm.ChaCha20Poly1305, &key, &nonce, &header, &mut message)?
# The buffer now holds ciphertext followed by a 16-byte tag.
print(message.len())?
std.crypto.open_in_place(AeadAlgorithm.ChaCha20Poly1305, &key, &nonce, &header, &mut message)?
print(message.to_text()?)?
Ok(())
28
meet at noon
Opening returns an error and zeros the buffer if authentication fails; do
not use its contents after an error. The ? above propagates that error.
Reserve 16 extra bytes before sealing when the buffer needs to be reused
without allocation. Opening never allocates.
Sign with a private seed
An Ed25519 signature allows someone with the public key to check a message without knowing the signing seed. The seed remains secret. A verifier also needs an authenticated source for the public key: accepting any key supplied alongside the signature proves no identity.
import std.bytes
import std.crypto
import std.secrets
type SigningError = std.bytes.BytesError | std.crypto.AdvancedError | std.secrets.SecretsError
def main() -> Result[(), IoError | AllocError | SigningError]:
seed = std.secrets.token_bytes(32)?
public_key = std.crypto.ed25519_public_key(&seed)?
message = std.bytes.from_text("release manifest")?
signature = std.crypto.ed25519_sign(&seed, &message)?
print(std.crypto.ed25519_verify(&public_key, &message, &signature)?)?
changed = std.bytes.from_text("modified manifest")?
print(std.crypto.ed25519_verify(&public_key, &changed, &signature)?)?
Ok(())
True
False
Signing can reuse its seed. Keys and signatures have fixed lengths; malformed
lengths are errors, while a well-sized invalid signature returns False. A
public key that anyone could sign for, one of
small order,
is the error SmallOrderKey.
Derive a key after an exchange
Ephemeral X25519 keys can each perform one agreement. Exchange their public keys first, authenticate the exchange through the application’s protocol, and derive an application key with HKDF. The example runs both peers locally; it demonstrates agreement and key derivation, without providing a network authentication protocol.
import std.bytes
import std.crypto
def main() -> Result[(), IoError | AllocError | std.bytes.BytesError | std.crypto.AdvancedError]:
mut alice = std.crypto.x25519()?
mut bob = std.crypto.x25519()?
alice_public = alice.public_key()?
bob_public = bob.public_key()?
alice_shared = alice.agree(&bob_public)?
bob_shared = bob.agree(&alice_public)?
salt = std.bytes.empty()?
context = std.bytes.from_text("example-v1 message key")?
alice_key = std.crypto.hkdf_sha256(&salt, &alice_shared, &context, 32)?
bob_key = std.crypto.hkdf_sha256(&salt, &bob_shared, &context, 32)?
print(std.crypto.equal(&alice_key, &bob_key))?
Ok(())
True
An agreement attempt consumes the private key even if the peer is rejected;
length and allocation errors leave it usable. HKDF separates key material
by purpose using context; it is not a password hashing function. None of
these byte buffers is automatically scrubbed when dropped. See the
key material and allocation contracts
before storing secrets or designing a protocol.
Log structured events
std.log records events: a level, a message, and typed fields. Where the
events go is up to the program. There is no global logger; you build one
around a sink and pass it to the code that logs.
A logger and a sink
A text sink writes each event as one line of key=value pairs to any
std.io writer. This one writes to standard
output so the lines appear below; a command-line tool would use
std.io.stderr() instead:
import std.io
import std.log
from std.log import Level, Clock
def main() -> Result[(), AllocError | IoError | std.log.LogError]:
sink = std.log.text_sink(std.io.stdout(), 1024)?
mut log = std.log.logger(sink, Level.Info, Clock.Off)
log.info("listening", [std.log.integer("port", 8080), std.log.text("host", "db.local")]?)?
log.debug("not shown", []?)?
log.warn("slow request", [std.log.floating("seconds", 1.5), std.log.boolean("retried", True)]?)?
print(log.emitted(), log.filtered())?
Ok(())
level=info msg="listening" port=8080 host="db.local"
level=warn msg="slow request" seconds=1.5e0 retried=true
2 1
The threshold Level.Info drops the debug event before it reaches the sink.
Text values are quoted and escaped, so one event is always one line.
Clock.Off leaves out the time; Clock.Wall stamps each event with the
current wall-clock time.
The fields of a dropped event were still built. When a field is costly to
compute, ask first with log.enabled(&Level.Debug).
Passing a logger to a library
Code that logs takes the logger as a parameter. Making it generic over the sink lets each caller pick the destination:
import std.log
from std.log import Level, Clock, Logger
def connect[S](log: &mut Logger[S], attempts: i64) -> Result[(), AllocError | std.log.LogError | OverflowError[i64]]:
for attempt in range(attempts):
log.warn("retrying", [std.log.integer("attempt", attempt + 1)]?)?
log.info("connected", []?)?
Ok(())
def main() -> Result[(), AllocError | IoError | std.log.LogError | AllocError | OverflowError[i64]]:
mut log = std.log.logger(std.log.memory_sink(10)?, Level.Trace, Clock.Off)
connect(&mut log, 2)?
for event in &log.sink.events:
print(std.log.level_name(&event.level), event.message, len(event.fields))?
Ok(())
warn retrying 1
warn retrying 1
info connected 0
A memory sink keeps events in a list, which suits tests. It holds at most the
capacity it was given; after that, logging returns the error Full and the
event is dropped.
Threads
Threads do not share one logger. Each builds its own around a channel sink, and
one thread writes the queued events out. A channel sink never waits: when the
queue is full, the event is dropped with Full, so slow output cannot stall the
rest of the program. See the log reference
for channel_sink, share, and drain.
Read and write TOML
std.toml parses TOML 1.0 documents, such as configuration files and project
manifests, into a value tree, and writes a tree back as TOML text. Every call
takes explicit limits.
Parse a configuration file
import std.toml
from std.toml import Entry, Toml
def show(entries: &list[Entry], indent: &str) -> Result[(), AllocError | IoError]:
for entry in entries:
name = (indent + entry.key)?
match &entry.value:
case Toml.Table(inner):
print((name + ":")?)?
show(inner, &(indent + " ")?)?
case Toml.Text(text):
print(name, "is the text", text)?
case Toml.Int(number):
print(name, "is the integer", *number)?
case Toml.Array(items):
print(name, "has", len(items), "items")?
case Toml.DateTime(moment):
print(name, "is a date in", moment.year)?
case _:
print(name, "is something else")?
Ok(())
def main() -> Result[(), AllocError | IoError | std.toml.TomlError]:
bounds = std.toml.limits()
text = "# Service settings\ntitle = \"cache\"\n\n[server]\nhost = \"0.0.0.0\"\nport = 8080\n\n[server.tls]\nenabled = true\n\n[[backend]]\nurl = \"http://a.internal\"\n\n[[backend]]\nurl = \"http://b.internal\"\n\n[release]\ndate = 2026-10-01\n"
value = std.toml.parse_text(text, &bounds)?
match &value:
case Toml.Table(root):
show(root, "")?
print(std.toml.find(root, "server"), std.toml.find(root, "client"))?
case _:
print("a document is always a table")?
Ok(())
title is the text cache
server:
host is the text 0.0.0.0
port is the integer 8080
tls:
enabled is something else
backend has 2 items
release:
date is a date in 2026
Option[i64].Some(1) Option[i64].Nothing
A document is always a Toml.Table: a list[Entry] in source order, where
each entry has a key and a value. [server] and [server.tls] become
nested tables, and each [[backend]] adds one table to the array backend.
find returns the index of the first entry with a key.
Toml has one variant per TOML type: Bool, Int, Float, Text,
DateTime, Array, and Table. Integers are i64. A DateTime holds the
date and time fields, its offset from UTC in minutes, and a kind that says
which parts the text gave: an offset date-time, a local date-time, a local
date, or a local time.
Errors say where
A TomlError carries a fault, the byte offset, and a line and column
that both start at 1.
import std.toml
from std.toml import Limits
def attempt(text: &str, bounds: &Limits) -> Result[(), IoError]:
match std.toml.parse_text(text, bounds):
case Ok(_):
print("ok")?
case Err(error):
print(error.fault, "at line", error.line, "column", error.column)?
Ok(())
def main() -> Result[(), AllocError | IoError]:
bounds = std.toml.limits()
attempt("name = \"Tom\"\nname = \"Pradyun\"", &bounds)?
attempt("[fruit]\na = 1\n[fruit]\nb = 2", &bounds)?
attempt("point = { x = 1 }\npoint.y = 2", &bounds)?
attempt("port = 08080", &bounds)?
attempt("day = 2026-02-29", &bounds)?
attempt("key = \"unterminated", &bounds)?
Ok(())
std.toml.TomlFault.DuplicateKey at line 2 column 1
std.toml.TomlFault.DuplicateTable at line 3 column 1
std.toml.TomlFault.KeyConflict at line 2 column 1
std.toml.TomlFault.InvalidNumber at line 1 column 8
std.toml.TomlFault.InvalidDateTime at line 1 column 7
std.toml.TomlFault.UnexpectedEnd at line 1 column 20
TOML forbids defining a key or table twice, and an inline table such as
{ x = 1 } is complete as written, so point.y cannot add to it. Integers may
not have leading zeros, and dates must exist. An error about a key or a table
points at the start of the line that repeats it.
Write a tree as TOML
import std.toml
from std.toml import Entry, Toml
def main() -> Result[(), AllocError | IoError | std.toml.TomlError]:
bounds = std.toml.limits()
mut package = list[Entry]()?
package.append(Entry("name", Toml.Text("player")))?
package.append(Entry("version", Toml.Text("0.1.0")))?
mut authors = list[Toml]()?
authors.append(Toml.Text("Ada"))?
package.append(Entry("authors", Toml.Array(authors)))?
mut root = list[Entry]()?
root.append(Entry("package", Toml.Table(package)))?
root.append(Entry("manifest-version", Toml.Int(1)))?
text = std.toml.encode_text(&Toml.Table(root), &bounds)?
print(text.rstrip()?)?
Ok(())
manifest-version = 1
[package]
name = "player"
version = "0.1.0"
authors = ["Ada"]
The writer puts each table’s plain keys first and its tables after them as
[section] headers, because a key written after a header would belong to that
header’s table. Parsing the output gives back the same values. The writer does
not keep comments or the original layout, so it suits generated files rather
than editing a file a person maintains.
Update a manifest while keeping comments
For a file maintained by a person, use a Document. Replacing a value keeps
the surrounding spelling and comments:
import std.toml
from std.toml import Toml
def main() -> Result[(), AllocError | IoError | std.toml.TomlError]:
source = "# release settings\n[package]\nversion = '0.1.0' # public version\n"
mut document = std.toml.document_text(source, &std.toml.limits())?
document.set(&["package", "version"]?, &Toml.Text("0.2.0"))?
document.insert(&["package", "publish"]?, &Toml.Bool(False))?
print(document.text()?.rstrip()?)?
Ok(())
# release settings
[package]
version = "0.2.0" # public version
publish = false
Paths list decoded keys, so a key containing a dot is one list element.
get reads an owned value; remove deletes a key or a whole section. Failed
edits leave the document unchanged. Arrays and inline tables are replaced as
whole values; this API does not edit individual array rows or inline fields.
See the editing reference for the exact
preservation rules and costs.
The std.toml reference lists every fault,
limit, and cost.
Compress bytes and streams
std.compress reads and writes gzip, zlib, raw DEFLATE and Brotli data. Whole
buffers go through compress and decompress; streams go through a
Compressor writer and a Decompressor reader from
std.io.
Whole buffers
compress takes a format, the data, and a level from 0 (fastest) to 9
(smallest). decompress takes a limit on the output size and fails rather
than produce more:
import std.bytes
import std.compress
from std.compress import Format
def main() -> Result[(), AllocError | IoError | std.compress.CompressError | std.bytes.BytesError]:
mut text = std.bytes.empty()?
for _ in range(0, 100):
text.extend(&std.bytes.from_text("to be or not to be ")?)?
packed = std.compress.compress(Format.Gzip, &text, 6)?
print(text.len())?
print(packed.len())?
plain = std.compress.decompress(Format.Gzip, &packed, 10000)?
print(plain == text)?
print(str.repr(std.compress.decompress(Format.Gzip, &packed, 1000))?)?
Ok(())
1900
59
True
Result[std.bytes.Bytes, std.compress.CompressError].Err(std.compress.CompressError.LimitExceeded)
Repetitive text shrinks a lot. That is also the danger in decompressing: a small input can expand a thousandfold. Always choose the limit from the memory the program can spend, never from the size of the input.
Brotli
Use Format.Brotli for Brotli files or HTTP br bodies. It has the same
functions and stream wrappers, with quality levels 0–11 instead of 0–9.
Quality 5 is a useful starting point. Always choose the decoded-output limit
from your application’s budget, independently of the compressed size.
import std.bytes
import std.compress
from std.compress import Format
def main() -> Result[(), IoError | AllocError | std.bytes.BytesError | std.compress.CompressError]:
source = std.bytes.from_text("a small Brotli message")?
packed = std.compress.compress(Format.Brotli, &source, 5)?
restored = std.compress.decompress(Format.Brotli, &packed, 1024)?
print(restored.to_text()?)?
Ok(())
a small Brotli message
Brotli allocates working memory during streaming and has no checksum; the
output limit does not cap all codec memory. For HTTP, check the actual
Content-Encoding before decoding. The
Brotli reference explains these
differences and HTTP handling.
Streams
A Compressor wraps any writer and is itself a writer, so write_all and
copy_all work with it. finish writes the end of the stream; without it
the output is incomplete. A Decompressor wraps any reader and is itself a
reader, decoding as it goes in fixed-size buffers:
import std.bytes
import std.compress
import std.io
from std.compress import Format
type CompressedStreamError = std.io.StreamError | std.io.Partial | std.compress.CompressError | std.bytes.BytesError
def main() -> Result[(), IoError | AllocError | CompressedStreamError | OverflowError[i64]]:
line = std.bytes.from_text("one line of a log\n")?
mut writer = std.compress.compressor(Format.Gzip, std.io.cursor(std.bytes.empty()?), 6)?
for _ in range(0, 1000):
std.io.write_all(&mut writer, &line)?
writer.finish()?
packed = writer.inner.data.copy()?
print(packed.len())?
mut reader = std.compress.decompressor(Format.Gzip, std.io.cursor(packed), 1000000)?
mut buffer = std.bytes.with_capacity(4096)?
buffer.resize(4096, 0u8)?
mut lines = 0
while True:
count = reader.read(&mut buffer, 0, 4096)?
if count == 0:
break
lines += buffer.count(10u8, 0, count)
print(lines)?
Ok(())
185
1000
The reader counts 18,000 bytes of lines while holding at most 4096 of them,
so the same loop handles a file much larger than memory. To compress a file,
wrap std.io.open_file(path, "w")? instead of a cursor.
Damaged input
gzip and zlib end with a checksum of the original data, so a damaged stream
is caught. decompress returns only an error. A Decompressor reports it
through the Reader protocol as a StreamError, and failure names the
exact cause:
import std.bytes
import std.compress
import std.io
from std.compress import Format
def main() -> Result[(), AllocError | IoError | std.compress.CompressError | std.bytes.BytesError | OverflowError[i64] | OverflowError[u8]]:
data = std.bytes.from_text("checked at the end")?
mut packed = std.compress.compress(Format.Gzip, &data, 6)?
last = packed.len() - 1
packed.set(last, packed.get(last) + 1u8)
print(str.repr(std.compress.decompress(Format.Gzip, &packed, 100))?)?
mut reader = std.compress.decompressor(Format.Gzip, std.io.cursor(packed), 100)?
mut out = std.bytes.empty()?
match std.io.read_up_to(&mut reader, &mut out, 100):
case Ok(_):
print("read it all")?
case Err(partial):
print(str.repr(partial.cause)?)?
print(str.repr(reader.failure())?)?
Ok(())
Result[std.bytes.Bytes, std.compress.CompressError].Err(std.compress.CompressError.Checksum)
std.io.StreamError.Io(IoError.InvalidInput)
Option[std.compress.CompressError].Some(std.compress.CompressError.Checksum)
The checksum comes last, so a reader may already have returned some bytes
before it finds the damage. Treat streamed output as unverified until read
returns 0.
See the std.compress reference for every
error, the rules for input after a stream, and memory costs.
Read and write archives
std.archive reads and writes TAR and ZIP archives over
std.io streams, and extracts them safely into
a new directory.
Write and list a TAR archive
A TarWriter wraps any writer. Each add_ call writes one entry, and
finish writes the end of the archive. A TarReader returns the entries in
order; between calls to next, it is a reader of the current entry’s
content:
import std.archive
import std.bytes
import std.io
import std.path
def main() -> Result[(), IoError | AllocError | std.io.Partial | std.archive.ArchiveError | std.bytes.BytesError]:
mut writer = std.archive.tar_writer(std.io.cursor(std.bytes.empty()?))?
writer.add_dir(&std.path.from_text("notes")?, 493, 1700000000)?
writer.add_file(&std.path.from_text("notes/todo.txt")?, &std.bytes.from_text("buy milk")?, 420, 1700000000)?
writer.finish()?
packed = writer.inner.data.copy()?
print(packed.len())?
mut reader = std.archive.tar_reader(std.io.cursor(packed))?
while True:
match reader.next()?:
case Some(entry):
mut content = std.bytes.empty()?
drop(std.io.read_up_to(&mut reader, &mut content, 1000)?)
print(decode_utf8(entry.name.as_bytes())?, entry.size)?
if content.len() > 0:
print(content.to_text()?)?
case Nothing:
break
Ok(())
2560
notes/ 0
notes/todo.txt 8
buy milk
TAR stores each header and each file’s content in 512-byte blocks and ends
with two empty blocks, so even a tiny archive takes a few kilobytes. To
compress it, wrap the writer in a
std.compress Compressor.
ZIP archives
A ZIP archive keeps a directory of its entries at the end, so a
ZipReader needs a stream it can seek: a file or a cursor. entries lists
everything; open selects one entry to read:
import std.archive
import std.bytes
import std.io
import std.path
from std.archive import Method
def main() -> Result[(), IoError | AllocError | std.io.Partial | std.archive.ArchiveError | std.bytes.BytesError]:
mut writer = std.archive.zip_writer(std.io.cursor(std.bytes.empty()?))?
mut text = std.bytes.empty()?
for _ in range(0, 50):
text.extend(&std.bytes.from_text("all work and no play ")?)?
writer.add_file(&std.path.from_text("story.txt")?, &text, Method.Deflate, 420, 1700000000)?
writer.add_file(&std.path.from_text("short.txt")?, &std.bytes.from_text("hi")?, Method.Stored, 420, 1700000000)?
writer.finish()?
mut reader = std.archive.zip_reader(std.io.cursor(writer.inner.data.copy()?), 100)?
entries = reader.entries()
for entry in entries:
print(decode_utf8(entry.name.as_bytes())?, entry.size, entry.compressed)?
reader.open(1)?
mut content = std.bytes.empty()?
drop(std.io.read_up_to(&mut reader, &mut content, 100)?)
print(content.to_text()?)?
Ok(())
story.txt 1050 35
short.txt 2 2
hi
The reader checks each entry’s size and CRC-32 when its content ends.
Extract safely
An archive is untrusted input. Its names may point outside the target
(../../.bashrc), its links may lead elsewhere, and a small archive may
expand into terabytes. extract_tar and extract_zip check every entry
against a Policy, write into a hidden staging directory, and rename it to
the destination only when everything succeeded:
import std.archive
import std.bytes
import std.fs
import std.io
import std.path
from std.sync import CancellationToken
type ArchiveOperationError = std.archive.ArchiveError | std.bytes.BytesError | std.fs.FsError
def main() -> Result[(), AllocError | IoError | ArchiveOperationError]:
parent = std.fs.root(&std.path.from_text(".")?)?
mut scratch = std.fs.temporary_dir(&parent, &std.path.from_text("archive-lesson-")?, 8)?
mut writer = std.archive.tar_writer(std.io.cursor(std.bytes.empty()?))?
writer.add_file(&std.path.from_text("site/index.html")?, &std.bytes.from_text("<h1>hi</h1>")?, 420, 0)?
writer.add_symlink(&std.path.from_text("site/secret")?, &std.path.from_text("/etc/passwd")?, 0)?
writer.finish()?
packed = writer.inner.data.copy()?
cancel = CancellationToken()?
mut policy = std.archive.policy()
mut first = std.archive.tar_reader(std.io.cursor(packed.copy()?))?
match std.archive.extract_tar(&mut first, scratch.root(), &std.path.from_text("out")?, &policy, &cancel):
case Ok(_):
print("extracted")?
case Err(error):
print(error)?
policy.skip_links = True
mut second = std.archive.tar_reader(std.io.cursor(packed))?
summary = std.archive.extract_tar(&mut second, scratch.root(), &std.path.from_text("out")?, &policy, &cancel)?
print(summary.files, summary.bytes, summary.skipped)?
std.fs.remove_tree(scratch.root(), &std.path.from_text("out")?, 10, 4, &cancel)?
scratch.cleanup()?
Ok(())
std.archive.ArchiveError.Link
1 11 1
The first attempt fails on the link and leaves nothing behind. The second skips it. Extracted files get private permissions (0600); stored modes and times are not applied.
The default policy allows 10000 entries, 1 GiB per file, 4 GiB in total, and a compression ratio of 100 for large ZIP entries. Lower the limits to what your program expects.
See the std.archive reference for the
format rules, every error, and memory costs.
Build a package with dependencies
A package is a directory with a plenty.toml manifest and sources under
src/. plenty new hello creates one; plenty new shapes --lib creates a
library. Here is a library package in shapes/:
manifest-version = 1
[package]
name = "shapes"
version = "0.1.0"
src/lib.plenty is the library: the module that other packages import.
pub class Square:
pub side: i64
pub def area(self) -> Result[i64, OverflowError[i64]]:
Ok(self.side * self.side)
An application beside it declares the library as a dependency. The key under
[dependencies] is the name that application code imports, and path is
relative to this manifest:
manifest-version = 1
[package]
name = "app"
version = "0.1.0"
[dependencies]
geometry = { path = "../shapes" }
src/main.plenty is the executable, named after the package:
import geometry
from report import show
def main() -> Result[(), AllocError | IoError | OverflowError[i64]]:
square = geometry.Square(4)
show(square.area()?)
Other files under src/ are the package’s own modules, imported as before:
pub def show(area: i64) -> Result[(), IoError]:
print("area:")?
print(area)?
Ok(())
Run plenty run anywhere inside app/:
plenty run
area:
16
plenty run compiles into target/optimized/app and runs it. Progress messages go
to standard error; program arguments follow --, as in plenty run -- a b.
Builds optimize by default. For shorter edit-run cycles in large projects, use
plenty run --fast-compile or plenty build --fast-compile; these write to
target/fast-compile/ and keep the same overflow, bounds and ownership checks.
Use the default profile when measuring runtime performance.
A later build with no changed source reports Fresh and reuses the executable.
The first build also writes plenty.lock, which records the resolved packages.
Commit it with the manifest. --locked makes a build fail instead of changing
it.
A dependency can also come from a Git repository, at a tag or a commit. The official time-zone package is one:
[dependencies]
tz = { git = "https://github.com/plentylang/tz.git", tag = "v0.3.0", version = "0.3.0" }
The first build fetches the repository into .pkg/ and records the exact
commit in plenty.lock. Later builds use that commit even if the tag moves
upstream, until plenty update resolves it again. The optional version
checks that the dependency’s own version is compatible. plenty add writes
these entries and updates the lockfile:
plenty add geometry --path ../shapes
plenty add tz --git https://github.com/plentylang/tz.git --tag v0.3.0
Several related packages can share one lockfile and target/ as a
workspace.
A dependency’s name belongs only to the package that declares it. app cannot
import what shapes depends on without declaring it too, and a local module
named geometry would be an error. Two packages may each define Square;
those are distinct types.
The dependency’s name also reaches its other modules. If shapes had
src/circles.plenty, import geometry.circles would bind it beside
import geometry, unless the library itself declares something named
circles; modules and imports
gives the rule.
Programs in tests/ are integration tests. Each one sees the package’s library
under the package name, and passes when main returns exit status zero:
from shapes import Square
def main() -> Result[i32, OverflowError[i64]]:
if Square(3).area()? == 9:
Ok(0i32)
else:
Ok(1i32)
plenty test
running 1 test
test area ... ok
test result: ok. 1 passed; 0 failed
A later lesson shows test functions and std.test.
plenty check type-checks every target without linking. It reaches only the
files that targets import, so it warns about any other .plenty file under
src/ and ends with a summary such as
Checked app v0.1.0: 1 executable; 2 of 2 files in src/. Keep test fixtures
and other files that are not package source outside src/. plenty clean
removes target/. Single files still run with plenty FILE; the compiler is
also available alone as plentyc.
Match text with regular expressions
std.regex compiles a pattern once into a Regex, then searches text with
it. Matching takes time proportional to the length of the text, whatever the
pattern: there is no backtracking, so no pattern can make a search run for
hours. compile takes Limits, which also bound the pattern itself, so a
pattern from an untrusted source is safe to compile.
Keep the compiled Regex when searching repeatedly. The library selects
prefix filters and faster matching engines
where the pattern permits them; captures, match ordering and resource limits
keep the same meaning.
Find matches
find returns the first match as byte offsets; text copies the matched
part. find_all returns every match and fails rather than return more than
its limit:
import std.regex
def main() -> Result[(), AllocError | IoError | std.regex.RegexError]:
limits = std.regex.limits()
year = std.regex.compile("\\b(19|20)\\d\\d\\b", &limits)?
text = "Released 1999, revised 2024, not 12345."
match year.find(text)?:
case Some(found):
print(found.start, found.end, found.text(text)?)?
case Nothing:
print("no year")?
print(len(year.find_all(text, 10)?))?
print(year.is_match("in 1850")?)?
print(str.repr(year.find_all(text, 1))?)?
Ok(())
9 13 1999
2
False
Result[list[std.regex.Match], std.regex.RegexError].Err(std.regex.RegexError.LimitExceeded(std.regex.Limit.Matches))
A Plenty string needs \\ for each backslash in the pattern. \d, \w,
\s and \b are ASCII-only; for other scripts, use properties such as
\p{Alphabetic}.
Groups and replacement
Parentheses capture. (?P<name>...) names a group. captures returns every
group of the first match; a group that did not take part is Nothing. In a
replacement template, $1 and ${name} insert a group and $$ inserts a
dollar sign:
import std.regex
def main() -> Result[(), AllocError | IoError | std.regex.RegexError]:
limits = std.regex.limits()
date = std.regex.compile("(?P<year>\\d{4})-(?P<month>\\d{2})", &limits)?
text = "from 2024-05 to 1999-12"
match date.captures(text)?:
case Some(groups):
match groups.name("year"):
case Some(group):
print(group.text(text)?)?
case Nothing:
print("group did not match")?
match groups.get(2):
case Some(group):
print(group.text(text)?)?
case Nothing:
print("group did not match")?
case Nothing:
pass
print(date.replace_all(text, "${month}/$1", 100)?)?
words = std.regex.compile("\\s*,\\s*", &limits)?
print(" | ".join(words.split("red , green,blue", 10)?)?)?
Ok(())
2024
05
from 05/2024 to 12/1999
red | green | blue
The last argument of replace_all and split is a limit on the result size
or the number of pieces. Write ${1}0 when a group number is followed by a
digit. A bare $name is an error: use ${name}.
Errors and limits
A pattern error reports its byte offset. A pattern that would build too large a matcher, and a search that would do too much work, are limit errors:
import std.regex
from std.regex import Limits
def main() -> Result[(), AllocError | IoError | std.regex.RegexError]:
limits = std.regex.limits()
print(str.repr(std.regex.compile("total: (\\d+", &limits))?)?
print(str.repr(std.regex.compile("(a{1000}){1000}", &limits))?)?
tight = Limits(1000, 16, 100, 1000, 1000000, 100)
word = std.regex.compile("\\w+!", &tight)?
print(str.repr(word.find("no exclamation mark in this sentence"))?)?
Ok(())
Result[std.regex.Regex, std.regex.RegexError].Err(std.regex.RegexError.Syntax(std.regex.SyntaxError(offset=7, message="unclosed group")))
Result[std.regex.Regex, std.regex.RegexError].Err(std.regex.RegexError.LimitExceeded(std.regex.Limit.Instructions))
Result[Option[std.regex.Match], std.regex.RegexError].Err(std.regex.RegexError.LimitExceeded(std.regex.Limit.Work))
The default limits suit patterns written into a program. For patterns from users, lower them to what a request may spend.
See the std.regex reference for the full
syntax, every limit, and searching bytes.
Write tests with std.test
plenty test runs a package’s tests. A file in tests/ holds either one
program with main, as in the packages lesson, or any number
of test_ functions. Each test function runs in its own process, so a trap in
one fails only that test.
Here is a small library to test:
manifest-version = 1
[package]
name = "stats"
version = "0.1.0"
from std.bytes import from_text
from std.io import Writer, Partial, write_all
from std.bytes import BytesError
pub def total(values: &list[i64]) -> Result[i64, OverflowError[i64]]:
mut sum = 0
for value in values:
sum += *value
Ok(sum)
pub def mean(values: &list[i64]) -> Result[f64, OverflowError[i64]]:
Ok(f64(total(values)?) / f64(len(values)))
pub def save(path: &str, values: &list[i64]) -> Result[(), AllocError | IoError | OverflowError[i64]]:
write_text(path, &str.from(total(values)?)?)?
Ok(())
pub def report[W: Writer](out: &mut W, values: &list[i64]) -> Result[(), AllocError | BytesError | Partial | OverflowError[i64]]:
line = from_text(&str.from(total(values)?)?)?
write_all(out, &line)?
Ok(())
A test function takes no parameters and returns () or Result[(), E].
std.test supplies assertions that return Result[(), TestError], so ?
stops the test at the first failure:
import std.bytes
import std.test
import std.io
from stats import mean, report, save, total
def test_total() -> Result[(), AllocError | IoError | std.test.TestError | OverflowError[i64]]:
values = [1, 2, 3]?
std.test.equal(&total(&values)?, &6)?
empty: list[i64] = []?
std.test.equal(&total(&empty)?, &0)?
Ok(())
def test_mean() -> Result[(), AllocError | IoError | std.test.TestError | OverflowError[i64]]:
values = [1, 2]?
std.test.close(mean(&values)?, 1.5, 0.000001)?
Ok(())
def test_save() -> Result[(), AllocError | IoError | std.test.TestError | OverflowError[i64]]:
scratch = std.test.scratch()?
path = scratch.file("total.txt")?
values = [4, 5]?
save(&path, &values)?
std.test.equal(&read_text(&path)?, &"9")?
Ok(())
def test_order_does_not_matter() -> Result[(), AllocError | IoError | std.test.TestError | OverflowError[i64]]:
mut rng = std.test.rng()?
mut values = [1, 2, 3, 4, 5]?
rng.shuffle(&mut values)
std.test.equal(&total(&values)?, &15)?
Ok(())
def test_report_survives_short_writes() -> Result[(), AllocError | std.bytes.BytesError | std.io.Partial | std.test.TestError | OverflowError[i64]]:
mut out = std.test.fake_stream(std.bytes.empty()?)?
out.limit(1)
values = [9, 10]?
report(&mut out, &values)?
std.test.equal(&out.output.to_text()?, &"19")?
Ok(())
def test_save_reports_a_full_disk() -> Result[(), AllocError | IoError | std.test.TestError | OverflowError[i64]]:
scratch = std.test.scratch()?
path = scratch.file("total.txt")?
values = [4, 5]?
std.test.fail_files_after(0u32, 28i32)?
saved = save(&path, &values)
std.test.restore_files()?
std.test.is_err(&saved)?
Ok(())
std.test.scratch() makes an empty directory under $TMPDIR that is removed
with everything in it when the Scratch value is dropped. std.test.rng() is
a generator seeded from plenty test --seed N, or from 0, so a run repeats
exactly.
report takes any std.io Writer, so the test hands it a fake stream that
accepts one byte per call. A file rarely takes a write in pieces, but a pipe or
socket does; the fake makes that case happen every time. fail_after makes a
chosen call fail, and std.test.fake_clock stands in for the clock in the same
way.
A real file fails rarely, so test_save_reports_a_full_disk makes it fail:
after fail_files_after(0u32, 28i32), every file call fails with error 28, the
system’s code for a full disk, until restore_files. fail_allocations_after
does the same for memory. plenty test enables these hooks automatically.
For an integration test that launches the ordinary application, explicitly use
plenty build --test-hooks and arm the budget at the operation being tested.
Default builds contain no hooks. See
failure injection and memory-limited runs.
plenty test
running 6 tests
test sums::test_total ... ok
test sums::test_mean ... ok
test sums::test_save ... ok
test sums::test_order_does_not_matter ... ok
test sums::test_report_survives_short_writes ... ok
test sums::test_save_reports_a_full_disk ... ok
test result: ok. 6 passed; 0 failed
Tests run in parallel, one per CPU, but are listed in the order they appear.
Each test is named FILE::FUNCTION. plenty test mean runs only the tests
whose names contain mean, and plenty test --list prints the names without
running anything.
A test’s output is kept and shown only if it fails. If total added one too
many, plenty test test_total would print:
running 1 test
test sums::test_total ... FAILED (returned Err)
failures:
---- sums::test_total output ----
sums.plenty:7:5: assertion failed: left == right
left: 7
right: 6
test result: FAILED. 0 passed; 1 failed; 5 filtered out
The command exits with status 1.
The assertion writes where it was called and what it found as it fails, and
returned Err says the test stopped there. A trap would read exit status 1 and show the trap’s
message. plenty test --no-capture lets output through as it happens, and
--timeout 10 fails any test still running after ten seconds.
A file defines either main or test_ functions, not both. Helpers that are
not tests need a name without the test_ prefix, or a module in a subdirectory
of tests/. See std.test for every assertion.
Compare text with std.diff
std.diff finds what changed between two versions of a text and reports it as
edit operations: runs that stayed equal, were deleted, or were inserted.
Show a unified diff
std.diff.lines compares line by line. unified formats the result the way
diff -u does, with the given number of unchanged lines around each change:
import std.diff
def main() -> Result[(), AllocError | IoError]:
old = "name = \"stats\"\nversion = \"0.1.0\"\nauthors = []\nlicense = \"MIT\"\n"
new = "name = \"stats\"\nversion = \"0.2.0\"\nauthors = []\nlicense = \"MIT\"\nreadme = \"README.md\"\n"
changes = std.diff.lines(old, new)?
print(changes.has_changes())?
print(changes.unified(1, "plenty.toml", "plenty.toml (edited)")?)?
Ok(())
True
--- plenty.toml
+++ plenty.toml (edited)
@@ -1,4 +1,5 @@
name = "stats"
-version = "0.1.0"
+version = "0.2.0"
authors = []
license = "MIT"
+readme = "README.md"
A hunk header @@ -1,4 +1,5 @@ says that four lines starting at line 1 of
the old text became five lines starting at line 1 of the new one. Equal texts
give empty text, so check has_changes first when that matters.
Walk the operations
words splits text into runs of whitespace and runs of other characters, and
chars into single characters. Each Op gives a range of old tokens and a
range of new tokens; a deletion has an empty new range and an insertion an
empty old range:
import std.diff
from std.diff import Kind
def main() -> Result[(), AllocError | IoError]:
changes = std.diff.words("the quick brown fox", "the quick red fox jumps")?
old = changes.old_tokens()
new = changes.new_tokens()
for op in changes.ops():
match op.kind:
case Kind.Equal:
pass
case Kind.Delete:
for index in range(op.old_start, op.old_end):
print("removed", str.repr(old[index])?)?
case Kind.Insert:
for index in range(op.new_start, op.new_end):
print("added", str.repr(new[index])?)?
Ok(())
removed "brown"
added "red"
added " "
added "jumps"
ops returns a reference, so bind the TextDiff to a name before calling it.
std.diff.diff compares any two lists whose elements can be dictionary keys,
and std.diff.diff_bytes compares Bytes.
The default algorithm finds the fewest deletions and insertions. Passing
Algorithm.Patience as the last argument anchors the comparison on lines that
occur once in each text, which often reads better for source code.
Finding the fewest edits takes time quadratic in the input size when two large
inputs have little in common. For large or untrusted input, pass a cost limit
such as std.diff.lines(old, new, max_cost=Some(512)): the work stays roughly
linear and the operations still turn the old text into the new one, but may
include more deletions and insertions than necessary. See the
std.diff reference for both algorithms, the
cost limit, and their costs.
Render Markdown as HTML
std.markdown reads CommonMark, the common core of
Markdown, and writes HTML. Between the two it offers a stream of events, so a
program can inspect or change a document before it is rendered.
Convert a document
to_html renders a whole document. Like other parsers in the standard
library, it takes limits on input size, nesting depth, and output size:
import std.markdown
from std.markdown import MarkdownError
def main() -> Result[(), AllocError | IoError | MarkdownError]:
bounds = std.markdown.limits()
text = "# Shopping\n\n- *fresh* bread\n- [cheese](https://example.com/cheese)\n"
print(std.markdown.to_html(text, &bounds)?)?
Ok(())
<h1>Shopping</h1>
<ul>
<li><em>fresh</em> bread</li>
<li><a href="https://example.com/cheese">cheese</a></li>
</ul>
The HTML ends with a line ending, so print leaves a blank line after it.
Walk the events
A parser yields Start and End events around each block and inline element,
with Text and other events between them. This program prints the text of
every second-level heading:
import std.markdown
from std.markdown import Event, MarkdownError, Tag
def main() -> Result[(), AllocError | IoError | MarkdownError]:
bounds = std.markdown.limits()
text = "# Plenty\n\n## Install\n\nRun `plenty FILE`.\n\n## Learn\n\nRead the *tutorial*.\n"
mut events = std.markdown.parser_text(text, &bounds)?
mut in_section = False
while True:
event = events.next()?
match event:
case Event.Start(tag):
match tag:
case Tag.Heading(level):
in_section = level == 2
case _:
pass
case Event.End(_):
in_section = False
case Event.Text(words):
if in_section:
print("-", words)?
case Event.Done:
break
case _:
pass
Ok(())
- Install
- Learn
Text holds the characters to show, with escapes and entities such as &
already resolved. The last event is Done.
Change events before rendering
Markdown lets raw HTML through unchanged. When the text comes from someone you do not trust, render the events yourself and turn raw HTML into text, which the renderer escapes:
import std.bytes
import std.markdown
from std.markdown import Event, MarkdownError
def main() -> Result[(), AllocError | IoError | std.bytes.BytesError | MarkdownError]:
bounds = std.markdown.limits()
text = "Hi <b onclick=\"steal()\">there</b>!\n\n<script>steal()</script>\n"
mut events = std.markdown.parser_text(text, &bounds)?
mut html = std.markdown.renderer(&bounds)?
while True:
event = events.next()?
match event:
case Event.Html(raw):
html.event(&Event.Text(raw))?
case Event.InlineHtml(raw):
html.event(&Event.Text(raw))?
case Event.Done:
break
case _ as other:
html.event(&other)?
print(html.take()?.to_text()?)?
Ok(())
<p>Hi <b onclick="steal()">there</b>!</p>
<script>steal()</script>
Link destinations are not checked either: [x](javascript:steal()) is a valid
link. A program that publishes untrusted Markdown should also inspect
Tag.Link and Tag.Image destinations.
std.markdown.write_html renders a parser’s events straight to a writer, such
as a file, without holding the whole page. See the
std.markdown reference for the events,
limits, errors, and exact HTML output.
Useful programs and resources
Read arguments and text, format results, and use standard streams and files. Combine these operations into a command-line application.
Command-line arguments
args() returns Result[list[str], IoError]. Index zero is the executable’s
invocation name, followed by the supplied arguments. Pass arguments to a compiled
binary, for example ./program first "two words"; quoted spaces remain within
one argument. Each call creates an independent list and reports allocation or
UTF-8 errors.
def count_arguments() -> Result[bool, IoError]:
values = args()?
Ok(len(values) >= 1)
def main() -> Result[(), AllocError | IoError]:
print(count_arguments()?)?
Ok(())
True
Parse typed arguments
Use std.cli when a tool needs options, positionals, generated help, and useful
usage errors. The schema explicitly chooses each argument’s type and whether it
is required or repeatable. Path values retain native bytes.
This small input-selection tool accepts a row limit and any number of paths.
The fixture models finder --limit -2 first.csv -- --last.csv. Real programs
use raw = args_bytes()? and parse(&schema, &raw, 1) to skip the executable name.
import std.cli
from std.cli import Spec, Kind, Value, parse
def main() -> Result[(), AllocError | IoError | std.cli.CliError]:
schema = [
Spec("--limit", Kind.Integer, False, False, False, "maximum rows"),
Spec("FILE", Kind.Path, True, True, True, "input files")
]?
raw = [encode_utf8("--limit")?, encode_utf8("-2")?,
encode_utf8("first.csv")?, encode_utf8("--")?,
encode_utf8("--last.csv")?]?
parsed = parse(&schema, &raw, 0)?
for entry in &parsed.entries:
match &entry.value:
case Value.Integer(limit):
print("limit:", limit)?
case Value.Path(path):
print("file:", path.to_string()?)?
case _:
pass
Ok(())
limit: -2
file: first.csv
file: --last.csv
Long options use separate values: --limit 12. A value following an option is
always consumed, even if it starts with -. The standalone -- ends option
recognition; subsequent tokens are positional. Missing required arguments,
unknown options, duplicate nonrepeatable options, and invalid integers are typed
errors. Optional specs produce no entry when absent, leaving defaults to you.
Handle help before running the tool. It is a successful result, not an error.
The following complete program uses a synthetic request so the lesson can run
without shell arguments. Replace raw and the start index as above in a binary.
from std.cli import Spec, Kind, parse, help, describe_for, exit_status
from std.error import Context, Subject
def run() -> Result[i32, AllocError | IoError]:
schema = [Spec("FILE", Kind.Path, True, True, False, "input file")]?
raw = [encode_utf8("--help")?]?
status = match parse(&schema, &raw, 0):
case Ok(parsed):
if parsed.help_requested:
match help(&schema, "finder"):
case Ok(text):
write_stdout(text)?
case Err(error):
Context("help", Subject.Absent, error).report()
return Ok(1i32)
0i32
case Err(error):
status = exit_status(&error)
context = Context("arguments", Subject.Absent, error)
context.report_with(describe_for(&context.cause, &schema))
status
Ok(status)
def main() -> i32:
match run():
case Ok(status):
status
case Err(error):
Context("argument setup", Subject.Absent, error).report()
1i32
Usage: finder [--help]
FILE <path> (required) input file
--help Show this help
--help bypasses missing-required checks, but an earlier bad argument still
fails. After --, or as the next token for a text/path option, --help is data.
The parser validates the schema before processing either form.
The run helper propagates setup and output failures. main reports
those failures and returns a nonzero exit status, just as it does for argument
and help errors.
std.cli lists all errors, costs, and grammar limits.
Show progress
Use std.cli.progress when a command does enough work that users need feedback.
Advance a bar after each successful unit of work and finish it explicitly.
The bar uses stderr only when it is a terminal, so redirected output remains
clean. The library limits redraws; an update on every item does not redraw on
every item.
import std.cli.progress as progress
from std.cli.progress import ProgressError
def main() -> Result[(), ProgressError | IoError | OverflowError[i64]]:
mut bar = progress.bar(5u64)
mut total = 0
for value in range(1, 6):
total += value * value
bar.advance()?
bar.finish()?
print("sum of squares:", total)?
Ok(())
sum of squares: 55
The calculation is deliberately short to keep the example runnable; a real
program might count imported records or processed files. For an unknown total,
use progress.spinner(): advance() counts completed units, and tick() requests
an animation step without changing the count. There is no background thread, so
a spinner only moves when your program updates it.
Propagate display errors as above unless your application deliberately treats
progress as optional. An early failure does not automatically finish the display;
finish before returning or writing diagnostics when a clean terminal line is
needed. Avoid mixing another progress bar or concurrent stderr logging with the
same display. The initial renderer assumes an ANSI terminal at least 80 columns
wide and uses compact approximate counts; bar.position() gives the exact count.
The reference describes rates, ETA, deterministic clock inputs and plain-text rendering.
Reading integers from text
Use a sized type’s parse method for external text. It borrows the string and
returns an allocation-free Result. Decimal ASCII digits and an optional sign
are accepted, with surrounding Unicode whitespace ignored. Numeric prefixes,
underscores, and non-ASCII digits are not accepted.
def main() -> Result[(), AllocError | IoError]:
print(str.repr(u8.parse(" 255 "))?)?
print(str.repr(u8.parse("256"))?)?
print(str.repr(i64.parse("hello"))?)?
Ok(())
Result[u8, ParseError].Ok(255)
Result[u8, ParseError].Err(ParseError.OutOfRange)
Result[i64, ParseError].Err(ParseError.Invalid)
Functions returning the same error type can propagate parsing failures with ?.
try_from reports a value that does not
fit with RangeError.OutOfRange. Match it explicitly to combine parsing,
calculation, and narrowing in a function returning ParseError:
def shade(percent: &str) -> Result[u8, ParseError]:
number = i64.parse(percent)?
scaled_result = checked:
number * 255
product = match scaled_result:
case Ok(value):
value
case Err(_):
return Err(ParseError.OutOfRange)
hundred: Positive[i64] = 100
scaled = product // hundred
match u8.try_from(scaled):
case Ok(value):
Ok(value)
case Err(error):
match error:
case RangeError.OutOfRange:
Err(ParseError.OutOfRange)
def main() -> Result[(), AllocError | IoError]:
print(str.repr(shade("50"))?)?
print(str.repr(shade("150"))?)?
print(str.repr(shade("half"))?)?
Ok(())
Result[u8, ParseError].Ok(127)
Result[u8, ParseError].Err(ParseError.OutOfRange)
Result[u8, ParseError].Err(ParseError.Invalid)
The local checked multiplication maps overflow to ParseError.OutOfRange,
keeping this conversion helper’s public error contract. The positive divisor
proves division cannot fail. No unchecked arithmetic or lossy cast is needed.
Reading floating-point text
Floating-point parsing accepts decimal/exponent forms and surrounding whitespace.
Overflow returns OutOfRange; underflow can round to zero. Explicit inf,
infinity, and nan are accepted case-insensitively, with optional signs.
def main() -> Result[(), AllocError | IoError]:
print(str.repr(f32.parse("1.25e2"))?)?
print(str.repr(f32.parse("1e100"))?)?
print(str.repr(f64.parse("-0.0"))?)?
Ok(())
Result[f32, ParseError].Ok(125.0)
Result[f32, ParseError].Err(ParseError.OutOfRange)
Result[f64, ParseError].Ok(-0.0)
Convert and format values
str.from converts numbers or booleans to text, returning
Result[str, AllocError]. Floats use the shortest text that parses back to the
same value.
def main() -> Result[(), AllocError | IoError]:
print(str.from(42)?)?
print(str.from(-0.0f32)?)?
print(str.from(True)?)?
Ok(())
42
-0.0
True
Inspect a value with str.repr
str.repr formats a borrowed value; print writes a value and newline,
reporting formatting allocation failures and output errors:
def main() -> Result[(), AllocError | IoError]:
values = [1, 2]?
print(str.repr(values)?)?
print(values)?
Ok(())
[1, 2]
[1, 2]
Formatting failure writes nothing. An output error can leave a partial write. The source remains usable; neither call copies a mutable collection.
The inner ? handles formatting; the outer ? handles printing. To inspect a
Result wrapper itself, pass it to str.repr:
def main() -> Result[(), AllocError | IoError]:
result: Result[i64, str] = Err("not ready")
print(str.repr(result)?)?
Ok(())
Result[i64, str].Err("not ready")
Split text into fields
Use text.split(separator) to split on an explicit, nonempty string. It
returns Result[list[str], AllocError]:
def main() -> Result[(), AllocError | IoError]:
line = "Ada::Bea::::"
parts = line.split("::")?
print(parts)?
print(str.repr(" / ".join(parts)?)?)?
print(line)?
print("".split(",")?)?
Ok(())
["Ada", "Bea", "", ""]
"Ada / Bea / / "
Ada::Bea::::
[""]
Adjacent and trailing separators preserve empty fields. Matches do not overlap:
"aaaaa".split("aa") succeeds with ["", "", "a"]. A separator that does not
occur produces a single piece containing the whole input.
Each piece owns independent storage. Splitting borrows both inputs and leaves them unchanged, including on allocation failure.
This program cannot produce its fields if splitting fails, so ? returns the
error from main. The same policy handles failures from joining, formatting,
and printing. Use a match instead when you have a useful recovery action.
An empty separator causes a runtime trap, not AllocError. Check
len(separator) > 0 for user-supplied separators. Whitespace splitting without
a separator and a maximum-split argument are unsupported.
Splitting text that is already in memory
splitlines() returns an independent list without line endings. Pass
True to keep the original endings. A final newline does not add an extra
empty line, and empty text produces an empty list. Unlike file reads, this method
also recognizes Unicode line/paragraph separators and the other Python-style
text line boundaries.
def main() -> Result[(), AllocError | IoError]:
print("first\r\n\nlast\n".splitlines()?)?
print("first\r\nlast".splitlines(True)?)?
print("".splitlines()?)?
Ok(())
["first", "", "last"]
["first\r\n", "last"]
[]
Writing text with recoverable errors
write_stdout(text) borrows a string and writes exactly its contents, without a
newline. Success returns the number of Unicode characters. An I/O error can
follow a partial write, so retrying the whole string can duplicate output.
def main() -> Result[(), AllocError | IoError]:
result = write_stdout("Hello!\n")
print(result?)?
Ok(())
Hello!
7
IoError.System(code) carries a native OS error code. Decoding, allocation, and
stream-state errors have separate variants; the
text I/O reference
lists them all.
Flushing output and reporting diagnostics
Use write_stderr(text) for diagnostics. It returns the same character-count
result as write_stdout. flush_stdout() and flush_stderr() return
Result[(), IoError]. Flush when the caller needs to observe buffered-write
errors or when displaying a prompt; a flush does not promise disk durability.
def report() -> Result[(), IoError]:
write_stdout("ready")?
flush_stdout()?
Ok(())
def main() -> Result[(), IoError]:
report()?
print("")?
Ok(())
ready
Reading a line
input() returns Result[Option[str], IoError]. It removes a trailing LF or
CRLF. Some("") means an empty line; Nothing means end-of-file. UTF-8 errors
and allocation failures return Err. A failure can consume input, so retrying
does not promise to repeat the same line. Input currently requires a Unix host.
For a prompt, call write_stdout and flush_stdout before input().
Call echo_line()? repeatedly until it returns False to process stdin. This
example’s main does not read input.
def echo_line() -> Result[bool, IoError]:
match input()?:
case Some(line):
write_stdout(line)?
write_stdout("\n")?
Ok(True)
case Nothing:
Ok(False)
def main() -> Result[(), IoError]:
print("echo_line is ready")?
Ok(())
echo_line is ready
Reading a UTF-8 file
read_text(path) opens, reads, and closes a file, returning
Result[str, IoError]. It requires Linux and strict UTF-8, translating CRLF and
bare CR to LF. Paths are relative to the process working directory. The handle
closes even when reading or allocating fails.
The &str parameter borrows literals and bindings implicitly;
configuration(&path) is also valid.
def configuration(path: &str) -> Result[str, IoError]:
read_text(path)
def main() -> Result[(), IoError]:
write_text("configuration.txt", "mode=fast\n")?
print(configuration("configuration.txt")?)?
path = "configuration.txt"
print(configuration(path)?)?
Ok(())
mode=fast
mode=fast
Writing a UTF-8 file
write_text(path, text) creates or replaces a file and returns the number of
Unicode characters written. It writes exact UTF-8 bytes without translating
newlines and closes the handle before returning. OS failures can leave truncated
or partially written contents; this is not an atomic save.
Run this in a scratch directory: it replaces plenty-example.txt.
def save_and_read() -> Result[str, IoError]:
write_text("plenty-example.txt", "Hello, é!\n")?
read_text("plenty-example.txt")
def main() -> Result[(), AllocError | IoError]:
print(str.repr(save_and_read())?)?
Ok(())
Result[str, IoError].Ok("Hello, é!\n")
Appending text
append_text(path, text) adds exact UTF-8 bytes to the end of a file, creating
it if necessary. It returns the number of characters appended. Errors can leave
partial output; concurrent appends are not guaranteed to be atomic records.
Run this in a scratch directory: it replaces plenty-log.txt before appending.
def log_example() -> Result[str, IoError]:
write_text("plenty-log.txt", "started\n")?
append_text("plenty-log.txt", "finished\n")?
read_text("plenty-log.txt")
def main() -> Result[(), AllocError | IoError]:
print(str.repr(log_example())?)?
Ok(())
Result[str, IoError].Ok("started\nfinished\n")
Open a file for a block
with open(path)? as file makes a stream available inside the indented block.
It closes on exit, including on return, ?, break, or continue. The stream
reference cannot escape the block.
open(path) opens an existing file for reading. open(path, "w") creates or
truncates a file, and open(path, "a") creates or appends. Each returns
Result[File, IoError].
Files move on assignment and cannot be copied.
read() returns text from the current position through EOF, translating CRLF
and bare CR to LF and validating UTF-8. At EOF it returns an empty string.
Run this in a scratch directory: it replaces plenty-reading.txt.
def read_example() -> Result[str, IoError]:
write_text("plenty-reading.txt", "hello\r\nworld")?
with open("plenty-reading.txt")? as file:
return Ok(file.read()?)
def main() -> Result[(), IoError]:
print(read_example()?)?
Ok(())
hello
world
The returned text owns its storage and survives closing the file. Read errors may consume input before failing. File I/O requires Linux.
Reading lines
readline() keeps the terminating newline, translating LF, CRLF, and bare CR
to "\n". An empty line is "\n"; EOF is "". The last line need not have a
terminating newline. Unlike input(), line reading retains that terminator.
Run this in a scratch directory: it replaces plenty-lines.txt.
def count_lines() -> Result[i64, IoError | OverflowError[i64]]:
write_text("plenty-lines.txt", "first\r\n\rfinal")?
mut count = 0
with open("plenty-lines.txt")? as file:
while True:
line = file.readline()?
if line == "":
break
count = count + 1
Ok(count)
def main() -> Result[(), AllocError | IoError | OverflowError[i64]]:
print(count_lines()?)?
Ok(())
3
readline() and read() share newline state. Their errors may consume input;
retrying is not a rollback. Direct for line in file iteration is unsupported.
For size-limited reads and saved positions, see the
text I/O reference.
Write through a file
Use a writable stream when several writes belong to one open file. write(text)
returns the number of Unicode characters written and preserves exact UTF-8 bytes,
including newlines. Mode "w" replaces a file; "a" appends to it.
Run this in a scratch directory: it replaces plenty-stream.txt.
def main() -> Result[(), IoError]:
with open("plenty-stream.txt", "w")? as file:
print(file.write("hello\n")?)?
file.write("world\n")?
file.close()?
with open("plenty-stream.txt", "a")? as file:
file.write("goodbye\n")?
print(read_text("plenty-stream.txt")?)?
Ok(())
6
hello
world
goodbye
Call close()? to report a close error; automatic cleanup cannot report it.
Exiting the block can then close the already closed file harmlessly.
For flushing, disk synchronization, and other detailed contracts, see the text I/O reference.
Context managers
Define __enter__ and __exit__ to make a class usable with with. Entry
supplies the as value; exit takes only a mutable receiver and returns ().
No inheritance or protocol declaration is needed.
class Message:
text: str
def __enter__(self: &mut Message) -> str:
drop(print("enter"))
self.text
def __exit__(self: &mut Message) -> ():
drop(print("exit"))
def main() -> Result[(), IoError]:
with Message("hello") as text:
print(text)?
print("after")?
Ok(())
enter
hello
exit
after
The manager moves into the block. Body locals and the entry value clean up before
__exit__; the manager drops afterward, including on early exits. An owned entry
value can move out, for example through return. The as binding is local to the
body; omit it for unit entry methods.
with acquire()?: propagates acquisition failure without entering the context.
The lifecycle trace uses drop(print(...)) deliberately: cleanup cannot return
an error, and a failed diagnostic write must not interrupt cleanup. For output
that must succeed, provide an explicit fallible operation and propagate its
result before leaving the scope.
Borrow and combine context managers
To keep a manager after the block, borrow it explicitly. The original binding is exclusively borrowed until exit completes.
class Stage:
marker: i64
def __enter__(self: &mut Stage) -> i64:
self.marker = 1
self.marker
def __exit__(self: &mut Stage) -> ():
self.marker = 11
def main() -> Result[(), IoError]:
mut stage = Stage(0)
with &mut stage as n:
print(n)?
print(stage.marker)?
Ok(())
1
11
Stage records lifecycle markers, not an accumulating counter: enter assigns
1 and exit assigns 11, including on repeated uses. Cleanup performs no
fallible arithmetic.
Several managers can share one with statement. They enter from left to right
and exit from right to left, just like nested blocks. Later manager expressions
can use earlier as bindings.
class Label:
text: str
def __enter__(self: &mut Label) -> str:
drop(print(self.text))
self.text
def __exit__(self: &mut Label) -> ():
drop(print(self.text))
def main() -> Result[(), IoError]:
with Label("first") as first, Label("second") as second:
print("body")?
Ok(())
first
second
body
second
first
Exit is infallible and cannot suppress errors. Check fallible writes or flushes
explicitly. Fatal traps do not run exits, and yield inside with is not yet
supported.
__enter__ can return a reference to the manager, giving the body access to its
methods and fields while with keeps ownership. The reference cannot escape.
class Counter:
count: i64
def __enter__(self: &mut Counter) -> &mut Counter:
&mut self
def __exit__(self: &mut Counter) -> ():
drop(print(self.count))
def main() -> Result[(), IoError]:
with Counter(1) as counter:
counter.count = 7
Ok(())
7
The lifecycle trace uses drop(print(...)) deliberately: cleanup cannot return
an error, and a failed diagnostic write must not interrupt cleanup. For output
that must succeed, provide an explicit fallible operation and propagate its
result before leaving the scope.
Build a small command-line application
Put one integer on each line of an input file. This program ignores blank lines, adds the integers, and writes a short report to an output file. Its two arguments are the input and output paths. Running it without arguments prints usage.
Test for EOF before stripping whitespace: a blank line should be ignored, not mistaken for the end of the file.
Save this module as report.plenty:
pub def summarize(path: &str) -> Result[str, AllocError | IoError | ParseError | OverflowError[i64]]:
mut count = 0
mut total = 0
with open(path)? as file:
while True:
line = file.readline()?
if line == "":
break
cleaned = line.strip()?
if cleaned == "":
continue
total = total + i64.parse(cleaned)?
count = count + 1
Ok(f"count={count} total={total}\n"?)
Save this application beside it as main.plenty:
from report import summarize
enum UsageError:
Arguments
type AppError = AllocError | IoError | ParseError | OverflowError[i64] | UsageError
def run() -> Result[(), AppError]:
arguments = args()?
if len(arguments) == 1:
print("usage: number-report INPUT OUTPUT")?
return Ok(())
if len(arguments) != 3:
write_stderr("usage: number-report INPUT OUTPUT\n")?
return Err(UsageError.Arguments)
report = summarize(arguments[1])?
write_text(arguments[2], report)?
print("saved report")?
Ok(())
def main() -> Result[(), AppError]:
run()
usage: number-report INPUT OUTPUT
Run the application with its arguments:
plenty script main.plenty numbers.txt report.txt
To make it directly executable on Unix, add #!/usr/bin/env -S plenty script
as its first line, then run chmod +x main.plenty. Runner options precede the
filename; everything after it belongs to your application. See
single-file scripts for optional
dependency metadata and locks.
For a retained native binary instead:
plenty --compile main.plenty -o number-report
./number-report numbers.txt report.txt
For an input containing 10, a blank line, and 20, the output file contains
count=2 total=30 followed by a newline. write_text replaces the output file;
choose a path whose contents may be replaced.
AppError preserves usage, parsing, I/O, allocation, and arithmetic errors.
Returning the error from main reports its cause on standard error and exits unsuccessfully.
run prints "saved report" only after writing succeeds. It propagates failures
to main so a failed operation cannot accidentally become a successful exit.
Values and their total must fit i64; overflow propagates as
OverflowError[i64]. The output file is not opened until the whole input has
been summarized successfully.
Checking the argument count directly keeps this two-path interface simple.
For named options and typed argument values, use the
typed command-line parser. The explicit readline loop
shows EOF handling for one file. For buffered
line iteration across multiple files or stdin, use
std.cli.fileinput; each loop item is a
Result that you handle with line?.
See the text I/O reference for file contracts and limits.
Check the application processing module
Test the processing module without command-line arguments. Save its companion
as report_demo.plenty:
pub def summarize(path: &str) -> Result[str, AllocError | IoError | ParseError | OverflowError[i64]]:
mut count = 0
mut total = 0
with open(path)? as file:
while True:
line = file.readline()?
if line == "":
break
cleaned = line.strip()?
if cleaned == "":
continue
total = total + i64.parse(cleaned)?
count = count + 1
Ok(f"count={count} total={total}\n"?)
Run this application in a scratch directory; it replaces numbers.txt and
report.txt:
from report_demo import summarize
def main() -> Result[(), AllocError | IoError | ParseError | OverflowError[i64]]:
write_text("numbers.txt", "10\r\n\r\n20\r\n")?
report = summarize("numbers.txt")?
write_text("report.txt", report)?
write_stdout(read_text("report.txt")?)?
Ok(())
count=2 total=30
Keep bytes separate from text
Use list[u8] when data may contain invalid UTF-8. Encoding text copies its exact
UTF-8 bytes; decoding validates every byte. Both operations observe their input,
so you can keep the original data.
def main() -> Result[(), IoError]:
text = "é\r\n"
bytes = encode_utf8(text)?
print(len(bytes))?
print(decode_utf8(&bytes)? == text)?
print(len(bytes))?
invalid = encode_utf8("x")?
mut raw = invalid
raw[0] = 255u8
match decode_utf8(&raw):
case Ok(_):
print("unexpected text")?
case Err(error):
print(error)?
Ok(())
4
True
4
IoError.Data(DataError.InvalidUtf8)
On Unix, filenames and executable arguments can contain arbitrary bytes except
NUL. Use args_bytes() to obtain those arguments, then pass a path byte list to
read_bytes(path) or write_bytes(path, contents). These helpers preserve file
contents and names without text decoding or newline conversion. They currently
support Linux. write_bytes creates or replaces its destination.
For supplied paths on stdin, use input_bytes(0u8). It returns one NUL-delimited
record as Some(record) or Nothing at EOF. Write records with
write_stdout_bytes(&record) followed by write_stdout_bytes(b"\0"). Finish
with flush_stdout()?. Empty records are distinct from EOF; a final nonempty
unterminated record is accepted.
CR, LF, and invalid UTF-8 remain part of the path. Keep the original bytes when
opening files, even if you separately decode or format them for display. The
finder experiment includes a working --null
mode. Its globs match bytes in that mode, so ? matches one byte.
See the binary I/O reference for partial effects, allocation failures, and remaining library limits.
Enumerate and classify paths
Use std.path to represent a native path and std.fs to enumerate it lazily.
Each generator item is a Result because advancement can fail after opening:
import std.fs as fs
import std.path as path
def main() -> Result[(), AllocError | IoError]:
write_text("sample.txt", "abc")?
root = path.from_text(".")?
for result in fs.entries(fs.read_dir(&root)?):
entry = result?
name = entry.file_name()?
if name.to_text()? == "sample.txt":
print(entry.file_type()? == fs.FileType.File)?
print(entry.metadata()?.size())?
Ok(())
True
3
entries consumes the directory owner. Exhaustion closes it; an early break,
return or propagated error drops it. Keep a named mutable ReadDir when you want
manual next() calls or explicit close(). Entries own their data and remain
valid after the directory closes. Enumeration includes dot-prefixed names and
has unspecified order.
entry.file_type() and entry.metadata() inspect a symlink itself.
fs.metadata(&path) follows its target; fs.symlink_metadata(&path) does not.
A dangling link can therefore be inspected without requiring its target.
Types distinguish regular files, directories, links and other native objects.
Path text conversion is checked. For arbitrary Unix filenames, use
name.as_bytes() and NUL-framed transport, avoiding text
decoding. These filesystem APIs currently require 64-bit Linux and statx for
metadata. See the directory reference.
Use fs.symlink to create a link and fs.canonicalize when you need its
resolved absolute name. Relative link targets are relative to the link’s parent:
import std.fs as fs
import std.path as path
def main() -> Result[(), IoError]:
write_text("original.txt", "data")?
target = path.from_text("original.txt")?
link = path.from_text("alias.txt")?
fs.symlink(&target, &link)?
print(fs.read_link(&link)?.to_text()?)?
resolved = fs.canonicalize(&link)?
print(resolved.is_absolute(), fs.metadata(&resolved)?.size())?
Ok(())
original.txt
True 4
Canonicalization requires existing components and can fail for dangling links or loops. Its result can become stale immediately: it does not secure a later open against replacement. For operations through a held directory, use rooted filesystem helpers. The metadata and links reference documents native path limits and error behavior.
Use temporary files and directories
Use std.tempfile when a task needs scratch storage. A with statement owns the
resource until its body ends and then cleans it up, including on an early return
or an error propagated with ?.
For a tool that needs a filename, use named_file. You can write through the
owner and pass its path to other code while the owner is alive:
import std.tempfile
import std.io
import std.fs
def main() -> Result[(), AllocError | IoError | std.fs.FsError | std.io.Partial]:
with std.tempfile.named_file(suffix=".txt")? as temporary:
std.io.write_all(temporary, &b"hello")?
print(read_text(&temporary.path().to_text()?)?)?
temporary.close()?
Ok(())
hello
The explicit close()? reports cleanup failures. Without it, leaving with
still attempts cleanup, but cannot report an error from that attempt.
For a stream with no filename, with std.tempfile.file()? as stream: lends a
std.io.FileStream instead.
A temporary directory can contain an entire working tree. Cleanup removes its contents recursively; symlinks do not cause traversal outside that tree:
import std.tempfile
import std.path
import std.fs
def main() -> Result[(), AllocError | IoError | std.fs.FsError]:
with std.tempfile.directory()? as scratch:
note = scratch.path().join(&std.path.from_text("note.txt")?)?
write_text(¬e.to_text()?, "working copy")?
print(read_text(¬e.to_text()?)?)?
scratch.cleanup()?
Ok(())
working copy
To preserve a named file or directory, call keep()? before cleanup. It returns
an owned path and transfers responsibility for removing it to your code.
Use the parent keyword argument to choose a directory explicitly. Otherwise,
the module uses the first nonempty TMPDIR, TEMP, or TMP, falling back to
/tmp when none is set.
See the reference for cleanup budgets, Linux filesystem requirements, and the trusted-parent requirement. For atomic publication of a completed file, continue with filesystem staging.
Stage files and clean temporary directories
Open a std.fs.Root to anchor operations to a directory. Child names are
single-component lossless paths: no slash, NUL, . or ... Child opens reject
symlinks. The initial root’s ancestors remain the caller’s trust decision.
An anonymous temporary file has no name until publish succeeds. Publication
refuses an existing destination. Returning early before publication closes the
file and releases its partial contents without deleting a named path.
This example uses a random private directory inside the working directory:
import std.io
import std.fs as fs
import std.path as path
from std.sync import CancellationToken
def main() -> Result[(), AllocError | IoError | std.io.StreamError | fs.FsError]:
current = path.from_text(".")?
parent = fs.root(¤t)?
prefix = path.from_text("plenty-example-")?
mut directory = fs.temporary_dir(&parent, &prefix, 8)?
original = path.from_text("original.txt")?
copied = path.from_text("copied.txt")?
mut staged = fs.temporary_file(directory.root())?
staged.stream.file.write("hello")?
staged.publish(&original)?
staged.close()?
cancel = CancellationToken()?
print(fs.copy_file(directory.root(), &original, &copied, 1024, &cancel, preserve=fs.CopyMetadata.PermissionsAndTimes)?)?
pattern = path.from_text("*.txt")?
names = fs.glob(directory.root(), &pattern, 10, 10, 1000)?
print(len(names))?
directory.root().remove_file(&original)?
directory.root().remove_file(&copied)?
directory.cleanup()?
Ok(())
5
2
These operations require Linux, a filesystem supporting O_TMPFILE, and
/proc/self/fd for publication. Unsupported environments return errors rather
than using a weaker fallback. Single-file copying uses bounded scratch storage, returns
errors for byte limits/cancellation, and leaves no partial destination name.
To preserve ordinary permission bits and access/modification times, pass
preserve=fs.CopyMetadata.PermissionsAndTimes to copy_file; ownership,
ACLs, xattrs and special permission bits are excluded.
Temporary-directory cleanup removes only an empty directory. Its destructor
is best-effort; call cleanup() when you need to observe failure. Other actors
must not rename entries in the parent while the temporary directory is live.
An identity check detects existing replacements but is not atomic with rmdir.
Use explicit bounded remove_tree only when intentionally deleting contents;
it can leave partial progress on cancellation or errors.
See the reference for root contracts, temporary-resource limits, and copy, deletion and glob budgets.
For a directory tree, use copy_tree with explicit entry, depth, byte and
handle budgets from copy_limits(). Its default copies contents only.
The destination must be new, symlinks and special files are rejected, and a
failure after creation leaves partial output for the caller to inspect or
remove. The recursive-copy example and recovery contract
show the complete call and explain TreeCopyError.Setup versus partial-output Partial/Entry errors.
For cooperating processes that must not operate on a resource simultaneously,
use std.fs.lock. Keep the returned owner alive
for the protected work. acquire_now reports contention immediately;
acquire_before takes a total deadline and a cancellation token. Use a stable
lockfile in a trusted directory and leave that file in place after releasing
the lock: removing it while another process owns it can split coordination
across different inodes. Locks close automatically, and normal subprocesses
do not inherit them across exec.
Create a private named file
When a file should be visible immediately and must not already exist, use
Root.create_file. Its default 0600 mode prevents group and other access from
creation; create_dir defaults to 0700. Both are restricted by the process
umask. An existing file, directory or symlink causes an error.
import std.io
import std.fs as fs
import std.path as path
def main() -> Result[(), AllocError | IoError | std.io.StreamError]:
parent = fs.root(&path.from_text(".")?)?
name = path.from_text("private-state")?
parent.create_dir(&name)?
directory = parent.open_dir(&name)?
token = path.from_text("token")?
mut file = directory.create_file(&token)?
file.file.write("secret")?
fs.set_file_unix_mode(&file, 0o400u32)?
file.close()?
directory.remove_file(&token)?
parent.remove_dir(&name)?
Ok(())
set_file_unix_mode changes the held file, so a replacement at its old path
cannot redirect the update. Root.set_unix_mode does the same for a directory.
Setting a mode does not close the handle or revoke existing access. Unlike
staging, a named file remains after a write failure; use the staging pattern
when readers must see only complete output. See the
permission contracts
for custom modes, ACLs, umask and errors.
Watch a directory for changes
Use std.fs.watch when a tool should notice that files changed: a local preview,
an indexer, or a configuration reloader. Watch the parent directory when another
program might replace a file by renaming a new version over it.
This example watches a temporary directory, writes a file, then inspects the queued batch. It handles overflow by requesting a full rescan instead of assuming every change was delivered:
import std.fs
import std.fs.watch
import std.path
import std.tempfile
def main() -> Result[(), AllocError | IoError | std.fs.FsError | std.fs.watch.WatchError]:
with std.tempfile.directory()? as scratch:
mut watcher = std.fs.watch.watcher()?
watcher.add(scratch.path())?
note = scratch.path().join(&std.path.from_text("note.txt")?)?
write_text(¬e.to_text()?, "hello")?
count = watcher.poll()?
for index in range(count):
event = watcher.event(index)?
if event.overflowed():
print("rescan the directory")?
elif event.created():
print("created", event.name.to_string()?)?
watcher.close()?
scratch.cleanup()?
Ok(())
created note.txt
poll() never waits. For a long-running program, use wait(&deadline, &cancel)
with a std.time deadline and cancellation token,
or register the watcher with a std.ready.Selector. Consume the batch before
the next call: event(index) copies its name, so saved events remain valid.
The default buffer bounds each read to 4 KiB, and polling does not allocate.
A directory watch covers its immediate children, not an entire tree. Events
may be coalesced and an event’s path may have changed again before you read it.
Treat notifications as a reason to check the current filesystem state, rather
than as a complete history. Names retain native bytes; the example uses
display() so a non-UTF-8 name does not prevent reporting the change.
See the reference for rename cookies, watch lifecycle events, overflow recovery, and resource limits.
Prune before descending
The small companion files provide two fixture directories. Run the program from
an empty work/ child of the directory containing these files; the root below
uses .. to reach the fixture without changing the working directory:
pub const value: i64 = 1
pub const value: i64 = 2
With a top-down walker, edit its directory list before asking for another row:
import std.fs as fs
import std.path as path
def main() -> Result[(), AllocError | IoError | fs.WalkError | OverflowError[i64]]:
root = path.from_text("../tree")?
mut walker = fs.walk(&root, True)?
while walker.next()?:
print(walker.current.path.to_text()?)?
mut index = len(walker.current.directories) - 1
while index >= 0:
if walker.current.directories[index].to_text()?.endswith("/pruned"):
drop(walker.current.directories.pop(index))
index = index - 1
walker.close()
print(walker.next()?)?
Ok(())
../tree
../tree/kept
False
False passed to fs.walk requests bottom-up order. Editing a bottom-up row
cannot undo descent. Child directory links appear in current.files, and are
never followed; a link to an ancestor cannot make this walker loop. An explicit
root may itself resolve through a link.
To continue siblings after an error, match walker.next() instead of using ?:
import std.fs as fs
import std.path as path
def main() -> Result[(), AllocError | IoError | fs.WalkError]:
root = path.from_text("missing-root")?
mut walker = fs.walk(&root, True)?
match walker.next():
case Ok(found):
print(found)?
case Err(error):
match error:
case fs.WalkError.Io(at, operation, cause):
print(at.to_text()?)?
print(operation)?
print(cause)?
case fs.WalkError.Allocation(cause):
print(cause)?
print(walker.next()?)?
Ok(())
missing-root
read_dir
IoError.System(2)
False
The failed subtree is already skipped. Another call continues any pending siblings. Propagating the error stops traversal with normal cleanup. Only the current scan owns a native directory stream; retaining results does not retain directory handles. See walking contracts.
Say which operation failed
IoError.System(2) says a file was missing, but not which file or which step
wanted it. std.error wraps an error in a Context holding an operation label
and, optionally, a path. The original error stays inside as cause, with its
type.
from std.error import Context, Subject, context, path_context, native_code
from std.path import Path, from_text
def read_limit(path: &Path) -> Result[list[u8], Context[IoError]]:
path_context(read_bytes(path.as_bytes()), "read limit", path)
def parse_limit(text: &str) -> Result[i64, Context[ParseError]]:
context(i64.parse(text), "parse limit")
def flush_output() -> Result[(), Context[IoError]]:
context(flush_stdout(), "flush output")
def limit(path: &Path) -> Result[i64, AllocError | IoError | Context[ParseError]]:
match read_limit(path):
case Ok(bytes):
Ok(parse_limit(&decode_utf8(bytes)?)?)
case Err(error):
print(error.describe()?)?
print(native_code(error.cause))?
Ok(10)
def run() -> Result[(), AllocError | IoError | Context[IoError] | Context[ParseError]]:
path = from_text("limit.txt")?
print(limit(&path)?)?
match parse_limit("ten"):
case Ok(value):
print(value)?
case Err(error):
print(error.describe()?)?
print(error.cause == ParseError.Invalid)?
flush_output()?
Ok(())
def main() -> i32:
match run():
case Ok(_):
0i32
case Err(cause):
Context("run", Subject.Absent, cause).report()
1i32
read limit "limit.txt": IoError.System(2)
Option[i32].Some(2)
10
parse limit: ParseError.Invalid
True
context(result, label) and path_context(result, label, &path) pass an Ok
through and wrap an Err, including operations whose success is (), as
flush_output shows. main uses the direct form,
Context(label, Subject.Absent, cause). Nothing is wrapped implicitly: ?
needs a compatible declared error, so read_limit returns Context[IoError]
and callers can include that context in an error sum.
describe() builds one line: the label, the quoted path if there is one, and
the cause. It allocates, so it returns a Result. error.cause is the untouched
typed error; match it, compare it, or ask native_code for the OS number.
report() is for the edge of the program. It flushes standard output and
writes error: plus the description to standard error. It returns nothing and
does not choose an exit status, so main returns i32 and decides. If memory
runs out while describing, report still writes the label with a fixed
message, and a failed path copy in path_context keeps the label and cause.
Here run preserves its allocation, I/O, and contextual errors in a sum.
The outer context adds the operation label without discarding the cause.
Return a Context or your own enum when a failure needs more explanation. See
the module reference for nesting contexts and
the rules standard modules follow.
For a human-readable cause, give its type the public std.report formatting
method. Contexts then format their operation labels and delegate to that cause:
from std.error import Context, Subject
from std.fmt import Formatter, FormatError
from std.report import to_string
pub enum SettingsError:
Invalid
pub def format[W](self, output: &mut Formatter[W]) -> Result[(), FormatError]:
output.text("invalid setting")
def main() -> Result[(), FormatError | IoError]:
inner = Context("parse", Subject.Absent, SettingsError.Invalid)
outer = Context("load settings", Subject.Absent, inner)
print(to_string(&outer, 120, 8)?)?
print(to_string(&outer, 120, 8)?)?
Ok(())
load settings: parse: invalid setting
load settings: parse: invalid setting
Formatting borrows both contexts and includes the cause once. Returning an
Err with this context type from main selects the same wording automatically,
with the entrypoint’s prefix and newline. A context whose cause has no formatting
method still uses structural entrypoint reporting. The existing describe,
describe_with, report and report_with methods keep their behavior.
Without a path, context formatting adds no allocation. A present path still
uses fallible Path.display; failure leaves the typed cause intact and may
leave part of the report already written. See
bounded reports for limits and failure handling.
Pack bytes in a compact buffer
For a binary record, frame, or file body, std.bytes.Bytes provides checked
byte-range operations and works with byte-oriented I/O. Its contiguous buffer
stores one byte per element. A list[u8] also packs its elements into bytes;
use its ordinary list operations and element references when those suit the task.
import std.bytes
from std.bytes import Bytes, BytesError
def checksum(view: &Bytes, start: i64, stop: i64) -> Result[i64, OverflowError[i64]]:
mut total = 0
for index in range(start, stop):
total += i64(view.get(index))
Ok(total)
def main() -> Result[(), AllocError | IoError | BytesError | OverflowError[i64]]:
mut frame = std.bytes.with_capacity(8)?
frame.push(2u8)?
frame.extend(&std.bytes.from_text("hi")?)?
frame.push(0u8)?
frame.push(255u8)?
print(frame.len(), frame.capacity(), frame)?
print(checksum(&frame, 1, 3)?)?
payload = frame.slice(1, 3)?
print(payload.to_text()?)?
print(str.repr(frame.to_text())?)?
Ok(())
5 8 b"\x02hi\x00\xff"
209
hi
Result[str, std.bytes.BytesError].Err(std.bytes.BytesError.InvalidUtf8)
Growth and copies can fail to allocate, so they return AllocError; to_text
returns BytesError. A range is start up to but not including stop, and
must lie inside the buffer: there are no negative offsets and nothing is
clamped. A range outside the buffer, such as checksum(&frame, 3, 9) on five
bytes, is a bug in the caller, so it stops the program with a
trap report
naming the method, Bytes.get: index is outside 0..len(), and your calling
line. Check a range that comes from input first and return your own error.
slice and copy make independent buffers; nothing is copied unless you ask. to_text succeeds only for valid UTF-8, and a zero byte
is ordinary data. Printing a buffer spells it as a byte literal, with \xhh
for bytes that are not printable ASCII; for hex output, see
printing bytes.
checksum shows how to work on part of a buffer: borrow it and pass the range.
While that borrow is in use the owner cannot change, so growth can never move
bytes out from under a reader:
import std.bytes
def main() -> Result[(), AllocError | IoError | std.bytes.BytesError]:
mut frame = std.bytes.from_text("hi")?
view = &frame
frame.push(0u8)?
print(view.len())?
Ok(())
conflicting borrow: cannot modify or exclusively borrow `frame` while it is borrowed
File-reading and argument builtins still return list[u8]; from_list
converts one with an explicit copy. The output builtins write_stdout_bytes,
write_stderr_bytes and write_bytes take a &[u8] view, so pass a buffer or
a range of it with no conversion:
import std.bytes
def main() -> Result[(), AllocError | IoError]:
mut record = std.bytes.from_list(&encode_utf8("name=")?)?
record.extend(b"\xff\0")?
write_stdout_bytes(&record[0:5])?
print(record.len(), record.find(61u8, 0, record.len()))?
Ok(())
name=7 Option[i64].Some(4)
A parser should not call get once per byte: each call costs far more than
the comparison it feeds. The scans search a whole range in one call and return
an absolute index. find_in stops at any byte of a set, find_not_in at the
end of a run, and matches_text compares in place without copying.
to_text_range copies one range out as checked text, with no intermediate
Bytes:
import std.bytes
def main() -> Result[(), AllocError | IoError | std.bytes.BytesError | OverflowError[i64]]:
line = std.bytes.from_text("name=Ada; role=engineer")?
stops = std.bytes.from_text("=;")?
blank = std.bytes.from_text(" ")?
mut at = 0
while at < line.len():
match line.find_in(&stops, at, line.len()):
case Some(index):
print(index, line.to_text_range(at, index)?)?
at = index + 1
case Nothing:
print("rest", line.to_text_range(at, line.len())?)?
at = line.len()
print(line.find_not_in(&blank, 9, line.len()))?
print(line.matches_text(11, "role"), line.count(32u8, 0, line.len()))?
Ok(())
4 name
8 Ada
15 role
rest engineer
Option[i64].Some(11)
True 2
Index syntax reads and writes single bytes, with the same rules as a list
index: a negative index counts from the end, and a missing one stops the
program with index out of bounds. Iterating over &buffer reads each byte
without consuming the buffer:
import std.bytes
def main() -> Result[(), AllocError | IoError | std.bytes.BytesError | OverflowError[u8] | OverflowError[i64]]:
mut word = std.bytes.from_text("plenty")?
word[0] = 80u8
word[-1] -= 32u8
print(word.to_text()?, word[1])?
mut vowels = 0
for b in &word:
if b == 101u8 or b == 89u8:
vowels += 1
print(vowels)?
word.fill(1, 5, 46u8)
print(word.to_text()?)?
Ok(())
PlentY 108
2
P....Y
get and set take the same checked index without counting from the end.
for b in &mut word yields &mut u8 references into the buffer, so
*b = 0u8 in its body clears every byte in place. A
slice such as word[1:3]? copies with the list
rules, clamping its bounds. The slice method keeps the range rule instead:
frame.slice(1, 3)? copies exactly that range, and a range outside the buffer
traps. Both report only AllocError.
Fixed binary data, such as a file signature or a protocol delimiter, is
written as a byte literal: b"..." holds ASCII characters, the string escapes,
and \xHH for any other byte. Its type is bytes, an immutable value in the
program’s read-only data, so it never allocates. It has a length, indexing, and
iteration:
import std.bytes
def main() -> Result[(), AllocError | IoError | OverflowError[i64]]:
magic = b"\x89PNG"
print(magic, len(magic), magic[1])?
mut total = 0
for byte in magic:
total += i64(byte)
print(total)?
Ok(())
b"\x89PNG" 4 80
366
Operations that only read another sequence of bytes, such as find_bytes,
matches, find_in, and extend, take a literal as readily as a buffer. Their
parameter is a &[u8]
slice view: a
literal, &buffer, or part of either, &buffer[a:b], converts to it, so each
call reads the bytes where they are and nothing is copied. Your own functions
can accept every form the same way:
import std.bytes
def starts_png(data: &[u8]) -> bool:
len(data) >= 4 and data[0] == 137u8 and data[1] == 80u8
def main() -> Result[(), AllocError | IoError]:
magic = b"\x89PNG"
mut header = std.bytes.from_bytes(&magic)?
header.extend(b"\r\n\x1a\n")?
print(header.find_bytes(b"\r\n", 0, header.len()))?
print(header.matches(0, &magic[:4]), header.find_in(b"\n\r", 0, header.len()))?
print(starts_png(&header), starts_png(&header[4:]), starts_png(b"GIF8"))?
Ok(())
Option[i64].Some(4)
True Option[i64].Some(4)
True False False
The literal is never converted into a Bytes behind your back. To change bytes,
or to call a Bytes method such as find on them, copy any view
with from_bytes, as header does; the copy allocates, so it returns a
Result. A slice of a literal, such as b"GET /index"[4:]?, is also an owned
Bytes. Stream writes in std.io accept a literal the same way, as the next
lessons show.
A byte literal accepts only ASCII characters, so b"é" is an error; use
from_text for the UTF-8 encoding of text. See
byte literals for the full rules.
See the module reference for every operation, its costs, and what views cannot do yet.
Build and normalize bounded text
std.text offers a reusable UTF-8 builder. Its limit counts bytes: é takes
two bytes and 🙂 takes four. A failed append leaves the old contents intact.
import std.text
import std.unicode
def main() -> Result[(), AllocError | IoError | std.text.TextError]:
mut message = std.text.builder(32)?
message.append("Hello, ")?
message.append("é🙂")?
print(message.to_text()?, message.byte_len())?
message.clear()
message.append("again")?
print(message.finish()?)?
words = ["one", "two", "three"]?
print(std.text.join(&words, " / ", 32)?)?
print(str.repr(std.text.hard_wrap("é🙂abc", 2, 32)?)?)?
print(std.unicode.version())?
print(std.unicode.nfc("é", 16)? == "é")?
print(std.unicode.nfkc("ffi", 16)?)?
Ok(())
Hello, é🙂 13
again
one / two / three
"é🙂\nab\nc"
17.0.0
True
ffi
Use to_text() for a snapshot when keeping the builder, and finish() when
done. finish consumes it even if allocating the final string fails; to_text
keeps it available for retry. Both currently copy the output bytes.
hard_wrap counts Unicode scalars, preserves LF, and can split a combining sequence;
it does not measure terminal columns or find word boundaries. Normalization
has a separate scalar limit covering the input and decomposed intermediate.
For example, decomposing é needs room for two scalars even though NFC can
compose them back to one. Every operation remains fallible: use ? or match
TextError to handle allocation, invalid limits, and output that exceeds them.
Normalization never happens automatically. NFC treats canonically equivalent
spellings consistently; NFKC also removes compatibility distinctions such as
the ligature above. Choose it only when that loss is intended. Byte decoding
still rejects malformed UTF-8 before normalization; embedded NUL is ordinary
text. See std.text and
std.unicode for costs and complete contracts.
For prose, use fill to wrap at word boundaries. shorten collapses whitespace
and truncates to one line; dedent removes a common space/tab margin. These
helpers use default input and output caps of 1 MiB, which you can change with
max_input_bytes and max_bytes.
import std.text
def main() -> Result[(), IoError | std.text.TextError]:
print(std.text.fill("one two three four", 9)?)?
print(std.text.shorten("one two three four", 13)?)?
print(std.text.indent(&std.text.dedent(" first\n second")?, "> ")?)?
Ok(())
one two
three
four
one two [...]
> first
> second
Widths count scalars rather than display columns. Use a reusable
TextWrapper
when you need indentation, tab options, or a maximum number of lines. Wrap
each paragraph separately: the default settings treat input newlines as spaces.
Read and write streams in pieces
A file, a pipe, or a terminal rarely moves all the bytes you ask for in one
call. std.io makes that visible: one read or write reports how many bytes
it moved, and helpers repeat the call until a range is complete.
A stream is any pub class with the right methods. This one hands out its
text at most three bytes at a time, as a slow pipe would:
import std.bytes
import std.io
from std.bytes import Bytes
from std.io import StreamError
pub class Trickle:
pub data: Bytes
pub at: i64
pub def read(self: &mut Trickle, buffer: &mut Bytes, start: i64, stop: i64) -> Result[i64, StreamError]:
if start < 0 or start > stop or stop > buffer.len():
return Err(StreamError.OutOfRange)
if self.at < 0 or self.at > self.data.len():
return Err(StreamError.OutOfRange)
# Valid ranges give nonnegative differences <= i64.MAX.
# count <= stop-start and <= data.len()-at, so both ends stay in range.
unchecked:
mut count = stop - start
if count > 3:
count = 3
if count > self.data.len() - self.at:
count = self.data.len() - self.at
buffer.overwrite(start, &self.data[self.at:self.at + count])
self.at += count
Ok(count)
def main() -> Result[(), AllocError | IoError | StreamError | std.bytes.BytesError]:
mut source = Trickle(std.bytes.from_text("0123456789")?, 0)
mut buffer = std.bytes.with_capacity(8)?
buffer.resize(8, 46u8)?
print(source.read(&mut buffer, 0, 8)?, buffer.to_text()?)?
print(str.repr(std.io.read_exact(&mut source, &mut buffer, 0, 4))?, buffer.to_text()?)?
print(str.repr(std.io.read_exact(&mut source, &mut buffer, 0, 8))?)?
print(source.read(&mut buffer, 0, 8)?)?
Ok(())
3 012.....
Result[(), std.io.Partial].Ok(()) 3456....
Result[(), std.io.Partial].Err(std.io.Partial(completed=3, cause=std.io.StreamError.UnexpectedEof))
0
read fills part of buffer[start:stop] and returns the count. A reader
checks the range itself and reports OutOfRange, as the
stream contract requires;
overwrite would otherwise trap on it. Three of eight
bytes is a short read, not an error; zero means end of input. read_exact
calls read until the range is full. When the input ends first, the error says
how many bytes arrived, and they are in the buffer.
Writing has the same shape. write may accept only part of a range, and
write_all continues until all of it is delivered. A BufWriter collects
small writes in a fixed-size buffer and passes them on in larger pieces:
import std.bytes
import std.io
def main() -> Result[(), IoError | AllocError | std.io.Partial | std.io.StreamError | std.bytes.BytesError | OverflowError[i64]]:
mut input = std.io.buffered_reader(std.io.cursor(std.bytes.from_text("red\ngreen\nblue")?), 4)?
mut output = std.io.buffered_writer(std.io.stdout(), 16)?
mut line = std.bytes.empty()?
mut count = 0
while input.read_until(10u8, &mut line, 64)? > 0:
count += 1
std.io.write_all(&mut output, b"line: ")?
std.io.write_all(&mut output, &line)?
line.clear()
std.io.write_all(&mut output, b"\n")?
output.flush()?
print(count, output.pending(), output.capacity())?
Ok(())
line: red
line: green
line: blue
3 0 16
write_all takes the byte literal b"line: " as readily as the buffer
line: a stream’s write reads a &[u8] view in place, so it also takes part
of a buffer, &line[:3]. A class of your own becomes a writer with a flush
method and a write method taking source: &[u8] that returns
how many bytes it accepted; the
module reference shows one.
cursor turns a Bytes into a stream, which is how to test stream code
without a file. read_until appends one line, including its newline, and never
more than the limit of 64 bytes, so a file with no newline cannot exhaust
memory. Both buffers are allocated once, with the capacity you pass, and never
grow. Output reaches the terminal when the buffer fills and at flush, so
flush before printing anything that must come after it.
Bytes held by a BufWriter are not written when it goes out of scope. Call
flush, or close_buffered for a stream that also needs closing:
import std.bytes
import std.io
from std.io import SeekFrom
type ByteTransferError = std.io.Partial | std.bytes.BytesError | std.io.StreamError
def main() -> Result[(), AllocError | IoError | ByteTransferError]:
mut file = std.io.buffered_writer(std.io.open_file("streams-lesson.bin", "w+")?, 8)?
record = std.bytes.from_list(&[1u8, 0u8, 255u8, 10u8, 7u8]?)?
std.io.write_all(&mut file, &record[:5])?
print(file.pending(), std.io.seek_writer(&mut file, SeekFrom.Start(2))?, file.pending())?
mut back = std.bytes.with_capacity(3)?
back.resize(3, 0u8)?
std.io.read_exact(&mut file.inner, &mut back, 0, 3)?
print(back.to_list()?)?
print(str.repr(std.io.close_buffered(&mut file))?)?
print(str.repr(file.flush())?)?
Ok(())
5 2 0
[255, 10, 7]
Result[(), std.io.CloseError].Ok(())
Result[(), std.io.StreamError].Err(std.io.StreamError.Io(IoError.Closed))
Five bytes wait in the buffer until seek_writer delivers them and moves the
file to byte 2. File streams carry raw bytes: 255 and the newline byte come
back unchanged. close_buffered flushes, closes even when the flush fails, and
reports which of the two went wrong.
See the module reference for every error, the copy helper, and the standard stream adapters.
A text filter over files or stdin
For a command-line text filter, let
std.cli.fileinput handle the buffered
reads, UTF-8 decoding, and owned files:
import std.cli.fileinput as fileinput
from std.cli.fileinput import FileInputError
def main() -> Result[(), IoError | AllocError | FileInputError]:
for line in fileinput.lines()?:
write_stdout(line?)?
flush_stdout()?
Ok(())
After compiling, pass filenames in order, or pipe text into the executable with
no arguments. A filename - selects stdin. The outer ? handles argument-list
construction; line? propagates an error while opening, reading, or decoding.
Use write_stdout, not print, because lines retain their endings. The final
line of a file need not have a newline.
For applications that parse flags, pass the chosen list[Path] to
fileinput.files(paths) instead: lines() does not parse options. Both APIs
use chunked buffering, not a system call for each character, and decode strict
UTF-8. Breaking out closes owned files, never stdin; any bytes read ahead into
the iterator’s buffer are discarded, so do not interleave other stdin readers.
For fixed-width in-memory fields, use the binary packet cursor instead of manually tracking offsets or expecting stream reads to be atomic.
A read range must satisfy 0 <= start <= stop <= buffer.len(). Invalid
buffer ranges trap before reading; see the stream contract.
Read and write binary packets
A Cursor keeps the byte position for a
packet. Its typed operations read or write one complete field, with explicit
byte order and no manual offset bookkeeping.
import std.bytes
import std.io
from std.encoding import Endian
from std.io import SeekFrom
def main() -> Result[(), AllocError | IoError | RangeError | std.bytes.BytesError | std.io.BinaryError | std.io.StreamError]:
payload = b"hello"
mut packet = std.io.cursor(std.bytes.empty()?)
packet.write_u8(1u8)?
# Check that a payload size fits its wire field before converting it.
packet.write_u32(u32.try_from(len(payload))?, Endian.Big)?
packet.write_f32(1.5f32, Endian.Big)?
packet.write_bytes(&payload, 64)?
drop(packet.seek(SeekFrom.Start(0))?)
version = packet.read_u8()?
length = i64(packet.read_u32(Endian.Big)?)
scale = packet.read_f32(Endian.Big)?
# A hostile length is rejected before allocating its owned payload copy.
body = packet.read_bytes(length, 64)?
packet.require_end()?
print(version, scale, body.to_text()?, packet.position())?
Ok(())
1 1.5 hello 14
The checked u32 conversion propagates RangeError if a payload length cannot
fit the wire field. read_bytes returns an owned copy because borrowed views
cannot escape a method. Scalar fields allocate nothing when storage already has capacity.
If a field is truncated, BinaryError.Truncated reports its position, requested
width, and available bytes. The failed operation leaves the cursor in place;
previous fields remain consumed. require_end detects unexpected trailing
bytes. Raw stream reads and writes and seek use the same position, so they
can be interleaved with typed fields when the format calls for them.
Read and write CSV records
std.csv reads a table one record at a time, so a large file never has to fit
in memory. A reader sits on any byte stream and fills the
same Record on every call:
import std.bytes
import std.csv
import std.io
type CsvInputError = ParseError | std.csv.CsvError | std.bytes.BytesError
def main() -> Result[(), AllocError | IoError | CsvInputError | OverflowError[i64]]:
text = "item,price,note\r\npen,3,\"blue, fine\"\r\nbook,12,\"says \"\"read me\"\"\"\r\ncard,2,\"two\nlines\"\r\n"
mut input = std.csv.reader(std.io.cursor(std.bytes.from_text(text)?), &std.csv.rfc4180(), 16)?
mut record = std.csv.record()?
input.read(&mut record)?
print(str.repr(record.texts()?)?)?
mut total = 0
while input.read(&mut record)?:
total += i64.parse(&record.text(1)?)?
print(record.number(), record.line(), record.text(0)?, str.repr(record.text(2)?)?)?
print(total, input.records())?
Ok(())
["item", "price", "note"]
2 2 pen "blue, fine"
3 3 book "says \"read me\""
4 4 card "two\nlines"
17 4
read returns True while there is another record and False at the end of
the input. The reader’s buffer is the 16 bytes passed to reader; records
longer than that are assembled across several fills. Quoted fields may hold
the delimiter, line breaks, and doubled quotes, and arrive without their
quoting. The first record is not special, so the program reads the header
itself.
Fields are bytes. text(index) checks one field as UTF-8 and returns a copy
you own; texts() copies them all. The record itself is overwritten by the
next read, which is why read takes it as &mut.
rfc4180() is the usual dialect: comma, double quote, and every record as
wide as the first. A record of another width is reported, but it has been read
and the reader carries on. Malformed input stops the reader for good:
import std.bytes
import std.csv
import std.io
from std.csv import CsvError
def main() -> Result[(), AllocError | IoError | CsvError | std.bytes.BytesError]:
text = "a,b\nc\nd,e\nf,g\"h\ni,j\n"
mut input = std.csv.reader(std.io.cursor(std.bytes.from_text(text)?), &std.csv.rfc4180(), 16)?
mut record = std.csv.record()?
mut more = True
while more:
outcome = input.read(&mut record)
match outcome:
case Ok(found):
if found:
print(str.repr(record.texts()?)?)?
else:
more = False
case Err(cause):
print(str.repr(&cause)?)?
match cause:
case CsvError.Width(_, _, _):
print("skipped", str.repr(record.texts()?)?)?
case _:
more = False
print(str.repr(input.read(&mut record))?)?
Ok(())
["a", "b"]
std.csv.CsvError.Width(2, 2, 1)
skipped ["c"]
["d", "e"]
std.csv.CsvError.Invalid(std.csv.Fault.BareQuote, std.csv.Position(record=4, field=1, line=4, offset=13))
Result[bool, std.csv.CsvError].Err(std.csv.CsvError.Stopped)
Width(2, 2, 1) says record 2 should have two fields and has one. The
Position names the record, the field index, the line, and the byte offset of
the stray quote. Record i,j is never reached: after a broken quote there is
no trustworthy place to resume.
A writer builds one record from fields and delivers it at end_record.
texts does both for a list. Change the dialect’s public fields before
creating the writer:
import std.bytes
import std.csv
import std.io
from std.csv import Newline
def main() -> Result[(), AllocError | IoError | std.bytes.BytesError | std.csv.CsvError]:
mut dialect = std.csv.rfc4180()
dialect.newline = Newline.Lf
dialect.max_field = 12
mut output = std.csv.writer(std.io.stdout(), &dialect)?
output.texts(&["item", "note"]?)?
output.texts(&["pen", "blue, fine"]?)?
output.text("book")?
output.text("says \"hi\"")?
output.end_record()?
print(str.repr(output.texts(&["card"]?))?)?
print(str.repr(output.text("much too long for a field"))?)?
raw = std.bytes.from_text("card:plain")?
output.field(&raw[:4])?
output.field(&raw[5:10])?
output.end_record()?
output.flush()?
print(output.records())?
Ok(())
item,note
pen,"blue, fine"
book,"says ""hi"""
Result[(), std.csv.CsvError].Err(std.csv.CsvError.Width(4, 2, 1))
Result[(), std.csv.CsvError].Err(std.csv.CsvError.Rejected(std.csv.Fault.FieldTooLong, 4, 0))
card,plain
4
Only fields that need quotes get them. The one-field record is refused by the
width policy and the oversized text by max_field; neither writes anything.
field adds a byte range without building a string first. Call flush when
you are done: a writer does not flush or close its stream when it goes out of
scope.
Limits apply to reading too. A field over max_field bytes or a record over
max_record is an error at the byte that does not fit, so a file with a
missing closing quote cannot consume all memory.
See the module reference for the exact grammar,
every error, the borrowed data() view, and what allocates.
Process XML events
std.xml reads one bounded event at a time from a
byte stream. Event text has already been checked as UTF-8
and entity references have been decoded. Match the events your application
needs, and keep reading through Eof to check the complete document:
import std.bytes
import std.io
import std.xml
from std.xml import Event
def main() -> Result[(), AllocError | IoError | std.bytes.BytesError | std.xml.XmlError]:
source = "<items><item>Tea & cake</item><item>Milk</item></items>"
mut input = std.xml.reader(std.io.cursor(std.bytes.from_text(source)?), &std.xml.limits())?
mut output = std.xml.writer(std.io.cursor(std.bytes.empty()?), &std.xml.limits())?
mut done = False
while not done:
event = input.next()?
match &event:
case Event.Text(text):
print(*text)?
case Event.Eof:
done = True
case _:
pass
output.write(&event)?
print(output.inner.data.to_text()?)?
Ok(())
Tea & cake
Milk
<items><item>Tea & cake</item><item>Milk</item></items>
The writer escapes values and checks the element structure. Passing Eof
checks that the root has closed and flushes its stream. When constructing
events yourself, call finish() after the final element. Events own their
strings, so retaining an event also retains its memory.
Choose limits for the input you accept. max_token bounds a complete tag,
comment, or text run; max_depth bounds nesting. The reference describes
all limits and errors. This example lets
errors propagate so malformed input and stream failures remain visible.
A failed file writer may have emitted partial output; applications that need
replacement only on success should manage a temporary destination.
This reader accepts UTF-8 XML 1.0 without a DTD. It never loads external entities. Namespace prefixes remain literal names, so applications needing namespace URI resolution must supply that policy themselves.
Read the environment and run other programs
std.env reads the variables a program was started with. std.process starts
another program from a list of arguments and collects what it prints:
import std.bytes
import std.io
import std.env
import std.process
from std.process import Stdio
def main() -> Result[(), AllocError | IoError | std.bytes.BytesError | std.io.StreamError]:
print(std.env.get("PLENTY_LESSON_UNSET")?)?
mut command = std.process.command("printf")?
command.arg("%s\n")?
command.arg("two words; echo $HOME | wc")?
command.stdout(Stdio.Piped)
mut child = command.spawn()?
output = child.output(4096)?
print(output.stdout.to_text()?)?
print(output.status.success(), output.status.code())?
Ok(())
Option[str].Nothing
two words; echo $HOME | wc
True Option[i32].Some(0)
An unset variable is Nothing, not an error. Each arg is one argument,
delivered byte for byte: the semicolon, the $HOME, and the | are printed,
not run, because no shell reads them. Stdio.Piped connects the child’s output
to a pipe, and output collects at most 4096 bytes of it and waits for the
child to end.
If a configuration supplies arguments as one string, use std.shlex.split
to recognize quoted words, then pass them individually to arg. Do not quote
arguments passed to arg itself; those quotes would become literal data.
import std.shlex
from std.shlex import ShlexError
def main() -> Result[(), ShlexError | AllocError | IoError]:
arguments = std.shlex.split("--title 'Two words' --literal '$HOME'")?
for argument in &arguments:
print(argument)?
Ok(())
--title
Two words
--literal
$HOME
split removes word quoting without expanding $HOME or interpreting pipes.
std.shlex also provides quote for the cases
where one argument must be written into a POSIX shell command string.
To find where to keep settings or caches, ask
std.fs.dirs rather than reading HOME
yourself: config_dir() and cache_dir() apply the XDG rules and return a
Path, or Nothing when the environment names no directory.
When shell syntax is what you want, ask for it by name. shell runs
/bin/sh -c with your script. communicate also feeds the child’s input:
import std.io
import std.bytes
import std.process
from std.process import Stdio
def main() -> Result[(), AllocError | IoError | std.io.StreamError | std.bytes.BytesError]:
mut command = std.process.shell("tr a-z A-Z; echo \"mode: $MODE\" >&2; exit 3")?
command.env("MODE", "loud")?
command.stdin(Stdio.Piped)
command.stdout(Stdio.Piped)
command.stderr(Stdio.Piped)
mut child = command.spawn()?
output = child.communicate(b"quiet words", 4096)?
print(output.stdout.to_text()?)?
print(output.stderr.to_text()?)?
print(output.status.success(), output.status.code())?
Ok(())
QUIET WORDS
mode: loud
False Option[i32].Some(3)
env sets a variable for this child only. communicate writes the input,
reads both outputs as they arrive so that no pipe fills up, and returns the
exit status. A nonzero exit code is a normal result: the program ran. Never
put text you did not write into a shell script; pass it as an argument or a
variable.
A program that cannot be started is an error, and a child that runs too long can be stopped:
import std.time
import std.process
def main() -> Result[(), AllocError | IoError | std.process.WaitError]:
mut missing = std.process.command("plenty-no-such-program")?
match missing.spawn():
case Ok(_):
print("started")?
case Err(error):
print(error)?
mut sleeper = std.process.command("sleep")?
sleeper.arg("60")?
mut child = sleeper.spawn()?
cancel = CancellationToken()?
match child.wait_timeout(&std.time.milliseconds(50u32), &cancel)?:
case Some(_):
print("ended")?
case Nothing:
print("still running")?
child.kill()?
status = child.wait()?
print(status.success(), status.code(), status.signal())?
Ok(())
IoError.System(2)
still running
False Option[i32].Nothing Option[i32].Some(9)
System(2) is the operating system’s “no such file”. wait_timeout gives up
after 50 milliseconds and leaves the child running. kill stops it, and wait
collects its status: ended by signal 9, with no exit code. Always wait for a
child you started, even after killing it; a dropped Child is not cleaned up.
See the references for std.env and
std.process, including pipes you read and
write yourself.
To inspect CPU, memory, disk or process information without launching a command, continue to system information.
Read system information
std.system reads Linux’s /proc files into typed values. Open the source once,
then ask for the information needed. Each query is fresh and fallible:
import std.system
from std.system import SystemError
def main() -> Result[(), IoError | SystemError]:
source = std.system.local()?
cpus = source.cpu_count()?
memory = source.memory()?
uptime = source.uptime()?
print("CPU information available:", cpus > 0)?
print("Memory information available:", memory.total_bytes > 0u64)?
print("Uptime available:", uptime.seconds >= 0.0)?
Ok(())
CPU information available: True
Memory information available: True
Uptime available: True
The example prints checks so its output stays the same on different machines.
A monitor can print cpus, memory.total_bytes and uptime.seconds directly.
There is no timer or background thread: another call reads another snapshot.
For memory headroom, prefer available_bytes to free_bytes: the kernel’s
available estimate includes memory it expects to reclaim. The field is optional
because not every source provides it. Here is how to keep absence distinct from
zero, using sample values so both paths are visible:
import std.system
def show_available(memory: &std.system.Memory) -> Result[(), IoError]:
match memory.available_bytes:
case Some(bytes):
print("available bytes:", bytes)?
case Nothing:
print("available memory is not reported")?
Ok(())
def main() -> Result[(), IoError]:
reported = std.system.Memory(8192u64, 1024u64, Some(4096u64), 0u64, 0u64)
absent = std.system.Memory(8192u64, 1024u64, Nothing, 0u64, 0u64)
show_available(&reported)?
show_available(&absent)?
Ok(())
available bytes: 4096
available memory is not reported
Reads have default byte limits; override them with a named argument such as
source.memory(max_bytes=65536). A file that exceeds the limit produces an error,
not a truncated record. open_procfs(&path) reads a different procfs mount or
captured directory, useful when testing a monitoring tool.
process_ids() returns a bounded, sorted list of visible process IDs.
process(pid) reads one status record. Processes can exit between those calls,
and access can be denied; handle the SystemError.Io cause rather than treating
the whole list as guaranteed live. Use std.process
to manage children your program starts.
block_devices() reports device and partition sizes; disk_io() reports
counters. Avoid summing whole disks with their partitions. Neither gives free
filesystem space. CPU and memory values can describe the host even inside a
container; they are not cgroup quota measurements or a recommendation for how
many workers to start.
See std.system for fields, units, error
variants and snapshot limitations.
Store records in SQLite
For a dedicated file of byte-keyed records without SQL, the next lesson’s
std.db.kv supplies a smaller interface over this same
engine. This lesson uses SQL to describe and query structured rows.
std.db.sqlite keeps data in a SQLite database: a single file, or memory.
Values travel only through prepared statements, so a value is never
mistaken for SQL.
Insert and query
execute runs fixed SQL such as a schema. prepare compiles one statement
with ? parameters; bind each by number, from 1, and run it. A query is
read a row at a time with step, and each column, numbered from 0, is read
with the method for its type:
import std.db.sqlite
def main() -> Result[(), AllocError | IoError | std.db.sqlite.DbError]:
mut db = std.db.sqlite.memory(65536)?
db.execute("CREATE TABLE books(title TEXT NOT NULL, year INTEGER, rating REAL)")?
mut insert = db.prepare("INSERT INTO books VALUES (?, ?, ?)")?
insert.bind_text(1, "Dune")?
insert.bind_integer(2, 1965)?
insert.bind_real(3, 4.3)?
insert.run()?
insert.bind_text(1, "x'); DROP TABLE books; --")?
insert.bind_null(2)?
insert.bind_real(3, 1.0)?
insert.run()?
mut query = db.prepare("SELECT title, year FROM books WHERE rating > ? ORDER BY rating")?
query.bind_real(1, 0.5)?
while query.step()?:
print(query.text(0)?, query.optional_integer(1)?)?
Ok(())
x'); DROP TABLE books; -- Option[i64].Nothing
Dune Option[i64].Some(1965)
The second title looks like an attack but is stored as plain text. Reads
copy each value out, so a str from text stays valid after the next
step. memory(65536) sets the largest text, blob or row the connection
will store or read; anything larger is TooLarge.
Types are checked
Each read requires the column’s stored type. A NULL needs optional_integer
or another optional read, and a number stored as text is not an integer:
import std.db.sqlite
def main() -> Result[(), AllocError | IoError | std.db.sqlite.DbError]:
mut db = std.db.sqlite.memory(65536)?
mut query = db.prepare("SELECT '42', NULL")?
query.step()?
print(str.repr(query.integer(0))?)?
print(str.repr(query.integer(1))?)?
print(str.repr(query.kind(0)?)?)?
Ok(())
Result[i64, std.db.sqlite.DbError].Err(std.db.sqlite.DbError.TypeMismatch(std.db.sqlite.ValueType.Text))
Result[i64, std.db.sqlite.DbError].Err(std.db.sqlite.DbError.TypeMismatch(std.db.sqlite.ValueType.Null))
std.db.sqlite.ValueType.Text
Transactions
begin returns a Transaction. commit keeps its changes; dropping it
without commit, including by returning early with ?, rolls them back.
SQLite failures carry SQLite’s own code and message:
import std.db.sqlite
from std.db.sqlite import Begin, Connection, DbError
def transfer(db: &mut Connection, amount: i64) -> Result[(), DbError | OverflowError[i64]]:
mut tx = db.begin(Begin.Immediate)?
mut change = db.prepare("UPDATE accounts SET balance = balance + ? WHERE name = ?")?
change.bind_integer(1, 0 - amount)?
change.bind_text(2, "alice")?
change.run()?
change.bind_integer(1, amount)?
change.bind_text(2, "bob")?
change.run()?
tx.commit()
def main() -> Result[(), AllocError | IoError | DbError | OverflowError[i64]]:
mut db = std.db.sqlite.memory(65536)?
db.execute("CREATE TABLE accounts(name TEXT PRIMARY KEY, balance INTEGER CHECK (balance >= 0))")?
db.execute("INSERT INTO accounts VALUES ('alice', 10), ('bob', 0)")?
transfer(&mut db, 4)?
match transfer(&mut db, 50):
case Ok(_):
print("moved")?
case Err(DbError.Sqlite(error)):
print(error.primary() == std.db.sqlite.constraint, error.message)?
case Err(_):
print("other")?
mut query = db.prepare("SELECT name, balance FROM accounts ORDER BY name")?
while query.step()?:
print(query.text(0)?, query.integer(1)?)?
Ok(())
True CHECK constraint failed: balance >= 0
alice 6
bob 4
The failed transfer left no trace: its transaction rolled back when
transfer returned the error.
For a file, use std.db.sqlite.connect(path, Mode.Create, limit). Another
connection holding a write lock makes a write fail at once with SQLite’s
busy code; set_busy_timeout makes it wait instead. See the
std.db.sqlite reference for every error,
the busy policy, and how connections are shared between threads.
SQLite’s FTS5 full-text search is built in. Applications that need a native
extension, such as vector search, can load a trusted shared library with
Connection.load_extension(path, entrypoint=""). This runs native code in the
application; SQL cannot enable extension loading. See
native extensions and full-text search
for the loading contract and connection lifetime.
Store byte-keyed records
Use std.db.kv when a program needs named maps of byte keys and values, and
ordered scans, without writing SQL. Use std.db.sqlite when SQL queries,
joins or your own database schema are part of the application.
Batch related writes in a transaction. A failed operation propagated by ?
drops the unfinished transaction and rolls the batch back:
import std.db.kv as kv
import std.bytes as buffers
from std.bytes import BytesError
def main() -> Result[(), AllocError | kv.KvError | BytesError | IoError]:
mut store = kv.memory()?
key = buffers.from_text("user:42")?
value = buffers.from_text("Ada")?
mut transaction = store.begin()?
store.put("names", &key, &value)?
transaction.commit()?
drop(transaction)
match store.get("names", &key)?:
case Some(name):
print(name.to_text()?)?
case Nothing:
print("not found")?
mut rows = store.scan_all("names", max_rows=20)?
while True:
match rows.next()?:
case Some(entry):
print(entry.key.to_text()?, entry.value.to_text()?)?
case Nothing:
break
drop(rows)
store.close()?
Ok(())
Ada
user:42 Ada
This example uses a private in-memory store. For persistence, replace
kv.memory() with kv.connect("records.db"); the default creates a new store or
opens an existing KV file. It rejects a database belonging to another schema.
Mode.ReadOnly opens an existing store without allowing writes.
Keys and values need not be text; the example converts them because names
are UTF-8. Every returned value owns its buffers. Scans require an explicit row
limit and copy one row at a time. Finish/drop a scan before modifying the
same Store or ending its transaction. Use scan(table, start, stop, max_rows)
for a byte range with inclusive start and exclusive stop.
The reference explains file compatibility, per-row size limits, lock conflicts and scan lifetimes. The engine comparison explains why this API uses SQLite rather than introducing a redb dependency.
Read and write HTTP messages
An HTTP/1.1 connection carries one message after another. Each message is a
head, which ends at a blank line, and an optional body. A reader must find
where each body ends, or it reads the next request as part of the last one.
std.http does that work over any std.io stream, so you can test it with
bytes in memory before any socket is involved.
import std.bytes
import std.http
import std.io
type MessageStreamError = std.http.HttpError | std.io.StreamError | std.bytes.BytesError
def main() -> Result[(), AllocError | IoError | MessageStreamError]:
wire = "POST /notes HTTP/1.1\r\nHost: example.com\r\nContent-Length: 5\r\n\r\nhelloGET /notes HTTP/1.1\r\nHost: example.com\r\nConnection: close\r\n\r\n"
limits = std.http.limits()
mut conn = std.io.buffered_reader(std.io.cursor(std.bytes.from_text(wire)?), 1024)?
while True:
match std.http.read_request(&mut conn, &limits)?:
case Nothing:
break
case Some(head):
framing = std.http.request_framing(&head)?
mut body = std.http.body_reader(conn, &framing, &limits)?
mut data = std.bytes.empty()?
std.http.read_to_end(&mut body, &mut data)?
print(head.method, head.target, str.repr(&framing)?, str.repr(data.to_text()?)?, std.http.request_keeps_alive(&head))?
conn = std.http.finish(&mut body)?
Ok(())
POST /notes std.http.Framing.Length(5) "hello" True
GET /notes std.http.Framing.Empty "" False
read_request reads one head and returns Nothing when the input ends
between messages. request_framing says how the body is delimited: here by
Content-Length, then not at all. The body reader takes the connection, so
nothing else can read from it until finish hands it back, positioned at the
next request. finish fails if the body was not read to its end. Every read
is bounded by limits(): line length, head size, header count, and body size.
Writing goes the other way. A chunked body is sent in pieces whose total size the writer need not know in advance:
import std.bytes
import std.http
import std.io
from std.http import Header
type MessageStreamError = std.io.StreamError | std.http.HttpError | std.bytes.BytesError
def main() -> Result[(), AllocError | IoError | MessageStreamError]:
mut sink = std.io.cursor(std.bytes.empty()?)
mut reply = std.http.response(200, "OK")?
reply.add("Transfer-Encoding", "chunked")?
std.http.write_response(&mut sink, &reply)?
text = std.bytes.from_text("first second")?
std.http.write_chunk(&mut sink, &text[:6])?
std.http.write_chunk(&mut sink, &text[6:12])?
std.http.write_last_chunk(&mut sink, &list[Header].new()?)?
print(str.repr(sink.data.to_text()?)?)?
mut forged = std.http.response(200, "OK")?
forged.add("Location", "/a\r\nSet-Cookie: x=1")?
print(str.repr(std.http.write_response(&mut sink, &forged))?)?
smuggled = "POST / HTTP/1.1\r\nContent-Length: 4\r\nTransfer-Encoding: chunked\r\n\r\n"
mut conn = std.io.buffered_reader(std.io.cursor(std.bytes.from_text(smuggled)?), 1024)?
match std.http.read_request(&mut conn, &std.http.limits())?:
case Some(head):
print(str.repr(std.http.request_framing(&head))?)?
case Nothing:
print("connection ended before a request")?
Ok(())
"HTTP/1.1 200 OK\r\nTransfer-Encoding: chunked\r\n\r\n6\r\nfirst \r\n6\r\nsecond\r\n0\r\n\r\n"
Result[(), std.http.HttpError].Err(std.http.HttpError.InvalidHeader)
Result[std.http.Framing, std.http.HttpError].Err(std.http.HttpError.ConflictingFraming)
The writer refuses a header value with a line break, which would let text
from a request add headers of its own. The reader refuses a message with both
Content-Length and Transfer-Encoding: two servers could disagree about
where its body ends, which is how request smuggling works. Neither guesses.
The same functions work over a std.net stream wrapped in a BufReader.
Deadlines stay with the socket calls. See
the module reference for every error, response
framing rules, and the body reader’s use as a std.io stream.
Formatting into a reusable writer
An f-string is the short way to build a
str; it compiles to the calls below. Use std.fmt directly when output must
go to a chosen byte writer, reuse storage, or stay within a byte budget. Numeric
widths count bytes and include the sign. Every call is fallible, and the byte
budget includes punctuation and padding.
import std.fmt
def main() -> Result[(), AllocError | IoError | std.fmt.FormatError]:
mut out = std.fmt.string_formatter(80, 8)?
out.text("count=")?
out.integer(12, 4)?
out.text(", label=")?
out.debug_text("one\ntwo")?
print(std.fmt.finish(&out)?)?
Ok(())
count= 12, label="one\ntwo"
Your type chooses separate display and debug methods. Both can be generic over
the output writer; callers need no trait object or format-string parser.
print and f"{x}" call display; str.repr, f"{x!r}" and the elements
of a printed list or other aggregate call debug. std.fmt.to_string and
to_debug call the same methods within a byte budget and depth you choose.
import std.fmt
from std.fmt import Formatter
pub class Label:
name: str
pub def display[W](self, out: &mut Formatter[W]) -> Result[(), FormatError]:
out.text(&self.name)
pub def debug[W](self, out: &mut Formatter[W]) -> Result[(), FormatError]:
out.text("Label(")?
out.debug_text(&self.name)?
out.text(")")
def main() -> Result[(), AllocError | IoError | FormatError]:
label = Label("ready")
print(label)?
print(str.repr(label)?)?
print([Label("a"), Label("b")]?)?
print(std.fmt.to_debug(&label, 80, 8)?)?
Ok(())
ready
Label("ready")
[Label("a"), Label("b")]
Label("ready")
Because these methods can fail, printing such a value returns
Result[(), IoError | FormatError], and str.repr and an f-string with such a
field add FormatError to AllocError. A type without display prints with its debug; one without
either prints its fields.
Enums use the same methods. Match the shared receiver to format a payload without consuming it:
import std.fmt
from std.fmt import Formatter
pub enum ReadError:
Missing
Invalid(i64)
pub def display[W](self, out: &mut Formatter[W]) -> Result[(), FormatError]:
match self:
case ReadError.Missing:
out.text("no reading")
case ReadError.Invalid(line):
out.text("invalid reading on line ")?
out.integer(*line, 0)
def main() -> Result[(), FormatError | IoError]:
error = ReadError.Invalid(12)
print(error)?
print(std.fmt.to_string(&error, 80, 8)?)?
Ok(())
invalid reading on line 12
invalid reading on line 12
Returning the enum from main does not call the method yet: the report still
uses the structural entrypoint report.
Inside such a method, out.format(f"...") writes an f-string straight to the
formatter, with no intermediate str. It returns Result[(), FormatError], and
the formatter’s byte budget and depth limit apply:
import std.fmt
from std.fmt import Formatter
pub class Reading:
line: i64
value: f64
pub def display[W](self, out: &mut Formatter[W]) -> Result[(), FormatError]:
out.format(f"line {self.line}: {self.value:.1f}")
def main() -> Result[(), AllocError | IoError | FormatError]:
reading = Reading(7, 21.25)
print(f"[{reading}] [{reading:>14}]"?)?
mut out = std.fmt.string_formatter(10, 8)?
match out.format(f"{reading}"):
case Err(FormatError.LimitExceeded):
print("kept", std.fmt.finish(&out)?)?
case _:
print("fits")?
Ok(())
[line 7: 21.2] [ line 7: 21.2]
kept line 7: 21
To stream, construct std.fmt.formatter(writer, scratch, max_bytes, max_depth)
with a std.io writer and initialized Bytes scratch of at least
32 bytes. Provision scratch and sink capacity before entering an allocation-free
section. Formatting itself then reuses that memory; user methods remain
responsible for their own allocations. Flush the public out.writer explicitly
when needed. On failure, out.completed() tells you how many bytes were accepted;
do not blindly restart a partially written record.
The reference describes aggregate callbacks, errors, escaping, depth limits and exact costs.
Formatting a bounded report
Use std.report to give a value a human-readable spelling through a writer-generic
formatter. The method borrows its receiver, so an error can be formatted again
or examined afterward.
import std.fmt
import std.report
from std.fmt import Formatter, FormatError
pub enum ReadingError:
Invalid(i64)
pub def format[W](self, output: &mut Formatter[W]) -> Result[(), FormatError]:
match self:
case ReadingError.Invalid(line):
output.text("invalid reading on line ")?
output.integer(*line)
def main() -> Result[(), FormatError | IoError]:
error = ReadingError.Invalid(12)
print(std.report.to_string(&error, 80, 8)?)?
print(std.report.to_string(&error, 80, 8)?)?
Ok(())
invalid reading on line 12
invalid reading on line 12
The method’s [W] lets it write to a string, stderr, or another writer.
The formatter supplies that type, so callers do not pass a type argument.
The public enum and its public format method satisfy the structural
std.fmt.Formattable protocol. A class uses the same visibility and method signature.
std.report.render(&value, &mut output) counts a nesting level and restores it
on success or error. Use it when a formatting method delegates to another value.
An anonymous sum supports the same reporting API when every member satisfies
std.fmt.Formattable. You can keep distinct error types and let the compiler
select the active member’s method:
import std.fmt
import std.report
from std.fmt import Formatter, FormatError
pub enum ReadingError:
Invalid(i64)
pub def format[W](self, output: &mut Formatter[W]) -> Result[(), FormatError]:
match self:
case ReadingError.Invalid(line):
output.text("invalid reading on line ")?
output.integer(*line)
pub enum ConnectionError:
Closed
pub def format[W](self, output: &mut Formatter[W]) -> Result[(), FormatError]:
output.text("connection closed")
def show(error: &(ReadingError | ConnectionError)) -> Result[(), FormatError | IoError]:
print(std.report.to_string(error, 80, 1)?)?
Ok(())
def main() -> Result[(), FormatError | IoError]:
reading: ReadingError | ConnectionError = ReadingError.Invalid(12)
connection: ReadingError | ConnectionError = ConnectionError.Closed
show(&reading)?
show(&connection)?
show(&reading)?
Ok(())
invalid reading on line 12
connection closed
invalid reading on line 12
Dispatch borrows the payload without allocating; string collection still
allocates. The original sum remains usable afterward. render counts one depth
level for the sum, with no extra level for selecting its member. Calling
error.format(&mut output) directly also dispatches, but does not count depth.
All members must have the required public method, even if the value currently
holds a different member. Other protocols are not automatically forwarded.
For stderr, construct std.fmt.stderr_formatter(max_bytes, max_depth) and
call render explicitly. Construction and scalar writes allocate no heap
memory, so simple report methods remain usable when allocation is unavailable.
Add any desired "error: " prefix and final "\n" with output.text yourself;
they count toward the byte limit. User methods can still allocate, so this
guarantee depends on what those methods do.
The collector has the same formatting contract but allocates fallibly:
import std.fmt
import std.report
from std.fmt import FormatError
def main() -> Result[(), FormatError | IoError]:
mut output = std.fmt.string_formatter(4, 0)?
output.text("a")?
match output.text("é🙂"):
case Err(_):
print("too long", output.completed())?
case Ok(_):
pass
output.text("é")?
print(std.fmt.finish(&output)?)?
Ok(())
too long 1
aé
Text checks its whole UTF-8 byte length before writing. Numeric output streams
digits, so a limit can leave part of a number. On stderr failure, inspect
completed() and has_failed(); subsequent writes return OutputClosed without
writing again. A limit error is recoverable, as the example shows.
When main returns Err, its concrete error type selects the reporting path.
A type with this public format method produces a human-readable report with
the prefix error: main returned and a final newline. A sum selects this path
only when all its members support it; other types retain structural reporting.
The automatic reporter allows 16,384 bytes and 64 nesting levels. If formatting
fails after writing part of the message, it appends [report unavailable] and
a newline without trying the method again. Stderr failures stop output.
Use std.fmt for floats, widths,
debug escaping or an arbitrary byte writer. The
reference lists the complete reporting API.
Generic code and protocols
Use type parameters when several types need the same algorithm or data shape. These lessons build on core functions, borrowing, classes, and enums. Recursive data is an optional extension at the end.
- Replace duplicated functions with a type parameter
- Infer or choose a specialization
- Requiring methods with a protocol
- Parameterize a class
- Parameterize an enum
- Parameterize a protocol
- Store different implementations of a protocol
- Give a method its own type parameters
- Keep a numeric bound in the type
- Calculate with quantities
- Build recursive data
- Use values through a box
- Store recursive children in a list
- Remove a child and clean up a recursive structure
- Traverse recursive data through a borrow
- Reborrow while following a recursive structure
- Preserve a chain when insertion fails
- Represent parent links with node IDs
Replace duplicated functions with a type parameter
Suppose the same operation is needed for two integer widths. Separate functions work, but repeat both the implementation and its interface:
def keep_small(value: u8) -> u8:
value
def keep_large(value: i64) -> i64:
value
def main() -> Result[(), IoError]:
print(keep_small(7u8))?
print(keep_large(9))?
Ok(())
7
9
A type parameter lets one declaration accept either width while keeping the argument and result types the same.
def identity[T](value: T) -> T:
value
def sum_to[T: IntType](stop: T) -> Result[T, OverflowError[T]]:
mut total: T = 0
for n in range[T](stop):
total = total + n
Ok(total)
def first[T](values: &list[T]) -> &T:
&values[0]
def main() -> Result[(), AllocError | IoError | OverflowError[u16]]:
print(sum_to(5u16)?)?
values = identity([3, 4]?)
print(first(&values))?
print(values)?
Ok(())
10
3
[3, 4]
Declare type parameters after the function name. Calls infer them from the
argument types, or you can supply them explicitly before the call.
Ownership still follows the concrete type: identity(values) transfers the
list, while a function taking &T borrows its argument.
IntType restricts a parameter to integer types, and FloatType to f32 and
f64. CopyType accepts any type whose values copy, such as numbers, str,
and tuples of them, and rejects lists and other values that move; see
CopyType. Unconstrained T is useful
when the body only moves, borrows, or uses operations supported by the chosen
concrete type. Each specialization is checked against those operations.
A type parameter can also stand for unit () inside a Result or Option.
This lets one helper handle both operations that return a value and operations
that only succeed or fail:
def keep_result[T, E](result: Result[T, E]) -> Result[T, E]:
result
def main() -> Result[(), AllocError | IoError]:
written = print("saved")
keep_result(written)?
Ok(())
saved
Here T is inferred as (). You can also write an explicit unit type argument,
such as keep_result[(), IoError](written). Unit still cannot be a direct
function parameter, so identity[()](()) is rejected.
Require collection keys
Use K: KeyType when a generic algorithm needs dictionary keys or set members.
It accepts integers, booleans, strings, bytes, and supported tuples, classes, enums and anonymous sums, including custom cleanup. The compiler checks all enum payloads; no hashing method is required.
enum Label:
Id(i64)
Name(str)
def frequencies[K: KeyType](values: list[K]) -> Result[dict[K, i64], AllocError | OverflowError[i64]]:
mut counts: dict[K, i64] = {}?
for value in values:
*counts.get_or_insert(value, 0)? += 1
Ok(counts)
def main() -> Result[(), AllocError | IoError | OverflowError[i64]]:
values = [Label.Id(7), Label.Name("7"), Label.Id(7)]?
counts = frequencies(values)?
print(counts[Label.Id(7)], counts[Label.Name("7")])?
Ok(())
2 1
The loop consumes the list and moves each key into get_or_insert, which
drops it when the key is already present. This also works for keys with custom
cleanup, which cannot be copied. Dictionary insertion and the count’s addition
remain fallible. Floats do not satisfy KeyType; see the
constraint reference.
Infer or choose a specialization
Inference matches the signature against the argument types. In first(&values),
the parameter is &list[T] and the argument is &list[i64], so T is i64.
When multiple arguments determine the same parameter, their types must agree.
Unsuffixed literals in generic positions use their normal defaults (i64 or
f64); another argument does not change them. Keep explicit type arguments when
you want them to guide literals: sum_to[u16](5) is equivalent to sum_to(5u16).
Types must be determined from arguments, not the expected return type. For
example, this function needs empty[u8]() because it has no input mentioning T:
def empty[T]() -> Result[list[T], AllocError]:
[]
def main() -> Result[(), AllocError | IoError]:
values = empty[u8]()?
print(values)?
Ok(())
[]
An annotation on the receiving binding does not change that rule:
def empty[T]() -> Result[list[T], AllocError]:
[]
def main() -> Result[(), AllocError | IoError]:
values: list[u8] = empty()?
Ok(())
cannot infer type parameter `T` for `empty` from its arguments
Calls with the same concrete types share one compiled specialization, including recursive calls. Independent uses can specialize many different helpers, or the same helper for many types. The compiler separately limits how deeply one new specialization requires another, how many one generic function has, and how many are created in total. These are compile-time resource limits; generic calls add no runtime allocation. See specialization limits for the current bounds.
Requiring methods with a protocol
A protocol names the methods a generic algorithm needs. A class or enum satisfies it by having matching methods; no inheritance or registration is required. Importing a protocol does not add methods to a type.
protocol Readable:
def read(self) -> str:
pass
class Message:
text: str
def read(self) -> str:
self.text
def read_message[T: Readable](source: &T) -> str:
source.read()
def main() -> Result[(), IoError]:
message = Message("hello")
print(read_message(&message))?
Ok(())
hello
Here &message determines T = Message; the compiler then checks the Readable
requirements. Writing read_message[Message](&message) explicitly also works.
self means a shared receiver. Write self: &mut ProtocolName when a required
method mutates the receiver. Method parameters, return types, and receiver
borrowing must match exactly. All required methods are checked at specialization,
including methods the generic function does not happen to use. Across modules,
the class methods must be visible to the generic function’s defining module.
That means both pub class Message and pub def read, even when the caller is
in Message’s own module. The caller does not lend private access to the helper.
If only the class is private, the call diagnostic asks you to declare it
pub class; private methods still need their own pub def.
For example, save this helper as reader.plenty:
pub protocol Readable:
def read(self) -> str:
pass
pub def read_message[T: Readable](source: &T) -> str:
source.read()
This application needs pub class Message, as well as its public method:
import reader
class Message:
pub def read(self) -> str:
"hello"
def main() -> ():
message = Message()
drop(reader.read_message(&message))
class `Message` is private to its defining module; declare it `pub class`
Protocols constrain class and enum type arguments; they cannot be stored as
values. Use source: &T with T: Readable, not source: Readable.
Parameterize a class
A generic class shares one declaration across concrete field types. Its methods can use the class’s type parameters.
class Cell[T]:
value: T
def get(self) -> T:
self.value
def replace(self: &mut Cell[T], value: T) -> ():
self.value = value
def read[T](cell: &Cell[T]) -> T:
cell.get()
def main() -> Result[(), IoError]:
mut count = Cell(7u8)
count.replace(9)
print(read(&count))?
text = Cell("hello")
print(text.get())?
Ok(())
9
hello
Constructors infer their parameters: Cell(7u8) and Cell[u8](7) have the
same type. With Cell(7), the unsuffixed integer defaults to i64.
This get method works for integers and immutable strings, which can be read
from a shared field. An owned list needs an explicit copy or a reference:
returning it would move it out of a borrowed field. A Cell can hold a list,
but its get cannot be called:
class Cell[T]:
value: T
def get(self) -> T:
self.value
def main() -> Result[(), AllocError | IoError]:
numbers = Cell([1, 2]?)
first = numbers.get()
Ok(())
lesson.plenty:9:13: `Cell[list[i64]].get` copies `T`, but `list[i64]` cannot be copied
lesson.plenty:5:9: note: in instantiation `Cell[list[i64]].get`: cannot move out of a field
A generic method is checked once for each set of type arguments it is used
with, not where it is written. Without the get call, the program is accepted.
The error is reported at the line of your code that first required
Cell[list[i64]].get, here the method call, and names the type argument that
the method cannot accept. The note below it gives the line in the definition
and the reason.
That wording follows the value: here self.value is declared as T. An
unrelated list created inside the method does not become a T value just
because the caller also chooses a list for T. For failures without a clear
parameter origin, the compiler instead reports that the instance’s definition
is rejected, followed by the operation’s error. See
instance diagnostics
for the attribution rules.
When you write a generic class for others to use, plenty --check --check-all-methods FILE also checks the methods your program never calls,
for every instance it creates of a generic type it declares. The
generics reference
describes the check.
Parameterize an enum
Use a type parameter when the same alternatives should carry different kinds of
values. Choose the concrete payload type in brackets. Here, T: IntType accepts
any supported sized integer. Omit the constraint to allow other stored types.
enum Choice[T: IntType]:
Empty
Value(T)
def main() -> Result[(), IoError]:
selected = Choice[u8].Value(7)
match selected:
case Choice.Empty:
print("empty")?
case Choice.Value(value):
print(value)?
Ok(())
7
A constructor names the instance, as in Choice[u8].Value(7). A pattern names
only the enum, as in case Choice.Value(value): the matched value already
fixes T, so a pattern takes no type arguments.
Parameterize a protocol
A protocol can describe the same operation for several result types. Classes satisfy it by providing matching methods.
protocol Readable[T]:
def read(self) -> T:
pass
class Cell[T]:
value: T
def read(self) -> T:
self.value
def read[T, R: Readable[T]](source: &R) -> T:
source.read()
def main() -> Result[(), IoError]:
cell = Cell(7u8)
print(read(&cell))?
Ok(())
7
The argument determines R = Cell[u8]; that class’s read signature determines
T = u8. Its return type and receiver borrowing must match the protocol.
Explicit read[u8, Cell[u8]](&cell) also works. Conflicting type evidence is
an error, rather than a reason to widen integers.
Store different implementations of a protocol
An enum wrapper works well when your program knows all the alternatives. A protocol box lets implementations be added independently: the collection only names the operations it needs.
protocol Counter:
def current(self) -> i64:
pass
def increment(self: &mut Counter) -> Result[(), OverflowError[i64]]:
pass
class Count:
value: i64
def current(self) -> i64:
self.value
def increment(self: &mut Count) -> Result[(), OverflowError[i64]]:
self.value += 1
Ok(())
enum Flag:
Off
On
def current(self) -> i64:
match self:
case Flag.Off:
0
case Flag.On:
1
def increment(self: &mut Flag) -> Result[(), OverflowError[i64]]:
*self = Flag.On
Ok(())
def main() -> Result[(), AllocError | IoError | OverflowError[i64]]:
mut items: list[Box[Counter]] = []?
items.append(Box[Counter](Count(10))?)?
items.append(Box[Counter](Flag.Off)?)?
for item in &mut items:
item.increment()?
print(item.current())?
Ok(())
11
1
Each Box[Counter](value)? moves its input into one heap allocation. The list
stores owners with a common interface. Its mutable loop borrows each owner, so
increment and current operate on the same instance. No match or cast is
needed. Use &items for a loop that only calls shared methods.
The protocol fixes each method’s contract. Count.increment can overflow, so
increment returns Result[(), OverflowError[i64]], and Flag declares the
same result even though it never fails.
The Box boundary is explicit: list[Counter] is not a stored value type, and
Box(Count(10))? preserves the concrete Count type. Allocation failure drops
the consumed input. Appending can also fail and consumes its new box under the
ordinary list contract.
Dictionary values and generic readers
The same owner type works as a dictionary value and through a protocol bound:
protocol Label:
def label(self) -> str:
pass
class Message:
text: str
def label(self) -> str:
self.text
enum Status:
Ready
def label(self) -> str:
return "ready"
def read[T: Label](value: &T) -> str:
value.label()
def main() -> Result[(), AllocError | IoError]:
items: dict[str, Box[Label]] = {
"message": Box[Label](Message("hello"))?,
"status": Box[Label](Status.Ready)?,
}?
for name, item in items.items():
print(name, read(item))?
Ok(())
message hello
status ready
Borrowing avoids copying the boxes. Their concrete payloads are destroyed when their owners leave scope, including any custom cleanup.
A box exposes only its protocol’s methods. This initial interface supports shared and mutable receivers, fixed protocol arguments, ordinary parameters, and owned results. Methods with their own type parameters, consuming receivers, or returned borrows cannot be boxed. Protocol boxes cannot be copied, used as set elements or dictionary keys, or transferred to worker threads. Payloads containing channel senders or receivers also need concrete owners: erasing them would hide endpoints from the compiler’s checks for waits that could deadlock. See the reference for the complete limits.
Give a method its own type parameters
A method may declare type parameters of its own, in addition to those on its class. Arguments determine them just as they do for generic functions. A parameter that appears only in the result must be chosen explicitly:
class Number:
value: i64
def convert[T: IntType](self) -> T:
T(self.value)
def main() -> Result[(), IoError]:
number = Number(7)
small: u8 = number.convert[u8]()
print(small)?
Ok(())
7
Method parameters cannot shadow class parameters. Constructors and destructors use only the class parameters. See the functions-as-values path for a generic method that accepts a callback.
Keep a numeric bound in the type
A configuration value often needs a fixed interval. Name that interval with
Bounded, then validate computed input once before passing it around:
type RetryCount = Bounded[i64, 0, 8]
def describe(retries: RetryCount) -> Result[(), IoError]:
print("Retries:", retries.get())
def main() -> Result[(), RangeError | IoError]:
describe(3)?
configured = 5
retries = RetryCount.try_from(configured)?
describe(retries)?
print(RetryCount.try_from(9).is_err())?
Ok(())
Retries: 3
Retries: 5
True
Both ends are included. The literal 3 is checked at compile time because the
parameter expects RetryCount. A computed integer goes through try_from,
which returns RangeError.OutOfRange when it does not fit. .get() extracts the
integer for calculations. A literal 9 passed directly to describe would be
a compile error.
Abstract over the bounds
The arguments 0 and 8 are compile-time values, not types. A generic function
can keep them as const parameters:
def interval[T: IntType, const lo: T, const hi: T](value: &Bounded[T, lo, hi]) -> (T, T):
(lo, hi)
def main() -> Result[(), IoError]:
retries: Bounded[i64, 0, 8] = 3
percentage: Bounded[u8, 0, 100] = 75
print(interval(&retries), interval(&percentage))?
Ok(())
(0, 8) (0, 100)
The argument’s type supplies T, lo and hi; no runtime integer becomes a
specialization argument. const lo: T uses the earlier T: IntType, so a u8
interval has u8 bounds. The values are available inside the body without
storing them alongside each integer.
User classes, enums and methods can also declare const parameters. Use them when the value belongs in type identity. A setting read from a file normally stays a runtime value: it cannot select a new type at runtime. The value-parameter reference gives argument restrictions, and bounded integers describes conversion and value behavior.
Calculate with quantities
Use Quantity[T, D] for a physical value whose dimension should be checked
in arithmetic. T is its numeric storage type and D is a compile-time
dimension. The std.units module provides the
SI dimensions, units and constants:
import std.units as u
from std.units import Velocity, m, km, s, minute
def double[D: Dimension](value: Quantity[f64, D]) -> Quantity[f64, D]:
value + value
def main() -> Result[(), IoError]:
distance = 1500.0 * m
elapsed = 2.0 * minute
speed: Quantity[f64, Velocity] = distance / elapsed
print(double(distance).to(km), "km")?
print(speed, speed.to(km / u.hour), "km/h")?
Ok(())
3.0 km
12.5 m/s 45.0 km/h
u.Length and u.Time are distinct base dimensions. Velocity is an alias
for Length / Time. Dividing the distance by the elapsed time computes this
dimension, so the speed annotation checks successfully. Adding a time to a
length would be a compile error. When a dimension error involves Velocity,
the diagnostic includes a note naming Velocity and its normal form,
std.units.Length * Pow[std.units.Time, -1]. This connects the normalized
dimension to the name used in your program.
Units are constants. km is a length whose stored value is 1000.0, and
2.0 * minute stores 120.0 seconds. Multiplying a number by a unit
constructs a quantity, and dividing one unit by another, as in km / u.hour,
gives another unit.
double infers its dimension parameter from its argument. The same function
works for a length, a time or a velocity. Dimension arguments add no runtime
fields: a quantity uses its numeric storage type’s native representation.
Construction and get add no numeric conversion, and float to performs the
same division as plain numbers. A dimension itself has no values:
length: u.Length is invalid.
.to(km) divides by the unit’s stored value and returns a plain f64.
.get() reads the coherent value directly. Both methods remove the dimension from the result, so keep quantities until the boundary
where you need a number. Printing a quantity includes its coherent-unit symbol.
The explicit "km" label accompanies the plain number returned by .to(km);
printing double(distance) directly would show 3000.0 m.
Use an f-string to choose numeric precision: f"{speed:.2f}"? produces
"12.50 m/s". As with other f-strings, include AllocError in the function’s
error type when propagating its allocation failure.
Scalars only scale quantities. Write 0.0 * m for a zero length; `distance
- 0.0
anddistance > 0.0are errors. Dividing a length by a length producesQuantity[f64, One]; use.get()` to obtain its dimensionless number.
std.units also has dimension-aware functions. u.sqrt of an area returns a
length, u.sum adds a list of quantities, and u.sin takes an angle such as
30.0 * u.degree. Temperatures in Celsius or Fahrenheit convert at the
boundary with u.from_celsius and u.to_celsius, because their zero is
offset.
Declare your own dimensions
Declare a base dimension for a quantity outside SI, or to keep two kinds of value apart that SI would give the same dimension. Give each base its coherent-unit symbol:
dimension Pixel("px")
dimension Point("pt")
dimension PixelDensity = Pixel / Point
const px: Quantity[f64, Pixel] = Quantity[f64, Pixel](1.0)
const pt: Quantity[f64, Point] = Quantity[f64, Point](1.0)
const retina: Quantity[f64, PixelDensity] = 2.0 * px / pt
def main() -> Result[(), IoError]:
width = 320.0 * pt
print(width * retina)?
Ok(())
640.0 px
Unit constants can be shared across functions or exported from a module with
pub const, as std.units does. Their initializers are evaluated at compile
time, including unit scaling and arithmetic between constants. The
constructor takes the value in the coherent unit.
Dimension expressions also support products, integer powers and exact roots.
Root[Pow[Length, 2], 2] is Length; fractional dimensions are rejected.
In a generic function’s parameter types, use concrete dimensions or bare
parameters. Expressions using parameters belong in return types or bodies.
See std.units for its units, constants and
functions, dimensions for algebra and
visibility, and quantities for
integer/domain storage, conversions, checked arithmetic and current limits.
Build recursive data
Chains and trees contain values of their own type. A directly embedded value
would have no finite size, so put the recursive field in a Box or collection.
A box owns one value on the heap.
This chain ends with Empty. Each Link owns a value and a box holding the
rest of the chain:
enum Chain[T]:
Empty
Link(T, Box[Chain[T]])
def total(chain: Chain[i64]) -> Result[i64, OverflowError[i64]]:
mut remaining = chain
mut answer = 0
while True:
match remaining:
case Chain.Empty:
break
case Chain.Link(value, rest):
answer = answer + value
remaining = rest
Ok(answer)
def main() -> Result[(), AllocError | IoError | OverflowError[i64]]:
mut chain = Chain[i64].Empty
for value in range(1, 4):
chain = Chain[i64].Link(value, Box(chain)?)
print(total(chain)?)?
Ok(())
6
Box(value) moves the value onto the heap and returns an allocation result.
Constructing the Link itself allocates nothing.
When a value of type T is required, a Box[T] moves its content out and frees
the box, without allocating. Here, remaining = rest expects Chain[i64], so
it unboxes rest. The conversion also works for arguments, returns, annotated
bindings, constructor fields, and collection elements. total consumes its chain;
the caller cannot use it afterward.
A boxed class’s fields and methods are reached as b.field and b.method(),
and where a function expects &T, you can pass a &Box[T]. Where no type is
required, a box stays a box: other = b moves the box itself. Operators and
conditions do not convert either. Write *b there to move the content out, or
&*b and &mut *b to borrow it.
Without the box, the compiler rejects the declaration:
enum Chain:
Empty
Link(i64, Chain)
def main() -> ():
pass
`Chain` contains itself and would have infinite size; store the recursive field in a `Box`, list, set, or dict
Use values through a box
Call collection and string methods through a box, including nested boxes.
Methods borrow its content; mutation requires a mut binding or an exclusive
reference:
def main() -> Result[(), AllocError | IoError]:
mut items = Box(Box([1, 2]?)?)?
items.append(3)?
match items.get(2):
case Some(value):
print(value)?
case Nothing:
print("missing item")?
text = Box(" hello ")?
print(text.strip()?)?
print(text.startswith(" "))?
Ok(())
3
hello
True
Read-only methods also work on temporary boxes. Bind a box with mut before
calling a mutating method.
The recursive-data lesson explains when a box transfers its contents and when it keeps ownership of them.
Store recursive children in a list
A list of children provides the indirection a recursive class needs; each
child needs no additional box. Save this module as forest.plenty:
pub class Node[T]:
pub value: T
pub children: list[Node[T]]
import forest
def main() -> Result[(), AllocError | IoError]:
mut root = forest.Node[i64](1, [forest.Node[i64](2, []?)]?)
root.children[0].value = 9
print(root.value)?
print(root.children[0].value)?
Ok(())
1
9
An empty children list terminates the tree. For a single optional child, use a
field such as next: Option[Box[Node]] and terminate it with Nothing.
Remove a child and clean up a recursive structure
To take an owned child out of a mutable class, leave a replacement behind.
replace(node.next, Nothing) returns the old tail without cloning or allocating:
class Link:
value: i64
next: Option[Box[Link]]
def main() -> Result[(), AllocError | IoError]:
mut chain = Some(Box(Link(1, Some(Box(Link(2, Nothing))?)))?)
while True:
match chain:
case Some(link):
mut node = link
print(node.value)?
chain = replace(node.next, Nothing)
case Nothing:
break
Ok(())
1
2
Replacing node.next with Nothing leaves that field empty when node drops.
Use another tail as the replacement to relink nodes. replace requires a mutable
field and evaluates the replacement before any destination indices.
Automatic cleanup can drop a deep boxed chain without growing the native stack. A recursive traversal you write can still overflow it; use a loop for unbounded depth.
To inspect a structure while keeping it, traverse it through a borrow.
Moving individual owned fields out without replacing them remains unsupported.
Automatic copy, equality, print, and str.repr also reject recursive data,
including containers holding it. Print selected scalar fields as above, or write
an explicit traversal.
Optional: traverse recursive data
Traverse a chain without consuming it by matching a borrow. Some(next) below
binds next as &Box[links.Node[i64]], which can be passed where a reference to
the node is expected.
pub class Node[T]:
pub value: T
pub next: Option[Box[Node[T]]]
import links
def total(node: &links.Node[i64]) -> Result[i64, OverflowError[i64]]:
match &node.next:
case Some(next):
Ok(node.value + total(next)?)
case Nothing:
Ok(node.value)
def increment(node: &mut links.Node[i64]) -> Result[(), OverflowError[i64]]:
node.value = node.value + 1
match &mut node.next:
case Some(next):
return increment(next)
case Nothing:
pass
Ok(())
def main() -> Result[(), AllocError | IoError | OverflowError[i64]]:
mut chain = links.Node[i64](3, Some(Box(links.Node[i64](6, Nothing))?))
print(total(&chain)?)?
increment(&mut chain)?
print(total(&chain)?)?
print(chain.value)?
Ok(())
9
11
4
match &mut value binds mutable payload references, allowing increment to
update each node while its caller keeps the chain.
increment ends with a tail call through a reference borrowed from its parameter,
so its stack does not grow. total adds after its recursive call returns, so it
uses native stack space for every node.
A loop avoids that. Declare a reference binding with mut and assign it the
next node. Such a binding may only be assigned a reference borrowed from itself,
which next is: it comes from match &node.next.
class Node:
value: i64
next: Option[Box[Node]]
def total(head: &Node) -> Result[i64, OverflowError[i64]]:
mut sum = 0
mut node = head
while True:
sum = sum + node.value
match &node.next:
case Some(next):
node = next
case Nothing:
break
Ok(sum)
def increment(head: &mut Node) -> Result[(), OverflowError[i64]]:
mut node = head
while True:
node.value = node.value + 1
match &mut node.next:
case Some(next):
node = next
case Nothing:
break
Ok(())
def main() -> Result[(), AllocError | IoError | OverflowError[i64]]:
mut chain = Node(3, Some(Box(Node(6, Nothing))?))
increment(&mut chain)?
print(total(&chain)?)?
Ok(())
11
The chain stays borrowed while node is used. The loop’s stack usage is
independent of chain length.
Walk recursive data with a reference
A reference binding declared with mut can advance through a recursive
structure by borrowing each next step from itself:
class Folder:
name: str
inside: list[Folder]
def innermost(top: &Folder) -> str:
mut folder = top
while len(folder.inside) > 0:
folder = &folder.inside[0]
folder.name
def main() -> Result[(), AllocError | IoError]:
notes = Folder("notes", []?)
work = Folder("work", [notes]?)
home = Folder("home", [work]?)
print(innermost(&home))?
Ok(())
notes
folder starts at top. Each &folder.inside[0] is borrowed from folder, so
the assignment is allowed. home stays borrowed until folder’s last use.
The binding can also be pointed at another place within the value it first borrowed, without going through itself. This retargets it: the old loan ends at the assignment. A search can remember its best element this way, without keeping an index:
def longest(words: &list[str]) -> &str:
mut best = &words[0]
for i in range(1, len(words)):
if words[i].byte_len() > best.byte_len():
best = &words[i]
best
def main() -> Result[(), AllocError | IoError]:
words = ["fig", "banana", "kiwi"]?
print(longest(&words))?
Ok(())
banana
best first borrows from words, so &words[i] is a valid new target. See
reference binding reassignment
for the full rule. Pointing it at a different value is rejected, even one of the
same type:
def main() -> Result[(), AllocError | IoError]:
first = [1, 2]?
second = [3]?
mut view = &first
view = &second
print(view)?
Ok(())
reference binding `view` can be reassigned only to a reference into the owner it first borrowed
Matching borrowed values also leaves the owner in place while inspecting its contents.
The same descendant-cursor pattern can lend exclusive access while updating each node. See the complete mutable chain walk. For branching traversal, a stack of references is currently unsupported; an arena with a worklist of IDs is an alternative.
Preserve a chain when insertion fails
A borrowed collection should usually keep its existing contents when insertion fails. Allocate the new node before detaching the old head:
class Node:
value: i64
next: Option[Box[Node]]
class Chain:
head: Option[Box[Node]]
def push(chain: &mut Chain, value: i64) -> Result[(), AllocError]:
mut node = Box(Node(value, Nothing))?
node.next = replace(chain.head, Nothing)
chain.head = Some(node)
Ok(())
def main() -> Result[(), AllocError | IoError]:
mut chain = Chain(Nothing)
push(&mut chain, 10)?
push(&mut chain, 20)?
match &chain.head:
case Some(head):
print(head.value)?
case Nothing:
print("empty")?
Ok(())
20
The only allocation is Box(...). After it succeeds, replace transfers the
old head and leaves Nothing behind; linking the nodes cannot fail or allocate.
If allocation fails, the existing chain has not changed.
Putting replace(chain.head, Nothing) inside the Box argument would detach
the chain first. Failed box allocation destroys its input, including that old
chain. Memory safety does not imply that a failed operation preserves data.
For a generic payload, also decide whether failure should destroy the incoming
value or return it to the caller.
The earlier consuming construction example abandons the whole local computation on failure. This example has a stronger contract because the caller retains the collection. See the complete linked-stack operations.
Represent parent links with node IDs
When children need to identify their parent, one list can own all the nodes. Links then contain indices, not references or owning boxes. This representation is often called an arena. Moving the list’s storage during growth does not change an index.
This append-only example stores children as IDs too:
class Node:
name: str
parent: Option[i64]
children: list[i64]
def main() -> Result[(), AllocError | IoError]:
mut nodes: list[Node] = []?
nodes.append(Node("root", Nothing, []?))?
nodes.append(Node("leaf", Some(0), []?))?
nodes[0].children.append(1)?
child = nodes[0].children[0]
match nodes[child].parent:
case Some(parent):
print(nodes[parent].name)?
case Nothing:
print("no parent")?
mut pending = [0]?
while True:
match pending.pop():
case Some(id):
print(nodes[id].name)?
for child_id in &nodes[id].children:
pending.append(*child_id)?
case Nothing:
break
Ok(())
root
root
leaf
The explicit worklist avoids recursive function calls. It owns integers, so it does not need to store references, which Plenty currently rejects. Its growth is fallible and its memory use depends on the tree’s shape. A graph with cycles needs visited-node tracking as well.
Keep this list append-only while IDs are live. Removing an element from the middle, sorting, or swapping elements changes which node an index denotes. Removing a graph edge does not destroy its target: the arena owns that node until explicitly removed or the whole arena is dropped. If IDs come from outside this controlled example, check their range before indexing.
For reusable slots, tombstones preserve indices and generation checks can
reject stale IDs after reuse. Those invariants belong in a reusable abstraction;
raw integers do not enforce them. The arena proposal
contains a tested prototype, not an available std.arena module.
This example exits and drops the arena if construction fails. An API that mutates a caller’s existing graph must reserve needed storage before changing parent/child links, or explicitly roll back partial changes. See failure-preserving insertion.
Functions as values
Pass a function when its caller should choose an operation, or store one for later. The first lessons need only the core path; generic signatures and records also use type parameters and constraints.
- Pass functions as values
- Anonymous functions
- Capture data and private state
- Accept functions and closures through one API
- Borrow in a closure
- Return a closure
- Generic function values
- Transform a cell with a generic method
- Store stateful callbacks
- Store owned closures
- Reduce a borrowed collection with a pipeline
- Transform a stream of values
- Keep callbacks in tuples and lists
- Name stored callbacks
- Store different captured callbacks
- Put a callback in an enum
- Consume a capture
- Consume a callback
- Consume owned data while borrowing local state
- Store consuming jobs
Pass functions as values
Use a function’s name without parentheses to pass it as a value. A Callable
annotation specifies its parameter types and return type. The value stores
only a code address. To accept a callback with captured state too, use a
generic callback parameter.
def double(value: i64) -> Result[i64, OverflowError[i64]]:
Ok(value * 2)
def apply(operation: Callable[[i64], Result[i64, OverflowError[i64]]], value: i64) -> Result[i64, OverflowError[i64]]:
operation(value)
def main() -> Result[(), IoError | OverflowError[i64]]:
operation = double
print(operation(6)?)?
print(apply(double, 9)?)?
Ok(())
12
18
The callback’s own return type includes OverflowError[i64]; apply
forwards that public Result and main propagates it with ?. A caller
does not change a callback’s arithmetic contract. See
checked arithmetic.
Function values can also be returned and reassigned through mut bindings.
Copying a function value copies its code address; no allocation is needed.
You can call a value selected from a collection directly:
def increment(value: i64) -> Result[i64, OverflowError[i64]]:
Ok(value + 1)
def main() -> Result[(), AllocError | IoError | OverflowError[i64]]:
handlers = [increment]?
index = 0
print(handlers[index](41)?)?
Ok(())
42
A callable can also borrow an argument and return a fallible unit result:
def increment(value: &mut i64) -> Result[(), OverflowError[i64]]:
*value = *value + 1
Ok(())
def main() -> Result[(), IoError | OverflowError[i64]]:
update: Callable[[&mut i64], Result[(), OverflowError[i64]]] = increment
mut value = 41
update(&mut value)?
print(value)?
Ok(())
42
Class constructors and payload-bearing enum variants can be passed the same way. Use the constructor directly when a callback only wraps a value:
class Reading:
value: i64
def main() -> Result[(), AllocError | IoError]:
readings: list[Reading] = iter(range(3)).map(Reading).collect()?
wrap: Callable[[i64], Option[i64]] = Some
print(readings, wrap(7))?
Ok(())
[Reading(value=0), Reading(value=1), Reading(value=2)] Option[i64].Some(7)
For a generic constructor, an annotation supplies the instance, as for wrap
above. Without that context, name it explicitly: Option[i64].Some.
Constructor values take every positional argument and retain any initializer
failure in their result. See constructor rules.
Anonymous functions
Use def without a name to create a function value. Parameters and the return
value need explicit types. Inside brackets, such as a call’s arguments, the body
is one expression and its value is returned:
def apply(f: Callable[[i64], Result[i64, OverflowError[i64]]], value: i64) -> Result[i64, OverflowError[i64]]:
f(value)
def main() -> Result[(), AllocError | IoError | OverflowError[i64]]:
print(apply(def(x: i64) -> Result[i64, OverflowError[i64]]: Ok(x * 2), 4)?)?
print(apply(
def(x: i64) -> Result[i64, OverflowError[i64]]: Ok(x + 1 if x > 0 else 0),
4,
)?)?
Ok(())
8
5
The body ends at the comma or closing bracket that ends the argument. Each
function has type Callable[[i64], Result[i64, OverflowError[i64]]] and can be
stored, returned, or passed to a callback parameter.
Outside brackets, the body is an indented block with the same rules as a named function:
def main() -> Result[(), IoError | OverflowError[i64]]:
transform = def(value: i64) -> Result[i64, OverflowError[i64]]:
adjusted = value * 2
if adjusted > 10:
return Ok(10)
Ok(adjusted + 1)
print(transform(3)?)?
print(transform(8)?)?
Ok(())
7
10
The position decides the form. A function with statements cannot be written
inside brackets; give it an indented body and a name first, as transform has:
def apply(f: Callable[[i64], Result[i64, OverflowError[i64]]], value: i64) -> Result[i64, OverflowError[i64]]:
f(value)
def main() -> ():
apply(def(x: i64) -> Result[i64, OverflowError[i64]]: return Ok(x * 2), 4)
inside brackets an anonymous function body is one expression, and its value is returned; remove `return`
Parentheses are brackets, so a one-expression function can still be bound:
double = (def(x: i64) -> Result[i64, OverflowError[i64]]: Ok(x * 2)). Python’s lambda is not available;
the compiler names the def spelling instead.
Capture data and private state
When an operation needs more than its call arguments, let its function value own that extra data. Use an explicit list to capture surrounding bindings:
def main() -> Result[(), AllocError | IoError | OverflowError[i64]]:
offset = 4
values = [10, 20]?
adjust = def [offset, values](index: i64) -> Result[i64, OverflowError[i64]]:
Ok(values[index] + offset)
print(adjust(0)?)?
print(adjust(1)?)?
Ok(())
14
24
values moves into the closure; the scalar offset copies. Each call borrows
the captures, so the list remains available for subsequent calls and drops with
the closure. Its environment is stored inline and has a different type from the
code-only Callable. A reusable closure cannot move captures out; borrow or
explicitly copy them when an independent owner is needed.
Use mut in the capture list for private state. A stateful closure also needs
a mutable binding to call:
def main() -> Result[(), IoError | OverflowError[i64]]:
count = 0
mut next = def [mut count]() -> Result[i64, OverflowError[i64]]:
count = count + 1
Ok(count)
print(next()?)?
print(next()?)?
print(count)?
Ok(())
1
2
0
The captured scalar changes independently of the original count.
Accept functions and closures through one API
Constrain a type parameter by its call signature to accept named functions and captured closures through one parameter. Borrow the callback when its caller should keep it:
def apply[T, E, F: Callable[[T], Result[T, E]]](f: &F, value: T) -> Result[T, E]:
f(value)
def double(value: i64) -> Result[i64, OverflowError[i64]]:
Ok(value * 2)
def main() -> Result[(), IoError | OverflowError[i64] | OverflowError[u8]]:
offset = 2u8
add = def [offset](value: u8) -> Result[u8, OverflowError[u8]]:
Ok(offset + value)
print(apply(&double, 3)?)?
print(apply(&add, 3u8)?)?
Ok(())
6
5
A short callback can be written in the call. An anonymous function needs
parentheses to be the operand of &:
def apply[T, E, F: Callable[[T], Result[T, E]]](f: &F, value: T) -> Result[T, E]:
f(value)
def main() -> Result[(), IoError | OverflowError[i64]]:
offset = 2
print(apply(&(def [offset](value: i64) -> Result[i64, OverflowError[i64]]: Ok(offset + value)), 3)?)?
print(apply(&(def(value: i64) -> Result[i64, OverflowError[i64]]: Ok(value * value)), 3)?)?
Ok(())
5
9
F preserves the callback’s concrete environment type. Both arguments agree
on T, the input and successful output type; E preserves the callback’s
error type. The caller handles the returned Result normally. Use &mut F
for a callback that changes its captured state. Borrowed captures stay protected throughout
the call.
Borrow a named function directly with &double, or with &mut double for a
&mut F parameter; a separate binding is optional. A function has no state, so
each &mut borrow takes its own copy of the code address.
A concrete parameter such as f: Callable[[i64], i64] holds only a function’s
code address. The constraint F: Callable[[i64], i64] instead checks whether
the actual type can be called with that signature; its captured state remains
inline. Adding & to the concrete Callable type does not make it accept
closures. Closure[...] describes a concrete environment, chiefly when
returning one from a factory.
Passing captured state to a code-only parameter explains the mismatch and suggests the generic form:
def apply(f: Callable[[i64], Result[i64, OverflowError[i64]]], value: i64) -> Result[i64, OverflowError[i64]]:
f(value)
def main() -> ():
offset = 3
add = def [offset](value: i64) -> Result[i64, OverflowError[i64]]:
Ok(value + offset)
apply(add, 4)
for a generic callback API, use `F: Callable[[i64], Result[i64, OverflowError[i64]]]` with `callback: &F`
Let the callback determine a separate result type
The first helpers require the input and successful output to have the same
type T. Give the complete result its own parameter U to allow other
transformations, including infallible callbacks.
The callback’s concrete signature determines U, including when it does not
appear in another argument. This example uses both forms of callback through
that shared interface:
def apply[T, U, F: Callable[[T], U]](callback: &F, value: T) -> U:
callback(value)
def double(value: i64) -> Result[i64, OverflowError[i64]]:
Ok(value * 2)
def main() -> Result[(), IoError | OverflowError[i64]]:
offset = 7
add = def [offset](value: i64) -> Result[i64, OverflowError[i64]]:
Ok(offset + value)
print(apply(&double, 10)?)?
print(apply(&add, 10)?)?
Ok(())
20
17
Borrow in a closure
A local callback may need to inspect data that its caller will use again.
Use & in a capture list to keep ownership outside the closure:
def main() -> Result[(), AllocError | IoError]:
mut values = [4]?
read = def [&values]() -> i64:
values[0]
print(read())?
moved = read
print(moved())?
values.append(5)?
print(values)?
Ok(())
4
4
[4, 5]
The borrow follows the closure into moved and ends after its last call.
Changing the list before a later call would conflict with that borrow. Borrowing
closures cannot be returned from their creating function or stored in collections
or sum types. Pass them to other functions through reference parameters:
def invoke(f: &Closure[[], i64]) -> i64:
f()
def main() -> Result[(), IoError]:
value = 7
read = def [&value]() -> i64:
value
print(invoke(read))?
Ok(())
7
The &Closure parameter borrows read implicitly; invoke(&read) is also valid.
The capture remains borrowed throughout invoke, including while its other
arguments are being evaluated. A closure can also explicitly borrow another
closure; the same rule protects all of their captured owners.
Use &mut to modify an existing binding rather than moving it into private state:
def main() -> Result[(), IoError | OverflowError[i64]]:
mut count = 0
mut increment = def [&mut count]() -> Result[(), OverflowError[i64]]:
count = count + 1
Ok(())
increment()?
increment()?
print(count)?
Ok(())
2
Both the source and closure bindings need mut. The original count becomes
available again after the last closure call. Reading it before another call would
conflict with the exclusive capture.
Return a closure
Return an owned closure to keep its captured data after its creating function
returns. Closure describes the returned call signature:
def counter(start: i64) -> Closure[[], Result[i64, OverflowError[i64]]]:
def [mut start]() -> Result[i64, OverflowError[i64]]:
start = start + 1
Ok(start)
def apply(f: &mut Closure[[], Result[i64, OverflowError[i64]]]) -> Result[i64, OverflowError[i64]]:
f()
def main() -> Result[(), IoError | OverflowError[i64]]:
mut first = counter(10)
mut second = counter(30)
print(apply(&mut first)?)?
print(second()?)?
print(first()?)?
Ok(())
11
31
12
apply needs an exclusive borrow because calling the counter changes its state.
For a read-only closure, use a shared parameter such as &Closure[[i64], Result[i64, OverflowError[i64]]].
Different anonymous expressions have different concrete environment types, even with identical signatures. A factory must return one concrete type on every path. Borrowing closures cannot escape their creating function. To return different owned shared-call environments, explicitly allocate a BoxedCallable.
An owned closure can capture another closure or be wrapped in Option or
Result:
def make(offset: i64) -> Option[Closure[[i64], Result[i64, OverflowError[i64]]]]:
inner = def [offset](value: i64) -> Result[i64, OverflowError[i64]]:
Ok(offset + value)
outer = def [inner](value: i64) -> Result[i64, OverflowError[i64]]:
Ok(inner(value)? * 2)
Some(outer)
def main() -> Result[(), IoError | OverflowError[i64]]:
match make(3):
case Some(calculate):
print(calculate(4)?)?
case Nothing:
print("no callback")?
Ok(())
14
Captured resources drop in reverse capture order, including nested environments.
You can call a returned closure immediately, for example counter(10)().
Its environment remains alive through the complete expression and is then
dropped. ? in a later argument also cleans up that temporary. Match an
optional closure before calling it, as in the example above.
Generic function values
A higher-order function can use a type parameter inside its callable signature. The compiler infers the type from the function value and the other arguments:
def increment(value: u8) -> Result[u8, OverflowError[u8]]:
Ok(value + 1)
def apply[T, U](operation: Callable[[T], U], value: T) -> U:
operation(value)
def main() -> Result[(), IoError | OverflowError[u8]]:
print(apply(increment, 41u8)?)?
Ok(())
42
Here T is u8. Passing 41i64 would conflict with increment’s signature.
To choose a generic function’s own type arguments, use brackets without calling it:
def identity[T](value: T) -> T:
value
def main() -> Result[(), IoError]:
keep = identity[u8]
print(keep(42))?
Ok(())
42
identity[u8] is a function value; identity[u8](42) calls that specialization.
A concrete callable annotation can also supply those type arguments:
def identity[T](value: T) -> T:
value
def main() -> Result[(), IoError]:
keep: Callable[[u8], u8] = identity
print(keep(42))?
Ok(())
42
This also works when a parameter or return type supplies the concrete signature.
Without that context, use explicit brackets such as identity[u8].
Transform a cell with a generic method
Give a method its own type parameters to let a callback transform Cell[T]
into Cell[U]:
class Cell[T]:
value: T
def map[U, F: Callable[[T], U]](self, transform: &F) -> Cell[U]:
Cell(transform(self.value))
def double(value: u8) -> u16:
# u8 is at most 255, so the doubled value is at most 510 in u16.
unchecked:
u16(value) * 2
def main() -> Result[(), IoError]:
original = Cell(7u8)
transform = double
changed = original.map(&transform)
print(changed.value)?
print(original.value)?
Ok(())
14
7
map borrows the original cell, so passing its field to the callback requires
a copyable T. Here, double determines the result type U = u16.
Store stateful callbacks
Pair explicit state with a named function when a registry needs different callback bodies with the same state type and call signature:
class Handler[State]:
state: State
callback: Callable[[&mut State, i64], Result[i64, OverflowError[i64]]]
def call(self: &mut Handler[State], value: i64) -> Result[i64, OverflowError[i64]]:
self.callback(&mut self.state, value)
def accumulate(total: &mut i64, value: i64) -> Result[i64, OverflowError[i64]]:
*total = *total + value
Ok(*total)
def subtract(total: &mut i64, value: i64) -> Result[i64, OverflowError[i64]]:
*total = *total - value
Ok(*total)
def main() -> Result[(), AllocError | IoError | OverflowError[i64]]:
mut handlers = [Handler(10, accumulate), Handler(100, subtract)]?
for handler in &mut handlers:
print(handler.call(3)?)?
print(handlers[0].call(2)?)?
Ok(())
13
97
15
Each record owns its state; call borrows it exclusively. Constructing the
records needs no allocation, but storing them in a list does.
For different state shapes, use an enum and handle its variants in the callback. Owned concrete closures can also be stored when their producer types agree.
See examples/callback_registry.plenty for a registry example.
Store owned closures
A generic record can own a callback’s captured state. You do not need to name the closure’s concrete type: the constructor infers it.
class Handler[F: Callable[[i64], Result[i64, OverflowError[i64]]]]:
callback: F
def call(self: &mut Handler[F], amount: i64) -> Result[i64, OverflowError[i64]]:
self.callback(amount)
def counter(total: i64) -> Closure[[i64], Result[i64, OverflowError[i64]]]:
def [mut total](amount: i64) -> Result[i64, OverflowError[i64]]:
total = total + amount
Ok(total)
def main() -> Result[(), IoError | OverflowError[i64]]:
mut first = Handler(counter(10))
mut second = Handler(counter(100))
print(first.call(3)?)?
print(second.call(7)?)?
print(first.callback(2)?)?
Ok(())
13
107
15
The Callable constraint checks the signature without erasing the concrete
environment type. Each record stores its environment inline.
Keep captured state owned when storing a callback. A closure borrowing a local
cannot enter a record. Moving the record transfers ownership; dropping it cleans
up captured resources. Closures do not support copy or equality.
Reduce a borrowed collection with a pipeline
iter pipelines also accept functions, so
they can transform items without making an intermediate collection. Borrow the
source when the caller should keep it.
class Payment:
name: str
cents: i64
def main() -> Result[(), AllocError | IoError | OverflowError[i64]]:
payments = [Payment("bob", 120), Payment("ada", 300), Payment("bob", 80)]?
selected = iter(&payments).filter(def(p: &Payment) -> bool: p.name == "bob")
total = selected.map(def(p: &Payment) -> i64: p.cents).sum()?
largest = iter(&payments).map(def(p: &Payment) -> i64: p.cents).max()
refund = iter(&payments).any(def(p: &Payment) -> bool: p.cents < 0)
print(total, largest, refund, len(payments))?
Ok(())
200 Option[i64].Some(300) False 3
selected holds a loan of payments. The filter and map run when sum
consumes the pipeline, in source order. A closure written directly in the chain,
like the filter and map here, is compiled into that loop rather than called.
The payments remain available afterwards.
Trying to consume selected again is an ownership error.
sum and max work as in Chain iteration steps with
iter; here they reduce
mapped values, since a Payment is neither a number nor a str. Likewise,
any and all take a predicate when the items are not bool. Use try_fold for
a fallible accumulation, or fold for an infallible one.
A method returning a borrowed list can be used directly as the source. Its owner remains borrowed while the pipeline is in use:
class Readings:
values: list[i64]
def samples(self) -> &list[i64]:
&self.values
def main() -> Result[(), AllocError | IoError | OverflowError[i64]]:
readings = Readings([10, 20, 30]?)
product = iter(readings.samples()).try_fold(1, def(acc: i64, n: &i64) -> Result[i64, OverflowError[i64]]: Ok(acc * *n))?
print(product)?
Ok(())
6000
try_fold starts with its first argument and passes the accumulator and each
item to the callback. An Ok supplies the next accumulator; the first Err
stops iteration and is returned to the caller.
Predicates borrow each item; map receives whatever the source yields. A
borrowed list yields references, so the map above takes &Payment. A range
yields integers by value:
def main() -> Result[(), IoError | OverflowError[i64]]:
for square in iter(range(10)).skip(2).take(3).map(def(n: i64) -> Result[i64, OverflowError[i64]]: Ok(n * n)):
print(square?)?
Ok(())
4
9
16
Use try_collect when a transformation can fail and the desired result is an
owned collection. The annotation chooses the collection, and ? preserves the
first parsing or allocation error:
def main() -> Result[(), AllocError | ParseError | IoError]:
lines = ["10", "20", "30"]?
values: list[i64] = iter(&lines).map(def(line: &str) -> Result[i64, ParseError]: i64.parse(line)).try_collect()?
print(values, len(lines))?
Ok(())
[10, 20, 30] 3
For an infallible transformation, use collect() with the same annotation.
try_fold and try_for_each propagate errors returned by their callbacks;
they do not automatically unwrap Result items.
After enumerate or zip, callbacks take two
item parameters. Predicates borrow
both components; mapping receives the index by value and the borrowed item:
def main() -> Result[(), AllocError | IoError]:
names = ["Ada", "Bob", "Cal"]?
indexed = iter(&names).enumerate(1)
selected = indexed.filter(def(i: &i64, name: &str) -> bool: *i != 2)
labels: dict[i64, str] = selected.map(def(i: i64, name: &str) -> Result[(i64, str), AllocError]: Ok((i, copy(name)?))).try_collect()?
print(labels[1], labels[3], len(names))?
Ok(())
Ada Cal 3
Copying a name can fail, so try_collect propagates that error as well as
dictionary allocation errors. The successful owned pairs become dictionary entries. Repeated keys
keep the last value. A pair fold takes the accumulator followed by both item
components; a tuple accumulator stays a single parameter. Zip stops at the
shorter input, pulling the left first, so it may consume one extra left item.
Use chain to continue with a second source of the same item type, and
flat_map when each item expands into a source of its own:
class Batch:
first: i64
stop: i64
def main() -> Result[(), AllocError | IoError | OverflowError[i64]]:
morning = [3, 1]?
evening = [4]?
total = iter(&morning).chain(&evening).sum()?
print(total)?
batches = [Batch(100, 102), Batch(200, 201)]?
for ticket in iter(&batches).flat_map(def(b: &Batch) -> range[i64]: range(b.first, b.stop)):
print(ticket)?
Ok(())
8
100
101
200
Both lists stay borrowed until the sum ends. The flat_map callback runs
once per batch, only after the previous batch’s tickets are used up, and
returns a range, so nothing is allocated. Batch stores start and stop
endpoints directly; constructing the range needs no endpoint arithmetic. A
callback may return any owned iterable, such as a list, range or generator; returning a list allocates one list
per item.
Lend a pipeline whose callbacks borrow
A pipeline lent to a function can use a closure that borrows, when the closure is written inside the adapter call:
def first(p: &mut Iter[i64]) -> Option[i64]:
next(p)
def main() -> Result[(), IoError]:
limit = 3
mut above = iter(range(9)).filter(def [&limit](x: &i64) -> bool: *x > limit)
print(first(&mut above))?
print(above.count())?
Ok(())
Option[i64].Some(4)
4
Storing the same closure in a variable first and passing the variable makes the pipeline impossible to lend:
def first(p: &mut Iter[i64]) -> Option[i64]:
next(p)
def main() -> Result[(), IoError]:
limit = 3
keep = def [&limit](x: &i64) -> bool:
*x > limit
mut above = iter(range(9)).filter(keep)
print(first(&mut above))?
Ok(())
it holds closure `keep`, which is stored in a variable and borrows its captures; write the closure literal inside the adapter call instead
A closure written in the chain is compiled into the pipeline, so its borrow of
limit becomes one of the pipeline’s own loans, which the caller keeps while
first runs. A stored closure is a value holding that reference inside it,
and a function receiving a lent pipeline cannot take such a value. Without the
lend, both forms work.
Keep a plain loop for several statements or error propagation, and use a comprehension when building a list is the clearest form. Pipelines are local: Transform a stream of values applies them to generators and shows how to return a stream from a function. See the pipeline reference for terminals and the current limits.
Transform a stream of values
A generator has the same pipeline methods as iter(...), so a stream of values
can be filtered, mapped and reduced without an intermediate list:
def numbers(stop: i64) -> Generator[i64]:
for n in range(stop):
yield n
def main() -> Result[(), IoError | OverflowError[i64]]:
two: NonZero[i64] = 2
evens = numbers(10).filter(def [two](n: &i64) -> bool: *n % two == 0)
for square in evens.map(def(n: i64) -> Result[i64, OverflowError[i64]]: Ok(n * n)):
n = square?
print(n)?
if n > 3:
break
Ok(())
0
4
The generator body and callbacks run only when the loop asks for a value, and
break drops the rest of the chain: numbers never makes 3. The adapters are
fused into the loop, so they add no allocation or generator frame of their own.
take limits how many items are requested from a generator. Callbacks can capture values, as with
any closure:
def naturals() -> Generator[i64]:
for n in range(i64.MAX):
yield n
def main() -> Result[(), IoError | OverflowError[i64]]:
offset = 100
for n in naturals().map(def [offset](n: i64) -> Result[i64, OverflowError[i64]]: Ok(n + offset)).take(3):
print(n?)?
Ok(())
100
101
102
naturals is bounded by the representable range rather than overflowing a
manual counter. take(3) requests only its first three values. The map returns
ordinary public Result items, which the loop propagates before printing.
Return a stream
A pipeline stays in the function that builds it. When a chain needs to be
returned or passed to another function, end it with .generator(). This
explicitly moves the source and captures into one resumable generator, with the
adapters still fused inside it:
def squares(stop: i64) -> Generator[Result[i64, OverflowError[i64]]]:
iter(range(stop)).map(def(n: i64) -> Result[i64, OverflowError[i64]]: Ok(n * n)).generator()
def total(values: Generator[Result[i64, OverflowError[i64]]]) -> Result[i64, OverflowError[i64]]:
values.try_fold(0, def(acc: i64, n: Result[i64, OverflowError[i64]]) -> Result[i64, OverflowError[i64]]: Ok(acc + n?))
def main() -> Result[(), IoError | OverflowError[i64]]:
print(total(squares(4))?)?
mut values = squares(3)
first = next(values).unwrap()?
second = next(values).unwrap()?
print(Some(first), Some(second))?
Ok(())
14
Option[i64].Some(0) Option[i64].Some(1)
The two unwrap calls assert only that squares(3) has at least two items;
the arithmetic Result in each item is still propagated with ?. For an
unknown-length stream, match next instead. A generator cannot propagate an
arithmetic error from its body: yielding Result items lets its consumer
decide how to handle failure. try_fold here handles each item as well as
its own addition.
Conversion does not visit any items. The returned generator owns its source and captures, so a pipeline over a borrowed list or with borrowed captures cannot be converted, even after mapping to owned values; return an owned collection instead. Dropping the generator early releases its remaining state.
Each next resumes the generator, which costs more than a fused local loop.
Consume a pipeline where it is built unless a resumable stream must cross a
function boundary. See the pipeline reference
for the full list of adapters and terminals.
Keep callbacks in tuples and lists
A tuple can hold callbacks with different concrete types. Indexing invokes a callback in place; unpacking transfers ownership to new local bindings.
def add(offset: i64) -> Closure[[i64], Result[i64, OverflowError[i64]]]:
def [offset](n: i64) -> Result[i64, OverflowError[i64]]:
Ok(offset + n)
def main() -> Result[(), IoError | OverflowError[i64]]:
pair = (add(10), add(100))
print(pair[0](2)?)?
first, second = pair
print(second(3)?)?
Ok(())
12
103
Calls to one factory also fit in a list. Each callback owns its own state.
def counter(total: i64) -> Closure[[], Result[i64, OverflowError[i64]]]:
def [mut total]() -> Result[i64, OverflowError[i64]]:
total = total + 1
Ok(total)
def main() -> Result[(), AllocError | IoError | OverflowError[i64]]:
mut callbacks = [counter(n) for n in range(3)]?
for callback in &mut callbacks:
print(callback()?)?
match callbacks.pop(0):
case Some(callback):
mut removed = callback
print(removed()?)?
case Nothing:
print("no callback")?
print(callbacks[0]()?)?
Ok(())
1
2
3
2
3
pop transfers the callback out of the list. A borrowed callback prevents
resizing or removing entries until its borrow ends.
Name stored callbacks
A dictionary can associate names with owned callbacks from one factory. Calling
a callback that changes its captured state needs a mutable borrow: use
get_mut to keep the callback registered,
or pop to take ownership.
def counter(total: i64) -> Closure[[], Result[i64, OverflowError[i64]]]:
def [mut total]() -> Result[i64, OverflowError[i64]]:
total = total + 1
Ok(total)
def main() -> Result[(), AllocError | IoError | OverflowError[i64]]:
mut callbacks = {"open": counter(10), "close": counter(100)}?
match callbacks.get_mut("open"):
case Some(callback):
print(callback()?)?
case Nothing:
print("no open handler")?
match callbacks.pop("close"):
case Some(callback):
mut owned = callback
print(owned()?)?
case Nothing:
print("no close handler")?
print(len(callbacks))?
Ok(())
11
101
1
Replacing a registered callback drops the old captured state. Inserting a new key
may grow the dictionary and returns Result. A live borrow of any callback
protects the dictionary against replacement, removal, and growth.
The callback itself cannot be a dictionary key or a set element: closures do not define equality or hashing. The dictionary’s value type preserves its concrete factory identity, rather than accepting every callback with the same signature. Use explicit boxing to store different shared-call environments behind one signature.
Store different captured callbacks
Concrete closures remember where they were defined. To put different closures
behind one signature, explicitly allocate a BoxedCallable. Its constructor
consumes the callback and returns Result, so ? handles allocation failure.
def choose(k: i64) -> Result[BoxedCallable[[i64], Result[i64, OverflowError[i64]]], AllocError]:
if k > 0:
add = def [k](value: i64) -> Result[i64, OverflowError[i64]]:
Ok(k + value)
return BoxedCallable(add)
span = range(10, 13)
multiply = def [span](value: i64) -> Result[i64, OverflowError[i64]]:
Ok(span[1] * value)
BoxedCallable(multiply)
def main() -> Result[(), AllocError | IoError | OverflowError[i64]]:
callbacks: dict[str, BoxedCallable[[i64], Result[i64, OverflowError[i64]]]] = {
"add": choose(3)?,
"multiply": choose(0)?,
}?
print(callbacks["add"](4)?)?
print(callbacks["multiply"](4)?)?
borrowed = &callbacks["add"]
print(borrowed(5)?)?
Ok(())
7
44
8
The factory’s branches capture different data but return the same boxed signature.
Calling borrows the stored owner, so repeated calls keep it registered and require
no extra allocation. Ordinary moves transfer ownership, and dropping or replacing
the owner drops its captures. Named functions can also use BoxedCallable(name).
This type currently accepts reusable shared-call callbacks with owned captures. Keep mutable, one-shot, borrowed, or generator-containing closures concrete. Callbacks returning references and worker transfer are also unsupported. Boxing does not make a callback copyable or comparable; use names as registry keys.
Put a callback in an enum
An enum can also own a callback. Match a borrow when you want to leave the owner in place, or match the value to transfer its environment into an arm.
enum Job[F: Callable[[], i64]]:
Run(F)
Empty
def package[F: Callable[[], i64]](callback: F) -> Job[F]:
Job[F].Run(callback)
def invoke[F: Callable[[], i64]](job: &Job[F]) -> i64:
match &job:
case Job.Run(callback):
callback()
case Job.Empty:
0
def main() -> Result[(), IoError]:
value = 42
callback = def [value]() -> i64:
value
job = package(callback)
print(invoke(&job))?
Ok(())
42
Consume a capture
A reusable closure borrows its captured state on each call. Sometimes the purpose
of the callback is to hand over an owned value. Mark that callback once:
def main() -> Result[(), AllocError | IoError]:
values = [2, 3, 5]?
take = def once [values]() -> list[i64]:
values
result = take()
print(result)?
Ok(())
[2, 3, 5]
take() consumes the callback and transfers its captured list to the result.
A second call is a use-after-move error. If take is never called, dropping it
cleans up the list.
Consume a callback
Use OnceCallable for a function that owns and calls its callback once. It accepts
named functions, reusable closures, and one-shot closures.
pub def run[T, F: OnceCallable[[], T]](callback: F) -> T:
mut owned = callback
owned()
import callbacks
def defer[T](value: T) -> OnceClosure[[], T]:
def once [value]() -> T:
value
def main() -> Result[(), AllocError | IoError]:
take = defer([3, 5]?)
print(callbacks.run(take))?
Ok(())
[3, 5]
OnceClosure describes the concrete value returned by defer. OnceCallable
describes what the generic run function can accept. Passing take transfers
ownership; calling it again afterward is an error. The mutable local inside
run also permits reusable callbacks that update private captured state.
Consume owned data while borrowing local state
For a callback that also borrows local state, invoke it in that local scope:
def main() -> Result[(), AllocError | IoError | OverflowError[i64]]:
mut count = 0
values = [8]?
take = def once [values, &mut count]() -> Result[list[i64], OverflowError[i64]]:
count = count + 1
Ok(values)
result = take()?
print(count)?
print(result)?
Ok(())
1
[8]
The borrow ends after the call. This callback cannot be passed to callbacks.run
because a borrowing environment cannot move across function boundaries.
Store consuming jobs
A stored job may need to hand its captured resources to its caller. Use the
existing def once form and extract the job before calling it.
def make_job(n: i64) -> Result[OnceClosure[[], list[i64]], AllocError]:
values = [n]?
job = def once [values]() -> list[i64]:
values
Ok(job)
def main() -> Result[(), AllocError | IoError]:
mut jobs = [make_job(10)?, make_job(20)?]?
match jobs.pop(0):
case Some(job):
print(job())?
case Nothing:
pass
for job in jobs:
print(job())?
Ok(())
[10]
[20]
Calling jobs[0]() would try to consume a borrowed entry and is rejected.
Extract an owned callback with pop, consuming iteration, consuming enum matching,
or tuple unpacking.
Dropping uncalled jobs cleans up their captures. Borrowed captures and generator-containing environments cannot be stored in these heap owners.
Concurrent and parallel programs
Use scoped threads for work that borrows local data, or an executor for many owned jobs. Captured jobs use the functions-as-values lessons; generic recovery helpers also use type parameters.
- Import concurrency explicitly
- Run scoped threads
- Borrow local data for a worker
- Count with atomics
- Share mutable state with a mutex
- Choose how workers share state
- Move jobs to workers
- Run stored jobs
- Send values between threads
- Avoid waits that hang
- Handle channel backpressure
- Bound your waits
- Share one deadline across waits
- Reuse workers with an executor
- Handle executor outcomes
- Share state with pool workers
- Cancel queued work and bound a future wait
- Stop work cooperatively
- Recover a job when submission cannot proceed
- Map fallible work in parallel
- Combine values in parallel
- Receive from either channel
- Wait for I/O with a deadline
- Serve a TCP connection
- Encrypt a TCP connection
- Verify clients and choose a protocol
- Serve HTTP requests
- Stop a server on a signal
- Read forms, cookies and forwarded clients
- Serve files and sign cookies
- Send HTTP requests
- Share one HTTP client between workers
- Talk over WebSockets
- Stream events to browsers
Import concurrency explicitly
The concurrency operations in these lessons also have explicit standard-library imports. Module aliases and item aliases name the same facilities and types as the familiar prelude spellings. Existing programs continue to work.
import std.thread as threads
from std.channel import channel as bounded, Receiver as Inbox
from std.concurrent import ThreadPoolExecutor as Pool
from std.sync import CancellationToken as Stop
def receive(inbox: Inbox[str]) -> Result[str, RecvError]:
inbox.recv()
def stopped(stop: Stop) -> bool:
stop.wait()
stop.is_cancelled()
type Errors = AllocError | IoError | SendError[str] | RecvError | SpawnError[()] | PoolError | FutureError | SubmitError[()]
def main() -> Result[(), Errors]:
sender, receiver = bounded[str](1)?
sender.send("hello")?
drop(sender)
with threads.spawn(receive, receiver).drop_job()? as task:
print(task.join()?)?
stop = Stop()?
with Pool(1, 1)? as pool:
future = pool.submit(stopped, stop.share()).drop_job()?
stop.cancel()
print(future.result()?)?
Ok(())
hello
True
Dropping the sender before joining permits a receiving worker to finish even
when it loops until disconnection. Imported names retain the same loan,
worker-effect, queue-bound and hang diagnostics as prelude names. Error types
can also be imported: from std.concurrent import FutureError, for example.
Importing an existing builtin under its own name is allowed only when it names
that exact builtin; unrelated imports still cannot replace it.
The module references list the complete surfaces:
std.thread,
std.channel,
std.concurrent, and
std.sync. Imports ship inside the compiler and
need no network or separately installed Plenty sources.
A failed submission returns its unstarted job inside SubmitError;
drop_job() drops it and leaves SubmitError[()], as the
executor lesson explains. A spawn of a named
function with owned arguments works the same way: it returns them inside
SpawnError, and drop_job() leaves SpawnError[()].
Run scoped threads
Use with spawn(...) to run a function on a native thread. The current thread
continues until join() waits for the result:
def squares(n: i64) -> Result[list[i64], AllocError | OverflowError[i64]]:
[i * i for i in range(n)]
def main() -> Result[(), SpawnError[()] | AllocError | IoError | OverflowError[i64]]:
with spawn(squares, 4).drop_job()? as first, spawn(squares, 3).drop_job()? as second:
print(first.join()?)?
print(second.join()?)?
Ok(())
[0, 1, 4, 9]
[0, 1, 4]
The ? after spawn(...) handles failure to create the thread. The ? after
join() handles the worker’s own Result, here allocation failure or
arithmetic overflow while building its list. They are separate failures. drop_job() is explained in
error types at thread start. Join consumes the task binding,
so it cannot be called twice.
Every started thread finishes before its with block exits. This includes
return, ?, break, and continue. If you omit join(), exit waits and drops
the unused result, including any application error. Join when the result matters.
A task cannot escape its block or be stored for later.
Scopes cannot forcibly cancel workers. Keep their computations finite or give them a shutdown protocol. Fatal traps terminate the process without unwinding.
Error types at thread start
A failed start reports SpawnError[F], which returns the unstarted job: here
the function and its arguments. drop_job() drops that job at the start, so the
error becomes SpawnError[()], one member for every start. Keep the job when
you want to retry it, as owned worker jobs shows.
Borrowed workers report SpawnError[()] too.
Like any ?, the start error widens into the enclosing Result’s error sum, so
main lists SpawnError[()] beside the worker’s AllocError and
OverflowError[i64], and the printing IoError. A sum without the start error is rejected.
Use module-level error aliases
to keep longer signatures readable, as in executor jobs. Each alias names that example’s propagated
errors; it does not change widening through ? or introduce an error wrapper.
Borrow local data for a worker
Borrow mutable inputs explicitly. Shared readers can coexist; an exclusive borrow prevents the parent from accessing the same data until the block ends. Join explicitly to propagate a worker’s arithmetic error:
def increment(value: &mut i64) -> Result[(), OverflowError[i64]]:
*value = *value + 1
Ok(())
def main() -> Result[(), SpawnError[()] | IoError | OverflowError[i64]]:
mut count = 4
with spawn(increment, &mut count)? as task:
task.join()?
print(count)?
Ok(())
5
The borrow lasts until the block ends, even after an explicit join. Reading or
modifying count inside the block conflicts with it; use a shorter block when
you need access back sooner.
Reusable closures work too. Borrow the closure itself so that a failed start leaves the job available to retry:
def main() -> Result[(), SpawnError[()] | IoError | AllocError]:
mut values = [1, 2]?
mut job = def [&mut values]() -> Result[i64, AllocError]:
values.append(3)?
Ok(len(values))
with spawn(&mut job)? as task:
print(task.join()?)?
print(values)?
Ok(())
3
[1, 2, 3]
For immutable captures, spawn(&job) allows shared access. Scoped work may
borrow or own its inputs, so a named worker can also receive an owned list:
def total(values: list[i64]) -> Result[i64, OverflowError[i64]]:
mut result = 0
for value in values:
result = result + value
Ok(result)
def main() -> Result[(), SpawnError[()] | IoError | AllocError | OverflowError[i64]]:
values = [2, 3, 4]?
with spawn(total, values).drop_job()? as task:
print(task.join()?)?
Ok(())
9
values moves and cannot be used again. Because every argument is owned, the
start reports SpawnError, like the scoped threads lesson’s
starts: without drop_job() a failed start would return values inside the
error. A start that borrows reports the job-free SpawnError[()], because an
error cannot hold a borrow. If such a start fails, its moved arguments drop on
the parent thread.
To preserve creation error details, put the scope in a helper and match its
result. SpawnError.Unavailable(job, code) contains the native error code.
The compiler checks worker helpers, captured data, and destructors. Foreign
calls, File values, indirect function calls, and generators are not yet eligible.
Worker code can use checked print, but output from different threads may
arrive in any order.
Count with atomics
A shared borrow lets several workers read the same data, but not change it. An
atomic from std.atomic is the exception: its methods update it through a
shared reference, one indivisible step at a time:
from std.atomic import Atomic, Ordering
def count(hits: &Atomic[u64], rounds: i64) -> ():
for _ in range(rounds):
hits.fetch_add(1, Ordering.Relaxed)
def main() -> Result[(), SpawnError[()] | IoError]:
hits = Atomic[u64](0)
with spawn(count, &hits, 100000)?, spawn(count, &hits, 100000)?:
pass
print(hits.load(Ordering.Acquire))?
Ok(())
200000
Atomic[u64](0) stores the counter inline in main’s frame; nothing is
allocated. fetch_add returns the value it replaced. It stops the
program on overflow, before storing anything.
Every operation names an Ordering. Relaxed only makes the update itself
indivisible. Release on a store and Acquire on a load make the writes before
the store visible to the thread that loads it. Use SeqCst when unsure. The
compiler rejects orderings an operation cannot use:
from std.atomic import Atomic, Ordering
def main() -> ():
ready = Atomic[bool](False)
ready.store(True, Ordering.Acquire)
store does not accept Ordering.Acquire; use Ordering.Relaxed, Ordering.Release, or Ordering.SeqCst
A class can hold atomics as fields, so one borrow shares several of them. Atomics
are not values to print or compare, so read each one with load:
from std.atomic import Atomic, Ordering
class Stats:
seen: Atomic[u32]
largest: Atomic[i64]
def record(stats: &Stats, start: i64) -> Result[(), OverflowError[i64]]:
for value in range(start, start + 1000):
stats.seen.fetch_add(1, Ordering.Relaxed)
stats.largest.fetch_max(value, Ordering.Relaxed)
Ok(())
def main() -> Result[(), SpawnError[()] | IoError | OverflowError[i64]]:
stats = Stats(seen=Atomic(0), largest=Atomic(i64.MIN))
with spawn(record, &stats, 0)? as first, spawn(record, &stats, 5000)? as second:
first.join()?
second.join()?
print(stats.seen.load(Ordering.Acquire))?
print(stats.largest.load(Ordering.Acquire))?
Ok(())
2000
5999
compare_exchange replaces a value only if it still equals the one you expect,
and returns Err with the actual value otherwise. See
std.atomic for every method.
Share mutable state with a mutex
An atomic keeps one value consistent. When several values must change together,
put them in a Mutex from std.sync. Only one thread at a time can hold its
lock, so no worker sees count updated without total:
from std.sync import Mutex
class Stats:
count: i64
total: i64
def record(stats: &Mutex[Stats], value: i64, rounds: i64) -> Result[(), OverflowError[i64]]:
for _ in range(rounds):
with stats.lock() as data:
count = data.count + 1
total = data.total + value
data.count = count
data.total = total
Ok(())
def main() -> Result[(), SpawnError[()] | IoError | OverflowError[i64]]:
stats = Mutex[Stats](Stats(0, 0))
with spawn(record, &stats, 3, 10000)? as first, spawn(record, &stats, 4, 10000)? as second:
first.join()?
second.join()?
final = stats.into_inner()
print(final.count)?
print(final.total)?
Ok(())
20000
70000
stats.lock() waits until no other thread holds the lock, then binds data, a
&mut Stats, for the block. Leaving the block unlocks, whether by reaching its
end, return, ?, break, or continue. Both new fields are computed before
either is stored, so arithmetic failure cannot leave a half-updated pair. Workers share &stats without mut,
because the lock, not the borrow, makes the access exclusive. After the workers
finish, into_inner() takes the value back without locking. Nothing here
allocates.
lock() works only as a with manager, so a lock cannot be left held by
mistake. data belongs to the block: it cannot be stored, returned, or used
after it.
Waiting under a lock for something that needs the same lock never finishes. The compiler rejects the waits it can see. Relocking a mutex inside its own block is one; joining a worker that can lock the same kind of mutex is another:
from std.sync import Mutex
def bump(counter: &Mutex[i64]) -> Result[(), OverflowError[i64]]:
with counter.lock() as value:
*value = *value + 1
Ok(())
def main() -> Result[(), SpawnError[()]]:
counter = Mutex(0)
with counter.lock() as value:
with spawn(bump, &counter)?:
pass
Ok(())
leaving this `with` block joins a task whose worker can lock a `Mutex[i64]` while this function holds a lock on a `Mutex[i64]`
note: the lock is acquired here; end the lock's `with` block before this one, or start the task outside it
Start the task first, then lock inside its block: the lock’s block ends, and unlocks, before the task is joined.
from std.sync import Mutex
def bump(counter: &Mutex[i64]) -> Result[(), OverflowError[i64]]:
with counter.lock() as value:
*value = *value + 1
Ok(())
def main() -> Result[(), SpawnError[()] | IoError | OverflowError[i64]]:
counter = Mutex(0)
with spawn(bump, &counter)? as task:
with counter.lock() as value:
*value = *value + 10
task.join()?
print(counter.into_inner())?
Ok(())
11
Nesting a lock inside the block of another mutex of the same type is also
rejected when either one is reached through a reference or a computed index:
two references can name one mutex. Two separately owned mutexes, such as two
local names, may nest. The compiler cannot see a relock inside a called
function, a join inside a function called while the lock is held, or two
threads taking two locks in opposite orders. Keep lock blocks short and take
nested locks in one fixed order. See
std.sync and
mutexes for the full rules.
Choose how workers share state
Prefer giving each worker what it needs and taking results back. Reach for shared mutable state only when workers must see each other’s changes while running.
| Need | Use |
|---|---|
| Inputs a worker reads or owns | Shared borrows, or move the values in |
| Results to collect | task.join(), or send them on a channel |
| One independent counter or flag | An atomic |
| Several values that must change together | A mutex |
| State that executor jobs share | A Shared handle to an atomic or a mutex |
| A request to stop | A cancellation token |
Here each worker sums its own slice of the work and returns the result, so nothing is shared and nothing needs a lock:
def total(start: i64, end: i64) -> Result[i64, OverflowError[i64]]:
mut sum = 0
for value in range(start, end):
sum = sum + value
Ok(sum)
def main() -> Result[(), SpawnError[()] | IoError | OverflowError[i64]]:
with spawn(total, 0, 500).drop_job()? as low, spawn(total, 500, 1000).drop_job()? as high:
print(low.join()? + high.join()?)?
Ok(())
499500
An atomic is the cheapest shared update, but each operation is separate: two atomics updated one after the other can be observed half-done. A mutex makes a whole block of updates appear at once, at the cost of making other threads wait.
Atomics and mutexes are shared by borrowing, so they work with scoped threads.
Executor jobs must own their inputs, so they cannot borrow either one. Give each
job its own owner of a Shared[Atomic[T]] or Shared[Mutex[T]] instead, or
send jobs their data and collect their results.
Move jobs to workers
An owned closure can move its inputs into a worker. Use a one-shot closure when the worker also moves those inputs into its result.
def main() -> Result[(), AllocError | IoError | SpawnError[OnceClosure[[], list[i64]]]]:
values = [3, 4]?
with spawn(def once [values]() -> list[i64]: values)? as task:
print(task.join())?
Ok(())
[3, 4]
If thread creation fails, the error is SpawnError[F].Unavailable(job, code),
which retains the complete job and the native error code. F is the closure’s
own type. The signature writes
OnceClosure[[], list[i64]], and ? fills in the concrete closure it
propagates, so main returns the unstarted job unchanged.
A signature names one closure there: a second spawn of a different closure
literal is rejected. When you do not need the job back, call drop_job() on
each start. It drops a failed start’s job right away, and the error becomes
SpawnError[()], which names no job, so one member covers every start:
def main() -> Result[(), AllocError | IoError | SpawnError[()]]:
values = [3, 4]?
with spawn(def once [values]() -> list[i64]: values).drop_job()? as first:
print(first.join())?
with spawn(def once () -> str: "done").drop_job()? as second:
print(second.join())?
Ok(())
[3, 4]
done
Dropping the job also drops the channel endpoints it captured, so they cannot keep a later join waiting; avoid waits that hang shows why that matters.
To keep the job, make each one with one factory function, or use a generic
helper, which names the job as F. Recover the job to retry or run it
locally, as this helper does:
def attempt[F: OnceCallable[[], i64]](job: F) -> Result[i64, SpawnError[F]]:
with spawn(job)? as task:
return Ok(task.join())
def run[F: OnceCallable[[], i64]](job: F) -> i64:
match attempt(job):
case Ok(value):
value
case Err(SpawnError.Unavailable(job, _)):
job()
def main() -> Result[(), IoError]:
n = 42
job = def once [n]() -> i64:
n
print(run(job))?
Ok(())
42
A reusable closure can also be moved into spawn; its captures are dropped
after the worker’s single invocation.
Run stored jobs
Stored reusable callbacks can run on scoped threads. Borrow them explicitly, just as you would borrow a local closure. Distinct tuple slots have separate loans.
def counter(total: i64) -> Closure[[], Result[i64, OverflowError[i64]]]:
def [mut total]() -> Result[i64, OverflowError[i64]]:
total = total + 1
Ok(total)
def main() -> Result[(), SpawnError[()] | IoError | OverflowError[i64]]:
mut jobs = (counter(10), counter(100))
with spawn(&mut jobs[0])? as first, spawn(&mut jobs[1])? as second:
print(first.join()?)?
print(second.join()?)?
print(jobs[0]()?)?
Ok(())
11
101
12
A failed start leaves its borrowed callback owned by the caller. Any earlier successful start is joined.
Workers can also return an owned callback through join():
def make(offset: i64) -> Closure[[i64], Result[i64, OverflowError[i64]]]:
def [offset](n: i64) -> Result[i64, OverflowError[i64]]:
Ok(offset + n)
def main() -> Result[(), SpawnError[()] | IoError | OverflowError[i64]]:
with spawn(make, 10).drop_job()? as task:
callback = task.join()
print(callback(3)?)?
Ok(())
13
List and dictionary entries can also supply a borrowed job. Except at distinct
literal list indices, their loans protect the whole collection. Even after join(), wait until the with block
ends before resizing or replacing that collection. Stored callbacks have the
same thread-eligibility restrictions as local ones.
Send values between threads
A channel transfers owned messages from senders to receivers. Give it a fixed capacity to bound queued memory: a sender waits when the queue is full, and a receiver waits when it is empty.
def produce(sender: Sender[i64]) -> Result[(), SendError[i64] | OverflowError[i64]]:
for n in range(5):
sender.send(n * n)?
Ok(())
def drain(inbox: &Receiver[i64]) -> Result[i64, OverflowError[i64]]:
mut total = 0
while True:
match inbox.recv():
case Ok(value):
total = total + value
case Err(_):
break
Ok(total)
def main() -> Result[(), SpawnError[()] | AllocError | IoError | OverflowError[i64]]:
sender, receiver = channel[i64](2)?
with spawn(produce, sender).drop_job()?, receiver as inbox:
print(drain(inbox)?)?
Ok(())
30
This introductory example deliberately leaves the producer result unclaimed to focus on endpoint guards; use the explicit join pattern when worker errors affect the outcome. The receiver’s accumulation error is still propagated.
spawn(produce, sender) moves the only sender into the producer. When the
producer returns, its sender drops; the receiver drains the queue, then reports
disconnection, which ends the loop. A blocking recv fails only on
disconnection, so case Err(_) is enough here.
The receiver guard is the last manager in the with statement. Managers exit
in reverse order, so however the body exits, the receiver drops before Plenty
joins the producer. A producer waiting for queue space then sees disconnection
instead of waiting forever.
Plenty requires this. A join reached while receiver is still alive could wait
forever for a producer blocked on a full queue, so the compiler rejects it:
def produce(sender: Sender[i64]) -> Result[(), SendError[i64] | OverflowError[i64]]:
for n in range(5):
sender.send(n * n)?
Ok(())
def main() -> Result[(), SpawnError[()] | AllocError | RecvError | IoError | OverflowError[i64]]:
sender, receiver = channel[i64](2)?
with spawn(produce, sender).drop_job()?:
print(receiver.recv()?)?
Ok(())
leaving this `with` block joins a task whose worker holds `Sender[i64]` while `receiver` still holds `Receiver[i64]`
The compiler compares message types rather than following each queue, so it also asks for this order in some programs that could not hang. Write the guard anyway; it costs nothing.
Wait for the producer’s result
The automatic join above drops the producer’s result. To check it with
task.join(), finish with the receiver first: put its guard inside the task’s
block, before any statement that can leave the block, and join after the guard
ends.
def produce(sender: Sender[i64]) -> Result[(), SendError[i64] | OverflowError[i64]]:
for n in range(5):
sender.send(n * n)?
Ok(())
def drain(inbox: &Receiver[i64]) -> Result[i64, OverflowError[i64]]:
mut total = 0
while True:
match inbox.recv():
case Ok(value):
total = total + value
case Err(_):
break
Ok(total)
def main() -> Result[(), SpawnError[()] | AllocError | IoError | SendError[i64] | OverflowError[i64]]:
sender, receiver = channel[i64](2)?
with spawn(produce, sender).drop_job()? as task:
mut total = 0
with receiver as inbox:
total = drain(inbox)?
task.join()?
print(total)?
Ok(())
30
total is declared before the inner guard because a binding made inside a
block ends with that block. task.join()? propagates the producer’s
SendError[i64] or OverflowError[i64], if any.
A ? before the guard leaves the block with the receiver still alive, so this
version is rejected:
def produce(sender: Sender[i64]) -> Result[(), SendError[i64] | OverflowError[i64]]:
for n in range(5):
sender.send(n * n)?
Ok(())
def drain(inbox: &Receiver[i64]) -> Result[i64, OverflowError[i64]]:
mut total = 0
while True:
match inbox.recv():
case Ok(value):
total = total + value
case Err(_):
break
Ok(total)
def main() -> Result[(), SpawnError[()] | AllocError | IoError | SendError[i64] | OverflowError[i64]]:
sender, receiver = channel[i64](2)?
with spawn(produce, sender).drop_job()? as task:
print("receiving")?
mut total = 0
with receiver as inbox:
total = drain(inbox)?
task.join()?
print(total)?
Ok(())
leaving this `with` block joins a task whose worker holds `Sender[i64]` while `receiver` still holds `Receiver[i64]`
`?` leaves the block here
Prefer a guard to drop(receiver): a drop protects only the paths that reach
it, and an earlier ? does not.
A receiving worker
The same order applies when the worker receives: an empty queue keeps it waiting while any sender survives. Guard the sender last, so every exit disconnects the queue before the join:
def consume(receiver: Receiver[i64]) -> ():
match receiver.recv():
case Ok(_):
pass
case Err(_):
pass
def main() -> Result[(), SpawnError[()] | AllocError | SendError[i64]]:
sender, receiver = channel[i64](1)?
with spawn(consume, receiver).drop_job()?, sender as outbox:
outbox.send(42)?
Ok(())
Leaving sender alive until after this block is rejected, even if the message
is already queued. For an explicit task.join(), end an inner sender guard
first, as with the receiver above.
Ownership and cost
Create additional handles explicitly with sender.share() or receiver.share().
Multiple receivers compete for messages on the same queue: each message is
delivered to one of them. Disconnection occurs only after the last handle on
that side drops, so every handle you keep needs a guard or a drop. The
wait habits lesson shows how to start several producers.
Channel construction can fail. Its positive capacity fixes the queue’s storage up front; sending and receiving introduce no further allocations. Messages still use their ordinary types: sending a list moves it, and receiving returns its ownership. A failed send returns the unsent value inside its error.
A failed start returns its owned arguments, including endpoints, inside
SpawnError, as in owned worker jobs. drop_job()
drops them at the start instead, so the error holds no endpoint;
wait habits shows why that matters.
Avoid waits that hang
A join waits for a worker. If the worker is blocked on a channel endpoint or a lock that the waiting function still holds, or waits for a cancel that the function has not made yet, the join never returns. Plenty rejects such waits at compile time. These habits keep your programs clear of the whole class:
- Move each endpoint into one worker as an argument:
spawn(produce, sender).drop_job()?. A failed start then drops it. - Guard each endpoint you keep as the last manager:
with spawn(produce, sender).drop_job()?, receiver as inbox:. - To wait inside the block, end the guard first. Open it before any statement that can leave the block, and wait after it ends.
- Wait in a statement of its own. An endpoint moved into the same expression is still held while the wait runs.
- Call
drop_job()on a start you will not retry. - Start workers outside a lock block, and lock inside them.
- Cancel a token on every exit when a worker waits for it:
with spawn(serve, stop.share()).drop_job()?, stop.cancel_on_exit():.
The channel shutdown lesson showed habits 1 to 3, and the mutex lesson showed habit 6. Cooperative cancellation shows habit 7. Executor pools follow the same habits, as the future wait lesson shows. This lesson covers several producers and failed starts.
Several producers
Each producer needs its own sender, and the receiver must outlive none of them.
One guard after several ? starts is not enough: when a later start fails, ?
joins the earlier producers before the guard exists.
def produce(sender: Sender[i64]) -> Result[(), SendError[i64] | OverflowError[i64]]:
for n in range(5):
sender.send(n * n)?
Ok(())
def drain(inbox: &Receiver[i64]) -> Result[i64, OverflowError[i64]]:
mut total = 0
while True:
match inbox.recv():
case Ok(value):
total = total + value
case Err(_):
break
Ok(total)
def main() -> Result[(), SpawnError[()] | AllocError | IoError | OverflowError[i64]]:
sender, receiver = channel[i64](2)?
second = sender.share()
with spawn(produce, sender).drop_job()?, spawn(produce, second).drop_job()?, receiver as inbox:
print(drain(inbox)?)?
Ok(())
leaving this `with` block joins a task whose worker holds `Sender[i64]` while `receiver` still holds `Receiver[i64]`
`?` leaves the block here
Give the receiver to a consuming worker instead, and start it first. Then guard the sender and hand each producer a share of it:
def produce(sender: Sender[i64]) -> Result[(), SendError[i64] | OverflowError[i64]]:
for n in range(5):
sender.send(n * n)?
Ok(())
def total(receiver: Receiver[i64]) -> Result[i64, OverflowError[i64]]:
mut sum = 0
while True:
match receiver.recv():
case Ok(value):
sum = sum + value
case Err(_):
break
Ok(sum)
def main() -> Result[(), SpawnError[()] | AllocError | IoError | OverflowError[i64]]:
sender, receiver = channel[i64](2)?
with spawn(total, receiver).drop_job()? as consumer:
with sender as outbox, spawn(produce, outbox.share()).drop_job()?, spawn(produce, outbox.share()).drop_job()?:
pass
print(consumer.join()?)?
Ok(())
60
Here the sender guard comes first: the producers send too, so only the
consumer, joined later, needs the sender gone. The inner block joins both
producers, then drops the last sender. The consumer sees disconnection and
returns, so consumer.join() after the block cannot hang. If a producer fails
to start, ? leaves through the same exits in the same order.
Failed starts keep their endpoints
A job that fails to start returns inside SpawnError[F], with every endpoint it
captured. ? keeps that error until the function returns, which is after the
block’s joins:
def report(receiver: Receiver[i64]) -> Result[(), IoError | OverflowError[i64]]:
mut total = 0
while True:
match receiver.recv():
case Ok(value):
total = total + value
case Err(_):
break
print(total)
def produce(sender: Sender[i64], count: i64) -> Result[(), SendError[i64] | OverflowError[i64]]:
for n in range(count):
sender.send(n * n)?
Ok(())
type Errors = AllocError | SpawnError[()] | SpawnError[OnceClosure[[], Result[(), SendError[i64] | OverflowError[i64]]]]
def main() -> Result[(), Errors]:
sender, receiver = channel[i64](2)?
count = 5
job = def once [sender, count]() -> Result[(), SendError[i64] | OverflowError[i64]]:
produce(sender, count)
with spawn(report, receiver).drop_job()?, spawn(job)?:
pass
Ok(())
leaving this `with` block joins a task whose worker holds `Receiver[i64]` while the error that `?` propagates still holds `Sender[i64]`
call `drop_job()` on the start
When you will not retry the job, call drop_job() on the start. It drops the
job, and its sender, before ? leaves, and the error becomes SpawnError[()]:
def report(receiver: Receiver[i64]) -> Result[(), IoError | OverflowError[i64]]:
mut total = 0
while True:
match receiver.recv():
case Ok(value):
total = total + value
case Err(_):
break
print(total)
def produce(sender: Sender[i64], count: i64) -> Result[(), SendError[i64] | OverflowError[i64]]:
for n in range(count):
sender.send(n * n)?
Ok(())
type Errors = AllocError | SpawnError[()]
def main() -> Result[(), Errors]:
sender, receiver = channel[i64](2)?
count = 5
job = def once [sender, count]() -> Result[(), SendError[i64] | OverflowError[i64]]:
produce(sender, count)
with spawn(report, receiver).drop_job()?, spawn(job).drop_job()?:
pass
Ok(())
30
A named start, spawn(produce, sender, count), binds the same kind of job from
its arguments, so it needs drop_job() here too.
The channel reference lists every way a function can hold an endpoint, and the programs the check rejects although they could not hang.
Handle channel backpressure
Use send_nowait when a full queue should return control to the caller. Its
Full error retains your message, so you can retry it after making space.
type MessageError = RecvError | SendError[list[i64]] | AllocError
def main() -> Result[(), AllocError | IoError | MessageError | OverflowError[i64]]:
sender, receiver = channel[list[i64]](1)?
sender.send([10]?)?
match sender.send_nowait([20, 30]?):
case Ok(_):
pass
case Err(SendError.Full(message)):
print(receiver.recv()?)?
sender.send(message)?
case Err(SendError.Disconnected(message)):
print(message)?
print(receiver.recv()?)?
drop(sender)
print(str.repr(receiver.recv())?)?
Ok(())
[10]
[20, 30]
Result[list[i64], RecvError].Err(RecvError.Disconnected)
recv_nowait() similarly reports RecvError.Empty when the channel is connected
but has no message ready. Neither nowait operation waits for the queue to change
state, though both synchronize access to the queue.
If you need only part of a stream, the receiver guard handles shutdown:
def produce(sender: Sender[i64]) -> Result[(), AllocError | IoError | SendError[i64] | OverflowError[i64]]:
for n in range(10, 100):
sender.send(n * n)?
Ok(())
def first_square() -> Result[i64, SpawnError[()] | AllocError | RecvError]:
sender, receiver = channel[i64](1)?
with spawn(produce, sender).drop_job()?, receiver as inbox:
return Ok(inbox.recv()?)
def main() -> Result[(), SpawnError[()] | AllocError | RecvError | IoError | OverflowError[i64]]:
print(first_square()?)?
Ok(())
100
Returning drops the receiver first. The producer’s next send reports
disconnection; its ? ends the worker. The task scope then joins it and discards
its unclaimed result.
Channel capacity has type Positive[i64]. A positive literal converts at
compile time; for configuration read at run time, pass
Positive.try_from(capacity)?. Construction then reports only allocation
failure. See integer domains.
Bound your waits
Use a timeout when your program needs control back even if no message arrives.
Timeouts are u64 milliseconds; zero performs an immediate check.
def main() -> Result[(), IoError | RecvTimeoutError | SendTimeoutError[i64] | AllocError]:
sender, receiver = channel[i64](1)?
match receiver.recv_timeout_ms(0):
case Ok(value):
print(value)?
case Err(RecvTimeoutError.TimedOut):
print("nothing yet")?
case Err(RecvTimeoutError.Disconnected):
print("senders finished")?
sender.send_timeout_ms(42, 0)?
print(receiver.recv_timeout_ms(0)?)?
Ok(())
nothing yet
42
send_timeout_ms(value, milliseconds) returns an unsent value in
SendTimeoutError[T].TimedOut(value) or .Disconnected(value). You can retry,
store, or drop it. A timeout does not discard the message.
Timeouts limit waiting, not total wall-clock execution. Lock contention and scheduling can delay return. A message already available at the check is received even with a zero budget. To bound several waits by one total budget, share a deadline.
Share one deadline across waits
A timeout bounds one wait. When a task makes several waits, give it a
Deadline from std.time: one total
budget that every wait draws from.
import std.sync
from std.time import Deadline, monotonic_now, deadline_after, milliseconds
def collect(inbox: &Receiver[i64], deadline: &Deadline, cancel: &CancellationToken) -> Result[i64, OverflowError[i64]]:
mut total = 0
while True:
match std.sync.recv_before(inbox, deadline, cancel):
case Ok(value):
total += value
case Err(std.sync.WaitError.TimedOut):
break
case Err(std.sync.WaitError.Disconnected) | Err(std.sync.WaitError.Cancelled):
break
Ok(total)
def main() -> Result[(), AllocError | IoError | RangeError | SendError[i64] | OverflowError[i64]]:
sender, receiver = channel[i64](4)?
sender.send(1)?
sender.send(2)?
sender.send(3)?
deadline = deadline_after(&monotonic_now(), &milliseconds(20))?
cancel = CancellationToken()?
print(collect(&receiver, &deadline, &cancel)?)?
print(deadline.expired(&monotonic_now()))?
print(deadline.remaining(&monotonic_now()).as_timeout_ms())?
Ok(())
6
True
0
recv_before checks caller cancellation between bounded waits and shares
one deadline. The loop makes four waits against one 20-millisecond budget instead of
granting each wait its own. remaining takes the current instant and returns
zero once the deadline has passed. A zero budget still receives a message that
is already queued, so the three sent values always arrive.
as_timeout_ms() is available for the low-level _timeout_ms methods.
It converts a span to the u64 milliseconds that
timed waits accept. It rounds up: a wait never returns
before its deadline, and a remainder below a millisecond does not turn the loop
into a zero-timeout spin.
Sleep that can be cancelled
sleep cannot be interrupted. To pause until either a span passes or someone
asks you to stop, wait on a
cancellation token with the span as the budget:
from std.time import seconds
def pause(stop: &CancellationToken) -> bool:
stop.wait_timeout_ms(seconds(3600).as_timeout_ms())
def main() -> Result[(), AllocError | IoError | SpawnError[()]]:
stop = CancellationToken()?
with spawn(pause, &stop)? as task:
stop.cancel()
print(task.join())?
Ok(())
True
The worker borrows the token and asks for an hour. It returns as soon as
cancel() runs, even if the request comes before the wait starts. The result
is true when cancellation ended the wait and false when the span elapsed.
Deadlines use the monotonic clock, so changing the system time neither extends nor shortens them. They limit waiting, not execution: a wait can return late on a busy machine. See the time reference.
Reuse workers with an executor
A thread pool reuses workers for many jobs. Set its worker count and queue capacity at construction:
def square(value: i64) -> Result[i64, OverflowError[i64]]:
Ok(value * value)
type PoolJobError = ParallelError[OverflowError[i64]] | FutureError | SubmitError[()] | PoolError
def main() -> Result[(), IoError | PoolJobError | OverflowError[i64]]:
with ThreadPoolExecutor(2, 4)? as pool:
first = pool.submit(square, 6).drop_job()?
second = pool.submit(square, 9).drop_job()?
print(first.result()??)?
print(second.result()??)?
print(pool.map_result(square, range(6))?)?
Ok(())
36
81
[0, 1, 4, 9, 16, 25]
submit returns an owned Future[Result[i64, OverflowError[i64]]];
result() waits and consumes it. The first ? handles the future wait,
and the second propagates the worker’s overflow error.
A failed submission returns SubmitError, which owns the unstarted job.
drop_job() drops that job, as for thread starts, and
leaves SubmitError[()], which still says why submission failed.
Submit jobs before waiting for their results to let them overlap. Leaving with
waits for accepted jobs and joins every worker, including on early return or
propagated error.
map_result returns successful values in input order, or the first worker
error in input order. map instead keeps every returned value, including
any Result wrappers. It accepts an owned list or an integer range
and a named function or owned reusable immutable closure, keeping a bounded
window of jobs outstanding. Job and
result storage can fail to allocate.
def main() -> Result[(), AllocError | IoError | ParallelError[OverflowError[i64]] | PoolError]:
with ThreadPoolExecutor(2, 2)? as pool:
factor = 3
scale = def [factor](value: i64) -> Result[i64, OverflowError[i64]]:
Ok(value * factor)
print(pool.map_result(scale, range(4))?)?
Ok(())
[0, 3, 6, 9]
Mapping consumes scale and shares its one immutable environment until all jobs
finish. Mutable, one-shot, or borrowed closures cannot be shared this way.
map_result accepts the same workers when they return Result[T, E].
You can submit a closure to move several values into a job explicitly:
type PoolJobError = FutureError | SubmitError[()] | PoolError
def main() -> Result[(), AllocError | IoError | PoolJobError]:
with ThreadPoolExecutor(2, 2)? as pool:
values = [3, 6, 9]?
future = pool.submit(def once [values]() -> list[i64]: values).drop_job()?
print(future.result()?)?
Ok(())
[3, 6, 9]
Pool jobs own their inputs; use spawn when a worker needs to borrow local data.
A worker may use channels, following the same habits as a
task, but cannot hold an executor or future.
Keep submission and future waits on the owning thread to avoid pool dependency
deadlocks. Files, native effects and foreign calls follow the same rules as for scoped workers.
Dropping a future leaves its job running; an unclaimed result is cleaned up. A future may also outlive the pool’s scope: shutdown finishes the job, and the handle retains the result until you retrieve or drop it.
Keep the job instead when you want to retry it or run it locally; a submission backpressure lesson shows how.
Handle executor outcomes
A job returning Result keeps its application errors separate from submission
failure or cancellation. Retrieve its result, then handle the application result:
def divide(value: i64) -> Result[i64, str]:
divisor = NonZero.try_from(value).map_error(def(error: RangeError) -> str: "zero divisor")?
# 24 // a nonzero i64 cannot overflow: the dividend is not i64.MIN.
unchecked:
Ok(24 // divisor)
type PoolJobError = PoolMapError | FutureError | SubmitError[()] | PoolError
def main() -> Result[(), AllocError | IoError | PoolJobError]:
with ThreadPoolExecutor(2, 2)? as pool:
future = pool.submit(divide, 0).drop_job()?
match future.result()?:
case Ok(value):
print(value)?
case Err(message):
print(message)?
print(pool.map(divide, [2, 0, 4]?)?)?
Ok(())
zero divisor
[Result[i64, str].Ok(12), Result[i64, str].Err("zero divisor"), Result[i64, str].Ok(6)]
The example prints the application error to show it; a program that cannot
continue propagates it instead. To propagate both layers from a single future,
write (future.result()?)?. Mapping preserves each application outcome in
input order.
Share state with pool workers
Scoped threads can borrow an atomic or a mutex, but executor jobs must own their
inputs. Shared[T] gives a value several owners. Hand each job its own owner
with share():
from std.atomic import Atomic, Ordering
from std.sync import Mutex
class Stats:
count: i64
total: i64
def work(hits: Shared[Atomic[u64]], stats: Shared[Mutex[Stats]], value: i64) -> Result[i64, OverflowError[i64]]:
for _ in range(1000):
hits.fetch_add(1, Ordering.Relaxed)
with stats.lock() as data:
count = data.count + 1
total = data.total + value
data.count = count
data.total = total
Ok(value)
type PoolJobError = FutureError | SubmitError[()] | PoolError
def main() -> Result[(), AllocError | IoError | PoolJobError | OverflowError[i64]]:
hits = Shared[Atomic[u64]](Atomic(0))?
stats = Shared[Mutex[Stats]](Mutex(Stats(0, 0)))?
with ThreadPoolExecutor(2, 4)? as pool:
first = pool.submit(work, hits.share(), stats.share(), 3).drop_job()?
second = pool.submit(work, hits.share(), stats.share(), 4).drop_job()?
print(first.result()?? + second.result()??)?
print(hits.load(Ordering.Acquire))?
with stats.lock() as data:
print(data.count)?
print(data.total)?
Ok(())
7
2000
2000
7000
Shared[T](value) moves the value into one allocation, which can fail.
share() returns another owner of the same value without allocating. Each
owner is dropped like any other value; the last one to drop, on whichever
thread, runs the value’s cleanup and frees it.
The handle lends its content: hits.fetch_add(...) and stats.lock() reach
the atomic and the mutex through it, and &*hits borrows the atomic itself.
The content is shared, never exclusive, so it changes only through an atomic
or a mutex:
class Stats:
count: i64
def main() -> Result[(), AllocError]:
stats = Shared(Stats(0))?
stats.count = 1
Ok(())
a Shared owner gives shared access only; mutate its content through an Atomic or a Mutex inside it
Handles cannot be compared or printed; read the content instead. Take another
owner with share(), not copy(). Copying a value that holds a handle, such
as a job’s (str, Shared[Atomic[u64]]) input, shares that handle the same way.
A value cannot hold a share of itself through a mutex, since it would never be
released: link back to it with an index into a list that owns the values.
The compiler’s lock checks see through handles. Shutting a pool down while
holding a lock on a mutex one of its jobs can reach is rejected, as is waiting
for such a job’s result. Wait after the lock block ends, as main reads
stats only after the pool’s block. Nesting
a lock through a handle inside a lock on another Mutex[T] of the same type is
rejected too, because two handles can name one mutex. See shared owners
for the full rules.
Cancel queued work and bound a future wait
future.cancel() returns True when it prevents a queued job from starting.
It cannot interrupt running work. A cancelled future reports
FutureError.Cancelled from result(). future.done() observes completion or
cancellation without consuming the handle. pool.cancel_pending() cancels
every job still queued without waiting, and a following pool.shutdown() waits
for running jobs; ordinary scope exit drains all work.
For a future, future.wait_timeout_ms(milliseconds) returns a boolean: true means
completed or cancelled. This only observes the future; call result() afterward
to take its result. A false return leaves the job running or queued.
Jobs that send
Shutdown and result() both wait for a job, so a job that sends follows the
same habits as a task. When you do not wait for its result
inside the block, guard the receiver as the last manager of the pool’s with:
def produce(sender: Sender[i64]) -> Result[(), SendError[i64] | OverflowError[i64]]:
for n in range(5):
sender.send(n * n)?
Ok(())
def drain(inbox: &Receiver[i64]) -> Result[i64, OverflowError[i64]]:
mut total = 0
while True:
match inbox.recv():
case Ok(value):
total = total + value
case Err(_):
break
Ok(total)
def main() -> Result[(), AllocError | IoError | PoolError | SubmitError[()] | OverflowError[i64]]:
sender, receiver = channel[i64](2)?
with ThreadPoolExecutor(1, 1)? as pool, receiver as inbox:
pool.submit(produce, sender).drop_job()?
print(drain(inbox)?)?
Ok(())
30
That guard drops the receiver at shutdown, which is too late for a result()
inside the block:
def produce(sender: Sender[i64]) -> Result[(), SendError[i64] | OverflowError[i64]]:
for n in range(5):
sender.send(n * n)?
Ok(())
def drain(inbox: &Receiver[i64]) -> Result[i64, OverflowError[i64]]:
mut total = 0
while True:
match inbox.recv():
case Ok(value):
total = total + value
case Err(_):
break
Ok(total)
def main() -> Result[(), AllocError | FutureError | IoError | PoolError | SendError[i64] | SubmitError[()] | OverflowError[i64]]:
sender, receiver = channel[i64](2)?
with ThreadPoolExecutor(1, 1)? as pool, receiver as inbox:
future = pool.submit(produce, sender).drop_job()?
print(drain(inbox)?)?
future.result()??
Ok(())
this result() waits for a job that holds `Sender[i64]` while `receiver` still holds `Receiver[i64]`
To wait for the result, end the receiver’s own guard first:
def produce(sender: Sender[i64]) -> Result[(), SendError[i64] | OverflowError[i64]]:
for n in range(5):
sender.send(n * n)?
Ok(())
def drain(inbox: &Receiver[i64]) -> Result[i64, OverflowError[i64]]:
mut total = 0
while True:
match inbox.recv():
case Ok(value):
total = total + value
case Err(_):
break
Ok(total)
def main() -> Result[(), AllocError | FutureError | IoError | PoolError | SendError[i64] | SubmitError[()] | OverflowError[i64]]:
sender, receiver = channel[i64](2)?
with ThreadPoolExecutor(1, 1)? as pool:
future = pool.submit(produce, sender).drop_job()?
mut total = 0
with receiver as inbox:
total = drain(inbox)?
future.result()??
print(total)?
Ok(())
30
A failed submit(...)? has submitted nothing, so it can come before the guard.
done() and wait_timeout_ms() do not prove the channel protocol to the checker:
the result wait is rejected while a matching receiver lives, even after they
report completion.
A worker that may run indefinitely needs a cooperative shutdown protocol: scope exit still waits after a timed wait returns.
Stop work cooperatively
Use a cancellation token to ask a running worker to stop. Share the token explicitly; all handles observe the same request.
def work(stop: CancellationToken, ready: Sender[i64]) -> Result[i64, SendError[i64]]:
ready.send(1)?
stop.wait()
# Release resources and return through ordinary control flow.
Ok(42)
type CancellableWorkError = SendError[i64] | FutureError | RecvError | SubmitError[()] | AllocError | PoolError
def main() -> Result[(), AllocError | IoError | CancellableWorkError]:
stop = CancellationToken()?
ready, started = channel[i64](1)?
with ThreadPoolExecutor(1, 1)? as pool, stop.cancel_on_exit():
future = pool.submit(work, stop.share(), ready).drop_job()?
with started as inbox:
inbox.recv()?
stop.cancel()
print(future.result()??)?
print(stop.is_cancelled())?
Ok(())
42
True
The worker reports that it started, then waits for the request.
stop.cancel_on_exit() cancels stop however the block ends, including
through a ?, so the pool’s shutdown never waits for a job that is still
waiting for the request; cancel on every exit shows
the mistake it prevents. The explicit stop.cancel() comes before
future.result(), which waits inside the block.
The receiver’s own guard ends before future.result() waits, and it also
covers the ? on recv(), as in
jobs that send.
Without the guard, an exit from the pool’s block would reach shutdown with
started alive, and Plenty rejects that:
def report(ready: Sender[i64]) -> Result[(), SendError[i64]]:
ready.send(1)
def main() -> Result[(), AllocError | PoolError | RecvError | SubmitError[()]]:
ready, started = channel[i64](1)?
with ThreadPoolExecutor(1, 1)? as pool:
pool.submit(report, ready).drop_job()?
started.recv()?
Ok(())
leaving its scope shuts down this pool after a job that holds `Sender[i64]` was submitted, while `started` still holds `Receiver[i64]`
As with task joins, the compiler compares message types rather than counting messages, so it asks for the guard even though this job’s one send always fits the queue.
Cancel on every exit
A worker that waits for a token keeps a join or pool shutdown waiting until
someone cancels it. When the cancel comes after a ?, return, or break,
that exit skips it, and Plenty rejects the program:
def serve(stop: CancellationToken) -> ():
stop.wait()
def prepare() -> Result[(), str]:
Ok(())
def main() -> Result[(), AllocError | SpawnError[()] | IoError | str]:
stop = CancellationToken()?
with spawn(serve, stop.share()).drop_job()?:
prepare()?
print("ready")?
stop.cancel()
Ok(())
leaving this `with` block joins a task that holds `stop` before this function cancels it
`?` leaves the block here
Add stop.cancel_on_exit() as the last manager, after the task. On every exit
it cancels stop before the task is joined, so the explicit cancel() is no
longer needed:
def serve(stop: CancellationToken) -> ():
stop.wait()
def prepare() -> Result[(), str]:
Ok(())
def main() -> Result[(), AllocError | SpawnError[()] | IoError | str]:
stop = CancellationToken()?
with spawn(serve, stop.share()).drop_job()?, stop.cancel_on_exit():
prepare()?
print("ready")?
Ok(())
ready
The guard holds a share of the token and allocates nothing.
The same mistake can happen the other way round. Here the parent waits for
the worker to cancel stop when it finishes, but a ? in the worker skips
the cancel:
def prepare() -> Result[(), str]:
Ok(())
def client(stop: CancellationToken) -> Result[(), str]:
prepare()?
stop.cancel()
Ok(())
def main() -> Result[(), AllocError | SpawnError[()] | IoError | str]:
stop = CancellationToken()?
with spawn(client, stop.share()).drop_job()? as task:
stop.wait()
task.join()?
Ok(())
`?` leaves this function without cancelling `stop`, which its normal end cancels
Put the worker’s body in with stop.cancel_on_exit():, so the parent’s
stop.wait() returns however the worker ends. task.join()? after the wait
then reports the worker’s error, if it had one:
def prepare() -> Result[(), str]:
Ok(())
def client(stop: CancellationToken) -> Result[(), str]:
with stop.cancel_on_exit():
prepare()?
Ok(())
def main() -> Result[(), AllocError | SpawnError[()] | IoError | str]:
stop = CancellationToken()?
with spawn(client, stop.share()).drop_job()? as task:
stop.wait()
task.join()?
print("client finished")?
Ok(())
client finished
The cancellation reference describes the check and its limits.
Checking for cancellation
Check stop.is_cancelled() between work units and return an appropriate result.
Cancellation does not interrupt instructions or skip cleanup.
cancel() is safe to repeat. Dropping a token handle does not cancel work.
wait_timeout_ms(milliseconds) waits for a request and returns whether it arrived.
A token does not wake an ordinary recv() call. Use a shutdown message or a
timed receive when a worker must periodically check for
cancellation. Automatic joining still waits for that worker to return.
In a receive loop, check is_cancelled() between bounded receives. Avoid a
zero-timeout retry loop that consumes a CPU while idle.
Recover a job when submission cannot proceed
submit waits for queue space. submit_nowait instead returns the unstarted job
in SubmitError.Full(job) when full. Every submission error owns the job, so a
generic helper can run it locally when submission is unavailable:
def run_or_submit[F: OnceCallable[[], i64]](
pool: &ThreadPoolExecutor, job: F
) -> Result[i64, FutureError]:
match pool.submit_nowait(job):
case Ok(future):
future.result()
case Err(
SubmitError.Full(job)
| SubmitError.Shutdown(job)
| SubmitError.OutOfMemory(job)
| SubmitError.CapacityOverflow(job)
):
Ok(job())
def main() -> Result[(), AllocError | IoError | FutureError | PoolError]:
with ThreadPoolExecutor(1, 1)? as pool:
pool.shutdown()
number = 42
job = def once [number]() -> i64:
number
print(run_or_submit(pool, job)?)?
Ok(())
42
When you do not need the job back, call drop_job() on the submission, as in
the executor lesson: the error keeps its variant
and names no job. Erasing it into Failure also drops the job but loses the
error type; that is a last resort.
Map fallible work in parallel
Use map_result when each worker returns a Result and you want either all the
successful values or one error. The error keeps its original type.
def square(n: i64) -> Result[i64, str | OverflowError[i64]]:
if n < 0:
Err("negative input")
else:
Ok(n * n)
def main() -> Result[(), AllocError | IoError | ParallelError[str | OverflowError[i64]] | PoolError]:
with ThreadPoolExecutor(2, 2)? as pool:
print(pool.map_result(square, range(5))?)?
match pool.map_result(square, [3, -1, -2]?):
case Ok(values):
print(values)?
case Err(ParallelError.Worker(error)):
match error:
case message: str:
print(message)?
case overflow: OverflowError[i64]:
return Err(ParallelError[str | OverflowError[i64]].Worker(overflow))
case Err(error):
return Err(error)
Ok(())
[0, 1, 4, 9, 16]
negative input
The fixed negative inputs deliberately demonstrate a handled worker error;
a program that cannot recover should propagate that error too. Allocation
failure, arithmetic overflow, and shutdown are not recovered here and leave main with an error.
Outputs keep input order. If several inputs fail, the earliest input’s error wins;
the fastest worker does not decide. map_result waits for accepted jobs and
cleans up partial results before returning. Later inputs may already have run, so
an error cannot undo their effects. Do not make a worker wait for an input that
might never be submitted.
Both map and map_result consume input lists and use a bounded job window.
Use ordinary map when you want a list containing every worker’s Result,
including all errors. Both operations can report allocation failure.
The worker preserves both its input error and arithmetic error in an
anonymous sum. Matching the str member
prints its message rather than the structural representation of the sum.
Combine values in parallel
reduce_tree combines adjacent pairs on the pool, then combines their results.
The function takes two values and returns either one value or a Result of that
value type. Give arithmetic callbacks their own fallible contract.
def add[T: IntType](left: T, right: T) -> Result[T, OverflowError[T]]:
Ok(left + right)
def main() -> Result[(), AllocError | IoError | PoolMapError | PoolError | OverflowError[i64]]:
with ThreadPoolExecutor(2, 2)? as pool:
print(pool.reduce_tree(add, range(1, 6))?)?
print(pool.reduce_tree(add, range(0))?)?
Ok(())
Option[i64].Some(15)
Option[i64].Nothing
Empty input returns Nothing; otherwise the result is Some(value). The outer
Result reports allocation failure, a closed pool, or the callback’s precise
error. A fallible callback gives Result[Option[T], PoolMapError | E]. Lists
are consumed without copying their elements; ranges remain ordinary copyable
values.
The tree is fixed: [a, b, c, d, e] becomes [f(a, b), f(c, d), e], then
[f(f(a, b), f(c, d)), e], then one value. More workers do not change that
grouping. An odd final element advances unchanged.
Choose an operation whose grouping is suitable for your task. Subtraction and floating-point addition can give different answers from a serial left-to-right loop. Keep a serial loop when that exact order matters. Worker side effects need not follow the tree order. For fallible callbacks, errors are selected by tree level, then by left-to-right pair, not by completion time. Already accepted jobs finish before the error returns; later levels do not run. Fallible callbacks use one synchronization round per tree level rather than the multi-level jobs available to infallible callbacks.
For larger inputs, reduction allocates scratch storage proportional to the input length and only a bounded window of jobs. Empty and singleton reductions allocate nothing. If allocation fails, the operation waits for accepted jobs and cleans up every intermediate value before returning.
Run a pipeline on a pool
.par(&pool) runs the rest of a pipeline on a
pool’s workers. Steps after it must be map, filter or filter_value, and the pipeline must
end in a terminal that combines the chunks: collect, try_collect, count,
any, all, sum, min, max or reduce.
def score(n: i64) -> i64:
seven: NonZero[i64] = 7
remainder = n % seven
# Floor remainder is 0..6, so its square fits even for arbitrary n.
unchecked:
remainder * remainder % seven
def positive(n: &i64) -> bool:
*n > 0
def main() -> Result[(), AllocError | IoError | PoolMapError | PoolError | OverflowError[i64]]:
with ThreadPoolExecutor(4, 4)? as pool:
scores = iter(range(0, 10)).par(&pool).map(score).filter(positive).collect()?
print(scores)?
print(iter(range(0, 10)).par(&pool).map(score).sum()?)?
Ok(())
[1, 4, 2, 2, 4, 1, 1, 4]
19
The pool splits the input into one contiguous chunk per worker, and each
chunk runs the fused steps as one job. Results come back in input order. Every
terminal returns a Result, because submitting work can fail or the pool can
be shut down. Integer sum also returns OverflowError[T] for the first
serial left-to-right overflowing addition, including its exact wrapped
payload. It retains the mapped numbers until every job has finished, then adds
them in input order; callbacks run once, not again to recover the error. This
costs temporary storage proportional to the surviving items. Caller arithmetic
modes do not change this public result.
A list source works the same way. iter(xs) consumes the list; iter(&xs)
borrows it, yields &i64 items as a serial pipeline does, and leaves xs
usable afterward. The chunks read the list in place instead of copying it.
def big(n: &i64) -> bool:
*n > 10
def main() -> Result[(), AllocError | IoError | PoolMapError | PoolError | OverflowError[i64]]:
with ThreadPoolExecutor(4, 4)? as pool:
xs = [3, 14, 15, 9, 26]?
print(iter(&xs).par(&pool).filter(big).count()?)?
print(iter(&xs).par(&pool).sum()?)?
print(xs)?
Ok(())
3
67
[3, 14, 15, 9, 26]
Callbacks run concurrently. Use named functions or reusable closures that own
their captures and do not mutate captured bindings. Side effects can occur in
any chunk order. Put enumerate before .par:
the chunk offsets carry the index. An overflowing enumeration index still
terminates the process (even in a checked: caller); it is separate from a
recoverable sum overflow.
The chunking is fixed by the input length and the pool’s worker count. A
floating-point sum adds each chunk, then adds the chunk totals, so it can
differ from a serial sum. reduce groups the same way: each chunk, then the
chunk results in order. Pass chunks=N to fix the count, for example
.par(&pool, chunks=8), when the same answer is needed from pools of any
size. chunks has type Positive[i64]; a computed count uses
Positive.try_from(count)?, which rejects zero or negative values before
the pipeline starts. The reference
lists the rules and error types.
Receive from either channel
select_recv waits for a message from either of two channels. Their message
types may differ; match the returned Selected value to find which one arrived.
Both receivers are borrowed implicitly; explicit & arguments also work.
type ChannelSelectionError = RecvError | SelectError | SendError[i64] | SendError[str] | AllocError
def main() -> Result[(), IoError | ChannelSelectionError]:
commands, control = channel[str](1)?
numbers, data = channel[i64](1)?
commands.send("stop")?
numbers.send(42)?
match select_recv(control, data)?:
case Selected.First(command):
print(command)?
case Selected.Second(number):
print(number)?
print(data.recv()?)?
Ok(())
stop
42
When both channels have messages, the first takes priority. The other message stays in its queue. This can be useful for shutdown commands, but constant traffic on the first channel can starve the second.
select_recv_nowait(&first, &second) returns SelectError.Empty if neither has a
message. select_recv_timeout_ms(&first, &second, milliseconds) instead waits up
to that budget and may return SelectError.TimedOut.
Selection skips a drained channel whose senders have all gone away. It reports
SelectError.Disconnected only when both channels are drained and disconnected.
The receivers remain yours after every outcome.
Serve a TCP connection
std.net opens TCP and UDP sockets. Every call
that waits for the network takes a deadline and a cancellation token, the same
as the bounded I/O waits, so a worker serving a connection
cannot hang forever.
import std.bytes
from std.bytes import Bytes
from std.net import NetError, Shutdown, TcpListener, connect, listen, loopback_v4
from std.sync import CancellationToken
from std.time import Deadline, deadline_after, milliseconds, monotonic_now
# Echoes one connection until the client stops sending.
def echo(listener: &TcpListener, deadline: &Deadline, cancel: &CancellationToken) -> Result[i64, AllocError | NetError | std.bytes.BytesError | OverflowError[i64]]:
mut stream = listener.accept(deadline, cancel)?
mut buffer = std.bytes.with_capacity(64)?
buffer.resize(64, 0u8)?
mut total = 0
while True:
count = stream.read_before(&mut buffer, 0, 64, deadline, cancel)?
if count == 0:
return Ok(total)
stream.write_all_before(&buffer[:count], deadline, cancel)?
total = total + count
Ok(total)
type SocketOperationError = RangeError | NetError | std.bytes.BytesError
def main() -> Result[(), SpawnError[()] | IoError | AllocError | SocketOperationError | OverflowError[i64]]:
cancel = CancellationToken()?
deadline = deadline_after(&monotonic_now(), &milliseconds(5000))?
mut listener = listen(&loopback_v4(0u16), 8)?
address = listener.local_address()?
with spawn(echo, &listener, &deadline, &cancel)? as server:
mut stream = connect(&address, &deadline, &cancel)?
message = std.bytes.from_text("hello")?
stream.write_all_before(&message, &deadline, &cancel)?
stream.shutdown(Shutdown.Write)?
mut reply = std.bytes.with_capacity(16)?
reply.resize(16, 0u8)?
count = stream.read_before(&mut reply, 0, 16, &deadline, &cancel)?
print(reply.to_text_range(0, count)?)?
print(server.join()?)?
listener.close()?
match connect(&address, &deadline, &cancel):
case Err(NetError.Refused):
print("refused")?
case _:
print("unexpected")?
Ok(())
hello
5
refused
Port 0 asks the system for a free port, and local_address reports which one
it chose. shutdown(Shutdown.Write) tells the server that no more data is
coming, so its read returns 0 and it returns how many bytes it echoed. After
the listener is closed, nothing listens on the port, so the last attempt is
NetError.Refused: an error value to handle, not a crash.
read_before makes one read, which may return fewer bytes than were sent; a
real client loops until it has what it needs.
Wait on many connections
A worker per connection is simple, but a server with many idle connections
can wait on all of them in one call with a
selector. Each socket is registered
under a key you choose, and wait reports which keys are ready:
import std.ready
import std.net
import std.bytes
from std.net import connect, listen, loopback_v4
from std.ready import Interest, selector
from std.sync import CancellationToken
from std.time import deadline_after, milliseconds, monotonic_now
type SocketOperationError = std.bytes.BytesError | std.ready.WaitError | std.net.NetError | RangeError
def main() -> Result[(), AllocError | IoError | SocketOperationError]:
cancel = CancellationToken()?
deadline = deadline_after(&monotonic_now(), &milliseconds(5000))?
listener = listen(&loopback_v4(0u16), 8)?
address = listener.local_address()?
quiet = connect(&address, &deadline, &cancel)?
mut chatty = connect(&address, &deadline, &cancel)?
first = listener.accept(&deadline, &cancel)?
mut second = listener.accept(&deadline, &cancel)?
mut chosen = selector(8)?
first.watch(&mut chosen, 1u64, Interest.Read)?
second.watch(&mut chosen, 2u64, Interest.Read)?
chatty.write_all_before(&b"x", &deadline, &cancel)?
mut buffer = std.bytes.with_capacity(16)?
buffer.resize(16, 0u8)?
while True:
count = chosen.wait(&deadline, &cancel)?
match second.read_now(&mut buffer, 0, 16):
case Ok(received):
print(count, chosen.event(0)?.key)?
print(buffer.to_text_range(0, received)?)?
break
case Err(std.net.NetError.WouldBlock):
continue
case Err(error):
return Err(error)
Ok(())
1 2
x
Only the second connection has input, so the wait reports one event, key 2.
read_now makes a single nonblocking read without another readiness wait.
If readiness has become stale, WouldBlock sends the loop back to the selector.
This example has only one possible sender and one byte of input. A server
dispatches each event by its key, handles zero from a nonempty read as EOF,
and keeps any unsent suffix after a partial write_now. accept_now similarly
accepts one queued connection or returns WouldBlock without waiting.
Name lookup, UDP datagrams and IPv6 addresses are in the
reference.
Encrypt a TCP connection
std.tls wraps a connected TCP stream in TLS. The client
checks that the server’s certificate chains to a trust root it chose and that
the certificate names the host it meant to reach. Every handshake, read, and
write takes the same deadline and cancellation token as the TCP calls.
A server needs a certificate chain and its private key, as PEM text.
roots_from_pem turns CA certificates into TrustRoots; real clients
usually call std.tls.system_roots() instead, which reads the operating
system’s bundle. This lesson embeds a test CA so that it runs offline.
import std.bytes
import std.tls
from std.net import NetError, TcpListener, connect, listen, loopback_v4
from std.sync import CancellationToken
from std.time import Deadline, deadline_after, milliseconds, monotonic_now
from std.tls import ServerConfig, TlsError
# Test-only certificates: a CA, and a leaf it signed for localhost and
# 127.0.0.1. Never use a published key outside an example.
def ca_pem() -> str:
return "-----BEGIN CERTIFICATE-----\nMIIBmjCCAT+gAwIBAgIULOQ2Qxr8CVZY2bG0uRYKKEdMD3swCgYIKoZIzj0EAwIw\nGTEXMBUGA1UEAwwOUGxlbnR5IFRlc3QgQ0EwIBcNMjQwMTAxMDAwMDAwWhgPMjEy\nNDAxMDEwMDAwMDBaMBkxFzAVBgNVBAMMDlBsZW50eSBUZXN0IENBMFkwEwYHKoZI\nzj0CAQYIKoZIzj0DAQcDQgAEkiB3Ogehrrt6YbQghwDPv6tkhE+z9zTCVih8fvtR\nZ1/691Nfh7/J9IgaNgNSTMJjGTolzGnjaCUDqoARlFyM+6NjMGEwHQYDVR0OBBYE\nFOTFBDgWXHj/qmVJDMlAHYCEZFX5MB8GA1UdIwQYMBaAFOTFBDgWXHj/qmVJDMlA\nHYCEZFX5MA8GA1UdEwEB/wQFMAMBAf8wDgYDVR0PAQH/BAQDAgIEMAoGCCqGSM49\nBAMCA0kAMEYCIQD41ATQnIjKkKMLQAb9AyoyOFmljYZyz348XJEIbqMgKwIhAL3m\nmWd8On5gvkfytspatH5mDWOcSyW+au7vuIC2gHhO\n-----END CERTIFICATE-----\n"
def leaf_pem() -> str:
return "-----BEGIN CERTIFICATE-----\nMIIBnDCCAUKgAwIBAgIBATAKBggqhkjOPQQDAjAZMRcwFQYDVQQDDA5QbGVudHkg\nVGVzdCBDQTAgFw0yNDAxMDEwMDAwMDBaGA8yMTI0MDEwMTAwMDAwMFowFDESMBAG\nA1UEAwwJbG9jYWxob3N0MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAE8THNQeTi\nJ6Eq0tSxtHW5Q4TCsbIqB3MOtM98ERyKgl6kHeHCeDZYKx8B+VRGT2jkWuTeN24Q\nljwUfh7uh7Iaz6N+MHwwGgYDVR0RBBMwEYIJbG9jYWxob3N0hwR/AAABMAkGA1Ud\nEwQCMAAwEwYDVR0lBAwwCgYIKwYBBQUHAwEwHQYDVR0OBBYEFEIFC2Fyd4cfAKTx\nMiNCMdDkqDNbMB8GA1UdIwQYMBaAFOTFBDgWXHj/qmVJDMlAHYCEZFX5MAoGCCqG\nSM49BAMCA0gAMEUCIAUF3HH43lnirgSN1Vr4twZrFmXKBxlbYNQni/h4ZaxhAiEA\n5rD/yE1B4IsbEGCMlyAeX507leXfGY9+wrHaxikU/w0=\n-----END CERTIFICATE-----\n"
def leaf_key() -> str:
return "-----BEGIN PRIVATE KEY-----\nMIGHAgEAMBMGByqGSM49AgEGCCqGSM49AwEHBG0wawIBAQQglvSEE3H7v+mInxSF\nAyWFDKYpzpXr20wH8mfFY3bCaB2hRANCAATxMc1B5OInoSrS1LG0dblDhMKxsioH\ncw60z3wRHIqCXqQd4cJ4NlgrHwH5VEZPaORa5N43bhCWPBR+Hu6HshrP\n-----END PRIVATE KEY-----\n"
# Answers one message on one connection.
def echo(listener: &TcpListener, config: &ServerConfig, deadline: &Deadline, cancel: &CancellationToken) -> Result[i64, AllocError | TlsError | std.bytes.BytesError]:
tcp = listener.accept(deadline, cancel).map_error(TlsError.Net)?
mut stream = std.tls.accept(tcp, config, deadline, cancel)?
mut buffer = std.bytes.with_capacity(64)?
buffer.resize(64, 0u8)?
count = stream.read_before(&mut buffer, 0, 64, deadline, cancel)?
stream.write_all_before(&buffer[:count], deadline, cancel)?
stream.close_before(deadline, cancel)?
Ok(count)
type TlsConnectionError = RangeError | TlsError | NetError | std.bytes.BytesError
def main() -> Result[(), SpawnError[()] | IoError | AllocError | TlsConnectionError]:
cancel = CancellationToken()?
deadline = deadline_after(&monotonic_now(), &milliseconds(5000))?
roots = std.tls.roots_from_pem(&ca_pem())?
client = std.tls.client_config(&roots)?
server = std.tls.server_config(&leaf_pem(), &leaf_key())?
listener = listen(&loopback_v4(0u16), 8)?
address = listener.local_address()?
with spawn(echo, &listener, &server, &deadline, &cancel)? as worker:
tcp = connect(&address, &deadline, &cancel)?
mut stream = std.tls.connect(tcp, &client, "localhost", &deadline, &cancel)?
message = std.bytes.from_text("hello")?
stream.write_all_before(&message, &deadline, &cancel)?
mut reply = std.bytes.with_capacity(16)?
reply.resize(16, 0u8)?
count = stream.read_before(&mut reply, 0, 16, &deadline, &cancel)?
print(reply.to_text_range(0, count)?)?
print(stream.read_before(&mut reply, 0, 16, &deadline, &cancel)?)?
stream.close_before(&deadline, &cancel)?
print(worker.join()?)?
with spawn(echo, &listener, &server, &deadline, &cancel)? as worker:
tcp = connect(&address, &deadline, &cancel)?
match std.tls.connect(tcp, &client, "example.test", &deadline, &cancel):
case Err(error):
print(error)?
case Ok(_):
print("unexpected")?
match worker.join():
case Err(error):
match error:
case tls: TlsError:
print(tls)?
case buffer: std.bytes.BytesError:
print(buffer)?
case cause: AllocError:
print(cause)?
case Ok(_):
print("unexpected")?
Ok(())
hello
0
5
std.tls.TlsError.WrongHost
std.tls.TlsError.Rejected
std.tls.connect takes ownership of the TCP stream and returns only after
the handshake, so a TlsStream is always verified. The second read returns
0 because the server sent close_notify before closing. A connection that
ends without it is TlsError.Truncated, since an attacker can cut a TCP
connection short.
The second client asks for example.test, which the certificate does not
name. It fails with WrongHost and tells the server, whose handshake fails
with Rejected. An unknown issuer is UntrustedCertificate, and an expired
certificate is ExpiredCertificate.
TlsStream also has read, write, flush, and close, so it works with
std.io helpers and std.http framing. Those
methods wait without a deadline.
See std.tls for the full reference.
Verify clients and choose a protocol
A server can ask the client for a certificate too. This is mutual TLS: both
sides prove who they are, so a service can accept only clients holding a key
that its own authority signed. Build such a configuration with a builder,
which gathers settings and checks them together in build.
The same builders negotiate an application protocol (ALPN), the name a
client and server agree on during the handshake, such as h2 for HTTP/2.
import std.bytes
import std.tls
from std.net import NetError, TcpListener, connect, listen, loopback_v4
from std.sync import CancellationToken
from std.time import Deadline, deadline_after, milliseconds, monotonic_now
from std.tls import TlsError, ClientCertificates, ClientConfig, ServerConfig, TlsStream
type ConnectionError = AllocError | IoError | NetError | TlsError | std.bytes.BytesError
# Test-only certificates: a server CA and a localhost leaf it signed, and a
# client CA and a client certificate it signed. Never use a published key
# outside an example.
def ca_pem() -> str:
return "-----BEGIN CERTIFICATE-----\nMIIBmTCCAT+gAwIBAgIUUResO5RQujUgwbO6wnWnB6CmmckwCgYIKoZIzj0EAwIw\nGTEXMBUGA1UEAwwOUGxlbnR5IFRlc3QgQ0EwIBcNMjQwMTAxMDAwMDAwWhgPMjEy\nNDAxMDEwMDAwMDBaMBkxFzAVBgNVBAMMDlBsZW50eSBUZXN0IENBMFkwEwYHKoZI\nzj0CAQYIKoZIzj0DAQcDQgAEnJV8Ts/yoTQC8df/IwhCPwjYuY+siUFEkwoyRG82\nFbjVeUydEKr7G3fG2hp21P46YwzRUsYal8Y/dZNwDPHKcKNjMGEwHQYDVR0OBBYE\nFIRt2tnXzddX1P2dNz1kPdQrraOuMB8GA1UdIwQYMBaAFIRt2tnXzddX1P2dNz1k\nPdQrraOuMA8GA1UdEwEB/wQFMAMBAf8wDgYDVR0PAQH/BAQDAgEGMAoGCCqGSM49\nBAMCA0gAMEUCIQCvPKJicaEPA6BwyR27QUz2f+h4RXSkJ9AX48/WuZ9K/QIgcV5e\nt2woOIP3sGU2PAopPcJVH2eC2068REEAlU7QFWc=\n-----END CERTIFICATE-----\n"
def leaf_pem() -> str:
return "-----BEGIN CERTIFICATE-----\nMIIBnDCCAUKgAwIBAgIBATAKBggqhkjOPQQDAjAZMRcwFQYDVQQDDA5QbGVudHkg\nVGVzdCBDQTAgFw0yNDAxMDEwMDAwMDBaGA8yMTI0MDEwMTAwMDAwMFowFDESMBAG\nA1UEAwwJbG9jYWxob3N0MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEh9wzUObb\nKhx8QP/mmT2p56niQY0XpFwOiXAC8qThDYNrqtZY+f+gzGFjFamRWJBgegZ7lTWc\n3XACctwTZV7F7aN+MHwwGgYDVR0RBBMwEYIJbG9jYWxob3N0hwR/AAABMBMGA1Ud\nJQQMMAoGCCsGAQUFBwMBMAkGA1UdEwQCMAAwHQYDVR0OBBYEFP2y8OrD2DL5XVhP\ntwRYSPXCUxblMB8GA1UdIwQYMBaAFIRt2tnXzddX1P2dNz1kPdQrraOuMAoGCCqG\nSM49BAMCA0gAMEUCIQD7b3qMvbfg125pcOxRnlgi9d1kA/XO06NcKtspGn/chAIg\nRsvsbrsb0St1++9xCNovakguzYFGDNOUGEecwkoMwRw=\n-----END CERTIFICATE-----\n"
def leaf_key() -> str:
return "-----BEGIN PRIVATE KEY-----\nMIGHAgEAMBMGByqGSM49AgEGCCqGSM49AwEHBG0wawIBAQQgkVCAyTuCJRiqL7JD\nJEWjEGpE5Jas61PHxV76daBTJaChRANCAASH3DNQ5tsqHHxA/+aZPannqeJBjRek\nXA6JcALypOENg2uq1lj5/6DMYWMVqZFYkGB6BnuVNZzdcAJy3BNlXsXt\n-----END PRIVATE KEY-----\n"
def client_ca_pem() -> str:
return "-----BEGIN CERTIFICATE-----\nMIIBnTCCAUOgAwIBAgIUdjiICLarpx0Q9y7yY6grXgRasaAwCgYIKoZIzj0EAwIw\nGzEZMBcGA1UEAwwQUGxlbnR5IENsaWVudCBDQTAgFw0yNDAxMDEwMDAwMDBaGA8y\nMTI0MDEwMTAwMDAwMFowGzEZMBcGA1UEAwwQUGxlbnR5IENsaWVudCBDQTBZMBMG\nByqGSM49AgEGCCqGSM49AwEHA0IABOMjuGk6yTyYHlHTSFqdxWo2pgdqU5/oUzGq\naI+Hwz342D5kZPevdjFf1uyS+wa8XAdRdGqcxoTfqMMKaaj5vw+jYzBhMB0GA1Ud\nDgQWBBS89JCv7pReUSMZUl8kRbl5orgjjjAfBgNVHSMEGDAWgBS89JCv7pReUSMZ\nUl8kRbl5orgjjjAPBgNVHRMBAf8EBTADAQH/MA4GA1UdDwEB/wQEAwIBBjAKBggq\nhkjOPQQDAgNIADBFAiEA5H1xwC1vK6Fx1Zn8HqI2Ob8wry5EdjCO3Usej/KfXG8C\nIF3oydaIMmuMWw6yVxTJ6XMLbvMf4TXIxlPt7uZwpBAc\n-----END CERTIFICATE-----\n"
def client_pem() -> str:
return "-----BEGIN CERTIFICATE-----\nMIIBhjCCASygAwIBAgIBATAKBggqhkjOPQQDAjAbMRkwFwYDVQQDDBBQbGVudHkg\nQ2xpZW50IENBMCAXDTI0MDEwMTAwMDAwMFoYDzIxMjQwMTAxMDAwMDAwWjAYMRYw\nFAYDVQQDDA1QbGVudHkgQ2xpZW50MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAE\nT5GBs10j6jCpjnrqvhM72PkArJoUAQHTk30vDaFUfzqLjJLvNnGG3mxcJ6nhmZIH\nl4K0HSop5C/aewN7v67bWaNiMGAwEwYDVR0lBAwwCgYIKwYBBQUHAwIwCQYDVR0T\nBAIwADAdBgNVHQ4EFgQUd3LfI5CS7q5ORpggbgx2rsLnrsQwHwYDVR0jBBgwFoAU\nvPSQr+6UXlEjGVJfJEW5eaK4I44wCgYIKoZIzj0EAwIDSAAwRQIgUAs5ByLBx0fx\nhNbzolmRLRLUgo0jwQJwhc3dINKcY2ECIQC5MGMpqLAhsdUm00Kdjco/wNwFiPq8\nUypR3xWb5//Lkw==\n-----END CERTIFICATE-----\n"
def client_key() -> str:
return "-----BEGIN PRIVATE KEY-----\nMIGHAgEAMBMGByqGSM49AgEGCCqGSM49AwEHBG0wawIBAQQgIVqyz+J6RT5Qaqpt\ndcpJwQnZR3muh8UkZ4GdOuUB1aShRANCAARPkYGzXSPqMKmOeuq+EzvY+QCsmhQB\nAdOTfS8NoVR/OouMku82cYbebFwnqeGZkgeXgrQdKinkL9p7A3u/rttZ\n-----END PRIVATE KEY-----\n"
# Reports who connected, then answers one message.
def answer(stream: &mut TlsStream, deadline: &Deadline, cancel: &CancellationToken) -> Result[(), ConnectionError]:
match stream.peer_certificate()?:
case Some(der):
print(der.len() > 0)?
case Nothing:
print("no certificate")?
print(stream.application_protocol()?)?
mut buffer = std.bytes.with_capacity(64)?
buffer.resize(64, 0u8)?
count = stream.read_before(&mut buffer, 0, 64, deadline, cancel)?
stream.write_all_before(&buffer[:count], deadline, cancel)?
stream.close_before(deadline, cancel)?
Ok(())
def greet(listener: &TcpListener, config: &ServerConfig, deadline: &Deadline, cancel: &CancellationToken) -> Result[(), ConnectionError]:
tcp = listener.accept(deadline, cancel)?
match std.tls.accept(tcp, config, deadline, cancel):
case Ok(opened):
mut stream = opened
answer(&mut stream, deadline, cancel)?
case Err(error):
print(error)?
Ok(())
# The client's outcome is printed after the server's, once the server is done.
def ask(listener: &TcpListener, server: &ServerConfig, client: &ClientConfig, deadline: &Deadline, cancel: &CancellationToken) -> Result[(), ConnectionError | SpawnError[()]]:
address = listener.local_address()?
mut reply = std.bytes.with_capacity(16)?
reply.resize(16, 0u8)?
with spawn(greet, listener, server, deadline, cancel)? as worker:
tcp = connect(&address, deadline, cancel)?
mut stream = std.tls.connect(tcp, client, "localhost", deadline, cancel)?
message = std.bytes.from_text("hello")?
stream.write_all_before(&message, deadline, cancel)?
outcome = stream.read_before(&mut reply, 0, 16, deadline, cancel)
worker.join()?
match outcome:
case Ok(count):
print(reply.to_text_range(0, count)?)?
case Err(error):
print(error)?
Ok(())
def main() -> Result[(), ConnectionError | SpawnError[()] | RangeError]:
cancel = CancellationToken()?
deadline = deadline_after(&monotonic_now(), &milliseconds(5000))?
mut server = std.tls.server_builder(&leaf_pem(), &leaf_key())?
server.verify_clients(&std.tls.roots_from_pem(&client_ca_pem())?, ClientCertificates.Required)
server.application_protocol("greet/1")?
server_config = server.build()?
roots = std.tls.roots_from_pem(&ca_pem())?
mut client = std.tls.client_builder(&roots)?
client.certificate(&client_pem(), &client_key())?
client.application_protocol("greet/1")?
client_config = client.build()?
listener = listen(&loopback_v4(0u16), 8)?
ask(&listener, &server_config, &client_config, &deadline, &cancel)?
ask(&listener, &server_config, &std.tls.client_config(&roots)?, &deadline, &cancel)?
Ok(())
True
Option[str].Some("greet/1")
hello
std.tls.TlsError.CertificateRequired
std.tls.TlsError.Rejected
verify_clients takes the roots that client certificates must chain to,
usually a private authority rather than the system bundle. With
ClientCertificates.Optional a client may present none, and the server’s
peer_certificate() is then Nothing. peer_certificate() returns the DER
bytes of the verified leaf, for code that maps a certificate to an account.
The second client has no certificate. The server fails with
CertificateRequired. In TLS 1.3 the client finishes its side of the
handshake before the server checks it, so the client learns of the refusal
at its first read, as Rejected.
A server picks the first of its protocol names that the client also offers,
and application_protocol() reports it on both sides. A client that offers
only names the server does not accept fails with Incompatible; one that
offers none gets Nothing.
Builders also choose the minimum TLS version and the cipher suites, add
revocation lists, and give a server several certificates to choose by the
name the client asked for. See std.tls.
ConnectionError preserves the concrete allocation, I/O, networking, TLS, and
byte-buffer errors from the connection helpers. ask adds SpawnError[()] for
the scoped start, and main adds the range errors from setup.
Serve HTTP requests
std.http.server runs an HTTP/1.1
server on a std.net listener. You give it a table of
routes and one handler; it accepts connections, frames requests with
std.http, and stops when a
cancellation token is cancelled.
Each route maps a method and a path pattern to a value of your own route type, normally an enum. The handler receives that value and matches on it.
For supervised connection heaps and event-driven handlers, see
Isolate HTTP connections with actors. The worker server
taught here keeps its sequential Exchange interface.
import std.http
import std.bytes
import std.http.server
import std.net
import std.time
from std.http.server import Exchange
from std.net import TcpStream
from std.sync import CancellationToken
pub enum Route:
Home
Hello(Greeting)
pub class Greeting:
pub opening: str
pub def handle(self, exchange: &mut Exchange[TcpStream]) -> Result[(), AllocError | std.http.HttpError]:
text = f"{self.opening}{exchange.params[0]}\n"?
exchange.send_text(200, &text)?
Ok(())
pub class App:
pub def handle(self, route: &Route, exchange: &mut Exchange[TcpStream]) -> Result[(), AllocError | std.http.HttpError]:
match route:
case Route.Home:
exchange.send_text(200, "home\n")?
case Route.Hello(greeting):
greeting.handle(exchange)?
Ok(())
type FetchError = AllocError | IoError | std.bytes.BytesError | std.net.NetError | RangeError
def get(port: u16, path: &str) -> Result[str, FetchError]:
never = CancellationToken()?
deadline = std.time.deadline_after(&std.time.monotonic_now(), &std.time.seconds(5u32))?
mut stream = std.net.connect(&std.net.loopback_v4(port), &deadline, &never)?
request = std.bytes.from_text(&f"GET {path} HTTP/1.1\r\nHost: localhost\r\nConnection: close\r\n\r\n"?)?
stream.write_all_before(&request, &deadline, &never)?
mut reply = std.bytes.empty()?
mut buffer = std.bytes.empty()?
buffer.resize(512, 0u8)?
while True:
count = stream.read_before(&mut buffer, 0, 512, &deadline, &never)?
if count == 0:
break
reply.extend(&buffer[:count])?
text = reply.to_text()?
status = text.split("\r\n")?[0]
body = text.split("\r\n\r\n")?[1].strip()?
Ok(f"{status} | {body}"?)
def client(port: u16, stop: CancellationToken) -> Result[(), FetchError]:
with stop.cancel_on_exit():
print(get(port, "/")?)?
print(get(port, "/index.html")?)?
print(get(port, "/hello/Ada%20L")?)?
print(get(port, "/missing")?)?
Ok(())
type ServerSetupError = std.http.server.ServerError | std.net.NetError
def main() -> Result[(), SpawnError[()] | IoError | AllocError | ServerSetupError | FetchError]:
mut routes = std.http.server.router[Route]()?
routes.add("GET", "/", Route.Home)?
routes.add("GET", "/index.html", Route.Home)?
routes.add("GET", "/hello/{name}", Route.Hello(Greeting("hello, ")))?
listener = std.net.listen(&std.net.loopback_v4(0u16), 16)?
port = listener.local_address()?.port
stop = CancellationToken()?
with spawn(client, port, stop.share()).drop_job()? as task:
report = std.http.server.serve(listener, &routes, &App(), &std.http.server.config(), &stop)?
task.join()?
print(report.requests)?
Ok(())
HTTP/1.1 200 OK | home
HTTP/1.1 200 OK | home
HTTP/1.1 200 OK | hello, Ada L
HTTP/1.1 404 Not Found | Not Found
4
The client cancels stop when it is done, which stops the server. Its body
is in with stop.cancel_on_exit():, so a failed request stops the server too,
and task.join()? after serve reports that failure; see
cancel on every exit. The
later HTTP lessons use the same shape.
FetchError names the client’s errors, shared by get and client.
ServerSetupError names this example’s server and listener errors. serve
already wraps its network failures in ServerError; the separate NetError
member is needed because main opens and inspects the listener directly.
The alias composes with allocation, printing and thread-start errors without
mapping their values.
router[Route]() makes an empty table for the route type Route. It
matches the request path against each pattern in the order the routes were
added. {name} captures one path segment, and the handler finds it,
percent-decoded, in exchange.params. The value given to add comes back
to handle as route, borrowed from the table, so the handler chooses
what to do with a match. A route the handler forgets is a compile error:
import std.http
from std.http.server import Exchange
from std.net import TcpStream
pub enum Route:
Home
Hello
pub class App:
pub def handle(self, route: &Route, exchange: &mut Exchange[TcpStream]) -> Result[(), std.http.HttpError]:
match route:
case Route.Home:
exchange.send_text(200, "home\n")?
Ok(())
def main() -> Result[(), AllocError | IoError]:
pass
Ok(())
non-exhaustive match; missing Route.Hello
A variant can carry the route’s own handler and its state, as Hello
carries a Greeting here; App only forwards to it. Several patterns may
share a variant, such as / and /index.html both adding Route.Home.
A path with no route gets 404 without reaching the handler, and a path
whose route has another method gets 405 with an Allow field.
The client sends a Host field because HTTP/1.1 requires exactly one: a
request without it, with two, or with a malformed value gets 400 before
routing. The handler finds the host the request names in
exchange.authority.
The handler is a pub class with a pub def handle, because serve, in
another module, calls it, and Route is a pub enum, because handle’s
signature names it. Every worker thread borrows the same handler and the
same table, so handle takes self and route by shared borrow and
cannot change them.
handle returns its own error type, here AllocError | std.http.HttpError
from the f-string and send_text. The server answers an Err of any type with
500 and counts it in report.handler_errors;
When a handler fails
lists the other cases.
serve runs on the calling thread until stop is cancelled. Here the client
thread cancels it after its last request. serve then stops accepting, lets
the connections it has finish within the drain timeout, joins its workers
and returns a Report of what it counted. config() gives the defaults:
four workers, a queue of sixteen waiting connections, and a deadline for
each phase of a request. A connection that arrives when the queue is full is
answered 503 at once instead of waiting.
A worker serves one request at a time, but it does not wait for a client to
send one. Between requests, and while a request head is still arriving, the
connection waits on the server’s own thread, on a
selector, and goes back to the
queue once the client has sent a whole head. A few workers can therefore
keep many idle clients open, and clients that send slowly cannot hold them
all. config.idle_connections caps how many connections may wait.
HTTPS
std.http.tls.serve is serve over TLS. It takes the
ServerConfig from the TLS lesson, and the handler’s exchanges carry a
TlsTransport instead of a TcpStream:
import std.http.tls
from std.http.tls import TlsTransport
pub class App:
pub def handle(self, route: &Route, exchange: &mut Exchange[TlsTransport]) -> Result[(), AllocError | std.http.HttpError]:
...
server = std.tls.server_config(&leaf_pem(), &leaf_key())?
report = std.http.tls.serve(listener, server, &routes, &App(), &config, &stop)?
Each connection must finish its handshake within
config.handshake_timeout of being accepted, so a client that connects and
says nothing is closed then. Between the client’s handshake messages the
connection waits on the server’s thread too. Routing, deadlines and shutdown work as for
plain HTTP.
Stop a server on a signal
A supervisor such as systemd, a container runtime or a shell stops a program
with SIGTERM, or SIGINT for Ctrl-C. By default either signal ends a
Plenty program at once, cutting off responses in flight.
std.signal turns them into an event, so
the HTTP server can drain instead:
import std.http
import std.http.server
import std.net
import std.process
import std.ready
import std.signal
from std.http.server import Exchange
from std.net import TcpStream
from std.signal import Signal
from std.sync import CancellationToken
pub enum Route:
Home
pub class Hello:
pub def handle(self, route: &Route, exchange: &mut Exchange[TcpStream]) -> Result[(), std.http.HttpError]:
exchange.send_text(200, "hello\n")?
Ok(())
# Plays the supervisor: a shell sends SIGTERM to its parent, this program.
def supervisor() -> Result[(), AllocError | IoError]:
mut command = std.process.shell("kill -TERM $PPID")?
command.spawn()?.wait()?
Ok(())
type ServerShutdownError = std.signal.SignalError | std.http.server.ServerError | std.net.NetError | std.ready.WaitError
def main() -> Result[(), SpawnError[()] | IoError | AllocError | ServerShutdownError]:
signals = std.signal.listen()?
mut routes = std.http.server.router[Route]()?
routes.add("GET", "/", Route.Home)?
listener = std.net.listen(&std.net.loopback_v4(0u16), 16)?
stop = CancellationToken()?
with spawn(std.signal.cancel_on, signals, stop.share()).drop_job()? as watcher, stop.cancel_on_exit(), spawn(supervisor).drop_job()? as stopper:
served = std.http.server.serve(listener, &routes, &Hello(), &std.http.server.config(), &stop)
stop.cancel()
stopper.join()?
match watcher.join()?:
case Some(Signal.Terminate):
print("stopping on SIGTERM")?
case Some(Signal.Interrupt):
print("stopping on SIGINT")?
case Nothing:
print("stopped without a signal")?
print(served?.requests)?
Ok(())
stopping on SIGTERM
0
std.signal.listen() catches both signals for the whole process and returns
a Signals handle on which they arrive. Call it on the main thread, before
the work it should stop; a worker may not call it, and a process listens only
once.
cancel_on runs on its own thread. It waits for a signal and cancels stop,
which begins the server’s shutdown:
serve stops accepting, finishes the requests it has within the drain
timeout, and returns its Report. If serve returns for another reason,
stop.cancel() ends the watcher instead, and it returns Nothing.
The watcher waits until a signal arrives or stop is cancelled, and the block
joins it as it exits. stop.cancel_on_exit() directly after the watcher
cancels stop before that join on every exit, including a failed
spawn(supervisor).drop_job()?; see cancel on every exit.
After the first signal both signals have their default action again, so a
second Ctrl-C or SIGTERM ends a slow drain at once. Closing or dropping
the Signals handle restores the default action too. Child processes
started by std.process
keep the default action.
A program that is not a server can wait on the handle directly with
signals.wait(&deadline, &cancel), or register it on a
selector with
signals.watch(&mut selector, key) and take the signal with
signals.received().
Read forms, cookies and forwarded clients
A handler for std.http.server gets the raw request.
Three modules read the common parts of it safely:
std.http.body reads forms, JSON and
uploads within limits, std.http.cookie
reads and writes cookies, and
std.http.proxy finds the client
behind a reverse proxy.
import std.bytes
import std.http
import std.http.body
import std.http.cookie
import std.http.proxy
import std.http.server
import std.net
import std.time
from std.http.body import FormLimits
from std.http.proxy import Forwarding, Proxies
from std.http.server import Exchange
from std.net import TcpStream
from std.sync import CancellationToken
pub enum Route:
Login
Me
pub class Site:
pub forms: FormLimits
pub proxies: Proxies
pub def handle(self, route: &Route, exchange: &mut Exchange[TcpStream]) -> Result[(), AllocError | std.http.HttpError | std.bytes.BytesError | std.http.cookie.CookieError | std.http.proxy.ProxyError | std.net.NetError]:
match route:
case Route.Login:
match std.http.body.read_form(exchange, &self.forms):
case Ok(fields):
match std.http.body.find(&fields, "name"):
case Some(index):
name = &fields[index].value
mut head = std.http.response(200, "OK")?
std.http.cookie.add(&mut head, &std.http.cookie.set_cookie("user", name)?)?
head.add("Content-Type", "text/plain")?
mut text = f"welcome {name}"?
match std.http.body.find(&fields, "city"):
case Some(city):
text = f"{text} from {fields[city].value}"?
case Nothing:
pass
exchange.respond(head, &std.bytes.from_text(&text)?)?
case Nothing:
exchange.send_text(400, "name is required")?
case Err(error):
std.http.body.reject(exchange, &error)?
case Route.Me:
cookies = std.http.cookie.cookies(&exchange.request.headers, 32)?
client = std.http.proxy.client_of(exchange, &self.proxies)?
address = std.net.format_ip(client.address)?
mut who = "nobody"
match std.http.cookie.find(&cookies, "user"):
case Some(index):
who = cookies[index].value
case Nothing:
pass
exchange.send_text(200, &f"{address} is {who}"?)?
Ok(())
def fetch(port: u16, request: &str) -> Result[str, AllocError | IoError | std.bytes.BytesError | std.net.NetError | RangeError]:
never = CancellationToken()?
deadline = std.time.deadline_after(&std.time.monotonic_now(), &std.time.seconds(5u32))?
mut stream = std.net.connect(&std.net.loopback_v4(port), &deadline, &never)?
data = std.bytes.from_text(request)?
stream.write_all_before(&data, &deadline, &never)?
mut reply = std.bytes.empty()?
mut buffer = std.bytes.empty()?
buffer.resize(512, 0u8)?
while True:
count = stream.read_before(&mut buffer, 0, 512, &deadline, &never)?
if count == 0:
break
reply.extend(&buffer[:count])?
text = reply.to_text()?
lines = text.split("\r\n")?
mut parts = [lines[0]]?
for line in lines:
if line.startswith("Set-Cookie"):
parts.append(line)?
parts.append(text.split("\r\n\r\n")?[1].strip()?)?
Ok(" | ".join(&parts)?)
def client(port: u16, stop: CancellationToken) -> Result[(), AllocError | IoError | RangeError | std.bytes.BytesError | std.net.NetError]:
with stop.cancel_on_exit():
form = "POST /login HTTP/1.1\r\nHost: localhost\r\nContent-Type: application/x-www-form-urlencoded\r\nContent-Length: 28\r\nConnection: close\r\n\r\nname=Ada&city=S%C3%A3o+Paulo"
print(fetch(port, form)?)?
print(fetch(port, "POST /login HTTP/1.1\r\nHost: localhost\r\nContent-Type: text/plain\r\nContent-Length: 3\r\nConnection: close\r\n\r\nAda")?)?
print(fetch(port, "GET /me HTTP/1.1\r\nHost: localhost\r\nCookie: theme=dark; user=Ada\r\nX-Forwarded-For: 198.51.100.4, 203.0.113.7\r\nConnection: close\r\n\r\n")?)?
Ok(())
type ProxySetupError = std.http.server.ServerError | std.http.proxy.ProxyError | std.net.NetError
def main() -> Result[(), SpawnError[()] | AllocError | ProxySetupError | IoError | RangeError | std.bytes.BytesError | std.net.NetError]:
mut routes = std.http.server.router[Route]()?
routes.add("POST", "/login", Route.Login)?
routes.add("GET", "/me", Route.Me)?
mut proxies = std.http.proxy.proxies(Forwarding.XForwardedFor)?
proxies.trust("127.0.0.1")?
site = Site(std.http.body.form_limits(), proxies)
listener = std.net.listen(&std.net.loopback_v4(0u16), 16)?
port = listener.local_address()?.port
stop = CancellationToken()?
with spawn(client, port, stop.share()).drop_job()? as task:
std.http.server.serve(listener, &routes, &site, &std.http.server.config(), &stop)?
task.join()?
Ok(())
HTTP/1.1 200 OK | Set-Cookie: user=Ada; Path=/; Secure; HttpOnly; SameSite=Lax | welcome Ada from São Paulo
HTTP/1.1 415 Unsupported Media Type | Unsupported Media Type
HTTP/1.1 200 OK | 203.0.113.7 is Ada
read_form checks that the request says it carries a form, refuses a body
over forms.max_bytes (64 KiB by default) or with more than
forms.max_fields fields, and decodes each name=value pair: + is a
space and %C3%A3 is the UTF-8 encoding of ã. It returns the pairs in order; find gives the
index of the first with a name. When it fails, reject answers with the
status that fits: 415 for the wrong type, 413 for too much, 400 for a
malformed body.
set_cookie checks the name and value and gives the cookie safe defaults:
Secure, so the browser sends it only over HTTPS, HttpOnly, so page
scripts cannot read it, and SameSite=Lax, so other sites cannot send it
with a form post. No value can add an attribute or another field: a name
with a space, such as Ada L, is an error rather than a broken header, so
encode free text before storing it in a cookie.
cookies reads every Cookie field. It skips pairs that are not valid
cookies instead of failing, because the field carries the cookies of every
application on the host.
The server sees its peer, 127.0.0.1. When a reverse proxy forwards
requests, the peer is the proxy and the client’s address is in
X-Forwarded-For, which anyone can write. client_of believes the header
only because the program trusts 127.0.0.1, and reads it from the right:
203.0.113.7 is the first address that is not a trusted proxy, so it is the
client. 198.51.100.4 came from the client itself and is ignored. With no
trust call, the header is ignored and the client is the peer.
Serve files and sign cookies
std.http.files answers a request
with a file from one directory, and
std.http.cookie can sign a cookie
so the server can tell whether a client changed it.
import std.bytes
import std.fs
import std.http
import std.http.cookie
import std.http.files
import std.io
import std.http.server
import std.net
import std.path
import std.secrets
import std.time
from std.bytes import Bytes
from std.fs import Root
from std.http.server import Exchange
from std.net import TcpStream
from std.sync import CancellationToken
pub enum Route:
Files
Login
Me
pub class Site:
pub public: Root
pub keys: list[Bytes]
pub def handle(self, route: &Route, exchange: &mut Exchange[TcpStream]) -> Result[(), AllocError | std.http.HttpError | std.bytes.BytesError | std.http.files.FilesError | std.http.cookie.CookieError]:
match route:
case Route.Files:
path = exchange.params[0]
std.http.files.serve(exchange, &self.public, &path)?
case Route.Login:
mut head = std.http.response(204, "No Content")?
cookie = std.http.cookie.signed_cookie(&self.keys, "user", "Ada")?
std.http.cookie.add(&mut head, &cookie)?
exchange.respond(head, &std.bytes.empty()?)?
case Route.Me:
cookies = std.http.cookie.cookies(&exchange.request.headers, 32)?
match std.http.cookie.find_signed(&cookies, &self.keys, "user")?:
case Some(name):
exchange.send_text(200, &"hello ".concat(&name)?)?
case Nothing:
exchange.send_text(401, "who are you?")?
Ok(())
type FetchError = AllocError | IoError | std.bytes.BytesError | std.net.NetError | RangeError
def fetch(port: u16, request: &str) -> Result[str, FetchError]:
never = CancellationToken()?
deadline = std.time.deadline_after(&std.time.monotonic_now(), &std.time.seconds(5u32))?
mut stream = std.net.connect(&std.net.loopback_v4(port), &deadline, &never)?
data = std.bytes.from_text(request)?
stream.write_all_before(&data, &deadline, &never)?
mut reply = std.bytes.empty()?
mut buffer = std.bytes.empty()?
buffer.resize(512, 0u8)?
while True:
count = stream.read_before(&mut buffer, 0, 512, &deadline, &never)?
if count == 0:
break
reply.extend(&buffer[:count])?
Ok(reply.to_text()?)
# The value of the field `name` in a response, or "".
def field(reply: &str, name: &str) -> Result[str, AllocError | IoError]:
for line in reply.split("\r\n")?:
if line.startswith(name):
return Ok(line.split(": ")?[1])
Ok("")
def get(path: &str, extra: &str) -> Result[str, AllocError]:
Ok(f"GET {path} HTTP/1.1\r\nHost: localhost\r\nConnection: close\r\n{extra}\r\n"?)
def client(port: u16, stop: CancellationToken) -> Result[(), FetchError]:
with stop.cancel_on_exit():
page = fetch(port, &get("/files/index.html", "")?)?
print(page.split("\r\n")?[0])?
print(field(&page, "Content-Type")?)?
print(page.split("\r\n\r\n")?[1].strip()?)?
tag = field(&page, "ETag")?
again = fetch(port, &get("/files/index.html", &f"If-None-Match: {tag}\r\n"?)?)?
print(again.split("\r\n")?[0])?
for path in ["/files/../secret.txt", "/files/%2e%2e/secret.txt", "/files/.env"]?:
print(fetch(port, &get(&path, "")?)?.split("\r\n")?[0])?
login = fetch(port, "POST /login HTTP/1.1\r\nHost: localhost\r\nContent-Length: 0\r\nConnection: close\r\n\r\n")?
cookie = field(&login, "Set-Cookie")?.split(";")?[0]
print(cookie.startswith("user=Ada."))?
print(fetch(port, &get("/me", &f"Cookie: {cookie}\r\n"?)?)?.split("\r\n\r\n")?[1])?
forged = cookie.replace("Ada", "Eve")?
print(fetch(port, &get("/me", &f"Cookie: {forged}\r\n"?)?)?.split("\r\n\r\n")?[1])?
Ok(())
type FileSiteError = std.fs.FsError | std.io.StreamError | std.http.server.ServerError | std.secrets.SecretsError | std.net.NetError
def main() -> Result[(), SpawnError[()] | IoError | AllocError | FileSiteError | FetchError]:
parent = std.fs.root(&std.path.from_text(".")?)?
mut directory = std.fs.temporary_dir(&parent, &std.path.from_text("plenty-site-")?, 8)?
for name in ["index.html", ".env"]?:
mut staged = std.fs.temporary_file(directory.root())?
staged.stream.file.write("<h1>Welcome</h1>\n")?
staged.publish(&std.path.from_text(name)?)?
staged.close()?
mut routes = std.http.server.router[Route]()?
routes.add("GET", "/files/{path...}", Route.Files)?
routes.add("POST", "/login", Route.Login)?
routes.add("GET", "/me", Route.Me)?
site = Site(parent.open_dir(directory.name())?, [std.secrets.token_bytes(32)?]?)
listener = std.net.listen(&std.net.loopback_v4(0u16), 16)?
port = listener.local_address()?.port
stop = CancellationToken()?
with spawn(client, port, stop.share()).drop_job()? as task:
std.http.server.serve(listener, &routes, &site, &std.http.server.config(), &stop)?
task.join()?
for name in ["index.html", ".env"]?:
directory.root().remove_file(&std.path.from_text(name)?)?
directory.cleanup()?
Ok(())
HTTP/1.1 200 OK
text/html; charset=utf-8
<h1>Welcome</h1>
HTTP/1.1 304 Not Modified
HTTP/1.1 404 Not Found
HTTP/1.1 404 Not Found
HTTP/1.1 404 Not Found
True
hello Ada
who are you?
serve takes the path the route captured, already percent-decoded, and
opens it one segment at a time below public. A segment that is empty or
starts with . is refused, so .. cannot climb out and hidden files such
as .env stay hidden, and a symbolic link is never followed, so a link
inside the directory cannot lead out of it either. Every refusal is a plain
404, which says nothing about what lies outside.
The type comes from the file’s extension, and the response carries an
ETag and Last-Modified built from the file’s size and modification
time. A browser that sends the tag back in If-None-Match gets 304 Not Modified with no body. The file itself is streamed in 16 KiB pieces, so a
large file does not need a large buffer.
signed_cookie appends a tag to the value: an HMAC of the cookie’s name and
value under the first key, so user=Ada.<tag>. find_signed checks the
tag and gives back Ada; a client that changes the value to Eve cannot
compute a matching tag without the key, so the forged cookie is treated as
missing. The value is still readable by the client: signing stops changes,
not reading.
Keys are at least 32 random bytes, kept secret and stable across restarts;
token_bytes here makes a new one per run, which signs out every client
when the program restarts. To replace a key, put the new one first and keep
the old one after it until old cookies have expired: find_signed accepts
any listed key, and signed_cookie always signs with the first.
Send HTTP requests
std.http.client sends HTTP/1.1
requests and keeps finished connections open for the next request to the
same server. Every call takes a deadline and a cancellation token, so a slow
server cannot hold the caller longer than it allows.
This program runs a small server on a loopback port and sends it three requests from another thread:
import std.bytes
import std.http
import std.http.client
import std.http.server
import std.net
import std.time
from std.http.server import Exchange
from std.net import TcpStream
from std.sync import CancellationToken
pub enum Route:
Home
Old
Echo
pub class Site:
pub def handle(self, route: &Route, exchange: &mut Exchange[TcpStream]) -> Result[(), AllocError | std.http.HttpError | std.bytes.BytesError]:
match route:
case Route.Home:
exchange.send_text(200, "home\n")?
case Route.Old:
mut head = std.http.response(301, "Moved Permanently")?
head.add("Location", "/")?
exchange.respond(head, &std.bytes.empty()?)?
case Route.Echo:
mut body = std.bytes.empty()?
exchange.read_all(&mut body)?
exchange.send(200, "text/plain", &body)?
Ok(())
type VisitError = AllocError | IoError | std.bytes.BytesError | std.http.client.ClientError | RangeError
def visit(port: u16, stop: CancellationToken) -> Result[(), VisitError]:
with stop.cancel_on_exit():
never = CancellationToken()?
deadline = std.time.deadline_after(&std.time.monotonic_now(), &std.time.seconds(5u32))?
base = f"http://127.0.0.1:{port}"?
client = std.http.client.client(std.http.client.config()?)?
home = client.get(&f"{base}/"?, &deadline, &never)?
print(home.status, home.body.to_text()?.strip()?)?
moved = client.get(&f"{base}/old"?, &deadline, &never)?
print(moved.status, moved.redirects, moved.url.endswith("/"))?
mut post = std.http.client.request("POST", &f"{base}/echo"?)?
post.add("Content-Type", "text/plain")?
post.body = std.bytes.from_text("ping")?
echoed = client.fetch(post, &deadline, &never)?
print(echoed.body.to_text()?)?
stats = client.stats()
print(stats.opened, stats.reused)?
Ok(())
def main() -> Result[(), SpawnError[()] | std.http.server.ServerError | std.net.NetError | AllocError | VisitError]:
mut routes = std.http.server.router[Route]()?
routes.add("GET", "/", Route.Home)?
routes.add("GET", "/old", Route.Old)?
routes.add("POST", "/echo", Route.Echo)?
listener = std.net.listen(&std.net.loopback_v4(0u16), 16)?
port = listener.local_address()?.port
stop = CancellationToken()?
with spawn(visit, port, stop.share()).drop_job()? as task:
drop(std.http.server.serve(listener, &routes, &Site(), &std.http.server.config(), &stop)?)
task.join()?
Ok(())
200 home
200 1 True
ping
1 3
client(config()) makes a client for plain http URLs with the default
limits and timeouts. get and fetch send a request, read the whole
response body into body, and put the connection back in the client’s
pool. All four requests here, counting the redirect, used one connection:
opened is 1 and reused is 3.
The client follows the 301 itself. redirects counts the hops and url
is the address that finally answered. When a redirect leaves the original
server, the client drops Authorization and Cookie so that credentials
meant for one server are not sent to another.
A request holds its method, URL, extra header fields and body. The client
writes Host and Content-Length itself and refuses requests that try to
set them.
deadline covers the whole exchange: connecting, sending, waiting for the
answer and reading the body. The config adds a limit for each phase, such
as head_timeout for the wait between sending a request and its response
starting. If either runs out, the call returns Net(TimedOut).
A request that fails because a reused connection was closed by the server
is sent again on a new connection, but only for methods that are safe to
repeat, such as GET and PUT. A POST is never sent twice; its error is
returned so the program can decide what to do.
For a body that should be processed incrementally, send returns a response
after its head arrives. Use read_body to read chunks, or
read_all_progress to append into your own buffer while retaining the count
appended before an error. Both preserve the full ClientError, including
cancellation and TLS failures. response.total() counts successfully read
payload bytes; response.trailers() is complete after response.is_done().
Give a completed response to client.finish to allow reuse. The
typed stream lesson explains how to compose
it with transfers through typed_response without a stored-error side channel.
Share one HTTP client between workers
A client keeps finished connections in a pool for the next
request to the same server. Every client method borrows the client shared, so
jobs on an executor can use one client through
Shared handles and reuse each other’s connections.
This program serves a loopback port from one thread and sends it requests from three pool jobs that share a client:
import std.http
import std.http.client
import std.http.server
import std.net
import std.time
from std.concurrent import ThreadPoolExecutor, Future, SubmitError, PoolError, FutureError
from std.http.client import Client, ClientError, Plain, TcpConnector
from std.http.server import Exchange
from std.net import TcpListener, TcpStream
from std.sync import CancellationToken
pub enum Route:
Home
pub class Site:
pub def handle(self, route: &Route, exchange: &mut Exchange[TcpStream]) -> Result[(), std.http.HttpError]:
exchange.send_text(200, "home\n")?
Ok(())
def serve(listener: TcpListener, stop: CancellationToken) -> Result[(), AllocError | std.http.server.ServerError]:
mut routes = std.http.server.router[Route]()?
routes.add("GET", "/", Route.Home)?
drop(std.http.server.serve(listener, &routes, &Site(), &std.http.server.config(), &stop)?)
Ok(())
def fetch_all(client: Shared[Client[Plain, TcpConnector]], url: str, count: i64) -> Result[i64, AllocError | ClientError | RangeError | OverflowError[i64]]:
never = CancellationToken()?
mut ok = 0
for _ in range(count):
deadline = std.time.deadline_after(&std.time.monotonic_now(), &std.time.seconds(5u32))?
if client.get(&url, &deadline, &never)?.status == 200:
ok += 1
Ok(ok)
type PoolJobError = PoolError | FutureError | SubmitError[()]
type ClientRunError = OverflowError[i64] | OverflowError[u64] | AllocError | IoError | SpawnError[()] | RangeError | ClientError | std.net.NetError | std.http.server.ServerError | PoolJobError
def main() -> Result[(), ClientRunError]:
listener = std.net.listen(&std.net.loopback_v4(0u16), 16)?
port = listener.local_address()?.port
stop = CancellationToken()?
with spawn(serve, listener, stop.share()).drop_job()? as server, stop.cancel_on_exit():
url = f"http://127.0.0.1:{port}/"?
client = Shared(std.http.client.client(std.http.client.config()?)?)?
mut answered = 0
with ThreadPoolExecutor(3, 8)? as pool:
a = pool.submit(fetch_all, client.share(), url, 10).drop_job()?
b = pool.submit(fetch_all, client.share(), url, 10).drop_job()?
c = pool.submit(fetch_all, client.share(), url, 10).drop_job()?
answered = a.result()?? + b.result()?? + c.result()??
stats = client.stats()
print(answered, stats.opened + stats.reused, stats.opened <= 3)?
print(stats.opened == u64(client.idle()) + stats.discarded)?
stop.cancel()
server.join()?
Ok(())
30 30 True
True
stop.cancel_on_exit() stops the server however the block ends, so a failed
request cannot leave the block’s join waiting for it. On success, stop.cancel()
ends the server before server.join()? checks its result.
Shared(client(config)?)? moves the client into one shared allocation;
client.share() gives each job its own owner, and client.get reaches the
client through the handle. Thirty requests went out, and every one either
opened a connection or reused one: opened + reused is 30. A job holds one
connection at a time and opens one only when none is idle, and a connection
one job parks is taken by whichever job asks next. Three jobs ran at once and
none of their connections closed, so at most three were opened.
The client holds its pool lock only while taking a connection out or putting one back. Connecting, sending, and waiting for a response happen with the lock released, so a job stuck on a slow server does not delay the others. Each request still has its own deadline and cancellation token.
stats() and idle() work through the handle too. These observation counters saturate at u64.MAX; the small fixed workload
here stays far below that limit. Before saturation, every connection the client opened is idle, in use, or counted
as discarded once it has closed, whatever closed it; with no request or
response holding one, opened is idle() plus discarded, as the last line
shows. The pool limits max_idle and max_idle_per_origin and the
idle_timeout apply to the shared pool as a whole; see
the client reference.
Send HTTPS requests
std.http.https gives the
HTTP client https URLs. It runs TLS
on each connection, checking that the server’s certificate chains to a
trusted root and names the host in the URL. Everything else is the same
client: pooled connections, deadlines, redirects and replays.
This program serves HTTPS on a loopback port with
std.http.tls and sends it two
requests from another thread. The second streams its body from a reader
instead of holding it in the request. Like the TLS lesson, it embeds a test
CA so that it runs offline.
import std.bytes
import std.http
import std.http.client
import std.http.https
import std.http.server
import std.http.tls
import std.io
import std.net
import std.time
import std.tls
from std.http.server import Exchange
from std.http.tls import TlsTransport
from std.sync import CancellationToken
# Test-only certificates: a CA, and a leaf it signed for localhost and
# 127.0.0.1. Never use a published key outside an example.
def ca_pem() -> str:
return "-----BEGIN CERTIFICATE-----\nMIIBmjCCAT+gAwIBAgIULOQ2Qxr8CVZY2bG0uRYKKEdMD3swCgYIKoZIzj0EAwIw\nGTEXMBUGA1UEAwwOUGxlbnR5IFRlc3QgQ0EwIBcNMjQwMTAxMDAwMDAwWhgPMjEy\nNDAxMDEwMDAwMDBaMBkxFzAVBgNVBAMMDlBsZW50eSBUZXN0IENBMFkwEwYHKoZI\nzj0CAQYIKoZIzj0DAQcDQgAEkiB3Ogehrrt6YbQghwDPv6tkhE+z9zTCVih8fvtR\nZ1/691Nfh7/J9IgaNgNSTMJjGTolzGnjaCUDqoARlFyM+6NjMGEwHQYDVR0OBBYE\nFOTFBDgWXHj/qmVJDMlAHYCEZFX5MB8GA1UdIwQYMBaAFOTFBDgWXHj/qmVJDMlA\nHYCEZFX5MA8GA1UdEwEB/wQFMAMBAf8wDgYDVR0PAQH/BAQDAgIEMAoGCCqGSM49\nBAMCA0kAMEYCIQD41ATQnIjKkKMLQAb9AyoyOFmljYZyz348XJEIbqMgKwIhAL3m\nmWd8On5gvkfytspatH5mDWOcSyW+au7vuIC2gHhO\n-----END CERTIFICATE-----\n"
def leaf_pem() -> str:
return "-----BEGIN CERTIFICATE-----\nMIIBnDCCAUKgAwIBAgIBATAKBggqhkjOPQQDAjAZMRcwFQYDVQQDDA5QbGVudHkg\nVGVzdCBDQTAgFw0yNDAxMDEwMDAwMDBaGA8yMTI0MDEwMTAwMDAwMFowFDESMBAG\nA1UEAwwJbG9jYWxob3N0MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAE8THNQeTi\nJ6Eq0tSxtHW5Q4TCsbIqB3MOtM98ERyKgl6kHeHCeDZYKx8B+VRGT2jkWuTeN24Q\nljwUfh7uh7Iaz6N+MHwwGgYDVR0RBBMwEYIJbG9jYWxob3N0hwR/AAABMAkGA1Ud\nEwQCMAAwEwYDVR0lBAwwCgYIKwYBBQUHAwEwHQYDVR0OBBYEFEIFC2Fyd4cfAKTx\nMiNCMdDkqDNbMB8GA1UdIwQYMBaAFOTFBDgWXHj/qmVJDMlAHYCEZFX5MAoGCCqG\nSM49BAMCA0gAMEUCIAUF3HH43lnirgSN1Vr4twZrFmXKBxlbYNQni/h4ZaxhAiEA\n5rD/yE1B4IsbEGCMlyAeX507leXfGY9+wrHaxikU/w0=\n-----END CERTIFICATE-----\n"
def leaf_key() -> str:
return "-----BEGIN PRIVATE KEY-----\nMIGHAgEAMBMGByqGSM49AgEGCCqGSM49AwEHBG0wawIBAQQglvSEE3H7v+mInxSF\nAyWFDKYpzpXr20wH8mfFY3bCaB2hRANCAATxMc1B5OInoSrS1LG0dblDhMKxsioH\ncw60z3wRHIqCXqQd4cJ4NlgrHwH5VEZPaORa5N43bhCWPBR+Hu6HshrP\n-----END PRIVATE KEY-----\n"
pub enum Route:
Count
pub class Site:
pub def handle(self, route: &Route, exchange: &mut Exchange[TlsTransport]) -> Result[(), AllocError | std.http.HttpError | std.bytes.BytesError]:
mut body = std.bytes.empty()?
exchange.read_all(&mut body)?
exchange.send_text(200, &f"{body.len()} bytes"?)?
Ok(())
type VisitError = AllocError | IoError | std.bytes.BytesError | std.http.client.ClientError | std.tls.TlsError | RangeError
def visit(port: u16, stop: CancellationToken) -> Result[(), VisitError]:
with stop.cancel_on_exit():
never = CancellationToken()?
deadline = std.time.deadline_after(&std.time.monotonic_now(), &std.time.seconds(5u32))?
base = f"https://localhost:{port}"?
roots = std.tls.roots_from_pem(&ca_pem())?
tls = std.tls.client_config(&roots)?
client = std.http.https.client(std.http.client.config()?, tls)?
home = client.get(&f"{base}/"?, &deadline, &never)?
print(home.status, home.body.to_text()?)?
mut source = std.io.cursor(std.bytes.from_text("a body read in pieces")?)
request = std.http.client.request("PUT", &f"{base}/upload"?)?
mut response = client.send_from(request, &mut source, &std.http.client.chunked(), &deadline, &never)?
mut answer = std.bytes.empty()?
response.read_all(&mut answer)?
print(response.status, answer.to_text()?)?
client.finish(response)
stats = client.stats()
print(stats.opened, stats.reused)?
Ok(())
type HttpsSetupError = std.http.server.ServerError | std.net.NetError | std.tls.TlsError
def main() -> Result[(), SpawnError[()] | AllocError | HttpsSetupError | VisitError]:
mut routes = std.http.server.router[Route]()?
routes.add("GET", "/", Route.Count)?
routes.add("PUT", "/upload", Route.Count)?
listener = std.net.listen(&std.net.loopback_v4(0u16), 16)?
port = listener.local_address()?.port
stop = CancellationToken()?
server = std.tls.server_config(&leaf_pem(), &leaf_key())?
with spawn(visit, port, stop.share()).drop_job()? as task:
drop(std.http.tls.serve(listener, server, &routes, &Site(), &std.http.server.config(), &stop)?)
task.join()?
Ok(())
200 0 bytes
200 21 bytes
1 1
std.http.https.client takes the client config and a TLS ClientConfig.
The client uses the URL’s host, here localhost, as the name the
certificate must carry. Verification cannot be turned off: to trust a
private server, build the ClientConfig from its CA as this program does,
and use std.http.https.system_client for public servers. A certificate
from an unknown CA fails with Tls(UntrustedCertificate), and one for
another host with Tls(WrongHost). Both requests used one TLS connection.
send_from reads the body from any std.io reader in 16 KiB pieces, so a
large upload never sits in memory whole. chunked() sends it in chunks of
unknown total length; sized(n) sends Content-Length: n and fails with
WrongBodyLength if the reader yields a different amount. Setting
expect_continue on the upload asks the server to approve the request
before the body is sent.
A streamed body can be read only once. The client still replays a request on a fresh connection when a pooled one was closed before any of the body was read, but it returns a redirect that would need the body again rather than following it.
With config.proxy set, an https request asks the proxy to open a tunnel
with CONNECT, then runs TLS to the server through it, so the proxy sees
neither the request nor the response.
The HTTPS support is a separate module: a program that imports only
std.http.client does not include the TLS library.
Isolate HTTP connections with actors
std.actor.http.server gives each
connection its own handler and heap. A contained trap loses that connection;
other connections can continue. Its qualification record
states which deployment checks have passed.
Start with a complete, bounded request handler. This endpoint counts requests on one connection. The ordinary HTTP client runs on a separate native thread; blocking client operations must not run inside an actor callback.
import std.actor.http.server
import std.bytes
import std.http
import std.http.client
import std.net
import std.time
from std.actor.http.server import Request, Response, ServerError
from std.sync import CancellationToken
type HandlerError = OverflowError[i64] | AllocError | std.http.HttpError | ServerError
pub class Counter:
pub requests: i64
pub def handle(self: &mut Counter, request: Request, response: &mut Response) -> Result[(), HandlerError]:
if request.target.path != "/":
response.send_text(404, "Not Found")?
return Ok(())
if request.head.method != "GET" and request.head.method != "HEAD":
mut head = std.http.response(405, "Method Not Allowed")?
head.add("Allow", "GET, HEAD")?
response.respond(head, b"")?
return Ok(())
self.requests += 1
response.send_text(200, &str.from(self.requests)?)?
Ok(())
def make_counter() -> Result[Counter, HandlerError]:
Ok(Counter(0))
type VisitError = AllocError | IoError | std.bytes.BytesError | std.http.client.ClientError | RangeError
def visit(port: u16, stop: CancellationToken) -> Result[(), VisitError]:
with stop.cancel_on_exit():
client = std.http.client.client(std.http.client.config()?)?
url = f"http://127.0.0.1:{port}/"?
deadline = std.time.deadline_after(&std.time.monotonic_now(), &std.time.seconds(5u32))?
never = CancellationToken()?
for _ in range(2):
reply = client.get(&url, &deadline, &never)?
print(reply.status, reply.body.to_text()?)?
Ok(())
def main() -> Result[(), SpawnError[()] | std.net.NetError | HandlerError | VisitError]:
listener = std.net.listen(&std.net.loopback_v4(0u16), 16)?
port = listener.local_address()?.port
stop = CancellationToken()?
with spawn(visit, port, stop.share()).drop_job()? as task:
report = std.actor.http.server.serve(listener, make_counter, &std.actor.http.server.config(), &stop)?
task.join()?
print(report.requests)?
Ok(())
200 1
200 2
2
make_counter runs inside the new connection’s heap. Two connections get
different counters; the client here reuses one connection. The mutable
handle receiver updates only that connection’s state. Store longer-lived
text as owned Bytes; the request’s str fields are available during the
callback but cannot become fields of the actor’s handler.
The handler checks the method as well as the path, includes Allow in its
405 response, and propagates response errors. HEAD follows the GET handler
but does not send body bytes. The client cancels the server even if a request
fails, and task.join()? preserves that failure rather than printing success.
The complete request body is bounded by protocol limits and the actor heap. A whole response must fit the configured output queue. Use the event interface for larger streams or multi-step nonblocking work: retain progress, handle short writes, and resume when output becomes writable.
Use the worker server when sequential handlers and its routing/application helpers suit the service. Use the actor server when connection isolation justifies an event-driven application model. Neither choice turns an infinite loop, stack overflow or foreign-library failure into a safely recoverable request; consult the reliability boundary.
Talk over WebSockets
A WebSocket starts as an HTTP request and then stays open, carrying
messages both ways. std.http.websocket
does both halves: accept turns a server exchange into a
WebSocket, and connect opens one as a client.
This program serves /chat on a loopback port. The handler echoes each
text message in upper case until the client closes. The client, on another
thread, sends two messages and then closes the connection.
import std.time
import std.bytes
import std.http.server
import std.http.websocket
import std.net
from std.http.server import Exchange
from std.http.websocket import Kind, Message, WebSocketError
from std.net import TcpStream
from std.sync import CancellationToken
from std.time import Deadline
type ChatError = WebSocketError | std.bytes.BytesError | RangeError | AllocError
def soon() -> Result[Deadline, RangeError]:
Ok(std.time.deadline_after(&std.time.monotonic_now(), &std.time.seconds(5u32))?)
pub enum Route:
Chat
pub class Echo:
pub def handle(self, route: &Route, exchange: &mut Exchange[TcpStream]) -> Result[(), ChatError]:
wait_cancel = exchange.cancel_token()
mut socket = std.http.websocket.accept(exchange, &std.http.websocket.config())?
mut message = std.bytes.empty()?
while True:
message.clear()
match socket.receive(&mut message, &soon()?, &wait_cancel)?:
case Message.Text:
shout = message.to_text()?.upper()?
socket.send_text(&shout, &soon()?, &wait_cancel)?
case Message.Binary:
socket.close(1003, "text only", &soon()?, &wait_cancel)?
return Ok(())
case Message.Close(code):
return Ok(())
Ok(())
def chat(port: u16, stop: CancellationToken) -> Result[(), ChatError | IoError]:
with stop.cancel_on_exit():
never = CancellationToken()?
url = f"ws://localhost:{port}/chat"?
mut socket = std.http.websocket.connect(&url, &std.http.websocket.config(), &soon()?, &never)?
mut reply = std.bytes.empty()?
for line in ["hello", "still there?"]?:
socket.send_text(&line, &soon()?, &never)?
reply.clear()
match socket.receive(&mut reply, &soon()?, &never)?:
case Message.Text:
print(reply.to_text()?)?
case _ as other:
print(other)?
socket.close(1000, "bye", &soon()?, &never)?
Ok(())
def main() -> Result[(), SpawnError[()] | AllocError | std.http.server.ServerError | std.net.NetError | ChatError | IoError]:
mut routes = std.http.server.router[Route]()?
routes.add("GET", "/chat", Route.Chat)?
listener = std.net.listen(&std.net.loopback_v4(0u16), 16)?
port = listener.local_address()?.port
stop = CancellationToken()?
with spawn(chat, port, stop.share()).drop_job()? as task:
drop(std.http.server.serve(listener, &routes, &Echo(), &std.http.server.config(), &stop)?)
task.join()?
Ok(())
HELLO
STILL THERE?
receive appends one whole message to a buffer you own and says what kind
it was. A message sent in several frames arrives joined; pings are answered
on the way. Config bounds each frame and each message, 64 KiB and 1 MiB by
default, and a peer that sends more is closed with code 1009 and the error
TooLarge. Text that is not UTF-8 closes the connection with 1007.
Every call takes a deadline. A handler waiting in receive also wakes when
the server begins to shut down, with Stopping; it should then close with
1001, “going away”. Each open WebSocket holds a server worker until its
handler returns, so set config.workers for the connections you expect.
close sends a close code and reason, waits for the other side’s close and
then shuts the connection down. When the other side closes first,
receive answers for you and returns Close with its code.
For wss URLs, std.http.wss.connect takes a TLS ClientConfig as the
HTTPS client does; on the server, accept works the same
on exchanges from std.http.tls.serve.
Stream events to browsers
Server-Sent Events keep one HTTP response open and send it a line-based
event each time something happens. Browsers read them with EventSource,
which reconnects by itself when the connection drops.
std.http.sse turns a
server exchange into an event stream.
This program serves /clock on a loopback port. The handler sends three
tick events a short interval apart, then an end event, and closes the
stream. The client, on another thread, reads the whole response with the
HTTP client and prints it without its final blank line.
import std.time
import std.bytes
import std.http
import std.http.client
import std.http.server
import std.http.sse
import std.net
from std.http.server import Exchange
from std.http.sse import SseError
from std.net import TcpStream
from std.sync import CancellationToken
from std.time import Deadline
def after(milliseconds: u32) -> Result[Deadline, RangeError]:
Ok(std.time.deadline_after(&std.time.monotonic_now(), &std.time.milliseconds(milliseconds))?)
pub enum Route:
Clock
pub class Clock:
pub def handle(self, route: &Route, exchange: &mut Exchange[TcpStream]) -> Result[(), SseError | AllocError | RangeError]:
wait_cancel = exchange.cancel_token()
mut events = std.http.sse.accept(exchange)?
events.retry(&std.time.milliseconds(2000u32), &after(1000)?, &wait_cancel)?
for n in range(3):
match events.wait(&after(20)?, &wait_cancel):
case Ok(_):
pass
case Err(SseError.Stopping):
events.send("server stopping", &after(1000)?, &wait_cancel, event="end")?
return Ok(())
case Err(error):
return Err(error)
id = f"{n}"?
events.send(&f"tick {n}\nall well"?, &after(1000)?, &wait_cancel, event="tick", id=Some(id))?
events.send("done", &after(1000)?, &wait_cancel, event="end")?
events.close()
Ok(())
type WatchError = AllocError | IoError | std.bytes.BytesError | std.http.client.ClientError | RangeError
def watch(port: u16, stop: CancellationToken) -> Result[(), WatchError]:
with stop.cancel_on_exit():
never = CancellationToken()?
client = std.http.client.client(std.http.client.config()?)?
reply = client.get(&f"http://127.0.0.1:{port}/clock"?, &after(5000)?, &never)?
match std.http.find(&reply.head.headers, "content-type"):
case Some(index):
print(reply.status, reply.head.headers[index].value)?
case Nothing:
print(reply.status)?
print(reply.body.to_text()?.strip()?)?
Ok(())
def main() -> Result[(), SpawnError[()] | std.http.server.ServerError | std.net.NetError | AllocError | WatchError]:
mut routes = std.http.server.router[Route]()?
routes.add("GET", "/clock", Route.Clock)?
listener = std.net.listen(&std.net.loopback_v4(0u16), 16)?
port = listener.local_address()?.port
stop = CancellationToken()?
with spawn(watch, port, stop.share()).drop_job()? as task:
drop(std.http.server.serve(listener, &routes, &Clock(), &std.http.server.config(), &stop)?)
task.join()?
Ok(())
200 text/event-stream
retry: 2000
event: tick
id: 0
data: tick 0
data: all well
event: tick
id: 1
data: tick 1
data: all well
event: tick
id: 2
data: tick 2
data: all well
event: end
data: done
Each event is an optional event name, an optional id, and one data
line for each line of its text, ended by an empty line. A browser’s
EventSource joins the data lines back together and calls the listener
for the event’s name, or message when it has none. retry tells it how
long to wait before reconnecting. Every call writes and flushes whole
events before it returns, within its deadline.
The response has no length: it ends when the handler closes the stream, so
it is not cut off by the server’s response_timeout. A client that
reconnects sends the last id it received, which the handler reads as
events.last_event_id to resume after it.
wait paces the stream. It returns Ok at its deadline, Disconnected as
soon as the client goes away, and Stopping when the server begins to shut
down; the stream is still open then, so the handler can say goodbye. A
send to a client that has gone away also fails with Disconnected. Each
open stream holds a server worker until its handler returns, so set
config.workers for the streams you expect.
Wait for I/O with a deadline
A worker can own or borrow a File like any other value. To keep it from
blocking forever on a pipe that never delivers, wait with
std.ready: every wait has a deadline, and a
cancellable wait checks its token at least every 10 milliseconds.
import std.bytes
import std.io
import std.process
from std.io import FileStream
from std.process import Stdio
from std.ready import WaitError, read_before
from std.sync import CancellationToken
from std.time import Deadline, deadline_after, milliseconds, monotonic_now, sleep
def first_bytes(stream: &mut FileStream, deadline: &Deadline, stop: &CancellationToken) -> Result[str, std.bytes.BytesError | AllocError]:
mut buffer = std.bytes.with_capacity(16)?
buffer.resize(16, 0u8)?
text = match read_before(stream, &mut buffer, 0, 16, deadline, stop):
case Ok(count):
str.from(count)?
case Err(WaitError.TimedOut):
"timed out"
case Err(WaitError.Cancelled):
"cancelled"
case Err(WaitError.Stream(_)):
"failed"
Ok(text)
type PipeWaitError = str | RangeError | std.bytes.BytesError
def main() -> Result[(), SpawnError[()] | IoError | AllocError | PipeWaitError]:
mut command = std.process.shell("sleep 10")?
command.stdout(Stdio.Piped)
mut child = command.spawn()?
pipe: Result[File, str] = child.take_stdout().ok_or("missing piped stdout")
mut quiet = std.io.from_file(pipe?)
never = CancellationToken()?
soon = deadline_after(&monotonic_now(), &milliseconds(20))?
with spawn(first_bytes, &mut quiet, &soon, &never)? as task:
print(task.join()?)?
stop = CancellationToken()?
later = deadline_after(&monotonic_now(), &milliseconds(60000))?
observer = stop.share()
with spawn(first_bytes, &mut quiet, &later, &observer)? as task:
sleep(&milliseconds(20))
stop.cancel()
print(task.join()?)?
child.kill()?
child.wait()?
Ok(())
timed out
cancelled
The child writes nothing, so the first worker’s 20-millisecond deadline passes. The second worker has a minute, but the parent cancels it, and the cancellation wakes its wait at once. Either way the worker returns normally and the scope joins it; the pipe and buffer are unchanged.
A worker may not change the working directory, because every thread resolves relative paths against it:
import std.env
import std.path
def worker() -> Result[(), AllocError | IoError]:
std.env.set_current_dir(&std.path.from_text("/")?)
def main() -> Result[(), SpawnError[()]]:
with spawn(worker).drop_job()?:
pass
Ok(())
changing the working directory affects every thread's relative paths
Change it on the main thread before starting workers, or give a child its own
directory with Command.current_dir. See
native effects on worker threads
for the rules each native operation follows.
Working with C libraries
These lessons require core modules, text, borrowing, resource owners, and Result.
They are independent of the generic, callback, and concurrency paths.
- Call C libraries
- Pass text to C
- Own a foreign handle
- Build a C library
- Report errors through an exported function
- Export owned objects
- Load a library at runtime
Call C libraries
Declare C functions in a .plentyi interface and import it as a module. C’s
abs uses int, which is i32 on Plenty’s supported target. The wrapper rejects
the minimum integer, which C cannot negate.
Save this as c_math.plentyi:
extern def abs_raw(value: i32) -> i32 = "abs"
pub def absolute(value: i32) -> Option[i32]:
if value == -2147483648i32:
return Nothing
Some(abs_raw(value))
import c_math
def main() -> Result[(), IoError]:
print(c_math.absolute(-42i32))?
print(c_math.absolute(-2147483648i32))?
Ok(())
Option[i32].Some(42)
Option[i32].Nothing
The C standard library is already linked. For another library, pass its archive
or driver options explicitly, for example --link-arg /path/to/libexample.a or
--link-arg -L/path/to/libs --link-arg -lexample. A shared library also needs to
be discoverable by the system loader when the application runs.
The interface author must check the C header and document pointer lifetimes,
ownership, and errors. Plenty does not verify the native implementation;
application code should use typed wrappers.
pub exposes a name to Plenty imports; it does not export a C symbol. See the
C interface reference for the supported ABI.
Pass text to C
Choose the text adapter to match C’s signature. as utf8 lends a pointer and
byte length without allocating. as c_string makes a terminated copy and reports
embedded NULs or allocation failure through CStrError.
Save this interface as c_text.plentyi:
# strlen reads only during the call and does not retain or return its input.
# C signature: size_t strlen(const char *text).
pub extern def byte_length(text: &str as c_string) -> Result[u64, CStrError] = "strlen"
import c_text
def main() -> Result[(), AllocError | IoError | CStrError]:
text = "é"
print(c_text.byte_length(&text)?)?
with_nul = "a\0b"
match c_text.byte_length(&with_nul):
case Ok(size):
print(size)?
case Err(error):
print(error)?
print(text)?
Ok(())
2
CStrError.EmbeddedNul
é
The C call returns a plain size_t, which is u64 on this target. The generated
adapter returns Result[u64, CStrError] because preparing the argument may fail.
The temporary buffer is freed after the call; the original text is unchanged.
CStrError.Allocation(error) reports a failed buffer allocation. These conversion
errors do not replace any error status that the C function itself returns.
An interface for uint64_t checksum(const uint8_t *data, size_t size) would
instead declare text: &str as utf8. It receives all bytes, including embedded
NULs, and needs no temporary copy. Neither adapter lets C retain or modify the
buffer. Use a binding with an explicit ownership contract for those cases.
Own a foreign handle
Own a C resource with a private opaque pointer and a destructor. This interface wraps a temporary C file, keeping raw operations private and reporting acquisition failure through its public factory.
Save this as c_scratch.plentyi:
opaque RawFile
extern def create_raw() -> RawFile = "tmpfile"
extern def close_raw(file: RawFile) -> i32 = "fclose"
extern def position_raw(file: RawFile) -> i64 = "ftell"
# fwrite borrows the bytes and file only for the call; it retains neither.
# The adapter expands to (data, byte_len), followed by count and file.
extern def write_raw(text: &str as utf8, count: u64, file: RawFile) -> u64 = "fwrite"
pub enum ScratchError:
Create
Write
Position
pub class Scratch:
raw: RawFile
def __init__(self) -> ():
self.raw = RawFile.null()
def __del__(self) -> ():
if not self.raw.is_null():
drop(close_raw(self.raw))
pub def write(self: &mut Scratch, text: &str) -> Result[(), ScratchError]:
if len(text) == 0:
return Ok(())
if write_raw(text, 1u64, self.raw) != 1u64:
return Err(ScratchError.Write)
Ok(())
pub def position(self: &mut Scratch) -> Result[i64, ScratchError]:
value = position_raw(self.raw)
if value < 0:
return Err(ScratchError.Position)
Ok(value)
pub def create() -> Result[Scratch, ScratchError]:
mut owner = Scratch()
owner.raw = create_raw()
if owner.raw.is_null():
return Err(ScratchError.Create)
Ok(owner)
import c_scratch
def main() -> Result[(), c_scratch.ScratchError | IoError]:
mut file = c_scratch.create()?
text = "é\0A"
file.write(&text)?
print(file.position()?)?
Ok(())
4
Initialize the owner to null before acquiring the resource, so cleanup is valid even if acquisition fails. After success, the destructor closes the file through its originating library, including on early exit. The owner cannot be copied.
The destructor discards close errors. Offer an explicit fallible close method
when callers need to handle them.
Native ownership transfer needs its own wrapper: clear the field when C consumes
the handle, and retain or return the owner when C leaves ownership with the caller.
Build a C library
Use export def to name the C symbol for a function. This calc.plenty module
can also be imported and called by an ordinary Plenty program:
pub enum CalcError:
Overflow
pub export def add(a: i32, b: i32) -> Result[i32, CalcError] = "calc_add":
"Add two checked integers."
result = checked:
a + b
result.map_error(def(error: OverflowError[i32]) -> CalcError: CalcError.Overflow)
pub export def increment(value: &mut i32) -> Result[(), CalcError] = "calc_increment":
result = checked:
*value + 1
*value = result.map_error(def(error: OverflowError[i32]) -> CalcError: CalcError.Overflow)?
Ok(())
import calc
def main() -> Result[(), IoError | calc.CalcError]:
print(calc.add(20, 22)?)?
mut value = 10i32
calc.increment(&mut value)?
print(value)?
mut maximum = i32.MAX
match calc.increment(&mut maximum):
case Err(calc.CalcError.Overflow):
print("overflow", maximum == i32.MAX)?
case Ok(_):
print("unexpected success")?
Ok(())
42
11
overflow True
Arithmetic errors are mapped locally to a payload-free error enum supported
by the C ABI. increment computes before assigning, so overflow leaves its
borrowed value unchanged. Builtin OverflowError[T] carries a payload and
cannot be exported directly. See exported error contracts.
pub makes the function accessible through source imports. export independently
creates a C entry point in a library build. Libraries do not need main.
Save the module as calc.plenty, create a build directory, and run:
plenty --shared-library calc.plenty --library-name calc -o build/libcalc.so
The output includes calc.h, with a status-returning calc_add declaration
and generated contract comments, and calc.plentyi, which Plenty consumers can import. Copy that generated
interface into the consuming project’s source root and link libcalc.so using
--link-arg; it replaces the source module there. C consumers include calc.h
and link the library.
If you only have the binary, recover its Plenty interface without loading it:
plenty --extract-interface build/libcalc.so --library-name calc -o calc.plentyi
plenty --verify-interface build/libcalc.so calc.plentyi
For a static archive, use --static-library instead. Pass the native arguments
listed in calc.link-args.txt after the archive when linking a C program.
For C callers, increment takes an int32_t * to an initialized value, borrowed
exclusively for the call. Plenty callers get this borrowing contract from the
generated interface.
Exports support numeric scalars, borrows, unit, supported Result values, and
owned class handles. Read the generated
header’s requirements even for simple APIs: runtime traps can terminate the
process, and current libraries require serialized calls on one caller thread.
See the library reference
for supported export types and packaging details.
Report errors through an exported function
Exports can return a numeric error through Result. Plenty callers use the
ordinary result; C callers receive a status and separate output pointers:
export def positive(value: i32) -> Result[i32, i32] = "calc_positive":
if value < 0:
Err(1)
else:
Ok(value)
def main() -> Result[(), AllocError | IoError | i32]:
print(positive(42)?)?
print(str.repr(positive(-1))?)?
Ok(())
42
Result[i32, i32].Err(1)
The generated C declaration is uint32_t calc_positive(int32_t p0, int32_t *out_ok, int32_t *out_error). Status 0 writes out_ok; status 1 writes
out_error. Its header explains output storage and borrowing requirements.
Name errors with an enum
A numeric code means little to a C caller. Return an enum whose variants have no payloads instead:
enum CalcError:
Empty
Negative
export def checked(value: i32) -> Result[i32, CalcError] = "calc_checked":
if value == 0:
Err(CalcError.Empty)
elif value < 0:
Err(CalcError.Negative)
else:
Ok(value)
def main() -> Result[(), AllocError | IoError | CalcError]:
print(checked(42)?)?
print(str.repr(checked(-1))?)?
Ok(())
42
Result[i32, CalcError].Err(CalcError.Negative)
C receives a uint32_t code, the variant’s position in the declaration. The
header names each code:
#define calc_CalcError_Empty UINT32_C(0)
#define calc_CalcError_Negative UINT32_C(1)
uint32_t calc_checked(int32_t p0, int32_t *out_ok, uint32_t *out_error);
The generated calc.plentyi declares its own CalcError, so Plenty consumers
match calc.CalcError variants. Enums with payloads and sums such as
AllocError | CalcError cannot cross the C boundary.
Export owned objects
C receives an exported class as an opaque handle. The generated header names its destroy function and ownership rules.
pub enum CalcError:
Overflow
pub class Counter:
value: i64
def __del__(self) -> ():
drop(print(self.value))
pub export def create(value: i64) -> Result[Counter, AllocError] = "counter_create":
Ok(Counter(value))
pub export def read(owner: &Counter) -> i64 = "counter_read":
owner.value
pub export def increment(owner: &mut Counter) -> Result[(), CalcError] = "counter_increment":
result = checked:
owner.value + 1
owner.value = result.map_error(def(error: OverflowError[i64]) -> CalcError: CalcError.Overflow)?
Ok(())
pub export def finish(owner: Counter) -> () = "counter_finish":
drop(owner)
import counter
def main() -> Result[(), AllocError | IoError | counter.CalcError]:
mut owner = counter.create(42)?
counter.increment(&mut owner)?
print(counter.read(&owner))?
print("ready")?
counter.finish(owner)
Ok(())
43
ready
43
Build counter.plenty with --shared-library --library-name counter to get a C
factory returning a status and writing a counter_Counter * output on success.
A class returned to C must be declared as Result[Class, AllocError], because
allocating its handle can fail. The library allocates the handle before running
the function, so a failure has no other effects. Release the output exactly once
with counter_Counter_destroy.
counter_read borrows a const counter_Counter *; counter_increment borrows a
counter_Counter * exclusively. Neither transfers ownership.
counter_finish consumes its handle. A C caller must not use or destroy that
handle again; a Plenty caller gets a moved-value error if it tries to reuse
owner. A consuming function keeps this rule even if it returns an error.
Plenty consumers get automatic cleanup through the generated counter.plentyi.
Keep the originating library loaded for every owner’s lifetime.
The lifecycle trace uses drop(print(...)) deliberately: cleanup cannot return
an error, and a failed diagnostic write must not interrupt cleanup. For output
that must succeed, provide an explicit fallible operation and propagate its
result before leaving the scope.
The fallible increment maps arithmetic overflow to the payload-free
CalcError.Overflow at the ABI boundary. It leaves the object unchanged on
failure; callers must handle its status before continuing.
Load a library at runtime
A program can choose a native library while it runs. Plenty still compiles the
function signatures ahead of time. Start with a small library, saved as
counter.plenty:
enum CalcError:
Overflow
class Counter:
value: i64
def __del__(self) -> ():
drop(print("released"))
export def create(value: i64) -> Result[Counter, AllocError] = "counter_create":
Ok(Counter(value))
export def read(counter: &Counter) -> i64 = "counter_read":
counter.value
export def increment(counter: &mut Counter) -> Result[(), CalcError] = "counter_increment":
result = checked:
counter.value + 1
counter.value = result.map_error(def(error: OverflowError[i64]) -> CalcError: CalcError.Overflow)?
Ok(())
Build the shared library and its runtime-loading module:
plenty --shared-library counter.plenty --library-name counter -o libcounter.so
plenty --runtime-interface counter.plenty --library-name counter -o plugin.plentyi
Now save this consumer beside plugin.plentyi and run it from the directory
containing libcounter.so. No library link arguments are needed.
import plugin
def main() -> Result[(), AllocError | IoError | LoadError | plugin.CalcError]:
path = "./libcounter.so"
library = plugin.load(&path)?
mut counter = library.create(42)?
library.increment(&mut counter)?
print(library.read(&counter))?
mut maximum = library.create(i64.MAX)?
match library.increment(&mut maximum):
case Err(plugin.CalcError.Overflow):
print("overflow", library.read(&maximum) == i64.MAX)?
case Ok(_):
print("unexpected success")?
drop(maximum)
drop(library)
drop(counter)
missing = "./missing-library.so"
match plugin.load(&missing):
case Ok(_):
print("unexpected library")?
case Err(error):
print(error)?
Ok(())
43
overflow True
released
released
LoadError.OpenFailed
Loading checks the library’s exact interface contract and every required export.
A changed contract reports LoadError.IncompatibleContract; a missing export
reports LoadError.MissingSymbol. Imports alone load nothing.
The loaded methods preserve the exports’ signatures, borrows, and errors.
Dropping library does not invalidate its objects: counter still uses the
matching destructor, and native code stays loaded until process exit.
Keep objects with the library instance that created them. Passing an object to a different loaded instance stops with a diagnostic before making the native call. Loading trusts native code: metadata checks compatibility, and library constructors may run before a later check reports an error.
The lifecycle trace uses drop(print(...)) deliberately: cleanup cannot return
an error, and a failed diagnostic write must not interrupt cleanup. For output
that must succeed, provide an explicit fallible operation and propagate its
result before leaving the scope.
The maximum-value object exercises the loaded error-status path and is
released separately. The fallible increment maps arithmetic overflow to the payload-free
CalcError.Overflow at the ABI boundary. It leaves the object unchanged on
failure; callers must handle its status before continuing.
Keep exploring
Choose a small program of your own: a report, a file converter, or a command-line checker. Start with the core features and add an optional path when you need it.
Use the reference for exact contracts and the implementation status to check feature support. The backlog records planned work.
Reference
The reference describes implemented behavior and current restrictions. Start with the language overview for the connecting ideas, or use the support index to find a contract.
- Language contract: source and modules, types, expressions, ownership, functions, generics, errors and resource scopes, and iteration define what programs mean.
- Built-in facilities and interoperability: built-in operations, I/O, concurrency, and C interfaces specify available operations, ownership transfer, allocation, and failure behavior.
- Standard library: imported
stdmodules, currentlystd.math, are written in Plenty and embedded in the compiler. - Toolchain and implementation: building programs explains commands and artifacts; architecture explains checking, native compilation, and runtime mechanisms.
- Comparisons: Plenty and Rust compares
ownership and what Rust’s
unsafepermits.
Learn Plenty teaches the language through executable examples. Supplementary examples cover additional operations. Design alternatives live in Proposals; future work lives in the backlog.
Language contract
These pages define source syntax and what programs mean. Read the overview, then follow source and modules, types, expressions, ownership, functions, generics, errors and resource scopes, and iteration.
Operation signatures and their effects belong to built-in facilities; compiler/runtime mechanisms belong to implementation.
Language overview
Plenty is a statically typed, ahead-of-time compiled language with Python-shaped syntax and Rust-influenced behaviours. Its design favors fast compilation, explicit interfaces, and predictable memory use.
Every function declares its parameter and return types. Local expressions can infer types, while generics reuse definitions at concrete types. Classes and enums have fixed declarations rather than dynamic attributes or inheritance.
Ownership makes copies, moves, shared access, and exclusive mutation explicit. Owned resources have deterministic cleanup. Recoverable errors are values; operations that allocate can report allocation failure without exceptions.
The language contract describes source organization, types, expressions, callable values, and iteration. Built-in operation contracts belong to facilities; building and executing programs belongs to the toolchain.
Source files and declarations
Spaces delimit indented suites. Tabs are rejected, and indentation must return
to an existing level. Blank lines and comment-only lines do not affect
indentation. Parentheses permit multiline expressions. Comments start with #.
Identifiers contain ASCII letters, digits, and underscores and cannot start with
a digit. Keywords and names beginning __plenty_ are reserved.
Diagnostics use one-based line and column positions.
Strings accept
single, double, or triple quotes; triple quotes permit physical newlines. The
first standalone string in an indented function body is its documentation. Use
return "text" when that first string should be the returned value. A b
prefix makes a byte literal, b"...", which holds ASCII
characters and \xHH escapes. Strings accept the
escapes \0, \n, \r, \t,
\\, \', \", and \u{H...} for a scalar by code point. Integer
literals are decimal, hexadecimal (0xFF), octal
(0o17), or binary (0b1010), with optional _ digit separators and width
suffixes.
Module scope contains functions, classes, enums, protocols, type aliases, constants, static assertions, imports, and target conditionals. Executable statements and mutable bindings belong inside functions; there are no global variables. Named functions belong at module scope; classes and enums can declare methods, and protocols declare method requirements. Anonymous functions can appear in function bodies; see functions.
A line holding only @name annotates the declaration below it. The set of
annotations is closed and compiler-known: @copy and @newtype on classes,
and @redacted on classes and enums.
See annotations.
Names follow lexical scope and prelude rules. Modules use
explicit imports, with visibility controlled by
pub. An application has a main entrypoint.
Keywords
A keyword cannot name a binding, parameter, field, function, type, or module. These keywords have a meaning in Plenty:
and, as, break, case, class, const, continue, def, dimension, elif,
else, enum, export, extern, False, for, from, if, import,
in, match, mut, not, opaque, or, pass, protocol, pub,
return, True, type, while, with, yield
These are reserved but have no meaning, so a Python or Rust habit is rejected instead of becoming a name:
async, await, lambda, None, struct, trait, try
None, try, and lambda have diagnostics that name the replacement:
Nothing, a Result, and an anonymous function.
self and is are not keywords. except, finally, and raise are not
reserved either, but a statement that begins like Python exception handling
is rejected.
Names and scopes
Module declarations are visible throughout their module. Functions can call later functions or recurse mutually; aliases and data declarations can refer forward. Duplicate declarations and redefinitions are rejected.
Function parameters and local bindings occupy lexical scopes. A declaration in
a branch does not escape that branch: using the name afterwards is an unknown
binding, and the diagnostic says that the earlier binding ended with its block.
Bind a mut local to a value before the block and assign it inside instead. Assigning an existing mutable local changes
that local across a branch join. Pattern bindings belong to their arm and may
shadow outer bindings. A new binding cannot read itself in its initializer.
Type lookup is separate from local expression names. A local can shadow a numeric cast name without changing type annotations. A binding shadowing an enum type qualifier is diagnosed rather than selecting a different constructor.
The built-in prelude is available without an import. It provides primitive and
container type names (including Box and Shared), standard sums and
variants (Option, Result, Some, Nothing, Ok, Err), callable and
generator types, and facility names for
files, errors, threads, channels, executors, and cancellation. FormatError,
the error of user formatting methods and of f-strings that run them, names
std.fmt.FormatError; a module that
declares or imports its own FormatError, such as std.report’s, uses that
one instead. Built-in operations
include copy, drop, len, range, text I/O, worker submission, and channel
construction and selection. Their contracts appear under
facilities and the relevant language section.
Top-level declarations cannot redefine built-ins. Locals can shadow expression
names; unqualified standard variant patterns still denote the standard variants.
Imported names are explicit private bindings. Importing a module or protocol does not activate methods or introduce transitive names.
An import binds its alias, or else its item or module path. That name cannot be
a built-in, a module declaration, or a name an earlier import bound. Importing
a standard declaration that is itself the built-in, such as
from std.concurrent import ThreadPoolExecutor, is allowed. The diagnostic
points at the bound name, says whether it conflicts with a prelude built-in or
with a declaration or import on a given line, and suggests renaming, binding
with as, or removing a redundant import. Function-local bindings may still
shadow imported names, as math = 3 inside a function can after
import std.math as math.
Modules and imports
std is a reserved toolchain namespace. The embedded modules
std.math, std.path
and std.fs are available without local
sources. Unknown std modules never
fall back to the source root. Local files/directories and declarations named
std, and imports binding a non-standard module as std, are rejected.
import package.module
import package.module as module_alias
from package.module import Name as Alias
Comma-separated imports are supported. Imports occur only at module scope and
execute no initializers. Relative imports, wildcards, re-exports, and implicit
transitive imports are unsupported. Directories are namespaces, without
__init__ execution.
The source root defaults to the entry file’s canonical directory; --module-root DIR selects another root. In a package,
the root is src/, and the first component of an import may instead be a
declared dependency alias, which names that dependency’s library; an alias may
not match a local top-level module. a.b resolves to <root>/a/b.plenty or the trusted
interface <root>/a/b.plentyi. Having both is ambiguous. Imported canonical
filenames and directories must have identifier components and cannot escape
the root. File/directory name collisions and import cycles are errors; a cycle
diagnostic names the actual dependency chain.
import a.b binds the path a.b, and a.b.f names a member of that module.
A binding cannot reuse a name the file declares or imports with from, and an
alias cannot enclose another binding: import x as a with import a.b is
rejected. Because a.plenty beside a/ is ambiguous, only a dependency alias
names a module with other modules beneath it: tz names the library, such as
src/lib.plenty, and tz.zones names src/zones.plenty of that dependency.
A file can import both under their own paths, in either order:
import tz
import tz.zones
tz.f then names the library’s f, and tz.zones.f the f of tz.zones.
If the library declares a member zones, public or private, tz.zones would
name both, so the later import is rejected with a diagnostic naming both
imports; import tz.zones as zones binds the module without the clash. The
same holds for a deeper path such as tz.rules.dst and a library member
rules. from tz import zones imports only a declaration of the library,
never the module beside it, since imports do not re-export. When the library
declares no zones and tz.zones is a module, the diagnostic says that
from imports declarations and suggests import tz.zones or
import tz.zones as zones. The same applies to a local namespace directory
(from util import helpers) and to std (from std import json).
A module path is not a value. Using tz.zones (or tz) in an expression or
type reports that it is a module and names one of its declarations, such as
tz.zones.offset; a path that encloses an imported module, such as
tz.rules with import tz.rules.dst, is reported as a module namespace. A
missing member of an imported module, such as tz.missing, reports that it
is private or is not a declared export.
Canonical paths identify modules, so imports through different aliases or diamond
dependencies preserve the same nominal types. Private declarations remain private
to their defining module, regardless of directory ancestry. All reachable modules
are checked before execution. Imported modules need no entrypoint, and their
main functions are ordinary functions.
The concurrency standard modules export existing builtin
identities. An explicit import such as from std.thread import spawn may bind
the matching builtin name; an import cannot replace a builtin with a different
symbol. Item and module aliases retain the same types and compiler safety checks.
Constants
const name: T = value declares an immutable value at module scope;
pub const exports it. T must be a concrete integer or float type, an
integer domain
such as Positive[i64], a Quantity[T, D],
bool, str, or bytes. The compiler evaluates the initializer, a constant
expression, and each read uses the result:
const KIB: i64 = 1 << 10
const MIB: i64 = KIB * KIB
const MASK: u8 = ~(1u8 << 3)
const RATIO: f32 = f32(MIB) / 3.0
const NAME: str = "plenty" + "-" + "lang"
const WIDE: bool = MIB > KIB and not (KIB == 0)
def main() -> Result[(), IoError]:
print(MIB)?
print(MASK)?
print(RATIO)?
print(NAME)?
print(WIDE)?
Ok(())
1048576
247
349525.34
plenty-lang
True
A constant expression contains literals, other constants (from this module or
imported, in any declaration order), numeric type constants such as u8.MAX or
f64.PI, numeric casts such as f32(x) and u8(x), parentheses, the unary
operators -, ~, and not, the arithmetic, bitwise, shift, comparison, and
logical operators, x if c else y, and + on two str values. Quantity
construction, quantity operators, .get() and floating .to(unit) also fold.
User function calls, f-strings, collections, and other expressions are rejected
with “not a constant expression”.
Evaluation follows the numeric rules of the same
expression at run time, so the folded value is the value the program would
compute: // floors, % takes the divisor’s sign, / divides floats only,
<< discards shifted-out bits, >> is arithmetic for signed types, and
operands must share a type. Unsuffixed literals take the type the context
expects. Where run time would trap, evaluation is a compile error instead:
overflow, division by zero, a shift count outside 0..width, and an out-of-range
float-to-integer cast. As at run time, x if c else y evaluates only the
selected branch, and and and or evaluate their right operand only when it
decides the result; the other operand is still type-checked, so
1 if D == 0 else 100 // D is valid when D is zero. A cycle
is reported with its path, for example constant A depends on itself: A -> B -> A.
An integer domain constant (NonZero[T], Positive[T] or NonNegative[T])
evaluates its initializer as T, and a value outside the domain is a compile
error. Each read is the domain value, as if the literal were written there.
Another initializer reads its integer with .get(), or names it alone as the
whole initializer of a constant of the same type:
const MAX_URL: NonNegative[i64] = 8192
const PAGE: Positive[i64] = MAX_URL.get() // 64
const FIRST: Positive[i64] = PAGE
assert PAGE.get() <= 1024
def pages(size: Positive[i64] = PAGE) -> i64:
MAX_URL.get() // size
def main() -> Result[(), IoError]:
print(pages(), pages(FIRST), MAX_URL)?
Ok(())
64 64 8192
Static assertions
assert CONDITION or assert CONDITION, "message" at module scope checks a
constant expression during compilation. A false condition fails the build with
static assertion failed, followed by the message when given. The message is
itself a constant str expression. An assertion declares no name and produces
no code.
const PAGE: i64 = 4096
assert PAGE % 64 == 0, "pages hold whole cache lines"
def main() -> Result[(), IoError | OverflowError[i64] | ZeroDivisionError]:
print(PAGE // 64)?
Ok(())
64
Module-scope assert uses the spelling of a Python assert statement, which
the compiler checks once instead of at run time. Plenty has no run-time
assert statement; inside a function, assert is an ordinary name.
Included files
include_str("path") initializes a str constant and include_bytes("path")
a bytes constant with a file’s contents:
const SCHEMA: str = include_str("sql/schema.sql")
const ICON: bytes = include_bytes("assets/icon.png")
The argument is one string literal: a path relative to the directory of the
source file that contains it. Absolute paths and paths that resolve outside the
package’s source root, including through .. or symbolic links, are errors.
include_str rejects a file that is not valid UTF-8 and names the first
invalid byte. Both forms initialize a const only and may appear in other
constant expressions, for example "> " + SCHEMA. They require a source file,
so source passed without a path, such as an editor buffer, cannot use them.
The contents become read-only data in the executable; reads allocate nothing. An included file is an input of the program cache: editing it rebuilds the program.
Binding and representation
Imports and aliases bind constants as values: import std.math as math permits
math.pi, while from std.math import pi permits pi. Constants are not
callables or mutable places; brackets on a constant index it, as in TABLE[n]
or tables.TABLE[n:]. Private constants follow the same visibility
rules as private functions. A local binding may shadow an imported constant.
Constant names share the declaration namespace with functions and types.
Constants require no module initialization or allocation. Each read of a
numeric or bool constant compiles to an immediate of the constant’s declared
type (an integer domain constant’s immediate is converted as a literal is), and each read of a str or bytes constant to a reference to read-only
data, in debug and release builds; there is no getter function, global
storage, or code run at import time. See
call-site substitution.
Visibility
pub exposes top-level functions, aliases, classes, enums, protocols, and
constants. Other declarations are private to their module. Class fields and methods require their
own pub; public enums expose every variant, but enum methods need their own pub.
Generic functions check access in their defining module, including protocol
requirements and calls on a type parameter. Calling a generic function does not
lend it the caller’s private access. To let a generic helper in another module
(such as std.io.read_exact) call a method, mark both the type pub class or
pub enum and the method pub def; a public method alone does not expose a private type.
Private fields and other private methods remain inaccessible. This keeps generic
code under the same lexical visibility rules as ordinary functions. A failing
protocol check is reported at the user’s call; nested instantiations retain the
outer call location and an instantiation note. Code within the type’s own module
can still use its private members through a locally defined generic function.
A generated field constructor is public only when its class and every field are
public. An explicit __init__ controls its own visibility. A public factory may
use private constructors inside their defining module. Construction through an
alias obeys the same access checks.
Public signatures, aliases, fields, and enum payloads cannot expose private nominal types, including through aliases or nested containers. Public generic contracts also require public arguments and constraint types; see generics.
Access checks apply to field reads, writes, borrows, methods, and construction.
Automatic destruction is unaffected by destructor privacy; direct lifecycle
calls are prohibited. Structural printing and equality include private fields,
so privacy does not hide data. pub does not specify a C ABI or binary export.
Target conditionals
if const selects code by facts about the compilation target. It may appear
at module scope, around declarations and imports, and in any function body,
around statements:
if const target.os == "windows":
import windows_console
def separator() -> str:
return "\\"
else:
def separator() -> str:
return "/"
def main() -> Result[(), IoError]:
if const target.pointer_width == 64:
limit = 1 << 40
else:
limit = 1 << 20
print(separator())?
print(limit)?
Ok(())
On a 64-bit Linux or macOS target, it prints:
/
1099511627776
The compiler decides the condition while parsing and keeps only the first arm
whose condition holds, or the else arm. Each continuation is written
elif const; an elif without const is an error. Unselected arms must
parse, but are not resolved or checked: they may import modules, call
functions, and name declarations that exist only on another target. An
unselected arm is not checked on the current target, so build on each target a
program supports.
In a function body, the selected statements take the conditional’s place in
the enclosing block; they open no new scope. A binding in the selected arm,
such as limit above, is visible after the conditional. The last statement
of a selected arm can be the function’s result.
Facts
| Fact | Known values |
|---|---|
target.os | "linux", "macos", "windows", "freebsd" |
target.arch | "x86_64", "aarch64" |
target.pointer_width | 32, 64 |
target.endian | "little", "big" |
The set is closed. An unknown fact, such as target.cpu, is an error, and so is
a comparison with a value outside the known values, such as "linx", so a
misspelling cannot silently select the else arm. The compiler currently
builds natively for GNU/Linux on x86-64 and AArch64: target.os is "linux",
target.arch is "x86_64" or "aarch64", pointer width is 64, and endian is
"little". The facts describe the packaged runtime, and an installed compiler
cannot select the other architecture with --target; see platform support.
Conditions
A condition compares one fact with a string or integer literal using == or
!=, and combines comparisons with and, or, not, parentheses, True,
and False. Any other expression is an error. A condition cannot name a
module constant: conditions are decided while parsing,
before names are resolved, so that unselected arms can name declarations that
do not exist. A constant can instead be defined differently in each arm.
Application entrypoint
A binary application requires exactly one module-level main in its entry
module, with no parameters and one of these return types:
| Return type | Process result |
|---|---|
() | Status zero |
i32 | The returned status |
Result[(), E] | Ok(()) gives zero; Err gives one |
Result[i32, E] | Ok(status) gives that status; Err gives one |
Transparent aliases are accepted. The wrapper calls main once. Small
nonnegative statuses are portable; an operating system may truncate others.
A returned Err is reported once on standard error after flushing standard
output. Its payload is destroyed exactly once, and reporting failure does not
change status one. Intermediate error propagation prints nothing. See
entrypoint reporting for the exact
format and fallback behavior.
Checking and compilation diagnose missing or invalid entrypoints before running
code or producing an output artifact. Both diagnostics list all four accepted
return types. Compilation errors execute nothing; runtime errors can leave
effects that already occurred. Library checking does not require main.
Types and data definitions
Primitive types are i8, i16, i32, i64, u8, u16, u32, u64, f32,
f64, bool, str, and bytes, the type of byte literals such
as b"\x89PNG". There is no built-in int; use a width or a
transparent alias. Numbers have local literal
inference and explicit conversions rather than implicit widening.
The unit type () denotes completion without data. It can be an expression,
function result, or enum or tuple component, including Option[()] and
Result[(), E]. Standalone unit bindings, parameters, collection elements, and
class fields are unsupported. A reference to unit, &() or &mut (), is an
ordinary reference: borrowed unit payloads have this type, and a &() parameter
lends a unit argument like any other temporary. There is no None type or value.
Tuples are structural products. Classes and enums are nominal: identical fields or payloads do not make separately declared types interchangeable. Import aliases preserve a declaration’s identity; transparent aliases create none. Generic instances with different arguments have different nominal identities.
Dimensions are compile-time types with no values. Nominal
bases and normalized products, quotients, powers and exact roots can be arguments
to generic parameters constrained by Dimension.
Type inference stays within local expressions and calls; every function interface declares its types. Collection, reference, callable, and generator types follow their respective facilities, ownership, function, and iteration contracts.
Data definitions may refer forward and recur through permitted
storage boundaries. References cannot be stored in classes,
enums, or tuples. Generators can occur inside standard Option and Result
wrappers, but not class fields, user enum payloads, or tuple components.
Quantities combine numeric storage with a compile-time dimension and check that arithmetic combines those dimensions correctly.
Numeric types and literals
For a fixed inclusive integer interval, use Bounded[T, lo, hi].
Unsuffixed integers default to i64; decimal or exponent literals default to
f64. Width suffixes include 7u8, 1f32, 1.5f32, and 1e-3f64. Decimal forms
such as .5 and 1. are accepted. Literals must fit their selected width.
Integer literals may also be hexadecimal (0x1F1E6), octal (0o755), or
binary (0b1010). The prefix letter and hexadecimal digits may be either case.
These forms are always integers: they take the integer suffixes and range
checks of decimal literals, as in 0xFFu8 and -0x80i8, and have no fraction,
exponent, or float suffix, so 0x1f32 is the integer 7986. Any literal may
separate digits with single underscores, as in 1_000_000 and 0xffff_ffff;
an underscore cannot lead, trail, repeat, or follow the prefix.
| Source | Diagnostic |
|---|---|
0x | `0x` needs at least one hexadecimal digit |
0b102 | invalid digit `2` in a binary literal |
0b1f32 | invalid digit `f` in a binary literal |
0x1u7 | invalid integer suffix |
0x_FF | invalid integer separator |
0x1FFu8 | integer literal out of range for u8 |
Annotations, parameters, returns, and typed arithmetic operands guide
unsuffixed literals within an expression. Already typed values and suffixed
literals must match exactly; no numeric widening occurs. Operands of different
numeric types are rejected with the cast that reconciles them, for example
cannot combine i64 with f64; convert with `f64(n)` . The suggested cast
never fails: an unsigned and a signed integer convert to the narrowest signed
type wider than the unsigned one, both operands if neither has that type
(i8 with u16 suggests i32 for each). u64 with a signed type has no such
type, so the message names no cast. Ordinary generic calls
have argument inference rules that do not infer
one argument’s literal width from another.
Explicit casts use numeric type names, for example u8(200). A cast to an
integer type requires the target to represent the value. An integer keeps its
mathematical value; one outside the target’s range traps, unlike the default
recoverable integer arithmetic errors. This covers narrowing, a negative value
cast to an unsigned type, and an unsigned value above a signed type’s maximum. A float is
truncated toward zero first: u8(255.9) is 255 and u8(-0.9) is 0. A float
whose truncated value is outside the target’s range, an infinity, or NaN is the
same runtime error. The message is integer cast out of range.
A cast whose operand is a numeric literal is checked at compile time, so
u8(300) and u8(300.7) are compile-time errors. A cast between integer types
where the target represents every source value, such as i64(small_u8), emits
no check. A failing check calls the runtime; a passing one does not, and no cast
allocates. A cast whose operand compiles to a constant, such as i64(u16.MAX)
or u8(i64(200)) in a function body, emits the converted constant with no
conversion or check, as the literal 65535 would. Such an operand outside the
target’s range, as in u8(i64(300)), is still the runtime error.
Casts to a float type never fail. Integer-to-float and narrowing float casts may
round, and an f64 beyond f32’s range becomes an infinity. Widening f32 to
f64 is exact.
Two other forms convert between integer types.
T.try_from(value)
returns a Result instead of stopping the program.
T.wrapping_from(value)
keeps the low bits and never fails.
All three conversions take a number, not a reference. Borrowed iteration such
as for b in &buffer yields &u8, so i64(b) is a compile-time error that
names the fix: numeric casts require a number, not a reference; read it with `*b` . A borrow written in the operand, as in i64(&n), is reported with
remove the `&` instead.
Unary minus requires a signed integer or a float; an unsigned operand is a
compile-time error, including a sign directly before a literal. Both -3u8
and -0u8 report unsigned negation, as does value: u8 = -0 when the literal
gets its type from context. Signed minimum literals such as -128i8 remain
valid, and floating-point negative zero is preserved.
Integer arithmetic failures return Err from the enclosing function by
default. Its declared Result error type must include the possible errors;
successful arithmetic still produces ordinary integers. Use an explicit
checked: block for local recovery, or
unchecked: where the programmer guarantees that the
arithmetic cannot fail.
// floors signed quotients, including negative operands; % is integer-only. / accepts
matching-width floats. Float arithmetic follows IEEE behavior for rounding,
signed zero, infinities, NaN, and underflow; arithmetic overflow may yield infinity.
Integers support bitwise &, |, ^, and unary ~, and shifts << and >>.
Operands must have the same integer type; a typed left operand guides an
unsuffixed count literal. Counts must be nonnegative and less than the integer
width; invalid counts trap before shifting. Left shifts discard high bits.
Signed right shifts extend the sign bit; unsigned right shifts fill with zero.
From lower to higher precedence: |, ^, &, shifts, addition/subtraction,
multiplication/division/modulo, unary operators. Comparisons bind below |.
Prefix & and &mut still create borrows.
value.wrapping_add(other), wrapping_sub, and wrapping_mul require matching
integer types and compute modulo two to the integer width. They allocate
nothing. Ordinary +, -, and * keep their overflow checks.
value.rotate_left(count) and value.rotate_right(count) rotate the bits of
an integer: bits leaving one end enter at the other, so none are lost. The count
has the receiver’s type, and a typed receiver guides an unsuffixed count literal.
Because no count loses bits, rotation never traps: the count is reduced modulo
the integer width, so rotating by the width is the identity and a negative
signed count rotates the other way. Signed values rotate their two’s-complement
bit pattern. Rotation allocates nothing.
These built-ins are selected for integer receivers; classes may define methods
with the same names and their own signatures.
The numeric facilities
provide the type constants T.MIN, T.MAX, T.PI, and T.E; pow on every
numeric type; and the float methods sqrt, exp, log, sin, cos,
floor, ceil, and fmod. A two-operand method requires one type, as the
operators do, and an integer result outside its type is the same overflow
error.
Comparisons require matching types; ordering accepts integers, floats, strings, and tuples of those. For scalar floats, NaN is unequal to every value, including itself, and all ordered comparisons with it are false. These rules also apply to structural equality and membership. Floats cannot be dictionary keys or set elements.
Function arithmetic contracts
def scale(value: i64, factor: i64) -> Result[i64, OverflowError[i64]]:
Ok(value * factor)
def main() -> Result[(), IoError]:
print(scale(3, 4).unwrap(), scale(i64.MAX, 2).is_err())?
Ok(())
12 True
Each named or anonymous function has its own boundary, independent of its
definition or call site. The declared error type must contain every escaping
arithmetic error; a plain or Option return cannot implicitly propagate them.
Explicit Ok, Err, returns and ? keep their ordinary meaning. Operators
do not produce individual Result wrappers or implicitly unwrap calls.
The operation/error table under checked blocks also applies to the function default. Error contracts follow operators and operand types, not range analysis. Locally handled errors do not have to appear in the function’s error type. Generators cannot propagate function-level arithmetic errors: handle a local checked result or explicitly opt out.
Coordinated repository adoption under B14 remains in progress. Public contracts are explicit: do not assume that an arbitrary callback consumer accepts a Result-returning callback.
Unchecked blocks
unchecked: states that integer arithmetic written in its lexical suite
does not overflow or divide by zero; that is the programmer’s precondition,
not the compiler’s. It yields the last expression’s ordinary type, without wrapping or unwrapping a
Result. It is also allowed as a statement suite, including an empty
pass suite and suites whose paths exit. Locals remain block scoped and
reference-valued suites are not supported.
def scale_in_bounds(value: i64, factor: i64) -> i64:
unchecked:
value * factor
The --unchecked MODE build setting selects what a plain unchecked: suite
does with its preconditions. unchecked trap: always checks them:
| Suite | --unchecked trap (default) | --unchecked assume |
|---|---|---|
unchecked: | Check; trap on a violation | Omit the checks; a violation is undefined behavior |
unchecked trap: | Check; trap on a violation | Check; trap on a violation |
A trap terminates the process; it never wraps or reaches an outer checked
boundary. In assume mode, overflow is not wrapping: a violation may produce a
silently wrong result or worse, and no crash is guaranteed. Use explicit
wrapping operations for modular arithmetic, and unchecked trap: or ordinary
validation when termination on bad input is part of the contract. The default
does not change with optimization or profile.
def checked_index(row: i64, width: i64, column: i64) -> i64:
unchecked trap:
row * width + column
The preconditions are representable results of +, -, *, negation and
pow, nonzero divisors and representable quotients for // and %, valid
shift counts, and in-range integer conversions. i64.MIN % -1 stays 0, and
a negative pow exponent still traps in either mode. Floating-point
arithmetic, allocation and explicit wrapping, saturating and checked methods
keep their contracts.
In assume mode, list, slice-view and Bytes indexing written inside a plain
unchecked: suite also requires 0 <= index < len(collection). Reads, writes,
augmented assignments and element borrows omit both bounds checks and
negative-index normalization. A negative or out-of-range index is undefined
behavior, even when the negative index would normally count from the end.
Default/trap builds retain normal negative indexing and out-of-range traps.
This does not change slices, collection methods such as swap, or indexing of
strings, immutable bytes, dictionaries, tuples or ranges.
Modes apply lexically. Nested checked: restores local recoverability; a
nested plain unchecked: follows the build setting, even inside
unchecked trap:; leaving any suite restores the outer mode. Named functions
and closures keep their own contracts wherever they are defined or called.
Calls still return their declared results, and ?, return, break and
continue keep their ordinary targets.
std always checks its unchecked: suites, so the setting cannot change
library behavior before its preconditions are audited. The setting is part of
the compile cache identity; there
is no plenty.toml key yet. Cranelift has no no-overflow instruction flags,
so assume mode removes checks but cannot give the backend further overflow
facts.
unchecked and trap are contextual: only unchecked: or unchecked trap:
followed by a newline opens a suite; unchecked: i64 = 0 remains a typed
binding. Suites support binding, assignment, return, function-tail and
match-arm-tail value positions.
Checked blocks
A checked: block turns the integer failures written inside it into a value.
Its body is an indented suite whose last expression is the block’s value T.
The block is an expression of type Result[T, E]:
def mean(total: i64, count: i64) -> Result[i64, OverflowError[i64] | ZeroDivisionError]:
checked:
total * 2 // count
If no operation fails, the block’s value is Ok(T). The first failing
operation exits the block with Err, and the rest of the block does not run.
E is the anonymous sum of the errors that the block’s
operators can produce:
| Operation inside the block | Error |
|---|---|
+, -, *, unary - and pow on T | OverflowError[T] |
// on a signed T with a T or NonZero[T] divisor (T.MIN // -1) | OverflowError[T] |
// and % with a plain integer divisor that may be zero | ZeroDivisionError |
OverflowError[T] has one field, wrapped: T, the exact result modulo
2N for an N-bit T. ZeroDivisionError has no fields. Both are
builtin classes; their names cannot be redefined. T.MIN % -1 is 0, inside
or outside a block. Shifts, casts, and float arithmetic are unchanged: they
neither add an error nor exit the block. A negative pow exponent still traps,
since it has no integer result. Float a.div_checked(b) returns
Result[F, ZeroDivisionError], failing for either signed zero divisor; it
behaves as a one-operation block and may appear anywhere.
A class pattern names each error and binds the wrapped result:
def halve_sum(a: i64, b: i64) -> i64:
result = checked:
(a + b) // 2
match result:
case Ok(half):
half
case Err(OverflowError(wrapped)):
# A positive divisor rules out zero and MIN // -1.
two: Positive[i64] = 2
wrapped // two
case Err(ZeroDivisionError()):
0
The error type follows the operators, not their operand values. n // 4
inside a block still adds ZeroDivisionError and OverflowError[i64]; the
code generator omits the branches a constant divisor rules out. A declared
Result whose error includes every possible failure is kept, so a block can
be the body of a function that returns Result[i64, OverflowError[i64] | ParseError].
The rule is lexical within one function. Compound assignment such as
x += 1, comprehensions, and operators that
@newtype classes
forward to their field report the field’s arithmetic error types. The nearest
explicit arithmetic mode controls failure routing.
Functions and closures defined inside the block keep their own declared
contracts. There is no private callback error channel or checked-block lifetime
restriction; ordinary ownership and borrow restrictions still apply. A callee’s
Result requires explicit handling, and ? exits the function rather than
being caught by the block.
Integer std.math reductions and serial pipeline
sum() return public overflow results; float reductions stay infallible.
sort_by_key accepts ordinary
fallible key callbacks and returns their error. Other callback consumers
require their own explicitly fallible contracts. Do not assume a call inherits
local checking or that an infallible callback signature
can accept recoverable arithmetic.
Control flow is otherwise unchanged. ? inside the block exits the function,
and return, break and continue behave as they do outside it. A failure
releases the values and locals the block owns, as ? does. It does not roll
back assignments the block already made to outer variables or elements, and an
assignment whose operation failed does not happen.
A block may appear wherever a match block
can produce a value: as the value of a binding or assignment, after return,
or as a function’s or case’s final expression. checked is not reserved: it
starts a block only when : and a newline follow it, so checked: i64 = 0
still declares a variable. A block whose value is discarded is rejected like
any unused Result. A block with no operation that can fail is a compile-time
error, as is one whose every path exits. A block cannot produce a reference.
Module constants are evaluated at compile time, where overflow is a
compile-time error. Arithmetic in a block is evaluated at run time even when
its operands are constants, so checked: around i8.MAX + 1i8 produces
Err(OverflowError[i8](wrapped=-128)).
Arithmetic uses plain values and a shared cold error exit at its nearest
recoverable boundary. A trapping unchecked: operation uses an inline trap
guard instead; assume mode emits the plain instruction. See the
design note.
Nonzero integers
NonZero[T], for any integer type T, holds a T that is not zero. Where
one is expected, so that the program cannot hold a zero, an integer literal
converts at compile time, including a negative one; 0 is a compile-time
error. Any other value is converted with NonZero.try_from(value), which
returns Result[NonZero[T], RangeError] and fails with
RangeError.OutOfRange for zero. T is the value’s type;
NonZero[T].try_from(value) names it and requires a T. get() reads the
integer back.
def per_group(total: i64, groups: NonZero[i64]) -> Result[i64, OverflowError[i64]]:
Ok(total // groups)
def main() -> Result[(), IoError | RangeError | OverflowError[i64]]:
three: NonZero[i64] = 3
print(per_group(12, three)?)?
print(per_group(12, -4)?)?
count = 5
five = NonZero.try_from(count)?
print(five * three, five + three, five > three)?
match NonZero.try_from(count - 5):
case Ok(n):
print(n)?
case Err(RangeError.OutOfRange):
print("zero")?
Ok(())
4
-3
15 8 True
zero
A value has the same representation as its integer and allocates nothing.
The constructor is private: NonZero(3) is a compile-time error that names
the two conversions. NonZero[T] copies, compares with ==, !=, <, <=,
>, and >=, prints and formats as its integer, and is a valid dictionary
key and set element.
Arithmetic keeps a result nonzero only where that follows from nonzero
operands. Unary - and +, and * between two NonZero[T] values, return
NonZero[T]; *= updates one. + and - between two values return T,
since their result can be zero. Overflow returns Err at the nearest
recoverable boundary, and violates the precondition of
unchecked blocks, so a product cannot wrap to zero. Other
operators between two NonZero values, including //, %, bitwise operators,
and shifts, are compile-time errors that suggest .get(). An integer literal beside a NonZero[T] operand
is a NonZero[T] too: step * 2 is a NonZero[i64], and step == 0 is a
compile-time error. Operands must otherwise have the same type; mixing
NonZero[T] with T is otherwise a type error whose hint shows try_from.
A plain T dividend accepts a NonZero[T] divisor: total // groups and
total % groups return T, and //= and %= update a T destination.
These forms omit the divide-by-zero check. Signed floor division still checks
T.MIN // -1 for overflow; T.MIN % -1 returns zero. In a checked: block,
signed division by NonZero[T] contributes only OverflowError[T]; modulo
and unsigned division contribute no failures. A block containing only
operations that cannot fail is rejected, as with other checked arithmetic.
A range step is NonZero[i64].
NonZero is a builtin name and cannot be redefined.
Positive and non-negative integers
Positive[T] holds an integer greater than zero; NonNegative[T] holds an
integer greater than or equal to zero. T must be i8, i16, i32 or
i64. For unsigned integers use NonZero[uN] or plain uN, respectively.
Floats and other non-integer types are rejected without a replacement suggestion.
Both are builtin names and cannot be redefined.
An unsuffixed integer literal converts where the domain type is expected,
including parameters, defaults, fields, returns and collection elements.
An out-of-domain literal is a compile-time error. A
module constant may have a domain type;
its value is checked the same way. A computed integer converts
with Positive.try_from(value) or NonNegative.try_from(value), returning
Result[Domain[T], RangeError]; invalid values give RangeError.OutOfRange.
The qualified forms Positive[T].try_from and NonNegative[T].try_from
require exactly T. Constructors and fields are private; .get() reads T.
There is no implicit conversion between a domain and its integer or another
domain.
def configured(value: i64) -> Result[Positive[i64], RangeError]:
Positive.try_from(value)
def main() -> Result[(), IoError | RangeError | OverflowError[i64]]:
size: Positive[i64] = 3
limit: NonNegative[i64] = 0
print(size + 2, size * 2, size - size, -size, limit)?
stride = NonZero.from(size)
print(list(range(0, 8, stride)).unwrap())?
print(NonNegative.from(configured(4)?))?
print(configured(0).is_err())?
Ok(())
5 6 0 -3 0
[0, 3, 6]
4
True
+, *, += and *= preserve the domain; subtraction and unary -
return T, and unary + preserves it. Overflow returns Err by default
and violates the precondition of unchecked blocks, so the
domain cannot be violated by wrapping. Comparisons accept the same domain type or a literal in its domain.
Other operators between domain values, including division, modulo, bitwise
operations and shifts, require .get().
A plain T dividend accepts a Positive[T] divisor directly with //, %,
//= and %=. The result is T. These forms omit both the zero check and
the signed MIN // -1 check, since a positive divisor can be neither zero
nor -1. They contribute no failures inside a checked: block. A
NonNegative[T] divisor still needs .get() or explicit conversion to a
nonzero domain, since zero is valid for it.
NonZero.from(positive) and NonNegative.from(positive) explicitly widen a
Positive[T]; they cannot fail or allocate. Qualified forms name T.
Conversion from a NonNegative to a Positive or NonZero uses
try_from(value.get()), since it may be zero.
Both domains have T’s representation, copy without allocation, print and
format as the integer, and work as dictionary keys and set elements. len()
continues to return i64; convert a computed length explicitly when a domain
is needed.
Bounded integers
Bounded[T, lo, hi] stores an integer T in the inclusive interval
lo <= value <= hi. T is any of the eight integer types. The two bounds are
value arguments of exactly that type. Bounds outside
T or a lower bound above the upper bound are compile errors, including when
an instance is used only in an alias, field or signature. Equal bounds are valid.
Use a type alias to name a domain:
type Percent = Bounded[u8, 0, 100]
def set_percent(value: Percent = 50) -> u8:
value.get()
def main() -> Result[(), IoError | RangeError]:
literal: Percent = 25
configured = 75u8
checked = Percent.try_from(configured)?
print(set_percent(literal), set_percent(100), checked.get())?
print(Percent.try_from(101).is_err())?
Ok(())
25 100 75
True
Construction and extraction
| Operation | Result |
|---|---|
Integer literal where Bounded[T, lo, hi] is expected | Checked at compile time; no allocation or Result |
Bounded[T, lo, hi].try_from(value) or an alias’s .try_from(value) | Result[Bounded[T, lo, hi], RangeError]; RangeError.OutOfRange outside the interval |
value.get() | The stored T, copied without allocation |
Expected literal contexts include annotated bindings, assignment, concrete
function parameters and returns, defaults, constructor fields and typed
collection elements. A suffixed literal must have exactly T. Plain computed
integers require try_from, even when the programmer knows they fit. A
try_from input is evaluated once and may be T or a reference to T.
There is no implicit conversion between different bounded types or from a
bounded value to its stored integer. The constructor and field are private.
def main() -> ():
percent: Bounded[u8, 0, 100] = 101
outside `Bounded[u8, 0, 100]`'s inclusive bounds
Value behavior
A bounded integer is a copyable private single-field class. It stores only T;
the bounds are type metadata and add no fields. It follows ordinary class
borrowing, equality, hashing and cleanup behavior. Equality requires matching
bounded types. Default formatting shows the class and its value; .get() gives
the ordinary integer’s formatting and ordering operations.
Arithmetic requires .get(). The compiler does not derive new intervals for
arithmetic or widen intervals automatically. Check the result again with
try_from if it must re-enter a bounded domain. Arithmetic on the extracted
integer retains ordinary overflow/error contracts.
Positive[T] and NonNegative[T] remain useful for their fixed lower bounds
and domain-preserving operations. Use Bounded when a ceiling or another lower
bound is part of the type. Existing standard-library limit signatures retain
their documented types and dynamic checks; the availability of Bounded does
not itself change them. Module constants may supply the bounds, but a module
constant cannot yet have a Bounded result type.
Byte literals and bytes
A byte literal spells fixed binary data: protocol magic numbers, framing, test
fixtures, search needles. Its type is the built-in immutable type bytes, the
byte counterpart of str.
def is_png(header: bytes) -> bool:
header == b"\x89PNG\r\n\x1a\n"
def main() -> Result[(), IoError]:
request = b"GET / HTTP/1.1\r\n"
print(len(request), request[0], request[-1])?
print(is_png(b"\x89PNG\r\n\x1a\n"))?
print(b'single "quotes"')?
Ok(())
16 71 10
True
b"single \"quotes\""
Spelling
A byte literal is b directly followed by a quoted string: b"...", b'...',
b"""...""", or b'''...'''. Triple quotes permit physical newlines, as for
strings. Each rule has one form:
- The prefix is a lowercase
b.B"..."is rejected with a diagnostic.bfollowed by anything other than a quote is an ordinary name, sobandBremain usable identifiers. bytesis a built-in type name likestr: no function, type, or import binding may take it, so writeimport std.bytes, notimport std.bytes as bytes. A local variable may still be namedbytes; the type has no associated functions, sobytes.namealways refers to that value.- The text holds only ASCII characters. Any other character is rejected at its
position with a hint to use
\xHHescapes orstd.bytes.from_text. A source character never stands for its UTF-8 encoding. - Escapes are those of string literals (
\0,\n,\r,\t,\\,\',\") plus\xHH, exactly two hexadecimal digits of either case, for any byte.\u{...}is rejected with a hint to write the scalar’s UTF-8 bytes as\xHH;\Nand every other escape are errors. - There are no other prefixes: no raw form (
rb,br) because there are no raw strings, and no formatted form (bf,fb). Build formatted text with a string and convert it withstd.bytes.from_text. - Adjacent literals are not concatenated, as for strings.
- A byte literal is not a
matchpattern; compare with==.
Values
A bytes value refers to read-only data emitted into the executable. Creating,
copying, passing, returning, storing, or comparing one never allocates, and it
has no owner to release: assignment copies the reference, like an integer. The
contents cannot change.
| Operation | Result |
|---|---|
len(value), value.len() | Number of bytes, i64 |
value[i] | The byte at i, u8; list index rules: negative counts from the end, a missing index stops the program with index out of bounds |
value[a:b:c] | A slice: Result[std.bytes.Bytes, AllocError], a new owned buffer; no import needed |
&value[a:b] | A shared slice view, &[u8], of the selected bytes; no copy |
for byte in value, reversed(value) | Each byte as u8 |
x in value | Whether the u8 x occurs |
==, != | Content equality with another bytes |
print(value) | The literal spelling, which reads back as the same bytes: printable ASCII, \\, \", \n, \r, \t, and \xhh for every other byte; see printing bytes |
bytes values can be elements, set members, dictionary keys, class fields, and
enum payloads, and can be captured and passed to scoped threads. len is the
only method; it lets bytes satisfy a protocol
that requires it. There are no ordering comparisons and no concatenation. A slice copies into an
owned Bytes rather than producing another bytes. bytes and
str are distinct: neither converts to the other implicitly.
Owned buffers
std.bytes.Bytes is the owned, growable buffer.
std.bytes.from_bytes(source: &[u8]) copies a literal, or any other byte view,
into a new Bytes and returns a Result, because the copy allocates.
Operations that only read bytes, such as find_bytes, matches, extend, and
every std.io, std.net, and std.tls stream write, take a
&[u8] view. A literal, a Bytes buffer,
a list[u8], or part of any of them converts to that view and is read in
place; nothing is copied, and there is no implicit conversion between bytes
and Bytes. Changing or growing bytes needs a Bytes.
The literal type is a separate prelude type rather than a static &Bytes view
so that the prelude does not depend on std.bytes: every program would
otherwise know the library class. The &[u8] view is what lets one parameter
accept every byte owner.
Type aliases
type Count = i32
def increment(value: Count) -> Count:
value + Count(1)
type Name = Type declares a transparent, non-generic module-level alias to a
supported type or another alias. Aliases are visible throughout the module and
may refer forward. Unknown targets, cycles, duplicate aliases, and declaration
or built-in name collisions are rejected even when the alias is unused.
Local aliases are unsupported.
An alias adds no nominal identity, validation, or runtime wrapper. Two aliases
for i32 are interchangeable with each other and i32. Existing storage and
visibility restrictions apply through aliases.
Aliases work in signatures and local annotations. Numeric aliases support their target’s cast syntax and conversion semantics. Boolean, string, and unit aliases add no constructors or conversions.
Aliases do not change literal defaults or introduce suffixes: x: Count = 42
uses the annotation as context, while 42Count is invalid. A local expression
name may shadow Count without changing its meaning in a type annotation.
Compile-time dimensions
A dimension identifies a physical dimension without introducing a runtime value. Declare a nominal base with its coherent-unit symbol, or a transparent derived dimension with an expression:
dimension Length("m")
dimension Time("s")
dimension Velocity = Length / Time
dimension Acceleration = Length / Pow[Time, 2]
@copy
class Sample[D: Dimension]:
value: f64
def reading[D: Dimension](sample: Sample[D]) -> f64:
sample.value
def main() -> Result[(), IoError]:
speed: Sample[Velocity] = Sample[Length / Time](3.0)
print(reading(speed))?
Ok(())
3.0
Sample demonstrates a dimension argument. Its field is an ordinary f64;
this class does not supply dimension-aware arithmetic or unit conversion.
For physical values and arithmetic, use the built-in
Quantity[T, D].
Declarations and identity
dimension is a keyword. Declarations belong at module scope and follow ordinary
module visibility: pub dimension Length("m") is
exported. Bases in different modules differ even when their names and symbols
match. Import aliases preserve identity. A public derived dimension or signature
cannot expose a private base dimension.
A derived declaration creates no new identity. Forward references work; cycles
are rejected. One is the built-in dimensionless dimension. Coherent-unit
symbols are retained as metadata; they do not select a conversion or change
dimension equality.
Dimensions have no values. A dimension can be an argument to a generic parameter
constrained by Dimension, but cannot be a parameter’s value type, a return
value, a field, a collection element, or an ordinary type alias target.
CopyType describes values and does not admit dimensions.
dimension Length("m")
def bad(length: Length) -> ():
pass
def main() -> ():
pass
dimension `Length` has no values
Algebra
| Expression | Meaning |
|---|---|
One | Empty product |
| Base, derived name, or dimension parameter | The named dimension |
A * B, A / B | Product or quotient, left-associative; parentheses group |
Pow[A, n] | Integer power |
Root[A, n] | Exact integer root; n must be nonzero |
n is an unsuffixed integer literal within i64 range, optionally negative.
Decimal, hexadecimal, octal, binary, and ordinary integer separators are
supported. Negative roots take the reciprocal of the corresponding positive
root. Every base exponent must divide exactly by the root degree.
Pow and Root are fallback type-operator names. A local declaration or import
with either name keeps its ordinary meaning; for example, importing std.fs.Root
as FsRoot leaves the dimension operator Root available.
The normal form is a sorted list of nominal bases with nonzero i8 exponents.
Products commute and associate when their intermediate exponents stay in range;
division cancels exactly. Each subexpression must fit -128..127: a later
cancellation does not rescue an overflowing intermediate result. Normal forms
use base names and Pow terms; aliases do not change type identity.
dimension Length("m")
dimension Invalid = Root[Pow[Length, 3], 2]
def main() -> ():
pass
not a whole dimension
Diagnostic alias notes
Type mismatches and quantity dimension errors show the normalized dimensions. When a matching derived alias is visible, a note connects that normal form to its source spelling:
dimension Length("m")
dimension Time("s")
dimension Velocity = Length / Time
def main() -> ():
speed: Quantity[f64, Velocity] = Quantity[f64, Length](1.0)
note: dimension alias `Velocity` = `Length * Pow[Time, -1]`
The first matching alias in declaration/import order is used. A namespace
import considers exported aliases in their declaration order. Notes use the
written import binding, such as Fast for from units import Velocity as Fast
or u.Velocity for import units as u. Private or unimported aliases from other
modules are excluded, as are names shadowed by generic parameters. Aliases of
One or of a single base work the same way.
The scope belongs to the expression or type that failed. A generic body error can therefore explain an alias in the defining module while the diagnostic also identifies the call that required that instance. Nested types and module constants receive the same notes. Errors encountered while resolving dimension declarations can explain aliases that have already resolved.
These notes do not change equality, specialization, coherent-unit output, or normalized quantity debug formatting.
Generic dimensions
D: Dimension accepts dimensions in functions, classes, and enums. A bare
dimension parameter in an argument type can be inferred structurally, as in
reading(sample). Explicit arguments use reading[Velocity](sample).
In a function parameter type, a dimension argument must be concrete or a bare
parameter. An expression involving a parameter, such as Sample[D / Time], is
rejected even for explicit calls. Return types and bodies may use dimension
expressions, evaluated after substitution for each instance. An invalid root
or overflowing exponent then reports the instantiation that required it.
Dimension arguments distinguish generic instances but add no fields or runtime metadata to their values. Specializations differing only in dimensions are not currently shared. See generic specialization.
Quantities
Quantity[T, D] stores a numeric value in the coherent unit of a
dimension. T must be an integer, a float, or an integer domain
such as Positive[i64]. Quantity values copy. Put domains inside the quantity:
Quantity[Positive[i64], Length], rather than Positive[Quantity[i64, Length]].
std.units provides the SI dimensions, units and
constants; the examples here declare their own dimensions to show the
underlying rules.
dimension Length("m")
dimension Time("s")
dimension Velocity = Length / Time
def main() -> Result[(), IoError]:
metre = Quantity[f64, Length](1.0)
kilometre = 1000.0 * metre
distance = 2500.0 * metre
elapsed = Quantity[f64, Time](100.0)
speed: Quantity[f64, Velocity] = distance / elapsed
print(distance.to(kilometre), speed.get())?
Ok(())
2.5 25.0
Construction and conversion
Quantity[T, D](value) accepts T, with ordinary literal inference and domain
literal validation. .get() returns the stored T. The field is private.
There is no implicit conversion to or from a plain number, including for
Quantity[T, One].
For floating storage, q.to(unit) returns (q / unit).get(). Both quantities
must have the same storage and dimension; the unit’s value determines the
scale. For integer storage, / is unavailable, so use explicit // and .get()
when integer floor conversion is intended.
Quantity[f32, D].from(q) and Quantity[f64, D].from(q) convert a quantity
of the same dimension and plain numeric storage, with ordinary numeric cast
semantics. They neither change dimensions nor infer a scale.
Quantity[Positive[T], D].try_from(n) returns
Result[Quantity[Positive[T], D], RangeError]; the same operation supports
NonNegative and NonZero. It takes an integer of the domain’s underlying
width and checks the domain exactly as its ordinary try_from does.
Operators
The storage operation follows the existing numeric rules, including
width checks, literal inference, domain restrictions, overflow and checked:.
The table’s R is the storage operation’s result type.
| Operation | Dimensions and result |
|---|---|
a + b, a - b | Equal dimensions; Quantity[R, D] |
a * b | Quantity[R, A * B] |
a / b, a // b | Quantity[R, A / B]; / requires floats, // integers |
a % b | Equal dimensions and integer storage; Quantity[R, D] |
q * k, k * q, q / k | Numeric scalar; Quantity[R, D] |
k / q | Quantity[R, One / D] |
+q, -q | Same dimension, storage result R |
==, !=, <, <=, >, >= | Equal dimensions; bool |
+=, -= | Equal dimensions and storage; result must fit the destination type |
*=, /= | Scalar only; result must fit the destination type |
A scalar cannot be added to or compared with a quantity, even when it is zero. Bitwise operators and other compound assignments are unavailable. Operands are evaluated once in source order. Compound assignments preserve ordinary borrow checks and evaluate a collection index once.
For p: Quantity[Positive[i64], Length], p + p keeps positive storage,
p * p has positive storage and dimension Pow[Length, 2], and p - p and
-p have plain i64 storage while retaining Length. A plain integer quantity
accepts a Positive or NonZero quantity divisor for // and %, with the
same proof-based checks as the underlying integers. Domain dividends retain
the underlying domain’s restrictions.
Inside checked:, failures remain OverflowError[T] and/or
ZeroDivisionError, as for the underlying arithmetic. Integer domains use
their underlying integer in OverflowError. Quantity construction and
rewrapping do not change arithmetic policy.
Module constants
Declare units and derived values with const or pub const. Use a concrete
Quantity[T, D] annotation; dimension aliases may appear in D. Initializers
can use the Quantity constructor, other constants (including imports and forward
references), numeric casts of extracted values, quantity operators, .get(),
and floating .to(unit). The compiler evaluates them before generating code.
dimension Length("m")
dimension Time("s")
const metre: Quantity[f64, Length] = Quantity[f64, Length](1.0)
const kilometre: Quantity[f64, Length] = 1000.0 * metre
const minute: Quantity[f64, Time] = 60.0 * Quantity[f64, Time](1.0)
const speed: Quantity[f64, Length / Time] = kilometre / minute
assert kilometre.to(metre) == 1000.0
def main() -> Result[(), IoError | AllocError]:
print(kilometre)?
print(f"{speed:.2f}"?)?
Ok(())
1000.0 m
16.67 m/s
Storage widths, dimensions and integer-domain results follow ordinary quantity
arithmetic. A domain constructor accepts a valid literal or a value already in
that domain. p - p still erases the storage domain, so a positive quantity
cannot be declared from that subtraction. .get() on a domain-backed quantity
returns the domain; call .get().get() to extract its plain integer.
Integer overflow and division by zero are compile errors. Conditional and
short-circuit expressions check both branches but evaluate only the selected
branch. A dimension mismatch in a skipped branch is still an error. Constant
cycles are rejected. User function calls, fallible conversions and formatting
are not constant expressions; for a float storage conversion, write
Quantity[f32, D](f32(q.get())). See
module constants for the common rules.
Formatting
print(q) and f"{q}" show the coherent value followed by a space and the
coherent-unit symbol: 1.3 m/s. Numeric format specs apply to the number,
including its padding; f"{q:08.2f}" gives 00001.30 m/s. Integer domains
format as their underlying integers. Dimensionless quantities have no suffix.
Products use *, powers use ^, and multiple denominator factors are
parenthesized: kg*m/s^2 or m/(kg*s). Symbols appear in the order their
base dimensions are declared within a module, with modules ordered by name.
std.units declares its bases in SI order, so a
newton prints as kg*m/s^2. The order is presentation only: distinct
dimensions with equal symbols remain distinct, and their symbols do not
cancel.
dimension Mass("kg")
dimension Length("m")
dimension Time("s")
def main() -> Result[(), IoError | AllocError]:
speed = Quantity[f64, Length / Time](1.3)
print(speed)?
print(f"{speed:.2f}"?)?
print(f"{speed!r}"?)?
momentum = Quantity[f64, Mass](2.0) * speed
print(momentum)?
print(f"{momentum!r}"?)?
Ok(())
1.3 m/s
1.30 m/s
Quantity[f64, Length * Pow[Time, -1]](1.3)
2.6 kg*m/s
Quantity[f64, Length * Mass * Pow[Time, -1]](2.6)
str.repr(q), !r and quantities inside aggregates show
Quantity[T, D](value), with the normalized dimension expression. That
expression orders bases by qualified name, as dimension identity does, so it
can differ from the symbol order. !a escapes
non-ASCII characters in that debug spelling. !s converts the display spelling
to text first, so f"{q!s:>12}" pads the whole number-and-unit text.
Quantities support display(out) and debug(out) with a mutable
std.fmt.Formatter, including through std.fmt.display,
debug, to_string and to_debug. Direct writer formatting and numeric
f-string fields write the number and static symbols without allocating a
quantity-specific string; ordinary formatter limits and writer errors apply.
String-producing operations still allocate. Printing retains IoError, and
str.repr retains AllocError; quantities do not add user formatting errors.
Nested format specs and c retain their ordinary FormatError possibility.
Representation and current limits
Quantity[T, D] uses T’s native value and parameter/result representation.
Dimensions add no runtime field, tag, or arithmetic operation. Construction and
get retag the coherent numeric value without a constructor call; float to
lowers to ordinary storage division. Integer-domain storage keeps the domain’s
existing representation and arithmetic contracts. Ordinary classes keep their
existing ABI.
Collections, aggregates, references, closures and generators use the same storage packing as their numeric payloads. Arithmetic and conversion do not allocate. The release instruction probe compares addition, multiplication and unit conversion with plain-number code. This does not promise that every surrounding function call or temporary is optimized away. Generic instances that differ only in dimension share one body unless they format a value or build a collection, whose type descriptor names the dimension.
Unit spellings live in static type metadata and compiled formatting calls, not
in each value. A single quantity carries no type pointer: the compiler knows
its dimension at every use and embeds the symbol in the formatting code. A
list already holds one pointer to its type descriptor, whatever its element
type, because the runtime’s shared printing, sorting, hashing and cleanup code
reads the element type through it. A list[Quantity[f64, D]] stores plain
8-byte f64 rows; only its descriptor, and through it the unit symbol, differs
from a list[f64]. Descriptors are static read-only data, so neither lists nor
values allocate type information. std.units.wrap_list and
strip_list therefore convert between the two
list types by rewriting one header word, leaving every element untouched. See
the remaining units work.
Type and dimension errors include scoped alias notes, while debug formatting keeps normalized dimensions.
Tuples
(a, b) creates a tuple; (value,) creates a one-component tuple. Parentheses
without a comma group an expression, and empty () is unit. Annotations accept
tuple[A, B] or (A, B). A one-component type requires (A,) or tuple[A];
(A) groups the type A, just as parentheses group an expression.
Tuples are structural products with fixed component types. Construction allocates nothing; component expressions retain their own allocation and error contracts. Tuples containing owned components move, while copyable tuples copy their components. Resolved owned closures can be stored; references and generators cannot.
A mut tuple binding permits mutable component borrows regardless of whether
its components are copyable. A copied tuple has independent inline storage;
changing the original does not change the copy. Components can be changed
through &mut pair[0] or a mutable payload match. Direct indexed assignment
(pair[0] = value) remains unsupported; assign through the reference instead.
pair[0] selects a component. Indices must be integer literals. A negative
literal counts from the end, as in Python: for a tuple of length n, pair[-k]
is pair[n - k], and pair[-0] is pair[0]. The same resolved component is
borrowed by &pair[-1] and &mut pair[-1]. An index outside -n to n - 1
is a compile error that states the tuple’s length. Owned components may be observed or explicitly copied; moving
them out requires whole-tuple unpacking:
number, text = (42, "cm")
number, text = 42, "cm"
Recursive unpacking transfers each selected component; _ destroys a discarded component.
The parentheses are optional on the right of an unpacking statement, and targets
may also be fields and collection elements: see
assignment to several targets.
Unpacking a tuple reference borrows the components in place instead. With
p: &(i64, str), n, s = p binds n = &p[0] and s = &p[1]; with a
&mut reference, or n, s = &mut pair, each name is a &mut reference to its
component, and _ skips one. The names are new immutable bindings: they cannot
reuse existing names or be declared mut. for n, s in &pairs unpacks each
element the same way, as described under
iterable values.
def bump(p: &mut (i64, str)) -> Result[(), OverflowError[i64]]:
n, _ = p
*n += 1
Ok(())
def main() -> Result[(), AllocError | IoError | OverflowError[i64]]:
mut pair = (1, "one")
bump(&mut pair)?
n, s = &pair
print(*n, s)?
Ok(())
2 one
A slice with integer-literal bounds and step
selects components at compile time: (1, "two", 3.0)[1:] has type
(str, f64), and pair[::-1] reverses a pair. It allocates nothing, so it
returns no Result and takes no ?. Slices of affine components are rejected
for now.
Nested tuple targets mirror the value’s shape: ((x, y), label) = value.
Parentheses without a comma group a target; (x,) requires one component.
The same shapes work for owned and borrowed loop elements, comprehension
bindings and generator-expression bindings. Borrowing a nested
tuple keeps component-precise loans, so sibling components remain independent.
Starred unpacking is unsupported. Unit components can be discarded
or used as unit expressions without enabling standalone unit storage.
An owned list also supports unpacking with an exact runtime length check at
each list position in a nested target; tuple positions use their statically known
arity. See collection rules.
Tuples of integers, bool, strings, and nested tuples of those types are
copyable dictionary keys and set members. Hashing and equality use their
contents, with no formatting or string-key encoding.
Tuples whose components are integers, floats, strings, or recursively orderable
tuples support <, <=, >, and >=. Both operands must have exactly the same
type. They compare lexicographically, using the first differing component, with
the same allocation-free total order as list.sort(). This also works through
references and in generic functions; comparison observes both operands.
Booleans, unit, classes, ordinary enums, and collections are not orderable fields.
Floating-point tuple fields use the sorting order: negative zero precedes positive
zero, and all NaNs compare equal in order and follow every number. Structural ==
and != still use IEEE equality. Consequently, (-0.0,) == (0.0,) and
(-0.0,) < (0.0,) are both true; a NaN tuple is <= itself but not == itself.
Scalar float operators keep their IEEE behavior.
The difference is deliberate. Tuple ordering exists to sort and rank records,
so it must be total: every pair of tuples is ordered, and < on tuples agrees
with sort(). IEEE ordering would make a tuple containing NaN unordered
against every other tuple. Plenty has one total order rather than separate
partial and total orders; the
ordering proposal
gives the rationale.
Distinct components of a named tuple can be borrowed independently. Reads of one component can continue while another is exclusively borrowed; whole-tuple access overlaps all components. See borrowing for projection and collection-element loan rules.
Anonymous sums
A | B holds one owned value of either whole type. It works in ordinary
bindings, fields, arguments and returns, including Result[T, A | B] errors.
The compiler stores a tag and the largest active payload inline; wrapping and
widening allocate nothing. Payload construction retains its own costs.
Aliases, order, duplicates and nested sums normalize: A | B, B | A and
(A | B) | A are identical; A | A is just A. Distinct nominal classes and
enums remain distinct, and enum variants never become separate members.
There is no stable tag numbering or foreign layout contract.
Structural formatting (print, str.repr, and std.fmt.represent) shows the
active member directly, without an anonymous wrapper. For example, a
Result[(), IoError | ThreadError] error can be represented as
Result[(), IoError | ThreadError].Err(ThreadError.System(11)). Type names use
the normalized A | B spelling. The structural fallback for an error returned
by main follows the same rule; named enums keep their usual qualified variants.
A closure signature such as OnceClosure[[], i64] names no concrete
environment, so it is a member only in a function’s declared return type,
directly or inside a member such as SpawnError[OnceClosure[[], i64]]. The
body’s returned or propagated closure resolves it; see
closure return inference.
| binds less tightly than references and type applications. Use &(A | B)
to borrow a sum. &A | B attempts to store a reference as a member and is
rejected, as are other members forbidden by owned storage rules. Callable
brackets delimit their types: Callable[[A | B], C | D]. Parentheses without
a comma group a type; (A,) is a singleton tuple and () is unit.
An expected type permits lossless injection of a member or widening to a
superset. This applies to annotated bindings, assignments, arguments,
constructors and returns. Result[T, E] can widen to Result[T, F] when
F contains every member of E; its success type must remain identical.
Other generic containers and references do not gain covariance. Conversions
evaluate their source once, move owned payloads and invoke no user conversion.
Public return types and binding types remain declared, never silently widened.
A constructor, tuple or collection literal takes its context from the one
member it can build, so [1.0, 2]? passed as list[f64] | Unset checks its
elements as f64, and []? or {}? needs no further annotation. Several
viable members, such as list[f64] | list[i64], are ambiguous and rejected;
bind the value with a member type first.
Match a whole member with case name: MemberType:. An ordinary wildcard
matches any remaining value. A member annotation names a single normalized
type; pattern | means alternatives, not a type union. Typed member patterns
are supported at the case position, not inside other patterns.
An enum variant pattern also selects its member directly, including inside
Err or another payload. The compiler tests the member before testing its
variant, so equal variant names in different enums remain distinct:
def parse(text: &str) -> Result[i64, AllocError | ParseError]:
Ok(i64.parse(text)?)
def main() -> Result[(), IoError]:
match parse("x"):
case Ok(value):
print(value)?
case Err(ParseError.Invalid):
print("not a number")?
case Err(ParseError.OutOfRange):
print("outside i64 range")?
case Err(_):
print("out of memory")?
Ok(())
not a number
Every member and each enum member’s variants must be covered, whether by variant patterns, a whole-member pattern, or a wildcard. Duplicate and unreachable cases are rejected; covering one variant does not cover its siblings. Payload literals still need a fallback for the other payload values.
The matched value supplies generic arguments, as in ordinary enum patterns.
If a qualifier names several members, such as Choice[i64] | Choice[str],
Choice.Value(value) is ambiguous even when only one payload type would fit
its subpattern. Use a whole-member pattern with an explicit type and match
that value’s variants inside the arm. Case guards remain unsupported.
An owned match moves noncopyable payloads. Matching &value or &mut value
borrows the active member with the same permission and lifetime. Only the
active payload is initialized or destroyed. Copyability and thread transfer
require every alternative to qualify. If generic substitution collapses
T | A to A, case value: A: binds that ordinary value; two equivalent
member arms are still duplicates. Generic inference rejects ambiguous member
substitutions; supply explicit type arguments in that case.
For reporting, a sum satisfies
std.fmt.Formattable when every normalized member
satisfies that protocol. Formatting dispatch borrows the active member without
allocation or consuming the sum. A missing or mismatched method on any member
rejects the whole bound, regardless of which member is active.
This version does not infer residual sums after recovery, introduce empty
sums, forward arbitrary member methods or protocols, or erase errors. See
propagation for ? and builtin
Failure, the last-resort error that erases its cause.
The anonymous-sum proposal records the design rationale and migration boundary. The composition overview distinguishes sums from deliberate error mapping and typed stream APIs.
Collection keys
A sum satisfies KeyType when every member is a supported key type.
Use set[i64 | str] or dict[i64 | str, V] directly. Hashing and equality
include the active member type and its structural value: 1, 1i8, True
and "1" stay distinct in a sum that admits them. Normalized aliases and
member order do not change identity. Keys with an unsupported member are
rejected even if that member is never inserted.
Insertion and lookup use the ordinary expected-type injection rules. Observed member values can be injected for lookup without moving them; their source remains borrowed through the operation. This does not make reference types covariant. Sums containing move-only classes keep their ownership rules, and stored keys are only exposed through shared borrows. Anonymous sums use the hash-table representation, including sums with finite members.
Classes and construction
A class declares typed fields and associated methods. It has no inheritance, dynamic attributes, class variables, properties, or runtime method lookup.
class Point:
x: i64
y: i64
def squared_length(self) -> Result[i64, OverflowError[i64]]:
Ok(self.x * self.x + self.y * self.y)
Without __init__, the generated constructor takes fields in declaration order.
Point(3, 4) and Point.new(3, 4) return Point directly. Construction itself
allocates nothing; arguments and initializer operations retain their own error
contracts. new is a reserved class member.
A class name without arguments is a function value
with the constructor’s positional signature: make = Point lets
make(3, 4) construct a point. A generic class needs an explicit instance,
such as Cell[i64], or a concrete expected Callable result naming the class
instance. Constructor values require all positional arguments; parameter names
and defaults belong to the statically resolved constructor call. Taking the
value allocates nothing and preserves constructor visibility and fallibility.
An explicit __init__ overrides the generated constructor and returns either
() or Result[(), E]. The fallible form makes both constructor spellings
and constructor function values return precisely Result[Class, E]. The declared
error type is independent of the caller’s arithmetic mode; successful fallible
initializers write explicit Ok(()). Non-unit Result success types are rejected.
Storage itself allocates nothing, so construction does not add AllocError.
Initializer operations and parameter defaults must declare or explicitly map
their actual errors into E, including allocation failures when applicable.
__del__ and __exit__ still require (); cleanup cannot propagate failure.
Class parameters in an initializer output, including a parameter naming the
whole Result, are substituted before validating that output. Unit class type
arguments remain unsupported; a concrete unit alias can still name an
initializer output in an otherwise valid generic class.
Fields may contain classes, enums, tuples, collections, and resolved owned
closures, but not references, generators, or unit.
Every successful initializer exit must initialize every field. Branches merge
their definitely initialized fields; a loop alone cannot establish initialization
because it may run zero times. Field assignments performed inside an if or
match block used as a binding or assignment’s value do not establish
initialization after that value block. Assign the selected value directly to
self.field instead; the outer assignment initializes that field. See
conditional values.
Initialized fields may be read. Borrowing or
passing the whole partially initialized instance is rejected, and initialization
cannot be delegated to another method. Constructors accept named arguments,
using field names for generated constructors and parameter names for explicit
initializers. Explicit initializer parameter defaults are inherited by both
constructor spellings. See call rules.
Field defaults, constructor overloading, and static methods are unsupported.
Failed initialization destroys initialized fields and remaining owned arguments and skips the class’s destructor. The destructor becomes active only after successful initialization; see cleanup.
Classes can define arithmetic operator hooks
such as __add__ and __neg__, compiled as direct method calls. Any other
__name__ method besides __init__, __del__, __enter__ and __exit__ is
rejected, so Python
special methods such as __str__ and __len__ cannot be silently ignored.
Class identity is nominal. Structural equality compares fields without calling
user methods. Printing uses Point(x=3, y=4) unless the class defines
display or debug. Comparison and hashing
hooks such as __eq__, __lt__ and __hash__ are rejected; see
equality, ordering and hashing.
Automatic equality and formatting reject recursive data. Instance moves, field access, and explicit
duplication follow ownership; receiver signatures
follow methods, and constructor access follows
visibility.
A class pattern such as
case Point(x, y=0): takes a class apart in match, written like the
generated constructor with every field named once.
One-field classes
A class with exactly one field and no __del__ is represented as that field.
Wrapping a value in such a class, reading its field, or moving the field out
does not copy or allocate, so a wrapper such as class Handle: _file: File
costs the same as the bare File. The field’s type is unrestricted: a scalar,
str, a collection, an enum, or another class. Nothing in the source changes:
the class keeps its name, methods, equality, and printing
(Handle(_file=...)), and the field keeps its
visibility. A one-field class with __del__ keeps a separate
record because its cleanup state needs storage. See
value representations.
A field whose type is another class stores that class’s fields inside the
owner, at a fixed position. Reading path.start.x costs the same as reading a
field of path itself, and moving or returning path copies its bytes once,
with no per-field address work. This is not visible in source: nested fields
still move, borrow, and run __del__ as other fields do.
Copy classes
@copy on the line before class declares a copy class. Its instances copy on
ordinary value use, as copyable tuples and enums do: assignment, owned argument
passing, return, collection insertion, capture, and by-value receivers
(self: Point) all leave the source usable. Copying duplicates the inline
fields and never allocates; string fields share immutable storage. copy(value)
still accepts the instance. Other classes remain affine.
@copy
class Point:
x: i64
y: i64
def main() -> Result[(), IoError]:
p = Point(1, 2)
mut q = p
q.x = 5
print(p, q)?
Ok(())
Point(x=1, y=2) Point(x=5, y=2)
The declaration is checked as a whole:
- Every field must copy: a scalar,
str, another@copyclass, or a tuple or enum whose components copy. The error names the field and its type. - The class cannot define
__del__, because each copy would run it.
A generic @copy class copies when its type arguments copy, as a tuple does.
Pair[i64] and Pair[str] copy; Pair[list[i64]] is accepted and moves. The
field rule is checked at the definition, whether or not the class is used: a
field must copy whenever every type argument copies. T, (T, U), Option[T]
and Pair[T] qualify; a field such as items: list[T] or b: Box[T] moves
for every argument and is reported at its declaration. A
CopyType bound (or IntType or
FloatType) restricts the class to instances that copy:
@copy
class Pair[T]:
left: T
right: T
def main() -> Result[(), AllocError | IoError]:
numbers = Pair(1, 2)
copied = numbers
lists = Pair([1]?, [2]?)
moved = lists
print(numbers, copied, moved)?
Ok(())
Pair[i64](left=1, right=2) Pair[i64](left=1, right=2) Pair[list[i64]](left=[1], right=[2])
Equality, printing, field access, and matching are unchanged. Classes are
collection keys when every field
is a supported key type. Custom __del__ cleanup is allowed, including in
nested fields; such keys cannot be copied for set algebra or key-view to_set().
Hashing reads their
fields structurally, just as equality does. Ordinary instances move into the
collection; @copy instances copy. __hash__ and __eq__ hooks are unsupported.
Annotations
An annotation is @name alone on a line directly before a declaration. The
set is closed and compiler-known; there are no user decorators. The
annotations are @copy, @newtype, and @redacted. @copy and @newtype
apply only to class declarations, including pub class; @redacted also
applies to enums. An unknown name, arguments, a repeated annotation, or an
annotation before any other declaration, member, or statement is an error.
@newtype makes a class forward its field’s operators; see
newtype forwarding. The
class must declare exactly one field and cannot define __init__. It can
have type parameters (@newtype class Wrapper[T]: value: T) and __del__.
@copy and @newtype are independent. @newtype does not imply @copy: a
@newtype class over i64 still moves on assignment unless it is also
@copy. Forwarded operators borrow their operands, so a + a and reusing a
after a + b work either way. A class can carry both annotations, in either
order, with the same meaning:
@newtype
@copy
class Seconds:
value: f64
def main() -> Result[(), IoError]:
a = Seconds(1.5)
b = a
print(a + b)?
Ok(())
Seconds(value=3.0)
Neither annotation changes layout or printing. Every class with one field and
no __del__ uses the one-field representation, with or
without @newtype, and a @newtype value prints as its class
(Seconds(value=3.0)). Annotation names follow
@ and do not reserve words: copy remains an ordinary name, and the builtin
copy(value) is unaffected.
Redacted types
@redacted on a class or enum hides its contents from every automatic
formatting path: print, str.repr, f-string fields, and the report main
writes for a returned error. The value renders as its type name followed by
(@redacted), at any nesting depth: in a list, dictionary, tuple, Option,
Result, enum payload, or another class’s field.
@redacted
class ApiKey:
secret: str
class Client:
host: str
key: ApiKey
def main() -> Result[(), AllocError | IoError]:
client = Client("example.com", ApiKey("hunter2"))
print(client)?
print([ApiKey("other")]?)?
print(client.key.secret.byte_len())?
Ok(())
Client(host="example.com", key=ApiKey(@redacted))
[ApiKey(@redacted)]
7
The rules:
- Redaction applies to the whole value. There is no per-field form; a class
that mixes secret and public data either is redacted entirely or keeps its
secret in a separate redacted field type, as
Clientdoes above. - The name includes type arguments and module qualification as ordinary
printing would: a generic
Secret[T]printsSecret[i64](@redacted). - A generic wrapper over a redacted type is not itself redacted;
Holder[ApiKey]printsHolder[ApiKey](item=ApiKey(@redacted)). A@newtypeover a redacted type likewise prints its own name with the redacted field. - In an f-string,
!r,!s,!a,=, and every format spec give the sameApiKey(@redacted); there is no conversion that reveals the contents. Padding and alignment apply to the marker. - Explicit formatting is unaffected: a
displayordebugmethod the type defines is used byprint,str.reprand f-strings as for any other type, and astd.reportFormattableimplementation still writes what it chooses. - Field access, equality, hashing, ordering, matching, moves, and layout are
unchanged.
@redactedcombines with@copyand@newtypein any order.
Redaction is a guard against accidental logging, not a security boundary: code that can read a field can still print it.
Enums
enum Reading:
Missing
Value(i64)
Invalid(str)
Enums are nominal module-level types. Variants have fixed positional payloads;
nullary declarations omit parentheses. Reading.Value(7) and
Reading.Value.new(7) return Reading directly. Nullary variants use
Reading.Missing or Reading.Missing.new().
Construction allocates nothing. Payload expressions evaluate first and transfer
their values into the enum; their own allocation and error contracts still apply.
Payloads can include primitive values, classes, collections, tuples, other enums,
unit, and resolved owned closures. An enum copies when every payload of every
variant copies, and moves otherwise; @copy does not apply to enums. See
the copy rule. References and generator-containing values
cannot be user enum payloads. Standard Option/Result wrappers can contain
generators, including through nested standard wrappers. A function result may
carry a reference in the Some or Ok payload; such a value is held only in a
local and never stored, as described in
optional and fallible references.
Constructors can qualify variants with an enum name, transparent alias, or explicit generic instance; patterns use the enum name or an alias and never type arguments. Public enums expose every variant.
A payload variant named without arguments is a
function value: Reading.Value has
type Callable[[i64], Reading], and a variant with two payloads takes two
arguments. Pass it where a callable is expected, for example
i64.parse(text).map_error(ConfigError.InvalidCount) or
iter(range(3)).map(Reading.Value). A generic enum needs an explicit instance,
as in Wrapped[i64].One, or a concrete expected Callable whose result names
that instance. This includes Option[i64].Some, Result[i64, str].Ok, and
Result[i64, str].Err. The prelude Some, Ok, and Err specialize from an
expected callable signature, for example
wrap: Callable[[i64], Option[i64]] = Some. Nullary variants remain values of
the enum, not functions. A constructor value cannot have unit or generator
payload parameters; use a closure for those signatures.
Enums with permitted element types can be list elements or dictionary values;
they can also be dictionary keys or set elements when every payload is a
supported key type. Hashing includes
the variant tag and payloads. Recursive enums remain excluded. Anonymous sums qualify when every member is a supported key type.
Equality compares nominal identity, variant, and payload contents, including IEEE
float comparisons; it never calls user methods, and comparison and hashing
hooks are rejected.
Printing uses qualified variant names unless the enum defines
display or debug. Automatic equality and formatting reject
recursive data types.
Matching selects variants and binds payloads. Standard Option and Result add prelude constructors and contextual inference. Generic enums follow generic data.
Methods
An enum can declare ordinary methods alongside its variants:
enum Reading:
Missing
Value(i64)
def number(self) -> i64:
match self:
case Reading.Missing:
0
case Reading.Value(value):
*value
Bare self borrows the enum; self: &mut Reading takes an exclusive borrow.
Matching that receiver borrows the active payload under the ordinary matching
rules. Methods may have their own generic parameters and use named arguments
and defaults under the ordinary method rules. Generic enum parameters come from
the receiver, as described in generic data.
Methods are private to their defining module unless declared pub def.
Enum methods follow the class rule for
reserved names: a
__name__ method must be an operator hook, since enums have no
lifecycle or context hooks.
Variant and method names must be distinct. Class lifecycle hooks are not enum
methods: variant construction and automatic payload destruction retain their
existing behavior. Anonymous sums cannot declare methods. They forward only the
std.fmt.Formattable contract when every member
satisfies it; arbitrary member methods and protocols are not forwarded.
Enums can satisfy structural protocols with
accessible matching methods. They can also supply display and debug methods
for std.fmt; builtin printing and entrypoint reports
continue to use structural rendering.
Recursive storage
Classes and user enums may refer to themselves or one another, including through
Option, Result, tuples, and transparent aliases. Every storage cycle must pass
through a heap boundary: Box[T], a list, a set, or a dictionary. A cycle without
one would require an infinite-sized value and is rejected.
class Node[T]:
value: T
next: Option[Box[Node[T]]]
Finite concrete generic instances can recur, including mutual instances with
different arguments. Alias-only cycles remain invalid. Endless type expansion,
such as a Node[T] containing a Node[list[T]], is rejected when specialized;
the diagnostic names the first three instances and points at the type
application that grows. Finite nesting deeper than 64 levels reports that limit.
These limits concern type descriptions rather than the number of nodes in a
runtime chain or tree.
Recursive values and aggregates containing them move. Automatic copy, equality
(including membership), and formatting reject types whose stored values can recur,
even when a particular value is shallow. Callable signatures do not constitute
stored recursive values. Consuming matching transfers payloads;
borrowed matching preserves the owner. See ownership.
Moving a field
through a Box, stored references, multiple mutable owners of one node, and
source ownership cycles are unsupported. A terminating variant,
Nothing, or an empty collection supplies a construction base case. A list can
own all nodes while integer IDs represent parent links, shared targets, or
cycles: such graph edges do not carry ownership or create loans. Index stability,
stale IDs, and graph consistency remain the application’s responsibility. See
representation choices.
Shared[T] nodes may form immutable
trees and DAGs, but not cycles: a Shared[T] whose content can reach it again
through a Mutex or channel is rejected, so back-references use these IDs.
Automatic destruction of recursive owners uses bounded stack space and allocates
nothing. Explicit recursion in user methods and destructor hooks still uses the
native stack. Box construction, content conversion, projections, and borrowing
are described under facilities.
The recursive representation proposal records the storage and destruction rationale. The owned-structure assessment compares linked lists, trees, and checked-index arenas using these facilities.
Expressions and evaluation
Arguments and ordinary binary operands evaluate once, from left to right.
and and or short-circuit; conditions and Boolean operators require bool
rather than truthiness. Conditional expressions evaluate only the selected
branch. Assignment evaluates its value before destination
indices.
Operator precedence, from highest to lowest:
| Group | Operators or forms |
|---|---|
| Postfix | Calls, member access, indexing, slicing, error propagation ? |
| Unary | +, -, bitwise not ~, dereference *, borrowing & and &mut |
| Multiplication | *, /, //, % |
| Addition | +, - |
| Shift | <<, >> |
| Bitwise and | & |
| Bitwise exclusive or | ^ |
| Bitwise or | | |
| Comparison and membership | ==, !=, <, >, <=, >=, in, not in |
| Boolean negation | not |
| Boolean conjunction | and |
| Boolean disjunction | or |
| Conditional | value_if_true if condition else value_if_false |
Arithmetic operators associate left to right. Parentheses override precedence.
Chained comparisons are rejected. There is no exponentiation operator: two
adjacent stars, as in a ** b, are rejected with a diagnostic that names
base.pow(exponent), whatever
the operand types. Multiplying by a dereference needs a space: a * *b. Numeric rules specify
type matching, casts, overflow, and float behavior. Membership follows the
container’s facilities contract.
Ordering operators accept matching integers, floats, strings, and
orderable tuples. Tuple operators use the same
lexicographic total order as sort(), including its signed-zero and NaN rules;
scalar float comparisons retain IEEE behavior.
Format strings build text from fields with static specs.
Suites and conditionals have value and exit rules. Enums support
exhaustive matching; loops provide repeated execution.
Postfix ? follows error propagation, and resource scopes
follow cleanup.
Nesting
Blocks and expressions nest to at most 64 levels in total. A function body is level 1. A suite is one level below the statement that owns it, and so are that statement’s expressions. An operand, argument, index, or element is one level below the expression that holds it, and parentheses add a level around what they enclose.
Operators group to the left, so a + b + c is (a + b) + c: every binary
operator, call, member access, index, or ? in a chain puts everything before
it one level deeper. A statement in a function body can chain 62 of them. Each
manager of a with statement is one
level around the managers after it and the body, and each for or if clause
of a comprehension is one level around the clauses after it and the element.
The elif arms of an if are not nested.
Deeper source is rejected at the token that reaches level 65, with
block and expression nesting exceeds the implementation limit of 64. Bind an
intermediate value to a name, group a long chain with parentheses, or move
inner blocks into a function. Types and
patterns have their own limits of 64.
Binding and assignment
name = expression first declares an immutable local with inferred type.
name: Type = expression adds an annotation. mut name = expression or
mut name: Type = expression declares a mutable local.
The value of a binding or a single-target assignment is an expression, an
if block, or a
match block. The selected suite
supplies a block’s value.
Later name = value assigns to an existing local and requires mut and the same
type. Repeating mut or an annotation for that name is a duplicate declaration.
Parameters are immutable, and declarations always have an initializer. Copies,
moves, and exclusive access follow ownership.
Field and indexed assignment accept nested writable paths such as
rows[0][-1] = 9 and records[0].values[1] = 7. The root must be mutable or an
exclusive reference. Existing list slots change in place, without copying or
allocating their containing owners. Dictionaries have no indexed assignment, so
an assignment path cannot write through dictionary[key]; use insert(key, value)?
to add or replace, or get_mut
to change a value in place.
Assignment proceeds in this order:
- Evaluate the right-hand value.
- Evaluate destination indices once each, from root to leaf.
- Resolve the destination using the resulting contents, checking bounds or keys.
- Replace the destination while holding exclusive access.
Index expressions can read or resize the destination. Their values are saved before any address is resolved, and no user expression runs between address resolution and the write. Replacing an owned value destroys its previous value exactly once. If an index propagates an error, a pending right-hand owner is cleaned up and no write occurs.
Whole-instance assignment through *reference = instance is rejected for class
references, including class-valued field references. Replace the owning binding
or a named field instead.
Augmented assignment
target += value, -=, *=, /=, //=, %= and integer bitwise updates
&=, |=, ^=, <<=, >>= update an existing numeric
destination. The destination can be a mutable binding, field, collection element,
or exclusive reference, including nested paths. The right side takes the
destination’s numeric type. Integer division uses //=, float division uses
/=, and %= currently accepts integers. Strings and collections do not
support augmented assignment; string concatenation remains fallible and explicit.
Inside a checked: block, an integer
update that overflows or divides by zero exits the block with Err before
writing back, so the destination keeps its previous value.
Unlike ordinary assignment, augmented assignment evaluates its target before the right side:
- Evaluate each target index once, from root to leaf, saving its value.
- Read the destination’s current scalar value.
- Evaluate the right side and apply the arithmetic operation.
- Write back to the same destination using the saved indices.
A shared loan protects the destination during right-side evaluation. The right
side may observe it (count += count), but cannot move, replace, or resize its
owner. Collection loans cover the whole collection unless distinct literal
list indices separate them. Index
expressions may mutate the owner before the destination is read. If an index or
the right side propagates an error, the update performs no write. Invalid list
indices trap as in ordinary indexed assignment; dictionary elements are not
destinations. The update itself
allocates nothing, introduces no binding, and cannot initialize an unread
constructor field.
User-defined classes and enums can provide explicit mutable in-place hooks; see operator overloading. Those calls use the method’s implementation rather than the scalar read/compute/write sequence.
Several targets
target, target = value, value assigns to two or more targets at once:
def main() -> Result[(), AllocError | IoError]:
mut items = [10, 20, 30]?
items[0], items[2] = items[2], items[0]
print(items)?
mut low = 1
mut high = 2
low, high = high, low
print((low, high))?
Ok(())
[30, 20, 10]
(2, 1)
The right side is one tuple. With several targets its parentheses are optional,
so a, b = b, a means a, b = (b, a); a single target still requires them. Its
component count must equal the target count.
- Evaluate the whole right side.
- Assign each component to its target, from left to right, exactly as a single assignment to that target would.
Because the right side is complete before the first write, targets can exchange
values. Each target evaluates its own indices when its turn comes, so an earlier
target can change a later target’s index: i, items[i] = 2, 5 writes items[2].
A target is a name, a field, a collection element, *reference, or _, which
destroys its component. A name follows the single-assignment rule: it assigns an
existing mut local or declares a new immutable one. mut a, b = value
declares mutable locals and accepts names or nested tuples of names only.
Tuple targets nest recursively: (a, (b, c)) = value requires the same tuple
shape. mut (a, b), c = ((1, 2), 3) and mut ((a, b), c) = ((1, 2), 3)
declare the same nested components; the first tuple remains one target when an
outer comma follows it. A singleton (a,) still requires one tuple component.
Leaves are assigned left to right after the whole right side is evaluated;
field, index and dereference leaves follow the same rules as flat targets.
Naming the same local twice, starred targets, and a closure capture as a name
target are rejected. In a constructor, a self.field target initializes its field, and
later targets may read it.
The right side obeys ordinary ownership, so reading an owned element such as a
nested list still cannot move it out of its collection. Exchange owned list
elements with swap. When a target
traps on a missing index or key, earlier targets have already been assigned.
Nothing is allocated.
Replacing a field
replace(owner.field, replacement) follows the same evaluation order but returns
the previous field value to the caller instead of destroying it. It requires the
same type, visibility, mutability, and loan checks. It supports nested fields,
boxed owners, and fields reached through collection elements. It allocates
nothing and keeps the field initialized. It also accepts a list element and
the target of a mutable reference, replace(*reference, replacement), but not a
whole binding or temporary receiver. A user declaration named replace
continues to resolve normally.
Use replace when the owner is reached through a reference or defines
__del__, or when it must stay usable as a whole. A field of an owned binding
can instead move out
and be assigned later.
User-defined arithmetic operators
Classes and user-defined enums can implement arithmetic with Python-style special methods. The compiler selects the method statically and emits an ordinary direct call. There is no runtime lookup, boxing, or allocation for dispatch.
| Operator | Left method | Reflected method | In-place method |
|---|---|---|---|
+ | __add__ | __radd__ | __iadd__ |
- | __sub__ | __rsub__ | __isub__ |
* | __mul__ | __rmul__ | __imul__ |
/ | __truediv__ | __rtruediv__ | __itruediv__ |
// | __floordiv__ | __rfloordiv__ | __ifloordiv__ |
% | __mod__ | __rmod__ | __imod__ |
& | __and__ | __rand__ | __iand__ |
| | __or__ | __ror__ | __ior__ |
^ | __xor__ | __rxor__ | __ixor__ |
<< | __lshift__ | __rlshift__ | __ilshift__ |
>> | __rshift__ | __rrshift__ | __irshift__ |
Unary +, -, and ~ call __pos__, __neg__, and __invert__.
Precedence and associativity are unchanged.
Signatures and resolution
A value-producing hook has a shared self, exactly one other parameter for
binary operators, and no other parameters for unary operators. The other
operand is a shared reference or copyable value. Its type can differ from the
receiver’s. The result is an owned value of any supported return type, including
Result; unit and reference results are rejected. Hooks cannot declare method
type parameters, parameter defaults, or yield. Generic classes and ordinary
generic functions can use operators after specialization.
For a + b, the compiler selects a.__add__ if declared; otherwise it selects
b.__radd__ if declared. A selected hook’s signature must match: a type error
does not cause fallback. There is no dynamic NotImplemented protocol, subtype
priority, or implicit conversion. A reflected method is allowed for operands
of the same type when the left method is absent. Ordinary member visibility
applies: exported APIs need pub hooks.
Both operands are evaluated exactly once, left to right, even for a reflected call. Shared operands stay borrowed through the call. Temporary owners survive the full expression and are destroyed normally; no implicit deep copy occurs. Methods can still be called explicitly.
An operation that allocates should return Result, just as a named method does.
Write (a + b)? to propagate failure: ? binds more tightly than +.
The operator syntax does not automatically propagate, unwrap, or discard errors.
In-place operations
target += value calls target.__iadd__(value); the other in-place names follow
the table. Its signature requires self: &mut T, one shared or copyable operand,
and -> (). It mutates the existing destination and does not replace it with a
returned owner. The target must be a writable binding, field, collection element,
or exclusive reference. Its indices are evaluated once before the right side.
The ordinary mutable-method receiver reservation prevents invalidating that place
while evaluating arguments.
There is no fallback from += to +. For an operation that creates a new value
or can fail, use explicit assignment (a = (a + b)?) or a named fallible method.
This keeps allocation and error handling visible. An in-place hook can perform
explicitly handled allocation inside its body; dispatch itself allocates nothing.
Newtype forwarding
A class annotated @newtype wraps exactly one field and forwards every
operator that the field’s type supports, so a wrapper keeps its field’s
arithmetic without one hook per operator:
@newtype
class Metres:
value: i64
def main() -> Result[(), IoError | OverflowError[i64]]:
mut total = Metres(3) + Metres(4)
total *= Metres(2)
print(total, -total, total > Metres(10))?
Ok(())
Metres(value=14) Metres(value=-14) True
a + b on two Metres values lowers to a.value + b.value through the
ordinary operator rules for the field type, then wraps the result:
- A result of the field’s type becomes the class:
Metres + Metresis aMetres. - A
Result[F, E]result, whereFis the field type, becomesResult[Class, E].@newtype class Name: text: strgivesName + Name -> Result[Name, AllocError], written(a + b)?. - Any other result is returned as is, such as the
f64from a field hook__mul__(self, other: &Vec2) -> f64. <,<=,>, and>=forward and returnbool.==and!=stay structural, as for every class.- Unary
+,-, and~forward in the same way. target += valueperforms the field’s own in-place operation,target.value += value.value, so it follows the field type’s+=.
Forwarding happens only when no hook is selected and both operands have the
same @newtype class. Typing stays nominal: Metres + i64, i64 + Metres,
and Metres + Feet are errors unless a hook accepts them; there is no
implicit conversion. An explicit hook replaces forwarding for its operator
only: a @newtype class with __add__ uses it for + and still forwards -.
Reflected hooks apply only to mixed operand types, so they never compete with
forwarding.
Each operand is evaluated once, left to right, and borrowed as a hook operand
would be; forwarding never moves or copies the wrapper. Overflow errors,
division rules, allocation failure, and other errors are those of the wrapped
operation. An operator the field type rejects is rejected with the field
type’s diagnostic: a list[i64] wrapper has no +, and an f64 wrapper has
no %. Forwarding allocates nothing beyond what the wrapped operation does.
Forwarded operations build the result through the class constructor, which is
why a @newtype class cannot define __init__. The declaration rules are in
annotations.
Equality, ordering and hashing
Equality, ordering and hashing never call user code:
==and!=compare fields structurally, including inside collections.<,<=,>and>=on a class come only from newtype forwarding.- Dictionary keys and set elements use built-in structural hashing. Classes are not keys yet; named enums with supported key payloads are accepted.
A class, enum or protocol that declares __eq__, __ne__, __lt__,
__le__, __gt__, __ge__, __cmp__, __hash__ or __key__ is rejected,
so a Python-style hook cannot compile and then be ignored. Rename a method
that is not meant as a hook, or compare explicitly:
class Money:
cents: i64
def same_amount(self, other: &Money) -> bool:
self.cents == other.cents
def main() -> Result[(), IoError]:
print(Money(1) == Money(2), Money(1).same_amount(&Money(1)))?
Ok(())
False True
User-defined ordering through one __cmp__ hook is planned under
B37, and user-defined keys under
B74; see the ordering
and hashing proposals.
Reserved method names
A method named __name__, with two underscores on each side, must be a hook
the compiler calls: an operator hook in the tables above, or, on a class,
__init__, __del__, __enter__ or __exit__. A class, enum or protocol that declares any
other such name is rejected, so a Python special method cannot compile and
then be ignored. The names stay free for hooks the language may add. Where
Plenty has its own form, the diagnostic names it:
| Python method | Plenty form |
|---|---|
__str__, __repr__, __format__ | display and debug methods |
__pow__ | a pow method, called as base.pow(exponent) |
__iter__, __next__ | a generator method |
__len__, __contains__ | an ordinary shared len method for len(value); explicit search methods for membership |
__bool__ | a predicate method such as is_empty |
__getitem__, __setitem__ | named methods such as get and set |
len(value) calls an ordinary
shared len(self) -> i64 method on classes and enums. Membership through in
and not in still uses only intrinsic operations; user-container membership
remains under B49. A name with underscores on one side
only, such as __size, is an ordinary method.
Boundaries
Custom truthiness, indexing, membership, assignment, dereference, **, and
@ operators are not implemented; their Python hooks are
reserved.
and, or, and not keep their boolean and short-circuit rules. Numeric
operators retain their overflow checks and division rules. Operators a newtype
forwards to an integer field report OverflowError of the field type through
the function’s declared Result
by default, through a local checked: block,
or as a violated precondition in unchecked:, just like the field’s own
operators. A user-defined
hook is an ordinary call with its own declared contract, independent of the
caller’s mode; a fallible hook still needs explicit Result handling.
See the operator tutorial, assignment rules, and implementation and performance notes.
Operator lowering and performance
Operator hooks use the same typed call operations, loan checks, argument ABI, caller attribution, and reachability/cache machinery as named calls. Reflected operations evaluate arguments in source order and reorder their already computed values for the call. They never rerun operand expressions. Dispatch uses the specialized nominal types; it does not add vtables or runtime type tests.
Keep four costs distinct:
- Dispatch: an ordinary direct call, eligible for the same bounded inlining as an explicitly named call.
- Borrowing: shared parameters lend existing storage without copying matrix payloads. The compiler may need an addressable temporary for a computed owner.
- Results: inline class result storage uses the normal return ABI. Returning a large matrix still requires whatever allocation its implementation specifies.
- Expression chains:
(a + b) + cinvokes two operations and may build two results. This implementation does not fuse loops, build expression templates, or silently change evaluation order or floating-point association. Named destination-writing methods remain useful for large numerical workloads.
Bounded optimization and evidence
The release leaf inliner accepts shared class receivers that only project scalar fields, alongside its existing scalar inputs and arithmetic. It still requires a scalar output, no locals, no calls or writes, and at most 24 non-marker typed operations. Tail calls to eligible leaves also substitute their body when the caller has no managed locals to clean up before the call. This benefits ordinary named methods as well as operator hooks. Managed aggregates, matrix allocation, and result-producing ordinary class constructors remain ordinary calls. Quantity values use their storage type’s scalar eligibility.
The regression probe compares a + b, a.__add__(b), and a.x + b.x for a
class containing one f64. Operator and named calls have identical native
instructions in both build modes. In release mode all three produce identical
12-instruction x86-64 bodies, with no call or jump to the hook. Before the
inliner extension, both call spellings added a five-instruction tail-call wrapper.
This is code-generation evidence, not a claim about matrix-kernel throughput.
Separate tests disable heap allocation across owned class addition and in-place
updates, and check destructor and failure paths.
Quantity erasure
The builtin Quantity carries its storage type’s native
ABI. Its constructor and numeric projection lower directly, with no call,
allocation, dimension tag, or constructor state. Direct get and float to
retain argument validation and receiver loans while avoiding addressable
receiver temporaries for named numeric values. Module unit constants substitute
their folded storage literal and type retag at each read.
tests/test_quantity_erasure.rs compares complete release instruction listings
for Quantity and plain f32/f64 functions: a + b, a * b, q.to(unit),
and q.to(km) with a constant coherent value of 1000.0. On x86-64 the paired
listings are identical, including parameter/result registers, with no Quantity
constructor, method, constant-getter or caller-context calls. The test uses the
host backend; native ARM execution remains part of ARM qualification rather
than evidence from this x86-64 run.
Other tests exercise every numeric width and representative integer domains through direct/indirect and tail calls, references, aggregates, collections, closures and generators in both build modes. Signed zero, NaN, infinity and worker transfers retain their numeric payloads. Existing allocation-denial, checked-arithmetic, formatting and diagnostic tests cover the language contracts. This evidence does not imply identical instructions for arbitrary programs or shared bodies for different dimension instances.
Design sources
Python supplies the familiar hook names and the separation between ordinary,
reflected, and in-place operations. Plenty deliberately uses static resolution,
shared value-producing receivers, and unit-returning mutable in-place methods
instead of Python’s dynamic NotImplemented and replacement-result behavior.
See the Python data model.
D’s operator specification rewrites operators into named member calls and has separate right-hand and assignment hooks. Its operator-string templates reduce declaration repetition; they do not themselves eliminate allocation or intermediate results. Plenty uses ordinary named methods without adding string-template machinery.
The C++ operator rules likewise transform operator notation to function calls while preserving the operator’s sequencing rules. That separation motivates keeping source evaluation order independent of which operand owns the selected hook. Inlining remains an optimization: even Clang’s always-inline attribute does not guarantee substitution. Plenty reuses its bounded leaf substitution rather than adding an operator-only optimizer or promising that every call vanishes.
Rust’s Add trait separates
the right-hand operand and output types. Plenty’s signatures likewise need not
return the receiver type; this permits scalar results and explicit Result
values without introducing associated types for this feature.
Suites, conditionals, and returns
A suite’s final expression supplies its value. Other expressions evaluate and
are discarded, except that implicitly discarding a Result is a compile error.
Handle it with ?, matching, or a function; drop(result) explicitly discards
it. This applies in non-final conditionals, matches, loops, resource scopes,
and generator bodies. It does not prohibit unused bindings, Option, or
aggregates containing Results.
Continuing branches of value-producing if/elif/else suites must agree on
type. An if without else can only produce unit. The inline conditional uses
value_if_true if condition else value_if_false; only its selected value is
evaluated.
An if takes any number of elif arms. The arms are alternatives at one level,
tested in order; they do not count toward the
nesting limit. An if written inside an else suite does.
def clamp_low(value: i64, minimum: i64) -> i64:
if value < minimum:
return minimum
value
return value exits the enclosing function from any suite. Bare return
returns unit. return if condition: returns its selected suite’s value.
return match value: returns the final expression of the
selected case. Every explicit return must match the declared result, including
returns in non-final or statically unchosen branches.
A returning branch contributes no value to paths that continue. Every continuing
function path must still produce the declared result; an if without else
cannot prove that all paths return. This analysis follows structure without
constant-condition folding. Statements after an explicit return or an exhaustive
conditional whose branches all return are unreachable errors.
break and continue likewise contribute no branch value when they exit a suite
inside a loop. Ordinary exits perform cleanup.
An if as a statement’s value
A block if can be the whole right-hand side of a binding or single-place
assignment, or follow return. Its selected suite’s final expression supplies
the value:
def classify(flag: bool) -> Result[i64, OverflowError[i64]]:
result = if flag:
base = 10
base + 1
else:
2
Ok(result * 2)
def main() -> Result[(), IoError | OverflowError[i64]]:
print(classify(True)?, classify(False)?)?
Ok(())
22 4
mut and annotations work as for ordinary bindings. An annotation or the
assigned destination’s type supplies the expected type to every continuing arm,
including every elif and else; this allows contextual Nothing and empty
collection literals. Continuing arms must agree. Arms ending in return,
break, or continue contribute no value. If every arm exits, write an ordinary
if statement instead of binding its nonexistent value. An absent else
produces unit on its unselected path; unit cannot be stored in a binding.
Conditions are evaluated in order; only the selected suite runs. Its locals
are cleaned up before the binding or assignment, including before evaluating
destination indices. ?, return, break, and continue
keep their enclosing function or loop. A binding or assignment cannot receive a
reference or a closure with borrowed captures from the block; produce an owned
value instead. These are the same lifetime and evaluation rules as a
match used as a value.
return if condition: returns each selected suite’s final expression as though
it ended with return expression. Returned references follow ordinary
returned-reference rules, and calls
in these final positions remain source-tail calls.
A suite without a value returns unit. Every path leaves the enclosing suite,
so a statement directly after the whole block is unreachable. An arm that
executes break or continue still exits to its enclosing loop rather than
returning from the function; code after that loop can run.
The block ends the statement. Block if and match remain unsupported inside
call arguments, operands, augmented assignments, or assignments to several
targets. Bind the selected value first, or use an inline conditional for two
single expressions.
Pattern matching
match evaluates its scrutinee once and selects a case. The scrutinee is an
enum, an anonymous sum, a class, a tuple, an integer, a
str, or a bool, or a reference to one:
enum Reading:
Missing
Value(i64)
def amount(reading: Reading) -> i64:
match reading:
case Reading.Missing:
0
case Reading.Value(number):
number
A case holds one pattern. A pattern is one of:
| Pattern | Matches |
|---|---|
_ | any value, binding nothing |
name | any payload, bound to an immutable local |
_ as name | the whole matched value, bound to an immutable local |
name: MemberType | a whole member of an anonymous sum; see member patterns |
Enum.Variant or Enum.Variant(p, ...) | that variant, with one pattern per payload position |
Some(p), Nothing, Ok(p), Err(p) | that standard variant |
(p, q) or (p,) | a tuple with one pattern per component |
Class(p, ..., field=p, ...) | a class value, with one pattern per field; see class patterns |
1, -1, 0u8 | an integer equal to the literal |
"text" | a str equal to the literal |
True, False | that bool |
p | q | a value either alternative matches |
A user-defined or library variant is always qualified by its enum’s name, a
module path to it, or an alias of the matched type. Some, Nothing, Ok, and
Err are never qualified. A pattern takes no type arguments, because the
matched value supplies them: case Choice.Value(v) matches a Choice[i64] and
a Choice[str] alike, and case Choice[i64].Value(v) is an error.
At an anonymous-sum position, an enum variant pattern selects that enum
member before testing its variant. This also works in nested payloads, such
as Err(ParseError.Invalid) for Result[T, AllocError | ParseError].
Coverage includes every variant of every member. A generic qualifier that
names several members is ambiguous; select one with a typed member pattern
before matching its variants. See anonymous sums.
A bare name binds only at a payload or field position; the case’s own pattern
is a variant, a class or tuple pattern, a literal, _, alternatives of those,
_ as name, or a typed member pattern. A bare name as the case’s own pattern is an error, and a payload name equal to a variant of
that payload’s enum is rejected, so a forgotten qualifier cannot silently bind
everything. A variant pattern at a Box[T] payload matches the box’s contents.
Patterns nest to at most 64 levels.
enum Shape:
Dot
Circle(i64)
Rect(i64, i64)
def size(shape: Option[Shape]) -> i64:
match shape:
case Some(Shape.Circle(side) | Shape.Rect(side, _)):
side
case Some(Shape.Dot) | Nothing:
0
A literal pattern matches an integer, str, or bool position: the scrutinee
itself, a payload, or the contents of a Box. An unsuffixed integer literal
takes the position’s type and must fit it; a suffixed one must name that type
exactly. Float literals are not patterns.
Tuple patterns
Tuple patterns are recursive fixed-arity products. (p,) matches a singleton;
(p) groups a pattern. Empty () is unit, not an empty tuple pattern; use _
for a unit component. A tuple pattern can appear at the root or inside enum,
class, tuple, or boxed payloads. Every component must have a pattern.
At an anonymous-sum position, it selects a unique tuple member with that arity.
If several members have the same arity, bind one with a typed member pattern
before matching its components.
def combine(left: Option[i64], right: Option[i64]) -> Result[i64, OverflowError[i64]]:
match (left, right):
case (Some(a), Some(b)):
Ok(a + b)
case (Some(a), Nothing):
Ok(a)
case (Nothing, Some(b)):
Ok(b)
case (Nothing, Nothing):
Ok(0)
def main() -> Result[(), IoError | OverflowError[i64]]:
print(combine(Some(2), Some(3))?)?
Ok(())
5
Coverage considers combinations of component patterns, so omitting
(Nothing, Nothing) above is an error. Alternatives retain the existing
256-combination expansion limit and decision-tree work limit. Matching owns and
moves each bound affine component once and destroys ignored components once.
Matching a reference instead binds component references, preserving disjoint
component loans.
Class patterns
A class pattern takes a class value apart, written as the class’s generated
constructor takes its fields: positional patterns in field-declaration order,
then field=pattern. It matches a class position, the contents of a Box, or
that class’s member of an anonymous sum.
class Point:
x: i64
y: i64
def axis(p: Point) -> Result[i64, OverflowError[i64]]:
match p:
case Point(0, y):
Ok(y)
case Point(x=x, y=0):
Ok(x)
case Point(x, y):
Ok(x + y)
The rules have no exceptions:
- Every field appears exactly once; ignore one with
_orfield=_. A class with no fields is writtenName(). - Positional patterns precede named ones. An unknown, repeated, or missing field, or one positional pattern too many, is an error.
- The class is named as in an expression: by its name, a module path, or an
import. As for enums, the matched value supplies type arguments, so
case Wrap(item)matches aWrap[i64]andcase Wrap[i64](item)is an error. - Each field must be visible where the pattern is written, so a class with a private field cannot be taken apart outside its module.
- A class pattern tests nothing itself: it is irrefutable when every subpattern is, and a nested literal or variant makes it refutable.
- A bare name that spells the class at its position, such as
case Err(OverflowError):, is rejected rather than bound, so the parentheses cannot be forgotten silently. - Owned matching moves bound fields into their bindings and destroys the rest,
as for an enum payload. A class with
__del__expects every field initialized, so an owned match cannot bind its fields; match a reference such as&valueto borrow them, or bind it whole. Borrowed matching binds field references, as for payloads.
A @newtype class or any one-field class is matched the same way; its pattern
costs no projection at run time. The error of a
checked: block is the motivating
case:
def total(a: i64, b: i64) -> i64:
result = checked:
a + b
match result:
case Ok(sum):
sum
case Err(OverflowError(wrapped)):
wrapped
def opcode(name: str) -> i64:
match name:
case "cpy":
1
case "inc" | "dec":
2
case _:
0
def describe(reading: Option[i64]) -> str:
match reading:
case Some(0):
"zero"
case Some(_):
"nonzero"
case Nothing:
"missing"
case _ as name: matches every value the earlier cases leave and binds the
whole matched value, for any supported scrutinee type. It does not make floats,
collections, or other unsupported scrutinees matchable.
It is accepted only as a case’s entire pattern: not as a | alternative,
and not at a payload position, where
the name alone binds. Because it matches everything, a case after it is
unreachable, and it is itself unreachable when earlier cases are exhaustive.
name cannot be a variant of the matched enum, _, or mut.
def forward(shape: Shape) -> i64:
match shape:
case Shape.Dot:
0
case _ as other:
size(Some(other))
Every alternative of p | q binds the same names with the same types, at any
depth. Cases are tried in order and the first match runs. Bindings stay within
the arm and may shadow outer names.
Coverage must be exhaustive; the error lists missing values as patterns, such
as missing Ok(Some(Shape.Dot)), Ok(Nothing) or missing Point(_, _). A case, or a single alternative,
that earlier patterns already cover is an error. Incorrect payload arity and
wrong enum identities are errors. A pattern expands to at most 256 combinations
of its alternatives.
Literals never cover an integer or str position: such a position also needs
_ or a payload name, and a match on an integer or str ends with case _:
or case _ as name:. True and False together cover a bool, so a _ after
both is unreachable. A
repeated literal is an unreachable case.
Matching preserves source-order behavior, including overlapping payload patterns.
Native dispatch uses a jump table for at least 256 distinct integer values or
enum tags forming one contiguous range, in any source order. Signed values and
all integer widths are supported; out-of-range values still take the wildcard
or defensive invalid-tag trap. Table storage grows with the case count.
Smaller or sparse sets use comparison chains; a str case compares lengths
and then bytes. These chains can be sensitive to which cases occur most often; the
dispatch measurements
explain the conservative table threshold. A match on at least 16 distinct str
literals instead binary-searches them once, so its cost grows with the
logarithm of the case count and does not depend on arm order; see the
string match measurements. Keep using match for clear
exhaustive branching; dispatch selection does not change its behavior.
Guards, float patterns, range patterns, and pattern as name
for any pattern other than the case’s own _ are unsupported. A scrutinee of any other type, such as a float or a
collection, is rejected with advice to use an if/elif chain.
Payload and whole-value bindings cannot be declared mut. To change an owned
payload, move it into a mutable local inside the arm (mut local = payload); to change it in
place, match &mut value.
Continuing arms must agree on their result type. Arms ending in return, break,
or continue contribute no join value. A function-tail match supplies the final
expression of its selected arm, just like a tail conditional. A match elsewhere
in a suite is a statement, and its arms’ values are discarded, unless it is
a statement’s value.
A match as a statement’s value
A match block can be the whole value of these statements, and of no other
position:
| Statement | Effect |
|---|---|
name = match value:, with mut or an annotation as for any binding | declares the local, or assigns an existing mut local |
place = match value:, where place is a field, a collection element, or *reference | assigns the place |
return match value: | returns from the function |
def area(shape: Shape) -> Result[i64, OverflowError[i64]]:
side: i64 = match shape:
case Shape.Dot:
return Ok(0)
case Shape.Circle(side):
side
case Shape.Rect(width, height):
if width < height:
width
else:
height
Ok(side * side)
The statement ends with the last case; nothing follows the block on its line. Patterns, exhaustiveness, and ownership of the scrutinee are those of any other match.
In a binding or assignment, the final expression of the selected case is the
value. A final if or match supplies the value of its selected suite.
- Continuing cases must agree on one type. An annotation, or the type of the
assigned local, is the expected type of every case’s final expression, as it
is for an ordinary right-hand side; without one, each case is inferred alone,
so
Nothingor[]?needs the annotation. - A case that ends in
return,break,continue, or a propagating?gives no value. At least one case must continue; a match whose cases all exit is rejected with advice to write it as a statement. - A case whose final statement has no value, such as
passor an assignment, makes the match unit, which cannot be bound. - The value cannot be a reference or a closure with borrowed captures, because
the case’s bindings end before the statement uses it. Produce an owned value,
or declare
mut nameand assign in each case. - Earlier expressions in a case are discarded as in any suite, so an unhandled
Resultthere is still an error.
Evaluation order is: the scrutinee; the selected case’s suite; cleanup of that case’s bindings and locals, in reverse declaration order, and of scrutinee parts no pattern bound; then the binding or assignment. An assignment evaluates destination indices and destroys the replaced value after all of that. An exit from a case performs the ordinary cleanup for that exit and assigns nothing.
return match value: returns the final expression of the selected case, exactly
as if that case ended with return expression. Every case therefore exits, a
call there is a source-tail call, and a
statement after the block is unreachable. A case that ends without a value
returns unit, which is an error unless the function returns unit. A returned
reference follows the rules of any other returned
reference.
Augmented assignment, assignment to several targets, call arguments, operands,
and every other expression position reject a match block; bind its value to a
name first. A block if supports the
same whole-statement value positions and retains these expression-position
restrictions.
Ownership
Owned matching consumes the scrutinee: bound payloads move into their bindings
and every part no pattern binds is destroyed. _ as name moves the whole value
into name, with no copy or allocation. Testing a literal copies the
integer, str, or bool it compares, so a later pattern can still bind or
move the value; for a str the copy shares the text and allocates nothing. Matching a shared or exclusive
reference binds shared or exclusive payload references instead; see
borrowed matching. Unit payload bindings produce unit
expressions and can be returned from a unit-returning function.
Loops and control transfers
while condition: repeats its suite while the Boolean condition is true.
for name in source: evaluates its source and visits the supported sequence.
Flat tuple unpacking is also accepted in the loop binding. Source kinds,
iteration loans, generators, and comprehensions are described under
iteration.
break exits the nearest loop. continue advances to its next iteration.
Both must appear within a loop. Statements following either transfer are
unreachable errors. Loops are unit-valued; loop else suites are unsupported.
Declarations inside a loop body do not establish values outside it. Assignments to existing mutable locals persist. A loop may run zero times and cannot alone prove that a function returns or that an initializer sets every field.
Control transfers destroy values whose ownership ends on that path. An explicit
return exits the enclosing function, while ? follows the enclosing callable’s
error propagation rule.
Format strings
An f-string is a string literal with the prefix f: f"...", f'...',
f"""...""", or f'''...'''. Literal text uses the ordinary
string escapes, including \u{H...}; {{ and }} are
literal braces, and a single } is an error. Each {...} is a replacement
field:
{field[=][!conversion][:spec]}
The expression f"..." has type Result[str, AllocError], like a collection
literal: s = f"x={x}"? or print(f"x={x}"?)?. Its type is
Result[str, AllocError | FormatError] when formatting can fail other than by
allocation: a field runs a user display or debug method,
directly or for an element of an aggregate, a spec has
nested fields, or a spec has type c. The choice
is made per site from the fields’ types; in a generic function it is made for
each instantiation. out.format(f"...") writes one
to an existing formatter instead. The
compiler parses the template and checks every spec while compiling. It lowers
the literal to calls on a std.fmt string formatter,
with no format-string interpreter at run time. A module that uses an f-string
loads std.fmt without an import. FormatError is a
prelude name, so a signature names it without an
import too (issue #632).
from std.fmt import Formatter
class Tag:
name: str
def display[W](self, out: &mut Formatter[W]) -> Result[(), FormatError]:
out.format(f"#{self.name}")
def plain(n: i64) -> Result[str, AllocError]:
pair = [n, n]?
Ok(f"n={n:>4} {pair}"?)
def tagged(tag: &Tag, width: i64) -> Result[str, AllocError | FormatError]:
Ok(f"{tag} {1:{width}}"?)
def main() -> Result[(), AllocError | IoError | FormatError]:
print(plain(7)?)?
print(tagged(&Tag("x"), 3)?)?
Ok(())
n= 7 [7, 7]
#x 1
A function that declares only FormatError for an f-string with built-in
spellings is rejected, and the diagnostic names AllocError as the
replacement.
Fields
A field is any expression, for example {p.x}, {xs[i + 1]},
{total / count:.2f}, {a > b}, {'yes' if ok else 'no'}, or
{parse(text)?}. As in Python, {a, b} is the tuple (a, b). A : or !
outside brackets ends the expression, so a lambda must be parenthesized.
Errors inside a field are reported at their position in the source.
All fields, including the nested fields of specs, are evaluated once, left to
right, before any text is built. Each value is lent to the formatting code, so
formatting moves nothing; a temporary value lives until the f-string’s text is
built. A ? in a field belongs to the enclosing function: on Err or
Nothing it returns from that function before formatting starts, exactly as
? in a call argument does. The Result of the f-string itself reports only
formatting failure, so f"{parse(s)?}"? has two independent ?s.
A field cannot contain a backslash, a comment, or the quote character that encloses the f-string. A field in a triple-quoted f-string may span lines; a line break in its spec is a character of the spec, such as a fill.
Only the f prefix starts an f-string. F"..." and the combinations bf/fb
are errors: build text with an f-string, then convert it to bytes.
{expr=} writes the field’s source text, including the = and any spaces
around it, then its value. Without a conversion or a spec the value is written
as with !r, so {name=} gives name="Ada".
Quantity fields apply numeric specs to
the stored number and append the coherent-unit symbol. Use !s to pad the
whole display text, or !r for Quantity[T, D](value) debug output.
Values and conversions
Without a conversion, a field is written as print writes it, including its
user formatting methods:
| Type | Spelling |
|---|---|
str | The text itself |
bool | True or False |
| Integers | Decimal |
f32, f64 | Shortest round-trip, as str.from |
A class or enum with a display method | That method, with the depth accounting of std.fmt.display |
A class or enum with only a debug method | That method |
Anything else, including bytes | The str.repr spelling, as print shows aggregates and b"..." values; elements with a debug method use it |
A field calls a top-level display or debug from the module where the
f-string is written, as a direct call there would. A private class, or a
private method, formats in its own module. In another module, including through
a generic function defined there, the class and the method must be public;
otherwise the field is rejected with the ordinary private-access diagnostic.
Methods of aggregate elements run as str.repr runs them, with their own
module’s access.
!s is the same spelling. !r writes str quoted and escaped as
Formatter.debug_text does, uses a type’s debug method when it has
one, and otherwise the str.repr spelling, including element debug methods. !a, like Python’s ascii(),
takes the !r text and spells each non-ASCII scalar as \u{hex} with
lowercase digits, the escape debug_text uses for controls:
f"{'café'!a}" is "caf\u{e9}", which is also a valid string literal. Python spells the same escapes \xe9,
\u2603, or \U0001f600, with single quotes.
A @redacted type without its own
display or debug method writes TypeName(@redacted) under every conversion and
spec, including !r, !a, and =.
Format specs
A spec follows Python’s format specification mini-language:
[[fill]align][sign][z][#][0][width][grouping][.precision][type]
alignis<(left),>(right),^(center; extra fill goes right), or=(fill between the sign and the digits, numbers only). Text defaults to left and numbers to right.fillis any single character.signis-(default),+, or a space.z(floats only) drops the sign of a result that rounds to zero, so{-0.0001:z.2f}is0.00;-infkeeps it.#adds0b,0o,0x, or0Xto integers, and keeps the point or trailing zeros of floats.0pads numbers with zeros after the sign and prefix. With grouping, the zeros are grouped too:{-1234:010,}gives-0,001,234. After an explicit alignment,0is the fill character.widthis a minimum count of Unicode scalars; output is never truncated to it.groupingis,or_: every three integer digits, or every four with_and typesx,X,o, andb.precisionis the digits after the point fore,f, and%, the significant digits forgand for a float with no type, and the maximum number of scalars for text.
Width and precision are decimal numbers of at most 4096.
| Value | Types | Other limits |
|---|---|---|
Text (str, bool, values with a conversion or no built-in spelling) | none, s | No sign, z, #, 0, grouping, or = |
| Integers | none, d, x, X, o, b, n, c | No precision or z; , only for decimal; c takes no sign, #, or grouping |
f32, f64 | none, e, E, f, F, g, G, %, n |
c writes the Unicode scalar whose code point is the integer, laid out like a
number: right-aligned by default, and 0 pads with zeros. A value that is not
a scalar (negative, a surrogate, or above U+10FFFF) makes the f-string return
Err(FormatError.InvalidSpec), so a field with c adds FormatError to the
f-string’s type. n is Python’s n in the C locale, which is
the locale Python formats with unless a program calls locale.setlocale: d
for integers and g for floats, with no grouping. Plenty has no locales, so
grouping with n is rejected rather than ignored.
Float types follow Python, including exponents with a sign and at least two
digits (1.23e+03), % multiplying by 100, and g choosing fixed or exponent
notation. Typed floats spell infinity and NaN as inf and nan (INF, NAN
for upper-case types). A float without a type or precision keeps the print
spelling (1e16, NaN); with a precision it behaves as Python’s .Ng
with at least one digit after the point. Because Plenty has a bool type, a
spec on a bool formats its text rather than the number 1 or 0.
An unsupported option for the field’s type is a compile error at the spec,
for example format spec `.2f` does not apply to i64: integers take no precision; convert with f64(...) to format a float. Convert explicitly to use
another type’s options.
Nested fields in a spec
A spec may contain fields, as in {x:{width}} or
{x:{fill}{align}{width}.{precision}f}. A nested field is an expression with
no =, conversion, spec, or further nesting. Python pastes nested values into
the spec text and parses the result at run time; Plenty instead parses the
spec while compiling, and each nested field supplies exactly one option,
chosen by its position and its type:
| Nested field | Type | Supplies |
|---|---|---|
First, before an alignment character or a second str field | str | The fill |
| First otherwise, or second after the fill | str | The alignment |
| Where the width’s digits would be | Any integer | The width |
After . | Any integer | The precision |
Any other nested field, such as an integer before the alignment or two adjacent
integer fields, is a compile error, as is 0 after a nested alignment (write
the fill instead). Everything written in the spec, including its type, is
checked against the field’s type as usual; {x:{w}.2f} with an integer x is
rejected because of .2f whatever w is.
The nested values are checked when the f-string runs, and an invalid one makes
it return Err(FormatError.InvalidSpec) before any text is built:
- an alignment must be
"",<,>,^, or=(=only for numbers);""means the default alignment; - a fill must be one scalar, or
""for the default space, and a non-empty fill needs a non-empty alignment; - a width or precision must be in
0..=4096.
For valid values the result is the one Python gives. Where Python would read a
pasted value differently, such as a str field holding "<10" or a fill with
an empty alignment, Plenty returns InvalidSpec rather than another layout.
Writing to a formatter
out.format(f"..."), where out is a std.fmt.Formatter[W]
or a &mut reference to one, writes the f-string to out without building a
str, and has type Result[(), FormatError]. It is the form to use in a
display or debug method:
pub def display[W](self, out: &mut Formatter[W]) -> Result[(), FormatError]:
out.format(f"({self.x}, {self.y})")
The argument must be an f-string literal; the compiler lowers the pair to
calls on out, so out’s byte budget, depth limit, and partial-progress rules
apply, and its scratch is reused. A field that needs a temporary string (a
spec on a value without a built-in spelling, !r on str with a spec, !a)
still allocates one. format is not an ordinary method of Formatter: it has
no other form, and a user type’s own format method is unaffected.
Cost and failure
The result of f"..." is built in a new string formatter whose output and
32-byte scratch are allocated for each evaluation; a float spelling longer than the scratch
grows it. A spec on a value without a built-in spelling first builds that
value’s text as a temporary string. There is no byte budget: an f-string is
bounded only by allocation failure, reported as AllocError whatever the
f-string’s type; a user method’s own FormatError.Allocation(...)
is reported as AllocError too. User display and debug
methods may nest through std.fmt up to 32 levels, after which the result is
FormatError.RecursionLimit. No partial string is returned on error.
Slices
A subscript whose brackets contain a top-level : is a slice. It copies the
selected elements into a new value and returns Result[_, AllocError], so the
copy is visible as a ? (issue #579). A tuple slice allocates nothing and
returns the tuple itself (see tuple slices):
def main() -> Result[(), AllocError | IoError]:
xs = [0, 1, 2, 3, 4, 5]?
print(xs[1:3]?)?
print(xs[-2:]?)?
print(xs[::2]?)?
print(xs[::-1]?)?
print("Aé🙂Z"[1:3]?)?
Ok(())
[1, 2]
[4, 5]
[0, 2, 4]
[5, 4, 3, 2, 1, 0]
é🙂
Syntax
slice = primary "[" [start] ":" [stop] [":" [step]] "]"
xs[:], xs[::], xs[a:], xs[:b], xs[::c] and xs[a:b:c] are all
slices. A slice has the precedence of indexing and the other postfix forms:
xs[1:3]?[0] indexes the copy and s[1:]?.upper()? calls a method on it.
A comma inside slice brackets is rejected: there are no multi-dimensional
subscripts. The receiver, then each written bound, evaluate once from left to
right. The receiver is only read, so a named receiver stays usable.
Bounds and steps
start and stop are i64 expressions or references to i64; other integer
types need a cast such as i64(n). Positions count elements, Unicode scalars
for str, bytes for Bytes and bytes, and rows for clist.
Out-of-range bounds clamp exactly as in Python. A negative bound counts from the end, and positions outside the receiver select nothing rather than trapping. Omitted bounds depend on the sign of the step: a positive step runs from the start to the end, and a negative step runs from the last element back to the first.
The step has type NonZero[i64] and defaults to 1, so a zero step cannot be
written or computed. A nonzero integer literal converts implicitly, and a literal
0 is a compile error. A computed step needs an explicit conversion, which
reports a zero step as RangeError:
def main() -> Result[(), AllocError | IoError | RangeError | OverflowError[i64] | ZeroDivisionError]:
xs = [0, 1, 2, 3, 4, 5]?
step = NonZero.try_from(len(xs) // 2)?
print(xs[::step]?)?
print(xs[100:]?)?
print(xs[-100:2]?)?
Ok(())
[0, 3]
[]
[0, 1]
Receivers and results
| Receiver | Result | Elements |
|---|---|---|
list[T] | Result[list[T], AllocError] | Duplicated with copy(); Shared[T] handles with share() |
str | Result[str, AllocError] | Unicode scalars |
std.bytes.Bytes | Result[std.bytes.Bytes, AllocError] | Bytes |
bytes | Result[std.bytes.Bytes, AllocError] | Bytes; no import needed |
clist[T] | Result[clist[T], AllocError] | Rows copied column by column |
| Tuple | The selected tuple, or (); no Result | Components selected at compile time |
A list slice of a nested list copies each selected inner list, so later changes
to the source do not reach the slice. A step-1 slice of a list whose elements
own no storage, such as list[i64], is one block copy. Every slice reserves its
result storage once, before copying; allocation failure returns
Err(AllocError.OutOfMemory) and leaves the receiver unchanged.
A list element type that copy() rejects, such as std.bytes.Bytes, is a
compile error. Transfer such elements out of an owned temporary with the
slice() method instead. A
Shared handle in an element, as in list[(str, Shared[T])], gains another
owner, as copy() does.
A bytes literal slice gives an owned std.bytes.Bytes. Slice syntax makes the
Bytes type available without an import std.bytes line; naming the type in a
signature still needs the import. The compiler loads std.bytes for this only
when some module of the program could produce a bytes value: one with a
b"..." literal, include_bytes, or the word bytes. A program that slices
only lists or text does not load it.
Tuple slices
Tuple slice bounds and step must be integer literals, because the result type
depends on them: (1, "two", 3.0)[1:] has type (str, f64). A slice that
selects no component is (). A tuple slice copies components inline and
allocates nothing, so it returns no Result and takes no ? (issue #625):
def main() -> Result[(), IoError]:
t = (1, "two", 3.0)
tail: (str, f64) = t[1:]
print(tail, t[::-1])?
Ok(())
("two", 3.0) (3.0, "two", 1)
A tuple whose selected components are affine is rejected for now; unpack it and copy the components instead.
A clist receiver must be a named place, and the slice is subject to the same borrow checks as other clist reads.
Slice views
&xs[a:b] borrows the selected elements in place, and &mut xs[a:b] borrows
them exclusively (issue #581). A view copies nothing and allocates nothing, so it
needs no ?. Use one to pass part of a list or buffer to a function:
def total(values: &[i64]) -> Result[i64, OverflowError[i64]]:
mut sum = 0
for x in values:
sum += *x
Ok(sum)
def main() -> Result[(), AllocError | IoError | OverflowError[i64]]:
mut xs = [1, 2, 3, 4, 5]?
print(total(&xs[1:4])?)?
tail = &mut xs[-2:]
tail[0] = 40
for x in &mut xs[:2]:
*x *= 10
print(xs)?
Ok(())
9
[10, 20, 3, 40, 5]
The bounds clamp as for a copying slice, so a view is empty rather than out of
range. A view selects adjacent elements, so it takes no step; copy every k-th
element with xs[a:b:k]?.
| Viewed value | Shared view | Exclusive view |
|---|---|---|
list[T] | &[T] | &mut [T] |
std.bytes.Bytes | &[u8] | &mut [u8] |
bytes | &[u8] | rejected: a literal is read-only |
| A view | A narrower view | A narrower view of an exclusive view |
Shared[list[T]] | &[T] of the content | rejected: the content is shared |
&[T] and &mut [T] are types: parameters and local annotations may name
them. A list of u8, a Bytes buffer, and a bytes literal all give &[u8],
so one &[u8] parameter reads any of them. The viewed value must be a named
place, such as xs, self.items, or rows[0], or a bytes or str
literal, as in &b"GET /"[:3]. A shared view argument may also view a call
result, as in w.write(&encode(x)?[2:]): the result lives until the end of the
statement. Bind a call result to a name before binding a view of it to a name,
and before passing an exclusive view of it.
| Operation | Meaning |
|---|---|
len(v) | Number of elements in the view |
v[i] | Element i of the view, with list index rules: a negative index counts from the end, and an index outside the view traps |
v[i] = x | Replace element i; needs an exclusive view in a mut binding |
v.swap(i, j) | Exchange elements i and j in place, with list.swap’s contract; needs an exclusive view |
&v[i], &mut v[i] | Reference to element i |
&v[a:b], &mut v[a:b] | Narrower view; bounds count from the start of v |
for x in v | &T per element, or &mut T for an exclusive view; reversed(v) and enumerate(v) also work |
iter(v) | A pipeline over the elements, with the items of for x in v |
v == w, v != w | Equal lengths and equal elements; both sides are views of one element type |
print(v), str.repr(v), f"{v}" | The elements as a list, [2, 3]; a &[u8] as a bytes literal, b"bc" |
In --unchecked assume builds, view indexing inside a plain
unchecked: suite assumes a
nonnegative index below the view’s own length, with no bounds check or
negative-index normalization. The view’s construction and swap method keep
their normal contracts.
A view’s length never changes, so v[i] in for i in range(len(v)) or
for i, x in enumerate(v) compiles without a bounds check, as described for
lists. An in-place reversal,
while lo < hi: starting with v.swap(lo, hi) or v[hi], checks its
first pass’s indices once, before the loop, as described there.
def even(x: &i64) -> bool:
two: NonZero[i64] = 2
*x % two == 0
def main() -> Result[(), AllocError | IoError]:
xs = [1, 2, 3, 4, 5]?
middle = &xs[1:4]
print(middle, iter(middle).filter(even).count())?
data = b"abc"
print(&data[1:])?
Ok(())
[2, 3, 4] 2
b"bc"
A view prints as a reference does, through to its elements; a Bytes buffer
itself keeps its own spelling. Other list operations, such as methods other
than swap, in, list(v), and copying slices of a view, are rejected. Copy
from the owner, xs[a:b]?, for an owned value.
v.swap(i, j) counts both indices within the view, so a negative index counts
from the view’s end, and an index outside the view traps before either element
moves. Both indices are evaluated before the view is borrowed, so they may read
it, as len(v) - 1 does. Equal indices leave the view unchanged. It accepts
every element type and allocates, copies, retains, and drops nothing. For
numbers and bool it compiles to the inline bounds checks, loads, and stores
of list.swap; other element types make one runtime call, which repairs inline
payload addresses. A shared view is read-only and has no swap:
def reverse(v: &mut [i64]) -> Result[(), OverflowError[i64]]:
mut lo = 0
mut hi = len(v) - 1
while lo < hi:
v.swap(lo, hi)
lo += 1
hi -= 1
Ok(())
def main() -> Result[(), AllocError | IoError | OverflowError[i64]]:
mut xs = [1, 2, 3, 4, 5]?
reverse(&mut xs[1:])?
print(xs)?
Ok(())
[1, 5, 4, 3, 2]
Views borrow their owner
A view holds a loan of its run of elements in its owner: a shared view a
shared loan, and an exclusive view an exclusive one, under the
borrowing rules. While a view is live, its
owner cannot grow, shrink, move, or be reassigned, and len(xs) and &xs
conflict with an exclusive view. Two views are disjoint when one stops where
the other starts, at one literal or non-mut i64 name, so &mut xs[:m] and
&mut xs[m:] can both be live. Overlapping or unprovable ranges, such as
&mut xs[:3] with &xs[2:], conflict.
Disjoint slice views
states the full rule and why it is sound.
Where a view type is expected, a borrow of its owner converts to the whole
view, as [:] would: &xs or &mut xs for a list, &buffer or a b"..."
literal for &[u8], and &handle for a Shared list. A reference binding to
an owner, such as an xs: &list[i64] parameter, converts too. The conversion
copies nothing and allocates nothing. Expected view types are parameters,
including a generic &[T], and annotated locals:
def total(values: &[i64]) -> Result[i64, OverflowError[i64]]:
mut sum = 0
for x in values:
sum += *x
Ok(sum)
def width(data: &[u8]) -> i64:
len(data)
def main() -> Result[(), AllocError | IoError | OverflowError[i64]]:
xs = [1, 2, 3]?
whole: &[i64] = &xs
print(total(&xs)?, total(&xs[1:])?, total(whole)?, width(b"abc"))?
Ok(())
6 5 6 3
&xs[:] keeps working. An owned list, total(xs), is not a view and is
rejected. A &T type parameter takes a reference to a whole value, not a
view. Standard-library operations that read bytes take
&[u8], so buffer.extend(&other[2:]) and
writer.write(b"ok") pass a view.
A function with a view parameter may return a reference into it: the first
reference or view parameter is the
origin of a returned reference.
Views may be passed to scoped workers.
Views are local until stored references land (backlog B11). A view cannot be returned, stored in a class field, tuple, collection, or enum payload, or captured by an anonymous function. Return or store the bounds, or an owned copy.
Substring views
&s[a:b] on a str borrows the selected text in place as a &substr
(issue #595). Like other views it copies nothing, allocates nothing, and takes
no ?. Its bounds are those of the copying slice s[a:b]?: they count
Unicode scalars, a negative bound counts from the end, and bounds outside the
text clamp. A view therefore always starts and ends on a scalar boundary, and
s[a:b]? == &s[a:b] for any a and b.
def vowels(text: &substr) -> Result[i64, OverflowError[i64]]:
mut n = 0
for c in text:
if c in "aeiouéö":
n += 1
Ok(n)
def main() -> Result[(), AllocError | IoError | OverflowError[i64]]:
s = "héllo, wörld"
word = &s[7:]
print(word, len(word), word.byte_len(), vowels(&s[:5])?)?
print(word == "wörld", &s[:1] < word, "ör" in word, word.find("l"))?
for c in reversed(&word[1:3]):
print(c)?
owned = word[:2]?
print(owned)?
Ok(())
wörld 5 6 2
True True True Option[i64].Some(3)
r
ö
wö
&substr is a separate type from &str. A &str refers to a whole str,
whose representation
stores its byte length and scalar count ahead of the text, and every str
operation reads them. A run inside a string has no such prefix, so a view
holds the run’s address and byte length instead. substr is only written as
&substr; there is no owned substr and no &mut substr, since a str is
immutable. A &substr parameter or annotated local also takes a whole string
as &s, a &str binding, or a str literal, converted as &s[:] would be.
| Operation | Meaning |
|---|---|
len(v) | Number of scalars; scans the view’s text, as positions do |
v.byte_len() | Number of UTF-8 bytes; constant time |
v[i] | Scalar i as a one-scalar str; a negative index counts from the end, and an index outside the view traps |
&v[a:b] | Narrower view; bounds count scalars from the start of v |
v[a:b]? | An owned copy of part of the view, Result[str, AllocError] |
for c in v | Each scalar as a one-scalar str; reversed(v), enumerate(v), and iter(v) also work |
v == w, v != w, <, <=, >, >= | Compare by Unicode scalar value, as str does; either side may be a str or a &substr |
needle in v, needle not in v | Substring membership; the needle is a str or a &substr |
v.startswith(x), v.endswith(x) | Exact prefix or suffix; x is a str or a &substr |
v.find(x) | First matching scalar position within the view, or Nothing |
print(v), f"{v}" | The text itself; str.repr(v) and f"{v!r}" quote it as a str |
Each scalar a loop or index yields fits in a string’s inline form, so these
operations allocate nothing. Finding a bound, len(v), v[i], and v.find(x)
scan the text before the position, as they do on a str. str methods other
than those listed are not available on a view; copy it with v[:]? to use one.
A str method does not take a &substr argument, and v in s for a str
s is written v in &s[:].
A substring view borrows its string as other views borrow their owner: while it is live, the string cannot be reassigned or moved, and the view stays local until stored references land (backlog B11).
Limits
| Source | Diagnostic |
|---|---|
ys: list[i64] = xs[1:] | slice returns Result[list[i64], AllocError]; use `?` to propagate allocation failure |
xs[::0] | slice step cannot be zero |
xs[::k] with k: i64 | slice step must be `NonZero[i64]`, not `i64` |
&xs[::2] | a slice view selects adjacent elements, so it takes no step |
&mut s[1:3] on a str | a `str` is immutable, so a substring view is shared only; write `&s[a:b]` |
def f(xs: &list[T]) -> &[T] | a function cannot return a slice view yet (backlog B11) |
total(xs) for total(v: &[i64]) | this parameter takes a slice view: pass `&xs` for a whole list or buffer, or `&xs[a:b]` for part of it |
x in v for a view v | slice views do not support `in` yet |
xs[1:3] = ys | slice assignment is not supported; assign elements by index, or build a new list |
m[1:2, 0] | a slice cannot have several dimensions |
t[n:] on a tuple | tuple slice bounds must be integer literals; the result type depends on them |
t[1:]? on a tuple | a tuple slice `t[a:b]` does not allocate, so it takes no `?`; write `t[a:b]` |
d[1:] on a dict | dict does not support slicing |
class Window:
rows: &[i64]
def main() -> ():
pass
slice views cannot be stored in class fields yet (backlog B11)
First-class slice(...) values are not implemented. The slice syntax proposal
records the design and alternatives.
Ownership, borrowing, and lifetimes
An owner controls a value’s lifetime. Copyable values can have independent owners; affine values transfer their ownership when moved. References lend access without taking ownership. Access permissions end at a loan’s last use; owned resources follow the specified destruction scopes.
- Bindings, moves, and copies
- Borrowing and reborrowing
- Reference binding reassignment
- Returned references
- Optional and fallible references
- Borrowed matching
- Deterministic destruction
Types define which values can be stored together. Iteration and resource scopes apply these same ownership rules to their hidden owners.
Bindings, moves, and copies
Bindings and mutable reassignment follow the declaration and assignment rules. Ordinary value use (assignment, owned argument passing, and return) either copies or transfers ownership according to the value’s type:
Numbers, bool, str, bytes, ranges, and named function values copy. A
tuple, Option, Result, enum, or anonymous sum (A | B) copies when all of
its parts copy. A class copies only when it is declared @copy, and a generic one only when its type arguments
also copy.
Every other value moves: collections, boxes, closures that capture,
generators, and other class instances.
A class of integers is still affine unless declared @copy. Copied strings may
share immutable storage. Inserting an affine value into a collection or
capturing it transfers ownership.
A moved-from binding cannot be read until a permitted reassignment initializes it.
copy(value) observes its argument and returns Result[T, AllocError] containing
an independent value. It recursively duplicates owned mutable contents;
immutable strings can share storage. A nested
Shared handle is not duplicated:
the copy holds another owner of the same content, as share() would.
copy(handle) on a handle itself is rejected in favour of handle.share().
Copying does not supply mutable aliases.
For a value that already copies, copy(value)? equals value; checking
reports it as a redundant-copy warning.
Generators, captured closures, recursive data, and types containing custom
destruction cannot be copied. Allocation
and destruction describe failure and cleanup.
Only paths that continue contribute to availability at a branch join. Moving on
some continuing paths makes a binding possibly moved; moving on all makes it
moved. Reinitializing a mutable binding restores availability. Every loop
backedge, including continue, must preserve outer owners available at loop
entry. Moving and then reinitializing is accepted; carrying the move to a
backedge is rejected. Break paths join the zero-iteration path.
Moving fields
An affine field can move out of a class instance held by an owned binding,
including an owned self receiver. The path from the binding to the field may
pass through further class fields, as in outer.inner.resource. Availability
is then tracked per field, with the same rules as bindings:
- The moved field, and any place inside it, cannot be used until it is assigned.
- The other fields remain usable and can move out too.
- The binding cannot be used as a whole while a field is moved: it cannot be
moved, passed, returned, borrowed, or used as a method receiver. Assigning
the field (
value.field = replacement) or the whole binding restores it. - At a branch join, a field moved on some paths is possibly moved. A field moved in a loop must be assigned again before the backedge.
- When the binding goes out of scope, or a path returns early, only the fields still initialized are destroyed, each exactly once.
class Resource:
name: str
def __del__(self) -> ():
print("close", self.name).unwrap()
class Pair:
left: Resource
right: Resource
def consume(resource: Resource) -> ():
print("consume", resource.name).unwrap()
def main() -> ():
mut pair = Pair(Resource("a"), Resource("b"))
consume(pair.left)
print(pair.right.name).unwrap()
pair.left = Resource("c")
whole = pair
print("end").unwrap()
consume a
close a
b
end
close c
close b
Using the binding as a whole before the field is assigned is rejected:
class Resource:
name: str
class Pair:
left: Resource
right: Resource
def main() -> ():
pair = Pair(Resource("a"), Resource("b"))
left = pair.left
whole = pair
use of partly moved binding `pair`: `pair.left` is moved; assign `pair.left` before using `pair` as a whole
Moving out needs no mut binding, as moving the whole value does not, but
assigning the field back does. Fields cannot move out:
- through a reference, including bare
selfand&mutreceivers; usereplace, which transfers the old value while leaving the field initialized; - of a class that defines
__del__, which receives the whole object and so needs every field initialized;replaceworks there too; - through a
Box, aSharedhandle, a collection element, a closure capture, or a temporary.
Tuple elements move by unpacking the whole tuple, and enum payloads by
consuming match. Use an explicit copy or a collection removal operation for
collection elements. Two list elements can be exchanged in place with
swap.
Borrowing and reborrowing
&place creates a shared reference &T; &mut place creates an exclusive
reference &mut T. Exclusive access requires a mutable owner or an existing
exclusive reference. A shared loan permits other shared reads; an exclusive
loan prohibits competing reads, writes, and borrows. Neither acquires ownership.
&mut str permits replacing a string binding, not mutating string bytes.
A reference binding can be initialized from an explicit borrow, a named
reference, or a reference-returning call. Naming or passing an existing
reference reborrows it. An exclusive parent can lend shared or exclusive access;
conflicting parent access is prohibited while the child is live. A shared
reference cannot be upgraded to exclusive access. Reference bindings are
immutable unless declared mut; that declaration controls reassignment of the
reference, independently of the referent’s shared or exclusive access mode.
Loans are checked through branches, loops, and early exits and normally end at their last use. Moving, replacing, or destroying an owner conflicts with any live loan into it. References cannot be stored in aggregates or retained across generator suspension. General lifetime annotations are unsupported. Fields can move out of an owned binding; see moving fields.
Class fields and fixed tuple indices are projected places. Disjoint fields can
be borrowed independently; access to an ancestor or whole owner overlaps its
descendants. &items[index] and &mut items[index] borrow list elements or
dictionary values in named storage, including projected collections. Indices
and keys follow ordinary indexing, including negative list indices and traps
for absent entries. Element loans at distinct nonnegative integer literal list
indices, such as xs[0] and xs[1], are disjoint. Every other element loan
protects the whole collection: a variable, computed, or negative index, and
every dictionary key. Growth, removal, replacement, movement, and whole-list
access such as len(xs) or &xs overlap every element loan.
&xs[a:b] and &mut xs[a:b] borrow a run of list or byte-buffer elements as
a slice view, &[T] or &mut [T].
A view’s loan covers its run of elements; see
disjoint slice views for when two runs are disjoint.
Nested fields and collection elements can be projected through these references.
Borrowing itself allocates nothing.
Disjoint slice views
Two views of one owner are disjoint when the checker can prove their runs do not overlap for every owner length. It accepts two forms:
- One view stops where the other starts, at the same integer literal or the
same non-
muti64binding:&mut xs[:m]and&mut xs[m:], or&mut xs[:-2]and&mut xs[-2:]. - Both bounds compared are literals of the same sign, and one view stops at or
before the other starts:
&mut xs[0:2]and&mut xs[4:6]. An empty view such as&xs[3:1]overlaps nothing.
A list element at a literal index k is disjoint from a view that starts at a
nonnegative literal above k or stops at a nonnegative literal at or below
k. Nested views and rows of one view follow the same rules relative to that
view. Every other pair overlaps, including bounds of different signs, two
different names, a mut binding, and a computed expression such as m + 1.
The rule is sound because a view clamps each bound to the owner’s length when
it is created, and the length cannot change while a view is live. A
nonnegative bound v becomes min(v, len) and a negative one max(len + v, 0);
both are monotone, so equal or ordered bounds of one sign stay equal or ordered
after clamping. A shared bound name reads one value for both views: assigning
it while a view bounded by it is live is rejected.
def main() -> Result[(), AllocError | IoError | OverflowError[i64] | ZeroDivisionError]:
mut xs = [1, 2, 3, 4, 5]?
m = len(xs) // 2
left = &mut xs[:m]
right = &mut xs[m:]
left[0] = right[0] + right[1]
print(xs)?
Ok(())
[7, 2, 3, 4, 5]
def main() -> Result[(), AllocError | IoError]:
mut xs = [1, 2, 3, 4, 5]?
left = &mut xs[:3]
right = &mut xs[2:]
left[0] = right[0]
Ok(())
conflicting borrow
The diagnostic names both views and states the rule. Rewrite the bounds to meet at one name, or end the first view before taking the second.
Distinct fields of one borrowed loop element are disjoint, as they are for any
other reference. Under for p in &mut pairs, swap(&mut p[0], &mut p[1])
borrows two fields of the current element; under for n, s in &mut pairs,
n and s are such fields. Borrowing the list or another element while they
are live still conflicts, because the loop element’s index is unknown.
Integer node IDs do not borrow their target collection. They permit sequential updates to different nodes, but do not make simultaneous indexed loans disjoint. Use the arena lesson for owned storage with parent links and a traversal worklist.
An ordinary &T parameter implicitly borrows an owned argument: inspect(value)
and inspect(&value) lend the same owner. An &mut T parameter requires
&mut value, except when forwarding an existing exclusive reference. Methods
borrow their receivers according to their declared self parameter. Generic
calls and callbacks follow the same signature-based rules.
A class method with an exclusive receiver reserves its place before evaluating
arguments, then activates exclusive access just before entering the method.
The receiver and arguments still evaluate once, left to right. During argument
evaluation the reserved place permits shared reads, but cannot be moved,
replaced, or mutably borrowed. Thus buffer.fill(0, buffer.len(), 0u8) works
even when buffer is an &mut Bytes: the length is a completed value before
the exclusive call starts. Generic methods use the same rule. A shared
reference into the receiver that remains live as an argument still conflicts
with activation; a nested mutation of that receiver also conflicts. Bind a
nested mutation’s result to a local before the outer call when sequential
mutations are intended. Explicit function arguments such as f(&mut x, &x)
do not receive this implicit receiver reservation.
Shared call arguments may be literals or owned temporaries. They evaluate once
in argument order and stay alive through later arguments and the containing
full expression, including cleanup on ?. A reference returned into such a
temporary can be used within that expression, but cannot be bound, returned,
or retained by a scoped worker. Mutable temporary arguments and general
temporary lifetime extension are unsupported.
Reference binding reassignment
A mutable reference binding covers the place it first borrowed, its first place. A list element’s first place is the whole list. The binding may be assigned, with an unchanged type:
- a reference borrowed through itself: a field, element, box content, borrowed-match payload, or the result of a reference-returning call that received it. The binding’s loan continues.
- another reference into its first place that does not go through the binding,
such as
cur = headorbest = &items[i]. This retargets the binding: the loan it held ends at the assignment, and later uses hold the new loan.
A reference to a different owner, or to a place of the same owner outside the
first place (&p.right after &mut p.left), is rejected. The first place is
borrowed while the binding is live, so the owner cannot change between a
retarget and a later use of the binding. On a path that skips a retarget, the
old loan continues; a reborrow of the old target keeps its own loan.
class Node:
value: i64
next: Option[Box[Node]]
def total(head: &Node) -> Result[i64, OverflowError[i64]]:
mut sum = 0
mut cur = head
while True:
sum = sum + cur.value
match &cur.next:
case Some(next):
cur = next
case Nothing:
break
Ok(sum)
def main() -> Result[(), AllocError | IoError | OverflowError[i64]]:
chain = Node(1, Some(Box(Node(2, Nothing))?))
print(total(&chain)?)?
Ok(())
3
This traversal uses constant stack space and allocates nothing after constructing the chain. Because a reassignable reference can point anywhere within its first target, loans through it protect that whole target. Two such field loans are not considered disjoint. An exclusive binding cannot be accessed while a live child reference conflicts with it. Immutable reference bindings keep precise field loans.
Retargeting lets a search remember its best element without an index:
def longest(words: &list[str]) -> &str:
mut best = &words[0]
for i in range(1, len(words)):
if words[i].byte_len() > best.byte_len():
best = &words[i]
best
def main() -> Result[(), AllocError | IoError]:
words = ["fig", "banana", "kiwi"]?
print(longest(&words))?
Ok(())
banana
Because the old loan ends at a retarget, a collection can grow after the old target’s last use and before the binding is pointed at a new element:
def main() -> Result[(), AllocError | IoError]:
mut values = [1, 2]?
mut slot = &mut values[0]
*slot = 10
values.append(3)?
slot = &mut values[2]
*slot = 30
print(values)?
Ok(())
[10, 2, 30]
Returned references
A function or method returning &T or &mut T borrows its result from its
first reference parameter, the origin. A slice view
parameter, &[T] or &mut [T], counts as a reference parameter. A function without a reference
parameter cannot return a reference, and a mutable result requires an &mut
origin. Every return path must originate in the origin; later reference
parameters, local owners, and owned parameters cannot escape. Direct parameter
references, reborrows, field and element projections, borrowed-match payloads,
and forwarding calls are supported. Branches may use explicit returns;
conditional reference expressions are unsupported.
class Pair:
first: i64
second: i64
def first(pair: &Pair) -> &i64:
&pair.first
def main() -> Result[(), IoError]:
pair = Pair(3, 4)
value = first(&pair)
print(*value)?
Ok(())
3
Later reference parameters are ordinary inputs. Their loans end when the call returns, so a temporary key or a borrow of another owner does not restrict the result. For a method, the origin is its reference receiver:
class Scores:
names: list[str]
points: list[i64]
def points_of(self: &mut Scores, name: &str) -> &mut i64:
for i, candidate in enumerate(&self.names):
if candidate == name:
return &mut self.points[i]
&mut self.points[-1]
def main() -> Result[(), AllocError | IoError | OverflowError[i64]]:
mut scores = Scores(["ada", "bob"]?, [3, 5]?)
mut name = "bob"
points = scores.points_of(&name)
name = "eve"
*points = *points + 10
print(scores.points)?
print(name)?
Ok(())
[3, 15]
eve
This small lookup assumes equally sized, nonempty lists and uses the last score as its fallback. Enumeration avoids manual index arithmetic while keeping the reference-returning method infallible. A general lookup can return an optional reference instead.
To return a borrow of a different argument, make it the first reference
parameter. A function cannot choose between two arguments; that relationship
needs annotations Plenty does not have. Signatures alone fix the origin, so
Callable types and generic functions follow the same rule.
The result extends its origin loan until its last use. A body that directly returns its parameter or a fixed field projection permits a precise returned field loan; the original call still requires access to its whole argument. Indexed origins, control-dependent returns, and other bodies conservatively protect the whole argument, even after further projection.
A &mut T result can be assigned through directly, without a binding:
*f(&mut xs) = value and *f(&mut xs) += value call f once and write its
referent. The call runs first, as in r = f(&mut xs); *r = value, so its loan
is live through the right-hand side: *f(&mut xs) = xs[1] is a conflicting
borrow. Read the value into a local first.
def head(xs: &mut [i64]) -> &mut i64:
&mut xs[0]
def main() -> Result[(), AllocError | IoError | OverflowError[i64]]:
mut xs = [1, 2, 3]?
*head(&mut xs[:]) = 9
*head(&mut xs[1:]) += 10
print(xs)?
Ok(())
[9, 12, 3]
A caller may bind a result from named storage and reborrow it. A temporary
origin survives only its full expression, the statement or condition that
evaluates it. A shared temporary passed to an ordinary function follows this
rule: len(view("text")) can use a returned &str, but r = view("text") is
rejected even when r is unused. A temporary method receiver, such as
make().label(), follows the same rule: the result may be used within the
statement, and the receiver drops at its end. Binding the result, or returning
it, is rejected:
class Point:
x: i64
name: str
def label(self) -> &str:
&self.name
def make(name: str) -> Point:
Point(0, name)
def main() -> Result[(), IoError]:
print(make("origin").label())?
print(len(make("east").label()))?
Ok(())
origin
4
A temporary receiver that itself borrows other storage, such as a returned
&Point in pick(&a).view(), cannot lend a returned reference: its loan of
that storage would end before the result is used. Bind the receiver first,
p = pick(&a), then call p.view().
A method on an element or field of a temporary, as in
[make("a")]?[0].label() or holder.items()[0].view(), borrows that element
in place. An owned temporary lives to the end of the statement, as a
temporary receiver does; a returned reference lends as a named reference does.
Returned references cannot enter aggregates, scoped workers with temporary origins, or generator suspension. No lifetime annotation or callee-body inference changes the signature’s first-reference origin.
A borrow that may be absent or may fail is returned as Option[&T] or
Result[&T, E] under the same origin rule; see
optional and fallible references.
Optional and fallible references
A function may return Option[&T], Option[&mut T], Result[&T, E], or
Result[&mut T, E]: a borrow that may be absent or may fail. The success
payload may also be another such enum, as in Result[Option[&T], E]. These
reference-carrying results follow the
returned-reference rule: the payload borrows the
first reference parameter, a function without one is rejected, and
Option[&mut T] requires an &mut origin.
class Item:
name: str
count: i64
def find(items: &mut list[Item], name: &str) -> Option[&mut Item]:
for item in items:
if item.name == name:
return Some(item)
Nothing
def main() -> Result[(), AllocError | IoError | OverflowError[i64]]:
mut items = [Item("ada", 1)]?
match find(&mut items, "bob"):
case Some(item):
item.count += 1
case Nothing:
items.append(Item("bob", 1))?
match find(&mut items, "ada"):
case Some(item):
item.count += 1
case Nothing:
pass
print(items[0].count)?
print(len(items))?
Ok(())
2
2
Nothing, Err(e), and Some or Ok around such a value hold no loan and
are always accepted. Every other return path must borrow the origin, as for
&T results. Err payloads cannot hold references: copy the part of the input
that the error reports.
Using the result
The caller’s origin loan lasts until the last use of the result or of any
reference taken from it. A Nothing or Err arm holds no payload, so it may
modify the origin. These forms take the reference out or inspect the result:
match resultbinds the payload reference directly;match &resultis rejected because the result already holds a reference.?propagatesNothingorErrand produces the reference.unwrap(),ok_or(e),is_some(),is_none(),is_ok(), andis_err().- Returning the result from a function with the same origin, directly or from a local.
- Printing and
str.repr, which show the referenced value:Option[&str].Some("ada").
unwrap_or(default) is rejected: it would choose between two borrows. Use
match.
Holding the result
A result can be bound to an immutable local. The local is a holder, not
storage: like a reference binding, it extends the origin loan to its last use.
Copying it shares that loan. A reference-carrying binding must hold a borrow,
so o: Option[&str] = Nothing is rejected.
A reference-carrying value is never stored. Tuples, lists and other
collections, class fields, enum payloads, generic type arguments, closure
captures, scoped workers, and values live across yield all reject it. Match
it first and store an owned copy or an index instead.
Current limits
mutlocals and parameters of a reference-carrying type are rejected. Pass the reference after matching.- A
matchused as a value cannot produce one. - Patterns cannot match nested variants through the carried reference:
case Some(Some(v))onOption[&Option[i64]]needs a secondmatch. - A returned reference from a temporary argument follows the returned-reference rule: the result cannot be bound to a local.
- Standard library lookups such as
dict.getstill return copies or trap; borrowed optional lookup methods are planned work.
Borrowed matching
match &value, or matching an existing &E, inspects an enum without consuming
it. Named payload bindings have type &T, even for scalar or unit payloads.
match &mut value, or matching an existing &mut E, binds &mut T payloads.
_ creates no binding and moves no borrowed payload. _ as name binds &E or
&mut E: a reborrow of the whole scrutinee, whose loan protects all of it. The scrutinee evaluates
once; patterns, coverage, and arm scope follow ordinary
matching. A binding inside a nested pattern is a
reference to that inner payload, through boxes as well, and a name bound by
p | q refers to the payload of whichever alternative matched.
Mutable matching supports standard sums and user enums, both copyable and affine. Mutating a copyable enum’s inline payload leaves independent copies unchanged. Matching and its payload borrows allocate nothing.
Payload loans descend from the scrutinee loan and protect the active variant against conflicting access, replacement, movement, or destruction. A nested binding’s loan also protects each enclosing variant. Disjoint payload fields can be borrowed mutably together; precise class-field origins remain precise, while collection and imprecise returned origins protect their whole target. Reading a discriminant conflicts with an existing exclusive loan into the enum.
Bindings exist only in their arm. Last-use checking permits replacement after
the final payload use, even within that arm. No live payload loan may cross
yield. Payload references can be returned under the
single reference-parameter rule; the returned
payload conservatively protects the caller’s whole argument. Borrowing a
temporary, returning a local owner’s payload, and storing payload references
in aggregates are rejected. A mut reference binding can be reassigned to a
payload borrowed from itself; the arm’s payload bindings are immutable.
A returned Option[&T] or Result[&T, E] already holds a reference: match it
by value, not with match &result. Its payload bindings are the carried
reference and extend the call’s origin loan; see
optional and fallible references.
Deterministic destruction
Live owned locals drop at lexical scope exit in reverse binding declaration
order, with inner scopes before outer scopes. return, ?, break, and
continue clean up the scopes they leave. Moving transfers the cleanup
obligation; moved or already dropped values are not dropped again. Replacement
destroys the previous initialized value before installing the new one; see
assignment for evaluation order.
drop(value) -> () consumes an owner early. It cannot destroy a borrowed
referent or consume an owner during a conflicting loan. Dropping a shared
immutable string releases that owner’s share, without freeing other owners’
storage. Ending a borrow at last use does not move observable resource
destruction away from its specified scope exit or explicit drop.
Owned expression temporaries drop at the end of their full expression unless transferred. A block transfers its result before destroying its other locals. Loop conditions and each comprehension iteration finish their own temporaries. An early propagation cleans pending evaluated operands before enclosing scope cleanup; later operands do not run.
A class may declare def __del__(self) -> () (an exclusive receiver) for
additional cleanup. The hook runs before automatic field cleanup. It cannot
return a recoverable error, yield, allow references to the dying value to
escape, or be called as an ordinary method. Because the hook receives every
field, fields of such a class cannot
move out. Fields clean up
automatically, skipping any field that has moved out:
| Owner | Cleanup order |
|---|---|
| Class fields and active enum payload fields | Declaration order |
| Tuple components | Component order |
| List elements | Index order |
| Dictionary entries | Insertion order, each key before its value |
| Closure captures | Reverse capture order |
A child finishes destruction before its next sibling begins. Recursive heap
cleanup requires no allocation or recursive native stack growth; source-level
recursive calls inside user hooks can still use the native stack. User cleanup
retains its own effects and allocation costs. Types with custom destruction,
including nested fields, cannot be copied. Explicit fallible shutdown should
return Result; a destructor provides fallback cleanup rather than reporting
shutdown failure.
Bounded-stack destruction does not make a recursive traversal bounded-stack. An explicit descendant cursor or worklist is still needed for unbounded-depth input. See linked-data examples.
Suspended or never-started generators drop their remaining captures and locals
without resuming the body. Statements following yield are not cleanup hooks.
Resource scopes insert exit calls into
ordinary cleanup. Tail transfers finish caller
cleanup before entering the callee. Fatal traps and process aborts do not unwind.
Functions and callable values
Functions declare complete signatures. Named functions and capture-free
anonymous functions have Callable types; explicit captures create concrete
closure environments. Reusable calls borrow an environment, while consuming
calls transfer it into the body.
- Signatures, methods, and calls
- Named and anonymous function values
- Closure environments
- Consuming and stored closures
- Boxed callable environments
- Recursion and tail calls
Generic constraints allow one callback API to accept different concrete callable representations without erasing their types.
Signatures, methods, and calls
Named functions are declared at module scope or as class or enum methods. Every
parameter and result has an explicit type, except the receiver shorthand
self. All signatures are available before bodies are checked, permitting
forward calls and mutual recursion. Nested named functions, overloads, and
redefinitions are unsupported.
def clamp_low(value: i64, minimum: i64) -> i64:
if value < minimum:
return minimum
value
def main() -> Result[(), IoError]:
print(clamp_low(2, 5))?
Ok(())
5
The final expression supplies the function’s result. return value exits from
any suite; bare return supplies unit. Every return must match the declared
type, and every continuing path must produce that result. The
branch and return rules define how continuing
paths join and unreachable statements are rejected.
A string literal that begins a body is the function’s documentation, not its
final expression. A str function whose body is only that string is rejected
with this explanation; write return "text" to return the string.
Calls evaluate the callee once, then arguments once from left to right.
Parameter types supply expected types, including numeric literal context.
Owned arguments copy or move; reference arguments follow
borrowing. A later argument’s ? cleans up
already evaluated arguments without entering the callee.
Named functions, class and enum methods, and class constructors accept
name=value arguments.
Positional arguments must come first; named arguments can follow in any order.
Unknown names, duplicate bindings (including a positional argument plus its
name), and missing required arguments are errors. Names bind statically; no
runtime dictionary is constructed. Supplied expressions still run in written
order, with the expected type of the parameter they name. self is supplied
by the method receiver and cannot also be named. Parameter names are part of
the source API.
Parameters may have defaults: def f(x: i64, scale: i64 = 2) -> i64:.
Required parameters must precede defaulted parameters. Each omitted expression
is evaluated afresh, after all supplied arguments, in declaration order. An
explicit value, including Nothing, skips that default. Names in defaults
resolve in the defining module; defaults cannot depend on parameters or self.
Concrete declarations are checked even when their defaults are never omitted;
generic defaults are checked on specialization.
Owned defaults produce fresh owners. For a borrowed parameter, the default
produces an owner of the referenced type and lends it for the call. For example,
values: &mut list[i64] = []? creates a new list for each omission. It requires
a compatible Result return type: allocation failure is explicit. ? inside
a default produces an error from the called function, not its caller. Evaluated
owners are cleaned up and the body is skipped on failure. Supplying every
argument does not change the return type. A returned reference cannot borrow
an omitted temporary; pass caller-owned storage instead.
Calls through callable values retain full positional arity. Builtins and enum payloads remain positional. Protocol requirements cannot declare defaults; named method calls in protocol-constrained generic bodies are currently rejected conservatively, including calls whose receivers happen to have a concrete type. Those bodies must also supply every method argument: an implementation’s defaults are not part of its protocol contract. Generic parameters must be inferable from supplied arguments or passed explicitly; omitted defaults do not drive type inference. Anonymous functions and C import/export declarations do not accept parameter defaults. No splat unpacking or variadic parameters are supported. Field defaults are separate from parameter defaults.
A method’s first parameter is self. Bare self borrows the enclosing class
or enum immutably in ordinary methods. Class lifecycle methods __init__ and
__del__ instead borrow &mut Class; enums have no lifecycle methods.
Mutating ordinary methods declare self: &mut Class (or the enum type);
their receiver must be a mutable owner or exclusive
reference. Methods resolve statically. Temporary
owned receivers are supported when no reference escapes. Attempting mutation
through an ordinary method’s bare self reports the explicit
self: &mut Class annotation needed for an exclusive receiver. For a generic
class, use its declared type parameters in that annotation, such as
self: &mut Cell[T]; the diagnostic points to these parameters rather than a
particular specialization. See
classes for initialization and field rules and
returned references for receiver
lifetime restrictions.
An explicit self: Class (or self: Enum) is a by-value receiver. Calling
value.finish(args) transfers an affine receiver just like passing it as the
first owned argument to a function; the old binding is unusable even when the
method returns an error. Copyable enums retain ordinary copy semantics. Generic
owners can use self: Cell[T] or the current-owner shorthand self: Cell.
There is no dynamic dispatch or allocation introduced by this call syntax.
The receiver is evaluated before the remaining arguments, including named arguments. Once moved, it cannot be used in a later argument; save needed values before the call. If a later argument propagates an error, the pending owned receiver is cleaned up. Two-phase mutable-receiver reservation does not apply. The callee owns cleanup unless it returns or otherwise transfers the receiver. To support retry, return the owner in the error payload explicitly.
Owned self is an immutable binding, like any value parameter; use mut owned = self to modify it. Consuming methods do not permit moving individual class
fields, consuming through an affine reference or indexed element, or returning
a borrow of the consumed owner’s storage. Constructor/destructor receivers and
operator-hook contracts are unchanged. See the
consuming-method lesson.
Named and anonymous function values
A named function can be assigned, stored, passed, and returned. To accept a
callback, take a type parameter bound by Callable and a parameter of type
&F, or &mut F when the callback changes a capture:
def apply[F: Callable[[i64], i64]](f: &F, value: i64) -> i64. This form
accepts named functions, capture-free anonymous functions, and
closures; see generic functions.
Parameter names do not affect a signature. Callable[[], T] takes no
arguments, and Callable[[T], ()] returns unit. Aliases may name these
signatures.
Callable[[parameter types], result] is also a value type: a code address with
no environment, the type of a named function or capture-free anonymous
function. A capturing closure also needs an inline environment, so a concrete
Callable parameter or field cannot hold one; use the bound form above, or a
boxed callable for storage. Function values are
copyable, display as <function>, and compare native code addresses rather
than behavior. They cannot be dictionary keys or set elements. A payload enum
variant written without arguments, such as AppError.Io, and a class
constructor, such as Point, are also function values; see
enums and classes.
Generic constructor values specialize from an explicit data instance or an
expected Callable, including prelude Some, Ok, and Err. Like named
functions, constructor values can be borrowed as &Point or &Reading.Value;
&mut borrows a fresh copy of the code address.
An arbitrary callee expression is allowed: choose()(n), (f)(n),
handlers[index](n), and record.callback(n). The callee evaluates before its
arguments. Brackets following a local binding or a
module constant index it; brackets following a
function declaration or module path provide type arguments. Following a field
or method on a value, brackets index when their contents start at a local
binding and otherwise provide method type arguments.
Take a generic function value with explicit specialization, such as identity[u8],
or let a concrete expected Callable from an annotation, parameter, or return
specialize the bare name. Both input and result positions provide evidence;
unconstrained type parameters need explicit arguments. Signatures do not
implicitly convert, and this does not introduce higher-rank polymorphism.
Generic inference also works inside callable
inputs and results.
An anonymous function is def(parameters) -> T: followed by a body. Its
position fixes the form of the body: outside brackets the body is an indented
block; inside (), [], or {} it is one expression, whose value is returned.
def apply[F: Callable[[i64], Result[i64, OverflowError[i64]]]](f: &F, value: i64) -> Result[i64, OverflowError[i64]]:
f(value)
def main() -> Result[(), IoError | OverflowError[i64]]:
twice = def(value: i64) -> Result[i64, OverflowError[i64]]:
Ok(value * 2)
print(twice(3)?)?
print(apply(&(def(value: i64) -> Result[i64, OverflowError[i64]]: Ok(value + 1)), 3)?)?
Ok(())
6
4
The one-expression body is the final expression of a block, so both forms create the same value and follow the same type, capture, and ownership rules. Only the layout differs:
- The expression extends as far as it can and ends at the
,or closing bracket of the enclosing element. Line breaks inside brackets do not end it. - Statements have no one-expression form:
return, a binding, an assignment, or a second expression is rejected. Give such a function an indented body and bind it to a name first. - A string body is the returned value, not documentation.
- Outside brackets a body on the
defline is rejected. Parentheses are brackets:twice = (def(value: i64) -> Result[i64, OverflowError[i64]]: Ok(value * 2)).
In both forms an anonymous function must be a complete expression: a bound or
returned value, an argument, an element, or the contents of parentheses. As the
operand of an operator, including & and &mut, it needs parentheses:
&(def [offset](value: i64) -> Result[i64, OverflowError[i64]]: Ok(value + offset)). Parameter and result types
are always written. lambda is reserved and rejected with the def spelling.
Capture-free anonymous functions have ordinary Callable types, can nest, and
can be returned. Their parameters and locals form a separate scope; enclosing
generic type parameters are substituted. Implicit capture of an enclosing
local, including one shadowing a module function, is rejected. Use an explicit
capture list for environments.
Callable signatures exclude generator frames and closure environment values.
A named function can be borrowed directly: apply(&double, value) or
callback = &double. Each borrowed declaration or explicit specialization,
such as &identity[u8], has one immutable code-address slot in the borrowing
function, initialized at entry without allocation. Its references follow the
ordinary local borrow and escape checks. A local binding shadowing a function
name takes precedence. A generator can borrow one for a call, but a reference
to that local slot cannot remain live across yield.
Generic reference bindings require explicit specialization;
a concrete callable parameter can instead supply the expected signature.
&mut double borrows a fresh copy of the code address, so a named function also
passes to a &mut F parameter; distinct &mut borrows of one declaration do
not conflict because a function has no state.
Callable constraints, as distinct from storage types, accept reusable named
functions and concrete captured closures.
Closure environments
def [captures](parameters) -> T: creates a reusable concrete environment.
Captures are explicit; no implicit deep copy occurs. The body takes either
anonymous function form: an indented block outside
brackets, one expression inside them.
| Capture | Access inside the body |
|---|---|
value | Own the value; read or borrow it |
mut value | Own private mutable state |
&value | Share a surrounding binding |
&mut value | Exclusively borrow a mutable surrounding binding |
Scalars copy into captures, immutable strings keep their sharing semantics, and affine owners move. An owned capture’s source need not be mutable; a borrowed mutable source must be. Captures cannot be redeclared or shadowed by loop, comprehension, unpacking, or match bindings inside the body.
def counter(start: i64) -> Closure[[], Result[i64, OverflowError[i64]]]:
def [mut start]() -> Result[i64, OverflowError[i64]]:
start = start + 1
Ok(start)
def main() -> Result[(), IoError | OverflowError[i64]]:
mut count = counter(2)
print(count()?)?
print(count()?)?
Ok(())
3
4
Every closure has its own declared arithmetic contract, independent of where
it is defined or called. A checked: block
or unchecked: suite around its definition does not change its body. There is
no private callback error channel or checked-block lifetime restriction; normal
capture ownership and borrow rules still apply. Callers handle a closure’s
Result explicitly, just as they handle a named function’s result.
Calls borrow the environment and may repeat. Mutable captures require an
exclusive environment borrow and a mutable closure binding or owner. Captured
owners can be borrowed and mutated as permitted but cannot be moved out; use
a consuming closure when ownership must leave the body. Assigning a closure
moves it; copy and equality are unavailable. Display is <closure>.
Closure[[parameter types], result] describes a concrete environment’s
signature. Its producer identity is retained: different closure expressions
do not unify merely because their signatures match. Factories and generic
consumers preserve that concrete layout. Prefer F: Callable[...] with &F
or &mut F for reusable callback APIs accepting both named functions and
closures; the generic constraint is separate from a code-only Callable value.
Diagnostics identify a concrete closure by its definition’s source file, line,
and column (line and column for source supplied without a filename). This
location describes its origin; different generic instantiations of the same
expression can still have distinct environment types.
Creating, moving, passing, returning, calling, and dropping inline environments
adds no heap allocation. Captured owners drop in reverse capture order.
Owned environments can capture other owned environments, ranges, and standard
sums, and can travel through Option/Result or suspended generator frames.
Reusable closures cannot capture generator frames. Body operations and capture
expressions retain their own allocation and error contracts.
Capture loans follow moves and remain live through environment borrowing and later argument evaluation. Shared captures prohibit owner mutation or destruction; exclusive captures also prohibit competing reads. Borrowing environments can be passed by reference, including through generic callbacks, but cannot cross a function boundary by value, escape their creating function, own-nest, enter sum/container storage, or suspend in generators. Borrowing another environment retains its transitive loans.
Calls can target temporaries, such as make(3)(4) or wrapped.unwrap()(4);
the callee survives the full expression and cleans up if an argument propagates.
Capturing and one-shot closures cannot return references or yield.
Environment nesting is limited to 64 levels and must fit the native stack
layout limit. Closure calls do not perform native tail transfers.
Consuming and stored closures
def once [captures](parameters) -> T: creates an affine one-shot environment.
Calling consumes it and transfers owned captures into the body, where they can
be moved into the result. mut value permits changing a capture during that
call; the closure binding itself need not be mutable. Even a capture-free
def once (...) is affine. Calling twice, through a reference, or through a
reusable Callable constraint is rejected.
OnceClosure[[inputs], result] describes its concrete signature and producer
identity. It does not unify with reusable Closure. The generic constraint
F: OnceCallable[[inputs], result] accepts named functions, reusable closures,
and one-shot closures. A consuming callback API takes F by value and may
bind a mutable local to accommodate mutable reusable closures. Specialization
still checks concrete move rules: calling a one-shot argument twice is rejected.
OnceCallable is a constraint, not a storage type.
A function returning OnceClosure or Closure, alone or inside a sum such as
Result, takes the concrete environment from a closure value it returns. The
signature alone does not specify the captured environment, so a body that
returns no closure value on any path, such as one that only returns Err, is
rejected with cannot infer the concrete one-shot closure returned by this function (or closure for a reusable one).
? returns its error, so it also resolves a closure signature in the declared
error, including inside a retained-job error such as SpawnError or
SubmitError and as an anonymous sum member.
The first closure returned or propagated fixes that position’s producer;
another producer is rejected. Several closures from one factory function share
its producer. To propagate distinct jobs, drop_job()
converts each error to the job-free SpawnError[()] or SubmitError[()]. A member pattern with the same signature, such as
case failed: SpawnError[OnceClosure[[], i64]]:, names the resolved member.
Only declared return types infer this way: a closure signature in a parameter’s
or local binding’s sum is rejected.
Uncalled environments drop captures in reverse order. After entry, the body
owns cleanup: moved result captures survive and the rest drop on exit. Argument
failure drops the pending environment without entering the body. One-shot
environments can own and resume concrete generator frames; mut source
permits advancing a captured frame. Creation, movement, invocation, and cleanup
of the inline environment allocate nothing.
One-shot borrowed captures permit a local consuming call. Their loans remain live through argument evaluation and invocation and may end afterward. Such environments cannot escape, pass by value across functions, own-nest, enter sums/containers, or suspend in a generator. Passing one by reference does not grant permission to consume it. Reference returns, yielding, and capture shadowing have the same restrictions as reusable closures.
Wholly owned concrete environments can be stored in generic class fields, tuple components, list elements, dictionary values, and generic enum payloads, including through standard sum wrappers. A signature annotation alone does not resolve a storage layout. Different tuple slots or enum variants may hold different producers; a list’s elements share one concrete producer type. Borrowed captures and environments containing generators cannot enter heap storage. A rejection names its cause: an unresolved signature, a borrowed capture, a captured generator, a reference, or a generator. Classes and enums themselves are inline; their construction adds no environment allocation. Collection construction and growth remain fallible. For different shared-call producers behind one signature, use an explicitly allocated BoxedCallable.
Calling a stored reusable callback borrows its field or element, exclusively for mutable captures. A live indexed callback loan protects the collection against invalidating mutation. Moving, replacing, and dropping storage retain ordinary ownership and cleanup rules. Values containing environments cannot be copied or compared; environments cannot be dictionary keys or set elements.
A stored one-shot callback must first be extracted as an owner, using pop,
consuming iteration or matching, or tuple unpacking. Calls through references
or indexed places cannot consume it. A class field can own such a callback;
move the field
out of an owned binding to call it. Invocation, extraction,
and cleanup add no allocation; growing its collection can allocate.
Boxed callable environments
BoxedCallable[[inputs], result] owns an explicitly allocated callback with a
structural signature. Unlike concrete Closure, its type does not include the
producer or capture layout. Different owned environments with the same signature
can share list elements, dictionary values, class fields, enum payloads, and
factory return types. Named Callable values can use the same owner type.
BoxedCallable(callback) consumes the callback and returns
Result[BoxedCallable[[inputs], result], AllocError]. Construction always attempts
one allocation, even for a named function or capture-free closure. There is no
implicit conversion or allocation when assigning or returning a callback. On
allocation failure the consumed environment and its captures are dropped.
The initial contract accepts reusable shared-call callbacks with wholly owned
captures. Borrowed captures, generator-containing environments, mutable closures,
one-shot closures, and callbacks returning references are rejected. Signatures
otherwise use the same rules as Callable, including reference parameters and
ordinary owned arguments, with at most 255 parameters plus the hidden environment
slot. A BoxedCallable satisfies a matching generic
Callable constraint; its affine ownership remains visible to specialization.
Invocation borrows the owner immutably, including through shared or mutable references, fields, and collection indexes. The binding need not be mutable. Temporary owners survive argument evaluation and the call, then drop normally. Live borrowed callbacks prevent moves, replacement, removal, and invalidating collection changes. Moving a box transfers its sole owner; copying, equality, membership equality, hashing, and use as a dictionary key or set element are unsupported, as for concrete closure environments.
Calls, moves, and cleanup allocate nothing beyond allocations explicitly
performed by the callback itself. The heap box retains its concrete environment
descriptor for relocation and destruction, together with a compiler-generated
adapter address. Each capture is dropped exactly once when its owner is replaced
or destroyed. Calls use that adapter and the public signature, so the caller
does not need the concrete capture layout. Formatting shows <boxed callable>.
Erasure does not certify captures or indirect effects for concurrency. Boxed callbacks, including nested fields and container elements, cannot cross worker or channel boundaries. Retain a concrete environment when worker eligibility must be checked. Mutable and consuming erased modes require a separate contract.
Recursion and tail calls
A source-tail call supplies the function’s result without further work. This
includes final expressions, explicit returns, selected conditional-expression
branches, tail if/match arms, the arms of return if, the cases of
return match, and result branches of and/or. An if or match block
bound to a local or used as an assignment’s value is not a tail position.
Arithmetic,
wrapping in Ok or Some, conversion, or ? after a call makes it non-tail.
Evaluating arguments before the call does not make it non-tail.
For a direct source-tail call within its caller’s direct recursion cycle, compilation requires an approved native tail transfer. Otherwise it reports the callee and the lifetime, resource-scope exit, or representation restriction at that call. Cycles are computed after concrete generic specialization and include both tail and non-tail direct edges. Different specializations are different nodes. A cycle that also has non-tail edges has no constant-stack guarantee.
A transfer evaluates arguments left to right, then drops pending temporaries
and the caller’s remaining owned parameters and locals in ordinary exit order,
then enters the callee. Moved arguments belong to the callee. Destructors on
unmoved locals run before the callee body and do not by themselves prevent a
transfer. An argument’s ? takes an ordinary error exit instead.
These restrictions retain the caller for an ordinary call:
- Any reference argument into the caller’s locals, owned parameters, or temporaries, including a receiver in that storage.
- A pending
withexit action after the call. - An owned inline argument, such as a range, record, closure, or generator frame.
- A result requiring conversion, wrapping, propagation, or incompatible result storage.
Nonrecursive ordinary fallbacks are legal. Reference arguments originating in the caller’s reference parameters may be forwarded, including reborrows, projections, borrowed-match payloads, borrowed iteration items, and forwarded reference results. Their owners belong to an older caller. Exact-compatible inline results can also transfer, including ranges, records, nested sums, and concrete generator frames. No hidden heap allocation extends argument lifetime.
Indirect Callable calls follow the same transfer restrictions but are outside
the direct recursion guarantee. Concrete closure invocation, worker starts,
generator construction, and generator bodies are outside that graph. Ordinary
recursive calls during generator resumption may consume stack. Explicit
direct-call edges in generated adapters participate; an indirect call into an
adapter does not create a direct edge. See the
tail-call ABI for implementation.
Generics and constraints
Functions, classes, enums, and methods can declare type parameters. They also
accept value parameters, such as const N: i64, for compile-time
integer and boolean specialization.
Type parameters are written as follows:
def identity[T](value: T) -> T:
value
def add[T: IntType](a: T, b: T) -> T:
a + b
Concrete uses preserve ordinary type and ownership rules. Generic bodies are checked for each used concrete instance. Calls remain statically dispatched; generic syntax adds no runtime type lookup or allocation. The toolchain documents specialization limits and caching.
Functions and methods accept unconstrained parameters, IntType, FloatType,
CopyType, KeyType, Dimension, structural protocol constraints, and callable constraints.
Dimension accepts compile-time dimensions, which
have no values. IntType accepts the
eight fixed-width integer types and FloatType accepts f32 and f64,
including aliases; each is a constraint rather than a value type. No constraint
admits both integers and floats; an unconstrained parameter does, because a
body is checked for each instance. Callable capabilities follow functions.
KeyType
KeyType accepts exactly the supported dictionary keys and set members:
integers and their supported domains, bool, str, bytes, and tuples,
classes, enums or anonymous sums whose every field, payload or member is a supported key type.
Floats, collections, boxes and recursive types remain excluded. Custom cleanup
is allowed, but copying operations still reject such keys. It is a built-in constraint, not a protocol or a stored value type;
no __hash__ method is needed or accepted.
Functions, methods, classes and enums can declare K: KeyType. Explicit and
inferred arguments are checked against the same admission rule as dict[K, V]
and set[K]. This makes the requirement visible and reports unsuitable
arguments at the generic boundary, even when its body does not hash anything.
Generic bodies remain checked per concrete instance: an unconstrained K
can still be used with collections when its concrete type is valid.
def distinct[K: KeyType](values: list[K]) -> Result[set[K], AllocError]:
mut result: set[K] = set()?
for value in values:
result.add(value)?
Ok(result)
def main() -> Result[(), AllocError | IoError]:
print(len(distinct(["red", "blue", "red"]?)?))?
Ok(())
2
KeyType implies neither CopyType nor eligibility for explicit copy().
The loop consumes its input and transfers each key, so it also works for keys
with custom cleanup. A generic body that copies keys or calls non-mutating set
algebra is rejected when specialized with such a key type.
def accept[K: KeyType](value: K) -> ():
pass
def main() -> ():
accept(1.5)
f64 does not satisfy KeyType
Finding an unsupported key field
When a collection or KeyType constraint rejects a structured key, the
diagnostic follows the first unsupported component in declaration order.
It names class fields, enum variants and payload positions, and tuple element
positions. Positions start at one. For example, a class field sample whose
second tuple element is f64 reports field sample -> tuple element 2: f64 is not a supported key type. Unsupported sum members
are named in the diagnostic path. This applies to type annotations, inferred collection
literals and conversions, and explicit or inferred generic arguments.
CopyType
CopyType accepts every type argument that copies under the
copy rule: numbers, bool,
str, bytes, ranges, NonZero, unit (), named function values, tuples,
enums, and anonymous sums whose parts copy, and @copy classes whose type
arguments copy. Closures, collections, boxes, and other classes do not satisfy
it. Type arguments are never references, so &T never satisfies it; a
&T parameter borrows a T that does.
def twice[T: CopyType](value: T) -> (T, T):
(value, value)
def main() -> Result[(), IoError]:
print(twice(3), twice("a"), twice((1, Some(2.5))))?
Ok(())
(3, 3) ("a", "a") ((1, Option[f64].Some(2.5)), (1, Option[f64].Some(2.5)))
def twice[T: CopyType](value: T) -> (T, T):
(value, value)
def main() -> Result[(), AllocError]:
pair = twice([1]?)
Ok(())
list[i64] does not satisfy CopyType: it moves on ordinary use
A body is checked per instance, so twice with an unconstrained T also
compiles for these calls. The bound states the requirement in the signature
and reports a moving argument at the call. Functions, methods,
classes, and enums accept it.
Argument inference determines parameters from inputs or a complete explicit argument list. Generic data and methods retain nominal identity. Protocols describe exact structural interfaces without inheritance or registration.
Generic function type arguments can be unit (), including when inferred from
Result or Option payloads. Substituted signatures and bodies retain ordinary
unit restrictions: unit can be a return type or built-in enum payload, but not
a direct function parameter or stored collection element. A &T or &mut T
parameter instantiated with () borrows unit, so a helper such as
std.test.is_err accepts a Result[(), E]. Explicit calls and specialized
function values also accept ().
Generic function type arguments cannot be references. A signature can borrow
T with &T or &mut T. A callable-constrained parameter can borrow a
concrete closure while keeping its capture loans live across the call. Data and
protocol arguments have their additional restrictions on their own pages.
Diagnostics from instances
A generic body is not checked where it is declared. Each used instance is checked once, and an error found in it is reported at the use that first required the instance, in code that is not itself a generic instance. A note gives the position in the definition, the instance being checked, and the reason:
main.plenty:7:9: cannot instantiate `outer[i64]`: its definition is rejected
pairs.plenty:5:13: note: in instantiation `pairs.scaled[i64]`: unknown binding `pairs.factor`
The first line names the instance the use asked for. The note names the instance whose definition failed; the two differ when generic code requires further instances, and the instances between them are not listed. Further notes belong to the reason. Names longer than 96 characters are cut short.
When the body is rejected only because of a type argument, the first line names that argument instead. Three causes are recognized: an implicit copy of an affine argument (moving it out of a field, an element, or a reference, or using a moved binding of it), an operator the argument does not support, and a method the argument does not have:
main.plenty:8:9: `outer[list[i64]]` requires `pairs.twice[list[i64]]`, which copies `T`, but `list[i64]` cannot be copied
pairs.plenty:2:13: note: in instantiation `pairs.twice[list[i64]]`: use of moved binding `value`
pairs.plenty:2:6: note: `value` is moved here
The note then describes the rejected operation without advice for changing the
body, which the caller may not own. A cause is attributed when the rejected
value comes from a type parameter and its type is exactly one parameter’s
argument. Equal concrete types alone do not establish that origin: an inferred
row = [1]? is not a T value merely because T = list[i64]. Conversely,
an unrelated concrete annotation does not hide a failure on an actual T.
The diagnostic tracks parameter origins through local aliases, borrows,
collection displays, and field or element projections whose declared shape
identifies the parameter. Projecting inside an opaque T does not establish
the origin of its contents. Attribution is conservative: mixed assignments,
untracked transformations such as call results, or two parameters with the
same concrete argument retain the first form. These rules affect diagnostic
wording only, not which programs are accepted.
This covers generic functions, generic methods, functions written inside them,
and the methods of a generic class or enum instance. A method of an instance is
checked when it is first used: called, including by a constructor call for
__init__, or required by a protocol bound that the instance
satisfies. A __del__ method is checked when the instance is first required,
because constructing a value arms it. An instance can therefore exist although
one of its methods would be rejected for it, as long as that method is not used:
main.plenty:9:13: `Cell[list[i64]].get` copies `T`, but `list[i64]` cannot be copied
main.plenty:5:9: note: in instantiation `Cell[list[i64]].get`: cannot move out of a field
A method required through a protocol bound is reported at the bounded use,
naming the bounded instance, such as accept[Holder[Guard]]. Unused methods are
not compiled for the instance either.
Since bodies are checked only per instance, a body error that no type argument causes, such as an unknown name, is reported in the first form, and not at all while the declaration is unused. Errors in the declaration itself (type parameters, constraints, signature types) and uses that do not satisfy a constraint are reported at one location. Instance field types, constraints on data arguments, and substituted method signatures are checked when the instance is first required.
Checking every method
Checking per use keeps compilation fast while code changes, but a library’s
tests then check a method for a type argument only if they use it. The check
modes accept --check-all-methods
(plenty --check, --check-module,
and plenty check): after the used
methods, every remaining method of every class and enum instance the program
creates is checked as if it were called. Such an error has no use, so it is
reported where the program first required the method’s type, such as a
constructor call or a type annotation, with a note naming the flag:
main.plenty:8:9: `Cell[list[i64]].get` copies `T`, but `list[i64]` cannot be copied
main.plenty:4:9: note: in instantiation `Cell[list[i64]].get`: cannot move out of a field
note: `Cell[list[i64]].get` is checked because of --check-all-methods; the program does not use it
The flag checks only types the program’s own package declares: the entry
file’s directory (or --root) for plenty --check, and the package’s src/
for plenty check. Generic types from std and from dependencies keep
per-use checking, because their authors chose which methods each instance
supports. LruCache[K, V].get copies V, for example, so an
LruCache[str, list[i64]] is accepted as long as the program does not call
get. An own type is still checked when its type arguments come from std.
The flag checks only instances: a generic class or enum with no instance stays unchecked, since its bodies are checked with concrete type arguments and there are none. Compiling does not accept the flag, so it never changes generated code or the compile cache.
Type argument inference
Calls supply all generic arguments, as in identity[list[i64]](values), or infer all
from inputs, as in identity(values). Partial explicit lists and inference from
an expected result type are unsupported. Output-only or unused parameters need
an explicit list.
Inference structurally matches concrete arguments: &T against &Message learns
T = Message; &dict[K, V] against &dict[str, u8] learns both parameters.
Collections, ranges, generators, tuples, sums, callable signatures, and nominal
generic class/enum arguments participate. Repeated parameters must agree after
alias resolution. Nominal matching requires the same declaration as well as
matching arguments. Value parameters are inferred from nominal
arguments by the same structural match: Bounded[T, lo, hi] against
Bounded[u8, 0, 100] learns all three parameters.
A shared-reference parameter supplies ordinary implicit borrowing, so a
parameter value: &T can accept an owner directly. Mutable borrowing still needs
&mut or an existing exclusive reference. Arguments evaluate once in source
order.
A generic parameter position supplies no expected type to its argument.
Unsuffixed integers default to i64 and floats to f64, independently of other
arguments or the result context. add(1u8, 2u8) works; add(1u8, 2) conflicts.
add[u8](1, 2) guides both literals. Nongeneric parameter positions retain their
ordinary contextual typing. Empty literals and incomplete sum constructors may
need an annotated binding or explicit arguments.
Callable constraints contribute evidence from a known callback signature:
F: Callable[[], T] can determine T from the callback result. Parameterized
protocol bounds likewise contribute evidence from accessible methods on the known
class. Evidence from all inputs and constraints must agree. Inference does not
widen numbers or search for an implementation class.
A generic function becomes a concrete function value using identity[u8] or
an expected concrete Callable signature. A fully concrete callable constraint
can also select a generic named function passed directly by value. This contextual
selection does not change ordinary generic-call inference. OnceCallable
constraints accept consuming callbacks with the same inference rules. Constraint
signature shapes are checked at declaration; concrete ownership capabilities are
checked when the body is specialized. Polymorphic function values are unsupported.
Value generic parameters
Functions, classes, enums and methods can specialize on compile-time integers
and booleans as well as types. Write const before a value parameter:
def selected[const Enabled: bool](value: i64) -> i64:
if Enabled:
return value
0
def main() -> Result[(), IoError]:
print(selected[True](7), selected[False](7))?
Ok(())
7 0
A value parameter’s declared type is one of i8, i16, i32, i64, u8,
u16, u32, u64 or bool. It may instead name a preceding type parameter
constrained by IntType, as in [T: IntType, const lo: T, const hi: T].
This dependent form keeps a bound’s width and signedness identical to T.
Aliases in the const parameter’s own type annotation are not supported.
Protocols and their requirement parameters do not accept const parameters.
Arguments and identity
Each value argument is an integer or boolean literal, an optionally negated
integer literal, a module constant (including an imported public constant), or
a bare value parameter being forwarded. Unsuffixed literals take the parameter’s
declared type; suffixed literals and constants must match it exactly. There is
no widening. Floating-point values, strings, calls, numeric type constants such
as i64.MAX, and expressions such as N + 1 are not value arguments. A module
constant may name a value computed by a supported constant initializer.
const maximum: i64 = 4 * 4
class Tag[const N: i64]:
def number(self) -> i64:
N
def number[const N: i64](value: &Tag[N]) -> i64:
value.number()
def main() -> Result[(), IoError]:
tag: Tag[16] = Tag[maximum]()
print(number(&tag))?
Ok(())
16
Instance identity compares resolved values, so Tag[maximum] and Tag[16]
are the same type. Different values remain different nominal types even when
no field uses the parameter. Aliases can name complete instances. Values add
no runtime fields and do not change copying, borrowing or cleanup rules.
A value argument cannot appear where a type argument is required.
Bodies and inference
Within a specialized body, a const parameter is an immutable value of its declared type. Reads become typed literals. Ordinary arithmetic on it uses the usual arithmetic contracts; const parameters do not make a function a compile-time evaluator. A local binding cannot reuse a generic parameter’s name.
Structural inference learns values from nominal arguments: Tag[N] matched
against Tag[16] supplies N = 16. Repeated evidence must agree. An ordinary
integer argument’s runtime value supplies no evidence, and an output-only
parameter needs an explicit argument. As with type parameters, supply all
arguments explicitly or infer all of them. Methods supply only their own
parameters; the receiver already fixes its class or enum arguments.
Each distinct argument combination uses the existing specialization cache and instance limits. Value argument expressions cannot grow recursively because bracket arithmetic is rejected.
Bounded[T, lo, hi] is the first built-in consumer.
Inline arrays, derived sizes in signatures, and general per-instance static
assertions are not implemented; B83 tracks that scope.
Generic data and methods
class Cell[T]:
value: T
def read(self) -> T:
self.value
enum Choice[T]:
Value(T)
Cell[u8](7) constructs a Cell[u8] directly using the ordinary class rules.
Class constructor parameters can be inferred from generated field order or an
explicit initializer’s signature: Cell(7u8) selects u8, while Cell(7) selects
i64. Output-only or phantom parameters need explicit arguments; a result
annotation does not influence argument inference.
A constructor’s enum qualifier requires explicit arguments:
Choice[u8].Value(7) constructs a concrete instance. A
pattern takes none, because the matched value
supplies them: case Choice.Value(value): matches it. Aliases can name
complete instances. Different arguments produce different nominal identities
even when a parameter is absent from every payload. This includes
value parameters: Tag[3] and Tag[4] are distinct instances.
Bare self uses the current concrete class or enum; an explicit mutable receiver is
self: &mut Cell[T] or self: &mut Choice[T]. A by-value receiver is
self: Cell[T] or self: Choice[T] (the current-owner spellings Cell and
Choice also infer their declared parameters). Ordinary methods may declare their own parameters and
integer, protocol, or callable constraints. Data parameters are already fixed
by the receiver; method parameters cannot shadow them. Lifecycle methods cannot
add parameters.
Explicit method calls use receiver.method[Types](arguments). Supply every method
parameter without repeating data parameters. Inferred method arguments follow
ordinary inference. Fields and temporary owned receivers support
generic method calls; returned references retain their ordinary named-receiver
and lifetime restrictions. Brackets beginning with a local binding index a
callable field instead, as in table.callbacks[index](argument).
Data parameters accept IntType, FloatType, CopyType, KeyType, Callable, and OnceCallable constraints;
protocol bounds on data are unsupported. Callable signatures may mention other
data parameters and contribute constructor inference evidence. Explicit instances
and aliases are checked too; unused declaration bounds still require valid
signature shapes.
Data type arguments cannot be unit, references, or generators. Resolved owned closures can be arguments; see stored callables. Generic fields, construction, moves, and destruction keep their ordinary data semantics, including recursive storage.
Public instances must expose public concrete arguments, including phantom parameters and types named only in callback constraints. Declaration and member visibility apply normally. A generic method does not currently satisfy a protocol’s monomorphic method requirement.
Structural protocols
protocol Readable[T]:
def read(self) -> T:
pass
def read[T, R: Readable[T]](source: &R) -> T:
source.read()
A protocol declares required ordinary method signatures with pass bodies.
A bound accepts a class or enum whose accessible inherent methods match every requirement,
even requirements unused by the generic body. Receiver borrowing, arity,
parameter types, and return type must match exactly; parameter names may differ.
There is no inheritance, registration, or import-driven method activation.
Bare self means a shared receiver. An owned requirement uses self: Readable[T]
or self: Readable; its implementation must also take its receiver by value.
Owned, shared and mutable receivers are distinct requirements. Mutable requirements use
self: &mut Readable[T] or self: &mut Readable. Within requirements the protocol’s
own name denotes the implementing type, including in returned references.
Protocol arguments substitute into all requirements. Once an input determines
the implementing type, its accessible concrete signatures supply inference
evidence for other parameters, including nested types and returns. Evidence must
agree with ordinary arguments and callable constraints. A method type that no
arguments can fit, such as -> () for -> Result[(), E], does not satisfy the
protocol. For a concrete implementation, a protocol parameter absent from every
requirement still needs explicit arguments. A Box[P[Args...]] supplies its
fixed arguments directly. Inference follows known signatures without searching for a
type or implementation.
Visibility is checked from the bound’s defining module before method signatures supply evidence. Public contracts cannot expose private protocols, requirement types, or concrete arguments. Importing a protocol adds no methods to a type.
A requirement may declare its own type parameters, each unconstrained or constrained by one protocol. An implementation then declares a generic method with as many parameters, the same constraints, and the same signature once its parameter names are replaced by the requirement’s; the names may differ. Calls through the bound instantiate the implementation for each argument type:
protocol Sized:
def size(self) -> i64:
pass
protocol Sink:
def put[S: Sized](self: &mut Sink, item: &S) -> Result[i64, OverflowError[i64]]:
pass
class Crate:
width: i64
def size(self) -> i64:
self.width
enum Parcel:
Letter
Packet(i64)
def size(self) -> i64:
match self:
case Parcel.Letter:
1
case Parcel.Packet(width):
*width
class Total:
sum: i64
def put[T: Sized](self: &mut Total, item: &T) -> Result[i64, OverflowError[i64]]:
self.sum += item.size()
Ok(self.sum)
def feed[K: Sink, S: Sized](sink: &mut K, item: &S) -> Result[i64, OverflowError[i64]]:
sink.put(item)
def main() -> Result[(), IoError | OverflowError[i64]]:
mut total = Total(0)
feed(&mut total, &Crate(3))?
feed(&mut total, &Parcel.Letter)?
print(feed(&mut total, &Parcel.Packet(4))?)?
Ok(())
8
Crate and Parcel both satisfy Sized, so feed accepts either. Use an
enum to model a sum type, a value that is one of several variants, and a class
to model a product type, a record that holds all of its fields. Either can meet
the same protocol.
A generic method cannot satisfy a requirement without type parameters, and a non-generic method cannot satisfy one with them.
Protocols constrain class and enum arguments, the built-in bytes, and an
explicit protocol box exposing the identical protocol.
A protocol alone is not a runtime value type. bytes has one
method, len(self) -> i64; other built-in types have none to offer. Protocol
parameters cannot have their own bounds; protocol arguments exclude unit,
references, generators, and closure environments. Fields, inheritance, multiple
bounds, defaults, associated types, and lifecycle requirements are unsupported.
Generic bodies are checked per concrete instance. Requirements express the
minimum interface rather than a separately checked abstract body. IntType,
FloatType, CopyType, and KeyType
remain separate built-in constraints.
std.fmt.Formattable additionally accepts an
anonymous sum when every normalized member satisfies its contract. The compiler
dispatches to the active member through a shared borrow without allocation.
This is specific to that standard-library protocol: a user-defined protocol with
the same signature does not gain sum forwarding.
Design history
The original protocol design records the shipped foundation. Protocol method bodies propose defaults, final methods, and explicit private-hook grants; those extensions are not implemented. For an existing-language solution, see library-owned operations. The boxed protocol design records the research and rationale for implemented explicit erased owners.
Boxed protocol objects
Box[P](value) -> Result[Box[P], AllocError] consumes a concrete class, enum,
or bytes value satisfying protocol P. Different implementations have the
same owner type, so a list[Box[P]] or dictionary with Box[P] values can call
protocol methods without matching. Protocol arguments must be fixed, for
example Box[Readable[i64]].
Construction makes one allocation, including for an empty payload. Failure destroys the consumed input. A static method table is shared by objects with the same concrete implementation and protocol. Calls, moves, and compiler cleanup add no allocation; user methods and destructors retain their own costs. Collection insertion retains its ordinary consumed-input failure contract.
Every method operates on the same concrete payload. Shared methods borrow it; mutable methods require a mutable binding or exclusive reference. Loans remain rooted in the owner, including through fields and collection elements. An active borrow prevents conflicting replacement, removal, mutation, or movement. Read-only calls on temporary owners keep them alive through the expression; mutable calls require a place.
Only protocol methods are visible. Box[P] satisfies the identical P bound,
including its fixed arguments. Generic inference can use those arguments without
knowing the hidden concrete implementation. Structural matching against another
protocol, even one with identical methods, is unsupported. Conformance checks
required methods from the protocol’s defining module; adapters grant no private
access.
Boxing eligibility
Every requirement must have a shared or mutable receiver and fixed argument
and result types. Ordinary owned arguments, borrowed arguments and slice views,
unit results, aggregate results and fallible results are supported. Calls take
positional arguments. The complete protocol is checked when Box[P] is used.
These requirements prevent boxing, while remaining valid for static generic use:
- Method-local type parameters.
- A consuming receiver.
- The implementing type outside the receiver, including nested occurrences.
- Returned references or views, including references inside
OptionorResult.
Generator-frame and concrete-closure signature types remain unsupported by the indirect-call ABI. Mutually recursive boxed interface signatures are diagnosed rather than expanded indefinitely.
std.fmt.Formattable has a method-local writer parameter and cannot be boxed.
The constructor accepts specialized generic classes and enums, but does not
introduce new builtin conformances or forward arbitrary anonymous sums.
Ownership and limits
Bare P, list[P], and &P remain unsupported value types. Use &Box[P] to
borrow an erased owner. Box(value) still makes a concrete Box[T]; annotations
and arguments never implicitly erase or allocate. Box[P](existing_box) is
rejected: construct from the concrete owner directly, or move an existing
Box[P] without another constructor.
Protocol boxes can be stored in ordinary aggregates and returned from factories.
They cannot be copied, compared for equality, hashed, used as keys/set elements,
or extracted with *box. Downcasts and protocol conversions are unsupported.
Automatic formatting writes <boxed protocol> without inspecting the payload.
These restrictions apply through containing aggregates too.
Native worker/channel transfer is rejected, including through nested containers.
A protocol box cannot conceal executors, futures, channel endpoints, or erased callback captures;
these would hide ownership obligations from static checks. Shared owners
cannot contain protocol boxes, because their erased payloads could conceal
ownership cycles or owners whose destruction waits. Indirect calls retain the
compiler’s conservative effect checks.
The channel restriction includes Sender[T] and Receiver[T] nested in classes,
enums or containers. Hiding an endpoint would prevent the compiler from checking
whether it can block a worker during a join, result wait or pool shutdown. Keep
such payloads in concrete owners such as Box[T], and drop the relevant endpoint
holder before waiting.
See the tutorial, concrete boxes, and design and research.
Errors, allocation, and resource scopes
Recoverable failure is a value: Option represents absence and Result
represents success or an error. ? propagates a failure through the enclosing
function with ordinary cleanup. Allocating operations expose checked failures;
fatal traps terminate without unwinding.
There are no exceptions. A try, except, raise, or finally statement is
rejected with a hint toward Result:
def main() -> ():
try:
pass
`try` is not supported; Plenty has no exceptions: return `Result[T, E]`, propagate an error with `?`, and handle it with `match`
Destruction defines cleanup order.
The entrypoint defines process exit status and
reporting of a main error. std.error adds operation
and path context around a typed error and reports it on standard error.
Standard sums and propagation
Option[T].unwrap_or(default: T) -> T selects a present payload or the default.
Option[T].ok_or(error: E) -> Result[T, E] converts Some(value) to Ok(value)
and Nothing to Err(error). Both consume owned receivers, evaluate receiver
then argument exactly once, and drop the unused argument. They add no
allocation and support affine payloads and arguments; their construction
expressions retain their own allocation contracts. Unit payloads and unit
errors are supported. Bare absence needs enough type context to determine T.
Conversion is explicit: use option.ok_or(error)? in a Result-returning
function; ? alone does not invent an error for absence.
Defaults and recovery
Result[T, E].unwrap_or(default: T) -> T returns the Ok payload or the
supplied default, discarding the error. Like the Option form, it evaluates the
receiver and default eagerly, in that order. In
option.unwrap_or(compute()?), compute runs and can propagate an error even
when the option is Some. Choose an eager default only when that is intended.
The lazy alternatives consume an owned callback and invoke it at most once:
| Method | Callback | When called | Result |
|---|---|---|---|
Option[T].unwrap_or_else(f) | () -> T | Nothing | T |
Result[T, E].unwrap_or_else(f) | E -> T | Err(error), passing the error | T |
Option[T].ok_or_else(f) | () -> E | Nothing | Result[T, E] |
Named functions, reusable closures, and owned one-shot closures use
OnceCallable.
Constructing a callback and evaluating its captures still happens eagerly;
only its body is conditional. Unused callbacks and their captures are dropped.
These operations add no allocation, deep copying, or implicit error propagation.
The existing restrictions on passing borrowed environments by value apply.
Use match for reference-carrying payloads and callbacks that cannot be passed
as owners.
Both sums also support map_or(default, mapper) and
map_or_else(fallback, mapper). The mapper consumes T and returns U.
map_or evaluates its default U eagerly. map_or_else invokes only the
selected callback: Option’s fallback takes no arguments, Result’s fallback
consumes E, and both return the same U as the mapper. Receiver and callback
expressions evaluate from left to right, once. No automatic flattening occurs.
For a fallible default, choose U = Result[T, E], wrap a present value with
Ok, and propagate after selection:
def default_number() -> Result[i64, ParseError]:
i64.parse("23")
def choose(value: Option[i64]) -> Result[i64, ParseError]:
Ok(value.map_or_else(default_number, Ok)?)
def main() -> Result[(), ParseError | IoError]:
print(choose(Some(7))?, choose(Nothing)?)?
Ok(())
7 23
These fallback semantics follow Rust’s Option and Result operations, with Plenty’s explicit allocation and callback contracts.
Deliberate assertions
Option[T].expect(message: str) -> T and
Result[T, E].expect(message: str) -> T return the success payload or trap with
the supplied message and the call site’s source location. The message is
evaluated eagerly, including on success. Reporting adds no allocation;
constructing a dynamic message retains its ordinary allocation contract.
The trap reports the message, without formatting the error payload, and uses
Plenty’s existing trap behavior rather than Rust panic unwinding. Prefer ?
and explicit recovery for expected failures.
Inspection and propagation
Result[T, E].is_ok() -> bool and is_err() report which variant a result
holds; Option[T].is_some() -> bool and is_none() do the same for an option.
They observe the receiver, so a named value remains usable afterwards and a
temporary is dropped at the end of the full expression. They take no arguments,
ignore the payload, and allocate nothing. Asking a Result for is_some or an
Option for is_ok is a compile-time error that names the available pair.
Classes may define methods with the same names.
Option[T] has Some(value) and Nothing; Result[T, E] has Ok(value)
and Err(error). These prelude variants are ordinary typed values. Some
can infer its type from its payload; Ok, Err, and Nothing need expected
context for their missing type arguments. Qualified constructors and aliases
are supported. Payload storage follows enum rules.
Constructing, passing, returning, matching, and propagating standard sum wrappers allocates nothing. Payload operations retain their own costs and ownership rules. Wrapping a generator or closure does not make it copyable. References cannot be sum payloads. Nested sums preserve all scalar values; there is no source-visible null or numeric sentinel reserved for absence.
Postfix value? evaluates once. Some(value) or Ok(value) produces its
payload. Nothing returns Nothing from the enclosing function;
Err(error) returns Err(error). The function must return the same sum
family; Result errors must be identical or widen losslessly into a declared
anonymous sum, except for explicit Failure
erasure below. Success types may differ. There are
no implicit Result/Option conversions. A unit success produces unit.
For explicit error conversion, result.map_error(mapper)
consumes Result[T, E] and returns Result[T, G], invoking the mapper only
for Err. It forwards Ok without copying and adds no allocation. A payload
variant is a mapper, as in print(line).map_error(AppError.Io)?. The
std.error.map_error(result, mapper) function has the same behavior. Neither
changes the propagation rule.
def first(values: &list[i64]) -> Option[i64]:
Some(values.get(0)?)
def checked_first(values: &list[i64]) -> Result[Option[i64], OverflowError[i64]]:
match first(values):
case Some(value):
Ok(Some(value + 1))
case Nothing:
Ok(Nothing)
def main() -> Result[(), AllocError | IoError | OverflowError[i64]]:
values = [3]?
print(checked_first(&values)?)?
Ok(())
Option[i64].Some(4)
Here first is a focused demonstration of Option propagation; an ordinary
first-element lookup can simply return values.get(0). Absence and arithmetic
failure are distinct: checked_first returns Ok(Nothing) for an empty list
and propagates overflow as Err, never as absence. ? does not implicitly
convert Option to Result.
Propagation may appear in arguments, conditions, loops, and comprehensions.
It composes with other postfix operations: values?[0] indexes the unwrapped
value. Failure releases already evaluated pending operands and initialized
locals, including hidden iterators/builders and entered resource scopes, in
ordinary cleanup order. Later operands and statements do not run. Error
payload ownership transfers to the caller. Generator bodies reject ? because
their declared return family is Generator[T]; explicitly match instead.
Prefer a concrete error or a declared sum when combining fallible operations,
so callers can inspect or report the cause. Builtin Failure is the last
resort, for when no precise error can be written. It is a copyable
allocation-free builtin enum containing only Failure.Unspecified. In a function returning Result[T, Failure], ?
accepts any Result error type. Failure consumes and drops the original error
payload, then returns Err(Failure.Unspecified) after normal cleanup. Success
keeps its original payload. Erasure keeps no details, allocates no wrapper,
does not catch traps, and does not convert Nothing. User cleanup retains
its own effects and allocations.
Erasure applies only at ?. Assignments, arguments, direct returns, and
explicit Err(payload) still require the declared error type or lossless
anonymous-sum membership/widening. Use
Err(Failure.Unspecified) to construct that error and Ok(()) for unit
success. An allocating expression propagated this way still has its own
AllocError type before propagation.
An expression statement cannot implicitly discard a Result, including in
non-final branches, loops, context blocks, or generator bodies. Handle it,
return it, or use explicit drop(result). Unused bindings, Option, and
aggregates containing Results are not rejected by this rule. Builtin print
also rejects a direct or borrowed Result; wrapper inspection uses
str.repr(result) with its formatting Result handled separately.
Allocation and traps
Source-level heap construction and allocating operations use checked paths.
AllocError has two allocation-free nullary variants:
AllocError.OutOfMemory for allocator rejection and
AllocError.CapacityOverflow for impossible sizes or layouts. Constructing,
copying, comparing, matching, or propagating it inside Result allocates nothing.
AllocError itself is not a family on which ? operates.
Class instances, tuples, and enum values are inline and construct directly
without allocation. Allocations in their component expressions keep their
own Results. A fallible class initializer can make construction return
Result[Class, AllocError]; class initialization
defines cleanup of a partially initialized instance. String literals, scalar
values, and ranges require no heap allocation.
Collections, boxes, and newly allocated dynamic text expose allocation errors.
Collection displays, comprehensions, constructors, growing updates, and
copy(value) return Result. Their APIs have no separate try_ naming family.
Successful reservation permits subsequent insertions within that capacity
without storage allocation. Failed growth preserves logical contents and order,
although capacity may have increased. Failed deep copy preserves the source
and destroys partial output. Exact APIs and input-transfer behavior are in
collections and
text.
Failure does not generally roll back argument effects, consumed generator steps, external input, or partial writes. Operations specify what is preserved and what has already been consumed. Allocation-free wrapper transport, borrowing, inline environments, generator frames, and cleanup bookkeeping do not make allocations performed by user expressions or destructors disappear. Cleanup on an allocation-error path must still handle its own fallible work. No public allocator switching or custom-memory-manager API is exposed.
value.unwrap() extracts a success payload without allocation, moving it
when owned. Err or Nothing deliberately traps. Invalid indexing, absent
dictionary keys in indexed access, a numeric cast to an integer type that
cannot represent the value, invalid range parameters, and an empty split
separator are also fatal runtime errors. Optional lookups provide recoverable
absence where supported. Integer overflow and division by zero return Err
from the function’s declared Result
by default, or from a local checked: block.
Only an explicit unchecked: suite
reaches arithmetic traps from source operators, where they currently enforce
its no-failure precondition. Explicitly trapping primitives
such as atomic fetch operations retain their contracts.
Traps terminate
the process without unwinding language scopes. They are not caught by Result
propagation or resource scopes.
Trap reports
A trap flushes standard output, writes its report to standard error, and exits
with status one. The line gives the position of the failing operation in the
form of a compile-time diagnostic, error: file:line:column: message:
error: main.plenty:7:11: index out of bounds: index 5, length 3
The position is the start of the expression that trapped: the left operand of an
arithmetic operator, the type name of a cast, the indexed collection, the
receiver of unwrap or split, or the range call. A trap in an imported function or a generic
instance reports the position in the function’s body, in the file that defines
it, not the call or instantiation. The file is relative to the
source root, tools/geometry.plenty for the
module tools.geometry, so an executable does not depend on the directory it
was built in. Compile-time diagnostics print the full path. Reporting a trap
allocates nothing.
When the failing body belongs to the embedded standard library, an additional line names the nearest non-std caller:
error: std/collections.plenty:20:10: index out of bounds: index 0, length 0
note: called from main.plenty:7:12
This caller is the actual invocation, including generic methods, indirect calls, tail calls, and nested standard-library helpers. A user callback that calls std becomes the nearest caller; a trap in the callback itself keeps its own ordinary report. A suspended std generator uses its current resume site, not its creation site. The note is omitted when there is no source caller on the current thread, such as a std function started directly as a worker. Caller context does not cross thread creation or executor submission. This is one caller note, not a stack trace; it does not change the trap message into an API-specific error.
Caller locations
The same nearest non-std caller is available to standard-library code that
reports a failure without trapping. std.test
assertions use it to begin their report with file:line:column: . The compiler
records it as described above, so a std function needs no extra parameter and
user code no annotation. Plenty code outside the standard library cannot read
it: the hook, __caller_location, is rejected with `__caller_location` is reserved for the standard library. Without a source caller on the current
thread the report has no position.
| Trap | Message |
|---|---|
Integer overflow in +, -, *, //, pow, or negation, inside unchecked: | integer overflow |
// or % by zero, inside unchecked: | division by zero |
Integer pow with a negative exponent | negative integer exponent |
| Shift count negative or not below the integer width | shift count outside integer width |
| Numeric cast to an integer type that cannot represent the value | integer cast out of range |
unwrap() on Nothing or Err | unwrap on Nothing, unwrap on Err |
List, text, range, or Bytes index out of range | index out of bounds: index I, length N |
Absent dictionary key in d[k] | dictionary key not found |
A range longer than i64 | range length exceeds i64 |
Empty split separator held in a variable; a literal "" is a compile error | string split requires a nonempty separator |
A standard-library argument outside its domain, such as a Bytes range past the length | The operation and argument, such as Bytes.get: index is outside 0..len() |
A bounds report prints the index as written, before a negative index counts from the end. A missing dictionary key is not printed. Slices never trap on their bounds: out-of-range bounds clamp as in Python.
Standard-library code raises the last kind with the statement
__trap("message"), whose argument is a string literal. It reports the
position of the statement in the std file, with the caller note, and never
returns, so it can end a branch of a function that returns a value. Code
outside the standard library is rejected with `__trap` is reserved for the standard library.
Stack overflow is reported separately and has no position. Trap sites describes how positions reach the runtime.
Resource scopes
with expression as name: evaluates an owned manager once and moves it into
a hidden mutable owner. A concrete class manager exposes visible methods
__enter__(self: &mut Class) -> T and __exit__(self: &mut Class) -> (),
with no other parameters. The entry result becomes a body-local binding.
Without as, an unused owned result drops before the body. Entry is an
ordinary call: it does not implicitly unwrap a Result. Perform fallible
acquisition with ? in the manager expression.
Comma-separated managers nest left to right, each one a level of nesting. Each acquisition and entry completes before evaluating the next expression, which can use earlier entry bindings. If a later acquisition propagates failure, only already entered managers exit.
On leaving the body, its locals drop in reverse order, then __exit__ runs
once, then the manager drops normally. Nested managers exit in reverse order.
Return values are evaluated and preserved before cleanup. return, ?,
break, and continue use this order for the scopes they leave; control
inside an inner loop does not exit an outer manager. __del__ is a separate
fallback hook, so exit and later destruction must be safe together. Exit is
not an implicit flush or durability guarantee; fallible operations expose
their own Results.
with &mut existing: instead reborrows a mutable named manager or class field.
The exclusive loan lasts through __exit__, preventing competing access,
replacement, and movement. After exit the caller retains its mutated manager;
its destructor remains at the owner’s normal scope boundary. An existing
exclusive reference must also be explicitly reborrowed with &mut.
__enter__ may return &T or &mut T under the
returned-reference contract.
The entry binding carries that loan and must finish before exit: it cannot
escape an owned context or stay live across a borrowed manager’s exit call.
Owned entry results may move out normally.
Resource-scope bookkeeping adds no runtime allocation; manager methods retain
their own costs. yield inside with is unsupported. Traps and process aborts
do not unwind managers. Intrinsic thread, channel, and executor managers use
the same scope-exit obligations; their shutdown contracts are described under
concurrency. mutex.lock() is an
intrinsic manager too: it borrows the mutex shared rather than moving it, binds
an exclusive reference to the value, and unlocks on every exit; see
mutexes.
Iteration and generators
Iteration either consumes an owned iterable or lends access to a collection. Comprehensions build fallible collection results. Generators own concrete frames and resume only when advanced; construction and frame management add no heap allocation.
- Iterable values and borrowing
- Comprehensions and collection construction
- Generators
- Enumeration and paired iteration
- Reverse traversal
- Local iterator pipelines
Loop control defines binding scope, break,
continue, and return checking. Collections
and text define their operations and ordering.
Iterable values and borrowing
for name in iterable: evaluates the iterable once. Owned collections and
generators transfer into the loop’s hidden owner; it drops on exhaustion or
early exit. There is no implicit snapshot. Ranges, strings, and copyable
elements retain their ordinary copy semantics.
Iterating a range, as the source of for, reversed, enumerate, zip, a
comprehension, or a generator expression, compiles to a counted loop. The
element count is computed once, before the first iteration, and no step calls
the runtime, so it costs no more than the equivalent while loop. This holds
for a range(...) call written directly as the source and for a range value
stored in a variable, field, or generator frame or returned by a call. The loop
copies the range when it starts: changing a bound variable, or assigning a new
range to the variable, inside the body does not change the count. A generator
expression’s outer source is stored in its frame, so each element also costs
one frame resume.
| Iterable | Yielded value |
|---|---|
| List | Element |
| Dictionary | Key, in insertion order |
| Set | Element, with no promised order |
| Range | Integer of the range’s element type |
| String | Unicode scalar value as str |
bytes | Byte as u8 |
std.bytes.Bytes | Byte as u8 |
| Generator | Declared yielded value |
Consuming list iteration transfers affine elements. for item in &list
preserves the list and always yields &T, including scalars and strings.
for item in &mut list yields &mut T, including scalar elements.
Existing collection references follow the same access mode. This includes
direct reference-returning calls: for item in owner.column()
and iter(owner.column()) need no intermediate reference binding. The call
runs once when the source is prepared, and its returned loan protects the owner
through the loop, including backedges, against
invalidating mutation, movement, or destruction. Element loans follow the
borrowing rules: only distinct literal list
indices are disjoint. Iteration
references cannot escape into stored references.
Borrowed temporary iterable owners remain alive through the loop and drop when its scope exits, including on early exit.
Borrowed dictionary and set iteration yields shared references to keys or
elements. Borrowed range, string, and bytes iteration also yields shared
references; each refers to a computed value in an iteration-local slot and
cannot escape that iteration. A Bytes buffer has addressable storage:
for b in &buffer and for b in &mut buffer loop over a
slice view of the whole buffer and
yield &u8 or &mut u8 into it. Comparisons and printing observe references uniformly;
dereference explicitly when passing an element to a function that takes a value
or storing it in a new collection. Other arithmetic observes numeric references
as well.
Borrowed generators cannot be loop sources; advance one with next(&mut g)
instead. Use for item in copy(values)? for an explicit independent snapshot
when the original must be mutated during iteration.
for key, value in dictionary.items(): borrows a named dictionary or field
and preserves insertion order. Keys are &K and values are &V, regardless
of whether their types are copyable. (&mut dictionary).items() or
&mut dictionary.items() instead yields &mut V values with shared &K
keys. These loops allocate
no item tuples or snapshot buffers. items() is a loop/comprehension intrinsic,
not a storable iterator, and requires two distinct binding names (or _).
Its source loan prohibits invalidating dictionary mutation.
dictionary.keys() creates a bindable local KeyView[K]; iteration yields
&K in the same insertion order, without allocation. dictionary.values()
is a loop/comprehension source yielding &V, including affine values;
(&mut dictionary).values() yields &mut V; see
updating every value.
Both protect the whole dictionary under their loans. Collect explicitly ([*key for key in d.keys()]?), or
use keys_snapshot() / values_snapshot() for owned lists and into_keys() /
into_values() to consume the owner. See dictionary views
for the key view’s set operations and storage restrictions.
Loop targets permit recursive tuple unpacking, such as for (x, y), label in rows.
Comprehensions and generator expressions accept the same targets. Each name is
a new immutable binding local to that iteration scope, shadowing any outer local
without changing it. A clause evaluates its source before introducing its own
bindings; filters and subsequent sources see the new bindings. Enclosing tuple
parentheses do not change the shape: for ((x, y), label) in rows is equivalent,
including over borrowed zip and dictionary item sources. Over borrowed elements, each name
borrows its component in place: for n, s in &pairs binds n as &item[0]
and s as &item[1], and for n, _ in &mut pairs makes n a &mut
reference, so *n += 1 changes the list. The names borrow distinct fields of
the element, so swap(n, s) under for n, s in &mut pairs passes both as
&mut arguments, as swap(&mut p[0], &mut p[1]) does for an element p of a
&mut loop. Borrowing the list or another element while they are live
conflicts. Iteration itself, including string
scalar production and generator next, adds no allocation; iterable creation
and body expressions keep their own contracts. There is no arbitrary user
iterator protocol, special-method dispatch, or async iteration.
Generator expressions construct
lazy owned frames and follow the generator iteration rules.
Comprehensions and collection construction
List, set, and dictionary comprehensions return
Result[collection, AllocError]. Multiple for and if clauses nest from
left to right, each one a level of
nesting. Each iterable evaluates when its enclosing iteration reaches
it; filters run before the result expression. Dictionary keys evaluate before
their values. Iteration uses the same consuming or borrowed contracts as
loops.
def main() -> Result[(), AllocError | IoError | OverflowError[i64]]:
values = [n * n for n in range(5) if n > 1]?
print(values)?
Ok(())
[4, 9, 16]
A comprehension with one for clause and any number of if clauses lowers to
the same loop as the equivalent pipeline:
[f(x) for x in s if p(x)] runs as iter(s).filter(p).map(f).collect() would,
with no hidden function. Its iterable is evaluated before the output is
allocated. Over a known-length range or list without filters, the output
reserves its full length once and never grows. Comprehensions with several
for clauses keep the nested loop.
Comprehension bindings have their own scope; the first iterable sees the enclosing scope. Flat tuple targets are supported. Conditional expressions in iterables or filters must be parenthesized. Parenthesized generator expressions produce lazy owned frames instead of collection Results. Async comprehensions are unsupported.
Output ownership and growth are fallible. On growth failure, the partial output,
current yielded owners, and active hidden iterators clean up. No iterator
advances and no later filter or element expression runs. Nested expressions
keep their own Result types and failure behavior. A nested ? propagates from
the enclosing function rather than becoming an error inside the surrounding
comprehension Result. Already performed side effects are not rolled back.
Collection displays [elements], {elements}, and {key: value} follow the
same recoverable output-allocation policy. Their owner allocates before entries
evaluate left to right; allocation failure prevents later entry evaluation.
Expected Result types supply element types for empty displays. See
collection construction for literal
types and duplicate-entry behavior.
list[T].from(source) and set[T].from(source) consume an owned collection,
range, or generator and return Result with AllocError. Failure destroys
partial output, the current element, and remaining source. Generator effects
before failure persist. Sets require hashable elements and keep the first
occurrence of each value. Borrowed sources and strings are unsupported by
these .from(...) methods. The call-style constructors additionally accept
strings and bytes; see collection conversion.
A directly written range in a comprehension can receive numeric element
context from the output annotation through direct arithmetic, for example
squares: list[u8] = [n * n for n in range(8)]?. This inference does not run
backward through arbitrary calls, filters, stored ranges, or explicit range
type arguments. Use range[u8](...) at those boundaries. Fixed-width
arithmetic follows its normal overflow rules.
Generators
A function containing yield declares Generator[T]. Ordinary call syntax
producer(arguments) constructs its owned frame; .new(arguments) is an
equivalent alias. Arguments evaluate first and copy or move into the frame.
Construction runs no body code and requires no heap allocation. Argument
expressions keep their own allocation and error contracts.
def numbers(stop: i64) -> Generator[i64]:
mut value = 0
while value < stop:
yield value
# value starts at zero and value < stop <= i64.MAX.
unchecked:
value = value + 1
def main() -> Result[(), IoError]:
for value in numbers(3):
print(value)?
Ok(())
0
1
2
Resumption executes until a statement-only yield value, bare return, or
fallthrough. Yield types are exact and owned payloads transfer to the caller.
Reference, generator, closure, and unit yields are rejected; a generator body
cannot return a value. An ordinary factory without yield may return another
frame by moving it. ? is unsupported in a generator body; match fallible results
explicitly. Recoverable arithmetic failures cannot propagate from a generator:
handle a
local checked: result, call a helper
that returns a Result and yield or match it, or use a small unchecked: suite
with a documented bound.
next(&mut g) -> Option[T] borrows a mutable generator owner or reborrows an
exclusive generator reference. next(g) is shorthand for named bindings and
exclusive references. Shared references, immutable owners, and mutable
temporaries cannot be advanced. The borrow lasts only through the call;
exhaustion is stable and produces Nothing on every later call. Resumption
and its Option wrapper add no allocation beyond body operations.
Loops, comprehensions, and iterable collection constructors consume frames.
break destroys a loop’s hidden frame without executing subsequent generator
statements. Assignment, arguments, and returns transfer ownership. Standard
Option/Result wrappers may own frames, including nested wrappers, and remain
affine. Frames cannot enter collections, class fields, tuples, user enum
payloads, or heap-stored closures. Printing, comparison, length, and membership
are undefined for frames.
Each producer and concrete specialization has a distinct frame identity.
Generator[T] describes the yield contract without erasing that identity.
Local annotations preserve the initializer’s producer, and parameters,
references, aliases, and standard sum wrappers specialize to actual frames.
A factory must resolve to one producer per specialization; equal yield types
do not allow different producers to join. Nothing or Err alone cannot
infer an absent producer. No hidden boxing makes frames interchangeable.
A suspended frame retains its live owned locals. Completion releases them;
dropping a suspended or never-started frame releases remaining locals and
captures without resuming the body or allocating. Short local borrows completed
within one resume are permitted, but generator parameters cannot be references
and no live loan may cross yield. yield inside with is unsupported.
There is no yield-from, send/throw, or async/await.
Fluent transformations
Generators expose the pipeline adapters and terminals:
numbers(100).filter(even).map(square).fold(0, add) consumes the generator
through a fused adapter loop. Source resumption retains its ordinary cost;
the fluent adapters do not each construct another suspended frame.
An owned pipeline can become a returnable resumable stream through explicit
.generator(). It transfers already prepared source and capture state into
one frame without advancing any item. Borrowed sources or callback captures
are ineligible, including when mapped items are owned. There is no implicit
conversion at a Generator[T] parameter or return annotation. The resulting
frame follows the specialization, suspension and cleanup rules above.
Generator expressions
(expression for name in source if condition) creates a concrete owned
Generator[T], with its yielded type inferred from expression. Multiple
for and if clauses have the same scopes and nesting limits as comprehensions.
The outer source evaluates once at creation; inner sources, filters, and the
element expression run only as the frame advances. A range outer source is
stored in the frame and advances as a counted loop. A sole call argument can
omit parentheses: list(n for n in range(4)). Arithmetic in an element or
filter follows the same generator restriction; a Result-returning helper may
produce Result items. With other arguments,
parenthesize the generator expression.
Inputs and referenced enclosing local values copy or move into the frame at
creation. Copyable captures are snapshots; later assignments do not change them.
Affine captures become unavailable to the caller. Captures are taken in order
of first use in the remaining clauses, then the yielded expression, after the
outer source. References cannot be captured, including a borrowed outer source;
use an owned source or copy(source)?. Adapter sources enumerate, zip, and
reversed evaluate their inputs at creation without allocating an adapter.
Construction and advancement add no heap allocation beyond the expressions they run. The same yield restrictions, propagation restrictions, destruction, and concrete-frame specialization rules apply as for generator functions. Dropping an unstarted frame releases its captures without evaluating filters or elements; short-circuit consumers destroy the remaining frame immediately.
Recursive inline layouts are rejected. Nested frame layouts are limited to 64 levels, native frame offsets to signed 32-bit sizes, a function’s native stack frame (which holds the inline frames it owns) to Cranelift’s 1 GiB, a compile-time error naming the function, and concrete consumer specializations to 16,384 per consumer and 65,536 per compilation, with the dependency depth limit of generic functions guarding expanding recursion (see specialization). These are implementation limits, not heap-backed fallback behavior. See generator implementation.
Enumeration and paired iteration
enumerate(iterable[, start: i64]) and zip(left, right) are intrinsic
sources for for loops and comprehensions, and can start local
pipelines. Pipelines also have .enumerate(start=0) and
.zip(right) methods. They compose the existing loop
plans and allocate no iterator object, snapshot, or item buffer. Inputs
evaluate once in source order; both zip inputs evaluate even if one is empty.
Owned items use inline tuples, allowing one tuple binding or two unpacked
bindings. Nested adapters work with owned items.
Enumeration yields (i64, element), starting at zero by default. Its counter
increments with checked arithmetic only after another element is available.
A last index of i64.MAX is valid; a further item traps at the adapter’s
source location. continue advances the source and keeps the count aligned.
Enumerating a forward list or view, as enumerate(xs), enumerate(&xs),
enumerate(v) or iter(&xs).enumerate(), without a start counts with the loop’s own row
position. That count cannot overflow and needs no separate counter. Indexing
the same &list[T] parameter or view with it then needs no
bounds check.
Zip yields (left element, right element) and stops when either input ends.
It tests the left condition before the right, including generator advancement.
When the right generator ends first, the left can have one unmatched pending
item; cleanup drops it. Generator bodies therefore may have one extra left
side effect. break, return, and propagation drop all remaining owned inputs.
Each input follows the ordinary iteration ownership rules.
Borrowed elements retain their reference types and source loans. Such adapters
require two unpacked bindings and bind the references directly, without
constructing a tuple containing references. Aliasing exclusive inputs and
invalidating mutation remain errors. An inner borrowed adapter cannot be
used as an outer adapter’s tuple-valued input. dictionary.items() yields
two references; it can start a pipeline but cannot become a nested tuple
containing references.
The calls do not construct first-class iterators, support arbitrary iterator
protocols, or implement zip strictness. Pipeline enumeration also accepts the
start keyword. Reverse traversal
can supply either input. A user function or callable
binding named enumerate or zip retains its normal call behavior.
The free forms and methods yield the same items: enumerate(&names, 1) and
iter(&names).enumerate(1) are interchangeable as a loop source or pipeline
start. Use the method form to enumerate or zip partway
through a chain, after a filter or skip, where it counts or pairs only the
items that reach it. reversed is not a pipeline adapter: reverse the source
instead, as in iter(reversed(&values)) or zip(reversed(&values), &values).
Reverse traversal
reversed(iterable) is a source for loops and comprehensions over a list,
string, bytes, std.bytes.Bytes, or range. It evaluates the source once and
visits its elements from last to first, without changing the source order or allocating a reversed
copy. Text is traversed by Unicode scalar value, with linear total byte
traversal; combining marks remain separate scalars. Ranges retain their
element type and original step, including descending and unsigned ranges.
The ordinary iteration ownership rules apply. An owned
list transfers its elements and drops its unvisited elements on early exit.
reversed(&values) yields shared element references; reversed(&mut values)
yields exclusive references for a list. A borrowed temporary source stays
alive until the loop scope exits. continue advances backward once, and
break, return, and propagation clean up the remaining source and current
item. Flat tuple unpacking and comprehensions work as in forward iteration.
Use reversed as an input to enumerate or zip, including
borrowed inputs, or as a pipeline source:
iter(reversed(&values)). Pipelines have no reversed method. Its own input must be a concrete list, string, byte literal,
byte buffer, or range; it cannot reverse enumerate, zip, a generator,
or a hash collection.
It does not create a stored iterator or dispatch to a user iterator protocol.
There are no free any and all functions. Test boolean items with the
pipeline terminals iter(flags).any() and
iter(flags).all(), or pass a predicate: iter(values).any(is_negative).
reversed may supply their input: iter(reversed(&flags)).all(). A call to
any(...) or all(...) that names no user function is an error naming these
replacements.
Functions and callable bindings named reversed, any, or all retain
their normal call behavior. Imported generic functions and generator bodies
can use reversed.
Local iterator pipelines
iter(source) starts a lazy, single-use pipeline. It needs no import. Source
expressions and callback captures are evaluated immediately, from left to right;
items are visited only when a loop, comprehension, or terminal consumes it.
The compiler fuses the adapters into the source’s ordinary iteration loop.
Generators expose the same adapters and terminals directly:
numbers(100).filter(even).map(square).collect() is equivalent to starting
with iter(numbers(100)). Each adapter extends the fused plan.
p = iter(&payments).filter(def(p: &Payment) -> bool: p.name == "bob")
total = p.map(def(p: &Payment) -> i64: p.cents).fold(0, add)
Use a plain loop when the body has several statements and a comprehension when building a collection is clearest. Pipelines are useful for borrowed transformations and reductions that would otherwise need nested generator adapters or an intermediate collection.
Sources and lifetime
Sources follow the ordinary iteration rules: owned
collections and generators move into the pipeline; borrowed collections retain
their loan. Integer ranges use counted loops. Text, bytes and collection views
keep their ordinary item types. A slice view
keeps its owner’s loan and yields &T, or &mut T for an exclusive view, as
in iter(&xs[1:4]).filter(even).count().
A source may be a reference-returning call, such as iter(owner.column()).
The call runs once during construction; its owner stays borrowed until the
pipeline is consumed or dropped. An intermediate reference binding is optional.
A local may have an Iter[T] annotation. Its concrete source and callback
identity remain part of its type. Consuming it, adapting it, or drop(p) moves
it; a second use is rejected. Dropping it releases the owned source and
captures without advancing a generator or calling a callback. Early exit from
an owning consumer releases the unconsumed source and captures.
Pipelines cannot be returned, passed by value, shared, captured, or put in
stored data. In generator functions, a pipeline must be consumed by a terminal
in the expression that creates it, such as yield iter(&flags).any(); a
pipeline local or a loop over a pipeline is rejected. A pipeline can be lent to a
&mut Iter[T] parameter, and an eligible owned pipeline can leave its scope
after explicit .generator() conversion.
Advancing one item
next(p) takes one item from a pipeline local in its creating scope and
returns Option[T]. The binding must be mut; next(&mut p) is equivalent.
The pipeline keeps its progress, so a later next, loop, adapter or terminal
continues after the items already taken:
mut rows = iter(lines).filter(def(l: &str) -> bool: l != "")
header = next(rows)
records: list[Record] = rows.map(parse_row).try_collect()?
next expands the fused step inline: there is no frame and no dispatch. It
returns Nothing once the pipeline is exhausted, and every later next,
loop or terminal sees an empty pipeline: take_while stays stopped after its
first rejected item, and zip stays stopped after either side ends. Items
must be storable in an Option, so a pipeline of references is rejected;
map to an owned value first. Loans held by the pipeline stay live until it is
consumed or dropped, so mutating a borrowed source between two next calls is
a borrow error. next on a consumed pipeline is a use of a moved binding.
Lending a pipeline
A function can take a pipeline as &mut Iter[T], consume part of it, and
return. The lender then continues after the items the borrower took:
def skip_header(rows: &mut Iter[str]) -> Result[(), CsvError]:
match next(rows):
case Some(header):
check_header(&header)
case Nothing:
Err(CsvError.Empty)
def main() -> Result[(), CsvError | AllocError | IoError]:
mut rows = iter(lines).filter(def(l: &str) -> bool: l != "")
skip_header(&mut rows)?
records: list[Record] = rows.map(parse_row).try_collect()?
...
The lender writes &mut p and the binding must be mut. Inside the borrower,
p supports next(p), for x in p, terminals, and adapters that build a
sub-pipeline consumed in the borrower. A break, return or ? inside the
borrower’s loop leaves the remaining items in the pipeline. The borrower may
lend p on, as f(p) or f(&mut p), including recursively. It cannot move,
return, store or capture p, and &Iter[T], by-value Iter[T] parameters
and Iter results are rejected, as are Iter parameters of exported and
foreign functions.
&mut Iter[T] names only the item type. Like a &mut Generator[T]
parameter, the borrower is compiled once per lent pipeline, as a frame
specialization; recursive and repeated lends of the same pipeline share it.
A method can take a lent pipeline the same way.
A generic borrower such as def first[T](p: &mut Iter[T]) -> Option[T] infers
type parameters only from the lent pipelines’ item types. A call that needs
any other type parameter, even one an ordinary argument would determine,
supplies all type arguments explicitly:
def tagged[T, K](p: &mut Iter[T], tag: K) -> Option[(K, T)]:
...
tagged(&mut p, "first") # error: cannot infer type parameter `K`
tagged[i64, str](&mut p, "first") # Some(("first", 0))
A borrower is checked only through the calls that lend it a pipeline. One that no call lends a pipeline to is not checked, like an unused generic function, so its errors appear when the first lending call is added.
The lender keeps the pipeline’s loans live across the call, so mutating a borrowed source while the pipeline is unfinished stays a borrow error. Lending a pipeline whose binding a consumer already moved is a use of a moved binding.
Costs and limits:
- The borrower receives one pointer per owned state slot (source cursor,
owned captures, counters,
chainphase,flat_mapinner source,zipandtake_whileflags) and the value of each reference the pipeline holds, as hidden arguments. It copies owned state into its own locals on entry and writes it back on every exit, including?,returnand an early loop exit. No pipeline state is heap-allocated, and a pipeline that is never lent pays nothing.next(p)in the borrower expands one step inline. - References the pipeline holds, such as a borrowed source or a borrowed capture of a closure literal, are read-only in the borrower; the lender’s loans cover them.
- A pipeline of references, such as
iter(&names), can be lent as&mut Iter[&T]. Each item borrows the lender’s source, which stays borrowed for the whole call, so the borrower can use items in loops, adapters and local closures, and lend the pipeline on. An item cannot leave the call: returning it or a reference derived from it is an error, andnext(p)on reference items is rejected as in the creating function. A generic borrower cannot take one, since type arguments cannot be references; write&mut Iter[&T]with a concreteT. - A pipeline cannot be lent when it holds a reference inside other state, such
as a closure with borrowed captures stored in a variable and then passed to
an adapter, or a reference into its own state. The diagnostic points at the
adapter argument; write the closure literal inside the adapter call instead,
as in
.filter(def [&limit](x: &i64) -> bool: *x > limit). Its captures then become references the lender’s loans cover. - A lent pipeline may need at most 64 state slots.
Returning a resumable stream
.generator() consumes a pipeline and transfers its prepared source, callback
environments, counters and other iteration state into one concrete
generator frame. It takes no arguments and adds no heap
allocation. It does not revisit source expressions, reconstruct callbacks, or
advance the source. Adapters remain fused inside the frame; next resumes
that frame until its next accepted item or exhaustion.
def squares(stop: i64) -> Generator[i64]:
iter(range(stop)).map(def(n: i64) -> i64: n * n).generator()
def total(values: Generator[i64]) -> i64:
values.fold(0, def(acc: i64, n: i64) -> i64: acc + n)
The result can be returned, passed to a generator-consuming helper, moved,
advanced with next, or used as another fluent source. Owned sources and
owned captures are required. Borrowed sources and borrowed captures are
rejected even when a later map produces owned items; conversion never copies
borrowed state. Items must satisfy the ordinary generator yield restrictions.
The existing concrete-producer and finite-frame rules apply, including the
restriction against joining unrelated generator producer types.
Dropping an unstarted or partly consumed frame releases all its owned state without invoking more callbacks. Exhaustion is stable. A zip frame preserves left-first pulls and drops an unmatched left item when the right input ends. Mutable owned captures keep their state between resumptions.
Conversion is always explicit. An expected Generator[T] type does not convert
an Iter[T], and a generator expression over an unconverted pipeline is not
an alternative conversion spelling. Prefer local consumption when a resumable
stream does not need to cross a function boundary.
Adapters
| Method | Behavior |
|---|---|
map(f) | Call (Item) -> U once per item; emit the returned value |
filter(f) | Retain items for which the shared-reference predicate is true |
take(n) | Emit at most n items; zero or negative n advances nothing |
skip(n) | Discard the first n items; zero or negative n skips nothing |
take_while(f) | Stop at the first false shared-reference predicate, dropping that item; stay stopped |
filter_value(value=True) | Retain items equal to value, compared with ==; no callback |
enumerate(start=0) | Emit the checked i64 index and item; count accepted items at this point in the chain |
zip(right) | Pull one accepted item from each input, left first; stop when either ends and stay stopped |
chain(other) | Emit every item of this pipeline, then every item of other |
flat_map(f) | Call (Item) -> source once per item; emit each item of the returned source |
Predicates receive &T for owned T, borrowed &T, and mutable &mut T
items. Mapping receives the item itself. Callback parameters and results require
explicit types. In serial pipelines, every reusable callable is accepted by
value: a named function, an anonymous function, a capturing closure with owned
or borrowed captures, a stored Callable value or parameter, a BoxedCallable, or a
class or payload-variant constructor.
Generic constructors need explicit instances here, for example
map(Option[i64].Some), because a pipeline does not supply an expected callback
result type. Parallel pipelines accept a narrower set of
callback forms.
One-shot closures are rejected because the pipeline calls the callback once per
item, and a reference such as &f is rejected: pass the function or closure
itself. map cannot currently return references.
A closure literal written directly as a callback is lowered into the consumer’s
loop instead of being called. It behaves as the call would: captures are
evaluated once, when the chain is built; borrowed captures stay borrowed until
the pipeline is consumed or dropped; owned captures are dropped with the
pipeline; and a mut capture keeps its state across items. A closure stored in
a local, and a literal whose body is more than one expression or contains ?,
match, or another closure literal, is called as before. This optimization
preserves each callback’s declared arithmetic contract.
Callbacks never inherit the consumer’s checked or unchecked mode. Arithmetic
callbacks keep their function boundary; use try_fold for a callback returning
Result[A, E]. An infallible callback must handle arithmetic locally or
contain an explicit, justified unchecked: suite.
Integer sum() returns Result[T, OverflowError[T]], including inside
unchecked:, and stops at the first overflow. Floating sum() returns an
ordinary float using sequential IEEE addition, including infinity on overflow.
filter_value(value) evaluates value once, at construction, and compares
each item with item == value in the loop; the item passes through unchanged.
The value takes the item type T as its expected type, so filter_value(2)
over u8 items needs no suffix, but it is not otherwise converted; a different
type is rejected at the call. For a borrowed &T or &mut T item, value may
be a T or a shared &T, and the comparison reads through both references
(*item == *value). A borrowed value keeps its loan until the pipeline is
consumed or dropped, like a borrowed source; moving or assigning its owner
while the pipeline is live is a borrow error. Owned items require a T.
Any type that == accepts is allowed, with the same semantics. In particular,
floats use IEEE equality: a NaN value keeps nothing and -0.0 keeps 0.0.
The argument defaults to True, so filter_value() keeps true bool items and
is a type error for any other item type. Pair items are rejected; use filter
with a two-parameter predicate. An owned value is dropped with the pipeline, and
a .generator() conversion moves it into the frame.
Free enumerate(source[, start]) and zip(left, right) also start pipelines.
dictionary.items() can be used directly in a chain or through iter.
A pair item spreads into two callback parameters: map and for_each receive
the components themselves; predicates receive shared references to both.
fold and try_fold take (accumulator, left, right), and reduce takes
(accumulator_pair, left, right). The accumulator is always one value, even
when it is a tuple. Only the item’s top level spreads. Owned nested pairs are
ordinary tuples; a nested pair containing references cannot be formed.
Zip pulls through each arm’s filters and skips independently. If the right
arm ends, a left item may already have been consumed; that unmatched item is
dropped. Ending a consumer drops both arms and their pending items. Enumeration
permits a final index of i64.MAX, trapping only if another item is available.
There is a 64-adapter/type-depth limit, alongside expression nesting limits.
chain(other) accepts any pipeline source, including another pipeline. Both
sides must yield exactly the same item type, pair shape included. The second
source is evaluated at construction but not advanced until the first ends. A
take_while that stops the first side moves on to the second; it does not end
the chain. An item borrowed from either side keeps both sources borrowed.
flat_map(f) calls f for an outer item only when the previous inner source
has ended. One rule applies: the callback returns any owned iterable (a list,
set, dictionary, str, bytes, range, or generator), and flat_map emits its
items as a for loop over that value would. A reference, a Result, a
non-iterable value, or a nested Iter is rejected; borrowed outer items are
allowed, but the returned source may not borrow them. Each inner source is released as
soon as it ends, and an unfinished one is released when the consumer exits or
the pipeline is dropped. Inner items are owned, so flat_map adds no loan.
Terminals
| Method | Result and behavior |
|---|---|
fold(initial, f) | Left fold with callback (A, Item) -> A |
reduce(f) | Left fold without an initial value; Option[Item], empty is Nothing |
count() | Checked i64 item count |
sum() | Integer Result[T, OverflowError[T]] (Ok(0) when empty); floating T (0.0 when empty) |
min() / max() | Least / greatest numeric or str item in Option[Item]; empty is Nothing; ties keep the first |
any(f) / all(f) | Short-circuit shared-reference predicates; empty is false / true |
any() / all() | Short-circuit over bool or &bool items; empty is false / true |
find(f) | First matching item in Option[Item], then release the rest |
for_each(f) | Call (Item) -> () for each item |
try_fold(initial, f) | Left fold with (A, Item) -> Result[A, E], stopping at the callback’s first error |
try_for_each(f) | Call (Item) -> Result[(), E], stopping at the callback’s first error |
collect() | Build an expected list[T], set[T], or dict[K, V]; return Result[Collection, AllocError] |
try_collect() | Collect Result[T, E] items, returning `Result[Collection, E |
generator() | Transfer an eligible owned plan into a lazy Generator[Item] |
any() and all() test each item itself, as filter_value() does; other
item types need a predicate. There are no free any and all functions.
find and reduce copy a borrowed &T or &mut T item when T is
copyable, such as a number or str, as sum, min, and max do:
iter(&values).find(f) returns Option[i64] for a list[i64], and a reduce
callback takes (i64, i64) -> i64. Other borrowed items cannot be stored in an
Option; map them to an owned value first.
sum, min, and max read a borrowed &T or &mut T item as its value
(*item), so iter(&values).max() returns Option[i64] for a list[i64] and
copies the selected str. They take no arguments and add no allocation. sum
accepts numeric items; min and max accept numeric and str items. Pairs,
tuples, and other types are rejected: map them to a supported item first, or use
fold or reduce. Float min and max follow IEEE 754-2019 minimum and
maximum: -0.0 orders below 0.0, and any NaN item makes the result
Some(NaN). The NaN returned is the first NaN item. They still consume every
item. std.math.min_all and max_all share this rule
and give the same result for the same items. A float sum adds in source
order with IEEE 754 addition, with no compensated summation: any NaN item
makes the sum NaN, and an empty float sum is 0.0. Integer sums return the first overflow’s
wrapped payload without allocation. Handle their ordinary result explicitly;
an outer checked: suite does not catch errors returned by a terminal.
Collecting needs an expected collection type, from a binding annotation, argument, or return type. There is no default collection:
values: list[i64] = iter(range(10)).map(square).collect()?
result: Result[list[i64], AllocError] = iter(range(10)).map(square).collect()
Borrowed items must be mapped to owned values before collection. Dictionary
collection consumes owned (K, V) pairs; the last value for a duplicate key
wins. try_collect into a dictionary needs Result[(K, V), E] items.
A failed collection drops its partial
output and the remaining owned source. try_collect stops at the first item
error. try_fold and try_for_each instead stop on callback errors: their
callbacks must explicitly handle any Result items. They add no allocation
error of their own.
Parallel pipelines
.par(&pool) runs the rest of a pipeline on a ThreadPoolExecutor’s workers.
The pool is borrowed: write &pool, or pass a &ThreadPoolExecutor value.
with ThreadPoolExecutor(8, 64)? as pool:
lengths = iter(paths).par(&pool).map(file_size).collect()?
best = iter(range(1, n)).par(&pool, chunks=16).map(score).reduce(better)?
| Part | Accepted |
|---|---|
| Source | an owned list[T], a borrowed &list[T], or a range[i64], through iter(source), iter(source).enumerate(start) or enumerate(source, start) |
Steps after par | map, filter, and filter_value |
| Callbacks | named functions, or reusable immutable closures that own their captures; constructor values (class or variant), stored capture-free closures and Callable values are not yet accepted |
| Terminals | collect, try_collect, count, any, all, sum, min, max, reduce |
enumerate must come before par. Other steps (take, skip, take_while,
zip, chain, flat_map) are rejected after par, as are
steps before it other than enumerate. Generators, other range types, and
lists reached through &mut are rejected. A parallel pipeline
must end in a terminal: it cannot be stored, looped over with for, converted
to a generator, or advanced with next. fold, try_fold, for_each, and
try_for_each are rejected because their callbacks have no associativity
guarantee or would run in no fixed order. Mutable, one-shot, and borrowing
closures are rejected; callbacks also pass the
worker eligibility checks.
Borrowed sources. As in a serial pipeline, iter(&xs).par(&pool) yields
&T items, and the list stays usable afterward. Every chunk reads its rows of
xs in place. collect, try_collect and reduce cannot store &T items;
map them to owned values first. sum, min, max and filter_value read
through the reference, as in the serial form.
An owned list is consumed: each chunk moves its items out of the list’s rows,
and the list is dropped after every chunk has finished, which drops any items a
chunk did not consume (after an early any/all exit or a callback error, or
in chunks never submitted). Each item is dropped exactly once.
filter_value(value=True) follows the serial rules: value is evaluated once,
on the calling thread, takes the item type, and is compared with ==. Every
chunk compares against that one value, so the items (or, for &T items, the
referenced values) must be copyable (scalars, str, and tuples of them); use filter for other item types. The value must
also pass the worker checks.
Chunks. The source is split into contiguous chunks, and each chunk runs the
fused steps as one pool job. By default the chunk count is the
pool’s worker count, read when the pipeline runs. .par(&pool, chunks=N) sets
it explicitly; N has type Positive[i64]. A positive literal converts at
compile time; zero or negative literals are compile errors. Convert a computed
count once with Positive.try_from(count)?, which returns RangeError.OutOfRange
for zero or negative values. An input with fewer items than the count gets
one chunk per item, so no chunk is empty. Chunk boundaries depend only on the
input length and the count, never on scheduling.
Terminals.
| Terminal | Result |
|---|---|
collect() | Result[list[U], PoolMapError], in input order |
try_collect() on Result[U, E] items | Result[list[U], ParallelError[E | AllocError]] |
count() | Result[i64, PoolMapError] |
any(keep), all(keep); any(), all() on bool items | Result[bool, PoolMapError] |
sum() on integer items | Result[T, PoolMapError | OverflowError[T]] |
sum() on float items | Result[T, PoolMapError] |
min(), max() | Result[Option[T], PoolMapError] |
reduce(f) | Result[Option[T], PoolMapError] for (T, T) -> T; Result[Option[T], PoolMapError | E] for (T, T) -> Result[T, E] |
try_collect reports the error at the lowest input index, across chunks, as
map_result does. Chunk
allocation failure inside a worker is reported as
ParallelError.Worker(AllocError). As in the serial form, any() and all()
without a predicate test bool items. any and all evaluate every chunk;
there is no early stop across chunks.
Integer sum retains each chunk’s mapped numeric values, waits for the jobs
to finish, then adds the values serially in input order. It returns the first
OverflowError[T], with the exact wrapped result of the failing addition,
whenever the serial left-to-right running total would leave the type’s range.
Later cancellation does not erase an overflow; an incoming negative total can
rescue a prefix that would overflow if its chunk started at zero. This ordinary
Result contract is independent of the caller’s default, checked: or
unchecked: mode. Map and filter callbacks are not replayed to find the error;
on a successful mapping pass each runs exactly once per item reaching it.
Allocation or submission failure can prevent some items from being visited.
Floating-point sum adds each chunk serially, then adds the chunk totals in
input order, preserving IEEE behavior (including infinity and NaN). It can
differ from the serial sum, since chunk totals regroup the additions; it is
deterministic for a given input length and chunk count.
With the default count, a different pool size can change the result; pass
chunks=N when the same answer is needed from any pool.
min and max take each chunk’s serial result and combine them in input
order, so they equal the serial results: the first NaN in input order wins,
and -0.0 is less than 0.0. Empty input gives Nothing.
reduce(f) takes a named function (T, T) -> T or
(T, T) -> Result[T, E]. The latter returns
Result[Option[T], PoolMapError | E], preserving E without wrapping it in
ParallelError. Like float sum, it reduces each
chunk serially, left to right, on its worker, then reduces the chunk results in
input order, skipping chunks with no items; empty input gives Nothing. No list
of all items is built. The result is deterministic for a given input length and
chunk count, but it can differ from the serial left fold when f is not
associative, and with the default count a different pool size can change it.
Code that needs one grouping for any pool or chunk count can collect the items
and use pool.reduce_tree,
whose fixed adjacent-pair tree does not depend on the worker count. Neither
reduction uses integer sum’s serial-prefix grouping.
For a fallible reducer, each chunk stops at its first callback error. All accepted chunk jobs drain before inspecting their Results. The first failed chunk in input order wins, before any cross-chunk combining calls. If every chunk succeeds, chunk results combine in input order until the first error. Infrastructure failure from the mapping pass returns its existing PoolMapError before inspecting chunk Results. Sources, captures, unused values and errors are released exactly once, and borrowed sources can be reused after return. Empty input and singleton chunks invoke no local combining callback; a single surviving item invokes no callback at all. Inline closures and indirect Callable reducers remain unsupported; a named generic function is accepted.
Errors and order. Every terminal reports PoolMapError.Shutdown for a
closed pool and PoolMapError.Allocation when chunk storage or job submission
cannot allocate. Callbacks may run in any order and on any worker. Every
submitted chunk finishes before the terminal returns, on success, on a callback
error and after a failed submission; this is what lets chunks read the source’s
rows without owning them. Integer sum overflow is checked only after that
mapping pass has drained. Partial outputs and captures are dropped once.
Enumeration index overflow remains a deliberate integer overflow process
trap, as in serial enumerate, even in a checked: caller. It is not a
recoverable integer-sum overflow: an arbitrary enumeration start can overflow
while splitting chunks or advancing their indices.
Arguments are evaluated in written order, as in a serial pipeline: the source,
the enumerate start, the pool and chunks arguments of par, then each later
step’s value or callback captures and the terminal’s arguments, left to right.
The pool stays borrowed until the terminal returns, so a later argument cannot
move it.
Cost. Splitting a list or range allocates one small list of chunk bounds
without copying the source’s rows. Each chunk is one pool job,
and its worker reads the items from the source’s rows. Collecting terminals allocate one
output list per chunk, then append each chunk’s storage to one reserved result
list. Integer sum also allocates a numeric output list per chunk and retains
all surviving mapped values until the ordered addition pass; this uses O(n)
temporary numeric storage, not just O(chunks) totals. It does not build a
flattened list. The scalar additions themselves allocate nothing, but the
parallel terminal is allocating and fallible. Float sum and reduce allocate
nothing beyond the chunk lists and the jobs.
Each chunk is still a pool job, so with fewer than a few thousand cheap items the job overhead exceeds the work. See the parallel pipeline measurements.
Cost
A local (non-parallel) pipeline itself adds no heap allocation or per-adapter
generator frame.
Source expressions, callbacks, and destructors retain their own costs. A
generator source still pays its normal resume cost. Collecting reserves once
for a known-length range or list pipeline containing only map, take,
skip, enumerate, or chain of two known-length sides; a filter, filter_value, zip,
flat_map, or unknown-length source grows as it goes.
A terminal over a chain runs one loop per side, so it tests no flag per
item; when zip or flat_map follows the chain, and in for loops,
collect, comprehensions, and generator conversion, one loop tests a flag per
item. flat_map adds the callback and
the inner source’s own setup per outer item: a list-returning callback
allocates one list per outer item, while a range-returning callback allocates
nothing.
filter_value adds one == comparison per item and no call; for scalar items
the optimized loop matches a hand-written if *item == value loop.
Closure literals in a chain add no call. Optimized native code also
substitutes small, straight-line scalar leaf functions and stored closures at
their call sites (at most 24 meaningful operations). Immutable scalar captures
can stay in registers. Arithmetic checks and source locations are preserved.
Larger named callbacks and stored closures retain direct calls; fusion alone
does not guarantee inline-loop performance for every body.
The reproducible benchmark
covers named callbacks, closure literals, scalar captures and a branching
callback.
Explicit conversion adds one suspension boundary for the whole pipeline, rather than one frame per adapter. A generator source inside that frame still has its own suspension boundary. Local fluent consumption keeps its fused loop without relying on generator-frame fusion as an optimization.
The proposal describes the broader
design; B75 tracks follow-up work. Pipelines
replace the removed std.iter generator functions.
Built-in facilities and interoperability
Operation contracts state what they consume or borrow, whether they allocate or block, and what remains after failure.
- Built-in types and operations: numbers, text, tuples, lists, dictionaries, sets, and ranges.
- Input, output, and files: arguments, streams, whole-file helpers, and owned file handles.
- Concurrency and parallelism: certified workers, ownership transfer, channels, pools, cancellation, and bounded waits.
- C interoperability: trusted imports, text and ownership adapters, library exports, metadata, and runtime loading.
These APIs follow the ownership and error/allocation rules. They need no import; modules that do are in the standard library.
Built-in types and operations
These operations are available through the prelude or compiler-known type and method names. Their contracts complement the language rules for types, ownership, and iteration.
- Numbers: parsing, checked integer conversion, limits and constants, powers, float functions, and scalar formatting.
- Quantities: numeric values with checked dimensions, coherent-value access and unit conversion.
- Formatting: representations and printed values.
- Tuples: fixed products and component access.
- Boxes: explicit heap ownership and content access.
- Shared owners: one value with several counted owners.
- Text: Unicode values and character lookup.
- Byte literals: immutable
bytesvalues, byte lookup, and iteration; distinct from ownedstd.bytes.Bytesbuffers. - Text search, building, and splitting.
- Collections: common behavior, followed by construction and copying, lists, dictionaries, and sets.
- Ranges: allocation-free integer sequences.
An operation’s allocation guarantee excludes constructing its arguments and
running user cleanup. Result and Option wrappers add no allocation. Operations
that observe inputs leave them available; owned arguments follow ordinary move
or copy rules. References are accepted where an operation observes a value,
unless its contract says otherwise.
Numbers
The integer types are i8, i16, i32, i64, u8, u16, u32, and u64;
the floating-point types are f32 and f64. Literal typing, arithmetic,
comparisons, and casts follow the numeric language rules.
There is no builtin int type or implicit numeric widening.
Parsing
T.parse(text: str) -> Result[T, ParseError] observes its input and allocates
nothing. Parsing trims Unicode White_Space. ParseError.Invalid means malformed
input; OutOfRange means a valid number outside the target’s range. Both errors
are allocation-free.
Integer parsing accepts an optional ASCII sign and decimal ASCII digits, and checks the requested width. Empty input, prefixes, underscores, and non-ASCII digits are invalid.
Float parsing accepts ASCII decimal/exponent forms and case-insensitive inf,
infinity, and nan, with an optional sign. It rounds directly to the requested
width. Finite overflow returns OutOfRange; underflow may round to zero. Signed
zero survives. Separators and hexadecimal forms are invalid.
Checked conversion
T.try_from(value) -> Result[T, RangeError] is defined for every integer type
T and takes one value of any integer type. It returns Ok with the same
mathematical value when T can represent it and Err(RangeError.OutOfRange)
otherwise. RangeError has just that one variant, so an exhaustive error match
needs one arm. Conversion allocates nothing and does not call the
runtime. Float and non-numeric arguments are compile-time errors, as is
try_from on a float type; float conversions use casts.
NonZero.try_from(value)
fails with the same error for zero.
Parsing still returns ParseError, which distinguishes malformed text from
an out-of-range number. A function that parses and then narrows can return
Result[T, ParseError | RangeError] and propagate both operations with ?,
preserving each cause. It can instead explicitly match the conversion error and
return Err(ParseError.OutOfRange) when that mapping is its intended contract;
there is no implicit conversion between these distinct error types.
Wrapping conversion
T.wrapping_from(value) -> T is defined for every integer type T and takes
one value of any integer type. It keeps the low bits of the value’s
two’s-complement representation, sign-extending a narrower signed value first,
so the result is the value modulo 2 to the power of T’s width:
u8.wrapping_from(300) is 44, u64.wrapping_from(-1) is
18446744073709551615, and i8.wrapping_from(200u8) is -56. It never fails,
allocates nothing, and does not call the runtime. Float and non-numeric
arguments are compile-time errors, as is wrapping_from on a float type.
The bare cast T(value) is the form that
requires the value to fit.
Limits and constants
T.MIN and T.MAX are the smallest and largest values of an integer type T.
Float types provide T.PI, T.E, T.TAU (nearest to π, e, and 2π), T.INF
(positive infinity), and T.NAN (a quiet NaN). Each constant has type T, so
u8.MAX - small needs no cast. Float types have no MIN or MAX, and integers
have no float constants. Names are upper-case; unknown names are compile-time
errors listing the supported constants.
Powers
base.pow(exponent) is defined on every numeric type and takes one exponent of
the receiver’s type. There is no ** operator.
For integers the result is exact. A result outside the type returns
Err(OverflowError[T]) from the enclosing function by default; its declared
Result must include that error. Inside a
checked: block, it exits
that block instead. Inside an
unchecked: suite, it
traps with integer overflow. A negative exponent always traps with
negative integer exponent, since it has no integer result. x.pow(0) is 1 for every x, including zero.
Convert to a float for a fractional or negative power.
For floats pow is the platform math library’s pow: (2.0).pow(0.5) is the
square root of two, and a negative base with a fractional exponent gives NaN.
An integer literal receiver needs parentheses, as in (2).pow(10), because
2. begins a float literal. A suffixed literal does not: 2u8.pow(7).
Float functions
These methods are defined on f32 and f64 and return the receiver’s type,
except div_checked, which returns a Result. None traps or allocates;
results outside a function’s domain are NaN or an
infinity, following IEEE 754 and the platform math library.
| Method | Result |
|---|---|
x.sqrt() | Square root; NaN for a negative x |
x.exp() | e to the power x |
x.exp2(), x.expm1() | 2 to the power x; e to the power x minus 1, accurate near zero |
x.log() | Natural logarithm; -inf for zero, NaN for a negative x |
x.log2(), x.log10(), x.log1p() | Base-2, base-10, and natural log of 1 + x (accurate near zero) |
x.sin(), x.cos(), x.tan() | Trigonometric functions of x radians |
x.asin(), x.acos(), x.atan(), y.atan2(x) | Inverse trigonometry in radians; atan2 preserves quadrant and signed zeros |
x.sinh(), x.cosh(), x.tanh() | Hyperbolic functions |
x.asinh(), x.acosh(), x.atanh() | Inverse hyperbolic functions |
x.cbrt() | Real cube root, including negative inputs |
x.floor(), x.ceil(), x.trunc() | Whole value toward negative infinity, positive infinity, or zero; still a float |
x.round() | Nearest whole value, choosing even at halfway values; preserves signed zero |
x.fabs(), x.copysign(y) | Absolute magnitude; magnitude with the sign bit of y |
x.pow(y) | x to the power y |
x.fmod(y) | Remainder of x / y with the sign of x; NaN when y is zero |
x.div_checked(y) | Result[F, ZeroDivisionError]: Ok(x / y), or Err when y is 0.0 or -0.0; an infinite or NaN y divides normally |
An integer receiver is a compile-time error naming the float cast. % remains
integer-only; fmod is the float remainder. floor and ceil return floats,
and the cast i64(x.floor()) produces the integer.
Each binary method takes one argument of the receiver’s type. Unary methods
take no arguments. Generic FloatType functions can use every method.
The imported std.math functions wrap these primitives with
recoverable domain and overflow errors and checked integer rounding.
Formatting
str.from(number_or_bool) -> Result[str, AllocError] observes one scalar.
Integers use decimal; floats use shortest round-trip debug formatting, including
a decimal point for whole finite values, -0.0, inf, -inf, and NaN;
booleans use True and False. str(value) is not a cast; its diagnostic
names str.from.
Formatting uses bounded temporary storage and fallible output construction; small text results require no heap allocation. Aggregate formatting and format specifications are unsupported here; an f-string applies a compile-time spec.
Tuple operations
Tuples provide fixed component access with an integer literal index; negative
literals count from the end, so t[-1] is the last component.
Construction, indexing, unpacking, and component-loan rules are defined in
tuple types.
Supported tuples compare structurally without allocation. copy duplicates copyable components and mutable contents fallibly; formatting displays supported component values. Types containing stored recursive data are excluded from these automatic traversals.
A tuple can store concrete owned callbacks of different types. Reusable calls borrow their indexed component; consuming calls require extraction by unpacking. See stored callable values.
Boxes
This page describes concrete Box[T]. Explicit Box[P](value)? for a protocol
creates a protocol object with
indirect method dispatch and no payload extraction or copying.
Box(value) -> Result[Box[T], AllocError] moves a value into one heap allocation.
Failure drops the input. Boxes are affine. Moving, borrowing, extracting content,
and destroying the box add no allocation; explicit copying and operations on its
content retain their own costs. Boxes provide a heap boundary for
recursive data. A box has one
owner; Shared[T] has several.
For insertion into a borrowed structure, allocate a detached node before moving existing contents into it. Otherwise a failed allocation can destroy the old contents as part of its input. The insertion lesson demonstrates preservation without adding a successful-path allocation.
Where a T is required, Box[T] converts to T by moving out the content and
freeing the box, without failure. This applies to arguments, returns, annotated
bindings, assignments, constructor fields, and collection elements. Borrowed
boxes convert to references to their contents. Nested boxes convert through
every level.
Without an expected content type, the box remains a box: other = box moves its
owner. Operators and conditions do not convert it. *box explicitly extracts
the content; &*box and &mut *box borrow it. Matching an owned box consumes it;
matching a borrowed box lends its content.
Class fields and methods are reached directly through boxes. Builtin list, dictionary, set, and string methods also reach through nested boxes. Calling a method on a boxed binding, field, or element borrows its content and preserves the box. Mutation requires a mutable owner or exclusive reference.
Read-only methods accept owned temporary boxes, keeping content alive through the expression. Mutation needs a binding, field, or element rather than a temporary receiver. Read-only builtin methods also accept boxed projections from temporary collections or classes: the temporary owner stays alive until the end of the full expression, without copying its content or requiring a named binding:
def main() -> Result[(), AllocError | IoError]:
print([Box([1, 2]?)?]?[0].get(1))?
Ok(())
Option[i64].Some(2)
A returned reference to a box still needs a named binding before calling a builtin method directly on it. See returned references for temporary origins and projected receivers.
Automatic copy, equality (including membership), print, and str.repr reject types whose stored values can recur, including through containers or standard sums. Use explicit traversal to inspect such values. Automatic box cleanup allocates nothing and does not recurse on the native stack; user methods and destructor hooks retain ordinary recursion and effect rules.
Shared owners
Shared[T] is one heap value with several owners. Each owner is an affine
handle; the value is released when the last handle drops, on whichever thread
drops it. Use it to give pool jobs, or any owners that cannot borrow from one
scope, the same atomic, mutex, or immutable value, or a value that synchronizes
itself, such as an HTTP client
and its connection pool. It is a prelude name, like Box.
| Operation | Result | Effect |
|---|---|---|
Shared[T](value), Shared(value) | Result[Shared[T], AllocError] | Move value into one new allocation |
handle.share() | Shared[T] | Another owner of the same value |
handle.field, handle.method(...) | as for T | Use the content through a shared borrow |
handle[i] | element | Copy a copyable element; borrow an affine one with &handle[i] |
&*handle | &T | Borrow the content |
*handle | T | Copy copyable content |
with handle.lock() as value: | binds value: &mut V | Lock a Shared[Mutex[V]]; see mutexes |
Construction makes one allocation, a counted box; failure drops the input and
returns AllocError. share() increments the count and allocates nothing.
Dropping a handle decrements it; the last drop runs T’s destructor once and
frees the allocation. The count is atomic, so handles may be moved into, shared
with, and dropped on any worker.
Access
A handle gives only shared access to its content. Every access borrows the
content through *handle, and that loan is rooted at the handle’s binding:
while a field reference, element reference, or &*handle is live, the handle
cannot be moved, dropped, or reassigned. The handle may be a &Shared[T]
reference; share() through it makes a new owner.
The content changes only through types that synchronize their own mutation:
an Atomic, a Mutex,
a channel endpoint, or a cancellation token. &mut *handle, assignment to a
field or element, &mut self methods, and mutating collection methods are
rejected. *handle copies copyable content; moving affine content out is
rejected, since other owners may still use it.
Where a &T is expected, &handle converts to &T as a borrowed
Box does: a &T parameter or an annotated r: &T = &handle
borrows the content, and the loan is rooted at the handle as &*handle’s is.
The same holds where a view is expected: &handle of a Shared[list[T]] is a
&[T]. The conversion is shared only, so &mut handle for a &mut T
parameter is rejected. There is no conversion from Shared[T] to T.
A temporary handle, such as Shared(5)?.share(), has no binding to root a
borrow at and must be named first.
Rules
- Handles have no equality, formatting, or
print, including when stored inside other values. Compare or print values read through the handle. copy()of a value that contains handles, such as a tuple, class, enum payload, list, or dictionary, copies its owned parts and gives each nested handle another owner, asshare()would. The content is not copied and the count is shared. Sharing a handle allocates nothing; the copy allocates only for its owned parts.copy(handle)on a handle itself is rejected: writehandle.share(). Slices copy their elements the same way.Tfollows the storage rules of a box: no references, generators, or borrowed environments.Tcannot reach aThreadPoolExecutororFuture. The last owner could drop on one of the pool’s own workers, which would then wait for its own pool.Tcannot reachShared[T]again through aMutexor a channel endpoint. Their contents change after construction, so a value could come to hold a share of itself and never be released. Immutable sharing, as in a tree or DAG ofSharednodes, is accepted: each node exists before any share of it. Link back to a parent or a sibling with an index into an owning list instead; see recursive data. There is no weak handle. For the same reason aBoxedCallablecannot be stored behind such a mutex or channel: its captures are not visible to the check.
Rules are checked at each construction, with the concrete T.
Threads
A handle crossing a thread boundary is checked like any other value: its
content must be worker-eligible. Because the last
owner can drop on any holder’s thread, the content’s destructors are checked as
if every holder could run them. Mutexes reachable through a handle count for the
compiler’s lock checks: a join, result wait, or pool shutdown
while holding a lock on a Mutex[V] that a worker can reach through a share is
rejected, and so is nesting a lock through a handle inside a lock on another
Mutex[V] of the same V, since two handles can name one mutex.
Lowering
A handle is a pointer to the same counted record as a box: a header with an
atomic reference count and destructor, the content’s type, and the content
slot. share() is a relaxed atomic increment; a release decrements with
release ordering, and the last owner acquires before destroying. Borrowing the
content is an address computation. See the
memory architecture.
Formatting values
str.repr(value) -> Result[str, AllocError] observes a printable value, including
references, and produces its representation. Strings are quoted and escaped;
aggregates display their structure. Direct Result wrappers can be inspected this
way. Formatting uses checked buffer growth and preserves the input on failure.
print(values...) -> Result[(), IoError] observes its arguments, writes a trailing
newline, and uses raw top-level strings. Printing an existing string needs no
temporary formatting allocation; other values use checked growth. Formatting
failure emits no bytes; output errors can leave a written prefix.
Zero arguments print a newline. Multiple arguments evaluate once from left to right before output, with one space between values and raw top-level strings. Their source loans remain live through argument evaluation and printing; later arguments cannot mutate an earlier observed owner. Printing keeps every owner intact and does not build an intermediate tuple or list. Multiple values use a checked formatting buffer, even when all arguments are strings.
Print rejects a direct or borrowed Result in any argument: handle it first, or intentionally
format the wrapper with str.repr(result) and handle that formatting Result.
Option and printable aggregates containing Results are allowed. Their Results
must be handled or explicitly dropped, following the
error rules.
Quantities display their
coherent number and unit at the top level. Debug output, including in
aggregates, uses Quantity[T, D](value). These builtin spellings keep the
ordinary print and str.repr error types.
User formatting methods
print, str.repr and f-strings share one formatting path. A class or enum
controls its spelling with two methods that write to a std.fmt.Formatter:
displaywrites the textprint(x)andf"{x}"produce at the top level;debugwrites the textstr.repr(x)andf"{x!r}"produce, and the spelling ofxas an element of any formatted aggregate.
A type without display uses its debug; a type without debug uses its
structure. Strings are raw at the top level and quoted inside aggregates, so
print(["a"]) writes ["a"].
from std.fmt import Formatter
class P:
n: i64
def display[W](self, out: &mut Formatter[W]) -> Result[(), FormatError]:
out.format(f"<{self.n}>")
def debug[W](self, out: &mut Formatter[W]) -> Result[(), FormatError]:
out.format(f"P#{self.n}")
def main() -> Result[(), AllocError | IoError | FormatError]:
p = P(3)
print(p)?
print(str.repr(p)?)?
print([P(3)]?)?
print(f"{p} {p!r}"?)?
Ok(())
<3>
P#3
[P#3]
<3> P#3
When a value’s type, or any type it holds, has one of these methods, the
method can fail, so the operation’s error widens with std.fmt.FormatError:
print returns Result[(), IoError | FormatError], str.repr returns
Result[str, AllocError | FormatError], and an f-string returns
Result[str, AllocError | FormatError]. Other values keep the narrower types:
Result[(), IoError], Result[str, AllocError] and Result[str, AllocError].
A method’s error is returned unchanged and nothing is printed. The method
runs with no byte budget beyond the operation’s own and the nesting depth of
an f-string field (32); a private type’s method runs with its own module’s
access, so an aggregate holding another module’s private type still formats.
display and debug are reserved for this form. Formatting a type whose
method of either name has another signature is a compile error that names
the type, the method and the required signature; rename the method if it
serves another purpose. std.log.Logger (a debug that logs) and
std.cli.Progress (a display that draws) are such types, so they cannot be
printed.
main’s error report and runtime trap messages do not call these methods;
they use structure (issue #634).
f-strings build a str
from fields, with Python format specs checked at compile time. A field without
a spec uses the same spelling as print.
A @redacted class or
enum without the method in use formats as its type name followed by (@redacted) in str.repr, print,
f-string fields with any conversion or spec, and main’s error report, at any
nesting depth. The marker uses the same checked buffer as ordinary formatting.
Automatic formatting rejects stored recursive types even when an individual value
is shallow. Print selected scalar fields or write an explicit traversal.
A top-level closure, generator or scoped task, and sums that hold one (for example
SpawnError[OnceClosure[[], i64]]), cannot be formatted by str.repr. The
diagnostic names the unformattable kind and, when it is nested, the containing
type: match the value and format its parts, or call a closure and format its
result. print renders a closure as an opaque marker.
Numeric spelling follows scalar formatting; stream behavior follows
standard output.
Text
str is immutable valid UTF-8 representing Unicode scalar values. Embedded
U+0000 is ordinary content (\0 is a literal escape). Equality and hashing
compare all bytes without normalization; strings have no public terminator.
Ordering (<, <=, >, >=) is lexicographic by Unicode scalar value,
matching the order used by list[str].sort(). Prefixes precede longer strings;
comparison observes both inputs and allocates nothing.
len(text) counts scalars, including combining marks individually, rather than
bytes or grapheme clusters. text.byte_len() -> i64 is the length of the UTF-8
encoding in bytes: len("héllo") is 5 and "héllo".byte_len() is 6. It
observes the receiver, takes constant time, and allocates nothing; use it to
enforce a size limit before copying or encoding text. str has no len
method, so the two counts cannot be confused at a call site.
s[a:b]? copies the scalars from a to b into a new str;
&s[a:b] borrows them in place as a &substr, which loops, compares, and
searches without allocating. Both count bounds in scalars; see
substring views.
String literals require no allocation. Results of at most seven UTF-8 bytes, including every single character and the empty string, also require no heap allocation. Longer constructed results use fallible storage. This threshold does not change the public type or the result type of an operation.
Escapes
String literals and f-string text accept the escapes \0, \n, \r, \t,
\\, \', \", and \u{H...}. The last names one Unicode scalar by code
point with 1 to 6 hexadecimal digits of either case, so invisible and combining
scalars stay visible in source:
def main() -> Result[(), IoError]:
print(len("e\u{301}"), "e\u{301}" == "é", "\u{1F600}".byte_len())?
Ok(())
2 False 4
It is the spelling the !a conversion
writes, so !a output pastes back into source as the same text. A surrogate
(\u{d800} to \u{dfff}) or a value above \u{10ffff} is not a scalar and is
rejected, as are \u without braces, an empty or longer digit run, and every
other escape, including Python’s \x, \uXXXX, \U, and \N{...}. A
byte literal rejects \u{...}: it holds
bytes, so write the UTF-8 encoding with \xHH.
| Source | Diagnostic |
|---|---|
"\u200d" | `\u` needs 1 to 6 hexadecimal digits in braces, as in `\u{200d}` |
"\u{d800}" | `\u{d800}` is a surrogate code point, not a Unicode scalar value |
"\u{110000}" | `\u{110000}` is beyond the largest Unicode scalar value, `\u{10ffff}` |
Character lookup
text[index] -> str accepts an i64 scalar index, with negative indices counted
from the end. A missing index traps. text.get(index) -> Option[str] instead
returns Nothing, including for either extreme i64 index.
Both observe the text and index once in source order, accept references, and produce an independent single-scalar string without allocation. ASCII lookup is constant-time; other text requires scanning to the scalar boundary. String iteration likewise yields one-scalar strings without allocation.
ord(text) -> Result[u32, ParseError] converts a one-scalar string to its
Unicode code point. Empty text or multiple scalars return ParseError.Invalid.
chr(code: u32) -> Result[str, ParseError] performs the reverse conversion;
surrogate code points and values above U+10FFFF return ParseError.OutOfRange.
Both allocate nothing, including for non-ASCII scalars. Unsuffixed chr
arguments infer u32; other integer widths require explicit conversion.
Indexing and iteration retain the existing single-scalar str representation;
there is no separate char type.
Classification
text.isascii() -> bool is true for empty text and for every ASCII character,
including controls and NUL. text.isspace() -> bool requires nonempty text whose
characters all have Unicode’s White_Space property. NUL and U+200B are not
whitespace. The bundled Unicode tables also govern trimming; Python’s additional
whitespace classifications are not promised.
text.isdigit() and text.isalpha() require nonempty text consisting entirely
of ASCII digits (0–9) or ASCII letters (A–Z, a–z), respectively.
They deliberately do not implement Python’s Unicode digit/letter predicates.
All classification methods observe their receiver, take no arguments, and allocate nothing.
text.lower() and text.upper() return Result[str, AllocError], applying
Unicode’s per-scalar case mappings without locale-specific or contextual rules.
Mappings may expand a scalar (ß becomes SS in uppercase). They observe the
receiver and allocate only the final output, fallibly; short results stay inline.
They do not normalize or implement case folding.
Other operations are grouped under search,
building, and splitting.
Text search
All these methods observe their strings once in source order, accept references, and allocate nothing. Matches are literal UTF-8, with no normalization, case folding, or regex interpretation. Optional bounds are unsupported.
| Operation | Result | Behavior |
|---|---|---|
text.startswith(prefix) | bool | Exact prefix; empty prefix always matches |
text.endswith(suffix) | bool | Exact suffix; empty suffix always matches |
text.find(needle) | Option[i64] | First matching scalar position, or Nothing |
text.rfind(needle) | Option[i64] | Last matching scalar position, or Nothing |
text.count(needle) | i64 | Number of non-overlapping matches, left to right |
Each method takes exactly one string argument. Prefix and suffix checks do not
accept tuples of alternatives. An empty needle matches at zero for find, at
len(text) for rfind, and at all len(text) + 1 scalar boundaries for count.
The last rfind match can overlap an earlier match. Converting a found byte
position to a scalar position scans the preceding prefix.
needle in text and needle not in text provide substring membership. A
substring view
&s[a:b] supports in, startswith, endswith, and find, with a str or
&substr argument.
Building text
These methods observe their inputs once, receiver first, and return
Result[str, AllocError]. Inputs remain unchanged on success and failure; the
result outlives them. References are accepted. Length/layout overflow returns
CapacityOverflow; storage exhaustion returns OutOfMemory. Only the final
text needs storage, with no intermediate strings; small outputs
require no heap allocation. The one exception is join over a generator,
described below.
| Operation | Behavior |
|---|---|
text.concat(other) or text + other | Concatenate two strings |
separator.join(parts) | Insert the separator between consecutive elements of an iterable of str |
text.repeat(count) | Repeat exact contents; count is i64; zero or negative yields empty |
text[start:stop:step] | Slice by scalar position, with Python’s clamping, omitted bounds, and steps |
text.slice(start, stop) | Forward scalar slice with required i64 bounds |
text.replace(old, new) | Replace all non-overlapping literal matches, left to right |
text.removeprefix(prefix) | Remove at most one exact prefix |
text.removesuffix(suffix) | Remove at most one exact suffix |
text.strip() | Remove Unicode White_Space at both ends |
text.lstrip() / text.rstrip() | Remove White_Space at the left/right end |
Empty input to repeat produces empty output for any count.
Joining
join accepts every iterable whose for element
is str, and joins exactly the pieces that loop would yield:
| Argument | Pieces | Allocations |
|---|---|---|
list[str] | Elements in order | One, for the result |
set[str] | Elements, in no promised order | One, for the result |
dict[str, V] | Keys in insertion order | One, for the result |
str | Unicode scalars, so "-".join("abc") is "a-b-c" | One, for the result |
Generator[str], including a generator expression | Yielded values | Scratch growth plus one for the result |
An empty iterable produces empty text; a single piece adds no separator. Empty
pieces are retained. Other element types are rejected: write an f-string or
convert with str.from to join numbers.
Collections and strings are observed, with or without &. join measures
them before allocating, then copies once.
A generator is one-shot, so join consumes it as a for loop does: a named
generator moves (sep.join(g)), and a reference to one is rejected. Its total
size is unknown in advance. Pieces are appended to a scratch buffer whose
capacity starts at the first piece’s size (at least 8 bytes) and at least
doubles each time it fills. The result is then copied into exact-size text and
the scratch is freed before join returns. Producing n > 8 bytes takes at
most ceil(log2(n / 8)) + 2 allocations and holds less than 3n bytes at the
peak. For example, four 10-byte pieces with ", " use scratch capacities of
10, 20, 40, and 80 bytes, then one 46-byte result. Each yielded piece is
released once it is copied. On failure the generator stops where it is; its
remaining state drops with it, and no partial text is returned. To bound the
cost, collect the pieces into a list first or use
std.text.join with a byte limit.
Choose by the shape of the text:
- an f-string for a fixed
number of pieces of mixed types, such as
f"{host}:{port}"?. It reportsAllocError, plusFormatErrorwhen a field runs a user formatting method or a spec is computed; concat(or+) for two strings in anAllocErrorfunction;joinfor any number of strings from one iterable.
slice includes start and excludes stop. Negative bounds count from the end;
bounds clamp to [0, len(text)], without overflow for extreme i64 values.
Reversed bounds yield empty text. ASCII bounds map directly to bytes; other
text requires scanning scalar boundaries. Slice syntax text[a:b:c]? follows the
same scalar positions and also accepts omitted bounds and a NonZero[i64] step;
text[::-1]? reverses the scalars.
replace observes receiver, old, and new in that order. Inserted text is not
searched again. An empty old string matches every scalar boundary, including
both ends; on empty input it inserts new once. Empty new text removes matches.
There is no count limit, regex interpretation, or normalization.
Prefix/suffix removal uses exact strings, including NUL, without repeated removal. Empty or missing patterns preserve the contents. Trimming takes no character-set argument, leaves interior text unchanged, and uses the same White_Space classification as isspace.
Splitting text
Both operations observe their text and arguments without consuming them. They
return independently owned lines or pieces in Result[list[str], AllocError].
All list storage and long output strings are constructed fallibly. Failure
reclaims the initialized result prefix and preserves the inputs.
Explicit separators
text.split(separator) requires one nonempty string, including a reference.
A literal "" separator is a compile error, split requires a nonempty separator. A computed empty separator traps; this invalid-argument error is
not an AllocError.
Maximum-split arguments are unsupported.
Matches are literal, non-overlapping, and scanned left to right. Leading,
trailing, and adjacent separators produce empty pieces. Empty input yields
[""]; no match yields one piece containing the input’s contents. Multibyte
separators and NUL work without normalization.
Whitespace
text.split() with no separator splits on runs of Unicode White_Space, using
the same bundled property as isspace() and trimming. It omits leading,
trailing, and repeated empty pieces; empty or whitespace-only text yields an
empty list. NUL and U+200B are ordinary content. The output remains fallible
and independently owned, like explicit-separator splitting.
Lines
text.splitlines() discards terminators. text.splitlines(True) retains each
original terminator without translation; the optional argument is bool and
positional.
Recognized boundaries are LF, CR, CRLF, vertical tab, form feed, U+001C–U+001E, U+0085, U+2028, and U+2029. Empty text yields no lines; interior blank lines are retained; a final terminator adds no extra empty line. This is broader than file universal-newline reading, which recognizes CR/LF only.
Collections
For primitive-field class records scanned by field, see columnar lists. Their packed columns and borrowed scalar pipelines have a deliberately smaller interface than ordinary lists.
list[T], dict[K, V], and set[T] have concrete element types, including
nested collections. Dictionary keys and set members are integers, bool, str,
bytes, or tuples, classes, enums and anonymous sums recursively containing those types.
Every class field and every payload of every variant must be a supported key
type. Hashing includes class fields, variant tags and payload contents,
independently of where the value is stored. Ordinary classes move into a
collection; @copy classes keep their usual copy behavior. Stored keys are
exposed only through shared borrows, so their identity cannot change in place.
Floats, collections, boxes and recursive types are not supported keys.
Keys with custom __del__ cleanup are supported, including nested cleanup,
but cannot be duplicated: copying collections, non-mutating set algebra and
key-view to_set() reject these keys. Borrowing and ownership transfers work normally. User-defined hash/equality hooks remain unsupported. See the
heterogeneous collections tutorial.
Unit elements are rejected. Sets of supported finite enums or tuples with at most 256 possible values
use bitsets instead; they do not hash or request entropy.
Dictionary and hash-backed set bucket selection uses SipHash-1-3 over the complete key,
the schedule Rust’s HashMap, CPython and Swift use. A str, bytes,
integer or bool key is hashed as its contents alone; keys nested in tuples, classes and
enums are framed by field count, variant tag and string length. One secret 128-bit key is
obtained from OS entropy on the process’s first hash operation and shared across
worker threads. Initialization and hashing allocate no runtime storage.
Constructing or reserving an empty collection does not need entropy. The first
hash may wait for the OS entropy source to initialize. If the OS cannot
provide the key, the operation traps with collection hashing requires OS entropy; there is no predictable fallback. Interrupted reads retry.
Randomized bucket placement makes precomputed hash-flooding inputs harder to construct; it does not guarantee worst-case constant-time lookup. Hash values are private implementation details, not persistent identifiers or cryptographic digests. No test-mode fixed seed is needed: dictionary insertion order is independent of buckets. Set iteration order remains unspecified.
Lists preserve order and duplicates. Dictionaries preserve first insertion order: replacing a value keeps its key’s position. Sets remove duplicates and promise no iteration order. Equality is structural: list order matters; dictionary insertion order and set order do not. Structural comparisons allocate nothing. Collections have no identity test.
len(collection) -> i64, in, and not in observe collections. Dictionary
membership tests keys. Indexed reads use i64 list indices (including negative
indices) or the dictionary’s key type; missing keys and out-of-bounds indices
trap with exit status
one, reporting the source position. An index is one expression evaluated once; multiple
comma-separated indices are rejected. std.bytes.Bytes
uses the list index rules for buffer[i] reads and writes.
len(value) also calls an accessible ordinary def len(self) -> i64 method
on a class or enum, including standard-library containers such as
Deque and Bytes.
It borrows the receiver, including a temporary, and evaluates it once. A mutable
or owned receiver, extra parameters, method type parameters, or another return
type does not meet this contract; call such methods directly instead. The method
body determines its cost and whether it allocates. Built-in collections retain
their intrinsic length operations. Shared arguments follow the ordinary
call borrowing rules:
len(&"ada"), len(&b"abc"), and len(&make_collection()) are valid.
Borrowing itself does not allocate; producing the argument may allocate.
in and not in do not dispatch to methods on classes or enums, even if they
have an ordinary contains method. Use the container’s explicit API; for
example, Bytes.find searches a byte range.
A type being iterable or indexable does not imply that it supports membership.
Collections move on assignment, owned calls, and returns. Independent duplication uses copy. Indexed reads do not extract affine owners: use an explicit element reference, an owned removal operation, or copying. Element loans protect the whole collection unless they use distinct nonnegative integer literal list indices. See borrowing and iteration.
Flat unpacking also consumes a list: a, b = values requires exactly two
elements and transfers their owners; _ drops an ignored element. A length
mismatch traps at the unpacking expression before any target is written.
Unpacking itself allocates nothing. Borrowed lists, nested targets, and starred
targets are unsupported; explicitly copy a list first to preserve it.
replace(collection[index], value) transfers the previous element out while
writing its replacement. It requires exclusive access, evaluates the replacement
before the target index, and adds no allocation. List indices must exist.
Dictionaries reject it, as they reject all
indexed writes; use replace(*value, replacement) on a
reference from get_mut.
Shared mutation rules
Mutation requires a named mut binding, mutable class field, or exclusive
reference. Arguments and indices evaluate before the target is exclusively
borrowed, so items.append(len(items)) is valid. Active conflicting loans
still reject mutation; temporary mutation receivers are unsupported.
Copying and borrowed lookup
Optional lookups named get, and peek for containers with an order such as
caches and heaps, return a copy: dictionary.get(key) -> Option[V] and
list.get(index) -> Option[T]. Indexing traps on a missing key or index
instead.
A copying get has no hidden cost. It accepts only
copyable values:
scalars, immutable strings, ranges, function values, instances of @copy
classes, and tuples or enums of those. Copying one is a fixed-size copy; a
string shares its immutable storage. Neither allocates nor duplicates a buffer.
Values that own storage, including every collection and every class instance
not declared @copy, are affine, and get rejects them
at compile time (“get cannot return owned dictionary values”), so owned data is
never copied by accident.
Copying is the default because the common case reads a small value and uses it
directly, as in counts.get(key, 0) + 1. A borrowing get would need a
dereference at each such use, and its get(key, default) form could not exist:
returning either the stored value or the default would choose between two
borrows, which returned references
do not allow.
Borrowed forms take a suffix: get_ref and peek_ref for shared references,
get_mut and peek_mut for exclusive ones. Builtin and standard library
containers follow the same rule, so the method name tells the reader whether
they hold a copy or a borrow. User functions can return
Option[&V]. Of the
builtin containers, only dictionaries provide an optional borrowed lookup,
get_mut; get_ref is not provided yet.
collection.clear() -> () removes all entries while retaining capacity and
allocating no runtime storage. Removed owners drop before return: lists in
element order, dictionaries in insertion order with keys before values, sets in
unspecified order. Independent immutable owners survive. Empty clear is harmless.
See construction and copying for constructors, growth, reservation, and failures, then lists, dictionaries, and sets for their operations.
Columnar lists
clist[T] stores each field of T in a contiguous column of native-width
values. Choose it for repeated scans of a few fields in many rows. Ordinary
list[T] remains useful for general objects and whole-record access.
T must be a fully defined, nonempty class containing only bool, signed or
unsigned 8/16/32/64-bit integers, f32, or f64. Custom destructors and fallible
initializers are rejected. Strings, nested classes, enums, nullable fields,
collections and references are not supported. Creating a clist or copying or
removing a row requires access to every field; field selection respects module privacy.
Construction and mutation
| Operation | Result and behavior |
|---|---|
clist[T]() or clist[T].new() | Empty owner, without allocating |
clist[T].with_capacity(n) | Result[clist[T], AllocError] |
len(items) or items.len() | Live row count as i64 |
items.capacity() | Row capacity as i64 |
items.reserve(additional) | Result[(), AllocError]; capacity for len(items) + additional rows |
items.append(row) | Result[(), AppendError[T]]; moves the row |
items.remove(i) | Allocation-free Option[T]; takes the row and preserves remaining order |
items.clear() | (); empties the owner without releasing its capacity |
items[i].field | Checked scalar read; negative indices count from the end |
items[i].field = value | Checked scalar replacement |
items[i].field += value | Numeric augmented assignment; evaluates the index once |
items.copy_row(i) | Allocation-free Option[T], independent of subsequent writes |
items[a:b:c] | Result[clist[T], AllocError]; a slice copying the selected rows; items[:]? duplicates the owner |
items.column(T.field) | Local, one-shot Iter[FieldType] borrowing the owner |
Capacity growth allocates replacement storage, copies each column, and publishes
the new storage only after success. Failure preserves length, capacity and all
existing values. Negative or overflowing capacity requests return
AllocError.CapacityOverflow. Capacity may exceed the requested size.
AppendError.OutOfMemory(row) and AppendError.CapacityOverflow(row) preserve
the complete submitted row for retry. Successful append moves its scalar fields
into the columns without rerunning an initializer. An append within reserved
capacity allocates nothing. The empty owner and all live rows are cleaned up on
every ordinary exit; moving a clist transfers its storage.
Field replacement evaluates the RHS before its indices, following ordinary
assignment. Enclosing owner indices precede the row index, and each is evaluated
once. Augmented assignment evaluates these indices, reads its current value,
then evaluates the RHS under a shared owner loan before writing. Out-of-bounds
indices trap at the source position; copy_row and remove return Nothing instead.
remove accepts an i64 index; negative indices count from the end. An invalid
index leaves the owner unchanged. The returned row is an independent ordinary
T, assembled without rerunning its initializer. Removal shifts later rows
left in every column. clear makes all columns empty. Both require mutable
access, allocate nothing, and retain capacity for later appends. Neither changes
the owner pointer or column starts. Removing a row requires access to every
field, just like copy_row; clearing does not expose any field.
A slice reserves exactly the selected row count, then copies each column; a
step-1 slice copies each column as one block. It reads every field, so it needs
the same access as copy_row, and the receiver must be a named clist.
Column iteration and ownership
T.field is a compile-time selector accepted only by column. It is not a
string lookup or a general reflection value. Each column plan borrows the entire
clist until consumed. While that loan is live, reserve, append, remove, clear, field writes,
movement and destruction of the owner are rejected. Several shared column plans
can be zipped together. Their lengths and row order agree.
Columns yield scalar values, not &Field references. The source remains
borrowed, but each element is loaded directly from its native-width buffer.
Ordinary Plenty references address tagged value slots; treating packed bytes as
those slots would be invalid. Use scalar callback parameters with existing
map, filter, fold, zip and other local pipeline operations.
Recreate a column plan for a second traversal. Plans cannot escape into stored
values or suspended generators.
Column iteration is currently rejected inside generator functions, even if the
scan would finish before a yield.
items[i], &items[i], packed field references and direct row-method calls are
unsupported: there is no contiguous T at that row position. copy_row creates
an ordinary owned T on which methods can run; writes to that copy do not write
back. Whole-clist copy, mutable column views and general
row iteration are not part of this initial interface. Equality compares row
values; formatting reports clist(len=N) without materializing rows.
Layout and cost
The current implementation uses one allocation for an owner header and all
columns. Every column contains tightly packed elements (bool uses one byte).
Capacity rounding aligns column starts; this is an internal layout, not an
Arrow, Parquet, C ABI or serialization contract. Growth copies all columns,
and can temporarily hold both old and new allocations.
Ordered removal moves (len - normalized_index - 1) values in each column,
so its cost grows with the number of following rows and the total native row
width. Removing the last row only gathers that row. Repeated removal near the
front can be quadratic; this is not a bulk filtering operation. clear takes
constant time because all fields are primitives with no destructors. It does
not erase old bytes, shrink storage, or allocate; subsequent appends overwrite
the newly live rows. Neither operation uses tombstones.
Column loops load the pointer and length before iteration. Checked arithmetic and floating-point rules are
unchanged. Indexed field reads and writes use checked native-width loads and
stores. copy_row checks its index once, then gathers the known fields directly
into fresh ordinary row storage without runtime schema dispatch. Each access
observes the current owner, so growth and mutation between accesses are safe.
Whole-row workloads can still cost more than ordinary lists: gathering separated
fields and moving the reconstructed object have costs of their own. The
access measurements
compare these paths with the initial implementation.
Automatic SIMD counts
With the default optimized profile, a column of i32 or u32 followed by one filter and
count() uses four-lane SIMD when the predicate compares its borrowed argument
against a constant or immutable by-value capture of the same type
(==, !=, <, <=, >, >=). Either operand order is recognized. The
predicate may also join two such comparisons with and, for example
*x >= low and *x < high. Captured bounds may be runtime values; the closure
retains its own snapshot. This bounded pattern accepts at most two captures,
both of the column’s scalar type. Named functions and closures are recognized.
For example:
items.column(T.score).filter(def(x: &i32) -> bool: *x > 5i32).count().
The ordinary scalar program remains valid with --fast-compile.
The compiler proves this small pattern from the checked operations; it requires no user annotation, temporary collection, allocation, aligned column start or lane-count bookkeeping. Full groups use unaligned-safe vector loads; the scalar loop handles the remaining zero to three rows. Counts use 64-bit accumulators, including for columns larger than a 32-bit lane counter could represent. The generated instructions use the existing x86-64 SSE2 baseline, so this slice needs neither a newer CPU nor runtime feature dispatch.
Borrowed or mutable captures, or, more than two comparisons, checked arithmetic
in predicates, additional adapters, other scalar types and other terminals
retain scalar execution. Only total comparisons qualify: eagerly combining their
SIMD masks cannot introduce effects or traps from a skipped and operand.
Side effects, traps and
floating-point order are unchanged. This is a bounded optimization, not general
auto-vectorization; parallel scans remain unimplemented. See the
validation and measurements.
See the tutorial and the design rationale and measurements.
Collection construction and copying
Constructors and displays
list[T].new(), dict[K, V].new(), and set[T].new() return
Result[collection, AllocError]. with_capacity(capacity: i64) has the same
result and reserves at least that many entries, including hash storage.
Aliases expose these constructors too. Empty construction allocates its owner;
negative counts and impossible layouts return CapacityOverflow; exhaustion
returns OutOfMemory. Partial storage is reclaimed on failure.
Small enum bitsets need only the owner allocation;
valid capacity hints require no additional storage because all possible members
already fit in the membership mask.
[elements], {key: value}, {elements}, and typed empty constructors such as
list[i64]() also return Result[collection, AllocError]. {} is a dictionary.
Entries must have identical types. Empty displays require contextual types,
such as the annotation in names: list[str] = []?; a typed constructor such as
list[str]()? needs none. Nonempty displays infer from their first entry unless
context supplies the type.
Already typed entries never implicitly narrow.
List displays reserve space for all entries before evaluating any of them;
their element buffer does not grow during insertion. Other displays and
comprehensions start with an empty owner and grow as needed. Entries evaluate
left to right after initial storage succeeds. A failed
insertion stops evaluation and destroys the initialized prefix; unevaluated
entries have no effects. Nested fallible expressions require their own ?.
A display’s error is always AllocError, so ? on it needs a function
returning Result[_, AllocError] or an error sum containing AllocError, such
as Result[_, AllocError | IoError] when it also prints. Otherwise match the
result and convert the error explicitly.
Propagation leaves the enclosing function, not merely the display. Comprehension
evaluation follows the iteration rules.
A directly iterated nonempty i64 list display with constant integer entries
at a constant step, such as for n in [2, 4, 6]?:, is lowered to a counted
loop without collection storage. Repeated entries, descending sequences,
singletons, .unwrap(), and reversed(...) are supported too, provided the
step fits i64. Borrowed sources, other element types, nonuniform sequences,
and expressions with effects retain ordinary materialization. This is an
allocation optimization: the source still needs valid Result handling,
including a compatible enclosing error type for ?. Use range when a
sequence is naturally described by bounds and a step.
Reservation and insertion
These exclusive methods return Result[(), AllocError]:
| Receiver | Operation | Behavior |
|---|---|---|
| Any collection | reserve(additional: i64) | Room for at least this many entries beyond current length |
| List | append(value: T) | Append one element |
| Set | add(value: T) | Insert if absent |
| Dictionary | insert(key: K, value: V) | Insert or replace without changing existing key order |
dictionary.get_or_insert(key: K, default: V) follows the same growth and
failure rules but returns Result[&mut V, AllocError]: a present key keeps its
entry and drops both arguments; see
dictionary lookup.
Growth reserves all required storage before mutation. Failure preserves contents,
order, and lookup behavior; capacity may change. Negative or impossible counts
return CapacityOverflow. A successful reserve permits the specified number of
subsequent insertions without storage allocation. Existing-key replacement and
duplicate set insertion do not grow storage. An equal incoming key is destroyed;
the existing key owner is retained. This can run custom cleanup even when the
collection length does not change.
Insertion consumes its arguments on both outcomes; failure drops owned inputs. Reserve before moving an input that must survive a storage failure. The Result remains fallible even when no growth is needed.
Conversion
list[T].from(source) and set[T].from(source) consume an owned collection,
range, or generator and return Result[collection, AllocError]. list(source)
and set(source) infer the element type; list[T](source) and set[T](source)
specify it explicitly. These call-style constructors also accept str and
bytes: list("aé")? produces ["a", "é"], and list(b"AB")? produces
[65u8, 66u8]. The .from(...) methods do not accept those two source types.
dict(d) transfers a dictionary.
Set conversion retains the first occurrence of each permitted member.
Failure destroys partial output, the current element, and the remaining source; generator effects already performed are not rolled back. Borrowed sources and pair-iterable dictionary construction are unsupported.
Explicit copying
copy(value) -> Result[T, AllocError] observes its operand once, including
references to T. Mutable contents are recursively duplicated; immutable
strings can share storage. Scalars and wholly immutable values need no
allocation. Classes with custom cleanup, aggregates containing them, and
generators are uncopyable. A nested Shared handle gains another
owner, as share() would, and its content is not copied, so content behind a
handle may be of any type. copy(handle) of a handle itself is rejected; write
handle.share().
A failed copy preserves the source and releases newly initialized contents. Partial records do not run whole-record destructors. Copying follows the ownership rules; it introduces no mutable aliases.
Lists
Lists use the common collection rules. Growth through
append, reserve, and extend is recoverable; indexed replacement allocates
nothing and traps on a missing index. Lists have no * repetition; a
comprehension such as [0.0 for _ in range(count)]? builds repeated elements.
With --unchecked assume, list indexing lexically inside a plain
unchecked: suite
requires a nonnegative index below the list length. Reads, writes and borrows
omit both the bounds check and negative-index normalization; violating this
precondition is undefined behavior. unchecked trap: and default builds keep
normal indexing behavior. The optimizations below describe checked accesses.
Inside for and while loops, scalar indexing of a named list already in
scope reads the list’s length and row address once, before the loop, and then
uses checked direct loads and stores. An immutable shared reference always
qualifies. An owned list, exclusive reference, or retargetable binding
qualifies only when the loop cannot resize, replace, or move the list: the loop
makes no call and performs no list operation other than scalar element reads,
writes, and length queries. Each list keeps its own bounds check, and a
negative index is still normalized; unrelated lists need not have equal
lengths. Generator bodies and other loops keep ordinary indexed access.
An access that cannot fail has no bounds check. In
for i in range(start, len(xs)) with a nonnegative constant start and
step 1, xs[i] cannot fail when xs is a &list[T] parameter or a
slice view: its
length cannot change while the function runs. The stop may also be a
local set from len(xs) earlier and not changed since, as in
n = len(xs) followed by for i in range(n). With the stop written as
len(xs) for the same list xs, an owned list, an exclusive reference, or a
shared reference made inside the function also qualifies when the loop uses
the row cache described above: the loop cannot change the list, so the cache
takes its length from the range’s stop; a length saved in a local before
the loop does not qualify for these lists. Indexing any other list with i,
a range with any other stop, or a loop that can resize the list keeps the
check, so a length read before a clear() still traps. The counter of
for i, x in enumerate(xs) over such a list or view qualifies the same way,
when the call has no start; indexing another list with it still checks the
bound but skips negative-index normalization.
for x in xs over an owned list reads the list’s length and rows once,
before the loop, like iteration over a borrowed list: the loop owns the
list, and no other code can reach it. Each element read has no bounds check,
and the position advances without an overflow check.
A while lo < hi loop over a slice view checks once, before the loop, when
its body begins by indexing the view with hi, and possibly lo, as
v.swap(lo, hi) or t += v[hi] does. The view must be one the loop does not
reassign, lo must start nonnegative and only grow, and hi must change only
by checked subtraction of constants. Each pass’s hi is then at most its
first value, and lo is below hi. The check runs only when the loop does,
and fails with the first pass’s message and site. A body that starts with any
other operation that can fail, such as t * 10 + v[hi], keeps its checks.
Any other for i in range(start, stop) loop with step 1 and a nonnegative
start is compiled twice when its body is small, makes no call, and has no
nested range loop, as in for i in range(n): dst[i] = src[i]. start is
nonnegative when it is a constant, a range counter, or a checked sum of such
values, as in for j in range(i + 1, n). One test before the loop compares
stop with the length of each list or view the body indexes by i or i + k,
for a constant k from 0 to 64, and checks each index that no pass changes,
such as bodies[i] in that inner loop. When every index is in bounds, the copy
without those checks runs; otherwise the checked copy runs, so a body that
breaks, returns, or fails before reaching a bad index behaves as before, and
the bad index traps with its usual message and site. A negative unchanged
index also selects the checked copy, which counts it from the end. The length
must be read once before the loop: a borrowed list or view, or a list that
uses the row cache described above. A division keeps its own checks in both
copies. The body limit, 256 operations, bounds the code size the second copy
adds. An overflow check on i + k with k > 1 remains.
A while loop without the lo < hi form above is compiled twice under the
same conditions. Either kind of loop may hold a nested while loop. Its test
covers each unchanged index and each nonnegative constant index, such as
perm[0] in while perm[0] != 0:; a negative constant index keeps its check
in both copies. A nested
loop also sees the lengths its enclosing loops never change: with
n = len(u), a for j in range(n) loop inside for i in range(n) reads
u[j] and u[i] without checks.
Storage costs
Scalar lists store elements at their declared widths: one byte for bool, i8
and u8; two for i16 and u16; four for i32, u32 and f32; eight for
i64, u64 and f64. Thus 128 live u8 elements occupy 128 bytes of row
storage, and 128 i64 elements occupy 1,024 bytes. The allocation also includes
a fixed owner, spare capacity from growth, and rounding to 16-byte allocation
units. reserve counts elements, not bytes.
Other elements use one 16-byte value slot plus any inline payload storage;
lists reserve no dictionary value slot. Scalar-only Option and Result
elements still take 16 bytes. Records and multi-field enum variants retain
16-byte field slots, so an optional variant with three bool fields takes
64 bytes per row. These are private implementation layouts, not an FFI ABI.
Element references remain valid under the existing borrowing rules, including
for packed scalar elements. See value representations.
Lookup and removal
| Operation | Result | Behavior |
|---|---|---|
items.get(index: i64) | Option[T] | Observe an element; Nothing outside the list |
items.pop() | Option[T] | Remove and own the last element |
items.pop(index: i64) | Option[T] | Remove and own the selected element |
items.count(value) | i64 | Number of equal elements |
items.find(value) / items.rfind(value) | Option[i64] | First/last matching position |
Indices can be references; negatives count from the end. get and pop handle
extreme i64 indices safely. Neither allocates; a missing pop leaves the list
unchanged. pop requires exclusive access, preserves remaining order and capacity,
and supports all list element types. Last-element removal touches no surviving
rows and is constant-time in the list length, including for inline elements.
Other positions shift and repair only subsequent rows. Extracted inline payloads
are copied into the caller’s own storage before another list mutation or
destruction; the freed capacity can be reused by a later append.
get observes the list and index once in source order. It supports only
non-affine values, including copyable enums; retained results survive replacement
or destruction of the list. Use pop or explicit references for affine elements.
There is no default argument.
Search/count accept exactly the element type, including references, and support
integers, floats, booleans, and strings. They observe both operands once, allocate
nothing, and use IEEE float equality (NaN never matches; signed zeros do).
Aggregate searches, optional bounds, and throwing index are unsupported.
Reordering and extending
items.reverse() -> () requires exclusive access and reorders all element types
without allocation, copying, retaining, or dropping them. Empty and singleton
lists are unchanged; later destruction follows the new order.
items.swap(i: i64, j: i64) -> () exchanges two elements under the same
contract: exclusive access, every element type, and no allocation, copy, retain,
or drop. Negative indices count from the end, and a missing index traps. Both
indices are evaluated before the list is borrowed, so they may read it. Equal
indices leave the list unchanged. swap is the way to exchange owned elements,
which cannot be moved out of indexed storage. On a list of numbers or bool,
swap compiles to inline bounds checks, loads, and stores with no runtime call;
other element types call the runtime once. An exclusive
slice view, &mut [T],
has the same swap, with indices counted within the view.
items.sort() -> () orders the elements ascending, in place, without allocating.
It requires exclusive access and one of these element types:
| Element type | Order |
|---|---|
| Integers | Numeric |
f32, f64 | Numeric, with -0.0 before 0.0 and every NaN after all numbers |
str | By code point; a prefix comes before the longer string |
| Tuples of these types | By the first component, then the next where those are equal |
Every other element type, including bool, classes, enums, and collections, is
a compile-time error. Elements that compare equal are indistinguishable, so the
result is fully determined. items.sort(True) orders descending; an omitted or
False argument orders ascending. Sorting takes O(n log n) comparisons.
Wide inline rows use three-way introsort with insertion sort for small partitions
and a heapsort fallback; rows without inline payloads use Rust’s unstable slice sort. Row swaps
repair inline addresses immediately, without retaining or dropping elements.
The wide-row algorithm uses O(log n) stack space and no heap workspace. A tuple
of numbers and strings has its field offsets resolved once per sort.
items.sort_by_key(key) -> () accepts any element type. The key is a named
function, a reusable shared-call closure, or an anonymous function written in
place, such as items.sort_by_key(def(x: &i64) -> i64: *x); it takes &T for the element type T and returns an
integer, float, string, or tuple of those. items.sort_by_key(key, True) orders
descending. The list is exclusively borrowed for the operation; callbacks
cannot also borrow it through a capture. Keys use the same total order as
sort, including signed zero and NaN placement. The runtime sorts the rows
with the wide-row introsort above, for every element width, and calls a
generated comparator that computes both keys. The sort is unstable: equal-key
elements may change relative order. It allocates no buffer and recomputes keys
during O(n log n) comparisons, so the callback should give consistent keys and
remains responsible for its own explicit allocations. An inconsistent key
yields an unspecified order but keeps every element.
A key returning Result[K, E] is also accepted: the sort then returns
Result[(), E] instead of (). The first error stops further key calls;
the runtime finishes its safe row moves and returns that error unchanged.
The list may already be reordered, but every element remains owned exactly
once. Both keys and errors may own strings. Callers handle this public result
normally, even inside unchecked:; no private callback channel is involved.
For example, a negating key declares
def(x: &i64) -> Result[i64, OverflowError[i64]]: Ok(-*x).
One-shot and mutable-call closures and reference-valued keys are unsupported.
Elements are moved by row exchange, with no copying or early destruction.
sorted(items) -> Result[list[T], AllocError] observes a list and explicitly
copies it before sorting by natural order. sorted(items, True) gives a
descending copy. The source survives either result; the element type must
support copying and natural ordering. For key ordering of a copy, use
mut result = copy(items)? followed by result.sort_by_key(key).
User-defined comparison functions and class ordering protocols are unsupported.
items.extend(other) -> Result[(), AllocError] consumes a same-typed owned list
on both outcomes. It reserves the entire additional count before transfer.
Success moves elements in order without cloning; failure preserves destination
contents and drops the source. Sufficient capacity, including empty sources,
needs no allocation. Source references, arbitrary iterables, and self-extension
by moving the destination are rejected. Use extend(copy(source)?) to preserve it.
Slices
Slice syntax items[start:stop:step]?
copies the selected elements into a new list with Python’s clamping, omitted
bounds, and negative steps. It duplicates elements with copy() and Shared
handles with share(), and leaves the source unchanged; element types that
copy() rejects are a compile error.
items.slice(start: i64, stop: i64) -> Result[list[T], AllocError] is the
transfer form for an owned temporary. It takes both
bounds, observes receiver and bounds once in order, and accepts references.
Start is inclusive; stop is exclusive. Negative bounds count from the end and
clamp to [0, len(items)]; reversed or extreme bounds safely yield the corresponding
clamped slice. It has no step.
Non-affine elements are retained into an independent list. Affine elements
require an owned temporary receiver: selected owners transfer, and the remainder
drops in source order. Failure drops the whole temporary; observed sources remain
unchanged. copy(items)?.slice(start, stop) explicitly preserves an affine source.
All output storage is reserved before transfer: two allocations for a nonempty
slice, one for an empty owner.
Sorting design alternatives
User-defined ordering and Timsort are unimplemented proposals for comparison semantics and stable sorting respectively. They do not change the current sorting contract above.
Dictionaries
Dictionaries use the common collection rules.
insert(key, value) adds a key or replaces its
value. Counting uses *counts.get_or_insert(key, 0)? += 1, which looks the
key up once; see lookup and removal. The ? exposes
allocation failure.
Dictionaries have no indexed assignment. dictionary[key] = value,
dictionary[key] += n, and &mut dictionary[key] are compile errors whose
messages name insert or get_mut: key presence is a run-time fact, so a
write through an absent key would trap where --check cannot see it.
Reading dictionary[key] remains, and traps on an absent key.
Writing through dictionary[key].field or calling a mutating method on
dictionary[key] is rejected the same way; use get_mut for one key, or
a mutable value loop for every value.
def main() -> Result[(), AllocError | IoError | OverflowError[i64]]:
mut stock = {"apples": 3}?
stock.insert("pears", 5)?
match stock.get_mut("apples"):
case Some(count):
*count -= 1
case Nothing:
pass
print(stock)?
Ok(())
{"apples": 2, "pears": 5}
Updating every value
for value in (&mut dictionary).values(): lends each value as &mut V in
insertion order; for key, value in (&mut dictionary).items(): adds a shared
&K key. &mut dictionary.items() is the same loop. Each step reaches the
next entry directly, with no key lookup, item tuple, or copy, so it works for
affine values and allocates nothing. Inside a function taking
d: &mut dict[K, V], write (&mut d).values(); d.values() stays a shared
loop.
class Player:
name: str
hp: i64
items: list[str]
def main() -> Result[(), AllocError | IoError | OverflowError[i64]]:
mut players = {"ada": Player("Ada", 3, []?), "lin": Player("Lin", 9, []?)}?
for player in (&mut players).values():
player.hp += 10
player.items.append("potion")?
for key, player in (&mut players).items():
if *key == "lin":
player.hp = 0
break
for key, player in players.items():
print(*key, player.hp, player.items)?
Ok(())
ada 13 ["potion"]
lin 0 ["potion"]
The loop holds an exclusive loan of the whole dictionary until it ends, so
the body cannot insert, remove, clear, read, or borrow the dictionary again,
including through get_mut or a nested loop; these are conflicting borrow
errors. break, return, and ? end the loan without dropping or moving any
value. The references are loop-local: storing one in a collection, assigning
it to an outer reference binding, or returning it from a function that does not
take the dictionary as its first reference parameter is rejected. A mutable
projection is an immediate loop source; (&mut dictionary).values() cannot be
bound to a name. There is no values_mut() or items_mut(): the exclusive
receiver selects the mutable loop, as for x in &mut xs does for a list, and
those names are compile errors that point here. To change values while also
adding or removing keys, collect the keys first with keys_snapshot() and use
get_mut per key.
Lookup and removal
dictionary.get(key) -> Option[V] observes receiver and key once in source
order, including references. It copies or retains only non-affine values;
collections, classes, and enums containing affine payloads are rejected.
Lookup allocates nothing. Results outlive source replacement or destruction.
Copying and borrowed lookup
explains why get copies and how borrowed lookups are named.
dictionary.get(key, default) -> V accepts a second positional argument of
the value type and returns it only when the key is absent. Receiver, key, and
default evaluate once in that order; the default is eager even for a present
key. Lookup still requires a copyable value and adds no allocation. Zero,
False, and an empty string are present values. This does not insert the key.
A stored Nothing produces Some(Nothing), distinct from a missing key.
dictionary.get_mut(key) -> Option[&mut V] lends the value for key
exclusively, or returns Nothing. It works for every value type, including
affine ones, and allocates nothing. The key is observed before the receiver is
borrowed. The result holds an exclusive loan of the whole dictionary until its
last use, as a reference-returning method’s result does, so the dictionary
cannot be read, mutated, or borrowed again meanwhile. Match on the result to
update a value in place: *count += 1 for a scalar, items.append(x)? for a
list, or point.x = 0 for a class. replace(*value, new) returns the previous
value and keeps the entry’s position.
dictionary.get_or_insert(key, default) -> Result[&mut V, AllocError] lends
the value for key exclusively, first inserting key and default when the
key is absent. Key and default evaluate in that order before the receiver is
borrowed, and both move into the call. A present key keeps its stored entry
and value; the call drops the incoming key and the unused default before
returning. Either way the key is hashed once, so counting or grouping needs one
lookup per item rather than the two of get_mut followed by insert. A
missing key may grow the dictionary as insert does; on AllocError the key
and default are dropped and the dictionary is unchanged. The result holds the
same whole-dictionary loan as get_mut’s. The default is eager: it is built
even when the key is present, so prefer get_mut when building it costs an
allocation and the key is usually present.
def main() -> Result[(), AllocError | IoError | OverflowError[i64]]:
mut counts: dict[str, i64] = {}?
for word in "to be or not to be".split()?:
*counts.get_or_insert(word, 0)? += 1
mut lengths: dict[i64, list[str]] = {}?
for word in counts.keys():
group = lengths.get_or_insert(len(*word), []?)?
group.append(*word)?
print(counts, lengths)?
Ok(())
{"to": 2, "be": 2, "or": 1, "not": 1} {2: ["to", "be", "or"], 3: ["not"]}
dictionary.pop(key) -> Option[V] requires exclusive access and supports all
permitted value types. A hit removes the entry, drops its stored key, and transfers
the value’s owner into Some; a miss returns Nothing without mutation.
The key is observed once before exclusive receiver access. Removal takes
exactly one key, with no default argument.
Removal allocates nothing, preserves capacity and remaining insertion order, and appends a reinserted key at the end. Expected lookup/removal cost is constant with well-distributed keys, plus hashing/comparison and cleanup; adversarial collisions can make it linear.
Snapshots and views
dictionary.keys() -> KeyView[K] creates a shared, local-only borrowed view
without allocating. A view can be bound to a local, iterated, and used with
builtin set operations. Iteration yields &K in insertion order; use
[copy(key)? for key in dictionary.keys()]? to collect owned keys explicitly,
including move-only class keys without custom cleanup. Keys containing __del__
can be borrowed here but cannot be copied; use into_keys() to transfer them.
len(view) is O(1), and key in view / not in use dictionary lookup.
The value holds a borrowed owner handle with a compiler-known key projection;
the owner retains its row descriptor and each loop keeps its own cursor.
== and != compare keys as sets, ignoring insertion order, against
KeyView[K] or set[K] with the same key type. Dictionaries may have different
value types. issubset, issuperset, and isdisjoint take &set[K] or
&KeyView[K] and return bool. These observations allocate nothing and have
no AllocError; relationships perform O(n) expected hash lookups (equality
first compares lengths). Empty views are subsets of every set-like operand
and disjoint from every operand.
union, intersection, difference, and symmetric_difference take those
same operands and return Result[set[K], AllocError]. to_set() materialises
the keys as that result type. Algebra reserves its result before inserting
keys and leaves both sources unchanged on failure. Move-only keys are copied
into independent result storage. All four algebra methods and to_set() reject
keys containing custom cleanup, because those keys cannot be copied.
Relationship checks remain available. Result order is unspecified.
Operators such as |, &, and <= are not part of the view API.
Set relationship and algebra methods accept mixed set/view operands in either
receiver order. set.intersection_update(view) and
set.difference_update(view) also accept key views. Equality accepts either operand order. These builtin
methods observe their operands, so an explicit & may be omitted without
moving a named set or view.
set(view) and list(view) are not supported. Use view.to_set(),
dictionary.keys_snapshot(), or [*key for key in view]? to choose the owned
result explicitly.
A bound view holds a shared loan on the whole dictionary until its last use.
Insertion, get_mut, get_or_insert, update, removal, clearing, and moving the owner
conflict with that loan, including through aliases. Views cannot be stored in
fields, collections, tuples, enum payloads, generators or worker/closure
captures, or passed to or returned from user functions. Binding a view of a
temporary dictionary is rejected; immediate iteration may borrow a temporary.
drop(view) and copy(view) are rejected like ordinary
references; end the loan by finishing the view’s last use.
Conditional view bindings are not supported. General stored/returned views
remain separate work.
dictionary.values() creates a shared local view yielding &V, without
allocating or copying affine payloads. Both values = dictionary.values()
and items = dictionary.items() can be bound and reused in loops; len(view)
is O(1). iter(values) and iter(items) start pipelines, with item callbacks
taking separate key and value parameters. These views have the same local-only
borrow restrictions as key views and do not provide set operations.
(&mut dictionary).values() yields &mut V in immediate loops; mutable views
cannot be bound. See updating every value.
Automatic box dereferencing applies to keys(), values() and items(),
including nested boxes and borrowed boxed dictionaries. Their loans protect
the boxed owner throughout the observation or loop.
dictionary.keys_snapshot() -> Result[list[K], AllocError] and
dictionary.values_snapshot() -> Result[list[V], AllocError] take no arguments and
preserve insertion order. Each returns a new list, retaining immutable element
storage. Observed dictionaries remain unchanged on failure, and snapshots
survive later source mutation or destruction.
A snapshot with affine selected elements requires an owned temporary: this
applies to keys_snapshot() for move-only keys and values_snapshot() for
move-only values. Those elements transfer and the temporary is consumed on
success or failure. Use copy(d)?.keys_snapshot() or
copy(d)?.values_snapshot() to preserve such a source explicitly. The unselected
side may be affine even when observing a named dictionary. Both snapshots
reserve their complete output before transfer: two allocations when nonempty, one for an empty owner.
dictionary.into_keys() -> Result[list[K], AllocError] and
dictionary.into_values() -> Result[list[V], AllocError] consume a named owner
or owned expression directly, including dictionaries with affine payloads.
They preserve insertion order and transfer the selected elements without copying.
The dictionary is consumed on success and failure. Output reservation happens
before any transfer; allocation failure drops the remaining source exactly once.
Use for value in dictionary.into_values()?: when the dictionary is no longer
needed. This is an owned snapshot, not a stored borrowed view.
dictionary.items() creates a shared local borrowed view, with an independent
cursor for each loop, and creates no item tuples or snapshot buffers. Shared views load
shared references to both keys and values, including scalars and strings;
mutable views lend mutable values while keys remain shared references. Its loop loans follow the
iteration rules.
Transferring an update
dictionary.update(other) -> Result[(), AllocError] requires exclusive access
and consumes a same-typed owned dictionary on every outcome. Existing keys keep
their position and key owner; incoming values replace old values. New keys append
in source insertion order.
All storage for absent keys is reserved before replacement or insertion. Failure preserves destination contents and lookup behavior, though capacity may change, and drops the source. Success transfers entries without copying; old values drop in source traversal order after replacement. Empty and replacement-only updates, or updates fitting reserved capacity, allocate no runtime storage.
Use update(copy(source)?) to preserve the source. References, pair iterables,
and keyword updates are unsupported.
Sets
Dictionary KeyView[K] values borrow
their dictionary’s keys without allocating and provide the same relationship
and algebra methods. Their operations accept &set[K] or &KeyView[K];
equality also compares a key view and a set in either operand order.
Sets use the common collection rules; members are integers,
booleans, strings, bytes, or tuples, classes, enums and anonymous sums recursively containing those types.
Iteration order is unspecified. add and reserve follow
the recoverable growth contract.
Removal and update
set.discard(value) -> bool requires exclusive access. It observes one
same-typed value or reference before borrowing the receiver, removes and releases
the stored member if present, and returns whether removal occurred. A miss changes
nothing. It allocates no storage and retains capacity. Set pop is unsupported.
set.update(other) -> Result[(), AllocError] consumes one same-typed owned set
on both outcomes. It reserves for absent members before changing membership.
Failure preserves destination contents and lookup behavior and drops the source;
success transfers new owners and releases duplicate incoming owners. Empty and
duplicate-only sources, or sufficient reserved capacity, need no allocation.
Use copy(source)? explicitly to retain a source. References and general or
multiple iterables are unsupported.
set.intersection_update(other) -> () keeps common members;
set.difference_update(other) -> () removes members found in other.
Both accept a same-key-type dictionary KeyView as well as a set. They
require exclusive destination access and observe the source before
borrowing the destination. The source remains borrowed during mutation; aliases
into the destination are rejected. Removed members drop, capacity remains, and
the operations allocate nothing. Use clear() to empty a set directly.
Set algebra
Each method takes exactly one same-key-type set or dictionary KeyView
(with an optional explicit borrow), observes both inputs,
and returns Result[set[T], AllocError]:
| Operation | Members selected |
|---|---|
set.union(other) | Present in either input |
set.intersection(other) | Present in both |
set.difference(other) | Present in the receiver, absent from other |
set.symmetric_difference(other) | Present in exactly one input |
Inputs may be the same set and remain unchanged on failure. Output is independent, though immutable strings can share storage. All storage for the actual result is reserved before copying members, including move-only class records. Hash-backed results use four allocations when nonempty, one for an empty owner. Small enum bitsets use one owner allocation regardless of membership. Arithmetic operator overloads and iterable conversions are not implied.
Keys containing __del__ cannot be duplicated, so all four non-mutating
algebra methods reject them, even for empty sets or a currently cleanup-free
variant. copy(set) is also rejected. Use consuming update to combine owners,
and intersection_update or difference_update to filter in place. These
operations preserve each retained key’s ownership and destroy removed or
incoming duplicate owners. Relationship checks only borrow and remain available.
Relationships
issubset(other), issuperset(other), and isdisjoint(other) return bool.
They take one same-key-type set or dictionary KeyView, accept references,
observe both operands once
left to right, and allocate nothing. Empty sets are subsets of every set and
disjoint from every set. Heterogeneous coercion and comparison operators are
unsupported.
Small enum bitsets
A set of a named enum or tuple with a supported finite domain of at most 256
distinct values uses a size-selected membership mask instead of rows and hash
buckets: one 64-bit word for 1–64 values, two words for 65–128, and four words for
129–256. Selection uses the complete type domain, not the number of current
members; clearing or draining a set preserves its selected width. This includes payload-free named enums and named enums or tuples composed recursively
of bool and other such enums or tuples. Count each variant’s payload
combinations, then add the variants: Empty plus Flag(bool) has three values;
a variant with six boolean fields has 64. Option[bool] also has three values.
This is automatic:
write set[Permission], with the same methods and Result handling as any set.
The owner still allocates once; construction, copying and algebra can therefore
return AllocError. Adding members, valid reservation, removal and in-place
updates need no further allocations. Invalid capacity requests still return
CapacityOverflow.
Membership, insertion and removal encode the payload into a dense index and operate on its bit. Encoding visits the active payload’s fields; length and relationships between two such sets use bit operations. Their union, intersection, difference and symmetric difference use word operations plus the result-owner allocation. Iteration visits present values; borrowed iteration still yields shared references, and the borrow checker still protects the source owner. Set iteration order remains unspecified; do not rely on the order seen with a particular representation.
Dictionary key views remain borrowed hash-table views. Their mixed operations
with bitsets work normally, and to_set() can produce a bitset. Dictionary
storage itself is unchanged. Domains larger than 256, integer or string payloads,
and classes keep the ordinary hash-set representation. Anonymous sums also use hash sets, even when their members have finite domains.
Recursive keys remain unsupported. The compiler emits immutable canonical values
for payload domains, including inline record storage, so borrowed iteration needs
no allocation or temporary value cache. This adds static data proportional to the
domain size and value layout. All three mask alternatives fit inside the
existing owner storage; small-domain operations process only their selected words. No target-specific SIMD support is required.
See the measurements and storage tradeoffs
for the per-owner savings and shared metadata costs.
Ranges
range(stop), range(start, stop), and range(start, stop, step) return
range[T] directly. Stop is exclusive; omitted start is zero and omitted step
is one. A range is a small copyable value: construction, passing, returning,
standard-sum wrapping, and repeated iteration allocate nothing. Iterating a
stored range is a counted loop,
like a direct range(...) source. Collecting its elements into a collection is
fallible.
range[T](...) selects any integer type. Start and stop have type T; step is
always NonZero[i64],
permitting descending unsigned ranges. Without explicit type arguments, a typed
bound or range annotation selects T; otherwise it defaults to i64. Bounds
must fit T, so an exclusive u8 stop of 256 is rejected.
A step literal such as -2 converts at compile time, and 0 is a
compile-time error. A computed step is converted first, which handles zero as
an error value:
def evens(stop: i64, stride: i64) -> Result[list[i64], AllocError | RangeError]:
step = NonZero.try_from(stride)?
Ok(list(range(0, stop, step))?)
def main() -> Result[(), AllocError | IoError | RangeError]:
print(evens(9, 2)?)?
print(list(range(5, 0, -2))?)?
Ok(())
[0, 2, 4, 6, 8]
[5, 3, 1]
Passing an i64 step is a compile-time error: range step must be `NonZero[i64]`, not `i64`; write a nonzero literal, or convert a value with `NonZero.try_from(step)?` . A range therefore never has a zero step. A length
exceeding i64 traps. Bounds and length handle the full u64 domain without
intermediate overflow. len, in, not in, and i64
indexing are supported; negative indices count from the end and missing indices
trap. Membership is constant-time.
In a directly written comprehension range, a numeric output annotation can guide the loop variable through direct arithmetic in the output expression:
squares: list[u8] = [n * n for n in range(8)]?
This context does not flow backward through arbitrary calls, filters, stored
ranges, or explicit range arguments. Use range[u8](...) at those boundaries.
Arithmetic retains its fixed-width rules.
Input, output, and files
Text I/O uses strict UTF-8; binary I/O preserves raw bytes.
Both use explicit IoError results. Input consumption
and external writes may have partial effects before failure; errors do not promise
rollback. Error values and their Result/Option wrappers allocate nothing.
Successful writes and ordinary flushing do not promise durable storage.
- Standard streams and arguments.
- Bytes and lossless Unix paths.
- Native paths, directory enumeration, metadata and links, and walking a tree.
- Whole-file helpers.
- Owned files and modes.
- Reading, writing and synchronization, and positions and truncation.
These APIs are prelude builtins. File I/O currently supports Linux;
unsupported hosts report IoError.Unsupported. Standard-stream input uses the
process descriptor on Unix. Configurable encodings, public OS
descriptors, and lazy file iteration are unsupported.
The imported std.fs/std.path modules add native paths, lazy directory streams,
metadata and a Plenty walker. The finder example
searches directory roots through these APIs.
I/O errors
IoError is an allocation-free enum:
| Variant | Meaning |
|---|---|
System(i32) | Native OS error code, only when the OS supplied one |
Data(DataError.InvalidUtf8) | Invalid or truncated UTF-8 |
Data(DataError.Allocation(AllocError)) | Recoverable buffer/result allocation failure |
InvalidMode | Unsupported computed open mode |
Closed | File is closed |
NotReadable | File lacks read capability |
NotWritable | File lacks write capability |
InvalidInput | Invalid argument, such as NUL in a path or negative truncation size |
Unsupported | Unsupported host operation |
Other | Failure without a native code or more specific variant |
Constructing, matching, and propagating these errors adds no allocation.
The ordinary error rules determine whether
details are retained or erased by builtin Failure.
File operations report Closed before mode errors. Reads and writes validate
capabilities even for zero-length work. Negative read limits mean unbounded
reading; they are valid, unlike negative truncation sizes.
An error can follow consumed input, a written prefix, or other external effects. The operation’s page states its cleanup and preservation guarantees.
Standard streams and arguments
Output
write_stdout(text: str) -> Result[i64, IoError] and write_stderr(text)
observe text and write its exact UTF-8 bytes without adding a newline. Success
returns the Unicode-scalar count. Errors can follow partial output.
Stderr writes go straight to descriptor 2, so a closed or read-only stderr is
IoError.System(9) (EBADF). Stdout writes are buffered by the Rust standard
library, which reports EBADF as success; a closed or read-only stdout
discards output without an error.
print(values...) -> Result[(), IoError] formats supported values separated by
spaces and adds a newline; no arguments produce an empty line;
see formatting for allocation and Result restrictions.
flush_stdout() and flush_stderr() return Result[(), IoError], exposing
deferred buffered-write errors. These operations retain the process-owned stream
handles. Success and flushing do not imply durable storage.
Input
input() -> Result[Option[str], IoError] reads one stdin line, removing LF or
CRLF and preserving other characters. Some("") is an empty line; Nothing
is EOF. Decoding is strict UTF-8.
Buffer growth and result construction are recoverable. Failure may consume a prefix or complete line; the reader does not read ahead into the next line. The current Unix backend reads the process descriptor directly. Prompt arguments, bare-CR newline translation and buffered stream objects are unsupported. Use input_bytes for binary records without newline conversion.
Arguments
args() -> Result[list[str], IoError] snapshots all executable arguments,
including the invocation name at index zero. Each call produces an independent
list with fallible text construction. Invalid UTF-8 is rejected rather than
replaced or escaped; failed construction releases partial output. OS argument
storage is only borrowed during construction.
Use args_bytes for lossless Unix arguments and paths.
Bytes and lossless Unix paths
Raw transport uses owned list[u8] values. These are ordinary mutable lists,
with checked indexing, borrowing, and fallible growth; the current representation
uses general list slots rather than a compact byte buffer. A Unix path is a byte
list used without decoding. Filesystem calls reject embedded NUL and otherwise
preserve every byte, including CR, LF, tabs, and invalid UTF-8.
All operations below are prelude builtins. Arguments are observed in source
order and remain available. Results use IoError; allocation failures are
IoError.Data(DataError.Allocation(...)).
| Operation | Result | Contract |
|---|---|---|
encode_utf8(text) | Result[list[u8], IoError] | Independent owned copy of the exact UTF-8 bytes |
decode_utf8(bytes) | Result[str, IoError] | Strict validation and independent text; invalid input returns DataError.InvalidUtf8 |
args_bytes() | Result[list[list[u8]], IoError] | Independent snapshot including invocation name at index zero; no decoding |
input_bytes(delimiter: u8) | Result[Option[list[u8]], IoError] | Read one stdin record, removing only its delimiter |
write_stdout_bytes(source: &[u8]) | Result[i64, IoError] | Exact bytes with no added framing; returns byte count |
write_stderr_bytes(source: &[u8]) | Result[i64, IoError] | Same contract on stderr |
read_bytes(path) | Result[list[u8], IoError] | Read whole file without decoding or newline translation |
write_bytes(path, source: &[u8]) | Result[i64, IoError] | Create or truncate, write exact bytes, check close, return byte count |
The writers read source as a slice view:
pass &bytes for a list[u8] or std.bytes.Bytes, &buffer[a:b] for part of
one, or a b"..." literal. Nothing is copied or converted.
input_bytes(0u8) provides NUL framing for Unix paths, whose names cannot contain
NUL. Some(empty) represents an empty record; Nothing means EOF before any
record byte. A final unterminated nonempty record is accepted. CR and LF have no
special meaning. Input reads no bytes beyond the delimiter and retries interrupted
reads. Allocation or I/O failure can consume part of a record; no rollback or
resynchronization is promised. Construction may fail even at EOF.
Writers read the view in place without allocating an output buffer. They retry
interrupted writes and complete all bytes on success, but errors can follow a
written prefix. Use the existing flush_stdout() or flush_stderr() to report
deferred output errors. Calls and NUL records are not atomic against concurrent
writers.
Binary file helpers use the same Linux open/creation/symlink rules as
text helpers. Path conversion is fallible and precedes opening,
so allocation failure cannot truncate a destination. No output allocation follows
opening. Reads release partial buffers and descriptors on error; successful file
operations check close errors. Success does not imply atomic replacement or durable
storage. File helpers on other hosts return IoError.Unsupported; raw stdin uses
the Unix process descriptor.
For display, decode explicitly or format the byte list with str.repr; keep the
original bytes for filesystem operations. Replacement decoding is never used.
The imported std.path module provides Linux Path owners over this
transport, and std.fs provides directory enumeration and
metadata. std.bytes provides compact buffers that convert
to and from list[u8], and std.io reads and writes them
through files and the process streams without that conversion. First-class byte
views and portable path representations remain within issue #49’s broader scope.
The finder example exercises raw arguments, matching,
and NUL input/output against real Unix filenames.
Whole-file helpers
Paths are UTF-8 strings, absolute or relative to the process working directory.
NUL in a path returns IoError.InvalidInput; ordinary symlink resolution applies.
Arguments are observed once in source order and remain available. All temporary
storage is fallible, and descriptors are closed on success and failure.
| Operation | Result | Behavior |
|---|---|---|
read_text(path) | Result[str, IoError] | Read a whole file, translating CRLF and bare CR to LF |
write_text(path, text) | Result[i64, IoError] | Create or truncate, then write exact UTF-8 |
append_text(path, text) | Result[i64, IoError] | Create or append exact UTF-8 without truncating |
read_text preserves content NUL, validates UTF-8 strictly, and does not remove
a BOM or use a locale. Path conversion, buffer growth, and output construction
are recoverable; errors reclaim partial buffers. It does not promise a snapshot
against concurrent writers.
write_text and append_text return the Unicode-scalar count. Path allocation
precedes opening, so allocation failure cannot truncate or append to the file.
No output allocation follows opening. OS errors can leave a truncated destination
or written prefix. Success checks close errors but does not imply synchronization
or atomic replacement.
Creation uses 0666 restricted by the process umask. Append creates a missing
file even for empty text. Each native write appends at the current end, but one
logical call may use several writes and is not an atomic concurrent record.
For explicit lifetime and cursor control, use File.
For raw path bytes and untranslated contents, use read_bytes and write_bytes.
Owned files and modes
The builtin accepts two positional call forms:
open(path: str) -> Result[File, IoError]open(path: str, mode: str) -> Result[File, IoError]
Both create an opaque, affine file owner. open(path) opens an existing file
for reading, equivalent to open(path, "r"). Like other builtins, open
takes positional arguments only: open(path, mode="w") is rejected. Named
functions and methods accept named arguments and defaults; see
signatures and calls.
Pass an explicit mode positionally. Literal modes are checked at
compile time; computed invalid modes return IoError.InvalidMode.
| Mode | Capabilities | Opening behavior |
|---|---|---|
r / r+ | Read / read and write | Existing file required |
w / w+ | Write / read and write | Create or truncate |
a / a+ | Write / read and write | Create or append |
x / x+ | Write / read and write | Atomic exclusive creation |
Exclusive creation fails for an existing path, including a symlink, without
truncation. a+ starts at EOF; other modes start at byte zero. Append writes use
OS append semantics even after seeking. Paths and creation permissions follow
the whole-file rules. Owner and path storage are allocated
fallibly before opening, so allocation failure cannot truncate the destination.
Files can move, return, live in aggregates, and be borrowed, but cannot be copied.
file.closed -> bool observes state. Printing shows File(open) or
File(closed); equality is owner identity rather than path equality.
file.readable() and file.writable() return Result[bool, IoError] through
shared access, without allocation. They report the open mode, not a guarantee
that a subsequent OS operation will succeed; closed files return Closed.
Closing and resource scopes
file.close() -> Result[(), IoError] requires exclusive access and is idempotent.
It marks the owner closed before closing its descriptor, reports native errors,
and allocates nothing. Automatic destruction closes any live descriptor without
reporting errors or allocating. Closing does not promise durability.
File is an intrinsic context manager: entry lends &mut File, exit closes it,
and destruction releases the owner. with &mut file as stream: leaves the original
owner closed afterward. Entry permits an already closed file; its operations
report Closed.
Exit discards close errors to preserve the body’s control flow. Call close()?
inside the body when those errors matter. All ordinary
resource-scope cleanup rules apply.
Reading files
All reads require exclusive access to a live readable File, even for zero limits. They use strict UTF-8, preserve NUL, and translate LF, CRLF, and bare CR to LF. Output is independent of the file. Buffers, long strings, and list growth are fallible; failures may consume input but leave the file valid for another operation or close. Invalid UTF-8 is reported after consuming input. No rollback or concurrent-file snapshot is promised.
| Operation | Result | Behavior |
|---|---|---|
file.read() | Result[str, IoError] | Read to EOF |
file.read(count: i64) | Same | Read at most count translated Unicode scalars |
file.readline() | Result[str, IoError] | Read one line, retaining its translated terminator |
file.readline(count: i64) | Same | Stop at count scalars or the first translated newline |
file.readlines() | Result[list[str], IoError] | Eagerly collect remaining lines with translated terminators |
Negative read counts mean read to EOF; negative readline counts mean a complete
line. Zero returns empty text without consuming input or a pending LF.
Positive limits never read past a scalar and can leave a partial line for the
next call.
EOF returns "" from read/readline. An empty line is "\n"; a final
unterminated line receives no added terminator. readlines returns an empty list
for empty input and adds no synthetic line at EOF. On failure it destroys the
collected prefix. Bounded readline controls memory better than eager readlines.
At CR, line reading returns immediately. The next read consumes an optional LF; it does not read beyond a line merely to decide whether CR was followed by LF. Saved positions preserve this translation state. File iteration, size hints, and configurable encodings are unsupported. To split text already in memory using broader Unicode line boundaries, use splitlines.
Writing and synchronization
file.write(text) -> Result[i64, IoError] observes text once before exclusive
receiver access. It writes exact UTF-8 without newline insertion or translation,
returning the Unicode-scalar count. The file must be live and writable even for
empty text. No runtime write buffer is allocated; errors can leave partial output.
file.writelines(lines: list[str]) -> Result[(), IoError] observes a list or
shared list reference once before exclusive access. It writes each string’s
exact bytes without separators, preserves the list, and allocates nothing after
argument construction. Empty lists still validate the file and capability.
Errors may leave a written prefix. General iterable sources are unsupported.
Update streams are unbuffered, so switching reads and writes requires no flush. Writes use physical byte positions and can overwrite part of a UTF-8 sequence; they do not insert text. Nonempty writes discard pending CRLF translation state. Append modes use OS append semantics, but a logical write need not be one atomic record against concurrent writers.
Flush and sync
file.flush() -> Result[(), IoError] flushes runtime buffers. Files are currently
unbuffered, so it validates the open state and normally does no extra work.
file.sync() -> Result[(), IoError] requests OS synchronization of data and
metadata. Both require exclusive access and report errors without allocating.
Neither close nor flush substitutes for sync. Filesystem, device, and directory entry durability rules still apply; sync is not an atomic-save operation.
Positions and truncation
file.tell() -> Result[u64, IoError] returns an opaque text-position cookie.
file.seek(cookie: u64) -> Result[(), IoError] restores it. Both require
exclusive access, allocate nothing, and report native seek errors, including
nonseekable streams.
Zero rewinds. Other cookies must come from tell() on the same stream with
unchanged contents. They preserve pending CRLF translation state, rather than
representing a plain byte offset. Arithmetic and persistence of cookies are
unsupported. A failed seek leaves decoder state unchanged; arbitrary cookies
remain memory-safe but can yield invalid text or surprising positions. Relative
and end-relative seeking are unsupported.
file.truncate(size: i64) -> Result[i64, IoError] changes a writable file’s
length in bytes, returns that length, and leaves the cursor unchanged.
Omitting size uses the current physical byte position, not a text cookie.
Negative sizes fail before mutation. The operation requires exclusive access,
allocates nothing, can fail at the OS, and does not synchronize storage.
Extending regular Linux files adds zero bytes. Shrinking can split UTF-8, which later reading reports as invalid data.
Concurrency and parallelism
There is no async/await or event loop in Plenty. The separate async/await exploration includes a standalone spike, not an implemented language feature or an adoption decision.
Concurrency is explicit. Native workers can overlap without an interpreter lock;
ordinary loops and comprehensions remain serial. Ownership transfers and checked
loans still apply across threads. Shared immutable storage does not permit
unsynchronized mutation; std.atomic values are
updated through shared references with atomic instructions, and a
mutex grants one lock holder at a time exclusive access to its value.
A Shared[T] handle gives several owners, such as
pool jobs, shared access to one value; the last owner, on any thread, drops it.
The Go-style scoped-task exploration assesses a possible lightweight alternative, including a fixed-stack native experiment. It introduces no implemented task API or change to these contracts.
Worker eligibility is a closed-world compiler check, not a user-defined marker
protocol. It examines concrete stored types, reachable helper and callback
bodies, captures, results, and destructors, including finite recursive type
graphs. Scalars, strings, ranges, collections, classes, enums, and Shared
handles are eligible when their contents and effects are eligible. A shared
value’s destructor is checked as if any holder’s thread could run it. Stored concrete callbacks retain
their eligibility evidence. Thin Callable values are certified against every
matching function address in the program.
Each native operation has a declared effect. Files, directory streams and child pipes cross threads under the ordinary ownership and loan rules; changing the working directory is rejected. Foreign pointers/calls, uncertified indirect callable effects, and generator frames are conservatively excluded. Imports do not grant thread permission. Ordinary checked standard-stream I/O is usable, with scheduler-dependent ordering. Owned transferred values cannot contain references or borrowed environments.
Scoped threads can borrow parent storage for the task’s lexical scope. Channels and executor jobs require wholly owned storable values. Executors and futures cannot enter worker code, including through containers or messages: worker-side submission and future waiting are unsupported.
- Scoped threads and consuming jobs.
- Bounded channels, including the recommended patterns that keep joins, result waits, and pool shutdowns from waiting on a live endpoint.
- Executors and shutdown, futures, and ordered mapping / fixed-tree reduction.
- Mutexes.
- Cancellation tokens and timed waits and selection.
- Native effects on worker threads and deadline-bounded I/O
through
std.ready.
Allocation guarantees exclude argument expressions, worker bodies, and user cleanup. Error wrappers add no allocation. Blocking operations offer no strict waiter fairness, and application communication can deadlock. Normal scope exits join workers; fatal traps terminate without unwinding. A nonterminating worker prevents its scope or executor from finishing. Worker stack exhaustion terminates the process rather than becoming a job error.
Scoped threads
with spawn(worker, arguments...)? as task: starts a native thread and joins it
before leaving the block. Named functions, including inferred generic instances,
use their ordinary typed parameters. spawn(&job) or spawn(&mut job) runs an
explicitly borrowed reusable zero-argument closure. A binding that already holds
a &Closure or &mut Closure, such as a parameter or a get_mut match binding,
is reborrowed for the task’s scope: spawn(callback) means spawn(&*callback),
or spawn(&mut *callback) when the closure has mutable captures, which requires a
&mut reference. Wholly owned jobs use consuming spawn.
task.join() waits and transfers the result exactly once, consuming the task
binding. A worker returning Result[T, E] yields that same result. Join itself
is infallible under the compiler/runtime contract; unexpected native join failure
terminates the process to avoid reclaiming storage still used by a worker.
Omitting as, or leaving a task unjoined, waits and drops its result at exit.
Application errors disappear with an unclaimed result; join explicitly to inspect
or propagate them. Owned results, including collections, recursive records, and
owned concrete callbacks, can transfer back. References, borrowed environments,
and generator frames cannot be returned from a worker.
Scope and loans
Task bindings are lexical join tokens: they cannot be copied, borrowed, returned,
stored, captured, or passed to a function. spawn must be the direct manager
expression, using ? or .unwrap(). Comma-separated managers form nested scopes.
Each successful task joins on normal completion, return, ?, break, and
continue. Failure to start a later task first joins earlier tasks. Detachment
and forced cancellation are unsupported. A join that could wait for a task
holding a token before the function’s later cancel is rejected; put
stop.cancel_on_exit() after the task to cancel it on every exit. See
cancel on every exit.
A join, explicit or automatic, is rejected while the function keeps a channel endpoint that the worker could be waiting on. See joins and live endpoints for the rule and its recommended patterns.
Argument loans and borrowed capture dependencies last until the end of the
with block, including after explicit join. Shared readers can coexist;
exclusive loans prohibit conflicting access, moves, or drops. Ordinary field
disjointness and enum-payload rules apply. yield inside a task scope is rejected.
Scoped work may borrow or own its inputs. Named arguments use ordinary call
semantics: &T and &mut T borrow, copyable values copy, and affine values move
into the worker. Pool jobs must own their inputs instead; only
.par chunks
read a borrowed list internally, because the terminal waits for every chunk
before it returns. Borrowing callbacks
from a collection protects the collection through scope exit; separate tuple
slots can have disjoint loans. All workers satisfy the
shared eligibility rules.
Start failures
A named start whose arguments are all owned or copied, or that has none, binds them into a
one-shot job, as pool.submit(function, arguments...) does,
and starts that job. Its start expression has result
Result[(), SpawnError[F]], where F is the bound
OnceClosure[[], R]; a failed start returns the job with every argument, as an
owned job start does. spawn(worker, sender) with an owned Sender
therefore returns the sender on failure. Call drop_job() on the start to drop
the arguments there instead.
A start that borrows, a named start with a &T or &mut T argument or
spawn(&job), returns the job-free SpawnError[()].Unavailable((), code), so
every start fails with SpawnError. A borrowed argument cannot be returned in
an error, so moved arguments of a borrowed start drop exactly once on the parent
thread, in reverse argument order; on success they belong to the worker. The
worker has not run and cannot have changed borrowed inputs; already evaluated
argument expressions keep their effects.
spawn(...)? follows ordinary
? propagation: the
enclosing Result’s error may be the start error or an anonymous sum containing
it.
Every nonzero pthread_create status is a start failure; none terminates the
process. The error carries it: SpawnError[F].Unavailable(job, code), with
code: i32. The usual code is EAGAIN (11),
for a thread or memory limit. The runtime passes no thread attributes, so other
codes come only from the environment: a seccomp filter that denies clone
surfaces its own code, such as EPERM (1).
Task bookkeeping and error handling allocate no Plenty heap storage; native thread stacks and OS bookkeeping still require resources. The current runtime supports joinable native threads on x86_64 Linux GNU.
Joining synchronizes worker completion and its cleanup, but does not synchronize
Linux’s later removal of the task from /proc/self/task. Diagnostic tools can
briefly observe an exiting task there after a successful join; this does not
mean the worker is still running Plenty code or still owns its dropped handles.
Consuming thread jobs
with spawn(job)? as task: moves a concrete wholly owned zero-argument closure
into a worker. Reusable and one-shot closures are accepted; def once permits
moving a capture into the result. Borrowed captures and generator frames are
excluded. A named start with owned arguments binds
them into such a job. The scoped-thread lifetime and join rules
still apply.
The start expression has result Result[(), SpawnError[F]], where F is the
concrete job type. Its one variant, SpawnError[F].Unavailable(job, code),
returns the complete unstarted job and the i32 native code after any
native start failure. Returning, matching,
dropping, and retrying the error and job allocate nothing. Capture expressions
have already run, but the worker has not. ? widens the error into an enclosing
sum containing SpawnError[F], which keeps the job. A kept job’s endpoints stay
alive through the joins that the ? exit performs, so the
live-endpoint join rule counts them. A generic signature names the job as
F. A nongeneric one writes its signature, such as
SpawnError[OnceClosure[[], list[i64]]], alone or in a sum; ? resolves that
member to the job it propagates, as for a
returned closure.
Each such member names one producer, so start several jobs from one factory
function or a generic helper.
Job-free errors
with spawn(job).drop_job()? as task: gives up the job instead. drop_job()
converts Result[(), SpawnError[F]] to Result[(), SpawnError[()]]: on failure
it drops the job and its captures at the start, and the error keeps its variant
and code, SpawnError[()].Unavailable((), code). When applied directly to
spawn, the compiler constructs this job-free error in the failed-start block,
without materializing an intermediate job-carrying result in the caller frame.
On success it changes nothing. One
SpawnError[()] member covers every job, so a function can start distinct
closures and propagate one precise error:
def main() -> Result[(), IoError | SpawnError[()]]:
with spawn(def once () -> i64: 1).drop_job()? as first:
print(first.join())?
with spawn(def once () -> str: "two").drop_job()? as second:
print(second.join())?
Ok(())
Because the job is gone before ? runs its exit’s joins, its endpoints cannot
block them. The conversion copies a tag and the code and allocates nothing. drop_job() also
applies to a SpawnError[F] value, or a Result carrying SubmitError[F], as
for pool submissions. Erasing the error into Failure drops the
job too, but loses the error type; use it only when no precise error fits.
After success, only the worker owns the captures. One-shot invocation moves them into the body; reusable invocation borrows them once and then drops the environment. Join transfers the result exactly once; automatic join drops an unclaimed result. Job storage and the ownership handoff require no heap allocation.
Bounded channels
channel[T](capacity: Positive[i64]) -> Result[tuple[Sender[T], Receiver[T]], AllocError]
constructs a bounded multi-producer, multi-consumer FIFO queue. Capacity must be
a Positive[i64]: positive literals convert at compile time; computed counts
use Positive.try_from(capacity)?. Zero and negative literals are compile
errors. Size overflow or exhaustion returns AllocError. Failed construction exposes no partial queue.
| Operation | Result | Behavior |
|---|---|---|
sender.send(value) | Result[(), SendError[T]] | Wait for space or receiver disconnection |
sender.send_nowait(value) | Same | Return Full(value) if no space |
receiver.recv() | Result[T, RecvError] | Wait for a message or sender disconnection |
receiver.recv_nowait() | Same | Return Empty if no message is ready |
endpoint.share() | Same endpoint type | Create another owner of the queue |
Methods observe endpoints; mut is unnecessary. Endpoint assignment moves;
share() explicitly retains the queue. Copy and equality are unsupported.
Sending moves affine messages. SendError[T].Disconnected(value) and
Full(value) own the complete unsent value; blocking send never returns Full.
Receiving transfers one message to one receiver. Shared receivers compete rather
than broadcast. Accepted messages follow insertion order, but concurrent senders
and waiters have no predetermined or strictly fair order.
Construction uses one allocation. Sharing, send, receive, waits, disconnection,
and error handling allocate nothing. nowait avoids waiting for queue space/data
but can contend on synchronization; it does not guarantee immediate execution.
Timed operations and receive selection are described separately.
Disconnection and cleanup
Dropping the last sender wakes receivers, which drain buffered messages before
returning RecvError.Disconnected. Dropping the last receiver wakes senders,
returns their pending messages as errors, and drops queued messages.
Endpoints are owned context managers. with receiver as inbox: moves the handle
into the scope, lends &Receiver[T], and drops that owner on every normal exit.
Other shared handles remain independent owners.
Workers must handle disconnection and terminate.
Joins and live endpoints
A worker waiting for queue space is released only by a receive or by the drop of the last receiver. A join that waits for such a worker while the joining function keeps a receiver alive, and does not receive, never returns. The compiler therefore rejects a join that could wait this way:
A scoped task whose worker can hold
Sender[T]cannot be joined while a live binding of the joining function can holdReceiver[T].Likewise, a worker that can hold
Receiver[T]cannot be joined while a live binding of the joining function can holdSender[T]: an empty queue could leave the worker waiting forever for a message or the last sender’s drop.
The rule applies to task.join() and to the automatic join on every exit from
the task’s with block: its end, return, ?, break, continue, and the
failed start of a later task in the same statement. A task that task.join()
has consumed on every path is not joined again at exit. A join is likewise
rejected while the joining function holds a lock on a Mutex[T] that the
worker can reach; see mutexes.
Recommended patterns
These patterns satisfy the rule on every path, and keep satisfying it when the block gains early exits. They are the canonical form; the tutorial teaches them in avoid waits that hang.
-
Move each endpoint into the worker that uses it, as a named argument:
spawn(produce, sender).drop_job()?. A failed start returns its owned arguments insideSpawnError;drop_job()drops them before?leaves. -
Guard each endpoint the function keeps as the last manager of the task’s
with. Contexts exit in reverse order, so every exit drops it before the join:with spawn(produce, sender).drop_job()?, receiver as inbox: ... with spawn(consume, receiver).drop_job()?, sender as outbox: ...Use a new name for the guard binding:
with outbox as outbox:is a duplicate binding. The blocked-join diagnostic suggests an unused name, adding a numeric suffix wheninboxoroutboxis already bound. Use that guard binding for endpoint operations inside the block. -
To join explicitly, end the endpoint’s guard first. Open the guard before any statement that can leave the block:
with spawn(produce, sender).drop_job()? as task: with receiver as inbox: ... result = task.join()drop(receiver)before the join also satisfies the rule, but only on paths that reach it: an earlier?still leaves with the receiver alive. Moving the receiver into another worker or function likewise ends the binding. -
For several producers, give the receiver to a consuming worker started first, and start the producers inside a sender guard:
with spawn(consume, receiver).drop_job()? as consumer: with sender as outbox, spawn(produce, outbox.share()).drop_job()?, spawn(produce, outbox.share()).drop_job()?: pass total = consumer.join()One receiver guard after several
?starts is rejected: a failed later start joins the earlier tasks before the guard exists. Alternatively, start the later producers in a function that takes the receiver by value. -
Call
drop_job()on a start the function will not retry:spawn(job).drop_job()?orspawn(produce, sender).drop_job()?. -
Wait in a statement of its own,
result = task.join(), and move an endpoint into a value or return it only after the block. -
Start tasks and pools outside lock blocks; see mutexes.
-
Cancel a token that a worker waits for on every exit. Like an endpoint guard,
stop.cancel_on_exit()goes directly after the task or pool:with spawn(serve, stop.share()).drop_job()?, stop.cancel_on_exit(): ...See cancel on every exit.
Pools follow the same patterns, with pool.submit(...) in place of the start;
see shutdown and
result waits.
What can hold an endpoint
A worker can hold a sender that is an argument or capture, a field or element
of one, behind a reference, or a message of a channel it holds. A binding
can hold a receiver the same way, except that a local reference is ignored
because its owner is checked instead. A &Receiver[T] parameter, or a parameter
that refers to a value holding one, always counts: its owner outlives the call.
The same holder rules apply to receiving workers and surviving senders, including
borrowed sender parameters, aggregates, and concrete callable captures. Other
shared handles count separately, and a drop on only one branch does not release
a handle on the other paths.
Besides live bindings, these values hold the endpoints moved into them:
| Holder | Held through | Rejected example | Fix |
|---|---|---|---|
The error that ? propagates | The function’s return, after the exit’s joins and pool shutdowns | with spawn(consumer)? as task, spawn(producer)?: where the producer closure captures the sender | spawn(producer).drop_job()? |
The value a return or final expression hands back | The joins and pool shutdowns of every block it leaves | return Err(sender) inside the consumer’s block | Drop the endpoint in the block, or return it after the block |
| A statement’s expression and the value it builds | Until the statement stores, drops, or tests the value | pair = (sender, task.join()) inside the consumer’s block | Join in an earlier statement |
The final expression of a match case used as a value | That case’s cleanup | A case whose value is an endpoint, ending a block that the endpoint blocks | Produce the endpoint after the block |
A failed closure start’s SpawnError[F] retains the unstarted job and every
endpoint it captured, so the first row also applies to an enclosing task block.
Erasing the error into Failure drops the job too, as a last resort.
The returned-value and expression checks follow the bindings that the
expression moves, not its type, so return Ok(()) of Result[(), Sender[i64]]
is accepted. A tuple, class, enum payload, Some, or call argument that
receives a moved endpoint counts. The check is conservative: an endpoint moved
into a call inside the expression counts even if the call consumes it, so
return Ok(consume(sender)) is rejected; call consume(sender) first. A wait
nested in an expression, such as task.join() or future.result()?, is
rejected while an operand evaluated before it holds a matching endpoint.
The expression check covers bindings, assignments, unpacking, discarded
expressions, yield, if, elif, and while conditions, for iterables,
match subjects, and with managers. A wait inside a called function is seen
only for a future or
pool passed to the call; a scoped
task cannot be passed to a function.
The diagnostic reports the join or the with statement, each binding,
propagated error, returned value, or unfinished expression that holds a matching
endpoint, the early exit that reaches the join, and, for an explicit join, where
the task starts.
Pool shutdown checks sending jobs against surviving receivers. See shutdown and live receivers.
What the rule does not decide
Endpoints are matched by message type, not by queue, and the compiler does not count messages. These safe programs are rejected:
- the parent receives everything the worker sends, or drains to disconnection, through a receiver that is still alive at the join;
- the live receiver belongs to a different queue of the same message type;
- the worker sends only with
send_nowaitorsend_timeout, or its sends always fit the queue’s capacity; - another thread also receives from the queue;
- the receiver’s caller-side owner would have received.
The receiving-worker rule likewise rejects a sender for an unrelated queue of
the same message type, a worker that only receives with recv_nowait or
recv_timeout, and a worker whose receives are already satisfied by buffered
messages or another thread. It does not count sends or receives.
The rule is not a deadlock-freedom guarantee. It does not cover:
- an endpoint captured by a
Callableor boxed callable held by the joining function, whose captures have no type; - endpoints held by the function’s callers through bindings it was not passed;
- a worker blocked for any other reason.
Dropping or guarding the surviving endpoints removes the rejection. Workers must still handle disconnection and terminate.
Messages
Messages must be concrete wholly owned storable values satisfying the worker checks, even for a channel used on one thread. Recursive data and sender-based reply paths are supported, but recursive message-type paths back to a receiver are rejected: a queue retaining its last receiver could prevent destruction. This conservative check also rejects noncyclic instances of such a type. Zero-capacity rendezvous is unsupported.
Executors and shutdown
ThreadPoolExecutor(max_workers: Positive[i64], queue_capacity: Positive[i64]) returns
Result[ThreadPoolExecutor, PoolError]. Both counts are Positive[i64]; positive literals convert at compile time,
and computed counts use Positive.try_from(count)?. Construction
starts all fixed native workers before success and provides a bounded FIFO queue.
Partial startup failure stops and joins workers already started. PoolError
has Allocation(AllocError) and Thread(ThreadError).
| Operation | Result | Behavior |
|---|---|---|
pool.submit(job) | Result[Future[T], SubmitError[F]] | Move an owned zero-argument job; wait for space |
pool.submit(function, arguments...) | Same, with a bound job type | Bind a statically named function and owned arguments |
pool.submit_nowait(...) | Same | Return Full(job) if the queue is full |
pool.cancel_pending() | () | Cancel every queued job; running jobs continue and the pool stays open |
pool.shutdown() | () | Stop accepting, drain accepted work, join workers |
cancel_pending() does not wait for running jobs. It removes the queued jobs,
so their queue slots are free for later submissions; their futures report
FutureError.Cancelled, and each job’s captures are dropped with its future.
To stop without running queued work, call pool.cancel_pending() and then
pool.shutdown(). shutdown() takes no argument; shutdown(True) is rejected
with a diagnostic naming cancel_pending().
Methods observe the pool; mut is unnecessary. Executors are affine and do not
support copying or equality. Jobs may differ in concrete argument/result types on
one pool. Reusable closures run once then release their environment; one-shot
closures transfer captures into their body. Named submissions infer ordinary
argument types and return a callable bound job on failure.
Inputs and results must be wholly owned and satisfy the worker checks.
Jobs share state through Shared[T] handles:
pass each job its own handle.share().
Borrowed captures/arguments, generator frames, and uncertified indirect effects
are rejected. Executors and futures cannot enter workers.
User jobs must own their inputs because a future can outlive the borrow that
would feed it. Parallel pipelines
are the one internal exception: .par chunk jobs read a list’s rows in place,
which is sound because every .par terminal waits for all submitted jobs before
returning, including after a failed submission. User-facing scoped pool jobs, a
deferred B22 extension, would build on the same guarantee. Channel protocols can
still deadlock.
SubmitError[F] has Full(job), Shutdown(job), OutOfMemory(job), and
CapacityOverflow(job). Each owns the complete unstarted job for retry,
synchronous invocation, or drop. A named submission’s job is a one-shot closure
binding the arguments, so pool.submit(stopped, token)? can propagate as
SubmitError[OnceClosure[[], bool]]; a signature names jobs as for
owned starts. pool.submit(job).drop_job()? instead drops the job
of a rejected submission at the call and gives SubmitError[()] with the same
variant, such as Shutdown(()); one such member covers every job, as for
job-free start errors. Blocking submit never returns Full. Shutdown is
reported before job allocation; waiting for space precedes allocation and transfer.
submit_nowait can contend on synchronization.
Construction uses one allocation; each accepted job needs one fallible cell. Status, cancellation, result transfer, waiting, and shutdown allocate nothing.
Lifetime
with ThreadPoolExecutor(workers, capacity)? as pool: owns the executor and
lends &ThreadPoolExecutor. Every normal exit drains accepted work and joins
workers; ordinary drop does the same. Explicit shutdown is idempotent and always
waits. Detachment and wait=False are unsupported; running jobs are not forcibly
killed. Tokens permit cooperative stopping; a shutdown that
could wait for a job holding a token before the function’s later cancel is
rejected, and with ThreadPoolExecutor(...)? as pool, stop.cancel_on_exit():
cancels it on every exit. See
cancel on every exit.
Workers must finish their communication before the scope can close. See futures for result ownership and parallel operations for batched work.
Shutdown and live receivers
Shutdown waits for every accepted job, and a job waiting for queue space is released only by a receive or by the drop of the last receiver. The compiler therefore applies the task join rule to pools:
A pool cannot be shut down while a live binding of the function can hold
Receiver[T], once the function has submitted a job that can holdSender[T].
A pool shuts down at pool.shutdown(), including through a
&ThreadPoolExecutor, at drop(pool), when its binding is reassigned, and
wherever its owner leaves scope: the end of its with block or of the function,
return, ?, break, and continue. A call that receives a pool, by value or
by reference, counts as a shutdown when a function it reaches, directly or
through further calls, calls shutdown() or owns a pool. A call through a
Callable, closure, or boxed callable value, or to a function that makes one,
is assumed to shut down a pool passed to it. Locals are released last declared first,
so a receiver bound after the pool is dropped before the pool shuts down.
Jobs are not matched to pools and are never assumed to have finished. A job
submitted with submit or submit_nowait counts at every later shutdown in the
function, and at every shutdown in a loop that contains the submission. A
submit(...)? or submit(...).drop_job()? that fails has submitted nothing on
the path that leaves.
map, map_result, and reduce_tree finish their jobs before returning and do
not count. What a job and a binding can hold is defined as for tasks.
The recommended patterns for tasks apply.
When the function does not wait for the job’s result inside the block, guard
the receiver as the last manager of the pool’s with, so every exit drops it
before the pool shuts down:
with ThreadPoolExecutor(2, 8)? as pool, receiver as inbox:
pool.submit(produce, sender).drop_job()?
...
That guard is still alive at a future.result() inside the block, which
its own check rejects. To wait for
the result, end the receiver’s guard first:
with ThreadPoolExecutor(2, 8)? as pool:
future = pool.submit(produce, sender).drop_job()?
with receiver as inbox:
...
result = future.result()
A channel created inside the pool’s block, and a drop(receiver) on every path
to the shutdown, also satisfy the rule. The
holders of a task join apply too: a
returned Ok(receiver), an unfinished pair = (receiver, future.result()?), or
a match case whose value is receiver holds it through the shutdown.
The diagnostic reports the shutdown or the pool’s binding, each binding,
returned value, or unfinished expression that holds a matching receiver, where
the job is submitted, and the early exit that reaches the shutdown.
The rule has the task rule’s false positives and gaps. It also rejects these safe programs:
- the job has finished, for example after
future.result()returned; - the job was submitted to a different pool than the one shutting down;
- a
submitfailure handled without?returned the job to the caller.
It does not cover:
- a job submitted by another function, through a
&ThreadPoolExecutoror a pool it returns; - a pool passed to a call inside a class, tuple, or collection;
- a
map,map_result, orreduce_treecallback that waits to send.
Blocking future retrieval has its own
check at result(), before any later receiver cleanup.
Shutdown and held locks
A job waiting for a mutex this function holds would keep the shutdown waiting
too. Submission therefore also records the guarded types of the Mutex[T]
values a job can reach, including through Shared handles:
A pool cannot be shut down while the function holds a lock on a
Mutex[T], once it has submitted a job that can reach aMutex[T]of the sameT.
Shutdown points, job counting, and the false positives are those of the
receiver rule; mutexes are matched by type, as for
joins under a lock. End the lock’s
with block before the pool shuts down.
Futures
Executor submission returns an affine Future[T] with one result consumer.
Futures can move and live in collections, classes, and enums, but cannot be copied,
compared, or passed into worker code. There is no shared-result handle,
result cloning, async/await, or event loop.
| Operation | Result | Ownership and waiting |
|---|---|---|
future.result() | Result[T, FutureError] | Consume the future and wait for its result |
future.done() | bool | Observe completion or cancellation |
future.cancel() | bool | Prevent pending work from starting |
future.wait_timeout_ms(timeout_ms: u64) | bool | Observe completion/cancellation within a millisecond budget |
Methods other than result observe their receiver without needing mut.
Cancellation succeeds only before execution begins. Repeated cancellation of a
cancelled job returns True; running or completed jobs return False. Successful
cancellation immediately makes done() true and result() return
Err(FutureError.Cancelled) without waiting. It does not immediately reclaim
queue space: captures drop after both the queue and public future release them.
Dropping a future neither cancels nor waits. Its job still finishes; the worker drops an unclaimed result. A future surviving executor shutdown retains the completed result until retrieval or drop. Futures do not retain the executor and cannot prevent shutdown. Mutable results transfer exactly once.
An application Result remains nested: a worker returning Result[T, E] yields
Future[Result[T, E]]. The first ? on result() handles cancellation; the
next handles application failure. Retrieval, cancellation, status, and error
transport allocate nothing. Timed observation does not cancel or consume work;
see timeout rules.
Scoped Task.join() returns T directly because scoped tasks cannot be
cancelled. Future.result() needs its outer Result to report cancellation;
for an application Result, (future.result()?)? deliberately handles two
different failure layers.
Result waits and live receivers
result() cannot wait for a locally submitted job holding Sender[T] while
the calling function still holds Receiver[T]. A full queue could keep the job
and the result wait blocked forever. End the receiver’s own guard before
retrieving the result:
with ThreadPoolExecutor(1, 1)? as pool:
future = pool.submit(produce, sender).drop_job()?
with receiver as inbox:
...
result = future.result()
A guard alongside the pool, with ThreadPoolExecutor(1, 1)? as pool, receiver as inbox:, disconnects only at pool shutdown, which is too late for a result
wait inside the block. A drop(receiver) on every path to the wait also
satisfies the rule. See the recommended patterns.
The diagnostic points to the wait, receiver holders, and submission. Tracking
follows local future moves, ?, unwrap(), drop_job(), extraction from
submission results, and calls that return a future. Separate futures with the same result type keep their own job origins.
Receivers dropped on every continuing path are accepted. done(), cancel(),
and wait_timeout_ms() are unaffected, and result retrieval still consumes the
future and preserves both cancellation and application errors.
The same wait is rejected while the function holds a lock on a Mutex[T] and
the job can reach a Mutex[T] of the same T, for example through a
Shared[Mutex[T]] share: the job could be waiting for that lock. End the lock’s
with block before waiting. Mutexes are matched by type.
This is a conservative check, not a proof of deadlock freedom: endpoints are matched by message type, not channel identity. Completion, cancellation, queue capacity, and earlier receives do not discharge the check. Assignments to the same local are conservatively combined.
A call that receives the future, by value or by reference, is checked as a wait
at the call when a function it reaches, directly or through further calls,
calls result() on a Future[T] of the same T. The diagnostic names the
call, and finished(future) below is rejected:
def finished(future: Future[()]) -> bool:
future.result().is_ok()
def main() -> Result[(), AllocError | PoolError | SubmitError[OnceClosure[[], ()]]]:
sender, receiver = channel[i64](1)?
with ThreadPoolExecutor(1, 1)? as pool:
future = pool.submit(produce, sender)?
ok = finished(future) # rejected: `receiver` is still live
drop(receiver)
Ok(())
Matching by type means a callee that waits for a different Future[T] also
counts. A call through a Callable, closure, or boxed callable value, or to a
function that makes one, is assumed to wait.
A future returned from a call keeps its jobs. When the call’s result can hold a
Future[T], directly or in a Result, Option, or other aggregate, it is
assumed to hold every future of the same T that its arguments carry, and
every job with a T result that the callee, or a function it reaches,
submits. Both waits below are rejected like waits for a local submission:
def submitted[T, F](result: Result[Future[T], SubmitError[F]]) -> Result[Future[T], str]:
match result:
case Ok(future):
Ok(future)
case Err(_):
Err("submission failed")
def start(pool: &ThreadPoolExecutor, sender: Sender[i64]) -> Result[Future[()], str]:
submitted(pool.submit(produce, sender))
def main() -> Result[(), AllocError | PoolError | str]:
with ThreadPoolExecutor(1, 1)? as pool:
sender, receiver = channel[i64](1)?
future = submitted(pool.submit(produce, sender))?
ok = future.result().is_ok() # rejected: `receiver` is still live
other_sender, other = channel[i64](1)?
later = start(&pool, other_sender)?
done = later.result().is_ok() # rejected: `other` is still live
drop(receiver)
drop(other)
Ok(())
The result is assumed to hold those futures even when the callee drops or waits for one of them. A call through a value keeps only its arguments’ jobs: jobs submitted by a callable, closure, or boxed callable value, or by a function compiled elsewhere, are not seen. Provenance is not tracked through collections, class fields, or multi-field aggregate storage; indirect submissions and blocking mapping callbacks retain the executor limits.
Ordered parallel mapping
pool.map(worker, inputs) -> Result[list[U], PoolMapError] accepts a statically
named function or a concrete reusable immutable closure (T) -> U, where U
is non-unit and owned storable data.
It consumes an owned list[T] or copies an integer range[T] normally. Generic
worker types are inferred. Ordinary worker restrictions apply.
A closure moves into the operation and must wholly own its captures. One
environment is shared by all jobs and remains pinned until every accepted job
finishes, including on empty input, allocation failure, worker error, and
shutdown. It drops once on the owning thread before the map call returns.
Captures are neither copied per input nor given separate heap environments.
Mutable and one-shot closures cannot be shared across jobs; borrowed closures
belong in scoped spawn. Concrete closures passed through an owned generic
F: Callable[[T], U] parameter remain eligible. Indirect code-address Callable
values lack worker-effect evidence and are rejected, even if initialized from a
named function; pass that function directly instead.
The result preserves input order regardless of completion order. Mapping is
eager: output capacity is reserved before execution, and at most
max_workers + queue_capacity jobs remain outstanding. Each job cell can fail
allocation. Memory is the output list plus that bounded window, rather than one
future per input.
PoolMapError has Allocation(AllocError) and Shutdown. Infrastructure
failure stops submissions, waits for accepted work, and drops partial outputs
and remaining inputs. The input list is consumed on every outcome; previous
effects remain visible. Allocation failure leaves the pool usable. A closed pool
returns Shutdown, including for empty input. Worker Result values remain ordinary
list elements.
Borrowed inputs, generators, multiple iterables, and streaming output are unsupported.
Fallible workers
pool.map_result(function, inputs) instead accepts (T) -> Result[U, E]
and returns Result[list[U], ParallelError[E]]. U must be non-unit; inputs,
outputs, and errors meet owned job eligibility. ParallelError[E] has
Allocation(AllocError), Shutdown, and Worker(E), with allocation-free error
transport.
Success follows input order. If several workers fail, the lowest input index’s error wins. A failed submission stops further submission; an earlier accepted input’s worker error takes precedence over that submission failure. Allocation failure before any submission returns Allocation; a closed pool returns Shutdown, including for empty input.
On observing a worker error, submission stops and all accepted jobs are drained. Partial successes, later errors, and unsubmitted inputs drop exactly once. Some later inputs may already have run, and effects are not undone. Workers must not depend on every input being scheduled. Output capacity and the bounded job window are reserved before execution; submitted cells remain fallible.
A parallel pipeline
builds on map and map_result: iter(items).par(&pool) maps and filters
contiguous chunks on the pool. Unlike pool.map, it also accepts a borrowed
list, iter(&items), whose chunks read the caller’s rows in place.
Fixed-tree reduction
pool.reduce_tree(function, inputs) -> Result[Option[T], PoolMapError] accepts
a statically named (T, T) -> T worker and an owned list or integer range.
A named (T, T) -> Result[T, E] worker instead gives
Result[Option[T], PoolMapError | E], preserving the ordinary public callback
error (including managed, affine payloads).
Generic types are inferred and worker eligibility applies. Lists
are consumed on every outcome; ranges retain copyable semantics. Empty input
returns Nothing; otherwise the result is Some. No identity is inserted or copied.
Each round combines adjacent pairs in left/right order; an odd final element
advances unchanged. Five inputs use f(f(f(a, b), f(c, d)), e). Worker count,
queue capacity, and completion order do not change the tree. A successful
nonempty input of length n invokes the worker exactly n - 1 times.
The rounds define the grouping, not the schedule. For infallible callbacks, one job reduces an aligned run of adjacent inputs, a power-of-two count chosen so there is about one run per worker, through as many rounds as the run spans; jobs then combine the run results pairwise. A job’s calls are serial, and runs proceed independently, so calls from different rounds can overlap in time.
This reassociates operations rather than performing a left fold. Floating-point
results can differ from serial accumulation; pure workers produce the same result
for the same input and target across pool sizes. Effect order is unspecified.
Workers must not depend on another pair being scheduled. Workers retain their
own arithmetic contracts.
General integer addition uses a callback returning
Result[T, OverflowError[T]] with explicit Ok(left + right). A caller’s
checked: or unchecked: mode never changes the callback or API type.
Fallible callbacks use separate adjacent-pair rounds, not multi-level run jobs.
All accepted jobs drain before returning. Callback errors are observed in
level order, then left-to-right pair order, never completion order. No later
level runs after an error. An accepted callback error takes precedence over a
later submission failure; without such an error the precise allocation or
shutdown failure is returned. Unused errors and intermediate values drop once.
This adds per-level synchronization and potentially more jobs than the
infallible optimized path, without changing successful grouping or replaying
callbacks. Empty and singleton input never invoke the callback.
A parallel pipeline’s
reduce groups by chunk instead, so its result can depend on the chunk count;
use reduce_tree when one grouping is needed for every pool. Unlike
reduce_tree, a pipeline can read a borrowed list: iter(&xs).par(&pool)
borrows xs and maps its items to owned values before reduce.
Empty and singleton reductions allocate nothing. Larger inputs need one fallible scratch buffer proportional to input length, including for ranges, a bounded job window, fallible job cells, and, when a job reduces more than one input, one small buffer of per-job run descriptors and pair scratch. Pair arguments transfer without tuple allocation or implicit copying.
Infrastructure failure stops submission, drains accepted jobs, and destroys
intermediate and unsubmitted values. A closed pool returns Shutdown even for empty
input. When the callback returns exactly T and T is itself a Result, those Results
remain ordinary elements: only a distinct Result[T, E] callback is fallible.
Mutexes
std.sync.Mutex[T] holds a value that several workers change through shared
references or shared owners, one lock holder at a
time. Use it when several values must
change together to keep an invariant; a single independent counter or flag is
an atomic. The operations are listed in
std.sync.
Locking
with place.lock() as value: borrows the mutex at place shared, waits until
no other thread holds the lock, and binds value: &mut T for the block. as
may be omitted. The place is a name, a field, an element, or a dereference,
including a &Mutex[T] parameter, or a Shared[Mutex[T]] handle, which locks
the mutex through *handle; a temporary is rejected. lock() anywhere
other than a with manager is a compile-time error naming that form.
The lock is released on every exit from the block: its end, return, ?,
break, and continue. Comma-separated managers nest, so
with a.lock() as x, b.lock() as y: takes a then b and releases b first.
value is an ordinary exclusive reference. Its loan belongs to the lock, not to
the mutex’s owner, which stays shared-borrowed until the block ends: the mutex
cannot be moved, replaced, or consumed while locked. Like any local reference,
value cannot be stored, returned, captured by an anonymous function, or used
after the block.
into_inner() consumes an owned mutex and returns its value without locking.
Ownership proves that no thread holds or waits for the lock.
Waits the compiler rejects
A lock waits as long as another holder keeps it. The compiler rejects these patterns that would wait forever:
- Relocking in the block. Locking a place that an enclosing lock block in
the same function already holds is an error. Places are compared as written:
mand*mmatch. - Nesting locks that could alias. Locking a
Mutex[T]inside a lock block that holds anotherMutex[T]of the sameTis an error when either place is reached through a reference, aSharedhandle, or a computed index (cells[i]). Such places can name one mutex (r = &a,b = a.share(),i == j), and only types are compared. Two owned places built from local names, fields, and constant indices, such asaandborcells[0]andcells[1], are distinct mutexes and may nest. Otherwise wrap one value in a distinct class. A relock inside a called function is not detected and deadlocks at run time. - Joining under a lock. A scoped task cannot be joined, by
task.join()or by leaving itswithblock, while the joining function holds a lock on aMutex[T]and the worker can reach aMutex[T]of the sameT: as an argument or capture, a field or element of one, or behind a reference. The worker could be waiting for that lock. Mutexes are matched by type, like the channel rule. Start the task outside the lock:with spawn(...)?:aroundwith m.lock():unlocks before joining. A join inside a function called while the lock is held is not detected and deadlocks at run time. - Waiting for a pool job under a lock. A job submitted to a
pool records the
Mutex[T]types it can reach, including throughSharedhandles.future.result()on its future, and a shutdown of any pool after it was submitted (shutdown(),drop, reassignment, or the end of the pool’s scope), is an error while the function holds a lock on aMutex[T]of a recordedT. A future is traced like the channel rule traces it, through local bindings and calls that return it. - Yielding under a lock.
yieldinside a lock block is an error: a suspended generator would keep the lock indefinitely.
Locking two mutexes in different orders on different threads can still deadlock; take nested locks in one fixed order.
No poisoning
A trap terminates the process without unwinding, so no program continues after
a holder fails mid-update. A Plenty mutex therefore has no poisoned state and
lock() has no error result.
Not provided
There is no try_lock, timed lock, reentrant lock, condition variable,
read-write lock, or fairness guarantee, and no unlocked access through
&mut Mutex[T] (Rust’s get_mut); lock it instead. Executor and pool jobs own
their inputs and cannot borrow a mutex; give each job a share of a
Shared[Mutex[T]] instead.
Lowering
The state word is 0 when unlocked, 1 when locked, and 2 when locked with possible waiters. Lock is an inline compare-exchange from 0 to 1; on failure the runtime exchanges in 2 and sleeps on the word with a Linux futex until an exchange finds 0. Unlock is an inline exchange to 0 that calls the runtime to wake one waiter only if it replaced 2. See the runtime mechanisms.
Cancellation tokens
CancellationToken() -> Result[CancellationToken, AllocError] creates an affine
owner of a shared cancellation state using one fallible allocation. Tokens may
cross threads and live in owned containers; assignment moves them. Copy and
equality are unsupported.
| Operation | Result | Behavior |
|---|---|---|
token.share() | CancellationToken | Another owner of the same state |
token.cancel() | () | Permanently request cancellation and wake all token waiters |
token.is_cancelled() | bool | Observe the request |
token.wait() | () | Wait until requested |
token.wait_timeout_ms(timeout_ms: u64) | bool | True if requested within the millisecond budget |
with token.cancel_on_exit(): | guard | Cancel on every exit from the block; see below |
All methods observe their receiver without requiring mut and allocate nothing.
Cancellation is idempotent. Dropping a handle does not request cancellation;
the last owner releases the state. Timeout does not consume or reset it. wait_timeout is also the cancellable
sleep; std.time supplies its
budget from a Duration.
Workers must check the token at safe boundaries, finish required cleanup, and return normally. A token imposes no return type or error variant. It does not change future state by itself: Future.cancel only prevents pending work from starting.
Ordinary blocking channel operations and foreign calls do not observe tokens. Use timed waits or an explicit shutdown message when work must periodically regain control. Scope exit and executor shutdown still join running workers, so cancellation-aware code must actually return, and the code that cancels must reach its cancel on every exit; see cancel on every exit.
Cancel on every exit
A task or pool job that waits for a token keeps its join, or its pool’s
shutdown, waiting until someone cancels the token. token.cancel_on_exit() is a
with guard that cancels the token on every exit from its block: its end,
return, ?, break, continue, and the failed start of a later manager in
the same statement. It holds a share of the token, allocates nothing, has no
value to bind with as, and is valid only as a with manager.
Contexts exit in reverse order, so put the guard directly after each task or pool that holds the token:
with spawn(serve, listener, stop.share()).drop_job()?, stop.cancel_on_exit():
...
with ThreadPoolExecutor(2, 8)? as pool, stop.cancel_on_exit():
...
To join inside the block, or to wait for a job’s result(), cancel first:
either call stop.cancel() before the wait, or end a nested guard before it:
with spawn(serve, stop.share()).drop_job()? as task:
with stop.cancel_on_exit():
...
task.join()
A worker that cancels a token when it finishes, so that its caller can stop
waiting, puts its body in with stop.cancel_on_exit():.
Waits the compiler rejects
The compiler rejects a wait that could skip the function’s own later cancel:
When a function cancels a token binding, a join of a task, a
result()of a job, or a pool shutdown that waits for work holding that token must come after a cancel on every path that reaches it.
A task or job holds the token when its arguments move, share or borrow that
binding, including a share() of it bound to another name. The rule covers
task.join(), future.result(), drop(pool), and the automatic join or
shutdown on every exit listed above. A guard placed before the task in the
header is rejected too, since it cancels after the join. The diagnostic names
the exit, the job’s start where it helps, and the later cancel, and suggests
the guard.
A worker is held to its own final cancel: when a function’s normal end always
cancels a token parameter, an earlier return or ? that skips the cancel is
rejected. A conditional cancel is not checked.
What the check does not decide
The check is local to one function and linear in its size. It does not see:
- tokens reached through a field, a
Shared[T], a class value built before the start, or a closure capture; - the tokens that
map,map_resultorreduce_treejobs hold; - a cancel made by a called function;
- a worker’s deadline or timed wait, so a worker that stops by itself is still checked;
- which pool runs which job: a function-exit shutdown counts every owned pool;
- every path through a loop: a loop body is walked at most twice.
A program outside these forms can still wait forever; the guard is the habit that avoids it.
Timed waits and selection
Prefer the typed deadline waits in std.sync
for cancellable application code. They take &Deadline and &CancellationToken.
The low-level primitives below use u64 milliseconds using a monotonic total budget, including repeated
wakeups. Zero checks immediately. Readiness or disconnection at the final check
wins over timeout. Lock contention, cleanup, and scheduling may delay return
beyond the budget; timeouts are not hard execution deadlines.
std.time converts a
Duration or the remainder of a shared Deadline to this budget.
| Operation | Result |
|---|---|
sender.send_timeout_ms(value, timeout_ms) | Result[(), SendTimeoutError[T]] |
receiver.recv_timeout_ms(timeout_ms) | Result[T, RecvTimeoutError] |
future.wait_timeout_ms(timeout_ms) | bool |
token.wait_timeout_ms(timeout_ms) | bool |
The original _timeout spellings remain compatibility aliases.
SendTimeoutError[T] has Disconnected(value) and TimedOut(value), both
returning ownership of the unsent message without copying or allocating.
RecvTimeoutError has Disconnected and TimedOut; buffered messages remain
receivable after the last sender drops. All channel waits borrow endpoints.
A future wait observes completion or cancellation without consuming its result or cancelling work. A token wait observes the permanent cancellation request without consuming or resetting it. Neither timed observation allocates.
Receive from either of two channels
| Operation | Result |
|---|---|
select_recv(&first, &second) | Result[Selected[A, B], SelectError] |
select_recv_nowait(&first, &second) | Same |
select_recv_timeout_ms(&first, &second, timeout_ms) | Same |
Selection borrows both receivers and transfers exactly one message.
Selected[A, B] has First(A) and Second(B) variants. Shared & markers are
optional, and two handles sharing one queue are accepted. The unselected queue
keeps its messages. Ordinary message eligibility applies.
SelectError.Empty means neither had a message during a nonblocking check;
TimedOut means the budget expired. Disconnected means both queues are drained
and have no senders. One disconnected queue does not stop waiting for the other.
Blocking selection returns only Disconnected.
The first receiver has priority whenever both are ready; sustained traffic can starve the second. Selection receives a message before returning, rather than reporting readiness that another receiver could invalidate. It allocates nothing.
Only two-channel receive selection is supported: no send cases, arbitrary channel sets, or cancellation-token case. Selected and ordinary receives compete for messages. Application protocols can still deadlock.
Native effects on worker threads
Worker certification asks every native operation a body can reach for its
effect. The compiler classifies each primitive in one exhaustive table
(src/effects.rs), so a new primitive cannot be added without stating its
effect. Standard modules have no exemption: a std helper is certified by
the same walk as user code, and a foreign call is still rejected.
| Effect | Operations | On a worker |
|---|---|---|
| Compute | Text, collections, bytes, hashing, compression, Unicode | Allowed |
| Clock | Monotonic and wall clocks, sleep | Allowed |
| Entropy | std.secrets reads | Allowed |
| Process stream | Standard input, output and error, print, args | Allowed; no ordering between threads |
| Path | open, read_text, metadata, read_dir, std.fs.root, std.fs.symlink, std.fs.canonicalize | Allowed; a relative path resolves against the working directory at the call |
| Handle | Methods of File, ReadDir and std.fs.Root, rooted mutation, std.ready waits and selectors, operations on an open socket | Allowed under the loan rules below |
| Environment | std.env.get, vars, current_dir | Allowed; reads only |
| Child | std.process spawn, wait, poll and signal | Allowed; the runtime’s child table is synchronized |
| Network | Opening a std.net socket, name lookup | Allowed; the kernel and C library synchronize sockets and the resolver |
| Process global | std.env.set_current_dir | Rejected |
| Signal disposition | std.signal.listen | Rejected |
Changing the working directory would change how every other thread resolves a
relative path, so a worker body, helper, callback or destructor that reaches
set_current_dir is rejected:
error: worker `worker` reaches `std.env.set_current_dir`: changing the working directory affects every thread's relative paths; change it on the main thread before starting workers
Catching termination signals changes how every thread’s process ends, so
std.signal.listen is rejected on a worker the same way. A worker may wait on
the Signals handle the main thread opened. Nothing in the language writes
the environment.
When the rejected function is reached through others, the diagnostic lists
them from the worker outward. A rejected type in a called function’s
signature or locals names that function and its path the same way. A generic function is named by the location of
its definition, and a method as Class.method:
error: worker `work` reaches `std.env.set_current_dir` through `outer` -> `Parser.parse` -> the generic function at std/error.plenty:7:5 -> `wrap`: changing the working directory affects every thread's relative paths; change it on the main thread before starting workers
Thin callable targets
A Callable stores a code address with a concrete signature. The compiler
collects every address of that signature in the closed program and certifies
all of their bodies, signature types, stored values and destructors. This
allows named and capture-free function values, including payload variant
constructors, in worker helpers such as result.map_error(Domain.Allocation).
Moving a callback through parameters, returned values, classes, lists or
channels preserves this check. Recursive signatures and bodies are inspected
once per identity.
This is conservative certification by signature. Taking an unsafe address on the main thread can reject a worker’s use of a different function with the same signature, even when that unsafe value never enters the worker. Standard library callbacks share this candidate set: an unsafe user callback with the same signature as a std error mapper can prevent a std worker from certifying. A conditional with one safe and one unsafe candidate is rejected too. An absent address set supplies no evidence and is rejected; a function signature alone is never an effect contract. A concrete captured closure has its own body and capture evidence and can avoid unrelated signatures, while retaining all checks on its actual effects.
Foreign interfaces cannot produce or reinterpret a Callable: their scalar
and opaque-pointer ABI is separate from Plenty function values. Erased
BoxedCallable environments remain ineligible. The direct worker-entry APIs
keep their existing requirements for statically named functions or concrete
closures; certification of thin values used inside workers does not add a
new spawn or parallel-worker entry form.
Native handles
File, directory streams (ReadDir), entries and metadata cross thread
boundaries like any other owned value:
- An owned handle can move into a worker, a channel or an executor job. The receiving thread closes it, explicitly or when its owner drops.
- A scoped worker can borrow a handle. Reading, writing, seeking and closing
need an exclusive loan, so the parent cannot use a handle a worker holds as
&mut File, and two workers cannot hold it mutably at once. - A shared loan (
&File) permits only operations that do not move the cursor or change the handle: theclosedproperty,readable,writable, rootedopenatoperations, andstd.readywaits. Several workers may wait on one shared handle. - A failed thread start drops each moved handle once, which closes it.
Foreign pointers (opaque types) remain ineligible: the compiler cannot know
whether a foreign resource tolerates use from another thread.
Blocking and cancellation
Blocking calls stay ordinary function calls; there is no async syntax. A
worker that must not block indefinitely waits with a deadline through
std.ready, which makes one bounded ppoll per step
on the handle and the cancellation token’s wake
descriptor, so cancelling the token wakes the wait at once. A deadline that passes leaves the handle and any borrowed buffer
intact; the caller decides whether to retry, close or return. A token still
does not interrupt an unbounded read, write, recv or child wait.
Resource use
Each worker is one native thread, started explicitly by spawn, an executor or
a parallel operation, so a program’s thread count is what it asks for. No
facility starts a thread per handle or per connection; a
std.ready selector lets one worker wait on
many. An idle std.ready waiter does not wake until its handle is ready, its
deadline passes or its token is cancelled. std.net.resolve is the exception:
each lookup runs on a short-lived helper thread, at most 32 at once. Descriptors are
released when their owner closes or drops them, on whichever thread that is.
Supported target: x86_64 Linux GNU. Elsewhere std.ready waits report
IoError.Unsupported.
C interoperability and libraries
Trusted .plentyi modules describe foreign ABI and lifetime promises. The
compiler checks declared types, ownership, and loans; binding authors and native
callers must supply matching implementations, valid storage, and obey the stated
aliasing rules. Foreign code can violate memory safety. No foreign unwinding or
nonlocal jump may cross Plenty frames.
- Trusted declarations and ABI types.
- Borrowed arguments and text adapters.
- Owning foreign resources.
- Exports and generated C contracts, including owned library objects.
- Embedded contracts and compatibility.
- Runtime loading and trusted loading primitives.
The native ABI targets are x86_64-unknown-linux-gnu and
aarch64-unknown-linux-gnu; the compiler, runtime and library object must match.
See ARM qualification for deployment
limits. Calls and owned exported
objects remain on their creating caller thread, with serialized access as required
by the generated C contract. Ordinary pub controls Plenty imports and does not
export a native symbol. Building and linking
is explicit; source imports alone neither link nor load a library.
Trusted declarations and ABI types
Imports resolve name.plenty or name.plentyi; having both is ambiguous.
.plentyi permits foreign declarations alongside ordinary Plenty types and
wrappers:
pub opaque Image
extern def load_raw(code: i32) -> Image = "image_load"
extern def free_raw(image: Image) -> () = "image_free"
pub extern def version() -> u32 = "image_version"
extern def requires fully typed parameters, a return type, and an explicit
C symbol or function-address parameter. Bodies, generics, varargs, defaults,
and implicit header translation are unsupported. pub only controls source
imports. Checking does not invoke native functions or inspect C headers.
The raw ABI supports i8–i64, u8–u64, f32, f64, nominal opaque
pointers, and unit for void results, plus the
explicit borrowing/text adapters. On x86-64 and AArch64 GNU/Linux,
these match C intN_t, uintN_t, float, double, and pointers. Binding aliases
can name C types: int is i32, long and ptrdiff_t are i64, and size_t
is u64. Bool, aggregate values, and private Plenty layouts have no raw ABI.
opaque Name declares a nominal pointer without a pointee layout. Distinct names
do not interchange. Values can be copied, compared for address equality, passed
back to C, or tested with .is_null(); Name.null() constructs null. They cannot
be dereferenced, converted to integers, or used in pointer arithmetic. They do
not own or free their pointees. Public signatures cannot expose private opaque
types. Prefer private pointers behind owning wrappers.
C symbols must be identifiers outside the main, plenty_, and __plenty_
namespaces, except the checked loading primitives.
Conflicting declarations are rejected. The binding author must supply the exact
ABI, lifetimes, permissions, and error/ownership rules. Callbacks and C records
are unsupported.
Calling a native address
opaque Function
extern def invoke(function: Function, value: i32) -> i32 = function
The identifier after = must name an opaque pointer parameter. It selects the
callee and is omitted from C arguments; remaining arguments use ordinary adapters.
The binding must provide a non-null address of the exact declared signature and
keep the originating code resident. Arbitrary addresses are not validated.
Keep this primitive behind a checked private wrapper; it is not a source-language
callable value.
Borrowed arguments and text adapters
Foreign borrowing is call-scoped. An extern def parameter &T or &mut T,
where T is a supported scalar or opaque pointer, becomes C const T * or T *
to initialized storage. Borrowing an opaque pointer passes a pointer to the
pointer, suitable for an output parameter.
Native code must not retain these addresses or write through a shared borrow. Mutable arguments obey exclusive borrowing; native code may access exactly the declared scalar’s bytes, with no assumptions about Plenty slot padding.
Text representations
extern def inspect(text: &str as utf8) -> u64 = "inspect_bytes"
extern def length(text: &str as c_string) -> Result[u64, CStrError] = "strlen"
| Adapter | Actual C arguments | Construction |
|---|---|---|
&str as utf8 | const uint8_t *data, size_t byte_len | No allocation; existing bytes including NUL |
&str as c_string | const char *data | One temporary terminated copy; rejects embedded NUL |
Both buffers are read-only and call-scoped: C cannot mutate, retain, or return an address into them. Empty UTF-8 views have length zero and must not be dereferenced. Ordinary strings retain their counted UTF-8 representation.
Any declaration with c_string must return Result[T, CStrError], with T a
supported C scalar/pointer or unit. The actual C return type is T; the adapter
wraps it in Ok. Conversion runs left to right before native entry. Embedded NUL
or allocation failure skips the call, releases earlier buffers, and returns Err;
normal return also releases all temporary buffers.
CStrError has allocation-free EmbeddedNul and Allocation(AllocError)
variants. It reports conversion failure rather than native status: wrappers must
interpret native errors separately.
Returned native strings are not implicitly decoded. Returned pointers need an independent lifetime. Retained/returned buffer loans and copying arbitrary C bytes directly into str are unsupported.
Strided numeric lists and byte views
&list[T] as strided and &mut list[T] as strided, for primitive numeric
T, pass three C arguments: a data pointer, size_t len, and
size_t stride_bytes. Element i lives at data + i * stride_bytes.
The pointer may be null for an empty list and must then not be dereferenced.
The stride describes Plenty storage; it is not necessarily sizeof(T).
The adapter does not allocate or pack a temporary collection. Native code
may read only each declared scalar’s bytes and may write those bytes only
through a mutable loan. It must preserve padding and runtime metadata,
cannot resize the list, and cannot retain or return the pointer. Use
memcpy or unaligned scalar accesses if the native type requires alignment.
No booleans, strings, objects, nested lists or nullable elements are admitted.
Exclusive borrowing rules still apply. Current numeric lists store naturally
sized elements; generated and runtime reads sign-extend narrow signed values
directly, so mutable views need no post-call normalization pass.
The optional dataframe engine uses this adapter
for bulk transfer.
&[u8] as strided uses the same three C arguments with a byte stride of 1.
It borrows a literal, a Bytes buffer, a list[u8], or a subview without
allocation or copying. The native call may read exactly len bytes and must
not retain the pointer. Empty views must not be dereferenced. Mutable byte
views and views of other element types are not admitted by this adapter;
mutable numeric lists retain the contract above.
Owning foreign resources
A public class can own a private opaque pointer and invoke its matching foreign
destroy function in __del__. It moves, cannot be copied, and cleans up on
ordinary scope exits and error propagation. Borrowing the class retains native
ownership. The binding chooses shared or exclusive methods from the native
library’s actual mutation and aliasing guarantees, not merely its use of const.
For fallible acquisition, construct an owner containing null first, then acquire into its pointer inside a Result-returning factory. Its destructor is active before any native resource exists, so partial acquisition or subsequent failure still cleans up. Class storage is inline and requires no allocation; allocation explicitly requested by fields or native acquisition retains its own contract.
The destructor guards null unless the native destroy operation accepts it, and uses the matching native release function rather than Plenty’s allocator. Native allocation failure is recoverable only if the library exposes such an error.
Transfer policies
- If C consumes unconditionally, move the owner into the wrapper and clear its pointer before calling C, on both success and failure paths.
- If C consumes only on success, clear the pointer on success and return the
intact owner on failure, for example as
Err(owner)inResult[(), Handle]. - Borrowed calls retain ownership and expose
&Handleor&mut Handleaccording to the permitted access. C must not retain borrowed addresses.
Raw declarations cannot prove these policies. The wrapper and native implementation must agree, including partial effects, null handling, and thread restrictions. For Plenty objects exported to C, use the generated handle contract.
Exports and generated C contracts
export def gives a top-level function an explicit C entry point in a library
build. It remains an ordinary Plenty callable; pub independently controls
imports. In application builds, export functions retain private native symbols.
export def add(a: i32, b: i32) -> i32 = "calc_add":
"Add two checked integers."
a + b
Parameters can be numeric scalars, owned exported classes, or shared/exclusive
references to either. Returns can be numeric, unit, or Result[T, E], where T is
numeric/unit and E is numeric, AllocError, ParseError, RangeError, Failure,
or a nongeneric user enum whose variants have no payloads.
Result[Class, AllocError] factories use owned handles.
Bool, other managed objects, raw pointers, returned references, generic exports,
and other aggregate results are unsupported; bodies can use ordinary features.
Enums with payload variants and error sums such as AllocError | ParseError
are rejected; sums are planned with the broader C representations (B27).
C symbols begin with the library name plus _. NAME_plenty_ and _plenty_
source names are reserved for generated metadata and compatibility adapters.
Duplicate symbols, import/export collisions, and generated-name collisions fail
checking or artifact validation. The same namespace applies to imported source
modules’ exports. Library builds require at least one export but no main.
Scalar borrows
&i32 becomes const int32_t *; &mut i32 becomes int32_t *.
Pointers must be non-null, aligned, initialized, and live for the call. Shared
arguments exclude mutation through every alias; exclusive arguments cannot overlap
another loan or any concurrent/reentrant access. The callee retains no storage.
Adapters read/write exactly the C type’s size, allocate nothing, and copy mutable values back on every normal return. A trap does not return and promises neither copy-back nor rollback. Generated Plenty wrappers preserve the reference types so ordinary loan checking applies.
Result outputs
Result exports return uint32_t status: 0 for Ok, 1 for Err. After input arguments
come T *out_ok (omitted for unit) and E *out_error. Both must be non-null,
aligned, writable, mutually disjoint, and disjoint from input loans; previous
contents need not be initialized. Only the selected output is written.
Err does not undo input mutations. Adaptation and Result reconstruction allocate
nothing and do not expose private sum layouts.
Enum errors use uint32_t *out_error. A variant’s code is its declaration
index. The builtin errors have these codes:
| Error | Codes |
|---|---|
| AllocError | OutOfMemory = 0; CapacityOverflow = 1 |
| ParseError | Invalid = 0; OutOfRange = 1 |
| RangeError | OutOfRange = 0 |
| Failure | Unspecified = 0 |
Prefer a precise enum; Failure is a last resort when no precise error exists.
For a user enum, the generated interface re-declares the enum, so it stays
self-contained, and its wrappers return that interface type:
enum CalcError:
Empty
Negative
export def check(value: i64) -> Result[f64, CalcError] = "calc_check":
...
The header names each code once per enum as LIBRARY_ENUM_VARIANT, before the
functions that use it:
/* CalcError codes, written to *out_error when a call returns 1. */
#define calc_CalcError_Empty UINT32_C(0)
#define calc_CalcError_Negative UINT32_C(1)
/* On Err, *out_error is one of the calc_CalcError_* codes. */
uint32_t calc_check(int64_t p0, double *out_ok, uint32_t *out_error);
Interfaces use the enum’s unqualified name, so two exported enums with the same
name, an enum named like an export or exported class, or a constant that
collides with another generated C identifier fails the build. Runtime loaders
also reserve Library and load.
Status and error code are distinct: code zero still means failure when status is one. Generated headers list these codes and caller requirements. Author docstrings appear as separate escaped documentation.
Headers require serialized calls on one caller thread and prohibit unwinding or nonlocal jumps across Plenty frames. Traps and explicit unwrap failures can terminate the process; adapters invent no recovery mechanism. Keep the generated header, interface, and binary together; see contracts and library artifacts.
Owned library objects
An export returning Result[Class, AllocError] publishes an opaque owned handle.
Its adapter reserves a box before invoking the Plenty body. Allocation failure
skips the body, releases consumed handle arguments, and reports AllocError. A
body Err frees the empty box. Class fields and layout remain private.
For library calc and class Resource, the C declarations include:
typedef struct calc_Resource calc_Resource;
uint32_t calc_create(int64_t p0, calc_Resource **out_ok, uint32_t *out_error);
void calc_Resource_destroy(calc_Resource *value);
Status zero returns one non-null owned handle; status one writes the error code without changing out_ok. Destroy each successful handle exactly once with its originating destroy function; never use free or another library instance’s function. Null destruction is a no-op. Destruction runs ordinary class and field cleanup and invalidates all aliases. No borrow or concurrent/reentrant access may remain; calls stay on the creating thread and code stays resident.
Generated .plentyi wrappers own private pointers and destroy them automatically.
Wrapper construction is inline and allocates nothing. Factory failure drops an
empty wrapper; ordinary moves, drop, scope exit, and ? apply. Owners cannot be
copied or expose the native object’s fields. Only allocation-error class factories
are supported; methods require explicit free-function exports.
Class names need distinct ASCII basenames across source modules. Handle types,
destroy functions, and generated namespaces cannot collide with user exports or
foreign imports. Names beginning plenty_ are reserved. For example, Thing and
Thing_destroy conflict as exported classes.
Borrowing
&Resource becomes a read-only call-scoped const calc_Resource *.
Shared arguments may alias, but no alias may mutate or destroy the live handle.
&mut Resource becomes an exclusive calc_Resource *: no other input or alias
may access it or its fields. Handles must belong to the same library instance.
Borrow adaptation allocates nothing.
Ownership remains with the caller. Whole-class replacement through a reference is rejected, but field changes survive Err; errors do not roll back mutations. Inputs and output storage must not overlap. Plenty wrappers preserve these loans.
Consumption
A by-value class argument consumes its C handle on entry, including when the call returns Err. It cannot alias another input or have outstanding borrows. The caller cannot later access or destroy it. A supported class result explicitly returns a new usable handle, even when the body returns the received instance.
Generated wrappers clear their private pointers immediately before native entry, preventing double destruction. The source call consumes arguments on every path; the factory’s box-allocation failure also cleans up consumed inputs.
Embedded contracts and compatibility
Library artifacts contain the exact generated UTF-8 .plentyi contract,
including its format and target. The versioned discovery function
NAME_plenty_interface_v1(size_t *length) returns immutable bytes and writes
their length. The bytes have no NUL terminator; callers cannot modify or free
them and must keep the code resident. Discovery allocates nothing.
Contracts survive supported ELF stripping and section garbage collection once their archive member is linked. Inspection reads dedicated contract sections, not arbitrary matching strings. It never loads or executes the binary. ELF objects, shared objects, executables, and regular static archives are supported; thin/compressed archives and malformed contracts are rejected without following thin-archive paths.
Inspection validates version, namespace/section agreement, C ABI, UTF-8, and target. Limits are 512 MiB per binary, 4 MiB per contract, 16 MiB total contract data, and 256 contracts. Duplicate namespaces are rejected even if bytes agree. Extraction preserves a previous output on failure and cannot overwrite the input. See build commands for extraction and verification.
Compatibility fingerprints
Generated interfaces carry a SHA-256 fingerprint and require
NAME_plenty_contract_v1_HASH before each native call. An incompatible binary
lacking that symbol fails linking or symbol resolution. The guard allocates
nothing; C headers declare it for optional C-side checking.
Signatures, borrowing, cleanup/adaptation rules, namespace, and target affect the fingerprint. Parameter spelling, docstrings, private layout, and bodies do not. Matching is conservative: adding an export changes the fingerprint. This is not a general stable Plenty binary ABI or a promise of compatibility between compiler/runtime builds when combining static runtimes.
Verification validates the exact contract and checks its guard symbol is defined; shared libraries must expose the guard dynamically. Metadata alone cannot pass. Older format-1 contracts without fingerprints can be extracted but must be regenerated for verification. Verification executes no library or linker code, and does not prevent later binary replacement: the normal symbol guard remains necessary.
Extracted contracts remain trusted .plentyi source. Metadata matching detects
contract mismatch, not malicious code, native bugs, or authenticity.
Runtime loading
A generated runtime interface exposes load(path: &str) -> Result[Library, LoadError]
and a Library method for each export. Importing it does not load native code;
the consumer needs no link argument for the loaded library. Generation currently
requires the original export source and differs from a linked .plentyi
interface. See commands.
Methods preserve supported numeric/unit inputs and results, scalar borrows, Result error types, and owned class handles. Borrowed input mutations remain visible on Err. The generated module is checked before publication.
Loading opens the requested library, compares exact discovery metadata, requires the fingerprint guard, and resolves every export/destructor before returning a usable table. Failed lookup never exposes a partial public table; its private lookup lease closes on every path. Calls use cached native addresses.
LoadError has allocation-free OutOfMemory, CapacityOverflow, InvalidPath,
OpenFailed, InvalidSymbol, MissingSymbol, and IncompatibleContract variants.
Path termination requires one fallible Plenty allocation. The private lease
wrapper and returned method table are inline classes; they allocate nothing.
Native constructors and the OS loader may allocate independently. Cached scalar
calls and adapters need no allocation beyond effects of the native callee.
Generation accepts at most 128 functions including destructors, and 240 parameter
slots per method; consuming owners require additional temporary slots.
Method names beginning _, new, and self, and exported class names Library and
load, are reserved. Native symbols must be shorter than 512 bytes; discovery
metadata must fit 4 MiB.
Owned objects and code residency
Returned class wrappers store their handle, originating identity, and exact destructor address. Inline wrapper construction allocates nothing; the native factory’s owned-handle allocation remains fallible. Factory Err cleans the empty wrapper. Consuming arguments transfer on both Ok and Err; successful class results arm a new wrapper.
Objects may outlive the Library table. Successfully loaded code and native static state remain resident until process exit, so their originating destructor remains callable. Before any class argument crosses the native boundary, methods compare its origin with their table. A different loaded instance traps with a diagnostic, even if contracts match; it is not a LoadError. Repeated loads of the same resident instance are compatible. Calls and objects stay on their creating thread.
A path containing / is absolute or relative to the process working directory;
bare names follow platform search rules. Native constructors can run during open
before a later check fails. Loading trusts native executable code; metadata is
compatibility checking, not authentication.
Trusted loading primitives
Trusted interfaces can implement their own runtime loader with these exact
versioned declarations. They are checked exceptions to the reserved plenty_
symbol namespace; opaque type names are chosen by the binding author.
opaque Lease
opaque Address
extern def open_raw(path: &str as utf8, output: &mut Lease) -> u32 = "plenty_library_open_v1"
extern def symbol_raw(lease: Lease, name: &str as utf8, output: &mut Address) -> u32 = "plenty_library_symbol_v1"
extern def close_raw(lease: Lease) -> () = "plenty_library_close_v1"
extern def check_raw(lease: Lease, discovery: &str as utf8, expected: &str as utf8) -> u32 = "plenty_library_contract_v1"
extern def require_origin(expected: Address, actual: Address) -> () = "plenty_library_require_origin_v1"
Open and lookup return zero and write output only on success. Errors are 1 (OutOfMemory), 2 (CapacityOverflow), 3 (InvalidPath), 4 (OpenFailed), 5 (InvalidSymbol), 6 (MissingSymbol), and 7 (IncompatibleContract). Paths must be nonempty without NUL; symbols must be ASCII C identifiers shorter than 512 bytes. Path conversion is fallible; lookup and error handling allocate nothing. The OS loader has independent memory management.
Successful open grants one lookup lease. Close consumes it exactly once; null is a no-op. Other operations need a live lease, and wrappers must close it on every exit, including partial lookup failure. Closing never unmaps successfully loaded code or native static state; addresses remain valid until process exit. The GNU/Linux implementation uses eager local binding. Imports alone do not open libraries, and native constructors may run before later checks fail.
Private opaque addresses can select typed indirect extern calls. The binding must verify every lookup, exact signature, ownership, and thread contract. Low-level lookup does not automatically verify metadata.
check_raw resolves the named discovery function with C signature
const uint8_t *(size_t *), then compares exact length and bytes with expected.
Expected metadata must be nonempty and at most 4 MiB. Null, wrong-sized, or
different discovery data returns IncompatibleContract; missing discovery returns
MissingSymbol. Comparison allocates nothing. Discovery must supply valid readable
memory; matching cannot prove memory safety or undo native constructor effects.
require_origin compares non-owning identities. Null expected identity or mismatch
terminates with the instance-mismatch diagnostic.
Standard library
The standard library is the set of modules under the reserved std namespace.
They are written in Plenty or expose existing compiler facilities, and are embedded in the compiler: importing one needs no
source installation, network, Cargo, or rustc. Native compilation still requires
the supported linker environment.
Using std.dataframe additionally requires its separately installed
optional native engine; unused imports need no engine.
A standard module must be imported. Names available without an import belong to the prelude and are specified under built-in facilities.
Modules
| Module | Contents | Error type |
|---|---|---|
std.actor | Supervised actors on one thread, each with its own heap, bounded mailbox and copied messages; a trap discards only that actor’s heap and files; timers, request-reply and restart | StartError, Undelivered[M], TimerError, RequestError[M] |
std.actor.net | TCP for actors: a listener that starts one actor per connection with admission limits, connections with bounded buffers, back-pressure and deadlines, and nonblocking connect and lookup | NetError, TimerError, BytesError |
std.actor.tls | TLS for actor connections: a std.tls session over a std.actor.net connection, with plaintext back-pressure and the session in the connection actor’s heap | TlsError, TimerError |
std.actor.http.server | HTTP/1.1 and HTTPS on isolated connection actors, bounded request/response handlers or resumable streaming events, deadlines and draining | ServerError |
std.archive | TAR and ZIP readers and writers over std.io, and extraction into a new directory under explicit path, link, and size limits | ArchiveError |
std.atomic | Inline atomic integers and booleans with compile-time checked orderings, and fences | None |
std.bytes | Compact owning byte buffers with checked ranges, explicit copies and validated text conversion | BytesError |
std.cache.lru | Fixed-capacity caches that evict the least recently used entry, with constant-time lookup, promotion and eviction | AllocError |
std.channel | Bounded channels, endpoints and selection | Channel/send/receive/selection errors |
std.cli | Explicit typed options/positionals, repeatable values, generated help and lossless path arguments | CliError |
std.cli.fileinput | Buffered fallible UTF-8 lines across native paths or command-line files, with stdin fallback | FileInputError |
std.cli.progress | Terminal-only stderr bars and spinners with counts, rates, ETA and throttled redraws | ProgressError |
std.collections | Owned binary heaps ordered by < or by a stored comparator | AllocError |
std.complex | Complex numbers over f64 with arithmetic operators, polar form, exp, log, sqrt, pow, and a+bi display | None |
std.compress | Bounded gzip, zlib, DEFLATE and Brotli compression of buffers and std.io streams, and CRC-32 | CompressError |
std.concurrent | Executors, futures and parallel operations | Pool/submission/future/parallel errors |
std.crypto | Digests, HMAC, BLAKE3 hashing/key derivation, constant-time comparison, AEAD encryption, Ed25519, X25519, and HKDF over std.bytes | CryptoError, AdvancedError |
std.csv | Streaming CSV readers and writers over std.io with explicit dialects, size limits, and positioned errors | CsvError |
std.dataframe | Optional Polars analytical expression plans, typed nullable columns, schema inspection, grouping, and CSV/Parquet I/O | DataFrameError |
std.db.kv | Ordered byte-key maps, atomic transactions and bounded scans over a versioned SQLite store | KvError |
std.db.sqlite | SQLite databases through prepared statements with typed parameters and reads, owned transactions that roll back on drop, and an explicit value-size limit | DbError |
std.deque | Owned generic ring buffers with constant-time removal at either end | AllocError |
std.diff | Line, word, character, byte, and list differences as edit operations, with Myers and patience algorithms and unified-diff output | AllocError |
std.encoding | Hex, base32/base64, MIME base64 and validated custom alphabets, plus fixed-width integer and IEEE 754 float packing with explicit byte order, over std.bytes | EncodingError |
std.env | Environment variables as text or bytes, a detached snapshot, and the working directory | IoError |
std.error | Explicit error mapping, operation/path context, descriptions, and standard-error reports | Mapper-selected error, Context[E], AllocError |
std.fmt | Writer-generic value/report formatting, bounded output and fallible strings | FormatError |
std.fs | Directories, metadata/walking, symbolic links and canonical paths, rooted mutation, owned temporary resources, bounded file/tree copy, metadata preservation, delete/glob | IoError, AllocError, WalkError, FsError, TreeCopyError |
std.fs.dirs | The home directory and the XDG configuration, cache, data, and state directories, from the environment | IoError |
std.fs.lock | Owned Linux shared/exclusive advisory locks, nonblocking and cancellable deadline acquisition | LockError |
std.fs.watch | Bounded Linux filesystem change batches, rename cookies, overflow and readiness waits | WatchError, WaitError |
std.graph | Directed and undirected graphs with node and edge data, stable handles, traversal, shortest paths, topological order, components and spanning trees | AllocError, GraphError |
std.http | Bounded HTTP/1.1 message heads and streamed Content-Length, chunked, and close-delimited bodies over std.io streams | HttpError |
std.http.decoder | Resumable bounded HTTP/1.x request framing over caller-supplied bytes, without I/O | HttpError |
std.http.body | Bounded request bodies for handlers: URL-encoded forms, JSON, and multipart uploads into unnamed temporary files | BodyError |
std.http.client | HTTP/1.1 client: pooled connections, deadlines and cancellation, bounded redirects, an explicit proxy, and replay of idempotent requests only | ClientError |
std.http.cookie | Cookie fields read leniently and Set-Cookie fields validated against injection, with secure defaults | CookieError |
std.http.files | Static files below a Root, opened one segment at a time without following links, with media types and conditional requests | FilesError |
std.http.https | std.http.client for https URLs through std.tls, verified by default, with CONNECT tunnels through a proxy | ClientError |
std.http.proxy | The client address and scheme behind explicitly trusted reverse proxies; deployment guidance | ProxyError |
std.http.server | HTTP/1.1 server: routing, a bounded worker pool and admission queue, idle connections parked off the workers, per-phase deadlines, and graceful shutdown | ServerError |
std.http.sse | Server-Sent Events from a server exchange: event, id, multi-line data, comments and retry, flushed per event, with disconnect and shutdown detection | SseError |
std.http.tls | std.http.server over std.tls, with handshake deadlines | ServerError |
std.http.typed | Typed head parsing and writes, fixed/chunked/close-delimited bodies, and guarded connection recovery with concrete causes and progress | FramingError[E], Partial[E] |
std.http.websocket | RFC 6455 WebSocket connections: the handshake as a server exchange or a ws client, bounded framed messages, pings and the closing handshake | WebSocketError |
std.http.wss | std.http.websocket clients for wss URLs through std.tls | WebSocketError |
std.image | Owned 8-bit pixel buffers, bounded PNG decoding and encoding, cropping, colour conversion and resizing | ImageError |
std.io | Partial-progress byte stream contracts, retry helpers, fixed-capacity buffering, file and process-stream adapters, and checked sequential binary fields in an owned memory cursor | StreamError, Partial, CopyError, CloseError, BinaryError |
std.io.typed | Byte stream protocols that carry a caller-chosen error type, buffering, adapters for std.io streams, and exact read, write, and copy helpers | StepError[E], Partial[E], CopyFailure[R, W] |
std.json | Strict JSON under depth, input, and output limits: resumable event decoder, validating encoder, and a value tree | JsonError |
std.jwt | JSON Web Tokens: compact JWS signing and verification with the algorithm fixed by the key, strict decoding, and explicit-clock claim checks | JwtError |
std.log | Leveled events with typed fields, delivered to caller-chosen sinks: logfmt lines, bounded memory, or a nonblocking channel | LogError |
std.markdown | CommonMark 0.31.2 parsing into events and HTML rendering, under input, depth, and output limits | MarkdownError |
std.template | Bounded Jinja-style text templates with explicit JSON values, HTML escaping, conditions, loops and named inheritance | TemplateError |
std.math | Floating-point functions, checked rounding, integer algorithms, integer and float helpers, and constants | MathError, OverflowError[T] |
std.mime | Media types: RFC 9110 parsing and formatting with parameters, media-range matching, and the file-extension table | ParseError, AllocError |
std.net | TCP listeners and streams, UDP sockets, IP addresses and name lookup, with deadlines and cancellation | NetError |
std.path | Lossless Linux paths, lexical operations, checked text conversion and escaped display | IoError, AllocError |
std.process | Children spawned from an argument vector with no implicit shell, piped streams, bounded output collection, signals, and explicit reaping | IoError, StreamError |
std.random | Caller-owned seeded generators with unbiased ranges, shuffling, and index sampling; not for secrets | RandomError |
std.ready | Readiness waits on pipes and other handles, bounded by a deadline and a cancellation token | WaitError |
std.regex | Linear-time regular expressions over text and bytes: search, captures, iteration, split, and replacement under explicit pattern, memory, and work limits | RegexError |
std.report | Bounded human-readable report rendering with std.fmt | FormatError |
std.secrets | Operating-system entropy: secret bytes, hex and base64url tokens, and unbiased integers, with no fallback source | SecretsError |
std.shlex | Shell-style word splitting and POSIX argument quoting, without execution or expansion | ShlexError, AllocError |
std.sketch | Bounded distinct counts, sampled quantiles and frequent-item bounds with versioned checkpoints | SketchError, AllocError, BytesError |
std.sketch.xor | Static approximate membership over numeric identities with bounded construction | SketchError, AllocError, BytesError |
std.sketch.cuckoo | Scalable approximate membership with bounded growth and receipt-directed deletion | SketchError, AllocError, BytesError |
std.signal | SIGTERM and SIGINT as an event to wait or select on, and a helper that cancels a token on one | SignalError, WaitError |
std.simd | Portable fixed-lane values, masks, checked and wrapping arithmetic, selection, and bounded list loads/stores | None |
std.stat | Descriptive f64 statistics, inclusive quantiles, and normal, uniform and exponential distributions with explicit RNG sampling | StatError |
std.plot | Layered charts from owned columns or record accessors, validated mappings, grouped statistics, and deterministic streaming SVG | PlotError |
std.linalg | Dense f64 matrices, products, solves, LU, QR, Cholesky, SVD and eigen decompositions through an optional faer engine | AllocError, LinalgError |
std.sync | Shared cooperative cancellation tokens | AllocError |
std.system | Bounded Linux procfs snapshots: CPUs, memory, load, uptime, block devices, disk I/O and process status | IoError, SystemError |
std.tempfile | Unnamed and named temporary files, recursively cleaned directories, context managers and explicit ownership transfer | IoError, FsError, StreamError, AllocError |
std.test | Assertions that explain their failure on standard error, self-removing scratch directories, the seeded generator for plenty test, and fake clocks and streams | TestError |
std.text | Bounded builders, joining, scalar/word wrapping, shortening, and indentation | TextError |
std.thread | Explicit scoped thread and owned-job imports | ThreadError, SpawnError[F] |
std.time | Checked durations, monotonic instants and deadlines, wall timestamps, and sleep | RangeError |
std.tls | TLS 1.2 and 1.3 client and server streams over std.net TCP, with verified certificates, client certificates, ALPN, deadlines, and cancellation | TlsError |
std.toml | Strict TOML 1.0 parsing, value-tree writing and format-preserving document edits, under depth, input, and output limits | TomlError |
std.unicode | Unicode 17 normalization, combining classes, and Mark classification | TextError |
std.units | SI dimensions, prefixed and common units, physical constants, and dimension-aware square roots, sums and trigonometry over Quantity | MathError, RangeError |
std.units.catalogue | Every SI prefix on every SI unit and the other named units of NIST SP 811 | None |
std.url | Strict RFC 3986 absolute URL parsing and formatting without normalization, component-aware percent encoding, and query pairs | UrlError |
std.uuid | UUID values, canonical text and network bytes, random v4 and timestamp-prefixed v7 generation | ParseError, BytesError, SecretsError, RangeError |
std.xml | Bounded UTF-8 XML event readers and validating stream writers, without DTDs or external entities | XmlError |
The generated signature index lists every public declaration of every module on one page. Standard library conventions states the naming, error, parameter and type rules public modules follow, and where modules still differ.
Planned modules appear only in the backlog. The tutorial introduces the library in Use the standard math library, Enumerate and classify paths, and Prune before descending.
Importing a standard module
Standard modules use the ordinary import forms:
import std.math
import std.math as math
from std.math import sqrt, pi
from std.math import MathError as Error
An import binds only the names it lists. import std.math requires the full
std.math.sqrt qualification; an alias shortens it.
std is reserved. Each of these is an error:
- a
stddirectory,std.plenty, orstd.plentyiin the source root; - a declaration named
std; - an import that binds a non-standard module as
std.
An unknown standard module, including a bare import std, is an error that
lists the available modules. It never falls back to a local file.
How standard modules are compiled
A standard module passes through ordinary parsing, visibility checks, type
checking, specialization, and native emission. Its private helpers stay private.
Repeated imports, through any alias or dependency path, share its nominal
types, so a MathError from one module equals a MathError from another.
The scope rationale
records when a module uses a native library instead of Plenty source.
The four concurrency modules export canonical builtin identities instead of wrapper functions. Their imports reach the existing ownership and worker checks directly, including scoped joins and blocking-wait diagnostics. Their prelude spellings remain available. Importing one of these under its own builtin name is permitted only when it denotes the same builtin; unrelated name collisions remain errors. Generic constructor and error aliases retain the same types.
Embedded modules have toolchain identities such as std/math.plenty,
independent of the installation path. A diagnostic located inside a standard
module, such as a note on a generic helper instantiated with an unsupported
type, uses that name. The source-only APIs (check_source, compile_source_*)
have no source root, so they can import std modules but not project modules.
The runtime packaging page describes how the sources ship with the compiler.
Reading a module page
Each module page gives every public function’s signature and states:
- whether it returns a plain value or a
Result, and which error variants it reports; - whether it borrows or consumes its arguments;
- whether it allocates;
- the limits that differ from the Python module of the same name.
Recoverable failures use typed errors: std.math defines MathError, while
std.path and std.fs share the native IoError and storage AllocError.
The filesystem walker adds WalkError to retain attempted paths and operations.
? propagates an error to a caller returning the same error type or
Failure.
std.error states the conventions modules follow for typed causes,
native codes, and allocation failure. Module functions accept
named arguments and defaults;
the signature index shows each default.
Acknowledgements
- The breadth of Python’s standard library motivated the module catalogue.
Standard library signatures
Every public declaration of each std module, one line each, so you can
check that an operation exists and what it returns before reading its
module page. The page states the contracts: ownership, allocation, error
variants, and limits. pub is omitted. Fields, methods, and variants are
indented under their owner. Summaries come from the
module table.
After changing a standard module, regenerate this page with
UPDATE_STDLIB_INDEX=1 cargo test --locked --test integration test_stdlib_index::.
Modules
Each section’s lines in this file, so one module can be read alone.
std.actor: 110-187std.actor.net: 189-252std.actor.tls: 254-283std.actor.http.server: 285-348std.archive: 350-416std.atomic: 418-427std.bytes: 429-476std.cache.lru: 478-500std.channel: 502-506std.cli: 508-538std.cli.fileinput: 540-549std.cli.progress: 551-575std.collections: 577-604std.complex: 606-632std.compress: 634-664std.concurrent: 666-670std.crypto: 672-753std.csv: 755-818std.dataframe: 820-1019std.db.kv: 1021-1053std.db.sqlite: 1055-1119std.deque: 1121-1135std.diff: 1137-1166std.encoding: 1168-1235std.env: 1237-1251std.error: 1253-1273std.fmt: 1275-1342std.fs: 1344-1443std.fs.dirs: 1445-1455std.fs.lock: 1457-1471std.fs.watch: 1473-1508std.graph: 1510-1598std.http: 1600-1670std.http.decoder: 1672-1689std.http.body: 1691-1726std.http.client: 1728-1838std.http.cookie: 1840-1872std.http.files: 1874-1887std.http.https: 1889-1903std.http.proxy: 1905-1926std.http.server: 1928-2013std.http.sse: 2015-2034std.http.tls: 2036-2053std.http.typed: 2055-2088std.http.websocket: 2090-2125std.http.wss: 2127-2133std.image: 2135-2181std.io: 2183-2289std.io.typed: 2291-2329std.json: 2331-2395std.jwt: 2397-2444std.log: 2446-2513std.markdown: 2515-2549std.template: 2551-2575std.math: 2577-2648std.mime: 2650-2676std.net: 2678-2760std.path: 2762-2779std.process: 2781-2830std.random: 2832-2853std.ready: 2855-2886std.regex: 2888-2941std.report: 2943-2950std.secrets: 2952-2978std.shlex: 2980-2989std.sketch: 2991-3029std.sketch.xor: 3031-3042std.sketch.cuckoo: 3044-3059std.signal: 3061-3078std.simd: 3080-3181std.stat: 3183-3216std.plot: 3218-3307std.linalg: 3309-3350std.sync: 3352-3370std.system: 3372-3430std.tempfile: 3432-3464std.test: 3466-3508std.text: 3510-3553std.thread: 3555-3559std.time: 3561-3605std.tls: 3607-3689std.toml: 3691-3740std.unicode: 3742-3754std.units: 3756-3993std.units.catalogue: 3995-4773std.url: 4775-4803std.uuid: 4805-4828std.xml: 4830-4863
std.actor
Supervised actors on one thread, each with its own heap, bounded mailbox and copied messages; a trap discards only that actor’s heap and files; timers, request-reply and restart. Errors: StartError, Undelivered[M], TimerError, RequestError[M]. Reference.
const max_heap_limit: i64 = 67108864
const max_spawns: i64 = 16
const max_capacity: i64 = 1048576
class Address
def key(self) -> u64
class Watch
def key(self) -> u64
class Timer
def key(self) -> u64
def tag(self) -> i64
class Request
def key(self) -> u64
enum Reply[M]
Answer(M) | TimedOut | Gone
enum TimerError
Capacity | Range
enum RequestError[M]
Full(M) | Gone(M) | Capacity(M)
enum Exit[E]
# Stopped by `Context.stop`, a `Stop` request, or a spawn hook that | Failed(E) | Trapped(TrapKind)
enum TrapKind
Memory | Index | Arithmetic | Unwrap | Other
enum Mail[M, E]
Message(M) | Ready(Watch, Readiness) | Down(Address, i64, Exit[E]) | Stop | Timer(Timer) | Request(Request, M) | Reply(Request, Reply[M])
enum Undelivered[M]
Full(M) | Gone(M)
enum StageError
Allocation(AllocError) | Wait(WaitError)
enum StartError
Unsupported | Capacity | Heaps | Mailbox
class Stats
started: i64
stopped: i64
failed: i64
trapped: i64
restarted: i64
delivered: i64
full: i64
gone: i64
lost: i64
def supported() -> bool
def heaps_live() -> i64
def heap_bytes_used() -> i64
def heap_bytes_committed() -> i64
class Context[M, E]
def address(self) -> Address
def stop(self: &mut Context[M, E]) -> ()
def heap_used(self) -> i64
def send(self: &mut Context[M, E], to: Address, message: M) -> Result[(), Undelivered[M]]
def stop_actor(self: &mut Context[M, E], to: Address) -> bool
def spawn(self: &mut Context[M, E], tag: i64, limit: Positive[i64]) -> Result[Address, StartError]
def spawn_isolated(self: &mut Context[M, E], tag: i64, limit: Positive[i64]) -> Result[Address, StartError]
def restart(self: &mut Context[M, E], child: Address, limit: i64, window: &Duration) -> bool
def after(self: &mut Context[M, E], delay: &Duration, tag: i64) -> Result[Timer, TimerError]
def every(self: &mut Context[M, E], period: &Duration, tag: i64) -> Result[Timer, TimerError]
def cancel(self: &mut Context[M, E], timer: Timer) -> bool
def request(self: &mut Context[M, E], to: Address, message: M, timeout: &Duration) -> Result[Request, RequestError[M]]
def reply(self: &mut Context[M, E], request: Request, message: M) -> Result[(), Undelivered[M]]
def watch[H: Watchable](self: &mut Context[M, E], handle: &H, interest: Interest) -> Result[Watch, WaitError]
def unwatch[H: Watchable](self: &mut Context[M, E], handle: &H) -> Result[(), WaitError]
class Stage[N, M, E]
def shutdown(self: &mut Stage[N, M, E]) -> ()
def start(self: &mut Stage[N, M, E], node: N, limit: Positive[i64]) -> Result[Address, StartError]
def send(self: &mut Stage[N, M, E], to: Address, message: M) -> Result[(), Undelivered[M]]
def stop_actor(self: &mut Stage[N, M, E], to: Address) -> bool
def add_timers(self: &mut Stage[N, M, E], count: i64) -> Result[(), AllocError]
def live(self) -> i64
def stats(self) -> Stats
def node(self, actor: Address) -> Option[&N]
def heap_used(self, actor: Address) -> Option[i64]
def run(self: &mut Stage[N, M, E], deadline: &Deadline, cancel: &CancellationToken) -> Result[(), WaitError]
def stage[N, M, E](capacity: i64, mailbox: i64, events: i64) -> Result[Stage[N, M, E], StageError]
std.actor.net
TCP for actors: a listener that starts one actor per connection with admission limits, connections with bounded buffers, back-pressure and deadlines, and nonblocking connect and lookup. Errors: NetError, TimerError, BytesError. Reference.
class Limits
input: Positive[i64]
output: Positive[i64]
idle: Duration
stall: Duration
enum Expired
Idle | Read | Write
class Events
mine: bool
received: i64
ended: bool
writable: bool
drained: bool
expired: Option[Expired]
class Connection
def start[M, E](self: &mut Connection, context: &mut Context[M, E]) -> Result[(), NetError | TimerError]
def handle[M, E](self: &mut Connection, mail: &Mail[M, E], context: &mut Context[M, E]) -> Result[Events, NetError | TimerError]
def input(self) -> &Bytes
def input_start(self) -> i64
def input_stop(self) -> i64
def available(self) -> i64
def consume[M, E](self: &mut Connection, count: i64, context: &mut Context[M, E]) -> Result[(), NetError | TimerError]
def pending(self) -> i64
def space(self) -> i64
def write[M, E](self: &mut Connection, source: &[u8], context: &mut Context[M, E]) -> Result[i64, NetError | TimerError]
def reserve(self: &mut Connection, wanted: NonNegative[i64]) -> Result[i64, NetError]
def output(self: &mut Connection) -> &mut Bytes
def output_stop(self) -> i64
def commit[M, E](self: &mut Connection, count: i64, context: &mut Context[M, E]) -> Result[(), NetError | TimerError]
def forward[M, E](self: &mut Connection, count: i64, context: &mut Context[M, E]) -> Result[i64, NetError | TimerError]
def read_within[M, E](self: &mut Connection, span: &Duration, context: &mut Context[M, E]) -> Result[(), NetError | TimerError]
def clear_read_deadline(self: &mut Connection) -> ()
def shutdown_write[M, E](self: &mut Connection, context: &mut Context[M, E]) -> Result[(), NetError | TimerError]
def close[M, E](self: &mut Connection, context: &mut Context[M, E]) -> Result[(), NetError]
def acknowledge_now(self: &mut Connection) -> ()
def reading(self) -> bool
def closed(self) -> bool
def stream(self) -> &TcpStream
def connection(stream: TcpStream, limits: &Limits) -> Result[Connection, AllocError]
enum Admission
Pause | Refuse
class Listener
def start[M, E](self: &mut Listener, context: &mut Context[M, E]) -> Result[(), NetError]
def handle[M, E](self: &mut Listener, mail: &Mail[M, E], context: &mut Context[M, E]) -> Result[bool, NetError]
def accept(self) -> Result[Option[Connection], NetError | AllocError]
def live(self) -> i64
def refused(self) -> i64
def tag(self) -> i64
def local_address(self) -> Result[SocketAddress, NetError]
def listener(socket: TcpListener, limits: Limits, capacity: Positive[i64], heap: Positive[i64], tag: i64, admission: Admission) -> Result[Listener, NetError]
class Dialing
def handle[M, E](self: &mut Dialing, mail: &Mail[M, E], context: &mut Context[M, E]) -> Result[Option[TcpStream], NetError]
def abandon[M, E](self: &mut Dialing, context: &mut Context[M, E]) -> Result[(), NetError]
def dial[M, E](address: &SocketAddress, timeout: &Duration, context: &mut Context[M, E]) -> Result[Dialing, NetError | TimerError]
class Resolving
def handle[M, E](self: &mut Resolving, mail: &Mail[M, E], context: &mut Context[M, E]) -> Result[Option[list[SocketAddress]], NetError]
def abandon[M, E](self: &mut Resolving, context: &mut Context[M, E]) -> Result[(), NetError]
def lookup[M, E](host: &str, port: u16, timeout: &Duration, context: &mut Context[M, E]) -> Result[Resolving, NetError | TimerError]
std.actor.tls
TLS for actor connections: a std.tls session over a std.actor.net connection, with plaintext back-pressure and the session in the connection actor’s heap. Errors: TlsError, TimerError. Reference.
class TlsConnection
def start[M, E](self: &mut TlsConnection, context: &mut Context[M, E]) -> Result[(), TlsError | TimerError]
def handle[M, E](self: &mut TlsConnection, mail: &Mail[M, E], context: &mut Context[M, E]) -> Result[Events, TlsError | TimerError]
def input(self) -> &Bytes
def input_start(self) -> i64
def input_stop(self) -> i64
def available(self) -> i64
def consume[M, E](self: &mut TlsConnection, count: i64, context: &mut Context[M, E]) -> Result[(), TlsError | TimerError]
def pending(self) -> i64
def space(self) -> i64
def write[M, E](self: &mut TlsConnection, source: &[u8], context: &mut Context[M, E]) -> Result[i64, TlsError | TimerError]
def forward[M, E](self: &mut TlsConnection, count: i64, context: &mut Context[M, E]) -> Result[i64, TlsError | TimerError]
def read_within[M, E](self: &mut TlsConnection, span: &Duration, context: &mut Context[M, E]) -> Result[(), TlsError | TimerError]
def clear_read_deadline(self: &mut TlsConnection) -> ()
def shutdown_write[M, E](self: &mut TlsConnection, context: &mut Context[M, E]) -> Result[(), TlsError | TimerError]
def close[M, E](self: &mut TlsConnection, context: &mut Context[M, E]) -> Result[(), TlsError]
def established(self) -> bool
def reading(self) -> bool
def closed(self) -> bool
def session(self) -> &Session
def connection(self) -> &Connection
def over(net: Connection, session: Session, input: Positive[i64]) -> Result[TlsConnection, TlsError]
def server(net: Connection, config: &ServerConfig, input: Positive[i64]) -> Result[TlsConnection, TlsError]
def client(net: Connection, config: &ClientConfig, server_name: &str, input: Positive[i64]) -> Result[TlsConnection, TlsError]
std.actor.http.server
HTTP/1.1 and HTTPS on isolated connection actors, bounded request/response handlers or resumable streaming events, deadlines and draining. Errors: ServerError. Reference.
enum ServerError
InvalidConfig | AdmissionStopped | Allocation(AllocError) | Bytes(BytesError) | Http(HttpError) | Net(NetError) | Tls(TlsError) | Timer(TimerError) | Wait(WaitError) | Actor(std.actor.StartError) | Range(RangeError) | Handler
enum Notice[M]
Message(M) | Drain | Resume | Completed | CompletedBy(Address) | Rejected
enum Event[M]
Head(RequestHead, RequestTarget) | Body(Bytes) | Trailers(list[std.http.Header]) | End | Writable | Actor(Mail[Notice[M], ServerError]) | Message(M) | Stopping
class Config
capacity: i64
heap_limit: i64
input_buffer: i64
output_buffer: i64
pending_output: i64
limits: Limits
idle_timeout: Duration
head_timeout: Duration
body_timeout: Duration
write_timeout: Duration
response_timeout: Duration
handshake_timeout: Duration
drain_timeout: Duration
max_requests: Positive[i64]
class Report
actors: Stats
requests: u64
rejected: u64
protocol_errors: u64
protocol Handler[M, E]
def receive(self: &mut Handler[M, E], event: Event[M], response: &mut Response, context: &mut Context[Notice[M], ServerError]) -> Result[(), E]
class Response
def is_stopping(self) -> bool
def close_after(self: &mut Response) -> ()
def shutdown_server(self: &mut Response) -> ()
def is_started(self) -> bool
def is_done(self) -> bool
def pending(self) -> i64
def space(self) -> i64
def start(self: &mut Response, head: ResponseHead) -> Result[(), ServerError]
def write(self: &mut Response, source: &[u8]) -> Result[i64, ServerError]
def finish(self: &mut Response) -> Result[(), ServerError]
def respond(self: &mut Response, head: ResponseHead, body: &[u8]) -> Result[(), ServerError]
def send(self: &mut Response, status: i64, content_type: &str, body: &[u8]) -> Result[(), ServerError]
def send_text(self: &mut Response, status: i64, text: &str) -> Result[(), ServerError]
def config() -> Config
class Node[F, H]
def started[M](self: &mut Node[F, H], context: &mut Context[Notice[M], ServerError]) -> Result[(), ServerError]
def receive[M](self: &mut Node[F, H], mail: Mail[Notice[M], ServerError], context: &mut Context[Notice[M], ServerError]) -> Result[(), ServerError]
def spawn[M](self, tag: i64, child: Address, context: &mut Context[Notice[M], ServerError]) -> Result[Option[Node[F, H]], ServerError]
class Request
head: RequestHead
target: RequestTarget
body: Bytes
trailers: list[std.http.Header]
protocol RequestHandler[E]
def handle(self: &mut RequestHandler[E], request: Request, response: &mut Response) -> Result[(), E]
def serve[F: Callable[[], Result[H, FE]], H: RequestHandler[HE], FE, HE](listener: TcpListener, factory: F, config: &Config, stop: &CancellationToken) -> Result[Report, ServerError]
def serve_tls[F: Callable[[], Result[H, FE]], H: RequestHandler[HE], FE, HE](listener: TcpListener, tls: ServerConfig, factory: F, config: &Config, stop: &CancellationToken) -> Result[Report, ServerError]
def serve_events[F: Callable[[], Result[H, FE]], H: Handler[M, HE], M, FE, HE](listener: TcpListener, factory: F, config: &Config, stop: &CancellationToken) -> Result[Report, ServerError]
def serve_tls_events[F: Callable[[], Result[H, FE]], H: Handler[M, HE], M, FE, HE](listener: TcpListener, tls: ServerConfig, factory: F, config: &Config, stop: &CancellationToken) -> Result[Report, ServerError]
std.archive
TAR and ZIP readers and writers over std.io, and extraction into a new directory under explicit path, link, and size limits. Errors: ArchiveError. Reference.
enum ArchiveError
Stream(StreamError) | Io(IoError) | Fs(FsError) | Allocation(AllocError) | Corrupt | Checksum | Truncated | Unsupported | UnsafePath | Link | Special | Duplicate | Exists | Limit | Cancelled | InvalidInput
enum EntryKind
File | Directory | Symlink | Hardlink | Other
class Entry
name: Path
kind: EntryKind
size: i64
compressed: i64
mode: i64
mtime: i64
link: Path
class Policy
max_entries: i64
max_entry_size: i64
max_total_size: i64
max_ratio: i64
max_depth: i64
max_name: i64
skip_links: bool
skip_special: bool
def policy() -> Policy
class Summary
entries: i64
files: i64
bytes: i64
skipped: i64
def stream_error(error: ArchiveError) -> StreamError
class TarReader[R]
inner: R
def failure(self) -> Option[ArchiveError]
def next(self: &mut TarReader[R]) -> Result[Option[Entry], ArchiveError]
def read(self: &mut TarReader[R], buffer: &mut Bytes, start: i64, stop: i64) -> Result[i64, StreamError]
def tar_reader[R](inner: R) -> Result[TarReader[R], ArchiveError]
class TarWriter[W]
inner: W
def add_file(self: &mut TarWriter[W], name: &Path, data: &[u8], mode: i64, mtime: i64) -> Result[(), ArchiveError]
def add_from[S](self: &mut TarWriter[W], name: &Path, size: i64, source: &mut S, mode: i64, mtime: i64) -> Result[(), ArchiveError]
def add_dir(self: &mut TarWriter[W], name: &Path, mode: i64, mtime: i64) -> Result[(), ArchiveError]
def add_symlink(self: &mut TarWriter[W], name: &Path, target: &Path, mtime: i64) -> Result[(), ArchiveError]
def finish(self: &mut TarWriter[W]) -> Result[(), ArchiveError]
def tar_writer[W](inner: W) -> Result[TarWriter[W], ArchiveError]
class ZipReader[R]
inner: R
def len(self) -> i64
def entries(self) -> &list[Entry]
def failure(self) -> Option[ArchiveError]
def open(self: &mut ZipReader[R], index: i64) -> Result[(), ArchiveError]
def read(self: &mut ZipReader[R], buffer: &mut Bytes, start: i64, stop: i64) -> Result[i64, StreamError]
def zip_reader[R](inner: R, max_entries: NonNegative[i64]) -> Result[ZipReader[R], ArchiveError]
enum Method
Stored | Deflate
class ZipWriter[W]
inner: W
def add_file(self: &mut ZipWriter[W], name: &Path, data: &[u8], method: Method, mode: i64, mtime: i64) -> Result[(), ArchiveError]
def add_dir(self: &mut ZipWriter[W], name: &Path, mode: i64, mtime: i64) -> Result[(), ArchiveError]
def add_symlink(self: &mut ZipWriter[W], name: &Path, target: &Path, mtime: i64) -> Result[(), ArchiveError]
def finish(self: &mut ZipWriter[W]) -> Result[(), ArchiveError]
def zip_writer[W](inner: W) -> Result[ZipWriter[W], ArchiveError]
def extract_tar[R](archive: &mut TarReader[R], parent: &Root, destination: &Path, policy: &Policy, cancel: &CancellationToken) -> Result[Summary, ArchiveError]
def extract_zip[R](archive: &mut ZipReader[R], parent: &Root, destination: &Path, policy: &Policy, cancel: &CancellationToken) -> Result[Summary, ArchiveError]
std.atomic
Inline atomic integers and booleans with compile-time checked orderings, and fences. Reference.
Compiler-provided: Atomic, fence. Their signatures are on the reference page.
enum Ordering
Relaxed | Release | Acquire | AcqRel | SeqCst
std.bytes
Compact owning byte buffers with checked ranges, explicit copies and validated text conversion. Errors: BytesError. Reference.
enum BytesError
InvalidUtf8 | Allocation(AllocError)
class Bytes
def len(self) -> i64
def is_empty(self) -> bool
def write_float(self: &mut Bytes, value: f64) -> i64
def write_float_spec(self: &mut Bytes, value: f64, style: i64, precision: i64, offset: i64) -> i64
def capacity(self) -> i64
def get(self, index: i64) -> u8
def set(self: &mut Bytes, index: i64, value: u8) -> ()
def push(self: &mut Bytes, value: u8) -> Result[(), AllocError]
def pop(self: &mut Bytes) -> Option[u8]
def reserve(self: &mut Bytes, additional: i64) -> Result[(), AllocError]
def resize(self: &mut Bytes, count: i64, fill: u8) -> Result[(), AllocError]
def truncate(self: &mut Bytes, count: i64) -> ()
def clear(self: &mut Bytes) -> ()
def fill(self: &mut Bytes, start: i64, stop: i64, value: u8) -> ()
def extend(self: &mut Bytes, source: &[u8]) -> Result[(), AllocError]
def overwrite(self: &mut Bytes, offset: i64, source: &[u8]) -> ()
def copy_within(self: &mut Bytes, start: i64, stop: i64, to: i64) -> ()
def slice(self, start: i64, stop: i64) -> Result[Bytes, AllocError]
def copy(self) -> Result[Bytes, AllocError]
def find(self, value: u8, start: i64, stop: i64) -> Option[i64]
def find_in(self, set: &[u8], start: i64, stop: i64) -> Option[i64]
def find_not_in(self, set: &[u8], start: i64, stop: i64) -> Option[i64]
def find_bytes(self, needle: &[u8], start: i64, stop: i64) -> Option[i64]
def count(self, value: u8, start: i64, stop: i64) -> i64
def matches(self, offset: i64, source: &[u8]) -> bool
def matches_text(self, offset: i64, text: &str) -> bool
def ascii_run(self, start: i64, stop: i64) -> i64
def utf8_run(self, start: i64, stop: i64) -> i64
def to_list(self) -> Result[list[u8], AllocError]
def to_text(self) -> Result[str, BytesError]
def to_text_range(self, start: i64, stop: i64) -> Result[str, BytesError]
def read_file(self: &mut Bytes, file: &mut File, start: i64, stop: i64) -> Result[i64, IoError]
def read_stdin(self: &mut Bytes, start: i64, stop: i64) -> Result[i64, IoError]
def empty() -> Result[Bytes, AllocError]
def with_capacity(capacity: i64) -> Result[Bytes, AllocError]
def from_list(values: &list[u8]) -> Result[Bytes, AllocError]
def from_text(text: &str) -> Result[Bytes, AllocError]
def from_bytes(source: &[u8]) -> Result[Bytes, AllocError]
def write_file(file: &mut File, source: &[u8]) -> Result[i64, IoError]
std.cache.lru
Fixed-capacity caches that evict the least recently used entry, with constant-time lookup, promotion and eviction. Errors: AllocError. Reference.
class LruCache[K, V]
def __init__(self, capacity: i64) -> Result[(), AllocError]
def capacity(self) -> i64
def len(self) -> i64
def is_empty(self) -> bool
def contains(self, key: &K) -> bool
def get(self: &mut LruCache[K, V], key: &K) -> Option[V]
def peek(self, key: &K) -> Option[V]
def get_ref(self: &mut LruCache[K, V], key: &K) -> Option[&V]
def get_mut(self: &mut LruCache[K, V], key: &K) -> Option[&mut V]
def peek_ref(self, key: &K) -> Option[&V]
def peek_mut(self: &mut LruCache[K, V], key: &K) -> Option[&mut V]
def promote(self: &mut LruCache[K, V], key: &K) -> bool
def put(self: &mut LruCache[K, V], key: K, value: V) -> Result[Option[(K, V)], AllocError]
def remove(self: &mut LruCache[K, V], key: &K) -> Option[V]
def pop_lru(self: &mut LruCache[K, V]) -> Option[(K, V)]
def clear(self: &mut LruCache[K, V]) -> ()
std.channel
Bounded channels, endpoints and selection. Errors: Channel/send/receive/selection errors. Reference.
Compiler-provided: channel, Sender, Receiver, SendError, RecvError, SendTimeoutError, RecvTimeoutError, Selected, SelectError, select_recv, select_recv_nowait, select_recv_timeout, select_recv_timeout_ms. Their signatures are on the reference page.
std.cli
Explicit typed options/positionals, repeatable values, generated help and lossless path arguments. Errors: CliError. Reference.
enum Kind
Flag | Integer | Text | Path
class Spec
name: str
kind: Kind
positional: bool
required: bool
repeatable: bool
help: str
enum Value
Flag | Integer(i64) | Text(str) | Path(Path)
class Entry
spec: i64
value: Value
class Parsed
help_requested: bool
entries: list[Entry]
enum CliError
Schema(i64) | UnknownOption(i64) | UnexpectedArgument(i64) | MissingValue(i64) | Duplicate(i64) | Missing(i64) | InvalidInteger(i64) | InvalidUtf8(i64) | Allocation(AllocError)
def validate(schema: &list[Spec]) -> Result[(), CliError]
def parse(schema: &list[Spec], arguments: &list[list[u8]], start: i64) -> Result[Parsed, CliError]
def help(schema: &list[Spec], program: &str) -> Result[str, CliError]
def exit_status(error: &CliError) -> i32
def describe(error: &CliError) -> Result[str, AllocError]
def describe_for(error: &CliError, schema: &list[Spec]) -> Result[str, AllocError]
std.cli.fileinput
Buffered fallible UTF-8 lines across native paths or command-line files, with stdin fallback. Errors: FileInputError. Reference.
enum FileInputError
Open(Option[Path], StreamError) | Read(Option[Path], StreamError) | Text(Option[Path], BytesError) | Close(Option[Path], StreamError)
def files(paths: list[Path]) -> Generator[Result[str, FileInputError]]
def lines() -> Result[Generator[Result[str, FileInputError]], IoError]
std.cli.progress
Terminal-only stderr bars and spinners with counts, rates, ETA and throttled redraws. Errors: ProgressError. Reference.
enum ProgressError
InvalidPosition | Finished | Output(FormatError)
class Progress
def position(self) -> u64
def total(self) -> Option[u64]
def is_finished(self) -> bool
def rate(self, now: &Instant) -> Option[f64]
def eta(self, now: &Instant) -> Option[f64]
def render[W](self, now: &Instant, output: &mut Formatter[W]) -> Result[(), FormatError]
def set_position_at(self: &mut Progress, position: u64, now: &Instant) -> Result[(), ProgressError]
def set_position(self: &mut Progress, position: u64) -> Result[(), ProgressError]
def advance(self: &mut Progress, count: u64 = 1u64) -> Result[(), ProgressError]
def tick(self: &mut Progress) -> Result[(), ProgressError]
def finish_at(self: &mut Progress, now: &Instant) -> Result[(), ProgressError]
def finish(self: &mut Progress) -> Result[(), ProgressError]
def bar_at(total: u64, now: &Instant) -> Progress
def bar(total: u64) -> Progress
def spinner_at(now: &Instant) -> Progress
def spinner() -> Progress
std.collections
Owned binary heaps ordered by < or by a stored comparator. Errors: AllocError. Reference.
class Heap[T]
def __init__(self) -> Result[(), AllocError]
def len(self) -> i64
def is_empty(self) -> bool
def reserve(self: &mut Heap[T], additional: i64) -> Result[(), AllocError]
def peek(self) -> Option[T]
def peek_ref(self) -> Option[&T]
def push(self: &mut Heap[T], value: T) -> Result[(), AllocError]
def pop(self: &mut Heap[T]) -> Option[T]
def clear(self: &mut Heap[T]) -> ()
class HeapBy[T, F: Callable[[&T, &T], bool]]
def __init__(self, before: F) -> Result[(), AllocError]
def len(self) -> i64
def is_empty(self) -> bool
def reserve(self: &mut HeapBy[T, F], additional: i64) -> Result[(), AllocError]
def peek(self) -> Option[T]
def peek_ref(self) -> Option[&T]
def push(self: &mut HeapBy[T, F], value: T) -> Result[(), AllocError]
def pop(self: &mut HeapBy[T, F]) -> Option[T]
def clear(self: &mut HeapBy[T, F]) -> ()
def drain[T](heap: Heap[T]) -> Generator[T]
def drain_by[T, F: Callable[[&T, &T], bool]](heap: HeapBy[T, F]) -> Generator[T]
std.complex
Complex numbers over f64 with arithmetic operators, polar form, exp, log, sqrt, pow, and a+bi display. Reference.
class Complex
re: f64
im: f64
def __add__(self, other: &Complex) -> Complex
def __sub__(self, other: &Complex) -> Complex
def __mul__(self, other: &Complex) -> Complex
def __truediv__(self, other: &Complex) -> Complex
def __neg__(self) -> Complex
def scale(self, factor: f64) -> Complex
def conj(self) -> Complex
def norm(self) -> f64
def abs(self) -> f64
def arg(self) -> f64
def to_polar(self) -> (f64, f64)
def exp(self) -> Complex
def log(self) -> Complex
def sqrt(self) -> Complex
def pow(self, exponent: &Complex) -> Complex
def powi(self, n: i64) -> Complex
def display[W](self, out: &mut Formatter[W]) -> Result[(), FormatError]
def from_polar(r: f64, theta: f64) -> Complex
std.compress
Bounded gzip, zlib, DEFLATE and Brotli compression of buffers and std.io streams, and CRC-32. Errors: CompressError. Reference.
enum Format
Deflate | Zlib | Gzip | Brotli
enum CompressError
OutOfRange | Corrupt | Checksum | Truncated | TrailingData | LimitExceeded | Allocation(AllocError)
def stream_error(error: CompressError) -> StreamError
class Inflater
def step(self: &mut Inflater, input: &[u8]) -> Result[i64, CompressError]
def complete(self) -> bool
def output(self) -> &Bytes
def clear(self: &mut Inflater) -> ()
def inflater(format: Format, limit: NonNegative[i64]) -> Result[Inflater, CompressError]
def compress(format: Format, data: &[u8], level: i64) -> Result[Bytes, CompressError]
def decompress(format: Format, data: &[u8], limit: NonNegative[i64]) -> Result[Bytes, CompressError]
def crc32(seed: i64, data: &[u8]) -> Result[i64, CompressError]
class Compressor[W]
inner: W
def write(self: &mut Compressor[W], source: &[u8]) -> Result[i64, StreamError]
def flush(self: &mut Compressor[W]) -> Result[(), StreamError]
def finish(self: &mut Compressor[W]) -> Result[(), StreamError]
def compressor[W](format: Format, inner: W, level: i64) -> Result[Compressor[W], CompressError]
class Decompressor[R]
inner: R
def failure(self) -> Option[CompressError]
def read(self: &mut Decompressor[R], buffer: &mut Bytes, start: i64, stop: i64) -> Result[i64, StreamError]
def decompressor[R](format: Format, inner: R, limit: NonNegative[i64]) -> Result[Decompressor[R], CompressError]
std.concurrent
Executors, futures and parallel operations. Errors: Pool/submission/future/parallel errors. Reference.
Compiler-provided: ThreadPoolExecutor, Future, PoolError, SubmitError, FutureError, PoolMapError, ParallelError. Their signatures are on the reference page.
std.crypto
Digests, HMAC, BLAKE3 hashing/key derivation, constant-time comparison, AEAD encryption, Ed25519, X25519, and HKDF over std.bytes. Errors: CryptoError, AdvancedError. Reference.
enum CryptoError
Finished | InvalidKeyLength | UnsupportedAlgorithm | Allocation(AllocError)
enum Algorithm
Sha256 | Sha512 | Sha1 | Blake3
def output_size(algorithm: Algorithm) -> i64
class Digest
def size(self) -> i64
def update(self: &mut Digest, data: &[u8]) -> Result[(), CryptoError]
def update_text(self: &mut Digest, text: &str) -> Result[(), CryptoError]
def finish(self: &mut Digest) -> Result[Bytes, CryptoError]
def finish_into(self: &mut Digest, target: &mut Bytes, offset: i64) -> Result[(), CryptoError]
class Mac
def size(self) -> i64
def update(self: &mut Mac, data: &[u8]) -> Result[(), CryptoError]
def update_text(self: &mut Mac, text: &str) -> Result[(), CryptoError]
def finish(self: &mut Mac) -> Result[Bytes, CryptoError]
def finish_into(self: &mut Mac, target: &mut Bytes, offset: i64) -> Result[(), CryptoError]
def verify(self: &mut Mac, expected: &[u8]) -> Result[bool, CryptoError]
def digest(algorithm: Algorithm) -> Result[Digest, CryptoError]
def hmac(algorithm: Algorithm, key: &[u8]) -> Result[Mac, CryptoError]
def sha256(data: &[u8]) -> Result[Bytes, CryptoError]
def sha512(data: &[u8]) -> Result[Bytes, CryptoError]
def blake3(data: &[u8]) -> Result[Bytes, CryptoError]
def blake3_keyed_hasher(key: &[u8]) -> Result[Mac, CryptoError]
def blake3_keyed(key: &[u8], data: &[u8]) -> Result[Bytes, CryptoError]
def blake3_derive_hasher(context: &str) -> Result[Digest, CryptoError]
def blake3_derive_key(context: &str, key_material: &[u8]) -> Result[Bytes, CryptoError]
def sha1(data: &[u8]) -> Result[Bytes, CryptoError]
def hmac_sha256(key: &[u8], data: &[u8]) -> Result[Bytes, CryptoError]
def hmac_sha512(key: &[u8], data: &[u8]) -> Result[Bytes, CryptoError]
def equal(a: &[u8], b: &[u8]) -> bool
enum AdvancedError
InvalidLength | AuthenticationFailed | InvalidPeer | Finished | RandomFailure | InvalidKey | WeakKey | SmallOrderKey | Allocation(AllocError)
def format[W](self, output: &mut Formatter[W]) -> Result[(), FormatError]
enum AeadAlgorithm
Aes256Gcm | ChaCha20Poly1305
def seal_in_place(algorithm: AeadAlgorithm, key: &[u8], nonce: &[u8], aad: &[u8], data: &mut Bytes) -> Result[(), AdvancedError]
def open_in_place(algorithm: AeadAlgorithm, key: &[u8], nonce: &[u8], aad: &[u8], data: &mut Bytes) -> Result[(), AdvancedError]
def ed25519_public_key_into(seed: &[u8], target: &mut Bytes) -> Result[(), AdvancedError]
def ed25519_public_key(seed: &[u8]) -> Result[Bytes, AdvancedError]
def ed25519_sign_into(seed: &[u8], message: &[u8], target: &mut Bytes) -> Result[(), AdvancedError]
def ed25519_sign(seed: &[u8], message: &[u8]) -> Result[Bytes, AdvancedError]
def ed25519_verify(public_key: &[u8], message: &[u8], signature: &[u8]) -> Result[bool, AdvancedError]
def hkdf_sha256_into(salt: &[u8], key_material: &[u8], info: &[u8], target: &mut Bytes) -> Result[(), AdvancedError]
def hkdf_sha256(salt: &[u8], key_material: &[u8], info: &[u8], size: NonNegative[i64]) -> Result[Bytes, AdvancedError]
class X25519Key
def public_key_into(self, target: &mut Bytes) -> Result[(), AdvancedError]
def public_key(self) -> Result[Bytes, AdvancedError]
def agree_into(self: &mut X25519Key, peer_public_key: &[u8], target: &mut Bytes) -> Result[(), AdvancedError]
def agree(self: &mut X25519Key, peer_public_key: &[u8]) -> Result[Bytes, AdvancedError]
def x25519() -> Result[X25519Key, AdvancedError]
enum RsaScheme
Pkcs1Sha256 | Pkcs1Sha384 | Pkcs1Sha512 | PssSha256 | PssSha384 | PssSha512
class RsaPublicKey
def signature_size(self) -> i64
def verify(self, scheme: RsaScheme, message: &[u8], signature: &[u8]) -> Result[bool, AdvancedError]
class RsaPrivateKey
def signature_size(self) -> i64
def public_key(self) -> Result[RsaPublicKey, AdvancedError]
def sign_into(self, scheme: RsaScheme, message: &[u8], target: &mut Bytes) -> Result[(), AdvancedError]
def sign(self, scheme: RsaScheme, message: &[u8]) -> Result[Bytes, AdvancedError]
class P256PublicKey
def verify(self, message: &[u8], signature: &[u8]) -> Result[bool, AdvancedError]
class P256PrivateKey
def public_key(self) -> Result[P256PublicKey, AdvancedError]
def sign_into(self, message: &[u8], target: &mut Bytes) -> Result[(), AdvancedError]
def sign(self, message: &[u8]) -> Result[Bytes, AdvancedError]
def rsa_public_key_from_spki(der: &[u8]) -> Result[RsaPublicKey, AdvancedError]
def rsa_public_key_from_components(n: &[u8], e: &[u8]) -> Result[RsaPublicKey, AdvancedError]
def rsa_private_key_from_pkcs8(der: &[u8]) -> Result[RsaPrivateKey, AdvancedError]
def p256_public_key_from_point(point: &[u8]) -> Result[P256PublicKey, AdvancedError]
def p256_public_key_from_coordinates(x: &[u8], y: &[u8]) -> Result[P256PublicKey, AdvancedError]
def p256_public_key_from_spki(der: &[u8]) -> Result[P256PublicKey, AdvancedError]
def p256_private_key_from_pkcs8(der: &[u8]) -> Result[P256PrivateKey, AdvancedError]
def ed25519_public_key_from_spki(der: &[u8]) -> Result[Bytes, AdvancedError]
def ed25519_seed_from_pkcs8(der: &[u8]) -> Result[Bytes, AdvancedError]
std.csv
Streaming CSV readers and writers over std.io with explicit dialects, size limits, and positioned errors. Errors: CsvError. Reference.
enum Fault
BareQuote | AfterQuote | UnterminatedQuote | FieldTooLong | RecordTooLong | TooManyFields
class Position
record: i64
field: i64
line: i64
offset: i64
enum CsvError
Invalid(Fault, Position) | Rejected(Fault, i64, i64) | Width(i64, i64, i64) | EmptyRecord(i64) | InvalidUtf8(i64, i64) | InvalidDialect | Stopped | Stream(StreamError) | Allocation(AllocError)
enum Width
Any | First | Exactly(i64)
enum Newline
Lf | CrLf
enum Quoting
Needed | Always
class Dialect
delimiter: u8
quote: u8
newline: Newline
quoting: Quoting
width: Width
skip_bom: bool
max_field: i64
max_record: i64
max_fields: i64
def rfc4180() -> Dialect
class Record
def len(self) -> i64
def number(self) -> i64
def line(self) -> i64
def data(self) -> &Bytes
def start(self, index: i64) -> i64
def stop(self, index: i64) -> i64
def bytes(self, index: i64) -> Result[Bytes, CsvError]
def append_to(self, index: i64, out: &mut Bytes) -> Result[(), CsvError]
def text(self, index: i64) -> Result[str, CsvError]
def texts(self) -> Result[list[str], CsvError]
def clear(self: &mut Record) -> ()
def record() -> Result[Record, CsvError]
class Reader[R]
inner: R
def records(self) -> i64
def line(self) -> i64
def offset(self) -> i64
def read(self: &mut Reader[R], record: &mut Record) -> Result[bool, CsvError]
def reader[R](inner: R, dialect: &Dialect, capacity: Positive[i64]) -> Result[Reader[R], CsvError]
class Writer[W]
inner: W
def records(self) -> i64
def pending(self) -> i64
def field(self: &mut Writer[W], source: &[u8]) -> Result[(), CsvError]
def text(self: &mut Writer[W], value: &str) -> Result[(), CsvError]
def discard(self: &mut Writer[W]) -> ()
def end_record(self: &mut Writer[W]) -> Result[(), CsvError]
def record(self: &mut Writer[W], record: &Record) -> Result[(), CsvError]
def texts(self: &mut Writer[W], fields: &list[str]) -> Result[(), CsvError]
def flush(self: &mut Writer[W]) -> Result[(), CsvError]
def writer[W](inner: W, dialect: &Dialect) -> Result[Writer[W], CsvError]
std.dataframe
Optional Polars analytical expression plans, typed nullable columns, schema inspection, grouping, and CSV/Parquet I/O. Errors: DataFrameError. Reference.
enum TimeUnit
Milliseconds | Microseconds | Nanoseconds
enum DType
I8 | I16 | I32 | U8 | U16 | U32 | U64 | F32 | I64 | F64 | Bool | String | Binary | Date | Datetime(TimeUnit, Option[str]) | List | Struct | Other(str)
class Field
name: str
dtype: DType
class SchemaNode
parent: i64
name: str
dtype: DType
class ListParts
values: Column
offsets: list[i64]
valid: list[bool]
class StructParts
fields: DataFrame
valid: list[bool]
enum JoinKind
Inner | Left | Full | Semi | Anti
enum AsOfStrategy
Backward | Forward | Nearest
class SortKey
def __init__(self, expression: Expr, descending: bool = False, nulls_last: bool = True) -> ()
enum DataFrameError
Schema(str) | Query(str) | Io(str) | NativePanic(str) | Null(str) | Bounds(str) | Allocation(AllocError) | Message(IoError)
class Column
def len(self) -> Result[i64, DataFrameError]
def to_frame(self) -> Result[DataFrame, DataFrameError]
def list_parts(self) -> Result[ListParts, DataFrameError]
def struct_parts(self) -> Result[StructParts, DataFrameError]
class DataFrame
def column(self, name: &str) -> Result[Column, DataFrameError]
def schema_tree(self) -> Result[list[SchemaNode], DataFrameError]
def height(self) -> Result[i64, DataFrameError]
def lazy(self) -> Result[Query, DataFrameError]
def write_parquet(self, path: &str) -> Result[(), DataFrameError]
def write_csv(self, path: &str, has_header: bool = True, separator: &str = ",", null_value: &str = "") -> Result[(), DataFrameError]
def get_date_days(self, name: &str, row: i64) -> Result[Option[i32], DataFrameError]
def get_datetime_ticks(self, name: &str, row: i64, unit: TimeUnit, utc: bool = False) -> Result[Option[i64], DataFrameError]
def values_date_days(self, name: &str) -> Result[list[i32], DataFrameError]
def values_date_days_nullable(self, name: &str) -> Result[list[Option[i32]], DataFrameError]
def values_datetime_ticks(self, name: &str, unit: TimeUnit, utc: bool = False) -> Result[list[i64], DataFrameError]
def values_datetime_ticks_nullable(self, name: &str, unit: TimeUnit, utc: bool = False) -> Result[list[Option[i64]], DataFrameError]
def get_i8(self, name: &str, row: i64) -> Result[Option[i8], DataFrameError]
def values_i8(self, name: &str) -> Result[list[i8], DataFrameError]
def values_i8_nullable(self, name: &str) -> Result[list[Option[i8]], DataFrameError]
def get_i16(self, name: &str, row: i64) -> Result[Option[i16], DataFrameError]
def values_i16(self, name: &str) -> Result[list[i16], DataFrameError]
def values_i16_nullable(self, name: &str) -> Result[list[Option[i16]], DataFrameError]
def get_i32(self, name: &str, row: i64) -> Result[Option[i32], DataFrameError]
def values_i32(self, name: &str) -> Result[list[i32], DataFrameError]
def values_i32_nullable(self, name: &str) -> Result[list[Option[i32]], DataFrameError]
def get_u8(self, name: &str, row: i64) -> Result[Option[u8], DataFrameError]
def values_u8(self, name: &str) -> Result[list[u8], DataFrameError]
def values_u8_nullable(self, name: &str) -> Result[list[Option[u8]], DataFrameError]
def get_u16(self, name: &str, row: i64) -> Result[Option[u16], DataFrameError]
def values_u16(self, name: &str) -> Result[list[u16], DataFrameError]
def values_u16_nullable(self, name: &str) -> Result[list[Option[u16]], DataFrameError]
def get_u32(self, name: &str, row: i64) -> Result[Option[u32], DataFrameError]
def values_u32(self, name: &str) -> Result[list[u32], DataFrameError]
def values_u32_nullable(self, name: &str) -> Result[list[Option[u32]], DataFrameError]
def get_u64(self, name: &str, row: i64) -> Result[Option[u64], DataFrameError]
def values_u64(self, name: &str) -> Result[list[u64], DataFrameError]
def values_u64_nullable(self, name: &str) -> Result[list[Option[u64]], DataFrameError]
def get_f32(self, name: &str, row: i64) -> Result[Option[f32], DataFrameError]
def values_f32(self, name: &str) -> Result[list[f32], DataFrameError]
def values_f32_nullable(self, name: &str) -> Result[list[Option[f32]], DataFrameError]
def get_i64(self, name: &str, row: i64) -> Result[Option[i64], DataFrameError]
def values_i64(self, name: &str) -> Result[list[i64], DataFrameError]
def get_f64(self, name: &str, row: i64) -> Result[Option[f64], DataFrameError]
def values_f64(self, name: &str) -> Result[list[f64], DataFrameError]
def schema(self) -> Result[list[Field], DataFrameError]
def get_bool(self, name: &str, row: i64) -> Result[Option[bool], DataFrameError]
def get_binary(self, name: &str, row: i64) -> Result[Option[Bytes], DataFrameError]
def values_binary(self, name: &str) -> Result[list[Bytes], DataFrameError]
def values_binary_nullable(self, name: &str) -> Result[list[Option[Bytes]], DataFrameError]
def get_str(self, name: &str, row: i64) -> Result[Option[str], DataFrameError]
def values_bool(self, name: &str) -> Result[list[bool], DataFrameError]
def values_str(self, name: &str) -> Result[list[str], DataFrameError]
def values_str_nullable(self, name: &str) -> Result[list[Option[str]], DataFrameError]
def values_i64_nullable(self, name: &str) -> Result[list[Option[i64]], DataFrameError]
def values_f64_nullable(self, name: &str) -> Result[list[Option[f64]], DataFrameError]
def values_bool_nullable(self, name: &str) -> Result[list[Option[bool]], DataFrameError]
class Expr
def list_len(self) -> Result[Expr, DataFrameError]
def list_get(self, index: i64) -> Result[Expr, DataFrameError]
def struct_field(self, name: &str) -> Result[Expr, DataFrameError]
def str_to_date(self, format: &str = "%Y-%m-%d", strict: bool = True) -> Result[Expr, DataFrameError]
def str_to_datetime(self, format: &str, unit: TimeUnit, utc: bool = False, strict: bool = True) -> Result[Expr, DataFrameError]
def dt_year(self) -> Result[Expr, DataFrameError]
def dt_month(self) -> Result[Expr, DataFrameError]
def dt_day(self) -> Result[Expr, DataFrameError]
def dt_hour(self) -> Result[Expr, DataFrameError]
def dt_minute(self) -> Result[Expr, DataFrameError]
def dt_second(self) -> Result[Expr, DataFrameError]
def dt_date(self) -> Result[Expr, DataFrameError]
def dt_format(self, format: &str) -> Result[Expr, DataFrameError]
def cast(self, dtype: DType, strict: bool = True) -> Result[Expr, DataFrameError]
def str_lower(self) -> Result[Expr, DataFrameError]
def str_upper(self) -> Result[Expr, DataFrameError]
def str_len(self) -> Result[Expr, DataFrameError]
def str_contains(self, pattern: &Expr) -> Result[Expr, DataFrameError]
def str_starts_with(self, prefix: &Expr) -> Result[Expr, DataFrameError]
def str_ends_with(self, suffix: &Expr) -> Result[Expr, DataFrameError]
def n_unique(self) -> Result[Expr, DataFrameError]
def over(self, partition_by: &list[Expr]) -> Result[Expr, DataFrameError]
def gt(self, other: &Expr) -> Result[Expr, DataFrameError]
def mul(self, other: &Expr) -> Result[Expr, DataFrameError]
def add(self, other: &Expr) -> Result[Expr, DataFrameError]
def sub(self, other: &Expr) -> Result[Expr, DataFrameError]
def div(self, other: &Expr) -> Result[Expr, DataFrameError]
def eq(self, other: &Expr) -> Result[Expr, DataFrameError]
def lt(self, other: &Expr) -> Result[Expr, DataFrameError]
def ge(self, other: &Expr) -> Result[Expr, DataFrameError]
def le(self, other: &Expr) -> Result[Expr, DataFrameError]
def and_(self, other: &Expr) -> Result[Expr, DataFrameError]
def or_(self, other: &Expr) -> Result[Expr, DataFrameError]
def fill_null(self, other: &Expr) -> Result[Expr, DataFrameError]
def sum(self) -> Result[Expr, DataFrameError]
def count(self) -> Result[Expr, DataFrameError]
def mean(self) -> Result[Expr, DataFrameError]
def min(self) -> Result[Expr, DataFrameError]
def max(self) -> Result[Expr, DataFrameError]
def is_null(self) -> Result[Expr, DataFrameError]
def alias(self, name: &str) -> Result[Expr, DataFrameError]
class Query
def schema_tree(self) -> Result[list[SchemaNode], DataFrameError]
def schema(self) -> Result[list[Field], DataFrameError]
def filter(self, predicate: &Expr) -> Result[Query, DataFrameError]
def with_columns(self, expressions: &list[Expr]) -> Result[Query, DataFrameError]
def select(self, expressions: &list[Expr]) -> Result[Query, DataFrameError]
def group_by(self, keys: &list[Expr], aggregates: &list[Expr]) -> Result[Query, DataFrameError]
def collect(self) -> Result[DataFrame, DataFrameError]
def collect_streaming(self) -> Result[DataFrame, DataFrameError]
def sink_parquet(self, path: &str) -> Result[(), DataFrameError]
def join(self, other: &Query, left_keys: &list[Expr], right_keys: &list[Expr], how: JoinKind = JoinKind.Inner, suffix: &str = "_right", nulls_equal: bool = False) -> Result[Query, DataFrameError]
def join_asof(self, other: &Query, left_on: &str, right_on: &str, strategy: AsOfStrategy = AsOfStrategy.Backward, tolerance: Option[i64] = Nothing, allow_exact_matches: bool = True, suffix: &str = "_right") -> Result[Query, DataFrameError]
def sort(self, keys: &list[SortKey]) -> Result[Query, DataFrameError]
def slice(self, offset: i64, length: i64) -> Result[Query, DataFrameError]
def limit(self, count: i64) -> Result[Query, DataFrameError]
def column_list(name: &str, values: &Column, offsets: &list[i64]) -> Result[Column, DataFrameError]
def column_list_nullable(name: &str, values: &Column, offsets: &list[i64], valid: &list[bool]) -> Result[Column, DataFrameError]
def column_struct(name: &str, fields: &DataFrame) -> Result[Column, DataFrameError]
def column_struct_nullable(name: &str, fields: &DataFrame, valid: &list[bool]) -> Result[Column, DataFrameError]
def frame(columns: &list[Column]) -> Result[DataFrame, DataFrameError]
def column_i8(name: &str, values: &list[i8]) -> Result[Column, DataFrameError]
def column_i8_nullable(name: &str, values: &list[Option[i8]]) -> Result[Column, DataFrameError]
def lit_i8(value: i8) -> Result[Expr, DataFrameError]
def column_i16(name: &str, values: &list[i16]) -> Result[Column, DataFrameError]
def column_i16_nullable(name: &str, values: &list[Option[i16]]) -> Result[Column, DataFrameError]
def lit_i16(value: i16) -> Result[Expr, DataFrameError]
def column_i32(name: &str, values: &list[i32]) -> Result[Column, DataFrameError]
def column_i32_nullable(name: &str, values: &list[Option[i32]]) -> Result[Column, DataFrameError]
def lit_i32(value: i32) -> Result[Expr, DataFrameError]
def column_u8(name: &str, values: &list[u8]) -> Result[Column, DataFrameError]
def column_u8_nullable(name: &str, values: &list[Option[u8]]) -> Result[Column, DataFrameError]
def lit_u8(value: u8) -> Result[Expr, DataFrameError]
def column_u16(name: &str, values: &list[u16]) -> Result[Column, DataFrameError]
def column_u16_nullable(name: &str, values: &list[Option[u16]]) -> Result[Column, DataFrameError]
def lit_u16(value: u16) -> Result[Expr, DataFrameError]
def column_u32(name: &str, values: &list[u32]) -> Result[Column, DataFrameError]
def column_u32_nullable(name: &str, values: &list[Option[u32]]) -> Result[Column, DataFrameError]
def lit_u32(value: u32) -> Result[Expr, DataFrameError]
def column_u64(name: &str, values: &list[u64]) -> Result[Column, DataFrameError]
def column_u64_nullable(name: &str, values: &list[Option[u64]]) -> Result[Column, DataFrameError]
def lit_u64(value: u64) -> Result[Expr, DataFrameError]
def column_f32(name: &str, values: &list[f32]) -> Result[Column, DataFrameError]
def column_f32_nullable(name: &str, values: &list[Option[f32]]) -> Result[Column, DataFrameError]
def lit_f32(value: f32) -> Result[Expr, DataFrameError]
def column_i64(name: &str, values: &list[i64]) -> Result[Column, DataFrameError]
def column_date(name: &str, days: &list[i32]) -> Result[Column, DataFrameError]
def column_date_nullable(name: &str, days: &list[Option[i32]]) -> Result[Column, DataFrameError]
def column_datetime(name: &str, ticks: &list[i64], unit: TimeUnit, utc: bool = False) -> Result[Column, DataFrameError]
def column_datetime_nullable(name: &str, ticks: &list[Option[i64]], unit: TimeUnit, utc: bool = False) -> Result[Column, DataFrameError]
def column_f64(name: &str, values: &list[f64]) -> Result[Column, DataFrameError]
def col(name: &str) -> Result[Expr, DataFrameError]
def lit(value: f64) -> Result[Expr, DataFrameError]
def lit_i64(value: i64) -> Result[Expr, DataFrameError]
def lit_str(value: &str) -> Result[Expr, DataFrameError]
def lit_bool(value: bool) -> Result[Expr, DataFrameError]
def lit_binary(value: &[u8]) -> Result[Expr, DataFrameError]
def when(predicate: &Expr, then_value: &Expr, otherwise: &Expr) -> Result[Expr, DataFrameError]
def null_f64() -> Result[Expr, DataFrameError]
def scan_parquet(path: &str) -> Result[Query, DataFrameError]
def scan_csv(path: &str, has_header: bool = True, separator: &str = ",", infer_rows: i64 = 100, null_value: Option[str] = Nothing, schema_overrides: Option[list[Field]] = Nothing) -> Result[Query, DataFrameError]
def column_bool(name: &str, values: &list[bool]) -> Result[Column, DataFrameError]
def column_binary(name: &str, values: &list[Bytes]) -> Result[Column, DataFrameError]
def column_binary_nullable(name: &str, values: &list[Option[Bytes]]) -> Result[Column, DataFrameError]
def column_str(name: &str, values: &list[str]) -> Result[Column, DataFrameError]
def column_i64_nullable(name: &str, values: &list[Option[i64]]) -> Result[Column, DataFrameError]
def column_f64_nullable(name: &str, values: &list[Option[f64]]) -> Result[Column, DataFrameError]
def column_bool_nullable(name: &str, values: &list[Option[bool]]) -> Result[Column, DataFrameError]
def column_str_nullable(name: &str, values: &list[Option[str]]) -> Result[Column, DataFrameError]
std.db.kv
Ordered byte-key maps, atomic transactions and bounded scans over a versioned SQLite store. Errors: KvError. Reference.
enum Mode
ReadOnly | ReadWrite | Create
enum KvError
Database(sqlite.DbError) | Allocation(AllocError) | InvalidName | ForeignDatabase | UnsupportedVersion(i64)
const format_version: i64 = 1
class Entry
key: Bytes
value: Bytes
class Scan
def next(self: &mut Scan) -> Result[Option[Entry], KvError]
class Transaction
def commit(self: &mut Transaction) -> Result[(), KvError]
def rollback(self: &mut Transaction) -> Result[(), KvError]
class Store
def begin(self: &mut Store, immediate: bool = True) -> Result[Transaction, KvError]
def set_busy_timeout(self: &mut Store, budget: &Duration) -> Result[(), KvError]
def close(self: &mut Store) -> Result[(), KvError]
def put(self: &mut Store, table: &str, key: &[u8], value: &[u8]) -> Result[(), KvError]
def get(self: &mut Store, table: &str, key: &[u8]) -> Result[Option[Bytes], KvError]
def contains(self: &mut Store, table: &str, key: &[u8]) -> Result[bool, KvError]
def remove(self: &mut Store, table: &str, key: &[u8]) -> Result[bool, KvError]
def clear(self: &mut Store, table: &str) -> Result[i64, KvError]
def len(self: &mut Store, table: &str) -> Result[i64, KvError]
def scan(self: &mut Store, table: &str, start: &[u8], stop: &[u8], max_rows: NonNegative[i64], after: bool = False, reverse: bool = False) -> Result[Scan, KvError]
def scan_all(self: &mut Store, table: &str, max_rows: NonNegative[i64], reverse: bool = False) -> Result[Scan, KvError]
def connect(path: &str, mode: Mode = Mode.Create, max_length: i64 = 1048576) -> Result[Store, KvError]
def memory(max_length: i64 = 1048576) -> Result[Store, KvError]
std.db.sqlite
SQLite databases through prepared statements with typed parameters and reads, owned transactions that roll back on drop, and an explicit value-size limit. Errors: DbError. Reference.
enum Mode
ReadOnly | ReadWrite | Create
enum Begin
Deferred | Immediate | Exclusive
enum ValueType
Integer | Real | Text | Blob | Null
class SqliteError
code: i64
message: str
def primary(self) -> i64
enum DbError
Sqlite(SqliteError) | Closed | InUse | EmptySql | TrailingSql | TransactionActive | TransactionFinished | InvalidPath | InvalidExtensionEntry | OutOfRange | NoRow | TypeMismatch(ValueType) | TooLarge | InvalidUtf8 | Allocation(AllocError)
const error: i64 = 1
const busy: i64 = 5
const locked: i64 = 6
const read_only: i64 = 8
const cant_open: i64 = 14
const constraint: i64 = 19
const mismatch: i64 = 20
const min_length_limit: i64 = 1024
const max_length_limit: i64 = 1000000000
class Connection
def execute(self: &mut Connection, sql: &str) -> Result[(), DbError]
def prepare(self: &mut Connection, sql: &str) -> Result[Statement, DbError]
def begin(self: &mut Connection, mode: Begin) -> Result[Transaction, DbError]
def changes(self) -> i64
def last_insert_rowid(self) -> i64
def set_busy_timeout(self: &mut Connection, budget: &Duration) -> Result[(), DbError]
def load_extension(self: &mut Connection, path: &str, entrypoint: &str = "") -> Result[(), DbError]
def close(self: &mut Connection) -> Result[(), DbError]
def connect(path: &str, mode: Mode, max_length: i64) -> Result[Connection, DbError]
def memory(max_length: i64) -> Result[Connection, DbError]
class Statement
def parameter_count(self) -> i64
def parameter_index(self, name: &str) -> Result[i64, DbError]
def bind_null(self: &mut Statement, index: i64) -> Result[(), DbError]
def bind_integer(self: &mut Statement, index: i64, value: i64) -> Result[(), DbError]
def bind_real(self: &mut Statement, index: i64, value: f64) -> Result[(), DbError]
def bind_text(self: &mut Statement, index: i64, value: &str) -> Result[(), DbError]
def bind_blob(self: &mut Statement, index: i64, value: &[u8]) -> Result[(), DbError]
def clear_bindings(self: &mut Statement) -> Result[(), DbError]
def step(self: &mut Statement) -> Result[bool, DbError]
def reset(self: &mut Statement) -> ()
def run(self: &mut Statement) -> Result[(), DbError]
def column_count(self) -> i64
def column_name(self, column: i64) -> Result[str, DbError]
def kind(self, column: i64) -> Result[ValueType, DbError]
def is_null(self, column: i64) -> Result[bool, DbError]
def integer(self, column: i64) -> Result[i64, DbError]
def real(self, column: i64) -> Result[f64, DbError]
def text(self, column: i64) -> Result[str, DbError]
def blob(self, column: i64) -> Result[Bytes, DbError]
def optional_integer(self, column: i64) -> Result[Option[i64], DbError]
def optional_real(self, column: i64) -> Result[Option[f64], DbError]
def optional_text(self, column: i64) -> Result[Option[str], DbError]
def optional_blob(self, column: i64) -> Result[Option[Bytes], DbError]
class Transaction
def commit(self: &mut Transaction) -> Result[(), DbError]
def rollback(self: &mut Transaction) -> Result[(), DbError]
std.deque
Owned generic ring buffers with constant-time removal at either end. Errors: AllocError. Reference.
class Deque[T]
def __init__(self) -> Result[(), AllocError]
def len(self) -> i64
def is_empty(self) -> bool
def append(self: &mut Deque[T], value: T) -> Result[(), AllocError]
def appendleft(self: &mut Deque[T], value: T) -> Result[(), AllocError]
def pop(self: &mut Deque[T]) -> Option[T]
def popleft(self: &mut Deque[T]) -> Option[T]
def clear(self: &mut Deque[T]) -> ()
std.diff
Line, word, character, byte, and list differences as edit operations, with Myers and patience algorithms and unified-diff output. Errors: AllocError. Reference.
enum Algorithm
Myers | Patience
enum Kind
Equal | Delete | Insert
class Op
kind: Kind
old_start: i64
old_end: i64
new_start: i64
new_end: i64
def old_len(self) -> Result[i64, OverflowError[i64]]
def new_len(self) -> Result[i64, OverflowError[i64]]
def is_equal(self) -> bool
def diff[T](old: &list[T], new: &list[T], algorithm: Algorithm = Algorithm.Myers, max_cost: Option[i64] = Nothing) -> Result[list[Op], AllocError]
def diff_bytes(old: &[u8], new: &[u8], algorithm: Algorithm = Algorithm.Myers, max_cost: Option[i64] = Nothing) -> Result[list[Op], AllocError]
class TextDiff
def ops(self) -> &list[Op]
def old_tokens(self) -> &list[str]
def new_tokens(self) -> &list[str]
def has_changes(self) -> bool
def unified(self, context: i64, old_name: &str, new_name: &str) -> Result[str, AllocError]
def lines(old: &str, new: &str, algorithm: Algorithm = Algorithm.Myers, max_cost: Option[i64] = Nothing) -> Result[TextDiff, AllocError]
def words(old: &str, new: &str, algorithm: Algorithm = Algorithm.Myers, max_cost: Option[i64] = Nothing) -> Result[TextDiff, AllocError]
def chars(old: &str, new: &str, algorithm: Algorithm = Algorithm.Myers, max_cost: Option[i64] = Nothing) -> Result[TextDiff, AllocError]
std.encoding
Hex, base32/base64, MIME base64 and validated custom alphabets, plus fixed-width integer and IEEE 754 float packing with explicit byte order, over std.bytes. Errors: EncodingError. Reference.
enum EncodingError
InvalidLength | InvalidByte(i64) | InvalidPadding(i64) | NonCanonical(i64) | InvalidAlphabet | InvalidPem(i64) | PemLabel | Allocation(AllocError)
enum Endian
Little | Big
class Encoding
def encode(self, source: &[u8]) -> Result[Bytes, EncodingError]
def decode(self, source: &[u8]) -> Result[Bytes, EncodingError]
def text(self, source: &[u8]) -> Result[str, EncodingError]
def from_text(self, text: &str) -> Result[Bytes, EncodingError]
def custom(symbols: &str, padding: Option[u8] = Some(61u8)) -> Result[Encoding, EncodingError]
def base32_encode(source: &[u8]) -> Result[Bytes, EncodingError]
def base32_decode(source: &[u8]) -> Result[Bytes, EncodingError]
def base32_text(source: &[u8]) -> Result[str, EncodingError]
def base32_from_text(text: &str) -> Result[Bytes, EncodingError]
def base32hex_encode(source: &[u8]) -> Result[Bytes, EncodingError]
def base32hex_decode(source: &[u8]) -> Result[Bytes, EncodingError]
def base32hex_text(source: &[u8]) -> Result[str, EncodingError]
def base32hex_from_text(text: &str) -> Result[Bytes, EncodingError]
def base64mime_encode(source: &[u8]) -> Result[Bytes, EncodingError]
def base64mime_decode(source: &[u8]) -> Result[Bytes, EncodingError]
def base64mime_text(source: &[u8]) -> Result[str, EncodingError]
def base64mime_from_text(text: &str) -> Result[Bytes, EncodingError]
def pem_encode(source: &[u8], label: &[u8]) -> Result[Bytes, EncodingError]
def pem_text(source: &[u8], label: &[u8]) -> Result[str, EncodingError]
def pem_decode(source: &[u8], label: &[u8]) -> Result[Bytes, EncodingError]
def pem_from_text(text: &str, label: &[u8]) -> Result[Bytes, EncodingError]
def hex_encode(source: &[u8]) -> Result[Bytes, EncodingError]
def hex_decode(source: &[u8]) -> Result[Bytes, EncodingError]
def hex_text(source: &[u8]) -> Result[str, EncodingError]
def hex_from_text(text: &str) -> Result[Bytes, EncodingError]
def base64_encode(source: &[u8]) -> Result[Bytes, EncodingError]
def base64_decode(source: &[u8]) -> Result[Bytes, EncodingError]
def base64_text(source: &[u8]) -> Result[str, EncodingError]
def base64_from_text(text: &str) -> Result[Bytes, EncodingError]
def base64url_encode(source: &[u8]) -> Result[Bytes, EncodingError]
def base64url_decode(source: &[u8]) -> Result[Bytes, EncodingError]
def base64url_text(source: &[u8]) -> Result[str, EncodingError]
def base64url_from_text(text: &str) -> Result[Bytes, EncodingError]
def pack_u16(target: &mut Bytes, offset: i64, value: u16, order: Endian) -> ()
def pack_u32(target: &mut Bytes, offset: i64, value: u32, order: Endian) -> ()
def pack_u64(target: &mut Bytes, offset: i64, value: u64, order: Endian) -> ()
def pack_i16(target: &mut Bytes, offset: i64, value: i16, order: Endian) -> ()
def pack_i32(target: &mut Bytes, offset: i64, value: i32, order: Endian) -> ()
def pack_i64(target: &mut Bytes, offset: i64, value: i64, order: Endian) -> ()
def append_u16(target: &mut Bytes, value: u16, order: Endian) -> Result[(), EncodingError]
def append_u32(target: &mut Bytes, value: u32, order: Endian) -> Result[(), EncodingError]
def append_u64(target: &mut Bytes, value: u64, order: Endian) -> Result[(), EncodingError]
def append_i16(target: &mut Bytes, value: i16, order: Endian) -> Result[(), EncodingError]
def append_i32(target: &mut Bytes, value: i32, order: Endian) -> Result[(), EncodingError]
def append_i64(target: &mut Bytes, value: i64, order: Endian) -> Result[(), EncodingError]
def unpack_u16(source: &[u8], offset: i64, order: Endian) -> u16
def unpack_u32(source: &[u8], offset: i64, order: Endian) -> u32
def unpack_u64(source: &[u8], offset: i64, order: Endian) -> u64
def unpack_i16(source: &[u8], offset: i64, order: Endian) -> i16
def unpack_i32(source: &[u8], offset: i64, order: Endian) -> i32
def unpack_i64(source: &[u8], offset: i64, order: Endian) -> i64
def pack_f32(target: &mut Bytes, offset: i64, value: f32, order: Endian) -> ()
def pack_f64(target: &mut Bytes, offset: i64, value: f64, order: Endian) -> ()
def append_f32(target: &mut Bytes, value: f32, order: Endian) -> Result[(), EncodingError]
def append_f64(target: &mut Bytes, value: f64, order: Endian) -> Result[(), EncodingError]
def unpack_f32(source: &[u8], offset: i64, order: Endian) -> f32
def unpack_f64(source: &[u8], offset: i64, order: Endian) -> f64
std.env
Environment variables as text or bytes, a detached snapshot, and the working directory. Errors: IoError. Reference.
class Variable
name: list[u8]
value: list[u8]
def valid_name(name: &list[u8]) -> bool
def get_bytes(name: &list[u8]) -> Result[Option[list[u8]], IoError]
def get(name: &str) -> Result[Option[str], IoError]
def vars() -> Result[list[Variable], IoError]
def current_dir() -> Result[Path, IoError]
def set_current_dir(path: &Path) -> Result[(), IoError]
std.error
Explicit error mapping, operation/path context, descriptions, and standard-error reports. Errors: Mapper-selected error, Context[E], AllocError. Reference.
def map_error[T, E, G, F: OnceCallable[[E], G]](result: Result[T, E], mapper: F) -> Result[T, G]
enum Subject
Absent | At(Path) | Unavailable(AllocError)
class Context[E]
operation: str
subject: Subject
cause: E
def format[W](self, output: &mut Formatter[W]) -> Result[(), FormatError]
def describe(self) -> Result[str, AllocError]
def describe_with(self, cause: &str) -> Result[str, AllocError]
def report(self) -> ()
def report_with(self, description: Result[str, AllocError]) -> ()
def context[T, E](result: Result[T, E], operation: str) -> Result[T, Context[E]]
def path_context[T, E](result: Result[T, E], operation: str, path: &Path) -> Result[T, Context[E]]
def native_code(error: IoError) -> Option[i32]
std.fmt
Writer-generic value/report formatting, bounded output and fallible strings. Errors: FormatError. Reference.
enum FormatError
InvalidSpec | LimitExceeded | RecursionLimit | Allocation(AllocError) | Output(StreamError) | OutputClosed
class Spec
fill: str
align: str
sign: str
coerce_zero: bool
alternate: bool
zero: bool
width: i64
grouping: str
precision: i64
kind: str
def spec(fill: str = " ", align: str = "", sign: str = "", coerce_zero: bool = False, alternate: bool = False, zero: bool = False, width: i64 = 0, grouping: str = "", precision: i64 = -1, kind: str = "") -> Spec
def spec_fill(fill: &str, align: &str) -> Result[str, FormatError]
def spec_align(align: &str) -> Result[str, FormatError]
def spec_count(value: i64) -> Result[i64, FormatError]
class Formatter[W]
writer: W
def completed(self) -> i64
def has_failed(self) -> bool
def text(self: &mut Formatter[W], value: &str) -> Result[(), FormatError]
def substr(self: &mut Formatter[W], value: &substr) -> Result[(), FormatError]
def boolean(self: &mut Formatter[W], value: bool) -> Result[(), FormatError]
def floating(self: &mut Formatter[W], value: f64, width: NonNegative[i64]) -> Result[(), FormatError]
def unsigned(self: &mut Formatter[W], value: u64, base: i64, width: NonNegative[i64] = 0) -> Result[(), FormatError]
def integer(self: &mut Formatter[W], value: i64, width: NonNegative[i64] = 0) -> Result[(), FormatError]
def debug_text(self: &mut Formatter[W], value: &str) -> Result[(), FormatError]
def debug_substr(self: &mut Formatter[W], value: &substr) -> Result[(), FormatError]
def ascii_text(self: &mut Formatter[W], value: &str) -> Result[(), FormatError]
def char_spec(self: &mut Formatter[W], code: i64, spec: &Spec) -> Result[(), FormatError]
def text_spec(self: &mut Formatter[W], value: &str, spec: &Spec) -> Result[(), FormatError]
def substr_spec(self: &mut Formatter[W], value: &substr, spec: &Spec) -> Result[(), FormatError]
def signed_spec(self: &mut Formatter[W], value: i64, spec: &Spec) -> Result[(), FormatError]
def unsigned_spec(self: &mut Formatter[W], value: u64, spec: &Spec) -> Result[(), FormatError]
def float_spec(self: &mut Formatter[W], value: f64, spec: &Spec) -> Result[(), FormatError]
def float32_spec(self: &mut Formatter[W], value: f32, spec: &Spec) -> Result[(), FormatError]
def enter(self: &mut Formatter[W]) -> Result[(), FormatError]
def leave(self: &mut Formatter[W]) -> ()
def formatter[W: Writer](writer: W, scratch: Bytes, max_bytes: NonNegative[i64], max_depth: NonNegative[i64]) -> Result[Formatter[W], FormatError]
def stderr_formatter(max_bytes: NonNegative[i64], max_depth: NonNegative[i64]) -> Result[Formatter[Stderr], FormatError]
def string_formatter(max_bytes: NonNegative[i64], max_depth: NonNegative[i64]) -> Result[Formatter[Cursor], FormatError]
def finish(output: &Formatter[Cursor]) -> Result[str, FormatError]
def display[T, W](value: &T, output: &mut Formatter[W]) -> Result[(), FormatError]
def debug[T, W](value: &T, output: &mut Formatter[W]) -> Result[(), FormatError]
def to_string[T](value: &T, max_bytes: NonNegative[i64], max_depth: NonNegative[i64]) -> Result[str, FormatError]
def to_debug[T](value: &T, max_bytes: NonNegative[i64], max_depth: NonNegative[i64]) -> Result[str, FormatError]
def represent[T, W](value: &T, output: &mut Formatter[W]) -> Result[(), FormatError]
def repr_string[T](value: &T) -> Result[str, FormatError]
def represent_view[T, W](value: &[T], output: &mut Formatter[W]) -> Result[(), FormatError]
def repr_view_string[T](value: &[T]) -> Result[str, FormatError]
def repr_text[W](spelled: Result[str, AllocError | FormatError], output: &mut Formatter[W]) -> Result[(), FormatError]
def repr_result(spelled: Result[str, AllocError | FormatError]) -> Result[str, FormatError]
def allocation_only(built: Result[str, FormatError]) -> Result[str, AllocError]
def allocation_or_format(built: Result[str, FormatError]) -> Result[str, AllocError | FormatError]
def debug_string(value: &str) -> Result[str, FormatError]
def repr_substr_string(value: &substr) -> Result[str, FormatError]
def ascii_string(value: &str) -> Result[str, FormatError]
def list_with[T, W, F: Callable[[&T, &mut Formatter[W]], Result[(), FormatError]]](values: &list[T], output: &mut Formatter[W], render: F) -> Result[(), FormatError]
def pair_with[A, B, W, F: Callable[[&A, &mut Formatter[W]], Result[(), FormatError]], G: Callable[[&B, &mut Formatter[W]], Result[(), FormatError]]](value: &(A, B), output: &mut Formatter[W], first: F, second: G) -> Result[(), FormatError]
protocol Formattable
def format[W](self, output: &mut Formatter[W]) -> Result[(), FormatError]
std.fs
Directories, metadata/walking, symbolic links and canonical paths, rooted mutation, owned temporary resources, bounded file/tree copy, metadata preservation, delete/glob. Errors: IoError, AllocError, WalkError, FsError, TreeCopyError. Reference.
enum FsError
Io(IoError) | Stream(StreamError) | Allocation(AllocError) | Entropy(SecretsError) | InvalidInput | Limit | Cancelled | Replaced | Collisions | UnsupportedType | Overlap | InvalidLimits
class Root
def open_dir(self, name: &Path) -> Result[Root, IoError]
def read_file(self, name: &Path) -> Result[FileStream, IoError]
def create_dir(self, name: &Path, unix_mode: u32 = 0o700u32) -> Result[(), IoError]
def create_file(self, name: &Path, unix_mode: u32 = 0o600u32) -> Result[FileStream, IoError]
def set_unix_mode(self, unix_mode: u32) -> Result[(), IoError]
def remove_file(self, name: &Path) -> Result[(), IoError]
def remove_dir(self, name: &Path) -> Result[(), IoError]
def move(self, source: &Path, destination: &Path) -> Result[(), IoError]
def names(self) -> Result[Names, IoError]
def set_file_unix_mode(stream: &FileStream, unix_mode: u32) -> Result[(), IoError]
def root(path: &Path) -> Result[Root, IoError]
class Names
def next(self: &mut Names) -> Result[Option[Path], IoError]
def close(self: &mut Names) -> Result[(), IoError]
class TempFile
stream: FileStream
def publish(self: &mut TempFile, name: &Path) -> Result[(), IoError]
def close(self: &mut TempFile) -> Result[(), StreamError]
def remove_published(self: &mut TempFile, name: &Path) -> Result[(), FsError]
def temporary_file(parent: &Root) -> Result[TempFile, IoError]
class TempDir
def name(self) -> &Path
def root(self) -> &Root
def cleanup(self: &mut TempDir) -> Result[(), FsError]
def keep(self: &mut TempDir) -> ()
def cleanup_tree(self: &mut TempDir, max_entries: Positive[i64], max_depth: Positive[i64], cancel: &CancellationToken) -> Result[(), FsError]
def temporary_dir_named(parent: &Root, name: &Path) -> Result[TempDir, FsError]
def temporary_dir(parent: &Root, prefix: &Path, attempts: Positive[i64]) -> Result[TempDir, FsError]
enum CopyMetadata
ContentsOnly | PermissionsAndTimes
def copy_from[R: Reader](source: &mut R, parent: &Root, destination: &Path, max_bytes: NonNegative[i64], cancel: &CancellationToken) -> Result[i64, FsError]
def copy_file(parent: &Root, source: &Path, destination: &Path, max_bytes: NonNegative[i64], cancel: &CancellationToken, preserve: CopyMetadata = CopyMetadata.ContentsOnly) -> Result[i64, FsError]
class CopyLimits
max_entries: i64
max_depth: i64
max_bytes: i64
max_open_handles: i64
def copy_limits() -> CopyLimits
enum TreeCopyError
Setup(FsError) | Partial(FsError) | Entry(Path, i64, FsError)
def copy_tree(source_parent: &Root, source: &Path, destination_parent: &Root, destination: &Path, limits: &CopyLimits, cancel: &CancellationToken, preserve: CopyMetadata = CopyMetadata.ContentsOnly) -> Result[i64, TreeCopyError]
def remove_tree(parent: &Root, name: &Path, max_entries: Positive[i64], max_depth: Positive[i64], cancel: &CancellationToken) -> Result[(), FsError]
def glob(parent: &Root, pattern: &Path, max_entries: NonNegative[i64], max_matches: NonNegative[i64], max_steps: NonNegative[i64]) -> Result[list[Path], FsError]
enum FileType
File | Directory | Symlink | Other
class Permissions
readonly: bool
unix_mode: u32
class Metadata
def file_type(self) -> FileType
def is_file(self) -> bool
def is_dir(self) -> bool
def is_symlink(self) -> bool
def size(self) -> u64
def permissions(self) -> Permissions
def modified(self) -> Option[Timestamp]
def accessed(self) -> Option[Timestamp]
def created(self) -> Option[Timestamp]
def unix_changed(self) -> Timestamp
def unix_uid(self) -> u32
def unix_gid(self) -> u32
def unix_device(self) -> u64
def unix_inode(self) -> u64
class DirEntry
def file_name(self) -> Result[Path, IoError]
def path(self) -> Result[Path, IoError]
def file_type(self) -> Result[FileType, IoError]
def metadata(self) -> Result[Metadata, IoError]
class ReadDir
def next(self: &mut ReadDir) -> Result[Option[DirEntry], IoError]
def close(self: &mut ReadDir) -> Result[(), IoError]
def closed(self) -> bool
def read_dir(path: &Path) -> Result[ReadDir, IoError]
def metadata(path: &Path) -> Result[Metadata, IoError]
def symlink_metadata(path: &Path) -> Result[Metadata, IoError]
def file_metadata(stream: &FileStream) -> Result[Metadata, IoError]
def read_link(path: &Path) -> Result[Path, IoError]
def canonicalize(path: &Path) -> Result[Path, IoError]
def symlink(target: &Path, link: &Path) -> Result[(), IoError]
def entries(owner: ReadDir) -> Generator[Result[DirEntry, IoError]]
enum WalkError
Io(Path, str, IoError) | Allocation(AllocError)
class WalkDir
path: Path
directories: list[Path]
files: list[DirEntry]
class Walk
current: WalkDir
def close(self: &mut Walk) -> ()
def next(self: &mut Walk) -> Result[bool, WalkError]
def walk(root: &Path, top_down: bool) -> Result[Walk, AllocError]
std.fs.dirs
The home directory and the XDG configuration, cache, data, and state directories, from the environment. Errors: IoError. Reference.
def home_dir() -> Result[Option[Path], IoError]
def config_dir() -> Result[Option[Path], IoError]
def cache_dir() -> Result[Option[Path], IoError]
def data_dir() -> Result[Option[Path], IoError]
def state_dir() -> Result[Option[Path], IoError]
std.fs.lock
Owned Linux shared/exclusive advisory locks, nonblocking and cancellable deadline acquisition. Errors: LockError. Reference.
enum Mode
Shared | Exclusive
enum LockError
WouldBlock | TimedOut | Cancelled | InvalidPath | Io(IoError) | Allocation(AllocError)
class Lock
def close(self: &mut Lock) -> Result[(), LockError]
def is_closed(self) -> bool
def acquire_now(path: &Path, mode: Mode) -> Result[Lock, LockError]
def acquire_before(path: &Path, mode: Mode, deadline: &Deadline, cancel: &CancellationToken) -> Result[Lock, LockError]
std.fs.watch
Bounded Linux filesystem change batches, rename cookies, overflow and readiness waits. Errors: WatchError, WaitError. Reference.
enum WatchError
OutOfRange | InvalidData | Io(IoError) | Wait(WaitError) | Allocation(AllocError)
class Event
watch: i32
cookie: u32
name: Path
def created(self) -> bool
def deleted(self) -> bool
def modified(self) -> bool
def metadata_changed(self) -> bool
def closed_after_write(self) -> bool
def moved_from(self) -> bool
def moved_to(self) -> bool
def deleted_self(self) -> bool
def moved_self(self) -> bool
def unmounted(self) -> bool
def overflowed(self) -> bool
def ignored(self) -> bool
def is_directory(self) -> bool
class Watcher
def add(self: &mut Watcher, path: &Path) -> Result[i32, WatchError]
def remove(self: &mut Watcher, watch: i32) -> Result[(), WatchError]
def poll(self: &mut Watcher) -> Result[i64, WatchError]
def wait(self: &mut Watcher, deadline: &Deadline, cancel: &CancellationToken) -> Result[i64, WatchError]
def count(self) -> i64
def event(self, index: i64) -> Result[Event, WatchError]
def register(self, selector: &mut Selector, key: u64) -> Result[(), WaitError]
def unregister(self, selector: &mut Selector) -> Result[(), WaitError]
def close(self: &mut Watcher) -> Result[(), WatchError]
def watcher(batch_bytes: i64 = 4096) -> Result[Watcher, WatchError]
std.graph
Directed and undirected graphs with node and edge data, stable handles, traversal, shortest paths, topological order, components and spanning trees. Errors: AllocError, GraphError. Reference.
type NodeId = i64
type EdgeId = i64
enum GraphError
Cycle(list[NodeId]) | NegativeCycle(list[NodeId]) | InvalidWeight(EdgeId) | InvalidEstimate(NodeId) | Allocation(AllocError)
class Walk
class Paths
def distance(self, node: NodeId) -> Option[f64]
def edge_to(self, node: NodeId) -> Option[EdgeId]
def path(self, node: NodeId) -> Result[list[NodeId], AllocError]
class DiGraph[N, E]
def __init__(self) -> Result[(), AllocError]
def node_count(self) -> i64
def edge_count(self) -> i64
def node_bound(self) -> i64
def edge_bound(self) -> i64
def reserve(self: &mut DiGraph[N, E], nodes: i64, edges: i64) -> Result[(), AllocError]
def add_node(self: &mut DiGraph[N, E], data: N) -> Result[NodeId, AllocError]
def add_edge(self: &mut DiGraph[N, E], source: NodeId, target: NodeId, data: E) -> Result[EdgeId, AllocError]
def remove_node(self: &mut DiGraph[N, E], node: NodeId) -> Option[N]
def remove_edge(self: &mut DiGraph[N, E], edge: EdgeId) -> Option[E]
def contains_node(self, node: NodeId) -> bool
def contains_edge(self, edge: EdgeId) -> bool
def node(self, node: NodeId) -> &N
def node_mut(self: &mut DiGraph[N, E], node: NodeId) -> &mut N
def edge(self, edge: EdgeId) -> &E
def edge_mut(self: &mut DiGraph[N, E], edge: EdgeId) -> &mut E
def get_node_ref(self, node: NodeId) -> Option[&N]
def get_node_mut(self: &mut DiGraph[N, E], node: NodeId) -> Option[&mut N]
def get_edge_ref(self, edge: EdgeId) -> Option[&E]
def get_edge_mut(self: &mut DiGraph[N, E], edge: EdgeId) -> Option[&mut E]
def source(self, edge: EdgeId) -> NodeId
def target(self, edge: EdgeId) -> NodeId
def find_edge(self, source: NodeId, target: NodeId) -> Option[EdgeId]
def nodes(self) -> Result[list[NodeId], AllocError]
def edges(self) -> Result[list[EdgeId], AllocError]
def successors(self, node: NodeId) -> Result[list[NodeId], AllocError]
def predecessors(self, node: NodeId) -> Result[list[NodeId], AllocError]
def outgoing(self, node: NodeId) -> Walk
def incoming(self, node: NodeId) -> Walk
def next_edge(self, walk: &mut Walk) -> Option[EdgeId]
def bfs(self, start: NodeId) -> Result[list[NodeId], AllocError]
def dfs(self, start: NodeId) -> Result[list[NodeId], AllocError]
def toposort(self) -> Result[list[NodeId], GraphError]
def strongly_connected_components(self) -> Result[list[list[NodeId]], AllocError]
def dijkstra[F: Callable[[&E], f64]](self, start: NodeId, weight: &F) -> Result[Paths, GraphError]
def astar[F: Callable[[&E], f64], H: Callable[[NodeId], f64]](self, start: NodeId, goal: NodeId, weight: &F, estimate: &H) -> Result[Option[(f64, list[NodeId])], GraphError]
def bellman_ford[F: Callable[[&E], f64]](self, start: NodeId, weight: &F) -> Result[Paths, GraphError]
class UnGraph[N, E]
def __init__(self) -> Result[(), AllocError]
def node_count(self) -> i64
def edge_count(self) -> i64
def node_bound(self) -> i64
def edge_bound(self) -> i64
def reserve(self: &mut UnGraph[N, E], nodes: i64, edges: i64) -> Result[(), AllocError]
def add_node(self: &mut UnGraph[N, E], data: N) -> Result[NodeId, AllocError]
def add_edge(self: &mut UnGraph[N, E], a: NodeId, b: NodeId, data: E) -> Result[EdgeId, AllocError]
def remove_node(self: &mut UnGraph[N, E], node: NodeId) -> Option[N]
def remove_edge(self: &mut UnGraph[N, E], edge: EdgeId) -> Option[E]
def contains_node(self, node: NodeId) -> bool
def contains_edge(self, edge: EdgeId) -> bool
def node(self, node: NodeId) -> &N
def node_mut(self: &mut UnGraph[N, E], node: NodeId) -> &mut N
def edge(self, edge: EdgeId) -> &E
def edge_mut(self: &mut UnGraph[N, E], edge: EdgeId) -> &mut E
def get_node_ref(self, node: NodeId) -> Option[&N]
def get_node_mut(self: &mut UnGraph[N, E], node: NodeId) -> Option[&mut N]
def get_edge_ref(self, edge: EdgeId) -> Option[&E]
def get_edge_mut(self: &mut UnGraph[N, E], edge: EdgeId) -> Option[&mut E]
def endpoints(self, edge: EdgeId) -> (NodeId, NodeId)
def opposite(self, edge: EdgeId, node: NodeId) -> NodeId
def find_edge(self, a: NodeId, b: NodeId) -> Option[EdgeId]
def nodes(self) -> Result[list[NodeId], AllocError]
def edges(self) -> Result[list[EdgeId], AllocError]
def neighbors(self, node: NodeId) -> Result[list[NodeId], AllocError]
def incident(self, node: NodeId) -> Walk
def next_edge(self, walk: &mut Walk) -> Option[EdgeId]
def bfs(self, start: NodeId) -> Result[list[NodeId], AllocError]
def dfs(self, start: NodeId) -> Result[list[NodeId], AllocError]
def connected_components(self) -> Result[list[list[NodeId]], AllocError]
def dijkstra[F: Callable[[&E], f64]](self, start: NodeId, weight: &F) -> Result[Paths, GraphError]
def astar[F: Callable[[&E], f64], H: Callable[[NodeId], f64]](self, start: NodeId, goal: NodeId, weight: &F, estimate: &H) -> Result[Option[(f64, list[NodeId])], GraphError]
def bellman_ford[F: Callable[[&E], f64]](self, start: NodeId, weight: &F) -> Result[Paths, GraphError]
def minimum_spanning_tree[F: Callable[[&E], f64]](self, weight: &F) -> Result[list[EdgeId], GraphError]
std.http
Bounded HTTP/1.1 message heads and streamed Content-Length, chunked, and close-delimited bodies over std.io streams. Errors: HttpError. Reference.
enum HttpError
Stream(StreamError) | Allocation(AllocError) | InvalidLimit | Truncated | LineTooLong | HeadTooLarge | TooManyHeaders | BareLineFeed | InvalidRequestLine | InvalidStatusLine | UnsupportedVersion | InvalidHeader | InvalidContentLength | DuplicateFraming | ConflictingFraming | UnsupportedTransferEncoding | InvalidChunk | BodyTooLarge | LengthMismatch | Unfinished | Broken | InvalidTarget | MissingHost | DuplicateHost | InvalidHost
class Limits
max_line: i64
max_head: i64
max_headers: NonNegative[i64]
max_body: NonNegative[i64]
def limits() -> Limits
class Header
name: str
value: str
class RequestHead
method: str
target: str
minor: i64
headers: list[Header]
def add(self: &mut RequestHead, name: str, value: str) -> Result[(), HttpError]
class ResponseHead
minor: i64
status: i64
reason: str
headers: list[Header]
def add(self: &mut ResponseHead, name: str, value: str) -> Result[(), HttpError]
enum Framing
Empty | Length(i64) | Chunked | UntilClose
def request(method: str, target: str) -> Result[RequestHead, HttpError]
def response(status: i64, reason: str) -> Result[ResponseHead, HttpError]
def find(headers: &list[Header], name: &str) -> Option[i64]
def same_name(left: &str, right: &str) -> bool
def parse_field(line: &[u8]) -> Result[Header, HttpError]
def parse_request_line(line: &[u8]) -> Result[RequestHead, HttpError]
def parse_status_line(line: &[u8]) -> Result[ResponseHead, HttpError]
def read_request[R: Reader](conn: &mut BufReader[R], limits: &Limits) -> Result[Option[RequestHead], HttpError]
def read_response[R: Reader](conn: &mut BufReader[R], limits: &Limits) -> Result[Option[ResponseHead], HttpError]
def request_framing(head: &RequestHead) -> Result[Framing, HttpError]
def response_framing(head: &ResponseHead, method: &str) -> Result[Framing, HttpError]
def request_keeps_alive(head: &RequestHead) -> bool
class RequestTarget
authority: str
path: str
query: str
def request_target(head: &RequestHead) -> Result[RequestTarget, HttpError]
def response_keeps_alive(head: &ResponseHead, framing: &Framing) -> bool
def parse_chunk_size(line: &[u8]) -> Result[i64, HttpError]
class BodyReader[R]
trailers: list[Header]
def total(self) -> i64
def is_done(self) -> bool
def take_failure(self: &mut BodyReader[R]) -> Option[HttpError]
def read_body(self: &mut BodyReader[R], buffer: &mut Bytes, start: i64, stop: i64) -> Result[i64, HttpError]
def read(self: &mut BodyReader[R], buffer: &mut Bytes, start: i64, stop: i64) -> Result[i64, StreamError]
def body_reader[R: Reader](conn: BufReader[R], framing: &Framing, limits: &Limits) -> Result[BodyReader[R], HttpError]
def read_to_end[R: Reader](body: &mut BodyReader[R], out: &mut Bytes) -> Result[i64, HttpError]
def discard[R: Reader](body: &mut BodyReader[R]) -> Result[i64, HttpError]
def finish[R](body: &mut BodyReader[R]) -> Result[BufReader[R], HttpError]
def release[R](body: &mut BodyReader[R]) -> Option[BufReader[R]]
def write_request[W: Writer](writer: &mut W, head: &RequestHead) -> Result[(), HttpError]
def encode_request(head: &RequestHead) -> Result[Bytes, HttpError]
def write_response[W: Writer](writer: &mut W, head: &ResponseHead) -> Result[(), HttpError]
def encode_response(head: &ResponseHead) -> Result[Bytes, HttpError]
def write_chunk[W: Writer](writer: &mut W, source: &[u8]) -> Result[(), HttpError]
def encode_chunk_size(length: NonNegative[i64]) -> Result[Bytes, HttpError]
def write_last_chunk[W: Writer](writer: &mut W, trailers: &list[Header]) -> Result[(), HttpError]
def encode_last_chunk(trailers: &list[Header]) -> Result[Bytes, HttpError]
def write_body[W: Writer](writer: &mut W, framing: &Framing, source: &[u8]) -> Result[(), HttpError]
std.http.decoder
Resumable bounded HTTP/1.x request framing over caller-supplied bytes, without I/O. Errors: HttpError. Reference.
const feed_budget: i64 = 4096
enum Event
Head(RequestHead, RequestTarget, Framing) | Body(Bytes) | Trailers(list[Header]) | End
class Decoder
def is_started(self) -> bool
def has_head(self) -> bool
def is_done(self) -> bool
def total(self) -> i64
def reset(self: &mut Decoder) -> Result[(), HttpError]
def finish(self: &mut Decoder) -> Result[(), HttpError]
def feed(self: &mut Decoder, source: &[u8]) -> Result[(i64, Option[Event]), HttpError]
def decoder(limits: &Limits) -> Result[Decoder, HttpError]
std.http.body
Bounded request bodies for handlers: URL-encoded forms, JSON, and multipart uploads into unnamed temporary files. Errors: BodyError. Reference.
enum BodyError
UnsupportedMediaType | TooLarge | TooManyFields | Malformed | InvalidUtf8 | Http(HttpError) | Json(JsonError) | Storage(StreamError) | Allocation(AllocError)
def status(error: &BodyError) -> i64
def reject[T: Transport](exchange: &mut Exchange[T], error: &BodyError) -> Result[(), HttpError]
class FormLimits
max_bytes: NonNegative[i64]
max_fields: NonNegative[i64]
def form_limits() -> FormLimits
def parse_form(source: &[u8], limits: &FormLimits) -> Result[list[Pair], BodyError]
def parse_form_text(text: &str, limits: &FormLimits) -> Result[list[Pair], BodyError]
def find(pairs: &list[Pair], key: &str) -> Option[i64]
def read_form[T: Transport](exchange: &mut Exchange[T], limits: &FormLimits) -> Result[list[Pair], BodyError]
def read_json[T: Transport](exchange: &mut Exchange[T], limits: &std.json.Limits) -> Result[Json, BodyError]
def send_json[T: Transport](exchange: &mut Exchange[T], status: i64, value: &Json, limits: &std.json.Limits) -> Result[(), BodyError]
class MultipartLimits
max_parts: NonNegative[i64]
max_head: NonNegative[i64]
max_field: NonNegative[i64]
max_file: NonNegative[i64]
max_body: NonNegative[i64]
def multipart_limits() -> MultipartLimits
enum Content
Text(str) | File(TempFile)
class Part
name: str
filename: Option[str]
content_type: str
size: i64
content: Content
def read_multipart[T: Transport](exchange: &mut Exchange[T], limits: &MultipartLimits, storage: &Root) -> Result[list[Part], BodyError]
std.http.client
HTTP/1.1 client: pooled connections, deadlines and cancellation, bounded redirects, an explicit proxy, and replay of idempotent requests only. Errors: ClientError. Reference.
enum ClientError
Net(NetError) | Http(HttpError) | Url(UrlError) | Allocation(AllocError) | InvalidConfig | InvalidUrl | UnsupportedScheme | InvalidRequest | InvalidRedirect | TooManyRedirects | UnexpectedInterim | Tls(TlsFailure) | TunnelRefused(i64) | Body(StreamError) | WrongBodyLength
enum TlsFailure
UntrustedCertificate | ExpiredCertificate | WrongHost | InvalidCertificate | RevokedCertificate | Rejected | Protocol | InvalidName | InvalidConfiguration | NoTrustRoots | CertificateRequired | Incompatible
class Tunnel
host: str
port: u16
authorization: Option[str]
limits: Limits
class Endpoint
scheme: str
host: str
port: u16
tunnel: Option[Tunnel]
protocol Transport
def read_before(self: &mut Transport, buffer: &mut Bytes, start: i64, stop: i64, deadline: &Deadline, cancel: &CancellationToken) -> Result[i64, ClientError]
def write_before(self: &mut Transport, source: &[u8], deadline: &Deadline, cancel: &CancellationToken) -> Result[i64, ClientError]
protocol Connector[T]
def open(self, endpoint: &Endpoint, deadline: &Deadline, cancel: &CancellationToken) -> Result[T, ClientError]
class Plain
def read_before(self: &mut Plain, buffer: &mut Bytes, start: i64, stop: i64, deadline: &Deadline, cancel: &CancellationToken) -> Result[i64, ClientError]
def write_before(self: &mut Plain, source: &[u8], deadline: &Deadline, cancel: &CancellationToken) -> Result[i64, ClientError]
class TcpConnector
def open(self, endpoint: &Endpoint, deadline: &Deadline, cancel: &CancellationToken) -> Result[Plain, ClientError]
def dial(endpoint: &Endpoint, deadline: &Deadline, cancel: &CancellationToken) -> Result[TcpStream, ClientError]
class Dialled
def read(self: &mut Dialled, buffer: &mut Bytes, start: i64, stop: i64) -> Result[i64, StepError[ClientError]]
def write(self: &mut Dialled, source: &[u8]) -> Result[i64, StepError[ClientError]]
class Proxy
endpoint: Endpoint
authorization: Option[str]
def proxy(url: &str) -> Result[Proxy, ClientError]
class Config
limits: Limits
connect_timeout: Duration
write_timeout: Duration
head_timeout: Duration
read_timeout: Duration
idle_timeout: Duration
continue_timeout: Duration
max_idle: i64
max_idle_per_origin: i64
redirects: i64
retries: i64
sensitive: list[str]
proxy: Option[Proxy]
def config() -> Result[Config, ClientError]
class Upload
length: Option[i64]
expect_continue: bool
def sized(length: i64) -> Upload
def chunked() -> Upload
class Unstreamed
def read(self: &mut Unstreamed, buffer: &mut Bytes, start: i64, stop: i64) -> Result[i64, StreamError]
class Stats
opened: u64
reused: u64
retried: u64
redirected: u64
discarded: u64
class Request
method: str
url: str
headers: list[Header]
body: Bytes
def add(self: &mut Request, name: str, value: str) -> Result[(), ClientError]
def request(method: &str, url: &str) -> Result[Request, ClientError]
class Fetched
status: i64
head: ResponseHead
url: str
redirects: i64
body: Bytes
class Wire[T]
def read(self: &mut Wire[T], buffer: &mut Bytes, start: i64, stop: i64) -> Result[i64, StepError[ClientError]]
def write(self: &mut Wire[T], source: &[u8]) -> Result[i64, StepError[ClientError]]
class Response[T]
status: i64
head: ResponseHead
url: str
redirects: i64
def is_done(self) -> bool
def read_body(self: &mut Response[T], buffer: &mut Bytes, start: i64, stop: i64) -> Result[i64, ClientError]
def read_all(self: &mut Response[T], out: &mut Bytes) -> Result[i64, ClientError]
def read_all_progress(self: &mut Response[T], out: &mut Bytes) -> Result[i64, Partial[ClientError]]
def total(self) -> i64
def trailers(self) -> &list[Header]
def read_typed(self: &mut Response[T], buffer: &mut Bytes, start: i64, stop: i64) -> Result[i64, StepError[ClientError]]
def read(self: &mut Response[T], buffer: &mut Bytes, start: i64, stop: i64) -> Result[i64, StreamError]
def take_failure(self: &mut Response[T]) -> Option[ClientError]
class TypedResponse[T]
def read(self: &mut TypedResponse[T], buffer: &mut Bytes, start: i64, stop: i64) -> Result[i64, StepError[ClientError]]
def typed_response[T](response: Response[T]) -> TypedResponse[T]
def unwrap_response[T](response: TypedResponse[T]) -> Response[T]
class Client[T, C]
def stats(self) -> Stats
def idle(self) -> i64
def close_idle(self) -> ()
def send(self, request: Request, deadline: &Deadline, cancel: &CancellationToken) -> Result[Response[T], ClientError]
def send_from[R: Reader](self, request: Request, body: &mut R, upload: &Upload, deadline: &Deadline, cancel: &CancellationToken) -> Result[Response[T], ClientError]
def fetch(self, request: Request, deadline: &Deadline, cancel: &CancellationToken) -> Result[Fetched, ClientError]
def get(self, url: &str, deadline: &Deadline, cancel: &CancellationToken) -> Result[Fetched, ClientError]
def finish(self, response: Response[T]) -> ()
def client(config: Config) -> Result[Client[Plain, TcpConnector], ClientError]
def connector[T: Transport, C: Connector[T]](opener: C, config: Config) -> Result[Client[T, C], ClientError]
std.http.cookie
Cookie fields read leniently and Set-Cookie fields validated against injection, with secure defaults. Errors: CookieError. Reference.
enum CookieError
InvalidName | InvalidValue | InvalidAttribute | TooManyCookies | InvalidKey | Http(HttpError) | Crypto(CryptoError) | Allocation(AllocError)
class Cookie
name: str
value: str
def cookies(headers: &list[Header], limit: i64) -> Result[list[Cookie], CookieError]
def find(cookies: &list[Cookie], name: &str) -> Option[i64]
enum SameSite
Strict | Lax | Unrestricted
class SetCookie
name: str
value: str
path: str
domain: str
max_age: Option[Duration]
secure: bool
http_only: bool
same_site: SameSite
def to_string(self) -> Result[str, CookieError]
def set_cookie(name: &str, value: &str) -> Result[SetCookie, CookieError]
def removal(name: &str) -> Result[SetCookie, CookieError]
def add(head: &mut ResponseHead, cookie: &SetCookie) -> Result[(), CookieError]
const min_key: i64 = 32
def sign(keys: &list[Bytes], name: &str, value: &str) -> Result[str, CookieError]
def verify(keys: &list[Bytes], name: &str, signed: &str) -> Result[Option[str], CookieError]
def find_signed(cookies: &list[Cookie], keys: &list[Bytes], name: &str) -> Result[Option[str], CookieError]
def signed_cookie(keys: &list[Bytes], name: &str, value: &str) -> Result[SetCookie, CookieError]
std.http.files
Static files below a Root, opened one segment at a time without following links, with media types and conditional requests. Errors: FilesError. Reference.
enum FilesError
Io(IoError) | Stream(StreamError) | Http(HttpError) | Truncated | Allocation(AllocError)
const max_path: i64 = 4096
def open_file(root: &Root, path: &str) -> Result[Option[FileStream], FilesError]
def media_type(name: &str) -> str
def http_date(timestamp: &Timestamp) -> Result[Option[str], FilesError]
def parse_http_date(text: &str) -> Result[Option[Timestamp], FilesError]
def serve[T: Transport](exchange: &mut Exchange[T], root: &Root, path: &str) -> Result[i64, FilesError]
std.http.https
std.http.client for https URLs through std.tls, verified by default, with CONNECT tunnels through a proxy. Errors: ClientError. Reference.
class Connection
def read_before(self: &mut Connection, buffer: &mut Bytes, start: i64, stop: i64, deadline: &Deadline, cancel: &CancellationToken) -> Result[i64, ClientError]
def write_before(self: &mut Connection, source: &[u8], deadline: &Deadline, cancel: &CancellationToken) -> Result[i64, ClientError]
def secure_error(error: TlsError) -> ClientError
class HttpsConnector
def open(self, endpoint: &Endpoint, deadline: &Deadline, cancel: &CancellationToken) -> Result[Connection, ClientError]
def connector(tls: ClientConfig) -> HttpsConnector
def client(config: Config, tls: ClientConfig) -> Result[Client[Connection, HttpsConnector], ClientError]
def system_client(config: Config) -> Result[Client[Connection, HttpsConnector], ClientError]
std.http.proxy
The client address and scheme behind explicitly trusted reverse proxies; deployment guidance. Errors: ProxyError. Reference.
enum ProxyError
InvalidNetwork | UnknownPeer | InvalidForwarded | TooManyHops | Allocation(AllocError)
enum Forwarding
# X-Forwarded-For, with the scheme from X-Forwarded-Proto. | XForwardedFor
class Proxies
header: Forwarding
max_hops: i64
def trust(self: &mut Proxies, network: &str) -> Result[(), ProxyError]
def trusts(self, ip: &IpAddress) -> bool
def proxies(header: Forwarding) -> Result[Proxies, ProxyError]
class Client
address: IpAddress
scheme: Option[str]
proxied: bool
def client(peer: &SocketAddress, headers: &list[Header], proxies: &Proxies) -> Result[Client, ProxyError]
def client_of[T: Transport](exchange: &Exchange[T], proxies: &Proxies) -> Result[Client, ProxyError]
std.http.server
HTTP/1.1 server: routing, a bounded worker pool and admission queue, idle connections parked off the workers, per-phase deadlines, and graceful shutdown. Errors: ServerError. Reference.
enum ServerError
InvalidConfig | InvalidPattern | DuplicateRoute | Allocation(AllocError) | Thread(SpawnError[()]) | Net(NetError)
protocol Transport
def read_before(self: &mut Transport, buffer: &mut Bytes, start: i64, stop: i64, deadline: &Deadline, cancel: &CancellationToken) -> Result[i64, NetError]
def write_before(self: &mut Transport, source: &[u8], deadline: &Deadline, cancel: &CancellationToken) -> Result[i64, NetError]
def shutdown(self: &mut Transport, direction: Shutdown) -> Result[(), NetError]
protocol Handler[T, R, E]
def handle(self, route: &R, exchange: &mut Exchange[T]) -> Result[(), E]
class Config
workers: i64
queue: i64
limits: Limits
idle_timeout: Duration
head_timeout: Duration
body_timeout: Duration
write_timeout: Duration
response_timeout: Duration
drain_timeout: Duration
max_requests: i64
events: Option[ChannelSink]
handshake_timeout: Duration
idle_connections: i64
def config() -> Config
class Report
connections: u64
rejected: u64
requests: u64
handler_errors: u64
protocol_errors: u64
timeouts: u64
cancelled: u64
accept_errors: u64
dropped_events: u64
enum Match
Found(i64, list[str]) | NotFound | MethodNotAllowed(str)
class Router[R]
def add(self: &mut Router[R], method: &str, pattern: &str, route: R) -> Result[(), ServerError]
def find(self, method: &str, path: &str) -> Result[Match, ServerError]
def route(self, index: i64) -> &R
def router[R]() -> Result[Router[R], ServerError]
class Conn[T]
def read(self: &mut Conn[T], buffer: &mut Bytes, start: i64, stop: i64) -> Result[i64, StreamError]
def write(self: &mut Conn[T], source: &[u8]) -> Result[i64, StreamError]
def flush(self: &mut Conn[T]) -> Result[(), StreamError]
class Exchange[T]
request: RequestHead
authority: str
path: str
query: str
params: list[str]
peer: Option[SocketAddress]
def stopping(self) -> bool
def cancelled(self) -> bool
def stop_token(self) -> CancellationToken
def cancel_token(self) -> CancellationToken
def status(self) -> i64
def sent(self) -> i64
def read_body(self: &mut Exchange[T], buffer: &mut Bytes, start: i64, stop: i64) -> Result[i64, HttpError]
def read_all(self: &mut Exchange[T], out: &mut Bytes) -> Result[i64, HttpError]
def respond(self: &mut Exchange[T], head: ResponseHead, body: &[u8]) -> Result[(), HttpError]
def send(self: &mut Exchange[T], status: i64, content_type: &str, body: &[u8]) -> Result[(), HttpError]
def send_text(self: &mut Exchange[T], status: i64, text: &str) -> Result[(), HttpError]
def start(self: &mut Exchange[T], head: ResponseHead) -> Result[(), HttpError]
def write(self: &mut Exchange[T], source: &[u8]) -> Result[(), HttpError]
def end(self: &mut Exchange[T]) -> Result[(), HttpError]
def switch_protocols(self: &mut Exchange[T], head: ResponseHead) -> Result[(T, Bytes), HttpError]
def detach(self: &mut Exchange[T], head: ResponseHead) -> Result[T, HttpError]
def reason(status: i64) -> str
def serve_connection[T: Transport, R, E, H: Handler[T, R, E]](stream: T, router: &Router[R], handler: &H, config: &Config, stop: &CancellationToken, cancel: &CancellationToken) -> Report
def serve_connection_from[T: Transport, R, E, H: Handler[T, R, E]](stream: T, peer: Option[SocketAddress], router: &Router[R], handler: &H, config: &Config, stop: &CancellationToken, cancel: &CancellationToken) -> Report
enum OpenError
TimedOut | Cancelled | Failed | Gone
protocol Opener[T]
def open(self, tcp: TcpStream) -> Result[T, OpenError]
def advance(self, stream: &mut T, deadline: &Deadline, cancel: &CancellationToken) -> Result[(), OpenError]
def handshakes(self) -> bool
def watch(self, stream: &T, selector: &mut Selector, key: u64) -> bool
def unwatch(self, stream: &T, selector: &mut Selector) -> ()
def serve[R, E, H: Handler[TcpStream, R, E]](listener: TcpListener, router: &Router[R], handler: &H, config: &Config, stop: &CancellationToken) -> Result[Report, ServerError]
def serve_with[T: Transport, O: Opener[T], R, E, H: Handler[T, R, E]](listener: TcpListener, opener: &O, router: &Router[R], handler: &H, config: &Config, stop: &CancellationToken) -> Result[Report, ServerError]
std.http.sse
Server-Sent Events from a server exchange: event, id, multi-line data, comments and retry, flushed per event, with disconnect and shutdown detection. Errors: SseError. Reference.
enum SseError
Net(NetError) | Allocation(AllocError) | Http(HttpError) | InvalidField | InvalidText | Disconnected | Closed | Stopping
class EventStream[T]
last_event_id: Option[str]
def is_open(self) -> bool
def send(self: &mut EventStream[T], data: &str, deadline: &Deadline, cancel: &CancellationToken, event: str = "", id: Option[str] = Nothing) -> Result[(), SseError]
def send_bytes(self: &mut EventStream[T], data: &[u8], deadline: &Deadline, cancel: &CancellationToken, event: str = "", id: Option[str] = Nothing) -> Result[(), SseError]
def comment(self: &mut EventStream[T], text: &str, deadline: &Deadline, cancel: &CancellationToken) -> Result[(), SseError]
def retry(self: &mut EventStream[T], span: &Duration, deadline: &Deadline, cancel: &CancellationToken) -> Result[(), SseError]
def wait(self: &mut EventStream[T], deadline: &Deadline, cancel: &CancellationToken) -> Result[(), SseError]
def close(self: &mut EventStream[T]) -> ()
def head() -> Result[ResponseHead, HttpError]
def accept[T: Transport](exchange: &mut Exchange[T]) -> Result[EventStream[T], SseError]
def accept_with[T: Transport](exchange: &mut Exchange[T], head: ResponseHead) -> Result[EventStream[T], SseError]
std.http.tls
std.http.server over std.tls, with handshake deadlines. Errors: ServerError. Reference.
class TlsTransport
stream: TlsStream
def read_before(self: &mut TlsTransport, buffer: &mut Bytes, start: i64, stop: i64, deadline: &Deadline, cancel: &CancellationToken) -> Result[i64, NetError]
def write_before(self: &mut TlsTransport, source: &[u8], deadline: &Deadline, cancel: &CancellationToken) -> Result[i64, NetError]
def shutdown(self: &mut TlsTransport, direction: Shutdown) -> Result[(), NetError]
class TlsOpener
def open(self, tcp: TcpStream) -> Result[TlsTransport, OpenError]
def advance(self, stream: &mut TlsTransport, deadline: &Deadline, cancel: &CancellationToken) -> Result[(), OpenError]
def handshakes(self) -> bool
def watch(self, stream: &TlsTransport, selector: &mut Selector, key: u64) -> bool
def unwatch(self, stream: &TlsTransport, selector: &mut Selector) -> ()
def serve[R, E, H: Handler[TlsTransport, R, E]](listener: TcpListener, tls: ServerConfig, router: &Router[R], handler: &H, config: &Config, stop: &CancellationToken) -> Result[Report, ServerError]
std.http.typed
Typed head parsing and writes, fixed/chunked/close-delimited bodies, and guarded connection recovery with concrete causes and progress. Errors: FramingError[E], Partial[E]. Reference.
enum FramingError[E]
Transport(E) | Truncated | InvalidProgress | OutOfRange | Broken | NotReusable | Protocol(HttpError)
def read_request[E, R: Reader[E]](reader: &mut R, limits: &Limits) -> Result[Option[RequestHead], Partial[FramingError[E]]]
def read_response[E, R: Reader[E]](reader: &mut R, limits: &Limits) -> Result[Option[ResponseHead], Partial[FramingError[E]]]
def write_request[E, W: Writer[E]](writer: &mut W, head: &RequestHead) -> Result[(), Partial[FramingError[E]]]
def write_response[E, W: Writer[E]](writer: &mut W, head: &ResponseHead) -> Result[(), Partial[FramingError[E]]]
def write_last_chunk[E, W: Writer[E]](writer: &mut W, trailers: &list[Header]) -> Result[(), Partial[FramingError[E]]]
def write_chunk[E, W: Writer[E]](writer: &mut W, source: &[u8]) -> Result[(), Partial[FramingError[E]]]
def fixed_body[E, R: Reader[E]](inner: R, length: NonNegative[i64]) -> Result[FixedBody[E, R], FramingError[E]]
class FixedBody[E, R]
def read(self: &mut FixedBody[E, R], buffer: &mut Bytes, start: i64, stop: i64) -> Result[i64, StepError[FramingError[E]]]
def chunked_body[E, R: Reader[E]](inner: R, limits: &Limits) -> Result[ChunkedBody[E, R], FramingError[E]]
class ChunkedBody[E, R]
trailers: list[Header]
def total(self) -> i64
def is_done(self) -> bool
def read(self: &mut ChunkedBody[E, R], buffer: &mut Bytes, start: i64, stop: i64) -> Result[i64, StepError[FramingError[E]]]
def close_body[E, R: Reader[E]](inner: R) -> CloseBody[E, R]
class CloseBody[E, R]
def read(self: &mut CloseBody[E, R], buffer: &mut Bytes, start: i64, stop: i64) -> Result[i64, StepError[FramingError[E]]]
def request_body[E, R: Reader[E]](inner: R, head: &RequestHead, limits: &Limits) -> Result[Body[E, R], FramingError[E]]
def response_body[E, R: Reader[E]](inner: R, head: &ResponseHead, method: &str, request_keep_alive: bool, limits: &Limits) -> Result[Body[E, R], FramingError[E]]
class Body[E, R]
def total(self) -> i64
def is_done(self) -> bool
def trailers(self) -> &list[Header]
def read(self: &mut Body[E, R], buffer: &mut Bytes, start: i64, stop: i64) -> Result[i64, StepError[FramingError[E]]]
def finish(self: &mut Body[E, R]) -> Result[R, FramingError[E]]
std.http.websocket
RFC 6455 WebSocket connections: the handshake as a server exchange or a ws client, bounded framed messages, pings and the closing handshake. Errors: WebSocketError. Reference.
enum WebSocketError
Net(NetError) | Allocation(AllocError) | Http(HttpError) | Connect(ClientError) | Entropy(StreamError) | InvalidConfig | InvalidUrl | UnsupportedScheme | OutOfRange | Refused(i64) | Rejected(i64) | InvalidHandshake | Protocol | TooLarge | InvalidText | Closed | Stopping
class Config
max_frame: Positive[i64]
max_message: Positive[i64]
subprotocol: Option[str]
def config() -> Config
enum Message
Text | Binary | Close(i64)
enum Kind
Text | Binary
class WebSocket[T]
subprotocol: Option[str]
def is_open(self) -> bool
def receive(self: &mut WebSocket[T], out: &mut Bytes, deadline: &Deadline, cancel: &CancellationToken) -> Result[Message, WebSocketError]
def send(self: &mut WebSocket[T], kind: Kind, data: &[u8], deadline: &Deadline, cancel: &CancellationToken) -> Result[(), WebSocketError]
def send_text(self: &mut WebSocket[T], text: &str, deadline: &Deadline, cancel: &CancellationToken) -> Result[(), WebSocketError]
def ping(self: &mut WebSocket[T], data: &[u8], deadline: &Deadline, cancel: &CancellationToken) -> Result[(), WebSocketError]
def close(self: &mut WebSocket[T], code: i64, reason: &str, deadline: &Deadline, cancel: &CancellationToken) -> Result[(), WebSocketError]
def accept[T: Transport](exchange: &mut Exchange[T], config: &Config) -> Result[WebSocket[T], WebSocketError]
class Target
secure: bool
host: str
port: u16
def target(url: &str) -> Result[Target, WebSocketError]
def dial(target: &Target, deadline: &Deadline, cancel: &CancellationToken) -> Result[TcpStream, WebSocketError]
class Wire[T]
def read(self: &mut Wire[T], buffer: &mut Bytes, start: i64, stop: i64) -> Result[i64, StreamError]
def handshake[T: Transport](stream: T, target: &Target, config: &Config, deadline: &Deadline, cancel: &CancellationToken) -> Result[WebSocket[T], WebSocketError]
def connect(url: &str, config: &Config, deadline: &Deadline, cancel: &CancellationToken) -> Result[WebSocket[TcpStream], WebSocketError]
std.http.wss
std.http.websocket clients for wss URLs through std.tls. Errors: WebSocketError. Reference.
def connect(url: &str, tls: &ClientConfig, config: &Config, deadline: &Deadline, cancel: &CancellationToken) -> Result[WebSocket[TlsTransport], WebSocketError]
std.image
Owned 8-bit pixel buffers, bounded PNG decoding and encoding, cropping, colour conversion and resizing. Errors: ImageError. Reference.
enum ImageError
OutOfRange | Corrupt | Checksum | Truncated | Unsupported | LimitExceeded | Allocation(AllocError)
enum ColorType
Gray | GrayAlpha | Rgb | Rgba
def channels(self) -> i64
def has_alpha(self) -> bool
enum Format
Png | Jpeg | Gif | WebP
enum Filter
Nearest | Bilinear
class Limits
max_width: NonNegative[i64]
max_height: NonNegative[i64]
max_bytes: NonNegative[i64]
def check(self, width: i64, height: i64, color: ColorType) -> Result[(), ImageError]
def limits() -> Limits
class Info
format: Format
width: i64
height: i64
color: ColorType
depth: i64
interlaced: bool
class Image
def width(self) -> i64
def height(self) -> i64
def color(self) -> ColorType
def pixels(self) -> &Bytes
def into_pixels(self: Image) -> Bytes
def copy(self) -> Result[Image, ImageError]
def get(self, x: i64, y: i64, channel: i64) -> Result[u8, ImageError]
def set(self: &mut Image, x: i64, y: i64, channel: i64, value: u8) -> Result[(), ImageError]
def crop(self, x: i64, y: i64, width: i64, height: i64) -> Result[Image, ImageError]
def convert(self, color: ColorType) -> Result[Image, ImageError]
def resize(self, width: i64, height: i64, filter: Filter) -> Result[Image, ImageError]
def new(width: Positive[i64], height: Positive[i64], color: ColorType) -> Result[Image, ImageError]
def from_pixels(width: Positive[i64], height: Positive[i64], color: ColorType, pixels: Bytes) -> Result[Image, ImageError]
def detect(data: &[u8]) -> Option[Format]
def info(data: &[u8]) -> Result[Info, ImageError]
def decode(data: &[u8], limits: &Limits) -> Result[Image, ImageError]
def encode_png(image: &Image, level: i64 = 6) -> Result[Bytes, ImageError]
std.io
Partial-progress byte stream contracts, retry helpers, fixed-capacity buffering, file and process-stream adapters, and checked sequential binary fields in an owned memory cursor. Errors: StreamError, Partial, CopyError, CloseError, BinaryError. Reference.
enum StreamError
Interrupted | UnexpectedEof | WriteZero | OutOfRange | InvalidProgress | Io(IoError) | Allocation(AllocError)
enum BinaryError
Truncated(i64, i64, i64) | TrailingData(i64, i64) | OutOfRange | LimitExceeded | Allocation(AllocError)
class Partial
completed: i64
cause: StreamError
enum CopyError
Read(i64, StreamError) | Write(i64, StreamError)
enum CloseError
Flush(StreamError) | Close(StreamError) | FlushAndClose(StreamError, StreamError)
enum SeekFrom
Start(i64) | Current(i64) | End(i64)
protocol Reader
def read(self: &mut Reader, buffer: &mut Bytes, start: i64, stop: i64) -> Result[i64, StreamError]
protocol Writer
def write(self: &mut Writer, source: &[u8]) -> Result[i64, StreamError]
def flush(self: &mut Writer) -> Result[(), StreamError]
protocol Seeker
def seek(self: &mut Seeker, target: SeekFrom) -> Result[i64, StreamError]
protocol Closer
def close(self: &mut Closer) -> Result[(), StreamError]
def classify(error: IoError) -> StreamError
def read_exact[R: Reader](reader: &mut R, buffer: &mut Bytes, start: i64, stop: i64) -> Result[(), Partial]
def write_all[W: Writer](writer: &mut W, source: &[u8]) -> Result[(), Partial]
def read_up_to[R: Reader](reader: &mut R, out: &mut Bytes, limit: NonNegative[i64]) -> Result[i64, Partial]
def copy_all[R: Reader, W: Writer](reader: &mut R, writer: &mut W, scratch: &mut Bytes) -> Result[i64, CopyError]
class BufReader[R]
inner: R
def capacity(self) -> i64
def buffered(self) -> i64
def read(self: &mut BufReader[R], buffer: &mut Bytes, start: i64, stop: i64) -> Result[i64, StreamError]
def read_until(self: &mut BufReader[R], delimiter: u8, out: &mut Bytes, limit: Positive[i64]) -> Result[i64, StreamError]
def buffered_reader[R](inner: R, capacity: Positive[i64]) -> Result[BufReader[R], StreamError]
def seek_reader[R: Seeker](reader: &mut BufReader[R], target: SeekFrom) -> Result[i64, StreamError]
class BufWriter[W]
inner: W
def capacity(self) -> i64
def pending(self) -> i64
def write(self: &mut BufWriter[W], source: &[u8]) -> Result[i64, StreamError]
def flush(self: &mut BufWriter[W]) -> Result[(), StreamError]
def buffered_writer[W](inner: W, capacity: Positive[i64]) -> Result[BufWriter[W], StreamError]
def seek_writer[W: Seeker](writer: &mut BufWriter[W], target: SeekFrom) -> Result[i64, StreamError]
def close_buffered[W: Closer](writer: &mut BufWriter[W]) -> Result[(), CloseError]
class Cursor
data: Bytes
def position(self) -> i64
def remaining(self) -> i64
def require_end(self) -> Result[(), BinaryError]
def read_bytes(self: &mut Cursor, count: i64, max_bytes: i64 = 1048576) -> Result[Bytes, BinaryError]
def write_bytes(self: &mut Cursor, source: &[u8], max_bytes: i64 = 1048576) -> Result[(), BinaryError]
def read_u8(self: &mut Cursor) -> Result[u8, BinaryError]
def write_u8(self: &mut Cursor, value: u8, max_bytes: i64 = 1048576) -> Result[(), BinaryError]
def read_i8(self: &mut Cursor) -> Result[i8, BinaryError]
def write_i8(self: &mut Cursor, value: i8, max_bytes: i64 = 1048576) -> Result[(), BinaryError]
def read_u16(self: &mut Cursor, order: Endian) -> Result[u16, BinaryError]
def write_u16(self: &mut Cursor, value: u16, order: Endian, max_bytes: i64 = 1048576) -> Result[(), BinaryError]
def read_i16(self: &mut Cursor, order: Endian) -> Result[i16, BinaryError]
def write_i16(self: &mut Cursor, value: i16, order: Endian, max_bytes: i64 = 1048576) -> Result[(), BinaryError]
def read_u32(self: &mut Cursor, order: Endian) -> Result[u32, BinaryError]
def write_u32(self: &mut Cursor, value: u32, order: Endian, max_bytes: i64 = 1048576) -> Result[(), BinaryError]
def read_i32(self: &mut Cursor, order: Endian) -> Result[i32, BinaryError]
def write_i32(self: &mut Cursor, value: i32, order: Endian, max_bytes: i64 = 1048576) -> Result[(), BinaryError]
def read_u64(self: &mut Cursor, order: Endian) -> Result[u64, BinaryError]
def write_u64(self: &mut Cursor, value: u64, order: Endian, max_bytes: i64 = 1048576) -> Result[(), BinaryError]
def read_i64(self: &mut Cursor, order: Endian) -> Result[i64, BinaryError]
def write_i64(self: &mut Cursor, value: i64, order: Endian, max_bytes: i64 = 1048576) -> Result[(), BinaryError]
def read_f32(self: &mut Cursor, order: Endian) -> Result[f32, BinaryError]
def write_f32(self: &mut Cursor, value: f32, order: Endian, max_bytes: i64 = 1048576) -> Result[(), BinaryError]
def read_f64(self: &mut Cursor, order: Endian) -> Result[f64, BinaryError]
def write_f64(self: &mut Cursor, value: f64, order: Endian, max_bytes: i64 = 1048576) -> Result[(), BinaryError]
def read(self: &mut Cursor, buffer: &mut Bytes, start: i64, stop: i64) -> Result[i64, StreamError]
def write(self: &mut Cursor, source: &[u8]) -> Result[i64, StreamError]
def flush(self: &mut Cursor) -> Result[(), StreamError]
def seek(self: &mut Cursor, target: SeekFrom) -> Result[i64, StreamError]
def cursor(data: Bytes) -> Cursor
class FileStream
file: File
def read(self: &mut FileStream, buffer: &mut Bytes, start: i64, stop: i64) -> Result[i64, StreamError]
def write(self: &mut FileStream, source: &[u8]) -> Result[i64, StreamError]
def flush(self: &mut FileStream) -> Result[(), StreamError]
def sync(self: &mut FileStream) -> Result[(), StreamError]
def seek(self: &mut FileStream, target: SeekFrom) -> Result[i64, StreamError]
def close(self: &mut FileStream) -> Result[(), StreamError]
def open_file(path: &str, mode: &str) -> Result[FileStream, StreamError]
def open_path(path: &Path, mode: &str) -> Result[FileStream, StreamError]
def from_file(file: File) -> FileStream
class Stdin
def is_terminal(self) -> bool
def read(self: &mut Stdin, buffer: &mut Bytes, start: i64, stop: i64) -> Result[i64, StreamError]
class Stdout
def is_terminal(self) -> bool
def write(self: &mut Stdout, source: &[u8]) -> Result[i64, StreamError]
def flush(self: &mut Stdout) -> Result[(), StreamError]
class Stderr
def is_terminal(self) -> bool
def write(self: &mut Stderr, source: &[u8]) -> Result[i64, StreamError]
def flush(self: &mut Stderr) -> Result[(), StreamError]
def stdin() -> Stdin
def stdout() -> Stdout
def stderr() -> Stderr
std.io.typed
Byte stream protocols that carry a caller-chosen error type, buffering, adapters for std.io streams, and exact read, write, and copy helpers. Errors: StepError[E], Partial[E], CopyFailure[R, W]. Reference.
enum StepError[E]
Interrupted | Source(E)
class LegacyReader[R]
def read(self: &mut LegacyReader[R], buffer: &mut Bytes, start: i64, stop: i64) -> Result[i64, StepError[StreamError]]
def reader[R: std.io.Reader](inner: R) -> LegacyReader[R]
def unwrap_reader[R](reader: LegacyReader[R]) -> R
class LegacyWriter[W]
def write(self: &mut LegacyWriter[W], buffer: &[u8]) -> Result[i64, StepError[StreamError]]
def writer[W: std.io.Writer](inner: W) -> LegacyWriter[W]
def unwrap_writer[W](writer: LegacyWriter[W]) -> W
protocol Reader[E]
def read(self: &mut Reader[E], buffer: &mut Bytes, start: i64, stop: i64) -> Result[i64, StepError[E]]
protocol Writer[E]
def write(self: &mut Writer[E], buffer: &[u8]) -> Result[i64, StepError[E]]
enum ReadFailure[E]
Source(E) | UnexpectedEof | InvalidProgress | OutOfRange
class BufReader[E, R]
inner: R
def capacity(self) -> i64
def buffered(self) -> i64
def read(self: &mut BufReader[E, R], buffer: &mut Bytes, start: i64, stop: i64) -> Result[i64, StepError[ReadFailure[E]]]
def buffered_reader[E, R: Reader[E]](inner: R, capacity: Positive[i64]) -> Result[BufReader[E, R], StreamError]
enum WriteFailure[E]
Source(E) | WriteZero | InvalidProgress
class Partial[E]
completed: i64
cause: E
enum CopyFailure[R, W]
Read(i64, ReadFailure[R]) | Write(i64, WriteFailure[W])
def within(start: i64, stop: i64, size: i64) -> bool
def read_exact[E, R: Reader[E]](reader: &mut R, buffer: &mut Bytes, start: i64, stop: i64) -> Result[(), Partial[ReadFailure[E]]]
def write_all[E, W: Writer[E]](writer: &mut W, buffer: &[u8]) -> Result[(), Partial[WriteFailure[E]]]
def copy_all[E, G, R: Reader[E], W: Writer[G]](reader: &mut R, writer: &mut W, scratch: &mut Bytes) -> Result[i64, CopyFailure[E, G]]
std.json
Strict JSON under depth, input, and output limits: resumable event decoder, validating encoder, and a value tree. Errors: JsonError. Reference.
enum JsonFault
UnexpectedByte | UnexpectedEnd | TrailingData | InvalidLiteral | InvalidNumber | NumberOutOfRange | InvalidEscape | InvalidSurrogate | ControlCharacter | InvalidUtf8 | DuplicateKey | DepthExceeded | InputTooLarge | OutputTooLarge | NonFiniteNumber | InvalidState | Stream(StreamError) | Allocation(AllocError)
class JsonError
fault: JsonFault
offset: i64
line: i64
column: i64
enum DuplicateKeys
Reject | Keep
class Limits
max_depth: NonNegative[i64]
max_input: NonNegative[i64]
max_output: NonNegative[i64]
duplicate_keys: DuplicateKeys
enum Json
Null | Bool(bool) | Int(i64) | Float(f64) | Text(str) | Array(list[Json]) | Object(list[Member])
class Member
key: str
value: Json
enum Event
Pending | End | Null | Bool(bool) | Number(str) | Text(str) | Key(str) | StartArray | EndArray | StartObject | EndObject
def limits() -> Limits
class Decoder
def offset(self) -> i64
def depth(self) -> i64
def reject(self, fault: JsonFault) -> JsonError
def feed(self: &mut Decoder, source: &[u8]) -> Result[(), JsonError]
def finish(self: &mut Decoder) -> ()
def next(self: &mut Decoder) -> Result[Event, JsonError]
def decoder(bounds: &Limits) -> Result[Decoder, JsonError]
def read_event[R: Reader](source: &mut Decoder, reader: &mut R) -> Result[Event, JsonError]
def parse(source: &[u8], bounds: &Limits) -> Result[Json, JsonError]
def parse_text(text: &str, bounds: &Limits) -> Result[Json, JsonError]
def read_value[R: Reader](reader: &mut R, bounds: &Limits) -> Result[Json, JsonError]
def find(members: &list[Member], key: &str) -> Option[i64]
class Encoder
def pending(self) -> i64
def written(self) -> i64
def is_complete(self) -> bool
def null(self: &mut Encoder) -> Result[(), JsonError]
def boolean(self: &mut Encoder, value: bool) -> Result[(), JsonError]
def integer(self: &mut Encoder, value: i64) -> Result[(), JsonError]
def unsigned(self: &mut Encoder, value: u64) -> Result[(), JsonError]
def decimal(self: &mut Encoder, value: f64) -> Result[(), JsonError]
def number(self: &mut Encoder, text: &str) -> Result[(), JsonError]
def text(self: &mut Encoder, value: &str) -> Result[(), JsonError]
def key(self: &mut Encoder, name: &str) -> Result[(), JsonError]
def start_array(self: &mut Encoder) -> Result[(), JsonError]
def end_array(self: &mut Encoder) -> Result[(), JsonError]
def start_object(self: &mut Encoder) -> Result[(), JsonError]
def end_object(self: &mut Encoder) -> Result[(), JsonError]
def event(self: &mut Encoder, event: &Event) -> Result[(), JsonError]
def value(self: &mut Encoder, value: &Json) -> Result[(), JsonError]
def take(self: &mut Encoder) -> Result[Bytes, JsonError]
def encoder(bounds: &Limits) -> Result[Encoder, JsonError]
def write_pending[W: Writer](source: &mut Encoder, writer: &mut W) -> Result[(), JsonError]
def encode(value: &Json, bounds: &Limits) -> Result[Bytes, JsonError]
def encode_text(value: &Json, bounds: &Limits) -> Result[str, JsonError]
def write_value[W: Writer](writer: &mut W, value: &Json, bounds: &Limits) -> Result[(), JsonError]
std.jwt
JSON Web Tokens: compact JWS signing and verification with the algorithm fixed by the key, strict decoding, and explicit-clock claim checks. Errors: JwtError. Reference.
enum Algorithm
Hs256 | Hs512 | Rs256 | Rs384 | Rs512 | Ps256 | Ps384 | Ps512 | Es256 | EdDsa
def name(self) -> str
enum Claim
Expiry | NotBefore | IssuedAt | Issuer | Audience
enum JwtError
Malformed | NotObject | TooLarge | UnsupportedAlgorithm | AlgorithmMismatch | CriticalHeader | InvalidSignature | Expired | NotYetValid | MissingClaim(Claim) | InvalidClaim(Claim) | WrongIssuer | WrongAudience | InvalidKey | InvalidValidation | RandomFailure | Allocation(AllocError)
def format[W](self, output: &mut Formatter[W]) -> Result[(), FormatError]
class SigningKey
def algorithm(self) -> Algorithm
def verifying_key(self) -> Result[VerifyingKey, JwtError]
class VerifyingKey
def algorithm(self) -> Algorithm
def hmac_signing_key(algorithm: Algorithm, secret: &[u8]) -> Result[SigningKey, JwtError]
def hmac_verifying_key(algorithm: Algorithm, secret: &[u8]) -> Result[VerifyingKey, JwtError]
def rsa_signing_key(algorithm: Algorithm, key: RsaPrivateKey) -> Result[SigningKey, JwtError]
def rsa_verifying_key(algorithm: Algorithm, key: RsaPublicKey) -> Result[VerifyingKey, JwtError]
def es256_signing_key(key: P256PrivateKey) -> SigningKey
def es256_verifying_key(key: P256PublicKey) -> VerifyingKey
def ed25519_signing_key(seed: &[u8]) -> Result[SigningKey, JwtError]
def ed25519_verifying_key(public_key: &[u8]) -> Result[VerifyingKey, JwtError]
const default_max_size: i64 = 8192
const max_leeway: i64 = 86400
const max_token_size: i64 = 1048576
class Validation
now: Timestamp
leeway: Duration
require_expiry: bool
issuer: Option[str]
audience: Option[str]
max_size: Positive[i64]
def validation(now: &Timestamp) -> Validation
class Header
algorithm: Algorithm
key_id: Option[str]
token_type: Option[str]
class Token
header: Header
claims: list[Member]
def decode_header(token: &str, max_size: Positive[i64] = 8192) -> Result[Header, JwtError]
def verify(token: &str, key: &VerifyingKey, rules: &Validation) -> Result[Token, JwtError]
def sign(key: &SigningKey, claims: &Json, key_id: Option[str] = Nothing) -> Result[str, JwtError]
std.log
Leveled events with typed fields, delivered to caller-chosen sinks: logfmt lines, bounded memory, or a nonblocking channel. Errors: LogError. Reference.
enum LogError
InvalidSpec | InvalidKey | UnknownLevel | TooLarge | Full | Closed | Format(FormatError) | Output(StreamError) | Allocation(AllocError)
enum Level
Trace | Debug | Info | Warn | Error
def level_name(level: &Level) -> str
def parse_level(name: &str) -> Result[Level, LogError]
enum Value
Text(str) | Integer(i64) | Unsigned(u64) | Floating(f64) | Boolean(bool)
class Field
key: str
value: Value
def text(key: str, value: str) -> Field
def integer(key: str, value: i64) -> Field
def unsigned(key: str, value: u64) -> Field
def floating(key: str, value: f64) -> Field
def boolean(key: str, value: bool) -> Field
class Event
level: Level
message: str
fields: list[Field]
time: Option[Timestamp]
def event(level: Level, message: str, fields: list[Field]) -> Event
protocol Sink
def emit(self: &mut Sink, event: Event) -> Result[(), LogError]
enum Clock
Off | Wall | Fixed(Timestamp)
class Logger[S]
sink: S
def enabled(self, level: &Level) -> bool
def emitted(self) -> u64
def filtered(self) -> u64
def failed(self) -> u64
def emit(self: &mut Logger[S], event: Event) -> Result[(), LogError]
def log(self: &mut Logger[S], level: Level, message: str, fields: list[Field]) -> Result[(), LogError]
def debug(self: &mut Logger[S], message: str, fields: list[Field]) -> Result[(), LogError]
def info(self: &mut Logger[S], message: str, fields: list[Field]) -> Result[(), LogError]
def warn(self: &mut Logger[S], message: str, fields: list[Field]) -> Result[(), LogError]
def error(self: &mut Logger[S], message: str, fields: list[Field]) -> Result[(), LogError]
def logger[S: Sink](sink: S, threshold: Level, clock: Clock) -> Logger[S]
class LineBuffer
data: Bytes
def write(self: &mut LineBuffer, source: &[u8]) -> Result[i64, StreamError]
def flush(self: &mut LineBuffer) -> Result[(), StreamError]
class TextSink[W]
writer: W
def emit(self: &mut TextSink[W], event: Event) -> Result[(), LogError]
def text_sink[W: Writer](writer: W, max_line: Positive[i64]) -> Result[TextSink[W], LogError]
class MemorySink
events: list[Event]
def emit(self: &mut MemorySink, event: Event) -> Result[(), LogError]
def memory_sink(capacity: NonNegative[i64]) -> Result[MemorySink, LogError]
class Discard
def emit(self: &mut Discard, event: Event) -> Result[(), LogError]
def discard() -> Discard
class ChannelSink
def share(self) -> ChannelSink
def emit(self: &mut ChannelSink, event: Event) -> Result[(), LogError]
def channel_sink(sender: Sender[Event]) -> ChannelSink
class Drained
delivered: u64
failed: u64
def drain[S: Sink](receiver: &Receiver[Event], sink: &mut S) -> Drained
std.markdown
CommonMark 0.31.2 parsing into events and HTML rendering, under input, depth, and output limits. Errors: MarkdownError. Reference.
enum MarkdownFault
InvalidUtf8 | InputTooLarge | OutputTooLarge | DepthExceeded | InvalidState | Stream(StreamError) | Allocation(AllocError)
class MarkdownError
fault: MarkdownFault
offset: i64
line: i64
class Limits
max_input: NonNegative[i64]
max_depth: NonNegative[i64]
max_output: NonNegative[i64]
enum Tag
Paragraph | Heading(i64) | BlockQuote | CodeBlock(str) | BulletList | OrderedList(i64) | Item | Emphasis | Strong | Link(str, str) | Image(str, str)
enum Event
Start(Tag) | End(Tag) | Text(str) | Code(str) | Html(str) | InlineHtml(str) | SoftBreak | HardBreak | Rule | Done
def limits() -> Limits
class Parser
def next(self: &mut Parser) -> Result[Event, MarkdownError]
def parser(source: &[u8], bounds: &Limits) -> Result[Parser, MarkdownError]
def parser_text(text: &str, bounds: &Limits) -> Result[Parser, MarkdownError]
class HtmlRenderer
def pending(self) -> i64
def written(self) -> i64
def event(self: &mut HtmlRenderer, event: &Event) -> Result[(), MarkdownError]
def take(self: &mut HtmlRenderer) -> Result[Bytes, MarkdownError]
def renderer(bounds: &Limits) -> Result[HtmlRenderer, MarkdownError]
def write_pending[W: Writer](source: &mut HtmlRenderer, writer: &mut W) -> Result[(), MarkdownError]
def write_html[W: Writer](source: &mut Parser, writer: &mut W, bounds: &Limits) -> Result[(), MarkdownError]
def render(source: &[u8], bounds: &Limits) -> Result[Bytes, MarkdownError]
def to_html(text: &str, bounds: &Limits) -> Result[str, MarkdownError]
std.template
Bounded Jinja-style text templates with explicit JSON values, HTML escaping, conditions, loops and named inheritance. Errors: TemplateError. Reference.
enum TemplateError
LimitExceeded | Syntax(i64) | UnknownTemplate(str) | DuplicateTemplate(str) | MissingValue(str) | TypeMismatch | InheritanceCycle | Allocation(AllocError) | Bytes(BytesError)
enum HtmlContext
Text | DoubleQuotedAttribute | SingleQuotedAttribute | TextAndAttributes
class Limits
max_source: NonNegative[i64]
max_tokens: NonNegative[i64]
max_templates: NonNegative[i64]
max_context_nodes: Positive[i64]
max_context_bytes: NonNegative[i64]
max_depth: Positive[i64]
max_steps: NonNegative[i64]
max_output: NonNegative[i64]
def limits() -> Limits
def escape(text: &str, context: HtmlContext, max_bytes: NonNegative[i64] = 16777216) -> Result[str, TemplateError]
class Engine
def add(self: &mut Engine, name: &str, source: &str) -> Result[(), TemplateError]
def render(self, name: &str, context: &Json, html: bool = True) -> Result[str, TemplateError]
def engine(bounds: &Limits) -> Result[Engine, TemplateError]
std.math
Floating-point functions, checked rounding, integer algorithms, integer and float helpers, and constants. Errors: MathError, OverflowError[T]. Reference.
enum MathError
Domain | Overflow
const pi: f64 = f64.PI
const e: f64 = f64.E
const tau: f64 = f64.TAU
const inf: f64 = f64.INF
const nan: f64 = f64.NAN
def isnan(x: f64) -> bool
def isinf(x: f64) -> bool
def isfinite(x: f64) -> bool
def fabs(x: f64) -> f64
def copysign(x: f64, y: f64) -> f64
def isclose(a: f64, b: f64, rel_tol: f64, abs_tol: f64) -> Result[bool, MathError]
def degrees(x: f64) -> f64
def radians(x: f64) -> f64
def sqrt(x: f64) -> Result[f64, MathError]
def exp(x: f64) -> Result[f64, MathError]
def exp2(x: f64) -> Result[f64, MathError]
def expm1(x: f64) -> Result[f64, MathError]
def log(x: f64) -> Result[f64, MathError]
def log2(x: f64) -> Result[f64, MathError]
def log10(x: f64) -> Result[f64, MathError]
def log1p(x: f64) -> Result[f64, MathError]
def log_base(x: f64, base: f64) -> Result[f64, MathError]
def sin(x: f64) -> Result[f64, MathError]
def cos(x: f64) -> Result[f64, MathError]
def tan(x: f64) -> Result[f64, MathError]
def asin(x: f64) -> Result[f64, MathError]
def acos(x: f64) -> Result[f64, MathError]
def atan(x: f64) -> Result[f64, MathError]
def atan2(y: f64, x: f64) -> Result[f64, MathError]
def sinh(x: f64) -> Result[f64, MathError]
def cosh(x: f64) -> Result[f64, MathError]
def tanh(x: f64) -> Result[f64, MathError]
def asinh(x: f64) -> Result[f64, MathError]
def acosh(x: f64) -> Result[f64, MathError]
def atanh(x: f64) -> Result[f64, MathError]
def cbrt(x: f64) -> Result[f64, MathError]
def pow(x: f64, y: f64) -> Result[f64, MathError]
def fmod(x: f64, y: f64) -> Result[f64, MathError]
def floor(x: f64) -> Result[i64, MathError]
def ceil(x: f64) -> Result[i64, MathError]
def trunc(x: f64) -> Result[i64, MathError]
def round(x: f64) -> Result[i64, MathError]
def divmod[T: IntType](a: T, b: T) -> Result[(T, T), MathError]
def divmod_float(a: f64, b: f64) -> Result[(f64, f64), MathError]
def factorial(n: NonNegative[i64]) -> Result[u64, MathError]
def gcd(a: u64, b: u64) -> u64
def lcm(a: u64, b: u64) -> Result[u64, MathError]
def isqrt(n: u64) -> u64
def perm(n: u64, k: u64) -> Result[u64, MathError]
def comb(n: u64, k: u64) -> Result[u64, MathError]
def abs[T: IntType](x: T) -> Result[T, OverflowError[T]]
def abs_float[T: FloatType](x: T) -> T
def min[T](a: T, b: T) -> T
def max[T](a: T, b: T) -> T
def min_all[T](items: &list[T]) -> Option[T]
def max_all[T](items: &list[T]) -> Option[T]
def sum[T: IntType](items: &list[T], start: T) -> Result[T, OverflowError[T]]
def prod[T: IntType](items: &list[T], start: T) -> Result[T, OverflowError[T]]
def sum_iter[I, T: IntType](items: &I, start: T) -> Result[T, OverflowError[T]]
def prod_iter[I, T: IntType](items: &I, start: T) -> Result[T, OverflowError[T]]
def sum_float[T: FloatType](items: &list[T], start: T) -> T
def prod_float[T: FloatType](items: &list[T], start: T) -> T
def sum_iter_float[I, T: FloatType](items: &I, start: T) -> T
def prod_iter_float[I, T: FloatType](items: &I, start: T) -> T
std.mime
Media types: RFC 9110 parsing and formatting with parameters, media-range matching, and the file-extension table. Errors: ParseError, AllocError. Reference.
enum MimeError
Invalid | Allocation(AllocError)
class Param
name: str
value: str
class MediaType
def type_name(self) -> &str
def subtype(self) -> &str
def params(self) -> &list[Param]
def param(self, name: &str) -> Option[str]
def suffix(self) -> Result[Option[str], AllocError]
def essence(self) -> Result[str, AllocError]
def is_wildcard(self) -> bool
def matches(self, pattern: &MediaType) -> bool
def set_param(self: &mut MediaType, name: &str, value: &str) -> Result[(), MimeError]
def remove_param(self: &mut MediaType, name: &str) -> bool
def to_string(self) -> Result[str, AllocError]
def parse(text: &str) -> Result[MediaType, MimeError]
def new(type_name: &str, subtype: &str) -> Result[MediaType, MimeError]
def from_extension(extension: &str) -> Option[str]
def from_file_name(name: &str) -> str
std.net
TCP listeners and streams, UDP sockets, IP addresses and name lookup, with deadlines and cancellation. Errors: NetError. Reference.
enum NetError
TimedOut | ConnectionTimedOut | WouldBlock | Cancelled | Refused | Reset | AddressInUse | AddressUnavailable | Unreachable | NameNotFound | TryAgain | InvalidAddress | HandleLimit | Stream(StreamError)
enum IpAddress
V4(u32) | V6(u64, u64)
class SocketAddress
ip: IpAddress
port: u16
scope: u32
def to_string(self) -> Result[str, NetError]
def socket_address(ip: IpAddress, port: u16) -> SocketAddress
def ipv4(a: u8, b: u8, c: u8, d: u8) -> IpAddress
def loopback_v4(port: u16) -> SocketAddress
def loopback_v6(port: u16) -> SocketAddress
def any_v4(port: u16) -> SocketAddress
def any_v6(port: u16) -> SocketAddress
def format_ip(ip: IpAddress) -> Result[str, NetError]
def parse_ip(text: &str) -> Result[IpAddress, NetError]
def parse_address(text: &str) -> Result[SocketAddress, NetError]
def resolve(host: &str, port: u16, deadline: &Deadline, cancel: &CancellationToken) -> Result[list[SocketAddress], NetError]
class Lookup
def descriptor(self) -> &File
def watch(self, selector: &mut Selector, key: u64) -> Result[(), NetError]
def unwatch(self, selector: &mut Selector) -> Result[(), NetError]
def finish_now(self: &mut Lookup) -> Result[list[SocketAddress], NetError]
def start_lookup(host: &str, port: u16) -> Result[Lookup, NetError]
enum Shutdown
Read | Write | Both
class TcpStream
def read(self: &mut TcpStream, buffer: &mut Bytes, start: i64, stop: i64) -> Result[i64, StreamError]
def write(self: &mut TcpStream, source: &[u8]) -> Result[i64, StreamError]
def flush(self: &mut TcpStream) -> Result[(), StreamError]
def close(self: &mut TcpStream) -> Result[(), StreamError]
def read_now(self: &mut TcpStream, buffer: &mut Bytes, start: i64, stop: i64) -> Result[i64, NetError]
def write_now(self: &mut TcpStream, source: &[u8]) -> Result[i64, NetError]
def read_before(self: &mut TcpStream, buffer: &mut Bytes, start: i64, stop: i64, deadline: &Deadline, cancel: &CancellationToken) -> Result[i64, NetError]
def write_before(self: &mut TcpStream, source: &[u8], deadline: &Deadline, cancel: &CancellationToken) -> Result[i64, NetError]
def write_all_before(self: &mut TcpStream, source: &[u8], deadline: &Deadline, cancel: &CancellationToken) -> Result[(), NetError]
def local_address(self) -> Result[SocketAddress, NetError]
def peer_address(self) -> Result[SocketAddress, NetError]
def shutdown(self: &mut TcpStream, direction: Shutdown) -> Result[(), NetError]
def descriptor(self) -> &File
def watch(self, selector: &mut Selector, key: u64, interest: Interest) -> Result[(), NetError]
def unwatch(self, selector: &mut Selector) -> Result[(), NetError]
def set_no_delay(self: &mut TcpStream, enabled: bool) -> Result[(), NetError]
def acknowledge_now(self: &mut TcpStream) -> Result[(), NetError]
class TcpListener
def local_address(self) -> Result[SocketAddress, NetError]
def descriptor(self) -> &File
def watch(self, selector: &mut Selector, key: u64) -> Result[(), NetError]
def unwatch(self, selector: &mut Selector) -> Result[(), NetError]
def accept_now(self) -> Result[TcpStream, NetError]
def accept(self, deadline: &Deadline, cancel: &CancellationToken) -> Result[TcpStream, NetError]
def close(self: &mut TcpListener) -> Result[(), NetError]
def listen(address: &SocketAddress, backlog: i64) -> Result[TcpListener, NetError]
def connect(address: &SocketAddress, deadline: &Deadline, cancel: &CancellationToken) -> Result[TcpStream, NetError]
class Connecting
def descriptor(self) -> &File
def watch(self, selector: &mut Selector, key: u64) -> Result[(), NetError]
def unwatch(self, selector: &mut Selector) -> Result[(), NetError]
def finish_now(self: &mut Connecting) -> Result[(), NetError]
def stream(self: Connecting) -> TcpStream
def start_connect(address: &SocketAddress) -> Result[Connecting, NetError]
class Datagram
size: i64
stored: i64
source: SocketAddress
def truncated(self) -> bool
class UdpSocket
def local_address(self) -> Result[SocketAddress, NetError]
def descriptor(self) -> &File
def watch(self, selector: &mut Selector, key: u64, interest: Interest) -> Result[(), NetError]
def unwatch(self, selector: &mut Selector) -> Result[(), NetError]
def set_broadcast(self: &mut UdpSocket, enabled: bool) -> Result[(), NetError]
def send_to(self: &mut UdpSocket, source: &[u8], target: &SocketAddress, deadline: &Deadline, cancel: &CancellationToken) -> Result[i64, NetError]
def receive_from(self: &mut UdpSocket, buffer: &mut Bytes, start: i64, stop: i64, deadline: &Deadline, cancel: &CancellationToken) -> Result[Datagram, NetError]
def close(self: &mut UdpSocket) -> Result[(), NetError]
def bind_udp(address: &SocketAddress) -> Result[UdpSocket, NetError]
std.path
Lossless Linux paths, lexical operations, checked text conversion and escaped display. Errors: IoError, AllocError. Reference.
class Path
def as_bytes(self) -> &list[u8]
def to_text(self) -> Result[str, IoError]
def to_string(self) -> Result[str, IoError]
def copy(self) -> Result[Path, AllocError]
def join(self, child: &Path) -> Result[Path, IoError]
def is_absolute(self) -> bool
def parent(self) -> Result[Option[Path], IoError]
def file_name(self) -> Result[Option[Path], IoError]
def components(self) -> Result[list[list[u8]], IoError]
def from_bytes(bytes: list[u8]) -> Path
def from_text(text: &str) -> Result[Path, IoError]
std.process
Children spawned from an argument vector with no implicit shell, piped streams, bounded output collection, signals, and explicit reaping. Errors: IoError, StreamError. Reference.
enum Stdio
Inherit | Null | Piped
class ExitStatus
def success(self) -> bool
def code(self) -> Option[i32]
def signal(self) -> Option[i32]
class Output
status: ExitStatus
stdout: Bytes
stderr: Bytes
class Command
def arg_bytes(self: &mut Command, value: &list[u8]) -> Result[(), IoError]
def arg(self: &mut Command, value: &str) -> Result[(), IoError]
def arg_path(self: &mut Command, value: &Path) -> Result[(), IoError]
def env_bytes(self: &mut Command, name: &list[u8], value: &list[u8]) -> Result[(), IoError]
def env(self: &mut Command, name: &str, value: &str) -> Result[(), IoError]
def env_remove_bytes(self: &mut Command, name: &list[u8]) -> Result[(), IoError]
def env_remove(self: &mut Command, name: &str) -> Result[(), IoError]
def env_clear(self: &mut Command) -> ()
def current_dir(self: &mut Command, path: &Path) -> Result[(), IoError]
def stdin(self: &mut Command, mode: Stdio) -> ()
def stdout(self: &mut Command, mode: Stdio) -> ()
def stderr(self: &mut Command, mode: Stdio) -> ()
def spawn(self) -> Result[Child, IoError]
def command_bytes(program: &list[u8]) -> Result[Command, IoError]
def command(program: &str) -> Result[Command, IoError]
def shell(script: &str) -> Result[Command, IoError]
class Child
def id(self) -> i64
def take_stdin(self: &mut Child) -> Option[File]
def take_stdout(self: &mut Child) -> Option[File]
def take_stderr(self: &mut Child) -> Option[File]
def close_stdin(self: &mut Child) -> Result[(), IoError]
def wait(self: &mut Child) -> Result[ExitStatus, IoError]
def try_wait(self: &mut Child) -> Result[Option[ExitStatus], IoError]
def wait_timeout(self: &mut Child, budget: &Duration, cancel: &CancellationToken) -> Result[Option[ExitStatus], WaitError]
def wait_before(self: &mut Child, deadline: &Deadline, cancel: &CancellationToken) -> Result[Option[ExitStatus], WaitError]
def signal(self, number: i32) -> Result[(), IoError]
def terminate(self) -> Result[(), IoError]
def kill(self) -> Result[(), IoError]
def communicate(self: &mut Child, input: &[u8], limit: NonNegative[i64]) -> Result[Output, StreamError]
def output(self: &mut Child, limit: NonNegative[i64]) -> Result[Output, StreamError]
enum WaitError
Io(IoError) | Cancelled
std.random
Caller-owned seeded generators with unbiased ranges, shuffling, and index sampling; not for secrets. Errors: RandomError. Reference.
enum RandomError
EmptyRange | ZeroState
class Rng
def next_u64(self: &mut Rng) -> u64
def next_u32(self: &mut Rng) -> u32
def next_bool(self: &mut Rng) -> bool
def next_f64(self: &mut Rng) -> f64
def below(self: &mut Rng, bound: u64) -> Result[u64, RandomError]
def between(self: &mut Rng, low: i64, high: i64) -> Result[i64, RandomError]
def index(self: &mut Rng, count: i64) -> Result[i64, RandomError]
def shuffle[T](self: &mut Rng, items: &mut list[T]) -> ()
def sample_indices(self: &mut Rng, count: i64, amount: i64) -> Result[list[i64], AllocError]
def fill(self: &mut Rng, target: &mut Bytes, start: i64, stop: i64) -> ()
def state(self) -> (u64, u64, u64, u64)
def seeded(seed: u64) -> Rng
def from_state(a: u64, b: u64, c: u64, d: u64) -> Result[Rng, RandomError]
std.ready
Readiness waits on pipes and other handles, bounded by a deadline and a cancellation token. Errors: WaitError. Reference.
enum Interest
Read | Write | ReadWrite
class Readiness
readable: bool
writable: bool
hangup: bool
error: bool
enum WaitError
TimedOut | Cancelled | Stream(StreamError)
def wait_now(file: &File, interest: Interest) -> Result[Readiness, WaitError]
def wait_until(file: &File, interest: Interest, deadline: &Deadline, cancel: &CancellationToken) -> Result[Readiness, WaitError]
def wait(file: &File, interest: Interest, deadline: &Deadline, cancel: &CancellationToken) -> Result[Readiness, WaitError]
def read_before(stream: &mut FileStream, buffer: &mut Bytes, start: i64, stop: i64, deadline: &Deadline, cancel: &CancellationToken) -> Result[i64, WaitError]
def write_before(stream: &mut FileStream, source: &[u8], deadline: &Deadline, cancel: &CancellationToken) -> Result[i64, WaitError]
protocol Watchable
def descriptor(self) -> &File
class Event
key: u64
readiness: Readiness
class Selector
def watch(self: &mut Selector, file: &File, key: u64, interest: Interest) -> Result[(), WaitError]
def unwatch(self: &mut Selector, file: &File) -> Result[(), WaitError]
def wait(self: &mut Selector, deadline: &Deadline, cancel: &CancellationToken) -> Result[i64, WaitError]
def event(self, index: i64) -> Result[Event, WaitError]
def close(self: &mut Selector) -> Result[(), WaitError]
def selector(capacity: i64) -> Result[Selector, WaitError]
std.regex
Linear-time regular expressions over text and bytes: search, captures, iteration, split, and replacement under explicit pattern, memory, and work limits. Errors: RegexError. Reference.
enum Limit
PatternBytes | Nesting | Repetition | Instructions | Memory | Work | Output | Matches
class SyntaxError
offset: i64
message: str
enum RegexError
Syntax(SyntaxError) | LimitExceeded(Limit) | OutOfRange | Allocation(AllocError)
class Limits
max_pattern_bytes: NonNegative[i64]
max_nesting: NonNegative[i64]
max_repeat: NonNegative[i64]
max_instructions: NonNegative[i64]
max_memory: NonNegative[i64]
max_work: NonNegative[i64]
def limits() -> Limits
class Match
start: i64
end: i64
def len(self) -> Result[i64, OverflowError[i64]]
def is_empty(self) -> bool
def text(self, haystack: &str) -> Result[str, RegexError]
def bytes(self, data: &[u8]) -> Result[Bytes, RegexError]
class Captures
def len(self) -> i64
def get(self, index: i64) -> Option[Match]
def name(self, name: &str) -> Option[Match]
class Cursor
class Regex
def group_count(self) -> i64
def group_index(self, name: &str) -> Option[i64]
def instructions(self) -> i64
def is_match(self, text: &str) -> Result[bool, RegexError]
def is_match_bytes(self, data: &[u8]) -> Result[bool, RegexError]
def find(self, text: &str) -> Result[Option[Match], RegexError]
def find_at(self, text: &str, start: i64) -> Result[Option[Match], RegexError]
def find_bytes(self, data: &[u8], start: i64) -> Result[Option[Match], RegexError]
def captures(self, text: &str) -> Result[Option[Captures], RegexError]
def captures_at(self, text: &str, start: i64) -> Result[Option[Captures], RegexError]
def captures_bytes(self, data: &[u8], start: i64) -> Result[Option[Captures], RegexError]
def cursor(self) -> Result[Cursor, RegexError]
def next_match(self, text: &str, cursor: &mut Cursor) -> Result[Option[Match], RegexError]
def next_match_bytes(self, data: &[u8], cursor: &mut Cursor) -> Result[Option[Match], RegexError]
def find_all(self, text: &str, max_matches: i64) -> Result[list[Match], RegexError]
def find_all_bytes(self, data: &[u8], max_matches: i64) -> Result[list[Match], RegexError]
def split(self, text: &str, max_pieces: i64) -> Result[list[str], RegexError]
def replace(self, text: &str, template: &str, max_bytes: i64) -> Result[str, RegexError]
def replace_all(self, text: &str, template: &str, max_bytes: i64) -> Result[str, RegexError]
def compile(pattern: &str, bounds: &Limits) -> Result[Regex, RegexError]
std.report
Bounded human-readable report rendering with std.fmt. Errors: FormatError. Reference.
def render[T: Formattable, W](value: &T, output: &mut Formatter[W]) -> Result[(), FormatError]
def to_string[T: Formattable](value: &T, max_bytes: NonNegative[i64], max_depth: NonNegative[i64]) -> Result[str, FormatError]
std.secrets
Operating-system entropy: secret bytes, hex and base64url tokens, and unbiased integers, with no fallback source. Errors: SecretsError. Reference.
enum SecretsError
OutOfRange | EmptyRange | Entropy(StreamError) | Allocation(AllocError)
class OsEntropy
def read(self: &mut OsEntropy, buffer: &mut Bytes, start: i64, stop: i64) -> Result[i64, StreamError]
def os_entropy() -> OsEntropy
class Source[R]
inner: R
def fill(self: &mut Source[R], target: &mut Bytes, start: i64, stop: i64) -> Result[(), SecretsError]
def token_bytes(self: &mut Source[R], count: i64) -> Result[Bytes, SecretsError]
def token_hex(self: &mut Source[R], count: i64) -> Result[str, SecretsError]
def token_urlsafe(self: &mut Source[R], count: i64) -> Result[str, SecretsError]
def next_u64(self: &mut Source[R]) -> Result[u64, SecretsError]
def below(self: &mut Source[R], bound: u64) -> Result[u64, SecretsError]
def source_from[R: Reader](inner: R) -> Source[R]
def os() -> Source[OsEntropy]
def fill(target: &mut Bytes, start: i64, stop: i64) -> Result[(), SecretsError]
def token_bytes(count: i64) -> Result[Bytes, SecretsError]
def token_hex(count: i64) -> Result[str, SecretsError]
def token_urlsafe(count: i64) -> Result[str, SecretsError]
def random_u64() -> Result[u64, SecretsError]
def below(bound: u64) -> Result[u64, SecretsError]
std.shlex
Shell-style word splitting and POSIX argument quoting, without execution or expansion. Errors: ShlexError, AllocError. Reference.
enum ShlexError
UnclosedQuote(i64) | TrailingEscape(i64) | Nul(i64) | Allocation(AllocError)
def split(text: &str, comments: bool = False) -> Result[list[str], ShlexError]
def quote(text: &str) -> Result[str, ShlexError]
std.sketch
Bounded distinct counts, sampled quantiles and frequent-item bounds with versioned checkpoints. Errors: SketchError, AllocError, BytesError. Reference.
enum SketchError
InvalidArgument | InvalidData | Incompatible | Empty | Limit | ConstructionFailed | Allocation(AllocError)
def hash_u64(key: u64, seed: u64 = 0u64) -> u64
class HyperLogLog
def precision(self) -> i64
def add(self: &mut HyperLogLog, key: u64) -> ()
def estimate(self) -> f64
def relative_standard_error(self) -> f64
def merge(self: &mut HyperLogLog, other: &HyperLogLog) -> Result[(), SketchError]
def to_bytes(self) -> Result[Bytes, AllocError]
def hyperloglog(precision: i64 = 12, seed: u64 = 0u64) -> Result[HyperLogLog, SketchError]
def hyperloglog_from_bytes(data: &[u8], max_precision: i64 = 18) -> Result[HyperLogLog, SketchError]
class Quantiles
def count(self) -> u64
def retained(self) -> i64
def add(self: &mut Quantiles, value: f64) -> Result[(), SketchError]
def quantile(self, fraction: f64) -> Result[f64, SketchError]
def to_bytes(self) -> Result[Bytes, AllocError]
def quantiles(capacity: Positive[i64] = 1024, seed: u64 = 0u64) -> Result[Quantiles, SketchError]
def quantiles_from_bytes(data: &[u8], max_capacity: Positive[i64] = 1048576) -> Result[Quantiles, SketchError]
class Frequency
key: u64
lower: u64
upper: u64
class FrequentItems
def count(self) -> u64
def error_bound(self) -> u64
def add(self: &mut FrequentItems, key: u64) -> Result[(), SketchError]
def frequency(self, key: u64) -> Frequency
def candidates(self) -> Result[list[Frequency], AllocError]
def to_bytes(self) -> Result[Bytes, AllocError]
def frequent_items(capacity: Positive[i64] = 128) -> Result[FrequentItems, SketchError]
def frequent_items_from_bytes(data: &[u8], max_capacity: Positive[i64] = 65536) -> Result[FrequentItems, SketchError]
std.sketch.xor
Static approximate membership over numeric identities with bounded construction. Errors: SketchError, AllocError, BytesError. Reference.
class XorFilter
def len(self) -> i64
def contains(self, key: u64) -> bool
def to_bytes(self) -> Result[Bytes, AllocError]
def build(keys: &list[u64], max_keys: NonNegative[i64], seed: u64 = 0u64) -> Result[XorFilter, SketchError]
def from_bytes(data: &[u8], max_keys: NonNegative[i64]) -> Result[XorFilter, SketchError]
std.sketch.cuckoo
Scalable approximate membership with bounded growth and receipt-directed deletion. Errors: SketchError, AllocError, BytesError. Reference.
class CuckooFilter
def len(self) -> u64
def layers(self) -> i64
def buckets(self) -> i64
def contains(self, key: u64) -> bool
def insert(self: &mut CuckooFilter, key: u64) -> Result[i64, SketchError]
def remove(self: &mut CuckooFilter, key: u64, layer_index: i64) -> Result[bool, SketchError]
def to_bytes(self) -> Result[Bytes, AllocError]
def cuckoo(initial_buckets: i64 = 64, max_buckets: Positive[i64] = 1048576, seed: u64 = 0u64) -> Result[CuckooFilter, SketchError]
def from_bytes(data: &[u8], max_buckets: Positive[i64]) -> Result[CuckooFilter, SketchError]
std.signal
SIGTERM and SIGINT as an event to wait or select on, and a helper that cancels a token on one. Errors: SignalError, WaitError. Reference.
enum Signal
Interrupt | Terminate
enum SignalError
AlreadyListening | Stream(StreamError)
class Signals
def wait(self: &mut Signals, deadline: &Deadline, cancel: &CancellationToken) -> Result[Signal, WaitError]
def received(self: &mut Signals) -> Result[Option[Signal], WaitError]
def watch(self, selector: &mut Selector, key: u64) -> Result[(), WaitError]
def unwatch(self, selector: &mut Selector) -> Result[(), WaitError]
def close(self: &mut Signals) -> Result[(), WaitError]
def listen() -> Result[Signals, SignalError]
def cancel_on(signals: Signals, stop: CancellationToken) -> Result[Option[Signal], WaitError]
std.simd
Portable fixed-lane values, masks, checked and wrapping arithmetic, selection, and bounded list loads/stores. Reference.
enum i32x4
Lanes(i32, i32, i32, i32)
def lane(self, index: i64) -> Option[i32]
def to_tuple(self) -> (i32, i32, i32, i32)
def add(self, other: &i32x4) -> Result[i32x4, OverflowError[i32]]
def sub(self, other: &i32x4) -> Result[i32x4, OverflowError[i32]]
def mul(self, other: &i32x4) -> Result[i32x4, OverflowError[i32]]
def wrapping_add(self, other: &i32x4) -> i32x4
def wrapping_sub(self, other: &i32x4) -> i32x4
def wrapping_mul(self, other: &i32x4) -> i32x4
def simd_eq(self, other: &i32x4) -> mask32x4
def simd_ne(self, other: &i32x4) -> mask32x4
def simd_lt(self, other: &i32x4) -> mask32x4
def simd_le(self, other: &i32x4) -> mask32x4
def simd_gt(self, other: &i32x4) -> mask32x4
def simd_ge(self, other: &i32x4) -> mask32x4
def wrapping_sum(self) -> i32
def store(self, values: &mut list[i32], offset: i64) -> bool
def splat_i32x4(value: i32) -> i32x4
def load_i32x4(values: &list[i32], offset: i64 = 0) -> Option[i32x4]
enum u32x4
Lanes(u32, u32, u32, u32)
def lane(self, index: i64) -> Option[u32]
def to_tuple(self) -> (u32, u32, u32, u32)
def add(self, other: &u32x4) -> Result[u32x4, OverflowError[u32]]
def sub(self, other: &u32x4) -> Result[u32x4, OverflowError[u32]]
def mul(self, other: &u32x4) -> Result[u32x4, OverflowError[u32]]
def wrapping_add(self, other: &u32x4) -> u32x4
def wrapping_sub(self, other: &u32x4) -> u32x4
def wrapping_mul(self, other: &u32x4) -> u32x4
def simd_eq(self, other: &u32x4) -> mask32x4
def simd_ne(self, other: &u32x4) -> mask32x4
def simd_lt(self, other: &u32x4) -> mask32x4
def simd_le(self, other: &u32x4) -> mask32x4
def simd_gt(self, other: &u32x4) -> mask32x4
def simd_ge(self, other: &u32x4) -> mask32x4
def wrapping_sum(self) -> u32
def store(self, values: &mut list[u32], offset: i64) -> bool
def splat_u32x4(value: u32) -> u32x4
def load_u32x4(values: &list[u32], offset: i64 = 0) -> Option[u32x4]
enum f32x4
Lanes(f32, f32, f32, f32)
def lane(self, index: i64) -> Option[f32]
def to_tuple(self) -> (f32, f32, f32, f32)
def add(self, other: &f32x4) -> f32x4
def sub(self, other: &f32x4) -> f32x4
def mul(self, other: &f32x4) -> f32x4
def div(self, other: &f32x4) -> f32x4
def simd_eq(self, other: &f32x4) -> mask32x4
def simd_ne(self, other: &f32x4) -> mask32x4
def simd_lt(self, other: &f32x4) -> mask32x4
def simd_le(self, other: &f32x4) -> mask32x4
def simd_gt(self, other: &f32x4) -> mask32x4
def simd_ge(self, other: &f32x4) -> mask32x4
def sum_ordered(self, initial: f32) -> f32
def store(self, values: &mut list[f32], offset: i64) -> bool
def splat_f32x4(value: f32) -> f32x4
def load_f32x4(values: &list[f32], offset: i64 = 0) -> Option[f32x4]
enum f64x2
Lanes(f64, f64)
def lane(self, index: i64) -> Option[f64]
def to_tuple(self) -> (f64, f64)
def add(self, other: &f64x2) -> f64x2
def sub(self, other: &f64x2) -> f64x2
def mul(self, other: &f64x2) -> f64x2
def div(self, other: &f64x2) -> f64x2
def simd_eq(self, other: &f64x2) -> mask64x2
def simd_ne(self, other: &f64x2) -> mask64x2
def simd_lt(self, other: &f64x2) -> mask64x2
def simd_le(self, other: &f64x2) -> mask64x2
def simd_gt(self, other: &f64x2) -> mask64x2
def simd_ge(self, other: &f64x2) -> mask64x2
def sum_ordered(self, initial: f64) -> f64
def store(self, values: &mut list[f64], offset: i64) -> bool
def splat_f64x2(value: f64) -> f64x2
def load_f64x2(values: &list[f64], offset: i64 = 0) -> Option[f64x2]
enum mask32x4
Lanes(bool, bool, bool, bool)
def any(self) -> bool
def all(self) -> bool
def count(self) -> i64
def select[T](self, yes: &T, no: &T) -> T
def invert(self) -> mask32x4
def and_mask(self, other: &mask32x4) -> mask32x4
def or_mask(self, other: &mask32x4) -> mask32x4
def xor_mask(self, other: &mask32x4) -> mask32x4
enum mask64x2
Lanes(bool, bool)
def any(self) -> bool
def all(self) -> bool
def count(self) -> i64
def select[T](self, yes: &T, no: &T) -> T
def invert(self) -> mask64x2
def and_mask(self, other: &mask64x2) -> mask64x2
def or_mask(self, other: &mask64x2) -> mask64x2
def xor_mask(self, other: &mask64x2) -> mask64x2
std.stat
Descriptive f64 statistics, inclusive quantiles, and normal, uniform and exponential distributions with explicit RNG sampling. Errors: StatError. Reference.
enum StatError
Empty | InsufficientData | NonFinite | Domain | Overflow | Allocation(AllocError)
def mean(values: &list[f64]) -> Result[f64, StatError]
def pstdev(values: &list[f64]) -> Result[f64, StatError]
def stdev(values: &list[f64]) -> Result[f64, StatError]
def pvariance(values: &list[f64]) -> Result[f64, StatError]
def variance(values: &list[f64]) -> Result[f64, StatError]
def quantile(values: &list[f64], p: f64) -> Result[f64, StatError]
def median(values: &list[f64]) -> Result[f64, StatError]
def quantiles(values: &list[f64], n: Positive[i64] = 4) -> Result[list[f64], StatError]
class Uniform
def pdf(self, x: f64) -> Result[f64, StatError]
def cdf(self, x: f64) -> Result[f64, StatError]
def inverse_cdf(self, p: f64) -> Result[f64, StatError]
def sample(self, rng: &mut random.Rng) -> f64
def uniform(low: f64 = 0.0, high: f64 = 1.0) -> Result[Uniform, StatError]
class Exponential
def pdf(self, x: f64) -> Result[f64, StatError]
def cdf(self, x: f64) -> Result[f64, StatError]
def inverse_cdf(self, p: f64) -> Result[f64, StatError]
def sample(self, rng: &mut random.Rng) -> Result[f64, StatError]
def exponential(rate: f64 = 1.0) -> Result[Exponential, StatError]
class Normal
def pdf(self, x: f64) -> Result[f64, StatError]
def cdf(self, x: f64) -> Result[f64, StatError]
def inverse_cdf(self, p: f64) -> Result[f64, StatError]
def sample(self, rng: &mut random.Rng) -> Result[f64, StatError]
def normal(mean: f64 = 0.0, sigma: f64 = 1.0) -> Result[Normal, StatError]
std.plot
Layered charts from owned columns or record accessors, validated mappings, grouped statistics, and deterministic streaming SVG. Errors: PlotError. Reference.
enum PlotError
Allocation(AllocError) | Write(i64, StreamError) | Io(StreamError) | UnknownColumn(str) | DuplicateColumn(str) | ShapeMismatch(str) | MissingAesthetic(str) | WrongKind(str) | Conflict(str) | InvalidSetting(str)
enum Column
Num(list[f64]) | Int(list[i64]) | Text(list[str]) | Bool(list[bool]) | MaybeNum(list[Option[f64]]) | MaybeText(list[Option[str]])
class Table
def share(self) -> Table
def height(self) -> i64
def names(self) -> Result[list[str], PlotError]
def table(values: list[(str, Column)]) -> Result[Table, PlotError]
def columns(values: &dict[str, list[f64]]) -> Result[Table, PlotError]
enum Field[T]
Num(str, BoxedCallable[[&T], f64]) | Int(str, BoxedCallable[[&T], i64]) | Text(str, BoxedCallable[[&T], str]) | Bool(str, BoxedCallable[[&T], bool])
def num[T, F: Callable[[&T], f64]](name: str, read: F) -> Result[Field[T], PlotError]
def integer[T, F: Callable[[&T], i64]](name: str, read: F) -> Result[Field[T], PlotError]
def text[T, F: Callable[[&T], str]](name: str, read: F) -> Result[Field[T], PlotError]
def flag[T, F: Callable[[&T], bool]](name: str, read: F) -> Result[Field[T], PlotError]
def rows[T](values: &list[T], fields: list[Field[T]]) -> Result[Table, PlotError]
class Factor
def factor(name: str) -> Factor
class Mapping
def aes(x: str | Factor = "", y: str | Factor = "", colour: str | Factor = "", color: str | Factor = "", fill: str | Factor = "", group: str = "", ymin: str = "", ymax: str = "") -> Mapping
enum Position
Identity | Stack | Dodge | Fill
def position_identity() -> Position
def position_stack() -> Position
def position_dodge() -> Position
def position_fill() -> Position
enum Units
In | Cm | Mm | Px
enum Legend
Right | Hidden
class Layer
class Labels
class Theme
enum Component
Layer(Layer) | Labels(Labels) | Theme(Theme) | Cartesian(Option[(f64, f64)], Option[(f64, f64)]) | Flip | ColourScale(bool, ColourScale) | PositionScale(bool, PositionScale) | Facet(Facet)
enum Scales
Fixed | Free | FreeX | FreeY
class Facet
def facet_wrap(facets: str, nrow: i64 = 0, ncol: i64 = 0, scales: Scales = Scales.Fixed) -> Component
def facet_grid(rows: str = "", cols: str = "", scales: Scales = Scales.Fixed) -> Component
enum ColourScale
Default | Manual(dict[str, str]) | Gradient(str, str)
def scale_colour_manual(values: dict[str, str]) -> Component
def scale_color_manual(values: dict[str, str]) -> Component
def scale_fill_manual(values: dict[str, str]) -> Component
def scale_colour_gradient(low: str = "#132B43", high: str = "#56B1F7") -> Component
def scale_color_gradient(low: str = "#132B43", high: str = "#56B1F7") -> Component
def scale_fill_gradient(low: str = "#132B43", high: str = "#56B1F7") -> Component
enum Unset
Unset
enum Transform
Identity | Log10 | Sqrt | Reverse
class PositionScale
def scale_x_continuous(name: str = "", breaks: Option[list[f64]] = Nothing, labels: Option[list[str]] = Nothing, limits: (f64, f64) | Unset = Unset.Unset, transform: Transform = Transform.Identity) -> Component
def scale_y_continuous(name: str = "", breaks: Option[list[f64]] = Nothing, labels: Option[list[str]] = Nothing, limits: (f64, f64) | Unset = Unset.Unset, transform: Transform = Transform.Identity) -> Component
def scale_x_discrete(name: str = "", limits: Option[list[str]] = Nothing, labels: Option[list[str]] = Nothing) -> Component
def scale_y_discrete(name: str = "", limits: Option[list[str]] = Nothing, labels: Option[list[str]] = Nothing) -> Component
def scale_x_log10(name: str = "", breaks: Option[list[f64]] = Nothing, labels: Option[list[str]] = Nothing, limits: (f64, f64) | Unset = Unset.Unset) -> Component
def scale_y_log10(name: str = "", breaks: Option[list[f64]] = Nothing, labels: Option[list[str]] = Nothing, limits: (f64, f64) | Unset = Unset.Unset) -> Component
def scale_x_sqrt(name: str = "", breaks: Option[list[f64]] = Nothing, labels: Option[list[str]] = Nothing, limits: (f64, f64) | Unset = Unset.Unset) -> Component
def scale_x_reverse(name: str = "", breaks: Option[list[f64]] = Nothing, labels: Option[list[str]] = Nothing, limits: (f64, f64) | Unset = Unset.Unset) -> Component
def scale_y_sqrt(name: str = "", breaks: Option[list[f64]] = Nothing, labels: Option[list[str]] = Nothing, limits: (f64, f64) | Unset = Unset.Unset) -> Component
def scale_y_reverse(name: str = "", breaks: Option[list[f64]] = Nothing, labels: Option[list[str]] = Nothing, limits: (f64, f64) | Unset = Unset.Unset) -> Component
def geom_point(colour: str = "", color: str = "", size: f64 | i64 = 1.5, alpha: f64 | i64 = 1.0, na_rm: bool = False, data: Table | Unset = Unset.Unset, mapping: Mapping | Unset = Unset.Unset, inherit_aes: bool = True) -> Component
def geom_line(colour: str = "", color: str = "", linewidth: f64 | i64 = 0.5, alpha: f64 | i64 = 1.0, na_rm: bool = False, data: Table | Unset = Unset.Unset, mapping: Mapping | Unset = Unset.Unset, inherit_aes: bool = True) -> Component
def geom_path(colour: str = "", color: str = "", linewidth: f64 | i64 = 0.5, alpha: f64 | i64 = 1.0, na_rm: bool = False, data: Table | Unset = Unset.Unset, mapping: Mapping | Unset = Unset.Unset, inherit_aes: bool = True) -> Component
def geom_col(colour: str = "", color: str = "", fill: str = "", width: f64 | i64 = 0.9, alpha: f64 | i64 = 1.0, position: Position = Position.Stack, na_rm: bool = False, data: Table | Unset = Unset.Unset, mapping: Mapping | Unset = Unset.Unset, inherit_aes: bool = True) -> Component
def geom_bar(colour: str = "", color: str = "", fill: str = "", width: f64 | i64 = 0.9, alpha: f64 | i64 = 1.0, position: Position = Position.Stack, na_rm: bool = False, data: Table | Unset = Unset.Unset, mapping: Mapping | Unset = Unset.Unset, inherit_aes: bool = True) -> Component
def geom_histogram(colour: str = "", color: str = "", fill: str = "", bins: i64 = 30, alpha: f64 | i64 = 1.0, position: Position = Position.Stack, na_rm: bool = False, data: Table | Unset = Unset.Unset, mapping: Mapping | Unset = Unset.Unset, inherit_aes: bool = True) -> Component
def geom_errorbar(colour: str = "", color: str = "", linewidth: f64 | i64 = 0.5, width: f64 | i64 = 0.2, alpha: f64 | i64 = 1.0, na_rm: bool = False, data: Table | Unset = Unset.Unset, mapping: Mapping | Unset = Unset.Unset, inherit_aes: bool = True) -> Component
def geom_linerange(colour: str = "", color: str = "", linewidth: f64 | i64 = 0.5, alpha: f64 | i64 = 1.0, na_rm: bool = False, data: Table | Unset = Unset.Unset, mapping: Mapping | Unset = Unset.Unset, inherit_aes: bool = True) -> Component
def geom_ribbon(colour: str = "", color: str = "", fill: str = "", linewidth: f64 | i64 = 0.5, alpha: f64 | i64 = 0.3, na_rm: bool = False, data: Table | Unset = Unset.Unset, mapping: Mapping | Unset = Unset.Unset, inherit_aes: bool = True) -> Component
def labs(title: str = "", x: str = "", y: str = "", colour: str = "", fill: str = "") -> Component
def theme_grey(base_size: f64 | i64 = 11, legend_position: Legend = Legend.Right) -> Component
def theme_minimal(base_size: f64 | i64 = 11, legend_position: Legend = Legend.Right) -> Component
def theme_classic(base_size: f64 | i64 = 11, legend_position: Legend = Legend.Right) -> Component
def coord_flip() -> Component
def coord_cartesian(xlim: (f64, f64) | Unset = Unset.Unset, ylim: (f64, f64) | Unset = Unset.Unset) -> Component
class Plot
def plus(self, components: list[Component]) -> Result[Plot, PlotError]
def write_svg[W: Writer](self, writer: &mut W, width: f64 = 6.0, height: f64 = 4.0, units: Units = Units.In, dpi: f64 = 300.0) -> Result[list[str], PlotError]
def save(self, path: &str, width: f64 = 6.0, height: f64 = 4.0, units: Units = Units.In, dpi: f64 = 300.0) -> Result[list[str], PlotError]
def ggplot(data: Table, mapping: Mapping, components: list[Component]) -> Result[Plot, PlotError]
std.linalg
Dense f64 matrices, products, solves, LU, QR, Cholesky, SVD and eigen decompositions through an optional faer engine. Errors: AllocError, LinalgError. Reference.
enum LinalgError
Shape | Bounds | NonFinite | Singular | NotSymmetric | NotPositiveDefinite | NoConvergence | Overflow | NativePanic | Allocation(AllocError)
class Matrix
def rows(self) -> i64
def cols(self) -> i64
def get(self, row: i64, col: i64) -> Result[f64, LinalgError]
def set(self: &mut Matrix, row: i64, col: i64, value: f64) -> Result[(), LinalgError]
def to_list(self) -> Result[list[f64], AllocError]
def transpose(self) -> Result[Matrix, AllocError]
def add(self, rhs: &Matrix) -> Result[Matrix, LinalgError]
def subtract(self, rhs: &Matrix) -> Result[Matrix, LinalgError]
def scale(self, factor: f64) -> Result[Matrix, LinalgError]
def negate(self) -> Result[Matrix, LinalgError]
def __add__(self, rhs: &Matrix) -> Result[Matrix, LinalgError]
def __sub__(self, rhs: &Matrix) -> Result[Matrix, LinalgError]
def __neg__(self) -> Result[Matrix, LinalgError]
def __mul__(self, rhs: &Matrix) -> Result[Matrix, LinalgError]
def __rmul__(self, factor: f64) -> Result[Matrix, LinalgError]
def matmul(self, rhs: &Matrix) -> Result[Matrix, LinalgError]
def lu(self) -> Result[Lu, LinalgError]
def solve(self, rhs: &Matrix) -> Result[Matrix, LinalgError]
def qr(self) -> Result[(Matrix, Matrix), LinalgError]
def cholesky(self) -> Result[Matrix, LinalgError]
def svd(self) -> Result[(Matrix, list[f64], Matrix), LinalgError]
def eigh(self) -> Result[(list[f64], Matrix), LinalgError]
def eig(self) -> Result[(list[f64], list[f64], Matrix, Matrix), LinalgError]
class Lu
def lower(self) -> Result[Matrix, LinalgError]
def upper(self) -> Result[Matrix, LinalgError]
def permutation(self) -> Result[list[i64], AllocError]
def solve(self, rhs: &Matrix) -> Result[Matrix, LinalgError]
def matrix(rows: i64, cols: i64, values: &list[f64]) -> Result[Matrix, LinalgError]
def vector(values: &list[f64]) -> Result[Matrix, LinalgError]
def zeros(rows: i64, cols: i64) -> Result[Matrix, LinalgError]
def identity(n: i64) -> Result[Matrix, LinalgError]
def diagonal(values: &list[f64]) -> Result[Matrix, LinalgError]
std.sync
Shared cooperative cancellation tokens. Errors: AllocError. Reference.
Compiler-provided: CancellationToken. Their signatures are on the reference page.
class Mutex[T]
def __init__(self, value: T) -> ()
enum WaitError
TimedOut | Cancelled | Disconnected
enum SendBeforeError[T]
TimedOut(T) | Cancelled(T) | Disconnected(T)
def recv_before[T](receiver: &Receiver[T], deadline: &Deadline, cancel: &CancellationToken) -> Result[T, WaitError]
def send_before[T](sender: &Sender[T], value: T, deadline: &Deadline, cancel: &CancellationToken) -> Result[(), SendBeforeError[T]]
def future_before[T](future: &Future[T], deadline: &Deadline, cancel: &CancellationToken) -> Result[(), WaitError]
def cancelled_before(token: &CancellationToken, deadline: &Deadline, cancel: &CancellationToken) -> Result[(), WaitError]
def select_recv_before[A, B](first: &Receiver[A], second: &Receiver[B], deadline: &Deadline, cancel: &CancellationToken) -> Result[Selected[A, B], WaitError]
std.system
Bounded Linux procfs snapshots: CPUs, memory, load, uptime, block devices, disk I/O and process status. Errors: IoError, SystemError. Reference.
enum SystemError
Io(IoError) | Stream(StreamError) | Allocation(AllocError) | Limit | Malformed(str)
class Memory
total_bytes: u64
free_bytes: u64
available_bytes: Option[u64]
swap_total_bytes: u64
swap_free_bytes: u64
class LoadAverage
one_minute: f64
five_minutes: f64
fifteen_minutes: f64
runnable_tasks: u64
total_tasks: u64
last_pid: u32
class Uptime
seconds: f64
idle_cpu_seconds: f64
class BlockDevice
major: u32
minor: u32
name: str
size_bytes: u64
class DiskIo
major: u32
minor: u32
name: str
reads_completed: u64
bytes_read: u64
writes_completed: u64
bytes_written: u64
in_flight: u64
busy_milliseconds: u64
class Process
pid: u32
parent_pid: u32
name: str
state: str
real_uid: u32
threads: u64
virtual_bytes: Option[u64]
resident_bytes: Option[u64]
class Procfs
def cpu_count(self, max_bytes: Positive[i64] = 1048576) -> Result[i64, SystemError]
def memory(self, max_bytes: Positive[i64] = 65536) -> Result[Memory, SystemError]
def load_average(self, max_bytes: Positive[i64] = 4096) -> Result[LoadAverage, SystemError]
def uptime(self, max_bytes: Positive[i64] = 4096) -> Result[Uptime, SystemError]
def block_devices(self, max_bytes: Positive[i64] = 1048576, max_devices: NonNegative[i64] = 4096) -> Result[list[BlockDevice], SystemError]
def disk_io(self, max_bytes: Positive[i64] = 1048576, max_devices: NonNegative[i64] = 4096) -> Result[list[DiskIo], SystemError]
def process_ids(self, max_entries: NonNegative[i64] = 65536) -> Result[list[u32], SystemError]
def process(self, pid: u32, max_bytes: Positive[i64] = 65536) -> Result[Process, SystemError]
def open_procfs(path: &Path) -> Result[Procfs, IoError]
def local() -> Result[Procfs, IoError]
std.tempfile
Unnamed and named temporary files, recursively cleaned directories, context managers and explicit ownership transfer. Errors: IoError, FsError, StreamError, AllocError. Reference.
def default_dir() -> Result[Path, IoError]
class TemporaryFile
def stream(self: &mut TemporaryFile) -> &mut FileStream
def close(self: &mut TemporaryFile) -> Result[(), StreamError]
def __enter__(self: &mut TemporaryFile) -> &mut FileStream
def __exit__(self: &mut TemporaryFile) -> ()
def file(parent: &Path = default_dir()?) -> Result[TemporaryFile, IoError]
class NamedTemporaryFile
def path(self) -> &Path
def read(self: &mut NamedTemporaryFile, buffer: &mut Bytes, start: i64, stop: i64) -> Result[i64, StreamError]
def write(self: &mut NamedTemporaryFile, source: &[u8]) -> Result[i64, StreamError]
def flush(self: &mut NamedTemporaryFile) -> Result[(), StreamError]
def sync(self: &mut NamedTemporaryFile) -> Result[(), StreamError]
def seek(self: &mut NamedTemporaryFile, target: SeekFrom) -> Result[i64, StreamError]
def close(self: &mut NamedTemporaryFile) -> Result[(), FsError]
def keep(self: &mut NamedTemporaryFile) -> Result[Path, AllocError]
def __enter__(self: &mut NamedTemporaryFile) -> &mut NamedTemporaryFile
def __exit__(self: &mut NamedTemporaryFile) -> ()
def named_file(parent: &Path = io_error(default_dir())?, prefix: &str = "plenty-", suffix: &str = "") -> Result[NamedTemporaryFile, FsError]
class TemporaryDirectory
def path(self) -> &Path
def root(self) -> &Root
def cleanup(self: &mut TemporaryDirectory) -> Result[(), FsError]
def keep(self: &mut TemporaryDirectory) -> Result[Path, AllocError]
def __enter__(self: &mut TemporaryDirectory) -> &mut TemporaryDirectory
def __exit__(self: &mut TemporaryDirectory) -> ()
def directory(parent: &Path = io_error(default_dir())?, prefix: &str = "plenty-", max_entries: Positive[i64] = 100000, max_depth: Positive[i64] = 64) -> Result[TemporaryDirectory, FsError]
std.test
Assertions that explain their failure on standard error, self-removing scratch directories, the seeded generator for plenty test, and fake clocks and streams. Errors: TestError. Reference.
enum TestError
Assertion | Fixture(FsError) | Allocation(AllocError) | InvalidSeed | Unavailable
def fail(message: &str) -> Result[(), TestError]
def check(condition: bool, message: &str) -> Result[(), TestError]
def equal[T](left: &T, right: &T) -> Result[(), TestError]
def not_equal[T](left: &T, right: &T) -> Result[(), TestError]
def less[T](left: &T, right: &T) -> Result[(), TestError]
def close(left: f64, right: f64, tolerance: f64) -> Result[(), TestError]
def is_ok[T, E](result: &Result[T, E]) -> Result[(), TestError]
def is_err[T, E](result: &Result[T, E]) -> Result[(), TestError]
def fail_allocations_after(count: u32) -> Result[(), TestError]
def restore_allocations() -> Result[(), TestError]
def fail_files_after(count: u32, code: i32) -> Result[(), TestError]
def restore_files() -> Result[(), TestError]
class Scratch
def path(self) -> &Path
def root(self) -> &Root
def file(self, name: &str) -> Result[str, TestError]
def cleanup(self: &mut Scratch) -> Result[(), TestError]
def scratch() -> Result[Scratch, TestError]
def seed() -> Result[u64, TestError]
def rng() -> Result[Rng, TestError]
class FakeClock
def now(self) -> Instant
def advance(self: &mut FakeClock, span: &Duration) -> Result[(), RangeError]
def set(self: &mut FakeClock, nanoseconds_after_origin: u64) -> ()
def fake_clock(nanoseconds_after_origin: u64) -> FakeClock
class FakeStream
input: Bytes
output: Bytes
def calls(self) -> i64
def limit(self: &mut FakeStream, bytes: i64) -> ()
def fail_after(self: &mut FakeStream, after: i64, error: StreamError) -> Result[(), OverflowError[i64]]
def read(self: &mut FakeStream, buffer: &mut Bytes, start: i64, stop: i64) -> Result[i64, StreamError]
def write(self: &mut FakeStream, source: &[u8]) -> Result[i64, StreamError]
def flush(self: &mut FakeStream) -> Result[(), StreamError]
def fake_stream(input: Bytes) -> Result[FakeStream, AllocError]
std.text
Bounded builders, joining, scalar/word wrapping, shortening, and indentation. Errors: TextError. Reference.
def find_in(text: &str, set: &[u8]) -> Option[i64]
def find_not_in(text: &str, set: &[u8]) -> Option[i64]
enum TextError
InvalidLimit | LimitExceeded | Allocation(AllocError) | Bytes(BytesError)
class Builder
def byte_len(self) -> i64
def limit(self) -> i64
def append(self: &mut Builder, text: &str) -> Result[(), TextError]
def clear(self: &mut Builder) -> ()
def to_text(self) -> Result[str, TextError]
def finish(self: Builder) -> Result[str, TextError]
def builder(max_bytes: NonNegative[i64]) -> Result[Builder, TextError]
def join(pieces: &list[str], separator: &str, max_bytes: NonNegative[i64]) -> Result[str, TextError]
def hard_wrap(text: &str, width: Positive[i64], max_bytes: NonNegative[i64]) -> Result[str, TextError]
class TextWrapper
width: Positive[i64]
initial_indent: str
subsequent_indent: str
expand_tabs: bool
replace_whitespace: bool
fix_sentence_endings: bool
break_long_words: bool
drop_whitespace: bool
break_on_hyphens: bool
tabsize: NonNegative[i64]
max_lines: Option[Positive[i64]]
placeholder: str
max_bytes: NonNegative[i64]
max_input_bytes: NonNegative[i64]
def wrap(self, text: &str) -> Result[list[str], TextError]
def fill(self, text: &str) -> Result[str, TextError]
def text_wrapper(width: Positive[i64] = 70, max_bytes: NonNegative[i64] = 1048576, max_input_bytes: NonNegative[i64] = 1048576) -> TextWrapper
def wrap(text: &str, width: Positive[i64] = 70, max_bytes: NonNegative[i64] = 1048576, max_input_bytes: NonNegative[i64] = 1048576) -> Result[list[str], TextError]
def fill(text: &str, width: Positive[i64] = 70, max_bytes: NonNegative[i64] = 1048576, max_input_bytes: NonNegative[i64] = 1048576) -> Result[str, TextError]
def shorten(text: &str, width: Positive[i64], placeholder: &str = " [...]", max_bytes: NonNegative[i64] = 1048576, max_input_bytes: NonNegative[i64] = 1048576, fix_sentence_endings: bool = False, break_long_words: bool = True, break_on_hyphens: bool = True) -> Result[str, TextError]
def dedent(text: &str, max_bytes: NonNegative[i64] = 1048576, max_input_bytes: NonNegative[i64] = 1048576) -> Result[str, TextError]
def indent(text: &str, prefix: &str, max_bytes: NonNegative[i64] = 1048576, max_input_bytes: NonNegative[i64] = 1048576) -> Result[str, TextError]
def indent_with[F: Callable[[&str], bool]](text: &str, prefix: &str, predicate: F, max_bytes: NonNegative[i64] = 1048576, max_input_bytes: NonNegative[i64] = 1048576) -> Result[str, TextError]
std.thread
Explicit scoped thread and owned-job imports. Errors: ThreadError, SpawnError[F]. Reference.
Compiler-provided: spawn, ThreadError, SpawnError. Their signatures are on the reference page.
std.time
Checked durations, monotonic instants and deadlines, wall timestamps, and sleep. Errors: RangeError. Reference.
class Duration
def as_nanoseconds(self) -> u64
def as_microseconds(self) -> u64
def as_milliseconds(self) -> u64
def as_seconds(self) -> u64
def as_timeout_ms(self) -> u64
def is_zero(self) -> bool
def checked_add(self, other: &Duration) -> Result[Duration, RangeError]
def checked_sub(self, other: &Duration) -> Result[Duration, RangeError]
def saturating_sub(self, other: &Duration) -> Duration
def checked_mul(self, factor: u64) -> Result[Duration, RangeError]
def nanoseconds(count: u64) -> Duration
def microseconds(count: u32) -> Duration
def milliseconds(count: u32) -> Duration
def seconds(count: u32) -> Duration
class Instant
def checked_add(self, span: &Duration) -> Result[Instant, RangeError]
def duration_since(self, earlier: &Instant) -> Result[Duration, RangeError]
def saturating_duration_since(self, earlier: &Instant) -> Duration
def monotonic_now() -> Instant
def instant_at(nanoseconds_after_origin: u64) -> Instant
protocol Clock
def now(self) -> Instant
class MonotonicClock
def now(self) -> Instant
def monotonic_clock() -> MonotonicClock
class Deadline
def expired(self, now: &Instant) -> bool
def remaining(self, now: &Instant) -> Duration
def deadline_at(at: &Instant) -> Deadline
def deadline_after(start: &Instant, budget: &Duration) -> Result[Deadline, RangeError]
class Timestamp
seconds: i64
nanoseconds: u32
def duration_since(self, earlier: &Timestamp) -> Result[Duration, RangeError]
def checked_add(self, span: &Duration) -> Result[Timestamp, RangeError]
def wall_now() -> Timestamp
def sleep(span: &Duration) -> ()
def sleep_until(deadline: &Deadline) -> ()
std.tls
TLS 1.2 and 1.3 client and server streams over std.net TCP, with verified certificates, client certificates, ALPN, deadlines, and cancellation. Errors: TlsError. Reference.
enum TlsError
Net(NetError) | UntrustedCertificate | ExpiredCertificate | WrongHost | InvalidCertificate | Rejected | Protocol | Truncated | InvalidName | InvalidConfiguration | NoTrustRoots | Closed | Allocation(AllocError) | RevokedCertificate | CertificateRequired | Incompatible
enum TlsVersion
Tls12 | Tls13
enum CipherSuite
Tls13Aes256GcmSha384 | Tls13Aes128GcmSha256 | Tls13Chacha20Poly1305Sha256 | EcdheEcdsaAes256GcmSha384 | EcdheEcdsaAes128GcmSha256 | EcdheEcdsaChacha20Poly1305Sha256 | EcdheRsaAes256GcmSha384 | EcdheRsaAes128GcmSha256 | EcdheRsaChacha20Poly1305Sha256
enum ClientCertificates
Required | Optional
class TrustRoots
def len(self) -> i64
def roots_from_pem(pem: &str) -> Result[TrustRoots, TlsError]
def system_roots() -> Result[TrustRoots, TlsError]
class ClientConfig
class ServerConfig
class ClientBuilder
def certificate(self: &mut ClientBuilder, chain_pem: &str, key_pem: &str) -> Result[(), TlsError]
def application_protocol(self: &mut ClientBuilder, name: &str) -> Result[(), TlsError]
def minimum_version(self: &mut ClientBuilder, version: TlsVersion) -> ()
def cipher_suite(self: &mut ClientBuilder, suite: CipherSuite) -> ()
def revocation_list(self: &mut ClientBuilder, crl_pem: &str) -> Result[(), TlsError]
def build(self) -> Result[ClientConfig, TlsError]
def client_builder(roots: &TrustRoots) -> Result[ClientBuilder, TlsError]
def client_config(roots: &TrustRoots) -> Result[ClientConfig, TlsError]
class ServerBuilder
def certificate(self: &mut ServerBuilder, chain_pem: &str, key_pem: &str) -> Result[(), TlsError]
def verify_clients(self: &mut ServerBuilder, roots: &TrustRoots, certificates: ClientCertificates) -> ()
def application_protocol(self: &mut ServerBuilder, name: &str) -> Result[(), TlsError]
def minimum_version(self: &mut ServerBuilder, version: TlsVersion) -> ()
def cipher_suite(self: &mut ServerBuilder, suite: CipherSuite) -> ()
def revocation_list(self: &mut ServerBuilder, crl_pem: &str) -> Result[(), TlsError]
def build(self) -> Result[ServerConfig, TlsError]
def server_builder(chain_pem: &str, key_pem: &str) -> Result[ServerBuilder, TlsError]
def server_config(chain_pem: &str, key_pem: &str) -> Result[ServerConfig, TlsError]
class TlsStream
def handshake_before(self: &mut TlsStream, deadline: &Deadline, cancel: &CancellationToken) -> Result[(), TlsError]
def try_handshake(self: &mut TlsStream) -> Result[bool, TlsError]
def try_flush(self: &mut TlsStream) -> Result[bool, TlsError]
def try_read(self: &mut TlsStream, buffer: &mut Bytes, start: i64, stop: i64) -> Result[Option[i64], TlsError]
def try_write(self: &mut TlsStream, source: &[u8]) -> Result[Option[i64], TlsError]
def interest(self) -> Interest
def watch(self, selector: &mut Selector, key: u64, interest: Interest) -> Result[(), TlsError]
def unwatch(self, selector: &mut Selector) -> Result[(), TlsError]
def read_before(self: &mut TlsStream, buffer: &mut Bytes, start: i64, stop: i64, deadline: &Deadline, cancel: &CancellationToken) -> Result[i64, TlsError]
def write_before(self: &mut TlsStream, source: &[u8], deadline: &Deadline, cancel: &CancellationToken) -> Result[i64, TlsError]
def write_all_before(self: &mut TlsStream, source: &[u8], deadline: &Deadline, cancel: &CancellationToken) -> Result[(), TlsError]
def close_before(self: &mut TlsStream, deadline: &Deadline, cancel: &CancellationToken) -> Result[(), TlsError]
def shutdown(self: &mut TlsStream, direction: Shutdown) -> Result[(), TlsError]
def read(self: &mut TlsStream, buffer: &mut Bytes, start: i64, stop: i64) -> Result[i64, StreamError]
def write(self: &mut TlsStream, source: &[u8]) -> Result[i64, StreamError]
def flush(self: &mut TlsStream) -> Result[(), StreamError]
def close(self: &mut TlsStream) -> Result[(), StreamError]
def application_protocol(self) -> Result[Option[str], TlsError]
def server_name(self) -> Result[Option[str], TlsError]
def peer_certificate(self) -> Result[Option[Bytes], TlsError]
def version(self) -> Option[TlsVersion]
def cipher_suite(self) -> Option[CipherSuite]
def start_connect(tcp: TcpStream, config: &ClientConfig, server_name: &str) -> Result[TlsStream, TlsError]
def start_accept(tcp: TcpStream, config: &ServerConfig) -> Result[TlsStream, TlsError]
def connect(tcp: TcpStream, config: &ClientConfig, server_name: &str, deadline: &Deadline, cancel: &CancellationToken) -> Result[TlsStream, TlsError]
def accept(tcp: TcpStream, config: &ServerConfig, deadline: &Deadline, cancel: &CancellationToken) -> Result[TlsStream, TlsError]
class Session
def feed(self: &mut Session, source: &[u8]) -> Result[i64, TlsError]
def feed_end(self: &mut Session) -> Result[(), TlsError]
def take(self: &mut Session, target: &mut Bytes, start: i64, stop: i64) -> Result[i64, TlsError]
def read(self: &mut Session, target: &mut Bytes, start: i64, stop: i64) -> Result[Option[i64], TlsError]
def write(self: &mut Session, source: &[u8]) -> Result[i64, TlsError]
def close(self: &mut Session) -> ()
def handshaking(self) -> bool
def wants_write(self) -> bool
def wants_read(self) -> bool
def application_protocol(self) -> Result[Option[str], TlsError]
def server_name(self) -> Result[Option[str], TlsError]
def peer_certificate(self) -> Result[Option[Bytes], TlsError]
def version(self) -> Option[TlsVersion]
def cipher_suite(self) -> Option[CipherSuite]
def client_session(config: &ClientConfig, server_name: &str) -> Result[Session, TlsError]
def server_session(config: &ServerConfig) -> Result[Session, TlsError]
std.toml
Strict TOML 1.0 parsing, value-tree writing and format-preserving document edits, under depth, input, and output limits. Errors: TomlError. Reference.
enum TomlFault
UnexpectedByte | UnexpectedEnd | InvalidUtf8 | ControlCharacter | InvalidEscape | InvalidNumber | NumberOutOfRange | InvalidDateTime | DuplicateKey | DuplicateTable | KeyConflict | DepthExceeded | InputTooLarge | OutputTooLarge | NotATable | Stream(StreamError) | Allocation(AllocError)
class TomlError
fault: TomlFault
offset: i64
line: i64
column: i64
class Limits
max_depth: NonNegative[i64]
max_input: NonNegative[i64]
max_output: NonNegative[i64]
enum DateTimeKind
OffsetDateTime | LocalDateTime | LocalDate | LocalTime
class DateTime
kind: DateTimeKind
year: i64
month: i64
day: i64
hour: i64
minute: i64
second: i64
nanosecond: i64
offset_minutes: i64
enum Toml
Bool(bool) | Int(i64) | Float(f64) | Text(str) | DateTime(DateTime) | Array(list[Toml]) | Table(list[Entry])
class Entry
key: str
value: Toml
def limits() -> Limits
def parse(source: &[u8], bounds: &Limits) -> Result[Toml, TomlError]
def parse_text(text: &str, bounds: &Limits) -> Result[Toml, TomlError]
def read_value[R: Reader](reader: &mut R, bounds: &Limits) -> Result[Toml, TomlError]
def find(entries: &list[Entry], key: &str) -> Option[i64]
def encode(value: &Toml, bounds: &Limits) -> Result[Bytes, TomlError]
def encode_text(value: &Toml, bounds: &Limits) -> Result[str, TomlError]
def write_value[W: Writer](writer: &mut W, value: &Toml, bounds: &Limits) -> Result[(), TomlError]
class Document
def text(self) -> Result[str, TomlError]
def get(self, path: &list[str]) -> Result[Option[Toml], TomlError]
def set(self: &mut Document, path: &list[str], value: &Toml) -> Result[(), TomlError]
def insert(self: &mut Document, path: &list[str], value: &Toml) -> Result[(), TomlError]
def remove(self: &mut Document, path: &list[str]) -> Result[bool, TomlError]
def document(source: &[u8], bounds: &Limits) -> Result[Document, TomlError]
def document_text(text: &str, bounds: &Limits) -> Result[Document, TomlError]
std.unicode
Unicode 17 normalization, combining classes, and Mark classification. Errors: TextError. Reference.
def version() -> str
def combining_class(scalar: &str) -> Option[u32]
def is_mark(scalar: &str) -> Option[bool]
def nfd(text: &str, max_scalars: NonNegative[i64]) -> Result[str, TextError]
def nfc(text: &str, max_scalars: NonNegative[i64]) -> Result[str, TextError]
def nfkd(text: &str, max_scalars: NonNegative[i64]) -> Result[str, TextError]
def nfkc(text: &str, max_scalars: NonNegative[i64]) -> Result[str, TextError]
std.units
SI dimensions, prefixed and common units, physical constants, and dimension-aware square roots, sums and trigonometry over Quantity. Errors: MathError, RangeError. Reference.
dimension Mass("kg")
dimension Length("m")
dimension Time("s")
dimension Current("A")
dimension Temperature("K")
dimension Amount("mol")
dimension Luminosity("cd")
dimension Area = Pow[Length, 2]
dimension Volume = Pow[Length, 3]
dimension Wavenumber = One / Length
dimension Velocity = Length / Time
dimension Acceleration = Length / Pow[Time, 2]
dimension Frequency = One / Time
dimension Force = Mass * Acceleration
dimension Pressure = Force / Area
dimension Energy = Force * Length
dimension Power = Energy / Time
dimension Momentum = Mass * Velocity
dimension Action = Energy * Time
dimension Density = Mass / Volume
dimension MassFlowRate = Mass / Time
dimension VolumeFlowRate = Volume / Time
dimension DynamicViscosity = Pressure * Time
dimension KinematicViscosity = Area / Time
dimension SurfaceTension = Force / Length
dimension HeatFlux = Power / Area
dimension SpecificEnergy = Energy / Mass
dimension Entropy = Energy / Temperature
dimension SpecificHeatCapacity = SpecificEnergy / Temperature
dimension ThermalConductivity = Power / (Length * Temperature)
dimension MolarMass = Mass / Amount
dimension MolarEnergy = Energy / Amount
dimension MolarHeatCapacity = Entropy / Amount
dimension Concentration = Amount / Volume
dimension PerAmount = One / Amount
dimension CatalyticActivity = Amount / Time
dimension Charge = Current * Time
dimension Voltage = Power / Current
dimension ElectricField = Voltage / Length
dimension Capacitance = Charge / Voltage
dimension Permittivity = Capacitance / Length
dimension Resistance = Voltage / Current
dimension Conductance = Current / Voltage
dimension MagneticFlux = Voltage * Time
dimension MagneticFluxDensity = MagneticFlux / Area
dimension Inductance = MagneticFlux / Current
dimension Permeability = Inductance / Length
dimension Illuminance = Luminosity / Area
dimension LuminousEfficacy = Luminosity / Power
dimension StefanBoltzmann = HeatFlux / Pow[Temperature, 4]
dimension Gravitation = Volume / (Mass * Pow[Time, 2])
const kg: Quantity[f64, Mass] = Quantity[f64, Mass](1.0)
const m: Quantity[f64, Length] = Quantity[f64, Length](1.0)
const s: Quantity[f64, Time] = Quantity[f64, Time](1.0)
const A: Quantity[f64, Current] = Quantity[f64, Current](1.0)
const K: Quantity[f64, Temperature] = Quantity[f64, Temperature](1.0)
const mol: Quantity[f64, Amount] = Quantity[f64, Amount](1.0)
const cd: Quantity[f64, Luminosity] = Quantity[f64, Luminosity](1.0)
const rad: Quantity[f64, One] = Quantity[f64, One](1.0)
const sr: Quantity[f64, One] = Quantity[f64, One](1.0)
const degree: Quantity[f64, One] = (f64.PI / 180.0) * rad
const arcminute: Quantity[f64, One] = degree / 60.0
const arcsecond: Quantity[f64, One] = degree / 3600.0
const Hz: Quantity[f64, Frequency] = 1.0 / s
const N: Quantity[f64, Force] = kg * m / (s * s)
const Pa: Quantity[f64, Pressure] = N / (m * m)
const J: Quantity[f64, Energy] = N * m
const W: Quantity[f64, Power] = J / s
const C: Quantity[f64, Charge] = A * s
const V: Quantity[f64, Voltage] = W / A
const F: Quantity[f64, Capacitance] = C / V
const ohm: Quantity[f64, Resistance] = V / A
const S: Quantity[f64, Conductance] = A / V
const Wb: Quantity[f64, MagneticFlux] = V * s
const T: Quantity[f64, MagneticFluxDensity] = Wb / (m * m)
const H: Quantity[f64, Inductance] = Wb / A
const lm: Quantity[f64, Luminosity] = cd * sr
const lx: Quantity[f64, Illuminance] = lm / (m * m)
const Bq: Quantity[f64, Frequency] = 1.0 / s
const Gy: Quantity[f64, SpecificEnergy] = J / kg
const Sv: Quantity[f64, SpecificEnergy] = J / kg
const kat: Quantity[f64, CatalyticActivity] = mol / s
const ug: Quantity[f64, Mass] = 1e-9 * kg
const mg: Quantity[f64, Mass] = 1e-6 * kg
const g: Quantity[f64, Mass] = 1e-3 * kg
const pm: Quantity[f64, Length] = 1e-12 * m
const nm: Quantity[f64, Length] = 1e-9 * m
const um: Quantity[f64, Length] = 1e-6 * m
const mm: Quantity[f64, Length] = 1e-3 * m
const cm: Quantity[f64, Length] = 1e-2 * m
const km: Quantity[f64, Length] = 1e3 * m
const ps: Quantity[f64, Time] = 1e-12 * s
const ns: Quantity[f64, Time] = 1e-9 * s
const us: Quantity[f64, Time] = 1e-6 * s
const ms: Quantity[f64, Time] = 1e-3 * s
const uA: Quantity[f64, Current] = 1e-6 * A
const mA: Quantity[f64, Current] = 1e-3 * A
const kA: Quantity[f64, Current] = 1e3 * A
const mK: Quantity[f64, Temperature] = 1e-3 * K
const umol: Quantity[f64, Amount] = 1e-6 * mol
const mmol: Quantity[f64, Amount] = 1e-3 * mol
const kmol: Quantity[f64, Amount] = 1e3 * mol
const kHz: Quantity[f64, Frequency] = 1e3 * Hz
const MHz: Quantity[f64, Frequency] = 1e6 * Hz
const GHz: Quantity[f64, Frequency] = 1e9 * Hz
const mN: Quantity[f64, Force] = 1e-3 * N
const kN: Quantity[f64, Force] = 1e3 * N
const MN: Quantity[f64, Force] = 1e6 * N
const hPa: Quantity[f64, Pressure] = 1e2 * Pa
const kPa: Quantity[f64, Pressure] = 1e3 * Pa
const MPa: Quantity[f64, Pressure] = 1e6 * Pa
const GPa: Quantity[f64, Pressure] = 1e9 * Pa
const mJ: Quantity[f64, Energy] = 1e-3 * J
const kJ: Quantity[f64, Energy] = 1e3 * J
const MJ: Quantity[f64, Energy] = 1e6 * J
const GJ: Quantity[f64, Energy] = 1e9 * J
const uW: Quantity[f64, Power] = 1e-6 * W
const mW: Quantity[f64, Power] = 1e-3 * W
const kW: Quantity[f64, Power] = 1e3 * W
const MW: Quantity[f64, Power] = 1e6 * W
const GW: Quantity[f64, Power] = 1e9 * W
const Wh: Quantity[f64, Energy] = 3600.0 * J
const kWh: Quantity[f64, Energy] = 3.6e6 * J
const MWh: Quantity[f64, Energy] = 3.6e9 * J
const GWh: Quantity[f64, Energy] = 3.6e12 * J
const nC: Quantity[f64, Charge] = 1e-9 * C
const uC: Quantity[f64, Charge] = 1e-6 * C
const mC: Quantity[f64, Charge] = 1e-3 * C
const uV: Quantity[f64, Voltage] = 1e-6 * V
const mV: Quantity[f64, Voltage] = 1e-3 * V
const kV: Quantity[f64, Voltage] = 1e3 * V
const MV: Quantity[f64, Voltage] = 1e6 * V
const pF: Quantity[f64, Capacitance] = 1e-12 * F
const nF: Quantity[f64, Capacitance] = 1e-9 * F
const uF: Quantity[f64, Capacitance] = 1e-6 * F
const mF: Quantity[f64, Capacitance] = 1e-3 * F
const kohm: Quantity[f64, Resistance] = 1e3 * ohm
const Mohm: Quantity[f64, Resistance] = 1e6 * ohm
const uS: Quantity[f64, Conductance] = 1e-6 * S
const mS: Quantity[f64, Conductance] = 1e-3 * S
const uT: Quantity[f64, MagneticFluxDensity] = 1e-6 * T
const mT: Quantity[f64, MagneticFluxDensity] = 1e-3 * T
const nH: Quantity[f64, Inductance] = 1e-9 * H
const uH: Quantity[f64, Inductance] = 1e-6 * H
const mH: Quantity[f64, Inductance] = 1e-3 * H
const delta_nu_Cs: Quantity[f64, Frequency] = 9192631770.0 * Hz
const c: Quantity[f64, Velocity] = 299792458.0 * (m / s)
const h: Quantity[f64, Action] = 6.62607015e-34 * (J * s)
const e: Quantity[f64, Charge] = 1.602176634e-19 * C
const k_B: Quantity[f64, Entropy] = 1.380649e-23 * (J / K)
const N_A: Quantity[f64, PerAmount] = 6.02214076e23 / mol
const K_cd: Quantity[f64, LuminousEfficacy] = 683.0 * (lm / W)
const hbar: Quantity[f64, Action] = h / f64.TAU
const R: Quantity[f64, MolarHeatCapacity] = N_A * k_B
const sigma: Quantity[f64, StefanBoltzmann] = 5.670374419e-8 * (W / (m * m * K * K * K * K))
const g_n: Quantity[f64, Acceleration] = 9.80665 * (m / (s * s))
const atm: Quantity[f64, Pressure] = 101325.0 * Pa
const G: Quantity[f64, Gravitation] = 6.67430e-11 * (m * m * m / (kg * s * s))
const m_e: Quantity[f64, Mass] = 9.1093837139e-31 * kg
const m_p: Quantity[f64, Mass] = 1.67262192595e-27 * kg
const m_u: Quantity[f64, Mass] = 1.66053906892e-27 * kg
const epsilon_0: Quantity[f64, Permittivity] = 8.8541878188e-12 * (F / m)
const mu_0: Quantity[f64, Permeability] = 1.25663706127e-6 * (H / m)
const minute: Quantity[f64, Time] = 60.0 * s
const hour: Quantity[f64, Time] = 3600.0 * s
const day: Quantity[f64, Time] = 86400.0 * s
const hectare: Quantity[f64, Area] = 1e4 * (m * m)
const litre: Quantity[f64, Volume] = 1e-3 * (m * m * m)
const millilitre: Quantity[f64, Volume] = 1e-6 * (m * m * m)
const cubic_metre: Quantity[f64, Volume] = m * m * m
const tonne: Quantity[f64, Mass] = 1e3 * kg
const dalton: Quantity[f64, Mass] = m_u
const eV: Quantity[f64, Energy] = e * V
const keV: Quantity[f64, Energy] = 1e3 * eV
const MeV: Quantity[f64, Energy] = 1e6 * eV
const GeV: Quantity[f64, Energy] = 1e9 * eV
const au: Quantity[f64, Length] = 149597870700.0 * m
const bar: Quantity[f64, Pressure] = 1e5 * Pa
const mbar: Quantity[f64, Pressure] = 1e2 * Pa
const rankine: Quantity[f64, Temperature] = K * 5.0 / 9.0
const inch: Quantity[f64, Length] = 0.0254 * m
const foot: Quantity[f64, Length] = 0.3048 * m
const yard: Quantity[f64, Length] = 0.9144 * m
const mile: Quantity[f64, Length] = 1609.344 * m
const nautical_mile: Quantity[f64, Length] = 1852.0 * m
const angstrom: Quantity[f64, Length] = 1e-10 * m
const light_year: Quantity[f64, Length] = 9460730472580800.0 * m
const parsec: Quantity[f64, Length] = (648000.0 / f64.PI) * au
const acre: Quantity[f64, Area] = 4046.8564224 * (m * m)
const gallon: Quantity[f64, Volume] = 3.785411784e-3 * (m * m * m)
const pound: Quantity[f64, Mass] = 0.45359237 * kg
const ounce: Quantity[f64, Mass] = pound / 16.0
const pound_force: Quantity[f64, Force] = 0.45359237 * 9.80665 * N
const knot: Quantity[f64, Velocity] = nautical_mile / hour
const km_per_hour: Quantity[f64, Velocity] = km / hour
const mph: Quantity[f64, Velocity] = mile / hour
const psi: Quantity[f64, Pressure] = pound_force / (inch * inch)
const torr: Quantity[f64, Pressure] = atm / 760.0
const mmHg: Quantity[f64, Pressure] = 133.322387415 * Pa
const calorie: Quantity[f64, Energy] = 4.184 * J
const kilocalorie: Quantity[f64, Energy] = 4184.0 * J
const btu: Quantity[f64, Energy] = 1055.05585262 * J
const horsepower: Quantity[f64, Power] = 550.0 * foot * pound_force / s
const week: Quantity[f64, Time] = 7.0 * day
const year: Quantity[f64, Time] = 365.0 * day
def abs[T: IntType, D: Dimension](x: Quantity[T, D]) -> Result[Quantity[T, D], OverflowError[T]]
def abs_float[T: FloatType, D: Dimension](x: Quantity[T, D]) -> Quantity[T, D]
def clamp[Num, D: Dimension](x: Quantity[Num, D], low: Quantity[Num, D], high: Quantity[Num, D]) -> Quantity[Num, D]
def sum[D: Dimension](items: &list[Quantity[f64, D]]) -> Quantity[f64, D]
def mean[D: Dimension](items: &list[Quantity[f64, D]]) -> Option[Quantity[f64, D]]
def wrap_list[D: Dimension](values: list[f64]) -> list[Quantity[f64, D]]
def strip_list[D: Dimension](items: list[Quantity[f64, D]]) -> list[f64]
def sqrt[D: Dimension](x: Quantity[f64, D]) -> Result[Quantity[f64, Root[D, 2]], MathError]
def cbrt[D: Dimension](x: Quantity[f64, D]) -> Result[Quantity[f64, Root[D, 3]], MathError]
def hypot[D: Dimension](a: Quantity[f64, D], b: Quantity[f64, D]) -> Quantity[f64, D]
def exp(x: Quantity[f64, One]) -> Result[Quantity[f64, One], MathError]
def log(x: Quantity[f64, One]) -> Result[Quantity[f64, One], MathError]
def log2(x: Quantity[f64, One]) -> Result[Quantity[f64, One], MathError]
def log10(x: Quantity[f64, One]) -> Result[Quantity[f64, One], MathError]
def sin(angle: Quantity[f64, One]) -> Result[Quantity[f64, One], MathError]
def cos(angle: Quantity[f64, One]) -> Result[Quantity[f64, One], MathError]
def tan(angle: Quantity[f64, One]) -> Result[Quantity[f64, One], MathError]
def asin(x: Quantity[f64, One]) -> Result[Quantity[f64, One], MathError]
def acos(x: Quantity[f64, One]) -> Result[Quantity[f64, One], MathError]
def atan(x: Quantity[f64, One]) -> Result[Quantity[f64, One], MathError]
def atan2[D: Dimension](y: Quantity[f64, D], x: Quantity[f64, D]) -> Result[Quantity[f64, One], MathError]
def from_celsius(degrees: f64) -> Quantity[f64, Temperature]
def to_celsius(t: Quantity[f64, Temperature]) -> f64
def from_fahrenheit(degrees: f64) -> Quantity[f64, Temperature]
def to_fahrenheit(t: Quantity[f64, Temperature]) -> f64
def from_duration(span: &Duration) -> Quantity[f64, Time]
def to_duration(t: Quantity[f64, Time]) -> Result[Duration, RangeError]
std.units.catalogue
Every SI prefix on every SI unit and the other named units of NIST SP 811. Reference.
dimension LinearDensity = u.Mass / u.Length
dimension ElectricDipoleMoment = u.Charge * u.Length
dimension MagneticFieldStrength = u.Current / u.Length
dimension Exposure = u.Charge / u.Mass
dimension Luminance = u.Luminosity / u.Area
const qg: Quantity[f64, u.Mass] = 1e-33 * u.kg
const rg: Quantity[f64, u.Mass] = 1e-30 * u.kg
const yg: Quantity[f64, u.Mass] = 1e-27 * u.kg
const zg: Quantity[f64, u.Mass] = 1e-24 * u.kg
const ag: Quantity[f64, u.Mass] = 1e-21 * u.kg
const fg: Quantity[f64, u.Mass] = 1e-18 * u.kg
const pg: Quantity[f64, u.Mass] = 1e-15 * u.kg
const ng: Quantity[f64, u.Mass] = 1e-12 * u.kg
const ug: Quantity[f64, u.Mass] = 1e-9 * u.kg
const mg: Quantity[f64, u.Mass] = 1e-6 * u.kg
const cg: Quantity[f64, u.Mass] = 1e-5 * u.kg
const dg: Quantity[f64, u.Mass] = 1e-4 * u.kg
const dag: Quantity[f64, u.Mass] = 1e-2 * u.kg
const hg: Quantity[f64, u.Mass] = 1e-1 * u.kg
const kg: Quantity[f64, u.Mass] = 1e0 * u.kg
const Mg: Quantity[f64, u.Mass] = 1e3 * u.kg
const Gg: Quantity[f64, u.Mass] = 1e6 * u.kg
const Tg: Quantity[f64, u.Mass] = 1e9 * u.kg
const Pg: Quantity[f64, u.Mass] = 1e12 * u.kg
const Eg: Quantity[f64, u.Mass] = 1e15 * u.kg
const Zg: Quantity[f64, u.Mass] = 1e18 * u.kg
const Yg: Quantity[f64, u.Mass] = 1e21 * u.kg
const Rg: Quantity[f64, u.Mass] = 1e24 * u.kg
const Qg: Quantity[f64, u.Mass] = 1e27 * u.kg
const qm: Quantity[f64, u.Length] = 1e-30 * u.m
const rm: Quantity[f64, u.Length] = 1e-27 * u.m
const ym: Quantity[f64, u.Length] = 1e-24 * u.m
const zm: Quantity[f64, u.Length] = 1e-21 * u.m
const am: Quantity[f64, u.Length] = 1e-18 * u.m
const fm: Quantity[f64, u.Length] = 1e-15 * u.m
const pm: Quantity[f64, u.Length] = 1e-12 * u.m
const nm: Quantity[f64, u.Length] = 1e-9 * u.m
const um: Quantity[f64, u.Length] = 1e-6 * u.m
const mm: Quantity[f64, u.Length] = 1e-3 * u.m
const cm: Quantity[f64, u.Length] = 1e-2 * u.m
const dm: Quantity[f64, u.Length] = 1e-1 * u.m
const dam: Quantity[f64, u.Length] = 1e1 * u.m
const hm: Quantity[f64, u.Length] = 1e2 * u.m
const km: Quantity[f64, u.Length] = 1e3 * u.m
const Mm: Quantity[f64, u.Length] = 1e6 * u.m
const Gm: Quantity[f64, u.Length] = 1e9 * u.m
const Tm: Quantity[f64, u.Length] = 1e12 * u.m
const Pm: Quantity[f64, u.Length] = 1e15 * u.m
const Em: Quantity[f64, u.Length] = 1e18 * u.m
const Zm: Quantity[f64, u.Length] = 1e21 * u.m
const Ym: Quantity[f64, u.Length] = 1e24 * u.m
const Rm: Quantity[f64, u.Length] = 1e27 * u.m
const Qm: Quantity[f64, u.Length] = 1e30 * u.m
const qs: Quantity[f64, u.Time] = 1e-30 * u.s
const rs: Quantity[f64, u.Time] = 1e-27 * u.s
const ys: Quantity[f64, u.Time] = 1e-24 * u.s
const zs: Quantity[f64, u.Time] = 1e-21 * u.s
const attosecond: Quantity[f64, u.Time] = 1e-18 * u.s
const fs: Quantity[f64, u.Time] = 1e-15 * u.s
const ps: Quantity[f64, u.Time] = 1e-12 * u.s
const ns: Quantity[f64, u.Time] = 1e-9 * u.s
const us: Quantity[f64, u.Time] = 1e-6 * u.s
const ms: Quantity[f64, u.Time] = 1e-3 * u.s
const cs: Quantity[f64, u.Time] = 1e-2 * u.s
const ds: Quantity[f64, u.Time] = 1e-1 * u.s
const das: Quantity[f64, u.Time] = 1e1 * u.s
const hs: Quantity[f64, u.Time] = 1e2 * u.s
const ks: Quantity[f64, u.Time] = 1e3 * u.s
const Ms: Quantity[f64, u.Time] = 1e6 * u.s
const Gs: Quantity[f64, u.Time] = 1e9 * u.s
const Ts: Quantity[f64, u.Time] = 1e12 * u.s
const Ps: Quantity[f64, u.Time] = 1e15 * u.s
const Es: Quantity[f64, u.Time] = 1e18 * u.s
const Zs: Quantity[f64, u.Time] = 1e21 * u.s
const Ys: Quantity[f64, u.Time] = 1e24 * u.s
const Rs: Quantity[f64, u.Time] = 1e27 * u.s
const Qs: Quantity[f64, u.Time] = 1e30 * u.s
const qA: Quantity[f64, u.Current] = 1e-30 * u.A
const rA: Quantity[f64, u.Current] = 1e-27 * u.A
const yA: Quantity[f64, u.Current] = 1e-24 * u.A
const zA: Quantity[f64, u.Current] = 1e-21 * u.A
const aA: Quantity[f64, u.Current] = 1e-18 * u.A
const fA: Quantity[f64, u.Current] = 1e-15 * u.A
const pA: Quantity[f64, u.Current] = 1e-12 * u.A
const nA: Quantity[f64, u.Current] = 1e-9 * u.A
const uA: Quantity[f64, u.Current] = 1e-6 * u.A
const mA: Quantity[f64, u.Current] = 1e-3 * u.A
const cA: Quantity[f64, u.Current] = 1e-2 * u.A
const dA: Quantity[f64, u.Current] = 1e-1 * u.A
const daA: Quantity[f64, u.Current] = 1e1 * u.A
const hA: Quantity[f64, u.Current] = 1e2 * u.A
const kA: Quantity[f64, u.Current] = 1e3 * u.A
const MA: Quantity[f64, u.Current] = 1e6 * u.A
const GA: Quantity[f64, u.Current] = 1e9 * u.A
const TA: Quantity[f64, u.Current] = 1e12 * u.A
const PA: Quantity[f64, u.Current] = 1e15 * u.A
const EA: Quantity[f64, u.Current] = 1e18 * u.A
const ZA: Quantity[f64, u.Current] = 1e21 * u.A
const YA: Quantity[f64, u.Current] = 1e24 * u.A
const RA: Quantity[f64, u.Current] = 1e27 * u.A
const QA: Quantity[f64, u.Current] = 1e30 * u.A
const qK: Quantity[f64, u.Temperature] = 1e-30 * u.K
const rK: Quantity[f64, u.Temperature] = 1e-27 * u.K
const yK: Quantity[f64, u.Temperature] = 1e-24 * u.K
const zK: Quantity[f64, u.Temperature] = 1e-21 * u.K
const aK: Quantity[f64, u.Temperature] = 1e-18 * u.K
const fK: Quantity[f64, u.Temperature] = 1e-15 * u.K
const pK: Quantity[f64, u.Temperature] = 1e-12 * u.K
const nK: Quantity[f64, u.Temperature] = 1e-9 * u.K
const uK: Quantity[f64, u.Temperature] = 1e-6 * u.K
const mK: Quantity[f64, u.Temperature] = 1e-3 * u.K
const cK: Quantity[f64, u.Temperature] = 1e-2 * u.K
const dK: Quantity[f64, u.Temperature] = 1e-1 * u.K
const daK: Quantity[f64, u.Temperature] = 1e1 * u.K
const hK: Quantity[f64, u.Temperature] = 1e2 * u.K
const kK: Quantity[f64, u.Temperature] = 1e3 * u.K
const MK: Quantity[f64, u.Temperature] = 1e6 * u.K
const GK: Quantity[f64, u.Temperature] = 1e9 * u.K
const TK: Quantity[f64, u.Temperature] = 1e12 * u.K
const PK: Quantity[f64, u.Temperature] = 1e15 * u.K
const EK: Quantity[f64, u.Temperature] = 1e18 * u.K
const ZK: Quantity[f64, u.Temperature] = 1e21 * u.K
const YK: Quantity[f64, u.Temperature] = 1e24 * u.K
const RK: Quantity[f64, u.Temperature] = 1e27 * u.K
const QK: Quantity[f64, u.Temperature] = 1e30 * u.K
const qmol: Quantity[f64, u.Amount] = 1e-30 * u.mol
const rmol: Quantity[f64, u.Amount] = 1e-27 * u.mol
const ymol: Quantity[f64, u.Amount] = 1e-24 * u.mol
const zmol: Quantity[f64, u.Amount] = 1e-21 * u.mol
const amol: Quantity[f64, u.Amount] = 1e-18 * u.mol
const fmol: Quantity[f64, u.Amount] = 1e-15 * u.mol
const pmol: Quantity[f64, u.Amount] = 1e-12 * u.mol
const nmol: Quantity[f64, u.Amount] = 1e-9 * u.mol
const umol: Quantity[f64, u.Amount] = 1e-6 * u.mol
const mmol: Quantity[f64, u.Amount] = 1e-3 * u.mol
const cmol: Quantity[f64, u.Amount] = 1e-2 * u.mol
const dmol: Quantity[f64, u.Amount] = 1e-1 * u.mol
const damol: Quantity[f64, u.Amount] = 1e1 * u.mol
const hmol: Quantity[f64, u.Amount] = 1e2 * u.mol
const kmol: Quantity[f64, u.Amount] = 1e3 * u.mol
const Mmol: Quantity[f64, u.Amount] = 1e6 * u.mol
const Gmol: Quantity[f64, u.Amount] = 1e9 * u.mol
const Tmol: Quantity[f64, u.Amount] = 1e12 * u.mol
const Pmol: Quantity[f64, u.Amount] = 1e15 * u.mol
const Emol: Quantity[f64, u.Amount] = 1e18 * u.mol
const Zmol: Quantity[f64, u.Amount] = 1e21 * u.mol
const Ymol: Quantity[f64, u.Amount] = 1e24 * u.mol
const Rmol: Quantity[f64, u.Amount] = 1e27 * u.mol
const Qmol: Quantity[f64, u.Amount] = 1e30 * u.mol
const qcd: Quantity[f64, u.Luminosity] = 1e-30 * u.cd
const rcd: Quantity[f64, u.Luminosity] = 1e-27 * u.cd
const ycd: Quantity[f64, u.Luminosity] = 1e-24 * u.cd
const zcd: Quantity[f64, u.Luminosity] = 1e-21 * u.cd
const acd: Quantity[f64, u.Luminosity] = 1e-18 * u.cd
const fcd: Quantity[f64, u.Luminosity] = 1e-15 * u.cd
const pcd: Quantity[f64, u.Luminosity] = 1e-12 * u.cd
const ncd: Quantity[f64, u.Luminosity] = 1e-9 * u.cd
const ucd: Quantity[f64, u.Luminosity] = 1e-6 * u.cd
const mcd: Quantity[f64, u.Luminosity] = 1e-3 * u.cd
const ccd: Quantity[f64, u.Luminosity] = 1e-2 * u.cd
const dcd: Quantity[f64, u.Luminosity] = 1e-1 * u.cd
const dacd: Quantity[f64, u.Luminosity] = 1e1 * u.cd
const hcd: Quantity[f64, u.Luminosity] = 1e2 * u.cd
const kcd: Quantity[f64, u.Luminosity] = 1e3 * u.cd
const Mcd: Quantity[f64, u.Luminosity] = 1e6 * u.cd
const Gcd: Quantity[f64, u.Luminosity] = 1e9 * u.cd
const Tcd: Quantity[f64, u.Luminosity] = 1e12 * u.cd
const Pcd: Quantity[f64, u.Luminosity] = 1e15 * u.cd
const Ecd: Quantity[f64, u.Luminosity] = 1e18 * u.cd
const Zcd: Quantity[f64, u.Luminosity] = 1e21 * u.cd
const Ycd: Quantity[f64, u.Luminosity] = 1e24 * u.cd
const Rcd: Quantity[f64, u.Luminosity] = 1e27 * u.cd
const Qcd: Quantity[f64, u.Luminosity] = 1e30 * u.cd
const qrad: Quantity[f64, One] = 1e-30 * u.rad
const rrad: Quantity[f64, One] = 1e-27 * u.rad
const yrad: Quantity[f64, One] = 1e-24 * u.rad
const zrad: Quantity[f64, One] = 1e-21 * u.rad
const arad: Quantity[f64, One] = 1e-18 * u.rad
const frad: Quantity[f64, One] = 1e-15 * u.rad
const prad: Quantity[f64, One] = 1e-12 * u.rad
const nrad: Quantity[f64, One] = 1e-9 * u.rad
const urad: Quantity[f64, One] = 1e-6 * u.rad
const mrad: Quantity[f64, One] = 1e-3 * u.rad
const crad: Quantity[f64, One] = 1e-2 * u.rad
const drad: Quantity[f64, One] = 1e-1 * u.rad
const darad: Quantity[f64, One] = 1e1 * u.rad
const hrad: Quantity[f64, One] = 1e2 * u.rad
const krad: Quantity[f64, One] = 1e3 * u.rad
const Mrad: Quantity[f64, One] = 1e6 * u.rad
const Grad: Quantity[f64, One] = 1e9 * u.rad
const Trad: Quantity[f64, One] = 1e12 * u.rad
const Prad: Quantity[f64, One] = 1e15 * u.rad
const Erad: Quantity[f64, One] = 1e18 * u.rad
const Zrad: Quantity[f64, One] = 1e21 * u.rad
const Yrad: Quantity[f64, One] = 1e24 * u.rad
const Rrad: Quantity[f64, One] = 1e27 * u.rad
const Qrad: Quantity[f64, One] = 1e30 * u.rad
const qHz: Quantity[f64, u.Frequency] = 1e-30 * u.Hz
const rHz: Quantity[f64, u.Frequency] = 1e-27 * u.Hz
const yHz: Quantity[f64, u.Frequency] = 1e-24 * u.Hz
const zHz: Quantity[f64, u.Frequency] = 1e-21 * u.Hz
const aHz: Quantity[f64, u.Frequency] = 1e-18 * u.Hz
const fHz: Quantity[f64, u.Frequency] = 1e-15 * u.Hz
const pHz: Quantity[f64, u.Frequency] = 1e-12 * u.Hz
const nHz: Quantity[f64, u.Frequency] = 1e-9 * u.Hz
const uHz: Quantity[f64, u.Frequency] = 1e-6 * u.Hz
const mHz: Quantity[f64, u.Frequency] = 1e-3 * u.Hz
const cHz: Quantity[f64, u.Frequency] = 1e-2 * u.Hz
const dHz: Quantity[f64, u.Frequency] = 1e-1 * u.Hz
const daHz: Quantity[f64, u.Frequency] = 1e1 * u.Hz
const hHz: Quantity[f64, u.Frequency] = 1e2 * u.Hz
const kHz: Quantity[f64, u.Frequency] = 1e3 * u.Hz
const MHz: Quantity[f64, u.Frequency] = 1e6 * u.Hz
const GHz: Quantity[f64, u.Frequency] = 1e9 * u.Hz
const THz: Quantity[f64, u.Frequency] = 1e12 * u.Hz
const PHz: Quantity[f64, u.Frequency] = 1e15 * u.Hz
const EHz: Quantity[f64, u.Frequency] = 1e18 * u.Hz
const ZHz: Quantity[f64, u.Frequency] = 1e21 * u.Hz
const YHz: Quantity[f64, u.Frequency] = 1e24 * u.Hz
const RHz: Quantity[f64, u.Frequency] = 1e27 * u.Hz
const QHz: Quantity[f64, u.Frequency] = 1e30 * u.Hz
const qN: Quantity[f64, u.Force] = 1e-30 * u.N
const rN: Quantity[f64, u.Force] = 1e-27 * u.N
const yN: Quantity[f64, u.Force] = 1e-24 * u.N
const zN: Quantity[f64, u.Force] = 1e-21 * u.N
const aN: Quantity[f64, u.Force] = 1e-18 * u.N
const fN: Quantity[f64, u.Force] = 1e-15 * u.N
const pN: Quantity[f64, u.Force] = 1e-12 * u.N
const nN: Quantity[f64, u.Force] = 1e-9 * u.N
const uN: Quantity[f64, u.Force] = 1e-6 * u.N
const mN: Quantity[f64, u.Force] = 1e-3 * u.N
const cN: Quantity[f64, u.Force] = 1e-2 * u.N
const dN: Quantity[f64, u.Force] = 1e-1 * u.N
const daN: Quantity[f64, u.Force] = 1e1 * u.N
const hN: Quantity[f64, u.Force] = 1e2 * u.N
const kN: Quantity[f64, u.Force] = 1e3 * u.N
const MN: Quantity[f64, u.Force] = 1e6 * u.N
const GN: Quantity[f64, u.Force] = 1e9 * u.N
const TN: Quantity[f64, u.Force] = 1e12 * u.N
const PN: Quantity[f64, u.Force] = 1e15 * u.N
const EN: Quantity[f64, u.Force] = 1e18 * u.N
const ZN: Quantity[f64, u.Force] = 1e21 * u.N
const YN: Quantity[f64, u.Force] = 1e24 * u.N
const RN: Quantity[f64, u.Force] = 1e27 * u.N
const QN: Quantity[f64, u.Force] = 1e30 * u.N
const qPa: Quantity[f64, u.Pressure] = 1e-30 * u.Pa
const rPa: Quantity[f64, u.Pressure] = 1e-27 * u.Pa
const yPa: Quantity[f64, u.Pressure] = 1e-24 * u.Pa
const zPa: Quantity[f64, u.Pressure] = 1e-21 * u.Pa
const aPa: Quantity[f64, u.Pressure] = 1e-18 * u.Pa
const fPa: Quantity[f64, u.Pressure] = 1e-15 * u.Pa
const pPa: Quantity[f64, u.Pressure] = 1e-12 * u.Pa
const nPa: Quantity[f64, u.Pressure] = 1e-9 * u.Pa
const uPa: Quantity[f64, u.Pressure] = 1e-6 * u.Pa
const mPa: Quantity[f64, u.Pressure] = 1e-3 * u.Pa
const cPa: Quantity[f64, u.Pressure] = 1e-2 * u.Pa
const dPa: Quantity[f64, u.Pressure] = 1e-1 * u.Pa
const daPa: Quantity[f64, u.Pressure] = 1e1 * u.Pa
const hPa: Quantity[f64, u.Pressure] = 1e2 * u.Pa
const kPa: Quantity[f64, u.Pressure] = 1e3 * u.Pa
const MPa: Quantity[f64, u.Pressure] = 1e6 * u.Pa
const GPa: Quantity[f64, u.Pressure] = 1e9 * u.Pa
const TPa: Quantity[f64, u.Pressure] = 1e12 * u.Pa
const PPa: Quantity[f64, u.Pressure] = 1e15 * u.Pa
const EPa: Quantity[f64, u.Pressure] = 1e18 * u.Pa
const ZPa: Quantity[f64, u.Pressure] = 1e21 * u.Pa
const YPa: Quantity[f64, u.Pressure] = 1e24 * u.Pa
const RPa: Quantity[f64, u.Pressure] = 1e27 * u.Pa
const QPa: Quantity[f64, u.Pressure] = 1e30 * u.Pa
const qJ: Quantity[f64, u.Energy] = 1e-30 * u.J
const rJ: Quantity[f64, u.Energy] = 1e-27 * u.J
const yJ: Quantity[f64, u.Energy] = 1e-24 * u.J
const zJ: Quantity[f64, u.Energy] = 1e-21 * u.J
const aJ: Quantity[f64, u.Energy] = 1e-18 * u.J
const fJ: Quantity[f64, u.Energy] = 1e-15 * u.J
const pJ: Quantity[f64, u.Energy] = 1e-12 * u.J
const nJ: Quantity[f64, u.Energy] = 1e-9 * u.J
const uJ: Quantity[f64, u.Energy] = 1e-6 * u.J
const mJ: Quantity[f64, u.Energy] = 1e-3 * u.J
const cJ: Quantity[f64, u.Energy] = 1e-2 * u.J
const dJ: Quantity[f64, u.Energy] = 1e-1 * u.J
const daJ: Quantity[f64, u.Energy] = 1e1 * u.J
const hJ: Quantity[f64, u.Energy] = 1e2 * u.J
const kJ: Quantity[f64, u.Energy] = 1e3 * u.J
const MJ: Quantity[f64, u.Energy] = 1e6 * u.J
const GJ: Quantity[f64, u.Energy] = 1e9 * u.J
const TJ: Quantity[f64, u.Energy] = 1e12 * u.J
const PJ: Quantity[f64, u.Energy] = 1e15 * u.J
const EJ: Quantity[f64, u.Energy] = 1e18 * u.J
const ZJ: Quantity[f64, u.Energy] = 1e21 * u.J
const YJ: Quantity[f64, u.Energy] = 1e24 * u.J
const RJ: Quantity[f64, u.Energy] = 1e27 * u.J
const QJ: Quantity[f64, u.Energy] = 1e30 * u.J
const qW: Quantity[f64, u.Power] = 1e-30 * u.W
const rW: Quantity[f64, u.Power] = 1e-27 * u.W
const yW: Quantity[f64, u.Power] = 1e-24 * u.W
const zW: Quantity[f64, u.Power] = 1e-21 * u.W
const aW: Quantity[f64, u.Power] = 1e-18 * u.W
const fW: Quantity[f64, u.Power] = 1e-15 * u.W
const pW: Quantity[f64, u.Power] = 1e-12 * u.W
const nW: Quantity[f64, u.Power] = 1e-9 * u.W
const uW: Quantity[f64, u.Power] = 1e-6 * u.W
const mW: Quantity[f64, u.Power] = 1e-3 * u.W
const cW: Quantity[f64, u.Power] = 1e-2 * u.W
const dW: Quantity[f64, u.Power] = 1e-1 * u.W
const daW: Quantity[f64, u.Power] = 1e1 * u.W
const hW: Quantity[f64, u.Power] = 1e2 * u.W
const kW: Quantity[f64, u.Power] = 1e3 * u.W
const MW: Quantity[f64, u.Power] = 1e6 * u.W
const GW: Quantity[f64, u.Power] = 1e9 * u.W
const TW: Quantity[f64, u.Power] = 1e12 * u.W
const PW: Quantity[f64, u.Power] = 1e15 * u.W
const EW: Quantity[f64, u.Power] = 1e18 * u.W
const ZW: Quantity[f64, u.Power] = 1e21 * u.W
const YW: Quantity[f64, u.Power] = 1e24 * u.W
const RW: Quantity[f64, u.Power] = 1e27 * u.W
const QW: Quantity[f64, u.Power] = 1e30 * u.W
const qC: Quantity[f64, u.Charge] = 1e-30 * u.C
const rC: Quantity[f64, u.Charge] = 1e-27 * u.C
const yC: Quantity[f64, u.Charge] = 1e-24 * u.C
const zC: Quantity[f64, u.Charge] = 1e-21 * u.C
const aC: Quantity[f64, u.Charge] = 1e-18 * u.C
const fC: Quantity[f64, u.Charge] = 1e-15 * u.C
const pC: Quantity[f64, u.Charge] = 1e-12 * u.C
const nC: Quantity[f64, u.Charge] = 1e-9 * u.C
const uC: Quantity[f64, u.Charge] = 1e-6 * u.C
const mC: Quantity[f64, u.Charge] = 1e-3 * u.C
const cC: Quantity[f64, u.Charge] = 1e-2 * u.C
const dC: Quantity[f64, u.Charge] = 1e-1 * u.C
const daC: Quantity[f64, u.Charge] = 1e1 * u.C
const hC: Quantity[f64, u.Charge] = 1e2 * u.C
const kC: Quantity[f64, u.Charge] = 1e3 * u.C
const MC: Quantity[f64, u.Charge] = 1e6 * u.C
const GC: Quantity[f64, u.Charge] = 1e9 * u.C
const TC: Quantity[f64, u.Charge] = 1e12 * u.C
const PC: Quantity[f64, u.Charge] = 1e15 * u.C
const EC: Quantity[f64, u.Charge] = 1e18 * u.C
const ZC: Quantity[f64, u.Charge] = 1e21 * u.C
const YC: Quantity[f64, u.Charge] = 1e24 * u.C
const RC: Quantity[f64, u.Charge] = 1e27 * u.C
const QC: Quantity[f64, u.Charge] = 1e30 * u.C
const qV: Quantity[f64, u.Voltage] = 1e-30 * u.V
const rV: Quantity[f64, u.Voltage] = 1e-27 * u.V
const yV: Quantity[f64, u.Voltage] = 1e-24 * u.V
const zV: Quantity[f64, u.Voltage] = 1e-21 * u.V
const aV: Quantity[f64, u.Voltage] = 1e-18 * u.V
const fV: Quantity[f64, u.Voltage] = 1e-15 * u.V
const pV: Quantity[f64, u.Voltage] = 1e-12 * u.V
const nV: Quantity[f64, u.Voltage] = 1e-9 * u.V
const uV: Quantity[f64, u.Voltage] = 1e-6 * u.V
const mV: Quantity[f64, u.Voltage] = 1e-3 * u.V
const cV: Quantity[f64, u.Voltage] = 1e-2 * u.V
const dV: Quantity[f64, u.Voltage] = 1e-1 * u.V
const daV: Quantity[f64, u.Voltage] = 1e1 * u.V
const hV: Quantity[f64, u.Voltage] = 1e2 * u.V
const kV: Quantity[f64, u.Voltage] = 1e3 * u.V
const MV: Quantity[f64, u.Voltage] = 1e6 * u.V
const GV: Quantity[f64, u.Voltage] = 1e9 * u.V
const TV: Quantity[f64, u.Voltage] = 1e12 * u.V
const PV: Quantity[f64, u.Voltage] = 1e15 * u.V
const EV: Quantity[f64, u.Voltage] = 1e18 * u.V
const ZV: Quantity[f64, u.Voltage] = 1e21 * u.V
const YV: Quantity[f64, u.Voltage] = 1e24 * u.V
const RV: Quantity[f64, u.Voltage] = 1e27 * u.V
const QV: Quantity[f64, u.Voltage] = 1e30 * u.V
const qF: Quantity[f64, u.Capacitance] = 1e-30 * u.F
const rF: Quantity[f64, u.Capacitance] = 1e-27 * u.F
const yF: Quantity[f64, u.Capacitance] = 1e-24 * u.F
const zF: Quantity[f64, u.Capacitance] = 1e-21 * u.F
const aF: Quantity[f64, u.Capacitance] = 1e-18 * u.F
const fF: Quantity[f64, u.Capacitance] = 1e-15 * u.F
const pF: Quantity[f64, u.Capacitance] = 1e-12 * u.F
const nF: Quantity[f64, u.Capacitance] = 1e-9 * u.F
const uF: Quantity[f64, u.Capacitance] = 1e-6 * u.F
const mF: Quantity[f64, u.Capacitance] = 1e-3 * u.F
const cF: Quantity[f64, u.Capacitance] = 1e-2 * u.F
const dF: Quantity[f64, u.Capacitance] = 1e-1 * u.F
const daF: Quantity[f64, u.Capacitance] = 1e1 * u.F
const hF: Quantity[f64, u.Capacitance] = 1e2 * u.F
const kF: Quantity[f64, u.Capacitance] = 1e3 * u.F
const MF: Quantity[f64, u.Capacitance] = 1e6 * u.F
const GF: Quantity[f64, u.Capacitance] = 1e9 * u.F
const TF: Quantity[f64, u.Capacitance] = 1e12 * u.F
const PF: Quantity[f64, u.Capacitance] = 1e15 * u.F
const EF: Quantity[f64, u.Capacitance] = 1e18 * u.F
const ZF: Quantity[f64, u.Capacitance] = 1e21 * u.F
const YF: Quantity[f64, u.Capacitance] = 1e24 * u.F
const RF: Quantity[f64, u.Capacitance] = 1e27 * u.F
const QF: Quantity[f64, u.Capacitance] = 1e30 * u.F
const qohm: Quantity[f64, u.Resistance] = 1e-30 * u.ohm
const rohm: Quantity[f64, u.Resistance] = 1e-27 * u.ohm
const yohm: Quantity[f64, u.Resistance] = 1e-24 * u.ohm
const zohm: Quantity[f64, u.Resistance] = 1e-21 * u.ohm
const aohm: Quantity[f64, u.Resistance] = 1e-18 * u.ohm
const fohm: Quantity[f64, u.Resistance] = 1e-15 * u.ohm
const pohm: Quantity[f64, u.Resistance] = 1e-12 * u.ohm
const nohm: Quantity[f64, u.Resistance] = 1e-9 * u.ohm
const uohm: Quantity[f64, u.Resistance] = 1e-6 * u.ohm
const milliohm: Quantity[f64, u.Resistance] = 1e-3 * u.ohm
const cohm: Quantity[f64, u.Resistance] = 1e-2 * u.ohm
const dohm: Quantity[f64, u.Resistance] = 1e-1 * u.ohm
const daohm: Quantity[f64, u.Resistance] = 1e1 * u.ohm
const hohm: Quantity[f64, u.Resistance] = 1e2 * u.ohm
const kohm: Quantity[f64, u.Resistance] = 1e3 * u.ohm
const Mohm: Quantity[f64, u.Resistance] = 1e6 * u.ohm
const Gohm: Quantity[f64, u.Resistance] = 1e9 * u.ohm
const Tohm: Quantity[f64, u.Resistance] = 1e12 * u.ohm
const Pohm: Quantity[f64, u.Resistance] = 1e15 * u.ohm
const Eohm: Quantity[f64, u.Resistance] = 1e18 * u.ohm
const Zohm: Quantity[f64, u.Resistance] = 1e21 * u.ohm
const Yohm: Quantity[f64, u.Resistance] = 1e24 * u.ohm
const Rohm: Quantity[f64, u.Resistance] = 1e27 * u.ohm
const Qohm: Quantity[f64, u.Resistance] = 1e30 * u.ohm
const qS: Quantity[f64, u.Conductance] = 1e-30 * u.S
const rS: Quantity[f64, u.Conductance] = 1e-27 * u.S
const yS: Quantity[f64, u.Conductance] = 1e-24 * u.S
const zS: Quantity[f64, u.Conductance] = 1e-21 * u.S
const aS: Quantity[f64, u.Conductance] = 1e-18 * u.S
const fS: Quantity[f64, u.Conductance] = 1e-15 * u.S
const pS: Quantity[f64, u.Conductance] = 1e-12 * u.S
const nS: Quantity[f64, u.Conductance] = 1e-9 * u.S
const uS: Quantity[f64, u.Conductance] = 1e-6 * u.S
const mS: Quantity[f64, u.Conductance] = 1e-3 * u.S
const cS: Quantity[f64, u.Conductance] = 1e-2 * u.S
const dS: Quantity[f64, u.Conductance] = 1e-1 * u.S
const daS: Quantity[f64, u.Conductance] = 1e1 * u.S
const hS: Quantity[f64, u.Conductance] = 1e2 * u.S
const kS: Quantity[f64, u.Conductance] = 1e3 * u.S
const MS: Quantity[f64, u.Conductance] = 1e6 * u.S
const GS: Quantity[f64, u.Conductance] = 1e9 * u.S
const TS: Quantity[f64, u.Conductance] = 1e12 * u.S
const PS: Quantity[f64, u.Conductance] = 1e15 * u.S
const ES: Quantity[f64, u.Conductance] = 1e18 * u.S
const ZS: Quantity[f64, u.Conductance] = 1e21 * u.S
const YS: Quantity[f64, u.Conductance] = 1e24 * u.S
const RS: Quantity[f64, u.Conductance] = 1e27 * u.S
const QS: Quantity[f64, u.Conductance] = 1e30 * u.S
const qWb: Quantity[f64, u.MagneticFlux] = 1e-30 * u.Wb
const rWb: Quantity[f64, u.MagneticFlux] = 1e-27 * u.Wb
const yWb: Quantity[f64, u.MagneticFlux] = 1e-24 * u.Wb
const zWb: Quantity[f64, u.MagneticFlux] = 1e-21 * u.Wb
const aWb: Quantity[f64, u.MagneticFlux] = 1e-18 * u.Wb
const fWb: Quantity[f64, u.MagneticFlux] = 1e-15 * u.Wb
const pWb: Quantity[f64, u.MagneticFlux] = 1e-12 * u.Wb
const nWb: Quantity[f64, u.MagneticFlux] = 1e-9 * u.Wb
const uWb: Quantity[f64, u.MagneticFlux] = 1e-6 * u.Wb
const mWb: Quantity[f64, u.MagneticFlux] = 1e-3 * u.Wb
const cWb: Quantity[f64, u.MagneticFlux] = 1e-2 * u.Wb
const dWb: Quantity[f64, u.MagneticFlux] = 1e-1 * u.Wb
const daWb: Quantity[f64, u.MagneticFlux] = 1e1 * u.Wb
const hWb: Quantity[f64, u.MagneticFlux] = 1e2 * u.Wb
const kWb: Quantity[f64, u.MagneticFlux] = 1e3 * u.Wb
const MWb: Quantity[f64, u.MagneticFlux] = 1e6 * u.Wb
const GWb: Quantity[f64, u.MagneticFlux] = 1e9 * u.Wb
const TWb: Quantity[f64, u.MagneticFlux] = 1e12 * u.Wb
const PWb: Quantity[f64, u.MagneticFlux] = 1e15 * u.Wb
const EWb: Quantity[f64, u.MagneticFlux] = 1e18 * u.Wb
const ZWb: Quantity[f64, u.MagneticFlux] = 1e21 * u.Wb
const YWb: Quantity[f64, u.MagneticFlux] = 1e24 * u.Wb
const RWb: Quantity[f64, u.MagneticFlux] = 1e27 * u.Wb
const QWb: Quantity[f64, u.MagneticFlux] = 1e30 * u.Wb
const qT: Quantity[f64, u.MagneticFluxDensity] = 1e-30 * u.T
const rT: Quantity[f64, u.MagneticFluxDensity] = 1e-27 * u.T
const yT: Quantity[f64, u.MagneticFluxDensity] = 1e-24 * u.T
const zT: Quantity[f64, u.MagneticFluxDensity] = 1e-21 * u.T
const aT: Quantity[f64, u.MagneticFluxDensity] = 1e-18 * u.T
const fT: Quantity[f64, u.MagneticFluxDensity] = 1e-15 * u.T
const pT: Quantity[f64, u.MagneticFluxDensity] = 1e-12 * u.T
const nT: Quantity[f64, u.MagneticFluxDensity] = 1e-9 * u.T
const uT: Quantity[f64, u.MagneticFluxDensity] = 1e-6 * u.T
const mT: Quantity[f64, u.MagneticFluxDensity] = 1e-3 * u.T
const cT: Quantity[f64, u.MagneticFluxDensity] = 1e-2 * u.T
const dT: Quantity[f64, u.MagneticFluxDensity] = 1e-1 * u.T
const daT: Quantity[f64, u.MagneticFluxDensity] = 1e1 * u.T
const hT: Quantity[f64, u.MagneticFluxDensity] = 1e2 * u.T
const kT: Quantity[f64, u.MagneticFluxDensity] = 1e3 * u.T
const MT: Quantity[f64, u.MagneticFluxDensity] = 1e6 * u.T
const GT: Quantity[f64, u.MagneticFluxDensity] = 1e9 * u.T
const TT: Quantity[f64, u.MagneticFluxDensity] = 1e12 * u.T
const PT: Quantity[f64, u.MagneticFluxDensity] = 1e15 * u.T
const ET: Quantity[f64, u.MagneticFluxDensity] = 1e18 * u.T
const ZT: Quantity[f64, u.MagneticFluxDensity] = 1e21 * u.T
const YT: Quantity[f64, u.MagneticFluxDensity] = 1e24 * u.T
const RT: Quantity[f64, u.MagneticFluxDensity] = 1e27 * u.T
const QT: Quantity[f64, u.MagneticFluxDensity] = 1e30 * u.T
const qH: Quantity[f64, u.Inductance] = 1e-30 * u.H
const rH: Quantity[f64, u.Inductance] = 1e-27 * u.H
const yH: Quantity[f64, u.Inductance] = 1e-24 * u.H
const zH: Quantity[f64, u.Inductance] = 1e-21 * u.H
const aH: Quantity[f64, u.Inductance] = 1e-18 * u.H
const fH: Quantity[f64, u.Inductance] = 1e-15 * u.H
const pH: Quantity[f64, u.Inductance] = 1e-12 * u.H
const nH: Quantity[f64, u.Inductance] = 1e-9 * u.H
const uH: Quantity[f64, u.Inductance] = 1e-6 * u.H
const mH: Quantity[f64, u.Inductance] = 1e-3 * u.H
const cH: Quantity[f64, u.Inductance] = 1e-2 * u.H
const dH: Quantity[f64, u.Inductance] = 1e-1 * u.H
const daH: Quantity[f64, u.Inductance] = 1e1 * u.H
const hH: Quantity[f64, u.Inductance] = 1e2 * u.H
const kH: Quantity[f64, u.Inductance] = 1e3 * u.H
const MH: Quantity[f64, u.Inductance] = 1e6 * u.H
const GH: Quantity[f64, u.Inductance] = 1e9 * u.H
const TH: Quantity[f64, u.Inductance] = 1e12 * u.H
const PH: Quantity[f64, u.Inductance] = 1e15 * u.H
const EH: Quantity[f64, u.Inductance] = 1e18 * u.H
const ZH: Quantity[f64, u.Inductance] = 1e21 * u.H
const YH: Quantity[f64, u.Inductance] = 1e24 * u.H
const RH: Quantity[f64, u.Inductance] = 1e27 * u.H
const QH: Quantity[f64, u.Inductance] = 1e30 * u.H
const qlm: Quantity[f64, u.Luminosity] = 1e-30 * u.lm
const rlm: Quantity[f64, u.Luminosity] = 1e-27 * u.lm
const ylm: Quantity[f64, u.Luminosity] = 1e-24 * u.lm
const zlm: Quantity[f64, u.Luminosity] = 1e-21 * u.lm
const alm: Quantity[f64, u.Luminosity] = 1e-18 * u.lm
const flm: Quantity[f64, u.Luminosity] = 1e-15 * u.lm
const plm: Quantity[f64, u.Luminosity] = 1e-12 * u.lm
const nlm: Quantity[f64, u.Luminosity] = 1e-9 * u.lm
const ulm: Quantity[f64, u.Luminosity] = 1e-6 * u.lm
const mlm: Quantity[f64, u.Luminosity] = 1e-3 * u.lm
const clm: Quantity[f64, u.Luminosity] = 1e-2 * u.lm
const dlm: Quantity[f64, u.Luminosity] = 1e-1 * u.lm
const dalm: Quantity[f64, u.Luminosity] = 1e1 * u.lm
const hlm: Quantity[f64, u.Luminosity] = 1e2 * u.lm
const klm: Quantity[f64, u.Luminosity] = 1e3 * u.lm
const Mlm: Quantity[f64, u.Luminosity] = 1e6 * u.lm
const Glm: Quantity[f64, u.Luminosity] = 1e9 * u.lm
const Tlm: Quantity[f64, u.Luminosity] = 1e12 * u.lm
const Plm: Quantity[f64, u.Luminosity] = 1e15 * u.lm
const Elm: Quantity[f64, u.Luminosity] = 1e18 * u.lm
const Zlm: Quantity[f64, u.Luminosity] = 1e21 * u.lm
const Ylm: Quantity[f64, u.Luminosity] = 1e24 * u.lm
const Rlm: Quantity[f64, u.Luminosity] = 1e27 * u.lm
const Qlm: Quantity[f64, u.Luminosity] = 1e30 * u.lm
const qlx: Quantity[f64, u.Illuminance] = 1e-30 * u.lx
const rlx: Quantity[f64, u.Illuminance] = 1e-27 * u.lx
const ylx: Quantity[f64, u.Illuminance] = 1e-24 * u.lx
const zlx: Quantity[f64, u.Illuminance] = 1e-21 * u.lx
const alx: Quantity[f64, u.Illuminance] = 1e-18 * u.lx
const flx: Quantity[f64, u.Illuminance] = 1e-15 * u.lx
const plx: Quantity[f64, u.Illuminance] = 1e-12 * u.lx
const nlx: Quantity[f64, u.Illuminance] = 1e-9 * u.lx
const ulx: Quantity[f64, u.Illuminance] = 1e-6 * u.lx
const mlx: Quantity[f64, u.Illuminance] = 1e-3 * u.lx
const clx: Quantity[f64, u.Illuminance] = 1e-2 * u.lx
const dlx: Quantity[f64, u.Illuminance] = 1e-1 * u.lx
const dalx: Quantity[f64, u.Illuminance] = 1e1 * u.lx
const hlx: Quantity[f64, u.Illuminance] = 1e2 * u.lx
const klx: Quantity[f64, u.Illuminance] = 1e3 * u.lx
const Mlx: Quantity[f64, u.Illuminance] = 1e6 * u.lx
const Glx: Quantity[f64, u.Illuminance] = 1e9 * u.lx
const Tlx: Quantity[f64, u.Illuminance] = 1e12 * u.lx
const Plx: Quantity[f64, u.Illuminance] = 1e15 * u.lx
const Elx: Quantity[f64, u.Illuminance] = 1e18 * u.lx
const Zlx: Quantity[f64, u.Illuminance] = 1e21 * u.lx
const Ylx: Quantity[f64, u.Illuminance] = 1e24 * u.lx
const Rlx: Quantity[f64, u.Illuminance] = 1e27 * u.lx
const Qlx: Quantity[f64, u.Illuminance] = 1e30 * u.lx
const qBq: Quantity[f64, u.Frequency] = 1e-30 * u.Bq
const rBq: Quantity[f64, u.Frequency] = 1e-27 * u.Bq
const yBq: Quantity[f64, u.Frequency] = 1e-24 * u.Bq
const zBq: Quantity[f64, u.Frequency] = 1e-21 * u.Bq
const aBq: Quantity[f64, u.Frequency] = 1e-18 * u.Bq
const fBq: Quantity[f64, u.Frequency] = 1e-15 * u.Bq
const pBq: Quantity[f64, u.Frequency] = 1e-12 * u.Bq
const nBq: Quantity[f64, u.Frequency] = 1e-9 * u.Bq
const uBq: Quantity[f64, u.Frequency] = 1e-6 * u.Bq
const mBq: Quantity[f64, u.Frequency] = 1e-3 * u.Bq
const cBq: Quantity[f64, u.Frequency] = 1e-2 * u.Bq
const dBq: Quantity[f64, u.Frequency] = 1e-1 * u.Bq
const daBq: Quantity[f64, u.Frequency] = 1e1 * u.Bq
const hBq: Quantity[f64, u.Frequency] = 1e2 * u.Bq
const kBq: Quantity[f64, u.Frequency] = 1e3 * u.Bq
const MBq: Quantity[f64, u.Frequency] = 1e6 * u.Bq
const GBq: Quantity[f64, u.Frequency] = 1e9 * u.Bq
const TBq: Quantity[f64, u.Frequency] = 1e12 * u.Bq
const PBq: Quantity[f64, u.Frequency] = 1e15 * u.Bq
const EBq: Quantity[f64, u.Frequency] = 1e18 * u.Bq
const ZBq: Quantity[f64, u.Frequency] = 1e21 * u.Bq
const YBq: Quantity[f64, u.Frequency] = 1e24 * u.Bq
const RBq: Quantity[f64, u.Frequency] = 1e27 * u.Bq
const QBq: Quantity[f64, u.Frequency] = 1e30 * u.Bq
const qGy: Quantity[f64, u.SpecificEnergy] = 1e-30 * u.Gy
const rGy: Quantity[f64, u.SpecificEnergy] = 1e-27 * u.Gy
const yGy: Quantity[f64, u.SpecificEnergy] = 1e-24 * u.Gy
const zGy: Quantity[f64, u.SpecificEnergy] = 1e-21 * u.Gy
const aGy: Quantity[f64, u.SpecificEnergy] = 1e-18 * u.Gy
const fGy: Quantity[f64, u.SpecificEnergy] = 1e-15 * u.Gy
const pGy: Quantity[f64, u.SpecificEnergy] = 1e-12 * u.Gy
const nGy: Quantity[f64, u.SpecificEnergy] = 1e-9 * u.Gy
const uGy: Quantity[f64, u.SpecificEnergy] = 1e-6 * u.Gy
const mGy: Quantity[f64, u.SpecificEnergy] = 1e-3 * u.Gy
const cGy: Quantity[f64, u.SpecificEnergy] = 1e-2 * u.Gy
const dGy: Quantity[f64, u.SpecificEnergy] = 1e-1 * u.Gy
const daGy: Quantity[f64, u.SpecificEnergy] = 1e1 * u.Gy
const hGy: Quantity[f64, u.SpecificEnergy] = 1e2 * u.Gy
const kGy: Quantity[f64, u.SpecificEnergy] = 1e3 * u.Gy
const MGy: Quantity[f64, u.SpecificEnergy] = 1e6 * u.Gy
const GGy: Quantity[f64, u.SpecificEnergy] = 1e9 * u.Gy
const TGy: Quantity[f64, u.SpecificEnergy] = 1e12 * u.Gy
const PGy: Quantity[f64, u.SpecificEnergy] = 1e15 * u.Gy
const EGy: Quantity[f64, u.SpecificEnergy] = 1e18 * u.Gy
const ZGy: Quantity[f64, u.SpecificEnergy] = 1e21 * u.Gy
const YGy: Quantity[f64, u.SpecificEnergy] = 1e24 * u.Gy
const RGy: Quantity[f64, u.SpecificEnergy] = 1e27 * u.Gy
const QGy: Quantity[f64, u.SpecificEnergy] = 1e30 * u.Gy
const qSv: Quantity[f64, u.SpecificEnergy] = 1e-30 * u.Sv
const rSv: Quantity[f64, u.SpecificEnergy] = 1e-27 * u.Sv
const ySv: Quantity[f64, u.SpecificEnergy] = 1e-24 * u.Sv
const zSv: Quantity[f64, u.SpecificEnergy] = 1e-21 * u.Sv
const aSv: Quantity[f64, u.SpecificEnergy] = 1e-18 * u.Sv
const fSv: Quantity[f64, u.SpecificEnergy] = 1e-15 * u.Sv
const pSv: Quantity[f64, u.SpecificEnergy] = 1e-12 * u.Sv
const nSv: Quantity[f64, u.SpecificEnergy] = 1e-9 * u.Sv
const uSv: Quantity[f64, u.SpecificEnergy] = 1e-6 * u.Sv
const mSv: Quantity[f64, u.SpecificEnergy] = 1e-3 * u.Sv
const cSv: Quantity[f64, u.SpecificEnergy] = 1e-2 * u.Sv
const dSv: Quantity[f64, u.SpecificEnergy] = 1e-1 * u.Sv
const daSv: Quantity[f64, u.SpecificEnergy] = 1e1 * u.Sv
const hSv: Quantity[f64, u.SpecificEnergy] = 1e2 * u.Sv
const kSv: Quantity[f64, u.SpecificEnergy] = 1e3 * u.Sv
const MSv: Quantity[f64, u.SpecificEnergy] = 1e6 * u.Sv
const GSv: Quantity[f64, u.SpecificEnergy] = 1e9 * u.Sv
const TSv: Quantity[f64, u.SpecificEnergy] = 1e12 * u.Sv
const PSv: Quantity[f64, u.SpecificEnergy] = 1e15 * u.Sv
const ESv: Quantity[f64, u.SpecificEnergy] = 1e18 * u.Sv
const ZSv: Quantity[f64, u.SpecificEnergy] = 1e21 * u.Sv
const YSv: Quantity[f64, u.SpecificEnergy] = 1e24 * u.Sv
const RSv: Quantity[f64, u.SpecificEnergy] = 1e27 * u.Sv
const QSv: Quantity[f64, u.SpecificEnergy] = 1e30 * u.Sv
const qkat: Quantity[f64, u.CatalyticActivity] = 1e-30 * u.kat
const rkat: Quantity[f64, u.CatalyticActivity] = 1e-27 * u.kat
const ykat: Quantity[f64, u.CatalyticActivity] = 1e-24 * u.kat
const zkat: Quantity[f64, u.CatalyticActivity] = 1e-21 * u.kat
const akat: Quantity[f64, u.CatalyticActivity] = 1e-18 * u.kat
const fkat: Quantity[f64, u.CatalyticActivity] = 1e-15 * u.kat
const pkat: Quantity[f64, u.CatalyticActivity] = 1e-12 * u.kat
const nkat: Quantity[f64, u.CatalyticActivity] = 1e-9 * u.kat
const ukat: Quantity[f64, u.CatalyticActivity] = 1e-6 * u.kat
const mkat: Quantity[f64, u.CatalyticActivity] = 1e-3 * u.kat
const ckat: Quantity[f64, u.CatalyticActivity] = 1e-2 * u.kat
const dkat: Quantity[f64, u.CatalyticActivity] = 1e-1 * u.kat
const dakat: Quantity[f64, u.CatalyticActivity] = 1e1 * u.kat
const hkat: Quantity[f64, u.CatalyticActivity] = 1e2 * u.kat
const kkat: Quantity[f64, u.CatalyticActivity] = 1e3 * u.kat
const Mkat: Quantity[f64, u.CatalyticActivity] = 1e6 * u.kat
const Gkat: Quantity[f64, u.CatalyticActivity] = 1e9 * u.kat
const Tkat: Quantity[f64, u.CatalyticActivity] = 1e12 * u.kat
const Pkat: Quantity[f64, u.CatalyticActivity] = 1e15 * u.kat
const Ekat: Quantity[f64, u.CatalyticActivity] = 1e18 * u.kat
const Zkat: Quantity[f64, u.CatalyticActivity] = 1e21 * u.kat
const Ykat: Quantity[f64, u.CatalyticActivity] = 1e24 * u.kat
const Rkat: Quantity[f64, u.CatalyticActivity] = 1e27 * u.kat
const Qkat: Quantity[f64, u.CatalyticActivity] = 1e30 * u.kat
const microlitre: Quantity[f64, u.Volume] = 1e-9 * (u.m * u.m * u.m)
const centilitre: Quantity[f64, u.Volume] = 1e-5 * (u.m * u.m * u.m)
const decilitre: Quantity[f64, u.Volume] = 1e-4 * (u.m * u.m * u.m)
const hectolitre: Quantity[f64, u.Volume] = 1e-1 * (u.m * u.m * u.m)
const kilolitre: Quantity[f64, u.Volume] = u.m * u.m * u.m
const megalitre: Quantity[f64, u.Volume] = 1e3 * (u.m * u.m * u.m)
const meV: Quantity[f64, u.Energy] = 1e-3 * u.eV
const TeV: Quantity[f64, u.Energy] = 1e12 * u.eV
const TWh: Quantity[f64, u.Energy] = 1e12 * u.Wh
const fermi: Quantity[f64, u.Length] = 1e-15 * u.m
const micron: Quantity[f64, u.Length] = 1e-6 * u.m
const mil: Quantity[f64, u.Length] = 1e-3 * u.inch
const microinch: Quantity[f64, u.Length] = 1e-6 * u.inch
const point: Quantity[f64, u.Length] = u.inch / 72.0
const pica: Quantity[f64, u.Length] = u.inch / 6.0
const printers_point: Quantity[f64, u.Length] = 0.013837 * u.inch
const printers_pica: Quantity[f64, u.Length] = 12.0 * printers_point
const survey_foot: Quantity[f64, u.Length] = (1200.0 / 3937.0) * u.m
const survey_mile: Quantity[f64, u.Length] = 5280.0 * survey_foot
const chain: Quantity[f64, u.Length] = 66.0 * survey_foot
const rod: Quantity[f64, u.Length] = 16.5 * survey_foot
const fathom: Quantity[f64, u.Length] = 6.0 * survey_foot
const are: Quantity[f64, u.Area] = 1e2 * (u.m * u.m)
const barn: Quantity[f64, u.Area] = 1e-28 * (u.m * u.m)
const circular_mil: Quantity[f64, u.Area] = (f64.PI / 4.0) * (mil * mil)
const survey_acre: Quantity[f64, u.Area] = 43560.0 * (survey_foot * survey_foot)
const darcy: Quantity[f64, u.Area] = u.cm * (1e-3 * u.Pa * u.s) * (u.cm * u.cm * u.cm / u.s) / (u.cm * u.cm * u.atm)
const stere: Quantity[f64, u.Volume] = u.m * u.m * u.m
const acre_foot: Quantity[f64, u.Volume] = survey_acre * survey_foot
const barrel: Quantity[f64, u.Volume] = 42.0 * u.gallon
const quart: Quantity[f64, u.Volume] = u.gallon / 4.0
const pint: Quantity[f64, u.Volume] = u.gallon / 8.0
const cup: Quantity[f64, u.Volume] = u.gallon / 16.0
const gill: Quantity[f64, u.Volume] = u.gallon / 32.0
const fluid_ounce: Quantity[f64, u.Volume] = u.gallon / 128.0
const tablespoon: Quantity[f64, u.Volume] = fluid_ounce / 2.0
const teaspoon: Quantity[f64, u.Volume] = fluid_ounce / 6.0
const bushel: Quantity[f64, u.Volume] = 2150.42 * (u.inch * u.inch * u.inch)
const peck: Quantity[f64, u.Volume] = bushel / 4.0
const dry_quart: Quantity[f64, u.Volume] = bushel / 32.0
const dry_pint: Quantity[f64, u.Volume] = bushel / 64.0
const imperial_gallon: Quantity[f64, u.Volume] = 4.54609e-3 * (u.m * u.m * u.m)
const imperial_gill: Quantity[f64, u.Volume] = imperial_gallon / 32.0
const imperial_fluid_ounce: Quantity[f64, u.Volume] = imperial_gallon / 160.0
const cord: Quantity[f64, u.Volume] = 128.0 * (u.foot * u.foot * u.foot)
const register_ton: Quantity[f64, u.Volume] = 100.0 * (u.foot * u.foot * u.foot)
const carat: Quantity[f64, u.Mass] = 200.0 * u.mg
const grain: Quantity[f64, u.Mass] = 64.79891 * u.mg
const pennyweight: Quantity[f64, u.Mass] = 24.0 * grain
const troy_ounce: Quantity[f64, u.Mass] = 480.0 * grain
const troy_pound: Quantity[f64, u.Mass] = 5760.0 * grain
const short_hundredweight: Quantity[f64, u.Mass] = 100.0 * u.pound
const long_hundredweight: Quantity[f64, u.Mass] = 112.0 * u.pound
const short_ton: Quantity[f64, u.Mass] = 2000.0 * u.pound
const long_ton: Quantity[f64, u.Mass] = 2240.0 * u.pound
const assay_ton: Quantity[f64, u.Mass] = (short_ton / troy_ounce) * u.mg
const slug: Quantity[f64, u.Mass] = u.pound_force * u.s * u.s / u.foot
const shake: Quantity[f64, u.Time] = 1e-8 * u.s
const sidereal_day: Quantity[f64, u.Time] = 86164.0905 * u.s
const sidereal_hour: Quantity[f64, u.Time] = sidereal_day / 24.0
const sidereal_minute: Quantity[f64, u.Time] = sidereal_hour / 60.0
const sidereal_second: Quantity[f64, u.Time] = sidereal_minute / 60.0
const sidereal_year: Quantity[f64, u.Time] = 365.256363004 * u.day
const tropical_year: Quantity[f64, u.Time] = 365.24219 * u.day
const dyne: Quantity[f64, u.Force] = 1e-5 * u.N
const kilogram_force: Quantity[f64, u.Force] = u.kg * u.g_n
const ounce_force: Quantity[f64, u.Force] = u.pound_force / 16.0
const poundal: Quantity[f64, u.Force] = u.pound * u.foot / (u.s * u.s)
const kip: Quantity[f64, u.Force] = 1e3 * u.pound_force
const ton_force: Quantity[f64, u.Force] = 2000.0 * u.pound_force
const technical_atmosphere: Quantity[f64, u.Pressure] = kilogram_force / (u.cm * u.cm)
const ksi: Quantity[f64, u.Pressure] = 1e3 * u.psi
const cmHg: Quantity[f64, u.Pressure] = 10.0 * u.mmHg
const inHg: Quantity[f64, u.Pressure] = 25.4 * u.mmHg
const ftHg: Quantity[f64, u.Pressure] = 304.8 * u.mmHg
const mmH2O: Quantity[f64, u.Pressure] = water_density * u.g_n * u.mm
const cmH2O: Quantity[f64, u.Pressure] = water_density * u.g_n * u.cm
const inH2O: Quantity[f64, u.Pressure] = water_density * u.g_n * u.inch
const ftH2O: Quantity[f64, u.Pressure] = water_density * u.g_n * u.foot
const erg: Quantity[f64, u.Energy] = 1e-7 * u.J
const calorie_IT: Quantity[f64, u.Energy] = 4.1868 * u.J
const kilocalorie_IT: Quantity[f64, u.Energy] = 4186.8 * u.J
const btu_th: Quantity[f64, u.Energy] = u.calorie / (u.g * u.K) * u.pound * u.rankine
const therm_EC: Quantity[f64, u.Energy] = 1.05506e8 * u.J
const therm_US: Quantity[f64, u.Energy] = 1.054804e8 * u.J
const quad: Quantity[f64, u.Energy] = 1e15 * u.btu
const ton_of_tnt: Quantity[f64, u.Energy] = 4.184e9 * u.J
const horsepower_electric: Quantity[f64, u.Power] = 746.0 * u.W
const horsepower_metric: Quantity[f64, u.Power] = 75.0 * kilogram_force * u.m / u.s
const ton_of_refrigeration: Quantity[f64, u.Power] = 12000.0 * u.btu / u.hour
const poise: Quantity[f64, u.DynamicViscosity] = 0.1 * u.Pa * u.s
const centipoise: Quantity[f64, u.DynamicViscosity] = 1e-3 * u.Pa * u.s
const rhe: Quantity[f64, One / u.DynamicViscosity] = 1.0 / poise
const stokes: Quantity[f64, u.KinematicViscosity] = 1e-4 * (u.m * u.m) / u.s
const centistokes: Quantity[f64, u.KinematicViscosity] = 1e-6 * (u.m * u.m) / u.s
const ampere_hour: Quantity[f64, u.Charge] = u.A * u.hour
const milliampere_hour: Quantity[f64, u.Charge] = u.mA * u.hour
const faraday: Quantity[f64, u.Charge] = u.e * u.N_A * u.mol
const mho: Quantity[f64, u.Conductance] = u.S
const debye: Quantity[f64, ElectricDipoleMoment] = 1e-21 * u.C * u.m * u.m / (u.c * u.s)
const gauss: Quantity[f64, u.MagneticFluxDensity] = 1e-4 * u.T
const gamma: Quantity[f64, u.MagneticFluxDensity] = 1e-9 * u.T
const maxwell: Quantity[f64, u.MagneticFlux] = 1e-8 * u.Wb
const oersted: Quantity[f64, MagneticFieldStrength] = (1e3 / (4.0 * f64.PI)) * u.A / u.m
const curie: Quantity[f64, u.Frequency] = 3.7e10 * u.Bq
const rad_dose: Quantity[f64, u.SpecificEnergy] = 1e-2 * u.Gy
const rem: Quantity[f64, u.SpecificEnergy] = 1e-2 * u.Sv
const roentgen: Quantity[f64, Exposure] = 2.58e-4 * u.C / u.kg
const footcandle: Quantity[f64, u.Illuminance] = u.lm / (u.foot * u.foot)
const phot: Quantity[f64, u.Illuminance] = u.lm / (u.cm * u.cm)
const stilb: Quantity[f64, Luminance] = u.cd / (u.cm * u.cm)
const lambert: Quantity[f64, Luminance] = u.cd / (f64.PI * u.cm * u.cm)
const footlambert: Quantity[f64, Luminance] = u.cd / (f64.PI * u.foot * u.foot)
const revolution: Quantity[f64, One] = 2.0 * f64.PI * u.rad
const gon: Quantity[f64, One] = (f64.PI / 200.0) * u.rad
const angular_mil: Quantity[f64, One] = (2.0 * f64.PI / 6400.0) * u.rad
const gal: Quantity[f64, u.Acceleration] = 1e-2 * u.m / (u.s * u.s)
const kayser: Quantity[f64, u.Wavenumber] = 1e2 / u.m
const tex: Quantity[f64, LinearDensity] = u.g / u.km
const denier: Quantity[f64, LinearDensity] = u.g / (9000.0 * u.m)
std.url
Strict RFC 3986 absolute URL parsing and formatting without normalization, component-aware percent encoding, and query pairs. Errors: UrlError. Reference.
enum UrlError
LimitExceeded | MissingScheme | MissingHost | InvalidByte(i64) | InvalidEscape(i64) | InvalidHost(i64) | InvalidPort(i64) | InvalidUtf8 | Allocation(AllocError)
enum Component
Userinfo | Segment | Path | Query | QueryPart | Fragment
class Url
scheme: str
userinfo: Option[str]
host: Option[str]
port: Option[i64]
path: str
query: Option[str]
fragment: Option[str]
def to_string(self) -> Result[str, UrlError]
class Pair
key: str
value: str
def parse(text: &str, max_bytes: NonNegative[i64]) -> Result[Url, UrlError]
def encode_bytes(source: &[u8], component: Component) -> Result[Bytes, UrlError]
def decode_bytes(source: &[u8]) -> Result[Bytes, UrlError]
def encode(text: &str, component: Component) -> Result[str, UrlError]
def decode(text: &str) -> Result[str, UrlError]
def query_pairs(query: &str) -> Result[list[Pair], UrlError]
def query_text(pairs: &list[Pair]) -> Result[str, UrlError]
std.uuid
UUID values, canonical text and network bytes, random v4 and timestamp-prefixed v7 generation. Errors: ParseError, BytesError, SecretsError, RangeError. Reference.
class Uuid
def as_u64_pair(self) -> (u64, u64)
def compare(self, other: &Uuid) -> i64
def is_nil(self) -> bool
def is_rfc_variant(self) -> bool
def version(self) -> Option[u8]
def unix_milliseconds(self) -> Option[u64]
def write_bytes(self, target: &mut Bytes, offset: i64) -> ()
def to_bytes(self) -> Result[Bytes, BytesError]
def to_string(self) -> Result[str, BytesError]
def nil() -> Uuid
def from_u64_pair(high: u64, low: u64) -> Uuid
def parse(text: &str) -> Result[Uuid, ParseError]
def from_bytes(source: &[u8]) -> Result[Uuid, ParseError]
def v4_from[R: Reader](source: &mut Source[R]) -> Result[Uuid, SecretsError]
def v4() -> Result[Uuid, SecretsError]
def v7_from[R: Reader](at: &Timestamp, source: &mut Source[R]) -> Result[Uuid, RangeError | SecretsError]
def v7() -> Result[Uuid, RangeError | SecretsError]
std.xml
Bounded UTF-8 XML event readers and validating stream writers, without DTDs or external entities. Errors: XmlError. Reference.
enum Fault
Syntax | UnexpectedEnd | InvalidName | InvalidCharacter | InvalidUtf8 | InvalidReference | UnknownEntity | DuplicateAttribute | MismatchedEnd | DoctypeForbidden | UnsupportedDeclaration | DepthExceeded | InputTooLarge | OutputTooLarge | TokenTooLarge | TooManyAttributes | InvalidState | Stopped | Allocation(AllocError) | Stream(StreamError)
class XmlError
fault: Fault
offset: i64
class Limits
max_depth: NonNegative[i64]
max_input: NonNegative[i64]
max_output: NonNegative[i64]
max_token: Positive[i64]
max_attributes: NonNegative[i64]
def limits() -> Limits
class Attribute
name: str
value: str
enum Event
Declaration(Option[bool]) | Start(str, list[Attribute]) | Empty(str, list[Attribute]) | End(str) | Text(str) | CData(str) | Comment(str) | ProcessingInstruction(str, str) | Eof
class Reader[R]
inner: R
def offset(self) -> i64
def next(self: &mut Reader[R]) -> Result[Event, XmlError]
def reader[R](inner: R, bounds: &Limits) -> Result[Reader[R], XmlError]
class Writer[W]
inner: W
def offset(self) -> i64
def write(self: &mut Writer[W], event: &Event) -> Result[(), XmlError]
def finish(self: &mut Writer[W]) -> Result[(), XmlError]
def writer[W](inner: W, bounds: &Limits) -> Result[Writer[W], XmlError]
Standard library conventions
These are the rules public std APIs follow. A new module, function, or method
follows them; a change that breaks one explains why on its module page. Each
rule records the majority or most recent practice in the
signature index. Where existing modules differ, the rule says
so and names the issue or backlog item that tracks the
difference; those modules are not examples to copy.
Packages outside std may use the same rules so their APIs read like the
standard library.
Implementation and visibility
- Write a module in Plenty when the functionality is reasonably small and straightforward to express. Wrap a Rust or C library only for a concrete reason, such as substantial complexity (SQLite, Polars) or specialist security requirements (cryptography). The wrapper keeps the same allocation-failure and error contracts as Plenty code.
- Everything declared
pubis public API: it appears in the signature index and is documented on the module page. Helpers stay private. - Native hooks are std-only compiler intrinsics, not methods or functions
of a std module. A std module calls the intrinsic (
__crypto_update,__compress_new) with&[u8]views for inputs and&mut Bytesoutput buffers whose ranges it checks itself; user code calling one is rejected as reserved for the standard library. - A module’s error types and protocols are imported from that module. The
prelude holds builtin names, plus
std.fmt.FormatError, which user formatting methods return (issue #632). - Each module page states ownership, allocation, error variants and limits
for every public function, as described in
Reading a module page, and ends with
## Acknowledgementswhen there is something to credit.
Naming
Constructors
| Form | Use | Examples |
|---|---|---|
| Lower-case noun of the product | Build a value or adapter from parts | std.io.cursor(data), std.json.decoder(&limits), std.fs.watch.watcher() |
limits(), config() | Default settings value | std.json.limits(), std.http.client.config() |
| Verb for an operating-system resource | Acquire a file, socket or process | open_file, connect, listen, dial |
from_<source> | Convert from another representation | std.bytes.from_text, std.complex.from_polar, std.io.from_file |
parse / parse_text | Decode a value from bytes / from str | std.json.parse, std.json.parse_text |
with_capacity(capacity) | Empty container with reserved storage | std.bytes.with_capacity |
__init__ | Containers built with the class name | Deque[T](), LruCache(capacity) |
A format defined over text alone takes parse(text: &str) (std.url,
std.uuid, std.mime). new is used only by std.image and std.mime; do
not add it.
Conversions and rendering
| Name | Meaning |
|---|---|
to_<type>() | New owned value; may allocate and fail |
as_<type>() | Cheap view or reinterpretation; never allocates |
to_text() | Validated conversion of byte content to str (Bytes, Path) |
to_string() | Canonical text of a structured value, as an owned str |
encode / encode_text | Serialize to Bytes / to str |
describe() | One-line error description as Result[str, AllocError] |
display[W](self, out) | Write to a std.fmt.Formatter[W]; print and f-string fields call it |
debug[W](self, out) | Write a diagnostic form; str.repr, !r and aggregate elements call it |
format[W](self, output) | Write to a std.fmt.Formatter[W]; the Formattable protocol for reports |
format is reserved for writing to a formatter. Url, MediaType,
SocketAddress and SetCookie return owned canonical text through to_string().
display and debug are reserved for the formatter form:
formatting a type whose method of either name has another signature is a
compile error. See Formatting.
A _text suffix names the str form of an operation whose main form takes
bytes (parse_text, encode_text, from_text). A _bytes suffix names the
byte form of an operation whose main form takes text (get_bytes,
command_bytes, send_bytes). The suffix does not yet predict the type:
std.path.from_bytes takes list[u8] and std.bytes.from_bytes takes
bytes (see Bytes).
Predicates
A method returning bool about the value’s state or contents starts with
is_ or has_: is_empty, is_terminal, is_done, is_cancelled.
Deadline.expired(now) is an established exception. Flags that report what an
event was (std.fs.watch created(), deleted()) are read like fields and
need no prefix.
Existing deviations: closed() on std.actor.net, std.actor.tls and
std.fs readers; is_finished() in std.cli.progress; is_complete() on
the JSON encoder; complete() on the inflater; cancelled() in
std.http.server. New code uses is_done() for “no further work or output”.
Sizes and counts
| Name | Meaning |
|---|---|
len() | Elements the value holds, as i64 |
is_empty() | len() == 0 |
capacity() | Elements storable without reallocating, as i64 |
count(value, ...) | Occurrences of value, as in Python’s list.count |
count() without an argument also reports a size in std.fs.watch (events in
the batch), std.simd masks (set lanes) and std.sketch (values observed).
New code uses len() for held elements and a descriptive noun for totals.
len(container) calls its accessible shared len(self) -> i64 method.
in does not dispatch to container methods; use explicit search operations
(issue #623; remaining protocol scope is B49).
Ending work
| Method | Contract |
|---|---|
flush(&mut self) | Write buffered output; the object stays usable |
finish(&mut self) -> Result[P, E] | Complete an encoder, writer or digest and return its product (Bytes, str, or ()); later writes fail |
close(&mut self) -> Result[(), E] | Release the resource and report the failure that release can produce |
close returns the type’s own error, except that a type implementing
std.io.Closer returns StreamError (issue #533 records why TcpStream.close
differs from TcpListener.close). Walk.close and the TLS Session.close
return ().
An adapter whose protocol cannot carry its own error records it for later:
failure(self) -> Option[E] returns a copy, and
take_failure(&mut self) -> Option[E] moves out an error that owns data. Do
not add a third spelling.
Readiness
A selectable type has
watch(self, selector: &mut Selector, key: u64, interest: Interest) -> Result[(), E]
(omitting interest when only one direction applies) and
unwatch(self, selector: &mut Selector) -> Result[(), E]. std.fs.watch
spells these register/unregister, HTTP server transports return bool,
and the actor Context.watch returns a Watch handle.
Errors
Domain error types
A module with failures of its own defines one public error named
<Module>Error (JsonError, NetError): an enum, or a class carrying an enum
plus position. Variants name the fault (Truncated, LimitExceeded,
InvalidUtf8). A cause from another module is kept intact in a variant named
after its source: Stream(StreamError), Io(IoError), Net(NetError),
Http(HttpError). std.error covers context and mapping.
Allocation failure
| Module failures | Error type |
|---|---|
| Allocation only | AllocError (std.deque, std.cache.lru, std.diff) |
| Allocation and domain faults | The domain error with an Allocation(AllocError) variant |
Do not return a union such as AllocError | GraphError; add the variant
instead. A function that only forwards another module’s error keeps that
error, as std.bytes constructors keep AllocError. An operation with two
unrelated causes names both in its own enum: std.actor.StageError is
Allocation(AllocError) or Wait(WaitError).
Invalid arguments
A size that cannot be allocated, negative included, is
AllocError.CapacityOverflow, or Allocation(CapacityOverflow) in a domain
error: std.bytes.with_capacity(-1), std.secrets.token_bytes(-1).
A position the caller computes against a value it holds, such as an index or
range into a buffer, or a count that no call could accept, traps when invalid,
as an out-of-range list index does. The message names the operation and the
argument (Bytes.get: index is outside 0..len()), and the
trap report
names the calling line. The operation’s result then names only what can still
fail: Bytes.get returns u8, Rng.sample_indices returns
Result[list[i64], AllocError]. std.bytes range methods, their
read_file and read_stdin ranges, and std.random counts and ranges follow
this rule. Standard-library code reports these with the std-only __trap
statement; a handle from the same object that has become invalid may trap the
same way (std.graph).
A limit, size or count whose fixed lower bound is 0 or 1 is a
NonNegative[i64] or Positive[i64] parameter or field, so the caller checks
it once with try_from and a literal is checked at compile time
(integer domains).
A fixed ceiling on such a value is checked at a direct std call, constructor,
or field update when its input is a literal or a module constant (including
constant arithmetic the compiler can evaluate). A value above it is a compile
error. Dynamic values still trap when the operation uses them, naming the
operation and the argument. Ceilings are declaration metadata: they do not
change Positive[i64] or NonNegative[i64], so passing a value through an
ordinary local, function, reference, or erased callable does not prove a ceiling.
Runtime guards remain necessary. See std ceiling checking
for the internal declaration mechanism and its scope.
Another run-time value the caller supplies (a limit or configuration) that the
operation cannot accept is an Err with a named variant: OutOfRange,
InvalidLimits, InvalidConfig. It is never clamped or ignored. Buffer
ranges, encoding and UUID offsets, crypto output offsets, and CSV field
indices trap. Builtins follow several policies
today (issue #629); see the
consistency proposal.
Returning moved inputs
An operation that takes ownership of a value and can fail before using it
(send, submit, enqueue, admit) returns the value in its error, generic over the
value’s type: Undelivered[M] and RequestError[M] in std.actor,
SendError[T] for channels,
SubmitError[F] for executors, SpawnError[F] for thread starts. Exception:
a scoped spawn with borrowed arguments reports the job-free SpawnError[()]
and drops its moved arguments, because an error cannot carry a loan.
Parameters
Ownership
Borrow (&T) what the call only reads. Use &mut T for state or a buffer it
updates. Take ownership (T) only when the result or a queue keeps the value:
std.io.cursor(data: Bytes), std.xml.reader(inner: R, ...). An adapter that
owns its inner stream exposes it as a public inner field so the caller can
recover it.
Time and cancellation
| Quantity | Type |
|---|---|
| Span | &Duration |
| Monotonic point | &Instant |
| Wall-clock point | &Timestamp |
| Blocking wait | deadline: &Deadline, cancel: &CancellationToken, the last two parameters |
| Server or loop shutdown | stop: &CancellationToken |
A method that also has a deadline-free form adds the deadline under a
_before suffix (read_before, handshake_before); a nonblocking form uses
_now (read_now, accept_now). An operation that reads the clock accepts
now: &Instant, or offers an _at variant that does, so tests can supply the
time with the fake clocks in std.test.
Typed spans and wall-clock values use Duration and Timestamp in SQLite,
key-value storage, JWT validation, cookies, SSE retry hints and HTTP dates.
The typed waits in std.sync provide deadline
and cancellation forms for compiler-owned channels, futures and tokens.
Low-level _timeout_ms forms remain available; their old _timeout names are
compatibility aliases. SSE’s optional event and id follow the required
deadline, cancel pair because required parameters precede defaults.
Limits
Every input-driven allocation has a cap.
- One or two caps: named
max_<thing>: NonNegative[i64]parameters with defaults (std.url.parse(text, max_bytes),std.db.kv.connect(..., max_length)). - Several caps: a
Limitsclass with publicmax_<thing>fields, alimits()function returning documented defaults, and alimits: &Limitsparameter. A module with several such classes qualifies both names (FormLimits,form_limits()). - Fields with a lower bound of
0or1useNonNegative[i64]orPositive[i64]; direct construction or field updates reject a statically known value above a fixed ceiling, and dynamic values trap when the operation starts. A relation between fields that no type states remainsInvalidLimits. An input that exceeds a limit isLimitExceededor a specific variant (TooLarge).
std.json, std.toml, std.markdown, std.template, std.regex and
std.xml name the parameter bounds: &Limits; std.http and std.image use
limits:. A parameter follows its type’s name, as config: &Config and
dialect: &Dialect do. std.actor.net.Limits has no limits() function and
no max_ prefixes, and InvalidLimit appears beside InvalidLimits.
Bytes
| Role | Type |
|---|---|
| Owned buffer, argument or result | Bytes |
| Read-only input | source: &[u8], accepting &buffer, &buffer[a:b], &list and b"..." literals |
| Caller’s output buffer | buffer: &mut Bytes, start: i64, stop: i64, a half-open range |
| A list of byte-sized numbers | list[u8] |
A &[u8] input carries its own range, so it takes no start and stop
and has no out-of-range error: the caller passes &buffer[a:b], whose bounds
clamp. Slice views
describe the conversion.
std.path, std.env, std.process and std.cli carry operating-system
strings as list[u8] (B39).
Streams
Byte streams implement the std.io protocols Reader, Writer,
Seeker and Closer: read(buffer, start, stop) -> Result[i64, StreamError]
reports partial progress. Read ranges outside 0 <= start <= stop <= buffer.len()
trap before transport access, including empty ranges and EOF; read_exact
validates the same bounds. A function borrows a stream as reader: &mut R
with R: Reader; an adapter takes inner: R.
std.io.typed is for callers that must recover the
transport’s own error type (std.http.typed); other modules do not add a
second protocol. Network streams add deadline and nonblocking forms
(read_before, read_now) that return the module’s error.
Booleans and enums
A choice between modes is an enum: Shutdown, Interest, Endian,
DuplicateKeys. A bool parameter is acceptable when its name says what
True means (enabled, descending, html), it has a default, and callers
pass it by name. Two bool parameters that combine into modes are an enum.
Exception: std.fs.walk(root, top_down) requires a positional bool. An
operation that offers a mode by a separate method instead, such as
ThreadPoolExecutor.cancel_pending() before shutdown(), needs neither.
Integer types
| Quantity | Type |
|---|---|
| Length, index, offset, size | i64, matching len() and indexing |
| Capacity, limit or count with a fixed lower bound | NonNegative[i64] (>= 0) or Positive[i64] (>= 1) |
| Hash, key, seed, bit pattern, selector key | u64 |
| Value fixed by a format or the OS (file size, inode, wire integer) | Its own width |
| Running statistics counter | u64 |
Builtin capacities and worker/chunk counts also use Positive[i64]:
channel(capacity), ThreadPoolExecutor(max_workers, queue_capacity) and
.par(pool, chunks=...). Allocation sizes (with_capacity, reserve) stay
i64 and report a negative size as CapacityOverflow. Results such as
len() stay i64. CuckooFilter.len() returns u64; low-level timed
waits still take u64 milliseconds.
Formatting
std.fmt.Formatter[W] and its FormatError serve value formatting and
reports. Import types, constructors, and the writer-generic Formattable
protocol from std.fmt; std.report provides rendering helpers.
- a value that f-strings display implements
display[W]; - an error that renders into a report implements
format[W](self, output: &mut Formatter[W]); - an owned canonical string is
to_string() -> Result[str, E]; reserveformatfor writing to a formatter (including compiler-handled f-strings). printand f-strings calldisplay;str.repr,!rand elements of formatted aggregates calldebug. A type withoutdisplayuses itsdebug, and one withoutdebuguses its structure. When a user method can run,printreturnsResult[(), IoError | FormatError], andstr.reprand f-strings returnResult[str, AllocError | FormatError]; otherwiseprintreportsIoErrorand both othersAllocError. See formatting.std.log.Logger.debugandstd.cli.Progress.displayhave other signatures, soprintrejects those types; call their methods instead.
Deviations
| Area | Deviation | Tracking |
|---|---|---|
| Invalid arguments | Builtin policies differ | Issue #629 |
| Containers | in does not dispatch to std container methods | B49; issue #623 records the length/diagnostic fix |
| Bytes | list[u8] OS strings | B39 |
| Streams | TcpStream.close returns StreamError | Issue #533 (intentional) |
| Limits | bounds: spelling; std.actor.net.Limits shape | Not yet tracked |
| Names | Predicates without is_; count() as size; watch shapes; new | Not yet tracked |
std.actor
Supervised actors on one thread. Every actor owns a heap: its handlers allocate there, and a trap in a handler discards that heap, closes the files the actor opened, and tells the actor’s parent. The rest of the program keeps running. Messages are copied into the receiver’s heap, mailboxes are bounded and allocated when the actor starts, and a send that cannot be delivered returns the message.
Implemented targets: x86-64 and AArch64 GNU/Linux. Elsewhere supported() is
False and starting an actor returns StartError.Unsupported. The ARM heap
and AAPCS64 recovery path are part of ARM qualification;
native hardware validation remains a release gate.
A boss starts two workers. One answers; the other has a bug and traps. The boss receives the answer and both exits:
from std.actor import Address, Context, Exit, Mail, heap_bytes_used, heaps_live, StageError, stage
from std.ready import WaitError
from std.sync import CancellationToken
from std.time import deadline_after, monotonic_now, seconds
type Fault = AllocError | IoError | OverflowError[i64]
pub enum Job:
Square(i64)
Answer(i64)
pub class Boss:
pub me: Option[Address]
pub ended: i64
pub class Worker:
pub boss: Address
pub enum Node:
Leading(Boss)
Working(Worker)
pub def started(self: &mut Node, context: &mut Context[Job, Fault]) -> Result[(), Fault]:
match self:
case Node.Leading(boss):
boss.me = Some(context.address())
good = context.spawn(1, 65536).unwrap()
bad = context.spawn(2, 65536).unwrap()
context.send(good, Job.Square(7)).unwrap()
context.send(bad, Job.Square(-1)).unwrap()
case Node.Working(_):
pass
Ok(())
pub def receive(self: &mut Node, mail: Mail[Job, Fault], context: &mut Context[Job, Fault]) -> Result[(), Fault]:
match self:
case Node.Leading(boss):
match mail:
case Mail.Message(Job.Answer(value)):
print("answer", value)?
case Mail.Down(_, _, Exit.Stopped):
print("worker stopped")?
boss.ended += 1
case Mail.Down(_, _, Exit.Trapped(kind)):
print("worker trapped", kind)?
boss.ended += 1
case _:
pass
if boss.ended == 2:
context.stop()
case Node.Working(worker):
match mail:
case Mail.Message(Job.Square(value)):
if value < 0:
# A bug: unwrapping Nothing traps.
missing: Option[i64] = Nothing
missing.unwrap()
context.send(worker.boss, Job.Answer(value * value)).unwrap()
context.stop()
case _:
pass
Ok(())
# Runs with the new child's heap current; see "The spawn hook".
pub def spawn(self, tag: i64, child: Address, context: &mut Context[Job, Fault]) -> Result[Option[Node], Fault]:
match self:
case Node.Leading(boss):
Ok(Some(Node.Working(Worker(boss.me.unwrap()))))
case Node.Working(_):
Ok(Nothing)
def main() -> Result[(), AllocError | IoError | StageError | WaitError]:
mut actors = stage[Node, Job, Fault](8, 4, 4)?
root = Node.Leading(Boss(Nothing, 0))
actors.start(root, 65536).unwrap()
actors.run(&deadline_after(&monotonic_now(), &seconds(5)).unwrap(), &CancellationToken()?)?
stats = actors.stats()
print(stats.started, stats.stopped, stats.trapped, heaps_live(), heap_bytes_used())?
Ok(())
answer 49
worker stopped
worker trapped std.actor.TrapKind.Unwrap
3 2 1 0 0
The trap also prints its usual message to standard error, followed by
note: contained in an actor; its heap is discarded.
For TCP servers and clients, std.actor.net provides a
listener with one actor per connection, bounded connection buffers, deadlines,
and nonblocking connect and lookup. The repository’s
examples/actor_echo.plenty is an echo server built on it; a connection whose
first byte is X traps, and only that connection closes.
Actors and stages
Stage.node(address) borrows a live node shared between calls to run.
Use it to read counters or copy an owned snapshot outside the actor’s heap.
It returns Nothing after the actor stops or traps, including when another
actor has reused its slot. End the borrow before running the stage, sending
mail, stopping an actor or dropping the stage; the ordinary borrowing rules
prevent those operations while the node is borrowed. The accessor does not
switch heaps or run actor code.
run checks its deadline between scheduling rounds even when mail remains
queued. It does not preempt a running handler: a blocking call or an infinite
loop still prevents the stage from making progress.
Stage.shutdown() immediately stops every live actor, releases queued mail,
timers and tracked handles, and discards their heaps. It does not deliver
Stop or Down callbacks or restart actors, and does not need mailbox space
or timer capacity. Node destructors run under containment.
It is idempotent; the empty stage can start new actors afterward, with its
statistics retained and old addresses invalidated. Dropping a stage calls
shutdown automatically, including on early returns after cancellation or
errors. Application-level draining belongs before this final cleanup. Like
other actor code, a destructor that blocks or loops is not preempted.
A Stage[N, M, E] runs actors of one node type N, message type M and error
type E. N is a public class or enum with three public methods:
| Method | Called |
|---|---|
started(self: &mut N, context: &mut Context[M, E]) -> Result[(), E] | Once, before any mail |
receive(self: &mut N, mail: Mail[M, E], context: &mut Context[M, E]) -> Result[(), E] | For each delivered mail |
spawn(self, tag: i64, child: Address, context: &mut Context[M, E]) -> Result[Option[N], E] | On the parent, after a handler that called Context.spawn returns; builds the child with the child’s heap current. Nothing stops the child at once. See the spawn hook |
M and E must be copyable with copy; delivery copies them into another
heap. Use an enum node type to run several kinds of actor on one stage, as in
the example.
The spawn hook
The spawn hook runs the parent’s code while the child’s heap is current, so
everything it allocates belongs to the child. Two rules keep the parent’s heap
and the child’s apart:
- The hook takes
selfas a shared borrow. It cannot store a value in the parent’s node, so nothing it allocates outlives the child’s heap there, and it cannot move a value out of the parent’s node into the child. - The node type holds only values that a shared borrow neither aliases nor
changes: numbers,
bool,bytes,Bytes, files and sockets, functions,std.tlsconfigurations and sessions, and lists, dicts, sets, boxes, classes and enums of these. Astrcopy shares its storage andMutex.lockgives&mutthrough a shared borrow, so the compiler rejects a node holdingstr,Shared, a lock, an atomic, a channel, a closure or a task, naming the type and the field’s holder:
error: an actor node cannot hold `str` (in `Node`): its `spawn` hook reads the node while the child's heap is current; keep text in `Bytes` and pass shared state in messages
The hook builds the child’s node from copies of what it reads, and creates
values the child owns, such as an accepted socket, inside the hook. A hook
declared self: &mut N is rejected with cannot borrow shared reference as mutable in the standard library’s run_spawn.
In the hook, Context.spawn and Context.spawn_isolated return
StartError.Capacity, and Context.stop has no effect. With ordinary spawn,
an error or trap in the hook ends the parent: the hook is the parent’s code.
An error is copied to the parent’s heap before the child’s heap is discarded.
Use Context.spawn_isolated(tag, heap_limit) when construction itself belongs
to the child’s failure boundary, such as admitting a network connection. A
hook error or trap ends only the attempted child and delivers
Down(child, tag, Exit.Failed(error)) or Exit.Trapped(kind) to its parent.
Returning Nothing delivers Exit.Stopped. The notification does not require
room in the parent’s mailbox. A failed setup releases the child’s heap,
tracked handles and timers before another attempt can reuse its slot.
Stats.started includes these admitted attempts, even if they never reach
their started method.
Child exits follow the mail already queued for the parent when the child
stopped, including the child’s final messages. Newer mail cannot postpone
Down indefinitely: an acceptor can recover retired connection slots even
while new readiness keeps arriving.
Isolation also applies when the stage rebuilds an isolated child under a
restart policy. A replacement hook that fails or traps ends the child and
notifies its parent immediately; it is not retried recursively. A replacement
hook returning Nothing reports Stopped. Ordinary child handler failures
continue to follow the configured restart policy.
| Export | Contract |
|---|---|
stage[N, M, E](capacity, mailbox, events) | Result[Stage[N, M, E], StageError]: at most capacity live actors, mailbox messages per actor, events ready handles per wait. Allocates every slot table now |
Stage.start(node, limit: Positive[i64]) | Result[Address, StartError]: a root actor with a heap of limit bytes. Allocates its mailbox now |
Stage.send(to, message) | Result[(), Undelivered[M]] |
Stage.stop_actor(to) | bool: queues Mail.Stop; False when the actor has stopped or its mailbox is full |
Stage.run(&deadline, &cancel) | Result[(), WaitError]: delivers mail and readiness until no actor is live; fails when the deadline passes or cancel is cancelled first |
Stage.live() | Live actors |
Stage.stats() | Stats |
Stage.heap_used(actor) | Option[i64]: bytes in the actor’s heap, Nothing once it has stopped |
Stage.add_timers(count) | Result[(), AllocError]: allocates count more records for timers and pending requests; a stage starts with none |
Context.address() | The running actor’s Address |
Context.send(to, message) | Result[(), Undelivered[M]] |
Context.stop() | Stops the running actor after this handler returns |
Context.stop_actor(to) | As Stage.stop_actor |
Context.spawn(tag, limit: Positive[i64]) | Result[Address, StartError]: reserves a child slot, heap and mailbox now; the parent’s spawn hook builds the child after the handler returns |
Context.restart(child, limit, &window) | bool: restarts the child when it fails or traps, at most limit times per window; see Restart |
Context.after(&delay, tag), every(&period, tag) | Result[Timer, TimerError]: delivers Mail.Timer once after delay, or every period |
Context.cancel(timer) | bool: no Mail.Timer for it follows; False when it has already ended |
Context.request(to, message, &timeout) | Result[Request, RequestError[M]]: queues Mail.Request; exactly one Mail.Reply follows |
Context.reply(request, message) | Result[(), Undelivered[M]]: answers a request this actor received |
Context.heap_used() | Bytes in the running actor’s heap |
Context.watch(&handle, interest) | Result[Watch, WaitError]: delivers the handle’s readiness as Mail.Ready(watch, readiness); see Watching handles |
Context.unwatch(&handle) | Result[(), WaitError]: stops delivering it |
supported() | Whether this target has actor heaps |
heaps_live(), heap_bytes_used(), heap_bytes_committed() | Process totals over all actor heaps |
max_heap_limit | 64 MiB, the largest heap limit; Stage.start, Context.spawn and Context.spawn_isolated trap above it |
max_spawns | 16 spawn requests per handler |
max_capacity | 1,048,576, the largest stage capacity; stage returns StageError.Allocation(CapacityOverflow) for a negative capacity or one above it |
| Type | Values |
|---|---|
Address | A slot and a generation. A stopped actor’s address is never reused; key() returns it as u64 |
Mail[M, E] | Message(M), Ready(Watch, Readiness), Down(Address, i64, Exit[E]) (the child, its spawn tag, its exit), Stop, Timer(Timer), Request(Request, M), Reply(Request, Reply[M]) |
Timer | tag(), and key() as u64 |
Watch | key() as u64, to tell watched handles apart; copyable |
Request | key() as u64, to match a reply to its request |
Reply[M] | Answer(M), TimedOut, Gone (the receiver stopped or restarted first) |
TimerError | Capacity (every timer record is in use), Range (a zero period, or a time out of range) |
RequestError[M] | Full(M), Gone(M) as for Undelivered; Capacity(M): no free timer record |
Exit[E] | Stopped, Failed(E), Trapped(TrapKind) |
TrapKind | Memory, Index, Arithmetic, Unwrap, Other; see Traps |
Undelivered[M] | Full(M): the mailbox is full or the receiver’s heap cannot hold the copy. Gone(M): the receiver has stopped |
StageError | Allocation(AllocError) (slot tables, or a capacity that is negative or over max_capacity: CapacityOverflow), Wait(WaitError) (the readiness selector) |
StartError | Unsupported, Capacity (no free slot, or max_spawns reached), Heaps (1024 heaps live or the reservation failed), Mailbox (its allocation failed) |
Stats | started, stopped, failed, trapped, restarted, delivered, full (refused sends, and readiness deferred by a full mailbox), gone (sends and readiness for stopped actors), lost (Down for a parent that had stopped) |
Delivery
Each round delivers one mail to every actor that was runnable when the round
began, then waits on the stage’s std.ready selector without blocking if any
actor is still runnable. Readiness is level-triggered: an actor has at most one
Mail.Ready queued per handle, and readiness that finds a full mailbox is
reported again on a later wait. Due timers and ended requests wait outside the mailbox and
alternate with mailbox deliveries, so they never need mailbox room and neither
kind delays the other indefinitely.
Stop stops the actor after its handler returns. An Err(error) from any
handler stops the actor and gives its parent Exit.Failed(error), copied into
the parent’s heap; if that copy does not fit, the parent sees
Exit.Trapped(TrapKind.Memory).
Destructors run for a stopped or failed actor’s node.
Supervision
Every actor started by Context.spawn has a parent; the parent receives
Mail.Down(child, tag, exit) after the child’s last message, so a reply always
arrives before the Down that follows it. tag is the one the child was
spawned with, so a parent can tell kinds of children apart without recording
their addresses. Down never needs mailbox room and
is never refused: the child’s slot holds the exit until the parent receives
it, so a stopped child occupies one unit of capacity until then. When the
parent has already stopped, the exit is dropped and counted in Stats.lost. A
stopping actor’s undelivered mail is dropped, and so are its children’s
pending exits.
A parent can also call Context.spawn again on Down, or ask the stage to
restart the child.
Watching handles
Context.watch(&handle, interest) delivers a handle’s readiness to the running
actor as Mail.Ready(watch, readiness). It takes any
std.ready.Watchable handle: std.net’s TcpStream,
TcpListener, UdpSocket, Connecting and Lookup. The returned Watch is
the one the mail carries, so an actor watching several handles compares
watch.key() to find which is ready. Watching a watched handle again changes
its interest and returns the same Watch. Context.unwatch(&handle) stops
delivery, and so does closing the handle.
The key is built from the handle’s descriptor number and the actor’s slot, so
watching allocates nothing. A closed handle’s number can be reused by a later
handle, which then has the same key; readiness already queued for the closed
one can reach the new one, and its nonblocking calls return WouldBlock.
Arithmetic limits
Observation-only Stats totals saturate at i64.MAX; saturation never changes
mail delivery, admission, errors or cleanup. Slot and timer/request generations
never wrap: after an identity with generation u32.MAX ends, its slot or record
is permanently retired. A pending Down still retains its payload and is
delivered (or dropped during parent cleanup) before the exhausted slot is
forgotten. Later admission uses the existing capacity errors when no reusable
slot or record remains. Timer-record table sizes beyond their 32-bit key index
range return AllocError.CapacityOverflow before appending.
Mailbox ordering counters are not lifetime totals. Before their representation
would overflow, the stage subtracts the delivered prefix from both counters and
all of that mailbox’s pending Down fences, clamping only already-passed fences
to zero. Ready exits stay ready and future exits retain their exact remaining
prefix distance. No callback or allocation runs during rebasing. Clearing an
actor drains its queued messages and child exits before counters are reset on
slot reuse; restarting also clears those queues.
Timers, requests and restart
A lead actor asks a counting service for its count, crashes it, and asks
again. The request queued behind the crash is answered Gone, since the
restart drops the old actor’s mail; the retry reaches the restarted service,
whose count starts again:
from std.actor import Address, Context, Mail, Reply, StageError, stage
from std.ready import WaitError
from std.sync import CancellationToken
from std.time import deadline_after, milliseconds, monotonic_now, seconds
type Fault = AllocError | IoError | OverflowError[i64]
pub enum Job:
Get
Count(i64)
Crash
pub class Lead:
pub service: Option[Address]
pub answers: i64
pub enum Node:
Leading(Lead)
Counting(i64)
pub def started(self: &mut Node, context: &mut Context[Job, Fault]) -> Result[(), Fault]:
match self:
case Node.Leading(lead):
service = context.spawn(0, 65536).unwrap()
# Up to 3 restarts a minute.
context.restart(service, 3, &seconds(60))
context.request(service, Job.Get, &seconds(1)).unwrap()
lead.service = Some(service)
case Node.Counting(_):
pass
Ok(())
pub def receive(self: &mut Node, mail: Mail[Job, Fault], context: &mut Context[Job, Fault]) -> Result[(), Fault]:
match self:
case Node.Leading(lead):
service = lead.service.unwrap()
match mail:
case Mail.Reply(_, Reply.Answer(Job.Count(count))):
print("count", count)?
lead.answers += 1
if lead.answers == 1:
context.send(service, Job.Crash).unwrap()
context.request(service, Job.Get, &seconds(1)).unwrap()
else:
context.after(&milliseconds(10), 0).unwrap()
case Mail.Reply(_, Reply.Gone):
print("gone")?
context.request(service, Job.Get, &seconds(1)).unwrap()
case Mail.Timer(_):
context.stop_actor(service)
case Mail.Down(_, _, exit):
print("service", exit)?
context.stop()
case _:
pass
case Node.Counting(count):
match mail:
case Mail.Request(request, Job.Get):
*count += 1
context.reply(request, Job.Count(*count)).unwrap()
case Mail.Message(Job.Crash):
missing: Option[i64] = Nothing
missing.unwrap()
case _:
pass
Ok(())
pub def spawn(self, tag: i64, child: Address, context: &mut Context[Job, Fault]) -> Result[Option[Node], Fault]:
Ok(Some(Node.Counting(0)))
def main() -> Result[(), AllocError | IoError | StageError | WaitError]:
mut actors = stage[Node, Job, Fault](2, 4, 1)?
# Records for timers and pending requests.
actors.add_timers(4)?
actors.start(Node.Leading(Lead(Nothing, 0)), 65536).unwrap()
actors.run(&deadline_after(&monotonic_now(), &seconds(5)).unwrap(), &CancellationToken()?)?
stats = actors.stats()
print(stats.trapped, stats.restarted)?
Ok(())
count 1
gone
count 1
service std.actor.Exit[AllocError | IoError | OverflowError[i64]].Stopped
1 1
Timers
Context.after(&delay, tag) delivers Mail.Timer(timer) once, and
Context.every(&period, tag) every period starting one period from now;
timer.tag() is the caller’s tag. A tick still undelivered when the next
falls due absorbs it, and ticks missed while the stage was busy are skipped
rather than delivered late in a burst. Context.cancel(timer) ends a timer,
and no Mail.Timer for it is delivered afterwards. An actor’s timers end when
it stops.
Timers need records, which Stage.add_timers(count) allocates outside every
actor heap; a stage starts with none, and after, every and request return
Capacity when every record is in use. Starting, firing and cancelling a
timer allocate nothing. Stage.run waits on the selector until the earlier of
its deadline and the next due timer; the timer’s wake-up is not a timeout.
An unrepresentable now + delay returns TimerError.Range before reserving
or scheduling a record. Repeating timers skip missed ticks and clamp their
next deadline to the last representable instant. Mailbox/runnable ring indices
and timer-heap indices use bounded arithmetic, including at representation
limits, rather than relying on a potentially overflowing index sum.
Request and reply
Context.request(to, message, &timeout) copies message into the receiver’s
heap like send, queues it as Mail.Request(request, message), and returns
the Request. The receiver answers with Context.reply(request, answer),
which copies the answer into the requester’s heap and queues it as
Mail.Reply(request, Reply.Answer(answer)) in the requester’s mailbox, after
the messages the receiver sent before it. Exactly one Mail.Reply reaches the
requester for each request:
Answer(M)when the receiver replies in time;TimedOutwhentimeoutpasses first;Gonewhen the receiver stops or restarts first.
A late or second reply, a reply to a request sent to another actor, and a
reply after the requester stopped return Undelivered.Gone. A pending
request holds a timer record until its reply. Handlers do not block, so the
requester handles other mail while it waits. Compare request.key() to match
replies when several requests are pending.
A request timeout beyond the representable clock range is clamped to the last instant, rather than returning an arithmetic error or consuming its message on an overflow path.
Restart
Context.restart(child, limit, &window) makes the stage restart a child of
the running actor when it fails or traps. Restart is opt-in per child; limit
0 turns it off. On a restart:
- The child’s queued mail, timers and child exits are dropped, requests sent
to it are answered
Gone, and its heap is discarded. - A heap of the child’s original limit is reserved, and the parent’s
spawnhook builds the child again with its original tag. The address stays, so other actors can keep sending to it. - The new child runs
startedbefore any mail.Stats.restartedcounts it.
The parent receives no mail for a restart. At most limit restarts happen per
window, counted from the first restart in the window. After that, or when
the hook returns Nothing or no heap can be reserved, the child stops and the
parent receives Down with its exit. When the hook fails or traps, the parent
ends as for any hook error, and the child’s exit is counted in Stats.lost.
Restart suits service actors that the hook can rebuild from the tag. An actor built around a resource the hook took once, such as an accepted connection, cannot be rebuilt: its socket was closed with its heap. Children that a restarted actor had started keep running without a parent.
Memory accounting
Each heap reserves a 64 MiB address range and commits pages as its size
classes grow. A heap’s limit counts the bytes of its live blocks, each rounded
up to its size class; an allocation over the limit fails like any other
allocation, as AllocError where the program handles it. A handler that
returns that error fails its actor:
from std.actor import Address, Context, Mail, heap_bytes_used, heaps_live, StageError, stage
from std.ready import WaitError
from std.sync import CancellationToken
from std.time import deadline_after, monotonic_now, seconds
type Fault = AllocError | IoError | OverflowError[i64]
pub class Greedy:
pub size: i64
pub def started(self: &mut Greedy, context: &mut Context[i64, Fault]) -> Result[(), Fault]:
# 8 bytes for each of a million values do not fit in 64 KiB.
values: list[i64] = [0 for _ in range(self.size)]?
print("unreachable", len(values))?
Ok(())
pub def receive(self: &mut Greedy, mail: Mail[i64, Fault], context: &mut Context[i64, Fault]) -> Result[(), Fault]:
Ok(())
pub def spawn(self, tag: i64, child: Address, context: &mut Context[i64, Fault]) -> Result[Option[Greedy], Fault]:
Ok(Nothing)
def main() -> Result[(), AllocError | IoError | StageError | WaitError]:
mut actors = stage[Greedy, i64, Fault](1, 1, 1)?
actors.start(Greedy(1000000), 65536).unwrap()
actors.run(&deadline_after(&monotonic_now(), &seconds(5)).unwrap(), &CancellationToken()?)?
print(actors.stats().failed, heaps_live(), heap_bytes_used())?
Ok(())
1 0 0
An allocation failure the handler does not handle, such as one inside a
runtime operation that cannot report it, is contained like a trap.
std.crypto’s RSA operations reserve their memory in the heap first and
return Allocation when it does not fit; see
RSA allocation.
heap_bytes_used() is the sum of every heap’s used bytes and
heap_bytes_committed() the pages made writable; discarding a heap returns
both. At most 1024 heaps are live per process.
The stage’s own tables, every mailbox, and Context live outside all actor
heaps and hold no pointer into one, so discarding a heap cannot leave a
dangling pointer in them. The root node passed to Stage.start was built
outside its heap: whatever it already owns stays outside and leaks if the
actor traps. Build large state in started instead.
Traps
A trap in a handler, including an allocation failure that cannot be returned:
- discards the actor’s heap, returning its memory;
- closes up to 4096 files and sockets the actor opened, which also removes them from the stage’s selector;
- gives the parent
Exit.Trapped(kind); - does not run destructors for the trapped actor’s node or the handler’s locals, since they lived in the discarded heap.
kind is a TrapKind:
| Kind | Cause |
|---|---|
Memory | An allocation failure the program cannot handle, or an exit that does not fit the parent’s heap |
Index | A list, string, byte or slice index out of bounds, or a missing dictionary key |
Arithmetic | Integer overflow, division by zero, an invalid shift or a lossy cast |
Unwrap | unwrap on Nothing or Err |
Other | Any other runtime error, such as a failed assertion |
The message appears on standard error only; reporting the kind allocates nothing. These are not contained and still end the process:
- stack overflow;
- internal runtime panics;
- an allocation failure while no handler is running.
A lock or other process-wide state held at the moment of a trap stays as it was. Allocations made by C libraries through FFI, such as SQLite’s, are outside actor heaps and are neither limited nor freed by a trap.
Limits
- One thread. Heaps belong to the thread that created the first one, and heap operations from another thread fail. Starting a thread in a handler traps the actor, since the thread could outlive the heap holding its task.
- An actor node cannot hold
str; useBytes, which also accepts text. - Copying a message copies owned values. Where
copywould share storage instead, astris duplicated and any other shared value stops the program with a runtime error. - One message type per stage; a request and its reply are both
M. - Restart is one-for-one: a child’s restart does not stop or restart its siblings, and the parent is not stopped when the restart limit is reached.
- Files beyond 4096 per process are not tracked and stay open after a trap.
- Handlers are not preempted: a handler that loops forever stalls the stage.
- A stage holds at most
max_capacity(1,048,576) actors, since a watch key holds a 20-bit slot.
Acknowledgements
- Parents receiving their children’s exits follow Erlang/OTP supervisors; no Erlang code is used.
- Uses the Rust
libccrate for the Linux page-size query used by actor heaps (MIT OR Apache-2.0).
std.actor.net
TCP for std.actor: a listener that starts one actor per
connection, a connection with bounded buffers and deadlines, and connect and
name lookup that do not block the stage. A connection’s buffers are allocated
in its actor’s heap when it is admitted; handling readiness and timers
allocates nothing. A trap in a connection actor closes that connection only.
Supported target: x86-64 Linux, as for std.actor.
Connection cursor arithmetic is bounded by the allocated input/output buffers;
invalid consume/commit/forward counts retain their existing NetError causes.
Output reservation compares the requested count with remaining space without
first adding it to the cursor, so even an i64.MAX request cannot overflow that
comparison. Deadlines beyond the clock range clamp to the last instant.
Listener.refused() is an observation-only total and saturates at i64.MAX;
the live-child count remains an exact admission count. A matching Down when
no child is live returns NetError.Stream(StreamError.Io(IoError.InvalidInput))
before changing that count.
A server echoes one connection. It also starts a client as its own child, under a different tag, which dials it, sends a request and prints the reply:
from std.actor import Address, Context, Mail, StartError, TimerError, heaps_live, StageError, stage
from std.actor.net import Admission, Connection, Dialing, Limits, Listener, connection, dial, listener
from std.bytes import BytesError
from std.net import NetError, SocketAddress, listen, loopback_v4
from std.ready import WaitError
from std.sync import CancellationToken
from std.time import deadline_after, monotonic_now, seconds
type Fault = AllocError | NetError | BytesError | TimerError | StartError | IoError | OverflowError[i64]
# Spawn tags: the listener's connections and the client.
const connection_tag: i64 = 0
const client_tag: i64 = 1
pub enum Step:
Waiting(SocketAddress)
Dialing(Dialing)
Talking(Connection)
# Connects, sends one request, and prints the reply.
pub class Client:
pub step: Step
pub def started(self: &mut Client, context: &mut Context[i64, Fault]) -> Result[(), Fault]:
match &self.step:
case Step.Waiting(address):
self.step = Step.Dialing(dial(address, &seconds(5), context)?)
case _:
pass
Ok(())
pub def receive(self: &mut Client, mail: &Mail[i64, Fault], context: &mut Context[i64, Fault]) -> Result[(), Fault]:
match &mut self.step:
case Step.Dialing(attempt):
match attempt.handle(mail, context)?:
case Some(stream):
mut call = connection(stream, &Limits(64, 64, seconds(5), seconds(5)))?
call.start(context)?
call.write(b"hello, actors", context)?
call.shutdown_write(context)?
self.step = Step.Talking(call)
case Nothing:
pass
case Step.Talking(call):
events = call.handle(mail, context)?
if events.ended:
print("reply", call.input().to_text_range(call.input_start(), call.input_stop())?)?
call.close(context)?
context.stop()
case Step.Waiting(_):
pass
Ok(())
# Serves connections and stops once its one connection and the client have
# both stopped.
pub class Server:
pub listener: Listener
pub ended: i64
pub enum Node:
Serving(Server)
Echoing(Connection)
Calling(Client)
pub def started(self: &mut Node, context: &mut Context[i64, Fault]) -> Result[(), Fault]:
match self:
case Node.Serving(server):
server.listener.start(context)?
context.spawn(client_tag, 65536)?
case Node.Echoing(echo):
echo.start(context)?
case Node.Calling(client):
client.started(context)?
Ok(())
pub def receive(self: &mut Node, mail: Mail[i64, Fault], context: &mut Context[i64, Fault]) -> Result[(), Fault]:
match self:
case Node.Serving(server):
server.listener.handle(&mail, context)?
match mail:
case Mail.Down(_, _, _):
server.ended += 1
if server.ended == 2:
context.stop()
case _:
pass
case Node.Echoing(echo):
events = echo.handle(&mail, context)?
if echo.available() > 0:
echo.forward(echo.available(), context)?
if events.ended and echo.available() == 0:
echo.shutdown_write(context)?
if not echo.reading() and echo.pending() == 0:
context.stop()
case Node.Calling(client):
client.receive(&mail, context)?
Ok(())
# Builds each child by its tag: a connection's actor, with its buffers in
# the child's heap, or the client.
pub def spawn(self, tag: i64, child: Address, context: &mut Context[i64, Fault]) -> Result[Option[Node], Fault]:
match self:
case Node.Serving(server):
if tag == client_tag:
return Ok(Some(Node.Calling(Client(Step.Waiting(server.listener.local_address()?)))))
match server.listener.accept()?:
case Some(echo):
Ok(Some(Node.Echoing(echo)))
case Nothing:
Ok(Nothing)
case _:
Ok(Nothing)
def main() -> Result[(), AllocError | IoError | StageError | WaitError | NetError]:
socket = listen(&loopback_v4(0u16), 16)?
limits = Limits(4096, 4096, seconds(30), seconds(30))
server = listener(socket, limits, 100, 65536, connection_tag, Admission.Refuse)?
mut actors = stage[Node, i64, Fault](8, 4, 8)?
actors.add_timers(8)?
actors.start(Node.Serving(Server(server, 0)), 65536).unwrap()
actors.run(&deadline_after(&monotonic_now(), &seconds(10)).unwrap(), &CancellationToken()?)?
stats = actors.stats()
print(stats.started, stats.stopped, heaps_live())?
Ok(())
reply hello, actors
3 3 0
The repository’s examples/actor_echo.plenty is a longer echo server: a
connection whose first byte is X traps, and only that connection closes.
Listening
A Listener belongs to one actor, which watches the socket and starts a child
actor for each connection. Call start from the actor’s started and pass
every mail to handle. On readiness, handle spawns one child with the
listener’s tag and heap limit; the actor’s spawn hook
then calls accept, which accepts the connection and allocates its buffers in
the child’s heap. A Down from a child with the listener’s tag frees its
place; handle ignores readiness of other handles and other children’s
Down, so the actor can watch other handles and start other children under
other tags, as the client in the example above is.
At most capacity connections are live. A connection that arrives at capacity,
or when the stage has no free slot or heap, is handled by Admission:
Admission | At capacity |
|---|---|
Pause | Stops accepting until a connection ends. Arrivals wait in the kernel’s backlog; once it is full the kernel drops further attempts, so peers wait on their own connect timeouts |
Refuse | Accepts and closes the connection at once, so the peer sees it end without data. refused() counts them |
Pause suits short connections and load bursts; Refuse tells peers at once,
which suits clients that retry elsewhere. At the descriptor limit
(NetError.HandleLimit), both stop accepting until a connection ends.
Connections
A Connection owns its stream and two buffers sized by Limits. Call start
from the actor’s started, and pass every mail to handle; it reads, writes
and checks deadlines, and returns what happened as Events:
| Field | Meaning |
|---|---|
mine | The mail was about this connection |
received | Bytes this mail added to the input |
ended | The peer finished sending; no more input will arrive |
writable | Output space freed after write accepted less than it was offered |
drained | All output has been sent |
expired | Option[Expired]: a deadline passed; see Deadlines |
handle acts only on Mail.Ready for its own socket, so one actor can hold
several connections. Each mail reads at most once and
writes at most four times, so a fast connection cannot hold the stage.
Readiness is level-triggered, so unread input brings another Mail.Ready.
Reading
Received bytes are input()[input_start():input_stop()], read in place. Call
consume(count) once they are handled. When the input buffer is full the
connection stops reading, so the kernel’s buffers fill and the peer’s sends
block in turn; consume resumes reading. A request larger than Limits.input
must be handled in parts or refused.
Writing and back-pressure
write(source, start, stop) copies as much of source[start:stop] as fits
into the output buffer, sends what the socket accepts at once, and returns the
count accepted. A short count is back-pressure: keep the rest and offer it
again when Events.writable is set. forward(count) moves received input to
the output, for echoing and relaying. pending() and space() report the
output buffer.
Closing
shutdown_write ends output once pending bytes are sent; the peer then reads
end of input, and reading continues. close closes the socket at once,
discarding unsent output. An actor whose connection is done stops with
Context.stop; dropping the connection closes the socket. A reset or other
socket failure is returned by handle as an error, and returning it ends the
actor with Exit.Failed.
Deadlines
| Deadline | Set by | Ends when |
|---|---|---|
| Idle | Limits.idle | No bytes moved either way for this long |
| Stall | Limits.stall | Pending output made no progress for this long, such as a peer that stopped reading |
| Read | read_within(&span) | Input did not finish before clear_read_deadline; bounds how long a request may take to arrive |
A zero duration turns a Limits deadline off. handle reports a passed
deadline once in Events.expired; the actor decides what to do, usually
close and stop. Idle and stall then restart from now, and the read deadline
is cleared.
Each connection uses one actor timer, re-armed only when it
fires or a sooner deadline starts, so moving data does not create or cancel
timers. A stage needs add_timers with one record per connection that has a
deadline, and one per dial or lookup with a timeout.
Outbound connections and lookup
dial(&address, &timeout, context) starts a nonblocking connect and returns a
Dialing; lookup(host, port, &timeout, context) starts a name lookup and
returns a Resolving. Pass every mail to their handle, which returns
Nothing until the result is ready, then the stream or the addresses, or the
error. A zero timeout waits indefinitely; a passed one is NetError.TimedOut.
abandon stops waiting. Make a Connection from a dialled stream with
connection(stream, &limits).
The lookup runs the system resolver on a helper thread, as
std.net.resolve does, so the stage keeps running. Its
request and reply are allocated outside the actor’s heap, so a trap in the
actor while the lookup runs does not affect the helper thread. At most 32
lookups run at once in a process; another fails with TryAgain. Calling
std.net.resolve or connect in a handler also works, but blocks every actor
on the stage until it returns.
Memory
A connection actor’s heap holds its node, both buffers and 192 bytes of other
state, measured with Context.heap_used() after started: 2240 bytes with
1 KiB buffers and 8384 with 4 KiB buffers. The heap limit passed to
listener reserves address space; memory is committed as the heap is used.
The stage’s slot, mailbox and timer records are allocated by stage and
add_timers. Admission allocates the buffers; nothing else on the
connection’s path allocates. lookup allocates its result list in the actor’s
heap.
Exports
| Export | Contract |
|---|---|
Limits(input, output, idle, stall) | Buffer sizes in bytes (Positive[i64]) and deadlines; see Deadlines |
listener(socket, limits, capacity: Positive[i64], heap: Positive[i64], tag, admission) | Result[Listener, NetError]; traps for a heap above max_heap_limit |
Listener.start(context) | Result[(), NetError] |
Listener.handle(&mail, context) | Result[bool, NetError]: True for mail the listener used |
Listener.accept() | Result[Option[Connection], NetError | AllocError]: in the spawn hook; Nothing when no connection waits, and the hook should return Nothing |
Listener.live(), refused(), tag(), local_address() | Admitted connections not yet ended, connections refused, the child tag, the bound address |
connection(stream, &limits) | Result[Connection, AllocError]: buffers in the current heap |
Connection.start(context) | Result[(), NetError | TimerError] |
Connection.handle(&mail, context) | Result[Events, NetError | TimerError] |
Connection.input(), input_start(), input_stop(), available() | The input buffer and the unconsumed range |
Connection.consume(count, context) | Result[(), NetError | TimerError] |
Connection.write(&source[start:stop], context) | Result[i64, NetError | TimerError]: bytes accepted; Stream(StreamError.Io(IoError.Closed)) after shutdown_write or close, and InvalidInput for a range outside source |
Connection.forward(count, context) | Result[i64, NetError | TimerError]: bytes moved from input to output |
Connection.pending(), space() | Unsent output bytes; bytes write would accept now |
Connection.reserve(wanted: NonNegative[i64]), output(), output_stop(), commit(count, context) | Produce output in place: reserve compacts and returns Result[i64, NetError], the room up to wanted; write into output() from output_stop(); commit sends the count bytes written. For layers such as std.actor.tls |
Connection.acknowledge_now() | Acknowledge received data at once (TCP_QUICKACK), as TcpStream.acknowledge_now in std.net; best effort |
Connection.read_within(&span, context), clear_read_deadline() | Set or clear the read deadline |
Connection.shutdown_write(context), close(context) | End output after pending bytes; close at once |
Connection.reading(), closed(), stream() | Whether input may still arrive; whether closed; the stream |
dial(&address, &timeout, context) | Result[Dialing, NetError | TimerError] |
Dialing.handle(&mail, context) | Result[Option[TcpStream], NetError]: Refused, Unreachable, TimedOut and other connect errors |
lookup(host, port, &timeout, context) | Result[Resolving, NetError | TimerError] |
Resolving.handle(&mail, context) | Result[Option[list[SocketAddress]], NetError]: NameNotFound, TryAgain, TimedOut |
Dialing.abandon(context), Resolving.abandon(context) | Result[(), NetError]: stop waiting and cancel the timer |
| Type | Values |
|---|---|
Admission | Pause, Refuse |
Expired | Idle, Read, Write (the stall deadline) |
Events | As in Connections; copyable |
Limits
- TCP only;
std.actor.tlsadds TLS. - The input and output buffers have fixed sizes. Input is read in place, so a
message must fit in
Limits.inputto be handled whole. - A trap closes the actor’s sockets without a graceful shutdown; the peer sees a reset or end of input.
std.actor.tls
Plaintext space accounts for TLS record overhead without an overflowing
round-up addition, including for i64.MAX free ciphertext bytes. Cursor updates
are bounded by the offered slices and allocated plaintext/ciphertext buffers;
the existing TLS, network and timer errors are unchanged.
TLS for std.actor.net connections. A TlsConnection runs a
std.tls session over a Connection: it moves ciphertext between
the session and the connection’s buffers, decrypts into a bounded plaintext
buffer, and reports the same Events. Nothing waits, so one stage serves
many TLS connections. Certificates, verification and configuration are
those of std.tls; cryptography is rustls.
Supported target: x86-64 Linux, as for std.actor.
A server echoes one TLS connection, and a client actor dials it, sends a request once the handshake completes and prints the reply:
from std.actor import Address, Context, Mail, TimerError, heaps_live, StageError, stage
from std.actor.net import Admission, Dialing, Limits, Listener, connection, dial, listener
from std.actor.tls import TlsConnection, client, server
from std.bytes import BytesError
from std.net import NetError, SocketAddress, listen, loopback_v4
from std.ready import WaitError
from std.sync import CancellationToken
from std.time import deadline_after, monotonic_now, seconds
from std.tls import ClientConfig, ServerConfig, TlsError, client_config, roots_from_pem, server_config
type Fault = AllocError | NetError | BytesError | TimerError | TlsError | IoError
# Test-only CA and a localhost leaf it signed; never reuse a published key.
const CA_PEM: str = "-----BEGIN CERTIFICATE-----\nMIIBmjCCAT+gAwIBAgIULOQ2Qxr8CVZY2bG0uRYKKEdMD3swCgYIKoZIzj0EAwIw\nGTEXMBUGA1UEAwwOUGxlbnR5IFRlc3QgQ0EwIBcNMjQwMTAxMDAwMDAwWhgPMjEy\nNDAxMDEwMDAwMDBaMBkxFzAVBgNVBAMMDlBsZW50eSBUZXN0IENBMFkwEwYHKoZI\nzj0CAQYIKoZIzj0DAQcDQgAEkiB3Ogehrrt6YbQghwDPv6tkhE+z9zTCVih8fvtR\nZ1/691Nfh7/J9IgaNgNSTMJjGTolzGnjaCUDqoARlFyM+6NjMGEwHQYDVR0OBBYE\nFOTFBDgWXHj/qmVJDMlAHYCEZFX5MB8GA1UdIwQYMBaAFOTFBDgWXHj/qmVJDMlA\nHYCEZFX5MA8GA1UdEwEB/wQFMAMBAf8wDgYDVR0PAQH/BAQDAgIEMAoGCCqGSM49\nBAMCA0kAMEYCIQD41ATQnIjKkKMLQAb9AyoyOFmljYZyz348XJEIbqMgKwIhAL3m\nmWd8On5gvkfytspatH5mDWOcSyW+au7vuIC2gHhO\n-----END CERTIFICATE-----\n"
const LEAF_PEM: str = "-----BEGIN CERTIFICATE-----\nMIIBnDCCAUKgAwIBAgIBATAKBggqhkjOPQQDAjAZMRcwFQYDVQQDDA5QbGVudHkg\nVGVzdCBDQTAgFw0yNDAxMDEwMDAwMDBaGA8yMTI0MDEwMTAwMDAwMFowFDESMBAG\nA1UEAwwJbG9jYWxob3N0MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAE8THNQeTi\nJ6Eq0tSxtHW5Q4TCsbIqB3MOtM98ERyKgl6kHeHCeDZYKx8B+VRGT2jkWuTeN24Q\nljwUfh7uh7Iaz6N+MHwwGgYDVR0RBBMwEYIJbG9jYWxob3N0hwR/AAABMAkGA1Ud\nEwQCMAAwEwYDVR0lBAwwCgYIKwYBBQUHAwEwHQYDVR0OBBYEFEIFC2Fyd4cfAKTx\nMiNCMdDkqDNbMB8GA1UdIwQYMBaAFOTFBDgWXHj/qmVJDMlAHYCEZFX5MAoGCCqG\nSM49BAMCA0gAMEUCIAUF3HH43lnirgSN1Vr4twZrFmXKBxlbYNQni/h4ZaxhAiEA\n5rD/yE1B4IsbEGCMlyAeX507leXfGY9+wrHaxikU/w0=\n-----END CERTIFICATE-----\n"
const LEAF_KEY: str = "-----BEGIN PRIVATE KEY-----\nMIGHAgEAMBMGByqGSM49AgEGCCqGSM49AwEHBG0wawIBAQQglvSEE3H7v+mInxSF\nAyWFDKYpzpXr20wH8mfFY3bCaB2hRANCAATxMc1B5OInoSrS1LG0dblDhMKxsioH\ncw60z3wRHIqCXqQd4cJ4NlgrHwH5VEZPaORa5N43bhCWPBR+Hu6HshrP\n-----END PRIVATE KEY-----\n"
pub enum Step:
Waiting(SocketAddress)
Dialing(Dialing)
Talking(TlsConnection)
# Connects, sends one request once the handshake completes, and prints the
# reply.
pub class Caller:
pub config: ClientConfig
pub step: Step
pub sent: bool
pub def started(self: &mut Caller, context: &mut Context[i64, Fault]) -> Result[(), Fault]:
match &self.step:
case Step.Waiting(address):
self.step = Step.Dialing(dial(address, &seconds(5), context)?)
case _:
pass
Ok(())
pub def receive(self: &mut Caller, mail: &Mail[i64, Fault], context: &mut Context[i64, Fault]) -> Result[(), Fault]:
match &mut self.step:
case Step.Dialing(attempt):
match attempt.handle(mail, context)?:
case Some(stream):
net = connection(stream, &Limits(4096, 4096, seconds(5), seconds(5)))?
# The session is allocated in this actor's heap.
mut call = client(net, &self.config, "localhost", 1024)?
call.start(context)?
self.step = Step.Talking(call)
case Nothing:
pass
case Step.Talking(call):
events = call.handle(mail, context)?
# `writable` first reports the end of the handshake.
if events.writable and not self.sent:
self.sent = True
# The empty output takes all 14 bytes.
call.write(b"hello over TLS", context)?
call.shutdown_write(context)?
if events.ended:
print("reply", call.input().to_text_range(call.input_start(), call.input_stop())?)?
call.close(context)?
context.stop()
case Step.Waiting(_):
pass
Ok(())
pub class Server:
pub listener: Listener
pub config: ServerConfig
pub enum Node:
Serving(Server)
Echoing(TlsConnection)
Calling(Caller)
pub def started(self: &mut Node, context: &mut Context[i64, Fault]) -> Result[(), Fault]:
match self:
case Node.Serving(serving):
serving.listener.start(context)?
case Node.Echoing(echo):
echo.start(context)?
case Node.Calling(caller):
caller.started(context)?
Ok(())
pub def receive(self: &mut Node, mail: Mail[i64, Fault], context: &mut Context[i64, Fault]) -> Result[(), Fault]:
match self:
case Node.Serving(serving):
serving.listener.handle(&mail, context)?
match mail:
# This server handles one connection.
case Mail.Down(_, _, _):
context.stop()
case _:
pass
case Node.Echoing(echo):
echo.handle(&mail, context)?
# Forwarding frees input room, so more may be decrypted at once.
while echo.available() > 0:
if echo.forward(echo.available(), context)? == 0:
break
if not echo.reading() and echo.available() == 0:
echo.shutdown_write(context)?
if echo.pending() == 0:
context.stop()
case Node.Calling(caller):
caller.receive(&mail, context)?
Ok(())
# Builds each connection's actor, with its buffers and TLS session in
# the child's heap.
pub def spawn(self, tag: i64, child: Address, context: &mut Context[i64, Fault]) -> Result[Option[Node], Fault]:
match self:
case Node.Serving(serving):
match serving.listener.accept()?:
case Some(net):
Ok(Some(Node.Echoing(server(net, &serving.config, 1024)?)))
case Nothing:
Ok(Nothing)
case _:
Ok(Nothing)
def main() -> Result[(), AllocError | IoError | StageError | WaitError | NetError | TlsError]:
socket = listen(&loopback_v4(0u16), 16)?
address = socket.local_address()?
limits = Limits(4096, 4096, seconds(30), seconds(30))
# Include each connection's private TLS configuration and full records.
serving = Server(listener(socket, limits, 100, 262144, 0, Admission.Refuse)?, server_config(&LEAF_PEM, &LEAF_KEY)?)
caller = Caller(client_config(&roots_from_pem(&CA_PEM)?)?, Step.Waiting(address), False)
mut actors = stage[Node, i64, Fault](8, 4, 8)?
actors.add_timers(8)?
actors.start(Node.Serving(serving), 65536).unwrap()
actors.start(Node.Calling(caller), 262144).unwrap()
actors.run(&deadline_after(&monotonic_now(), &seconds(10)).unwrap(), &CancellationToken()?)?
stats = actors.stats()
print(stats.started, stats.stopped, heaps_live())?
Ok(())
reply hello over TLS
3 3 0
Connections
Make a TlsConnection from an admitted or dialled Connection with
server(net, &config, input) or client(net, &config, server_name, input),
where input is the size of the decrypted input buffer. Call them where the
connection’s actor heap is current, such as the spawn hook or a handler, so
the session lives in that heap. over(net, session, input) takes a session
made with std.tls.client_session or server_session.
Call start from the actor’s started, or once a dialled connection is
made; a client then queues its first handshake flight. Pass every mail to
handle. Its Events count plaintext:
| Field | Meaning |
|---|---|
mine | The mail was about this connection |
received | Decrypted bytes this mail added to the input |
ended | The peer sent close_notify |
writable | The handshake completed, or space freed after write accepted less than it was offered |
drained | All ciphertext has been sent |
expired | A deadline passed, as for Connection |
A TLS failure, such as an untrusted certificate or a peer’s alert, is
returned by handle as a TlsError, after the alert rustls queued is sent
if the socket takes it at once. A peer that ends its stream without
close_notify is Truncated, as in std.tls. Returning the error ends the
actor with Exit.Failed.
Reading
Decrypted bytes are input()[input_start():input_stop()], read in place.
consume(count) releases them. Releasing input room decrypts what waited,
so available() may grow again without new mail: handle input in a loop
until available() is 0, as the example’s echo does. When the plaintext
buffer is full the connection stops decrypting, its ciphertext buffer fills,
and the connection stops reading the socket, so back-pressure reaches the
peer as for std.actor.net.
reading() is false once close_notify or the end of the stream has
arrived.
Writing and back-pressure
write(source, start, stop) encrypts as much of source[start:stop] as
space() allows into the connection’s output, sends what the socket takes
at once, and returns the plaintext count accepted. space() is 0 during the
handshake, after shutdown_write, and while the session holds ciphertext
that did not fit the output; otherwise it is the free output less 32 bytes
of record overhead per 16 KiB. A short count is back-pressure: offer the
rest when Events.writable is set. forward(count) encrypts decrypted
input back out, for echoing and relaying. pending() is ciphertext not yet
sent.
Closing
shutdown_write queues close_notify, then ends the TCP write side once it
is sent; reading continues until the peer’s close_notify. close closes
the socket at once without close_notify. Stop the actor once
reading() is false, input is handled and pending() is 0.
Deadlines
Limits deadlines and read_within work as for
std.actor.net and count ciphertext progress. Set
read_within in started to bound the handshake and the first request, and
clear_read_deadline once established() is true.
Memory
| Item | Where | Size |
|---|---|---|
ClientConfig, ServerConfig, trust roots, builders | Their allocating heap (outside actors when built in main) | Owned trust anchors, configuration recipe and parsed configuration |
| Session | The connection actor’s heap | Its own parsed configuration, keys and rustls state; record buffers grow with record size |
| Ciphertext buffers | The connection actor’s heap | Limits.input and Limits.output |
| Plaintext input | The connection actor’s heap | The input argument |
Historical measurements before heap-local configuration reconstruction (issue #641),
using Context.heap_used() on an echo server: with 1 KiB buffers,
an established connection uses 12,944 bytes. After echoing 64 KiB it holds
17,168 bytes when the peer sent 1000-byte records, and 41,744 when it sent
full 16 KiB records, which rustls buffers whole. With 4 KiB buffers the
figures are 22,160 and 26,512 for 1000-byte records, and 50,960 for full
records. A peer controls its record size, so size the heap passed to
listener for full records: while a full record arrives rustls grows its
record buffer to a 32 KiB allocation, so 64 KiB per connection is not
enough. 80 KiB served full records with 1 KiB buffers, and 96 KiB with
4 KiB buffers. These historical figures and heap sizes exclude the configuration
graph now owned by each actor session. Measure the current application and its
certificate/root set before choosing a heap limit.
The HTTP server TLS comparison records current process RSS and handshake timings after reconstruction, with an ECDSA P-256 server certificate. Those resident-memory measurements do not replace the echo application’s per-actor heap measurements above.
rustls allocates through the Rust allocator, which in a handler is the
current actor’s heap. A session that exceeds its heap traps that actor
only, and its heap is discarded with the session; other connections
continue. Outside actors the same failure stops the program, as described
for std.tls.
Configurations keep an owned recipe: certificate chains and encoded private keys, trust anchors, revocation lists, application protocols, version and cipher choices. Opening an actor session reconstructs its complete rustls configuration from that recipe inside the connection actor’s heap. It retains no reference into the parent’s configuration, even during a handshake. The parent may drop or replace its configuration after construction without invalidating the session.
A trap, including allocation failure partway through reconstruction or a handshake,
discards the local graph with the heap. Cleanup does not call a destructor on
potentially interrupted rustls state. Repeated fresh configurations and trapped
sessions therefore do not pin retired configuration generations. Builders, roots
and configurations constructed inside an actor also belong to that actor’s heap.
An actor node may hold an immutable configuration and lend it to its spawn hook;
other std.tls handles cannot cross the actor boundary. The usual
Stage.start seed ownership limit still applies:
objects built outside the heap remain outside. If the root actor itself traps,
its prebuilt configuration can leak just like any other prebuilt owned resource.
Construct the root’s owned resources in started when its own trap recovery
must reclaim them; connection traps do not retain the root’s configuration.
This isolation costs a configuration reconstruction and private-key parsing for each actor session, and the session’s heap must include its trust anchors, certificates, keys and verification state. Large system root bundles or CRLs can substantially increase that cost. Outside actors, sessions still share the parsed configuration. Configurations themselves retain both a recipe and parsed state; building them validates the key and builds the parsed state. Retained encoded private keys are zeroized on normal destruction; trap cleanup discards actor pages.
Exports
| Export | Contract |
|---|---|
server(net, &config, input: Positive[i64]) | Result[TlsConnection, TlsError]: a server session in the current heap |
client(net, &config, server_name, input: Positive[i64]) | Result[TlsConnection, TlsError]: verifies the server against config and server_name |
over(net, session, input: Positive[i64]) | Result[TlsConnection, TlsError] |
TlsConnection.start(context) | Result[(), TlsError | TimerError] |
TlsConnection.handle(&mail, context) | Result[Events, TlsError | TimerError] |
TlsConnection.input(), input_start(), input_stop(), available() | The plaintext buffer and the unconsumed range |
TlsConnection.consume(count, context) | Result[(), TlsError | TimerError]; may decrypt more |
TlsConnection.write(&source[start:stop], context) | Result[i64, TlsError | TimerError]: plaintext accepted; Closed after shutdown_write or close |
TlsConnection.forward(count, context) | Result[i64, TlsError | TimerError]: plaintext moved from input to output |
TlsConnection.pending(), space() | Unsent ciphertext; plaintext write would accept now |
TlsConnection.read_within(&span, context), clear_read_deadline() | Set or clear the read deadline |
TlsConnection.shutdown_write(context), close(context) | Send close_notify then end output; close at once |
TlsConnection.established(), reading(), closed() | Handshake complete; input may still arrive; socket closed |
TlsConnection.session(), connection() | The Session, for the negotiated protocol, version and peer certificate; the Connection |
Limits
consumeandforwardmay decrypt more input without new mail; loop untilavailable()is 0.- Plaintext compaction uses
Bytes.copy_withinto move the unconsumed range in place without allocating. - No session resumption, as in
std.tls. std.actor.http.serverbuilds HTTP/1.1 over these sessions; its qualification is separate.
std.actor.http.server
HTTP/1.1 with one supervised actor per connection, over plain TCP or TLS. The server uses the actor networking, actor TLS and resumable HTTP decoder modules. Application callbacks run on one thread. A contained trap discards the connection’s heap and closes its tracked sockets; admission and unrelated connections continue.
The qualification record separates tested behavior from outstanding production-readiness gates. Qualification currently targets x86-64 GNU/Linux. HTTP/2 and HTTP/3 are outside this implementation.
Choosing a server
Both HTTP servers remain supported, with different execution contracts.
| Choice | std.http.server | std.actor.http.server |
|---|---|---|
| Ordinary handlers | Sequential Exchange operations, shared handler | Complete bounded Request, mutable handler per connection |
| Streaming | Sequential reads and writes on workers | Events, partial writes and explicit resumable state |
| Scheduling | Bounded worker pool; idle heads wait off workers | One nonblocking stage; callbacks run to completion |
| Contained failures | Returned errors; runtime traps end the process | Returned errors, handler traps and actor heap exhaustion |
| Existing helpers | Router, uploads, files, WebSockets and SSE | HTTP framing and response primitives; application dispatch |
Choose using the application’s failure model, I/O needs and measured workload. The comparison harness does not assume that either implementation is faster. The actor server does not make blocking database or HTTP-client operations nonblocking.
Complete requests
Start with the runnable actor HTTP lesson. The simple entry points are:
| Entry point | Contract |
|---|---|
serve(listener, factory, &config, &stop) | Serve plain TCP until shutdown; returns Result[Report, ServerError] |
serve_tls(listener, tls, factory, &config, &stop) | The same lifecycle using an owned std.tls.ServerConfig |
config() | Default configuration, without starting a server |
The server takes ownership of the listener and factory, and of a supplied TLS
configuration. A failed setup releases these resources; they are not returned
for retry. This differs from a queue’s failed submission: serve owns the
server lifecycle, including cleanup after an error. Configuration and the stop
token are borrowed for the call.
factory is a named function returning Result[H, FE]. The server calls it
inside each connection’s heap. H implements RequestHandler[E]:
def handle(self: &mut H, request: Request, response: &mut Response) -> Result[(), E]
The handler persists across requests on that connection. It is not shared
across connections. Factory and handler error types may differ. Its stored
fields must obey the actor heap boundary:
use owned Bytes for persistent text, not str, shared owners or locks.
Closures with captured state do not cross that boundary. Build connection
state in the factory instead of allocating it outside the actor heap.
Request owns head: RequestHead, target: RequestTarget, body: Bytes and
trailers: list[Header]. These values are temporary callback inputs. The
complete request body is buffered up to config.limits.max_body, subject also
to the actor’s heap limit. Route using the validated target and method; the
worker server’s Exchange and stored Router are not actor state.
Use Response.send_text, send, or respond for a whole bounded response.
It must fit in pending_output together with its encoded head and framing
reserve. Larger bodies use the streaming interface.
Returning without a response produces 500; an unfinished response is
finished before the callback completes, which can fail on a length mismatch.
Configuration and bounds
| Field | Default | Meaning |
|---|---|---|
capacity | 256 | At most this many connection actors; accepted range 1–1000 |
heap_limit | 4 MiB | Budget per connection; accepted range 256 KiB–64 MiB |
input_buffer, output_buffer | 4096 each | TCP ciphertext or plaintext transport buffers; at least 64 bytes |
pending_output | 65536 | Application response queue, including its head; at least 512 bytes |
limits | std.http.limits() | Maximum line, head, header count and decoded request body |
idle_timeout | 5 s | Waiting for a request |
head_timeout | 10 s | Receiving a request head |
body_timeout | 30 s | Receiving its body |
write_timeout | 10 s | Stalled transport output |
response_timeout | 60 s | Completing a response |
handshake_timeout | 10 s | Establishing TLS |
drain_timeout | 5 s | Grace period before forced stage shutdown |
max_requests | 1000 | Maximum requests per connection, a Positive[i64] |
Zero phase durations disable those phase timers. A zero drain duration instead
requests immediate teardown. The independent heap, buffer and protocol limits
all apply: increasing max_body alone does not increase available memory.
The runtime has at most 1024 live actor heaps per process, including the server’s
acceptor and any other stages. Capacity is not a promise that admission can
always reserve another heap or descriptor.
At capacity or when admission cannot reserve a child, the server closes a newly
accepted connection. It does not promise an HTTP 503 response for overload.
HTTP parse refusals use bounded lingering close so unread request bytes do not
immediately turn a completed response into a TCP reset. Its private lingering
state records only whether the 16 MiB drain threshold has been reached, not an
unbounded byte total; transport consumption still uses the full available input.
Errors, reports and shutdown
ServerError distinguishes invalid configuration and loss of the acceptor,
and preserves allocation, byte-buffer, HTTP, network, TLS, timer, selector,
actor-start and time-range causes in named variants. Application failures are
classified as Handler; application error values are not sent to the peer.
Report application-specific diagnostics before returning them when needed,
using an operation that respects the nonblocking callback contract.
A handler Err produces a generic 500 when a response has not started.
After output has started, an error closes the connection rather than adding a
second response. A trap closes the connection without a recoverable HTTP
answer. Isolated construction covers failures before the handler starts too.
It does not retry a partially handled request.
Report.actors is the underlying actor Stats; it includes the acceptor and
admitted construction attempts. requests counts completed responses,
rejected counts admission refusals, and protocol_errors counts reported
parse failures. These observation-only counters saturate at u64.MAX;
completion credit, admission and shutdown still progress normally at that limit.
A selector or setup failure returns Err instead of a report.
A connection submits one completion at a time and waits for admission to
acknowledge it before starting another request. A full admission mailbox uses
a private 1 ms retry timer instead of runnable retries on every scheduling
round. Live application messages and timers continue to dispatch while it
waits; retirement retains pending completion and shutdown control. If the
shared timer pool is exhausted, runnable retries preserve control delivery.
Admission retains at most one pending acknowledgement per connection and retries
it when the child’s mailbox is full. This bounds the added queue by capacity;
the stage reserves one additional timer record per connection. Phase deadlines
and the external forced-drain deadline continue to apply.
Notice.CompletedBy(Address) submits completion credit and Notice.Completed
returns it. These variants, Resume, Rejected and Drain are reserved for
server control; applications send Notice.Message(value). The additional enum
variant requires updating exhaustive matches over Notice.
Cancel the external stop token, or call Response.shutdown_server() from an
authorized application route, to start draining. Admission stops, idle
connections close, and active responses get the configured grace period.
The external driver then calls stage shutdown,
even if application mailboxes or timers are full. Early error returns also
drop the stage and reclaim its live heaps. Use std.signal.cancel_on
outside the stage to turn process signals into cancellation.
Reliability boundary
Callbacks and destructors are not preempted. Blocking I/O or an infinite loop can stall the whole stage, including its shutdown checks. Stack overflow and internal runtime panics remain process-fatal. Traps do not roll back external effects or run user destructors; foreign-library allocations and process-wide locks have the actor runtime’s limits.
TLS sessions reconstruct their configuration inside the connection actor’s heap; traps reclaim it without retaining retired configurations. Configuration rotation and repeated fresh server lifecycles can therefore release old TLS configurations. Size connection heaps for the local certificate, key, trust-anchor and CRL graph; see TLS memory ownership. WebSocket upgrades, dedicated actor SSE helpers, blocking-work offload and multithreaded stages are not provided by this module.
Scheduler adapter
Node[F, H] is an opaque actor-dispatch adapter with private state. Its public
started, receive and spawn methods implement the lifecycle expected by
std.actor, using Context[Notice[M], ServerError]
and returning ServerError. spawn returns an optional child node. These
methods advance the server’s admission or connection state; applications use
the serve entry points instead of constructing or driving nodes directly.
The type and lifecycle methods are public because the existing generic actor
dispatcher checks their visibility across module boundaries. This is a
visibility exception, not a new raw runtime hook.
Streaming actor HTTP
std.actor.http.server offers serve_events and
serve_tls_events for requests or responses that should progress in bounded
parts. They take the same listener, factory, configuration and stop token as
the complete-request entry points, and return Result[Report, ServerError].
The TLS form additionally owns a std.tls.ServerConfig.
The factory creates one mutable Handler[M, E] per connection. Its callbacks
and helper functions retain their own arithmetic contracts: include escaping
integer arithmetic errors in their declared result errors, independently of the
server’s ServerError contract.
def receive(self: &mut H, event: Event[M], response: &mut Response,
context: &mut Context[Notice[M], ServerError]) -> Result[(), E]
Events and stored state
| Event | Meaning |
|---|---|
Head(head, target) | Validated request head and target |
Body(bytes) | Owned decoded body bytes, bounded by the decoder’s per-feed budget |
Trailers(fields) | Validated chunked-body trailer fields |
End | The complete request body has arrived |
Writable | Response output progressed; offer previously unaccepted bytes again |
Message(value) | A user Notice.Message(value) sent through the actor context |
Actor(mail) | Unconsumed actor readiness, user timer, request/reply or child-exit mail |
Stopping | One notification when this connection starts draining; finish pending work promptly |
Keep owned byte buffers, numeric state and actor-safe connection objects in
the handler. Text in a head or trailer event is for that callback; copy needed
text into Bytes before returning. Events are owned, so the handler decides
what body bytes to retain, consume or drop. Retained application data counts
against the connection’s heap budget.
Context uses Notice[M] as its message type and ServerError as its actor
error type. Wrap user messages in Notice.Message. Other Notice variants
belong to server lifecycle/accounting and are not application commands.
Timer tags below zero are reserved by the server; use nonnegative tags for
application timers. Event.Actor preserves the original Mail, so existing
std.actor.net and std.actor.tls state machines
can consume readiness and timers without reconstructing them.
For an outbound operation, retain a Dialing or Connection in the handler,
start it from a request callback, and pass its Event.Actor(mail) values to
the networking object’s handle method with the supplied context. Watch
the relevant handle and keep application transitions bounded. A blocking
std.http.client.fetch inside the callback still blocks every actor.
Custom child spawning from a connection handler is not supported; the server’s
spawn hook is for connection admission. Application timers share a bounded
pool and can return TimerError.Capacity.
Writing responses
| Operation | Contract |
|---|---|
start(head) | Validate and queue an owned response head; Result[(), ServerError] |
write(source: &[u8]) | Queue part of source; returns accepted byte count or ServerError |
finish() | Complete framing; a declared length must match; Result[(), ServerError] |
respond(head, &body) | Queue and finish one whole response; it must fit |
send(status, &content_type, &body) | Whole response with an automatically encoded head |
send_text(status, &text) | Whole UTF-8 plain-text response |
pending() | Bytes waiting in the application queue, excluding transport/TLS buffers |
space() | Payload space presently available, after framing reserve |
is_started(), is_done(), is_stopping() | Whether framing began, finished, or shutdown was requested |
close_after() | Disable reuse after this response |
shutdown_server() | Start server-wide draining; this response gets the shutdown grace period |
All whole-response operations return Result[(), ServerError]. start and
respond consume their heads; a failed call does not return the head for retry.
Byte inputs are borrowed and accepted bytes are copied into the bounded queue.
A successful write returns 0 <= count <= len(source). A short successful
write is back-pressure. Retain the suffix and resume it on Writable; do not
call finish until every intended byte was accepted. For an output offset
with 0 <= offset <= body.len(), adding the accepted count from body[offset:]
stays within that same bound. Update the offset only after a successful write.
Do not spin on a zero-byte write. Transport progress drives subsequent events,
and request decoding pauses when response capacity is exhausted. Application
state needs its own bound too: a callback must not keep appending to a separate
buffer while the response is blocked.
start accepts a Content-Length header, or selects chunked output for
HTTP/1.1 when the length is absent. HTTP/1.0 uses close-delimited output in that
case. Do not supply Transfer-Encoding yourself. respond supplies the
content length and rejects caller-supplied framing headers. Bodyless statuses
and HEAD obey their framing rules; HEAD consumes offered body bytes without
sending them. Invalid lengths and sequencing return HTTP errors.
A response can begin while request bytes are still arriving. The server keeps
delivering body events under back-pressure, but conservatively closes that
connection after the response rather than reusing it. Finishing a response
before request End abandons the unread body and uses bounded lingering close.
For reusable ordinary request/response traffic, finish reading the request
before starting the response.
The server bounds decoder advancement per actor turn. A handler must likewise
bound its own work; actor heap isolation does not impose a CPU-time budget.
Direct 101 upgrades and the worker server’s Exchange-based WebSocket/SSE
adapters are not supported by this interface.
Actor HTTP qualification
This record covers std.actor.http.server.
Its core HTTP/TLS implementation has passed the checks recorded below.
It is not yet labelled production-ready: the platform/sanitizer scope
below is deliberately limited. TLS configuration retention after traps (issue #641)
is addressed by heap-local reconstruction. The worker server’s qualification
is separate evidence and remains in force for that implementation.
Gates
| Gate | Current evidence |
|---|---|
| Shared protocol grammar | Resumable decoder reuses existing head/target/framing rules; focused decoder and existing HTTP regression suites passed during development |
| Admission isolation | Actor regression tests cover constructor error/trap/empty result, restart-constructor failure, full parent mailbox, and heap/timer/descriptor recovery |
| Bounded teardown | Actor tests cover full-mailbox shutdown, idempotence, reuse, and repeated live-stage drop with trapping node destructors |
| Real HTTP/TLS connections | test_std_actor_http_server passed with runtime/allocation checks, including TLS close notification, keep-alive and nonblocking outbound TCP |
| Corpus and deterministic mutations | test_std_actor_http_corpus passed: 80 shared requests sent in 1-byte and 64-byte writes, plus 600 mutations |
| Coverage-guided fuzzing | Incremental decoder: 1,688,343 libFuzzer executions in 61 seconds, seed 592601, zero findings; coverage instrumentation, not ASan/TSan. Whole actor lifecycle is covered by socket tests rather than this target |
| Slow clients and output back-pressure | Socket tests passed for partial heads/bodies, handshakes and slow readers |
| Shutdown under load and signals | Forced drain with a busy mailbox passed; SIGTERM completes an in-flight streamed response; HTTP/TLS application shutdown is tested, including a TLS peer omitting close notification |
| Repeated failure/resource stability | Repeated handler traps, heap exhaustion, factory failures and 5,120 connections at concurrency 16 pass; fresh-configuration trap/normal-drop/construction-OOM regressions check actor heaps and process allocation balance |
| TLS and reverse proxies | curl 8.18.0, nginx 1.30.6 and Caddy 2.11.7 passed streamed/sized/Expect uploads, chunked replies, keep-alive, repeated 413 refusals and proxy TLS verification |
| Installed/relocated toolchain | Copied compiler alone into an empty environment, compiled HTTP/TLS applications, removed compiler and source, then ran both native binaries through body/echo/trap/recovery/shutdown probes successfully |
| Comparative measurements | Original HTTP/TLS comparison: 540,000 successful requests; refreshed TLS comparison: 270,000 successful requests and 9,000 verified handshakes after issues #641/#644; actor mixed-fault checks passed |
This table records evidence rather than inheriting a pass from a shared component. The B85 backlog entry owns remaining work. The design records the two-server goal and acceptance rationale. The remaining platform scope is x86-64 GNU/Linux; no macOS, ARM, ASan or TSan result is implied by these checks.
The actor-specific suites are in tests/test_std_actor_http_server.rs and
tests/test_std_actor_http_corpus.rs; decoder tests are in
tests/test_std_http_decoder.rs. Run the integrated suites with
cargo test --locked --all-features --test integration test_std_actor_http_
and the decoder with the test_std_http_decoder:: filter. These results do
not establish behavior on untested platforms.
Performance comparison
examples/http_server_bench.py selects the worker or actor implementation,
with corresponding plain and TLS fixtures. It uses the same small response,
16 KiB response, POST echo, keep-alive and new-connection workloads. Compiler
hash, revision, source, CPU affinity, settings, failures, throughput, latency,
CPU time and memory are recorded with the results. Failed or incomplete
requests make the driver fail; they are not a clean throughput result.
For an equal-core comparison, use one worker and one actor stage on the same
server CPU, with the client on a separate CPU. Run each at least three times,
alternating implementations against fresh processes. Compare the ordinary
four-worker configuration separately; it has a different CPU budget. Match
idle connection counts explicitly (--idle 984 fits both), and keep load-test
concurrency within both admission limits. Use --tls for HTTPS.
Performance under faults is a separate measurement: combine healthy traffic with trapped handlers, heap exhaustion, slow clients and overload, and record healthy-request latency, recovery, lost connections and resource growth. Clean-request throughput alone does not measure the actor server’s purpose. Machine load and client bottlenecks must accompany any reported comparison.
The refreshed TLS comparison records current throughput, handshake latency, idle memory and trap recovery after issues #641 and #644; the original comparison below remains historical evidence.
Recorded comparison
These measurements predate the heap-local TLS configuration reconstruction for issue #641. Their source/compiler provenance and results remain unchanged. They do not establish current TLS setup costs or memory use; actor sessions now own a parsed configuration and rebuild their keys and verification state on construction. Plain HTTP does not use that path.
Measured on 2026-11-01 at source revision 12c021c9, with compiler SHA-256
53e6b1c791bd104ba9795a739b91211025d2b10201ed757ca0e053cfc5ccc8d4.
Native fixtures were compiled with --release, on a Ryzen 7 7840HS,
Linux 7.2.11 x86-64. One-worker and actor servers used CPU 0; four workers used
CPUs 0, 2, 4 and 6 (four physical cores). Clients used separate physical cores
8, 10, 12 and 14. No compilation, qualification suite or fault load ran alongside
these measurements. Machine load snapshots are retained in the
raw results and reproduction sources.
Each configuration ran three times in alternating worker/actor order against fresh processes: 5,000 requests per workload per run, with 984 connections for the separate idle measurement. These are short loopback measurements, not sustained deployment-capacity claims. ApacheBench measured ordinary workloads; four curl processes measured new TLS connections. The result artifact contains all sample ranges, p50/p99 latency, server CPU time, RSS, failures and settings.
Median requests/second (all measured requests completed successfully):
HTTP
| Workload | One worker / one core | Actor / one core | Four workers / four cores |
|---|---|---|---|
| Small, 1 keep-alive client | 21,779 | 30,078 | 23,433 |
| Small, 16 keep-alive clients | 25,545 | 27,703 | 96,802 |
| Small, 64 keep-alive clients | 24,515 | 13,597 | 82,129 |
| New connection, 16 clients | 15,160 | 12,215 | 15,035 |
| 16 KiB response, 16 clients | 18,575 | 13,402 | 72,903 |
| 1 KiB POST echo, 16 clients | 24,215 | 25,071 | 98,404 |
HTTPS
| Workload | One worker / one core | Actor / one core | Four workers / four cores |
|---|---|---|---|
| Small, 1 keep-alive client | 19,195 | 26,605 | 21,324 |
| Small, 16 keep-alive clients | 22,568 | 25,430 | 79,376 |
| Small, 64 keep-alive clients | 20,663 | 11,976 | 65,670 |
| New connection, 16 clients | 3,796 | 4,109 | 4,285 |
| 16 KiB response, 16 clients | 14,902 | 11,859 | 53,654 |
| 1 KiB POST echo, 16 clients | 20,939 | 23,104 | 81,723 |
The actor wins several small-response cases on one core, while the worker wins at 64 clients and for 16 KiB responses. Four worker cores provide much higher concurrent throughput. HTTP p99 at 16 small-response clients was 1.013 ms for one worker, 1.625 ms for the actor, and 0.286 ms for four workers; a throughput improvement need not improve tail latency.
Median incremental idle memory per connection was 0.86 KiB (one worker), 21.73 KiB (actor), and 1.13 KiB (four workers) for HTTP. With TLS it was 41.07, 45.86 and 41.35 KiB respectively. These are resident-memory differences for this fixture, not the configured maximum actor heap reservation.
The 64-client actor HTTP samples recorded roughly 617,000 full-mailbox retry attempts for 5,000 completed responses, without lost messages or failed HTTP requests. These are the original measurements before the completion-credit fix for issue #644; the retry comparison records the changed behavior. The raw actor statistics also distinguish abrupt TLS peer closure from an HTTP request failure: curl/ApacheBench can receive the complete response and close without TLS close notification.
Reproduce a single configuration from the repository root; change
--implementation actor to --implementation worker --workers 1, or use
--implementation worker --workers 4 --server-cpus 0,2,4,6, and add --tls
for HTTPS:
python3 examples/http_server_bench.py --compiler target/debug/plenty \
--implementation actor --server-cpus 0 --client-cpus 8,10,12,14 \
--requests 5000 --idle 984 --runs 3 --output target/actor-http-comparison
For the recorded samples, each invocation used --runs 1 and the six
configurations alternated for three rounds. Adapt CPU affinity to the machine.
Completion retry comparison
Issue #644 replaces repeated runnable completion retries with one outstanding completion per connection, an admission acknowledgement, and a private 1 ms retry after a full admission mailbox. Retrying a full child acknowledgement is bounded by connection capacity; application mail and timers remain live.
The raw comparison freezes the
module before the change at 5fad016c, the changed module, the shared fixture,
and a replay script. Both snapshots were imported as the same local server
module and compiled with the same compiler/runtime and --release;
only the fixture’s module imports changed. The compiler SHA-256 is
7da9d5da6d3bbab8a5ae08ea18f5d8fa7ce755e38f6cb0bedf0654cc82875dcf.
This is a controlled HTTP comparison on x86-64 GNU/Linux, not a new TLS or
worker-server qualification. CPU affinities remain server 0, clients
8,10,12,14. Three alternating before/after runs of all six workloads completed
180,000 measured requests with zero failures or missing responses. Each run’s
actor statistics include its additional /stop request and final drain;
requests was exactly 5,001.
| HTTP workload | Before median requests/s | After median requests/s |
|---|---|---|
| Keep-alive, 1 client | 27,524 | 27,664 |
| Keep-alive, 16 clients | 26,087 | 34,867 |
| Keep-alive, 64 clients | 12,054 | 31,511 |
| New connection, 16 clients | 10,812 | 10,449 |
| Keep-alive 16 KiB response, 16 clients | 12,228 | 13,422 |
| Keep-alive 1 KiB POST echo, 16 clients | 23,155 | 29,999 |
At 64 clients, median full-mailbox attempts fell from 616,899 to 6,707 and mail deliveries from 319,895 to 26,840. This improves throughput, but p99 latency rose from 12.464 to 16.854 ms in this workload. The retry delay and credit exchange are a throughput/latency tradeoff; this does not establish an across-the-board latency improvement. Incremental idle RSS with 984 connections was 21.73 KiB before and 21.80 KiB after per connection; setup RSS and raw runs are in the artifact. No task-local builds ran during measurement; unrelated activity on the shared host was not controlled, and the recorded load snapshot was taken after the run. Short local runs do not establish long-term soak behavior or precise performance across environments.
Replay the frozen sources from a checkout with a compiler, ApacheBench and
taskset; adapt the script’s affinities on other machines:
python3 - <<'PYREPLAY'
import json
from pathlib import Path
record = json.loads(Path("book/src/assets/actor-http-retry-comparison.json").read_text())
Path("/tmp/actor-http-retry-comparison.py").write_text(record["reproduction_script"])
PYREPLAY
python3 /tmp/actor-http-retry-comparison.py \
book/src/assets/actor-http-retry-comparison.json target/debug/plenty \
target/actor-http-retry-comparison
Mixed failures
The separate fault workload placed 32 stalled request heads beside four healthy clients and 100 intentional handler traps in each of four batches. All 4,000 healthy requests and 400 post-batch recovery requests succeeded on the actor server; all 400 trapping connections ended without an HTTP response. RSS stayed at 4,136 KiB and descriptor count at six after every batch. The server then answered its shutdown request and exited successfully, with zero rejected admissions or lost messages. Healthy batch p99 was 1.20–1.28 ms.
Both worker configurations passed their ordinary and stalled-head baselines, then exited on the first intentional runtime trap, as their execution contract specifies. This comparison demonstrates trap containment, not immunity to arbitrary native faults. CPU 6 and client CPU 7 are SMT siblings in this separate probe, so its latency is descriptive failure evidence rather than a throughput comparison. The result artifact embeds the temporary fault harness and its two fixtures, along with compiler and native-binary hashes.
Actor HTTP TLS comparison
Current matched measurements for std.http.server and
std.actor.http.server. The
actor qualification record retains the original
comparison and the platform limits.
Measured on 2026-11-01 at source revision c397a845, after the private TLS
configuration reconstruction (issue #641) and completion-credit changes
(issue #644). The raw results and replay sources
retain all samples, source fixtures, the driver, a reproduction script, an
independent validation script, compiler/native hashes and machine load snapshots.
Compiler SHA-256:
13c6ed9514305a6d67592417265034555283bb7e8635188dcddc77091c907aa6.
The compiler was built with cargo build --locked --bin plenty; native fixtures
used --release --compile with PLENTY_CACHE=off.
The Ryzen 7 7840HS host ran Linux 7.2.11 x86-64 GNU/Linux. One worker and the single actor stage used CPU 0; four workers used physical cores 0, 2, 4 and 6. Clients used physical cores 8, 10, 12 and 14. Three rounds alternated worker/actor/four-worker order, reversing it in the middle round. Every workload started a fresh server, with 5,000 requests at the stated concurrency. All 270,000 measured requests completed with zero failures or missing responses. Idle measurements used 984 connections in each configuration. No task-local builds or search indexing ran during measurement; unrelated shared-host activity was uncontrolled. The host’s one-minute load average was 2.88 before and 2.72 after the complete run.
HTTPS throughput and latency
Median requests/second across three samples:
| Workload | One worker / one core | Actor / one core | Four workers / four cores |
|---|---|---|---|
| Small, 1 keep-alive client | 18,876 | 25,956 | 20,925 |
| Small, 16 keep-alive clients | 21,874 | 34,379 | 89,255 |
| Small, 64 keep-alive clients | 19,955 | 28,641 | 64,121 |
| New connection, 16 clients | 3,788 | 3,794 | 4,465 |
| 16 KiB response, 16 clients | 14,279 | 13,332 | 51,283 |
| 1 KiB POST echo, 16 clients | 21,051 | 30,171 | 84,747 |
The actor leads the equal-core small-response and echo cases in these samples; one worker leads the 16 KiB case, and four worker cores have substantially higher concurrent throughput. New-connection throughput is nearly equal between the single-core configurations. This is a current comparison of the implementations, not an isolated before/after measurement of either fix.
Median sample p99 latency in milliseconds (new-connection latency includes TLS and the HTTP response):
| Workload | One worker | Actor | Four workers |
|---|---|---|---|
| Small, 1 keep-alive client | 0.092 | 0.064 | 0.085 |
| Small, 16 keep-alive clients | 1.576 | 0.579 | 0.443 |
| Small, 64 keep-alive clients | 28.137 | 22.583 | 19.484 |
| New connection, 16 clients | 5.313 | 5.394 | 4.324 |
| 16 KiB response, 16 clients | 1.817 | 1.441 | 0.575 |
| 1 KiB POST echo, 16 clients | 0.981 | 0.629 | 0.367 |
The 64-client p99 values are high for all configurations. The three p99 samples range from 24.349–32.037 ms for one worker, 22.431–22.710 ms for the actor, and 16.729–24.064 ms for four workers. Short loopback runs on a shared host do not establish a service latency objective or sustained deployment capacity. ApacheBench drives keep-alive workloads; four parallel curl processes drive new connections, without certificate verification, as in the historical comparison.
TLS handshake and idle memory
A separate Python probe made 1,000 new connections at concurrency 16 per sample,
three samples per configuration: 9,000 successful verified TLS handshakes and
validated HTTP responses. It used one shared client SSL context loaded with the
fixture CA. Every session negotiated TLS 1.3 with TLS_AES_256_GCM_SHA384 and
reported no session reuse. Timing begins after TCP connect and ends when the
client’s TLS handshake returns, before the HTTP request. It includes client
verification, scheduling and library overhead; it does not isolate server CPU
cost or prove that the server has consumed the client’s Finished message.
| Measurement | One worker | Actor | Four workers |
|---|---|---|---|
| Median sample handshake p50, ms | 1.782 | 3.630 | 1.383 |
| Median sample handshake p99, ms | 3.489 | 5.443 | 5.192 |
| Median incremental idle RSS, KiB/connection | 41.12 | 57.80 | 41.42 |
The actor fixture uses 16.68 KiB more incremental idle RSS than one worker, including its private per-session configuration, for this ECDSA P-256 certificate fixture. This comparison does not isolate configuration memory from other actor overhead. Idle RSS is a process resident-memory difference with connection capacity raised to 1,000, not an actor heap high-water mark or a general heap-sizing recommendation. The historical 45.86 KiB actor TLS figure predates the ownership change. Different certificate chains, root sets, record sizes and application state need their own measurement.
TLS failures and recovery
A separate descriptive probe combined 32 stalled TLS request heads, four healthy
clients and 100 intentional handler traps per batch. Across four actor batches,
all 4,000 healthy requests and 400 post-batch recovery requests succeeded;
all 400 trap requests ended without a successful response. Healthy-request p99,
including TLS setup, ranged from 1.699 to 2.957 ms. After each batch RSS was
5,128 KiB and the descriptor count was six; before fault injection RSS was
5,124 KiB. Shutdown succeeded with exactly 4,501 completed requests (including
100 baseline requests and /stop), 400 trapped actors, zero rejected admissions
and zero lost messages. Abrupt client TLS closes also increment actor failure
counts; they do not mean those complete HTTP responses failed.
Both worker configurations completed their 100-request baselines, then exited with status 1 on the first handler trap; neither served recovery requests. This demonstrates the actor containment contract for these injected traps. It is one fault run per configuration, with a fixed root TLS configuration, not a configuration-rotation soak, an allocation-failure test, or a new platform qualification. The actor server’s production-readiness limits remain unchanged.
Replay from the recorded source revision with ApacheBench, curl, taskset and a compiler built there. Extract the scripts from this artifact before switching to that checkout, then pass its path and compiler to the replay script:
python3 - <<'PYREPLAY'
import json
from pathlib import Path
record = json.loads(Path("book/src/assets/actor-https-comparison.json").read_text())
Path("/tmp/actor-https-replay.py").write_text(record["reproduction_script"])
Path("/tmp/actor-https-validate.py").write_text(record["validation_script"])
PYREPLAY
TLS_CHECKOUT=/path/to/checkout-of-c397a845
python3 /tmp/actor-https-replay.py "$TLS_CHECKOUT" \
"$TLS_CHECKOUT/target/debug/plenty" /tmp/actor-https-replayed
python3 /tmp/actor-https-validate.py /tmp/actor-https-replayed/results.json
Adapt CPU affinities to the host. Keep newly recorded results separate from the published artifact; numerical RSS stability and negotiated-cipher assertions in the validation script describe this fixture and platform, and may need adjustment elsewhere.
std.archive
import std.archive reads and writes TAR and ZIP archives over
std.io streams, and extracts them into a new directory under a
std.fs Root. It has one error type, ArchiveError.
The module is ordinary Plenty over std.bytes,
std.compress (DEFLATE and CRC-32) and std.fs; it adds no
runtime code. It is available on the supported x86-64 Linux GNU target.
import std.archive
import std.bytes
import std.io
import std.path as path
def main() -> Result[(), IoError | AllocError | std.archive.ArchiveError | std.bytes.BytesError]:
mut writer = std.archive.tar_writer(std.io.cursor(std.bytes.empty()?))?
writer.add_dir(&path.from_text("docs")?, 493, 1700000000)?
readme = std.bytes.from_text("read me first")?
writer.add_file(&path.from_text("docs/README")?, &readme, 420, 1700000000)?
writer.finish()?
mut reader = std.archive.tar_reader(std.io.cursor(writer.inner.data.copy()?))?
while True:
match reader.next()?:
case Some(entry):
print(entry.name.to_string()?, entry.kind, entry.size)?
case Nothing:
break
Ok(())
docs/ std.archive.EntryKind.Directory 0
docs/README std.archive.EntryKind.File 13
Functions
| Function | Result | Contract |
|---|---|---|
tar_reader(inner: R) | Result[TarReader[R], ArchiveError] | A streaming TAR reader over any Reader |
tar_writer(inner: W) | Result[TarWriter[W], ArchiveError] | A streaming TAR writer over any Writer |
zip_reader(inner: R, max_entries: NonNegative[i64]) | Result[ZipReader[R], ArchiveError] | Reads the central directory of a seekable inner |
zip_writer(inner: W) | Result[ZipWriter[W], ArchiveError] | A ZIP writer over any Writer; it never seeks |
policy() | Policy | The default extraction limits |
extract_tar(archive: &mut TarReader[R], parent: &Root, destination: &Path, policy: &Policy, cancel: &CancellationToken) | Result[Summary, ArchiveError] | Extract every remaining entry into the new directory parent/destination |
extract_zip(archive: &mut ZipReader[R], parent: &Root, destination: &Path, policy: &Policy, cancel: &CancellationToken) | Result[Summary, ArchiveError] | Extract every entry into the new directory parent/destination |
stream_error(error: ArchiveError) | StreamError | How a reader reports error through the Reader protocol |
Entries
pub enum EntryKind:
File
Directory
Symlink
Hardlink
Other
pub class Entry:
pub name: Path
pub kind: EntryKind
pub size: i64
pub compressed: i64
pub mode: i64
pub mtime: i64
pub link: Path
name holds the raw bytes stored in the archive; it is not checked or
decoded until extraction. size is the length of the content, compressed
its length in the archive (equal to size in TAR), mode the Unix
permission bits (0 to 4095, or 0 for ZIP entries made on other systems),
and mtime the modification time in Unix seconds. link is a TAR link
target; ZIP stores a symbolic link’s target as its content. Other covers
devices, FIFOs, and unknown TAR types.
TarReader
TarReader[R] reads POSIX ustar, pax, GNU, and V7 archives. inner is
public.
| Method | Result | Contract |
|---|---|---|
next() | Result[Option[Entry], ArchiveError] | The next entry, skipping unread content; Nothing at the end |
read(buffer: &mut Bytes, start: i64, stop: i64) | Result[i64, StreamError] | The current entry’s content; 0 at its end |
failure() | Option[ArchiveError] | The failure that ended the archive, if any |
Format rules:
- Each header’s checksum is verified; old writers’ signed sums are accepted.
A mismatch is
Checksum. - Numbers are octal, or GNU base-256 for values beyond 8 GiB. Any other
digit, a negative base-256 value, or an overflow is
Corrupt. - Long names and link targets come from the ustar prefix, from GNU
LandKentries, and from paxpathandlinkpathrecords; paxsizereplaces the header size. Other pax keys and global pax headers are ignored. A pax header or GNU long name larger than 64 KiB isLimit. - Only regular files and unknown types have content; links, directories, and devices have none, whatever their size field says.
- The archive ends with two zero blocks. Input that ends before them is
Truncated; one zero block followed by anything else isCorrupt.
A malformed header ends the archive: next and read repeat the error, and
failure names it. An error from inner is returned unchanged and leaves
the reader usable, so Interrupted may be retried.
TarWriter
TarWriter[W] writes POSIX ustar headers with user and group 0 and no
owner names. A name or link target longer than 100 bytes is stored in a
pax extended header; a size beyond 8 GiB uses GNU base-256.
| Method | Result | Contract |
|---|---|---|
add_file(name: &Path, data: &[u8], mode: i64, mtime: i64) | Result[(), ArchiveError] | A regular file holding data |
add_from(name: &Path, size: i64, source: &mut S, mode: i64, mtime: i64) | Result[(), ArchiveError] | A regular file of exactly size bytes read from source |
add_dir(name: &Path, mode: i64, mtime: i64) | Result[(), ArchiveError] | A directory |
add_symlink(name: &Path, target: &Path, mtime: i64) | Result[(), ArchiveError] | A symbolic link; target is stored as given |
finish() | Result[(), ArchiveError] | Write the end blocks and flush inner |
Names are normalised: empty and . components are dropped, and directories
get a trailing /. A name that is empty after that, absolute, or contains
.., NUL, or a backslash is UnsafePath, so a writer cannot produce an
archive that extraction would refuse. A mode outside 0 to 4095, a negative
size or time, a link target that is empty or contains NUL, or any call
after finish is InvalidInput. A source that ends early in add_from is
Truncated and leaves the archive unusable.
An error while writing an entry or the end blocks to inner, including a
Truncated source, leaves the archive unusable: that call returns the error,
and every later add_* or finish call returns InvalidInput without
writing.
ZipReader
ZipReader[R] needs a stream with seek, such as FileStream or Cursor.
zip_reader locates the end record, reads the whole central directory, and
keeps one Entry per member; more than max_entries entries is Limit.
inner is public.
| Method | Result | Contract |
|---|---|---|
len() | i64 | The number of entries |
entries() | &list[Entry] | Every entry, in central-directory order |
open(index: i64) | Result[(), ArchiveError] | Select entry index for read |
read(buffer: &mut Bytes, start: i64, stop: i64) | Result[i64, StreamError] | The selected entry’s content; 0 at its end |
failure() | Option[ArchiveError] | The failure that ended the selected entry, if any |
Format rules:
- The end record must be the last one whose comment reaches the end of the input exactly. ZIP64 end records are read when a count, size, or offset is saturated, and ZIP64 extra fields supply each entry’s sizes and offset.
- Spanned (multi-disk) archives are
Unsupported. - Entries may not overlap: each entry’s local header, name, and data must fit
before the next entry’s local header or the central directory. Overlapping
entries, the basis of some ZIP bombs, are
Corrupt. - The local header must repeat the central name. Only methods 0 (stored)
and 8 (DEFLATE) are read; others and encrypted entries are
Unsupported. - The content must be exactly the declared size, its CRC-32 must match
(
Checksum), and a DEFLATE stream must end exactly at the declared compressed size (Corrupt). - An entry is a directory when its name ends in
/or its attributes say so; Unix attributes (host 3) also givemode, symbolic links, and special files. Times are DOS times in local time, read as UTC.
open clears an earlier failure. A content failure ends the entry: read
repeats it and failure names it. CRC and size are checked when the content
ends, so treat bytes as unverified until read returns 0. Errors from
inner are returned unchanged, as in TarReader.
ZipWriter
ZipWriter[W] writes each entry’s header, then its data, and the central
directory at finish. It never seeks, so it can write to a pipe.
| Method | Result | Contract |
|---|---|---|
add_file(name: &Path, data: &[u8], method: Method, mode: i64, mtime: i64) | Result[(), ArchiveError] | A regular file; Method.Deflate compresses at level 6, Method.Stored does not |
add_dir(name: &Path, mode: i64, mtime: i64) | Result[(), ArchiveError] | A directory |
add_symlink(name: &Path, target: &Path, mtime: i64) | Result[(), ArchiveError] | A symbolic link |
finish() | Result[(), ArchiveError] | Write the central directory and end records, and flush inner |
Names and link targets follow the TarWriter rules. A name with non-ASCII
bytes that is valid UTF-8 sets the UTF-8 flag. Attributes are Unix (host 3) with mode.
Times are stored as DOS times, clamped to 1980 through 2107, with
two-second resolution. ZIP64 fields are written for an entry whose size or
offset reaches 2³²−1, and ZIP64 end records when there are 65535 or more
entries or the central directory lies beyond 2³²−1.
An error while writing an entry or the directory to inner leaves the
archive unusable: that call returns the error, and every later add_* or
finish call returns InvalidInput without writing.
Offsets and entry counts remain exact i64 values. An entry or final directory
whose complete output extent cannot be represented returns
Stream(StreamError.OutOfRange) before new I/O, directory mutation, or the
failed-writer latch. This preflight failure leaves the writer usable and
preserves previously written output; it does not wrap or saturate progress.
Extraction
extract_tar and extract_zip create destination, a single name, under
parent. Nothing appears at destination until every entry has been
extracted:
- A staging directory named
.plenty-extract-and 32 random hex digits is created underparent. - For each entry, in archive order: check
cancel, the entry count, and the name; refuse or skip links and special files; check the size limits; then create the directory or write the file inside the staging directory. Files are written throughstd.fs.copy_from, so each is staged anonymously and published with a no-replace link. - The staging directory is renamed to
destinationwithout replacing anything. An existingdestinationisExists.
On any error the staging tree is removed and the first error returned.
Removal allocates; if it fails too, the staging directory keeps its
.plenty-extract- name for the caller to remove. Resource bookkeeping is exact
and includes newly created intermediate directories, not just archive entries.
Before creating a directory or copying a file, extraction checks that its
creation count and the two cleanup traversal slots remain representable;
exhaustion is Limit before that creation/copy. Cleanup still preserves the
original extraction error.
Every path is resolved through std.fs.Root handles that refuse symbolic
links, so an entry cannot reach outside the staging directory. Name rules:
- An absolute name, a
..component, NUL, or a backslash isUnsafePath. - Empty and
.components are dropped; a file whose name is then empty isUnsafePath, and such a directory is ignored. - A name longer than
max_namebytes, or with more thanmax_depthcomponents, isLimit. - Two entries with the same file name, or a file and a directory with the
same name, are
Duplicate. Repeated directory entries are allowed. - Names are byte strings; they are not decoded or normalised.
Files are created with mode 0600 and directories with 0700. Stored modes, times, and owners are not applied.
Policy
pub class Policy:
pub max_entries: i64
pub max_entry_size: i64
pub max_total_size: i64
pub max_ratio: i64
pub max_depth: i64
pub max_name: i64
pub skip_links: bool
pub skip_special: bool
| Field | Default | Meaning |
|---|---|---|
max_entries | 10000 | Entries of any kind, including skipped ones |
max_entry_size | 1 GiB | Bytes in one file |
max_total_size | 4 GiB | Bytes in all files |
max_ratio | 100 | Largest size / compressed for ZIP entries over 1 MiB |
max_depth | 32 | Components in a name, 1 to 62 |
max_name | 4096 | Bytes in a name |
skip_links | False | Skip symbolic and hard links instead of failing with Link |
skip_special | False | Skip devices, FIFOs, and unknown types instead of failing with Special |
Sizes are checked against the declared size before any byte is written, and
a file that turns out longer or shorter than declared fails. A field out of
range is InvalidInput. The ratio check applies to ZIP entries only: a
compressed TAR stream is bounded by its std.compress decompressor’s limit.
Summary reports entries (all entries), files, bytes (file content
written), and skipped.
Errors
pub enum ArchiveError:
Stream(StreamError)
Io(IoError)
Fs(FsError)
Allocation(AllocError)
Corrupt
Checksum
Truncated
Unsupported
UnsafePath
Link
Special
Duplicate
Exists
Limit
Cancelled
InvalidInput
| Variant | Meaning |
|---|---|
Stream(cause) | inner or a source stream failed |
Io(cause) | A directory operation failed |
Fs(cause) | A std.fs failure with no closer variant |
Allocation(cause) | A buffer, name, or list could not be allocated |
Corrupt | Malformed archive structure |
Checksum | A TAR header checksum or ZIP CRC-32 mismatch |
Truncated | The input ended inside the archive |
Unsupported | Encryption, a compression method other than stored or DEFLATE, or a spanned ZIP |
UnsafePath | A name that could escape the destination |
Link | A link entry, when skip_links is false |
Special | A device, FIFO, or unknown entry, when skip_special is false |
Duplicate | A name already extracted |
Exists | destination already exists |
Limit | A policy or format limit was exceeded |
Cancelled | cancel was set |
InvalidInput | A bad argument, policy, or destination, or a writer call after finish or a failed write |
stream_error maps Stream(cause) to cause, Truncated to
UnexpectedEof, Allocation(cause) to Allocation(cause), Io(cause) to
Io(cause), and everything else to Io(IoError.InvalidInput).
Ownership and threads
Readers and writers own inner and their buffers; dropping one releases them
and does not flush or close inner. None can be copied. The module has no
global state, passes the ordinary worker-effect checks, and may be used from
any thread.
Costs
| Value | Allocates |
|---|---|
TarReader, TarWriter | One 512-byte block |
ZipReader | Up to 64 KiB to find the end record, one Entry and record per member, a 16 KiB input buffer, and an Inflater (about 62 KiB) |
ZipReader.open | A fresh Inflater for a DEFLATE entry |
ZipWriter | The central directory, about 50 bytes plus the name per entry |
ZipWriter.add_file | The compressed copy of data for Method.Deflate |
| Extraction | Each entry’s name components, one directory handle per level, and an 8 KiB copy buffer per file |
Reading content allocates nothing; time is linear in the archive size. A TAR archive is read once, front to back; a ZIP archive is read once for the central directory and once per opened entry.
Limits
- No link creation: links are refused or skipped.
- No restoring of modes, times, or owners.
- No appending to an existing archive or rewriting entries.
- No streaming ZIP writes: each
add_fileholds the whole entry in memory, and data descriptors (sizes after the data) are not written. - No ZIP encryption, and no methods other than stored and DEFLATE.
- No sparse TAR files, and no TAR archives split across volumes.
- TAR reader content lengths are at most 2⁶² bytes, keeping block padding
representable. PAX decimal fields accept nonnegative values through
i64.MAX(9223372036854775807); larger values and invalid digits areCorrupt. This numeric range does not raise the content-length limit. Octal accumulation subtracts ASCII bias before adding a digit; fixed-block checksums do not wrap. - An unrepresentable PAX output record length is
Allocation(AllocError.CapacityOverflow)before appending the record. - One target, x86-64 Linux GNU.
Acknowledgements
- Streaming readers and writers are modelled on Go’s
archive/tarandarchive/zip, with Python’starfileandzipfileas further references.
std.atomic
Inline atomic integers and booleans that scoped workers update through shared
references. Exports Atomic[T], Ordering, and fence. They are not prelude
names: each use needs an import, through any item or module alias. Supported
target: x86_64 Linux GNU.
from std.atomic import Atomic, Ordering
def work(hits: &Atomic[u64]) -> ():
for _ in range(1000):
hits.fetch_add(1, Ordering.Relaxed)
def main() -> Result[(), SpawnError[()] | IoError]:
hits = Atomic[u64](0)
with spawn(work, &hits)?, spawn(work, &hits)?:
pass
print(hits.load(Ordering.Acquire))?
Ok(())
2000
Atomic[T]
T is bool, i8, i16, i32, i64, u8, u16, u32, or u64; any
other element type is a compile-time error. Atomic[T](value) returns an
Atomic[T] directly. Atomic(value) takes T from the expected type, such
as an annotation or a field, or else from the value.
An atomic is stored inline wherever its owner stores values: a local, a class field, a tuple or enum payload, or a collection element. Construction and every operation allocate nothing and call no runtime function.
An atomic is affine: it moves and cannot be copied. It has no equality, no
ordering, and no formatting; read its value with load. *r cannot read it and
*r = ... cannot replace it. Moving or reassigning an unborrowed atomic is
ordinary.
Methods take the receiver by shared borrow, so a &Atomic[T] given to several
workers can update it. The receiver must be a place: a name, field, element, or
* dereference. An operation on a temporary is rejected, except into_inner.
| Method | Result | Notes |
|---|---|---|
load(order) | T | |
store(value, order) | () | |
swap(value, order) | T | previous value |
compare_exchange(current, new, success, failure) | Result[T, T] | Ok(previous) when it equaled current and was replaced, else Err(actual) |
fetch_add(value, order), fetch_sub | T | integer only; overflow traps before storing |
fetch_wrapping_add(value, order), fetch_wrapping_sub | T | integer only; wraps |
fetch_and, fetch_or, fetch_xor, fetch_nand | T | |
fetch_max(value, order), fetch_min | T | integer only; compares by T’s signedness |
fetch_not(order) | bool | bool only |
into_inner() | T | consumes an owned atomic |
Every fetch_ method returns the previous value. fetch_add and fetch_sub
use the ordinary integer overflow trap at the call’s position. They compute
the checked result from the value last read and store it with a
compare-exchange, retrying if another thread changed the value, so no thread
observes a wrapped value. They trap on the value they read even if another thread
changes it afterwards: that is a valid ordering of the operations, and it keeps
the guarantee that no wrapped value is stored, unlike Rust’s wrapping
fetch_add. The other methods lower to one atomic instruction,
or a compare-exchange loop where the target has none. No operation blocks or
takes a lock; a compare-exchange loop retries while other threads keep changing
the value, so it is lock-free but not wait-free.
Ordering
Ordering is an enum: Relaxed, Release, Acquire, AcqRel, SeqCst.
Each ordering argument is written as Ordering.<variant>, through any alias. A
variable, call, or other expression is a compile-time error, as is an ordering
the operation does not allow:
| Operation | Allowed orderings |
|---|---|
load | Relaxed, Acquire, SeqCst |
store | Relaxed, Release, SeqCst |
| Read-modify-write methods | all five |
compare_exchange success | all five |
compare_exchange failure | Relaxed, Acquire, SeqCst |
fence | Release, Acquire, AcqRel, SeqCst |
The orderings mean what they mean in C++ and Rust. Every operation currently
compiles to a sequentially consistent instruction, which is at least as strong
as any requested ordering, so a program cannot rely on a weaker ordering being
weaker. Ordering values can also be stored and matched like any enum value.
fence
fence(order) -> () orders memory operations around it without an atomic
location. It compiles to a full sequentially consistent fence.
Sharing
Scoped threads borrow an atomic, or a class holding atomics, for the task’s
scope. The parent may keep calling methods on it while the workers run. Executor
jobs own their inputs, so they share an atomic through a
Shared[Atomic[T]] handle: each job
receives its own handle.share() and calls methods on it directly.
Not provided: compare_exchange_weak (identical to compare_exchange on the
supported target), fetch_update (write a compare_exchange loop),
get_mut, as_ptr, compiler-only fences, and pointer atomics.
For a lesson see count with atomics; the architecture page describes the lowering.
Acknowledgements
- Operations and orderings follow Rust’s
std::sync::atomic.
std.bytes
import std.bytes provides Bytes, an owning contiguous buffer that stores one
byte per element, and BytesError. Use Bytes for
checked byte-range operations and byte-oriented I/O. A list[u8] also stores
one byte per element and provides ordinary list operations and element
references; its capacity is rounded to allocation units as described under
list storage costs.
Bytes are never text: converting to str is an explicit, validated operation.
The module is embedded Plenty code over a private native storage owner. It is available on the supported x86-64 Linux GNU target.
Constructing
| Function | Result | Contract |
|---|---|---|
std.bytes.empty() | Result[Bytes, AllocError] | Empty buffer with no reserved capacity |
std.bytes.with_capacity(capacity) | Result[Bytes, AllocError] | Empty buffer with exactly capacity bytes reserved; negative is CapacityOverflow |
std.bytes.from_list(values: &list[u8]) | Result[Bytes, AllocError] | Independent copy of the list’s bytes |
std.bytes.from_text(text: &str) | Result[Bytes, AllocError] | Independent copy of the exact UTF-8 bytes |
std.bytes.from_bytes(source: &[u8]) | Result[Bytes, AllocError] | Independent copy of a byte view: a literal, Bytes, list[u8], or part of one |
Bytes has no public constructor and its storage field is private. Every
constructor allocates and can report Allocation.
A byte literal such as b"\r\n" has the separate, immutable prelude type
bytes, which lives in read-only data and never allocates. Operations that only
read another byte sequence take a &[u8] view, so they accept a
literal, a buffer, a list, or part of any of them without copying.
len(buffer) calls its shared len method and returns the number of bytes,
just as len does for a byte literal or byte slice view. Membership operators
are unsupported on Bytes; use the explicit search methods below.
Operations
start, stop, index, offset, count, and sizes are i64; elements are u8.
| Method | Receiver | Result | Contract |
|---|---|---|---|
len() | shared | i64 | Number of bytes |
is_empty() | shared | bool | Whether the length is zero |
capacity() | shared | i64 | Bytes storable without reallocating |
get(index) | shared | u8 | Read one byte |
find(value, start, stop) | shared | Option[i64] | First index of value within the range |
find_in(set, start, stop) | shared | Option[i64] | First index holding any byte of set |
find_not_in(set, start, stop) | shared | Option[i64] | First index holding no byte of set: the end of a run |
find_bytes(needle, start, stop) | shared | Option[i64] | First index where all of needle occurs inside the range |
count(value, start, stop) | shared | i64 | Occurrences of value within the range |
matches(offset, source) | shared | bool | Whether source occurs at offset |
matches_text(offset, text: &str) | shared | bool | Whether the UTF-8 bytes of text occur at offset |
ascii_run(start, stop) | shared | i64 | Index after the run of bytes below 128 that begins at start |
utf8_run(start, stop) | shared | i64 | Index after the longest valid UTF-8 that begins at start |
slice(start, stop) | shared | Result[Bytes, AllocError] | Independent copy of a range, with exact capacity |
copy() | shared | Result[Bytes, AllocError] | Independent copy of the whole buffer |
to_list() | shared | Result[list[u8], AllocError] | Independent list[u8] for the transport builtins |
to_text() | shared | Result[str, BytesError] | Strictly validated UTF-8 copy; otherwise InvalidUtf8 |
to_text_range(start, stop) | shared | Result[str, BytesError] | to_text of one range, validated and copied once with no intermediate Bytes; a range that splits a UTF-8 sequence is InvalidUtf8 |
set(index, value) | exclusive | () | Replace one byte |
write_float(value: f64) | exclusive | i64 | Write shortest scientific ASCII into existing bytes at zero; requires len() >= 32 and traps otherwise; returns bytes written, never allocates or changes length |
write_float_spec(value: f64, style: i64, precision: i64, offset: i64) | exclusive | i64 | Write a Python format-spec spelling at offset for std.fmt: style 1 fixed, 2 exponent, 3 general, 4 general keeping .0, 5 f64 and 6 f32 print spelling, plus 16 for #; a negative value starts with -. Returns bytes written, or the negated total length needed when shorter, writing nothing. Other styles, precision outside 0..=4096, or an offset past the length trap. Never allocates |
fill(start, stop, value) | exclusive | () | Replace every byte of a range |
overwrite(offset, source) | exclusive | () | Copy source over existing bytes at offset; never grows |
copy_within(start, stop, to) | exclusive | () | Copy self[start:stop] to to in the same buffer, like memmove; the ranges may overlap; never grows |
push(value) | exclusive | Result[(), AllocError] | Append one byte |
extend(source) | exclusive | Result[(), AllocError] | Append source; pass &other[a:b] to append part of a buffer |
reserve(additional) | exclusive | Result[(), AllocError] | Ensure room for additional more bytes |
resize(count, fill) | exclusive | Result[(), AllocError] | Set the length, appending fill or dropping the tail |
truncate(count) | exclusive | () | Shorten to count; a negative count or one longer than the length traps |
pop() | exclusive | Option[u8] | Remove and return the last byte, or Nothing |
clear() | exclusive | () | Set the length to zero, keeping capacity |
source, set, and needle parameters are &[u8] views. == and !=
compare contents. A buffer
prints as a byte literal.
Bytes is a resource: assignment and argument passing move it, and the builtin
copy rejects it. Duplication is always the fallible copy() or slice()
method. It has no hashing. A buffer can be stored in collections, sent through
channels, and moved into worker threads.
Printing
print, str.repr, and f-string fields spell a Bytes buffer, a bytes
literal, and a &[u8] view the same way: as a byte literal that reads back as
the same bytes. Printable ASCII appears as itself, except that a backslash,
double quote, LF, CR, and TAB are \\, \", \n, \r, and \t; every other
byte is \xhh with lowercase digits. This matches Python’s bytes repr, except that the quotes are always
double. The private storage field never appears, also inside a class, list, or
Option.
import std.bytes
import std.encoding
from std.bytes import BytesError
from std.encoding import EncodingError
def main() -> Result[(), AllocError | IoError | BytesError | EncodingError]:
frame = std.bytes.from_bytes(b"GET\t\"/\"\r\n\x00\xff")?
print(frame)?
print(&frame[0:3], [std.bytes.from_text("ok")?]?)?
print(std.encoding.hex_text(&frame)?)?
Ok(())
b"GET\t\"/\"\r\n\x00\xff"
b"GET" [b"ok"]
47455409222f220d0a00ff
The literal spelling suits data that is mostly ASCII, such as protocol text.
For binary data such as digests, keys, or packed records, write the hex form
with std.encoding.hex_text, two lowercase digits per byte, or
format each byte explicitly.
Byte views
An operation that only reads bytes takes &[u8], a
slice view: a pointer and a
length. &buffer, &buffer[a:b], &list, a b"..." literal, and a view held
by the caller all convert to it with no copy and no allocation. The range is
part of the view, so these operations take no start and stop and report no
range error: view bounds clamp, and a negative bound counts from the end.
import std.bytes
def ends_line(line: &[u8]) -> bool:
count = len(line)
count >= 2 and line[-2] == 13u8 and line[-1] == 10u8
def main() -> Result[(), AllocError | std.bytes.BytesError | IoError]:
mut head = std.bytes.from_text("Host: example")?
head.extend(b"\r\n")?
mut name = std.bytes.empty()?
name.extend(&head[:4])?
print(head.find_bytes(b": ", 0, head.len()), ends_line(&head), ends_line(b"no"), name)?
Ok(())
Option[i64].Some(4) True False b"Host"
The operations taking &[u8] are extend, overwrite, find_in,
find_not_in, find_bytes, and matches here, the stream writes in
std.io, std.net, and std.tls, and the
read-only inputs of std.crypto. Each is one compiled body for
every byte owner; native operations read the view’s bytes in place. A view has
len(), indexing, slicing, ==, and iteration, but no methods. The receiver
of a Bytes method is always a buffer: copy a literal or view with from_bytes to
scan it with find or count, or to change it.
std.bytes.ByteSource, the protocol these operations took before, is
removed; naming it is a compile error that points to &[u8].
Index syntax and iteration
buffer[i] reads one byte as u8 and buffer[i] = value replaces one. They
follow the list index rules, not the
method range rule: a negative index counts from the end, and
a missing index traps with
exit status one and index out of bounds: index I, length N at the source
position. Use get and set when an out-of-range index is an expected outcome.
Inside a plain unchecked: suite under --unchecked assume, index syntax
requires 0 <= i < len(buffer) and omits bounds checks and negative-index
normalization. Violations are undefined behavior. Default builds,
unchecked trap: and buffer methods retain their checks; see
unchecked blocks.
Writes need a mut binding, a mutable field, or &mut Bytes, and evaluate as
indexed assignment does: the right
side, then the index, then the exclusive loan. buffer[i] += 1u8 evaluates the
index once. Indexing works on named buffers, fields, references, and temporary
values; it never grows the buffer and allocates nothing.
for b in buffer consumes the buffer and yields each u8. for b in &buffer
and for b in &mut buffer keep it and loop over a slice view of the
whole buffer, as &buffer[:] does: each &u8 or &mut u8 points into the
buffer, so *b += 1u8 changes it. reversed(&buffer) walks from the end. The
loop holds the view’s loan, so its body cannot grow, shrink, or move the
buffer, and the body of a shared loop cannot write to it.
Slice syntax buffer[a:b:c]? copies the
selected bytes into a new Bytes with Python’s clamping and returns
Result[Bytes, AllocError]; it also slices bytes literals. The slice(start, stop) method keeps the range rule: a range outside the
buffer traps instead of clamping. A slice view,
&buffer[a:b], borrows the bytes without a copy.
Scanning
find, find_in, find_not_in, find_bytes, count, matches,
matches_text, ascii_run, and utf8_run each make one native call over
their range. A parser uses them to step from one significant byte to the next
without a get call per byte.
- A found index is absolute, never relative to
start.Nothingmeans the range holds no match. - A
setis a byte view whose elements are the members; order and repeats do not matter. An empty set matches no byte, sofind_inreturnsNothingandfind_not_inreturnsstartfor a nonempty range. Build a set once and reuse it. The set, needle, or source may be the buffer being scanned. find_bytesreports a match only when the whole needle lies inside the range. An empty needle is found atstart. It takes time linear in the range and needle lengths, even on repetitive input, and does not allocate.matchesandmatches_textrequire0 <= offset <= len. When fewer bytes remain afteroffsetthan the comparison needs, the result isFalse, not an error. An empty comparison isTrue.ascii_runandutf8_runreturn an index fromstarttostop; equal tostartwhen the first byte does not qualify and tostopwhen every byte does.utf8_runaccepts exactly whatto_textaccepts and excludes a sequence thatstopcuts short.
The range rule
Internal index arithmetic uses small explicit unchecked: regions only where
validated ranges or native scan bounds prove the result fits in i64.
Growable sizes remain native allocation results, and invalid ranges still trap
before native access. This does not make caller arithmetic
unchecked: index updates such as buffer[i] += 1u8 follow the caller’s ordinary
arithmetic contract.
Every range must satisfy 0 <= start <= stop <= len, and every index passed to
a method 0 <= index < len. Methods have no negative indexing and no clamping,
unlike list slicing and
index syntax. A violation is a caller bug: it
traps with a
message naming the method and argument, such as
Bytes.get: index is outside 0..len(), and the report’s note names the
calling line. Check a range that comes from input before passing it. A zero-length range is valid
anywhere from 0 to len inclusive. overwrite additionally requires
offset + len(source) <= len of the destination, and copy_within requires
0 <= to and to + (stop - start) <= len. matches and
matches_text take an offset from 0 to len inclusive and report a short
remainder as False. The same rule applies to the range arguments of
read_file and read_stdin and of
to_text_range.
Caller buffer ranges in stream reads, encoding offsets, UUID writes, crypto output offsets, and CSV field positions also trap; see the invalid-argument convention.
Errors
pub enum BytesError:
InvalidUtf8
Allocation(AllocError)
The constructors and the methods that can fail only to allocate (push,
extend, reserve, resize, copy, slice, to_list) return
AllocError; a negative reserve or resize count is CapacityOverflow.
to_text and to_text_range return BytesError for InvalidUtf8. A range
outside the buffer traps. The only other result shape is
pop, whose Option[u8] reports an empty buffer.
Allocation carries the storage AllocError: OutOfMemory when the allocator
refuses, CapacityOverflow when the requested total cannot be a size. A failed
operation leaves the buffer’s length and contents unchanged and releases
anything it allocated.
Views
A view of a buffer is an ordinary reference
plus explicit offsets: &Bytes is a shared view and &mut Bytes an exclusive
one. A function that works on part of a buffer takes the reference and a
start/stop pair and uses the checked operations above.
The existing loan rules therefore define what may happen while a view exists:
- while a
&Bytesis live, the owner cannot be mutated, grown, cleared, moved, or exclusively borrowed; - while a
&mut Bytesis live, the owner cannot be read, borrowed, or moved; - a buffer cannot be extended or overwritten from itself, because the
exclusive receiver conflicts with the shared source. Use
copy_withinto move a range inside one buffer, or copy the range first.
These are compile-time conflicting borrow errors. Growth may relocate the
storage, but only a live loan can hold an address into it, and growth conflicts
with every live loan, so relocation cannot invalidate anything a program holds.
A slice view,
&buffer[a:b] or &mut buffer[a:b], carries its own bounds: it is a &[u8]
or &mut [u8] of the selected bytes, and holds the same whole-buffer loan as
&Bytes or &mut Bytes. It supports len, index reads and writes, loops,
iter pipelines, narrower views, and ==, and prints as a bytes literal,
b"bc". A &[u8] parameter also accepts views of list[u8] and bytes
literals. It has no Bytes methods: a scan’s receiver is the buffer and its
range, while the operations that read another byte sequence take
the view itself. A view cannot be stored or returned yet; that needs stored
references (backlog B11).
Costs
Storage is one byte per element plus one fixed-size owner. with_capacity,
slice, copy, and the from_ constructors reserve exactly what they need.
Growth through push, extend, and resize is amortized and
may over-reserve; reserve requests an exact total.
| Allocates | Does not allocate |
|---|---|
Constructors, slice, copy, to_list, to_text, to_text_range | len, is_empty, capacity, get, index syntax, iteration, every scan, ==, transport |
push, extend, resize, reserve beyond capacity | The same operations within capacity |
set, fill, overwrite, copy_within, truncate, pop, clear |
get, set, index syntax, push within capacity, pop, truncate, and
clear take constant time. Range operations take time linear in the range;
extend, overwrite, copy_within, slice, copy, resize, and fill are
single native bulk operations.
The scans run natively. find and find_in with a set of at most four bytes
examine eight bytes per step. find_in with a larger set and find_not_in
build a 256-entry table on each call, then examine one byte per step. count,
ascii_run, utf8_run, matches, and matches_text are linear in the bytes
examined. find_bytes searches for the needle’s first byte and compares at
each candidate: linear for ordinary data, and at most the range length times
the needle length when the needle’s first byte repeats throughout the range.
Measured with a release compiler on x86-64 Linux, 8,388,608 elements, pinned to one core:
| Program | Bytes | list[u8] |
|---|---|---|
| Build and copy once: peak resident memory | 18.9 MB | 526.8 MB |
| Build and copy once: wall time | 0.01 s (resize + copy) | 1.4 s (append loop + copy) |
| Build, then read every element: peak resident memory | 10.8 MB | 264.6 MB |
| Build, then read every element: wall time | 0.7 s (resize + get loop) | 1.6 s (append loop + index loop) |
| Append one element at a time | 0.6 s (push loop) | included above |
An empty program peaks at 2.4 MB. A single-element call costs roughly 70 to 85 ns here, so prefer the bulk operations; the saving is storage and copying.
A scan call costs about the same as a get call, however long its range.
Finding the only comma at the end of a 67,108,864-byte buffer, same machine,
pinned to one core:
| Program | Wall time |
|---|---|
find(44u8, 0, len) | 4 ms |
find_in with a four-byte set | 18 ms |
find_not_in with a 94-byte set | 15 ms |
count(44u8, 0, len) | 17 ms |
A get loop over the same range | 2.3 s |
Index syntax and iteration read the storage through a borrowed reference rather
than calling a std.bytes method. Reading every element of an 8,388,608-byte
buffer, same machine, pinned to one core:
| Loop | Wall time |
|---|---|
buffer.get(i)? over range(n) | 0.56 s |
buffer[i] over range(n) | 0.43 s |
for b in &buffer | 0.15 s |
fill over 67,108,864 bytes takes about 5 ms.
Transport
The read methods fill a range of a buffer from a descriptor, and the module
functions write a byte view to one, without an intermediate list. Each makes at
most one native call, returns the bytes moved as Result[i64, IoError], and
never changes a buffer’s length or capacity.
| Operation | Contract |
|---|---|
buffer.read_file(file: &mut File, start, stop) | Read into buffer[start:stop]; zero for a nonempty range is end of input |
buffer.read_stdin(start, stop) | Read standard input into buffer[start:stop] |
std.bytes.write_file(file: &mut File, source: &[u8]) | Write a prefix of source |
A count below the requested length is normal. A read range outside the buffer
traps under the range rule; a write takes a view, which carries its own range.
A closed file is Closed, and the open mode is enforced with NotReadable
and NotWritable. An interrupted call is IoError.System(4). File transfers
are raw bytes with no newline translation. write_file is a module function,
not a method, so a b"..." literal or a subview needs no owned buffer. For the
process streams, the prelude’s
write_stdout_bytes and write_stderr_bytes
take the same &[u8] view and write all of it.
Programs normally use these through std.io, which adds retry, end
of input, seeking, and bounded buffering on top.
Bytes has no native back-end methods. std.secrets,
std.crypto, std.compress,
std.db.sqlite, std.net, std.ready and
std.tls call compiler intrinsics (__crypto_*, __compress_*,
__tls_*, __sqlite_*, __net_*, __ready_*, __io_entropy_read). They
read byte inputs as &[u8] views, passed as a pointer and a length with no
copy. Output buffers are &mut Bytes operands whose ranges Plenty checks
first; native handles are held in byte storage.
These intrinsics are reserved for the standard library: user code calling one
is rejected with `__crypto_update` is reserved for the standard library.
Import the module that owns them instead.
The older binary I/O builtins and
std.path still take and return list[u8]. Convert explicitly with
from_list and to_list; each conversion allocates and copies.
Acknowledgements
find_bytesuses the two-way string-matching algorithm by Maxime Crochemore and Dominique Perrin (1991), as Rust’scorestring search does.- Word-at-a-time scans use the zero-byte test from Sean Eron Anderson’s Bit Twiddling Hacks, which credits a 1987 newsgroup post by Alan Mycroft.
std.cache.lru
from std.cache.lru import LruCache provides a fixed-capacity cache that
evicts its least recently used entry. It is ordinary embedded Plenty code over
a private dictionary and lists. Keys follow the
dictionary key rules: integers,
bool, str, bytes, or tuples of those. Values may be any type a list can
store, including owners such as lists and classes with destructors.
from std.cache.lru import LruCache
def main() -> Result[(), AllocError | IoError]:
mut names = LruCache[i64, str](2)?
drop(names.put(1, "Ada")?)
drop(names.put(2, "Grace")?)
print(names.get(&1), names.peek(&4))?
match names.put(3, "Edsger")?:
case Some(entry):
key, name = entry
print("evicted", key, name)?
case Nothing:
pass
print(names.contains(&2), names.len())?
Ok(())
Option[str].Some("Ada") Option[str].Nothing
evicted 2 Grace
False 2
Operations
| Operation | Result | Recency | Contract |
|---|---|---|---|
LruCache[K, V](capacity) | Result[LruCache[K, V], AllocError] | Reserve all storage for capacity entries | |
cache.capacity() | i64 | The construction capacity | |
cache.len() | i64 | Live entry count | |
cache.is_empty() | bool | Whether the count is zero | |
cache.contains(&key) | bool | Unchanged | Membership test |
cache.get(&key) | Option[V] | Promotes | Copy the value; copyable V only |
cache.peek(&key) | Option[V] | Unchanged | Copy the value; copyable V only |
cache.get_ref(&key) | Option[&V] | Promotes | Borrow the value |
cache.get_mut(&key) | Option[&mut V] | Promotes | Borrow the value mutably |
cache.peek_ref(&key) | Option[&V] | Unchanged | Borrow the value |
cache.peek_mut(&key) | Option[&mut V] | Unchanged | Borrow the value mutably |
cache.promote(&key) | bool | Promotes | Mark used; False if absent |
cache.put(key, value) | Result[Option[(K, V)], AllocError] | Promotes | Insert or replace, reporting the displaced entry |
cache.remove(&key) | Option[V] | Remove and transfer the value | |
cache.pop_lru() | Option[(K, V)] | Remove and transfer the least recent entry | |
cache.clear() | () | Drop entries from least to most recent |
Mutation, promotion included, requires a mutable owner or exclusive reference.
put with a new key makes it the most recent entry. If the cache was full, it
removes the least recent entry and returns it as Some((key, value)). With an
existing key, put keeps the stored key, replaces the value, promotes the
entry and returns Some((key, old_value)). Otherwise it returns Nothing.
This follows push in the Rust lru crate. Compare the returned key with
your own to tell eviction from replacement.
Copying and borrowed access
get and peek copy the value, like dictionary.get; see
copying and borrowed lookup.
They accept only copyable values,
such as integers, strings and tuples of those. A cache may still store
lists, classes and other affine values: each method of a generic class is
checked when it is first called, so only a call to get or peek with such
a value type is rejected. The error points at the call and names the
rejected instance, such as LruCache[i64, list[i64]].get, with a note
that its body cannot move out of indexed storage.
For affine values, borrow instead. get_ref and peek_ref return
Option[&V]; get_mut and peek_mut return Option[&mut V]. All four return
Nothing for an absent key. A borrowed result follows the
optional reference rules:
it borrows the whole cache until its last use, so the cache cannot change
meanwhile. Match it, or use ? or unwrap(), to reach the value:
from std.cache.lru import LruCache
def main() -> Result[(), AllocError | IoError]:
mut pages = LruCache[str, list[str]](2)?
drop(pages.put("home", ["intro"]?)?)
match pages.get_mut("home"):
case Some(lines):
lines.append("news")?
case Nothing:
pass
match pages.peek_ref("home"):
case Some(lines):
print(len(lines), lines)?
case Nothing:
print("missing")?
print(pages.peek_ref("about").is_none())?
Ok(())
2 ["intro", "news"]
True
Capacity
Capacity is fixed. Zero is valid and stores nothing: put returns its own
entry as Some((key, value)) at once, matching functools.lru_cache with
maxsize=0. A negative capacity, or i64.MAX, returns
AllocError.CapacityOverflow. Construction reserves storage for
capacity + 1 entries, so a large capacity costs memory before any entry
is stored.
Errors and costs
AllocError is the only error. Construction allocates everything the cache
will use; it returns OutOfMemory or CapacityOverflow and releases partial
storage on failure. After construction, no operation allocates. put still
returns a Result, because insertion into the reserved collections remains
fallible. On failure the cache is unchanged and the incoming key and value
are dropped.
Expected cost is constant for lookup, promotion, insertion, eviction and
removal, apart from hashing and value cleanup. Adversarial key collisions can
make dictionary lookup linear. clear and dropping the cache are linear.
The cache stores each key twice: in its private dictionary and in its recency slots. Keys are always copyable, so this needs no explicit copy.
Limits
There is no iteration in recency order, resizing, statistics, expiry or
thread-safety contract. A memoizing decorator such as functools.lru_cache
is out of scope: Plenty has no decorators. Wrap a function in a
cache explicitly with get and put.
Acknowledgements
- Behavior modelled on Python’s
functools.lru_cache, including no caching atmaxsize=0. - API modelled on the Rust
lrucrate:putreporting the displaced entry follows itspush, along withpeek,promoteandpop_lru.
std.channel
Explicit imports for bounded channels and timed waits and selection. Supported target: x86_64 Linux GNU. Imports are allocation-free aliases of the existing prelude facilities, with identical generic type identity.
| Export | Contract |
|---|---|
channel[T](capacity: Positive[i64]) | Result[tuple[Sender[T], Receiver[T]], AllocError] |
Sender[T], Receiver[T] | Owned endpoints; explicit share() retains the queue |
select_recv(&first, &second) | Result[Selected[A, B], SelectError] |
select_recv_nowait(&first, &second) | Same, without waiting for messages |
select_recv_timeout(&first, &second, timeout_ms: u64) | Same, with a millisecond budget |
SendError[T] | Full(value), Disconnected(value) |
RecvError | Empty, Disconnected |
SendTimeoutError[T] | TimedOut(value), Disconnected(value) |
RecvTimeoutError | TimedOut, Disconnected |
Selected[A, B] | First(value), Second(value) |
SelectError | Empty, TimedOut, Disconnected |
A worker sums messages until the sender’s scope drops it:
from std.channel import channel, Receiver, SendError
from std.thread import spawn, SpawnError
def total(inbox: Receiver[i64]) -> Result[i64, OverflowError[i64]]:
mut sum = 0
while True:
match inbox.recv():
case Ok(value):
sum += value
case Err(_):
break
Ok(sum)
def main() -> Result[(), AllocError | SendError[i64] | SpawnError[()] | IoError | OverflowError[i64]]:
sender, inbox = channel[i64](2)?
with spawn(total, inbox).drop_job()? as task:
with sender as outbox:
for n in range(1, 6):
outbox.send(n * 10)?
print(task.join()?)?
Ok(())
150
The linked facility pages specify every endpoint method. Capacity must be
a Positive[i64] and bounds queued messages. Computed capacities use
Positive.try_from(count)?; invalid literals are compile errors. Construction
returns only AllocError for capacity overflow or exhaustion. Construction makes one fallible allocation;
send, receive, share, error transport and selection allocate nothing. Messages
move without deep copying; rejected sends return the original value. Queue
operations have constant bookkeeping plus value transfer, with unbounded lock
and scheduling latency; destroying queued values costs their cleanup. Last
receiver drop destroys queued values and wakes blocked senders; last sender drop
allows buffered messages to drain before disconnection.
Messages must be wholly owned and worker-eligible. References, uncertified
foreign handles, and recursive receiver retention are still rejected. Imports
do not weaken the checks for live opposite endpoints across joins or future
waits. Timeouts are budgets, not hard deadlines. Use
std.time to compute remaining milliseconds; tokens do not interrupt
an ordinary channel wait. No unbounded channel or arbitrary selector is added.
Acknowledgements
- Contracts follow Rust’s
sync_channeland crossbeam-channel; no Crossbeam code is used.
std.cli
For terminal progress bars and spinners, see std.cli.progress.
std.cli parses an explicit schema into typed owned values. Its policy is checked
Plenty; it uses the existing byte arguments, strict UTF-8 decoder, integer parser,
and lossless Path. No reflection, native parser, package download, or
additional link dependency is involved. Supported target: x86_64 Linux GNU.
from std.cli import CliError, Kind, Spec, Value, parse
def main() -> Result[(), AllocError | IoError | CliError]:
schema = [
Spec("--count", Kind.Integer, False, False, False, "times to greet"),
Spec("--loud", Kind.Flag, False, False, False, "shout"),
Spec("NAME", Kind.Text, True, True, False, "who to greet")
]?
# A real tool passes args_bytes()? with start 1 to skip the program name.
raw = [encode_utf8("--count")?, encode_utf8("3")?, encode_utf8("Ada")?]?
parsed = parse(&schema, &raw, 0)?
for entry in &parsed.entries:
print(schema[entry.spec].name, entry.value)?
Ok(())
--count std.cli.Value.Integer(3)
NAME std.cli.Value.Text("Ada")
Schema and results
Construct Spec(name: str, kind: Kind, positional: bool, required: bool, repeatable: bool, help: str). All fields are public. Kind is Flag, Integer,
Text, or Path; there are no implicit defaults or conversions.
Option names include their leading --; positional names are display labels.
Names contain ASCII letters, digits, _, and interior -. Empty/duplicate names,
the reserved --help, positional flags, required positionals after optional ones,
and positionals after a repeatable positional are invalid. Only the final
positional can repeat. Options can appear between positionals in the schema.
validate(schema: &list[Spec]) -> Result[(), CliError] checks these rules. Both
parsing and help generation validate first, including when input requests help.
parse(schema: &list[Spec], arguments: &list[list[u8]], start: i64) -> Result[Parsed, CliError] borrows both inputs. Pass the unmodified args_bytes()
snapshot with start=1 to skip the invocation name, or fixture tokens with
start=0. A start outside 0..=len(arguments) is UnexpectedArgument(start).
Indices in errors are absolute indices in these inputs, never relative to start.
Parsed has public help_requested: bool and entries: list[Entry] fields.
Each Entry has spec: i64 (the schema index) and value: Value. Entries preserve
argument encounter order, including interleaved repeated options. Match Value:
| Variant | Meaning |
|---|---|
Flag | The option was present |
Integer(i64) | Checked decimal integer; invalid or overflowing input fails |
Text(str) | Owned, strictly decoded UTF-8, including an empty string |
Path(Path) | Owned exact native bytes, including invalid UTF-8 |
Absent optional specs have no entry; applications supply their own defaults.
Neither inputs nor returned data borrow each other. Raw access through args()
and args_bytes() remains available.
Token policy
- Only exact long options are supported:
--count 12, not-c, abbreviations, combined short flags, or--count=12. - A non-flag option consumes exactly its next token, even
--,--help, or another option’s name. End of input isMissingValue; a noninteger value for an integer option isInvalidInteger. This makes negative values unambiguous. - An unconsumed
--ends option recognition. Every later token is positional, including--help. A lone-, an empty argument, and tokens beginning with a minus followed by a digit are also positional before termination. Other dash-prefixed tokens are options; use--before a path like-file. --helpreturns immediately withhelp_requested=Trueand any preceding entries. It skips requiredness checks and later tokens, but does not override earlier errors. As an option value or after--, it is ordinary data.- Nonrepeatable duplicate options fail. Required specs need at least one entry. Positionals fill schema order; a repeatable final positional takes the rest.
Help, errors, and exit codes
help(schema: &list[Spec], program: &str) -> Result[str, CliError] generates a
usage heading and schema-ordered descriptions, with types and required/repeatable
markers plus --help. It borrows its inputs and returns owned text.
CliError distinguishes Schema(schema_index), UnknownOption(argument_index),
UnexpectedArgument(argument_index), MissingValue(schema_index),
Duplicate(schema_index), Missing(schema_index), InvalidInteger(argument_index),
InvalidUtf8(argument_index), and Allocation(AllocError).
Paths never go through text decoding. Integer/text values and option names do;
invalid UTF-8 is rejected without replacement decoding.
describe(error: &CliError) -> Result[str, AllocError] provides a readable
diagnostic with the relevant index. A UI may match the typed error to also show
schema[index].name or an escaped path from arguments[index].
describe_for(error: &CliError, schema: &list[Spec]) -> Result[str, AllocError]
uses schema names for schema-index errors, such as missing required argument FILE; it safely falls back to the numeric index when it is outside the schema.
exit_status(error: &CliError) -> i32 returns 2 for usage errors and 1 for schema
or allocation failures. Neither function writes or exits. Applications return 0
for help/success, report errors to stderr, and return the selected status. See
std.error for nonallocating fallback reporting.
Costs and limits
Schema validation uses quadratic name comparisons and allocates temporary name
bytes. For S schema entries and A arguments, parsing takes O(S² + A·S) comparisons
plus decoding/copying input bytes. It allocates S counters, returned entries, and
owned text/path payloads. Integer/flag payloads need no additional owned storage.
Partial output is released on failure; all these allocation failures are typed.
Argument indices and per-spec counts are bounded by the supplied argument list
length; counting never wraps or saturates. The existing CliError contract
therefore needs no arithmetic error variant.
Help repeatedly concatenates, with worst-case quadratic copying in output size.
describe may allocate; the allocation-error message itself is static.
There are no globals or OS handles, and input is only observed. Values follow ordinary Plenty ownership and worker-send rules; imports receive the same checks as application code. There are no subcommands, short aliases, float/bool-string conversion, automatic environment defaults, or process-exit side effects.
The typed arguments lesson
shows a reusable data-tool schema and explicit error handling.
The repository’s examples/countbytes.plenty is a complete byte-counting tool:
compile it with plenty --compile examples/countbytes.plenty -o countbytes, then
run ./countbytes --total FILE.... It reads each file fully and handles parsing,
help, path context, and exit statuses explicitly.
Acknowledgements
std.cli.fileinput
Read text lines from successive files or standard input without loading whole
files. The module is implemented in Plenty over std.io, with owned
fallible generators, not global state.
Recommended loop
lines() snapshots native command-line arguments after the executable name.
With no arguments it reads stdin. Use line? inside the loop: the outer ?
handles argument construction, while each item can report opening, reading,
decoding, or allocation failure.
import std.cli.fileinput as fileinput
from std.cli.fileinput import FileInputError
def main() -> Result[(), IoError | AllocError | FileInputError]:
for line in fileinput.lines()?:
write_stdout(line?)?
flush_stdout()?
Ok(())
Compile this filter and run ./filter first.txt second.txt, or pipe input into
./filter. write_stdout does not add a second newline. print would.
Explicit paths and options
| API | Result | Behavior |
|---|---|---|
lines() | Result[Generator[Result[str, FileInputError]], IoError] | Raw argv after the executable name; no option parsing. Failure to allocate the path list is IoError.Data(DataError.Allocation(...)), as for print |
files(paths: list[Path]) | Generator[Result[str, FileInputError]] | Consumes explicit native paths in order; an empty list selects stdin |
Applications with flags should parse them with std.cli, collect the
selected Path values, and call files. The module never treats application
flags specially: lines() attempts to open even --help as a filename.
Path preserves native bytes, including non-UTF-8 filenames. Only an exact
one-byte - selects stdin; use ./- for a file literally named -.
import std.cli.fileinput as fileinput
import std.path as paths
from std.cli.fileinput import FileInputError
def main() -> Result[(), IoError | AllocError | FileInputError]:
write_text("first.txt", "one\ntwo")?
write_text("second.txt", "three\n")?
names = [paths.from_text("first.txt")?, paths.from_text("second.txt")?]?
for line in fileinput.files(names):
print(str.repr(line?)?)?
Ok(())
"one\n"
"two"
"three\n"
Line, buffering, and ownership contracts
- Lines end at LF, retaining that LF and any preceding CR. Lone CR is ordinary text. The last nonempty, unterminated line is returned without adding LF. No line is synthesized at EOF, and empty files yield nothing.
- Decoding is strict UTF-8, including stdin. A completed line is validated as a whole, so multibyte characters may cross read-buffer boundaries. Invalid input is an error, not replacement text. There is no encoding option.
- Files open lazily, one at a time, using ordinary path and symlink semantics.
Each file and each stdin occurrence has an 8192-byte
BufReader. Unread bytes survive yields. Delimiter scanning advances through each chunk once; line accumulation uses amortized buffer growth, then one text conversion. Storage is the owned path list, a fixed read buffer, and the current line (including temporary bytes during conversion), not the entire file. A single very long line can still exhaust memory; there is no line-length limit. - Each
-reads the same process stream from its current position. After EOF, another-normally yields nothing; it can read new data if the underlying stream has subsequently been reset or an interactive source resumes. - Exhaustion explicitly closes each owned file and reports a close error.
break, return, propagation, or dropping the generator closes any remaining file through ordinary ownership cleanup; close errors during destruction cannot be reported. Stdin is never closed. - Buffering reads ahead. Do not mix an active iterator with other stdin readers. Dropping it early discards its unread buffered bytes; a fresh iterator cannot recover those bytes. This applies even though descriptor 0 stays open.
Errors
FileInputError has Open(Option[Path], StreamError),
Read(Option[Path], StreamError), Text(Option[Path], BytesError), and
Close(Option[Path], StreamError) variants. The path is the exact owned native
path supplied by the caller (Some even for -); Nothing identifies implicit
stdin. Open includes allocating the read buffer. Text includes allocating
line storage and strict UTF-8 conversion; line growth during reading can also
return Read(..., StreamError.Allocation(...)).
Underlying errors are preserved, including OS codes and allocation causes.
An error yields one Err and terminates the whole sequence, not just that
file. Owned file resources are released before yielding a read or text error.
Already returned lines remain valid; failed partial lines are not returned and
consumed input cannot be rolled back. Handle or propagate each item’s Result;
a loop does not propagate it automatically.
This is core input only: no rewriting, backups, open hooks, filename/line-number tracking, skip-file operation, or shared global iterator. Unlike Python’s default text mode, it does not perform universal-newline translation.
Acknowledgements
- Python
fileinputmodels the ordered-files, stdin fallback, dash sentinel, and retained-line-ending API. This is an original Plenty implementation, not copied Python code.
std.cli.progress
Bars and spinners for long-running command-line work, implemented in Plenty over
std.io, std.time and std.report.
The only new native operation asks whether a process stream is a terminal.
import std.cli.progress as progress
from std.cli.progress import ProgressError
def main() -> Result[(), ProgressError]:
mut bar = progress.bar(100u64)
for item in range(100):
# Process one item here, then count it.
bar.advance()?
bar.finish()?
Ok(())
Construction and updates
bar(total: u64) and spinner() return an owned Progress, initially at zero.
A bar has a known total; a spinner counts work without predicting a total.
Construction does not draw. Neither creates a worker or timer thread.
| Method | Contract |
|---|---|
advance(count: u64 = 1u64) | Add completed work; checked for overflow and the bar’s total |
set_position(position: u64) | Set an absolute count; it must not decrease or exceed the total |
tick() | Request a redraw without changing the count; useful for a spinner |
finish() | Draw the final count and a newline, ignoring the throttle; repeated calls do nothing |
position() / total() | Exact u64 count / Option[u64] total |
is_finished() | Whether finished or stopped by a display error |
Updates and finish return Result[(), ProgressError]. Finishing early preserves
the actual count: it does not claim remaining work was done. A zero-total bar is
already full, but still needs finish() if it was drawn. Updates after finishing
return Finished; invalid positions/overflow return InvalidPosition and leave
the count unchanged. Output failure preserves the accepted count, stops further
updates, and returns Output(std.fmt.FormatError) with its original cause.
Counts never wrap or saturate: an advance beyond u64.MAX is
InvalidPosition, even for a spinner. This check happens before changing the
count or drawing; a finished progress object still returns Finished first.
Display and metrics
Output goes only to stderr, and only if stderr is a terminal both at construction and before each draw. Redirected runs emit nothing, including on finish; stdout stays available for application results. Closed/undetectable streams count as nonterminals. The first update draws immediately. Subsequent updates draw at most every 100 ms, using a monotonic clock; skipped updates still count completed work. The spinner advances its frame on actual redraws, not every update call. If a later query finds redirected stderr, that progress display stays hidden.
The default line uses a ten-cell ASCII bar or a four-frame spinner, counts,
average whole items/second and estimated whole seconds remaining. Counts and
metrics use truncated decimal suffixes k, M, G, T, P, E to keep lines
short, including at u64.MAX. The exact counts remain available through getters.
The line is under 80 columns. Interactive rendering assumes an ANSI-compatible
terminal at least 80 columns wide: carriage return and erase-line replace the
previous frame. Cursor visibility and terminal modes are never changed.
rate(now: &Instant) -> Option[f64] is the average count per second since
construction. eta(now: &Instant) -> Option[f64] is remaining seconds at that
average. Zero elapsed time yields no rate; an unknown total or zero rate yields
no ETA, displayed as --. Completed known totals have ETA zero. Estimates use
floating-point arithmetic; displayed values truncate and saturate to u64.
Queries use the supplied time even after finishing; they do not freeze metrics.
For tests or an application-owned clock, bar_at(total, now), spinner_at(now),
set_position_at(position, now) and finish_at(now) take a borrowed Instant.
Use one timeline consistently. Backwards elapsed time saturates at zero;
backwards redraw timestamps cannot bypass the throttle.
render[W](now, output: &mut Formatter[W]) writes the plain line without newline or
cursor controls and leaves redraw state unchanged. A string formatter allows
snapshot testing; formatter errors propagate unchanged.
Costs and limits
Progress state is fixed-size. Normal drawing uses the bounded reporting formatter, without constructing temporary strings. Hidden and throttled updates skip formatting. Stderr writes may fail or block; a frame consists of multiple small writes and is not atomic with concurrent output. Keep one progress owner on one thread, and finish it before writing other diagnostics to stderr. Dropping it does not write: explicitly finish before normal early returns if a clean line is needed. An output failure may leave a partial line.
This initial API has no automatic ticking, multiple-bar coordination, templates, labels, colors, terminal resizing or iterator adapters. It follows the manually updated, terminal-only stderr model of indicatif, without embedding it. See Show progress for a small example.
Acknowledgements
- Design follows indicatif 0.18.6; no indicatif code is used.
std.collections
from std.collections import Heap, HeapBy, drain, drain_by provides owned
binary heaps (priority queues). They are ordinary embedded Plenty code over a
private list[T]; no native code or runtime primitive is involved.
| Type | First element | Element types |
|---|---|---|
Heap[T] | Least by < | Integers, floats, str, and tuples of those |
HeapBy[T, F] | The one the comparator puts before all others | Any storable type |
Heap[T] uses the element type’s own <. Classes, ordinary enums, collections,
and tuples containing unordered fields are rejected at compile time with
“operator < does not accept” the type. A scalar NaN element compares false
against everything and leaves the removal order unspecified. Tuples use the
lexicographic total sorting order, including
signed zeros and NaNs, so Heap[(i64, str)] orders by priority then text.
HeapBy(before) stores a comparator F: Callable[[&T, &T], bool]: a named
function or an owned closure. before(a, b) returns whether a must leave
ahead of b; it must be a consistent strict order and must return False for
equal elements. Reverse the test for a greatest-first queue. The comparator is
inferred from its signature, as in HeapBy(urgent)?. A closure with borrowed
captures cannot be stored.
from std.collections import Heap, drain
def main() -> Result[(), AllocError | IoError]:
mut tasks = Heap[(i64, str)]()?
tasks.push((3, "write docs"))?
tasks.push((1, "fix crash"))?
tasks.push((2, "review"))?
print(tasks.peek())?
for priority, name in drain(tasks):
print(priority, name)?
Ok(())
Option[tuple[i64, str]].Some((1, "fix crash"))
1 fix crash
2 review
3 write docs
Operations
Both types share one interface. heap is Heap[T] or HeapBy[T, F].
| Operation | Result | Contract |
|---|---|---|
Heap[T]() | Result[Heap[T], AllocError] | Construct an empty heap |
HeapBy(before) | Result[HeapBy[T, F], AllocError] | Construct an empty heap owning its comparator |
heap.len() | i64 | Observe the element count |
heap.is_empty() | bool | Observe whether the count is zero |
heap.reserve(additional) | Result[(), AllocError] | Make room for at least additional: i64 more elements |
heap.push(value) | Result[(), AllocError] | Consume a value and insert it |
heap.peek() | Option[T] | Copy the first element; copyable T only |
heap.peek_ref() | Option[&T] | Borrow the first element |
heap.pop() | Option[T] | Remove and transfer the first element |
heap.clear() | () | Destroy all elements, retaining capacity |
drain(heap) | Generator[T] | Consume a Heap[T], yielding elements in removal order |
drain_by(heap) | Generator[T] | The same for a HeapBy[T, F] |
Mutation requires a mutable owner or exclusive reference. On an empty heap,
pop, peek and peek_ref return Nothing. peek follows the
copying lookup rule:
it type-checks only for a copyable T, and is checked where it is called, so a
heap of class instances uses peek_ref. The reference from peek_ref borrows
the heap, so the heap cannot change while it is live. There is no peek_mut:
changing the first element in place could break the heap order.
Elements that compare equal leave in an unspecified order. For first-in, first-out ties, store an increasing sequence number in the element and compare it after the priority.
Ownership and cleanup
A heap owns its elements and moves like any other owner; duplicate it only
with an explicit copy. push takes ownership, and pop transfers it back.
Reordering exchanges rows in place: no element is copied, retained, or dropped
while it is in the heap. References, generators, and unit cannot be elements.
Dropping or clearing a heap destroys the remaining elements once each, in
storage order, not priority order. A drain frame owns the heap: abandoning
the frame early destroys the elements not yet yielded.
A heap holds no OS resource and no shared state. It has no thread-safety contract of its own; it is an ordinary owned value of its element type.
Errors and costs
AllocError is the only error. push allocates only when the backing list must grow and reserve
only when the requested room is missing. On failure both leave the existing
contents and order unchanged, and push drops the incoming value. Every other
operation, including drain, allocates nothing.
| Operation | Time for n elements |
|---|---|
len, is_empty, peek, peek_ref | Constant |
push | At most floor(log2 n) comparisons; a growing push also moves the backing list |
pop | At most 2 * floor(log2 n) comparisons |
clear, drop | Linear, plus element cleanup |
| Draining all elements | O(n log n) comparisons |
Supported target: x86_64-unknown-linux-gnu. See the runnable
priority queue lesson.
Limits
There is no construction from an existing list, bulk insertion, merging, indexed access, borrowed iteration, removal of an arbitrary element, or priority update. A comparator cannot fail; one that reports inconsistent answers gives an unspecified order but cannot break memory safety.
Acknowledgements
- Modelled on Python’s
heapq, which also puts the least element first, and Rust’sBinaryHeap.
std.complex
from std.complex import Complex provides complex numbers with f64 real and
imaginary parts. The module is ordinary embedded Plenty: arithmetic is written
with operator hooks and the
functions use the f64 numeric methods. Nothing in it allocates or fails.
from std.complex import Complex, from_polar
def main() -> Result[(), IoError | AllocError | FormatError]:
a = Complex(1.5, 2.0)
b = Complex(-0.5, 4.0)
print(f"{a + b} {a - b} {a * b} {-a}"?)?
print(f"{a / b}"?)?
print(f"{b.conj()} {Complex(3.0, 4.0).abs()} {Complex(3.0, 4.0).norm()}"?)?
print(a == Complex(1.5, 2.0), a != b, a.re, a.im)?
print(a)?
Ok(())
1.0+6.0i 2.0-2.0i -8.75+5.0i -1.5-2.0i
0.4461538461538462-0.4307692307692308i
-0.5-4.0i 5.0 25.0
True True 1.5 2.0
1.5+2.0i
Complex(re, im) is the generated constructor, and re and im are public
fields. Complex is a copy class:
assignment, by-value argument passing, return, and collection insertion copy
its two f64 parts and leave the source usable. Copying never allocates.
Operators and methods borrow their operands, which reads the parts in place
instead of copying them into the call. There are no in-place hooks; accumulate
with total = total + term on a mut binding.
from std.complex import Complex
def double(z: Complex) -> Complex:
z + z
def main() -> Result[(), AllocError | IoError]:
a = Complex(1.0, 2.0)
mut b = a
b.im = -2.0
zs = [a, b, double(a)]?
print(a == zs[0], b == zs[1], zs[2].re, len(zs))?
Ok(())
True True 2.0 3
Operations
| Operation | Result | Contract |
|---|---|---|
a + b, a - b, a * b, a / b | Complex | Arithmetic on two Complex operands |
-a | Complex | Negate both parts |
a == b, a != b | bool | Structural: both parts compare as f64 |
z.scale(factor: f64) | Complex | Multiply both parts by a real factor |
z.conj() | Complex | Complex conjugate |
z.abs() | f64 | Magnitude, scaled like hypot |
z.norm() | f64 | Squared magnitude re * re + im * im |
z.arg() | f64 | Phase angle im.atan2(re), in [-pi, pi] |
z.to_polar() | (f64, f64) | (z.abs(), z.arg()) |
from_polar(r: f64, theta: f64) | Complex | Complex(r * cos(theta), r * sin(theta)) |
z.exp() | Complex | Complex exponential |
z.log() | Complex | Principal natural logarithm: abs().log() + arg() i |
z.sqrt() | Complex | Principal square root: nonnegative real part |
z.pow(exponent: &Complex) | Complex | z.powi(n) for a real integral exponent n with abs(n) <= 100; otherwise the principal value of exp(exponent * log(z)) |
z.powi(n: i64) | Complex | z multiplied by itself n times by repeated squaring; a negative n gives 1 / z.powi(-n) |
Both operands of an operator must be Complex; there are no mixed operations
with f64. Operator hooks dispatch on the left operand’s type and cannot be
overloaded, so z * 2.0 cannot share __mul__ with z * w. Use
z.scale(2.0) for a real factor, or Complex(2.0, 0.0) as an operand.
scale is also the more exact choice for non-finite values: `Complex(inf, 1.0)
- Complex(2.0, 0.0)
computesinf * 0.0for its imaginary part and gives NaN, whilescale(2.0)givesinf+2.0i`.
Multiplication uses the textbook formula. Division uses Smith’s algorithm,
which scales by the larger divisor part so that c * c + d * d never overflows
or underflows. abs scales the same way, so Complex(1e300, 1e300).abs() is
finite. norm does not scale and overflows first.
from std.complex import Complex, from_polar
import std.math as math
def main() -> Result[(), IoError | AllocError | FormatError | math.MathError]:
z = from_polar(2.0, 0.5)
r, theta = z.to_polar()
print(math.isclose(r, 2.0, 1e-15, 0.0)?, math.isclose(theta, 0.5, 1e-15, 0.0)?)?
print(f"{Complex(-4.0, 0.0).sqrt()} {Complex(3.0, 4.0).sqrt()}"?)?
print(f"{Complex(-1.0, 0.0).log()}"?)?
i = Complex(0.0, 1.0)
print(f"{i.pow(i)} {Complex(0.0, 0.0).pow(Complex(2.0, 0.0))}"?)?
print(f"{Complex(1.0, 1.0).pow(Complex(2.0, 0.0))} {Complex(1.0, 1.0).powi(-2)}"?)?
print(f"{Complex(0.0, f64.PI).exp()}"?)?
Ok(())
True True
0.0+2.0i 2.0+1.0i
0.0+3.141592653589793i
0.20787957635076193+0.0i 0.0+0.0i
0.0+2.0i 0.0-0.5i
-1.0+1.2246467991473532e-16i
The functions are computed from f64 primitives, so results carry their
rounding: exp(pi i) has a tiny imaginary part. Compare results with a
tolerance, for example math.isclose on each part.
pow uses the method of Python’s complex.__pow__. A real integral exponent n with
abs(n) <= 100 uses powi, which multiplies by repeated squaring, so small
powers of exactly representable values are exact: (1+1i).pow(2) is 0+2i.
Each squaring can roughly double the relative error, so larger or
non-integral exponents use exp(exponent * log(z)) and inherit the rounding
of exp and log. A zero exponent gives 1+0i, including for a NaN base, as
f64.pow does. A zero base gives 0+0i for a real positive exponent;
otherwise it follows f64 division and log, so 0.pow(-2) is inf+NaNi
rather than an error. Call powi directly for an i64 exponent of any size,
including i64.MIN. Its negative-exponent magnitude is formed without signed
overflow; the result remains an infallible Complex under checked defaults.
Floating-point behavior
Complex arithmetic follows f64 arithmetic: it never traps and never returns
Result. Overflow gives infinity and invalid operations give NaN, which then
propagates. This matches the / operator on f64 and the numeric methods, and
lets operators compose. It differs from the std.math functions,
which report domain and overflow errors; check parts with math.isfinite when
a result must be finite.
Dividing by Complex(0.0, 0.0) divides each part of the numerator by the real
zero, exactly as f64 division does: (1+2i) / 0 is inf+infi, and 0 / 0 is
NaN+NaNi. Python raises ZeroDivisionError instead.
Signed zero is preserved where the formulas define it. log of zero is
-inf plus the zero’s arg, and sqrt of a signed zero keeps the sign of the
imaginary part. Complex(-1.0, 0.0).arg() is pi and Complex(-1.0, -0.0).arg()
is -pi, so the branch cut of log, sqrt and pow along the negative real
axis follows the sign of the imaginary zero. abs returns infinity when either
part is infinite, even if the other is NaN.
from std.complex import Complex
def main() -> Result[(), IoError | AllocError | FormatError]:
print(f"{Complex(1.0, 2.0) / Complex(0.0, 0.0)} {Complex(0.0, 0.0) / Complex(0.0, 0.0)}"?)?
print(f"{Complex(-4.0, 0.0).sqrt()} {Complex(-4.0, -0.0).sqrt()}"?)?
print(f"{Complex(0.0, 0.0).log()} {Complex(f64.INF, f64.NAN).abs()}"?)?
print(Complex(0.0, 0.0) == Complex(-0.0, 0.0), Complex(f64.NAN, 0.0) == Complex(f64.NAN, 0.0))?
Ok(())
inf+infi NaN+NaNi
0.0+2.0i 0.0-2.0i
-inf+0.0i inf
True False
Equality is the language’s structural class equality, not a hook: 0.0 equals
-0.0, and a NaN part makes values unequal, as for f64.
Formatting
Complex has a public display method, so print,
f-strings and
std.fmt.to_string write re+imi or re-imi, each part in the
f64 print spelling: 1.5+2.0i, 1.0-0.5i, NaN+infi. A negative zero
imaginary part is written -0.0i. It has no debug method, so str.repr,
!r and list elements use the structural spelling
std.complex.Complex(re=1.5, im=2.0). A format
spec applies to the whole text; format the parts directly, as in
f"{z.re:.3f}{z.im:+.3f}i", to control precision.
Acknowledgements
- API modelled on Python’s
complexandcmath;normfollows C++std::norm. - Division uses Robert L. Smith’s Algorithm 116: Complex division (Communications of the ACM, 1962).
std.compress
import std.compress compresses and decompresses DEFLATE (RFC 1951), zlib
(RFC 1950), gzip (RFC 1952) and Brotli (RFC 7932) data, whole std.bytes buffers at
once or as std.io streams. It has one error type, CompressError.
DEFLATE codecs and CRC-32 use zlib-rs; Brotli uses the
reference codec with fallible allocation callbacks, bundled into the runtime.
The Plenty module owns buffering, the output limit, and the policy for input
that follows a stream. The module is available on the supported x86-64 Linux
GNU target.
import std.bytes
import std.compress
from std.compress import Format
def main() -> Result[(), IoError | AllocError | std.bytes.BytesError | std.compress.CompressError]:
mut log = std.bytes.empty()?
for _ in range(50):
log.extend(&std.bytes.from_text("GET /index.html 200\n")?)?
packed = std.compress.compress(Format.Gzip, &log, 6)?
print(log.len(), packed.len() < log.len())?
restored = std.compress.decompress(Format.Gzip, &packed, 4096)?
print(restored == log)?
Ok(())
1000 True
True
Functions
| Function | Result | Contract |
|---|---|---|
compress(format: Format, data: &[u8], level: i64) | Result[Bytes, CompressError] | All of data, compressed |
decompress(format: Format, data: &[u8], limit: NonNegative[i64]) | Result[Bytes, CompressError] | The contents of data, at most limit bytes |
compressor(format: Format, inner: W, level: i64) | Result[Compressor[W], CompressError] | A writer that compresses into inner |
decompressor(format: Format, inner: R, limit: NonNegative[i64]) | Result[Decompressor[R], CompressError] | A reader of the decompressed contents of inner |
inflater(format: Format, limit: NonNegative[i64]) | Result[Inflater, CompressError] | A step-by-step decoder; see Inflater |
crc32(seed: i64, data: &[u8]) | Result[i64, CompressError] | The CRC-32 of data, continuing from seed |
stream_error(error: CompressError) | StreamError | How a Decompressor reports error |
Format is Deflate (raw, no header or checksum), Zlib (Adler-32),
Gzip (CRC-32 and length), or Brotli (no checksum). For DEFLATE formats,
level runs from 0, which stores the data
uncompressed in blocks, to 9, which is smallest and slowest; 6 is a usual
choice. Brotli takes qualities 0–11; 5 is a useful starting point, while 10–11
spend substantially more time and memory. A level outside its format’s range
is OutOfRange.
crc32 is the gzip and zip checksum; start a fresh sum with seed 0. The
check value of the nine bytes 123456789 is 3421780262. seed must lie in
0 to 2³²−1. CRC-32 retains its native 32-bit polynomial checksum semantics;
checked integer defaults do not change the checksum algorithm.
Input policy
A decoder accepts exactly one stream, with these rules:
- Truncation. Input that ends inside a stream is
Truncated. - Concatenated gzip members. When a gzip member ends and the next byte is
31 (the first byte of the gzip magic), the next member is decoded and its
output appended, as
gzip -ddoes forcat a.gz b.gz. Each member’s CRC and length are checked. - Trailing data. Any other byte after the end of a stream is
TrailingData, including the zero padding some tools append to gzip files. Raw DEFLATE, zlib and Brotli streams never continue. - Integrity. A gzip CRC-32 or length mismatch, a zlib Adler-32 mismatch,
and a gzip header CRC mismatch are
Checksum. Raw DEFLATE and Brotli have no checksum. Any other malformed input, a zlib stream that needs a preset dictionary, or a stream of another format isCorrupt. - Output limit. Producing more than
limitbytes in total, counted across gzip members, isLimitExceeded. Exactlylimitbytes succeeds. Choose the limit from what the program can hold, not from the input size: a few kilobytes of DEFLATE can expand by a factor of 1000. Decoded progress stays exact: exceeding thei64representation also reportsLimitExceeded, rather than wrapping or saturating the counter.
Gzip header fields (name, comment, modification time, extra data) are validated and discarded; the encoder writes a minimal header with no name and time 0, so equal input gives equal output.
Compressor
Compressor[W] has the Writer methods, so write_all,
copy_all, and anything else generic over Writer accept it. inner is
public.
| Method | Result | Contract |
|---|---|---|
write(source: &[u8]) | Result[i64, StreamError] | Compress part of source, a view of a Bytes, a list or a b"..." literal; the count is the input consumed |
flush() | Result[(), StreamError] | Deliver everything written so far as complete blocks, then flush inner |
finish() | Result[(), StreamError] | Write the stream’s end and checksum, then flush inner |
Output reaches inner in order, on later write calls and in flush and
finish. A write first delivers pending output from the previous one; if
that fails, the error is returned and none of source is consumed. A failure
keeps the undelivered bytes, so calling the method again resumes where it
stopped. Interrupted inner writes are retried; a zero-byte inner write is
WriteZero.
Call finish before using the result. Dropping a compressor without it
leaves inner holding an incomplete stream, which a decoder reports as
Truncated. After finish, write is StreamError.Io(IoError.Closed) and
finish and flush only deliver what is left. Each flush ends the current
block, which costs some compression ratio; flush at message boundaries, not
after every write.
Decompressor
Decompressor[R] has the Reader method read(buffer, start, stop), which returns at least one decoded byte, or 0 once the stream
and the input have both ended. inner is public.
| Method | Result | Contract |
|---|---|---|
read(buffer: &mut Bytes, start: i64, stop: i64) | Result[i64, StreamError] | Decode into buffer[start:stop] |
failure() | Option[CompressError] | The decoding failure that ended the stream, if any |
An error from inner is returned unchanged and leaves the reader usable, so
Interrupted and other transient failures may be retried. A decoding failure
ends the stream: read returns it through the Reader protocol, and every
later call repeats it. failure gives the precise cause, which
stream_error maps as follows:
CompressError | StreamError |
|---|---|
Truncated | UnexpectedEof |
Allocation(cause) | Allocation(cause) |
OutOfRange | OutOfRange |
Corrupt, Checksum, TrailingData, LimitExceeded | Io(IoError.InvalidInput) |
gzip and zlib store their checksum after the data, so a reader may already
have returned bytes when it finds a Checksum failure. Treat decoded output
as unverified until read has returned 0. Bytes decoded in the step that
fails are discarded. decompress returns nothing on any error.
Inflater
Inflater is the decoder inside Decompressor, for a reader that owns its
input and cannot hand over an inner stream, as
std.archive does for ZIP entries. It applies the same
input policy and limit.
| Method | Result | Contract |
|---|---|---|
step(input: &[u8]) | Result[i64, CompressError] | Decode from input; returns the count consumed |
output() | &Bytes | The output of the last step, at most 16384 bytes |
clear() | () | Empty the pending output |
complete() | bool | Whether the input so far is a complete stream |
Empty the output with clear before each step. After consuming the input,
call step(b"") with an empty view until complete() or a step produces no output:
Brotli can have pending decoded bytes after consuming its final input byte.
A step that returns 0 with no output needs more input unless complete. Discard
the Inflater after any error; do not rely on retries of codec errors.
Errors
pub enum CompressError:
OutOfRange
Corrupt
Checksum
Truncated
TrailingData
LimitExceeded
Allocation(AllocError)
| Variant | Meaning |
|---|---|
OutOfRange | A level or seed out of bounds |
Corrupt | Malformed compressed data |
Checksum | Well-formed data whose checksum or length does not match |
Truncated | The input ended inside a stream |
TrailingData | Bytes after the end of the stream; see Input policy |
LimitExceeded | More than limit bytes of output |
Allocation(cause) | A codec state, buffer, or output could not be allocated |
Ownership and threads
A Compressor or Decompressor owns its inner stream, its codec state,
and its buffers; dropping it releases them all, finished or not, and does not
flush or close inner. Neither can be copied. The module has no global state,
passes the ordinary worker-effect checks, and may be used from any thread;
moving a stream to a worker moves its ownership.
The codec state is held in a Bytes whose contents are the pending output.
The __compress_* intrinsics that operate on it are
reserved for the standard library. Given a buffer not
made by std.compress, they stop the program with
operand is not a std.compress state. Both native engines report a successful
consumed count within the offered input view and keep pending output within the
reserved capacity. The Plenty adapters rely on these bounds for their buffer
cursors; these native wrapper contracts are unchanged by checked arithmetic
defaults. Inner stream progress is still validated separately:
InvalidProgress does not advance the cursor or discard the pending tail.
Costs
For the DEFLATE formats, memory is fixed when a stream is made and does not grow with the data. Brotli’s different allocation contract is described separately:
| Value | Allocates |
|---|---|
Compressor | 371 KiB of zlib-rs state at every level (window, hash chains, symbol buffer) and a 16 KiB output buffer |
Decompressor | 46 KiB of zlib-rs state (including the 32 KiB window), a 16 KiB output buffer, and a 16 KiB input buffer |
compress, decompress | One codec state and buffer as above, and the result, which grows by doubling |
| Operation | Allocates | Time |
|---|---|---|
Compressor.write, flush, finish | Nothing | Linear in the input; higher levels are slower |
Decompressor.read | Nothing | Linear in the output |
crc32 | Nothing | Linear in the view |
Inputs are read directly from the caller’s buffer; output is copied once,
from the codec’s buffer to inner or the caller’s buffer. Allocation failures
are reported as Allocation; dropping a failed or unfinished codec releases
its state and buffers.
Supply
The runtime builds zlib-rs from an unmodified copy under
plenty-runtime/vendor, so building the compiler needs no network and an
installed compiler needs neither the sources nor a Rust toolchain:
| Crate | Version | Licence | Maintainer |
|---|---|---|---|
zlib-rs | 0.6.8 | Zlib | Trifecta Tech Foundation, as part of zlib-rs / libz-rs-sys |
zlib-rs has no dependencies. It is a Rust port of zlib with zlib-ng’s
optimisations, and reports allocation failure as an error rather than
aborting, which Allocation relies on. The runtime calls its zlib-style
interface (deflate::init, inflate::inflate, and so on), enabled by the
crate’s __internal-api feature.
The licence file is kept with the sources. The version is pinned in
Cargo.lock, and a test fails if the two disagree. Updating it is the
runtime maintainers’ job, with each release that fixes a defect: replace the
directory with the new release, run cargo update -p zlib-rs --precise VERSION, and rerun tests/test_std_compress.rs, whose fixtures come from
CPython’s zlib and gzip and from the system gzip tool. The
__internal-api interface is not covered by semver, so read the release’s
changes to deflate.rs and inflate.rs.
Executables are linked with --gc-sections, and the codecs are reached only
through their own runtime entry symbols, so a program that does not use
std.compress contains none of them.
Limits
- No bzip2, xz, zstd, or lz4. ZIP and TAR are in
std.archive. - No preset dictionaries, compression strategies, or window and memory settings; DEFLATE encoders use a 32 KiB window and zlib’s default memory level, and Brotli uses the standard 4 MiB encoder window.
- No access to gzip header fields.
- Trailing padding after gzip members is an error, not ignored.
- No seeking within compressed data.
- One target, x86-64 Linux GNU.
Acknowledgements
- Wraps zlib-rs 0.6.8, vendored in the runtime (Zlib).
Brotli streams
Use std.compress.Format.Brotli with the same whole-buffer functions,
Compressor, Decompressor and Inflater as the
other compression formats. The format is
RFC 7932, also used by HTTP’s
Content-Encoding: br.
import std.bytes
import std.compress
from std.compress import Format, CompressError
from std.bytes import BytesError
def main() -> Result[(), AllocError | BytesError | CompressError | IoError]:
original = std.bytes.from_text("A Brotli message")?
encoded = std.compress.compress(Format.Brotli, &original, 5)?
decoded = std.compress.decompress(Format.Brotli, &encoded, 1024)?
print(decoded.to_text()?)?
Ok(())
A Brotli message
Quality ranges from 0 (fastest) to 11 (most effort). It is a compression-effort setting, not a guarantee that each higher setting produces a smaller result. Start with 5 for general use; choose 10–11 when extra compression work is acceptable, such as preparing static assets. Quality 0 still compresses; it does not mean the stored-block mode of DEFLATE level 0. The encoder uses the standard 4 MiB window. The decoder accepts standard windows up to 16 MiB; the nonstandard large-window extension and custom dictionaries are disabled.
flush() makes all written input decodable without ending the stream.
finish() writes its final marker; repeated successful finishes are harmless.
Short/interrupted inner I/O and retrying an inner write failure preserve pending
output. Codec failures, including allocation failures, require discarding that
codec. A whole-buffer call returns no partial value on any error.
The decoder rejects trailing data, including a second Brotli stream. Only gzip
has the concatenated-member policy. Truncated input is Truncated; malformed
input is Corrupt. Brotli has no checksum: a changed stream can still be valid
and decode to changed data. Use transport/application integrity when required.
Memory and failures
The Plenty input/output staging buffers remain 16 KiB. Unlike the DEFLATE
implementation, Brotli allocates codec working memory during write, read,
flush and finish. Working memory depends on quality, stream window and block
structure; a decoder may allocate its history before hitting the output limit.
The output limit is not a total-memory quota. It applies to all produced output,
including bytes buffered internally after the last input byte has been consumed.
The native callbacks use Plenty’s fallible allocator and matching deallocator,
so failure is CompressError.Allocation(AllocError.OutOfMemory) (or its
StreamError.Allocation projection), not process termination. Failed and
unfinished states release every native allocation on drop. There is no
allocation-free-after-setup guarantee for Brotli.
The low-level Inflater caller must drain output even when all input has been
consumed: clear its output, call step(b""), and repeat until
complete or no output is produced. decompress and Decompressor do this
automatically. An incomplete decoder with no more input/output is truncated.
HTTP
The HTTP client continues to expose the encoded entity body; it does not
negotiate or decode content automatically. If your request sends
Accept-Encoding: br, check the actual response’s Content-Encoding and use
decompress(Format.Brotli, &fetched.body, decoded_limit) for a single br
coding. Bound the encoded HTTP body and the decoded output independently.
Multiple content codings must be decoded in reverse application order; do not
assume every response is Brotli merely because the request accepted it.
Servers can compress response bytes with the same function and send
Content-Encoding: br. The content length describes the compressed bytes.
Choose a representation from the request’s accepted encodings, and send
Vary: Accept-Encoding when the representation varies. Streaming compression
uses the existing Writer API; no new HTTP framing or automatic negotiation is
introduced here.
Supply and update policy
The pinned, unmodified reference C codec 1.2.1 is in
plenty-runtime/native/brotli-1.2.1, with its MIT license and source archive
digest. It includes the decoder block-switch validation fix for
CVE-2026-111497; the vendored source must not be replaced with an unpatched
older version. Release third-party notices include its license.
The shared build helper compiles the same C sources for runtime unit tests
and both compiler-embedded runtime archives. The explicit
BROTLI_ENCODER_CLEANUP_ON_OOM flag is mandatory: without it upstream selects
exit-on-allocation-failure mode. Allocation-failure tests check this behavior.
An installed Plenty compiler needs neither a system Brotli library nor the
source tree. Unused compression is removed by native section garbage collection.
This complex codec is wrapped rather than rewritten in Plenty. The Rust
brotli 9.1.0 allocator interface was inspected, but its encoder accesses
allocation results without a failure check (for example RingBufferInitBuffer).
Returning empty memory can panic; unwinding is not an allocation-error strategy
for Plenty’s abort-on-panic runtime. The C interface has explicit nullable
callbacks and recoverable allocation handling.
Updates require source/license/digest and security review, plus all codec, external-interop, malformed-input, allocation-failure, cleanup, executable-size and installed-compiler checks. Output bytes may change with a codec update; the interoperability and error contracts are the stable API.
Acknowledgements
- Wraps the Brotli reference C codec 1.2.1, vendored in the runtime (MIT).
std.concurrent
Explicit imports for executors,
futures,
parallel map, and
tree reduction.
Exports: ThreadPoolExecutor, Future[T], PoolError, SubmitError[F],
FutureError, PoolMapError, and ParallelError[E]. Imports preserve prelude
identity and add no allocation or wrapper. Supported target: x86_64 Linux GNU.
from std.concurrent import ThreadPoolExecutor, PoolError, FutureError, SubmitError
def square(n: i64) -> Result[i64, OverflowError[i64]]:
Ok(n * n)
def main() -> Result[(), AllocError | IoError | PoolError | FutureError | SubmitError[()] | OverflowError[i64]]:
with ThreadPoolExecutor(2, 4)? as pool:
first = pool.submit(square, 7).drop_job()?
second = pool.submit(square, 8).drop_job()?
print(first.result()??, second.result()??)?
Ok(())
49 64
ThreadPoolExecutor(workers: Positive[i64], queue_capacity: Positive[i64]) returns
Result[ThreadPoolExecutor, PoolError]; both arguments are positive domain values. Computed counts use
Positive.try_from(count)?; invalid literals are compile errors.
PoolError has Allocation(AllocError) and Thread(ThreadError).
The linked pages specify submission, mapping, reduction and shutdown signatures.
Accepted jobs and results are wholly owned; borrowing belongs in scoped threads.
Pools and futures cannot enter workers. All reachable code still passes worker
effect checking.
Construction makes one allocation and starts the requested OS workers. Each
accepted job needs one fallible cell; a full queue bounds outstanding queued
jobs and can block submit. submit_nowait reports SubmitError.Full(job);
other variants are Shutdown(job), OutOfMemory(job), and CapacityOverflow(job).
Recovery retains ownership without a hidden clone. submit(...).drop_job()
drops a rejected job at the call and reports the job-free SubmitError[()]. Waiting, result transfer,
status and cancellation allocate nothing. Queue bookkeeping is constant apart
from job/result transfer and cleanup; waiting and shutdown have no bounded
latency. Map allocates outputs and uses a bounded work window; reduction costs
and ordering are specified in its facility page.
Future[T].result() consumes the future and returns Result[T, FutureError].
When a worker returns a Result, that is the success payload: the example’s
?? handles the future error and then the worker’s overflow error separately.
Cancellation before execution yields FutureError.Cancelled; it never forcibly
stops running code. done(), cancel(), and wait_timeout_ms(timeout_ms) observe
the same state as the prelude methods. Use std.sync tokens for
cooperative stopping. Pool exit drains and joins; dropping a future does not
cancel its job. Live-receiver shutdown and result-wait diagnostics also apply
through imports. There is no second future or cancellation model.
Acknowledgements
ThreadPoolExecutoris modelled on Python’sconcurrent.futures.
std.crypto
Cryptography here is upstream code, used as-is. Plenty implements no cryptographic algorithm. Every hash, MAC, cipher, signature, key agreement, key derivation and constant-time comparison is the upstream Rust implementation, vendored unmodified and compiled into the runtime: the RustCrypto crates,
subtle, the BLAKE3 team’sblake3, andring. This is deliberate. Constant-time code, side-channel resistance and algorithm correctness stay entirely with those projects and their review and testing. The Plenty side validates arguments, owns buffers, parses key encodings and reports errors. Versions and licences are under Supply and Acknowledgements.
import std.crypto computes BLAKE3, SHA-256, SHA-512 and legacy SHA-1 digests and HMAC tags over
std.bytes buffers, and compares byte buffers in constant time.
The digest and HMAC operations use CryptoError.
The module also provides authenticated encryption, Ed25519 signatures,
X25519 key agreement, and HKDF and
RSA and ECDSA signatures with key import, with
AdvancedError.
The module is available on the supported x86-64 Linux GNU target.
import std.bytes
import std.crypto
import std.encoding
from std.bytes import BytesError
from std.crypto import Algorithm, CryptoError
from std.encoding import EncodingError
def main() -> Result[(), IoError | AllocError | BytesError | CryptoError | EncodingError]:
message = std.bytes.from_text("hello")?
digest = std.crypto.sha256(&message)?
print(std.encoding.hex_text(&digest)?)?
key = std.bytes.from_text("server key")?
tag = std.crypto.hmac_sha256(&key, &message)?
# A receiver recomputes the tag and compares in constant time.
mut check = std.crypto.hmac(Algorithm.Sha256, &key)?
check.update_text("hello")?
print(tag.len(), check.verify(&tag)?)?
Ok(())
2cf24dba5fb0a30e26e83b2ac5b9e29e1b161e5c1fa7425e73043362938b9824
32 True
Functions
See BLAKE3 hashing and key derivation for its one-shot, incremental keyed-hash and derivation functions.
| Function | Result | Contract |
|---|---|---|
sha256(data: &[u8]) | Result[Bytes, CryptoError] | The 32-byte SHA-256 digest of data |
sha512(data: &[u8]) | Result[Bytes, CryptoError] | The 64-byte SHA-512 digest of data |
sha1(data: &[u8]) | Result[Bytes, CryptoError] | The 20-byte SHA-1 digest of data; legacy, see below |
hmac_sha256(key: &[u8], data: &[u8]) | Result[Bytes, CryptoError] | The 32-byte HMAC-SHA-256 tag |
hmac_sha512(key: &[u8], data: &[u8]) | Result[Bytes, CryptoError] | The 64-byte HMAC-SHA-512 tag |
digest(algorithm: Algorithm) | Result[Digest, CryptoError] | A new incremental digest |
hmac(algorithm: Algorithm, key: &[u8]) | Result[Mac, CryptoError] | A new incremental HMAC under key |
equal(a: &[u8], b: &[u8]) | bool | Whether a and b hold the same bytes, in constant time |
output_size(algorithm: Algorithm) | i64 | 32 for Sha256 and Blake3, 64 for Sha512, 20 for Sha1 |
Algorithm is Sha256, Sha512, Sha1 or Blake3. HMAC supports the SHA
algorithms; BLAKE3 has its own keyed mode. HMAC follows RFC 2104 and accepts a key
of any length, including empty; a key longer than the hash block (64 bytes for
SHA-256 and SHA-1, 128 for SHA-512) is hashed first. Choose a key of at least the output
size from std.secrets.
SHA-1 is broken for collision resistance: an attacker can make two inputs with the same digest. It is here only for protocols that fix it, such as the WebSocket handshake and Git object names. Do not use it for signatures, integrity checks against an adversary, or new designs.
Incremental states
Digest and Mac have the same methods. Feeding input in any number of
pieces gives the same output as one call over the joined input.
| Method | Result | Contract |
|---|---|---|
size() | i64 | The output size in bytes |
update(data: &[u8]) | Result[(), CryptoError] | Absorb data; pass &buffer[a:b] for part of a buffer |
update_text(text: &str) | Result[(), CryptoError] | Absorb the UTF-8 bytes of text |
finish() | Result[Bytes, CryptoError] | The output in a new buffer |
finish_into(target: &mut Bytes, offset: i64) | Result[(), CryptoError] | Write the output at target[offset:offset + size()] |
Mac.verify(expected: &[u8]) | Result[bool, CryptoError] | Finish, and compare the tag with expected in constant time |
update and verify read a &[u8] view, whose bounds
clamp; finish_into takes its offset under the
range rule. Each state finishes once: finish,
finish_into, and verify end it, and every later call is Finished. When
finish cannot allocate its result, the state is left unfinished and the call
may be retried. A tag whose length differs from size(), such as a truncated
tag, does not verify; verify does not accept truncation.
The fields are private, so a state can only be made by the module’s constructors.
Both are redacted: printing one shows std.crypto.Digest(@redacted) or
std.crypto.Mac(@redacted).
Digest.finish_into and Mac.finish_into require the complete digest to fit
at offset. Invalid offsets trap with finish_into: offset outside target buffer
before touching the buffer or finalizing the state. A valid output range on
an already-finished state still returns CryptoError.Finished.
Errors
pub enum CryptoError:
Finished
InvalidKeyLength
UnsupportedAlgorithm
Allocation(AllocError)
| Variant | Meaning |
|---|---|
Finished | The state has already produced its output |
InvalidKeyLength | A BLAKE3 key does not contain exactly 32 bytes |
UnsupportedAlgorithm | hmac was called with Algorithm.Blake3; use native keyed BLAKE3 |
Allocation(cause) | A state or output buffer could not be allocated |
Constant-time comparison
equal and Mac.verify use subtle::ConstantTimeEq (through
Mac::verify_slice for HMAC). Their running time depends on the lengths of
the operands, not on their contents or where they first differ. Lengths are
not secret: comparing buffers of different lengths returns False at once.
Digest has no verify; compare a finished digest with equal.
== on Bytes stops at the first difference. Never use it, or a loop of
get calls, to check a tag, token, or password hash: its timing tells an
attacker how many leading bytes of a guess are right.
The guarantee is that of subtle 2.6: comparisons combine bytes without
branches and pass the result through an optimisation barrier. It is a
best-effort property of compiled code, not a hardware guarantee, and it covers
only the comparison, not how the program reacts to the answer.
Key material
A Mac holds its key’s padded forms or BLAKE3 key state. Finishing it and dropping it overwrite
that state with volatile writes before the memory is released. Plenty does not
scrub the rest:
- The key
Bytespassed tohmacbelongs to the caller, who must protect it; Plenty does not zero memory on drop. - Outputs from
finishare ordinary buffers. - Copies the hashing code makes on the stack while it runs are not cleared.
update_textreads astr, which is never scrubbed.
Ownership and threads
A state owns one runtime object and nothing else; it holds no descriptor and
needs no closing. Inputs are shared loans and finish_into’s target is an
exclusive loan. States cannot be copied. The module has no global state,
passes the ordinary worker-effect checks, and may be used from any thread;
moving a state to a worker moves its ownership.
update and finish_into take buffers, not states, as operands. The
__crypto_* intrinsics behind them are
reserved for the standard library. Given a buffer not
made by std.crypto, they stop the program with
operand is not a std.crypto state.
Costs
| Operation | Allocates | Time |
|---|---|---|
SHA digest, hmac | One state object of 368 bytes plus the allocation header | Constant, plus one hash of the key when it exceeds a block |
update, update_text | Nothing | Linear in the input |
finish_into, verify | Nothing | Final blocks; BLAKE3 also folds its bounded tree stack |
finish | One output buffer | As finish_into |
sha256 and the other one-shots | A state and the output | Linear in the input |
equal | Nothing | Linear in the length; buffers of different lengths return at once |
Input is never copied: update hashes directly from the buffer’s storage.
For BLAKE3 state and finalization costs, see its allocation contract.
SHA-256 uses the x86-64 SHA extensions and SHA-512 uses AVX2 when the
processor reports them at run time, and portable code otherwise; the result is
the same.
Supply
The runtime builds these crates from unmodified copies under
plenty-runtime/vendor, so building the compiler needs no network and an
installed compiler needs neither the sources nor a Rust toolchain:
| Crate | Version | Licence |
|---|---|---|
sha2 | 0.10.9 | MIT or Apache-2.0 |
sha1 | 0.10.6 | MIT or Apache-2.0 |
hmac | 0.12.1 | MIT or Apache-2.0 |
subtle | 2.6.1 | BSD-3-Clause |
digest, block-buffer, crypto-common | 0.10.7, 0.10.4, 0.1.7 | MIT or Apache-2.0 |
generic-array, typenum | 0.14.7, 1.20.1 | MIT; MIT or Apache-2.0 |
cpufeatures, cfg-if | 0.2.17, 1.0.4 | MIT or Apache-2.0 |
Each crate’s licence file is kept with its sources. The same versions are
pinned in Cargo.lock, and a test fails if the two disagree. To update one,
replace its directory with the new release, run cargo update -p CRATE --precise VERSION, and rerun the known-answer tests in
tests/test_std_crypto.rs.
Executables are linked with --gc-sections, and the hashing code is reached
only through its own runtime entry symbols, so a program that does not use
std.crypto contains none of it. The link-args.txt written by
--emit-runtime includes that flag.
Limits
- No password hashing or password KDFs (PBKDF2, Argon2).
- No MD5, SHA-3, BLAKE2, or truncated SHA-512 variants.
- No zero-on-drop buffer type; see Key material.
- One target, x86-64 Linux GNU.
Arithmetic and output bounds
Finalization validates the caller offset before subtracting it from the target length or touching the digest state. Only that bounds-proven arithmetic opts out of recoverable integer errors; invalid offsets still trap, and allocation and finished-state errors keep their existing contracts.
Acknowledgements
-
Wraps the BLAKE3 team’s
blake31.8.7 (CC0-1.0 OR Apache-2.0 OR Apache-2.0 WITH LLVM-exception), with dependency credits. -
Wraps the RustCrypto
sha20.10.9,sha10.10.6 andhmac0.12.1 crates (MIT OR Apache-2.0) andsubtle2.6.1 (BSD-3-Clause), vendored in the runtime. -
Wraps
ring0.17.14 (Apache-2.0 AND ISC) for AEAD, Ed25519, X25519, and HKDF and RSA and ECDSA, using the runtime’s existing vendored copy.
BLAKE3 hashing and key derivation
std.crypto provides BLAKE3’s three native modes. Each produces
32 bytes and accepts std.bytes.Bytes input. The incremental forms use the
same Digest and Mac methods and range rules as SHA and HMAC.
Upstream code, used as-is. Every mode is the BLAKE3 team’s
blake3crate, vendored unmodified; Plenty implements none of the hash. Seestd.cryptofor this policy.
| Function | Result | Contract |
|---|---|---|
blake3(data: &[u8]) | Result[Bytes, CryptoError] | Unkeyed hash |
digest(Algorithm.Blake3) | Result[Digest, CryptoError] | Incremental unkeyed hash |
blake3_keyed(key: &[u8], data: &[u8]) | Result[Bytes, CryptoError] | Native keyed hash; exactly 32 key bytes |
blake3_keyed_hasher(key: &[u8]) | Result[Mac, CryptoError] | Incremental keyed hash, including constant-time verify |
blake3_derive_key(context: &str, key_material: &[u8]) | Result[Bytes, CryptoError] | Derive a key from high-entropy material under a context |
blake3_derive_hasher(context: &str) | Result[Digest, CryptoError] | Incremental key derivation |
output_size(Algorithm.Blake3) is 32. A keyed constructor rejects any other
key length with CryptoError.InvalidKeyLength. Generate keys with
std.secrets.token_bytes(32). This keyed mode is BLAKE3’s own
construction; hmac(Algorithm.Blake3, key) returns
CryptoError.UnsupportedAlgorithm.
For derivation, choose a fixed, globally unique, application-specific context
that identifies the purpose, such as "example.org messenger 2026-10-27 session key".
The context is public UTF-8 text, not a salt, password or secret key. Different
purposes should use different contexts. The key material must already have
sufficient entropy: this API is not password hashing or password stretching.
State, allocation and implementation
Each constructor makes one fallible state allocation. BLAKE3 has a larger fixed-size inline tree stack than SHA; it uses a separate object layout, so adding this algorithm does not increase SHA/HMAC state allocations. Context hashing in the derivation constructor needs no extra allocation.
update, update_text, finish_into and Mac.verify allocate nothing,
including when input crosses tree-chunk boundaries. finish allocates an
output buffer; one-shot functions allocate a state and output buffer. An
output allocation failure leaves the state unfinished and retryable. Invalid
ranges also leave it unchanged. A successful finish or a verification
(including a mismatch) ends the state.
The runtime wraps the BLAKE3 team’s blake3 crate 1.8.7 rather than
implementing a cryptographic primitive. Both runtime builds use its pure
feature without std or Rayon: Rust SIMD implementations select SSE2,
SSE4.1 or AVX2 at runtime on supported processors, with a portable fallback.
There are no worker threads, assembly or AVX-512 paths in this configuration.
Only the standard 32-byte output is exposed; extendable output is not exposed.
Finishing or dropping a BLAKE3 state overwrites the stored state using volatile writes. This includes keyed and derivation modes. Caller-owned keys, input buffers, result buffers and temporary upstream stack copies are not scrubbed; see key material.
The unmodified sources and licenses are vendored alongside arrayvec 0.7.8,
constant_time_eq 0.4.2 and cpufeatures 0.3.1. Release notices include these
Cargo dependencies. Tests use the upstream 1.8.7 known-answer vectors for
all three modes and allocation-failure instrumentation for the Plenty API.
Acknowledgements
- Wraps the BLAKE3 team’s
blake31.8.7 implementation and uses its published test vectors (CC0-1.0 OR Apache-2.0 OR Apache-2.0 WITH LLVM-exception); code and licenses are vendored in the runtime. - BLAKE3’s vendored dependencies are
arrayvec0.7.8 for its inline tree stack (MIT OR Apache-2.0),constant_time_eq0.4.2 for comparison support (CC0-1.0 OR MIT-0 OR Apache-2.0), and RustCrypto’scpufeatures0.3.1 for processor detection (MIT OR Apache-2.0).
Authenticated encryption, signatures, and key agreement
std.crypto provides AES-256-GCM, ChaCha20-Poly1305,
Ed25519, ephemeral X25519, and HKDF-SHA256 over Bytes.
Upstream code, used as-is. These are unmodified
ringprimitives. Plenty implements argument validation, buffer ownership and error handling, and no cryptography: the algorithms and their side-channel resistance are deliberately left toring. Seestd.cryptofor this policy.
All functions return Result[_, AdvancedError]. This error is separate from
CryptoError, which describes the digest/HMAC API. See the
signature index for complete declarations and the
encryption tutorial
for runnable examples.
Authenticated encryption
AeadAlgorithm has Aes256Gcm and ChaCha20Poly1305. Both use a 32-byte
secret key, a 12-byte nonce, and a 16-byte authentication tag.
seal_in_place(algorithm, key: &[u8], nonce: &[u8], aad: &[u8],
data: &mut Bytes) -> Result[(), AdvancedError]
open_in_place(algorithm, key: &[u8], nonce: &[u8], aad: &[u8],
data: &mut Bytes) -> Result[(), AdvancedError]
seal_in_place replaces plaintext with ciphertext followed by its tag. It
reserves/grows the buffer before encrypting; reserve 16 additional bytes in
advance to avoid allocation. open_in_place authenticates and decrypts that
format, removing the tag on success. Associated data (aad) is authenticated
but never encrypted or appended; the recipient must supply identical bytes.
Empty plaintext and empty associated data are supported. These APIs cap
plaintext at 68,719,476,704 bytes for both algorithms.
Never reuse a nonce with the same key. The API does not remember nonces
or manage counters, persistent state, key rotation, replay protection, or
message framing. Use a protocol that specifies these. Keys should come from
std.secrets or an appropriate key derivation scheme.
Invalid lengths leave data unchanged. If growth fails, plaintext remains
unchanged. Authentication failure zeros the entire ciphertext/tag buffer,
retains its original length, and returns AuthenticationFailed. It never
returns unauthenticated plaintext. Successful opening allocates nothing.
The other inputs and output must be separate buffers under the loan rules.
Ed25519
| Function | Output |
|---|---|
ed25519_public_key(seed) | A new 32-byte public key |
ed25519_public_key_into(seed, target) | Writes an exactly 32-byte buffer |
ed25519_sign(seed, message) | A new 64-byte signature |
ed25519_sign_into(seed, message, target) | Writes an exactly 64-byte buffer |
ed25519_verify(public_key, message, signature) | True for a valid signature; False for a mismatch; SmallOrderKey for a small-order key |
Every input is a &[u8] view; target is &mut Bytes. A seed must be exactly
32 cryptographically random bytes. Generate one with std.secrets.token_bytes(32);
its public key may be shared, but its seed must remain secret. Signing is
deterministic, and a signing seed can sign multiple messages. Verification
requires exactly 32 public-key bytes and 64 signature bytes; other lengths
return InvalidLength, whereas correctly sized malformed keys/signatures
return False. This is plain Ed25519, without a prehash or context mode.
Public keys must be authenticated by the application’s protocol.
Small-order public keys
ed25519_verify and SPKI import reject a
small-order public key with SmallOrderKey. Such a key is a point whose
order divides 8; no private key stands behind it, so anyone can make
signatures that verify under it. Under the identity point, the signature
R = B, S = 1 verifies for every message. The rule covers all 14
encodings of the eight points, including the non-canonical ones (y = p and
y = p + 1) and both sign bits. Verification checks lengths first, so a
small-order key with a wrong-length signature is still InvalidLength. The
result is an error rather than False because no answer about the signature
is meaningful for such a key.
import std.bytes
import std.crypto
import std.fmt
import std.encoding
import std.report
def main() -> Result[(), IoError | AllocError | std.bytes.BytesError | std.encoding.EncodingError | std.fmt.FormatError]:
identity = std.encoding.hex_from_text("0100000000000000000000000000000000000000000000000000000000000000")?
forged = std.encoding.hex_from_text("58666666666666666666666666666666666666666666666666666666666666660100000000000000000000000000000000000000000000000000000000000000")?
message = std.bytes.from_text("any message")?
match std.crypto.ed25519_verify(&identity, &message, &forged):
case Ok(valid):
print(valid)?
case Err(error):
print(std.report.to_string(&error, 256, 4)?)?
Ok(())
Ed25519 public key has small order: signatures can verify under it without any private key, so it is rejected
RFC 8032 permits these keys and
ring accepts them, so a peer that publishes one interoperates with RFC 8032
verifiers but not with Plenty. libsodium rejects the same set.
Mixed-order keys, a prime-order point plus a small-order one, are accepted.
Signing under one still needs the private key of its prime-order part, so
they do not allow forgery. They do let a key holder produce signatures that
verifiers with and without the cofactor in their equation judge differently.
Rejecting them needs a scalar multiplication by the group order, which ring
does not expose; libsodium’s verification accepts them too. The
Wycheproof vectors pin both rules.
X25519
x25519() -> Result[X25519Key, AdvancedError] generates an ephemeral private
key with ring’s operating-system random source. The private key has no
export/import operation. An X25519Key has these methods:
| Method | Contract |
|---|---|
public_key() | Returns a new 32-byte buffer |
public_key_into(target: &mut Bytes) | Writes an exactly 32-byte buffer |
agree(peer_public_key: &[u8]) | Consumes the key’s agreement capability and returns 32 shared-secret bytes |
agree_into(peer_public_key: &[u8], target: &mut Bytes) | Same, into an exactly 32-byte buffer |
Both agreement methods require a mutable receiver. An agreement attempt
consumes the private key even when the peer is rejected. A peer producing an
all-zero shared secret is InvalidPeer; agree_into zeros its output on
that failure. Later agreement or public-key requests return Finished.
Invalid lengths and output allocation failure leave the key usable and
existing output unchanged. Exchange public keys before calling agree.
X25519Key is redacted: printing one shows
std.crypto.X25519Key(@redacted).
X25519 alone does not authenticate the peer. Authenticate the exchange through a suitable protocol and pass the shared secret through HKDF with protocol-specific salt and context before using it as a symmetric key.
HKDF-SHA256
hkdf_sha256(salt: &[u8], key_material: &[u8], info: &[u8], size: NonNegative[i64])
-> Result[Bytes, AdvancedError]
hkdf_sha256_into(salt: &[u8], key_material: &[u8], info: &[u8],
target: &mut Bytes) -> Result[(), AdvancedError]
These perform HKDF extract followed by expand. Salt and context (info) can
be empty. Output length is 0 through 8160 bytes inclusive. hkdf_sha256
traps for a larger size; the _into form uses the existing target length and
returns InvalidLength for one over 8160, leaving the output unchanged. Use context to separate keys with different purposes. HKDF is
not password hashing and does not make a weak password into a strong key.
Errors, allocation, and key material
AdvancedError | Meaning |
|---|---|
InvalidLength | Incorrect key/nonce/tag/output length or output beyond the supported bound |
AuthenticationFailed | AEAD authentication failed |
InvalidPeer | X25519 rejected the peer key |
Finished | X25519 key already used for an agreement attempt |
RandomFailure | OS entropy or X25519 key/public-key generation failed |
InvalidKey, WeakKey | Raised only by RSA, ECDSA and key import |
SmallOrderKey | An Ed25519 public key has small order |
Allocation(AllocError) | A fallible buffer or private-state allocation failed |
AdvancedError satisfies std.fmt.Formattable: each variant
formats as a fixed sentence stating its cause, without key material. An
entrypoint report uses it when
main returns a bare AdvancedError; an error sum uses it only when every
member is formattable.
The _into functions, verification, and opening allocate nothing. Sealing
allocates only if growth exceeds capacity. Functions returning Bytes
allocate their result. X25519 creation fallibly allocates its state and a
short-lived 32-byte public-key buffer to validate generation before returning.
Failure at any of these allocation points is reported as Allocation.
The chosen ring paths use fixed-size stack key objects, signatures, tags,
and HKDF state. They do not use its allocating PKCS#8 or append-tag APIs.
Tests enforce the allocation-free paths with the runtime’s allocation guard.
The X25519 state’s private-key slot is overwritten with volatile zero writes
when consumed or dropped. This does not scrub copies in registers or on the
stack, caller-owned seeds/keys/shared-secret buffers, or allocator caches.
Bytes does not zero its storage on drop; see key material.
There are no password KDFs, streaming AEAD, or persistent private-key stores
in this API. Ed25519 keys can be read from PKCS #8 and SubjectPublicKeyInfo
with the key import functions. It supports the project’s
x86-64 Linux GNU target. Runtime entry symbols keep unused primitive paths
eligible for linker garbage collection; ring is also used by std.tls.
Arithmetic bounds
AEAD validates protocol lengths before adding or removing its 16-byte tag. These bounded operations cannot overflow; existing invalid-length, allocation, and authentication errors remain unchanged, including validation and allocation before encryption mutates plaintext.
Acknowledgements
- Wraps
ring0.17.14 (Apache-2.0 AND ISC), already vendored for TLS, for all primitives on this page. Its complete license text and third-party notices are included by the release notice generator through the Cargo dependency graph. - RFC 8032 supplies the Ed25519 known-answer test.
- RFC 5869 supplies the HKDF-SHA256 known-answer test.
- RFC 8439 supplies the ChaCha20-Poly1305 known-answer test.
- The published GCM test cases supply the AES-256-GCM known-answer test.
RSA and ECDSA signatures and key import
std.crypto signs and verifies with RSA (PKCS #1 v1.5 and PSS)
and ECDSA over P-256, and imports public and private keys from the standard
DER encodings, JWK components, and (through std.encoding)
PEM. These are the signature schemes of JOSE and JWT: RS256, PS256 and ES256,
with the SHA-384 and SHA-512 RSA variants. Ed25519 keys import here too and
sign with the Ed25519 functions.
Upstream code, used as-is. Signing, verification and private-key parsing are unmodified
ringprimitives; Plenty implements no cryptographic arithmetic. Plenty parses public-key encodings and checks RSA key sizes itself, in checked code, so a public key reachesringonly after its structure is validated. Seestd.cryptofor this policy. Every function returnsResult[_, AdvancedError]. See the signature index for complete declarations.
import std.bytes
import std.crypto
import std.encoding
from std.bytes import BytesError
from std.crypto import AdvancedError
from std.encoding import EncodingError
def main() -> Result[(), IoError | AllocError | BytesError | AdvancedError | EncodingError]:
# The ES256 example from RFC 7515 appendix A.3: a JWK's x and y, the
# signing input, and its fixed 64-byte signature.
x = std.encoding.base64url_from_text("f83OJ3D2xF1Bg8vub9tLe1gHMzV76e8Tus9uPHvRVEU")?
y = std.encoding.base64url_from_text("x_FEzRu9m36HLN_tue659LNpXW6pCyStikYjKIWI5a0")?
key = std.crypto.p256_public_key_from_coordinates(&x, &y)?
mut message = std.bytes.from_text("eyJhbGciOiJFUzI1NiJ9.eyJpc3MiOiJqb2UiLA0KICJleHAiOjEzMDA4MTkzODAsDQogImh0dHA6Ly9leGFtcGxlLmNvbS9pc19yb290Ijp0cnVlfQ")?
signature = std.encoding.base64url_from_text("DtEhU3ljbEg8L38VWAfUAqOyKAM6-Xx-F4GawxaepmXFCgfTjDxw5djxLa8ISlSApmWQxfKTUJqPP3-Kg6NU1Q")?
print(key.verify(&message, &signature)?)?
message.set(0, 0u8)
print(key.verify(&message, &signature)?)?
Ok(())
True
False
Key types
Each key type is an opaque class for one algorithm. There is no conversion
between them, no way to read a private key’s material back out, and no way to
use one as an HMAC key or Ed25519 seed. The constructors are private: a key
exists only after an import function validated it. Keys cannot be copied.
Private keys are redacted: printing one shows
std.crypto.RsaPrivateKey(@redacted) or std.crypto.P256PrivateKey(@redacted).
| Type | Holds | Methods |
|---|---|---|
RsaPublicKey | A 2048- to 8192-bit modulus and its exponent | signature_size(), verify(scheme, message, signature) |
RsaPrivateKey | A 2048- to 4096-bit two-prime key, held by the runtime | signature_size(), public_key(), sign(scheme, message), sign_into(scheme, message, target) |
P256PublicKey | A point on P-256 | verify(message, signature) |
P256PrivateKey | A P-256 scalar and its public point, held by the runtime | public_key(), sign(message), sign_into(message, target) |
RsaScheme selects the RSA padding and hash for each call:
RsaScheme | JOSE alg | Padding |
|---|---|---|
Pkcs1Sha256, Pkcs1Sha384, Pkcs1Sha512 | RS256, RS384, RS512 | PKCS #1 v1.5 (RFC 8017 section 8.2) |
PssSha256, PssSha384, PssSha512 | PS256, PS384, PS512 | PSS with MGF1 over the same hash and a salt of the hash length (RFC 7518 section 3.5) |
P-256 keys use ECDSA with SHA-256 only (JOSE ES256). Choose the scheme from the key’s configured algorithm, never from an unauthenticated message header.
Import
| Function | Input | Result |
|---|---|---|
rsa_public_key_from_spki(der) | SubjectPublicKeyInfo with rsaEncryption and NULL parameters | RsaPublicKey |
rsa_public_key_from_components(n, e) | Big-endian modulus and exponent, as a JWK’s decoded n and e | RsaPublicKey |
rsa_private_key_from_pkcs8(der) | Unencrypted PKCS #8 v1 holding an RSAPrivateKey | RsaPrivateKey |
p256_public_key_from_spki(der) | SubjectPublicKeyInfo with id-ecPublicKey and the named curve prime256v1 | P256PublicKey |
p256_public_key_from_point(point) | A 65-byte uncompressed SEC1 point: 0x04, x, y | P256PublicKey |
p256_public_key_from_coordinates(x, y) | 32-byte big-endian coordinates, as a JWK’s decoded x and y | P256PublicKey |
p256_private_key_from_pkcs8(der) | Unencrypted PKCS #8 v1 holding an ECPrivateKey with its public key | P256PrivateKey |
ed25519_public_key_from_spki(der) | SubjectPublicKeyInfo with id-Ed25519; a small-order key is SmallOrderKey | The 32-byte public key, as Bytes |
ed25519_seed_from_pkcs8(der) | Unencrypted PKCS #8 v1 or v2 holding an Ed25519 private key | The 32-byte seed, as Bytes |
Every input is &Bytes. These are the forms openssl genpkey and
openssl pkey -pubout write. For PEM, decode with
std.encoding.pem_decode(source, b"PUBLIC KEY") or
b"PRIVATE KEY" first; the label must match. Encrypted PKCS #8, the
PKCS #1 RSA PRIVATE KEY and SEC1 EC PRIVATE KEY forms, and X.509
certificates are not accepted.
DER is parsed strictly: lengths must be definite and minimal, integers minimal
and positive, and nothing may follow the outer element. The algorithm
identifier must be exactly the expected encoding, so another algorithm,
another curve, explicit curve parameters, or RSA-PSS-restricted keys
(id-RSASSA-PSS) are rejected. Public points must be uncompressed and on the
curve. RSA public keys must have an odd modulus of 2048 to 8192 bits without
a leading zero byte and an odd exponent from 3 to 233 - 1; private
keys are limited by ring to 2048 to 4096 bits, an exponent of at least
65537, and two primes, and their components must be consistent. A P-256
PKCS #8 key must include its public key, which must match the scalar.
An Ed25519 v2 key may include its public key, which must match the seed.
Validating a public point runs one ephemeral P-256 key agreement, because
ring has no separate point check; it needs operating-system randomness and
can fail with RandomFailure.
Signing and verification
verify returns True for a valid signature and False for any signature
that does not verify, including malformed values of the right length. Only
misuse is an error: a signature whose length is wrong for the key is
InvalidLength. An RSA signature is exactly signature_size() bytes, the
modulus length. An ECDSA signature is the fixed 64-byte r || s form that JOSE
uses (RFC 7518 section 3.4),
so a DER-encoded ECDSA signature never verifies; it is InvalidLength.
sign returns a new buffer; sign_into writes a target of exactly the
signature size and otherwise returns InvalidLength without writing.
PKCS #1 v1.5 signatures are deterministic. PSS draws a random salt and ECDSA
a random nonce (hedged by ring with the key and message), so repeated
signatures differ, and both can fail with RandomFailure. A PSS failure zeros
the target.
Errors
AdvancedError | Meaning here |
|---|---|
InvalidKey | A key encoding is malformed, uses another algorithm or curve, or breaks a size or consistency rule other than the minimum |
WeakKey | An RSA modulus is shorter than 2048 bits |
SmallOrderKey | An Ed25519 public key has small order |
InvalidLength | A signature or signing target has the wrong length for the key |
RandomFailure | Operating-system randomness failed during point validation, PSS or ECDSA signing, or private-key import |
Allocation(AllocError) | A fallible buffer or key-state allocation failed, or memory for an RSA operation could not be reserved |
These extend the existing AdvancedError rather than adding a type, because
the operations are the same family as Ed25519 and share its length, entropy
and allocation failures; one error type keeps a JWT verifier’s error set small.
InvalidKey and WeakKey carry no payload: no offset, field name or value,
so an error message cannot reveal anything about a private key.
Allocation and key material
Each import allocates its result fallibly: a public key’s validated DER or
point, a private key’s runtime state object, or a returned Ed25519 key or seed.
rsa_public_key_from_components builds the DER from n and e. P-256
signing into an existing target and P-256 verification allocate nothing;
tests enforce this with the runtime’s allocation guard, as they do for
ring’s P-256 private-key parsing and point validation.
ring’s RSA operations allocate big-number buffers through the Rust global
allocator, which cannot report failure. Before each one the runtime reserves a
fixed block, then runs ring with that thread’s allocations taken from it:
| Operation | Reserved | When | Released |
|---|---|---|---|
rsa_private_key_from_pkcs8 | 12 KiB | before parsing | when the key is dropped; at once if parsing fails |
RsaPrivateKey.sign, sign_into | 8 KiB | each call | when the call returns |
RsaPublicKey.verify | 12 KiB | each call | when the call returns |
signature_size and public_key reserve nothing; public_key allocates only
its result. Each size is at least twice the most that ring 0.17.14 uses for
the largest key it accepts: a 4096-bit private modulus or an 8192-bit public
one. ring makes a fixed number of buffers per operation, each at most
1 KiB, so the use does not depend on the message or the key’s value. A test
measures it and fails if a ring update outgrows a block.
If the block cannot be reserved, or ring asks for more than it holds, the
call returns Allocation. A failure inside ring abandons its frames, which
hold only the block and the stack, so nothing else needs repair; the block
is then released. The key, the target and the program stay usable; a failed
signature leaves the target zeroed. The blocks come from the current heap: in
an actor handler they count against the actor’s
heap limit.
A private key’s runtime state is overwritten with volatile zero writes when it
is dropped. For P-256 that covers the scalar and nonce key, which ring keeps
inline. For RSA, the key’s reserved block, which holds the primes, exponents
and coefficients, is zeroed before it is freed, as is each call’s block. A
key may be used, and dropped, on any thread. Not scrubbed:
- the caller’s DER or PEM input buffers, and the intermediate base64 and
decoded buffers that
std.encoding.pem_decodereleases; - the seed returned by
ed25519_seed_from_pkcs8, which belongs to the caller (on rejection, the function zeros its own copy); - copies
ringmakes in registers or on the stack while it runs.
Bytes does not zero its storage on drop; see
key material.
Test vectors
Every valid and invalid case in these
Wycheproof files gives the expected
result, with the key imported through each available form (SPKI, JWK
members, and the raw point or key):
| File | Cases | Result |
|---|---|---|
rsa_signature_2048_sha256, _2048_sha384, _2048_sha512, _3072_sha256, _4096_sha512 | 258 to 259 each | All pass |
rsa_pss_2048_sha256_mgf1_32, _2048_sha384_mgf1_48, _4096_sha512_mgf1_64 | 108, 141, 179 | All pass |
ecdsa_secp256r1_sha256_p1363 | 262 | All pass |
ecdsa_secp256r1_sha256 (DER signatures) | 484 | None verifies: each is InvalidLength |
ed25519 | 210 | All pass |
Cases marked acceptable are pinned to the observed behavior:
- The one PKCS #1 v1.5 case per file whose DigestInfo omits the NULL
parameters (
MissingNull) is rejected. - The 8 Ed25519 cases with small-order public keys (tcId 152 to 159) are
SmallOrderKey, from SPKI import and from verification. The 21 with mixed-order keys (160 to 180) verify. See small-order public keys.
The extracts and their generator, with the upstream commit and file digests,
are in tests/fixtures/crypto/wycheproof.
Limits
- No key generation for RSA or P-256, and no private-key export.
- No P-384, P-521, secp256k1, or ECDSA with DER signatures.
- No encrypted PKCS #8, PKCS #1 or SEC1 private-key forms, or certificates.
- No JWK parsing: decode the JSON and base64url members and pass the bytes.
- No RSA encryption.
- One target, x86-64 Linux GNU.
Arithmetic bounds
DER cursors stay within validated ordered buffer spans; length encodings use at
most three bytes. RSA import rejects moduli longer than 1024 bytes before bit
counting, and bounds exponents to five bytes. Consequently bit counts and DER
output extents cannot overflow. Small arithmetic opt-outs preserve InvalidKey
and WeakKey rather than exposing machine-arithmetic errors for malformed keys.
Acknowledgements
- Wraps
ring0.17.14 (Apache-2.0 AND ISC), already vendored for TLS, for RSA and ECDSA signing, verification, and private-key parsing. - Project Wycheproof (C2SP/wycheproof, Apache-2.0) supplies the RSA, ECDSA and Ed25519 test vectors.
- RFC 7515 appendices A.2 and A.3 supply the RS256 and ES256 known-answer tests, and RFC 8037 appendix A the Ed25519 one.
- Encodings follow RFC 5280 (SubjectPublicKeyInfo), RFC 5208 and RFC 5958 (PKCS #8), RFC 5480 and RFC 5915 (elliptic-curve keys), RFC 8410 (Ed25519 keys), and RFC 8017 (RSA).
std.csv
import std.csv reads and writes delimited records over
std.io byte streams. A Reader fills one reusable Record per
call from a fixed buffer. A Writer encodes one record at a time. Neither
loads a whole file, and every size is bounded by the Dialect.
The module is checked Plenty over std.bytes and std.io, with no
native calls of its own. It is available on the supported x86-64 Linux GNU
target.
import std.bytes
import std.csv
import std.io
from std.csv import Newline
def main() -> Result[(), AllocError | IoError | ParseError | std.bytes.BytesError | std.csv.CsvError | OverflowError[i64]]:
text = "item,price\r\npen,3\r\n\"book, large\",12\r\n"
mut input = std.csv.reader(std.io.cursor(std.bytes.from_text(text)?), &std.csv.rfc4180(), 4096)?
mut record = std.csv.record()?
input.read(&mut record)?
mut total = 0
while input.read(&mut record)?:
total += i64.parse(&record.text(1)?)?
mut dialect = std.csv.rfc4180()
dialect.newline = Newline.Lf
mut output = std.csv.writer(std.io.stdout(), &dialect)?
output.texts(&["rows", "total"]?)?
output.texts(&[str.from(input.records() - 1)?, str.from(total)?]?)?
output.flush()?
Ok(())
rows,total
2,15
Format
The grammar is RFC 4180 with a configurable delimiter and quote byte. There is one accepted form; nothing is trimmed, skipped, or repaired.
- A record ends at LF, CRLF, or CR. The last record may end at end of input.
- Fields are separated by the delimiter.
a,has two fields, the second empty. - An empty line is one record with one empty field. Empty input has no records.
- A field that starts with the quote byte is quoted and ends at the next unpaired quote. Inside it, the delimiter, CR, and LF are data, and two quotes are one quote byte.
- A closing quote must be followed by the delimiter, a record end, or end of input.
- A quote byte anywhere else in an unquoted field is an error.
- Spaces are data, including those next to a delimiter or quote.
There are no headers, comments, or escape characters other than the doubled quote. The first record is an ordinary record.
Dialect
Dialect is a class with public fields. std.csv.rfc4180() returns the
values in the third column; change fields before creating a reader or writer.
| Field | Type | rfc4180() | Meaning |
|---|---|---|---|
delimiter | u8 | 44u8 (,) | Field separator |
quote | u8 | 34u8 (") | Quote byte |
newline | Newline | CrLf | Record end a Writer emits: Lf or CrLf |
quoting | Quoting | Needed | Needed quotes a field that contains the delimiter, the quote, CR, or LF; Always quotes every field |
width | Width | First | Row-width policy, below |
skip_bom | bool | False | A Reader discards a leading UTF-8 byte-order mark |
max_field | i64 | 65536 | Most bytes in one unescaped field |
max_record | i64 | 1048576 | Most bytes in the unescaped fields of one record |
max_fields | i64 | 4096 | Most fields in one record |
delimiter and quote must be different ASCII bytes other than CR and LF,
each limit at least one, and Width.Exactly(count) at least one. Otherwise
reader and writer return InvalidDialect. A reader accepts all three
record ends whatever newline says, and ignores quoting.
Width | Policy |
|---|---|
Any | Each record keeps its own field count |
First | Every record has the field count of the first |
Exactly(count) | Every record has count fields |
Fields are bytes. The module never decodes them on its own: Record.text
validates one field as UTF-8 when asked, and a writer emits the bytes it is
given. Other encodings are read with Record.bytes and converted by the
program.
Errors
CsvError | Meaning |
|---|---|
Invalid(Fault, Position) | A reader found malformed or oversized input |
Rejected(Fault, record, field) | A writer refused a field that exceeds a limit |
Width(record, expected, found) | A record’s field count breaks the width policy |
EmptyRecord(record) | A writer was asked to end a record with no fields |
InvalidUtf8(record, field) | Record.text found bytes that are not UTF-8 |
InvalidDialect | The dialect breaks the rules above |
Stopped | The reader already failed |
Stream(StreamError) | The inner stream failed |
Allocation(AllocError) | A record or buffer could not grow |
Fault | Reader | Writer |
|---|---|---|
BareQuote | A quote inside an unquoted field | |
AfterQuote | Data between a closing quote and the next delimiter or record end | |
UnterminatedQuote | Input ended inside a quoted field | |
FieldTooLong | A field exceeds max_field | The same |
RecordTooLong | A record exceeds max_record | The same |
TooManyFields | A record exceeds max_fields | The same |
Position locates the byte a reader stopped at:
| Field | Counts from | Meaning |
|---|---|---|
record | 1 | The record being read; matches Record.number() |
field | 0 | The field being read; matches the index Record.text takes |
line | 1 | The physical line, counting record ends and line breaks inside quotes; CRLF counts once |
offset | 0 | The byte offset in the stream; for UnterminatedQuote, the input length |
A limit fault points at the first byte that does not fit. Record numbers and field indexes in the other variants use the same bases.
Reading
| Function | Result |
|---|---|
reader(inner: R, dialect: &Dialect, capacity: Positive[i64]) | Result[Reader[R], CsvError] |
record() | Result[Record, CsvError] |
reader takes ownership of inner, copies the dialect, and allocates one
buffer of exactly capacity bytes, which never grows. capacity is
Positive[i64], so 0 is a compile error. inner needs the Reader method of std.io. The reader does
its own buffering, so inner needs no BufReader.
Reader[R] | Result | Contract |
|---|---|---|
read(record: &mut Record) | Result[bool, CsvError] | Replace record with the next record; False is end of input |
records() | i64 | Records returned so far |
line() | i64 | The current line, from one |
offset() | i64 | Bytes consumed from the stream |
read retries Interrupted and continues across short reads; a record may
span any number of buffer fills. After False, the record is empty and every
later call returns False without reading the stream again.
Record, line and byte positions are semantic i64 counters, not saturating
diagnostics. If the next position is unrepresentable, reading returns
Stream(StreamError.OutOfRange), empties the record and stops the reader.
Record number i64.MAX is still processable when the preceding count is
i64.MAX - 1. A failing line increment leaves the parser cursor before that
newline; CRLF still counts as one line, inside and outside quotes. The inner
stream may already have read ahead: parser consumption is distinct from
transport progress. Buffer fills are capped by the remaining representable
byte span; at zero room, a reader can refuse before an EOF probe.
A Width error is reported after the whole record has been read: record
holds it, and the next read continues with the following record. Every
other error empties record and stops the reader; later calls
return Stopped. A stopped reader cannot resynchronize, because a malformed
quote leaves no reliable record boundary.
inner is a public field, for calls such as input.inner.close(). Reading it
directly skips bytes the reader already holds.
Records
A Record stores the unescaped fields end to end in one buffer and one list
of field ends. A reader overwrites it, keeping its capacity.
Record | Result | Contract |
|---|---|---|
len() | i64 | Number of fields |
number() | i64 | The record’s number from one; zero before the first read |
line() | i64 | The line on which the record started |
text(index) | Result[str, CsvError] | An owned copy of one field, validated as UTF-8 |
texts() | Result[list[str], CsvError] | An owned list of every field as text |
bytes(index) | Result[Bytes, CsvError] | An owned copy of one field |
append_to(index, out: &mut Bytes) | Result[(), CsvError] | Append one field to the caller’s buffer |
data() | &Bytes | Every field’s bytes, borrowed |
start(index), stop(index) | i64 | Field index is data()[start:stop] |
clear() | () | Remove the fields; number() and line() are kept |
An index outside 0 <= index < len() traps before reading or appending.
The message names Record.start or Record.stop and index outside record;
bytes, text, and append_to validate through those position methods.
data() is the borrowed view. It and the offsets describe the current
contents only. Reader.read needs &mut Record, so the compiler rejects a
program that still holds the view across the next read. text, texts,
bytes, and append_to return or fill storage the caller owns, which
survives any number of reads.
Writing
| Function | Result |
|---|---|
writer(inner: W, dialect: &Dialect) | Result[Writer[W], CsvError] |
writer takes ownership of inner, which needs the Writer methods of
std.io, and copies the dialect.
Writer[W] | Result | Contract |
|---|---|---|
field(source: &[u8]) | Result[(), CsvError] | Add source as the next field |
text(value: &str) | Result[(), CsvError] | Add the UTF-8 bytes of value as the next field |
end_record() | Result[(), CsvError] | Finish the record in progress and deliver it |
record(record: &Record) | Result[(), CsvError] | Add every field of record, then end the record |
texts(fields: &list[str]) | Result[(), CsvError] | Add every text, then end the record |
discard() | () | Drop the record in progress |
flush() | Result[(), CsvError] | Deliver held bytes, then flush the inner stream |
records() | i64 | Records finished so far |
pending() | i64 | Bytes of finished records the stream has not accepted |
A record in progress lives in the writer’s buffer and reaches the stream only
at end_record, as one run of writes that retries Interrupted and short
counts. A record with one empty field is written as two quote bytes, so that
it is not an empty line. end_record with no fields is EmptyRecord.
Failures leave a defined state:
RejectedandAllocationfromfieldortextadd nothing; the record in progress keeps its earlier fields.Widthfromend_recorddiscards the record in progress. Nothing of it was written. WithWidth.First, the first finished record sets the count.- A transport
Streamerror fromend_recordmeans the record is finished and counted, and its undelivered bytes are held. The nextfield,end_record, orflushdelivers them first and fails again if the stream still refuses. recordandtextsdiscard the record in progress on any error. AStreamerror still leaves the finished record held.
Position exhaustion is distinct from that ordinary transport retry: after
record i64.MAX is finished, field, text and end_record reject an
unrepresentable next record as Stream(StreamError.OutOfRange) before
encoding or implicitly draining any pending output. Already-finished pending
bytes and their count remain intact; explicit flush still delivers them.
No counter wraps, saturates, or adds a public arithmetic error type. Internal
field/content and pending-buffer arithmetic uses validated remaining budgets
and held-buffer bounds; scan endpoints clamp room before adding it to a cursor.
Nothing is written when a Writer is dropped. End the last record and call
flush. The writer does not close inner; it is the public field inner.
Ownership and threads
A reader or writer owns its stream and has one user at a time: every
operation that moves data takes &mut self. A Record is an ordinary owned
value, independent of the reader that filled it, and can be moved to another
thread. The module holds no global state and adds no synchronization.
Costs
Reading examines each input byte once and copies field bytes into the record in runs. Writing examines each field byte once to decide quoting, and once more when quoting to double quote bytes. Both are linear in the bytes processed. Field access by index is constant time.
The reader and writer find the next delimiter, quote, or line break with the
std.bytes scans, so the work in Plenty code is per
field and per quote, not per byte. Measured with a release compiler on x86-64
Linux, pinned to one core, reading 64 MB from standard input into one reused
Record:
| Input | Before the scans | Now |
|---|---|---|
| 3,234,608 fields of about 19 bytes, some quoted | 4.73 s (14 MB/s) | 2.00 s (32 MB/s) |
| 376,760 fields of about 180 bytes, some quoted | 3.37 s (19 MB/s) | 0.30 s (213 MB/s) |
Short fields are bounded by the cost of the calls made per field, about 600 ns, not by scanning.
| Allocates | Does not allocate |
|---|---|
reader: one buffer; record, writer: empty storage | Reader.read once the record’s capacity covers the record |
Reader.read, when a record outgrows the Record | Record.append_to within the capacity of out |
Record.text and bytes: one copy; texts: one list and a copy per field | Record.data, start, stop, len, clear |
Writer.text: one temporary copy of the text | Writer.field, record, end_record, flush within the writer’s capacity |
Writer.field, when a record outgrows the writer’s buffer | Writer.discard, records, pending |
A Record holds at most max_record bytes of content and max_fields
offsets. A reader consumes at most 2 * max_record + 3 * max_fields + 2
bytes for one record before returning it or a limit fault. A writer holds at
most one finished record that the stream refused and one record in progress,
each at most that many bytes. Storage grows geometrically, so capacity can be
up to twice the content.
Limits
- Only x86-64 Linux GNU is supported.
- There is no lenient mode: stray quotes, and data after a closing quote, are errors, and a reader does not continue after them.
- There are no header mapping, typed columns, comment lines, or whitespace
trimming. Convert fields with the type’s
parsemethod. - Only the UTF-8 byte-order mark is recognized, and only with
skip_bom. A writer never emits one. Other encodings need conversion by the program. delimiterandquoteare single ASCII bytes.- A stream error that could be retried, such as
EAGAINon a nonblocking descriptor, stops the reader like any other. - End of input is final: a reader does not look for data appended later.
Writer.textcopies the text once to reach its bytes. Usefieldwith a reusedBytesto avoid the copy.
std.dataframe
Polars-backed analytical dataframes, expression plans and CSV/Parquet files. The optional, separately compiled engine is reused by every Plenty program. See installation and deployment and the tutorial. The daily sales report application combines ingestion, checked catalogue joins, row quarantine and typed exports.
Use clist[T] for a lightweight mutable columnar collection with a
compile-time class schema and Plenty arithmetic. Use std.dataframe for
analytical queries and file-backed datasets with runtime schemas. Neither
replaces ordinary lists.
import std.dataframe as df
def main() -> Result[(), df.DataFrameError | AllocError | IoError]:
data = df.frame([
df.column_str("station", ["north", "south", "north", "south"]?)?,
df.column_f64("reading", [12.0, 30.0, 18.0, 24.0]?)?
]?)?
recent = df.col("reading")?.gt(df.lit(15.0)?)?
keys = [df.col("station")?]?
averages = [df.col("reading")?.mean()?.alias("average")?]?
result = data.lazy()?.filter(recent)?.group_by(keys, averages)?.sort(
[df.SortKey(df.col("station")?)]?
)?.collect()?
print(result.values_str("station")?)?
print(result.values_f64("average")?)?
Ok(())
["north", "south"]
[18.0, 27.0]
Construction and extraction
column_i64(name: &str, values: &list[i64]) and
column_f64(name: &str, values: &list[f64]) return
Result[Column, DataFrameError]. Each makes one native call and copies the
borrowed numeric values into a native contiguous column. Names are copied;
inputs remain usable. No input pointer survives the call.
frame(columns: &list[Column]) -> Result[DataFrame, DataFrameError]
shares the columns’ native buffers. Duplicate names and unequal lengths fail.
An empty column list creates a zero-row, zero-column frame. Zero-length
typed columns retain their types. This engine supports at most
2**32 - 1 rows.
DataFrame.height() returns Result[i64, DataFrameError].
get_i64(name, row) and get_f64(name, row) return
Result[Option[i64], DataFrameError] and
Result[Option[f64], DataFrameError]. Bounds, missing names and mismatched
types fail; a valid null cell returns Ok(Nothing).
values_i64(name) and values_f64(name) return a Plenty list in a
Result. They allocate initialized Plenty storage, then copy the whole
column in one native call. A column containing any nulls fails with
DataFrameError.Null; use values_i64_nullable or values_f64_nullable
to preserve missing values, or fill_null when replacement is appropriate.
Strings, booleans, binary payloads, nullable input/output and schema inspection are described
in Column types and schemas. No implicit casts or
zero-copy borrowing occur at the Plenty boundary.
Logical Date and Datetime columns add checked temporal
interchange, parsing, formatting and calendar expressions.
Expression plans
Every operation below returns a Result[..., DataFrameError].
The signature index gives exact declarations.
| Operation | Result value |
|---|---|
col(name), lit(f64), lit_i64(i64), lit_str(text), lit_bool(value), lit_binary(payload: &[u8]), null_f64() | Expr |
when(predicate, then_value, otherwise), expr.cast(dtype, strict=True) | Expr |
expr.str_lower/str_upper/str_len(), str_contains/str_starts_with/str_ends_with(pattern) | Expr |
expr.n_unique(), expr.over(partition_by) | Expr |
expr.add/sub/mul/div/gt/eq/lt/ge/le(&other) | Expr |
expr.and_(&other), or_(&other), fill_null(&other) | Expr |
expr.sum/count/mean/min/max/is_null() | Expr |
expr.alias(name) | Expr |
frame.lazy(), scan_parquet(path) | Query |
query.filter(&predicate) | Query |
query.with_columns(&expressions), select(&expressions) | Query |
query.group_by(&keys, &aggregates) | Query |
query.join(&other, &left_keys, &right_keys, how=JoinKind.Inner, suffix="_right", nulls_equal=False) | Query |
query.join_asof(&other, "key", "key", strategy=AsOfStrategy.Backward, tolerance=Nothing, allow_exact_matches=True, suffix="_right") | Query |
query.sort(&keys), slice(offset, length), limit(count) | Query |
query.collect(), collect_streaming() | DataFrame |
scan_csv(path, ...) | Query |
frame.write_csv(path, ...), query.sink_parquet(path) | () |
frame.write_parquet(path) | () |
See expression contracts for conditional evaluation, explicit casts, string operations, distinct counts and partitioned windows.
Expressions describe native Polars operations; they are not opaque Plenty
callbacks. Polars can optimize a complete plan, push projections/filters
into Parquet scans, and use its worker pool. A scan is lazy; errors can
arise during construction or collection. with_columns replaces matching
names or adds new columns. Expressions in one call see the input schema;
chain calls when a later expression depends on a newly derived column.
Selection projects or aggregates. Grouping supports multiple keys and
aggregates, but does not promise group order. Add sort to the plan when
an application needs a particular order.
Owners and plans are immutable through this public API. Methods borrow their receivers; resulting plans clone native handles and share buffers. Dropping the original columns, frame or expressions does not invalidate a derived query. Each handle is freed by the engine that allocated it. There are no public raw pointers or user-created native handles.
Joins, ordering and row windows
For sorted integer keys, as-of joins attach the previous, next or nearest right row with an optional distance bound. Unlike equality joins, they produce at most one match for each left row.
These operations return new immutable query plans; neither input is consumed or mutated. They execute inside Polars, with no Plenty row callbacks or intermediate conversion into Plenty lists. Input owners, key expressions and sort specifications may be dropped after constructing the resulting plan.
JoinKind offers Inner, Left, Full, Semi and Anti equality joins.
Inner/left/full retain both sets of key columns; semi/anti are membership
queries returning only left columns. See equality join contracts
for duplicate/null matching, key retention, suffix collisions and validation.
SortKey(expression: Expr, descending: bool = False, nulls_last: bool = True)
owns its expression; constructing the wrapper itself is infallible.
query.sort(keys: &list[SortKey]) orders lexicographically, with the first
key most significant. Every key has independent direction and null placement.
Null placement is independent of direction: nulls_last=True puts nulls last
even in descending order. NaN follows Polars value ordering. Equal keys retain
their input order; that does not make ties from an unordered join/group
deterministic. Include an additional identifying key when that matters.
An empty sort list fails with Schema.
slice(offset: i64, length: i64) selects a row window.
The second argument is a count, unlike Plenty list slice(start, stop).
Negative offsets count from the end. The requested interval is intersected
with the available rows: slice(-2, 10) keeps the last two rows, while
slice(-100, 2) on a three-row frame is empty. Offsets beyond the end also
produce empty frames. length must be in 0..4294967295 inclusive;
negative or larger lengths fail immediately with DataFrameError.Bounds.
limit(count) is slice(0, count), with the same length checks.
Sort first when a page or top-N result needs a defined ordering.
See CSV and streaming I/O for file options, direct sinks, create-new output behavior and buffering limits.
Semantics and limits
Polars 0.55.2 semantics apply inside expressions, not Plenty scalar
semantics. Schemas are checked at runtime. Input/extraction supports
all Plenty numeric widths (i8/i16/i32/i64, u8/u16/u32/u64, f32/f64),
str, bool, Binary (std.bytes.Bytes), Date and naive/UTC Datetime,
including nullable columns.
Lists and structs have column-oriented construction and extraction,
native access expressions and structured schema inspection. Other unexposed native
types may pass through Parquet queries and appear in schema inspection, but cannot
be extracted through this API. Paths are UTF-8 strings. There is no automatic
lambda translation, mutation API, cloud credential configuration or stable
serialization of plans.
Null is missing data, distinct from floating-point NaN. Comparisons with
null yield null, which filtering drops. Polars orders NaN above finite
numbers: NaN > 50.0 may keep a row that a Plenty scalar comparison would
drop. Aggregations generally skip nulls; count counts non-null values and
returns i64. Floating-point sums may differ with execution order and
parallelism. Group order is unspecified. Native integer arithmetic and
reductions are not Plenty’s checked arithmetic: overflow must not be
assumed to return a recoverable error. Validate input ranges where needed.
DataFrameError retains native messages in Schema, Query, Io,
NativePanic, Null, and Bounds. Polars errors, including some file
errors, use Query; direct filesystem errors use Io.
Offset-table lengths and packed string byte totals are checked on the Plenty
side; an unrepresentable extent returns
DataFrameError.Allocation(AllocError.CapacityOverflow) before allocating the
affected table or advancing the total. This does not change the native engine’s
row limit or its allocation-failure policy. Copying a native
error message can itself fail with Allocation(AllocError) or
Message(IoError). Plenty-side list allocation is recoverable.
Native Rust/Polars allocation failure can abort the process.
Caught Rust panics become errors where unwinding is available; this does
not make arbitrary native failures recoverable.
POLARS_MAX_THREADS configures the engine’s global pool before first use.
It is independent of Plenty executors. Avoid oversubscribing both pools
when running many analytical queries concurrently.
Acknowledgements
- Wraps Polars 0.55.2, built into the optional dataframe engine library (MIT).
Dataframe column types and schemas
The optional dataframe engine supports these boundary
types. Construction and extraction return Result[..., DataFrameError];
names and dtypes are checked at runtime. No implicit cast is performed.
import std.dataframe as df
def main() -> Result[(), df.DataFrameError | AllocError | IoError]:
labels: list[Option[str]] = [Some("low"), Nothing]?
data = df.frame([
df.column_u8("level", [3u8, 200u8]?)?,
df.column_str_nullable("label", labels)?
]?)?
for field in data.schema()?:
print(field.name, field.dtype)?
print(data.values_u8("level")?)?
print(data.values_str_nullable("label")?)?
print(data.get_str("label", 1)?)?
Ok(())
level std.dataframe.DType.U8
label std.dataframe.DType.String
[3, 200]
[Option[str].Some("low"), Option[str].Nothing]
Option[str].Nothing
| Column dtype | Non-nullable constructor | Non-nullable extraction |
|---|---|---|
i8, i16, i32 | column_i8/i16/i32(name, values) | frame.values_i8/i16/i32(name) |
i64 | column_i64(name, values) | frame.values_i64(name) |
u8, u16, u32, u64 | column_u8/u16/u32/u64(name, values) | frame.values_u8/u16/u32/u64(name) |
f32 | column_f32(name, values) | frame.values_f32(name) |
f64 | column_f64(name, values) | frame.values_f64(name) |
str | column_str(name, values) | frame.values_str(name) |
bool | column_bool(name, values) | frame.values_bool(name) |
Binary (std.bytes.Bytes) | column_binary(name, values) | frame.values_binary(name) |
Logical Date and Datetime columns have separate checked constructors/getters with explicit epoch-day or timestamp-unit interchange.
Constructors borrow list[T]; extraction returns an owned list[T].
All scalar types also have column_TYPE_nullable(name, &list[Option[T]]) and
frame.values_TYPE_nullable(name) -> Result[list[Option[T]], DataFrameError].
Use these when nulls are meaningful. Non-nullable extraction fails with
DataFrameError.Null if any cell is null, even if the column was originally
constructed without nulls and a later join introduced them.
get_TYPE (for each numeric type, str, bool and binary) accepts a column name and
zero-based row index and returns Result[Option[T], DataFrameError].
A null cell is Ok(Nothing); an unknown name, wrong dtype or invalid index
is an error. Empty and all-null typed columns retain their declared dtype.
Native columns support at most 2**32 - 1 rows. String input currently
has the slightly lower limit 2**32 - 2: its offset buffer needs one
entry beyond the number of rows. Binary input has the same offset limit and
currently accepts at most 2**32 - 1 packed payload bytes per call.
Numeric input, extraction and Parquet round trips preserve signedness and
width. In particular, u64.MAX never passes through an i64 or a float.
Extraction requires the exact dtype: use an explicit expression cast when
conversion is intended. Native operations can change a result dtype (for
example, a reduction may widen a narrow integer); inspect the result schema
and choose its matching getter. Polars arithmetic and overflow semantics still
apply inside queries.
lit_i8/i16/i32/i64, lit_u8/u16/u32/u64 and lit_f32 create expressions
with those exact types; lit(value) remains the f64 literal constructor.
Text, booleans and missing values
Strings are UTF-8, preserving Unicode scalars and embedded NUL bytes.
There is no normalization or locale conversion. Empty text is Some(""),
not Nothing. Booleans remain booleans; they are not implicitly converted
to integers. Floating-point NaN and infinity remain present floating-point
values. In particular, Some(f64.NAN) does not become null.
Both inputs and outputs own their data independently of the other side. Constructing a column never retains pointers into a Plenty list. A query may outlive its original frame, and extracted lists may outlive the native frame or query. Native owners are released by their allocating engine.
Existing non-nullable numeric paths transfer a whole column in one native call. Nullable numeric and boolean paths pack scalar values and validity bytes, then transfer whole buffers. String input joins UTF-8 text and constructs byte offsets and validity; output takes an owned native snapshot, copies packed bytes/offsets/validity in bulk and decodes each byte range into an ordinary Plenty string. This is copying, with temporary buffers and additional per-string allocations, not Arrow zero-copy interchange. It avoids per-cell foreign calls, but conversion can dominate small queries. All these allocations remain subject to the native OOM limitation.
Binary payloads
Binary columns contain arbitrary byte sequences, including embedded NUL and
invalid UTF-8. Their Plenty values are owned std.bytes.Bytes
buffers. Some(empty_buffer) is a present zero-byte payload; Nothing is
missing data. Constructors borrow buffers and copy their contents. Extraction
returns independent mutable buffers; changing them cannot change the frame.
lit_binary(value: &[u8]) copies a constant payload into a native expression.
It accepts literals, buffers, byte lists and subviews without a Plenty-side
allocation; the native expression owns its copy after the call. Use it with
equality, filtering, or fill_null. cast(DType.Binary) explicitly
converts strings to UTF-8 bytes. Binary-to-string casts require valid UTF-8;
the pinned engine rejects invalid sequences even with strict=False.
Binary columns round-trip
through Parquet, but are not supported by the CSV interface.
Transfers pack bytes, offsets, and validity into whole-column native calls.
Construction passes the packed Bytes as a borrowed byte view, without a
second payload copy into list[u8]. Extraction copies through a packed
Bytes before allocating each result buffer.
These are copying APIs with temporary memory overhead, not zero-copy views;
select/filter in Polars before extracting large payloads.
See the binary-column lesson.
Inspect a schema before collecting
DataFrame.schema() and Query.schema() return
Result[list[Field], DataFrameError], in column order.
Each Field has public name: str and dtype: DType fields.
DType has I8, I16, I32, I64, U8, U16, U32, U64,
F32, F64, Bool, String, Binary, Date,
Datetime(TimeUnit, Option[str]), List, Struct and Other(str) variants.
Datetime metadata identifies its unit and optional timezone label; construction
and extraction currently support naive and UTC values only.
Other(description) preserves the engine’s native dtype description,
including unexposed numeric or fixed-size array types. It does not imply that
a type can be extracted or that the description is a stable serialization
format. Do not parse that text to manufacture a cast.
Query schema inspection resolves the native plan without collecting all rows. A Parquet scan may read file metadata, and invalid paths, missing columns or incompatible expressions may fail at this stage. It is not a promise that later collection will succeed: data and I/O failures can still occur. Inspection returns independent metadata and does not mutate or consume the query.
See the runnable lesson and the signature index.
Nested columns add List and Struct dtype variants,
column-oriented construction/extraction and structured schema_tree() inspection.
Nested dataframe columns
Lists and structs compose from existing native columns. They do not require a
dynamic Plenty object for every row. Constructors borrow their inputs; returned
columns, extracted parts and query plans remain usable after those inputs drop.
All operations below return Result[..., DataFrameError].
import std.dataframe as df
def main() -> Result[(), df.DataFrameError | AllocError | IoError]:
items = df.column_i64("item", [1, 2, 3, 4, 5]?)?
# Offsets split the items into rows [1, 2, 3], [] and [4, 5].
baskets = df.column_list("basket", items, [0, 3, 3, 5]?)?
report = baskets.to_frame()?.lazy()?.select([
df.col("basket")?.list_len()?.alias("count")?,
df.col("basket")?.list_get(0)?.alias("first")?
]?)?.collect()?
print(report.values_i64("count")?)?
print(report.values_i64_nullable("first")?)?
Ok(())
[3, 0, 2]
[Option[i64].Some(1), Option[i64].Nothing, Option[i64].Some(4)]
Construct lists and structs
column_list(name, values, offsets) groups a flattened child Column into
variable-length lists. For n rows, offsets contains n + 1 integers: it
starts at zero, never decreases, and ends at the child column length. Row i
contains child values in [offsets[i], offsets[i + 1]). Equal offsets make an
empty list. The child dtype, including nested types, is preserved even when
the child is empty or entirely null. Its column name is not part of the list
element schema.
column_list_nullable(name, values, offsets, valid) adds a list[bool] with
one entry per parent row. A false entry means a null list, not an empty list.
Null rows must have equal consecutive offsets; hidden children are rejected.
Child nulls are independent: a present list can contain null elements.
The non-nullable constructor only promises present parent lists.
The extra offset entry limits list construction to 2**32 - 2 parent rows;
the flattened child retains the engine’s 2**32 - 1 row limit.
column_struct(name, fields) takes a child DataFrame. Its column names,
order and dtypes become the struct fields, and its height becomes the parent
row count. At least one field is required. Zero-row typed structs work.
column_struct_nullable(name, fields, valid) takes one boolean per parent row.
Null parents mask their fields; values underneath null parents are not retained
as meaningful data. A present struct with all-null fields remains distinct
from a null struct.
Input lists and validity buffers are copied into native storage. Struct fields can share immutable native buffers. Constructors validate lengths and offsets; malformed input returns an error rather than indexing unchecked memory.
Extract column-oriented parts
frame.column(name) returns an independent Column handle sharing native
buffers. column.to_frame() wraps it in a single-column dataframe with its
existing name; use that frame’s ordinary typed extraction methods when values
must enter Plenty. column.len() returns its row count.
column.list_parts() returns ListParts with:
| Field | Meaning |
|---|---|
values: Column | Flattened child values, named item |
offsets: list[i64] | Canonical offsets starting at zero |
valid: list[bool] | Parent presence, independently of child nulls |
Null parent ranges are empty. Extraction may compact/rechunk native child
buffers; it is not a zero-copy promise. It transfers offsets and validity in
bulk, and leaves the child data native until explicitly extracted. The returned
parts can be passed back to column_list_nullable to reconstruct the column.
column.struct_parts() returns StructParts(fields: DataFrame, valid: list[bool]). The child dataframe retains field order and types; null parent
rows have null child fields. The mask is necessary to distinguish a null parent
from a present struct whose fields are all null. These parts can be passed to
column_struct_nullable. Either method rejects the wrong column dtype.
Query nested values
| Expression | Behavior |
|---|---|
expr.list_len() | i64 element count; null list stays null, empty list yields zero |
expr.list_get(index) | Element at a fixed i64 index; negative indexes count from the end; out of bounds yields null |
expr.struct_field(name) | Named child field; null parent yields null; missing field is an error |
These are native Polars expressions and compose with filtering, projection, aggregation and other expressions. An extracted element can itself be a list or struct. Operations validate their input schema during native plan resolution or collection; creating an expression does not prove that a later input fits it. Nested casts, list-element callbacks and row-exploding operations are not exposed by this initial interface.
Structured schema inspection
Existing schema() reports DType.List and DType.Struct for nested columns.
DataFrame.schema_tree() and Query.schema_tree() return a preorder
list[SchemaNode], where each node has parent: i64, name: str, and
dtype: DType. Root columns have parent -1; other parents index an earlier
node. Struct children retain field names/order; a list has one child named
item. This synthetic name cannot collide with a struct field because its
parent identifies its role.
The flat representation avoids recursive DType ownership: existing scalar
dtype values stay copyable, printable and comparable. It supports arbitrary
combinations of lists and structs without parsing DType.Other descriptions.
Other native dtypes remain Other leaves. Schema inspection does not collect
rows, although a file scan can read metadata and fail.
Parquet compatibility
Lists and structs preserve their semantic types, values and parent/child nulls through the pinned Polars Parquet reader/writer. Physical encoding and unused values hidden beneath null parents need not survive. Fixed-size arrays, arbitrary map semantics and other unexposed dtypes are not promised here.
Polars 0.55.2 has no native Map dtype. Its Parquet schema conversion reads maps
as list/struct data; its Arrow metadata conversion explicitly maps Map to a
large list of key/value structs (polars-parquet,
src/arrow/read/schema/{convert,metadata}.rs). Use the schema tree and ordinary
list/struct operations for that representation. No dictionary uniqueness,
key ordering, or preservation of the Parquet MAP annotation on write is
promised. This is a supported Polars representation, not a new Plenty Map API.
See the runnable lesson and column types. The existing native allocation-failure and 32-bit row-count limits also apply to flattened child columns.
Dataframe expressions
These operations extend the native expression plans.
Every function and method below returns Result[Expr, DataFrameError].
They build immutable Polars expressions, not per-row Plenty callbacks.
Receivers and expression arguments are borrowed; returned plans remain valid
after those inputs are dropped. Invalid column names and incompatible dtypes
can fail during schema resolution or collection.
import std.dataframe as df
def main() -> Result[(), df.DataFrameError | AllocError | IoError]:
data = df.frame([
df.column_str("name", ["Ada", "lin"]?)?,
df.column_i64("score", [92, 61]?)?
]?)?
result = df.when(
df.col("score")?.ge(df.lit_i64(70)?)?,
df.lit_str("pass")?,
df.lit_str("retry")?
)?.alias("result")?
report = data.lazy()?.with_columns([df.col("name")?.str_upper()?, result]?)?.collect()?
print(report.values_str("name")?)?
print(report.values_str("result")?)?
Ok(())
["ADA", "LIN"]
["pass", "retry"]
Literals and conditions
lit_str(value: &str) copies a UTF-8 literal into the native plan;
lit_bool(value: bool) creates a boolean literal. Alongside lit(f64)
and lit_i64(i64), these distinguish literal values from col(name).
lit_binary(value: &[u8]) copies an arbitrary byte payload into
the native plan without decoding it as text. Pass a b"..." literal,
&buffer, &buffer[a:b], or &list_of_u8. The view reaches the native
copy directly; the Plenty facade allocates no temporary payload.
when(predicate, then_value, otherwise) accepts three expressions.
For each row, a true boolean predicate selects then_value; false or null
selects otherwise. Polars determines a common output dtype for the two
branches, so branches can undergo native coercion. An incompatible predicate
or branch type produces a query error.
This is not short-circuit evaluation. Both branches must be valid
independently: a condition cannot guard an invalid cast, missing column or
other failing expression. Use an appropriate non-strict cast or validate the
data separately. Nested when expressions can represent several choices,
but this API does not translate Plenty if statements automatically.
Explicit casts
expr.cast(dtype: DType, strict: bool = True) supports
all numeric DType variants (I8/I16/I32/I64, U8/U16/U32/U64, F32/F64),
Bool, String and Binary.
Temporal targets and DType.Other(...) fail immediately with DataFrameError.Schema;
native dtype descriptions are not executable cast specifications.
Use the explicit temporal parsing and calendar methods
to construct dates/datetimes in a query.
The default strict mode reports invalid conversions as a query error.
With strict=False, failed value conversions become null when Polars
supports that conversion. It does not make every source/target pair legal;
an unsupported conversion can still fail. Existing nulls remain null.
Strict does not mean lossless. For example, casting a finite fractional
float to an integer truncates toward zero, and integer-to-float conversion
can lose precision. Numeric-to-boolean converts zero to false and nonzero
values to true. String parsing and boolean formatting follow Polars, not
Plenty’s scalar parsing/formatting rules. Use the typed nullable getters
when a non-strict cast can introduce missing values.
String-to-binary casts preserve the UTF-8 bytes. Binary-to-string casts validate
UTF-8: the pinned engine rejects invalid sequences even with strict=False.
Extraction itself never performs these conversions.
String expressions
| Method | Result dtype | Meaning |
|---|---|---|
str_lower(), str_upper() | String | Unicode lowercase or uppercase |
str_len() | I64 | Number of Unicode scalar values |
str_contains(pattern: &Expr) | Bool | Contains a literal substring |
str_starts_with(prefix: &Expr) | Bool | Starts with the literal text |
str_ends_with(suffix: &Expr) | Bool | Ends with the literal text |
Use df.lit_str("text")? for a constant pattern, or df.col("pattern")?
for row-wise patterns. Null text or a null pattern produces null. A null
predicate is dropped by filter. The receiver must be a string expression.
Use string patterns for predictable text semantics: Polars can coerce a
primitive pattern to text, but does not similarly accept a numeric receiver.
This native expression coercion does not change the strict
dtype checks on Plenty’s column construction and extraction boundary.
Contains is literal: ".", "*" and "[" have no regex meaning.
The engine enables Polars’ regex crate feature because its literal
contains implementation is behind that feature; no regex API is exposed here.
Character counts are neither UTF-8 byte counts nor grapheme counts.
Combining marks count separately. Casing can change the number of scalars
(for example uppercase ß becomes SS); no normalization or
locale-specific collation is promised. Embedded NUL bytes remain text.
Distinct counts and partition windows
expr.n_unique() computes an exact distinct count and returns I64.
Unlike count(), it counts null as one distinct value. Empty input has
zero distinct values; all-null nonempty input has one. It can be used in
select, group_by aggregates or a window expression.
expr.over(partition_by: &list[Expr]) builds a native group-to-rows window.
For example, df.col("amount")?.sum()?.over([df.col("team")?]?)?
computes each team’s total and broadcasts it back to that team’s rows,
without collapsing the frame or converting it into Plenty objects.
Multiple key expressions form composite partitions; null keys group together.
The key list must be nonempty. Empty keys fail immediately with Schema;
missing keys and invalid expressions fail when resolving or executing.
Scalar aggregates map to every row in the partition. Non-scalar expressions
must satisfy Polars’ group-to-rows length requirements. Output maps back to
the input row positions; it does not introduce a sort order.
This increment has no window ordering, frame bounds, rolling operations,
cumulative sums or ranking methods. Sorting a query does not turn sum().over
into a running sum: it still sees the entire partition. Floating reductions
retain the existing native rounding/order caveats.
See the runnable lesson and exact signatures.
Dataframe equality joins
Query.join(other, left_keys, right_keys, how=JoinKind.Inner, suffix="_right", nulls_equal=False) returns a new immutable query.
It borrows both plans and expression lists; the resulting query remains
valid after those owners are dropped.
Key lists must be nonempty and equally long. Multiple expressions form
composite keys, matching every component. Key dtypes must be compatible;
there is no implicit Plenty-side cast. Empty/unequal lists fail immediately
with DataFrameError.Schema; missing columns, incompatible dtypes and
other native plan errors can fail during schema resolution or collection.
| Kind | Rows kept | Output columns |
|---|---|---|
Inner | Every matching left/right pair | Both sides |
Left | Matching pairs, plus each unmatched left row | Both sides; unmatched right cells are null |
Full | Matching pairs, plus each unmatched row from either side | Both sides; absent-side cells are null |
Semi | Each left row with at least one right match | Left side only |
Anti | Each left row with no right match | Left side only |
import std.dataframe as df
def main() -> Result[(), df.DataFrameError | AllocError | IoError]:
orders = df.frame([df.column_i64("product", [3, 1, 2, 1]?)?, df.column_i64("units", [5, 2, 1, 3]?)?]?)?
prices = df.frame([df.column_i64("product", [1, 2]?)?, df.column_f64("price", [10.0, 25.0]?)?]?)?
keys = [df.col("product")?]?
order = [df.SortKey(df.col("product")?), df.SortKey(df.col("units")?)]?
priced = orders.lazy()?.join(prices.lazy()?, keys, keys)?.sort(order)?.collect()?
print(priced.values_i64("product")?, priced.values_f64("price")?)?
unpriced = orders.lazy()?.join(prices.lazy()?, keys, keys, how=df.JoinKind.Anti)?.collect()?
print(unpriced.values_i64("product")?)?
Ok(())
[1, 1, 2] [10.0, 10.0, 25.0]
[3]
Multiplicity and validation
Inner, left and full joins are many-to-many. Two equal left keys and three equal right keys yield six matching pairs. No uniqueness check is performed; these are not validated one-to-one or many-to-one lookups.
Semi and anti joins test membership. Duplicate right matches do not multiply the output. Duplicate left rows remain distinct input rows and are retained or removed independently. For example, two matching left rows survive a semi join as two rows even when the right side contains three copies of the key. These joins are not deduplication operations.
An empty left input produces no semi/anti rows. An empty right input makes a semi join empty and an anti join retain all left rows. A full join with one empty side retains the other side, padding the absent columns with nulls.
Nulls and order
Null keys do not match by default. Thus a null left key is excluded by semi,
retained by anti, and remains unmatched in a full join.
With nulls_equal=True, nulls in corresponding key positions compare equal,
including inside composite keys. Pair-producing joins still multiply
duplicate null-key matches; membership joins still retain each left row at
most once.
NaN remains a native floating-point value governed by Polars equality, not a null. No join kind promises output order. Apply explicit sorting when an application needs a reproducible presentation, including a tie-breaker when keys alone are insufficient.
Keys, names and suffixes
Inner, left and full joins retain both sides’ key columns without automatic
coalescing. A right column whose name is also present on the left gets the
suffix, including keys. If this still leaves duplicate output names, the query
fails with DataFrameError.Query; it does not overwrite a column.
For an unmatched right row in a full join, the left key is null while the
right key keeps its value. To create a common display key, derive it explicitly:
df.col("id")?.fill_null(df.col("id_right")?)?.alias("joined_id")?.
Keep the originals when their provenance matters. This is a normal expression
step, not a different join coalescing policy.
Semi and anti joins emit only left columns, so they neither append right keys nor suffix right names. The suffix argument has no effect for them, including an empty suffix. Right-side name collisions are therefore harmless for these membership joins.
The same expression-key and ownership rules apply to all five kinds. Native query execution supports ordinary and explicit streaming collection; streaming is not a promise of bounded join memory. For nearby-key matching, use integer as-of joins. Temporal/grouped as-of joins and uniqueness-validation options are not exposed.
See membership and full-join examples, joining and ranking, and the signature index.
Integer as-of joins
Query.join_asof(other, left_on, right_on, strategy=AsOfStrategy.Backward, tolerance=Nothing, allow_exact_matches=True, suffix="_right") matches each
left row to at most one nearby right key. It returns a new immutable native
query and borrows both inputs. It remains usable after their owners are dropped.
left_on and right_on are column names. Both columns must have dtype I64;
there is no implicit cast. Derive a named key with with_columns when needed.
Floating-point, date/time, grouped and composite as-of keys are not supported.
Use equality joins for exact multi-column matching.
import std.dataframe as df
def main() -> Result[(), df.DataFrameError | AllocError | IoError]:
trades = df.frame([df.column_i64("second", [3, 7]?)?]?)?
quotes = df.frame([
df.column_i64("second", [1, 5, 6]?)?,
df.column_f64("bid", [9.5, 10.0, 10.5]?)?
]?)?
result = trades.lazy()?.join_asof(quotes.lazy()?, "second", "second")?.collect()?
print(result.values_f64("bid")?)?
Ok(())
[9.5, 10.5]
Ordering and matching
Both inputs must be sorted ascending by their key, with nulls first. Sorting
is not automatic: use sort([df.SortKey(df.col("key")?, nulls_last=False)]?)
when the source is not already ordered.
The native execution checks sortedness and rejects an unsorted input with
DataFrameError.Query. The requirement applies to the inputs of the join,
after any preceding filters or key transformations.
| Strategy | Right key chosen |
|---|---|
Backward | Greatest key less than or equal to the left key |
Forward | Smallest key greater than or equal to the left key |
Nearest | Key with the smallest absolute integer distance |
Set allow_exact_matches=False to exclude equal keys. tolerance=Some(n)
adds an inclusive distance bound of n key units; Nothing means no bound.
The bound must be nonnegative and fit in i64. Some(0) permits only exact
matches, so disabling exact matches with zero tolerance yields no matches.
Integer distance comparisons handle the full signed i64 range without
wrapping at its endpoints.
Backward selects the last right row at the chosen key; forward selects the first. Nearest breaks equal-distance ties toward the greater key and selects the last duplicate at its chosen key. These rules refer to right input order: when duplicate selection matters, explicitly sort by the key and a suitable secondary column before joining.
Null left keys are unmatched; null right keys are skipped. Null keys never match one another. Nulls-first placement is validated consistently, including for inputs whose native metadata already marks them sorted. Every left row survives once, including duplicates and unmatched rows. An empty right input supplies null right fields; an empty left input yields no rows.
Output, failures and costs
Both key columns remain in the output. Right column names that collide with left names receive the suffix. An unresolved collision is a query error; columns are not silently overwritten. Unmatched right-side cells are null, so use nullable extraction unless matching is guaranteed.
Negative tolerances and unsupported key dtypes fail with
DataFrameError.Schema. Missing columns, invalid native plans, suffix
collisions and unsorted data are errors rather than unmatched rows. Some
errors arise while the plan is built; data-dependent failures arise when
it is collected or written. Native allocation/panic limits are the same as
for other dataframe operations.
The join remains a Polars query-plan node; no Plenty per-row callback or
intermediate Plenty collection is used. A native lazy validation step before
each input scans its entire key column at execution time and returns the frame
unchanged. These steps materialize both inputs and block optimizations from
moving operations across validation. Put intended filters, projections and
sorting before join_asof to reduce that work.
The validation steps compensate for a pinned Polars streaming path that can accept an unsorted right input despite its sortedness-check option. Ordinary and explicit streaming collection therefore have the same checked contract, but as-of joins are not bounded-memory streaming operations. Plan creation resolves schemas and does not eagerly collect the datasets.
Rebuild the optional engine to obtain its
asof_join feature. Ordinary programs still do not link Polars.
See the sensor lookup example and signature index.
Dates and datetimes
Temporal columns have logical types, not merely integer labels. Their
construction and extraction use explicit physical units at the Plenty boundary;
native parsing and calendar expressions handle ordinary analytical workflows.
All operations return Result[..., DataFrameError].
import std.dataframe as df
def main() -> Result[(), df.DataFrameError | AllocError | IoError]:
source = df.frame([df.column_str("text", ["2024-02-29", "1970-01-02"]?)?]?)?
day = df.col("day")?
result = source.lazy()?.select([
df.col("text")?.str_to_date()?.alias("day")?
]?)?.with_columns([
day.dt_year()?.alias("year")?,
day.dt_format("%d/%m")?.alias("label")?
]?)?.collect()?
print(result.values_date_days("day")?)?
print(result.values_i64("year")?)?
print(result.values_str("label")?)?
Ok(())
[19782, 1]
[2024, 1970]
["29/02", "02/01"]
Units, timezones and the boundary
| Logical type | Construction | Extraction |
|---|---|---|
| Date | column_date(name, days: &list[i32]) | values_date_days(name) / get_date_days(name, row) |
| Datetime | column_datetime(name, ticks: &list[i64], unit, utc=False) | values_datetime_ticks(name, unit, utc=False) / get_datetime_ticks(name, row, unit, utc=False) |
Dates count proleptic Gregorian calendar days since 1970-01-01. Datetimes count
ticks from 1970-01-01 00:00:00. TimeUnit.Milliseconds, Microseconds and
Nanoseconds mean 1,000, 1,000,000 and 1,000,000,000 ticks per second.
Negative values precede the epoch. No unit is inferred or converted by a getter.
Date extraction returns i32; datetime extraction returns i64.
utc=False describes a timezone-free civil datetime. utc=True describes
UTC; it does not convert a local wall-clock value into UTC. No host timezone
is consulted. Named-zone localization, timezone conversion, daylight-saving
ambiguities and leap seconds are not supported in this foundation.
std.time.Timestamp uses seconds plus
nanoseconds and is not implicitly converted to these column representations.
Each constructor also has a _nullable form accepting list[Option[T]].
Each bulk getter has a _nullable form returning list[Option[T]]; the plain
getter rejects any null with DataFrameError.Null. Scalar getters return
Option[T] inside Result. Empty and all-null columns retain their dtype.
Transfers copy whole buffers; nullable values use a separate validity buffer.
Construction and typed extraction validate the engine’s representable calendar
range. Not every i32 day or i64 millisecond/microsecond value is a valid
calendar date. The full i64 nanosecond range is representable. Invalid values
produce errors rather than wrapping; null payload bits are ignored.
Wrong dtype, unit or UTC metadata is an error, not an implicit conversion.
Schemas report DType.Date or DType.Datetime(unit, timezone: Option[str]).
Nothing is naive and Some("UTC") is UTC. Other timezone labels from files
are preserved honestly, but the typed temporal getters and calendar operations
reject them. Schema inspection is not a promise that every reported type or
value can be extracted. Generic get_i64 does not discard datetime metadata.
Parse strings and extract calendar fields
| Expression method | Output |
|---|---|
str_to_date(format="%Y-%m-%d", strict=True) | Date |
str_to_datetime(format, unit, utc=False, strict=True) | Datetime with the specified unit/zone |
dt_year(), dt_month(), dt_day() | I64 calendar component |
dt_hour(), dt_minute(), dt_second() | I64 time component of a Datetime |
dt_date() | Date portion of a Datetime |
dt_format(format) | String |
Formats use Chrono directives such as %Y-%m-%d, %H:%M:%S and %.f
(optional fractional seconds). Parsing consumes the whole string. Omitted
clock fields default to zero. Invalid values and out-of-range conversions
fail with Query in strict mode or become null with strict=False.
Existing nulls remain null. Parsing leap-second :60 is rejected.
Fractional precision beyond the selected unit is discarded toward the earlier
instant: -1 nanosecond becomes -1 millisecond, not zero. Choose nanoseconds
when that precision matters; their calendar range is much smaller than that
of milliseconds or microseconds. Nanosecond overflow is checked explicitly.
Invalid formats and timezone directives (%z, %Z, %+, including modified
forms) are rejected. A literal Z is just text; use it only when the data is
explicitly UTC. Date-only values cannot supply time-of-day formatting fields.
Unicode literals in formats are preserved. Calendar components use the stored
naive/UTC interpretation and return null for null inputs.
Parsing and range checks execute as checked native column expressions inside Polars plans, with no Plenty callback or per-cell FFI. These validation/parsing nodes are opaque to the optimizer; calendar kernels remain native Polars operations. The checked parser avoids overflow and leap-second normalization in the pinned native parser. Imported invalid calendar values fail explicitly before calendar extraction or formatting.
Parquet preserves temporal dtypes and metadata. CSV does not: format columns
explicitly before writing and parse the resulting strings after reading.
CSV schema overrides and Expr.cast accept numeric, boolean and string target
types; they do not construct temporal types. As-of joins remain integer-key
operations. See the runnable lesson
and signature index.
Acknowledgements
- Date parsing, calendar validation and format directives use Chrono 0.4.45, built into the dataframe engine (MIT OR Apache-2.0).
CSV and streaming dataframe I/O
These operations use the optional Polars engine.
They return Result[..., DataFrameError], retain native error messages,
and leave query/frame owners usable after each call.
import std.dataframe as df
def main() -> Result[(), df.DataFrameError | AllocError | IoError]:
data = df.frame([
df.column_str("city", ["Oslo", "Lima", "Pune"]?)?,
df.column_i64("population", [709, 10883, 7764]?)?
]?)?
data.write_csv("cities.csv")?
query = df.scan_csv("cities.csv", schema_overrides=Some(data.schema()?))?
large = query.filter(df.col("population")?.gt(df.lit_i64(1000)?)?)?.collect()?
print(large.values_str("city")?)?
Ok(())
["Lima", "Pune"]
Scan a local CSV file
scan_csv(path, has_header=True, separator=",", infer_rows=100, null_value=Nothing, schema_overrides=Nothing) returns a Query.
The path names one file; wildcard characters are literal, not a glob.
| Option | Contract |
|---|---|
has_header: bool | First record supplies names; false uses column_1, column_2, … |
separator: &str | One ASCII byte, excluding NUL, double quote, CR and LF; tab is allowed |
infer_rows: i64 | Positive: sample that many rows; zero: infer strings; -1: inspect the whole file |
null_value: Option[str] | One optional exact token treated as null in every column |
schema_overrides: Option[list[Field]] | Named overrides using any numeric DType, Bool or String; other columns retain inference |
Overrides reuse schema fields. The optional list is
consumed, so schema_overrides=Some(frame.schema()?) is convenient for
CSV round-trips that must preserve these scalar override types. Duplicate or
unknown override names, temporal targets and DType.Other fail with Schema.
For temporal columns, explicitly format to strings
before CSV output and parse those strings after input; Parquet preserves their metadata.
Overrides select types by name; they are not a full schema or a column
rename/reorder facility. Headerless empty input has no width/names to infer.
CSV carries text, not a durable typed schema. Quoting does not guarantee
that an inferred column stays a string; specify a String override for
identifiers such as "001". A sample can infer an integer and then fail
on a later non-integer value. Use overrides, increase the sample, or read
strings and cast explicitly.
Whole-file inference costs a full scan before query execution.
Input must be valid UTF-8. Double quotes delimit quoted fields and doubled quotes escape quote characters. Quoted separators and newlines are supported. Malformed records, invalid typed values and invalid UTF-8 produce errors; the facade does not enable lossy decoding, ignored parse errors or ragged-row truncation. A scan may inspect schema/sample data immediately; errors can occur during creation, schema inspection or later execution. The file must remain available while the query executes.
Empty strings and null tokens
By default, an unquoted empty field is null; a quoted empty string field
is present empty text. This preserves Nothing versus Some("") in
typed string round-trips. Floating NaN remains a present floating value.
A custom null token also matches quoted fields: with null_value=Some("NA"),
both NA and "NA" become null. Choose a token that does not occur as a
real value. CSV quoting cannot disambiguate this collision. Empty/missing
fields still follow the native missing-is-null policy.
Write a collected frame as CSV
frame.write_csv(path, has_header=True, separator=",", null_value="")
returns Result[(), DataFrameError]. It writes a new local UTF-8 file,
with double-quote escaping and LF record endings. This starts from an
already collected dataframe; it is not a query sink.
The output null token is written verbatim. It cannot contain the separator,
double quote, CR or LF; invalid options fail before creating a file.
The empty default writes null as an unquoted empty field and preserves
present empty strings through quoting. Read a custom token back by passing
the same token to scan_csv. CSV does not preserve types by itself.
Execute with the streaming engine
query.collect_streaming() returns a DataFrame and explicitly selects
Polars’ streaming engine. It still materializes the entire final result.
It is useful when streaming intermediate work helps, not when the final
result itself cannot fit in memory. Ordinary collect() retains its native
automatic execution policy.
query.sink_parquet(path) executes a native Parquet file-sink plan using
the streaming engine and returns (). The wrapper does not collect a
complete dataframe and then call a writer. The opened output file is passed
directly to Polars, and intermediate rows remain inside the native plan.
For example, a CSV scan can filter/project and write Parquet without
converting rows into Plenty values or retaining the whole output frame.
Default native Parquet compression and row-group settings apply.
Neither method guarantees constant or bounded memory for arbitrary plans. Sorts, joins, high-cardinality groups, distinct counts and partition windows can buffer substantial data, and the streaming planner can use in-memory stages for some operations. A lazy frame built from Plenty columns already holds its source data in memory. Unsupported operations or resource failures can still error, and native OOM can abort. No batch iterator, memory budget, spill-directory control or cloud configuration is exposed by this increment.
The sink preserves the plan’s order where defined; it does not make unordered group/join output deterministic. Sort explicitly when output order matters, accepting the associated buffering cost.
Output ownership and failures
The new write_csv and sink_parquet methods create a new file.
An existing path, including a symlink or an alias of a lazy input, fails with
Io without truncation. Parent directories must already exist.
The sink keeps the opened file descriptor through execution instead of
reopening its path. Neither method mutates or consumes the input owner.
After creation, an I/O or query error may leave a partial new file. These
methods do not provide atomic publication, automatic cleanup or an fsync
durability guarantee. Choose a fresh output name; explicitly manage cleanup
or publication in the application when necessary.
The older DataFrame.write_parquet retains its documented overwrite behavior.
Native Polars failures use DataFrameError.Query; direct file creation
failures use Io, and invalid facade options use Schema. Other native
failure and allocation contracts are unchanged.
See the runnable lesson.
For row-oriented parsing over Plenty streams without Polars, use
std.csv.
Install the dataframe engine
The ordinary compiler build does not compile Polars. Importing
std.dataframe can be type-checked without its engine. Linking needs the
engine only when retained native code reaches a dataframe operation,
including calls through dependencies. An unused import or unused local
helper does not add a Polars dynamic dependency.
Use the release engine
Linux release archives include
bin/libplenty_dataframe_v1.so and its bin/dataframe-notices/. The compiler
finds the library beside itself, so no separate installation is needed. Set
PLENTY_DATAFRAME_ENGINE only to select a different library. The notices
directory holds engine.json, license files and dependency notices for
redistribution. The manifest records the Plenty version, source commit, target, ABI generation,
library checksum and minimum glibc. It is descriptive metadata, not a loader
configuration or a compiler-enforced version check.
The prebuilt engine supports x86-64 and ARM64 GNU/Linux with glibc 2.28 or later
and libgcc_s.so.1, with no AVX2 requirement. Rust, Cargo and a source checkout
are unnecessary. The experimental Windows x86-64 release also includes
bin/plenty_dataframe_v1.dll and bin/dataframe-notices/, requiring Windows 10
or Windows Server 2016 or later and the MSVC runtime. Its manifest records
minimum_windows instead of minimum_glibc. The experimental macOS ARM64
release includes bin/libplenty_dataframe_v1.dylib and the same notices layout,
requiring macOS 11 or later. Its manifest records minimum_macos.
musl/Alpine and other native targets remain unsupported.
The compiler still needs its linking prerequisites.
See shipping dataframe applications for runtime
library placement, notices and compatibility checks.
Build from source
For the library plus local --dist metadata/licenses, use just engine dataframe.
just build builds the compiler and both engines; see the
local build recipes.
The lower-level command below installs only the library.
From a Plenty source checkout, with a stable Rust toolchain:
python3 scripts/build_dataframe_engine.py --out target/release
Build the compiler separately with cargo build --release --locked.
The script pins Polars 0.55.2 with lazy, parquet, csv, strings, regex,
temporal, dtype-date, dtype-datetime, semi_anti_join and asof_join,
uses three Cargo jobs by default, and installs
libplenty_dataframe_v1.so. Use
--target-dir PATH to reuse engine build artifacts and --jobs N to
control build concurrency. It forces baseline x86-64 code generation
and the system C linker, overriding ambient CPU flags; AVX2 is not required.
The script supports native x86-64/ARM64 GNU/Linux builds, Linux cross targets,
and native x86_64-pc-windows-msvc and aarch64-apple-darwin builds. On Windows it uses MSVC, installs
plenty_dataframe_v1.dll, and avoids ELF linker flags. On macOS it builds for
Apple M1 with MACOSX_DEPLOYMENT_TARGET=11.0, installs
libplenty_dataframe_v1.dylib with identity
@rpath/libplenty_dataframe_v1.dylib, and uses the native C linker.
Build on a system with a compatible glibc version for the deployment host.
This source-build command does not impose the release’s glibc 2.28 baseline;
the release workflow builds and audits the engine inside its pinned container.
The extra string features support the native expression API. Polars 0.55.2’s
string dispatch imports its optional time module even without string-to-date
operations, so this feature combination also needs temporal to compile.
The date/datetime dtype features support temporal columns and expressions.
Checked parsing and calendar validation also use the pinned Chrono dependency.
Ordinary compiler/runtime builds still do not build Polars.
The parquet feature already enables Polars’ streaming engine; CSV support
adds only csv, not a second execution engine. The semi_anti_join feature
supports native membership joins; asof_join supports checked integer-key
as-of matching. Temporal and grouped as-of APIs are not exposed.
Discovery and compatibility
The dataframe engine follows the general native engine contract.
The compiler looks beside its own executable, or at the file named by
PLENTY_DATAFRAME_ENGINE. An explicit environment path takes precedence.
The Rust compilation API also offers CompileOptions.dataframe_engine.
Missing engines produce an installation hint at link time.
export PLENTY_DATAFRAME_ENGINE=/opt/plenty/libplenty_dataframe_v1.so
plenty --compile analysis.plenty -o analysis
The resulting executable contains a versioned shared-library dependency,
not a copy of the entire engine. For shipping, prefer
distribution output: --compile analysis.plenty --dist my-app
copies the selected engine and its packaged notices, with only $ORIGIN lookup.
The ordinary -o output above has RUNPATH searching its own directory
and the engine’s build-time directory. Deploy the executable beside the
matching libplenty_dataframe_v1.so, or install that library in the
loader’s search path. Keep compiler bindings and engine from the same
Plenty release; the v1 name denotes the C ABI generation, not compatibility
with arbitrary future Polars builds. Directly emitted objects also require
this engine when externally linked. Engine directories containing a colon
are rejected because ELF RUNPATH uses that character as a separator.
Rebuild/reinstall the engine when updating the compiler’s dataframe API.
New query operations add version-1 symbols; an older engine cannot satisfy
their native links even though its filename is unchanged.
The system loader’s LD_LIBRARY_PATH and LD_PRELOAD can override RUNPATH;
keep stale engine copies out of those paths when switching installations.
Normal programs need neither this library nor Rust. They do not gain a
DT_NEEDED entry for Polars. A project fingerprint includes the selected
engine path and metadata, so replacing or relocating it triggers relinking.
Already-linked executables still need their dynamic dependencies at run time.
macOS application linking
The macOS compiler uses the same option/environment/beside-compiler discovery
precedence and links the selected ARM64 dylib directly. Ordinary compilation
copies it beside the executable, as described in
engine copies beside applications, which
preserves old file mappings and cached signatures. Invalid selected engines report the build
script needed to produce a compatible dylib. --dist adds the validated manifest, licenses
and notices, and checks the copied dylib identity and checksum. The application
uses @loader_path to resolve the library’s @rpath identity from its own
directory, without a fallback to the original build directory.
Release packaging checks ARM64 Mach-O headers, a minimum macOS version no newer than 11, system-only dependencies, the versioned dylib identity, facade exports and the code signature. Apple tools verify the linker-generated ad-hoc signature; the release remains unnotarized. Keep the dylib unmodified and beside the executable when moving the application. Native CI tests Parquet queries and application-only relocation.
Windows application linking
The Windows compiler selects the DLL through the same option/environment/beside-
compiler precedence. It derives an MSVC import library from that DLL’s exported
C symbols using lib.exe, supplied by Visual Studio Build Tools. Ordinary
compilation copies the DLL beside the output executable; --dist includes the
DLL and validated notices. Programs use Windows DLL lookup, with the application
directory searched by the loader. Keep the DLL beside the executable, and use
trusted application directories and PATH entries. Type-checking needs no DLL.
Engine copies beside applications
On Windows and macOS, ordinary compilation links the selected engine and, after
the link succeeds, places a copy beside the executable. A failed link leaves an
existing application and its engine unchanged. When the existing copy already
has the same contents, it is not rewritten. Otherwise the new engine is copied
to a fresh file in the output directory and renamed over the old one, so
programs that have the old engine open keep it. Windows refuses to replace a DLL
that a running program has loaded; the error names the file, and closing the
program and compiling again replaces it. plenty FILE runs the program from a
temporary directory and hard-links the engine there when the file system
allows, instead of copying it.
Development checks
To exercise the optional component after building it:
export PLENTY_DATAFRAME_ENGINE="$PWD/target/release/libplenty_dataframe_v1.so"
export PLENTY_DATAFRAME_TESTS=1
cargo test --locked --test integration test_std_dataframe::
cargo test --locked --test integration test_tutorial::dataframe_examples_with_optional_engine
Regular tutorial checks type-check dataframe examples without the engine. The separate dataframe CI job builds it and executes those examples and the real-engine integration tests. These tests cover bulk transfer, query plans, nulls, errors and Parquet, as well as absence of the dynamic dependency from programs that do not use it.
Ship a dataframe application
A compiled program that uses std.dataframe needs
libplenty_dataframe_v1.so on Linux, libplenty_dataframe_v1.dylib on macOS,
or plenty_dataframe_v1.dll on Windows at run time. Ship the same engine you compiled
against, from the matching Plenty release, beside your executable. End
users need the host C libraries, but no Plenty compiler, Rust, Cargo or cc.
Programs without reachable dataframe operations need no engine.
Bundle the executable and engine
With a release compiler, the engine and its notices are
already in bin/:
plenty --compile analysis.plenty --dist my-app
./my-app/analysis
my-app must not exist. Plenty stages and publishes the executable, the exact
engine used for linking, and its packaged notices together. No reachable engine
calls means no engine files; reachable linear algebra calls also add the linalg
engine and its notices. See application distributions
for metadata checks, supported input layouts, local-build packaging and CLI
restrictions. A bare installed library without notices is not enough for --dist.
The resulting directory can be archived, installed elsewhere and renamed:
my-app/
analysis
libplenty_dataframe_v1.so
dataframe-notices/
engine.json
LICENSE-MIT
LICENSE-APACHE
THIRD-PARTY-NOTICES.md
licenses/rust/...
Keep the engine notices with distributions of the library, alongside any notices your own application needs. The release archive’s README contains the same distribution command for users who have no source checkout.
Test on the oldest supported system
The prebuilt engine targets x86-64 and ARM64 Linux, glibc 2.28 or later and
libgcc_s.so.1. The experimental Windows x86-64 engine requires Windows 10 or Windows Server
2016 or later and the MSVC runtime. The experimental macOS ARM64 engine
requires macOS 11 or later. musl/Alpine remains unsupported.
Your executable links against the build machine’s C libraries and can require
a newer glibc. Compile and test the application on the oldest distribution
you support; a baseline-compatible engine alone does not make your executable
portable. No particular memory bound is implied by the shared-library format.
Test the bundle in a clean machine or container without your original Plenty installation. Run it from a different working directory, then move the bundle and run it again. Test a missing-library failure too. The release smoke tests exercise app-only relocation and a native Parquet query on each supported test distribution after removing the compiler and original engine location.
How library lookup works
This applies to every native engine.
On macOS ARM64, --dist produces analysis,
libplenty_dataframe_v1.dylib and dataframe-notices/. Ordinary compilation
also copies the dylib beside the executable. The dylib identifies itself as
@rpath/libplenty_dataframe_v1.dylib; the executable searches @loader_path
(its own directory). No original build directory is retained. Keep the
linker-signed dylib unmodified. Loader environment variables such as
DYLD_LIBRARY_PATH can affect lookup; remove them when testing relocation.
On Windows, --dist produces analysis.exe, plenty_dataframe_v1.dll and
dataframe-notices/. Ordinary compilation also copies the DLL beside its
executable. Windows loader search rules apply; the application directory
supplies the bundled DLL. Test with the compiler directory removed from PATH.
On Linux, distribution output embeds an ELF RUNPATH of only the executable’s own
directory ($ORIGIN), so a missing bundled library cannot fall back to the
original engine directory. A sibling lib/ is not searched automatically.
For manual deployment with ordinary --compile analysis.plenty -o analysis,
copy the exact selected library beside the executable and all the archive’s
notices into dataframe-notices/, as in the layout above. Ordinary output still
searches $ORIGIN then the build-time engine directory; that fallback can
mask a missing bundled library on the development machine. Prefer --dist for
shipping applications.
PLENTY_DATAFRAME_ENGINE chooses an engine at compile time; setting it when
running an already-built application does not affect library lookup.
LD_LIBRARY_PATH and LD_PRELOAD can override RUNPATH, so remove stale
overrides when testing or switching releases. Keep the application directory
and library search paths trusted. System-wide library installation is also
possible through the operating system’s loader configuration.
The v1 filename denotes the C ABI generation, not a promise that every
release exports every operation. New operations can add symbols without
changing this name. Keep releases matched and do not substitute a differently
built Polars library. engine.json records the engine’s origin; --dist checks release/target/ABI
metadata and the copied library checksum. Ordinary compilation does not validate
its version; matching metadata is not proof of same-source identity for local builds.
| Failure | What to check |
|---|---|
cannot open shared object file naming libplenty_dataframe_v1.so | Put the matching library beside the executable |
undefined reference or symbol lookup error naming pldf_v1_ | Check for an old engine or a loader environment override |
GLIBC_2.xx not found | Check both the application and engine’s platform requirements; rebuild on the oldest supported system |
Directly emitted object files need an appropriate external link command and loader configuration; the automatic RUNPATH described here applies to executables linked by Plenty.
std.db.sqlite
import std.db.sqlite stores and queries data in SQLite databases through
prepared statements with typed parameters and typed column reads. It has one
error type, DbError.
For named byte-key maps without application SQL, use std.db.kv,
which shares this engine. Use this module for relational schemas, SQL queries,
or reusable prepared statements. See the storage comparison.
SQLite is vendored C code compiled into the runtime. The Plenty module owns the error model, row state, value limits, and transaction ownership. It is not an ORM: SQL is written by the program, and values are bound, never spliced into SQL text. The module is available on the supported x86-64 Linux GNU target.
import std.db.sqlite as sqlite
def main() -> Result[(), AllocError | IoError | sqlite.DbError]:
mut db = sqlite.connect("shop.db", sqlite.Mode.Create, 65536)?
db.execute("CREATE TABLE items(name TEXT NOT NULL, price INTEGER NOT NULL)")?
mut insert = db.prepare("INSERT INTO items VALUES (?, ?)")?
for name, price in [("pen", 3), ("book", 12), ("lamp", 25)]?:
insert.bind_text(1, name)?
insert.bind_integer(2, price)?
insert.run()?
mut query = db.prepare("SELECT name, price FROM items WHERE price > ? ORDER BY price")?
query.bind_integer(1, 5)?
while query.step()?:
print(query.text(0)?, query.integer(1)?)?
Ok(())
book 12
lamp 25
Opening
| Function | Result | Contract |
|---|---|---|
connect(path: &str, mode: Mode, max_length: i64) | Result[Connection, DbError] | The database file at path |
memory(max_length: i64) | Result[Connection, DbError] | A new private database in memory, gone when the connection closes |
Mode is ReadOnly, ReadWrite (the file must exist), or Create. A
missing file without Create is SQLite’s cant_open (14). path is a file
name, not a URI; "", ":memory:", and paths containing a NUL byte are
InvalidPath.
max_length is the largest text, blob, or row, in bytes, that the
connection stores or reads. It must lie in min_length_limit (1024) to
max_length_limit (1,000,000,000, SQLite’s own maximum); otherwise
OutOfRange. Binding, computing, or reading a larger value is TooLarge.
Choose it from the memory the program can spend: every text or blob read is
copied into a new str or Bytes.
Connection
| Method | Result | Contract |
|---|---|---|
execute(sql: &str) | Result[(), DbError] | Run every statement in sql to completion, discarding rows |
prepare(sql: &str) | Result[Statement, DbError] | Compile exactly one statement |
begin(mode: Begin) | Result[Transaction, DbError] | Start a transaction |
changes() | i64 | Rows changed by the most recent INSERT, UPDATE, or DELETE |
last_insert_rowid() | i64 | The rowid of the most recent successful INSERT |
set_busy_timeout(budget: &Duration) | Result[(), DbError] | How long to retry a locked database; see Busy databases |
load_extension(path: &str, entrypoint: &str = "") | Result[(), DbError] | Load a trusted native SQLite extension into this connection |
close() | Result[(), DbError] | Close now |
execute takes no parameters and is meant for schema and fixed SQL.
prepare rejects SQL with no statement (EmptySql) or more than one
(TrailingSql); a trailing ;, whitespace, or comment is allowed.
close is InUse while a statement or transaction made from the
connection is alive, and leaves it open. After a successful close, every
method that reaches SQLite is Closed, changes and last_insert_rowid
return 0, and a second close succeeds. Dropping a connection without
close closes it once its last statement and transaction are gone.
Native extensions and full-text search
load_extension loads a native shared library for the supported target. Only
load libraries you trust: their initialization and SQL functions execute native
code in your process. The extension must use SQLite’s extension API table;
it does not need a separately linked SQLite library. Use an explicit filesystem
path to select the library. SQLite may try platform suffixes if it cannot open
that path directly.
An empty entrypoint asks SQLite to discover the initialization symbol;
otherwise it names the exact exported symbol (for example, sqlite3_vec_init).
Empty or NUL-containing paths are InvalidPath; a NUL in the entrypoint is
InvalidExtensionEntry. Loader failures return Sqlite with SQLite’s loader
message. Closed is returned if the connection has been closed.
The native loading API is enabled only during the call, under the connection
mutex, and disabled again on success or failure. SQL’s load_extension() stays
disabled. Extensions belong to the connection and must be loaded again when
reopening a database that uses them. There is no public enable/disable toggle.
FTS5 is built in: CREATE VIRTUAL TABLE ... USING fts5(...), MATCH, and
bm25(...) work without loading an extension. Vector extensions such as
sqlite-vec are installed separately; Plenty does not download them.
Statement
Parameters are numbered from 1 and columns from 0.
| Method | Result | Contract |
|---|---|---|
parameter_count() | i64 | The largest parameter number |
parameter_index(name: &str) | Result[i64, DbError] | The number of :name, @name, or $name, prefix included; unknown is OutOfRange |
bind_null(index) | Result[(), DbError] | |
bind_integer(index, value: i64) | Result[(), DbError] | |
bind_real(index, value: f64) | Result[(), DbError] | NaN is OutOfRange |
bind_text(index, value: &str) | Result[(), DbError] | Copied |
bind_blob(index, value: &[u8]) | Result[(), DbError] | Copied |
clear_bindings() | Result[(), DbError] | Every parameter back to NULL |
step() | Result[bool, DbError] | Advance: True with a row current, False once finished |
reset() | () | Rewind, keeping the bindings |
run() | Result[(), DbError] | Reset, step to the end discarding rows, reset again |
column_count() | i64 | Columns in each row |
column_name(column) | Result[str, DbError] | The column’s name or AS alias |
kind(column) | Result[ValueType, DbError] | The storage class of the value in the current row |
is_null(column) | Result[bool, DbError] | |
integer(column), real(column), text(column), blob(column) | Result[i64 / f64 / str / Bytes, DbError] | The value, which must have that storage class |
optional_integer(column) and the other optional_ reads | Result[Option[...], DbError] | Nothing for NULL, otherwise as the strict read |
An index outside the statement’s parameters or columns is OutOfRange.
Binding after a step rewinds the statement first, so a statement is reused
by binding new values and stepping again. A statement that has finished
keeps returning False from step until reset or a new binding. A
failed step rewinds it.
Reads require a current row (NoRow otherwise) and never convert: SQLite’s
storage class must match the read, or the error is
TypeMismatch(found), where found is the stored class. ValueType is
Integer, Real, Text, Blob, or Null. A column’s declared type does
not decide its storage class; SQLite’s type affinity does, so an INTEGER
column given the text '7' stores the integer 7, but a TEXT column given
the integer 7 stores the text '7'. STRICT tables still allow lossless
conversion, including '7' into an INTEGER column; they reject values that
cannot be losslessly converted to the declared type. Text that is not valid
UTF-8 is InvalidUtf8.
Every text and blob read is copied out, so values stay valid after the next
step, reset, or the statement’s drop. There are no borrowed row views.
Transactions
Begin is Deferred (locks taken on first use), Immediate (write lock at
once), or Exclusive.
| Method | Result | Contract |
|---|---|---|
commit() | Result[(), DbError] | Make the changes durable |
rollback() | Result[(), DbError] | Discard the changes |
A connection has at most one transaction: begin while one is open is
TransactionActive. Dropping a transaction that was neither committed nor
rolled back rolls it back, so ? out of a function undoes its work.
A finished transaction, or one ended by SQL such as COMMIT run through
execute, is TransactionFinished on commit and rollback. A failed
commit that SQLite leaves open, such as one that is busy, may be retried
or rolled back.
Savepoints are available through execute("SAVEPOINT name") inside a
transaction.
Busy databases
A write needs a lock on the database file. When another connection holds
it, the default is to fail at once with SQLite’s busy (5) code.
set_busy_timeout(&budget) takes a std.time.Duration. Zero restores the
default; positive fractional milliseconds round up. A rounded budget above
2,147,483,647 milliseconds is OutOfRange. Use Begin.Immediate for transactions that
write, so a conflict is found at begin rather than partway through.
Errors
pub enum DbError:
Sqlite(SqliteError)
Closed
InUse
EmptySql
TrailingSql
TransactionActive
TransactionFinished
InvalidPath
InvalidExtensionEntry
OutOfRange
NoRow
TypeMismatch(ValueType)
TooLarge
InvalidUtf8
Allocation(AllocError)
| Variant | Meaning |
|---|---|
Sqlite(error) | A failure reported by SQLite: error.code is its extended result code, error.message its message |
Closed | The connection was closed |
InUse | close with statements or a transaction alive |
EmptySql, TrailingSql | prepare given no statement or more than one |
TransactionActive | begin with a transaction already open |
TransactionFinished | commit or rollback on a finished transaction |
InvalidPath | An empty, ":memory:", or NUL-containing path |
InvalidExtensionEntry | An extension entrypoint containing NUL |
OutOfRange | A parameter, column, limit, or timeout out of range, a NaN, or an unknown parameter name |
NoRow | A read with no current row |
TypeMismatch(found) | A read of the wrong storage class |
TooLarge | A value or row over the connection’s max_length |
InvalidUtf8 | Stored text that is not UTF-8 |
Allocation(cause) | The runtime or SQLite could not allocate |
SqliteError.primary() is the primary result code, the extended code modulo
256, using Plenty modulo with a positive divisor (for example, a manually
constructed SqliteError with code -1 has primary 255). Internal negative
native sentinel codes are decoded into
their existing DbError variants before primary-code classification.
The module names the common ones as constants: error (1), busy (5),
locked (6), read_only (8), cant_open (14), constraint (19), and
mismatch (20). For example, a NOT NULL violation has code 1299 and primary
code 19. SQLite’s own nomem, toobig, and range codes become
Allocation, TooLarge, and OutOfRange.
Ownership and threads
A Connection, Statement, or Transaction is an owned value; none can be
copied. A statement or transaction keeps its connection’s SQLite handle
alive, so a statement may outlive the Connection value it came from.
Dropping a statement finalizes it.
SQLite is compiled thread-safe and every connection is opened in serialized mode, so a connection, statement, or transaction may be moved to a worker thread and used there. Opening and loading an extension are path effects; other operations act on a handle the caller owns. Extension code must itself support the threads on which its functions are used. SQLite’s messages are copied per thread, so concurrent errors on different threads do not overwrite each other.
The handles are ByteStorage values. The __sqlite_* primitives are
reserved for the standard library and exist only for this
module. Given a handle not made by it, they stop the program with
operand is not a std.db.sqlite connection (or statement, or
transaction).
Costs
| Operation | Allocates |
|---|---|
connect, memory | A copy of the path, the connection owner, and SQLite’s connection state and page cache |
load_extension | Copies of the path and entrypoint, plus loader and extension allocations |
prepare, begin | The statement or transaction owner and SQLite’s compiled program |
bind_*, step, reset, kind, integer, real | Nothing in the runtime; SQLite copies bound text and blobs |
text, blob, column_name | The returned value |
SQLite allocates through the C allocator, outside Plenty’s allocation
accounting and failure injection; its own allocation failures are reported
as Allocation.
Supply
The runtime compiles the SQLite amalgamation from an unmodified copy under
plenty-runtime/native/sqlite-3.54.0, so building the compiler needs a C
compiler (cc, or CC) but no network, and an installed compiler needs
neither the sources nor a system SQLite:
| Library | Version | Licence | Source |
|---|---|---|---|
| SQLite | 3.54.0 | Public domain | sqlite-amalgamation-3540000.zip from sqlite.org, SHA3-256 7b670a62fdfbd672b75fef004cb703c8a3e87d3a5cc7d675b4a08337004a2d93 |
It is built with SQLITE_THREADSAFE=1, SQLITE_DEFAULT_MEMSTATUS=0,
SQLITE_DQS=0 (double-quoted strings are identifiers only),
SQLITE_LIKE_DOESNT_MATCH_BLOBS, SQLITE_OMIT_DEPRECATED,
SQLITE_ENABLE_FTS5, and SQLITE_OMIT_SHARED_CACHE. Native extension loading
is available through Connection.load_extension; SQL loading stays disabled.
Updating it is the runtime maintainers’ job, with each SQLite release that
fixes a defect: replace the directory with the new amalgamation, verify its
published SHA3-256, update the directory name in build.rs and the table
above, and rerun tests/test_std_sqlite.rs, which checks that the vendored
header, this page, and sqlite_version() agree.
Executables are linked with --gc-sections, SQLite is compiled with
function and data sections, and it is reached only through its own runtime
entry symbols, so a program that does not use std.db.sqlite contains none of it.
A program that uses SQLite includes the database engine and its enabled features.
Limits
- No connection pool, ORM, or query builder.
- No user-defined SQL functions, collations, hooks, or virtual tables.
- No incremental blob I/O, backup API, or online serialization.
- No URI file names.
- No borrowed row views: every text and blob read is a copy.
- SQLite’s allocations are outside Plenty’s allocation accounting.
- One target, x86-64 Linux GNU.
Acknowledgements
- Wraps SQLite 3.54.0, vendored as the C amalgamation and compiled into the runtime (public domain).
std.db.kv
import std.db.kv provides persistent, ordered byte-key storage without SQL
in application code. The small public API is implemented in Plenty over the
existing std.db.sqlite engine. It adds no native dependency.
See the engine decision for the SQLite/redb comparison.
import std.bytes as buffers
import std.db.kv as kv
from std.bytes import BytesError
def main() -> Result[(), IoError | AllocError | BytesError | kv.KvError]:
mut store = kv.connect("settings.db")?
store.put("config", &buffers.from_text("theme")?, &buffers.from_text("dark")?)?
store.close()?
mut reopened = kv.connect("settings.db", mode=kv.Mode.ReadOnly)?
match reopened.get("config", &buffers.from_text("theme")?)?:
case Some(value):
print(value.to_text()?)?
case Nothing:
print("unset")?
print(reopened.len("config")?)?
reopened.close()?
Ok(())
dark
1
Stores and tables
connect(path, mode=Mode.Create, max_length=1048576) returns
Result[Store, KvError]. Modes are Create, ReadWrite (existing file) and
ReadOnly. memory(max_length=1048576) creates a private in-memory store.
Paths are UTF-8 strings with the same validation as SQLite paths.
Logical table names are nonempty strings, bound as data rather than interpolated
into SQL. Each table is an independent namespace. It appears on first put;
an empty table and a nonexistent table behave identically. Keys and values
are byte strings, including empty ones, NULs and non-UTF-8 bytes: inputs are
&[u8] views and results are owned std.bytes.Bytes.
Keys sort lexicographically by unsigned bytes, shorter equal prefixes first.
Integer encodings must be chosen deliberately if numerical ordering matters.
| Store method | Result | Behavior |
|---|---|---|
put(table, key, value) | Result[(), KvError] | Insert or replace one key |
get(table, key) | Result[Option[Bytes], KvError] | Copy out the value, or Nothing |
contains(table, key) | Result[bool, KvError] | Test presence without copying the value |
remove(table, key) | Result[bool, KvError] | True if an entry was removed |
clear(table) | Result[i64, KvError] | Remove all entries; return their count |
len(table) | Result[i64, KvError] | Count entries (a table scan, not a cached counter) |
set_busy_timeout(budget: &Duration) | Result[(), KvError] | Retry lock conflicts for this duration (rounded up to milliseconds); default zero |
close() | Result[(), KvError] | Close, or Database(InUse) while transactions/scans still own handles |
All methods take &mut Store. Key/value arguments are borrowed and copied
into SQLite; returned values are owned and survive the next operation. Ordinary
methods prepare a statement per call. Use transactions to amortize commit I/O.
Transactions
store.begin(immediate=True) returns a Transaction. Its commit() and
rollback() return Result[(), KvError]. Dropping an unfinished transaction
rolls back, including when ? exits a function. Every operation on that Store
participates until completion. Outside a transaction each write auto-commits.
Only one transaction may be active per Store. Immediate mode takes the write
lock at the beginning; immediate=False defers locks until first use and is
appropriate for a read transaction. Deferred does not prohibit writes;
use Mode.ReadOnly for a read-only connection. A busy commit can be retried
or rolled back according to SQLite’s transaction contract.
Different Store connections follow SQLite isolation and locking. This module uses SQLite’s default rollback journal rather than enabling WAL automatically. Readers may delay a writer’s commit. No background thread, automatic retry loop or connection pool is introduced.
Bounded scans
scan(table, start, stop, max_rows: NonNegative[i64], after=False, reverse=False) returns a
Scan over [start, stop); after=True excludes start too, allowing an
ascending scan to resume after its last returned key. reverse=True changes
the order without changing the bounds. Reversed/equal bounds yield no rows.
scan_all(table, max_rows, reverse=False) has no key bounds, so it can include
every byte key, including keys composed entirely of 0xff bytes.
The row limit is required and is a NonNegative[i64]; zero yields nothing.
A limit truncates the scan, not an error; it does not promise that no more
matching rows exist. Scan.next() returns Result[Option[Entry], KvError];
each Entry owns public key and value buffers. Only one row is copied per
call. A copy allocation failure preserves the pending row for retry; a SQLite
step failure terminates that scan, avoiding silent replay of earlier rows.
Repeated calls after exhaustion return Nothing.
The snapshot starts at the first next, not construction. Finish/drop scans
before writing through the same Store or ending its transaction: SQLite does
not promise a stable scan when its own connection changes the rows being read.
Separate Store connections supply normal SQLite snapshot isolation. A Scan
keeps its underlying connection alive even if the Store is dropped. Drop the
Scan to release its statement and any read lock promptly.
Limits, errors and allocation
max_length is SQLite’s per-text/blob/encoded-row limit, between 1024 and
1,000,000,000 bytes. It includes row encoding overhead; the maximum value
payload is therefore smaller. It is not a total database size, page-cache
memory cap or execution deadline. Callers choose scan row limits independently.
KvError includes InvalidName, ForeignDatabase,
UnsupportedVersion(version), Allocation(AllocError) and
Database(std.db.sqlite.DbError). SQLite allocation errors are flattened to
KvError.Allocation; other database failures retain their structured cause,
including busy, read-only, I/O, closed handles and overlarge values.
Plenty-owned handle/row allocations participate in allocation accounting and
failure injection. SQLite’s C allocations are outside those hooks, but its
SQLITE_NOMEM failures are returned through the same allocation error.
File compatibility
Files use SQLite’s documented format and a versioned Plenty schema:
PRAGMA application_id = 1347177302; -- ASCII PLKV
PRAGMA user_version = 1;
CREATE TABLE plenty_kv(
table_name TEXT NOT NULL,
key BLOB NOT NULL,
value BLOB NOT NULL,
PRIMARY KEY(table_name,key)
) STRICT, WITHOUT ROWID;
Mode.Create initializes only an empty, untagged SQLite database, in one
transaction. It does not repurpose an unrelated database. Opening a known
application ID with an unsupported version fails without migrating it. Version
1 remains readable across Plenty releases; future incompatible changes require
an explicit migration/export path rather than silently rewriting old files.
SQLite tools can inspect a store directly; use SQLite 3.37+ for STRICT tables.
Manual schema/tag changes are unsupported, and the application ID is a format
tag, not authentication of an untrusted database.
Durability, recovery and journals remain SQLite’s responsibility. Do not copy an active database by copying just its main file: its journal may be needed for recovery. This API does not yet supply online backup or compaction methods. See SQLite file format and atomic commit.
The tutorial shows a batch transaction and bounded scan. Direct SQL, alternate engines, prefix helpers, zero-copy values and automatic typed serialization are outside this API.
Choosing key-value storage
std.db.kv is a convenience interface over SQLite, not a new database engine.
The implementation keeps byte-key policy in Plenty and reuses SQLite for
storage, transactions, locking and crash recovery.
| Choice | Best fit | Tradeoffs |
|---|---|---|
std.db.kv | A dedicated file of named byte-key maps, ordered scans and atomic batches | Small API; copied byte values; prepares statements per call; only the documented KV schema |
std.db.sqlite | SQL queries, relational tables, joins, indexes, shared application schemas, or tuned prepared-statement reuse | More application SQL, but access to the same mature engine and tools |
| redb (not shipped by Plenty) | An application wanting a specialized embedded ordered KV engine | Rust-native typed/zero-copy access and MVCC; another file format and dependency; its allocation contract currently prevents a thin Plenty wrapper |
SQLite implements ordered-key storage through
WITHOUT ROWID. Our composite primary
key is (table_name,key), so scans can seek to a namespace and byte range.
This avoids a separate rowid table plus primary-key index. SQL parsing,
binding and copied-out values still cost work. Direct SQLite can reuse prepared
statements for a hot loop; the initial KV convenience API does not cache them.
Batch transactions often matter more than individual call overhead for writes.
No benchmark here establishes that SQLite is faster or slower than redb for
your workload; durability settings, batches, cache and key sizes must match.
Why not wrap redb now?
The evaluated crate is redb 4.4.0, downloaded for source inspection only; it is not a build dependency. Its published design uses copy-on-write B+trees with concurrent snapshot readers and a writer. Its README describes a stable file format with reasonable efforts toward future upgrades. Those are useful properties for a dedicated KV engine.
However, its ordinary database creation and page-cache paths contain
infallible Box::new, Arc::new, vector allocation and collection into
Arc<[u8]>. Concrete examples are Builder::new/database setup in src/db.rs
and zero_filled_arc/read_direct in
src/tree_store/page_store/cached_file.rs. There is no fallible-allocation
boundary around those paths that a small Plenty wrapper can use. Returning
Result for database I/O does not make those allocations fallible, and
catching a panic does not recover from allocator aborts.
See the published crate source
and crate description.
Meeting Plenty’s requirement would need a substantial fallible-allocation
audit/change to the engine, not just converting its public error enum.
SQLite already reports allocation failures via SQLITE_NOMEM, which Plenty’s
existing adapter maps to AllocError. Reusing it is a much smaller change
than building and validating another durable storage engine in Plenty.
This choice gives Plenty users the requested KV ergonomics now. It does not promise redb-compatible files, redb-level zero-copy reads, or backend-neutral file interchange. The KV reference publishes the actual SQLite schema and compatibility rules.
std.deque
from std.deque import Deque provides an owned generic ring buffer. It is
ordinary embedded Plenty code with private list[Option[T]] storage; it does
not change list indexing or list.pop(0) costs.
| Operation | Result | Contract |
|---|---|---|
Deque[T]() | Result[Deque[T], AllocError] | Construct an empty deque |
len(queue) or queue.len() | i64 | Observe the live element count |
queue.is_empty() | bool | Observe whether the count is zero |
queue.append(value) | Result[(), AllocError] | Consume a value and append on the right |
queue.appendleft(value) | Result[(), AllocError] | Consume a value and append on the left |
queue.pop() | Option[T] | Remove and transfer the rightmost element |
queue.popleft() | Option[T] | Remove and transfer the leftmost element |
queue.clear() | () | Destroy elements from right to left, retaining capacity |
from std.deque import Deque
def main() -> Result[(), AllocError | IoError]:
mut queue = Deque[str]()?
queue.append("b")?
queue.append("c")?
queue.appendleft("a")?
print(len(queue), queue.pop())?
while True:
match queue.popleft():
case Some(item):
print(item)?
case Nothing:
break
Ok(())
3 Option[str].Some("c")
a
b
Mutation requires a mutable owner or exclusive reference. Empty removals return
Nothing. Stored elements follow ordinary collection storage restrictions,
including affine owners; references, generators, and unit elements are unsupported.
End removal takes constant time apart from payload cleanup and allocates nothing. Appending is amortized constant time; growth takes linear time and doubles capacity, starting at eight slots. It reserves and initializes the full new buffer before transferring any existing element. Allocation or capacity failure preserves existing contents and drops the incoming value. Appending within capacity, removing, and clearing allocate no runtime storage.
The deque moves and requires explicit copy for independent duplication. It
has no indexed access, membership operator, iteration protocol, bounded capacity, rotation, or bulk
extension API. To consume it, repeatedly pop an end. See the runnable
queue lesson.
Acknowledgements
- API modelled on Python’s
collections.deque.
std.diff
import std.diff compares two sequences and returns the edit operations that
turn the old one into the new one. It diffs generic lists, bytes, and text split
into lines, words, or Unicode scalars, and formats text differences as unified
diffs. It is ordinary embedded Plenty code: every allocation failure is reported
as AllocError. Unrepresentable frontier storage sizes return
AllocError.CapacityOverflow before frontier allocation.
| Operation | Result | Contract |
|---|---|---|
std.diff.diff(&old, &new, algorithm, max_cost) | Result[list[Op], AllocError] | Edit operations between two list[T]; T must be usable as a dictionary key |
std.diff.diff_bytes(&old, &new, algorithm, max_cost) | Result[list[Op], AllocError] | Edit operations between the bytes of two Bytes values |
std.diff.lines(old, new, algorithm, max_cost) | Result[TextDiff, AllocError] | Compare &str texts line by line |
std.diff.words(old, new, algorithm, max_cost) | Result[TextDiff, AllocError] | Compare runs of whitespace and runs of other scalars |
std.diff.chars(old, new, algorithm, max_cost) | Result[TextDiff, AllocError] | Compare Unicode scalar by scalar |
text_diff.ops() | &list[Op] | The operations, indexing the tokens |
text_diff.old_tokens(), text_diff.new_tokens() | &list[str] | Each side’s tokens |
text_diff.has_changes() | bool | False when the texts are equal |
text_diff.unified(context, old_name, new_name) | Result[str, AllocError] | Unified diff text, or "" for equal texts |
algorithm is optional and defaults to Algorithm.Myers. max_cost is an
optional Option[i64] that defaults to Nothing, no limit; see
Cost limit. Import Algorithm, Kind, Op, and TextDiff
from std.diff to name them.
import std.diff
def main() -> Result[(), AllocError | IoError]:
old = "apples\nbread\nmilk\n"
new = "apples\nbutter\nmilk\neggs\n"
changes = std.diff.lines(old, new)?
print(changes.has_changes(), len(changes.ops()))?
print(changes.unified(1, "a/list.txt", "b/list.txt")?.rstrip()?)?
Ok(())
True 5
--- a/list.txt
+++ b/list.txt
@@ -1,3 +1,4 @@
apples
-bread
+butter
milk
+eggs
Operations
An Op has public kind, old_start, old_end, new_start, and new_end
fields. It says that old[old_start:old_end] becomes new[new_start:new_end].
op.old_len() and op.new_len() return
Result[i64, OverflowError[i64]]. They subtract the public endpoints as given,
including representable negative lengths; an unrepresentable difference returns
the precise wrapped overflow value. op.is_equal() returns bool.
Operations produced by the diff algorithms have nonnegative, representable lengths.
kind | Old range | New range |
|---|---|---|
Kind.Equal | Nonempty | Equal elements, same length |
Kind.Delete | Nonempty, removed | Empty, at the position of the removal |
Kind.Insert | Empty, at the insertion position | Nonempty, added |
Operations are in order and cover both sequences without gaps: each starts
where the previous one ended. No operation is empty, two adjacent operations
never have the same kind, and a deletion comes before an adjacent insertion.
There is no separate replace kind; a replacement is a deletion followed by an
insertion. Equal inputs give one Equal operation, or none when both are empty.
Algorithms
Algorithm.Myers finds a shortest edit script: the number of equal elements is
the length of a longest common subsequence. It uses the linear-space
divide-and-conquer form of Myers’ algorithm, after removing the common prefix
and suffix. It takes O((N+M)·D) time and O(N+M) space, where D is the number of
deleted and inserted elements. Small changes to large inputs are fast; two
large inputs with nothing in common take time quadratic in their size unless
max_cost limits the search.
Algorithm.Patience matches the elements that occur exactly once in each
sequence, keeps the longest increasing run of those matches as anchors, and
runs Myers between anchors. The script is valid but not always shortest. It
often reads better for source code, where repeated lines such as braces and
blank lines otherwise pull unrelated blocks into alignment.
Both algorithms first intern elements to integer ids through a dictionary, so
comparisons during the search are integer comparisons. Text tokens are str
copies that share storage with each other but not with the input text.
Cost limit
max_cost=Some(c) bounds the work for inputs that differ a lot, such as large
untrusted texts. The result is still a valid script, but may not be shortest.
- Each step of the divide-and-conquer search looks for the middle of a shortest
path through one range. With a limit, that search stops once its forward and
backward paths together reach about
cdeletions and insertions. - When the search stops, the range is split at the end of the search path
that covers the most elements, as git does without
--minimal. The part that path covers is within the limit and is diffed exactly; the rest is diffed again with the same limit. Elements on opposite sides of the split are never matched. - If a shortest script for the whole input has at most
cdeletions and insertions, the limit never fires and the result is the same as without it. - Time is roughly proportional to (N+M)·
cinstead of (N+M)·D. Space is unchanged. - A negative
ccounts as zero. Values up to 2 behave alike: each search stops after one edit in each direction.
Algorithm.Patience applies the limit both to matching the unique elements and
to each Myers search between anchors. The limit does not change which elements
are compared or how text is split into tokens.
For two texts of 3000 lines with no line in common, lines takes about 300 ms
without a limit, 15 ms with max_cost=Some(256), and grows linearly with the
input under the limit. Git stops each search after at least 256 edits in each
direction, which corresponds to max_cost=Some(512).
Tokens
Each side’s tokens concatenate to its original text.
linessplits after each LF. A line keeps its LF, and a CR before it, so a final line without an LF differs from the same line with one.wordsalternates runs of whitespace (str.isspace) and runs of other scalars.charsmakes one token per Unicode scalar, not per grapheme cluster.
Unified format
unified follows the output of GNU diff -U:
- The text starts with
--- old_nameand+++ new_namelines. No timestamp is added. - Each hunk header is
@@ -start,count +start,count @@, with 1-based starts. A count of 1 is omitted, and an empty range gives the line before it as its start, as in@@ -0,0 +1 @@. - Each token is written as one line prefixed by a space,
-, or+. A token that does not end in LF is followed by an LF and the line\ No newline at end of file. contextunchanged tokens surround each change. Changes separated by at most2 * contextunchanged tokens share a hunk. A negative context counts as zero. Eveni64.MAXcontext is accepted without overflowing the hunk comparison.
unified is meant for lines. For word or scalar diffs, every token still
becomes its own output line.
There is no inline (word-level) rendering, patch parsing or application, or three-way merge. See the runnable diff lesson.
Acknowledgements
- Myers and patience implementations derived from similar 3.2.1 by Armin Ronacher and contributors (Apache-2.0).
- Uses the linear-space variant of Myers’ O(ND) algorithm (1986).
- Patience diff was devised by Bram Cohen.
- Unified output matches GNU diffutils
diff -U.
std.encoding
import std.encoding provides hex, base32, base64, PEM and custom codecs and
fixed-width integer and IEEE 754 float packing over std.bytes
buffers, with one error type, EncodingError.
Byte order, range checks, growth and byte manipulation are checked Plenty code. Four std-only compiler intrinsics reinterpret same-width integer/float bits with a native bitcast; they add no runtime library or dependency. Bytes are accessed through validated buffer operations. Availability matches integer packing on the supported targets; the float tests are qualified on x86-64 Linux GNU.
These are binary encodings. They do not normalize, transcode, or validate text;
turning decoded bytes into a str is the separate, strict
Bytes.to_text conversion.
import std.bytes
import std.encoding
from std.encoding import EncodingError, Endian
def main() -> Result[(), AllocError | IoError | std.bytes.BytesError | EncodingError]:
data = std.bytes.from_text("hi!")?
print(std.encoding.hex_text(&data)?)?
print(std.encoding.base64_text(&data)?)?
print(std.encoding.base64_from_text("aGkh")?.to_text()?)?
mut header = std.bytes.with_capacity(4)?
std.encoding.append_u32(&mut header, 258u32, Endian.Big)?
print(std.encoding.hex_text(&header)?)?
print(std.encoding.unpack_u32(&header, 0, Endian.Big))?
Ok(())
686921
aGkh
hi!
00000102
258
Codecs
source is a byte view &[u8]: pass &buffer for a whole Bytes,
&buffer[start:stop] for part of it, or a b"..." literal.
| Function | Result | Contract |
|---|---|---|
hex_encode(source) | Result[Bytes, EncodingError] | Two lowercase digits per byte |
hex_decode(source) | Result[Bytes, EncodingError] | Digits of either case; even length |
base64_encode(source) | Result[Bytes, EncodingError] | RFC 4648 section 4, padded |
base64_decode(source) | Result[Bytes, EncodingError] | The padded form only |
base64url_encode(source) | Result[Bytes, EncodingError] | RFC 4648 section 5, unpadded |
base64url_decode(source) | Result[Bytes, EncodingError] | The unpadded form only |
hex_text, base64_text, base64url_text (source) | Result[str, EncodingError] | The same encoding as a str |
hex_from_text, base64_from_text, base64url_from_text (text: &str) | Result[Bytes, EncodingError] | Decode the whole text |
Each result is a new buffer whose capacity equals its length. Encoders produce ASCII only.
For base32/base32hex, MIME line wrapping, PEM, and reusable custom alphabets, see additional codecs. They use the same views, result types, and byte indexes as the functions here.
Accepted forms
The codecs in this table accept one form. They never skip, trim, or repair input: whitespace, line breaks, and any other byte outside the alphabet are errors.
| Codec | Alphabet | Length | Padding |
|---|---|---|---|
| hex | 0-9, a-f, A-F; encoder writes lowercase | Even | None |
| base64 | A-Z, a-z, 0-9, +, / | Multiple of 4 | = required to complete the last group |
| base64url | A-Z, a-z, 0-9, -, _ | Any except 1 more than a multiple of 4 | = rejected |
Base64 text must be canonical: bits of the last symbol that encode no byte must be zero, so each byte sequence has exactly one accepted text per codec. The empty range encodes and decodes to an empty buffer.
Decoder checks, in order:
- The range, then the length.
- For base64, up to two trailing
=are set aside as padding. - Bytes are examined in order. The first one outside the alphabet is
InvalidPaddingif it is=andInvalidByteotherwise. - The trailing bits of a final partial group.
Integer packing
value has the type in the function name; offset is i64; order is
Endian.Little or Endian.Big. There is no default or native order.
| Function | Result | Contract |
|---|---|---|
pack_u16, pack_u32, pack_u64, pack_i16, pack_i32, pack_i64 (target: &mut Bytes, offset, value, order) | () | Overwrite the bytes at offset; never grows |
append_u16, append_u32, append_u64, append_i16, append_i32, append_i64 (target: &mut Bytes, value, order) | Result[(), EncodingError] | Grow by exactly the width |
unpack_u16, unpack_u32, unpack_u64, unpack_i16, unpack_i32, unpack_i64 (source: &[u8], offset, order) | u16, and so on | Read the bytes at offset |
Endian.Big puts the most significant byte at the lowest offset. Signed values
use two’s complement. pack_ and unpack_ require
0 <= offset <= len - width; a short buffer or invalid offset traps before anything is
read or written. Bytes are accessed one at a time, so no offset has an
alignment requirement. A single byte needs no order: use Bytes.get, set,
and push.
Floating-point packing
| Function | Result | Contract |
|---|---|---|
pack_f32, pack_f64 (target: &mut Bytes, offset: i64, value, order: Endian) | () | Overwrite exactly four/eight existing bytes; never grows |
append_f32, append_f64 (target: &mut Bytes, value, order: Endian) | Result[(), EncodingError] | Append exactly four/eight bytes |
unpack_f32, unpack_f64 (source: &[u8], offset: i64, order: Endian) | f32, f64 | Read exactly four/eight bytes |
The value type is the width in the name: f32 is IEEE 754 binary32 and f64
is binary64. Endianness is explicit, with no native-order default, alignment,
padding, or object-layout serialization. Sources use the same byte views as
integer unpacking: a whole buffer, a borrowed range, or a byte literal.
These operations reinterpret bits without numeric conversion, promotion, arithmetic, or text formatting. They preserve every representation, including positive and negative zero, infinities, subnormals, and quiet/signaling NaNs with their signs and payloads. Every input bit pattern is accepted. Subsequent arithmetic on a decoded float can change a NaN representation and lies outside this codec’s guarantee. Compare re-encoded bytes when testing representations; float equality cannot distinguish signed zeros or verify NaNs.
The integer bounds and failure guarantees apply unchanged. pack_ and
unpack_ require 0 <= offset <= len - width, including at unaligned offsets
inside a larger record. Negative, excessively large, and truncated ranges
trap before reading or modifying bytes. The messages are pack: offset outside target buffer
and unpack: offset outside source buffer. Packing leaves length,
capacity, and surrounding bytes unchanged. Appending keeps the existing prefix;
when growth fails, contents and length remain unchanged and the error is
Allocation(cause). Range checks avoid overflowing offset arithmetic.
Errors
pub enum EncodingError:
InvalidLength
InvalidByte(i64)
InvalidPadding(i64)
NonCanonical(i64)
InvalidAlphabet
InvalidPem(i64)
PemLabel
Allocation(AllocError)
| Variant | Meaning |
|---|---|
InvalidLength | The encoded length is impossible for the codec |
InvalidByte(index) | The byte at index is outside the alphabet |
InvalidPadding(index) | A configured padding byte at index where the codec does not accept it |
NonCanonical(index) | The symbol at index has nonzero bits that encode no byte |
InvalidAlphabet | A custom alphabet or padding symbol violates its construction rules |
InvalidPem(index) | The PEM framing is malformed at index |
PemLabel | The PEM label is not the expected one, or the expected label is invalid |
Allocation(cause) | Storage for the result could not be allocated, including an unrepresentable expanded size (CapacityOverflow) |
An index is a position in the source view; for the
_from_text forms it is a UTF-8 byte offset into the text. Every fallible
operation returns Result[T, EncodingError]. A failed operation leaves its
target unchanged and releases anything it allocated. Wrap a result with
std.error.context to name the operation.
Hex, base64, MIME and PEM expansion sizes are checked before allocating output.
They return Allocation(CapacityOverflow) rather than trapping on integer
overflow. Bounded decoder cursors, fixed-width packing and alphabet arithmetic
use explicit unchecked: regions whose source comments state their bounds.
PEM decoding records the validated body endpoint instead of reconstructing it
with a potentially overflowing size formula. Invalid input keeps its existing
codec error and source offset; callers’ arithmetic modes do not change these
contracts.
Ownership and threads
Sources are shared loans and targets exclusive loans, so the view rules apply: a buffer cannot be packed into while a view of it is live, and cannot be encoded into itself. The module holds no global state. Its functions pass the ordinary worker-effect checks and may run in any thread.
Costs
This table covers the original hex/base64 codecs and integer/float packing. See additional codec costs for custom table construction and MIME’s temporary buffers.
| Operation | Allocates | Time |
|---|---|---|
_encode, _decode | One buffer of exactly the output size, after the range and length checks | Linear in the range |
_text | The encoded buffer, then a str copy of it | Linear |
_from_text | A Bytes copy of the text, then the decoded buffer | Linear |
append_ | Only when capacity is exceeded, as Bytes.resize | Constant, amortized |
pack_, unpack_ | Nothing | Constant |
Output sizes for n input bytes: hex 2n; base64 4 * ceil(n / 3);
base64url ceil(4n / 3). A decoder allocates its output before it examines
the alphabet, so rejected input pays for that buffer, which is released before
returning.
Every byte goes through a checked single-element Bytes call. Measured with a
release compiler on x86-64 Linux, pinned to one core, on an 8,388,608-byte
input:
| Operation | Wall time |
|---|---|
hex_encode | 1.9 s |
hex_decode of the 16,777,216 digits | 2.5 s |
base64_encode | 1.3 s |
base64_decode of the 11,184,812 symbols | 1.3 s |
1,048,576 pack_u64 and 1,048,576 unpack_u64 | 2.9 s |
That is a few megabytes per second. Bulk native primitives would be faster and are not implemented.
Limits
- No streaming or incremental codecs: input and output are whole buffers.
- No arbitrary line widths, automatic case folding, or permissive recovery from damaged encodings.
- No variable-length integers, 128-bit integers, or additional float formats.
- No record-layout description: pack fields one call at a time.
Acknowledgements
- RFC 4648 specifies the base16, base32 and base64 formats implemented by the codecs.
- RFC 7468 specifies the textual encapsulation and label syntax used by PEM encoding.
- IEEE 754 defines the binary32 and binary64 representations used by float packing; no standard text or external implementation is shipped.
Base32, MIME, PEM and custom alphabets
These operations extend std.encoding. They are implemented in
Plenty over Bytes, without another runtime primitive or native
dependency. Inputs are whole buffers; there is no streaming state.
import std.bytes
import std.encoding
from std.encoding import EncodingError
def main() -> Result[(), AllocError | IoError | std.bytes.BytesError | EncodingError]:
data = std.bytes.from_text("hello")?
print(std.encoding.base32_text(&data)?)?
print(std.encoding.base32_from_text("NBSWY3DP")?.to_text()?)?
lower = std.encoding.custom("abcdefghijklmnopqrstuvwxyz234567", Nothing)?
text = lower.text(&data)?
print(text, lower.from_text(&text)?.to_text()?)?
Ok(())
NBSWY3DP
hello
nbswy3dp hello
Base32
base32 uses the uppercase alphabet ABCDEFGHIJKLMNOPQRSTUVWXYZ234567 from
RFC 4648 section 6.
base32hex uses 0123456789ABCDEFGHIJKLMNOPQRSTUV from section 7. Both require
= padding to complete eight-symbol groups. They are case-sensitive and do
not accept whitespace, 0/1 substitutions, or nonzero unused trailing bits.
Function suffix, for both base32 and base32hex | Result |
|---|---|
_encode(source: &[u8]) | Result[Bytes, EncodingError] |
_decode(source: &[u8]) | Result[Bytes, EncodingError] |
_text(source: &[u8]) | Result[str, EncodingError] |
_from_text(text: &str) | Result[Bytes, EncodingError] |
For example, f encodes as MY====== or CO======. Unpadded forms are
available through custom below; a decoder never guesses the padding policy.
The empty input encodes as empty, not a group of padding characters.
Range validation comes first. Padded lengths must be a multiple of eight.
After stripping trailing padding, the number of symbols must describe whole
bytes with fewer unused bits than a symbol. Padding must fill exactly one
partial group. Symbols and trailing bits are then checked in input order.
Impossible lengths produce InvalidLength; excessive or misplaced padding
produces InvalidPadding(index) when the preceding length checks pass.
MIME line wrapping
base64mime_encode, base64mime_decode, base64mime_text and
base64mime_from_text have the corresponding base64 signatures.
The encoder produces padded standard base64 with CRLF between lines of at most 76 symbols, following the line width of RFC 2045 section 6.8. It adds no final CRLF and emits nothing for empty input. Add the surrounding message headers and any final line termination in the enclosing protocol.
The decoder skips only ASCII space, tab, CR and LF. It then applies ordinary
strict base64 length, alphabet, padding and trailing-bit validation. It accepts
unwrapped text and does not enforce a line width. This is deliberately narrower
than RFC 2045’s permissive recovery from arbitrary non-alphabet bytes: punctuation,
non-ASCII whitespace, and damaged padding remain errors. Error indexes still
refer to the original buffer, including skipped whitespace; _from_text indexes
are UTF-8 byte offsets. It does not parse MIME headers or multipart messages.
PEM
pem_encode(source: &[u8], label: &[u8])
-> Result[Bytes, EncodingError]
pem_text(source: &[u8], label: &[u8])
-> Result[str, EncodingError]
pem_decode(source: &[u8], label: &[u8])
-> Result[Bytes, EncodingError]
pem_from_text(text: &str, label: &[u8]) -> Result[Bytes, EncodingError]
These read and write one block in the strict form of
RFC 7468 (section 3,
stricttextualmsg), the textual wrapping of DER keys and certificates.
label is the text between BEGIN and the dashes, such as b"PUBLIC KEY"
or b"PRIVATE KEY"; a bytes literal costs no allocation. The decoder takes
the label it expects, so a private key cannot be read where a public key was
meant.
import std.bytes
import std.encoding
from std.encoding import EncodingError
def main() -> Result[(), AllocError | IoError | std.bytes.BytesError | EncodingError]:
pem = "-----BEGIN PUBLIC KEY-----\nMCowBQYDK2VwAyEA11qYAYKxCrfVS/7TyWQHOg7hcvPapiMlrwIaaPcHURo=\n-----END PUBLIC KEY-----\n"
der = std.encoding.pem_from_text(pem, b"PUBLIC KEY")?
print(der.len(), std.encoding.pem_text(&der, b"PUBLIC KEY")? == pem)?
print(str.repr(std.encoding.pem_from_text(pem, b"PRIVATE KEY"))?)?
Ok(())
44 True
Result[std.bytes.Bytes, std.encoding.EncodingError].Err(std.encoding.EncodingError.PemLabel)
The decoder accepts exactly:
-----BEGIN LABEL-----at the start of the view, with the expected label;- one line terminator, LF or CRLF, used for every line that follows;
- padded standard base64 in lines of exactly 64 symbols, except a final line of 1 to 64 symbols, with at least one symbol in all;
-----END LABEL-----with the same label;- at most one final line terminator, then the end of the view.
Explanatory text around the block, RFC 1421 headers such as Proc-Type,
blank lines, spaces, tabs, and other line widths are errors. A label that is
not an RFC 7468 label (printable ASCII except -, with single spaces or
hyphens only between other characters) or a BEGIN label other than label is
PemLabel. Other structural errors are InvalidPem(index); base64 errors give
the same variants as base64_decode. Every index is a position in source.
The decoder does not interpret the DER it returns; the
key import functions do.
The encoder writes that form with LF terminators, including a final one. An
empty range has no strict PEM form and is InvalidLength.
Reusable custom encoding
custom(symbols: &str, padding: Option[u8] = Some(61u8))
-> Result[Encoding, EncodingError]
symbols lists digits in increasing value order. It must contain 2, 4, 8, 16,
32 or 64 distinct printable ASCII bytes (! through ~). A padding byte must
also be printable ASCII and must not occur in the alphabet. Invalid settings
return InvalidAlphabet. Some(61u8) means =; Nothing forbids padding.
With no configured padding, = may itself be an alphabet symbol.
Encoding owns its validated tables and has four methods:
| Method | Result |
|---|---|
encode(source: &[u8]) | Result[Bytes, EncodingError] |
decode(source: &[u8]) | Result[Bytes, EncodingError] |
text(source: &[u8]) | Result[str, EncodingError] |
from_text(text: &str) | Result[Bytes, EncodingError] |
Bits are read most significant first. Padding completes the smallest group of
symbols representing whole bytes. Decoding accepts only the selected alphabet
and policy, with zero unused trailing bits; it does not fold case, translate
lookalike characters, ignore whitespace, or automatically detect a radix.
If no padding is configured, a byte outside the alphabet is InvalidByte,
including =. If configured, misplaced occurrences of that padding byte are
InvalidPadding.
For a known protocol, prefer the named codec. Construct a custom encoding once and reuse it when the protocol requires a different alphabet or padding. The tutorial demonstrates this.
Allocation and limits
Base32 and custom encode/decode allocate one result buffer of exactly its final
length. Fixed base32 alphabets need no temporary table. custom allocates its
alphabet and a 128-byte decoding table once; its shared methods do not mutate
them. Construction and all codec allocations report EncodingError.Allocation.
The source and existing encoding remain unchanged on failure.
MIME encoding holds ordinary base64 plus its wrapped result; MIME decoding
holds a whitespace-free copy plus its decoded result. These are whole-buffer
copies, not zero-copy adapters. _text additionally copies output into a str;
_from_text additionally copies input into Bytes, as with existing codecs.
PEM encoding holds ordinary base64 plus the framed result; PEM decoding holds
the base64 body without line terminators plus its decoded result. A decoded
private key is therefore left in freed buffers that are not zeroed; see
key material.
All operations take linear time in input length. Base32 output is
8 * ceil(n / 5) bytes. Unpadded custom output is ceil(8n / bits_per_symbol);
padded output rounds that to a whole symbol group. New radix encoders reject
unrepresentable size arithmetic with InvalidLength. No custom bit order,
case folding, ignored-character configuration, or arbitrary wrapping is exposed.
std.env
import std.env reads the process environment and the working directory.
Names, values, and directories are operating-system byte strings; text is a
checked view of them.
Encoding and validation are checked Plenty; four native primitives read a
variable, snapshot the environment, and read or change the working directory.
The module is available on the supported x86-64 Linux GNU target; elsewhere
every operation reports IoError.Unsupported.
import std.env
def main() -> Result[(), IoError]:
mode = std.env.get("PLENTY_EXAMPLE_MODE")?.unwrap_or("development")
print("mode:", mode)?
cwd = std.env.current_dir()?
print(cwd.is_absolute())?
Ok(())
mode: development
True
Variables
| Function | Result | Contract |
|---|---|---|
get(name: &str) | Result[Option[str], IoError] | The value as text; Nothing when unset |
get_bytes(name: &list[u8]) | Result[Option[list[u8]], IoError] | The value unchanged |
vars() | Result[list[Variable], IoError] | A snapshot of every variable |
valid_name(name: &list[u8]) | bool | Nonempty, with no = and no NUL |
Variable has two public list[u8] fields, name and value.
- An unset variable is
Ok(Nothing). A variable set to the empty string isOk(Some("")). - A name that is not
valid_nameisIoError.InvalidInput: no variable can have it. getreports a value that is not UTF-8 asIoError.Data(DataError.InvalidUtf8). It never replaces or drops bytes; read such a value withget_bytes.varsreturns each variable once, in the order the process stores them. The snapshot is detached: it does not follow later changes.- Each result is a new allocation; failure is
IoError.Data(DataError.Allocation(...)).
There is no operation that sets or removes a variable of the running program.
The environment is process-wide state that other threads and native code read
without synchronization. Give a child its variables through
Command.env instead.
Native code embedding Plenty must likewise avoid concurrent mutation of the
process environment while runtime calls read it. Both single-variable lookup
and snapshots copy directly into fallible Plenty storage, without intermediate
host strings or snapshots.
Command-line arguments are the prelude’s
args and args_bytes, and typed
options are std.cli.
Working directory
| Function | Result | Contract |
|---|---|---|
current_dir() | Result[Path, IoError] | The absolute working directory |
set_current_dir(path: &Path) | Result[(), IoError] | Change it for the whole process |
Both use std.path.Path, so a directory name that is not UTF-8
round-trips. current_dir fails with the system error when the directory has
been removed. set_current_dir reports a missing directory as
IoError.System(2) and leaves the working directory unchanged.
current_dir() grows a fallible native buffer and reports allocation failure
as IoError.Data(DataError.Allocation(...)).
The working directory is shared by every thread, and relative paths resolve
against it at the moment of each call. Prefer absolute paths, or
Command.current_dir for a child, over
changing it in a program with more than one thread. A worker thread may not
call set_current_dir at all; the compiler rejects any worker that can reach
it (see native effects).
Limits
- Linux x86-64 GNU only.
- No mutation of the program’s own environment.
- No executable-path lookup. The home and XDG base directories are
std.fs.dirs; temporary-directory selection isstd.tempfile.default_dir.
Acknowledgements
- API modelled on Rust’s
std::env; theget/get_bytespair follows Python’sos.getenvandos.getenvb.
std.error — explicit mapping and contextual errors
Import std.error. A Context[E] records which operation failed and, when
there is one, the path it acted on, around an unchanged typed cause E. The
module is ordinary embedded Plenty code over Result
and std.path; it adds no exceptions, implicit conversions, or
runtime support.
pub enum Subject:
Absent
At(Path)
Unavailable(AllocError)
pub class Context[E]:
pub operation: str
pub subject: Subject
pub cause: E
| Operation | Result | Contract |
|---|---|---|
map_error(result, mapper) | Result[T, G] | Consumes Result[T, E]; invokes an owned OnceCallable[[E], G] only on Err; same as the result.map_error(mapper) method |
Context(operation, subject, cause) | Context[E] | Takes ownership of all three; allocates nothing |
context(result, operation) | Result[T, Context[E]] | Wraps an Err with Subject.Absent; allocates nothing |
path_context(result, operation, &path) | Result[T, Context[E]] | Wraps an Err, copying the path only then |
error.describe() | Result[str, AllocError] | One-line text ending in str.repr of the cause |
error.describe_with(&cause_text) | Result[str, AllocError] | The same text with a caller-supplied cause |
error.format(&mut output) | Result[(), FormatError] | Borrowed std.report formatting when E satisfies Formattable |
error.report() | () | Writes error: and the description to standard error |
error.report_with(description) | () | The same for a description Result the caller built |
native_code(error) | Option[i32] | The OS code of IoError.System(code); otherwise Nothing |
context and path_context consume the Result and return Ok payloads
unchanged, including () from operations such as flush_stdout() and
File.close(). Conversion is always this explicit call: ? neither wraps an error
in a Context nor unwraps one, and the
identical-error rule
applies to Context[E] as to any error type. A Context moves, because it can
own a path. Its fields are public: read error.cause or match it to recover the
typed cause, at any nesting depth. ? in a function returning
Result[T, Failure], the last-resort erased error, discards a Context like any
other error; list Context[E] in the sum to keep it.
from std.error import Context, path_context
from std.path import Path, from_text
def read_settings(path: &Path) -> Result[list[u8], Context[IoError]]:
path_context(read_bytes(path.as_bytes()), "read settings", path)
def main() -> Result[(), AllocError | IoError]:
path = from_text("settings.toml")?
match read_settings(&path):
case Ok(bytes):
print(len(bytes), "bytes of settings")?
case Err(error):
# A missing settings file is expected: report it and use defaults.
print(error.describe()?)?
print(error.cause == IoError.System(2i32))?
print("using default settings")?
Ok(())
read settings "settings.toml": IoError.System(2)
True
using default settings
Explicit error mapping
result.map_error(mapper) consumes Result[T, E] and returns Result[T, G],
where mapper is an owned OnceCallable[[E], G]. A one-payload enum variant is
the usual mapper; use a named classifier when several error variants need
different policies, or a closure to add other payloads or captured context:
i64.parse(text).map_error(ConfigError.Parse)?
print(line).map_error(def [path](cause: IoError) -> AppError: AppError.Write(path, cause))?
The library function
map_error[T, E, G, F: OnceCallable[[E], G]](result: Result[T, E], mapper: F)
has the same behavior; the method is a compiler-provided Result method, not
an import. A class or enum that defines its own map_error keeps it.
The receiver and mapper expressions run once, receiver first, even on success.
The mapper runs exactly once on Err, taking ownership of the original error;
on Ok, the success payload moves through unchanged and the unused mapper is
dropped. Named functions, reusable closures (including mutable environments),
and consuming closures are accepted. Captures and affine payloads follow normal
cleanup rules, including early return while constructing the mapper.
Mapping adds no allocation or boxing. Callback construction, execution, and
destruction retain their ordinary effects. If the callback returns
Result[G, X], the output is Result[T, Result[G, X]]: nothing is flattened
and no cause-preservation policy is invented. Use context
or path_context when their explicit context-allocation policy fits.
? still requires identical error types (except legacy Failure erasure);
the conversion is visible at the map_error call. The mapper must be owned and
take exactly the error type: a reference, a variant with several payloads, or
a callable returning unit is rejected. .map_err(...) is not implemented.
See the lesson.
Worker-reachable calls accept thin named or anonymous function values,
including variant constructors such as ConfigError.Parse, when every function
address of that callable signature in the program passes
worker-effect certification.
The compiler checks every candidate body and its stored types; one safe target
never excuses an unsafe one. A main-thread-only address with the same signature
can conservatively prevent certification. A concrete closure retains its own
body and capture evidence; it cannot bypass the checks either.
Allocation failure
Attaching context never replaces the cause. path_context copies the borrowed
path only on Err; if that copy fails, the context holds
Subject.Unavailable(alloc_error) with the original operation and cause. To
avoid the copy, move an owned path into Context(operation, Subject.At(path), cause).
The Ok path of both helpers, construction, and field access allocate nothing.
describe allocates the cause’s representation, the escaped path, and
intermediate strings, and returns AllocError if any step fails. The context
is unchanged either way.
Description format
operation: cause
operation "escaped path": cause
operation <path unavailable>: cause
The path uses Path.display: printable ASCII is kept, backslash and
double quote are escaped, and every other byte is \xhh, so a description is
one line of ASCII apart from the operation and cause. describe renders the
cause as str.repr does; a native
code appears as IoError.System(2).
A nested context is a cause like any other, so describe shows its structural
representation. Describe the inner context first and pass the text outward:
outer.describe_with(&outer.cause.describe()?)
That gives load settings: read_bytes "missing/settings": IoError.System(2).
An application error enum can match its variants and call the method that fits
each payload. std.fmt.Formattable provides borrowed formatting
for application types. Context[E] supports this protocol when E does:
it emits the operation and subject, then calls render on the typed cause
exactly once. Nested contexts therefore produce nested operation labels without
duplicating the cause. Each nested render counts toward the depth limit.
This conditional method is compiler-supported for the embedded std.error
type; it does not introduce conditional methods for arbitrary generic classes.
An unformattable cause keeps the existing description APIs and structural
entrypoint reporting. describe and describe_with retain their existing
behavior even when the cause has a format method.
Formatting an absent or unavailable subject adds no allocation. A present path
uses the allocating Path.display; failure is returned as
FormatError.Allocation(...), preserving the cause and any
already-written prefix. Operation and cause text are emitted unchanged.
Reports
report flushes standard output, then writes error: , the description, and a
newline to standard error in one write. It returns unit: flush and write
failures are dropped, because standard error is the last channel. The caller
keeps the error and chooses the exit status, normally by returning i32 from
main.
Neither fallback allocates:
| Condition | Standard error |
|---|---|
| The description cannot be allocated | error: operation: no memory to describe the error |
| The description exists but the whole line cannot be allocated | The normal line, written in three pieces |
report_with(description) applies the same rules to a description the caller
built, such as the nested one above; pass the Result unhandled.
Returning Err(context) from main produces the
entrypoint report. It selects the
typed formatting method when the cause is formattable, otherwise renders the
context structurally and shows a path as its byte list. The explicit report
and report_with APIs keep their description-based behavior.
Conventions for standard modules
- An operation returns its precise error type:
IoError,AllocError, or the module’s own enum. It does not attach context, format text, or allocate to describe a failure. - A module enum that can fail from allocation has a variant carrying the
AllocError, asWalkError.AllocationandDataError.Allocationdo. Allocation failure is never folded intoOtherorFailure. - A native code is kept as
IoError.System(code)and never translated to text in the error value. - A composite operation that must say which step or path failed returns
Context[E], or an enum whose variants hold contexts. The operation label is a string literal naming the public operation, such asread_dir. - Failing to allocate context reports the original cause, with
Subject.Unavailable. - Only an application decides to print. Library code returns errors;
reportbelongs at the boundary that owns the exit status.
WalkError.Io predates this module and keeps its own
(path, operation, cause) payload.
Limits
describeandreportrequire a cause typestr.repraccepts. A cause holding recursive data or a generator is rejected where one of them is first called for thatContext[E].- No OS description text is added to a native code.
- The legacy
reportandreport_withmethods usewrite_stderr: a pipe whose reader has gone terminates the process withSIGPIPE. Thestd.reportformatter and entrypoint reporter suppress it and preserve normal cleanup. - A piecewise fallback report is not atomic against concurrent writers.
Subjecthas no network address or other non-path variant.
A Context[E] can cross threads when E can. Every operation takes time linear
in the text it produces. Supported target: x86_64 Linux GNU. The lesson is
Say which operation failed.
Acknowledgements
std.fmt
std.fmt separates user-defined display from debug output and streams UTF-8 to
any static std.io.Writer. It leaves builtin print unchanged.
Formatting policy is checked Plenty; the native primitives only write
floating-point spellings into existing byte storage. There is no runtime
reflection, format-string interpreter, or dynamic writer object.
f-strings compile to calls on
this module, and out.format(f"...") writes one to a formatter out without an
intermediate str.
The same formatter serves std.report: its writer-generic
Formattable.format protocol shares these methods and errors.
stderr_formatter(max_bytes, max_depth) creates a Formatter[Stderr] without
allocating; text, boolean and integer output on that path also needs no heap
storage. Advanced numeric formatting acquires scratch lazily.
import std.fmt
from std.fmt import Formatter
pub class Point:
x: i64
y: i64
pub def display[W](self, out: &mut Formatter[W]) -> Result[(), FormatError]:
out.text("(")?
out.integer(self.x, 0)?
out.text(", ")?
out.integer(self.y, 0)?
out.text(")")
def main() -> Result[(), AllocError | IoError | FormatError]:
print(std.fmt.to_string(&Point(3, -4), 80, 8)?)?
mut out = std.fmt.string_formatter(80, 8)?
out.text("total:")?
out.float_spec(1234.5, &std.fmt.spec(width=10, grouping=",", precision=2, kind="f"))?
print(std.fmt.finish(&out)?)?
Ok(())
(3, -4)
total: 1,234.50
Writers and strings
formatter(writer, scratch, max_bytes, max_depth) consumes a writer and a
Bytes scratch buffer and returns Result[Formatter[W], FormatError].
Scratch must already have length at least 32, not just capacity. Both
limits are NonNegative[i64], so a negative literal is a compile error. Depth
is at most 64, since nested display calls recurse on the native stack. A shorter
scratch or a larger depth traps with std.fmt.formatter: scratch is shorter than 32 bytes or std.fmt.formatter: max_depth is above 64; stderr_formatter and
string_formatter trap the same way for depth. Construction consumes both
arguments even on failure.
The formatter owns them; its public writer field lets the caller flush, close,
or inspect the underlying stream. Formatting never flushes or closes it.
string_formatter(max_bytes, max_depth) creates a growable in-memory cursor and
scratch. Text emissions on this collector stage their complete UTF-8 spelling
in reusable scratch before one cursor write, preserving the output on failed
allocation. finish(&output) validates and copies its current bytes into a str;
the formatter remains usable. Both are fallible. to_string(&value, max_bytes, max_depth) and to_debug(...) construct a formatter, invoke the corresponding
user contract below, and return a string. They return no partial string on error.
Use a formatter directly to inspect partial progress or reuse storage.
All formatting methods use an exclusive receiver and return
Result[(), FormatError]:
| Method | Form |
|---|---|
text(&str) | Exact UTF-8, including embedded NUL |
debug_text(&str) | Double quotes; escapes ", \, LF, CR, TAB; other C0/C1 controls use lowercase \u{hex}; other Unicode is preserved |
ascii_text(&str) | The text with each non-ASCII scalar as lowercase \u{hex}; ASCII is unchanged |
substr(&substr), debug_substr(&substr) | text and debug_text for a substring view, without copying it |
boolean(bool) | True or False |
integer(i64, width: NonNegative[i64] = 0) | Signed decimal, including i64.MIN |
unsigned(u64, base, width: NonNegative[i64] = 0) | Base 2, 8, 10, or 16, lowercase digits, no prefix |
floating(f64, width: NonNegative[i64]) | Shortest round-trip scientific decimal, e.g. 1.25e0, -0e0, inf, -inf, NaN |
Smaller integer/float types can be explicitly converted. Width is a minimum
byte count, padded on the left with ASCII spaces, including any sign.
It never truncates. An unsupported base returns InvalidSpec before output;
convert a computed width with NonNegative.try_from(width)?.
These numeric spellings are ASCII, so byte and scalar widths coincide; no
grapheme or terminal-cell width is promised. Text has no implicit padding.
floating has no precision or locale controls; promoting f32 to f64
spells the promoted value. The spec methods below add Python’s options. There
is no format-spec parser: a Spec is built from its fields, and f-strings
parse their specs while compiling.
Format specs
Spec holds one field’s options as public fields, in this order: fill (one
scalar), align ("", "<", ">", "^", "="), sign ("", "+",
"-", " "), coerce_zero (z), alternate (#), zero (0), width,
grouping ("", ",", "_"), precision (-1 when absent), and kind
(the type letter or ""). spec(...) builds one with keyword defaults, for
example spec(width=8, precision=2, kind="f"). Width and precision are at most
4096.
| Method | Options |
|---|---|
text_spec(&str, &Spec) | Fill, alignment (default left), width; precision truncates. Widths count scalars |
signed_spec(i64, &Spec), unsigned_spec(u64, &Spec) | Types d, x, X, o, b; # prefixes; sign; , (decimal) or _ grouping; 0 grouped zero padding; default right |
signed_spec, unsigned_spec with kind n | As d, without grouping |
char_spec(i64, &Spec) | Kind c: the scalar with that code point; fill, alignment (default right), width, 0 padding; InvalidSpec for a value that is not a Unicode scalar |
float_spec(f64, &Spec), float32_spec(f32, &Spec) | Types e, E, f, F, g, G, %, n (as g, without grouping), or none; precision (default 6 when typed); #; sign; coerce_zero drops the sign of a result whose digits are all zero; grouping; 0 padding |
Their spellings match Python’s format, except that a float with neither type
nor precision uses print’s spelling. A combination the method’s type does not
accept, such as a sign on text or a precision on an integer, returns
InvalidSpec before output. f-strings reject these at compile time.
Spec numbers are spelled in scratch at offset 8. A spelling longer than the
scratch, such as a 64-digit binary integer or 1e300 with f, first grows the
scratch fallibly (Allocation(...)); preallocate scratch to keep
formatting allocation-free. represent(&value, &mut out) writes the str.repr
spelling of any value str.repr accepts, which allocates a temporary string.
A Bytes buffer, bytes literal, or &[u8] view spells as a byte literal
(printing bytes); write hex with
std.encoding.hex_text.
repr_string(&value), debug_string(&text), and ascii_string(&text) return
the str.repr, debug_text, and ascii_text spellings as fallible strings.
A &T parameter cannot take a slice view,
so represent_view(v, &mut out) and repr_view_string(v) do the same for a
&[T]; f-strings call them for view fields. For a &substr, f-strings call
substr, substr_spec(v, &spec) (the text_spec options), and
repr_substr_string(v), the debug_substr spelling as a fallible string.
When a field’s str.repr spelling runs a debug method, the f-string calls
str.repr at its own site and passes the result to repr_text(spelled, &mut out) or repr_result(spelled), which turn AllocError | FormatError into a
FormatError.
An f-string builds its text as Result[str, FormatError] and converts the
result at its site: allocation_only(built) gives Result[str, AllocError]
when no field runs a user method or has a computed spec, and
allocation_or_format(built) gives Result[str, AllocError | FormatError]
otherwise. Both report Allocation(cause) as cause. allocation_only traps on any other
FormatError, which the compiler’s choice rules out.
f-strings check the values of a spec’s nested fields with three functions, each
returning InvalidSpec for a value the spec cannot take: spec_align(&align)
accepts "", <, >, ^, and =; spec_fill(&fill, &align) turns "" into
the default space and otherwise requires one scalar and a non-empty alignment;
spec_count(n) accepts a width or precision in 0..=4096.
Static user contracts and aggregates
A public class or enum opts in by providing either or both public methods:
pub def display[W](self, out: &mut Formatter[W]) -> Result[(), FormatError]:
...
pub def debug[W](self, out: &mut Formatter[W]) -> Result[(), FormatError]:
...
std.fmt.display(&value, &mut out) and debug(...) check the selected method at
each concrete generic instantiation. These helpers use ordinary structural
calls rather than named protocol bounds. Missing/private methods, wrong
signatures, moves, conflicting loans and worker effects get ordinary compiler
diagnostics. to_string/to_debug use these user methods; primitive scalars use
the explicit methods in the table instead of acquiring hidden methods.
An enum method can match its borrowed receiver to render the active payload.
print, str.repr and f-strings call the same methods, as
user formatting methods
describe; each call there starts a fresh depth of 32. Entrypoint error reports
do not call them, and anonymous sums do not automatically forward them to
members.
list_with(&values, &mut out, render) writes [a, b] using a statically typed
callback (&T, &mut Formatter[W]) -> Result[(), FormatError] for each borrowed
element. pair_with(&pair, &mut out, first, second) similarly writes (a, b)
with separate callbacks for a heterogeneous pair. Callbacks choose display or
debug and may recursively use these helpers. Other tuple arities, maps, enums
and classes compose these primitives explicitly; there is no automatic derive.
See the formatting lesson.
Display, debug, list and pair entrypoints each enter one depth level and restore
it on success or error. Depth zero permits scalars but rejects those
entrypoints. Custom recursive code must recurse through these entrypoints or
pair enter()/leave() itself; ordinary direct method recursion cannot be
policed. A custom renderer can still loop or deliberately emit unbounded work.
Errors and partial output
FormatError has InvalidSpec, LimitExceeded, RecursionLimit,
Allocation(AllocError), Output(StreamError), and OutputClosed variants.
String, scratch and sink allocation failures are Allocation(...).
Terminal stderr errors set has_failed(); later emissions on that formatter
return OutputClosed without writing. Other destinations retain their writer’s
recovery policy: a writer error does not make the formatter sticky. Allocation
and byte/depth limit failures do not set has_failed(). completed() is the exact count accepted by the
writer since construction, including a successful prefix before any error.
The writer’s positive short counts are retried, Interrupted is retried, zero
is WriteZero, and negative/oversized counts are InvalidProgress. An Err
must mean that call accepted no bytes, as required by std.io. There is no
rollback or implicit retry of an entire failed formatting operation. Retrying
it can duplicate its successful prefix. A partial OS write can split UTF-8;
finish then reports Bytes(InvalidUtf8) until valid text is restored.
The budget for text is checked for the whole UTF-8 input before output;
other emissions check each scalar, ASCII byte, or float spelling.
Limit errors never exceed it; previous output remains. Budgets count UTF-8 bytes,
escapes, delimiters and padding, not input characters. Direct writes through the
public writer field are outside the counter/budget contract. Callers must not
alter its position or contents during a formatting operation.
Formatting internals use explicit unchecked: regions for arithmetic already
bounded by validated output budgets, scratch positions, radix digit counts,
format widths/precision or recursion limits. Unrepresentable user input does
not acquire a hidden arithmetic trap: existing InvalidSpec, LimitExceeded,
allocation and output errors retain their contracts. User display/debug methods
still have their own declared arithmetic boundaries, independent of formatter
implementation modes.
Cost and availability
After construction, formatting scalars performs no allocation when the
writer does not allocate. The caller can preallocate a Cursor buffer to make
this true for in-memory streams. Caller-provided scratch is reused for scalar streaming; string
formatters may grow scratch to stage a whole text emission. String output grows explicitly and finish copies the complete bytes. A custom
renderer/callback can allocate, and its ordinary effects are not suppressed.
Text is linear in input bytes plus output bytes, integer work is bounded by 64
digits plus padding, and f64 conversion has fixed bounded scratch work plus
padding. A spec float costs time linear in its precision and magnitude digits. A padded float may be converted twice because padding reuses scratch.
List work is linear in element count plus callback work. At most one Unicode
scalar (four bytes), one ASCII byte, or one float spelling (at most 32 bytes) is
offered per write on a caller-provided writer; wrap a syscall writer in
BufWriter to batch these calls. A string formatter submits a whole text
emission to its cursor. Direct stderr text submits the unconsumed UTF-8 suffix.
Memory is caller scratch plus writer state and bounded recursive call frames.
Formatters are affine because they own Bytes; copying is rejected. Loans and
worker transfers follow the underlying writer’s normal ownership/effect rules.
The embedded module works offline with a relocated matching toolchain on the
supported x86-64 Linux GNU target and adds no external library dependency.
Acknowledgements
- Format specs follow Python’s format specification mini-language.
std.fs — directory enumeration
Import std.fs and pass a borrowed Path.
Metadata and links covers native-byte symlink creation and
canonicalize resolution as well as file observations.
For owned temporary resources, copy/move/remove and bounded glob filtering,
see rooted helpers and file/tree copying. The home and XDG base directories are in
std.fs.dirs. For change notifications, see
std.fs.watch. For cooperating-process locking, see
std.fs.lock.
read_dir(&path) -> Result[ReadDir, IoError] opens one directory immediately.
ReadDir is affine and owns a lazy native stream. next() requires a mutable
owner and returns Result[Option[DirEntry], IoError]: Some is one entry,
Nothing is EOF or an already closed stream, and Err is an advancement failure
after opening. Any advancement error, including allocation failure, terminates
and closes the stream. Exhaustion also closes it before returning EOF.
import std.fs as fs
import std.path as path
def main() -> Result[(), AllocError | IoError]:
write_text("a.txt", "alpha")?
write_text("b.txt", "beta")?
root = path.from_text(".")?
mut names: list[str] = []?
for result in fs.entries(fs.read_dir(&root)?):
entry = result?
name = entry.file_name()?.to_text()?
if name.endswith(".txt"):
names.append(name)?
names.sort()
print(names)?
Ok(())
["a.txt", "b.txt"]
close() -> Result[(), IoError] explicitly relinquishes the stream; repeated
close is harmless, including after a close error. closed() -> bool observes
the state. Drop closes any remaining stream without allocating or reporting a
close error. Scope exit, return, ?, and dropping a suspended consuming generator
release their owned stream normally.
entries(read_dir(&root)?) consumes the owner and returns
Generator[Result[DirEntry, IoError]]. Handle each item’s Result explicitly.
The existing generator/iteration system supplies deterministic cleanup on an
early break; no separate fallible for protocol is introduced. Breaking out
of a manual while leaves its named ReadDir alive until close or scope exit.
Only structural . and .. are excluded. Hidden names remain visible, order
is unspecified native order, and enumeration is not a snapshot. Sorting and
ignore rules belong to callers. Additions/removals during enumeration can affect
which entries appear.
DirEntry operation | Result |
|---|---|
file_name() | Fallible detached Path containing the exact native name |
path() | Fallible detached Path joined to the supplied root |
file_type() | Result[FileType, IoError], without following a link |
metadata() | Fresh Result[Metadata, IoError], without following a link |
Entries own copied path bytes and cached native type information; they never
borrow libc’s reused readdir buffer or retain a directory descriptor. Keeping
entries in a list survives advancing, closing, or dropping the stream.
Name/path copies can fail without invalidating the entry. Relative roots remain
relative, and the joined path is not canonicalized.
Type-only queries use d_type when available. An unknown native type triggers a
non-following metadata lookup and propagates any lookup error. FileType is
File, Directory, Symlink, or Other; sockets, FIFOs and devices are Other,
never regular files. Cached native types can become stale; entry metadata is
fresh on every call. See metadata and links.
Python’s os.scandir and Rust’s read_dir/DirEntry provide the API baseline.
The runtime follows their Unix errno-reset/EOF distinction but copies detached
entry data fallibly instead of adopting Rust DirEntry’s descriptor retention.
Native enumeration currently supports 64-bit Linux; unsupported hosts return
IoError.Unsupported. See walking for tree traversal.
Acknowledgements
- Directory API modelled on Python’s
os.scandirand Rust’sstd::fs::read_dir. - Uses the Rust
libccrate for target-specific Linux filesystem syscalls and flags (MIT OR Apache-2.0).
Metadata and links
In std.fs, metadata(&path) follows symbolic links, while
symlink_metadata(&path) inspects the link itself. Both return
Result[Metadata, IoError]; missing paths, inaccessible paths and broken link
targets preserve their OS error. DirEntry.metadata() follows the latter rule.
No primitive converts an error to False. Broken links remain enumerable and
inspectable with non-following operations.
import std.fs as fs
import std.path as path
def main() -> Result[(), AllocError | IoError]:
write_text("notes.txt", "hello")?
notes = path.from_text("notes.txt")?
info = fs.metadata(¬es)?
print(info.is_file(), info.is_dir(), info.size())?
current = path.from_text(".")?
print(fs.metadata(¤t)?.file_type())?
missing = path.from_text("missing.txt")?
match fs.metadata(&missing):
case Ok(_):
print("found")?
case Err(error):
print(error)?
Ok(())
True False 5
std.fs.FileType.Directory
IoError.System(2)
Metadata is an affine, detached observation, not an open file or a promise about a later operation. No metadata cache is kept on an entry. Request another path-level observation to refresh it; replacing or changing a file can make earlier data stale.
file_metadata(&stream) -> Result[Metadata, IoError] observes an open
FileStream itself (statx with AT_EMPTY_PATH), not whatever its path names
now. A file opened through a Root keeps that confinement:
renaming or replacing the path afterwards cannot redirect the observation.
| Metadata operation | Value |
|---|---|
file_type() | FileType.File, .Directory, .Symlink, or .Other |
is_file(), is_dir(), is_symlink() | Predicates on that observation |
size() | u64 size in bytes, also for links and other native types |
permissions() | Permissions(readonly: bool, unix_mode: u32) |
modified(), accessed(), created() | Option[Timestamp] |
unix_changed() | Unix status-change Timestamp, not creation time |
unix_uid(), unix_gid() | u32 Unix identity fields |
unix_device(), unix_inode() | u64 Unix directory-identity fields |
Timestamp is std.time.Timestamp; import the name from
std.time. It has signed i64 seconds since the Unix epoch and u32 nanoseconds
in [0, 1_000_000_000). Negative times use floor seconds: half a second before
the epoch is (-1, 500_000_000). Nothing means the filesystem did not supply
that timestamp. Unsupported creation time is never filled with Unix change time.
Metadata getters do not allocate. readonly means no Unix write mode bits;
it does not account for ACLs, privileges or mount policy and does not establish
that a later operation will succeed.
read_link(&path) -> Result[Path, IoError] returns the stored target exactly,
including a relative target or invalid UTF-8. It does not resolve it, make it
absolute or require that the target exists. Buffer growth and copying are
fallible; a full native buffer is retried rather than silently truncated.
Creating and resolving links
symlink(target: &Path, link: &Path) -> Result[(), IoError] creates a new
symbolic link containing the target’s exact bytes. A relative target is
interpreted relative to the link’s parent when followed, not relative to the
caller’s working directory. The target need not exist. An existing destination,
including a dangling link, causes an error without replacing it. Parent
components of link follow ordinary filesystem resolution; this is not a
rooted no-follow operation. Use Root and TempFile for their
separate confinement and publication contracts.
canonicalize(path: &Path) -> Result[Path, IoError] returns a new absolute
native-byte path after resolving symbolic links, . and ... Components are
resolved in filesystem order: link/.. refers to the target’s parent. Every
component must exist; dangling links and missing components return the native
error, a non-directory intermediate component or a trailing slash on a file
returns ENOTDIR, and loops or excessive link traversal return ELOOP.
Empty paths fail. No text decoding or lossy replacement occurs.
Both functions borrow their inputs, are usable on worker threads, and resolve
relative input paths against the process working directory. Embedded NUL bytes
return IoError.InvalidInput. On the current 64-bit Linux backend each input
and the canonical output must fit in PATH_MAX including the terminating NUL
(4095 content bytes); excess returns IoError.System(ENAMETOOLONG). Input
lengths are checked before copying. Native path buffers are bounded stack
storage; symlink needs no Plenty heap allocation. Canonical output allocation
is fallible and reports IoError.Data(DataError.Allocation(...)); OS/C-library
failures retain IoError.System(errno). Unsupported hosts return
IoError.Unsupported.
Canonicalization is a transient observation, not a security boundary or a
snapshot. Concurrent rename, removal or replacement can change resolution or
invalidate the returned name immediately. A later open need not access the same
object; use held handles and descriptor-rooted operations when identity matters.
The narrow native wrappers use symlink and caller-buffer realpath so libc
implements platform link traversal, directory checks and error behavior instead
of duplicating that resolver in Plenty. No realpath(NULL) allocation escapes
into the language.
The baseline is Rust’s explicit metadata/symlink_metadata split and
non-following DirEntry behavior. This differs from Python’s default followed
DirEntry predicates/stat; those defaults are not mixed. The native backend uses
Linux’s stable 256-byte statx ABI, also used by Rust’s Unix fs backend, to avoid
std’s infallible long-path CString allocation. It requires 64-bit Linux and
libc/kernel statx support; absent kernel support or required basic fields returns
IoError.Unsupported. Native layouts and descriptors are private.
Acknowledgements
-
Uses the C library’s
symlinkandrealpathfor native link creation and filesystem resolution. -
Follows Rust’s
metadata/symlink_metadatasplit and its Unixstatxbackend.
To update ordinary permission bits, use the held-handle operations
Root.set_unix_mode and set_file_unix_mode.
Changing a Permissions observation’s fields does not change the filesystem.
Walking a tree
std.fs.walk(&root, top_down) -> Result[Walk, AllocError] constructs an explicit
Plenty walker over directory enumeration. It copies the root
and allocates its pending stack, but opens no directory until advancement.
walker.next() -> Result[bool, WalkError] updates walker.current when it returns
True. False means exhaustion or explicit close. walker.close() clears pending
work and current entry lists without allocation and is repeatable.
import std.fs as fs
import std.path as path
def main() -> Result[(), AllocError | IoError | fs.WalkError]:
current = path.from_text(".")?
root = fs.root(¤t)?
root.create_dir(&path.from_text("site")?)?
root.open_dir(&path.from_text("site")?)?.create_dir(&path.from_text("posts")?)?
write_text("site/index.html", "home")?
write_text("site/posts/first.html", "hello")?
top = path.from_text("site")?
mut walker = fs.walk(&top, False)?
while walker.next()?:
print(walker.current.path.to_string()?, len(walker.current.files))?
Ok(())
site/posts 1
site 1
current is a WalkDir with path: Path, directories: list[Path], and
files: list[DirEntry]. Directories contains only non-followed directory paths;
files contains every other native type, including directory links, dangling
links, FIFOs and sockets. Non-directory does not mean regular file.
In top-down mode, next yields the parent before any child is opened. Remove or
reorder current.directories before advancing again; the walker descends only
into the remaining paths, in their resulting order. It consumes that list on
the next call. Thus pruning happens before even attempting access to an
unreadable child. In bottom-up mode, children have already been visited when
the parent is yielded, so editing its directories cannot prune prior descent.
Native sibling order remains unspecified; there is no implicit sorting.
The walker never follows discovered directory links. An explicitly supplied root can resolve through a link, just as ordinary path lookup can. This initial walker has no follow-links option or visited-identity set; an ancestor link therefore does not cause recursive descent. Relative roots retain their spelling and the walker never changes the working directory.
WalkError.Io(path, operation, cause) retains the attempted Path, a fixed
operation label (read_dir, next, file_type, or entry.path), and concrete
IoError. The failing frame has already been removed. Handle Err and call next
again to skip that subtree and continue pending siblings; propagate with ?
to stop and drop the walker. A partially scanned directory produces no partial
row. WalkError.Allocation(error) reports fallible stack/list/path storage;
no completeness guarantee applies after an allocation failure. If allocating
the error’s entry path also fails, the context reports the containing root and
entry.path instead.
Traversal uses a Plenty list of frames and no recursive native call stack.
At most one directory stream is open during a scan, and none is retained between
next calls. A row allocates one directory’s lists. Top-down storage includes
pending sibling paths along the active traversal depth and the current row;
bottom-up additionally retains parent rows along that depth. It streams the
tree rather than collecting the entire tree in advance; a very wide directory
still needs fallible per-directory storage. Pending-stack reservation counts
only the frames about to be appended, not frames already held. An
unrepresentable additional frame count returns
WalkError.Allocation(AllocError.CapacityOverflow) before appending.
Python os.walk supplies top-down/bottom-up and pruning precedent. Grouping
directory symlinks with files follows pathlib.Path.walk with link following
disabled. The explicit next/error continuation is Plenty’s adaptation of an
error callback; failures are visible by default. The implementation is embedded
Plenty source in stdlib/std/fs.plenty, not a separate native traversal engine.
Acknowledgements
- Follows Python’s
os.walkandpathlib.Path.walk.
Rooted filesystem helpers
std.fs mutation helpers operate on an owned Root, opened by
root(&path) -> Result[Root, IoError]. The handle anchors operations to that
directory even if its pathname changes. Opening rejects a final symlink,
including a path with trailing slashes. Ancestor components of the initial
root path are resolved normally: selecting that root is the caller’s trust
decision, not a sandbox or a canonicalization operation.
import std.fs as fs
import std.path as path
def main() -> Result[(), AllocError | IoError]:
current = path.from_text(".")?
root = fs.root(¤t)?
drafts = path.from_text("drafts")?
final = path.from_text("final")?
root.create_dir(&drafts)?
root.move(&drafts, &final)?
reports = root.open_dir(&final)?
reports.create_dir(&path.from_text("2024")?)?
mut names = reports.names()?
while True:
match names.next()?:
case Some(name):
print(name.to_string()?)?
case Nothing:
break
Ok(())
2024
Every child name is a lossless Path containing one nonempty component.
Slash, NUL, . and .. are rejected. Child directory/file opens use
O_NOFOLLOW; symlinks and symlink cycles cannot escape through these opens.
Mounted directories are not a boundary. A directory moved after opening stays
accessible through its handle. These rules do not provide isolation from a
process with permission to rename entries in the same tree.
| Root method | Behavior |
|---|---|
open_dir(&name) | New independently owned Root; final symlink refused |
read_file(&name) | Read-only std.io.FileStream; regular files only |
create_dir(&name, unix_mode=0o700u32) | Exclusive directory creation; mode restricted by umask |
create_file(&name, unix_mode=0o600u32) | Exclusive regular-file creation; owned read/write FileStream, mode restricted by umask |
set_unix_mode(unix_mode) | Change ordinary permission bits on this held directory |
remove_file(&name) | Unlink one non-directory, including a symlink itself |
remove_dir(&name) | Remove one empty directory; no link following |
move(&source, &destination) | Same-root atomic renameat2(RENAME_NOREPLACE) |
names() | Owned lazy Names; next() yields Result[Option[Path], IoError] |
Move refuses every existing destination, including a dangling link. It moves
a source symlink itself; there is no replacement or cross-root/cross-device
copy-and-delete fallback. Removal has ordinary unlink/rmdir permissions and
race semantics. Names.close() is fallible and repeatable. EOF and errors close
the stream; drop closes it as well. Names are detached, native-order byte
values, including hidden names. No metadata observation through an unanchored
joined path is exposed by this interface.
Root, names, streams and temporary owners are affine. An owned Root or
FileStream can move to a worker; creation and permission operations are
handle effects permitted there. Sharing a handle by loan still follows the
worker resource rules.
Shared Root loans allow pathname operations within a thread, since they do not
mutate a cursor. Filesystem state is external mutable state; this does not synchronize
concurrent processes. Every syscall retains its native IoError; buffer and
owner allocation is fallible. Paths require O(name length) temporary storage.
Native primitives make narrow OS requests; traversal, budgets, retries and
temporary-resource lifecycle are ordinary checked Plenty code.
Current target: x86_64 Linux GNU. No new dependency or linker requirement is introduced. The matching installed compiler embeds the module and runtime; imports work offline after relocation. See temporary resources and bounded copying, deletion and globbing.
Private creation and permission changes
Use Root.create_file for a named file that must not already exist. It
returns Result[FileStream, IoError], opened for reading and writing; close it
through FileStream.close() or let its owner drop. Allocation for the path and
owner happens before creation. Neither allocation failure nor an existing
entry leaves a new file. Every existing entry, including a dangling symlink,
causes an error without truncation, replacement or permission changes. A
successful creation remains named after close, drop or a later write failure.
Use anonymous staging when partial contents must stay unnamed.
create_dir returns Result[(), IoError] and creates only the named directory,
not its parents. Existing directories and symlinks fail too. Open the result
with open_dir when a held directory handle is needed. Creation and this
subsequent open are not atomic; the parent must be trusted against concurrent
entry replacement, as for the other rooted helpers.
Both creation methods accept unix_mode: u32. The default modes (0600 for a
file, 0700 for a directory) prohibit group/other access from the instant of
creation. The mode goes directly to openat/mkdirat, with no create-then-chmod
window. Ordinary Unix umask/default-ACL rules apply: requested bits are an
upper bound, and the process umask can remove them when no default ACL applies.
The library never reads or changes the process-global umask. Directory setgid
inheritance and existing filesystem ACL policy are controlled by the OS.
To change permissions later, call Root.set_unix_mode(unix_mode) on a held
directory or fs.set_file_unix_mode(&stream, unix_mode) on a held FileStream.
Both return Result[(), IoError], perform no input-sized allocation and call
fchmod on the descriptor. Umask does not restrict these updates. They neither
reopen a pathname nor follow a newly substituted symlink. The operation changes
the inode shared by all of its hard links, even after renaming or unlinking.
It does not revoke already-open descriptors, change ownership, or promise an
ACL-free filesystem. The OS enforces ownership/capability checks and ACL effects.
Modes outside 0o000u32..0o777u32 return IoError.InvalidInput before any
filesystem change: file-type bits, sticky and set-ID bits are deliberately
unsupported. Modes are not silently masked. OS failures retain their native
IoError.System code; closed streams return IoError.Closed. Creation can
also return an allocation error through IoError.Data(DataError.Allocation(...)). These APIs have
the same 64-bit Linux scope as the rooted helpers; unsupported hosts return
IoError.Unsupported. Native code is limited to descriptor creation and
fchmod, because these operations require OS syscalls; mode validation and
public policy remain Plenty code.
Temporary resources
For ordinary scratch storage with with support, prefer
std.tempfile. These lower-level owners provide rooted staging
and explicit publication.
temporary_file(&root) -> Result[TempFile, IoError] creates an unnamed
O_TMPFILE inode in the root’s filesystem, mode 0600 restricted by umask.
Its public stream: std.io.FileStream is readable and writable; seek back
to read what was written. publish(&name) adds a
name with atomic no-replace linkat; existing files, directories and dangling
links are refused. A successful publication can occur once. It leaves the
stream open; close() reports close errors, and ordinary drop closes it.
Closing/dropping before publication releases the unnamed inode without an
unlink-by-name race. After publication the name persists.
import std.fs as fs
import std.io
import std.path as path
def main() -> Result[(), AllocError | IoError | std.io.StreamError | std.io.Partial]:
current = path.from_text(".")?
root = fs.root(¤t)?
mut staged = fs.temporary_file(&root)?
std.io.write_all(&mut staged.stream, &b"complete contents")?
staged.stream.sync()?
staged.publish(&path.from_text("config.txt")?)?
staged.close()?
print(read_text("config.txt")?)?
Ok(())
complete contents
remove_published(&name) -> Result[(), FsError] explicitly removes a published
name after comparing it with the still-open stream’s device and inode. A missing
name succeeds; a different regular file returns FsError.Replaced, and symlinks
are refused. The stream remains open. Publication is still allowed only once.
As with directory cleanup, identity checking and removal are not atomic against
hostile concurrent renames. This operation supports std.tempfile ownership.
Publication uses /proc/self/fd so unprivileged users need no special
capability. The filesystem must support O_TMPFILE, and procfs must be
available for publication. Unsupported filesystems and unavailable procfs
produce IoError (normally a native System code); there is no named-file
fallback. A successful publish is not a durability promise; call
stream.sync() before publishing if needed. Directory durability is not
provided here. Temporary creation itself copies no metadata.
copy_file and copy_tree can preserve ordinary permission
bits and access/modification times before completing their copy operation.
temporary_dir_named(&root, &name) exclusively creates a 0700 directory and
returns Result[TempDir, FsError]. A collision is reported and never removed.
temporary_dir(&root, &prefix, attempts: Positive[i64]) appends 32 lowercase
hex digits from 16 OS secret bytes, retrying only EEXIST up to attempts. An
attempts above 1024 traps. Prefixes are at most 200 bytes without slash/NUL; no UTF-8 conversion is imposed.
Entropy errors are reported without weakening the name source.
TempDir.name() and root() lend its name and directory handle. Explicit
cleanup() removes only an empty directory and reports errors. It is
idempotent after success. keep() disarms cleanup; the directory remains.
The destructor attempts the same empty-only cleanup and ignores errors.
This low-level destructor never recursively deletes contents. Fatal traps/process termination
do not unwind; explicit cleanup is required when its success matters.
Cleanup’s native name conversion can allocate: if allocation remains exhausted
during drop, even an empty directory can remain. File/directory handles still
close without allocation. Creation reserves names and owner storage before
mkdir, so allocation failure during construction leaves no directory behind.
cleanup_tree(max_entries: Positive[i64], max_depth: Positive[i64], &cancel) -> Result[(), FsError] is an
explicit recursive alternative, using the bounded deletion
budgets. It checks identity before traversing the held directory and again
before removing its entry. Successful cleanup disarms the owner. Failure can
leave partial cleanup and leaves it armed. std.tempfile.TemporaryDirectory
uses this operation for its best-effort destructor.
Temporary directories require a parent whose entries other actors do not
rename while the owner is live. Cleanup compares the held directory’s device
and inode to the named directory and reports FsError.Replaced for a changed
identity. A missing name or symlink is an I/O error. This catches replacements
already visible at the check; Linux has no compare-inode-and-unlink syscall,
so the check and rmdir are not atomic against a concurrent rename. Empty
replacement directories can still be removed by a racing actor. Nonempty
replacements are never traversed by the empty-only cleanup. A child-open failure
after mkdir attempts empty-only rollback under the same parent-trust contract.
Temporary names/owners and copies are allocated fallibly. A temporary file holds its file descriptor and a duplicate root descriptor. A temporary directory holds its directory and parent descriptors plus its name bytes. See root rules and bounded operations.
Acknowledgements
Bounded copy, deletion and glob filtering
copy_file(&root, &source, &destination, max_bytes: NonNegative[i64], &cancel, preserve=CopyMetadata.ContentsOnly) copies one
regular file to a new name in the same root and returns Result[i64, FsError].
copy_from(&mut reader, &root, &destination, max_bytes: NonNegative[i64], &cancel) accepts any
checked std.io.Reader. Both stage into an anonymous temporary file and
publish only after EOF, flushing, and a final cancellation observation.
They never replace an existing name. copy_file optionally preserves selected
metadata as described in copy metadata; copy_from has no
filesystem metadata source. Recursive copying
uses the same policy with explicit tree budgets and partial-output errors. Limits, cancellation, read/write errors,
allocation failures and publication errors leave no partial destination name.
Unnamed partial data is released by descriptor destruction, not path unlink.
import std.fs as fs
import std.path as path
from std.sync import CancellationToken
def main() -> Result[(), AllocError | IoError | fs.FsError]:
write_text("report.txt", "quarterly totals")?
current = path.from_text(".")?
root = fs.root(¤t)?
cancel = CancellationToken()?
source = path.from_text("report.txt")?
backup = path.from_text("report.bak.txt")?
print(fs.copy_file(&root, &source, &backup, 4096, &cancel)?)?
pattern = path.from_text("report*")?
matches = fs.glob(&root, &pattern, 100, 10, 10000)?
print(len(matches))?
Ok(())
16
2
The loop uses one 8192-byte buffer, retries interrupted reads, checks reader
progress, and checks cancellation before and after each read and before
publication. Writes use std.io.write_all. Cancellation is cooperative,
cannot interrupt an in-progress syscall/write-all, and may race a successful
publication after the final check. Exceeding the byte bound reports Limit. One read of up to 8192 bytes can consume beyond the
bound before rejecting it. A concurrently changing source is not a snapshot.
The count is bytes published. Costs are O(bytes) time and O(8192 + path lengths)
scratch storage, without a whole-file allocation. A successful result does not
promise durable storage. See temporary-file requirements.
remove_tree(&root, &name, max_entries: Positive[i64], max_depth: Positive[i64], &cancel)
explicitly removes a child directory tree. A max_depth above 64 traps. The root directory counts as one entry and one level. Each
encountered child consumes the shared entry budget. Depth is checked before
opening a recursive frame. File/link unlinking and empty-directory removal
use the anchored parent handle. Links (including cycles) are unlinked and
never descended into; the initial name must itself be a real directory.
Deletion is not transactional: cancellation, limits, permissions, concurrent changes or I/O errors can leave a partially removed tree. It checks cancellation between entries. Descent keeps O(depth) directory descriptors and O(depth * name length) path storage, without materializing the tree. It never accepts root/dot/dot-dot names. The initial root’s ancestors, mount points and rename races have the Root contract; do not use this as an adversarial sandbox. A directory identity check precedes the final rmdir, with the same non-atomic race limitation as temporary-directory cleanup. Other processes must not relocate entries during destructive traversal.
glob(&root, &pattern, max_entries, max_matches, max_steps), each budget a
NonNegative[i64], returns
Result[list[Path], FsError] for names directly inside the root. * matches
any byte sequence, ? one byte, and every other byte is literal (including
brackets, backslash and leading dot). Slash/NUL and patterns over 4096 bytes
are invalid; there is no recursive **, character class, escaping, sorting or
Unicode normalization. Consecutive stars retain their ordinary star meaning.
Invalid UTF-8 names remain lossless.
Entry, result and matching-step budgets are independent limits.
Each examined entry and each matcher iteration consumes its budget; exceeding
one returns Limit, releasing partial results. Greedy star backtracking takes
O(pattern length * name length) worst-case time, bounded by max_steps across
the entire scan. Storage is O(returned name bytes) plus one streaming entry.
Enumeration is native order, includes hidden names, and is not a snapshot.
FsError distinguishes Io, Stream, Allocation, Entropy, InvalidInput,
Limit, Cancelled, Replaced, Collisions, UnsupportedType, Overlap
and InvalidLimits. Basic Root methods preserve
IoError directly. Drop is best-effort cleanup, never proof of completion.
File metadata and recursive copying
All operations here are rooted, component-only, no-follow operations on 64-bit Linux. They retain the Root trust and rename contract: concurrent renames can invalidate identity checks, and these operations are not an adversarial sandbox. Filesystem capabilities, mount topology and permissions remain subject to the operating system. The implementation is in Plenty over native handle observation and metadata syscalls.
Copy metadata
CopyMetadata has two modes:
ContentsOnly, the default: file data only; new files use mode 0600 restricted by umask and directories use 0700 restricted by umask.PermissionsAndTimes: strictly attempt the source’s ordinary permission bits (mode & 0o777) and access/modification timestamps. Read the source metadata from its opened handle before reading contents or enumerating a directory. Apply timestamps and then permissions to the destination handle. Directory metadata is applied after children, so copying a read-only source directory does not make its destination unwritable before the children are copied.
The second mode excludes ownership, setuid/setgid/sticky bits, ACLs, extended
attributes, creation time and inode change time. It does not claim Python
copy2 parity. Destination ACLs/xattrs inherited from its filesystem are not
copied from the source or stripped by this API. Timestamp precision and permission
semantics depend on the destination filesystem; syscall success is the guarantee,
not preservation of precision that filesystem cannot represent. A missing source
timestamp or a failed metadata syscall returns an error; there is no silent
best-effort downgrade. The observation predates copy reads that may update source
atime. Changes to the source after observation are not reflected consistently:
this is not a filesystem snapshot.
copy_file(..., &cancel, preserve=CopyMetadata.PermissionsAndTimes) retains
its anonymous staging and no-replace contract. Metadata failure
leaves no published destination. Existing calls without preserve retain their
contents-only behavior. The optional named policy follows cancellation to preserve
that call shape; there is no required positional boolean. copy_from accepts an
arbitrary reader and therefore has no metadata policy.
Recursive copying
copy_tree(source_parent: &Root, source: &Path,
destination_parent: &Root, destination: &Path,
limits: &CopyLimits, cancel: &CancellationToken,
preserve: CopyMetadata = CopyMetadata.ContentsOnly)
-> Result[i64, TreeCopyError]
The source must name a real directory. The destination must be a new component;
existing files, directories and dangling symlinks are refused. Source and target
parents may differ, including different filesystems. An opened source directory
cannot contain the destination parent: a bounded descriptor-based ancestor walk
rejects that overlap, including the same directory opened through another root.
An existing destination that contains the source is refused by exclusive creation.
The traversal also compares source directory identities with the original
destination parent before descent. This catches a bind mount of a source
descendant whose namespace parents hide that ancestry; it reports an Entry
overlap error after destination creation, leaving partial output. No destination
parent is descended into or recursively copied into itself.
The ancestry walk holds at most two transient handles and stops after 4096 levels;
a longer or changing ancestry produces an I/O invalid-input error. Mount namespace
changes and hostile renames are outside the Root stability contract.
Only real directories and regular files are supported. Symlinks, including
relative, dangling and cyclic links, and special files return
FsError.UnsupportedType; no link is recreated or followed. Native byte names,
hidden entries, empty directories and binary contents remain lossless. A type
observation is followed by a no-follow typed open. Concurrent replacements can
cause an error or change which same-type inode is copied; no snapshot is promised.
Enumeration is streaming and native-order, not a materialized/sorted tree.
copy_limits() returns a CopyLimits with public fields:
| Field | Default | Contract |
|---|---|---|
max_entries | 10000 | Source root and every encountered child count once; positive |
max_depth | 32 | Root is level 1; valid range 1–64 |
max_bytes | 67108864 | Total bytes successfully copied across all files; nonnegative |
max_open_handles | 128 | Conservative operation-owned descriptor budget; at least 11 |
Invalid limits fail before destination creation. Descent additionally requires
3 * level + 8 <= max_open_handles, covering source/destination directories,
streaming enumeration and file staging; caller-owned handles are outside this
budget. Budgets are shared across the traversal. Exceeding an entry, depth,
handle or byte budget reports FsError.Limit. The returned count is copied file
data bytes, with zero for an empty tree. Files are transferred with one shared
8192-byte buffer, so time is O(entries + bytes) and storage is O(8192 + depth *
name length), with O(depth) descriptors. Individual names are bounded by the
native directory-entry limit. An over-limit file can consume one read beyond
its remaining allowance before being rejected, without publishing that file.
Cancellation is checked before creation, between entries, during file transfer,
and after traversal. It is cooperative and cannot interrupt a syscall or
write_all; completion may race cancellation after the final observation.
No durability guarantee is made.
Partial output and recovery
TreeCopyError.Setup(FsError) means the destination was not created by this
operation. TreeCopyError.Partial(FsError) means its newly created destination
may remain with a partial tree. TreeCopyError.Entry(name, depth, cause)
also means partial output; it transfers the immediate failing child name and its
1-based tree depth, preserving the cause without allocating while reporting an
error. It does not carry the full relative path. Errors while enumerating or
finishing a child directory identify that directory; failures on the source
root itself use Partial. Earlier copied files remain complete; the current
file is staged anonymously and is absent on failure before publication. Empty
or populated directories may remain, and metadata can be partly applied to them.
The caller retains both destination root and name and can inspect the tree or
explicitly call remove_tree with separate cleanup budgets and a fresh token.
There is no automatic recursive cleanup that could hide a second failure, erase
progress, or pretend rollback is guaranteed. Concurrent actors must not replace
or mutate the destination while copying or cleaning it. An error opening the
newly created destination is also Partial.
import std.fs as fs
import std.path as path
from std.sync import CancellationToken
def main() -> Result[(), AllocError | IoError | fs.FsError | fs.TreeCopyError]:
root = fs.root(&path.from_text(".")?)?
source = path.from_text("assets")?
root.create_dir(&source)?
write_text("assets/icon.bin", "binary data")?
destination = path.from_text("assets-backup")?
limits = fs.copy_limits()
cancel = CancellationToken()?
print(fs.copy_tree(&root, &source, &root, &destination, &limits, &cancel)?)?
print(read_text("assets-backup/icon.bin")?)?
Ok(())
11
binary data
Owned names, metadata observations, buffers, and native owners allocate fallibly. Allocation failure preserves the same file-publication and tree-partial contracts; held descriptors close on ordinary return/drop without allocating. Fatal traps or termination do not unwind. Roots must remain accessible under the selected permissions for any later cleanup; preserving read-only directory permissions can require the caller to restore writable permissions outside this API.
Acknowledgements
- Python’s
shutil.copytreeandcopy2provide the API precedent. The Plenty implementation uses explicit budgets, rooted no-follow operations and a narrower declared metadata policy; no Python source code is shipped.
User directories
import std.fs.dirs finds the user’s home directory and the per-user base
directories where a program keeps its configuration, caches, data, and state.
It is written in Plenty over std.env and returns
Path values.
| Function | Variable | Otherwise |
|---|---|---|
home_dir() | HOME | Nothing |
config_dir() | XDG_CONFIG_HOME | $HOME/.config |
cache_dir() | XDG_CACHE_HOME | $HOME/.cache |
data_dir() | XDG_DATA_HOME | $HOME/.local/share |
state_dir() | XDG_STATE_HOME | $HOME/.local/state |
Each returns Result[Option[Path], IoError]. The rules follow the
XDG Base Directory specification
and the Rust dirs crate on Linux:
- An empty variable counts as unset.
- An
XDG_*value is used only when it is absolute. The specification calls a relative value invalid, so it is ignored and the default belowHOMEis used. HOMEis used as set, without requiring an absolute path.- With no usable
XDG_*value and noHOME, the result isNothing.
import std.fs.dirs
from std.path import from_text
def main() -> Result[(), IoError]:
match std.fs.dirs.config_dir()?:
case Some(config):
settings = config.join(&from_text("myapp/settings.toml")?)?
# For example /home/ada/.config/myapp/settings.toml
print(settings.is_absolute())?
case Nothing:
print("no configuration directory")?
Ok(())
True
Paths keep the variable’s exact bytes, so a directory name that is not UTF-8
round-trips. Results are not cached: each call reads the environment again.
Nothing is created or checked; a returned directory may not exist yet. Create
it with the rooted helpers when the program first writes there.
Allocation failure is IoError.Data(DataError.Allocation(...)).
Limits
- x86-64 Linux GNU only, like
std.env; elsewhere each call reportsIoError.Unsupported. - No password-database fallback: when
HOMEis unset,dirsconsultsgetpwuid_r, buthome_dir()returnsNothing. - No runtime, executable, or system-wide (
XDG_CONFIG_DIRS,XDG_DATA_DIRS) directories.
Acknowledgements
- Follows the XDG Base Directory specification as the Rust
dirscrate applies it on Linux.
std.fs.lock — advisory file locks
std.fs.lock coordinates cooperating processes through Linux flock locks
on a regular file. It supports shared readers and an exclusive owner, with
nonblocking acquisition or a total deadline and cancellation token. The lock
is advisory: other processes can still read or write without acquiring it.
import std.fs.lock as lock
import std.path as path
from std.fs.lock import Mode, LockError
def main() -> Result[(), IoError | LockError]:
file = path.from_text("job.lock")?
mut owner = lock.acquire_now(&file, Mode.Exclusive)?
print(lock.acquire_now(&file, Mode.Shared).is_err())?
owner.close()?
another = lock.acquire_now(&file, Mode.Shared)?
print(another.is_closed())?
Ok(())
True
False
Acquisition
| Function | Result and behavior |
|---|---|
acquire_now(path: &Path, mode: Mode) | Result[Lock, LockError]: one nonblocking attempt; contention returns WouldBlock |
acquire_before(path: &Path, mode: Mode, deadline: &Deadline, cancel: &CancellationToken) | Result[Lock, LockError]: retries contention until acquired, cancelled or timed out |
Mode.Shared permits other shared owners. Mode.Exclusive excludes both
modes. Each call opens its own file description, so independent acquisitions
in the same process contend too. There is no upgrade/downgrade operation or
fairness guarantee. Use a consistent acquisition order for multiple resources.
A missing file is created with ordinary permission bits 0600, restricted by
the process umask. An existing file is opened read/write without truncating
its contents or changing its permissions. No writes are made. The final path
component must not be a symlink and the opened object must be a regular file;
opening a FIFO cannot block. Ancestor components follow normal OS resolution.
Use a trusted parent directory. Do not unlink or replace a live lockfile:
locks protect the opened inode, and a replacement would create an independent
lock domain. These functions never remove the lockfile, even after failure.
The borrowed native-byte Path must contain 1–4096 bytes and no NUL, otherwise
InvalidPath is returned before opening. OS path limits still apply. The
implementation allocates bounded path scratch and one owned File; allocation
failure returns Allocation(AllocError). The small lock facade adds no buffer
or queue. Failures close any descriptor already opened.
Waiting checks cancellation before deadline expiry, and both before opening
and before every lock attempt. An already expired deadline returns TimedOut
even if the lock would be available. After contention it waits on the token
for at most ten milliseconds or the remaining budget, whichever is smaller;
cancellation wakes that wait. EINTR from a lock attempt retries after checking
the token and deadline; the nonblocking form preserves it as Io(IoError.System).
Filesystem opening and metadata syscalls themselves cannot be interrupted by
the token or bounded by the deadline, including on remote filesystems.
Completion can race cancellation after the final observation.
Ownership and release
Lock is an affine owner with a private descriptor. It can move to a worker
or return from one; dropping it in that worker releases its ownership normally.
close(self: &mut Lock) -> Result[(), LockError]relinquishes this descriptor. Repeated close is harmless, including after a close error.is_closed(self) -> boolreports whether this owner has been closed.- Drop closes the descriptor without allocating or reporting a close error,
including on return,
?, and scope exit. It never issuesLOCK_UN.
On local Linux filesystems a lock belongs to the open file description.
Closing its final descriptor, or process death without surviving inherited
copies, releases it. A descriptor inherited through fork retains the lock
until that copy is also closed, even if the original owner closes or dies.
An explicit LOCK_UN would instead unlock all copies, which is why this API
has no separate unlock operation.
Descriptors are always created close-on-exec. Normal std.process
children do not retain them across exec. This API does not expose a raw
file descriptor or a switch to inherit it; explicit child-fd selection is
outside this API’s implemented scope. Applications requiring a lock to outlive
a runner through an exec’d child must keep their existing inheritance strategy.
This API is supported on 64-bit Linux; other targets return
Io(IoError.Unsupported). Network filesystems can emulate or restrict flock
and have different failure/mandatory-I/O semantics; consult the mounted
filesystem’s contract. It does not provide a distributed lease or fencing token.
Errors
LockError has WouldBlock, TimedOut, Cancelled, InvalidPath,
Io(IoError), and Allocation(AllocError). Native errors retain their OS code,
including access denial, unsupported locking, symlinks, and missing parents.
Closing only reports Io failures. Cancellation/timeouts consume no caller
resource: paths, deadlines, and tokens are borrowed, and no partial owner escapes.
Acknowledgements
- Linux flock defines the advisory locking and open-file-description lifetime semantics.
- libc, MIT OR Apache-2.0: the narrow native wrappers use its Linux syscall bindings. Waiting and error policy are Plenty.
std.fs.watch — filesystem changes
import std.fs.watch observes Linux files and directories through inotify.
It owns one nonblocking, close-on-exec descriptor, starts no threads, and uses
std.ready for deadlines, cancellation, and selector integration.
For a runnable introduction, see watch a directory.
import std.fs.watch as watch
import std.path as path
def main() -> Result[(), AllocError | IoError | watch.WatchError]:
mut watcher = watch.watcher()?
current = path.from_text(".")?
watcher.add(¤t)?
write_text("log.txt", "started")?
count = watcher.poll()?
for index in range(count):
event = watcher.event(index)?
if event.overflowed():
print("rescan")?
elif event.created():
print("created", event.name.to_string()?)?
elif event.closed_after_write():
print("written", event.name.to_string()?)?
watcher.close()?
Ok(())
created log.txt
written log.txt
Batches and ownership
watcher(batch_bytes=4096) -> Result[Watcher, WatchError] allocates a reusable
byte buffer and offset table. batch_bytes must be 272..65536: 272 accommodates
the longest Linux filename plus its event header. One read returns at most
batch_bytes // 16 events, with fewer when filenames occupy buffer space.
The offset table reserves that many ordinary list[i64] slots, in addition
to the compact byte buffer. This is a bound on each user-space batch, not on
the independent kernel queue or total number of watched paths.
| Operation | Behavior |
|---|---|
add(path: &Path) | Result[i32, WatchError]: watch the file or immediate directory children; return its watch ID |
remove(watch: i32) | Result[(), WatchError]: remove that watch; unknown/already removed IDs are errors |
poll() | Result[i64, WatchError]: replace the last batch with one nonblocking read; zero means no events or an interrupted read |
wait(deadline: &Deadline, cancel: &CancellationToken) | Result[i64, WatchError]: poll, then wait until a batch is available, cancelled, or timed out |
count() | Number of events in the last successful batch |
event(index: i64) | Result[Event, WatchError]: copy one event and its native filename |
register(selector: &mut Selector, key: u64) | Result[(), WaitError]: register the descriptor for read readiness |
unregister(selector: &mut Selector) | Result[(), WaitError]: remove that selector registration |
close() | Result[(), WatchError]: discard the batch and close all watches; repeated close is harmless |
poll and wait reuse their buffers without allocating. Events that do not fit
remain in the kernel queue for later calls; a full user-space batch is not
overflow. Consume the current batch before the next poll, wait, or close,
which invalidates it. event is an O(1) lookup plus filename copying. A failed
copy leaves the batch intact for retry, and a successful event owns its name
independently of the watcher. Dropping the watcher closes the descriptor and
all watches automatically.
Cancellation requested before wait wins over a ready event; otherwise an
already queued event wins over an expired deadline. After a selector reports
the watcher’s key readable, call poll. Closing the watcher removes its
descriptor from selectors automatically.
Event fields
Each Event exposes watch: i32, cookie: u32, and name: Path. The name is
the immediate child’s native filename bytes, not an absolute path. It is
empty for events about the watched object itself. Use Path.join with the
appropriate watched directory when a path is needed; name.to_text() validates
UTF-8, while name.to_string() provides display text.
Predicates report the event’s flags:
created(),deleted(),modified(),metadata_changed()andclosed_after_write()describe ordinary changes.moved_from()andmoved_to()identify rename endpoints. Their nonzerocookievalues pair endpoints within this watcher; either endpoint may be outside the watched directories or in a later batch. They need not be adjacent.deleted_self()andmoved_self()describe the watched object itself.ignored()says the kernel removed the watch, after explicit removal, deletion or unmount.unmounted()reports filesystem unmounting.overflowed()means kernel events were lost. Its watch ID is -1 and its filename is empty. Rescan the watched state before trusting an incremental cache again; overflow is returned as an event, not hidden by a successful read.is_directory()identifies directory events where the kernel provides it.
Adding the same inode again, including through another hard link, returns its existing watch ID. Paths follow symlinks. Watches follow inodes, not replaceable pathnames: to detect a program replacing a file atomically, watch its parent directory. Directory watches are not recursive. Reading/opening files does not generate events in this API; it requests only the change flags above.
Kernel notifications can coalesce, and are not an exact operation count or a transaction log. Changes through some remote filesystems, pseudo-filesystems, or memory mappings may not be reported. Events can be stale by the time they are read. Removal does not erase already queued events, and IDs can eventually be reused; use a fresh watcher after rebuilding state if stale IDs would be ambiguous. There is no automatic rescan, recursive registration, rename pairing, debouncing, or background callback execution. These native semantics follow the Linux inotify contract.
Errors and scope
WatchError distinguishes OutOfRange (buffer size/event index), InvalidData
(malformed native record), Io(IoError), Wait(WaitError), and
Allocation(AllocError). Native operations preserve IoError including its
allocation errors; facade buffer/filename allocations use Allocation.
Kernel resource limits, missing paths and denied access remain native errors.
Record offsets are bounded by the validated 65536-byte maximum batch and each
complete header/payload; malformed lengths remain InvalidData, not arithmetic
errors. Native signed watch IDs, including the overflow sentinel -1, are preserved.
No wrapped crate introduces infallible heap allocation. Watchers can move into
worker tasks; their mutable methods require exclusive borrowing.
The native implementation currently targets x86-64 Linux. Unsupported hosts
report IoError.Unsupported; there is no polling fallback. This module shares
the existing File, Bytes, path, and readiness runtime facilities, adding only
three operations for inotify creation, registration and removal.
Acknowledgements
- The Rust
notifycrate served as a design reference; no code is taken from it.
std.graph
from std.graph import DiGraph, UnGraph provides owned directed and
undirected graphs that store a value of type N on every node and E on
every edge, with traversal, shortest-path, ordering, component and spanning
tree algorithms. It is ordinary embedded Plenty code over lists; there is no
native dependency.
Graph kinds
| Type | Edges | Algorithms only on this kind |
|---|---|---|
DiGraph[N, E] | Run from source to target | toposort, strongly_connected_components, predecessors, incoming |
UnGraph[N, E] | Join two endpoints symmetrically | connected_components, minimum_spanning_tree, endpoints, opposite |
The kind is part of the type, so calling a directed-only algorithm on an
undirected graph is a compile-time error rather than a runtime surprise.
Both kinds permit parallel edges and self-loops. N and E follow ordinary
collection storage rules: affine owners are allowed, references and generators
are not, and unit () is not yet a supported type argument; use a small
placeholder such as bool for a graph without data.
Handles
NodeId and EdgeId are public aliases of i64. The n-th node or edge added
to a graph has handle n, counting from zero. Handles are plain integers so
that they copy; nothing stops passing an EdgeId where a NodeId is
expected, or a handle from one graph to another.
Removal never moves or reuses a handle. remove_node and remove_edge
leave a vacant slot, so every other handle keeps naming the same node or edge
and a removed handle stays invalid for the graph’s lifetime. The alternative,
swap-removal as in petgraph’s Graph, silently renames the last node to the
removed handle, so a handle saved earlier refers to a different node with no
way to detect it. Reusing vacant slots would need a generation counter in each
handle to catch stale ones. The cost of this rule is that memory stays
proportional to the number of nodes and edges ever added, not the number
alive; a graph with heavy churn should be rebuilt.
node_count() and edge_count() count live entries. node_bound() and
edge_bound() are one past the largest handle issued, so
for node in range(graph.node_bound()) with contains_node(node) visits live
nodes without allocating. nodes() and edges() return live handles in
increasing order.
Operations that take a handle trap when it is negative, out of range, or
removed, as indexing does. The trap is an index-out-of-bounds report inside
std.graph with a note: called from line naming the caller. remove_node,
remove_edge, contains_node, contains_edge and the get_* getters never
trap: removal of an invalid handle returns Nothing, and so does a checked
getter, so a possibly stale handle needs one lookup rather than a
contains_node test followed by node.
Building and reading
| Operation | Result | Contract |
|---|---|---|
DiGraph[N, E](), UnGraph[N, E]() | Result[Graph, AllocError] | An empty graph |
graph.reserve(nodes, edges) | Result[(), AllocError] | Room for that many more nodes and edges |
graph.add_node(data) | Result[NodeId, AllocError] | Consume data into a new node |
graph.add_edge(a, b, data) | Result[EdgeId, AllocError] | Consume data into a new edge; traps for an invalid endpoint |
graph.remove_node(node) | Option[N] | Remove every incident edge, then the node, transferring its data |
graph.remove_edge(edge) | Option[E] | Remove the edge, transferring its data |
graph.node(node), graph.edge(edge) | &N, &E | Borrow the data |
graph.node_mut(node), graph.edge_mut(edge) | &mut N, &mut E | Borrow the data exclusively |
graph.get_node_ref(node), graph.get_edge_ref(edge) | Option[&N], Option[&E] | Borrow the data; Nothing for an invalid handle |
graph.get_node_mut(node), graph.get_edge_mut(edge) | Option[&mut N], Option[&mut E] | Borrow the data exclusively; Nothing for an invalid handle |
graph.find_edge(a, b) | Option[EdgeId] | The earliest live edge from a to b (either way for UnGraph) |
di.source(edge), di.target(edge) | NodeId | Endpoints of a directed edge |
un.endpoints(edge) | (NodeId, NodeId) | Endpoints in the order given to add_edge |
un.opposite(edge, node) | NodeId | The endpoint that is not node (node itself for a self-loop) |
di.successors(node), di.predecessors(node) | Result[list[NodeId], AllocError] | Targets of outgoing edges, sources of incoming edges |
un.neighbors(node) | Result[list[NodeId], AllocError] | The other endpoint of every incident edge |
Neighbor lists follow edge insertion order and repeat a node once per parallel
edge. An undirected self-loop appears once in its node’s neighbors; a directed
self-loop appears in both successors and predecessors.
Each node heads linked chains threaded through its edges, as in petgraph’s adjacency list. Adding a node or an edge appends one slot and allocates only when that list grows, doubling its capacity. Allocation failure leaves the graph unchanged and drops the consumed data. Lookups, mutation through references, removal and walks allocate nothing. Removing an edge takes time proportional to the degrees of its endpoints; removing a node takes time proportional to the degrees of the node and its neighbors.
copy(graph)? duplicates the whole graph, including vacant slots, so handles
remain valid in the copy. Graphs move like other owned collections.
Allocation-free walks
di.outgoing(node), di.incoming(node) and un.incident(node) return a
Walk: a small cursor that holds no borrow of the graph. Pass it back with
graph.next_edge(&mut walk), which returns Option[EdgeId] in insertion
order:
from std.graph import UnGraph
def main() -> Result[(), AllocError | IoError]:
mut roads = UnGraph[str, i64]()?
home = roads.add_node("home")?
shop = roads.add_node("shop")?
park = roads.add_node("park")?
roads.add_edge(home, shop, 4)?
roads.add_edge(park, home, 2)?
mut walk = roads.incident(home)
while True:
match roads.next_edge(&mut walk):
case Some(edge):
other = roads.opposite(edge, home)
print(*roads.node(other), *roads.edge(edge))?
case Nothing:
break
Ok(())
shop 4
park 2
Edges added after the walk began at the end of the chain are seen; removing the edge the walk would visit next ends the walk early.
Traversal and components
| Operation | Result | Order |
|---|---|---|
graph.bfs(start) | Result[list[NodeId], AllocError] | Breadth-first, neighbors in insertion order |
graph.dfs(start) | Result[list[NodeId], AllocError] | Recursive depth-first preorder |
un.connected_components() | Result[list[list[NodeId]], AllocError] | Components by smallest handle, each in bfs order |
di.strongly_connected_components() | Result[list[list[NodeId]], AllocError] | Reverse topological: a component precedes those that reach it |
Traversal returns every node reachable from start, which comes first. Directed
traversal follows outgoing edges. dfs produces the order of a recursive search
but uses an explicit stack, so a path of any length cannot overflow the call
stack. Strongly connected components use iterative Tarjan; each component lists its
nodes in discovery order. Roots are tried in handle order. Each algorithm takes
O(V + E) time and O(V) extra space, where V is node_bound().
Topological order
di.toposort() returns Result[list[NodeId], GraphError]. It
uses Kahn’s algorithm, starting from source nodes in handle order, so the
result is deterministic. If the graph has a cycle, the error is
GraphError.Cycle(nodes): a list [a, b, ..., z] such that the graph has
edges a -> b, …, z -> a. A self-loop reports [node]. Finding the cycle
takes O(V + E) time after the sort fails.
Shortest paths
Weights come from a callback, weight: &F with F: Callable[[&E], f64], so
edge data can hold anything and the same graph can be searched by several
metrics. Integer weights convert with f64(value); they are exact up to 2^53.
| Operation | Result | Weights |
|---|---|---|
graph.dijkstra(start, weight) | Result[Paths, GraphError] | Non-negative; +inf acts as no edge |
graph.astar(start, goal, weight, estimate) | Result[Option[(f64, list[NodeId])], GraphError] | As Dijkstra |
graph.bellman_ford(start, weight) | Result[Paths, GraphError] | Any value except NaN and -inf |
Paths answers queries for every node:
| Method | Result |
|---|---|
paths.distance(node) | Option[f64]: total weight, or Nothing when unreachable |
paths.edge_to(node) | Option[EdgeId]: the last edge on the path; Nothing for the start |
paths.path(node) | Result[list[NodeId], AllocError]: start to node, or empty when unreachable |
Undirected edges can be used in either direction. Handles added after the search are reported unreachable.
Dijkstra checks each weight when it first examines that edge and returns
GraphError.InvalidWeight(edge) for a negative or NaN weight. It uses a
binary heap from std.collections and takes
O((V + E) log E) time. astar stops at goal and returns the cost and path,
or Nothing when goal is unreachable. estimate: &H with
H: Callable[[NodeId], f64] must never exceed the true remaining cost for
the result to be shortest; a NaN or negative estimate returns
GraphError.InvalidEstimate(node). Nodes may be expanded more than once when
an admissible estimate is inconsistent.
Bellman–Ford allows negative weights and takes O(V × E) time. A negative cycle
reachable from start returns GraphError.NegativeCycle(nodes) in the same
form as Cycle. In an undirected graph, any negative edge reachable from
start is such a cycle, reported as its two endpoints.
Minimum spanning tree
un.minimum_spanning_tree(weight) returns Result[list[EdgeId], GraphError]: a minimum spanning forest, with one tree per connected
component. It uses Kruskal’s algorithm with a union–find, so edges come out in
increasing weight, ties in handle order. Self-loops are never chosen. Negative
weights are allowed; a NaN weight returns GraphError.InvalidWeight(edge).
It takes O(E log E) time and O(V + E) space.
Errors
GraphError has five variants:
| Variant | Raised by |
|---|---|
Cycle(list[NodeId]) | toposort |
NegativeCycle(list[NodeId]) | bellman_ford |
InvalidWeight(EdgeId) | dijkstra, astar, bellman_ford, minimum_spanning_tree |
InvalidEstimate(NodeId) | astar |
Allocation(AllocError) | toposort, dijkstra, astar, bellman_ford, minimum_spanning_tree |
These algorithms return Allocation(cause) when their working lists cannot
grow; the traversals and other operations without a domain error return
AllocError. Weights
and estimates are calculated by user callbacks, which are called only while the
algorithm runs.
Limits
There is no graph-wide iteration protocol, compaction, subgraph, isomorphism,
max-flow, matching, or file-format support. Weakly connected components of a
directed graph, Floyd–Warshall, Prim, and petgraph’s dominators are
candidates that need a use case. Floating-point path costs use ordinary f64
addition, so very long paths accumulate rounding.
Acknowledgements
- API modelled on petgraph: node and edge handles,
StableGraph-style vacant slots, chained adjacency lists and thealgoset. No code was copied. - Topological order uses Kahn’s algorithm (1962).
- Strongly connected components use Tarjan’s algorithm (1972).
- Shortest paths use Dijkstra’s algorithm (1959), A* by Hart, Nilsson and Raphael (1968), and Bellman–Ford (1958), with negative-cycle recovery as described by cp-algorithms.
- Minimum spanning forests use Kruskal’s algorithm (1956).
std.http
import std.http reads and writes HTTP/1.1 messages (RFC 9112) over
std.io streams: request and response heads, Content-Length,
chunked and close-delimited bodies, and the connection reuse rules. It is the
framing layer only. std.http.server builds a server on
it and std.http.client a client; their HTTPS adapters use
std.tls.
std.http.decoder exposes the same request grammar as a
bounded byte-feed state machine without I/O. The
actor HTTP server uses it to suspend between
readiness events while keeping parsing state in its connection heap.
For body transfers that must preserve a concrete transport error type, the
additive std.http.typed readers support fixed-length, chunked
and close-delimited framing over std.io.typed. They assume validated heads;
the clients and servers above continue to use this module’s framing API.
The module is checked Plenty in stdlib/std/http.plenty; it adds no runtime
primitives. The same code runs over a std.net TcpStream, a
Cursor in a test, or any other Reader and Writer.
import std.bytes
import std.http
import std.io
from std.http import HttpError
def main() -> Result[(), AllocError | IoError | HttpError | std.io.StreamError | std.bytes.BytesError]:
wire = "POST /notes HTTP/1.1\r\nHost: example.com\r\nContent-Length: 5\r\n\r\nhello"
limits = std.http.limits()
mut conn = std.io.buffered_reader(std.io.cursor(std.bytes.from_text(wire)?), 1024)?
head = std.http.read_request(&mut conn, &limits)?.ok_or(HttpError.Truncated)?
framing = std.http.request_framing(&head)?
mut body = std.http.body_reader(conn, &framing, &limits)?
mut data = std.bytes.empty()?
std.http.read_to_end(&mut body, &mut data)?
print(head.method, head.target, data.to_text()?)?
mut reply = std.http.response(201, "Created")?
reply.add("Content-Length", "0")?
print(str.repr(std.http.encode_response(&reply)?.to_text()?)?)?
Ok(())
POST /notes hello
"HTTP/1.1 201 Created\r\nContent-Length: 0\r\n\r\n"
Messages
| Item | Meaning |
|---|---|
Header(name: str, value: str) | One field line |
RequestHead(method, target, minor: i64, headers: list[Header]) | minor is the HTTP/1.x minor version, 0 or 1 |
ResponseHead(minor, status: i64, reason: str, headers) | |
request(method, target), response(status, reason) | Result with a head using HTTP/1.1 and no fields; fails only when allocating the field list fails |
head.add(name, value) | Result[(), HttpError]: append a field |
find(headers: &list[Header], name: &str) | Option[i64]: index of the first field named name, ignoring ASCII case |
same_name(left: &str, right: &str) | bool: whether two names are equal ignoring ASCII case; never allocates, and non-ASCII text never matches |
Parsed names keep the case they arrived in. Values have surrounding spaces and tabs removed. Fields stay in arrival order and repeated names stay separate; nothing is combined.
Limits
Limits(max_line, max_head, max_headers, max_body) bounds every read:
| Field | Bounds | limits() |
|---|---|---|
max_line | One request, status, field or chunk-size line, including its CRLF | 8192 |
max_head | All bytes of a head, blank lines before a request line included; also a trailer section | 65536 |
max_headers | Field lines in a head or trailer section | 100 |
max_body | Decoded body bytes | 1048576 |
max_headers and max_body are NonNegative[i64], so a negative count is a
type error; read them with .get(). max_line and max_head must be at least
2, or the call fails with InvalidLimit. Memory use is one line buffer of at most
max_line bytes plus the parsed fields; the body is never held unless the
caller collects it.
Reading heads
| Function | Result |
|---|---|
read_request(conn: &mut BufReader[R], limits) | Result[Option[RequestHead], HttpError] |
read_response(conn: &mut BufReader[R], limits) | Result[Option[ResponseHead], HttpError] |
Nothing means the input ended before the message’s first byte, as when the
peer closes an idle connection; an end inside a head is Truncated. Blank
lines before a request line are skipped within max_head, and input that
ends after only blank lines is Nothing. A 1xx response is
returned like any other; read again for the final response.
Parsing is strict. Every line ends in CRLF; a bare LF is BareLineFeed, and a
CR inside a line is invalid. A request line is a token method, one space, a
target of visible ASCII, one space, and HTTP/1.0 or HTTP/1.1. A status
line is the version, a space, three digits from 100, and an optional space
and reason; HTTP/1.1 200 with no reason is accepted. A well-formed other
version such as HTTP/2.0 is UnsupportedVersion. A field line is a token
name, a colon with no space before it, and a value of visible bytes, spaces
and tabs that is valid UTF-8. A line starting with a space or tab (an obsolete
line fold) is InvalidHeader.
parse_request_line(line: &[u8]), parse_status_line(line: &[u8]) and
parse_field(line: &[u8]) expose the same grammar for custom readers. line
excludes CRLF; pass &buffer[:size] for a prefix. These helpers do not read streams or enforce head budgets.
For typed transports, prefer std.http.typed.read_request and
read_response, which enforce those budgets and retain causes
and consumed-byte progress.
Framing
Framing is Empty, Length(i64), Chunked, or UntilClose.
| Function | Result |
|---|---|
request_framing(head: &RequestHead) | Result[Framing, HttpError]; no framing field is Empty |
response_framing(head: &ResponseHead, method: &str) | Result[Framing, HttpError]; no framing field is UntilClose |
request_keeps_alive(head) | HTTP/1.1 unless Connection: close; HTTP/1.0 only with Connection: keep-alive |
response_keeps_alive(head, framing: &Framing) | As for requests, and never after an UntilClose body |
method is the method of the request a response answers. Responses to
HEAD, 1xx, 204 and 304 responses, and 2xx responses to CONNECT are Empty
whatever their fields say.
Framing that two implementations could read differently is an error, never a guess, because that disagreement is how request smuggling works:
| Fields | Error |
|---|---|
Content-Length and Transfer-Encoding together | ConflictingFraming |
| Either field twice, even with equal values | DuplicateFraming |
Content-Length other than decimal digits (+5, 1,1, empty), or above i64 | InvalidContentLength |
Transfer-Encoding other than exactly chunked, or in HTTP/1.0 | UnsupportedTransferEncoding |
The framing functions check both fields even where the status decides the framing, so a malformed response is rejected consistently.
Targets and Host
| Function | Result |
|---|---|
request_target(head: &RequestHead) | Result[RequestTarget, HttpError]: the checked authority, path and query |
RequestTarget has authority, path and query, all str. path and
query are still percent-encoded; query follows the first ? and is
empty without one, and an empty absolute-form path is /.
request_target applies the server rules of RFC 9112 section 3.2. The
parser reads a head without them, so a client or proxy can still read any
well-formed message; a server calls this before routing, as
std.http.server does.
| Request | Result |
|---|---|
HTTP/1.1 without Host | MissingHost |
More than one Host field, in any version | DuplicateHost |
A Host value other than uri-host [ ":" port ] | InvalidHost |
Origin-form target (/path?query) | authority is the Host value; empty for HTTP/1.0 without one |
Absolute-form target (http://authority/path?query, or https, scheme in any case) | authority is the target’s, replacing Host |
Absolute-form with userinfo (@), an empty host, or another scheme | InvalidTarget |
Asterisk-form (OPTIONS *) or authority-form (CONNECT host:port) | InvalidTarget |
Host is checked even when an absolute-form authority replaces it. An empty
Host value is valid; a client sends one when the target has no authority.
A host is a reg-name of unreserved, sub-delimiter and percent-encoded bytes,
or an IP literal in brackets. An IPv6 literal is checked for its characters,
not parsed as an address; an IPvFuture literal must match
"v" 1*HEXDIG "." 1*(unreserved / sub-delims / ":"). A port is digits.
Decimal Content-Length and hexadecimal chunk-size accumulation use local
checked results. Every nonnegative value through i64.MAX is representable;
larger values return InvalidContentLength or InvalidChunk, not an arithmetic
trap. Body limits are enforced separately, so a representable length can still
be BodyTooLarge. Bounded line, header and body cursors use documented
bounds-proven arithmetic opt-outs.
Reading bodies
| Function or method | Result |
|---|---|
body_reader(conn: BufReader[R], framing: &Framing, limits) | Result[BodyReader[R], HttpError]; takes the connection |
body.read_body(buffer, start, stop) | Result[i64, HttpError]: decoded bytes; 0 from a non-empty range is the end of the body, and an empty range returns 0 without reading |
body.read(buffer, start, stop) | The std.io Reader method |
read_to_end(body, out: &mut Bytes) | Result[i64, HttpError]: append the rest |
discard(body) | Result[i64, HttpError]: read and drop the rest |
finish(body: &mut BodyReader[R]) | Result[BufReader[R], HttpError]: the connection, at the next message |
release(body) | Option[BufReader[R]]: the connection in any state |
body.trailers | Trailer fields, once a chunked body has ended |
body.total(), body.is_done() | Bytes delivered; whether the end was reached |
body.take_failure() | Option[HttpError]: the framing error behind a read failure |
The body reader owns the connection, so nothing can read the next message
while a body is in progress. finish returns it only after the body’s end,
including buffered bytes of pipelined messages; a body with bytes left is
Unfinished and a failed one is Broken. release returns the connection
regardless, for writing a final error response or closing; its read position
is then unknown, so it must not carry another message.
A Length body larger than max_body fails before reading. A chunked body
fails as soon as a chunk size would pass max_body, and an UntilClose body
as soon as a byte arrives past it. Chunk sizes are hexadecimal without signs
or leading spaces. Chunk extensions after ; are checked for control bytes
and ignored. Each chunk’s data must be followed by CRLF.
parse_chunk_size(line: &[u8]) exposes that chunk-size grammar for custom
framed readers. It reads a &[u8] view of one line excluding CRLF, such as
&line[:size], and returns Result[i64, HttpError] without allocating.
Extensions are checked and ignored. Invalid syntax or a size exceeding
i64 returns InvalidChunk. Callers enforce their own line and body limits.
Through read, a body is an ordinary Reader for std.io.copy_all and
similar helpers. Stream and allocation errors keep their StreamError;
Truncated becomes UnexpectedEof; any other framing error becomes
Io(IoError.InvalidInput) and is kept for take_failure. After any error
the body reports only Broken.
read_to_end checks its next output-buffer extent before growing the buffer
or reading another chunk. An unrepresentable extent is
Allocation(AllocError.CapacityOverflow) and leaves the existing output and
connection position unchanged for that iteration. Earlier appended bytes remain.
Writing
| Function | Result |
|---|---|
write_request(writer: &mut W, head) | Result[(), HttpError] |
write_response(writer, head) | Result[(), HttpError] |
write_chunk(writer, source: &[u8]) | One chunk; an empty view writes nothing |
write_last_chunk(writer, trailers: &list[Header]) | The zero-size chunk and trailer section |
write_body(writer, framing, source: &[u8]) | A whole body under framing; Length and Empty must match len(source) |
Writers validate before writing and write nothing when validation fails:
token methods and names, targets of visible ASCII, reasons and values without
CR, LF or other control bytes, values without surrounding blanks, status 100–999,
and the framing rules above. A value cannot add a field of its own. A stream
failure may leave part of a message delivered. Heads are written with one
write_all; a chunk with three. The caller flushes.
encode_request(head), encode_response(head), encode_last_chunk(trailers)
and encode_chunk_size(length: NonNegative[i64]) return Result[Bytes, HttpError].
They perform the same validation and serialization without writing, sharing
implementation with the typed writers. A chunk length of zero encodes 0\r\n without the final trailer-section terminator.
Errors
| Variant | Cause |
|---|---|
Stream(StreamError) | The underlying stream failed, or a range was out of bounds |
Allocation(AllocError) | Memory for a line, field, or output ran out |
InvalidLimit | max_line or max_head below 2 |
Truncated | Input ended inside a head, a body, or a chunk |
LineTooLong, HeadTooLarge, TooManyHeaders, BodyTooLarge | A limit was reached |
BareLineFeed | A line ended in LF without CR |
InvalidRequestLine, InvalidStatusLine, InvalidHeader | Syntax outside the grammar above |
UnsupportedVersion | Not HTTP/1.0 or HTTP/1.1 |
InvalidContentLength, DuplicateFraming, ConflictingFraming, UnsupportedTransferEncoding | See Framing |
InvalidChunk | A chunk-size line or chunk terminator outside the grammar |
LengthMismatch | write_body given a range that does not match the framing |
Unfinished, Broken | finish before the end, or after a failure |
InvalidTarget, MissingHost, DuplicateHost, InvalidHost | See Targets and Host |
Costs, ownership and threads
- Reading a head is linear in its bytes. It allocates one line buffer, which
grows to the longest line read (at most
max_line), plus onestrper method, target, reason, name and value, and the field list. - A body reader allocates its line buffer and trailer list once. Body bytes
are copied once, from the
BufReaderinto the caller’s buffer; nothing is retained.read_to_endgrowsout4096 bytes at a time;discarduses one 4096-byte scratch buffer. - A writer builds each head or chunk-size line in one
Bytesand writes it; chunk data is written from the caller’s buffer without copying. - Heads,
LimitsandFramingare ordinary owned values. ABodyReaderowns its connection, so it moves to a worker with it whenRcan. Nothing is shared or global, and no call starts a thread or waits beyond the underlying stream’s own calls. - Supported targets are those of
std.io: x86_64 Linux GNU.
Example
The tutorial lesson reads
pipelined requests from memory and writes a chunked response. The tests in
tests/test_std_http.rs serve several requests on one loopback connection.
Limits
- HTTP/1.0 and HTTP/1.1 only. No HTTP/2, or
100-continuehandling beyond returning the interim response. Upgrades are not handled here:std.http.serverhands a101connection tostd.http.websocket, anddetachhands a long-lived response tostd.http.sse. - No connection pool, redirects, proxies, cookies, or TLS here; the server
is
std.http.serverand the clientstd.http.client. Deadlines are the caller’s: usestd.net’s*_beforecalls, or read with a deadline-awareReader. - Only the
chunkedtransfer coding. A body with another coding is rejected, not decoded. read_requestreturns the target as text;request_targetsplits the forms a server accepts. Usestd.urlfor other URLs.- Field values must be UTF-8; a peer that sends Latin-1 bytes is rejected.
Acknowledgements
- Bundling a client and a server follows the scope of Go’s
net/http.
std.http.decoder
A resumable HTTP/1.x request decoder with no I/O. Feed it whatever bytes are
available now, and retain the unconsumed suffix. It shares request-head,
target, framing, chunk-size and trailer validation with std.http.
Its stored state contains owned bytes rather than shared strings, so it can
live inside an actor node.
decoder(&limits) returns Result[Decoder, HttpError]. The limits are copied;
max_line or max_head below 2 returns InvalidLimit.
Allocation failures are HttpError.Allocation. Header bytes and lines are
bounded by those limits; body chunks are at most feed_budget (4096) bytes.
import std.http
import std.http.decoder
from std.http.decoder import Event
def main() -> Result[(), std.http.HttpError | IoError | OverflowError[i64]]:
mut decoder = std.http.decoder.decoder(&std.http.limits())?
wire = b"POST /echo HTTP/1.1\r\nHost: localhost\r\nContent-Length: 4\r\n\r\nping"
mut at = 0
while not decoder.is_done():
consumed, event = decoder.feed(&wire[at:])?
at += consumed
match event:
case Some(Event.Head(head, _, _)):
print(head.method)?
case Some(Event.Body(body)):
print(body.len())?
case Some(Event.End):
print("complete")?
case _:
pass
decoder.finish()?
Ok(())
POST
4
complete
This example already owns the complete request, so it can loop until End.
A readiness loop must instead yield when no currently available input can
advance the decoder. Bound the number of feed calls per scheduler turn too;
the per-call byte budget alone does not bound an unbounded caller loop.
Progress and events
feed(source: &[u8]) reads a &[u8] view and returns
Result[(consumed, Option[Event]), HttpError]. consumed counts from the
start of the view, not from the start of the caller’s buffer. Advance the source cursor by exactly that
count. Neither a short count nor Nothing implies EOF.
| Event | Meaning |
|---|---|
Head(RequestHead, RequestTarget, Framing) | Validated request metadata and body framing |
Body(Bytes) | Owned decoded body bytes |
Trailers(list[Header]) | Validated trailer fields after a chunked body |
End | Request complete; no pipelined bytes have been consumed |
An empty feed can emit a pending End. The returned head/trailer events own
text, but the decoder does not retain that text after the call. Callers that
keep events across actor callbacks must respect the actor’s storage boundary.
finish() reports explicit input EOF. It succeeds after a complete body, or
before any nonblank request has begun, and reports truncated or malformed
input otherwise. If body completion is pending, collect its End with an
empty feed. finish does not make an incomplete request reusable.
State and reuse
| Method | Contract |
|---|---|
is_started() | Nonblank request input has arrived, even if the line is partial |
has_head() | A valid head was emitted and the decoder has not failed |
is_done() | End was emitted |
total() | Decoded body bytes consumed for this request |
reset() | Begin another request after End; otherwise Unfinished |
Reset only when the application is ready to process pipelined input. Framing
and allocation errors poison the decoder; discard it instead of resuming
with a suffix whose request boundary is no longer trustworthy. Source
ranges follow the ordinary byte-view rules; feed
takes an already-formed view, not separate range parameters. The decoder never
reads a socket, waits, or owns the caller’s input buffer.
Internal cursor arithmetic uses explicit opt-outs whose bounds come from the
4096-byte feed budget, validated CRLF lines and the copied head/body limits.
Numeric length parsing keeps the shared parser’s recoverable errors. Calling
from another arithmetic mode does not change the decoder’s HttpError contract.
std.http.body
import std.http.body reads request bodies inside a
std.http.server handler: URL-encoded forms, JSON
documents and multipart uploads. Every reader checks the request’s
Content-Type, works within caller limits, and fails with a BodyError
that maps to a response status. The module is checked Plenty in
stdlib/std/http/body.plenty and adds no runtime primitives. The lesson
Read forms, cookies and forwarded clients
uses it.
import std.http.body
from std.http.body import BodyError
def main() -> Result[(), BodyError | IoError]:
limits = std.http.body.form_limits()
pairs = std.http.body.parse_form_text("name=Ada+Lovelace&lang=en&lang=fr", &limits)?
for pair in &pairs:
print(pair.key, pair.value)?
match std.http.body.find(&pairs, "name"):
case Some(index):
print(pairs[index].value)?
case Nothing:
print("no name")?
match std.http.body.parse_form_text("bad=%zz", &limits):
case Err(error):
print(error, std.http.body.status(&error))?
case Ok(_):
print("accepted")?
Ok(())
name Ada Lovelace
lang en
lang fr
Ada Lovelace
std.http.body.BodyError.Malformed 400
Errors
| Variant | Meaning | status |
|---|---|---|
UnsupportedMediaType | Content-Type missing, malformed or not the expected type | 415 |
TooLarge | A byte limit was passed, or Content-Length declares more than it | 413 |
TooManyFields | More fields or parts than allowed | 413 |
Malformed | Bad percent-escape, multipart structure, boundary or part header | 400 |
InvalidUtf8 | A decoded name or value, or a multipart field, is not UTF-8 | 400 |
Http(HttpError) | Reading the body failed; BodyTooLarge is 413, Stream (timeout, cancellation, reset) 408 | 413, 408, 400 or 500 |
Json(JsonError) | The document is invalid (400), over max_input (413), or fails on allocation (500) | 400, 413 or 500 |
Storage(StreamError) | Writing an upload to its temporary file failed | 500 |
Allocation(AllocError) | Memory ran out, including while writing an upload to its file | 500 |
status(&error) -> i64 gives the status, and
reject(exchange, &error) -> Result[(), HttpError] answers with it and its
reason as plain text. The server’s own fallback only maps a failed body
read, so a handler that wants 415 or 400 for a bad form calls reject
instead of returning the error.
Forms
| Item | Meaning |
|---|---|
FormLimits(max_bytes, max_fields) | Public NonNegative[i64] fields; form_limits() is 64 KiB and 256 fields |
read_form(exchange, &limits) | Result[list[Pair], BodyError]: read an application/x-www-form-urlencoded body |
parse_form(&bytes, &limits) | Decode a form already in memory |
parse_form_text(&text, &limits) | Decode text, such as exchange.query |
find(&pairs, key) | Option[i64]: index of the first pair named key |
Pair(key, value) is std.url’s. Pairs keep their order and
duplicates. + decodes to a space and %hh to a byte; a % without two
hex digits is Malformed, and the decoded bytes must be UTF-8. Empty pieces
between & are skipped, and a piece without = has an empty value.
Parameters such as charset on the media type are accepted and ignored:
forms are UTF-8. As in RFC 9110, an empty parameter (a/b; or a/b;;c=d)
is allowed; a parameter without = makes the media type malformed.
read_form refuses a declared Content-Length over max_bytes before
reading, and stops a chunked body as soon as it passes the limit. Byte totals
that cannot be represented as i64 are also TooLarge, not arithmetic
errors. Multipart part totals are checked before appending or writing the
next piece; whole-body totals are checked after reading the next chunk
but before retaining it.
JSON
| Item | Meaning |
|---|---|
read_json(exchange, &limits) | Result[Json, BodyError]: read and parse a JSON body |
send_json(exchange, status, &value, &limits) | Result[(), BodyError]: encode and send as application/json |
limits is a std.json.Limits; max_input bounds the body,
and depth, duplicate keys and output size follow it. The request must be
application/json or another application/...+json type.
Multipart uploads
| Item | Meaning |
|---|---|
MultipartLimits(max_parts, max_head, max_field, max_file, max_body) | Public NonNegative[i64] fields; multipart_limits() is 64 parts, 4 KiB heads, 64 KiB fields, 8 MiB files, 16 MiB in all |
read_multipart(exchange, &limits, storage: &Root) | Result[list[Part], BodyError]: read a multipart/form-data body |
Part | name, filename: Option[str], content_type, size, content: Content |
Content | Text(str) for a field, File(TempFile) for an upload |
max_head bounds one part’s header section, max_field a field held in
memory, max_file one upload, and max_body every byte read, including the
preamble, delimiters and epilogue. A declared Content-Length over
max_body is refused before reading.
A part with a filename parameter is an upload. Its content is streamed into
a new unnamed temporary file in storage as it arrives,
never held whole in memory, and the file is rewound before it is returned:
read it through file.stream after matching Content.File(file), or keep
it with file.publish(&name). Because
the file has no name until published, a request that fails, a connection
that is cancelled, and a part the handler drops all leave nothing on disk.
The descriptors are closed when the TempFiles drop. Each upload holds two
descriptors until then, so max_parts also bounds descriptor use.
filename is what the client sent. It is not a path and is not cleaned; do
not use it to name a file without checking it, and prefer a name the program
chooses.
Other parts are fields and must be UTF-8. A part’s Content-Type defaults
to text/plain and is reported, not checked. Each part needs a
Content-Disposition: form-data field with a name. The boundary must be
1 to 70 characters from RFC 2046’s set; per that RFC, the content of a part
cannot contain the delimiter line, so a part whose content holds
CRLF--boundary ends there. Content-Transfer-Encoding is ignored, as
RFC 7578 deprecates it.
Limits
- Bodies are read whole for forms and JSON; use
exchange.read_bodydirectly for streamed processing. - Multipart fields are not streamed; their size is bounded by
max_field. - No
multipart/mixednesting, and no character sets other than UTF-8.
std.http.client
import std.http.client sends HTTP/1.1 requests over pooled keep-alive
connections. Each call has a caller deadline and cancellation token and a
timeout for each phase. Redirects are bounded and drop credentials when the
origin changes. A proxy is used only when configured, and a request is
replayed only when that is safe. Request bodies are in memory or streamed
from a std.io Reader. Head parsing, writes and body framing use
std.http.typed, with shared std.http validation, and
sockets from std.net; std.http.https adds https
URLs through std.tls. Both modules are checked Plenty in
stdlib/std/http/ and add no runtime primitives. The tutorial lessons
Send HTTP requests,
Share one HTTP client between workers
and Send HTTPS requests show
complete programs.
import std.bytes
import std.http
import std.http.client
import std.http.server
import std.net
import std.time
from std.http.client import ClientError
from std.http.server import Exchange
from std.net import TcpStream
from std.sync import CancellationToken
pub enum Route:
Echo
pub class Echoer:
pub def handle(self, route: &Route, exchange: &mut Exchange[TcpStream]) -> Result[(), AllocError | std.http.HttpError | std.bytes.BytesError]:
mut body = std.bytes.empty()?
exchange.read_all(&mut body)?
exchange.send(200, "text/plain", &body)
type VisitError = AllocError | IoError | std.bytes.BytesError | ClientError | RangeError
def visit(port: u16, stop: CancellationToken) -> Result[(), VisitError]:
with stop.cancel_on_exit():
never = CancellationToken()?
deadline = std.time.deadline_after(&std.time.monotonic_now(), &std.time.seconds(5u32))?
client = std.http.client.client(std.http.client.config()?)?
mut post = std.http.client.request("POST", &f"http://127.0.0.1:{port}/echo"?)?
post.add("Content-Type", "text/plain")?
post.body = std.bytes.from_text("ping")?
reply = client.fetch(post, &deadline, &never)?
print(reply.status, reply.body.to_text()?)?
Ok(())
def main() -> Result[(), SpawnError[()] | std.http.server.ServerError | std.net.NetError | VisitError]:
mut routes = std.http.server.router[Route]()?
routes.add("POST", "/echo", Route.Echo)?
listener = std.net.listen(&std.net.loopback_v4(0u16), 16)?
port = listener.local_address()?.port
stop = CancellationToken()?
with spawn(visit, port, stop.share()).drop_job()? as task:
drop(std.http.server.serve(listener, &routes, &Echoer(), &std.http.server.config(), &stop)?)
task.join()?
Ok(())
200 ping
Clients
| Item | Meaning |
|---|---|
client(config: Config) | Result[Client[Plain, TcpConnector], ClientError]: plain http over TCP |
connector[T: Transport, C: Connector[T]](opener: C, config: Config) | Result[Client[T, C], ClientError]: connections come from opener; give both type arguments |
client.send(request, &deadline, &cancel) | Result[Response[T], ClientError]: the final response head, after redirects |
client.send_from(request, &mut body, &upload, &deadline, &cancel) | send with the body read from body, a std.io Reader; see Streamed bodies |
client.fetch(request, &deadline, &cancel) | Result[Fetched, ClientError]: send, read the whole body, finish |
client.get(url: &str, &deadline, &cancel) | fetch of a GET |
client.finish(response) | Pool the connection if the body was read to its end and both sides allow reuse; otherwise close it |
client.close_idle() | Close the connections pooled when it is called |
client.idle() | i64: pooled connections |
client.stats() | Stats |
Both constructors fail with InvalidConfig for a config outside the bounds
below. Every method borrows the client shared, so one client serves many
threads through Shared handles; see
Sharing a client between threads.
deadline bounds the whole exchange, including later reads of the body of
the Response that send returns. cancel stops it: a wait in progress
ends with Net(Cancelled). A request whose token is already cancelled, or
whose deadline has passed, fails with Net(Cancelled) or Net(TimedOut)
before it takes a connection, so it leaves pooled connections alone.
Requests and responses
| Item | Meaning |
|---|---|
request(method: &str, url: &str) | Result[Request, ClientError] with no fields and an empty body |
Request.method, url, headers: list[Header], body: Bytes | Public fields |
request.add(name, value) | Result[(), ClientError]: append a field |
Response.status, head: ResponseHead, url, redirects | Final status and head, the URL that answered, and redirects followed |
response.read_body(&mut buffer, start, stop) | Result[i64, ClientError]: part of the body; 0 is the end |
response.read_all(&mut out) | Result[i64, ClientError]: append the rest, within limits.max_body |
response.read_all_progress(&mut out) | Result[i64, Partial[ClientError]]: append the rest and retain the count appended by this call if it fails |
response.read_typed(...) | Result[i64, StepError[ClientError]]: read with the complete returned cause |
typed_response(response), unwrap_response(stream) | Owning TypedResponse[T] adapter for std.io.typed.Reader[ClientError]; unwrap before client.finish |
response.total(), response.trailers() | Payload bytes successfully read, and borrowed trailer fields (complete after is_done) |
response.read(...), take_failure() | A std.io Reader; a client failure is Io(InvalidInput) and take_failure returns it, a short body is UnexpectedEof |
response.is_done() | The whole body has been read |
Fetched.status, head, url, redirects, body: Bytes | A whole response |
The client writes Host from the URL, and Content-Length when the body is
not empty or the method is POST, PUT or PATCH. An in-memory body is
sent with Content-Length. A request that sets
Host, Content-Length, Transfer-Encoding, Connection, Keep-Alive,
Upgrade, TE, Trailer, Expect, Proxy-Authorization or
Proxy-Connection fails with InvalidRequest, as do CONNECT and a method
that is not an HTTP token. These checks run before any connection opens.
URLs must be absolute http or https URLs with a host. Credentials in the
URL (user:pass@) are InvalidUrl; send an Authorization field instead.
The fragment is not sent. The TCP connector refuses an https URL with
UnsupportedScheme; use std.http.https.
Interim 1xx responses are skipped, up to 16; 101 Switching Protocols is
UnexpectedInterim. A response to HEAD, and a 204 or 304, has no body.
Typed response reads preserve network, TLS, cancellation and framing causes
directly. read_all_progress counts bytes appended after the output’s existing
prefix. It reserves output capacity before consuming payload; on failure the
prefix and appended bytes remain available, the connection closes, and
total() retains previously returned payload progress. The existing read_all
returns the same cause while omitting the count. send and fetch also retain
their established ClientError return types; use the lower-level typed HTTP
operations when per-head or per-write partial counts are needed.
The legacy read method still implements std.io.Reader and uses
take_failure for causes that StreamError cannot represent. Bridging that
method with std.io.typed.reader cannot recover the erased cause; use
typed_response instead. Internal head parsing, body decoding and CONNECT
tunnelling no longer store transport errors for later retrieval.
Streamed bodies
send_from reads the request body from any std.io Reader, 16 KiB at a
time, and never holds more of it. The request’s own body must be empty,
or the call fails with InvalidRequest before connecting.
| Item | Meaning |
|---|---|
sized(length: i64) | Upload sent with Content-Length: length |
chunked() | Upload sent with Transfer-Encoding: chunked and no trailers |
Upload.length: Option[i64], expect_continue: bool | Public fields; expect_continue is False from both constructors |
With sized, the reader must yield exactly length bytes: more, or an end
before length, is WrongBodyLength and the connection is closed. A
negative length is InvalidRequest. A reader failure is Body(StreamError),
or Allocation. Explicit reader interruption is retried; negative or oversized
successful counts are rejected as Body(InvalidProgress).
With expect_continue, the head carries Expect: 100-continue and the
client waits up to config.continue_timeout before sending the body:
| The server | The client |
|---|---|
Sends 100 Continue | Sends the body, then reads the final response |
Sends nothing before continue_timeout | Sends the body anyway, as RFC 9110 allows |
Sends a final response first, such as 417 or 401 | Returns it without sending the body; the connection is closed, not pooled |
A 417 Expectation Failed is returned like any other status; the client
does not retry without the expectation, since the reader may not be
restartable.
Connections and the pool
After fetch, or finish on a fully read response, the connection waits in
the pool for another request to the same scheme, host and port (and proxy).
The most recently used connection is taken first. A connection is closed
instead of pooled when:
- either side sent
Connection: close, or the response is HTTP/1.0 without keep-alive, or its body ran to the end of the connection; - the body was not read to its end, or reading it failed;
- bytes beyond the response are already buffered;
- the pool already holds
max_idle_per_originconnections for that origin, ormax_idlein all (the oldest is closed to make room).
Dropping a Response without finish closes its connection. Before reusing
a pooled connection the client checks, without waiting, that nothing has
arrived on it: a connection the server closed, or one with unexpected
bytes, is closed and another is used. A check that cancellation cuts short
puts the connection back. A connection idle longer than idle_timeout is
closed when the pool is next used.
Sharing a client between threads
A client’s pool is one value behind a std.sync mutex, and its counters are
atomics, so threads share a client by sharing the value:
client = Shared(std.http.client.client(config)?)?
job = pool.submit(fetch_all, client.share(), url)?
Each job owns a handle and calls get, send, finish and the rest through
it; a connection one thread parks is reused by whichever thread asks next.
The lock is held only while a connection is taken out of the pool or put back
in: connecting, TLS, writing, waiting for a response and reading a body all
run with the lock released, as do closing any connection and the readiness
probe before reuse. A request to a slow origin therefore does not delay
requests to another through the same client. close_idle takes the
connections out one at a time, oldest first, and stops after as many as were
idle when it was called, so other threads parking connections cannot keep
it running.
max_idle, max_idle_per_origin and idle_timeout bound the shared pool as a
whole, and Stats counts every event exactly once. Every request sent,
counting replays and redirects, went on a connection counted in opened or in
reused, so opened + reused is the number of requests sent. Every
connection counted in opened is idle, held by a request or a Response, or
counted in discarded once closed, whatever closed it; so after all jobs
finish and every response has been finished or dropped, opened is idle()
plus discarded. stats() reads each counter without the lock, so a snapshot
taken while jobs run may show one counter ahead of another.
Sharing a client is subject to the Shared rules: the connector’s type must
be able to cross threads. TcpConnector and HttpsConnector, whose TLS
configuration is immutable, qualify. A single-threaded client pays an
uncontended lock and unlock per checkout and return, and an atomic increment
per counted event; neither allocates.
Replaying requests
A server may close an idle connection at the moment the client reuses it.
When a request on a reused connection fails before any byte of a response
arrives, by end of input, a reset, or a failed write, the client sends it
again on a new connection, at most retries times, but only if:
- the method is
GET,HEAD,OPTIONS,TRACE,PUTorDELETE, and - the body is the request’s in-memory
body, or a streamed body of which nothing has been read yet (for example, while waiting for100 Continue).
POST, PATCH and other methods are never replayed, since the server may
have acted on them. A failure on a new connection, or after response bytes,
is returned, and so is a timeout or cancellation.
Redirects
A 301, 302, 303, 307 or 308 response with a Location field is
followed, at most redirects times; one more is TooManyRedirects. With
redirects = 0 the redirect response itself is returned.
| Status | Next request |
|---|---|
303 | GET without a body (HEAD stays HEAD) |
301, 302 after POST | GET without a body |
301, 302 otherwise, 307, 308 | Same method and body |
When the body is dropped, so are Content-* fields. A redirect that keeps
the body is not followed for a streamed body, which cannot be read twice:
send_from returns the 301, 302, 307 or 308 response itself. Location may be
absolute, scheme-relative (//host/path), absolute-path or relative; it is
resolved against the current URL and . and .. segments are removed.
A location that does not parse is InvalidRedirect, another scheme is
UnsupportedScheme, and a move from https to http is InvalidRedirect.
When the scheme, host or port changes, the next request and every later one
lose Authorization, Cookie, and every field named in config.sensitive.
They are not restored on a redirect back. The body of a redirect response is
read and discarded, up to 64 KiB, so the connection can be reused; a longer
body closes it.
Proxies
| Item | Meaning |
|---|---|
proxy(url: &str) | Result[Proxy, ClientError] for http://host:port, no path or credentials |
Proxy.endpoint, authorization: Option[str] | Where to connect, and the Proxy-Authorization value |
With config.proxy set, every http request goes to the proxy with an
absolute-form target (GET http://host/path HTTP/1.1). An https request
opens a tunnel instead: the client sends CONNECT host:port HTTP/1.1 to the
proxy, and after a 2xx reply runs TLS to the origin through it, verifying
the origin’s certificate, and sends an origin-form request. A non-2xx
reply is TunnelRefused(status); its body is not read. authorization is
sent to the proxy as Proxy-Authorization, on CONNECT for a tunnel, and
never to an origin. Nothing is read from the environment: HTTP_PROXY and
NO_PROXY are ignored.
Configuration
config() returns a Config with every field public:
| Field | Meaning | config() |
|---|---|---|
limits | std.http Limits for response heads and bodies, including a proxy’s CONNECT reply | std.http.limits() |
connect_timeout | Resolve the host and connect | 10 s |
write_timeout | Any one write | 10 s |
head_timeout | From the end of the request to the end of the final response head | 30 s |
read_timeout | Any one read | 30 s |
idle_timeout | Keep a pooled connection | 30 s |
continue_timeout | Wait for 100 Continue before sending a streamed body anyway | 1 s |
max_idle | Pooled connections in all, at least 0; 0 sends Connection: close | 16 |
max_idle_per_origin | Pooled connections for one origin, at least 0 | 4 |
redirects | Redirects to follow, 0 to 100 | 10 |
retries | Replays of a safe request, 0 to 10 | 1 |
sensitive | Further field names dropped across origins | empty |
proxy | Option[Proxy] | Nothing |
Every wait ends at the earliest of the caller’s deadline, its phase timeout
and its per-operation timeout. Internal deadline addition saturates at
u64.MAX; its bounds are explicit, and debug/optimized boundary tests verify
it without allocations. Retry, redirect, response, upload and pool cursor
arithmetic uses documented opt-outs justified by configuration, body budgets
and validated progress. User connectors and transports retain independent
declared arithmetic contracts; these implementation modes do not extend into
their callbacks.
Transports
| Item | Meaning |
|---|---|
protocol Transport | read_before and write_before with TcpStream’s parameters, returning Result[i64, ClientError]; write_before takes its source as &[u8] |
protocol Connector[T] | open(self, endpoint: &Endpoint, deadline, cancel) -> Result[T, ClientError]; the connector is borrowed shared, since threads sharing a client open connections at the same time |
Endpoint(scheme, host, port: u16, tunnel: Option[Tunnel]) | Where to connect: the origin, or the proxy; host is lowercase without brackets |
Tunnel(host, port: u16, authorization: Option[str], limits: Limits) | The proxy to reach the origin through with CONNECT; limits bounds the proxy’s reply head |
dial(&endpoint, &deadline, &cancel) | Result[TcpStream, ClientError]: a TCP connection to the endpoint, through its tunnel if it has one |
Plain, TcpConnector | A TCP connection; the connector refuses https and tunnels with UnsupportedScheme |
A connector decides how to reach an endpoint. dial resolves the host and
tries each address in turn. An in-memory connector makes code that uses a
client testable without sockets; one that hands out scripted connections
keeps them in a Mutex, since open borrows it shared. connector checks
both type arguments against the protocols where it is called, so a connector
whose open takes self: &mut ... is reported there; like any argument of a
bound in another module, a connector and a transport must be pub classes.
HTTPS
import std.http.https supplies a connector that opens http endpoints over
TCP and https ones over std.tls. It is a separate module so
that a program using only http does not depend on the TLS stack.
| Item | Meaning |
|---|---|
client(config: Config, tls: ClientConfig) | Result[Client[Connection, HttpsConnector], ClientError] |
system_client(config: Config) | client with the operating system’s trust roots |
connector(tls: ClientConfig) | HttpsConnector, for std.http.client.connector |
secure_error(error: TlsError) | The ClientError a TLS failure becomes |
Verification is always on: the server’s certificate must chain to the roots
in tls and name the URL’s host, a DNS name or IP address, which is also
sent as the server name (SNI). There is no switch to skip verification;
trust a private CA by building tls from its certificate. Client
certificates, protocol versions and revocation lists come from tls. If
tls offers ALPN protocols, a server that picks anything but http/1.1 is
Tls(Incompatible).
A pooled https connection is keyed by scheme, host and port like any
other, so it is reused only for the same server name. A redirect from
https to http is InvalidRedirect.
TLS failures map to ClientError:
TlsError | ClientError |
|---|---|
Net(Stream(Allocation(a))) | Allocation(a) |
Any other Net(e) | Net(e) |
Truncated | Net(Stream(UnexpectedEof)), so a closed pooled connection is replayed as a TCP one is |
Closed | Net(Stream(Io(Closed))) |
Allocation(e) | Allocation(e) |
| Any certificate, handshake or configuration failure | Tls(TlsFailure) with the same variant name |
Errors
Net(TimedOut) means a caller or configured phase deadline passed.
Net(ConnectionTimedOut) means the kernel gave up on the connection; it is
propagated as a connection failure, including through HTTPS, rather than
treated as the end of the 100 Continue waiting period.
ClientError is Net(NetError) for connection failures, timeouts and
cancellation; Http(HttpError) for malformed or oversized responses, with
Truncated for a connection that closed early; Url(UrlError);
Allocation; Tls(TlsFailure) for a TLS handshake or session failure;
TunnelRefused(status) for a proxy that refused CONNECT;
Body(StreamError) for a streamed body’s reader failure; and
InvalidConfig, InvalidUrl, UnsupportedScheme, InvalidRequest,
InvalidRedirect, TooManyRedirects, UnexpectedInterim and
WrongBodyLength. TlsFailure has the variants UntrustedCertificate,
ExpiredCertificate, WrongHost, InvalidCertificate,
RevokedCertificate, Rejected, Protocol, InvalidName,
InvalidConfiguration, NoTrustRoots, CertificateRequired and
Incompatible; see TlsError. No variant carries a URL or field value, so printing an
error does not reveal credentials.
Stats counts, as u64: opened connections; reused, requests sent on a
pooled connection; retried replays; redirected hops; and discarded
connections, those closed for any reason: the response or server ended
reuse, the body was not read to its end, the exchange failed or was
replayed, the connection was stale, expired, over a pool limit or had unread
bytes buffered, or close_idle closed it. See
Sharing a client between threads for the
totals they keep. These counters retain atomic fetch_add’s
wrapping behavior; checked-default arithmetic does not change that primitive’s
contract.
Limits
- No cookie jar. Read
Set-Cookiefields fromresponse.headand send aCookiefield yourself; the client drops it when the origin changes. - No content decoding. The client does not send
Accept-Encoding, so servers normally send identity bodies. A program that asks forgziporbrchecks the response’s actual coding and decodes the body itself withstd.compressunder an independent decoded-size limit. See Brotli and HTTP. - No HTTP/2, and no proxy for
httpsother than anhttpproxy withCONNECT. - No protocol upgrades on pooled connections. WebSocket clients are in
std.http.websocket, which opens connections withdial. - Connecting tries addresses one after another, not in parallel.
Acknowledgements
- Design references: Go’s
net/httpClientand Python’srequests, including dropping credentials on cross-origin redirects.
std.http.cookie
import std.http.cookie reads the Cookie fields of a request and writes
Set-Cookie fields (RFC 6265). Reading is lenient; writing is strict, so no
value can add an attribute or a header field. The module is checked Plenty
in stdlib/std/http/cookie.plenty and adds no runtime primitives; signing
uses std.crypto.
import std.time
import std.http.cookie
from std.http import Header
from std.http.cookie import CookieError
def main() -> Result[(), CookieError | AllocError | IoError]:
headers = [Header("Cookie", "theme=dark; bad value; lang=\"en\"")]?
received = std.http.cookie.cookies(&headers, 16)?
for cookie in &received:
print(cookie.name, cookie.value)?
mut reply = std.http.cookie.set_cookie("theme", "light")?
reply.max_age = Some(std.time.seconds(3600u32))
print(reply.to_string()?)?
print(std.http.cookie.removal("lang")?.to_string()?)?
Ok(())
theme dark
lang en
theme=light; Path=/; Max-Age=3600; Secure; HttpOnly; SameSite=Lax
lang=; Path=/; Max-Age=0; Secure; HttpOnly; SameSite=Lax
Reading
| Item | Meaning |
|---|---|
Cookie(name, value) | One cookie a client sent |
cookies(&headers, limit) | Result[list[Cookie], CookieError]: every valid cookie in the Cookie fields, in order |
find(&cookies, name) | Option[i64]: index of the first cookie named name |
A pair whose name is not a token or whose value has a byte outside RFC
6265’s cookie-octets (control bytes, space, ", ,, ;, \) is skipped,
as is a piece without =. The field carries cookies set by every
application on the host, and one malformed cookie must not make every
request fail. A value wrapped in double quotes loses them. More than limit
valid cookies is TooManyCookies. Each field is copied once into a byte
buffer and checked with bulk scans; only valid cookies are copied out as
text, so a skipped pair allocates nothing.
Values are not decoded. A cookie’s value is whatever the client sends; check it, or sign it, before trusting it.
Writing
| Item | Meaning |
|---|---|
set_cookie(name, value) | Result[SetCookie, CookieError]: a cookie with the defaults below |
removal(name) | Result with a cookie that deletes name: empty value, Max-Age=0; CookieError for an invalid name |
SetCookie | Public fields name, value, path, domain, max_age: Option[Duration], secure, http_only, same_site: SameSite |
cookie.to_string() | Result[str, CookieError]: the field value, after checking every part |
add(&mut head, &cookie) | Append a Set-Cookie field to a ResponseHead |
SameSite | Strict, Lax, or Unrestricted (SameSite=None) |
set_cookie gives Path=/, Secure, HttpOnly, SameSite=Lax and no
Max-Age, which ends the cookie with the browser session. Change the
fields to relax them; to_string checks the result each time:
| Rule | Error |
|---|---|
| Name is a non-empty token | InvalidName |
| Value has only cookie-octets | InvalidValue |
path is empty or starts with / and has only visible ASCII and space, no ; | InvalidAttribute |
domain has only letters, digits, - and ., and does not start or end with . | InvalidAttribute |
max_age is a nonnegative Duration | Positive fractions round up to whole seconds; zero removes the cookie |
SameSite=None needs secure | InvalidAttribute |
A __Secure- name needs secure | InvalidAttribute |
A __Host- name needs secure, path / and no domain | InvalidAttribute |
Rounding max_age is representable even for the largest Duration
(u64.MAX nanoseconds becomes 18446744074 seconds).
Empty path and domain are left out. CookieError also has
TooManyCookies, Http(HttpError) from add, Crypto(CryptoError) from
the HMAC in the signing functions, and Allocation.
Signing
| Item | Meaning |
|---|---|
sign(&keys, name, value) | Result[str, CookieError]: value.<tag> |
verify(&keys, name, signed) | Result[Option[str], CookieError]: the value when the tag matches under any key, else Nothing |
signed_cookie(&keys, name, value) | Result[SetCookie, CookieError]: set_cookie(name, sign(...)) |
find_signed(&cookies, &keys, name) | Result[Option[str], CookieError]: verify on each cookie named name, in order; the first whose tag verifies; Nothing when none does |
min_key | 32: shortest key, in bytes |
The tag is HMAC-SHA256 over name=value under a key, in unpadded base64url
(43 characters) after a .. The name is covered, so a value signed for one
cookie is refused under another. keys is a list[Bytes]: sign uses the
first, and verify tries every key without stopping at a match, each with
the constant-time comparison of Mac.verify, so the time taken depends only
on the number of keys. To rotate, put the new key first and keep the old
one listed until cookies signed with it have expired.
An empty list or a key shorter than min_key is InvalidKey; sign also
checks the name and value as set_cookie does. A malformed or mismatched
tag is Nothing, not an error. Signing allocates the result text and a
fixed-size HMAC state; verification allocates the decoded tag and the
returned value, and nothing per key beyond the HMAC state.
Signing stops changes, not reading: the value travels in clear. It does not expire or bind the cookie to a client; put an expiry time in the value, or keep sessions on the server, when that matters.
Limits
- No
Expires(usemax_age),Partitioned, orPriorityattributes. - No encryption; a signed value is still readable. A session identifier
can instead be a random token from
std.secrets, looked up on the server. Set-Cookiefields sent by another server are not parsed.
std.http.files
import std.http.files answers std.http.server requests
with files below one directory. The module is checked Plenty in
stdlib/std/http/files.plenty; it uses std.fs Root handles and
file_metadata, and adds no runtime primitives of its own.
| Item | Meaning |
|---|---|
serve(exchange, &root, path) | Result[i64, FilesError]: answer the request with the file at path below root; the status sent |
open_file(&root, path) | Result[Option[FileStream], FilesError]: the regular file at path, or Nothing when it is refused or missing |
media_type(name) | str: the Content-Type for name’s extension, from std.mime |
http_date(timestamp: &Timestamp) | Result[Option[str], FilesError]: an IMF-fixdate such as Sun, 06 Nov 1994 08:49:37 GMT; Nothing before 1970 or after 9999 |
parse_http_date(text) | Result[Option[Timestamp], FilesError]: a timestamp at a whole second, or Nothing unless text is an exact IMF-fixdate |
max_path | 4096: longest accepted path, in bytes |
routes.add("GET", "/static/{path...}", 0)?
...
path = exchange.params[0]
std.http.files.serve(exchange, &self.public, &path)?
import std.time
import std.fs
import std.io
import std.http.files
import std.path
from std.http.files import FilesError
def main() -> Result[(), AllocError | IoError | std.fs.FsError | std.io.StreamError | FilesError]:
parent = std.fs.root(&std.path.from_text(".")?)?
mut site = std.fs.temporary_dir(&parent, &std.path.from_text("site-")?, 8)?
for name in ["index.html", ".env"]?:
mut staged = std.fs.temporary_file(site.root())?
staged.stream.file.write("<h1>Welcome</h1>\n")?
staged.publish(&std.path.from_text(name)?)?
staged.close()?
for path in ["index.html", ".env", "../index.html", "missing.css"]?:
found = std.http.files.open_file(site.root(), &path)?
print(path, found.is_some(), std.http.files.media_type(&path))?
print(std.http.files.http_date(&std.time.Timestamp(784111777, 0u32))?)?
for name in ["index.html", ".env"]?:
site.root().remove_file(&std.path.from_text(name)?)?
site.cleanup()?
Ok(())
index.html True text/html; charset=utf-8
.env False application/octet-stream
../index.html False text/html; charset=utf-8
missing.css False text/css; charset=utf-8
Option[str].Some("Sun, 06 Nov 1994 08:49:37 GMT")
Confinement
path is relative and already percent-decoded, as a {name...} capture
is. open_file splits it on / and opens each segment through the previous
directory’s Root, so every component is opened with O_NOFOLLOW relative
to its parent. A path is refused when:
- it is empty or longer than
max_path, - a segment is empty, which covers an absolute path and
//, - a segment starts with
., which covers.,..and hidden files such as.envor.git, - it contains a NUL byte,
- any component is a symbolic link, the last is not a regular file (a
directory, FIFO or device), or opening a component fails because it does
not exist, is not a directory where one is needed, is a directory where a
file is needed, has too long a name, or is not permitted (
ENOENT,ENOTDIR,EISDIR,ENAMETOOLONG,EACCES,ELOOP,ENXIO).
Any other open failure, such as EMFILE, EIO or ENOMEM, is a server
fault rather than a refusal: open_file and serve return
FilesError.Io.
Cleaning the path as a string would not be enough: a link inside the
directory can lead out of it, and a path checked before opening can be
replaced before it is opened. Here the open itself refuses links, and the
file is opened non-blocking so a FIFO cannot stall the worker. Every
refusal is the same 404 Not Found, so a client learns nothing about what
lies outside the root or which entries are hidden.
Responses
| Request | Response |
|---|---|
Not GET or HEAD | 405 with Allow: GET, HEAD |
| Refused or missing path | 404, plain text |
If-None-Match lists the file’s tag, or is * | 304 with ETag and Last-Modified, no body |
No If-None-Match, and If-Modified-Since is not before the file’s time | 304 |
| Otherwise | 200 with Content-Type, Content-Length, X-Content-Type-Options: nosniff, ETag, Last-Modified; HEAD sends no body |
The tag is weak, W/"size-seconds.nanoseconds" from the open file’s size
and modification time, compared weakly as RFC 9110 requires for
If-None-Match. If-None-Match takes precedence over If-Modified-Since,
which is ignored unless it is an exact IMF-fixdate (the obsolete RFC 850 and
asctime forms are refused). Size and time come from std.fs.file_metadata
on the open file, so they describe what is sent even if the path is replaced
meanwhile. A metadata size above i64.MAX is rejected as
FilesError.Http(HttpError.InvalidContentLength) before validators are built,
even when a conditional request would otherwise answer 304. No wrapped size,
invalid header or file-body read is produced on that path.
The body is read in 16 KiB pieces into one buffer and written as it is
read, so memory does not depend on the file’s size. Exactly the size seen at
open is sent. A file that shrinks while it is sent is Truncated after the
head has gone out; the server then closes the connection.
media_type(name) is std.mime.from_file_name(name):
it compares the extension after the last . without case, allocates
nothing, and returns application/octet-stream for an extension not in the
table.
Errors
FilesError is Io(IoError) from opening a path component with an error
outside the refusal list under Confinement or from reading
the open file’s metadata, Stream(StreamError) from reading it,
Http(HttpError) from response framing/writing or an unrepresentable file
size, Truncated, and Allocation.
A refused or missing path is not an error.
Limits
- No
Rangerequests (206), noIf-Range, and no strong tags. - No directory index: a path naming a directory is
404, soindex.htmlmust be named in the route or the handler. - No compression, precompressed variants, or
Cache-Control; a handler that needs them builds the response itself fromopen_file, or a reverse proxy adds them. - A file is read with ordinary blocking reads on the worker; a slow filesystem holds the worker.
- Linux only, like
std.fs.
Acknowledgements
media_typereturns types registered in the IANA Media Types registry, throughstd.mime.
std.http.proxy
import std.http.proxy finds the client of a request that came through
reverse proxies. A forwarding header is believed only from a proxy the
program names. The module is checked Plenty in
stdlib/std/http/proxy.plenty and adds no runtime primitives. This page
also gives the deployment guidance for std.http.server.
import std.http.proxy
import std.net
from std.http import Header
from std.http.proxy import Forwarding, ProxyError
def main() -> Result[(), ProxyError | std.net.NetError | AllocError | IoError]:
mut proxies = std.http.proxy.proxies(Forwarding.XForwardedFor)?
proxies.trust("10.0.0.0/8")?
headers = [Header("X-Forwarded-For", "198.51.100.7, 10.0.0.3"), Header("X-Forwarded-Proto", "https")]?
for peer_text in ["10.0.0.2:443", "203.0.113.9:443"]?:
peer = std.net.parse_address(&peer_text)?
found = std.http.proxy.client(&peer, &headers, &proxies)?
print(std.net.format_ip(found.address)?, found.scheme, found.proxied)?
Ok(())
198.51.100.7 Option[str].Some("https") True
203.0.113.9 Option[str].Nothing False
Clients
| Item | Meaning |
|---|---|
proxies(header: Forwarding) | Result[Proxies, ProxyError]: trusts nobody yet |
proxies.trust(network) | Trust 10.0.0.0/8, fd00::/8 or one address |
proxies.trusts(&ip) | bool |
proxies.max_hops | Forwarding entries allowed in one request; 16 |
Forwarding | XForwardedFor (with X-Forwarded-Proto) or Forwarded (RFC 7239) |
client(&peer, &headers, &proxies) | Result[Client, ProxyError] |
client_of(exchange, &proxies) | client for an exchange’s peer and fields |
Client | address: IpAddress, scheme: Option[str], proxied: bool |
The rule:
- If the peer is not trusted, it is the client, and the forwarding headers are ignored.
- Otherwise the entries of the chosen header are read from the right. Each trusted address is a proxy; the first untrusted one is the client.
- If every entry is trusted, the leftmost is the client. If there are no entries, the peer is.
Entries left of the client were written by the client. Under
X-Forwarded-For they are never parsed as addresses, but they still count
toward max_hops. Under Forwarded every element is syntax-checked and
counted, so a malformed element or too many elements anywhere in the header
fails. The other header is always ignored, so a
deployment that uses Forwarded cannot be fooled with X-Forwarded-For.
An entry is 192.0.2.1, 192.0.2.1:8080, 2001:db8::1 or
[2001:db8::1]:8080; the port is dropped. An entry that must be read but is
none of these, such as unknown or an obfuscated _name, fails with
InvalidForwarded, as does a malformed Forwarded element. More than
max_hops entries is TooManyHops. Empty comma-separated pieces are skipped.
Quoted escapes must have a following byte; a trailing backslash is
InvalidForwarded. Scanners stop at the end of a field without advancing
an index beyond its length, including at the i64 representation limit.
An IPv4-mapped IPv6 address, as a
dual-stack listener reports IPv4 peers, is treated as the IPv4 address.
scheme is http or https, lowercased, when a trusted proxy reported it:
the client’s element’s proto= for Forwarded, or the last
X-Forwarded-Proto value, which the nearest proxy wrote. Any other scheme is
InvalidForwarded. It is Nothing when the peer is not trusted.
trust refuses a network with bits set below its prefix length
(10.0.0.1/8) with InvalidNetwork, so a typo cannot trust more than
intended. client_of fails with UnknownPeer when the exchange has no peer,
as with serve_connection on a custom transport.
Deploying behind a reverse proxy
std.http.server serves plain HTTP/1.1. For a public service, run it
behind a reverse proxy that terminates TLS:
- Listen on loopback or a private address, never directly on the public interface. Then every peer is a proxy.
- Configure the proxy to overwrite, not append to, the header the program reads, or to append only its own peer address. Trust exactly the proxy’s addresses.
- Let the proxy enforce request rates and connection limits per client; the server’s limits are per connection and per worker.
- Keep
config.limitsand the body limits below the proxy’s, so the server answers oversized requests itself. - Cookies default to
Secure, which browsers send only over HTTPS. The program sees plain HTTP from the proxy; checkschemerather than assuming.
nginx and Caddy configurations tested in front of the server are in
tests/fixtures/proxy/; the results are under
reverse proxies.
Serving HTTPS directly
Without a proxy, serve TLS from the program with
std.http.tls. Then it faces clients itself:
- Listen on the public address only through
std.http.tls.serve; a plainserveon the same host should only redirect or be absent. - No proxy is trusted, so leave
Proxiesempty:client_ofreturns the peer. - Keep the server’s limits, timeouts and
idle_connectionstight; nothing in front enforces per-client rates. - Cookies keep their
Securedefault, and the scheme is alwayshttps.
type HandlerError = std.http.HttpError | std.http.files.FilesError | std.http.cookie.CookieError | std.bytes.BytesError
type SetupError = std.http.server.ServerError | std.bytes.BytesError | AllocError | IoError | std.tls.TlsError | std.net.NetError
pub enum Route:
Files
Login
Me
pub class Site:
pub public: Root
pub keys: list[Bytes]
pub def handle(self, route: &Route, exchange: &mut Exchange[TlsTransport]) -> Result[(), HandlerError]:
match route:
case Route.Files:
path = exchange.params[0]
std.http.files.serve(exchange, &self.public, &path)?
case Route.Login:
mut head = std.http.response(204, "No Content")?
std.http.cookie.add(&mut head, &std.http.cookie.signed_cookie(&self.keys, "session", "user42")?)?
exchange.respond(head, &std.bytes.empty()?)?
case Route.Me:
found = std.http.cookie.cookies(&exchange.request.headers, 8)?
match std.http.cookie.find_signed(&found, &self.keys, "session")?:
case Some(user):
exchange.send_text(200, &user)?
case Nothing:
exchange.send_text(401, "sign in first")?
Ok(())
def main() -> Result[(), SetupError]:
mut table = std.http.server.router[Route]()?
table.add("GET", "/s/{path...}", Route.Files)?
table.add("POST", "/login", Route.Login)?
table.add("GET", "/me", Route.Me)?
keys = [std.bytes.from_text(&read_text("/etc/site/session.key")?)?]?
site = Site(std.fs.root(&std.path.from_text("/srv/site")?)?, keys)
tls = std.tls.server_config(&read_text("/etc/site/chain.pem")?, &read_text("/etc/site/key.pem")?)?
listener = std.net.listen(&std.net.parse_address("0.0.0.0:443")?, 128)?
stop = CancellationToken()?
report = std.http.tls.serve(listener, tls, &table, &site, &std.http.server.config(), &stop)?
Ok(())
tests/test_std_http_files.rs runs this program on loopback with the test
certificates in tests/fixtures/tls.
Static files
std.http.files serves files below a Root. It opens
the request path one segment at a time through directory handles, refuses
.., hidden names, links and anything but regular files, and answers
ETag and If-Modified-Since; see its limits (no Range, no directory
index). A reverse proxy can still serve static files when it is already in
front.
std.http.server
import std.http.server serves HTTP/1.1 over std.net TCP
connections: method and path routing, a fixed pool of worker threads fed by a
bounded admission queue, a deadline for each phase of a request, and
graceful shutdown. Framing comes from std.http.
import std.http.tls serves HTTPS through std.tls. Both modules
are checked Plenty in stdlib/std/http/ and add no runtime primitives. The tutorial lesson Serve HTTP requests
shows a complete server. Qualification records
which production gates the server passes, its measured baselines, and
deployment guidance; the server is production-ready.
The separate std.actor.http.server uses supervised
connection actors and nonblocking callbacks. Its
qualification is independent; the two modules
offer different execution and failure models and remain separate public APIs.
import std.bytes
import std.http
import std.http.client
import std.http.server
import std.net
import std.time
from std.http.server import Exchange
from std.net import TcpStream
from std.sync import CancellationToken
pub enum Route:
Hello
pub class Greeter:
pub def handle(self, route: &Route, exchange: &mut Exchange[TcpStream]) -> Result[(), std.http.HttpError | AllocError]:
name = exchange.params[0]
exchange.send_text(200, &f"hello, {name}"?)?
Ok(())
type VisitError = AllocError | IoError | std.bytes.BytesError | std.http.client.ClientError | RangeError
def visit(port: u16, stop: CancellationToken) -> Result[(), VisitError]:
with stop.cancel_on_exit():
never = CancellationToken()?
deadline = std.time.deadline_after(&std.time.monotonic_now(), &std.time.seconds(5u32))?
client = std.http.client.client(std.http.client.config()?)?
for path in ["/hello/Ada", "/missing"]?:
reply = client.get(&f"http://127.0.0.1:{port}{path}"?, &deadline, &never)?
print(reply.status, reply.body.to_text()?.strip()?)?
Ok(())
def main() -> Result[(), SpawnError[()] | std.http.server.ServerError | std.net.NetError | VisitError]:
mut routes = std.http.server.router[Route]()?
routes.add("GET", "/hello/{name}", Route.Hello)?
listener = std.net.listen(&std.net.loopback_v4(0u16), 16)?
port = listener.local_address()?.port
stop = CancellationToken()?
with spawn(visit, port, stop.share()).drop_job()? as task:
drop(std.http.server.serve(listener, &routes, &Greeter(), &std.http.server.config(), &stop)?)
task.join()?
Ok(())
200 hello, Ada
404 Not Found
Serving
| Item | Meaning |
|---|---|
serve[R, E, H: Handler[TcpStream, R, E]](listener: TcpListener, router: &Router[R], handler: &H, config: &Config, stop: &CancellationToken) | Result[Report, ServerError]: serve until stop is cancelled, then shut down |
std.http.tls.serve[R, E, H: Handler[TlsTransport, R, E]](listener, tls: ServerConfig, router, handler, config, stop) | serve over TLS; see HTTPS |
serve_with[T: Transport, O: Opener[T], R, E, H: Handler[T, R, E]](listener, opener: &O, router, handler, config, stop) | serve with a transport that opener makes from each accepted TcpStream |
serve_connection[T: Transport, R, E, H: Handler[T, R, E]](stream: T, router, handler, config, stop, cancel: &CancellationToken) | Report: serve every request on one connection, on the calling thread |
serve_connection_from[T, R, E, H](stream: T, peer: Option[SocketAddress], router, handler, config, stop, cancel) | serve_connection that records the connection’s peer address in each exchange |
protocol Handler[T, R, E] | def handle(self, route: &R, exchange: &mut Exchange[T]) -> Result[(), E] |
protocol Transport | read_before, write_before and shutdown with the signatures of TcpStream’s; write_before takes its source as &[u8] |
reason(status: i64) | str: the reason phrase for a common status, or "" |
E is the handler’s own error type: HttpError, or a sum such as
AllocError | HttpError. Calls infer it from handle, as they
infer R from the router; a generic function that passes a handler on
declares it too, as in def run[E, H: Handler[TcpStream, Route, E]](handler: &H).
The server needs nothing from the error: it answers and counts every Err
alike (see When a handler fails) and drops it on
the worker. A return type other than Result[(), E] does not satisfy
Handler:
App does not satisfy std.http.server.Handler: signature of `handle` does not match.
serve takes ownership of the listener and runs on the calling thread. It
starts config.workers scoped worker threads, then accepts connections on
its own thread and passes each to the queue once it has input (see
Idle connections). Every worker shares one borrow of the router, the
handler and the config, so handle takes self and route by shared borrow
and cannot change them; shared state needs a synchronized type such as a
channel or token. A handler class and its handle method must be pub,
because serve calls them from another module, and so must the route type
its signature names. The route type and the handler must be usable on
worker threads, under the ordinary
worker eligibility check.
serve fails only before serving: InvalidConfig for a config outside the
bounds below, Allocation or Thread if a channel or worker cannot be made,
or Net if its selector or wake-up socket cannot be set up.
A worker that cannot start stops the workers already running. After that,
every problem is answered on the wire where possible, counted in the
Report, and logged to config.events; nothing aborts the server.
serve_connection serves one already-open stream. Any class with the three
Transport methods works, and an in-memory stream makes handlers testable
without sockets; its shutdown can do nothing. The server calls shutdown
only with Shutdown.Write, before a lingering close. Its exchanges have
no peer; serve_connection_from takes one, for a transport that knows its
peer or a test that plays one. stop and
cancel have the meanings described under Shutdown.
Routing
| Item | Meaning |
|---|---|
router[R]() | Result[Router[R], ServerError]: an empty table whose routes carry values of type R |
router.add(method, pattern, route: R) | Result[(), ServerError]: append a route |
router.find(method, path) | Result[Match, ServerError]: the route for a request |
router.route(index) | &R: the value of the route at index, in the order routes were added |
Match.Found(index, params: list[str]) | index is the matched route’s; params holds the captures in pattern order, still percent-encoded |
Match.NotFound | No pattern matches the path |
Match.MethodNotAllowed(allow: str) | Patterns match but not for this method; allow lists the methods, comma-separated |
A pattern starts with /. Each segment is literal text, {name}, which
captures one nonempty segment, or, last only, {name...}, which captures the
rest of the path, possibly empty, slashes included. Names are letters,
digits and _. method is a method name or * for any method. A bad
method or pattern fails with InvalidPattern, and a route with the same
method and the same shape as an earlier one with DuplicateRoute.
Routes are tried in the order they were added; the first match wins, so add
specific routes before general ones. A GET route also answers HEAD, with
the body left out. Matching compares the path exactly: no case folding, no
trailing-slash redirect and no .. normalization. The query string is not
part of the path.
Before routing, the server checks the target and Host with
std.http.request_target and answers 400
when they break RFC 9112 section 3.2: an HTTP/1.1 request needs exactly one
Host, no request may have two, and a Host value must be a host with an
optional port. An absolute-form target (GET http://example.com/a HTTP/1.1)
is routed on its path, and its authority replaces Host. The check lives in
std.http so any server built on it applies the same rules; the parser
itself stays permissive so clients and proxies can read any well-formed
message.
The server answers 404, or 405 with an Allow field, without calling the
handler. A capture that does not percent-decode to UTF-8 gets 400.
Route values
Each route carries a value of the application’s route type R, normally an
enum with one variant per route. The server passes the matched route’s value
to handle as route, borrowed from the router, and the handler dispatches
with a match. Matching is exhaustive, so a variant the handler does not
answer is a compile error naming it (non-exhaustive match; missing Route.Hello(_)), and a handler whose route type is not the router’s is
rejected at the serve call (conflicting types for R).
pub enum Route:
Home
Hello(Greeting)
pub class App:
pub def handle(self, route: &Route, exchange: &mut Exchange[TcpStream]) -> Result[(), AllocError | HttpError]:
match route:
case Route.Home:
exchange.send_text(200, "home\n")?
case Route.Hello(greeting):
greeting.handle(exchange)?
Ok(())
mut routes = std.http.server.router[Route]()?
routes.add("GET", "/", Route.Home)?
routes.add("GET", "/index.html", Route.Home)?
routes.add("GET", "/hello/{name}", Route.Hello(Greeting("hello, ")))?
A variant’s payload can hold the route’s own handler object and its state,
as Hello holds a Greeting; it is used through a shared borrow, like the
handler. Several patterns may carry equal values, so two paths can share one
variant; DuplicateRoute depends only on method and pattern shape. router
needs its type argument written out: router[Route]().
The server does not generate the dispatch. A route enum cannot implement
Handler itself, because enums have no methods and protocols constrain only
classes, and a router of handler objects of different classes would need
existential protocol values, which Plenty does not have. One match in one
handler class keeps dispatch static and checked.
Exchanges
The handler receives an Exchange[T] for one request.
| Field or method | Meaning |
|---|---|
request: RequestHead | Method, target, version and fields as read |
authority: str | host[:port] the request names: an absolute-form target’s authority, else the Host value; empty only for HTTP/1.0 without Host |
path, query | The target’s path and the text after its first ?, both still percent-encoded; an absolute-form target’s scheme and authority are removed |
params: list[str] | Captures, percent-decoded |
peer: Option[SocketAddress] | The address the connection came from: a reverse proxy’s, when there is one (see std.http.proxy) |
read_body(&mut buffer, start, stop) | Result[i64, HttpError]: part of the request body; 0 from a non-empty range is the end, and an empty range returns 0 without reading |
read_all(&mut out) | Result[i64, HttpError]: append the rest of the body, within limits.max_body |
send(status, content_type, &body) | Whole response with Content-Type and Content-Length |
send_text(status, text) | send with text/plain; charset=utf-8 |
respond(head, &body) | Whole response with the caller’s head; Content-Length is added |
start(head) | Begin a streamed response |
write(source: &[u8]) | Send and flush part of a streamed body |
end() | Finish a streamed response |
stopping() | Shutdown has begun; a long response should end soon |
cancelled() | The drain deadline has passed; reads and writes now fail |
status(), sent() | Status sent (0 before), body bytes written (0 for HEAD) |
stop_token(), cancel_token() | Shares of the tokens behind stopping() and cancelled(), for work that outlives a call |
switch_protocols(head) | Result[(T, Bytes), HttpError]: answer 101 and take the connection; see Switching protocols |
detach(head) | Result[T, HttpError]: answer with a close-delimited body and take the connection; see Detaching a connection |
The response methods return Result[(), HttpError]. A handler answers once.
respond refuses a head that declares Content-Length or
Transfer-Encoding (ConflictingFraming), and a body on 204 or 304
(LengthMismatch). start streams exactly Content-Length bytes when the
head declares it; otherwise it adds chunked framing for HTTP/1.1 or closes
the connection after the body for HTTP/1.0. Writing past a declared length,
or ending short of it, is LengthMismatch; a HEAD response sends no body,
so it may end without writing. A streamed response the handler
leaves unended is ended after it returns. Streamed body progress remains exact:
a write that would make sent() exceed i64.MAX returns BodyTooLarge
before output or response-state mutation, leaving the earlier progress intact.
HEAD does not accumulate body bytes.
The first body read sends 100 Continue when the request asked for it. A
response sent before the body is read to its end closes the connection after
the response, since the next request cannot be found.
exchange.send_text(200, &exchange.query) is rejected: send_text borrows
the whole exchange exclusively while exchange.query is borrowed. Bind the
field first: query = exchange.query copies the str, sharing its storage.
Switching protocols
switch_protocols(head) sends head, which must have status 101, as
given, and hands the connection to the caller with any input that arrived
after the request head. The head carries the Upgrade and Connection
fields the new protocol needs. The request body must be empty or read to its
end (Unfinished otherwise), and the exchange must be unanswered (Broken
otherwise). The exchange then counts as answered; the server neither reuses
nor closes the connection, and the handler owns it until it returns or
drops it. The connection still occupies the worker for as long as the
handler runs. std.http.websocket’s accept is built on
it.
Detaching a connection
detach(head) sends head with Connection: close added and hands the
connection to the caller, which writes a body that ends when it closes the
connection: a long-lived stream that the server’s response_timeout would
otherwise cut off. The status must be 200 to 999 (InvalidStatusLine
otherwise) and allow a body (LengthMismatch for 204 and 304), and the
head must not declare Content-Length or Transfer-Encoding
(ConflictingFraming). A HEAD request cannot detach (Broken), since its
response has no body; answer it with start and end. The rules for the
request body and the handler’s ownership are those of switch_protocols;
input that arrived after the request head is dropped. A refused head leaves
the exchange unanswered. To watch for shutdown, take shares of
stop_token() and cancel_token() and pass them to the stream’s waits.
std.http.sse’s accept is built on it.
When a handler fails
| Handler outcome | Response | Counted as |
|---|---|---|
Ok after answering | As sent | requests |
Ok without answering | 500 | handler_errors |
Err before answering, after a body read failed | 413 (too large), 408 (timed out) or 400 | protocol_errors |
Err before answering, otherwise | 500, connection closed | handler_errors |
Err after a streamed response started | Connection closed mid-body | handler_errors |
Configuration
config() returns a Config with every field public:
| Field | Meaning | config() |
|---|---|---|
workers | Worker threads, 1 to 1024 | 4 |
queue | Accepted connections waiting for a worker, at least 1 | 16 |
limits | std.http Limits for every request | std.http.limits() |
idle_timeout | Wait for the first byte of the next request | 5 s |
head_timeout | Read a request head, from its first byte | 10 s |
body_timeout | Read a request body, from the end of its head | 30 s |
write_timeout | Any one write | 10 s |
response_timeout | Whole response, from the end of the request head | 60 s |
drain_timeout | Shutdown grace for active connections | 5 s |
max_requests | Requests on one connection before it closes, at least 1 | 1000 |
idle_connections | Connections waiting on the accepting thread for a request, the rest of its head or the rest of a handshake, 0 to 1,000,000 | 256 |
handshake_timeout | HTTPS only: finish the TLS handshake, from the accept | 10 s |
events | Option[ChannelSink] for std.log events | Nothing |
Limits must be positive (max_body may be 0).
Idle connections
Unless idle_connections is 0, a worker does not wait for a client to
send a request. A connection waiting
for its first byte, for the next request after a keep-alive response, or
for the rest of a request head is parked on the accepting thread’s
std.ready selector instead of holding a worker. When the
first input arrives it joins the queue, and a worker reads what has
arrived. If that is not yet a whole head, the worker hands the connection
back with those bytes; the accepting thread then reads the rest as it
arrives and queues the connection once the head is complete. A worker
serves the request and, if the connection stays open with no next head
complete, returns it to be parked again. Ten keep-alive clients can
therefore share one worker, and clients that send heads slowly hold no
worker at all.
A head is complete for this purpose once its final empty line has arrived,
or once it is bound to fail: a line over max_line, a head over
max_head, a line ending in a bare LF, or a field line past
max_headers. A parked connection with a partial head holds those bytes,
at most max_head (64 KiB by default) and usually a 4 KiB buffer.
Parked connections close at idle_timeout, or answer 408 at
head_timeout once a head has started.
idle_connections caps the parked connections, including those with
partial heads, so it bounds how many slow clients the server waits for: a
new connection is refused when it is reached. A connection partway through
a head or a handshake that a worker hands back is parked even past the cap,
since it was admitted while there was room; at most
queue + 2 * workers + 64 such connections can be outside the parking
slots at once. With 0, nothing is parked: each connection goes to the queue at
once and a worker holds it until idle_timeout, reading each head as it
arrives, as a thread-per-connection server would.
HTTPS
import std.http.tls
import std.tls
tls = std.tls.server_config(&chain_pem, &key_pem)?
report = std.http.tls.serve(listener, tls, &table, &App(), &config, &stop)?
std.http.tls.serve takes a std.tls ServerConfig and calls a
handler of type Handler[TlsTransport, R, E]; TlsTransport.stream is the
TlsStream. Each handshake must finish within handshake_timeout of the
accept. Workers advance it as the client’s messages arrive, and between
them the connection waits parked, so a client that stalls mid-handshake
holds no worker. The deadline covers time spent parked and queued, so a
client that connects and stays silent is closed at that deadline, not at
idle_timeout. After the handshake, connections are served and parked like
plain ones: a TLS stream is only parked when it holds no decrypted input,
and a partial head is decrypted and gathered on the accepting thread. The
plain serve does not import std.tls.
serve_with accepts any Opener[T], the protocol behind std.http.tls:
| Method | Meaning |
|---|---|
open(self, tcp: TcpStream) | Result[T, OpenError]: the transport, made on a worker without waiting for the peer |
advance(self, stream: &mut T, deadline: &Deadline, cancel: &CancellationToken) | Result[(), OpenError]: advance the opening handshake until it completes or deadline passes (TimedOut); with a deadline already passed, do only what is possible without waiting |
handshakes(self) | bool: use handshake_timeout for new connections and close, not 503, those refused |
watch(self, stream: &T, selector: &mut Selector, key: u64) | bool: register a stream for what it waits for; called only after a read or advance that did not wait has timed out, so no input is buffered |
unwatch(self, stream: &T, selector: &mut Selector) | Remove it |
A worker calls advance with an expired deadline and parks the connection
while it returns TimedOut; with idle_connections set to 0, or once
shutdown has begun, it waits up to handshake_timeout instead. OpenError
is TimedOut, Cancelled, Failed (counted in protocol_errors) or
Gone (the client left or refused the handshake; not counted).
Deadlines and faults
Every read and write waits under a deadline, so a slow or silent client cannot hold a worker longer than the phase allows. With parking on, a worker waits for neither a request head nor a handshake (see Idle connections).
| Situation | Server action |
|---|---|
No request starts within idle_timeout | Close quietly |
HTTPS: no handshake within handshake_timeout of the accept | Close; counted in timeouts |
| HTTPS: the client closes or sends a fatal alert during the handshake, for example because it rejects the certificate | Close; not counted |
| HTTPS: handshake fails otherwise, for example a plaintext request | Close; counted in protocol_errors |
idle_connections connections already waiting | A new connection gets 503 with Retry-After: 1 (HTTPS: is closed); counted in rejected. A served connection going idle is closed; one partway through a head or a handshake is parked anyway |
Head not complete within head_timeout | 408, close; counted in timeouts |
| Malformed head, or a line, head or header count over a limit | 400, or 431 for size limits, 505 for an unsupported version, 501 for an unsupported transfer coding; close |
Ambiguous framing (Content-Length with Transfer-Encoding) | 400, close |
Declared Content-Length over max_body | 413 before reading, close |
HTTP/1.1 without Host, several Host fields, or an invalid Host value | 400, close |
Target neither origin-form (/path) nor http/https absolute-form | 400, close |
| Part of a head arrives, then nothing | Parked without a worker; 408 from the accepting thread at head_timeout, close; counted in timeouts |
Body read past body_timeout, or a write past its deadline | Close; counted in timeouts |
| Queue full | 503 with Retry-After: 1 (HTTPS: close without a response), close; counted in rejected |
accept fails, for example out of file handles | Count in accept_errors, pause 10 ms, keep accepting |
Responses written by the server itself carry Connection: close. Allocation
failure while serving closes that connection.
Lingering close
Closing a socket while request bytes are still unread makes the kernel
reset the connection, and a client that has not yet read the answer can
lose it. So after a response that may leave request input unread, the
server ends its output instead (Shutdown.Write; over HTTPS, close_notify
first) and discards input until the client ends its own, 16 MiB have been
discarded, or 2 s (at most write_timeout) have passed, and then closes. The
client reads the whole answer and a clean end. The budget is large because a
reverse proxy keeps forwarding a refused body up to its own limit, and
answers 502 instead of the refusal if the connection resets under it. This applies to the 400,
408, 413 and 431 refusals above, to a handler failure answered with a
body-read status, and to a handler response sent before the request body
was read to its end.
With parking on, the discarding happens on the accepting thread: the
connection is parked in a slot (and so counts toward idle_connections)
and closed when its input ends, its budget is spent, or its deadline
passes. With idle_connections 0, under serve_connection, or once
shutdown has begun, the worker discards before taking its next connection.
A parked 408 from the accepting thread lingers the same way. Shutdown
closes lingering connections that are parked at once.
Bytes of a pipelined request that arrived with the previous one start that
request: its head_timeout runs from the end of the previous response, and
an end or stall inside it is answered (400 or 408) rather than closed
quietly. Blank lines alone do not start a request.
A head is parsed once it is complete (see
Idle connections), so a malformed line followed by
nothing gets 408 at head_timeout, not 400 at once. 408 and 503
answers written on the accepting thread get a write budget of
write_timeout, at most 100 ms.
Accepted connections have Nagle’s algorithm turned off (TCP_NODELAY).
Responses are buffered and flushed whole, so this sends no extra small
segments, and a head followed by a body larger than the 4 KiB write buffer
is not held back for a delayed acknowledgement. Over HTTPS the server
also acknowledges the client’s TLS Finished at once, so a client that
leaves Nagle’s algorithm on sends its first request without waiting for a
delayed acknowledgement (see std.tls).
Shutdown
Cancelling stop begins shutdown:
servestops accepting and closes the listener.- Connections already queued are still served. A connection waiting idle
for its next request, or parked mid-handshake, closes at once; one whose
request has started arriving is handed to a worker, which waits for the
rest of its head, and served. Every response carries
Connection: close. - Handlers see
stopping()and should finish long responses. - At
drain_timeout,servecancels the second token. Every further read or write fails withCancelled,cancelled()is true, and the connection closes; counted incancelled. servejoins every worker and returns the mergedReport. Every accepted connection is closed by then. A client may still be reacting to that close:servereturning says nothing about the client’s progress, so join a client task before relying on its output.
Cancellation is cooperative. A handler that loops without reading, writing
or checking cancelled() keeps its worker, and serve waits for it.
To drain when a supervisor sends SIGTERM (or on Ctrl-C), run
std.signal.cancel_on on a thread beside
serve: it cancels stop when the signal arrives. A second signal ends the
process at once.
Reports and events
Report counts, as u64: connections, rejected, requests,
handler_errors, protocol_errors, timeouts, cancelled,
accept_errors and dropped_events. These observation-only counters saturate
at u64.MAX, both when incremented and when worker reports are merged;
exhausting a counter does not interrupt serving or change an operation’s error.
Body-byte progress and framing limits do not saturate.
With events set, each answered request emits an Info event request
with fields method, target, status, bytes (body bytes sent) and
micros (from head to end of response). Faults emit Warn events: bad request, bad request body, handler failed, handler sent no response,
timed out, cancelled, queue full, too many connections, handshake timed out, handshake failed and accept failed, most with a
status field. An event the sink cannot take is dropped and counted in
dropped_events; logging never blocks serving.
Limits
- The accepting thread waits on one selector of at most 64 events per wait, and a parked connection costs a list slot, room for one free-slot index, and a selector registration. Parking reuses a vacated slot in constant time; each wait still scans every slot for expired deadlines. A worker serves one connection at a time, from a complete head to the end of the response.
idle_connectionsbounds the slow clients the server waits for: with that many partial heads parked, new connections get503until theirhead_timeoutpasses. Raise it, and the descriptor limit, for exposed servers; each parked partial head holds at mostmax_headbytes.- HTTPS connections end without a TLS
close_notify, except those closed lingering. Responses are length-delimited or chunked, so clients do not depend on it, except for a streamed response to HTTP/1.0, whose end is the close and so cannot be told from truncation. - HTTPS has the
std.tlslimits. The server speaks only HTTP/1.x, so itsServerConfigshould offer no ALPN protocol buthttp/1.1. - No
Dateheader, compression, request trailers in the exchange, or HTTP/2. A connection taken byswitch_protocolsordetachholds its worker until the handler returns, so long-lived WebSockets and event streams count againstworkers. Forms, JSON, uploads, cookies and forwarded clients are instd.http.body,std.http.cookieandstd.http.proxy; static files instd.http.files. - A
503rejection reads only bytes that have already arrived. If more of the request arrives after the close, the client may see a reset instead. - A lingering close discards at most 16 MiB in at most
2 s. A client still sending past that, such as a large upload refused
with
413withoutExpect: 100-continue, can still see a reset; pastidle_connections, a refusal closes at once. - Captures in
paramsare decoded;pathandqueryare not. Hostand absolute-form authorities are checked against the URI grammar, not against names the server answers for. A handler that serves only some hosts comparesexchange.authorityitself.OPTIONS *andCONNECTtargets get400.- A trap in a handler ends the process; see handler errors and traps.
Acknowledgements
- Route patterns (
{name},{name...}) follow Go’snet/httpServeMuxpatterns.
HTTP server qualification
This page records what was checked before std.http.server
was labelled production-ready, what that showed, and how to deploy it.
The server is production-ready: every gate passes; see Gates. HTTP/2 and HTTP/3 need
separate adoption decisions and are not part of these gates.
The suites are in tests/test_std_http_server_qualification.rs, with the
request corpus in tests/fixtures/http_server/, and in
tests/test_std_http_server_proxy.rs, which runs the server behind nginx and
Caddy in Docker (see Reverse proxies). The benchmark is
examples/http_server_bench.py. Everything runs on loopback; only the proxy
tests need a network, once, to pull their images.
The current side-by-side TLS comparison records worker and actor throughput, handshake latency, idle RSS and trap recovery after actor TLS heap isolation and completion-credit changes. The original baselines on this page remain historical measurements of their recorded revision.
Gates
| Gate | Status | Evidence |
|---|---|---|
| Parser conformance and a retained corpus | Pass | About 80 corpus requests, each replayed in 64-byte reads and one byte at a time; 600 mutated copies. Includes the RFC 9112 section 3.2 Host rules and absolute-form targets |
| Fuzzing | Pass | Coverage-guided libFuzzer runs of the whole connection loop, seeded from the corpus, with every response status checked: 16.9 million inputs in 30 minutes, no finding. A short run is part of the test suite (test_fuzz.rs), beside deterministic corpus mutations here and in test_std_http.rs. See Fuzzing |
| Interoperability | Pass | curl over plain HTTP and TLS (reuse, sized, chunked and 100-continue uploads, HTTP/1.0, HEAD, chunked and 64 MiB responses, 404/405); std.http.client (pooling, streamed chunked upload, large downloads) |
| Slow clients | Pass | Idle and silent clients, partial request heads, partial TLS handshakes, slow readers and slow body senders are cut off at their deadlines without holding the server; heads and handshakes wait parked, not on a worker. See Slow request heads |
| Refusals without a reset | Pass | Refusals of requests not read to the end (431, 413, 400, a failed body read), each followed by more input, end with the whole answer and a clean end, over plain TCP (parked and on a worker) and TLS (close_notify). See Refusals and resets |
| Shutdown under sustained load | Pass | Six clients sending keep-alive requests while shutdown begins, each reusing its connection, with responses still arriving after the stop signal: every response received is whole, the server’s count matches, and serve returns within the drain deadline |
| Supervisor stop | Pass | With std.signal.cancel_on, SIGTERM during a streamed response begins shutdown: the response ends whole, serve returns its Report and the program exits 0. A second signal ends the program at once (test_std_signal.rs) |
| Allocation failure | Pass | Every allocation on the accepting thread fails in turn; request handling fails in turn through serve_connection (test_std_http_server.rs). No trap, no leak, and the request is served once allocation succeeds |
| Resource leaks | Pass | No descriptor survives a load round or a fault round (bad heads, truncated heads, silent and rejected clients), and the allocation-checks runtime finds no live heap after either |
| Measured baselines | Published | Baselines, over plain HTTP and HTTPS |
| Unused backends | Pass | A JSON service using the server and client links none of rustls/ring, SQLite or zlib-rs |
| Relocated offline toolchain | Pass | The compiler binary, copied alone into a fresh directory and run with an empty environment except PATH, builds and runs a JSON service. Installed alone as PREFIX/bin/plenty with a fresh home, it fetches a package’s Git dependency once; with the repository deleted and git replaced by a failing stub, plenty build --offline --locked and plenty run --offline build a CLI/data tool (std.cli, std.csv, std.json, typed std.error reports and exit statuses) and a JSON service over the same dependency. Both run and link no unused backend |
| TLS and proxy deployment | Pass | Direct TLS is tested with curl and std.tls clients. Behind nginx 1.30.6 and Caddy 2.11.7: keep-alive on both sides of the proxy, buffered and streamed bodies, 100-continue, forwarded Host and X-Forwarded-*, TLS terminated at the proxy, slow heads, a slow upstream, refusals and a drain on SIGTERM. Opt-in locally (PLENTY_DOCKER_TESTS=1) and run by CI; see Reverse proxies |
Supported platforms
x86-64 Linux with the GNU C library, the only target of
std.net and std.ready (epoll). Results here were
taken on Linux 7.2 (Fedora 44). No other target is supported.
Application error composition
The installed offline package test also checks B14/B71 through its shared CSV
summary dependency. The CLI reports invalid integer data with its record number
and ParseError, malformed row width with the complete CsvError, and a missing
file with its nested stream/I/O cause. Each report retains the operation and path,
prints no partial JSON, and exits 1; argument errors still exit 2 and help exits 0.
The JSON service returns those same causes as Context[SummaryError] in its
handler error sum and reports them before propagation. One client repeats all
three failures and a successful summary three times. The test checks identical
CLI/service diagnostics, generic 500 responses without private error details,
successful requests after failures, nine handler errors among twelve requests,
and no protocol errors. After the client, workers, CSV readers and listener have
dropped, the process has the same number of open descriptors as before the run.
This qualifies application composition over the CSV/HTTP APIs. The client now
uses typed head parsing, writes and fixed/chunked/close-delimited framing.
Separate typed-stream and client checks cover exact partial progress, original
transport causes, interruption, bounded heads, trailers, compatibility readers
and connection recovery; HTTPS checks cover verified and tunnelled requests.
Handler errors and traps
A handler reports a problem by returning Err; the server answers it, counts
it and goes on serving (see When a handler fails).
A fatal runtime error is different. An index out of range, integer overflow,
division by zero, unwrap of an Err, or worker stack exhaustion terminates
the whole process at once, without unwinding, with a message such as
error: main.plenty:3:11: index out of bounds: index 7, length 3 and exit
status 1. Every connection is cut off and no Report is returned.
So in a handler:
- Check indexes and arithmetic on anything that comes from a request, use
optional lookups and
?, and returnErrrather than callingunwrap. - Run the server under a supervisor that restarts it (systemd
Restart=on-failure, a container restart policy). - Allocation failure is not a trap: it closes the affected connection.
Baselines
examples/http_server_bench.py builds examples/http_server_bench.plenty
with --release, runs it with 4 workers on CPUs 0, 2, 4 and 6 and
ApacheBench 2.3 on CPUs 8, 10, 12 and 14 (one hardware thread per physical
core, so at most ten threads are busy), and repeats each workload against a
fresh server. With --tls it builds examples/https_server_bench.plenty
instead, the same handler served by std.http.tls with the ECDSA P-256
certificate in tests/fixtures/tls.
Measured on 2026-10-18 at revision 32c48c0, built by rustc 1.99.0 (target
x86_64-unknown-linux-gnu (CPU baseline: x86-64/SSE2)), on an AMD Ryzen 7
7840HS w/ Radeon 780M Graphics with Linux 7.2.9-201.fc44.x86_64, three runs
per workload. Other builds were running (load average about 4.5 at the
start of each run), so treat the ranges as noise bounds.
| Workload | Requests/s, median (range) | p50 / p99 latency, ms | Peak RSS, KiB | Failed |
|---|---|---|---|---|
| keep-alive, 1 client | 5560 (5536–5662) | 0.17 / 0.36 | 6212 | 0 |
| keep-alive, 16 clients | 33314 (32772–33388) | 0.10 / 2.96 | 6212 | 0 |
| keep-alive, 64 clients | 32853 (29653–32959) | 0.10 / 9.03 | 6236 | 0 |
| new connection per request, 16 clients | 16106 (16085–16137) | 0.93 / 1.64 | 6212 | 0 |
| keep-alive 16 KiB response, 16 clients | 24430 (24323–25589) | 0.14 / 4.30 | 6152 | 0 |
| keep-alive 1 KiB POST echo, 16 clients | 25850 (25783–25897) | 0.16 / 1.84 | 6160 | 0 |
HTTPS
The same workloads over TLS 1.3 (TLS_AES_256_GCM_SHA384, X25519, ECDSA
P-256 server key; rustls 0.23.46 with ring), measured in the same session:
| Workload | Requests/s, median (range) | p50 / p99 latency, ms | Peak RSS, KiB | Failed |
|---|---|---|---|---|
| keep-alive, 1 client | 3891 (3872–3913) | 0.20 / 0.51 | 8196 | 0 |
| keep-alive, 16 clients | 25892 (25038–26697) | 0.13 / 2.05 | 9352 | 0 |
| keep-alive, 64 clients | 24474 (24459–24516) | 3.33 / 6.19 | 11136 | 0 |
| new connection per request, 16 clients (curl) | 4196 (3939–4235) | 1.43 / 3.76 | 8792 | 0 |
| keep-alive 16 KiB response, 16 clients | 17858 (17382–19415) | 0.19 / 3.69 | 9268 | 0 |
| keep-alive 1 KiB POST echo, 16 clients | 19606 (18753–19782) | 0.91 / 1.49 | 9292 | 0 |
Each new HTTPS connection is a full handshake: the server keeps no session
cache and sends no tickets, so clients cannot resume. ApacheBench cannot
measure that workload: it counts connections closed without close_notify
as failed. The script runs curl 8.18.0 (OpenSSL 3.5.9) instead: four processes on the
client CPUs, each with --parallel-max 4, TCP_NODELAY (curl’s default)
and no certificate verification. The same curl method against the plain
server gave 21397 (21225–22032) requests/s, p50 / p99 0.67 / 1.05 ms, so the
handshake, not the client, sets the HTTPS rate.
With 1000 parked keep-alive connections the server’s resident memory grew
from 4364 KiB to 6680 KiB over plain HTTP, about 2.3 KiB per idle
connection, and from 5928 KiB to 48552 KiB over HTTPS, about 42.6 KiB per
idle connection: each TLS session keeps its rustls buffers and keys. Size
idle_connections for HTTPS with that in mind.
Latency is per request as the client sees it. Peak RSS is the server’s
VmHWM. 16k is a 16 KiB response; the POST echoes a 1 KiB body. Each
keep-alive connection closes after max_requests (1000) requests, so
ApacheBench reconnects. An earlier plain run, on 2026-10-17 at load average
about 8, gave 9239 to 16744 requests/s for the same workloads; the
difference is machine load, not a server change.
Limits of these numbers: one machine, loopback only, a client that is itself a bottleneck at high rates, small fixed responses, one certificate type and no TLS session resumption. They are a regression baseline, not a capacity plan.
Before this qualification, a response whose head and body went out as two writes (any body over the 4 KiB write buffer) waited about 40 ms for a delayed acknowledgement: the 16 KiB workload ran at 388 requests/s. The server now turns off Nagle’s algorithm on accepted connections.
Until issue #148 was fixed, a client that leaves Nagle’s algorithm on
(ApacheBench, Python’s socket) waited about 40 ms on each new HTTPS
connection: a TLS 1.3 server sends nothing after the client’s Finished, so
the client held its first request until the kernel’s delayed
acknowledgement. ab -n 20 -c 1 without -k took 42–44 ms per request.
The server now acknowledges the Finished at once; on 2026-10-18 the same
client took 0.76 ms per request over 200 requests (unpinned, not the
baseline method above).
Slow request heads
Until issue #132 was fixed, a worker read each request head as it arrived,
so a client that sent part of a head held a worker until head_timeout.
Now a partial head, like a partial TLS handshake, waits on the accepting
thread (see Idle connections).
Measured on 2026-10-17 with the scenario of slow_heads_do_not_hold_workers
below, also run with the defaults: clients that each send
GET / HTTP/1.1\r\nHost: x and stall, then one complete request.
| Configuration | Before | After |
|---|---|---|
2 workers, head_timeout 1 s, 6 slow clients | Answered after 2954 ms | Answered within 2 ms |
config() defaults (4 workers, 10 s), 4 slow clients | Answered after 9959 ms | Answered within 2 ms |
config() defaults, 64 slow clients | 503 at once; 44 slow clients got 503, the last 408 came after 50 s | Answered within 2 ms; all 64 got 408 at 10 s |
The slow clients still count against idle_connections (default 256):
with that many parked, new plaintext connections get 503 until slots become
available (new HTTPS connections close). Already-admitted partial heads and
handshakes returned by workers can park beyond that admission cap; see the
bounded overflow and parking-disabled behavior.
With 1000 parked partial heads the server’s resident memory grew
from 4548 KiB to 6396 KiB, about 1.9 KiB per connection. Size
idle_connections and the descriptor limit for the slow clients an exposed
server must outlast. Keep-alive and new-connection throughput showed no
change beyond run-to-run noise on a loaded machine (load average about 8).
slow_heads_do_not_hold_workers in the qualification suite and
partial_heads_wait_without_a_worker and
https_partial_handshakes_and_heads_wait_without_a_worker in
test_std_http_server.rs check this.
Refusals and resets
Until issue #138 was fixed, a refusal of a request the server had not read
to the end closed the socket with request bytes unread, so Linux reset the
connection. Measured on 2026-10-17 against
examples/http_server_bench.plenty, ten connections per case, each
sending the request and then reading to the end:
| Request | Before | After |
|---|---|---|
Head with a 20000-byte field (431) | 10 resets | 10 clean ends |
Content-Length over max_body and 200 KB of body (413) | 10 resets | 10 clean ends |
Malformed field line and 100 KB more (400) | 10 resets | 10 clean ends |
Bad chunk size and 100 KB more (400 from a failed body read) | 10 resets | 10 clean ends |
The results were the same with IDLE=0, where the worker lingers. Before
the fix a client reading to the end saw ECONNRESET, sometimes after
reading the answer. Now the
server ends its output, discards up to 16 MiB (64 KiB until the proxy tests
below) for up to 2 s, then closes
(see Lingering close).
refusals_end_cleanly_instead_of_resetting and
tls_refusals_end_with_close_notify in the qualification suite check this.
Fuzzing
fuzz/http_server/main.plenty is a libFuzzer target:
plenty --fuzz counts every block the compiled program runs, including the
standard library’s, and libFuzzer keeps the inputs that reach new ones. Each
input is one client connection served by serve_connection over an
in-memory stream. Its first byte selects the read size (1 to 64 bytes),
small limits (a 64-byte line, a 256-byte head, 4 fields, a 16-byte body)
and a limit of two requests per connection; the rest is what the client
sends. The routes are those of the corpus. Response bodies never repeat the
request, so the target finds every response head in what the server wrote
and traps on a status outside 100, 200, 400, 404, 405, 413, 431, 500, 501
and 505. Traps of the server itself, stack exhaustion and hangs over 10 s
are findings too.
fuzz/http_server/run.py builds the target with --release, seeds it with
every corpus case in four variants (64-byte reads, 7-byte reads, 1-byte reads,
7-byte reads with small limits) and runs it with the HTTP dictionary fuzz/http_server/http.dict.
Findings go to WORK/findings; the default WORK is
target/fuzz-http-server. Findings persist across runs: the script lists
only the files the current run added, and exits nonzero only for those or for
a libFuzzer failure.
cargo build --release --locked
python3 fuzz/http_server/run.py --time 1800 --jobs 2
target/fuzz-http-server/http_server_fuzzer FINDING # replay one finding
On 2026-10-18, on the machine of the baselines, two jobs ran
for 30 minutes each (rustc 1.99.0, clang 22.1.8’s libFuzzer), starting from
the corpus of an earlier 10-minute run: 16.9 million inputs, about 4,700 per
second per job, with no finding. 3,777 of the program’s 13,878 blocks were
reached, nearly all within the first seconds; most of the rest belong to
functions the target never calls, such as serve and TLS, which a fuzz
build keeps.
To act on a finding, replay it, reduce it with -minimize_crash=1 -runs=100000,
and handle it as a parser defect under
Security fixes: its request joins the
corpus, and so the seeds.
Not covered: TLS records (rustls and ring are fuzzed upstream), the
std.ready parking of real sockets, and timeouts, since the in-memory stream
never waits.
Reverse proxies
tests/test_std_http_server_proxy.rs puts nginx and Caddy, each in its stock
image, in front of the server, and drives them with curl and raw sockets. The
server runs on the host (it links the host’s C library, so it does not run on
Alpine); the proxy container uses --network host and reaches it on
loopback. The configurations are tests/fixtures/proxy/nginx.conf and
tests/fixtures/proxy/Caddyfile; the server is
tests/fixtures/proxy/server.plenty.
| Image | Digest |
|---|---|
nginx:1.30.6-alpine3.24 | sha256:970364228b1e4f932abc22d321932816782cca8a20b85a77552a7cfe9acfb3b5 |
caddy:2.11.7-alpine | sha256:d8542f48d34a9cf4e4c11a478865229840e87e4c96ea3f439101f31a5d35f75f |
Each test runs against both proxies. Clients connect from 127.0.0.2, so the
handler can tell them from the proxy at 127.0.0.1.
The harness reserves distinct TCP ports and the TLS port’s UDP socket through
configuration setup, then releases them immediately before starting Docker.
Caddy also opens that UDP port for HTTP/3; this is not HTTP/3 qualification.
Process-local logical leases retain both port numbers across that handoff and
until the proxy’s Drop cleanup completes, including startup failure and unwind.
Concurrent cooperating fixtures reject an already claimed port in either role
and select another pair; only registry operations hold the mutex, not Docker
startup or scenarios. Deterministic non-Docker tests force reuse during the
socket-free gap, check cross-role exclusion and RAII release, and stress eight
concurrent reservers (issue #719). These are helper tests, not renewed real-proxy
qualification. A separate exact batch on the reviewed port-ownership source
passes all eight worker and two actor nginx/Caddy cases with two test threads
(local PR #918), including actual container removal and unchanged foreign
container state. This is bounded proxy qualification, not a whole-workspace or
performance result. Unrelated processes remain outside this ownership registry.
Host networking still has a socket handoff window, so readiness checks a unique
proxy-local response over both HTTP and verified TLS, without adding upstream
requests. An unexpected HTTP status or identity fails immediately rather than
being retried. Shutdown setup failures retain response headers and body,
listener/upstream ports, container logs and upstream stderr. These checks harden
endpoint identification; they do not establish the cause of the intermittent
pre-shutdown 400 reported in issue #647.
| Test | Checks |
|---|---|
*_forwarding_keep_alive_and_tls | exchange.authority and Host keep the client’s Host; X-Forwarded-Proto is http or https; client_of, trusting only 127.0.0.1, finds the client and ignores a spoofed X-Forwarded-For; curl reuses its connection to the proxy; twelve requests reach the server over at most three connections; TLS terminated at the proxy, with a 700 KB upload |
*_bodies_and_continue | Sized, chunked and Expect: 100-continue uploads of 700 KB and a 1 MiB chunked download, through a buffering and a streaming location; a client that waits for 100 Continue before sending; a streamed response reaches the client before the handler ends it |
*_slow_clients_timeouts_and_refusals | A partial request head is closed by the proxy at its 2 s header timeout and never reaches the server; a handler slower than the proxy’s 2 s upstream timeout gets 504 and the next request is served; five rounds of 413 refusals of 2 MiB bodies and early answers to 1000 KiB bodies the handler does not read, sized and chunked, buffered and streamed, all reach the client |
*_shutdown_drains | SIGTERM while a response streams through the proxy: it ends whole, the server exits 0 with its report, and the proxy then answers 502 |
Differences between the proxies, none of them a server fault:
- nginx appends to a client’s
X-Forwarded-For(203.0.113.9, 127.0.0.2); Caddy, trusting no proxies by default, replaces it (127.0.0.2). With only the proxy trusted,client_ofgives127.0.0.2for both. - nginx sends
100 Continueitself and buffers the body. Caddy streams it and also relays the server’s own100 Continue, so the client sees two interim responses, which RFC 9110 section 15.2 requires clients to accept. - Caddy offers HTTP/2 to TLS clients and speaks HTTP/1.1 to the server.
- After relaying an answer sent before a large body was read, Caddy closes
the client’s connection with the rest of the body unread, so a client
still sending may see a reset after the answer. Curl can stop on its
upload error before reading the final response. The Caddy refusal and
early-answer checks therefore receive concurrently with the upload and
require the final status, a single correct
Content-Length, and the exact complete body before accepting a reset. A lone100 Continue, a truncated answer, or a timeout still fails. The nginx checks require curl to exit successfully. The other curl interoperability checks remain strict. - nginx reuses connections to the server only with an
upstreamblock withkeepalive,proxy_http_version 1.1and an emptyConnectionheader, as in the test configuration.
The tests found one server defect (issue #142). A refusal, or an answer
sent before the body was read, closed lingering after discarding at most
64 KiB. A proxy forwards the whole body regardless, so it was often still
sending when the server closed; the connection reset under it, and it
answered 502 instead of passing on the server’s answer. The budget is now 16 MiB, still
within 2 s. Twenty requests per case, measured on 2026-10-17:
| Request | Proxy | 64 KiB budget | 16 MiB budget |
|---|---|---|---|
2 MiB sized body, 413 | nginx, buffering | 14 of 20 502 | 0 |
2 MiB chunked body, 413 | nginx, buffering | 15 of 20 502 | 0 |
2 MiB sized body, 413 | nginx, streaming | 2 of 20 502 | 0 |
1000 KiB sized body, early 200 | nginx, buffering | 1 of 20 502 | 0 |
1000 KiB chunked body, early 200 | nginx, buffering | 7 of 20 502 | 0 |
2 MiB bodies, 413, four variants | Caddy | 1 to 3 of 20 502 each | 0 |
1000 KiB chunked body, early 200 | Caddy | 1 of 20 502 | 0 |
With 200 KiB bodies neither proxy saw a 502 under either budget.
Deployment
For a public service, put a reverse proxy in front of the server (see
deploying behind a reverse proxy).
The proxy also absorbs slow clients beyond what idle_connections bounds:
- Buffer whole request heads, and preferably bodies, before forwarding
(nginx does both by default; keep
proxy_request_buffering on).tests/fixtures/proxy/has tested nginx and Caddy configurations. - Connect to the server over loopback or a private network with
proxy_http_version 1.1and keep-alive, so connections are reused. - Set the proxy’s client timeouts and connection limits per client; the server’s limits are per connection.
The server itself:
- Listen on loopback or a private address. Serve
HTTPS directly only when no proxy
is possible, and then keep
head_timeoutandhandshake_timeoutshort andidle_connectionsabove the number of slow clients expected. - Raise the descriptor limit (
LimitNOFILEin systemd) aboveidle_connections + queue + workersplus the program’s own files; when it runs out,acceptfails, is counted inaccept_errorsand retried every 10 ms. - Set
eventsto astd.logchannel sink and export theReportcounts;rejected,timeoutsandaccept_errorsshow overload. - Drain on
SIGTERMwithstd.signal, and set the supervisor’s stop timeout (TimeoutStopSecin systemd) abovedrain_timeout.
Security fixes and backend updates
The HTTP modules are checked Plenty in the compiler’s embedded standard library; a fix reaches a program when it is rebuilt with a fixed compiler. For a defect in parsing, framing or limits:
- Record it in the issue register with severity
high, and add the triggering request totests/fixtures/http_server/corpus.txtwith its correct answer, so it stays fixed. - Fix it in
stdlib/std/http.plentyorstdlib/std/http/, and run this suite withtest_std_http.rs,test_std_http_server.rs,test_std_http_client.rsandtest_std_http_helpers.rs. - Rebuild and redeploy every program that serves HTTP.
TLS uses vendored rustls and ring, pinned in Cargo.lock. Follow rustls and
ring advisories and update them as described under
TLS supply, then run test_std_tls.rs,
test_std_http_https.rs and the TLS tests here. No other native backend is
linked into a program that only serves HTTP.
Reproducing
cargo test --locked --all-features --test integration test_std_http_server_qualification::
PLENTY_DOCKER_TESTS=1 cargo test --locked --test integration test_std_http_server_proxy::
cargo build --release --locked
python3 examples/http_server_bench.py --compiler target/release/plenty --output target/http-bench --runs 3
python3 examples/http_server_bench.py --compiler target/release/plenty --output target/https-bench --runs 3 --tls
python3 fuzz/http_server/run.py --time 1800 --jobs 2
The curl tests are skipped when curl is not installed. The proxy tests
need Docker and curl and are skipped, with a message, unless
PLENTY_DOCKER_TESTS=1 (see reverse-proxy tests);
the proxy_tests job in .github/workflows/ci.yml runs them on every push
and pull request. The benchmark needs ApacheBench (ab) and taskset, and
with --tls also curl. Fuzzing needs clang’s libFuzzer; without it the
libFuzzer tests in test_fuzz.rs print a note and pass, unless
PLENTY_FUZZ_TESTS=1. Busy machines distort the benchmark: check
the load average first, and keep the total number of busy threads at or
below the number of physical cores.
std.http.sse
import std.http.sse sends
Server-Sent Events,
the text/event-stream format browsers read with EventSource, from a
std.http.server handler. The handler turns its exchange
into an event stream that outlives the server’s response deadline, then
sends events until it closes the stream, the client goes away, or the
server shuts down. The module is checked Plenty in
stdlib/std/http/sse.plenty and adds no runtime primitives. The tutorial
lesson Stream events to browsers
shows a complete program. There is no client-side reader. Line-scan and write-progress offsets stay
within their respective buffers, including the final line break; no wrapping
or saturating lifetime byte counter is used.
import std.time
import std.bytes
import std.http.client
import std.http.server
import std.http.sse
import std.net
from std.http.server import Exchange
from std.http.sse import SseError
from std.net import TcpStream
from std.sync import CancellationToken
pub enum Route:
Events
pub class Feed:
pub def handle(self, route: &Route, exchange: &mut Exchange[TcpStream]) -> Result[(), SseError | RangeError]:
wait_cancel = exchange.cancel_token()
deadline = std.time.deadline_after(&std.time.monotonic_now(), &std.time.seconds(1u32))?
mut events = std.http.sse.accept(exchange)?
events.send("first line\nsecond line", &deadline, &wait_cancel, event="note", id=Some("1"))?
events.send("done", &deadline, &wait_cancel)?
events.close()
Ok(())
type WatchError = AllocError | IoError | std.bytes.BytesError | std.http.client.ClientError | RangeError
def watch(port: u16, stop: CancellationToken) -> Result[(), WatchError]:
with stop.cancel_on_exit():
never = CancellationToken()?
deadline = std.time.deadline_after(&std.time.monotonic_now(), &std.time.seconds(5u32))?
client = std.http.client.client(std.http.client.config()?)?
reply = client.get(&f"http://127.0.0.1:{port}/events"?, &deadline, &never)?
print(reply.body.to_text()?.strip()?)?
Ok(())
def main() -> Result[(), SpawnError[()] | std.http.server.ServerError | std.net.NetError | WatchError]:
mut routes = std.http.server.router[Route]()?
routes.add("GET", "/events", Route.Events)?
listener = std.net.listen(&std.net.loopback_v4(0u16), 16)?
port = listener.local_address()?.port
stop = CancellationToken()?
with spawn(watch, port, stop.share()).drop_job()? as task:
drop(std.http.server.serve(listener, &routes, &Feed(), &std.http.server.config(), &stop)?)
task.join()?
Ok(())
event: note
id: 1
data: first line
data: second line
data: done
Opening a stream
| Item | Meaning |
|---|---|
accept(exchange: &mut Exchange[T]) | Result[EventStream[T], SseError]: answer with head() and return the stream |
accept_with(exchange, head: ResponseHead) | accept with the caller’s 200 head, such as one with CORS fields |
head() | Result[ResponseHead, HttpError]: the head accept sends |
The response is:
HTTP/1.1 200 OK
Content-Type: text/event-stream
Cache-Control: no-cache
Connection: close
accept_with adds Content-Type and Cache-Control unless the head names
them, and refuses a status other than 200
(Http(InvalidStatusLine)) and a head that declares Content-Length or
Transfer-Encoding (Http(ConflictingFraming)). A refused head leaves the
exchange unanswered, so the handler can still answer it. The request body
must be empty or read to its end (Http(Unfinished)).
The body is delimited by the connection’s close rather than chunked: the
connection leaves the server through
Exchange.detach, so the server’s
response_timeout no longer applies, and dropping or closing the stream
closes the connection. A HEAD request is answered with the head alone,
with the server’s usual framing, and accept returns Closed.
stream.last_event_id is the request’s Last-Event-ID field, which a
reconnecting EventSource sends with the last id it received, or
Nothing. A handler that keeps a history can resume after it.
Sending
| Method | Meaning |
|---|---|
send(&data, &deadline, &cancel, event="", id=Nothing) | Send one event |
send_bytes(data: &[u8], &deadline, &cancel, event="", id=Nothing) | send of UTF-8 data, without copying it first |
comment(&text, &deadline, &cancel) | Send comment lines, which clients ignore |
retry(span: &Duration, &deadline, &cancel) | Ask the client to wait this long before reconnecting |
wait(&deadline, &cancel) | Wait until deadline, watching for a disconnect and for shutdown |
close() | End the response; closing a closed stream does nothing |
is_open() | Whether events can still be sent |
These return Result[(), SseError], except close and is_open. Each
sending call writes and flushes whole lines before it returns, within its
deadline.
An event is written as the format’s readers parse it:
| Part | Lines |
|---|---|
event | event: name, unless empty; the client’s default type is message |
id | id: value when given; Some("") resets the client’s last event id |
data | data: line for each line; CRLF, CR and LF all end a line |
| end | an empty line, which dispatches the event |
send("a\nb", &deadline, &cancel, event="update", id=Some("7")) writes
event: update\nid: 7\ndata: a\ndata: b\n\n. Data ending with a line break
gets a last, empty data: line, so the client sees the trailing line feed.
An event name or id holding CR or LF, or an id holding NUL, is
InvalidField; data that is not UTF-8 is InvalidText. These checks happen before
anything is written, and the stream stays open.
comment("ping", &deadline, &cancel) writes : ping\n, one line per line of text.
A comment is the usual keep-alive: proxies close quiet connections, and a
write is how a server notices a client that has gone away.
retry takes a nonnegative Duration; positive fractions round up to whole
milliseconds on the wire. The deadline/cancellation pair precedes the optional
event and id arguments because required parameters precede defaults.
All waits accept caller cancellation. They preserve the retained server shutdown
tokens, checking the caller between native waits of at most 10 ms; this interval
is not a scheduling guarantee. Cancellation before a send leaves the stream
open and writes nothing. Cancellation during a partial write closes it.
wait returns Net(Cancelled) without closing on caller cancellation, so the
handler can choose whether to send a final event with another token.
Disconnects and shutdown
A client that closes or resets the connection makes the next write fail
with Disconnected. wait reads from the connection while it waits, so it
returns Disconnected as soon as the client closes, without a write.
Anything the client sends is discarded. A write that times out, or fails
part-way through an event, also closes the stream, since the client is left
inside an event. Every connection error closes the stream; later calls fail
with Closed. Stopping, the field checks above and allocation failures
leave it open.
The stream keeps the server’s shutdown tokens. wait ends with Stopping
when shutdown begins, and the stream stays open so the handler can send a
last event; a later wait returns Stopping at once. When the drain
deadline passes, writes and wait end with Net(Cancelled).
A handler sends events between waits:
while True:
events.send(&next_update()?, &soon()?, &wait_cancel)?
match events.wait(&in_one_second()?, &wait_cancel):
case Ok(_):
pass
case Err(SseError.Stopping):
events.send("bye", &soon()?, &wait_cancel, event="shutdown")?
return Ok(())
case Err(error):
return Err(error)
Errors
SseError has these cases:
| Case | Meaning |
|---|---|
Net(NetError) | The connection failed, timed out, or was cancelled |
Allocation(AllocError) | A buffer could not grow |
Http(HttpError) | The head was refused or could not be written |
InvalidField | An event name or id with a line break, or an id with NUL |
InvalidText | Data that is not UTF-8 |
Disconnected | The client closed the connection |
Closed | The stream was closed, or failed earlier |
Stopping | Server shutdown began while waiting |
Limits
- An open stream holds its server worker until the handler returns, as a
WebSocket does, so each one counts against
workers. Route event streams to handlers that return when the stream ends. - One caller at a time: the methods borrow the stream exclusively, so one thread cannot wait while another sends. A handler that needs events from other threads receives them through a channel between waits.
- No event history: resuming after
last_event_idis the handler’s job. - No client-side reader.
- A stream holds a 256-byte output buffer, grown to the largest event sent, and a 512-byte buffer for discarded input.
std.http.typed
HTTP head parsing, writing and body framing over
std.io.typed.Reader[E] and Writer[E] preserve the
transport cause in FramingError[E], with no stored-error retrieval API.
Use request_body or response_body with a parsed HTTP head to validate
framing, select the decoder, enforce body limits and recover a completed
persistent connection. The lower-level adapters also remain available for
callers that have already validated Content-Length and method/status semantics.
import std.bytes
import std.http
import std.io
import std.io.typed
from std.http.typed import FramingError, read_request, request_body
from std.io.typed import CopyFailure, Partial, ReadFailure, copy_all
type HttpCause = FramingError[ReadFailure[std.io.StreamError]]
type ReadBodyError = Partial[HttpCause] | HttpCause | CopyFailure[HttpCause, std.io.StreamError]
def main() -> Result[(), AllocError | IoError | std.bytes.BytesError | std.io.StreamError | std.http.HttpError | ReadBodyError]:
wire = std.bytes.from_text("POST /notes HTTP/1.1\r\nHost: example.com\r\nTransfer-Encoding: chunked\r\n\r\n5\r\nhello\r\n0\r\n\r\n")?
limits = std.http.limits()
mut conn = std.io.typed.buffered_reader(std.io.typed.reader(std.io.cursor(wire)), 256)?
head = read_request(&mut conn, &limits)?.ok_or(std.http.HttpError.Truncated)?
mut body = request_body(conn, &head, &limits)?
mut output = std.io.typed.writer(std.io.cursor(std.bytes.empty()?))
mut scratch = std.bytes.with_capacity(16)?
scratch.resize(16, 0u8)?
copy_all(&mut body, &mut output, &mut scratch)?
print(head.method, head.target, std.io.typed.unwrap_writer(output).data.to_text()?)?
print(body.total(), body.is_done())?
# The connection comes back only after a complete, reusable message.
drop(body.finish()?)
Ok(())
POST /notes hello
5 True
Reading and writing heads
read_request(reader, limits) and read_response(reader, limits) return
Result[Option[RequestHead or ResponseHead], Partial[FramingError[E]]].
They share the legacy parser’s grammar, UTF-8 validation and Limits policy.
Nothing means EOF before a message; request parsing permits leading blank
lines within the head budget. Response parsing returns each interim response
separately. Framing fields and request routing remain separate validation steps.
On failure, completed counts bytes consumed for this head, including CRLF and
leading request blank lines. It excludes prefetched bytes still in a buffered
reader. Transport errors remain Transport(E); malformed syntax and limits
are Protocol(HttpError), and an incomplete head is Truncated.
Explicit interruption retries without losing the partially read line. Other
errors terminate this parse; the count is diagnostic information, not a token
for resuming a partially parsed head.
These functions request one byte at a time and never consume body bytes. For
sockets, use std.io.typed.buffered_reader so those requests are served from a
buffer; its ReadFailure[E] envelope preserves the original transport payload.
Pass the same buffered reader to the body constructor to retain read-ahead.
write_request(writer, head), write_response(writer, head),
write_chunk(writer, source: &[u8]) and
write_last_chunk(writer, trailers) return
Result[(), Partial[FramingError[E]]]. Head/trailer serialization shares
std.http’s validation and completes before any write. Chunk
payload is written directly from the source without copying it. completed
counts wire bytes delivered by that operation, including chunk metadata.
Transport errors and invalid progress remain distinct; a zero write is
Protocol(HttpError.Stream(StreamError.WriteZero)). Partial output must not
be blindly replayed. Empty chunks write nothing; write_last_chunk ends a
chunked body.
Wire-progress extents are checked before delivery. In particular, a chunk’s
size line, payload and final CRLF must all fit i64; representation failure
returns Partial(0, FramingError.OutOfRange) before writing any of that chunk.
It is not invalid transport progress. Existing transport failures retain their
payload and exact previously delivered count.
Head, body and trailer cursor arithmetic uses explicit opt-outs justified by validated ranges, byte budgets and accepted progress. A caller’s arithmetic mode does not change these public result contracts.
From a parsed head
request_body(inner, head, limits) and
response_body(inner, head, method, request_keep_alive, limits) return
Result[Body[E, R], FramingError[E]], where R satisfies Reader[E].
The transport must be positioned immediately after the head. Pass the whole
buffered reader, including any prefetched body or next-message bytes; taking
its inner stream would lose those bytes.
These constructors call request_framing or response_framing,
respectively. Duplicate/conflicting framing, unsupported transfer coding and
invalid lengths remain Protocol(HttpError) failures, including on HEAD and
bodyless responses. HEAD, 1xx, 204, 304 and successful CONNECT semantics select
an empty body. Requests without declared framing are empty; responses without
declared framing are close-delimited. The constructors accept parsed heads:
they do not parse bytes, revalidate arbitrary hand-built head syntax, validate
request routing/Host, or consume interim responses on the caller’s behalf.
Every mode checks Limits at construction. A declared payload length above
max_body fails before any read. Chunked and close-delimited payloads are
bounded while reading. At exactly the close-delimited limit, the next nonempty
read probes one byte to distinguish EOF from an oversized body; that probe
is not delivered or counted, and an oversized body becomes broken. Empty reads
never probe. HEAD/bodyless responses do not spend a body budget on their
declared representation length.
Body.read follows the same typed reader contract as the low-level adapters.
total() counts payload bytes returned successfully; is_done() is true only
after successful completion. trailers() borrows the chunked trailer list
(empty for other modes); it is complete only once is_done() is true.
Interruption preserves state; failures retain concrete transport causes and
previously delivered progress through the typed stream helpers. Invalid ranges
trap before changing body state or accessing the transport. Other source/framing failures break it.
body.finish() returns Result[R, FramingError[E]]. It returns the complete
transport, with buffered bytes preserved, only after a persistent fixed-length,
chunked or empty body completes. An unfinished body reports
Protocol(HttpError.Unfinished) and remains readable. A failed or previously
finished body reports Broken. Successful extraction happens exactly once;
subsequent reads report Broken and is_done() becomes false.
NotReusable means completion did not permit another HTTP message: the
connection is close-delimited, has a closing Connection policy, is HTTP/1.0
without keep-alive, is a 101 upgrade or is a successful CONNECT tunnel.
For responses, pass the originating request’s keep-alive policy as
request_keep_alive; both sides must allow persistence. This is message-boundary
recovery, not pooling: after an interim response, the next message is still
the final response; after reading a request, the server must also honor its
response’s closing policy. Upgraded/tunnel streams are deliberately not exposed
by this API. Rejected finish calls retain ownership; dropping the body closes
or drops its transport, including on incomplete/error paths. Constructors take
ownership too, so a construction failure drops the supplied transport.
Low-level framing
| Constructor | Framing |
|---|---|
fixed_body(inner, length: NonNegative[i64]) | Result[FixedBody[E, R], FramingError[E]] |
close_body(inner) | CloseBody[E, R]; reads until transport EOF |
chunked_body(inner, limits) | Result[ChunkedBody[E, R], FramingError[E]]; bounded chunked decoding with std.http.Limits |
All readers expose read(buffer, start, stop) returning
Result[i64, StepError[FramingError[E]]]. Invalid caller buffer ranges trap
before checking body state or accessing the transport, as required by the
Reader contract. A fixed body caps each read to
its remaining length and reports Truncated on premature EOF. It never
consumes bytes after the body. A close-delimited body treats EOF as successful
completion and remembers it, avoiding further transport reads. A
close-delimited connection must never be reused.
FramingError distinguishes Transport(E), Truncated,
InvalidProgress, OutOfRange, Broken, NotReusable, and Protocol(HttpError). The latter
retains chunk syntax, limit, trailer and framing-allocation errors using the
existing HttpError variants; transport errors always use
Transport(E). Interruption is forwarded as
a retry signal without poisoning the body. Transport errors and invalid
progress make it broken; later reads report Broken. Truncation also breaks
a fixed or chunked body. Chunk syntax, limit and trailer failures also break
the chunked body. Invalid buffer ranges do not consume input or break a healthy
body. Valid empty reads return zero without reading or marking EOF.
Chunked bodies
chunked_body requires a validated head selecting chunked transfer coding.
It checks Limits at construction (max_line and max_head at least two;
the types of max_headers and max_body already exclude negative values). Chunk-size lines and trailer fields
include CRLF in their line limit. max_head bounds the entire trailer section,
including its final CRLF; max_headers bounds trailer fields, and max_body
bounds decoded payload. A chunk exceeding the remaining body budget is rejected
before its payload is read. Hex sizes are overflow-checked.
The decoder requires CRLF and uses the same extension and field policy as
std.http: extensions after ; are ignored after rejecting control bytes
other than tab; field names must be tokens, surrounding spaces/tabs are removed
from values, and values must be valid UTF-8 without other control bytes or DEL.
Obsolete folded fields are rejected. Trailers are metadata, not a second head:
they do not change framing or replace validated head fields. Parsed fields are
available in body.trailers; treat them as complete only after body.is_done().
body.total() counts decoded payload bytes returned to the caller.
Parsing resumes after Interrupted even within a size line, a CRLF, or a trailer.
A successful payload read returns immediately, before parsing subsequent
framing. Consequently read_exact and copy_all retain all previously returned
payload progress when later metadata or transport fails. Metadata bytes do not
count as delivered bytes. As with other readers, an unsuccessful read may have
modified scratch storage; only an Ok(count) identifies payload to use.
The terminal blank trailer line completes the body. Metadata reads request one
byte at a time, and payload reads are capped to the current chunk; the reader
does not request any bytes from the next message. Repeated completed reads
return zero without consulting the transport. Empty reads do not advance parsing
or imply completion. Construction reserves one line buffer of max_line bytes;
trailer strings and their list allocate fallibly while reading. This simple
decoder prioritizes bounded, resumable behavior; it has no internal read-ahead
buffer or throughput qualification yet.
The adapters own their transports. Connection pooling, TLS setup and
cancellation polling belong to std.http.client, which uses
these typed head and body paths, including CONNECT and streamed uploads.
Cancellation delivered as a typed transport cause survives intact. The
client’s legacy Response.read/take_failure bridge remains for compatibility;
its internal transport and framing paths do not retrieve stored errors.
The low-level adapters do not expose
their transports; use the parsed-head constructors and Body.finish for
guarded connection recovery. Bridging a legacy head reader preserves only the
error type that the legacy reader itself returns.
std.http.websocket
import std.http.websocket speaks the WebSocket protocol of
RFC 6455: the opening handshake,
as a std.http.server exchange that switches protocols
or as a client, then framed text and binary messages, ping and pong, and
the closing handshake. std.http.wss adds wss clients through
std.tls. Both modules are checked Plenty in stdlib/std/http/
and add no runtime primitives; the handshake’s SHA-1 comes from
std.crypto. The tutorial lesson
Talk over WebSockets shows a
complete program.
import std.time
import std.bytes
import std.http.server
import std.http.websocket
import std.net
from std.http.server import Exchange
from std.http.websocket import Message, WebSocketError
from std.net import TcpStream
from std.sync import CancellationToken
from std.time import Deadline
type ChatError = WebSocketError | std.bytes.BytesError | RangeError | AllocError
def soon() -> Result[Deadline, RangeError]:
Ok(std.time.deadline_after(&std.time.monotonic_now(), &std.time.seconds(5u32))?)
pub enum Route:
Chat
pub class Shout:
pub def handle(self, route: &Route, exchange: &mut Exchange[TcpStream]) -> Result[(), ChatError]:
wait_cancel = exchange.cancel_token()
mut socket = std.http.websocket.accept(exchange, &std.http.websocket.config())?
mut message = std.bytes.empty()?
while True:
message.clear()
match socket.receive(&mut message, &soon()?, &wait_cancel)?:
case Message.Close(_):
return Ok(())
case _:
socket.send_text(&message.to_text()?.upper()?, &soon()?, &wait_cancel)?
Ok(())
def chat(port: u16, stop: CancellationToken) -> Result[(), ChatError | IoError]:
with stop.cancel_on_exit():
never = CancellationToken()?
url = f"ws://127.0.0.1:{port}/chat"?
mut socket = std.http.websocket.connect(&url, &std.http.websocket.config(), &soon()?, &never)?
socket.send_text("hello", &soon()?, &never)?
mut reply = std.bytes.empty()?
print(socket.receive(&mut reply, &soon()?, &never)?, reply.to_text()?)?
socket.close(1000, "bye", &soon()?, &never)?
Ok(())
def main() -> Result[(), SpawnError[()] | std.http.server.ServerError | std.net.NetError | ChatError | IoError]:
mut routes = std.http.server.router[Route]()?
routes.add("GET", "/chat", Route.Chat)?
listener = std.net.listen(&std.net.loopback_v4(0u16), 16)?
port = listener.local_address()?.port
stop = CancellationToken()?
with spawn(chat, port, stop.share()).drop_job()? as task:
drop(std.http.server.serve(listener, &routes, &Shout(), &std.http.server.config(), &stop)?)
task.join()?
Ok(())
std.http.websocket.Message.Text HELLO
Connections
WebSocket[T] is one open connection over a stream T that reads and
writes like a server Transport: TcpStream, or
TlsTransport for TLS.
| Item | Meaning |
|---|---|
socket.receive(&mut out, &deadline, &wait_cancel) | Result[Message, WebSocketError]: append the next whole message to out |
socket.send(kind, data: &[u8], &deadline, &wait_cancel) | Result[(), WebSocketError]: send data as one Kind.Text or Kind.Binary message |
socket.send_text(&text, &deadline, &wait_cancel) | send of text, copied into a buffer first |
socket.ping(data: &[u8], &deadline, &wait_cancel) | Send a ping with at most 125 bytes of payload |
socket.close(code, &reason, &deadline, &wait_cancel) | Run the closing handshake, then shut the stream down |
socket.is_open() | Whether messages can still be sent |
socket.subprotocol | Option[str]: the subprotocol both sides agreed on |
Message is Text, Binary, or Close(code). Text has been checked as
UTF-8. A message sent in several frames is joined before receive returns
it. Pings are answered with a pong and pongs are skipped while receiving.
When the peer closes, receive answers with the same code and returns
Close with the peer’s code, or 1005 when it gave none, and its reason
appended to out; the connection is then closed.
send splits a message longer than max_frame into several frames. A
Text message must be UTF-8 (InvalidText). Client frames are masked with
a fresh key from std.secrets for each frame, as RFC 6455
requires; server frames are not.
close accepts the codes a peer may send: 1000 to 1003, 1007 to 1014, and
3000 to 4999, with a reason of at most 123 bytes (OutOfRange otherwise).
It sends the close, skips any messages still arriving until the peer’s
close, and shuts the stream down. Closing a connection whose closing
handshake already finished returns Ok.
On an error, out keeps only what it held before the call. Every error
except Stopping and OutOfRange closes the connection; later calls fail
with Closed. A protocol fault first sends a close: 1002 for Protocol,
1009 for TooLarge and 1007 for InvalidText.
Each call has its own deadline. A connection holds one 4 KiB input buffer,
grown only to hold input that arrived with the handshake, and one 4 KiB
output buffer; a sent message is copied through the output buffer in
pieces, so send allocates nothing for any message size.
Configuration
config() returns the defaults.
| Field | Default | Meaning |
|---|---|---|
max_frame | 65536 | Largest frame payload received, and the size send splits messages into |
max_message | 1048576 | Largest message receive joins |
subprotocol | Nothing | The subprotocol a client offers, or a server accepts when the client offers it |
Both maxima are Positive[i64], so a zero or negative maximum is a type error.
They bound payload bytes, not the prefix already held by the caller’s buffer.
An unrepresentable combined buffer extent is
Allocation(AllocError.CapacityOverflow), not TooLarge; as with other receive
errors, the original prefix is retained and the connection is closed. Frame
splitting and mask positions stay within their source ranges even when the
configured maximum is i64.MAX.
Servers
accept(exchange: &mut Exchange[T], config: &Config) returns
Result[WebSocket[T], WebSocketError]. It checks that the request is a
handshake: GET over HTTP/1.1 with no body, Upgrade: websocket,
Connection: upgrade, Sec-WebSocket-Version: 13 and one
Sec-WebSocket-Key of 16 bytes in base64. Field names and the Upgrade
and Connection tokens are compared without case. A request that fails
these checks is answered and accept returns Refused(status):
| Request | Answer |
|---|---|
Missing websocket in Upgrade, or another version | 426 Upgrade Required with Upgrade: websocket and Sec-WebSocket-Version: 13 |
| Any other fault | 400 Bad Request |
Otherwise it answers 101 Switching Protocols with the accept value, and
the configured subprotocol when the client listed it, and takes the
connection with
switch_protocols. The server no
longer counts or closes the connection; dropping the WebSocket closes it.
The WebSocket keeps the server’s shutdown tokens. A receive waiting for
the next message ends with Stopping when shutdown begins, and the
connection stays open, so the handler can close with 1001. A wait in the
middle of a message, or in send or close, ends with Net(Cancelled)
when the drain deadline passes.
The handler keeps its worker for as long as the WebSocket is open, so each
open connection counts against workers. Route a WebSocket to a handler
that returns when the connection closes.
Clients
| Item | Meaning |
|---|---|
connect(url: &str, &config, &deadline, &cancel) | Result[WebSocket[TcpStream], WebSocketError]: dial and handshake a ws URL |
target(url: &str) | Result[Target, WebSocketError]: parse a ws or wss URL |
dial(&target, &deadline, &cancel) | Result[TcpStream, WebSocketError]: a TCP connection to the target |
handshake(stream, &target, &config, &deadline, &cancel) | Result[WebSocket[T], WebSocketError]: the client handshake over a connected stream |
Target.secure, host, port | wss or not; the host, lowercase without IPv6 brackets; the port, 80 or 443 by default |
target rejects URLs with another scheme (UnsupportedScheme), with
credentials or a fragment, without a host, or with port 0
(InvalidUrl). The request target is the URL’s path, / when empty, and
its query.
The handshake sends a random 16-byte key and checks the server’s answer: a
status other than 101 is Rejected(status), and a 101 without the
Upgrade and Connection tokens, with the wrong accept value, with
extensions, or with a subprotocol the client did not offer is
InvalidHandshake. The deadline bounds the handshake; later calls take
their own. cancel ends every later wait with Net(Cancelled).
connect returns UnsupportedScheme for wss URLs: use
std.http.wss.
WSS
import std.http.wss runs the client handshake over TLS.
| Item | Meaning |
|---|---|
connect(url: &str, &tls: ClientConfig, &config, &deadline, &cancel) | Result[WebSocket[TlsTransport], WebSocketError] for a wss URL |
The server’s certificate is verified against tls’s roots with the URL’s
host as the name, as for HTTPS. A TLS failure is
Connect(ClientError.Tls(...)). A ws URL is UnsupportedScheme. The
module is separate so that a program using only ws does not link TLS. On
the server side, accept takes exchanges from
std.http.tls.serve unchanged.
Every message operation accepts a caller cancellation token after its deadline.
Pre-cancellation returns Net(Cancelled) before changing the connection.
Cancellation while receiving or writing closes the connection, since a partial
frame cannot safely be resumed as a different operation. The retained server
shutdown tokens keep their existing behavior. Native waits keep their shutdown
wakeups and check the caller token between slices of at most 10 ms; scheduling
and lock contention can delay observation beyond that interval.
Errors
WebSocketError has these cases:
| Case | Meaning |
|---|---|
Net(NetError) | The stream failed, timed out, was cancelled, or ended early |
Allocation(AllocError) | A buffer could not grow |
Http(HttpError) | The handshake’s HTTP head could not be read or written |
Connect(ClientError) | The client could not connect, including TLS failures |
Entropy(StreamError) | No random bytes for a key or mask |
InvalidConfig, InvalidUrl, UnsupportedScheme | See above |
OutOfRange | A ping payload, close code or reason out of bounds |
Refused(status) | The server refused a request that was not a handshake |
Rejected(status) | The server answered the client with another status |
InvalidHandshake | A 101 answer that does not complete the handshake |
Protocol | The peer broke the framing rules |
TooLarge | A frame or message over the configured limits |
InvalidText | Text that is not UTF-8 |
Closed | The connection has closed |
Stopping | Server shutdown began while waiting for a message |
Limits
- No extensions:
permessage-deflateis neither offered nor accepted. - No proxy for clients;
connectdials the URL’s host directly. - One caller at a time:
receiveand the sending methods all borrow the connection exclusively, so one thread cannot wait for a message while another sends. - A server WebSocket holds a worker until its handler returns.
receiveplaces a control frame’s payload inoutwhile handling it, sooutbriefly grows by up to 125 bytes.- A close answer echoes the peer’s code without its reason.
send_textcopies its text;sendsends bytes the caller already holds.
std.image
import std.image holds images as owned buffers of 8-bit pixels, decodes and
encodes PNG files, and crops, converts and resizes images. It has one error
type, ImageError.
The module is ordinary Plenty over std.bytes and
std.compress, which supplies zlib and CRC-32; it adds no
runtime code. Decoding checks the image’s dimensions against caller-supplied
limits before it allocates. It is available on the supported x86-64 Linux GNU
target.
import std.image
from std.image import ColorType, Filter, ImageError
def main() -> Result[(), IoError | ImageError]:
# A 4 x 2 RGB image: a red top row over a black bottom row.
mut image = std.image.new(4, 2, ColorType.Rgb)?
for x in range(4):
image.set(x, 0, 0, 255u8)?
png = std.image.encode_png(&image)?
decoded = std.image.decode(&png, &std.image.limits())?
print(decoded.width(), decoded.height(), decoded.pixels() == image.pixels())?
small = decoded.resize(2, 1, Filter.Bilinear)?
print(small.get(0, 0, 0)?, small.get(0, 0, 1)?)?
gray = decoded.convert(ColorType.Gray)?
print(gray.get(0, 0, 0)?, gray.get(0, 1, 0)?)?
Ok(())
4 2 True
128 0
54 0
Functions
| Function | Result | Contract |
|---|---|---|
new(width: Positive[i64], height: Positive[i64], color: ColorType) | Result[Image, ImageError] | A zeroed image: black, and transparent if it has alpha |
from_pixels(width: Positive[i64], height: Positive[i64], color: ColorType, pixels: Bytes) | Result[Image, ImageError] | Takes pixels as the buffer, without copying |
detect(data: &[u8]) | Option[Format] | The format named by the first bytes of data |
info(data: &[u8]) | Result[Info, ImageError] | The header of a PNG file, without decoding pixels |
decode(data: &[u8], limits: &Limits) | Result[Image, ImageError] | The image in a whole PNG file, within limits |
encode_png(image: &Image, level: i64 = 6) | Result[Bytes, ImageError] | A PNG file of image, compressed at zlib level |
limits() | Limits | The default limits; see Decoding limits |
Width and height lie in 1 to 2³¹−1, the PNG maximum, so every image can be
encoded. new and from_pixels trap on a dimension above 2³¹−1; convert a
computed size with Positive.try_from, or check it first with
Limits.check. An image whose size in bytes does not fit in i64 is
LimitExceeded, including a colour conversion or mixed-dimension bilinear
intermediate whose pixel extent is unrepresentable. from_pixels needs a buffer of exactly
width * height * channels bytes, otherwise OutOfRange.
detect recognises PNG, JPEG, GIF and WebP signatures and returns Nothing
for anything else. Only PNG is decoded: info and decode return
Unsupported for any input that detect does not name as PNG, including an
empty or partial signature.
Format is Png, Jpeg, Gif or WebP. Filter is Nearest or
Bilinear; see Resizing.
Info is a @copy class with public fields:
| Field | Meaning |
|---|---|
format: Format | Always Png |
width: i64, height: i64 | The dimensions in pixels |
color: ColorType | The colour type decode produces |
depth: i64 | The stored bits per sample: 1, 2, 4, 8 or 16 |
interlaced: bool | Whether the file is Adam7-interlaced |
Images
ColorType is Gray, GrayAlpha, Rgb or Rgba. Its channels() method
returns 1 to 4 and has_alpha() whether the last channel is alpha. Channels
are interleaved in the order of the name. An Image stores rows top to
bottom and pixels left to right, width * channels bytes per row with no
padding, so the byte of channel c at (x, y) is at
(y * width + x) * channels + c. Its fields are private, so the buffer length
always matches the dimensions.
| Method | Result | Contract |
|---|---|---|
width(), height() | i64 | The dimensions in pixels |
color() | ColorType | The colour type |
pixels() | &Bytes | The buffer, borrowed |
into_pixels(self: Image) | Bytes | Gives up the image for its buffer, without copying |
get(x, y, channel) | Result[u8, ImageError] | One sample |
set(self: &mut Image, x, y, channel, value: u8) | Result[(), ImageError] | Replaces one sample |
copy() | Result[Image, ImageError] | An independent image with the same pixels |
crop(x, y, width, height) | Result[Image, ImageError] | A copy of the rectangle whose top-left pixel is (x, y) |
convert(color: ColorType) | Result[Image, ImageError] | A copy in another colour type; see Colour conversion |
resize(width, height, filter: Filter) | Result[Image, ImageError] | A resampled copy; see Resizing |
A coordinate or channel outside the image, or a rectangle that is empty or
extends past an edge, is OutOfRange. The builtin copy(image) is rejected
because the buffer is a resource; use the method.
Decoding limits
decode reads the header and checks it against a Limits before it
allocates anything, so a small file that declares a large image fails at once
with LimitExceeded:
| Field | Bound |
|---|---|
max_width: NonNegative[i64] | Width in pixels |
max_height: NonNegative[i64] | Height in pixels |
max_bytes: NonNegative[i64] | The decoded buffer, width * height * channels bytes |
std.image.limits() returns 16384 × 16384 pixels and 256 MiB, enough for an
8192 × 8192 Rgba image. Construct Limits(max_width, max_height, max_bytes)
for other bounds. max_bytes counts the channels of the decoded colour type:
a palette image with transparency decodes to Rgba, four bytes per pixel.
check(width, height, color) applies the same
test to any dimensions, for example before new or resize with sizes taken
from input.
The limits bound what decode allocates: besides the result, it holds two
scanlines of at most 8 × width + 1 bytes each, a palette of at most 1 KiB
and an Inflater of about 62 KiB. The result is
reserved at full size once the header passes. A non-interlaced image is
decoded into it row by row, so a file that declares a large image but holds
little data writes little memory; an interlaced image is zero-filled first,
because its passes write scattered pixels.
decode takes the whole file in memory, so bound the input when reading it
as well. Read one byte more than the bound to tell a file that fits from one
that does not:
import std.bytes
import std.image
import std.io
from std.bytes import Bytes
from std.image import ColorType, ImageError, Limits
type Problem = AllocError | IoError | std.bytes.BytesError | std.io.StreamError | std.io.Partial | ImageError | OverflowError[i64]
# The whole file, or LimitExceeded if it is longer than `limit` bytes.
def load(path: &str, limit: NonNegative[i64]) -> Result[Bytes, Problem]:
mut file = std.io.open_file(path, "r")?
mut data = std.bytes.empty()?
read = std.io.read_up_to(&mut file, &mut data, limit + 1)?
file.close()?
if read > limit.get():
return Err(ImageError.LimitExceeded)
Ok(data)
def main() -> Result[(), Problem]:
banner = std.image.new(3000, 100, ColorType.Rgb)?
png = std.image.encode_png(&banner)?
mut out = std.io.open_file("banner.png", "w")?
std.io.write_all(&mut out, &png)?
out.close()?
data = load("banner.png", 1048576)?
header = std.image.info(&data)?
print(header.width, header.height, header.depth)?
previews = Limits(1024, 1024, 4194304)
match std.image.decode(&data, &previews):
case Ok(image):
print(image.width())?
case Err(ImageError.LimitExceeded):
print("too large for a preview")?
case Err(error):
return Err(error)
Ok(())
3000 100 8
too large for a preview
Decoding
decode accepts every PNG colour type, bit depth and filter, interlaced or
not, and produces 8 bits per channel:
| PNG colour type | Decodes to | With tRNS |
|---|---|---|
| Grayscale, 1 to 16 bits | Gray | GrayAlpha |
| Truecolour, 8 or 16 bits | Rgb | Rgba |
| Indexed, 1 to 8 bits | Rgb | Rgba |
| Grayscale with alpha, 8 or 16 bits | GrayAlpha | Not allowed |
| Truecolour with alpha, 8 or 16 bits | Rgba | Not allowed |
Samples of 1, 2 and 4 bits scale exactly to 0–255. A 16-bit sample v
becomes (v + 128) // 257, the nearest 8-bit level. A tRNS colour key is
compared with the stored samples at full depth, so a 16-bit key matches only
its exact colour. Palette entries without a tRNS alpha are opaque.
The file must follow the PNG structure exactly; anything else is an error:
- Signature and header.
IHDRmust come first, with a valid size, colour type and depth pair, compression and filter method 0, and interlace method 0 or 1. - Chunks. Every chunk’s CRC is checked: a mismatch is
Checksum. A chunk length over 2³¹−1 or a type that is not four ASCII letters isCorrupt. An unknown critical chunk, one whose type starts with an uppercase letter, isUnsupported; unknown ancillary chunks are skipped after their CRC check. - Order.
PLTEandtRNScome before the image data, once each,PLTEfirst, and only for colour types that allow them; an indexed image needsPLTE. TheIDATchunks are consecutive, and their contents form one zlib stream.IENDfollows, with no data. Bytes afterIENDare ignored. - Image data. The zlib Adler-32 is checked: a mismatch is
Checksum. A stream that ends early, holds more data than the header describes, or contains an invalid filter type or palette index isCorrupt. - Truncation. Input that ends inside a chunk or before
IENDisTruncated.
Ancillary chunks are ignored, including gAMA, cHRM, sRGB, iCCP,
bKGD, pHYs, text and time chunks, and the animation chunks of APNG, so an
animated PNG decodes to its default image.
info reads the signature, IHDR and the chunks before the image data with
the same checks, and allocates nothing.
Encoding
encode_png writes an 8-bit, non-interlaced PNG of the image’s colour type:
the signature, IHDR, the image data in IDAT chunks of about 64 KiB, and
IEND, with no ancillary chunks. level is the zlib level, 0 to 9;
anything else is OutOfRange. PNG filter differences and reconstruction use
explicit modulo-256 byte arithmetic; cursor and filter-score arithmetic stays
bounded by the validated dimensions and held buffers. Level 0 stores rows unfiltered and
uncompressed. Above 0, each row is filtered all five ways and the one with
the smallest sum of absolute differences is kept, the heuristic the PNG
specification recommends. Equal images and levels give equal files.
Colour conversion
convert copies the image into another colour type:
- Colour to gray uses ITU-R BT.709 luma of the stored values,
(2126 × red + 7152 × green + 722 × blue + 5000) // 10000. - Gray to colour copies the gray level into all three colour channels.
- Dropping alpha discards it, without blending against a background.
- Adding alpha makes every pixel opaque.
Converting to the image’s own colour type returns a copy. Values are used as stored, without gamma decoding.
Resizing
resize takes target dimensions under the rules of new. It does not apply
Limits; check sizes taken from input with Limits.check first. Resizing to
the same dimensions returns a copy.
Nearestcopies, for each target pixel, the source pixel under its centre.Bilinearweights source pixels with a triangle filter, one axis at a time. When shrinking, the filter widens by the reduction factor so every source pixel contributes, which avoids the aliasing of point sampling. Weights are fixed-point with 22 fractional bits, following Pillow’sImage.resize.
Images with alpha are resampled with premultiplied colour, so transparent pixels do not tint their neighbours; a fully transparent result has colour 0. Bilinear results equal Pillow’s for images without alpha and are within 1 with alpha, where the two round premultiplied values differently.
Errors
pub enum ImageError:
OutOfRange
Corrupt
Checksum
Truncated
Unsupported
LimitExceeded
Allocation(AllocError)
| Variant | Meaning |
|---|---|
OutOfRange | A dimension, coordinate, channel, rectangle, buffer length, or level out of bounds |
Corrupt | A malformed PNG file; see Decoding |
Checksum | A chunk CRC or zlib Adler-32 mismatch |
Truncated | The input ended inside a chunk or before IEND |
Unsupported | A format other than PNG, or an unknown critical chunk |
LimitExceeded | An image outside the Limits, or an unrepresentable pixel or decompressed-pass extent |
Allocation(cause) | A buffer could not be allocated |
Ownership and threads
An Image owns its buffer; dropping it releases the buffer. from_pixels
and into_pixels move a buffer in and out without copying, and pixels
borrows it. Every other operation that returns an image returns a new one
and leaves its source unchanged. The module has no global state, passes the
ordinary worker-effect checks, and may be used from any thread.
Costs
| Operation | Allocates |
|---|---|
new, copy, crop, convert, resize with Nearest | The result |
resize with Bilinear | The result; for each resampled axis, 16 bytes plus 8 bytes per tap for each target position, with 4 taps when enlarging and about 2 × the reduction factor + 2 when shrinking; and an intermediate image of the target width and source height when the height changes and either the width changes or the image has alpha |
decode | The result, two scanlines, the palette, and an Inflater; see Decoding limits |
encode_png | The file, which grows by doubling; a zlib Compressor (about 387 KiB); five filtered copies of a row and a zero row, or one row at level 0; and about 64 KiB of compressed data before each IDAT is framed |
detect, info, from_pixels, into_pixels, get, set, Limits.check | Nothing |
Decoding is linear in the file size and pixel count. Encoding is linear in the pixel count, with zlib slower at higher levels. A bilinear resize costs one multiply per tap per target sample on each axis. Pixel loops are scalar Plenty without SIMD. On a 3000 × 2000 RGB image, decoding took about twice as long as Pillow, encoding at level 6 about 2.7 times, and bilinear shrinking to a third about 6 times.
The module is compiled only into programs that import it.
Limits
- PNG only: JPEG, GIF and WebP are detected but
Unsupported. - 8 bits per channel. 16-bit samples are reduced on decode, and there is no 16-bit, palette, low-depth or interlaced output.
- No colour management: gamma, chromaticity, sRGB and ICC chunks are ignored, and conversion and resizing work on stored values, not linear light.
- No metadata: text, time, physical size and background chunks are neither returned nor written. No APNG animation frames.
- Whole files only: no decoding from a
std.ioreader or row by row. - Only nearest and bilinear filters; no box, bicubic or Lanczos.
- No rotation, flipping, compositing or drawing.
- One target, x86-64 Linux GNU.
Acknowledgements
- The W3C PNG Third Edition supplies the format, the filter-selection heuristic and the Adam7 interlace by Adam M. Costello; the decoder and encoder are implemented independently.
- The Paeth filter is from Alan W. Paeth, “Image File Compression Made Easy”, Graphics Gems II (1991).
- Bilinear weights follow Pillow’s resampling; the module does not wrap or ship its code. Tests compare decoding, encoding and bilinear resizing with Pillow when it is installed.
- Gray conversion uses the ITU-R BT.709 weights, as the Rust
imagecrate does.
std.io
import std.io provides byte streams with explicit partial progress: the
stream contracts, helpers that retry until a range is complete, fixed-capacity
buffering, and adapters for files, the process streams, and memory. Data moves
through std.bytes.Bytes ranges; no operation creates a list[u8].
Writes read a &[u8] view, so a buffer, part of one, or
a b"..." literal is written as it is, without a copy.
The module is embedded Plenty code over native transport and terminal queries. It is available on the supported x86-64 Linux GNU target.
Stream contracts
A stream is any pub class or enum whose pub methods match one of these protocols.
Generic calls specialize for each concrete type. Eligible interfaces can also
use explicit protocol boxes, whose
method calls dispatch indirectly; this does not introduce an implicit stream
object type.
| Protocol | Method | Result |
|---|---|---|
Reader | read(self: &mut, buffer: &mut Bytes, start: i64, stop: i64) | Result[i64, StreamError] |
Writer | write(self: &mut, source: &[u8]) | Result[i64, StreamError] |
Writer | flush(self: &mut) | Result[(), StreamError] |
Seeker | seek(self: &mut, target: SeekFrom) | Result[i64, StreamError] |
Closer | close(self: &mut) | Result[(), StreamError] |
One call makes at most one attempt and returns the number of bytes it moved.
readfills a prefix ofbuffer[start:stop]and never changes the buffer’s length. A count below the range length is a short read, not an error. Zero for a nonempty range means end of input.writeconsumes a prefix ofsource. A short count is normal. Zero for a nonempty source means the stream accepted nothing;write_allreports that asWriteZeroinstead of looping. To write part of a buffer, pass&buffer[a:b].- An empty range or source returns
Ok(0)and says nothing about end of input. - An error means the call moved no bytes. A stream that fails after moving some bytes returns the count and reports the failure on the next call.
Interruptedmeans nothing moved and the same call can be repeated.seekreturns the new byte offset from the start.
A read range must satisfy 0 <= start <= stop <= buffer.len(). An invalid
range traps with read: start/stop outside buffer before touching the stream,
even at EOF or for an empty range. read_exact applies the same rule with
read_exact: start/stop outside buffer. A write source is a view,
whose bounds clamp, so it has no range error. A stream that returns a negative
count or more than it was offered is reported as InvalidProgress by every
helper and buffer, which then stop using that count.
An implementation declares write with the same source: &[u8] parameter and
passes the view, or part of it, on to operations that read one, such as
Bytes.extend or another stream’s write:
import std.bytes
import std.io
from std.bytes import Bytes
from std.io import StreamError
pub class Tally:
pub data: Bytes
pub def write(self: &mut Tally, source: &[u8]) -> Result[i64, StreamError]:
match self.data.extend(source):
case Ok(_):
Ok(len(source))
case Err(cause):
Err(StreamError.Allocation(cause))
pub def flush(self: &mut Tally) -> Result[(), StreamError]:
Ok(())
def main() -> Result[(), AllocError | std.bytes.BytesError | std.io.Partial | IoError]:
mut tally = Tally(std.bytes.empty()?)
std.io.write_all(&mut tally, b"ab")?
tail = std.bytes.from_text("cdef")?
std.io.write_all(&mut tally, &tail[:2])?
print(tally.data.to_text()?)?
Ok(())
abcd
A helper names the one protocol it needs, such as [R: Reader]. Protocols
cannot be combined in one bound, so close_buffered requires Closer and
relies on the buffer’s own use of write and flush.
Errors
StreamError | Meaning |
|---|---|
Interrupted | The native call was interrupted (EINTR); retry |
UnexpectedEof | Input ended before read_exact filled its range |
WriteZero | A writer accepted no bytes of a nonempty range |
OutOfRange | A seek target or other stream range is invalid |
InvalidProgress | A stream reported an impossible byte count |
Io(IoError) | Any other native failure, including Closed, NotReadable, NotWritable, and System(code) |
Allocation(AllocError) | A buffer could not be allocated or grown |
std.io.classify(error: IoError) performs this mapping for code that wraps
another native operation.
Helpers that take several steps keep the progress made before a failure:
| Type | Contents |
|---|---|
Partial | completed: i64 bytes moved, then cause: StreamError |
CopyError.Read(count, cause) | The reader failed after count bytes reached the writer |
CopyError.Write(count, cause) | The writer failed after accepting count bytes |
CloseError.Flush(cause) | Flushing failed; the stream was closed |
CloseError.Close(cause) | Flushing succeeded; closing failed |
CloseError.FlushAndClose(flush, close) | Both failed |
Seek and binary-field extent calculations handle checked arithmetic through
these declared errors. read_up_to reports an unrepresentable buffer extent
as Partial(completed, Allocation(CapacityOverflow)) before growing the buffer.
If another copy chunk would make the delivered total exceed i64.MAX,
copy_all returns CopyError.Read(completed, OutOfRange) before delivering
that chunk. The reader may already have consumed it; copying is not
transactional or resumable. Internal cursor arithmetic uses explicit opt-outs
only after validating ranges, progress and the applicable byte budget.
These types do not convert to each other. Match the result, or use ? in a
function whose error sum lists each of them.
Helpers
| Function | Result | Contract |
|---|---|---|
read_exact(reader, buffer, start, stop) | Result[(), Partial] | Fill the whole range. End of input first is UnexpectedEof |
write_all(writer, source: &[u8]) | Result[(), Partial] | Deliver all of source. No progress is WriteZero. Does not flush |
read_up_to(reader, out, limit: NonNegative[i64]) | Result[i64, Partial] | Append at most limit bytes to out; fewer only at end of input |
copy_all(reader, writer, scratch) | Result[i64, CopyError] | Move bytes until end of input through the whole of scratch. Does not flush |
All four retry Interrupted and continue after short progress. They stop at
the first other error and return the bytes moved so far. After a failed
read_exact, buffer[start:start + completed] holds the bytes read. After a
failed read_up_to, out has grown by exactly completed bytes.
read_exact, write_all, and copy_all do not allocate. copy_all needs a
nonempty scratch, otherwise CopyError.Read(0, OutOfRange); its length is the
transfer size. read_up_to grows out by at most 8192 bytes ahead of the data
it has received and removes unused bytes before returning, so memory follows
the input actually read, not limit.
copy is not used as a name because the builtin copy already exists.
Bounded buffering
| Function | Result |
|---|---|
buffered_reader(inner: R, capacity: Positive[i64]) | Result[BufReader[R], StreamError] |
buffered_writer(inner: W, capacity: Positive[i64]) | Result[BufWriter[W], StreamError] |
Each takes ownership of inner and allocates one buffer of exactly capacity
bytes. The buffer never grows. The only failure is allocating it. A computed
capacity converts with Positive.try_from.
BufReader[R] | Result | Contract |
|---|---|---|
read(buffer, start, stop) | Result[i64, StreamError] | Serve held bytes, otherwise make one inner read |
read_until(delimiter, out, limit: Positive[i64]) | Result[i64, StreamError] | Append through the first delimiter, at most limit bytes |
buffered() | i64 | Bytes held and not yet consumed |
capacity() | i64 | The fixed buffer size |
read makes at most one inner read per call. A request at least as large as
the capacity, made while the buffer is empty, goes straight to the inner
stream. read_until returns the count appended: zero is end of input, and a
count whose last byte is not the delimiter means the limit or end of input came
first. It retries Interrupted. After an
error, bytes already appended stay in out and bytes not yet appended stay in
the buffer.
BufWriter[W] | Result | Contract |
|---|---|---|
write(source) | Result[i64, StreamError] | Hold source, delivering held bytes first when it does not fit |
flush() | Result[(), StreamError] | Deliver every held byte, then flush the inner stream |
pending() | i64 | Bytes held and not yet delivered |
capacity() | i64 | The fixed buffer size |
Held bytes never exceed the capacity and reach the inner stream in order. A
range at least as large as the capacity is passed to the inner stream directly
after the held bytes, and may be accepted short. Delivery retries Interrupted
and short writes. When delivery fails, the undelivered bytes stay held and the
range passed to write is not accepted; a later flush resumes at the first
undelivered byte.
Nothing is written when a BufWriter is dropped. Call flush or
close_buffered; otherwise held bytes are discarded.
| Function | Result | Contract |
|---|---|---|
seek_reader(reader, target) | Result[i64, StreamError] | Discard held input and seek. Current is relative to the bytes consumed |
seek_writer(writer, target) | Result[i64, StreamError] | Deliver held bytes, then seek |
close_buffered(writer) | Result[(), CloseError] | Flush, then close the inner stream even if the flush failed |
Both classes expose the wrapped stream as the public field inner, for calls
such as reader.inner.close(). Reading or writing inner directly bypasses
the buffer. The language cannot move a field out of a consumed value, so there
is no method returning the inner stream; exchange it with
replace(buffer.inner, other).
Adapters
| Constructor | Stream | Protocols |
|---|---|---|
open_file(path: &str, mode: &str) | Result[FileStream, StreamError] | Reader, Writer, Seeker, Closer |
open_path(path: &Path, mode: &str) | Result[FileStream, StreamError] | the same, preserving native path bytes |
from_file(file: File) | FileStream | the same |
stdin() | Stdin | Reader |
stdout(), stderr() | Stdout, Stderr | Writer |
cursor(data: Bytes) | Cursor | Reader, Writer, Seeker |
open_path accepts a std.path.Path without decoding its bytes.
It has the same modes, permissions, symlink behavior, and ownership as
open_file. Embedded NUL is StreamError.Io(IoError.InvalidInput) rather than
a truncated filename; an invalid mode is StreamError.Io(IoError.InvalidMode).
Owner and path allocations are fallible and occur before opening or truncating.
Use std.cli.fileinput for buffered fallible text-line
iteration across multiple native paths or stdin.
FileStream owns a File, public as file.
read and write each make one native call on the descriptor, with no
newline translation or text decoding. seek positions by byte offset and
returns the new offset; this differs from the opaque text cookies of
File.tell. SeekFrom.Start with a negative offset is OutOfRange; another
target the system rejects is Io(IoError.System(code)). flush and sync
forward to the file. close reports the native close error once and is Ok
when repeated; other operations on a closed file are Io(IoError.Closed).
The open mode is enforced: Io(IoError.NotReadable) or Io(IoError.NotWritable).
Dropping an unclosed FileStream closes the file and discards any close error.
Stdin, Stdout, and Stderr hold no state and never close the process
descriptors. Stdin.read makes one native read. Stdout and Stderr share the
process streams used by print and the text writers, so output keeps its order
across both; flush delivers buffered stdout. Stderr is unbuffered: each
Stderr.write makes one native write on descriptor 2, so a closed or read-only
stderr is Io(IoError.System(9)) (EBADF).
Each process stream has is_terminal() -> bool. It asks the OS on each call and
returns false for redirected, closed or undetectable streams. This query neither
reads nor writes bytes and does not imply ANSI escape support or a terminal width.
std.cli.progress uses it to keep redirected stderr clean.
Cursor reads from and writes to an owned buffer, public as data, at
position(). Reads stop at the end of the buffer. Writes replace existing
bytes and extend the buffer at its end; writing past the end first fills the
gap with zeros. seek accepts any nonnegative position. A write can report
Allocation; it reserves before changing existing bytes.
The same cursor also provides checked sequential binary fields,
bounded owned payload reads, and remaining/require_end validation. These
methods share its position while retaining stricter per-field failure atomicity.
Ownership and threads
Transport methods take &mut self, so a stream has one user at a time and a
buffer passed to read cannot be the stream’s own storage. A read needs
&mut Bytes; a write borrows its source shared. A buffered adapter owns its
inner stream, and the inner stream cannot be used through another name while
the adapter exists. Sharing one stream between threads requires moving it or
placing it in a lock; the module adds no synchronization of its own. Writes to
Stdout and Stderr from several threads are each atomic with respect to the
process buffer, but their order is unspecified.
Costs
Each read or write on FileStream or Stdin is one system call. Buffered
adapters make one bulk copy per call between the caller’s range and the fixed
buffer. read_until searches held bytes with Bytes.find, one native scan
per call. Seeking a BufReader discards up to capacity bytes already read.
| Allocates | Does not allocate |
|---|---|
buffered_reader, buffered_writer: one buffer | Every BufReader and BufWriter method except read_until |
read_up_to and read_until, when out lacks capacity | read_exact, write_all, copy_all, seek_reader, seek_writer, close_buffered |
Cursor.write beyond the buffer’s capacity | FileStream, Stdin, Stdout, Stderr operations |
Limits
- Only x86-64 Linux GNU is supported. Nonblocking descriptors are not handled:
EAGAINarrives asIo(IoError.System(11)). - Stdout writes pass through the Rust standard library’s buffered stdout,
which reports
EBADFas success: writes to a closed or read-only stdout succeed and discard their bytes. - Streams are bytes only. Text decoding, line iteration over streams, and
formatted writes are not provided; use
read_untilandBytes.to_text. - There are no vectored operations, no read-ahead peeking, and no stream object that erases the concrete class.
- A class cannot declare which protocols its parameter satisfies. A wrong
inner stream is rejected at
buffered_readerorbuffered_writerwith a note naming the missing method, not with a protocol diagnostic. - The native calls
__io_file_read,__io_file_write,__io_stdin_read,__io_std_write, and__io_file_seekare private transport for the standard library, reached throughBytes.read_fileandread_stdin,std.bytes.write_file,Stdout.write/Stderr.write, andFileStream.seek.__io_is_terminalis the private process-stream terminal query.
Acknowledgements
- Stream protocols follow Go’s
iopackage (Reader,Writer,Seeker,Closer) and Rust’sstd::io::Readandstd::io::Write.
Sequential binary fields in a Cursor
std.io.Cursor owns a Bytes buffer and one byte position. Its typed
methods use std.encoding to read and write fixed-width fields
without caller-maintained offsets. Raw read, write, and seek share this
same position; raw reads still allow short reads. No second cursor, stored
borrow, native serialization library, or independent position is involved.
Scalar fields
read_u8() and read_i8() take no byte order. For each of u16, i16,
u32, i32, u64, i64, f32, and f64, read_<type>(order: Endian) selects explicit
little or big endian. The matching writes are
write_<type>(value, order, max_bytes: i64 = 1048576); one-byte writes omit
order. There is no native-order default or alignment requirement. Signed
integers use two’s complement. Floating fields use exact IEEE 754 binary32 or
binary64 bits, preserving signed zero, subnormals, infinities and NaN payloads
without arithmetic or width conversion, matching std.encoding.
All typed methods return Result[..., BinaryError]. A successful field
advances by its exact width. Failed fields leave position, contents, and length
unchanged. Earlier successful operations remain committed: this is per-field
atomicity, not record rollback. This guarantee is specific to memory; generic
stream helpers can report externally visible partial progress.
The byte order belongs to the operation, matching the underlying codecs and
making mixed-endian formats explicit without another mutable configuration.
Existing buffers are owned by cursor(data); typed operations borrow the
cursor for one call. Mutating its public data changes what subsequent reads
see without changing the position.
Payloads, bounds, and errors
remaining() -> i64returns bytes after the current position, or zero after seeking past the end.require_end() -> Result[(), BinaryError]rejects remaining bytes without advancing. A position at or beyond the end succeeds.read_bytes(count: i64, max_bytes: i64 = 1048576)returns an ownedBytescopy. It checks the cap and available input before allocation. Returned borrowed views are not supported by the language. An empty read returns an empty owned buffer without moving, even after a seek past the end.write_bytes(source: &[u8], max_bytes: i64 = 1048576)borrows a literal,Bytes, list, or subview and writes its complete range. An empty write does not fill a seek gap or move position.
Every typed write’s max_bytes bounds the final buffer length, including an
existing larger buffer and zero-filled seek gaps. Negative caps/counts and
position arithmetic overflow are OutOfRange; a valid size exceeding the cap
is LimitExceeded. Seek itself remains unrestricted except for its existing
nonnegative checked i64 position contract. Writing beyond the current end
fills the gap with zeros after successfully allocating the whole extent.
BinaryError variant | Meaning |
|---|---|
Truncated(position, needed, remaining) | The complete requested field is unavailable; all three values are byte counts/offsets |
TrailingData(position, remaining) | require_end found unconsumed bytes |
OutOfRange | Negative count/cap or position arithmetic overflow |
LimitExceeded | Requested owned copy or final write buffer exceeds its cap |
Allocation(AllocError) | Buffer creation/growth failed |
Scalar reads and writes within capacity allocate nothing. Growth uses Bytes
allocation, and owned payload reads allocate one buffer and copy count bytes.
Scalar operations are constant time; payload copies and zero-filled gaps take
time proportional to their size. The caps bound logical bytes, not allocator
metadata or resident memory. No synchronization or hidden worker restrictions
are added; the owning cursor can be moved to one worker as other owned buffers
can. Callers cannot borrow its data as a write source while borrowing the
cursor mutably.
See the binary packet tutorial for a runnable example.
std.io.typed
Typed byte streams preserve a caller-selected error payload without side
storage. They complement std.io; existing streams keep their APIs.
import std.bytes
import std.io
import std.io.typed as typed
from std.io.typed import Partial, ReadFailure
def main() -> Result[(), IoError | AllocError | std.bytes.BytesError | Partial[ReadFailure[std.io.StreamError]]]:
mut source = typed.reader(std.io.cursor(std.bytes.from_text("header:body")?))
mut buffer = std.bytes.with_capacity(6)?
buffer.resize(6, 0u8)?
typed.read_exact(&mut source, &mut buffer, 0, 6)?
print(buffer.to_text()?)?
match typed.read_exact(&mut source, &mut buffer, 0, 6):
case Ok(_):
print("complete")?
case Err(partial):
print(partial.completed, partial.cause)?
Ok(())
header
5 std.io.typed.ReadFailure[std.io.StreamError].UnexpectedEof
Reader[E].read(&mut self, &mut Bytes, start, stop) and
Writer[E].write(&mut self, &[u8]) return Result[i64, StepError[E]]. The
read range is half-open and must satisfy 0 <= start <= stop <= buffer.len();
an invalid range traps with read: start/stop outside buffer before consulting
the transport. read_exact traps with read_exact: start/stop outside buffer.
A writer takes the bytes as a view. A successful call reports
the bytes moved; a failed call reports no progress from that attempt.
StepError.Interrupted requests retry. Source(E) preserves every other
cause, including cancellation and allocation failure. Reader Ok(0) is EOF
for a nonempty range. Writers need only write, not flush.
| Operation | Result |
|---|---|
read_exact(reader, buffer, start, stop) | Result[(), Partial[ReadFailure[E]]] |
write_all(writer, buffer: &[u8]) | Result[(), Partial[WriteFailure[E]]] |
copy_all(reader, writer, scratch) | Result[i64, CopyFailure[E, G]] |
Helpers retry only Interrupted. Partial carries completed: i64 and
cause. ReadFailure[E] distinguishes Source(E), UnexpectedEof,
InvalidProgress, and OutOfRange; WriteFailure[E] has Source(E),
WriteZero and InvalidProgress. Negative or oversized counts are invalid
progress. Invalid read ranges and empty reads or writes do not call the
underlying stream.
CopyFailure.Read(completed, ReadFailure[E]) and
CopyFailure.Write(completed, WriteFailure[G]) preserve the failing operation
and independent source/destination error types. Counts mean bytes delivered to
the writer, including a short write before failure. A read chunk may already
have been consumed when writing fails: this API does not promise resumable
copying or retain unwritten bytes. Empty scratch is an OutOfRange read
failure. A chunk that would make the delivered total exceed i64.MAX is
Read(completed, ReadFailure.OutOfRange) before writing that chunk; it is not
an invalid count from the transport. Internal cursor updates use documented
bounds-proven opt-outs, without changing the stream’s declared error type.
Helpers allocate nothing and require caller-provided storage.
buffered_reader(inner, capacity: Positive[i64]) returns
Result[BufReader[E, R], StreamError] for a Reader[E]; construction
allocates the buffer, which is its only failure. Its read returns
StepError[ReadFailure[E]], preserving a transport cause as
Source(ReadFailure.Source(E)) and forwarding Interrupted. Invalid ranges
and empty reads do not call the transport. Invalid inner counts become
ReadFailure.InvalidProgress. Reads allocate nothing; capacity() and
buffered() report the buffer size and unread bytes.
The public inner field permits transport-specific operations, but consuming
it discards prefetched bytes. Keep the whole buffered reader when handing a
connection from head parsing to body decoding or to the next HTTP message.
reader(inner) wraps an existing std.io.Reader as
LegacyReader[R]; writer(inner) wraps a std.io.Writer as
LegacyWriter[W]. Their typed cause is StreamError. Only its
Interrupted variant becomes a retry signal; other variants survive intact.
unwrap_reader(bridge) and unwrap_writer(bridge) consume the bridge and
return the owned stream, allowing
explicit flushing, seeking or closing. Copying does not flush or close.
These bridges cannot recover causes already erased by a legacy adapter:
For HTTP client responses, prefer std.http.client.typed_response(response)
over bridging Response.read: its typed cause is the complete ClientError.
The legacy Response.read/take_failure pair remains a compatibility API.
See typed HTTP framing and the tutorial.
std.json
import std.json decodes and encodes JSON (RFC 8259) under explicit limits,
with one error type, JsonError. It has two levels:
- Events: a resumable
Decoderand a validatingEncoderthat never hold the document. Typed codecs are ordinary functions over them. - Value tree:
Json, built from and written through the same events.
The module is checked Plenty code with no native primitives of its own. It
reads and writes std.bytes buffers and std.io streams,
and is available on the supported x86-64 Linux GNU target.
import std.bytes
import std.json
from std.json import Json
def main() -> Result[(), AllocError | IoError | std.bytes.BytesError | std.json.JsonError]:
bounds = std.json.limits()
value = std.json.parse_text("{\"name\": \"Ada\", \"langs\": [\"en\", \"fr\"]}", &bounds)?
match &value:
case Json.Object(members):
for member in members:
match &member.value:
case Json.Text(text):
print(member.key, "is", text)?
case Json.Array(items):
print(member.key, "has", len(items), "items")?
case _:
print(member.key, "is another type")?
case _:
print("not an object")?
mut out = std.json.encoder(&bounds)?
out.start_object()?
out.key("id")?
out.integer(7)?
out.key("person")?
out.value(&value)?
out.end_object()?
print(out.take()?.to_text()?)?
Ok(())
name is Ada
langs has 2 items
{"id":7,"person":{"name":"Ada","langs":["en","fr"]}}
Accepted text
The decoder accepts exactly RFC 8259 and one document per input.
| Rule | Behavior |
|---|---|
| Encoding | UTF-8, validated strictly; no byte-order mark |
| Whitespace | Space, tab, \n, \r between tokens only |
| Top level | Any value, then only whitespace; anything else is TrailingData |
| Extensions | None: no comments, trailing commas, single quotes, NaN, Infinity, hex, or leading + |
| Numbers | `-?(0 |
| Strings | Bytes below 32 must be escaped; escapes are \" \\ \/ \b \f \n \r \t \uXXXX with digits of either case |
| Surrogates | A \uD800-\uDBFF escape must be followed by a \uDC00-\uDFFF escape; the pair is one scalar. Anything else is InvalidSurrogate |
| Duplicate keys | Compared after unescaping. Rejected by the tree unless Limits says Keep; never checked by the event decoder |
The encoder writes compact text: no whitespace, members in call order. In
strings it escapes ", \, and bytes below 32 (\b \f \n \r \t, otherwise
\u00xx in lowercase); every other scalar, including / and non-ASCII, is
written as UTF-8.
Limits
pub class Limits:
pub max_depth: NonNegative[i64]
pub max_input: NonNegative[i64]
pub max_output: NonNegative[i64]
pub duplicate_keys: DuplicateKeys # Reject or Keep
limits() returns Limits(64, 67108864, 67108864, DuplicateKeys.Reject). The
fields are public; adjust a copy. A literal converts on assignment; a computed
value needs NonNegative.try_from(value)?. Every constructor and whole-document
function takes &Limits; decoder and encoder trap when max_depth is above
1024, so the whole-document functions built on them do too.
| Field | Bounds | When exceeded |
|---|---|---|
max_depth | Arrays and objects open at once. 0 allows scalars only | DepthExceeded at the opening bracket |
max_input | Total bytes given to one decoder | InputTooLarge; the offset is the limit |
max_output | Total bytes one encoder produces, including bytes already taken | OutputTooLarge |
duplicate_keys | Whether the tree accepts a repeated key in one object | DuplicateKey at the repeated key |
The depth limit also bounds the encoder’s recursion over a Json tree, so a
tree built by hand cannot overflow the stack.
Input and output counters are bounded by the remaining declared budget before
addition. Taking or writing output does not reset that budget. Bounded cursor,
hex-digit, surrogate and UTF-8 state arithmetic uses explicit, locally justified
unchecked: scopes; malformed values still return the JSON faults below,
not arithmetic errors. No public arithmetic error type is added.
Errors
pub class JsonError:
pub fault: JsonFault
pub offset: i64
pub line: i64
pub column: i64
For decoding, offset is the byte offset from 0, and line and column start
at 1. Only \n ends a line. A column is one Unicode scalar, not one byte. A
syntax fault points at the offending byte; a fault found after a token was read
(DuplicateKey, NumberOutOfRange) points at the token’s first byte.
UnexpectedEnd points one past the last byte. Position arithmetic is checked:
if the next byte or ASCII run would make an offset, line, or column
unrepresentable as i64, decoding returns InputTooLarge at the current
position without consuming that byte or run. This also applies when
max_input itself is i64.MAX; the one-based line and column need their own
representation checks.
For encoding, offset is the number of bytes produced when the call failed and
line and column are 0. A failed constructor reports 0, 0, 0.
JsonFault | Meaning |
|---|---|
UnexpectedByte | A byte that cannot start or continue the document here |
UnexpectedEnd | Input ended inside a document, or was empty |
TrailingData | A non-whitespace byte after the document |
InvalidLiteral | A misspelled null, true, or false |
InvalidNumber | A byte that breaks the number grammar; for Encoder.number, text that is not a JSON number |
NumberOutOfRange | An integer outside i64 or a decimal outside finite f64, in the tree |
InvalidEscape | An unknown escape or a non-hex digit in \uXXXX |
InvalidSurrogate | An unpaired surrogate escape |
ControlCharacter | An unescaped byte below 32 in a string |
InvalidUtf8 | Malformed, overlong, surrogate, or out-of-range UTF-8 in a string |
DuplicateKey | A repeated key under DuplicateKeys.Reject |
DepthExceeded | Nesting beyond max_depth |
InputTooLarge, OutputTooLarge | A size limit was reached; InputTooLarge also covers an unrepresentable input position |
NonFiniteNumber | A NaN or infinite f64 given to the encoder |
InvalidState | A call that the decoder’s or encoder’s state does not allow |
Stream(StreamError) | A reader or writer failed; InvalidProgress if it reported an impossible count |
Allocation(AllocError) | Memory was refused |
After any error, a decoder or encoder is
poisoned: every later call reports InvalidState. Partial trees and buffers are
released before the error returns.
Events
pub enum Event:
Pending
End
Null
Bool(bool)
Number(str)
Text(str)
Key(str)
StartArray
EndArray
StartObject
EndObject
Number holds the validated source text, so a codec chooses the type and loses
nothing: i64.parse, u64.parse, f64.parse, or the text itself. Text and
Key hold the unescaped string.
Decoder
decoder(bounds: &Limits) -> Result[Decoder, JsonError].
| Method | Receiver | Result | Contract |
|---|---|---|---|
feed(source: &[u8]) | exclusive | Result[(), JsonError] | Copy a byte view into the unexamined input |
finish() | exclusive | () | Declare that no more input will arrive |
next() | exclusive | Result[Event, JsonError] | The next event |
offset() | shared | i64 | Bytes examined |
depth() | shared | i64 | Arrays and objects open |
reject(fault: JsonFault) | shared | JsonError | An error placed at the first byte of the most recent event’s token |
next returns Pending when every fed byte has been examined and the document
is incomplete; feed more or call finish. After finish it returns End once
the document and trailing whitespace are consumed, and End again on every
later call. A number at the end of the fed input stays pending until the next
byte or finish shows where it ends. Input may be split anywhere, including
inside an escape or a UTF-8 sequence.
read_event[R: Reader](source: &mut Decoder, reader: &mut R) -> Result[Event, JsonError]
drives a decoder from a stream. It reads up to 4096 bytes whenever next is
Pending, retries Interrupted, treats a zero-length read as end of input, and
never returns Pending. Do not mix it with feed on one decoder.
A typed codec reads the events it expects and uses reject for values it will
not accept:
def coordinate(decoder: &mut std.json.Decoder) -> Result[i64, JsonError]:
match decoder.next()?:
case Event.Number(text):
match i64.parse(text):
case Ok(value):
Ok(value)
case Err(_):
Err(decoder.reject(JsonFault.NumberOutOfRange))
case _:
Err(decoder.reject(JsonFault.UnexpectedByte))
Encoder
encoder(bounds: &Limits) -> Result[Encoder, JsonError]. Every writing method
takes an exclusive receiver and returns Result[(), JsonError].
| Method | Writes |
|---|---|
null(), boolean(value: bool) | A literal |
integer(value: i64), unsigned(value: u64) | Decimal digits |
decimal(value: f64) | The shortest text that parses back to the same value, as str.from; non-finite is NonFiniteNumber |
number(text: &str) | The text unchanged, after checking the number grammar |
text(value: &str) | A string |
key(name: &str) | A member name; only valid where an object expects one |
start_array(), end_array(), start_object(), end_object() | Brackets; commas and colons are inserted automatically |
event(event: &Event) | The call matching a decoder event; Pending and End write nothing |
value(value: &Json) | A whole tree as one value |
| Method | Result | Contract |
|---|---|---|
pending() | i64 | Bytes produced and not yet taken |
written() | i64 | Bytes produced in total |
is_complete() | bool | Exactly one top-level value has been written and closed |
take() | Result[Bytes, JsonError] | Move the pending bytes out |
The encoder accepts only call sequences that form one document: a key outside
an object, a value where a key belongs, a mismatched close, or a second
top-level value is InvalidState. It does not check member names for
duplicates.
write_pending[W: Writer](source: &mut Encoder, writer: &mut W) -> Result[(), JsonError]
writes every pending byte with std.io.write_all and keeps the buffer’s
capacity. It does not flush. If the writer fails, the bytes it accepted are
removed and the rest stay pending.
Value tree
pub enum Json:
Null
Bool(bool)
Int(i64)
Float(f64)
Text(str)
Array(list[Json])
Object(list[Member])
pub class Member:
pub key: str
pub value: Json
| Function | Result | Contract |
|---|---|---|
parse(source: &[u8], bounds) | Result[Json, JsonError] | Decode one document from a byte view |
parse_text(text: &str, bounds) | Result[Json, JsonError] | The same, from a str |
read_value[R: Reader](reader: &mut R, bounds) | Result[Json, JsonError] | Decode one document, reading to end of input |
encode(value: &Json, bounds) | Result[Bytes, JsonError] | Compact encoding |
encode_text(value: &Json, bounds) | Result[str, JsonError] | The same, as a str |
write_value[W: Writer](writer: &mut W, value: &Json, bounds) | Result[(), JsonError] | Encode the whole value, then write it; no flush |
find(members: &list[Member], key: &str) | Option[i64] | Index of the first member named key |
A number without a fraction or exponent becomes Int and must fit i64;
otherwise the document is rejected with NumberOutOfRange, never rounded. Any
other number becomes Float, the nearest f64, and must be finite: 1e309 is
NumberOutOfRange, while 1e-400 is 0.0. -0 is Int(0); -0.0 keeps its
sign. Float values are written with a fraction or exponent (100.0, 1e21),
so a tree keeps its Int/Float distinction through a round trip.
Objects keep members in source order, including duplicates under Keep.
find is a linear scan; build a dict when an object is searched often.
Ownership and threads
A Json owns its whole tree; moving it moves everything and dropping it
releases everything. Json, Decoder, and Encoder are single-owner values
with no shared or global state. They can be moved into a worker thread, and the
module’s functions pass the ordinary worker-effect checks. One decoder or
encoder cannot be used from two threads at once, since every operation needs an
exclusive loan.
Sources are shared loans: feed and parse copy what they need and keep no
reference to the caller’s bytes. Events own their strings.
Json is recursive, so the builtins that require a finite layout walk reject
it: ==, copy, print, and str.repr do not accept a Json. Compare and
display trees through encode_text.
Costs
| Operation | Allocates | Time |
|---|---|---|
decoder | Three empty buffers | Constant |
feed | Grows the window by the view; storage is reused once the window is drained | Linear in the view |
next | One str per Number, Text, or Key event; one byte of stack per open container; the scratch buffer grows to the longest token | Linear in the bytes examined |
read_event | A 4096-byte window, once | As next, plus the reads |
parse, read_value | The events, one list per container, one set of keys per object under Reject, and for parse one copy of the input | Linear in the input, expected |
| Encoder calls | Grow one output buffer; one byte of stack per open container; number text for integer, unsigned, decimal | Linear in the bytes written |
encode, write_value | The whole output buffer | Linear in the output |
encode_text | The output buffer, then a str copy | Linear |
find | Nothing | Linear in the member count |
The decoder and the tree builder use no recursion. Encoder.value recurses once
per nesting level, at most max_depth (1024) deep.
An event loop that drains next to Pending before each feed, or uses
read_event, holds only the current chunk, the current token, and the depth
stack, whatever the document size. A tree costs more than its text: each value
is an enum cell and each string and container its own allocation.
The decoder takes a run of unescaped ASCII string content, and the encoder a
run of bytes that need no escape, with one std.bytes scan
each. Structural bytes, whitespace, numbers, literals, escapes, and non-ASCII
string bytes still go through a checked single-element Bytes call. Measured
with a release compiler on x86-64 Linux, pinned to one core, on an
8,388,872-byte document of 34,393 objects (687,862 events):
| Operation | Before the scans | Now |
|---|---|---|
Decoder.next over the whole buffer | 2.55 s | 1.58 s |
parse | 3.59 s | 2.65 s |
parse, then encode of the tree | 4.56 s | 3.28 s |
That is roughly 2 to 5 megabytes per second. The remaining time is the calls made per token and per event, not byte access.
Limits
- Compact output only: no pretty-printing or key sorting.
- One document per input: no JSON Lines or concatenated documents. Split the input and use one decoder per document.
- The tree holds integers as
i64and other numbers asf64. Larger integers and exact decimals are available only asEvent.Numbertext. write_valuebuilds the whole encoding before writing. Use anEncoderwithwrite_pendingfor bounded output memory.- No derived codecs: a typed codec is written by hand over events.
- No tree equality or path lookup helpers.
std.jwt
import std.jwt signs and verifies JSON Web Tokens (RFC 7519) in the compact
JWS form (RFC 7515), with the RFC 7518 and RFC 8037 algorithms that
std.crypto provides. Every error is a JwtError.
Upstream code, used as-is. Every MAC and signature is computed and checked by
std.crypto, which uses unmodifiedringand RustCrypto primitives;std.jwtis checked Plenty code that only parses, validates and dispatches. Seestd.cryptofor this policy.
import std.time
import std.json
import std.jwt
import std.secrets
from std.json import Json, JsonError
from std.jwt import Algorithm, JwtError
from std.secrets import SecretsError
def main() -> Result[(), AllocError | IoError | SecretsError | JsonError | JwtError]:
secret = std.secrets.token_bytes(32)?
signer = std.jwt.hmac_signing_key(Algorithm.Hs256, &secret)?
claims = std.json.parse_text("{\"sub\":\"ada\",\"aud\":\"api\",\"exp\":4102444800}", &std.json.limits())?
token = std.jwt.sign(&signer, &claims)?
# The verifying key fixes the algorithm; the header only has to agree.
key = signer.verifying_key()?
mut rules = std.jwt.validation(&std.time.wall_now())
rules.leeway = std.time.seconds(30u32)
rules.audience = Some("api")
verified = std.jwt.verify(&token, &key, &rules)?
print(verified.header.algorithm.name())?
for member in &verified.claims:
match &member.value:
case Json.Text(text):
print(member.key, text)?
case _:
pass
# Any change to the signed bytes fails the signature check.
forged = token.replace("eyJzdWIiOiJhZGEi", "eyJzdWIiOiJib2Ii")?
match std.jwt.verify(&forged, &key, &rules):
case Err(JwtError.InvalidSignature):
print("forged token rejected")?
case _:
print("unexpected")?
Ok(())
HS256
sub ada
aud api
forged token rejected
See the signature index for complete declarations.
Keys and algorithms
A key is bound to one algorithm when it is made. sign uses that algorithm,
and verify accepts a token only if its header’s alg names the same one.
The header never selects the algorithm or the key, so none, an HMAC token
checked against an RSA public key, and other algorithm-substitution attacks
fail before any cryptography runs.
Algorithm | Name | Signing key | Verifying key |
|---|---|---|---|
Hs256, Hs512 | HS256, HS512 | hmac_signing_key(algorithm, &secret) | hmac_verifying_key(algorithm, &secret) |
Rs256, Rs384, Rs512 | RS256 to RS512 | rsa_signing_key(algorithm, RsaPrivateKey) | rsa_verifying_key(algorithm, RsaPublicKey) |
Ps256, Ps384, Ps512 | PS256 to PS512 | rsa_signing_key(algorithm, RsaPrivateKey) | rsa_verifying_key(algorithm, RsaPublicKey) |
Es256 | ES256 | es256_signing_key(P256PrivateKey) | es256_verifying_key(P256PublicKey) |
EdDsa | EdDSA (Ed25519) | ed25519_signing_key(&seed) | ed25519_verifying_key(&public_key) |
Each constructor returns Result[_, JwtError], except the infallible ES256
ones. InvalidKey reports:
- an algorithm the constructor does not handle, such as
rsa_signing_key(Algorithm.Hs256, ...); - an HMAC secret shorter than the hash output: 32 bytes for HS256, 64 for
HS512 (RFC 7518 section 3.2). Generate one with
std.secrets.token_bytes; - an Ed25519 seed or public key that is not 32 bytes, or a small-order public key, which could verify forged signatures.
HMAC secrets and Ed25519 keys are copied. RSA and P-256 keys come from
std.crypto key import, which enforces their sizes and
encodings, and are moved in. SigningKey.verifying_key() returns the matching
verifying key: a copy of the secret, or the public half. algorithm() returns
a key’s algorithm and Algorithm.name() its JOSE name. Both key types are
redacted: printing one shows std.jwt.SigningKey(@redacted) or
std.jwt.VerifyingKey(@redacted), never the secret.
HS384 and ES384 are not provided: std.crypto has no SHA-384 HMAC or P-384
curve. Their names are UnsupportedAlgorithm.
Verifying
verify(&token, &key, &rules) returns a Token with the decoded header and
the claims as a list[Member] from std.json. It
checks, in order:
rulesare in range, otherwiseInvalidValidation. Amax_sizeabovemax_token_sizetraps.- The token is at most
rules.max_sizebytes, otherwiseTooLarge. Nothing is decoded before this check. - The token is three segments separated by dots, each strict unpadded
base64url (RFC 4648 section 5) with zero trailing bits. Padding, the
standard alphabet, whitespace, and any other byte are
Malformed. - The header is a JSON object (
NotObject) of strict JSON without duplicate members (Malformed).alg,kidandtypmust be strings when present. A missingalgisMalformed;noneor an unknown name isUnsupportedAlgorithm. Acritmember isCriticalHeader, since no extensions are understood (RFC 7515 section 4.1.11). Other members, includingjwk,jku,x5uandx5c, are ignored: they never supply a key. algequals the key’s algorithm, otherwiseAlgorithmMismatch.- The signature verifies over the first two segments, otherwise
InvalidSignature. HMAC tags are compared in constant time bystd.crypto’sMac.verify. - Only now are the claims parsed, with the same strictness as the header: they must be a JSON object without duplicate members.
- The claims satisfy the time, issuer and audience rules below.
decode_header(&token, max_size: Positive[i64] = 8192) performs steps 2 to 4
without a key, for choosing a key by kid; a max_size above
max_token_size traps. Its result is unauthenticated: use it only to pick
among keys the program already trusts, then call verify.
Validation rules
std.jwt.validation(now) returns a Validation with these public fields:
| Field | Default | Meaning |
|---|---|---|
now: Timestamp | the argument | A valid wall timestamp at or after the Unix epoch, such as std.time.wall_now() |
leeway: Duration | zero | Allowed clock skew, up to max_leeway seconds (86400); subsecond precision is retained |
require_expiry | True | Whether a token without exp is MissingClaim(Claim.Expiry) |
issuer | Nothing | When set, iss must be present and equal it |
audience | Nothing | The audience this program accepts |
max_size | default_max_size (8192) | Token limit in bytes, a Positive[i64] of at most max_token_size (1 MiB) |
The clock is explicit so tests and programs choose its source. The claims are checked as RFC 7519 section 4.1 describes:
exp:Expiredoncenow - leeway >= exp. A fractional value is rounded down.nbfandiat:NotYetValidwhen more thanleewayseconds in the future. Fractional values are rounded up. Aniatin the future is rejected because the token cannot have been issued yet.- A time claim that is not a number, or is beyond the
i64range, isInvalidClaimwith itsClaim. iss, whenissueris set: missing isMissingClaim(Claim.Issuer), a non-string isInvalidClaim(Claim.Issuer), and a different string isWrongIssuer. The comparison is exact.audis a string or an array of strings; anything else isInvalidClaim(Claim.Audience). It must containaudience, otherwiseWrongAudience. A token withaudisWrongAudiencewhenaudienceisNothing, since the program is not among its recipients (RFC 7519 section 4.1.3). Whenaudienceis set, a missingaudisMissingClaim(Claim.Audience).
Other claims, including sub and jti, are returned unchecked. typ is
returned in Header.token_type and not checked.
Signing
sign(&key, &claims, key_id) returns the compact token as str. claims
must be a Json.Object, otherwise NotObject; it is encoded compactly in
member order, and a value JSON cannot represent, such as a NaN, is
Malformed. The header is {"alg":"...","typ":"JWT"}, with kid added when
key_id is Some. HS, RS and EdDSA signatures are deterministic; PS and ES
signatures use operating-system randomness, and its failure is
RandomFailure. sign adds no claims: set exp and iat yourself.
Errors
JwtError has a std.report format method; no message
repeats token contents or key material.
| Variant | Meaning |
|---|---|
Malformed | Not three strict base64url segments of strict JSON, a header without alg, or claims that cannot be encoded |
NotObject | The header or claims are not a JSON object |
TooLarge | The token exceeds max_size |
UnsupportedAlgorithm | none or an algorithm this module does not provide |
AlgorithmMismatch | The header’s alg is not the key’s algorithm |
CriticalHeader | The header has crit |
InvalidSignature | The signature or MAC does not verify |
Expired, NotYetValid | The time claims reject now |
MissingClaim(Claim), InvalidClaim(Claim) | A required claim is absent, or a claim has the wrong type or range |
WrongIssuer, WrongAudience | iss or aud does not match |
InvalidKey | A key does not suit its algorithm |
InvalidValidation | now or leeway is out of range |
RandomFailure | Randomness failed while signing |
Allocation(AllocError) | An allocation failed |
Claim is Expiry, NotBefore, IssuedAt, Issuer or Audience.
Memory
verify allocates a copy of the token, the decoded header, claims and
signature, and the claims tree; sign allocates the encoded segments and the
result. All of these are bounded by max_size when verifying and by the
claims when signing. Every allocation is fallible and reported as
Allocation; tests fail each allocation in turn and check that no other
error results and nothing leaks.
Memory-constrained programs should use ES256 or EdDSA, which allocate nothing
inside std.crypto beyond the signature they return. HMAC allocates one
368-byte state per operation. RSA reserves
up to 12 KiB per private key while it lives, plus 8 KiB per signature and
12 KiB per verification, and reports a failed reservation as Allocation;
see RSA and ECDSA keys.
Limits
- Only compact JWS. JWE (encrypted tokens), the JSON serialization, unencoded payloads (RFC 7797) and nested tokens are not supported.
- No JWK or JWKS handling: import keys through
std.crypto, which accepts JWK components. - No HS384, ES384, ES512 or
ES256K. - Header nesting is limited to 4 levels and claims to 32.
Arithmetic bounds
Dot offsets refer to found bytes inside the size-limited token. Time comparisons
validate a nonnegative clock and at most max_leeway seconds of leeway before
subtracting or scaling; large future dates are rejected before nanosecond
multiplication. These bounds-proven arithmetic opt-outs keep the existing JWT
validation errors, including fractional-second comparisons at integer limits.
Acknowledgements
- Token format, claims and algorithms follow RFC 7519 (JWT), RFC 7515 (JWS), RFC 7518 (JWA) and RFC 8037 (EdDSA in JOSE). Tests use the RFC 7515 appendix A.1 to A.3 and RFC 8037 appendix A.4 examples.
- The key-bound API and validation settings were modelled on the Rust
jsonwebtokencrate; no code is used. - Cryptography is
std.crypto’s; see its acknowledgements.
std.log
import std.log provides leveled events with typed fields and a static sink
contract. There is no global logger: a program builds a Logger around a sink
it chooses and passes the logger, or the sink, to the code that logs. A library
that logs takes &mut Logger[S] as a generic parameter, so its caller decides
the destination, threshold, and clock. The module is checked Plenty over
std.io, std.fmt, and std.time, with no
native code of its own. It is available on the supported x86-64 Linux GNU
target.
import std.io
import std.log
from std.log import Clock, Level, LogError
def main() -> Result[(), AllocError | IoError | LogError]:
# Programs usually log to std.io.stderr() with Clock.Wall timestamps.
sink = std.log.text_sink(std.io.stdout(), 4096)?
mut log = std.log.logger(sink, Level.Info, Clock.Off)
log.info("listening", [std.log.integer("port", 8080), std.log.text("host", "db")]?)?
log.debug("below the threshold", []?)?
log.warn("slow request", [std.log.floating("seconds", 2.5)]?)?
print(log.emitted(), log.filtered())?
Ok(())
level=info msg="listening" port=8080 host="db"
level=warn msg="slow request" seconds=2.5e0
2 1
Events
| Type | Contents |
|---|---|
Level | Trace, Debug, Info, Warn, Error, in increasing severity |
Value | Text(str), Integer(i64), Unsigned(u64), Floating(f64), Boolean(bool) |
Field | key: str, value: Value |
Event | level: Level, message: str, fields: list[Field], time: Option[Timestamp] |
All fields are public. text(key, value), integer, unsigned, floating,
and boolean build a Field; event(level, message, fields) builds an
unstamped Event. An event is wholly owned, so a sink may keep it or move it to
another thread. Fields keep the order the caller gave.
level_name(&level) returns the lowercase name, such as "warn".
parse_level(&name) accepts exactly those five names and returns
UnknownLevel for anything else, including other capitalizations.
Loggers
logger(sink, threshold: Level, clock: Clock) -> Logger[S] takes ownership of
sink. The sink is public as the field sink.
Logger[S] | Result | Contract |
|---|---|---|
enabled(&level) | bool | Whether an event at level would reach the sink |
emit(event) | Result[(), LogError] | Filter, stamp, and deliver one event |
log(level, message, fields) | Result[(), LogError] | Build and emit an event |
debug, info, warn, error (message, fields) | Result[(), LogError] | log at that level |
emitted(), filtered(), failed() | u64 | Events delivered, below the threshold, and rejected by the sink |
The diagnostic u64 event counters saturate at u64.MAX; exhaustion never
changes filtering, delivery, or the sink error returned to the caller.
An event below the threshold is dropped and counted as filtered; that is
Ok(()), not an error. The fields of a filtered event have already been
built, so guard expensive fields with enabled first.
Clock | Stamp for an event whose time is Nothing |
|---|---|
Off | None; time stays Nothing |
Wall | std.time.wall_now() at emission |
Fixed(Timestamp) | That timestamp, for reproducible output and tests |
An event that already has a time keeps it, so replayed or forwarded events
are not restamped. Stamping happens after filtering.
A sink error is returned unchanged and counted as failed. The logger does not
retry, buffer, or fall back to another sink. The caller chooses whether a
logging failure matters: propagate it, or drop the result and inspect
failed() later.
Sinks
A sink is any pub class with this method, resolved at compile time:
| Protocol | Method | Result |
|---|---|---|
Sink | emit(self: &mut, event: Event) | Result[(), LogError] |
The sink owns the event from then on, even on failure.
| Constructor | Sink | Delivery |
|---|---|---|
text_sink(writer: W, max_line: Positive[i64]) | Result[TextSink[W], LogError] | One logfmt line per event to any Writer |
memory_sink(capacity: NonNegative[i64]) | Result[MemorySink, LogError] | Keeps up to capacity events in the public list events |
channel_sink(sender: Sender[Event]) | ChannelSink | Queues the event without waiting |
discard() | Discard | Accepts and drops every event |
Text lines
TextSink writes logfmt: space-separated key=value pairs, then a newline.
time=1700000000.000001234 level=info msg="listening" port=8080 host="db" tls=false load=2.5e-1
timeappears only for a stamped event: seconds since the Unix epoch, a dot, and exactly nine digits of nanoseconds. Atimewhosenanosecondsis 1,000,000,000 or more returnsInvalidSpecand writes nothing.levelis the lowercase name.msgand text values use thedebug_textform: double quotes, with",\, LF, CR, TAB, and other control characters escaped, so one event is always one line.- Integers are decimal. Floats use the shortest round-trip scientific form
of
std.fmt. Booleans aretrueandfalse. - A key must be nonempty ASCII letters, digits,
_,., or-, and must not betime,level, ormsg. OtherwiseemitreturnsInvalidKeyand writes nothing.
max_line bounds one line in bytes, newline included; 0 is a compile error.
An event whose line would be longer returns TooLarge and
writes nothing: lines are never truncated. Each line is built in a buffer
reserved at construction, then delivered with one
write_all followed by flush. A write error is
Output(cause); part of the line may already have reached the writer.
TextSink exposes the writer as the public field writer.
For a command-line program, use text_sink(std.io.stderr(), ...). Lines then
share the process buffer with print, keep their order relative to it, and
are flushed per event.
Memory
MemorySink reserves room for capacity events at construction. Once full,
emit returns Full and the event is dropped; the list never grows. Tests
inspect sink.events directly.
Channels and threads
ChannelSink sends with send_nowait on a bounded channel. A
full queue returns Full and drops the event, so a slow consumer never blocks
the logging thread. A queue with no receiver returns Closed. share()
returns another handle on the same queue for another thread’s logger.
Drained.delivered and Drained.failed also saturate at u64.MAX, without
stopping the consumer or replacing a sink result.
drain(receiver: &Receiver[Event], sink: &mut S) -> Drained delivers queued
events to sink until every sender is gone. It waits for events. A sink
error is counted in Drained.failed and draining continues;
Drained.delivered counts the rest. Run it on one thread to serialize output
from many producers:
def consume(receiver: Receiver[Event]) -> Result[Drained, LogError]:
mut lines = std.log.text_sink(std.io.stderr(), 4096)?
Ok(std.log.drain(&receiver, &mut lines))
The usual join rules
apply: a ChannelSink holds a sender, so drop every handle before joining
the draining thread.
Errors
LogError | Meaning |
|---|---|
InvalidSpec | An event time with out-of-range nanoseconds |
InvalidKey | A field key TextSink cannot write unambiguously |
UnknownLevel | parse_level received another name |
TooLarge | The line would exceed max_line |
Full | A bounded sink had no room; the event was dropped |
Closed | The channel has no receiver |
Format(FormatError) | std.fmt rejected a value |
Output(StreamError) | The writer failed |
Allocation(AllocError) | A sink’s storage could not be reserved |
Ownership and threads
Every emission takes &mut, so a logger and its sink have one user at a time.
Threads do not share one logger: each thread builds its own Logger around a
ChannelSink handle from share(), and one thread drains the queue. Events
move through the channel without copying. A Logger holding a TextSink over
a stream can also be moved into a worker like any owned value.
Costs
| Allocates | Does not allocate |
|---|---|
| Building fields and the field list, as the caller writes them | Logger.emit and the shorthand methods |
text_sink: a 32-byte scratch buffer and a max_line line buffer | TextSink.emit, apart from the writer’s own storage |
memory_sink: a list of capacity events | MemorySink.emit |
channel(...): the queue, made by the caller | ChannelSink.emit, share, drain |
TextSink.emit is linear in the line length; each byte passes through
std.fmt and is copied once into the line buffer. The line then costs one
write_all and one flush: over stderr() that is one buffered write, and
over a FileStream at least one system call. Dropping an event frees its
fields.
Limits
- Only x86-64 Linux GNU is supported.
- There is no JSON output, no sampling or rate limiting, no per-module
thresholds, and no span or context propagation. Compose these from
Eventand a custom sink. - Sinks are static: a logger cannot switch sinks at run time, and there is no sink object that erases the concrete class. Fan-out to two destinations is a custom sink that writes to both.
LineBufferis public only becausestd.fmtwrites into it; it is not meant for direct use.
Acknowledgements
- Typed fields and caller-chosen sinks are modelled on Go’s
log/slog; the five levels match the Rusttracingcrate.
std.markdown
import std.markdown parses CommonMark 0.31.2
into a stream of events and renders events as HTML, under explicit limits and
with one error type, MarkdownError. It has two levels:
- Events: a
ParseryieldsStart,End, text, and break events in document order. Programs can inspect, filter, or rewrite them. - HTML: an
HtmlRendererturns events into HTML text.to_html,render, andwrite_htmlconnect a parser to a renderer.
The module is checked Plenty code with no native primitives of its own. It
follows the algorithms of the reference implementation, commonmark.js, and
renders the HTML that the specification’s examples show. It reads
std.bytes buffers, writes std.io streams, and is
available on the supported x86-64 Linux GNU target.
Accepted text
| Rule | Behavior |
|---|---|
| Syntax | CommonMark 0.31.2: every block and inline construct of the specification, and nothing more |
| Extensions | None: no tables, task lists, strikethrough, footnotes, extended autolinks, or front matter |
| Encoding | UTF-8, validated strictly; invalid input is InvalidUtf8 at the first bad byte |
| Line endings | \n, \r\n, and \r all end a line |
NUL | Replaced by U+FFFD, as the specification requires |
| Raw HTML | HTML blocks and inline tags pass through unchanged as Html and InlineHtml events |
| Unicode | Punctuation and whitespace classes for emphasis come from Unicode 16.0.0; link labels match after Unicode case folding by lowering then raising |
| Entities | All HTML5 named character references, and decimal and hexadecimal references |
Raw HTML is not sanitized. For untrusted input, drop or escape Html and
InlineHtml events before rendering, and check link and image destinations,
which may use any scheme, including javascript:.
Limits
pub class Limits:
pub max_input: NonNegative[i64]
pub max_depth: NonNegative[i64]
pub max_output: NonNegative[i64]
limits() returns Limits(67108864, 64, 67108864). The fields are public;
adjust a copy. Every constructor and whole-document function takes &Limits.
A literal converts on assignment; a computed value needs
NonNegative.try_from(value)?.
| Field | Bounds | When exceeded |
|---|---|---|
max_input | Bytes of Markdown given to one parser | InputTooLarge; the offset is the limit |
max_depth | Block quotes and list items open at once. 0 allows neither | DepthExceeded on the line that would exceed it |
max_output | Total bytes one renderer produces, including bytes already taken or written | OutputTooLarge |
Inline nesting, such as emphasis inside links inside emphasis, is not limited
by max_depth: the parser and renderer walk it without recursion. Running time
is linear in the input, including on the inputs that make naive parsers
quadratic, such as long runs of unclosed brackets, backticks, or HTML comments.
Link destinations may nest parentheses 32 deep, as in cmark.
Errors
pub enum MarkdownFault:
InvalidUtf8
InputTooLarge
OutputTooLarge
DepthExceeded
InvalidState
Stream(StreamError)
Allocation(AllocError)
pub class MarkdownError:
pub fault: MarkdownFault
pub offset: i64
pub line: i64
| Fault | offset | line |
|---|---|---|
InvalidUtf8 | The first invalid input byte | 0 |
InputTooLarge | max_input for the raw input limit; 0 (unavailable) for tab-expanded column exhaustion | 0 |
DepthExceeded | The start of the offending line in the input | That line, from 1 |
OutputTooLarge, Stream | Bytes the renderer had produced | 0 |
| Others | 0, or bytes produced for a renderer | 0 |
After an error from next or event, later calls to that method return
InvalidState. Renderer output already produced remains available through
take or write_pending; a stream failure from write_pending can be retried.
Tab-expanded columns remain exact. If advancing a column is unrepresentable,
parsing returns InputTooLarge with offset and line both 0: a raw input
location is unavailable, not known to be byte 0. Allocation extents that cannot
be represented return Allocation(AllocError.CapacityOverflow).
Events
pub enum Tag:
Paragraph
Heading(i64) # level 1 to 6
BlockQuote
CodeBlock(str) # info string; empty for indented code
BulletList
OrderedList(i64) # start number
Item
Emphasis
Strong
Link(str, str) # destination, title
Image(str, str) # destination, title
pub enum Event:
Start(Tag)
End(Tag)
Text(str)
Code(str) # inline code span
Html(str) # raw HTML block
InlineHtml(str) # raw inline tag, comment, or declaration
SoftBreak
HardBreak
Rule
Done
pub def parser(source: &[u8], bounds: &Limits) -> Result[Parser, MarkdownError]
pub def parser_text(text: &str, bounds: &Limits) -> Result[Parser, MarkdownError]
Parser.next(self: &mut Parser) -> Result[Event, MarkdownError]
parser validates the input and parses its block structure: paragraphs,
headings, lists, quotes, code, and link reference definitions. It copies what
it needs, so source stays free. next parses inline content one paragraph
or heading at a time, so a parser holds the block tree and one leaf’s inline
events, not the whole event stream.
Every Start has a matching End with an equal tag. Text, entities, and
backslash escapes are resolved: Text holds the characters to show, and
adjacent text arrives as one event. A code block’s content, including its final
line ending, is one Text event between Start(CodeBlock) and
End(CodeBlock); an empty block has none. Html holds a raw block without its
final line ending.
Paragraphs directly inside items of a tight list produce no Paragraph events,
so a renderer need not track tightness. Link reference definitions produce no
events; references to them arrive as Link and Image with the definition’s
destination and title. Inside Image, the events between Start and End
are its description; HTML renders them as plain text in alt.
After the last event, next returns Done, and keeps returning it.
HTML
pub def renderer(bounds: &Limits) -> Result[HtmlRenderer, MarkdownError]
HtmlRenderer.event(self: &mut HtmlRenderer, event: &Event) -> Result[(), MarkdownError]
HtmlRenderer.pending(self) -> i64
HtmlRenderer.written(self) -> i64
HtmlRenderer.take(self: &mut HtmlRenderer) -> Result[Bytes, MarkdownError]
pub def write_pending[W: Writer](source: &mut HtmlRenderer, writer: &mut W) -> Result[(), MarkdownError]
pub def write_html[W: Writer](source: &mut Parser, writer: &mut W, bounds: &Limits) -> Result[(), MarkdownError]
pub def render(source: &[u8], bounds: &Limits) -> Result[Bytes, MarkdownError]
pub def to_html(text: &str, bounds: &Limits) -> Result[str, MarkdownError]
A renderer appends the HTML for each event to an owned buffer. pending counts
bytes not yet taken; written counts every byte produced. take returns the
buffer and leaves an empty one. write_pending writes the buffer to a writer;
on failure only the bytes the writer refused stay pending, so a retry cannot
repeat output. write_html renders every remaining event of a parser, writing
whenever 64 KiB is pending; neither function flushes.
The output matches the specification’s examples:
| Event | HTML |
|---|---|
Paragraph, Heading(n) | <p>…</p>, <hn>…</hn> |
BlockQuote | <blockquote> and </blockquote> on their own lines |
BulletList, OrderedList(n) | <ul>, <ol>, or <ol start="n"> when n is not 1 |
Item | <li>…</li> |
CodeBlock(info) | <pre><code class="language-w">…</code></pre>, where w is the first word of info; no class when info is empty |
Emphasis, Strong | <em>, <strong> |
Link(d, t) | <a href="d" title="t">; no title when t is empty |
Image(d, t) | <img src="d" alt="…" title="t" /> |
Code(s) | <code>s</code> |
Html(s), InlineHtml(s) | s, unescaped |
SoftBreak, HardBreak, Rule | a line ending, <br />, <hr /> |
Text and attributes escape &, <, >, and ". Destinations in href and
src are percent-encoded except for ASCII letters, digits, and
!#$%()*+,-./:;=?@_~^, with & written as & and ' as '.
Rewritten events must retain document order and valid tags. A heading level
outside 1..6, or unrepresentable nested image depth, returns InvalidState
without emitting bytes for that failing event. Earlier output and its exact
written total remain intact; the renderer is then broken. Output-budget
exhaustion, including unrepresentable byte totals, returns OutputTooLarge
without wrapping or saturating the count.
Each block starts on its own line, and the output ends with a line ending unless it is empty.
Example
import std.markdown
from std.markdown import Event, MarkdownError, Tag
def main() -> Result[(), AllocError | IoError | MarkdownError]:
bounds = std.markdown.limits()
text = "# Notes\n\nSee [the guide](/guide) and *more*.\n"
print(std.markdown.to_html(text, &bounds)?)?
mut events = std.markdown.parser_text(text, &bounds)?
while True:
event = events.next()?
match event:
case Event.Start(tag):
match tag:
case Tag.Link(destination, _):
print("link to", destination)?
case _:
pass
case Event.Done:
break
case _:
pass
Ok(())
<h1>Notes</h1>
<p>See <a href="/guide">the guide</a> and <em>more</em>.</p>
link to /guide
See the runnable Markdown lesson.
Implementation notes
The Unicode punctuation ranges and the HTML5 entity table are generated into
the module by scripts/gen_markdown_tables.py from Python’s unicodedata and
html.entities; rerun it to move to a newer Unicode version.
The parser and renderer are derived from commonmark.js 0.31.2, Copyright (c)
2014 John MacFarlane, under the BSD-2-Clause license. The punctuation table
derives from the Unicode Character Database (Unicode License v3) and the entity
table from the WHATWG HTML Standard (BSD-3-Clause for portions in source code).
Release archives include these notices in THIRD-PARTY-NOTICES.md.
Acknowledgements
- Parser and renderer ported from commonmark.js 0.31.2 (BSD-2-Clause).
- Punctuation table generated from the Unicode Character Database 16.0.0 (Unicode-3.0).
- Entity table generated from the WHATWG HTML Standard named character references (BSD-3-Clause).
- The link parenthesis nesting limit follows cmark.
- The event API takes pulldown-cmark as a design reference; no code is taken from it.
std.template
Bounded text templates, implemented in Plenty. Register named sources once with
Engine.add, then render them repeatedly with explicit std.json.Json objects.
No files are opened and no Plenty code, shell commands or methods are evaluated.
import std.template
from std.template import TemplateError
from std.json import Json, Member
def main() -> Result[(), TemplateError | AllocError | IoError]:
bounds = std.template.limits()
mut templates = std.template.engine(&bounds)?
templates.add("greeting", "<p>Hello {{ name }}!</p>")?
values = Json.Object([Member("name", Json.Text("Ada & Grace"))]?)
print(templates.render("greeting", &values)?)?
Ok(())
<p>Hello Ada & Grace!</p>
Language
The syntax is a small Jinja-style subset, using Tera as a syntax reference, not a Tera/Jinja compatibility implementation.
| Form | Meaning |
|---|---|
{{ person.name }} | Look up an object field; items.0 indexes an array |
{{ 'hello' }}, {{ 42 }}, {{ true }} | Text, numeric, boolean and null literals |
{{ name | upper }} | Apply a filter; filters can be chained |
{# comment #} | Ignore a comment |
{% if active %}yes{% else %}no{% endif %} | Conditional, with optional else |
{% if not active %}...{% endif %} | Negate a condition |
{% for item in items %}{{ item.name }}{% else %}empty{% endfor %} | Array loop; optional else renders only for an empty array |
{% block title %}Title{% endblock %} | Define a top-level named block |
{% extends 'base' %} | Inherit from a registered template |
Names use ASCII letters, digits and underscores, starting with a letter or
underscore. A path’s later segments may be nonnegative array indices. Text
literals support \\, \', \", \n, \r and \t. Conditions treat
null, false, zero and empty text/arrays/objects as false. Missing fields are
errors, including in conditions. Arrays and objects cannot be interpolated
directly; loop over an array or select a scalar field.
The innermost loop provides loop.index (one-based), loop.index0, loop.first,
loop.last and loop.length. A loop variable shadows an outer value with the
same name and is removed when that loop ends. The name loop is reserved for
loop metadata. Loops accept a single variable and an array path, without spaces
inside that path. Object iteration is not supported.
Filters are upper, lower, trim, length, escape and safe. Text filters
require text. length counts Unicode scalars for text and entries for arrays
or objects. Filters do not accept arguments. Arithmetic, comparisons, boolean
operators other than not, calls, bracket indexing, elif, assignments,
includes, macros, raw blocks and whitespace-trimming delimiters are not part
of this version. Unsupported syntax is rejected when registering a template,
including inside branches that would never execute.
Escaping and inheritance
render(name, context, html=True) escapes &, <, >, " and ' in
interpolated values. Literal template text is trusted. escape marks its
escaped result safe, avoiding double escaping; safe explicitly trusts the
value. upper and lower remove that trust. Use html=False for plain-text
emails or generated text.
This escaping is for HTML text and quoted ordinary attributes. It does not
validate URLs or make values safe inside JavaScript, CSS, tag names, unquoted
attributes or raw HTML. Template authors control safe and literal markup;
do not treat untrusted templates as a markup sanitizer.
Escaping individual values
Use escape(text: &str, context: HtmlContext, max_bytes: NonNegative[i64] = 16777216)
when assembling a fragment directly or matching another renderer’s exact output.
It returns Result[str, TemplateError]. Choose the context from the position
where the value will appear; the escaper does not parse surrounding HTML.
HtmlContext | Escapes | Intended position |
|---|---|---|
Text | &, <, > | Ordinary HTML text children |
DoubleQuotedAttribute | &, " | Inside a double-quoted ordinary attribute value |
SingleQuotedAttribute | &, ' | Inside a single-quoted ordinary attribute value |
TextAndAttributes | &, <, >, ", ' | The conservative policy used by render and the escape filter |
Entities are &, <, >, " and ', respectively.
Other UTF-8 bytes are preserved. The attribute modes do not add delimiters;
keep the matching quotes around the complete attribute value. A text-escaped
value can still contain quotes and is unsuitable for an attribute without
escaping its original input for that attribute’s context.
from std.template import escape, HtmlContext, TemplateError
def main() -> Result[(), TemplateError | AllocError | IoError]:
name = "Ada & \"friends\""
title = escape(&name, HtmlContext.DoubleQuotedAttribute, max_bytes=128)?
child = escape(&name, HtmlContext.Text, max_bytes=128)?
print(f"<span title=\"{title}\">{child}</span>"?)?
Ok(())
<span title="Ada & "friends"">Ada & "friends"</span>
The input is borrowed and never modified; success returns ordinary owned text,
with no trust marker. Passing that text through Json.Text into render
applies normal escaping again. Prefer raw input for template values and these
helpers for directly assembled fragments. safe remains an explicit trust
assertion by the template author; it does not check an escaping context.
Escaping treats the input as text, including existing entities: & becomes
&amp;. Call it once on the original value for the intended context;
repeated calls are not idempotent. The new function does not change filter
trust: escape uses TextAndAttributes, upper/lower revoke trust, and
trim preserves it.
max_bytes caps UTF-8 output, including entity expansion. Oversized input is
rejected before allocation because escaping cannot shrink it. Each input byte
is scanned once; replacement entities are appended directly and are not
revisited. The output cap therefore also bounds source bytes scanned and
copied. The function may allocate source/output buffers and returned text;
allocation failures are TemplateError.Allocation or an allocation cause
inside TemplateError.Bytes. Exceeding the cap returns LimitExceeded, with
no partial output. A zero cap accepts only empty text; a dynamic cap uses
NonNegative.try_from(value)?.
These contexts do not support HTML comments, tag/attribute names, unquoted attributes, scripts, styles or URL validation. Choosing an enum value is explicit caller responsibility, not proof that its surrounding HTML is valid.
Inheritance
extends must precede every non-whitespace, non-comment construct. An extending
template contains only whitespace, comments and top-level blocks. Blocks cannot
be nested or repeated within a template. The most-derived definition replaces
its ancestor’s entire block; no super() call is provided. Blocks absent from
the base’s layout are not rendered. Parent names are literal registry names,
not filesystem paths. Missing parents and inheritance cycles are render errors.
Values, limits and errors
The root context must be Json.Object. Build values explicitly with
Json and Member, or supply a value from the JSON decoder.
Object keys must be unique and float values finite throughout the context.
Values are read-only; rendering builds an indexed view and does not modify or
deep-copy the caller’s tree. This API needs no serialization derivation. A
future derived serializer can provide the same JSON data model without changing
template syntax; derived serialization
is currently a proposal, not an implemented conversion.
limits() returns:
| Field | Type | Default | Bounds |
|---|---|---|---|
max_source | NonNegative[i64] | 1,048,576 | Total UTF-8 bytes of registered names and sources |
max_tokens | NonNegative[i64] | 16,384 | Total compiled tokens across templates |
max_templates | NonNegative[i64] | 128 | Registered template count |
max_context_nodes | Positive[i64] | 65,536 | Values in a render context, including its root |
max_context_bytes | NonNegative[i64] | 16,777,216 | UTF-8 bytes of context keys and text values |
max_depth | Positive[i64] | 64 | Syntax/context/render nesting and inheritance chain; at most 64 |
max_steps | NonNegative[i64] | 1,000,000 | Render operations, including context construction, lookup and loop visits |
max_output | NonNegative[i64] | 16,777,216 | UTF-8 output bytes and transformed text values |
A literal converts where a field is expected; a computed value needs
Positive.try_from(value)? or NonNegative.try_from(value)?.
engine(&bounds) traps when max_depth is above 64 and copies the settings. add compiles and validates before
registering; a failure leaves the engine unchanged. Names cannot be empty or
registered twice. A render failure returns no partial output and leaves the
engine reusable. Limits bound sizes and work counts, not an exact heap-byte
quota or wall-clock deadline. Budget totals remain exact: an unrepresentable
source, token, context-byte or work total returns TemplateError.LimitExceeded,
without wrapping or saturating. Registration commits its totals only after the
compiled template is appended successfully. Text filters may allocate intermediate
text.
TemplateError distinguishes exceeded limits, syntax with a
zero-based source location, missing/duplicate template names, missing values,
type mismatches, cycles, allocation failures and byte-buffer errors. Syntax
locations are UTF-8 byte offsets at the containing template construct, rather
than at an individual expression character. Allocation errors remain explicit, including those
wrapped by BytesError.
Acknowledgements
std.math
std.math provides floating-point functions with recoverable domain and
overflow errors, checked rounding to integers, integer algorithms, generic
numeric helpers, and constants. Its source is stdlib/std/math.plenty.
Importing, the reserved std namespace, and compilation are common
to every standard module.
import std.math as math
from std.math import pi, MathError
Domain algorithms report MathError.Domain and MathError.Overflow.
Integer magnitude and reduction helpers preserve OverflowError[T], including
its wrapped numeric payload. No function in this module allocates.
import std.math as math
from std.math import MathError
def main() -> Result[(), AllocError | IoError | MathError]:
print(math.sqrt(2.0)?, math.log10(1000.0)?, math.round(2.5)?)?
print(str.repr(math.sqrt(-1.0))?)?
print(math.isclose(0.1 + 0.2, 0.3, 1e-9, 0.0)?)?
print(math.gcd(84u64, 36u64), math.comb(10u64, 3u64)?)?
print(math.max(3, 9), math.abs_float(-4.5), math.pi)?
Ok(())
1.4142135623730951 3.0 2
Result[f64, std.math.MathError].Err(std.math.MathError.Domain)
True
12 120
9 4.5 3.141592653589793
Floating-point functions
Inputs are f64, following Python’s double-precision math surface. Explicitly
convert integers and f32 inputs. Angles use radians.
| Functions | Result |
|---|---|
sqrt(x), cbrt(x) | Result[f64, MathError] |
exp(x), exp2(x), expm1(x) | Result[f64, MathError] |
log(x), log2(x), log10(x), log1p(x), log_base(x, base) | Result[f64, MathError] |
sin(x), cos(x), tan(x), asin(x), acos(x), atan(x), atan2(y, x) | Result[f64, MathError] |
sinh(x), cosh(x), tanh(x), asinh(x), acosh(x), atanh(x) | Result[f64, MathError] |
pow(x, y), fmod(x, y) | Result[f64, MathError] |
floor(x), ceil(x), trunc(x), round(x) | Result[i64, MathError] |
divmod_float(a, b) | Result[(f64, f64), MathError] |
fabs(x), copysign(x, y), degrees(x), radians(x) | f64 |
isnan(x), isinf(x), isfinite(x) | bool |
isclose(a, b, rel_tol, abs_tol) | Result[bool, MathError] |
MathError.Domain reports invalid real operations, including negative square
roots, nonpositive logarithms, infinite trigonometric inputs, and zero finite
negative powers. MathError.Overflow reports finite inputs whose result
exceeds the floating or integer result range. Underflow succeeds. NaNs
propagate through floating results; rounding NaN is a domain error. Rounding
infinity or a value outside [-2**63, 2**63) overflows. Floating primitives
preserve signed zero; fabs clears its sign and copysign observes it.
Predicates and angular conversions have no recoverable failure. Angular
conversions follow IEEE arithmetic, including overflow to infinity. Integer
.pow remains a distinct exact operation with the language’s recoverable
integer arithmetic contract.
round rounds to the nearest integer, choosing the even integer at a halfway
value: round(2.5) is Ok(2) and round(3.5) is Ok(4). It takes no decimal
places argument. The numeric .round() method keeps the float type and signed
zero; the library function checks the rounded value before converting to i64.
divmod_float returns the floor quotient and a remainder with the divisor’s
sign, deriving both from the same floating remainder to avoid inconsistent
results near integral quotients. Zero remainders retain the divisor’s sign;
zero quotients retain the sign of a / b. A zero divisor or any non-finite
input is Domain; a quotient beyond the finite float range is Overflow.
Floating rounding means a == q * b + r need not hold exactly.
isclose follows Python’s weak symmetric comparison: exact equality succeeds
first, unequal infinities fail, and otherwise either relative tolerance or the
absolute tolerance may accept the difference. Negative tolerances produce
Domain; NaNs are never close. Both tolerances are required parameters;
1e-9, 0.0 matches Python’s defaults.
The module exports pi, e, tau, inf, and nan as immutable f64
module constants. They can also be imported
individually. Numeric types retain f64.PI, f64.E, f64.TAU, f64.INF, and
f64.NAN, with the same names on f32.
Integer algorithms and generic helpers
| Function | Contract |
|---|---|
factorial(n: NonNegative[i64]) -> Result[u64, MathError] | Inputs above 20 overflow |
gcd(a: u64, b: u64) -> u64 | Euclidean algorithm; gcd(0, 0) is zero |
lcm(a: u64, b: u64) -> Result[u64, MathError] | Either zero input gives zero; checked multiplication |
isqrt(n: u64) -> u64 | Exact floor square root over the entire u64 range |
perm(n: u64, k: u64) -> Result[u64, MathError] | Ordered selections; k > n gives zero |
comb(n: u64, k: u64) -> Result[u64, MathError] | Unordered selections; k > n gives zero; cancels factors before multiplication |
abs[T: IntType](x: T) -> Result[T, OverflowError[T]] | Integer magnitude; a signed minimum returns its wrapped overflow payload |
abs_float[T: FloatType](x: T) -> T | Floating magnitude; clears the sign of zero |
min[T](a: T, b: T) -> T, max[T](a: T, b: T) -> T | Compare two owned comparable values; equal values retain the first |
min_all[T](items: &list[T]) -> Option[T], max_all[T](items: &list[T]) -> Option[T] | Copy the first extremum; Nothing for an empty list; elements must be copyable and comparable |
sum[T: IntType](items: &list[T], start: T) -> Result[T, OverflowError[T]], prod[T: IntType](items: &list[T], start: T) -> Result[T, OverflowError[T]] | Sequential integer fold; first overflow returns its wrapped payload; empty lists return Ok(start) |
sum_float[T: FloatType](items: &list[T], start: T) -> T, prod_float[T: FloatType](items: &list[T], start: T) -> T | Sequential IEEE floating fold; empty lists return the explicit start |
sum_iter[I, T: IntType](items: &I, start: T) -> Result[T, OverflowError[T]], prod_iter[I, T: IntType](items: &I, start: T) -> Result[T, OverflowError[T]] | Borrowed list, set, range, or dictionary-key integer fold |
sum_iter_float[I, T: FloatType](items: &I, start: T) -> T, prod_iter_float[I, T: FloatType](items: &I, start: T) -> T | Borrowed floating fold; elements must match the start type |
divmod[T: IntType](a: T, b: T) -> Result[(T, T), MathError] | Same-type integer floor quotient and remainder; zero divisor is Domain, signed minimum divided by -1 is Overflow |
Integer combinatorics report Overflow when the result cannot fit u64.
Unlike Python, Plenty has no arbitrary-precision integer type. gcd and lcm
take two unsigned inputs; convert signed magnitudes explicitly. Generic helpers
specialize against operations in their bodies, with normal language errors for
unsupported types. They allocate nothing. Integer helpers return ordinary
public Result values even when called inside unchecked:; caller modes
do not change a callee’s contract. sum_float uses sequential IEEE 754
addition, with no compensated summation guarantee; any NaN element makes the
sum NaN, and overflow may produce infinity. The separate float names avoid
requiring impossible integer error handling from float callers.
Combinatorics and divmod handle local arithmetic results and preserve their
established MathError contract. The infallible unsigned gcd remainder and
isqrt loop use explicit unchecked: regions with bounds documented in
source. These are proven-bounded operations, not wrapping arithmetic.
Collection helpers borrow their inputs and preserve them. For f32 and f64,
min, max, min_all, and max_all follow IEEE 754-2019 minimum and
maximum: -0.0 orders below 0.0, so min(0.0, -0.0) is -0.0 and
max(-0.0, 0.0) is 0.0, and any NaN input makes the result NaN (Some(NaN)
for the list helpers), regardless of its position. The NaN returned is the
first NaN input. The pipeline min() and max()
terminals share this rule.
Other types use ordinary comparisons and retain the first of equal values.
Tuple elements retain their existing lexicographic total
order, including the ordering of NaN fields;
they are not scalar NaNs. Dictionary
folds visit keys in insertion order; set order is unspecified. The fold start is
always explicit and determines the result type. Generators and custom iterator
protocols are not borrowed iterables; strings have nonnumeric elements.
Runtime boundary and current limits
Only numeric methods and constants are compiler/runtime primitives. Platform
transcendentals, cancellation-sensitive expm1/log1p, and IEEE sign operations
need accurate scalar operations that ordinary arithmetic cannot substitute for.
The existing numeric-method boundary exposes these to f32, f64, and generic
FloatType functions, so user libraries can use them too. Algorithms, predicates,
conversions, tolerance comparisons, guards, and error construction remain Plenty
source. Generic arithmetic and integer combinatorics need no runtime additions.
This first module is inspired by CPython’s Doc/library/math.rst and
Modules/mathmodule.c, particularly its domain/overflow policy and isclose.
It is not the complete Python math API. Missing operations include hypot,
dist, fsum, sumprod, frexp, ldexp, nextafter, ulp, remainder,
fma, special functions, and newer pi-scaled trigonometry. Accurate summation
and bit-level float decomposition need separate contracts and tests; simple
arithmetic substitutes must not imply Python’s numerical guarantees.
Arbitrary constant expressions and custom iterator protocols remain language
design questions. % remains integer-only; use fmod or
divmod_float for floats. Powers use .pow() or math.pow; ** is rejected
with a diagnostic naming .pow(). Remaining scheduling belongs in the
backlog.
Acknowledgements
- API modelled on Python’s
math, including CPython’s domain and overflow policy.
std.mime
import std.mime parses, builds, formats and compares media types such as
text/html; charset=utf-8, and maps file extensions to a Content-Type.
The module is checked Plenty in stdlib/std/mime.plenty and links no native
code. std.http.files uses its extension table. The
signature index lists the complete API.
import std.mime
from std.mime import MimeError
def main() -> Result[(), AllocError | IoError | MimeError]:
value = std.mime.parse("Text/HTML; Charset=\"UTF-8\"; title=\"a \\\"b\\\"\"")?
print(value.to_string()?)?
print(value.type_name(), value.subtype(), value.param("charset"), value.param("TITLE"))?
print(value.matches(&std.mime.parse("text/*")?), value.matches(&std.mime.parse("image/*")?))?
mut upload = std.mime.new("multipart", "form-data")?
upload.set_param("boundary", "x7 y")?
print(upload.to_string()?)?
print(std.mime.from_file_name("logo.SVG"), std.mime.from_extension("woff2"))?
Ok(())
text/html; charset=utf-8; title="a \"b\""
text html Option[str].Some("utf-8") Option[str].Some("a \"b\"")
True False
multipart/form-data; boundary="x7 y"
image/svg+xml Option[str].Some("font/woff2")
Media types
| Item | Meaning |
|---|---|
parse(text) | Result[MediaType, MimeError]: the media type in text |
new(type_name, subtype) | Result[MediaType, MimeError]: a media type without parameters; both names must be tokens |
value.type_name(), value.subtype() | &str: the lowercase names |
value.params() | &list[Param]: parameters in order; each Param has name and value |
value.param(name) | Option[str]: the value of the parameter name, compared without case |
value.essence() | Result[str, AllocError]: type/subtype without parameters |
value.suffix() | Result[Option[str], AllocError]: the structured syntax suffix after the last +, such as json in application/ld+json |
value.is_wildcard() | bool: the type or subtype is * |
value.matches(&pattern) | bool: value falls within the media range pattern |
value.set_param(name, value) | Result[(), MimeError]: add a parameter, or replace the value of one with the same name |
value.remove_param(name) | bool: remove the parameter name; False when absent |
value.to_string() | Result[str, AllocError]: canonical text |
MimeError has two variants: Invalid, for text that is not a media type
or a name or value that is not allowed, and Allocation(AllocError).
parse follows the media-type rule of
RFC 9110 section 8.3.1:
media-type = type "/" subtype *( OWS ";" OWS [ parameter ] )
parameter = token "=" ( token / quoted-string )
Spaces and tabs are allowed around the whole text and around each ;, and
empty parameters such as a/b;;x=1 are skipped. A quoted-string may hold
tab, space, visible ASCII and non-ASCII text, and \ escapes the next
character; the stored value is unquoted and unescaped. Everything else
returns MimeError.Invalid: a missing or empty type or subtype, a
character outside a token, a parameter without = or value, an unclosed
quote, text after a closing quote, and a parameter name given twice
(compared without case, as
RFC 6838 section 4.3
requires).
Type, subtype and parameter names are case-insensitive, so they are stored
in lowercase. Parameter values keep their case, except charset, whose
values are case-insensitive and are stored in lowercase. == therefore
treats Text/HTML;Charset=UTF-8 and text/html; charset=utf-8 as equal. It
still compares parameters in order, so a/b; x=1; y=2 and
a/b; y=2; x=1 differ.
to_string writes type/subtype, then ; name=value for each parameter in
order. A value that is empty or not a token is written as a quoted-string,
with " and \ escaped. Parsing formatted text gives an equal value.
set_param requires a token name and a value without control characters
other than tab, or returns MimeError.Invalid and leaves the value
unchanged. It lowercases the name, and the value of charset.
Media ranges
value.matches(&pattern) reads pattern as a media range, as in an
Accept header: */* matches every type, text/* every text subtype,
and any other type and subtype must be equal. Each parameter of pattern
must appear in value with the same value; extra parameters of value are
ignored. So text/html; charset=utf-8 matches text/* and
text/html; charset=utf-8, but not text/html; level=1.
std.mime does not parse whole Accept headers: splitting the list and
reading q weights belongs to the HTTP layer.
File extensions
| Item | Meaning |
|---|---|
from_extension(extension) | Option[str]: the Content-Type for an extension given without its dot, or Nothing |
from_file_name(name) | str: the Content-Type for the extension after the last . of name, or application/octet-stream |
Both compare without ASCII case and allocate nothing. A name with no .,
such as html, has no extension; .html has the extension html. The
table is fixed:
| Extension | Type |
|---|---|
html, htm | text/html; charset=utf-8 |
css | text/css; charset=utf-8 |
js, mjs | text/javascript; charset=utf-8 |
json | application/json |
txt | text/plain; charset=utf-8 |
csv | text/csv; charset=utf-8 |
xml | application/xml |
svg | image/svg+xml |
png, jpg, jpeg, gif, webp, ico | image/png, image/jpeg, image/gif, image/webp, image/vnd.microsoft.icon |
wasm | application/wasm |
pdf | application/pdf |
woff2, woff | font/woff2, font/woff |
The result is Content-Type text, ready for a response header; pass it to
parse for a MediaType.
Allocation
parse copies the text once into a byte buffer and allocates the names,
values and parameter list of the result. new, set_param, essence,
suffix and to_string allocate their results. Every allocation failure
returns AllocError and leaves existing values unchanged. to_string checks
its fragment-list capacity before allocating; an unrepresentable capacity
returns AllocError.CapacityOverflow. Accessors,
param, matches, is_wildcard, remove_param and the extension lookups
allocate nothing.
Limits
- No length limit: the caller bounds input, as HTTP header limits do.
- No content sniffing, and no lookup from a media type back to extensions.
- The extension table is the fixed list above, not the full IANA registry.
Acknowledgements
- API shape modelled on Rust’s
mimecrate. - Extension types are registered in the IANA Media Types registry.
std.net
import std.net provides TCP listeners and streams, UDP sockets, IPv4 and IPv6
addresses, and name lookup. Its deadline operations take a
Deadline and a cancellation token and wait through
std.ready. Immediate TCP operations make one nonblocking attempt;
the plain std.io methods wait without a deadline. There is no required event
loop or async syntax. Use deadline operations to keep a worker’s waits bounded.
The runtime supplies one primitive per socket call (opcodes 410–418).
Selectors and cancellable waits come from std.ready (opcodes 419–422).
Addresses, waits, retries and error naming are checked Plenty in
stdlib/std/net.plenty.
import std.bytes
from std.io import StreamError
from std.net import NetError, TcpListener, connect, listen, loopback_v4
from std.sync import CancellationToken
from std.time import Deadline, deadline_after, milliseconds, monotonic_now
def greet(listener: &TcpListener, deadline: &Deadline, cancel: &CancellationToken) -> Result[(), AllocError | NetError | StreamError | std.bytes.BytesError]:
mut stream = listener.accept(deadline, cancel)?
reply = std.bytes.from_text("hello from the server")?
stream.write_all_before(&reply, deadline, cancel)?
stream.close()
def main() -> Result[(), SpawnError[()] | IoError | AllocError | RangeError | NetError | StreamError | std.bytes.BytesError]:
cancel = CancellationToken()?
deadline = deadline_after(&monotonic_now(), &milliseconds(5000))?
listener = listen(&loopback_v4(0u16), 8)?
address = listener.local_address()?
with spawn(greet, &listener, &deadline, &cancel)? as server:
mut stream = connect(&address, &deadline, &cancel)?
mut buffer = std.bytes.with_capacity(64)?
buffer.resize(64, 0u8)?
count = stream.read_before(&mut buffer, 0, 64, &deadline, &cancel)?
print(buffer.to_text_range(0, count)?)?
server.join()?
Ok(())
hello from the server
Addresses
| Item | Meaning |
|---|---|
IpAddress.V4(bits: u32) | An IPv4 address; V4(0x7f000001) is 127.0.0.1 |
IpAddress.V6(high: u64, low: u64) | An IPv6 address as two 64-bit halves in network order |
SocketAddress(ip, port: u16, scope: u32) | An address and port; scope selects a link-local IPv6 interface and is otherwise 0 |
ipv4(a, b, c, d) | IpAddress from four u8 parts |
socket_address(ip, port) | SocketAddress with scope 0 |
loopback_v4(port), loopback_v6(port) | 127.0.0.1 and ::1 |
any_v4(port), any_v6(port) | 0.0.0.0 and ::, which bind every interface of the family |
parse_ip(text) | Result[IpAddress, NetError] |
parse_address(text) | Result[SocketAddress, NetError] from 1.2.3.4:80, [::1]:80 or [fe80::1%2]:80 |
format_ip(ip) | Result[str, NetError] |
address.to_string() | Result[str, NetError], which parse_address reads back |
IpAddress is copyable; SocketAddress is a class and moves. Parsing is
strict and repairs nothing: IPv4 needs four decimal parts of at most three
digits without leading zeros; IPv6 accepts hex groups, at most one ::, and a
dotted IPv4 tail. Ports and scopes are decimal. Anything else is
InvalidAddress. IPv6 text follows RFC 5952: lowercase, no leading zeros, the
first longest run of two or more zero groups written ::, and IPv4-mapped
addresses in dotted form (::ffff:192.0.2.1).
Errors
NetError names the outcomes a caller handles differently:
| Variant | Cause |
|---|---|
TimedOut | The caller’s deadline passed |
ConnectionTimedOut | The kernel gave up on the connection (ETIMEDOUT) |
WouldBlock | An immediate TCP operation cannot make progress now (EAGAIN/EWOULDBLOCK); wait for readiness before trying again |
Cancelled | The token was cancelled |
Refused | Nothing listens at the address (ECONNREFUSED) |
Reset | The peer reset or closed the connection while data was moving (ECONNRESET, EPIPE) |
AddressInUse | Another socket is bound to the address (EADDRINUSE) |
AddressUnavailable | The address is not local, or IPv6 is unavailable (EADDRNOTAVAIL) |
Unreachable | No route to the network or host |
NameNotFound | The resolver found no address for the name |
TryAgain | The resolver failed temporarily (EAI_AGAIN), or 32 lookups are already running |
HandleLimit | The process or system has no free descriptors (EMFILE, ENFILE) |
InvalidAddress | Address text outside the grammar, or an empty or NUL-containing host name |
Stream(StreamError) | Any other failure, with its native code in StreamError.Io(IoError.System(code)), or allocation failure |
TimedOut ends the current wait; it does not mark an established connection
as failed, so the caller may retry with another deadline. ConnectionTimedOut
is a connection failure: close that stream rather than retrying a wait on it.
Both distinctions survive the network-error wrappers in
std.tls and the HTTP client.
TCP
| Function or method | Result |
|---|---|
listen(address: &SocketAddress, backlog: i64) | Result[TcpListener, NetError] |
listener.accept(deadline, cancel) | Result[TcpStream, NetError] |
listener.accept_now() | Result[TcpStream, NetError]: one nonblocking accept attempt |
listener.local_address() | Result[SocketAddress, NetError] |
listener.close() | Result[(), NetError] |
connect(address: &SocketAddress, deadline, cancel) | Result[TcpStream, NetError] |
start_connect(address: &SocketAddress) | Result[Connecting, NetError]: starts a connection without waiting |
connecting.watch(&mut selector, key), unwatch(&mut selector) | Result[(), NetError]: the attempt is ready when it can be finished |
connecting.finish_now() | Result[(), NetError]: WouldBlock while the attempt continues, then success or why it failed |
connecting.stream() | TcpStream: consumes a finished attempt |
descriptor() on TcpStream, TcpListener, UdpSocket, Connecting, Lookup | &File: the handle a selector waits on, for std.ready.Watchable |
stream.read_now(buffer, start, stop) | Result[i64, NetError]: one nonblocking read attempt into &mut Bytes |
stream.write_now(source: &[u8]) | Result[i64, NetError]: one nonblocking write attempt from a &[u8] view |
stream.read_before(buffer, start, stop, deadline, cancel) | Result[i64, NetError]: one read; 0 is end of input |
stream.write_before(source: &[u8], deadline, cancel) | Result[i64, NetError]: offer all of source in one write and return the count accepted; source is a &[u8] view |
stream.write_all_before(source: &[u8], deadline, cancel) | Result[(), NetError] |
stream.read, write, flush | The std.io Reader and Writer methods, returning StreamError on failure; read and write block without a bound |
stream.close() | Result[(), StreamError], as required by std.io.Closer |
stream.shutdown(direction: Shutdown) | Result[(), NetError]; Shutdown.Read, Write or Both |
stream.set_no_delay(enabled: bool) | Result[(), NetError]: send small writes immediately |
stream.acknowledge_now() | Result[(), NetError]: acknowledge data already received at once, not after the kernel’s delay (Linux TCP_QUICKACK); later data is acknowledged as usual |
stream.local_address(), peer_address() | Result[SocketAddress, NetError] |
- Port 0 binds a free port;
local_addressreports it. backlogbounds connections waiting foracceptand is clamped to 1..4096; the kernel may round it. Listeners setSO_REUSEADDR, so a port inTIME_WAITcan be reused, but two listeners on one port areAddressInUse. An IPv6 listener accepts only IPv6 connections.acceptborrows the listener shared, so several workers can accept from one listener; each connection goes to exactly one of them. A connection reset before it was accepted is skipped.- At the descriptor limit,
acceptreturnsHandleLimitand the connection stays queued: once the server closes a descriptor, the nextaccepttakes it. A server can back off or shed idle connections instead of failing. - Linux queues up to
backlog+ 1 unaccepted connections (measured on loopback for backlogs 1, 2, 8 and 64). Further connection attempts are dropped, not refused, soconnectwaits until its deadline and returnsTimedOut. The kernel also capsbacklogatnet.core.somaxconn. connectstarts a non-blocking connection and waits for it within the deadline. A refused or unreachable peer is an error value.- Accepted and connected descriptors stay nonblocking.
read_beforeandwrite_beforewait for readiness, attempt the transfer, and retryEAGAINor interruption against the original deadline and cancellation token. A stale readiness report or a full send buffer cannot turn that transfer into an unbounded kernel wait. Plainreadandwriteretain their blockingstd.iocontract by waiting without a deadline when the socket would block. A read may return fewer bytes than were sent, and a write may accept fewer than offered: loop, or usewrite_all_before. write_beforeoffers all ofsourcewithout a chunk-size cap. The nonblocking write accepts only what currently fits in the kernel’s send buffer; a successful count is progress, not a guarantee that the whole view was sent. Pass&buffer[a:b]to write part of a buffer; view bounds clamp.- Writing to a connection the peer has closed is
Reset, neverSIGPIPE. - On
TimedOutorCancelledthe socket and the caller’s buffer are unchanged, apart from what earlier steps ofwrite_all_beforesent.
Closing a stream
TcpStream.close deliberately returns StreamError: its signature satisfies
the std.io.Closer protocol. Together with Reader and
Writer, this lets a TCP stream work with generic I/O helpers, including
close_buffered, which flushes a BufWriter[TcpStream]
and closes its underlying stream. Changing close to return NetError would
break that protocol contract. Listeners and UDP sockets do not implement
Closer; their close methods return NetError.
A function that uses deadline operations and explicitly closes a TCP stream
can propagate both error types with NetError | StreamError, as greet does
above. Keeping both types preserves the original close failure. Dropping a
stream also releases its descriptor, but cannot report a close error.
Waiting on many sockets
stream.watch(selector, key, interest), listener.watch(selector, key) and
socket.watch(selector, key, interest) register a socket with a
std.ready selector; unwatch(selector) removes it. A
listener is readable while a connection waits to be accepted. After
selector.wait, each Event names a ready socket by its key. Use
accept_now, read_now or write_now to attempt the operation without a
second readiness wait. These methods also work without a prior selector wait.
Each immediate method makes at most one socket operation. It never polls,
waits for readiness, or retries interruption. WouldBlock means that the
accept queue is empty, no input is available, or the send buffer is full.
It remains possible after a readiness report; return to the selector so other
connections can make progress. No bytes move on WouldBlock. An interrupted
attempt returns Stream(StreamError.Interrupted), and an aborted accept
returns the native Stream(StreamError.Io(IoError.System(103))) (ECONNABORTED)
instead of silently taking another connection. The caller decides when to retry.
A successful read or write may transfer only a prefix of the requested range.
Keep the write offset and unsent bytes between events; subscribe to write
readiness only while output remains. Zero from a nonempty read range is EOF,
not WouldBlock. Empty ranges return zero. Ranges are checked before I/O:
0 <= start <= stop <= buffer.len() for reads, or source.len() for writes;
invalid ranges return Stream(StreamError.OutOfRange) and transfer nothing.
Immediate methods take no deadline or cancellation token. Apply the server’s deadline and cancellation policy to its selector waits and between bounded batches of attempts. The deadline-based methods remain useful when one worker can wait for that connection alone.
mut chosen = selector(64).unwrap()
listener.watch(&mut chosen, 0u64)?
# ... accept connections, then stream.watch(&mut chosen, key, Interest.Read)
count = chosen.wait(&deadline, &cancel)?
for index in range(count):
key = chosen.event(index)?.key
The selector tutorial demonstrates reading the reported connection and handling stale readiness.
UDP
| Function or method | Result |
|---|---|
bind_udp(address: &SocketAddress) | Result[UdpSocket, NetError] |
socket.send_to(source: &[u8], target: &SocketAddress, deadline, cancel) | Result[i64, NetError]; source is a &[u8] view |
socket.receive_from(buffer, start, stop, deadline, cancel) | Result[Datagram, NetError] |
socket.set_broadcast(enabled: bool) | Result[(), NetError] |
socket.local_address(), close() | As for TCP |
A Datagram has public fields size (the datagram’s full length), stored
(the bytes written into buffer[start:stop]) and source, and a
truncated() method. A datagram longer than the buffer is truncated, never
split; the rest is discarded. A datagram too large to send is
Stream(StreamError.Io(IoError.System(90))) (EMSGSIZE).
Name lookup
resolve(host: &str, port: u16, deadline: &Deadline, cancel: &CancellationToken)
returns Result[list[SocketAddress], NetError]. An IP literal returns that
address without a lookup. Otherwise the system resolver (getaddrinfo) is
called once and its addresses are returned in its order, each with port. No
address is NameNotFound.
The C resolver cannot be interrupted, so the lookup runs on a helper thread
with a 1 MiB stack, which writes its result to a socket pair. resolve waits
on the reading end within the deadline and the token, so it returns
TimedOut or Cancelled on time even while the resolver is still working.
An abandoned lookup finishes on the resolver’s own timeouts
(/etc/resolv.conf), then discards its result and exits. At most 32 lookups
run at once in a process, counting abandoned ones; another is TryAgain.
start_lookup(host: &str, port: u16) returns Result[Lookup, NetError]: the
same lookup without waiting. lookup.watch(&mut selector, key) reports it
ready when the reply arrives, and lookup.finish_now() returns WouldBlock
until then and the addresses or error afterwards. Dropping an unfinished
Lookup abandons it. std.actor.net
uses start_connect and start_lookup so actors connect and resolve without
blocking their stage.
The resolver’s result list is allocated by the C library and freed on the helper thread. Test builds with allocation checks wait at exit for every helper thread, abandoned ones included, to free its memory.
Effects and ownership
Opening a socket and resolving a name have the Network effect; operations
on an existing socket have the Handle effect. Both are allowed on workers;
see native effects on worker threads.
A listener, stream or socket owns one descriptor, opened close-on-exec, and
closes it when explicitly closed or dropped. Only resolve and start_lookup start a thread,
as above.
Example
The tutorial lesson serves an echo connection on a worker and handles a refused connection.
Address arithmetic is bounded by validated record ranges, fixed-width words
and decimal/hexadecimal digit counts. Resolver reply-buffer growth checks its
next extent before allocation or consuming input; an unrepresentable extent is
Stream(StreamError.Allocation(AllocError.CapacityOverflow)). These internal
checks preserve the public NetError and stream contracts regardless of the
caller’s arithmetic mode.
Limits
- x86_64 Linux GNU only; elsewhere every operation reports
Stream(StreamError.Io(IoError.Unsupported)). - No socket timeouts beyond deadlines, multicast, or socket options other
than no-delay and broadcast. TLS is in
std.tls. - An abandoned name lookup keeps its helper thread until the resolver gives up.
Unix-domain sockets
std.net does not provide Unix-domain sockets, by decision rather than as an
omission. A path socket leaves a file that someone must remove, and a stale
one makes the next bind fail, so it needs a cleanup policy that ownership
of the socket alone does not give. The abstract namespace avoids files but is
Linux-only. Their distinctive features, passing descriptors and peer
credentials, would move ownership of native handles between processes, which
the ownership rules do not yet
describe. Local communication with a child is covered by
std.process pipes, and with another local process by TCP on
loopback. Revisit this when a scheduled use case, such as talking to a
system daemon, needs them.
std.path — native paths
Import std.path. Path owns an affine list[u8], preserving exact Linux path
bytes, including invalid UTF-8, spaces, tabs, CR and LF. It is not a compact byte
buffer or a Windows path abstraction. from_bytes(bytes) transfers the list
without allocating. from_text(&text) explicitly encodes UTF-8 and returns
Result[Path, IoError]. NUL can occur in a lexical Path; filesystem operations
reject it with IoError.InvalidInput.
import std.path as path
def main() -> Result[(), AllocError | IoError]:
base = path.from_text("data/reports")?
child = path.from_text("2024/summary.csv")?
full = base.join(&child)?
print(full.to_string()?, full.is_absolute())?
print(full.file_name()?.ok_or(IoError.InvalidInput)?.to_string()?)?
print(full.parent()?.ok_or(IoError.InvalidInput)?.to_string()?)?
print(len(full.components()?))?
raw = path.from_bytes([110u8, 111u8, 255u8]?)
print(raw.to_string()?, raw.to_text().is_err())?
Ok(())
data/reports/2024/summary.csv False
summary.csv
data/reports/2024
4
no\xff True
| Operation | Result |
|---|---|
path.as_bytes() | Shared borrow of the existing list[u8]; no allocation |
path.to_text() | Checked Result[str, IoError]; never replacement-decodes |
path.to_string() | Escaped ASCII Result[str, IoError] for diagnostics |
path.copy() | Detached Result[Path, AllocError] |
path.join(&child) | Result[Path, IoError]; absolute child replaces the base |
path.is_absolute() | Whether its Linux spelling begins with / |
path.parent(), path.file_name() | Result[Option[Path], IoError] |
path.components() | Result[list[list[u8]], IoError] with lexical components |
Lexical operations use Rust’s Unix Path rules. Components remove redundant
separators and interior . components; an initial relative . is retained.
They preserve .., including across a symlink. Joining preserves the base’s
spelling and relative root; it does not canonicalize, resolve links, test
existence, or change the working directory. A root has no parent or filename;
a has the empty relative path as its parent. All owned output buffers are
fallible. OS ABI details stay private.
Display is explicit: to_string() preserves printable ASCII, escapes backslash
and double quote, and emits other bytes as \xhh. This representation keeps
invalid UTF-8 and control bytes visible without replacement decoding; it is not
an input parser. to_text() validates, while as_bytes() allows exact binary
transport. Automatic record formatting (print(path), str.repr(path)) is
diagnostic list formatting, not a round-trippable filesystem spelling. The
method is to_string() rather than display() because print and f-strings
call a display method only in its
formatter form. Use NUL framing with
binary transport when passing arbitrary Linux paths.
The Rust implementation baseline is library/std/src/path.rs; Python’s
PurePath/Path separation motivates keeping lexical operations distinct from
filesystem observations. Broader buffers and portable path
representations remain B12/B39 candidates; issue #76 owns
compact buffers/views. Closed issue #61 records the initial raw transport,
and issue #49 records the catalogue’s decomposition rather than implementation.
Acknowledgements
- Lexical rules follow Rust’s
std::path::Path; the lexical/filesystem split follows Python’sPurePath/Path.
std.process
import std.process starts child processes from an argument vector and owns
them until they are reaped. Arguments, variables, and directories are
operating-system byte strings; pipes carry std.bytes data.
No shell takes part unless the program is one. An argument reaches the child
exactly as given: spaces, quotes, $, ;, |, *, and newlines are ordinary
bytes. shell is the one, named way to run shell
syntax. std.shlex can split a shell-style argument string before
passing each resulting word to arg; it does not execute or expand shell syntax.
Request encoding, pipe draining, status decoding, and the reaping rules are
checked Plenty. Five native primitives spawn, hand over a pipe end, wait,
signal, and poll. The module is available on the supported x86-64 Linux GNU
target; elsewhere spawn reports IoError.Unsupported.
import std.process
from std.bytes import BytesError
from std.io import StreamError
from std.process import Stdio
def main() -> Result[(), AllocError | IoError | StreamError | BytesError]:
mut echo = std.process.command("echo")?
echo.arg("hello from a child")?
echo.stdout(Stdio.Piped)
mut child = echo.spawn()?
output = child.output(4096)?
print(output.status.success(), output.stdout.to_text()?.strip()?)?
Ok(())
True hello from a child
Building a command
| Function | Result | Contract |
|---|---|---|
command(program: &str) | Result[Command, IoError] | A command with no arguments |
command_bytes(program: &list[u8]) | Result[Command, IoError] | The same for a name that is not UTF-8 |
shell(script: &str) | Result[Command, IoError] | /bin/sh -c script |
A program name containing / is a path. Any other name is looked up in the
directories of the child’s PATH. The name is also the child’s argv[0].
shell passes script to /bin/sh as one argument. The script is shell
syntax, so text placed in it is interpreted: never build one from untrusted
input. Pass such data as an argument or a variable instead.
Command methods take self: &mut Command:
| Method | Result | Effect |
|---|---|---|
arg(value: &str) | Result[(), IoError] | Append one argument |
arg_bytes(value: &list[u8]) | Result[(), IoError] | Append one argument of any bytes |
arg_path(value: &Path) | Result[(), IoError] | Append a path losslessly |
env(name: &str, value: &str) | Result[(), IoError] | Set a variable for the child |
env_bytes(name: &list[u8], value: &list[u8]) | Result[(), IoError] | The same for any bytes |
env_remove(name: &str), env_remove_bytes(name: &list[u8]) | Result[(), IoError] | The child does not inherit it |
env_clear() | () | The child inherits nothing; earlier edits are dropped |
current_dir(path: &Path) | Result[(), IoError] | The child’s working directory |
stdin(mode: Stdio), stdout(mode: Stdio), stderr(mode: Stdio) | () | Choose a stream |
- A child inherits the parent’s environment and working directory unless the command says otherwise. Edits apply in order, so the last one for a name wins. Nothing changes in the parent.
- A piece containing a NUL byte, an empty program or directory, and a variable
name that is empty or contains
=areIoError.InvalidInput. A failed method leaves the command exactly as it was. - An empty argument is valid and is passed as an empty argument.
- Allocation failure is
IoError.Data(DataError.Allocation(...)). Unrepresentable argument, environment and combined request sizes preserveAllocError.CapacityOverflowin that cause, before mutation or starting a child. - Appending a terminated byte string reserves only its new bytes and NUL, not bytes already held. Existing capacity can satisfy the operation without an allocation; failed growth leaves the original command unchanged.
Stdio selects each standard stream:
| Mode | The child’s stream |
|---|---|
Stdio.Inherit | The parent’s own stream. The default |
Stdio.Null | /dev/null |
Stdio.Piped | A new pipe whose other end the Child owns |
Spawning
command.spawn() borrows the command and returns Result[Child, IoError]. A
command is reusable: each call starts another child.
A program that cannot be started is an error value, not a child that exits
with a conventional code: a missing program or directory is
IoError.System(2), a file without execute permission IoError.System(13).
A failed spawn leaves no process and no open pipe.
Pipe ends are close-on-exec, so one child never inherits another child’s pipes. Other descriptors the program opened are close-on-exec as well.
Close-on-exec still leaves a short window: a child forked by one thread holds
a copy of every descriptor open at that moment until its exec closes them. If
another thread had a program open for writing then, running that program just
after closing it fails with “text file busy” (IoError.System(26)). spawn
retries that error after waits of 1, 2, 4, 8, 16, 32 and 64 ms, then reports
it, so a busy program delays spawn by at most 127 ms. Spawns in one program
take turns, and a retrying spawn keeps its turn while it waits, so no new child
can pick up a descriptor meanwhile; other threads’ child operations other than
a blocking wait wait with it. spawn takes no deadline or cancellation
token. Waiting allocates nothing; each attempt repeats the brief host
allocation described under Limits. A program another process keeps
open for writing still fails after the last wait.
Child
A Child is the single owner of one process. It is not copyable.
| Method | Result | Contract |
|---|---|---|
id() | i64 | The process id |
take_stdin(), take_stdout(), take_stderr() | Option[File] | Move a pipe end out, once |
close_stdin() | Result[(), IoError] | End the child’s input |
wait() | Result[ExitStatus, IoError] | Close stdin, block until the child ends, reap it |
try_wait() | Result[Option[ExitStatus], IoError] | Reap if ended; Nothing while running |
wait_timeout(budget: &Duration, cancel: &CancellationToken) | Result[Option[ExitStatus], WaitError] | Nothing when the budget ran out |
signal(number: i32) | Result[(), IoError] | Send signal 1 to 64 |
terminate() | Result[(), IoError] | SIGTERM: a request the child may ignore |
kill() | Result[(), IoError] | SIGKILL: cannot be caught |
communicate(input: &[u8], limit: NonNegative[i64]) | Result[Output, StreamError] | Feed stdin, collect both outputs, wait |
output(limit: NonNegative[i64]) | Result[Output, StreamError] | communicate with no input |
Methods that reap or move a pipe take self: &mut Child; id and the signal
methods borrow.
Pipes
A taken pipe end is an ordinary File: stdin
is write-only, stdout and stderr are read-only, and none can seek. Use
Bytes.read_file and Bytes.write_file directly, or wrap the end
with std.io.from_file for the retry helpers and buffering.
- A read returns the bytes that have arrived, possibly fewer than asked, and 0 at end of input, which is when the child and every process that inherited the stream have closed it.
- A write to a child that no longer reads is
IoError.System(32)(EPIPE). The program is not sentSIGPIPEfor it. - The child sees end of input only when stdin is closed.
waitandcommunicateclose a stdin theChildstill holds; a taken end is the caller’s to close.
Reading one pipe to the end while the child blocks writing another, or writing
a large input while the child blocks writing output, is a deadlock: a pipe
holds 64 KiB. Use communicate, which serves all three in one loop.
Waiting and status
ExitStatus describes how the child ended:
| Method | Result | Contract |
|---|---|---|
success() | bool | Exit code 0 |
code() | Option[i32] | The exit code, 0 to 255; Nothing if a signal ended it |
signal() | Option[i32] | The terminating signal; Nothing if it exited |
- The first successful
wait,try_wait, orwait_timeoutthat sees the child ended reaps it. The status is kept, and later calls return it without asking the operating system. waitblocks without a limit. It does not drain pipes: a child blocked writing to a full pipe that nobody reads never ends.wait_before(deadline: &Deadline, cancel: &CancellationToken)pollstry_waitwith pauses of at most 10 ms.wait_timeoutconstructs one total deadline from a duration and delegates to it. Both leave stdin open and the child running when cancelled or timed out.WaitError.Cancelledonly ends this wait; it never sends a signal.WaitError.Iopreserves operating-system errors; an overflowing budget isWaitError.Io(IoError.InvalidInput).- Signals reach only a child that has not been reaped. Afterwards the methods
return
Ok(())and send nothing, so a number can never reach an unrelated process that received the same id. A number outside 1 to 64 isIoError.InvalidInput. - A signalled child still has to be waited for.
Collecting output
communicate writes input to a piped stdin, appends piped stdout and stderr
to two new buffers as data arrives, closes each stream at its end, then waits.
Output has public fields status: ExitStatus, stdout: Bytes, and
stderr: Bytes.
- Streams that are not piped, or whose end was taken, are skipped and their buffers stay empty.
limitbounds each output separately. A stream that produces more isStreamError.OutOfRange; at mostlimit + 1of its bytes were buffered. The child is then still running and still owned: kill or wait for it.- A child that stops reading ends the feed without an error. Its status says what happened.
- Interrupted calls are retried. Other failures are the
StreamErrorof the failing transfer. - Memory is the two outputs, grown in 8 KiB steps. Input is written in 4 KiB
pieces and is not copied. A next-buffer extent that cannot fit
i64returnsStreamError.Allocation(AllocError.CapacityOverflow)before growing or reading that chunk, including when the extra limit-probe byte is unrepresentable.
Readiness flags and process identifiers use bit masks. Argument counts, status subtraction and pipe cursors use explicit arithmetic opt-outs justified by successful storage growth, validated statuses and concrete file progress. Caller arithmetic modes do not change these boundaries.
Ownership and reaping
Plenty has no user destructors, so dropping a Child closes the pipe ends it
still holds and does nothing else: the process is neither stopped nor reaped.
Its runtime record, and its zombie once it ends, last until the program exits.
A build with runtime allocation checks reports that record as a leak. Every
child should end in a wait, try_wait, or wait_timeout that returns a
status.
A child that outlives the program is adopted by the system as usual.
Limits
- Linux x86-64 GNU only. Starting a child uses the host allocator briefly for its argument vector; that one allocation is outside fallible accounting.
- No redirection of a stream to a file or to another child, and no merging of
stderr into stdout.
shellcan do both. - Timed child waits poll; scheduling and lock contention can delay return beyond the 10 ms polling interval.
- No process groups, sessions, user or group changes, resource limits, or detached children.
communicateholds whole outputs in memory. For streaming, take the pipes and give each its own thread.
Acknowledgements
- API modelled on Rust’s
std::process::Commandand Python’ssubprocess, whosecommunicateit follows.
std.random
import std.random provides deterministic pseudo-random generators that the
caller creates, seeds, owns, and passes explicitly, with one error type,
RandomError.
The module is checked Plenty code with no native primitives. It never reads the operating system, the clock, or any process-wide state: there is no global generator and no automatic seeding. It is available on the supported x86-64 Linux GNU target.
A generator’s output is predictable from its seed, and from a few outputs. Do
not use it for keys, tokens, passwords, or anything an adversary must not
guess; use std.secrets. Collection hashing is unrelated to both
modules.
import std.random
from std.random import RandomError
def main() -> Result[(), AllocError | IoError | RandomError]:
mut rng = std.random.seeded(2026u64)
print(rng.between(1, 7)?, rng.between(1, 7)?, rng.next_bool())?
mut deck = ["A", "B", "C", "D", "E"]?
rng.shuffle(&mut deck)
print(deck)?
print(rng.sample_indices(10, 3)?)?
mut replay = std.random.seeded(2026u64)
print(replay.between(1, 7)?)?
Ok(())
6 5 True
["D", "A", "E", "C", "B"]
[9, 7, 1]
6
Generators
| Function | Result | Contract |
|---|---|---|
seeded(seed: u64) | Rng | Expand the seed into a full state; every seed is valid |
from_state(a: u64, b: u64, c: u64, d: u64) | Result[Rng, RandomError] | Restore a state taken from state(); all zeros is ZeroState |
Rng has private fields, so these two functions are the only way to make one.
To seed from the operating system, pass
std.secrets.random_u64() to seeded and record the seed if the
run must be repeatable.
Methods that draw take self: &mut Rng.
| Method | Result | Contract |
|---|---|---|
next_u64() | u64 | One step; all 64 bits are uniform |
next_u32() | u32 | The high half of one step |
next_bool() | bool | The top bit of one step |
next_f64() | f64 | A multiple of 2^-53 in [0, 1) |
below(bound: u64) | Result[u64, RandomError] | Uniform in [0, bound); zero is EmptyRange |
between(low: i64, high: i64) | Result[i64, RandomError] | Uniform in [low, high); low >= high is EmptyRange |
index(count: i64) | Result[i64, RandomError] | Uniform index into count elements; zero is EmptyRange, negative traps |
shuffle[T](items: &mut list[T]) | () | Uniform permutation in place |
sample_indices(count: i64, amount: i64) | Result[list[i64], AllocError] | amount distinct indices below count, in selection order; a negative count or amount outside 0..=count traps |
fill(target: &mut Bytes, start: i64, stop: i64) | () | Overwrite target[start:stop] |
state() | (u64, u64, u64, u64) | The current state words; takes &Rng |
Ranges are half-open and never clamp. between covers any ordered pair of
i64 bounds, including i64.MIN to i64.MAX. fill follows the
range rule. An argument that no call could accept,
such as a negative count, is a caller bug and
traps with a
message naming the method; an empty range is EmptyRange because it can come
from data, such as an empty list.
below, between, and index are unbiased: values of a step that would make
some results more likely are rejected and another step is drawn. A draw takes
fewer than two steps on average for every bound.
shuffle is Fisher-Yates. It exchanges elements with swap, so it works for
any element type and never copies or drops an element. To pick k of n
elements, call sample_indices(n, k) and index the collection.
A rejected request (EmptyRange or a failed allocation) does not advance the generator. Lists of fewer than two elements and empty fill
ranges also consume no steps.
Algorithm and stability
Rng is xoshiro256** 1.0: 256 bits of state and a period of 2^256 - 1.
seeded fills the state with four successive SplitMix64 outputs, as the
algorithm’s authors recommend.
The sequences are version-stable. For a given seed or state, every method above returns the same values in every Plenty release, on every thread and run. The derived methods are part of that contract:
| Method | Definition |
|---|---|
next_u32 | next_u64() >> 32 |
next_bool | Bit 63 of next_u64() |
next_f64 | (next_u64() >> 11) * 2^-53 |
below(bound) | Draw next_u64() until the value is at least 2^64 mod bound; return it modulo bound |
between(low, high) | low + below(high - low), computed in u64 |
shuffle | For last from len - 1 down to 1: swap last with below(last + 1) |
sample_indices | Over the list 0..count, for position from 0 below amount: swap position with position + below(count - position); return the first amount |
fill | Each step supplies eight bytes, least significant first; a final partial group takes the low bytes of one more step |
A different algorithm would be a new constructor, not a change to seeded.
Errors
pub enum RandomError:
EmptyRange
ZeroState
| Variant | Meaning |
|---|---|
EmptyRange | The requested range holds no value |
ZeroState | from_state was given the all-zero state, which never advances |
Bounded draws validate NonZero[u64] before rejection sampling. Generator
mixing still uses explicit wrapping operations; list/sample/fill cursors use
documented bounds-proven arithmetic opt-outs. Neither changes the version-stable
draw sequence, failure-state guarantees or public error types.
Ownership and threads
A generator is an ordinary owned value of four u64 words stored inline.
Drawing needs an exclusive loan, so two parts of a program cannot interleave
draws from one generator by accident; give each its own, seeded differently.
The module holds no global state, passes the ordinary worker-effect checks,
and a generator can be moved into a worker thread. There is no built-in way to
derive independent streams from one seed: use distinct seeds.
Costs
| Operation | Allocates | Time |
|---|---|---|
seeded, from_state, state | Nothing | Constant |
next_u64, next_u32, next_bool, next_f64, below, between, index | Nothing | Constant; expected constant for the bounded draws |
shuffle | Nothing | Linear, one bounded draw per element |
fill | Nothing | Linear, one checked Bytes.set per byte |
sample_indices(count, amount) | One list of count indices | Linear in count |
sample_indices is proportional to count, not amount. Throughput has not
been measured on an idle machine.
Limits
- One algorithm. No jump or split functions for parallel streams.
- Normal, exponential and floating uniform distributions are provided by
std.stat, using an explicitly suppliedRng. Weighted choice is not provided. - No
f32draws and no inclusive ranges. sample_indicesreturns indices; there is no element-sampling or reservoir form.
Acknowledgements
- Uses the xoshiro256** generator by David Blackman and Sebastiano Vigna, seeded with SplitMix64 by Guy Steele, Doug Lea and Christine Flood.
std.stat
import std.stat supplies descriptive statistics over &list[f64] and
normal, uniform and exponential distributions. It is checked Plenty code;
there is no native statistics dependency or global random state.
import std.random
import std.stat
from std.stat import StatError
def main() -> Result[(), AllocError | IoError | StatError]:
values = [2.0, 4.0, 4.0, 4.0, 5.0, 5.0, 7.0, 9.0]?
print(std.stat.mean(&values)?, std.stat.median(&values)?, std.stat.pstdev(&values)?)?
print(std.stat.quantiles(&values)?)?
scores = std.stat.normal(100.0, 15.0)?
print(scores.cdf(100.0)?, scores.inverse_cdf(0.5)?)?
mut rng = std.random.seeded(42u64)
draw = scores.sample(&mut rng)?
print(draw > 0.0 and draw < 200.0)?
Ok(())
5.0 4.5 2.0
[4.0, 4.5, 5.5]
0.5 100.0
True
Observations
All observations must be finite. Empty data returns StatError.Empty, and
NaN or infinity returns NonFinite. Convert integer observations explicitly
to f64; integers beyond 2^53 may lose precision during conversion.
| Function | Meaning | Allocation |
|---|---|---|
mean(values) | Arithmetic mean | None |
pvariance(values), pstdev(values) | Population variance/deviation, denominator n | None |
variance(values), stdev(values) | Sample variance/deviation, denominator n-1; needs two observations | None |
median(values) | Middle value, or midpoint of the middle pair | Copy and sort |
quantile(values, p) | Inclusive linear interpolation, 0 <= p <= 1 | Copy and sort |
quantiles(values, n: Positive[i64] = 4) | The n-1 cuts at i/n | Copy/sort plus output list |
Quantiles use Hyndman–Fan type 7: interpolate at sorted index p*(length-1).
This corresponds to Python’s statistics.quantiles(method="inclusive"),
not its default exclusive method. A singleton yields its one value at
every probability. n=1 produces an empty list after validating the data.
The original list is never mutated. Sorting takes O(n log n) time and O(n)
additional space; the other functions take O(n) time and constant space.
Mean uses scaled, compensated summation. Deviation uses a corrected two-pass
calculation with scaled differences. These avoid many overflow/cancellation
problems of summing raw values or squares, but are still approximate f64
calculations, not exact arithmetic. Underflow may round to zero. An
unrepresentable result is Overflow; deviation can succeed when variance
overflows. A one-element sample returns InsufficientData.
Distributions
Constructors validate parameters and return Result[Distribution, StatError]:
| Constructor | Parameters |
|---|---|
normal(mean=0.0, sigma=1.0) | Finite mean and strictly positive finite standard deviation |
uniform(low=0.0, high=1.0) | Finite ordered bounds; width and reciprocal width must be representable |
exponential(rate=1.0) | Strictly positive finite rate |
All three provide pdf(x), cdf(x), and inverse_cdf(p), returning
Result[f64, StatError]. NaN observations are rejected; infinite observations
have their usual limiting densities and probabilities. Probabilities must be
finite and in [0,1]. Normal inverse endpoints are negative/positive infinity;
exponential’s upper endpoint is infinity. Uniform inverse endpoints are its
bounds, and its density includes both bounds (irrelevant to integrals).
Parameters are private and cannot be changed after construction.
Normal tails use a central power series and Laplace’s continued fraction for
the Mills ratio; inversion bisects the smaller tail. Regression checks cover
central values, tails through probability 1e-100 and inverse round trips with
a 2e-13 relative tolerance. This is a tested tolerance, not a correctly-rounded
guarantee for every f64. Upper-tail CDFs eventually round to one; lower tails
underflow near 39 standard deviations. Inversion uses 64 iterations of the
bounded tail evaluation and is intended for ordinary statistical use, not
high-throughput vector analysis. Exponential uses expm1/log1p to retain
small-probability precision. Finite-result arithmetic overflow is reported.
Sampling and reproducibility
Pass a mutable std.random.Rng to sample. Uniform returns
f64; normal and exponential return Result[f64, StatError] because the
transformed value can overflow. An error leaves the generator unchanged.
All constructors, queries and sampling operations allocate nothing.
| Distribution | Draw policy |
|---|---|
| Uniform | One next_f64() draw u; interpolate between bounds |
| Exponential | One draw u; -log1p(-u)/rate |
| Normal | Two draws u,v; Box–Muller mean + sigma*sqrt(-2*log1p(-u))*cos(tau*v); no cached partner |
The draw counts and these transformations are version-stable; adding
distributions does not change any std.random sequence. Transcendental
functions follow the platform math implementation, so samples are reproducible
within the same math environment, not promised bit-identical across platforms
or libm updates. Rounding can put a uniform sample exactly at the upper bound.
No sampling function reads the clock, OS entropy, or a global generator.
Errors and scope
StatError has Empty, InsufficientData, NonFinite, Domain, Overflow
and Allocation(AllocError). Invalid distribution parameters and probabilities
are Domain; failures to allocate sorted/output lists
retain the original allocation cause, including CapacityOverflow for an
unrepresentable quantile output allocation. Validated list lengths, positive
quantile counts and fixed iteration budgets bound internal integer arithmetic;
no primitive arithmetic errors are added to the public contracts. These
operations have no hidden retry.
This module does not provide hypothesis tests, regression, weighted statistics,
discrete distributions, arbitrary precision or dataframe aggregation. Use
std.dataframe for bulk analytical queries.
Algorithm references: Python statistics, statrs distributions, and Lee’s analysis of the Laplace fraction. See the statistics tutorial for examples.
Acknowledgements
- API modelled on Python’s
statistics; distributions follow statrs. - Quantiles use type 7 of Hyndman and Fan (1996).
- Normal tails use Laplace’s continued fraction, following Lee’s analysis.
std.plot
std.plot draws layered charts as deterministic SVG files or byte streams.
It is implemented in Plenty and needs no plotting runtime, R installation,
display server, or external shared library. Import it with import std.plot as gg.
This is the core/SVG portion of the plotting proposal, not the complete first milestone or a drop-in replacement for ggplot2. See the plotting tutorial for runnable examples.
import std.plot as gg
from std.plot import Column, PlotError
def main() -> Result[(), AllocError | IoError | PlotError]:
data = gg.table([
("day", Column.Num([1.0, 2.0, 3.0, 4.0]?)),
("visits", Column.Num([120.0, 150.0, 90.0, 180.0]?)),
]?)?
chart = gg.ggplot(data, gg.aes("day", "visits"), [
gg.geom_line(),
gg.geom_point(),
gg.labs(title="Daily visits"),
]?)?
notes = chart.save("visits.svg", width=6.0, height=4.0)?
print(notes)?
svg = read_text("visits.svg")?
print(svg.startswith("<svg"), "Daily visits" in svg)?
Ok(())
[]
True True
Data and ownership
table(list[(str, Column)]) consumes named columns without copying their lists.
Columns have equal lengths and unique nonempty names; an empty table has height
zero.
Column.Num, Int, Text, and Bool contain lists of f64, i64, str,
and bool. MaybeNum and MaybeText contain lists of Option[f64] and
Option[str]. Table.height() and Table.names() inspect the table.
Table.share() gives another immutable owner of the same columns without
allocating. ggplot consumes a table. Plot.plus(components) borrows the
original plot and returns an independent specification, sharing its table and
copying its mappings and components. Both plots remain usable.
columns(&dictionary) copies a dict[str, list[f64]], retaining its key order.
rows(&records, fields) evaluates typed accessors and builds columns. Construct
fields with num, integer, text, and flag; each takes a name and an owned
callback such as def(row: &Reading) -> f64: row.temperature. These constructors
allocate a boxed callback. rows consumes the field list and retains no callbacks
or borrowed records. This also supports lists of tuples and derived columns.
Mappings and components
aes(x, y, colour=, color=, fill=, group=, ymin=, ymax=) maps column
names, not expressions.
colour and color are aliases; supplying both nonempty names is an error.
An empty name leaves that mapping unset. Numeric columns are continuous; text
and bool columns are discrete. factor("name") makes a numeric column discrete.
Numeric factors sort by numeric value; other levels sort by their string value.
Int positions and continuous colours convert to f64, with its usual precision
limit; factor labels preserve exact integers.
ggplot(table, mapping, components) eagerly checks names, required aesthetics,
column kinds, and component settings. Constants belong in a geom, such as
geom_point(colour="red"); a colour
name inside aes would instead be looked up as a column.
Every geom accepts data=, mapping=, and inherit_aes=True. Omitted data
uses the plot table; supplied data is an owned table and can have different
columns and a different row count. A layer mapping overrides only its nonempty
aesthetics; other aesthetics inherit the plot mapping. Use
inherit_aes=False to start with an empty mapping. Empty names do not clear
individual inherited aesthetics. Inherited names are looked up in the layer’s
table, so a summary table without the plot’s grouping column normally needs
inherit_aes=False. Only effective layer mappings are validated.
Position and colour scales train across all layers, so a category has the same
position and colour wherever it appears. Mixing discrete and continuous
mappings on one scale is an error. plus preserves the original specification
and shares the immutable plot and layer tables.
| Constructor | Behavior |
|---|---|
geom_point | Points; requires x and y; size is a diameter in millimetres |
geom_line | Connects points sorted by x within each group |
geom_path | Connects points in input order within each group |
geom_col | Bars with supplied continuous y heights |
geom_bar | Counts rows at each x, per group; omit y |
geom_histogram | Counts continuous x in equal-width bins; omit y |
geom_errorbar | Supplied ymin/ymax intervals with caps |
geom_linerange | Supplied ymin/ymax intervals without caps |
geom_ribbon | Filled band between ymin and ymax, sorted by x within groups |
See uncertainty and intervals for bound validation, missing-value gaps, grouping, and transformations. These geoms draw supplied bounds; they do not estimate confidence intervals.
Each geom accepts colour/color, alpha, and na_rm; bars also accept fill
and position. Lines and interval geoms use linewidth; ribbons also accept
fill. Numeric settings accept integer or float
arguments. Supported constant colours are black, white, red, green, blue, grey,
gray, orange, purple, yellow, pink, brown, navy, transparent, and #RRGGBB or
#RRGGBBAA. More R colour names are not yet supported.
Discrete colour/fill mappings default to an eight-colour palette, repeated for
additional levels. Continuous mappings default to an RGB gradient from #132B43
to #56B1F7. Legends are automatic. This is not yet ggplot2’s variable-size HCL
palette or its continuous colour-bar guide.
Manual palettes and gradients
scale_colour_manual(values) and scale_fill_manual(values) consume a
dict[str, str] mapping level labels to colours. For example,
gg.scale_fill_manual({"North": "navy", "South": "#D55E00"}?) keeps each
region’s colour stable when rows are reordered or a region is absent.
Dictionary order does not change the sorted legend order. Every observed level
must have an entry; extra entries are allowed. Numeric categories need factor
and keys matching their displayed labels. Invalid colours, missing entries, and
continuous mappings return errors from ggplot or plus.
scale_colour_gradient(low="#132B43", high="#56B1F7") and
scale_fill_gradient(...) set continuous endpoints. Both endpoints must be
six-digit #RRGGBB colours, with either hex letter case. Interpolation is in
encoded RGB channels, truncating to integer channels; it is not ggplot2’s
Lab interpolation. The finite data minimum and maximum map to the endpoints;
a constant column maps to the midpoint. Legend swatches use the same mapping.
Discrete data with a gradient returns WrongKind.
The scale_color_manual and scale_color_gradient spellings are aliases.
The last scale component for each aesthetic wins, regardless of its position
relative to geoms. plus can replace a scale without changing the original
plot. Colour and fill are independent; different palettes get separate legends
even when they map the same column. Constant geom colours override mappings.
Missing values are still omitted, rather than given a palette colour. Scale
limits, custom breaks, na_value, and palette functions are not yet exposed.
Default groups follow the interaction of discrete mappings; an explicit group
replaces this rule. Bar positions use colour/fill groups independently of x.
position_identity, position_stack, position_fill, and position_dodge
select unadjusted, stacked, proportion, and side-by-side bars. Positive and
negative stacks accumulate separately. Stacks follow reverse group order,
matching the top-to-bottom discrete legend. Dodge reserves one slot for every
group, including when a group is absent at a particular x. Bar width is in
data units (default 0.9); automatic numeric resolution-based widths are deferred.
Histogram bins defaults to 30 and may be 1–10,000. Bins span the finite x range;
they are left-closed/right-open, except that the final bin includes the maximum.
An identical-value input uses a unit-width bin centred on that value. Empty bins
do not produce rectangles. This deliberately small binning API does not yet
implement ggplot2’s boundary/centre/binwidth alignment or computed statistics.
Count/bin layers require discrete colour/fill mappings. If an explicit group
combines different mapped colours in one output bar, rendering reports a conflict
instead of selecting one arbitrarily.
Rows with missing or non-finite mapped values are omitted and counted in returned
notes; na_rm=True suppresses the note. These observation-only diagnostic
counts saturate at i64.MAX; semantic row counts, groups, and numerical
aggregates do not saturate. A missing y interrupts a line/path
within its group. Numeric data magnitudes above 1e100 are rejected during construction.
Labels, coordinates, and themes
facet_wrap and facet_grid split layers into panels, with fixed or free
position scales and shared colour legends. See facets for
panel ordering, independent statistics and repeated reference layers.
labs(title=, x=, y=, colour=, fill=) overrides nonempty labels; default axis and
legend labels come from column names. Later nonempty label settings win.
theme_grey, theme_minimal, and theme_classic select grey-panel, white-grid,
and plain-axis styles. They accept base_size in points and legend_position
(Legend.Right or Legend.Hidden). A later theme replaces the earlier theme.
These are compact styles, not the proposal’s complete theme-element tree.
coord_flip() exchanges the displayed axes. coord_cartesian(xlim=(low, high), ylim=(low, high)) clips and zooms after counting and stacking, preserving
statistics. Omit either pair to use its automatic range. Only one coordinate
component is allowed. Coordinate limits are expressed in original data units,
including on transformed axes. See position scales for
transforms, category order, breaks, labels, and the distinction between scale
limits and coordinate zoom.
Layout plans text boxes before writing any SVG. Margins adapt to axis labels,
titles and a right-hand legend. Titles, tick labels and legend labels wrap at
word boundaries; an overlong word splits between Unicode scalars. Explicit
newlines start new lines, and other whitespace is normalised. Tick boxes must
not overlap. A chart that cannot fit returns InvalidSetting before writing
to the supplied writer; increase dimensions, shorten labels, use fewer breaks,
or hide the legend. No ticks or legend entries are silently removed.
SVG text names Liberation Sans, Arial, Helvetica, sans-serif; no fonts or
font metrics ship. Deterministic character advances define each text box, and
SVG textLength with lengthAdjust="spacingAndGlyphs" constrains the displayed
width to that box. This may stretch/compress glyphs with a fallback font.
The guarantee concerns the planned boxes in conforming SVG viewers, not
identical glyph outlines across fonts. Complex-script shaping, font-specific
vertical metrics, rotated ticks, and multi-column legends remain unsupported.
Output and failures
plot.save("chart.svg", width=6.0, height=4.0, units=Units.In, dpi=300.0)
writes SVG and returns Result[list[str], PlotError]. write_svg(&mut writer, ...)
accepts any std.io.Writer and also returns notes. Both borrow the plot,
so it can be rendered repeatedly. Units.Cm, Mm, and Px are also supported;
dpi converts pixel dimensions to physical dimensions and never rasterises SVG.
Dimensions must be between 2 and 100 inches on each axis.
Rendering prepares per-layer rows, writes SVG fragments directly to the writer,
and flushes it. save uses a 16 KiB file buffer. It does not collect the complete
SVG unless the caller selects a memory cursor. Output uses two-decimal coordinates, stable element order and
identifiers, and no timestamps or machine-specific font discovery. Identical
inputs with the same toolchain produce identical bytes.
save truncates an existing destination, including before layout preflight.
A rendering, allocation, or I/O failure
can leave partial output; it is not an atomic file replacement. For transactional
publication, write a temporary file and rename it after success. A stream failure
is PlotError.Write(completed, cause), where completed counts bytes in the
failing fragment, not the entire plot. Flush/close failures use Io.
Other errors are UnknownColumn, DuplicateColumn, ShapeMismatch,
MissingAesthetic, WrongKind, Conflict, InvalidSetting, Allocation,
and Bytes. No library messages are printed. ggplot catches mapping
and setting mistakes; rendering can additionally reject unsupported numeric
magnification, XML-forbidden text, or insufficient layout space. Allocation
failure is always returned; no infallible native plotting allocator is involved.
Current scope
Not yet implemented: after_stat, statistical
smoothing, date scales and other colour scales, full themes and font
metrics, PNG/PDF, dataframe input, or record reflection. These remain part of
the accepted proposal; priorities are tracked in B80.
Acknowledgements
- ggplot2 supplies the model for the public names, layered composition, aesthetic mappings, grouping, and stacked-bar semantics. Its source was consulted during implementation; no ggplot2 code or data ships.
- Hadley Wickham’s A Layered Grammar of Graphics describes the data/mapping/layer/scale/coordinate separation used by the module.
Plot position scales
std.plot trains shared axes across every layer’s effective data
and mapping. The last scale component for each axis wins, including components
added with plus. Scale names supply axis titles; labs overrides them.
import std.bytes
import std.io
import std.plot as gg
from std.bytes import BytesError
from std.plot import Column, PlotError
def main() -> Result[(), AllocError | IoError | PlotError | BytesError]:
data = gg.table([
("requests", Column.Num([10.0, 100.0, 1000.0, 10000.0]?)),
("latency", Column.Num([2.0, 3.0, 5.0, 9.0]?)),
]?)?
chart = gg.ggplot(data, gg.aes("requests", "latency"), [
gg.geom_point(),
gg.scale_x_log10(
name="Requests per second",
breaks=Some([10.0, 1000.0]?),
labels=Some(["10", "1k"]?),
),
gg.scale_y_continuous(name="Latency (ms)", limits=(0.0, 10.0)),
]?)?
mut output = std.io.cursor(std.bytes.empty()?)
drop(chart.write_svg(&mut output)?)
svg = output.data.to_text()?
print(">1k<" in svg, ">100<" in svg, "Latency (ms)" in svg)?
Ok(())
True False True
Continuous axes
scale_x_continuous and scale_y_continuous accept name, breaks
(Option[list[f64]]), labels (Option[list[str]]), limits=(low, high), and
transform. Transform.Identity, Log10, Sqrt, and Reverse are
supported. The scale_{x,y}_log10, _sqrt, and _reverse helpers select
those transforms directly.
Breaks and limits use original data units; ticks are positioned using the
transform and labelled in original units. Explicit labels require explicit
breaks of equal length. Breaks must be distinct, finite and in the transform
domain. Omit breaks (or pass Nothing) for automatic ticks; use
breaks=Some([1.0, 10.0, 100.0]?) for specific values and
breaks=Some([]?) to hide ticks. Supply labels as Some([... ]?).
Limits must be finite, increasing and
inside the transform domain. Without breaks, ticks use 1/2/5 steps in
transformed space and automatic ranges have padding.
Explicit labels allow units, percentages, or date strings. Automatic calendar breaks, date columns and formatting callbacks are not implemented.
Discrete axes
scale_x_discrete and scale_y_discrete accept name, limits (an
optional ordered list of distinct level strings), and labels. Pass, for
example, limits=Some(["South", "North"]?). Limits set category
order, retain named unused categories, and omit rows in unlisted categories.
Labels must match the resulting level list. Without limits, levels are sorted
as described under mappings; numeric factors
keep numeric ordering.
Limits and transformations affect statistics
Continuous scale limits omit out-of-range input values before statistics.
Values outside a transform’s domain (for example zero on a log axis) are
omitted too, with removal notes unless na_rm=True. Missing/omitted values
interrupt lines. Discrete limits similarly exclude unlisted categories.
Histograms bin transformed x values. Supplied bar heights are transformed before stacking; computed counts are transformed before position adjustment. Consequently log-transformed stacks sum logarithms, not original heights; use unstacked bars when that is the intended interpretation. Bar baselines are zero in transformed space (one in original units for log10).
coord_cartesian limits only zoom after these computations and use original
data units even with transformed scales. Use coordinate limits when statistics
must still include values outside the view. coord_flip exchanges the displayed
axes after applying their scales; x scale settings still describe the x mapping.
Plot facets
std.plot can split a plot into labelled panels. Facets use column
names, not formulas.
facet_wrap("site", ncol=2)wraps one facet variable across a rectangular layout.nrowandncoldefault to zero (automatic). If both are supplied, their product must accommodate every panel.facet_grid(rows="region", cols="period")uses one variable per dimension. Either dimension can be omitted. Every row/column combination is retained, including combinations with no observations.
Levels sort using the mapping rules: text lexically, numbers numerically.
Missing/non-finite facet values form a separate panel labelled (missing);
they are not discarded as missing aesthetics. A literal string "(missing)"
remains a distinct category. Headings show the variable name and its value.
The last facet component wins, including when added with plus.
Layers and statistics
Facet columns are read from each layer’s effective table, independently of
aes. At least one layer must contain all requested facet variables.
A layer missing a facet variable repeats across its levels. This makes a
reference line without a site column appear in every site panel. A partially
specified grid layer repeats only across the missing dimension.
Counts, histogram bins and bar positions are computed separately per panel. Colour/fill scales train over all layers and panels and use one shared legend. Position scales are trained before statistics; their final ranges include panel-specific counts, stacks and interval bounds.
Fixed and free axes
Both constructors accept scales=Scales.Fixed (the default), Free,
FreeX, or FreeY.
For wrapped facets, free axes train independently in each panel. In a grid, x scales are shared within each column and y scales within each row, even when free. This retains grid alignment. Explicit scale or coordinate limits still apply to every panel. Discrete scale limits retain their explicit order. Fixed histogram x scales use common bin boundaries; free x scales train their bin ranges independently within each scale group.
coord_flip exchanges displayed axes after scale training. FreeX always
refers to the x mapping, so its displayed direction changes under flipping.
Layout and limits
All panel layouts are preflighted before any SVG is written. Panel sizes are equal, with common margins so plotting areas align. Axis titles and ticks repeat in each panel. The plot title and legend appear once. Increase output dimensions when labels or panels cannot fit; panels are not silently omitted.
There is a maximum of 1,000 panels. Multiple variables in one wrap/dimension, custom labellers, explicit facet ordering, proportional panel space, margins panels, and interior-axis suppression are not implemented. To choose an order, use a numeric grouping column and provide its meaning in the surrounding report.
Rendering materialises panel tables with fallible allocations and shares text values; selections containing every row share the original table. Broadcast layers still repeat their prepared geometry. Memory and preparation time therefore grow with the number of panels and repeated observations; this is not a lazy dataframe view. SVG fragments are still streamed.
Plot uncertainty and intervals
std.plot draws caller-supplied lower and upper bounds:
geom_errorbardraws a vertical interval with end caps.geom_linerangedraws the interval without caps.geom_ribbonfills the area between bounds, sorted by x within each group.
Map bounds with aes(x="time", ymin="lower", ymax="upper"). A y mapping is
optional and ignored by these geoms, so they can inherit a centre-value mapping
shared with points or a line. All three accept layer data/mapping overrides,
colour, alpha, linewidth and na_rm. Ribbons also accept fill.
They do not compute statistical estimates, errors or confidence levels.
Bounds and transformations
Both bounds must be continuous columns. Finite ymin > ymax is an error at
construction. Equal bounds are valid. Both endpoints contribute to axis
training, including in facets.
Position transforms and limits apply to each bound. A row with a missing,
non-finite, out-of-domain or scale-excluded endpoint is omitted, with a note
unless na_rm=True. Use coordinate limits to clip a visible interval without
removing its data. Reversed axes and coord_flip preserve the interval.
Linewidth is in millimetres. Error-bar width is the total cap width in
transformed x units (category spacing for discrete x); it defaults to 0.2
and currently must be in (0, 1]. On a log x axis it therefore describes a
distance in log space. To display horizontal intervals, use coord_flip.
Independent xmin/xmax mappings are not implemented.
Ribbon grouping and gaps
Ribbons require continuous x. Discrete colour/fill mappings or an explicit group define separate bands. Colour and fill must be constant within each continuous segment of a group; differing styles produce a rendering error instead of arbitrarily choosing one. Duplicate x values within a segment also produce an error. Use distinct x values for a well-defined band.
Missing bounds at a finite x split a ribbon into separate polygons. A segment needs at least two valid rows to form a polygon. Coordinates are joined by straight edges; no smoothing, interpolation between groups, or statistical fitting occurs. Ribbons use identity positioning, not stacking.
std.linalg — dense linear algebra
std.linalg provides finite f64 matrices, column vectors, multiplication,
linear solves and decompositions. Storage and the public API are written in
Plenty; the substantial numerical algorithms use faer 0.24.4 through an
optional shared engine. Constructing, inspecting and
transposing, adding, subtracting and scaling matrices works without installing
that engine.
import std.linalg as la
def main() -> Result[(), AllocError | IoError | la.LinalgError]:
a = la.matrix(2, 2, &[2.0, 1.0, 1.0, 1.0]?)?
b = la.vector(&[5.0, 3.0]?)?
x = a.solve(&b)?
print(x.to_list()?)?
gram = (a.transpose()? * a)?
print(gram.rows(), gram.cols(), gram.to_list()?)?
print((2.0 * a)?.get(0, 0)?)?
Ok(())
[2.0, 1.0]
2 2 [5.0, 3.0, 3.0, 2.0]
4.0
Matrices and vectors
matrix(rows, cols, values) copies a flat row-major list. Both dimensions
must be positive and the list must have exactly rows * cols entries.
vector(values) creates an n × 1 matrix. zeros(rows, cols), identity(n)
and diagonal(values) create common matrices. Inputs must be finite.
Matrix stores its values privately in column-major order. rows() and
cols() return dimensions; get(row, col) and mutable set(row, col, value)
use zero-based indices and reject negative or out-of-range indices. to_list()
returns a row-major copy. transpose() and copy(matrix) allocate independent
copies. Neither returns a borrowed view. Matrices have ordinary Plenty ownership
and can be passed, returned, or stored in other classes.
Arithmetic
Arithmetic borrows its operands and returns a fresh matrix in
Result[Matrix, LinalgError]. Use (a + b)?: parentheses put
error propagation around the complete operation. Vectors follow the same rules
because they are one-column matrices.
| Expression | Named equivalent | Meaning |
|---|---|---|
(a + b)? | a.add(&b)? | Elementwise addition; identical shapes required |
(a - b)? | a.subtract(&b)? | Elementwise subtraction; identical shapes required |
(-a)? | a.negate()? | Negate every entry |
(a * b)? | a.matmul(&b)? | Matrix product, not elementwise multiplication |
(2.0 * a)? | a.scale(2.0)? | Multiply every entry by a finite f64 scalar |
Scalar multiplication uses scalar-left syntax. a * 2.0 is unsupported:
the current language permits one __mul__ signature, which Matrix reserves
for matrix products. Use a.scale(2.0)? when scalar-right order reads better.
There is no broadcasting, elementwise multiplication, scalar addition, or
matrix division. Nonfinite scale factors return NonFinite; nonfinite computed
entries return Overflow.
No in-place arithmetic hooks are provided: these operations can fail, whereas
+= and similar hooks must return unit. Explicitly assign a = (a + b)? or
call a named method. Each operator allocates its own result; chains are not
fused and intermediate results consume storage. Dispatch is a direct borrowed
method call with no dispatch allocation or implicit operand copy, but the
result-producing wrappers are not promised to inline.
Products, solves and factors
| Operation | Result and contract |
|---|---|
a.matmul(&b) | Matrix product; a.cols() == b.rows() |
a.solve(&b) | Solve square A X = B with partial-pivot LU; multiple RHS columns supported |
a.lu() | Reusable Lu; rectangular matrices supported |
factor.solve(&b) | Reuse a square LU factorization |
factor.lower() / upper() | Explicit m × k unit-lower and k × n upper factors; k = min(m,n) |
factor.permutation() | Row indices p such that A[p, :] = L U |
a.qr() | (q, r), full m × m orthogonal Q and m × n upper-trapezoidal R; A = Q R |
a.cholesky() | Lower L with A = L Lᵀ; requires symmetric positive-definite A |
a.svd() | (u, values, v), thin factors with k columns; A = U diag(values) Vᵀ |
a.eigh() | (values, vectors) for symmetric A; eigenvectors are columns and values ascend |
a.eig() | (real, imag, vectors_real, vectors_imag) for any square real A; right eigenvectors are columns |
SVD singular values are nonnegative and descending. General eigenvalues have
no promised order. eig() represents complex results with separate real and
imaginary parts; it does not require a complex-number scalar type. Decomposition
signs/phases, eigenvalue order in eig(), and bases in repeated eigenspaces are
not stable identifiers. Compare residuals rather than exact factor entries.
Every operation preserves its inputs, including on failure. Results own their
storage. solve is preferable to forming an explicit inverse; factor once when
solving repeatedly. There are no implicit broadcasting,
indexing overloads, sparse matrices, empty matrices, or GPU operations.
Errors and numerical limits
Operations that can fail only to allocate, such as to_list and transpose,
return AllocError; the others report it as LinalgError.Allocation(cause).
Plenty matrix allocations and native scratch allocations are
fallible. Errors carry no dynamically allocated diagnostic message.
LinalgError distinguishes Shape, Bounds, NonFinite, Singular,
NotSymmetric, NotPositiveDefinite, NoConvergence, Overflow,
NativePanic, and Allocation(AllocError). Nonpositive dimensions and products
above i64.MAX // 16 return Shape before allocation; these checks do not
expose primitive arithmetic errors. Internal index arithmetic is bounded by
that validated extent. Invalid indices return Bounds before reading or
mutating storage.
Constructors/setters reject NaN and infinity; numerical results containing
NaN or infinity report Overflow. Symmetry is checked with exact f64 equality
between the two triangles; symmetrize deliberately if your data is approximate.
LU reports Singular on an exactly zero diagonal pivot at solve time. Near
singularity is not reliably diagnosed: a finite result can be inaccurate.
No automatic regularization, rank tolerance, conditioning estimate, least-squares
solver or pseudoinverse is provided. General eigenvectors of a defective matrix
need not form a basis. Scaling and residual checks remain the caller’s numerical
responsibility. Results may differ in their final bits across CPUs/releases.
Why Plenty uses a faer fork
Plenty uses cjrh/faer-rs’s caller-workspace branch,
based on faer 0.24.4 at 0ecd846.
Upstream’s dense, triangular and indexed GEMM dispatches can allocate packing
buffers infallibly inside the private-gemm-x86 backend. A failure can abort
the process before Plenty can return AllocError, even when matrix outputs
and decomposition scratch were allocated fallibly.
The workspace API change
adds fallible caller-owned packing buffers and a scoped borrow used by all three
dispatches. It avoids initializing the backend’s owning thread-local cache,
uses allocation-free cache discovery, and restores nested scopes on unwinding.
On CPUs without the optimized x86 path, scoped calls use allocation-free
nano-gemm. Plenty enters the scope with Par::Seq; the fork does not make every
upstream owned/parallel API fallible. Assembly kernels and numerical algorithms
are unchanged. The fork includes the small backend as a path dependency because
part of the change belongs in that separate upstream crate.
Plenty separately reuses these workspaces in a fixed four-slot pool. That performance optimization is in the engine, not a second fork kernel variant. It avoids paying for a large reservation on every small operation while keeping allocation failure recoverable on cold/concurrent misses. See the engine contract for its bounded retained-memory cost. The earlier approach of disabling optimized GEMM is no longer used.
Measured on the development x86-64 machine, correctness-checked warm matrix products retained approximately upstream throughput. Seven-sample medians for 16-byte-strided inputs were:
| Square matrix | Fork with workspace reuse | Unmodified faer | Earlier disabled-GEMM workaround |
|---|---|---|---|
| 32 × 32 | 0.00482 ms | 0.00473 ms | 0.00483 ms |
| 128 × 128 | 0.0947 ms | 0.0942 ms | 0.136 ms |
| 512 × 512 | 4.43 ms | 4.59 ms | 12.12 ms |
These are machine-specific measurements, not a promised speedup over upstream.
The same release compiler, baseline x86-64 target, wrapper and CPU affinity
were used; matrix creation was outside timing and workspace acquisition was
inside. A separate matched 700-product run at 512 × 512 counted 29.51 billion
user instructions for both fork and upstream, versus 92.04 billion for the
workaround (including identical benchmark setup and validation). This supports
restoring the optimized path rather than attributing the improvement solely to
CPU timing noise. Cold workspace allocation is outside these warm measurements.
The reproducible driver is plenty-linalg/bench.py.
The branch is selected in plenty-linalg/Cargo.toml; Cargo.lock pins the exact
revision, including the backend (currently 8e1e786). The fork’s design/provenance note
keeps the source import separate from the API change for upstream review.
The intent is to return to upstream releases when equivalent workspace support
is available. Dependency updates must recheck allocation denial, numerical
residuals, concurrent calls and the non-AVX fallback before changing the pin.
See the linear algebra lesson for runnable solve and factorization examples, and the generated signatures for every declaration.
Install and ship the linear algebra engine
Native operations in std.linalg require
libplenty_linalg_v1.so. Linux release archives
include it in bin/ beside the compiler, with bin/linalg-notices/ holding
engine.json and dependency notices. The compiler finds it there without
configuration; set PLENTY_LINALG_ENGINE to an absolute path to select a
different library from the same release. An embedded compiler can set
CompileOptions.linalg_engine, which takes precedence. Type checking and
unreachable native calls need no engine. Pure matrix construction/access does
not link it either. Programs reaching a native operation link only this engine,
without pulling in the Polars engine or faer code into the runtime archive.
From a source checkout, build it once:
For the library plus local --dist metadata/licenses, use just engine linalg.
just build builds the compiler and both engines; see the
local build recipes.
The lower-level command below installs only the library.
python3 -B scripts/build_native_engine.py linalg --out target/release
The isolated plenty-linalg/ Cargo workspace has a pinned lockfile. The build
targets baseline x86-64; compatible SIMD is selected by the underlying kernels.
It runs sequentially and creates no thread pool. The large algorithm and SIMD
implementation justifies a native dependency instead of reimplementing it in
Plenty.
Allocation contract
The facade owns matrices in Plenty lists. Engine calls borrow strided numeric
views for one call and retain no handles or references. Faer’s low-level APIs
receive scratch allocated with MemBuffer::try_new; other scratch buffers use
Vec::try_reserve_exact. High-level infallible allocating faer owners are never
used. Native scratch lives outside Plenty’s allocation accounting, but allocation
failure is reported as AllocError.OutOfMemory.
The faer fork adds fallibly allocated, caller-owned packing workspaces. The engine acquires one before entering a kernel, then supplies it to all three optimized GEMM dispatch paths through a scoped borrow. This avoids initializing the backend’s owning thread-local cache. Rayon is disabled. Native tests deny allocations, including on the first call, and exercise blocked decompositions and CPUs without AVX.
A fixed four-slot pool reuses workspaces. Its mutex is held only while taking
or returning a buffer, never during numerical computation; concurrent calls
receive independent buffers. The pool itself allocates nothing. A cache miss
allocates fallibly, and a full pool drops returned excess buffers. The baseline
x86 build retains at most roughly 16 MiB of packing storage until process exit;
active concurrent calls may temporarily require additional buffers. Output and
decomposition-scratch allocation costs remain visible through Result.
Source builds fetch the fork once, with the exact Git revision pinned in
plenty-linalg/Cargo.lock. For offline builds, first run
cargo fetch --locked --manifest-path plenty-linalg/Cargo.toml while online,
then preserve both Cargo’s registry and Git caches. Prebuilt engine users need
neither Git nor a source checkout.
Unexpected Rust panics are caught at the C boundary and reported as
NativePanic; this is a guard against internal bugs, not an OOM recovery
mechanism. Engine operations write fresh Plenty-owned output buffers which are
discarded on error, leaving caller inputs intact.
Deployment
Prefer explicit distribution output. With a release compiler, the bundled engine and notices need no configuration:
plenty --compile analysis.plenty --dist my-app
./my-app/analysis
my-app must not exist. Plenty copies only reached engines, validates the
packaged metadata and library checksum, and includes all engine notices:
my-app/
analysis
libplenty_linalg_v1.so
linalg-notices/
engine.json
LICENSE-MIT
LICENSE-APACHE
THIRD-PARTY-NOTICES.md
licenses/rust/...
Distribution output searches only $ORIGIN (the executable’s directory),
with no build-machine fallback. Ordinary --compile FILE -o OUT still searches
$ORIGIN then the build-time engine directory; manual deployment requires copying
the exact library and notices into the same layout. A sibling lib/ directory is not searched automatically.
PLENTY_LINALG_ENGINE affects compilation, not an already-built program.
Check relocation without the original installation and without stale
LD_LIBRARY_PATH or LD_PRELOAD. Users need the host C libraries, not Rust,
Cargo, Python, or Plenty. Missing plla_v1_ symbols indicate an old engine.
Prebuilt releases target x86-64 and ARM64 GNU/Linux with glibc 2.28 and libgcc_s.so.1.
Compile and test your application on your oldest supported distribution: a
compatible engine cannot lower the application’s own glibc requirement.
See the shared native-engine contract.
std.ready
import std.ready waits for a native handle, or any of several handles, to
become readable or writable,
bounded by a Deadline and optionally by a
cancellation token. It is the blocking model for I/O on worker
threads: each worker makes bounded blocking waits, and no event loop, callback
registration or async syntax is involved. See
native effects on worker threads.
The runtime supplies four primitives (opcodes 419–422): a single ppoll on a
handle’s descriptor with a nanosecond timeout, and opening, registering with
and waiting on a selector. Deadlines, interruption retries, cancellation and
event decoding are checked Plenty in stdlib/std/ready.plenty.
import std.bytes
import std.io
import std.process
from std.process import Stdio
from std.ready import WaitError, read_before
from std.sync import CancellationToken
from std.time import deadline_after, milliseconds, monotonic_now
def main() -> Result[(), AllocError | IoError | RangeError | WaitError | std.bytes.BytesError | str]:
mut command = std.process.shell("echo ready")?
command.stdout(Stdio.Piped)
mut child = command.spawn()?
pipe: Result[File, str] = child.take_stdout().ok_or("missing piped stdout")
mut output = std.io.from_file(pipe?)
cancel = CancellationToken()?
deadline = deadline_after(&monotonic_now(), &milliseconds(5000))?
mut buffer = std.bytes.with_capacity(16)?
buffer.resize(16, 0u8)?
count = read_before(&mut output, &mut buffer, 0, 16, &deadline, &cancel)?
print(count, buffer.to_text_range(0, count)?.strip()?)?
child.wait()?
Ok(())
6 ready
Types
| Type | Contents |
|---|---|
Interest | Read, Write or ReadWrite |
Readiness | Public bool fields readable, writable, hangup, error |
WaitError | TimedOut, Cancelled, or Stream(StreamError) |
Selector | An owned set of registered handles; see Selectors |
Event | Public fields key: u64 and readiness: Readiness |
Watchable | Protocol: descriptor(self) -> &File, the handle a selector waits on |
hangup and error are reported whatever the interest. On Linux a pipe whose
writer has closed reports hangup without readable; a read then returns end
of input. error means a read or write will report the pending error.
Functions
| Function | Result |
|---|---|
wait_now(file: &File, interest: Interest) | Result[Readiness, WaitError]: one immediate poll |
wait_until(file: &File, interest: Interest, deadline: &Deadline, cancel: &CancellationToken) | Result[Readiness, WaitError] |
wait(file: &File, interest: Interest, deadline: &Deadline, cancel: &CancellationToken) | Result[Readiness, WaitError] |
read_before(stream: &mut FileStream, buffer: &mut Bytes, start: i64, stop: i64, deadline: &Deadline, cancel: &CancellationToken) | Result[i64, WaitError] |
write_before(stream: &mut FileStream, source: &[u8], deadline: &Deadline, cancel: &CancellationToken) | Result[i64, WaitError] |
selector(capacity: i64) | Result[Selector, WaitError] |
wait_until is a compatibility alias of wait, with the same cancellation
semantics. wait_now does not block or allocate a token; no readiness is
TimedOut.
- A handle that is already ready succeeds, even when the deadline has passed.
Otherwise the wait returns
TimedOutonce the deadline passes. - A token cancelled before the call returns
Cancelledeven when the handle is ready. One cancelled during the wait wakes it at once: the wait polls the token’s wake descriptor alongside the handle, andcancel()makes that descriptor readable. On loopback, 50 cancellations of a blockedacceptwoke it in 27 microseconds on average and 80 at worst. - An interrupted wait (
EINTR) is retried against the same deadline. - A closed
FileisStream(StreamError.Io(IoError.Closed)). An invalid descriptor is reported the same way. read_beforewaits for input and then makes oneFileStream.read; zero is end of input.write_beforewaits for space and then makes one write of at most 4096 bytes. This cap bounds pipe writes; it does not guarantee space in a blocking socket’s send buffer. It returns the bytes written; loop to write more.- A wait borrows the handle and allocates nothing. On
TimedOutorCancelled, the handle, the stream position and the caller’s buffer are unchanged.
Regular files are always ready; readiness is useful for pipes, terminals and
sockets. These generic FileStream transfer helpers neither change
descriptor mode nor retry a transfer after stale readiness. A blocking
descriptor can still block after a readiness report, for example when another
reader consumed the input. Use std.net’s transfer methods for
TCP: its descriptors stay nonblocking and its deadline methods retry against
the original budget.
Selectors
A selector waits on many handles in one call, so one worker can serve many idle connections. It owns one epoll descriptor and an event buffer of 16 bytes per event; it starts no thread.
| Method | Result |
|---|---|
selector.watch(file: &File, key: u64, interest: Interest) | Result[(), WaitError]: register, or change the key and interest |
selector.unwatch(file: &File) | Result[(), WaitError]; a handle that is not registered is not an error |
selector.wait(deadline, cancel) | Result[i64, WaitError]: how many events are ready, at least 1 |
selector.event(index: i64) | Result[Event, WaitError]: event index of the last wait |
selector.close() | Result[(), WaitError] |
capacityis the most events one wait reports, clamped to 1..64. Further ready handles stay ready for the next wait.- Registrations are level-triggered: a handle is reported by every wait while it is ready, until it is read, written or unwatched. Closing a registered handle removes it.
waitfollows the single-handle rules: ready handles are reported even when the deadline has passed, an earlier cancellation wins, a later one wakes the wait at once, andEINTRis retried.- An index outside the last wait’s count is
Stream(StreamError.OutOfRange). watch,unwatchandwaitneed the selector exclusively (&mut); handles are borrowed shared only for the call. A registered handle may still be read, written or closed by its owner.std.netstreams, listeners and UDP sockets havewatchandunwatchmethods that reportNetError.Watchablenames a handle by theFileit waits on.std.net’sTcpStream,TcpListener,UdpSocket,ConnectingandLookupimplement it, andstd.actor’sContext.watchtakes anyWatchable.
For TCP, follow an event with accept_now, read_now or
write_now. They make one nonblocking attempt
without a second ppoll. If the event is stale, NetError.WouldBlock returns
control to the selector loop instead of blocking other connections. Keep an
offset after a partial write, and stop watching for write readiness when the
output is drained. EOF is zero from a nonempty read range, not WouldBlock.
examples/net_wait_bench.plenty measures one worker serving N idle loopback
connections, one ready at a time, with a selector against a loop that polls
each connection with an expired deadline. Mean round trip in nanoseconds over
20,000 one-byte echoes, pinned to eight cores:
| Connections | Selector | Polling loop |
|---|---|---|
| 1 | 28,668 | 21,958 |
| 16 | 29,143 | 30,192 |
| 128 | 30,048 | 134,935 |
| 1024 | 31,515 | 1,099,546 |
The selector’s cost does not grow with idle connections; the polling loop’s grows linearly.
Readiness flags are tested with bit masks, not integer division. Selector storage and event offsets use documented bounded arithmetic: at most 64 records of 16 bytes. Caller arithmetic modes do not alter these contracts.
Cancellation descriptors
A token’s first cancellable wait creates its wake descriptor, an eventfd
opened close-on-exec, which the token keeps until it is dropped. A program
therefore holds one extra descriptor per token that has waited. If the
descriptor cannot be created (for example at the descriptor limit), the wait
falls back to blocking at most 10 milliseconds at a time and rechecking the
token, as before.
Example
from std.ready import WaitError, read_before
from std.sync import CancellationToken
from std.time import deadline_after, milliseconds, monotonic_now
def drain(stream: FileStream, cancel: CancellationToken) -> Result[i64, WaitError | OverflowError[i64]]:
mut owned = stream
mut buffer = std.bytes.with_capacity(64).unwrap()
buffer.resize(64, 0u8).unwrap()
deadline = deadline_after(&monotonic_now(), &milliseconds(5000)).unwrap()
mut total = 0
while True:
count = read_before(&mut owned, &mut buffer, 0, 64, &deadline, &cancel)?
if count == 0:
return Ok(total)
total = total + count
Ok(total)
The tutorial lesson runs a complete program.
Limits
- x86_64 Linux GNU only; elsewhere every wait reports
IoError.Unsupported. - Selectors are epoll instances; there is no edge-triggered or one-shot mode, and a wait reports at most 64 events.
- A token does not wake an unbounded
read,writeor childwait. - Sockets and name lookup are in
std.net.
Acknowledgements
Selectoris modelled on Python’sselectorsmodule and the Rustmiocrate, from whichInterestand caller-chosen event keys come.
std.regex
import std.regex compiles regular expressions and searches str text and
std.bytes buffers with them. A search runs in time linear in
the input for a compiled pattern: there is no backtracking. Every resource a
pattern or search can consume has an explicit limit. Pattern and search errors
use RegexError; arithmetic on freely constructed Match offsets has the
separate contract described below.
The parser, compiler and matching engines are Plenty code in the module:
a Pike VM, bounded ASCII prefix filters, a lazy DFA and a one-pass matcher.
The runtime supplies only primitives: Unicode properties and single-scalar
case mappings from the Rust toolchain’s Unicode tables, and reads of borrowed
UTF-8 by byte offset. They do not allocate, except the one that copies a byte
range into a new str, which can fail with an allocation error.
Alphabetic, Lowercase, Uppercase, White_Space, Numeric, Control,
and case mappings use the Unicode version in the Rust compiler that builds
Plenty (char::UNICODE_VERSION). Their results may change when that toolchain
is upgraded. Mark uses the independently pinned Unicode 17.0.0 tables in
std.unicode, as does that module’s normalization. The two
data versions need not match; ASCII classes and byte-offset rules are unchanged
by Unicode table upgrades.
import std.regex
from std.regex import RegexError
def main() -> Result[(), AllocError | IoError | RegexError]:
date = std.regex.compile("(?P<year>\\d{4})-(?P<month>\\d{2})-(?P<day>\\d{2})", &std.regex.limits())?
text = "Released 2024-03-15, patched 2024-04-02."
print(date.is_match(text)?)?
match date.captures(text)?:
case Some(groups):
match groups.name("year"):
case Some(year):
print("year", year.text(text)?)?
case Nothing:
print("no year")?
case Nothing:
print("no date")?
print(len(date.find_all(text, 10)?))?
print(date.replace_all(text, "${day}/${month}/${year}", 1000)?)?
Ok(())
True
year 2024
2
Released 15/03/2024, patched 02/04/2024.
Functions and types
| Item | Result | Contract |
|---|---|---|
limits() | Limits | The default limits, listed below |
compile(pattern: &str, bounds: &Limits) | Result[Regex, RegexError] | The compiled pattern, or its first error |
Limits(max_pattern_bytes, max_nesting, max_repeat, max_instructions, max_memory, max_work) has public NonNegative[i64] fields in that order. A
literal converts where a field is expected; a computed value needs
NonNegative.try_from(value)?.
| Field | Default | Bounds | Exceeded as |
|---|---|---|---|
max_pattern_bytes | 65536 | Pattern length in bytes | Limit.PatternBytes |
max_nesting | 128 | Depth of nested groups; at most 1000 | Limit.Nesting |
max_repeat | 1000 | Each count in {n}, {n,}, {n,m} | Limit.Repetition |
max_instructions | 100000 | Program instructions plus class items, after repetitions are expanded | Limit.Instructions |
max_memory | 67108864 | Matcher scratch bytes for one search | Limit.Memory |
max_work | 100000000 | Matcher steps in one call | Limit.Work |
compile traps when max_nesting is above 1000. Matcher
base scratch memory is 8 × (7n + 2ns + 2s + 8) bytes for n instructions and s
capture slots (two per group, counting the whole match); compile checks it
for the largest s the regex can use, so no later search fails on memory.
An eligible search may also allocate a 66,048-byte DFA cache, only when it
fits in the remaining max_memory budget. Smaller budgets select the VM
without rejecting an otherwise valid pattern. The VM scratch remains
available for fallback; all allocations remain fallible.
The estimate is checked before comparison: an unrepresentable word estimate
also exceeds every representable byte budget and reports Limit.Memory.
Actual unrepresentable allocation extents report
Allocation(AllocError.CapacityOverflow) before any matcher array is allocated.
Work is checked against its remaining budget before increment, including when
max_work is i64.MAX; exhaustion remains Limit.Work, not an arithmetic
error. Repetition-count overflow likewise remains Limit.Repetition.
RegexError is:
| Variant | Meaning |
|---|---|
Syntax(SyntaxError) | Invalid pattern or replacement template; SyntaxError(offset, message) gives the byte offset into it |
LimitExceeded(Limit) | A limit was exceeded; also Limit.Output and Limit.Matches for the per-call limits below |
OutOfRange | A start offset outside the input or inside a UTF-8 sequence, a Match outside its haystack, or a Cursor from another regex |
Allocation(AllocError) | Allocation failed |
Regex
All searches are leftmost-first, as in Perl and the Rust regex crate: of
the matches that start earliest, the one the pattern prefers wins
(a|ab matches a in ab; greedy repetitions prefer more).
| Method | Result | Contract |
|---|---|---|
group_count() | i64 | Groups, counting group 0 (the whole match) |
group_index(name: &str) | Option[i64] | The number of a named group |
instructions() | i64 | Program size, for choosing max_work |
is_match(text: &str) | Result[bool, RegexError] | Whether any match exists; stops at the first |
find(text: &str) | Result[Option[Match], RegexError] | The first match |
find_at(text: &str, start: i64) | Result[Option[Match], RegexError] | The first match at or after start |
captures(text: &str) | Result[Option[Captures], RegexError] | The groups of the first match |
captures_at(text: &str, start: i64) | Result[Option[Captures], RegexError] | As captures, from start |
cursor() | Result[Cursor, RegexError] | Iteration state for next_match |
next_match(text: &str, cursor: &mut Cursor) | Result[Option[Match], RegexError] | The next match, or Nothing after the last |
find_all(text: &str, max_matches: i64) | Result[list[Match], RegexError] | Every match in order |
split(text: &str, max_pieces: i64) | Result[list[str], RegexError] | The text between matches |
replace(text: &str, template: &str, max_bytes: i64) | Result[str, RegexError] | text with the first match replaced |
replace_all(text: &str, template: &str, max_bytes: i64) | Result[str, RegexError] | text with every match replaced |
find_at and captures_at start at a scalar boundary; assertions such as
^ and \b still see the text before start. Iteration (next_match,
find_all, split, replace_all) follows the Rust regex crate: after an
empty match, the next search starts one scalar later, and an empty match
where the previous match ended is skipped. So a* on baaa matches at
0–0 and 1–4, and split of "añ" by the empty pattern gives "", "a", "ñ", "".
Per-call limits: find_all fails with Limit.Matches rather than return more
than max_matches matches, and split rather than return more than
max_pieces pieces. replace and replace_all fail with Limit.Output when
the result would exceed max_bytes bytes.
A replacement template inserts group n for $n or ${n}, a named group
for ${name}, and $ for $$; a group that did not take part inserts
nothing. Any other $, including $name, an unknown group, or an unclosed
${, is a Syntax error at the $. The template is checked before any
search.
Bytes
is_match_bytes(data), find_bytes(data, start),
captures_bytes(data, start), next_match_bytes(data, cursor), and
find_all_bytes(data, max_matches) search a Bytes buffer. Valid UTF-8
sequences are scalars as in text; each byte of an invalid sequence (including
an encoded surrogate) is one position that nothing matches, not even
\P{Any} or [^a]. Offsets are byte offsets into data, and start may be
any offset from 0 to data.len(); a start inside a sequence makes its
remaining bytes invalid positions.
Match, Captures and Cursor
Match has public fields start and end (byte offsets, end exclusive) and
methods len() -> Result[i64, OverflowError[i64]], is_empty(),
text(haystack: &str) and bytes(data: &[u8]). len() checks end - start:
public fields may contain arbitrary offsets, so representable negative lengths
are retained and unrepresentable differences return an arithmetic error.
The text and byte methods still copy the matched part and report OutOfRange
for a haystack it does not fit. Matches returned by a successful search have
ordered, nonnegative offsets and a representable length.
Captures has len() (equal to group_count()), get(index: i64) and
name(name: &str), each Option[Match]. Group 0 is the whole match; a group
outside the pattern or not taking part is Nothing. A group inside a
repetition holds its last iteration.
Capture indexing validates the group number before multiplying it by two;
out-of-range indices, including i64.MAX, return Nothing. Internal opt-outs
are bounded by validated scalar widths, pattern/input spans, configured nesting,
successful instruction emission and preflighted matcher array extents. Unlimited
replacement does not maintain a redundant total-replacements counter.
A Cursor holds the matcher scratch and position, so a loop over
next_match allocates scratch only when it is created. Use it with the regex
that made it; a different pattern, memory budget or work budget returns
OutOfRange before consulting cached transitions.
Pattern syntax
| Syntax | Meaning |
|---|---|
x, \x for punctuation x | A literal scalar |
\n \t \r \f \v, \xHH, \x{H...} | Control characters and a scalar by hexadecimal value |
. | Any scalar except \n; any scalar under s |
[abc], [a-z], [^...] | A class, range, or negated class; - is literal first or last, ] must be escaped |
\d \D | ASCII digit [0-9], or not |
\w \W | ASCII word scalar [0-9A-Za-z_], or not |
\s \S | ASCII space [\t\n\v\f\r ], or not |
\p{Name} \P{Name} | Has the Unicode property, or not; also inside classes |
xy, x|y | Sequence; alternatives, the first preferred |
x* x+ x? | 0 or more, 1 or more, 0 or 1; greedy |
x{n} x{n,} x{n,m} | Counted repetition |
x*? x+? x?? x{n,m}? | Lazy: prefer fewer |
(x), (?P<name>x), (?<name>x) | Capturing group; names match [A-Za-z_][A-Za-z0-9_]* and are unique |
(?:x), (?flags:x) | Non-capturing group, optionally with flags |
(?flags) | Set flags until the end of the enclosing group |
^ $ | Start and end of text; of a line under m |
\A \z | Start and end of text |
\b \B | ASCII word boundary, or not |
Properties are Alphabetic, Lowercase, Uppercase, White_Space,
Numeric, Control, Mark (general category M), ASCII, and Any. Flags
are i (case-insensitive), m (multi-line ^ $), and s (. matches
\n); (?i-m) sets i and clears m.
Case-insensitive matching uses simple, one-scalar case mappings: a scalar
matches another when their lowercase or uppercase mappings agree, so (?i)k
matches the Kelvin sign and (?i)σ matches Σ and ς, but (?i)ss does
not match ß.
These are errors rather than silently different behavior: look-around
((?=, (?!, (?<=, (?<!), backreferences (\1), nested classes and
POSIX classes ([[:alpha:]]), an empty class ([]), a repetition of a
repetition (a**; write (?:a*)*), a range with a class end ([\d-z]), the
x flag, and unknown escapes, properties, or flags. A pattern with an
unmatched ) or ( is an error too.
Costs
Search is linear in the input for a fixed compiled pattern. The Pike VM
visits each instruction at most once per input position in each of its two
sets. Accelerators can add a bounded scan before transferring to the VM;
there is no unbounded backtracking. max_work caps matcher steps in each
call: find, captures, and is_match run one search; find_all, split,
and replace_all count every search they make; each next_match call is
counted separately. A native literal batch reserves the sum of its maximum
search spans before starting, charges each needle’s actual search span, and
retains that charge across fallback. Even a match near the beginning can charge
an entire window; insufficient budget returns Limit.Work before the batch.
Short tails use the scalar filter. Work includes accelerated byte scans and uncached DFA
transitions, and is retained across fallback. Exact step counts depend on
the engine and its cache. Iteration can rescan text after a match, so choose
max_work for the whole call.
The compiler selects these accelerators without changing match semantics:
- A case-sensitive literal prefix uses substring search in text. For programs of at most 256 instructions, a bit-parallel filter can instead recognize up to eight mandatory ASCII positions across alternatives and positive range classes. It admits possible matches for the matching engine to confirm; nullable, case-insensitive or non-ASCII paths shorten or disable this filter. Up to eight paths can occupy separate eight-bit lanes, preventing false candidates made by mixing alternatives. Extraction visits at most 2,048 instructions and keeps at most 17 pending paths; exceeding those bounds retains the position-wise union.
- When the extracted paths expand to at most eight ASCII literals of the same
three-to-eight-byte width, a 64-byte scalar probe returns nearby matches
promptly. Longer scans then use native substring search in windows
of at most 1,024 bytes. Windows overlap by
width - 1, and all literals are searched before selecting the earliest candidate. The engine still confirms candidates and resolves alternative priority. Literal storage is bounded by eight short strings; it is compiled metadata, not per-search scratch. Wider classes or excessive path products retain the scalar filter. - For ASCII-only programs of at most 62 instructions without assertions, a lazy DFA answers existence searches. When every match has the same byte length, it also supplies match offsets for searches, cursors and replacements. At its initial state it uses the same mandatory-prefix filter to skip impossible starts. Its 64-state cache is local to the matcher and reused by a cursor. Cache exhaustion disables the DFA for that matcher and resumes the VM.
- For small programs without assertions, a one-pass table accelerates searches that need only whole-match offsets when each ASCII byte has at most one preferred consuming path. It retains greedy/lazy ordering. Non-ASCII input transfers to the VM. Failed anchored attempts can consume at most one input length before fallback, preventing quadratic rescanning. The table has at most 62 × 129 entries and is built only with sufficient memory headroom.
ASCII accelerators borrow the text’s UTF-8 bytes once per search and read the view directly, without allocating or calling the runtime for each byte. Scalar decoding and assertions retain their Unicode boundary semantics.
Captures requiring subgroup positions use the Pike VM, with prefix filtering
where applicable. Unsupported patterns continue to use the VM. Each search
allocates scratch; a Cursor reuses it. No new public API or opt-in flag is
needed. The regex-redux measurements
exercise cleaning, counting and replacement with the same patterns as the
Rust regex reference; remaining gaps include Unicode DFA transitions and
the remaining scalar scan lowering (issue #760).
Choices
The module uses its own matcher rather than the Rust regex crate because
that crate allocates infallibly and would abort on allocation failure, and
its Unicode tables would enlarge every program that imports std.regex.
The narrower memchr::memmem::find primitive uses constant space and no
allocation. Its SIMD byte search and portable fallbacks justify native code;
Plenty’s current SIMD API has no byte-vector search operations. Literal
extraction, batching, work limits and all matching policy remain Plenty. The
syntax is the subset of the Rust crate’s syntax with the same meaning, except
that Perl classes and \b are ASCII-only, which the tests check against
the crate.
Acknowledgements
-
Syntax, match semantics and the prefilter/fallback strategy follow the Rust
regexcrate 1.12.3. -
Matching uses Rob Pike’s virtual machine, as described by Russ Cox in “Regular Expression Matching: the Virtual Machine Approach” (2009).
-
Native byte substring search uses
memchr2.8.0 (MIT OR Unlicense), including its SIMD implementations and portable fallbacks.
std.report
std.report provides bounded human-readable reports using the same
std.fmt.Formatter[W] and FormatError as value display and
f-strings. It provides rendering helpers for the canonical std.fmt.Formattable
protocol; it does not own another formatter, protocol, or error family. Import
formatter types and constructors from std.fmt.
import std.report
from std.fmt import FormatError, Formatter
pub class DiskFull:
path: str
needed: u64
pub def format[W](self, output: &mut Formatter[W]) -> Result[(), FormatError]:
output.text("disk full writing ")?
output.text(&self.path)?
output.text(": need ")?
output.unsigned(self.needed, 10)?
output.text(" more bytes")
def main() -> Result[(), IoError | FormatError]:
error = DiskFull("out.csv", 4096u64)
print(std.report.to_string(&error, 256, 4)?)?
Ok(())
disk full writing out.csv: need 4096 more bytes
Formatting contract
The structural Formattable protocol requires a public shared-receiver method
on a public class or enum, since its bound is checked in std.fmt:
pub def format[W](self, output: &mut Formatter[W]) -> Result[(), FormatError]:
pass
The writer parameter belongs to the method. A single implementation can render
to a string, stderr, or a caller-owned writer, and can call all the scalar and
format-spec methods described in std.fmt. For example,
output.float_spec(value, &std.fmt.spec(precision=2, kind="f")) is available
inside a report. Scratch for advanced numbers can allocate; this does not alter
the allocation-free text/integer/boolean stderr path.
render[T: Formattable, W](&T, &mut Formatter[W]) borrows the value, enters one
depth level, calls format, and restores depth even on error. Depth is in
0..=64; zero permits direct scalar writes but rejects render. Nested values
should use render to participate in the depth limit. Direct user-method calls
do not count depth automatically.
An anonymous sum satisfies
Formattable when every normalized member satisfies it. Dispatch borrows the
active member and calls its writer-generic format method without moving the
payload or allocating a wrapper. The sum adds no nesting level beyond the one
entered by render. Aliases, reordered members and singleton normalization
preserve this behavior. Direct sum.format(&mut output) uses the same dispatch
without entering a depth level. A missing, private, or mismatched method on any
member fails the bound, even when that member is inactive. This forwarding is
specific to std.fmt.Formattable; other protocols, including display/debug,
are not automatically forwarded.
Helpers and destinations
| Operation | Behavior |
|---|---|
std.report.render(&value, &mut output) | Borrowed report into any Formatter[W] |
std.report.to_string(&value, max_bytes: NonNegative[i64], max_depth: NonNegative[i64]) | Create a string formatter, render, and finish |
std.fmt.formatter(writer, scratch, max_bytes, max_depth) | Transfer a writer and caller-provided scratch into a formatter |
std.fmt.string_formatter(max_bytes, max_depth) | Create an allocating Formatter[Cursor] |
std.fmt.stderr_formatter(max_bytes, max_depth) | Create an allocation-free Formatter[Stderr] |
std.fmt.finish(&output) | Copy a string formatter into an owned str; takes &Formatter[Cursor] |
The formatter does not add punctuation, flush stdout, close a stream, or lock a
whole record. Each write counts toward its byte budget. completed() records
the accepted prefix. has_failed() records terminal stderr failure; subsequent
emissions on that formatter return FormatError.OutputClosed without writing. Other writer
destinations retain their own recovery policy and do not set this flag.
Formatting limits and allocation failures leave the formatter usable. See
errors and partial output for shared errors
and retry rules. In particular, finishing stderr is rejected by its type.
For allocation-free stderr reports, use text, integer, unsigned, and
boolean. Text writes use the native descriptor-2 operation directly, retry
interruptions, and count successful short writes. A closed or read-only stderr
returns Output(StreamError.Io(IoError.System(9))) (EBADF). On supported
x86-64 Linux the write suppresses SIGPIPE for that operation, so a closed pipe
returns an error and normal cleanup runs. Advanced formatting lazily acquires
scratch and can fail with Allocation(...).
The string formatter stages a complete UTF-8 text emission before calling its
cursor. Failed staging or cursor growth preserves the valid UTF-8 output and
byte count. finish borrows the formatter, so an allocation failure can be
retried. A whole record can still have a successful prefix before a later
emission fails; do not replay it blindly.
Current boundaries
This protocol selects human-readable
entrypoint reports; types without it
retain structural rendering. std.error.Context[E] supports the protocol when
its cause does. display/debug remain the value presentation hooks; sharing
the formatter does not change which hook an entrypoint chooses. The
tutorial shows enum and sum
reporting.
The completed reporting proposal records the original reporting design and its later unification.
std.secrets
import std.secrets returns unpredictable bytes from the operating system as
buffers, text tokens, and integers, with one error type, SecretsError.
Use it for session tokens, keys, nonces, and password-reset links. For
repeatable simulation and sampling use std.random; the two
modules share no state and no code path.
Entropy comes from the operating system. Plenty implements no random generator for secrets: native OS services supply every byte.
Policy is checked Plenty code over one native primitive, the intrinsic
__io_entropy_read, which is
reserved for the standard library and makes a single OS
request. Linux uses getrandom(2) (glibc 2.25 or later, Linux 3.17 or later).
The macOS runtime uses getentropy, with at most 256 bytes per request;
Windows uses the system-preferred BCryptGenRandom generator, with at most
u32::MAX bytes per request. Short counts are normal and the Plenty layer
retries until its requested range is full. See platform support
for compiler qualification; an entropy backend alone does not qualify a port.
import std.secrets
from std.secrets import SecretsError
def main() -> Result[(), IoError | SecretsError | OverflowError[u64]]:
session = std.secrets.token_urlsafe(32)?
reset_code = std.secrets.token_hex(16)?
roll = std.secrets.below(6u64)? + 1u64
# The values differ on every run; their shapes do not.
print(len(session), len(reset_code), roll >= 1u64 and roll <= 6u64)?
Ok(())
43 32 True
Functions
| Function | Result | Contract |
|---|---|---|
fill(target: &mut Bytes, start: i64, stop: i64) | Result[(), SecretsError] | Overwrite target[start:stop] with secret bytes |
token_bytes(count: i64) | Result[Bytes, SecretsError] | A new buffer of exactly count secret bytes |
token_hex(count: i64) | Result[str, SecretsError] | count secret bytes as 2 * count lowercase hex digits |
token_urlsafe(count: i64) | Result[str, SecretsError] | count secret bytes as unpadded base64url, ceil(4 * count / 3) characters |
random_u64() | Result[u64, SecretsError] | A uniform u64 |
below(bound: u64) | Result[u64, SecretsError] | Uniform in [0, bound) without modulo bias; zero is EmptyRange |
count is a number of bytes, not characters, and has no default: choose it for
the secret’s purpose. Thirty-two bytes is ample for a bearer token. A negative
count is a size that cannot be allocated, so it is
Allocation(AllocError.CapacityOverflow); zero gives an empty result without consulting the
system. fill follows the range rule. The text
forms use the std.encoding codecs.
Failure policy
- No fallback. If the system cannot supply entropy, the operation returns
SecretsError.Entropy. It never substitutes the clock, an address, a seeded generator, or/dev/urandom. - Blocking. Linux waits until its kernel generator has been initialized, which matters only very early in boot. The API has no nonblocking form.
- Short and interrupted requests are retried until the range is full. A
request that reports zero bytes for a nonempty range is
UnexpectedEof; a count larger than the request isInvalidProgress. - No partial secrets. When
fillfails, the whole requested range is zero. A failed token call releases its buffer and returns only the error. - Scrubbing.
token_hex,token_urlsafe,random_u64, andbelowzero their intermediate byte buffer before releasing it. The values they return, and atoken_bytesbuffer, are the caller’s to protect: Plenty does not zero memory on drop, and the encoder’s own intermediate buffer is not scrubbed.
Sources
The functions above are shorthand for methods on Source[R], which applies
the same policy to any byte reader.
| Item | Contract |
|---|---|
os() | Source[OsEntropy], the operating-system source |
source_from[R: Reader](inner: R) | Source[R] over a std.io reader |
Source[R].fill, token_bytes, token_hex, token_urlsafe, next_u64, below | As the functions above, with self: &mut Source[R]; next_u64 is random_u64 |
Source[R].inner | The reader, public so its owner can inspect or replace it |
os_entropy() | OsEntropy, a std.io Reader whose read makes one request and may return a short count |
source_from exists to test code against scripted short reads and failures,
and to use a hardware device opened as a stream. The secrecy of its output is
exactly that of the reader supplied; only os() is backed by the kernel
generator. OsEntropy has a private field and can only be made by
os_entropy() and os().
next_u64 reads eight bytes as a little-endian integer. below draws
next_u64() until the value is at least 2^64 mod bound and returns it modulo
bound.
Errors
pub enum SecretsError:
OutOfRange
EmptyRange
Entropy(StreamError)
Allocation(AllocError)
| Variant | Meaning |
|---|---|
OutOfRange | A range outside the buffer; the source was not consulted |
EmptyRange | below(0) |
Entropy(cause) | The source failed; cause is the StreamError |
Allocation(cause) | A token or scratch buffer could not be allocated, including CapacityOverflow for a negative count |
From the operating system, cause is StreamError.Io(IoError.System(code))
with the getrandom error number: for example 38 (ENOSYS) on a kernel
without the call, or under a sandbox that denies it.
Ownership and threads
Targets are exclusive loans. Source[OsEntropy] and OsEntropy hold no
descriptor and no buffered bytes; creating one costs nothing and they need no
closing. The module has no global state, passes the ordinary worker-effect
checks, and may be used from any thread. Each request goes to the kernel, so
threads never share generator state in the process, and a forked child cannot
repeat its parent’s values.
below validates its bound into NonZero[u64] before rejection sampling, so
its remainder operations are statically infallible. Reading an eight-byte word
uses one documented bounded shift-offset opt-out. Neither changes the entropy
source, retry policy, zeroization or public error type.
Costs
| Operation | Allocates | System requests |
|---|---|---|
fill | Nothing | One per 33,554,431 bytes at most; one for up to 256 bytes |
token_bytes(count) | One buffer of count bytes | As fill |
token_hex, token_urlsafe | The token buffer, the encoded buffer, and the str | As fill |
random_u64 | One 8-byte scratch buffer | One |
below | As random_u64, per attempt | Fewer than two expected |
Nothing is buffered between calls, so every operation is at least one system
call. To draw many small values, fill one buffer and decode it, or seed a
std.random generator when the values need not be secret.
Throughput has not been measured on an idle machine.
Limits
- Linux only, through
getrandom(2); there is no/dev/urandompath for kernels older than 3.17. - No nonblocking or deadline-bounded request.
- No key derivation; see backlog B67. Digests, HMAC, and
constant-time comparison are in
std.crypto. - No zero-on-drop buffer type.
- No choice or shuffle helpers; combine
belowwith indexing.
std.shlex
std.shlex splits text containing shell-style words and quotes one argument
for a POSIX shell. It is written in Plenty and never starts a process, reads
the environment, or expands shell expressions.
Prefer std.process.Command.arg when you
already have individual arguments: pass them directly, without quoting.
Use split for an argument string supplied by a configuration file or user.
Interface
| Function | Result | Contract |
|---|---|---|
split(text: &str, comments: bool = False) | Result[list[str], ShlexError] | Remove word quoting and escapes, returning owned arguments |
quote(text: &str) | Result[str, ShlexError] | Encode one argument for a POSIX shell |
import std.shlex
from std.shlex import ShlexError
def main() -> Result[(), ShlexError | AllocError | IoError]:
print(std.shlex.split("--name 'Ada Lovelace' --literal '$HOME'")?)?
print(std.shlex.quote("two words")?)?
Ok(())
["--name", "Ada Lovelace", "--literal", "$HOME"]
'two words'
Splitting rules
- Space, tab, carriage return and line feed separate unquoted words. Other
Unicode whitespace is ordinary data; this whitespace convention follows
Python’s
shlexrather than a shell’s command grammar. - Single quotes preserve everything until the next single quote. Double
quotes preserve whitespace and single quotes. Adjacent quoted and unquoted
parts join into one word, and
''or""creates an empty word. - Outside quotes, backslash preserves the next scalar. Inside double quotes,
it escapes only
$, backtick, double quote, backslash and line feed; before anything else, the backslash is retained. - Backslash followed by line feed is removed outside single quotes. Alone, that continuation does not create an empty word.
- With
comments=True, an unquoted#at the start of a word skips the rest of that line.a#b, quoted hashes and escaped hashes stay literal. Comments are disabled by default.
There is no variable, command, arithmetic, tilde or wildcard expansion.
Operators such as ;, |, > and && are ordinary word contents, not a
pipeline or a command separator. Dollar-single-quoted strings, substitutions,
here-documents and shell grammar are not supported. This is not a shell
script validator, nor an exact implementation of Python’s shlex modes.
Quoting and errors
quote leaves nonempty ASCII words containing only letters, digits and
_@%+=:,./- unchanged. Otherwise it uses single quotes and escapes embedded
single quotes by closing and reopening the quoted sections. Empty text becomes
''. For every accepted input, split(quote(text)?)? returns exactly one
word equal to text.
Quoted text is for one argument in a noninteractive POSIX shell (sh -c or a
script), not for an entire script or for Windows shells. Control characters
remain literal; do not paste or feed this output to an interactive shell.
It does not make an executable name, option,
path, or shell program appropriate for an application. Prefer argument-vector
spawning; when a shell is needed, passing data through arguments or environment
variables also avoids building scripts from data.
ShlexError has UnclosedQuote(position), TrailingEscape(position),
Nul(position) and Allocation(AllocError). Positions are zero-based Unicode scalar indices, matching
Plenty string indexing. The quote/escape position points to its opening
character. Both functions reject NUL, including in a comment, because POSIX
process arguments cannot contain it. Allocation failures return
Allocation(cause); no partial result is exposed and intermediate storage is released.
Splitting takes linear time and uses a reusable scalar-piece list plus the returned words. Quoting takes linear time and may allocate intermediate text when escaping apostrophes. These are whole-string APIs; callers handling untrusted, arbitrarily large input should bound that input before parsing.
The API draws on Python’s shlex
and the Rust shlex crate, with the
specific splitting rules above rather than a compatibility mode switch.
Acknowledgements
- API modelled on Python’s
shlexand the Rustshlexcrate 2.0.1.
std.sketch
Bounded summaries for streams too large to retain. These pure-Plenty algorithms own their memory and random state. They do not wrap Apache DataSketches or share its serialized formats. Also see xor filters and cuckoo filters.
from std.sketch import SketchError, hyperloglog, hash_u64, quantiles
def main() -> Result[(), SketchError | AllocError | IoError | ZeroDivisionError]:
mut users = hyperloglog()?
mut latency = quantiles()?
for i in range(10000):
user = u64(i % 2500)
users.add(hash_u64(user))
latency.add(f64(i % 100))?
print(f"about {users.estimate():.0f} distinct users"?)?
print("median", latency.quantile(0.5)?, "max", latency.quantile(1.0)?)?
Ok(())
about 2511 distinct users
median 47.0 max 99.0
Distinct identities
hyperloglog(precision=12, seed=0) creates a dense HyperLogLog. Precision is
4–18; storage is exactly 2**precision packed one-byte registers plus Bytes metadata.
add(key: u64) and merge(other) allocate nothing. Duplicate keys are harmless.
Merge requires equal precision and seed and leaves the receiver unchanged on
SketchError.Incompatible. Estimate and merge cost O(registers).
estimate() uses the original HyperLogLog harmonic estimate with linear counting
for small cardinalities. relative_standard_error() returns 1.04/sqrt(m):
an asymptotic standard error, not a guaranteed error bound. There is no HLL++
bias table or near-64-bit-universe correction. Use this for ordinary cardinality
estimation, not exact counts or estimates approaching the hash-space size.
See the HyperLogLog paper.
Keys are numeric identities. hash_u64(key, seed=0) is a stable SplitMix64 mixer,
not a cryptographic hash. The algorithms are designed for ordinary data, not
adversarially chosen keys. Applications mapping text or compound values to u64
must choose and persist their own stable hash convention; collisions in that
mapping cannot be undone by a sketch.
Quantiles
quantiles(capacity: Positive[i64] = 1024, seed=0) creates Quantiles, a uniform sample without
replacement using reservoir Algorithm R.
Capacity is up to 1,048,576 finite f64 values; a larger capacity traps. add(value) rejects NaN/infinities;
count() is the number of accepted values and retained() is at most capacity.
Construction reserves the sample buffer; updates allocate nothing, maintain it
in sorted order, and cost O(capacity) when replacing a sample. Queries cost O(1).
quantile(fraction) accepts a finite number in [0,1]. Empty streams return
SketchError.Empty. Endpoints return the exact minimum/maximum. Other fractions
use the nearest-rank order statistic of the retained sample (no interpolation).
Results are exact before sampling starts. Afterwards they are probabilistic;
there is no deterministic rank-error guarantee or tail-relative guarantee.
Increase capacity to improve accuracy. This first API has no quantile merge.
It is a simple reservoir, not KLL, REQ, or a t-digest.
The sketch owns a std.random.Rng. The seed, accepted input order, and capacity
determine its sample. Serializing and restoring preserves continued sampling.
The random generator’s existing version-stability contract is unchanged.
Frequent identities
frequent_items(capacity: Positive[i64] = 128) creates a FrequentItems
Misra–Gries summary with up to 65,536 counters; a larger capacity traps. It accepts unit-weight u64 identities; there is no weighted
update or merge in this API. Construction allocates two fixed arrays.
add(key) and frequency(key) allocate nothing and cost O(capacity).
frequency(key) returns Frequency(key, lower, upper) with deterministic bounds
on the true count. error_bound() is the number of full counter decrements,
at most count() // (capacity + 1). A missing candidate has lower bound zero
and upper bound error_bound(). candidates() allocates a list of retained
nonzero counters in storage order, not frequency order. Every key whose true
frequency exceeds error_bound() appears in that list. A second pass over the
original data can compute exact counts for these candidates.
Errors and persistence
InvalidArgument covers invalid configuration or nonfinite input. Count overflow
returns Limit without accepting the update: these numerical counters stay exact,
never wrap or saturate, and a rejected update leaves samples, extrema, RNG state
and frequency bounds unchanged. Nonfinite quantile input is still
InvalidArgument, even when the count is exhausted. Constructors, readers and updates
that can also fail for another reason report allocation failure as
SketchError.Allocation(cause); candidates() and to_bytes() fail only to
allocate and return AllocError. All allocation failures remain recoverable.
to_bytes() writes a versioned Plenty format: PSH1, PSQ1, or PSF1.
Integers use little-endian u64 words. Quantile values use 32-byte, space-padded,
shortest round-trip scientific ASCII slots; this preserves finite f64 values,
including signed zero, without host-endian assumptions. These are checkpoint
formats, not interchange formats with third-party sketch libraries.
hyperloglog_from_bytes(data, max_precision=18),
quantiles_from_bytes(data, max_capacity: Positive[i64] = 1048576), and
frequent_items_from_bytes(data, max_capacity: Positive[i64] = 65536) reject unknown versions,
bad lengths, invalid state and trailing bytes. Pass a smaller caller limit when
reading untrusted checkpoints. All three readers take a &[u8] view. Validated precision/capacity bounds
keep checkpoint sizes and offsets representable; frequent-item accounting is
validated against the serialized total before adding each counter.
Formats are version-tagged: future incompatible formats need a new version and
must not silently reinterpret these bytes. They contain no checksum or
authentication; structurally valid corruption can change estimates.
The crate audit for issues #496–#498 found infallible allocations in
datasketches 0.5.0 (hll/array8.rs, req/compactor.rs), xorf 0.13.0
(prelude/mod.rs), and scalable_cuckoo_filter 0.5.1 (bits.rs). Small direct
implementations keep Plenty’s recoverable allocation policy without a native
dependency or ABI.
Acknowledgements
- Uses the HyperLogLog algorithm by Philippe Flajolet, Éric Fusy, Olivier Gandouet and Frédéric Meunier (2007).
- Uses reservoir Algorithm R as described by Jeffrey Vitter (1985).
std.sketch.xor
A static approximate-membership filter over u64 keys. build(keys, max_keys: NonNegative[i64], seed=0) accepts duplicates, sorts/deduplicates a fallible copy, and returns an
XorFilter. The input is not mutated. max_keys bounds input entries, not
just distinct keys; a max_keys above 16,777,216 traps.
from std.sketch import SketchError
import std.sketch.xor as xor
def main() -> Result[(), SketchError | AllocError | IoError]:
keys = [10u64, 20u64, 30u64, 20u64]?
filter = xor.build(&keys, 1000)?
print(filter.len(), filter.contains(20u64), filter.contains(25u64))?
Ok(())
3 True False
contains(key) allocates nothing and reads three fingerprint positions. Inserted
keys have no false negatives for an intact filter. Absent keys may return true:
approximately 1/256 with well-distributed hashes. This is not an exact set,
security boundary, or an adversarial-key guarantee. len() counts distinct
input keys; the empty filter always returns false.
Construction uses three disjoint blocks, graph peeling, and reverse XOR8
fingerprint assignment. The persistent filter has 3 * (1 + unique_count // 2)
one-byte slots, plus metadata. At the maximum key count, each block has
8,388,609 slots. Position reduction multiplies a 32-bit hash fragment by this
bounded block size without overflow; only the SplitMix64 mixer and attempt-seed
schedule use explicit modular arithmetic. Scratch memory is O(input entries + slots),
considerably larger than the final filter. Sorting is O(n log n), followed by
up to 32 O(n) peeling attempts. Failure to peel returns
SketchError.ConstructionFailed; allocation failures return
SketchError.Allocation(cause).
See Graf and Lemire’s xor-filter paper.
This first implementation deliberately uses more construction slack than a tuned xor/fuse filter. It is immutable after construction. For changing data, see cuckoo filters.
to_bytes() returns Result[Bytes, AllocError]. from_bytes(data, max_keys: NonNegative[i64]) reads a &[u8] view and returns SketchError.
The PSX1 format stores seed, unique count, block size (little-endian u64), then
fingerprint bytes. Readers validate the count/shape and exact length before
allocation, and reject values beyond the caller’s limit. It is a Plenty-specific
versioned checkpoint, not the xorf crate’s representation. It has no integrity
checksum: valid-looking corruption can introduce false negatives.
Acknowledgements
- Uses the xor filter by Thomas Mueller Graf and Daniel Lemire (2020).
std.sketch.cuckoo
cuckoo(initial_buckets=64, max_buckets: Positive[i64] = 1048576, seed=0) creates a scalable
approximate-membership filter for u64 keys. Initial buckets must be a power of
two between 1 and 1,048,576. The total bucket limit is at least that large; a
limit above 16,777,216 traps. Each bucket has four two-byte fingerprints. Layers double
geometrically; the caller’s limit counts buckets across all layers. The
validated bucket budget bounds byte offsets and the live occurrence count
(at most four occurrences per bucket); these sizes and counts do not wrap or
saturate.
insert(key) returns the layer index receiving that occurrence. Every call
inserts an occurrence, including duplicates. contains(key) searches all
layers. len() counts live inserted occurrences, not distinct keys;
layers() and buckets() expose current storage growth.
An existing layer is tried with at most 256 eviction steps. A failed eviction
chain is rolled back completely before trying another layer or allocating a
new one. Allocation failure or SketchError.Limit preserves the filter and its
membership. Construction allocates bounded eviction scratch; insertions into
existing layers and membership/deletion queries allocate nothing.
Keep the layer index that insert returns; remove needs it:
from std.sketch import SketchError
from std.sketch.cuckoo import cuckoo
def main() -> Result[(), SketchError | AllocError | IoError]:
mut seen = cuckoo()?
receipt = seen.insert(42u64)?
seen.insert(7u64)?
print(seen.len(), seen.contains(42u64))?
print(seen.remove(42u64, receipt)?, seen.len(), seen.contains(7u64))?
Ok(())
2 True
True 1 True
Deletion requires the insertion receipt
Keep the index returned by insertion alongside the authoritative record.
remove(key, layer_index) requires a known live occurrence inserted into that
layer of this filter. It returns Result[bool, SketchError]; an invalid layer
index is InvalidArgument. The layer indices survive serialization and remain
stable during growth. Empty layers are retained.
A positive contains is insufficient evidence for deletion: it may be a false
positive. Also, a key may falsely match an earlier layer even when it really
exists in a later layer. Supplying the receipt avoids that cross-layer ambiguity.
Removing an absent occurrence, reusing a receipt, or using the wrong filter/layer
can erase another key’s fingerprint and cause false negatives. Do not deduplicate
inserts using contains; doing so breaks deletion accounting. Applications that
cannot retain this information should use the filter only for membership checks.
Within a layer, equal fingerprints sharing a bucket share both candidate buckets, so deleting one of the corresponding known occurrences remains sound. The fingerprints do not identify which colliding occurrence was removed.
Accuracy, cost and persistence
Intact filters have no false negatives under the deletion precondition. False positives are approximately 8/65536 per full layer for ordinary hashes, increasing with the number of layers. Zero fingerprints map to one, causing a small bias. This is not an adversarial-key or security guarantee. Queries cost O(layers), with up to eight slot comparisons per layer. Growth initializes a new array and may be slower than a successful insertion into an existing layer.
to_bytes() writes PSC1, seed, total configured limit, initial bucket count,
layer count, and occurrence count as little-endian u64, followed by little-endian
u16 slots in layer order. from_bytes(data, max_buckets: Positive[i64]) reads a &[u8] view,
validates shape, counts, length and the caller limit before constructing storage,
and returns SketchError. The caller limit bounds the serialized
configured maximum, not only today’s occupancy. Serialization fails only to
allocate and returns AllocError. The format is versioned, Plenty-specific and unauthenticated.
Acknowledgements
- Uses the cuckoo filter from “Cuckoo Filter: Practically Better Than Bloom” by Bin Fan, David G. Andersen, Michael Kaminsky and Michael D. Mitzenmacher (2014).
- Layered growth takes the Rust
scalable_cuckoo_filtercrate as a design reference; no code is taken from it.
std.signal
import std.signal lets a program receive SIGTERM and SIGINT as an
event instead of being ended by them. Supervisors such as systemd and
container runtimes stop a service with SIGTERM; a terminal sends SIGINT
on Ctrl-C. A program that never calls listen keeps the default action:
either signal ends it at once.
The runtime supplies one primitive (opcode 500): it installs the handler and
returns the program’s end of a socket pair. Waiting, decoding and the
cancel_on helper are checked Plenty in stdlib/std/signal.plenty.
import std.process
import std.signal
from std.ready import WaitError
from std.signal import SignalError
from std.sync import CancellationToken
from std.time import deadline_after, monotonic_now, seconds
def main() -> Result[(), AllocError | IoError | RangeError | SignalError | WaitError]:
mut signals = std.signal.listen()?
# Play the supervisor: a shell sends SIGTERM to its parent, this program.
mut kill = std.process.shell("kill -TERM $PPID")?
kill.spawn()?.wait()?
deadline = deadline_after(&monotonic_now(), &seconds(5))?
received = signals.wait(&deadline, &CancellationToken()?)?
print("caught", received)?
signals.close()?
Ok(())
caught std.signal.Signal.Terminate
Types and functions
| Item | Contents |
|---|---|
Signal | Interrupt (SIGINT) or Terminate (SIGTERM) |
SignalError | AlreadyListening, or Stream(StreamError) |
Signals | The handle on which caught signals arrive |
listen() | Result[Signals, SignalError] |
cancel_on(signals: Signals, stop: CancellationToken) | Result[Option[Signal], WaitError] |
| Method | Result |
|---|---|
signals.wait(deadline: &Deadline, cancel: &CancellationToken) | Result[Signal, WaitError]: the next signal |
signals.received() | Result[Option[Signal], WaitError]: a signal already caught, without waiting |
signals.watch(selector: &mut Selector, key: u64) | Result[(), WaitError]: register for reading |
signals.unwatch(selector: &mut Selector) | Result[(), WaitError] |
signals.close() | Result[(), WaitError] |
wait follows the std.ready rules for deadlines and
cancellation. After a selector reports the handle,
received takes the signal. WaitError is the std.ready type.
Behavior
listencatches both signals for the whole process, whichever thread they arrive on. Each caught signal is sent as one byte on a socket pair; the handler makes no allocation and takes no lock.- A process listens once. A second
listenisAlreadyListening, even after the first handle is closed. - Only the main thread may call
listen: a worker that reaches it is rejected at compile time, likestd.env.set_current_dir. See native effects. A worker may own or borrow theSignalshandle and wait on it. - After the first signal, both signals have their default action again, so a second one ends the process at once. A drain that takes too long can be cut short with a second Ctrl-C.
- Closing or dropping the handle puts the default action back in effect: the next signal ends the process as if it had never listened.
- The handler is installed with
SA_RESTART, so most interrupted system calls resume.std.readywaits retryEINTRagainst their deadline. - Child processes started by
std.processstart with the default action:execresets caught signals, and the signal mask is not changed. A Ctrl-C at a terminal goes to the whole foreground process group, children included. - The handle holds one descriptor, opened close-on-exec. The runtime keeps the other end for the life of the process.
Stopping a server
cancel_on waits for a signal and cancels stop, returning which signal it
was. If something else cancels stop first, it returns Nothing. Run it on
its own thread beside std.http.server.serve,
whose shutdown and drain begin when stop is cancelled:
signals = std.signal.listen()?
stop = CancellationToken()?
with spawn(std.signal.cancel_on, signals, stop.share()).drop_job()? as watcher, stop.cancel_on_exit():
served = std.http.server.serve(listener, &routes, &handler, &config, &stop)
stop.cancel()
watcher.join()?
report = served?
Leaving the block joins the watcher, which waits until a signal arrives or
stop is cancelled. stop.cancel_on_exit() cancels stop on every exit, and
the explicit stop.cancel() comes before the in-block join; see
cancel on every exit. The
tutorial lesson runs a
complete program.
Limits
- x86-64 and AArch64 GNU/Linux; elsewhere
listenisStream(StreamError.Io(IoError.Unsupported)). - Only
SIGTERMandSIGINT. There is noSIGHUPreload, no other signal, and no way to catch a signal more than once. - A library compiled from Plenty does not catch signals unless its code
calls
listen; the host’s dispositions are its own.
Acknowledgements
- Uses D. J. Bernstein’s self-pipe trick, with a socket pair in place of the pipe.
- Uses the Rust
libccrate for Linux signal action and socket bindings (MIT OR Apache-2.0).
std.simd — portable explicit lanes
Use this module when an algorithm needs explicit lane operations. Start with
ordinary column pipelines for bulk clist queries: their
automatic SIMD
handles vector widths and tails without changing the source algorithm.
Values and construction
i32x4, u32x4, f32x4, and f64x2 are nominal, copyable values. Their names
describe scalar types and logical lane counts, not an instruction set. Construct
them with i32x4.Lanes(a, b, c, d) or the matching module factory, for example
splat_i32x4(value). Construction, arithmetic, comparison and selection allocate
nothing. Values can be passed, returned, nested in other values and stored in
collections under the ordinary enum rules.
lane(index) returns Option[Scalar]; indices outside 0..lane_count return
Nothing, including negative indices. to_tuple() copies the logical lanes in
order, without allocation. Ordinary == compares complete values; use the
simd_* methods below when each lane needs its own comparison result.
Construct distinct lanes, then add the same value to each with explicit wrapping:
import std.simd as simd
def main() -> Result[(), IoError]:
values = simd.u32x4.Lanes(0u32, 1u32, 2u32, u32.MAX)
print(values.wrapping_add(simd.splat_u32x4(1u32)).to_tuple())?
Ok(())
(1, 2, 3, 0)
Arithmetic and masks
| Operation | Contract |
|---|---|
add, sub, mul | Integer vectors return Result[Vector, OverflowError[Lane]]; float vectors stay infallible |
Float div | Scalar IEEE division, including infinities and NaNs |
Integer wrapping_add, wrapping_sub, wrapping_mul | Modular lane arithmetic |
Integer wrapping_sum() | Modular sum of all lanes |
Float sum_ordered(initial) | Left fold from initial, in increasing lane order |
simd_eq/ne/lt/le/gt/ge(other) | Lane comparisons returning a compatible mask |
Integer checked operations execute in lane order and return the first overflowing
lane’s scalar OverflowError, including its wrapped value. Callers handle this
ordinary Result; an unchecked: caller does not change the method’s contract. Floating operations do not implicitly fuse or
reassociate. NaN comparisons, signed zero and subnormals follow the target’s
scalar rules; NaN payload identity is not promised.
The integer reduction API is explicitly wrapping_sum(): it associates as
(((lane0 + lane1) + lane2) + lane3), wrapping each addition. There is no
ordinary fallible integer reduction method. Float sum_ordered(initial) also
folds left in lane order, without reassociation.
Comparisons on 32-bit lanes produce mask32x4; f64x2 produces mask64x2.
Masks contain logical booleans, constructed directly with Lanes(...) or by
comparison. There are no integer-mask conversions or public native bit encoding.
mask.select(yes, no) selects each lane from two vectors of the same type and
the matching mask shape. It evaluates both operands before selection: it does
not suppress side effects, allocation failures or traps while computing them.
any(), all() and count() summarize a mask. invert(), and_mask(other),
or_mask(other) and xor_mask(other) combine logical lanes.
Select a floor for the lanes below it; the input stays unchanged:
import std.simd as simd
def main() -> Result[(), IoError]:
values = simd.f32x4.Lanes(18.0f32, 21.0f32, 16.0f32, 24.0f32)
floor = simd.splat_f32x4(20.0f32)
below = values.simd_lt(floor)
print(below.count(), below.select(floor, values).to_tuple())?
Ok(())
2 (20.0, 21.0, 20.0, 24.0)
Like other generic Plenty helpers, select[T] forwards structurally to
yes.selected(mask, no). The module’s vector implementations of that helper
are private. A user-defined type may supply its own public selected method;
its effects and allocation behavior then belong to that implementation, and
the compiler does not replace it with a vector intrinsic.
Safe list access
load_i32x4(values, offset=0) and the corresponding factories for other vector
types accept a shared list borrow and return Option[Vector]. They return
Nothing unless a whole vector fits. vector.store(&mut values, offset) returns
True after replacing all lanes, or False without changing the list if the
whole vector does not fit. Negative offsets are invalid. Bounds are checked
before adding lane offsets, including for i64.MAX.
These operations neither allocate nor grow the list. They access ordinary list elements individually; lists are not packed native arrays. Returned vectors own copies of their scalar lanes, so no borrow persists after loading. Stores take an exclusive borrow and preserve length and capacity. Process full groups and then the scalar tail explicitly when writing a manual kernel.
Handle a load that may not fit. A failed store leaves every element unchanged:
import std.simd as simd
def main() -> Result[(), AllocError | IoError | OverflowError[i32]]:
mut values = [1i32, 2i32, 3i32, 4i32]?
match simd.load_i32x4(values):
case Some(lanes):
print(lanes.store(&mut values, 1))?
print(lanes.add(simd.splat_i32x4(10i32))?.store(&mut values, 0))?
case Nothing:
print("A full vector does not fit")?
print(values)?
Ok(())
False
True
[11, 12, 13, 14]
Rust relationship and initial limits
The model follows Rust’s portable std::simd: fixed logical lanes, splats,
per-lane arithmetic and simd_* comparisons, masks, and selection. This first
Plenty module has concrete types rather than Simd[T, N] and uses methods rather
than overloaded arithmetic operators. Enum factories use module functions and
the Lanes variant because Plenty has no static enum methods. Unlike Rust’s
wrapping integer operators, add/sub/mul preserve Plenty’s checked arithmetic;
modular operations have explicit wrapping_* names.
The implementation uses existing inline enums, whose payload stores each lane in a tagged scalar slot. A four-lane payload currently occupies 64 bytes, plus the ordinary value handle; this is not a packed 16-byte vector or a native register ABI. Mask variants expose logical booleans, not hardware masks. Neither vector nor mask layout is a C, serialization, Arrow or Parquet contract.
On x86-64, the default optimized profile lowers wrapping arithmetic, float arithmetic, comparisons
and selection to baseline 128-bit SIMD. Integer wrapping_sum reduces lanes
directly without calling the scalar module body. Numeric vector locals without mutable
borrows stay in vector registers across supported operations and loops. Values
materialize in ordinary tagged storage at calls, containers and observable
borrows; parameters, masks, generators and mutable-borrowed locals retain that
storage path. This is an internal optimization, not a different ownership rule.
Checked integer arithmetic, mask summaries, ordered float reductions and list access keep
their ordinary scalar implementations. On other compiler hosts, the module’s
scalar bodies remain available. The AArch64 implementation currently uses these
scalar bodies, including in optimized builds; ARM vector acceleration is separate
work.
The same source semantics apply with and without optimization. No runtime ISA
selection or newer CPU feature is required.
Storage boundaries, copying and lane assembly can dominate short operations.
Explicit SIMD is not a blanket speedup over scalar code, and this module does
not replace the packed-column lowering used by clist. Wider vectors, packed views, generic
lane counts, gathers/scatters, casts, shifts and integer division are not in this
initial surface. See the tutorial
and the implementation evidence.
Acknowledgements
- Design follows Rust’s portable
std::simd.
std.sync
Exports CancellationToken, the existing
cooperative cancellation facility,
and Mutex[T]. The imported and prelude CancellationToken types are
identical; no wrapper, allocation, or new effect permission is introduced by
importing. Mutex is not a prelude name: each use needs an import, through any
item or module alias. Supported target: x86_64 Linux GNU.
CancellationToken() -> Result[CancellationToken, AllocError] creates an owned
shared state in one fallible allocation.
| Method | Result | Effect |
|---|---|---|
share() | CancellationToken | Explicitly retain the same state |
cancel() | () | Permanently request cancellation and wake token waiters |
is_cancelled() | bool | Observe the request |
wait() | () | Wait until cancellation is requested |
wait_timeout_ms(timeout_ms: u64) | bool | Observe cancellation within the budget |
cancel_on_exit() | with guard | Cancel on every exit from the with block; see cancel on every exit |
Methods borrow the handle, allocate nothing, and copy no captured data. Share and observation have constant bookkeeping; cancellation also wakes waiters. Lock contention and scheduling have no bounded latency. Tokens are safe to move or explicitly share into eligible workers; borrowed handles retain ordinary loans. Dropping the last handle releases the state but does not request stopping. Cancellation is idempotent and does not reset.
A token does not interrupt instructions, cancel a future, or wake a channel
wait. Workers must return cooperatively, using bounded waits where needed.
std.time supplies deadline arithmetic and timeout conversion.
Typed deadline waits
Prefer these waits when a caller supplies a total budget and cancellation.
They borrow the handles, allocate nothing, and preserve the deadline across
wakeups. deadline and cancel are the last two parameters.
| Operation | Result |
|---|---|
recv_before(&receiver, &deadline, &cancel) | Result[T, WaitError] |
send_before(&sender, value, &deadline, &cancel) | Result[(), SendBeforeError[T]] |
select_recv_before(&first, &second, &deadline, &cancel) | Result[Selected[A, B], WaitError] |
future_before(&future, &deadline, &cancel) | Result[(), WaitError] |
cancelled_before(&token, &deadline, &cancel) | Result[(), WaitError] |
WaitError has TimedOut, Cancelled and Disconnected (channels only).
Every SendBeforeError[T] variant (TimedOut, Cancelled, Disconnected)
returns the original unsent value, without copying it. Cancellation visible
before an attempt wins even over readiness. Otherwise an already-ready value
or disconnection wins over an expired deadline. Selection keeps the builtin’s
first-receiver priority. Future observation leaves its result owned by the
future and does not cancel its worker. Token observation leaves its state intact.
Channel and future primitives cannot wake on a separate token, so these wrappers sleep in slices of at most 10 ms, rechecking cancellation between slices. This is a polling interval, not a hard latency guarantee under scheduler or lock contention. Positive fractional milliseconds round up when passed to the native wait, so a final slice can overrun by less than 1 ms. No deadline is restarted.
Low-level millisecond methods have explicit unit names: send_timeout_ms,
recv_timeout_ms, wait_timeout_ms on futures and tokens, and
select_recv_timeout_ms. The original _timeout spellings remain compatibility
aliases with the same millisecond units. They do not accept a separate token.
Mutex[T]
A value that workers update through shared references or
Shared[Mutex[T]] handles, one lock
holder at a time. See mutexes for the rules
and the deadlocks the compiler rejects.
from std.sync import Mutex
class Stats:
count: i64
total: i64
def record(stats: &Mutex[Stats], value: i64) -> Result[(), OverflowError[i64]]:
with stats.lock() as data:
data.count = data.count + 1
data.total = data.total + value
Ok(())
def main() -> Result[(), SpawnError[()] | IoError | OverflowError[i64]]:
stats = Mutex[Stats](Stats(0, 0))
with spawn(record, &stats, 3)? as first, spawn(record, &stats, 4)? as second:
first.join()?
second.join()?
final = stats.into_inner()
print(final.total)?
Ok(())
7
The explicit joins handle each worker’s arithmetic Result; merely leaving a spawn scope waits for completion but does not propagate the worker’s error. An arithmetic failure unlocks the mutex on exit, retaining earlier mutations.
| Operation | Result | Effect |
|---|---|---|
Mutex[T](value), Mutex(value) | Mutex[T] | Store value unlocked, inline |
with mutex.lock() as value: | binds value: &mut T | Wait for the lock, hold it for the block |
mutex.into_inner() | T | Consume an owned mutex and return its value |
T is any type a generic class field can hold: no references, generators, or
borrowed environments. A mutex is affine, with no copy, equality, or
formatting. It is stored inline (a 32-bit state word beside T); construction,
locking, and unlocking allocate nothing. An uncontended lock and unlock are
inline atomic instructions; a contended one waits in the kernel.
lock() is valid only as a with manager. It borrows the mutex shared, so no
mut is needed, and every exit from the block unlocks. There is no
try_lock, timed lock, poisoning, reentrant locking, or fairness guarantee.
into_inner() needs no lock: owning the mutex proves no other thread can hold
it.
Atomic integers and booleans are in std.atomic. The shared
owner Shared[T] is a prelude type.
Semaphores, condition variables, and read-write locks are not exported.
Acknowledgements
Mutexuses mutex 2 from Ulrich Drepper’s “Futexes Are Tricky”.- Uses the Rust
libccrate for the target-specific Linux futex syscall number (MIT OR Apache-2.0).
std.system
Bounded Linux system-information snapshots, implemented in Plenty over
std.fs and std.io. No subprocess, background sampler,
native dependency, or global cache is involved.
local() -> Result[Procfs, IoError]
open_procfs(path: &std.path.Path) -> Result[Procfs, IoError]
local() opens /proc. open_procfs opens another procfs mount or a directory
of captured files; this also supports deterministic tests. Procfs owns a
directory handle and closes it on drop. It resolves reads relative to that
handle, following the rooted filesystem API’s no-follow rules.
import std.system
from std.path import from_text
from std.system import SystemError
def main() -> Result[(), IoError | SystemError]:
# A captured procfs file gives fixed values; std.system.local() reads /proc.
write_text("loadavg", "0.50 0.25 0.10 2/180 4242\n")?
source = std.system.open_procfs(&from_text(".")?)?
load = source.load_average()?
print(load.one_minute, load.runnable_tasks, load.last_pid)?
live = std.system.local()?
print("CPUs found:", live.cpu_count()? > 0)?
Ok(())
0.5 2 4242
CPUs found: True
Queries
Every method returns Result[T, SystemError]. Calls read fresh data; two calls
are not one atomic snapshot. Limits are named optional arguments.
| Method | Result | Source | Defaults |
|---|---|---|---|
cpu_count() | i64 online logical CPU records | stat | max_bytes=1048576 |
memory() | Memory | meminfo | max_bytes=65536 |
load_average() | LoadAverage | loadavg | max_bytes=4096 |
uptime() | Uptime | uptime | max_bytes=4096 |
block_devices() | list[BlockDevice] | partitions | max_bytes=1048576, max_devices=4096 |
disk_io() | list[DiskIo] | diskstats | max_bytes=1048576, max_devices=4096 |
process_ids() | sorted list[u32] | numeric directory entries | max_entries=65536 |
process(pid: u32) | Process | PID/status | max_bytes=65536 |
The generated signature index lists every public record field. For example:
source = std.system.local()?
memory = source.memory(max_bytes=65536)?
print(memory.total_bytes, memory.available_bytes)?
Units and meaning
Memoryreports total/free physical bytes, optional available physical bytes, and total/free swap bytes.MemAvailableis a kernel estimate;Nothingmeans the field is unavailable, not zero. Free memory excludes reclaimable caches and is usually not the useful estimate of allocation headroom.LoadAveragehas 1-, 5-, and 15-minute load values, runnable/total task counts, and the last created PID. Load is not CPU-utilization percent.Uptime.secondsis elapsed boot time, including suspend according to procfs.idle_cpu_secondssums idle time over CPUs and can exceed elapsed time.BlockDevicegives major/minor device numbers, a name, andsize_bytes. Whole disks and partitions can appear; summing them double-counts storage./proc/partitionsreports 1024-byte blocks, so capacity has that granularity. This is not mounted filesystem capacity or free space.DiskIogives reads/writes completed, bytes read/written, currently in-flight requests, and busy milliseconds. Sector counts are converted using Linux’s fixed 512-byte accounting unit. These are raw counters, not rates. Counters can reset on reboot/device replacement or wrap; detect resets before computing differences. Whole disks and partitions overlap here too. New trailing diskstats fields are ignored; the modern eleven-counter prefix is required.Processgives PID, parent PID, the kernel-reported name and state code, real UID, thread count, and optional virtual/resident bytes. Name escaping fromstatusis preserved, including literal backslash escapes; it is not a command line or executable path. Missing memory fields, for example on kernel threads, yieldNothing. Resident memory is an approximate kernel counter.
The CPU count is not an affinity/cgroup-aware worker-pool size. Memory, load, uptime and devices describe what the mounted files expose, often the host even in a container. PIDs follow the mount’s PID namespace and permissions. This API does not infer a container’s effective CPU or memory quota.
Limits, failures, and races
max_bytes is a Positive[i64]; i64.MAX traps, as does process with pid
0. Entry and device bounds are NonNegative[i64]. A byte-limited read probes one extra byte to distinguish
exact-limit EOF from truncation, since proc files often report size zero.
Exceeding any bound returns Limit, never a partial successful result.
process_ids counts all visited directory entries toward its bound,
including non-PID names. Parsing and result storage allocate fallibly; byte
limits bound input size, not exact peak memory.
enum SystemError:
Io(IoError)
Stream(std.io.StreamError)
Allocation(AllocError)
Limit
Malformed(str)
I/O errors keep their original cause, including missing and inaccessible files.
Allocation errors can also occur inside the existing Io/Stream envelopes.
Invalid UTF-8, missing required fields, duplicate selected keys, invalid numbers
and unit-conversion overflow return Malformed(field_or_source). Unknown
meminfo/status keys are ignored. Numeric conversions never truncate or wrap.
Scaling checks the full u64 product and preserves the field-specific
Malformed error on overflow; counts and byte-limit probe offsets are bounded
by their validated limits rather than saturated.
Enumerating PIDs does not read each process’s status or silently skip failures.
Call process(pid) for the records wanted, choosing how to handle disappearance
or access denial. It opens the PID directory once before reading status, avoiding
mixing files from a reused path; the process may still exit and the read may fail.
A returned PID is neither a stable identity nor authority to signal a process.
This first interface has no CPU-utilization sampling, filesystem free-space
query, command-line decoding, cgroup accounting, or process control. For children
owned by the program, use std.process.
Formats follow the kernel’s procfs documentation, disk I/O accounting, and partitions ABI. See the tutorial.
std.tempfile
Owned temporary files and directories, with automatic cleanup and explicit
fallible cleanup methods. Prefer this module for scratch storage; use
std.fs for descriptor-rooted staging and atomic publication.
import std.fs
import std.path as path
import std.tempfile as tempfile
def main() -> Result[(), AllocError | IoError | std.fs.FsError]:
parent = path.from_text(".")?
with tempfile.directory(&parent, prefix="build-")? as scratch:
note = scratch.path().join(&path.from_text("note.txt")?)?
write_text(¬e.to_text()?, "draft")?
print(read_text(¬e.to_text()?)?)?
scratch.cleanup()?
print(std.fs.metadata(scratch.path()).is_err())?
Ok(())
draft
True
Constructors
| Function | Result | Defaults |
|---|---|---|
default_dir() | Result[Path, IoError] | First nonempty TMPDIR, TEMP, TMP; otherwise /tmp |
file(parent: &Path = default_dir()?) | Result[TemporaryFile, IoError] | Unnamed file |
named_file(parent, prefix="plenty-", suffix="") | Result[NamedTemporaryFile, FsError] | Parent is default_dir() |
directory(parent, prefix="plenty-", max_entries: Positive[i64] = 100000, max_depth: Positive[i64] = 64) | Result[TemporaryDirectory, FsError] | Parent is default_dir() |
parent is a borrowed Path; prefix and suffix are borrowed strings. Explicit
parents bypass environment selection. Relative parents are resolved against the
current directory. Environment paths preserve native bytes. Selection is fresh
on every call, without caching or a trial file: an unusable configured directory
causes construction to fail, rather than silently falling back elsewhere.
Named resources use 16 random bytes encoded as 32 hexadecimal characters, between
the prefix and suffix. Prefix and suffix together may occupy at most 200 UTF-8
bytes and must contain neither / nor NUL. These are byte limits, not scalar
counts; invalid components or a combined length above 200 return
FsError.InvalidInput before creating a resource. Directories accept only a prefix.
Creation retries up to 16 name collisions. Files are created with owner-only
permissions, directories with owner-only access (subject to the process umask).
No API returns an uncreated, supposedly unique filename.
Files
TemporaryFile.stream() lends a mutable std.io.FileStream. Its context manager
also yields that stream. close() returns Result[(), StreamError]; dropping the
owner closes the descriptor. The file has no pathname.
NamedTemporaryFile.path() returns a borrowed absolute Path, usable by another
program while the owner is alive. The file supports read, write, flush,
sync, and seek with the same signatures as FileStream, and works with
std.io reader/writer helpers. Its context manager yields the named
owner itself. The underlying descriptor stays private for cleanup identity checks.
close() returns Result[(), FsError]: it removes the name, then closes the
stream. An already removed name is accepted. A different file at that name is
rejected; a failed removal leaves the owner armed for another cleanup attempt.
keep() returns Result[Path, AllocError] and disarms removal after successfully
copying the path. The stream remains open until closed or dropped. Call keep()
before cleanup; it does not recreate an already removed file.
Directories
TemporaryDirectory.path() returns a borrowed absolute Path; root() borrows
its descriptor-rooted std.fs.Root. Its context manager yields the directory
owner. cleanup() returns Result[(), FsError] and recursively removes contents
and the directory. Successful cleanup is idempotent. keep() returns an owned
path and disarms cleanup, allocating the path before changing ownership policy.
Each cleanup attempt is bounded by max_entries and max_depth, both
Positive[i64]; directory traps on a max_depth above 64. The root directory counts toward both budgets. A limit or I/O failure
can leave a partially removed tree; cleanup remains armed for a later attempt.
Symlinks are removed without traversing their targets. Before traversal, the
parent entry must still identify the owned directory; otherwise cleanup returns
FsError.Replaced without traversing the replacement.
Cleanup and platform limits
Context exits and destructors perform best-effort cleanup. Use explicit
close()? or cleanup()? inside the context when failure must be reported.
Normal return and ? propagation run cleanup; process termination and runtime
traps do not promise destructor execution. Cleanup itself can fail to allocate.
These APIs currently use the Linux facilities of std.fs: unnamed and named
files require filesystem O_TMPFILE support; named publication additionally
requires /proc/self/fd. There is no fallback that exposes an intermediate
predictable name. Memory-to-disk spooling is not provided.
Parents must be trusted against concurrent hostile mutation. Identity checks detect replaced entries, but are not atomic with deletion; they do not make cleanup safe against an attacker racing renames. Named files may also be removed by external temporary-directory cleaners. Keep the owner alive while another program uses its path.
See the temporary-resource tutorial for complete examples.
Acknowledgements
std.test
import std.test provides assertions, fixtures, and test doubles for the
tests that plenty test runs,
with one error type, TestError. The module is checked Plenty code. Any
program may import it; it depends on the runner only for reading
PLENTY_TEST_SEED and for failure injection.
A test file with main passes when it exits with status zero, so a plain
program can run assertions:
import std.test
from std.test import TestError
def test_split() -> Result[(), AllocError | TestError]:
parts = "a,b,,c".split(",")?
std.test.equal(&len(parts), &4)?
std.test.equal(&parts[2], &"")?
std.test.check(parts[0] < parts[1], "parts keep their order")?
Ok(())
def main() -> Result[(), AllocError | IoError | TestError]:
test_split()?
print("test_split passed")?
Ok(())
test_split passed
Assertions
An assertion that holds returns Ok(()) without allocating. One that fails
flushes standard output, writes the position of the call, assertion failed: ,
and an explanation to standard error, and returns Err(TestError.Assertion).
The explanation is written at once, so it survives propagation or handling by
the caller. Tests can return TestError or a sum containing it.
parse.plenty:12:5: assertion failed: left == right
left: 4
right: 5
The file is relative to its source root, tests/ for a test file, as in
trap reports.
The position is the user code that
called the assertion, even through
standard-library helpers: the
caller location
that trap notes use. A user helper that wraps an assertion reports the helper’s
line, not its caller’s.
| Function | Passes when | Failure explanation |
|---|---|---|
fail(message: &str) | never | message |
check(condition: bool, message: &str) | condition | message |
equal[T](left: &T, right: &T) | left == right | left == right, then left: and right: lines |
not_equal[T](left: &T, right: &T) | left != right | left != right, then a both: line |
less[T](left: &T, right: &T) | left < right | left < right, then left and right lines |
close(left: f64, right: f64, tolerance: f64) | left == right or abs(left - right) <= tolerance | the three values; NaN on either side fails, equal infinities pass |
is_ok[T, E](result: &Result[T, E]) | result is Ok | result is Ok, then an error: line |
is_err[T, E](result: &Result[T, E]) | result is Err | result is Err, then a value: line |
Values are shown with str.repr, so T must support equality or ordering as
the function needs. A value whose type str.repr rejects, such as a
Shared[T] owner or a std.sync.Mutex,
is shown as <value cannot be formatted>; is_ok and is_err therefore accept
every Result, whatever the branch they do not expect holds. Showing a value
allocates; when that fails the line reads <no memory to show the value>. A
failure to write to standard error is ignored.
Under plenty test, a test that returns Err(TestError.Assertion), or the
Failure.Unspecified that ? makes of it, gets no further error: line: the
explanation above is the whole report.
Fixtures
| Function | Result | Contract |
|---|---|---|
scratch() | Result[Scratch, TestError] | A new empty directory named plenty-test- plus 32 random hex digits, created with mode 0700 under $TMPDIR when it is an absolute path, otherwise under /tmp |
seed() | Result[u64, TestError] | PLENTY_TEST_SEED as a decimal u64, or 0 when it is unset; any other value writes a message to standard error and returns TestError.InvalidSeed |
rng() | Result[Rng, TestError] | std.random.seeded(seed()?) |
plenty test --seed N sets PLENTY_TEST_SEED for every test it runs, so a
failure seen under one seed repeats under it.
Scratch owns its directory:
| Method | Result | Contract |
|---|---|---|
path() | &Path | The directory’s absolute path |
root() | &Root | A std.fs.Root anchored at the directory |
file(name: &str) | Result[str, TestError] | The absolute path text of name inside the directory, for builtins such as write_text and open |
cleanup() | Result[(), TestError] | Remove the directory and everything in it; later calls do nothing. Takes &mut Scratch |
Dropping a Scratch runs cleanup and ignores its error. Removal uses
std.fs.remove_tree with budgets of 100000 entries and depth 64;
beyond them cleanup reports TestError.Fixture(FsError.Limit) and leaves the
rest in place. A process that traps or is killed skips destructors, so its
scratch directory remains, except under plenty test, which gives each test
its own TMPDIR and removes it when the test ends.
Test doubles
Code that reads the clock or moves bytes can take its source as a protocol bound, so a test passes a double and a program passes the real thing. Calls are resolved at compile time; a double costs nothing in the program that does not use it.
fake_clock(nanoseconds_after_origin: u64) returns a FakeClock, a
std.time.Clock whose time moves only when the test moves it.
Its instants are on the instant_at timeline.
| Method | Result | Contract |
|---|---|---|
now() | Instant | The current fake instant |
advance(span: &Duration) | Result[(), RangeError] | Move forward by span; past the end of the timeline is RangeError.OutOfRange and nothing moves. Takes &mut FakeClock |
set(nanoseconds_after_origin: u64) | () | Move to any instant, including backwards. Takes &mut FakeClock |
fake_stream(input: Bytes) returns Result[FakeStream, AllocError], a
std.io Reader and Writer over memory. Reads
return input from its start; writes append to output. Both fields are
public. It injects the partial progress and failures that a file or socket
produces only under load:
| Method | Result | Contract |
|---|---|---|
limit(bytes: i64) | () | Each later read or write moves at most bytes bytes; 0 or less removes the limit |
fail_after(after: i64, error: StreamError) | Result[(), OverflowError[i64]] | The call after calls from now (0 is the next) returns Err(error) and moves nothing. One failure is pending at a time; a later request replaces it |
calls() | i64 | read, write, and flush calls made so far, including failed ones |
read, write, flush | as std.io | The stream protocol methods |
A range outside its buffer traps with read: start/stop outside buffer before the call is
counted. flush does nothing but count and fail when told. Writing allocates
as output grows; that failure is StreamError.Allocation.
fail_after checks the absolute call index before replacing the pending fault.
On overflow it returns the wrapped OverflowError[i64] and changes neither
schedule nor counter; callers must handle its result. read, write and
flush return StreamError.OutOfRange without changing data, counter or pending
fault if the call counter cannot advance. No caller arithmetic mode changes
these boundaries. Existing signed offsets are retained: a representable past
index does not match a future call.
from std.time import Clock, Instant
def overdue[C: Clock](clock: &C, since: &Instant) -> bool:
clock.now().saturating_duration_since(since).as_seconds() > 30
def test_overdue() -> Result[(), std.test.TestError | RangeError]:
mut clock = std.test.fake_clock(0u64)
since = clock.now()
std.test.check(not overdue(&clock, &since), "fresh")?
clock.advance(&std.time.seconds(31u32))?
std.test.check(overdue(&clock, &since), "after 31s")?
Ok(())
A program calls the same function with std.time.monotonic_clock().
Failure injection
Executables that plenty test builds, or that are compiled with --test-hooks,
can make the C library’s allocation and
file calls fail, so a test reaches the error paths of code that allocates or
does file I/O. Injection belongs to the calling thread.
| Function | Effect |
|---|---|
fail_allocations_after(count: u32) | After count more allocations succeed, each fails until restored: the operation reports AllocError.OutOfMemory |
restore_allocations() | Allocations succeed again |
fail_files_after(count: u32, code: i32) | After count more file calls succeed, each fails with errno code until restored: the operation reports IoError.System(code) |
restore_files() | File calls succeed again |
All four return Result[(), TestError]. A later call replaces the budget.
Counted file calls are opening any path, and reading, writing, syncing, or
truncating a regular file; standard streams, pipes, and sockets never fail, so
a failing test can still report. Other filesystem calls, such as renaming or
listing a directory, are not counted. How many calls an operation makes is not
part of its contract, so a test should fail from the first call (count 0) or
check only that some call failed. A failure that ? propagates leaves the
budget in place; restore it before an assertion that must allocate to show its
values.
def test_save_reports_a_full_disk() -> Result[(), std.test.TestError]:
scratch = std.test.scratch()?
name = scratch.file("state.txt")?
std.test.fail_files_after(0u32, 28i32)?
saved = write_text(&name, "state")
std.test.restore_files()?
full: Result[i64, IoError] = Err(IoError.System(28i32))
std.test.equal(&saved, &full)?
Ok(())
Use plenty --compile server.plenty --test-hooks -o server to exercise a real
program’s refusal paths from an external client. The program can read its own
test configuration and arm a budget immediately before admission, then restore
it before reporting the result. plenty build --test-hooks and
plenty run --test-hooks enable the same controls for project executables.
This also enables file failure injection. The flag does not arm any failures
by itself. Keep it out of production builds.
Without this opt-in or plenty test, these functions write
error: failure injection needs plenty test or --test-hooks to standard error
and return Err(TestError.Unavailable). Default executables contain no
interposition code. Project and compiler caches distinguish instrumented builds
from default builds.
Memory-limited integration runs
On Linux, playpen can bound the memory of the real executable and its descendants through the systemd user manager:
playpen --memory-limit 128M --memory-swap-max 0 --private-tmp false -- ./server
Build before launching, so the limit measures the program rather than the compiler. Disabling the private temporary directory lets a test driver share temporary fixtures with the program. Use a suitable limit for the workload and check both the exit status and the application’s expected response.
The cgroup limit is an integration resource ceiling: exhausting it can kill
the process instead of returning AllocError. Use --test-hooks and
fail_allocations_after for deterministic error-path checks, including under
playpen. The budget is a count of allocation attempts, not bytes or a live-heap
limit; it cannot measure peak memory or prove a program stays below a byte
budget. Use an external memory profiler for those measurements.
Errors
| Variant | Meaning |
|---|---|
Assertion | An assertion failed; its explanation is already on standard error |
Fixture(FsError) | Creating or removing a scratch directory failed, or a name or path was invalid |
Allocation(AllocError) | Memory for a fixture ran out |
InvalidSeed | PLENTY_TEST_SEED is set but is not a decimal u64 |
Unavailable | Failure injection was requested without plenty test or --test-hooks |
Limits
- Injected failures cover allocation and the file calls listed above, on the
calling thread only. Allocation on other threads, such as executor workers,
and network calls are not injected; use
FakeStreamat the stream boundary. - Calls the C library makes internally, such as
fopeninside another C function, are not intercepted; only the program’s and runtime’s own calls are.
std.text
Bounded reusable text construction, joining, scalar wrapping, and byte-set
scans. Word wrapping and indentation add Python-style
paragraph tools. Import std.text; every allocating operation returns
Result[..., TextError].
| Operation | Behavior |
|---|---|
builder(max_bytes: NonNegative[i64]) | New empty Builder |
Builder.byte_len(), Builder.limit() | Current UTF-8 byte count and configured maximum |
Builder.append(text: &str) | Append a copy, or leave contents unchanged on failure |
Builder.clear() | Empty the builder, retaining its allocation |
Builder.to_text() | Independent UTF-8 string snapshot; builder remains reusable |
Builder.finish() | Final UTF-8 string; consumes the builder on success or failure |
join(pieces: &list[str], separator: &str, max_bytes: NonNegative[i64]) | Separator between pieces, including empty pieces; empty list yields empty text |
hard_wrap(text: &str, width: Positive[i64], max_bytes: NonNegative[i64]) | Hard wrap at width Unicode scalars, inserting LF |
find_in(text: &str, set: &[u8]) | Option[i64]: scalar position of the first ASCII scalar whose byte is in set |
find_not_in(text: &str, set: &[u8]) | Option[i64]: scalar position of the first scalar that is not such a member: the end of a run |
TextError has InvalidLimit, LimitExceeded, Allocation(AllocError), and
Bytes(BytesError) variants. Limits and widths are domain types, so
InvalidLimit only reports wrapper indentation or a
placeholder that cannot fit. A literal converts where a limit is expected; a
computed value needs NonNegative.try_from(value)?. Byte-buffer failures preserve their typed cause;
normalization uses the direct allocation variant. Limits apply to output bytes,
not characters. Appending checks the complete input size without allocating,
then appends atomically. Appends of at most four UTF-8 bytes reserve capacity
before writing directly, without a temporary encoding; larger appends encode a
temporary buffer. Empty appends allocate nothing. Invalid UTF-8 must be
handled at byte decoding; str is always valid UTF-8. Embedded NUL
is preserved. Public builder operations preserve complete UTF-8.
Hard wrapping preserves every input scalar and existing LF, including final LF.
It never inserts an extra LF before an existing LF. It does not collapse spaces,
find word boundaries, calculate terminal display width, or keep grapheme clusters
together: a combining mark can begin a new line. Width and string positions are
scalar counts; the output limit and byte_len are byte counts.
Segmentation and display width
The official unicode_text
package provides UAX #29 grapheme
cluster, word and sentence boundaries, and terminal display width following the
Rust unicode-width crate, with Unicode 17.0.0 data matching
std.unicode. It is plain Plenty; add it as a
git dependency:
[dependencies]
unicode_text = { git = "https://github.com/plentylang/unicode-text.git", tag = "v0.1.0" }
import unicode_text
from unicode_text import Ambiguous
def main() -> Result[(), AllocError | IoError]:
text = "Hi 👩🔬! 日本語のテキスト。"
print(unicode_text.grapheme_list(&text)?)?
print(unicode_text.word_list(&text)?)?
for span in unicode_text.sentences(text):
print(span.start, span.stop)?
print(unicode_text.width(&text, Ambiguous.Narrow))?
print(unicode_text.width("±1", Ambiguous.Wide))?
Ok(())
["H", "i", " ", "👩🔬", "!", " ", "日", "本", "語", "の", "テ", "キ", "ス", "ト", "。"]
["Hi", "日", "本", "語", "の", "テキスト"]
0 8
8 17
25
3
graphemes, word_bounds, words and sentences are generators of Span
values carrying scalar positions and UTF-8 byte offsets, and allocate nothing;
the *_list forms return Result[list[str], AllocError]. width takes an
explicit Ambiguous.Narrow or Ambiguous.Wide for East Asian Ambiguous
characters and, like unicode-width, counts each control scalar as one column;
scalar_width returns Nothing for a control. The package README lists the
complete API, policies and data update procedure.
find_in and find_not_in are the str forms of the
Bytes scans: set is a Bytes buffer or a bytes
literal, borrowed, and the whole text is scanned in one native call that
allocates nothing. Only ASCII scalars can be members; set bytes from 0x80
match nothing, so a non-ASCII scalar never matches find_in and always stops
find_not_in. Results are scalar positions, as for str.find, and there are no
bounds arguments. Empty text gives Nothing; an empty set makes find_in
Nothing and find_not_in Some(0) for nonempty text. Use them to validate a
character class without a per-scalar loop:
import std.text
def token(text: &str) -> bool:
text.byte_len() > 0 and std.text.find_not_in(text, b"!#$%&'*+-.^_`|~0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz").is_none()
def main() -> Result[(), AllocError | IoError]:
print(token("GET"), token("G T"), token("GÉT"), std.text.find_in("a=b", b"="))?
Ok(())
True False False Option[i64].Some(1)
find_in costs O(text bytes), plus counting the scalars before a match;
find_not_in costs O(bytes up to the first non-member).
The builder owns a compact Bytes buffer. finish(self: Builder) consumes it;
use to_text if the builder must survive an allocation failure for retry or
remain available for reuse. Both operations currently copy the bytes into a
new string; finish does not promise zero-copy conversion. Other methods borrow it; append and
clear require exclusive access, and ordinary loan checking rejects mutation
while shared access remains live. There is no global mutable state or thread
affinity. Appending takes O(input bytes) amortized time; inputs longer than four
UTF-8 bytes need a temporary encoding. Small appends reuse spare capacity, and
reserve geometrically when they grow; growth can temporarily retain old and replacement allocations.
Storage capacity grows geometrically and may exceed the logical byte limit;
the limit is not an exact resident-memory quota. to_text copies O(output bytes).
Joining and hard wrapping take O(input + output bytes) amortized time, plus output
storage. Hard wrapping writes scalars directly into geometrically grown storage.
Algorithms are checked Plenty using std.bytes; no new OS backend is required.
Sources and storage primitives are bundled for offline compilation with a
relocated matching toolchain on x86_64 Linux GNU. See the
runnable lesson.
See word wrapping and indentation for wrap, fill,
shorten, dedent, indent, indent_with, and reusable TextWrapper.
Internal scalar and UTF-8 cursors use explicit arithmetic opt-outs justified
by actual input spans and checked builder budgets. Hyphen scanning clamps
lookahead to the input and adds an ASCII run’s length to the byte cursor,
rather than forming a potentially overflowing sum of unrelated offsets.
Width/placeholder validation and input/output limits retain their existing
TextError contracts; caller arithmetic modes do not change them.
Acknowledgements
-
Builderis modelled on Go’sstrings.Builder;joinfollows Python’sstr.join. -
Python’s
textwrapsupplies the paragraph algorithms and API model; adapted CPython code is PSF-2.0, credited on the paragraph tools page.
Word wrapping and indentation
The paragraph tools in std.text follow Python’s
textwrap. They operate on
owned UTF-8 strings through borrowed inputs, return independent results, and
report failures through TextError. wrap finds word boundaries and returns
lines; hard_wrap inserts literal line breaks at scalar counts.
Quick use
import std.text
def main() -> Result[(), IoError | std.text.TextError]:
print(std.text.wrap("Hello there, how are you?", 12)?)?
print(std.text.fill("Hello there, how are you?", 12)?)?
print(std.text.shorten("Hello world!", 11)?)?
print(std.text.indent(&std.text.dedent(" first\n second")?, "> ")?)?
Ok(())
["Hello there,", "how are you?"]
Hello there,
how are you?
Hello [...]
> first
> second
| Function | Result and policy |
|---|---|
wrap(text, width=70, max_bytes=1048576, max_input_bytes=1048576) | list[str], without appended line terminators |
fill(text, width=70, max_bytes=1048576, max_input_bytes=1048576) | The same lines joined with LF |
shorten(text, width, placeholder=" [...]", max_bytes=1048576, max_input_bytes=1048576, fix_sentence_endings=False, break_long_words=True, break_on_hyphens=True) | Collapse Unicode whitespace, then keep enough whole chunks to fit one line with the placeholder when truncated |
dedent(text, max_bytes=1048576, max_input_bytes=1048576) | Remove a common literal space/tab prefix from nonblank LF-delimited lines |
indent(text, prefix, max_bytes=1048576, max_input_bytes=1048576) | Prefix each nonblank line, retaining original line endings |
indent_with(text, prefix, predicate, max_bytes=1048576, max_input_bytes=1048576) | Prefix lines selected by a Callable[[&str], bool] |
Text and prefix arguments are borrowed &str. Boolean options on shorten
are independent flags; pass changed flags by name. shorten first collapses
runs of Unicode White_Space to single ASCII spaces and trims the ends. Its
wrapping and truncation then follow the same rules as a one-line wrapper;
turning off long-word breaking does not disable the width used for truncation.
dedent ignores whitespace-only lines when finding the common margin and
normalizes their bodies to empty strings. Tabs and spaces are different
characters; a tab is not expanded to a number of spaces. It preserves LF
positions and the rest of nonblank lines, including their CR characters.
indent uses the builtin splitlines rules,
including CRLF and Unicode line separators. A final terminator does not create
an extra line. The predicate sees each full line, including its terminator,
and is invoked once per line, in order. To indent even blank lines:
import std.text
def main() -> Result[(), AllocError | IoError | std.text.TextError]:
result = std.text.indent_with("one\n\ntwo\n", "+ ", def(line: &str) -> bool: True)?
print(str.repr(result)?)?
Ok(())
"+ one\n+ \n+ two\n"
Reusable configuration
text_wrapper(width=70, max_bytes=1048576, max_input_bytes=1048576) returns a
TextWrapper without allocating. Its public fields below may be changed
between calls. wrap(text) and fill(text) borrow both the wrapper and input;
they retain neither results nor input and do not modify configuration.
| Field | Default | Meaning |
|---|---|---|
width: Positive[i64] | 70 | Line width in Unicode scalars, including indentation |
initial_indent: str | "" | Prefix on the first produced line |
subsequent_indent: str | "" | Prefix on other lines |
expand_tabs: bool | True | Expand tabs to tab stops before wrapping |
tabsize: NonNegative[i64] | 8 | Tab stop interval; zero deletes tabs when expansion is enabled |
replace_whitespace: bool | True | Replace ASCII tab/LF/VT/FF/CR with spaces after expansion |
fix_sentence_endings: bool | False | Change a single inter-chunk space after an ASCII lowercase sentence ending to two spaces |
break_long_words: bool | True | Split chunks that cannot fit on a line |
drop_whitespace: bool | True | Discard whitespace chunks at line ends and at the start of subsequent lines |
break_on_hyphens: bool | True | Prefer compound-hyphen and double-dash boundaries |
max_lines: Option[Positive[i64]] | Nothing | Optional truncation limit, with a placeholder on the last line |
placeholder: str | " [...]" | Truncation suffix; leading whitespace is removed when it must stand alone |
max_bytes: NonNegative[i64] | 1048576 | Total UTF-8 output bytes |
max_input_bytes: NonNegative[i64] | 1048576 | Input and expanded intermediate byte limit |
import std.text
def main() -> Result[(), IoError | std.text.TextError]:
mut wrapper = std.text.text_wrapper(16)
wrapper.initial_indent = "* "
wrapper.subsequent_indent = " "
print(wrapper.fill("one two three four five")?)?
wrapper.max_lines = Some(1)
wrapper.placeholder = "..."
print(wrapper.fill("one two three four five")?)?
Ok(())
* one two three
four five
* one two...
Wrap a single paragraph at a time. With whitespace replacement enabled, LF
becomes a space. With replacement disabled, original control characters may
remain inside a returned line. Leading paragraph whitespace is preserved when
it fits with content, and empty or whitespace-only paragraphs normally produce
no lines. Whitespace within a paragraph is otherwise preserved, rather than
collapsed. Disable break_long_words to keep long words intact even when they
exceed width. Indentation at least as wide as width can also exceed it:
long-word breaking still consumes at least one scalar per iteration.
These operations count scalars, not terminal columns or grapheme clusters;
combining marks may begin a line. Hyphen scanning uses the existing runtime
Unicode Alphabetic and Numeric properties plus underscore. These properties
follow the Rust toolchain’s Unicode version, as in std.regex,
and are not a promise of exact CPython Unicode-category equivalence. Ordinary
word splitting recognizes ASCII whitespace only; nonbreaking spaces stay inside
chunks. Blank-line classification, trimming, and shorten use Plenty’s
Unicode White_Space property, which excludes Python’s extra U+001C–U+001F
whitespace classifications. Sentence detection is the intentionally limited
English heuristic of lowercase ASCII letter, ., !, or ?, optionally a
closing quote.
Errors, limits, and cost
All operations return Result[..., TextError]. Widths and max_lines are
Positive[i64]; byte limits and tabsize are NonNegative[i64], so a literal
outside those ranges does not compile and a computed value needs
Positive.try_from(value)? or NonNegative.try_from(value)?. Indentation or a
placeholder that cannot fit the width is InvalidLimit. A valid cap exceeded by
input, expansion or output is LimitExceeded. Invalid configurations are
checked when an operation is called; constructing or editing a wrapper is
infallible. Indent strings and an active placeholder are each bounded by
max_input_bytes, as is the prefix for indent and indent_with.
max_bytes counts the sum of the UTF-8 bytes in wrap/TextWrapper.wrap
lines. For fill, the inserted LF bytes count too. max_input_bytes bounds the
original input and tab-expanded intermediate separately, so even shorten
rejects an input above that cap instead of allocating an unbounded temporary.
A huge tab size is checked against the cap before expanding. Chunk and line
counts are bounded by the input scalar count; sentence correction can add at
most one extra space per input scalar. The limits bound logical data, rather
than exact resident memory: list slots, geometric capacities, temporary string
copies, and old allocations during growth also take space.
Scanning and copying are O(input + expanded input + output), with storage
proportional to the capped intermediate, output, and per-chunk/per-line metadata,
plus four bytes per expanded input scalar for the splitter’s lookaround array.
A split long word retains scalar and UTF-8 byte cursors, avoiding repeated
suffix scans and copies without another scalar array. Unchanged whitespace
preparation shares the immutable input; tab expansion fills each space run in
one bulk operation. Complete chunks are reused during line
assembly, and dedent keeps only the common ASCII margin rather than allocating
one list per line. shorten uses native whitespace splitting and joining and
releases that word list before wrapping. Small builder appends write bytes
directly into geometrically grown storage; empty appends allocate nothing.
Allocation failure is recoverable as Allocation(AllocError) or
Bytes(BytesError) with the original byte-buffer cause. Failure discards the
partial result and preserves borrowed inputs and wrapper configuration; a
predicate’s own effects are not undone. fill materializes the bounded line
list before joining it, so it retains line metadata and an additional output
copy. No global state, native worker, regex engine, or additional dependency is
needed.
Performance qualification
examples/textwrap_bench.plenty in the repository provides seven repeatable
workloads. Optimized Plenty executables were compared
against commit 7d253e5b on Linux x86-64, AMD Ryzen 7 7840HS, pinned to CPU 0.
Each number below is the median time per operation from three alternating
before/after process pairs, each doing three operations, at workload scale
262144 (approximately 256 KiB of input). Input construction and printing are
outside the timed region; both binaries produced the same byte-count checksums.
Both used 8 MiB input/output limits to cover expanded tabs.
| Workload | Before | After | Speedup |
|---|---|---|---|
| Prose fill | 109.39 ms | 27.71 ms | 3.95× |
| Long Unicode word | 31.67 ms | 7.02 ms | 4.51× |
| Long dash run | 94.71 ms | 15.70 ms | 6.03× |
| Tab-heavy fill | 296.89 ms | 101.99 ms | 2.91× |
| Dedent | 12.56 ms | 3.62 ms | 3.47× |
| Indent | 7.60 ms | 3.73 ms | 2.04× |
| Shorten | 110.22 ms | 18.96 ms | 5.81× |
The 16, 64 and 256 KiB scale probes showed approximately linear growth. At the largest scale, peak whole-process RSS for dash wrapping fell from 15,684 to 4,072 KiB, and tab wrapping from 27,132 to 14,032 KiB; RSS includes input construction, allocator retention and all three repetitions. At scale 65536, allocation-failure budget probes required 28,912 successful checked allocations for prose fill instead of 236,498, and 6,725 for shorten instead of 309,813. These counts measure Plenty’s injected allocation checks, not operating-system allocation calls. The remaining costs include eager chunk/line storage and encoding larger pieces for the builder. These are local measurements; elapsed time and allocator retention depend on the platform.
Acknowledgements
- CPython
textwrapprovides the algorithms and API model, adapted to bounded Plenty storage and a scanner instead of regex splitting. Copyright (C) 1999–2001 Gregory P. Ward; Copyright (C) 2002 Python Software Foundation; PSF-2.0. The complete license is bundled in release third-party notices.
std.thread
Explicit imports for scoped threads.
Exports: spawn, ThreadError, and SpawnError[F]. These are the same compiler
facilities and types available in the prelude, including through module and
item aliases. Importing allocates nothing and introduces no wrapper or new
execution model. Supported target: x86_64 Linux GNU.
from std.thread import spawn, SpawnError
def total(values: &list[i64], start: i64, stop: i64) -> Result[i64, OverflowError[i64]]:
mut sum = 0
for i in range(start, stop):
sum += values[i]
Ok(sum)
def main() -> Result[(), AllocError | IoError | SpawnError[()] | OverflowError[i64]]:
values = [n for n in range(1, 101)]?
with spawn(total, &values, 0, 50)? as low, spawn(total, &values, 50, 100)? as high:
print(low.join()? + high.join()?)?
Ok(())
5050
Use with spawn(worker, arguments...)? as task: for a named function, or
with spawn(job)? as task: for an owned zero-argument job. task.join() waits
and moves the result once; leaving the scope joins automatically. Tasks remain
lexical handles and cannot escape. Borrowed arguments retain their loans until
join. The ordinary worker eligibility, foreign-effect, and channel hang checks
apply to every imported spelling.
A start with only owned or copied arguments, spawn(job) or
spawn(worker, arguments...), reports SpawnError[F].Unavailable(job, code),
retaining the job for recovery; a named start’s job binds its arguments.
.drop_job() on the start drops the job and reports the job-free
SpawnError[()]. A start that borrows, such as spawn(total, &values, 0, 50),
reports SpawnError[()].Unavailable((), code) and drops its moved arguments,
since a loan cannot be returned in an error. ThreadError remains the payload
of PoolError.Thread. See
owned jobs and
start failures. Job handoff and bookkeeping
allocate no Plenty heap storage; native thread creation uses OS resources.
There is no hidden clone. Start/join bookkeeping has fixed overhead apart from
argument/result transfer and cleanup, but scheduling and joining have no bounded
latency. A join waits as long as the worker needs.
Scoped spawn(...)? propagates the start error like any ?: the enclosing
Result’s error must be the start error, or an anonymous sum containing it.
spawn(...).unwrap() traps on a failed start.
The prelude spellings remain supported. There is no detached thread, global pool, forced interruption, or import-only exemption from ownership checks. For examples see explicit concurrency imports.
std.time
std.time provides nonnegative spans, monotonic instants, deadlines, and
wall-clock timestamps as four distinct types, with clock reads and sleep. The
arithmetic is ordinary embedded Plenty code. The runtime supplies only two
clock reads and one sleep. No operation allocates.
Every time value is a small class, so it is
affine: pass it by reference, as every operation here does, and use copy for
an independent value. == compares values structurally. Ordering has no
operator; compare as_nanoseconds() results or use the operations below.
from std.time import deadline_after, instant_at, milliseconds, monotonic_now, seconds, sleep
def main() -> Result[(), RangeError | IoError]:
start = monotonic_now()
sleep(&milliseconds(20))
elapsed = monotonic_now().duration_since(&start)?
print(elapsed.as_milliseconds() >= 20)?
# A fake timeline makes deadline arithmetic exact.
deadline = deadline_after(&instant_at(0), &seconds(5))?
later = instant_at(seconds(2).as_nanoseconds())
print(deadline.remaining(&later).as_milliseconds(), deadline.expired(&later))?
Ok(())
True
3000 False
Durations
A Duration is a span of u64 nanoseconds, from zero to just over 584 years.
A negative span is unrepresentable.
| Operation | Result | Contract |
|---|---|---|
nanoseconds(count: u64) | Duration | Exact |
microseconds(count: u32), milliseconds(count: u32), seconds(count: u32) | Duration | Exact; a u32 count always fits |
span.as_nanoseconds() | u64 | Exact |
span.as_microseconds(), as_milliseconds(), as_seconds() | u64 | Whole units, discarding the remainder |
span.as_timeout_ms() | u64 | Milliseconds rounded up |
span.is_zero() | bool | |
span.checked_add(&other) | Result[Duration, RangeError] | Error above the maximum |
span.checked_sub(&other) | Result[Duration, RangeError] | Error when other is longer |
span.saturating_sub(&other) | Duration | Zero when other is longer |
span.checked_mul(factor: u64) | Result[Duration, RangeError] | Error above the maximum |
Constructors never fail and arithmetic never traps or wraps. Build a span of
minutes or longer by multiplying, as in seconds(60).checked_mul(5). There is
no float conversion, division, or text format.
Instants and deadlines
An Instant is a point on a monotonic timeline: it never moves backwards and
is unaffected by changes to the system clock. Its origin is unspecified, so an
instant means something only relative to another from the same timeline.
| Operation | Result | Contract |
|---|---|---|
monotonic_now() | Instant | Read the process’s monotonic clock |
instant_at(nanoseconds_after_origin: u64) | Instant | A point on a caller-defined timeline |
at.checked_add(&span) | Result[Instant, RangeError] | Error past the end of the timeline |
at.duration_since(&earlier) | Result[Duration, RangeError] | Error when earlier is later |
at.saturating_duration_since(&earlier) | Duration | Zero when earlier is later |
A Deadline is the instant at which a total budget ends. Code that receives a
&Deadline can make several waits without restarting the budget.
| Operation | Result | Contract |
|---|---|---|
deadline_after(&start, &budget) | Result[Deadline, RangeError] | Error past the end of the timeline |
deadline_at(&at) | Deadline | |
deadline.expired(&now) | bool | True once now reaches the deadline |
deadline.remaining(&now) | Duration | Zero once expired |
A zero budget is expired at its own start. Deadline operations take the current
instant as an argument and read no clock, so their results depend only on their
inputs. instant_at supplies the fake clock for simulations and tests; mixing
its instants with monotonic_now() readings is meaningless and undetected.
The monotonic clock is Linux CLOCK_MONOTONIC, reported as nanoseconds since
the process’s first reading. It does not advance while the machine is
suspended. Readings are comparable across threads of one process and not
across processes.
Clocks
Clock is the protocol for code that reads the time through a value instead of
calling monotonic_now(), so a test can control what it sees:
pub protocol Clock:
def now(self) -> Instant:
pass
monotonic_clock() returns a MonotonicClock, whose now() is
monotonic_now(); it holds no state. std.test.fake_clock
returns a clock that moves only when told. A function written as
def f[C: Clock](clock: &C) is compiled once per clock type, with no dynamic
dispatch.
Wall-clock timestamps
A Timestamp has public fields seconds: i64, counted from the Unix epoch in
UTC, and nanoseconds: u32 in [0, 1_000_000_000). Times before the epoch use
floor seconds: half a second before it is Timestamp(-1, 500000000).
std.fs metadata returns this type.
| Operation | Result | Contract |
|---|---|---|
wall_now() | Timestamp | Read the system clock once |
Timestamp(seconds, nanoseconds) | Timestamp | Unchecked field construction |
stamp.duration_since(&earlier) | Result[Duration, RangeError] | Error when earlier is later, or the span exceeds a Duration |
stamp.checked_add(&span) | Result[Timestamp, RangeError] | Error past i64 seconds |
The system clock can be stepped in either direction, by an administrator or by
time synchronization. A later reading may therefore be earlier, and
duration_since reports that as an error instead of a negative or wrapped
span. Measure elapsed time and timeouts with instants. A timestamp cannot be
converted to an instant or compared with one.
A timestamp whose nanoseconds is out of range is an error in both operations.
There is no subtraction of a span, leap-second table, parsing, or formatting
here; the tz package provides calendars, zones, and text.
Civil time and time zones
Calendar dates, wall-clock times, IANA time zones, and RFC 3339 text live in
the official tz package, not in std.
It bundles a pinned release of the IANA time-zone database as Plenty source,
so results do not depend on the host’s zone files, and a program that never
looks up a zone links no zone data. Its API follows Rust’s jiff. Add it as a
Git dependency:
[dependencies]
tz = { git = "https://github.com/plentylang/tz.git", tag = "v0.3.0" }
A tz.Zoned holds a Timestamp with its zone. Adding calendar units keeps
the wall-clock reading; a local time that a transition skips or repeats is
resolved by the tz.Disambiguation the caller passes:
import tz
def main() -> Result[(), tz.TzError | FormatError | AllocError | IoError]:
new_york = tz.zone("America/New_York")?
london = tz.zone("Europe/London")?
meeting = tz.datetime(2024, 3, 9, 9, 30)?.to_zoned(&new_york, tz.Disambiguation.Reject)?
next_week = meeting.checked_add(&tz.span(weeks=1), tz.Disambiguation.Reject)?
print(f"{next_week}"?)?
print(f"{next_week.in_zone(&london)?}"?)?
print(meeting.span_until(&next_week).hours())?
Ok(())
2024-03-16T09:30:00-04:00[America/New_York]
2024-03-16T13:30:00+00:00[Europe/London]
167
span_until measures elapsed time: the 167 hours above include the hour the
DST transition skips. zoned.until(&other, largest, disambiguation) and
since instead give a difference in calendar units up to largest
(tz.Unit.Year down to Nanosecond), computed on the wall clock as jiff
does; calendar units need both values in the same zone.
tz.zoned(&stamp, &zone) and zoned.timestamp() convert between the two
types.
tz.zone(name) can return any zone, so it links the whole database (about
150 KB). The tz.zones module has one function per canonical zone, and each
links only that zone’s data. The functions cannot fail:
import tz
from tz.zones import america_new_york
def main() -> Result[(), tz.TzError | AllocError | IoError]:
new_york = america_new_york()
meeting = tz.datetime(2024, 3, 10, 2, 30)?.to_zoned(&new_york, tz.Disambiguation.Compatible)?
print(meeting.offset().seconds())?
Ok(())
This prints -14400: 02:30 falls in that day’s gap, and Compatible moves it
forward to 03:30 EDT. The package README lists the API, how zone names map to
function names, its data version, and how the data is regenerated.
Sleeping and timed waits
| Operation | Result | Contract |
|---|---|---|
sleep(&span) | () | Block this thread for at least the span |
sleep_until(&deadline) | () | Block until the deadline; return at once if expired |
Sleep measures the monotonic clock, resumes after signals, and cannot be cancelled. Scheduling may extend it; nothing shortens it. Workers may sleep and read both clocks.
Prefer std.sync’s typed waits:
recv_before, send_before, select_recv_before, future_before and
cancelled_before take a shared deadline and a cancellation token. The wrappers
keep one total budget and preserve unsent values on every failure.
Low-level timed waits have
explicit _timeout_ms names and accept u64 milliseconds. as_timeout_ms()
rounds up so a positive remainder does not become a zero-budget retry loop.
For example, token.wait_timeout_ms(span.as_timeout_ms()) observes that token
without a second cancellation source. The old _timeout names remain aliases.
An expired deadline yields a zero budget, which those operations define as an immediate check where readiness still wins.
Errors, costs, and limits
Additions, products and non-saturating differences handle local checked
arithmetic and map unrepresentable results to the existing
RangeError.OutOfRange. Infallible unit conversions, saturating differences
and fractional timestamp arithmetic use explicit unchecked: suites with
bounds documented in source. Their guards prove nonzero divisors and
representable results; no operation relies on silent wrapping except the
explicit two’s-complement difference used for ordered timestamp seconds.
Every recoverable failure is the built-in
RangeError.OutOfRange:
the result is not representable in its type. Time arithmetic therefore
propagates with ? alongside T.try_from. Clock reads and sleep cannot fail.
Each operation takes constant time and allocates nothing, including clock reads and sleep. Values hold no OS resource and may move between threads.
Supported target: x86_64-unknown-linux-gnu. See the lessons
Measure time and wait and
Share one deadline across waits.
Design decisions
- Four types, no conversions between clocks. A span, a monotonic point, a budget end, and a wall time have different valid operations. Distinct types reject using a wall time for a timeout at compile time.
- Integer nanoseconds. One exact representation with no rounding rule.
Unit constructors take
u32so that construction is infallible; arithmetic is where overflow can occur, and it is always checked. - No hidden clock reads.
expiredandremainingtakenow. A function’s clock reads are visible at its call sites, and a fake clock needs no protocol, global state, or test-only mode. - Explicit rounding names.
as_millisecondsdiscards andas_timeout_msrounds up. Neither is a default. - Existing waits are unchanged. Millisecond budgets, cancellation tokens, and their wakeup rules remain the single mechanism; this module supplies the arithmetic that feeds them.
- One timestamp type.
std.fsreusesTimestampinstead of declaring its own, so file times and clock readings subtract directly.
Acknowledgements
Duration,Instantand their checked and saturating operations are modelled on Rust’sstd::time.
std.tls
import std.tls runs TLS 1.2 and 1.3 over a std.net TCP stream.
A client verifies the server’s certificate chain against chosen trust roots
and checks that it names the server; a server presents a certificate chain
and key, and may ask clients for theirs. Handshakes, reads, and writes take the same deadline and
cancellation token as the TCP stream under them. The module has one error
type, TlsError, and is available on the supported x86-64 Linux GNU target.
Upstream code, used as-is. The TLS protocol, certificate verification and all cryptography are rustls with its
ringprovider, compiled into the runtime. Plenty implements none of them; this is deliberate, as forstd.crypto.
A rustls session never touches a socket: std.tls moves ciphertext between the session and the TCP
stream in checked Plenty, so every wait is a std.net wait.
import std.bytes
import std.tls
from std.net import NetError, TcpListener, connect, listen, loopback_v4
from std.sync import CancellationToken
from std.time import Deadline, deadline_after, milliseconds, monotonic_now
from std.tls import ServerConfig, TlsError
# Test-only CA and a localhost leaf it signed; never reuse a published key.
const CA_PEM: str = "-----BEGIN CERTIFICATE-----\nMIIBmjCCAT+gAwIBAgIULOQ2Qxr8CVZY2bG0uRYKKEdMD3swCgYIKoZIzj0EAwIw\nGTEXMBUGA1UEAwwOUGxlbnR5IFRlc3QgQ0EwIBcNMjQwMTAxMDAwMDAwWhgPMjEy\nNDAxMDEwMDAwMDBaMBkxFzAVBgNVBAMMDlBsZW50eSBUZXN0IENBMFkwEwYHKoZI\nzj0CAQYIKoZIzj0DAQcDQgAEkiB3Ogehrrt6YbQghwDPv6tkhE+z9zTCVih8fvtR\nZ1/691Nfh7/J9IgaNgNSTMJjGTolzGnjaCUDqoARlFyM+6NjMGEwHQYDVR0OBBYE\nFOTFBDgWXHj/qmVJDMlAHYCEZFX5MB8GA1UdIwQYMBaAFOTFBDgWXHj/qmVJDMlA\nHYCEZFX5MA8GA1UdEwEB/wQFMAMBAf8wDgYDVR0PAQH/BAQDAgIEMAoGCCqGSM49\nBAMCA0kAMEYCIQD41ATQnIjKkKMLQAb9AyoyOFmljYZyz348XJEIbqMgKwIhAL3m\nmWd8On5gvkfytspatH5mDWOcSyW+au7vuIC2gHhO\n-----END CERTIFICATE-----\n"
const LEAF_PEM: str = "-----BEGIN CERTIFICATE-----\nMIIBnDCCAUKgAwIBAgIBATAKBggqhkjOPQQDAjAZMRcwFQYDVQQDDA5QbGVudHkg\nVGVzdCBDQTAgFw0yNDAxMDEwMDAwMDBaGA8yMTI0MDEwMTAwMDAwMFowFDESMBAG\nA1UEAwwJbG9jYWxob3N0MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAE8THNQeTi\nJ6Eq0tSxtHW5Q4TCsbIqB3MOtM98ERyKgl6kHeHCeDZYKx8B+VRGT2jkWuTeN24Q\nljwUfh7uh7Iaz6N+MHwwGgYDVR0RBBMwEYIJbG9jYWxob3N0hwR/AAABMAkGA1Ud\nEwQCMAAwEwYDVR0lBAwwCgYIKwYBBQUHAwEwHQYDVR0OBBYEFEIFC2Fyd4cfAKTx\nMiNCMdDkqDNbMB8GA1UdIwQYMBaAFOTFBDgWXHj/qmVJDMlAHYCEZFX5MAoGCCqG\nSM49BAMCA0gAMEUCIAUF3HH43lnirgSN1Vr4twZrFmXKBxlbYNQni/h4ZaxhAiEA\n5rD/yE1B4IsbEGCMlyAeX507leXfGY9+wrHaxikU/w0=\n-----END CERTIFICATE-----\n"
const LEAF_KEY: str = "-----BEGIN PRIVATE KEY-----\nMIGHAgEAMBMGByqGSM49AgEGCCqGSM49AwEHBG0wawIBAQQglvSEE3H7v+mInxSF\nAyWFDKYpzpXr20wH8mfFY3bCaB2hRANCAATxMc1B5OInoSrS1LG0dblDhMKxsioH\ncw60z3wRHIqCXqQd4cJ4NlgrHwH5VEZPaORa5N43bhCWPBR+Hu6HshrP\n-----END PRIVATE KEY-----\n"
def serve(listener: &TcpListener, config: &ServerConfig, deadline: &Deadline, cancel: &CancellationToken) -> Result[(), AllocError | TlsError | NetError | std.bytes.BytesError]:
mut stream = std.tls.accept(listener.accept(deadline, cancel)?, config, deadline, cancel)?
reply = std.bytes.from_text("hello over TLS")?
stream.write_all_before(&reply, deadline, cancel)?
stream.close_before(deadline, cancel)
def main() -> Result[(), SpawnError[()] | IoError | AllocError | RangeError | TlsError | NetError | std.bytes.BytesError]:
cancel = CancellationToken()?
deadline = deadline_after(&monotonic_now(), &milliseconds(5000))?
client = std.tls.client_config(&std.tls.roots_from_pem(&CA_PEM)?)?
server = std.tls.server_config(&LEAF_PEM, &LEAF_KEY)?
listener = listen(&loopback_v4(0u16), 8)?
address = listener.local_address()?
with spawn(serve, &listener, &server, &deadline, &cancel)? as worker:
tcp = connect(&address, &deadline, &cancel)?
mut stream = std.tls.connect(tcp, &client, "localhost", &deadline, &cancel)?
mut buffer = std.bytes.with_capacity(64)?
buffer.resize(64, 0u8)?
count = stream.read_before(&mut buffer, 0, 64, &deadline, &cancel)?
print(buffer.to_text_range(0, count)?)?
worker.join()?
Ok(())
hello over TLS
Ciphertext cursors are bounded by their fixed buffers, and each native TLS or
TCP transfer advances by at most the remaining slice. Their small arithmetic
opt-outs do not change TlsError, partial progress after a failed wait, or
closing the TCP stream after a failed close-notify.
Trust roots and configuration
| Function | Result | Contract |
|---|---|---|
roots_from_pem(pem: &str) | Result[TrustRoots, TlsError] | The CERTIFICATE blocks of pem as trust anchors |
system_roots() | Result[TrustRoots, TlsError] | The operating system’s certificate bundle |
client_config(roots: &TrustRoots) | Result[ClientConfig, TlsError] | Client settings verifying servers against roots |
server_config(chain_pem: &str, key_pem: &str) | Result[ServerConfig, TlsError] | Server settings presenting chain_pem, leaf first, with its private key |
roots_from_pem skips blocks of other types. PEM that does not decode is
InvalidCertificate. A decoded certificate that cannot serve as a trust
anchor is skipped, since system bundles may hold some; when none is usable
the result is NoTrustRoots. TrustRoots.len() counts the anchors kept.
system_roots reads the first readable file of:
/etc/ssl/certs/ca-certificates.crt(Debian, Ubuntu, Arch, Alpine)/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem(Fedora, RHEL)/etc/ssl/ca-bundle.pem(openSUSE)/etc/ssl/cert.pem
None readable is NoTrustRoots. It does not consult environment variables;
read another bundle with read_text and pass it to roots_from_pem. The
bundle is read at the call, so a long-running program sees updates only when
it builds new roots.
server_config accepts a PKCS#8, PKCS#1, or SEC1 private key. A chain or key
that does not parse, an empty chain, or a key that does not match the leaf
is InvalidConfiguration. Servers do not ask clients for certificates.
Both configurations use TLS 1.2 and 1.3 with rustls’s default cipher suites and key-exchange groups. Neither resumes sessions: clients keep no session cache and servers issue no tickets. A configuration is immutable and may be borrowed by any number of connections and workers.
A TLS 1.3 server therefore sends nothing after the client’s Finished. So
that a client with Nagle’s algorithm on does not hold its first request
until the kernel’s delayed acknowledgement (about 40 ms on Linux), a server
stream acknowledges the client’s last handshake flight at once (Linux
TCP_QUICKACK), unless a reply or the client’s first request is already
there to carry the acknowledgement.
Builders
A builder gathers further settings; build checks them together and returns
the configuration. A builder with no further calls builds the same
configuration as client_config or server_config.
| Function | Result |
|---|---|
client_builder(roots: &TrustRoots) | Result[ClientBuilder, TlsError] |
server_builder(chain_pem: &str, key_pem: &str) | Result[ServerBuilder, TlsError] |
| Method | On | Contract |
|---|---|---|
certificate(chain_pem: &str, key_pem: &str) -> Result[(), TlsError] | both | Client: the chain presented when a server asks. Server: another chain to choose by name |
verify_clients(roots: &TrustRoots, certificates: ClientCertificates) -> () | server | Ask clients for a certificate and verify it against roots |
application_protocol(name: &str) -> Result[(), TlsError] | both | Offer or accept name in ALPN, after the names already given |
minimum_version(version: TlsVersion) -> () | both | Tls13 refuses TLS 1.2 peers |
cipher_suite(suite: CipherSuite) -> () | both | Allow suite; once any is allowed, only allowed suites are used |
revocation_list(crl_pem: &str) -> Result[(), TlsError] | both | Reject peer certificates that a CRL in crl_pem names |
build() -> Result[ClientConfig or ServerConfig, TlsError] | both | The configuration |
pub enum ClientCertificates:
Required
Optional
pub enum TlsVersion:
Tls12
Tls13
pub enum CipherSuite:
Tls13Aes256GcmSha384
Tls13Aes128GcmSha256
Tls13Chacha20Poly1305Sha256
EcdheEcdsaAes256GcmSha384
EcdheEcdsaAes128GcmSha256
EcdheEcdsaChacha20Poly1305Sha256
EcdheRsaAes256GcmSha384
EcdheRsaAes128GcmSha256
EcdheRsaChacha20Poly1305Sha256
Client certificates. With verify_clients, a server asks for a client
certificate and verifies its chain against roots, which are usually a
private authority rather than the system bundle. With Required, a client
that presents none fails with CertificateRequired on the server. With
Optional, it is accepted, and peer_certificate() is Nothing. A client
has at most one chain: a second certificate call is InvalidConfiguration.
In TLS 1.3 the client finishes its handshake before the server has checked
its certificate, so a rejected client sees Rejected at its first read.
Certificates by name. A server with several chains presents the first whose leaf names the host the client asked for through server name indication (SNI), or the first chain when none does or the client sent no name (clients connecting by IP address send none).
Application protocols. A server picks the first of its names, in the
order given, that the client offers. A client that offers names, none of
which the server accepts, fails with Incompatible on the server and
Rejected on the client. When either side gives no names, none is chosen. A
name is 1 to 255 bytes; others are InvalidConfiguration.
Versions and suites. Suites are the ring provider’s, listed in its
default order of preference. The client’s preference decides among suites
both sides allow. Allowed suites that cover no allowed version are
InvalidConfiguration at build. Peers with no version or suite in common
fail with Incompatible, or Rejected on the side that receives the alert.
Revocation lists. revocation_list adds each X509 CRL block of
crl_pem; PEM without one is InvalidConfiguration. A certificate that a
list from its issuer names is RevokedCertificate. Certificates whose
issuer has no list are accepted, and a list’s next-update time is not
enforced, so a stale list still applies. A server
revocation_list without verify_clients is InvalidConfiguration at
build. Lists are read once; load fresh ones by building a new
configuration.
Connecting
| Function | Result | Contract |
|---|---|---|
connect(tcp: TcpStream, config: &ClientConfig, server_name: &str, deadline: &Deadline, cancel: &CancellationToken) | Result[TlsStream, TlsError] | Complete a client handshake over tcp |
accept(tcp: TcpStream, config: &ServerConfig, deadline: &Deadline, cancel: &CancellationToken) | Result[TlsStream, TlsError] | Complete a server handshake over tcp |
Both take ownership of the TCP stream and return only after the handshake
completes. server_name is a DNS name or an IP address; the server’s leaf
certificate must name it in its subject alternative names. A name that is
neither is InvalidName. Certificates are checked against the wall clock at
the handshake.
When the handshake fails for a TLS reason, the alert rustls prepared is sent
if the deadline allows, so the peer sees Rejected. On any failure the TCP
stream is closed when the partial session is dropped.
start_connect(tcp, config, server_name) and start_accept(tcp, config)
return a stream without waiting. Its handshake then runs inside its first
reads and writes, in handshake_before, or in try_handshake; no
application data passes before the peer is verified.
Streams
| Method | Result | Contract |
|---|---|---|
handshake_before(deadline: &Deadline, cancel: &CancellationToken) | Result[(), TlsError] | Finish the handshake. One that times out keeps its progress for a later call, so with a deadline already passed it does only the work possible without waiting |
read_before(buffer: &mut Bytes, start: i64, stop: i64, deadline: &Deadline, cancel: &CancellationToken) | Result[i64, TlsError] | Decrypted bytes into buffer[start:stop], waiting for at least one |
write_before(source: &[u8], deadline: &Deadline, cancel: &CancellationToken) | Result[i64, TlsError] | Encrypt and send at most 16 KiB of source; the count taken |
write_all_before(source: &[u8], deadline: &Deadline, cancel: &CancellationToken) | Result[(), TlsError] | Send all of source |
close_before(deadline: &Deadline, cancel: &CancellationToken) | Result[(), TlsError] | Send close_notify, then close the TCP stream |
shutdown(direction: Shutdown) | Result[(), TlsError] | End the TCP stream in one direction without closing it; ending writes sends close_notify first without waiting, and ciphertext the socket does not take at once is lost. Reading continues |
read(buffer: &mut Bytes, start: i64, stop: i64) | Result[i64, StreamError] | std.io Reader; waits without a deadline |
write(source: &[u8]) | Result[i64, StreamError] | std.io Writer; waits without a deadline |
flush() | Result[(), StreamError] | Nothing to do: writes are sent as they are made |
close() | Result[(), StreamError] | close_before without a deadline |
application_protocol() | Result[Option[str], TlsError] | The protocol chosen through ALPN |
server_name() | Result[Option[str], TlsError] | On a server, the name the client sent through SNI; Nothing on a client |
peer_certificate() | Result[Option[Bytes], TlsError] | The DER encoding of the peer’s verified leaf certificate |
version() | Option[TlsVersion] | The negotiated version |
cipher_suite() | Option[CipherSuite] | The negotiated suite |
Each source is a &[u8] view: a Bytes buffer, part of one, or a b"..."
literal, written without a copy.
The session details are Nothing until the handshake has settled them.
peer_certificate() is Nothing on a server that did not verify clients or
whose client presented none.
Ranges follow the range rule. A read of zero bytes
means the peer sent close_notify. A connection that ends without one is
Truncated (StreamError.UnexpectedEof through read), because an attacker
can cut a TCP connection short. Code that frames its own messages, such as
Content-Length bodies, may treat a truncation after the last expected byte
as an ordinary end.
close_before does not wait for the peer’s close_notify. It closes the TCP
stream even when sending close_notify fails, and then returns that error.
After a close, writes are Closed. A failed write has sent an unknown prefix
of its range.
The std.io methods block as TcpStream’s do, so TlsStream
works with std.io helpers and std.http framing. Through them,
TLS failures appear as StreamError.Io(IoError.Other); use the _before
methods to see the TlsError.
Without waiting
These methods never wait, so one thread can drive many streams from a
std.ready selector:
| Method | Result | Contract |
|---|---|---|
try_handshake() | Result[bool, TlsError] | Advance the handshake; true once complete |
try_read(buffer: &mut Bytes, start: i64, stop: i64) | Result[Option[i64], TlsError] | Decrypted bytes if any can be had now; Some(0) is close_notify |
try_write(source: &[u8]) | Result[Option[i64], TlsError] | Encrypt at most 16 KiB; Nothing, taking nothing, while the handshake is incomplete or earlier ciphertext is unsent |
try_flush() | Result[bool, TlsError] | Send buffered ciphertext; true once none is left |
interest() | Interest | What to wait for: ReadWrite while ciphertext waits to be sent, else Read |
watch(selector: &mut Selector, key: u64, interest: Interest) | Result[(), TlsError] | Register the TCP stream |
unwatch(selector: &mut Selector) | Result[(), TlsError] | Remove it |
A try_ result of Nothing or false means the stream needs the socket.
These methods use TCP’s immediate operations,
without polling with an expired deadline. WouldBlock or interruption leaves
the TLS progress buffered and returns Nothing or false; other network
errors, including ConnectionTimedOut, propagate.
Do all the work available without waiting, then register interest()
again and wait: rustls may hold decrypted bytes the socket no longer
signals, and a write may leave ciphertext that only a writable socket can
take. Ciphertext a try_ call could not send stays buffered for the next
call. Registrations are level-triggered, so ReadWrite on an idle stream
wakes at once; register Read once interest() says so.
A read_before or handshake_before that times out at its deadline has
also done all the work available, so a stream may be watched after one.
std.http.server parks HTTPS
connections this way, between handshake messages and while idle.
Sessions
A Session is a TLS session without a socket: the caller moves ciphertext
through its own buffers. It never waits and owns no descriptor, so it suits
event loops that already own the socket.
std.actor.tls runs one over an actor connection.
client_session(&config, server_name) and server_session(&config) make
one with the same verification as connect and accept.
| Method | Result | Contract |
|---|---|---|
feed(source: &[u8]) | Result[i64, TlsError] | Process received ciphertext; the count used, short while decrypted bytes wait for read |
feed_end() | Result[(), TlsError] | The peer’s stream ended; without close_notify, read then fails with Truncated |
take(target: &mut Bytes, start: i64, stop: i64) | Result[i64, TlsError] | Ciphertext to send; 0 when none waits |
read(target: &mut Bytes, start: i64, stop: i64) | Result[Option[i64], TlsError] | Decrypted bytes; Nothing while none is ready, Some(0) for close_notify |
write(source: &[u8]) | Result[i64, TlsError] | Encrypt at most 16 KiB; short once 64 KiB of ciphertext waits for take; Closed after close |
close() | () | Queue close_notify for take |
handshaking(), wants_write(), wants_read() | bool | Handshake incomplete; ciphertext waits for take; more ciphertext is useful |
application_protocol, server_name, peer_certificate, version and
cipher_suite are as for TlsStream. While the handshake runs, write
holds plaintext in the session; check handshaking() first to bound it. On
a TLS failure the session has queued an alert: send what take returns
before closing the socket. A session is allocated where it is made; in an
actor handler that is the actor’s heap.
Read ranges follow the Reader contract: invalid buffer bounds trap
with read: start/stop outside buffer before I/O or handshake work. The
nonblocking session take operation also validates its caller buffer first
and traps with take: start/stop outside buffer. Empty ranges must still
be within the buffer.
Errors
pub enum TlsError:
Net(NetError)
UntrustedCertificate
ExpiredCertificate
WrongHost
InvalidCertificate
Rejected
Protocol
Truncated
InvalidName
InvalidConfiguration
NoTrustRoots
Closed
Allocation(AllocError)
RevokedCertificate
CertificateRequired
Incompatible
| Variant | Meaning |
|---|---|
Net(cause) | The TCP operation ended, including deadline TimedOut, kernel ConnectionTimedOut, and Cancelled |
UntrustedCertificate | The peer’s chain does not lead to a trust root, or a signature is wrong |
ExpiredCertificate | A certificate in the chain is expired or not yet valid |
WrongHost | The leaf certificate does not name server_name |
InvalidCertificate | A certificate is malformed or unusable for TLS, or PEM does not decode |
Rejected | The peer sent a fatal alert, usually because it rejected our certificate |
Protocol | The peer broke the TLS protocol |
Truncated | The connection ended without close_notify |
InvalidName | server_name is neither a DNS name nor an IP address |
InvalidConfiguration | A chain, key, name, list, or combination of builder settings is unusable |
NoTrustRoots | No usable trust anchor was found |
Closed | The stream was closed by this side |
Allocation(cause) | A buffer or runtime object could not be allocated |
RevokedCertificate | A revocation list names the peer’s certificate |
CertificateRequired | A server requiring client certificates received none |
Incompatible | The peers share no version, cipher suite, or application protocol |
Deadlines and cancellation
Every _before method passes its deadline and token to each std.net call
it makes, so a handshake against a silent peer ends with
Net(NetError.TimedOut) and a cancelled token with Net(NetError.Cancelled).
The kernel’s connection timeout is Net(NetError.ConnectionTimedOut) and is
not resumable. The try_ methods also return this error; they do not mistake it
for a socket that would block.
Use one deadline for a whole exchange, as with std.net. Ciphertext that a
timed-out write did not send stays buffered and goes out with the next call,
but which plaintext the peer has received is unknown; after a timeout in the
middle of an exchange, close or drop the stream unless the protocol can
recover.
Ownership and threads
TrustRoots, ClientConfig, ServerConfig, TlsStream, and Session each own one
runtime object; dropping it releases the object, and dropping a stream closes
its socket without close_notify. The configurations, builders, streams and
sessions are redacted: printing one shows its type name
and (@redacted), never key or session state. A stream may move to a worker. Trust roots,
builders and configurations belong to the heap that allocates them. Outside
actors, sessions share the parsed configuration. An actor session
reconstructs a heap-local configuration from owned source material so a trap
cannot retain another heap’s configuration. Starting a
session draws randomness and verifying a certificate reads the clock; both are
allowed in workers.
The __tls_* intrinsics behind the session are
reserved for the standard library. Writing uses
__tls_write, which reads the caller’s &[u8] view in place.
Costs
| Item | Memory |
|---|---|
TlsStream | Two 16 KiB ciphertext buffers, plus rustls’s session state and its buffer for the incoming record (up to about 17 KiB) |
Session | rustls’s session state and its record buffers; see std.actor.tls for measurements |
| Configuration | Owned certificate/key/trust/policy recipe plus parsed configuration; shared by sessions outside actors |
TrustRoots | The parsed anchors; a full system bundle holds about 120 |
rustls allocates through the runtime’s Rust allocator, not Plenty’s fallible
allocation, so an allocation failure inside rustls stops the program.
rustls has no fallible allocation interface, and stable Rust aborts when the
global allocator fails, so this is not recoverable. Inside an actor, exhaustion instead traps that actor and discards its heap,
including partial TLS construction. Allocation failures in std.tls itself are
Allocation. Actor sessions also pay for reconstructing a private configuration,
including key parsing; see actor TLS memory.
Plaintext is copied once into rustls and ciphertext once into the outgoing buffer; incoming ciphertext is decrypted into rustls’s buffer and copied to the caller.
Supply
The runtime builds these crates from unmodified copies under
plenty-runtime/vendor (ring’s pregenerated assembly is trimmed to the
x86-64 ELF files), so building the compiler needs no network:
| Crate | Version | Licence |
|---|---|---|
rustls | 0.23.46 | Apache-2.0, ISC, or MIT |
rustls-webpki | 0.103.15 | ISC |
rustls-pki-types | 1.15.1 | MIT or Apache-2.0 |
ring | 0.17.14 | Apache-2.0 and ISC, with OpenSSL and BoringSSL notices |
untrusted, zeroize, once_cell | 0.9.0, 1.9.1, 1.21.4 | ISC; MIT or Apache-2.0; MIT or Apache-2.0 |
getrandom, libc | 0.2.17, 0.2.186 | MIT or Apache-2.0 |
ring’s C and assembly sources are compiled by build.rs with the system C
compiler into a static archive, as SQLite’s are. The versions are pinned in
Cargo.lock, and a test fails if the two disagree. To update, replace the
directories with new releases, run cargo update -p CRATE --precise VERSION,
and rerun tests/test_std_tls.rs. Follow rustls security advisories: an
update is a change to the compiler, and programs pick it up when rebuilt.
Programs whose main never reaches std.tls code contain none of rustls or
ring, even when they import a module that imports std.tls; the linker drops
the unreachable functions and the TLS runtime only they call.
Example
The tutorial lesson serves an echo over TLS on a worker and shows a client rejecting the wrong host name. Mutual TLS adds client certificates and application protocols.
Limits
- No certificate pinning; verify
peer_certificate()bytes yourself. - No session resumption or 0-RTT data: a shared session cache is memory without a bound or an owner, and 0-RTT data can be replayed.
- No OCSP: rustls does not check stapled responses, and fetching them would reach the network behind the program’s back. Revocation is by CRL only.
- No choice of key-exchange groups, and no TLS 1.0 or 1.1.
- Outside actors, allocation failure inside rustls stops the program; actor TLS contains it by discarding the actor heap.
- HTTPS in
std.httpis not built in; wrap aTlsStreamwith its framing. - One target, x86-64 Linux GNU.
Acknowledgements
-
Wraps rustls 0.23.46 with the ring 0.17.14 crypto provider and rustls-webpki 0.103.15, vendored in the runtime; see Supply for licences.
-
The CA bundle paths are drawn from the Linux list in Go’s
crypto/x509. -
Uses zeroize 1.9.1 (MIT or Apache-2.0) to clear retained encoded private keys on normal destruction.
std.toml
import std.toml parses TOML 1.0 documents into a
value tree and writes a tree back as TOML text, under explicit limits, with one
error type, TomlError.
For editing human-maintained files, document_text and document create a
Document that preserves the source layout. Use the value
tree writer below for generated files.
The module is checked Plenty code with no native primitives of its own. It
reads and writes std.bytes buffers and std.io streams,
and is available on the supported x86-64 Linux GNU target.
import std.toml
from std.toml import Toml
def main() -> Result[(), AllocError | IoError | std.toml.TomlError]:
bounds = std.toml.limits()
config = std.toml.parse_text("title = \"demo\"\n\n[server]\nport = 8080\nhosts = [\"a\", \"b\"]\n", &bounds)?
match &config:
case Toml.Table(entries):
for entry in entries:
match &entry.value:
case Toml.Text(text):
print(entry.key, "is", text)?
case Toml.Table(fields):
print(entry.key, "has", len(fields), "entries")?
case _:
print(entry.key, "is another type")?
case _:
print("not a table")?
print(std.toml.encode_text(&config, &bounds)?.rstrip()?)?
Ok(())
title is demo
server has 2 entries
title = "demo"
[server]
port = 8080
hosts = ["a", "b"]
Accepted text
The parser accepts TOML 1.0 and rejects everything else, including TOML 1.1 additions.
| Rule | Behavior |
|---|---|
| Encoding | UTF-8, validated strictly before parsing; no byte-order mark |
| Newlines | \n or \r\n; a \r alone is an error everywhere |
| Comments | # to end of line; control characters other than tab are ControlCharacter |
| Keys | Bare (A-Za-z0-9_-), basic-quoted, or literal-quoted parts joined by ., with optional spaces or tabs around each . |
| Strings | Basic, literal, multi-line basic, and multi-line literal. A newline right after an opening """ or ''' is dropped; \r\n inside a multi-line string is kept as written |
| Escapes | \b \t \n \f \r \" \\ \uXXXX \UXXXXXXXX, hex digits of either case, Unicode scalar values only. In a multi-line basic string, \ before a line end removes the line end and the whitespace and newlines that follow |
| Control characters | Bytes 0-8, 10-31, and 127 are rejected in every string and comment, except newlines in multi-line strings |
| Integers | Decimal with optional sign, or 0x, 0o, 0b with lowercase prefix and no sign. _ only between digits; no leading zeros. Must fit i64 |
| Floats | Integer part, then a fraction, an exponent, or both; inf, nan with optional sign. Must be finite unless written as inf or nan |
| Booleans | true, false |
| Date-times | RFC 3339 with T, t, or one space between date and time; seconds required; Z, z, or ±HH:MM offset. Fractional digits beyond nanoseconds are truncated. Year 0000-9999; the day must exist in its month; second 60 is accepted at any time |
| Arrays | Mixed element types, newlines and comments between elements, optional trailing comma |
| Inline tables | One line, no trailing comma; complete as written |
Tables follow the definition rules of the specification:
| Situation | Fault |
|---|---|
| A key assigned twice in one table | DuplicateKey |
A [table] header repeated, or naming a table already made by dotted keys or an inline table | DuplicateTable |
| A header or dotted key that passes through a value, an inline table, or an array that is not an array of tables | KeyConflict |
A dotted key that adds to a table made by a [header] other than the current one | KeyConflict |
[[name]] where name is not an array of tables | KeyConflict |
A header may name a table that an earlier header created implicitly, such as
[a] after [a.b], once. A header path that passes through an array of
tables continues in its last table.
Value tree
pub enum Toml:
Bool(bool)
Int(i64)
Float(f64)
Text(str)
DateTime(DateTime)
Array(list[Toml])
Table(list[Entry])
pub class Entry:
pub key: str
pub value: Toml
pub enum DateTimeKind:
OffsetDateTime
LocalDateTime
LocalDate
LocalTime
pub class DateTime:
pub kind: DateTimeKind
pub year: i64
pub month: i64
pub day: i64
pub hour: i64
pub minute: i64
pub second: i64
pub nanosecond: i64
pub offset_minutes: i64
A document is a Toml.Table. Tables keep their entries in the order each key
first appeared in the text, including keys added later by a header, so
[a.b] before [a] gives a the entry b first. An array of tables is an
Array of Table values. DateTime fields that its kind does not use are 0;
offset_minutes is minutes east of UTC, and Z is 0.
| Function | Result | Contract |
|---|---|---|
limits() | Limits | The defaults below |
parse(source: &[u8], bounds: &Limits) | Result[Toml, TomlError] | Parse one document from a buffer |
parse_text(text: &str, bounds) | Result[Toml, TomlError] | The same, from a str |
read_value[R: Reader](reader: &mut R, bounds) | Result[Toml, TomlError] | Read to end of input, then parse |
encode(value: &Toml, bounds) | Result[Bytes, TomlError] | TOML text for a table |
encode_text(value: &Toml, bounds) | Result[str, TomlError] | The same, as a str |
write_value[W: Writer](writer: &mut W, value: &Toml, bounds) | Result[(), TomlError] | Encode the whole value, then write it; no flush |
find(entries: &list[Entry], key: &str) | Option[i64] | Index of the first entry named key |
read_value reads chunks of up to 4096 bytes, retries Interrupted, and
treats a zero-length read as end of input. It reads at most one byte past
max_input before reporting InputTooLarge.
Writing
The writer produces text that parses back to the same values:
- Each table lists its plain entries as
key = valuelines first, then each table entry as a[path]section and each non-empty array whose elements are all tables as[[path]]sections. A section header is omitted for a non-empty table whose entries are all sections. Inside one group, entries keep their order. The plain-first order is required: a plain key written after a header would belong to that header’s table. - Other arrays and tables nested in them are written inline on one line.
- Keys are bare when they are non-empty and use only bare-key bytes; otherwise they are basic strings.
- Strings are basic strings.
",\, and control characters are escaped (\b \t \n \f \r, otherwise\u00XXin uppercase hex); every other scalar is written as UTF-8. - Floats are the shortest text that parses back to the same value, as
str.from, with.0added when that text has no point or exponent; infinities and NaN areinf,-inf, andnan. - Date-times use
T,Zfor offset 0, and the shortest fraction that keeps the nanoseconds.
Comments, blank lines, key quoting, number spellings, and inline-or-section
choices of the source are not kept. The writer rejects a root that is not a
table (NotATable), a table with a repeated key (DuplicateKey), and a
DateTime whose fields are out of range or whose unused fields are not 0
(InvalidDateTime).
Limits
pub class Limits:
pub max_depth: NonNegative[i64]
pub max_input: NonNegative[i64]
pub max_output: NonNegative[i64]
limits() returns Limits(64, 67108864, 67108864). The fields are public;
adjust a copy. A literal converts on assignment; a computed value needs
NonNegative.try_from(value)?. Every function takes &Limits and traps when
max_depth is above 1024.
| Field | Bounds | When exceeded |
|---|---|---|
max_depth | Tables and arrays nested inside the document’s root table. 0 allows only plain values at the root. An array of tables counts two levels: the array and each table | DepthExceeded |
max_input | Bytes in one document | InputTooLarge; the offset is the limit |
max_output | Bytes one encoding produces | OutputTooLarge |
The depth limit also bounds the parser’s and writer’s recursion, so neither a document nor a tree built by hand can overflow the stack.
Input cursor advances consume validated bytes, and lookahead beyond the integer
representation is treated as beyond the input. Bounded cursor, digit and
calendar arithmetic uses explicit opt-outs with local range proofs. Stream
reads check the full allocation extent, including the extra byte used to detect
oversize input, before resizing or invoking the reader; an unrepresentable
extent reports Allocation(AllocError.CapacityOverflow). Existing parser,
output-size and atomic document-edit errors remain unchanged.
Errors
pub class TomlError:
pub fault: TomlFault
pub offset: i64
pub line: i64
pub column: i64
For parsing, offset is the byte offset from 0, and line and column start
at 1. Only \n ends a line. A column is one Unicode scalar, not one byte.
If a one-based line or column cannot fit in i64, the fault is
InputTooLarge at the still-valid byte offset, with both line and column 0
meaning unavailable. Coordinates are not saturated or wrapped. A
syntax fault points at the offending byte. A fault about a whole key or header
(DuplicateKey, DuplicateTable, KeyConflict, and a DepthExceeded that a
key or header causes) points at the first byte of its key or [. A
DepthExceeded that an array or inline table causes points at its opening
bracket, and a fault about a number or date-time at the value’s first byte.
UnexpectedEnd points one past the last byte. An allocation failure before
scanning starts points at offset 0, line 1, column 1; one while assembling the
value after a successful scan points one past the last byte.
For writing and for stream failures, offset is the number of bytes produced,
written, or read when the call failed, and line and column are 0.
NotATable reports 0, 0, 0.
TomlFault | Meaning |
|---|---|
UnexpectedByte | A byte that cannot start or continue the document here |
UnexpectedEnd | Input ended inside a key, value, or header |
InvalidUtf8 | Malformed, overlong, surrogate, or out-of-range UTF-8, anywhere |
ControlCharacter | A forbidden control character in a string, key, or comment |
InvalidEscape | An unknown escape, a bad hex digit, or an escape that is not a Unicode scalar |
InvalidNumber | A token that breaks the integer or float grammar |
NumberOutOfRange | An integer outside i64 or a float outside finite f64 |
InvalidDateTime | A malformed date-time, or one naming a day or time that does not exist |
DuplicateKey | A key defined twice in one table |
DuplicateTable | A table defined twice |
KeyConflict | A key or header that would extend a value or a closed table |
DepthExceeded | Nesting beyond max_depth |
InputTooLarge, OutputTooLarge | A size limit was reached |
NotATable | encode was given a root that is not a table |
Stream(StreamError) | A reader or writer failed; InvalidProgress if it reported an impossible count |
Allocation(AllocError) | Memory was refused |
Partial trees and buffers are released before an error returns.
Ownership and threads
A Toml owns its whole tree; moving it moves everything and dropping it
releases everything. There is no shared or global state, so trees can be moved
into a worker thread. parse copies its source and keeps no reference to it.
Toml is recursive, so the builtins that require a finite layout walk reject
it: ==, copy, print, and str.repr do not accept a Toml. Compare and
display trees through encode_text. DateTime is an ordinary class and can be
printed and compared.
Costs
| Operation | Allocates | Time |
|---|---|---|
parse, parse_text | One copy of the input; one node per key, table, and value; one index entry per named key; one scratch buffer as long as the longest key, string, or number; then the tree | Linear in the input, expected |
read_value | The input buffer, grown in chunks of 4096 bytes, then as parse without the copy | As parse, plus the reads |
encode, write_value | The whole output buffer; one key set per table with two or more entries; a temporary copy of each key and string and a temporary str per float, each released before the next | Linear in the output, expected |
encode_text | The output buffer, then a str copy | Linear |
find | Nothing | Linear in the entry count |
The parser takes runs of plain string, comment, and bare-key bytes with one
std.bytes scan each. A table lookup goes through an index
keyed by parent and name, so a table with many keys costs the same per key as
a small one. Measured with a release compiler on x86-64 Linux, parsing and then
writing a 2,688,896-byte document of 20,000 [[item]] tables and one table of
20,000 keys takes about 2 seconds, close to std.json on input of the same
size and shape.
Limits of this module
- The value-tree writer normalizes formatting; use
Documentfor edits that preserve unrelated source text. - No event or streaming interface: the whole document and its tree are held in memory. TOML’s tables can be extended anywhere in a file, so no part of the tree is final before the end of input.
- Integers are
i64and floatsf64, as TOML 1.0 requires of implementations; a larger integer is an error, never rounded. - No tree equality, path lookup, or typed decoding helpers; a typed reader is
written by hand over the tree with
find. - TOML 1.1 additions (newlines and trailing commas in inline tables,
\e,\xHH, optional seconds) are rejected.
Editing TOML documents
std.toml.Document retains validated TOML 1.0 source text and a parser index.
It takes inspiration from toml_edit, but is
implemented in checked Plenty with fallible allocation, alongside the
existing parser and writer.
document(source: &[u8], bounds: &Limits) -> Result[Document, TomlError]
document_text(text: &str, bounds: &Limits) -> Result[Document, TomlError]
Document.text() -> Result[str, TomlError]
Document.get(path: &list[str]) -> Result[Option[Toml], TomlError]
Document.set(path: &list[str], value: &Toml) -> Result[(), TomlError]
Document.insert(path: &list[str], value: &Toml) -> Result[(), TomlError]
Document.remove(path: &list[str]) -> Result[bool, TomlError]
Mutation methods take &mut Document. Paths contain decoded key components:
["package", "version"] addresses package.version, whereas ["a.b"]
addresses the single quoted key "a.b". get returns an owned value, or
Nothing for a missing key. An empty path reads the complete root table;
mutating the root is rejected with KeyConflict.
set replaces an existing value or inserts a missing one. insert requires
absence (DuplicateKey otherwise). For insertion of a scalar or array, its
parent table must already exist. Insert a Toml.Table to create a section,
including empty tables and implicit ancestors. Existing section tables can
only be replaced with tables; replacing a section with a scalar is rejected.
remove returns whether a key existed and removes a table’s descendant
statements, even when its sections are separated in the source.
import std.toml
from std.toml import Toml
def main() -> Result[(), AllocError | IoError | std.toml.TomlError]:
bounds = std.toml.limits()
source = "[package]\nname = \"demo\" # keep this comment\nversion = \"0.1.0\"\n"
mut document = std.toml.document_text(source, &bounds)?
document.set(&["package", "version"]?, &Toml.Text("0.2.0"))?
document.insert(&["package", "edition"]?, &Toml.Int(2026))?
print(document.text()?.rstrip()?)?
Ok(())
[package]
name = "demo" # keep this comment
version = "0.2.0"
edition = 2026
Preservation contract
- Parsing followed by
textreturns the exact original source, including comments, whitespace, key spelling, line endings and absence of a final newline. - Replacing an assignment changes only its value token. Its key, spaces around
=, trailing comment and newline remain byte-for-byte unchanged. The new value uses the normal writer’s spelling; replacing an array or inline table replaces its entire token, including comments inside that token. - Insertion adds a canonically formatted entry at the end of its parent section, or appends a new section. New text uses LF line endings; original CRLF bytes remain untouched. A missing newline at an insertion boundary is added.
- Removing a key or section removes the complete associated statement lines, including their trailing comments. Standalone comments and blank lines remain exactly where they were relative to other surviving text. The editor does not guess which standalone comment belongs to a removed key.
- Replacing a section removes its old statements and appends a newly formatted section. Unrelated statements and standalone comments retain their bytes.
Boundaries and costs
Arrays, including arrays of tables, are atomic values. get can read them and
remove can delete an entire array of tables, but a path through an array returns
KeyConflict; there is no implicit “last row” selection. Inline-table fields
can be read individually, but edits must replace the whole inline table. They
have no independent document statements to insert or remove. There is no
comment-editing, key-renaming, array-index path, or custom-formatting API.
Each mutation builds a candidate source and reparses it before committing.
All errors, including TomlFault.Allocation(AllocError), leave the document
unchanged. Unsupported paths return KeyConflict; syntax conflicts discovered
by validation retain the parser’s specific fault and candidate-source location.
This is an in-memory editing API, not an atomic file-writing API.
The document retains the source, arena and statement spans. Editing costs a
full source copy and parse; get copies the requested tree. This deliberately
favours a small, checked implementation for manifests over large-document
incremental editing. Limits are captured at construction: input and depth
limits apply to both original and candidate text, and output limits bound
candidate construction. No Rust dependency or infallible allocator wrapper is
introduced.
Acknowledgements
- Design follows the Rust
toml_editcrate.
std.xml
std.xml reads and writes bounded XML 1.0 events over std.io
byte streams. It is implemented in Plenty so buffer, string, attribute-list,
and nesting-stack allocation failures remain explicit. It keeps a current
token and a stack of open names, without building a document tree.
Events and format
Attribute(name: str, value: str) has public fields. Events own their strings
and attribute lists; they remain valid after another reader call.
Event | Meaning |
|---|---|
Declaration(Option[bool]) | XML 1.0 declaration; optional standalone yes/no |
Start(str, list[Attribute]) | Open an element with ordered attributes |
Empty(str, list[Attribute]) | Empty element, without a separate end event |
End(str) | Close the matching element |
Text(str) | Decoded character data, including whitespace |
CData(str) | CDATA content without delimiters |
Comment(str) | Comment content without delimiters |
ProcessingInstruction(str, str) | Target and data; separator whitespace is omitted |
Eof | Complete, well-formed document |
Input must be UTF-8, optionally with a leading UTF-8 BOM. A declaration is
optional; when present it must be first, specify version 1.0, and use UTF-8
if it names an encoding. The standalone flag is retained. There must be
exactly one root element; comments, processing instructions, and XML whitespace
may surround it. Names use XML 1.0 Fifth Edition character ranges. Duplicate
attribute names and mismatched closing names are errors.
Literal CRLF and CR normalize to LF. Literal whitespace in attributes becomes
spaces. The five predefined entities (amp, lt, gt, apos, quot) and
decimal/hexadecimal character references are decoded in text and attributes;
references to tab, LF, or CR in attributes retain those characters. Comments,
CDATA, and processing instructions do not expand references. XML-forbidden
characters, malformed UTF-8, and malformed references are rejected.
DTD declarations, including internal subsets and external entities, are
rejected. The module never loads an external resource. Names with colons and
xmlns attributes are preserved literally: it does not resolve namespaces or
check namespace bindings. It does not validate against a schema or DTD, build
a DOM, query XPath, canonicalize XML, or support encodings other than UTF-8.
Events preserve content rather than original spelling, quote choices, entity
spellings, or declaration formatting.
Limits
limits() returns a Limits with public fields. Constructors copy the limits;
changing the original afterwards has no effect.
| Field | Type | Default | Meaning |
|---|---|---|---|
max_depth | NonNegative[i64] | 64 | Open nesting including an empty element; at most 1024 |
max_input | NonNegative[i64] | 67108864 | Total input bytes consumed, including BOM |
max_output | NonNegative[i64] | 67108864 | Total encoded bytes written |
max_token | Positive[i64] | 1048576 | Raw bytes per reader event, or encoded bytes per writer event, including markup |
max_attributes | NonNegative[i64] | 4096 | Attributes per element |
A literal converts where a field is expected; a computed value needs
NonNegative.try_from(value)? or Positive.try_from(value)?. reader and
writer trap when max_depth is above 1024. A zero depth permits no root and therefore no complete document. The input
limit applies to readers and the output limit to writers. The XML declaration
has its own fixed grammar of at most three fields. The BOM is checked against
the token limit separately from the first event.
A reader uses a fixed 4096-byte read buffer and may read ahead that far. Size
limits constrain consumption, rather than the inner stream’s position. Text
between markup tokens is one event and must fit max_token; it is not split
into arbitrarily sized chunks. Memory is bounded by the buffer, current event,
and at most max_depth retained names, plus events retained by the caller.
These limits bound input-controlled work and storage, not elapsed time or a
stream that repeatedly reports interruption.
Input/output position sums are bounded by their remaining byte budgets before
advancing, even when a configured budget is i64.MAX. Internal cursor and
suffix arithmetic uses explicit opt-outs justified by held-buffer bounds;
character-reference arithmetic is guarded by the Unicode scalar ceiling.
Oversized numeric references remain InvalidReference, and byte/token
exhaustion remains InputTooLarge, OutputTooLarge, or TokenTooLarge rather
than exposing arithmetic errors. Partial writer progress retains the original
transport cause and exact accepted-byte offset.
Reading and writing
reader[R](inner: R, bounds: &Limits) -> Result[Reader[R], XmlError] owns a
stream with a public read method matching std.io.Reader.
Reader.next(&mut self) -> Result[Event, XmlError] retries interrupted reads,
checks reported progress, and returns Eof repeatedly after successful EOF.
Reader.offset() counts consumed input bytes, before line normalization.
writer[W](inner: W, bounds: &Limits) -> Result[Writer[W], XmlError] owns a
stream with public write and flush methods matching std.io.Writer.
Writer.write(&mut self, event: &Event) -> Result[(), XmlError] validates and
encodes one event before writing it. It checks nesting, root count, attribute
uniqueness, names, characters, and event ordering. Text and attribute values
are escaped; attribute tab/LF/CR and text CR inside elements are written as
character references. Comments containing -- or ending in -, CDATA
containing ]]>, and processing instructions containing ?> are rejected.
Raw CR in comments, CDATA, processing instructions, or document whitespace
still follows XML line normalization when read back.
Writer.finish(&mut self) -> Result[(), XmlError] verifies that the root was
completed and flushes the stream. It is idempotent after success. Writing
Eof calls finish; further writes are rejected. Writer.offset() counts
bytes accepted by the inner writer. Both wrappers expose their owned inner
stream; avoid reading or writing it directly while using the wrapper, since
that bypasses the wrapper’s position and document state.
A writer can stream a reader’s events straight through, but this is not an atomic file replacement: earlier events and a partially written event remain visible on failure. Use a separately managed temporary destination when that matters. See the XML lesson.
Errors
XmlError has public fields fault: Fault and offset: i64. The offset is
zero-based consumed input or accepted output bytes at detection. Validation
may detect a malformed token after consuming it; this is not necessarily the
byte containing the original mistake.
Fault distinguishes syntax, unexpected EOF, invalid names/characters/UTF-8,
invalid references, unknown entities, duplicate attributes, mismatched ends,
forbidden doctypes, unsupported declarations, each resource bound, invalid
writer state, and Stopped. Allocation(AllocError) and
Stream(StreamError) retain their causes, including stream allocation errors.
All allocation in the module is fallible. A malformed or oversized reader
event, failed writer event, or failed finish permanently stops that wrapper;
subsequent operations return Stopped. Successfully finished writers instead
reject additional writes with InvalidState.
Acknowledgements
- W3C XML 1.0 Fifth Edition supplies the character, name, document, reference, and normalization rules implemented independently.
- quick-xml informs the streaming event API; the module does not wrap or ship its code.
std.unicode
Normalization and a deliberately small classification surface, pinned to
Unicode 17.0.0 independently of the host Rust compiler. version() returns the
string "17.0.0".
| Function | Result |
|---|---|
nfd(text: &str, max_scalars: NonNegative[i64]) | Canonical decomposition |
nfc(text: &str, max_scalars: NonNegative[i64]) | Canonical decomposition and composition |
nfkd(text: &str, max_scalars: NonNegative[i64]) | Compatibility decomposition |
nfkc(text: &str, max_scalars: NonNegative[i64]) | Compatibility decomposition and canonical composition |
combining_class(scalar: &str) | Option[u32]: canonical combining class of exactly one scalar |
is_mark(scalar: &str) | Option[bool]: General_Category is Mark for exactly one scalar |
import std.unicode
from std.text import TextError
def main() -> Result[(), AllocError | IoError | TextError]:
decomposed = std.unicode.nfd("café", 64)?
print(len("café"), len(decomposed), std.unicode.nfc(&decomposed, 64)? == "café")?
accent = decomposed[4]
print(std.unicode.is_mark(&accent), std.unicode.combining_class(&accent))?
print(std.unicode.nfkc("file ①", 64)?)?
Ok(())
4 5 True
Option[bool].Some(True) Option[u32].Some(230)
file 1
Normalization returns Result[str, std.text.TextError]. It borrows input and
returns independent text. The explicit scalar limit bounds the input, fully
decomposed intermediate, and result. Exceeding any gives LimitExceeded. The
limit is NonNegative[i64], and a limit above i64.MAX // 4 traps. An empty string
fits a zero limit. Output UTF-8 is consequently at most four times the limit.
Normalization’s explicit arithmetic opt-outs rely on that validated ceiling,
successfully appended scalar counts, and positive insertion-sort indices.
They do not bypass scalar limits or allocation failures, and normalization adds
no arithmetic error variants to TextError.
The limit counts Unicode scalars, not bytes or grapheme clusters, and canonical
composition does not let an oversized decomposition bypass it.
Classification allocates nothing and returns Nothing for zero or multiple
scalars. Mark and nonzero combining class are distinct properties. This module
does not offer locale collation, case folding, or general categories beyond
Mark. Grapheme, word and sentence segmentation and display width are in the
official unicode_text package, described with
text segmentation and width. Existing
string casing and ASCII classification remain the text builtins.
Compatibility normalization can erase meaningful presentation distinctions;
callers choose it explicitly. No implicit normalization occurs in equality,
hashing, paths, decoding, or text builders.
Stable canonical ordering and blocked composition follow
UAX #15, including Hangul and composition
exclusions. Checked Plenty performs the algorithms; allocation-free scalar
lookups reuse bundled unicode-normalization 0.1.25 tables and primitives.
Canonical ordering uses stable insertion: O(n²) worst-case time for adversarial
descending combining runs, O(n) otherwise, where n is decomposed scalar count.
Storage is O(n), with two scalar lists and bounded UTF-8 construction; allocation
failure is recoverable. Choose a suitably small limit for untrusted input.
No hidden global state or thread affinity is introduced; ordinary ownership
and worker-effect checking still apply.
The complete Unicode 17 NormalizationTest.txt fixture checks all twenty
normalization equalities for each row through one compiled Plenty harness.
Unicode upgrades are deliberate source changes: replace the pinned upstream
sources, update the version function, fixture and licenses together, and rerun
the corpus, allocation tests, tutorial tests and packaging tests. Do not derive
table versions from the host toolchain. Vendoring preserves the dependency-free
runtime build and avoids the upstream iterator’s infallible scratch allocation.
Provenance and exact update steps are recorded beside the vendored sources.
The native tables add no external libraries or build-time network access and
ship inside both runtime archives. Supported target: x86_64 Linux GNU. See
the lesson and
std.text for bounded output construction and error variants.
Acknowledgements
- Normalization uses tables and lookups from unicode-normalization 0.1.25, vendored in the runtime (MIT OR Apache-2.0), with data from the Unicode Character Database 17.0.0 (Unicode-3.0).
std.units
import std.units as u provides the SI base dimensions, named derived
dimensions, units, physical constants and functions over the built-in
Quantity[T, D]. The module is ordinary
embedded Plenty: dimensions are pub dimension declarations and units are
pub const quantities. Nothing in it allocates.
import std.units as u
from std.units import Velocity, km, hour, m, s
def main() -> Result[(), IoError | AllocError]:
speed: Quantity[f64, Velocity] = 90.0 * km / hour
braking = 6.0 * m / (s * s)
distance = speed * speed / (2.0 * braking)
print(speed, speed.to(u.knot))?
print(f"{distance.to(m):.1f} m, {(speed / braking).to(s):.2f} s"?)?
Ok(())
25.0 m/s 48.59611231101511
52.1 m, 4.17 s
Every unit is a Quantity[f64, D] holding its size in the coherent SI unit,
so km stores 1000.0. Construction multiplies by the unit and to divides
by it; both are folded at compile time when the operands are constants. A
conversion between two non-coherent units divides two rounded values, so it
can round: u.foot.to(u.inch) is 12.000000000000002. The stored values
themselves are the nearest f64 to each exact definition.
Printing shows the coherent value and SI symbols in SI order (kg, m, s, A, K,
mol, cd), so a newton prints as kg*m/s^2; see
formatting.
Dimensions
| Base | Coherent unit |
|---|---|
Mass | kg |
Length | m |
Time | s |
Current | A |
Temperature | K |
Amount | mol |
Luminosity | cd |
Derived dimensions are aliases; each names a different dimension, so diagnostics can name it:
| Group | Dimensions |
|---|---|
| Geometry and motion | Area, Volume, Wavenumber, Velocity, Acceleration, Frequency, Momentum |
| Mechanics | Force, Pressure, Energy, Power, Action, Density, SurfaceTension, Gravitation |
| Flow | MassFlowRate, VolumeFlowRate, DynamicViscosity, KinematicViscosity |
| Heat | HeatFlux, SpecificEnergy, Entropy, SpecificHeatCapacity, ThermalConductivity, StefanBoltzmann |
| Chemistry | MolarMass, MolarEnergy, MolarHeatCapacity, Concentration, PerAmount, CatalyticActivity |
| Electromagnetism | Charge, Voltage, ElectricField, Capacitance, Permittivity, Resistance, Conductance, MagneticFlux, MagneticFluxDensity, Inductance, Permeability |
| Light | Illuminance, LuminousEfficacy |
SI gives some different kinds of quantity one dimension. Energy also
describes torque, Frequency also describes activity in becquerels, and
SpecificEnergy also describes absorbed and equivalent dose. A program that
needs to keep such kinds apart declares its own base dimension.
Units
SI units, their prefixed forms, eV and Wh use their symbols. Other units
use names. Micro is written u: um, us, uF.
| Dimension | Units |
|---|---|
Mass | ug, mg, g, kg, tonne, dalton, pound, ounce |
Length | pm, nm, um, mm, cm, m, km, inch, foot, yard, mile, nautical_mile, angstrom, au, light_year, parsec |
Time | ps, ns, us, ms, s, minute, hour, day, week, year |
Current | uA, mA, A, kA |
Temperature | mK, K, rankine |
Amount | umol, mmol, mol, kmol |
Luminosity | cd, lm |
One | rad, sr, degree, arcminute, arcsecond |
Area, Volume | hectare, acre; litre, millilitre, cubic_metre, gallon (US liquid) |
Velocity | km_per_hour, mph, knot |
Frequency | Hz, kHz, MHz, GHz, Bq |
Force | mN, N, kN, MN, pound_force |
Pressure | Pa, hPa, kPa, MPa, GPa, mbar, bar, atm, torr, mmHg, psi |
Energy | mJ, J, kJ, MJ, GJ, Wh, kWh, MWh, GWh, eV, keV, MeV, GeV, calorie, kilocalorie, btu |
Power | uW, mW, W, kW, MW, GW, horsepower |
Charge | nC, uC, mC, C |
Voltage | uV, mV, V, kV, MV |
Capacitance | pF, nF, uF, mF, F |
Resistance, Conductance | ohm, kohm, Mohm; uS, mS, S |
| Magnetism | Wb; uT, mT, T; nH, uH, mH, H |
| Other | lx, Gy, Sv, kat |
There is no milliohm constant, because mohm could be read as milli or mega;
write 1e-3 * u.ohm. Angles are dimensionless, as in SI, so u.degree is
Quantity[f64, One] holding π/180. calorie is the thermochemical calorie
(4.184 J), btu the International Table BTU, horsepower the mechanical
horsepower (550 ft·lbf/s), mmHg the conventional millimetre of mercury and
year 365 days. Celsius and Fahrenheit are not units here; see
temperature.
std.units.catalogue adds every SI prefix on every SI
unit and the other named units of NIST SP 811. It is a separate module so that
only programs importing it pay for its 768 constants.
Constants
| Constant | Value | Status |
|---|---|---|
delta_nu_Cs | 9192631770 Hz, caesium hyperfine frequency | Exact SI defining constant |
c | 299792458 m/s, speed of light | Exact SI defining constant |
h | 6.62607015e-34 J·s, Planck constant | Exact SI defining constant |
e | 1.602176634e-19 C, elementary charge | Exact SI defining constant |
k_B | 1.380649e-23 J/K, Boltzmann constant | Exact SI defining constant |
N_A | 6.02214076e23 /mol, Avogadro constant | Exact SI defining constant |
K_cd | 683 lm/W, luminous efficacy | Exact SI defining constant |
hbar | h / (2π) | Exact, rounded to f64 |
R | N_A * k_B, molar gas constant | Exact, rounded to f64 |
sigma | 5.670374419e-8 W/(m²·K⁴), Stefan–Boltzmann constant | Exact value truncated as CODATA lists it |
g_n | 9.80665 m/s², standard gravity | Exact by definition |
atm | 101325 Pa, standard atmosphere | Exact by definition |
G | 6.67430e-11 m³/(kg·s²), gravitational constant | CODATA 2022 |
m_e, m_p, m_u | Electron, proton and atomic mass constant | CODATA 2022 |
epsilon_0, mu_0 | Vacuum permittivity and permeability | CODATA 2022 |
Functions
| Function | Result | Contract |
|---|---|---|
abs[T: IntType, D](x) | Result[Quantity[T, D], OverflowError[T]] | std.math.abs of integer storage; a signed minimum returns its overflow |
abs_float[T: FloatType, D](x) | Quantity[T, D] | std.math.abs_float of floating storage |
clamp[Num, D](x, low, high) | Quantity[Num, D] | math.min(math.max(x, low), high), so low > high gives high; any storage |
sum[D](items: &list[Quantity[f64, D]]) | Quantity[f64, D] | Neumaier compensated sum; an empty list gives zero |
mean[D](items: &list[Quantity[f64, D]]) | Option[Quantity[f64, D]] | sum(items) / len(items); Nothing for an empty list |
sqrt[D](x: Quantity[f64, D]) | Result[Quantity[f64, Root[D, 2]], MathError] | std.math.sqrt, with its errors |
cbrt[D](x: Quantity[f64, D]) | Result[Quantity[f64, Root[D, 3]], MathError] | std.math.cbrt, with its errors |
hypot[D](a, b) | Quantity[f64, D] | sqrt(a*a + b*b) scaled to avoid intermediate overflow; infinity if either side is infinite |
exp, log, log2, log10 | Result[Quantity[f64, One], MathError] | Take Quantity[f64, One]; std.math errors |
sin, cos, tan | Result[Quantity[f64, One], MathError] | Take an angle as Quantity[f64, One] in radians |
asin, acos, atan | Result[Quantity[f64, One], MathError] | Return an angle in radians |
atan2[D](y, x) | Result[Quantity[f64, One], MathError] | Both arguments have dimension D; the angle of (x, y) |
Every argument is a copied quantity, except sum and mean, which borrow the
list. Functions built on std.math take f64 storage, as std.math does.
abs takes integer storage and abs_float floating storage, as in
std.math; clamp accepts any storage. The generic std.math
functions min, max, min_all and max_all already accept quantities, so
this module does not repeat them. math.sum and math.prod take plain
integers and math.abs a plain integer, so they do not accept quantities.
A square or cube root whose dimension is not whole is a compile error at the call that needs it:
import std.units as u
import std.math as math
def main() -> Result[(), math.MathError]:
side = u.sqrt(2.0 * u.litre)?
Ok(())
`Root[Pow[std.units.Length, 3], 2]`: not a whole dimension
A root derives its dimension, so a formula’s result is checked. Falling
through a tunnel across a uniform-density Earth is simple harmonic motion: the
trip to the centre takes a quarter period, (π/2)·sqrt(R/g), and the speed
there is sqrt(g·R):
import std.units as u
from std.units import km, m, s, minute
import std.math as math
def main() -> Result[(), IoError | AllocError | math.MathError]:
radius = 6371.0 * km
to_centre = f64.PI / 2.0 * u.sqrt(radius / u.g_n)?
top_speed = u.sqrt(u.g_n * radius)?
print(f"{to_centre.to(minute):.1f} min, {top_speed.to(km / s):.2f} km/s"?)?
Ok(())
21.1 min, 7.90 km/s
Dimensionless results stay quantities, so an angle or a ratio combines with
other quantities without .get():
import std.units as u
from std.units import m
import std.math as math
def main() -> Result[(), IoError | AllocError | math.MathError]:
rise = 3.0 * m
run = 4.0 * m
angle = u.atan2(rise, run)?
slope = u.hypot(rise, run)
print(f"{angle.to(u.degree):.2f} degrees, {slope}"?)?
print(slope * u.cos(angle)?, u.sqrt(rise * run)?)?
lengths = [rise, run, slope]?
print(u.sum(&lengths), u.mean(&lengths))?
Ok(())
36.87 degrees, 5.0 m
4.0 m 3.4641016151377544 m
12.0 m Option[Quantity[f64, std.units.Length]].Some(Quantity[f64, std.units.Length](4.0))
Lists
| Function | Result | Contract |
|---|---|---|
wrap_list[D](values: list[f64]) | list[Quantity[f64, D]] | Consumes values and returns the same list with quantity elements |
strip_list[D](items: list[Quantity[f64, D]]) | list[f64] | Consumes items and returns the same list with plain elements |
Use these at the boundary with code that produces or expects plain numbers.
Both are infallible and run in constant time: a quantity row is stored exactly
as its f64, so the conversion replaces only the list’s type descriptor. No
element is read or copied, and nothing is allocated; see
representation.
D is given explicitly to wrap_list and inferred by strip_list:
import std.units as u
from std.units import Length, km
def main() -> Result[(), IoError | AllocError]:
laps = u.wrap_list[Length]([400.0, 400.0, 1609.344]?)
total = u.sum(&laps)
print(f"{total.to(km):.3f} km"?)?
metres = u.strip_list(laps)
print(metres)?
Ok(())
2.409 km
[400.0, 400.0, 1609.344]
Temperature and time
| Function | Result | Contract |
|---|---|---|
from_celsius(degrees: f64) | Quantity[f64, Temperature] | degrees + 273.15 kelvin |
to_celsius(t) | f64 | t in kelvin minus 273.15 |
from_fahrenheit(degrees: f64) | Quantity[f64, Temperature] | Through Celsius: (degrees - 32) * 5/9 + 273.15 |
to_fahrenheit(t) | f64 | Through Celsius: (t - 273.15) * 9/5 + 32 |
from_duration(span: &Duration) | Quantity[f64, Time] | Seconds in f64 |
to_duration(t: Quantity[f64, Time]) | Result[Duration, RangeError] | Rounds to the nearest nanosecond; RangeError.OutOfRange for a negative, NaN or too-large time |
Celsius and Fahrenheit have offset zeros, so they are not multiplicative units.
They convert at the boundary to an absolute temperature; a temperature
difference in degrees Celsius has the same size as one in kelvin. Going
through Celsius keeps whole Celsius values exact: from_fahrenheit(212.0) is
373.15 K.
A std.time.Duration counts nanoseconds as u64. from_duration
converts whole seconds and the remainder separately, but the result is still
an f64: spans longer than about 104 days (2^53 nanoseconds) no longer resolve
single nanoseconds.
import std.units as u
import std.time as time
def main() -> Result[(), IoError | RangeError]:
boil = u.from_fahrenheit(212.0)
print(boil, u.to_celsius(boil), u.to_fahrenheit(u.from_celsius(-40.0)))?
timeout = u.to_duration(1.5 * u.minute)?
print(timeout.as_seconds(), u.from_duration(&time.milliseconds(250)))?
print(u.to_duration(-1.0 * u.s).is_err())?
Ok(())
373.15 K 100.0 -40.0
90 0.25 s
True
Limits
- Units and functions use
f64storage. A quantity withf32storage converts withQuantity[f32, D].from(q). wrap_listandstrip_listtakef64storage only.- Named units are not printed: a joule prints as
kg*m^2/s^2.
Acknowledgements
- Exact unit definitions are from the SI Brochure (9th edition) and NIST SP 811, appendix B.
- Measured constants are the CODATA 2022 recommended values.
- misu and Frink informed the choice of units; Frink’s samples suggested the tunnel example.
std.units.catalogue
import std.units.catalogue as uc adds every SI prefix on every SI unit and
the other named units of NIST SP 811
(2008), appendix B, to std.units. Its units are the same
Quantity[f64, D] values over the std.units dimensions, so they mix freely
with std.units units and constants. Nothing in it allocates.
import std.units as u
import std.units.catalogue as uc
def main() -> Result[(), IoError | AllocError]:
gold = 3.0 * uc.troy_ounce
print(f"{gold.to(u.g):.2f} g"?)?
battery = 3000.0 * uc.milliampere_hour * (3.7 * u.V)
print(f"{battery.to(u.Wh):.1f} Wh"?)?
lot = 2.5 * uc.survey_acre
print(f"{lot.to(u.acre):.6f} international acres"?)?
print(uc.Gm.to(u.km), uc.fs.to(u.ps))?
Ok(())
93.31 g
11.1 Wh
2.500010 international acres
1000000.0 0.001
The module holds 768 constants. Constant evaluation costs about 10 ms of compile time for a program that imports it, and nothing for one that does not.
Prefixed SI units
The module defines each of the 24 SI prefixes, quecto (q, 10⁻³⁰) to quetta
(Q, 10³⁰), on each SI base unit and each coherent derived unit with a special
name except the steradian and the degree Celsius: g (prefixes apply to the
gram), m, s, A, K, mol, cd, rad, Hz, N, Pa, J, W, C,
V, F, ohm, S, Wb, T, H, lm, lx, Bq, Gy, Sv and kat.
Names follow std.units: prefix symbol then unit symbol, with micro written
u and deca da, as in ug, dam, GPa and Mohm. Each value is a
power-of-ten literal times the coherent unit, so it is the nearest f64 to the
exact multiple.
Two prefixed symbols cannot be names. The attosecond is attosecond, because
as is a keyword, and the milliohm is milliohm, because std.units rejects
mohm as readable as milli or mega.
Prefixed forms that std.units also defines, such as km, hold the same
values. The catalogue also adds microlitre, centilitre, decilitre,
hectolitre, kilolitre, megalitre, meV, TeV and TWh.
Named units
| Group | Units |
|---|---|
| Length | fermi, micron, mil, microinch, point, pica, printers_point, printers_pica, survey_foot, survey_mile, chain, rod, fathom |
| Area | are, barn, circular_mil, survey_acre, darcy |
| Volume | stere, acre_foot, barrel, quart, pint, cup, gill, fluid_ounce, tablespoon, teaspoon, bushel, peck, dry_quart, dry_pint, imperial_gallon, imperial_gill, imperial_fluid_ounce, cord, register_ton |
| Mass | carat, grain, pennyweight, troy_ounce, troy_pound, short_hundredweight, long_hundredweight, short_ton, long_ton, assay_ton, slug |
| Time | shake, sidereal_day, sidereal_hour, sidereal_minute, sidereal_second, sidereal_year, tropical_year |
| Force | dyne, kilogram_force, ounce_force, poundal, kip, ton_force |
| Pressure | technical_atmosphere, ksi, cmHg, inHg, ftHg, mmH2O, cmH2O, inH2O, ftH2O |
| Energy | erg, calorie_IT, kilocalorie_IT, btu_th, therm_EC, therm_US, quad, ton_of_tnt |
| Power | horsepower_electric, horsepower_metric, ton_of_refrigeration |
| Viscosity | poise, centipoise, rhe, stokes, centistokes |
| Electricity and magnetism | ampere_hour, milliampere_hour, faraday, mho, debye, gauss, gamma, maxwell, oersted |
| Radiation | curie, rad_dose, rem, roentgen |
| Light | footcandle, phot, stilb, lambert, footlambert |
| Angle | revolution, gon, angular_mil |
| Other | gal, kayser, tex, denier |
Each unit is built from its definition rather than from SP 811’s rounded
factor: survey_foot is 1200/3937 m, survey_acre 43,560 square survey feet,
btu_th the thermochemical calorie per gram kelvin times a pound and a degree
Fahrenheit, and slug one pound-force second squared per foot. A unit whose
size is measured rather than defined, such as sidereal_day (86,164.0905 s)
or tropical_year (365.24219 days, J2000), uses the current value.
Some units need a choice of name or variant:
pointandpicaare the desktop-publishing point (1/72 in) and pica (1/6 in);printers_point(0.013837 in) andprinters_picaare the older American printer’s units.rad_doseis the rad of absorbed dose (0.01 Gy).u.radis the radian.galis the gal of acceleration (0.01 m/s²); the US gallon isu.gallon.quart,pint,cup,gill,fluid_ounce,tablespoonandteaspoonare US liquid measures;dry_quart,dry_pint,peckandbushelare US dry measures. Imperial measures have animperial_prefix.calorie_ITandkilocalorie_ITare International Table calories;u.calorieis the thermochemical calorie.btu_this the thermochemical BTU;u.btuis the International Table BTU.- The conventional mercury and water columns (
cmHg,inH2Oand so on) use 13,595.1 kg/m³ and 1,000 kg/m³ under standard gravity. faradayis e·N_A·mol, exact in the 2019 SI; SP 811 printed the earlier CODATA value.
Compound units have no constants: write u.btu * u.inch / (u.s * u.foot * u.foot * u.rankine) for a BTU-inch per second square foot degree Fahrenheit.
Square and cubic customary units are products too, such as u.foot * u.foot.
Dimensions
The catalogue declares the dimensions its units need that std.units lacks.
Each names a distinct normal form, as the std.units aliases do.
| Dimension | Definition | Units |
|---|---|---|
LinearDensity | u.Mass / u.Length | tex, denier |
ElectricDipoleMoment | u.Charge * u.Length | debye |
MagneticFieldStrength | u.Current / u.Length | oersted |
Exposure | u.Charge / u.Mass | roentgen |
Luminance | u.Luminosity / u.Area | stilb, lambert, footlambert |
rhe has type Quantity[f64, One / u.DynamicViscosity].
Omissions
- The CGS electrostatic and electromagnetic units (
abampere,statvolt,biot,franklin,gilbert,unit poleand the rest), whose systems give charge different dimensions. The gauss, maxwell and oersted remain, because their SI equivalents are unambiguous and they are still in use. - Variants defined at a historical reference temperature: the mean, 39 °F, 59 °F and 60 °F BTUs, the mean, 15 °C and 20 °C calories, the perm, and mercury and water columns at a stated temperature.
- Units without a fixed definition: the boiler and water horsepowers and the clo.
- Duplicates of
std.unitsunits under another name, such as the metric ton and kilopond.
Checks
Tests compare every catalogue unit, and the named units of std.units, with
the factor SP 811 table B.8 prints: within half a unit in its last digit for a
rounded factor, and within four ulps for an exact one. Another test checks
that each prefix exists on each SI unit with the right value and dimension.
Acknowledgements
- Units and exact factors are from NIST SP 811 (2008), appendix B, a US government work, and the SI Brochure (9th edition); the ronna, quetta, ronto and quecto prefixes were added by the 27th CGPM (2022).
- Frink’s unit file served as a checklist; no text was taken from it.
std.url
import std.url parses and formats absolute URLs, percent-encodes data for a
named URL component, and splits and builds query strings. It has one error
type, UrlError.
The module is checked Plenty code over std.bytes, with no native
primitives of its own. It is available on the supported x86-64 Linux GNU
target. A URL is not a filesystem path: std.path is a separate
type and nothing converts between them.
import std.url
from std.url import Component, Pair, UrlError
def main() -> Result[(), AllocError | IoError | UrlError]:
url = std.url.parse("https://example.com:8443/search?q=caf%C3%A9&page=2#top", 2048)?
print(url.scheme, url.host, url.port, url.path)?
match &url.query:
case Some(query):
for pair in std.url.query_pairs(query)?:
print(pair.key, "=", pair.value)?
case Nothing:
print("no query")?
print(std.url.encode("a b/c", Component.Segment)?)?
print(std.url.query_text(&[Pair("q", "fish & chips"), Pair("n", "1")]?)?)?
Ok(())
https Option[str].Some("example.com") Option[i64].Some(8443) /search
q = café
page = 2
a%20b%2Fc
q=fish%20%26%20chips&n=1
The accepted syntax
std.url implements the generic syntax of RFC 3986, with these stated
differences. It is not the WHATWG URL Standard and does not imitate browsers.
| Rule | Behavior |
|---|---|
| Scheme | Required. Relative references such as /a/b, //host/a, and ?q are MissingScheme |
| Characters | ASCII only, exactly the characters RFC 3986 allows in each component. Spaces, controls, "<>\^{} and the backtick, [/] outside an address literal, and all non-ASCII text are InvalidByte |
| Escapes | % must be followed by two hex digits of either case, or the URL is InvalidEscape |
| Userinfo | Everything before the first @ of the authority. A second @ is an error, never a choice of host |
| Host | A registered name, or [IPv6 address], or [v…] (IPvFuture). Empty is allowed. A dotted IPv4 address is an ordinary registered name |
| IPv6 | The RFC 3986 IPv6address grammar: one optional ::, an optional trailing dotted IPv4 address whose octets have no leading zeros. Zone identifiers (%25eth0) are rejected |
| Port | Narrower than the RFC: one to five digits, no leading zero, at most 65535. An empty port (http://h:/) is InvalidPort |
| Internationalized names | No IDNA processing. A Unicode host must already be Punycode (xn--...) or percent-encoded UTF-8 |
| Normalization | None. Case, . and .. segments, escapes, default ports, and empty components are kept exactly as written |
Because nothing is normalized and the port has one spelling, every accepted URL formats back to the identical text.
Url
pub class Url:
pub scheme: str
pub userinfo: Option[str]
pub host: Option[str]
pub port: Option[i64]
pub path: str
pub query: Option[str]
pub fragment: Option[str]
Components hold the text between the delimiters, still percent-encoded and
without the delimiters themselves. Nothing means the delimiter was absent;
Some("") means it was present with nothing after it, so s:, s:?, and
s:// are three different values. host is Nothing exactly when the URL has
no // authority. An address literal keeps its brackets: [::1].
| Function | Result | Contract |
|---|---|---|
std.url.parse(text: &str, max_bytes: NonNegative[i64]) | Result[Url, UrlError] | Split and validate text, whose UTF-8 length must not exceed max_bytes |
url.to_string() | Result[str, UrlError] | Validate every component and join them |
parse checks the length first: LimitExceeded for a longer text. It then
reports the first violation from left to right.
to_string exists so that a Url built or edited by a program cannot produce
text that parses differently. It applies the rules of parse to each field,
and also returns:
MissingHostwhenuserinfoorportis present withouthost;InvalidPortwhenportis outside0to65535;InvalidBytewhenhostis present and a nonemptypathdoes not start with/, orhostis absent andpathstarts with//.
A successful to_string result always parses back to equal components.
Compare URLs only after deciding what equivalence means for your scheme. The
module does not lowercase schemes or hosts and does not decode escapes, so
HTTP://A/%7e and http://a/~ are different values.
Percent encoding
| Function | Result | Contract |
|---|---|---|
encode(text: &str, component) | Result[str, UrlError] | Escape the UTF-8 bytes of text for component |
encode_bytes(source: &[u8], component) | Result[Bytes, UrlError] | The same for a byte view |
decode(text: &str) | Result[str, UrlError] | Replace each %hh; the result must be UTF-8 |
decode_bytes(source: &[u8]) | Result[Bytes, UrlError] | Replace each %hh in a byte view; an InvalidEscape index is relative to the view |
Ranges follow the std.bytes range rule.
Letters, digits, and -._~ are never escaped. % is always escaped, so
encoding text that already contains escapes encodes it again. Escapes are
written with uppercase hex digits. Each Component additionally keeps:
| Component | Also kept | Use for |
|---|---|---|
Userinfo | !$&'()*+,;= | A user name or a password; : is escaped, so join them yourself |
Segment | !$&'()*+,;=:@ | One path segment; / is escaped |
Path | !$&'()*+,;=:@/ | A whole path whose / are separators |
Query | !$&'()*+,;=:@/? | A whole query string |
QueryPart | !$'()*,;:@/? | One key or one value; &, =, and + are escaped |
Fragment | !$&'()*+,;=:@/? | A fragment |
Userinfo output is also valid in a registered host name.
Decoding is the same for every component: %hh becomes one byte and every
other byte is copied. + stays +; the application/x-www-form-urlencoded
reading of + as a space is not implemented. decode returns InvalidUtf8
when the bytes are not UTF-8; use decode_bytes for binary data.
Decoded text can contain any character, including /, ?, NUL, and line
breaks. Split a URL into components before decoding, and never parse decoded
text as a URL or a path.
Query strings
pub class Pair:
pub key: str
pub value: str
| Function | Result | Contract |
|---|---|---|
query_pairs(query: &str) | Result[list[Pair], UrlError] | Split at &, split each piece at its first =, decode both sides |
query_text(pairs: &list[Pair]) | Result[str, UrlError] | Join key=value with &, each side encoded as QueryPart |
Every &-separated piece is one pair, in order, including empty pieces and
repeated keys. A piece without = has an empty value, so a and a= read the
same; query_text always writes the =. The empty query has no pairs.
query_pairs accepts any byte that is not part of a malformed escape; pass it a
Url.query to get the stricter component check.
Errors
pub enum UrlError:
LimitExceeded
MissingScheme
MissingHost
InvalidByte(i64)
InvalidEscape(i64)
InvalidHost(i64)
InvalidPort(i64)
InvalidUtf8
Allocation(AllocError)
| Variant | Meaning |
|---|---|
LimitExceeded | The text is longer than max_bytes |
MissingScheme | No scheme and : before the first /, ?, or # |
MissingHost | to_string was given userinfo or a port without a host |
InvalidByte(index) | The byte at index is not allowed where it appears |
InvalidEscape(index) | The % at index is not followed by two hex digits |
InvalidHost(index) | The address literal starting at index is malformed, or the byte at index follows its ] and is not : |
InvalidPort(index) | The port starting at index is not an accepted number |
InvalidUtf8 | Decoded bytes are not UTF-8 |
Allocation(cause) | Storage could not be allocated |
An index is a UTF-8 byte offset into the input. For to_string it is an offset
into the text that would have been produced; for the byte-range functions it
is a position in the source buffer, not relative to start. A failed
operation releases anything it allocated. Wrap a result with
std.error.context to name the operation.
Ownership and threads
Url and Pair own their strings, so they are resources: assignment moves
them, and the ordinary loan rules
apply to their fields. All inputs are shared loans. The module holds no global
state; its functions pass the ordinary worker-effect checks and may run in any
thread.
Costs
Every operation is linear in its input and allocates in proportion to it.
| Operation | Allocates |
|---|---|
parse | One copy of the text, then one string per present component. A text of more than max_bytes UTF-8 bytes is rejected before the copy, so the copy never exceeds max_bytes |
to_string | One growing buffer, a temporary per component, and the result |
encode_bytes, decode_bytes | One buffer of exactly the output size, after validation |
encode, decode | A Bytes copy of the text, the coded buffer, then the str |
query_pairs | A copy of the query, the list, and two strings per pair |
query_text | One growing buffer and two temporaries per pair |
Encoded output is at most three times its input. The encoder counts literal
bytes, then checks 3 * escaped + literals before allocating its result.
Unrepresentable output sizes are Allocation(AllocError.CapacityOverflow).
Parser, decoder and query cursors use documented bounds-proven arithmetic;
terminal literal runs and final query pairs stop at the input endpoint instead
of advancing past it. A serializer error offset that cannot fit i64 also
returns allocation-capacity overflow. Caller arithmetic modes do not change
these contracts.
Every byte goes through checked single-element Bytes calls, several times
per byte. An earlier release-compiler measurement, before checked-default
adoption, on x86-64 Linux pinned to one core found that parsing one 91-byte URL
with all seven components took about 49 µs:
roughly half a microsecond per input byte.
Limits
- No relative references and no reference resolution (RFC 3986 section 5).
- No normalization or equivalence: no case folding, dot-segment removal, escape normalization, or default-port knowledge.
- No IDNA or Punycode conversion, and no scheme-specific rules such as
httprequiring a host. - No IPv6 zone identifiers. IPv4 addresses are not distinguished from registered names.
- No form encoding (
+as a space) and no;pair separator. - Query pairs are a list. There is no keyed lookup helper; scan the list.
std.uuid
UUID values for records, requests and files, with random v4 and timestamp-prefixed
v7 generation. The implementation is checked Plenty using
std.bytes, std.secrets and std.time.
No extra native library is linked. Start with the
tutorial; the
signature index lists the complete API.
import std.bytes
import std.secrets
import std.uuid
def main() -> Result[(), AllocError | IoError | ParseError | std.bytes.BytesError | std.secrets.SecretsError]:
id = std.uuid.parse("6BA7B810-9DAD-11D1-80B4-00C04FD430C8")?
print(id.to_string()?, id.version(), id.is_rfc_variant())?
same = std.uuid.from_bytes(&id.to_bytes()?)?
print(same == id)?
fresh = std.uuid.v4()?
print(len(fresh.to_string()?), fresh.version(), fresh == id)?
Ok(())
6ba7b810-9dad-11d1-80b4-00c04fd430c8 Option[u8].Some(1) True
True
36 Option[u8].Some(4) False
Values and conversion
Uuid owns two private u64 fields in network order: most significant half
first. It has ordinary class move semantics; == and != compare the complete
128-bit value. All methods borrow their receiver.
| Operation | Result and contract |
|---|---|
parse(text: &str) | Result[Uuid, ParseError]; exactly 36 ASCII characters in 8-4-4-4-12 hyphenated hex form, accepting either digit case |
from_bytes(source: &[u8]) | Result[Uuid, ParseError]; exactly 16 bytes, in big-endian/network order; accepts a view of Bytes, a list or a byte literal |
from_u64_pair(high: u64, low: u64) | Uuid; preserves the supplied halves |
nil() | Uuid; all zero bits |
id.to_string() | Result[str, BytesError]; lowercase hyphenated text |
id.to_bytes() | Result[Bytes, BytesError]; an owned 16-byte buffer |
id.write_bytes(target: &mut Bytes, offset: i64) | (); overwrites exactly 16 existing bytes; no growth or allocation |
id.as_u64_pair() | (u64, u64); a copyable, numerically ordered sorting key |
id.compare(other: &Uuid) | i64; -1, 0, or 1 in unsigned 128-bit order |
id.is_nil() | bool; all bits are zero |
id.is_rfc_variant() | bool; the two leading variant bits are 10 |
id.version() | Option[u8]; the raw four-bit version for the RFC variant, otherwise Nothing |
id.unix_milliseconds() | Option[u64]; the leading 48 bits for RFC-variant v7, otherwise Nothing |
Parsing and byte conversion preserve all bits, including nil, max, unrecognized
versions and non-RFC variants. Parsing does not validate generation history.
Malformed text and wrong byte lengths return ParseError.Invalid. There is no
trimming, URN/braced/compact spelling, little-endian GUID convention, or implicit
normalization. write_bytes validates the whole range before writing; invalid
offsets trap with Uuid.write_bytes: offset outside target buffer before writing.
Parsing, reading bytes, comparison, and numeric construction do not allocate.
to_bytes allocates its result; to_string uses a 36-byte scratch buffer and
allocates its result. Allocation failures are BytesError.Allocation.
Normal print(id) retains structural class formatting: use
print(id.to_string()?)? for canonical text. Class relational operators and
dictionary keys are not available; use compare, or sort_by_key with
as_u64_pair(). Use canonical text when a dictionary key is needed.
Generation
v4() -> Result[Uuid, SecretsError]
v7() -> Result[Uuid, RangeError | SecretsError]
v4 obtains 16 OS-random bytes and sets the version and variant, leaving 122
random bits. v7 reads wall_now() once and obtains 10 random bytes: its layout
is a 48-bit Unix millisecond timestamp and 74 random bits, plus version/variant.
Both propagate allocation and entropy errors from std.secrets; neither falls
back to a pseudorandom seed, and neither maintains a process-wide counter.
The clock value must have nonnegative seconds, nanoseconds below one billion,
and a millisecond value at most 281474976710655. Sub-millisecond fractions
are rounded down; invalid timestamps return RangeError.OutOfRange before
requesting entropy. Generation uses a short-lived entropy buffer; a batch
generator with reusable scratch is not part of this API.
v7 sorts by its recorded millisecond, not by call order. Values in the same
millisecond have random order, and a backwards wall-clock adjustment can yield
a smaller UUID. There is no strict monotonicity or mathematical uniqueness
guarantee. The timestamp is visible to anyone reading the identifier. UUIDs are
identifiers, not authentication credentials; use std.secrets tokens for that
purpose.
For deterministic tests or an explicitly chosen entropy device:
v4_from[R: Reader](source: &mut Source[R]) -> Result[Uuid, SecretsError]
v7_from[R: Reader](at: &Timestamp, source: &mut Source[R]) -> Result[Uuid, RangeError | SecretsError]
Source is std.secrets.Source, constructed with source_from(reader) or
os(). These functions apply the same masking and timestamp checks as the
convenience calls. Production callers must supply cryptographic entropy;
replaying a test source can repeat identifiers. Validation happens before any
read, and short/interrupted reads and failures follow the secrets source’s
existing policy. No clock or entropy callback is stored in a UUID.
References and scope
The layout and test vectors follow RFC 9562,
especially sections 5.4/5.7 and appendices A.3/A.6. API references are
Python’s uuid and
Rust’s uuid::Uuid.
Plenty deliberately accepts fewer text spellings than either reference and
does not adopt Python’s v7 counter or Rust’s stateful now_v7 ordering policy.
Name-based v3/v5 and v1/v6/v8 generators are not implemented.
Arithmetic bounds
Parsing and formatting use fixed 36-byte text and 16-byte wire layouts. Output offsets are validated before subtraction or mutation, and UUIDv7 validates the 48-bit millisecond range before requesting entropy. Bounds-proven arithmetic opt-outs preserve existing parse, range, entropy, and allocation errors without adding arithmetic errors to the public API.
Acknowledgements
- API modelled on Python’s
uuidand Rust’suuid::Uuid.
Toolchain and implementation
Building and running covers compiler commands, native linking, supported targets, and library artifacts.
Architecture describes the frontend, checking, specialization, Cranelift emission, runtime packaging, storage, and cleanup mechanisms. The support index links to implemented contracts and their principal limits.
Contributor practices live in Development.
Building and running programs
Plenty compiles ahead of time through Cranelift. Checking needs no native linker;
running and executable builds use a cc-compatible driver and the runtime archive
embedded in the compiler. Program builds invoke neither Cargo nor rustc.
- Installing a release: release archives, requirements, and
PATHsetup. - Commands: checking, running, executable and library output.
- Warnings: lints that checking reports, such as
redundant-copy. - Neovim: file detection, syntax highlighting, editing defaults, and language server setup.
- Language server:
plenty lspdiagnostics, hover, go to definition, and completion. - Packages:
plenty.toml, path dependencies,plenty.lock, profiles, and theplenty build,run,check, andtestcommands. - Application exits: statuses and returned-error reports.
- Native linking and targets: driver arguments, object output, runtime extraction, and supported platforms.
- Native engines: when a large native library ships as a separate shared library, and the contract it follows.
- Application distributions: explicit relocatable executable,
engine and notices bundles with
--dist. - Library artifacts: generated files, contract inspection, and publication guarantees.
- Fuzzing: coverage-instrumented libFuzzer builds with
--fuzz.
For source-level entrypoint rules, see source and modules. For runtime packaging, see architecture.
Installing a release
Each Plenty release on the project’s GitHub Releases page provides a compiler archive
for each Linux target, such as plenty-VERSION-x86_64-unknown-linux-gnu.tar.gz
or plenty-VERSION-aarch64-unknown-linux-gnu.tar.gz, and its .sha256 checksum.
Download the archive matching your machine’s architecture.
Extract it anywhere and put its bin/ directory on PATH. No installer, root
access, fixed location, Rust, Cargo, or source checkout is needed. The README
in the archive repeats these instructions.
The Linux archive includes the native engines for std.dataframe and
std.linalg in bin/, where the compiler finds them. Programs that do not reach
an engine’s native operations do not depend on it. To ship a program that does,
follow application deployment.
For a step-by-step download guide and persistent PATH examples for Bash, Zsh, and Fish, start with Setting up.
plenty-VERSION-x86_64-unknown-linux-gnu/
bin/plenty compiler and project tool
bin/plentyc single-file compiler
bin/libplenty_dataframe_v1.so
Polars engine for std.dataframe
bin/dataframe-notices/ its manifest and licenses
bin/libplenty_linalg_v1.so faer engine for std.linalg
bin/linalg-notices/ its manifest and licenses
README.md requirements and installation
LICENSE-MIT Plenty's license: MIT or Apache-2.0,
LICENSE-APACHE at your option
THIRD-PARTY-NOTICES.md licenses of software built into the compiler
licenses/rust/ the Rust standard library's notices
The standard-library sources and ordinary runtime are built into the
executables, so the compiler works from any directory. Dataframe and linear
algebra operations use the shared engines in bin/; keep them beside the
compiler.
Requirements
| Requirement | Detail |
|---|---|
| Platform | x86-64 or ARM64 (AArch64) Linux with glibc 2.28 or later: RHEL/AlmaLinux/Rocky 8, Debian 10, Ubuntu 18.10, Fedora 29, or newer |
| Kernel | The Rust standard library’s minimum, 3.2 on x86-64 and 4.1 on ARM64; old kernels are not tested |
| CPU | Any x86-64 CPU (the original SSE2 instruction set) or any ARMv8-A CPU; the compiler and generated programs use only that baseline |
| Linking | A cc driver, linker, startup files, and C library development files |
Rust’s AArch64 Linux target sets the 4.1 kernel minimum. ARM64 archives are built and tested on native ARM runners; page sizes other than 4 KiB are not tested.
musl distributions such as Alpine are outside the native target set; see native linking and targets.
Plenty generates machine code itself and runs cc only to link it with the
embedded runtime and the system C library, as described in
native linking. It is not a self-contained toolchain. Install the
linking prerequisites:
# Debian, Ubuntu
sudo apt-get update && sudo apt-get install gcc libc6-dev
# Fedora, RHEL, AlmaLinux, Rocky Linux
sudo dnf install gcc tar gzip
# Arch Linux
sudo pacman -Syu --needed gcc
Release checks run these commands in clean AlmaLinux 8.10 (glibc 2.28), Debian 12 and 13, Ubuntu 24.04, Fedora 44, and Arch Linux containers for x86-64, and AlmaLinux 8.10 and Debian 13 for ARM64, then compile and run programs from the extracted archive. Each archive’s README lists the systems it was tested on. AlmaLinux 8 is the glibc 2.28 baseline; installing packages does not make an older system compatible.
Experimental macOS and Windows downloads
Releases also publish experimental compiler archives for macOS on Apple silicon and for 64-bit Windows:
| Archive | Requirements |
|---|---|
plenty-VERSION-aarch64-apple-darwin.tar.gz | macOS 11 or later on Apple silicon; the Xcode Command Line Tools (xcode-select --install) for linking |
plenty-VERSION-x86_64-pc-windows-msvc.zip | Windows 10, Windows Server 2016, or later on x86-64; Visual Studio Build Tools with the “Desktop development with C++” workload and the Windows SDK, which also supply the MSVC runtime the compiler uses |
Both contain bin/plenty, bin/plentyc (.exe on Windows), the README,
licenses and notices. Both include the dataframe engine and
bin/dataframe-notices/: bin/libplenty_dataframe_v1.dylib on macOS,
bin/plenty_dataframe_v1.dll on Windows. Linear algebra remains unavailable
on these two platforms. The release
workflow builds each archive natively, runs the compiler/runtime unit tests and
platform-boundary checks, then extracts it on a macOS 15 or Windows Server 2025
runner with Rust and Cargo removed from PATH, and compiles and runs a small
program. Both additionally exercise a dataframe Parquet query and relocatable
application distribution. Projects, most standard-library services such as SQLite and TLS, and
the diagnostics for a missing linker are not yet tested there, and several OS
services are incomplete; see platform support.
The executables are not notarized or code-signed for distribution. The macOS
executables carry the linker’s ad-hoc signature; a browser download is
quarantined, so run xattr -dr com.apple.quarantine on the extracted
directory, or download with curl, which does not quarantine files. Windows may show a SmartScreen warning. The
archive’s README gives installation and troubleshooting steps for each system.
Install and use
cd ~/Downloads
sha256sum -c plenty-VERSION-x86_64-unknown-linux-gnu.tar.gz.sha256
tar -xzf plenty-VERSION-x86_64-unknown-linux-gnu.tar.gz
export PATH="$HOME/Downloads/plenty-VERSION-x86_64-unknown-linux-gnu/bin:$PATH"
plenty --version
plenty --compile hello.plenty -o hello
./hello
Add the export line to your shell’s startup file to keep it. The directory
can be moved or renamed, and paths may contain spaces; update PATH to match.
Plenty writes nothing inside it: the compiled-function cache
lives in ~/.cache/plenty, and temporary files in TMPDIR.
plenty --version prints the compiler’s version and target:
plenty X.Y.Z (x86_64-unknown-linux-gnu), with X.Y.Z replaced by the installed
version (including any pre-release suffix).
Upgrade and switch versions
Each release extracts to its own versioned directory, so several can be
installed side by side. The first plenty on PATH is the one that runs:
point PATH at another release’s bin/ to switch, and check with
plenty --version or command -v plenty. Uninstall by deleting the directory.
Troubleshooting
| Symptom | Cause and fix |
|---|---|
failed to invoke linker driver `cc`, mentioning gcc or clang | No C compiler on PATH. Install the prerequisites, or pass --linker clang or another cc-compatible driver |
linker driver `cc` failed naming Scrt1.o or crt1.o, -lc, -lgcc_s, or -lpthread | The C library development files are missing; install the prerequisites (libc6-dev on Debian and Ubuntu) |
GLIBC_2.xx not found when starting plenty | The system is older than glibc 2.28 |
plenty --check FILE needs no linker.
Programs you build
Generated programs link against the C library of the machine that compiles
them, through its cc. A program built on a new distribution can need a newer
glibc than Plenty itself and fail on an older system. To deploy to older
systems, compile there, or in a container of the oldest supported
distribution, as the release itself is built.
Maintainers: see releasing Plenty.
Neovim support
The repository includes a runtime plugin in editors/nvim. It detects
*.plenty files, highlights source syntax using the active colorscheme, sets
four-space soft tabs, supplies # comment formatting, and configures the
language server for Neovim’s built-in LSP client.
Add this to init.lua, replacing the path with your Plenty checkout:
vim.opt.runtimepath:prepend('/absolute/path/to/plenty/editors/nvim')
vim.cmd('filetype plugin on')
vim.cmd('syntax enable')
Restart Neovim and open examples/classes.plenty. Check :set filetype? if
highlighting is missing; it should report plenty. The plugin directory’s
README also describes native package installation and the headless test commands.
Extension detection takes precedence over generic content detection, including
for source beginning with comments. An already-open buffer can be corrected with
:setlocal filetype=plenty after the runtime path is available.
Highlighting includes keywords, primitive and collection types, function
declarations, annotations such as @copy, numbers and suffixes, strings, byte
literals and escapes, operators, and comments.
Single- and double-quoted strings stay on one line; triple-quoted strings span
lines. See source syntax. Highlighting
uses regular expressions and does not perform semantic checks. Reserved words
can be colored even when their construct is not implemented.
Language server
With Neovim 0.11 or later, editors/nvim/lsp/plenty.lua configures the
server. Enable it after the runtime path above:
vim.lsp.enable('plenty')
-- If `plenty` is not on PATH:
-- vim.lsp.config('plenty', { cmd = { '/path/to/plenty', 'lsp' } })
vim.api.nvim_create_autocmd('LspAttach', {
callback = function(event)
local client = vim.lsp.get_client_by_id(event.data.client_id)
if not client or client.name ~= 'plenty' then
return
end
vim.lsp.inlay_hint.enable(true, { bufnr = event.buf })
vim.keymap.set('n', 'gd', vim.lsp.buf.definition, { buffer = event.buf })
vim.lsp.completion.enable(true, client.id, event.buf, { autotrigger = true })
end,
})
K shows the hover. gd goes to the definition, into a read-only copy of a
std module when the definition is there; CTRL-] does the same through
Neovim’s default LSP tag function. Errors appear as diagnostics; ]d and [d
move between them. Completion triggers after . and with CTRL-X CTRL-O.
With nvim-cmp instead of the built-in completion, leave out the
vim.lsp.completion.enable line and add the nvim_lsp source.
Neovim 0.10 has no vim.lsp.config; start the server from a FileType
autocommand instead:
vim.api.nvim_create_autocmd('FileType', {
pattern = 'plenty',
callback = function()
vim.lsp.start({
name = 'plenty',
cmd = { 'plenty', 'lsp' },
root_dir = vim.fs.root(0, { 'plenty.toml', '.git' }),
})
end,
})
Inferred type hints
The LspAttach example enables inferred types inline for Plenty buffers.
Neovim 0.10 or later provides
vim.lsp.inlay_hint.enable.
With the 0.10 startup example, add the same LspAttach callback but omit the
vim.lsp.completion.enable line (that completion API requires 0.11).
To enable or disable hints in an already-attached buffer, run:
:lua vim.lsp.inlay_hint.enable(true, { bufnr = 0 })
:lua vim.lsp.inlay_hint.enable(false, { bufnr = 0 })
A binding such as count = 42 displays an inferred : i64 after count.
These labels do not change the file. An explicit count: i64 = 42 has no
redundant hint. If hints are missing, check :checkhealth vim.lsp for an attached
Plenty server, ensure the server binary includes inlay-hint support, and resolve
check errors. See the hint coverage and limits.
No editor-wide setting is changed by the server.
Formatting, automatic block indentation, and Tree-sitter are not provided.
Language server
plenty lsp serves the Language Server Protocol
on standard input and output. It checks files with the compiler’s own front
end, so its errors and names match plenty check and plentyc --check. Neovim
is the supported editor; see Neovim support for its setup.
Features
| Request | Behavior |
|---|---|
| Diagnostics | The file’s check errors and warnings, published when it is opened, saved, or edited (after a 150 ms pause) |
| Hover | The declaration as written and its docstring, as Markdown |
| Go to definition | The declaration’s name, in the same file, another module, a dependency, or std |
| Inlay hints | Inferred types after unannotated simple local binding names, when the current check succeeds |
| Completion | Names in scope and keywords; fields, methods, enum variants, and module members after . |
Hover and definition work on functions, methods, classes, enums, enum
variants, protocols, fields, constants, type aliases, imported names, modules
in an import, locals, and parameters. A hover shows a function’s full header,
including type parameters, the receiver (self, &mut self), defaults, and
the return type, and pub when it is written. Hover on a local or parameter
shows its type as the checker inferred it. A method call resolves through its
receiver’s type, so p.dist() opens the method of p’s class. Completion
items carry the declaration and the docstring’s first line.
A docstring is the string that begins a function body; see signatures and calls. Classes, enums, and fields have no docstrings.
Inferred types on code lines
Enable inlay hints in your editor to see, for example, : i64 after the
name in count = 42. These are editor-rendered annotations, not source edits;
you can keep writing inferred bindings without inserting type annotations.
Other language servers provide inferred type annotations through this same
LSP 3.17 inlay-hint request.
The editor requests hints for a document range and decides how to display them.
See Neovim setup to enable them.
Plenty supplies type hints for new simple local bindings, including mut,
using the checker’s type spelling as hover does. Explicit annotations,
reassignments, parameters, tuple destructuring, loop and pattern bindings do
not get hints. Compiler-internal and producer-specific types (such as local
pipelines and capturing closures) are omitted. An uninstantiated generic body has no
checked hints; when instantiations disagree on a binding’s type, it is omitted.
Hints are withheld while the current check has errors, rather than guessing
or displaying types and positions from an older version of the source.
The server advertises inlayHintProvider with no resolve support and returns
complete string labels of kind Type, without text edits or extra padding.
Hint positions are immediately after the binding name, within the requested
half-open range, using the negotiated UTF-8, UTF-16 (default), or UTF-32
position encoding. Hint requests reuse the document’s analysis; a pending edit
is checked first, just as for hover. No parameter-name hints are provided.
How a file is checked
The server checks the file being edited as the entry of a compilation, with
the unsaved text of every open file in place of its saved text. A file that
defines main is checked as an application, and any other file as a library
module, as plentyc --check-module does. Inside a package (a plenty.toml
above the file’s src/), imports resolve through the package graph without
fetching or changing plenty.lock; run plenty fetch once for Git
dependencies. Outside a package, imports resolve from the file’s directory.
Checking recovers from an error in one function: the function keeps its signature, the other functions are still checked, and each error is published. Errors that stop a whole module, such as a syntax error, an unresolved import, or an invalid class declaration, are published alone; while one is present, hover, definition, and completion use the names found by the file’s last check that got further.
Definitions in an embedded std module open as a read-only copy of its
source, written to std-sources/VERSION/std/ below $PLENTY_CACHE_DIR,
$XDG_CACHE_HOME/plenty, or ~/.cache/plenty. The copies are ordinary files,
so any editor can open them, and they are rewritten when the compiler’s copy
differs.
Limits
- Names inside a generic function or method are resolved only where it is instantiated with concrete types; a method called on a protocol-bounded parameter opens the method of the class it was instantiated with, not the protocol’s requirement. A method of a generic class is instantiated only where it is used. An uninstantiated generic body resolves its module-level names and locals only.
- Builtin functions, methods of built-in types such as
listandstr, andOk,Err,Some, andNothinghave no declaration to show. - Variant names in
casepatterns are not resolved. - Locals offered by completion are those bound in the enclosing function above the cursor, by line.
- An error in a module other than the open file is shown on the file’s first line, with the other file’s position in the message.
- Each check reads the whole program again; there is no incremental reuse
between checks. A program importing
std.http.server,std.json, and five other largestdmodules checks in about 0.2 s with a release build. - Find references, document symbols, and signature help are not implemented.
- A message body that is not valid JSON is answered with a
-32700parse error and the session continues. A header without a validContent-Lengthleaves the stream unframed, so the server exits as if the input had ended. - After
shutdown, the server answers any other request with a-32600invalid-request error and ignores notifications untilexit.
Other editors
Any client that starts a server with a command line and speaks LSP over stdio
can use plenty lsp for files with the .plenty extension. No VS Code
extension is provided; a generic LSP client extension configured with the
command plenty lsp works.
Compiler commands
These commands compile single files. plentyc accepts exactly these commands;
plenty accepts them too, alongside the project commands.
| Command | Behavior |
|---|---|
plenty FILE | Reuse a cached executable or compile/link, run privately, then remove the temporary directory |
plenty --check FILE | Check an application without emitting or executing code |
plenty --check-module FILE | Check a module and imports without requiring main |
plenty --compile FILE -o OUT | Produce a native executable (dynamic dependencies remain) |
plenty --compile FILE --dist DIR | Publish a new application distribution with only reached engines and their notices |
plenty --emit-object FILE -o OUT | Emit one application object, including imported modules; requires main |
plenty --dump STAGES FILE -o DIR | Write each function’s IR and machine code as text; --dump-std and --dump-filter TEXT select functions |
plenty --fuzz FILE -o OUT | Produce a coverage-instrumented libFuzzer executable that runs main once per input; --libfuzzer PATH selects libFuzzer |
plenty --shared-library FILE --library-name NAME -o OUT | Build a shared C library with generated header and interface |
plenty --static-library FILE --library-name NAME -o OUT | Build a static C library with generated header and interface |
plenty --extract-interface LIBRARY --library-name NAME -o OUT | Extract embedded source metadata without executing native code |
plenty --verify-interface LIBRARY INTERFACE | Verify a generated contract and compatibility symbol |
plenty --runtime-interface FILE --library-name NAME -o OUT.plentyi | Generate a typed runtime loader from checked export source, without loading a library or invoking native tools |
plenty --emit-runtime DIR | Extract libplenty_runtime.a, link-args.txt, and target.txt |
plenty --print-target | Print the packaged runtime’s target and CPU baseline |
plenty --version (-V) | Print the compiler’s version and target: plenty X.Y.Z (x86_64-unknown-linux-gnu), where X.Y.Z is the installed version |
Checking modes print warnings on stderr; they do not change the exit status.
--check-all-methods with --check or --check-module also checks every
method of every instance the program creates of a generic class or enum it
declares, including methods nothing uses, which are otherwise
checked only when used.
Types from std keep per-use checking.
Other modes reject it: compiling would emit those methods, and a reused cached
object would skip their check.
Code generation is optimized by default, including plentyc, object and shared
library output, fuzz targets and dumps. --fast-compile skips speed optimizations
for shorter edit-run cycles in large projects; it preserves all language checks.
--release remains a compatibility alias for the optimized default. See
profiles.
--unchecked MODE selects what plain
unchecked: suites
outside std do with their arithmetic and indexing preconditions: trap, the default,
checks them; assume omits the checks, so a violation is undefined behavior.
Assumed list, slice-view and Bytes indices must be nonnegative and in range;
bounds checks and negative-index normalization are omitted.
It applies to compiling modes, including plenty build, run, test and
scripts; checking modes reject it.
--test-hooks with FILE or --compile enables
failure injection in an ordinary
executable. It does not arm failures automatically. Default builds have no hooks;
library, object-only, fuzz, and checking modes reject this flag.
--compile and --shared-library output is
stripped of its symbol table and
debug sections; --no-strip keeps them.
Compiled functions are reused from the
compile cache, and so is a whole program’s
object when none of its sources changed. --no-cache compiles without it,
--no-program-cache reuses functions but not programs or executables, --cache-stats
reports its hits and misses after compiling, and plenty cache stats and
plenty cache clean inspect and empty it.
With no arguments, plenty displays help. --module-root DIR selects the source
root for file commands. --package-graph GRAPH instead resolves imports through
a package graph file for FILE, --check,
--check-module, and --compile. Library builds do not require main; their exported
contracts follow the C interface rules.
Compile-and-run inherits standard streams, environment, and working directory.
It propagates the child’s exit code; Unix signal termination becomes 128 plus
the signal number. Temporary objects and runtime files are removed on success
and failure. On supported Linux toolchains, single-file runs also reuse a
cached executable by
default, avoiding linking when its inputs match. Other runs link normally.
Unix scripts can use #!/usr/bin/env plenty as their first line and be made
executable with chmod +x FILE; hashbang launches use the same caches.
For argument forwarding, optional embedded manifests and isolated dependency
resolution, use plenty script, with the Unix shebang
#!/usr/bin/env -S plenty script. The compiler shortcuts above remain unchanged.
Warnings
A warning reports code the checker accepts but that has a simpler correct form. Warnings never reject a program and never change an exit status.
Checking reports them: plenty --check, plenty --check-module,
plenty check, and the language server. Compiling and
running do not, because a program cache hit
skips the frontend that finds them. Run a check to see a program’s warnings.
The command line prints each warning on stderr after the check passes, sorted by file and position:
main.plenty:3:14: warning: `copy(original)?` is redundant: `str` copies on assignment and when passed or returned; write `original` [redundant-copy]
The bracketed name is the lint’s stable identifier. The language server publishes the same text with warning severity and the name as the diagnostic code. A check that fails reports only its errors.
There is no option to disable a warning or silence one site. Every warning has a replacement that keeps the program’s behavior; apply it.
std code and the bodies of generic functions and generic class or enum
methods report no warnings: a generic body is checked per instance, and code
that is redundant for one type argument can be needed for another.
Lints
| Name | Reports |
|---|---|
redundant-copy | copy(x)? when the type of x already copies |
redundant-copy
A value whose type copies
is copied by plain use: assigning, passing, or returning it. copy(x)? then
produces the same value as x, with a Result and an AllocError path that
cannot add anything. The lint uses the checker’s own copy rule for the
instantiated type, so str, numbers, bool, bytes, copyable tuples and
sums, and @copy classes such as Point[i64] are reported, and a list or
ordinary class is not.
It reports copy(x)? only when writing x instead keeps the type, evaluates
the same expression once, and moves nothing. x must be:
- a local that is not a reference or view;
- a field path from such a local, such as
record.name; a field read through a reference, such asself.namein aselfmethod, is left alone; - a list, dictionary, or tuple element, such as
words[i]; - a string, number, boolean, or bytes literal.
For a name or field path the message spells the replacement. These are left alone:
copy(x)used as aResult, for example matched or stored, rather than propagated with?;- a reference argument such as
copy(r)?forr: &str, orcopy(&x)?;*rreads a copyable value through a reference; - a call or other computed argument, such as
copy(f())?; copy(x)?of a type that moves, which makes an independent value.
def main() -> Result[(), AllocError | IoError]:
original = "hello"
copied = original
print(original, copied)?
Ok(())
hello hello
Removing a copy(...)? can leave a function’s AllocError unused. The
function still checks; narrow its declared error type if nothing else needs it.
Single-file scripts
plenty script compiles a standalone application and runs it without creating a
package. It requires the normal native compiler/linker environment and an explicit
main, not top-level execution.
#!/usr/bin/env -S plenty script
def main() -> Result[(), IoError]:
print("Hello, Plenty!")?
Ok(())
Hello, Plenty!
Save it as hello.plenty and run:
plenty script hello.plenty 123
chmod +x hello.plenty
./hello.plenty 123
Arguments and modes
plenty script [OPTIONS] [--] FILE [ARGS...]
plenty script --check hello.plenty
plenty script --compile -o ./hello hello.plenty
Runner options precede FILE. Everything after it is passed unchanged to the
application, including flags, empty strings and a literal --. -- before FILE
allows filenames beginning with -. Thus plenty script --help displays runner
help, while ./hello.plenty --help asks the application for help.
On Unix, args() starts with
the invoked script path, including a symlink launcher’s spelling;
args_bytes() preserves native argument
bytes without UTF-8 conversion. Retained binaries have normal executable argument
zero. Streams, environment and working directory are inherited; imports do not
change the working directory. Child exit codes and Unix signal status follow
ordinary compile-and-run.
--check and --compile -o OUT resolve the same dependencies and preserve source
diagnostics, but reject program arguments and never run the application.
Scripts optimize by default; --fast-compile skips optimization for shorter
edit-run cycles. --release remains a compatibility alias for the default.
--unchecked MODE selects the unchecked arithmetic mode. --offline, --locked,
--verbose, --no-strip and --no-cache retain their ordinary meanings.
--cache-stats reports artifact reuse. Build status goes to stderr; a fresh cached
run is quiet unless reporting was requested.
Optional embedded manifest
A script may have one ---plenty line, TOML metadata, and one closing --- line
before its declarations. The optional shebang must be the first line; blank lines
may precede the header. Delimiters are unindented lines with no extra whitespace.
Duplicate, misplaced and unterminated headers are errors. CRLF is supported.
Imported modules cannot contain script manifests. plenty FILE and plentyc
remain compiler interfaces: an embedded header directs their users here rather
than fetching dependencies.
For example, given an existing library package at ../util:
#!/usr/bin/env -S plenty script
---plenty
[package]
name = "helper"
[dependencies]
util = { path = "../util", version = "^0.1.0" }
---
from util import answer
def main() -> Result[(), IoError]:
print(answer())?
Ok(())
The package manifest dependency schema applies, including Git
tag/rev, subdir, and version requirements. No registry or version-only lookup
is introduced. manifest-version defaults to 1; [package] is optional, with
version 0.0.0 and a name derived from the lowercase filename stem. Invalid name
characters become _; names needing an initial identifier character or colliding
with reserved std receive a script_ prefix. Explicit names use package
validation. Description, authors, license and minimum plenty-version are
supported. Unknown keys, workspaces and explicit library/binary targets are rejected.
There is exactly one target: the original script. Neighboring targets and enclosing
workspaces are never discovered.
The physical script directory is both the module root and the base of relative dependencies. A symlink launcher uses resources beside its target. Sibling modules are supported, with normal reserved-name, alias-conflict and cycle checks. A script with sibling modules or path dependencies is not self-contained when copied. Diagnostics retain original filenames, lines and body byte positions. Editors recognize the header; dependency completion uses already cached resolution without fetching. A script run with only an explicit durable lock may need an ordinary cached-lock run before editor dependency resolution is available.
Storage and locks
Generated dependency state lives under scripts/state/HASH/ in PLENTY_CACHE_DIR,
or $XDG_CACHE_HOME/plenty, or ~/.cache/plenty. HASH covers the canonical script
path, not its basename. Unix directories are created private to the user. Nothing
is generated beside the source by default, so read-only script directories work.
Per-script file locks serialize resolution; locks are released before running the
application. Successful object and executable entries are published atomically;
runs execute private copies, never a shared writable binary. Failed builds do not
run an earlier executable.
Dependencies use a cached plenty.lock, retained across body edits and reconciled
when dependencies change. --locked rejects missing or stale locks; --offline
uses cached dependency sources only. A dependency-free first run needs no lock and
works with both flags. Artifact caching can be disabled without disabling the
persistent dependency state needed by these flags.
For a durable, shareable lock:
plenty script --lockfile ./hello.lock hello.plenty
plenty script --locked --offline --lockfile ./hello.lock hello.plenty
The option path is relative to the caller and must be outside the Plenty cache so cleanup cannot remove it; local source paths inside the lock are relative to the physical script directory. Cached locks are not portable reproducibility guarantees: clearing the cache or moving a script can resolve a moved tag differently. Full revisions do not pin mutable tags in transitive dependencies; path dependencies remain live trees.
plenty cache stats includes script dependency state, and plenty cache clean
removes it as well as compiled artifacts. Cleanup takes each namespace’s lock;
small coordination lock files remain so concurrent launches keep using the same
lock. Automatic artifact-size trimming does not evict dependency state. Durable
locks outside managed cache state are not removed. Executable reuse has the
existing Linux toolchain eligibility limits;
other toolchains link normally. Unix shebang/argument behavior is tested on Linux;
non-Unix script argument zero is the generated executable name.
This is not a sandbox or interpreter, and does not provide a script test command. For a multi-target application, portable package locks, or package test discovery, use packages. See the design rationale for alternatives.
Packages and project commands
A package is a directory containing plenty.toml. Project commands search the
current directory and its parents for the nearest manifest, then for an
enclosing workspace. The compiler receives a fixed package
graph; imports never fetch or resolve packages.
Commands
| Command | Behavior |
|---|---|
plenty new PATH [--lib] [--name NAME] | Create PATH with a manifest, src/main.plenty (or src/lib.plenty), and .gitignore |
plenty init [--lib] [--name NAME] | Create a package in the current directory; refuses to replace a manifest or source |
plenty check [--locked] [--offline] [--check-all-methods] | Check the library (without requiring main) and every executable, and report unchecked sources; --check-all-methods also checks unused methods of the package’s own generic instances, not those of std or dependency types; prints each package’s warnings once |
plenty build [--fast-compile] [--no-strip] [--locked] [--offline] | Check the library and compile every executable to target/PROFILE/NAME |
plenty run [--fast-compile] [--no-strip] [--locked] [--offline] [--bin NAME] [-- ARGS...] | Build one executable and run it with ARGS |
plenty test [--fast-compile] [--locked] [--offline] [--no-capture] [--seed N] [--list] [--jobs N] [--timeout SECONDS] [FILTER...] | Compile and run the package’s tests |
plenty add [ALIAS] SOURCE [--version REQ] [--offline] | Add a dependency to the manifest and lock it |
plenty fetch [--locked] [--offline] | Resolve, download, and lock every dependency without compiling |
plenty update [PACKAGE...] [--offline] | Move Git dependencies to the commits their tags and revisions name now |
plenty graph [--locked] [--offline] | Print the package graph file for plentyc --package-graph |
plenty clean | Remove target/ |
plenty lsp | Serve the language server on stdin and stdout |
--locked fails instead of creating or changing plenty.lock. --offline
never runs git fetch; it uses only the package cache in .pkg/. Executables
are stripped of their symbol table
and debug sections in both profiles; --no-strip keeps them.
build --test-hooks and run --test-hooks enable
failure injection in project
executables. Switching this flag rebuilds the executable. These instrumented
builds retain symbols, like plenty test, and are intended for integration tests.
The package name defaults to the directory name and must be a lowercase
identifier other than std. Status lines go to standard error.
plenty run inherits standard streams, environment, and working directory, and
exits with the program’s status; Unix signal termination becomes 128 plus the
signal number. With several executables, --bin NAME is required.
Any other first argument is a compiler command: plenty FILE
still compiles and runs one file. A file named like a project command needs a
path, such as plenty ./run. The plentyc binary provides only the compiler
commands.
Check coverage
A target’s check covers the files its import graph reaches: its entry, the
package modules it imports, and the dependency libraries those import.
plenty check checks every target, then reports each .plenty file below the
package’s src/ that no target reached. Such a file was not parsed or
type-checked, so it may contain errors:
Checking tools v1.0.0
warning: src/util/orphan.plenty is not imported by any target of `tools`; it was not checked
Checked tools v1.0.0: library and 2 executables; 4 of 5 files in src/
The warning does not change the exit status. The summary names the targets
checked and counts the files below src/ that they reached. Test files are
not targets: plenty test checks tests/ when it compiles them, and
plenty check ignores that directory. Every .plenty file below src/ is
package source, which build reuse also fingerprints; keep
fixtures, generated samples and other intentionally unbuilt .plenty files in
another directory, such as tests/fixtures/ or fixtures/. Files without the
.plenty extension are never reported.
Tests
Each file directly in tests/ is a test file; files in subdirectories are
helper modules. A test file defines either main or test_ functions, not
both, and defining neither is an error:
- A file with
mainis one test, named by the file stem, that passes when the program exits with status zero. - Each module-level function whose name starts with
test_is a test namedFILE::FUNCTION. It takes no parameters and returns()orResult[(), E]written out, not through an alias; any other signature is an error. It passes when it returns normally or returnsOk. A returnedErris written to standard error aserror: NAME returned ERROR, exceptFailure.Unspecifiedandstd.test.TestError.Assertion: the first carries nothing to show, and the second follows an explanationstd.testalready wrote.
Plenty has no attributes or reflection, so plenty test finds test functions
by parsing each file, then compiles the file with a generated main that calls
the function named by its one argument. Every test runs in its own process, so
a trap, abort, or leaked state fails or affects only that test. The generated
main exits 0 when the test passes and 3 when it returns Err; a trap exits 1.
A test file imports the package’s library under the package name, the
package’s dependencies under their aliases, and its own helpers under tests/.
Because those names are dependency aliases, a test file or helper cannot be
named after the package or one of its dependencies.
plenty test compiles every selected test file to
target/PROFILE/tests/PACKAGE/STEM before running any. These executables also
link the hooks behind std.test failure injection,
which wrap the C library’s allocation and file calls; plenty build and
plenty run never link them. Each test runs with the
package directory as its working directory, empty standard input, and TMPDIR
set to a new empty directory that the runner removes, with everything in it,
when the test ends. That removal also covers scratch directories of a test
that traps or times out, which skip destructors. The per-test directories live
in one plenty-test-run- directory under the runner’s own temporary directory.
Tests run in parallel, at most one per available CPU unless --jobs N says
otherwise. Results are printed in discovery order whatever order the tests
finish in, so repeated runs print the same lines. Tests that share files in the
package directory must use --jobs 1 or their own names.
Each test’s standard output and error are captured together and printed under
failures: only if it fails; --no-capture passes them through instead, and
parallel tests’ output then interleaves. The command prints one line per test
and a summary, and exits nonzero if any test fails:
running 2 tests
test area::test_square ... ok
test area::test_negative ... FAILED (returned Err)
failures:
---- area::test_negative output ----
assertion failed: left == right
left: -9
right: 9
test result: FAILED. 1 passed; 1 failed
| Option | Effect |
|---|---|
FILTER... | Run only the tests whose names contain one of the filters; the summary counts the rest as filtered out |
--list | Print the selected test names, one per line, without compiling |
--no-capture | Let test output through as it is written |
--seed N | Set PLENTY_TEST_SEED=N for every test; std.test seeds its generator from it |
--jobs N | Run at most N tests at once; N is at least 1 |
--timeout SECONDS | Kill a test still running after SECONDS (at least 1) and fail it as timed out after SECONDSs; processes it started are not killed |
--fast-compile, --locked, --offline | As for plenty build |
A failing test’s line gives the reason: returned Err, exit status N (a
trap is status 1), terminated by signal N, or timed out after Ns. Without
--timeout a test can run forever.
Profiles
| Profile | Selected by | Output | Code generation |
|---|---|---|---|
| optimized | default | target/optimized/ | Cranelift opt_level=speed and optimized call-site passes |
| fast-compile | --fast-compile | target/fast-compile/ | Cranelift opt_level=none; skips optimized call-site passes |
Profiles change code generation only. Overflow, bounds, ownership, and every
other language check behave identically; neither profile adds debug information.
Use --fast-compile for shorter edit-run cycles in large projects. Measure runtime
performance with the optimized default. Compiled functions, program objects and
executables remain cached separately by profile.
plentyc FILE and plentyc --compile FILE -o OUT also optimize by default;
--fast-compile selects the same unoptimized code generation for single files.
--release is accepted as a compatibility alias for the optimized default.
These are Plenty profiles, independent of Cargo’s profiles for building the compiler.
Build reuse
build, run, and test skip a target whose output exists and whose inputs
are unchanged since its last successful build; the package’s status line then
reads Fresh instead of Compiling. build reuses a library check the same
way; check always checks again. The inputs are the content of every file below each graph
package’s src/ (not only the imported ones), the package graph, the entry
file, the target kind, the profile and linker options, and the compiler
executable (path, size, and modification time; the compiler embeds std and
the runtime). Fingerprints live in target/PROFILE/.fingerprint/. A build
forgets the old fingerprint before it starts, so a failed or interrupted build
is never reused. Reuse is per target: a stale target is recompiled whole.
Manifest
Language editions are not supported; edition is an unknown manifest key.
The editions proposal explores
per-package source compatibility independently of plenty-version and
manifest-version.
manifest-version = 1
[package]
name = "app"
version = "0.1.0"
description = "optional"
authors = ["optional"]
license = "optional"
plenty-version = "0.1.0"
[lib]
path = "src/lib.plenty"
[[bin]]
name = "app"
path = "src/main.plenty"
[dependencies]
util = { path = "../util", version = "^0.1.0" }
[dependencies.audio]
git = "https://example.com/audio.git"
tag = "v1.2.0"
subdir = "packages/audio"
manifest-version and [package] with name and version are required.
version and plenty-version are MAJOR.MINOR.PATCH. plenty-version is the
minimum toolchain version; an older toolchain rejects the package. The numbers
above are sample package versions and compatibility requirements, not the
current compiler version. plenty new and plenty init start the new package at
0.1.0 and set plenty-version from the compiler that creates it. Unknown keys
and tables are errors, so a misspelled setting is never ignored. The manifest is
a subset of TOML 1.0: basic and literal strings, decimal integers, booleans,
arrays, tables, inline tables, and arrays of tables. Dotted keys, dates, floats,
and multi-line strings are errors.
Targets default to src/lib.plenty (the library) and src/main.plenty (an
executable named after the package) when those files exist. [lib] and
[[bin]] replace the defaults; their paths are .plenty files below src/. A
package needs at least one target. A library entry may define main; it is an
ordinary function there.
Dependencies
Each [dependencies] key is an alias: the only name under which that package’s
code imports the dependency’s library. import util binds the library module,
and import util.shapes reads src/shapes.plenty in the dependency. Aliases are
lowercase identifiers. std cannot be an alias, and an alias that matches a
local top-level module or directory is an error.
path is relative to the declaring manifest and names a live directory
containing plenty.toml. A git dependency is described below.
version is an optional requirement on the dependency’s version, checked
after resolution; it never selects a version. X.Y.Z and ^X.Y.Z accept
versions at least X.Y.Z with the same leftmost nonzero component (^0.2.1
accepts 0.2.5 but not 0.3.0), =X.Y.Z accepts only that version, and
>=X.Y.Z accepts that version or later. A mismatch names both packages:
error: package `app` requires `util` ^2.0.0, but /work/util is version 1.4.0
Resolution reads every manifest before compiling. A graph contains one package per name: two dependency paths reaching the same directory share one package and its types, whatever their aliases; two different directories with the same package name are an error that names both paths. Package cycles are errors. Dependencies are not transitive: a package imports only what it declares.
Each package’s sources live in its own src/ directory and cannot import files
of another package except through a declared alias. The same module path in two
packages names two modules, so Point in two packages is two distinct types.
Diagnostics name a dependency’s declarations as PACKAGE::MODULE.Name, such
as util::lib.Point; the root package’s names are unqualified.
Git dependencies
[dependencies]
tz = { git = "https://github.com/plentylang/tz.git", tag = "v0.3.0" }
codec = { git = "https://example.com/media.git", rev = "4f2a9c1", subdir = "codec" }
A git dependency needs exactly one of tag (a tag name) or rev (7 to 40
lowercase hexadecimal digits of a commit). subdir is the package’s directory
inside the repository, relative and without ... Plenty runs the system git
program; any URL it accepts works, including file:// and SSH. Credentials
come from Git’s own configuration; Plenty never prompts.
The first resolution fetches every branch and tag of the repository into a
bare repository under .pkg/git/db/, records the commit that the tag or
revision names in plenty.lock, and extracts that commit’s tree into
.pkg/git/checkouts/REPO/COMMIT/. Later commands build the locked commit,
even if the tag has moved upstream; they fetch only when the cache lacks the
commit, as after deleting .pkg/. Each repository is fetched at most once per
command. Snapshots are extracted beside their final path and renamed into
place, so an interrupted fetch never leaves a partial snapshot. .pkg/ is
generated: the .gitignore that plenty new writes excludes it.
plenty update resolves every tag and revision again and rewrites the
lockfile; plenty update NAME... resolves only the named packages. It prints
each changed commit.
A path dependency inside a Git package names another package of the same
commit, and cannot leave the repository. A graph contains one commit per
package: if two dependents reach the same repository directory at different
commits, resolution fails and names both chains:
error: conflicting requirements for package `geo` from https://example.com/mono.git:
app requires tag=v2 (commit c25fe24762e9)
app -> shapes requires commit 421b7a9fd85a through a path in the same repository
a graph may contain one commit per package
| Situation | Behavior |
|---|---|
--offline and the repository or commit is not cached | Error naming the URL; nothing is fetched |
--locked and the lockfile has no commit for a requirement | Error before any fetch |
| The locked commit is gone upstream and not cached | Error suggesting plenty update |
A locked commit is not 40 lowercase hexadecimal digits | Error: the lockfile is malformed; delete it to resolve again |
| The tag or revision does not exist | Error naming the URL and tag or revision |
git is not installed | Error saying Git dependencies need git on PATH |
Adding dependencies
plenty add appends one entry to the [dependencies] table of the package
containing the current directory, keeping the rest of the file unchanged, then
resolves and locks the graph. If resolution fails, the manifest is restored.
plenty add --path ../util
plenty add tz --git https://github.com/plentylang/tz.git --tag v0.3.0 --version 0.3.0
plenty add codec --git https://example.com/media.git --rev 4f2a9c1 --subdir codec
--path is relative to the current directory and is written relative to the
manifest. The alias defaults to the dependency’s package name; adding an alias
that already exists is an error.
Workspaces
A workspace is a directory whose plenty.toml lists member packages:
manifest-version = 1
[workspace]
members = ["app", "util"]
Members are directories below the workspace, without ... The same manifest
may also contain [package], which makes the workspace directory a member too.
Members share one plenty.lock, one .pkg/, and one target/ in the
workspace directory, and depend on each other through ordinary path
dependencies.
A command run in the workspace directory acts on every member: check and
build process each in manifest order, run chooses among all their
executables (whose names must be unique), and test prefixes each test name
with PACKAGE/. A command run inside a member acts on that member only, with
the shared lockfile and outputs. plenty add and plenty graph need a member.
Lockfile
Every project command writes plenty.lock beside the root manifest (or in the
workspace directory) when it is missing or out of date. It lists each
package’s name, version, source, and dependencies as alias/package pairs: the
root package first, then the others sorted by name. A source is root,
path+RELATIVE from the lockfile’s directory, or git+URL with the subdir,
the full commit, and the selectors (such as tag=v1.2.0) that resolved to
it. It contains no absolute paths, so a checkout can move. Writes replace the
file atomically. --locked fails instead of creating or changing it. A
lockfile that cannot be read is an error; deleting it resolves every
dependency again.
Package graph files
plenty graph prints the resolved graph of the current package as TOML, which
plentyc --package-graph FILE reads in place of --module-root:
graph-version = 1
[[package]]
identity = "app"
root = "/work/app/src"
dependencies = { util = "util" }
[[package]]
identity = "util"
root = "/work/util/src"
library = "/work/util/src/lib.plenty"
dependencies = {}
The first package contains the entry file. identity is unique and qualifies
the package’s declarations in diagnostics; dependencies maps each alias to an
identity whose package has a library. Relative paths are relative to the
file. --package-graph works with FILE, --check, --check-module, and
--compile. It lets another build system compile a package without
plenty.toml, and makes the compiler’s input visible when debugging
resolution.
Application exit and error reports
The native wrapper calls the application’s main once. Unit and Ok(()) produce
status zero, i32 and Ok(i32) supply the status, and Err produces one. The
operating system may truncate statuses; small nonnegative values are portable.
Entrypoint signatures are defined in the
language contract.
A returned Err produces one report, identically for compile-and-run and a
compiled executable. The wrapper flushes standard output, writes one line to
standard error, destroys the error payload once, and returns one. Intermediate
? propagation prints nothing.
The line begins error: main returned . When the concrete error type satisfies
std.fmt.Formattable, its borrowed format method
supplies the message. An anonymous sum takes this path only when every member
satisfies the protocol. Otherwise the payload is rendered as
str.repr would render it. Errors needing an explanation add one after a colon:
IoError.System(code) uses the OS description, payload-free IoError variants
use fixed descriptions, and Failure states that it retains no original details.
Choose a concrete error type when the report should identify the cause.
Reporting is best effort. On the structural path, if allocation fails or the
type cannot be rendered (for example it contains a generator or recursive data),
the wrapper writes the
allocation-free fallback error: main returned Err. Standard-error write failure,
including a closed descriptor or a pipe without a reader, is ignored. Every path
still destroys the payload and returns one.
Human-readable reporting allows 16,384 bytes for the prefix, message and final
newline, with a maximum rendering depth of 64. A formatting or budget failure
appends [report unavailable] and a newline, outside that byte budget. The
already-written prefix remains; the formatter is never called again and its
error is not reported recursively. A terminal stderr failure stops output
without attempting the suffix. These writes suppress SIGPIPE and need no heap
allocation, though a user formatter may allocate. Reports stream in pieces and
can interleave with other writers; they are not atomic records.
User-supplied text is emitted unchanged, so it may itself contain newlines.
Context[E] uses human-readable reporting when its
cause supports it, printing the operation/path followed by the cause once.
Other contexts retain structural reporting. Reporting does not change which
types may be returned in Err or the behavior of intermediate propagation.
Compilation/checking errors execute no program code. Runtime errors can leave effects already performed. Traps and stack overflow do not unwind.
Native linking and targets
Native emission implements x86_64-unknown-linux-gnu and
aarch64-unknown-linux-gnu. ARM deployment qualification is described in
Linux AArch64 development. --target TRIPLE
requires an exact match between the requested target, packaged runtime, and
Cranelift ISA before emission. Unsupported explicit targets fail before source
loading. This flag does not enable cross-compilation. Checking is independent of
native emission unless a target is explicitly requested.
--linker PATH selects a cc-compatible driver; the default is cc on PATH.
Repeat --link-arg ARG to pass individual arguments after program/runtime inputs
and before the runtime’s native dependencies. Static-library builds use
--archiver PATH, defaulting to ar; it must support deterministic rcsD mode.
With no --linker or --link-arg override, Plenty prefers mold, then lld,
then the driver’s default linker. It looks for executable ld.mold or ld.lld
on PATH and checks that cc -fuse-ld=NAME -Wl,--version succeeds before
selecting one. An unavailable or rejected candidate falls through to the next
choice. A failure linking the actual program is reported without retrying with
a different linker.
Discovery runs once per compiler process; set PATH and the driver’s
environment before starting it. The probe’s result is also recorded in the
compile cache directory, under
DIR/linker-probe/, so later processes skip the probe. A record’s key covers
PATH, COMPILER_PATH, and the resolved path, size and modification time of
cc and of each ld.mold and ld.lld found on PATH: installing, removing,
upgrading or touching any of them causes a fresh probe. The directory keeps the
64 newest records. With the cache off (PLENTY_CACHE=off, --no-cache), each
process probes again. A linker that cc finds elsewhere than PATH, such as
its own installation directory, is not part of the key; after changing one,
remove DIR/linker-probe/.
PLENTY_LINKER_THREADS=N limits the automatically selected linker to N
threads: it adds -Wl,--thread-count=N for mold and -Wl,--threads=N for
lld. Unset or empty keeps the linker’s default, and a value that is not a
positive integer fails the link. It does not apply to a linker selected with
--link-arg or found through COMPILER_PATH. mold uses every core by default,
which is fastest for one link; when many links run at once, as in a parallel
test run, each link’s threads compete with the others, and one thread per link
finished 400 links at 8 at a time in about 60% of the time.
Pass --verbose to report the effective driver command. --linker cc opts out
of automatic selection and uses the driver’s configured default; to pin a
particular linker, use e.g. --link-arg -fuse-ld=bfd or
--link-arg -fuse-ld=mold. Any explicit --link-arg, even -lm, disables
automatic selection. These options apply to both compiler binaries, including
shared libraries and fuzz executables. Object emission does not probe a linker.
In the Rust API, CompileOptions::auto_linker = false opts out; non-default
drivers and nonempty link_args also disable discovery. verbose reports the
command. The caller must explicitly disable auto_linker when selecting cc
itself, since assigning that default value cannot record an explicit override.
Symbols and debug information
Executables and shared libraries are linked with -Wl,--strip-all, which
drops the symbol table (.symtab, .strtab) and every .debug_* section.
Generated code carries no debug information; these sections come from the Rust
runtime archive and its precompiled standard library, and were about three
quarters of a hello-world executable:
| Program | Unstripped | Stripped |
|---|---|---|
empty main | 2,727,008 bytes | 302,280 bytes |
print("hello") | 3,289,728 bytes | 789,776 bytes |
Measured on x86-64 with mold. Stripping leaves the loaded segments unchanged (size
reports the same text, data and bss), so it affects file size, not
memory use or speed.
Nothing reads these sections at run time. Trap reports find the failed check
through the compiler’s site and trap-instruction tables, which the linker resolves as
relocations, not through symbols; the runtime prints no backtraces. A shared
library keeps its dynamic symbol table (.dynsym), so its exports and
--verify-interface are unaffected, and its embedded .plenty.interface.*
metadata is not a debug section.
--no-strip keeps both, for --compile and --shared-library; plenty build
and plenty run accept it too. For single-file compilation with plenty or
plentyc, --fast-compile and --no-strip may appear in either order before or
after --compile FILE, including between the mode and filename.
Use --no-strip to inspect an executable with nm,
perf, or a debugger, or to split the debug information into a separate
file:
plenty --compile --no-strip app.plenty -o app
objcopy --only-keep-debug app app.debug
objcopy --strip-all --add-gnu-debuglink=app.debug app
--fuzz targets, plenty test executables, and explicit --test-hooks builds
are never stripped: they run
and are debugged where they are built. Static libraries are archives of
unlinked objects, so the final link decides; --emit-object likewise leaves
stripping to the external link (see below). In the Rust API,
CompileOptions::strip is true by default.
Stripping is part of the linker’s deterministic output and does not change the reproducibility contract below; package build fingerprints include it.
Size budgets
tests/test_executable_size.rs fails when an empty program or a hello program
grows past its stripped-file or .text budget. The measurements and current
limits live beside BUDGETS in that file. x86-64 and ARM64 have separate limits,
as do ordinary runtimes and the runtime-checks feature’s allocation accounting.
Compare both sizes: linker layout and segment alignment can increase the file
without increasing its code. ARM files with a separate read-only LOAD segment
after executable code and 64 KiB segment alignment use a separately measured
file-size budget. Compact layouts keep their smaller file limits, and both
layouts keep the same .text limits.
A failure preserves the measured executable, source, and layout.txt in the
directory printed by the test. The report records sizes, section offsets,
segment alignment, ELF producer comments, runtime-checks mode, Git revision,
and available toolchain versions. Missing version commands are recorded rather
than preventing artifact retention. Set PLENTY_SIZE_ARTIFACT_DIR to a dedicated
directory to retain successful measurements too; each sample gets its own
subdirectory. CI uploads these samples as executable-size-ARCH-shard-N
artifacts with seven-day retention, so comparisons can use the actual runner
ELFs. Failed Linux release builds also upload their samples as
release-size-evidence-TARGET artifacts with fourteen-day retention. Local
failures and explicitly requested samples remain until removed.
Tool versions queried during the test describe the current environment;
ELF producer comments can identify the actual linker (mold and lld record
their versions there).
The same file checks, for each std module with runtime native code, that
hello links none of that code, that importing any std module links none of
it, and that a program calling the module does link it. Its NATIVE_MODULES
table names each module, a program, and the symbols that prove the code is
linked: the module’s entry symbols
and, for a vendored library, part of its symbol names. A new native wrapper
adds a row. Raising a budget is a reviewed change that records the new
measurement beside BUDGETS.
Reproducible artifacts
Generated code uses the original x86-64/SSE2 baseline or generic ARMv8-A,
regardless of the compiler machine’s CPU extensions. ARM does not require
optional LSE, SVE or crypto instructions. --print-target reports the native
triple and CPU baseline. Reproducibility applies within one target and fixed
toolchain; artifacts from different architectures have different bytes.
With the same compiler binary, source tree, module contents, and options,
application objects and extracted runtime artifacts are independent of source
directory, output name, time, user, and code-generation worker count. Static
archives use deterministic member timestamps, owner IDs, and modes.
Executables and shared libraries additionally depend on the linker driver,
its options, startup objects, and native libraries. Automatic selection makes
the available linkers and driver environment inputs too: pin the linker for
reproducible builds across machines. Package build fingerprints include the
automatically selected linker name. Different versions of the same tool still
require a clean rebuild after changing the native toolchain. Determinism is
checked with the locally selected toolchain; other linker implementations are
separate inputs. A program that reaches a native engine also
depends on the engine’s directory, which ordinary output records in RUNPATH.
Distribution output instead uses only $ORIGIN and copies
the selected engines and their notices.
Rebuilding the compiler also depends on rustc and its precompiled standard
library. Runtime compilation remaps project, output, and sysroot source paths;
it cannot rewrite paths already compiled into the Rust standard library.
Changing the compiler or any of these toolchain inputs can change output bytes.
Paths and arguments are passed directly, without a shell, expansion, or word
splitting. There is no CC environment override. The linker driver must accept
object/archive inputs, Unix native flags, and -o OUT; GCC and Clang do. Drivers
with different conventions need a wrapper. Choosing a driver does not expand
target support. Failures report its identity, status, and diagnostics.
External build systems
--emit-object requires no linker and exports plenty_main. Link exactly one
application object, followed by the runtime archive extracted from the same
compiler build and its native arguments, one per line in link-args.txt. The
first argument is -Wl,--gc-sections. The object puts each function and data
object in its own section, so the linker drops every function main cannot
reach, including the unused functions of imported std modules, and then the
runtime code only they call, such as the std.crypto
hashes, the std.compress codecs and the
std.tls stack. The
runtime supplies C main. link-args.txt lists only what linking requires,
so it does not strip; add -Wl,--strip-all (or -s) to match plenty --compile. Neither object emission nor runtime extraction needs
Cargo or rustc on PATH.
Ordinary Plenty functions, including pub functions, use private symbols and
calling conventions. Application object output does not expose a C library API
or provide separate Plenty module compilation; use a library build for exports.
The Rust API exposes CompileOptions, executable compilation with options,
compile_source_to_object, compile_file_to_object, and emit_runtime. Runtime
extraction returns archive and argument paths/values for build-system integration.
Native engines
An engine is a large native library behind a std module that ships outside
the runtime archive, as a separate shared library. Only programs whose
retained code reaches the module link it. std.dataframe
(Polars) and std.linalg (faer, issue #490)
are engines today; images (issue #476) is another candidate.
When a module is an engine
Code in the runtime archive costs nothing to programs that do not reach it:
the linker drops it with --gc-sections (see linking),
provided only the module’s own entry symbols reach it (issue #501). It still
grows the compiler, which embeds the archive, and every compiler build
compiles it. So a native module stays in the runtime archive unless one of
these holds:
- Size. A program reaching one ordinary operation of the module grows by
more than 2 MB, measured as the stripped executable minus the stripped
print("hello")executable. - Dependencies. Its crates need a native toolchain beyond rustc and
cc, a system library outside the release baseline, or a build that adds more than a minute to a clean release build of the compiler. - Release cadence. It must be rebuilt or upgraded independently of the compiler, for example to follow a fast-moving third-party release.
Measured on 2026-10-26 with stripped release builds:
| Code | Size |
|---|---|
print("hello") executable | 789,584 bytes |
plus a std.tls client configuration | 2,075,096 bytes (+1.3 MB) |
| runtime archive embedded in the compiler | 15.1 MiB |
Polars dataframe engine, libplenty_dataframe_v1.so | 89,553,952 bytes |
std.tls, the largest runtime module, stays in the archive; the dataframe
engine qualifies on size and build time. A new wrapper’s issue or PR records the
same measurement before choosing.
Shared libraries, not static archives
Engines are shared libraries. A separately supplied static archive, passed to the linker only when reached, was considered and rejected:
- Linking. An engine written in Rust carries its own copy of Rust’s standard library, allocator shims and panic runtime. The runtime archive already defines these and installs Plenty’s global allocator, so two Rust static archives in one link either collide or must be built as one crate graph, which is the runtime archive again. A shared engine keeps its copy private and exports only its facade symbols.
- Executable size. A static engine adds the code each program reaches.
Large engines dispatch internally (query planners, codec tables), so a
program reaching one operation retains most of the engine, in every
executable. A shared engine adds only a
DT_NEEDEDentry and the imported symbols, and one copy serves every program on the host. - Deployment. The cost of a shared engine: the application ships the library beside the executable (distribution output automates this).
- Reproducibility. Under the reproducibility contract
an executable that reaches an engine also depends on the engine’s directory,
which ordinary output records in
RUNPATH(distribution output uses only$ORIGIN), and on the facade’s symbols, but not on the engine’s bytes. A static engine would make every executable depend on the engine library’s bytes, which would then need the runtime’s deterministic build too.
Revisit static archives only for an engine with no Rust standard library or allocator of its own.
The engine contract
A second engine follows std.dataframe in each of these.
| Part | Rule |
|---|---|
| Facade | stdlib/std/NAME.plentyi, a trusted interface whose extern def declarations name the engine’s C symbols. Plenty code in the facade is ordinary std code. |
| Symbols | Every export starts with a prefix unique to the engine and its ABI generation, such as pldf_v1_. The library exports nothing else. |
| ABI generation | libplenty_NAME_vN.so, also its SONAME. Adding an operation adds symbols and keeps N. Changing or removing a symbol, a layout or an ownership rule starts N+1, with a new prefix and file name, so existing executables keep loading vN. |
| Reachability | The compiler follows the object’s relocations from its roots, as --gc-sections does. An unused import, unused helper or unused facade function needs no engine and adds no DT_NEEDED. Windows executable emission omits unreachable user helpers before COFF import resolution; explicit objects, libraries and dumps retain user definitions (emission policy). |
| Discovery at compile time | CompileOptions field, then PLENTY_NAME_ENGINE, then libplenty_NAME_vN.so beside the compiler executable. Ordinary output rejects : in the engine directory (an ELF RUNPATH separator). Build fingerprints include the selected path, size and modification time. |
| Discovery at run time | RUNPATH of $ORIGIN, then the engine’s directory at link time for ordinary output; distribution output uses only $ORIGIN. The loader’s LD_LIBRARY_PATH and system configuration also apply. |
| Missing engine | Linking fails, naming the module, the path tried, how to build or install the engine, and the environment variable. Type-checking never needs the engine. |
| Mismatched engine | An older engine lacking a symbol fails at link time (undefined reference to PREFIX...) or at load time (symbol lookup error). Release packaging rejects an engine whose exports do not cover the facade’s declarations. A different N is a different file and is never substituted. |
| Errors | Each call returns an error code; the engine catches panics at the C boundary and reports them as an error. Handles are freed by the engine that allocated them. Engine allocations are outside Plenty’s allocation accounting; the module’s reference page states whether allocation failure is recoverable. |
| Build | A separate Cargo workspace, plenty-NAME/, with its own Cargo.lock, built by a script outside the compiler build, for the selected x86-64/SSE2 or ARMv8-A target and the release glibc. |
| Release | The matching compiler archive bundles the target’s engines in bin/ beside NAME-notices/, with engine.json (Plenty version, source commit, target, ABI generation, library checksum and platform minimum), Plenty and dependency licenses, and Rust standard-library notices. --dist validates metadata and the copied-library checksum and includes the notices; ordinary compilation does not read engine.json. |
| Tests | The ordinary suite checks that unreached uses link no engine and that a reached use without one reports the missing engine. A separate CI job builds the engine and runs the real-engine tests, gated by PLENTY_NAME_TESTS=1. |
macOS dataframe engine
The experimental macOS ARM64 archive bundles libplenty_dataframe_v1.dylib
and dataframe-notices/. Discovery uses the same option/environment precedence,
then the dylib beside the compiler. The dylib has versioned identity
@rpath/libplenty_dataframe_v1.dylib; applications use @loader_path to find
their sibling copy. Ordinary compilation copies the dylib beside its output;
distribution output includes
validated metadata and notices. Engine builds target Apple M1 and macOS 11.
Packaging audits the deployment minimum, dependencies, exports and ad-hoc
signature. The linear algebra engine remains Linux-only.
Windows dataframe engine
The experimental Windows x86-64 archive bundles plenty_dataframe_v1.dll and
dataframe-notices/. Discovery uses the same option/environment precedence,
then the DLL beside the compiler. MSVC lib.exe generates an import library
from the DLL’s C exports. Ordinary compilation copies the DLL beside the output;
distribution output includes
its validated notices. Runtime lookup follows Windows loader rules rather than
ELF RUNPATH. The linear algebra engine remains Linux-only.
Adding a second engine
The compiler side is a table: an entry in ENGINES (src/toolchain.rs)
gives the facade module, library, symbol prefix, environment variable and
install hint, and drives reachability, link arguments, the facade’s
.plentyi resolution and build fingerprints. The CompileOptions override
is still a named field per engine.
The shared build and packaging entry points are scripts/build_native_engine.py
and scripts/release/package-engine.py, each taking an engine name. scripts/build_dataframe_engine.py
remains compatible. Packaging checks facade exports and the engine SONAME,
generates component-specific dependency notices, and installs the library with
its NAME-notices/ directory, in the compiler archive’s bin/ for releases. The native-engine CI matrix builds and
tests each engine separately; engine tests never make normal compiler builds
depend on either engine workspace.
Application distributions
Use explicit distribution output when shipping an application:
plenty --compile analysis.plenty --dist dist/analysis
./dist/analysis/analysis
plentyc accepts the same command. The executable is named after the source
file’s stem plus the native executable suffix (analysis.plenty becomes
analysis.exe on Windows and analysis on Unix), not the destination directory.
Dots in the stem are preserved: analysis.v2.plenty becomes analysis.v2.exe
on Windows. Paths may contain spaces. --fast-compile, --no-strip, --module-root and
--package-graph work as for ordinary executable compilation. --dist requires
--compile FILE and rejects -o/--output, extra --link-arg
arguments and other output modes. A custom --linker must still produce the
required lookup policy for the native platform.
The destination must not exist, even as an empty directory, file or dangling
symlink. Choose a new name for each build. Plenty stages the output beside the
destination and publishes the complete directory without replacing an existing
path, including one created concurrently. Compilation, linking, missing notices
or publication failures leave no partial destination; temporary staging is
removed on normal error exits. Parent directories may be created. Abrupt process
termination can leave a hidden .plenty-dist-* staging directory, not a published
bundle.
Contents and engine selection
A program without reachable native-engine calls produces only its executable. Otherwise Plenty copies only the engines selected for that link, using the ordinary engine discovery and reachability rules. Unused imports and unreachable helpers do not add engines. Each used engine adds:
dist/analysis/
analysis
libplenty_NAME_v1.so
NAME-notices/
engine.json
LICENSE-MIT
LICENSE-APACHE
THIRD-PARTY-NOTICES.md
licenses/...
NAME is dataframe or linalg; a program may need both. Executable names that
collide with engine or notices names are rejected. Application assets, application
licenses and non-engine dynamic dependencies are not collected automatically.
Add your own required files before distributing the directory.
The selected engine needs its packaged notices. Supported input layouts are:
- A library beside
NAME-notices/, which holdsengine.json, license files andlicenses/. Linux release archives install both engines this way inbin/, and the compiler selects them without configuration. lib/libplenty_NAME_v1.sowith the same files in the parent oflib/. SetPLENTY_NAME_ENGINEto the library’s absolute path.
Plenty checks manifest format 1, compiler release version, native target, ABI generation and library name; requires source-commit and minimum-glibc provenance; and verifies the copied library’s SHA-256 and SONAME. It links that staged copy, so the bundled bytes are the bytes selected for linking. Missing files, empty license directories, symlinks within notices, malformed metadata and mismatches fail explicitly. No notices are fabricated or silently omitted.
This is consistency checking, not signature verification or proof that a local
engine was built from the compiler’s source revision. Use trusted, matching
release artifacts or build and package from the same checkout. Ordinary
--compile FILE -o OUT does not require or validate engine metadata.
macOS dataframe distributions
On macOS ARM64, the same command produces analysis,
libplenty_dataframe_v1.dylib and dataframe-notices/. The executable links the
dylib’s versioned @rpath identity and searches only @loader_path for the
bundled engine. Ordinary compilation also copies the dylib beside its output,
using the same lookup policy. Neither form retains the original engine path.
The manifest requires minimum_macos instead of minimum_glibc. The compiler
checks the copied library’s SHA-256, ARM64 Mach-O kind, versioned dylib identity,
signature presence and absence of build-machine dependencies/search paths.
Release packaging additionally verifies signatures with codesign, deployment
minimums and facade exports. Applications can require a newer OS than the
engine: compile and test on the oldest system you intend to support.
macOS archives bundle the dataframe engine only and remain unnotarized.
Windows dataframe distributions
On Windows x86-64, the same command produces analysis.exe,
plenty_dataframe_v1.dll and dataframe-notices/. The DLL uses Windows loader
lookup from the application directory. The compiler derives an import library
with MSVC lib.exe; that build artifact is not needed by end users. Metadata
requires minimum_windows instead of minimum_glibc; the copied library must
be x86-64 PE and match its SHA-256. There is no ELF SONAME or RUNPATH on Windows.
The Windows archive bundles only the dataframe engine.
Package a local engine
For local development, use just build for the compiler and both engines, or
just engine linalg / just engine dataframe for one engine. These recipes
install the library and matching NAME-notices/ in target/release/, ready for
--dist. They record the engine’s actual glibc requirements without claiming
the portable release baseline; see the local build workflow.
A bare library installed by build_native_engine.py is enough for ordinary
compilation, but not distribution output. Add local notices to it with:
python3 -B scripts/release/package-engine.py linalg \
--engine target/release/libplenty_linalg_v1.so --out-dir target/release --local-install
For a portable engine, package a baseline-compatible build from the same
checkout with the release packager, which the release build also uses to
install engines in the archive’s bin/ (substitute dataframe and its library
for the Polars engine):
python3 -B scripts/release/package-engine.py linalg \
--engine /path/to/libplenty_linalg_v1.so --out-dir target/engine-bundle
export PLENTY_LINALG_ENGINE="$PWD/target/engine-bundle/libplenty_linalg_v1.so"
plenty --compile analysis.plenty --dist dist/analysis
The packager needs the Rust standard-library license texts and engine dependency
sources to generate notices, and audits facade exports, SONAME, native
dependencies and the glibc 2.28 baseline. A library built on a newer host may fail
that audit. Build in the pinned release environment instead; the existing
bash scripts/release/local-check.sh --skip-smoke builds and packages locally
with Docker, without publishing. See local release checks.
Do not relabel another release’s metadata to bypass checks.
Relocation and platform limits
On Linux, bundled executables use only $ORIGIN (the executable’s own directory) for
engine RUNPATH, never the original engine directory. A sibling lib/ is
not searched. Ordinary builds retain $ORIGIN followed by the build-time engine
directory. PLENTY_NAME_ENGINE affects compilation only; loader configuration,
LD_LIBRARY_PATH and LD_PRELOAD still apply at run time. Keep those paths trusted.
Move the whole bundle, run it from another working directory without loader overrides, and test that removing an engine makes loading fail even on the build machine. End users need no Plenty, Rust or C compiler, but still need compatible host C libraries. This is not static linking or OS-independent packaging: native emission uses the installed compiler’s native target. Building on a newer system can raise the executable’s glibc requirement. Build and test on the oldest system you support; engine metadata alone cannot establish portability.
Library artifacts and publication
Library builds produce a binary, C header, source interface, and native link arguments. Export ownership and ABI rules belong in the C interface contract.
Embedded metadata can be extracted without loading or executing the binary. Explicit interface verification checks the contract and its compatibility symbol; see embedded contracts.
For embedding applications, read_library_interfaces(path) returns all
LibraryInterface entries sorted by library name, including exact source and
target; extract_library_interface(path, name, output) selects one for output.
verify_library_interface(binary, interface) performs explicit verification
without executing a linker or library code.
Publication guarantees
The compiler stages a library’s files in a private directory beside the requested output, on the destination filesystem. It checks destinations before building and before publication. Each must be absent or an ordinary file; symlinks, directories, and paths overwriting loaded source or imported interfaces are rejected.
Compilation, native tool, argument, and staging failures preserve previous files. Publication replaces files by individual atomic renames and keeps backups until the complete set is installed. An installation error rolls back replacements and removes new outputs. If rollback fails, the diagnostic identifies the retained recovery directory containing backups.
Publication is not a crash-atomic transaction across files. Build systems must exclude concurrent writers and coordinate readers. A crash or power loss can leave a mixed set; fingerprints and verification help detect it.
Interface extraction uses the same mechanism for its single output. It rejects symlink destinations and cannot overwrite the input binary. Native tools receive paths as individual arguments, including paths with spaces or commas.
Fuzzing
plenty --fuzz FILE -o OUT builds a libFuzzer
executable from a program. libFuzzer runs the program’s main once per
input, watches which code each input reaches, and keeps the inputs that reach
new code to mutate further. Inputs that trap are saved as findings.
plenty --fuzz main.plenty -o fuzzer
./fuzzer -max_total_time=60 corpus/
./fuzzer crash-1060186d20321afe84b2d22eeca505a577c686c6 # replay one input
Fuzz targets optimize by default. --fast-compile before --fuzz skips
optimization when compilation latency matters more than fuzz throughput. --libfuzzer PATH
names libFuzzer’s archive. Discovery asks clang -print-file-name=libclang_rt.fuzzer_no_main.a first, then the native target’s
suffixed name (libclang_rt.fuzzer_no_main-aarch64.a or
libclang_rt.fuzzer_no_main-x86_64.a). It accepts only a successful query naming
an existing absolute file. LLVM’s compiler-rt supplies the archive: install
compiler-rt on Fedora, or clang and libclang-rt-dev on Debian/Ubuntu.
The build fails with a message when neither name resolves. When clang ships a
separate libclang_rt.fuzzer_interceptors archive beside the selected runtime,
Plenty links its matching generic or target-suffixed companion so libFuzzer can
learn the strings compared by the program. It never substitutes another target’s
companion. --linker and --link-arg apply as for
--compile. The libFuzzer flags are LLVM’s; -help=1 lists them.
Writing a target
The program reads the input through two C functions that the fuzz driver
defines. Declare them in a trusted module beside the program, for example
fuzz_input.plentyi:
pub extern def size() -> u64 = "fuzz_input_size"
pub extern def byte(at: u64) -> u8 = "fuzz_input_byte"
byte returns 0 past the end. A target is an ordinary program that imports
this module, rebuilds the data it needs from the bytes, and calls the code
under test:
import fuzz_input
import std.bytes
def main() -> Result[(), std.bytes.BytesError]:
mut data = std.bytes.empty()?
mut at = 0u64
while at < fuzz_input.size():
data.push(fuzz_input.byte(at))?
at += 1u64
# parse `data` here
Ok(())
A finding is a fatal runtime error such as an index out
of range or unwrap of an Err, stack exhaustion on the thread running
inputs (see the driver), or main ending with a nonzero
status, including Err. libFuzzer reports the first as “fuzz target
exited” and the others as “deadly signal”, saves the input, and exits
with status 77. To check a property, return Err when it fails.
Each input runs the whole program, as the runtime’s main would, in the same
process, so state outside the program, such as files it writes, persists
between inputs. A target that serves a network protocol should feed the input
through an in-memory stream rather than a socket; fuzz/http_server/main.plenty
does this for std.http.server.
Instrumentation
Every block of every function, including the standard library’s, increments
its own byte counter when it runs. The counters and a table of their
functions are laid out as clang’s -fsanitize-coverage=inline-8bit-counters,pc-table
lays them out, in sections __sancov_cntrs and __sancov_pcs; the driver
passes both to libFuzzer. Fuzz targets keep their symbol table and debug
sections, unlike --compile output. cov: in libFuzzer’s output counts the blocks
reached, and ft: the features, which also distinguish how often a block
ran.
Blocks are counted, not edges: a branch whose arms meet without running a
block of their own adds nothing. Integer comparisons are not traced. String
and byte-string equality reach memcmp, and libFuzzer’s interceptors,
linked from libclang_rt.fuzzer_interceptors.a beside the --libfuzzer
archive when it exists, report the strings compared, which libFuzzer then
tries in inputs. A dictionary (-dict=FILE) of the format’s keywords helps
with the rest.
Instrumented functions are cached like any other; their entries are separate
from those of ordinary builds. Fuzz builds keep the counters and table of
every function, so --gc-sections drops nothing they compile. Ordinary
builds contain neither the counters nor the driver.
The driver
plenty-runtime/native/fuzz_driver.c is compiled with the compiler and
linked only into fuzz builds, with -Wl,--wrap=main: the C library starts
the driver, which registers the counters and calls libFuzzer’s
LLVMFuzzerRunDriver, and each input calls the runtime’s main. libFuzzer
is C++, so the link adds -lstdc++.
Parallel runs use -jobs=N -workers=N, independent processes sharing the
corpus directory. With the interceptors linked, libFuzzer’s -fork mode
crashes its own parent process within seconds (clang 22.1.8), reporting a
“deadly signal” with an empty input; it is not a finding.
On Linux, libFuzzer’s RSS limit uses getrusage(RUSAGE_SELF).ru_maxrss,
the process’s peak resident memory. Linux preserves this peak across exec,
so a target launched directly by a large process can report an OOM before
doing meaningful work. The integration tests first exec a small Python
interpreter, then spawn the target from that interpreter’s small address
space. An exec alone does not reset the peak. The tests retain a 2048 MiB
per-target limit and report resource-limit failures (oom-* / timeout-*)
separately from expected crash-* findings with exit status 77. The HTTP
fuzz runner already spawns its target from Python.
See LLVM’s peak RSS implementation
and resource-limit handling.
libFuzzer’s handlers for SIGSEGV and SIGBUS, which save the input, run
on the faulting thread’s stack. So that they still run when a program has
exhausted its stack, the driver gives the thread running inputs a 256 KiB
alternate stack and moves the handlers onto it. The runtime’s own
stack-overflow message is not printed in fuzz builds.
Only the thread running inputs has the alternate stack. A stack overflow
on a thread the program spawns kills the process without saving the input.
Without AddressSanitizer libFuzzer has no leak checking or stack traces. Plenty code needs neither for memory safety; a trap prints its source position before libFuzzer’s report.
The target is the installed compiler’s GNU/Linux architecture (see linking). ARM libFuzzer/tool discovery and sanitizers require separate qualification; a successful ordinary ARM build does not prove their availability.
Compiler and runtime architecture
source → lexer/parser → name and alias resolution → signatures → checking
↓
typed operation IR
↓
Cranelift → object → linker
↑
embedded Rust runtime
- Frontend and checking: types, ownership, access CFGs, and IR.
- Native emission: specialization, batching, and performance evidence.
- Value representations: inline storage, standard sums, and relocation.
- Memory and destruction: owners, reference counts, and allocator provenance.
- Collections: storage, hash order, and current costs.
- Generators and closure environments: frames and captures.
- Tail-call ABI: approved transfers and hidden result storage.
- Inspecting generated code:
--dumptext for each function’s IR and machine code. - Runtime packaging: archives, platform checks, stack overflow, and Rust panics.
- Concurrent mechanisms: pinned storage, locks, and waiters.
- Native interfaces: adapters, metadata, and resident code.
- Support index: links to implemented contracts and major restrictions.
The opt-in compiler phase profiler reports lowering setup/body/finalization, ownership analysis subphases, and native compilation/code collection/context cleanup. Benchmark workloads can be selected independently; profiling is never activated by the feature alone.
These pages describe mechanisms. Source rules live in the language contract, and operation guarantees live beside the built-in APIs.
Frontend and checking
frontend.rs handles name resolution, local inference, mutability, and
lowering to typed operations. Tokens and the lexer (frontend/lexer.rs), the
syntax tree (frontend/syntax.rs) and the core parser (frontend/parser.rs)
read no lowering state; feature modules add their own parsing beside their
lowering.
Type references retain source positions. Aliases resolve through an iterative,
cached dependency walk before signatures are collected; alias-only cycles are
rejected without recursive traversal of long chains.
Nominal definitions reserve identities before resolving fields. Weak stored
edges and table-owning handles represent recursive type graphs without compiler
memory cycles. Concrete type facts and native metadata use finite graph walks.
Cloning a structural type shares its already-owned child storage. When a field
escapes a nominal table, cloning promotes its weak edges and copies only the
structural paths that contain them. This is a single pass through the structural
type, without following nominal table links; stored definitions still keep weak
edges so sharing cannot create a table ownership cycle. Callable and boxed
callable signatures follow the same rule: a signature is shared unless an input
or the output holds a weak edge.
A complete class or enum caches its facts; a collection or box combines its
contents’ facts instead of walking again.
Aliases normalize to Ty values and generate no runtime operations.
Generic instance arguments distinguish Type(Ty) from typed constant values;
values never become storage types. The constant evaluator also serves value
arguments during type resolution, before ordinary module-constant folding.
Quantity constant output types resolve lazily with cycle detection. The
Bounded checker validates each concrete
interval and lowers admitted literals and checked conversions to private
single-field class construction.
The compiler’s own hash tables (src/hash.rs) use FxHash rather than std’s
SipHash. Their keys are names and types from the program being compiled, so
flood resistance buys nothing, and SipHash was about 11% of checking
instructions (issue #302). Iteration order stays unspecified.
Module loading canonicalizes qualified names and checks imported modules together before producing one application object. Import traversal is bounded to depth 128 and 4,096 modules. This is not separate module compilation.
A module is lexed one top-level item at a time. The parser holds the tokens of the item it is parsing and of the next one, so a module’s tokens never all exist beside its syntax tree (issue #54). A lexical error anywhere in a module is still reported before any syntax error in it.
The whole syntax tree is live when lowering starts, so its layout sets much of checking’s peak memory. Blocks, call arguments and parameters are collected on parser scratch stacks and stored at their exact length. Every node holds a 40-byte token that names its module by an interned file number, not a reference-counted path (issue #422).
op.rs defines backend-neutral structured operations and an independent checker.
Its compile-time operand stack is private compiler machinery. CompiledFn carries
a signature, documentation, body, and typed local slots. Unit produces zero stack
values. Loads and stores address slots; exhaustive branches produce conditional values.
The checker collects every signature on one thread, then checks function bodies on all available cores: a body reads only the signature table. It reports the first error in program order whichever body failed first, so diagnostics do not depend on the core count. Operations, types, and signatures are shared between threads without copying.
Name resolution of a module’s functions, and of its classes’ fields and methods, runs on all cores when the module has at least 256 functions or 64 classes: each reads only the module’s scope. Frame layout and generator operation validation after lowering also run on all cores. Both report the first error in source or program order, as a serial pass would. Threads share generator, closure, and nominal layout caches. A cached layout records how deep its frames and environments nest, and is reused only where it cannot hide the nesting limit of 64; otherwise the type is measured again. Each function’s layout error is therefore the same whichever thread measured a type first (issue #126).
Parsing, the rest of name resolution, type resolution, class expansion, generic preparation, signatures, and lowering bodies to operations run on one thread (issue #59). Type resolution can create generic data instances in a table shared by every module, and lowering interns strings, records generic instances, and infers returned fields in tables shared by every body. Generic preparation resolves each concrete function’s parameter and return types once; registering the function’s signature reuses them.
Three ways to run these phases on several threads were measured and not adopted. Each gained little and raised peak memory, because each thread’s allocator keeps its own free memory:
- Parsing one module in pieces: for 5,000 units, parsing took 0.09 s instead of 0.21 s, lowering took 0.06 s longer, and peak memory rose from 417 MB to 538 MB.
- Parsing imported modules in parallel, a level of imports at a time: for a program whose imports load 21 standard modules (12,549 lines), parsing took 9 ms instead of 23 ms and lowering 5 ms longer; the check was no faster, and peak memory rose from 28 MB to 35 MB.
- Lowering bodies speculatively on all cores and keeping them in queue order: checking 5,000 units was about 9% faster and object emission no faster, and peak memory rose 12–14%.
Measuring compilation gives the time each phase takes.
Control flow and access
Return, TailCall, Break, and Continue terminate paths. The checker joins
only continuing branches, validates transfers against signatures or loop entry
stacks, and rejects operations after guaranteed exits. Loops lower to headers,
bodies, and exits; the for increment precedes each continue.
An if statement lowers to one operation that holds its elif arms as a list,
each with its own condition, so no pass recurses once per arm and a chain costs
linear time. Every pass does recurse once per level of block or expression
nesting, as do the clone and drop of the syntax tree. The parser therefore
bounds that depth to 64 and
later passes rely on it instead of checking again.
Ownership checking tracks definite slot availability through branches and loop backedges. Borrow checking flattens typed access facts into a separate source-level CFG with branch successors, early exits, loop edges, and stable binding-place IDs. Each CFG node stores up to two successors inline; only matches with more than two arms allocate a separate successor slice. Edges retain source branch order, including duplicate successors when empty arms join the same continuation. Loan liveness reaches a fixed point before native lowering. Functions without loan facts skip this analysis. It never runs on Cranelift IR and needs no whole-program alias analysis or per-call inspection of function bodies.
Native lowering still consumes structured operations. Cranelift SSA variables
represent slots; stores define variables and sealed joins reconcile paths.
Only slots whose address is taken (&x, &mut x) or whose values have inline
storage live in the function’s stack frame; the frame holds those slots alone.
Augmented assignment of a local (x += 1) takes no address, and starting a
thread adds no slot: task storage has its own frame space and a worker reaches
parent slots only through borrowed arguments. Generators keep every slot in
memory.
Source spans and marker operations carry diagnostic provenance without changing
move or loan rules. A generic instance also records the use that first required
it, so that lowering and ownership errors in its body are
reported at that use.
Std ceiling checking
frontend/ceilings.rs checks fixed upper bounds without introducing value
generics or another integer type. Embedded std source can append
__max(1024) to a parameter or field declared Positive[i64] or
NonNegative[i64]. The decimal non-negative bound is metadata on its
TypeRef; resolved types, layout, equality and callable signatures are
unchanged. The marker is rejected in user declarations and is hidden in public
editor signatures and the generated std signature index.
Direct function, method and generated class-constructor calls check their
supplied arguments before named/default wrappers replace expressions with local
names. Defaults are checked at their declaration. Direct field assignments and
replace(owner.field, value) use the same check, including through borrowed or
boxed owners. The module-constant evaluator handles literals, module constants
(including integer-domain and Quantity constants), and supported constant
operators/casts. Constant folding runs after type resolution so quantities
retain normalized dimensions and storage types during evaluation.
Ordinary local bindings, arbitrary function calls and explicit fallible domain
conversions are not constant evaluation. Indirect callable invocations and
writes through a separately borrowed field do not retain this metadata.
Existing std runtime guards cover values that reach operations through those
paths; this mechanism does not claim a proof-carrying bounded integer type.
The annotations cover the fixed ceilings enforced by the std limit-domain
migration: format/JSON/TOML/XML/template/regex nesting, fs/tempfile cleanup depth
and attempts, sketch construction limits, unicode scalar limits, JWT token size,
HKDF output size, image dimensions, actor heaps and procfs read sizes. Decoded sketch
max_capacity has no ceiling. General bounded integers and lower bounds above
one remain B96.
Embedding APIs
Source-string checking and compilation accept Plenty source and reject imports.
File APIs (check_file, check_module_file, and file compilation) resolve imports
from an explicit optional root. Checking performs no execution or native emission.
Editor queries
ide::analyze, behind plenty lsp, runs the same load, resolution, and
lowering with a per-thread recorder switched on (src/frontend/ide.rs).
Module loading records modules and imports; name resolution records
declarations, module-level names, and locals with their binding positions;
lowering records what needs types: methods resolved through the receiver,
fields, enum variants, and the types of locals. An ordinary compilation
records nothing. Expressions keep only the position of their first token, so
member expressions also carry the position of the member name. In this mode an
error in one function’s resolution or lowering is recorded and checking moves
to the next function. See the language server.
Specialization and native emission
Generic declarations produce cached concrete specializations. Names, types, and operations are resolved before Cranelift receives IR; runtime metadata describes known storage and operations, not dynamic inference. Value parameters enter the same instance keys as typed integer or boolean constants, distinct from type arguments. Equal resolved constants share an instance even when spelled through different module constants; different values consume separate instances. Substitution turns body reads into typed literals and adds no runtime parameter.
Placeholder identities and queued concrete signatures break recursive checking
dependencies. Each generic function or method allows 16,384 instances. A
dependency chain of newly required instances may be 256 steps longer than the
number of generic functions and methods declared plus generator consumers
specialized. A program has finitely many generic declarations, so expansion
without end must grow the instances of one of them without end. A chain in
which no declaration repeats is no longer than that number, so only repeats
can exceed the depth limit: a finite chain through distinct generic functions,
however long, does not reach it (issue #435). Independent calls start at depth one, even when they specialize
the same template with different arguments. A new instance inherits its first
requiring caller’s depth plus one; lifted closures and generated call wrappers
preserve that depth. Repeated calls, ordinary recursion, and alias-equivalent
arguments reuse cached specializations without spending any limit. The depth
check guards expansion, rather than counting the unrelated generics in a
library. It is a conservative implementation bound, not a proof of infinite
recursion. The per-template limit bounds expansion that branches within the
depth limit, such as f[T] requiring both f[list[T]] and f[Box[T]]. A
separate budget of 262,144 instances per compilation bounds the resources of
expansion that drags other generic functions along, and reports a
resource-budget diagnostic (issue #428).
Generic data and generator consumers follow the same scheme (issue #430). Each
generic class or enum allows 16,384 instances, with definitions nested at most
64 deep, and a compilation allows 65,536 generic data instances; Option
instances do not count. Each function with a Generator parameter allows
16,384 concrete frame specializations, which share the dependency depth limit
of generic functions, and a compilation allows 65,536. Type nesting is still
bounded separately.
A generic class or enum instance records its methods without lowering them
(issue #515). Lowering a body queues each instance method it names through a
call, function address, callable adapter, worker, or destructor, which is the
set native reachability follows. A protocol bound queues the methods it
matched, and __del__ is queued with the instance. Methods never queued are
neither checked nor emitted, so they create no instances, strings, or cache
entries. --check-all-methods, accepted only by check modes, queues every
method still unqueued of a type declared outside std and dependencies once
the queue empties, repeating while those bodies
create instances (issue #517).
Before native preparation, emission removes unreachable definitions from the
embedded standard library (issue #400). It keeps every user definition, C export
and generator as a root, then follows direct/tail calls, function
addresses, closure and boxed-callable adapters, thread/executor workers and sort
key adapters through nested operations. A standard class’s __del__ is retained
when a retained body arms the class after construction (issue #506). Release runs
the hook only for an armed instance, so this covers values dropped through
fields, containers, payloads, captures and queues; a descriptor of a class that
no retained body arms has no hook. Standard-library provenance comes from
the embedded-module loader, not a filename or symbol prefix. Generated foreign-call
wrappers preserve that provenance too (issue #688): an unused standard-library
engine import must not leave native references for the COFF linker to resolve.
Windows executable builds use a narrower root set for their temporary COFF
object (issue #690): entry operations and C exports, followed through the same
function references and concrete generator types in live signatures and locals.
Unreachable user helpers and generators are omitted together with the native
imports only they need. COFF resolves undefined symbols even in sections its
linker will discard, so section garbage collection alone cannot do this. A
reached foreign call keeps its real import; no fallback definition is generated.
All bodies still receive frontend and ownership checks. This applies in both
fast-compile and optimized executable builds, so an unreachable user function has no
executable symbol or breakpoint location on Windows. Explicit --emit-object,
library objects and --dump preserve their existing user-definition and generator
roots for inspection and external linking. External COFF consumers of such an
object must provide its native imports, including those in unused helpers.
Retained definitions keep their declaration order. All frontend and ownership checks run before this step, including checks of bodies subsequently omitted. This saves native work and object space; it does not avoid parsing or lowering imported modules. See the edit-build measurements.
Each function’s trap records are merged into one read-only source-site table. On AArch64, functions after the table’s first record address their own base through a local symbol alias into that same table. Their GOT relocations have zero addends: an ARM GOT relocation selects a GOT slot, so a placement-time offset on that relocation would not add to the pointer loaded from the slot. Aliases are finalized before object emission, preserving the same source records for cold, cached, and cache-disabled compilation without runtime pointer fixups.
Native emission builds and compiles each retained function on any available core. The main thread places finished functions in the object in declaration order, preserving byte-for-byte output across core counts. There is no worker-count flag. Tests compare one, two, and eight workers. A function compiled before, by this or any other program, is taken from the compiled-function cache instead; the object is the same either way.
For ordinary user functions, each worker reuses the previous compilation’s
cleared Cranelift Function buffers as well as its builder and compilation
contexts. Function names, signatures, instructions, source locations and symbol
references are rebuilt; only storage survives. Both cache hits and misses clear
the function before reuse. Storage lasts until the worker finishes, and a large
function can leave larger buffers for later small functions. Generator bodies
and generated adapters still construct fresh IR. See the
allocation measurements.
Workers claim one job at a time to balance functions of different sizes, but send completed results in batches of up to 32 jobs. This reduces result-channel wakeups on warm-cache builds. A partial batch is sent when a worker finishes or stops after an error; the main thread still places each job in declaration order. Batching can delay placement until the worker fills its batch and retains up to 31 additional completed jobs per worker before sending them.
With the cache enabled, a result keeps a shared owner of the cache’s immutable machine-code bytes until placement, avoiding a separate allocation and copy for each job. The object builder still copies those bytes into its own sections. Relocations remain specific to each function and are resolved independently. This adds no cache retention: decoded stencils already live until the compilation ends. With caching disabled, each result owns its machine-code buffer.
The plenty and plentyc executables allocate with mimalloc (version 2,
without transparent huge pages) instead of glibc malloc (issue #315). Workers
free operations the main thread allocated and grow IR buffers at a high rate;
glibc serializes such frees and growth on per-arena locks, so threads slept on
each other. Emitting 1,000 units from a warm cache on 16 hardware threads made
about 86,000 voluntary context switches with glibc and 480 with mimalloc. The
compiler library does not choose an allocator, so tests and other programs
that call it in-process keep their own.
A command-line check (--check, --check-module, and --package-graph --check) leaks the checked program instead of freeing it, since the process
exits next (issue #321). On the 1,000-unit benchmark this saves 2.0% of
instructions and about 4% of cycles; peak memory is unchanged. Only the
single-file compiler interface enables it, through
CheckOptions::skip_teardown: plenty lsp, project commands, tests and
library callers free every program. Other allocation strategies
were measured on 2026-10-20 and not adopted:
- Leaking the operation tables and object at the end of emission saved 0.02%
of
--emit-objectinstructions: workers already free each function’s operations once its code is built. - A never-freeing bump allocator, the upper bound for arenas, saved 10.5% of checking instructions and 2.7% of emission instructions, but took 30–45% more task time from page faults, and raised peak memory from 88 MB to 354 MB for checking and from 97 MB to 1.9 GB for emission.
- The allocator itself takes 8% of checking cycles and 6% of emission cycles. The largest sources in checking are lexer tokens and type clones in the frontend, tracked by issue #302; most emission allocation is inside Cranelift, whose per-function register allocator context Plenty cannot reuse without patching it (issue #153).
A body can need symbols that are declared on first use: type descriptors, its trap-site table, foreign imports. Their order decides the object’s bytes, so a worker building IR does not declare them. It names each in a placeholder namespace, and the main thread declares them in declaration order as it places the function, then points the function’s relocations at them. If any function fails, emission reports the error of the first failing function in that order.
Match dispatch
Integer literal and lowered enum-tag dispatch uses a jump table when at least 256 distinct keys form one contiguous range (issue #395). Other sets retain source-order comparison chains. This conservative threshold follows the dense-match measurements; small tables regressed for some input distributions.
The lowerer sorts keys with their original arm blocks, checks contiguity and
rejects duplicates or an early wildcard before selecting a table. It subtracts
the minimum key with wrapping arithmetic at the scrutinee’s width, then checks
the unsigned index against the last valid entry before reducing to the
32-bit index Cranelift’s br_table takes. This preserves negative ranges,
ranges crossing zero, unsigned maxima, and full 256-value byte ranges without
letting distant 64-bit values alias an entry. Storage is proportional to the
number of cases, never an unbounded sparse key span.
The default edge remains the wildcard or the defensive invalid-tag trap. Only dispatch changes: arm bodies, payload loads, ownership cleanup, source-order payload subpatterns, joins and tail exits use their existing lowering paths. No source syntax, allocation behavior or target CPU requirement changes.
A str match whose arms before the fallback are at least 16 distinct literals
searches one table instead of comparing each literal (issue #565). Codegen
sorts the literals by byte length and then bytes, keeping the first source arm
of any repeated literal, and emits a read-only data object of n + 1 32-bit
offsets followed by the bytes. plenty_str_match binary-searches it and
returns the entry index, or n for no match; br_table maps the index to the
arm block, with n going to the wildcard. The scrutinee is released once
before the branch. Other str matches keep their plenty_str_eq chain; see
the string match measurements.
A match on a standard sum whose arms test only tags 0 and 1, with any other arm a wildcard, reads a one-bit tag field. It tests that bit and branches, with no compare against each tag and no invalid-tag trap (issue #757).
When the scrutinee of a sum match is a local held in a register, each tag arm records the local’s tag for its body:
- An arm that rebuilds the matched variant around its own payload, in a sum
of the same width, reuses the matched value instead of splitting and
re-tagging it.
case Ok(value): Ok(value)in a generic helper that changes only the error type is the common case. - In the arm of a managed sum’s field-free variant 0, such as
Nothing, the local holds zero, so lowering writes the shared empty slot and the local’s later release emits nothing.
Splitting a sum built from two words, or widened from one, reuses those words
instead of emitting isplit, which Cranelift’s optimizer does not look
through, so a tag shifted out and back in folds away.
Symbol references
Code is compiled position-independent (Cranelift’s is_pic), so the object
links into a PIE or a shared library. How code refers to a symbol depends on
where the symbol is defined (issue #372):
| Target | Reference | Relocation |
|---|---|---|
| Program function, runtime helper | call rel32 | R_X86_64_PLT32 |
| Data the object defines locally (literals, trap-site tables, type metadata, coverage counters) | lea sym(%rip) | R_X86_64_PC32 |
Trusted C declaration, memmove, memset | load from the GOT, indirect call | R_X86_64_GOTPCREL |
| Address of a program function | load from the GOT | R_X86_64_GOTPCREL |
Runtime helpers are marked colocated: the runtime archive is linked into every executable and shared library made from the object, and a shared library’s version script makes the runtime’s symbols local, so the linker resolves the call directly, without a PLT entry. A C library may be shared, so its functions stay behind the GOT.
Under is_pic, Cranelift loads every symbol address from the GOT, even a
colocated one, and the object’s plain R_X86_64_GOTPCREL relocations are not
relaxed by the linker. After compiling a function, the compiler therefore
rewrites each GOT load of colocated data into a lea with a PC-relative
relocation (src/codegen/relax.rs), the rewrite a linker applies to
R_X86_64_REX_GOTPCRELX. Only data no other module can preempt is
colocated. The rewrite depends only on the code and the function’s IR, so it
applies equally to cached code, and the cache key, which covers
the IR, still identifies the result. Function addresses, taken for function
values, closures, generators, and runtime callbacks, are not rewritten.
On examples/http_server_bench.plenty with the standard library (779
functions, release), this removed 42,389 of 42,760 GOT loads and 35,220 of
35,590 indirect calls, and reduced machine code from 1,888,952 to 1,730,245
bytes (8.4%).
The worker count is the number of processors the process may run on. Restricting
its CPU affinity, for example with taskset, restricts the workers.
A function’s operations are released once its IR is built, and the object is written to its file one section at a time. Emission therefore holds neither every function’s operations nor a second copy of the object (issue #54).
Cranelift’s verifier checks compiler-generated IR, not source programs. It runs in debug compiler builds, including tests; release builds omit it.
The compiler uses Cranelift’s backtracking register allocator. Its time grows
with the square of a function’s branch arguments, the values its branches pass
to block parameters: two for each variable assigned in a conditional arm, for
example. A function with more than 2,048 branch arguments is compiled with
Cranelift’s single-pass allocator instead, whose time is linear (issue #56).
The largest function in the test suite has 156. Generated code reaches the
limit: about a thousand conditional assignments, two thousand match arms, or
a thousand-element collection literal in one function. In measurement such a
function’s machine code was 60% to 135% larger and took 1.1 to 3.6 times the
CPU cycles to run. Applied to whole programs, the single-pass allocator reduced
emission CPU time but increased code size and slowed native workloads, so no
other function uses it. The choice depends on the function alone, so it is the
same for every core count. tests/test_large_functions.rs runs functions over
the limit.
Identical code folding
A local function whose placed code equals an earlier function’s shares that
function’s code instead of getting its own copy. Code is equal when its bytes,
relocations, requested symbols, trap-site records and checks are equal; calls
compare by callee after folding, so a caller of folded functions folds too
when its callees were placed first. Quantity instances that differ only in
dimension are the common case, since
dimensions are erased:
sum[Length] and sum[Time] in std.units compile
to one body. An instance that formats a value or builds a collection keeps its
own body, because its type descriptor names the dimension.
A folded function is defined empty, and once the object module is finished its
symbol is moved to the shared code, so every call and table entry reaches it.
On Mach-O the moved symbol is marked N_ALT_ENTRY, keeping the code one
linker atom. Exported functions and coverage builds keep every body. So does
every function used as a value: function values
compare code addresses, so two distinct functions never compare equal. Other
functions may still fold into one. In --dump asm, a folded function lists
; shares the code of SYMBOL instead of instructions.
Call-site substitution
Before any body is lowered, src/codegen/small_calls.rs replaces two kinds of
call with the callee’s operations (issue #363):
- Constants. A read of a module
constbecomes a literal of the constant’s declared type, so width, sign and float bits are those of the declaration. An integer domain constant such asPositive[i64]becomes its literal and the domain’s constructor call, as a literal argument lowers. This happens in every build, and the constant has no function in the object. - Leaf predicates, in optimized builds only. A function whose parameters
and result are numbers or
bool, with no locals, and whose body is at most 12 operations drawn from parameter reads, literals, constants, comparisons,not,and,or, and matches onboolor integer literals. Nothing in such a body can trap, call, allocate, or touch memory. The caller stores the arguments in new local slots and runs the body; calls to one predicate in a body share its slots. The definition is still emitted, since a function value may name it.
Arithmetic, which can overflow, any call, and recursion keep the call.
Generator bodies keep predicate calls, since their frame layout is fixed by
then, as does plenty_main.
The substituted operations report no source position, so a later trap or call in the caller reports the same position and call chain as before. They become part of the caller’s IR, so the compiled-function cache key changes when a constant’s value or a predicate’s body does.
Call-site inlining
In optimized builds, src/codegen/inline_calls.rs lowers some calls by
lowering the callee’s typed body in the caller (issues #548 and #722). The
body keeps its own locals, source position, trap sites and cleanup, so traps
and ? failures behave as the call would; only its final return becomes a
jump. A callee is inlined when it is not recursive, not a generator, has no
tail call and defines no function, and its operations, counting the callees
inlined into it, stay within a bound:
| Callee | Bound |
|---|---|
A class’s new or __init__ | 64 operations |
| Any function with one direct call site | 160 operations |
| Any other function | 16 operations, or 24 while its call sites times its operations stay within 256 |
The larger shared bound (issue #770) covers bounds-checked accessors such as
Bytes.get and Bytes.set and short arithmetic helpers called in hot loops,
whose call would otherwise also keep the loop around it from being versioned.
The product bound keeps a helper called from dozens of places, as in the
regular-expression parser, a call, so code size and compile time grow little.
A tail call is inlined only when the caller has no managed locals or has just one tail-call site; otherwise it retains the shared tail-exit cleanup, keeping code growth linear in the number of owned locals and exits. C string adapter failures also return through the inlined callee’s exit, so the caller can handle the error.
An inlined function’s own body is built only if placed code or data still
refers to it, for example a call from a generator, a function value, a
protocol table or a thread adapter; exported functions are always built.
Otherwise the copy at its call site is its only code, so inlining a function
called once costs no extra compile time. The symbol of a body not built
stays in the object as a single trap instruction, so code outside Plenty
that calls it, such as C linked against the object, stops with SIGILL
rather than running into the next function. --dump builds keep every body.
Last-use moves
After substitution, src/codegen/last_use.rs turns the last read of a local
into a move (issue #388). A read of a local counts a new owner, and the
local’s own owner is released later, when it is reassigned or the function
returns. When no later operation can read the local, the read takes its value
and leaves an empty slot. A match on an owned parameter and a payload binding
read once are the common cases.
An empty slot holds zero, and the release of a value known to be zero emits nothing (issue #391). Every write of an empty slot in a function uses the same one-word and two-word zero constants from its entry block, so where all paths into a join leave a slot empty, the slot’s value there is that constant and its later release emits nothing. A slot moved from on only some paths keeps its release, which tests for zero before calling the runtime. In user code the test also skips the call-chain bookkeeping that names the release site, so an empty slot costs a compare and branch.
A local whose address is taken lives in a stack slot, and lowering reads it
back with a load, so it cannot see the zero there. Before compiling a
function, src/codegen/empty_slots.rs finds stack slots whose address is only
loaded from or stored to, and tracks which of their bytes hold zero on every
path; a call does not change them. A zero test or one-word release that reads
such bytes is removed (issue #549). A constructed value whose fields have all
been moved out is the common case.
A backward liveness pass over each function finds these reads. Loops are
solved to a fixed point; break, continue, early returns and the error path
of ? are followed. Only managed copyable types qualify: they have no
destructor, so ending the last owner earlier changes no output. Generator
bodies are skipped. A borrowed local moves only at a read after which none of
its loans is live, by the loan liveness of the ownership checker in
src/ownership.rs (issue #755). Reading a key after its get_mut lookup has
ended moves it into insert; a read followed by a use of a reference to the
local still copies. A local with any borrow that records no loan is never
moved. A one-word read that feeds a collection query keeps the borrowed read
from issue #379, which costs less than a move.
When a match reads the tag of a local it has just copied, the copy’s counting
is removed, so testing an owned scrutinee costs no runtime call. The pass runs
after the ownership checker, so diagnostics are unchanged.
Constants in loops
In optimized builds Cranelift rematerializes an integer or float constant in each
block that uses it rather than keeping it in a register across a loop: this
saves a register for the loop’s lifetime at the cost of one instruction per use.
On x86-64 a constant that is the right operand of a float operation is read from
the constant pool. A left float operand, or a 64-bit integer constant in a
wrapping_add, is built with movabs (plus movq for a float) each time it is
used (issue #370).
Lowering therefore moves a non-NaN constant to the right of float + and *,
which gives the same bits: only the choice between two NaN operands depends on
order. 2.0 * x and x * 2.0 become x + x, which rounds, overflows, and
propagates signed zero, infinities and NaN identically. No other float
operation is rewritten; there is no reassociation, fusion, or fast-math mode.
A non-zero constant on the left of - or /, which cannot move, is emitted as
lane 0 of a vconst vector constant (issue #380). Cranelift does not
rematerialize a vconst, and x86-64 loads it from the constant pool with one
movupd, so 1.5 - x reads its constant without a general-purpose register.
The operation and its operand bits are unchanged. Zero keeps its f64const,
which lowers to a register-clearing xorpd, and a constant whose other operand
is also constant is left for Cranelift to fold. In a throughput-bound loop with
four c - x terms (taskset, perf stat, 10⁹ iterations, Ryzen 7 7840HS) the
change lowers cycles from 8.8 to 5.4 billion, matching the same loop with the
constants passed as arguments; in a loop bounded by a dependency chain the
movabs was off the chain and the cycles do not change.
x.fabs() and x.copysign(y) are bit masks rather than runtime calls: band
with the magnitude mask, and bor of that with band(y, sign mask). Each
mask is lane 0 of a vconst, loaded from the constant pool with one movupd;
Cranelift’s own fabs and fcopysign build their masks with movabs + movq.
Since Cranelift 0.137 the x86-64 backend keeps both operands of a scalar float
band, bor or bxor in registers, because a memory operand reads 128 bits,
so an f64const mask would also be built with movabs (issue #439).
Like Rust’s abs and copysign, they keep a NaN payload. A constant receiver
or sign argument is folded into the masks while lowering.
Float negation -x is bxor with the pooled sign mask rather than Cranelift’s
fneg, whose x86-64 lowering builds the mask with movabs + movq at each
use (issue #385). Both flip only the sign bit, NaN payload included. Lowering
folds a constant operand and -(-x); Cranelift’s other fneg rewrites, such
as (-x) * (-y) to x * y, no longer apply. In a throughput-bound loop with
four s + -(x * a) terms the change lowers cycles from 7.7 to 4.4 billion and
instructions per iteration from 30 to 22.
Integer constants are left as Cranelift places them. In the
examples/cpu_codegen_audit mix kernel the per-iteration movabs is off the
loop’s dependency chain: passing the constant as an argument instead removes
the instruction but not a cycle (7.05 billion cycles either way for 10⁹
iterations). Removing the rematerialization would need a Cranelift change.
Float rounding
x.floor(), x.ceil(), x.trunc() and x.round() are inline SSE2
sequences, equal bit for bit to Rust’s floor, ceil, trunc and
round_ties_even (issue #386). Generated code targets the
SSE2 baseline, where
Cranelift’s floor, ceil, trunc and nearest are library calls rather
than SSE4.1 roundsd.
For a magnitude a below 2⁵² (2²³ for f32), (a + 2⁵²) - 2⁵² is a
rounded to a whole number, ties to even, because numbers in [2⁵², 2⁵³) are
spaced 1 apart. round uses that directly. trunc steps back one when the
rounded magnitude exceeds a; floor and ceil apply the sign of x and step
one down or up when the rounded value lies past x. The step is a vector
compare (cmppd) whose lane mask selects the constant 1, so no branch depends
on the fraction. ORing in the sign bit of x gives -0.0 for a negative x
whose result is zero. A larger magnitude or an infinity is already whole and is
returned unchanged, behind the one branch; a NaN passes through the arithmetic,
which quiets it and keeps its sign and payload. A constant receiver is folded
while lowering.
The calls went through plenty_f64_math to Rust’s libm, which rounds with
integer operations and data-dependent branches, and forced live float registers
onto the stack. With --release, taskset on one core, perf stat, Ryzen 7
7840HS, best of five:
| Loop (2×10⁸ iterations) | call (G cycles) | inline (G cycles) |
|---|---|---|
Two floor terms, steady step | 4.0–4.9 | 1.8–2.0 |
One floor of a random value | 5.2 | 2.0 |
One round of a random value | 6.8 | 1.8 |
Dependency chain y - y.floor() | 7.3 | 5.0 |
Dependency chain y - y.round() | 10.4 | 3.4 |
ceil and trunc match floor. A chain whose rounding the call’s branches
predict exactly, such as floor of values always in [0, 1), can be faster as
a call, because the prediction takes the call off the chain. Each use adds
about fifteen Cranelift instructions to compile: a file of 2,000 one-line
functions with four rounding calls each takes 55% (release) to 85% (debug)
more compiler instructions.
Loop layout
A loop header whose condition ends in a compare branches to the exit on the
complemented compare, so the exit is the branch’s first successor (issue #767,
lower_loop_copy in src/codegen/collections.rs). Cranelift orders blocks by
a depth-first search that visits the first successor first, then emits them in
reverse postorder. With the body as the first successor, every loop body lands
after all code reachable from its exit, at the end of the enclosing loop. A
value used after an inner loop and live across it then sits in a register
through the out-of-line body, while the header keeps it on the stack: regalloc2
reloads it at the start of each pass. With the exit first, the body follows its
header. In fannkuch-redux this removes reloads from three hot inner loops.
A condition that is not a compare, such as the join of a short-circuit and,
keeps the body as the first successor: inverting it would cost a compare and a
second branch on each pass.
Loops are also rotated into do-while form (issue #726). The header becomes a
guard that tests the condition once and branches to the exit or the body. The
body ends in a latch that tests the condition again and branches back to the
body or to the exit, so a pass ends in one conditional branch instead of a jump
to the header and the header’s branch. continue goes to the latch. Each copy
of a versioned loop is rotated on its own; the guard keeps the exit-first
branch above.
The condition is lowered twice, so rotation is limited to conditions whose
second copy adds no observable work: reads of scalar, bool and view locals,
arithmetic, compares, casts, not, short-circuit and/or, and a view’s
length and items (rotatable). Both copies carry the condition’s source
position, so a failing index or overflow reports the same error and location
from either copy, and they evaluate in the same order. A condition with a call,
a list’s length, a collection method or a managed value keeps the header test.
A loop’s own index facts (the lo < hi relation and in-range counters in
src/codegen/induction.rs) describe values read in the body. Neither copy of
the condition records them, since the guard’s values also reach the exit when
the loop does not run (issue #816). An unrotated loop records none: its body
reads the header’s values, which reach the exit too.
A body that never falls through and has no continue gets no latch.
Rotation removes one instruction per pass from tight loops: the reverse
kernel’s swap loop matches C’s eight instructions. In loops that make calls,
regalloc2 can place a split’s moves on the latch’s back edge instead; fasta
gains about 0.7% instructions this way.
Values carried to shared exits
A function’s ? failures, returns and tail calls share their cleanup and
release code: each exit jumps into a chain of shared blocks, passing the
values it carries (src/codegen/exits.rs). regalloc2 merges a block
parameter with the arguments passed to it when their live ranges do not
overlap. Parameters that every exit passes to join each exit’s values, and
the values those share registers with, into one bundle spanning the
function. regalloc2 splits such a bundle a few times, then cuts it into
minimal pieces whose spill choice ignores loop depth, so a loop’s counter or
total ended up stored and reloaded on every pass.
In a function with a loop, counting loops of callees inlined into it, an exit
therefore stores its carried values in a stack slot and jumps to a reload
block, which loads them and enters the shared blocks (issues #771 and #811).
The loads address the slot from a different base than the stores, so
Cranelift’s alias analysis cannot replace a lone exit’s loads with the stored
values and reform the merge. A loop-free function passes parameters: it has
no hot loop to protect, and a hot return path, such as a constructor called
in a loop, would pay a store and a load. break and continue pass
parameters too. Staging returns removes 4.4% of n-body’s instructions and
1.6% of fannkuch-redux’s.
Loop counter checks
Integer arithmetic keeps its overflow and division checks, and Cranelift does
not remove them, since it does not track value ranges. The lowerer drops two
kinds that a while loop’s own condition makes impossible (issue #369,
src/codegen/induction.rs). Both apply to a counter kept in a CLIF variable
whose condition compares it with another local or a literal, as in
while i < n or while i >= 0:
- Step overflow. The first write to the counter in the body, if it is a
top-level
i = i + kori = i - kwith a literalk, runs at most once per pass, with the condition still true.i < nputsiat least one below the type’s maximum, soi + 1cannot overflow; a literal bound also admits larger steps that stay in range, andi > nadmitsi - 1. Such a step is a plain add. Later writes, steps inside nested loops, and steps such asi <= nwith a variablenkeep their check. When the bound is also a local, its own first step is proven the same way, so inwhile lo < hibothlo += 1andhi -= 1are plain (issue #720). - Signed division. A signed counter that enters the loop holding a
nonnegative literal, and whose every write in the loop adds a nonnegative
literal, is never negative, since each add is checked or proven.
//and%of such a counter by a nonnegative literal or counter use unsigned division, with no floor correction orMIN // -1check. The zero-divisor check stays.
A for loop over a range(start, stop) call with step 1 and an i64 or
u64 element gets both facts from the range instead of a condition: every
value of the loop variable i is below stop, so a top-level i + 1 in the
body is a plain add, and a nonnegative start keeps i nonnegative. The
lowerer accepts the loop only when its hidden counters start from that
range’s computed length, start, and step, and nothing in the body writes them
except the loop’s own step before continue. Reversed ranges, other steps,
narrower elements, and range values in locals keep every check.
Counters in memory, borrowed locals and generator frames, are not analyzed.
for loops over ranges step with wrapping arithmetic. A range over
an integer type narrower than i64, or one with a positive literal step and
a start known nonnegative, cannot have more than i64.MAX elements, so its
length needs no overflow check (issue #721).
Short circuits in conditions
a or b lowers to a match on a whose arms push True or evaluate b; the
arms meet in a join block that takes the bool as a parameter. Cranelift does
not thread branches through block parameters, so an if, while, match, or
enclosing and/or that consumes the result would test the join parameter
again, even on edges where an arm passed a constant.
Before Cranelift runs, src/codegen/threading.rs retargets each edge into a
block that only passes its parameters on or branches on them, when no other
block uses its values and it has no instruction that traps, touches memory, or
calls. An edge whose arguments decide the block’s brif goes straight to the
chosen target; a jump whose argument is the unknown condition, or its not, becomes that
brif, with the targets swapped for not (issue #377). Only branches move, so
operands still run in order, only when needed, with their traps and ?
propagation. A short-circuit result stored in a variable and read after the
condition keeps its join block.
Measuring compilation
A local run of compiler-benchmark used 40,000 functions, three requested runs, and disabled caches. The machine and run date were not recorded, so these numbers are not independently reproducible:
| Compiler | Build + run (µs/function) |
|---|---|
| Clang 22.1.8 | 172 |
| Plenty 0.2.4 | 233 |
| Zig 0.16.0 | 269 |
| GCC 16.2.1 | 367 |
| Go 1.26.10 | 440 |
Plenty’s build took approximately 9.3 seconds, with 176 MB peak RSS. Generated-code optimization was off; only the non-generic Plenty workload was implemented. All four adapter tests passed, including native-result verification.
examples/compile_bench.rs generates units containing a protocol, class, enum,
and three functions. It reports median checking time, optionally full AOT/linking,
and a fourfold-input scaling ratio. Functions and methods independently declare
precise arithmetic errors, use explicit successful Ok returns and propagate
fallible callees without wrapping each operator. The current template generates
40,003/160,003 lines for 1000/4000 units; see its
functional validation and controlled comparison.
Historical measurements retain their original source shapes.
tests/test_compile_scaling.rs guards against
superlinear checking of classes, functions, protocols, enum variants, and
match cases.
For time and allocations per compiler phase in one run, build with the
profile feature; see Profiling the compiler.
For the 2026-10-10 release measurement on an AMD Ryzen 7 7840HS, x86_64 Linux, 16 hardware threads: 4,000 units / 156,003 lines checked in 1.61 s and 16,000 units / 624,003 lines in 6.69 s. Parallel emission and omitting release-build verification reduced full AOT of 1,000 units / 39,003 lines from 3.25 s to 0.87 s, and object emission of 5,000 units / 195,003 lines from 16.9 s to 4.35 s (2.6 s checking). With one core the latter took 13.2 s, isolating verifier removal. These are workload measurements, not general latency guarantees.
The 2026-10-12 release measurement used the same machine restricted to its
eight physical cores, and 5,000 units / 195,003 lines. Checking function bodies
on all cores reduced the independent check from 0.48 s to 0.09 s and --check
from 1.81 s to 1.44 s (five runs each). Object emission then spends, in seconds:
| Phase | Threads | Time |
|---|---|---|
| Parsing and name resolution | 1 | 0.31 |
| Type resolution, class expansion, generic preparation, signatures | 1 | 0.35 |
| Lowering bodies to operations | 1 | 0.41 |
| Layout, operation validation, rest of the frontend | 1 | 0.13 |
| Independent check | 8 | 0.09 |
| Tail-call preparation; declaring functions, strings, adapters | 1 | 0.19 |
| Building IR and compiling it | 8 | 2.1 |
| Serializing and writing the object | 1 | 0.22 |
On one core, building IR takes 0.95 s and compiling it 13.1 s. On eight, the compile step holds every core: the main thread spends 0.05 s defining compiled functions and otherwise builds IR or compiles.
With all 16 hardware threads the same emission took 9.9 s, against 4.2 s on
the eight cores. The instruction count was the same; above about ten busy
threads this machine ran at 1.1 GHz instead of 3.9 GHz. Measure on a fixed set
of cores, for example taskset -c 0,2,4,6,8,10,12,14. The 2026-10-20
measurement below did not reproduce the slowdown.
The 2026-10-15 release measurement used the same eight cores while other builds loaded the machine, with paired runs of the compiler before and after building IR on worker threads. Object emission of 5,000 units took 6.0 to 6.5 s before and 5.7 to 5.9 s after (three runs each); 20,000 units took 25.8 to 26.7 s and 23.2 to 24.4 s (two each). CPU time and peak memory were unchanged. The per-function trap-site tables added 0.4 MB of read-only data to the 5,000-unit object (66.6 MB) and 0.2% to a linked 1,000-unit program’s code and data. The remaining serial work is the frontend, about 1.9 s of the 5,000-unit emission.
The 2026-10-16 release measurement used the same eight cores, again beside
other builds, with paired runs before and after resolving names, validating
frames, preparing tail calls, and finding thread and executor adapters on all
cores. For 5,000 units these phases took 48, 19, 35, and 20 ms instead of 99,
85, 47, and 35 ms. --check took 1.77 to 2.02 s before and 1.62 to 1.84 s
after, and object emission 6.1 to 7.9 s before and 6.0 to 6.3 s after (five
runs each).
Peak memory was unchanged. On 1,000 units the difference was within noise.
The 2026-10-20 release measurement compared glibc malloc with mimalloc in
paired, alternating runs of --emit-object (no linking), with a load average
of 4 to 7 from other builds. Medians of three runs for 5,000 units, wall time
with user and system CPU time in seconds:
| Cache | Cores | glibc | mimalloc |
|---|---|---|---|
| Off | 8 physical | 2.26 (7.7 + 0.41) | 1.98 (7.2 + 0.37) |
| Off | 16 hardware threads | 2.07 (10.7 + 0.61) | 1.74 (9.9 + 0.45) |
| Warm | 8 physical | 2.11 (3.7 + 1.55) | 1.42 (2.4 + 0.49) |
| Warm | 16 hardware threads | 2.13 (4.8 + 3.16) | 1.39 (2.9 + 0.57) |
With mimalloc, voluntary context switches fell from 556,000 to 1,200 for the warm 16-thread build, and peak memory was 375 MB against 388 to 414 MB. For 1,000 units the warm build took 0.26 s instead of 0.37 s on 16 threads, and peak memory rose from 92 to 110 MB. Sixteen threads were as fast as eight or faster, so the worker count stays one per available processor. In this run the clock stayed near 4 GHz on 16 threads; sixteen threads executed the same instructions as eight in 31% more cycles, from sharing cores. Parsing, type resolution, class expansion, generic preparation, signatures, and lowering remain on one thread, about 0.75 s of the 5,000-unit frontend, as do declaring functions (0.1 s) and writing the object.
The 2026-10-17 release measurement used eight logical processors on four
cores (taskset -c 0-7), beside other builds, before and after resolving each
declaration once: class templates are copied without method bodies, concrete
types are defined without copying their declarations, and signatures are
registered from the types generic preparation resolved. Checking 5,000 units
spent, in milliseconds (medians of nine runs each):
| Phase | Threads | Before | After |
|---|---|---|---|
| Parsing | 1 | 279 | 279 |
| Name resolution | 1 and all | 53 | 53 |
| Type resolution | 1 | 107 | 86 |
| Public API check and class expansion | 1 | 63 | 64 |
| Generic preparation | 1 | 140 | 149 |
| Signature registration | 1 | 85 | 28 |
| Lowering bodies to operations | 1 | 465 | 462 |
| Frame validation | all | 24 | 25 |
| Independent check | all | 128 | 134 |
| Freeing the program | 1 | 121 | 124 |
--check took 1.51 s before and 1.46 s after (eight runs each); 1,000 units
took 0.30 s and 0.28 s (ten runs each), and a program whose imports load 21
standard modules 119 ms and 112 ms (twenty runs each). Peak memory fell from
406 MB to 404 MB at 5,000 units. Object emission of 5,000 units took 5.25 s
and 5.21 s (medians of six runs each), within noise.
Running the declaration phases, from type resolution to signature registration, on several threads could save about 0.1 s of their 0.33 s at most. Type resolution creates nominal identities and generic instances in program order, and registration fills the signature table; only resolving signatures and expanding classes read the tables without changing them, and they would need thread-safe type tables. The parallel designs measured for parsing and lowering are in Frontend and checking. Freeing the program after the independent check costs 0.12 s, which a process about to exit could leave to the operating system.
Report build mode, machine, input size, repetitions, and included phases with new results. Measure checking, IR construction, native emission, runtime extraction, and linking separately when assessing compiler changes.
Value representations
Classes, tuples, and enum payloads keep fields inline in their owner’s storage. Native slots contain ordinary value bits plus any required inline payload bytes. Moves and copies relocate private addresses into the destination; addresses never extend the lifetime of departed frames. Immutable metadata describes field layout, ownership, relocation, equality, and formatting. Concrete layout facts are cached.
Runtime record traversal advances a running slot offset, including each field’s inline payload and any class cleanup state word. A complete pass takes one width lookup per field, without allocating an offset table. Hashing, equality, ordering, formatting, copying, relocation, and retain/release use this traversal. Copy failure cleanup walks the completed prefix backwards with a decreasing offset. This bounds offset work per pass; nested payload work and repeated visits while queued destruction resumes remain separate costs. Strings, the most common heap field, are freed during the pass and cause no resumption.
A class with exactly one field and no __del__ is represented as that field:
same slot, same bytes, no record and no internal address. Constructing it,
reading or moving its field, copying it, and releasing it are the field’s own
operations; Wrap(file) costs nothing beyond file. Its runtime descriptor
carries the field’s layout under the class’s name, so formatting still prints
Wrap(x=1) and columnar lists still see one column. A one-field class with
__del__ needs the cleanup state word and keeps the record layout, as do
classes with two or more fields. The
flat class layout proposal records
why the record holds an address and the layout contract C interop depends on.
A field whose type is a class with a record (after looking through one-field
wrappers) is stored flat when its owner is a class record, a tuple, or an
enum variant with two or more fields: its record bytes sit at a fixed offset
in the owner’s record, with no slot word of its own. Reading, borrowing,
matching or writing owner.inner.x adds constant offsets to the owner’s record
address; moving the owner copies its bytes with no address fixes for flat
fields, so a tuple or variant of such classes moves as plain bytes. An
all-zero flat record is an empty field, so moving a field out zeroes its bytes
and releasing an empty field does nothing. The rule has these exceptions, which
keep the slot word followed by the record:
- top-level class values (locals, list elements, boxes, captures), because runtime operations and sum payloads receive one 128-bit value;
- the payload of a single-field variant, such as
Some(point)orOk(point), which is that one 128-bit value with the sum’s tags stacked in its high word: the record follows it, and moving the sum fixes only that one address, as moving a bare class value does; - field-less classes without
__del__, which have no record bytes to place.
A reference to a class value names the 16 bytes before its record whether or
not a slot word is stored there: readers use address + 16 without loading
the word, writers copy the record bytes, and writing an empty value zeroes
them. A __del__ hook receives such a reference and must not read the word.
The runtime release walk clears a finished flat record so a resumed walk skips
it.
Compiled code avoids the runtime walk when the type makes it unnecessary. A payload with no internal addresses, such as a range, a flat record, or a closure environment without inline captures, is copied as plain bytes; a sum whose payloads are all such is copied by testing its tag. An all-zero value is an empty ownership slot, such as a moved-from local, and its release makes no runtime call. These paths keep high-frequency helpers, such as error mappers called per byte, out of runtime operation calls.
Record copies and clears of up to 192 bytes are emitted inline as 16-byte
vector loads and stores rather than memmove and memset calls. A local moved
straight into a call argument, with only operand pushes between the move and
the call, is passed at its own address instead of a fresh copy: the callee
copies its arguments on entry, so the caller’s storage is not read again.
Inlined constructors, generator producers and portable SIMD operations keep
argument addresses, so moves into them still copy.
A class with a record also has a plan in its runtime descriptor: the word
offsets of the strings it owns, including those of flat class fields, whether
those strings are all it owns, and whether its bytes hold any address into
themselves. When the strings are all it owns (no __del__ at any depth, no
other managed field), the runtime releases the record by freeing those strings
directly instead of walking its fields; freeing a string runs no user code, so
the order is unobservable. A record holding no internal addresses moves without
relocation: only its owner’s value word changes. Collection rows use both, and
a list whose row class owns nothing skips its per-row release entirely.
References use two native words: an address and packed inline-sum tag-offset information. They point into typed local, frame, field, or collection storage; source loans determine when those locations remain usable. Function values are code addresses without capture environments. Returned-reference analysis caches precise field-origin summaries when available and otherwise uses conservative owner projections.
Scalar references read and write only the scalar’s declared bytes, whether they address a packed list element, a local slot, or a projected enum payload. Slot padding is not part of the scalar value; runtime equality, hashing and formatting ignore it. Projected writes leave ancestor tags unchanged.
Generator resumption reserves space for the complete inline yield layout after the runtime result word. The shared runtime-call scratch area grows for larger records and nested payloads; yielding a record does not impose a fixed-size limit.
Lists
Dense lists pack primitive numeric and boolean elements into 1-, 2-, 4- or
8-byte rows with natural alignment. Other rows contain a 16-byte value slot
and its inline payload, with no unused dictionary value slot. The runtime
keeps a 16-byte-aligned buffer, rounding allocation sizes up to whole words;
its exposed length counts initialized bytes. Borrowed scalar indexing uses
the concrete element width to calculate row counts and addresses. General
borrowed iteration obtains the runtime row stride. Native strided FFI views
use that same stride, and signed reads extend their declared bytes without a
post-call repair pass. List storage costs
describe the remaining space used by sum and record elements.
Strings
A str is one 64-bit word. Heap and literal pointers address a 32-byte prefix:
a 16-byte managed header, byte length, and scalar count, followed by UTF-8 bytes
without a terminator. Literal storage is immortal and eight-byte aligned.
A set low bit marks an inline string. Its low byte stores length << 1 | 1;
the remaining seven bytes store UTF-8. These values own no heap allocation and
retain/release ignores them. Byte equality, hashing, and printing handle both
representations identically. Source-visible text behavior belongs in
text operations.
A &substr is two
words, like a &[u8]: the address of its first UTF-8 byte and its byte count.
It has no prefix, so it is not a &str. A view of an inline string addresses
the bytes in the borrowed string’s slot, which its loan keeps in place.
Standard sums
Option and Result use a 128-bit value: a 64-bit terminal payload and a 64-bit
path of binary tags, outermost first. Nested sums add bits without boxing, bounded
by the 64-level type limit. The payload is scalar bits, an owned heap handle, or
a private address of inline data. No numeric sentinel or null pointer is reserved
as a source value. LoadError uses three terminal tag bits for seven variants.
Scalar/pointer sums use 16-byte addressable slots. Inline payload bytes follow the
value word; slots grow only when the payload requires it. Native results use
caller-owned storage, and relocation preserves sum tags. Runtime aggregate helpers
receive aligned slot pointers instead of relying on a C u128 calling convention.
A one-word operand is stored as its word and a zero word, so no two-word zero
stays live across the call (issue #757).
These layouts are private and are not the public FFI ABI.
Ranges
Ranges contain 32 bytes of bounds, step, and length. Stored ranges use 48-byte slots: ordinary value storage plus inline range data. Collection relocation and frame/result moves repair embedded addresses. Range creation and return need no separate owner allocation; see ranges.
Memory and destruction
Each live managed operand, local, field, and capture has an owner. Moving an affine value clears its source slot. Loading a copyable inline value copies its storage and retains contained strings. Stores stage the new value before releasing the old owner. Source cleanup order is defined by destruction, not reference counts.
Runtime objects share {atomic u64 refs, destroy_callback}. Heap objects begin
with one reference; literals use an immortal count. A
Shared[T] handle is a box whose
count is shared by its source-level owners. Native helpers borrow inputs
and return owned managed results, including retained projections and builder aliases.
Buffers have explicit owners; type metadata is immutable program data.
A class instance owns its fields in a record inside its owner’s slot; the slot’s
value word is the record’s address, which moves relocate. A class field, tuple
component or multi-field variant field of class type is stored flat inside its
owner’s record, so moving the owner copies bytes without relocating that field;
an all-zero flat record marks a moved-out or released field. A one-field class
without __del__ has no record: the instance is its field, so moving it or
moving the field out copies the field’s bits and releases nothing. See
value representations.
Retain uses relaxed atomic updates. Final release acquires preceding releases before destruction; the final thread owns the destruction queue entry. This protects ownership counts, not concurrent mutable payload access or foreign resource transfer. Source permissions still come from borrow and worker checks.
Sole-owner release
Release first loads the count. An immortal count returns. A count of 1 skips the locked decrement: an acquire fence follows the load and the object is destroyed, leaving the dead count at 1 until the destruction queue reuses the word. Other counts take the release decrement.
This is sound because the releaser consumes a counted reference it owns, so the count is at least 1 and other counted references or borrows account for the rest:
- Every count operation by another thread comes from a counted reference or a borrow of one. A borrow of the releaser’s own reference ends before the release, through a scoped-worker join or a job or channel handoff, which orders that thread’s retains before the load. By coherence the load then reads their increments or a later value, so it cannot read a stale 1.
- When the load reads 1, no other counted reference exists, and no new one can appear: creating one needs the releaser’s reference, which it owns exclusively. A retain racing with this release would already race with destruction under an atomic decrement, so borrow and worker checks exclude it.
- The 1 was written by the initial store or by another owner’s release decrement. The acquire fence synchronizes with that decrement, so the other owner’s payload use happens before destruction, as under the atomic path.
Shared[T]handles each own a count, so two live handles keep it at 2 or more. Borrowed contents are released only when the last handle destroys the box or under a mutex.- The destruction queue writes its link only after the release has decided to
destroy, as before. Immortal literals keep
u64::MAXand are never written, so read-only literal data stays valid. Copies into another actor heap check the count only for the immortal value.
Retain has no such path. It also runs through shared borrows, which carry no
count: a scoped worker and its parent copying one string, or two jobs reading
the same Shared[list[str]], can both read 1, and a plain store of 2 would
lose an increment. Retain therefore always uses an atomic update. Compiled
code changes counts only by calling plenty_retain and plenty_release.
An iterative queue destroys owned graphs without recursive native-stack growth.
A record, list, or box being destroyed hands each heap child to the queue and
resumes its walk after that child is gone, which keeps cleanup in declaration
order. A string owns nothing and runs no user code, so the walk frees it in
place instead. A class whose fields own only strings, at any depth and without
__del__, is released in one step from the string offsets its descriptor
records; see value representations.
Inline generator cleanup uses the count handoff but finishes synchronously while
its enclosing storage is live. Private helper aliases do not authorize mutable
source aliases. Traps end the process without unwinding language scopes.
Box destruction
When a box’s content releases without running code, the free order of its children cannot be observed. Its descriptor then carries a compiled release of the content slot, and the runtime calls that function instead of walking the content’s descriptor. The compiler builds one for each boxed type that is:
- a string, collection, box, or another value released with one
plenty_release; - a sum such as
Option[Box[T]]in which one variant owns a single such value; or - a class without
__del__whose fields are of those kinds or own nothing.
The function reads each owned child at its fixed offset and passes it to
plenty_release. During destruction that call only queues the child, so deep
trees still drop without growing the native stack. Content whose release runs
a destructor keeps the ordered walk described above.
Box(value) for content with inline storage, such as a class record, asks
the runtime only for the allocation. Compiled code then copies the content into
the box’s slot. If the allocation fails, the runtime releases the value the box
would have owned. The slot is not zeroed first, just as a local’s stack slot is
not: the copy writes the value word and the payload that word names, such as the
current variant’s fields of a sum, and every reader of the box reads the value
word first and then only that payload. Bytes past a smaller sum variant stay
unwritten and are never read.
Allocator provenance
Runtime objects and container buffers both come from the global allocator, and an object allocation carries no prefix before its header: with only one allocator, freeing needs no record of which one created an object. A box is a 16-byte header, its descriptor word and a padding word, then its content slot. Public allocator selection and scoped allocator lifetimes are unsupported. Adding them (backlog B13) must record provenance again without enlarging objects of the global allocator; a 16-byte prefix made each binary-trees node a 112-byte malloc chunk rather than 96 (issue #814).
Checked buffer growth and partial-construction cleanup implement the
recoverable allocation contract.
The compiler’s runtime-checks feature enables the embedded runtime’s
allocation-checks instrumentation for native integration tests. Allocation-free
regions count successful allocations, including those freed inside the region;
failed attempts do not count. Failure injection is separate. See
runtime validation.
Collection storage and costs
collection.rs supplies static operation signatures. Typed loops and builders
lower to structured operations; Cranelift calls a fixed runtime ABI using
128-bit slots and immutable element metadata. No per-element compilation or
dynamic type inference occurs.
Lists use dense storage with constant-time indexing. Primitive scalar rows use
their declared widths; other rows use a value slot plus their inline payload.
The buffer is a byte prefix of an aligned Vec<u128> allocation. Inline records,
tuples, and enum payloads occupy contiguous rows; buffer moves relocate private addresses.
Mutation grows buffers in place. Explicit copy duplicates owned contents.
Literals and comprehensions build in amortized linear time under ordinary hash
distribution; operation contracts retain their documented allocation failures.
Hash storage
Dictionary/set hash buckets point to stable row slots. A separate fallibly allocated buffer uses two machine words per reserved slot (16 bytes on the native target) for insertion-order links or the dead-row free list. Occupancy does not depend on payload bits. Removal unlinks a row and repairs its probe cluster by backward shifts, without moving payloads or scanning the whole table.
Each bucket is one 64-bit word. For a table of 2^s buckets, its low s bits
hold the row slot and the bits above hold the low 64 - s bits of the key’s
hash, so storing hashes costs no memory. A probe compares keys only where those
bits match its hash. Growth rebuilds the table from stored bits without reading
or hashing keys, and removal finds a shifted bucket’s home from them; both fall
back to rehashing keys only above 2^32 buckets (issue #761). Load stays at most
one half, so the slot field always fits.
Stored bits also serve another table at least as long, which needs no more of
them (issue #817). A copy whose rows and table length match its source copies
the bucket array. Set algebra probes each operand with the other’s stored bits
and indexes its result from them, update reuses the consumed source’s bits,
and intersection_update and difference_update repair the index in place.
Where a target table is shorter than the source’s, each key is hashed once; a
copied dictionary with removed rows rehashes its keys.
New keys reuse free slots at the logical tail; replacement preserves position. Live-row cursors make traversal, snapshots, copying, formatting, and destruction follow insertion order without visiting dead rows. Capacity follows peak occupancy and explicit reservation, and persists until destruction. Hashes use SipHash-1-3 with a process-wide secret key obtained from OS entropy. The key is shared across worker threads; dictionary insertion order remains independent of bucket placement. See the hashing contract for initialization, failure behavior, and complexity limits.
Top-level str, bytes, integer and bool keys hash their contents
directly. Tuple keys recursively hash scalar fields, string bytes, and byte-literal
contents using their static metadata. Inline storage addresses never participate in the hash, so copied
keys, relocated rows, lookup temporaries, and newly constructed equal strings
all retain the same hash. Tuple hashing and equality allocate nothing.
std.deque builds a ring buffer in Plenty over a private list of optional
payloads. End removal replaces a slot with Nothing and transfers its previous
owner. Growth doubles the slot count and initializes a replacement buffer
before moving live entries; list removal behavior is unchanged.
std.collections keeps each binary heap in a private list and restores heap
order with swap, so sifting never copies or drops a payload.
std.bytes does not use collection slots. Its private owner holds a contiguous
one-byte-per-element buffer, reached only through range-checked runtime
operations that never expose an address. See std.bytes.
Reservation obtains bucket, link, and row capacity before publishing indices or transferring owners. Failure preserves contents and order, though capacity may have grown. Relocation ignores dead rows. A popped inline payload stays readable until mutation or receiver release; codegen snapshots it before either.
Narrow native helpers
Integer, float and bool element reads and writes through a list place are
inline machine code, with no runtime call (issue #509). This applies to any
list place: a local, a field (self.xs[i], a.b.xs[i] = v), a borrowed
receiver, or an inner row (grid[i][j] += 1), in any loop. Codegen loads the
list’s first-row address, stored byte count and popped-row flag from the owner
at the offsets in plenty-runtime/src/list_layout.rs, which the runtime
asserts against its repr(C) Collection and Entries; a scalar row is 1, 2,
4 or 8 bytes according to its type. One unsigned comparison accepts an index
in 0..len; any other index takes a cold path that adds the length and checks
again, so a nonnegative index needs no normalization (issue #715). A constant
index, or one known nonnegative, takes the single comparison only, and both
paths join on the row address (issue #720). An access to the same rows,
length and index values as a check earlier in the block, or in the chain of
check and trap-guard blocks leading to it, reuses that check’s address, since
it cannot fail if the first passed (issue #721). It then
loads or stores the declared bytes. Reads preserve payload bits before
narrowing or bitcasting, and writes store the same bits a write through an
element reference would. An index out of bounds calls
plenty_trap_list_index, which reports the index as written, the length and
the site, as the generic path does. The failure block stores the index and
length to the function’s scratch slot and passes its address: as register
arguments, they would pin the loop’s values to the call’s argument registers
and add moves to every iteration. The length of a borrowed scalar list is
read the same way. The list’s loan keeps the rows in place for each access.
Outside the loop caches below, codegen does not cache the address across source
operations, so growth between accesses is observed.
A for or while loop caches the length and first-row address of a named
scalar list, or of a scalar-list field of a named class value
(self.items[i]), defined before it (issues #429, #715 and #717). The frontend
emits the header reads before the loop and lowers xs[i] in the loop to
ListScalarGetView or ListScalarSetView, which take the cached values as
operands. A list of any other element type gets the same cache for the row
references that field access and element references take (bodies[i].x,
&mut bodies[i]): ElementRefView takes the cached values and computes the
row address inline (issue #739). An immutable shared reference cannot change during the loop. For an
owned list or owner, an exclusive reference, or a retargetable binding, the
frontend checks the finished loop’s operations against a list that cannot
resize or replace any list (arithmetic, control flow, loan bookkeeping, ?,
scalar row access, row references, length queries, view construction, calls, and no write to
the binding). A direct call is allowed when none of its arguments can resize
the list: Plenty has no globals, so a callee reaches a list only through its
arguments. Shared borrows and &mut passed to a slice view cannot resize it,
nor can &mut to an unrelated owned local. The loop falls back for a &mut
list or class argument, including a method receiver, that reaches the cached
list or its owner or that is borrowed from a reference parameter; for a
by-value argument that holds references; for indirect and closure calls; and
for any other operation. The fallback rewrites those accesses to ListScalarGetRef,
ListScalarSetRef and ElementRef, which read the header at each access. The cache is held in
ordinary locals rather than in loads marked readonly: Cranelift treats a
readonly load as valid for the whole function, which a later append would
violate.
The length of a borrowed list of any element type is read from the header
inline: the stored byte count divided by the constant row size, a shift when
it is a power of two, minus the popped-row flag. The length of an owned
temporary uses plenty_list_len, and indexing an owned temporary scalar list or iterating one by value uses
plenty_list_scalar_get, bypassing the argument block of a
runtime operation. These helpers take a live owner
pointer and an i64 index. Named receivers are borrowed without temporary
retain/release; temporary owners are released after obtaining the result.
Text and bytes queries that return a scalar (len, byte_len, isascii,
indexing, ord) call one narrow helper each, such as plenty_text_len or
plenty_bytes_item; a bytes literal index calls plenty_byte_str_item.
plenty_text_ord returns the code point, or u64::MAX when the text is not
one scalar, and codegen builds the Result inline. Results, bounds traps, and
sites match the generic operations’. A query of a borrowed place reads through
the reference. A query of an owned local or full-expression temporary reads the
value in its slot: codegen removes the retain the load added and skips the
matching release, since nothing runs between the load and the helper call.
startswith, endswith, find, rfind, and count call
plenty_text_starts_with and its siblings with both texts in registers;
plenty_text_find and plenty_text_rfind return u64::MAX for no match, and
codegen builds the Option inline. These searches, in, ==, <, and the
runtime operations whose operands are all texts or scalars (replace,
upper, string concatenation, and so on) read in place each text operand that
its source still holds: a copy loaded from a local or read through a
reference, or a string literal. Codegen tracks these copies across the
operand pushes that run no code and release nothing (loads, literals,
references, last-use moves, and stack swaps); it removes each copy’s retain and
skips its release. Any other op, including a conditional or call that computes
an operand, ends the tracking.
Runtime operations that only read their operands read every such copy in
place, whatever its type: dictionary get, get_mut, and indexing; in on a
list, dictionary, set, range, or text; list find, rfind, and count; set
subset, superset, and disjoint tests; and Bytes reads, writes, and scans that
do not resize the buffer. In counts.get_mut(&key) the key is not counted, nor
is a dictionary read through a shared reference, and a Bytes method’s read of
self._owner counts nothing. These operations run no Plenty code (keys hash
and compare structurally) and release or replace no stored value. Operations
that can, such as pop, discard, and inserts, keep the copy’s count, because
a stored value they release may be the copy’s source.
Inserts store their operands without adjusting counts (issue #762). Every
operand codegen passes is owned, whether its read moved it or counted a copy,
so insert, add, append, and a get_or_insert miss make that count the
collection’s, and a failed insert releases it once. A get_or_insert hit
keeps the stored entry: the runtime releases affine operands, and codegen
tests a hit flag in the result and releases copyable ones inline. A mutation
of a receiver through &mut (counts.insert(k, v) on a parameter or local,
pop, discard, clear, set updates, reverse, sort, and so on) reads the
collection’s owner in place under the exclusive loan, so the receiver is
neither retained nor released. clear, reverse, sort, swap, and the set
updates return unit rather than an alias of the receiver, so they make no
count change either (issue #813). Literals, comprehensions, and collect()
do the same: each element moves into the insert from a hidden local, and the
collection under construction is loaded from its owner directly before the
insert, so a counted element costs no retain or release (issue #826).
intersection_update and difference_update also read their set argument in
place when its source still holds it (issue #825). That source can be a &set
parameter, a local, or a temporary parked in an expression local because its
keys have __del__. The argument stays under its shared loan, and a removed
member’s __del__ reaches only that member, since keys cannot hold
collections. The receiver must be a local, a &mut parameter, or a field of
one. An element receiver such as grid[0] still counts the argument (issue
#831).
An element reference (&xs[i], &mut xs[i], &mut grid[i][j], and the
implicit one under bodies[i].x) of any element type checks the index inline
like scalar access and computes the row’s address, rows + index * row_bytes,
where row_bytes is the element’s constant row size (issue #739). An index
out of bounds calls plenty_trap_list_index with the same message as other
list indexing. That address is the reference; reads and writes through it are direct loads and stores. Dictionary
element references still use a runtime operation.
A borrowed list loop (for x in &xs, for x in &mut xs, reversed(&xs), or a
loop over a list reference) iterates through the reference without copying the
list’s owner. Before the first iteration it reads the length and the
first row’s address from the header and uses the element’s constant row size
as the distance between rows. Each element reference is
then rows + index * row_bytes, the slot a generic element reference names, so
a scalar read is one load with no call. The loop holds the list’s loan from its
first condition to its last body use; no mutation can grow, shrink, or move the
rows meanwhile, which is what makes the cached length and address valid. Element
values keep reference semantics: writes through &mut land in the list, and an
element can be passed on or returned under the list’s loan.
std.bytes element reads and writes and Bytes.len use plenty_bytes_item,
plenty_bytes_set_item, and plenty_bytes_len. Bytes.push calls
plenty_bytes_push with its storage owner read through the exclusive field
reference, so appending a byte neither retains nor releases the owner and
reaches the allocator only when the buffer grows. Text len, byte_len,
isascii, scalar indexing, and the forward and reverse iteration cursors use
plenty_text_* helpers. A text or byte owner reached through a place (a local,
parameter, field, or row) is read through a shared reference under the
caller’s loan, so the query neither retains nor releases it; other receivers
are released after the call. A scalar text result is always inline. Indexing
traps report the same site and message as the generic path.
Managed reads and some nested projections retain the generic ownership path.
String length is cached, UTF-8 scalar indexing scans boundaries, and text iteration
uses a byte cursor. Compiler-generated builder/iterator locals count toward the
256-slot limit. examples/collection_bench.py and examples/collection-bench.md
provide reproducible workloads and measured costs.
Generator frames and closure environments
Each generator producer/specialization has a concrete frame layout, resume
function, and immutable descriptor. A frame owns parameters and locals in typed
slots, plus slot metadata, continuation state, resume callback, and reentrancy guard.
Its internal resume ABI is (frame, out_slot) -> ready; yields transfer owned
values, completion clears slots, and destruction releases remaining captures.
Ordinary slots use 16 bytes; inline payloads add their complete storage. Standard sums reserve the largest possible payload, and frames can contain other frames. Constructors and returning functions use caller-owned output storage. Moving nested frames repairs addresses; cleanup completes before surrounding storage expires.
Layout caches are per concrete producer. Recursive inline layouts are rejected, nesting is limited to 64 levels, native offsets fit signed 32-bit sizes, and consumer specialization is limited to 256 per compilation. Unresolved factory returns are rechecked with inferred context so all paths agree on concrete storage.
Yield requires an empty residual operand stack. Lowering adds resume-dispatch
edges to continuation blocks; all locals are frame-backed, so SSA values need
not survive invocations. Dispatch currently uses a linear comparison chain.
There is no interpreter, native-stack suspension, or eager yield collection.
Iteration wraps resume results in inline Option values.
Closures
Capture environments are concrete inline layouts with nominal producer identities. Consumers specialize for their actual layout; matching signatures do not erase identity or introduce boxing. Owned factories return into caller storage. Nested environment moves repair addresses, including environments stored in generator frames or collection rows.
Reusable invocation borrows an environment; consuming invocation transfers it. Borrowed capture loans remain source-level facts through environment moves and generic consumers. Captures are released in reverse capture order; an environment whose captures need no cleanup, such as a capture-less error mapper, is not retained or released at all. Cached facts and visited-type traversal avoid repeatedly expanding shared capture/type graphs. Nesting is limited to 64 levels and storage must fit the native stack-layout limit. The callable contract defines permitted captures, storage, and invocation; the generator contract defines suspension and lifetime restrictions.
Arithmetic boundaries and callbacks
Every named function and closure has an independent declared return contract.
The frontend’s CheckedFunction operation gives arithmetic failures a shared
exit through that public Result; it does not wrap successful source returns.
An explicit checked: expression instead constructs its own local Result.
unchecked: lowers its lexical suite’s operators with trap guards, which
validate its no-failure precondition. Nested
function definitions start their own recoverable boundary in either mode.
See function arithmetic contracts.
There is no private callback error channel, CheckedUnwrap operation, or
checked-block lifetime restriction. Ordinary calls, generic callback consumers,
and substituted pipeline callbacks preserve the callback’s declared Result
and its normal ownership rules. Consumers must handle fallible callbacks
explicitly; a caller’s arithmetic mode never changes a callee’s ABI.
Serial integer pipeline sum() also has a public overflow Result, rather
than inheriting the consumer’s local mode.
Runtime key sorting uses one caller-owned first-error cell per invocation. A failed key evaluation saves its error, releases earlier keys, and returns an equal comparison. Later comparisons skip user callbacks. The sorter finishes its ownership-safe row moves, then its caller forwards the saved error. This preserves its bounded sorting complexity and allocation behavior without unwinding across Rust frames; a failure does not roll back row order.
Tail-call ABI
Native Plenty functions use Cranelift’s Tail convention. Validation and lowering
share a checked representation plan. Approved transfers emit return_call or
return_call_indirect; disagreement at lowering is an internal compiler error.
Unsupported transfers become ordinary calls before lowering. Direct source-tail
calls in a recursion cycle instead fail checking when no safe plan exists; see
the function contract.
| Representation | Native transfer |
|---|---|
| Scalar/heap-owner arguments and exact-compatible result | Supported |
| Exact-compatible inline result, including ranges, records, sums, concrete frames | Supported through incoming result storage |
| Owned inline argument, including range, record, closure, or frame | Unsupported: its bytes must outlive frame removal |
| Reference argument | Supported only when ForwardsReferences proves every passed reference originates in a reference parameter |
Indirect Callable call | Same result, owned-input, and reference checks |
| Conversion, wrapping, propagation, or incompatible result area | Ordinary call; remaining work follows return |
Concrete closure invocation uses its environment adapter and does not perform a native tail transfer. ABI capabilities do not expand allowed callable signatures.
Storage and cleanup
An inline result uses caller-supplied storage. The plan checks native scalar representations, payload capacity, alignment, and component offsets; exact source types ensure compatible nested relocation metadata. A transfer forwards the incoming hidden pointer unchanged. Final return relocates the result there before local cleanup. Source code cannot borrow this hidden result area.
Arguments are staged before caller-owned locals are released. Different argument
counts, registers, and stack argument sizes are handled by the calling convention.
Owned inline arguments currently pass addresses into caller storage. Cranelift’s
StructArgument can copy bytes but cannot relocate Plenty’s nested addresses, so
a raw copy cannot safely remove the frame. No hidden heap allocation extends it.
Ordinary fallback keeps the frame alive and uses staged snapshots; unborrowed
locals still receive the source-defined pre-call cleanup.
Compile cache
Every compile used to lower every function a program reaches, including all
of an imported std module, from IR to machine code. Most of that code is the
same from one program to the next, and Plenty’s objects are reproducible
(native emission): the same input, compiler, and target give
the same bytes. The compiler therefore keeps a local, content-addressed cache
of compiled functions and reuses an entry whenever a function it is compiling
has one.
Reusing functions still runs the frontend and builds every function’s IR: with every function cached, these were 45% of a warm build of a large program (issue #304). The cache therefore also keeps each program’s object, which a build whose sources have not changed uses without running the frontend or code generation. Single-file runs also keep linked executables, avoiding repeated linking.
What is cached
The unit is one compiled function: its machine code, alignment, and relocations, as Cranelift serializes them. Generic instantiations are functions of their own, so they are cached like any other.
Cranelift compiles a function’s stencil: the IR without the identities of the external symbols it names. The IR refers to a callee, a type descriptor, or a string literal by an index into a small table belonging to the function; the table is applied to the compiled code afterwards, and the placing thread resolves the function’s requested symbols in the object. A cached entry holds the code for the stencil, with relocations against those indices. On a hit the compiler still builds the function’s IR, so the table and the requested symbols are this program’s, and applies them to the cached code. Two programs whose functions have the same stencil but print different types, call different functions, or sit at different source positions therefore share one entry and each get their own symbols.
The key
An entry’s name is a SHA-256 of:
- Cranelift’s key for the function: the stencil, the target triple, and every ISA flag, which includes the optimization level, the register allocator (chosen per function, see emission), the verifier setting, and the frame and probe settings;
- the cache format version;
- the compiler’s version and Cranelift’s version.
The compiled code for a stencil depends on nothing else: the compiler’s own work ends with the IR, which the key covers, and what it does with the result (rewriting GOT loads of local data, see symbol references, then placing bytes and relocations in the object) is the same whether they came from Cranelift or the cache. Entries hold Cranelift’s output before that rewrite. The compiler’s build identity, such as a Git hash, is deliberately not part of the key: it would empty the cache at every commit without excluding any stale entry. A change to Plenty’s code generation changes the IR it builds, and so the key.
Every damaged or stale entry is a miss, never an error: an entry carries a checksum of its payload, and Cranelift rejects entries written by another version of itself.
Plenty buffers the key’s Hash byte stream before feeding it to SHA-256.
The byte stream and serialized entries remain compatible with the pinned
Cranelift version (0.137.0); unit tests compare buffered keys to Cranelift’s
keys and exercise entry exchange in both directions. Dependency upgrades
must retain these checks: the upstream key implementation is not a stable
serialization format.
Within one compilation, each distinct key has a lazily initialized memory entry. Concurrent requests for that key share its disk read, decoding, or compilation; unrelated keys can initialize in parallel. The memory entry keeps only machine bytes, alignment, and unresolved relocations. Relocations are resolved using each current function’s symbol table and function ID; source locations and other parameterized Cranelift metadata are not reused. Cranelift’s optional cache-check mode continues through its original path, including recompilation and comparison of full metadata.
This saves duplicate decoding at the cost of retaining all distinct machine code and relocation buffers until the compilation’s cache is dropped. There is no separate memory eviction limit. In-memory reuse counts as a hit, but adds no disk bytes read; the reported time saved is still an estimate.
Program objects
A build of an application (plenty FILE, --compile, --emit-object,
plenty build, plenty run, and plenty test) looks up its program before
loading any module. An entry holds the object the build wrote and the inputs
it read. Library builds, --check, and legacy-syntax programs do not use
program entries.
The entry’s name is a SHA-256 of:
- the cache format version, the compiler’s and Cranelift’s versions, and the compiler’s identity: its executable’s path, size and modification time;
- a digest of every standard module’s text, which the compiler embeds;
- the target, the optimization level, and whether the build has
plenty testhooks or fuzzing coverage, and the executable reachability policy (Windows temporary objects omit unreachable user helpers; inspectable objects retain them); - the request: the canonical entry file and the package graph (each package’s
canonical root, identity, library and dependencies), and the
mainthatplenty testappends; or, for a source compiled from memory, its text.
Unlike a function’s stencil, the frontend’s output depends on the compiler’s
whole code, so the key names the compiler binary rather than its version. A
rebuilt compiler has a new modification time, so every build after a compiler
change, including a change to std, misses. The binary is not hashed: that
would cost about as much as the warm build it saves.
The name does not cover the imported modules, which a build finds only by
parsing. The entry records them instead: each source file read, with a
SHA-256 of its text, the package whose root it resolves against, and where
each of its imports resolved. A file read by
include_str or include_bytes
is recorded the same way, with no imports. A lookup uses the entry only if every recorded
file still has that text and every recorded import still resolves to the same
file or standard module. Resolving an import again repeats every existence
check the loader made, so a new file that would make an import ambiguous, or
change which file it names, is noticed though the program never read it. The
package checks of a build, such as reserved names and overlapping roots, run
again too. Anything else (a missing, damaged, or undecodable entry, an edited
file, a changed resolution) is a miss: the build runs and its object replaces
the entry. Each request therefore keeps one entry, its latest build’s.
The first build of a request stores only an empty entry, a marker that it was built; a later build of it stores its object. So the third build of an unchanged program is the first hit, and after an edit, the first rebuild stores and the next one hits. A program built only once, such as each temporary program of the test suite, costs a few bytes rather than its object. Without markers, one run of the test suite stored 3,870 objects (1.3 GB, beside 194 MB of compiled functions); with them it stores 438 objects and 2,925 markers (96 MB).
plenty script records its successful object on the
first build, so its first unchanged rerun can skip both compilation and linking.
The frontend and code generation are a function of these inputs: no phase reads the environment, the clock, or any file other than the modules it loads, and the object is the same for any number of cores. A program hit therefore writes the bytes a build would. Only a successful build stores an entry, so a program hit never hides a diagnostic. Linking still runs on an object hit unless a single-file run also hits the executable cache.
Every edit to a module the program reads misses, including a change to a function body that no other module’s code depends on: the frontend checks a program’s modules together (see frontend), so there is no smaller unit to keep. See later phases.
--no-program-cache (CompileOptions::program_cache = false) keeps
reusing compiled functions but neither uses nor records program entries.
compile_bench uses it, so its builds measure the frontend and code
generation. With --cache-stats, the line begins program hit or
program miss when the build consulted the program cache; a hit reports no
function hits or misses.
Single-file executables
plenty FILE and plentyc FILE, including Unix hashbang launches such as
#!/usr/bin/env plenty, cache their linked executable by default.
plenty script, including #!/usr/bin/env -S plenty script, uses the same
executable reuse contract. No extra
flag is needed. The first successful link stores an entry; an unchanged run
can reuse it immediately. The object/function caches still run before this
lookup, so this avoids linking, not the object cache’s initial warm-up.
The executable key covers the object bytes, compiler/runtime identity, working
directory, complete environment, effective link arguments, and the selected
C driver and linker’s identities. The linker writes a dependency file naming
its native inputs, including startup objects, libraries and linker scripts.
Entries record those inputs’ resolved paths, sizes, timestamps and file
identities, plus library/tool search directories (including absent ones),
GCC specs and collect2. Replacing a native input or adding a library to a
watched search directory invalidates the entry. This uses filesystem metadata,
not content hashing of system libraries; it is not a defense against deliberate
filesystem tampering. Environment changes conservatively miss even when they
would only affect the running program.
A hit verifies the checksummed entry, validates native inputs, and writes a
private executable with owner-only execute permissions in the run’s temporary
directory. It does not execute the shared cache file: concurrent replacement,
eviction, or a cache filesystem mounted noexec cannot disrupt a running
program. Publication uses the cache’s temporary-file/rename protocol. Failed
links never publish an entry; missing, damaged or unwritable entries fall back
to linking. The program always runs again; output and runtime data are not
cached. Streams, working directory, environment and exit status are unchanged.
The bundled runtime archive is extracted only when linking runs, after this lookup misses or is bypassed. Its bytes are already covered by the compiler identity, so a hit needs no temporary runtime archive. The object and private executable are still restored, and toolchain/native-input checks still run. See the warm startup measurements for the attributed cost and reproducible comparison (issue #699).
Executable reuse currently supports Linux’s default GCC-based cc driver
(a native ELF binary whose resolved name identifies GCC) with mold, lld or
GNU ld when the selected linker supports --dependency-file. Before looking
up an executable, Plenty asks GCC to resolve the selected linker and requires
it to be a native ELF binary. It also rejects non-native private linker
candidates and collect2 helpers in GCC’s search directories. A shell wrapper
cannot serve cache hits merely by forwarding --help or --version: its
external configuration and delegated tools are not depfile inputs (issue #676).
Clang, wrapper drivers and wrapper linkers bypass executable reuse and link
normally because their implicit inputs are not tracked.
Explicit drivers/link arguments, non-Linux hosts, test-hook and
fuzz builds, and redirected/injected toolchains (COMPILER_PATH,
GCC_EXEC_PREFIX, LIBRARY_PATH, LD_PRELOAD, LD_AUDIT) retain ordinary
linking. Explicit --compile outputs and project plenty run do not use this
layer. --no-cache, PLENTY_CACHE=off and --no-program-cache disable it.
--cache-stats adds an executable cache: hit or miss line when consulted;
--verbose prints a link command only when linking actually runs.
Location and controls
| Setting | Effect |
|---|---|
PLENTY_CACHE_DIR=DIR | Use DIR. The default is $XDG_CACHE_HOME/plenty, else ~/.cache/plenty |
PLENTY_CACHE=off, --no-cache | Compile without the cache; the flag applies to one compiler command |
--no-program-cache | Reuse compiled functions, but not program objects or executables |
PLENTY_CACHE_LIMIT=SIZE | Keep at most SIZE bytes (512M, 2G, or a byte count); the default is 1 GiB |
--cache-stats | Print whether the program hit, function hits and misses, bytes read and written, and an estimate of the time saved |
plenty cache stats | Print the directory, its entry count (functions, programs and executables) and size, and the limit |
plenty cache clean | Remove every entry |
The same directory also holds the automatic linker probe’s results under
DIR/linker-probe/ (Native linking); they are not
cache entries, and plenty cache clean leaves them.
Function entries live under DIR/fn/, program entries under
DIR/program/, and executable entries under DIR/executable/, sharded by the first byte of the hash. Each is written to a
temporary name in its shard and renamed, so compilers running at once in
several processes and threads share one directory without a lock and never
read a partial entry. Two of them missing the same function write the same
bytes; two building one program at once each write a complete entry, and the
later rename wins.
A compile that wrote entries trims the directory to the limit when no sweep
ran in the last hour (DIR/sweep records the last one), removing the least
recently used entries of any kind first, by the later of each file’s
access and modification time. Linux updates access times lazily, so recency
is coarse, and no hit writes anything. A program entry with an object is
about the size of that object (1.7 MB for a program that imports
std.http.server).
If the directory cannot be created or written, the compiler prints one warning and compiles without the cache for the rest of the process; a read-only directory still serves hits. A program entry that cannot be written is skipped without a warning, so a read-only directory that serves every function does not warn on each build. The cache changes no diagnostic: code generation never reaches a program with errors, and only successful builds store programs.
The test suite compiles thousands of programs; the workspace’s
.cargo/config.toml points PLENTY_CACHE_DIR at target/plenty-cache, so
every test run of a worktree shares one cache and none touches the user’s
~/.cache. Tests that exercise the cache use directories of their own.
Trust
The cache holds machine code that the compiler will link into programs. A local directory is trusted like the compiler binary itself: anyone who can write it can already replace the compiler. The checksum detects damage, not tampering. Sharing a cache between users, or over a network, needs the trust model described below, which this phase does not implement.
Later phases
These are design notes, not implemented behavior.
The bounded module-cache investigation records current edit profiles, a scalar native-artifact spike, and a runnable counterexample to signature-only invalidation. It changes no cache behavior.
Per-module artifacts. Program objects make a no-change rebuild nearly
free, but any edit runs the whole frontend again, std modules included. A
per-module cache would keep, for each module:
- its interface: the resolved public declarations other modules can see (function signatures, class, enum and protocol definitions with their type parameters and layouts, aliases, exports, and access rules), including any fact that lowering infers from a body and a caller relies on, such as the fields a function returns;
- its body: the lowered, checked operations of its non-generic functions, with its string literals as a table of its own, and the generic functions and data it defines, in the resolved form that instantiation needs;
- the generic instances it requests, with their type and value arguments.
Its key would be a SHA-256 of the module’s text, the compiler’s identity and settings, and the interface hashes of its direct imports. A body edit then rebuilds only the edited module; dependents rebuild only when an interface they import changes. An instance of a generic depends on the generic’s body, not just its interface, so an instance would be keyed by the defining module’s body hash as well. Native call-site substitution also creates body dependencies that invalidation must cover. A program build would combine the modules’ bodies and instances in a fixed order, deduplicating instances and renumbering strings, so the object stays byte-identical to an uncached build.
The frontend cannot produce these yet: it checks all modules as one program. Type resolution creates generic data instances in a table shared by every module, lowering interns strings and records instances and returned fields in tables shared by every body, and signatures are collected into one whole-program table (see frontend). The operations and types also have no serialized form; nominal types are graphs with weak edges. The work is: give lowering a per-module view built from imported interfaces, make instance and string tables per module and merge them afterwards, serialize operations and types, and keep program-order diagnostics.
No smaller artifact pays for itself (issue #317). Loading a stored tree allocates every node, as building it does, so a copy of a phase’s result bounds what loading it costs:
| Phase output (release compiler, system allocator) | Building it | Copying it |
|---|---|---|
| Syntax trees of the 1,000-unit program | 63 ms to parse | 40 ms |
| Operations of the 1,000-unit program (7,001 functions) | 109 ms to lower | 15 ms |
Operations of a program importing std.http.server (605 functions) | 19 ms to lower | 0.5 ms |
Stored syntax trees would save at most a third of parsing, about 5% of a
warm build after an edit. Name resolution rewrites those trees in place and
costs less than a copy of them (20 ms for the 1,000 units split into ten
modules). Only lowered operations are much cheaper to load than to build, so
a module artifact must hold them, and that needs the lowering work above. A
program that is one module, such as compile_bench’s, would need the same
artifacts per function.
For std, lowering only the functions a program reaches would cut work in
every build, cold or warm: in a program that imports std.http.server and
prints, the calls in its operations reach 64 of the 222 std functions
(drop hooks and thread, executor and sort adapters, not counted, reach more),
and in examples/http_server_bench.plenty 155 of 230.
Native emission now omits unreachable standard definitions after checking (issue #400; see emission). This reduces object size and native work without a cache redesign. It does not implement selective frontend lowering or module artifacts. The module-edit probe checks unchanged, body, interface and generic-body edits against uncached objects and provides fresh-process time and peak-memory measurements for further work.
Remote and shared cache. The same keys over HTTP to a content-addressed store would serve CI and teams. It must be optional and opt-in, readable by default and written only by trusted builders such as CI, perhaps through the Bazel Remote Execution API’s cache protocol or a minimal GET/PUT. Hashes are not a trust boundary: an entry is machine code that will run. Trust only configured servers, and verify entries by recompiling a sample in CI.
Measurements
Release compiler, eight cores, warm page cache (issue #144):
| Program | No cache | Cold cache | Warm cache |
|---|---|---|---|
import std.http.server and print (576 functions) | 0.14 s wall, 0.67 s CPU, 65 MB | 0.15 s, 0.75 s, 68 MB | 0.06 s, 0.14 s, 24 MB |
compile_bench 1,000 units (7,002 functions, 16 MB of entries) | 0.86 s wall, 4.6 s CPU, 102 MB | 0.80 s, 4.1 s, 103 MB | 0.47 s, 1.5 s, 94 MB |
A one-line edit to the 1,000-unit program misses one function. A cold run already has hits: functions of one program with the same stencil share an entry.
A program hit (issue #317; plenty --compile, release compiler, function
cache warm in both cases, paired runs on a loaded host):
| Program | Function cache only | Program hit |
|---|---|---|
compile_bench 1,000 units as a file | 3.59 G instructions, 1.72 s CPU, 0.46 s wall | 0.22 G, 0.24 s, 0.08 s |
import std.http.server and print | 0.75 G, 0.40 s, 0.13 s | 0.19 G, 0.23 s, 0.07 s |
What remains on a hit is mostly linking. A build after an edit misses; it also hashes the sources and writes the entry, about 1% more instructions for the 1,000-unit program.
A build that misses its program entry, as after an edit, runs the whole
frontend even with every function cached (--emit-object, profile build,
eight cores):
| Program | Build | Parsing | Name resolution | Lowering | Object emission |
|---|---|---|---|---|---|
| 1,000 units in ten modules | 267 ms | 41 ms | 20 ms | 103 ms | 90 ms |
import std.http.server and print | 51 ms | 9 ms | 3 ms | 17 ms | 19 ms |
With plenty --compile, these builds take 3.16 G and 0.62 G instructions,
and 3.08 G for the 1,000 units as one module.
Inspecting generated code
plenty --dump STAGES FILE -o DIR writes what the compiler produced for each
function as plain text, for reading, grep, and diff. It compiles the
program as --emit-object would, with no object written and nothing linked.
plenty --dump all app.plenty -o dump/
plenty --fast-compile --dump clif-opt,asm --dump-filter parse app.plenty -o dump/
plenty --dump asm --dump-std app.plenty -o - | less
| Option | Effect |
|---|---|
STAGES | all, or a comma list of ops, clif, clif-opt, asm |
-o DIR | Write index.txt and one STAGE.txt per stage into DIR, creating it; stage files from an earlier dump that this one does not write are removed |
-o - | Print the files to standard output, each after a #### NAME line |
--fast-compile | Dump unoptimized code; the default dump uses the optimized profile |
--dump-std | Include retained standard-library functions; unreachable standard definitions are omitted before native emission. By default only the program’s own modules are dumped |
--dump-filter TEXT | Dump only functions whose name or symbol contains TEXT |
--module-root DIR | Resolve imports as for the other file commands |
Stages
| Stage | Contents |
|---|---|
ops | The typed operation IR the frontend hands to code generation, after call-site substitution: each local slot’s type, then one operation per line, with nested bodies indented |
clif | Cranelift IR as lowered from the operations, before Cranelift’s passes |
clif-opt | Cranelift IR after its passes, as compiled to machine code |
asm | Final machine code with byte offsets: x86-64 AT&T syntax or AArch64 instructions |
With --fast-compile, Cranelift runs at opt_level=none, so clif-opt shows
legalized rather than optimized IR; the native emission page
describes what optimization does. Operations are printed with the compiler’s
own debug form and change with it; they are for comparison, not parsing.
Files
Every stage file holds one section per function, in the order the compiler places functions in the object, which does not depend on the worker count:
For example, an x86-64 dump of a function with explicit unchecked:
arithmetic can contain:
==== twice symbol=__plenty_fn_twice kind=function source=app.plenty:1
0: push %rbp
1: mov %rsp,%rbp
4: mov %rdi,%rax
# line 2
7: imulq 0x20
e: jo 0x0000000000000019
14: mov %rbp,%rsp
17: pop %rbp
18: ret
19: ud2 # plenty_trap_overflow
1b: .byte 0x00,0x00,0x00,0x00,0x00,0x28,0x00,0x00,0x00,0x00,0x00,0x00,0x00
The header names the function as source shows it (Point.length,
first[i64], main.<closure@26:11> for a closure at line 26, column 11 of
main), its object symbol, its kind, and where it is defined. Kinds are
function, drop and resume (a class’s destructor and a generator’s
resume function, keyed NAME@drop and NAME@resume), entry
(plenty_main), and adapter (thread, executor and sort trampolines). Keys are unique; a repeated name gets
a #N suffix.
clifandclif-optmark each change of source line with; line N, and name each external function and data symbol on its declaration line.- A check’s trap in an executable names the runtime helper its
trap table entry reaches:
trap user4 ; plenty_trap_div_zeroin CLIF,ud2 # plenty_trap_overflowin x86asm, or an AArch64udfwith the same helper comment. asmmarks source lines with# line N, names the symbol each relocated operand refers to, and shows the constants Cranelift places after the code as.byterows. x86 RIP-relative addresses are offsets within the function. AArch64 offsets advance by four bytes per instruction, and PC-relative operands retain the decoder’s displacement notation. Unknown ARM encodings appear as.instwords, preserving their bytes. A function that shares another’s code shows; shares the code of SYMBOLinstead.- Data symbols carry what they hold: a string literal’s text, a type’s
metadata, or the object’s trap-site table, with the function’s record count
and its offset into the table (
.Ldata1 (trap sites: 1)+44).
index.txt starts with the compiler version, target, optimization level,
stages, scope, and filter, then lists one row per dumped function, largest
machine code first:
# bytes ops clif opt kind lib source symbol key
56 6 7 7 function user app.plenty:1 __plenty_fn_twice twice
ops counts operations, including nested ones, and clif and opt count
Cranelift instructions before and after its passes; adapters have no
operations and show -.
lib is user or std.
Guarantees and limits
- The output is deterministic: the same compiler and source produce the same text for any worker count and cache state. Source files appear relative to the module root, so the checkout’s location does not matter.
--dumpbypasses the compile cache and the program cache: every function is compiled and nothing is stored. Source line markers are part of the function’s cache key, so ordinary builds never carry them; they do not change the machine code.- Line markers carry a line, not a column. Code Cranelift moves or merges keeps the line of the operation it came from.
- Offsets and data symbol numbers shift when unrelated code changes, which adds
noise to a
diffofasmbetween revisions; diffclif-optor compare the index first. - Dumps use the compiler’s native target. ARM bytes are decoded as A64 words, including zero-valued UDF traps; they never pass through the x86 decoder. The platform contract applies to compilation generally.
The compiler phase profiler measures
where compile time goes; --dump shows what that time produced.
Runtime packaging
plenty-runtime is a Rust workspace crate using the standard library. Its
vendored dependencies include the RustCrypto crates behind
std.crypto, zlib-rs behind
std.compress, rustls and its ring provider
behind std.tls, SQLite’s C amalgamation behind
std.db.sqlite, and siphasher for
collection hashing.
Its private C ABI accepts fixed-width scalars and pointers to typed
storage. Generated code supplies valid initialized fields, bounded immutable
metadata, and checked ownership transfers; Rust owns buffers and narrowly scoped
unsafe operations handle raw storage, packed pointers, counts, and callbacks.
The compiler build invokes the selected rustc directly for Cargo’s target,
avoiding recursive Cargo invocation and build-lock contention. It builds optimized
ThinLTO, abort-on-panic static archives and embeds them with native link requirements.
Program compilation extracts the archive and links through a cc driver. Runtime
source and the Rust toolchain are not needed by an installed compiler.
Direct native dependency builds use the pinned cc crate through
plenty-runtime/native/cc_build.rs for target compiler and archive selection.
SQLite and Brotli use native C options; ring selects its upstream assembly for
the target OS and architecture, including pregenerated COFF objects on Windows
x86-64. The GNU archive-symbol parser and panic-symbol redirects are Linux-only.
The standalone build-tool and source-selection tests do not by themselves
establish a working native compiler port.
Standard library Plenty sources are embedded in the compiler as well. std.math
is parsed and checked as an ordinary module, with a stable toolchain identity;
it is never loaded from the user’s source root. Its algorithms and error policy
remain Plenty code, while numeric methods supply IEEE/platform primitives.
See the standard library and the math contract.
The application archive supplies C main, calling plenty_main. A separate
library archive omits both startup and that dependency; shared-library export
maps hide runtime symbols. Both archives are built with the compiler.
Native emission validates the native target triple,
Cranelift’s ISA and pointer type, and the packaged runtime target. Current layouts
depend on 64-bit pointers, 128-bit slots, little-endian metadata, the native C ABI
and fixed descriptor offsets. Cranelift names both Linux C conventions SystemV;
on AArch64 this implements AAPCS64, including aligned u128 register pairs.
Ordinary Plenty functions keep the private Tail convention, with native adapters
at foreign boundaries. Layout assertions and native tests protect these
boundaries; selecting a linker does not validate a new platform.
Target identities
The compiler’s private src/target.rs module owns the admitted target table:
triple, architecture, OS, object format, pointer width, byte order and CPU
baseline. Frontend target facts and binary compatibility checks consume that
description. Tool discovery and command execution remain in toolchain;
target facts do not own compiler state or runtime OS services.
Cargo’s HOST runs the build script; Cargo’s TARGET identifies the compiler
executable’s host. Those may differ when cross-building the compiler. Native
Plenty compilation requires the compiler host, packaged runtime and requested
program target to match. That check preserves cross-building the compiler
without offering cross-compilation through an installed Plenty compiler.
The target table includes both GNU/Linux targets plus experimental
aarch64-apple-darwin and x86_64-pc-windows-msvc paths. Experimental emission
uses AppleAarch64 or WindowsFastcall at C boundaries, Mach-O or COFF objects,
and native linker syntax. Mach-O interface records have aligned length frames
so combined metadata sections retain each library. These paths are being
qualified in native CI; they are not new release guarantees. See
platform support.
Runtime OS boundaries
The runtime keeps its compiler-facing ABI independent of native handles.
threads stores the native thread identity in one opaque word and delegates
creation and joining to Unix pthread or Windows HANDLE implementations.
Cancellation tokens keep their wake resource behind the native module; Linux
uses eventfd. Other systems retain the existing unavailable-readiness-wake
result, while ordinary token waiting uses the shared condition variable.
The generated mutex fast path remains the same atomic state protocol. Its contended wait and wake operations use Linux futexes, Windows address waits, or public pthread primitives on macOS. The macOS implementation shares static parking storage and broadcasts on wake, avoiding private Darwin ulock APIs and per-mutex allocations. Unrelated contended mutexes can wake together; performance under heavy contention still needs native measurement.
Signal register access is selected by OS and architecture. Darwin’s machine context is a pointer to a separate record; Linux’s is inline. The POSIX trap and stack-overflow handlers use native errno access and signal conventions. macOS stack exhaustion can arrive as SIGBUS or SIGSEGV, and its alternate stack reservation accounts for Darwin’s larger minimum. Existing host handlers retain their signals. Windows uses vectored exception handlers and a per-thread stack guarantee. Only registered checked-trap addresses are redirected; unrelated exceptions continue through the host chain. Library threads do not install these handlers.
Resident-library loading also has native backends. POSIX uses libc’s native loader flags; Windows uses UTF-16 paths, native symbol lookup and module pinning. The Windows byte-path boundary accepts UTF-8 and rejects invalid byte sequences; it does not yet represent filenames containing unpaired UTF-16 surrogates. Closing a lookup lease preserves the resident-code contract.
These boundaries and experimental output paths do not establish macOS or Windows standard-library parity or release support. The native qualification workflow tests synchronization separately so failures elsewhere in the runtime do not prevent exercising these primitives. Native Windows exception subprocess tests pass in CI. Object and linker support, actor recovery, OS services and end-to-end Plenty execution remain qualification requirements in the portability plan.
Entry symbols
A standard module’s native code is reachable only through its own entry
symbols. Generated code calls each runtime operation N through its own
exported function plenty_op_<N>(arguments, descriptor, site); no dispatcher
shared between modules takes an operation number. --gc-sections keeps only the
entries a program calls and the code they reach, so a program that never
calls std.net, std.process, std.crypto, std.compress, std.tls or
std.db.sqlite links none of their runtime code. A dispatcher shared by
several modules would keep every module it reaches (issue #501). hello world
calls two entries. BUDGETS in tests/test_executable_size.rs bounds the
size of hello world and an empty program, and NATIVE_MODULES there checks
this for every module with native code; a new module adds a row there (see
size budgets).
The argument block holds three 16-byte inputs at offsets 0, 16 and 32, the result at 48, and further operands at 64 and 80. The compiler always initializes the first three input words, using zero for omitted inputs. It initializes offsets 64 and 80 only for actual fourth and fifth operands; the result and unused scratch words may be uninitialized. Runtime handlers must not read those spare words, even if an optimizer would discard the value.
The compiler and runtime share the scratch-operand count in
plenty-runtime/src/collection_opcodes.rs. Code generation checks its actual
native operand count against that contract, including hidden operands such as
a column field index. The runtime reads only the declared scratch inputs and
supplies zero for unused elements of its local operand array. Adding a native
operation with more than three operands requires updating this shared contract.
No extra caller stores initialize spare words. Each entry inlines its constant
operation number and handler, so the operand count and handler selection fold
away rather than adding a runtime arity lookup.
plenty_control, plenty_executor and plenty_channel keep an operation
number. Each serves one facility (cancellation tokens, executors and channels
respectively), so a program that does not use it does not reach it.
Narrow helpers such as plenty_list_len and plenty_text_find take their operands in registers;
see narrow native helpers.
Boxed callable owners reuse the heap box prefix: a destruction header, concrete
content descriptor at byte 16, shared-call adapter address at byte 24, and typed
content slot at byte 32. Runtime operation 128 consumes an environment,
allocates its box, and returns Result; failure drops the concrete content.
Erased descriptors use kind J for managed ownership and opaque formatting, while
the heap prefix retains the concrete box descriptor for relocation and cleanup.
Protocol objects reuse this prefix with a static method-table address at byte
24 and descriptor kind j. Each table entry is an adapter receiving a reference
to the same concrete payload slot. Mutable signatures preserve exclusive loans
in typed IR. Symbol requests intern tables by their ordered adapters, with native
function relocations resolved in deterministic emission order. Reachability
includes every adapter and its concrete method. Concrete descriptors retain
relocation and destructor behavior independently of the erased public type.
Stack overflow
An executable exhausting its stack reports error: stack overflow on standard
error and aborts with SIGABRT (shell status 134). This applies to its main,
scoped-worker, and executor-worker threads. It does not unwind destructors or
context exits. Libraries leave signal handling to their host and register no
alternate signal stack on threads they create.
Executable startup installs a SIGSEGV handler unless one already exists. Each
thread reserves an alternate signal stack in its own entry frame: 16 KiB on
x86-64 and 64 KiB on AArch64, which has larger saved register contexts. It removes
the stack when Plenty code returns, with no heap allocation or mapping. Startup
checks the runtime page size and kernel minimum signal-stack requirement before
installing the handler; an inadequate reserved stack leaves the default signal
behavior in place. ARM qualification currently covers 4 KiB pages.
Generated frames larger than a page probe each page. The handler recognizes a
data fault below the entry frame and no more than one page below the faulting
stack pointer. AArch64 loop probes leave SP unchanged, so the compiler also
registers their exact store addresses in the read-only trap table. At those
addresses only, the handler checks the negative probe offset in x16 against the
frame size in x17 and uses the resulting probe address as the stack frontier.
Other faults retain the default action. Foreign code skipping stack pages is
outside this detection. Stack sizes remain the system’s limits, and core-file
policy follows the system.
Rust panics
A Rust panic in the runtime is a runtime defect, not a Plenty error. An
executable reports it as error: internal runtime panic at FILE:LINE:COLUMN: MESSAGE on standard error and aborts with SIGABRT, without unwinding or a
backtrace. A failed Rust allocation reports error: memory allocation of N bytes failed and aborts, unless an actor handler contains it as a trap or it
happens inside a reserved arena call, which returns an error instead: see
RSA allocation.
Plenty’s own traps and errors are unaffected.
std’s panic handler would link its panic hook and backtrace symbolizer into
every executable, even with abort-on-panic. The application archive’s link
arguments therefore redirect std’s rust_begin_unwind and
__rust_alloc_error_handler to the runtime’s handlers with -Wl,--defsym, and
--gc-sections drops std’s versions. Measured on 2026-10-26, hello world’s
.text is 69 KB with the redirect and 271 KB without it. Libraries keep std’s
handlers, since their host links them.
Trap sites
An operation that can trap passes its runtime helper a pointer to its source position. Positions are sixteen-byte records (line, column, the offset to a length-prefixed file name, and an embedded-standard-library flag) in one read-only table per object. Each function numbers its own records, deduplicated by position and provenance, so its IR is built on any thread with the offsets known; the thread placing the function appends them to the object’s table and carries the function’s base offset in its relocation, and file names follow all the records, each named once. The module loader supplies provenance; a filename that resembles a standard module cannot acquire it. File names are relative to the source root, which keeps objects independent of the source directory. The table is a local symbol with no relocations, so each Plenty library in a program keeps its own.
Inline checks (integer overflow, division by zero, shift counts, float-to-integer
casts, unwrap) branch to a cold trap block; checks of one kind at one
position share the block. In an executable the block is a single x86 ud2 or
AArch64 udf, with its own Cranelift trap code, and the object holds a trap table mapping
each such address to the check’s kind and site record. Its addresses are
32-bit offsets the linker resolves, so a PIE applies no relocations to it
and its pages stay unread until a check fails. plenty_main
registers the table, and startup installs a SIGILL handler. It uses the
alternate signal stack by default; if a custom SIGILL action was already
installed, its SA_ONSTACK and SA_RESTART choices and signal mask are retained.
Unrelated signals invoke that saved action without removing Plenty’s
checked-trap handler. On a kernel-raised SIGILL at a listed check address the
handler resumes the thread in the check’s helper with the site as its
argument, so the report, exit status and thread behaviour match a direct
call. x86 restores a call-compatible stack; AArch64 restores PC, aligned SP and
argument registers for a native C helper entry. Any other
SIGILL goes to the previous action. The table lists only
functions reachable from exported functions and destructors, so linker
garbage collection still removes the rest. A function has 247 trap codes;
later checks, and every check in a library or --fuzz build,
compute the pointer in the cold block and call the helper. Either way the
passing path gains nothing. A check whose
condition the compiler computes from constants and finds false, such as the
divisor’s zero test in x // 10 or the count test in x >> 13, emits no branch
or trap call, in debug and release builds. A check that may fire, or always
fires, keeps both. Collection helpers
that check an index, key, range parameter, or separator take it as an argument:
one address computation per call. An operation entry writes its result at a
fixed offset in the argument block, which frees an argument for the site. Iteration reads and operations that cannot trap pass null. The runtime
reads the record only when the trap fires and formats the report directly to
standard error.
For objects containing embedded std functions, user calls into std save and
restore a thread-local pointer to the nearest user call site, except calls to
std accessors that only read a field, such as NonNegative[i64].get, which
cannot trap or run other code. User code also
saves and restores it around generator advancement, collection operations that
can run callbacks, and cleanup that can run compiled code: a class __del__,
a generator, or a closure, channel or executor, which may own either. Releasing strings,
byte storage, and records, boxes or collections of only those runs the runtime
alone, which neither traps nor reads the context, so those releases skip it
(issue #750). Std code saves and restores it only around
indirect calls and direct calls to functions that may change it: a function
may change the context when it tail-calls std from user code, makes an indirect
tail call, calls such a function in tail position, or is a generator or a
function without Plenty source. The runtime restores the context itself after
each destructor, generator resumption, and sort-key comparison it runs, so no
runtime helper changes it. Direct
user-to-user calls need no context operations. Tail calls transfer the pointer without retaining
the discarded frame. A std trap prints the original site and a caller note;
ordinary user traps ignore this context. Site records are immutable, so the
context needs no allocation. Worker threads start without a caller, and executor
jobs clear the context between jobs. The pointer names a site, not a stack trace.
A null site prints the message alone. Stack overflow, the token front end, and runtime contract failures such as a library object from another loaded instance have no site.
Validation
Native tests and compile-time layout assertions check public helper signatures
and private layouts. Miri exercises runtime storage with exposed-provenance
semantics for packed pointers; it cannot execute generated Cranelift code.
runtime-checks enables allocation counting and failure injection. Allocation-free
markers count successful allocations on the calling thread and keep a separate
baseline for each thread; failed attempts do not count, and failure injection is
also thread-local. Live-heap checkpoints count
bytes across the process, so workers must be idle or joined and any retained
worker storage counts toward the checkpoint’s allowance. All workers and their
results must be cleaned up before the final allocation-balance check. A relocated
compiler test rejects per-program C/Rust runtime compilation.
The runtime contributor guide lists validation commands.
Concurrent runtime mechanisms
Worker eligibility is a closed-world compiler check over finite type graphs, reachable helpers, concrete callbacks, and destructors. Visited identities bound recursive walks. Atomically counted owners can cross certified boundaries; counts alone do not permit mutable sharing. The concurrency contracts define allowed effects, owned jobs, blocking, and shutdown.
Each native primitive (CollectionOp) reports an effect from one exhaustive
match in src/effects.rs; the certification walk rejects a worker that can
reach a denied effect and names the call. Primitives without a native contract,
such as foreign calls, are rejected separately. See
native effects on worker threads.
Tasks and queues
Scoped tasks pin argument/result storage in the parent’s frame. Generated C adapters invoke typed workers and publish results there. POSIX join must complete before reclaiming storage; unexpected join failure terminates the process to avoid a worker using departed storage. Task bookkeeping does not allocate a Plenty owner; native thread stacks and OS bookkeeping still use resources.
Channels allocate a pinned core/ring whose slots reserve complete message layouts.
Endpoint values reference counted headers. One mutex protects predicates and
disconnect flags; separate condition variables serve readers and writers.
Disconnect wakes all waiters. Receive relocates values before slot reuse, and
user destructors run outside locks. nowait can still contend on a mutex.
Executors keep pinned worker records and a bounded ring. Partial startup failure joins started workers. Each accepted job has one fallibly allocated cell for its concrete environment and result; queue/worker and future owners use separate atomic counts. Queue and job state have separate locks. User callbacks/destructors run outside them; pool destruction drains and joins.
Cancellation and selection
Tokens combine an atomic cancellation flag with a mutex/condition-variable handshake for waiter registration. Timed waits track a monotonic elapsed budget, so wakeups do not restart it and huge millisecond values need no overflowing absolute deadline.
Receive selection registers pinned stack waiter nodes in participating queues and uses a per-call condition variable. Queue-to-waiter lock ordering synchronizes notification; selection transfers a value under its queue lock and removes all registrations on every return. It needs no global selection lock or per-wait heap allocation.
Atomics
std.atomic operations compile inline, with no runtime call. A method borrows
its receiver’s place, so a local atomic lives in the function’s frame and every
thread addresses the same slot. The payload is the slot’s low word, naturally
aligned because slots are at least 8-byte aligned. Each access is a Cranelift
atomic instruction (atomic_load, atomic_store, atomic_rmw, atomic_cas,
fence) of the element’s exact width, never a plain load the optimizer could
merge. Checked fetch_add/fetch_sub loop on atomic_cas and trap before
storing. Cranelift’s atomics are all sequentially consistent, so each requested
Ordering is validated at compile time and then strengthened. On x86-64 a
load is a mov, a store and a fence add mfence, and read-modify-write
operations use lock-prefixed instructions or a lock cmpxchg loop.
Mutexes
std.sync.Mutex[T] is the generic class in stdlib/std/sync.plenty: an
Atomic[u32] state word and the value, stored inline, so moves, destruction,
and worker eligibility are ordinary class behavior. Its fields are private;
lock() and into_inner() are compiler intrinsics. A lock block stores the
mutex reference in a hidden guard local and roots the value’s exclusive loan
there, while the mutex’s own shared loan stays live until the block’s exit.
The ownership checker treats the guard as live from the lock to the unlock and
rejects a join reached while a live guard’s T matches a Mutex[T] the
worker’s inputs can reach. Each pool submission records the Mutex[T] types
its job input can reach, beside the sender message types, in the function’s
submission list and in the future’s local provenance; result() and each
shutdown run the same guard scan against them. Reachability walks through
Shared[...] contents.
Shared owners
Shared[T] reuses the Box[T] runtime record and its type metadata kind: a
counted header, the content type, and a 32-byte-aligned content slot. The
frontend emits BoxOp::Share (the box allocation entry point) and
BoxOp::SharedRef, a shared-only projection to the content slot; it never
emits the box’s move-out or mutable projection for a handle. share() is the
ordinary retain of a loaded owner. Construction rejects contents that reach an
executor or future, or that reach the same Shared[T] through a Mutex or a
channel endpoint.
Locking follows mutex 2 of Drepper’s “Futexes Are Tricky”: state 0 is
unlocked, 1 locked, and 2 locked with possible waiters. Lock is an inline
atomic_cas from 0 to 1. On failure, plenty_mutex_lock_slow exchanges in 2
and, until the exchange returns 0, sleeps with FUTEX_WAIT_PRIVATE on value 2;
EINTR and EAGAIN retry the exchange. Unlock is an inline atomic_rmw
exchange to 0, and calls plenty_mutex_wake (FUTEX_WAKE_PRIVATE of one
waiter) only when it replaced 2. The runtime issues the futex system call
directly. Nothing allocates; a held lock is never abandoned by unwinding,
because traps terminate the process.
Native interfaces and metadata
Trusted interface declarations lower to host C calls, keeping Rust collection and enum layouts behind private runtime helpers. Scalar output adapters use call-scoped native storage; UTF-8 adapters lend bytes and length, and C-string adapters create terminated temporary buffers. Owned exports use native boxes with generated matching destruction. The interop contract defines what foreign code must preserve; these adapters cannot enforce foreign behavior.
Generated export interfaces and headers describe names, ABI types, ownership, and error conventions. Library artifacts embed interface metadata and a SHA-256 compatibility symbol. Bounded compile-time extraction reads metadata without loading the binary; explicit verification compares its declared contract.
Typed runtime loaders cache resolved symbol addresses and check exact metadata
and object-instance identity. Runtime primitives use eager local GNU/Linux
loading and RTLD_NODELETE. Closing a lookup lease releases lookup ownership
while resolved code/static state remain resident until process exit. Dynamic
loading executes library constructors; it is not sandboxed metadata inspection.
Loader-owned memory and constructors are outside Plenty’s allocator accounting.
Support index
The reference describes the supported subset, not Python’s full API. Detailed restrictions live with their contracts.
| Area | Contract and principal limits |
|---|---|
| Source and modules | Declarations, imports, visibility, and entrypoints; compile-time constant expressions, static assertions, and included files; if const target conditionals over a closed set of target facts; one source root per package, no import cycles or re-exports |
| Packages | plenty.toml, path and Git dependencies pinned in plenty.lock, version requirements, workspaces, plenty add/fetch/update, per-target build reuse, optimized/fast-compile profiles, plenty test with per-process test functions run in parallel under optional timeouts, and plentyc --package-graph; no registry or version selection |
| Types and data | Types, including byte literals of type bytes; fixed layouts and recursive data through heap boundaries; read-only std.bytes operations and stream writes take a &[u8] view of bytes, Bytes or list[u8] without copying |
| Expressions | Operators, assignment, and control flow; blocks and expressions nest to 64 levels; f-strings with any expression as a field and compile-time Python format specs, whose nested fields supply only fill, alignment, width, and precision; copying slices xs[a:b:c]? of lists, text, bytes, tuples, and clists; local slice views &xs[a:b] of lists and bytes, typed &[T], and of text, typed &substr, with &xs, &s, and literals converting to the whole view where a view is expected; no slice assignment |
| Ownership | Moves, checked borrows, returned references including Option[&T] and Result[&T, E], and cleanup; general stored references unsupported |
| Functions | Named functions and concrete closures; named arguments and fresh per-call defaults for static calls, with callable/builtin/protocol restrictions; explicit fallible shared-call erasure |
| Generics | Concrete specialization and structural protocols, including integer/bool value parameters and Bounded, and requirements with their own protocol-constrained type parameters; methods of a generic class or enum instance are checked and compiled only when used; explicit boxed protocol dispatch with shared/mutable receivers |
| Dimensions | Compile-time dimensions: nominal bases, normalized algebra, exact roots and Dimension generic arguments; Quantity construction, numeric/domain operators, conversions, coherent-unit formatting, module constants and scoped alias diagnostic notes. Storage-type native ABI with release instruction evidence |
| Errors and resources | Sums, propagation, allocation, and context scopes; traps do not unwind; explicit std.error result mapping and contexts |
| Iteration | Loops, comprehensions, and native generators; local serial pipelines with borrowed sources, fused adapters including chain, flat_map, and callback-free filter_value, reductions, next(p), lending as &mut Iter[T], and parallel pipelines on a pool with .par(&pool) over owned or borrowed lists and ranges; no reference yield or public borrow across suspension |
| Built-ins | Numeric, text, and collection operations; explicitly fallible allocating operations; columnar lists for primitive-field classes with borrowed scalar pipelines |
| Standard library | Embedded std.math, std.path, std.fs, std.deque, and std.bytes; typed errors, Linux paths/metadata and walking, owned double-ended queues, and compact byte buffers. std.time adds checked durations, monotonic deadlines, and wall timestamps. std.io adds partial-progress byte streams, retry helpers, and fixed-capacity buffering over files, process streams, and memory. std.encoding adds strict hex/base64 codecs and explicit-endian integer packing. std.url adds strict RFC 3986 URL parsing, component-aware percent encoding, and query pairs. std.collections adds binary heaps. std.complex adds f64 complex numbers with arithmetic operators. std.cache.lru adds fixed-capacity least-recently-used caches. std.csv adds bounded streaming CSV records and writers. std.random adds caller-owned seeded generators and std.secrets OS-backed secret bytes and tokens. std.crypto adds digests, HMAC, BLAKE3 hashing/key derivation, constant-time comparison, and AEAD, Ed25519, X25519, and HKDF. std.compress adds bounded gzip, zlib, and DEFLATE streams. std.archive adds TAR and ZIP reading, writing, and confined extraction. std.regex adds linear-time regular expressions with explicit resource limits. std.db.sqlite adds SQLite databases through typed prepared statements and owned transactions. std.json adds strict, bounded JSON: a resumable event decoder, a validating encoder, and a value tree. std.log adds leveled structured events with explicit, bounded sinks. std.toml adds strict, bounded TOML 1.0 parsing and writing over a value tree. std.env reads variables and the working directory, and std.process spawns owned children from argument vectors without a shell, with pipes, bounded output collection, signals, and explicit reaping. std.ready adds readiness waits on one handle or many (selectors) under a deadline and a cancellation token that wakes them directly. std.actor runs supervised actors over std.ready on x86-64 and AArch64 GNU/Linux, each with a limited heap that a trap discards, bounded mailboxes of copied messages, timers, request-reply and opt-in restart. std.signal delivers SIGTERM and SIGINT as an event, so a server can drain when its supervisor stops it. std.net adds TCP and UDP sockets, IP address parsing, and cancellable name lookup under the same deadlines. std.tls adds verified TLS client and server streams over those sockets, with client certificates, ALPN, SNI certificate selection, revocation lists, and selector-driven use. std.http adds bounded HTTP/1.1 message framing with streamed bodies and strict rejection of ambiguous framing. std.http.server serves it, and HTTPS through std.tls, with typed routes, bounded workers and admission, idle connections parked on a selector, per-phase deadlines, and graceful shutdown; its qualification lists the production gates it passes. std.http.client sends requests over pooled connections with deadlines, bounded redirects, an explicit proxy, and streamed uploads, shares one pool between threads through Shared, and std.http.https adds verified https with CONNECT tunnels. std.http.body, std.http.cookie and std.http.proxy add bounded forms, JSON and multipart uploads, validated cookies, and trusted-proxy client addresses for handlers. std.http.files serves static files confined to a root directory; cookies can be signed. std.test adds assertions that report their caller’s position, fixtures, fake clocks and streams, and allocation and file failure injection for plenty test. std.diff adds Myers and patience differences over lists, bytes, and text tokens, with unified-diff output |
| I/O | Text/byte streams, raw Unix paths, and owned UTF-8 files; direct file iteration unsupported |
| Concurrency | Threads, channels, pools, futures, and parallel operations, with atomics and mutexes shared through scoped-thread borrows or atomically counted Shared[T] owners; certified worker effects, serial ordinary loops; no weak handles |
| Interoperability | Trusted C imports, exports, metadata, and runtime loading |
| Editor support | Language server with diagnostics, hover, go to definition, and completion, configured for Neovim; generic bodies resolve per instantiation |
| Native toolchain | Commands and artifacts, and per-function IR and assembly dumps; native x86_64-unknown-linux-gnu and aarch64-unknown-linux-gnu, linking with the matching host cc; release requirements and ARM qualification distinguish implementation from tested deployment environments |
The ARM path uses generic ARMv8-A instructions, scalar std.simd, Linux ELF,
and the AAPCS64 C ABI. It is a release platform: the release workflow builds the
compiler and both optional engines in the glibc 2.28 image on native ARM
runners, runs the full suite there, and smoke-tests the archives in AlmaLinux 8
and Debian 13. Kernels older than the one tested, page sizes other than 4 KiB
and other distributions are not tested; see the
ARM development notes. x86-64 retains
its SSE2 baseline. Selecting --target or a linker does not turn an installed
compiler into a cross-compiler.
Native emission for aarch64-apple-darwin and x86_64-pc-windows-msvc is
experimental. It includes target ABI, Mach-O/COFF objects and native linker
invocation. Release packaging includes the dataframe dylib on macOS ARM64 and
the dataframe DLL on Windows x86-64, with their metadata and notices. The linear
algebra engine remains Linux-only. Native release/portability workflows check
Parquet queries and application distributions after relocation. Native CI and the
release workflow pass compiler/runtime builds and unit tests, platform-boundary
checks, and compilation and execution of a generated callback/result program on
both operating systems. Runtime and standard-library capabilities remain
incomplete, including actor recovery and several OS services, and these checks
do not establish standard-library parity; see
experimental macOS and Windows downloads.
There is no interpreter, REPL, JIT, async/await, inheritance, dynamic attributes, or exception unwinding. Future work and priorities remain in the backlog.
Runtime contributor guide
plenty-runtime is the dependency-free Rust workspace crate behind generated
native code. The book’s compiler/runtime architecture pages describe packaging,
representations, memory, and synchronization. This guide records boundary
obligations and validation for runtime contributors.
Boundary invariants
- Helpers use the host C calling convention with fixed-width scalars and typed
storage pointers. Rust
Vecand enum layouts never cross that ABI. - Generated code supplies initialized fields, bounded immutable descriptors, valid pointer lifetimes, and checked ownership transfers. Internal zero slots mean moved or uninitialized storage, never source-level nullable references.
- Borrowing helpers return owned results. Consuming helpers explicitly transfer messages, jobs, or results; count changes alone do not confer mutable access.
- Moves of inline payloads repair their embedded addresses. Inline frame cleanup completes before owner-provided storage expires. User callbacks and destructors run outside synchronization locks and outstanding mutable runtime views.
- Allocation uses checked layouts and guards for partial construction. Buffers retain matching allocation/deallocation provenance. Traps and Rust panics terminate rather than unwind through generated frames.
- Layout assertions and native integration tests protect compiler/runtime agreement. An internal layout is not a public foreign ABI.
Validation
cargo test --workspace --all-features
cargo clippy --workspace --all-targets --all-features -- -D warnings
MIRIFLAGS=-Zmiri-permissive-provenance cargo +nightly miri test -p plenty-runtime
cargo test -p plenty-runtime tests the crate independently. Its Cargo library
omits the process entrypoint; embedded application builds enable it separately.
runtime-checks enables allocation counting and failure injection in the
compiler’s archive. Allocation-free regions count every successful allocation,
zeroed allocation, and reallocation on the calling thread, including temporary
allocations already freed. Failed attempts, injected or real, obtain no memory
and do not count, so failure injection can run inside a region.
Each thread has its own region baseline, so workers can use overlapping regions.
Failure injection is also local to the calling thread and covers allocation,
zeroed allocation, and reallocation without allocating inside the instrumentation.
Native tests check cleanup, retained source state, and retries after failure.
Normal compiler builds omit these checkpoints. The runtime crate’s
allocation-checks feature enables its counting allocator and gated ABI unit
tests. --all-features selects both; workspace --features runtime-checks alone
does not enable the runtime crate’s feature.
The __test_small_live_heap__ marker and final allocation-balance check count
live bytes across the process, including storage transferred between threads.
At a live-heap checkpoint, workers must be idle or joined and any worker-owned
heap storage must fit within the 4096-byte allowance. Join all workers and drop
their results before the final allocation-balance check.
Miri checks layouts, metadata, ownership, relocation, and reentrant destruction. Packed pointer slots require exposed-provenance semantics; this does not establish strict-provenance correctness. Native tests remain necessary because Miri cannot execute Cranelift-generated machine code.
For instrumented archives, PLENTY_RUNTIME_RUSTFLAGS supplies whitespace-separated
rustc flags. Compiler RUSTFLAGS are not copied implicitly. A nightly address
sanitizer build can use -Zsanitizer=address -Clto=off with an ASan-capable linker
wrapper. Record the flags and tested native boundary with validation results.
Comparisons
These pages compare Plenty with other languages, one page per language. They describe implemented Plenty behavior and link to the reference pages that define it. Plans and rejected alternatives are in the backlog and proposals.
- Rust: ownership, borrowing and what
unsafepermits.
Rust
Plenty’s ownership model is Rust-influenced: values have one owner, borrows are checked, and destruction is deterministic. See ownership, borrowing, and lifetimes. This page records where the two languages differ.
Unsafe code
Plenty has no unsafe blocks. Rust’s unsafe permits five operations that
safe Rust rejects; Plenty code can perform none of them, except calling C
through trusted declarations.
unsafe Rust permits | Plenty |
|---|---|
Dereferencing raw pointers (*const T, *mut T), with pointer arithmetic, null or dangling pointers, and aliased &mut | Not permitted. * dereferences only checked references (&T, &mut T). The only raw pointers are opaque foreign handles, which cannot be dereferenced, converted to integers or offset. |
Calling unsafe functions: C functions, transmute, get_unchecked, manual allocation, uninitialized memory | C functions are declared in .plentyi modules, whose author vouches for the C side. Bit reinterpretation is limited to four std-only integer/float intrinsics used by std.encoding. The other operations do not exist. |
| Reading and writing mutable statics | Not permitted. There are no global variables; module scope holds only constants. |
Implementing unsafe traits such as Send and Sync | Not permitted. Worker eligibility is a closed-world compiler check of stored types, reachable bodies and native effects; code cannot assert it. |
Reading union fields | No unions. Enums and anonymous sums are tagged and checked when matched. |
Rust’s inline assembly (asm!) also requires unsafe. Plenty has no inline
assembly.
Where unchecked code lives
Code that Rust would write inside unsafe lives outside Plenty source:
- trusted
.plentyideclarations, where a wrong declaration or a faulty C library can violate memory safety; - the Rust runtime, whose
scoped
unsafecode handles raw storage, packed pointers, counts and callbacks; - compiler intrinsics used by the standard library.
A defect in one of these, or in the compiler, can corrupt memory. Plenty code cannot.
Consequences
- Buffer overruns. List, text, range and
Bytesindexing is bounds-checked, and an out-of-range index stops the program withindex out of bounds: index I, length N(traps). No option disables these checks. Slices clamp their bounds instead of trapping. Integer overflow traps outsidechecked:, so an overflowed length cannot size a buffer. - References. A reference is created from existing storage, such as a variable, a list element or a slice, and cannot outlive it. Creating one from an index goes through the bounds check.
- Limits. A custom allocator, an intrusive linked list, an unchecked fast path or memory-mapped device access cannot be written in Plenty. Write it in C or Rust and call it through C interoperability, or add it to the runtime.
Supplementary examples
These examples complement the core learning path. Use them to look up a particular operation once you understand ownership, borrowing, and recoverable errors.
See collections and iteration, owned files, and text stream positions for the corresponding contracts and limits.
The syntax cheat sheet summarizes everyday syntax on one page and lists the Python habits Plenty rejects.
The finder experiment tests these facilities against a realistic CLI task and records where the implemented surface still blocks it.
Linked data structures compares owned chains and trees with arena IDs, including fallible insertion and allocation-free rotation.
Syntax cheat sheet
A compact summary of everyday Plenty, for readers who already know Python. Every program on this page runs as a test, so the forms shown are current. Each section links to the reference, which remains authoritative. The tutorial teaches the same material in order, and the signature index lists the standard library.
Builds optimize by default. Use plenty --fast-compile FILE,
plenty script --fast-compile FILE, or plenty run --fast-compile for shorter
edit-run cycles in large projects; runtime checks stay enabled. Package output
is in target/optimized/ by default or target/fast-compile/ with that flag.
Programs and printing
def main() -> Result[(), AllocError | IoError]:
# print returns Result[(), IoError]; `?` propagates the error.
print("hello", 42)?
# Collection literals allocate, so they return Result[_, AllocError].
names = ["Ada", "Grace"]?
print(len(names), names)?
Ok(())
hello 42
2 ["Ada", "Grace"]
Executable statements live in functions; module scope holds only declarations,
imports, const values, and compile-time assert checks. main returns (), i32,
Result[(), E], or Result[i32, E]. A Result cannot be silently discarded:
use ?, match, or drop(result). See entrypoint,
source files, and branches.
Bindings and numbers
import std.math as math
def main() -> Result[(), IoError | ParseError | OverflowError[i64] | ZeroDivisionError]:
mut count = 0
count += 1
small = 7u8
ratio: f64 = 1.5
total = f64(count) + ratio
print(count, small, total)?
print(7 // 2, -7 // 2, 7 % 2, 7.0 / 2.0)?
base = 2
print(base.pow(10), (3).pow(2), math.abs(-4)?, math.max(3, 9))?
print(i64.MAX, u8(200), i64.parse(" 12 ")?)?
Ok(())
1 7 2.5
3 -4 1 3.5
1024 9 4 9
9223372036854775807 200 12
- Bindings are immutable;
mutdeclares a mutable one, once. - There is no
int: unsuffixed integers arei64, floatsf64. Suffixes:7u8,1.5f32. Integers may be hexadecimal, octal, or binary (0xFFu8,0o755,0b1010) and use_separators (1_000_000). - Types never widen implicitly; cast with
f64(n),u8(n)(traps if out of range),T.try_from(n), orT.wrapping_from(n).T.parse(text)returnsResult[T, ParseError]. //floors and%is integer-only. Integer arithmetic returns errors from the function: includeOverflowError[T]and, for plain divisors,ZeroDivisionErrorin itsResultsignature. Still writeOk(...)on success; operators themselves return ordinary numbers. No**: usex.pow(n).abs,min,max,sumcome fromstd.math.
Use checked: for local recovery instead of returning arithmetic errors from
the function. Its last line is the Ok value:
def scaled(n: i64, d: i64) -> Result[i64, OverflowError[i64] | ZeroDivisionError]:
checked:
n * 1000 // d
def main() -> Result[(), IoError]:
print(scaled(7, 2).unwrap(), scaled(7, 0).is_err(), scaled(i64.MAX, 1).is_err())?
Ok(())
3500 True True
-
Positive[T](> 0) andNonNegative[T](>= 0) accept signed integerT. Literals convert where expected; computed values usePositive.try_from(n)?orNonNegative.try_from(n)?. Read with.get(); widen a Positive withNonZero.from(p)orNonNegative.from(p). See integer domains. -
Bounded[T, lo, hi]accepts the eight integer types and inclusive compile-time bounds. Literals convert where expected; an alias such astype Percent = Bounded[u8, 0, 100]providesPercent.try_from(n)?for runtime inputs. Read.get()for arithmetic. Value parameters useconst N: i64or[T: IntType, const N: T]. -
Each function and closure has its own declared arithmetic contract, independent of its definition or call site. There is no private callback channel. Handle a callee’s
Resultnormally. -
unchecked:asserts that its arithmetic cannot fail and yields an ordinary suite value, not aResult. A violation traps by default and is undefined behavior with--unchecked assume;unchecked trap:always traps. Use wrapping operations for modular arithmetic. Keep opt-outs small and document their bounds. -
?,return,break, andcontinueinside it act on the function or loop. -
Float arithmetic stays IEEE: overflow may produce infinity and division by zero may produce infinity or NaN. For recoverable zero division,
x.div_checked(y)isResult[F, ZeroDivisionError].
See numbers and assignment.
Text
def main() -> Result[(), AllocError | IoError]:
name = "Ada"
greeting = ("Hello, " + name)?
print(greeting)?
print(f"{name} has {len(name)} letters; pi is {f64.PI:.2f}"?)?
words = "a,b,,c".split(",")?
print(words, "-".join(&words)?, " x ".strip()?)?
print("hello".find("l"), "hello".startswith("he"), "ll" in "hello")?
print(name[0], name[-1], "hello"[1:3]?, str.from(42)?)?
Ok(())
Hello, Ada
Ada has 3 letters; pi is 3.14
["a", "b", "", "c"] a-b--c x
Option[i64].Some(2) True True
A a el 42
str is immutable. Operations that build text return
Result[str, AllocError]; ? binds tighter than +, so write (a + b)?.
An f-string has type Result[str, AllocError], or
Result[str, AllocError | FormatError] when a field runs a display or
debug method or a spec is computed. Escapes are \n, \t,
\r, \0, \\, quotes, and \u{H...} ("e\u{301}"); there is no \x or
\N{...}. len counts Unicode scalars;
text.byte_len() counts bytes. A slice
such as name[1:]? or name[::-1]? copies, so it needs ?. See
text,
building text, and
format strings.
Collections
def main() -> Result[(), AllocError | IoError | OverflowError[i64] | ZeroDivisionError]:
mut items = [3, 1, 2]?
items.append(4)?
items.sort()
print(items, items[0], items[-1], items.get(10))?
print(items[1:3]?, items[::-1]?, items.pop(), 2 in items)?
mut counts: dict[str, i64] = {}?
for word in "a b a".split()?:
*counts.get_or_insert(word, 0)? += 1
counts.insert("b", 5)?
match counts.get_mut("a"):
case Some(count):
*count += 10
case Nothing:
pass
print(counts, counts.get("z"))?
for word, count in counts.items():
print(word, *count)?
print(len(counts.keys()), "a" in counts.keys())?
mut seen = {1, 2}?
seen.add(2)?
squares = [n * n for n in range(5) if n % 2 == 0]?
print(len(seen), squares)?
Ok(())
[1, 2, 3, 4] 1 4 Option[i64].Nothing
[2, 3] [4, 3, 2, 1] Option[i64].Some(4) True
{"a": 12, "b": 5} Option[i64].Nothing
a 12
b 5
2 True
2 [0, 4, 16]
Displays, comprehensions, append, insert, and add can fail to allocate,
so they return Result. Empty displays need a type: xs: list[i64] = []?.
Indexing a missing element or key traps; get returns an Option.
Dictionaries have no indexed assignment: insert adds or replaces,
get_mut returns Option[&mut V] to change a value in place, and
get_or_insert(k, default)? returns &mut V after inserting a missing key.
for v in (&mut d).values(): and for k, v in (&mut d).items(): change
every value in place. See
collections,
lists, and
dictionaries.
Tuples
def div_mod(a: i64, b: i64) -> Result[(i64, i64), OverflowError[i64] | ZeroDivisionError]:
Ok((a // b, a % b))
def main() -> Result[(), AllocError | IoError | OverflowError[i64] | ZeroDivisionError]:
pair = (42, "cm")
single: (i64,) = (7,)
print(pair, pair[0], pair[-1], single[0])?
quotient, remainder = div_mod(7, 2)?
print(quotient, remainder, (1, "b") < (1, "c"))?
mut point = (1, 2)
x = &mut point[0]
*x += 10
print(point)?
for n, unit in [(1, "kg"), (2, "m")]?:
print(n, unit)?
Ok(())
(42, "cm") 42 cm 7
3 1 True
(11, 2)
1 kg
2 m
Tuples allocate nothing. Indices are integer literals; negative ones count from
the end, so pair[-1] is pair[1]. There is no pair[0] = value, so assign
through &mut pair[0]. Moving an owned component out needs whole-tuple
unpacking, including nested targets such as (x, y), label = value.
Starred targets are unsupported. match accepts tuple patterns such as
case (Some(a), Some(b)):. See
tuples and
assignment to several targets.
Control flow and matching
def classify(n: i64) -> str:
match n:
case 0:
"zero"
case 1 | 2 | 3:
"small"
case _:
"many"
def main() -> Result[(), IoError | OverflowError[i64] | ZeroDivisionError]:
for i in range(0, 10, 4):
print(i, classify(i))?
mut n = 4
while n > 0:
n -= 1
if n == 2:
continue
elif n == 0:
break
print("n", n)?
parity = "even" if n % 2 == 0 else "odd"
print(parity)?
for index, letter in enumerate("hi"):
print(index, letter)?
Ok(())
0 zero
4 many
8 many
n 3
n 1
even
0 h
1 i
Block if and match can also be the whole value of a binding, a single-place
assignment, or return; see conditional values.
A suite’s final expression is its value, so return is only needed for early
exits. Conditions must be bool; there is no truthiness. Chained comparisons
and loop else are rejected. match works on enums, classes, tuples, integers, str,
and bool, must be exhaustive, and has no guards. Tuple patterns nest within
enum/class payloads and each other. See
matching and
loops.
Functions and closures
def scale(value: i64, factor: i64 = 2) -> Result[i64, OverflowError[i64]]:
Ok(value * factor)
def increment(value: i64) -> Result[i64, OverflowError[i64]]:
Ok(value + 1)
def apply[F: Callable[[i64], Result[i64, OverflowError[i64]]]](f: &F, value: i64) -> Result[i64, OverflowError[i64]]:
f(value)
def main() -> Result[(), IoError | OverflowError[i64]]:
print(scale(5)?, scale(5, factor=10)?)?
offset = 100
shift = def [offset](value: i64) -> Result[i64, OverflowError[i64]]:
Ok(value + offset)
print(apply(&increment, 1)?, apply(&shift, 1)?)?
print(apply(&(def(value: i64) -> Result[i64, OverflowError[i64]]: Ok(value * 3)), 2)?)?
Ok(())
10 50
2 101
6
Every parameter and result has a type. Anonymous functions are
def(params) -> T:; lambda is rejected. Captures are explicit:
def [x, mut y, &z, &mut w](...). Accept callbacks as F: Callable[...]
with &F, which takes named functions and closures; use &mut F for a
closure that changes a mut capture. See
signatures,
function values, and
closures.
Classes and enums
For primitive-field classes, clist[T]() creates allocation-free empty
columnar storage.
items.append(row) returns Result[(), AppendError[T]]; items[i].field
reads a scalar, items.copy_row(i) returns Option[T], and
items.column(T.field) is a borrowed local pipeline yielding scalar values.
Whole-row and packed-field references are unsupported.
@copy
class Point:
x: f64
y: f64
class Counter:
name: str
hits: i64
def bump(self: &mut Counter) -> Result[(), OverflowError[i64]]:
self.hits += 1
Ok(())
def report(self) -> i64:
self.hits
enum Shape:
Dot
Circle(f64)
Rect(f64, f64)
def area(self) -> f64:
match self:
case Shape.Dot:
0.0
case Shape.Circle(radius):
f64.PI * *radius * *radius
case Shape.Rect(width, height):
*width * *height
def unused(counter: &Counter) -> bool:
match counter:
case Counter(_, 0):
True
case Counter(name=_, hits=_):
False
def main() -> Result[(), IoError | OverflowError[i64]]:
mut counter = Counter("clicks", 0)
counter.bump()?
counter.bump()?
print(counter.report(), counter)?
print(Shape.Rect(2.0, 3.0).area(), Shape.Dot)?
origin = Point(0.0, 0.0)
mut shifted = origin
shifted.x = 1.5
print(origin, shifted)?
print(unused(&counter))?
Ok(())
2 Counter(name="clicks", hits=2)
6.0 Shape.Dot
Point(x=0.0, y=0.0) Point(x=1.5, y=0.0)
False
Classes have typed fields and a generated positional constructor; there is no
inheritance. An explicit __init__ returns () for a plain class result or
Result[(), E] for precisely Result[Class, E], with explicit Ok(()) on
success. Construction adds no allocation error; initializer operations and
parameter defaults must declare or map their own errors. Bare self is a shared borrow; mutating methods declare
self: &mut Class. Explicit self: Class transfers an owned receiver into the
method, just like a by-value function parameter; the caller cannot reuse an
affine receiver. @copy before class makes instances copy like integers; every
field must copy, and the class cannot define __del__. A generic @copy class
copies when its type arguments copy, and T: CopyType requires an argument
that copies (copy classes). @newtype on a one-field class forwards the field’s operators between two values of that class (newtype forwarding). A field can move out of an owned binding or receiver, after
which the object cannot be used as a whole until that field is assigned again;
see moving fields.
Variants are always qualified (Shape.Dot), and matching a
borrowed enum binds references, so payloads are dereferenced with *.
case Counter(name, hits=0) takes a class apart, written like its constructor, with
every field named once
(class patterns). See
classes, enums,
and borrowed matching.
Classes and enums can define arithmetic hooks
such as __add__(self, other: &T) -> T, __rmul__, and __neg__.
Operators borrow their operands and dispatch statically. Fallible hooks return
Result: write (a + b)?. In-place hooks such as
__iadd__(self: &mut T, other: &T) -> () mutate an existing destination;
there is no automatic += fallback. Equality remains structural. Other
__name__ methods, such as __eq__, __str__ and __len__, are
rejected:
write display/debug for formatting and named methods for the rest.
Compile-time dimensions
Use dimension for a base or a derived dimension, and D: Dimension for an
argument that carries no runtime value:
dimension Length("m")
dimension Time("s")
dimension Velocity = Length / Time
const metre: Quantity[f64, Length] = Quantity[f64, Length](1.0)
def main() -> Result[(), IoError]:
distance = 12.0 * metre
elapsed = Quantity[f64, Time](4.0)
speed: Quantity[f64, Velocity] = distance / elapsed
print(speed)?
Ok(())
3.0 m/s
Products and quotients normalize; Pow[A, n] and Root[A, n] require integer
exponents and exact roots. One is dimensionless. A dimension cannot be a value
type. Quantity[T, D] checks dimensions in arithmetic and .get() reads its
stored number. For floats, .to(unit) divides by a quantity of the same type.
std.units provides SI dimensions, units such as u.km
and constants; declare your own only for other bases.
See dimensions and
quantities.
Option, Result, and errors
def parse_pair(text: str) -> Result[(i64, i64), AllocError | ParseError]:
parts = text.split(",")?
if len(parts) != 2:
return Err(ParseError.Invalid)
Ok((i64.parse(parts[0])?, i64.parse(parts[1])?))
def first_even(values: &list[i64]) -> Option[i64]:
two: NonZero[i64] = 2
for value in values:
if *value % two == 0:
return Some(*value)
Nothing
def main() -> Result[(), AllocError | IoError | ParseError]:
print(parse_pair("3,4")?)?
match parse_pair("3"):
case Ok(pair):
print(pair)?
case Err(error):
print("bad input:", error)?
values = [1, 3, 6]?
print(first_even(&values), first_even(&values).unwrap_or(0))?
print(first_even(&values).ok_or(ParseError.Invalid)?)?
Ok(())
(3, 4)
bad input: ParseError.Invalid
Option[i64].Some(6) 6
6
There is no None and no exceptions: absence is Option[T]
(Some(x)/Nothing), failure is Result[T, E] (Ok(x)/Err(e)). ?
returns early on Nothing or Err; the function’s error type must equal the
source’s or contain it in an A | B union. ? never converts an Option into
a Result: use ok_or(error)?. To wrap an error, map it explicitly:
print(line).map_error(AppError.Io)?. unwrap() traps. print shows the full
enum name, as in Option[i64].Some(6). See
propagation and
anonymous sums.
Ownership and borrowing
def total(values: &list[i64]) -> Result[i64, OverflowError[i64]]:
mut sum = 0
for value in values:
sum += *value
Ok(sum)
def double_all(values: &mut list[i64]) -> Result[(), OverflowError[i64]]:
for value in values:
*value *= 2
Ok(())
def main() -> Result[(), AllocError | IoError | OverflowError[i64]]:
mut values = [1, 2, 3]?
print(total(&values)?, total(values)?)?
double_all(&mut values)?
backup = copy(values)?
moved = values
print(moved, backup)?
label = "team"
other = label
print(label, other)?
Ok(())
6 6
[2, 4, 6] [2, 4, 6]
team team
Numbers, bool, str, bytes, ranges, and named function values copy. A
tuple, Option, Result, enum, or anonymous sum (A | B) copies when all of
its parts copy. A class copies only when it is declared @copy, and a generic one only when its type arguments
also copy.
Every other value moves: collections, boxes, closures that capture,
generators, and other class instances.
After moved = values, values is unusable.
str assignment keeps both names usable without copy; immutable storage may
be shared. copy(x)? explicitly duplicates eligible owned mutable contents.
An eligible ordinary class does not need @copy for this explicit operation;
nested Shared handles retain the same content.
Custom destruction, recursive data, generators and captured closures restrict
copying. &x lends shared access and
&mut x exclusive access; a &T parameter borrows an owner implicitly, but
&mut must be written. Iterating &list yields &T, so dereference with *.
References cannot be stored in classes, enums, tuples, or collections. See
moves and copies and
borrowing.
Generators
def countdown(start: i64) -> Generator[i64]:
mut n = start
while n > 0:
yield n
# The loop proves 1 <= n <= i64.MAX, so decrement cannot overflow.
unchecked:
n -= 1
def main() -> Result[(), AllocError | IoError]:
for n in countdown(3):
print(n)?
print(list(countdown(4))?, countdown(5).any(def(n: &i64) -> bool: *n == 2))?
Ok(())
3
2
1
[4, 3, 2, 1] True
A function containing yield returns Generator[T]; its body cannot use ?.
Generators have the pipeline methods, such as
any(predicate). See generators.
Local pipelines
Local transformations can use iter(source).map(callback).filter(predicate)
with a loop or a terminal such as fold(initial, callback) or count().
Borrowed sources and callback captures stay borrowed until the pipeline is
consumed. Generators expose the same fluent methods directly. An unfinished
pipeline stays local; .generator() explicitly transfers owned sources and
owned captures into one resumable frame that can be passed or returned. See
local iterator pipelines.
enumerate and zip produce pair items: mapping takes two parameters,
predicates borrow both, and folds take the accumulator followed by the pair’s
components. Collect owned pairs into an expected dict[K, V] with collect(). a.chain(b) continues with a
second source of the same item type; flat_map(f) emits the items of the owned
source f returns for each item. filter_value(v) keeps items equal to v
(True by default) without a callback; borrowed items accept v or &v.
def add_square(total: i64, value: i64) -> Result[i64, OverflowError[i64]]:
Ok(total + value * value)
def add_result(total: i64, value: Result[i64, OverflowError[i64]]) -> Result[i64, OverflowError[i64]]:
Ok(total + value?)
def squares(stop: i64) -> Generator[Result[i64, OverflowError[i64]]]:
iter(range(stop)).map(def(n: i64) -> Result[i64, OverflowError[i64]]: Ok(n * n)).generator()
def main() -> Result[(), IoError | OverflowError[i64]]:
values = iter(range(5))
print(values.try_fold(0, add_square)?)?
print(squares(4).try_fold(0, add_result)?)?
Ok(())
30
14
Python habits that Plenty rejects
| Python | Plenty |
|---|---|
| Statements at module level | Put them in main |
int, float | i64, f64, or another width |
None | Nothing in an Option[T] |
try/except, raise | Return Result; handle with ? or match |
xs = [1, 2] | xs = [1, 2]? |
print(x) as a statement | print(x)? |
"a" + b | ("a" + b)?, or an f-string with ? |
x ** n | x.pow(n) |
range(a, b, step) with a computed step | range(a, b, NonZero.try_from(step)?); a literal step converts |
abs(x), min, max, sum(xs) | Integer std.math.abs(x)?, min, max, sum(&xs, 0)?; float abs_float / sum_float |
if items: | if len(items) > 0: |
a < b < c | a < b and b < c |
xs[1:3], s[1:3] | xs[1:3]?, s[1:3]?: a slice allocates a copy; &xs[1:3] borrows a list view, and &s[1:3] a &substr, in place |
lambda x: x + 1 | def(x: i64) -> Result[i64, OverflowError[i64]]: Ok(x + 1) inside brackets |
map(f, xs), filter(p, xs), reduce(f, xs, init) | iter(&xs).filter(p).map(f), .fold(init, f); no import |
any(flags), all(p(x) for x in xs) | iter(&flags).any(), iter(&xs).all(p) |
d[k] = v | d.insert(k, v)?, which adds or replaces |
d[k] += 1 | *d.get_or_insert(k, 0)? += 1, or *v += 1 on Some(v) from d.get_mut(k) |
case x if x > 0: | Test inside the arm |
class B(A): | Composition or a protocol |
| Global variables | Pass values as parameters; const holds a number, quantity, bool, str, or bytes computed at compile time |
y = xs, then using xs | y = copy(xs)?, or borrow with &xs |
A name first bound inside an if, loop, or match arm, used after it | Bind name = if condition: (or name = match value:), or use a if cond else b; the binding ends with its block. Only if const arms open no scope |
x is None | x.is_none(), or match on Some/Nothing |
"-" * 3 | "-".repeat(3)? |
del xs[i] | xs.pop(i), which returns an Option |
assert x in a function | Return an error: if not x: return Err(...) |
xs.sort(reverse=True) | xs.sort(True) |
Three of the most common rejections:
def main() -> ():
print("hello")
return
cannot implicitly discard a Result
def main() -> ():
x = 2 ** 3
exponentiation with `**` is not supported; use `base.pow(exponent)`
def main() -> Result[(), AllocError | IoError]:
values = [1, 2]?
moved = values
print(values)?
Ok(())
use of moved binding `values`
Run plenty --check FILE to type-check without building.
Finder feasibility experiment
The repository’s examples/finder/ attempts a small fd-style search tool using
the current language. Its README records the assessment, commands, validation,
and remaining limits. main.plenty and matching.plenty implement a working
filter for supplied paths, with literal basename search, */? wildcards,
full-path matching, hidden-component filtering, and CLI errors.
cargo build --offline
target/debug/plenty --compile examples/finder/main.plenty -o /tmp/plenty-finder
printf 'src/main.plenty\nnotes.txt\n' | /tmp/plenty-finder --paths --glob '*.plenty'
python3 -B examples/finder/check.py -v
The program processes one UTF-8 path per line. It borrows strings while matching,
avoids allocating basename slices, and preserves concrete I/O errors through
Result[i32, IoError | OverflowError[i64]]. ? in a glob matches one Unicode scalar. Matching is
case-sensitive; omitted patterns match all nonempty visible paths.
Without --paths, the finder searches [PATTERN [ROOT]], defaulting to ..
It uses the embedded std.fs walker over native enumeration, prunes hidden
directories before descent, and never follows discovered directory links.
Directories, regular files, symlinks and other types are eligible output.
An explicit root can itself resolve through a link. Traversal errors stop with
the attempted path, operation and concrete IoError. Python supplies test fixtures
and an independent traversal oracle, not the application’s candidate paths.
The capability probes now check working directory and metadata APIs.
Use --null for lossless Unix paths: args_bytes() preserves raw patterns,
input_bytes(0u8) reads NUL records, and write_stdout_bytes emits exact names
followed by NUL. CR, LF, and invalid UTF-8 are preserved. Matching borrows
list[u8]; in this mode ? matches one byte, so UTF-8 é takes two ? tokens.
The default text mode retains its Unicode-scalar and line semantics.
Issue #61’s transport acceptance cases are covered by a real Unix fixture with spaces, tabs, CR/LF, and invalid UTF-8, plus raw argument patterns and a byte wildcard oracle. Native traversal preserves those names through Path, joining and filesystem lookup. Broader compact byte buffers and portable path/library APIs remain within issue #49.
See the current standard stream and argument, binary transport, paths, directory enumeration, walking, text, and foreign text adapter contracts.
Choose a representation for linked data
Start with the ownership and operations the application needs:
| Need | Starting representation | Important cost or constraint |
|---|---|---|
| Stack or sequential data | list[T] | Contiguous storage; growth can allocate. |
| Queue with two ends | std.deque | Ring buffer; amortized constant-time insertion and constant-time removal. |
| Tree with uniquely owned children | Classes/enums with boxes or child lists | Subtrees move as owners; boxes allocate per node. |
| Parent/back links, shared targets, or cycles | List owning nodes plus IDs | IDs need validation; removal/reuse needs an identity policy. |
| Frequent rewiring at known nodes | Linked owners or arena links | Finding the node may dominate; measure allocation and locality costs. |
These are starting points, not performance rankings. The current box runtime has a 32-byte header before the payload, in addition to allocator overhead and alignment. Treat this as an implementation detail, not a stable ABI or proof that one representation wins. See memory architecture.
- Push, pop, and walk an owned chain.
- Rotate a tree without allocating new boxes.
- Build balanced search trees: complete B-tree and left-leaning red-black sets, including deletion and invariant checks.
- Retrieve strings with a trie: Unicode keys, prefix retrieval, longest-prefix matching, and pruning deletion.
- Build parent links and an iterative worklist.
Boxes allow direct field access and transfer without explicit dereferences in
many contexts. They do not permit two owners of the same node. Current borrowing
rules also reject stored references and simultaneous mutable loans into distinct
indices of one list. Sequential updates such as nodes[a].next = Some(b) work;
the IDs themselves borrow nothing.
Automatic destruction of recursive owners uses bounded native stack space and does not allocate. Recursive user traversal and destructor hooks do not inherit that guarantee. A descendant reference cursor handles a chain; a non-consuming general tree traversal needs a supported worklist representation. Arena IDs are one option when a list of borrowed nodes is unavailable.
An intrusive collection embeds its links in the application object. It can support multiple memberships and, in pointer-based designs, may need stable addresses and externally managed lifetimes. Arena-owned objects with integer links cover some of these uses, but do not promise stable physical addresses. The design assessment discusses this boundary and the candidate standard-library abstraction.
Push, pop, and walk an owned chain
Each node owns its successor. Insertion allocates before transferring the old head so a failed allocation preserves the caller’s chain. Pop transfers the successor out before destroying the removed node.
pub class Node:
pub value: i64
pub next: Option[Box[Node]]
pub class Linked:
pub head: Option[Box[Node]]
pub def push(xs: &mut Linked, value: i64) -> Result[(), AllocError]:
mut node = Box(Node(value, Nothing))?
node.next = replace(xs.head, Nothing)
xs.head = Some(node)
Ok(())
pub def pop(xs: &mut Linked) -> Option[i64]:
match replace(xs.head, Nothing):
case Some(node):
mut removed: Node = node
xs.head = replace(removed.next, Nothing)
Some(removed.value)
case Nothing:
Nothing
pub def bump(xs: &mut Linked) -> Result[(), OverflowError[i64]]:
match &mut xs.head:
case Some(head):
mut cursor: &mut Node = head
while True:
cursor.value += 1
match &mut cursor.next:
case Some(next):
cursor = next
case Nothing:
break
case Nothing:
pass
Ok(())
from linked import Linked, push, pop, bump
def main() -> Result[(), AllocError | IoError | OverflowError[i64]]:
mut xs = Linked(Nothing)
for value in range(1, 4):
push(&mut xs, value)?
bump(&mut xs)?
print(pop(&mut xs))?
print(pop(&mut xs))?
print(pop(&mut xs))?
print(pop(&mut xs))?
Ok(())
Option[i64].Some(4)
Option[i64].Some(3)
Option[i64].Some(2)
Option[i64].Nothing
cursor advances by borrowing from its current referent. It does not create
stored references or require a recursive call. Pop and this traversal allocate
nothing. bump propagates overflow and retains increments already performed;
it does not roll back the chain. Each successful push allocates one box; its
failure destroys the incoming value, not the existing chain.
This is a teaching module with public links, not a packaged collection that enforces an abstract API. Use a contiguous list for an ordinary stack unless the workload benefits from separate nodes. See representation choices.
Rotate a tree without allocating new boxes
A right rotation moves the left child into the root position and transfers
its right subtree to the old root’s left. Keep both nodes boxed: converting a
box to a Tree value would free that box and require allocation to box it again.
pub class Tree:
pub value: i64
pub left: Option[Box[Tree]]
pub right: Option[Box[Tree]]
pub def rotate_right(root: Box[Tree]) -> Box[Tree]:
mut old = root
match replace(old.left, Nothing):
case Some(left):
mut pivot = left
old.left = replace(pivot.right, Nothing)
pivot.right = Some(old)
pivot
case Nothing:
old
from tree import Tree, rotate_right
def main() -> Result[(), AllocError | IoError]:
left = Box(Tree(1, Nothing, Nothing))?
root = Box(Tree(2, Some(left), Nothing))?
rotated = rotate_right(root)
print(rotated.value)?
match &rotated.right:
case Some(right):
print(right.value)?
case Nothing:
print("no right child")?
Ok(())
1
2
The rotation takes constant time and allocates nothing. If there is no left child, it returns the original owner unchanged. It preserves in-order node order; it does not by itself balance a tree or validate search-tree ordering.
Removing a subtree similarly uses replace(root.left, Nothing). The returned
owner can be reattached or dropped. Parent references cannot be stored in these
nodes; use arena IDs when parent
navigation is required.
Build balanced search trees
A search tree keeps keys ordered while its shape changes. These examples implement two balanced sets directly in Plenty:
- A B-tree with minimum degree 2: each node holds up to three keys and four children, also called a 2–3–4 tree.
- A left-leaning red-black tree: each node holds one key, two child IDs, and a color bit.
Both implement insertion, membership, deletion, length, and sorted traversal.
They store distinct i64 keys, including the full signed range. Duplicate
insertion and missing-key removal return False without changing the tree.
These are educational implementations, not std APIs or benchmark results.
Use a built-in set[T] when ordering is unnecessary.
Run the implementations
From a repository checkout:
plenty book/src/design/examples/data-structures/balanced-trees/main.plenty
plenty book/src/design/examples/data-structures/balanced-trees/checks.plenty
Or save the B-tree module,
red-black module, and
demo in the same directory, then run
plenty main.plenty. Save checks.plenty there
to run the invariant and model checks as well.
The included source is the actual demo compiled by the integration tests:
import btree
import red_black
def main() -> Result[(), AllocError | IoError]:
mut b = btree.create()?
mut r = red_black.create()?
for key in [40, 10, 70, 20, 60, 30, 50]?:
drop(btree.insert(&mut b, key)?)
drop(red_black.insert(&mut r, key)?)
print(btree.ordered(&b)?)?
print(red_black.ordered(&r)?)?
print(btree.insert(&mut b, 40)?, red_black.insert(&mut r, 40)?)?
print(btree.has(&b, 30), red_black.has(&r, 99))?
print(btree.remove(&mut b, 40), red_black.remove(&mut r, 40))?
print(btree.remove(&mut b, 99), red_black.remove(&mut r, 99))?
print(btree.ordered(&b)?)?
print(red_black.ordered(&r)?)?
print(btree.valid(&b)?, red_black.valid(&r)?)?
Ok(())
Expected output:
[10, 20, 30, 40, 50, 60, 70]
[10, 20, 30, 40, 50, 60, 70]
False False
True False
True True
False False
[10, 20, 30, 50, 60, 70]
[10, 20, 30, 50, 60, 70]
True True
Shared interface and ownership
Both modules expose the same operations:
| Operation | Result | Allocation |
|---|---|---|
create() | Result[Tree, AllocError] | Empty owner creation can fail. |
insert(&mut tree, key) | Result[bool, AllocError] | May grow storage; True means newly inserted. |
has(&tree, key) | bool | None. |
remove(&mut tree, key) | bool | None; True means removed. |
length(&tree) | i64 | None. |
ordered(&tree) | Result[list[i64], AllocError] | Allocates an owned sorted result. |
valid(&tree) | Result[bool, AllocError] | Allocates visitation marks for diagnostics. |
A Tree exclusively owns a private list of nodes. Links are private indices,
not references or owning edges. -1 means no child; it is never used as a
list index by a valid operation. Updating links in separate statements avoids
simultaneous mutable loans into the same list. Neither implementation needs
stored references, parent pointers, unsafe operations, or a foreign library.
Deleted nodes join an internal free list. Reusing them needs no generation counter because node IDs never leave the module. Surviving IDs remain valid when the backing list grows. Storage is retained for reuse rather than shrunk; dropping the tree releases all of it. Neither type is a copyable value.
Structural index/count arithmetic uses small unchecked: regions with bounds
maintained by these private representations, not caller-supplied numeric values.
The B-tree additionally caps its key count at i64.MAX: inserting a new key at
that representational capacity returns AllocError.CapacityOverflow before
allocation or mutation. Duplicate insertion still returns Ok(False) there.
One key per live node bounds the red-black tree’s count by its node-list length.
Validate more than sorted output
checks.plenty compares both trees against a sorted-list model after every
operation and runs each implementation’s structural validator. It checks:
- All 14,400 pairs of insertion and deletion permutations for five keys,
including
i64’s minimum and maximum; repeated insertions, repeated removals, empty trees, and refill after complete deletion. - Four 257-key scenarios using ascending, descending, and alternating-extreme orders, including root growth and contraction, followed by slot reuse.
- 16,000 deterministic mixed operations across four seeds, followed by draining every remaining key.
The validators check strict ordering, key counts, reachability, duplicate or cyclic links, and separation of reachable nodes from the free list. The B-tree also checks occupancy and equal leaf depth; the red-black tree checks its color and black-height invariants.
The Rust integration harness additionally sweeps allocation-failure points before insertion, checks retry after failure, and verifies deletion and reuse with allocation disabled:
cargo test --locked --features runtime-checks --test integration test_data_structure_patterns::balanced_tree
These tests are regression evidence, not a proof over every possible history. The validators assume storage created by the module, not hostile serialized input. Recursive traversal and validation use native stack proportional to tree height; ordinary balanced operations keep that height logarithmic.
Deliberate limits
There are no generic comparators, key/value maps, range cursors, rank/select, persistence, concurrency, disk pages, or externally reusable handles. The B-tree’s tiny fixed degree makes split/merge cases readable; its separately allocated per-node key and child lists are not a cache-tuned storage design. Growing either backing list can move many rows, so insertion is not a strict worst-case logarithmic-time operation in this representation.
See representation choices and the linked-list lesson for the ownership trade-offs behind index-based links.
Split, borrow, and merge in a B-tree
The complete B-tree implementation is a
minimum-degree-2 B-tree, or 2–3–4 tree. Its nonempty root and every other
live node contain one to three keys. An internal node with n keys has n + 1
children; all leaves have the same depth. Unlike a B+ tree, keys can live in
internal nodes, and leaves have no sibling chain.
Use the balanced-tree demo and checks to run it.
Storage
Each node owns two fixed-length lists: three key slots and four child-ID slots.
n selects the live key prefix. Unused key slots may retain old scalar values;
unused child slots in live nodes are -1. A leaf has no child IDs. The root,
size, free-list head, and node storage belong to one private Tree owner.
The free pool retains deleted nodes and their two lists. Taking a slot resets its child IDs; releasing a slot adds it to the pool without moving other nodes. No public operation exposes a node ID.
Insert: split before descending
insert first searches for a duplicate. For a new key, it measures the current
height and prepares enough spare nodes for every possible split on the descent.
A full root needs two slots: a new root and a right sibling. Each remaining
level needs at most one sibling. With height counted in node levels, reserving
height + 1 spare nodes covers the empty tree and both cases.
Preparation is the only fallible mutation phase. It allocates detached empty nodes and links successful allocations into the free pool. Failure can retain extra spare storage, but does not change the root, reachable shape, keys, or size. A retry can reuse those spares. Once preparation succeeds, rewiring, key shifts, and taking spare nodes allocate nothing.
For a full child [a, b, c], split:
- Leaves
[a]in the original node. - Moves
[c]and the appropriate children into a spare sibling. - Moves
binto the parent, shifting parent keys and child IDs to make room.
Splitting a full root first ensures the parent always has room. Splitting each full child before descending means insertion reaches a nonfull leaf, where an in-place key shift creates the final slot.
Delete: do not descend into a one-key node
remove first confirms membership, so a missing-key removal changes nothing.
The internal erase helper therefore assumes it will find its key.
At a leaf, shift keys left and decrement n. At an internal matching key:
- If the left child has at least two keys, replace the key with its predecessor and delete that predecessor from the left subtree.
- Otherwise, if the right child has at least two keys, use its successor.
- Otherwise, merge the two one-key children with the separating key, then delete within the merged three-key node.
When descending toward a key, a one-key child needs help first:
- Borrow from the left: move the parent’s separator into the child’s first slot, move the sibling’s largest key into the parent, and transfer the sibling’s last child if these are internal nodes.
- Borrow from the right: perform the symmetric transfer using the sibling’s smallest key and first child.
- Merge: if neither sibling can lend, join two one-key children around a parent separator. Return the unused sibling slot to the free pool.
This preserves the minimum occupancy without a separate upward underflow pass. If the root becomes empty, its only child becomes the root; deleting the final leaf key makes the tree empty. The old root slot is released in both cases.
Costs and validation
At fixed degree, search and deletion take O(log n) time and deletion allocates
nothing. Insertion performs O(log n) tree work and prepares at most O(log n)
spare nodes; occasional backing-list growth costs O(n), giving amortized
O(log n) insertion rather than a worst-case bound. ordered visits every key
in O(n) time and allocates its result. Recursive deletion/traversal uses
O(log n) native stack space for trees maintained by these operations.
Storage follows the historical peak plus the split reserve, not just the current
key count. valid allocates and scans visitation marks for all retained slots,
so its cost follows retained storage. It verifies occupancy, ordered bounds,
equal leaf depth, active child counts, total size, and free-pool membership.
The test driver exercises both borrow directions, predecessor/successor replacement, merges, root changes, and reuse through permutation, ordered, and mixed histories. These are tests of a teaching implementation, not performance qualification for a database or filesystem.
Acknowledgements
- Cormen, Leiserson, Rivest, and Stein, Introduction to Algorithms: the published B-tree split and top-down borrow/merge algorithms. The Plenty implementation is original; no external source code or data is bundled.
Rotate and recolor a red-black tree
The complete red-black implementation uses left-leaning red-black (LLRB) links. It is a red-black tree with an extra orientation rule, not the parent-pointer variant with symmetric insertion and double-black deletion cases. Use the demo and checks to run it alongside the B-tree.
Invariants
Each node has one i64 key, two child IDs, and a red bit describing its
incoming link. Missing children are black. After each public operation:
- The root is black.
- Keys obey strict binary-search ordering.
- No red link leans right.
- A red node has no red child.
- Every path from a node to a missing child has the same number of black links.
A black node connected to a left red child represents a logical three-node containing two keys. Collapsing these red links gives a balanced 2–3 tree. The degree-2 B-tree instead allows three keys in a node; the two examples are related balanced representations, not identical encodings.
Ownership and rotations
The tree owns a list of private scalar-field nodes. Child links are indices, so a rotation updates existing nodes in place without allocating boxes or storing references. Deleted slots join a private free list and are reset before reuse.
A left rotation promotes the right child, transfers its left subtree to the old root’s right, and makes the old root its left child. The promoted node inherits the old root’s color; the old root becomes red. A right rotation is symmetric. These changes preserve key order but are only part of balancing.
flip toggles a node’s color and both child colors. Unlike setting colors to
constants, toggling supports both splitting and joining logical nodes. Internal
callers ensure both children exist before invoking it.
Insert
After rejecting a duplicate, insert allocates or reuses one detached red node
before changing any tree links or colors. Failed growth therefore preserves
the existing set, shape, and size.
The recursive put links that node at its binary-search position, then calls
balance while returning toward the root. The repair sequence is:
- Rotate left for a red right link.
- Rotate right for two consecutive left red links.
- Flip colors when both children are red.
The shared repair helper also handles deletion’s transient shapes. The public operation finally paints the root black. Once the detached node exists, this phase allocates nothing.
Delete
remove first confirms membership, so its internal descent can assume the key
exists. If both root children are black, it temporarily paints the root red to
permit a color transfer without reducing the final black height.
Before descending into a black child whose left child is also black,
move_red_left or move_red_right transfers a red link toward that child.
Color flips merge logical nodes temporarily; rotations can borrow from a
sibling. These are top-down preparations, not an upward double-black repair.
When deleting a matching internal key, replace it with the smallest key in its
right subtree, then use delete_min to remove that successor node. A matching
terminal node is released to the free list. balance repairs links on return,
and the remaining root is painted black. Deletion and all its rotations and
color changes allocate nothing.
Costs and validation
Balancing bounds height by O(log n). Membership and removal take O(log n)
time; recursive update and traversal use O(log n) native stack space.
Insertion performs logarithmic tree work, but growing the backing list can
move O(n) rows, so the representation provides amortized O(log n) insertion,
not a strict worst-case bound. Sorted collection visits O(n) keys and
allocates an owned list.
Free slots retain storage up to the historical peak. valid allocates and
scans visitation marks across that retained storage. It independently checks
ordering, black root, no right red link, no consecutive red links, equal black
height, size, reachability, and free-list separation.
See the model and invariant tests for permutations, mixed operations, allocation failures, and allocation-free removal and reuse. This implementation is intentionally a set of integer keys, without generic comparators, ranks, or external node handles.
Acknowledgements
- Robert Sedgewick and Kevin Wayne, Algorithms: Balanced Search Trees: the published left-leaning representation, rotations, color flips, and top-down deletion algorithm. The Plenty implementation is original; no external source code or data is bundled.
Retrieve strings with a trie
A trie, or retrieval tree, shares common prefixes between keys. Each edge
here represents one Unicode scalar. The path c → a → r represents "car";
a terminal marker distinguishes that stored key from a prefix that merely leads
to "cart". The permanent root represents the empty string.
This example implements a set of strings directly in Plenty, including exact
lookup, insertion, pruning deletion, prefix enumeration, and longest-prefix
matching. It uses private node IDs rather than stored references. It is an
educational implementation, not a new std API.
Run the example
From a repository checkout:
plenty book/src/design/examples/data-structures/trie/main.plenty
plenty book/src/design/examples/data-structures/trie/checks.plenty
Or save the implementation, demo, and
test driver in the same directory, then run
plenty main.plenty or plenty checks.plenty. These are the canonical source
files compiled by the Rust regression tests. The demo below includes its source
rather than maintaining another copy:
import trie
def main() -> Result[(), AllocError | IoError]:
mut words = trie.create()?
for key in ["", "car", "cart", "cat", "dog", "café", "cafe\u{301}", "猫", "猫咪"]?:
drop(trie.insert(&mut words, key)?)
print(trie.ordered(&words)?)?
print(trie.has(&words, "car"), trie.has(&words, "ca"))?
print(trie.has_prefix(&words, "ca"), trie.has_prefix(&words, "z"))?
print(trie.with_prefix(&words, "car")?)?
print(trie.with_prefix(&words, "猫")?)?
print(trie.longest_prefix(&words, "cartwheel"))?
print(trie.longest_prefix(&words, "unknown"))?
print(trie.insert(&mut words, "car")?)?
print(trie.remove(&mut words, "car"), trie.has(&words, "cart"))?
print(trie.with_prefix(&words, "car")?)?
print(trie.remove(&mut words, ""))?
print(trie.longest_prefix(&words, "unknown"))?
print(trie.valid(&words)?)?
Ok(())
Expected output (the visually similar café spellings are distinct keys):
["", "café", "café", "car", "cart", "cat", "dog", "猫", "猫咪"]
True False
True False
["car", "cart"]
["猫", "猫咪"]
Option[str].Some("cart")
Option[str].Some("")
False
True True
["cart"]
True
Option[str].Nothing
True
Interface
| Operation | Result | Meaning |
|---|---|---|
create() | Result[Trie, AllocError] | Create an empty set with a permanent root. |
length(&tree) | i64 | Count stored keys, including the empty string if present. |
insert(&mut tree, key) | Result[bool, AllocError] | True for a new key; duplicates return False. |
has(&tree, key) | bool | Test for a complete stored key, not just a path. |
remove(&mut tree, key) | bool | Delete a key and prune unused suffix nodes; missing keys return False. |
has_prefix(&tree, prefix) | bool | Whether any stored key starts with the prefix. |
with_prefix(&tree, prefix) | Result[list[str], AllocError] | All matching keys in scalar-lexicographic order, including the prefix itself if stored. |
ordered(&tree) | Result[list[str], AllocError] | All keys; equivalent to with_prefix(&tree, ""). |
longest_prefix(&tree, text) | Option[str] | The longest stored key that prefixes the text. |
valid(&tree) | Result[bool, AllocError] | Check structure, terminals, pruning, and free-list membership. |
All key, prefix, and text parameters borrow &str; the implementation keeps
immutable string handles only when needed. Exact lookup, prefix existence,
longest-prefix matching, and deletion allocate nothing. Insertion can allocate
node storage; enumeration allocates its result list; validation allocates
visitation marks and a worklist.
For an empty set, has_prefix(&tree, "") is False, enumeration returns an
empty list, and longest_prefix returns Nothing. If "" is stored,
longest-prefix matching returns at least Some(""), even when no nonempty key
matches. Removing a key does not remove its longer descendants or shorter
stored prefixes.
Unicode and ownership
Matching is exact and case-sensitive. There is no normalization, case folding,
locale collation, or grapheme-cluster segmentation. "é" and "e\u{301}"
follow different paths. A combining mark is its own edge; an emoji outside the
basic multilingual plane is one scalar, not several byte edges. Embedded NUL
and the maximum Unicode scalar are accepted like other valid string content.
Terminal nodes retain full keys as immutable str handles, so enumeration
need not rebuild each string from its path. Returned strings remain valid after
deletion, slot reuse, or dropping the trie. They share immutable storage rather
than copying key bytes; an inserted heap-backed key can therefore keep that
storage alive. See text storage and iteration.
Validation
The model driver compares the trie with a sorted-list set and independent prefix/longest-prefix scans. It checks 14,400 pairs of five-key insertion/deletion permutations, duplicates, missing removals, complete pruning and refill, plus 8,000 mixed operations over Unicode, case-sensitive, empty, NUL-containing, and prefix-related keys. A high-fanout case inserts 256 distinct scalar edges at both root and interior levels in reverse order, then alternates first/middle/last sibling deletion. A 10,000-scalar key checks returned strings after the original key, query, and trie owners are destroyed, with same-sized allocation pressure before comparison against independently rebuilt text.
The integration tests additionally sweep insertion and prefix-result allocation failures, retry failed insertion, disable allocation during lookup/pruning/reuse, and exercise a 100,000-scalar key under a 256 KiB native stack on Linux:
cargo test --locked --features runtime-checks --test integration test_data_structure_patterns::trie
Read trie storage, updates, and costs for the implementation and its limits. Compare the balanced search trees when ordering alone, rather than shared-prefix retrieval, is the main requirement.
Trie storage, updates, and costs
The runnable trie uses a private list[Node]. Each node contains a
one-scalar edge label, a parent ID, first-child and next-sibling IDs, an optional
stored key, and free-list bookkeeping. The root occupies slot zero and has no
edge label. -1 means no link, never a valid index to dereference.
Ordered child links
A parent’s children form a singly linked list ordered by scalar label. To find an edge, scan siblings until its label matches or exceeds the desired scalar. This keeps storage and rewiring simple, but costs a linear scan in the parent’s fanout; it is not the constant-time edge lookup sometimes assumed for tries.
One owner controls all nodes. IDs borrow nothing, and each mutation updates one field at a time. Backing-list growth can move rows without changing IDs. Parent links support upward pruning and iterative traversal without stored references or a stack of borrowed nodes. No node ID escapes the module, so slot reuse needs no public generation or cross-collection identity scheme.
Insert without damaging the trie on allocation failure
Insertion makes two streaming passes over the input scalars:
- Follow existing edges until the first missing edge, then count the remaining
missing suffix nodes. An existing terminal returns
Falseimmediately. - After storage preparation succeeds, walk again, insert the missing edges in sibling order, and set the final node’s stored key.
Preparation appends empty nodes to a detached free pool until it has enough slots for the entire missing suffix. Failure may retain spare capacity and free nodes, but cannot change reachable links, terminals, or key count. A retry can reuse successful partial preparation. Taking slots, rewiring, and copying an immutable string handle allocate nothing once preparation succeeds.
Inserting the empty key, or marking an already-existing prefix node terminal,
needs no new nodes. No repeated key[index] scanning is used: string iteration
visits Unicode scalars directly.
Remove and prune without allocating
Locate the terminal; missing paths and nonterminal prefixes return False.
Clear its stored key, then walk upward while the current node is a nonterminal
leaf. Unlink that node from its parent’s child list, clear its fields, and put
its slot on the free list. Stop at the permanent root or a node still needed by
another key.
For example, removing "car" leaves the path for "cart". Removing "cart"
afterward may prune the r and t suffix, but must preserve the shared ca
path when "cat" is stored. Removing "" changes only the root’s terminal.
No backing-list entry is removed or shifted.
Prefix and longest-prefix traversal
Prefix enumeration first locates the prefix node. It then performs iterative preorder over that subtree: emit the terminal before visiting ordered children, use sibling links to advance, and parent links to backtrack. Stop before following the starting node’s sibling. This produces lexicographically ordered keys without sorting or rebuilding text, and without crossing into unrelated prefixes. The empty prefix starts at the root and visits the whole trie.
Longest-prefix matching follows the query’s scalar edges while remembering the last terminal encountered. It returns that terminal’s stored string handle, not a newly allocated slice of the query. The root terminal naturally handles an empty-string fallback.
All production operations are iterative. Trie depth follows key length, so the logarithmic-height stack bound of a balanced search tree would not be sufficient here. The validator also uses an explicit heap worklist, not recursion.
Complexity and retained memory
Let m be the input’s scalar length and d the maximum fanout encountered.
Membership, prefix existence, longest-prefix lookup, and the tree work of
insertion/removal are bounded by O(m × d), plus streaming the input. Pruning
also scans affected sibling lists. A missing-suffix insert may prepare O(m)
slots; occasional backing-list growth can move all retained rows, so it is not
strictly bounded by key length on each call.
After locating a prefix, enumeration takes O(v + k) work for v visited nodes
and k emitted handles, plus result-list allocation. Its traversal state uses
constant native stack and no auxiliary traversal list. Output is unbounded:
callers asking for the empty prefix collect the whole set.
Storage follows the historical peak number of scalar-prefix nodes, including unused slots from partial preparation, rather than the current number of keys. Deletion releases stored string handles and clears labels in pruned nodes but retains node capacity for reuse. Terminal keys also retain their string storage; this example is not a compressed radix trie or a minimal dictionary automaton.
valid allocates marks/workspace over retained storage. It checks sorted unique
sibling labels, valid live IDs, parent consistency, reachability without cycles
or duplicate edges, no unpruned nonterminal leaves, terminal paths and key count,
and free/live separation. It then searches each stored key to verify its path,
so its cost includes all those searches; it is not a constant-time health check
or a parser for hostile serialized state.
Deliberate limits
This is a set, not a generic key/value map. There are no result limits, lazy cursors, wildcard/fuzzy search, compressed edges, persistence, concurrency, or normalization policy. A sorted child vector, dictionary, or radix representation could trade allocation and implementation complexity for faster high-fanout queries; measure a workload before choosing one. No new standard-library contract or language feature is needed for this example.
Collection operation examples
Use these examples when a program needs a particular collection operation. Start with Collections, ownership, and borrowing for ownership and borrowing.
- Construct collection literals
- Reserve collection capacity
- Copy while preserving the source
- Take dictionary snapshots
- Remove set members
- Reverse a list
- Sort a list and exchange elements
- Clear and reuse storage
- Extend a list
- Update dictionaries
- Update sets
- Compare sets
- Combine sets
- Filter sets in place
- Search lists
- Slice lists
- Use optional collection results
- Choose numeric types in expressions
- Observe Unicode string values
- Combine different error types
Handle collection allocation failures
list[T]() creates an empty list and returns Result[list[T], AllocError].
Sets and dictionaries work
the same way: set[T]() and dict[K, V]().
If you know how many entries you need, use with_capacity(n). The collection
starts empty and has space for at least n entries, including any hash table.
Here construction and insertion both propagate errors to the caller:
def answers() -> Result[list[i64], AllocError]:
mut values = list[i64].with_capacity(2)?
values.append(21)?
values.append(42)?
Ok(values)
def main() -> Result[(), IoError]:
match answers():
case Ok(values):
print(values)?
case Err(AllocError.OutOfMemory):
print("not enough memory")?
case Err(AllocError.CapacityOverflow):
print("requested capacity is too large")?
Ok(())
[21, 42]
On failure, construction reclaims any memory it already obtained.
Type aliases work too: after type Numbers = list[i64],
you can write Numbers().
append for a list, add for a set, and insert for a dictionary
each return Result[(), AllocError].
reserve(n) reserves space for at least n additional entries; it is available
on all three collection types and returns the same result type.
def add_answers(values: &mut list[i64]) -> Result[(), AllocError]:
values.reserve(2)?
values.append(21)?
values.append(42)?
Ok(())
def main() -> Result[(), AllocError | IoError]:
mut values: list[i64] = []?
match add_answers(&mut values):
case Ok(done):
print(values)?
case Err(AllocError.OutOfMemory):
print("not enough memory")?
case Err(AllocError.CapacityOverflow):
print("requested capacity is too large")?
Ok(())
[21, 42]
AllocError has two variants, neither of which allocates.
OutOfMemory means the allocator rejected a request. CapacityOverflow means
the requested size cannot be represented; a negative reservation is also a
capacity error. You can handle a failure without terminating the program:
def main() -> Result[(), AllocError | IoError]:
mut values = [1, 2]?
match values.reserve(-1):
case Ok(done):
print("reserved")?
case Err(error):
print(error)?
print(values)?
Ok(())
AllocError.CapacityOverflow
[1, 2]
On failure the collection keeps its existing contents. An insertion consumes its arguments even when it fails, so an owned argument is cleaned up rather than returned to you. Reserve first if you want to keep an item until storage is ready. Successful reservation covers collection storage, including a dictionary or set’s hash table; constructing the elements themselves may still allocate.
These methods require a mutable receiver. Dictionary
insert(key, value) replaces an existing value or adds a new entry. Adding an
existing set element or replacing a dictionary entry needs no storage growth.
Handle allocating arguments too: copy([1, 2]?)? checks both construction and
copying. Explicit .unwrap() chooses to terminate if a result is an error.
Empty a collection and reuse its storage
Lists, dictionaries, and sets have clear() -> (). The method requires mutable
access, drops the contents, and retains capacity for later insertions:
def main() -> Result[(), AllocError | IoError]:
mut items = [10, 20]?
items.clear()
print(items)?
items.append(30)?
print(items)?
mut scores = {"Ada": 10}?
scores.clear()
print(scores)?
Ok(())
[]
[30]
{}
Clearing itself allocates no storage. Owned elements are destroyed before the method returns, in list order or dictionary insertion order; custom destructors may have their own effects and allocations. Clearing an empty collection is fine.
Collection literals
Collection literals directly return Result[collection, AllocError]:
def build() -> Result[list[list[i64]], AllocError]:
[[1, 2]?, [3]?]
def main() -> Result[(), AllocError | IoError]:
print({"answer": 42}?)?
print(build()?)?
Ok(())
{"answer": 42}
[[1, 2], [3]]
Construction stops at its first allocation failure, drops the partial collection,
and skips later entries. The nested ? operations extract each inner list and
propagate its failure to build.
Empty displays need context: values: Result[list[i64], AllocError] = [], or
values: list[i64] = []? inside a function returning Result[..., AllocError].
Byte keys and iterable conversion
Byte literals and tuples of hashable values can be dictionary keys or set members. Call-style list constructors can collect text scalars or literal bytes:
def main() -> Result[(), AllocError | IoError]:
names = {b"GIF8": "gif"}?
seen = {(b"tag", 1), (b"tag", 1)}?
print(names[b"GIF8"], len(seen))?
print(list("aé")?)?
print(list[u8](b"AB")?)?
Ok(())
gif 1
["a", "é"]
[65, 66]
The .from(...) methods accept collections, ranges, and generators,
but not text or bytes. See collection conversion
for the supported constructor forms.
Copy without losing the source on failure
copy(value) returns
Result[T, AllocError]. It observes the source, so your original value remains
usable whether copying succeeds or fails. Use ? to keep the success path short:
def extended(source: &list[i64]) -> Result[list[i64], AllocError]:
mut result = copy(source)?
result.append(30)?
Ok(result)
def main() -> Result[(), AllocError | IoError]:
original = [10, 20]?
print(extended(&original)?)?
print(original)?
Ok(())
[10, 20, 30]
[10, 20]
This also works for nested collections, ordinary classes, and enum payloads. If copying a later field or element fails, the partial copy is cleaned up and the original remains unchanged. Scalars and immutable values, including strings, need no allocation to copy; their existing immutable storage can be shared.
Generators and classes with
custom cleanup cannot be copied, including when nested in another value.
The operation covers duplication itself. In copy([1, 2]?)?, the first ?
handles construction and the second handles copying.
Take dictionary snapshots with recoverable allocation
keys() and values() build new lists in dictionary insertion order,
returning Result[list[T], AllocError]:
def names(scores: &dict[str, i64]) -> Result[list[str], AllocError]:
result = scores.keys_snapshot()?
Ok(result)
def main() -> Result[(), AllocError | IoError]:
mut scores = {"Ada": 10, "Bea": 20}?
saved = scores.values_snapshot()?
scores.insert("Ada", 30)?
print(names(&scores)?)?
print(saved)?
print(scores)?
Ok(())
["Ada", "Bea"]
[10, 20]
{"Ada": 30, "Bea": 20}
For numbers, strings, and other immutable values, the dictionary remains usable
after either outcome. String storage is shared, so the new list does not copy
string contents. The snapshot remains valid after the source
changes or leaves scope. An empty dictionary produces Ok([]) if its new list
header can be allocated.
For owned values such as lists or classes, values() requires an owned
temporary. Request duplication explicitly to keep the
original:
def rows(data: &dict[str, list[i64]]) -> Result[list[list[i64]], AllocError]:
copy(data)?.values_snapshot()
def main() -> Result[(), AllocError | IoError]:
data = {"first": [1, 2]?, "second": [3]?}?
print(rows(&data)?)?
print(data)?
Ok(())
[[1, 2], [3]]
{"first": [1, 2], "second": [3]}
Calling make_dictionary().values_snapshot() instead transfers owned values from
that temporary into the list. If allocation fails, the temporary and its values
are cleaned up. keys_snapshot() never takes the dictionary’s values, so it works on
borrowed dictionaries regardless of their value type.
Both operations reserve all list storage before retaining or transferring any
elements. They report OutOfMemory or CapacityOverflow through AllocError.
Update a dictionary by transferring another dictionary
destination.update(source) returns Result[(), AllocError], replacing
existing values and adding new entries. Existing keys keep their positions;
new keys appear in source insertion order:
def merge(destination: &mut dict[str, i64], source: dict[str, i64]) -> Result[(), AllocError]:
destination.update(source)
def main() -> Result[(), AllocError | IoError]:
mut scores = {"Ada": 10, "Bea": 20}?
changes = {"Bea": 25, "Cam": 30, "Ada": 15}?
merge(&mut scores, changes)?
print(scores)?
Ok(())
{"Ada": 15, "Bea": 25, "Cam": 30}
The source is consumed on both outcomes. On allocation failure, the destination
stays unchanged and source owners are dropped. Use
destination.update(copy(source)?) to keep the source. Storage for every
new entry is reserved before any value is replaced. Old destination values are
dropped during successful replacement; their destructors keep their own effects.
Updating only existing keys needs no new runtime storage. Only same-typed
dictionaries are accepted; pair iterables and keyword arguments are deferred.
Extend a list by transferring another list
items.extend(other) returns Result[(), AllocError]. It moves all elements
from another list of the same type after reserving space:
def combined() -> Result[list[i64], AllocError]:
mut items = list[i64]()?
more = [10, 20, 30]?
items.extend(more)?
Ok(items)
def main() -> Result[(), AllocError | IoError]:
print(combined()?)?
Ok(())
[10, 20, 30]
The source is consumed even when reservation fails; its elements are then cleaned
up and the destination stays unchanged. To retain a source, explicitly write
items.extend(copy(source)?). References and general iterables are not
accepted as sources yet. The destination needs mutable access. Existing capacity
is reused, and moving elements does not clone or allocate their contents.
The source literal also returns an allocation result; its ? propagates that
failure before extend runs.
Search and count list elements
For lists of integers, floats, booleans, or strings, these queries borrow their inputs and allocate nothing:
def main() -> Result[(), AllocError | IoError]:
names = ["Ada", "Lin", "Ada"]?
print(names.count("Ada"))?
print(names.find("Ada"))?
print(names.rfind("Ada"))?
print(names.find("missing"))?
Ok(())
2
Option[i64].Some(0)
Option[i64].Some(2)
Option[i64].Nothing
Searches return optional zero-based positions. The query must match the element type exactly. Float comparisons follow IEEE rules: NaN never matches, and positive and negative zero match each other. Searching lists of aggregates is not yet supported.
Take a list slice
A slice items[start:stop:step]
returns a new list inside a Result. Start is included and stop is excluded.
Negative bounds count from the end, omitted bounds run to the ends, a negative
step walks backwards, and bounds outside the list clamp to its ends:
def middle(items: &list[i64]) -> Result[list[i64], AllocError]:
items[1:-1]
def main() -> Result[(), AllocError | IoError]:
items = [10, 20, 30, 40]?
print(middle(&items)?)?
print(items[-100:100]?)?
print(items[3:1]?)?
print(items[::-2]?)?
print(items)?
Ok(())
[20, 30]
[10, 20, 30, 40]
[]
[40, 20]
[10, 20, 30, 40]
Bounds must be i64, and the step is a NonZero[i64]. Even an empty result
can fail to allocate its list header; use ? or match to handle AllocError.
The slice copies each selected element with copy(), so nested lists are
duplicated and the source keeps its elements. Element types that cannot be
copied, such as std.bytes.Bytes, are rejected. For those, call
items.slice(start, stop) on an owned temporary, such as a list returned by a
function: it transfers the selected elements and drops the rest. Use
copy(items)?.slice(start, stop) only when the elements can be copied anyway.
If allocation fails, the temporary is cleaned up in full; a borrowed source
remains unchanged.
Preserve different error causes
Declare a sum of the errors an operation can return. Each ? injects its
original error into that sum, preserving the variant and its payload:
type BuildError = ParseError | AllocError | IoError | OverflowError[u8]
def work(text: str) -> Result[list[u8], BuildError]:
n = u8.parse(text)? # ParseError
values = [n, n + 1u8]? # AllocError and OverflowError[u8]
print("built values")? # IoError
Ok(values)
def main() -> Result[(), AllocError | IoError]:
print(str.repr(work("41"))?)?
match work("invalid"):
case Ok(values):
print(values)?
case Err(error):
match error:
case cause: ParseError:
print(cause == ParseError.Invalid)?
case cause: AllocError:
print(cause)?
case cause: IoError:
print(cause)?
case cause: OverflowError[u8]:
print(cause)?
Ok(())
built values
Result[list[u8], AllocError | IoError | OverflowError[u8] | ParseError].Ok([41, 42])
True
The alias names the combination; it does not wrap the errors or allocate.
Matching a member gives the original concrete value. ? also widens an error
sum into a larger declared sum without dropping its cause. Normal scope cleanup
runs on the error path.
A main returning Err reports the error on standard error and exits with
status one. Use ? for propagation and match for recovery. .unwrap() is an
explicit choice to terminate immediately on Err or Nothing, without normal
scope cleanup. Neither sums nor ? catch runtime traps or turn Option.Nothing
into an error.
Builtin Failure discards the original cause. It is a last resort, for when
no precise error can be written; these errors combine without it. The
error tutorial describes its
behavior.
Choosing numeric types in expressions
Type arguments go before the call: range[u8](8). A collection annotation or
function return type can also guide a directly written range comprehension.
def squares() -> Result[list[u8], AllocError | OverflowError[u8] | ZeroDivisionError]:
[n * n for n in range(8) if n % 2 == 0]
def main() -> Result[(), AllocError | IoError | OverflowError[u8] | OverflowError[u16] | ZeroDivisionError]:
print(squares()?)?
print([n * n for n in range[u8](8) if n % 2 == 0]?)?
small: list[u16] = [n + 1 for n in range(3)]?
print(small)?
print(list(range[u8](5, 0, -2))?)?
fraction: f32 = 3.5
print(fraction * 2)?
Ok(())
[0, 4, 16, 36]
[0, 4, 16, 36]
[1, 2, 3]
[5, 3, 1]
7.0
Context guides unsuffixed literals, never changes an existing value’s type.
x: u8 = 256 fails, as does assigning an i64 binding to a u8 binding. Explicit
suffixes remain authoritative. A range’s bounds must fit its element type;
range[u8](256) therefore fails even though its stop is exclusive.
Inference stays within straightforward expressions. It does not infer a range
type backward through filters, arbitrary calls, or a previously stored range.
Use explicit range[T] in those cases. There is no silent numeric widening or
narrowing.
Find and validate collection values
Option[T] has two variants: Some(T) and Nothing.
Some, Nothing, Ok, and Err are built in and need no type prefix.
Nothing belongs to one concrete option type; it is not a universal null.
def first_positive(values: list[i64]) -> Option[i64]:
for value in values:
if value > 0:
return Some(value)
Nothing
def main() -> Result[(), AllocError | IoError]:
for values in [[-1, 0]?, [-1, 42]?]?:
match first_positive(values):
case Some(value):
print(value)?
case Nothing:
print("not found")?
Ok(())
not found
42
Use Result[T, E] when the absent result has an explanation. Its variants
are Ok(T) and Err(E). This example intentionally maps arithmetic errors
to explanatory strings; a numeric API can instead preserve the builtin error
types. The return signature supplies both types:
def divide(left: i64, right: i64) -> Result[i64, str]:
if right == 0:
return Err("division by zero")
result = checked:
left // right
match result:
case Ok(value):
Ok(value)
case Err(OverflowError(_)):
Err("quotient out of range")
case Err(ZeroDivisionError()):
Err("division by zero")
def main() -> Result[(), AllocError | IoError]:
for result in [divide(8, 2), divide(8, 0)]?:
match result:
case Ok(value):
print(value)?
case Err(message):
print(message)?
Ok(())
4
division by zero
Validate text in a collection
For an operation that can fail but has no success data, return Result[(), E]
and construct success with Ok(()):
def validate(name: str) -> Result[(), str]:
if len(name) == 0:
return Err("name is empty")
Ok(())
def main() -> Result[(), AllocError | IoError]:
for result in [validate("Plenty"), validate("")]?:
match result:
case Ok(_):
print("valid")?
case Err(message):
print(message)?
Ok(())
valid
name is empty
Unit is a real success payload, distinct from the absence represented by
Nothing. Option[()] also works. Standalone unit bindings and unit parameters
remain unsupported. User-defined enum variants still use their enum’s prefix,
even if a variant happens to be named Ok or Some.
Program entrypoints and native compilation
Running and linking
Plenty compiles ahead of time. Running a file compiles a
temporary native executable, runs it, and removes it afterward. Both running
and compiling require a cc-compatible linker driver. The default is cc on
PATH; add --linker clang or --linker /path/to/driver to select another.
Repeat --link-arg ARG for extra native link arguments. Checking needs no linker.
The Rust runtime is packaged with Plenty; compiling Plenty programs needs
neither cargo nor rustc.
Entrypoints and exit status
Execution starts by calling main once. Every binary application must declare
main with no parameters. It can return (), i32, Result[(), E], or
Result[i32, E]. Ok(()) produces exit status zero and Ok(status) uses that
status. An Err is reported on standard error, then dropped, and the program
exits with status one. For example, a main that returns the error from reading
a missing file prints
error: main returned IoError.System(2): No such file or directory.
The error is printed once, when main returns it, not at each ? on the way.
The () form
finishes successfully with exit status zero. The i32 form returns a process
exit status: zero means success, and nonzero means failure. Use an i32 literal
such as 0i32 or 1i32; return annotations also guide unsuffixed literals.
Keep executable statements inside functions. Module scope contains def,
class, enum, and type declarations, plus imports. Bindings inside main
are local to it.
The modules part explains imports and pub visibility.
Owned locals are dropped before the program exits, including when main returns
a nonzero status. Operating systems
limit the range of observable exit statuses; use small nonnegative codes for
portable command-line programs.
You can also choose the successful process status explicitly:
def main() -> Result[i32, IoError]:
print("ready")?
Ok(0i32)
ready
Calling a function at module scope is an error, even if main is also declared:
def main() -> ():
print("ready").unwrap()
main()
executable statements are not allowed at module scope
Keeping an executable or linking externally
Use --compile when you want to keep the executable and run it repeatedly
without compiling again. The executable does not need Plenty installed.
For an external build system, use plenty --emit-object hello.plenty -o hello.o
and plenty --emit-runtime runtime. Link the object with
runtime/libplenty_runtime.a followed by the arguments in runtime/link-args.txt.
These two commands do not invoke a linker. They emit a complete application,
including its imports; ordinary pub functions are not C exports.
Native compilation currently supports x86_64-unknown-linux-gnu. Use
plenty --print-target to inspect the packaged target. An explicit
--target TRIPLE must match it; choosing a linker does not enable cross-compilation.
Reverse a list in place
reverse() mutates a list through a mut binding or &mut reference and returns
(). It allocates nothing and moves no elements out of the list:
def flip(items: &mut list[i64]) -> ():
items.reverse()
def main() -> Result[(), AllocError | IoError]:
mut items = [10, 20, 30]?
flip(&mut items)
print(items)?
items.reverse()
print(items)?
Ok(())
[30, 20, 10]
[10, 20, 30]
Owned elements such as classes and nested lists work too. Reversing does not copy or destroy them; eventual list cleanup follows their new order. To keep the original order separately, explicitly copy the list first.
Sort a list and exchange elements
sort() orders a list in place without allocating. Tuples sort by their first
component, so sorting (start, stop) spans puts overlapping spans next to each
other. Merging them is then one pass:
def merge(spans: &mut list[(i64, i64)]) -> Result[list[(i64, i64)], AllocError]:
spans.sort()
mut merged: list[(i64, i64)] = []?
for i in range(len(spans)):
start, stop = spans[i]
if len(merged) > 0 and start <= merged[-1][1]:
if stop > merged[-1][1]:
merged[-1] = (merged[-1][0], stop)
else:
merged.append((start, stop))?
Ok(merged)
def main() -> Result[(), AllocError | IoError]:
mut spans = [(8, 10), (1, 3), (15, 18), (2, 6)]?
print(merge(&mut spans)?)?
print(spans)?
Ok(())
[(1, 6), (8, 10), (15, 18)]
[(1, 3), (2, 6), (8, 10), (15, 18)]
swap(i, j) exchanges two elements of any type in place. Here it moves the row
with the largest leading value to the top, as elimination with pivoting does,
without copying a row:
def pivot(rows: &mut list[list[i64]]) -> ():
mut best = 0
for i in range(1, len(rows)):
if rows[i][0] > rows[best][0]:
best = i
rows.swap(0, best)
def main() -> Result[(), AllocError | IoError]:
mut rows = [[1, 2]?, [7, 8]?, [4, 5]?]?
pivot(&mut rows)
print(rows)?
Ok(())
[[7, 8], [1, 2], [4, 5]]
Copyable elements can also be exchanged by
assigning several targets:
items[i], items[j] = items[j], items[i].
Combine sets without consuming them
union returns an independent set inside a Result. Both inputs are borrowed:
def merged(a: &set[i64], b: &set[i64]) -> Result[set[i64], AllocError]:
a.union(b)
def main() -> Result[(), AllocError | IoError]:
a = {1, 2}?
b = {2, 3}?
match merged(&a, &b):
case Ok(values):
print(len(values))?
print(1 in values and 3 in values)?
case Err(error):
print(error)?
print(len(a))?
Ok(())
3
True
2
An allocation failure preserves the inputs. Set iteration order is unspecified.
Find common members with intersection:
def main() -> Result[(), AllocError | IoError]:
match {1, 2}?.intersection({2, 3}?):
case Ok(common):
print(len(common))?
print(2 in common)?
case Err(error):
print(error)?
Ok(())
1
True
Use difference to remove another set’s members from a new result:
def main() -> Result[(), AllocError | IoError]:
wanted = {1, 2, 3}?
completed = {2, 3, 4}?
match wanted.difference(completed):
case Ok(remaining):
print(len(remaining))?
print(1 in remaining)?
case Err(error):
print(error)?
print(len(wanted))?
Ok(())
1
True
3
symmetric_difference keeps members found in exactly one of the two inputs:
def main() -> Result[(), AllocError | IoError]:
match {1, 2}?.symmetric_difference({2, 3}?):
case Ok(changed):
print(len(changed))?
print(1 in changed and 3 in changed)?
print(2 in changed)?
case Err(error):
print(error)?
Ok(())
2
True
False
Compare sets
Set relationships borrow both sets and allocate nothing:
def main() -> Result[(), AllocError | IoError]:
needed = {"read"}?
available = {"read", "write"}?
print(needed.issubset(available))?
print(available.issuperset(needed))?
print(needed.isdisjoint({"write"}?))?
print(len(needed))?
Ok(())
True
True
True
1
Both operands must have the same set type. Empty sets are subsets of every set and disjoint from every set.
Filter a set in place
intersection_update retains common members in a mutable set. It borrows the
other set, returns unit, and reuses existing storage without allocating:
def main() -> Result[(), AllocError | IoError]:
mut selected = {1, 2, 3}?
allowed = {2, 3, 4}?
selected.intersection_update(allowed)
print(len(selected))?
print(1 in selected)?
print(2 in selected and 3 in selected)?
print(len(allowed))?
Ok(())
2
False
True
3
The source must be a different set: it stays borrowed while the destination is
mutated. Use intersection when you want an independent result instead.
Use difference_update to remove the other set’s members in place:
def main() -> Result[(), AllocError | IoError]:
mut pending = {1, 2, 3}?
done = {2, 3, 4}?
pending.difference_update(done)
print(len(pending))?
print(1 in pending)?
print(len(done))?
Ok(())
1
True
3
This also allocates nothing and keeps the source usable. To remove every member
without needing another set, use clear().
Remove set members
Use values.discard(value) to remove a member from a mutable set. It returns
True if the member was present and removed, or False if it was absent:
def main() -> Result[(), AllocError | IoError]:
mut names = {"Ada", "Bea"}?
name = "Ada"
print(names.discard(name))?
print(names.discard(name))?
print(name)?
print("Bea" in names)?
print(len(names))?
names.add(name)?
print(len(names))?
Ok(())
True
False
Ada
True
1
2
The argument is observed, so name remains usable. Discarding a stored zero,
False, or empty string still returns True.
Removal allocates nothing and preserves capacity for reuse. Like dictionary
removal, it has expected constant cost for well-distributed keys, apart from
hashing and cleanup; heavy hash collisions can still slow it down.
Use an exclusive reference when removing members through a function parameter.
Update a set by transferring another set
Sets also support update(source) -> Result[(), AllocError]. Members already
present are kept, and missing members transfer from the source:
def main() -> Result[(), AllocError | IoError]:
mut names = {"Ada", "Bea"}?
more = {"Bea", "Cam"}?
names.update(more)?
print(len(names))?
print("Ada" in names and "Bea" in names and "Cam" in names)?
Ok(())
3
True
The source must be a set of the same type and is consumed on both outcomes.
Use names.update(copy(source)?) to preserve it. Failed reservation leaves
the destination unchanged and cleans up the consumed source. Duplicate-only and
empty sources need no new runtime storage; existing capacity can cover new
members too. Set iteration order remains unspecified.
Cleanup before a tail call
A function that ends with a call makes a tail call. Plenty evaluates the arguments, cleans up the function’s remaining values, and only then enters the called function:
class Resource:
name: str
def __del__(self) -> ():
print(("release " + self.name).unwrap()).unwrap()
def finish(total: i64) -> Result[i64, IoError | OverflowError[i64]]:
print("finish")?
Ok(total)
def step(total: i64) -> Result[i64, IoError | OverflowError[i64]]:
scratch = Resource("scratch")
finish(total + 1)
def main() -> Result[(), AllocError | IoError | OverflowError[i64]]:
print(step(41)?)?
Ok(())
release scratch
finish
42
step and finish deliberately share a Result type, including the arithmetic
error that can originate in step, so forwarding the result remains a tail call.
scratch is released before finish runs. A value passed as an argument moves
to the called function, which cleans it up instead. Tail recursion therefore
keeps its constant stack use when each step owns a resource. A call that borrows
one of the function’s values, such as inspect(&scratch) or a method call on it,
cannot release that value first: it runs as an ordinary call and cleanup follows.
A reference the function received as a parameter refers to a value outside the
function, so passing it on is still a tail call.
The destructor’s printing is only an observation aid for this cleanup example;
its explicit unwrap() terminates if that diagnostic output fails. Ordinary
work, including finish, propagates its errors instead.
Repeat work with tail recursion
A tail-recursive function can repeat work without growing the call stack. Ordinary range loops are usually the first choice for repetition. This page explains recursion when a function naturally calls itself.
def sum_to(n: i64, total: i64) -> Result[i64, OverflowError[i64]]:
if n <= 0:
return Ok(total)
sum_to(n - 1, total + n)
def main() -> Result[(), IoError | OverflowError[i64]]:
print(sum_to(100, 0)?)?
Ok(())
5050
The recursive call is the last operation on that path. Plenty reuses its call
frame in native code. An explicit
return sum_to(n - 1, total + n) works too. In contrast, 1 + recurse(...)
still has addition to do after the call and is not a tail call.
Returning a range also works: the result’s storage belongs to the original caller and stays alive throughout the recursion.
def countdown(n: i64) -> Result[range[i64], OverflowError[i64]]:
if n == 0:
return Ok(range(2, 5))
countdown(n - 1)
def main() -> Result[(), IoError | OverflowError[i64]]:
print(countdown(1000)?)?
Ok(())
range(2, 5, 1)
Passing an owned range, record, or generator as an argument still needs an ordinary call and consumes native stack space. A direct recursive tail call with such an argument is rejected as described below. Returning one alone does not impose that limit.
A direct recursive call in tail position must use a native tail transfer.
This also applies to mutual recursion: first calls second, which calls
back into first. Plenty reports an error when it cannot remove the caller’s
frame safely. For example, the following call borrows a local that needs that
frame to stay alive:
def walk(value: &i64, remaining: i64) -> Result[i64, OverflowError[i64]]:
if remaining == 0:
return Ok(*value)
child = remaining - 1
walk(child, remaining - 1)
def main() -> ():
pass
recursive call to `walk` is in tail position but cannot be a tail call: the reference argument borrows local `child`
Pass a reference borrowed from the caller, carry owned scalar state, or use a loop when the next step needs local storage. Non-tail recursion still uses the native stack. Calls through function values are outside this direct-recursion diagnostic; using a callback does not promise bounded stack use.
Text and Unicode values in a collection
Strings store explicit lengths, so an embedded NUL does not end the text.
Indexing returns one Unicode scalar as a str; negative indices count from
the end. Each scalar fits inline in the string value, so indexing does not
allocate. General text can share immutable storage.
def main() -> Result[(), AllocError | IoError]:
text = "é\0😀"
print(len(text))?
print(text[-1])?
print("\0" in text)?
print([text]?)?
Ok(())
3
😀
True
["é\0😀"]
Text and file examples
Use these examples to look up text methods and advanced file operations. The text I/O reference lists signatures and platform limits.
Text methods borrow their inputs. Queries that return scalars or Option
allocate nothing; operations producing independent strings or collections
report allocation failure in a Result.
Bounded stream reads
Read a bounded number of characters with read(count). The count is Unicode
characters, not UTF-8 bytes. Zero consumes nothing; a negative count reads to EOF.
def demo() -> Result[(), IoError]:
write_text("bounded.txt", "é🦀hello")?
with open("bounded.txt")? as file:
print(file.read(2)?)?
print(file.read()?)?
Ok(())
def main() -> Result[(), IoError]:
demo()?
Ok(())
é🦀
hello
readline(count) limits a line read in the same way. A long line can arrive in
several pieces; a negative count reads the rest of the line.
def demo() -> Result[(), IoError]:
write_text("lines.txt", "abcd\nnext")?
with open("lines.txt")? as file:
print(file.readline(2)?)?
print(file.readline(2)?)?
print(file.readline(2)? == "\n")?
Ok(())
def main() -> Result[(), IoError]:
demo()?
Ok(())
ab
cd
True
Checking stream capabilities
Use readable() and writable() to inspect an open file through a shared reference.
Closed files return an error; closed itself remains an infallible property.
def demo() -> Result[(), IoError]:
with open("capabilities.txt", "w")? as file:
print(file.readable()?)?
print(file.writable()?)?
Ok(())
def main() -> Result[(), IoError]:
demo()?
Ok(())
False
True
Observe file closing
Files close automatically when dropped. Call close() explicitly to observe
a close error. A closed file remains a valid value without an open handle.
Run this in a scratch directory: it replaces plenty-handle.txt.
def inspect_file() -> Result[(), IoError]:
mut file = open("plenty-handle.txt", "w")?
print(file.closed)?
file.close()?
print(file.closed)?
file.close()?
Ok(())
def main() -> Result[(), IoError]:
inspect_file()?
Ok(())
False
True
Closing twice succeeds. Automatic close cannot report errors or promise durable writes.
Creating a file without overwriting
Use mode "x" when an existing path should be an error. This check happens
atomically during creation, so there is no separate existence-check race.
def demo() -> Result[(), IoError]:
with open("new.txt", "x")? as file:
file.write("first version")?
print(read_text("new.txt")?)?
Ok(())
def main() -> Result[(), IoError]:
demo()?
Ok(())
first version
Add + to an open mode to enable both reading and writing. r+ preserves an
existing file, w+ truncates or creates, x+ creates exclusively, and a+
starts at EOF and always appends writes. Writes overwrite bytes at the current
position; they do not insert characters. Use care with multibyte text.
def demo() -> Result[(), IoError]:
write_text("update.txt", "hello")?
with open("update.txt", "r+")? as file:
file.read(1)?
file.write("a")?
print(read_text("update.txt")?)?
Ok(())
def main() -> Result[(), IoError]:
demo()?
Ok(())
hallo
Saving and restoring text positions
tell() returns a u64 position cookie. Pass it back to seek() on the same
file to resume reading there, or use seek(0u64) to rewind. Do not calculate with
these values: they include newline state and are not byte or character counts.
Saved positions are only meaningful while the file contents remain unchanged.
def demo() -> Result[(), IoError]:
write_text("positions.txt", "one\r\ntwo")?
with open("positions.txt")? as file:
file.readline()?
saved = file.tell()?
print(file.read()?)?
file.seek(saved)?
print(file.read()?)?
Ok(())
def main() -> Result[(), IoError]:
demo()?
Ok(())
two
two
truncate(size) resizes a writable file in bytes and returns its new length.
With no size, it truncates at the current physical position. Neither form moves
the cursor. A text-position cookie is not a byte size. Cutting through a multibyte
character makes the file invalid UTF-8; a later read reports that error.
def demo() -> Result[(), IoError]:
write_text("short.txt", "keep rest")?
with open("short.txt", "r+")? as file:
file.read(4)?
print(file.truncate()?)?
print(read_text("short.txt")?)?
Ok(())
def main() -> Result[(), IoError]:
demo()?
Ok(())
4
keep
Flush and synchronize writes
write(text) returns a character count and preserves exact bytes, including
newlines. flush() flushes runtime buffers; files are currently unbuffered.
Use sync() when you need to request that the OS synchronize file contents and
metadata. Closing or flushing alone does not make that request.
Run this in a scratch directory: it replaces plenty-stream.txt.
def write_example() -> Result[(), IoError]:
with open("plenty-stream.txt", "w")? as file:
print(file.write("hello\n")?)?
file.flush()?
file.sync()?
file.close()?
with open("plenty-stream.txt", "a")? as file:
file.write("goodbye\n")?
Ok(())
def main() -> Result[(), IoError]:
write_example()?
Ok(())
6
Explicit close lets you propagate its error. Context exit then closes the already closed owner harmlessly. Writing can leave a prefix after an OS error, and append calls are not guaranteed to be atomic records across processes. Synchronization still follows your filesystem’s durability rules.
Reading a collection of lines
readlines() collects the remaining lines into an independent list[str],
keeping their translated newlines. At EOF it returns an empty list. Failures
release the partial list, but may have consumed input. For large or untrusted
files, bounded readline(count) lets you control memory use instead.
def demo() -> Result[list[str], IoError]:
write_text("collection.txt", "first\r\n\nlast")?
with open("collection.txt")? as file:
return file.readlines()
def main() -> Result[(), IoError]:
print(demo()?)?
Ok(())
["first\n", "\n", "last"]
writelines(lines) writes a list[str] without consuming it. It inserts no
newlines: include them in the strings when wanted. The operation returns
Result[(), IoError]; an error may leave a partially written file.
def demo() -> Result[(), AllocError | IoError]:
lines = ["one\n", "two\n"]?
with open("written.txt", "w")? as file:
file.writelines(lines)?
print(len(lines))?
with open("written.txt")? as file:
print(file.readlines()?)?
Ok(())
def main() -> Result[(), AllocError | IoError]:
demo()?
Ok(())
2
["one\n", "two\n"]
Classify text without allocating
isascii() accepts only ASCII characters, including controls and NUL.
isspace() checks for nonempty text containing only Unicode whitespace:
def main() -> Result[(), IoError]:
print("hello".isascii())?
print("é".isascii())?
print("".isascii())?
print(" \t\n".isspace())?
print(" x ".isspace())?
print("".isspace())?
Ok(())
True
False
True
True
False
False
Both queries borrow their receiver and allocate nothing. Whitespace means the
same Unicode White_Space characters removed by strip; NUL and zero-width
space do not count.
Remove an exact prefix or suffix
These methods remove one complete match at the chosen end. They return an
independent string inside a Result, leaving the original usable:
def title(name: &str) -> Result[str, AllocError]:
name.removeprefix("draft-")?.removesuffix(".txt")
def main() -> Result[(), AllocError | IoError]:
name = "draft-notes.txt"
print(title(&name)?)?
print("abab".removeprefix("ab")?)?
print("notes.txt".removesuffix(".csv")?)?
Ok(())
notes
ab
notes.txt
Empty patterns do nothing. Each method can fail to allocate even when the
contents stay unchanged. Use strip for surrounding whitespace instead.
Check a text prefix or suffix
startswith and endswith borrow strings and return bool without allocating.
Matches are literal and case-sensitive; an empty prefix or suffix always matches.
def main() -> Result[(), IoError]:
name = "report.plenty"
print(name.startswith("report"))?
print(name.endswith(".plenty"))?
print(name.endswith(".PLENTY"))?
print("".startswith(""))?
Ok(())
True
True
False
True
Each method takes one string, including a reference. Bounds and lists or tuples of alternative patterns are not supported yet.
Find a substring
find and rfind return the first or last match as Option[i64]. Unlike Python’s
-1 sentinel, a missing match is Nothing. Positions count Unicode scalars:
def main() -> Result[(), IoError]:
text = "é🙂é🙂"
print(text.find("🙂"))?
print(text.rfind("🙂"))?
print(text.find("missing"))?
print(text.rfind(""))?
Ok(())
Option[i64].Some(1)
Option[i64].Some(3)
Option[i64].Nothing
Option[i64].Some(4)
Both operations borrow their inputs and allocate nothing. Use match or ? in
an Option-returning function. An empty needle matches at the beginning for
find and at the end for rfind. Each takes one literal pattern; optional
bounds and regexes are not supported.
Count occurrences
text.count(needle) returns an i64 without allocating. Matches are literal
and do not overlap. An empty needle counts Unicode scalar boundaries:
def main() -> Result[(), IoError]:
print("banana".count("ana"))?
print("aaaaa".count("aa"))?
print("é🙂".count(""))?
print("".count(""))?
print("text".count("missing"))?
Ok(())
1
2
3
1
0
The method borrows both strings and takes exactly one pattern argument.
Trim surrounding whitespace
strip(), lstrip(), and rstrip() remove Unicode whitespace
from both ends, the left end, or the right end. Each returns Result[str, AllocError]:
def main() -> Result[(), AllocError | IoError]:
text = " é🙂 "
print(str.repr(text.strip())?)?
print(str.repr(text.lstrip())?)?
print(str.repr(text.rstrip())?)?
print(str.repr(" \t\n".strip())?)?
Ok(())
Result[str, AllocError].Ok("é🙂")
Result[str, AllocError].Ok("é🙂 ")
Result[str, AllocError].Ok(" é🙂")
Result[str, AllocError].Ok("")
They borrow the source and allocate only the final independent string. Empty and unchanged results still need that allocation. Interior whitespace is kept; NUL and zero-width space are not trimmed. These methods use Unicode White_Space and take no arguments; custom character sets are not supported yet.
Repeat text
text.repeat(count) accepts an i64 and returns Result[str, AllocError].
Zero and negative counts produce an empty string:
def main() -> Result[(), AllocError | IoError]:
print(str.repr("é🙂".repeat(3))?)?
print(str.repr("x".repeat(0))?)?
print(str.repr("x".repeat(-2))?)?
print(str.repr("x".repeat(9223372036854775807))?)?
Ok(())
Result[str, AllocError].Ok("é🙂é🙂é🙂")
Result[str, AllocError].Ok("")
Result[str, AllocError].Ok("")
Result[str, AllocError].Err(AllocError.CapacityOverflow)
The source stays usable. The runtime checks the complete size before allocating one independent output string; even empty and single-copy outputs can fail to allocate. String multiplication syntax is not supported yet.
Slice text by Unicode scalar position
A slice of a string returns
Result[str, AllocError], with the same clamping rules as lists. Positions
count Unicode scalar values, just like len and indexing:
def main() -> Result[(), AllocError | IoError]:
text = "Aé🙂Z"
print(str.repr(text[1:-1])?)?
print(str.repr(text[-100:100])?)?
print(str.repr(text[3:1])?)?
print(str.repr(text[::-1])?)?
print(str.repr(text.slice(1, -1))?)?
print(text)?
Ok(())
Result[str, AllocError].Ok("é🙂")
Result[str, AllocError].Ok("Aé🙂Z")
Result[str, AllocError].Ok("")
Result[str, AllocError].Ok("Z🙂éA")
Result[str, AllocError].Ok("é🙂")
Aé🙂Z
text.slice(start, stop) is the method form, with required bounds and no step.
The result owns its UTF-8 bytes and outlives the source. The operation creates
only the final string, including for an empty or whole-string slice, so even
these cases can return Err(AllocError.OutOfMemory). Source strings are never
modified. Combining marks count separately; positions are not grapheme clusters
or byte offsets.
Replace literal text with recoverable allocation
text.replace(old, new) replaces every non-overlapping match, scanning left
to right, and returns Result[str, AllocError]. It borrows all three strings:
def rename(text: &str) -> Result[str, AllocError]:
updated = text.replace("Plenty", "plenty")?
Ok(updated)
def main() -> Result[(), AllocError | IoError]:
original = "Plenty is Plenty"
print(rename(&original)?)?
print(original)?
print("aaaaa".replace("aa", "X")?)?
print("Aé".replace("", "-")?)?
print("banana".replace("na", "")?)?
Ok(())
plenty is plenty
Plenty is Plenty
XXa
-A-é-
ba
An empty search string inserts the replacement before, between, and after Unicode scalars. An empty replacement removes matches. Matches are literal and inserted text is not searched again; regexes and a replacement-count limit are not supported. The result owns its bytes and survives destruction of every input.
The runtime checks the complete output size and allocates only the final string.
Even a call with no matches, or an empty result, makes that allocation and can
return OutOfMemory; unrepresentable sizes return CapacityOverflow. Failure
leaves all inputs unchanged.
Build text with recoverable allocation
Use text.concat(other) to combine two strings. To combine a list or another
iterable of strings, use separator.join(parts). Both return Result[str, AllocError]:
def greeting(names: &list[str]) -> Result[str, AllocError]:
joined = ", ".join(names)?
"Hello, ".concat(joined)?.concat("!")
def main() -> Result[(), AllocError | IoError]:
names = ["Ada", "Bea"]?
print(greeting(&names)?)?
print(names)?
print("-".join([]?)?)?
Ok(())
Hello, Ada, Bea!
["Ada", "Bea"]
The methods borrow their inputs.
Joining first calculates the final size and then allocates one output buffer;
it does not build a succession of intermediate strings. An empty list gives an
empty string, and a one-element list adds no separator. Empty pieces still count:
joining ["a", "", "b"] with "-" gives "a--b".
UTF-8 characters and embedded \0 are preserved. join also accepts a set,
a dictionary (its keys), a string (its scalars), or a generator of strings;
see building text for
each form’s allocations.
Look up a character without trapping
Use text.get(index) when the index might be out of range. It returns
Option[str]: Some(character) when the index exists, Nothing otherwise.
def show_character(text: &str, index: i64) -> Result[(), IoError]:
match text.get(index):
case Some(value):
print(value)?
case Nothing:
print("missing")?
Ok(())
def main() -> Result[(), IoError]:
text = "café🙂"
show_character(&text, -1)?
show_character(&text, 3)?
show_character(&text, 5)?
print(text)?
Ok(())
🙂
é
missing
café🙂
Indices are i64 and count Unicode scalars, as with text[index]. Negative
indices count backward from the end; -1 selects the last scalar. Empty strings
and indices outside either end return Nothing. Neither form allocates: a
character is a short string stored inline in the value, independent of the
original, which remains available.
Ordinary text[index] terminates the program for a missing index. Indexing
ASCII text takes constant time; other text is scanned from the start to reach
the requested scalar, so use for character in text to traverse it.
Backlog
This is the single maintained list of remaining language, compiler, runtime, standard-library and tooling work. Entries are candidates, not promises about syntax or delivery dates. Row order is a catalogue, not a priority list.
Delivered milestones and retired entries live in Completed work. A B ID can appear there for its delivered subset and here for its remaining extensions; IDs are never renumbered or reused. The implementation status and reference describe what works today. Proposals record rationale and alternatives, not another work queue.
The experimental Windows x86-64 dataframe build and release packaging are recorded in completed work. The macOS ARM64 dataframe subset is recorded in completed work. Broader native OS qualification remains in B18.7.
Current position
B34’s bounded cleanup/review, B71’s typed HTTP acceptance batch, B76’s initial dataframe datatype expansion, the initial B39–B70 library slices and B17 package tooling have landed. Their delivery records are in Completed work. Preserve their installed-toolchain, CLI/data, JSON-service, mutual-TLS and HTTP qualification checks during further work.
Issue severity takes precedence over the batches below. Defects, reproductions,
claims and implementation/review status belong in the shared issues.db.
Nothing on this page assigns ownership or approves an API without a scoped
agreement and acceptance checks.
Next batches
- B76 — analytical workloads: decimal columns follow the
std.decimal proposal after its core module ships. Use the pinned Polars engine’s
Parquet capabilities; fixed-size arrays, dictionary semantics, nested casts
and broader list operations need separate scope. Use the
daily sales application
and
examples/sales_report/as evidence: truly large or rejection-heavy workloads should guide multi-output streaming. Temporal/grouped as-of matching and time-bucketed/rolling aggregations need a separate workload before expanding the API. - B23/B76 — measured SIMD extensions: packed views, wider shapes, broader predicates, maps/folds, runtime ISA selection and parallel scans need bounded legality/performance evidence. Account for code placement using the retained placement research.
- B71 — additive error ergonomics: reporting conveniences remain independent candidates, not prerequisites for typed HTTP composition.
Further library breadth, B11 stored-reference extensions and B72 state/linear-obligation experiments remain candidates requiring a concrete use case and an agreed contract. This sequence preserves the existing batch scope; it does not schedule all catalogue entries.
Deliberate deferrals
- B19 — module caching and frontend scale: serial efficiency, streaming lowering/emission and per-module caching need measurements and a focused batch. The module-cache investigation rejected separately compiled modules and stored syntax trees. A lowered-module cache needs imported interfaces and invalidation by body-derived facts; its broad-import benchmark offered about 30 ms of frontend work in a 44 ms body-edit build. Revisit for frontend-bound std-heavy applications, using module-edit measurements as the baseline. Remote caching waits on a local per-module contract.
- B22 — broader concurrency: richer mapping/effects, streaming results, rendezvous, larger or send-side selection remain deferred. Existing channel and executor checks do not establish deadlock freedom.
- Automatic parallelization (B24), optional backends (B20), stored-reference relationships (B11), broader callback erasure (B32) and automatic deep recursive operations (B33) require their own contracts and acceptance checks.
Scope of follow-on work
B22 — task models. The stackful-task investigation and async/await exploration do not commit Plenty to either implementation. Before adoption, compare the same bounded TCP/HTTP workload: I/O cancellation/registration lifetimes, allocation and failed-start preservation, stack budgets, suspension/loan checks, memory and latency. Fixed stacks trade pinning/relocation for reservations and fatal unchecked exhaustion. Retaining native concurrency remains an option. Prefer library APIs over new syntax; require bounded queues/backpressure, failed-send ownership, joins on all exits, cooperative cancellation and partial-result cleanup. Scoped generic tasks need not wait for heterogeneous callback storage in B32.
B32 — additional callback representations. Evaluate registry/event-handler use cases and specify invocation, move/drop, allocation failure and capture lifetimes before adding erased modes. Escaping borrowed captures depend on B11; C ABI callbacks and foreign-library lifetimes remain B28.
Work items
Language, compiler and tooling
| ID | Remaining work | Scope / dependency |
|---|---|---|
| B11 | Broader borrowing | Stored/returned references and views; returning a borrow chosen among several reference parameters; symbolic-index collection loans; field-precise loans through mut reference bindings. Optional/fallible references: phase 2 (mut locals and reference-carrying parameters), nested patterns through the carried reference, match expressions producing these values, dict.get_ref and list.get_ref/get_mut. Preserve owner/variant stability and temporary lifetimes; measure precision against compile-time cost. Proposal; delivered subset, including the len call-borrowing fix for issue #683. |
| B12 | Practical I/O and iteration | Reusable iterator/view protocols, stored/returned slice and substring views (B11), portable path semantics (B18). For substrings: str and std text APIs accepting &substr, more view methods and cached scalar counts. Further adapters need use cases; pipeline-specific extensions are B75. Delivered subset. |
| B13 | Public allocator control | Global/default and per-container selection, allocator state/lifetimes and buffer provenance. Recoverable allocation is delivered; allocator design remains proposed. Bounded/inline-capacity containers depend on B83 value parameters and fixed arrays. |
| B14 | Typed error composition | Finish unchecked arithmetic modes: a plenty.toml setting and a stdlib precondition audit, so std can follow --unchecked assume. Broader Option/Result adapters beyond the delivered fallback/mapping helpers (issues #800/#820), typed composition of parsing/conversion errors and remaining checked shift/cast operations. Preserve precise errors and use Failure only as a last resort. Human-readable reporting extensions are B71. Checked-default core, assumed list/view/Bytes indexing (issue #764), coordinated repository/benchmark-template adoption and bounded validation are recorded in the delivered subset and validation record. |
| B16 | Inference and value conveniences | Expected-result inference, partial explicit type arguments, contextual literals across generic arguments; NonZero[T] module constants; saturating conversion; standalone unit bindings/parameters/storage. Require concrete use cases/contracts, not automatic extensions of today’s inference. Parsing/conversion error composition is B14. Delivered subset, including block-if statement values (issue #674) and tuple patterns/nested unpacking (issue #675). |
| B17 | Modules, tooling and packaging | Shared per-user Git cache; semver selection over tags; per-module incremental objects (B19); re-exports and import-cycle policy. LSP: find references, document symbols, signature help, case pattern variants and incremental module re-checking. Preserve deterministic lookup, standalone compilation and reproducibility. Release artifacts: four targets via scripts/release/target.sh, with manual dry runs of the release workflow. Further targets need runtime/linker/artifact tests (B18); std breadth is B38. Exploratory workflows are listed under tooling investigations. Delivered subset, including the completed script design and optimized build defaults. |
| B18 | Windows and additional targets | Native GNU/Linux AArch64 compiler/runtime implementation is present alongside x86-64: baseline ARMv8-A code generation, target-aware native dependencies, Linux bindings, UDF traps, stack handling and AAPCS64 actor recovery. Qualification includes host SIGILL coexistence (issue #588), first actor-heap ownership (issue #589), and native libFuzzer discovery (issue #591). Development and emulated qualification records the reproducible harness; B17 owns artifact delivery. Native ARM CI passes the compiler/runtime suites, both optional engines and Docker proxy tests. Linux ARM64 is a release platform: archives are built and tested on native ARM runners and smoke-tested on AlmaLinux 8 and Debian 13. The v0.8.9 artifact audit confirms glibc 2.28 compatibility for both Linux targets (issue #730). Remaining ARM gates: dedicated hardware concurrency/signal stress, minimum kernel and page-size qualification. Packaged linalg integration and tutorial tests pass under the ARM guest; that local artifact does not qualify the release libc floor. The updated cross-platform assessment, interface/source map and toolchain experiments guide ARM macOS and x86-64 Windows; Windows ARM64 remains deferred. Changing the linker alone does not add cross-compilation. The Cosmopolitan/APE proposal recommends a bounded x86-64 compatibility experiment and identical-artifact non-Linux qualification before deciding on a restricted profile or broader port. |
| B19 | Measured codegen and storage | See the remaining performance work; completed measurements and optimizations are in the performance record. |
| B20 | Optional backends | Evaluate the GCC proposal before adopting another backend/common IR. The Rust-source investigation includes a checked scalar emitter; ownership-bearing APIs remain hand-lowered probes and need emitted semantic parity. Cranelift stays the production backend. Rust dependencies are B17. |
| B22 | Native concurrency extensions | User-facing scoped pool jobs borrowing the submitting scope; broader iterator/closure mapping, streaming results, richer certified effects/types, rendezvous and broader selection. Mutable, one-shot, borrowed, generator-containing and indirect map callbacks need separate scope. Preserve fallible submission and deterministic shutdown. Task-model gates are under follow-on scope. Delivered subset. |
| B23 | SIMD extensions | Packed storage/views, broader vector shapes, further predicates, total maps/modular folds, runtime ISA selection and parallel scans. Require separate evidence; hide alignment/tail/target bookkeeping, preserve checked arithmetic and ordered floating folds, and do not treat generic lists as packed arrays. Spikes/backend gates; delivered subset. |
| B24 | Automatic parallelization | Deferred beyond B34. Require evidence that effects, cleanup, allocation failures and result ordering remain correct; a thread-count setting alone is insufficient. |
| B27 | Expanded C representations | Export text/byte buffers and views, error sums of payload-free enums, payload-carrying errors, recoverable returned ownership and explicit class-method adapters. Keep allocator provenance, output initialization and error-path transfer visible. Owned export contract. |
| B28 | Library loading/lifecycle | Generate loaders from extracted contracts without original export source; improve third-party C binding tooling and error detail. Unloading, retained/foreign-thread callbacks, C record layout and header-assisted bindings need separate contracts. Current resident/thread-bound loaders. |
| B29 | Further generic constraints | Protocol bounds on data, generic methods satisfying protocols, combined bounds/composition, generic aliases and enum constructor inference. Check bodies against declared bounds so argument-independent errors stay at definitions; preserve explicit lookup and bounded specialization. Library-owned operations favors composition/capabilities; protocol access grants remain deferred. Protocol bodies records agreed default/final dispatch, member-specific grants and definition-time contract checking, not automatic injection; no implementation scheduled. Explicit Box[P] owners, shared/mutable dispatch and fixed-argument inference are delivered; see the contract. Delivered implementation and investigation records. |
| B32 | Broader callback storage | Mutable/consuming/borrowed/worker-bound erased modes and generator-containing environments in heap owners need their own contracts. Escaping borrowed storage depends on B11; see follow-on scope. Delivered subset. |
| B33 | Deep recursive operations | Iterative fallible copy/format work storage and an explicit bounded-memory contract for structural equality/membership, without hidden allocation failure. Preserve cleanup, shared-string ownership and float/NaN behavior. Rationale. |
| B36 | Named-call extensions | Finer protocol receiver tracking, deliberate builtin argument names and structured initialization for large records. Splats, variadics and //* markers remain in the unpacking proposal. Delivered defaults/named calls. |
| B37 | Ordering extensions | Cached/fallible sort keys, stable sorting and user-defined ordering without hidden allocation. The ordering proposal recommends __cmp__/Order for comparisons/equality and runtime sorting, not dictionary keys. Evaluate Timsort/Powersort with fallible scratch space; naming/migration remain decisions. Use measured runtime-sort bounds (issue #331). Until user ordering lands, __cmp__ and the __eq__/__lt__ family are compile errors (issue #613). Delivered subset. |
| B38 | Standard-library breadth | See library follow-ups and ergonomics. Further breadth requires a use case and scoped contract. Prefer Plenty unless native complexity/security justifies a wrapper; use the engine criteria for large native libraries. Delivered modules. |
Tooling investigations
These belong to B17 and are not scheduled implementation commitments:
-
Compiler boundaries: structured diagnostics, scoped recorder/teardown policy, the
opand syntax-layer splits and grouped pipeline state are delivered (issues #691–#695). A compilation session and further lowering-state extraction wait for a consumer. Module replay remains B19, not a wholesale frontend rewrite. -
Single-file script manifests, isolated dependency resolution, durable locks and argument forwarding are delivered. Warm executable hits also skip runtime extraction (issue #699). Broader executable-cache toolchain coverage remains B17/B18; a script test command would require its own entry-point/discovery design.
-
plenty-manager: a Plenty-written tool in a separateplentylang/plenty-managerrepository for compiler installation, explicit updates, side-by-side versions and project pins. Settle release authentication and qualify manager hosts before delivery. -
Language editions: wait for a concrete incompatible change; prove mixed-edition compilation with shared types/runtime before edition or migration tooling.
-
Rust crate dependencies: three prototypes compare Rust ABI, generated C adapters and Rust-source lowering. The next gate is real Plenty project integration with checked foreign owners and a coherent runtime build; generated C adapters are the proposal’s recommendation.
-
Local semantic search: improve chunk boundaries and measure retrieval quality before deciding on committed index snapshots. Prefer Plenty for the interface and record language defects in
issues.db.
Remaining performance work
B19 is evidence-driven; these are candidates, not an approved optimization batch. Use profiling, faster-linking evidence and the completed measurements as baselines.
- Generated code: retain/release for text operands computed by calls or loaded in another block; dispatcher copies of sums with inline-record payloads; code size and compilation/link latency. Revisit match density/size policies only with application/CPU evidence. An opt-in CPU target such as x86-64-v2 requires a decision against B17’s fixed-baseline reproducibility contract.
- Layout/cleanup: natural record-field packing beyond 16-byte slots, compact
sum tags/niches, relocation/release plans for records holding ranges/sums/other
managed fields, records over 192 bytes and tuple release plans. Measure inline
copies beyond current runtime-copy paths; owned temporaries and nonscalar list
rows still use runtime helpers. Flat-layout rationale.
Box size (issue #814): a box’s class-record slot still keeps a value word that
only addresses the record after it, and
Option[Box[T]]keeps a tag word rather than a null niche; dropping both would take a binary-trees node from a 96-byte malloc chunk to 64 (C: 32). A size-class free list for fixed-size boxes is unmeasured. - Serial frontend efficiency: module-scope name hashing, builtin-sum
interning and
Ty/nominal-handle clone traffic (about 10% of check samples with allocation excluded; issue #696). The signature map and nominal fact/lock costs measured below 2% and are not candidates. Whole-program table/cache misses still increase per-function cost at scale (issue #57); interned names or per-function batch processing need measurements. Further type-representation changes follow the completed issue #302 investigations. Measured attribution records the profile; existing frontend function symbols and type-fact caches are not missing features. - Peak memory: lower and emit one function at a time instead of retaining full syntax/operation sets at phase boundaries (issue #54). IR construction’s measured 7% of emission CPU limits scaling only above about fourteen cores.
- Caching/lowering: imported-interface lowering with explicit outputs and
serialized operations/types; body-derived invalidation and per-function
granularity for one-module programs (issue #317). Separately evaluate lowering
only reachable
stdfunctions. A trusted opt-in remote cache follows a local per-module contract, not before it. Prove fresh-environment replay using explicit module outputs before persistence. See deferrals. - Upstream gates: regalloc2’s quadratic block-parameter handling needs an upstream report/fix before removing the 2,048-argument fallback. Reopen issue #425’s deferred arithmetic-check work only when Cranelift changes unfused overflow branches, frame setup or callee-save spills; do not treat closed investigations as active claims.
Stdlib-driven language ergonomics
| ID | Remaining work | Scope / dependency |
|---|---|---|
| B71 | Error presentation | Benchmark-template migration after active comparisons, and separately scoped conveniences such as field-level redaction (for example URL userinfo, beyond shipped class/enum redaction). Case guards, residual-set inference, empty sums, arbitrary protocol forwarding and dynamic error objects remain deferred. Delivered composition/HTTP work. |
| B72 | State/ownership ergonomics | Explore typestate and runtime-state enums. Linear obligations, resolution scopes and integration boundaries do not approve implementation or blanket std conversion. Real HTTP integration of the owned-response experiment remains unexplored. Field moves through Box/Shared, per-element tuple tracking, partial destruction for __del__ classes and whole-object destructuring need contracts. Delivered subset. |
| B73 | Arenas/linked data | Before std.arena, replace caller namespaces with a safe arena-identity contract covering copy/concurrency/exhaustion, review callbacks/movement cost, and qualify deletion/reuse, bad handles, affine cleanup, OOM preservation and tree/graph consumers. Keep linked lists/trees as examples until a workload justifies an API; the linked-list lesson covers owned chains and index-based bidirectional links (delivery), and balanced-tree examples demonstrate B-tree/red-black insert/delete and invariant checks (delivery); the Unicode trie adds iterative prefix retrieval and pruning (delivery). Direct disjoint loans/worklists relate to B11; no intrusive-pointer extension implied. Assessment/prototype. |
| B75 | Pipeline extensions | Early stop for parallel any/all and integer range sources beyond range[i64]. Broader reusable iterator/view relationships are B12/B11. All four original stages are delivered; they are not outstanding tasks. |
| B76 | Columnar extensions | Decimal columns after the B86 core module; temporal/grouped/floating as-of keys, time-bucketed/rolling aggregations, fixed-size arrays, dictionary semantics, nested casts and broader list operations need workload evidence. Validate use cases before static-schema/zero-copy machinery. For clist: row views/method adaptation, field references, affine fields, chunks, statistics, tombstones/vacuum and interchange. Broader SIMD/parallel scans need legality/performance evidence (B23); borrowing B11/B12, pipelines B75. Delivered dataframe/clist subsets. |
| B77 | Derived serialization | Accepted design, not implementation: JSON-shaped std.serial; hooks/compiler-expanded entry points; class @serial; TOML; docs. SerialError[E] retains format error/field path. Measure compile time/throughput before derive. Relates to B29/B51; reuse the closed annotation mechanism. |
| B85 | Actor extensions | Extend the recorded actor HTTP qualification beyond x86-64 GNU/Linux. Matched normal/fault comparisons are recorded; actor WebSocket/SSE adapters remain; WebSocket uses B94’s shared session and driver/upgrade contract; blocking-work offload proposes bounded native workers, independent job ownership and completion delivery. The budget/stack probe validates native suspension with heap/trap restoration; compiler checkpoints, shared-context ownership, bounded stack admission, safe suspension regions, overflow/platform qualification and fairness/overhead measurements remain. Multi-thread stages; sliding-window/all-for-one restart, parent restart notices and an actor-specific &mut self spawn diagnostic remain. macOS heaps wait on B18. Actor HTTP design, delivered actors/network/TLS/HTTP. |
| B80 | Plotting | Finish the accepted core milestone: remaining scales, smoothing, broader facet controls and font-measured layout. Then broader geoms/PNG, dates, reflected records and a separate dataframe bridge. Delivered subset. |
| B83 | Inline arrays and value-generic extensions | Value parameters and Bounded are delivered. Remaining accepted design: inline array[T, N], allocation-free literals at generic parameters and per-instance assertions; then BoundedList (B13) and measured internal SIMD rewrite (B23). Derived sizes in signatures remain deferred. |
| B86 | Exact decimals | std.decimal proposal: allocation-free 38-digit Decimal in Plenty with exact operators and explicit rounding. Stages: u64.widening_mul, std-only @arithmetic methods/hooks, __cmp__ (B37) and a ZeroTest bound for NonZero; core module; __key__ key policy (reconcile with the delivered structural-key contract) and a format-spec hook; Polars decimal columns (B76) and JSON/CSV interchange. Arbitrary precision only on evidence. |
| B87 | Std conventions | Public std APIs follow Standard library conventions. Align the listed deviations as they are scheduled (booleans with B36, byte inputs with B39); file issues for the untracked ones: bounds: vs limits:, AllocError unions, predicate names and count() sizes, watch shapes. Delivered: ThreadPoolExecutor.cancel_pending() replaces shutdown(bool) (issue #631); every thread start fails with SpawnError, and owned starts return their job (issue #630); native hooks are std-only intrinsics taking &Bytes, so std.bytes has no public hook methods (issue #633). |
| B88 | Copy rule | @copy on enums; undeclared enums affine after a migration that annotates every currently copyable enum. Generic enums use B84’s conditional copying and CopyType bound. Delivered: one copy rule in the reference, cheat sheet and skill, checked by tests/test_copy_rule.rs (issue #636). Applied copy/Shared guidance is delivered in issue #810. Proposal. |
| B89 | Formatting path | Phases 1 and 2 done (issue #615): print/f"{x}" use display; str.repr/!r/aggregate elements use debug; private types format in their module; wrong-signature display/debug is a compile error. Remaining: main’s report uses display; the shared formatter needed for it is delivered (issue #634). Proposal; issue #615. |
| B91 | Iteration sources | Adapters only on iterators; iter() for every source; membership on std containers stays B49. Delivered: len method dispatch (issue #623); uniform local dictionary views (issue #624); find/reduce copy borrowed copyable items, and iterator methods on non-iterators suggest iter(...) (issues #621, #622). Proposal. |
| B93 | Callable forms | Delivered: reusable callable forms in serial pipelines and sort_by_key, and &mut named functions (issues #617, #619, #620); constructor values, including generic classes and Some/Ok/Err (issue #618); lexical checked callbacks with scoped failure propagation (issue #614). Remaining: par callbacks for stored capture-free closures and Callable values, including class and variant constructors. Callable names stay (issue #24). Proposal. |
| B96 | Bounded integer adoption | The Bounded[T, lo, hi] type, literal checks and checked runtime construction are delivered. Remaining: adopt it for std.http.Limits.max_line/max_head (>= 2) and fixed std ceilings. Existing ceiling metadata (issue #686) still checks static std inputs; dynamic values still trap under those APIs. |
Units of measure
| ID | Remaining work | Scope / dependency |
|---|---|---|
| B97 | Units plot axis labels | std.plot axis labels from quantity units. Deferred; design open, including choosing a display unit that matches the values’ scale (km rather than m for long distances). Candidate design: plot axis labels. Compiler dimensions, Quantity operations, formatting, constants, scoped diagnostics, native erasure, std.units with zero-copy wrap_list/strip_list, the std.units.catalogue of SP 811 units, and shared bodies for instances that differ only in dimension are delivered. Design: units of measure proposal. A Positive[f64] float domain is a separate follow-up. |
Runtime-independent protocol support
| ID | Remaining work | Scope / dependency |
|---|---|---|
| B94 | Sans-I/O networking protocols | Agreed codec/session/driver direction, not implemented as a general convention. Validate with WebSocket: extract bounded framing/session/handshake logic; migrate the existing blocking/WSS driver without silent behavior changes; add the B85 actor driver with explicit upgrade ownership, entropy supply, timers, backpressure and shutdown. Follow the session contract and acceptance gates; retain existing regressions and add deterministic/fuzz, real TCP/TLS and actor resource/fairness evidence. No universal protocol interface, new scheduler or wholesale HTTP rewrite. Broader protocols require separate scope under B38. |
Standard library follow-ups
Only outstanding extensions are listed here. Initial delivery records, issue/PR references and reference links live in Completed library work. B IDs keep their original scope; a closed initial issue does not approve a follow-up. All candidates still require scope and acceptance checks.
| ID | Area | Remaining candidates |
|---|---|---|
| B38 | Additional breadth | std.bigint (issue #481): Plenty implementation per the bigint proposal; needs u64 widening-multiply and add-with-carry builtins. XML namespace URI resolution and tree/query APIs; recursive filesystem watches/automatic reconciliation; accurate summation/norms, float decomposition/adjacent values and special functions (the numerics core proposal schedules summation, norms and special functions in its first milestone, with optimization, integration, ODEs, interpolation and FFT under std.numeric; it replaces issues #492 and #493); constant function calls and bytes concatenation. Linalg conditioning, least squares and sparse storage need separate scope. Image: GIF/LZW/frames, baseline/progressive JPEG/encoding and WebP/VP8/VP8L/RIFF, then 16-bit buffers, streaming decode, metadata, resampling filters and the measured 2–6× Pillow speed gap; implement in Plenty under the same Limits. Broader statistics need numerical contracts; sketches need mergeable quantiles/frequent items, stronger tail accuracy and workload-led xor/fuse tuning. System: cgroup accounting, CPU utilization sampling and filesystem free space. |
| B39 | Bytes | Stored/returned views (B11); OS strings in std.path, std.env, std.process and std.cli are list[u8]. Text APIs accepting &substr are B12. Dataframe byte-view inputs delivered. |
| B40 | Binary streams | Unscheduled nonblocking descriptors, vectored I/O, peeking and erased streams. |
| B41 | Error context | Native-cause OS description text (B14/B71). |
| B43 | Processes | Native process-readiness wakeups for deadline waits, file/child redirection, stderr merging, dropped-child reaping policy and process groups. Polling deadline/cancellation waits delivered. |
| B46 | Logs | JSON-lines sink, sampling/rate limits, per-module thresholds and span context. |
| B47 | Text/Unicode | Grapheme-aware text wrapping and broader classification; keep the external unicode_text package’s Unicode version aligned with std.unicode. Delivered paragraph tools. |
| B48 | Regex | Broader DFA/one-pass coverage (Unicode and assertions), remaining scalar scan lowering (issue #760); full Unicode case folding, script and general-category properties. Bounded ASCII accelerators delivered (issue #742). |
| B49 | Collections/iterators | User iterator protocol (B12, needs B11 stored references); in on user and std containers; len() method dispatch delivered (record, issue #623), Python __len__/__contains__ names remain reserved (issue #645); further collections need use cases. |
| B50 | Random/secrets | Jump/split streams, distributions/float ranges, weighted choice/element sampling, scratch-free secret integers and a zero-on-drop buffer. |
| B51 | JSON | Bulk scans outside string runs, pretty-printing, JSON Lines and derived codecs (B77). |
| B52 | TOML | Independent conformance checks (toml-test/reference crate); evaluate TOML 1.1. |
| B53 | CSV | Lenient parsing/recovery, header mapping/typed columns, non-UTF-8 encodings, multibyte delimiters and throughput benchmarks. |
| B54 | Encodings | Bulk native codec/packing throughput, streaming codecs and varints. IEEE float packing delivered (issue #656). |
| B55 | Filesystem | Advisory locking delivered (issue #774); explicit child-fd inheritance remains issue #777. Passwd fallback, XDG_RUNTIME_DIR and XDG_*_DIRS search lists. |
| B56 | Concurrency primitives | Condition variables, rwlocks and semaphores only with a concrete use case. |
| B60 | URLs | Relative resolution, normalization/equivalence, IDNA and form encoding. |
| B63 | HTTP client | WebSocket permessage-deflate and proxy tunnels; cookie jars/content decoding were declined, not scheduled. |
| B64 | HTTP server | SSE client reader; waking SSE streams from channel messages rather than polling. |
| B65 | HTTP helpers | Static-file Range requests and directory indexes. |
| B66 | HTTP qualification | Generated route dispatch without user match waits on enum methods or existential protocol values; preserve existing qualification suites and proxy fixture ownership (issue #719; delivered scope). |
| B67 | Crypto | P-384/JWT ES384 and SHA-384 HMAC/JWT HS384; password KDFs need separate use-case/API review. |
| B68 | Compression | Gzip header access, zstd and trailing-zero tolerance; archives remain B69. |
| B69 | Archives | PAX decimal range correction delivered (issue #705). Remaining: policy-controlled link creation, mode/time restoration and streaming ZIP writes with data descriptors. |
| B70 | SQLite | Runtime allocator integration for accounting/failure injection, user-defined functions and backup API. |
| B78 | Diffs | Line-first pass for char/word diffs and diffs in std.test failures (B57). Performance baselines belong to B19. |
| B79 | Markdown | GFM tables, strikethrough/task lists and parser throughput beyond the recorded ~1 MB/s. |
| B82 | Graphs | Weak components, Prim/Floyd–Warshall, flow/matching, compaction, non-f64 weights and () type arguments. |
Reconciled completed work
The delivery records and retired entries moved to Completed work. This heading remains as a link destination; completed-only B IDs no longer appear as work to select here.
Keeping this list useful
- Add, prioritize or split future work here. Keep IDs stable and link design detail rather than copying queues into proposals, tutorials or reference pages.
- Settle a candidate’s scope and acceptance checks before implementation. Table order does not approve APIs or require every extension.
- Before syntax/builtins, state the user problem, why ordinary library code is insufficient, and ownership, allocation, compilation and teaching costs.
- On completion, move the delivered scope to Completed work, update reference/status/runnable lessons, and leave only genuine outstanding extensions here with a link to their delivery record. Do not mark an entire mixed item done because its initial slice shipped.
- Issues, claims and review state stay in
issues.db; reference pages describe implemented behavior. Use the development workflow for ongoing testing, documentation and performance practices. - Historical proposals may contain old syntax/dependencies; introductory notices must say so. Future priorities belong here, not on completed-work pages.
A JIT and an interpreter remain out of scope, not waiting at the bottom of this backlog. Async/await remains unadopted and unscheduled under B22’s investigation gates.
B18 — macOS and Windows implementation sequence
The portability design records rationale and toolchain evidence; this section owns the work sequence. Target ARM64 macOS and x86-64 Windows, preserving both Linux architectures. Use isolated child worktrees based on the portability branch, with component ownership agreed before concurrent edits. No local Mac/Windows machine is assumed; provision CI execution before treating either port as tested.
| Work package | Dependency / owned components | Completion evidence |
|---|---|---|
| B18.1 — backend and toolchain decisions | Standalone probes; pinned Cranelift, Zig/cargo-zigbuild, C/Rust adapters; compare Windows MSVC and GNU | Real Mach-O/COFF output and native CI execution of mixed ABI/Tail/large-frame fixtures; record SDK/CRT/deployment choice and unwind strategy. Signal restart qualification no longer assumes sleep-based reader scheduling (issue #689). Cross-link success alone is insufficient. |
| B18.2 — target and host separation | B18.1 findings; toolchain, frontend facts, cache/runtime identity | One target description, explicit build-host/compiler-host/output-target identities; incompatible combinations rejected; existing Linux suites pass. |
| B18.3 — object and linking interfaces | B18.2; object metadata/exports/source/trap/coverage records, drivers/distribution | ELF implementation preserved; Mach-O/COFF metadata, retention, exports and relocation round trips pass; no GNU options leak into other drivers. |
| B18.4 — dependency and runtime builds | B18.1–2; build.rs, native sources, test hooks, engine helper | Both runtime modes and native dependencies match target; Cargo and embedded builds agree; choose direct-build versus orchestrated Cargo using a measured prototype; no recursive Cargo lock contention. |
| B18.5 — runtime core | B18.3–4; memory, errors, entropy, threads, mutex/parking, signals/exceptions, actor recovery | Minimal native Plenty program, C callbacks, fallible allocation, secure randomness, stack protection and recovery tests pass on each OS; no success stubs or hidden allocation regressions. |
| B18.6 — OS contracts and services | Design decisions begin alongside B18.1; implementations depend on B18.5; paths/resources/files/process/readiness/watch | Lossless names, full process status, rooted filesystem security and resource-aware cancellation tests; explicit watch/shutdown capabilities; separate compatible portable contracts from Linux-only APIs. |
| B18.7 — library parity | B18.6; networking/DNS/TLS/HTTP, SQLite/KV, crypto/compression, dataframe/linalg | Real native execution, including both optional engines and CPU fallbacks; count skipped/early-return opt-ins separately; publish exact capability gaps before any preview. |
| B18.8 — developer workflows | B18.3–6; package CLI/cache/LSP/fuzz/export tools and tests | Extracted installation handles spaces/Unicode, literal arguments, replacement races, diagnostics and library hosts; explicitly report unqualified fuzz/sanitizer paths. |
| B18.9 — release admission | B18.7–8; native CI, packaging, notices, minimum-version audits, signing policy | Offline clean-machine compilation with documented prerequisites; tested target/engine artifacts, baseline OS/SDK/CRT evidence and signing/unsigned-preview policy. Preserve Linux ARM’s outstanding hardware/release gates. Experimental macOS ARM64 (.tar.gz, macOS 11+) and Windows x86-64 (.zip, Windows 10+, dynamic MSVC CRT) archives include their dataframe engines and notices, with native unit tests, executable/library audits and clean-runner Parquet/application-relocation smoke tests. macOS uses ad-hoc signatures without notarization; Windows remains unsigned. Native execution evidence still depends on CI, including qualification after the engine version-lookup fix (issue #687) and unused native-wrapper and executable-helper pruning (issues #688 and #690) and native distribution executable suffixes (issue #697). Remaining: library-parity smoke tests (B18.7), project workflows (B18.8), signing/notarization. |
Current evidence: the native CI smoke suite passes on macOS ARM64, Windows x86-64 MSVC and Linux x86-64. The first B18.2 increment centralizes target facts and compiler-host/runtime validation; it does not implement new object formats or OS services. This is a partial B18.1 gate: large frames, unwind, Rust boundaries and runtime dependencies still need qualification. Both new OS targets are experimental; neither is release-qualified. The shared Apple archive-command fix addresses issue #600; native build-tool tests run before the OS port build and on both Linux architectures. The first B18.5 increment separates native mutex waiting and thread handles, hides cancellation wake storage, and adds Darwin signal-context and stack-handler support. Standalone native synchronization tests run before the complete compiler build. Windows exception dispatch and per-thread stack guarantees are implemented with standalone subprocess tests that pass on the native Windows runner. Native OS entropy fixes dictionary initialization on non-Linux hosts (issue #604); Windows no-replace bundle publication and portable graph fixtures address issues #605 and #606. The initial B18.3 increment enables experimental Mach-O/COFF application output, native C conventions and linker syntax, with framed Mach-O interface metadata. Native qualification now passes full compiler/runtime builds, unit tests, platform-boundary checks and generated-program callback/result probes on macOS ARM64 and Windows x86-64. Linux x86-64 and ARM64 CI also passes all integration shards, both optional engines and Docker proxy tests. Full new-OS standard-library parity and release admission remain gates; these results do not complete B18.5–9.
B18.1 and B18.6 design research can run together; B18.3 and B18.4 can proceed in separate worktrees after the common description settles. Keep independent OS implementations separate after their contracts are agreed. A failing native probe changes the design before target admission. User-facing cross-compilation, Intel macOS and Windows ARM64 remain separate decisions, not implied outcomes.
Completed work
Delivery history and retired backlog entries live here, separate from the backlog, which is the single maintained list of future work.
- Language and tooling: language foundations, ownership, calls, value parameters and bounded integers, project tooling and concurrency.
- Checked-default validation: branch-scoped semantic/native qualification, controlled measurements and limits.
- Libraries and applications: shipped library slices, HTTP acceptance, dataframes, plotting and actors.
- Compiler performance and SIMD: delivered optimizations, measurements and investigations that were closed or deferred.
B IDs are stable. A record here means the described subset shipped or its investigation concluded; it does not mean every conceivable extension of that B item is implemented. Mixed items keep their remaining scope only in the backlog, linked from these records. A completed investigation may have rejected an implementation rather than shipped one.
These are concise delivery records, not a second reference or a future-work
queue. The reference owns current behavior and limits;
implementation status summarizes
support. Local issue/PR numbers identify records in the shared issues.db, not
GitHub records. Reproductions, claims and review state remain there.
Reconciliation history
The backlog’s full reconciliation on 2026-10-22 covered local main and merged
local PRs through #477, reference/tutorial content and the shared issues register.
Compiler/columnar follow-ups were checked on 2026-10-24 through #531. Later
merges added the delivered subsets recorded on these pages. The split from the
active backlog was made on 2026-10-31; it does not independently requalify every
historical claim or declare all remaining extensions complete.
B34’s bounded cleanup/review, B71’s typed HTTP acceptance, B76’s initial numeric, binary and nested dataframe expansion, B17 package tooling and the initial B39–B70 library slices have landed. Their acceptance checks remain ongoing regression requirements, not unfinished milestones.
Maintaining the split
When work ships, update its reference, implementation status and runnable lessons. Add a short delivery record here with issue/PR evidence and a reference link. Remove the delivered scope from the active backlog; if an item still has extensions, retain its ID there and link both ways. Do not copy outstanding task lists into these pages or recycle retired B IDs.
Typed time and cancellable waits
Delivered B42.
Issue #635 aligns database busy timeouts, JWT validation, cookie lifetimes,
HTTP dates and SSE retry hints with Duration and Timestamp. WebSocket, SSE,
readiness and child deadline waits accept caller cancellation while preserving
server shutdown behavior. std.sync
provides typed channel, future, token and two-channel selection waits; explicit
millisecond primitives remain as low-level compatibility APIs (B42).
Completed language and tooling work
Completed-work index · Remaining work
These records describe delivered subsets and concluded decisions. References own current contracts; extensions of a mixed B item stay in the backlog.
Package check coverage (issue #805)
plenty check now warns about each .plenty file below a package’s src/
that no target imports, and ends with a summary of the targets checked and the
src/ files they reached. Check coverage
documents the rule: fixtures and other unbuilt sources belong outside src/.
Coarse check timing reports
Local check reports retain full-suite compilation/execution timings and local CI step durations, including failures, as Markdown and JSON. CI reports summarize Actions jobs and steps automatically and support historical run comparisons, keeping parallel runner time separate from elapsed span.
Embedded-runtime dependency reuse (issue #829)
Embedded-runtime Rust dependency caching retains the 27 fixed vendored rlibs across runtime edits and feature-only rebuilds, with source/toolchain content keys and artifact validation. Test shards restore these dependencies independently of the compiler workspace cache. Native C compilation and both runtime archives still rebuild; this change does not eliminate the remaining build-script cost (tracked in local issue #832). A local probe reduced the vendored stage from 18.95 to 0.46 seconds; whole-build timing was inconclusive because the remaining stages varied. Hosted savings have not been measured.
Native-engine dependency caches (issue #827)
Optional-engine CI caches use separate compiler and engine target directories. Compiler cache cleanup can no longer traverse the engine target and remove its Polars or faer dependencies. A new engine cache key repopulates the corrected layout; actual hosted cache-hit savings still need measurement. Local engine-build defaults and installed-library paths are unchanged.
Linux release glibc verification (issue #730)
The v0.8.9 artifact audit confirms the published glibc 2.28 baseline for both Linux architectures, including both optional engines. It separates the ARM build tool’s 2.30 requirement from release requirements and records baseline execution and qualification limits.
Agent skill refresh
The Plenty agent skill now distinguishes trapping and assumed unchecked arithmetic, routes readers to boxed protocols, structural collection keys and quantities, and uses the signature index’s module navigation. Published skill assemblies include the language reference so those contracts remain accessible outside the compiler checkout. Issue #810 adds task-oriented capability discovery and a versioned gap-verification workflow, with executable typed partial-error and cancellable-shutdown examples. Applied examples distinguish string assignment, explicit class copying and nested Shared ownership, and package guidance covers module privacy, one-way imports, cycles, target/test coverage and current editor limits. Publication bundles the linked concurrency and editor setup contracts as well as the existing API pages. Issue #738 recalculates signature-index module ranges after publication transforms, so line-based reads select complete sections in the installed skill too.
Experiment arithmetic migration
Local issue #737 updates stored and generated experiment inputs for checked integer arithmetic. The fannkuch kernels propagate overflow errors; infallible Rust ABI comparisons retain explicit trapping suites. The experiment smoke check covers source inputs, expected rejections and generated fixtures without claiming to refresh historical performance measurements.
Optional-engine check build reuse (issue #828)
Engine CI checks
and just ci dataframe linalg use runtime-checks consistently for module
tests and runnable tutorial examples. The sales report uses the CLI already
built for the integration tests. These checks reuse one feature configuration
and avoid a separate sales-report compiler build. Allocation-failure checks
remain enabled, and the
dedicated default-feature build retains its separate coverage.
Range-trap fixture reuse (issue #830)
The I/O range and fixed-width encoding trap tests compile 18 reusable fixtures instead of 154 separate programs. Each invalid input still runs in its own child process and checks exit status and the operation-specific trap text. Argument dispatch selects among the original literal call sites, preserving constant-input coverage, empty-input checks and reader/transport variants. The fixtures retain optimized compilation and runtime checks.
B83 value parameters and B96 bounded integers
Value parameters specialize functions,
methods, classes and enums on integer and boolean constants. Resolved values
participate in nominal identity and structural inference; typed substitution
makes them available in bodies without runtime fields. A const parameter can
use a preceding T: IntType, supporting signed and unsigned bound widths.
Bounded[T, lo, hi] checks inclusive
bounds at instantiation, admits in-range literals in expected contexts, and
provides try_from returning RangeError for computed input and get for
explicit extraction. It reuses private copy-class storage and ownership.
Regression tests cover widths, boundaries, aliases, imports, privacy, generic
functions/methods/enums, inference, body forwarding, expected literal contexts,
copying and moves. The runnable tutorial teaches validation at input boundaries.
This slice uses Bounded as the first consumer rather than requiring arrays.
Remaining B83/B96 scope stays in the backlog.
B97 compile-time dimensions
Dimension declarations and algebra
provide nominal bases, transparent derived aliases, One, normalized products
and quotients, integer powers, and exact roots. Dimensions are interned types
without values and are admitted by the Dimension generic constraint.
Generic input expressions remain structurally inferable; output/body expressions
are checked per instance. Tests compare random expressions against a dense
reference evaluator and cover exponent boundaries, imports, privacy and generic
diagnostics.
The second stage of issue #701 adds Quantity core:
coherent-value construction, get, float to/from, domain try_from,
dimension-aware operators and compound assignments. Numeric widths, domain
preservation/erasure and checked failures use existing arithmetic rules.
Tests cover storage/result dimensions, rejection rules, evaluation order,
collection/reference destinations and allocation denial.
The third stage adds coherent-unit display and numeric format specs, normalized
quantity debug output in aggregates, and writer methods through std.fmt.
Unit spellings are static metadata, with no per-value dimension tag. Tests cover
numeric widths and domains, normalized symbols, conversions, nested specs,
writer limits and allocation denial.
The fourth stage adds compile-time Quantity module constants: construction, quantity arithmetic, coherent-value extraction and float unit conversion. Dimensions and storage domains survive evaluation; overflow and invalid constant expressions are compile errors. Imports, forward references, static assertions and skipped branches follow the existing constant rules.
The fifth stage adds scoped dimension-alias notes to type mismatches, quantity operators, constant checks, generic inference and dimension errors. Notes select the first visible matching alias in declaration/import order, respecting import bindings, visibility and generic-parameter shadowing. Canonical type identity and runtime formatting remain unchanged; tests cover nested types, defining-module scope, editor checks and concurrent compilations.
The sixth stage completes issue #701’s native erasure: Quantity parameters and
results use their storage type’s ABI, construction/projection are direct, and
unit constants substitute folded values. The x86-64 release probe produces
identical instructions to plain f32/f64 for addition, multiplication and
conversion by a parameter or constant unit. Tests cover numeric bits across
calls, references, aggregates, closures, generators and workers in both modes.
Domain-backed quantity aliases also resolve safely before storage definitions
are finalized.
std.units is embedded Plenty: the seven SI base
dimensions, distinct derived aliases, prefixed SI and common exact units, the
exact SI defining and CODATA 2022 constants, and dimension-aware sqrt,
cbrt, hypot, sum, mean, abs, clamp and std.math wrappers. Celsius,
Fahrenheit and std.time.Duration convert at the boundary; to_duration
rejects negative, NaN and out-of-range times. Symbols print in declaration
order within a module, so std.units quantities read kg*m/s^2.
wrap_list and strip_list convert between list[f64] and
list[Quantity[f64, D]] by replacing the list header’s type descriptor, with
no copy or allocation.
Instances that differ only in dimension
share one body:
the object places identical code once, for any function not used as a value.
Quantity[T, D](x) written with explicit type arguments, as generic bodies
must, now retags its value as other constructions do, instead of calling the
constructor; that made std.units generic bodies such as sum and hypot
identical across dimensions.
std.units.catalogue adds every SI
prefix on every SI unit and the other named units of NIST SP 811 appendix B,
built from their definitions and tested against SP 811’s factors. It is a
separate module so its 768 constants cost compile time only where imported.
B97 retains plot axis labels from units.
Heterogeneous collections tutorial
The heterogeneous collections section follows classes and enums. Runnable lessons teach enum wrappers with shared methods and anonymous-sum lists, sets, and dictionary keys and values. The lessons show mixed keys, mixed values, and both together, while preserving distinct member identities. Separate typed collections remain an alternative for independent lookup domains. Lessons also cover custom-cleanup keys, transferring keys and borrowing views without duplicating cleanup obligations.
Unicode trie example
The runnable trie stores exact
Unicode strings, including the empty key, with insertion, lookup, pruning
deletion, sorted prefix enumeration, and longest-prefix matching. Private
parent/sibling IDs support iterative operations and slot reuse. Model/invariant
checks cover permutation pairs, Unicode mixed histories and 256-way sibling
lists. Retained-result checks drop original key/query/trie owners before reuse
pressure and comparison. Allocation-failure sweeps cover insertion and prefix
results, and a Linux small-stack test handles
a 100,000-scalar key. This is a teaching set, not a std API, compressed trie,
or normalization policy. B73’s general arena qualification remains outstanding.
Balanced-tree examples
Executable B-tree and left-leaning red-black sets
implement integer-key insertion, lookup, deletion, ordered traversal and private
slot reuse directly in Plenty. A sorted-list model and independent structural
validators check exhaustive five-key permutation pairs, multi-level trees and
mixed histories. Runtime allocation-failure sweeps check preservation and retry;
deletion and reuse are checked with allocation disabled. These are educational
examples, not new std APIs or storage-engine performance qualification.
B73’s general arena identity/deletion qualification remains outstanding.
Linked-list tutorial
The linked-list lesson demonstrates owned singly linked push/pop and mutable traversal, then index-based doubly linked append and bidirectional traversal. Executable examples preserve existing links on allocation failure. Deletion, slot reuse and public handle validation are explicitly outside this teaching implementation; B73 remains unchanged.
Documentation interpolation colours
The default Ayu documentation theme colours f-string interpolation expressions and their braces cyan, distinct from surrounding string text. Nested tokens retain their syntax colours; other themes are unchanged.
B11: Borrowing
Borrowed enum matching/payload loans, reference rebinding within the original
place, precise first-reference return origins and disjoint literal-index loans
landed (issues #1/#30/#41, local PRs #488/#490). Optional/fallible returned
references (Option[&T]/Result[&T, E]) shipped as phase 1 (issue #520,
local PR #622). dict.get_mut replaced dictionary indexed assignment
(issue #580, local PR #696); heap and graph reference getters shipped
(local PR #629).
Local dictionary key/value views (issue #396), list/byte slice views
(issue #581), mutable Bytes loops (issue #594), proven disjoint slice/loop-field
loans (issue #593) and scalar-indexed &substr substring views (issue #595)
are implemented. Methods on owned temporaries can lend within the statement
(issue #597), calls returning mutable references can be assigned through
(issue #598), and method receivers projected from temporaries borrow in place
(issue #599). len also normalizes explicit shared borrows of literals and
owned temporaries, preserving full-expression cleanup without allocating for
the borrow itself (issue #683).
See borrowing, returned references, optional references and slices/views. Outstanding relationships remain B11 in the backlog.
B49: Length method dispatch
len(value) calls an accessible ordinary shared len(self) -> i64 method on
classes and enums, including standard Deque, Bytes, and heap containers
(issue #623). Owners, references, and temporaries follow ordinary method
borrowing and cleanup. Unsupported length and membership diagnostics name the
actual operand type rather than claiming every iterable supports the operation.
See the collection contracts.
User-container membership and iterator protocols remain B49 in the
backlog; Python __len__ and __contains__ names remain reserved.
B12: I/O and iteration
Text-file iteration, uniform borrowed iteration and borrowed temporary source
lifetimes shipped (issues #30/#75). Allocation-free enumerate/zip/reversed
sources and boolean reductions (issue #35) now compose through B75 pipelines;
free any/all were removed (local PR #562). Binary buffering is B39/B40;
lossless Unix arguments/binary I/O (issue #61), native Linux std.path and
local dictionary views are delivered. The finder exercises NUL-separated input
and output with --null; directory traversal/metadata shipped for issue #60.
See iteration and native paths. B12 in the backlog owns reusable/escaping views and portable paths.
B14: Errors
General anonymous sums preserve
payloads, normalize membership and support injection/widening. Precise errors,
? and explicit success values replaced the former try/try_ allocation
conventions; Failure is a last resort (issue #342), not the recommended default.
Option.unwrap_or and ok_or shipped as part of issue #35.
Sum fallback conveniences
(issues #800/#820) add Result.unwrap_or, lazy unwrap_or_else on both sums,
Option.ok_or_else, and map_or/map_or_else on both sums. Owned consuming
callbacks preserve conditional evaluation and cleanup; returning a Result from
both mapping branches handles a fallible fallback without special propagation.
Both sums also provide message-bearing expect with caller-located traps. B71 records typed
presentation and Result.map_error/variant mapper values (issue #556).
Arithmetic hooks shipped for
classes/enums, including reflected and explicit in-place methods (issue #512).
One-field @newtype forwarding preserves nominal typing and explicit-hook
precedence (issue #555). checked: blocks
return typed integer overflow/zero-division errors for arithmetic written in the
block; float div_checked and nontrapping MIN % -1 accompanied them
(issue #557). The lexical private callback channel from issue #614 (B93)
is superseded by independent declared function/closure error contracts in the
checked-default compiler core.
That core also supports lexical unchecked: suites. Their
build modes followed
(local PR #924): --unchecked trap checks the preconditions by default,
--unchecked assume omits the checks outside std, and unchecked trap:
always traps; caches key on the mode. Issue #764 extends assumed suites to list,
slice-view and Bytes index reads, writes and borrows: indices must be
nonnegative and in range, and both bounds checks and negative normalization
are omitted. Trapping scopes, other indexing and std keep their contracts.
The private
callback channel is removed. Initial std migration covers math helpers, bytes
and time: integer reductions preserve typed overflow payloads, separate float
helpers retain infallible IEEE values, serial pipeline sums return public
results, and key sorting transports fallible callbacks’ ordinary errors.
Encoding sizes, stream extents, formatting counters, filesystem budgets and
entropy-source offsets now preserve their existing error contracts or use
bounds-proven opt-outs. Filesystem walks reserve only newly appended frames
(issue #702); an allocation-failure regression distinguishes failure before
reservation from failure while scanning a popped sibling. Coordinated adoption
of remaining std modules, tests/examples and documentation is not complete
and remains B14.
Core HTTP decimal/chunk sizes, networking/readiness, bounded text operations,
deterministic random helpers and testing streams are migrated. HTTP accepts the
full representable Content-Length range without overflowing an ASCII-digit
intermediate (issue #703); larger lengths remain protocol errors. HTTP and DNS
reply-buffer extents report existing allocation-capacity errors before consuming
another chunk. FakeStream.fail_after now returns a precise overflow result and
preserves the previous schedule on failure. Counter exhaustion returns stream
OutOfRange before mutation. Representation-limit probes exercise the actual
private helpers without huge allocations, and RNG golden/reference sequences,
text-wrap compatibility/allocation tests and testing-toolchain checks pass.
URL output sizing, process request/output extents and incremental HTTP decoder arithmetic are migrated. URL expansion checks its final representation before allocation; terminal scans stop at the input endpoint. Process argument encoding reserves only additional bytes (issue #704), with zero-allocation spare-capacity and failed-mutation tests in debug and optimized output. Existing allocation causes remain intact. The process and readiness suites pass; incremental decoder parity tests pass apart from the still-unmigrated actor integration.
Typed HTTP framing and client arithmetic now preserve existing typed transport causes and progress. Complete chunk wire extents are checked before delivery; representation failure is OutOfRange, separate from invalid transport counts. Boundary tests use an affine cause type without constructing huge buffers and verify allocation-free arithmetic in debug and optimized output. Client deadline saturation, bounded counters and callback independence are explicit. The I/O range integration and typed composition/prototype examples pass; those cross-module dependencies were resolved by the coordinated checkpoint below.
JSON position arithmetic now reports the existing InputTooLarge fault before advancing a failing byte or ASCII-run cursor; input/output budgets and partial output progress remain intact. Unicode normalization arithmetic has explicit scalar-ceiling, successful-append and positive-index bounds. Both public APIs are unchanged. The component’s 35 runtime-check tests pass on an isolated build target, including optimized/unoptimized boundaries and the Unicode normalization corpus; independent review found no defects. A separately reviewed TOML/regex checkpoint adds 40 passing component tests and preserves parse errors, document edit atomicity, regex work/memory limits and replacement error precedence. Public Match lengths now expose typed subtraction overflow; TOML location exhaustion uses InputTooLarge with unavailable zero coordinates at the valid offset. XML/CSV and broader adoption were not yet included at that checkpoint.
The coordinated numerical, transport, server and actor checkpoint is integrated with independent source reviews. Ordinary integer SIMD operations return lane-typed overflow results; floats and explicit wrapping operations retain their contracts. Complex powers cover the full signed exponent range; matrix construction/indexing and statistics keep their typed errors. Diagnostic logger, server-report and actor counters saturate under the user-approved policy. Exact streamed progress, allocation extents, crypto/DER/UUID and JWT time bounds retain existing errors rather than hiding overflow.
Actor identity generations never wrap: exhausted slots and clock records retire. Mailbox rebasing preserves every target-owned child-exit fence and prefix order; affine callback arithmetic errors reach the parent without leaking actor heaps. Combined targeted runtime-check validation reports 733 passes, two failures in still-unmigrated HTTP body/cookie dependencies and one pre-existing ignored test. Full crypto, JWT, HTTPS, standalone server/client and actor TLS tests now pass together. The optional native-engine linalg bodies were initially unrun; broad documentation examples, remaining modules and full measurements were not yet included at that checkpoint.
Reviewed CLI/progress arithmetic now preserves exact work counts and output-failure state; the countbytes example exposes its aggregate overflow contract. Dataframe extents retain allocation-capacity causes, and plot counts/layout/rendering have explicit bounds without changing numerical aggregates. All 48 plot tests pass; the initial 13 dataframe native-engine self-skips were later replaced by real engine execution: all 17 dataframe and nine linalg tests pass, including native relocation and allocation cleanup after fixture-only arithmetic fixes. HTTP body/cookie helpers preserve semantic TooLarge errors and multipart/upload ownership. Actor HTTP has equivalent bounded lingering state without clamping actual consumed bytes; its 26 TCP/TLS/control/error tests pass. Remaining HTTP core failures still reach websocket. A reviewed proxy/helper checkpoint restores all 19 helper tests, preserving descriptor/cancellation/RFC/bulk-scan assertions. XML/CSV arithmetic preserves exact positions and explicit pending-output flushing at representation exhaustion. Temporary/system/watch/SQLite wrappers and compression retain their domain errors and resource/progress contracts; real procfs/inotify/SQLite and gzip/Brotli interop tests run. Actor HTTP interop retains curl and all 600 corpus mutations. Optional sqlite-vec and Docker cases remain unrun at this checkpoint.
The final reviewed std arithmetic checkpoints cover archive, HTTP files/SSE/
WebSocket, MIME/template/markdown, sketches and diff/image. Public mutable diff
lengths return overflow Results. File metadata above i64.MAX is rejected before
conditional validators (issue #706); all 11 file adapter tests now pass after
MIME integration. The combined final component selection has 143 scoped passes,
followed by image 23, Markdown 10, math 23 and registry/index 2 passes. The index
reports no remaining diagnosed module. New byte-ring PNG filter operations are
explicitly wrapping; parser columns, numerical sketch totals and semantic output
progress remain checked rather than saturated. Independent review prompted a
Markdown source-comment correction and an image OOM fixture change to exercise
conversion rather than copying; both were applied and validated.
These are focused checkpoints, not full-suite/native/export/performance acceptance.
Reviewed parallel integer sums now expose precise first-overflow Results after
workers drain, preserving serial-prefix behavior and callbacks-once semantics.
All 30 pipeline tests pass. The initial implementation retains O(n) numeric values
and combines them serially; no performance improvement is claimed. Generated
split/count helpers have source bounds and exact extent errors; enumeration keeps
its explicit trapping boundary.
Reviewed early/advanced lessons, reference and skill examples use public callback
Results and explicit worker error handling. The reviewed generic deque, heap,
graph and LRU arithmetic checkpoint preserves public declarations and exact
allocation/ownership contracts; its 49 targeted tests pass. Full executable
documentation now has all 20 tests passing, rather than the initial 3 / 17.
Reviewed fallible tree/pipeline reducers preserve ordinary callback Results,
typed worker-error ownership and deterministic selection; 24 parallel and 37
parallel-pipeline tests pass. Pair adapters reconstruct only the existing
two-field split and retain affine field/error cleanup. Registry success alone did not cover these
generic instantiations. Native export/FFI fixture adoption preserves scalar and
fieldless error ABIs, C/C++ guard-page assertions and exact resource cleanup;
its 23 component tests pass. Repository fixtures/examples, full native and
all-features validation were not yet complete at that checkpoint. An earlier all-features
baseline compiled successfully but reported 4,465 passes, 1,109 failures and
eight ignores; 27 confirmed self-skips are included in the reported passes.
That snapshot predates the generic/native follow-ups and is not a current pass.
A newer complete gate at ff52bd7e compiled with 4,813 reported passes, 813
failures and eight ignores, including the same 27 optional self-skips. No failed
outer name is new relative to the older baseline, and no std-source diagnostic
remains. Reviewed borrowing, tail/stack, control/trap and flow/shared fixture
batches subsequently pass all 568 parent-run cases, including 258 identical
failed names from that snapshot. No fresh whole-suite pass is inferred from
these scoped results. Reviewed concurrency and collection fixture batches add
183 and 428 passing parent-run cases with native worker/guard-page/ownership
oracles retained. The pre-existing propagated numeric-literal context defect
(issue #710) is repaired through a non-evaluating success-payload hint. A rebuilt
main baseline rejects the original source; normal/release native regressions and
the unchanged unsuffixed-literal callable oracle now pass. Parent checks cover
all 109 callable and ten literal-form cases. One unoptimized range-generator
oracle was precisely updated with explicit user approval to require its bounded
helper and caller-provenance calls; this does not establish zero-call or
performance equivalence.
Reviewed generalized initializer errors preserve exactly the declared E, including
generic whole-result parameters, aliases, named defaults and constructor values.
Construction adds no AllocError implicitly. Partial initialization and affine
error/default ownership are tested in both native modes; all 114 constructor/
checked-default cases pass. Issue #709’s partial-self closure-capture guard is
integrated, while destructor/context exits remain infallible and C-export error
layouts are unchanged. Generator/serial-pipeline fixture adoption has 184 passing
cases with suspension, fusion, allocation and drop oracles intact; its corrected
imported-callback test separately proves inlining and isolated cold/warm/edit cache
behavior. HTTP qualification retains real TCP/TLS/curl/corpus/installed-toolchain
assertions, with 15 passing cases and precise aggregate-overflow context. These are
scoped checkpoints, not a substitute for the red whole-workspace baseline.
The historical af8aa15d gate had 5,375 reported passes, 259 failures and
eight ignores. After the reviewed final fixture and example batches, the complete
host gate at 2ef36dce passes 5,650 tests with zero failures and eight existing
ignores. All 259 exact failed names now pass, along with sixteen new outer tests;
all 27 known optional self-skips execute their bodies. Real native dataframe,
linalg, tutorial engine, Docker, libFuzzer and Neovim execution is recorded.
Existing external engines were not rebuilt; transient host-tool conditionals are
not exhaustively traced. Two selected ignored trials pass separately. This green
snapshot does not establish performance acceptance or validate the pending shared
Unit checked-suite repair (issue #712). Reviewed shipped examples and structure companions pass 20 structure tests and all three
exhaustive-example tests. The explicitly run sqlite-vec transaction/inference
preservation trial passes with deterministic test inference, not a real embedding
model. Local skill assembly passes 19 sync tests and 152-page/837-target link
validation; no publication, agent trial or performance acceptance is inferred.
A native adapter audit found and repaired inline sort-error ownership (issue
#711): typed caller-owned storage now receives the payload before the comparator
returns. Reviewed native range/managed-error and layout regressions pass in both
modes without heap allocation; all 21 checked-default cases pass after parent
integration. Native probes did not reproduce wrong output before the repair;
the pre-repair structural regression distinguishes the lifetime/layout fix.
The independently reviewed Unit checked-suite verifier repair (issue #712) is
also integrated: Unit success preserves its entry type stack, while nonunit
behavior remains unchanged. Five verifier tests and four native both-mode tests
cover mutations, cleanup, propagation and floating division. A new comparison
baseline fork contains the identical production repair, explicitly labelled
“main + #712 fix”; final common-fix build and full-gate validation remain separate.
Reviewed call-retention probes preserve their arithmetic and error oracles and
require actual optimized native leaf calls, not source-policy inference.
Independent final-pin untimed review accepts all 124 check/native-build
configurations, 284 exact runtime cases and six optimized native call gates on
the common-fix baseline/candidate pair. The full 1000/4000-unit corpora compile
and execute their designated checksum entrypoints in all four modes and both
generated profiles. All 24 cleanup outcomes pass; four C/I zero-allocation
intervals have four genuinely allocating controls. These bounded calling-thread
checks do not establish allocation totals, timing, size or whole-suite acceptance.
Reviewed checkpoint continuation observes all 5,667 registered pairs across
stages: 5,659 passes and eight normal ignores, including all 27 optional bodies.
The separate uninterrupted post-#712 run completes both zero-test doctest packages
but fails one Caddy readiness assertion: concurrent fixtures selected the same
HTTP port. Its 5,658 passes, one failure and eight ignores remain a failed gate;
issue #719 tracks a separate fixture fix, not an established arithmetic regression.
A later user-approved isolated fork applies only that reviewed Rust fixture patch,
leaving production compiler/runtime/std and native Plenty fixtures unchanged.
Its uninterrupted host gate passes 5,669 tests with zero failures and eight normal
ignores, including all 27 optional bodies and ten added fixture helpers; both
actual zero-test doctest packages complete. Independent review accepts this bounded
host qualification. Neither PR branch is merged or rewritten, and the earlier
failed run is not relabelled. Missing wrapper classifications, sampled-process
limits, older compatible engines and the disclosed non-test pre-clean observation
gap remain explicit limits.
Exploratory runtime/compiler/code smokes pass 16 runtime samples, 32 compiler
samples and 12 code reports, retaining full corpus sizes, native oracles, optimized
call gates and allocation-proof integrity. One sample without warmups is not
performance acceptance. The first two-job profile attempt remains blocked before
either producer by issue #731’s authenticated-system-tool alias defect. The
reviewed repair preserves strict non-tool alias rejection and consumes the exact
authenticated record bytes; 102 profile and 182 combined controller mocks pass.
The new external driver passes independent review and 58 synthetic methods, then
a fresh two-job smoke succeeds.
The full approved schedules complete serially: 2466 timed general jobs (366 warmups and 2100 samples), all 22 code reports and all 16 official profile jobs. Exact attributed requests and approximate binary-unit byte displays remain separate metrics. Independent evidence review finds no issues with collection, not a general speedup or whole-proposal acceptance. Existing optimized code is inspected in 20 representative native symbols: guard/result/cleanup/return structures are accounted for, including the C→I corpus’s extra widening/tag test and shared-cleanup differences. A bounded native/export audit establishes no new defect; neither review claims universal ABI coverage.
The compiler benchmark template now uses precise independent arithmetic Results, without changing the original workload, controls or five tests. Four new tests bring its example suite to nine passes, including actual 1000/4000-unit checking, native checksums and mutation/failure paths. Fifteen arithmetic integration tests, formatting and strict workspace/all-target Clippy pass. Frozen comparison inputs remain unchanged. See collected results and limits for distributions, attribution, source bridging and bounded code analysis. Separate pre-existing archive-decimal and PNG-framing limits were recorded as issues #705 and #707, outside these arithmetic checkpoints. The archive decimal boundary fix addresses #705.
Class match patterns shipped with issue #558. Outstanding conveniences stay B14/B71 in the backlog.
B16: Values and syntax
Block if/elif/else supports whole-RHS bindings, single-place assignments,
and explicit returns with the same contextual types, cleanup and lifetime
restrictions as match-valued statements (issue #674). Arbitrary expression
nesting remains intentionally unsupported; see
conditional values.
Tuple component borrows (issue #68), once-only numeric augmented-assignment
index evaluation (issue #32), checked casts/explicit wrapping conversions
(issues #39/#65), RangeError conversions and unsigned-negation checks
(issues #66/#67) shipped. Copying slice syntax
and NonZero[i64] range/slice steps shipped (issues #579/#584, local PR #698).
Negative literal tuple indices are supported (issue #583); nested Shared
handles gain owners in copies/slices (issue #582).
Hex/octal/binary integer literals and \u{...} string escapes shipped
(issues #563/#564). @redacted classes/enums format as TypeName(@redacted)
through automatic paths, including std secret-bearing types (issue #586).
Bytes now prints a round-tripping byte literal (issue #596).
See numbers,
text escapes and
redacted types.
Recursive tuple match patterns and nested tuple unpacking in declarations,
assignments, loops, comprehensions and generator-expression targets shipped
(issue #675). Product coverage,
affine cleanup and borrowed component loans use the existing pattern and tuple
machinery. Guards, general aliases and starred targets remain unsupported; see
matching and
tuples.
Remaining inference/value candidates stay B16 in the backlog.
B17: Tooling and packaging
plenty script delivers executable
shebang scripts, lossless Unix argument forwarding, strict optional embedded
manifests, isolated path/Git resolution, cached and durable locks, and original-source
diagnostics. Per-script private cache state and resolution locks keep generated files
out of source directories; warm runs reuse objects and eligible Linux executables.
Check/compile modes, editor header handling and cache inspection/cleanup accompany
the runner. A script test command and broader executable-cache platforms remain
outside this delivery. The completed proposal
records the rationale (local PR #851).
File imports/visibility, reserved embedded std, the plentyc/plenty split,
strict manifests, pinned Git/path dependencies, locks/offline mode, workspaces,
project commands/tests, per-target build reuse and --package-graph shipped
(issue #45, local PRs #124/#130). Native artifacts use a fixed baseline,
deterministic archives and runtime path remapping (issue #58). Release archives
and distribution smoke tests are established (issue #151, local PR #187);
releasing owns the current workflow.
Default stripping/--no-strip (issue #500, local PR #575), removal of linked
Rust panic/backtrace code (issue #503, local PR #576), executable size budgets
(issue #502, local PR #580) and reachability-rooted destructors
(issue #506, local PR #581) reduce unused native linkage. Release/no-strip
option ordering is fixed (issue #590, local PR #732).
Single-file compile-and-run now caches linked executables by default on
supported Linux toolchains, including hashbang launches. Native dependency
and toolchain identities invalidate reuse; entries share cache inspection,
cleanup and eviction. Linker wrappers, including GCC-private overrides, are
excluded before executable lookup (issue #676). Wrapper regressions isolate GCC’s
private search paths and verify linker selection, including the manylinux GCC
toolchain that prefers its private ld over PATH (issue #679). Explicit build
outputs, project runs, custom linkers, non-Linux hosts and instrumented builds
still link normally.
See the
executable cache contract.
The language server
provides diagnostics, hover, definition, completion and simple-local inferred
type hints, with Neovim integration (issues #155/#554, local PR #196).
Compile-time constants/static assertions, embedded data and if const target.*
shipped (local PR #628). Module constant indexing and import/module diagnostics
were improved (issues #562/#568/#587).
The local semantic-search trial has repository extraction, incremental synchronization/deletion detection and hybrid queries in Plenty; a pinned Python/FastEmbed subprocess performs inference. Packages and linking own current contracts. Outstanding tooling/investigation scope stays B17 in the backlog.
Engine-notices packaging reads the Plenty version directly from the root
Cargo.toml with Python’s TOML parser, avoiding the nested Bash version lookup
that failed in Windows release CI (issue #687). Regression fixtures cover release
and prerelease versions, LF/CRLF manifests, unrelated version keys and a repository
path with spaces/Unicode. These local fixture checks do not substitute for a
native Windows release build.
Generated standard-library foreign wrappers retain their loader provenance for
unused-definition pruning (issue #688). Bare unused std.dataframe/std.linalg
imports emit no engine symbols in debug or optimized objects; required native
calls retain ordinary engine discovery. This fixes the Windows release smoke
failure after packaging; native Windows CI must still confirm the archive run.
Windows executable emission also omits unreachable user helpers and generators
from its temporary COFF object (issue #690), eliminating native imports only
those bodies require. Explicit objects, library objects and dumps retain their
inspection contract; missing reached imports remain linker errors. Actual COFF
links cover the unused-helper case and a reached-call negative control, in
both optimization modes. Native Windows execution remains a CI qualification
requirement. See the emission policy.
Distribution output appends the native executable suffix to the source stem
(issue #697), producing analysis.exe on Windows while preserving dotted stems
and Unix names. Filename regression tests cover both suffix conventions; the
existing native archive smoke test still checks dataframe distribution and
relocation. Native Windows execution remains a CI qualification requirement.
B21: Documentation authoring
Executable Markdown checked by tests/test_tutorial.rs remains the lesson
source; generated literate lessons were evaluated and not adopted
(local PR #537). See the documentation design.
This is a concluded decision, not an implementation waiting in the backlog.
B22: Native concurrency
Scoped native threads, structural eligibility/reachable effect checks, consuming jobs with failed-start ownership, bounded MPMC channels, executors, ordered fallible map/tree reduction, cooperative cancellation, timed waits and receive-pair selection complete the committed batch. Named workers accept owned arguments and map callbacks share eligible immutable reusable environments (issue #28). Sending-task join/shutdown guards (issues #12/#74), receiving-worker join checks (issue #108) and blocking future-result checks (issue #109) landed.
Atomics/mutexes can be shared through scoped loans or Shared, with cycle and
lock-held wait checks (issue #326, local PR #366); HTTP client sharing followed
(local PR #369). Parallel terminals drain before return and borrow list sources
in place (issue #455, local PR #571).
See threads, channels, executors and parallel operations. Broader concurrency/task models stay B22 in the backlog.
B29: Generic checking
Callable bounds on data, use-site instantiation diagnostics and per-value
argument blame (including aliases/projections) shipped
(issues #40/#518/#519/#526, local PRs #620/#638). Generic class/enum methods are
checked/emitted when used or required by a protocol/destructor
(issue #515, local PR #616); --check-all-methods checks the program’s own
instances, not dependency/std types (issue #517, local PR #621).
See generic data.
The library-owned operations investigation
completed in issue #145 (local PR #177): existing owning wrappers and optional
capabilities meet the assessed use cases; no language or stdlib change was approved.
Future default/final methods and private-hook grants belong to the separate
protocol-body proposal.
Additional bounds/default-body contracts remain B29 in the backlog.
The boxed protocol investigation records a
proposed Box[P] contract, compiler/runtime integration map and qualification
criteria. Four original probes established the then-current rejection of protocol-valued
boxes, concrete boxed method calls, and single-method erasure of class/enum
instances through BoxedCallable. The subsequent implementation adds explicit
Box[P](value)?, shared/mutable
method tables over one concrete payload, fixed protocol argument inference,
identical-protocol generic bounds, aggregate storage and descriptor-driven
cleanup. Executable lessons cover heterogeneous lists and dictionary values.
Borrowed outputs, consuming or generic requirements, copying, keys and native
transfer remain unsupported. Shared contents and hidden executor/future, channel-endpoint or
erased-callback payloads are conservatively rejected. See the
implemented contract.
B32: Callback storage
Owned concrete reusable/consuming environments and worker eligibility work in
ordinary aggregates. Explicit fallible BoxedCallable supports heterogeneous
owned shared-call storage (issue #25). See the
storage decision and
boxed callable reference.
Additional erased modes stay B32 in the backlog.
B36: Named arguments and defaults
Named functions/methods/constructors have static name binding, source-order argument evaluation, fresh per-call defaults, callee-local propagation and nonescaping default loans. See call rules and the completed design. Additional call forms stay B36 in the backlog.
B37: Ordering
Natural list sorting, swap and multi-target assignment (issue #31), lexicographic
strings (issue #22), allocation-free unstable sort_by_key/fallible sorted
(issue #64) and tuple relations/heap ordering (issue #111) shipped. Runtime
introsort with heapsort fallback uses generated native key comparators and
cached tuple field offsets (issue #331).
See lists.
Additional ordering contracts stay B37 in the backlog.
B72: State and ownership
HTTP assembly no longer needs placeholder heads, empty-string copies or dummy
Results (issues #134–#136). Owned method receivers reuse by-value call rules;
std.text.Builder.finish() adopts them. Field moves from owned locals/self
are path-tracked with reassignment and cleanup rules (issue #534).
The owned-response experiment
validated retry ownership, partial-write reporting and exactly-once cleanup,
without implementing real HTTP integration. See field moves.
Remaining experiments/ownership extensions stay B72 in the backlog.
B75: Fluent pipelines
All four proposal stages shipped: local
borrowed/owned fused plans, adapters/pair callbacks and fallible terminals;
explicit owned .generator() conversion; chain/flat_map and callback-free
filtering/reductions; resumable/lendable plans; and explicit chunked parallel
terminals. std.iter was removed in favor of this surface (local PR #545).
Generator methods resolve concrete producers (issue #401), known lengths reserve
capacity, and affine cleanup/callback performance fixes landed
(issues #398/#399/#452/#455).
Delivery evidence includes local PRs #540/#543/#546–#548/#551/#552/#554–#556/#560/#562; issue #371 is fixed. See pipelines and callback measurements. Only the remaining extensions are B75 in the backlog.
B84: Copy classes
Nongeneric @copy classes copy on ordinary use when every field copies and no
__del__ is declared (issue #535, local PR #655). This also introduced the
closed declaration-annotation mechanism. See copy classes.
Generic @copy classes copy when their type arguments copy, and a field that moves for every argument is rejected at the class; the built-in CopyType bound requires a type argument that copies (local PR #785). Dictionary keys remain B74 in the backlog.
B90: Owner-to-view conversion
&xs, &mut xs, &s, reference bindings and b"..."/str literals convert
to whole views at view types, and &Shared[T] converts to &T like Box
(local PR #784). Byte-reading std APIs, including the prelude byte writers and
every public read-only &Bytes input, take &[u8], and ByteSource is
removed; &b"..."[a:b] views literal data and a shared view argument can view
a call result (local PR #800). See slice views
and the proposal.
Retired foundations
These original B IDs have no separate active initial-delivery task. A delivery may have later evolved; use linked references for today’s behavior, particularly dictionary writes and exact target support.
| ID | Delivered scope / decision |
|---|---|
| B01 | Linker selection via CLI/Rust options, cc-compatible drivers, verbose output, argument fidelity and deterministic native/mock tests; mold/lld preference (issue #314). Linking. |
| B02 | Application object output and embedded-runtime extraction without a linker on PATH, with external-driver tests. Commands. |
| B03 | Validation of the packaged x86-64 Linux GNU target, runtime triple and pointer layout before emission. This does not claim additional targets shipped. Runtime packaging. |
| B04 | Trusted .plentyi C scalar/opaque-pointer imports and native ABI fixtures. Declarations. |
| B05 | Ownership-aware pointer/UTF-8/C-string adapters, opaque handles, cleanup and partial/failing transfer fixtures. Adapters. |
| B06 | Static/shared C exports, scalar/Result adapters, factories/destruction, transfer contracts, embedded interfaces, SHA-256 guards and verification. Exports. |
| B07 | Typed resident runtime loaders, exact metadata/symbol checks, ownership-preserving adapters and allocation-free error tests. Loading. |
| B08 | Function values, explicit reusable/consuming closure environments, factories/inference, callable constraints, captured generator frames and lifecycle cleanup. Closures. |
| B09 | Initial generic classes/enums, IntType bounds, inference, generic methods, structural contracts, specialization limits/reuse and allocation/drop checks. Generics. |
| B10 | Box/collection-bounded self/mutual recursive source types and aliases, finite type graphs, and allocation-free cleanup of 100,000-node chains on a 256 KiB stack. Recursive data. |
| B15 | Qualified/nested/or/scalar patterns, inferred generic arguments, whole-scrutinee case _ as name and match-valued bindings/assignments/returns (issues #36/#114/#118). Omitting user enum qualifiers was declined. Matching. |
| B25 | Concrete inline generator frames, producer specialization and no-allocation construction/move/resume/drop tests. Resume scratch storage covers the full inline yield layout, including large records (issue #768). Generators. |
| B26 | Copyable inline ranges with allocation-free calls, indexing, membership and repeat traversal. Ranges. |
| B30 | Ordinary nested updates without enclosing copies/allocations, with saved-index/RHS ordering and alias/failure cleanup tests. Dictionary updates subsequently moved to insert/get_mut (issue #580). Assignment. |
| B31 | Callable-simplicity audit retained Callable/Closure, ordinary generic callback records, mismatch guidance and closure source locations (issue #24). |
| B34 | Bounded post-concurrency cleanup/review through local main 736fe80: ownership, diagnostics, inline storage, cleanup, typed reporting, collection conveniences, compiler scaling, reproducibility and docs. Later channel checks are recorded under B22. |
| B35 | Inline classes/tuples/enums with allocation-free storage construction, recursive heap boundaries and boxed C handles (issue #17). Fallible argument expressions/custom initializers retain their failures. Classes. |
Other retired premises: deterministic runtime cleanup replaces leaked string/heap storage; strings already carry lengths and embedded NULs; the precompiled runtime is embedded, not rebuilt from C per program. Function type inference, unit type arguments in function specializations (issue #110), catch-all matching and the mdBook migration are delivered. Integer bitwise operations, checked shift counts and explicit wrapping arithmetic shipped for issue #33.
Outstanding work is not duplicated here.
Constructor function values
B93’s constructor subset (issue #618): class constructors and payload-bearing
variants, including Some, Ok, and Err, are positional function values.
Concrete expected callable results specialize generic constructors; explicit
instances work without context. Pipelines accept constructor values, and shared
or mutable callback arguments can borrow their code addresses. Fallible class
initializers keep Result[Class, AllocError] as their callable result. See
function values and
remaining B93 scope.
B95: Integer domain types
Positive[T] and NonNegative[T] for signed integers now provide literal
conversion, fallible run-time conversion, private construction, .get(),
domain-preserving arithmetic with checked overflow, ordinary comparisons,
copying, scalar formatting and hashing. Explicit NonZero.from(positive) and
NonNegative.from(positive) widen without failure or allocation.
Invalid element diagnostics suggest NonZero[uN] or uN only for unsigned
integers, not floats or other unsupported types (issue #700).
See integer domains.
Plain integer dividends accept NonZero[T] and Positive[T] divisors for
//, % and augmented assignment, preserving floor semantics while omitting
the zero guard (issue #664). A NonZero divisor retains signed division
overflow handling; a Positive divisor also removes the impossible -1
overflow/remainder guard. Checked blocks infer only the remaining failures.
The std migration is B95: Std limit domains.
B95: Positive concurrency counts
Builtin channel capacity, ThreadPoolExecutor worker/queue counts and
parallel pipeline chunks now take Positive[i64] (issue #665). Literals
are checked at compile time; computed counts use Positive.try_from once.
Channel construction returns AllocError; PoolError retains only
Allocation(AllocError) and Thread(ThreadError). Parallel chunks need no
run-time positivity trap. Existing queue ownership, startup rollback, bounds
and allocation-failure contracts are preserved.
See channels, executors and parallel chunks. The remaining std APIs are B95: Std limit domains.
B95: Std limit domains
Std parameters and Limits fields with a fixed lower bound of 0 or 1 take
NonNegative[i64] or Positive[i64], completing migration groups A–C of the
domain types proposal. This covers
url, kv, markdown, json, toml, xml, regex, template, text, fmt, report, log,
csv, io, process, fs, tempfile, system, archive, compress, image, stat, math,
sketch, http (client, body, form, multipart, typed, websocket), jwt, crypto
and the actor, actor networking, TLS and HTTP server entry points.
The now-impossible Err cases are gone: UrlError.InvalidLimit,
BodyError.InvalidLimit, KvError.InvalidLimit, xml Fault.InvalidLimits,
CsvError.OutOfRange, BytesError.OutOfRange, StartError.Limit and
SystemError.InvalidInput among them. Fixed ceilings (json and xml depth 1024,
regex nesting 1000, template and fmt depth 64, hkdf size 8160, actor heap
limit, image dimensions) trap with a message naming the operation and
argument. std.actor.net.connection returns AllocError. The
conventions state the rule
for new APIs.
std.http.Limits.max_line/max_head (>= 2) stay i64 until Bounded
(B96). Fixed std ceilings now reject statically known out-of-range arguments
(issue #686), including module-domain constants, named arguments, generated
config constructors, direct field updates and replace of a field. Std-only
metadata preserves the existing integer types and runtime traps for dynamic
values; indirect calls and writes through separately borrowed fields keep those
runtime checks. See the checking mechanism.
Std first used private getter functions for constant limits;
domain-typed module constants (issue #685) replaced them.
Local dictionary views
B91’s dictionary-view consistency slice (issue #624) makes values() and
items() reusable shared local views with constant-time len and pipeline
sources. Set observations, algebra, and filter mutations accept dictionary
key views. Views keep their dictionary borrowed and remain local-only; mutable
value/item iteration stays immediate. See dictionary views
and the remaining B91 scope.
Issue #803 documents (&mut d).values() and (&mut d).items() as the way to
update every value
without a per-key get_mut. values_mut(), items_mut() and iter_mut(),
indexed field or method writes, and writes through a shared value loop now
name that form in their diagnostics.
Structural key diagnostics
Rejected dictionary keys, set members, key views and KeyType arguments now
identify the first unsupported nested component in declaration order. Paths
name class fields, enum variants and one-based payload/tuple positions; custom
cleanup and anonymous sums have specific explanations. Key admission remains
unchanged. See the generic constraint reference
and B74 for remaining key design decisions.
Optimized builds by default (issue #723)
All compilation entry points now optimize generated code by default. The
fast-compile profile
trades runtime optimization for shorter compilation without changing language
checks. Package output uses target/optimized/ and target/fast-compile/;
--release remains an alias for the default. Profile-separated cache reuse and
explicit two-profile regression tests accompany the change. Integration with
call-site inlining preserves caller handling of C string conversion errors
(issue #724) and shared cleanup for tail calls with managed locals (issue #725).
Checked-default validation
Branch-scoped validation for local PR #886. This records implemented adoption and bounded evidence, not a merge, universal ABI certification or performance superiority. The proposal records the design; implementation and validation records its integration history.
Compared contracts
The frozen baseline is 8e8fccd9: main + the common #712 Unit-verifier fix, not
pristine or current main. The frozen candidate is 830763c9. Compiler, runtime,
flags and native-tool provenance were authenticated separately for each role.
Later controller repairs and benchmark-template adoption did not rebuild or
replace these comparison compilers.
| Mode | Compiler | Arithmetic contract |
|---|---|---|
| T | Baseline | Original unadorned trapping arithmetic |
| C | Baseline | Original explicit checked recovery |
| I | Candidate | Implicit function-level checked arithmetic |
| U | Candidate | Explicit trapping unchecked: |
T/U and C/I compare matching behavior. Trap-process and handled-error timings
are not interchangeable. Operators produce plain successful values; functions
and callbacks independently declare their public Result contracts.
Semantic and native qualification
The final common-fix native matrix passes 124 checks, 124 native builds, 284 exact runtime cases, six optimized call-retention gates and 24 cleanup cases. All 16 full 1000/4000-unit corpus configurations compile and execute their designated checksum entrypoints: 1004057 and 16016057. Four C/I calling-thread zero-allocation intervals have four genuinely allocating positive controls. These checks are not an absolute allocator census or an execution of every function in a corpus.
The uninterrupted post-repair host gate on isolated validation revision
8ca6fc5a passes 5669 tests, zero failures and eight normal ignores, with both
actual zero-test doctest packages completed. All 27 required optional bodies
execute, including real proxy, dataframe and linalg cases. Real HTTP fuzzing
requires at least 20000 completed iterations; headless Neovim assertions execute.
That host revision contains only the separately reviewed Rust proxy-fixture
patch from local PR #918 over the checked-default source. Production compiler,
runtime, std, Cargo and native Plenty fixtures remain unchanged from 830763c9.
This is not a direct gate of either untouched PR branch or all current main.
Ten authenticated same-ID container removals and subsequent absences are proved;
missing final wrapper classifications remain unknown. Older external engine
builds establish compatibility, not fresh-engine qualification. Ten non-test
example paths’ pre-clean absences remain unobserved; critical build freshness is
separately verified.
The earlier uninterrupted gate remains failed: 5658 passed, one Caddy readiness failure and eight ignores. Two fixtures selected the same port. Later success does not relabel that historical result.
Repository benchmark template
The compiler benchmark generator adopts precise independent Result, Ok and
callee ? contracts, with no per-operator wrapping. Original options, defaults,
1000/4000 scaling, declaration/operation order, warmups, timing boundaries, cache
policy and all five existing tests are unchanged. All 1000 generated unit bodies
match the frozen I corpus; only main replaces its old Failure with the precise
I/O, allocation, overflow and zero-division union. Frozen inputs are unchanged.
Four new tests bring example coverage to nine passing tests. Both complete 1000/4000 sources type-check, freshly compile and execute exact original checksums. Failure cases preserve earlier field mutation and require independent callee/main error declarations. Fifteen arithmetic integration tests, formatting and strict workspace/all-target Clippy pass. A separate target reuses inactive dependencies only after cleaning both compiler/runtime packages and checking critical artifact absence. These are functional tests, not new performance data.
Final helper CI initially reports 333 tests with 60 errors and three normal skips:
the historical-template hash guard correctly rejects the newly adopted live
example. Issue #732 separates the inputs: an exact full historical Rust fixture
retains SHA 14de3ba2, authenticated in one byte read before extraction. Current
example changes cannot affect it; missing/tampered fixtures fail closed without a
fallback. All 62 regenerated source files and 142 complete case expectations
remain byte/value-identical to the frozen native inputs. The failed CI log and
old helper remain preserved; focused 84 acceptance and 102 profile tests pass.
No comparison compiler, frozen input, measured result or receipt is refreshed.
Controlled general collection
One serial, interleaved schedule retains 2466 timed invocations:
- Runtime: 270 warmups and 1620 samples; five warmups and 30 samples per group.
- Compiler: 96 warmups and 480 samples; three warmups and 15 samples per group.
- Total: 366 warmups and 2100 post-warmup samples; 86 complete groups and 75 available paired-ratio distributions.
All slots pass their original oracles. The 350 timed exit-1 arithmetic traps
are required outcomes, not waived failures. Recovered repeated errors remain
separate outcomes. No sample, warmup, failure or outlier was discarded. Duplicate
final command-log metarecords are not additional executions. All four general
phases pass; the helper’s packet_complete=false and acceptance_complete=false
remain unchanged. Collection success is not whole-proposal acceptance.
Selected paired wall ratios are median [p10, p90], not confidence intervals:
| Runtime success case | U/T | I/C |
|---|---|---|
| Tight loop | 0.986 [0.880, 1.076] | 0.982 [0.906, 1.052] |
| Calls | 0.993 [0.877, 1.141] | 1.003 [0.886, 1.144] |
| Text full output | 0.978 [0.903, 1.065] | 0.982 [0.926, 1.072] |
| Text success | 0.964 [0.948, 0.986] | 0.982 [0.964, 1.004] |
| Compiler action | 1000 units U/T; I/C | 4000 units U/T; I/C |
|---|---|---|
| Check | 1.002; 0.985 | 1.012; 0.976 |
| Debug object | 1.009; 0.981 | 1.009; 0.987 |
| Debug AOT | 1.002; 0.986 | 1.000; 0.990 |
| Release AOT | 1.015; 0.990 | 1.008; 0.989 |
Complete distributions, including all error paths and available C/T and I/U
pairs, remain in the local review evidence. Within-group (p90-p10)/median
ranges from 2.43–55.13% for runtime and 1.12–14.77% for compiler groups. These
whole-process observations do not establish a general speedup: startup, parsing,
printing and teardown remain included. Compiler wall time includes the GNU-time
wrapper; its maximum observed 327952 KiB RSS covers the compiler and descendants,
not the whole machine or Rust compiler-build cost.
Code and allocation evidence
All 22 code reports pass, retaining 44 debug-object/optimized-linked-ELF
structures and 110 nonempty compiler dumps. Corpus IR excerpts select
work999/work3999; this is not a function-by-function audit of all generated
functions. Linked ELF includes runtime/startup effects and is not an optimized
application-object measurement.
Read-only optimized-IR/native analysis inspects 20 actual linked symbols: kernel, caller, text consumer and each corpus’s designated work function in all four modes. Counts below cover only each named symbol, excluding callees:
| Target | Overflow guards T/U; C/I | Result-tag tests T/U; C; I | Release sites T/U; C; I |
|---|---|---|---|
| Tight kernel | 1; 1 | 0; 0; 0 | 0; 0; 0 |
| Caller | 2; 2 | 0; 1; 1 | 0; 0; 0 |
| Text consumer | 1; 1 | 1; 1; 1 | 1; 2; 2 |
Each work999 / work3999 | 6; 2 | 1; 5; 6 | 2; 5; 4 |
The first three targets match within each behavior-preserving pair. Selected successful loop passes contain nine instructions in every tight kernel, and 21 in T/U versus 24 in C/I callers, excluding the retained leaf body. These are static path counts, not measured instruction totals or a cross-contract speed comparison. Arithmetic uses plain values and shared failure exits, not successful per-operator Result objects. Text’s second C/I release site is failure cleanup, not a second release on success.
Both corpus sizes independently show one additional C→I Result widening/tag test
and one fewer release site: I sends normal list cleanup through the common exit.
Stack reservation changes from 0x120 to 0x110. The six-versus-two local guard
count does not mean lost protection: T/U inline squared-length arithmetic;
C/I retain that independently fallible call. Earlier field updates and ordered
cleanup remain before later failure handling.
Selected native returns use RAX alone for T/U kernel/caller values, and RAX/RCX for recoverable kernel/caller and every text/corpus result. Optimized CLIF agrees; no behavior-preserving pair changes its return carrier or introduces visible hidden return storage. This does not prove all functions or ABIs equivalent.
One official full-profile invocation passes all 16 separate compiler jobs:
T/C/I/U × 1000/4000 × count/bytes, normal --emit-object. Every report retains
39 allocation and 39 timing rows, including all five required minimum phases
and role-specific source locations.
| Mode | Units | Exact attributed requests | Approximate attributed bytes |
|---|---|---|---|
| T | 1000 | 3341485 | 706.2 MB |
| T | 4000 | 13373869 | 2.7 GB |
| C | 1000 | 4388551 | 894.7 MB |
| C | 4000 | 17528900 | 3.5 GB |
| I | 1000 | 4349233 | 885.6 MB |
| I | 4000 | 17382219 | 3.4 GB |
| U | 1000 | 3349154 | 709.0 MB |
| U | 4000 | 13414450 | 2.8 GB |
Counts are exact attributed successful allocator requests, including realloc new-size requests above mimalloc. Bytes are formatted APPROXIMATE values using binary 1024 units, not recovered exact totals. Count and byte values come from separate processes: do not divide them to infer an observed average size. Exclusive attribution excludes nested measured guards on the same thread but includes unmeasured callees; worker attribution is not a complete process census. Calls mean phase invocations. Inclusive, potentially parallel profiling timings must not be summed as latency or compared with uninstrumented timings.
All eight approved C→I/T→U comparisons retain 312 common-phase entries: 118 exact count fractions, 118 approximate byte ratios and 76 unavailable zero-denominator ratios. Added/removed phase maps are retained; no missing phase was dropped. No retained-heap, generated-runtime allocation or instrumentation-equivalence claim follows.
The first two-job profiling attempt remains blocked before either producer ran because of issue #731’s system-tool hard-link guard. The reviewed correction admits only authenticated native-tool context, binds fresh link-count/ctime and consumes the exact authenticated record bytes. Source, proof, helper and output identities keep strict nonalias rules. A new independently reviewed driver, 58 synthetic regressions, a successful two-job smoke and the later full16 run are separate evidence; the original failure remains unchanged.
Observation limits
Collection was serial during a user-reserved window, without other managed builds, tests or writers. Fresh pre/post checks retained ordinary desktop activity without noise subtraction. Endpoint aggregate idle exceeded 98% for general collection and 97% for profiling; endpoints do not prove continuous inactivity. No cache flushing, affinity/governor changes or foreign-process signals were used. All raw reports, streams, artifacts, warmups and samples remain retained.
The bounded native/export audit preserves typed caller-owned error storage,
Unit entry-stack validation, ordered cleanup/draining and earlier successful
mutation. Executor-reduction opcode 9’s seven initialized operands are distinct
from the separate collection dispatcher’s five-operand limit. Existing C adapters
retain explicit status/output marshalling and reject unsupported error layouts;
no general cross-language native Result ABI is invented.
This is x86-64 host evidence, not universal cross-ISA, sanitizer, all-FFI or all-Rust-arithmetic coverage. Independent archive-decimal and PNG-framing defects were recorded as issues #705 and #707, outside this arithmetic adoption. The archive decimal boundary fix addresses #705.
Completed library and application work
Completed-work index · Remaining work
The initial library slices below shipped. “Complete” refers to that bounded slice, not every extension of a module. The backlog owns all remaining candidates; linked references define current behavior and platform limits.
B38: Library breadth
Issue #49’s initial B39–B70 slices are delivered. Modules are implemented in Plenty unless native complexity/security warrants a wrapper (issue #505, local PR #578); large native dependencies use separately packaged engines (issue #504, local PR #579). The catalogue rationale records the scope decision, not a second task list.
| Delivery | Contract / evidence |
|---|---|
| XML | Bounded UTF-8 events and validating writes without DTDs/external entities (issue #462). |
| Advisory file locks | Owned shared/exclusive Linux locks, nonblocking acquisition and cancellable deadline waits; close-only lifetime preserves inherited descriptions (issue #774). |
| Filesystem watch | Bounded Linux inotify batches, overflow reporting and readiness integration (issue #484). |
| Templates | Bounded Jinja-style interpolation, filters, conditions, named inheritance and HTML escaping over explicit JSON (issue #499); reusable bounded text and quote-specific attribute escaping (issue #823). |
| Linalg | Finite f64 matrices, solves and dense decompositions over allocation-safe faer kernels, with engine-parameterized build/packaging/CI (issue #490). |
| MIME | RFC 9110 media parsing/building/formatting/matching and the extension table used by HTTP files (issue #485). |
| Images | Plenty over std.compress: bounded owned 8-bit buffers, all PNG colour types/depths/Adam7 decoding, encoding, crop, conversion and nearest/bilinear resizing; dimensions/bytes checked before allocation (issue #476). |
| Statistics | Finite-f64 descriptions, inclusive quantiles and normal/uniform/exponential distributions with explicit fixed-draw sampling (issue #494). |
| Sketches | Dense HyperLogLog, reservoir quantiles, Misra–Gries bounds, static xor/scalable cuckoo filters, bounded memory and versioned checkpoints (issues #496–#498). |
| Key/value database | Named ordered byte maps, transactions and bounded scans in Plenty over SQLite; versioned file schema and engine decision (issue #495). |
| UUID | Canonical text/network bytes, OS-backed v4/v7 and allocation-free numeric sort keys (issue #461). |
| System | Bounded Linux procfs snapshots for CPU/memory/load/uptime/block devices/disk I/O/processes (issue #487). |
Remaining breadth is B38 in the backlog.
Issue #705 corrects the archive PAX decimal parser
to accept every nonnegative i64, including its final eight values. Boundary
tests cover both optimization modes, overflow and malformed digits; the separate
TAR content-length cap remains 2⁶² bytes.
B39–B70: Initial delivery slices
The local issues listed here recorded acceptance checks, ownership/allocation contracts and prerequisites. They are completion evidence, not active task assignments. Extensions retain their original B IDs in the standard-library follow-ups.
| ID / local issue | Delivered subset |
|---|---|
| B39 / #76 | Bytes: owned buffers, checked ranges, explicit copies, borrowed access, list-style indexing, loops, native fill and bulk scans (issue #115). Copying slices (issue #579), borrowed &[u8] subranges and mutable borrowed loops (issues #581/#594) shipped. Static bytes literals and ByteSource support in byte operations and std stream writes shipped (issue #156, local PRs #198/#203). Original lossless Unix transport remains list[u8] (issue #61). |
| B40 / #77 | Binary streams: statically dispatched reader/writer/seek/close, explicit partial progress, exact/bounded reads/writes/copy, fixed buffering and distinct flush/close failures, with file/process/memory adapters over Bytes. Sequential binary Cursor fields add checked reads/writes, payload caps and contextual truncation (issue #658). |
| B41 / #78 | Error context: typed operation/path/cause, explicit conversion, descriptions, stderr fallback reports, unit Results (issue #110), Formattable causes and readable entrypoint reports. |
| B42 / #79 | Time: checked durations, monotonic instants/deadlines, timestamps and sleep; typed std time APIs and cancellable waits (issue #635). Calendar/time-zone/RFC 3339 support is the official external plentylang/tz package (issue #463); Zoned.until/since calendar differences shipped in v0.3.0 (issue #566). |
| B43 / #80 | Environment/process: lossless variables/snapshots/cwd, argument-vector spawning without a shell, child env/cwd, piped streams, bounded deadlock-free collection, exit/signal status, termination and reaping. Shlex splits/quotes shell words without executing/expanding (issue #478). |
| B44 / #81 | CLI: validated schemas, typed/repeatable arguments, --, help/errors and lossless paths. Progress adds terminal-only throttled bars/spinners/counts/rates/ETA (issue #477); fileinput adds buffered fallible line iteration over native/raw command-line paths and stdin fallback (issue #553). |
| B45 / #82 | Formatting: static display/debug contracts, scalar/callback aggregate formatting, bounded partial-progress streaming and fallible strings. F-strings add field expressions, conversions, debug =, checked Python-style specs and nested fill/alignment/width/precision, returning FormatError; formatter output avoids intermediate strings (issue #157, local PRs #200/#202). |
| B46 / #83 | Logs: leveled typed events and generic caller-selected sinks (logfmt text, bounded memory and nonblocking channel/drain), clock policies and bounded lines. |
| B47 / #84 | Text builders/join/scalar wrapping, Python-style paragraph tools (issue #654), and Unicode 17 normalization, combining-class/Mark classification. UAX #29 segmentation/UAX #11 width ship in external plentylang/unicode-text (issue #465). |
| B48 / #85 | Regex: Plenty Pike VM with explicit pattern/nesting/repetition/instruction/memory/work/output/match limits; text/bytes captures, iteration, split and replacement. |
| B49 / #86 | Collections: natural/comparator heaps and generator drains, including tuple ordering/heap values (issue #111). B75 pipelines superseded the former std.iter adapters. |
| B50 / #87 | Random: owned version-stable seeded xoshiro256** generators, unbiased ranges, shuffle/index sampling/byte fill. Secrets: OS-only random fill/tokens/unbiased integers over getrandom, with no fallback. |
| B51 / #88 | JSON: strict RFC 8259 event decoder/validating encoder/value tree, depth/input/output/duplicate-key limits, located errors and std.io integration. |
| B52 / #89 | TOML: bounded strict TOML 1.0 parser/writer with located errors. Format-preserving edits add atomic in-memory changes and explicit path limits (issue #475). |
| B53 / #90 | CSV: streaming reusable-record reader/writer, explicit delimiter/quote/newline/width/size policies, strict RFC 4180 quoting and positioned errors. |
| B54 / #91/#489/#656 | Encoding: strict hex/base64/base64url/base32/base32hex, MIME wrapping, validated alphabets/padding and explicit-endian integer/bit-exact IEEE binary32/binary64 packing. |
| B55 / #92 | Filesystem: descriptor-rooted no-follow operations, anonymous publication, bounded copy/delete and byte globs; tempfile resources with explicit retention/cleanup. Recursive copying and selected metadata preservation add shared bounds, explicit partial-tree recovery and strict ordinary permissions/timestamps (issues #659/#660). Directories resolves HOME/XDG config/cache/data/state (issue #472), with Linux trusted-parent/publication limits documented. Explicit private creation modes and held-handle permission updates avoid create-then-chmod windows (issue #773). Link creation and canonicalization preserve native bytes and OS errors on workers, with explicit path limits and no confinement guarantee (issue #775). |
| B56 / #93 | Explicit thread/sync/channel/concurrent modules preserve builtin identities and ownership/effect guarantees. Atomics and mutexes shipped (local PR #362). |
| B57 / #94 | Test support: located assertions, Scratch, seeded RNG, fake clock/stream failures and short transfers; parallel package test processes, filters/list/capture/jobs/timeouts/seeds, per-test TMPDIR and allocation/file-I/O failure injection (local PR #142). |
| B58 / #95 | Every native primitive declares a worker effect, without std-wide exemption. Readiness supports bounded waits/cancellation and many-handle selection with direct token wakeups. |
| B59 / #96 | Sockets/DNS: owned TCP/UDP, bounded deadline/token resolver, descriptor-limit errors and measured accept bounds (local PRs #135/#140); immediate accept_now/read_now/write_now report WouldBlock for selector-driven TLS (issue #527). Initial platform: x86-64 Linux; Unix-domain sockets were declined. |
| B60 / #97 | URL: strict RFC 3986 absolute parsing/validated formatting, component-aware percent encoding, query pairs and bulk scans (issue #115). |
| B61 / #98 | TLS: verified TLS 1.2/1.3 client/server streams over std.net using vendored rustls/ring; PEM/system roots, deadlines/cancellation, certificate errors, client certs, ALPN/SNI, version/cipher choice, CRLs and selector-driven try operations (local PRs #145/#147). Recoverable rustls allocation, resumption/0-RTT and OCSP were declined for this slice. |
| B62 / #99 | HTTP/1.1: bounded head read/write and content-length/chunked/close-delimited framing, strict ambiguous-frame rejection and mutation corpus (local PR #139). |
| B63 / #100 | HTTP client: pooling, bounded deadlines/redirects, cross-origin credential stripping, proxies and idempotent replay; verified HTTPS, CONNECT, streamed uploads/Expect (local PRs #146/#154). Shared-client locking/atomics (issue #326, local PR #369) and pool accounting/cancellation fixes (issue #333, local PR #378) landed. ws/wss clients shipped (issue #467, local PR #593). Cookie jar/content decoding were declined; callers retain their own policies. |
| B64 / #101 | HTTP server: ordered typed routing, bounded admission/fixed workers, deadlines/logging, transport abstraction, graceful shutdown, parked idle keep-alives and accept-timed HTTPS handshakes (local PRs #143/#148). WebSocket upgrades/shutdown (issue #466, local PR #593) and detached close-delimited SSE shipped (issue #468, local PR #597). |
| B65 / #102 | Bounded forms/JSON/multipart temporary uploads, validated signed cookies/rotation, reverse-proxy/direct-TLS guidance and root-confined conditional static files (local PRs #144/#152). See body, cookie, proxy and files. |
| B66 / #103 | HTTP production qualification: framing mutations, curl/TLS interop, slow clients/load/shutdown, allocation/descriptor/heap failures, relocated tools and backend linkage (local PR #157). Host/absolute-form validation, parked heads/handshakes, signal drain, lingering close and TLS Finished acknowledgement fixes landed (issues #131–#133/#138/#142/#148). Nginx/Caddy reverse-proxy Docker checks, installed CLI/data and JSON-service workflows, HTTPS baselines and libFuzzer integration shipped (local PRs #171/#180/#183/#186). Production-ready label: 2026-10-18. Typed application routes shipped (issue #159, local PR #199). Proxy fixture port leases prevent cooperating same-process HTTP/TLS reuse across socket handoff and through cleanup (issue #719); deterministic non-Docker ownership tests cover concurrency and unwind. Separate parallel native qualification passes all ten worker/actor nginx/Caddy cases with authenticated removals and unchanged foreign containers (local PR #918), not a whole-workspace gate. External bind races remain; see reverse proxies. |
| B67 / #104 | Crypto: SHA-1 protocol use, SHA-256/512, HMAC and constant-time comparison via RustCrypto; BLAKE3 incremental/keyed/derivation (issues #473/#483). AEAD, Ed25519, ephemeral X25519 and HKDF use ring (issue #464). RSA/P-256, SPKI/PKCS#8/JWK import, strict PEM and Wycheproof checks shipped (issue #561); JWT builds on them (issue #488, local PR #697). RSA arenas make allocation failure explicit (issue #567); Ed25519 small-order rejection uses SmallOrderKey, while mixed-order keys remain accepted (issue #569). |
| B68 / #105 | Compression: bounded gzip/zlib/DEFLATE buffers/streams over zlib-rs, member/trailing-data/checksum policies and fixed per-stream memory. Brotli qualities 0–11 support bounded streams and fallible native allocation (issue #491). |
| B69 / #106 | Archives: TAR ustar/pax/GNU and ZIP/ZIP64 stored/DEFLATE streaming with root-confined staging extraction under explicit policy. |
| B70 / #107 | SQLite: vendored 3.54.0, typed statements, owned rollback-on-drop transactions, copied values and explicit length limits. |
B48: Regex acceleration
Issue #742 adds bounded ASCII prefix filters across alternatives and classes,
a lazy DFA for existence and fixed-length match offsets, and a one-pass engine
for unambiguous ASCII transitions. The Pike VM preserves subgroup captures and
handles unsupported patterns, Unicode fallback and exhausted DFA caches. Work
limits include accelerated scans and fallbacks; optional scratch fits within
the existing memory budget, and allocations remain fallible. Issue #752 adds
pattern and search-budget compatibility checks before a cursor reuses cached
transitions. Issue #753 removes per-byte text calls from the ASCII engines
by borrowing UTF-8 storage once per search, iterates byte slices directly in
the prefix scanner, and applies that filter at DFA restart states. The full
regex-redux workload uses 44.36% fewer instructions than the merged #742
implementation. Issue #758 adds separate bit lanes for small alternative sets
and bounded native substring batches for at most eight short ASCII prefixes.
The unchanged full workload uses another 11.39% fewer instructions than main
d951457d, with identical reference output. Scalar fallback lowering is
isolated with CLIF variants and tracked in issue #760.
The unchanged regex-redux program measures cleaning, all nine variant counts,
and five substitutions against the Rust regex reference. Reproducible
instruction-count results and the C/Plenty prefix-loop comparison are recorded
in experiments/regex/README.md. See the regex costs and coverage
and benchmark workflow. B48 retains broader
engine coverage, scalar scan lowering and Unicode feature expansion.
B71: Error composition and HTTP
The language/reporting foundation and precise-error migration are delivered:
general sums, enum methods, structural protocols, Formattable, typed context,
entrypoint reports and typed HTTP handlers. std.error.map_error and
Result.map_error accept payload variants as function values (issue #556).
Explicit retained-job errors/drop_job() preserve start/submission causes;
Failure remains a last resort.
Typed I/O supplies stream helpers/legacy bridges. Typed HTTP body readers preserve concrete causes and partial progress, validate parsed framing and guard connection recovery. Typed parsing/writes/buffering serve the whole HTTP client, including CONNECT, uploads/redirects/pooling; stored-error retrieval was removed while the legacy Reader bridge and typed response composition remain.
Installed CLI/data and JSON-service workflows validate causes, reports, exit status, recovery and cleanup (local PR #472). See the composition evidence.
Issue #634 unifies report and display formatting on
std.fmt.Formatter[W] and FormatError. Report methods use a method-local
writer parameter, including anonymous-sum dispatch and conditional typed
contexts. Reports can use arbitrary writers and floating-point format specs;
stderr text/integer/boolean reports still allocate no memory. Canonical owned
text methods on URLs, media types, socket addresses and Set-Cookie values are
to_string(). See report formatting.
Remaining presentation/inference candidates stay B71 in the backlog.
B76: Columnar data
The optional dataframe foundation includes bulk numeric/string/bool construction/extraction, nullable schema inspection, conditional/cast/string/distinct/window expression plans, grouping, five join kinds, stable sorting/windows and CSV/Parquet streaming (issues #448–#451). The Polars engine is reached-only and separately packaged; version-matched release archives, compatibility/notices and deployment checks shipped (issue #456). Integer-key checked as-of joins shipped (issue #453); Date/Datetime units, naive/UTC metadata, nullable epochs and calendar formatting followed (issue #457). Numeric widths, owned binary columns and nested list/struct schema/access/ Parquet interchange shipped with issues #458–#460, preserving parent/child nulls. The original spike records its evidence.
The sales report application
and examples/sales_report/ exercise catalogue validation, CSV/partitioned
Parquet, nullable cleanup, UTC daily grouping and audited exports, with independent
checks and an optional scaled workload. Classified rows are materialized once.
Primitive-field clist provides native-width packed columns, allocation-free empty owners, transactional reserve/append, scalar indexed fields, explicit row copies and borrowed scalar-valued pipelines. Ordered removal/clear retain capacity without allocating. Direct indexed access avoids per-field runtime dispatch (issue #432); bounded optimized integer count/filter SIMD is recorded under B23. It remains distinct from Polars-backed dataframes. Columnar evidence records whole-row tradeoffs. Remaining scope stays B76 in the backlog.
B78: Diffs
Myers/patience diffs over lists, Bytes and
line/word/char tokens produce GNU-compatible unified output (issue #469,
local PR #589). max_cost optionally bounds Myers work (issue #510,
local PR #596). The scalar-list codegen improvement is recorded under
B19, not an unfinished diff fix.
Extensions stay B78 in the backlog.
B79: Markdown
Markdown is a Plenty port of commonmark.js, targeting CommonMark 0.31.2 with events/HTML and input/depth/output limits (issue #470, local PR #595). Extensions stay B79 in the backlog.
B80: Plotting
The partial plotting implementation delivers owned tables/record accessors, layer-specific data/inherited mappings, nine geoms (including supplied errors/ranges/ribbons), grouped counts/binning/bar positions, wrapped/grid facets, transformed continuous/discrete position scales with breaks/ labels, palettes/gradients, basic themes/legends, wrapped text layout preflight and streaming SVG. This is not a claim that the accepted core milestone is finished; its remaining scope stays B80 in the backlog.
B81: LRU cache
LRU implements fixed-capacity expected-O(1) operations in Plenty with no allocation after construction (issue #482, local PR #613). Copying getters followed use-site generic checking (issues #515/#516, local PR #619); optional reference/mutable getters replaced callback/trapping getters (local PR #623). No separate outstanding B81 item was recorded; future additions need their own use case/scope.
B82: Graphs
Graphs deliver stable directed/undirected handles, BFS/DFS, topological sort, SCC/components, Dijkstra, A*, Bellman–Ford and Kruskal (issue #486, local PR #614). Extensions stay B82 in the backlog.
B85: Actors
Actors deliver supervised single-thread execution, per-actor heaps on x86-64 and AArch64 GNU/Linux, bounded copied-message mailboxes and trap containment (issue #536, local PR #657). Typed trap exits, heap-safe spawn checks, allocation-free timers, timeout/gone request-reply and opt-in one-for-one fixed-window restart shipped (issue #537, local PR #658).
Actor networking adds actor-per-connection listeners, pause/refuse admission, bounded back-pressure buffers, one-timer idle/stall/read deadlines and nonblocking dial/lookup (issue #538, local PR #660). Actor TLS uses a nonblocking TLS session (issue #543, local PR #664). General Watchable handles, named readiness, spawn tags on Down and in-place connection-buffer compaction shipped (issues #540–#542/#545, local PRs #661/#662).
std.actor.http.server adds a second
HTTP/1.1 server with isolated connection construction, bounded complete-request
handlers, resumable TCP/TLS streaming, deadlines and draining. The shared
std.http.decoder supplies byte-feed
request framing. Stage observations, isolated spawning and immediate/destructor
shutdown support its lifecycle; runnable deadline and live-stage cleanup fixes
address issues #638 and #639. Fair child-exit delivery and safe runnable-slot
reuse under connection churn address issues #640 and #642. Scalar-wrapper list
references use their actual storage width (issue #643), preventing neighboring
actor-address entries from being overwritten. The qualification record
separates implementation delivery from production readiness and comparative
measurements. The worker HTTP server remains independently supported.
The qualification record includes
540,000 successful matched HTTP/TLS benchmark requests, a mixed-failure run
with 400 contained traps and stable resources, and relocated native binaries.
TLS sessions reconstruct a private configuration graph within their actor heap,
so trapped sessions and repeated configuration rotation release retired generations
without dropping interrupted rustls state (issue #641). This adds per-session
configuration memory and setup work; the original TLS comparison predates that
change. Completion acknowledgements and bounded retry timers reduce concurrent
keep-alive mailbox retry overhead while preserving application mail, completion
accounting and shutdown control (issue #644). The qualification record retains
the original comparison and adds a controlled before/after HTTP measurement. A
refreshed TLS comparison
after both fixes records 270,000 successful requests, 9,000 verified handshakes,
current idle RSS and 400 contained traps with stable after-batch resources.
A test-only budget and retained-stack investigation exercises the production heap router and trap boundary on two guarded stacks. Explicit checkpoints demonstrate peer progress, retained nested local borrows, trap recovery after resumption and cooperative cancellation with cleanup. It adds no production scheduling behavior; compiler insertion and the stage’s shared-context ownership remain unimplemented. See the probe evidence and limits.
Remaining stages stay B85 in the backlog.
Standard-library argument and error contracts
B92’s allocation/error-type work (local PR #804) gives std domain errors an
Allocation(AllocError) cause, narrows allocation-only byte operations to
AllocError, and reports invalid caller positions with named traps. Stream
reads and read_exact validate buffer ranges before transport access, EOF,
handshake work, or stored errors; TLS session transfers follow the same rule
(issue #666). Encoding fields, UUID output, crypto finalization and CSV field
positions trap on invalid caller offsets or indices; operations with no other
failure return plain values (issue #667).
Unused byte-error adapters and unreachable range variants have been removed
(issue #668). Real failures remain recoverable: truncation, malformed encoded
text, allocation failure, transport failures and the capacities, limits and
seek targets that still return named errors. BytesError.OutOfRange, kept
then for actor networking connection limits, was removed by
B95.
See the invalid-argument convention
and byte-stream contract.
B92’s remaining worker-certification subset is delivered in issue #661. Thin
function and variant-constructor values are certified against all matching
function addresses in the closed program. The 30 pure allocation adapters
introduced by local PR #804 now use direct map_error calls at 227 current
call sites, including nested domain-error wrapping. Helpers that attach
partial-progress context retain that behavior. See
worker callable certification
for its signature-wide conservatism and remaining erased-callable limits.
Static integer domains were completed by B95.
Dataframe byte-view inputs (issue #657)
std.dataframe.lit_binary accepts &[u8] from literals, owned buffers, byte
lists and subviews. Its C strided adapter passes address, length and stride
1 directly, with no Plenty-side payload allocation. Binary-column constructors
pass their packed Bytes through that adapter too, removing the additional
list[u8] payload copy. Polars still owns a native copy; extraction still
builds independent owned buffers. Mutable byte views remain unsupported by
this adapter. See binary payloads
and C borrowing.
Windows x86-64 dataframe packaging
The experimental Windows release builds the Polars dataframe engine natively
and bundles plenty_dataframe_v1.dll with its licenses. The compiler generates
an MSVC import library from the selected DLL, copies the DLL beside ordinary
applications, and includes validated metadata and notices in --dist output.
Release and portability workflows exercise Parquet queries and app relocation;
Windows execution still requires native runner validation. Linear algebra and
macOS engine packaging remain outside this delivery. See the
engine installation reference.
macOS ARM64 dataframe packaging
The experimental Apple silicon release builds the Polars dataframe engine for
Apple M1 and macOS 11, and bundles libplenty_dataframe_v1.dylib with its
metadata and licenses. The dylib has a versioned @rpath identity; applications
find their sibling copy through @loader_path. Ordinary output copies the
dylib through a temporary sibling and atomic rename, preserving already-open
engine files when rebuilding (issue #678); --dist adds validated notices and
checks library identity and checksum. Selected-engine validation errors include
the supported rebuild command.
Packaging checks native dependencies, exports, deployment minimum and ad-hoc
code signature. Native workflows exercise Parquet queries, moved installations
and application-only relocation; native execution remains for CI to verify.
The shared native archive writer now gives libraries and notices mode 0644, resolving issue #677 for both Windows and macOS bundles. Broader B18.7 library qualification, linear algebra on these platforms and signing/notarization remain outside this delivery. See macOS engine installation.
Completed compiler performance and SIMD work
Completed-work index · Remaining work
These are delivery/investigation records, not current benchmark guarantees or an optimization queue. Workloads, methods and current measurements belong to profiling; remaining B19/B23/B74 scope belongs only to the backlog.
B19: Generated code and storage
- The CPU audit (
examples/cpu_codegen_audit/, local PR #431) led to issues #360–#392.--dumpexposes operations/CLIF/assembly (issue #360). Unborrowed locals stay in registers, narrow helpers replace dispatcher calls in borrowed loops/byte/text operations, last owned reads move, held text avoids retain/ release, and SSE2 rounding is inline (issue #386). - Separate
plenty_op_<N>symbols permit native section GC. The measured stripped hello executable fell from 790 KB to 356 KB (issue #501, local PR #577). Default stripping and lean runtime linkage are also recorded under B17. - Dense integer/tag matches use bounded jump tables at 256+ distinct values (issue #395; measurements). String matches with 16+ literals binary-search one sorted table (issue #565; measurements); other shapes retain chains.
- Inline class field copies/address rewrites and destructor-free release for up to eight owned fields shipped (issue #544, local PR #663). One-field destructor-free classes use their field representation (issue #547, local PR #666); class fields are flat and strings release in place (local PR #668).
- Inline copies/clears up to 192 bytes, in-place moved arguments and descriptor string-offset cleanup plans shipped (issue #550, local PR #669). Class components of tuples/multifield variants are flat; single-field Option/Result payloads keep the address layout (issue #551, local PR #670). See flat-layout evidence.
- Release builds inline small constructors/
__init__/new, avoid release of fields initialized first and omit provably zero releases (issues #548/#549, local PRs #665/#667). - Tail
list.pop()avoids survivor relocation; other positions repair only the suffix (issue #52). Numeric/bool rows use natural 1/2/4/8-byte widths and lists omit the dictionary value slot (issue #531). - Borrowed integer/float/bool list rows use inline reads/writes with unchanged bounds reporting. The recorded 3k-line dissimilar diff fell from 265 to 100 ms (issue #509, local PR #598; measurements).
- Collection access/removal, equality and string indexing work shipped (issues #18–#21). Layout and runtime constraints still follow the reference, not these snapshots.
B19: Compiler scale and caching
-
Warm single-file executable hits skip bundled-runtime extraction (issue #699). The measured no-op shebang median fell from 16.35 to 6.56 ms while preserving source/toolchain/native-input checks. See the startup measurements and method.
-
Cached collection element facts (issue #6), linear declaration/large-enum match checking (issues #53/#71), parallel native emission and the large-function single-pass allocation fallback (issues #55/#56) shipped. More than 2,048 branch arguments select the fallback with a code-size/speed tradeoff.
-
Operation checking/native IR construction/compilation run across cores with byte-identical output for any worker count (local PR #131); lookup, frame validation, tail-call preparation and adapter scans also parallelize (local PR #136). Declarations resolve once (issue #59, local PR #158).
-
Compiler peak memory was measured at about 1.8 KB/source line for checking and 2.8 KB for object emission after exact-length syntax lists/40-byte tokens, duplicate-body removal and earlier release of tokens/operations (issues #422/#54). Whole-program check work is linear in instructions and std imports no longer enable per-function whole-program work (issue #57).
-
Content-addressed compiled-function reuse shipped (issue #144, local PR #176). Buffered cache hashing/shared decoded stencils reduce repeated work (issue #316). Whole-object cache validation includes module text/import resolution (issue #317 stage 1, local PR #385).
-
Generic naming is linear; template/program instance limits and dependency depth bounds permit finite declaration chains (issues #428/#430/#435, local PR #516). Templates release when no pending lowering needs them, reducing generic/ordinary checking memory from 1.92× to 1.37× in the recorded 200×200 case (issue #434, local PR #521). Instance-time substitution, moved template bodies, template-ID cache keys, symbol-keyed requirements and cheaper parser tokens reduced the checking instruction ratio to 1.37× (local PR #534).
-
Cranelift 0.137’s reused regalloc2 context reduced debug compile instructions 4.2–4.7% in recorded workloads (issues #153/#421, local PR #528). Reusing cleared Cranelift Functions reduced ordinary construction allocation from 20.3 to 4.9 KB/function (measurements).
-
Batched codegen results preserve deterministic output with fewer messages (local PR #368). Mimalloc removed glibc arena-lock contention, so the measured 16-thread run was no slower than eight (issue #315, local PR #386). Linker preference chooses mold/lld while preserving overrides (issue #314, local PR #367).
Compiler architecture assessment
A source-based refactoring proposal
records the compiler boundary assessment, file/symbol maps, alternatives and
validation gates. Its six steps are delivered: structured diagnostics (issue
#691), scoped recorder and teardown state (#692), the op and syntax-layer
splits (#693, #694), grouped pipeline state (#695) and a frontend lookup
profile that changed no representation (#696). The legacy stack syntax is
removed (#698). No speedup is claimed; remaining scope stays in the backlog.
B19: Profiling and concluded investigations
--features profile exposes hotpath timing/calls/allocations per phase
(issue #150, local PR #188); finer lowering/ownership/native rows and isolated
compile_bench workloads supported investigations #301–#305 (local PR #343).
| Investigation | Recorded outcome |
|---|---|
| #301 | Avoiding 128-bit slot split/join shifts halved AOT instructions and reduced register-allocation work 70% (local PR #351). |
| #303 | Borrowed loan-graph operations and liveness-buffer reuse cut ownership-check instructions 40% (local PR #352). |
| #302 | Shared token/nominal names first cut checking instructions 8.8% and allocations 25% (local PR #357); shared structural/callable children and FxHash tables brought the broader check-workload reduction to about 20% (local PRs #383/#389). Investigation closed. |
| #305 | The mixed-workload checking ratio was attributed to cache/TLB misses, not algorithmic work. |
| #304 | Warm-cache work was measured at roughly 45% frontend and 18% linking; linking experiments record levers. |
| #59 | Parallel parsing, speculative lowering and declaration parallelism were evaluated but not adopted: small gains and higher peak memory. Declaration reuse landed. |
| #321 | Mimalloc and skipped command-line teardown were retained; arenas/bump allocation were measured and not adopted (local PR #388). |
| #317 | Stored syntax trees/separately compiled module artifacts did not pay; the investigation identified imported-interface lowering as a prerequisite for a useful lowered-module cache (local PR #392). See module-cache investigation and module-edit baselines. |
| #436 | Substitution-environment lowering offered at most 2.6% and was set aside. Instance registration/header tokens explain the remaining generic gap. Closed. |
| #437 | Code placement affects SIMD measurements; retained placement research, no compiler fix. Closed as investigated/deferred. |
| #440 | Cranelift alias regions gave no gain and no sound general partition; reference targets can span frames/rows/fields/payloads. test_memory_order guards against unsound regions (local PR #529). |
| #441 | Cranelift 0.137 dead-store elimination was qualified against call/atomic cross-thread handoffs; release concurrency/coverage tests and >1 GiB frame diagnostics guard the contract (local PR #530). |
These outcomes are not active claims. Any revisit follows B19 in the backlog, with new evidence.
B19: Traps and arithmetic lowering
One object-wide trap-site table and cold function check blocks reduced the
recorded 200×200 object size 34% and single-core emission 9%
(issue #420, local PR #505). Debug cmp; jcc overflow checks reduced that
benchmark’s text 5.5% (issue #425, local PR #525). Cranelift 0.137 fuses unsigned
add/all multiply checks and keeps pooled float-mask lanes in registers
(issue #439, local PR #528).
PC-relative trap tables and executable ud2/SIGILL resume-to-report helpers
reduced recorded text 14% debug/11% release and compile instructions 7–9%
(issue #425, local PR #531). Libraries/fuzz/coverage retain helper calls.
Issue #425 was closed as deferred to upstream changes, not as proof that every
check/frame/spill cost disappeared. The runtime reference
owns the contract; upstream revisit gates are in B19 of the backlog.
The Linux foreign-handler fixture repeats signal delivery until the tested read
completes, instead of assuming a scheduling delay puts the reader in the syscall
(issue #689). Restarting reads receive a pipe byte after repeated signals;
non-restarting reads must report EINTR, with a bounded watchdog. Delayed-entry
regressions retain the later checked-overflow report in debug and optimized builds.
This changes test synchronization, not runtime handler behavior.
B23: SIMD
Automatic integer-column filter/count
uses baseline four-lane i32/u32 SIMD with scalar tails in optimized builds.
One/two total comparisons joined by and, inclusive/exclusive ranges, reversed
operands and up to two immutable same-type scalar captures are supported.
The initial std.simd module provides four concrete numeric vector shapes, masks, checked/wrapping arithmetic, selection, safe list access and scalar fallback. Bounded lowering/local promotion avoid repeated materialization in supported kernels. Direct integer wrapping reductions avoid the spill reproduced in issue #444. These are delivered subsets, not automatic packed generic arrays or a backend switch. Remaining extensions stay B23/B76 in the backlog. This delivers the initial SIMD stage of the memory and parallelism assessment, which remains partly implemented for its public allocator design.
B74: Collection hashing
Named enums with supported key payloads now use structural hashing in sets and dictionary keys. Variant tags and all payload fields participate; nested named enums, tuples and classes, including custom cleanup, are supported. Anonymous sums with supported members also qualify; their active member tag and structural value define identity. Floats, boxes and recursive types remain excluded. These general key types use keyed runtime hashing. Supported finite enum and tuple domains with at most 256 values use the bitset specialization described here. See collections.
Collections use SipHash-2-4 with a per-process OS key (issue #329; issue #747
later moved to SipHash-1-3 with direct str, bytes and scalar key paths), and record
traversal performs linear field-offset work per pass (issue #330). The built-in KeyType constraint now exposes the same key admission rule on
functions, methods, classes and enums; see generics.
Custom-cleanup keys use the same structural identity without making their
cleanup obligations copyable.
Class keys hash their fields rather than their addresses. Move-only class keys,
and enums containing them, transfer on insertion and owned extraction; lookups
and views borrow them. Set algebra and key-view to_set() create independent
copies, and affine keys_snapshot() requires an owned temporary, matching
values_snapshot(). Keys containing custom cleanup support borrowing,
transferring updates, in-place filtering and owned extraction. Non-mutating
algebra, to_set() and explicit copying reject them. Duplicate insertion keeps
the existing key owner and destroys the incoming owner. Regression tests cover
nested cleanup, generic boundaries, duplicate ownership and allocation failures.
Small enum sets
The initial specialization used one membership word for payload-free named enums with up to 64 variants, without entry/order/hash buffers or entropy reads. The wider implementation extends this to 256. The collection owner still allocates once. Membership, length, insertion/removal, same-type set relationships and algebra use bit operations. Shared iteration uses immutable canonical enum values, while normal source loans remain enforced. Mixed operations with dictionary key views retain their existing semantics.
Allocation-failure tests cover the remaining owner allocation and allocation-free
mutation; boundary tests cover 1, 63, 64 and 65 variants. These initial
measurements cover payload-free enums at revision f4aba2de. Reproduction at
that revision is
cargo build --locked --bin plenty (without instrumentation features), then
python3 scripts/bench_enum_sets.py target/debug/plenty --output /tmp/enum-sets.json:
the same first 64 variants are exercised in a 64-variant bitset and a 65-variant
hash set. It checks equal results and alternates execution order. Native programs
use --release; measurements are observations, not performance assertions.
Measured on x86-64 AMD Ryzen 7 7840HS, with 200,000 rounds and median of five runs per case. Each lookup round probes 64 values; each mutation round adds and removes all 64. Union combines two disjoint 32-member sets; copy duplicates a 32-member set. These timings describe these workloads on this machine, not a general speed guarantee.
| Workload | Bitset (seconds) | Hash fallback (seconds) | Speedup |
|---|---|---|---|
| Lookup | 0.081969 | 0.357319 | 4.36× |
| Add and discard | 0.339045 | 1.598463 | 4.71× |
| Union | 0.012670 | 1.210755 | 95.56× |
| Copy | 0.013672 | 0.461085 | 33.72× |
Finite payload enum sets
The finite-payload stage (82bd5231) extended bitsets to named enums and tuples
whose complete domains contain at most 64 values, recursively composed of booleans and named enums/tuples.
The compiler checks sums and products against the limit before generating
canonical values and variant offsets. Membership encodes the active payload with
mixed-radix indexing; algebra still operates directly on the mask. Immutable,
16-byte-aligned metadata includes nested inline records, preserving stable shared
references without runtime caches or allocations. The runtime type descriptor
now carries an optional finite-domain descriptor.
Tests cover nested multi-field enum/tuple values, every member of a 64-value
domain, the 65-value hash fallback, singleton/Option payloads, mixed dictionary views,
copying and iteration after source destruction, and allocation failure. The
benchmark at that revision also supports --payload to compare a six-boolean variant (64 values)
with the same variant plus an unused extra case (65 values). Integer/string
payloads and classes keep hash storage; the wider implementation
extends the finite-domain threshold to 256.
On the same AMD Ryzen 7 7840HS, 200,000 rounds and medians of five runs with
--payload produced the following observations. The workload counts match the
payload-free benchmark, but each key now carries six boolean fields.
These are measurements of these workloads, not general speed guarantees.
| Workload | Finite bitset (seconds) | Hash fallback (seconds) | Speedup |
|---|---|---|---|
| Lookup | 0.584369 | 1.961403 | 3.36× |
| Add and discard | 1.594221 | 7.373249 | 4.62× |
| Union | 0.011405 | 4.927148 | 432.02× |
| Copy | 0.015694 | 1.980686 | 126.21× |
Wide enum bitsets
Finite enum and tuple domains select their mask width from the complete type domain: one 64-bit word for 1–64 values, two words for 65–128, and four words for 129–256. Operations dispatch to the selected width, including scalar membership, popcount and algebra for the smallest domains. Clearing or draining preserves the selection; sparse membership never narrows it. All alternatives fit within the existing insertion-order storage, so collection owners do not grow. The canonical payload-free tag table grows from 64 to 256 entries (one shared 4 KiB table); finite payload metadata remains proportional to domain size and inline layout. Wider masks do not change key admission or dictionary storage. No target-specific SIMD is needed.
Boundary tests cover 64/65, 128/129 and 256/257 values, including hash fallback. Randomized membership and mixed-end iteration tests exercise every word; native checks cover nested enum/tuple payloads, set algebra, dictionary views, copied values, owner failure and allocation-free mutation.
Requested heap storage measured with the runtime’s actual buffer capacities (excluding allocator overhead and immutable type metadata):
| Members | Bitset bytes | Payload-free hash-set bytes |
|---|---|---|
| 0 | 176 | 176 |
| 4 | 176 | 496 |
| 64 | 176 | 4,272 |
| 128 | 176 | 8,368 |
| 256 | 176 | 16,560 |
Both owners are 176 bytes on this x86-64 build. The hash comparison uses a
257-variant enum and inserts the same tags without pre-reservation. Payload
hash rows can require more storage; canonical payload data is shared per type,
not part of these per-owner heap figures. For example, one eight-boolean variant
emits 256 × 16-byte canonical slots plus 256 × 128-byte payload records: 36 KiB
of shared value data, plus descriptors and alignment. This can outweigh the
heap saving for a small number of sparse sets; the optimization trades static
data for faster operations and smaller per-instance storage. Reproduce with
cargo test -p plenty-runtime wide_enum_set_storage_measurements -- --nocapture.
The benchmark accepts --size 64|128|256, --payload, --members, and
--workloads. Without a baseline compiler it adds unused variants to force a
257-value hash domain. --baseline-compiler instead compares identical programs
against an older compiler; this also checks the existing 64-value specialization.
Every result is checked, execution order alternates, and JSON records individual
wall and child CPU times as well as medians.
The wide-domain comparison uses the pre-change 64-value compiler, native
--release compilation without runtime instrumentation, and an AMD Ryzen 7
7840HS. Reported ratios use median user-plus-system CPU time over five runs.
Other work was running on the host; CPU time excludes scheduling waits but
cannot remove effects on clock speeds and caches. These are workload-specific
observations, not speed guarantees.
For the full-domain cases, 50,000 rounds each probe all values, add and remove all values, union two disjoint half-domain sets, or copy a half-domain set. Payload cases use seven or eight boolean fields. Sparse cases use four values spread across the 256-value domain (two per input set) and 1,000,000 rounds. Reproduce a full-domain case with:
cargo build --locked --bin plenty
python3 scripts/bench_enum_sets.py target/debug/plenty \
--baseline-compiler /path/to/pre-change/plenty \
--size 256 --payload --rounds 50000 --runs 5 --output /tmp/wide-enums.json
Use --members 4 --rounds 1000000 for sparse cases, and omit --payload for
payload-free enums.
Speedups over the older hash-backed implementation:
| Domain / workload values | Lookup | Add/discard | Union | Copy |
|---|---|---|---|---|
| 128, payload-free | 3.75× | 4.23× | 136.24× | 115.01× |
| 128, boolean payload | 4.15× | 4.46× | 696.99× | 410.92× |
| 256, payload-free | 3.56× | 4.12× | 198.95× | 192.41× |
| 256, boolean payload | 4.38× | 4.48× | 1342.00× | 849.87× |
| 256, four payload-free values | 2.91× | 3.64× | 4.47× | 8.90× |
| 256, four payload values | 3.64× | 3.79× | 26.00× | 22.92× |
Raw measurements retain every wall/CPU sample, medians, settings, compiler hashes and the 64-value comparison. The gains at both wider sizes and in sparse cases support the 256-value cutoff.
The 64-value comparison uses seven alternating runs: 2,000,000 rounds for union and copy, 200,000 for lookup and mutation. Relative throughput changes against the old scalar implementation were lookup +0.3%, add/discard +3.5%, union +1.4%, copy +163.6%. These are measured medians, not guarantees of changes this small. Width is encoded once in the storage alternative, with scalar operations for small domains. Copying reuses the validated representation and allocates only the new owner, avoiding storage reclassification and unnecessary reservation or membership counting.
Proposals
Proposals record design rationale and rejected alternatives. The reference defines current behavior; the backlog tracks future work. If a proposal disagrees with the reference, follow the reference.
Status
Each main proposal opens with a status note linking to the governing reference, identifying differences in what shipped, and naming remaining backlog items. Companion pages identify their parent and whether their examples are hypothetical.
| Status | Meaning |
|---|---|
| Implemented | The design shipped. The page keeps the rationale. |
| Partly implemented | Some of it shipped. The block says which parts. |
| Not implemented | Nothing shipped. The page is the design record for future work. |
| Investigation complete | An assessment reached a decision; it does not imply feature implementation. |
Proposal text retains its original wording: it may describe shipped features as future work or show syntax that never shipped. Proposal code is not compiled by the test suite.
When a change implements or alters a design, update its status note. Move a
proposal to completed/ when its agreed scope ships or its
investigation concludes without an implementation commitment;
retain its rationale and update links and book navigation. Alternatives and
future extensions need not ship for the agreed scope to be complete. Keep
partly implemented and unimplemented proposals here; retain superseded or
abandoned designs separately in archived/ when needed.
The status scan used reference, implementation, and regression coverage at
local main commit 74fa81d8 on 2026-10-20; it did not rerun every feature
test. Later reviews reconciled HTTP acceptance, error composition, and anonymous
sums at 4e40c4fc on 2026-10-23, and entry points, modules, executable
lessons, and the next language phase at 161e1c63 on 2026-10-24.
Find a design by topic
Companion examples and evidence do not imply additional implementation commitments.
Completed
See Completed proposals for shipped designs and their reference pages. Further extensions remain tracked in the backlog.
Partly implemented
| Proposal | Shipped | Remaining |
|---|---|---|
| Value generic parameters | Value parameters and Bounded | B83 arrays/assertions; B96 std adoption |
| Recursive owned data | Recursive storage through Box[T] | B11, B33 |
| Typed ranges, generic calls, and expression-local inference | Typed ranges, contextual literals, explicit type arguments | B12, B16, B32 |
| C interop and dynamic libraries | Imports, linking, exports, and resident loading | B18, B27, B28 |
| Generated library contracts and C headers | Exports, contracts | B27, B28 |
| Fallible memory, parallel execution, and SIMD | Fallible allocation, concurrency, initial SIMD | B13 allocator control; B23 SIMD extensions; B24 automatic parallelization; B83 fixed arrays |
| Standard library scope | Embedded library, including data formats, TLS, HTTP, SQLite, signals and testing | Remaining B38–B70 scope |
| Keyword arguments, defaults, and static unpacking | Named calls and fresh defaults under the narrower design | B36: splats and signature markers remain separate candidates |
| Units of measure | Dimensions and Quantity[T, D], erased to their storage type (issue #701) | B97: std.units, sharing bodies that differ only in dimension |
Not implemented
| Proposal | Backlog |
|---|---|
| Runtime-independent networking protocols — agreed codec/session/driver direction; complete shared WebSocket session and both drivers remain proposed | B94, B85 |
| Actor budgets and retained stacks — test-only heap/trap probe; compiler checkpoints and stage ownership integration remain unimplemented | B85 |
| Blocking work for actors — bounded native workers, independent job ownership and event-based completion | B85 |
| Async/await exploration — owned movable frames, fallible admission and synchronous cancellation; isolated spike only | Unscheduled exploration; async implementation remains out of scope pending an adoption decision |
| Cosmopolitan and Actually Portable Executables — bounded compatibility experiment before any runtime/platform commitment | B18, B17 |
| Package language editions — exploratory compatibility mechanism; no editions implemented | B17 |
| Using Rust crates in Plenty — three prototypes favor generated C adapters over a Cargo/rustc graph; production integration remains | B17, related B20 |
| Compiling Plenty to Rust source — checked-IR scalar emitter, native companion round-trip and Rust build-cost measurements; ownership probes remain hand-lowered, no production backend | B20, related B17 |
| Linear obligations, including aggregate and error rules — opt-in non-abandonment | B72 (issue #149) |
| Synchronous resolution scopes — consuming scope policy; depends on linear obligations | B72 (issue #149) |
| Linearity, typestate, and execution responsibility — shared boundaries and deferred worker/async contracts | B72 (issue #149) |
GCC backend (plenty_codegen_gcc) | B20 |
Portable std.simd and automatic pipeline SIMD — small portable API, bounded safe transformations, and backend spike evidence | B23; related B12, B19, B24, B75 |
| Columnar lists — explicit field-column storage, a bounded object facade, and native layout/ownership spikes | B76; related B11, B12, B23, B75 |
User-defined ordering — one three-way __cmp__ hook | B37 |
| Timsort for stable, adaptive sorting — fallible workspace, stability, and compatibility choices | B37 (issue #331) |
User-defined dictionary keys and set members — derived structural hashing and a KeyType bound | Structural class/enum/anonymous-sum keys, including custom cleanup, plus KeyType; copying operations require duplicable keys |
Derived serialization for classes and enums — @serial derives hooks against a JSON-shaped std.serial data model shared by JSON and TOML | None yet; candidate under B51/B38 (issue #471) |
Returned enums that carry references — Option[&T] and Result[&T, E] returned under the first-reference-parameter rule, held in locals, never stored | None yet; extends B11 (issue #516) |
| Go-style scoped tasks, with runtime choices and native spike evidence — exploratory fixed-stack alternative; no lightweight scheduler adopted | B22 investigation; implementation uncommitted |
| Arbitrary-precision integers — Plenty limbs with an inline small value; malachite rejected on allocation failure and LGPL | B38 (issue #481) |
| Thread-local reference counts — single-threaded phase first, then Lean/Koka-style thread-shared marks at certified boundaries; measured atomic count traffic | None yet; related issues #743–#746 |
| Faster linking — measured link costs; caching the executable, running the linker directly and reusing the extracted runtime archive; object-format changes do not pay | B19; related B17 |
Checked arithmetic by default — function-level typed errors, explicit unchecked: with a validated no-failure precondition and retained local checked: recovery; core/repository adoption and bounded validation complete; local review and merge pending | B14 |
Exact decimal arithmetic — allocation-free 38-digit Decimal in Plenty, scale kept with numeric equality as in Python, exact operators under the integer trap and checked: rules, NonZero[Decimal] divisors, explicit rounding | None yet; related B76, B37, B74 |
Numerics core — std.numeric in Plenty over std.linalg; a scoped SciPy subset in three milestones; basin and numeris not adopted | B38 (replaces issues #492, #493) |
| Six rules for a consistent core — one copy rule, one formatting path, owner-to-view conversion, adapters on iterators only, error types that name only what can fail, every callable form as a callback | None yet; proposes B88–B93; related B12, B49, B84 (issues #614–#636) |
Completed investigations
| Assessment | Decision / related backlog |
|---|---|
| Native macOS, Windows, and AArch64 support | Native ports need target-aware builds, ABI validation and OS/stdlib contracts before release packaging. B18, with B17/B12 dependencies. |
| Protocol method bodies | Core default/final dispatch and member-specific grants agreed; definition-time contract checking required. No automatic method injection. Not implemented or scheduled (B29). |
| Owned structures and checked node identities | Tested patterns and arena prototype; production identity/access contract remains B73, broader borrowing B11. |
| Typestate and owned transitions | Existing owned transitions and their limits; linear non-abandonment is a separate proposed extension (B72). |
Cosmopolitan and Actually Portable Executables
Status: Not implemented. A bounded feasibility spike demonstrated two programs, not supported Cosmopolitan output. This proposal recommends an evidence-gated compatibility experiment before a platform commitment. Current behavior remains in runtime packaging and linking. Future work belongs to B18/B17 in the backlog.
Cosmopolitan offers a different deployment route from the native-platform assessment: generated programs could use Cosmopolitan Libc and the Actually Portable Executable (APE) container to run on multiple operating systems. It does not automatically make Plenty’s GNU/Linux runtime portable. Keep native GNU/Linux output unchanged and make any eventual Cosmo output explicitly opt-in.
The recommendation is not to implement a broad runtime port now. First establish a reproducible x86-64 runtime compatibility result, including failure paths, threads and execution of the identical artifact on a non-Linux OS. Then choose a restricted feature profile, a larger runtime port, or no support. The runtime design describes the unresolved build and ABI choices; evidence and qualification records what was actually measured and the go/no-go criteria. The proposed target facts and code-generation contract prevents host-Linux branch selection; a self-contained harness specification defines the first implementation deliverable without relying on scratch state.
Multi-OS is not multi-architecture
The spike used x86_64-unknown-cosmo-cc: its successful Plenty artifacts contain
x86-64 code only. Multi-OS output on one ISA is useful independently of fat
output. The ordinary cosmocc C workflow compiles both x86-64 and AArch64 and
combines the results with apelink; it cannot manufacture AArch64 Plenty code
from an x86-64 object. True fat Plenty output requires two qualified codegen
paths, two compatible runtimes and dependencies, and packaging and testing of
both slices.
Native Linux AArch64 work is being developed independently (local PR #706,
not merged into the inspected main). Coordinate target descriptions, ABI
checks, platform boundaries and tests with that work when available. Do not
copy unmerged code or treat its Linux GNU runtime, signal context or release
qualification as Cosmopolitan support. Native Windows/MSVC and Darwin ABIs are
also different targets from the C ABI offered by Cosmo.
Support matrix and deployment contract
This matrix separates upstream scope from Plenty evidence. It is not a support announcement or a promise of every OS × ISA combination.
| Environment / artifact | Evidence | Proposed admission |
|---|---|---|
| Native Linux x86-64 GNU | Implemented baseline at inspected main 2851ef34 | Preserve existing support |
| Cosmo x86-64 on Linux | Empty and print/allocation programs run; checked trap fails | Compatibility experiment only |
| Same Cosmo x86-64 artifact on Windows, macOS or BSD | No Plenty execution tested | One real non-Linux OS required for experiment; each named OS/version needs separate qualification |
| Cosmo AArch64 on any OS | No Plenty evidence | Separate ABI/runtime feasibility gate |
| Combined x86-64 + AArch64 APE | No Plenty evidence | Only after both slices qualify independently |
| BIOS / bare metal | Upstream release mentions BIOS; no Plenty evidence | Non-goal for this proposal |
The official toolchain documentation names Linux, macOS, Windows, FreeBSD, OpenBSD and NetBSD and describes the two ISA toolchains. These upstream claims do not establish Plenty’s OS versions, CPU baseline, capabilities or deployment guarantees. Choose and record those per qualified configuration rather than inheriting an upstream support list.
APE needs a working execution route. On the spike host a shell launched the
successful programs, but direct Python subprocess execution returned ENOEXEC.
The official documentation describes shell fallback, loader extraction into
TMPDIR/HOME, optional system loader/binfmt configuration, and Apple Silicon
loader installation/build requirements. Test writable/executable extraction
locations, restricted environments and installed-loader operation explicitly.
Do not require privileged registration by default or silently alter the host.
Do not claim every deployment is a zero-prerequisite direct execve launch.
Windows shell requirements for the toolchain wrappers are not a claim that
every final Windows executable needs a POSIX shell.
The compiler should initially remain a native GNU/Linux tool producing an experimental artifact. Porting the compiler executable itself to APE, making all native compiler hosts cross-compile Cosmo, and fat output are separate questions, not prerequisites for the bounded experiment.
Alternatives and non-goals
- Keep native ports only. Lowest additional maintenance; loses single-file multi-OS deployment. Prefer this if compatibility or qualification gates fail.
- Linker override alone. Useful for investigation, rejected as a supported design: the spike already demonstrates a trap failure and ABI mismatch.
- Restricted Cosmo runtime. Recommended candidate after the experiment; explicitly admit a small tested capability set instead of advertising the whole standard library. This still requires correct runtime fundamentals.
- Broad runtime and dependency port. Potentially useful, but only justified after measuring Rust std compatibility and concrete module/dependency costs.
- Dedicated Rust/Cosmo target or reduced runtime without Rust std. Could remove GNU assumptions, but carries substantial toolchain/runtime maintenance; compare against selective compatibility shims using evidence, not ideology.
This proposal does not replace Cranelift, change runtime language semantics, weaken fallible allocation/security contracts, implement native MSVC/Darwin ports, provide BIOS services, promise arbitrary dynamic C libraries, or bundle a new SDK into every compiler distribution. It supplies no production effort estimate.
Decision and risk policy
The major risks are latent GNU Rust std assumptions, signal/context corruption, TLS and synchronization bugs, incompatible native dependencies, and insufficient real-OS runners. A tiny program’s dead-section elimination can hide all of them. Loader policy and toolchain supply-chain maintenance add deployment risk even when program semantics pass. A restricted profile reduces exposed facilities, not the need to establish memory safety and thread lifecycle correctness.
Decisions about the final runtime build model, admitted capabilities, minimum OS/CPU versions, public option spelling and toolchain redistribution remain behind the qualification gates. Passing a gate permits a design decision; it does not silently authorize broader support.
Cosmo target, runtime and driver design
Part of the Cosmopolitan proposal. Recommendations here are not implemented behavior; build choices remain subject to evidence gates.
Separate target identity from the final linker
Current src/toolchain.rs::validate_target_pair accepts only the packaged
x86_64-unknown-linux-gnu target. src/codegen.rs builds a host ISA and uses
System V at exported runtime boundaries. build.rs builds and embeds Rust
static archives for Cargo’s target; selecting --linker does not change those
compile-time assumptions. See current runtime packaging.
An eventual target descriptor should distinguish compiler host, generated ISA, Cosmo C ABI, ELF object format, APE final container, runtime profile and toolchain identity. Do not describe the generated program as GNU/Linux merely because the compiler runs there. Preserve the internal Plenty Tail convention separately from Cosmo C calls. Verify entrypoints, helpers, callbacks, aggregate returns, i128/libcalls, stack alignment, atomics, relocations and direct/indirect Tail calls; familiar ELF and x86-64 are not a complete ABI proof.
For the experiment, retain object/runtime extraction and external link harnesses
rather than prematurely committing to a public target spelling. For production,
choose an explicit output-target option integrated with existing target selection;
--linker alone must never imply support. Host execution of a generated APE is
a separate operation from compilation, with a documented loader selection policy.
The proposed compile-time target and code-generation contract
specifies host-independent target.os = "cosmo" selection and the early red-zone
audit. Neither target facts nor object code can be corrected by final link flags.
Runtime build choices to resolve experimentally
Do not assume linking GNU-built Rust std into Cosmo is generally safe. Rust Linux cfgs, libc layouts, TLS, entropy, filesystem and thread implementations can survive final linking unchanged. Conversely, the successful allocation probe is evidence against declaring every GNU object unusable or requiring every dependency to be rebuilt without inspection.
Compare these bounded choices:
| Candidate | Required evidence / cost |
|---|---|
| Retain selected GNU-built Rust components with Cosmo-header C shims | Audit reachable std/dependency paths and generated code; prove no incompatible OS layouts/syscalls/TLS assumptions escape the boundary |
| Build a dedicated Rust/Cosmo runtime configuration and compatible dependencies | Establish target/std availability or maintained patches, build-script behavior, ABI, reproducible/offline builds and ownership of long-term maintenance |
| Reduce OS-facing Rust std use in a restricted runtime | Measure how much replacement is actually necessary; preserve allocation, panic, reporting and thread contracts rather than invent a second language runtime |
The first candidate is an experiment, not an endorsed distribution architecture. A shim fixes bindings it owns, not arbitrary std internals. If those internals cannot be bounded, stop or move to an explicitly approved runtime build project. Reconcile the chosen build with the native runtime build assessment, including separate output directories and consistent application/library modes. Do not recursively invoke Cargo against the enclosing build directory.
A narrow platform boundary
Prefer C compiled against pinned Cosmo headers where the libc ABI is complex or has runtime-valued constants. Expose small fixed-width/pointer interfaces to Rust; keep platform structs private to C, with build-time size/alignment and native round-trip checks. This is justified by ABI translation, not a reason to wrap ordinary Plenty library algorithms.
| Boundary | Required treatment |
|---|---|
| Signals and saved contexts | Replace GNU sigaction, masks, flags, alternate-stack and register-context assumptions in traps.rs and stack_overflow.rs; install/chaining/restore and saved-PC/SP access belong behind the boundary |
| Trap reporting and stack guards | Preserve source-site diagnostics and exit semantics, inline page probes and main/worker alternate-stack lifecycle; distinguish owned traps/guard faults from foreign faults |
| TLS and threads | Audit Rust TLS lowering plus Cosmo fixups, initialization/destructors, pthread entry/attributes, stack bounds, join and start-failure cleanup; test per-thread allocation/trap state isolation |
| Synchronization and syscalls | Replace or validate mutex.rs’s raw Linux syscall 202/futex assumptions; use qualified Cosmo primitives, testing wakeups, contention, cancellation and memory ordering |
| Filesystem and processes | Audit filesystem.rs statx/readdir64 layouts, flags, paths, errors, rooted operations and process inheritance/cleanup; do not substitute unsafe path traversal for secure operations |
| Network, readiness and watch | Audit pthread attributes in net.rs, sockets, DNS, clocks, nonblocking/error semantics and Linux inotify/readiness bindings; admit capabilities individually |
| Foreign libraries | Existing dlopen/ELF residency/metadata assumptions in libraries.rs are not an APE portability contract; reject until independently specified and qualified |
Signal-handler work must include async-signal-safety, previous-handler coexistence and unknown-fault forwarding. Prefer existing helper-call checked traps as an explicit experimental fallback if necessary, but still test the signal path; disabling stack guards is not an acceptable fallback. Libraries must continue to respect host handlers. Actors’ recovery and per-actor heaps need their own qualification rather than inheriting success from a main-thread trap.
Cosmo’s TLS fixups may adapt eligible GNU instruction patterns; the spike observed GS accesses, not a TLS failure. Neither that observation nor an isolated thread-local read proves Rust TLS destructor and worker lifecycle compatibility. Test scoped workers, executors, repeated spawn/join, foreign-thread entry if admitted, thread-local failure injection and trap caller-site reset between jobs.
Capabilities and dependencies
A candidate initial profile is executable-only core language, collections, allocation, reporting and basic standard output. Threads must be tested in the compatibility experiment even if public concurrency is withheld initially. Filesystem, processes, network/TLS, watches/readiness, actors, dynamic libraries, fuzz/coverage modes and optional engines start unqualified, not implicitly enabled. The final profile is a gate decision, not a promise that this list is sufficient.
Use target/profile capability checking to reject unavailable operations before linking, with the operation and missing capability named. Define whether the check applies to reachable code or all checked declarations; test generic and indirect calls so unused embedded std declarations do not disable every program. For capabilities genuinely determined at runtime, return an explicit existing unsupported error where the API permits it; otherwise reject compilation. Never provide success stubs, silently use a Linux implementation on another OS, or drop security guarantees. If the current error API cannot express absence, settle that API before admitting the facility.
Audit SQLite, ring/rustls, RustCrypto, compression and entropy paths separately, including C/assembly inputs and target cfg/build-script outputs. Pure algorithms may need only validation; native OS/assembly objects may need compatible builds or replacement. Retain specialist crypto libraries and their constant-time and entropy contracts. Optional dataframe/linalg engines require separately compatible builds and independent tests; shipping a Linux engine does not make it multi-OS. Dead stripping of uncalled modules is not a capability enforcement mechanism.
Archive identity and link UX
Package archives with a verified manifest: compiler/runtime contract identity, ISA and CPU baseline, C ABI/layout version, Cosmo release/hash, profile/features, Rust toolchain and dependency lock identity, application/library mode and native link inputs. Reject a GNU archive, wrong ISA/profile, incompatible ABI or stale cache entry before emission/linking. Extend existing target metadata and cache identity rather than create a second source of truth. Runtime extraction must include the manifest and exact link argument arrays.
A qualified driver should discover or accept a pinned Cosmo toolchain, validate
its version/layout, construct profile-specific arguments and preserve paths with
spaces. The spike’s missing -lutil justifies auditing native flags, not deleting
all GNU flags or asserting unwinding works because -lgcc_s resolves. Preserve
panic-abort, custom panic/OOM reporting, section retention and trap tables.
Test stripping/debug sidecars with Cosmo tools instead of assuming host strip
understands the final container.
Diagnostics should distinguish missing driver, incompatible archive/toolchain,
unavailable capability, linker failure and launch-route failure. Verbose output
should expose target/profile, toolchain identity and complete command arguments.
Do not hide ENOEXEC by treating a program as arbitrary shell text; choose a
documented loader/fallback route and preserve child arguments, status and errors.
Initially reject library exports/dynamic loading and unqualified tooling modes
explicitly, rather than emitting artifacts with untested contracts.
Pinning, provenance and licensing
The spike used Cosmopolitan 4.0.2 (bundled GCC 14.1.0). Use that as a reproduction baseline, not an assertion it is the best current release. Before implementation, review upstream changes/advisories and select an exact version, archive hash and source revision. Record compiler/runtime source, Rust version, Cargo.lock, patches, host tools, flags and artifact hashes. Fetch only declared inputs in a separate preparation step; exercise offline builds and reproducibility twice.
Start with an externally supplied pinned toolchain; bundling is a separate packaging decision. The official toolchain README’s Legal section distinguishes the GCC-based toolchain from permissively licensed runtime inputs and describes embedded notices. Audit actual shipped components and redistribution obligations, including loaders and any compiler sources/patches; do not label the whole SDK ISC or assume embedded notices replace Plenty’s release obligations. If code or data ships, update module headers where applicable, release third-party notices and supply documentation. This proposal copies no upstream implementation.
Cosmo compile-time target and code-generation contract
Proposed, not implemented. Part of the Cosmopolitan proposal. These contracts precede runtime qualification, not merely final packaging.
Build target is not the runtime operating system
Today src/frontend/target.rs derives facts from the packaged native triple,
recognizes a closed set of target.os values, and discards unselected branches
while parsing, before resolution and checking. See the implemented
target conditionals.
Using that native fact source for an APE would silently select Linux-specific
code on a Linux compiler host, even when the artifact later runs elsewhere.
Recommend adding the closed value "cosmo" for target.os and deriving all
facts from the explicit output-target descriptor, never the compiler host.
Here os denotes the compile-time platform contract, not the executing kernel.
For the proposed x86-64 profile the facts are os = "cosmo", arch = "x86_64",
pointer_width = 64, and endian = "little". A later qualified AArch64 slice
would have arch = "aarch64"; fat output evaluates conditionals independently
for each slice. This does not introduce runtime dispatch or a runtime OS API.
Runtime detection facilities are outside this proposal.
For Cosmo, existing comparisons with "linux", "macos", "windows" and
"freebsd" remain valid but equality is false and inequality true. A chain
selects its first true arm, or its else, just as native conditionals do.
Thus an existing Linux/else program selects its else arm: that arm must be
portable or explicitly reject Cosmo through capability checking. An else arm
is not evidence of portability. Unknown spellings remain errors. Native targets
retain their existing values and branch behavior; recognizing "cosmo" adds a
value to the closed set, not an alias for any native OS. Older compilers will
reject that new value when evaluated, so using it requires the new compiler.
Unselected arms remain syntactically parsed but are not resolved/type-checked;
this proposal does not change that language rule.
Required tests cover top-level selected/unselected imports (including absent modules), declarations and function-body branches, first-match/else and no-match chains, all native OS equality/inequality comparisons, boolean combinations, unknown-value diagnostics and native regressions. Compare identical Cosmo selection across compiler hosts wherever those hosts are supported; unit tests must independently vary host and output descriptors even during Linux-only experimentation. Cache tests must isolate native versus Cosmo and per-ISA/profile entries, including a source whose selected import changes. Include the output facts/conditional-semantics version in the shared target/cache identity so parsed or compiled results cannot be reused under a different platform contract.
Reject host-OS aliasing: it defeats multi-OS output. Reject treating target.os
as a runtime expression: discarded imports cannot be restored at launch.
Reject initially banning all OS conditions in favor of the explicit Cosmo value:
a dedicated arm gives portable libraries a direct migration path while keeping
native-only arms unselected. Capability rejection still guards unsupported calls
in the selected code; OS comparison alone is not a capability check.
Early x86-64 red-zone and code-generation gate
The retained 4.0.2 x86_64-unknown-cosmo-cc wrapper (around lines 119–128)
passes -mno-red-zone, except in its Linux-only optlinux mode, which selects
-mred-zone. Multi-OS output must not assume optlinux compatibility. In contrast,
build.rs invokes rustc with Cargo’s GNU target and no explicit no-redzone
option. This is a compatibility risk requiring inspection, not evidence that
red-zone use caused the observed trap failure. The signal-layout mismatch is
measured; its causal role in that failure remains a hypothesis too.
Before any runtime-compatibility acceptance, audit Plenty-generated objects, Rust runtime code, bundled Rust std/compiler-builtins, vendored Rust crates and native dependency/assembly objects. Retain effective backend settings, rustc arguments/target specifications, C/assembly build commands, archive-member and reachable-symbol inventories, and disassembly covering leaf functions, spills, stack probes and signal/worker paths. Establish whether reachable functions use storage below the current stack pointer without reserving it, accounting for control flow and inlining/LTO; a textual search for negative offsets alone is not a proof. Unreachable members need a recorded reachability argument and profile enforcement, not an assumption that all dependencies disappear.
The recommended multi-OS contract is no red-zone reliance in reachable code.
Verify the pinned Cranelift configuration and emitted code; configure/rebuild
Plenty objects if necessary. Use -C no-redzone=yes for rebuilt Rust components
and compatible C/assembly settings where required. A final Cosmo linker flag
cannot retroactively recompile emitted Plenty objects, Rust std or archives.
If an incompatible reachable member is found, rebuild its owning component
(including std/compiler-builtins if implicated), replace it, or reject the
capability/profile; then repeat the audit and behavioral tests. Do not assert
that every Rust dependency must be rebuilt without that evidence. If provenance
or analysis cannot establish compatibility, the gate stays closed. Record the
verified code-generation policy and affected input hashes in archive/cache
identity so changing the policy invalidates old objects.
Self-contained baseline harness specification
First implementation deliverable, not implemented here. All paths and invocations below are proposed. Part of the Cosmopolitan proposal. This specifies reproduction, not a passing portability test or another work queue.
Inputs and offline preparation
Check in scripts/experiments/cosmopolitan/ with a README, a Python 3 driver,
a machine-readable input manifest and fixtures/. No invocation may depend on
the original /tmp directory, prebuilt compiler or unpublished scratch files.
The driver must use argument arrays, isolated output directories and explicit
compiler/tool paths, including paths with spaces. It must not modify the host’s
loader registration, main checkout or global configuration.
Pin the baseline source to 2851ef3481f70dec580eb76d2651b86524d16dfb and its
tracked Cargo.lock; record the actual source tree hash and any patch diff.
Pin Cosmopolitan 4.0.2 from
the release archive,
SHA-256 85b8c37a406d862e656ad4ec14be9f6ce474c1b436b9615e91a55208aced3f44
(bundled GCC 14.1.0). Retain the versioned upstream README, driver, signal/TLS
headers and fixup source with URLs, release/source identities and hashes.
See official sources.
The original installed rustc was 1.99.1 (58cf8f9c3 2026-10-13) and host cc
was GCC 16.2.1 20260819; neither identifies the toolchain that built the retained
Plenty binary. The harness implementation must resolve a complete Rust toolchain
pin (compiler, Cargo, target std and component hashes) and native host-tool inputs
in its manifest before running; it must not substitute a moving stable or
pretend to reconstruct the prebuilt binary. If those exact host versions cannot
be provisioned, record the deliberate replacement and treat output differences
as discrepancies to investigate. The retained compiler hash on the
evidence page is comparison metadata, not a required input.
Proposed preparation invocation (not an existing command):
python3 scripts/experiments/cosmopolitan/run.py prepare --manifest MANIFEST --inputs INPUTS
python3 scripts/experiments/cosmopolitan/run.py baseline --offline --inputs INPUTS --output OUTPUT
prepare is the only network-enabled phase: verify downloads, prepare the pinned
source checkout and Rust/Cargo dependency cache/vendor inputs and host tools,
and write a hash inventory. baseline --offline verifies that inventory, builds
with cargo build --locked --offline --release --bin plenty in isolated build
storage, and runs without network. Missing/corrupt inputs fail before building;
no implicit downloads or fallback to a cached prebuilt compiler. Repeat in two
clean output directories and compare input/object/archive hashes, recording any
nondeterminism rather than requiring unexplained executable byte identity.
Complete original source fixture set
The following text is the required retained fixture content, not new runnable book examples. Preserve filenames and line breaks for source-site diagnostics. No upstream implementation code is copied by this proposal.
fixtures/hello.c:
#include <stdio.h>
int main(void) { puts("cosmo C control"); return 0; }
fixtures/empty.plenty:
def main() -> i32:
0i32
fixtures/print-alloc.plenty:
def main() -> Result[(), AllocError | IoError]:
xs = [1, 2, 3]?
print("cosmo Plenty", xs)?
Ok(())
fixtures/trap.plenty:
def divide(n: i64) -> i64:
1 // n
def main() -> Result[(), IoError]:
print(divide(0))?
Ok(())
fixtures/abi-control.c:
#include <stdio.h>
#include <stddef.h>
#include <signal.h>
int main(void) { printf("sigaction size=%zu flags=%zu mask=%zu restorer=%zu\n", sizeof(struct sigaction), offsetof(struct sigaction,sa_flags), offsetof(struct sigaction,sa_mask), offsetof(struct sigaction,sa_restorer)); return 0; }
Command and artifact contract
Run the fixtures from an output-local copy so diagnostics use trap.plenty.
Record complete commands, environment/tool versions, elapsed times, stdout,
stderr and raw subprocess status separately for every case:
- Build
hello.cwithcosmocc/bin/cosmocc hello.c -o hello.com; shell launch must printcosmo C controland exit 0. This C control can be dual-ISA; the Plenty tests use onlycosmocc/bin/x86_64-unknown-cosmo-cc. - From the fresh compiler, extract
--emit-runtime runtime. For each of empty, print-alloc and trap, use--compile NAME.plenty --no-cache -o NAME.nativeand--emit-object NAME.plenty --no-cache -o NAME.o. Native empty exits 0; print-alloc exits 0 withcosmo Plenty [1, 2, 3]plus newline; trap exits 1 witherror: trap.plenty:2:5: division by zeroon stderr. - Test unchanged
--compile NAME.plenty --no-cache --no-strip --verbose --linker ABSOLUTE_COSMO_CC -o NAME.as-is.comfor empty and print-alloc. Baseline expects linker exit 1 mentioning missing-lutil. - Read extracted
runtime/link-args.txtas one argument per line. Link eachNAME.o,runtime/libplenty_runtime.a, all extracted arguments except exact-lutilentries, then-o NAME.filtered.com, using the x86-64 driver. Record original and filtered arrays; do not delete any other flags. Baseline expects all three links to succeed, empty/print output as native, and trap raw status -4 (SIGILL) with no diagnostic. Disable core dumps and run viash -c 'exec ./NAME.filtered.com'with an eight-second timeout, preserving child status rather than relying on a shell’s textual signal report. - Compile/run
abi-control.cwith nativeccand the x86-64 Cosmo driver. Baseline output is respectivelysigaction size=152 flags=136 mask=8 restorer=144andsigaction size=32 flags=8 mask=24 restorer=16, each with newline/status 0. - Attempt direct Python subprocess execution of hello.com and the two successful
Plenty APEs. Record errno separately from child status. The spike observed
ENOEXEC(8); loader/binfmt-equipped hosts may differ and must be identified, not reconfigured to manufacture the original result. Also record successful shell launch routes.
Retain sources, manifest, compiler/runtime/object/executable hashes, extracted archives and link arguments, logs, disassembly/relocations and a JSON case report with expected/actual outcomes, discrepancy classification and launch route. Capture failures per case and continue independent cases; skip dependent cases with explicit reasons. Unexpected failures, timeouts, missing evidence or baseline discrepancies make the harness exit nonzero. Expected linker and trap failures are recorded baseline matches, not an early shell exit or portability success. A changed result from fresh source must be investigated, never silently blessed.
A separate future acceptance mode must require traps to diagnose at the source
site and exit 1, plus the signal/stack/thread tests in the
qualification gates. It must not
inherit the baseline’s expected SIGILL as a pass. This proposal adds no harness,
executes none of these cases, and claims no new experiment result.
Cosmo evidence and qualification gates
Part of the Cosmopolitan proposal. Spike observations are not implementation or support claims. Gates define required evidence, not a parallel priority list; scheduling remains B18/B17 in the backlog.
Retained experiment and provenance
The completed spike inspected main 2851ef3481f70dec580eb76d2651b86524d16dfb in
/tmp/plenty-cosmopolitan-20260718. Its report is retained locally at
/tmp/plenty-cosmopolitan-spike-report.md, with inputs, logs and binaries in
/tmp/plenty-cosmopolitan-20260718-artifacts. These are local evidence locations,
not required book assets or reproducible public downloads. The investigation
worktree remains unchanged. The observations below preserve the essential
results even when those scratch paths are unavailable.
The experiment used a retained prebuilt Plenty compiler and its embedded
runtime, not a fresh build from the inspected source. It reported Plenty
0.7.1, x86_64-unknown-linux-gnu; retained binary SHA-256:
489623df196e84fd201417dae5a389574af184b4c7e1dc78f8a74c05e186d48f.
The downloaded Cosmopolitan 4.0.2 archive SHA-256 was
85b8c37a406d862e656ad4ec14be9f6ce474c1b436b9615e91a55208aced3f44.
Source inspection and binary execution therefore have different provenance.
Only Linux x86-64 was tested. No Windows, macOS, BSD, AArch64, BIOS, networking, worker-thread, native engine or unwind qualification was performed. No compiler/runtime source was changed by the spike.
| Probe | Retained evidence | Observed result |
|---|---|---|
C control with ordinary cosmocc | c-control-build.log, c-control-run.log | Shell launch prints control message; not a Plenty runtime test |
| Native Plenty empty and print/list allocation | empty.native-*.log, print-alloc.native-*.log | Build/run succeed |
| Existing Plenty linker override | print-alloc.cosmo-build.log, empty.cosmo-build.log | Link fails: cannot find -lutil |
Exact emitted object + extracted runtime, omit only -lutil | *.filtered-command.txt, *.filtered-build.log, empty.filtered-run.log, print-alloc.filtered-run.log | Empty and print/list allocation link and run; output cosmo Plenty [1, 2, 3] |
| Division by zero | trap.native.run.log, trap.filtered.com.run.log | Native reports source-site error and exits 1; Cosmo exits -4 (SIGILL) with no diagnostic |
| C sigaction layout measurement | abi.native.log, abi.cosmo.log, abi-control.c | GNU size/flags/mask/restorer offsets 152/136/8/144; Cosmo 32/8/24/16 |
| Direct subprocess execution | direct-exec.log | C control and both successful Plenty APEs return ENOEXEC; shell fallback succeeds |
The ELF64 x86-64 objects linked with their measured PC-relative relocations. This establishes basic object/link compatibility for these inputs, not complete runtime compatibility. The signal-layout mismatch is measured; that it explains the trap failure is a strong hypothesis, not an isolated causal result from a patched runtime. Successful startup does not prove handlers were installed correctly. TLS fixup/source inspection and observed GS accesses leave TLS compatibility unresolved rather than demonstrating either universal safety or universal failure.
reproduce.sh retains consolidated commands, but was not rerun as a suite and
ends with an expected failing as-is link. It is not a passing regression test.
Do not rerun expensive probes merely to write this proposal; the first approved
implementation experiment must replace the prebuilt provenance gap. Its first
deliverable is the checked-in self-contained harness,
including all source fixtures, pinned/offline inputs and per-case failure records.
Staged go/no-go criteria
Each stage requires review of the evidence before expanding scope. Stop with a recorded limitation if acceptance fails; do not expand an allowlist to bypass it.
| Gate | Bounded experiment / acceptance criteria | Decision enabled |
|---|---|---|
| Reproducible baseline | Implement the harness contract first; fresh locked compiler/runtime, pinned offline inputs and retained sources/commands/artifacts; record expected linker/trap failures separately from passing acceptance | Confirm the spike applies to reproducible source, or investigate the discrepancy before proceeding |
| Target and code-generation compatibility | Establish the target-fact and red-zone contract: host-independent branch/import selection, target/cache isolation, effective codegen settings and reachable-object audit across Plenty, Rust runtime/std and native dependencies; rebuild or reject implicated components before runtime acceptance | Admit only code-generation inputs with a defensible multi-OS ABI contract; no final-link-only fix |
| x86-64 runtime compatibility | Cosmo-header signal/context shim; checked traps with site/status, unknown-signal forwarding, stack overflow on main and worker threads, page probing, allocation failure/panic/OOM reporting, repeated thread start/join/failure, isolated TLS/destructors and contended synchronization | Decide whether a bounded GNU-component compatibility model is defensible; otherwise stop or approve a dedicated runtime-build investigation |
| Real multi-OS execution | Transfer the exact same hashed x86-64 artifact to at least one real non-Linux OS; run core success/failure/thread tests through a documented loader route, without rebuilding it per OS | Decide whether to adopt a restricted executable profile; Linux-only success is insufficient |
| Restricted preview integration | Reviewed capability set, explicit negative tests, target/archive/cache validation, pinned driver and reproducible offline packaging; preserve native GNU tests | Admit only named profile/OS/version/CPU configurations as experimental, with limitations |
| Broader support qualification | Module/dependency and OS security semantics pass independently; extracted-release tests on every claimed OS/ISA/version baseline, deployment policy and notices complete | Support announcement only for the qualified matrix; no automatic admission of upstream’s entire list |
| Optional fat output | Independent AArch64 Cosmo ABI/runtime qualification, matching features and contracts, apelink packaging, identical combined artifact exercised on real x86-64 and AArch64 systems | Decide whether fat output merits its maintenance cost; not a prerequisite for x86-64 multi-OS support |
The runtime compatibility gate must not disable trap reporting or stack protection to obtain a green result. If signal-context rewriting cannot be made safe, compare helper-call checks explicitly while retaining proper stack fault handling; document the performance/semantic consequences before proceeding. No requirement here implies stack-overflow recovery: preserve the current executable diagnostic and abort behavior, and separately qualify any admitted actor trap containment.
Testing and CI design
Keep native GNU/Linux regression jobs unchanged. Add a separately named, opt-in experiment lane before release jobs: build the compiler fresh, record provenance, compile a behavioral corpus once, and publish hashes and artifacts for real-OS jobs. Non-Linux jobs consume those artifacts, not recompile convenient OS-specific replacements. Cross builds and emulators supplement ABI debugging; they cannot establish real signal, loader, filesystem-policy or scheduling behavior.
Exercise debug/release, cache on/off, paths with spaces, failure injection, large/recursive frames, integer and collection traps, foreign unknown faults, handler coexistence, thread startup failure and repeated cleanup. Test allocation balance and per-thread state after jobs. Check profile rejection via direct, indirect and generic calls, unavailable optional engines, wrong runtime manifests, stale caches, missing tools and direct-exec/loader errors. Add full compiler and runtime all-feature suites and executable documentation tests for admitted features; explicitly label exclusions instead of broadly skipping non-Linux failures. Library mode requires separate host-coexistence and ABI tests if admitted.
For filesystem/network expansion, include adversarial rooted paths, native names, resource leaks, cancellation/readiness, DNS, TLS entropy and handshake errors, process cleanup and unavailable facilities. Reuse the contracts in the native OS semantics assessment, not guessed Linux errno mappings. Check absence of accidental dynamic GNU/Linux dependencies and unqualified syscall paths in the final reachable runtime.
Release qualification additionally needs a clean extracted-package test with only documented tools, no Rust/Cargo or network for end users, correct status propagation, loader extraction failure/noexec cases, and OS download/launch policy including signing where required. Record which machines, versions, CPUs and routes actually ran. CI configuration alone does not qualify a platform; real runners and retained passing results are a go/no-go dependency.
Official sources and source-credit boundary
The spike checked official sources over HTTPS and retained copies. These links are versioned where possible; no additional external platform claim is needed for this proposal:
- Cosmopolitan 4.0.2 release: downloaded toolchain baseline and upstream scope.
- Toolchain README at 4.0.2: per-ISA drivers,
apelink, loader paths, installation/gotchas and Legal section. Retained bundle copy:cosmocc/README.md. - Inspected current toolchain README snapshot: context only, not a moving version pin; retained as
official-current-toolchain.md. - sigaction definition: checked against the C layout probe; retained as
official-sigaction.h. - TLS header and object fixups: TIB and eligible TLS instruction adaptation; fixup source retained as
official-fixupobj.c. - APE source: loader/container mechanics; retained as
official-ape.S.
The spike’s ape-readme.md is a 404, not supporting evidence. No upstream code,
data, toolchain or executable is added to the repository by this documentation
proposal; implementation-stage redistribution requires the audit described in
pinning and licensing.
Native macOS, Windows, and AArch64 support
Status: Research and design update (2026-10-31). Linux AArch64 is implemented in the ARM feature branch, with qualification limits recorded in the development guide. macOS and Windows remain proposed. This document does not announce their support. Priorities and implementation work packages live in B18 of the backlog, with B17 delivery and B12 path-contract dependencies.
Related design records cover FFI and dynamic libraries, generated export contracts, and the independent GCC backend candidate. This assessment concerns complete native toolchains, not only instruction selection.
Plenty can plausibly reach these platforms while keeping Cranelift, but this is a compiler/runtime/stdlib port, not just a release matrix expansion. Start with native builds: a compiler running on one OS/architecture generates programs for that same OS/architecture. Cross-compiling the compiler with Cargo is a separate build-system concern; compiling a Plenty program for a different target would additionally need a target ISA, runtime, SDK and linker. Defer that user feature until native ports are qualified.
Proposed scope
| Native host and generated-program target | Present evidence | Proposed treatment |
|---|---|---|
x86_64-unknown-linux-gnu | Explicitly admitted, tested and released | Preserve existing baseline throughout |
aarch64-unknown-linux-gnu | Implemented; extensive QEMU execution, native/release gates outstanding | Preserve alongside x86-64 Linux during refactoring |
aarch64-apple-darwin | Rejected by Plenty; native Cranelift/C ABI smoke tests pass in CI | First macOS port after common target/build work |
x86_64-apple-darwin | Not admitted | Deferred; no automatic promise from the ARM macOS port |
x86_64-pc-windows-msvc / x86_64-pc-windows-gnu | Neither admitted; native MSVC Cranelift/C smoke tests pass, GNU remains separate | Compare SDK/CRT/dependency feasibility before choosing one initial Windows ABI |
aarch64-pc-windows-msvc | Rejected today; ARM64 COFF relocation coverage is an additional backend risk | Separate feasibility gate before committing to a release |
All proposed targets are 64-bit little-endian. ARM32, big-endian AArch64, Android, iOS, Linux musl, ARM64EC, and universal macOS binaries are outside this initial scope. AArch64 support is not a promise to support every ARM board, OS or ABI. WSL runs the Linux build; it is not native Windows support. Minimum OS/kernel versions and CPU baselines must be chosen and tested per target; do not inherit Rust’s support tiers as Plenty’s own guarantees.
Evidence baseline
This update inspects Plenty 869e1cb8087c126ec26a620149bd74d2290c4ca7 and
Cranelift 0.137.0. The October 19 assessment described an older x86-only
checkout; its assumptions about hard-coded native calling conventions and
x86-only crypto assembly are superseded by the ARM implementation.
The current compiler has a two-target description, derives native C calling
conventions through Cranelift, and builds matching ARM runtime dependencies.
The target foundation centralizes frontend facts and binary identity in
src/target.rs, and checks the compiler host against the packaged runtime.
Metadata layout and object handling still implement ELF, and runtime services
remain Linux-specific. This is a useful starting point, not a finished OS
abstraction.
The immediate experimental scope is ARM64 macOS and x86-64 Windows. Preserve the Linux milestone in Git; perform independent work in child worktrees and merge validated increments into the portability branch. No local Mac or Windows machine is assumed: actual OS execution requires CI runners. Configuring a job or cross-linking an artifact is not execution evidence.
Recommended architecture and decision boundaries
Keep portable language semantics and checked Plenty library policy above a small target-specific native boundary. Centralize target data: triple, pointer and scalar layout, C ABI, object format, linker flavor, artifact suffixes, runtime identity, supported facilities and minimum platform. Each source of this information must agree; an allowlist expansion is the last enablement step, after validation, not the first workaround.
Use separate runtime platform modules for Linux, Darwin and Windows operations.
Prefer Rust std where its allocation, progress and error behavior meet Plenty’s
contracts; use reviewed OS bindings where they do not. Avoid copying raw Linux
constants into a generic module or translating every OS failure to a guessed
Linux errno. Unsupported facilities need an explicit, documented outcome;
security-sensitive operations must fail closed rather than silently weaken.
The detailed assessment is split into:
- Interface boundaries and source mapping.
- Compiler, runtime build, and native ABI.
- Standard library and OS semantics.
- Qualification, delivery, and reference implementations.
- Native ABI and Zig toolchain experiments.
Study both OS ports before fixing shared interfaces, then establish a minimal macOS path and the selected Windows path. Port OS services against explicit contracts before claiming parity. Keep Cranelift as the production backend; Zig is a candidate C/dependency compiler and linker driver. User-facing Plenty cross-compilation remains a separate admission decision.
Interface boundaries and source mapping
Proposed organization, based on Plenty
869e1cb8, with the shared target foundation now extracted intosrc/target.rs. No new target is admitted by this design. Work packages and priorities belong to B18.
Separate three identities
Cargo’s build host runs build scripts. The compiler host runs Plenty and its linker driver. The generated-program target supplies the ISA, ABI, runtime and system libraries. Native releases currently equate the latter two; cross-building the Rust compiler does not change that relationship. Preserve separate identities even while enforcing equality at admission.
Use immutable target facts for triple, OS, architecture, object format, native calling convention, minimum system version and artifact naming. Derive these from one checked description; do not duplicate target-name comparisons in the frontend, cache, package tooling and release scripts. Cache identity must include runtime/ABI compatibility and features, not just the CPU name.
Focused interfaces
| Owner | Shared operation | Implementation boundary |
|---|---|---|
| Compiler host/tool invocation | Discover tools; execute an argument vector and report errors | Host process/filesystem APIs; explicit target linker/SDK configuration |
| Target codegen | Build ISA; describe native calls; enumerate special probe sites; decode instructions | Reuse Cranelift TargetIsa; small Plenty-specific architecture helpers |
| Object emission and inspection | Retain/read interface records; resolve source/trap records; export symbols | ELF, Mach-O and COFF implementations owning their symbol/section state |
| Linking/distribution | Construct a link plan; locate dependencies; validate an installed engine | Driver-specific argument arrays and format-specific dependency inspection |
| Runtime OS services | Open/close/wait/spawn; obtain entropy; classify errors | Linux, Darwin and Windows service implementations with typed resource ownership |
| Fault recovery | Read saved PC/SP; enter a helper; recover an actor or terminate safely | OS-and-architecture context/exception implementations, selected at build time |
| Plenty standard library | Portable policy, ownership and meaningful results | Native backend normalization; if const only at deliberate source boundaries |
A single object combining these responsibilities would couple compiler state to runtime mechanics. Prefer modules with identical functions for build-time choices, small traits for genuinely interchangeable behavior, and data for facts. Keep stateful symbol/relocation work inside the emission component; passing the entire compiler through a platform trait would defeat the boundary. No global platform initialization, allocation or dynamic dispatch should be introduced into signal handlers merely to satisfy this organization.
Plenty protocols provide static constraints where a library genuinely accepts multiple providers. They need not be threaded through every application to use files or networking. Platform selection belongs behind public standard-library interfaces. Protocols do not supply missing OS behavior or dynamic dispatch.
Compiler changes with concrete owners
| Current location | Dependency to hide | Acceptance evidence |
|---|---|---|
src/target.rs, src/frontend/target.rs::fact | Shared facts now cover the two Linux targets; OS admission and linker policy remain separate work | Frontend facts agree with runtime manifest and emitted object for every admitted target |
src/codegen/output.rs::finish, src/library_metadata.rs | GNU ELF flags and ELF-only metadata reading | Format-aware objects; metadata survives executable/static/shared library dead stripping |
src/codegen/exports.rs, sites.rs, traps.rs, coverage.rs | Section names, relocation and linker retention assumptions | Source/trap/coverage lookup and exported contracts after relocation and installation |
src/toolchain.rs, src/library.rs, src/toolchain/distribution.rs | GNU options, ELF SONAME/RUNPATH and .so names | Driver argument tests plus real link/load/relocation tests with spaces in paths |
src/codegen/stack_probes.rs, runtime signal_context.rs | Pinned ARM instruction sequence and Linux context fields | Registered probes only; native overflow and unrelated-fault tests; handler coexistence |
build.rs, engine/release helpers | Direct dependency builds and Linux assembly/tool conventions | Matching compiler/runtime/native objects; reproducible offline build and artifact audit |
Cranelift 0.137 provides AppleAarch64 and WindowsFastcall, and Mach-O/COFF
object emission. That is feasibility evidence, not a working Plenty port. Keep
internal Tail calls distinct from native adapters. Test narrow signed/unsigned
arguments, mixed float/integer register exhaustion, u128 and callbacks against
both Rust and the chosen C toolchain. Windows GNU and MSVC are separate cases.
Audit unwind information explicitly: Plenty uses define_function_bytes, which
bypasses the object backend’s automatic unwind processing. The pinned backend
also has limitations for automatic Windows and Mach-O unwind emission. Retaining
panic-abort does not settle OS exception/stack-walking requirements. Native stack
probes, handler coexistence and actor recovery are separate admission gates.
Runtime and language contracts
| Plenty surface / current dependency | Reference implementation to inspect | Proposed boundary and required tests |
|---|---|---|
files.rs, aggregates/filesystem.rs; Unix descriptors, no-follow, temporary publication | Rust sys/fs/windows.rs; CPython Modules/posixmodule.c, Python/fileutils.c | Typed resources and rooted operations; rename/link races, sharing, delete-open behavior, atomic publication and confinement |
stdlib/std/path.plenty, environment and arguments; native Unix bytes | Rust sys/os_str, sys/process/windows.rs; CPython Lib/ntpath.py, _winapi.c | Lossless native names and explicit conversions; UNC/drive paths, unpaired wide units, invalid Unix UTF-8, Unicode and spaces |
aggregates/readiness.rs, mutex.rs, cancellation; epoll/eventfd/futex | Rust sys/sync, sys/net; CPython Modules/selectmodule.c | Wait/wake contracts covering sockets, pipes and cancellation; lost wakeups, EOF, stale registrations, timeout and handle reuse |
stdlib/std/fs/watch.plenty, aggregates/watch.rs; inotify records/masks/cookies | CPython selector backends help waiting, but do not supply equivalent watch semantics | Normalize events below portable policy; decide rename pairing, overflow and recursive behavior without inventing guarantees |
aggregates/net.rs, stdlib/std/net.plenty, report.plenty; Unix handles and numeric errno | Rust sys/net/connection/socket/windows.rs; CPython Modules/socketmodule.c | Distinguish socket ownership and OS error domains; partial I/O, would-block, DNS lifetime, SIGPIPE and cancellation |
aggregates/process.rs, files.rs; spawn/pipes/inheritance | Rust sys/process/windows.rs; CPython Modules/_winapi.c, _posixsubprocess.c | Literal arguments, explicit inherited handles, bounded pipe draining, missing executable, timeout, termination and reap |
threads.rs, actor_heap.rs, signals.rs, stack_overflow.rs | Rust sys/thread/windows.rs, sys/pal/{unix,windows}/stack_overflow* | Fallible startup without hidden allocation; native fault/exception recovery, cleanup, shutdown notifications and concurrent first use |
entropy.rs, crypto/TLS/compression dependencies | Rust sys/random/{apple,windows}.rs; CPython Python/bootstrap_hash.c | Fallible secure entropy, no weak fallback; known-answer tests and CPU-dispatch checks |
Reference paths are relative to the read-only checkouts and revisions recorded in delivery. They identify patterns and tests to study, not code licensed for uncredited copying. Prefer Rust facilities where their fallibility, allocation and progress semantics fit; borrowed implementation code/data needs its license and attribution handled explicitly.
The most consequential API decisions are Windows native path representation, readiness versus completion, filesystem event guarantees, OS error domains and shutdown signals. Record these before implementing backends. Security guarantees must survive the port; an unavailable operation must fail explicitly instead of silently following symlinks, losing filename data or weakening entropy.
Contracts requiring decisions before mechanical refactoring
stdlib/std/process.plenty encodes normal exits as 0–255 and POSIX signals as
256 plus the signal number. Windows native 32-bit exit codes require a distinct
representation; preserve full status and distinguish termination from normal
exit. Startup argument capture in text_io.rs also assumes C argc/argv bytes;
compare Rust’s GetCommandLineW handling before choosing Windows behavior.
The generated mutex fast path assumes a u32 state word with states 0/1/2, while the runtime promises no allocation. Rust’s Darwin mutex implementation uses pthread primitives, and its Darwin thread parking uses public libdispatch semaphores rather than private address-wait APIs. A direct substitution must therefore prove layout, initialization and allocation contracts or change the Plenty representation/lowering deliberately. Rust’s Windows implementation uses address-based waiting; establish the supported Windows baseline before reuse.
control::Token stores a Unix OwnedFd even outside Linux-only operations.
Separating function bodies alone will not make Windows compile: resource
representations must also become platform-specific. Filesystem watching needs
an explicit choice between retaining the Linux-native API and introducing a
portable change/rescan contract; equivalent-looking notifications are not proof
of equal rename, overflow or event-order guarantees.
Compiler, runtime build, and native ABI
Part of the cross-platform assessment. Proposed changes below are not implemented support.
Build the compiler and the runtime for the same machine
build.rs emits application and library runtime archives, native test hooks
and a fuzz driver. The ARM work added target-aware C/archive tool discovery,
matching Linux ARM ring assembly and dependency settings. Darwin/Windows
selection still needs implementation: the current ring lists select ELF/Linux
assembly, and direct Rust builds reproduce dependency build-script behavior.
Introduce one checked native-toolchain description consumed by runtime and native dependency builds. Distinguish Cargo HOST (build scripts run here), Cargo TARGET (compiler executable and embedded runtime run here), and any future generated-code target. Use native CI initially. For later cross-builds, require matching C tools, assembler, archiver, Rust target libraries and sysroot; reject inconsistent inputs before producing a package.
Prefer a separate, orchestrated runtime build using upstream Cargo build scripts
and a locked dependency graph, then embed verified artifacts into the compiler.
This avoids maintaining copies of ring/libc/getrandom platform logic. It must
preserve the current self-contained compiler distribution, both runtime modes,
offline builds, feature agreement and reproducibility. Do not invoke Cargo
recursively against the same target directory from build.rs and risk lock
contention. Compare this approach with a dedicated target-aware native build
helper before changing the build architecture. If direct rustc builds remain,
explicitly reproduce and test every dependency’s target-specific build-script
outputs, features, native inputs and licenses.
Audit SQLite compilation and platform defines, compression CPU feature paths, crypto entropy/provider dependencies, native test hooks and libFuzzer separately. The runtime workspace’s Cargo build and the embedded build must behave alike. On Windows choose compatible CRT linkage and a documented SDK/toolchain; allocations crossing library boundaries must still be freed by their owner. Keep Rust panic-abort and Plenty’s no-unwinding contract. Do not introduce an implicit Rust installation requirement for release users.
Code generation and ABI correctness
Keep the internal Plenty Tail calling convention distinct from the target C ABI. Derive native signatures from the target and audit all boundaries: runtime helpers, entrypoints, threads, callbacks, imports, exports, dynamic library discovery, and compiler-generated adapters. Replacing one SystemV assignment does not cover the others. Small integer extension, booleans, float arguments, aggregate returns, hidden return pointers, stack arguments and callee-saved registers need C/Rust/Plenty round trips on each target.
Retain Plenty’s specified layouts where possible, but separately establish
foreign layouts. Windows’ C long is not a Unix 64-bit long; OS structs,
socket handles and pthread_t cannot be inferred from pointer size. Check i128
and floating-point lowering/libcalls, alignment of aggregates and atomics,
thread entry storage, and library contract hashes against the selected ABI.
CPU baselines must not accidentally depend on the release builder’s processor.
Validate direct and indirect tail calls with spilled arguments, deep recursion,
large stack frames, closures, generators and worker threads. Keep stack probing
and guard-page protection: disabling them to make a build pass is not a port.
plenty-runtime/src/stack_overflow.rs requires separate architecture/OS handling,
including Darwin fault context and Windows exception/stack guarantees. Library
runtimes must continue respecting the embedding process’s handlers. A fallback
may preserve safe termination without identical diagnostic text, but must be
specified and tested rather than misclassifying arbitrary memory faults.
Objects, libraries, and linker drivers
| Concern | Linux GNU | Darwin | Windows MSVC |
|---|---|---|---|
| Object / executable | ELF .o, conventional executable name | Mach-O .o, conventional executable name | COFF .obj, PE .exe |
| Native libraries | .a, .so | .a, .dylib | static .lib, .dll plus import library |
| Driver and SDK | GCC/Clang with libc development files | Apple Clang and selected macOS SDK | MSVC or qualified clang-cl/lld-link with Windows SDK |
| Retention/export policy | Existing GNU sections/version scripts | Mach-O section/segment limits, exported symbols and dead stripping | COFF sections, export tables/import libraries and retention rules |
Represent link arguments as argument arrays, including paths with spaces;
current runtime link strings are whitespace-split. Add driver-specific command
construction and discovery rather than making MSVC pretend to be cc -o.
Audit startup, system libraries, minimum OS flags, executable naming, object
archive determinism and diagnostics for missing toolchains. LibFuzzer and
sanitizers may initially be explicitly unavailable where not qualified.
src/library_metadata.rs requires ELF for the admitted Linux architectures and
.plenty.interface.* sections. Define target-specific metadata sections and
validate architecture, endianness, bounds and contract identity for Mach-O and
COFF, including archive members. Mach-O naming constraints make simply reusing
the full ELF section name unsuitable. Enable the required object reader features
deliberately. Test object-only output, interface extraction, C headers, static
exports and dynamic exports independently of ordinary executables.
Runtime loading currently uses dlopen, dlsym, dlclose and GNU
RTLD_NODELETE. Darwin needs its own flags/lifetime implementation; Windows
needs wide-path loader calls, safe DLL search rules, exports and handle ownership.
Preserve the existing invariant that callable code stays resident after lookup
leases close, along with allocator provenance and instance identity checks.
If residency cannot be provided safely, reject the facility until it can; do
not map the existing close operation to an unload that leaves dangling code.
Compiler tools are part of the port
Audit src/project.rs, src/cache.rs, src/lsp.rs and test helpers for Unix
exit status, executable suffixes, chmod/symlinks, replacement/rename semantics,
open-file deletion, shell invocations and cache directories. Cache keys already
include target/ISA information; preserve it and verify runtime, ABI and feature
changes invalidate artifact reuse. Test concurrent writes on Windows, where
open-file sharing rules differ, and choose platform-native cache locations while
preserving explicit overrides.
The LSP manually parses file:// URIs and emits paths through lossy strings.
Add correct drive-letter/UNC handling, escaping, case behavior and non-ASCII
round trips; keep OS-native paths until presentation. Test package roots with
spaces, Unicode and mixed separators, Git dependencies, command arguments,
timeouts and child cleanup. Compiler startup and --check are necessary but
insufficient acceptance tests for a working developer installation.
Standard library and OS semantics
Part of the cross-platform assessment. These are proposed contracts to resolve before implementing native ports.
Paths, text, and filesystem guarantees
std.path.Path stores native Linux bytes, exposes as_bytes/from_bytes, and
checks an absolute path by its initial slash. std.env and filesystem/process
helpers also exchange native bytes. Windows uses UTF-16 OS strings, drive roots,
UNC paths and device names; replacing separators or converting invalid text
lossily would change filenames and could undermine confinement.
Choose a native-path representation with lossless round trips before Windows
implementation. Recommended direction: an opaque OS-native Path and explicit
text conversion; keep raw Unix-byte constructors/accessors platform-specific,
and provide an explicit Windows wide-unit boundary if needed. Consider a
documented lossless internal encoding, but never describe that encoding as UTF-8
text or portable filesystem bytes. This is a public API migration requiring
compatibility decisions, tutorial examples and tests, not merely a private FFI
change. Reject unsupported conversions explicitly. Process arguments and
environment names/values need the same policy; source code remains UTF-8.
Preserve lexical operations separately from filesystem resolution. Test drive relative paths, UNC/device prefixes, reserved names, trailing dots/spaces, case-sensitive and case-insensitive volumes, Unicode normalization, long paths, symlinks and Windows reparse points. macOS paths are not automatically case-sensitive just because they are Unix paths. Archive member names are a portable format namespace; extraction into native paths must reject traversal and collisions under the destination filesystem’s rules.
aggregates/filesystem.rs uses descriptor-rooted no-follow operations and Linux
O_TMPFILE plus /proc/self/fd publication. Darwin needs an alternative
temporary-file/publication strategy. Windows needs owned directory/file handles,
reparse-point policy, sharing/deletion rules and an honest atomic replacement
contract. Security must be evaluated under concurrent rename/link replacement,
not only clean-directory tests. Never replace rooted operations with unchecked
string concatenation. If a promised no-follow/atomic operation has no adequate
backend, return a documented unsupported result. HTTP static files, multipart
uploads and archive extraction inherit these requirements.
Handles, errors, readiness, and networking
The current runtime uses Rust File/Unix descriptors for several resource kinds.
Windows HANDLE, Winsock SOCKET and C-runtime descriptors are distinct; an
integer cast and a common close are not a resource abstraction. Introduce
typed owned native resources behind existing language ownership, with distinct
close/read/write/wait behavior and cancellation-safe lifetimes. Audit runtime
tags and compiler intrinsics that currently assume a file-shaped resource.
std.net and std.report classify numeric Linux errno values. Move portable
classification to the native boundary and preserve original OS domain/code
for diagnostics. Decide how existing public IoError/NetError variants expose
that information without losing partial progress. Darwin errno, Win32 errors
and Winsock errors are not interchangeable. Broken pipe, interrupted operation,
would-block, timeout and cancellation must retain their specified distinctions.
Review error formatting (strerror_r variants included), newline behavior,
binary/text console I/O and SIGPIPE suppression.
| Facility | Current assumption | Proposed port boundary |
|---|---|---|
std.ready, cancellation | epoll/poll and Linux eventfd | Darwin kqueue or qualified poll/wakeup implementation; Windows design for sockets and pipes/files plus wakeable cancellation |
std.net, DNS | Linux socket constants and hand-written structs; pthread resolver helper | OS bindings, Winsock initialization/error mapping, bounded resolver lifetime and platform address conversion |
std.signal | Linux signal delivery on both admitted architectures | Darwin signal backend; explicitly define Windows console/service shutdown capabilities instead of pretending POSIX signals exist |
std.process | Linux implementation, Unix status/signals and pipes | Native argument-vector spawn, inheritance control, Windows quoting/handle lists, termination/reaping and bounded output |
std.secrets | Linux getrandom; unsupported elsewhere | Fallible OS entropy primitive per platform, preserving no-weak-fallback guarantee |
std.time | Rust clocks plus deadline contracts | Verify monotonicity, precision, overflow, suspend behavior and timed waits per backend |
Windows IOCP is completion-based, while the public ready API is readiness-based.
Choose the bridge deliberately: a readiness abstraction for supported resources,
or a completion-backed adapter that owns buffers and preserves lifetimes. A
socket-only select/poll replacement does not implement waiting on process
pipes and cancellation tokens. Account for handle limits, stale registrations,
EOF/hangup, cancellation races and deadline rounding. macOS also needs tests
for event removal and descriptor reuse; sharing a Unix API shape is insufficient.
Threads, TLS, and higher layers
threads.rs directly declares pthread functions and fixes task storage to a
machine-word thread identifier. Use target-specific thread creation/join and
entry ABI while preserving pinned caller-owned storage, fallible startup and
join-before-reclaim. Rust’s boxed thread startup is a useful semantic reference,
not an automatic replacement for Plenty’s allocation contract. Review stack
sizes, TLS initialization and cleanup, atomics/alignment and memory ordering.
AArch64’s weaker ordering makes native concurrency stress especially important;
channels and executors must not rely on x86 behavior.
Rustls protocol logic is broadly reusable, but the vendored ring build, entropy, socket progress, readiness and certificate paths must first work. Preserve verified TLS, caller-selected trust roots, mutual TLS, ALPN, SNI and revocation contracts. Native OS certificate stores would be a separate trust-policy feature, not an implicit change in a port. Requalify HTTP pooling, shutdown, deadlines and file serving after lower layers pass; parser tests alone do not prove an HTTP server port.
Collections, JSON/TOML/CSV, URL parsing, encoding and much of regex/compression should need fewer OS changes, but still require target execution, allocation failure and integer/Unicode compatibility checks. SQLite requires its correct OS VFS/build configuration, locking, durable-write and path tests. Math/float edge behavior, endian packing and CPU-dispatched crypto/compression need their own fixtures. Avoid blanket gating that turns a green test suite into an undocumented subset of the standard library.
A limited preview may explicitly omit unqualified modules. Before a general release, publish a capability matrix and ensure unavailable operations are rejected predictably without unresolved symbols, compiler panic, unsafe fallback or silent loss of guarantees. The current reference remains authoritative until each implemented port updates its contracts and learner-visible examples.
Qualification, delivery, and reference implementations
Part of the cross-platform assessment. These are dependency gates and design criteria; priorities remain in B18 of the backlog, not a second work queue.
Evidence required before calling a target supported
| Gate | Prerequisites and acceptance evidence |
|---|---|
| Native backend experiment | Build an object, link and execute C/runtime calls, aggregate returns, indirect Tail calls, large frames and library metadata on the exact OS/ISA. Probe ARM64 COFF relocations before promising Windows ARM64. |
| Target/build boundary | Compiler and both embedded runtimes report matching target/ABI; target-aware native dependencies and drivers work offline; incompatible targets are rejected early. Both Linux architectures remain green. |
| Runtime core | Allocation failures, reporting, stack guards/overflow, thread start failure/join, channels, atomics, files and foreign ABI round trips pass natively. No unsafe success stubs. |
| OS and stdlib | Native paths and resources, rooted filesystem guarantees, process cleanup, cancellation/readiness, DNS, TLS and HTTP pass the platform capability contract. |
| User tooling | Package build/run/test, cache races, LSP path round trips, static/dynamic libraries and supported fuzz tooling are exercised from an extracted release. |
| Distribution | Fresh machine with documented native-linker prerequisites, no Rust/Cargo and no network compiles and runs representative programs; dependency/OS baseline audits and signed-package policy pass. |
The Linux AArch64 experiment can retain Linux semantics while exposing ABI,
native assembly and memory-ordering assumptions. Darwin ports then exercise
Mach-O, Apple ABI and Unix OS differences. Windows follows the decisions about
paths, handles and readiness; Windows ARM64 depends on both the Windows port
and backend feasibility. OS implementations can proceed independently once the
shared contracts settle. Native release jobs must wait for their corresponding
qualification gates; a compiler that prints --version is not enough.
CI and regression strategy
Add native test runners for each admitted tuple, with explicit toolchain/SDK and minimum-version policy. Emulators and cross-compilation can supplement coverage but do not replace real OS execution for thread scheduling, sockets, file races, signals, DLL loading or installer behavior. Runner availability and cost are implementation-time decisions. Plan for CI runners; the user confirmed no local Mac or Windows machines should be assumed. This assessment did not provision runners or publish any workflow.
Inventory Linux-specific fixtures: /proc/self/fd, /tmp, shell scripts,
descriptor counts, errno numbers, ELF tooling, chmod, signals, executable suffixes
and symlink privileges. Keep portable behavioral tests common and replace
platform observations with equivalent checks. For example, verify resource
return to an observed baseline, using descriptor counts on Linux and qualified
handle accounting on Windows. Preserve adversarial leak/race/security tests;
do not just skip every failure under a broad non-Linux condition.
Run compiler and runtime workspace suites with all features, tutorial examples, allocation injection and relevant release-helper tests. Test debug and release generated programs, cache on/off, library hosts and parallel execution. Include non-ASCII and non-text native names, paths with spaces, inheritance leaks, timeouts, repeated cancellation, DNS cleanup, TLS handshake errors and HTTP shutdown. Keep environment-dependent Unicode/version assertions semantic rather than tied to incidental toolchain output.
Release packaging and user prerequisites
Extend the existing tag-verified, locked, draft-then-publish workflow to build one archive per admitted tuple. Preserve tag/version checks, immutable published artifacts, checksums, licenses and third-party notices; the ARM work already parameterizes Linux compiler and engine artifacts and notices by target. No crates.io publishing is needed or proposed. Keep build, test and publication permissions separated.
Retain .tar.gz on Unix; use .zip on Windows, with plenty.exe and
plentyc.exe. Update packaging/smoke helpers that assume one Linux suffix or
archive glob. Every archive must contain target-specific installation guidance
and the same required resources/notices. Test extraction and PATH setup as an
ordinary user; .exe naming must propagate into packages and test runners.
For Linux AArch64 preserve and qualify the configured glibc 2.28 baseline and ARM dynamic-loader/dependency allowlist. Minimum-baseline release execution remains outstanding; configuration alone is not qualification. For macOS choose deployment targets, audit Mach-O architectures/install names and linked libraries, and test Apple Command Line Tools/SDK discovery. For Windows choose supported OS, SDK and CRT requirements; audit PE imports and run from a clean environment with only the documented native tools. The compiler still needs a native linker to build programs; bundling a compiler driver/SDK would be a distinct distribution/licensing decision.
The user-facing support promise must distinguish a full native port from a preview with explicit unavailable modules. A successful compiler build must not silently turn skipped actor/engine/watch tests into parity evidence.
Decide macOS Developer ID signing/notarization and Windows Authenticode policy before public downloads. Credentials, account access and signing runners are prerequisites for that release stage, not assumptions this proposal fulfills. Separate unsigned reproducible build checks from signed-artifact verification and document timestamp/signature differences. Test macOS quarantine/Gatekeeper and Windows download/launch behavior; signing is not a promise that every reputation prompt disappears. A preview must state unsigned status honestly.
Inspected reference implementations
The local checkouts were used read-only, without copying their implementations. Paths below are relative to the named checkout, making them useful even when the repos are not adjacent to a reader’s Plenty checkout. These snapshots are design evidence, not claims about the latest upstream support policy.
Rust, ../rust/, commit
080cc27146389dbb5c8772c476b1937be9c6110f:
| Inspected path | Transferable lesson |
|---|---|
library/std/src/sys/pal/mod.rs | Select platform implementations beneath a shared interface |
library/std/src/sys/fs/windows.rs | Wide-path file APIs and owned handles, rather than Unix descriptor emulation |
library/std/src/sys/process/windows.rs | Native UTF-16 environment/command handling and argument quoting require explicit policy |
library/std/src/sys/net/connection/socket/windows.rs | Winsock resource ownership and socket errors differ from file handles |
library/std/src/sys/thread/windows.rs | Native entry ABI, startup failure cleanup and join ownership; do not copy its boxed allocation strategy blindly |
library/std/src/sys/pal/unix/stack_overflow/{mod,handler_signal}.rs, library/std/src/sys/pal/windows/stack_overflow.rs | Stack protection/diagnostics need OS-specific setup and handler coexistence |
library/std/src/sys/random/{apple,windows}.rs | Entropy source selection depends on platform; adapt to Plenty’s fallible contract instead of copying panic behavior |
compiler/rustc_target/src/spec/targets/{aarch64_apple_darwin,aarch64_pc_windows_msvc}.rs | Architecture, ABI, baseline and atomic capabilities belong in target descriptions; rustc support does not prove Cranelift coverage |
CPython, ../cpython/, commit
b3714a5fb7a3aadd72134ee0710b22519c1df09a:
| Inspected path | Transferable lesson |
|---|---|
Modules/posixmodule.c, Python/fileutils.c | File operations and lossless native names need platform boundaries; Python’s surrogate policies are precedents, not a requirement to alter Plenty strings |
Modules/socketmodule.c | Separate socket closing/error behavior and platform capability detection |
Modules/selectmodule.c | poll/epoll/kqueue and Windows socket selection are different capabilities; one selector call cannot cover every resource |
Modules/_winapi.c | Windows process/pipe/handle mechanics deserve a dedicated implementation |
Python/bootstrap_hash.c | OS entropy paths, size limits and failure handling; never weaken randomness for portability |
Lib/ntpath.py | Drive, UNC and native path rules need a distinct model from POSIX slash operations |
These references show patterns to adapt within Plenty’s ownership, bounded resource and explicit-allocation rules. They do not license assuming equal semantics across systems. No Rust or CPython changes are proposed.
Decisions still open
Before implementation, settle the supported OS/CPU baselines, Windows native path API migration, readiness/completion bridge, and reduced-preview capability policy. Then choose runtime build orchestration, target-specific tool discovery, and the scope of static/dynamic library support at first release. Confirm ARM64 COFF behavior with an executable experiment. Signing credentials and native CI capacity can be arranged in parallel, but are required before qualified releases.
The highest uncertainty is Windows resource/security semantics and backend coverage, followed by maintaining fallible allocation across portable helpers. Linux AArch64 now supplies the shared-baseline regression target; its native hardware and release gates remain open. The immediate port experiments are ARM macOS and x86-64 Windows, with MSVC versus GNU still a decision gate. No effort estimate is reliable until those focused experiments establish the actual backend and SDK constraints.
Native ABI and Zig toolchain experiments
Evidence from 2026-10-31. These are small toolchain probes, not Plenty port qualification. Decisions and implementation sequence belong to B18; interface boundaries describe the design.
Native Cranelift CI smoke tests
The standalone platform-probes repository
now runs push-triggered native jobs on macOS ARM64, Windows x86-64 MSVC and
Linux x86-64. At probe commit ded48e7d6570dba951a954d0de09e1e93749fc14,
all three jobs passed.
The fixture uses Rust 1.99.1 and Cranelift 0.137.0; each job checks its host
triple, emits a native object, links a C harness using the platform compiler,
and executes seven assertions. Objects, executables and logs are CI artifacts.
The assertions cover ten integer arguments, mixed integer/f64 arguments,
signed/unsigned narrow arguments, generated-to-C calls for those signatures,
and a native entry calling an internal Tail function. Native CI confirms
these paths execute on Apple AArch64 and WindowsFastcall, rather than merely
producing a Mach-O or PE file. This fixture does not compile Plenty itself.
The initial run exposed two harness configuration errors. Enabling Cranelift
is_pic=true removed Darwin’s rejected absolute text relocations; Plenty
already enables that setting. Replacing filesystem canonicalization with
ordinary absolute output paths avoided MSVC driver mishandling of Windows
verbatim path prefixes. Both corrections were verified by the successful run.
This is partial B18.1 evidence. Indirect/tail-return calls, large frames and stack probes, u128/aggregates, Rust ABI boundaries, unwind/exception handling, minimum OS versions and full runtime/dependency builds remain unqualified. Windows GNU is not covered by this native suite; MSVC is the working probe baseline, not yet a qualified Plenty release contract.
The Plenty repository’s portability.yml workflow additionally builds the actual
workspace on these macOS and Windows runners and runs unit tests if compilation
succeeds. On macOS it also runs the AArch64 stack-probe integration tests. It preserves build logs and reports failures while the ports are
incomplete. Existing ci.yml jobs exercise Linux x86-64 and ARM64, including
optional engines and Docker proxy tests. A successful standalone probe is not
substituted for any of these compiler/runtime checks.
Two independent uses
cargo-zigbuild can help build the Rust compiler, its runtime and C dependencies.
Separately, Plenty could invoke zig cc as the driver linking Cranelift-produced
objects with a matching runtime. The latter needs Plenty’s own target-aware link
plan and artifact selection; cargo-zigbuild does not add that user feature.
Cranelift remains responsible for Plenty machine-code generation.
The important Zig mechanism is more than a linker executable: it carries target information, libc headers/sources or interface stubs, compiler support routines, and logic for producing target CRT/import artifacts. These inputs still have coverage and version limits. Full system SDKs and external libraries may remain necessary. Avoid promising arbitrary host-to-target builds based on a simple program, or confusing producing an executable with testing it on that OS.
Source and tool identities
| Input | Inspected identity |
|---|---|
| Plenty | 869e1cb8087c126ec26a620149bd74d2290c4ca7 |
| Local Zig source | 4dd65e63f23569782f1560d40217cc74aae8b6cb (development source) |
| Executed Zig | Installed /bin/zig, version 0.16.0; distinct from the source checkout |
| Executed cargo-zigbuild | 0.24.0; upstream tag resolves to 2ffd80658c96b5ff971085dd509e58e5b441f1c8 |
| Executed Rust | 1.99.1 |
The Zig checkout’s src/Compilation.zig loads bundled Darwin libSystem.tbd
and SDKSettings.json from lib/libc/darwin, and builds MinGW CRT/import
libraries. Its src/main.zig has native MSVC library discovery. These are
specific toolchain facilities, not evidence for portable Plenty runtime behavior.
The cargo-zigbuild documentation
describes target-qualified compiler wrappers, glibc-version selection,
SDKROOT, deployment targets and limitations. Its integration must reach all
of Plenty’s direct C/archive commands, not only Cargo’s final Rust linker.
Pin the tool version and inspect produced imports rather than assuming the
requested libc baseline is sufficient proof.
Executed small probes
An isolated dependency-free Cargo package in /tmp/plenty-zig-port-probe
compiles probe.c using CC_<target_with_underscores>, archives it using
AR_<target_with_underscores>, and links it into a Rust executable. Its C
function calls puts and returns its integer argument plus nine; Rust calls it
with two and asserts eleven. The helper invokes each tool as an executable
path, matching a relevant constraint of Plenty’s direct native build.
| Experiment | Observed result | What it establishes |
|---|---|---|
Rust+C aarch64-unknown-linux-gnu.2.28 | Build succeeded; ELF imported glibc symbols no newer than 2.28; QEMU Cortex-A53 execution exited zero and printed both messages | Working wrapper/archiver handoff and small C ABI call; execution used glibc 2.41, so minimum-libc execution is unproved |
Rust+C aarch64-apple-darwin, deployment 13.0, no SDKROOT | Build succeeded; ARM64 Mach-O reports minimum 13.0, SDK 26.4 and libSystem dependency | Zig bundled stubs suffice for this tiny artifact; no macOS execution occurred |
| Darwin CoreFoundation header | Compilation failed without full SDK headers | Bundled libSystem stubs are not a complete Apple SDK |
Windows MSVC C using stdio.h | Compilation failed without SDK/CRT headers | Local environment does not supply the required MSVC development inputs |
| Standalone Windows GNU C | Produced x86-64 PE32+ with UCRT API DLL imports | Small MinGW artifact construction only; no Windows/Wine execution, and no Rust GNU target installed for a Rust+C probe |
Logs, sources and artifacts remain in that task-owned temporary directory.
linux-run.json records the exact QEMU command and exit status. The successful
builds do not compile Plenty’s embedded runtime, ring/SQLite/compression,
optional engines, generated Tail calls or exception recovery. Those are
explicit subsequent experiments, not inferred successes.
Reproduction inputs
The original fixture and exact environment are recorded in
/tmp/plenty-zig-port-probe/REPORT.md. The executed build commands, from that
fixture directory, were:
export CARGO_ZIGBUILD_CACHE_DIR="$PWD/wrappers"
export ZIG_GLOBAL_CACHE_DIR="$PWD/zig-cache"
export CARGO_TARGET_DIR="$PWD/target"
export CARGO_BUILD_JOBS=1
cargo zigbuild --offline --target aarch64-unknown-linux-gnu.2.28 -v
MACOSX_DEPLOYMENT_TARGET=13.0 cargo zigbuild --offline --target aarch64-apple-darwin -v
cargo zigbuild --offline --target x86_64-pc-windows-msvc -v
zig cc -target aarch64-macos -c framework.c -o framework.o
zig cc -target x86_64-windows-gnu windows.c -o windows-gnu.exe
The first two commands succeeded; the MSVC and framework commands failed as
recorded above. framework.c includes CoreFoundation/CoreFoundation.h and
references CFGetTypeID(kCFBooleanTrue); windows.c prints one message from
main using stdio.h. All dependencies are system/toolchain inputs; no crates
were downloaded for the fixture. The QEMU command was:
qemu-aarch64 -cpu cortex-a53 \
-L /tmp/plenty-arm-toolchain/aarch64-buildroot-linux-gnu/sysroot \
target/aarch64-unknown-linux-gnu/debug/plenty-zig-port-probe
Recorded SHA-256 artifact identities (build paths affect reproducibility):
| Artifact | SHA-256 |
|---|---|
| Linux Rust+C ELF | 9e248f61e6cedff324c9e0c7e25eb757432d8348bce96f5b8bd0a19d6ded360f |
| Darwin Rust+C Mach-O | 78784e5bc56a76a2f98552b486706120a6121816a72ecfe7e5eb27bc0a7804d1 |
| Windows GNU C PE | 46b4e2498719abb500ba3f6a35b114e1b827882eb2bc04a665f1de2c3e32948c |
Admission experiments
Run these in isolated probe packages before expanding Plenty’s target allowlist:
- Use pinned Cranelift to emit native wrappers around direct/indirect
Tailcalls, spilled arguments and large frames for each candidate triple. Link and execute on CI; inspect actual relocations, stack probes and unwind records. - Build both actual embedded runtime variants and their C/assembly dependencies with the proposed toolchain. Compare Cargo and direct-build configurations, panic-abort aliases, archive member architectures, imports and notices.
- Round-trip signed/unsigned narrow integers, u128, floats, stack arguments and callbacks through Rust/C/Plenty. Include separate Windows GNU and MSVC fixtures before selecting one distribution contract.
- Exercise interface/source/trap/coverage metadata through static/shared links, dead stripping, relocated installation and library hosts. Verify resident code lifetime after loader leases close.
- Build and execute dataframe/linalg using the supported packaging helper. Audit platform library identities/search paths and allocator ownership.
- Repeat using pinned full Apple SDK/Windows SDK inputs and minimum supported systems. Audit imports and execute offline from extracted packages without Rust/Cargo. Cross-linking remains supplementary to native CI execution.
Decisions informed by the probes
Keep native macOS/Windows runners as the correctness baseline. Choose the Windows ABI after actual dependency and runtime probes; native MSVC is a useful first candidate, while MinGW is a separate CRT/link/distribution choice. A successful GNU C executable does not establish MSVC compatibility.
For macOS, distinguish a libSystem-only artifact from SDK/framework-dependent builds and record deployment target explicitly. For Linux, retain the existing glibc floor unless evidence warrants changing it; the user permits raising the floor to 2.30, but these probes do not require that change.
Do not bundle Zig, SDKs or libraries into Plenty releases without deciding installation size, redistribution/license obligations, offline behavior and version pinning. First prove whether invoking an installed driver meets the native-port goal. User-facing cross-compilation can later reuse the boundaries without forcing a toolchain-bundling decision into this port.
Standard library scope
Status: Partly implemented. The standard library reference owns the shipped contracts: collections, text/data formats, files/processes, streams, networking, TLS, HTTP clients/servers and helpers, crypto, SQLite, logging, signals, and testing. The catalogue below is the original scope, not a list of modules still missing. Issue #49 retains the original catalogue and rationale; it is closed by decomposition, not implementation. Remaining delivery is recorded only in backlog B38–B70. Module names and unimplemented signatures remain design candidates. This page adds no language syntax or runtime API.
The goal is useful CLI/data tools and network services without third-party application packages. Python’s library breadth and Go’s combined HTTP client/server scope motivated issue #49; compatibility with their APIs is not a requirement. A production HTTP server remains an explicit bundled-library goal, with separate acceptance criteria.
Boundaries
Keep the prelude small and import larger facilities under std. Ship source
and approved native dependencies with the matching toolchain; ordinary builds
must import the library offline after installation or relocation. Importing an
unused module must not impose unrelated backend/linker requirements on a program.
Issue #45 continues to own package distribution, manifests and project commands.
Ordinary algorithms, containers, parsers, routing and policy belong in checked Plenty source. Standard imports pass the same ownership and worker-effect checks as user modules. Runtime boundaries cover allocation, handles/syscalls, clocks, entropy, sockets/polling, native synchronization and approved bindings. Reuse maintained TLS/crypto, SQLite and suitable compression/math primitives; record backend versions, licenses, supported targets and update responsibility. Plenty or native sets when a native library is chosen. Library purity must not weaken correctness or security.
Current support is x86_64 Linux GNU. Lossless Linux Path and list[u8] binary
transport have shipped; neither is a portable path abstraction or compact byte
view. Further target support requires explicit platform contracts and tests.
Plenty or native
Write a std module in Plenty unless a native library is clearly better: cryptography, a large established engine, or performance the compiler cannot reach. A Rust crate named in an issue is a reference for API and test vectors unless the issue makes that case. Issue #505 records this decision.
Plenty code costs less in an executable:
- Removal. Emission drops unreachable std functions one at a time before
native code is generated (issue #400),
and release builds substitute small leaf functions at the call site
(issue #363).
Native runtime code is removed only at link time, by
--gc-sections, and only when nothing kept still calls it. A central runtime dispatcher keeps every operation it can reach (issue #501). - Allocation failure. Plenty code reports it as
AllocErrorby construction. A wrapped crate must be checked, since Rust’s global allocator aborts on failure. rustls is the recorded exception (std.tlscosts); new wrappers get none.
A native wrapper must:
- report allocation failure as
AllocError, through a fallible interface or a caller-supplied allocator, aszlib-rsdoes forstd.compress. If the crate cannot, write the needed part in Plenty instead; - give each operation its own runtime entry symbol, as the
std.cryptodigests have, so the linker drops what a program does not call; - add no case to a shared runtime dispatcher;
- add a row to
NATIVE_MODULESintests/test_executable_size.rs, which checks that hello links none of the wrapper’s code and a program using it does (size budgets); - when large, ship as a separate engine needed only by programs that import
it, following the engine contract
that the
std.dataframeengine uses.
Module implementation choices
The policy applied to issues #461–#499. “Decide” means the issue settles the choice with a measurement or allocation check before implementation.
| Issue | Module | Choice | Reason |
|---|---|---|---|
| #461 | std.uuid | Plenty | Implemented: byte formatting over std.secrets and std.time |
| #462 | std.xml | Plenty | Implemented bounded UTF-8 event reader/writer; no DTDs or external entities. Pure Plenty keeps allocation failures explicit. |
| #463 | Time zones | Plenty | Package; packages carry only Plenty source |
| #464 | AEAD, signatures, key exchange | Native | Cryptography; ring is already linked for std.tls |
| #465 | Unicode segmentation | Plenty | Package; generated tables |
| #466, #467 | WebSocket | Plenty | Framing over std.http and std.tls |
| #468 | Server-Sent Events | Plenty | Text encoding over a response stream |
| #469 | std.diff | Plenty | Myers and patience are short algorithms over lists |
| #470 | std.markdown | Plenty | Parser and renderer; pulldown-cmark allocates infallibly |
| #472 | User directories | Plenty | XDG rules over std.env |
| #473 | SHA-1 | Native | Cryptography; RustCrypto sha1 allocates nothing, like sha2 |
| #474 | Bytes.find_bytes | Native | SIMD search the compiler cannot match; memchr allocates nothing |
| #475 | std.toml edits | Plenty | std.toml is Plenty; toml_edit allocates infallibly |
| #476 | std.image | Plenty | Implemented: PNG decoding and encoding over std.compress, crop, conversion and resize, with limits checked before allocating; image allocates infallibly. JPEG, GIF and WebP deferred |
| #477 | Progress bar | Plenty | Terminal drawing over std.io |
| #478 | std.shlex | Plenty | Small tokenizer |
| #480 | std.memmap | Plenty | A few system calls behind small runtime entries |
| #481 | std.bigint | Decide | Measure Plenty limb arithmetic against native call cost; num-bigint allocates infallibly |
| #482 | LRU cache | Plenty | dict and std.deque |
| #483 | BLAKE3 | Native | Cryptography with SIMD; confirm hashing allocates nothing |
| #484 | inotify | Plenty | A few system calls behind small runtime entries |
| #485 | std.mime | Plenty | Parsing and a lookup table |
| #486 | std.graph | Plenty | User data on nodes and edges; algorithms over lists |
| #487 | System information | Plenty | Text files under /proc and /sys |
| #488 | std.jwt | Plenty | Composes std.json, std.encoding and std.crypto |
| #489 | base32 | Plenty | Same as the existing codecs |
| #490 | Linear algebra | Native | Large engine with SIMD kernels; separate engine; check faer allocation first |
| #491 | Brotli | Native | Large codec; the brotli crate takes an allocator, so failure is reported |
| #492 | Optimization | Plenty | Closed; replaced by the numerics core proposal: solvers in Plenty over std.linalg |
| #493 | numeris evaluation | Plenty | Closed; not adopted; std.linalg keeps faer |
| #494 | Statistics | Plenty | Formulas over lists; sampling keeps std.random’s stable sequence |
| #495 | Key-value store | Decide | Large engine with its own file format; std.sqlite already covers transactions and ordered keys |
| #496 | Sketches | Plenty | Small algorithms over arrays and hashes |
| #497 | Xor filters | Plenty | Short construction over a fixed key set |
| #498 | Cuckoo filters | Plenty | A small hash table |
| #499 | std.template | Plenty | Already chosen; tera renders from serde values |
Issue #471 designs derived serialization and makes no implementation choice. Issue #479 is closed.
Shared contracts
Establish binary buffers, partial-progress I/O, contextual errors and monotonic deadlines before consumers invent incompatible alternatives. Preserve typed Result errors, explicit fallible allocation and existing cooperative cancellation. Specify EOF, short progress, interruption, close/flush errors and cleanup after partial I/O, cancellation or allocation failure.
Structural protocols are statically dispatched. A proposed API must not assume associated types, existential storage, reflection or automatic codec derivation. Use implementable generics, explicit error/value enums and typed codecs. Consuming/borrowing, copied/streamed and in-place/allocating operations need distinct documented contracts. Binary data is not UTF-8 text; filesystem paths are not URLs; simulation randomness is not cryptographic entropy.
Every adopted module publishes its scope, errors, allocation/copy costs,
ownership/thread safety, supported platforms and complexity where relevant.
Add small reference pages, runnable tutorial examples and SUMMARY entries with
the implementation. Use local fixtures, fixed seeds, fake streams/clocks,
failure injection, external format fixtures and retained parser fuzz corpora.
Test Plenty libraries through plenty test when issue #45 supplies its runner,
alongside native primitive/lifecycle tests.
Capability checks and further breadth
The initial CLI/data check reads arguments/environment, loads TOML/JSON, walks files, processes CSV and reports contextual errors/structured logs. Existing math, text primitives, lossless paths, walking, sorting and deque support count toward it. It does not wait for a network execution model. The service check fetches verified HTTPS with cancellation/bounded streaming, persists records in SQLite and serves a routed JSON API meeting the HTTP acceptance gates.
Accurate summation/norms, float decomposition/adjacent values and special functions need numerical contracts; basic math has already shipped. Optional scope still includes timezone/calendar data, statistics, decimal/big-integer/ complex arithmetic, MIME types, HTML escaping/context-aware templates, WebSockets, diff tools, terminals and signals. Evaluate each against a concrete use case before adding an implementation issue/backlog entry. HTTP/2 is a follow-on protocol candidate; HTTP/3 needs a separate decision.
GUI toolkits, mail/FTP stacks, interpreter/debugger internals, arbitrary runtime loading and pickle-style serialization are outside the initial catalogue. YAML, an ORM and a complete web framework also require their own case. Closing the umbrella does not broaden these commitments or claim delivery.
Arbitrary-precision integers (std.bigint)
Status: Not implemented. This page compares ways to provide issue #481’s
std.bigint and recommends one. It ships no module, compiler change or API.
Scheduling belongs to B38. The current integer contract is
numeric types; library operators are
user-defined arithmetic operators.
Recommendation
Write std.bigint in Plenty, in three steps:
- Plenty module. A
BigIntwith an inlinei64for small values andu64limbs for large ones. Schoolbook and Karatsuba multiplication, Knuth division and divide-and-conquer radix conversion, ported fromnum-bigint(MIT OR Apache-2.0) with credit. Every allocation is an ordinary fallible Plenty allocation. - Compiler builtins for limb arithmetic.
u64.widening_mul(Craneliftumulhiplusimul) and an add-with-carry form. The measured limb loop is 4.6× slower than Rust without them. - Optional native kernel, only if measured to pay. A runtime entry for large multiplication (Toom-3 or an FFT) that writes into an output and scratch slice the Plenty caller has already allocated. It allocates nothing itself, so it needs no allocation-failure handling. Write it in permissively licensed Rust; do not use malachite.
Malachite does not fit. On allocation failure it aborts the process. It is LGPL-3.0-only, and the runtime archive is statically linked into user executables. It adds 1.2 MB to every executable that uses it and 35 s to a clean build. Its multiplication beats the alternatives only above about 10⁴ bits, and only through its AVX2 FFT on x86-64; the portable path was slower than every other crate, and ARM64 was not measured. It also parses decimal text 2–6× faster.
Requirements
- Fallible allocation. Every allocation in
stdreportsAllocError; a Rust allocation failure in the runtime aborts the process (Rust panics). - License. Plenty is MIT OR Apache-2.0. No current dependency is used under a copyleft license. The only license expressions that mention LGPL also offer MIT OR Apache-2.0.
- Size and build cost. Code in the runtime archive costs nothing to programs that do not reach it, but grows the compiler and its build. The engine criteria move a module out of the archive above 2 MB per program or one minute of build time.
- Platforms. Linux x86-64 and ARM64, macOS ARM64, Windows x86-64 MSVC (cross-platform assessment).
- Project preference. Implement in Plenty unless native complexity or
security justifies a wrapper (B38,
AGENTS.md).
Allocation failure in candidate crates
Each crate’s source was read and run with a global allocator that refuses allocations over 4 KiB, then asked to square a 65,536-bit number.
| Crate | License | How it allocates | On failure |
|---|---|---|---|
malachite-nz 0.14.0 | LGPL-3.0-only | Vec throughout (about 1,500 vec!/push/with_capacity sites, no try_reserve); the FFT allocates internally | handle_alloc_error: abort |
num-bigint 0.5.1 | MIT OR Apache-2.0 | Vec<u64>, no try_reserve | abort |
ibig 0.3.6 | MIT OR Apache-2.0 | alloc::alloc in memory.rs/buffer.rs, null handled by handle_alloc_error | abort |
dashu-int 0.6.2 | MIT OR Apache-2.0 | alloc::alloc in memory.rs/buffer.rs; null calls panic_out_of_memory() | panic, which is an abort in the panic=abort runtime |
rug 1.30 / GMP 6.3 | LGPL-3.0+ / LGPL-3.0 or GPL-2.0 | mp_set_memory_functions | GMP’s manual: allocation functions “must terminate program execution. A longjmp or throwing a C++ exception will have undefined results.” |
No crate exposes an allocator parameter or a fallible API. Malachite’s
limb-level functions with caller-supplied scratch, such as
limbs_mul_greater_to_out, are crate-private unless the test_build feature
is enabled, and its FFT path allocates regardless.
Can the runtime’s arena catch the abort?
The runtime can already contain one infallible Rust library: ring’s RSA
operations run inside actor_heap::within with a pre-reserved arena. An
allocation failure there jumps back and reports AllocError
(RSA allocation).
Applying this to malachite would fail for these reasons:
- Platforms. The arena router exists only on Linux x86-64 and ARM64. On
other targets
withinruns the job directly and a failure aborts. - Value-dependent memory.
ringuses a fixed amount of memory per operation, so its arena has a fixed, tested size. A bignum operation’s memory grows with its operands (the FFT path uses O(n log n)), so each call must estimate a bound and reserve it first. - Results cannot leave the arena.
withinforbids handing back allocations. Every resultNaturalwould have to be copied into Plenty storage before the arena is freed: an extra O(n) copy and twice the peak memory. - dashu panics instead of calling the allocation-error handler, so the arena cannot catch its failures at all.
An engine (shared library) does not help either: Rust’s Vec aborts there
too, and the linalg engine
reports allocation failure only because it uses fallible faer APIs and a fork.
Forking malachite (about 650,000 lines with malachite-base) to make it
fallible is not maintainable.
License: LGPL-3.0 in a statically linked runtime
The runtime archive, including any Rust crate it wraps, is statically linked into each executable that reaches the module. Under LGPL-3.0 §4, everyone who distributes such an executable conveys a “Combined Work” and must:
- give prominent notice and ship the GPL-3.0 and LGPL-3.0 texts;
- use terms that do not restrict modification of the library or reverse engineering to debug such modifications;
- either supply the application in a form the recipient can relink with a modified malachite (object files or source), or load malachite through a shared-library mechanism;
- for “User Products”, supply installation information.
These obligations fall on every Plenty user who distributes a binary that uses
std.bigint, not only on the Plenty project. Relinking is also hard in
practice: the Rust code is monomorphized and inlined into the runtime archive,
so the user would need the whole runtime archive rebuilt against the modified
malachite. A shared engine (libplenty_bigint_v1.so) would meet the
relinking condition with §4(d)(1), but it still leaves the allocation-failure
problem, costs a dynamic call per operation and must be shipped with the
application. GMP/rug raises the same issues. GMP also needs m4 and make
to build and does not support Windows MSVC (MSYS2/MinGW only).
num-bigint, ibig and dashu are MIT OR Apache-2.0, so their algorithms can
be ported into Plenty with credit in the module header and its
## Acknowledgements section.
Measurements
Measured 2026-10-31 on an AMD Ryzen 7 7840HS, rustc 1.99.1, release build with
panic=abort and codegen-units=1, pinned to one core. Inputs are
pseudo-random. The scratch crate lives under the worktree’s target/ directory
and is not committed.
Throughput
Nanoseconds per operation. “Small” is 10 million &a + &b or &a * &b on
values that fit in a word, producing a new value each time.
| Operation | num-bigint | malachite | dashu | ibig |
|---|---|---|---|---|
| small add | 8.2 | 7.3 | 7.2 | 1.1 |
| small multiply | 7.6 | 5.8 | 6.0 | 5.1 |
| multiply 256 bits | 30 | 33 | 48 | 29 |
| multiply 4,096 bits | 2,296 | 1,947 | 1,730 | 2,129 |
| multiply 65,536 bits | 175,398 | 59,958 | 145,030 | 154,566 |
| multiply 1,048,576 bits | 11,081,585 | 1,891,493 | 7,316,193 | 9,840,681 |
| 20000! by repeated multiply | 21.8 ms | 26.0 ms | 24.2 ms | 19.8 ms |
| format 20000! (77,338 digits) | 3.7 ms | 2.4 ms | 3.2 ms | 3.5 ms |
| parse 20000! | 4.2 ms | 0.70 ms | 1.5 ms | 1.7 ms |
Malachite is 2.4–6× faster at 65,536 bits and above, and much faster at
parsing. Up to 4,096 bits and on small values it is no faster than the others.
Its large-size speed comes from an f64 FFT that uses AVX2 and FMA only when
the std feature enables runtime CPU detection. With default-features = false (the portable path), 65,536-bit multiplication took 837,473 ns, slower
than every other crate. The ARM64 path uses the wide crate and was not
measured.
Plenty limb arithmetic
The same schoolbook multiplication was written in Plenty, compiled with
--release. Plenty has no u128 or high-half multiply, so each 64×64-bit
product uses four 32-bit multiplies. The Rust baseline uses u128.
| Size | Rust u128 schoolbook | Plenty, product in a helper function | Plenty, product inlined by hand |
|---|---|---|---|
| 256 bits | 23 ns | 403 ns | 357 ns |
| 4,096 bits | 4,879 ns | 33,705 ns | 22,389 ns |
| 65,536 bits | 929,096 ns | 7,814,643 ns | 4,878,989 ns |
The inlined inner loop costs 5.5 ns per limb product against Rust’s 1.2 ns. The
small-size cost is mostly fixed overhead: allocating and filling the result
list and the call. Holding 32-bit limbs in u64 slots, so that each product
needs one multiply, was slower (43,463 ns at 4,096 bits). The remaining cost is
in the checked adds and the indexing, not the multiply. A widening-multiply
builtin and a carry form would remove the emulation and most of the overflow
checks.
A Plenty small-value fast path is as fast as the crates’ small paths:
enum Big: Small(i64) | Large(list[u64]) with an __add__ hook that tries a
checked: add takes 8.7 ns per (acc + one)?. Calling a native crate would
add a call into the runtime and an owned native handle per result, on top of
the crate’s own 6–8 ns.
Executable size and build time
The program parses one number, squares it and prints it. Sizes are of a stripped Rust executable; the baseline prints its argument.
| Crate | Executable | Added | Clean build, 8 jobs |
|---|---|---|---|
| none | 339,416 B | — | — |
| num-bigint | 411,912 B | +72 KB | 2.4 s |
| ibig | 408,856 B | +69 KB | 2.5 s |
| dashu-int | 472,400 B | +133 KB | 4.5 s |
malachite-nz (std) | 1,543,816 B | +1.20 MB | 35 s |
Malachite would remain under the 2 MB engine threshold. Its build time would use more than half of the one-minute budget, and every compiler build would pay it. A Plenty module adds only the functions a program reaches; it needs no Rust build.
Options compared
| Plenty module (+ builtins, + optional kernel) | Wrap malachite | Wrap num-bigint / ibig / dashu | GMP via rug | |
|---|---|---|---|---|
AllocError on failure | Yes, on every target | No; arena on Linux only, with copy-out and per-call bounds | No; same arena limits; dashu cannot be contained | No (undefined if unwound) |
| License | MIT OR Apache-2.0 (ported, credited) | LGPL-3.0-only obligations on every distributor | Permissive | LGPL/GPL, same obligations |
| Size per program | Only reached functions | +1.2 MB | +70–130 KB | GMP static library |
| Compiler build | None | +35 s | +2–5 s | C build with m4 |
| Platforms | All targets | All, AVX2 FFT on x86-64 only | All | No Windows MSVC |
| Small values | 8.7 ns measured, no allocation | Native call + handle + 6–8 ns | Same | Same |
| Large values | Slower until builtins/kernel; 4.6× behind Rust schoolbook today | Fastest above 10⁴ bits | Comparable to each other | Fastest |
API direction
These signatures are a sketch for the implementation PR to settle.
from std.bigint import BigInt
class BigInt # not a copy class; copy(x)? duplicates the limbs
def __add__(self, other: &BigInt) -> Result[BigInt, AllocError]
def __radd__(self, other: i64) -> Result[BigInt, AllocError] # i64 + BigInt
def __sub__, __mul__, __floordiv__, __mod__, __neg__, __lshift__, __rshift__,
__and__, __or__, __xor__ # same shape
def add_assign(self: &mut BigInt, other: &BigInt) -> Result[(), AllocError]
def cmp(self, other: &BigInt) -> i64 # until __cmp__ (B37)
def to_i64(self) -> Result[i64, RangeError]
def to_f64(self) -> f64
def display[W](self, out: &mut Formatter[W]) -> Result[(), FormatError]
def to_str(self, radix: i64) -> Result[str, AllocError]
def from_i64(value: i64) -> BigInt # no allocation
def parse(text: &str, radix: i64) -> Result[BigInt, ParseError | AllocError]
- Ownership. Operator hooks take a shared
selfand a shared or copyable operand, and return an owned result.a + bborrows both operands and never consumes them. A result that fits ini64is stored inline and allocates nothing, but the hook still returnsResult, so callers write(a + b)?. - In-place operators.
__iadd__must return(), so it cannot reportAllocError. Provide named fallible methods (add_assign) instead, which is the reference’s own advice (“use explicit assignment or a named fallible method”). - Division. Use floor semantics, as
//and%have oni64. A zero divisor traps, as it does fori64. A checked variant can returnZeroDivisionError. - Equality and ordering.
==is structural, so it is correct only if the representation is canonical. Every value that fits ini64isSmall, and limbs carry no high zeros. Classes have no ordering operators until the user-defined ordering proposal’s__cmp__lands, socmpis a method for now.BigIntcannot be a dictionary key until user-type hashing lands. - Limits. Text conversion of attacker-controlled input should either use
subquadratic algorithms or take a digit limit, like other parsers’
Limits.
Open questions
BigInt + i64. Each operator has one left hook, so__add__cannot accept both&BigIntandi64.i64 + BigIntworks through__radd__. For the other order, callers writea + from_i64(n), which allocates nothing, unless hooks gain overloading or a mixed-operand sum type.- Names and signatures of the limb builtins (
widening_mul -> (u64, u64),carrying_add(other, carry: bool) -> (u64, bool)), and whether Cranelift lowers them on ARM64 as cheaply as on x86-64. - The size threshold above which a native multiplication kernel is worth its code, measured after the builtins land.
- Whether small helper functions in the limb loop need inlining. The measured
call to
mul_widecost 1.5× in the 4,096-bit loop.
Acknowledgements
num-bigint(MIT OR Apache-2.0): the reference implementation suggested by issue #481; the recommended source for ported algorithms.malachite(LGPL-3.0-only),ibiganddashu(MIT OR Apache-2.0), and GMP (LGPL-3.0 or GPL-2.0): evaluated and measured here; no code is used.
Checked arithmetic by default
Status: Core and repository adoption implemented; bounded validation complete. Function-level arithmetic boundaries, lexical
unchecked:, independent closure contracts, std/callback APIs, fixtures, shipped examples, compiler benchmark templates and teaching/reference changes are implemented on this branch. Validation evidence records the isolated post-repair host gate, controlled timings/profiles, representative optimized code and native/export audit, with explicit source and observation limits. Neither local PR #886 nor its proxy-fixture dependency #918 is merged. This is not whole-proposal performance or merge acceptance. Remaining B14 conveniences stay in the backlog.
Recommendation
Make every ordinary function body a boundary for recoverable integer arithmetic
failures. Operators still produce ordinary numbers on success; their failure
returns Err from the function, whose declared Result error type must include
it. Callers handle that ordinary Result with ?, match, or another existing
operation. There is no ? after each arithmetic operator.
Keep checked: as an explicit, value-producing local recovery boundary, both
in ordinary code and inside unchecked:. Give unchecked: the value of its
suite, without adding a Result wrapper. Inside it, the programmer guarantees
that affected arithmetic does not fail. This proposal validates that
precondition with today’s arithmetic traps.
This improves recoverability by default and makes deliberate arithmetic traps searchable. It does not remove machine checks yet: terminating on overflow requires detecting it. “Unchecked” means unchecked by the typed arithmetic error mechanism, never silent wrapping.
Plenty has no external users or dependent systems to preserve. Adopt this as a coordinated breaking change across the compiler, standard library, tests and documentation. Existing syntax and APIs are inputs to the design, not compatibility constraints; no transition mode or edition mechanism is required.
Function contracts
def scale(value: i64, factor: i64) -> Result[i64, OverflowError[i64]]:
product = value * factor
Ok(product)
def invoice(quantity: i64, price: i64) -> Result[i64, OverflowError[i64]]:
Ok(scale(quantity, price)?)
The multiplication has type i64; failure returns
Err(OverflowError[i64](wrapped=...)) from scale. A signature of -> i64
is rejected for this body. The compiler reports the failing operation and the
missing error type, with suggestions to declare a Result, handle a local
checked: result, or deliberately opt out.
The implicit function boundary controls failure routing, not success-value
conversion. Keep ordinary explicit Ok(...), Err(...), tail expressions,
and return rules. Do not wrap an existing Result into another Result,
auto-unwrap calls, infer public return types, or accept a bare i64 as the
success return of Result[i64, E]. This is deliberately more precise than
textually wrapping an entire function in today’s checked: expression.
Only failures that can escape this function boundary must appear in its error
type. A locally handled checked: error need not appear; an error propagated
with ? does. Wider declared error unions remain permitted. Functions with no
escaping arithmetic failures can still return a plain type, including ().
An Option-returning function cannot silently turn overflow into Nothing:
handle the arithmetic locally, change the return contract, or opt out.
The initial scope is exactly today’s checked integer operation set:
| Operation | Error contributed |
|---|---|
Integer +, -, *, unary -, pow | OverflowError[T] |
Signed // with a plain or NonZero[T] divisor | OverflowError[T] |
// and % with a plain integer divisor | ZeroDivisionError |
Unsigned division has no overflow case. NonZero[T] removes the zero error;
Positive[T] also removes signed division overflow. Compound assignment and
forwarded integer @newtype operations follow the same rules and field error
types as today. Type checking follows operators and operand types, not general
range analysis: even n // 4 keeps today’s conservative error set. Optimizers
may remove proven-unnecessary branches without changing that contract.
Float arithmetic, float div_checked, shifts, casts, negative integer pow
exponents, indexing, allocation, I/O, and explicit unwrap() retain their
existing contracts. Module constants still reject overflow at compile time.
This is a proposal for recoverable integer arithmetic by default, not a promise
that all runtime failures become recoverable.
The opt-out block
def scale_in_bounds(value: i64, factor: i64) -> i64:
unchecked:
value * factor
On success this returns an i64. As implemented here, overflow terminates the
process in debug and optimized builds alike. It cannot reach an outer checked
boundary as an Err; there is no unwinding or rollback.
Termination is the first implementation of the precondition, not its permanent
meaning. A dependent follow-up will add a compiler mode that assumes the
precondition and omits the checks, so a violation becomes undefined behavior,
not wrapping. Trapping stays the default, and a separate unchecked trap:
form will always check and trap. Code that uses traps to reject invalid input
must move to that form or validate explicitly before the mode can apply.
unchecked is contextual, like checked: only unchecked: followed by a
newline opens a suite; an existing typed binding unchecked: i64 = 0 remains
valid. Support value positions already available to checked:: a binding,
assignment, return value, function tail, or match-arm tail. Also support a
standalone suite for mutations and control flow. Locals remain block scoped.
A value-producing suite yields its final expression’s type T, not
Result[T, E]. If that expression already has type Result, it stays a
Result and must be handled normally. A statement suite follows ordinary
statement/discard rules; it cannot silently discard an owned value or a
Result. An empty suite with pass, or an all-exiting suite, is allowed in
statement position. No reference-valued block extension is proposed initially.
?, return, break, and continue retain their ordinary function/loop
targets. In particular, unchecked: does not catch or trap a callee’s Err,
disable allocation handling, or make an unchecked call variant:
def caller(value: i64) -> Result[i64, OverflowError[i64]]:
answer = unchecked:
scale(value, 2)?
Ok(answer)
scale still returns its declared error and ? still propagates it. To get
trapping arithmetic in a helper, the helper’s implementation must explicitly
contain the opt-out. Inlining cannot change that behavior.
Keeping checked:
Retain its Result[T, E] expression semantics for arithmetic in the suite.
Functions and closures defined there have their own declared error boundaries,
as described in functions and callbacks.
The block remains useful for recovering or mapping errors before they reach
the function signature, even when the default is checked. Within unchecked:,
it also restores recoverability for a smaller region:
def mixed(a: i64, b: i64) -> Result[i64, OverflowError[i64]]:
result = unchecked:
offset = a + 1
sensitive = checked:
offset * b
sensitive
result
The addition traps on overflow; multiplication produces the local Err, which
is returned normally. The outer unchecked suite passes through that Result
without trapping or wrapping it. A match on sensitive could instead recover
locally and let mixed return a plain i64.
| Lexical context of an integer operation | Failure destination |
|---|---|
| Function body, without a nearer explicit mode | Function’s declared Result |
Explicit checked: | That block’s Result |
Explicit unchecked: | Precondition violation; currently process termination |
checked: inside unchecked: | Inner block’s Result |
unchecked: inside checked: | Precondition violation; currently process termination |
Nested unchecked: is redundant but legal. Leaving any suite restores its
outer mode. A failure in an explicit checked block releases that block’s locals;
an implicit function failure releases function-owned values as ? does.
Earlier mutations are retained, and a failing compound assignment performs no
write. Neither boundary is transactional.
Keep today’s explicit checked: restrictions initially: unused results,
reference results, blocks with no failing arithmetic, and blocks whose every
path exits are rejected. Do not apply the last two restrictions to the implicit
function boundary: ordinary infallible and all-returning functions remain valid.
Auditability and cost
Search for unchecked: to locate explicit arithmetic opt-outs, then review why
each region cannot fail and whether its extent is minimal.
Use a source-aware search to distinguish syntax from comments and text. Prefer
comments explaining bounds or measured hot paths over generic “faster” claims.
Real-world confidence in bounds is useful evidence, but today’s check traps if
that confidence proves wrong, and an assumed-valid build would not.
This search is not a complete trap audit: indexing, unwrap(), negative powers,
foreign code and existing runtime primitives remain separate trap sources. Std
implementations must expose deliberate arithmetic opt-outs in source or documented
native contracts; hiding unchanged trapping reductions behind safe-looking calls
would weaken the proposal’s benefit.
Default code can reuse today’s straight-line arithmetic plus cold error exits.
It need not materialize a Result for each operation. unchecked: can avoid
error payload construction, recoverable cleanup paths and some return ABI costs;
it cannot promise check-free arithmetic or a speedup. Retaining checked: does
not itself imply runtime mode switches: the mode is lexical.
See implementation and validation for callback boundaries, repository-wide changes and the evidence needed before adoption.
Checked defaults: implementation and validation
Companion to checked arithmetic by default. Repository adoption and bounded validation complete. Collected results and limits cover the isolated host gate, all controlled timing/profile jobs, representative optimized code, native/export audit and compiler benchmark-template adoption. This is not universal performance/ABI or merge acceptance. Remaining B14 conveniences stay in the backlog.
Current implementation
Function bodies collect arithmetic failures against their declared public
Result type, without changing successful return rules. unchecked: selects
trapping operators and leaves suite values unwrapped. Nested functions no
longer inherit a private callback effect. Generator arithmetic receives a
targeted diagnostic; ordinary native tail returns remain supported.
tests/test_checked_defaults.rs exercises the new contracts, all eight
integer widths, nesting, mutable and stored callbacks, worker error transport,
fallible key sorting, cleanup, skipped compound writes, generators and tail
recursion. The private callback error channel and its lifetime restrictions
are removed; its tests are replaced by public-contract coverage. The retained
local arithmetic suite passes.
The initial library migration covers std.math, std.bytes, std.time,
std.encoding, std.io, std.io.typed, std.fmt, std.fs, std.env,
std.secrets, core std.http, std.net, std.ready, std.text, std.random,
std.test, std.url, std.process, std.http.decoder, std.http.typed
and std.http.client, plus std.json, std.unicode, std.complex, std.stat,
std.linalg, std.simd, std.log, std.crypto, std.tls, std.uuid,
std.jwt, std.http.server, std.toml, std.regex, std.cli, CLI progress,
std.dataframe, std.plot, HTTP body/cookie/proxy helpers, std.xml,
std.csv, std.tempfile, std.system, std.fs.watch, SQLite/KV wrappers,
std.compress, and actor core/TCP/TLS/HTTP-server adapters.
Integer math magnitude/reductions return Result[T, OverflowError[T]];
separate *_float helpers retain IEEE float values. Serial pipeline sum()
returns a public result for integer items. Sorting accepts Result[K, E]
key callbacks and returns Result[(), E]. Bounds-proven byte/time operations
use small documented opt-outs; fallible size and time calculations retain
their existing error contracts. Encoding expansion sizes are checked before
allocation, stream position/total limits return their declared errors, and
filesystem walk reservation no longer counts pending frames twice (issue #702).
Pure capacity helpers are tested at representation limits without allocating
huge buffers. The HTTP decimal accumulator subtracts ASCII bias before checked
multiply/add (issue #703), accepting the complete i64 range while mapping larger
lengths to protocol errors. FakeStream.fail_after returns a public
Result[(), OverflowError[i64]]; failures preserve its pending schedule.
Counter exhaustion in the stream protocol maps to OutOfRange before mutation.
URL expansion and process extents check representation limits before allocating
or consuming another chunk. Process argument encoding reserves additional bytes
without counting held bytes twice (issue #704). Incremental decoder counters are
bounded by its per-feed, line, head and body limits. Typed chunk writers
preflight the complete wire extent before delivery, preserving progress and
transport causes. Client callbacks retain their own contracts; deadline
saturation and bounded pool/retry/body arithmetic are explicit. No blanket unchecked:
conversion was applied.
JSON retains its input/output budgets and maps one-based position exhaustion
to JsonFault.InputTooLarge before advancing the failing cursor. Partial output
compaction preserves delivered progress without allocating. Unicode normalization
uses explicit scalar-ceiling, successful-append and positive-index bounds. The
JSON/Unicode component passed 35 isolated-target runtime-check tests, including
debug/optimized representation boundaries and the Unicode normalization corpus,
and received an independent source review. A separately reviewed TOML/regex
checkpoint preserves parser faults, edit atomicity, regex work/memory limits
and replacement validation order. Public mutable regex matches expose length
subtraction as Result[i64, OverflowError[i64]]; unavailable TOML coordinates
report InputTooLarge at the valid offset with zero line/column, following the
user’s explicit choices. Its 40 isolated-target tests pass, with six added
boundary tests in both optimization modes. A separately reviewed XML/CSV
checkpoint preserves parser faults and stream ownership. CSV position exhaustion
returns the existing stream OutOfRange error; preflight keeps offsets exact,
record MAX valid and explicit flushing of already-finished bytes available.
The reviewed numerical checkpoint preserves complex powers at i64.MIN, matrix
shape/index invariants and statistical capacity causes. Ordinary integer SIMD
arithmetic returns lane-typed overflow results; explicit wrapping and IEEE float
APIs remain unchanged. The SIMD tutorial and generated declarations match.
Observation-only logger, server-report and actor statistics saturate at their
existing maximum, following the user’s explicit policy; semantic progress stays
exact and fallible. Server output and unread extents preflight before delivery
or buffer growth. TLS/crypto/UUID/JWT retain their existing public errors with
source-visible buffer, DER, timestamp and time/leeway bounds.
Actor slots and timer/request records retire rather than wrap exhausted identity generations. Mailbox counters rebase all linked target-owned exit fences while preserving message-prefix ordering. An affine arithmetic error reaches its parent with both actor heaps reclaimed. Complete actor TLS, crypto, JWT, HTTPS and server/client tests pass after dependency integration. Combined targeted validation has 733 passes, two failures in unmigrated HTTP body/cookie, and one pre-existing ignored test. The two native-engine linalg bodies were still unrun at that checkpoint; the follow-up native-engine results are recorded below. At that checkpoint the registry still reported 29 failing module analyses; broad tutorial validation remains incomplete. These results are not all-features or performance acceptance.
A later reviewed checkpoint adds bounded CLI parsing/progress, checked dataframe extents, plot arithmetic and body/cookie byte budgets. Actor HTTP keeps exact response accounting and equivalent bounded lingering-threshold state; complete TCP/TLS/control/error tests pass. Initial optional-engine self-skips were followed by actual local-engine runs: all 17 dataframe and nine linalg tests pass, including relocation and allocation cleanup after fixture-only arithmetic fixes. This does not replace the native arithmetic audit. Reviewed proxy/helper fixtures restore descriptor, cancellation, RFC and bulk-scan checks; actor HTTP interop retains real curl and 600-mutation corpus behavior. Temporary/system/watch/SQLite wrappers and compression preserve existing errors and resource/progress contracts. Compression’s live gzip/Brotli interop runs; SQLite’s optional vector extension and Docker cases remain unrun. At that checkpoint the registry had 13 failing module analyses. The final reviewed std checkpoints now cover archive, diff/image, HTTP files/SSE/WebSocket, MIME/template/markdown and sketches. The registry/index passes both tests with no diagnosed modules; public mutable diff lengths return overflow Results. Focused HTTP files tests now execute all 11 filesystem/conditional/stream cases, not merely extracted size helpers. Oversized metadata returns InvalidContentLength before validators, following the user’s explicit policy (issue #706). Image and Markdown preserve format/domain errors, checked extents, exact parser and output progress, and cleanup. Their 23 and 10 tests pass respectively; representation-limit probes supplement, not replace, actual codec/parser tests.
The reviewed parallel integer sum now returns an ordinary overflow/pool Result, retaining serial-prefix semantics, the precise first wrapped payload and one callback invocation per item. Mapping drains before ordered addition and source cleanup; O(n) mapped storage and a serial combination pass are deliberate, unbenchmarked costs. Floats retain chunk grouping. Enumerate index overflow keeps its explicitly documented trapping contract. All 30 pipeline tests and both owned tutorial examples pass after integration.
Reviewed tutorial, reference and skill adoption uses independent public callback Results, explicit joins and precise propagation. Full executable documentation improves from 3 passing / 17 failing tests to 19 passing / 1 failing after the reviewed generic deque/heap/graph/LRU checkpoint (49 targeted passes). Its std public declarations, allocation causes and ownership contracts stay unchanged. Reviewed fallible fixed-tree and pipeline reducers now accept ordinary Result callbacks, including reconstructed pair items with exactly-once affine cleanup. Their public error unions retain pool failures and callback errors; deterministic tree-level/pair selection and chunk-fold policies remain distinct from integer sum’s serial prefixes. All 20 tutorial/skill tests, 24 parallel tests and 37 parallel-pipeline tests pass after integration. Registry analysis does not instantiate all generic bodies; executable coverage was needed in addition to its two passing tests. Reviewed native export/FFI fixtures restore 23 tests without widening C scalar/fieldless-error signatures or weakening native oracles.
Adoption remains incomplete. Repository fixtures/examples, full-suite/native
arithmetic audits and performance measurements remain.
A complete earlier all-features baseline compiled, including experimental emitter
and profiling features, but reported 4,465 passes, 1,109 failures and eight ignores.
The passes include 27 confirmed optional self-skips. Most failures reject old
fixture arithmetic contracts; some hit the generic std bodies adapted since that
snapshot. A newer complete gate at ff52bd7e compiled successfully and reported
4,813 passes, 813 failures and eight ignores, again including 27 optional
self-skips. It has no newly failing outer name relative to that baseline and no
remaining std-source diagnostic. Subsequently reviewed borrowing, tail/stack,
control/trap and flow/shared batches pass all 568 parent-run tests, including
258 identical failed names from that newer snapshot. This is scoped evidence,
not a measured current-head whole-suite result or acceptance. Public Result
adoption also exposed the pre-existing contextual numeric-literal defect in issue
#710. A reviewed non-evaluating success-payload hint fixes it without changing the
original unsuffixed-call-result oracle. A rebuilt main baseline rejects that
source; native normal/release regressions now pass, and all 109 callable fixtures
and ten literal-form tests pass after parent integration. The user approved deterministic fallible-tree
errors by level then left-to-right pair, accepting level-by-level scheduling while
infallible reducers retain their optimization. Ordinary Result[(), E] initializers now produce exactly Result[Class, E], without
adding AllocError unless declared. Reviewed generic-output deferral preserves
whole-result parameters, aliases, defaults and constructor values. All 114
constructor/checked-default tests pass after integration, including partial-field,
affine-error, default-failure and native inline/noninline cleanup. Issue #709’s
partial-self capture guard is integrated; captures are checked when created,
without crediting deferred closure-body writes. Infallible destructor/context-exit
protocols and existing C-export error-layout restrictions remain unchanged.
The tuple-adapter review correction is integrated with owned/enumerated/mapped
pair, cardinality, managed-error and allocation-failure regressions. Earlier
scalar-only passes did not establish that case; the correction received separate
review before integration. Further reviewed concurrency and collection batches
pass 183 and 428 parent-run cases while retaining actual worker, guard-page,
allocation and ownership oracles. The user explicitly approved updating one
unoptimized range-generator oracle to require the exact bounded helper and
caller-provenance calls. It retains counted-source/no-iterator-allocation checks
and native output, not a zero-call or performance-equivalence claim.
The historical gate at af8aa15d had 5,375 reported passes, 259 failures and
eight ignores, including 27 optional self-skips. After the reviewed final fixture
and example adoption, the complete host gate at 2ef36dce passes: 5,650 passed,
zero failed and eight existing ignores. Every one of those 259 failed names now
passes; sixteen additional outer tests pass separately. All 27 known optional
self-skips execute their bodies, including the selected dataframe/linalg engines,
tutorial engine jobs and ten Docker cases. The real libFuzzer and Neovim bodies
also execute. Two explicitly selected ignored trials pass afterward and are not
folded into the whole-gate totals. External engines are compatible existing
artifacts, not newly source-built engines; transient host-tool conditionals are
not exhaustively traced. This is host test acceptance, not performance acceptance.
Reviewed shipped-example and companion adoption subsequently passes all 20
structure tests and all three exhaustive-example tests, with the original stress,
allocation and ownership oracles intact. An explicitly run sqlite-vec trial passes
its real transactional example using deterministic test inference; this does not
qualify a real embedding model. A local release-skill assembly passes 19 sync tests
and checks 152 Markdown pages/837 local targets without missing links; no remote
publication or agent trial is implied. The bounded native/generated-adapter
audit’s inline sort-error ownership hazard (issue #711) is repaired with a
caller-owned typed result slot. Relocation precedes comparator return, without
heap allocation or restricting the public error type. Independent review and
21 parent-run checked-default tests cover range/nested managed errors, first and
second-key failure, skipped callbacks, exact payload/drop counts and normal/release
storage-layout checks. The pre-repair native probes happened to print correctly;
the discriminating failure was structural, so no reproduced crash is claimed.
The integrated whole gate is green at that immutable snapshot. Measurements
remain pending. The independently reviewed #712 repair is integrated. The fresh
post-repair gate was safety-aborted by an external Docker evidence-wrapper regex
error before any container launch: 4,089 passed, two wrapper-caused failures and
one ignore were observed; 1,575 registered tests were unobserved. All 25
checked-default tests and the five new verifier tests passed, but this partial
run is not a complete gate. A second independently reviewed external harness
attempt also stopped before container launch because the installed Docker emits
lowercase absence diagnostics: 3,691 passes, no completed failure rows and one
ignore were observed; 1,975 registered tests remained unobserved. Its corrected
process monitor fully waited owned descendants, but that does not make the run
green. Both failed attempts remain intact. Live diagnostic compatibility must be
prevalidated before another complete attempt.
Untimed native execution of the initial acceptance harness completed 110 builds and 260 exact-oracle runtime cases, but ten source checks failed and sixteen planned cases remained blocked. Reviewed harness corrections fix the corpus error union and add honest mode selection. The reviewed shared Unit checked-suite repair (issue #712) now preserves the exact entry type stack for Unit success, without changing nonunit behavior or codegen. Five verifier and four native normal/release regressions cover strict stack checking, ordered mutations, destruction/propagation and floating division. The same reviewed production hunk is committed on a new baseline fork, labelled “main + #712 fix”; original pins and failed-probe records remain unchanged.
Reviewed call-heavy probes now name the live multiplication result before adding the bias, preserving operations and error payloads while retaining a genuine leaf call. All four optimized modes and both recovery variants pass actual native call-site extraction on the historical syntax pair. The controller fails closed when extraction cannot qualify a call; an additional I-side cleanup case proves no rollback after an earlier successful mutation. These historical results are now supplemented by independently reviewed final-pin untimed validation on the common-fix baseline and candidate: all 124 source/profile configurations pass checks and native builds, all 284 runtime cases pass their exact oracles, and all six optimized executable call-retention gates pass. Both full corpus sizes execute their designated checksum entrypoints in all four modes and both generated profiles. All 24 cleanup cases pass, including C’s formerly blocked Unit/no-rollback cases. Four C/I allocation-free intervals and four real allocating controls prove active calling-thread successful-allocation checks, not a process-wide allocation census. No timing or size acceptance is inferred. Both matching perf/check build pairs and the genuine allocation-profiling build pair also pass preparation. The new baseline also executes the previously rejected original C cleanup source in both native modes. Profiling will report exact allocation counts and clearly label formatted byte totals approximate; no profiling or timing samples have yet been collected. The guarded measurement controller and separate profiling adapter are now independently reviewed and integrated. The combined Python suite passes 311 tests with three existing skips; their execution paths remain mock-validated, not measured. Runtime timings will be whole-process costs; compiler costs cover checks, debug objects and AOT builds. Optimized linked-ELF sizes are distinct from optimized object bytes, and profiling timings are not uninstrumented latency.
Checkpointed validation retains exact target/test identities on the unchanged verified build. After reviewed external-wrapper corrections, all 5,667 registered pairs have observed outcomes across stages: 5,659 passes and eight normal ignores, including all 27 required optional bodies. Passed prefixes were not replayed to complete that checkpoint. Cross-stage coverage is not an uninterrupted gate.
The separate final uninterrupted command completed all targets and both actual zero-test doctest packages, but failed: 5,658 passed, one failed and eight normally ignored; Cargo returned 101, with full wait and no wrapper STOP. Two concurrent Caddy fixtures selected the same HTTP port, and readiness correctly rejected the other fixture’s identity. Issue #719 tracks this pre-existing fixture race in a separately authorized fixing task. No compiler/arithmetic regression is established by that failure. All ten observed containers had completed exact-ID removals; foreign container and image identities were unchanged. Missing final wrapper classifications remain unknown, not Docker success.
The user then approved applying only the reviewed Rust fixture patch to an
isolated validation fork. Its entire proxy-test file equals the separately
reviewed fix; all compiler, runtime, std, Cargo and native Plenty fixture files
remain byte-identical to production revision 830763c9. On validation revision
8ca6fc5a, one uninterrupted gate passes 5,669 tests, zero failures and eight
normal ignores, including all 27 required optional bodies and ten new fixture
helpers. Both actual zero-test doctest packages complete. Independent evidence
review accepts this isolated host-gate result, not either unchanged PR branch or
a merge. The earlier failed run remains failed evidence.
All ten real proxy containers have authenticated same-ID removals and subsequent absence; foreign container and image identities are unchanged. Missing wrapper classifications stay unknown. Critical compiler/runtime/test build freshness is verified; ten non-test example paths’ pre-clean absence remains unobserved. Existing external engines remain older compatible builds, not fresh source builds. Process sampling is not exhaustive tracing or a controlled performance baseline.
Exploratory collection passed 16 runtime samples, 32 compiler samples and 12 code reports, with no warmups and one sample per selected case. These smokes qualified collection contracts, not speedups or acceptance-quality distributions. The first separate two-job profile attempt remains stopped before either producer ran: its identity guard rejected a packaged GCC hard link (issue #731).
The reviewed correction authenticates recorded native-tool identities, binds fresh link-count/ctime observations and preserves strict nonalias rules for inputs, proofs, frozen binaries and outputs. It consumes the exact authenticated record bytes, with deterministic record-swap regressions. All 102 profile and 182 combined controller mock tests pass. A new external driver then passes independent review and 58 synthetic methods, followed by a fresh successful two-job smoke. These do not relabel the original blocked attempt.
The full approved general schedule completes once: 2466 timed jobs, comprising 366 warmups and 2100 samples, plus qualification and code inspection. All 86 groups, 75 available paired distributions and 22 code reports retain their raw streams, resources and artifacts. No outlier or failure was dropped. All 350 required arithmetic traps correctly exit 1. The official full allocation-profile schedule separately completes all 16 jobs and retains 1248 phase rows; exact attributed request counts and approximate binary-unit byte displays are distinct from process allocation totals or retained heap. Independent evidence review finds no issues with these bounded collection components, not whole-proposal performance acceptance. The helper’s packet/acceptance-complete flags remain false.
Representative optimized-IR/native analysis accounts for arithmetic guards, Result construction/tests, cleanup and return carriers in 20 selected symbols. Trapping and recoverable pairs preserve their intended contracts. The C→I corpus has an additional Result widening/tag test, one fewer cleanup call site and a smaller local frame; this is not a measured speedup or universal ABI conclusion. The separate bounded native/export source audit finds no defect in changed typed storage, entry stacks, executor operands, partial mutation or explicit C adapters.
The repository compiler benchmark template is now mechanically adopted with independent precise errors and unchanged workload/measurement controls. Nine example tests, including full 1000/4000-unit type-check/native checksum cases and partial-mutation failures, pass alongside 15 arithmetic tests, formatting and strict workspace/all-target Clippy. These functional checks are not new timing samples. Frozen comparison inputs remain unchanged.
The validation record gives paired timing distributions, attributed allocation values, static code counts, source bridging and observation limits. Neither PR is merged; the proxy fixture patch remains only in the explicitly approved isolated validation fork. Independent pre-existing archive-decimal and PNG-framing limits are recorded as issues #705 and #707, not silently changed by arithmetic migration.
Reuse and limits
Source inspection at local main revision 96f6fc42 found useful existing
machinery in src/frontend/checked.rs, src/codegen/checked.rs, and
src/numeric.rs: a stack of checked scopes, per-operation error collection,
cleanup paths, and cold exits producing a typed error. Scope records already
distinguish function forwarding from arithmetic checking. The parser already
recognizes contextual checked: suites in value positions.
Adding an explicit trapping mode and retaining nested checked: should be a
contained extension of that machinery. Represent the nearest arithmetic mode
explicitly; do not merely search past an unchecked scope for an outer checked
target. Function failures use the declared public Result ABI. Remove the
private wrapper and forwarding machinery for lexical checked callbacks under
the uniform function rule below. Success returns, ?, early exits, cleanup and
tail calls need auditing.
The whole change is not a small default toggle. Most arithmetic-bearing functions currently declare plain returns, and generic/protocol bodies, callbacks, generators, runtime helpers and std signatures depend on that. The core implementation reuses these paths; no timing experiment has been performed.
Functions and callbacks
Every named function and anonymous function has its own default error boundary,
regardless of the mode surrounding its definition or call. Its declared return
type must cover its escaping arithmetic errors. A checked: or unchecked:
suite affects arithmetic in that function body, not a nested function’s body.
Inlining must preserve this distinction.
Remove the special hidden error channel for closures defined inside explicit
checked: blocks. That behavior was implemented for issue #614 under the old
opt-in model; retaining it would give closures two different error mechanisms
depending on their definition site. With checked defaults, an ordinary declared
Result provides the same recoverability and a visible contract. Existing
scope-bound callback behavior is not a compatibility requirement.
def make_incrementer() -> Callable[[i64], Result[i64, OverflowError[i64]]]:
def(value: i64) -> Result[i64, OverflowError[i64]]:
Ok(value + 1)
This closure needs no surrounding checked block or private error channel. Its
arithmetic contract does not tie its lifetime to its definition scope; ordinary
capture ownership and borrow rules still apply. A closure defined inside
unchecked: likewise defaults to recoverable arithmetic. To opt its
arithmetic out, put unchecked: inside the closure body, where it is visible.
Calls returning Result still require explicit handling, even within checked:.
Using ? there exits the function as usual; it does not turn the call’s error
into the enclosing block’s result. Local recovery from a callee uses match or
ordinary Result operations. The cost of the uniform rule is that a checked
block cannot transparently collect arithmetic errors from an arbitrary callback
invocation; the callback and its consumer expose and handle them normally.
Update callback consumers to support the appropriate fallible return contracts.
Review pipelines, sorting, generic helpers, worker jobs and boxed callables;
change their APIs where that produces the clearest design. Do not preserve
infallible callback signatures by blanket-inserting unchecked:. Worker error
transport and resource ownership still need explicit contracts, but no lexical
checked-block lifetime or private callback effect crosses those boundaries.
Special boundaries and library code
- Generators: today’s
Generator[T]cannot return a function-levelErrfrom its suspended body. Reject unhandled recoverable arithmetic there with a targeted diagnostic; locally handle achecked:result or use a deliberateunchecked:. A fallible generator protocol is not implied by this proposal. - Destructors and protocol returns: revise protocols where fallible results fit their purpose. Where execution semantics require infallibility, such as cleanup with no error recipient, recover locally or explicitly opt out. Do not preserve a fixed return merely because it is the current API.
- FFI and exports: choose explicit ABI contracts for fallible operations.
Wrappers for foreign APIs must respect those APIs; Plenty-owned exports can
change. Checked defaults do not invent a cross-language
ResultABI. Native implementations need their own documented arithmetic audit because they are not Plenty source suites. - Generic functions and recursion: validate against declared returns and bounds, with specialization diagnostics where needed. Do not discover public signatures by a call-graph fixed point or change effects with inlining.
- Std and builtins: apply the same default to Plenty implementations. Audit
builtin integer reductions such as
sum, which currently trap regardless of surroundingchecked:. Prefer a public fallible contract where arithmetic can fail; retain a trap only as an explicit, documented choice. Fallible user operator hooks remain ordinary calls returningResult, not implicit effects. - Internal compiler/runtime arithmetic: compiler implementation language operations do not acquire Plenty source semantics. Generated operations that implement a public primitive must nevertheless honor that primitive’s contract.
These are implementation and execution-contract questions. Changing user-function lowering alone does not make all library arithmetic recoverable. Any public std signature changes need a std-conventions review and regenerated signature index as part of implementation.
Coordinated repository changes and teaching
Plenty has no external users or dependent systems. Breaking syntax, APIs and behavior is acceptable: select the clearest resulting design and update the repository together. There is no compatibility release gate, edition dependency, legacy mode, deprecation period or external migration plan.
Implementation still needs to update arithmetic-bearing functions, callback
consumers, std implementations, examples and tests. This is ordinary work to
make the repository obey the new contract, not a reason to preserve an inferior
API. Diagnostics should explain missing arithmetic errors and the choices of
propagation, local recovery or an explicit unchecked: precondition. Do not insert unchecked
blocks throughout the repository merely to make the build pass.
Explicit checked expressions remain useful for local recovery. Recovery
calculations can themselves fail and need the same scrutiny. Use a contextual
unchecked: suite consistently with the block grammar; identifier compatibility
is not a requirement if a different grammar proves clearer.
Teach simple arithmetic in a fallible entry point, with a short explanation of
Result and Ok(()), then introduce local recovery and deliberate opt-outs.
Update lessons and application examples to recommend the new behavior directly;
no old-to-new migration lesson is needed. Keep tutorial/reference examples on
current behavior until implementation lands; proposal examples use text fences.
Validation contract
Acceptance needs semantic tests for function propagation, precise unions,
missing-error diagnostics, locally handled errors, all nesting directions,
value/statement suites, and ordinary calls returning Result inside unchecked
code. Exercise every integer width, division proofs, MIN // -1, MIN % -1,
negative powers, constant operands, compound assignments and forwarded newtypes.
Ownership tests must cover resources and temporaries on failure, partial
mutation, skipped writes, ?, explicit returns, loop exits and tail recursion.
Callback tests must verify that named and anonymous functions keep their own
declared error contracts when defined or called in either block mode. Cover
stored and returned callbacks, generic borrowers, generators, worker boundaries
and ordinary borrow-based escape restrictions; remove tests for the retired
private checked channel. Inlining and optimization
must preserve the lexical contract. Trapping tests need child processes and must
verify termination in both debug and optimized builds.
Compare today’s unadorned trapping arithmetic, today’s explicit checked block, proposed implicit checked function, and proposed unchecked block on equivalent workloads. Inspect emitted IR/object code as well as timings: count guards, result construction, cleanup and return ABI differences. Measure tight loops, call-heavy kernels and realistic std consumers on success and failure paths. Verify the no-allocation arithmetic contract and report compiler/build-time and code-size costs. No speedup claim is justified before these measurements.
The original checked-arithmetic design
records why local PR #678, which returned a Result from every operator, was
abandoned. Preserve its lesson: use plain values between operators and a shared
failure exit, rather than reviving per-operator wrapping and unwrapping.
Adoption also requires updated executable tutorial/reference examples, updated arithmetic and trap documentation, the std signature index, the full relevant compiler/runtime tests and the complete docs CI checks. The proposal itself changes none of those implemented contracts.
Unchecked arithmetic modes
Status: Partly implemented.
--unchecked trap|assumeandunchecked trap:are implemented; see the numeric reference, which is authoritative. Aplenty.tomlsetting and the stdlib precondition audit remain under B14 in the backlog; until that audit,stdalways traps. Issue #764 also extends the mode to list, slice-view andBytesindex syntax. This builds on checked arithmetic by default.
Contract
Plain unchecked: transfers responsibility for arithmetic preconditions to the
programmer. A compiler setting selects whether those preconditions are checked
at runtime or assumed valid. Trapping is the initial default in every build
profile; selecting optimization alone must not enable undefined behavior.
| Source form | Global trapping mode | Global assumed-valid mode |
|---|---|---|
| Ordinary arithmetic | Declared recoverable arithmetic contract | Same |
checked: | Local recoverable arithmetic contract | Same |
unchecked: | Check arithmetic and trap on violation | Assume arithmetic preconditions; omit their checks |
unchecked trap: | Check arithmetic and trap on violation | Check arithmetic and trap on violation |
Both unchecked forms produce an ordinary success value, without adding
arithmetic errors to the function’s return type. unchecked trap: makes
termination on an arithmetic violation a permanent source-level guarantee.
Plain unchecked: must not promise termination independently of configuration.
The diagnostic mode catches violated obligations without changing signatures;
passing that mode’s tests is evidence, not proof for all possible inputs.
In assumed-valid mode, overflow is not allowed and is not wrapping. Violating an arithmetic precondition is undefined behavior: the compiler may remove branches or transform loops on the assumption that violation never happens. Silent wrong results and memory corruption are possible; a crash is not guaranteed. An actual machine instruction wrapping does not define the source expression’s result. Use explicit wrapping operations for modular arithmetic; explicit wrapping, saturating and checked methods retain their own contracts in either mode.
Overflow obligations
The no-overflow obligation applies uniformly to signed and unsigned integers.
Every intermediate result of ordinary integer arithmetic must be representable
in that operation’s type, not merely the final result of the expression.
For example, a * b // c violates the contract if a * b overflows even when
the final mathematical quotient fits. Use a wider type, an algorithm that avoids
the overflowing intermediate, or explicit wrapping when modular arithmetic is
appropriate to the algorithm.
| Arithmetic contract | Overflow detection required? | May assume overflow never occurs? |
|---|---|---|
| Explicit wrapping operation | No | No |
Ordinary arithmetic in assumed-valid unchecked: | No | Yes |
Ordinary arithmetic in trapping unchecked: or unchecked trap: | Yes, unless proven unnecessary | Only on paths where checking or proof establishes it |
Wrapping already permits arithmetic without overflow checks. Undefined overflow
adds optimization freedom: for example, the compiler may treat x + 1 > x as
true when the addition carries the no-overflow obligation. This is not a valid
way to detect overflow after the operation. The benefit beyond check omission
depends on the backend and workload; the source contract does not guarantee a
speed improvement over wrapping.
Arithmetic scope and boundaries
The operation contract must specify which preconditions a block transfers: representable integer addition, subtraction, multiplication, negation and powers; valid integer divisors and representable quotients; valid shift counts; and the existing validity conditions of numeric conversions and integer power exponents. Preserve Plenty’s existing successful arithmetic semantics, including floor division and modulo. Do not import a machine instruction’s different semantics or forbidden cases as language rules. In particular, a defined signed minimum remainder by minus one must remain zero if that is the existing contract.
IEEE floating-point arithmetic retains its existing behavior, including NaN and infinity. Float-to-integer conversion retains its existing valid-input rounding rules; its range/validity obligations must be specified separately from ordinary floating arithmetic. Compile-time invalid constants remain compile errors. No new recoverable shift or cast API is implied by this proposal.
Issue #764 extends the implemented mode to list, slice-view and Bytes
index syntax: plain assumed suites require nonnegative, in-range indices and
omit bounds checks and negative-index normalization. Trapping suites preserve
normal negative indexing. Other indexing and collection methods remain checked.
Allocation failure, capacity/layout validation for memory allocation,
ownership, borrowing, I/O failures and explicit unwrap()
are outside this feature. Memory-management arithmetic must still establish
its own safety invariants. The global mode is not permission to skip allocator
validation or to assume an allocation succeeded.
Modes apply lexically to arithmetic, not dynamically across calls. The nearest
explicit arithmetic suite controls an operation; leaving it restores the outer
mode. A nested plain unchecked: uses the global setting, even inside
unchecked trap:. A nested checked: restores local recovery. Named functions
and closures keep their independent contracts; merely defining or invoking one
inside an unchecked suite does not change its arithmetic mode. Inlining must
preserve these boundaries. ?, returns, loop exits and ordinary Result values
keep their existing meanings.
The setting must have explicit, documented CLI/package configuration and
precedence. Its spelling is an implementation decision. It must participate in
compiled-artifact and function-cache identities, including dependency builds,
so changing the setting cannot reuse code with the opposite contract. Prebuilt
libraries retain their compiled semantics; a caller setting cannot retroactively
change them. A source unchecked trap: guarantee must survive all build modes,
worker lowering, specialization and cross-module optimization.
Standard library rationale
Validate public inputs, then use a small plain unchecked region only where the
validated inputs and maintained invariants establish its preconditions. If a
block intentionally rejects invalid input by trapping, express that guarantee
with unchecked trap: or explicit validation. Merely enclosing existing code
in an unchecked suite does not establish a bound.
Classify each audited operation by intent: established bounds permit plain
unchecked:, intentional modular arithmetic requires explicit wrapping, and
invalid-input rejection requires unchecked trap: or explicit validation.
Review arithmetic opt-outs already introduced by checked-default adoption before enabling assumed-valid mode for the stdlib. Preserve resource-failure mapping, earlier successful mutations, callback error propagation and ownership cleanup on defined paths. Use the repository’s consistency-check procedure for a repository-wide audit; proposal examples and historical fixtures must be identified separately from executable code.
Validation rationale
The modes need independent evidence. Invalid arithmetic belongs in trapping-mode tests, which must check the trap and source location without widening signatures. Never execute undefined arithmetic in assumed-valid mode to assert a particular crash, wrapped value or output. Its runtime tests use valid boundary inputs; static IR/native inspection establishes check omission and supported assumptions. Explicit trapping suites need invalid-input tests in both global modes. Cover signed and unsigned overflow obligations, including an overflowing intermediate with a representable final mathematical result, in trapping tests. Test explicit wrapping operations at overflowing inputs in both global modes and both unchecked block forms: those operations remain defined.
Check nested modes, callbacks, generators, compound writes, short-circuit paths, integer widths, divisors, shifts, conversions, powers and mode-sensitive cache reuse. Verify that allocation, indexing outside assumed suites and explicit checked/wrapping operations retain their contracts. Compiler backends must preserve valid-input semantics; omitting guards alone does not establish that all desired optimizer assumptions are represented.
Follow the emitted-code procedure: isolate small kernels, compare equivalent C contracts, attribute remaining instructions and measure optimized builds by instruction count. The programmer’s explicit precondition is the premise for removing a check in this feature; ordinary safe code still needs a compiler proof. Report backend limitations and measured results rather than promising that unchecked code is always faster. This contract is permission to optimize, not a speed guarantee.
Recursive owned data
Status: Partly implemented. Recursive classes, enums, aliases, and finite generic instances are implemented, with moves, matching, borrowing, and allocation-free drop. Recursive storage and boxes own the contract.
Shipped differently: this page first proposed that classes and enums stay heap records, so recursion needed no
Box. Classes and enums are now inline values, and every storage cycle must pass throughBox[T]or a collection. The source model below describes what shipped.Remaining: deep copy, equality, and formatting of recursive values (B33); stored references (B11). Shared mutable graphs, weak edges, and a garbage collector are out of scope.
For practical linked-list and tree patterns, see the owned-structure assessment. Its checked-index arena experiment addresses graph identity and access; this proposal addresses finite storage and destruction. An index arena is not a new borrowing capability.
Source model and finite storage
Classes, tuples, and enums store their fields inline, so a type that contained
itself directly would have infinite size. A recursive field goes through a heap
boundary: Box[T], a list, a set, or a dictionary. A cycle without one is
rejected, and the diagnostic names the boundaries to use.
class Node:
value: i64
next: Option[Box[Node]]
enum Tree:
Leaf(i64)
Branch(Box[Tree], Box[Tree])
Node(1, Nothing) and Tree.Leaf(1) allocate nothing and cannot fail. Some(node)
is an inline ownership wrapper. Box(value)? is the one allocation per link, and
its failure drops the value it consumed. Lists of children use their existing
fallible allocation API.
Keep transparent alias cycles invalid: type Loop = Option[Loop] has no nominal
ownership boundary and must not be expanded forever. Separately distinguish a
finite nominal recursion from infinitely growing generic instantiations such as
Node[T] containing Node[list[T]]. Neither a type spelling’s nesting limit nor
the runtime value’s depth is a substitute for that distinction.
Ownership and destruction
Recursive classes and enums move under the existing rules. Aggregates containing recursive values are also affine, and automatic deep copying is rejected. Mutable collections or classes already make their containing values affine. Storing an owner inside its own descendant must fail the normal move/borrow checks. There are no stored references or weak edges in this design, so source ownership cycles are not introduced by recursive types.
Drop uses the existing intrusive destruction queue and must allocate nothing, even for a very deep chain. A class hook still runs before its fields, and explicit nested drops in a hook finish synchronously. Queue links reuse dead header storage only after atomic final-release ownership has been established. Recursive values do not become transferable across threads merely because their counts are atomic; the worker eligibility rules apply to every reachable field and destructor effect.
Compiler identity and bounded analysis
Use a compilation-owned table of nominal definitions. Type edges refer to stable
identities; the table owns definitions, avoiding an Rc cycle that leaks compiler memory.
Reserve identities before resolving fields, then finalize immutable definitions.
Aliases resolve to those identities. Type equality and hashing compare nominal
identity and concrete generic arguments, never recursively expanded fields.
Compute properties over the finite type graph, with explicit monotone rules for copyability and destructor presence; a temporary placeholder must not cache a wrong fact. The implementation uses visited identities and leaf elimination to find recursive storage and aggregates that reach it. More detailed component analysis may support future thread effects without changing source syntax. Keep finite inline-layout checks distinct from heap graph recursion. Emit native descriptor IDs before their edges, as the current metadata emitter already does. Every metadata/visibility/debug traversal needs visited identities or a bounded worklist. Declaration errors should name the actual dependency cycle.
Operations that need their own depth contract
Construction, movement, matching, field borrowing, and iterative drop are
implemented. Deep copy, structural equality, hashing, and formatting recurse
through runtime values, so on recursive data they would turn ordinary valid
values into stack overflows. The compiler rejects them for any type whose stored
values can recur.
B33 lifts that restriction. Prefer iterative fallible work storage for already-fallible copy and formatting. Equality and other allocation-free operations cannot silently add a fallible allocation; they need a bounded-memory traversal. Direct user-written iterative traversals remain possible. This restriction is preferable to a hidden small maximum on the depth of every recursive value.
The backlog tracks implementation and acceptance work; this page records the representation and behavioral constraints, not a second task queue.
Owned structures and checked node identities
Status: Investigation complete. Owned-chain and tree-rotation examples, an arena lesson, and a checked-source prototype exercise existing Plenty. No
std.arenaAPI or new borrowing capability has shipped. The recursive-storage, borrowing, and box references own current behavior. B73 in the backlog owns arena delivery; B11 owns broader borrowed traversal and collection-loan precision.
For representation and destruction rationale, see recursive owned data. This assessment focuses on how to write and manipulate those structures using today’s language.
Make the correct representation easy to use
The goal is that a safe, efficient implementation should also be the obvious and simplest implementation for its workload. Memory safety alone is not enough: failure can destroy data, stale indices can identify the wrong object, and a recursive traversal can exhaust the native stack even when automatic cleanup is bounded-stack.
The investigation distinguishes uniquely owned chains/trees from parent links, shared subgraphs, and intrusive collections. Direct box field access reduces syntax but does not change their ownership requirements. A generic arena is a better first library candidate than separate general-purpose list, tree, and graph APIs: it centralizes identity, deletion, and checked access without fixing application-specific edge semantics.
Evidence and limits
The examples and tests/test_data_structure_patterns.rs preserve these findings:
| Experiment | Result |
|---|---|
| Owned linked stack | Fallible push, ownership-transferring pop, borrowed iteration, and mutable iteration work. |
| Tree rotation | Existing boxes can be rewired without allocating, including transfer of the pivot’s middle subtree. |
| Deep chain | One million nodes traversed, mutated, and explicitly dropped with allocations disabled under a 256 KiB native stack on Linux. This is a regression check, not a throughput benchmark. |
| Failed insertion | Allocating an empty node before linking preserves the existing chain. Detaching into a fallible box constructor does not. |
| Arena links | Indices express back/parent links and shared targets without ownership cycles. The TOML parser already uses this representation. |
| Borrowing limits | Stored references, including a worklist of borrowed tree nodes, are unsupported. Different list indices do not establish disjoint mutable loans. |
Bounded-stack cleanup is a useful semantic advantage, but recursive user calls and destructor hooks still consume native stack. The current box runtime carries a 32-byte header before its payload; simpler syntax is not evidence of lower per-node memory cost. No comparative timing benchmark was performed.
For ordinary stacks use a list; for queues use the existing
std.deque. Its growth already preserves existing
contents on failure. Keep owned chains and tree rotations as
teaching examples until an
application motivates a stronger packaged collection API.
Candidate arena contract
A candidate std.arena would own node values and issue typed handles. Handles
identify nodes; they neither own nodes nor keep them alive. The arena validates
identity before granting temporary access. Deletion invalidates the handle;
slot reuse must not make an old handle identify a replacement.
- Typed handles distinguish payload types. Arena identity must also distinguish separate instances with the same payload type, including after destruction.
- Growing storage preserves logical IDs but does not guarantee stable addresses.
- Reusing vacant slots changes a generation. Exhausted slots retire permanently rather than wrapping. Arena-identity exhaustion needs an equally explicit rule.
- Failed insertion preserves all existing elements and handles. The initial
candidate consumes the incoming value on failure, following
Deque.append; an ownership-returning variant is a separate contract to evaluate. - Lookup and removal reject invalid/stale handles through a checked result. They do not trap just because a handle has expired.
- Node removal does not silently rewrite inbound edges. Applications either remove those edges or handle a failed lookup. Unreachable nodes remain owned until removed or the arena is dropped.
- Safe access to two nodes rejects equal or invalid handles before invoking the callback. A callback error restores storage ownership, but does not roll back mutations already performed by the callback.
Returning Option[&T] or a tuple of mutable references is not an available
implementation strategy today. Scoped callbacks returning owned results fit the
current storage rules. Their usability and move costs need evaluation before a
public API is frozen. A separate language change could eventually offer direct
disjoint loans; the library should not require one merely to demonstrate value.
Prototype
The prototype assessment describes the executable module
in examples/arena/arena.plenty, its API, validation, and deliberate namespace
limitation. It is ordinary Plenty and is not embedded under std.
This distinction matters: requiring callers to invent unique namespace numbers would move a correctness problem into application code. The prototype proves generation validation and scoped access, not a satisfactory identity issuer. A production constructor needs automatically unique identity within a documented domain. A non-reusing runtime token is one possible narrow primitive; an owned issuer with enforced domain separation is another. Concurrency, copying, serialization, and exhaustion determine which contract is sound and convenient.
std.atomic and
std.sync.Mutex now provide synchronized mutation
through scoped shared borrows. An atomic counter could serve one identity issuer,
but separate counters can issue the same numbers: issuer uniqueness and lifetime
remain part of the arena contract. Atomics do not by themselves provide shared
ownership of an issuer or arena.
Issue #326 separately explores a counted shared owner for user data and a thread-shared HTTP pool. Its eventual access, cycle, and destruction rules may offer another representation for shared nodes or a shared identity issuer. These examples and the arena prototype do not depend on that design, and do not choose its cycle policy. Reassess the representation guidance when that feature ships; retain arenas for workloads that need checked logical identity and explicit removal independently of shared-owner lifetimes.
Rust and intrusive collections
Rust also supports safe owned singly linked lists and trees. It already provides
field autodereferencing, so Plenty’s omitted dereferences are a modest ergonomic
difference. Recoverable allocation is another difference: Plenty’s box constructor
returns Result and destroys its input on failure. Compare failure contracts,
not only the shortest successful-path expression.
An intrusive collection embeds links in application objects, potentially for multiple memberships. Pointer-based implementations can need stable addresses and lifetimes governed outside the collection. These requirements differ from an ordinary owning doubly linked list. An arena can emulate many link topologies while changing physical-address and lifetime semantics. It is not a replacement for all pointer-intrusive use cases.
Safe libraries can encapsulate trusted implementation machinery even when the
source language has no unsafe keyword. Add such machinery only when a concrete
workload needs stable-address/external-lifetime semantics. There is no evidence
here about which kind of list programmers complain about most.
Relevant precedents, consulted for the original investigation: Rust linked-list guidance, checked disjoint slice access, slotmap’s versioned keys, and intrusive-collections.
The documentation teaches implemented behavior. This proposal preserves the representation decisions and alternatives; priorities and delivery work remain only in B73/B11 in the backlog.
Arena prototype assessment
This accompanies the arena design assessment. It is an experiment in existing language facilities, not a stable standard-library contract.
Run the demo from the repository root:
cargo run --locked --bin plenty -- examples/arena/main.plenty
examples/arena/arena.plenty owns list[Slot[T]]; each slot stores an optional
value, a generation, and a free-list link. Key[T] carries the namespace, slot,
and generation, with no stored T. The demo transfers between two integers,
removes one, reuses its slot, and shows that the old handle fails lookup.
examples/arena/parent_links.plenty stores typed parent handles inside nodes,
duplicates a handle without copying its node, and detects a removed parent.
| Prototype operation | Result and behavior |
|---|---|
Arena[T](namespace, generation_limit=i64.MAX) | Fallible construction; namespace uniqueness is the caller’s responsibility. |
insert(value) | Result[Key[T], AllocError]; reuse a vacant slot or grow storage. |
contains(key) | Check namespace, bounds, generation, and occupancy. |
remove(key) | Option[T]; transfer the owner and invalidate its handle. |
read(key, callback) | Option[R]; callback receives &T and returns a storable result. |
update(key, callback) | Option[R]; callback receives &mut T. |
update_two(a, b, callback) | Option[R]; reject identical slots or invalid handles, then lend both values to a callback. |
len() | Number of live values, excluding vacant or retired slots. |
key.duplicate() | Another key for the same node, without allocation or payload copying. |
The module uses a custom empty destructor to forbid explicit copying of arenas;
otherwise copying storage would duplicate its namespace. Moving an arena
preserves its keys. Key.duplicate reconstructs three integers; it avoids the
fallible generic aggregate copy API and never duplicates a node payload.
The reduced generation limit is an experimental test seam for exhaustion, not
a proposed production tuning knob. Slots retire on exhaustion without wrapping.
Two-node updates without simultaneous indexed loans
update_two validates first, moves both values into local owners, lends those
locals to the callback, then restores them before returning the callback result.
The exclusive arena borrow prevents the callback from re-entering that arena.
References cannot escape through the stored Option[R] result.
This does not allocate, copy payloads, or run their destructors during transfer.
It works for affine payloads. It does physically move values, potentially
several times for a large inline T; it does not promise stable addresses or
zero memory traffic. A callback returning Err still restores both values, with
its mutations intact. Fatal traps do not unwind; restoration after a process
abort is not promised.
Borrowing distinct indices directly remains rejected. The prototype shows a library alternative, not a change to collection loans. It deliberately avoids temporarily allocating a second container or requiring copied payloads.
Failure and cost boundaries
Reusing a free slot and removing a value allocate nothing. Fresh slots use fallible list append; existing contents and keys survive failure, while the incoming value is destroyed. The free list lives in vacant slots, so removal does not need an allocation. Retired slots are retained as empty storage.
Lookup, removal, and reuse take constant time apart from payload operations; growth has list allocation/movement costs. Callback work and returned values retain their own costs and failures. An arena does not enforce tree/graph invariants, collect unreachable nodes, or repair incoming links.
What is proved, and what is missing
Native tests cover successful access, deletion/reuse, stale and foreign-namespace
rejection, generation exhaustion, affine destruction, callback errors,
allocation-free reuse/pair access, and preservation during failed growth.
The demo and module are exercised by tests/test_data_structure_patterns.rs.
Namespaces are supplied integers. Two arenas given the same namespace can accept
one another’s keys if the slot and generation coincide. Negative namespace
values are not special. This is an explicit prototype limitation, not safe
default instance identity. The production design must solve it before this can
become std.arena. The module also does not supply serialization, iteration,
compaction, shared concurrent access, or a node-address guarantee.
These findings support the generic arena direction while leaving its production identity and access API open for review. Delivery and acceptance criteria live in B73 of the backlog.
Typed ranges, generic calls, and expression-local inference
Status: Partly implemented. Recommended 2026-10-06. Typed ranges, contextual numeric literals,
IntType, explicit type arguments, and anonymousdefshipped; see ranges, numbers, generic inference, and function values.Shipped differently: the compiler passes expected types down as hints; it has no inference variables or deferred defaulting. A generic parameter position gives a literal no context. An integer literal is accepted in a float context. Generic bodies are checked per instance.
Remaining: expected-result inference, partial explicit type arguments, and literals across generic arguments (B16); an iterator protocol (B12); mutable/consuming erased callbacks (B32). Explicit fallible shared-call erasure already supports heterogeneous shared-call storage. Inclusive ranges and indented anonymous bodies inside delimiters are unscheduled.
The completed callable surface review and stored-environment decision own the callback design history. Allocation-free ranges record representation, separate from the inference choices discussed here.
Recommendation
Yes: use range[u8](8) and support contextual inference from list[u8].
Square brackets should be the ordinary explicit type-argument syntax for
generic functions, including qualified calls such as numbers.range[u8](8).
No extra separator is needed before the brackets.
The proposal was that all three examples below produce list[u8]:
squares = [n * n for n in range[u8](8) if n % 2 == 0]?
squares: list[u8] = [n * n for n in range(8) if n % 2 == 0]?
squares = [n * n for n in range(8) if n % 2u8 == 0]? # rejected
The first states the range’s type directly; the second states the desired collection type. Those are the clearest forms to teach, and both shipped. The third needs the constraint solving proposed below and is rejected: a filter operand does not select the range type.
Numeric suffixes such as 2u8, 42i32, 3.5f32, and 1e-3f64 already work.
Keep them as exact type constraints. The new convenience is contextual typing
of unsuffixed literals, not implicit numeric conversions.
Infer before defaulting or generating operations
At proposal time the compiler lowered a comprehension’s iterable before its result expression and assigned concrete numeric literal types immediately. It could not make the second and third examples work just by accepting a new call spelling. Introduce a typed-expression inference step before lowering to operations:
- Instantiate a generic signature with fresh inference variables, or substitute the explicit type arguments supplied by the caller.
- Collect constraints from declared types, explicit literal suffixes, argument and result types, operators, branches, and comprehension bindings/filters.
- Solve equalities and check bounds. Only then default unconstrained integer
variables to
i64and floating-point variables tof64. - Validate literal representability and lower concrete types to existing IR.
For the annotated comprehension, the result element must be u8. Multiplication
has equal operand/result types, so both uses of n must be u8. That determines
the range’s element type. Its stop literal 8, the remainder divisor 2, and
the comparison operand 0 can then be checked as u8 literals.
For the suffix example, % requires equal operand types: 2u8 constrains n
to u8. That propagates to the iterable and multiplication. Comparison produces
bool but still requires its numeric operands to have the same type.
No constraint source silently overrides another. For example, an annotated
list[u8] conflicts with range[u16](8) when the element expression is n * n.
An explicit cast in the element expression can make that intentional instead.
Keep the inference region small: an initializer, an assignment RHS, or a return expression, including nested calls, comprehensions, and expression-valued branches. A function’s declared return type constrains its tail expression and each explicit return. Function interfaces still require parameter and return types. Do not infer signatures from callers or inspect callee bodies to infer call-site types.
Once an unannotated local initializer has been resolved and defaulted, its type is fixed. A later use does not retroactively specialize it:
values = range(8) # defaults to range[i64]
squares: list[u8] = [n * n for n in values] # error
Use values = range[u8](8) at the earlier binding instead. Likewise, a declared
return type constrains a comprehension used directly as the final expression;
it does not reach back through an already defaulted local initializer. This
boundary keeps diagnostics predictable and avoids whole-function inference.
An output annotation also cannot determine every input in an arbitrary chain:
values: list[u8] = [u8(n) for n in range(8)]
Here the cast explicitly separates input and output types. The range can still
default to i64. Types flow through the known signatures of operations, not
through guesses about their behavior.
Literal context is not conversion
Under the proposed rules, x: u8 = 8 and x: f32 = 3.5 become valid because
the literal is initially unresolved. However:
limit: i64 = 8
values = range[u8](limit) # error: an existing i64 is not a u8
values = range[u8](8i64) # error: an explicit suffix fixes the type
values = range[u8](300) # error: the literal cannot fit in u8
Retain explicit casts for numeric conversions. Initially keep integer-shaped
unsuffixed literals in the integer family and decimal/exponent literals in the
floating-point family. To spell a floating-point whole number, use 1.0 or an
explicit suffix such as 1f32, which is already supported.
Contextual literals are a deliberate change to the current language contract.
Some examples that currently teach small: f32 = 1.5 as a type error must become
successful examples when this is implemented. Transparent aliases should supply
the same context as their targets; they still need not create new literal suffixes.
Integer arithmetic remains checked at its inferred width: u8 multiplication
must not secretly widen just because the expression appears in a comprehension.
IntType is a compile-time constraint
The sketched declaration is workable:
def range[T: IntType](stop: T) -> range[T]:
...
Here the return spelling describes a concrete built-in iterable. The example
is a signature sketch, not a requirement to implement this intrinsic in Plenty
source immediately. A user-written yielding function would naturally return
Generator[T] using the existing generator model once generic functions exist.
Define IntType initially as a compiler-known constraint admitting exactly
i8, i16, i32, i64, u8, u16, u32, and u64, including aliases of
those types. It excludes bool and floats. It is neither a runtime sum type nor
a type alias for a union. Its contract grants the operations required for integer
code: comparisons, checked arithmetic, and representable integer literals.
This does not require user-defined arithmetic protocols in the first release. Ordinary structural protocols remain the plan for user-defined method contracts. Keep these mechanisms distinct; a method name alone cannot establish built-in integer representation or arithmetic semantics.
Do not introduce a runtime-erased Iterator[T] merely to express range.
The existing language distinguishes concrete iterable values and generators;
an iterator protocol can later describe shared operations and borrowing rules.
Returning a concrete range preserves cheap repeated iteration and indexing,
which differ from consuming a generator.
Initially specialize reachable generic calls and reuse an instance for each declaration/type-argument combination. Check generic bodies against their bounds. Do not add overload search, implicit trait imports, compile-time arbitrary execution, or unconstrained specialization as part of typed ranges.
Range boundaries and descending unsigned iteration
The one-argument form is straightforward: both implicit zero and stop have
type T, and yielded elements have type T. Inferring T from a typed argument
should work too: range(8u8) implies range[u8](8).
Recommend that the three-argument form use T for start and stop, but a
separate signed i64 for step. This allows:
range[u8](8, 0, -1) # yields u8 values 8 through 1
The step is a displacement, not an element. Zero remains invalid. This deliberately
limits the step magnitude to the supported i64 domain even for u64 elements;
larger displacements can be a later extension. The concrete intrinsic may retain
its existing one-, two-, and three-argument forms without committing the language
to general overloads or default arguments yet.
Keep exclusive endpoints in T initially. Therefore range[u8](256) is a
compile-time error, and an ascending exclusive u8 range cannot include 255.
Do not silently widen the bound or wrap it to zero. A future inclusive-range API
can cover the full domain; a separately typed bound would be an alternative design
but would no longer have the simple stop: T signature above.
Iteration must check whether another element exists before an increment that
would overflow the element type. For example, range[u8](254, 255, 2) yields
only 254 and terminates normally. Use widened or checked unsigned-magnitude
calculations for bounds, differences, length, and stepping, including u64 and
the most negative i64 step. Do not lower these cases through signed i64
element arithmetic. If a length cannot fit the language’s length result type,
report the limit rather than wrapping; specify that behavior alongside the API.
Reserve space for functions as values and closures
Generic application and indexing share brackets. Resolve f[T](x) as generic
application when f names a generic callable; retain callbacks[i](x) for an
indexed callable value once functions become values. The parser must preserve
this distinction for name/type resolution, rather than assuming that every
bracket followed by a call denotes type arguments. Type arguments resolve in
the type namespace, with the same explicit import rules as annotations.
Multiline anonymous functions are compatible with this direction. A promising
spelling, deliberately still provisional, is an anonymous def expression:
transform = def(value: i64) -> i64:
adjusted = value * 2
adjusted + 1
Keep its parameters and return type explicit, like a named function. Expected return types then constrain literals and generic calls in its body. Captures need no separately repeated type declarations: their types come from bindings.
An initial closure model should distinguish:
- Read-only borrowing of captured owners.
- Exclusive borrowing when the closure mutates a captured owner.
- Explicit ownership capture for a closure that outlives the surrounding scope.
- Captures consumed during a call, making the closure callable at most once.
The implemented capture syntax and limits are recorded in the closure reference and consuming closure reference. Do not infer an invisible deep copy, or decide capture ownership by whether a later compiler pass happens to let a closure escape. A borrowing closure cannot outlive its owners, and captured borrows must remain active while future calls may use them. Owned captures drop with the closure, subject to moves and consumption. The implementation extends loan liveness with environment dependencies, rather than only lowering code and data.
Use concrete environment layouts and static call contracts for generic callable parameters where possible. Avoid mandatory heap allocation, reference counting, or dynamic dispatch for every closure. A heterogeneous collection of callbacks would need an explicit representation decision later.
Anonymous suites nested inside call arguments also require a parser design: the current lexer suppresses indentation inside parentheses. Preserve the goal of multiline anonymous functions in expression positions without claiming the assignment example above settles nested suite termination or comma placement.
Suggested delivery
- Add contextual numeric literal inference and a typed-expression phase with deferred defaults. Cover initializers, calls, returns, branches, filters, and comprehension elements, including conflicting and unrepresentable constraints.
- Add explicit function type arguments and typed range representation/lowering, using the same generic-call model intended for user functions. Test unsigned descent, boundary termination, maximum-width values, and empty ranges.
- Expand to explicitly bounded generic functions and cached instantiation.
A first
IntTypeconstraint can precede general structural protocols. - Add callable values and closure capture checking, with a separately reviewed multiline expression grammar. The range work must not preclude these features.
This records the original design sequence. Typed ranges, contextual inference, generic functions, and structural protocols now have implemented paths; see the reference for their exact limits. Remaining inference and closure work is tracked only in the backlog. The learning guide continues to teach implemented behavior with runnable examples.
Package language editions
Status: Not implemented. Exploratory candidate under B17, not an implementation commitment. Plenty currently has one language mode; its manifest rejects
edition. The package reference describes implemented behavior. All edition names, settings and examples below are hypothetical.
Assessment
Plenty could plausibly support packages written in different language editions in one compilation. The existing package-aware module loader is a useful foundation. A narrow mechanism for syntax and name-resolution changes looks like moderate compiler/tooling work, with little reason for extra runtime cost. It is not just a manifest field, and it is not a cheap way to preserve arbitrary historical language behavior.
Recommendation: keep this option open, but wait for a concrete incompatible change before implementing it. Prefer additive features and contextual keywords where they solve the problem. If an edition becomes worthwhile, require the first change to demonstrate mixed-edition interoperability using one type, ownership and runtime model. Do not promise a release cadence or permanent support policy before understanding that maintenance cost.
What editions buy
An application should be able to adopt a new spelling or resolution rule without waiting for every dependency to migrate. Conversely, an older application should be able to use a newer-edition library, provided its compiler supports both. The application selects its own edition, not the edition of the whole graph.
Rust uses opt-in editions for otherwise incompatible language changes. Crates from different editions interoperate and lower to a common internal representation. That constrains which changes can be edition-specific; it does not require a separate runtime per edition. See the Rust Edition Guide.
For Plenty, distinguish four independent settings:
| Setting | Meaning |
|---|---|
Package version | Version of the library/application and its API |
plenty-version | Minimum compiler version needed to build the package |
manifest-version | Format of the package metadata |
Proposed edition | Rules used to interpret this package’s source |
A new compiler can support old and new editions. An old edition may still use new additive features, and an old compiler need not understand a new edition. Editions do not pin compiler bugs, standard-library releases, dependency versions or native ABI compatibility. They would not make a new compiler link arbitrary objects emitted by old compilers: the initial contract is compilation from source with one supported toolchain and runtime.
A package author’s experience
Use illustrative names e1 and e2, not proposed release dates:
# app/plenty.toml
manifest-version = 1
[package]
name = "app"
version = "0.1.0"
edition = "e2"
[dependencies]
legacy = { path = "../legacy", version = "^0.1.0" }
The legacy package independently declares edition = "e1". All its modules,
library targets, executable targets and tests use that edition; importing it
does not reinterpret its source as e2. Workspace members choose independently.
Start without per-file or per-target edition overrides.
For a deliberately hypothetical change, suppose e2 reserves the word memo
while e1 permits it as a local name:
# legacy/src/lib.plenty — e1
pub def answer() -> int:
memo = 42
return memo
# app/src/main.plenty — e2
import legacy
def main() -> Result[(), IoError]:
print(legacy.answer())?
Ok(())
The application’s call works; only writing a local named memo in the e2
application is rejected. Migrating legacy could rename its local variable,
then change its manifest. No caller needs to change its edition.
Public names make even this example harder: an e1 library might export a
function named memo. Before reserving it, Plenty would need a way for e2
callers to name that existing API, such as contextual member-name parsing or
escaped identifiers. Neither spelling is selected here. Renaming a public API
is not a harmless automatic migration. A new keyword that makes old exports
inaccessible fails the interoperability requirement.
Boundaries that keep this manageable
| Candidate change | Edition suitability |
|---|---|
| New reserved spelling, old syntax translated to existing operations | Promising, with public-name access and migration rules |
| Different import lookup or implicit names | Possible, if resolved in the defining package before combination |
| New opt-in feature that accepts all existing code | Usually no edition needed |
| Changed type identity, value layout, calling convention or allocator contract | Outside the narrow design; needs a shared representation or a separate interoperability design |
| Weaker historical borrow/safety checks, different destruction or concurrency guarantees | Do not preserve through this mechanism; cross-package safety obligations must still agree |
| Removing a standard-library API or changing its result/error type | Ordinary API compatibility problem; an edition label alone does not solve it |
Every package must expose types and callable contracts that the other editions understand. Protocol satisfaction, borrows, generic arguments and errors must continue to compose. A generic function’s body must retain its defining package’s interpretation when specialized for a caller from another edition; the caller must not change how that body resolves names. Prefer resolving and normalizing edition differences before shared checking and specialization.
Safety fixes apply to every edition even if previously accepted code stops compiling. More general compatibility exceptions need an explicit policy; “old edition” cannot mean “all past compiler behavior is frozen”.
Embedded std should use the edition selected by the compiler distribution,
not the importing package’s edition. Its exported contracts remain usable from
every supported edition. Generated test harnesses and interface source also
need an explicit interpretation; never inherit an unrelated caller’s setting.
Selection and tooling
If adopted, missing edition metadata should mean a fixed baseline, never “whatever is newest today”. New-package commands should write the latest supported edition explicitly. Reject unknown editions with the package name and supported values; do not silently choose a nearby version. Validate the minimum compiler requirement independently.
Standalone compilation needs an equivalent explicit option, provisionally
plentyc --edition e2 FILE, with the same fixed baseline when absent. A package
graph supplies editions per package; reject a conflicting global override
instead of applying it to dependencies. Compiler library APIs, the language
server and documentation checks must use the same selection rules.
The proposed single-file script header could carry
[package].edition. It should share the package schema and default rather than
invent another source directive. An edition does not require a script runner.
Resolution continues to choose package sources/versions. Edition is metadata of the selected package, not a request to compile that dependency in multiple editions. A lockfile is not the authority for source interpretation. Generated package graphs must preserve the edition, and build reuse must be invalidated when it changes, including when no source bytes change.
Fit with the current implementation
This is a source inspection at local main 2d6998b4, not an implemented
prototype or a measured delivery estimate:
| Area | Existing foundation and required work |
|---|---|
src/project/manifest.rs | Strict keys and minimum compiler metadata exist; tests explicitly reject edition. Add validated metadata, defaults and generator support. |
src/project.rs | SourcePackage/SourceGraph already carry package identity, roots and dependency edges. Carry edition through resolution, graph serialization and synthetic test packages. Version the graph format so older readers cannot silently drop semantics; preserve the old format as baseline input if supported. |
src/frontend/modules.rs | Loader::file already receives a package index. parse/parse_buffered create a lexer/parser without an edition. Pass the owning edition through both buffered parsing paths and retain it wherever interpretation remains deferred. |
| Lexer, parser and checker | Gate only the intended source rule and normalize it; audit generic specialization, interfaces and source-only entry points for caller-dependent interpretation. |
src/project/fingerprint.rs and compiler caches | Target fingerprints cover sources, selected graph fields and compiler identity, but currently have no edition. Include effective editions in relevant build/cache inputs and version affected formats. |
| CLI, IDE and test tooling | Agree on package discovery, explicit standalone selection, generated code and diagnostics. Migration tooling is additional work, not supplied by storing the field. |
Module loading currently deduplicates by ModuleSource. If one physical file
can be assigned to different package contexts, implementation must either
reject conflicting ownership/editions or include that context in module
identity. Taking whichever edition happened to load first would be incorrect.
The inexpensive part is plumbing a value to the parser. The larger recurring cost is maintaining old rules and proving that their combinations still work. A syntax-only first edition should not need a second backend, runtime or dependency resolver. Rules that spread throughout checking or require adapters at package boundaries are a reason to reconsider the change, not evidence that editions in general are easy.
Evidence required before adoption
An implementation experiment should use a real proposed incompatible rule and prove old-to-new and new-to-old calls, including a diamond graph. It should cover public names that collide with new keywords, shared class/enum values, protocol satisfaction, borrows and cross-package generic specialization. Merely accepting two manifest values does not establish mixed-edition support.
Check omitted/unknown editions, independent workspace members, test harnesses,
embedded std, interfaces, direct compiler/API use and IDE diagnostics. Warm
builds must invalidate after changing only a root or dependency edition; graph
round trips must retain it. Run existing baseline tests unchanged and measure
compile-time/cache effects before claiming negligible cost.
Migration should first make source valid in both editions, then change the manifest and rerun checks/tests. Rust’s migration workflow uses compatibility diagnostics and verified edits; it also documents limits for generated code and configurations. Plenty could begin with diagnostics and a short manual migration guide for a small change. Automated public-API rewrites or claims of complete migration coverage need substantially more evidence.
Before committing, choose the first change, edition naming, baseline/default policy, supported-edition lifetime, and public-name escape policy. With no pressing incompatible change, retaining one language mode is the simpler choice. Scheduling and priority remain solely in the backlog.
Plenty toolchain manager
Status: Not implemented. Proposed under B17. The tool is named
plenty-manager, written in Plenty, and intended for a new, separate repository namedplenty-managerin theplentylangorganization on GitHub. This proposal does not create that repository. Commands, configuration and storage below are proposed interfaces.
Motivation and existing behavior
Installing, upgrading and switching compilers should not require editing shell startup files for every release. A project should be able to commit the exact compiler version used by its developers and CI, while older projects keep their own versions.
Today, release archives are
relocatable and can coexist in versioned directories. The first compiler on
PATH runs. The compiler embeds ordinary standard-library sources and runtime
code; Linux archives also carry the native engines in bin/. These properties make a
manager possible without changing how compilation works.
The package manifest’s
plenty-version is a
minimum supported compiler version, not a selection instruction. Preserve that
meaning. Dependency locking, compiler selection and proposed
language editions solve different problems.
Rustup’s toolchain selection is the precedent: executable proxies select an installation before Cargo or the compiler runs. Adopt the small useful subset, not its complete component, channel or override system.
Ownership and implementation
Implement the manager and its executable proxies in Plenty in the separate
plentylang/plenty-manager repository. Keep compiler release production and
archive compatibility in the compiler repository. Agree on a versioned release
metadata contract between them; neither repository should need to import the
other’s implementation.
Distribute native manager executables, so installing Plenty does not first require Plenty, Rust, Cargo or a source checkout. Maintainers build the manager with an explicitly pinned, already published Plenty compiler. Retain documented manual archive installation as the bootstrap and recovery path. A manager release must not require the compiler release it is intended to install.
Use Plenty’s existing process, filesystem, HTTP/TLS, hashing and configuration facilities where suitable. Audit the required APIs and platform behavior before implementation; this proposal is not evidence that secure extraction, file locking or executable replacement already has every needed primitive. Keep manageable application logic in Plenty. Security-sensitive verification may use existing audited libraries through standard-library facilities rather than new cryptographic implementations. Record discovered compiler/library defects in the issues register, not as silent installer workarounds.
First user-facing interface
Illustrative versions below are placeholders, not published release claims.
plenty-manager install X.Y.Z
plenty-manager list
plenty-manager default X.Y.Z
plenty-manager show
plenty-manager run X.Y.Z -- plenty build
plenty-manager remove X.Y.Z
plenty-manager update
install fetches an exact version for the native host and leaves all existing
versions intact. It does not change an existing default. On a fresh installation,
setting the default is an explicit step. list shows installed versions and the
default; show reports the effective version, selection source and executable
path for the current directory, including a missing selected installation.
run explicitly selects an installed version for plenty or plentyc, with
remaining arguments forwarded literally. It does not invoke a shell or search
PATH for the selected compiler. Unknown tool names are rejected.
update explicitly discovers and installs the newest stable compiler supported
on the host, then makes it the user default. It reports the resolved exact
version, preserves previous installations, and never edits project pins. It
is the only initial interface that resolves a moving release name. Interrupted
or failed updates leave the old default working. Manager self-update is not
part of this command or the initial scope.
remove rejects removal of the current default until another default is set.
It cannot discover every project pin: warn about that limitation and report a
clear missing-toolchain error when such a project is later used. Removal must
not damage running tools; platforms that cannot remove an in-use installation
must fail safely or defer cleanup.
Project pin and selection
Use a separate, committed file rather than overloading the library’s minimum compatibility requirement:
# plenty-toolchain.toml — proposed
[toolchain]
version = "X.Y.Z"
The initial schema accepts one exact published MAJOR.MINOR.PATCH version.
Reject ranges, moving channels, paths, unknown keys and malformed files rather
than guessing. Pre-release channels and local development toolchains are
separate extensions, not implicit initial behavior.
A proxy selects, in order:
- An explicit
plenty-manager run VERSION -- TOOL ...selection. PLENTY_TOOLCHAIN, containing an exact version.- The nearest
plenty-toolchain.toml, searching from the caller’s current directory upward to the filesystem root. - The user’s configured default.
Invalid higher-priority selections are errors, not reasons to fall back. With
no selection, explain how to install and select a default. With a selection
that is not installed, show the exact plenty-manager install VERSION command.
Neither proxy invocation nor a project file causes a network request.
Discovery deliberately follows the working directory, not a source filename or
--manifest-path. For a project addressed from elsewhere, enter its directory
or use an explicit selection. Workspace members normally share a root pin;
a nearer member pin overrides it when invoked there. A single compilation
still uses one compiler for the complete dependency graph. Dependency manifests
and pins never switch that compiler during resolution.
A newer compiler remains free to reject a package for other compatibility
reasons. The selected compiler continues to enforce every package’s minimum
plenty-version; the manager does not solve a second dependency graph or
silently select a newer compiler to satisfy it.
Proxy and installation layout
Add one per-user manager bin directory to PATH during explicit setup. It
contains plenty-manager and proxies named plenty and plentyc (with native
executable suffixes where required). Managed releases live outside that
namespace, keyed by exact version and native host triple. The proxy invokes
the absolute compiler path, never itself through PATH.
Choose platform-appropriate user data/configuration directories, with an
explicit PLENTY_MANAGER_HOME override for CI and portable installations.
Keep downloaded staging files distinct from complete installations. Do not
write manager state inside project checkouts, require root, or modify shell
startup files without consent. Keep archive-based installations usable outside
the manager.
Forward arguments, working directory, environment, streams and process results
without reinterpretation. On Unix, prefer process replacement for transparent
signal handling; where a supervising process is necessary, preserve status,
interruption and child lifetime. --version is handled by the selected compiler;
plenty-manager show explains the dispatch. Preserve the explicit selection in
child environments so nested tool invocations cannot accidentally switch back
to a project pin or default.
Install the compiler, embedded runtime and standard library as one release. Optional native engines must match the release and host compatibility contract; do not reuse an arbitrary engine from another installation. Initial delivery may omit engine management, but must say so and retain documented manual engine installation. The manager does not install system linkers, SDKs, libc development files or unsupported cross-compilation targets.
Download trust, persistence and failure behavior
Download only from the configured official distribution endpoint over verified TLS. Release metadata must identify its schema version, exact compiler version, host triple, archive URL, size and cryptographic digest. Resolve moving updates to that immutable identity before installation. Reject unsupported hosts before downloading; available compiler archives do not prove that the Plenty-written manager is qualified on the same platform.
A checksum served beside an archive detects corruption but is not independent authentication. Before shipping automatic installation, settle authenticated release metadata or signed artifacts, trust-root distribution and rotation, and downgrade/replay policy. Do not describe HTTPS plus adjacent checksums as a complete supply-chain security design. Use reviewed verification primitives.
Bound metadata and download sizes. Treat archives as untrusted: reject path traversal, absolute paths, escaping links, duplicate/conflicting entries and unexpected file types; validate the expected layout before publication. Never execute downloaded hooks. Avoid shell construction from versions, filenames or URLs, and do not execute the installed compiler as a substitute for verification.
Use private staging directories, per-installation locking and atomic publication on the destination filesystem. Update defaults atomically only after successful installation. Concurrent installs must agree on one complete result; failed verification, disk exhaustion or interruption must never expose a partial compiler. Retain enough metadata to identify each installed artifact and reject conflicting content under an existing release identity. Define safe recovery for stale locks and staging files without deleting active work.
Dispatch, inspection, removal and execution work offline. Explicit installation and update fail clearly without network access; no command silently substitutes another compiler. A compiler pin improves repeatability but does not pin Git sources, native engines, linker/SDK versions or the system C library. See build reproducibility.
Delivery and acceptance
Keep priority and scheduling solely in B17. Before implementation, agree on the release-metadata trust contract, supported manager hosts, storage layout and optional-engine scope. Creating the separate GitHub repository and publishing binaries are later, explicit operations, not part of adopting this document.
The first delivery should prove exact-version installation, side-by-side use, proxies, project pins, explicit overrides, default changes, explicit updates, removal and recovery. Require integration tests for:
- Two compiler releases used by separate projects without changing
PATH, and manual archive installations still working. - Selection precedence, malformed/missing pins, workspace/member discovery, dependency-pin isolation and calls from outside a project directory.
- Offline invocation with network access prohibited, including nested tools.
- Argument fidelity, spaces/Unicode in paths, supported native argument bytes, exit status, signals and interruption on each admitted host.
- Corrupt downloads, invalid trust metadata, hostile archives, unsupported hosts, disk exhaustion, concurrent installation/removal and interrupted update.
- A downloaded manager installing its build compiler from a clean machine without Plenty or Rust already installed, plus manual recovery without it.
- Optional engine/version isolation if engine management is included.
Document platform qualification, limitations and the compiler version used to build each manager release. Defer nightly/beta channels, target/component matrices, directory-override databases, automatic fetching on dispatch, self-update, arbitrary mirrors and automatic project-pin rewriting. A separate manager keeps these concerns out of the compiler while preserving a useful, small initial tool.
Faster linking
Status: Not implemented. This page records a measured study of what the compiler can do to make linking faster, and the levers it recommends. Linking and the compilation cache own the current contract. Compile and link latency is tracked under B19; no levers here are scheduled.
Summary
Linking costs a near-constant 18–19 ms per build, whatever the program’s size.
That is about 86% of a warm --compile of hello world, 27% of an edit loop on
examples/semantic_search, and 7% of the compile_bench 1000-unit workload.
Plenty’s own object adds only 0–2 ms of that, even at 10,000 sections. The
remaining cost is in how the compiler drives the linker and supplies the
runtime: the cc driver chain, writing the runtime archive on every link,
reading that archive, and the linker’s own startup. Changing the object’s
format does not pay; changing the link procedure does.
Recommended levers
Measured (M) or estimated (E). Times are hyperfine medians pinned to cores 0–7 on x86-64 Fedora with gcc 16.2, mold 2.40.4, LLD 23.1.1, GNU ld 2.46.1 and wild 0.11.0. Differences under about 1 ms are noise.
| # | Lever | Saving | Cost |
|---|---|---|---|
| 1 | Cache the linked executable next to the cached object. On a whole-program cache hit, run or copy the stored executable. | E: unchanged hello rerun 20.5 → about 2.5 ms (−88%); semantic_search 23.5 → about 5 ms (−80%). Only unchanged reruns benefit. | Low. Extends the whole-object cache (issue #317 stage 1). |
| 2 | Run the linker directly instead of through cc. Run cc -### once during the existing linker probe, store the resolved command, then run ld.mold or ld.lld with it. | M: −3.5 ms per link. The cc → collect2 → linker chain adds 3–4.5 ms, including about 1.5 ms loading the LTO plugin. Clang as cc costs 12 ms more than gcc. | Medium. The replayed command must match the driver’s. Fall back to cc when parsing fails or --linker/--link-arg is given. macOS and AArch64 need their own replay. |
| 3 | Extract the 17 MB runtime archive once into the cache directory, keyed by compiler build, instead of writing it to a temporary directory on every link. | M: −2 ms per link; peak memory −16 MB (hello 28 → 11.6 MB). | Low. Keep temporary extraction when the cache is off. |
| 4 | Strip debug sections and embedded LLVM bitcode from the embedded runtime archive at compiler build time. | M: archive 17.4 → 9.9 MB; compiler binary −7.5 MB. No link-time change. | Low. --no-strip output loses runtime debug information unless an unstripped archive is kept for it. |
| 5 | Link run-only builds (plenty FILE, plenty test) against a shared runtime library. Not for --compile. | M: mold link 12.5 → 8.5 ms (hello), 14.3 → 9.3 ms (semantic_search); each run starts 0.3 ms slower. | Medium to high. A second runtime artifact, and run-only executables link differently from shipped ones. Only after 1–3. |
| 6 | Track wild as a future linker. | M: direct link 5.4 / 7.6 / 6.1 ms against mold’s 12.5 / 14.3 / 13.7 (hello / semantic_search / bench1000). | Not ready: its --defsym cannot pull archive members, and it removes far less dead code (hello 393 KB against 116 KB), failing size budgets. |
Levers 2 and 3 were prototyped together (M):
| Program | Warm --compile | Edit loop |
|---|---|---|
| hello | 20.5 → 15.2 ms (−26%) | – |
actor_echo | – | 55 → 47.7 ms (−13%) |
semantic_search | 23.5 → 17.8 ms (−24%) | 69 → 62 ms (−10%) |
| bench1000 | 26.5 → 22.2 ms (−16%) | 263 → 257 ms (−2.5%) |
After 2 and 3, mold still takes about 11 ms for hello: about 5 ms is the cost of linking an empty C program (crt files, libc, libm, libgcc_s), about 4 ms is process startup and its thread pool, and about 6 ms is the runtime archive (419 members, 24,632 sections). Only levers 5 and 6 reduce that further.
Measurements
End to end (M, ms). The edit loop disables the whole-program cache and keeps the function cache warm.
| Program | Emit object | Warm --compile | Edit loop | Cache off | Link share of edit loop |
|---|---|---|---|---|---|
| hello | 2.4 | 21.3 | 22.2 | 28.4 | 86% |
actor_echo | 3.6 | 21.9 | 52.6 | 66.7 | 35% |
semantic_search | 4.7 | 23.9 | 70.8 | 91.1 | 27% |
http_server_bench | 4.6 | 22.7 | 82.4 | 118.9 | 22% |
| bench1000 | 7.9 | 27.3 | 261.7 | 345.1 | 7% |
Link only, by linker (M, ms). The first three go through cc -fuse-ld=X, as
Plenty does today; the rest run the linker directly.
| Program | bfd (cc) | mold (cc) | lld (cc) | bfd | mold | lld | wild |
|---|---|---|---|---|---|---|---|
empty C main | 14.0 | 11.8 | – | 18.5 | 9.0 | 8.4 | 4.3 |
| hello | 58.8 | 15.6 | 16.4 | 50.6 | 12.4 | 13.0 | 5.4 |
semantic_search | 113.3 | 17.9 | 20.1 | 110.3 | 15.4 | 17.3 | 7.6 |
| bench1000 | 66.4 | 16.2 | 17.5 | 59.4 | 14.2 | 15.5 | 6.1 |
On one core, for parallel plenty test runs (M): mold 16.9, lld 16.3, wild 8.9
and bfd 46.9 ms per hello link. Unpinned, mold uses 58 ms of CPU for 12 ms of
wall time, which supports setting PLENTY_LINKER_THREADS=1 for parallel runs.
What the linker receives (M):
| hello.o | semantic_search.o | bench1000.o | |
|---|---|---|---|
| Size | 12.7 KB | 898 KB | 3.9 MB |
| Sections | 75 | 2,042 | 10,052 |
R_X86_64_PLT32 | 4 | 6,558 | 16,005 |
R_X86_64_PC32 | 4 | 3,113 | 9,012 |
R_X86_64_64 | 47 | 984 | 26 |
R_X86_64_GOTPCREL | 0 | 12 | 0 |
The objects have no COMDAT groups, mergeable sections, debug information or
.eh_frame. The runtime archive is 17.4 MB in 419 members: 5.8 MB of debug
information and its relocations, 1.7 MB of embedded bitcode, 3.4 MB of code,
1.0 MB of read-only data and 0.3 MB of .eh_frame. Executables are not
position-independent on this system and carry 139 dynamic relocations.
mold’s phases for hello (M, ms): 11 in total; reading inputs 5; symbol work before copying 5 (merging strings 2–3, dead-code removal 0, relocation scan 0–1); creating the output 1; copying 0.
Levers that do not pay
| Lever | Result (M unless marked) |
|---|---|
Linker flags (--build-id=none, -z norelro, no --eh-frame-hdr) | ±0.5 ms |
| Static linking | 10 ms slower: it pulls in static glibc |
Pre-linking the runtime into one object with ld -r, stripped or not | 0.5–1.5 ms slower: section count, not member count, is the cost |
Removing duplicate -l stub libraries | No change |
| Linker thread counts | 4 ≈ 8 ≈ 16 threads for one link; the default is fine |
| PIE instead of non-PIE | 3,000 more relocations, 0.3 ms; default-PIE distributions are no slower |
| Removing unreachable code in the compiler | At most 1 ms of link time. Still worth doing for compile time: most of bench1000’s 262 ms edit loop compiles functions nothing calls |
| Several objects for parallel linkers | E: none; mold and lld already parallelize within one object |
Common object-format suggestions
These ideas help some toolchains; none applies to Plenty’s output.
- Deduplicating types before writing the object. In C++ the linker merges
repeated COMDAT groups and debug types from many translation units. Plenty
compiles one unit and emits each generic instance once, so its objects have no
COMDAT groups, debug types or duplicate symbols. Identical data sections total
10 KB in
semantic_search. String merging time comes from the runtime’s members. E: under 0.1 ms. - Page-aligned sections. bfd, lld, mold and wild all copy each input section into the output and patch it; none maps input sections directly, so input alignment has no effect. Larger alignment would only pad the output.
- Fewer, larger sections. Prototyped with one text section per object: link
time changed by ±1 ms and objects were 15–17% smaller, but
--gc-sectionscould no longer drop dead code, so bench1000’s executable grew from 341 KB to 2.39 MB. Keep one section per function. The runtime’s per-operation symbols (issue #501) are in the archive and unaffected. - Pre-computed relocations. Runtime calls are already direct (issue #372), executables are not position-independent here, and the relocation scan takes 0–1 ms for 25,000 relocations. Resolving calls at emit time would need one section, losing dead-code removal. E: under 1 ms.
- Dependency hints for parallel linkers. No ELF linker reads such hints and
ELF has no convention for them. lld’s
--symbol-ordering-fileand mold’s--section-ordercontrol output layout, not link speed.
Reproducibility and portability
B17 requires reproducible output.
- Levers 1 and 2 leave output bytes unchanged if the replayed command is
identical; an object-identity test should check it. The cache key must include
the link command and linker identity, which the linker probe already records.
Replaying
cc -###captures distribution differences such as multiarch paths and default PIE instead of hard-coding them. - Levers 3 and 4 do not change stripped executables. Lever 4 removes runtime
debug information from
--no-stripoutput. - Levers 5 and 6 change output bytes. They suit run-only builds, or a separately pinned linker input.
- macOS (ld64) and AArch64 need their own versions of lever 2, with
ccas the fallback.
C interop and dynamic libraries
Status: Partly implemented. Researched 2026-10-05. C imports through trusted interface files, static and shared linking, C exports, and typed resident runtime loading are implemented. The interop reference and linking own the contract; the C libraries lessons teach it.
Shipped differently: the
foreign library,symbol,bind, andptr[const c_char]sketches below are not Plenty syntax. A.plentyifile declaresopaque Nameandextern def f(...) -> T = "symbol". Pointers are&Tand&mut T, and text crosses as&str as utf8or&str as c_string. There are no built-inc_intaliases. Linking is configured on the command line (--link-arg,--linker), not in source.Remaining: unloading, callbacks, C records, and header-assisted bindings (B28); text and byte buffers, views, and broader error types (B27); more targets (B18).
The native-platform assessment expands the target/ABI requirements; a GCC backend would be a separate compiler backend choice, not automatic OS or FFI portability.
Plenty should support ordinary C libraries without making every application
author become an FFI expert. A binding author supplies an explicit, trusted
contract; application code uses typed functions, ownership, borrowing, Result,
and automatic cleanup. The contract belongs in an importable interface module.
Static archives, linked shared libraries, and libraries opened at runtime should
use the same contract representation.
This does not require weakening ordinary Plenty ownership rules. It requires a well-defined place where the compiler trusts external claims. A mistaken contract or a buggy native library can still corrupt memory; annotations cannot prove C implementation behavior. Ease of use comes from reusable bindings and generated adapters, rather than pretending that the boundary is checked end to end.
What existing systems teach us
Rust supports C shared-library linking and producing cdylib artifacts. Loading
a library at runtime is also possible, for example through libloading. The
important limitation is that signatures and ownership promises must be correct;
dynamic libraries are not fundamentally incompatible with Rust safety. We should
avoid attributing ecosystem usage patterns to an inability to use C DLLs.
Rust linkage reference,
libloading Library API.
Cython’s .pxd files demonstrate the value of reusable declaration modules and
compile-time namespacing. They are a useful aesthetic model for Plenty interface
files, though Plenty needs additional ownership and lifetime contracts.
Cython external C declarations.
Zig provides C type primitives, C-compatible records, exports, and header
translation. Its documentation explicitly requires matching target and C compiler
flags during translation: long, packing, enum representations, and other ABI
details depend on them. Reusing a mature C frontend is attractive; copying the
surface syntax alone would not provide this machinery. zig translate-c produces
Zig code, not a ready-made Cranelift interface or ownership specification.
Zig C interop.
ctypes offers a convenient model for loading libraries, typed function lookup,
and conversions. It also documents that invalid non-null pointers can crash the
process. Plenty should retain the convenience while resolving known signatures
ahead of time and checking its side of each contract.
Python ctypes.
CO3 assessment (2026-10-22)
CO3: The Language of FFI demonstrates native-facing declarations lowered to C companion types, conversion adapters, explicit ownership transfer, and finite generic dispatch. Its example switches a consumer between an ordinary dependency and C ABI bindings through build configuration. This supports Plenty’s existing separation of the useful language API from its native calling representation; it is not evidence that arbitrary C libraries share one ownership convention.
The article’s ambition to represent every source type is broader than Plenty’s near-term needs. It explicitly identifies missing captured-closure support, import/header verification, and C header generation. Treat these as design ideas, not demonstrated solutions. The CO3 0.8.2 API documents generated companions, conversion checks, and configurable conversion failure handling; this assessment does not audit their implementation or establish cross-version ABI compatibility. Plenty already generates headers and embedded contracts for its supported subset.
The useful extension is to make conversion policy as explicit as ownership:
| Boundary operation | Proposed Plenty policy |
|---|---|
| Borrow | Keep the owner alive; specify extent, access and non-retention. No implicit copy. |
| Copy into receiver storage | Opt into a conversion that leaves the original owned value with its sender; expose allocation/validation failure and release partially converted values. |
| Transfer | Move the owner with its exact originating release operation; describe consumption on both success and failure. |
CO3 defaults owned values such as vectors to borrowing at the boundary and cloning in the receiver. Plenty should borrow the distinction, not that implicit default: hidden allocation and copying would obscure fallibility and cost. B27 buffer adapters should distinguish a borrowed byte view from an independently owned copy without exporting Plenty collection layouts. Copying may solve an allocator mismatch, but cannot manufacture validity for a dangling source or justify reading storage while foreign code mutates it.
Header-assisted work in B28 should also offer conformance checking, not only binding generation: compare the declared native signature and record layout with the library’s authoritative header under the exact target, defines, include paths, packing and compiler options. Expose the lowered C declaration and conversions for binding review. A mismatch must identify the native argument/layout at fault; a match establishes ABI agreement, not ownership, retention or error semantics. Keep this an optional tool using a mature C frontend, outside ordinary builds.
For future callbacks, reuse ordinary import/export conversion and failure rules
for arguments and results in both directions. A generated C entry thunk is
distinct from an ordinary Plenty function pointer or a captured closure. Begin
with the synchronous context-pointer model below; specify context ownership,
cleanup and failure behavior before adding captured or retained callbacks. Do not
adopt CO3’s default conversion-failure panic as a recoverable Plenty Result:
representation errors need an explicit status policy where supported, while traps
still terminate. Checks of tags, lengths or encoding cannot establish pointer
liveness, provenance or the absence of conflicting foreign aliases.
Separate source visibility, binary linkage, and loading
These are three independent decisions:
| Concern | Proposed meaning |
|---|---|
pub | Another Plenty module may name the declaration. |
| C export | Emit an explicitly named, C-compatible externally visible symbol. |
| Static dependency | Link an archive into the application or library. |
| Shared dependency | Link against a shared library; the platform loader loads it. |
| Runtime loading | Open a selected library at runtime and resolve a known interface. |
Making a function pub must not automatically freeze its native ABI. Ordinary
Plenty functions can keep compiler-private conventions and symbol mangling.
Export declarations generate C ABI entry adapters and, eventually, C headers.
Likewise, source imports should not have incidental side effects that load DLLs.
Runtime loading is an explicit operation returning Result.
The accepted generated export contract design adds
versioned Plenty metadata alongside the ordinary C ABI. Adapters, metadata, and
C headers with precise ownership, borrowing, cleanup, and error comments come
from one checked export representation. The initial export subset
implements numeric and Result APIs, owned and borrowed handles, and typed
loading.
Interface files use the .plentyi suffix and are imported through the same
absolute module namespace as Plenty source. The compiler distinguishes them by
suffix; it does not decide how to interpret a file from its contents.
Two layers of declarations
Keep the actual C signature separate from the ergonomic Plenty signature. A C
function returning an integer and writing through an output pointer should not
be falsely declared as returning a native Plenty Result. The compiler generates
the conversion between these layers.
Illustrative interface sketch, deliberately not executable:
# graphics.plentyi — proposed concepts, grammar undecided
foreign library graphics:
abi = "C"
opaque Image
symbol image_load(path: ptr[const c_char], out: ptr[ptr[Image]]) -> c_int
symbol image_destroy(image: ptr[Image]) -> ()
# A declarative adapter, not the actual C function signature:
pub bind load(path: str) -> Result[owned Image, ImageError]:
call image_load(c_string(path), out image)
success when return == 0
on success own image with image_destroy
on failure image is null
An initial implementation should support a smaller contract grammar and handwritten
adapter bodies before attempting this full declarative convenience. ptr, const,
owned, symbol, and bind here name concepts, not agreed Plenty keywords.
A pointer-to-immutable-C-data is not automatically a Plenty immutable reference:
some other C alias may mutate it. The binding must establish that the stronger
Plenty borrowing guarantee holds, or copy the data into a Plenty value. const
alone neither proves this guarantee nor says how long memory remains valid.
What an interface contract must express
| Topic | Information the compiler or adapter needs |
|---|---|
| Native ABI | Target, calling convention, symbol name, scalar widths, argument/result lowering. |
| Buffer extent | Pointer plus length/capacity, units, alignment, and initialized extent. |
| Nullability | Whether null is permitted and whether it means absence, empty buffer, or failure. |
| Access | Read/write permissions, exclusivity where required, and whether native code retains an alias. |
| Lifetime | Call-only access, borrowing from a named owner, or independent owned result. |
| Transfer | Who owns an input or output on both success and failure paths. |
| Release | Exact destructor/deallocator, library instance, allocator context, size/alignment when needed. |
| Failure | Status mapping, immediate errno capture where specified, initialized outputs on failure. |
| Callbacks | Signature, borrowed/retained context, registration lifetime, reentrancy, thread of invocation. |
| Threads | Thread affinity, concurrent-call permissions, movable/shareable handles. |
| Control flow | No unwinding or nonlocal jump through Plenty frames. |
Useful default: pointer access lasts only for the call and native code does not retain the pointer. This must be an explicit binding-author promise, not an inference from an unannotated C header. Contracts absent from the declaration leave a raw operation requiring an explicit trusted/unsafe adapter.
Ownership transfer on failure matters. If a C call consumes a handle even when it
returns an error, the source value remains moved on both paths. If a call accepts
ownership only on success, an adapter must preserve or return the original owner
on failure. A single owned marker without conditional semantics is insufficient.
Borrowed results initially require a named owner and simple lifetime relation, such as “valid while this image remains borrowed.” Reject relationships the checker cannot express rather than silently extending the lifetime. This is a reason to design FFI contracts alongside public borrowing, not after freezing it.
Representation and allocation
Normal Plenty class, enum, str, list, dict, set, Option, and Result
representations remain private to the compiler/runtime. They should never become
C-compatible merely because a function is exported. Current managed objects have
ownership headers and runtime metadata; C code does not know those conventions.
Provide a separate opt-in foreign record representation later, with target C layout, fixed fields, alignment rules, and compile-time layout checks. Initially support scalar values and pointers to opaque objects, then plain records passed by pointer. Records passed by value, unions, bitfields, packed layouts, variadic calls, and platform-specific calling conventions need deliberate ABI work.
Foreign type aliases such as c_int, c_long, c_size_t, and c_char belong in
an explicit FFI module. They describe target C types; they do not reintroduce
ambiguous native Plenty int. C enum values and boolean representations also need
declared conversion rules rather than unchecked reinterpretation.
The single native str stays UTF-8 with a length and embedded-NUL support. A C
string adapter validates or rejects embedded NUL, creates a terminated buffer
when required, and reports allocation failure. Returned bytes need explicit
encoding, length, and ownership. No automatic assumption that arbitrary C bytes
are valid UTF-8. A byte-buffer/view API should precede useful buffer-oriented FFI.
Never free a foreign allocation using Plenty’s global allocator by default. Owned foreign handles carry their exact release operation and any required allocator context. Exported APIs provide matching release functions for their own outputs. Crossing DLL boundaries on systems with different runtime heaps makes this particularly important. Custom allocators and their lifetimes should use this same provenance model, rather than a separate incompatible mechanism.
A fallible adapter must release partially acquired foreign resources if subsequent conversion fails. Conversely, an allocation error result cannot reliably rescue a native library that itself aborts on allocation failure. Bindings must document which failures can actually be returned. Native error channels cannot be guessed from names or inferred solely from C signatures.
Dynamic library lifetime and callbacks
The easiest first runtime-loading policy is to keep successfully loaded libraries
resident until process exit. That avoids dangling function pointers, borrowed
static data, callbacks, destructors, and native background threads after unload.
Libraries can still be loaded lazily and report missing symbols as Result errors.
Unload support can follow once all these lifetime relationships are representable.
If unloading is later supported, the library owner must outlive every function handle, owned foreign value, destructor thunk, callback registration, and borrowed piece of static data. Merely finishing the last direct function call is not enough. Even a registration destructor may need to wait for in-flight callbacks before releasing its context. Loader APIs themselves can run native initialization code; opening a library is not a passive metadata read. libloading loading and symbol contracts.
Start callbacks with synchronous, same-thread calls and an explicit context pointer. Reentrant callbacks must obey active borrows; they cannot acquire another mutable borrow of the object already lent to C. Retained or foreign-thread callbacks require a later registration/lifetime model and thread-safe runtime support. Do not mark handles transferable between threads just because they are represented by pointers.
Foreign exceptions, longjmp, and Rust unwinding through Plenty frames are outside
the initial contract. C++ adapters catch exceptions on their side and return a
status. Plenty currently terminates on runtime traps and its Rust runtime uses
abort-on-panic; library exports initially share this limitation. A status-returning
wrapper does not turn those aborts into recoverable errors.
Rust FFI and unwinding.
Compiler and runtime implications
Repository evidence at proposal time:
src/codegen.rsemits native object files, enables position independence, usesCallConv::Tailfor Plenty functions, and explicitly usesSystemVfor runtime calls. Foreign calls need target-derived C conventions; hard-coded System V is not a portable DLL implementation.build.rsembeds a host-target runtime archive. Shared-library output needs a reviewed runtime link mode, exported-symbol policy, and startup path without the executable’smain/plenty_maindependency.plenty-runtime/README.mddocuments atomic counts, thread-local destruction, and compiler-private 64-bit slots. Native threads cannot safely share current Plenty objects merely because an FFI function spawns them.
Cranelift supplies native calling conventions and indirect calls. Its Tail
convention is not a stable external ABI. Generate C entry/exit adapters around
Plenty functions and preserve the internal convention. C aggregate classification
and target layout remain frontend responsibilities; an IR function signature alone
does not translate arbitrary C headers.
Cranelift CallConv.
Do not require a JIT or libffi for runtime loading of statically known signatures.
Resolve addresses once, then emit typed indirect calls in the AOT program. Arbitrary
runtime-discovered signatures are outside scope. Keep header translation as an
optional, cacheable binding-generation tool so ordinary Plenty builds remain fast
and do not depend on a bundled C frontend.
Suggested delivery sequence
- Reserve distinct internal, C import, and C export ABI concepts now. Keep native object layouts private and module visibility separate from binary exports.
- Add target-correct scalar/pointer C imports, static/shared link configuration, explicit raw adapters, opaque owned handles, and call-scoped buffer contracts.
- Add C exports and static/shared-library outputs with generated headers, matching destroy functions, versioned symbol naming, and C integration tests.
- Add explicit runtime loading of known interfaces, initially without unloading.
- Add record layout, retained callbacks, header-assisted binding generation, and more expressive borrowed results as independent extensions.
These stages record the original progression; the status and reference links above identify what has shipped. Apply the CO3 lessons within the existing B27 buffer/view work and B28 tooling/callback work, without making universal type lowering or generic dispatch prerequisites. The export qualification strategy defines the evidence required for each additional boundary type.
Every stage needs native integration tests built against a small C fixture: successful/error transfers, partial initialization, matching destruction, embedded-NUL rejection, library lifetime, and reentrant callbacks. Validate each supported ABI with actual C callers/callees; metadata parsing tests alone are insufficient.
Interface naming and format, trusted-boundary syntax, and shared-library exports are settled. Further host ABIs (B18) and borrowed buffer types (B27) remain open. None requires adopting Python object layouts, a garbage collector, Rust trait lookup, or a public Plenty native ABI.
Using Rust crates in Plenty
Status: Not implemented; comparative prototype research (2026-10-23). B17 owns dependency tooling; backend alternatives relate to B20. Current packages, linking and runtime packaging are unchanged. The prototypes are experiments, not a new supported compiler mode.
Recommendation
The most tractable way to let Plenty projects use Rust crates is generated
C-ABI adapters over a Cargo-built Rust dependency graph, with Rust-owned opaque
objects and checked Plenty-facing bindings. Users select Rust APIs, not C APIs;
upstream crate authors need neither rewrite their implementation nor provide C
exports. Cargo and rustc still consume ordinary rlib build artifacts.
This broadens the original investigation beyond avoiding C ABI. Executed prototypes show that Rust ABI does not eliminate the ownership/marshalling layer for Plenty’s different value representations, and it adds compiler-version and lowering obligations. A Rust-source backend removes that boundary for supported types, but preserving the whole Plenty language/runtime is a much larger job. The recommendation concerns implementation cost and maintenance, not a claimed universal performance winner.
Keep native Rust ABI as an opt-in research route. The separate Rust-source backend proposal explores Rust-consumable libraries and native interoperability if this becomes a central goal. Evidence and measurements describe all three; the project/value contract sets the intended user-facing boundary.
What the three prototypes establish
All paths execute one source workload containing scalar calls, Unicode/NUL text,
a generic identity operation, a Rust regex owner with query and destruction,
and a local Rust Future polled from Pending to Ready. Source is checked by the
existing Plenty frontend against generated binding signatures, then a small
restricted parser lowers its actual operations. Both native routes emit a real
Cranelift object. The third emits actual Rust source, not a substituted result.
| Route | Demonstrated | Remaining production work |
|---|---|---|
| Rust-ABI adapters | Real Cranelift calls to pinned scalar/pointer Rust-ABI functions, debug/release adapter builds; separate rustc-private ABI queries work. | Connect matched compiler ABI information to lowering, pin compiler/helper identities, validate every admitted type and upgrade. |
| Generated C adapters | Same selected Rust APIs and ownership lifecycle, generated adapters and caller declarations, stable Rust compiler interfaces. | Integrate generated modules/contracts with Plenty project tooling, error/allocation policy and runtime linking. |
| Plenty-to-Rust source | Checked restricted Plenty calls and arithmetic become Rust calls, native Rust owners and borrows; no opaque-handle adapter boxes. | Backend for checked IR, complete Plenty representations/semantics, std/runtime parity, diagnostics and compile-time qualification. |
A separate pinned nightly helper queried FnAbi: i64 was direct, &str a
pair, and a 24-byte repr(C) record passed indirectly by pointer under Rust ABI
but on the stack under C ABI. repr(C) fixes layout, not the function calling
convention. This makes compiler-assisted Rust ABI handling plausible, while
showing why the successful scalar calls must not be generalized. The helper
observes a different compiler build from the stable executable tests and is not
connected to their Cranelift lowering.
rlib consumption and final linking
An rlib contains compiler-specific metadata and code, potentially LLVM bitcode. It is not an API schema or a bundle of all upstream Rust dependencies. Generic instances may be generated downstream; rustc must participate. Metadata from another compiler version is rejected. Libraries and metadata, monomorphization.
Rust’s supported mixed-language linking routes are a rustc-owned final link or
one aggregate Rust staticlib linked by a foreign driver. Passing arbitrary
rlibs directly to a foreign linker remains unsupported. Artifact format and
calling convention are independent: C-ABI functions can live in rlibs.
Both adapter prototypes use rustc to link their rlibs and the Cranelift object.
Linkage.
For production, prefer a Cargo-managed final Rust target that includes the generated adapters, dependency graph, rebuilt Plenty runtime and application object. Alternatively produce one aggregate staticlib for today’s linker. Neither route means converting upstream crates to C libraries individually.
Recommended boundary
Plenty manifest + selected Rust APIs / small Rust companion module
|
generated bindings
/ \
Plenty checked foreign owners Rust C-ABI adapters
| ordinary Rust calls
Cranelift object Cargo dependency rlibs
\ /
one runtime graph, rustc final link
Generate exact-width scalar/pointer/length interfaces and explicit output/error contracts. Keep complex Rust values opaque; Rust constructs, borrows, mutates and drops them. Instantiate selected generic operations inside Rust. Plenty enforces owner moves, exclusive/shared loans and drop timing across calls. Binding metadata must specify these rules, not merely the machine signature. Automatic discovery can assist API selection, but Cargo metadata and symbol lists are insufficient to derive every lifetime/effect contract safely.
A companion .rs module is a practical escape hatch for trait-heavy APIs,
callbacks and async executors. It can present a small reusable operation to
Plenty while calling an unchanged crate. Publish/cache adapters independently
of individual applications where their API and crate-version contract allows;
do not make every user write unsafe pointer glue. Unsupported APIs should give
a precise diagnostic and suggest a facade, not silently erase types/errors.
The prototypes’ async operation is a future with explicit polling, not async Plenty syntax or a reactor integration. C ABI does not prevent async use: an opaque operation can expose start/poll/cancel with owned state. Real executors, wakers, callbacks, retained buffers and cancellation completion need explicit lifecycle contracts before general async crate support.
Runtime and compiler integration
Today src/codegen.rs::link_object extracts an embedded staticlib and
src/toolchain.rs::link invokes a cc driver. build.rs::build_runtime builds
Rust std, vendored dependencies and an abort-on-panic runtime into that archive.
Appending another independent Rust staticlib risks conflicting allocator,
panic/std and native library implementations.
For Rust-enabled projects, ship the matching Plenty runtime source/build recipe and rebuild it in the same Rust graph. Reuse the runtime variant without C main and a Rust launcher, preserving Plenty application initialization and exit status; or produce one aggregate staticlib with the existing startup contract. Keep today’s embedded-runtime mode for pure Plenty projects. None of the three experiments integrates the real Plenty runtime into its final executable.
Project resolution and caching belong in src/project.rs; imports receive a
resolved graph rather than fetching during type checking. Add foreign-owner
types and loan/drop operations before backend lowering. Map generated Rust
diagnostics back to binding requests. Stable C ABI removes dependence on
compiler-private lowering, not the need for compatible Rust artifacts
within each Cargo build.
The Rust-ABI alternative would obtain a monomorphic adapter’s FnAbi and layout
from a matched helper and translate every accepted pass mode and attribute.
Reject unsupported cases. Compiler APIs are unstable; rustup’s rustc-dev and
llvm-tools enable investigation but do not make a portable ABI contract.
ABI rules,
FnAbi,
external drivers.
Other boundaries and next evidence
Wasm components could supply isolation and a defined value-transfer interface, but need a Wasm runtime, supported crate targets, bindings and copy/lifetime policies. A subprocess/RPC boundary offers strong isolation and independent toolchains at the cost of serialization, process lifecycle and latency. These are useful for coarse operations, not substitutes for cheap in-process calls. Neither was prototyped here.
The next adoption gate is one real Plenty executable using a generated adapter, recoverable constructor errors, deterministic cleanup and a unified runtime graph. Then qualify a native build-script dependency, realistic async facade, offline rebuilds and version updates. Source-backend expansion and full native Rust ABI lowering remain separate decisions. The current evidence supports generated C adapters as the smallest route to useful crate reuse; it does not justify promising that every crate API will be directly importable.
Proposed Rust dependency contract
Companion to Rust crate dependencies. All syntax and behavior below are proposed, not implemented.
Project declaration
Distinguish Cargo crates from Plenty packages. One possible spelling is:
[rust-dependencies]
matcher = { package = "regex", version = "1", default-features = false, features = ["std"] }
[rust]
toolchain = "<project-selected supported Rust toolchain>"
bindings = "rust-bindings.toml"
This names a candidate, not a qualified crate. Registry versions, local paths and pinned Git sources should reuse Cargo semantics. Generate a deterministic private workspace with explicit manifest, target, profile and toolchain. Cargo owns feature resolution and transitive dependencies, including host proc-macros/build scripts. Use versioned metadata and build JSON for actual artifact paths, not guessed hash-suffixed filenames. Cargo external tools.
The recommended generated C-adapter route uses stable Rust interfaces. Pin the project toolchain for reproducibility; an exact compiler-private helper match is required only for the experimental Rust-ABI route. A Rust companion module can simplify unsupported APIs while the generator supplies the ABI glue.
Commit a Cargo-format companion lockfile, for example plenty.rust.lock, and
record its digest and toolchain identity in the Plenty lock/build contract.
--locked forbids changing either lock. --offline applies to both ecosystems
and fails on missing sources/components. Imports never silently install a
toolchain. Cargo builds execute dependency build scripts and proc-macros; this
is not a data-only import.
Cache keys include compiler/runtime identity, adapter/schema versions, full rustc identity, target/CPU, Cargo lock and resolved features, profile/panic/LTO flags, relevant environment/configuration and source/build inputs. Honor Cargo invalidation; a lockfile is not a fingerprint of mutable path sources. Rebuild after toolchain changes. Arbitrary prebuilt rlibs without a verifiable compatible graph are unsupported.
Leave native flags, search paths and build-script ordering with Cargo through
the final Rust target. Its links constraint prevents competing packages from
claiming the same native library. Account for the runtime’s vendored native
dependencies in that graph, rather than bypassing the checks.
Build scripts.
API selection and value rules
A binding specification selects public Rust paths, concrete type arguments and Plenty-facing signatures. rustc checks generated Rust calls, including traits, visibility and enabled features. Cargo metadata and symbol lists alone cannot describe callable APIs. Safe Rust signatures do not establish the validity of raw-pointer adapters: the generated boundary must enforce its own preconditions.
A first useful surface could expose a constructor, borrowed query and destructor for an opaque Rust matcher/parser, imported as a generated Plenty module under the dependency alias. A small Rust facade can simplify unsupported APIs without translating the crate. Automatic discovery can later prefill binding selections.
| Rust shape | Initial proposed treatment |
|---|---|
| Integers/floats/bool | Exact-width mapping, checked conversions and bool validity; target-check platform-sized integers. |
&str, &[u8] inputs | Scoped validated pointer/length views when storage permits, otherwise explicit copies; no retention after return. |
Owned records, String, Vec<T> | Opaque move-only owners, constructed/dropped in Rust; explicit copy-out where needed. |
Option<T>, Result<T,E> | Destructure in Rust into specified tags and initialized outputs; preserve typed/opaque errors, never reinterpret enum layout. |
| Generic calls | Selected concrete Rust instantiations; no automatic Rust trait implementation for arbitrary Plenty classes. |
| Shared/mutable methods | Shared/exclusive loans on the owner; forbid aliases and use after consuming calls. |
| Borrowed returns | Initially reject or explicitly copy; general Rust lifetime relationships exceed current Plenty returned-borrow rules. |
| Closures, trait objects, async, retained callbacks | No automatic general import; allow a reviewed Rust facade with a defined environment/lifetime/executor contract. Explicit future polling works in the prototype, but no production executor contract exists. Macros run in Rust, not as Plenty syntax. |
| Unsafe APIs/raw pointers | Separately audited safe facade, not automatic safe exposure. |
Boxing an opaque owner is an explicit fallible bridge allocation. Rust crates
may allocate internally and abort on OOM: they do not automatically gain
Plenty’s recoverable AllocError guarantee. Document copies and allocation
effects. Deallocation stays with the originating runtime even if both use the
system allocator. Drop exactly once on normal/error paths; invalidate consumed
handles and specify output initialization separately for success and failure.
Rust values are not automatically Plenty-copyable or worker-safe. Start with
thread-bound owners; later worker exposure needs compiler-checked Send/Sync
and compatible Plenty rules. Pinned/self-referential objects need an explicit
storage contract. Rust checks the adapter’s local borrows; Plenty must check
lifetimes and aliasing across successive calls.
Failure and qualification
Use abort-on-panic initially, matching today’s runtime. Panic terminates the
process rather than becoming Result; catch_unwind cannot catch an abort.
No Rust unwind may escape through Plenty frames. A later catch-at-adapter policy
needs compatible unwind builds and proof for every call and destructor.
Qualification needs compile-fail tests for unsupported APIs, aliasing and toolchain mismatches; drop counters across moves/errors; output-initialization checks; embedded-NUL/non-ASCII text; narrow integers and return lowering; panic in a subprocess; generics; a native dependency; and Rust owners alongside Plenty allocations. Check duplicate runtime/native symbols and cold/offline builds. The prototype evidence covers a subset: real Cranelift callers, generated adapters, a restricted Rust-source path, move rejection, cleanup, UTF-8/NUL input, generic calls, Pending/Ready polling and arithmetic traps. Production project/runtime integration and the broader matrix remain unimplemented.
Three Rust crate prototype results
Companion to Using Rust crates in Plenty.
Reproducible code, commands and checked result snapshots are in the repository’s
experiments/rust-crates/ directory; start with its README.md.
These are bounded experiments against Plenty d7332b6c, not shipped features.
One workload, three paths
The original Plenty source executes scalar arithmetic, Unicode/NUL text length,
generic identity, construction/query/drop of a Rust regex::Regex owner,
and a local pinned Future polled twice. regex 1.13.1 is an actual external
crate resolved by the experiment’s Cargo.lock. The future is a synthetic fixture
that returns Pending once, then Ready; there is no network I/O or executor.
The existing Plenty parser/type checker checks the source against generated
signature-only stubs. Those stubs model move-only classes and loans, never run,
and do not automatically prove the selected Rust API matches the declaration.
A second restricted parser converts the same source into an operation plan.
It supports one run(i64) -> i64 function, simple/mutable local bindings from
selected function calls, name/integer/text arguments, borrow markers and a final
sum of names. It rejects branches, loops, arbitrary expressions and unknown APIs.
The Rust-ABI and C-ABI paths emit real Cranelift machine code from that plan, calling generated Rust adapters. The generator changes the ABI keyword; both otherwise use the same scalar/pointer interfaces and Rust-owned handles. Both link ordinary rlibs through rustc. Cranelift’s scalar lowering is experimentally matched to stable rustc 1.99.0 on x86_64 Linux, not derived from compiler metadata.
The source backend emits Rust statements and checked arithmetic from the plan, calling the shared Rust fixture directly. Rust owns native values and borrows; there are no handle-box adapters on that path. This is a real source-lowering experiment, but it does not consume Plenty’s internal checked IR, compile arbitrary Plenty, reproduce its string/allocation model, or execute its runtime. Passing the public checker before a second parser is a prototype compromise.
Correctness and failure observations
All three paths passed with adapter/generated-source optimization levels 0 and
3. For inputs -100, -1, 0, 1, 41 and 1000, the independently specified answer is
3 * input + 11; live Rust owner counters return to zero. The workload includes
four Unicode scalars in five UTF-8 bytes (é, NUL, a, b) and two regex matches.
The Future’s first/second poll yield 0/1; finish requires Ready and a second poll
after Ready is forbidden. Unit tests separately assert these states and dropping
an unfinished Future. This does not qualify foreign callback/cancellation races.
An inserted use-after-move is rejected by the real Plenty checker. Large inputs trap on integer-sum overflow in all executable paths: Cranelift exits by SIGILL, generated Rust by SIGABRT. The harness detects an incorrectly successful wrap followed by its fallback panic, rather than counting any abort as a pass. This preserves the limited no-wrap contract, not identical Plenty trap diagnostics.
Rust unit tests, Python grammar rejection/literal tests, formatting and Clippy passed. No compiler/runtime implementation was changed, and no full Plenty regression suite was run. Native dependency build scripts, real async I/O, fallible constructor errors, OOM recovery, aggregate ABI calls and cross-target compatibility remain untested.
Compiler-derived ABI observation
After installing rustc-dev and llvm-tools, the separate helper built and ran
on nightly-2026-10-01 (21b707e3f, Rust 1.101.0-nightly). It invokes
fn_abi_of_instance after analysis:
| Function input | Observed pass mode |
|---|---|
Rust ABI i64 | Direct, size/alignment 8 |
Rust ABI &str | Pair, size 16/alignment 8 |
Rust ABI 24-byte repr(C) record | Indirect, Pointer |
| C ABI the same record | Indirect, OnStack |
Both record returns are indirect. This is concrete evidence that the query is accessible and layout alone does not determine calling convention. The helper’s results are not fed into the stable Cranelift caller: compiler identities differ, and complete pass-mode translation, attributes, monomorphic API discovery and compatibility checking remain work. Missing initial components were an environment issue, not evidence the approach was infeasible. FnAbi, PassMode.
Cost observations and interpretation
See the committed measurement snapshot at experiments/rust-crates/RESULTS.md
for exact values and all samples. The initial isolated dependency/tool build took
85 seconds; subsequent fixture builds were warm. Adapter-plus-link builds were
roughly a tenth of a second. Cranelift object emission took a few milliseconds.
These are different stages, not cold-build comparisons of production backends.
Each timing has three runs, with 2,000 whole workloads or two million scalar iterations, black-boxing inputs inside the loop and outputs. Source-mode Rust may inline; native calls cross an object boundary. Scalar results measure the whole small loop/call, not a calibrated standalone ABI transition. The shared fixture and regex dependencies remain debug-built even when adapters are optimized. CPU contention/frequency, ordering and debug dependencies make these exploratory measurements, not a backend speed ranking. No benefit of Rust ABI over C ABI was established.
Both native adapter paths borrow existing UTF-8 bytes without copying payloads and add two opaque-owner Box allocations per workload. Source mode avoids those two boxes; all paths retain the future’s internal pin allocation and regex’s own allocations. These are source-level accounting observations, not allocator instrumentation or a general zero-copy guarantee.
Handwritten glue consists of a ten-function API-selection table and the shared Rust facade/fixture. Unsafe adapter bodies, native caller signatures and checked source stubs are generated; no handwritten C wrapper or C header is needed. The Rust-source route calls the same facade directly. Both still rely on a trusted hand-selected type/ownership contract rather than automatic reflection.
What changes the recommendation
All three routes can reach the same useful Rust operations. Rust ABI did not remove conversion or ownership glue, while the helper revealed additional ABI machinery that must be maintained. C adapters can expose ordinary crates and future handles using stable compiler interfaces. Rust-source lowering is concise for this grammar and avoids handle boxes, but a complete backend must preserve Plenty’s fallible allocation, strings, sums, borrowing, cleanup, std and diagnostics. Therefore generated C adapters are the most tractable next production experiment; the others remain viable research directions with larger scope.
Compiling Plenty to Rust source
Status: Restricted checked-IR emitter spike; no production backend (2026-10-24). B20 tracks optional backends; B17 tracks Rust dependency tooling. The current compiler still emits native code through Cranelift. No production command, target, Rust import syntax or compatibility guarantee is introduced here.
Recommendation
The checked-emitter experiment now generates a restricted scalar library from Plenty’s actual checked operations, including a Rust → Plenty → Rust companion round-trip. It measures generated Rust build costs separately from the earlier hand-lowered ownership probes. References, records, enums and dynamic owners remain outside that emitter.
The evidence supports continued bounded exploration of Rust-hosted Plenty libraries. Emitting a Rust crate could let Rust call Plenty functions, and Plenty call selected Rust APIs, with ordinary Rust owners, borrows, enums and callbacks. rustc would own type layout, calling conventions, generic codegen and final linking within that build. Users could write application logic in Plenty while retaining a Rust application’s ecosystem and tooling.
The opportunity is real; “seamless” needs a precise boundary. Producing valid
Rust is easier than preserving Plenty’s semantics and exposing a natural
Rust API. Replacing runtime values with Vec, String and derived Drop
everywhere would change allocation, copying and cleanup behavior. Keeping all
current runtime representations would preserve more behavior but expose a
Plenty runtime API to Rust users. Prefer a small shared native-type subset,
with explicit wrappers for richer Plenty values.
The Rust crate investigation already includes an actual restricted Plenty-to-Rust emitter alongside two native adapter routes. This proposal develops the source-backend direction as its own architectural choice. It does not overturn that investigation’s lower-cost recommendation of generated C adapters for adding Rust dependencies to today’s compiler.
Proposed compilation boundary
Plenty source ── parse / resolve / type and ownership checks
|
typed source structure + places, loans,
cleanup decisions and source locations
|
generated Rust crate
/ \
Plenty support crate selected Rust APIs
\ /
Cargo + rustc
|
Rust host library or executable
Keep Plenty’s own checker authoritative. Do not translate unchecked source and use Rust errors as Plenty’s type system: their admitted borrows and contracts differ. The independent Rust check is valuable defense and catches emitter mistakes, but a supported Plenty program rejected by emitted Rust is a backend problem, not a reason to silently redefine the language.
Start from the existing structured operations
and their construction sites, not Cranelift IR or assembly. src/op.rs already
retains typed slots, structured branches/loops, Loan, Access, Site, Try
cleanup and DropLocal operations. It is not devoid of ownership information.
However, some source constructs are specialized/desugared and runtime operations
already assume Plenty layouts. No public checked-IR emission API exists today.
The next technical question is whether these facts suffice to reconstruct a small safe Rust body, or whether a thin typed source representation must be retained before representation-specific lowering. Prefer preserving the missing facts at that boundary to reverse-engineering lifetimes from machine slots. Do not commit to a new universal IR before the small slice answers that question.
User-facing direction
The first useful product would be a Rust-consumable library containing selected
Plenty functions. A generated crate would have stable exported Rust names and
an explicit API manifest; internal monomorphized names remain private. Rust
can supply native data, call these functions and inspect native Results.
Selected Rust imports would use checked signatures or a small Rust companion
module whose ownership/effects are declared. A raw symbol list is insufficient.
Build integration can use an explicit Plenty generation step followed by
Cargo, or a build script writing generated modules to OUT_DIR with complete
input invalidation. Cargo already supports this kind of
source generation.
Separate generation from dependency resolution so checking does not unexpectedly
fetch or execute build scripts. Pin the supported Rust toolchain/edition and
lock dependencies; build one coherent runtime/dependency graph.
This is source interoperability inside a coordinated Rust build, not a stable
Rust binary ABI or a promise to mix arbitrary rlibs. The
Rust ABI has no stability guarantee.
Native Cranelift Plenty objects cannot directly treat emitted Rust layouts as
their existing layouts. Mixed-backend calls still need a defined boundary.
Why pursue it, and what it does not buy automatically
| Potential benefit | Qualification |
|---|---|
| Rust calls Plenty as a normal crate | Public types and lifetime contracts must be designed, not just printed. |
| Plenty calls Rust without C marshalling | Only admitted types/APIs share that representation; ownership and effects still need checking. |
| LLVM optimization and Rust targets through rustc | No speedup or new Plenty target is established until runtime support, ABI boundaries and generated code are tested. |
| Cargo packaging, profiling and debugging | Generated-source locations need mapping back to Plenty; dependency build cost remains. |
| Access to trait-heavy or async libraries | A companion Rust facade can help now; arbitrary traits, macros and futures do not become Plenty features merely by changing backends. |
The fast Cranelift path remains valuable. A Rust-source route is a stronger candidate for ecosystem integration or an optional optimizing build than a demonstrated replacement for interactive compilation. Two frontends plus Rust type checking and optimization add work; incremental Cargo builds may amortize some of it. Measure cold, unchanged and edited builds separately, including generation, dependency build, Rust compilation, link time and peak memory.
The semantic contract describes the hard boundaries. Probe evidence and adoption criteria explain what has run and what would justify a first implementation. Future work and priority remain in the backlog, not this proposal.
Rust-source semantic and interoperability contract
This is the proposed contract for the Rust-source backend, not implemented behavior. Preserve the language first; make unsupported slices explicit rather than accepting a program with different semantics.
Values and API shape
| Plenty concept | Candidate Rust representation and constraint |
|---|---|
| Sized numeric scalars and bool | Corresponding Rust scalar; emit explicit checked operations/casts where required. |
| Simple scalar-field classes | Named Rust structs, normally move-only even when fields are Copy. Do not derive traits that weaken Plenty’s ownership contract automatically. |
| Named enums, Option, Result | Native enums; explicit conversions for Plenty error widening and pattern tests. Stable public wrapper names for anonymous sums. |
| Shared/exclusive references | Rust references with generated lifetimes from checked place relationships. A returned first-parameter borrow gets that parameter’s lifetime. |
| Borrowed primitive list data | &[T] / &mut [T] at selected Rust-facing boundaries; prove owner/lifetime stability and indexing behavior. |
| Owned collections | Initially preserve a support-crate wrapper; optional Vec conversion must state ownership, allocation failure and whether it copies. |
| Plenty immutable strings | Preserve share/retain behavior in a wrapper and borrow UTF-8 as &str where valid. String is not a transparent replacement. |
| Arbitrary Rust owners | Native Rust values only for explicitly admitted APIs with known loan/drop/effect contracts. Rust generic/trait declarations require more than an ABI signature. |
Do not transmute current runtime values to native Rust enums or collections. Plenty’s sum tags, owner handles, string headers and inline relocation rules are private representations. A source backend may choose different internal layouts, but std/runtime helpers must agree and every crossing must be checked. Initially isolate portable helpers behind a support crate rather than copying the whole runtime or linking two independent runtime/allocator graphs.
Ownership and destruction
Ordinary lexical locals and many ? exits fit Rust RAII. The probes validate
one such sequence, not the full contract. Plenty specifies closure capture
cleanup order and allocation-free bounded-stack recursive heap destruction;
ordinary Rust captures and recursively nested owners do not automatically
provide those guarantees. Rust’s
destructor rules
also make declaration order, temporaries and partial moves consequential.
Generate explicit capture structs/scopes and cleanup helpers when necessary.
Avoid blanket Option<T> slots, hidden clones, Rc<RefCell<T>> or heap boxes
as a convenient way to appease the borrow checker.
Checked place relationships must survive lowering. A concrete example is
Plenty’s simultaneous loans at distinct constant list indices: direct Rust
index borrows fail with E0499. A
split_at_mut
translation works for the tested pair. General lowering must preserve argument
evaluation, bounds failure sites, intervening effects and loan scopes; the
two-index sketch is not a general algorithm. If unavoidable unsafe helpers are
introduced, centralize and audit their place/aliasing preconditions. Do not
emit pervasive raw pointers and call the result safe Rust interoperability.
Rust and Plenty impose different limits on stored/returned references. Continue to enforce Plenty’s current limits even when Rust could represent more. Conversely, a backend must either translate an admitted Plenty construct or diagnose its unsupported backend subset before emitting code.
Arithmetic, errors and allocation
Plain Rust + in release mode is not the implementation of Plenty checked
addition. Emit checked_add plus the Plenty fatal trap path, and analogous
handling for other arithmetic, division, shifts and narrowing casts. Preserve
floating-point comparison and operation order; an optimization backend does not
authorize reassociation. Trap helpers need Plenty source/caller locations and
must not allocate on their failure path.
Native Result<T, E> is a good representation, but Plenty’s automatic error-sum
widening needs generated mappings rather than accidental From implementations
or boxing everything into dyn Error. Public error enums should implement the
appropriate Rust traits and retain concrete causes. Rust panics are not
recoverable Plenty error variants by default.
Fallible construction must reserve before evaluating list elements, propagate
failure and destroy only initialized values. Rust’s
Vec::try_reserve
can implement one part, but Vec::new() does not perform Plenty’s empty-owner
allocation. vec!, implicit cloning and unconstrained collection operations
can introduce infallible allocation. Preserve CapacityOverflow versus
OutOfMemory with explicit layout validation and an appropriate allocation
wrapper; simply exposing TryReserveError is not equivalent. External Rust
crates may allocate infallibly internally, regardless of generated code quality.
Their contract must say so; do not claim arbitrary Rust APIs satisfy Plenty’s
recoverable-allocation policy.
Panics, recursion and suspension
Plenty fatal traps do not
unwind. The probe’s default Rust panic runs a destructor that Plenty’s overflow
trap does not. Emit an aborting trap helper. For the initial foreign-call slice,
require an explicitly agreed
panic=abort build graph
or reject integration; never silently change a Rust host’s panic policy.
Supporting an unwinding host later needs a separate boundary contract, including
callbacks and destructor panics. Wrapping a whole generated function in
catch_unwind
is insufficient: cleanup may already have run before the catch, and aborting
panics are not caught.
Plenty also has required direct recursive tail transfers. An ordinary generated Rust call cannot be assumed to preserve that guarantee. Translate admitted self-tail recursion to a loop, use an explicit dispatch machine for supported mutual cycles, or reject the subset until implemented. Generic specialization must precede that analysis where it does today. Generators need explicit frame, suspension and cleanup lowering; a Rust iterator or async future is not automatically the same lifecycle. Concurrency, generators, boxed closures and runtime-specific intrinsics are outside a first library slice.
Generic and tooling boundaries
The present frontend specializes generics before native lowering. Emitting those instances as private Rust functions is simpler than recovering generic templates, but only exposes selected monomorphic APIs to Rust. Exporting a generic Plenty function as a Rust generic requires retaining the template and translating protocol bounds, not mechanically replacing bracket syntax. Rust trait coherence/orphan rules still apply to generated implementations; newtype facades may be necessary for foreign traits/types. Rust implementation rules.
Keep selected imports explicit first. Cargo metadata describes packages, not complete type/loan semantics. Automatic API discovery, procedural macros, associated types and async executors remain independent integration problems. The compiler should track a generated-byte-range to Plenty-span map, consume rustc JSON diagnostics, and distinguish user API mismatch from emitter failure. Path remapping alone cannot map generated Rust expressions back to Plenty. Runtime trap sites and debugger stepping are separate obligations. Neither source mapping nor debugger support has been implemented by this exploration.
Rust-source probe evidence and adoption criteria
This accompanies the proposal and semantic contract. Results are bounded evidence, not a completed backend or a performance comparison.
The subsequent checked-emitter spike adds actual checked-IR generation and compile/link measurements. The hand-lowered ownership results below remain separate evidence; they are not implemented by that scalar emitter.
What is already generated
The Rust dependency experiment has
an actual restricted emitter in experiments/rust-crates/run.py. It checks
Plenty against declared API stubs, reparses a small accepted subset and emits
calls, borrows and checked arithmetic to Rust source. A real regex owner and
synthetic future execute through that path. That establishes feasibility of
source emission with native Rust owners; it does not consume checked compiler
IR or implement the language. Its earlier results were not rerun for this
proposal and should retain their original measurement qualifications.
New semantic probes
experiments/rust-source-backend/ contains dependency-free Rust candidate
implementations, a separate Rust-host integration test, paired real Plenty
programs and a runner. The Rust is hand-lowered, deliberately isolating
semantic questions from implementing a second parser or a production emitter.
Run python3 run.py --plenty /absolute/path/to/plenty from that directory.
Inputs, expected observations and the recorded snapshot are in its README and
RESULTS.md; machine-readable observations are written under ignored target/.
Tested on x86_64 Linux with Plenty 0.3.0 from main 00d32a13, rustc 1.99.0
(b940084d7, LLVM 23.1.1), edition 2024. The local Rust checkout inspected for
Vec reservation and panic/abort implementation was 080cc27146389dbb5c8772c476b1937be9c6110f,
distinct from the tested compiler revision. It was used as source evidence,
not built or assumed identical to the installed toolchain.
| Question | Observation | Limit |
|---|---|---|
| Can Rust consume native-looking APIs? | Separate Rust test crate passes a native vector borrow, Cursor and consuming closure; returned element points into the original vector. | API declarations are handwritten, not inferred. No general Rust crate importer. |
| Do scope cleanup and propagation align? | Both ordinary/debug and release Plenty/Rust print 33, 2, 1, 3, 4: numeric result, reverse lexical cleanup on ?, explicit moved-owner drop, final marker. | One concrete control-flow path; not closure/generator/recursive cleanup coverage. |
| Does checked overflow preserve fatal behavior? | Explicit checked-add/abort fails in both modes with no destructor output, as Plenty does. | Trap formatting, exact exit status, stack guard and source attribution are not reproduced. |
| Can ordinary Rust panic stand in for a trap? | No: default panic prints destructor trace 99; separate panic=abort build does not. | Does not qualify arbitrary dependencies, callbacks or unwinding Rust hosts. |
| Does native fallible reservation help? | Impossible capacity fails before any element callback; normal construction succeeds. | Capacity overflow only; no allocator fault injection or Plenty owner/error representation parity. |
| Can all admitted loans be printed literally? | No: distinct constant list-index loans pass Plenty but naive Rust fails E0499; safe split translation passes. | Only a two-element constant-index case. |
| Does Rust still check owner misuse? | Moved-vector access is rejected with E0382. | Redundant checking cannot replace Plenty checks or prove the emitter correct. |
Two library tests and one Rust-host integration test pass in debug and release. The runner checks four differential executions, two rejected Rust sources and three panic observations. Cargo formatting and Clippy with warnings denied pass. No production compiler source changed, no external dependencies were added, and no cross-target or performance result is claimed.
Decision gate for a first backend slice
The checked-emitter experiment now satisfies the scalar part of the original gate: real checked-frontend emission, native companion calls, source mapping and build measurements. The remaining ownership-bearing gate is to admit simple move-only scalar records, named enums/Results, structured branches/loops, local borrows and a returned first-parameter borrow. Expose monomorphic exports and a small explicitly declared Rust companion API. Exercise a genuine Rust host that calls an exported Plenty function which in turn calls that companion. Restrict recursion and resource features until their contracts have a translation; unsupported cases must fail explicitly.
The adoption decision should require all of the following evidence:
- Differential tests against Cranelift for admitted semantics, including overflow, early exits, drop order, moves, aliases and fault-injected allocation if owned dynamic storage is admitted.
- A public Rust API usable without unsafe calls or per-call marshalling glue, with documented ownership and panic behavior and no hidden copy workaround.
- Source-located Plenty diagnostics for unsupported inputs, plus at least one generated rustc error mapped back to the triggering Plenty source.
- Cold/warm/edit build and representative runtime measurements, separately attributing source generation, Rust checking/optimization, dependency builds and linking. Compare the same accepted workloads, not tiny warm Cargo no-ops.
- A documented list of retained frontend facts, support-runtime dependencies, unsafe helper invariants if any, and costs of keeping both backends correct.
If useful native APIs require broadly changing Plenty’s semantics or pervasive unchecked lowering, stop expansion and prefer the generated-adapter route. If safe monomorphic library exports work with a small support layer, that alone can be worthwhile without committing to a whole-language compiler replacement. This gate describes the technical decision; scheduling remains in B20.
Checked emitter and Rust build costs
This follows the Rust-source proposal. The experiment
in experiments/rust-source-backend/emitter/ generates Rust from the real checked
frontend. It is deliberately smaller than the ownership-rich hand-lowered
probes: its purpose is to test a real emission boundary and the
cost of compiling the resulting source, not to declare a production backend.
Implemented experiment
Build the developer driver with cargo build --locked --release --features rust-source-spike --example rust_source_spike --bin plenty. The hidden emission
API is absent from normal builds. It loads a normal Plenty module, runs frontend and
independent operation/ownership checking on the complete input, then selects
explicit export roots and their reachable functions. There is no second parser,
alternative Plenty type system, public backend flag, or shared-IR redesign.
| Area | Admitted slice |
|---|---|
| API/storage | Native signed/unsigned integers through 64 bits, bool, unit results |
| Computation | Checked add/sub/mul, floor division/modulo, wrapping add/sub/mul, integer conversions, comparisons |
| Control flow | Structured branches, scalar matches, short circuiting, while, break/continue, returns and direct calls |
| Generics | Concrete scalar specializations already produced by Plenty; no generated Rust generic functions |
| Recursion | Self-tail calls become an explicit loop with evaluated argument temporaries; other recursion is rejected |
| Rust cooperation | Explicit checked stub → Rust companion replacement, called from a native Rust host through an exported Plenty function |
| Diagnostics | Source-located unsupported operations; sidecar line map demonstrated against JSON rustc type errors |
Every accepted value is a Copy scalar. This avoids inventing a storage, allocation or lifetime model. References, records, enums/Results, dynamic owners, floats, generators and concurrency are rejected, not silently converted into Rust types. Input imports still use Plenty’s normal resolver; only reachable admitted functions are emitted. All input bodies are checked before pruning.
The checked Op representation is sufficient for this slice: typed slots,
structured control flow, specializations and source sites survive lowering.
Snapshot temporaries preserve evaluation order; scalar branch values become
Rust tuple joins. Already-validated ownership-only markers have no emitted
action for this Copy-only storage. That does not establish that the same
approach suffices for references or destruction.
Generated Rust has no dependencies beyond the installed standard library, no
unsafe code, heap marshalling, trait-object dispatch, macros or derives.
Checked failures call an aborting helper, and the generated module requires
panic=abort. It does not reproduce Plenty’s trap wording/status, stack guards,
runtime caller traces or debugger source information. A companion is an explicit
integration contract, not automatic discovery of arbitrary Rust APIs.
Measurement method
run.py checks the semantic slice and bench.py measures it. Every benchmark
variant compiles a real Rust library, compiles/links a Rust host, and runs it;
its exit-status checksum must match the same Plenty workload compiled through Cranelift.
An edited incremental library is checked against a freshly compiled native
program too. The source generator creates 1, 100 or 1,000 scalar functions,
each with twelve bounded multiply/add/modulo steps, plus an aggregate loop.
The Rust library boundary prevents the host’s fixed input from being folded
through the whole library. Library compilation and the tiny host’s
compilation/linking are separate measurements; the latter is not a pure linker
timer. --emit=metadata is another full invocation to bound checking costs,
not an exact phase subtraction. No linker-only or peak-memory claim is made.
Each artifact-cold sample starts a new compiler process without incremental
reuse. OS pages, the installed Rust sysroot and Plenty runtime artifacts are
warm. The native baseline sets PLENTY_CACHE=off; it is not Plenty’s warm
compiled-function-cache path. Explicit Rust incremental results distinguish
first build, unchanged source and a real first-function body edit.
All Rust variants use edition 2024, aborting panics, no debug information,
embed-bitcode=no, the installed default linker, and no cross-crate LTO, nightly options,
third-party compiler cache or external crates. The
rustc codegen documentation
describes codegen-unit parallelism, incremental reuse and omitting unneeded
embedded bitcode. The table below is measured evidence on this machine, not
a promise that these choices dominate for every crate.
Rustc’s default local ThinLTO across optimized codegen units remains enabled.
Observed build tradeoffs
Recorded 2026-10-24 on a Ryzen 7 7840HS, Linux x86_64, rustc 1.99.0
(b940084d7, LLVM 23.1.1), Plenty base be665bb0 plus this experiment.
Three samples per cell; compiler processes were restricted to CPUs 0–7
(four physical cores). The small/flat development runs were quiet. During the
large optimized/grouped runs, another worker performed validation on separate
cores 4–7 (CPUs 8–15). Later validation exposed substantial tmpfs occupancy
(roughly 32 GB) and a full 8 GB swap allocation; completed build artifacts were
moved off tmpfs before the final measurements. Per-sample memory pressure and
frequencies were not recorded for these exploratory snapshots, so shared
thermals and memory contention can confound their ratios. The shape effects
exceed sample spread, but that alone does not rule out changing machine load.
Raw samples are flat-measurements.json and
grouped-measurements.json in the experiment directory.
| Functions | Native dev compile/link | Rust inline-check library, opt 0 | Rust helper library, opt 0 | Rust host compile/link after helper library |
|---|---|---|---|---|
| 1 | 41 ms | 33 ms | 24 ms | 49 ms |
| 100 | 55 ms | 509 ms | 154 ms | 46 ms |
| 1,000 | 229 ms | 5,300 ms | 1,452 ms | 48 ms |
The separate Plenty check/emission process adds approximately 2/13/125 ms for these helper cases. These are independently measured stages, not totals from a single timer. Generating one monomorphic helper per arithmetic operation/type reduces repeated Rust checking work and emitted source size (1,000 functions: 6.4 MB inline versus 2.8 MB with helpers). One versus sixteen codegen units did not materially help the flat development case. A separate Rust metadata-only invocation took about 1.26 s versus 1.45 s for the helper library, suggesting that much of this cost precedes native code generation. This is not an exact phase partition.
Flat optimized output exposes a much larger problem. The experiment also emits
groups of 32 sorted function symbols into Rust modules, and optionally applies
#[inline(never)] to source functions while leaving arithmetic helpers eligible
for normal optimization. This is not a Plenty module cache. Body edits preserve
these groups; inserting symbols can renumber them.
| Helper emission, opt 3 | 100 functions: library build | 1,000 functions: library build | 1,000 functions: execution |
|---|---|---|---|
| Flat, ordinary inlining | 1,081 ms | 81,920 ms | 4.78 ms |
| Groups of 32 | 391 ms | 4,074 ms | 4.89 ms |
| Flat, source functions not inlined | 758 ms | 10,255 ms | 6.16 ms |
| Groups of 32, source functions not inlined | 337 ms | 2,901 ms | 4.60 ms |
Execution includes process startup and 100 aggregate iterations. Native release
took 5.27 ms in the flat snapshot; these tiny absolute times do not establish
a general Rust runtime advantage. Development helper execution took 21.89 ms
versus native development’s 10.45 ms. Full binary sizes, including linked runtime
material, are in the snapshots; they are not generated .text sizes.
Grouping and inlining control offer a credible way to avoid pathological optimized build costs without sacrificing this workload’s runtime. The exact LLVM attribution was not profiled; these emission comparisons suggest a useful direction, but the machine conditions prevent attributing the full ratio to one change or a specific optimizer pass. Grouping made the development build slower here, so there is no single winning configuration.
For the 1,000-function development library, explicit incremental reuse reduced the flat first/unchanged/body-edit rustc times to 2,148/795/1,092 ms. Grouped output measured 1,655/656/697 ms. The edit timings are single observations, with a checksum checked against a fresh native build. Add emission and host compile/link for an application edit; these are not full rebuild times or Cargo no-op timings. Rust still reads and checks the generated crate.
Final end-to-end comparison
finalist-measurements.json adds a single timer around checking/emission,
Rust library compilation and host compilation/linking, with three independent
samples. After the user corrected the CPU frequency limitation, the two finalist
configurations and matching native baselines were rerun in a coordinated quiet
window on 2026-10-24. Sampled frequencies were 3.25–4.09 GHz. Workloads, three
repetitions, 100 aggregate iterations, CPU affinity and compiler flags are
unchanged. The raw snapshot records the command, source commit and environment.
The initial exploratory emission-shape measurements were not rerun.
| Functions | Native dev | Rust helper dev, end to end | Native release | Rust grouped/non-inlined release, end to end |
|---|---|---|---|---|
| 1 | 42 ms | 83 ms | 42 ms | 94 ms |
| 100 | 61 ms | 246 ms | 65 ms | 343 ms |
The previous roughly 544 MHz results are preserved separately in
finalist-throttled-measurements.json and superseded by this table; the two
hardware regimes are not pooled. The fresh samples support practical subsecond
builds for these small Rust libraries, while the native path remains faster.
These two small synthetic workloads do not set a general build-time budget.
Recommendation: continue the Rust-hosted library experiment with compact monomorphic helpers and an explicit development/optimized emission policy. Small libraries can have practical build times, and stable function grouping can prevent very expensive optimization. This evidence does not make rustc a replacement for Cranelift’s fast build path. Native ownership-bearing API emission remains the next semantic gate; more backend breadth should not be inferred from the scalar benchmark.
Validation and limits
The differential suite covers all admitted integer widths, narrowing/widening,
wrapping arithmetic, negative floor/modulo and MIN % -1, branch values,
short-circuit skipped traps, loop exits, scalar pattern matching, specialized
generics, unit calls and a 100,001-step self-tail recursion with swapped arguments.
An independent Python oracle additionally checks every one of the 65,280
nonzero-divisor i8 pairs for floor/modulo, in native dev/release and generated
Rust opt 0/3 with both helper shapes. The sole overflowing quotient is skipped
there and checked separately as a fatal case; its modulo is still checked.
Overflow, zero division/modulo, division overflow and checked casts fail in both
native and generated debug/release builds. Exact signal numbers are not compared.
Rust → generated Plenty → Rust companion execution is checked with native integer signatures. A wrong Rust companion type produces a real rustc error mapped back to its Plenty declaration. Invalid Plenty in an unexported body still fails frontend checking; references, unsupported numeric types and unsupported recursion are rejected. A default unwinding Rust build is rejected.
These measurements cannot establish owner cleanup, native Rust borrowing, fallible allocation, collection interoperability or arbitrary crate support. They qualify a restricted compiler experiment; the broader proposal’s adoption criteria remain open. Scheduling stays in B20.
Generated library contracts and C headers
Status: Partly implemented. Accepted 2026-10-07. Numeric and
Resultexports, owned and borrowed handles, generated headers, embedded interfaces, extraction, compatibility fingerprints, and typed resident loading are implemented. Exports, owned exports, and contracts own the shipped contract. C interop records the wider boundary design.Shipped differently: there is no standalone manifest. The contract is a generated
.plentyiinterface, embedded in the binary and written besideNAME.handNAME.link-args.txt. The image API below is an illustrative sketch: generated headers differ in naming and in howResultexports report status. Exported error types include numeric values,AllocError,ParseError,RangeError,Failure, and nongeneric payload-free user enums; payload-bearing enums and error sums remain unsupported.Remaining: text and byte buffers, returned views, and broader error types (B27); callbacks, unloading, and loaders generated from extracted contracts (B28).
One contract, several consumers
A Plenty-produced static or shared library should expose an ordinary C ABI and
carry a richer Plenty interface. Generate its C entry adapters, C header,
machine-readable .plentyi contract, and header contract comments from the same
checked export representation. Consumers using Plenty get ownership checking and
automatic cleanup; other languages get explicit instructions for satisfying the
same contract. They do not need a Plenty compiler to call the C exports.
The generated contract describes both the actual C signature and its Plenty
interpretation. For example, a status code and output handle may represent a Result containing
an owned object; its release symbol supplies automatic destruction. This does not
expose Plenty’s private Result, class, or collection layouts as C layouts.
Borrow checking uses the interface without reanalyzing the library implementation.
Keep opaque handles and explicit C representations at this boundary. An owner
must be released through its originating library and allocator context. Ordinary
pub declarations do not become C exports or acquire a stable binary layout.
Generic exports need concrete specializations; metadata alone cannot instantiate
new machine code in an AOT consumer.
Comments are part of the exported interface
Generate precise, human-readable comments beside each declaration. Use stable templates driven by the contract, rather than inferred prose or a separately maintained description of ownership. Author-written purpose, examples, and domain-specific notes can supplement these generated facts, with their source distinguished from the generated requirements and guarantees.
| Subject | What the header must explain when applicable |
|---|---|
| Inputs | Nullability, alignment, initialized extent, valid ranges, and required object/library identity. |
| Borrowing | Read/write access, alias restrictions, duration, whether pointers are retained, and whether callbacks can access the same storage. |
| Ownership | Whether an argument is borrowed or consumed; who owns every value on each success and failure path. |
| Outputs | Which outputs are initialized for each status, whether they must be initialized before the call, and whether previous contents are released. |
| Destruction | Exact release function and context, whether null is accepted, which aliases become invalid, and why ordinary free is inappropriate. |
| Returned views | The owner keeping a view alive and the mutations, calls, or destruction that invalidate it. |
| Text and buffers | Encoding, byte/element units, lengths and capacities, terminators, embedded NUL rules, and null behavior for empty buffers. |
| Failure | Status meanings, allocation failure behavior, partial effects, and whether input/output state is preserved. |
| Execution | Thread affinity, concurrent-call and reentrancy rules, callback lifetime, and prohibitions on unwinding or nonlocal jumps across the boundary. |
| Library lifetime | How long code, release functions, borrowed data, and callbacks must remain available. |
Describe obligations and guarantees separately. For example, an exclusive borrow
requires the caller to prevent conflicting access during the call; the callee
guarantees not to retain that borrow afterward. A C const qualifier alone does
not communicate these requirements. Avoid vague statements such as “caller owns
the result” when error paths or borrowed subobjects change what that means.
Illustrative generated header fragment, not an implemented image API:
#include <stdint.h>
typedef struct ImagesImage ImagesImage;
#define IMAGES_OK UINT32_C(0)
#define IMAGES_INVALID_SIZE UINT32_C(1)
#define IMAGES_ALLOCATION_FAILED UINT32_C(2)
/**
* Requires:
* - image is a live, non-null handle from this library instance.
* - Call on the thread that created image.
* - No other access to image or its storage occurs during this call,
* including access through aliases or reentrant callbacks.
* - No outstanding borrowed view into image's storage exists.
* Guarantees:
* - Borrows image exclusively for this call; retains no new alias.
* - Ownership stays with the caller on every return path.
* - IMAGES_OK: image has the requested dimensions.
* - IMAGES_INVALID_SIZE: width or height is outside 1..65535 inclusive.
* - IMAGES_ALLOCATION_FAILED: dimensions are valid but storage allocation fails.
* - Both failure statuses leave image unchanged.
* - These are the only returned status codes; runtime traps terminate
* the process and are not converted to status codes.
*/
uint32_t images_resize(ImagesImage *image, uint32_t width, uint32_t height);
/**
* Requires:
* - image is null, or a live owned handle from this library instance.
* - For a live handle, call on its creating thread with no outstanding
* borrows and no concurrent or reentrant access to image or its storage.
* - The originating library remains loaded throughout this call.
* Guarantees:
* - Null is a no-op; otherwise consumes and destroys image.
* - All aliases and views into the destroyed object become invalid.
* - Use this function exactly once per owned handle; never use free().
*/
void images_destroy(ImagesImage *image);
The unchanged-on-error promise above is a property of this illustrative API, not a default for all exports. Emit only guarantees established by the checked implementation or explicitly declared trusted dependencies. Do not invent rollback, thread safety, non-retention, or error recovery from a C signature or a function name. Unsupported contracts must produce export diagnostics rather than silently generate a safe Plenty view. Explain trusted foreign assumptions where they are part of the public contract.
Plenty checks its callers against these contracts. Foreign callers must uphold the documented obligations themselves. Metadata and comments cannot stop invalid C pointer use; runtime detection, where provided, is a separate API guarantee.
Packaging and compatibility
Emit the versioned .plentyi interface beside the library and embed the same
contract in a dedicated binary section/resource or static-archive member; no
second manifest format is needed. Include target/ABI information,
stable exported type identities, contract format version, and an interface
fingerprint. Preserve discovery metadata through supported stripping, archive
extraction, and linker garbage collection. Read metadata at compile time without
loading or executing library code, and validate it as untrusted file input.
Runtime loading uses a library-specific versioned discovery symbol or function table and checks the expected contract before exposing callable handles. Unique symbol namespaces also permit multiple libraries in one static link. The fingerprint detects interface mismatches; it is not authentication or proof of a foreign implementation’s behavior. AOT clients compile against known interfaces; metadata does not provide arbitrary runtime specialization.
Keep loaded libraries resident initially. Any future unloading must account for owners, release functions, borrowed data, callbacks, and in-flight calls. Library output also needs a runtime packaging mode without application startup symbols; metadata does not solve allocator or runtime-state composition by itself.
Validation requirements
Compare generated header contracts with their .plentyi facts, including ownership
on failure and output initialization. Compile and run C callers against generated
headers for static and shared libraries, and test the corresponding Plenty
consumer’s borrow and cleanup behavior. Cover unsupported contracts, incompatible
metadata, allocation failures, matching destruction, and the supported strip/link
settings. These checks belong with export and loader implementation, not solely
with documentation snapshots.
Qualifying new boundary types
The CO3 assessment reinforces generating native representations and conversion code together. For Plenty, extend this to a reviewable contract for each supported conversion, rather than promising that every language type automatically has a public ABI. Before admitting a new B27 buffer, view or error representation, specify:
- Its native declaration, target layout and valid values independently of the private Plenty layout; include pointer/length relations and tag/payload rules.
- Whether each direction borrows, copies or transfers, with allocator and library provenance, invalidation, and cleanup on every conversion failure.
- Which preconditions remain trusted and which the adapter checks; define the error result or termination behavior before converting invalid input into a language value. Validate readable storage only after its foreign validity obligations are satisfied; generated checks cannot safely probe arbitrary pointers.
- The semantic facts included in compatibility checking. A borrow-to-transfer change can break callers despite an identical C prototype. New lowering rules must update the contract/fingerprint scheme as necessary, with explicit treatment of old consumers; a stable-looking C layout alone is insufficient.
Qualify the same representative API through a C caller, a Plenty consumer of the generated interface, and typed runtime loading where supported. Exercise static and shared linking, valid success and error paths, allocation failure, and exact destruction. For malformed-value tests, use valid backing storage with invalid tags or encoding, and check size overflow before dereferencing buffers. A plausible length exceeding the real allocation remains a foreign contract violation, not something the adapter can generally detect. Use compiler rejection tests for forbidden lifetime escapes. Test callbacks in both directions when that capability is added. Compare results and ownership effects, not only emitted header text or metadata. Existing C fixtures are the starting point, not a reason to defer B27 until all B28 facilities exist.
CO3’s finite specialization and tagged-dispatch examples are useful alternatives for a later concrete generic-export need. Plenty should initially require named, concrete wrappers. If specialization is added, record a finite exported set, stable symbol names and full lowered contracts. Shared machine representation does not imply interchangeable ownership or destructor semantics; tagged dispatch also needs validated tags and an explicit unknown-tag policy. Neither generic export syntax nor a universal companion-type framework is required for buffers, views, header verification, or the first callbacks.
Fallible memory, parallel execution, and SIMD
Status: Partly implemented; retained for B13 allocator design. Originally researched after
b690afeon 2026-10-05 against Cranelift0.131.1; that backend evidence is historical, not the current dependency. The design sketches below are not current API documentation.
- Recoverable allocation: implemented. Allocating operations expose allocation errors through
Result, and?propagates; see allocation and traps. Public allocator selection, budgets, and bounded containers remain B13.- Parallel execution: implemented. See concurrency. Automatic parallelization is B24.
- SIMD: initial subsets implemented. Explicit
std.simdsupplies four numeric vector shapes, masks, scalar fallback and bounded native lowering. Integer-column filter/count has automatic SIMD. Broader shapes, packed views and transformations remain B23.This page remains the B13 design record. The newer portable
std.simdproposal refines B23 with an API contract, automatic pipeline SIMD and measured backend evidence. The completed allocation audit and native concurrency contract record the delivered subsets. Checked-index arenas solve node identity; they are distinct from the allocator-lifetime proposal here.Shipped differently:
AllocErrorhas two variants,OutOfMemoryandCapacityOverflow. Class, tuple, enum, generator, and range storage is inline and needs no allocation; component expressions and fallible class initializers retain their own errors..unwrap()exists;.expect(...)does not. Only object headers record an allocator; container buffers use the global one. Threads start through a scopedwithmanager, and a pool isThreadPoolExecutor(max_workers, queue_capacity)withmap,map_result, andreduce_tree.
Current delivery and remaining design
The original recoverable-allocation foundation and explicit native concurrency have shipped. B23’s delivery record describes the initial explicit and automatic SIMD subsets. Generic collections are not packed numeric arrays, and ordinary loops remain serial.
Public allocator switching, per-container allocator handles, allocator lifetimes, and buffer provenance remain unimplemented. Objects and container buffers use the global allocator; objects no longer carry an allocator prefix (issue #814), so allocator selection must record provenance without enlarging global objects. Fixed inline arrays and constant value parameters are separately accepted but unimplemented under B83; bounded containers build on that design. These remaining contracts keep this proposal partly implemented. Scope and priorities belong in B13/B23/B24/B83.
The allocation, concurrency and SIMD sections preserve the original rationale and alternatives. Where they describe a shipped area, the linked references and completed audits take precedence over their historical sketches.
Original recommendation
Make recoverable allocation failure a language contract, with Result and ?
as the normal way to handle it. This requires more than making collection growth
fallible: error values, metadata, cleanup, and runtime helpers must also work
without an allocation succeeding. Establish this foundation before advertising
OOM recovery, custom allocators, or embedded suitability.
Keep the public model simple: one str, ordinary collections using a configured
default allocator, and explicit constructors accepting an allocator when needed.
Start parallelism with owned tasks, bounded channels, and scoped borrowing; add
explicit parallel collection operations after those are sound. Start SIMD with
a small set of explicit fixed-width vectors. Neither automatic parallelization
nor a general auto-vectorizer should be necessary for the basic language.
Historical implementation audit
This snapshot predates fallible allocation, inline sums/ranges/generators, static type metadata, and atomic ownership counts. It explains the original proposal’s motivation, not current behavior. See the implementation status and native concurrency audit for the current contracts.
| Area | Original behavior and implication |
|---|---|
| Raw object allocation | plenty-runtime/src/memory.rs::allocate calls alloc_zeroed, then handle_alloc_error on null; layout overflow exits through fail. |
| Collection growth | aggregates.rs::insert uses try_reserve for entries, but turns failure into process termination. Hash-table rebuilding uses an infallible vec! allocation. |
| Type metadata | Descriptors are parsed into String, Vec, and Rc<Type> at runtime. Even constructing a small collection can allocate metadata. |
| Classes and enum values | Records have owned heap allocations. Standard Result and Option use enum machinery; constructing an error is not an allocation-free escape hatch. |
| Text and I/O | Dynamic text, input buffering, formatting, and collection rendering allocate. String literals are immortal compiler data. |
| Indexing and ranges | String indexing/iteration creates a new managed one-scalar string; constructing a range allocates metadata and a managed range object. |
| Structural equality | aggregates.rs::equal memoizes aggregate pairs in a Rust HashSet, avoiding exponential work on shared graphs whose floating-point contents prevent identity shortcuts. Equality and membership can therefore allocate internally. |
| Generators | Creating the frame allocates; suspended values own resources. Returning Option from iteration also uses enum machinery. |
| Destruction | An intrusive thread-local queue avoids allocating a separate worklist while traversing owned object graphs. User __del__ code can still allocate. |
| Sharing | Object reference counts are non-atomic u64; metadata uses non-atomic Rc. Moving a collection does not prove its reachable immutable strings are unshared. |
Rust does have fallible APIs: Vec::try_reserve preserves existing contents on
failure. The familiar infallible APIs choose a different policy, and the standard
allocation-error handler normally aborts with std. Plenty can expose a
different policy while using Rust underneath, but must select or implement
fallible operations consistently. Rust Vec,
allocation error handling.
Recoverable allocation contract
This is the original design rationale. The
completed allocation audit records the delivered
per-path behavior, including inline classes, sums, ranges and generator frames.
The third error category, fallible generator-frame construction and heap-backed
class construction discussed here were not adopted; .expect(...) remains
unimplemented. None is an additional implementation commitment.
Allocation-free error transport comes first
Introduce a small, non-owning AllocError value distinguishing exhaustion,
capacity overflow, and an unsupported allocation layout. Its data should be
fixed-size integers/tags, not an allocated message or backtrace. It can be
formatted later if resources permit; an emergency reporting path writes a
static message directly without growing a buffer.
Represent ordinary finite Result[T, E] and Option[T] values without a new heap
allocation for their wrapper: use an inline tag/payload layout, register tuples,
or a caller-provided result slot according to a private Plenty ABI. Moving an
already-created payload into Ok or Err must not allocate. This is a concrete
reason to generalize the current single-word managed-value representation.
Recursive values still need explicit indirection; neither inline errors nor
stack placement imply arbitrary recursive types have a finite layout.
Implementing ? before this representation change is useful for normal errors.
It must not be described as OOM-safe while its generated error path still creates
heap-backed enum objects. Compiler-emitted immutable type metadata is the next
useful change: replacing runtime descriptor parsing removes hidden allocation
and a future cross-thread Rc hazard at the same time.
Which operations return Result?
Recommend fallible public signatures for operations whose contract permits allocation, even on calls that happen to fit existing capacity. Their type must not change with optimization level, input size, or whether the compiler managed to place a particular object on the stack.
For example, a future items.append(value)? propagates AllocError, and a future
copy(value)? can fail when duplicating owned data. Explicit .expect(...)-style
termination may be offered for applications choosing that policy; it is not a
silent default selected by release builds. A terminal failure is distinct from a
recoverable allocation error and does not promise user cleanup.
There is a syntax decision to make before changing existing programs: allocating
literals, comprehensions, string concatenation, heap-backed class construction,
and generator creation have no explicit fallible method today. A consistent
candidate is for these expressions to produce a Result, with explicit ? at
the use site, including ([f(x) for x in source])?. This makes allocation visible
but changes many familiar expressions and needs learner examples before adoption.
An alternative is a narrowly specified allocation effect with an explicit
handler boundary; it adds more language machinery and should not be quietly
introduced as implicit ? insertion. The Result with explicit ? form was
chosen.
The required behavior is clearer than the syntax: a program must be able to
handle every language-runtime allocation failure without an unadvertised abort.
Audit these paths together, not just list.append:
| Allocation site | Required recovery behavior |
|---|---|
| Lists, dicts, sets | Reserve entries and hash storage before committing a mutation; report checked-size overflow separately from exhaustion. |
| Comprehensions | Destroy the partial result and pending owned temporaries on failure. Already-completed I/O or other external side effects are not rolled back. |
| Strings | Concatenation, formatting, conversion, slicing that copies, and input buffering expose failure; literals and merely retaining existing immutable storage need no allocation. |
| String indexing and ranges | Remove their internal allocation through a suitable representation, or expose failure even though these operations currently look like simple value expressions. |
| Equality and membership | Replace allocating memoization with an allocation-free strategy, or make its failure explicit. A bool result must not conceal an abort when temporary comparison storage is exhausted. |
| Classes and enum payloads | Clean up only successfully initialized fields if construction fails. No call to a whole-object custom destructor on a partially initialized object. |
copy | Destroy a partially built deep copy; retain the original unchanged. Resource objects with custom cleanup keep their existing copy restrictions. |
| Generators | Frame creation is fallible. Allocation errors in the body need an explicit yielded Result or a designed fallible-iteration contract, not an indistinguishable end-of-stream. |
| Metadata and dispatch | Prefer emitted static tables. Any remaining lazy caches or adapter allocation must return errors or be preallocated explicitly. |
| Printing and diagnostics | Stream through a nonallocating writer or propagate allocation/I/O errors. Formatting an error must not recursively allocate another error. |
| Scheduling and channels | Pool creation, task records, queue storage, thread stacks, and message construction have distinct fallible creation/submission contracts. |
| FFI | An external allocator/library can still abort; an imported declaration must not claim recoverable failure unless its implementation provides it. |
Mutation, moved inputs, and cleanup on failure
Guarantee the strong property for allocator-aware collection mutation: if reserve, rehash, or allocation fails, the existing collection’s logical contents remain unchanged. Capacity may have increased during preparatory work. A replace operation must not destroy the old value until all required allocation succeeds. Allocator failure must leave the original allocation valid on failed resize.
Recommend a simple initial ownership policy: an ordinary owning append consumes
its argument on the call, including failure, and destroys it if it was not stored.
The caller cannot conditionally reuse the moved binding after observing Err.
Document this prominently. A later recovering variant can return the uninserted
item together with AllocError, but that composite error must itself have an
allocation-free representation. Another explicit workflow is reserve first, then
move the value into already available capacity. Do not hide input cloning in the
error path.
Destruction itself must not allocate. The existing intrusive queue is a useful
starting point, not a complete proof: custom destructors, allocator callbacks,
formatting used by cleanup, and cleanup of partially initialized values all need
auditing. A destructor has no caller to receive Result; keep __del__ and scope
exit hooks infallible and make fallible close/flush an explicit operation before
drop. OOM-safe cleanup must handle any fallible work locally and continue releasing
owned resources. A later compiler-checked noalloc effect could verify critical
cleanup and request-rejection paths; initially keep this a documented runtime
contract with tests, not an unimplemented guarantee. See the companion
context-manager proposal.
The runtime needs a failure-injection allocator: fail allocation number N for every N in a test, then verify unchanged source values, balanced resources, partially initialized cleanup, and no secondary allocation on the failure path. Include failures during hash growth, nested copies, generators, output, and user cleanup. Success-only accounting and sanitizer tests cannot establish these failure guarantees.
What recovery can and cannot guarantee
An allocator reports only failures it can observe. On an overcommitting OS,
successful reservation can be followed by process termination when pages are
touched; a Result cannot intercept the kernel killing the process. Bounded
allocators and request budgets make admission control more useful; reserved and
committed-memory policy remains platform-specific. Linux explicitly documents
multiple overcommit policies. Linux overcommit accounting.
Compiler OOM, process startup, stack overflow, and third-party C libraries are separate limits. Do not advertise survival of arbitrary system-wide exhaustion. For services, the practical guarantee is narrower and useful: reject work when a controlled allocator cannot satisfy a request, clean up, and continue using previously reserved service resources. An application needs a preallocated path for that rejection response too.
Custom allocators without multiplying everyday types
The user’s proposed separation is viable. Prefer one ordinary constructor with
an optional allocator argument over unrelated list_custom/dict_custom names.
Exact syntax depends on generics, but conceptually list[T](allocator=pool) uses
the same operations as list[T](); literals and comprehensions use the configured
default. An explicit collection builder can later direct a comprehension into a
particular allocator without introducing ambient thread-local allocation scope.
Use a small runtime allocator handle containing a context pointer and operations for allocate, grow/resize, and deallocate. The contract includes alignment, checked sizes, thread eligibility, failure behavior, and ownership of the handle. This is compatible with statically known adapters for user classes; it does not require every ordinary collection type to expose an allocator generic parameter. The tradeoff is a stored handle and indirect allocation calls. Specialized allocator-generic collections can remain a later optimization.
Rust’s per-object Allocator contract is useful prior art, including matching
deallocation and allocator lifetime, but the inspected stable documentation still
marks it an unstable API. Plenty can implement its own stable runtime interface
without exposing Rust’s allocator ABI. Rust’s GlobalAlloc also permits an
implementation that aborts instead of returning null: a Plenty allocator
advertised as recoverable must impose the stronger return-failure rule.
Rust Allocator,
Rust GlobalAlloc.
The difficult part is lifetime and provenance, not constructor spelling:
- Each object and internal buffer remembers the allocator that created it, and frees through that allocator even if the default later changes. Growth stays with that allocator. Start with default selection fixed at startup to avoid unnecessary ambient state and initialization races.
- The allocator’s context must outlive every live allocation and destructor using it. Start with process-lifetime or explicitly retained allocator handles. Handle creation is fallible and its own allocation must not form an ownership cycle.
- A scoped arena is a borrow-checking feature: forbid reset/destruction while any arena-backed value or reference can survive. Deferring resettable arenas is better than using a runtime pointer that silently dangles after a scope ends.
- An arena does not make resource cleanup optional. Destructors of files and other resources still run; bulk byte reclamation cannot replace them.
- Inserting an existing string or nested list into a pool-backed list does not
migrate its storage. The outer buffer uses the pool; each child retains its own
allocator. A future explicit
copy_into(pool)must define which reachable storage it duplicates and may fail partway through. copyshould default to the source allocator; changing allocators is explicit. No pointer allocated in a DLL may be passed to another DLL’s unrelated free function. FFI ownership contracts need the matching release operation and its library lifetime; Plenty’s allocator handle remains a private representation.- Thread-confined allocators make their values thread-confined. Merely making a reference count atomic cannot make an allocator or its destructor thread-safe.
Supporting arbitrary allocator callbacks is itself a trusted low-level extension; the compiler cannot prove that foreign callbacks honor alignment and lifetime. Built-in budget/pool allocators should be the first supported implementations.
Fixed capacity and inline storage
The ideas in heapless and smallvec solve different problems. heapless
containers store fixed-capacity backing storage inline and report capacity
exhaustion; SmallVec stores a small prefix inline and spills to the heap when
needed. Inline means inside the containing value, not necessarily on the stack.
heapless,
SmallVec.
The original sketches were Array[T, N], BoundedList[T, N], and
SmallList[T, N], with constant generic capacities. The accepted
value-generics design now specifies array[T, N], but is
not implemented; the bounded and small-list names here remain illustrative.
A bounded container returns a capacity error without allocating; a small list
still needs fallible spill/growth. Element construction can allocate even when
the container does not. A borrowed slice/view should let APIs accept their
contents without copying into the ordinary heap-backed list. Defer this family
until basic constant parameters exist; inline aggregates have shipped. It should
not complicate the first custom-allocator interface.
Threads, channels, and parallel collections
This section preserves the original concurrency design. Scoped threads, bounded channels, executors, ordered mapping and fixed-tree reduction are implemented; the concurrency reference owns current APIs, worker eligibility, ordering and shutdown contracts. The suggestions below are not additional pending concurrency work.
Runtime prerequisites
The native concurrency audit records the current runtime and boundary contract. Type metadata is static and immutable, and object counts now use atomic ownership handoff, including immutable strings: a list move may leave a string alias in the originating thread. Atomic ownership alone does not make a mutable payload safe to share or grant foreign thread permissions.
Retain static immutable type metadata and use atomic reference counts for storage
that can be shared between threads, including the single public str type.
Benchmark the cost before choosing whether all headers should use the same atomic
implementation or whether internal unshared objects can keep a cheaper path.
Either choice preserves one user-visible str type. Revisit the destructor queue
algorithm when changing the count field: zero-to-destruction handoff, memory
ordering, and recycling the dead count as a link need a coherent atomic design.
Define compiler-known structural properties for transferable ownership and shareable immutable borrows. These are safety facts, not protocols users can satisfy merely by writing a method with the right name. Ordinary methods remain explicitly resolved; importing a module must not enable extra thread eligibility. Check all captured values recursively, including allocator handles, generator frames, destructor requirements, and FFI handles with thread affinity.
Initial user model
- Owned tasks move captures and results. Thread creation returns
Result, and join is explicit; no silent detached thread holding resources beyondmain. - Scoped tasks may borrow until the scope joins every task on every exit path,
including
?. Shared borrows need thread-shareable values; mutable borrows require disjoint storage proven by the checker. Scoped threads are established prior art in Rust’sthread::scope. - Bounded channels own preallocated queue storage. Sending transfers ownership; failed or disconnected send returns the undelivered value. Creation and payload construction are fallible; preallocating the queue does not eliminate allocations inside payloads. Specify closure, blocking, and shutdown behavior before adding unbounded channels.
- A configurable thread pool supports explicit parallel map/collect/reduce. Keep the executor API replaceable: the language need not promise Rayon as a dependency, and any chosen executor must meet the allocation contract.
Rayon demonstrates explicit parallel iterators and configurable pools, including
num_threads; its parallel fallible traversal may observe multiple errors and
does not specify which one is returned. These are useful comparisons, not a
reason to inherit unspecified language behavior accidentally.
Rayon pool configuration,
Rayon fallible traversal.
For an initial Plenty parallel collection operation, preserve input order in the successful result. If multiple items fail, returning the failure at the lowest input index is easier to reproduce; it requires completing earlier in-flight work before reporting it. Cleanup drops all partial outputs and joins started tasks. It cannot undo I/O already performed by workers. A simpler first-completed error mode is an alternative, but must be explicitly documented if selected. Dictionary/set construction must still resolve duplicates using source order. The order of destructor side effects across threads is not the serial iteration order and must not be promised accidentally.
Cancellation is cooperative and happens at defined boundaries, not by killing
threads during mutation or drop. Restrict break/continue in higher-level
parallel loop forms until their semantics are specified. Floating-point
reductions need an explicit order policy: reassociation changes results; a
parallel sum cannot silently claim ordinary left-to-right arithmetic semantics.
Automatic parallelization is deferred. Borrow independence alone does not prove equivalence: allocation failures, I/O, externally visible destructor order, floating-point reductions, and first-error ordering are observable. Start with explicit opt-in operations, measure workloads, then consider an effect analysis for a small deterministic subset. A runtime thread-count setting is reasonable for these operations, but should not silently change ordinary loop semantics.
SIMD that fits Cranelift
This is the original backend assessment, not an outstanding initial SIMD plan. The explicit implementation record describes the shipped API and representation limits; the portable SIMD proposal separates delivered subsets from broader transformations. Current values use copyable inline enums, with bounded local promotion to vector registers, rather than a public packed-vector layout or vector C ABI. The initial automatic integer-column filter/count slice is separate from general auto-vectorization.
Cranelift represents lane-typed vectors and provides vector arithmetic, but an IR type’s existence does not promise every instruction or width works on every backend. The upstream IR guide explicitly discusses target-dependent support. Cranelift IR, instruction builder.
The installed cranelift-codegen 0.131.1 source was checked directly:
src/ir/types.rs tests cover I32X4 and F64X2; src/isa/x64/lower.isle includes
fadd rules for F32X4/F64X2; src/isa/aarch64/lower.isle lowers vector fadd.
This is evidence for a small 128-bit starting set, not a blanket claim about
256/512-bit vectors, scalable vectors, arbitrary lane counts, or all operations.
The docs.rs pages for that exact crate version were unavailable to the research
tool, so the source inspection, rather than latest-version docs, establishes the
version-specific finding.
Propose explicitly imported vector types/functions first: 128-bit shapes such as
four f32 lanes or two f64 lanes, with splat, checked load/store, arithmetic,
comparisons, masks, and lane extraction. Names such as simd.f32x4 are sketches.
Represent vectors inline in SSA/registers or stack slots; do not heap-box each
vector as general managed objects were boxed at the time of this assessment.
Efficient vectorized kernels
also need contiguous typed buffers/views: the current generic collection entry
storage is not a guarantee of a contiguous C array of lane values.
Specify exact integer overflow behavior, mask representation, lane order, alignment, bounds, and floating-point behavior. Scalar fallback must preserve those semantics, not merely produce approximately similar answers. Unaligned loads are a useful safe default where supported; never read past a slice end to fill a vector, even if discarded lanes appear unused. Keep scalar tails explicit at first. SIMD reduction order and NaNs must follow the documented operation; relaxed reassociation and fused arithmetic require opt-in semantics.
Compile a documented baseline target with scalar fallback for unsupported operations. Offer explicit native-target compilation separately, recording the required CPU features; shipping native-target binaries to an older CPU can fail. Runtime feature dispatch and multiple compiled variants are later library/compiler features with code-size and compile-time costs. SIMD vectors should not cross the initial public C ABI by value: use pointers and lengths until each target’s ABI and alignment rules are deliberately supported.
Before adopting a vector operation, test native and scalar implementations on boundaries, NaNs/infinities, overflow, short/unaligned buffers, and each supported target. Inspect generated code and measure throughput and compilation time. Explicit vector lowering is compatible with Plenty’s fast-compilation goal; global loop dependence analysis and speculative vectorization are not required.
Original implementation sequence
This sequence records the original dependencies, not the current work queue. Delivery status is reconciled below it; remaining scope lives in the backlog.
- Add ordinary
?and clarify cleanup on propagated errors. Explicitly document that current heap-backedResultis not yet an OOM-recovery mechanism. - Design allocation-free finite sum/aggregate layouts and emitted static type metadata; measure compile time and ABI complexity before wider runtime changes.
- Introduce a fallible allocation ABI and failure-injection tests, then migrate every runtime allocation path. Settle allocating-expression syntax with small tutorial examples before changing all public APIs.
- Add a configured default allocator and explicit per-container allocator handles, starting with a bounded budget allocator. Keep scoped resettable arenas deferred until lifetimes can be enforced.
- Add contiguous buffers/views and bounded inline containers as generic/value support permits. These also establish a practical foundation for SIMD.
- Audit thread safety and add transfer/share checking, atomic shared ownership, owned/scoped tasks, and bounded channels. Then explicit pool-based parallel collection operations with documented ordering and failure behavior.
- Add the narrow tested SIMD surface. Automatic parallelization, auto-vectorization, allocator specialization, and runtime CPU multiversioning remain optional later work rather than prerequisites for useful programs.
Steps 1-3 and 6 are complete; their descriptions of the then-current runtime are historical. Step 4 remains B13. Step 5 is mixed: inline aggregates, list/byte slice views and packed columnar storage have shipped, while fixed inline arrays and value parameters remain B83 and bounded containers remain B13. Step 7’s initial explicit vectors and bounded automatic integer-column SIMD have shipped; B23 tracks extensions, not an unimplemented SIMD foundation.
Allocating-expression syntax, the private sum-value lowering contract, and task error ordering are settled. Existing collection operations also define ownership on failure; see the collection reference. Any future recovering-insertion API would be a separate design, not an unresolved contract for today’s operations. Allocator handle lifetimes and buffer routing remain open B13 design questions. Published SIMD operations already have scalar contracts; broader operations require separate decisions in the portable SIMD design.
The private lowering contract is not a frozen ABI or a public C representation; it must leave room for future FFI, vector values, target-specific calling conventions, and further representation improvements.
Thread-local reference counts
Status: Not implemented; stage 1 measured and rejected. Design note for issue #746. Stage 1 was built (issue #751, local PR #944) and removed every locked count instruction from k-nucleotide without a measurable cycle gain, so the PR was closed and stage 2 is not recommended; see measured result. The memory and destruction and concurrent runtime pages own the current contract. Issue #744 (no count pair around borrowed arguments) and issue #745 (sole-owner release) have landed.
Summary
Every runtime count update is a lock-prefixed read-modify-write, although
most objects never leave their thread. This note proposed the Lean 4 and Koka
design, a per-object thread-shared mark set at certified thread boundaries,
starting with a process-wide single-threaded phase that needs no marking.
Stage 1 was built and removed every locked count instruction from k-nucleotide, with no measurable change in cycles. On the measured processor the locked instructions are not the cost, so neither stage is recommended. The design below is kept as the record of what was considered.
The note also found that the retain half of issue #745’s fast path is unsound without a thread-locality guarantee; see the sole-owner fast path.
Measured result
Stage 1 was implemented as described below (issue #751, local PR #944): a
live-thread count raised only in plenty_thread_start and lowered only after
plenty_thread_join returns, with plain count updates while it is zero and a
runtime-checks trap for count updates from unregistered threads. Measured
against main at d7a70ede (issues #744 and #745 landed) on an AMD Ryzen 7
7840HS (Zen 4), with callgrind and perf stat -e cycles:u pinned to one core:
| Workload | Locked count instructions | Instructions | Cycles |
|---|---|---|---|
| k-nucleotide (fasta 50,000) | 1,399,543 to 0 | +0.2% | Within noise (under 2%) |
| reverse-complement | About 0 before and after | Unchanged | Within noise |
| Parallel k-nucleotide (7 pool jobs) | Unchanged: workers are live | +0.4% | Within noise |
| Retain/release loop, 50M iterations | All removed | About 3 more per operation | Unchanged |
In the loop benchmark, plenty_retain and plenty_release fell from 46% to
7% of cycle samples while total cycles stayed the same.
The premise of this note was wrong for this processor. Issue #743 attributed
k-nucleotide’s 26.5% of cycles in plenty_release to the lock decq, because
96% of that function’s samples fell on the instruction after it. An
uncontended locked operation on a cache line the thread already owns is cheap
on Zen 4; the samples land after the first access to the object’s header,
which is the likely real cost, attributed to the next instruction. This has
not been confirmed with cache-miss counters, and processors with costlier
atomics (some AArch64 cores, older x86) were not measured.
Consequences:
- Stage 1 adds about three instructions per count operation for no measured gain, so it was not merged.
- Stage 2 removes the same locked instructions on parallel programs and adds marking walks; on this evidence it is not worth building.
- The cost to attack is touching object headers: fewer count operations (issue #755, a last-use move after a borrow), fewer separately allocated objects, and confirming header misses with cache counters before further work on count atomicity.
Why counts exist and why they are atomic
Plenty has single ownership at source level, but runtime objects begin with
{AtomicU64 refs, destroy} (plenty-runtime/src/memory.rs). A count above
one arises from:
- copying an immutable heap
str, or an inline value containing one, which retains instead of copying; Shared[T]handles, whoseshare()retains one box;- native helpers that return retained projections and builder aliases, and the retain/release pairs still emitted around some borrowed arguments and receivers (issues #744, #741).
Strings of up to seven bytes are inline and own nothing; their count calls
return before touching memory. Literals use the immortal count u64::MAX.
Counts are atomic because counted owners can cross certified boundaries, and
because a worker can retain through a borrow: a scoped worker that copies a
str out of a borrowed list[str] increments a count the parent may be
incrementing at the same time. A release site cannot tell whether that ever
happened to its object.
Release uses fetch_sub(Release) and an acquire fence on the final count;
retain uses a relaxed compare-and-swap loop that rejects overflow. After the
final release the count word becomes the destruction queue’s link
(plenty_release, continue_after), on the destroying thread only.
Where an object can reach another thread
| Boundary | What crosses | Concurrent count access? |
|---|---|---|
Scoped spawn with &T arguments or spawn(&job) captures | Borrowed parent graph | Yes: worker and parent both retain |
Scoped spawn with moved, copied or &mut T arguments | Owned or exclusively borrowed graph | Yes: a str inside can share an object with a parent local |
| Scoped join result | Worker result | No: the worker’s frame is gone and join orders it |
pool.submit and consuming spawn | Job environment | Yes, as for moved arguments |
| Executor map’s shared environment | One environment borrowed by every job | Yes |
| Future result | Job result | No: the adapter finishes, including environment cleanup, before Phase::Ready is published under the job lock (executors.rs) |
| Channel send | Message | Yes |
.par pipelines, ordered map, fixed-tree reduce | Borrowed list and callback captures | Yes |
Shared[T] construction | Contents, and the box itself | Yes: every holder retains the box and copies from the contents |
Store under a Mutex lock | The stored value | Yes, when the mutex is reachable from another thread |
std.actor | Messages are copied into the receiver’s heap; all actors run on one thread | No |
| FFI | ForeignPtr cannot enter workers; exports require one caller thread | No |
| Globals | None exist; constants are immortal | No |
| Runtime-internal threads | The detached DNS lookup thread in aggregates/net.rs holds only Rust data | No |
Every Plenty thread starts in threads::plenty_thread_start (scoped tasks and
executor workers alike). threading::check in src/threading.rs already
walks each crossing type for eligibility, with src/effects.rs classifying
the native operations a worker can reach; both are compile-time type walks.
Marking adds a runtime walk over values of the same types.
Prior art
| System | Mechanism | Fit for Plenty |
|---|---|---|
| Lean 4 | m_rc > 0 single-threaded, < 0 multi-threaded, 0 persistent. lean_dec_ref inlines the rc > 1 case; lean_dec_ref_cold does the atomic fetch_add on negative counts. lean_mark_mt walks the graph with a worklist and negates counts; alloc_task calls it on a task’s closure, and task results are marked when resolved. Marks are never removed. Lean copies a shared object instead of updating it in place, so a marked object never points to an unmarked one. | Good. Boundaries are explicit and compiler-known. Plenty mutates in place, so it needs a store barrier where Lean copies. |
| Koka (Perceus) | Follows Lean: thread-shared objects have negative counts, so one inlined rc <= 1 test takes the slow path for both “shared” and “last reference”. tshare marks recursively; starting a thread marks its argument; storing into a thread-shared mutable reference is the other sharing operation. | Same as Lean; Koka’s mutable-reference rule corresponds to Plenty’s Mutex stores. |
| Biased RC (Choi, Shull, Torrellas, PACT 2018); CPython 3.13 free-threaded (PEP 703) | Each object has an owner thread with a plain local count; other threads use an atomic shared count. A negative shared count queues the object for the owner to merge. Swift prototype: RC operations over twice as fast, client programs 22.5% faster on average. CPython uses separate ob_tid, ob_ref_local and ob_ref_shared fields, immortal objects and deferred counting. | Poor. Needs an owner id per object and a thread-id comparison per operation; the 16-byte header has no room, and owner-thread merge queues conflict with the destruction queue’s reuse of the count word and with pool threads that finish. Its advantage, no walk at a boundary, matters less when the compiler knows every boundary. |
| Swift | Atomic counts on every class reference. OSSA copy_value/destroy_value and copy propagation delete redundant pairs. -assume-single-threaded makes all reference-counting instructions nonatomic, unchecked. Ungar, Grove and Franke (DLS 2017) report ARC taking up to 80% of time and propose dynamic atomicity: a store barrier plus generated tracers that mark objects reachable from shared storage. | Plenty’s current model. Pair elimination is issue #744. The phase flag below is a checked, dynamic -assume-single-threaded. Dynamic atomicity is the per-object design with a store barrier. |
| Rust | Rc is non-atomic and not Send; Arc is atomic. Owned values have no count. | Would split str into local and sendable forms, a user-visible distinction the language avoids. Shared[T] already plays Arc’s role. |
| Nim ARC/ORC | Counts are never atomic; isolate checks at compile time that a subgraph has no outside references, and Isolated[T] moves it between threads. --mm:atomicArc is the thread-safe variant. | Moved Plenty values are not isolated, because str copies share storage. Copying strings at each boundary would add hidden allocation, and Shared[T] contents are read by several threads anyway. |
Recommendation
Stage 1: single-threaded phase
Keep a process-wide count of live Plenty threads other than the main one.
plenty_thread_start increments it before pthread_create; the parent
decrements it after the corresponding join or pool shutdown. While it is
zero, retain and release use plain (relaxed load and store) updates:
release(o):
if inert(o): return
r = o.refs.load(relaxed)
if r == IMMORTAL: return
if live_threads.load(relaxed) == 0:
if r == 1: queue o for destruction # no fence: no other thread
else: o.refs.store(r - 1, relaxed)
return
atomic path as today
Soundness: only the main thread runs while the count is zero, and it alone
changes the count from zero. Its plain updates before a pthread_create
happen before everything the new thread does; every update while workers are
live is atomic; pthread_join orders every worker update before the plain
updates that follow. A pool’s idle workers keep the count above zero until
shutdown joins them. The destruction queue, continue_after, immortal counts
and actor heaps are unchanged.
Cost: one load of a read-mostly global and a predictable branch per count operation, next to the existing inert and immortal tests. No marking, no header change, no codegen change. Programs that start threads lose nothing while workers are live.
Stage 2: per-object thread-shared mark
Only if a parallel workload shows atomic count cost while workers run.
- Encoding. Bit 62 of
refsmarks a thread-shared object;u64::MAXstays immortal. One unsigned compare,refs < SHARED_BIT, selects the plain path for both retain and release; shared and immortal objects take the slow path, where immortal is tested first. Counts stay below2^62 - 1. - Marking. At each “yes” boundary in the table above, the compiler emits
a call to a marking walk for the crossing value’s type. A type that cannot
reach a heap
str, aSharedbox or an aliased helper result needs no walk:list[i64]orBytescosts nothing,list[str]visits each heap string. The walk uses the destruction queue’s worklist pattern, not recursion, and plain stores: the objects are thread-local until the boundary publishes them. - Store barrier. Plenty mutates in place, so a marked container can later
receive an unmarked value. Stage 2 marks values stored under a
Mutexlock, and walks do not stop at already-marked objects. Stopping early would need a barrier on every store into a marked container, as in Ungar et al. - No unmarking. A marked object stays marked, as in Lean and Koka: after a join another thread may still own a copy received through a channel.
- Destruction. The final release of a marked object keeps today’s release/acquire handoff. Its children are marked, so the destroying thread updates them atomically. The queue link overwrites the mark only in a dead object, on its destroying thread.
Costs: the walk at each crossing, proportional to the reachable heap strings,
boxes and aliases (a scoped spawn over a large &list[str] walks all of it,
once per spawn); a compare per count operation, which replaces the lock
instruction on the common path; the Mutex store barrier.
The sole-owner fast path
Issue #745 proposes that a release reading refs == 1 destroy without an
atomic operation, and that a retain reading 1 store 2. The release half is
sound: the releasing owner holds the only count, and no borrow of that owner
can be live on another thread while it is released, so no other thread can
retain concurrently.
The retain half is not. Retains happen through borrows, not only from owners:
parent owns s: str # refs == 1
spawn(worker, &s) # worker copies s: reads 1, stores 2
copy s on the parent # concurrently reads 1, stores 2
Both threads store 2 for three owners, and the object is freed while one
copy is live. The same happens with two pool jobs copying an element of a
Shared[list[str]]. A plain retain needs a thread-locality guarantee, which
stage 1 provides for the single-threaded phase.
Expected gain
Executed lock-prefixed count instructions, counted with callgrind
--dump-instr=yes --compress-pos=no at the addresses of lock decq in
plenty_release and lock cmpxchg in plenty_retain, on main at
fe11d0a2 (release compiler, --no-strip):
| Program (input) | Release calls | Atomic releases | Final releases | Retain calls | Atomic retains |
|---|---|---|---|---|---|
| k-nucleotide (fasta 50,000) | 4,320,706 | 1,815,225 | 222,025 | 2,569,772 | 1,315,181 |
| reverse-complement (fasta 50,000) | 1,536,704 | 1,536,704 | 3 | 1,536,701 | 1,536,701 |
| fasta (100,000) | 1,033,401 | 1,033,401 | 7 | 1,033,392 | 1,033,392 |
| regex-redux (fasta 50,000) | 412,613 | 281,838 | 246 | 232,686 | 231,533 |
The other benchmarks perform under 1,000 count operations each. The calls not counted as atomic are inline strings and literals.
- Almost every atomic operation is a non-final pair at
refs >= 2. In k-nucleotide the call sites incount_framesare the pair aroundget_mut(&key)(issue #744) and pairs around borrowedBytesreceivers; in reverse-complement all of them are theBytes.get/Bytes.setreceiver pairs of issue #741. Issue #745 handles the final releases: 12% of k-nucleotide’s atomic releases, almost none elsewhere. - Once those pairs are gone, stage 1 removes whatever atomic operations remain in these single-threaded programs. The size of that remainder is unknown until issues #744, #741 and #745 land; from the call sites it is expected to be small, so the additional gain on k-nucleotide and reverse-complement is likely a few percent at most.
- The upper bound, if pair removal covered none of them: in k-nucleotide at
250,000,
plenty_releaseandplenty_retaintake 30.4% of cycles (issue #743); in reverse-complement about 60% (issue #741). - Stage 2 gains nothing on these programs over stage 1. Its gain on parallel programs is unmeasured; BRC’s 22.5% average for Swift client programs is an indication, not an estimate for Plenty.
What to measure
- Before and after each stage, the executed
lockcount-instruction totals above, the final-release count, and callgrind instruction totals for k-nucleotide and reverse-complement; thenbench.py runtimings. - A concurrency-heavy workload with workers live during the counting:
a parallel k-nucleotide counting each frame on a pool job, as the Rust #1
reference does; a channel pipeline of
strmessages; and readers of aShared[dict[str, i64]]. Stage 1 should leave these unchanged; they decide whether stage 2 is worth building. - For stage 2, the marking cost: spawns in a loop over a large borrowed
list[str], and channel sends oflist[str]messages. - Correctness: runtime tests in which several threads retain through a borrow, including the scenario above, and a stage-1 test that the counter returns to zero only after join and pool shutdown.
Decisions
- Stage 1 was accepted as the first step, then measured and rejected; see measured result.
- The live-thread count was to return to zero after the last join, decremented
by the joining thread after
joinreturns; detached or unjoined threads keep it raised. - Issue #745 dropped the retain fast path and landed the release half only.
- The parallel k-nucleotide variant in
benchmarks/game(local PR #940) is the threaded workload. - Any future thread that runs Plenty code, such as the proposed
actor blocking-work threads, must start through
plenty_thread_start. This only matters if count modes return.
Sources
- Lean 4 runtime:
lean.h(lean_is_st,lean_is_mt,lean_dec_ref) andobject.cpp(lean_mark_mt,alloc_task,lean_dec_ref_cold); Ullrich and de Moura, Counting Immutable Beans, IFL 2019. - Reinking, Xie, de Moura and Leijen, Perceus: Garbage Free Reference Counting with Reuse, PLDI 2021, section 2.7.2 of the technical report.
- Choi, Shull and Torrellas, Biased Reference Counting: Minimizing Atomic Operations in Garbage Collection, PACT 2018, doi:10.1145/3243176.3243195.
- PEP 703: making the global interpreter lock optional in CPython.
- Ungar, Grove and Franke, Dynamic Atomicity: Optimizing Swift Memory Management, DLS 2017, doi:10.1145/3133841.3133843.
- Swift: SIL ownership,
lib/SILOptimizer/Transforms/CopyPropagation.cpp, and thetest/SILOptimizer/assume_single_threaded.siltest. - Rust:
std::rc::Rcandstd::sync::Arc. - Nim: memory management and
std/isolation.
Portable SIMD without everyday SIMD bookkeeping
Status: Initial slices implemented; broader design remains proposed. The explicit module now supplies concrete portable lanes, masks, scalar fallback and bounded native lowering. See its implementation evidence for API differences and representation limits. This proposal for B23 was researched against Plenty
7654eccaon 2026-10-23. The current pipeline contract implements serial fusion and bounded scalar callback substitution. The first implemented automatic SIMD slice is integer-column filter/count; the broader API extensions and recognition patterns below remain proposals. This refines the SIMD portion of the earlier memory and parallelism proposal.
Recommendation
Give ordinary numeric code SIMD benefits without making its author select an instruction set, vector width, mask layout, alignment, or tail strategy. Introduce two cooperating facilities:
- A small portable
std.simdmodule for algorithms that genuinely need explicit lanes, comparisons and selection. Follow Rust’s portable semantics and value model, not its entire trait, pointer and intrinsic surface. - A bounded compiler transformation for recognizable fluent pipelines. Prove
that a transformation preserves the existing scalar program; otherwise keep
the scalar program. No
.simd()marker or purity assertion should be required for the safe common cases.
An ordinary pipeline should be the recommended starting point. Explicit SIMD is an advanced tool, not a prerequisite for efficient arithmetic. Compile-time diagnostics can explain missed opportunities without making correctness depend on which optimizer or CPU is selected.
The API contract places bounds, alignment and representation complexity in the library/compiler. The automatic transformation design states exactly what the compiler must prove. The spike evidence includes reproducible native tests and backend comparisons.
Portability is a semantic promise
Public names describe lanes and operations, never SSE, AVX or NEON. A fixed logical vector can lower to one instruction, multiple narrower instructions, or scalar operations. The chosen lowering must preserve the same lane results, bounds behavior and arithmetic failures. “Portable” does not mean every target has equal speed or an implemented Plenty runtime today.
Start with tested 128-bit logical shapes and a scalar implementation of every published operation. Keep masks opaque and vector layout out of the public ABI. Avoid arbitrary lane-count generics, scalable-vector types, native register exposure and CPU multiversioning in the first API. Those are separate design choices, not hidden prerequisites.
Contiguous numeric storage matters more than vector syntax. Existing generic lists are not promised to be packed arrays of lane values. Do not silently copy them into scratch arrays to claim automatic SIMD. Counted ranges can benefit without new storage; packed numeric buffers/views need a deliberate B12/B19 contract before direct vector loads become an everyday list optimization.
Relationship to automatic parallelization
Both transformations need source bounds, visible callback bodies, effect summaries, and a cost model. Share that analysis and explanation machinery with B24 and the fluent pipeline design, but keep the decisions separate. SIMD runs on the current thread and does not introduce scheduling, thread transfer, cancellation, synchronization or thread-pool overhead.
“Serial” continues to mean serial observable semantics: the implementation may evaluate independent, total scalar operations in vector lanes. This clarifies the earlier proposal’s “always serial” wording without enabling implicit parallel workers. Existing explicit pool operations retain their own ordering contract; vectorizing inside a worker is an independent later opportunity.
Cranelift or LLVM?
Keep the API and candidate analysis backend independent. The spike shows that
Cranelift can generate useful 128-bit SIMD for x86-64 and AArch64 when supplied
vector IR. It also shows real limits: direct 256-bit i32x8 was rejected on
both tested targets, scalar CLIF loops stayed scalar, and LLVM produced materially
better code for a simple constant-multiply kernel. See the evidence page for
measurements, tool versions and the limits of that comparison.
Cranelift therefore does not block the initial facility, but fast compilation alone is not a reason to accept persistently poor hardware utilization. Do not build a growing general optimizer in Plenty just to avoid evaluating LLVM. Use these adoption gates before a production implementation commits to a backend:
| Evidence | Decision it should trigger |
|---|---|
| Bounded recognition plus 128-bit lowering reaches useful speedups at low compilation cost | Ship a narrow Cranelift path with scalar fallback |
| A small, semantics-preserving local rewrite closes a measured gap | Add it only with differential tests and a bounded complexity budget |
| Important workloads need wider/scalable vectors, gathers, sophisticated dependence analysis or repeated backend-specific rewrites | Prototype the same typed kernel representation through LLVM; reconsider backend timing |
| LLVM remains substantially faster across representative application kernels after equivalent arithmetic and alias contracts | Consider an optimizing build mode or an earlier primary-backend switch |
| LLVM benefits are isolated, compile latency or packaging costs dominate | Retain Cranelift and explicit specialized kernels; do not maintain two full backends by default |
For the decision, measure cold and incremental end-to-end compilation, optimizer time, executable size, runtime and memory on multiple input sizes and targets. Include bandwidth-bound, reduction, branchy/filter, and small-input workloads. Compare equivalent safety checks, target features, alias assumptions and floating semantics. This spike is a reason to investigate LLVM, not a migration benchmark. The earlier GCC backend investigation is related context, not a dependency of either SIMD approach.
Implementation boundaries and acceptance
B23 in the backlog owns prioritization. A sensible dependency order is a private lane-value representation with scalar oracle, the small public module, then narrowly recognized automatic candidates. Range reductions can be investigated alongside the module; neither depends on automatic threading. Packed-buffer algorithms depend on the storage contract, not on finishing all allocator work.
Before adoption, require native/scalar differential tests, target capability tests, tail and alignment checks, preserved diagnostics, and code-generation evidence. Run the AArch64 cases on hardware before claiming execution support. Teach ordinary pipelines first when the transformation ships, then show explicit vectors only where they clarify a real algorithm. Until then, keep these sketches out of the reference, signature index and executable tutorial.
A small portable std.simd surface
This records the proposed explicit API for portable SIMD. The initial module implements a subset using existing copyable enums, module factories and named operations; its implementation assessment explains differences from this sketch. Packed views, static type factories, operator syntax and fully opaque logical masks below remain proposed.
Values users can understand
Start with concrete types such as simd.f32x4, simd.f64x2, simd.i32x4 and
simd.u32x4. Each is an inline fixed-size value with ordinary copy semantics,
not a heap object or a promise of one hardware register. Qualify additional
integer widths only when their operations and scalar fallback are tested.
Keep a closed initial set rather than making users learn const generics or
backend capability traits. Named aliases can survive a later generic design.
Rust’s Simd<T, N> is a useful model: lane operations, splats, comparisons,
selection and portable load/store behavior. Its full API is not the starting
scope. In particular, do not expose raw pointer vectors, unsafe loads, arbitrary
gather/scatter, dynamic swizzles or architecture-specific intrinsics by default.
| Surface | Proposed contract |
|---|---|
splat(value) | Repeat one scalar into all lanes; allocation free |
Fixed-arity new(...), lane(index) | Construct without allocating a list; lane index is checked |
Floating +, -, *, / | Scalar lane semantics, without implicit fusion/reassociation |
Integer +, -, * | Checked like ordinary Plenty; first implementation may scalarize checks |
Integer wrapping_add/sub/mul | Explicit modular arithmetic, matching existing scalar methods |
eq, lt, le, etc. | Produce an opaque lane mask, not one boolean |
mask.select(yes, no) | Select already-computed lanes; this is not lazy conditional execution |
mask.any(), mask.all(), mask.count() | Boolean/count summaries; no exposed integer encoding |
load(view, offset), store(view, offset) | Check a complete vector fits; accept scalar-aligned memory; no overread |
wrapping_sum() on integer vectors | Explicit modular reduction, independent of grouping |
sum_ordered(initial) on float vectors | Scalar left fold in increasing lane order; no speedup promised |
Each comparison mask is compatible only with vectors of the corresponding lane count/shape; there are no implicit integer-mask casts. Avoid operator-overloading ambiguity around equality returning a mask: explicit comparison methods suffice. Mask bit representation, alignment and calling convention are private.
Keep Plenty’s arithmetic contract
Rust SIMD integer operators wrap. Plenty’s ordinary integer operators trap on overflow, so copying Rust’s operator behavior would make changing a scalar into a vector silently change the algorithm. Keep checked operators, expose wrapping by its existing name, and allow checked vectors to use scalar fallback initially. For an explicit checked operation, report the lowest failing lane and operation site. Do not publish partially written stores after a failed bounds check.
No initial integer division, shifts with new count rules, saturating casts, approximate reciprocal, fused multiply-add, or min/max with underspecified NaN behavior. Add each only with a complete scalar contract and target tests. Explicit fused or relaxed operations can be considered later; their names must make the semantic difference visible.
Floating lanes follow Plenty’s scalar rules for signed zero, infinities, NaNs and subnormals on a supported target. Do not promise bit-identical NaN payloads across platforms beyond the scalar contract. Rust documents subnormal flushing on some older architectures; Plenty must either provide an appropriate fallback or document the target’s scalar contract before enabling SIMD there. Do not silently weaken scalar semantics to accommodate an instruction.
Bounds, tails and ownership belong below the algorithm
Safe load/store accept the eventual checked contiguous numeric view, retaining its owner and borrow lifetime. No caller-supplied alignment proof is needed. Loads do not allocate; stores require an exclusive mutable view and validate all destination bounds before writing. The initial C ABI uses pointers/lengths and scalar elements, never a vector passed by value.
For manual kernels, provide a library block-view helper once packed views exist:
conceptually simd.blocks4(view) splits a borrow into full four-lane blocks and
a scalar tail. Each block has a safe load()/store() pair. The helper checks
the split once, does not allocate or create overlapping mutable loans, and
requires the tail to be handled explicitly. This is an ergonomic target, not
an implemented borrowing API. Do not require every user to write alignment
prefixes or masks for ordinary remainders.
Prefer full blocks plus scalar tail over padded masked evaluation initially. Padding memory safely is insufficient if inactive lanes divide by zero or trap on a cast. A future masked-load API needs an explicit inactive value and memory contract; selection alone does not suppress evaluation of either operand.
For everyday algorithms, automatic pipeline lowering performs the split and tail handling invisibly. Users reach for explicit vectors only when lane relationships are part of the algorithm. If common examples require repeated mask and tail bookkeeping, improve the safe helper or the automatic pattern before broadening the intrinsic catalog.
Target policy and explanations
Use a documented deployment baseline by default. A native-target build may require newer CPU features, but that is a build contract, not a different source API. Runtime dispatch, feature probing and multiple compiled variants remain separate extensions with code-size and startup costs.
Lack of a SIMD instruction must never make a valid portable program fail to compile: split or scalarize. A vector IR type’s existence is not a support check. Query a tested operation/type capability table before lowering and exercise its fallback path in tests. A backend compilation error should be a compiler defect, not the user’s mechanism for selecting vector widths.
Initial explicit SIMD implementation
This implements the first std.simd module,
following the API proposal with deliberate limits. It is separate from
the compiler’s packed-column automatic SIMD.
API and representation decision
Rust’s local source checkout at 080cc271463 supplied the model, particularly
library/portable-simd/crates/core_simd/src/vector.rs, masks.rs, and ops.rs:
logical lane values, splats, simd_* comparisons and mask selection. Plenty
preserves checked integer arithmetic under add/sub/mul; its wrapping_*
methods explicitly request Rust-style modular lane arithmetic. No Rust source
was copied into the implementation.
The first types are concrete i32x4, u32x4, f32x4, f64x2, mask32x4, and
mask64x2. They use existing copyable inline enums, so ordinary ownership,
borrowing, aggregates and function boundaries work without new syntax or a new
runtime ABI. Module splat/load factories and the Lanes constructor replace
Rust’s static methods. Logical mask booleans are public; native mask encoding
is not. Safe list loads/stores assemble ordinary list elements and require a
full vector, rather than promising packed numeric list storage.
Every published operation has an ordinary Plenty scalar body. The optimized x86-64 path substitutes only trusted embedded methods with exact signatures, using a closed table of 128-bit operations supported by the baseline backend. Other hosts retain the scalar bodies. This adds no supported target or runtime ISA dispatch. Checked integers and ordered float folds remain scalar.
Integer wrapping_sum lowers directly to lane extraction and modular adds.
This avoids both canonical argument materialization and an ABI call at loop
exits; the second accumulator in separate reductions no longer has to remain
live across the first reduction’s call (issue #444).
Keeping local values in registers
A first prototype emitted vector instructions but materialized every result as four tagged scalar slots. Its integer recurrence took about 172 instructions per iteration versus 13 for four scalar accumulators, and roughly 28 times the elapsed time. That was an architectural failure, not an acceptable SIMD result.
The implementation therefore promotes exact numeric-vector locals to private vector SSA variables. Parameters, generators, masks, and any local that is mutably borrowed or moved retain canonical storage. Shared borrows materialize valid snapshots; the ownership checker prevents source mutation during the loan. Fallback calls and container operations still receive normal enum values.
Cranelift conservatively retains stores to unused stack slots. The lowerer records its private snapshot instructions, roots snapshots whose SSA results reach any ordinary instruction, propagates dependencies between snapshots, and removes only unobserved groups. Calls, returns and block arguments are all uses. The vector-producing arithmetic is outside these groups. Thus supported local chains and loops retain vector values without repeatedly scattering lanes. This is bounded to the explicit module, not a general escape optimizer.
Externally stored four-lane values still have 64-byte tagged payloads plus the ordinary handle. The public API promises logical lanes and no allocation, not a packed 16-byte representation, one register, or a foreign vector ABI.
Evidence and limits
The reproducible experiments in experiments/simd/explicit/README.md in the repository
exercise checked results, dependency chains, and throughput on the same host.
They retain unfavorable cases rather than selecting only wins. On the initial
Ryzen 7 7840HS runs, eliminating materialization reduced the integer recurrence
to approximately 14 instructions per iteration. Its SSE2 multiply sequence was
still slower than four scalar accumulators. A single four-lane float recurrence
was close to scalar speed; four independent vectors of divide/add work showed
a useful throughput gain. Retained final timings and counter samples live with
the experiment.
The integer multiplication assessment isolates baseline instruction synthesis, dependency chains, newer-ISA behavior and an independent exit-shape spill limitation with the same integer workload.
The module is useful for explicit lane algorithms, but it does not make generic lists packed or remove all ABI costs. Measure the actual algorithm. Packed views, wider vectors and more capable target lowering remain separate decisions; the implementation does not implicitly relax floating-point order to improve a benchmark.
Differential tests cover integer endpoints and all comparisons, modular and checked arithmetic, distinct float lanes, NaNs, signed zero, subnormals, mask selection, full-fit bounds, no partial stores, nested values, copy independence, mutable projections/captures, shared references, loops/joins and scalar fallback. Code-generation checks require baseline vector arithmetic while checked integer operations remain scalar.
Why the integer multiply probe is slower
The initial module keeps the recurrence in vector registers, but baseline x86-64 lacks a single instruction for all four low 32-bit products. The original probe compares four independent scalar accumulators with one vector accumulator. Both compute exactly the same four modular multiply/add recurrences.
What actually executes
Optimized CLIF contains imul.i32x4 followed by iadd. Cranelift 0.137.0’s
isa/x64/lower.isle (lines 1168–1183, source commit
ebe5e8df1f4e12c5fc2f5954926881d8d61367d8) selects this baseline sequence:
pshufd value, odd-lane-order
pshufd factor, odd-lane-order
pmuludq original-value, original-factor
pshufd even-products, low-halves
pmuludq odd-values, odd-factors
pshufd odd-products, low-halves
punpckldq even-low-products, odd-low-products
paddd increment, result
The scalar loop has four independent imul plus add/lea pairs. The vector
loop has shuffle, multiply, shuffle, interleave and add dependencies before its
next iteration. Similar instruction totals therefore do not imply similar
critical paths. The hot loop has no tagged-lane gathering or scattering. Its
one invariant stack load is bookkeeping; canonical vector construction and
final extraction are outside the loop.
An experimental compiler enabled only has_sse41; the production compiler
still uses its existing baseline. The same CLIF then becomes pmulld plus
paddd. Intel documents PMULLD as the SSE4 operation retaining each product’s
low 32 bits; Rust’s matching intrinsic requires SSE4.1. See the
Intel intrinsic reference, page 136
and Rust intrinsic implementation.
Same workload, controlled comparisons
Final measurements on the Ryzen 7 7840HS use one million iterations per timing sample and ten million for counters. The numbers below are median user-space cycles per iteration from three counter runs, including small process overhead; they are observations for these loops, not instruction-latency specifications.
| Kernel | Scalar | Baseline SIMD | Experimental SSE4.1 SIMD |
|---|---|---|---|
| Four lanes, constant factor | 4.08 | 7.12 | 4.06 |
| Four lanes, runtime factor | 4.07 | 7.11 | 4.05 |
| Eight lanes, two independent vectors | 8.09 | 8.17 | 4.27 |
The runtime-factor variant removes scalar immediate-multiply specialization; its result is essentially unchanged. Two independent vectors hide more of the dependency cost: baseline throughput reaches scalar parity, and SSE4.1 is about 1.9 times faster. A single SSE4.1 vector only reaches parity with four scalar chains. Fewer instructions alone are not a speedup guarantee.
Instructions per iteration for the runtime-factor case are approximately
13/14/8 for four lanes and 21/22/10 for eight lanes (scalar/baseline/SSE4.1).
All variants check identical modular results. Reproduction sources, raw samples,
counter output and load context live in the repository’s
experiments/simd/explicit/ directory; start with its README.md.
The original exit-shape spill and its fix
The first eight-lane variant finished with
v0.wrapping_sum().wrapping_add(v1.wrapping_sum()). Its generated loop spills
one vector with a movdqu load/store pair, costing about 16.2 baseline cycles
or 13.0 SSE4.1 cycles. Changing only that final expression to
v0.wrapping_add(v1).wrapping_sum() removes the spill and yields the table above.
This rearrangement is valid for modular addition; it is not a license to
reassociate checked or floating arithmetic.
The spill-free loop uses only six XMM registers. Issue #444 traced the important distinction: the second vector remained live across the first reduction’s scalar ABI call and caller-location hooks. On the tested System V target, XMM registers are caller-clobbered, and the allocator placed that vector’s spill across the loop. The combined reduction had no second live vector at its single reduction call.
The fix lowers the existing integer wrapping_sum directly to lane extraction
and modular additions. It removes the reduction calls and their argument
materialization, so the original separate-sums spelling also keeps both vectors
in registers. It does not change floating reductions, arithmetic order, ABI or
ISA policy, or claim to fix general register-allocation heuristics. The
integer_spill.plenty probe retains the original source shape; before/after
measurements are in spill-measurements.json in the experiment directory.
With the baseline ISA unchanged, the separate-sums kernel improves from about
16.2 to 8.1 cycles per iteration (3.369 to 1.676 ms per million iterations),
reaching scalar parity. The single-vector multiply dependency cost remains.
The recommendation is to accept the portable baseline with these measured limits. The original slowdown is explained by its instruction set and dependent chain, rather than lingering tagged storage in that hot loop. Newer target selection can improve throughput, but requires a separate explicit compilation or dispatch policy. It should not be silently enabled to make this benchmark win.
Automatic SIMD for recognizable pipelines
Part of the portable SIMD proposal. This is a bounded proposed transformation, not implemented auto-vectorization. Fluent pipelines already preserve source and callback structure; the compiler should use that information before it disappears into low-level control flow.
Start with code people already write
These are current Plenty spellings; the proposed optimization changes only their implementation:
total = iter(range(0u32, 1000u32)).map(
def(x: u32) -> u32: x.wrapping_mul(3u32).wrapping_add(7u32)
).fold(0u32, def(acc: u32, x: u32) -> u32: acc.wrapping_add(x))
even = iter(range(0u32, 1000u32)).filter(
def(x: &u32) -> bool: x & 1u32 == 0u32
).count()
The first has a pointwise modular map and associative modular reduction. The
second needs a comparison mask and a count, not a compacted intermediate list.
Neither needs heap allocation, a thread pool, or a new annotation. The checked
i64 count is safe only when the source bound proves it fits. Guarded induction
must avoid overflowing while constructing the final vector of range values.
| Pattern | Admission rule and initial disposition |
|---|---|
| Sized range → total scalar maps → wrapping integer fold | Recognize exact modular reduction; vector accumulators plus ordered scalar remainder |
| Sized range or packed read-only view → bitwise/comparison filter → count | Mask count; bound the result to i64; no materialized filtered collection |
| Packed numeric input → total pointwise map → distinct preallocated destination | Vector loads/stores once views and exclusive destination proofs exist |
| Equal-domain zip of packed sources → pointwise map | Later bounded extension; preserve left-first consumption and prove both lengths before evaluation |
skip/take on a sized pure source before maps | Normalize bounds only when it preserves evaluation and count rules |
| Floating left fold, ordinary checked integer fold | Preserve order; exclude reassociation without a proof or a separately specified operation |
filter(...).map(...), filter(...).collect(), take_while, any, find | Defer: evaluation suppression, compaction or early stopping needs additional reasoning |
| Generators, indirect calls, I/O, mutable captures, allocation, ownership cleanup, atomics | Scalar fallback in the first implementation |
A recognized floating map can preserve each element’s multiply-then-add order; a floating reduction cannot freely change its accumulation tree. Even ordinary integer addition is not freely associative under overflow trapping: a sequential prefix may overflow although a regrouped sum fits. Do not infer safety from a final result bound alone.
Proof before profitability
For each candidate establish all of these facts:
- Source: finite length, element type, traversal order and storage stride are known. Prove each full-block access lies within the source; use scalar tails. No speculative reads into a later page or adjacent object.
- Body: a bounded visible expression graph uses only allowed total scalar operations and immutable scalar captures. Reject unknown calls rather than trusting a user purity assertion. Purity alone does not establish totality.
- Effects: no observable callback order, writes through aliases, allocation, cleanup or failure can move across a block boundary. Preserving only the successful result is insufficient.
- Terminal: a specific reduction law or destination layout is known.
A function named
addis not evidence of associativity. Match typed operations after substitution; retain scalar fallback for arbitraryfoldcallbacks. - Target and cost: every widened operation has legal lowering or a useful fallback. Trip count, load stride, scalar remainder, instruction expansion and code size justify the transformed loop.
Known-length collect is not automatically admitted. It allocates, may reserve
or grow, and has ownership/failure behavior. First implement preallocated numeric
destinations or nonallocating reductions. Later preserve the existing reservation
point and capacity policy before changing its inner loop; do not introduce an
infallible hidden temporary array.
Checked element maps could later use a vector failure mask and replay the first failing block scalarly, but only when replay has no effects and reproduces the first source operation and item that fail. This is substantially harder than wrapping arithmetic and is not an initial fast path. Never run speculative trapping instructions and then attempt to recover after selection.
Placement in the compiler
Current src/frontend/pipelines.rs builds an Iteration body while adapting a
pipeline; it does not retain an optimizer-ready list of typed stages. Extend the
plan with bounded source/adapter/terminal metadata instead of reconstructing
source expressions from arbitrary CLIF loops. Keep a scalar lowering as the
semantic oracle. Preserve ownership, loan and cleanup validation before any
optimization decision.
Small callback substitution already exists in src/codegen/leaf_calls.rs,
small_calls.rs and scalar_captures.rs. Reuse its bounded body visibility,
but do not confuse “eligible for substitution” with “nontrapping and safe to
speculate.” A separate whitelist/effect summary must certify widening. Analysis
may happen after specialization, with plan identity and source sites retained
through substitution. Cache keys must include the transformed plan, target
features and optimization policy.
Lower an admitted plan into a backend-neutral kernel description: logical lane
operations, source bounds, output/reduction policy, scalar tail and original
sites. Cranelift can receive explicit vector IR; LLVM can receive equivalent
scalar loops with truthful alias and arithmetic facts, or explicit vectors.
No noalias, nsw or nuw annotation may be guessed to coax better code.
Keep the analysis bounded by chain/body size; do not add whole-program alias analysis or arbitrary loop-dependence solving as a prerequisite. Share summaries with automatic parallelization, but neither optimization grants the other’s safety proof.
Make behavior easy to understand
Optimization misses are not errors. Offer an opt-in report with the source span and one actionable reason, for example:
SIMD: map/filter-count uses four lanes; scalar tail for remaining items
SIMD skipped: this callback can trap on integer overflow
SIMD skipped: ordered floating fold must preserve accumulation order
SIMD skipped: source layout does not provide contiguous numeric lanes
SIMD skipped: seven items are below this target's profitable threshold
The report and a compiler option to disable automatic SIMD are debugging tools,
not new language semantics. Do not require users to read IR or choose an ISA to
understand a missed optimization. Explicit std.simd remains available where an
algorithm needs lane control, with the same portable contract and scalar fallback.
Acceptance includes empty and short inputs, every remainder, range boundaries, unaligned views, overflow and failure ordering, NaNs and signed zero, callbacks with effects, zip length mismatch, scalar fallback, and disabled-optimization equivalence. Correct results alone are not evidence of SIMD: inspect generated instructions and benchmark enough sizes to reveal overhead and memory limits.
SIMD spike evidence and backend decision
Evidence for the portable SIMD proposal, collected on 2026-10-23. These experiments do not implement a Plenty vectorizer or the proposed module. They separate current-language semantics, explicit backend lowering, and another optimizer’s recognition of equivalent scalar loops.
This page retains the original 0.131.1 experiment. The implemented clist slice has separate Cranelift 0.137 validation and current measurements; use those for the production backend’s present limits.
Reproduce
The checked-in experiment (experiments/simd/README.md in the repository) contains a
locked isolated Rust package, C comparison/verification harness, a Plenty
pipeline program, and a runner that preserves objects, assembly, compiler
remarks, exit status and command timings. No changes to the production compiler
or runtime are required.
Measured environment: AMD Ryzen 7 7840HS, Linux x86-64; Rust 1.99.0
(b940084d7, LLVM 23.1.1); Clang 22.1.8 (Fedora 22.1.8-4.fc44);
Cranelift codegen/frontend/module/object 0.131.1. The isolated lockfile
resolves some Cranelift helper crates to 0.131.3; it is separate from Plenty’s
root lockfile. Plenty was freshly built from the task worktree based on
7654ecca. Default Clang configuration is disabled. Both comparisons use
baseline targets, not -march=native; no fast-math or floating contraction.
Current Plenty chains
pipelines.plenty runs these actual fluent chains:
| Input and terminal | Result | What it establishes |
|---|---|---|
range(0u32, 1000u32) → wrapping multiply/add map → wrapping fold | 1505500 | Current syntax and scalar modular oracle for a narrow recognition candidate |
Same range → even predicate → count() | 500 | A filter need not materialize or compact data when the terminal is a count |
Borrowed [1e20f32, 1.0f32, -1e20f32, 1.0f32] → floating left fold | 1.0 | Reassociation changes observable results; a regrouped computation gives 2.0 |
The worktree compiler produced 1505500 500 1.0, checked by the runner.
This validates scalar semantics and realistic input patterns, not recognition
or a speedup inside Plenty. The C/CLIF kernels below use packed arrays, whereas
these Plenty range pipelines need no array and a general Plenty list does not
promise that representation. Bridging the typed plan to vector IR remains work.
Native execution and generated code
The Rust probe builds scalar and four-lane CLIF loops for x * 3 + 7 on u32
and f32, at opt_level=speed. Both use the same loop construction; the
four-lane version is supplied explicitly. The C wrapper runs full blocks then
the scalar remainder. The C comparison gives LLVM ordinary loops over restrict
input/output pointers. These pointers are in fact disjoint in the harness.
Native x86-64 checks passed for:
- Integer lengths 0–129, all four-lane remainders, an input offset by one scalar from malloc alignment, modular overflow, and an output sentinel after the end.
- Floating lengths 0–9 containing signed zeros, infinities, NaN and minimum subnormals. Finite results agree bitwise; NaNs are compared by classification.
- LLVM filter-count results against a scalar count and the ordered floating fold.
These are focused probes, not exhaustive memory-safety tests: a sentinel detects overwrites, not every possible overread, and sanitizers cannot automatically instrument the generated CLIF object. Guard-page tests and fuller floating coverage are adoption requirements. The generated kernel requires a lane-multiple length; only its wrapper accepts arbitrary lengths.
| Probe | x86-64 | AArch64 |
|---|---|---|
| Scalar CLIF arithmetic map | Executed; scalar integer/mulss/addss | Object compiled only |
| Four-lane CLIF map | Executed; packed integer operations and mulps/addps | Object compiled only |
| Eight-lane CLIF map | Rejected: Unexpected SSA-value type: i32x8 | Same rejection |
| LLVM scalar-source maps | Vectorization width 4; packed arithmetic verified | Assembly shows four-lane arithmetic |
| LLVM scalar-source filter-count | Vectorized; executed and checked | Vectorization remark and emitted assembly only |
| LLVM strict floating fold | Scalar ordered adds; executed | Remark says vectorized, but assembly widens loads and retains ordered scalar adds |
The eight-lane probe stops at its first integer kernel, so it does not independently
test f32x8 or all target feature settings. AArch64 was not executed. No claim
of a production Plenty AArch64 runtime follows from successful object emission.
The floating-fold example shows why an optimizer remark alone is insufficient:
vector loads do not imply a reassociated vector reduction.
Small runtime comparison
Integer map, nanoseconds per item, best of five trials. Each trial processes roughly 16 million items through repeated calls; output contributes to a volatile sink. Allocations and compilation are outside the timed region.
| Items per call | Cranelift scalar | Cranelift four lanes + tail | LLVM from scalar source |
|---|---|---|---|
| 7 | 0.533 | 0.544 | 0.448 |
| 1,024 | 0.420 | 0.171 | 0.062 |
| 262,144 | 0.419 | 0.159 | 0.070 |
For the larger sizes, explicit SIMD improves the CLIF loop by roughly 2.5–2.6×; LLVM is another roughly 2.3–2.8× faster in this kernel. For seven items, vector setup and the tail erase the benefit. Earlier runs varied by several percent, with more variation for tiny calls. The CPU was not pinned and frequency was not controlled; best-of-five suppresses interference but is not a confidence interval. This is a hot repeated microbenchmark, not a cold-cache bandwidth test or representative application performance claim.
Assembly helps explain the gap: baseline CLIF expands four-lane multiplication
into pmuludq plus shuffles, and reloads constants in the loop. LLVM strength
reduces multiplication by three into adds and interleaves two vectors. This
compares the output of each optimizer on the straightforward equivalent kernel,
not matched instruction sequences. A local strength reduction or stronger
target baseline might narrow the gap; neither was measured here. It would be
premature to infer a general backend ranking from one arithmetic map.
The runner also records process elapsed times: the warm Cranelift object-emission invocations were around 1–3 ms for two tiny functions, while Clang invocations were tens of milliseconds for four C functions. These include process startup and different frontend/object work. They are not a controlled compilation latency comparison. A backend decision needs matched IR and end-to-end Plenty incremental/cold compilation measurements.
Source evidence and consequences
Rust checkout ../rust/ was inspected at
080cc27146389dbb5c8772c476b1937be9c6110f, under
library/portable-simd/crates/core_simd/src/:
core_simd_docs.mddescribes architecture-independent semantics, scalar lowering and older-target subnormal exceptions.vector.rsdefinesSimd<T, N>, documents wrapping integer operators, and implements checked/fallback and masked loads/stores.mod.rsseparates numerical, comparison, pointer and mask facilities. Plenty can adopt the useful value model without exposing the entire surface.
Installed Cranelift 0.131.1 source in src/isa/x64/lower.isle and
src/isa/aarch64/lower.isle contains vector arithmetic lowering, including
F32X4 and vector multiplication rules. Source inspection motivated the probe;
actual object emission is stronger evidence for these exact tested operations.
It is not evidence for every instruction on every vector type.
Plenty’s src/frontend/pipelines.rs, src/codegen/leaf_calls.rs,
src/codegen/small_calls.rs and src/codegen/scalar_captures.rs establish the
existing fusion and bounded substitution paths. src/codegen/collections.rs
still routes scalar list access through runtime helpers, so callback substitution
alone cannot provide contiguous bulk loads. The current
number contract keeps checked
arithmetic distinct from wrapping methods.
The evidence supports a small portable layer and narrowly recognized pipelines. It also gives a concrete reason to evaluate LLVM early if broader hardware utilization becomes a requirement. It does not justify growing an unbounded Plenty optimizer, promising auto-SIMD for arbitrary chains, or switching backends before a representative comparison.
Columnar lists with an object-shaped interface
Status: Initial primitive-field implementation. See the current reference for the supported interface. It uses packed native columns and borrowed, scalar-valued local pipelines. The original generated Plenty spike remains historical evidence. Extensions in this proposal are not implied by the initial implementation; B76 tracks remaining candidates.
Recommend an explicit clist[T] storage choice for repeated scans of a few
fields of many records. Keep ordinary class declarations and ordinary
list[T] unchanged. Start with flat, fixed-width scalar fields and a small
indexed-field/column interface; earn a richer object facade through separate
ownership and code-generation evidence. Do not require an ECS, dataframe
query language, automatic parallelism, or a new backend to try this idea.
class Particle:
x: f64
y: f64
mass: f64
kind: i64
mut particles = clist[Particle]()
particles.append(Particle(1.0, 2.0, 3.0, 0))?
particles[0].x = 4.0
This is a storage decision at the collection declaration, not a change to
Particle everywhere. The empty owner requires no allocation; the first append
is fallible. Append errors retain the submitted row; see the
ownership and API rationale.
The original layout evidence and Plenty implementation spike support the narrow-field use case and expose tradeoffs. The Plenty experiment establishes bounded scalar scan benefits with ordinary list-backed columns, not packed primitive storage, a final chunk size, or safe general row references. The native implementation follow-up records the packed representation and its comparison with that baseline.
What changes in memory
For four particles, ordinary array-of-structures (AoS) storage groups each record’s fields. Structure-of-arrays (SoA) groups each field’s values:
AoS: [x0 y0 m0 k0] [x1 y1 m1 k1] [x2 y2 m2 k2] [x3 y3 m3 k3]
SoA: x: [x0 x1 x2 x3] y: [y0 y1 y2 y3]
m: [m0 m1 m2 m3] k: [k0 k1 k2 k3]
Scanning mass then brings useful values into cache without the other fields.
This concerns cache lines, prefetching and memory bandwidth; it need not involve
OS paging. Contiguous columns also make ordinary scalar loops simpler and give
vectorization a better input representation. Neither SIMD nor multithreading
is implied by choosing the layout.
Whole-row access is not uniformly slower. Sequential full-row work can stream several columns efficiently, and compiler vectorization can change the result. Random whole-row access often needs several unrelated cache lines. Appending touches more buffers but can remain amortized constant time; deletion costs depend on ordering and identity promises. Measure the actual workload rather than naming a universally superior layout.
Plenty’s current runtime already keeps inline class payloads in type-sized
collection rows (plenty-runtime/src/entries.rs); it is not simply a list of
heap-object pointers. Its u128 storage words and bookkeeping also differ
from tightly packed native structs. A clist implementation must decide how
packed scalar columns cross that representation boundary. Reusing eight
ordinary lists is a useful semantic prototype, not evidence of packed columns.
Learn from adjacent systems
| Precedent | Useful part | Boundary for Plenty |
|---|---|---|
| Arrow columnar format | Struct arrays have equal-length child arrays; primitive columns support sequential scans | Arrow is an in-memory interchange layout, not an ownership checker. Adopting SoA alone does not give Arrow-compatible strings, validity bitmaps, alignment or zero-copy interchange. |
| Parquet row groups | Divide rows into groups, each containing a chunk per column | Parquet describes a file format. Its disk-oriented group sizes and compression/page machinery are not defaults for an in-memory container. |
| Flecs archetypes | Group entities with the same components for efficient iteration | Component columns need not split each component’s fields. A fixed-schema clist needs neither dynamic archetype migration nor an entity scheduler. |
| soa-rs | Separate generated row-reference and column-slice types from owned rows | Its references contain references to individual fields. Plenty currently forbids stored aggregate references; this cannot be copied as an ordinary library class today. |
| DuckDB zonemaps | Conservative min/max bounds can skip groups, especially when values are clustered | Arbitrary Plenty callbacks can have effects and traps. Skipping their execution needs a recognized safe predicate or an explicit query operation. |
Small initial scope
The initial candidate is a compiler-recognized collection for a fully concrete class whose fields are integers, floating-point numbers or booleans, with no custom destructor. Normal construction may happen before insertion; clist does not rerun a constructor per field. Custom constructors do not automatically justify exposing otherwise private fields. Exclude nested aggregates, text, collections, owned closures and recursive fields until their movement and destruction contracts are proved.
Provide length, fallible reserve/append, scalar field reads and writes,
explicit whole-row materialization/removal, and a typed immutable column
projection feeding existing fluent operations. Do not add a second family of
map_field, filter_field, and reduce_field methods. The detailed
contract makes the incomplete facade visible.
Start with dense flat columns. Preserve insertion order; use explicit ordered
removal and defer cheap tombstones. Keep physical storage private so fixed-size
chunks can be evaluated later without changing the element type. The
chunking and deletion assessment records how to
extend it without turning clist into a database prematurely.
Fit with pipelines and SIMD
Column projections should be borrowed sources for the existing
fluent pipeline model, with zip for multi-column work.
The source carries type, length, stride and ownership information; callbacks
keep ordinary evaluation order, traps and allocation behavior. A projection
must not allocate a temporary list. The first implementation may use scalar
loads and still benefit from fewer fetched bytes.
The SIMD proposal can later recognize safe kernels over these sources. Integer overflow must retain Plenty’s checked semantics unless the program explicitly uses wrapping operations; floating reductions stay ordered. Contiguous storage does not authorize reassociation, speculative callback execution, or automatic threads. The original layout experiment uses LLVM-generated native kernels; the implementation follow-up uses Plenty and Cranelift. Neither is a controlled backend comparison. The existing SIMD evidence remains the backend comparison. Avoid building a dataframe optimizer inside Plenty simply to compensate for a backend limitation.
Adoption gate
There is no demonstrated fundamental layout blocker. There is a substantial
semantic boundary between cheap field access and a transparent list[T]
replacement. Before accepting production implementation, require:
- A small generated layout path whose scans touch only selected columns, with bounded compile-time and code-size cost per class specialization.
- Scalar differential tests against ordinary lists, including empty/tail inputs, checked arithmetic, field mutation and ordered removal.
- Allocation failure at every required reserve, unchanged logical contents, exactly-once destruction, and useful unsupported-element diagnostics.
- Compile-fail borrowing tests for mutation/reallocation with live projections, private-field access, whole-row borrowing and callback escape.
- End-to-end Plenty benchmarks for narrow and full-row work, growth and deletion, including small inputs and at least one non-x86 execution target before making portable performance claims.
The proposal is a reason to try that bounded implementation, not a commitment to the later row facade, stats, vacuum, or interchange APIs.
Columnar-list ownership and API boundary
This records the original proposed contract for columnar lists. The implemented subset uses scalar-valued column pipelines and ordered removal, and does not expose packed field references. The alternatives below remain design rationale.
An honest object facade
items[i].field can lower to a field-column address at row i, evaluating the
index once and preserving normal bounds diagnostics. It need not create an
object or a proxy allocation. A full &items[i], however, cannot be an ordinary
&T: there is no contiguous T at that location. This is an ABI and lifetime
fact, not something a clever overload can hide.
For the initial version, support direct scalar field reads/writes and explicit
copy_row(i) -> Option[T], which reconstructs an owned class from scalar fields
without allocation. This is a deliberate duplication operation even though
classes themselves remain affine. An out-of-range index returns Nothing.
The copied class can be passed to ordinary methods; changes to it do not write
back. Do not implicitly gather/scatter around a method call: methods may return
borrows, invoke callbacks or exit early, and hidden writeback changes observable
semantics. Plain items[i], ordinary whole-row borrows, and direct row-method
calls should give a diagnostic pointing to field access or copy_row.
A later local row binding could allow for row in &items: row.mass, using an
ephemeral compiler-known projected place, not a stored T or &T. That would
need its own call, reborrow, escape and generator-suspension rules. Existing
local pipelines show that restricted compiler-known values are possible, but
do not implement this row representation. Treat general row iteration and
method adaptation as separate work rather than presenting them as free syntax.
Candidate minimal operations
| Operation | Meaning |
|---|---|
clist[T]() | Empty owner; no column allocation until needed |
len(items) | Live row count, identical for every column |
items.reserve(additional) | Fallible capacity for that many additional rows; no logical change on failure |
items.append(value) | Move one complete T; return Result[(), AppendError[T]], retaining the intact submitted row on failure |
items[i].field | Checked scalar access, with ordinary field visibility; negative indices follow list conventions |
items[i].field = value | Replace one field; keep the index and RHS evaluation order consistent with existing assignment |
items.copy_row(i) | Allocation-free Option[T] for the initial scalar-only class subset |
items.remove(i) | Option[T]; materialize and transfer the removed row, shift all columns together, preserve order |
items.column(T.field) | Borrow a typed immutable sequence of that field’s values, without allocating |
T.field in the last row would be a new compile-time field selector, not a
runtime string lookup or arbitrary class-variable expression. It must carry
the owning class and field type and respect visibility at the use site.
The compiler already knows class fields, but it does not expose selectors or
generic reflection. This proposal requires frontend support rather than a
magic generic library implementation. Concrete generic class instances may be
eligible after checking all instantiated field types; no runtime schema changes.
For example, this proposed pipeline uses scalar field references, not a fabricated class reference:
def add(total: f64, mass: &f64) -> f64:
total + *mass
total = iter(particles.column(Particle.mass)).fold(0.0, add)
Extend existing Iter source lowering for a read-only column view. Keep it
local, borrowing the owner until consumption; do not require a general stored
reference feature first. Two immutable column views can feed zip; they have
the same length and row order. No public raw slice pointer or alignment promise
is necessary. If chunks arrive later, a column view remains a logical sequence,
not a promise of one allocation spanning all rows.
Loans and mutation
Initially, any live projection or field reference protects the entire clist
against structural changes: append, reserve, removal, clear, movement and drop.
Do not advertise stable addresses merely because a chunk currently has room.
Scalar field references may use genuine &Field/&mut Field because that
individual field has contiguous storage. Begin with conservative owner-wide
conflicts; field- or chunk-disjoint mutable views require a separate proof.
Even two integer indices do not create simultaneous mutable access by themselves.
Column views cannot escape into stored classes, enums, collections or suspended generators. Shared access from scoped workers must obey existing worker eligibility rules; the initial proposal does not add parallel mutation or automatic partitioning. Row positions are not entity identities: ordered removal shifts later indices, just as in a dense list.
Allocation and ownership transaction
An append reserves every affected buffer before moving any field out of the input row. Only after all reserves succeed may it scatter fields and publish the new length. The commit phase must be infallible and call no user code. If a later reserve fails, earlier capacity increases may remain; logical contents, length and input ownership must not change. Overflow when computing byte capacities follows the allocation-error contract. Never use independent fallible pushes after publishing the first column.
The failed append’s typed error owns the complete T, permitting retry or
explicit destruction. A convenience that discards it can be evaluated later;
do not silently lose an affine owner. The Rust spike checks this transaction
boundary with injected reserve failures; it is not a proof of Plenty’s future
allocator implementation.
The scalar-only restriction makes movement and row reconstruction simple. Supporting affine fields later requires compiler-owned field extraction that does not introduce unrestricted user partial moves. It must transfer/drop each field exactly once. Custom class destructors need a whole-row receiver and defined ordering relative to fields; initially reject such classes instead of silently skipping or duplicating their destructor.
Storing str handles in a column would make handles, not string bytes,
contiguous. Packing variable-width payloads needs offsets, invalidation and
mutation contracts of its own. Nested lists, nullable values, enums and recursive
data likewise do not automatically become Arrow arrays. Conversion to/from
ordinary lists or Arrow would be explicit and potentially allocating; no FFI
layout, zero-copy conversion or serialization compatibility is promised.
Chunks, deletion, and statistics
These are extensions considered for columnar lists. The initial candidate stays dense and flat. Keeping these choices separate lets early measurements answer whether the basic storage interface is worthwhile.
Fixed-size chunks
A chunk contains the same bounded number of rows in every field column. A 1,024-row chunk of eight 64-bit fields holds 64 KiB of payload; a 4,096-row chunk holds 256 KiB. Those are experiment sizes, not chosen public defaults. A scan walks the selected column of each chunk and then advances to the next chunk.
Chunking can bound relocation, permit incremental allocation and release whole empty chunks. It also adds a directory, allocation count, tails, address calculations and transitions between buffers. A single aligned chunk allocation with generated column offsets is an alternative to one allocation per field; the spike uses the latter and does not settle the former. Tune byte footprint as well as row count, since class width varies.
Parquet’s row-group model is a useful analogy for grouping rows across columns, but its pages are encoded file data. In-memory chunk size should follow workload/cache measurements, not Parquet’s disk-oriented recommendations. The measured flat SoA scans beat both tested chunk sizes; chunking needs a growth, mutation or pruning benefit to earn its cost.
Deletion is an identity decision first
| Policy | Benefit | Cost or observable change |
|---|---|---|
| Ordered dense removal | Familiar list positions and iteration | Shift later values in every field |
| Swap with last row | Constant-sized transfer | Changes order and the moved row’s position |
| Tombstones | Avoid immediate relocation | Holes complicate indexing, scans and storage reclamation |
| Generational handles | Identity can survive relocation | Mapping storage, stale-handle checks and generation exhaustion |
Do not give clist[i] both dense-list and physical-slot meanings. A tombstoned
design needs either an explicit sparse-slot/handle API or rank/select metadata
to map logical live indices to storage. The latter has costs even before vacuum.
Do not reuse a tombstone for a new appended row if iteration promises insertion
order, unless an ordering structure preserves that promise. A bitmap branch can
also inhibit vectorization; processing dense live runs or vector masks is a
later optimization.
If tombstones are adopted, deletion should normally destroy owned payloads at deletion time, clear liveness and decrement the live count. Vacuum must move only live owners, never read destroyed slots and never run their destructors again. Keeping resources alive until vacuum would be a different, visible ownership contract, not a storage detail.
An explicit vacuum() is preferable to an unexplained latency spike. Specify
whether it compacts within chunks, across chunks, or also releases spare
capacity. In-place stable compaction need not allocate, but it may not return
buffer capacity to the allocator. Repacking into fewer allocations can fail;
reserve destinations first and retain a valid container on failure. Outstanding
loans prohibit either operation. A slot-based API must say that vacuum
invalidates positions, or use handles whose mapping is updated. A default
threshold cannot substitute for that identity contract.
The ownership experiment exhausts all 256 deletion masks of eight rows, including repeated deletion and repeated vacuum, with counted affine owners. It validates immediate destruction and stable compaction, not a production bitmap, chunk directory, relocation handle scheme, or deletion-speed claim.
Statistics need a validity contract
DuckDB’s zonemap guidance explains why per-group bounds help clustered data much more than shuffled data. Parquet page indexes similarly separate bounds metadata from payload reads to skip irrelevant pages. For clist, start with optional conservative integer bounds for explicit field comparisons, only after chunking itself is justified.
Deleting rows may leave old min/max values as a wider conservative bound: that loses pruning opportunities but does not hide a matching live row. Appending or mutating a value outside the bound must widen it or mark the statistics invalid. A mutable field loan must mark the affected stats dirty before exposing the address; recompute on demand or scan without pruning. NaNs, signed zero, nullable values, string ordering and unknown comparisons need explicit rules; “invalid/unknown means scan” is the safe fallback.
Exact sums and counts have stronger maintenance requirements. Cached sums cannot silently change checked-overflow traps or floating-point reduction order. Nor may a filter skip an arbitrary callback that prints, mutates a capture, allocates or traps. Recognize a bounded effect-free field comparison or expose an explicit typed query API; otherwise run the ordinary pipeline. This is the same semantic restraint required by automatic SIMD.
The spike’s min-bound test finds one candidate group out of eight for sorted integers and all eight for a permutation of the same values, with identical results. It does not time metadata construction or claim a general query speedup. Compression, secondary indexes, joins, transactional updates and automatic query planning remain outside this container proposal.
Columnar-list spike evidence
This accompanies the columnar-list proposal. Sources
and reproduction commands live in experiments/columnar-list/README.md in
the repository. These are bounded feasibility experiments, not a production
container, Plenty performance results, or a backend comparison.
This page records the original 2026-10-23 experiment. The subsequent generated Plenty implementation spike tests real runtime failure paths, borrowing, and Cranelift-generated scans separately.
Layout experiment
The dependency-free Rust program compares an array of 64-byte records (eight
u64 fields), eight flat columns, and groups of 1,024 or 4,096 rows with one
allocation per field. Every layout contains identical generated values.
The kernels sum one field, filter on one field and sum another, mix all eight
fields in sequential row order, and perform the same whole-row work in an
odd-stride permutation. That permutation visits every row once; it is a
repeatable dispersed-access workload, not a model of all random workloads.
All arithmetic deliberately wraps. This is equivalent across the experiment, but does not model Plenty’s default overflow traps. Checked integer and floating-point kernels need separate end-to-end validation before adoption. Timed paths allocate nothing; construction is outside the scan measurements. Append and vacuum are correctness experiments, not performance measurements.
On 2026-10-23: AMD Ryzen 7 7840HS, Linux 7.2.11, rustc 1.99.0
(b940084d7), LLVM 23.1.1, native x86-64 target, CPU affinity 4. Per-core
L1d is 32 KiB, L2 is 1 MiB, and shared L3 is 16 MiB. Each layout has
32 KiB, 2 MiB or 64 MiB of row payload at the three tested lengths; the
single selected column has 4 KiB, 256 KiB or 8 MiB. All four layouts coexist.
Release builds use -C target-cpu=native; a second build also disables loop
and SLP vectorization. Seven samples rotate layout order; each sample traverses
4,194,304 rows in total. References and outputs cross black_box, every result
is checked, and individual scan functions are not inlined into the timed
repetition. These are repeated warm scans with no cache flush. Frequency is
not locked and the machine is shared: load average was approximately 8.1–9.7
on 16 logical CPUs. Treat close rankings and exact ratios as provisional.
Large input: median nanoseconds per row
1,048,576 rows; lower is better. “Scalar” means both LLVM vectorizers disabled, not a simulation of Cranelift. Native may use all supported host instructions.
| Build / workload | AoS | Flat SoA | Chunk 1,024 | Chunk 4,096 |
|---|---|---|---|---|
| Scalar: sum one field | 2.463 | 0.251 | 0.446 | 0.334 |
| Scalar: filter two fields | 1.363 | 0.318 | 0.817 | 0.513 |
| Scalar: all fields, sequential | 1.904 | 3.522 | 3.006 | 2.626 |
| Scalar: all fields, permuted | 24.146 | 47.179 | 58.057 | 52.378 |
| Native: sum one field | 4.217 | 0.078 | 0.203 | 0.154 |
| Native: filter two fields | 6.895 | 0.292 | 0.736 | 0.458 |
| Native: all fields, sequential | 10.285 | 2.254 | 2.790 | 2.385 |
| Native: all fields, permuted | 22.904 | 49.426 | 56.298 | 57.228 |
Sample ranges matter: scalar AoS sequential whole-row work ranged 1.707–2.325 ns/row and flat SoA 3.453–4.638. Native permuted work ranged 21.686–27.734 for AoS and 45.205–64.939 for flat SoA. This is enough to identify tradeoffs, not to choose final tuning parameters.
Size sensitivity: one-field scan
| Rows | Scalar AoS | Scalar flat SoA | Native AoS | Native flat SoA |
|---|---|---|---|---|
| 512 | 0.403 | 0.240 | 0.560 | 0.040 |
| 32,768 | 0.810 | 0.256 | 0.735 | 0.064 |
| 1,048,576 | 2.463 | 0.251 | 4.217 | 0.078 |
The large single-column working set fits shared L3 while the complete AoS does not. Cache residency and reduced traffic are part of the benefit, but these warm-scan numbers must not be advertised as sustained DRAM bandwidth or cold-scan speedups. No hardware cache-miss counters were collected.
Code generation changes the story
Inspection of emitted assembly found vpgatherqq in the native AoS narrow,
filter and sequential whole-row kernels. Flat SoA uses contiguous vector
loads/arithmetic (vpaddq, and vprolq for whole-row mixing), without those
gathers. With both vectorizers disabled, these kernel bodies contain no vector
instructions. Chunk scan functions call the corresponding SoA kernels.
Thus the striking native ratios combine layout, cache residency and a poor AoS gather strategy on this compiler/CPU. Scalar results still favor columns for narrow work, but the sequential whole-row ranking reverses. Do not claim that all full-object access is slower in SoA or that LLVM always improves a loop. The experiment also charges flat SoA’s generated indexed row access its bounds checks and chunked random access its directory/division work. Future Plenty lowering may optimize those differently.
Flat columns won the measured narrow scans. Fixed chunks did not provide a scan-speed reason to adopt them here; growth limits, metadata pruning and deletion behavior need separate measurements. This supports starting flat, without foreclosing chunked storage later.
Ownership and metadata experiments
Four Rust tests check:
- Equal row values and scan results at lengths 0, 1, 2, 7, 1,023, 1,024, 1,025 and 4,097, including chunk tails.
- Failure before either required column reserve: no partial live row, both
owned fields returned intact, then destroyed exactly once. Real
try_reserveerrors share this path. Injection is a deterministic gate, not process-wide allocator failure instrumentation. - Every deletion mask of eight rows: owners destroyed at deletion, repeat
deletion harmless, stable in-place vacuum preserves row alignment/order,
repeat vacuum harmless and no duplicate destruction. Safe
Optionslots model holes; production bitmap/uninitialized storage is not implemented. - Conservative min-bound pruning over identical sorted/permuted values:
a
< 64query visits 1/8 groups when sorted and 8/8 when dispersed, returning the same sum. Statistics are rebuilt for the test, not maintained under mutation. This is a correctness demonstration, not a timing result.
These tests reduce implementation risk but leave real runtime allocation failure, destructor ordering, chunk relocation, and borrowing diagnostics for the production gate. There is no benchmark of noncopyable/string fields, concurrent updates, growth, tombstone scan density or memory reclamation.
Current Plenty feasibility
Using a compiler freshly built from 45798945, columns.plenty creates two
ordinary lists, reserves both before appending, and zips borrowed columns into
a fold. It prints 90. This is checked language code with ordinary collection
storage and no generated schema support. It establishes that a concrete SoA
wrapper need not wait for clist, not that its elements are packed native words.
reject-row-reference.plenty fails with references cannot be stored in class fields. reject-clist.plenty fails with `clist` is not a generic function.
The rejection fixtures document the baseline only, not new defects. The
implementation inspected in plenty-runtime/src/entries.rs stores collection
rows in u128 words; the borrowed-list row helpers in
plenty-runtime/src/aggregates.rs expose contiguous row addresses. A columnar
source needs new field/projection lowering and must not masquerade as that ABI.
All four native tests pass, including in release mode; formatting and Clippy
pass for the isolated crate. The Plenty checks are reproducible through
check_plenty.py. Existing language/reference behavior is unchanged.
Generated Plenty column-container spike
The primitive-only restriction makes a useful column container feasible with Plenty’s existing ownership rules. A generated facade passes actual allocation failure and borrow checks, and a one-field scan is substantially faster than ordinary row storage. It also exposes expensive indexed access and whole-row reconstruction. These are reasons to implement direct column projection carefully, not to ship the generated wrapper unchanged.
This 2026-10-24 follow-up to the proposal uses Plenty
00d32a13. Sources, commands and preserved timing samples are in
experiments/columnar-list/plenty/ in the repository. The initial spike changed
no compiler or runtime; the indexed-access follow-up below records the later
compiler fixes. clist[T] remains proposed.
What was implemented
An experiment-only generator reads one public, nongeneric, field-only class.
It accepts bool, signed/unsigned 8/16/32/64-bit integers, f32 and f64;
it rejects methods, destructors, private fields, nested/owned fields and empty
schemas. For Particle it emits a checked Plenty ClistParticle class with
one private list per field, length, reservation, append, scalar getters/setters,
whole-row copying, and borrowed column methods.
mut items = ClistParticle()?
items.append(Particle(1, 2, 3, 4, 5, 6, 7, 8))?
xs = items.column_x()
total = iter(xs).fold(0, add)
This is specialization into existing language constructs, not a new parser feature or an ordinary generic library implementation. Each column uses current 16-byte scalar list slots; the eight-field row list uses 128-byte row payloads. Neither is packed native-word storage. Empty construction allocates eight list handles; this differs from the proposed allocation-free empty owner.
Append reserves every column before publishing any value. A failure returns an
error containing the intact submitted row. Earlier capacity growth may remain,
but lengths and existing values are unchanged. Subsequent scalar appends cannot
allocate after reservation; the generator makes that invariant explicit with
unwrap(). Row reconstruction creates an independent value without allocation.
It never manufactures &Particle pointing across unrelated allocations.
Ownership and failure evidence
The freshly built compiler with runtime-checks passes 35 generated fixtures
(31 executing programs and four compile-fail cases):
- Every empty-constructor allocation and every append reserve can fail, both before the first row and while growing eight existing rows. Failed append returns all eight input fields, and retry succeeds. Runtime allocation accounting finishes balanced on every exit.
- Reserved append, row reconstruction and a borrowed two-column
zip/foldperform no successful allocation within checked regions. - Differential tests against
list[Particle]cover lengths 0, 1, 2, 7, 8, 9, 33 and 1,025, field mutation, every field’s value and negative indexing. Invalid row copies returnNothing, including extreme signed indices. - All eleven accepted primitive types round-trip, including signed extrema, unsigned maxima and floating negative zero.
- Live column/pipeline loans reject owner mutation; moved owners cannot be reused; callers cannot mutate private backing lists independently.
Two generator unit tests reject unsupported schemas. The three original Plenty feasibility probes still pass. No custom destructor, removal, stable row handle, stored view, cross-thread operation or affine-field semantics are implemented. Column loans conservatively borrow the entire owner.
Issue #427 recorded a frontend limitation found here. Its follow-up now accepts
iter(items.column_x()) directly and retains the returned loan through
consumption. The fixture suite includes this positive case and rejects mutation
while such a pipeline holds the owner.
End-to-end scalar measurements
AMD Ryzen 7 7840HS, x86-64 Linux, CPU 4, Plenty --release, current Cranelift
0.131.1. Eight checked i64 fields per row; values repeat every 1,024 rows.
Both layouts remain allocated. Seven samples follow one warmup, each sample
repeats a scan 20 times, alternating layout order. Every checksum is checked
independently in Python. No cache flush, frequency lock, counters or non-x86
execution was used. These are warm scans on a shared machine.
Median nanoseconds per row; lower is better:
| Kernel | 8,192 rows: list | 8,192: columns | 262,144 rows: list | 262,144: columns |
|---|---|---|---|---|
| One-field sum | 1.155 | 0.661 | 4.075 | 0.711 |
| Two-field filter/fold | 2.030 | 2.605 | 2.884 | 2.630 |
| All fields, direct indexed columns | 2.483 | 34.096 | 3.210 | 36.340 |
All fields, copy_row sequential | 2.537 | 66.190 | 3.535 | 66.882 |
All fields, copy_row permuted | 6.341 | 67.314 | 49.377 | 169.975 |
The clear result is the one-field sum: about 1.7× faster at the small size and 5.7× at the large size. Large rows occupy 32 MiB of payload, while one column occupies 4 MiB; shared L3 is 16 MiB. Cache residency is part of the benefit. The large one-column samples ranged 0.687–1.389 ns/row versus 3.556–4.651 for the row list. Two-column results are mixed and close enough at the large size that they do not establish a robust improvement.
The all-field comparison includes real facade costs, not just memory layout. Assembly inspection at the original spike revision showed:
- The one-column fold hoists length, base and stride, then directly loads and adds one scalar per iteration. There is no callback call in its hot loop.
- Two-column
ziphoists both sources, but calls the scalar fold callback for each pair. Checked arithmetic and original evaluation order remain intact. - Direct indexing of eight borrowed columns called
plenty_list_scalar_geteight times per row. Issue #429 records this cost. copy_rowadds a method call, reconstruction/Option handling and cleanup machinery. Avoiding reconstruction roughly halved the sequential cost there, but indexed access dominated the direct-column version.
The reserved construction pass took 145 ns/row for the row list and 454 ns/row for columns at the large size. This is one observation per layout, including allocation and append, not a repeated growth benchmark. Eight reserve/check/ append paths per row are a visible cost of this generated implementation.
Indexed-access follow-up (#429)
Ordinary for loops now cache length, row pointer and stride for immutable
shared scalar-list bindings already in scope. Indexed reads use direct loads
with separate bounds checks for every column, including negative indices.
No runtime container layout is exposed, equal column lengths are not assumed,
and the original helper handles the cold out-of-bounds diagnostic. Mutable,
retargetable and owned sources keep their existing path.
Repeating the same benchmark against baseline 1c1b8ca1, without overlapping
task builds, gave these medians (ns/row):
| Kernel | Rows | Before | After |
|---|---|---|---|
| Eight indexed columns | 8,192 | 33.248 | 5.653 |
| Eight indexed columns | 262,144 | 37.310 | 8.896 |
| One-column fold | 262,144 | 0.648 | 0.647 |
| Whole-row reconstruction | 262,144 | 64.465 | 63.062 |
The targeted large scan improved about 4.2×, but still trails ordinary row iteration (3.113 ns/row in the after run). Reconstruction is unchanged by this optimization. The eight-column function grew from 804 to 1,856 machine-code bytes; the two-column issue reproduction grew from 371 to 568. Cold compile/link medians were 91.47 versus 91.09 ms for the full benchmark and 47.26 versus 47.23 ms for the small reproduction: no meaningful compilation-time change in these seven alternating samples, using debug-built compilers and release output. The size increase is a real tradeoff, not evidence for applying this everywhere.
experiments/columnar-list/plenty/indexed-measurements.json retains raw samples,
compiler hashes and methodology. Reproduce runtime results with the existing
bench.py command in that directory’s README against each compiler.
Use --release --dump asm --dump-filter soa_stream
to inspect code size and the call-free hot loop; time --release --compile
with PLENTY_CACHE=off and PLENTY_LINKER_THREADS=1 for cold compilation.
Consequence for the design
Proceed with the primitive-only restriction and explicit borrowed column operations. The spike validates their ownership model and demonstrates a useful scalar scan benefit without SIMD or a backend switch. Ordinary list storage remains preferable for the measured whole-row work.
A production implementation needs a deliberate representation and projection path: packed primitive buffers if compact storage is promised, an empty owner without eight heap handles, and direct checked loads instead of repeated helper calls for recognized field scans. These findings concern frontend/runtime representation as well as backend optimization; they do not establish a need to replace Cranelift with LLVM. The remaining adoption scope belongs in B76, including removal, richer classes and portability checks.
Initial native column implementation
The current interface implements
the bounded primitive-field version of clist[T]. The earlier
generated-container experiment supplied the ownership
and performance baseline; its eight ordinary lists are not the native layout.
This page preserves the initial implementation’s measurements. The subsequent indexed-access work removes generic dispatch from field access and row reconstruction; consult that report for the current comparison.
Decisions at the representation boundary
One allocation holds a header and a contiguous buffer for each field. Native
widths are 1, 2, 4 or 8 bytes, including one-byte booleans. Rounding row capacity
to a multiple of eight aligns every column start. This avoids separate column
allocation failures: reserve prepares a complete replacement before publishing
it, and append advances the single row count only after all fields are written.
Failed append returns the unchanged input object in AppendError[T].
Ordinary Plenty references point to tagged value slots, so they cannot safely
point into these packed buffers. A column therefore yields scalar values
through a local Iter[FieldType] while borrowing the entire owner. Its pointer
and length are loaded before the loop; iteration emits native scalar loads.
Existing fluent operations consume these values without new column-specific
map/filter/reduce methods. A live plan prevents owner mutation or movement.
copy_row materializes an ordinary scalar-field object without allocation.
There is no general &T row view, no packed field reference, and no mutable
column plan. The initial indexed field access was checked through runtime helpers.
Strings, nested classes, enums, custom destructors and fallible initializers
are rejected. These restrictions make ownership and failure cleanup explicit
without committing to a future reference representation.
Reproducing the comparison
Build without allocation instrumentation, then run the extended original probe:
cargo build --locked --release --bin plenty
python3 -B experiments/columnar-list/plenty/bench.py target/release/plenty \
--native-clist --skip-generated --out /tmp/plenty-native-columns \
--rows 8192 262144 --repeats 20 --samples 7 --cpu 8
The runner checks every checksum, discards the first sample, alternates layout
order, and retains both owners during measurement. This final comparison uses
native clist and ordinary list[Particle]; it omits the earlier generated
eight-list spike. The earlier three-owner run is superseded, so its absolute
timings are not mixed into this two-owner comparison.
Each row has eight i64 fields. The kernels scan one field, filter using two
fields, read every field by index, materialize sequential complete rows, and
materialize rows in a fixed permutation. The historical random kernel uses
(i * 65537) % rows: this is a predictable permutation, and degenerates to
sequential order at 8192 rows. It is not an arbitrary random-access workload.
Construction is measured separately.
These are scalar x86-64 measurements. No automatic SIMD or parallel execution is introduced, and no other architecture’s performance is inferred. Contiguous loads give later vectorization and partitioning work a useful representation; they do not remove checked arithmetic, effects, aliasing or scheduling questions.
Results and limits
experiments/columnar-list/plenty/native-measurements.json retains all samples,
compiler hash, frequency samples and host information. These measurements use
reviewed commit d6e9a2cf on an AMD Ryzen 7 7840HS, pinned to CPU 8 with the
performance governor, after competing builds and timings finished. Frequency
samples taken every 250 ms during execution ranged from 4.089 to 4.093 GHz;
before-run samples ranged from 3.271 to 3.511 GHz. This replaces the initial
throttled final numbers. It remains a workload-specific measurement on one host.
Median nanoseconds per row, over seven measured samples:
| Rows | Operation | Ordinary row list | Native clist |
|---|---|---|---|
| 8,192 | One-field fold | 1.463 | 0.560 |
| 8,192 | Two-field filtered fold | 2.355 | 2.753 |
| 8,192 | All fields by index | 3.021 | 116.088 |
| 8,192 | Sequential whole rows | 3.047 | 96.809 |
| 8,192 | Permuted whole rows | 7.520 | 96.842 |
| 262,144 | One-field fold | 4.470 | 0.495 |
| 262,144 | Two-field filtered fold | 4.061 | 3.787 |
| 262,144 | All fields by index | 3.694 | 117.517 |
| 262,144 | Sequential whole rows | 3.971 | 96.098 |
| 262,144 | Permuted whole rows | 49.437 | 132.525 |
The ordinary all-fields and sequential whole-row controls borrow rows directly; the column kernels respectively index fields and materialize rows. They perform the same calculation using the available interfaces, so this compares practical access paths rather than isolating memory layout alone.
The large one-field scan was about 9.0× faster than the row list. The two-field median was only about 7% faster, with overlapping samples: native 3.660–3.845 ns/row and ordinary 3.648–5.448. This is not convincing evidence of a two-field speedup; the small case was about 17% slower. The earlier 4.3× two-field claim is withdrawn. Native columns use 64 bytes of payload per eight-i64 row, compared with 128 tagged-slot bytes in the row list; one large native column occupies 2 MiB. These layout and code-generation benefits depend on the calculation.
Whole-row work is a substantial weakness of this first implementation: sequential row copying was about 24× slower, and indexed eight-field work about 32× slower than the row-list controls. The native indexed kernel makes eight generic dispatcher calls per row. Reconstruction walks the schema, recomputes column prefixes and snapshots tagged row slots. These are current lowering costs, not inherent costs of columnar storage. Issue #432 records this performance concern; the follow-up addresses that dispatch overhead.
Reserved construction took about 174 / 145 ns per row for the large ordinary / native collections. These are single construction samples, not a stable construction benchmark. They do not justify a general append-speed claim.
Generated code and correctness
nm -S on the large-case executable gives these kernel sizes in bytes, excluding
shared runtime helpers and measurement wrappers:
| Kernel | Ordinary row list | Native clist |
|---|---|---|
| One-field fold | 161 | 294 |
| Two-field filtered fold | 235 | 620 |
| All fields by index | 362 | 1,269 |
| Sequential whole rows | 362 | 831 |
| Permuted whole rows | 560 | 936 |
The native one-field assembly has two setup helper calls, followed by a direct eight-byte indexed load/add and the existing overflow check. There is no helper call per element and no SIMD instruction in that loop. Its larger setup code is another reason not to assume a win for tiny collections.
Twenty-one focused cases cover scalar widths and floating-point bits, generic schemas, module privacy, owner loans, nested enum storage, source evaluation order, overflow/allocation failure recovery, and allocation-free reserved work. The full workspace suite with all features, Clippy with all targets/features, tutorial examples, Markdown links and the book build pass. The whole-row performance limitation motivated the subsequent indexed-access work.
Direct indexed column access
Issue #432 identified generic runtime dispatch as the dominant cost of indexed
clist fields and copy_row. The initial implementation
already emitted direct loads for column pipelines, but indexed eight-field work
made eight dispatcher calls per row. Row copying walked the schema, recalculated
column prefixes, and copied the resulting tagged row again.
Implementation and constraints
The compiler now calculates column prefixes from the checked primitive schema.
Reads and writes use native-width memory operations. A row copy checks its index
once, loads the fields, and writes directly into fresh ordinary row storage.
The existing column-pipeline setup is unchanged. The compiler and runtime share
layout offsets, checked against the runtime’s repr(C) header.
Null owners are checked before reading the header. Negative indices are
normalized without signed overflow, and an unsigned comparison rejects both
still-negative and too-large indices. Indexed access retains its source-located
trap through a small cold helper; copy_row returns Nothing for missing rows.
Field types determine the actual load/store width and row-slot encoding, including
signed narrow integers, unsigned maxima and floating-point bit patterns.
There is no new frontend analysis or pointer cache across source operations. Each operation reads the current owner, so reserve, append and replacement between operations cannot leave stale metadata. Existing loans, evaluation order and checked arithmetic are unchanged. General loop hoisting, automatic SIMD and parallel execution are outside this change.
Instruction-count evidence
experiments/columnar-list/plenty/count_access.py runs the same indexed and
sequential-copy kernels as bench.py, one at a time, for 200 passes over 8,192
rows. Linux perf stat counts user-mode instructions and cycles in three
alternating before/after samples, pinned to CPU 8. Every checksum is checked.
Compilation finishes before counting. This is useful under CPU contention;
cycles and elapsed time still require a separate quiet timing comparison.
Counts include startup, construction, output and teardown. A zero-pass control reports their cost separately; the table does not subtract it or claim isolated kernel counts.
experiments/columnar-list/plenty/indexed-instructions.json retains the samples,
compiler/source/binary hashes and counter scope. The baseline is local commit
697aedb5626d74b7f97f1a51bcd3334823df5744.
| Workload | Before instructions | After instructions | Reduction |
|---|---|---|---|
| Construction-only control | 15,870,296 | 15,870,567 | Essentially unchanged |
| Eight indexed fields | 3,043,642,962 | 255,083,611 | 11.9× fewer |
| Sequential row copies | 2,391,559,687 | 240,339,982 | 10.0× fewer |
The counts varied by fewer than 400 instructions across the three samples for each case. They support the removed-work explanation, rather than substituting for elapsed-time verification.
Reproduce with two compilers built without allocation instrumentation:
python3 -B experiments/columnar-list/plenty/count_access.py \
/path/to/before/plenty /path/to/after/plenty --out /tmp/clist-counts \
--rows 8192 --repeats 200 --samples 3 --cpu 8
The reference and tutorial continue to recommend column pipelines for selected
field scans. Whole-row reconstruction still gathers separate fields and moves
ordinary tagged row storage; removing dispatch does not make this free or prove
that clist is universally faster than an ordinary row list.
Final timing verification
experiments/columnar-list/plenty/native-indexed-measurements.json retains final samples,
compiler/source/executable hashes, kernel sizes, frequency readings and rejected
attempts. These use the same AMD Ryzen 7 7840HS, pinned to CPU 8. Both compilers
were built without allocation instrumentation. The runner waited for competing
builds/tests to finish, checked host processes every 250 ms during execution,
and rejected one large after-run when a build began. Earlier interfered timings
are not mixed into these results. This excludes detected build interference,
not all operating-system noise. Median sampled frequency was about 4.092 GHz;
the artifact retains occasional lower readings as well.
The existing bench.py program checks every checksum, warms up once, alternates
the ordinary/native layout order, and measures seven samples. Large cases use
20 repetitions; tiny cases use 100,000. All executables were prepared before
the final timing pass. Median nanoseconds per row:
| Rows | Indexed fields before | Indexed fields after | Row copy before | Row copy after |
|---|---|---|---|---|
| 1 | 119.778 | 7.496 | 97.596 | 14.830 |
| 8 | 105.703 | 6.477 | 80.813 | 12.457 |
| 64 | 99.223 | 6.519 | 80.060 | 12.780 |
| 8,192 | 115.951 | 6.419 | 94.516 | 12.331 |
| 262,144 | 117.983 | 6.526 | 96.029 | 12.558 |
At 262,144 rows, indexing improved about 18.1× and sequential copying about
7.6×. The after-run ordinary-row controls were 3.985 and 4.031 ns/row:
indexed clist work is still about 1.6× slower and row copying about 3.1× slower.
The copy kernel still moves the unwrapped ordinary row into local storage with
a 128-byte memmove. Removing schema dispatch does not remove that cost.
The large indexed function shrank from 1,269 to 991 bytes; the copy function grew from 831 to 961 bytes because its field loads are now explicit. These sizes exclude shared helpers. Neither valid indexed access nor reconstruction calls the generic collection dispatcher. Cold range-validation code elsewhere in the benchmark remains separate from the per-row access path.
The one-column control is a limitation: its large median changed from 0.497 to 0.743 ns/row, despite identical emitted kernel assembly and unchanged 294-byte function size. The 8,192-row case similarly changed from 0.494 to 0.736 ns/row. Code placement/alignment is a candidate explanation, not an isolated causal result. An exploratory change to inline column setup was removed; the final patch retains the original pipeline setup and loop. This observation is why instruction counts alone do not justify a universal speedup claim. The target indexed/copy improvements remain clear in both counters and elapsed time. Issue #437 records the scan-throughput follow-up separately.
Bounded placement check (issue #437)
The follow-up found reproducible address sensitivity, so the observation cannot
be dismissed as timing noise alone. probe_scan_layout.py compiles the one-field
workload once and relinks that same object with GNU ld at eight text-section
addresses. Linker alignment produces four distinct function placements. Each
binary performs 20,000 scans of 8,192 rows; checksums are checked, order is shuffled
with seed 437, and one warmup precedes three measured samples per placement.
Two independent runs produced the same object hash and 294-byte scan function.
| Function address modulo 64 | Cycles/row across the two runs | First-run time, ns/row |
|---|---|---|
| 9 | 2.045–2.066 | 0.500–0.502 |
| 25 | 2.048–2.065 | 0.505–0.507 |
| 41 | 3.048–3.070 | 0.747 |
| 57 | 4.055–4.084 | 0.996–0.997 |
Retired instruction counts remain essentially identical. Hardware counters cover the whole process; internal elapsed timing covers the repeated folds. The second run has noisier elapsed times, while the cycle groups remain distinct. Modulo 64 is an address label here, not a claim about the CPU’s exact fetch-block size. Instruction-fetch/decode alignment is a plausible mechanism, but the probe moves the whole text section and does not isolate a specific CPU front-end effect or prove the precise cause in the original two binaries.
The probe reused an available compiler executable instead of rebuilding Rust;
its hash, the object hash and all samples are retained in
experiments/columnar-list/plenty/scan-layout-measurements.json. Reproduce on
x86-64 Linux with GNU ld, nm, and access to user-mode perf counters:
python3 -B experiments/columnar-list/plenty/probe_scan_layout.py \
/path/to/plenty --out /tmp/clist-scan-placement
This completes the bounded investigation. No compiler change is proposed. A universal padding/alignment policy would need broader benchmarks and code-size tradeoffs; issue #437 remains a low-priority performance concern. The placement experiment is evidence, not a hard-coded alignment recommendation.
The subsequent function-alignment evaluation tests 16-, 32- and 64-byte boundaries against short scans, a two-column fold and an unrelated scalar loop. It rejects a blanket alignment change: some placements improve while others regress.
Reproduce the timing programs on each compiler, without competing builds:
python3 -B experiments/columnar-list/plenty/bench.py /path/to/plenty \
--native-clist --skip-generated --out /tmp/clist-access-large \
--rows 8192 262144 --repeats 20 --samples 7 --cpu 8
python3 -B experiments/columnar-list/plenty/bench.py /path/to/plenty \
--native-clist --skip-generated --out /tmp/clist-access-small \
--rows 1 8 64 --repeats 100000 --samples 7 --cpu 8
The full all-features workspace suite, Clippy, formatting, documentation links and book build pass. Focused cases cover development/optimized modes, mixed primitive widths, growth, copies returned across calls, independent copies, extreme indices, source-located traps, and absence of generic reconstruction dispatch. Existing tests retain evaluation-order, enum-projection, allocation failure and owner-loan coverage.
Function-alignment follow-up
Issue #437’s placement probe found a repeatable scan-throughput difference with identical function bytes. The next experiment tested a small potential remedy: raise the alignment of every generated function to 16, 32 or 64 bytes.
Do not adopt a blanket function-alignment change from these results. It improves some placements and worsens others. In particular, 64-byte alignment makes the one-column scan consistently slower than its best unaligned placement. This follow-up changes the experiment and documentation, not compiler behavior. Issue #437 remains open at low priority.
The subsequent online research connects the observed cycle groups to documented fetch-block constraints and a closely matching upstream incident. It finds no verified small remedy and recommends deferring further optimization rather than applying blanket alignment.
Available backend control
The pinned Cranelift 0.131.1 exposes log2_min_function_alignment. Its x86-64
backend specifies a minimum alignment of one byte and a preferred alignment of
32 bytes, but ObjectModule uses the compiled buffer’s required alignment.
Our baseline object therefore has byte-aligned functions (some sections need
eight-byte alignment for constants). Preferred alignment is not a guarantee
that the object linker places every function on a 32-byte boundary.
This version has no public loop-header alignment control. Its
bb_padding_log2_minus_one option inserts dummy zero bytes for code-generator
stress testing; it is not a production loop-alignment setting. These observations
come from the pinned crate’s shared/settings.rs, isa/x64/inst/mod.rs,
machinst/buffer.rs, and the object backend, rather than an assumption about
what another version or Wasmtime does.
Controlled comparison
The compiler was freshly built from 68108734 without allocation instrumentation.
Each workload was compiled once. GNU objcopy then raised all function-section
alignments, and GNU ld linked each variant at eight requested text addresses.
The baseline retains the original object, including constant-pool alignment.
The probe verifies unchanged function-section contents and actual function
addresses. It never inserts instructions or changes arithmetic semantics.
Measurements use the AMD Ryzen 7 7840HS, pinned to CPU 8, after the compiler build finished. Host-visible process checks found no competing builds around the measurement runs. Each sweep shuffles variant order with seed 437, excludes one warmup, and takes three samples per variant; every checksum passed. This is a small single-machine rejection test, not a representative architecture survey.
Ranges below are the minimum and maximum per-placement median cycles/row. Counters cover startup, construction, repeated calls and teardown; internal timers cover repeated calls. In the short scan, call/setup costs are substantial.
| Workload | Original alignment | 16 bytes | 32 bytes | 64 bytes |
|---|---|---|---|---|
| One-column fold, 8,192 rows | 2.04–4.07 | 2.04–3.06 | 2.04–3.05 | 3.04–3.11 |
| Two-column filtered fold, 8,192 rows | 11.17–11.72 | 10.87–12.71 | 10.76–12.24 | 10.79–10.91 |
| Checked scalar range sum, 8,192 iterations | 3.01–3.02 | 2.01–3.02 | 2.01–3.02 | 3.01–3.02 |
| One-column fold, 8 rows | 6.60–8.68 | 6.73–7.70 | 6.48–7.91 | 7.85–7.90 |
The one-column scan’s first sweep took 0.409–0.819 ns/row with original alignment, 0.409–0.618 with 32-byte alignment, and 0.609–0.624 with 64-byte alignment. A second complete sweep reproduced the same groups: 2.04–4.06 cycles/row originally, 2.04–3.06 at 32 bytes, and 3.04–3.06 at 64 bytes. Use these fresh within-run comparisons; earlier reports were measured at different CPU speeds. Across the first sweep’s alignments, retired instructions varied by fewer than 200 out of about 1.33 billion. Identical instruction counts are not identical throughput.
The scan function remains 294 bytes. At matching requested text addresses, 16-byte alignment adds 80 bytes to the linked text section, 32 adds 192–208, and 64 adds 416–464. Executable file size happens to remain unchanged because of existing segment padding. Across the other controls, text growth remains under 500 bytes; throughput tradeoffs, not this small size cost, reject the blanket change. Larger applications could accumulate more padding.
Interpretation and reproduction
Function alignment fixes the function’s starting boundary, not the hot loop’s
boundary within it. The one-column loop has a condition at function offset
0xc6 and a body at 0xfd; aligning the function to 64 bytes does not align
either block. Moving both blocks can improve or worsen instruction delivery.
The exact CPU mechanism is still not isolated, and the probe moves other code
as well. These results do not justify a benchmark-specific offset or NOP patch.
A further fix would need loop-aware backend work and broader validation; changing the global minimum alignment alone does not resolve this issue. That exceeds this bounded follow-up. No new user-facing flag is proposed.
The probe and all samples are in experiments/columnar-list/plenty/:
probe_scan_layout.py and alignment-policy-measurements.json. Requires x86-64
Linux, GNU binutils, a C linker driver, and access to user-mode perf counters.
After building the compiler, run the following sequentially without competing
builds. Repeat the first command with a new output directory to check stability.
python3 -B experiments/columnar-list/plenty/probe_scan_layout.py target/release/plenty \
--out /tmp/alignment-one --alignments 1 16 32 64
python3 -B experiments/columnar-list/plenty/probe_scan_layout.py target/release/plenty \
--out /tmp/alignment-two --alignments 1 16 32 64 --kernel two --repeats 5000
python3 -B experiments/columnar-list/plenty/probe_scan_layout.py target/release/plenty \
--out /tmp/alignment-scalar --alignments 1 16 32 64 --kernel scalar
python3 -B experiments/columnar-list/plenty/probe_scan_layout.py target/release/plenty \
--out /tmp/alignment-tiny --alignments 1 16 32 64 --rows 8 --repeats 1000000
Why identical scan code can run at different speeds
Online follow-up to issue #437, researched on 2026-10-24. The research found a closely matching upstream incident and a plausible explanation of the measured cycle groups, but no verified small fix or dependency upgrade. My recommendation is to close this investigation as understood enough to defer, retaining its reproducer and evidence. That is not a claim that performance was fixed or that the exact CPU mechanism has been proved.
This report adds no benchmark runs or compiler changes. It interprets the existing measurements and inspects the retained scan’s disassembly.
What AMD documents
AMD’s Zen4 guide describes next-address generation in aligned 64-byte blocks; a branch target near the end leaves a short block and reduces fetch bandwidth. For very hot loops it discusses aligning the loop’s start or end, and keeping branch placement within predictor limits. These concern the loop itself, not merely the enclosing function’s address. See sections 2.8.1.1 and 2.8.3 of AMD publication 57647, revision 1.01, April 2023 (mirrored PDF).
The same guide distinguishes that next-address path from traditional instruction fetch/decode: section 2.9 specifies 32-byte fetches aligned to 16-byte boundaries and contained within a 64-byte block. Section 2.9.1 describes a decoded-op cache that bypasses ordinary decoding and has its own entry and transition limits. Consequently, warm instruction caches and equal retired-instruction counts do not establish equal instruction-delivery throughput. The document is AMD’s primary material, hosted by a mirror; AMD’s current document endpoint returned an authorization error during this research.
A simple model fits the retained scan
For ordinary successful iterations, the 294-byte function executes two separated instruction regions:
- The condition and taken branch occupy offsets
0xc6..0xceinclusive. - The body, untaken overflow branch and jump back occupy
0xfd..0x117inclusive.
The function’s exit epilogue lies between them. Consider a deliberately simple
model: count the 64-byte blocks touched by each region separately, then add
the counts. Do not count only unique cache lines across the whole loop: a taken
branch starts the next region. For function address modulo 64 equal to d, a
region [a, b] touches floor((d+b)/64) - floor((d+a)/64) + 1 blocks.
| Function address modulo 64 | Condition fragments | Body fragments | Sum | Measured cycles/row group |
|---|---|---|---|---|
| 9, 16, 25, 32 | 1 | 1 | 2 | about 2 |
| 0, 41, 48 | 1 | 2 | 3 | about 3 |
| 57 | 2 | 2 | 4 | about 4 |
This arithmetic matches the rounded median cycle count of all 64 variants in
the retained one and repeat sweeps in
experiments/columnar-list/plenty/alignment-policy-measurements.json. It also
explains why 64-byte function alignment selected the three-cycle group: it fixes
d = 0, leaving the body to straddle a boundary. This is new analysis of existing
data, not a new experiment or an AMD-specified cycle formula.
Inference, not attribution: the correspondence makes instruction delivery a strong candidate. It does not separate fetch/prediction sequencing, op-cache packing, decode/fusion behavior, or other placement effects. The experiment moved other code too, and collected no counters isolating these mechanisms. Do not generalize this two-region formula to arbitrary loops or label it an instruction cache miss penalty.
A close Cranelift precedent
Wasmtime upstream issue 8573
reported roughly twice the execution time for identical hot-function bytes after
a different function changed size. It used an Intel i5-9500T, so it is an analogy,
not a diagnosis of our AMD CPU. Maintainers specifically identified a loop split
around its exit epilogue, suggested better block layout, and discussed partial
fetch blocks. The eventual mitigation was to prefer 32-byte function alignment;
the current x86-64 backend source
still links that incident beside preferred: 32 and minimum: 1.
This precedent is unusually close to our assembly shape. It also explains why simply adopting that historical mitigation is insufficient here: the local evaluation already tested 32-byte alignment and found both favorable and unfavorable placements. We have not identified a new fix hidden behind that source comment.
As checked on the research date, the published
Cranelift shared settings
identified themselves as 0.138.0-dev. They expose minimum function alignment
and dummy basic-block padding for debugging, but no loop-alignment setting.
The latter padding remains unsuitable as a production fix. This API check and
the upstream searches found no verified upgrade remedy; they do not prove that
newer code generation could never improve this case. No newer backend was built
or timed for this research.
Decision
There is useful explanatory evidence, so the outcome is not “nothing was found.” There is still no bounded, validated compiler change to make. Closing issue #437 as investigated/deferred is reasonable under the requested effort limit. Preserve the address-sensitive benchmark as evidence; do not describe closure as a performance fix, dismiss the observation as noise, or adopt global padding.
If this becomes important again, the concrete question is whether keeping the hot loop together, or applying loop-aware alignment in the backend, improves multiple workloads and architectures. The upstream incident supports that direction, but does not supply a drop-in patch. It does not justify switching backends solely for this microbenchmark.
For future causal diagnosis, AMD’s uProf pipeline-utilization documentation separates front-end bandwidth and latency bottlenecks, branch misprediction, and SMT contention. Those measurements would distinguish hypotheses better than repeating cycles and instruction counts alone. They were not collected here, and no additional local measurement campaign is proposed by this report.
Automatic SIMD filter/count
The first automatic SIMD implementation recognizes one i32 or u32 column,
one filter comparing its borrowed item with a same-type constant, and count().
It runs in optimized (--release) native builds. The
reference
defines the supported pattern; the tutorial
shows ordinary source code. There is no public SIMD API or new syntax.
The subsequent runtime-bound extension accepts immutable scalar captures and
two total comparisons joined by and. The reference above describes current
behavior; the measurements below retain the original constant-only slice.
Why this boundary
The initial recognition inspected the checked loop and a bounded scalar leaf predicate. It accepts exactly one comparison, either in a named function or a closure without captures. Calls, arithmetic, extra adapters and other shapes keep their existing scalar path. The original scalar loop is retained for the tail. No callback effects, traps, floating-point operations or allocation failures are reordered.
The source has a stable borrowed pointer and length. Fresh zero counters and
the known increment establish that the count fits i64. Four-lane loads only
run below the length rounded down to a multiple of four; they have no alignment
promise. Comparison masks widen to two 64-bit accumulators before summing, so
large columns cannot overflow 32-bit lane counters. The final scalar item local
is preserved as well as index and count.
Cranelift 0.137.0 emits 128-bit SSE2 instructions for this pattern. The current machine (AMD Ryzen 7 7840HS) has AVX2 and several AVX-512 features, but this slice does not need them. Using the existing target baseline keeps binaries portable among supported x86-64 machines and avoids runtime dispatch overhead and policy. This is not a claim that Cranelift can exploit all this CPU’s wider vector features; fresh backend upgrade probes establish the current width boundary and distinguish useful new backend features.
Validation and reproduction
Integration tests compare all six signed/unsigned comparisons over lengths 0–129 with scalar expectations, including extreme bit patterns, empty owners, column offsets and every four-lane remainder. Separate cases retain effects, captures, checked arithmetic and additional adapters on the scalar path. Dump tests check actual vector IR and packed comparison instructions.
The experiment in experiments/simd/clist/ compiles the actual Plenty pipelines,
then links a C fixture against their generated functions. It constructs read-only
internal-ABI owners solely to place columns immediately before a protected page.
Lengths 0–129 and a null empty owner pass. Odd lengths need four poison bytes
before the guard to keep the header aligned; scalar-oracle comparisons detect
counting that padding. This fixture is not a supported foreign clist interface.
Run from the repository root:
cargo build --locked --release --bin plenty
python3 experiments/simd/clist/run.py target/release/plenty --out /tmp/clist-simd-check --verify-only
python3 experiments/simd/clist/run.py target/release/plenty --out /tmp/clist-simd-bench
The retained benchmark measurements used a captured threshold to retain the scalar pipeline and a
constant threshold to enable SIMD, both comparing against five. It builds one
object and relinks it at four text addresses, 16 bytes apart, then shuffles
scalar/SIMD executions. Each variant has one warmup and three measured samples.
Internal timings exclude input construction; perf instructions/cycles include
startup and construction. Tiny calls therefore include substantial fixed scan
setup. This is a hot repeated microbenchmark, not an application or memory-
bandwidth claim. Code placement matters, as issue #437’s research
demonstrated; do not select just the best scalar/SIMD address pair. The current
runner uses an explicit scalar column loop as its reference, since captured
thresholds now qualify for SIMD; rerunning it measures that updated comparison.
Final measurements
Measured on 2026-10-24 on the Ryzen 7 7840HS, pinned to logical CPU 8, with a
fresh Cranelift 0.137.0 release compiler incorporating main c6e3f8b9, including
the executable trap-table changes. Host process monitoring sampled 16 times at half-second intervals
throughout the final run and observed no concurrent cargo/rustc work.
The retained experiments/simd/clist/measurements.json records all 72 measured
executions, checksums, actual function addresses, instruction/cycle counts,
compiler/object hashes and the method. Earlier exploratory runs are not mixed in.
Each cell below spans the four placement medians, not individual samples:
| Rows per call | Scalar ns/row | SIMD ns/row | Scalar instructions/row | SIMD instructions/row |
|---|---|---|---|---|
| 7 | 2.157–2.480 | 1.745–2.034 | 42.724 | 38.010 |
| 8,192 | 0.668–1.001 | 0.116–0.224 | 13.037 | 3.037 |
| 262,144 | 0.682–1.007 | 0.115–0.224 | 13.058 | 3.058 |
At matched link placements, the two larger sizes improve 3.94–5.93× in elapsed time with roughly 4.3 times fewer instructions. SIMD uses about 0.54–1.03 cycles/row, while the scalar control varies between about 3.0 and 4.6. The small case improves only 1.19–1.24×; setup costs dominate and the captured scalar threshold has extra fixed setup. This compares two paths in the same compiler, not identical source compiled before and after the change.
The previous 0.131.1 run is retained as
experiments/simd/clist/measurements-cranelift-0.131.json. The new vector loop
retains its comparison constant in a register and executes about 0.25 fewer
instructions per row on the larger inputs. Placement sensitivity is stronger
in this run, so the upgrade does not imply a uniform elapsed-time improvement.
The SIMD function occupies 592 bytes including its constant pool; the scalar
control occupies 513 bytes. Assembly contains unaligned movdqu, pcmpgtd,
mask shuffles and paddq/psubq, followed by the scalar tail. The comparison
does not call a predicate per row. The gains justify this small recognition
slice; they do not establish a general backend ranking, wider-vector benefit,
or speedup for other predicates and workloads.
SIMD after the Cranelift 0.137 upgrade
The automatic column filter/count slice was rebuilt and revalidated against
Cranelift 0.137.0, after merging local main through c6e3f8b9. The original PR used
0.131.1. Production lowering needed only the new MemFlagsData type and the
explicit signed-immediate builder methods (band_imm_s, iadd_imm_s,
ishl_imm_s, icmp_imm_s). The loop’s legality checks, four-lane width, baseline
target and scalar tail are unchanged.
Fresh evidence
Post-integration validation passed 145 compiler tests, 3,787 integration tests (two ignored), 183 runtime tests (four ignored) and five example tests. The full workspace/all-features run exposed issue #442 in the newly merged trap-table test: heap-to-stack offsets can exceed its signed 32-bit range. Keeping its synthetic code/sites/table in one allocation fixes the test without changing production behavior; the entire runtime suite then passed in both default and all-features configurations. This includes signed/unsigned comparison and fallback tests, generated-code checks and protected-page scans at lengths 0–129. Clippy passed for all workspace targets with warnings denied. Both compiler profiles were rebuilt from an empty task target directory; compiler debug information was disabled to keep build storage bounded.
The final four-placement measurements use the new release compiler. Its SIMD loop keeps the comparison constant in a register instead of reloading it each four-row iteration. The measured large scans execute about 0.25 fewer instructions per row. At the initial backend upgrade, scalar control assembly was unchanged while SIMD register allocation and block layout changed. Subsequent main changes replace executable trap helpers with PC-relative trap tables, shrinking the SIMD/scalar functions from 608/525 to 592/513 bytes. Measurements were repeated after integration; retired instructions per scanned row remain the same. Direct reference-store changes introduce no new alias-region assumptions. We do not attribute all elapsed-time differences to one optimizer pass.
Vector-width probe
examples/clist_simd_backend_probe.rs compiles a load/compare/store function
with the verifier enabled and the workspace’s pinned dependencies. The
experiment instructions (experiments/simd/clist/README.md in the repository)
reproduce it. On the Ryzen 7 7840HS:
| IR type | Baseline x86-64 | Detected native features |
|---|---|---|
i32x4 (128 bits) | Compiles, SSE encodings | Compiles, VEX encodings using XMM registers |
i32x8 (256 bits) | Rejected | Rejected |
i32x16 (512 bits) | Rejected | Rejected |
Both rejected widths report Unexpected SSA-value type. This is a compile-only
probe, independent of Plenty’s pattern recognition. The pinned upstream
x86 register-class selection
accepts vector SSA values up to 128 bits. Native feature detection changes
available instructions but does not remove that restriction. The versioned
upstream file was checked against the installed crate source.
Useful upstream changes
Cranelift 0.137 corresponds to Wasmtime v50. Release notes across v45–v50 describe optimizer and target improvements. In particular, v48 adds dead-store elimination in alias analysis and VNNI dot-product lowering; v50 improves alias analysis and introduces initial APX support. APX is not a wider SIMD-vector facility. Winch SIMD entries concern a different compiler backend.
The pinned x86 source has new AVX-VNNI and AVX-512-VNNI feature flags and dot-product patterns for packed signed/unsigned byte multiplication and accumulation. These could help a future quantized or packed-byte column calculation, provided its semantics and target selection are established. They do not replace this integer comparison/count kernel. No VNNI lowering, runtime dispatch or public SIMD API is added here; broader priorities remain in the backlog.
The upgrade improves the backend we already use without requiring a broader ISA policy for this slice. It does not demonstrate general auto-vectorization or unlock ordinary 256/512-bit vector SSA operations on x86.
SIMD counts with runtime bounds
The next bounded step after constant comparisons recognizes
immutable runtime scalar captures and two total comparisons joined by and.
This makes queries such as “count scores between these two input limits” use the
same four-lane SSE2 path without exposing vectors, alignment, tails or hardware
selection in the source. The tutorial
shows the supported idiom; the reference
defines its limits.
Legality boundary
Predicate recognition now has separate bounded metadata rather than expanding
general leaf-function inlining to control flow. It accepts one comparison or
the exact checked-IR short-circuit form for two comparisons joined by and.
Both compare the item with a same-type constant or an immutable scalar capture;
either operand order is accepted. Captures are limited to two by-value i32 or
u32 values matching the column. The actual closure environment is read once
before the vector loop, preserving its captured snapshot rather than rereading
the original source binding.
Two total comparisons can be evaluated eagerly as vector masks because neither
can call, allocate, mutate, trap or read externally mutable state. Combining
their masks therefore preserves observable short-circuit behavior. Borrowed
captures, mutable closures, arithmetic, calls, or, extra comparisons and extra
pipeline adapters retain the scalar path. The original scalar loop handles the
tail, with the same bounds and 64-bit count guarantees as before.
Validation
Tests compare normal and optimized scans with scalar expectations for all six comparisons, inclusive/exclusive range endpoints, reversed/equal bounds, lengths 0–33, and explicit signed/unsigned minimum, maximum and sign-bit boundaries. They check snapshot preservation after the source binding changes and across successive calls. Short-circuit side effects and skipped overflowing arithmetic remain scalar. IR/assembly checks confirm vector emission for runtime, named, constant/captured mixed and reversed comparisons, and scalar fallback for borrowed captures, arithmetic and disjunction.
The protected-page fixture now covers constant, runtime-threshold and range scans over lengths 0–129 and a null empty owner. Each scenario checks actual compiled Plenty functions against a scalar oracle.
Performance method
experiments/simd/clist/run.py --scenario threshold and --scenario range
compare optimized pipelines with explicit scalar column loops. Both receive
their bounds as runtime function arguments from the C fixture. The scalar
reference has no per-row callback call; measurements therefore do not inflate
the SIMD benefit with callback overhead. The original constant-only reports
remain historical evidence, since their captured scalar reference now qualifies
for SIMD.
Each scenario uses one compiled object, four verified code placements, randomized execution order, one excluded warmup and three samples per variant. Internal elapsed time covers repeated scans; instruction/cycle counts also include fixture startup. Small scans expose setup/tail costs, and large hot scans are not a claim about cold-data bandwidth or application-wide speedup.
Final measurements
Measured on 2026-10-24 on the Ryzen 7 7840HS, pinned to CPU 8, with the final
Cranelift 0.137.0 release compiler. Two monitoring windows each sampled host
processes 24 times at half-second intervals during the respective runs and saw
no concurrent cargo or rustc. Raw reports are retained as
experiments/simd/clist/measurements-runtime-threshold.json and
measurements-runtime-range.json, including compiler/object hashes, checksums,
function addresses and all 144 measured executions across the two scenarios.
Each range spans the four placement medians:
| Query | Rows | Scalar ns/row | SIMD ns/row | Scalar instructions/row | SIMD instructions/row |
|---|---|---|---|---|---|
| Runtime threshold | 7 | 1.613–1.980 | 2.152–2.212 | 36.867 | 40.010 |
| Runtime threshold | 8,192 | 0.625–0.931 | 0.107–0.214 | 11.034 | 3.039 |
| Runtime threshold | 262,144 | 0.623–0.943 | 0.108–0.215 | 11.058 | 3.058 |
| Runtime range | 7 | 1.957–2.301 | 2.978–3.055 | 41.581 | 53.438 |
| Runtime range | 8,192 | 0.856–1.318 | 0.162–0.218 | 15.034 | 4.542 |
| Runtime range | 262,144 | 0.845–1.270 | 0.160–0.212 | 15.063 | 4.563 |
At matched placements, the large threshold scans are 2.91–8.71× faster and range scans 4.41–8.01× faster than the explicit scalar loops. Instruction reductions are stable across placements; elapsed time remains placement-sensitive, so the whole range matters. Seven-row pipeline scans are 12–34% slower for thresholds and 30–56% slower for ranges than those loops: closure/vector setup and scalar tails dominate. This does not compare against the old scalar callback pipeline, and is not evidence that every small query should vectorize. No new profitability threshold is inferred from these three sizes alone.
Generated function sizes are 640 bytes for the threshold pipeline versus 309 for its scalar loop, and 752 versus 352 for the range. The extension keeps the existing SSE2 baseline and does not add target dispatch or parallel execution.
Dataframes backed by Polars
Status: A bounded foundation is implemented as
std.dataframe. It uses a separately built optional Polars engine, bulk numeric transfers and explicit expression plans.clistremains a distinct lightweight mutable collection. The text below records the original spike and its rationale; the reference describes the supported API. See the original measurements and B76 in the backlog for future priorities.
Why use an existing engine?
The original columnar-list vision overlaps with a typed dataframe: ordinary records describe the data, columns provide efficient storage, and bulk operations expose parallelism. Polars already supplies a much wider analytical engine, including expressions, joins, grouping, file readers and query planning. Its lazy optimizer can push filters and column selection into scans and simplify expressions. Embedding it lets Plenty focus on a clear, safe language interface instead of implementing another analytical engine. Polars lazy optimizations.
The tested boundary
The spike pins Polars 0.55.2, disables default features and enables lazy
and parquet. Their transitive features are substantial; this is not a tiny
dependency. A separately compiled Rust shared library owns DataFrame,
LazyFrame, expressions and typed column builders. An ordinary Plenty trusted
interface wraps opaque handles in move-only classes with destructors. Existing
foreign-call support is sufficient: there are no compiler or runtime changes.
Plenty creates an expression tree; Polars optimizes and evaluates it. No Plenty callback runs once per row. For example, the experimental module supports:
groups = [0, 1, 0, 1]?
values = [10.0, 60.0, 80.0, 90.0]?
data = df.from_lists(&groups, &values)?
predicate = df.col("value")?.gt(df.lit(50.0)?)?
weighted = df.col("value")?.mul(df.lit(2.0)?)?.alias("weighted")?
total = df.col("weighted")?.sum()?.alias("total")?
result = data.lazy()?.filter(&predicate)?.with_column(&weighted)?
.group_by_sum(df.col("group")?, &total)?.collect()?
This illustrates the API, not new multiline syntax. The runnable, single-line
chain is in experiments/polars-embedding/example.plenty. It produces totals
160 and 300, then repeats the operation through a written and scanned Parquet
file. Polars runs inside the same process, with no Python or subprocess query
execution. Cranelift compiles the Plenty orchestration; Rust/LLVM compiles the
engine separately. Ordinary Plenty compilation does not invoke Rust or Cargo.
Types and semantics need their own design
The spike has typed i64 and f64 input builders and checked typed extraction.
It does not implement DataFrame[T]: column names and query result schemas
are checked by Polars at runtime. Only a two-column constructor and a small
expression subset are exposed. A production typed facade must distinguish
statically known schemas from schemas discovered in files.
Null extraction becomes Option[T], independently from query failure in
Result. Floating-point NaN remains a value; null is not converted to NaN or
zero. Polars defines the query’s arithmetic, aggregation, comparison and ordering
semantics. These are not silently promised to match an ordinary Plenty loop:
parallel floating reductions can change rounding, groups have unspecified order,
nulls require explicit policies, and integer overflow needs a deliberate contract.
The executable check observes that Polars keeps NaN in a value > 50.0 filter,
whereas Plenty’s ordinary floating comparison returns false. Translating an
ordinary Plenty predicate into this expression without an explicit semantic
boundary would therefore change the program’s result.
The spike only builds floating multiplication and aggregates; it does not claim
checked Plenty integer arithmetic inside a query.
Polars data types.
Handles retain Polars-owned data after the Plenty source frame dies. Inputs are
borrowed by calls, query operations clone native handles, and errors leave
inputs owned by the caller. Native destruction uses the native allocator.
Recoverable failures currently have coarse error categories. Ordinary Rust
panics are caught at the boundary, but allocation failure may abort the process:
Result does not imply Plenty’s recoverable AllocError contract. Rich error
messages and a documented allocation policy remain important production choices.
Copies and deployment
Current foreign adapters support scalars and borrowed text, but no numeric slice
borrowing. The spike copies lists element by element into native typed builders,
then materializes Arrow columns. Small results are copied back one scalar at a
time. This is intentionally measured. A bulk numeric-buffer adapter would avoid
per-element calls and intermediate Option vectors even without zero-copy.
True zero-copy needs compatible layout, an owner that outlives the array, correct
alignment and null representation, and safe mutation restrictions. Ordinary
Plenty lists use tagged slots; they are not Arrow buffers. Even clist columns
cannot simply be handed over without an ownership/release protocol. Parquet
scans avoid the initial Plenty-list conversion altogether.
The engine is optional and prebuilt, with a fixed feature set and its own deployment artifact. The measured build uses x86-64-v3; this is a chosen spike hardware floor, not runtime ISA selection or Plenty’s normal baseline. A product must choose supported hardware/targets and package compatible binaries. Statically adding another independently built Rust runtime archive is not assumed safe; the shared-library boundary keeps allocation and Rust ownership on their own side.
Keep current clist behavior available while evaluating this direction. The
spike tests analytical functionality, not row-mutation compatibility, full-schema
typing, API completeness, or a migration. Its results can support prioritizing
Polars integration without claiming that every small mutable columnar collection
should pay for a query engine.
Polars embedding evidence
The proposal describes the boundary and semantic limits.
Reproduction scripts and raw observations are in experiments/polars-embedding/.
The experiment is separate from the compiler’s Cargo workspace and dependency
graph. Normal Plenty builds do not compile or link Polars.
What runs
- A Plenty-native facade constructs two typed columns from Plenty lists.
- Explicit expressions filter values, derive a column, group by a key and sum.
- The same query runs on a native lazy Parquet scan, returning values to Plenty.
- Negative cases cover missing columns, mismatched input lengths, bounds and wrong-type extraction. Null and NaN remain distinct, negative zero survives extraction, empty schemas work, and a query outlives its source builders/frame.
- Private constructors prevent source programs from manufacturing invalid native owners. Rust tests exercise null handles, unchanged outputs after errors and caught ordinary panics.
Measurement method
run.py compares the same deterministic data and equivalent filter/count or
filter/multiply/group/sum work in Plenty with Polars, direct Rust with Polars,
and a hand-written Plenty clist implementation. Input generation, import and
repeated query execution are separate measurements. The grouped clist code
specializes to eight known integer groups; it is not a general hash grouping
implementation. All results are checked against an independent checksum.
Polars queries are built once and collected repeatedly, with five warmups. Query
times include collection and copying the scalar or eight aggregated values out.
They do not include input conversion or startup. Trials rotate implementation
order. One-thread and four-thread settings use distinct physical cores; clist
remains serial. Tiny and million-row inputs expose dispatch overhead as well as
scan throughput. The startup experiment measures process launch with warm file
cache, not cold disk load or the first query’s thread-pool initialization.
The built shared engine is copied beside Plenty binaries and found through
$ORIGIN. Compilation timings measure optimized whole-program compile/link
with caches disabled and a repeated cache hit with the engine already built.
They are not a claim of separate Plenty module compilation.
Observations
Measured on 2026-10-25, AMD Ryzen 7 7840HS, Rust 1.99/LLVM 23.1.1. The engine uses explicit x86-64-v3 flags, stable Rust and no Polars nightly feature. Plenty programs retain their normal x86-64 baseline. CPU affinity is 2 for one thread and 2,4,6,8 for four threads. Seven rotated trials are recorded, with 500 query repetitions at 1,024 rows and 20 repetitions at one million rows.
| Rows | Query | Polars threads | Plenty + Polars | Direct Rust + Polars | Plenty clist |
|---|---|---|---|---|---|
| 1,024 | Filter/count | 1 | 41.3 µs | 39.8 µs | 0.96 µs |
| 1,024 | Filter/group/sum | 1 | 72.9 µs | 70.1 µs | 4.11 µs |
| 1,000,000 | Filter/count | 1 | 0.728 ms | 0.756 ms | 0.935 ms |
| 1,000,000 | Filter/group/sum | 1 | 4.574 ms | 7.982 ms | 3.941 ms |
| 1,000,000 | Filter/count | 4 | 0.710 ms | 0.709 ms | 0.961 ms |
| 1,000,000 | Filter/group/sum | 4 | 2.721 ms | 2.723 ms | 3.922 ms |
The large four-thread grouped query is about 1.44× faster than this specialized
clist loop, while small queries pay substantial engine overhead. The count
comparison uses f64, not clist’s SIMD-specialized integer filter/count path.
The wrapper and direct Rust paths are close in most cases. One-thread large
grouping is an unresolved exception: the separate direct Rust executable is
consistently slower. Removing an unnecessary rename from its plan did not remove
the difference. These are distinct native binaries and construction paths; this
bounded spike does not attribute that difference to wrapper overhead or claim
that calling Polars from Plenty speeds up Polars itself.
For one million rows, constructing the two Plenty input lists costs about
95–96 ms, then copying into Polars costs about 36 ms. Direct Rust constructs its
inputs in about 5.8 ms and imports them in about 6.3 ms. Building clist from the
same Plenty lists costs about 41 ms. A single sub-millisecond query does not
amortize these conversions. Keep data inside the engine across operations, read
files directly where appropriate, and evaluate bulk transfer before promising
efficient mixed Plenty/Polars workloads.
The instruction-count cross-check covers the whole process, including input
generation/import, 100 grouped queries and Polars’ five warmups. It records
about 9.62 billion user instructions for Plenty + Polars and 12.18 billion for
clist, yet the corresponding cycles are about 2.78 and 2.46 billion. These are
not isolated kernel counts or a substitute for the measured query times.
Build and distribution costs
The clean optimized engine build takes 279.9 seconds at three Cargo jobs, with dependencies already downloaded. GNU time reports 1,709,584 KiB maximum process RSS; this is not total parallel-build memory. The lockfile contains 332 packages including the adapter. A later direct-baseline-only rebuild takes 1.12 seconds. The clean-build observation predates removal of an unused helper and that baseline adjustment; it measures the same dependency and engine scope.
The shared library is 90.4 MiB, stripped of debug information. A Plenty
benchmark executable is about 3.2 MiB and uses that shared artifact. The
optimized Plenty/Polars benchmark compiles and links in about 53 ms without
caches, or 41 ms with a warm cache; clist is about 49/41 ms. These
process-level times include compiler startup and runtime extraction. They do
not include building the engine. This is a good separation of engine build cost
from everyday Plenty compilation, but a material distribution dependency.
Warm-file-cache launch with a single expression construction is about 2.79 ms,
versus 1.15 ms for the plain executable. That includes subprocess and
taskset overhead; it does not measure a cold launch or first-query latency.
Assessment
Embedding is tractable with the existing language boundary. Explicit expressions reach Polars’ optimizer, broad analytical operations and Parquet work without a compiler change, and the engine can be shipped precompiled. The results support prioritizing this direction over building those analytical facilities ourselves.
They do not establish a drop-in replacement for clist. Scalar/list transfer,
typed versus discovered schemas, null/NaN/overflow semantics, diagnostics and
native allocation failure need deliberate contracts. Tiny collection operations
remain much cheaper without a query engine. Current clist users need not
migrate as a consequence of this experiment.
Exploring async/await without exposed pinning
Status: Not implemented; exploratory proposal with a standalone native spike. Async/await is not accepted or scheduled for Plenty. The user requested this investigation while the language can still change freely, despite its original deliberate exclusion of async/await. Keeping it out remains a valid outcome. The backlog records that boundary; concurrency and generators describe what works today. Nothing here changes the tutorial or production compiler.
The question and provisional answer
Could Python-shaped async syntax coexist with Plenty’s explicit, fallible
allocation and deterministic ownership, without exposing Pin or requiring
asynchronous destructors? A restricted design looks possible: owned, concrete
movable frames; no surviving borrow across suspension; explicit task admission;
and synchronous cancellation only for operations whose pending work can be
safely abandoned. The native spike demonstrates those
mechanics for a tiny language. It does not demonstrate that the restrictions
are ergonomic enough for HTTP/TLS or that a production scheduler is affordable.
This could improve simplicity for Plenty’s narrower contract, not improve on Python or Rust in general. In particular, avoiding arbitrary borrows across await gives up a major capability of Rust’s design. Refusing implicit async cleanup gives up automatic completion of some protocol obligations. Moving those costs into adapters or a runtime does not make them disappear.
What to borrow from Python and Rust
Python separates a coroutine from a scheduled task: calling an async function
creates a coroutine, while awaiting or scheduling drives it. async def and
prefix await fit Plenty’s surface syntax. Python also specifies separate
asynchronous context-manager operations; these are more than alternate spelling
for ordinary deterministic drop. PEP 492
describes that syntax and protocol separation.
Python task groups provide an attractive execution-ownership model: their exit
waits for child tasks, and failures initiate sibling cancellation. Cancellation
is delivered through CancelledError, can be suppressed, and needs careful
handling in cleanup. Task creation can also opt into eager execution, so Python
is not a mandate for one immutable scheduling rule. Plenty should specify lazy
construction and task admission itself. Python task documentation
is the comparison source, consulted 2026-10-22.
Rust’s await machinery polls a child and suspends its parent on Pending;
its completion output can itself be a Result. That separation of suspension
from failure suits Plenty. Rust’s await reference
and Future contract
also make readiness notification and the poll interface explicit.
Rust permits address-sensitive generated state, including a reference to
another field in the same future. Pinning protects the resulting address
invariant; it is not arbitrary syntax that can simply be deleted.
The pinning documentation explains that
motivation. Rust’s AsyncDrop remains an experimental nightly API in the
documentation consulted.
This proposal avoids relying on it and makes no claim that Rust’s cleanup
design is settled or inherently impossible to simplify.
Candidate surface and evaluation order
The following is proposed Plenty syntax, not accepted source:
async def fetch(request: Request) -> Result[Response, TransportError | AllocError]:
response = (await exchange(request))?
Ok(response)
fetch(request) evaluates arguments immediately and moves/copies them into a
concrete Coroutine[Result[Response, ...]]. It does not execute the body or
schedule anything. Frame construction is inline and adds no allocation; argument
expressions retain their own failures. await consumes a coroutine and produces
its declared result. It may finish immediately or suspend; it does not imply
another task, a thread, a failure, or a fairness yield. Parentheses deliberately
leave await/? precedence undecided. ? propagates only the result error and
destroys active owned locals through the ordinary cleanup path.
Like today’s Generator[T], Coroutine[T] would retain a concrete producer
identity rather than secretly box different producers into one type. Returning
coroutines from factories, generic specialization, frame-size limits, recursive
async calls and containers need explicit contracts. Initially reject recursive
inline layouts and erased/dynamic coroutine storage. A coroutine value is affine:
it cannot be copied or awaited twice. Dropping an unstarted one destroys captures
without executing the body; this is a deliberate difference from Python’s
unawaited-coroutine warning behavior.
async would be legal only on functions in the initial scope. await would be
legal only inside them, plus a library root runner that consumes a coroutine.
No implicit async main, async iteration, async closures, arbitrary awaitable
protocol, async generators or async with follows from adopting this core.
These restrictions keep a possible first implementation independently assessable.
Scheduling is a separate choice
Direct await is sequential composition. To overlap work, an explicit bounded
scope admits multiple roots. A candidate API is
scope.start(coroutine) -> Result[Task[T], StartError[Coroutine[T]]]: capacity or
allocation failure returns the entire unstarted owner. The scope owns execution;
dropping a public handle must not orphan tasks. Results need their own owned,
allocation-audited storage and a policy when unclaimed. Prefer one result
consumer over implicit reference-counted shared results.
The minimal affine scope can synchronously cancel its remaining abort-safe children at exit. A stronger scope promising asynchronous graceful completion needs explicit awaited closure or retained responsibility elsewhere. It cannot silently borrow an enclosing stack and let that stack return. The distinction is developed in ownership and cleanup.
Existing Future[T] represents
an executor job’s result and has different drop/cancellation rules. Do not
reinterpret it as a coroutine or make .result() secretly await. Blocking
code called from async remains blocking. A future adapter must submit work
fallibly, wake the async consumer safely, preserve result/error ownership, and
define cancellation when the worker continues. No such adapter exists here.
Alternatives and the decision still needed
| Design | Benefit | Cost or unresolved boundary |
|---|---|---|
| Keep current threads, bounded executors and readiness APIs | No second calling convention or async ecosystem; existing fallible admission | Many mostly idle connections may need manual state machines or many native stacks; measure a real workload |
| Owned movable async frames (spike) | Inline composition, no pin API or frame allocation per call | No stored loan across suspension; ownership-return I/O can be cumbersome |
| Runtime-owned stable frames | Potentially allow broader borrows without user-facing pin types | Fallible per-frame/arena reservation, lifetime proof, no relocation, fragmentation; address stability is internalized, not removed |
| Stackful tasks / goroutine-style syntax | Synchronous-looking call chains can suspend deeply | Fixed stacks consume reserved memory and can exhaust; growing/moving stacks must preserve pointers, FFI and allocation failures; blocking calls and task admission still need policies |
| General Rust-like borrowed futures | Broad zero-copy composition and explicit low-level control | Address-sensitive frame projection and poll contracts recreate the complexity we want to avoid |
The independent stackful-task exploration investigates bounded fallible stack creation, guard pages and overflow behavior, cross-thread movement, foreign calls and cancellation. Stackful tasks are an alternative to syntax-directed stackless transformation, not a consequence of it. Current OS workers already provide one stackful baseline. Neither syntax alone solves protocol cleanup.
Before considering adoption, compare at least one bounded connection workload and one file/worker workflow using existing APIs and each plausible prototype. Measure allocations, reserved/live memory, throughput, latency, cancellation latency, compile time and API friction. The spike establishes neither a performance win nor a requirement for async. The main open decision is whether owned-operation ergonomics justify a second execution model; a full borrowing redesign should not be smuggled in as a small syntax convenience.
Suspension, allocation and cleanup boundaries
Companion to the unadopted async/await exploration. These are candidate constraints, not implemented Plenty contracts. The spike exercises only the small subset explicitly identified there.
Movable state without pinning
An inline frame can move between polls only if moving it cannot invalidate a
surviving reference. Scalars, owned heap handles and owned child frames fit that
rule when children obey it recursively. A short borrow during one poll is fine
if it ends before returning Pending. A pointer into an independently owned
heap buffer can stay valid when its handle moves, but owner/lifetime and
concurrent-access rules still need proof; a pointer into the frame cannot.
Start from the restrictions already used for generators: no borrowed parameters/captures, and no live loan across suspension. Extend the check through awaited children and hidden operation state, not just syntactically obvious local variables. A child awaiting a borrow into its parent’s frame makes the combined frame address-sensitive even if the parent source declares no named reference. Compiler-created temporaries must be checked too.
async def invalid() -> Result[(), IoError]:
mut bytes = ...
count = await read_into(&mut bytes) # rejected under the movable design
Ok(())
The straightforward alternative transfers owned state:
async def receive(stream: Stream, buffer: Bytes) -> ReadOutcome:
await read_owned(stream, buffer)
ReadOutcome = ReadOk(Stream, Bytes, Count)
| ReadErr(Stream, Bytes, Cause)
These names are illustrative. An error may need to distinguish a reusable stream from one whose framing is uncertain. Losing the stream on cancellation may be safe locally but undesirable; an ownership-return cancellation API can preserve it. Returning buffer/stream owners repeatedly is the chief ergonomic cost, especially with nested protocol layers and multiple buffers. Ordinary borrowed readers cannot automatically become async readers. The spike does not validate this API or Plenty’s generic ownership checker.
Reject frame-address escapes through FFI, callbacks, readiness registrations, self-references and user-defined awaitables until they have checked contracts. Use stable task identities or independently owned registration records for wake notifications; never retain an inline frame pointer in the reactor. Deregister before freeing the associated storage, including stale wakeups and concurrent completion races. If an OS operation retains a buffer until completion, dropping the frame cannot free that buffer merely because cancellation was requested.
A stable-storage alternative can hide address stability behind a consuming
owner API. That removes Pin from user spelling, but still requires non-moving
storage and lifetime/drop-order proofs inside the compiler/runtime. It also
changes fallible construction and task admission. Neither the spike nor existing
generator implementation proves that alternative sound.
An allocation ledger, including failures
| Boundary | Candidate contract | Evidence / remaining work |
|---|---|---|
| Inline coroutine creation | No new heap allocation; argument failures remain visible | Scalar captures demonstrated; owned captures and general layouts unimplemented |
| Nested await | Child frame stored inline; poll/resume bookkeeping does not allocate | Demonstrated, including nested state moved after suspension |
| Body allocation | Ordinary Result with AllocError, propagated explicitly | Spike uses a numeric status and tests injected failures; Plenty Result lowering unimplemented |
| Root admission | Reserve slot/result/wake storage before transferring owner; return owner on failure | One fallible root allocation and fixed two-slot queue demonstrated |
| Queue growth / saturation | Explicit fallible reservation or fixed capacity with a capacity error | Fixed capacity demonstrated; no dynamic growth |
| Wake/registration/timer storage | Reserve fallibly before registration; one owned cleanup path | Not implemented; cannot assume that waking or cloning a waker is free |
| Erased coroutine or result | Explicit fallible boxing/storage API | Not implemented; no implicit boxing fallback |
| Cancellation/drop | No allocator-dependent success path for mandatory synchronous release | Memory-only resources demonstrated; native handles and user cleanup unaudited |
The guarantee concerns coroutine machinery; caller expressions and user cleanup retain their own effects. Compile-time layout computation and compiler memory are outside the generated program’s allocation contract. Inline frames can be large even with no heap allocation. Set layout/nesting limits and diagnose excess rather than silently box. A runtime stack limit is not a recoverable heap allocation error. Recursive async calls need an explicit heap indirection or an initial rejection, as recursive generator layouts already do.
Cancellation is not asynchronous destruction
For the narrow candidate, dropping a suspended coroutine synchronously destroys
its live child and owned locals in reverse acquisition order. It does not resume
source statements, invoke arbitrary await, allocate an emergency cleanup task,
or promise that a remote peer rolled back work. The spike uses exactly this
rule. Scope cancellation can stop and destroy a suspended task on the executor
thread; cancellation cannot interrupt a currently executing poll or a blocking
foreign call. Long CPU work needs explicit cooperative checkpoints.
Only operations with a safe synchronous fallback can use that rule. Closing a
local socket may be an acceptable fallback to skipping a graceful protocol
shutdown; leaving a kernel operation referencing freed memory is never one.
The relevant property is not just a destructor returning (): it includes
in-flight registrations, external buffer ownership and the cost of unregistering.
A potentially blocking destructor also defeats bounded cancellation latency.
Do not generalize from the spike’s free() to all existing Plenty values.
An explicit await finish(owner) can attempt graceful shutdown and return its
errors. The program still needs an admissible fallback if it is cancelled during
that await. Initially exclude awaiting inside with, async context managers,
async destructors and operations with mandatory suspending cleanup. This is a
restriction, not a promise that all cleanup becomes simple.
For a protocol requiring completion, a scope/runtime must retain and drive the
owner to a permitted terminal state, or return it to someone responsible. If
that can suspend, synchronous scope exit cannot provide a nonblocking completion
guarantee. Choices are explicit awaited closure, blocking exit with possible
deadlock, or transfer to an explicitly owned supervisor. Each needs failure and
shutdown rules; none belongs invisibly in drop.
This agrees with linear integration: affine
abandonment cannot carry a future linear obligation. async with alone supplies
a location for cleanup code, not an owner guaranteed to execute it. Initial async
frames should exclude non-discardable values if the separate linear proposal
lands. Cancellation shields can delay a request, but cannot prove termination,
repair allocation failure, or outlive a borrowed resource.
Scope and error questions still open
A production scope needs rules for the first child error, siblings still running, cancellation result ownership, unclaimed results and multiple errors. Fail-fast cancellation can be useful, but aggregating errors must not introduce infallible allocations. A bounded result array, first error plus explicitly discardable later results, or a fallible collection step are distinct policies.
The experiment’s task cancellation returns a separate CANCELLED status and
ordinary child errors keep their numeric code. In Plenty, await coroutine
should return the coroutine’s declared output; awaiting a task handle may need
an additional scheduler cancellation result. Do not silently inject cancellation
into every function’s error set or pretend graceful cancellation is just a
transport error. The spelling and exact ownership of those outcomes remain open.
Native async/await spike evidence
Companion to the unadopted proposal. Evidence collected
on 2026-10-22 against local main at 9c994af0. This is a standalone compiler
experiment under experiments/async-await,
not a modification to the Plenty parser, checker, code generator or runtime.
What actually executes
A small Python program parses a restricted Python-shaped source grammar and emits C state machines. Python supplies an AST, not coroutine execution. A C11 compiler produces a native executable; a C harness polls it, moves suspended frames, injects allocation failures and checks cleanup. No Rust futures, pinning, GC, async library, hidden thread or manually substituted source result is used. The source really contains async functions and await expressions:
async def leaf(value: i64) -> i64:
resource = acquire()
result = await tick(value)
return result + 1
async def parent(value: i64) -> i64:
resource = acquire()
first = await leaf(value)
second = await tick(first)
return second + 1
This syntax is accepted only by the spike. It is neither ordinary Python
(the intrinsics and i64 are experiment-specific) nor current Plenty. Numeric
error propagation is implicit in this small compiler; the proposed Plenty
Result/? syntax is not implemented. There is no I/O backend: tick returns
pending once and then its value. Real nonblocking I/O and registration lifetimes
remain a separate adoption gate.
Each await has an initialization state and a resume state. Initialization runs once; pending returns preserve the child frame and all live locals. Ready results flow to the next statement in the same poll. Cleanup flags distinguish initialized resources/children from untouched fields; completion and error both release them. A child frame contains no pointer to its parent. The harness moves frames by transferring their bytes and invalidating the source, preserving owned heap handles. C does not enforce ownership; the restricted generator and trusted harness do. This is mechanical evidence, not a proof about arbitrary Plenty programs.
Reproduce and inspect
From the task/repository root:
python3 -B experiments/async-await/test_spike.py
SANITIZE=1 python3 -B experiments/async-await/test_spike.py
python3 -B experiments/async-await/lower.py experiments/async-await/program.async
The first two commands compile and execute temporary artifacts and report test
status. The third prints generated C for inspection. No dependencies beyond
Python 3 and a C11 compiler are required; CC selects the compiler. See the
README for the exact subset and
host/compiler allocation boundary.
Validation passed with the default C compiler and with Clang’s address,
undefined-behavior and leak sanitizers. The default GCC sanitizer link failed
because its runtime shared libraries were unavailable; Clang supplied a working
alternative. LeakSanitizer required execution outside the sandbox’s ptrace
wrapper. Both Python test methods passed, including twelve rejection subcases
and all native harness assertions. Book link checking and mdbook build book
also passed. The production compiler/runtime suite was not run: neither their
code nor executable tutorial examples changed.
Observations and their limits
| Experiment | Observed result | What it supports |
|---|---|---|
| Create/drop without polling | Zero allocation attempts and no body effects | Lazy inline construction for scalar captures |
Nested parent(40) | Two suspensions, then 42; two body resource allocations | Await composition needs no additional frame allocations |
| Move while parent and child are suspended | Resumes correctly; child then parent release exactly once | This restricted pointer-free frame layout can move after suspension |
| Drop before start / at each suspension | Zero surviving allocations; drop makes no allocation attempts | Synchronous cancellation works for the experiment’s memory-only owners |
| Child failure | Error code 71 survives parent propagation; both resources released | Error and completion cleanup paths can share a state-machine structure |
| Fail allocation 1, 2 or 3 | Admission failure returns intact frame; parent/child failures release acquired owners | Allocation failure need not abort or leak in these paths |
| Fail admission of a suspended frame | Byte-identical source still owns both resources; retry then cancellation releases everything | Reservation-before-transfer works beyond unstarted frames |
| Two roots and a third submission | Both progress; third gets capacity failure without allocation or transfer | Bounded explicit scheduling can preserve failed-submission ownership |
| Close scope while pending and allocator disabled | All remaining resources/tasks released without allocating | A restricted affine scope can cancel synchronously |
| Twelve unsupported-source cases | Rejected before C compilation | Borrowed parameters, recursion, loops, async context managers, escapes/copies and invalid calls are not silently accepted |
On this x86-64 host, frame sizes were: leaf 56 bytes, parent 128 bytes, failing leaf 56 bytes, propagating parent 96 bytes; two-slot scope 96 bytes. The successful scheduled parent performs three allocations: one root frame and two explicit 16-byte body resources. Direct polling performs only the latter two. The generated parent retains separate storage for its two await sites instead of overlaying disjoint lifetimes. Frame-size reduction is possible but not demonstrated. These figures are layout observations, not throughput or memory comparisons with Rust, Python, threads or real Plenty.
What changed our assessment
The basic state machine and failure rollback are small when every suspended value is owned and destruction is synchronous. That makes a constrained language extension plausible. The experiment also exposes why the restriction matters: admitting even one child borrow into its parent would invalidate its simple relocation test. A stable allocation, offset-based references with careful external-pointer rules, or a pin-like invariant would then be necessary.
The substantial unanswered work is at the library boundary. A buffer-owning read API must return owners on failure and cancellation; a reactor must never retain a movable frame pointer; an in-flight native operation may outlive a cancel request. None of those problems is exercised by a synthetic tick. Likewise, memory-only drop cannot establish protocol completion. The result is a reason to evaluate owned-operation ergonomics, not evidence that we have eliminated the hard parts of asynchronous I/O.
Production work would touch parsing/AST, concrete coroutine identity and generic
specialization, suspension-aware loans, return/error lowering, partial
initialization and cleanup, native-effect certification, diagnostics/cache
identity, runtime wake/task storage, every exposed async I/O adapter and the
tutorial. Existing generator machinery may be reusable, but its absence of
body ?, return values and nested await means this is not just a new keyword.
The spike omits branches, loops, arbitrary captures/results and user-defined
destructors; it estimates no calendar schedule or compiler performance.
Until a realistic owned I/O prototype and a comparison with current workers justify those costs, leaving async/await out remains the conservative choice.
Exploring Go-style tasks with Plenty’s ownership rules
Status: Not implemented; exploratory proposal with a standalone native spike. Lightweight tasks are neither adopted nor scheduled for Plenty. The backlog records the investigation; the concurrency reference describes the existing native workers. The production compiler, runtime and tutorial are unchanged. Keeping the current concurrency model is a valid outcome.
The question and provisional answer
Can ordinary functions suspend on communication, retain their local borrows, and clean up synchronously, without an exposed pinning API or async destructors? Scoped tasks on immovable, fixed-size stacks look like a plausible smaller model. The spike exercises actual stack switching, nested calls with stack-local references, bounded communication, cancellation and fallible admission. It does not establish a production implementation or the scalability normally associated with Go.
This is simpler at the function boundary than movable async frames: the call stack retains ordinary locals and nested calls; a callee can keep borrowing its caller’s local across a suspension. The simplification costs a stack reservation per task, fatal exhaustion unless another design is chosen, suspension effects that must still be checked, and a scheduler-aware I/O boundary. Full Go-style growable-stack M:N execution is a much larger commitment. The runtime choices separate these options.
What to borrow from Go
Go starts ordinary function calls concurrently, and its channels provide typed
communication. Arguments are evaluated in the starting goroutine. The call’s
return values are discarded by a go statement. These are language semantics,
not a promise about which operating-system thread runs the function.
Source: Go specification.
For Plenty, retain ordinary function bodies and explicit communication, while
using a fallible library operation that returns a scoped result handle. Spawning
must preserve ownership on failure. Starting a task cannot silently succeed when
there is no stack or queue capacity; detached tasks would also conflict with the
initial lifetime/cleanup contract. No go expression, unbounded scheduler queue,
automatic GC, implicit channel allocation or process-wide hidden executor follows
from this proposal.
Go’s scheduler distributes goroutines over worker threads, with separate goroutine, worker and processor state. Its runtime includes parking, syscall transitions and work distribution. That implementation is evidence of the work behind the surface, not a minimum dependency for a single-worker experiment. Source: runtime scheduler.
Candidate surface
Prefer library APIs and familiar with, explicit captures, Result/?, concrete
jobs and consuming result handles. This sketch is proposed API spelling;
neither std.task nor these error types exist:
import std.task as task
def answer(value: i64) -> Result[i64, task.Cancelled]:
task.checkpoint()?
Ok(value + 1)
def compute() -> Result[i64, task.ScopeError | task.StartError[()] | task.Cancelled]:
with task.scope(max_tasks=8, stack_bytes=65536)? as scope:
with scope.spawn(def once () -> Result[i64, task.Cancelled]: answer(41)).drop_job()? as work:
return Ok(work.join()?)
The sketch omits typed channel construction; it is not a proposed rewrite of the
existing std.thread or std.sync APIs. Names and whether scope/handle contexts
should be nested remain open. Keep task types distinct until their ownership and
wait behavior actually satisfy the native-thread contracts.
The parent evaluates captures and arguments first. scope.spawn(job) attempts
admission, commits ownership only on success and schedules the body for later
execution. Failure returns the unstarted concrete job, following
consuming native jobs.
An allocation-free .drop_job() conversion would discard that job deliberately.
A scope reserves its task table and ready/wait bookkeeping fallibly. Capacity
exhaustion is a separate Full(job) result; spawn_nowait is the initial policy.
Waiting for capacity while holding a dependency can deadlock, so blocking
admission needs an explicit later contract. A completed slot may be reused only
after its result/handle lifetime is resolved; the spike does not implement reuse.
Channels have explicit positive capacity, fallible construction and owned messages. A successful send commits ownership exactly once. Full/nonblocking, closed and cancelled send errors return the unsent value without allocating; a parked send retains its value in its task until it commits or returns it. Wait nodes are reserved with admission, and park/wake/cancel never grow a queue. Dropping the last sender wakes receivers to observe closure; closing the receiver side wakes senders with their values. Cancellation is checked before a new transfer commits; an already committed transfer is not rolled back. Returning a cancelled unsent value must still respect linear obligations. Rendezvous and general selection are outside the first candidate. These are proposed contracts, not behaviors proved by the integer-only spike.
join() drives or yields to the owning scheduler until completion, then moves
the result once. Returning a Result is ordinary task output, not a request to
cancel siblings. Scope teardown joins all admitted children on every normal exit,
including ?; unclaimed results are dropped once. Parent errors cannot erase
child ownership or silently discard unresolved linear obligations. Existing
linear integration remains a separate proposed contract.
Loans, suspension and migration
An ordinary stack remains at the same virtual address throughout task execution. Task-local loans and loans down its synchronous call chain may survive a yield without becoming movable self-referential frame fields. No public pin wrapper is needed. The runtime still promises address stability internally; hiding the promise does not remove its implementation obligations.
Initially accept wholly owned concrete jobs/results and prohibit references in messages, escaping callbacks and stored task handles. Within one task, preserve ordinary lifetime, exclusive-loan, owner-move and container-invalidation checks. Interleaving on one OS thread never authorizes two exclusive loans. Adding parent loans later should reuse scoped-worker reasoning: join before the owner dies, prevent conflicting parent accesses throughout the scope, and reject cycles. Do not infer those rules from the C experiment, which has no borrow checker.
Start with one scheduler bound to one native thread. Cross-thread task creation, handle transfer and migration are excluded. Owned inputs must still satisfy a defined task effect/eligibility check; existing native-worker certification does not prove that an operation is safe to suspend. A future M:N mode needs structural thread eligibility for captures, every live local, references, callbacks and destructors, and a contract for thread-local state and foreign handles. Merely making the initial closure transferable is insufficient.
Ordinary syntax can hide significant suspension. Infer a separate transitive
may suspend effect for task operations and their callers; validate scheduler
context at every reachable entry. A task-only helper must be rejected from an
ordinary non-task entry. Calls through stored callbacks require evidence too.
Reject suspension while a mutex guard, an uncertified foreign call, or a
destructor is active. A pure helper can be called in either context. These checks
may be a substantial compiler addition even without async/await keywords.
Cancellation and deterministic cleanup
Cancellation is a request, observed at task-aware waits and explicit checkpoints.
It wakes parked channel/I/O waits and returns a typed cancellation error. The
function then returns through ordinary cleanup and ? paths. Cleanup runs
synchronously on that task’s retained stack before join releases the stack.
Destructors cannot suspend, recursively drive the scheduler or allocate hidden
task storage. Explicit fallible finishing operations handle protocol shutdown
before ordinary destruction. A linear obligation cannot be abandoned merely
because cancellation was requested.
Do not free a suspended live stack, asynchronously throw into arbitrary native
code, use longjmp across destructors, or treat signal-handler unwinding as safe
cancellation. A running task may ignore cancellation; scope exit can consequently
wait forever. Fatal traps retain Plenty’s existing no-unwind behavior.
Synchronous destruction also does not imply nonblocking destruction: a blocking
close/drop path can stall this worker. Real adapters need that audit.
Go’s pipeline guidance explicitly teaches cancellation when downstream consumers stop early; convenient spawning and channels do not arrange application lifetime automatically. Plenty should make the scope boundary explicit and preserve its existing endpoint/join deadlock checks. Those checks are not a deadlock proof. Source: Go pipelines and cancellation.
Comparison with the async exploration
The independent async/await exploration describes restricted movable frames, with its ownership contract and native spike evidence. Both experiments are available for comparison; neither adds a production feature or depends on the other’s implementation.
| Question | Restricted movable async frames in that exploration | Fixed-stack task candidate here |
|---|---|---|
| Suspension syntax | Explicit await inside async functions | Ordinary calls; inferred suspension effects |
| References across suspension | Initially excluded, including hidden child borrows | Task-local call-chain borrows can remain at stable addresses |
| Storage | Concrete frames inline; explicit fallible root admission | Explicit fallible task record and a whole stack reservation |
| Destruction | Synchronous cleanup of initialized frame fields | Ordinary synchronous stack cleanup after cooperative return |
| Cancellation | Pending operation must support safe abandonment | Task must resume, observe cancellation and return |
| Existing blocking functions | Need an owned nonblocking adapter or blocking pool | Also stall the worker unless adapted or offloaded |
| Runtime portability | State machines can use ordinary native calling conventions | Stack switching requires ABI/tooling/platform integration |
Both candidates avoid exposing pinning by narrowing their contracts. Neither establishes general async-drop, cancellation latency, linear resolution, real I/O suitability or a concurrency feature ready for adoption. The async spike’s small frame layouts and this spike’s fixed stacks are observations of different workloads, not a fair performance comparison. A full Go emulation is not an established simpler alternative to either restricted candidate.
Decision still open
The experiment makes fixed-stack tasks worth comparing with the async candidate, especially when borrowed-buffer helper chains matter. Adoption needs the same bounded TCP/HTTP workload, cancellation and failed-start tests, a realistic stack budget, I/O registration-lifetime checks and compiler effect/loan experiments. Measure throughput, p95 latency, committed/resident/virtual memory, admission failure and cleanup work under equal limits. Those acceptance gates are recorded only in the backlog. Keeping native workers, adopting a narrow task library, choosing restricted async, or adopting neither all remain possible.
Stack, scheduler and allocation choices
Companion to the Go-style task exploration. All task designs here are candidates; today’s native workers remain the implemented contract.
Four different implementation commitments
| Model | Simplification | Cost and boundary |
|---|---|---|
| Existing native workers/executors | OS manages stacks/scheduling; ordinary synchronous code and checked ownership | Each running worker uses a native thread; executor queues jobs, rather than suspending arbitrary call stacks |
| Fixed-stack cooperative tasks on one worker | Stable stack addresses; no relocation maps; explicit bounded scheduling | Per-task stack budget, no automatic preemption, task-aware blocking, fatal unchecked overflow |
| Fixed-stack tasks over several workers | Potential task concurrency and parallelism without stack copying | Cross-thread eligibility/migration, synchronization, wake races, TLS/FFI, work distribution, shutdown |
| Growable-stack M:N goroutines | Small initial stacks and scalable ordinary call chains | Compiler stack checks, relocation/segmentation, failure semantics, syscall handoff, preemption, pointer maps and tooling |
The one-worker experiment is stackful M:1 execution. It is not an M:N scheduler or a native-thread pool. Its stack switching shows that ordinary C calls can suspend below their callers, without manually translating every call into an async state machine. It demonstrates no million-task scalability, parallelism or end-to-end I/O speedup.
Stack choices and recoverable allocation
Go’s runtime checks stack bounds in generated code, allocates replacement stack storage, copies stacks and adjusts pointers. Its stack allocator has fatal OOM paths. Go’s stack machinery is integrated with its pointer metadata and runtime; removing GC does not supply the relocation machinery for free. Source: runtime stack implementation.
For Plenty, assess these independently:
- Fixed guarded stacks: choose size and maximum task count explicitly. Allocate task records and mapped stacks before transferring the job; failed reservation is an admission error. This avoids later stack-growth allocation and keeps interior addresses stable. It wastes address space and commits pages as touched; a guard trap still terminates unless a separate safe failure path exists. Large stack frames can skip a guard without stack probes.
- Reserved, incrementally committed stacks: stable virtual addresses can
reduce initial resident cost, but later commitment can fail. Reserving address
space or successful
mmapdoes not guarantee physical memory on Linux with overcommit. Signal-driven commitment does not automatically yield a typed, recoverable allocation error or safe cleanup. Platforms also have different reservation/commit contracts. - Movable contiguous stacks: compiler-generated pointer maps and provenance rules must account for interior pointers, registers, channel wait records, callbacks and foreign pointers. Stable heap allocation alone does not repair stack pointers. Such metadata can exist without tracing GC, but writing it is new compiler/runtime work. Restricting every borrow across every possible growth point largely sacrifices the benefit over restricted async.
- Segmented stacks: old addresses can stay stable, but calls/returns need an alternate ABI, segment metadata, foreign-call bridges and reclamation rules. Segment growth still allocates inside an ordinary call chain.
A recoverable stack-growth failure cannot simply appear as AllocError in an
arbitrary function declared -> i64. Choices are to propagate a new effect and
error through all possible growth points, reserve a compiler-proven complete
budget, restrict call graphs/recursion, or retain fatal exhaustion. Each changes
the source contract or implementation substantially. This proposal provisionally
uses fallible explicit admission with fatal unchecked stack exhaustion,
matching today’s native-worker exhaustion boundary. If the requirement is that
all later stack capacity failures must be recoverable, this candidate is
incomplete and should not be adopted on the present evidence.
Allocation ledger
Fallible allocation means all explicit recoverable resource requests are reported; it is not a promise that OS process termination can be recovered. The candidate must advertise the fixed-stack limitation directly.
| Request | Failure ownership and recovery |
|---|---|
| Scope task table, ready queue and wait records | Fallible construction; no live children on failed construction; preallocate bounded bookkeeping |
| Task record, stack reservation and native context setup | Return complete unstarted job; unwind partial setup before admission; no body execution |
| Slot capacity exhausted | Return Full(job) without allocating, distinct from resource failure |
| Bounded channel buffer and readiness registrations | Explicit fallible construction/registration, rollback on failure; no hidden allocation on park/wake/cancel |
| Task body allocation | Ordinary typed Result; return through normal initialized-resource cleanup |
| Result storage | Reserve with admission or keep result in task cell; preserve concrete size/type and exactly-once transfer/drop |
| Optional blocking worker submission | Explicit bounded fallible handoff, preserving job/inputs on failure |
| Fixed stack exhausted / OS kills process under memory pressure | Fatal boundary; no claim of a returned job error or reliable cleanup |
The spike accounts for queue, task record, stack mapping and body allocations. It does not implement real readiness registration, result objects, OS worker creation or operating-system physical-memory admission guarantees.
I/O, foreign calls and fairness
Today’s std.ready waits use a bounded native
poll; today’s channel/thread/executor waits block native threads. Running these
unchanged inside one cooperative worker would block every task on that worker.
The native operations are not automatically task-aware because ordinary syntax
looks similar.
A candidate task backend needs nonblocking handle operations, owned readiness registrations and cancellation that removes those registrations before the buffer/handle can drop. Borrowed stack buffers can remain stable while parked, but late completion must never access them after return or cancellation. A kernel operation that cannot be synchronously cancelled needs a separate owned operation/result lifetime or explicit completion; stackful syntax does not solve that problem. Partial read/write progress and typed transport errors must survive.
Initially reject blocking OS/FFI calls in task bodies through effect checks, or
provide an explicit bounded native-worker offload with owned transferable data.
An offload must be joined/completed before freeing captured resources; requesting
cancellation does not terminate its foreign call. DNS, file I/O, TLS and user
destructors need separate audits. Native thread TLS, errno, signal masks,
stack bounds and thread-affine native handles need defined treatment before any
migration; a scheduler cannot assume that saving CPU registers solves them.
Cooperative scheduling offers no preemption of CPU loops, arbitrary FFI or
destructors. Task-aware channel waits and explicit checkpoints provide known
suspension points; inference makes their reachability visible to the compiler.
Document fairness and bounded work per turn. A later preemptive scheduler must
define safe points, stack scanning and lock/runtime critical sections; forced
cancellation still must not interrupt cleanup. Debuggers, profiler unwinding,
sanitizers, guard pages, ABI register state and supported targets also need
qualification. Linux ucontext is only an experiment backend, not a portability
plan.
GC-free lifetime and scope exit
Stack and task records have one owning scope; channels transfer owned values; reference-counted shared owners follow today’s cycle restrictions. Ready/wait queues contain non-owning task identities whose validity is protected by the scope. No task survives scope completion. Drop order is explicit: request optional cancellation, resume children until ordinary return, transfer/drop results and captures, detach wait records, then reclaim task stacks and the scope. Do not destroy a scope while its scheduler context is still referenced.
A child waiting forever or ignoring cancellation prevents scope exit. The runtime can detect some states with no ready task and no external registrations, but cannot prove general deadlock freedom or safely free live stacks as a fallback. Joining parents/children and bounded channel dependencies retain the existing analysis problems. Smaller surface syntax changes none of these lifetime facts.
Native stackful-task spike evidence
Companion to the Go-style task exploration.
The isolated source lives in experiments/stackful-tasks/ at the repository
root. It is a C experiment, not accepted Plenty syntax or production code.
What actually executes
Linux/glibc getcontext/makecontext/swapcontext run two task call stacks on
one native thread. Each task owns a 64 KiB usable stack, surrounded by two
inaccessible pages, plus a separate heap task record. Ordinary C helper calls
perform channel waits and nested cooperative suspension. The scheduler returns
to its own stack between turns. No native thread is created; there is no GC,
async library, frame relocation, custom forced unwinding or language compiler.
The scope supports two lifetime task slots and one explicitly allocated channel slot. It has preallocated task identities, implicit ready order by table scan, and states for ready, send wait, receive wait and done. A send on a full channel or receive on an empty channel parks, and communication wakes the opposite waiters. This is a tiny test scheduler, with no slot reuse, endpoint closure, selection or general-purpose channel API. Messages are integers; affine-value transfer, failed-send ownership, endpoint closure races and linear obligations are not tested.
Admission allocates a task record, maps the guarded stack, makes its usable
pages accessible and prepares its native context. The input pointer changes
ownership only after those operations succeed. The failure injector covers
queue allocation, task-record allocation and stack mapping; real mmap,
mprotect and context-setup failures are checked, but only allocation sites are
injected. Capacity failure allocates nothing and retains the job. Stack mapping
failure releases partial task storage, and the original job can be retried.
Cancellation marks a task and makes parked tasks ready. Channel operations observe the request and return normally. The cleanup worker also yields from a nested helper holding a reference to a caller’s volatile stack local, mutates that local on resumption, and checks the result before exiting. It explicitly releases initialized heap resources in reverse order on success, cancellation and body allocation failure. Destruction refuses to reclaim any unfinished stack. These C cleanup paths model the candidate control flow; they do not implement Plenty destructors, its borrow checker, linearity or Result lowering.
Reproduce
From the task checkout, with a C11 compiler and Linux/glibc:
python3 -B experiments/stackful-tasks/test_spike.py
CC=clang SANITIZE=1 python3 -B experiments/stackful-tasks/test_spike.py
The runner creates a temporary executable, compiles with -O2 -Wall -Wextra -Werror -pedantic, runs assertions with a timeout, and removes the executable.
SANITIZE=1 adds UndefinedBehaviorSanitizer; it does not claim complete
alternate-stack instrumentation or AddressSanitizer coverage. The bounded
experiment is intended as a mechanically inspectable counterexample to needing
movable frames for every suspension, not a benchmark.
Observations
Observed on x86-64 Linux/glibc:
| Check | Observed result |
|---|---|
| 1,000 values through capacity-one channel | Sum 500,500; 999 send parks and 999 receive parks |
| Reference to stack local across nested suspension | Same retained local is read/modified after resume |
| Task record / usable stack / full mapping | 1,016 bytes / 65,536 bytes / 73,728 bytes with 4 KiB pages |
| Queue / task-record / stack allocation failures | Failure returned; partial admission storage released; unstarted job retained and retried |
| Full task table | Job retained, allocation attempt count unchanged |
| Cancellation before body entry | Captured job dropped by scope; body never starts |
| Cancellation of parked sender and receiver | Task resumes and returns cancellation before scope reclamation |
| Success / cancel / either body allocation failure | Initialized resources released once in reverse order; tracked live allocations and mappings return to zero |
| No ready task | Step reports no progress; explicit cancellation wakes test waiters |
Two 64 KiB stacks are enough for these tiny call chains. This is not a safe general default. At 10,000 tasks the demonstrated mapping size alone would reserve about 703 MiB of virtual address space, excluding task records, allocator metadata and other storage. This is arithmetic from the observed mapping size, not a scalability run or resident-memory measurement. Successful mapping does not guarantee later physical-memory availability. The guards are installed, but no overflow-recovery experiment is claimed; overflowing or skipping guards is outside the safe experiment contract.
The finite no-yield example returns without scheduling a peer; it exercises no preemption mechanism. There is no real blocking-I/O demo, readiness backend, fairness/latency measurement, syscall handoff, FFI migration, or scheduler performance comparison. The source’s explicit swap points already establish that a blocked native call cannot yield by magic.
What this changes in the design discussion
Actual stack retention makes an ordinary borrowed helper chain possible across suspension without a public pinning type. Synchronous cancellation needs resumption and return; stack retention does not provide safe arbitrary stack destruction. The experiment consequently supports a narrow scoped library candidate, while leaving the most consequential deployment questions open: stack budgets and exhaustion, real I/O lifetime, transitive suspension checks, thread eligibility, bounded cancellation and tooling portability.
The sibling async experiment avoids whole-stack reservations and demonstrates moving restricted owned frames, while prohibiting surviving references. This experiment retains references by making the stack immovable and paying for it. The experiments expose different costs; they do not establish a winner.
Runtime-independent networking protocols
Status: Proposed; not implemented as a general convention. The three-layer direction is agreed: codec, sans-I/O session, runtime driver. This proposal specifies the intended contracts, not new APIs available today. Existing TLS sessions and the resumable HTTP decoder demonstrate parts of the design; they do not imply that all current networking modules already use it. Remaining scope and priorities belong to B94 and B85 in the backlog.
Companion pages describe the session contract, runtime drivers, and WebSocket validation and acceptance. All interface sketches on these pages are hypothetical, not executable Plenty or final public declarations.
Problem and decision
Plenty supports both sequential networking and supervised actor networking. A protocol implementation should not need separate copies of its framing, validation, exchange state, close rules and retransmission logic for each. Generic socket wrappers help share code across TCP and TLS, but do not remove the assumption that a protocol operation can wait on its caller’s stack.
Adopt three layers for new stateful networking protocols:
- Codec: parse and serialize protocol representations incrementally.
- Session: own protocol state and advance it from explicit inputs without performing I/O or waiting.
- Driver: connect a session to a runtime, transport and application API.
Share the protocol semantics, not one universal execution model. Sequential callers retain convenient operations such as receiving a message; actor callers receive events and submit commands without blocking the stage. Both drive the same session implementation.
Existing foundations and boundaries
| Existing component | What it establishes | What it does not establish |
|---|---|---|
std.tls.Session | Socket-free TLS state, explicit ciphertext input/output, plaintext operations and EOF | A universal protocol interface or actor-independent configuration ownership |
std.http.decoder | Incremental request framing, consumed-byte counts, explicit EOF and bounded feeds | A complete HTTP client/server session including response sequencing and lifecycle |
std.actor.net | Bounded socket buffers, readiness, timers, nonblocking connection setup | Application-protocol grammar or session rules |
std.http.websocket | Implemented RFC 6455 behavior and a sequential API over generic transports | An actor WebSocket adapter or a public socket-free WebSocket session |
The actor HTTP server uses the resumable decoder. The worker HTTP server still uses stream-oriented request reading; sharing parsing rules is not proof that both servers share all protocol orchestration. Existing qualification remains necessary during any migration.
Responsibilities
Codec
A codec recognizes or produces wire representations. It handles such matters as lengths, field encodings, frame headers, masking and incremental UTF-8 validation where appropriate. It accepts partial input and preserves enough state to resume; it must not require a complete peer-controlled frame to be buffered before checking its advertised size.
A codec does not decide when a socket should be read, when an actor should run, or whether an application may send a message in the current session state. Keep codecs private unless a real independent consumer needs their API. Three architectural layers do not require three public modules.
Session
A session enforces protocol semantics across codec operations: legal message sequences, negotiated settings, fragmentation, acknowledgements, control responses, closure and protocol timers. It accepts application commands, received data, explicit EOF and supplied time. It produces application events, wire output, timer requirements and terminal status.
No session owns a socket, selector, actor context, thread, cancellation token or runtime task. It neither waits nor obtains the current time. It can allocate within documented limits; sans-I/O does not mean allocation-free or immutable. State transitions happen through bounded, nonblocking calls.
Driver
A driver owns transport progress, readiness registration, waiting, cancellation, application delivery and runtime shutdown. It translates network EOF into session EOF and runs protocol timers when requested. It retains partially sent output and resumes it without loss or duplication.
Connection establishment, DNS and TLS configuration are driver/composition concerns. Protocol-specific decisions such as an acknowledgement timeout or WebSocket close response belong in the session. Application idle policies, server admission and a caller’s operation deadline belong outside it.
Architectural dependency rules
The permitted direction is driver → session → codec. Codec/session code may use ordinary data structures, byte operations and bounded computation, but must not import transport, actor or blocking-wait facilities. Review actual calls as well as imports: a callback or helper that secretly reads a socket still violates the boundary.
Protocol layers compose through bytes and explicit state, not inheritance. For example, a WebSocket session can sit above a TLS session, with a driver moving plaintext and ciphertext between their bounded buffers. A simpler blocking driver can instead use the existing TLS transport. This does not require changing every layer to the same representation.
Do not add a universal Protocol, global event enum, runtime registry or
standard protocol executor initially. TCP byte streams and UDP datagrams have
different contracts. Multiplexed protocols may need stream identifiers,
connection-wide flow control and endpoint-level state. Share small helpers only
after independent drivers demonstrate identical mechanics.
Public API direction
Keep existing convenience modules and behavior while extracting their protocol
logic. A proposed WebSocket session could live under
std.http.websocket.session; an actor adapter could live under
std.actor.http.websocket. These are candidate paths, not reserved names.
The codec may remain an implementation detail of the session module.
The session API should be useful to custom event loops and tests, not a public hook exposed solely so another std module can compile. Its module reference must document progress, ownership, allocation, errors and limits. Check exact signatures against stdlib conventions before implementation; regenerate the signature index only when public declarations actually change.
An initial API can favor owned events and copying into caller-owned output buffers. This is easier to specify under current borrowing and actor rules than retained views or callbacks into a mutably borrowed session. A zero-copy API is not a prerequisite; copies and memory costs must still be measured.
Implementation policy
Implement small, straightforward protocol machinery in checked Plenty. Use an established native library where substantial complexity or security expertise justifies it, preferably through its sans-I/O boundary. TLS remains backed by its existing security implementation; this proposal does not call for rewriting TLS, certificate validation, cryptographic primitives or QUIC in Plenty.
A native wrapper must retain input/output bounds, allocator provenance, error causes and actor cleanup guarantees. A library that performs hidden I/O is not made sans-I/O by placing a different name around it. A library whose unbounded allocation or global state cannot meet Plenty’s contracts requires a separate justification or must not be adopted.
Alternatives
| Alternative | Useful for | Why it is not the general answer |
|---|---|---|
Generic Reader/Writer or transport | Sharing a sequential driver across TCP, TLS and memory | A blocked read or WouldBlock does not preserve an actor callback’s execution position |
| Shared codecs only | Simple framing and stateless request formats | Stateful validation and automatic responses can still diverge between drivers |
| One dedicated protocol actor behind every API | Actor-native services | Sequential callers acquire an implicit runtime, mailbox and failure model |
| Worker offload of blocking implementations | Existing libraries without incremental interfaces | Waiting still consumes a worker; it is compatibility, not shared nonblocking protocol machinery |
| Coroutines or effects | Potential future sequential driver ergonomics | Requires suspension/ownership/cancellation machinery and does not remove backpressure requirements |
| One universal protocol interface | Potential common driver helpers | Prematurely forces stream, datagram, multiplexing and external-service differences into one shape |
The blocking-work proposal remains complementary. No async/await, stackful-task, effect, stored-reference or linearity feature is required for this design.
Scope and consequences
Use WebSocket to validate a complete shared session and both drivers. This is not approval to rewrite HTTP, introduce every common protocol, or change the actor scheduler. Existing modules migrate only with retained behavior and protocol-specific evidence. A future DNS, MQTT, SMTP or QUIC design should use the same separation while choosing its own session shape.
Benefits are a single location for protocol fixes, deterministic state-machine tests, reusable custom-runtime integration and independent runtime evolution. Costs are explicit state, more detailed progress contracts, bounded queues and some runtime-specific driver code. Shared protocol code does not guarantee equal throughput, equal memory use or identical application APIs.
Design references
- Sans-I/O guidance: separation of network protocol behavior from I/O mechanisms.
- h11: HTTP/1.1 as an I/O-independent event/state machine.
- wsproto: a socket-free WebSocket implementation as a design precedent, not a dependency or proposed port.
- Quinn protocol connections: explicit datagram, transmit and timer integration for a more complex protocol family.
These references inform the architecture; this documentation change imports no external implementation or data.
Session progress, ownership and resource contract
Companion to Runtime-independent networking protocols. Status: Proposed contract, not implemented APIs. The notation below describes operations, not final Plenty declarations. Priorities remain in the backlog.
Inputs and outputs
A stream session needs these conceptual operations. A concrete protocol may combine them when doing so leaves every progress and ownership rule explicit.
| Operation | Input | Observable result |
|---|---|---|
| Construct | Role, configuration, limits and initial protocol data | Valid session or a named configuration/allocation failure |
| Feed | Read-only byte view | Consumed-byte count, optional event and reason advancement stopped |
| Submit | Application command | Accepted command, partial byte acceptance, or refusal preserving ownership |
| Take output | Caller-owned mutable byte buffer and range | Count of wire bytes transferred out of the session |
| Advance time | Supplied monotonic instant | Bounded timer transitions, possibly producing events/output |
| Finish input | Explicit stream EOF | End-of-input transition or truncation failure |
| Inspect | No external effects | Pending work, next protocol deadline and lifecycle state |
Read-only bytes use &[u8]; owned buffers use Bytes; caller output uses
&mut Bytes and a half-open start, stop range. Counts and capacities use
i64. Time uses Instant, Duration and Deadline, never integer milliseconds.
No nonwaiting session operation takes a cancellation token merely to resemble
a blocking API.
An event-oriented session may offer a separate event-draining operation rather
than return one event from feed. Either arrangement must bound the number and
size of events retained internally. Returning an unbounded list of all events
from one input is not acceptable.
Progress must be unambiguous
A feed reports a count in 0..source.len(). The caller advances by precisely
that count and retains the remainder unchanged. A short count does not imply
EOF, and a call can consume bytes without producing an application event.
An empty feed may expose already-pending work but never announces EOF.
Distinguish at least these stop conditions, whether through a result enum or separate documented queries:
- Needs input: no internal work can advance without more network data.
- Needs output space: pending output must be transferred before advancement.
- Needs event delivery: the application must take a pending event.
- Needs external result: a named operation is awaiting a supplied result.
- Budget exhausted: work remains runnable; arrange a continuation.
- Terminal: no future protocol work is possible.
A pending event can be progress even when the consumed count is zero. Conversely,
0 consumed with no event and no change in pending work is not permission to
spin. A driver must wait for the identified dependency or yield after exhausting
its budget. If multiple dependencies exist, report enough information to choose
interests without repeatedly probing unchanged state.
Do not automatically reset a completed exchange if bytes for another exchange remain in the source. A protocol must specify when the application can admit the next exchange; HTTP pipelining is an example where this affects response order.
Input ownership and persistent state
feed borrows only for that call. Before returning, the session either consumes
bytes into owned, bounded state or leaves them for the caller. It never retains
a pointer into a caller buffer. Header parsing must not keep a view alive while
a driver compacts, grows or reuses that buffer.
Initially favor owned byte events. An event moved out belongs to its recipient;
the session cannot mutate it later. Ownership does not mean unrestricted actor
storage: an owned event containing shared str or another actor-ineligible type
may be usable only during a callback. The concrete reference must distinguish
transient events from persistable events and explain any conversion required.
Construct session buffers in the connection actor’s heap, not outside it and then move them into a child. Immutable configuration, foreign handles and secret material need the same provenance review as actor TLS. Sans-I/O is not an exemption from actor heap rules.
Command admission and retry safety
An application command is not accepted until the session reserves the resources needed to retain it or has made the documented partial progress. Backpressure is normal flow control, not a malformed-peer error.
Prefer borrowing command payloads when copying/encoding during the call. For a partial byte-oriented submission, return the accepted count; the caller retains the unsent suffix. For an ownership-taking command that can be refused before use, return the original command with the refusal cause, following std’s returned-input convention. Do not silently discard a moved message because the output queue is full.
Admission must have one specified outcome on allocation failure: either the command is entirely unaccepted, or an explicit progress value identifies the accepted prefix. Never return a bare failure after accepting an unknown amount and invite the caller to retry the entire message. Session mutation that can fail midway needs transactional preparation or a documented terminal failure.
For complete-message operations, accepting means local ownership/encoding responsibility, not successful peer delivery. Drivers must not invent delivery acknowledgements from successful socket writes.
Wire output and partial writes
Separate output transfer from transport delivery:
takecopies/moves a bounded prefix into a driver-owned buffer and removes exactly that prefix from the session’s pending output.- The driver owns those bytes until the transport accepts them or the connection terminates. A short socket write advances only its send cursor.
- The driver drains that cursor before sending later bytes on the same stream.
A temporary unwritable socket does not permit calling take again into the
same storage and overwriting unsent data. A future borrowed-output/commit API
could avoid a copy but needs its own borrow and failure contract; it is not
required here.
A session cannot infer that the peer has received a close frame just because
take returned its bytes. Where protocol state needs to distinguish queued
output from output accepted by the transport, the driver must provide an
explicit ordered flush milestone. No public API may claim graceful closure
solely because the session’s internal output queue is empty.
Time and external dependencies
Supply monotonic now explicitly for time-sensitive transitions. The session
returns an optional next protocol deadline; a timer notification means
“advance using this instant”, not “assume the old deadline is still current”.
Recompute the next deadline after every state change. Stale timer wakes must be
harmless, and decreasing instants must produce a named invalid-time error
rather than move protocol time backwards. Tests use supplied fake time.
Caller operation deadlines and cancellation remain in the driver. A keepalive, retransmission or close-handshake timeout belongs in the session; a server-wide drain deadline can force termination earlier without pretending the protocol completed gracefully.
Do not obtain entropy, resolve names, read credentials or invoke arbitrary application code inside a session transition. Supply data up front or request an explicit bounded external result. A request carries a checked identity so a late or duplicate result cannot satisfy a different operation. The requesting session specifies whether it can do other work while waiting. External failure preserves its cause and follows a documented retry/terminal policy.
Security-sensitive randomness must still come from the existing secure entropy facility. Supplying entropy explicitly makes tests reproducible; it does not justify a predictable seed or random stream in production.
Resource bounds and fairness
Each session documents separate caps for:
- incomplete headers, frame payloads and assembled messages;
- queued application commands and undelivered events, in bytes and count;
- protocol output, including automatic control traffic;
- codec scratch space and retained metadata;
- active exchanges, streams or external requests where applicable.
The total bound includes codec state, session queues, driver buffers and transport/TLS buffers. Moving bytes between layers must not double-count them as free capacity or hide multiple full-sized copies. Document peak memory, not just the session’s most visible buffer. Validate configurations before use; never silently clamp a bad size or overflow a combined-capacity calculation.
Check advertised lengths before allocating or converting to a machine index.
Queue saturation pauses progress; an oversized peer object produces a named
limit error. Allocation failures are preserved in the module’s domain error as
Allocation(AllocError).
A bounded input byte count is insufficient by itself. Bound parser scanning, event emission, internal transitions and output production per call; bound the number of calls and bytes copied per actor turn. Work left after a budget requires a scheduled continuation, not a wait for network readiness that might never recur. Limits must also cover many tiny control frames and fragmented messages, not just large payloads.
Reserve bounded space for mandatory control output where the protocol requires it. Define what happens if that reserve is exhausted: stop input, reject new application output or terminate according to the protocol. Never let a full application queue cause unbounded automatic replies or deadlock a required close response. For stream protocols, preserve wire order: a control response cannot be inserted into a frame already partly emitted.
EOF, errors and final states
Distinguish no bytes available, read-side EOF, orderly protocol closure, protocol failure, transport failure and administrative abort. A transport half-close does not automatically close the opposite direction; the protocol determines whether remaining output is useful. Dropping state performs resource cleanup only and must not wait to send a graceful close.
A session’s native error describes protocol/configuration/allocation/external input failures, not fictitious network failures. The driver preserves actual transport causes in its own error mapping. Keep existing public error variants where compatibility requires them; a new pure-session API need not inherit network variants that it cannot produce.
After a fatal parse error, do not resume from an untrusted frame boundary. A session may enter a bounded failing state to emit a protocol alert/close; the original error remains available while that output drains. It then becomes terminal. Terminal means no further output, event or timeout can advance it; failed-with-output-pending is not terminal. Document which inspection/draining operations remain permitted after failure and reject other operations with a named state error.
Blocking and actor protocol drivers
Companion to Runtime-independent networking protocols. Status: Proposed architecture, not implemented adapters. Drivers follow the session contract. Work and priority live in the backlog.
Shared mechanics, different execution
Both drivers must preserve the same input/output order and protocol state. They need not expose identical application methods or share all control flow. A sequential receive can wait; an actor receive reports an event and returns. Connection establishment, transport ownership and application dispatch belong in the driver, while frame validity and legal state transitions stay in the session.
Useful mechanics to extract after both implementations exist include cursor management, bounded buffer transfer and a single bounded advancement step. Do not hide waiting, actor contexts or runtime callbacks in those helpers. Keep protocol-specific decisions out of both drivers even if copying them would be shorter than extracting the correct session operation.
Blocking driver
A sequential operation conceptually does the following:
check caller cancellation and deadline before admitting a new command
submit the command, or remember its explicitly unaccepted suffix
repeat within a bounded advancement batch:
deliver pending events relevant to the operation
move session output into a bounded driver send buffer
write pending transport output and retain any unsent suffix
feed buffered input and retain any unconsumed suffix
advance protocol time using supplied monotonic now
retry admissible command progress
if operation is not complete:
wait for useful transport readiness or the earliest relevant deadline
perform bounded reads/writes, then resume advancement
This is an ordering sketch, not an instruction to drain every queue before checking another. Balance both directions: bidirectional protocols can deadlock if a driver insists on completing a large write before processing a peer’s control input. Conversely, do not read beyond bounded capacity merely because input is ready.
Use existing deadline-aware networking and cancellation mechanisms rather than polling or adding a hidden event runtime. The wait ends at the earliest of the caller’s deadline, the next protocol timer, and any driver shutdown deadline. If protocol progress requires simultaneous interests, use the existing readiness facilities rather than serial full-duration read and write waits. A generic blocking transport remains useful inside this driver where its capabilities actually suffice.
An operation completes according to its documented local semantics: command admitted, bytes flushed, message received or close handshake finished. These are different milestones. Output remaining after a receive operation must have a specified owner and flush policy; automatic pongs cannot remain queued forever because the application stopped calling receive.
Do not silently start a background thread to solve that problem. State whether the API makes progress only during calls, offers a drive/flush operation, or requires an explicit service loop for autonomous keepalive behavior. Preserve existing sequential API guarantees during extraction.
Cancellation and retry
Before admission, pre-cancellation leaves the session unchanged. Once any part of an operation has affected the wire, a timeout/cancellation cannot generally be retried as a fresh operation. The driver either exposes explicit resumable progress or terminates the connection according to its existing contract.
For WebSocket, preserve the current rule that cancellation mid-frame closes the connection rather than inviting an unsafe resend. Retaining internal state makes correct resumption possible in principle; it does not change a public method’s cancellation promise without a separate compatibility decision.
An abort discards remaining output and releases the transport. A graceful close is a separate bounded operation driven until its completion condition or a specified deadline. Neither destructor cleanup nor actor trap cleanup may wait for it.
Actor driver
A connection actor constructs its session and buffers inside its own heap. It owns the transport adapter, session, input cursor, output cursor and protocol timer state. It reacts to:
| Trigger | Driver action |
|---|---|
| Network readiness | Run bounded transport progress, then bounded session advancement |
| Application message/command | Attempt admission without waiting and return an explicit result |
| Protocol timer | Advance with the current instant, ignore stale schedules, recompute deadline |
| Continuation | Resume already-runnable bounded work without requiring new socket readiness |
| Application capacity restored | Resume event delivery and any paused session/input progress |
| Stop/drain request | Reject new work, drive bounded protocol closure, then force abort at the drain deadline |
Readiness is evidence that an attempt may progress, not a promise that it will.
WouldBlock is normal. Enable writable interest only while transport output
actually waits; permanently watching a writable socket can spin the stage.
When input/event capacity is exhausted, pause the appropriate intake and retain
unconsumed input. Resume explicitly when capacity returns.
If a turn budget expires while buffered input or internal transitions remain, arrange a continuation through a supported actor wake mechanism. Do not wait for another readiness event: the data may already have left the kernel. Bound continuation scheduling so one connection does not fill its own mailbox with redundant wakeups. Timer/admission failure is a reported failure, not permission to drop a required wake.
Handlers run to completion. The actor driver must not call blocking DNS, HTTP, database, entropy acquisition or user callbacks with unspecified blocking behavior from its protocol advancement path. Use existing nonblocking facilities or an explicitly supported external-service integration; do not assume the proposed blocking-work pool already exists.
An actor’s bounded mailbox is not an automatic delivery guarantee. If command submission or event forwarding encounters a full mailbox, preserve ownership and report backpressure. Avoid forwarding every protocol event through another mailbox when direct bounded delivery to the owning actor is sufficient.
Layering over TLS
The straightforward choices are:
- Blocking WebSocket → existing TLS transport → TCP.
- Actor WebSocket → actor TLS → actor TCP.
- Custom runtime → WebSocket session → TLS session → caller-owned socket.
Every intermediate layer can retain partial input/output. Pump the composed layers until an external dependency or work budget stops progress. Each layer must identify pending work without assuming that lack of new socket input means lack of plaintext or protocol events. Include plaintext, ciphertext and protocol buffers in the connection’s peak-memory accounting.
A TLS close notification is not a WebSocket close frame. Preserve the ordering of WebSocket close, protocol flush, TLS close and socket release, with a bounded fallback for peers that stop responding. TLS alerts and protocol close output may still need draining after an error, but the driver must retain the original cause and never wait indefinitely to deliver them.
Datagram and multiplexed protocols
Do not adapt UDP by concatenating packets into a stream. A datagram session receives one packet with source/destination metadata and produces packet output with its destination and relevant metadata. Define atomic packet admission, maximum datagram size, truncation handling and endpoint demultiplexing.
An endpoint may own routing state shared across several sessions. Multiplexed sessions can require per-stream credit, connection credit and reset events. Retransmission/congestion timers belong in that protocol implementation, not a generic driver. These distinctions are reasons to reuse the architecture rather than impose a universal session interface.
Platform and failure claims
A socket-free session can be portable without making its driver portable. Keep the existing actor platform limits explicit until native qualification proves otherwise. Do not broaden supported targets based only on session tests.
Returned errors, contained actor traps and process-fatal runtime failures remain different outcomes. Session extraction does not strengthen the actor sandbox. Test resource recovery after actor heap exhaustion and ensure native session state owns nothing in a discarded or unrelated heap. No hidden global protocol service should turn a connection-local failure into shared corrupted state.
WebSocket validation and acceptance
Companion to Runtime-independent networking protocols. Status: Proposed validation scope, not implemented session or actor support. This page defines the evidence needed to validate the architecture. The backlog, B94 with B85, owns implementation sequencing and priorities; the cases below are acceptance criteria, not a separate work queue.
Why WebSocket
WebSocket is a useful first complete session because it combines incremental framing with bidirectional commands, fragmentation, control responses, text validation and a closing handshake. It exercises more than a passive decoder without introducing QUIC’s transport/security complexity. Plenty already has an implemented sequential WebSocket API, which supplies a behavioral baseline and a migration obligation.
The initial scope is existing RFC 6455 client/server behavior over TCP and TLS,
plus an actor integration. It excludes permessage-deflate, proxy tunnels,
HTTP/2 extended CONNECT, new scheduling models and automatic migration of all
HTTP machinery. No additional wire features are required to validate the
three-layer design.
Codec responsibilities
The incremental codec handles frame headers, extended lengths, mask position across input slices and payload encoding/decoding. Validate reserved bits, opcodes, canonical length encodings and role-specific masking before accepting an invalid frame. Check frame and message bounds before allocating based on advertised lengths. Control frames remain at most 125 bytes and cannot be fragmented.
The codec can emit bounded payload chunks rather than assemble a complete message. Message limits are enforced across all fragments, including when the application consumes chunks and no single retained buffer is large. UTF-8 validation carries state across network fragments and continuation frames; validating each received byte slice independently is incorrect.
Session responsibilities
The session has explicit client/server roles and state for fragmentation, message size/text validation, automatic pong generation and closure. It rejects continuations without an open fragmented message, overlapping data messages, and data sends that are no longer legal after closure starts.
A conceptual input/output vocabulary is:
Inputs: received bytes, send text/binary, ping, begin close,
supplied entropy result, input EOF, supplied time, output flush milestone
Outputs: message chunks and boundaries, ping/pong notices, peer close,
failure/termination status, wire bytes, entropy request, next deadline
This is a contract sketch, not a commitment to these exact events. The blocking
adapter can assemble chunks into its existing complete-message API. Actor
applications should be able to stream within bounds instead of requiring a
whole maximum-sized message in their heap. Do not expose a chunked text event
as independently valid str when a UTF-8 scalar can span chunks; use bytes and
message-level validation semantics.
Specify when incoming pings/pongs are observable, which replies are automatic, and what happens when the application is not draining events. A bounded control reserve must allow required replies without unbounded allocation. If continuing input would exceed that reserve, stop consuming input or terminate explicitly. No driver implements its own ping/pong or fragmentation policy.
Entropy without hidden I/O
Clients need unpredictable handshake nonces and fresh unpredictable frame masks. This must not become a hidden entropy read in the codec/session or a blocking call on the actor stage. The driver/external-service boundary supplies securely obtained nonce/mask material through a narrowly documented operation.
For each accepted outbound client frame, bind a fresh mask to that frame before its header or payload can be emitted. If entropy is unavailable, leave the command unaccepted or retain it in an explicitly bounded pending state. Retrying must not duplicate a frame, reuse a previous frame’s mask or consume unrelated entropy results. Bound pending requests and reject late/duplicate identities. Test providers may supply fixed material; production providers must use Plenty’s existing secure entropy facility.
Before shipping actor client support, demonstrate an actual nonblocking or explicitly off-stage entropy path. This proposal does not claim that the current entropy API already supplies one. If that requires runtime work, record and review the concrete capability separately; do not weaken the sans-I/O boundary or silently rely on the unimplemented general blocking-work pool.
HTTP upgrade boundary
Opening-handshake validation and generation are protocol logic. Socket reading, HTTP exchange ownership and actor dispatch are driver concerns. Reuse the same handshake validators/builders from both drivers; they may be helpers separate from the post-upgrade frame session, provided no protocol rules are duplicated.
Retain current checks for method/version/body, upgrade tokens, key/version, accept value, subprotocol and unsupported extensions. Preserve existing server refusal status and client error behavior unless a separate change explicitly justifies it. Do not accept unsupported extensions merely because their frames look superficially valid.
Upgrade transfers ownership of the transport and every already-read byte beyond the HTTP head. Feed that suffix exactly once into the WebSocket session. It may contain several frames or a partial frame, including when the HTTP response and first frame arrive in one read. HTTP must not continue parsing bytes after a successful transfer.
For actor HTTP, define a supported ownership/lifecycle handoff before exposing
upgrades. The current actor server lists upgrades as remaining work; constructing
a frame session alone does not implement that integration. Specify how watches,
timers, shutdown ownership and connection accounting transfer, including failure
between sending 101 and completing the handoff.
Closing and compatibility
Track local close queued/flushed, peer close received, transport EOF and timeout as distinct facts. Echo/response behavior, close-code validation and reason validation must follow the existing contract and RFC. A forced abort is not a successful close handshake. Simultaneous close must not generate endless responses, and a close timeout must not discard the original failure cause.
Keep std.http.websocket and std.http.wss source-compatible where practical.
Preserve bounded frame/message behavior, refusal/rejection distinctions,
cancellation behavior, server stop tokens, worker ownership and conditional TLS
linking. Current whole-message buffering or exclusive borrowing may remain in
the convenience driver even if the session supports finer-grained progress.
Do not use extraction as an excuse to change these promises silently.
Reference/tutorial updates ship with the implementation: explain choosing the blocking or actor driver, show recommended error handling and shutdown, and include runnable examples for both. The session reference must explain custom runtime integration, progress and memory bounds. The proposal itself is not a substitute for those implemented-behavior pages.
Deterministic session evidence
Run the same wire corpus without sockets, threads, real clocks or production entropy. Include:
- Every split point for representative headers, extended lengths, masks, control payloads, UTF-8 sequences and close frames; one-byte and coalesced feeds.
- Fragmented text/binary messages with interleaved controls, invalid opcodes, bad masks, invalid UTF-8, invalid lengths and limits exceeded across fragments.
- Empty feeds, EOF at each incomplete boundary, repeated inspection after terminal states and rejected operations after failure/close.
- Zero/small output capacity, partial takes, partial command admission, saturated event/output queues and exactly-once retry behavior.
- Stale/duplicate external results, unavailable entropy, stale timer wakes, decreasing time, simultaneous close and a peer that never completes close.
- Allocation failure at each fallible growth/admission point, preserving ownership and reporting known progress or terminal failure.
Property tests vary byte segmentation, output capacity and legal pump scheduling while preserving input order. Equivalent schedules must have equivalent message and terminal semantics; chunk boundaries and batching may differ. Assert that reported consumption stays in range, queue bounds hold, output order is stable, and no no-progress state spins. Replayable traces contain inputs, commands, supplied time and test entropy, not secrets from production sessions.
Fuzz codec/session transitions and external-result ordering. Treat an unexpected trap, hang, bound violation or consumption inconsistency as a finding. Mutating only complete frames misses incremental-state defects. Independent RFC vectors and another implementation are needed as checks: both drivers can agree because they share the same bug.
Driver and composition evidence
Use one scripted peer corpus against both drivers and compare normalized application events, wire validity and terminal outcomes. Cover real TCP/TLS, partial writes, slow readers, read/write interdependence, EOF/half-close, cancellation, close deadlines and both upgrade directions. Include HTTP heads coalesced with WebSocket bytes and TLS records split at unrelated boundaries.
Actor tests must demonstrate bounded work per turn, progress without fresh network readiness after a budget yield, output/event backpressure recovery, correct timer rearming, full-mailbox handling, graceful/forced shutdown and unrelated-connection progress under load. Repeat handler traps and heap exhaustion; compare descriptor, live-heap and native-allocation baselines after cleanup. Verify transport ownership is recovered or closed exactly once on every upgrade-construction failure.
Retain existing HTTP, WebSocket, WSS and actor networking/TLS regressions. Measure throughput, latency, CPU and peak memory with matched payloads, connection counts, limits, TLS configuration and CPU allocation. Include idle connections, large/fragmented messages and control-heavy traffic. Report measured copy and buffering costs; do not claim an improvement from architecture alone. Actor platform support requires real platform evidence, not just portable session tests.
Acceptance and boundaries
The convention is validated when one complete WebSocket implementation supplies both blocking and actor drivers, with protocol rules only in the shared codec/session/handshake helpers; all documented progress/resource contracts have tests; existing convenience behavior is retained or explicitly reviewed; and reference, signatures and executable tutorials match the implementation.
The first extraction can be reviewed independently of the actor handoff, but it must not be called complete cross-runtime support until actor upgrades, client entropy and lifecycle integration are demonstrated. Any unresolved gate remains visible in the backlog and PR evidence. No production-readiness, zero-copy or cross-platform claim follows automatically.
Protocol specification
- RFC 6455: opening handshake, framing, masking, control frames, UTF-8 and closing rules. No RFC code or data is imported by this proposal.
An HTTP server on supervised actors
Status: Implementation and qualification in progress (B85). This proposal records the intended second server; it does not declare the actor server production-ready. The existing worker server remains supported independently. The backlog owns remaining work.
Two supported servers
std.http.server keeps sequential handlers, bounded workers and its existing
qualification. std.actor.http.server uses one supervised actor per connection,
with bounded heaps, mailboxes and socket buffers. A contained handler trap loses
that connection while unrelated connections and admission remain available.
Applications choose the execution and failure model that suits their work.
The actor server should make ordinary bounded request/response handlers simple. A separate event handler interface handles streaming input/output, timer wakes, actor messages and shutdown without blocking the stage. Request heads are owned transient callback inputs; persistent application state obeys the existing actor heap rules. Factory and connection initialization run in the child heap. No compiler heap-boundary relaxation is part of this design. The separate blocking-work proposal describes bounded background workers for existing blocking clients, without blocking callbacks.
Protocol and lifecycle
Share HTTP line, field, target and framing rules with std.http. A resumable
byte-feed decoder retains no shared strings across actor callbacks and consumes
a bounded amount of input per turn. Partial bodies and output resume on readiness;
waiting does not occupy a native worker. Plain TCP and TLS use the existing actor
network layers. Bounds cover admission, headers, bodies, output, actor heap use,
requests per connection, idle time, handshake, request phases and writes.
Shutdown stops admission, finishes bounded active responses during a grace period, closes idle connections and then forcibly retires remaining connection actors. A stop signal must not bypass the drain phase. Refusals preserve the existing bounded lingering-close policy where possible.
The first supported implementation uses one stage/thread and the existing 1024-live-heap process ceiling. Handlers run to completion and cannot safely perform blocking operations or unbounded CPU work. Stack overflow and internal runtime panics remain process-fatal; actor containment is not a sandbox. Shared foreign state and TLS configuration ownership need explicit repeated-failure checks. HTTP/2, HTTP/3 and multi-thread stages are separate scope.
Qualification and comparison
Production readiness requires retained parser corpus parity (including fragmentation, malformed framing, pipelining and trailers), real TCP/TLS clients, slow heads/bodies/readers/handshakes, bounded admission, repeated handler traps and allocation exhaustion, descriptor/heap baselines, graceful shutdown under load, reverse proxies, fuzzing that treats an unexpected contained parser trap as a finding, and documented platform limits.
Run the worker and actor servers on the same machine, compiler revision, requests, response sizes, concurrency and CPU allocations. Publish both an equal-core comparison and each model’s normal configuration where useful. Record throughput, latency, memory, failures and failure isolation. Simplicity, speed and resilience differences are hypotheses to measure, not promises inferred from the concurrency model. The benchmark must stay below each implementation’s configured connection limit.
Actor budgets and retained stacks
Status: Exploratory; no production scheduling change. A test-only native
probe validates stack retention with the existing actor heap router and trap
recovery. It does not insert compiler checkpoints or drive std.actor.Stage.
The actor reference still specifies handlers that
run to completion. Remaining implementation and qualification scope belongs to
B85 in the backlog. See the probe evidence.
Question and provisional decision
Could compiler-inserted budget checks let an actor yield inside ordinary nested calls, without changing application methods into generators?
The mechanics are plausible with immovable, guarded stacks on one scheduler
thread. A checkpoint decrements the active actor’s approximate work budget. On
exhaustion it saves the actor’s execution and runtime state, switches to the
scheduler, and later continues after the check. Helper calls keep their normal
calling convention. The runtime must separately preserve state currently stored
per native thread, and the stage must stop lending the same exclusive Context
to overlapping handler activations. These are substantive integration changes.
This is a narrower investigation than general stackful tasks: actor state already has an isolation boundary, mailboxes remain bounded, and a suspended callback remains the actor’s only active callback. The older C task spike proves nested stack retention; the new Rust probe exercises actual actor trap recovery and allocation routing. Closed local PRs #639/#642 explored socket-parking fibers before the implemented actor API; they did not validate budget checks or the current stage’s ownership contract. No unmerged source from those PRs is used here.
Scheduling and compiler boundary
Checks belong on executed loop cycles and recursive call cycles, including tail calls. A check only at the outer actor entry misses an unbounded helper loop. Calls through callbacks and generators also need coverage; inlining, loop specialization, SIMD lowering and cached object reuse must preserve that coverage. Existing lowering has distinct loop, tail-call and callback paths, so adding one check to one loop emitter would not establish fairness.
An initial budget can count approximate work between checks. It is not a time limit: a single large operation may have substantial cost. Long runtime operations need chunking, their own safe checkpoints or worker offload. Measure both uncontended check overhead and peer progress under expensive operations before choosing a charging policy. The probe’s quantum of 17 is only a deterministic test parameter.
One possible implementation shares compiled helpers and makes a checkpoint a no-op outside an actor, while honoring an explicit non-suspending region inside one. Actor-specialized helpers could reduce unrelated overhead but increase code and cache variants. Neither option has been implemented or measured. Every checkpoint must be visible to code generation as an ordinary call with the appropriate memory effects; switching stacks does not relax aliasing rules.
Do not switch inside allocation, heap-to-heap message copying, stage queue mutation, foreign-library arenas, or synchronous destruction. Compiled code invoked by a runtime helper can reach another checkpoint: excluding only Rust function bodies is insufficient. Transitive compiler effects or a runtime non-suspending depth, with audited entry/exit and trap restoration, must protect these regions. Deferral can itself be unbounded unless the protected operation has a bounded cost or an offload path.
Actor and scheduler ownership
Today Stage.deliver passes &mut self._nodes[index] and
&mut self._context to __actor_run. Its adapter arguments and result storage
are slots in the caller’s native stack. Context owns the shared mailboxes,
ready queue, selector, timer table, and current actor/spawn/stop fields.
A suspended callback cannot retain that exclusive context while the scheduler
and another callback access it. Saving _self alone would repair the visible
identity but leave overlapping exclusive loans. A candidate replacement is one
context capability per activation, holding actor-local identity and pending
commands, with stage access restricted to short non-suspending operations. Its
interior access needs an explicit compiler/runtime ownership contract. A raw
pointer hidden in the library is not sufficient proof of soundness.
Node slots, callback arguments/results, and context storage must stay at stable addresses until the callback finishes or traps. Scheduler dispatch frames must return independently of suspended actors. Keep suspended callbacks runnable even with empty mailboxes; enqueue new mail without starting another callback for that actor. Run spawn hooks, settlement and restart policy at their defined lifecycle points. Do not restart an actor or destroy its stack while a suspended activation still references its state.
Runtime, failure and resources
The probe places plenty_actor_run and its recovery frame on the actor’s own
stack. This lets the existing nonlocal trap path return within that actor after
several suspensions. The scheduler’s stack is separate. Each transition restores
the intended allocation heap, recovery pointer, caller source location, drop
queue and last-trap state. Arena calls and active destruction are excluded from
probe checkpoints. Other TLS, callback, lock and FFI boundaries still need an
audit; the demonstrated snapshot is not an exhaustive runtime contract.
The actor owning the stack also owns the trap boundary. Normal completion and
cooperative cancellation return through ordinary cleanup; a contained trap
abandons frames without destructors, following existing actor semantics, then
discards the heap. Resuming a trapped or completed context is forbidden.
Scheduler deadlines and cancellation can be examined between quanta, but
delivering Mail.Stop does not interrupt a callback already in progress.
Cancellation, stage teardown and any forced-discard policy need a separate
defined contract; stack switching does not make arbitrary stack abandonment safe.
The experiment uses fixed stacks with inaccessible guards. Stack admission must be fallible and bounded in a production implementation, with a separately visible stack-memory cost. Virtual reservation is not resident usage or a guarantee of later physical memory. Guard pages do not prove that oversized frames cannot skip guards. Existing stack-overflow reporting assumes the native stack and its alternate-signal-stack reservation; alternate stacks require explicit bounds and signal-handler qualification. Overflow remains process-fatal under the current contract. Recoverable stack overflow is not established here.
The probe uses glibc swapcontext, which preserves more state than a minimal
ABI switch and may involve a system call for signal masks. It is a correctness
vehicle, not the selected production switch. A custom switch needs callee-saved
register and floating-point/control-state tests, stack alignment, platform
hardening, debugger/unwinder and sanitizer integration. The experiment qualifies
only its host x86-64 GNU/Linux configuration; AArch64 and other targets are not
covered. Thread migration is outside this candidate.
Blocking native operations still require nonblocking actor adapters or bounded blocking-work offload. A checkpoint cannot interrupt a blocked syscall or arbitrary native computation. Where possible, compiler checks should reject known blocking operations from actor-reachable code, including indirect calls, with explicitly audited offload as the escape.
Assessment
Retained stacks avoid a generator transformation throughout the call graph and support actor-local borrows across nested suspension. The actual heap router and trap boundary can coexist with suspension when their state is restored. This is enough to justify a compiler/stage integration prototype, without adopting a production scheduler or claiming BEAM-equivalent fairness. The ownership of the shared stage context is the largest architectural question exposed by this probe. The backlog records the remaining gates.
Actor stack-switching probe evidence
Companion to Actor budgets and retained stacks.
Status: Test-only native probe; compiler insertion and stage integration are
not implemented. Source: experiments/actor-stacks/probe.rs, included inside
the runtime’s actor heap module only in x86-64 GNU/Linux test builds.
What executes
Two callbacks run on separate fixed 64 KiB stacks, with an inaccessible page
at each end. A small round-robin harness alternates them whenever an explicit
checkpoint exhausts a budget of 17. Checkpoints model intended compiler output;
no compiler pass inserts them. Each callback enters the actual
plenty_actor_run, so its production recovery record lives on its own stack.
The probe uses the real routing allocator, heap limits and unwrap trap helper.
The harness saves/restores allocation routing, recovery, caller-location and last-trap TLS, plus the production drop queue. It refuses checkpoints during arena calls or active destruction. The drop queue remains empty in these fixtures; nonempty queue switching is not tested. Caller-location identities are distinct and checked after resumption. Retained heap allocations are checked for ownership and contents, and an oversized allocation returns null within each actor’s limit. A borrowed stack local survives nested helper suspension.
The harness uses stable raw-pointer records rather than keeping an exclusive
Rust reference to scheduler storage across a switch. It does not model a sound
replacement for Plenty’s shared Context, compiler adapter storage, mailbox
delivery, readiness, timers, supervision or restart. It cannot establish the
borrow checker’s acceptance of a new context design.
Reproduce
From the task checkout:
cargo test --locked -p plenty-runtime actor_stack_switching_probe -- --ignored --nocapture --test-threads=1
The probe is ignored during ordinary suites because it owns the process-wide actor heap table; run it alone. An external timeout is appropriate for detecting switch failures. Its scheduler also asserts a finite maximum turn count. The default fixture has finite loops; it does not execute an infinite loop.
Checks and interpretation
| Case | Assertion |
|---|---|
| Budget disabled | Peer first runs after the first actor completes all 10,000 iterations |
| Budget enabled | Peer first runs after 17 iterations; each actor completes 10,000 iterations with 588 yields |
| Trap after resumption | First actor traps after three yields, returns Unwrap, runs no local destructor; peer completes and runs its destructor once |
| Cooperative cancellation | Scheduler requests cancellation after eight dispatches; both callbacks resume, observe it and return through cleanup before completing their loops |
| Allocation routing and limits | Small allocations belong to the running actor before/after switches; oversized allocation is refused |
| Reclamation | Each case discards both heaps and unmaps both stacks; live heap count, used bytes and committed heap bytes return to zero |
These are deterministic work/progress checks, not throughput or latency
benchmarks. The cancellation flag is an experiment-only mechanism, not
Mail.Stop or a new public cancellation API. The trap fixture deliberately
leaves its allocation and destructor behind; existing actor heap discard
reclaims that allocation without unwinding the stack. Runtime panics are not
recovered.
Observed on x86-64 GNU/Linux on 2026-11-02: all four cases passed in debug,
release and debug with allocation-checks. The budgeted pair performed 588
yields each; the trap case returned Unwrap at iteration 52 while its peer
completed 10,000 iterations. Cancellation returned both callbacks at iteration
68 with one destructor each. The oversized-allocation check calls Router
directly because an unused standard Rust allocation can be optimized away.
Validation also passed the ordinary runtime suite (187 passed, 5 ignored),
workspace/all-target Clippy with warnings denied, workspace formatting and an
explicit rustfmt check of the included probe. Three networking tests initially
failed under sandbox restrictions; the runtime suite passed when rerun with
local networking available. Documentation validation passed source links,
151 script tests (3 skipped), and the complete site using mdBook 0.5.4
(1,444 default and 722 development HTML pages). No compiler integration suite
was run: production code generation and executable Plenty examples are unchanged.
No stack admission fault injection, guard-page overflow, real socket workload, platform register conformance, signal delivery, blocked-native-call latency or complete sanitizer coverage is established. The stackful runtime uses glibc contexts, so it does not benchmark the cost of a future minimal register switch. Only a compiler prototype can validate coverage of optimized loops, recursion, tail calls, callbacks, generators and cached objects.
Blocking work for actors
Status: Proposed; not implemented. This is a design for the blocking-work part of B85, not a new scheduling or thread API available today. Priorities and remaining scope live in the backlog. The companion pages cover ownership and failure and runtime integration and qualification.
Purpose and existing limits
An actor HTTP handler should be able to call an existing blocking HTTP client, database driver or file operation without stopping unrelated connections. The callback submits work, returns to the stage, and later receives a completion event. Waiting occupies a worker thread, not the stage thread.
Today actor callbacks run to
completion on one thread. Ordinary thread creation inside a callback traps;
scoped threads also join before scope exit. Actor nodes cannot retain ordinary
thread, channel, executor or shared-state handles. Wrapping spawn or a future
in a Plenty class cannot bridge these constraints safely. The existing
actor DNS lookup
already demonstrates helper-thread completion independent of an actor heap,
but is a specialized operation, not a general offload API.
Prefer actor networking and actor TLS for protocols that already have nonblocking state machines. Offload is a compatibility path for blocking APIs; it is not a promise that more threads improve throughput.
Recommended API direction
Use an explicitly created, bounded OS-thread pool, owned outside the stage and
attached before the stage runs. No implicit global pool or thread per request.
The first version accepts named, capture-free worker functions and owned
Bytes request/result payloads. A typed error remains available; arbitrary
closures, borrowed arguments and generic object-graph transfer are excluded.
The following is a contract sketch, not executable Plenty or final generic
syntax. Suggested module: std.actor.blocking.
Limits(max_workers: i64, max_jobs: i64, max_input_bytes: i64,
max_output_bytes: i64, max_worker_bytes: i64)
limits() -> Limits
open_pool(limits: &Limits) -> Result[Pool, BlockingError]
Pool.attach(stage) -> Result[PoolKey, BlockingError]
worker(input: Bytes, deadline: &Deadline, cancel: &CancellationToken)
-> Result[Bytes, E]
Context.submit_blocking(pool: PoolKey, worker, input: Bytes,
deadline: &Deadline)
-> Result[Job[E], SubmitError[Bytes]]
Job[E].take_now(context) -> Result[Option[Result[Bytes, E]], BlockingError]
Job[E].abandon(context) -> Result[(), BlockingError]
Pool is not actor state. PoolKey is a checked non-owning capability scoped
to its attachment; Job[E] is a narrowly certified opaque actor handle. Neither
grants a worker access to Context or the stage. An attach operation must not
make Pool destruction race a live attachment. Final names and receiver syntax
must be checked against stdlib conventions
and the compiler’s generic callback representation during implementation.
submit_blocking never waits for a free worker or queue slot. It checks the
pool/stage identity, input size and deadline, prepares an
independent input snapshot, and atomically admits a job. Function eligibility is
a compile-time check, not a run-time escape hatch. Admission errors return
the original input in SubmitError[Bytes], with the reason (Capacity,
Closed, InvalidPool, TimedOut, LimitExceeded or allocation failure).
BlockingError embeds Allocation(AllocError) rather than an allocation union.
Worker errors E retain their type and cause; they are not flattened into text.
An accepted job reserves one of max_jobs records until its worker resources
and result are reclaimed, including after its caller abandons it. The bound
covers queued, running and completed-but-unclaimed jobs, not just the waiting
queue. Invalid sizes and unsupported configurations fail explicitly. Defaults
must be documented and measured before adoption; the API does not derive a
worker count from incoming traffic.
HTTP handler experience
Use the event HTTP interface:
- On request
End, build a boundedBytesdescription of the upstream request, submit a named worker and retain itsJob[E]and application correlation ID. - Return without joining, polling in a loop or waiting for the result.
- On
Event.Actor(Mail.Blocking(job_id))(proposed), calltake_nowfor the matching job, then build the response from its result. - On disconnect, request timeout or
Stopping, abandon outstanding jobs. Apply the application’s existing HTTP error policy for upstream failure or capacity exhaustion; do not automatically retry a side-effecting operation.
The worker constructs its own client/configuration, decodes input, invokes the existing blocking fetch with the supplied deadline and cancellation token, and returns bounded bytes or a transferable typed error. Do not claim every existing client error is transferable: eligibility must be checked, with an explicit owned error adapter where needed. The streaming interface means asynchronous response timing; the initial offload API itself returns one bounded result, not an unbounded response stream.
A database wrapper follows the same pattern but must satisfy the driver’s thread-affinity and effect contracts. Persistent database sessions and connection pools need a separately designed worker-owned resource facility; an arbitrary connection cannot simply move out of an actor into successive pool threads.
Decisions and alternatives
A bounded pool amortizes thread creation and makes saturation visible. One OS
thread per request allows incoming traffic to exhaust threads and stacks.
Blocking join inside a callback defeats the purpose. Moving the blocking call
to another actor on the same stage also stalls that stage. Supporting several
stage threads would not make a blocking call cooperative and is independent.
Use byte payloads initially to avoid promising safe deep transfer of arbitrary
actor-owned values. A named function avoids storing actor captures in a job.
The compiler can reuse existing worker-effect checks, but must add an explicit
actor-to-worker boundary rather than exempting ordinary spawn from its guard.
Full typed requests, streaming results and retained worker resources require
separate contracts. This proposal does not introduce async/await.
Blocking-work ownership and failure
This accompanies the proposed actor blocking-work API. None of these contracts are implemented yet.
Three allocation domains
An actor’s heap can disappear after any callback traps. Moving a Bytes header
into a native job does not make its backing storage independent. Submission
must copy the payload into a separately owned job allocation before publishing
it to workers. The copy is explicit in the contract and bounded by
max_input_bytes; failure leaves the original input recoverable. Metadata,
cancellation state, wake handles and error storage must likewise contain no
pointers into an actor heap.
Workers use ordinary thread-owned allocations under a proposed per-job allocation budget, never the actor heap table. The worker creates its own client, TLS configuration, request strings and any foreign handles. Native code must not temporarily switch a background thread into an actor’s heap. The existing single-owner actor heap rule remains intact.
Completion retains an independent result until the stage claims it. Only the
stage thread copies the result into the receiving actor’s heap. Output bytes
and typed errors together must fit max_output_bytes; transfer can also fail
if that actor has insufficient space. Report a distinguishable transfer error
when recoverable; an actor allocation trap must still retire the actor and
reclaim the independent completion. No worker frees or reads retired actor
storage, and no actor owns a pointer into a worker’s temporary stack.
max_worker_bytes requires real enforcement in the Plenty allocator path,
including worker-created client state and reallocation. Merely limiting the
returned body does not bound allocations made while fetching it. Reuse or extend
allocation-failure propagation so budget exhaustion is a recoverable job failure;
existing actor trap/OOM machinery is not a worker implementation. Audit the
client’s header/body/decompression limits as well. Foreign allocations and OS
thread stacks are outside this allocator budget and must be documented, bounded
where the wrapped API permits, and included in qualification measurements.
Eligibility and API capability
The first version uses capture-free named functions. Apply the existing worker eligibility checks to the function, transitive calls, errors, results and destructors. Owned bytes do not make an otherwise forbidden foreign effect safe. Reject borrowed state, uncertified indirect effects, actor addresses used as cross-thread senders, actor heap operations, executor submission and worker-side future waits.
E must additionally be independently owned, bounded and transferable into an
actor node under the actor type rules. Derive/check the transfer operations from
concrete types; do not introduce an unchecked public marker or raw-pointer API.
Restrict unsupported error types with a useful diagnostic. Account for the size
of any owned error payload, and retain the cause through an explicit adapter.
A byte-only input/result API does not silently authorize shared strings, locks,
channels or arbitrary handles in actor nodes.
The only new actor-storable types are the certified job handle and pool key. A handle identifies a registry entry; it does not own a native thread or a shared allocation directly. Dropping it marks the waiter abandoned without joining. Trap retirement must do the same through stage bookkeeping, because user destructors do not run on contained traps. Pool shutdown must remain safe even if no user-level job handle destructor ran.
Completion identity and cleanup
Use the stage attachment identity, actor slot generation and job generation when routing a completion. A recycled actor slot, new stage instance or new HTTP request must never receive an old result. Validate the result’s concrete error/transfer type against the job record before decoding it.
Exactly one owner wins a claim-versus-abandon race. Claimed results are consumed once; failed transfer destroys the source once. Abandoned queued jobs are removed before execution when possible. Abandoned running jobs retain their slot and cancellation state until they finish, then destroy output and release resources without posting to the retired actor. Cancellation cannot promise that an accepted operation has had no external effects.
Cleanup of worker-owned values, including abandoned results and errors, runs on a worker/reaper path whose lifetimes and capacity are bounded by the pool. Unbounded user cleanup must not run on the stage thread. The first version’s payload/error restrictions must permit bounded transfer and cleanup on the actor side. Reclamation cannot require a new queue allocation or mailbox slot.
Deadlines, cancellation and reliability
Submission takes an absolute Deadline and copies its monotonic value. Queue
waiting counts against it. Expired queued work must not start. The supervisor
signals a worker-owned cancellation token when the job is abandoned, its deadline
passes or the pool starts cancelling. A worker may inspect that token or pass it
to a cooperating library. The actor does not retain a shared token in its node.
Stopping the wait is prompt; stopping a blocking operation is cooperative.
A foreign call that ignores cancellation can continue after the HTTP request,
actor and server have ended. Its slot remains occupied, so repeated timed-out
requests cannot create unlimited replacement threads. The registry separates the actor’s terminal wait outcome from physical job
completion: deadline expiry can notify the actor with BlockingError.TimedOut
even while an uncooperative worker retains the job slot. No new per-expiry job
record is allocated. Late worker output is discarded after that outcome wins.
Serialize outcome publication, deadline expiry and abandonment under registry synchronization. At publication, compare the monotonic clock with the copied deadline: a result published at or after expiry loses to timeout, even if the stage has not processed its timer yet. A result published before expiry remains a result if the actor observes it later. Explicit abandonment suppresses further user notification and makes an already-queued notice stale. Tests must cover these orderings with controlled time and barriers rather than scheduler luck.
An ordinary returned Err(E) is a job outcome. A late or abandoned result is
reclaimed. Neither actor supervision nor thread isolation contains arbitrary
worker traps, stack overflow, native crashes or process-wide lock corruption.
The first version retains ordinary native worker fatal-trap semantics and must
say so prominently: code that would be contained inside an actor may instead
terminate the process when offloaded. Do not catch a runtime panic and report a
successful containment guarantee. Use a separate process when that isolation is
required. External database writes and HTTP side effects are never rolled back.
Blocking-work runtime and qualification
This records implementation guidance for the proposed actor blocking-work API, not a second work queue. B85 in the backlog owns scheduling and outstanding scope.
Runtime division
Implement policy, limits and the HTTP adapter examples in Plenty where practical. A small native boundary is necessary for thread creation, independently owned job storage, synchronization, allocator budgets, compiler-certified transfer and selector wake-ups. Ordinary Plenty code cannot safely construct these lifetimes from actor heaps and lexical thread handles. Reuse existing runtime thread and selector primitives where their contracts fit; do not add public raw std hooks.
Before coding, map the specialized DNS helper, actor retirement, selector handle tracking, native worker checks, callable lowering and allocator budget paths. The generic bridge is new work, not a wrapper around DNS’s private payloads. An existing executor cannot be assumed safe to retain or destroy in an actor. Sharing executor internals requires the same ownership and shutdown guarantees.
Bounded registry and wake protocol
Allocate max_jobs registry entries and completion capacity when opening the
pool; validate overflow and allocation failures. Each entry has an identity,
state, owner generation, deadline, cancellation flag, transfer descriptor and
bounded payload storage. Reserve a small allocation-free infrastructure-error
outcome per entry, so failure to allocate an ordinary output can still notify
the actor. Opening a pool must roll back already-started workers on partial
startup failure. Native synchronization must have short critical sections;
never execute work, user cleanup, blocking I/O or wait for capacity under a
stage-visible lock. No stage operation waits on worker completion.
Use the following lifecycle, with abandonment tracked independently once running:
Free -> Queued -> Running -> Completed -> Claiming -> Reclaiming -> Free
Queued -- cancelled before start --> Reclaiming
Running -- abandoned --> Running (no waiter) -- finished --> Reclaiming
Completed -- abandoned --> Reclaiming
A worker publishes its result before signalling the selector wake handle. The signal is a hint to drain the registry, not the sole record of completion. Coalesced or saturated wakes must not lose results. The stage drains a bounded number of completions per turn and schedules another internal turn if records remain; otherwise a completion flood can starve HTTP sockets and timers.
Deliver Mail.Blocking(job_id) through reserved control state, independent of
ordinary mailbox capacity and application timers. One reserved notification per
outstanding job is sufficient. Retry notification scheduling while the job is
live if necessary; do not silently discard it on Full. take_now may see a
pending job and return Ok(Nothing); repeated reads after consumption or explicit
abandonment return a named error. Stale already-queued notices are harmless and
never read reclaimed storage. Actor retirement cancels all its registry waiters.
Budget planning must count registry metadata, independent input/output copies,
worker scratch allocations, actor-side result copies, thread stacks and native
library allocations separately. max_jobs bounds retained input/output records;
max_workers bounds concurrent scratch budgets and stacks. Queued input must
not also acquire a full worker budget before execution. Measure transient
transfer peaks and cleanup backlog, not just steady-state body sizes.
Ready results consume admission capacity until claimed or reclaimed, bounding slow consumers. Scope attachment to one stage initially; one pool shared by many stages needs additional fairness and wake-routing contracts. Permit separate pools for different services so a stuck database need not exhaust HTTP-client workers, with process-wide resource budgets remaining the application’s concern.
Lifetime and shutdown
The outer driver owns the pool and a stage attachment. An actor cannot close or join the pool. HTTP serving needs an explicit optional pool attachment accepted by event-server entry points or a server setup object; do not hide a pool in a capture that violates the current factory/node rules. Preserve existing server entry points and defaults. This integration/API shape is an implementation design gate, not an assumed existing hook.
Expose nonblocking admission closure and explicit drain-versus-cancel modes. During HTTP draining, application completion events continue alongside socket progress until the grace deadline. After that, retiring connection actors abandons their jobs and detaches their response destinations. The pool must outlive running jobs; stage storage and wake handles cannot be freed while a worker may still access them. Use independently retained completion state and an explicit detach handshake, not borrowed stage pointers.
Final pool shutdown/join runs outside callbacks, with deadline: &Deadline and
cancel: &CancellationToken on a waiting API. If that wait times out, return an
error while retaining the owner for retry; it does not authorize freeing running
job storage. No implicit detached workers. Final owner destruction must join
remaining workers and can therefore wait indefinitely for uncooperative calls;
document this even when an explicit earlier wait was bounded. Do not promise a
bounded whole-process shutdown for arbitrary blocking libraries. Releasing a
stage attachment must only sever routing and signal cancellation, without joining.
Evidence required for adoption
Use a controlled local upstream and barriers/latches rather than relying on scheduler luck or an Internet endpoint. Tests should establish:
- An HTTP request waits on a blocked worker while unrelated connections complete, timers fire and cancellation is observed by the stage.
max_workersbounds concurrent execution;max_jobsincludes completed and abandoned-running work. Saturation returns original input without running it.- Actor traps, ordinary stop, disconnect, slot reuse and stage destruction before completion leave no dangling reads, misdelivery, lost cleanup or leaked heaps.
- Full mailboxes, exhausted application timers, coalesced wakes and completion bursts preserve notification and fair progress without busy polling.
- Deadline expiry before start, completion/abandon races and shutdown while queued, running or ready produce one outcome and one cleanup. A call ignoring its token holds its slot until the controlled upstream releases it.
- Allocation failure at admission, worker execution, publication and actor result transfer is recoverable where specified and never publishes a partial job.
- Compile-fail tests reject actor captures, borrowed payloads, forbidden effects, unsafe error types and ordinary thread spawning in actor callbacks. Valid fetch adapters pass the normal worker checks without an exemption.
- Returned typed upstream errors survive transfer. Fatal worker behavior is qualified in a subprocess and documented independently of actor containment.
- Repeated create/run/drain/drop cycles return threads, descriptors and tracked allocations to baseline; race/sanitizer checks cover abandonment and wake teardown.
An executable tutorial should implement an actor HTTP endpoint backed by a local blocking fetch, including capacity exhaustion, upstream failure, timeout and shutdown. Measure latency of unaffected requests, queue delay, throughput, CPU, thread count and peak memory against nonblocking actor I/O and the worker HTTP server. Report actual platform coverage; support must follow native thread and selector qualification rather than claiming portability from a Linux result.
Questions to settle during implementation
The safety contracts above are requirements; representation details still need prototypes: concrete generic error-transfer lowering, the bounded worker allocator, reserved completion-mail scheduling, the stage attachment lifetime, HTTP server pool injection, and errors/results that require thread-affine destruction. The initial version may narrow eligible errors further if safe bounded transfer cannot be established. Do not widen accepted types or drop allocation guarantees merely to make the first fetch example compile.
Keyword arguments, defaults, and static unpacking
Revised direction, 2026-10-17: Named arguments and fresh defaults is the preferred narrow proposal for B36. It separates named calls and defaults from splat unpacking and signature markers, and implements fresh fallible defaults. This older document retains the broader alternatives; its constant-only restriction and combined first stage are not the new direction.
Status: Partly implemented (B36). Named calls and fresh per-call defaults ship for named functions, methods and constructors. Splats, variadics and positional-only/keyword-only signature markers remain unimplemented. Signatures and calls owns the current contract, including callable and protocol restrictions. The present tense below records the older, broader proposed design; its constant-only defaults are superseded by fresh expression defaults.
Python’s keyword arguments, defaults, and */** unpacking make calls and
literals easier to read and write. Plenty adopts them only where the compiler
can resolve them completely. This is not a compatibility goal: a Python form
that needs runtime arity checks, runtime key lookup, dynamic typing, or hidden
allocation is rejected, not approximated.
In this role * and ** are unpacking syntax, not operators. Binary a * b
and a ** b keep their arithmetic meaning.
Rules every form must satisfy
- Static resolution. Every call is rewritten during checking into one positional call to a known declaration. Codegen and the runtime never see names, defaults, or unpacking at call sites.
- No runtime call failure. Arity, names, and types are checked at compile time. No form can fail at run time because a sequence has the wrong length or a mapping lacks a key.
- No hidden allocation. A form that allocates already has a
Resulttype at that site (a collection display). A call never allocates to pass its arguments. - One rule, no exceptions. Each form has one meaning, and the conditions for accepting it can be stated in a sentence.
Keyword arguments
Any parameter of a named function, method, generic function, or class constructor can be passed by name unless it is positional-only:
def scale(value: i64, factor: i64, offset: i64) -> i64:
value * factor + offset
scale(3, offset=1, factor=2)
- Positional arguments come before keyword arguments.
- Each parameter is bound exactly once. Unknown names and duplicates are errors.
- Arguments are evaluated and moved left to right in source order, as in Python, not in parameter order. A call’s effects follow the text.
- Expected-type inference works as it does for positional arguments:
factor=2againstfactor: u8produces au8.
Parameter names become part of a declaration’s public interface: renaming a
pub function’s parameter can break callers. Positional-only parameters (below)
let an author opt out.
Function values stay positional. Callable[[A], B] has no parameter names,
and this proposal does not add them. A call through a function value accepts
positional arguments only. A keyword call needs a statically known declaration.
Generated constructors take field names as keywords:
Point(x=3, y=4). This needs no new mechanism, since the generated
constructor’s parameters are the fields.
Builtins follow the same rules. Builtins with optional arguments, such as
splitlines and open, become ordinary signatures with a keyword and a
default.
Protocol requirements. The reference says implementation parameter names may differ from the protocol’s. Keep that. A generic body checks keyword calls against the protocol requirement’s names and markers, then reaches the implementation positionally. Implementation names never matter through a bound.
Positional-only and keyword-only markers
Python’s / and bare * markers are adopted unchanged:
def clamp(value: i64, /, low: i64, high: i64, *, wrap: bool) -> i64:
...
Parameters before / are positional-only. Parameters after * are
keyword-only. Keyword-only parameters suit Boolean flags and options, where a
bare positional True hides intent.
Default values
def split(text: &str, separator: &str = ",", limit: i64 = -1) -> Result[list[str], AllocError]:
A default is an allocation-free constant expression: a numeric, bool, or
str literal, unit, Nothing, or a nullary enum variant that needs no
allocation. A default cannot reference other parameters, globals, or calls.
Inside these limits, the moment of evaluation (definition time or call time) has
no visible effect, so Python’s shared mutable default problem cannot occur.
- A positional parameter with a default cannot be followed by a positional parameter without one. Keyword-only parameters can appear in any order.
- A parameter whose type mentions a type parameter cannot have a default. A default must check against a concrete type, never per specialization.
- The compiler inserts omitted defaults during the positional rewrite.
Class field defaults use the same constant rule and become defaults of the generated constructor. They are currently deferred in the class reference and can land with this work or later.
Unpacking at call sites
*tuple expands a tuple value into positional arguments:
point = (3, 4)
distance(*point)
The operand’s static type must be a tuple, so its arity and component types are known. The tuple is evaluated once at its source position and consumed like a whole-tuple unpacking. Several starred tuples may appear in one call. No storage is allocated.
**record expands a class value into keyword arguments named by its fields:
def draw(x: i64, y: i64, *, color: Color) -> ():
draw(**origin, color=Color.Red)
- The operand’s static type must be a class. Its visible fields must each match a parameter name. An extra field is an error, not an ignored value.
- The record is consumed and its fields move into the call. Moving fields out
of a class needs a whole-value destructuring rule that the language does not
have yet. As a hard rule,
**rejects classes with a custom__del__, since their fields cannot be separated from the destructor. - Remaining parameters come from other arguments or defaults.
Rejected: *list, *generator, or any operand whose length is not in its
type, used for fixed parameters. **dict in any position. Both need runtime
checks that could fail during the call.
Variadic parameters: *args: T
def total(*values: i64) -> i64:
mut sum = 0
for value in values:
sum = sum + value
sum
total(1, 2, 3)
The extra arguments must all have one element type T. At each call site the
count is known, so the caller places the evaluated arguments in a stack buffer
in its own frame and passes a pointer and length. Inside the body, values is a
read-only borrowed view of that buffer. The caller drops the elements after the
call returns. No heap allocation is involved.
- Generic
*values: TinfersTfrom all extra arguments. Under the existing inference rule, all evidence must agree. - A list can supply the whole variadic group,
total(*numbers), by passing a borrowed view of its storage. Mixing it with other extra arguments, as intotal(1, *numbers), is rejected because it would need a buffer of unknown size. - Heterogeneous variadics (
*argsof mixed types) need variadic generics and are out of scope. - Taking a variadic function as a
Callablevalue is deferred until the view type has a stable spelling for callable signatures. *argshas no connection to C varargs, which stay excluded fromextern def.
Dependency: a borrowed sequence view type. Plenty has no slice or span type
yet, and a view should be designed for its own sake (string, list, and buffer
views) before *args relies on it. A consuming variant, where the callee owns
the elements, can follow once owned views exist.
No **kwargs parameters
Python’s **kwargs is a dict[str, Any]. It needs dynamic typing and allocates
a dictionary on each call. Plenty does not add it.
Its main uses already have static alternatives:
- Option bags: use keyword-only parameters with defaults, or take an
explicit options class with field defaults:
render(page, opts=Options(dense=True)). - Forwarding: a wrapper takes
opts: Optionsand passes**optsto the inner call.
A future “keyword group” parameter that exposes a class’s fields as keyword-only parameters without constructing the class is possible. Defer it until a concrete forwarding need shows that the alternatives above are insufficient.
Unpacking in displays
Collection displays already return Result, so spreading into them adds no new
failure path. A tuple display is inline and allocates nothing:
both = [*first, *second]?
members = {*left, *right}?
merged = {**defaults, **overrides}?
extended = (*pair, 3)
- In a list or set display,
*sourceaccepts the owned sources thatlist[T].fromaccepts: a collection, range, or generator. The source is consumed. Usecopy(source)?to keep the input, asupdatealready requires. - In a dict display,
**sourceaccepts only an owneddict[K, V]of the same type. Later keys replace earlier ones, matchingdict.update. - In a tuple display,
*sourceaccepts only a tuple, so the result arity is static. - Construction stays left to right. The first failure releases the initialized prefix and drops evaluated sources not yet transferred; later entries are not evaluated. Implementations may sum known source lengths to reserve once.
return *head, lastandyield *head, lastare tuple displays and follow the tuple rule.
Starred targets in bindings
first, *_ = triple ignores the remaining tuple components and drops their
owners. It is static and allocation-free, so it is accepted.
first, *rest = triple with a named rest was rejected while tuples were
heap records: rest would have been a new allocation inside a binding statement
that has no place for ?. Tuples are now inline, so the form is allocation-free
and the rejection needs revisiting.
head, *tail = items on a list is rejected permanently as a binding: the list’s
length is not in its type. The checked form is a sequence pattern, where a length
mismatch is an explicit branch, not a hidden failure:
match items:
case [head, *tail]:
...
case []:
...
tail would be a borrowed view. This depends on the same view type as *args
and on sequence patterns, which match does not support yet. Mapping patterns
with **rest are rejected.
Rejected Python forms
| Form | Reason |
|---|---|
**kwargs parameter | Dynamic typing and a per-call dictionary allocation |
f(**mapping) with a dict | Keys are checked only at run time |
f(*items) into fixed parameters, for an operand whose length is not in its type | Arity is checked only at run time |
f(1, *items) into *args | Needs a buffer of unknown size |
Mixed-type *args | Needs variadic generics |
| Defaults that are arbitrary expressions | Hidden allocation and effects; Python’s evaluation-time trap |
head, *tail = items on a list | Runtime length check in a statement |
case {**rest} mapping patterns | Runtime key sets |
Keyword calls through Callable values | Callable types carry no parameter names |
Suggested stages
- Keyword arguments,
/and*markers, constant defaults, keyword constructors, and builtin special cases moved onto ordinary signatures. This stage is frontend-only. *tupleand**recordat call sites, and*_in tuple bindings.- Spreads in list, set, dict, and tuple displays.
- A borrowed sequence view type, then
*args: T. - Sequence patterns with
*rest, after views and sequencematch.
Each stage updates the function reference, the tutorial lessons that teach calls, and its backlog entry (B36).
Open questions
- The spelling of the view type that
*argsand*restbind. - Whether
**recordshould accept classes with__del__once a whole-value destructuring rule exists. - Whether protocol requirements may declare defaults, or only names and markers.
- Whether class field defaults ship in stage 1 or wait for a separate change.
Returned enums that carry references
Status: Phase 1 implemented (issue #520). Functions and methods can return
Option[&T], Option[&mut T], Result[&T, E], and nested success payloads of
these; callers match, propagate, unwrap, inspect, print, forward, or hold the
result in an immutable local. The reference is authoritative:
optional and fallible references.
Phase 2 (mut holders, parameters) and phase 3 (pipelines) are not
implemented, and standard library getters such as dict.get_ref remain
follow-up work under B11. This page is the design record; examples below may
predate the implemented diagnostics.
Problem
“Find it, or say it is absent” is one of the most common things a program does. In Plenty today a function cannot return a borrow that may be absent:
def find(items: &list[Item], name: &str) -> Option[&Item]:
# ^ references cannot be stored in enum payloads
Every container that holds non-copyable values works around this, and each workaround costs something:
| Workaround | Where it is used | Cost |
|---|---|---|
Callback getters, get_with(&key, &f) -> Option[R] | std.cache.lru get_with, peek_with | A callback per lookup; no ?, return or break from inside it; an extra type parameter in every signature |
| Trapping getters plus a membership test | LruCache.get_ref/peek_mut, Heap.peek, graph.node/edge | Two lookups (contains then the getter); a missing key is a crash rather than a value |
| Return an index, then index again | std.http.find, std.http.cookie.find, std.http.body.find, std.json.find, std.toml.find | A second bounds check; the index is not tied to the collection, and private layouts (LRU slots) cannot expose one |
| Copying lookup | dict.get, list.get | Rejects affine values (“get cannot return owned dictionary values”) |
| Rejected outright | next(p), find and reduce on pipelines of &T items | “references cannot be stored in a find result; map to an owned value first” |
The lookup lesson currently tells
learners “Optional borrowed lookup is not implemented”. std.graph traps on an
invalid handle by calling itself with an out-of-range index, because a getter
cannot report absence.
Plenty avoids lifetime annotations with two rules: a returned reference borrows
from the first reference parameter (the origin), and references never enter
aggregates. This proposal keeps both. It adds one narrow exception: an
Option or Result may carry a reference as a value in flight, produced by
a call under the origin rule and consumed locally, but never stored.
What exists today
Probes ran on local main at 13579f12 with a debug compiler
(plenty --check).
| Probe | Result | Consequence |
|---|---|---|
def find(xs: &list[Item], n: &str) -> Option[&Item] | Rejected: references cannot be stored in enum payloads | The motivating signature |
-> Result[&str, i64], o = Some(&xs[0]), o: Option[&i64] = Nothing | Same rejection | Construction and annotation share one barrier (Ty::contains_reference) |
enum Found: Hit(&str) | Rejected at the declaration | User enums use the storage-barrier check |
-> (&str, i64), t = (find(&xs, "a"), 1) | Rejected: references cannot be stored in tuples | Tuples are aggregates; out of scope here |
Callable[[&str], Option[&str]] | Same rejection | Callable signatures share the type check |
def pick[T](x: T) -> Option[T] called with &s | Rejected: reference type arguments are not supported | A type parameter is never a reference; generic user enums cannot carry one |
d.get("a") with list[i64] values | Rejected: get cannot return owned dictionary values | Affine values have no optional lookup |
next(p), iter(&xs).find(f) with &str items | Rejected: references cannot be stored in a next/find result | Pipelines hit the same wall |
Return a &mut element on one path, xs.append(...) on the other | Accepted and runs | Loans already end per path, so get-or-insert shapes work |
def touch(xs: &mut list[i64]) -> &i64, then read xs while the result is live | Accepted | A shared result from an &mut origin holds a shared loan |
Payload reference returned from match &node.next | Accepted | Borrowed-match payloads already satisfy the origin rule |
Relevant implementation facts:
- Representation. Wide values (enums, classes) and references are 128-bit
slots. A reference’s low word is a slot address; its high word is a tag
shift, non-zero only after projecting into a nested sum payload
(
src/codegen/references.rs). Enum tags stack in the high word: wrapping is(payload_tags << tag_bits) | tag, unwrapping shifts back (src/codegen/enums.rs,wrap_tagged,tagged_payload). There is no niche optimization. - Origins. The checker picks an origin only when the result type is
Ty::Ref(src/frontend.rs,return_origin);check_return_referencecompares each return path’s loan root with it, andcall_reference_resultgives the caller a child loan of argument 0’s loan (src/frontend/references.rs). The ownership pass itself only seesLoan/UseLoan/EndLoanops, so it is type-agnostic. - Local loan carriers that are not references already exist.
KeyViewvalues andIter[T]pipelines hold loans per slot throughreference_locals; neither can be returned. ?and builtins.?consumes the operand and moves the payload out (lower_try).unwrap,unwrap_or,ok_or,is_someand friends are compiler builtins (src/frontend/collections.rs), not generic Plenty methods; neither?norunwrapforwards a loan today. There is nomaporand_thenonOption/Result, and noif let/while letform:matchis the only pattern construct.
Precedents
| Language | Optional borrow | How it stays sound |
|---|---|---|
| Rust | Option<&T>, Result<&T, E>, any enum | Lifetime parameters, elided for one input reference; references may be stored anywhere a lifetime allows |
| Swift | No borrowed optionals; Optional<T> copies or retains | Reference counting; inout/borrowing parameters cannot be returned |
| Hylo (Val) | No first-class references; subscript yields a projection for a scope | Projections cannot escape their scope |
Rust’s lifetime elision rule (“one input reference: the output borrows it”) is Plenty’s origin rule. Rust needs annotations only for the cases this proposal continues to reject: several possible origins and stored references.
Proposal
The rule
A reference-carrying enum is an Option or Result type in which a
variant’s payload is a reference, directly or through further Option/Result
payloads: Option[&T], Option[&mut T], Result[&T, E],
Result[Option[&T], E]. Its reference payloads behave exactly like a returned
reference, and the value itself is never stored:
- Produced by a call or a local construction. A function or method may
declare a reference-carrying enum as its return type. Every return path,
including each
?that propagates, must originate in the first reference parameter under the existing returned-reference rule.Nothingand payload-free variants originate in nothing and are always accepted. A function without a reference parameter cannot declare one, and an&mutpayload requires an&mutorigin. - Held by a local. The caller may bind the result to a local, which holds
the origin loan until the last use of the local and of every reference
extracted from it.
Some(&place),Ok(&place)andErr(&place)may be bound to a local or returned; the binding holds that place’s loan, asr = &placedoes. - Consumed by inspection. It may be matched, propagated with
?, tested (is_some,is_ok, …), unwrapped, printed, or passed straight toreturn. - Never stored. It cannot enter a tuple, class field, list, dictionary,
set,
Box, closure capture, another enum value, a scoped worker, a channel, or a live state acrossyield. It cannot be a type argument other than the payload position ofOption/Result, solist[Option[&T]],Generator[Option[&T]]andBox[Option[&T]]stay rejected.
Rule 4 is the existing “references never enter aggregates” rule with no
exception: the enum is the reference’s wrapper, not a container it is stored
in. A nested type such as Result[Option[&T], E] is written in one signature
and produced as one value; wrapping an existing Option[&T] local in Ok(...)
at a return is permitted for the same reason, and is the only place an enum
value may appear inside another.
class Item:
name: str
tags: list[str]
def find(items: &list[Item], name: &str) -> Option[&Item]:
for item in items:
if item.name == name:
return Some(item)
Nothing
def main() -> Result[(), AllocError | IoError]:
mut items = [Item("ada", ["x"]?)]?
match find(&items, "ada"):
case Some(item):
print(item.tags)? # `item: &Item`; `items` is borrowed here
case Nothing:
items.append(Item("ada", []?))? # the loan ended at the match
Ok(())
Which enums qualify
| Option | For | Against |
|---|---|---|
Option only | Smallest change; covers get, find, first | Fallible borrowed lookups (Result[Option[&V], E], a parser returning Result[&Token, ParseError]) remain impossible |
Option and Result, nested in each other (recommended) | Covers infallible and fallible lookups; both are builtin sums whose methods are compiler builtins, so no generic instantiation with a reference type argument is needed; ? already treats them specially | Two types to document instead of one |
| Any enum, including user enums | Uniform | User enum payloads come from declarations or type parameters. A declaration naming &T would be the first user type with a reference field, which reads as “references can be stored”. A generic Lookup[&V] needs reference type arguments, which Plenty does not support and which would make every generic body a potential reference holder |
Recommendation: Option and Result only. User enums keep the rule that
declarations and type arguments never contain references. Revisit only with a
concrete case that Result[Option[&T], E] cannot express.
Anonymous error sums (AllocError | IoError) are Result error types and may
not contain references; an error that borrows its input would escape through
? into unrelated callers’ error unions.
Matching, ? and builtins
matchby value on a reference-carrying enum binds payload names as the reference type itself (item: &Item,slot: &mut V). Each binding is a child of the enum local’s loan, like a borrowed-match payload. In an arm without payload bindings, the enum’s last use is the scrutinee, so the loan ends there; this is what lets theNothingarm above mutateitems.match &ois rejected: a reference to a reference-carrying enum would be a reference to a reference, which Plenty does not have. The diagnostic says to matchodirectly. Matching a sharedOption[&T]by value copies only the reference and leavesousable.?returns early onNothing/Errand yields the payload reference, which inherits the operand’s loan:item = find(&items, "ada")?is a reference binding from a call result. Propagation is a return path: in a function returningOption[&T],?onNothingreturnsNothing, which borrows nothing, and the extracted reference must still reach any returnedSomefrom the origin.is_some,is_none,is_ok,is_errread the tag; they are uses of the value and extend its loan to that point only.unwrap()yields the reference with the operand’s loan and traps on absence, the same as indexing.ok_or(error)turnsOption[&T]intoResult[&T, E]with an owned error; the loan carries over.unwrap_or(default)with a reference default is rejected. The result could borrow either the option’s origin ordefault, which is a conditional reference expression; those are unsupported, and supporting them only here would give one builtin a rule no function signature can state. The diagnostic suggestsmatch.- Printing reads through the reference (
Option[&str].Some("ada")). Equality is not extended in phase 1.
match is the only pattern form; if conditional forms such as if let are
added, they bind payloads as match does. Should map/and_then combinators be added later, they follow from the
signature rule: the receiver is the first reference-carrying parameter, so
o.map(f) may return a borrow only through f’s own origin, its argument.
Err payloads
Recommendation: in Result, only the Ok payload may carry a reference.
Error values are routinely widened into anonymous sums, printed by main, and
propagated far from their origin; a borrowed error would make ? in an
ordinary Result[(), IoError | E] caller fail with an ownership error. Errors
that need to describe input should copy the relevant part, as they do today.
&mut payloads and aliasing
Option[&mut T] behaves as an exclusive reference. It is affine: a by-value
match, ?, unwrap, passing it, or returning it moves it, and the extracted
&mut T takes over the loan. While the value or its payload binding is live,
the origin is exclusively borrowed, so two get_mut results from one
dictionary cannot be live together (dictionary keys already protect the whole
collection). A shared result from an &mut origin holds only a shared loan,
as today: cache.get_ref(&k) may promote the entry through &mut self and still
let the caller read cache.len() while the result is live.
Get-or-insert works without new rules, because loans already end per path:
def counter(counts: &mut dict[str, list[i64]], key: &str) -> Result[&mut list[i64], AllocError]:
match counts.get_mut(key):
case Some(found):
return Ok(found)
case Nothing:
pass
counts.insert(*key, []?)?
Ok(&mut counts[key])
Loans, last use and temporaries
The loan model is the one returned references already use. A
reference-carrying local is a loan holder like a reference local; payload
bindings, ? results and unwrap results are child loans. The origin loan
ends at the last use of all of them on each path. A precise field origin stays
precise; indexed and control-dependent bodies protect the whole argument, as
they do for &T results.
Temporaries follow the returned-reference rules unchanged. A method returning
a reference-carrying enum needs a named receiver or a place in named storage.
With a temporary shared argument the result is usable only within its full
expression: print(lookup(&row(), "id").is_some())? works; binding the result
is rejected.
Mutable locals
A mut reference-carrying local may be reassigned, which is how a search
remembers its best match without an index:
def longest(words: &list[str]) -> Option[&str]:
mut best: Option[&str] = Nothing
for word in words:
match best:
case Some(current):
if word.byte_len() > current.byte_len():
best = Some(word)
case Nothing:
best = Some(word)
best
It follows reference rebinding:
the first reference it holds fixes its first place, later Some(...)
assignments must stay within that place, and assigning Nothing keeps the
first place. A binding initialized with Nothing takes its first place from its
first Some assignment. Recommendation: phase 2, after immutable locals have
shipped, because it touches the rebinding checker.
Parameters, generics and Callable
- Parameters. Recommendation: allow reference-carrying enum parameters in
phase 2. They give optional borrowed arguments
(
def render(node: &Node, parent: Option[&Node])) and let helpers take a lookup result. Such a parameter counts as a reference parameter when choosing the origin, sodef or_first(found: Option[&T], xs: &list[T]) -> &Tmay return onlyfound’s payload. Passing a shared one copies it; passing an&mutone moves it. - Generics. Type parameters still never bind references, so
Option[T]withT = &Ustays rejected; a generic function must spellOption[&T]. Its body is checked per instance, as reference-returning generic functions are now. Generic class methods such asLruCache[K, V].get_ref -> Option[&V]need no copyable bound, so they work for every instantiation regardless of issue #515. That issue still matters for copying accessors; this proposal does not depend on it. Callable.Callable[[&K], Option[&V]]is accepted, with the origin fixed by the first reference-carrying parameter, as for&Vresults.
Layout and code generation
No runtime cost and no new representation. An Option[&T] is one 128-bit slot,
the same size as Option[i64] and &T:
| Value | Low word | High word |
|---|---|---|
&T | slot address | tag shift |
Some(r) | slot address | (shift << 1) | 1 |
Nothing | 0 | 0 |
Ok(r) in Result[&T, E] | slot address | shift << 1 |
Err(e) | e’s word | (e.tags << 1) | 1 |
This is the existing nested-tag scheme: wrap_tagged and tagged_payload
already restore a payload’s high word exactly, so construction is a shift and
an or, and a match tests bit 0. The shift needs at most 7 bits, so layout
depth accounting charges a reference payload 7 tag bits within the existing
64-bit tag budget.
A null-address niche (the Rust layout) is rejected: the slot is 128 bits either
way, so it saves nothing, and it would add a special case to every tag read
(sum_tag, Unwrap, lower_try, widen_sum). Values are not stored, so no
heap or owner-local storage is involved, and there is no cleanup: a
reference-carrying enum is never managed.
Diagnostics
Each rejection names the value, the place, and the fix:
| Case | Message |
|---|---|
| Stored in an aggregate | Option[&Item] carries a reference and cannot be stored in a list; match it and store an owned copy or an index |
| Class field or user enum declaration | references cannot be stored in class fields; Option[&T] and Result[&T, E] may only be returned or held in locals |
| No reference parameter | a function returning Option[&Item] needs a reference parameter to borrow from |
| Wrong origin | this Some borrows fallback, but Option[&Item] results borrow the first reference parameter, items |
&mut payload from a shared origin | Option[&mut V] requires the first reference parameter to be &mut |
Reference in Err | Result errors cannot carry references; copy the part of the input the error reports |
match &o | o already holds a reference; match o directly |
unwrap_or(&other) | unwrap_or cannot choose between two borrows; use match |
Across yield, into a scoped worker, into a closure | found holds a borrow of items and cannot cross yield / be sent to a worker / be captured; match it first and pass the reference |
| Temporary origin | lookup(...) borrows a temporary argument; bind the argument to a local first |
| Type argument | reference type arguments are not supported; write Option[&T] in the signature (existing message, reused) |
Teaching
- Look up a value that may be absent
replaces “Optional borrowed lookup is not implemented” with a short
dict.get_refsection:getcopies,get_refborrows, and the dictionary stays borrowed until the payload’s last use. Readers already know element borrows from the preceding lessons. - Return a reference to a field
gains “Return a reference that may be absent”, writing
findwithOption[&Item]and showing?on it. This is the first point where readers write their own returning functions with references. - The reference gains a page after returned references, “Optional and fallible references”, stating rules 1 to 4, the layout, and the rejected cases. Returned references and borrowing link to it instead of saying references never enter enums.
The rule to teach is one sentence: a lookup can return Option[&T]; treat
the result like the reference it holds, and never store it.
Standard library changes
Naming follows one rule across builtin and std containers (decided after this
proposal was accepted; see
copying and borrowed lookup):
indexing and plain accessors trap; get and peek return a copy in an
Option; get_ref, peek_ref, and get_mut/peek_mut return borrows.
| Module | Today | Proposed |
|---|---|---|
builtin dict | get(k) -> Option[V] copies non-affine values | Add get_ref(k) -> Option[&V] and get_mut(k) -> Option[&mut V] for every value type |
builtin list | get(i) -> Option[T] copies | Add get_ref(i) and get_mut(i); get_ref(0) and get_ref(-1) replace a separate first/last |
std.cache.lru | get_with, peek_with, trapping get_ref, peek_mut | Copying get(&mut self, k) -> Option[V] (promotes) and peek(k) -> Option[V], possible now that methods are checked only when used (issue #515); get_ref -> Option[&V] (promotes), get_mut -> Option[&mut V], peek_ref -> Option[&V], peek_mut -> Option[&mut V]; remove the callback and trapping forms (issue #516) |
std.graph | node/edge trap; callers test contains_node first | Keep trapping node/edge (handle indexing); add get_node_ref, get_node_mut, get_edge_ref, get_edge_mut |
std.collections | Heap.peek, HeapBy.peek trap when empty | peek_ref() -> Option[&T], and a copying peek() -> Option[T] for copyable T, matching pop() -> Option[T] |
std.deque | No access without removal | Add get_ref(i)/get_mut(i) if a use case appears |
std.http, std.http.cookie, std.http.body, std.json, std.toml | find(...) -> Option[i64] | find(...) -> Option[&Header] (and the matching element type); callers currently index immediately |
| pipelines | next, find, reduce reject &T items | Phase 3: return Option[&T] borrowing the pipeline’s source |
Not affected: regex Captures.get (copyable Match), env.get (owned
text), and toml.Document.get, whose arena nodes are not Toml values and
which therefore still builds a copy.
Std has no stability promise yet, so the replaced forms are removed in the same change rather than deprecated, keeping one obvious way.
Alternatives considered
| Alternative | Why rejected |
|---|---|
| Status quo with callbacks | Every container repeats the work; callbacks cannot ?, return or break; signatures gain R and F parameters |
| Membership test plus trapping getter | Two lookups per access; a forgotten test is a crash rather than a type error |
| Handles and indices | A second bounds check, no relation between index and collection, and impossible for private layouts and dictionaries |
A dedicated optional-reference type (&T? or Ref[T]) | New syntax and a type learners must map back to Option; match and ? already handle Option |
| Any enum carrying references | Needs reference fields in declarations or reference type arguments; see Which enums qualify |
| References in all aggregates with lifetime parameters | Breaks the no-annotation design goal and adds checker and learner cost far beyond these use cases. This proposal does not preclude a later B11 decision on stored references |
Returning Option[&T] but forbidding local bindings | Forces every lookup into a match scrutinee; ? could not bind its result |
Implementation plan
- Types and checking. Add a
carries_loanpredicate (references and reference-carrying enums) and use it where origin selection,return_origin,check_return_reference, trailing-expression checks andcall_reference_resultnow testTy::Ref. Admit reference payloads inOption/Resultresolution only for return types and local annotations; keep the storage barrier for every other position, with the messages above. Make?,unwrapandok_orforward the operand’s loan, and by-value matches create child payload loans. Charge 7 tag bits in layout depth. Tests: acceptance, every rejection row, loan ends per arm,&mutaliasing, temporaries, generic per-instance checking, and codegen of nestedResult[Option[&T], E]. - Builtins and docs.
dict.get_ref/get_mut,list.get_ref/get_mut; the new reference page; the two tutorial sections. - Std adoption.
std.cache.lru(issue #516),std.graph,Heap.peek, thefindfunctions, each with its reference page and tests. - Extensions.
mutreference-carrying locals, reference-carrying parameters, then pipelinenext/find/reduceover&Titems.
Each step is usable alone; step 1 with Option alone would already serve user
code.
Open questions
- Scope:
Optiononly, orOptionandResult? Recommendation: both, with nesting, because fallible borrowed lookups are common and the two share one implementation. - May
Errcarry a reference? Recommendation: no; seeErrpayloads. - Nesting
Result[Option[&T], E], includingOk(found)at a return. Recommendation: allow. It is still one value produced by one call, and rule 4 continues to reject every stored position. - Local construction (
found = Some(&xs[0])). Recommendation: allow in phase 1; it isr = &xs[0]plus a tag.mutreassignment waits for phase 2. - Reference-carrying parameters. Recommendation: allow in phase 2, with the parameter counting toward origin selection.
- Builtin names:
get_ref/get_mut, or changegetto borrow affine values? Recommendation: new names. One method whose result is a copy fori64and a borrow forlist[i64]would hide which one the reader has. Decided:get/peekcopy andget_ref/peek_ref/*_mutborrow, in every container, includingstd.cache.lru. - Trapping accessors: replace or keep beside
get_*? Recommendation: changeHeap.peekto returnOption; keepgraph.node/edgetrapping like indexing and addget_node_ref/get_edge_ref; remove the LRU callback and trapping getters. unwrap_orwith a reference default. Recommendation: reject, as conditional reference expressions are.
Typestate and owned transitions
Status: Investigation complete. This explores existing facilities and limits, not a new language feature. Moves and destruction define current behavior. Scheduling belongs in B72.
The owned response experiment refreshes this assessment, with native cleanup/retry checks and compiler rejections. Explicit owned receivers now support method-shaped transitions, and field moves out of owned bindings are implemented (issue #534).
The separate linear-obligations proposal (issue #149) addresses non-abandonment of an owner, including retry errors. Typestate can remain affine when abandoning a state is harmless. See the integration assessment for a protocol example combining state transitions with obligations, worker effects, and execution responsibility.
What Plenty can express now
Typestate makes an operation available only for an object in a particular
state type. Distinct nominal classes plus consuming free functions express a
basic form today: prepare(draft: Draft) -> Ready moves the draft, and
send(ready: Ready) cannot accept an unprepared draft. The ownership checker
prevents reuse of the consumed binding. Private fields/constructors and module
visibility must prevent callers fabricating the supposedly validated state.
A small Draft to Ready probe type-checked with the available local debug
compiler during this assessment. This was a smoke check of that binary, not a
fresh build or an exhaustive typestate conformance test. The design basis is
the current ownership,
classes, and
function contracts.
Ordinary methods borrow their receiver by default. Explicit self: Class or
self: Enum uses the same by-value rules as a function argument. For affine
states, a successful call or an error both consume the previous binding; a
retryable error must explicitly carry the owner back. Copyable enums still copy.
There are no specialized method declarations for Connection[Open] versus
Connection[Closed]; distinct classes remain the conservative starting point.
Generic instance methods are now checked only when used (issue #515), so an unused method need not work for every argument type. This improves flexibility but does not add explicit state-specialized method sets. Owned receivers are an independent feature, documented with method contracts.
Moving an entire owner is supported, and so is moving an affine field out of
an owned binding (issue #534; see field moves). A transition that rebuilds a wrapper
around an owned connection can move the connection field out of the consumed
state. A class with __del__ still needs an Option plus replace, an enum
whose payload can be consumed, or API restructuring.
That friction is relevant to whether a typestate design is pleasant, not a
reason to weaken initialization guarantees indiscriminately.
The HTTP client now demonstrates the API-restructuring option (issue #134,
commit 2772825). run returns the response head and an Incoming body owner
as tuple components. fetch can consume the body and move the head into its
result without constructing a fallible placeholder; send assembles the public
Response only when needed. attempt likewise carries its head separately.
This removes the observed placeholder work using existing tuple moves, without
adding partial class moves or compile-time response states.
Actual stdlib use
Area at baseline bef41b1 | Existing approach | Assessment |
|---|---|---|
http.client.Response, Client.finish | Response owns the connection; finish consumes it | Ownership expresses a lifecycle boundary, but body completion is checked at runtime |
http.server.Exchange.start/write/end | One class with integer state, remaining length and flags | Not compile-time typestate; invalid order returns an error |
http.BodyReader | Runtime state machine over input-dependent chunk framing | Runtime states are natural; enums can improve internal invariants |
json.Decoder | Runtime token/Unicode/container states | Arbitrary input cannot reasonably become a compile-time sequence of state types |
json.Builder.Frame | Payload enum for array/object accumulation | Good state representation, distinct from public typestate |
db.sqlite.Transaction.commit/rollback | Mutable methods; a busy commit leaves the transaction retryable | Resource ownership with dynamic lifecycle, not consuming state transitions |
The stdlib therefore uses ownership more broadly than typestate. An enum eliminating an invalid combination of internal fields and a public API rejecting an invalid operation sequence at compile time are complementary techniques.
Useful experiments and tradeoffs
For server responses, compare an API where beginning a response consumes an
unanswered handle and returns a writer, and ending consumes that writer. This
can rule out writing before start or ending twice. Runtime checks still enforce
declared content length, I/O failures, and cancellation. Separate size/chunked
writers only if they simplify callers rather than multiplying types and generic
parameters. Existing handlers borrow Exchange, so a consuming design also
requires deciding how request-body access and handler ownership interact.
For builders, use distinct stages only for meaningful required choices. Optional configuration normally suits a single options class with validation. An explicit constructor establishing JSON decoder invariants is preferable to a 28-stage builder mirroring fields.
Failure ownership is part of every transition’s contract. A failed consuming transition can destroy the resource, or return the still-valid owner alongside the error for retry. SQLite’s busy-commit behavior demonstrates why blindly changing a mutable method to a consuming function can lose useful behavior. Cancellation and partial external effects likewise need explicit outcomes.
Avoid placing every connection phase in a distinct public type if callers must immediately put those types back into an enum for queues, pools, or runtime routing. Compare end-to-end handler code, not just the impressive compile-time rejection of one small example.
Validation before adoption
For each experiment, demonstrate one useful compile-time rejection, legal retry after failure, preservation of partial-progress reporting, and exactly-once cleanup. Check that public construction cannot forge states. Measure new public types, caller branching, placeholder replacements, and code size. Keep runtime enums for input-driven parsers; prototype typestate at coarse ownership boundaries. No blanket stdlib conversion is proposed.
Linear obligations
Status: Not implemented; proposed design for review. Issue #149, tracked under B72. No syntax or standard-library conversion is approved. The current contracts remain ownership, destruction, and error propagation.
This proposal defines an opt-in ownership obligation. Its aggregate and error rules are part of the same semantic core. A separate scope-resolution proposal adds automatic synchronous finishing. The integration contract compares proposal boundaries, effects, typestate, and future concurrency work. All code fences in these pages are pseudocode, including names, annotations, signatures, and outcome types; none are runnable Plenty examples.
Purpose and guarantee
An affine owner may be consumed at most once and may be discarded. A linear owner must be consumed exactly once on an ordinary completed execution path: resolve its obligation through a permitted terminal operation, or transfer it to another owner that inherits the same requirement. Reads, borrows, and intermediate operations are not consumption. They may happen repeatedly.
This makes obligations such as responding to a request or finishing a local transaction visible across calls and storage. “Run the entirety of a state machine” means not abandoning an outstanding obligation before an allowed terminal outcome. It does not mean visiting every state, forcing success, or proving termination. A retry loop may run forever while retaining its owner.
The goals are to prevent silent abandonment, preserve explicit failure ownership, and let ordinary affine programs retain their behavior. Non-goals are async destruction, general effect syntax, state-specialized methods, blanket stdlib migration, or stronger unwinding. Traps and process aborts remain outside the guarantee. It cannot establish fairness, deadlock freedom, successful I/O, knowledge of remote state, or recovery after a crash.
Baseline checked for this proposal
The reference and implementation were inspected at local commit 82544a4.
The source locations below are audit pointers, not additional specifications.
| Current behavior | Evidence and consequence |
|---|---|
| Classes, collections, captured closures, and aggregates with affine components move; ordinary scope exit may destroy them | Ownership reference above; src/ownership.rs. Moving already transfers ownership, but does not forbid abandonment. |
Cleanup covers return, ?, break, continue, replacement, and expression temporaries | Destruction reference above. __del__ cannot return a recoverable error or yield; suspended generator drop does not resume its body. |
? may erase an error into Failure.Unspecified, destroying the original | Error reference above; src/sum.rs. The unused-Result diagnostic does not cover unused bindings or wrappers and permits explicit drop. |
Owned with calls borrowed __exit__ -> (), then drops its manager | Resource scopes; src/frontend/contexts.rs. This is not yet a consuming resolver. |
| Worker checking follows reachable native effects, concrete callbacks, and destructors | Effects; src/effects.rs, src/threading.rs. It rejects uncertified indirect/foreign calls, including through stdlib helpers. |
| Dropping a future does not cancel or wait; its worker retains execution responsibility | Futures, executors; plenty-runtime/src/executors.rs, especially run_worker, stop, and destroy_job. Cancelled captures and unclaimed results can still be destroyed. |
| SQLite commit/rollback borrow the transaction; a busy commit permits retry | stdlib/std/db/sqlite.plenty, Transaction; typestate exploration. Existing fallback rollback is not a new linear contract. |
Proposed ownership rules
Introduce a discardability capability independent of ordinary copying.
Discardable[T] below is a schematic capability name, not chosen syntax.
Existing types remain discardable unless they contain an obligation or opt in
to nominal linearity. A linear value cannot be implicitly copied or duplicated
by copy; explicit duplication of a resource must create separately specified
obligations. A borrow is not an owner and does not acquire this obligation.
The checker tracks live obligations through its ownership control-flow graph. An owned argument transfers responsibility before the callee runs; the callee’s body must itself resolve or transfer it. Returning an owner is a valid transfer, including in an error. Passing it to an empty function is not an escape hatch: that function’s exit is rejected. Tail calls must resolve or transfer all other caller obligations before entering the callee.
| Site | Proposed rule |
|---|---|
End of block/function, explicit return, or ? failure | Every owner in the scopes left must be resolved or transferred. Check the error path separately from success, including pending evaluated operands. |
| Branch join | Check each outgoing path. An obligation cannot disappear on one path; represent conditional remaining ownership explicitly with a sum. Existing moved/possibly-moved availability still governs subsequent uses. |
Loop backedge / continue | Preserve the existing outer-owner availability rule. Resolve iteration-local obligations before leaving their scope; an outer obligation can remain in its slot across iterations, or be transferred and replaced with a new valid owner. |
break and zero iterations | Check the exited scopes and each resulting join. Do not infer that a collection was drained merely because its loop ended. |
| Ignored result, wildcard binding, expression temporary | Reject implicit destruction of an obligation, even when no named binding exists. Observing a temporary does not consume it. |
| Assignment or replacement | Plain assignment cannot destroy an old obligation. An exchange returning the old owner may transfer it; the returned owner must then be handled. Check failure while evaluating the replacement first. |
Explicit drop, aggregate destruction, fallback __del__ | Require discardability. Memory reclamation alone is not an authorized protocol finish. |
| Ordinary allocation/call failure | Responsibility stays with a live caller owner, returns in the error, or has already been resolved under an explicit contract. It cannot vanish during rollback of partially constructed values. |
| Fatal trap, abort, or nonreturning computation | No unwinding or liveness promise is added. A function which never returns has not completed an ordinary path. |
Borrow checking continues to forbid consumption while a conflicting loan is
live. A mutable borrower cannot silently remove the referent’s obligation;
ordinary replacement must leave a valid owner. Resolving a borrowed value into
a different state requires an explicit checked state/storage contract, not an
assumption that existing &mut methods can consume their receiver.
Construction and terminal authority
Nominal opt-in must carry an abstract obligation, not merely rely on having a
linear field. Otherwise unpacking an all-integer Transaction could silently
turn it into ordinary values. Conceptually it has a noncopyable seal whose
ownership follows transitions; the seal need not occupy runtime storage.
Seals and registry-authorized terminals are this proposal’s conservative design candidate, not prerequisites of every linear type system. An alternative could let each type’s defining module declare terminal operations as part of its trusted abstraction contract. That is more extensible but expands the authority to assert meaningful completion. The registry approach makes the initial trust boundary auditable at the cost of limiting new standalone resource protocols until a trusted-extension mechanism is designed.
Safe construction introduces that seal only through the type’s authorized constructor. Private fields/constructors prevent fabricated protocol states. Representation unpacking transfers every field and the seal; it does not resolve the seal. A safe transition passes it to its successor. An ordinary consumer, including one in the defining module, cannot discard it.
Terminal elimination needs a small, explicit trusted boundary. Initially the
compiler/runtime contract registry may authorize particular constructors and
terminal primitives, with an ownership/effect contract for each outcome.
Safe wrappers must call those primitives or transfer the seal; an empty wrapper
is rejected. Merely naming a function commit, marking it consuming, or adding
an empty destructor grants no terminal authority. There is no safe general
forget, user-declared empty eliminator, or FFI cast that removes an obligation.
This cannot prove the external truth of a primitive’s claim: the implementation of “response sent” or “local transaction released” belongs to the trusted resource contract. Incorrect trusted annotations can invalidate the guarantee, just as incorrect native ownership metadata can today. Review and test that boundary explicitly. A user-extensible trusted-declaration mechanism remains open; this proposal does not invent a general unsafe language or let arbitrary safe libraries mint terminal witnesses. Ordinary libraries can compose existing certified obligations without new authority.
Field extraction is a real dependency for ergonomic wrappers. Moving a field out of an owned binding is now implemented (issue #534; see moving fields), except for classes with a custom destructor and through references. All-fields deconstruction is not implemented; the smallest candidate remains consuming deconstruction within the declaring module, preserving any nominal seal and allowed only when no custom destructor must still observe the whole. General partial initialization, partial moves through borrows, and destructors on half-moved objects are outside this proposal. Whole-tuple moves and consuming enum matching can support early experiments without that extension.
Implementation and conformance outline
First derive discardability and obligation containment recursively in type
metadata, then attach outstanding-owner information to ownership checking and
every cleanup edge. Add capability checks at generic instantiations and native
operations before allowing linear values into existing containers or workers.
Validate return/error ownership and terminal annotations before lowering cleanup.
Only proven discardable storage may reach ordinary generated drop glue.
Audit temporary, constructor, callable, and allocation rollback paths as well as
visible drop calls; suppressing a destructor would merely leak an obligation.
This ordering describes semantic dependencies, not a second work queue; B72 remains the planning record. Runtime resource conversions need their own API decisions and are not included in this documentation change.
| Positive conformance case | Negative case / diagnostic requirement |
|---|---|
| Borrow a transaction repeatedly, then finish it | Exit with it live: identify the acquisition, remaining obligation, and exact exit. |
| Return the owner or transfer it to a checked helper | Empty helper: report its owned parameter and unresolved function exit. |
| Resolve on both branches, or return remaining ownership in a sum | Resolve only on one path: show the unresolved predecessor, not merely “unused value.” |
| Retain an outer owner through a retry loop; finish iteration-local owners | continue leaks a local or moves an outer owner without replacement. |
| Exchange an owner and resolve the returned old value | Overwrite, ignored temporary, or drop: identify the implicit/explicit discard. |
| Pass the same seal through a valid state transition | Reuse after move, fabricate a state, or discard a deconstructed seal. |
The companion pages add error, aggregate, scope, and worker cases. Native tests must inject allocation and submission failures and count transfers/terminal calls, including cancelled work, so every acquired seal is either returned or resolved once. Compile-fail tests must cover implicit cleanup as well as source operations. These are future conformance requirements, not tests run for this proposal.
Alternatives and open decisions
An expanded unused-value warning is cheaper but cannot enforce obligations across wrappers and calls. Affine RAII remains preferable when silent fallback cleanup is an adequate resource contract. Requiring all owners to be linear would impose unnecessary handling on ordinary containers and existing programs. Typestate alone and a combined effects redesign are assessed in the integration contract.
Proposed decisions are opt-in linearity, recursive non-discardability, checked transfer, and explicit terminal authority. Open design choices are declaration and capability syntax, how trusted contracts may be extended, the smallest consuming-deconstruction surface, and how much path-sensitive empty-container proof to expose. None may weaken error-path or aggregate preservation. Async completion and general effect polymorphism are explicitly deferred.
Linear values, aggregates, and errors
Status: Not implemented; companion to linear obligations. Issue #149, B72. This page is part of the core ownership proposal, not an independent feature. The current sums and ownership contracts remain authoritative. All examples are speculative pseudocode.
Wrapping preserves responsibility
Discardability is structural and conservative: a tuple, enum, class, box, collection, closure, or error that can own a linear value cannot be freely discarded. A nominally linear wrapper additionally retains its own seal. Wrapping, allocating, or erasing representation must not make an obligation disappear. Recursive types need a fixed-point containment calculation.
| Owner | Permitted handling / required restriction |
|---|---|
| Tuple | Move as a whole or unpack every component; ignored linear components are errors. |
Enum / Option / Result | Match by consumption and handle each payload. A payload-free arm can finish normally after the match proves that arm. |
| Class / box | Transfer whole owner, or use checked consuming deconstruction. A custom destructor is not permission to drop linear fields. |
| List / dictionary / set | Insert and remove by transfer with ownership-preserving failure. Clear, eviction, replacement, duplicate-key/value rejection, and container drop require discardable discarded items or an explicit resolver. |
| Capturing closure | Capturing by move transfers the obligation. A one-shot call must resolve/return it; destroying an uncalled closure is rejected. Borrowed captures keep existing lifetime rules. |
| Reusable closure | Invoking it does not consume its environment. A linear capture remains owed even after calls; an explicit final consuming operation would be needed. |
| Lazy generator / erased callable | Reject linear captures, retained locals, or erased obligations until their destruction and suspension contracts preserve them. The current generator drop does not run the rest of its body. |
Option[Transaction] is conservatively non-discardable even if a particular
value is Nothing. Consuming matching proves the payload-free branch locally;
it does not grant the entire type discardability. Similarly an empty linear
collection needs a checked consuming empty-finalization operation, rather than
a rule that drops arbitrary list[Transaction]. Returning an iterator’s
unconsumed remainder preserves ownership; early break from an iterator that
silently destroys the remainder must be rejected. Borrowed iteration is useful
without consuming anything, and consequently does not finish the collection.
Existing generic bodies that may discard a T need a Discardable[T]
requirement. Transfer-only identity, pairing, and error forwarding need none.
Instantiations with ordinary existing types retain today’s behavior. Initially
the existing specialization machinery can check the requirement at use and
point to the discarding body; explicit bound syntax and declaration-time generic
checking are separate ergonomic decisions. Containers themselves need not
require discardable elements: only operations which destroy elements do.
This affects builtin APIs, not just user-defined generic functions:
unwrap_or(default)discards the default when a present payload wins; the existing signature needs discardableT. Use an explicit match that handles both owners, or a new API returning the unselected alternative.ok_or(error)drops the error onSome, so requires discardableE; it may still transfer a linearTintoOk. Construction of a temporary option or error follows the same failure rules.is_ok()and similar observers do not discharge obligations. Observing a linear temporary and then destroying it is an error.map_errormust check the mapper’s handling of the owned error. Mapping a live-transaction error to an ordinary code cannot silently erase its payload.
Allocation and partial construction
A consuming allocating API cannot simply return AllocError after losing its
input. It must reserve capacity before transfer, preserve caller ownership via
an explicitly checked operation, return the owner in the failure payload, or
invoke an authorized resolver under a documented policy. An ordinary owned call
transfers its arguments: restoring a moved caller binding is not implicit.
For example, speculative insert_owned(list, item) may return either the
updated list or (list, item, AllocError). An insertion through a borrowed list
can return the uninserted item on failure. Dictionary insertion must return or
resolve a replaced value and any unused incoming key. Constructors must account
for earlier initialized fields when a later field expression propagates error.
Closure allocation failures must return owned captures or occur before transfer.
Hidden builders and argument temporaries are subject to the same rules.
For an existing native operation whose failure path releases an argument, reject linear instantiation until its contract is extended. “The allocator failed” is an ordinary error, not permission to abandon protocol state.
Transaction ownership on failure
Consider the intentionally rejected pseudocode:
tx = begin_transaction(connection)?
insert(&mut tx, record)? # error returns while tx is still owed
commit(tx)?
Handle insertion failure by explicitly rolling back, returning an error that
owns tx, or entering a checked resolution scope
before the fallible operation. Acquisition must also specify ownership on
failure: a consumed connection is returned or terminally closed; a borrow stays
with its owner. No hidden success-only ownership contract is sufficient.
Commit APIs have two different valid shapes:
commit_local(tx: Transaction) -> Result[Receipt, TerminalCommitError]
# Every result has resolved the local obligation; no retry owner survives.
commit_retryable(tx: Transaction) -> Result[Receipt, CommitError]
CommitError = Retry(Transaction, DbError) | Finished(TerminalCommitError)
# Retry retains the live transaction; Finished does not.
These are alternative contracts, not two names for the current SQLite method. The existing typestate assessment explains why SQLite’s busy commit must remain retryable. Its current mutable method already preserves the transaction; a consuming redesign must carry it in the retry result. The error enum above is itself non-discardable.
match commit_retryable(tx):
Ok(receipt): return Ok(receipt)
Err(Retry(tx, cause)):
return Err(NeedsDecision(tx, cause)) # caller now owns the obligation
Err(Finished(cause)):
return Err(TerminalFailure(cause))
A matching Result error type may propagate this obligation with ?, provided
other live locals are handled. Propagation to current Failure.Unspecified
must be rejected when it would destroy an obligation. A diagnostic should name
the retained transaction, the error conversion, and the original error type;
suggest preserving the typed error or resolving its owner first. Matching a
terminal arm can still permit ordinary error erasure afterward.
Rollback is not automatically infallible. A retryable rollback must likewise
return ownership; an authorized local-release operation can instead finish
with a reported failure. “Local handle released” and “remote rollback confirmed”
are different contracts. For a remote transaction with lost acknowledgements,
an UnknownOutcome may be a terminal local outcome if the API explicitly
permits it. If reconciliation is required, it must carry a live recovery
obligation. The type system cannot invent knowledge the protocol lacks.
Conformance probes
Accept typed retry errors passed through several helpers, complete enum matches,
owned removal with remainder returned, and allocation failure returning every
input. Reject nested wrappers dropped through a generic helper, a discarded
closure containing a transaction, wildcard patterns hiding obligations,
replacement of a linear dictionary value, and partial iteration losing its
remainder. Test both unwrap_or branches and ok_or’s unused error path.
Failure injection must cover every allocation point after the first transfer,
including partially built objects and closures. Diagnostics should trace the
containment path (for example SubmitError.job.capture.tx) and the cleanup
edge, rather than merely saying a large aggregate is noncopyable. Discardability
is the missing capability; copying and borrowing are separate concerns.
Synchronous resolution scopes
Status: Not implemented; separate proposal depending on linear obligations. Issue #149, B72. Current resource scopes and destruction do not provide the consuming-finalizer contract proposed here. All examples are pseudocode.
Why a separate extension
Manual transfer or resolution is a complete synchronous linearity rule, but a transaction around several fallible calls needs repetitive handling of every early exit. A manager can take responsibility, lend access to the body, and run an authorized resolver before the scope returns. This is cleanup policy on top of ownership; neither a new effect language nor async cleanup is necessary.
Python context managers illustrate the related control-flow guarantee:
following successful __enter__, ordinary body exits invoke __exit__. That
mechanism alone neither checks non-abandonment across calls/storage nor requires
every resource to enter a context. It does not statically stop escaped aliases,
promise cleanup success, or survive abrupt process termination. Plenty already
adds borrowing restrictions; linearity would additionally check the manager’s
own responsibility.
Proposed narrow contract
A resolution scope owns a hidden linear manager and registers a consuming
exit operation before lending body access. On each ordinary exit, body-local
owners finish first, borrowed access ends, then the manager is consumed exactly
once. The operation must resolve or transfer its payload on every returning
path. Its normal receiver is owned, unlike today’s borrowed __exit__.
with resolving(tx, release_local_transaction) as access:
insert(access, record)?
update(access, other_record)?
# every ordinary body exit consumes the manager and resolves tx
The illustrative resolver is explicitly a local-release policy that may attempt rollback but promises no remote success. There is no implicit commit-on-success policy. A library can offer a clearly named rollback/local-release guard; the language should not guess the business meaning of normal scope exit.
Acquisition and registration form one ownership contract: failure returns the unregistered owner, or occurs before transfer. Once registered, even a failed entry operation must return the owner or run the resolver. Do not copy today’s successful-entry rule in a way that leaks an already-owned obligation. Nested scopes finish in reverse order. If a later manager acquisition fails, earlier registered managers still resolve, and the failed acquisition’s returned owner must itself be handled.
The body receives a loan bounded by the manager. It cannot move the manager, replace its unresolved payload, retain the loan past exit, or return references to owned manager storage. Return values may be prepared before cleanup only if they obey these lifetime rules. A body-local linear owner is not automatically the manager’s responsibility unless explicitly transferred into it.
An explicit consuming finish inside the body needs a checked manager-state
transition that proves the exit operation has nothing left to resolve. The
first design may reject this and require manual ownership for commit/retry
flows. Assuming that a boolean finished flag makes a linear field disappear
would undermine the core rule. Similarly, borrowed with &mut existing cannot
consume the caller’s owner; adapting it requires a separate state contract.
Worked example: cancel_on_exit()
token.cancel_on_exit(), implemented for issue #355, is an instance of the
narrow unit-exit form specialized to one resolver. Its manager owns a share of
a CancellationToken; its exit operation is cancel(), which is synchronous,
infallible, returns unit and allocates nothing. Every ordinary exit, including
?, return, break, continue and a failed later manager in the same
header, runs it exactly once, in reverse manager order:
with spawn(serve, listener, stop.share())?, stop.cancel_on_exit():
prepare()?
# every exit cancels stop, then joins the task
The compiler check that pairs with it is the
cancellation rule:
when a function later cancels a token, a join or pool shutdown that waits for
work holding it must come after a cancel on every path. The guard satisfies
that rule by construction. It does not need linearity: the obligation is local
to one function and its exits, and cancel() cannot fail, so there is no
unresolved owner to return. It shows the shape a library resolver could take
once consuming exits exist, and why the infallible case is the first one to
make automatic.
Failure determines the available surface
The narrow automatic-exit form requires a synchronous resolver that returns unit after reaching an allowed terminal local outcome. It may block, and it may internally retry. If the only correct resolver returns a still-live owner on failure, this form is unavailable. Swallowing that failure is not resolution.
Today’s unit-returning __exit__ is sufficient to express an infallible
side effect, but not sufficient to prove that a borrowed manager’s linear
payload was consumed. A dedicated consuming-finalizer lowering (or a later
verified state-changing exit contract) is necessary. The hidden owner must not
subsequently reach ordinary drop with a live seal. This extension does not
change all existing managers or reinterpret existing __exit__ hooks.
Fallible finishing should initially remain explicit. A future combinator can
make error precedence and retained ownership visible instead of silently
overriding a pending return or propagating an exit error through today’s with:
resolve_scope(owner, body, resolver) -> ScopeOutcome[BodyValue, BodyError, R]
ScopeOutcome = Resolved(Result[BodyValue, BodyError], ResolutionReport)
| Unresolved(Result[BodyValue, BodyError], R, ResolveError)
Unresolved retains the owner R; the whole outcome is non-discardable. Both
body failure and resolution failure survive. If the body value/error is itself
linear, it remains owed in either branch. If allocation is needed to store an
outcome, its failure contract must retain these values too. The body borrows
the owner and returns a result to the combinator; it cannot return a borrowed
view of that owner. This sketch does not introduce nonlocal callback returns.
This explicit outcome is the recommended direction for recoverable resolution
failure, but callback syntax and a library implementation are open. A future
fallible with syntax would need equally explicit rules for simultaneous errors,
prepared return values, nested manager failures, and unresolved owners. It is
not a prerequisite for core linearity or the narrow unit-exit form.
Destructors, effects, and execution limits
An ordinary __del__ remains fallback memory/resource cleanup. It cannot be
used to launder a linear field into discardable storage. After terminal
resolution, remaining ordinary fields may be destroyed in their existing order.
This avoids requiring recoverable errors, suspension, or a stronger destructor
model merely to express meaningful finishing.
Resolver calls, including compiler-inserted ones, belong in the existing reachable-effect graph. A resolver forbidden on a worker makes that scope ineligible there; see the integration example. Opaque higher-order resolver effects remain subject to current certification limits. There is no generic “cleanup is always allowed” exception.
No bounded completion is promised: scope exit can wait indefinitely. An
immediate timeout return and completed cleanup cannot both be unconditional.
Transferring responsibility to a longer-lived scope is possible only if it owns
all needed resources and retains no loans whose owners have already returned.
Async cleanup expressibility and the runtime obligation to drive it need a
distinct proposal; neither Python async with nor a hypothetical async
destructor alone supplies that obligation.
Conformance and unresolved choices
Test normal return, ?, break, continue, nested acquisition failure, and explicit
body failure with a counted resolver: exactly one permitted resolution for each
registered owner, body loans ended first, and unchanged cleanup order for
ordinary values. Reject escaping loans, destroying an unentered manager,
calling an empty resolver, consuming through a borrowed manager, and a resolver
returning while still owning its payload. Verify worker diagnostics include
inserted exit calls and their origin scopes.
The consuming-exit declaration surface, safe explicit finish within a scope, and the fallible outcome combinator remain open decisions. The proposed defaults are explicit policy, no implicit commit, no ignored recoverable resolution error, and no async requirement.
Linearity, typestate, and execution responsibility
Status: Not implemented; integration assessment for issue #149. Planning remains in B72. This page connects linear obligations, resolution scopes, and the existing typestate exploration; it does not approve an effects or concurrency redesign. Examples below are speculative pseudocode.
Recommendation and boundaries
Use small proposals with explicit interfaces: a linear ownership core (including its inseparable aggregate/error rules), a separate synchronous resolution-scope extension, and this integration contract. Keep the existing typestate exploration independent. No new effect syntax is required for concrete synchronous consumers. Propose effect-polymorphic resolution or suspension only when a concrete use cannot fit current certification or execution ownership.
| Organization considered | Assessment |
|---|---|
| One linear-types proposal, existing typestate exploration, narrow effects section | Viable for manual synchronous resolution. Automatic scope policy and future execution ownership would make the document carry separable decisions. |
| Separate linearity/typestate/effect extensions with a small common contract | Preferred where there is an actual extension. Keep typestate’s current assessment; split scope policy now. Do not manufacture an effects extension when reachable-call checking suffices. |
| Combined linearity, typestate, effects, and concurrency proposal | Shared examples help, but there is no demonstrated semantic need for one adoption unit. It would force syntax/runtime decisions that synchronous linearity does not depend on. |
The pages form independent review units where possible, not claims of complete implementation independence. Aggregate cleanup and failure ownership cannot be deferred from the linear core: a single discarded wrapper defeats the guarantee. Scope resolution depends on core preservation and terminal authority. Worker support depends on auditing runtime discard paths. Async completion depends on a responsible execution owner and suspension rules; it is not a core prerequisite.
Responsibility and dependency matrix
| Responsibility | Implemented baseline | Proposed owner / dependency | Deferred |
|---|---|---|---|
| Ownership and permission to discard | Affine moves, borrows, destruction | Linear core adds non-discardability, seal transfer, and checks at every cleanup edge | General user-defined trusted terminal declarations |
| Legal state transitions | Nominal classes, consuming functions/methods, runtime enums | Existing typestate exploration; linear states additionally preserve seals | State-specialized generics |
| Permitted effects / worker eligibility | Reachable native-effect walk, concrete callable/destructor checks | Reuse it for terminal calls and inserted resolvers; no stdlib exemption | General indirect-effect certification / effect polymorphism |
| Cleanup policy | Deterministic fallback drop, unit borrowed __exit__ | Separate consuming scope resolver; policy explicitly named | Fallible exit syntax and async destructors |
| Execution responsibility / suspension | Native worker retention, joins, cooperative cancellation | Initially restrict discard-capable APIs; new completion contract needs runtime ownership audit | Suspended computation ownership and driving async cleanup |
| Failure ownership | Result payload transfer, lossy Failure erasure, owned submission errors | Linear core preserves live owners in errors and allocation rollback; each API defines terminal versus retry outcomes | Automatic recovery policy and remote-state certainty |
Counterexamples that test the split
A synchronous transaction with commit(tx) and rollback(tx) can carry a
linear obligation using one nominal type and consuming free functions. It needs
neither state-specialized methods nor effect annotations. Conversely,
prepare(Draft) -> Ready is useful affine typestate even if discarding a draft
or prepared value is harmless. Linearizing every state would add obligations
without a corresponding product requirement.
Effects determine where an operation may execute, not whether it eventually
executes. A worker-certified rollback may block forever; a locally permitted
main-thread-only resolver can be meaningful even though worker certification
rejects it. A function marked consuming is still not a terminal operation unless
its ownership contract accounts for the seal and all failure paths.
Linear typestate with explicit errors
Consider a protocol whose contract requires a response or completed local cancellation after acceptance:
send(Request) -> Result[InFlight, SendError]
SendError = NotAccepted(Request, Cause) | Uncertain(InFlight, Cause)
receive(InFlight) -> Result[Response, ReceiveError]
ReceiveError = Retry(InFlight, Cause) | Finished(TerminalFailure)
request_cancel(InFlight) -> Result[Cancelling, CancelStartError]
CancelStartError = Retry(InFlight, Cause) | Finished(TerminalFailure)
finish_cancel(Cancelling) -> Result[Cancelled, CancelFinishError]
CancelFinishError = Retry(Cancelling, Cause) | Finished(TerminalFailure)
Here Request is affine before acceptance; InFlight and Cancelling are
linear and carry the same obligation through transitions. Response,
Cancelled, and TerminalFailure are ordinary values because this example’s
contract considers their outcomes terminal. Another protocol may require a
linear response body or recovery token instead. Uncertain acceptance cannot be
reported as definitely unstarted when work may already exist remotely.
Typestate rejects calling finish_cancel on InFlight or receiving after the
handle becomes Cancelling. Linearity rejects simply returning after a cancel
request or discarding Retry(Cancelling, cause). Either mechanism alone misses
the other’s error: an affine state may be abandoned; one linear runtime-state
handle may still reject operation order only at runtime. Together they prevent
abandonment and statically restrict these transitions, without proving that
finish_cancel ever returns or that a remote peer received the cancellation.
Constructors must not let clients forge accepted or cancelled states. Moves prevent reuse of old handles, and terminal authority defines allowed outcomes. Use consuming free functions in experiments; current ordinary methods borrow receivers. For queues and runtime routing, an enum owning either linear state may be simpler than carrying state parameters through every helper. Input-driven parsers generally still suit runtime enums. See the existing typestate page for HTTP response lifecycles, field-extraction friction, and SQLite retry behavior.
Worker effects and hidden cleanup
Suppose an explicitly chosen resolver restores a process working directory:
restore(scope: DirectoryObligation) -> ():
while True:
match std.env.set_current_dir(scope.saved_path):
Ok(()):
authorized_restored(scope)
return
Err(cause):
record_error(cause) # retain scope and retry
worker(owner: DirectoryObligation):
with resolving(owner, restore):
compute()
This worker must be rejected: its required exit reaches a process-global effect.
The example deliberately retries restoration failures and may never return;
it does not silently treat failed restoration as success.
The current effect checker
already follows ordinary calls and destructors; generated consuming resolver
calls must be visible in that same graph. A diagnostic should identify the
submission/spawn, captured obligation, resolution scope, and call chain to
set_current_dir, with the existing reason that it affects every thread.
Report an ownership error as well if a proposed alternative merely drops the
owner to avoid the forbidden call.
Do not classify all linear owners as main-thread-only. In principle a worker can compute while borrowing the data and leave resolution with a parent, or return the obligation through an ownership-preserving join protocol. Reject at worker certification when the reachable body/required cleanup needs the effect, and at transfer when the chosen worker API cannot preserve ownership on all paths. For the initial core, existing APIs that can discard such owners are ineligible regardless of whether the particular body appears safe.
Concrete resolvers with known bodies need no new effect annotations. A generic resolver specialized to a concrete callable can use existing certification only where the resulting calls are inspectable. Current indirect and boxed callable restrictions still apply; an erased resolver or unresolved higher-order call cannot claim worker safety just because its type says it consumes an owner. Such use needs either a statically inspectable adapter or a separately justified effect-contract extension. The limitation must appear at the requiring worker use and the hidden resolver call, not as an exemption for library cleanup.
Native workers and future completion
Current Future is affine. Dropping it neither cancels the job nor waits: accepted running work retains a worker owner and proceeds. Executor shutdown joins running workers, and ordinary shutdown drains accepted queued jobs. Cancellation of running work is cooperative; setting a token does not force termination or change a future’s result by itself.
These facts are useful execution machinery but do not make every stored value
linear-safe. Preserve today’s APIs for discardable data. The first linear core
should reject linear captures/results in existing thread, executor, channel,
and lazy-frame paths that can destroy them, until a path-complete contract is
implemented. This is a proposed restriction for new types, not a claim that
today’s compiler knows Discardable.
| Existing path | Obligation-preserving extension needed before admitting linear values |
|---|---|
| Thread-start failure destroys moved inputs | Return complete inputs or invoke an authorized failure resolver. |
| Executor submission fails | SubmitError[F] already returns the complete job; preserve its captures, reject lossy erasure/drop, and audit allocation before transfer. |
Queued job cancelled / shutdown(True) | Current cancellation skips execution and later destroys captures. Return them to a responsible owner or run a certified resolver; marking the future done is not resolution. |
| Job runs normally | Its body and reusable-environment cleanup must resolve/transfer every capture, on success and failure. |
| Public future dropped / result unclaimed | Current runtime may destroy the result. Require a new non-discardable result handle or an explicit runtime-owned result resolver. |
| Scope exits / executor shuts down | Joining proves workers returned, not that a returned obligation was handled. Retained result storage needs a continuing owner. |
| Channel disconnect, queued-message cleanup | Return or resolve linear messages; destroying a queue is not enough. |
A possible Completion[T] would specify those responsibilities explicitly;
it must not silently redefine Future[T]. A joinable synchronous implementation
could precede async/await. Submission failures preserve the job; cancellation
must return it or complete its resolution; terminal results transfer to the
consumer or an authorized resolver. All hidden resolvers require effect checks
on the thread where they actually run, and must run without scheduler locks.
In future async work, moving a completion into .await merely transfers the
obligation to the suspended frame. Dropping that frame cannot abandon it. A
responsible scope/runtime must continue driving the computation or its
cancellation to an allowed terminal outcome. async with or an async destructor
provides a place to write suspending cleanup; neither ensures someone keeps
executing it. The separate async/await exploration now
examines suspension and execution ownership, but its affine spike does not
support linear values or establish non-abandonment. Such syntax still needs a
complete obligation-preserving runtime contract before claiming that guarantee.
A timeout can return immediately only by retaining responsibility elsewhere or returning the live owner. The responsible scope must retain all required owned resources and outlive any retained loans; a background cleanup cannot use a borrow into a caller that already returned. Completion can still wait forever, deadlock, or fail remotely. No fairness, termination, crash recovery, or bounded cancellation guarantee follows from linear ownership.
Integration checks before adoption
Demonstrate a synchronous linear transaction without new typestate/effect syntax, and an independently useful affine typestate API. Test the protocol above for illegal order, dropped retry states, and legal ownership-preserving retries. Worker compile-fail cases must include direct consumers, helper chains, inserted exit calls, and uncertified indirect resolvers. Positive worker tests must use permitted effects and an audited transfer protocol.
For every new worker protocol, inject start/submission failure, cancel queued work, drop the result consumer, exercise shutdown with pending/running work, and verify captures/results are resolved or returned exactly once. Include timeouts that return a live owner and reject a scope escaping required loans. Do not use a successful join or a cancellation flag as evidence of protocol completion. These checks bound any later implementation; this proposal adds no runtime or language behavior.
Protocol method bodies: defaults, final methods, and private hooks
Status: core design agreed; not implemented. This is the implemented
protocol methods proposal: “implemented” means a method with a body, not a
feature that has shipped. The compiler still requires protocol methods to have
pass bodies; current protocols
remain authoritative. Future prioritization belongs to B29.
Start with the agreed rules
and short programs. The feasibility
comparison below and implementation evidence
record the investigation against d39e6ec on 2026-10-20. Alternatives such as
automatic method injection were considered, not adopted.
Assessment: shared bodies with explicitly selected protocol calls are a small-to-medium frontend feature. Convenient calls through a protocol bound are a medium feature. Automatically adding methods to every structurally matching object, especially with private hooks, is a substantially larger language-design change. None inherently requires dynamic dispatch, an object wrapper, a new runtime representation, or a different borrow checker.
Those estimates concern sharing and lookup alone. The subsequently agreed definition-time contract checking adds substantial frontend work; the complete design with that checking and private-hook grants is not a small syntactic change.
This investigates sharing implementations. The earlier library-owned operations assessment addresses preventing callers from bypassing an algorithm; it remains relevant, because ordinary overridable defaults do not provide that guarantee.
The follow-up final-method and visibility design records the agreed core semantics: defaults honor overrides consistently, final methods reject same-named inherent methods at the structural boundary, and private hooks use member-specific grants while preserving module privacy. Three short programs illustrate the rules. The final-name check is a modest addition once provided-method selection exists; hidden external hooks require additional access-control machinery. Some specification work remains, and no compiler implementation is approved here.
Protocol bodies must also be checked against their declared contract at the definition, rather than accepted because a concrete class has extra members. This agreed requirement increases the scope beyond a generic-function lowering: the existing per-instance checker is not sufficient for a complete implementation.
One example, three distinct choices
Hypothetical syntax, not executable Plenty:
protocol DocumentStorage:
def write(self: &mut DocumentStorage, document: &str) -> Result[(), IoError]:
pass
def save(self: &mut DocumentStorage, document: &str) -> Result[(), IoError]:
validate(document)?
self.write(document)?
record_success()
Ok(())
FileStorage and DatabaseStorage would supply write. The application wants
to call the shared save. Treating an AST body consisting solely of pass as
a requirement is straightforward; an intentional no-op implementation would
need an explicit unit-returning body. Comments do not change the AST body.
There are three independent questions:
- Sharing: where may the application call the supplied
save? - Replacement: may a concrete type supply its own
saveinstead? - Access: may the application call
writedirectly?
Implementing the first does not imply either answer to the other two. In particular, the example above deliberately uses ordinary accessible hooks. Cross-module private hooks need a separate access design.
Options and relative cost
All new spellings below are illustrative.
| Design | Example call | Scope and assessment |
|---|---|---|
| Existing bounded function | save(&mut files, document) | Works today. Sharing has no language implementation cost. |
| Protocol-owned operation | DocumentStorage.save(&mut files, document) | Small-to-medium. Explicitly selects the protocol body, lowers to a generic function, and adds no inherent method. |
| Default selected through a bound | storage.save(document) inside S: DocumentStorage code | Medium. Needs protocol-aware method resolution, missing-default handling, override rules, and receiver provenance through specialization. |
| Explicit adoption on a type | An illustrative use DocumentStorage declaration in FileStorage | Medium-to-large. A local place to opt into methods and diagnose conflicts, but introduces a new relationship beyond structural matching. |
| Automatic structural extension | files.save(document) wherever a matching protocol is visible | Large relative to the above. Needs candidate discovery, overlap rules, inference limits, and a decision about whether imports change available methods. |
| Final provided method | Reject a matching object with a same-named inherent method | Modest addition to provided-method selection: check the final-name restriction at conformance and consistently select the protocol body. |
| Private hooks across modules | Access controlled separately from structural matching | Separate substantial feature. Needs explicit authority, cross-module rules, and guarantees that survive every call path. |
These are engineering judgments from source inspection and experiments, not calendar estimates or measurements of a completed implementation. A narrow compiler spike would be small; a supported feature includes diagnostics, modules, generic combinations, tooling, and regression coverage.
The protocol-owned operation is closest to ordinary generic-function sugar:
# Conceptual lowering of DocumentStorage.save:
def __document_storage_save[S: DocumentStorage](storage: &mut S,
document: &str) -> Result[(), IoError]:
validate(document)?
storage.write(document)?
record_success()
Ok(())
Here DocumentStorage checks only its required hooks; implemented operations
do not require same-named inherent methods. The generated name is an internal
detail. Calls must preserve the protocol definition’s module visibility.
This design has deliberately fixed selection: DocumentStorage.save(...)
calls that body even when the object has an inherent save. It is a provided
operation, not a fallback override. It avoids global method searching and
preserves the existing rule that importing a protocol does not change an
object’s inherent methods.
It does not, by itself, give the application the convenient files.save(...)
spelling. A bounded-call design could add that later, but must deliberately
choose whether it denotes the fixed operation or an overridable default.
The important design traps
Defaults are not a validation boundary. If a concrete save replaces the
default, it can omit validation. Explicitly selecting a fixed protocol body
protects that call’s algorithm; it does not prohibit calling a public write
or a separate concrete save. An owning wrapper remains the current-language
way to expose only the intended operation.
Structural compatibility is not implementation selection. Two protocols
may both require write but provide different save algorithms. A concrete
object can match both without selecting either. A qualified call is unambiguous.
For bound-based calls, the bound should identify the protocol; searching every
matching protocol makes imports and unrelated declarations affect lookup.
Inherent overrides need exact rules. If defaults are replaceable, decide
whether a same-named but incompatible method is an error or is ignored. Prefer
an error over silently choosing the default. Also specify calls between defaults:
does save_twice invoke an override of save, or its original protocol body?
Do not let these answers depend accidentally on specialization order.
Bodies must be checked against the contract. Existing generic bodies are
checked per concrete instance and may use methods not listed in their bound.
That behavior is not acceptable for provided protocol bodies in the agreed
design. Abstract checking is required, including through aliases, helpers and
closures; a syntactic scan for self.foo is insufficient. This is additional
frontend work beyond the demonstrated generic-function lowering.
Inference needs real evidence. Required concrete signatures can determine protocol parameters. A supplied body cannot become circular evidence for its own signature. If a protocol parameter occurs only in a provided operation, require explicit arguments or other independent evidence. Avoid searching other protocol defaults to manufacture conformance.
Privacy is lexical today. A library-defined shared body cannot access an external class’s private method. Automatically granting that access because a public protocol happens to match would cross an existing encapsulation boundary without the class author’s consent. If pursued, access grants should identify the protocol/member explicitly and be distinct from declaring conformance. “Private to the class’s module” also differs from “callable only by protocol bodies,” which the original request might require.
Precedents
Python’s structural protocols provide a particularly relevant warning: implicit structural matching does not supply default implementations; explicit subclassing does. Its existing inheritance mechanism selects the shared code. Python typing specification.
Rust permits trait bodies as defaults that implementations may replace. Individual trait items do not have independent visibility qualifiers. Rust Reference.
Swift protocol extensions supply defaults; a conforming type’s implementation of a requirement takes precedence. Constrained extension overlap also requires selection rules. This is useful precedent for both the feature and its semantic cost, not a lookup model Plenty can copy unchanged. Swift language guide source.
Suggested direction
The selected direction uses overridable defaults and @final bodies, selected
through a bound or protocol qualification. Both forms honor overrides for
non-final methods, including calls inside other protocol methods. Objects do
not acquire inherent methods automatically. Matching requires signature and
visibility equality, and cross-module private hooks require a
@protocol.Protocol annotation on the hook method; module privacy otherwise
stays unchanged. Final methods are annotated @final.
The earlier qualified-call alternative that always invokes the original body is not selected. The design page separates agreed rules from the remaining specification work. Definition-time checking against the declared protocol contract is mandatory.
The implementation map and probes show what already works and where the remaining compiler work lies. No new feature is scheduled by this sketch.
Final methods and hidden hooks
Status: core design agreed; not implemented. This refines the
feasibility sketch using @final and
@protocol.Name method annotations. Short programs explore the resulting
programming model. Future prioritization remains in B29.
Separate implementation policy from call visibility
The agreed design has three implementation policies:
| Declaration | Requirement on a matching class |
|---|---|
def operation(...): pass | Supply an inherent method matching the signature and visibility. |
def operation(...): <body> | Supply an override matching the signature and visibility, or use the protocol body. |
@final above def operation(...): <body> | Do not declare an inherent method of that name; use the protocol body. |
Visibility is independent: pub permits application calls through the bound;
a non-public hook is available to authorized protocol code. A final method
requires a body; @final on a pass body is contradictory and is rejected.
An explicit unit-returning body can represent an implemented no-op.
@final is written on the line above the method, like Python’s
typing.final. final communicates that an implementation cannot be
replaced; fixed would emphasize ownership of the algorithm, but resolves no
additional semantic issue. Method annotations defines
the @ syntax.
Enforce final at the structural boundary
Matching S: DocumentStorage requires:
- Every required method exists with exactly matching signature and visibility, and any necessary access grant.
- Every supplied override matches the protocol signature and visibility exactly.
- No inherent method has the name of a final protocol method.
Check the entire contract even if the generic function never calls save.
Report the failed bound at its use, with notes identifying both the final
declaration and the conflicting concrete declaration.
The class remains valid on its own. It simply does not satisfy this protocol.
No global scan of all classes and protocols is needed. Explicit
DocumentStorage.save(&mut value, ...) enforces the same complete
contract rather than offer a way around conformance.
Name-only conflict detection follows the agreed rule: a same-named inherent method fails the match, including an incompatible signature or private method. There is no visibility-based exception to the final restriction. An internal class edit can therefore break conformance. Field/method name collisions still need an explicit rule before implementation.
Exact signature matching uses Plenty’s existing structural signature rules, including receiver borrowing, type parameters and return type; parameter names need not be identical. Visibility is an additional equality requirement.
Once protocol-aware selection exists, this final-name check is a relatively small addition to conformance checking. It does not require the broader access grant machinery below. It also does not resolve how calls find protocol methods.
A coherent initial lookup rule
Select methods through an explicit bound and through protocol qualification, without automatically adding methods to concrete objects:
def persist[S: DocumentStorage](storage: &mut S, text: &str):
storage.save(text) # Final protocol implementation.
DocumentStorage.save(&mut files, text) # Same contract and selection.
files.save(text) # No injected inherent method.
For an ordinary default, calls through the bound choose the compatible inherent override if present. For a final method they always choose the protocol body. Calls from a final body to a replaceable method follow the same rule. A qualified call selects the same operation, including its override, to avoid two meanings of the public API. Explicitly invoking the fallback body would need a separate feature; it is outside the agreed design.
The earlier alternative where qualification always selects the original
protocol body was not selected. Fixed selection is expressed with final;
ordinary implemented methods provide overridable defaults.
A class author supplies hooks and optional overrides. Protocol code operates on that existing object, subject to normal ownership and borrowing. It adds no fields, ownership, allocation, or runtime wrapper. However, methods available through a bound can differ from those available on the concrete object. This needs teaching and completion support; the examples make the difference visible.
What non-public means
The desired interface is:
pub protocol DocumentStorage:
def write(self: &mut DocumentStorage, document: &str) -> Result[(), IoError]:
pass
@final
pub def save(self: &mut DocumentStorage, document: &str) -> Result[(), IoError]:
validate(document)?
self.write(document)?
record_success()
Ok(())
write is the hidden customization hook, and save is the public entry point.
Three permissions must be distinguished:
- Calling
writethrough aDocumentStoragebound. - Calling the concrete object’s inherent
write. - Letting a protocol defined elsewhere call that concrete private method.
Visibility matching requires the concrete write to be non-public too; a
public implementation fails conformance. Existing module privacy remains:
code in the class’s defining module can still call its private methods.
Class-only privacy and literal “only protocol bodies may call this method”
were considered and not selected.
For the third, structural matching alone should not authorize access to
unrelated private members. Otherwise application code could declare its own
protocol with a private write requirement and a public forwarding method,
then use matching to obtain the same access. This would defeat existing
module privacy.
The agreed class-side declaration annotates the hook itself:
pub class FileStorage:
@protocol.DocumentStorage
def write(...):
...
This grants methods defined inside the named protocol access to this specific hook. It grants no access to other private methods or fields. A hook used by two protocols carries two annotations, one per line, in any order.
The compiler checks the annotation where the class is defined. The named
protocol must declare a non-public requirement with the method’s name, and the
method must match that requirement’s signature and visibility exactly;
otherwise the annotation is an error at the method. This reports a mismatched
hook at its declaration instead of at a later bound. The annotation grants
access only: it does not declare conformance, and classes without private hooks
still match structurally with no annotation. Any method defined
inside DocumentStorage, including a later addition, may call write.
Unrelated protocols with identical signatures receive no permission.
The grant is lexical, not inherited from the caller. A standalone helper in the protocol’s module does not receive it merely because a protocol method calls the helper. Keep a hook-calling helper as a private implemented protocol method when it needs the grant. Ordinary functions in the concrete class’s module still have their existing module access. If the class and protocol share a module, that ordinary access remains available independently of the grant.
Broad grants to any protocol, module-wide switches, and a grant to all of one
protocol’s requirements were considered and not selected. Member-specific
permission makes expansion deliberate. An earlier class-level
allow DocumentStorage.write declaration was replaced by the method annotation,
which keeps the grant beside the hook and needs no repeated method name.
The grant is the one place a class must import a protocol: its module imports the protocol’s module. Import cycles are compile errors that name the dependency chain, and imports run no code, so Plenty has no partially initialized modules. A cycle would need the protocol’s module to import the class’s module. Protocol bodies are checked against the protocol’s own contract, so they never name a concrete class. Keep the contract in a module independent of concrete backends; anything that names a backend, such as a default factory, belongs in a third module. Public hooks still need no import.
No access scheme prevents an implementation from exposing a separate operation or performing its own I/O; the guarantee concerns the declared interface.
Method annotations
@ introduces a compiler annotation on the line above a def. The set is
closed: @final on protocol methods and @protocol.Name on class methods,
where Name resolves as a protocol in the class’s module (for example
@protocol.DocumentStorage or @protocol.library.DocumentStorage). Any other
annotation is an error. Annotations change no signature, body or call; they
state facts the compiler checks, so their order does not matter.
final and protocol are reserved words, so annotation names cannot collide
with user declarations. User-defined decorators are deferred and may never be
added: generic wrappers need variadic generics (B36), Plenty runs no code at
import, and a wrapper closure could allocate and change the signature. A later
decorator design must keep the wrapped signature unchanged and must not alter
the meaning of these annotations.
The annotation mechanism now exists, with one member: @copy on class
declarations (copy classes,
issue #535). The parser reads each @name alone on its line against the closed
set and rejects unknown names, arguments, repeats, and annotations where no
declaration accepts them. Names after @ are matched apart from identifiers, so
@copy reserves nothing and the builtin copy(value) keeps its name; @final
and @protocol.Name likewise do not need their names reserved. Adding them
means adding members to that set and accepting annotations before def in
protocol and class bodies, which the parser rejects today.
Check provided bodies against the contract
Agreed requirement: check protocol bodies against their declared contract,
without depending on extra members of any particular implementing class.
Every method or operation used on the implementing type must be justified by
that contract and the ordinary operations available for that type parameter.
A call to an undeclared self.bonus() is a definition-time error even if every
currently used class happens to have a compatible bonus.
This applies to default and final bodies alike, including aliases, nested closures, and calls through helpers. Helpers must have sufficient declared constraints for the call to be checked; unconstrained per-instance checking must not become an escape from the protocol contract. Member access grants authorize access to declared hooks; they do not add undeclared members.
Report contract violations at the protocol definition. Concrete specialization
can still perform lowering and concrete checks, but cannot make an invalid
protocol body valid by discovering additional methods or fields on the object.
This requires abstract contract checking beyond today’s per-instance generic
body checking. A syntactic scan for self.method is insufficient.
Whether to extend the same discipline to all ordinary generic functions is outside this decision. The implementation must nevertheless enforce it across helper calls originating in a protocol body.
Remaining specification work
The core behavior above is settled. This is not yet a complete implementation specification or approval to implement the feature.
Compatibility: today’s protocol requirements use bare def and remain
callable from bounded generic functions. Reinterpreting every bare def as a
hidden hook requires a deliberate stdlib, example, and tutorial migration.
Adding a final member can also invalidate existing structural matches.
Abstract checking machinery: specify how required, default and final methods, method-local bounds, helper contracts, and definition-site access are represented during checking before a concrete implementing type is known. Contract checking is required, not an optional later enhancement.
Inference and overlap: identify the protocol from the bound, not from the concrete receiver alone. Two bounds on different parameters can refer to the same concrete type and select different provided bodies. Combined bounds remain outside this design; qualification selects a protocol without global discovery. Rules for underdetermined type arguments and receiver aliases need precise specification and regression cases.
Access and collision edge cases: specify access in closures lexically nested inside protocol methods, and fields sharing a provided method name. These should be resolved explicitly rather than left to accidental behavior of generated functions.
State and validity: final freezes an algorithm, not its results. Hook contracts still define partial failure, side effects, and reentrancy. Persistent accounting/state must live in the object or a wrapper; a protocol supplies no new storage.
The @final and @protocol.Name spellings are agreed. Diagnostic wording can
be decided without reopening dispatch.
Short programs with final protocol methods
Hypothetical Plenty. These complete sketches illustrate the agreed core rules; the current compiler does not accept them. The research harness runs handwritten current-language equivalents of their algorithms, not the new dispatch or conformance rules.
The examples keep hooks public to isolate implementation selection. The private-hook variation at the end adds the separate cross-module access question.
Saving a document: required hook, final operation
The protocol author supplies validation once:
enum SaveError:
Empty
protocol DocumentStorage:
pub def write(self: &mut DocumentStorage, document: &str) -> Result[(), SaveError]:
pass
@final
pub def save(self: &mut DocumentStorage, document: &str) -> Result[(), SaveError]:
if document.byte_len() == 0:
return Err(SaveError.Empty)
self.write(document)?
Ok(())
class Memory:
writes: i64
pub def write(self: &mut Memory, document: &str) -> Result[(), SaveError]:
self.writes += 1
Ok(())
def persist[S: DocumentStorage](storage: &mut S, document: &str) -> Result[(), SaveError]:
storage.save(document)
def main() -> Result[(), IoError]:
mut memory = Memory(0)
match persist(&mut memory, ""):
case Ok(_):
print("unexpected")?
case Err(_):
print("empty rejected")?
match persist(&mut memory, "hello"):
case Ok(_):
print("saved")?
case Err(_):
print("unexpected")?
print(memory.writes)?
Ok(())
Expected output: empty rejected, saved, 1. The class author writes
only storage behavior. The application uses persist, which explicitly names
the contract and can call its final save.
If Memory adds its own save, the call to persist is rejected, even if
that inherent method validates correctly. It is the attempted structural match,
not the class declaration, that violates this candidate’s contract.
DocumentStorage.save(&mut memory, "hello") would be the qualified spelling.
memory.save("hello") does not gain a new inherent method. Calling
memory.write("hello") remains possible in this public-hook example.
Scoring: final operation calls a replaceable default
protocol Scored:
pub def base_score(self) -> i64:
pass
pub def score(self) -> i64:
self.base_score()
@final
pub def eligible(self) -> bool:
self.score() >= 10
class Basic:
points: i64
pub def base_score(self) -> i64:
self.points
class Bonus:
points: i64
pub def base_score(self) -> i64:
self.points
pub def score(self) -> i64:
self.points + 5
def qualifies[S: Scored](item: &S) -> bool:
item.eligible()
def main() -> Result[(), IoError]:
basic = Basic(7)
bonus = Bonus(7)
print(qualifies(&basic))?
print(qualifies(&bonus))?
Ok(())
Expected output: False, True. The final threshold algorithm calls the
selected score: the protocol fallback for Basic, the override for Bonus.
It does not freeze every method reachable from its body.
From the class author’s perspective, Bonus.score is still an ordinary method.
It may be called directly. Basic has no inherent score; uniform application
code uses a bound or Scored.score(&basic). This asymmetry is the main ergonomic
cost of avoiding automatic injection, and should be evaluated with users.
Same object, two protocols, different provided behavior
protocol Celsius:
pub def celsius(self) -> i64:
pass
@final
pub def display(self) -> i64:
self.celsius()
protocol Fahrenheit:
pub def celsius(self) -> i64:
pass
@final
pub def display(self) -> i64:
self.celsius() * 9 // 5 + 32
class Sensor:
reading: i64
pub def celsius(self) -> i64:
self.reading
def show[T: Celsius, U: Fahrenheit](left: &T, right: &U) -> Result[(), IoError]:
alias = left
print(alias.display())?
print(right.display())?
Ok(())
def main() -> Result[(), IoError]:
sensor = Sensor(20)
show(&sensor, &sensor)?
Ok(())
Expected output: 20, 68. Both generic arguments are concretely Sensor,
but their bounds select different implementations. The alias must retain the
left parameter’s protocol context. Neither default is installed globally on
Sensor, and neither protocol overrides the other.
This is why simply registering a synthetic Sensor.display when first
encountering a bound would be incorrect. It would make later calls depend on
which protocol was instantiated first.
Making the storage hook private across modules
This variation is a syntax sketch, not another complete runnable program:
# library.plenty
pub protocol DocumentStorage:
def write(self: &mut DocumentStorage, document: &str) -> Result[(), IoError]:
pass
@final
pub def save(self: &mut DocumentStorage, document: &str) -> Result[(), IoError]:
# Validate, call self.write(document), record success.
...
# backend.plenty
from library import DocumentStorage
pub class FileStorage:
@protocol.DocumentStorage # Agreed access-grant syntax; not implemented.
def write(self: &mut FileStorage, document: &str) -> Result[(), IoError]:
# Perform the actual file operation.
...
# application.plenty
def persist[S: DocumentStorage](storage: &mut S, document: &str) -> Result[(), IoError]:
storage.save(document) # Allowed.
# storage.write(document) would be rejected.
Here the class author explicitly permits the real library’s protocol to call
write; the application cannot create an equivalent protocol to acquire that
permission. The compiler checks the annotation where FileStorage is defined:
DocumentStorage must declare a non-public write with the same signature. No separate implementation block is required. Ordinary module
privacy still lets code inside backend.plenty call its own hook. All methods
defined in DocumentStorage may use the grant, but standalone helpers in
library.plenty do not inherit it. Class-only privacy was not selected.
The imports run one way: backend.plenty imports library.plenty, and the
application imports both. library.plenty never names FileStorage, because
protocol bodies are checked against the protocol’s own contract.
Protocol body implementation map and probes
This supports the feasibility sketch. Source
locations and results describe base d39e6ec, not a compiler with defaults.
The experiments use handwritten generic-function lowerings; they do not
implement new syntax, method selection, final-name checks, or private access grants.
Existing compiler machinery
| Area | Current implementation | Implication |
|---|---|---|
| Parsing | src/frontend/protocols.rs, Parser::protocol_decl | Uses the ordinary function parser and stores Vec<Function>, including bodies. An explicit guard rejects any body other than one Pass. Recognition is easy; removing the guard alone does not compile or register bodies. |
| Module resolution | src/frontend/modules.rs, protocol loop in resolve | Already walks each requirement through Scope::function and validates public signature types. Generated bodies must retain this definition-site namespace and access context, rather than borrowing the concrete class’s privileges. |
| Conformance | src/frontend/protocols.rs, check | Every method is currently required. Provided operations need separation from requirements; overridable defaults additionally need missing-method fallback and compatible-override checks. Include generic requirements and the bytes special case. |
| Inference | src/frontend/generics.rs, Engine::infer_bounds | Reads concrete inherent signatures for protocol requirements. Must avoid trying to read a nonexistent inherent default or inferring its parameters from itself. |
| Specialization | src/frontend/generics.rs, prepare and Engine::specialize | Generic templates, type substitution, cached instances, and specialization budgets already exist. A protocol body can become a template parameterized by implementing type, protocol arguments, and method-local parameters. |
| Calls | src/frontend/classes.rs, class_method | Derives the concrete owner/method symbol, checks access, then selects concrete or generic inherent methods. No general protocol-default fallback exists. |
| Access | src/frontend/modules.rs, check_member_at | Private access compares the member’s owner module with a source token. There is no protocol/member access-grant model. |
| Tooling | src/frontend/ide.rs, protocol declarations and method_call | Navigation needs the selected protocol body or override, not an invented concrete member. A feature must preserve source locations in generated templates and diagnostics. |
For explicit protocol operations, introduce distinct callable symbols, lower their bodies into existing generic templates, and resolve the qualified protocol call to that symbol. Keep conformance checks independent of compiling the body. Ordinary static calls can then flow through the existing IR/backend. There is no inherent need for protocol values, witness objects, vtables, boxing, or additional heap allocations. These are architectural conclusions, not benchmarked performance claims.
For bound-based syntax, more information must survive specialization.
Engine::specialize substitutes concrete types and clears function type
parameters. protocol_instances records a function-level flag used for
restrictions on named/default arguments; it is not a mapping from individual
receivers to protocol evidence.
Consider left: &T with T: P and right: &U with U: Q, instantiated with
the same concrete type. Choosing a default from the concrete type alone cannot
distinguish left.operation() from right.operation(). Aliases, returned
references, and captured receivers must preserve whichever selection semantics
the design promises. One possible approach is to resolve calls to explicit
protocol operations before erasing that information, but this needs more than
a textual receiver-name rewrite.
Reproducible experiments
From a checkout containing the script, build its compiler and run:
cargo build --locked --bin plenty
python3 -B scripts/probes/protocol_methods.py target/debug/plenty
The script creates isolated temporary modules, disables the compiled-function cache, and checks exact output or an expected diagnostic. It is a research harness, not a new language conformance suite.
| Probe | Observed result |
|---|---|
Library save and save_twice, external mutable backend, application in a third module | Runs; output -1, 3, 10, -999. Negative input is rejected before writing; nested shared calls update state; an inherent save remains a distinct operation returning -999. |
Same backend with non-public write | Rejected as private to its defining module. |
| Application imports library-private validation helper | Rejected as private. |
| A protocol method with a real body | Rejected by the existing default-implementation guard. |
Application calls memory.save when only the library function exists | Rejected as an unknown method. |
| Parameterized source protocol, concrete error inference, enum implementation | Runs; output 42. |
| Required generic method with its own protocol bound, called twice by shared code | Runs; output 14. |
| Shared generic function forwards a hook’s returned reference | Runs; output 8. |
| Mutation while that returned reference remains live | Rejected by borrowing checks. |
| Generic body calls a concrete method absent from its declared protocol | Runs; output 99, confirming per-instance checking rather than abstract contract checking. |
| Final storage candidate lowered to a generic function | Runs; output empty rejected, saved, 1; validation prevents the first write. |
| Final eligibility body with default and overridden score, lowered to two selected bodies | Runs; output False, True. Selection is handwritten, not inferred by a new compiler rule. |
| Two protocol operations on the same concrete sensor, including a receiver alias | Runs; output 20, 68; the lowering explicitly preserves the two selections. |
All 13 probes passed with the compiler built from the exploration worktree. Eight executed native programs and five checked expected compile failures. No compiler/runtime code was changed. These probes establish reuse of existing generics and ownership behavior; they do not establish that adding fallback lookup is a small patch.
The last three correspond to the candidate programs. Their handwritten lowering tests the algorithms and borrowing but cannot validate automatic selection, alias provenance, or rejection of final-method conflicts. Those remain compiler work. The cross-module private-hook sketch is not executed.
The accepted behavior now requires definition-time contract checking. The probe
that accepts an undeclared extra demonstrates a limitation of reusing today’s
generic checker, not behavior permitted in a future protocol body. The successful
lowerings establish executable algorithms and ownership behavior, but do not
validate this additional checker requirement.
Remaining validation if implemented
An implementation would need focused tests for qualified and bound-based
calls, imported aliases, two matching protocols, incompatible overrides,
calls between provided methods, recursion and specialization reuse/limits,
parameter inference without circular evidence, generic data and enum receivers,
bytes, mutable receiver reservations, returned loans and temporary receivers,
closures, diagnostics, and navigation. Changing body selection must also
invalidate the appropriate compiled artifacts.
New syntax should preserve existing rules for method-local parameters, parameter defaults and positional calls unless those changes are separately designed. An all-provided protocol and mutually recursive defaults need explicit rules: accepting their structural shape does not prove their algorithm terminates. Do not recursively compile bodies merely to decide conformance.
The body-lowering experiment did not test generated compiler symbols, actual default dispatch, abstract body checking, access grants, compile-time overhead, or binary-size changes. Full workspace tests and Clippy were not run for this documentation/Python-harness change; they are required for a Rust implementation.
Boxed protocol objects
Status: implemented with conservative erasure limits. Explicit Box[P]
owners support shared/mutable dispatch, fixed protocol arguments and generic
consumers of the same protocol. The implemented contract
is authoritative. This proposal retains the research and design rationale.
Shared cannot contain protocol objects, and construction rejects payloads
concealing executors, futures, channel endpoints or erased callbacks. Returned borrows, consuming
requirements, method-local generics and native transfer remain unsupported.
Generator-frame/concrete-closure signature types and recursive boxed interface
signatures are also rejected. See the retained measurements.
The investigation used local main at bb97038b on 2026-11-03. It found a
credible implementation path through the existing boxed-callable machinery,
not a working implementation of boxed protocols. Read the
public contract,
implementation design,
research evidence and runnable probes, and
qualification criteria for the details.
Intended behavior
A program should be able to store instances of different classes and enums, iterate over them, and call the methods declared by a common protocol without matching or casting. Every method on an object must reach the same underlying instance, including changes made through mutable methods.
Complete example of the implemented interface:
protocol MyProtocol:
def hey(self) -> i64:
pass
class Counter:
value: i64
def hey(self) -> i64:
self.value
enum Message:
Ready
Count(i64)
def hey(self) -> i64:
match self:
case Message.Ready:
0
case Message.Count(value):
*value
def main() -> Result[(), AllocError | IoError]:
mut items: list[Box[MyProtocol]] = []?
items.append(Box[MyProtocol](../Counter(10))?)?
items.append(Box[MyProtocol](../Message.Count(20))?)?
for item in &items:
print(item.hey())?
Ok(())
Expected output is 10 then 20. The list stores one common owner type;
each owner retains its concrete payload and the methods needed to operate on
it. The collection does not need a declaration listing every implementation.
Dictionary values work the same way, with ordinary supported keys:
items: dict[str, Box[MyProtocol]] = {
"counter": Box[MyProtocol](../Counter(10))?,
"message": Box[MyProtocol](../Message.Ready)?,
}?
for name, item in items.items():
print(name, item.hey())?
Protocol boxes are not dictionary keys or set elements. See the current heterogeneous collection lessons for implemented enum and anonymous-sum alternatives.
Why Box is explicit
The proposed spelling is list[Box[MyProtocol]], not list[MyProtocol].
MyProtocol remains a compile-time contract rather than a standalone stored
value type. The Box boundary supplies owned storage for an unknown concrete
layout, while Box[MyProtocol](../value)? exposes allocation and its failure.
This is a deliberate language choice, not a mathematical requirement. An unboxed-looking protocol value could be defined as an implicitly boxed owner, a borrowed view, or a fixed-capacity inline container. Each would need a different storage and lifetime contract. None is implied by this proposal.
Ordinary Box(value)? continues to produce Box[T] for the concrete T.
An annotation or collection insertion does not silently allocate or erase it.
The explicit protocol argument selects erasure. Direct P, list[P], and
bare &P remain unsupported as value types in the initial design; functions
can accept &Box[P] or an existing generic T: P bound.
Recommended scope
Support one protocol per box, shared and mutable receiver methods, fixed protocol type arguments, owned storage in ordinary aggregates, and existing generic consumers of that same protocol. Require every method to meet the boxing eligibility rules. Keep structural conformance and explicit allocation; introduce neither inheritance nor registration of implementations.
The initial contract excludes method-local generics, implementing-type occurrences outside the receiver, consuming receivers, returned borrows, downcasting, protocol-to-protocol conversions, copying, equality, and native worker/channel transfer. These restrictions do not reduce what protocols can express in today’s static generic calls.
Alternatives and tradeoffs
| Representation | Best fit | Consequence |
|---|---|---|
list[Item] with an enum wrapper | A known, finite set of alternatives | Exhaustive matching and inline payload storage; adding a variant changes the wrapper. |
list[A | B] | A small explicit set, dispatched by matching | No wrapper declaration; arbitrary user protocols are not forwarded today. |
list[Box[P]] | Implementations supplied independently of the collection | Explicit per-object allocation and indirect method calls; consumers know only P. |
list[BoxedCallable[[], i64]] | One reusable operation per object | Works today with an owned closure adapter; it does not provide several named methods over one mutable instance. |
An automatically generated enum of every implementation in the program would make the set depend on global discovery. Its layout, matching, compilation and module behavior need a separate design; it is not the mechanism proposed here.
A separate name such as BoxedProtocol[P], or a dyn P spelling inside Box,
could emphasize erasure. Prefer the requested Box[P] spelling, provided the
compiler and documentation clearly distinguish its restrictions from concrete
Box[T]. In particular, an erased payload cannot be extracted by value.
Feasibility assessment
This is a moderate, cross-cutting compiler feature. Type checking, generated adapters, indirect calls and concrete destruction already have useful precedents. Existing list/dictionary storage should accept a fixed owner representation without a new collection implementation.
The difficult work is preserving borrowing and effect guarantees after the concrete type disappears from the caller, not allocating memory or selecting a function pointer. The estimate is architectural judgment, not a calendar estimate or a measured prototype result.
This is independent of protocol method bodies.
Defaults, final methods, private-hook grants, and abstract checking of supplied
bodies need their own implementation. Boxed objects can initially dispatch the
accessible inherent methods required by protocols with pass bodies.
Boxed protocol object contract
Implementation note: the current reference describes delivered behavior. This page preserves the original design and qualification plan; research-era implementation statements below are historical. This is the proposed public contract for boxed protocols. It is not implemented. All new syntax in this page is illustrative.
Identity and construction
Box[P[Args...]] is an affine owner of one concrete value satisfying P[Args...].
Protocol identity includes its resolved declaration and fixed type arguments;
import aliases do not create new identities. Distinct protocol declarations
remain distinct boxed types even when their signatures match.
Box[P](../../boxed-protocols/value) consumes an owned concrete value, verifies conformance, and
returns Result[Box[P], AllocError]. It attempts one allocation for the box,
including for a zero-field class or payload-free enum. Method tables are
static compiler output, not a second allocation per instance. Construction of
the input and operations within its methods retain their own allocation costs.
Allocation failure destroys the consumed value, as ordinary Box(value) does.
If the box is constructed successfully but insertion into a collection fails,
the collection operation’s existing ownership contract applies to that box.
The proposal must not promise that a failed insertion returns its argument.
Programs preserving existing state should allocate detached replacement values
before moving state into them, as taught in
recursive insertion.
There is no implicit concrete-to-erased conversion on assignment, argument
passing, return, or collection insertion. Box[P](../../boxed-protocols/value) is the explicit
boundary. Allocation-free conversion from an existing Box[T] is outside the
initial scope: do not accidentally unbox/rebox it via today’s expected-type
coercions. Initially reject boxed inputs to this constructor and suggest
boxing the concrete owner directly. Moving an existing Box[P] needs no
constructor and performs no allocation.
The admitted concrete sources are ordinary storage-safe classes and enums,
including specialized generic instances, and the existing bytes conformance
case. Do not introduce new builtin conformances as part of boxing. Existing
storage barriers still reject borrowed fields/captures, generators and other
values that cannot enter a heap owner. An arbitrary anonymous sum does not gain
protocol forwarding merely because it is boxed.
Eligible protocols
A protocol is eligible when every requirement can be called with a fixed signature after substituting its protocol arguments and erasing its receiver. Check eligibility at a boxed-type use, not at protocol declaration: a protocol that is ineligible for boxing remains valid for static generic use.
| Requirement | Initial rule and reason |
|---|---|
def hey(self) -> i64 | Supported; bare self is a shared borrow. |
def set(self: &mut P, value: i64) -> () | Supported; the object must be exclusively borrowed. |
| Ordinary owned and borrowed arguments, including slice views | Supported under existing call and storage rules; arguments are not retained by erasure. |
Concrete owned results, including Result[T, E] | Supported; exact fallibility and payload types are part of the signature. |
Protocol-level parameters, such as Readable[i64] | Supported once every argument is fixed; they determine one concrete table shape. |
Method-level parameters, such as put[T](../../boxed-protocols/...) | Rejected; an ordinary finite table cannot supply every future specialization. |
| Protocol name outside the receiver, including nested occurrences | Rejected; that name denotes the hidden implementing type under current protocol rules. |
Owned receiver self: P | Rejected initially; consuming an erased payload requires a distinct transfer/cleanup contract. |
Returned references or views, including Option[&T] and Result[&T, E] | Rejected initially; dispatch must retain the origin loan before these are admitted. |
For example, compare(self, other: &P) requires another instance of the same
hidden concrete type. Two Box[P] values need not contain the same type, so
replacing that parameter with another protocol box would change the contract.
Similarly, returning P means returning the concrete implementing type today;
it must not silently become an allocating boxed result. Detect these uses
through aliases and nested signatures, before replacing the receiver type.
The current std.fmt.Formattable protocol has a method-local writer parameter.
It is therefore ineligible for this initial boxed representation. An erased
writer interface or a different reporting contract would be a separate design;
this proposal does not automatically create a general boxed error type.
Reject the entire boxed protocol if any requirement is ineligible; do not make the callable method set depend on which methods a particular program uses. Diagnostics should identify the protocol, offending requirement, and reason.
Calls, mutation and loans
object.method(args) selects a method declared by the boxed protocol. The
compiler knows its signature; only the implementation address varies at runtime.
Concrete fields and additional inherent methods are not accessible through the
box. No runtime name lookup or conformance check is needed on a call.
Shared calls borrow the owner, including through &Box[P], a list element,
a dictionary value or a class field. Mutable calls require &mut Box[P] or
an exclusively accessible mutable owner. Multiple methods share one payload:
protocol Counter:
def increment(self: &mut Counter) -> ():
pass
def current(self) -> i64:
pass
def advance(items: &mut list[Box[Counter]]) -> Result[(), IoError]:
for item in items:
item.increment()
print(item.current())?
Ok(())
The read observes the increment made to that same instance. This must not be implemented as separate independently captured copies for each method.
Loans remain rooted in the owner and its collection/field projection. A live borrow prevents conflicting mutation, replacement, removal, moving or dropping the owner, and invalidating collection changes. Preserve ordinary argument evaluation and receiver reservation rules: erasure grants no extra aliasing. Initially make no field-precision claim across indirect calls.
Read-only calls on temporary owners retain them until the full expression ends. Mutable calls follow existing boxed-receiver place requirements. Calls, owner moves, and compiler-provided cleanup add no allocation; user methods and destructor hooks retain their own behavior and effects.
Integration with generic consumers
Recommend that Box[P] satisfy the same P bound through its exposed methods:
def greet[T: MyProtocol](value: &T) -> i64:
value.hey()
def use(value: &Box[MyProtocol]) -> i64:
greet(value)
Here T is Box[MyProtocol]; specialization compiles a forwarding indirect
call, not a new specialization for each hidden payload. Protocol arguments
come from the box’s identity, including when generic inference needs them.
Moving a box into a generic owner parameter still transfers its owner normally.
This is deliberately restricted to the identical resolved protocol and arguments initially. Matching another protocol from an exposed method subset, combining protocols, and converting between erased protocols need separate rules. Do not discover extra capabilities by examining the hidden payload.
Current generic bodies are checked per concrete instance. A body that relies on
extra inherent methods may work for a concrete class yet fail for Box[P],
which exposes only P. Supporting boxed objects does not require broadening
that visible interface or silently implementing abstract generic-body checking.
Visibility and other operations
Use the existing protocol conformance visibility boundary, with the protocol’s
defining module as the bound site. Generated adapters must not gain private
access merely because they are emitted beside a concrete method. Public erased
signatures must expose accessible protocols and signature types. A library may
hide its concrete implementation behind a public Box[P] without exposing that
implementation’s fields or type name.
Support owners in lists, dictionary values, class fields, enum payloads, tuples,
Option, Result, parameters and factory returns, subject to ordinary aggregate
restrictions. Dictionary reading must preserve ownership: use borrowing/index
operations supported for affine values rather than implying get copies a box.
Do not support extraction of an erased payload with *object, bare &P or
&mut P values, downcasts, type inspection, implicit copying, copy, equality,
equality-based membership, hashing, or use as keys/set elements initially.
Existing Box[T] behavior remains available for concrete T. Proposed automatic
formatting is an opaque marker such as <boxed protocol MyProtocol>; it must
not inspect private fields or invoke arbitrary formatting methods implicitly.
Reject native worker/channel transfer, including through nested containers.
Erasure hides both payload eligibility and indirect method/destructor effects.
Also apply conservative shared-owner cycle checks: mutable storage of an erased
object must not conceal a back-reference that existing Shared checks would
reject. Actor/task handoff and allocator provenance need explicit qualification;
support for local calls does not certify them.
Boxed protocol implementation design
Implementation note: the current reference describes delivered behavior. This page preserves the original design and qualification plan; research-era implementation statements below are historical.
This supports the boxed protocol proposal. The source
map describes base bb97038b; representation and IR names below are suggestions,
not implemented APIs. Scheduling belongs to B29 in the backlog.
Reuse a concrete owner with an erased call interface
The existing BoxedCallable separates three facts that protocol objects also
need: a common public type, a concrete owned payload, and an adapter that knows
the payload’s actual layout. For protocols, replace the single call adapter
with a static table containing one adapter per eligible method.
A candidate representation is:
Box[P] owner
|
v
heap record
ownership/allocation header
concrete payload descriptor ------> layout, relocation and destruction
protocol method-table pointer ----> method 0 adapter
payload storage method 1 adapter
...
The public type has a fixed owner representation regardless of the payload’s size. Every table entry receives the same payload storage address. Adapters recover the concrete receiver representation and invoke the selected inherent method. Neither table lookup nor receiver adaptation allocates or copies the payload. Keep the descriptor needed for destruction even if the protocol has no cleanup-related method; lifecycle hooks are not protocol requirements.
plenty-runtime/src/aggregates.rs already has a Boxed header with a concrete
descriptor, a word currently called padding, and a payload slot at byte 32.
Boxed callable construction stores its adapter in that word. A protocol-table
address could occupy the analogous word for a distinct erased owner kind.
This suggests a compact implementation, not a frozen ABI or permission to
reinterpret a callable as a protocol object. Validate alignment, relocation,
destruction and target layouts before committing to offsets.
The existing allocator and descriptor-driven iterative box cleanup are the preferred foundation. Do not replace them with a new reference-counting or recursive destruction system. Explicit copying of an ordinary box must never become an accidental way to copy a protocol object.
Frontend and type identity
Introduce an explicit internal type, illustratively Ty::BoxedProtocol, whose
identity carries the resolved protocol and instantiated method signatures.
Keep the concrete payload type on construction/adapters, not in the erased
public identity. This avoids pretending that P has an ordinary inline size
or passing it through operations that assume every Ty::Box(T) can yield T.
Resolve Box[P[Args...]] through protocol-aware type resolution before the
ordinary concrete Box path tries to resolve its content as a value type.
Preserve imported aliases, visibility, generic substitution and canonical
identity. A protocol name alone must not start resolving as a stored value.
Explicit Box[P](../../boxed-protocols/value) construction also needs frontend handling; the current
Box(value) builtin merely infers its concrete content type.
Perform eligibility checking after resolving protocol arguments, while retaining the implementing-type marker needed to detect forbidden non-receiver uses. Then check the concrete input with the existing structural conformance rules. Keep receiver mutability, exact signatures and module access checks; method names alone do not establish conformance. Use positional method arguments initially, following current protocol-bound call restrictions, rather than inventing a keyword/default mapping from implementation parameter names.
Register adapters and the table for each canonical (concrete T, protocol P, protocol arguments) combination. Repeated construction should share emitted
code and data. Cache emitted artifacts separately from access decisions:
visibility must still be checked in each appropriate source context.
Materialize every required concrete method, including methods of generic
classes/enums, so no table entry points to an uninstantiated definition.
For T: P instantiated with Box[P], synthesize the exposed signatures with
the receiver replaced by a borrow of the owner and forward method calls through
the table. Inference should read the box’s fixed protocol arguments, not inspect
hidden concrete types. Do not loosen generic matching or protocol identity to
make this case fit by accident.
Typed IR and native lowering
Use explicit typed operations, illustratively ProtocolBoxNew and
ProtocolBoxCall, rather than smuggling an erased owner through concrete-box
operations. The IR checker must independently validate input ownership,
protocol identity, method index, receiver mutability, arguments, results and
adapter signatures. Construction consumes its concrete input and produces
the exact allocation-error result.
Calls load the table and statically selected method entry, then use the existing
indirect-call ABI machinery. The code generator may erase the receiver address
at the ABI boundary, as boxed callable lowering does, but the typed IR and loan
checker must retain the real shared/exclusive owner contract. Shared-call
BoxedCallable code is a precedent, not a drop-in mutable-call implementation.
Emit tables with native data/function relocations. Track adapter addresses and their callees in reachability analysis, including executable pruning on Windows. The compilation cache must include protocol identity, signatures, table order, receiver mode and selected implementation bodies. A method reorder or signature change must not reuse a caller expecting the old slot layout. Use deterministic table order, initially declaration order; no stable external table ABI is promised. Foreign export/import of these owners is outside the initial scope.
Integration map
| Area | Existing locations | Required integration |
|---|---|---|
| Types and construction | src/frontend.rs, src/op.rs, src/boxed.rs | Distinguish boxed protocols from concrete box extraction and type identity. |
| Protocols and generics | src/frontend/protocols.rs, src/frontend/generics.rs | Eligibility, conformance, fixed-argument inference, adapter specialization. |
| Method dispatch | src/frontend/collections.rs, src/frontend/classes.rs | Resolve methods of erased owners through fields, references, collection projections and temporaries. |
| Callback precedent | src/frontend/callables.rs, src/codegen/closures.rs | Generated adapters, fixed-signature indirect calls, fallible owner construction. |
| Independent verification | src/op/check.rs | Construction/call signature checks and receiver ownership checks. |
| Ownership and effects | src/ownership.rs, src/ownership/calls.rs, src/ownership/futures.rs, src/frontend/provenance.rs | Preserve loans and treat unknown indirect effects conservatively. |
| Runtime storage | plenty-runtime/src/aggregates.rs, src/codegen/metadata.rs | Concrete drop/relocation descriptor plus static method-table identity. |
| Reachability and artifacts | src/codegen/reachability.rs, src/threading.rs, codegen cache/dump paths | Retain adapters, serialize identities and reject uncertified transfer. |
| Shared owners | src/frontend/shared.rs | Prevent hidden payloads from bypassing cycle and executor/future ownership checks. |
| Modules and tooling | src/frontend/modules.rs, src/frontend/ide.rs | Public-interface validation, diagnostic blame and navigation to protocol requirements. |
All type walkers need an audit: storage barriers, contained owners, copying, equality, formatting, generic substitution, references, checked effects, serialization and FFI eligibility. An exhaustive type-enum match helps find sites, but a wildcard arm can silently apply an unsafe default.
Safety boundaries that erasure must preserve
Conformance proves signatures, not effects. A method may call back into user code, allocate, wait, acquire a lock or access another owner. Existing indirect call analysis is conservative; a protocol method table must not accidentally look like an effect-free primitive. Audit lock-held waits, live future/task obligations and reentrant arguments as well as ordinary borrow conflicts.
The existing BoxedCallable rejection at worker/channel boundaries is a useful
initial policy. Hiding a non-transferable object inside a list or a class must
not make it transferable. A future transferable protocol owner would need
certification at construction for payloads, reachable methods and destructors;
the presence of a matching protocol is not that certification.
Shared ownership has a second erasure hazard: a payload stored behind mutable
state can contain a share of its enclosing owner. Existing Shared traversal
already treats erased callbacks conservatively across such edges. Extend that
policy to protocol objects and audit hidden executors/futures too. Do not claim
that the ordinary box descriptor alone makes these static checks unnecessary.
Returned borrows are plausible later because Plenty ties a method’s reference result to its receiver. They still require forwarding the caller’s owner loan through an unknown implementation, including optional/fallible borrows and temporary receivers. Reject every reference-carrying output in the initial contract rather than copying only the direct-reference guard from boxed callables. Borrow-carrying output must be recognized structurally.
Relationship to protocol bodies
The protocol-body proposal specifies defaults,
final operations and private-hook grants. It is not a prerequisite here. The
initial table contains concrete inherent implementations of pass requirements.
If both features ship, table construction should use the same resolved selection rules as static bound calls. Default calls must honor overrides consistently; final methods and access grants must retain their declared authority. Avoid a second, independent method-selection algorithm in the backend. Abstract checking of supplied bodies remains part of that proposal, not a capability established by the single-method erasure probe.
Boxed protocol research evidence
This records the investigation behind boxed protocols.
The inspected baseline is local main at bb97038b, dated 2026-11-03. The
local debug compiler reported plenty 0.8.8 (x86_64-unknown-linux-gnu).
No protocol-object implementation or performance measurement was produced by
that original investigation. The current implementation is described in the
reference; the harness
now tests protocol objects by default. Use --baseline with the research-era
compiler to reproduce the historical rejection below.
Implementation measurements
The implementation was measured on x86_64 Linux with optimized native output
(--release). The retained scripts/probes/boxed_protocols_bench.py constructs
20,000 alternating one-field/two-field objects, performs 200 shared-read passes
(4 million calls), and destroys the list. Each phase uses std.time; the table
reports medians of five runs in milliseconds, excluding process startup.
Construction includes list growth. Every representation checks the same checksum.
| Representation | Construction | Shared reads | Cleanup | Executable bytes |
|---|---|---|---|---|
| inline enum | 1.870 | 7.731 | 0.272 | 183,896 |
| concrete box | 2.345 | 7.522 | 0.587 | 184,712 |
| boxed callable | 3.501 | 22.213 | 0.903 | 186,344 |
| boxed protocol | 2.024 | 16.210 | 0.689 | 185,112 |
The protocol representation trades indirect-call cost for an interface that can accept independently supplied implementations. In this workload its shared-read phase takes about twice the inline-enum/concrete-box time, and less than the boxed callback adapter. These are local microbenchmarks, not general speed promises: payload sizes differ between representations, methods do very little work, and host load changes the results. Mutation performance, large method tables and other native targets were not measured. Native execution qualification here is x86_64 Linux; no macOS, Windows or AArch64 run is claimed.
The harness also reports one cold compiler invocation per representation and linked executable size; compiler timings are not repeated measurements. Run it with a compiler built from this checkout:
python3 -B scripts/probes/boxed_protocols_bench.py target/debug/plenty
Reproducible compiler probes
The retained harness is scripts/probes/boxed_protocols.py. It writes isolated
temporary programs, disables the compiled-function cache with PLENTY_CACHE=off,
executes positive cases and checks expected diagnostic text in negative cases.
It requires no third-party Python packages.
From the checkout whose behavior is being investigated:
cargo build --locked --bin plenty
python3 -B scripts/probes/boxed_protocols.py --baseline target/debug/plenty
The investigation ran the harness with the existing debug compiler in the primary checkout; no compiler/runtime sources differ between that baseline and this proposal branch. All four probes passed: two native programs and two expected type-checking failures. The rejection probes characterize this baseline, not permanent requirements; update them when B29 implements boxing.
| Probe | Observed result | What it establishes |
|---|---|---|
Declare items: list[Box[MyProtocol]] = []? | Rejected: unknown type MyProtocol (the diagnostic quotes the name with backticks). | Protocols currently do not resolve as boxed value types. |
Iterate over list[Box[A]] and call item.hey() | Runs; prints 10. | Concrete boxes already support method calls through borrowed list elements. |
Append Box[B] to that list[Box[A]] | Rejected: expected Box[A], got Box[B]. | Heap storage alone does not erase the concrete type. |
Adapt a class and an enum satisfying MyProtocol into boxed callbacks | Runs; prints 10 then 20. | Existing generic conformance, owned capture and callback erasure compose for one shared method. |
Both implementations used by the probes:
protocol MyProtocol:
def hey(self) -> i64:
pass
class A:
value: i64
def hey(self) -> i64:
self.value
enum B:
Value(i64)
def hey(self) -> i64:
match self:
case B.Value(value):
*value
The positive erasure probe adds this current-language code:
def erase[T: MyProtocol](value: T) -> Result[BoxedCallable[[], i64], AllocError]:
callback = def [value]() -> i64:
value.hey()
BoxedCallable(callback)
def main() -> Result[(), AllocError | IoError]:
mut items: list[BoxedCallable[[], i64]] = []?
items.append(erase(A(10))?)?
items.append(erase(B.Value(20))?)?
for item in &items:
print(item())?
Ok(())
The protocol requirement is checked when specializing erase. The resulting
callbacks have the same public signature despite different concrete captured
values. Calling the callback reaches the appropriate hey implementation.
This does not test multiple named methods, mutable erased receivers,
protocol-table generation, Box[P] satisfying a generic bound, destruction
under injected allocation failure, or returned borrows. Separate callbacks
capturing separate objects would not establish that several methods operate on
the same state. Existing BoxedCallable also rejects mutable and consuming
callbacks; the successful probe does not remove that limit.
Source findings
| Finding at the baseline | Source evidence | Consequence for the proposal |
|---|---|---|
| Protocols are stored as required method declarations. | src/frontend/protocols.rs, Protocol and Parser::protocol_decl | Reuse declarations and exact conformance checks; do not assume a runtime protocol type already exists. |
| The protocol’s own name substitutes to the implementing type. | protocols::check inserts the concrete type into its alias environment. | Preserve a marker long enough to reject non-receiver occurrences before erasure. |
| Protocol conformance checks accessible methods and receiver/signature equality. | protocols::check, including generic-requirement matching | Boxing must not weaken static guarantees or bypass module access. |
Ordinary Box[T] requires a resolved concrete content type. | src/frontend.rs, TypeRef::resolve and the Box constructor path | Both annotation resolution and explicit protocol construction need new handling. |
The IR has Box, BoxedCallable and Callable, but no protocol-object type. | src/op.rs, Ty, BoxedCallableNew, BoxedCallableCall | A distinct erased type/operation path is the safer starting point. |
| Boxed callbacks generate concrete receiver adapters. | src/frontend/callables.rs, Lower::boxed_callable | Multi-method objects can reuse the adapter approach without allocating one callback per method. |
| Calls use a hidden environment address and a fixed public signature. | src/codegen/closures.rs, lower_boxed_callable_call | There is already native indirect-call support; the proposed extension selects an entry from a static table. |
| The allocated box retains its concrete descriptor. | plenty-runtime/src/aggregates.rs, Boxed, try_box_new, box_destroy, operation 128 | Erased callers need not know layout to preserve correct relocation and destruction. |
| Metadata emission supports function-address relocations. | src/codegen/metadata.rs, define | Method tables can be immutable native data with adapter addresses. |
| Boxed callable eligibility and call signatures are independently checked. | src/op/check.rs, BoxedCallableNew/BoxedCallableCall arms | New protocol operations need equivalent validation, especially for mutable receivers. |
| Erasure does not certify worker eligibility. | src/threading.rs, Ty::BoxedCallable rejection | Preserve conservative nested transfer restrictions until a capability contract exists. |
| Shared-cycle checks treat mutable paths containing erased callbacks specially. | src/frontend/shared.rs, rejection | Hidden protocol payloads require the same safety audit. |
| Indirect calls affect ownership/effect summaries and function reachability. | src/ownership/calls.rs, src/ownership/futures.rs, src/codegen/reachability.rs | A callable table is not merely a runtime representation change. |
The box layout contains a word used for the boxed callback’s adapter; its payload slot begins at byte 32 in the inspected implementation. Those are current source facts, not language promises. The suggestion to store a protocol-table pointer there is an inference requiring its own runtime and target tests.
Existing tests and local design history
tests/test_boxed_callables.rs already covers heterogeneous collections and
factory returns, generic callable constraints, shared calls, owned/borrowed
arguments, inline results, rejection of unsupported modes, ownership and
destruction. Runtime tests in plenty-runtime/src/tests.rs include
callable_boxes_preserve_adapter_concrete_descriptor_and_relocated_captures
and the allocation-checks test
failed_callable_box_drops_the_concrete_environment_once.
These tests were inspected, not rerun as part of this investigation; their
existence is a useful implementation template, not new validation evidence.
Local issue #25 and merged local PR #39 record the shipped
BoxedCallable work. The issue is closed; its resolution names merge commit
b25a2309368070c90fb69a42a4ccbc1480857717 and implementation commit
bc6b1f79d7bb4959671fde60c75c4ff46d91bdf5. Its description and PR explain
explicit allocation, concrete descriptor retention, and deliberately excluded
mutable/consuming/borrowed/worker-crossing modes. These numbers refer to the
shared local issues.db, not remote records.
The original protocol design explicitly deferred existential interface values and called out method tables, ownership and restrictions on implementing-type/generic signatures. The error-composition investigation separates static bounds, finite error sums and erased error objects; allocation while handling allocation failure remains a particular concern.
The boxed callable reference
is authoritative for the shipped erasure contract. The
protocol reference confirms
that arbitrary anonymous sums do not forward user protocols, while
std.fmt.Formattable has specific sum forwarding. Its writer-generic method is
documented in reporting.
Do not infer general protocol-object support from that special case.
Semantic search was checked fresh against the primary checkout before drafting. Related hits were verified against current source or local records. Some old HTTP route discussions predate enum methods; their historical limitations are not evidence that current enums lack methods. The native class/enum probe above establishes the relevant current behavior directly.
Confidence and limits
The feasibility conclusion is supported by current type checking, successful single-method erasure, and inspected ownership/runtime mechanisms. It does not establish a finished mutable protocol-object ABI, safe capability transfer, cross-platform qualification, code-size impact, or dispatch performance. The qualification criteria distinguish those obligations from the four baseline probes. No compiler/runtime feature has shipped in this work.
Boxed protocol qualification criteria
Implementation note: the current reference describes delivered behavior. This page preserves the original design and qualification plan; research-era implementation statements below are historical. These are acceptance criteria for the boxed protocol design, not a second work queue or a claim of implementation. B29 in the backlog owns scheduling. The research evidence records what was actually run.
Suggested implementation sequence
An internal spike can start with one shared receiver method and two concrete payload layouts. Prove construction, list iteration, indirect calls and correct drop before generalizing to several methods. Reuse descriptor-driven storage and generated adapters rather than building a second owner runtime.
Then implement eligibility, parameterized protocol identity, exact conformance, generic-bound integration and mutable receiver loans. A shared-only spike is useful evidence but does not meet the proposed shared-and-mutable contract. Complete type walkers, effects, visibility, diagnostics and tooling before presenting the feature as supported.
Keep these stages inside one agreed implementation scope or explicitly narrow the public contract before shipping a subset. Do not silently omit a method from the table because its ownership handling is unfinished.
Required behavior and regression coverage
| Area | Acceptance evidence |
|---|---|
| Heterogeneous storage | A class and an enum with different layouts share a list, dictionary values, fields, enum payloads and factory result type. Borrowed iteration calls the correct implementation without matching. |
| Shared state | At least two methods, including a mutable method, operate on one instance. Repeated calls and moving the box preserve that state. Shared references cannot call mutable methods. |
| Generic use | Box[P] works through T: P, fixed protocol arguments infer correctly, aliases preserve identity, and extra concrete methods remain inaccessible. |
| ABI | Scalar/unit/inline aggregate results; owned and reference/view arguments; fallible results; receiver modes; parameter count limits and argument evaluation order. |
| Loans | Nested projections, mutable list iteration, dictionary get_mut, temporaries, reentrant argument evaluation, and rejection of moves/removals/replacement during live loans. |
| Lifetime and destruction | Collection growth, reorder, extraction, replacement, early return and ordinary drop destroy each concrete payload exactly once, including custom destructors and nested owners. |
| Allocation failure | Inject failure at box construction and collection growth/insertion; verify the existing consumed-input contract, no leaks/double drops and valid remaining owners. |
| Eligibility diagnostics | Reject generic requirements, consuming receivers, all returned-borrow forms, and implementing-type occurrences outside the receiver, including through aliases and nested types. Existing static generic uses remain valid. |
| Privacy | Cross-module public protocol/private implementation, inaccessible required methods, hidden signature types, imported aliases, and no capability gained from adapter emission context. |
| Unsupported operations | Reject bare protocol values, implicit erasure, boxed-input construction, extraction, copying/equality/hashing, keys/set elements, downcasts and protocol conversions. Nested aggregate operations obey the same restrictions. |
| Capabilities and effects | Reject uncertified worker/channel transfer at every nesting level. Audit Shared cycles, hidden executor/future ownership, indirect blocking effects, locks and actor/task handoff. |
| Artifacts and tooling | Retain every adapter and destructor through pruning/linking; correct IR dumps, diagnostics/navigation, deterministic table layout and cache invalidation after protocol or implementation changes. |
Independent IR tests should reject malformed protocol calls even if normal frontend lowering would not emit them. Runtime tests should protect the owner header, table address, payload relocation and destruction invariants, mirroring the concrete descriptor tests for boxed callables.
Performance questions
Compare equivalent enum-wrapper, concrete boxed, boxed callback and boxed protocol workloads. Keep payloads, allocation counts and method work comparable; an inline enum versus one allocation per object measures storage as well as dispatch. Separate construction, iteration, mutation and destruction costs.
Measure payload-size variation, homogeneous versus mixed implementations, method count and number of concrete implementations. Record native code/table size and compilation time as well as runtime and allocation counts. Verify that moves, method dispatch and compiler-provided cleanup introduce no extra allocations. Do not promise a precise instruction count or overhead before measurement; optimizers may remove some indirect calls in particular programs.
Documentation and release qualification
On implementation, update protocol and box references, ownership/returned-borrow limits, generic conformance, architecture/support and runtime representation notes. Update broad statements such as “no dynamic dispatch” to distinguish static generic calls from explicitly erased owners. Teach runnable list and dictionary-value examples after classes/enums, retaining enum wrappers as the clear finite-alternative option. No new heterogeneous key/set lesson is implied.
Use the repository’s required formatting/lint and compiler/runtime test commands, including allocation-checks coverage; test executable lessons and all docs CI checks. Qualify native object relocations and calls on supported targets before claiming portability. Keep proposal status, delivery records and B29’s remaining scope aligned with the exact subset that ships.
Decisions to revisit only with evidence
The recommended initial contract makes concrete choices: explicit Box[P],
shared/mutable methods, no consuming/returned-borrow methods, and no certified
thread transfer. Implementation may uncover reasons to revise those choices;
record such revisions before relaxing a safety boundary.
The exact internal layout, table-sharing cache and diagnostic wording remain implementation choices. Bare borrowed protocol views, allocation-free erasure of existing boxes, cross-protocol conversion, method-generic dispatch, downcasting, copying and erased error reporting are outside this scope. Their absence does not block iteration and ordinary protocol method calls.
GCC backend (plenty_codegen_gcc)
Status: Not implemented (B20). The compiler has no
--backendoption. This page is the only record of the GCC backend design. API facts were checked against thegccjit7.1.0 crate source and the libgccjit 17 documentation on 2026-10-07.Out of date since then: line numbers, file sizes, and test counts refer to the tree on 2026-10-07. The Cranelift backend has grown from about 2,840 lines in five files to about 4,900 in
src/codegen.rsand fifteen submodules, plussrc/tail_abi.rs. A revised plan must cover what was added: parallel function emission (src/codegen/output.rs), which libgccjit’s global lock constrains; closures, threads, channels, executors, exports, and foreign calls; and inline aggregate values. Native emission supports onlyx86_64-unknown-linux-gnu(B18).
This backend design is separate from the native-platform assessment: another code generator does not by itself port the runtime, standard library, or foreign-call contracts.
Goal
Add a second AOT backend that uses GCC, through libgccjit, to make native
code. Cranelift stays the default backend. The model is
rustc_codegen_gcc.
┌── codegen_cranelift ──> CLIF ──────┐
Op ── op::check ── codegen_ssa┤ ├──> .o ──cc──> executable
└── codegen_gcc ──> libgccjit trees ─┘
Why
- GCC’s optimizer can make faster code, and with
-Os, smaller code. - GCC supports more CPU targets than Cranelift.
- It does not change runtime memory use. That comes from the value and heap design, not the backend.
Cost
- System dependency.
libgccjit.somust be installed to use the backend. Use the crate’sdlopenfeature (see below) so the default build and the default binary do not need it. - One target per install. One
libgccjitbuild makes code only for the target GCC was built for. To cross-compile, you need a separate libgccjit for each target. - Slower compiles, more compiler RAM than Cranelift.
- Two backends to maintain. Every new
Opneeds two lowerings, unless you first add a shared lowering layer (see Phase 1). - License. The
gccjitcrate and libgccjit are GPL-3.0. See “License”.
License
Not legal advice. This is the common reading.
Plan: license Plenty as MIT OR Apache-2.0. The repo has no license file now.
What works:
- All of Plenty’s own source can be
MIT OR Apache-2.0. Both licenses are compatible with GPL-3.0, so GPL code can sit beside them. - User programs are not affected. The runtime that is linked into compiled Plenty programs is Plenty’s own MIT/Apache code. Machine code that GCC makes is not covered by GCC’s license.
What does not work:
- A distributed
plentybinary built withgcc-backendincludes thegccjitcrate (GPL-3.0). That binary must be distributed under GPL-3.0 terms, including an offer of its source. dlopendoes not fix this. It removes the link-time need forlibgccjit.so, but the crate’s Rust code is still compiled into the binary.
Options:
- Keep
gcc-backendoff by default (recommended). Default binaries stay pure MIT/Apache. Only builds with the GCC backend carry the GPL. Say this in the README. - Write your own bindings for the about 40 libgccjit functions you need, instead of using the GPL crate. libgccjit itself is still GPL-3.0, and the FSF treats dynamic loading as a combined work. This option is weaker than it looks.
- Emit C and run
gccas a separate program. This avoids the question completely, because Plenty does not link GCC. It is not therustc_codegen_gccmodel.
Setup:
- Set
license = "MIT OR Apache-2.0"inCargo.tomlandplenty-runtime/Cargo.toml. - Add
LICENSE-MITandLICENSE-APACHEat the repo root.
How rustc_codegen_gcc does it (the model)
rustc_codegen_ssaholds the shared lowering. It calls builder traits (BuilderMethods,ConstMethods, …).- Each backend (
_llvm,_cranelift,_gcc) implements those traits. - The
_gcccrate uses thegccjitRust crate (bindings tolibgccjit). - It uses a patched GCC fork. Plenty does not need the fork. See “The
gccjitcrate” below.
The gccjit crate: facts that save time
Crate: gccjit = "7.1.0" (depends on gccjit_sys 4.1.0). Repo:
https://github.com/rust-lang/gccjit.rs. Docs on docs.rs are built with the
master feature, so docs.rs shows functions that you cannot use.
Features:
| Feature | Use it? | Why |
|---|---|---|
master | No | Needs the rustc_codegen_gcc GCC fork. Gates try/catch, attributes, get_target_info, new_temp, global_set_readonly, vector ops, get_error_count. Plenty needs none of these |
dlopen | Yes | Loads libgccjit at run time with gccjit::load(c"libgccjit.so.0"). Without it, the crate links -lgccjit, and every build of plenty needs libgccjit installed |
Every API this plan needs is available without master:
Context::compile_to_file(OutputKind::ObjectFile, ..),
RValue::set_require_tail_call, Block::end_with_switch, Context::new_case,
Function::get_address, Context::get_builtin_function,
LValue::global_set_initializer (bytes), LValue::global_set_initializer_rvalue,
Context::new_struct_constructor, LValue::set_alignment,
Context::add_command_line_option, Context::dump_reproducer_to_file.
Linkage maps directly (gccjit::FunctionType):
Cranelift Linkage | Used for | FunctionType |
|---|---|---|
Export | plenty_main | Exported |
Local | user functions, resume bodies | Internal |
Import | runtime helpers | Extern |
Error handling gotchas:
- In debug builds (this includes
cargo test), the crate panics after any call ifget_last_error()returns text. Withoutmaster, it cannot read the error count, so it panics on warnings too. Do not emit code that makes GCC warn. If you must, wrap the GCC backend instd::panic::catch_unwindand turn the panic into aResulterror. - In release builds, it does not panic. Check
ctx.get_first_error()aftercompile_to_file, or a failed compile is silent. - Any call that returns NULL panics in all builds (
panic_on_null).
libgccjit rejects unreachable blocks by default. The Cranelift lowering
makes blocks that have no predecessor (for example the block after a
TailCall or after a trap). Call ctx.set_allow_unreachable_blocks(true), or
do not create those blocks.
Optimization levels: OptimizationLevel has None, Limited, Standard,
Aggressive (-O0 to -O3). There is no -Os variant. Use
ctx.add_command_line_option("-Os").
Concurrency: libgccjit holds a global lock while it compiles. Parallel tests that use the GCC backend run their GCC compiles one at a time. They are correct, but slower.
Assembler: libgccjit writes assembly and runs the assembler. as
(binutils) must be on PATH. Plenty already needs cc, so this is normally
true.
Minimum libgccjit version
The libgccjit docs list features by ABI tag, not by GCC version. The tags this plan needs:
| ABI tag | Adds | Needed for |
|---|---|---|
LIBGCCJIT_ABI_3 | end_with_switch, new_case | Generator resume dispatch |
LIBGCCJIT_ABI_6 | set_bool_require_tail_call | Mandatory TCO |
LIBGCCJIT_ABI_9 | function_get_address | Resume and drop callbacks |
LIBGCCJIT_ABI_14 | global_set_initializer (bytes) | String literals |
LIBGCCJIT_ABI_19 | global_set_initializer_rvalue, struct/array constructors | Metadata blobs with pointers |
LIBGCCJIT_ABI_20 | INT128_T / UINT128_T, type_get_size | Inline sums (I128) |
LIBGCCJIT_ABI_24 | lvalue_set_alignment | set_align(8) on data |
So the minimum is LIBGCCJIT_ABI_24. With dlopen, gccjit::load fails
if the library lacks any symbol that the non-master crate binds, so the
crate may need a newer tag than this. Check what is installed:
objdump -T /usr/lib64/libgccjit.so.0 | grep -o 'LIBGCCJIT_ABI_[0-9]*' | sort -uV | tail -1
Current state (what the GCC backend must match)
The Cranelift backend is about 2,840 lines:
| File | Lines | Contents |
|---|---|---|
src/codegen.rs | 2029 | Driver, runtime declarations, functions, Lowerer |
src/codegen/enums.rs | 313 | Sum types |
src/codegen/generators.rs | 231 | Resumable bodies |
src/codegen/collections.rs | 134 | Collection ops |
src/codegen/metadata.rs | 133 | Static type metadata blobs |
Where to look first:
| What | Location |
|---|---|
| Entry point for all AOT builds | compile_ops_to_executable, src/codegen.rs:156 |
| Object emission, passes 1–3 | compile_to_object, src/codegen.rs:212 |
| ISA flags | host_isa, src/codegen.rs:273 |
| Runtime helper table (34 imports) | Runtime / declare_runtime, src/codegen.rs:293, :346 |
| User function signature | user_fn_signature, src/codegen.rs:528 |
| Plenty type to machine type | clif_type, src/codegen.rs:836 |
| Trap sequence | trap_if, src/codegen.rs:1477 |
| Tail call | lower_tail_call, src/codegen.rs:1858 |
Resume signature, func_addr | src/codegen/generators.rs:12, :72 |
| Data with pointers | define, src/codegen/metadata.rs:9 |
Features it uses, and the libgccjit equivalent:
| Cranelift feature | libgccjit equivalent | Notes |
|---|---|---|
| Object file output | compile_to_file(OutputKind::ObjectFile, ..) | Link step with cc stays the same |
| Runtime calls (SystemV) | new_function(.., FunctionType::Extern, ..) | Same plenty-runtime archive |
| Compile-time stack of SSA values | Store each pushed value in a local | See “rvalues are trees” |
CallConv::Tail + return_call | new_call(..) + set_require_tail_call(true), then end_with_return | Highest risk. See Risks |
Multiple return values (sig.outputs is a list) | One struct type per distinct output list; return the struct | C has one return value. Cache struct types by output list |
Variable | Function::new_local | Direct match |
Block params at match join | One local per param; assign before end_with_jump | GCC’s SSA pass removes them |
brif, jump | end_with_conditional, end_with_jump | Direct match |
| Generator resume dispatch | end_with_switch + new_case | Continuation index becomes a case value |
func_addr (resume, drop_callback) | Function::get_address | |
*_overflow checked arithmetic | get_builtin_function("__builtin_add_overflow") etc. | Type-generic. If it fails, see Risks |
trap after trap helper | Call helper, then __builtin_unreachable | Helper does not return |
select | Two blocks and a local | No ternary rvalue without master |
fcvt_to_sint_sat / _uint_sat | Explicit NaN check, compares, and clamps | A plain C cast is undefined when out of range |
I128 (inline sums) | new_c_type(CType::UInt128t) | Must match Rust u128 ABI in the runtime |
| Stack slots | Local array or struct + LValue::get_address | |
| Static string data | u8 array global + global_set_initializer | |
| Data that points to other data | Struct global + new_struct_constructor + global_set_initializer_rvalue | See “Metadata blobs” |
set_align(8) | LValue::set_alignment(8) | |
is_pic, preserve_frame_pointers | add_command_line_option("-fPIC") | Frame pointers are a Cranelift tail-call need only. Confirm in Phase 0 that the object links as a PIE |
rvalues are trees, not SSA values
In Cranelift, a Value is computed once. You can use it many times.
In libgccjit, an rvalue is an expression tree. If you use the same rvalue
twice, GCC computes it twice, and side effects (such as runtime calls) happen
twice. Rule for the GCC backend: assign each pushed value to a new local, and
push the local. This is how rustc_codegen_gcc handles it too. GCC’s
optimizer removes the extra locals.
Metadata blobs
metadata.rs makes each blob as bytes plus a list of (offset, DataId)
pointer slots. To build the same blob in libgccjit:
- Sort the pointer offsets.
- Make a struct type. Bytes between pointer slots become
u8[N]fields. Each pointer slot becomes avoid *field. - Build a
new_struct_constructorwith array constructors for the byte fields (one rvalue per byte: a string literal cannot fill achararray) andget_addressof the target global for each pointer field. - Set alignment 8.
Phase 1 should make the bytes-plus-slots form the backend-neutral output of
metadata.rs, so both backends use the same layout code.
Undefined behaviour
GCC optimizes on C undefined behaviour. Cranelift does not. Rules:
- Do wrapping arithmetic on unsigned types, then cast back.
- Check shift amounts before you shift.
- Add
-fno-strict-aliasingwithadd_command_line_option. The runtime and the generated code read the same memory through different types.
Phases
Phase 0: spike (small)
Goal: prove the toolchain works before you change the design.
sudo dnf install libgccjit libgccjit-devel. Neither is installed now. System GCC is 16.2. Run theobjdumpcommand above.- In a scratch crate with
gccjit = { version = "7.1", features = ["dlopen"] }:- write an object file with one exported
plenty_main, - call one runtime helper (for example
plenty_print_str), - link it with
libplenty_runtime.athroughcc. Extract the archive withplenty --emit-runtime DIR.
- write an object file with one exported
- Test mandatory tail calls, at
-O0and-O2:- self tail call, mutual tail call,
- a callee with more arguments than the caller (more than 6 integer arguments on x86-64, so some go on the stack),
- a callee that returns a struct larger than 16 bytes (returned in memory),
- a callee that takes or returns
unsigned __int128.
- Test
__builtin_add_overflowthroughget_builtin_functiononu8,i32, andi64. - Test a global whose initializer holds the address of another global.
- Build in debug mode and confirm that no step panics on a warning.
Exit: a short note on which features work, and which need a fallback.
Phase 1: backend seam (large, no behaviour change)
Goal: split the shared lowering from the Cranelift-specific code, like
rustc_codegen_ssa.
- Move backend-neutral work out of
codegen.rs:collect_user_fns,collect_str_ids,needs_local_addresses,- the runtime helper table (names and signatures as plain data),
- metadata blob layout (bytes plus pointer slots).
- Define builder traits for what
Lowererneeds. Start from the feature table: constants, arithmetic, checked arithmetic, compares, casts, calls, tail calls, blocks, branches, switch, locals, memory load/store, data addresses. - Make
Lowerergeneric over the builder. Implement the traits for Cranelift. - All existing tests must pass with no changes.
Design notes:
- gccjit handles carry a
'ctxlifetime (RValue<'ctx>,Block<'ctx>). Give the traits a lifetime parameter, asrustc_codegen_gccdoes withCodegenCx<'gcc, 'tcx>. Do this in the first version of the traits, or you must change every signature later. - Model “value” in the trait as something that a backend can read many times
with no side effects. Cranelift returns its SSA
Value. GCC returns a local. - Model block params as “declare join values, assign, jump”. Cranelift maps this to block params; GCC maps it to locals.
This is the largest phase. It is also useful on its own: it makes new Op
lowerings easier to write and review.
Alternative: skip Phase 1 and copy Lowerer into a GCC version. This is faster
to start, but every later Op change needs two edits that can drift apart. Use
this only if you want a throwaway experiment.
Phase 2: GCC backend, core (medium)
Put it in src/codegen/gcc/ behind a Cargo feature gcc-backend (optional
dependency on gccjit with dlopen).
Order of work, smallest first:
- Integers, booleans,
Displayof numbers,plenty_main, exit status. - User functions, calls, multiple returns, tail calls.
match, patterns, join locals.- Strings and static data.
- Checked arithmetic and traps.
- Floats and saturating casts.
Crate layout note: a separate plenty-codegen-gcc workspace crate would need
Op and Ty from plenty, and plenty would need the backend. Cargo does not
allow that cycle. Use a module behind a feature now. Split out a plenty-core
crate later only if you need it.
Phase 3: GCC backend, full coverage (medium)
- Sum types (
enums.rs), includingI128inline sums. - Collections.
- Metadata blobs with pointers.
- Generators (switch-based resume,
get_addressfor callbacks).
Phase 4: CLI and tests (small)
- CLI:
--backend cranelift|gcc. Defaultcranelift. If the binary was built withoutgcc-backend, orgccjit::loadfails, give a clear error. - Tests: all 20 test files that compile programs reach
compile_ops_to_executable(src/codegen.rs:156), most throughtests/support/mod.rs. Read aPLENTY_BACKENDenv var there, and one change runs the full AOT suite on GCC:PLENTY_BACKEND=gcc cargo test --features gcc-backend. - Run the GCC suite at
-O0and-O2. Undefined behaviour bugs show only at-O2. - CI: add one job that installs libgccjit and runs the command above.
- Docs: update the build table in
CLAUDE.mdand the toolchain reference.
Phase 5: options (optional)
-O0/-O2/-Osflag. Useset_optimization_levelfor-O0–-O3andadd_command_line_option("-Os")for size.- Debugging:
dump_reproducer_to_file(path)writes a C program that rebuilds the same context.set_dump_code_on_compile(true)prints the assembly.set_dump_initial_gimple(true)prints GCC’s view of your trees. Put these behind an env var such asPLENTY_GCC_DUMP.
Risks
| Risk | Impact | Mitigation |
|---|---|---|
GCC cannot always do a required tail call. On x86-64 SysV, a callee with more stack arguments than its caller fails. Cranelift’s Tail convention handles this; the C ABI does not. | Compile error for valid Plenty programs | Phase 0 test. If it fails: pass arguments through a pointer to a frame struct when they do not fit in registers, or turn tail calls into a loop with a dispatcher (trampoline) |
| Tail calls to functions with multiple returns. A struct return larger than 16 bytes goes through a hidden pointer, and GCC may refuse the tail call | Compile error | Phase 0 test. If it fails: pass the caller’s return pointer explicitly as a parameter |
__builtin_*_overflow does not work through get_builtin_function | No checked arithmetic | Use the typed builtins (__builtin_sadd_overflow, __builtin_saddl_overflow, __builtin_uaddll_overflow, …). For 8- and 16-bit types, compute in 32 bits and check the range |
Installed libgccjit is older than LIBGCCJIT_ABI_24 | No metadata alignment; gccjit::load fails | Check with objdump in Phase 0. Show the required tag in the load error |
| Debug-build panics on GCC warnings | Test crashes with a GCC message | Fix the code that causes the warning; catch_unwind at the backend boundary |
| Undefined behaviour in generated trees | Wrong code only at -O2 | Rules above; run the suite at -O0 and -O2 |
I128 ABI mismatch with the Rust runtime | Corrupt sum values | Phase 0 test with a runtime call that takes and returns u128 |
| Two backends drift | Test failures on one backend only | Phase 1 seam; run all AOT tests on both backends |
Effort
| Phase | Size | Note |
|---|---|---|
| 0 Spike | Small | A few evenings. It decides if the plan works |
| 1 Seam | Large | Touches all 2,840 backend lines; no new features |
| 2 Core | Medium | |
| 3 Full | Medium | Generators and metadata are the hard parts |
| 4 CLI/tests | Small | One env var covers the whole suite |
The total is about the same as writing the Cranelift backend again. Phase 1
returns some of that cost, because each later backend or Op is cheaper.
Open questions
- Do you want Phase 1 (shared lowering), or a separate copy of
Lowererfor an experiment? - Which license option (see “License”) do you want for GCC-enabled builds?
- Is cross-compilation a goal? If yes, the “one target per libgccjit” limit matters, and you need a separate libgccjit for each target.
- Should the GCC backend be in the default CI, or in an optional job only?
User-defined ordering
Status: Not implemented. Classes and enums have structural == and no
ordering. The current contracts are classes,
enums, tuple ordering,
list sorting and
std.collections. Remaining work belongs in
B37. Equality is shared with the sibling
hashing proposal.
Timsort is a separate algorithm and stability proposal. A comparison hook determines the order; it does not choose the sorting algorithm, require workspace allocation, or guarantee stability.
Problem
Ordering accepts integers, floats, str, and tuples of those. Every other type
is rejected: a < b on a class reports “operator < does not accept”, and
list.sort(), sorted, Heap[T], and the std.math helpers min and max
inherit that limit. The workarounds are per call site:
items.sort_by_key(key)with a key function returning an orderable tuple;HeapBy(before)with a stored comparator;- an ad hoc method such as
as_nanoseconds()instd.time, which says “Ordering has no operator”.
A key cannot be a reference, so a key holding a str field copies the string
on every comparison. A type that should have one natural order states it again
at every sort, heap, and comparison, and the statements can disagree. Classes
are not orderable tuple components, so (priority, task) cannot be ordered
when task is a class.
Only __init__, __del__, __enter__, and __exit__ are recognized hooks
today; any other dunder is an ordinary method.
Recommendation
A class or enum defines its order with one three-way method:
class Version:
major: i64
minor: i64
label: str
def __cmp__(self, other: &Version) -> Order:
order = self.major.compare(&other.major)
if order != Order.Equal:
return order
order = self.minor.compare(&other.minor)
if order != Order.Equal:
return order
self.label.compare(&other.label)
Order is a prelude enum with nullary variants Less, Equal, and Greater.
Every orderable built-in value gains x.compare(&y) -> Order, using the
existing total sorting order. A type with __cmp__ is orderable everywhere an
orderable built-in is: all six comparison operators, sort, sorted, keys of
sort_by_key, Heap[T], tuple components, and generic bodies that use <.
Rules
- The signature is exactly
def __cmp__(self, other: &T) -> Order, whereTis the declaring type (for a generic type, the same instance). It has no type parameters, defaults, orResult. Classes and enums may define it; anonymous sums may not. - All six operators derive from it.
a < bis__cmp__(&a, &b) == Less;a <= bis!= Greater;a > banda >= bare the mirrored tests;a == bis== Equal;a != bis!= Equal. Individual operators cannot be defined. - Defining
__cmp__replaces structural==and!=for that type. There is one source of truth for equality and order, and they cannot disagree. __cmp__is not called by name.a.compare(&b)is the spelling, for user and built-in types alike. This follows the existing rule that rejects explicit__init__/__del__calls.- Like
__del__, the hook belongs to the type: it applies wherever the type is visible, independent of the method’s own visibility. - The order must be total and consistent: antisymmetric, transitive, and
stable for unchanged values. The compiler cannot check this. A violation
gives an unspecified order, as an inconsistent
HeapBycomparator does today, but cannot break memory safety. - Its body may perform only computation, as the worker effect walk classifies it: no I/O, clock, entropy, or process effects.
Alternatives compared
| (a) six dunders | (b) one __cmp__ | (c) derived | (d) comparators only | |
|---|---|---|---|---|
| Methods written | Up to six, or a total_ordering rule | One | None | One per call site |
| Operators can disagree | Yes | No | No | Not applicable |
== consistent with order | By convention | By construction | By construction | No relation |
| Lexicographic composition | Two calls per field (a < b, b < a) | One call per field | Generated | Key tuples |
| Field order is semantic | No | No | Yes | No |
| New surface | Six hook names | One hook, Order, compare | Opt-in syntax or none | None |
(a) Python’s six rich comparisons (__eq__, __ne__, __lt__, __le__,
__gt__, __ge__). Python lets each disagree and returns NotImplemented for
mixed types; functools.total_ordering fills in the missing ones. A Plenty
version would need a rule for which subsets are legal, a consistency rule it
cannot check, and checker work for six signatures per type. Sorting needs only
<, but composing an order over fields needs equality as well, so a tuple
(priority, task) would call __lt__ twice per tied component. (CPython’s C
API combines these into one tp_richcompare slot with an operator argument,
which PyO3 exposes as __richcmp__; that is a dispatch detail, not a
Python-level method.)
(b) One three-way method, as in Rust’s Ord::cmp, Python 2’s __cmp__,
and C++’s operator<=>. One method answers every operator and composes
lexicographically in one call per component. The cost of deriving an operator
is one compare of the returned tag (see Costs).
(c) Derived structural ordering, lexicographic over fields in declaration
order like tuples. Automatic ordering would make every class orderable,
including types where order is meaningless (instants from different clocks,
handles), and would make reordering fields a silent behavior change. Opt-in
derivation needs attribute or decorator syntax that Plenty does not have, and
B45 already declines automatic derive for formatting.
Defer it; a later derive, if any, generates an ordinary __cmp__.
(d) Comparators only (status quo). It keeps the language unchanged, but
leaves <, min, max, Heap[T], and tuple components unavailable, repeats
the order at each use, and makes str keys copy per comparison. Keep
sort_by_key and HeapBy for orders other than the natural one.
Total and partial order
Plenty has one total order for sorting and tuple operators, and IEEE behavior
for scalar float operators. Rust splits PartialOrd from Ord so that floats
can have operators but not sort; learners then meet total_cmp anyway.
Plenty should not add that split. __cmp__ returns one of three answers, so
every user order is total. x.compare(&y) on floats uses the sorting order:
-0.0 before 0.0, every NaN equal to every other NaN and after all numbers.
A class wrapping an f64 and comparing with compare therefore has
NaN == NaN under its own ==, matching its order. That differs from scalar
f64 ==, deliberately: the class’s equality follows its declared order.
There is no Unordered variant. A type with values that genuinely do not
compare (instants from different clocks) should not define __cmp__.
Equality
Classes and enums currently use structural equality, which the
hashing proposal also depends on: a hash must agree
with ==. Rule 3 makes __cmp__ the definition of equality for an ordered
type. The hashing proposal admits only structural equality and hashing, so a
type that defines __cmp__ is not a key type: its == is user code that a
structural hash cannot follow. To key a dictionary by such a value, key on a
projection of the fields __cmp__ compares. A type without __cmp__ keeps
structural == unchanged and stays eligible as a key.
Making __cmp__ define == also lifts a limit: structural equality rejects
recursive data, but a written __cmp__ may recurse explicitly through
Box[T] fields, giving such types both equality and order.
Use sites
Operators and generic code. Generic bodies are checked per concrete
instance (protocols), so
std.math.min, max, std.test.less, and Heap[T] accept ordered user types
without signature changes. min_all/max_all still require copyable elements,
and class instances are affine unless declared @copy, so they remain
unavailable for other classes.
Protocols. No Ordered protocol is needed for generic bodies. Protocols
constrain only classes, enums, and bytes, so a protocol could not admit i64
today. A user protocol may still require __cmp__ to restrict a parameter to
ordered user types; whether built-in orderable types should satisfy such a
protocol is an open question.
Sorting. items.sort() and sorted(items) accept element types with
__cmp__; sort_by_key accepts keys of such types. The sort itself allocates
nothing; allocation inside __cmp__ is the type’s own, as for key callbacks.
Tuples. A tuple whose components are orderable, including ordered user
types, is orderable. Heap[(i64, Task)] orders by priority, then by Task.
Dictionaries and sets. Unchanged here; see the hashing proposal.
Ownership
Both operands are shared borrows: self and other: &T. A comparison never
moves, copies, or drops an operand. a < b on owned values borrows both for
the call, as tuple comparison observes them today; comparing &T values passes
the references through. Because the receiver is shared, __cmp__ cannot
mutate either value, so a sort cannot change the elements it is ordering. The
list remains exclusively borrowed by sort; the hook receives element loans
derived from that borrow, as sort_by_key callbacks do.
Errors, allocation, and workers
__cmp__ cannot fail: it returns Order, not Result, so ? is unavailable
inside it. Allocation in Plenty is always fallible and visible as a Result,
so a comparison that allocates must handle the failure locally. Traps
(overflow, indexing) remain traps and abort the sort like any other trap.
Rule 7 restricts the body to computation. This keeps comparisons pure for
learners, lets sort keep its documented contract apart from the hook’s own
allocation, and makes every ordered type usable from workers. It reuses the
existing closed-world walk in src/effects.rs and threading::check, which
already certifies every reachable body; no new eligibility mechanism is needed.
Whether to also forbid allocation is an open question.
Costs
Measured today
A probe on this branch (--release, Cranelift 0.131, 1,000,000 pairs, single
core, median of seven runs, construction subtracted; checksums equal):
| Sort | Path | Time |
|---|---|---|
list[i64].sort() | Runtime, Rust sort_unstable_by over scalar rows | 78 ms |
list[(i64, i64)].sort() | Runtime heapsort, descriptor-interpreted compare | 1,444 ms |
list[Point].sort_by_key(key) | Plenty-source heapsort, runtime OrderCompare per comparison | 4,142 ms |
list[Point] heapsort in Plenty, direct cmp(&Point, &Point) -> i64 | Models the hook lowered through a generated sort | 2,473 ms |
perf attributes the last row’s time to runtime calls for element access and
row exchange (plenty_collection 48%, Payload::relocate 15%,
plenty_retain/plenty_release 19%, collection_at 7%). The comparison
function itself is 4%; replacing the call with inline field compares did not
improve the time. The tuple sort spends 27% in the runtime’s interpretive
compare (plenty-runtime/src/aggregates.rs), which dispatches on the
descriptor kind and recurses per field on every comparison. Sizes: the direct
comparison function is 400 bytes, a specialized Plenty heapsort 2.8 KB, and a
sort_by_key instantiation 3.7 KB.
The comparison call is not the bottleneck; how the sort reaches the elements is.
Issue #331 later moved sort_by_key into the runtime: it sorts rows with the
wide-row introsort and calls a generated native comparator, with no
per-access runtime call or reference count change. Natural tuple sorts resolve
field offsets once per sort, and moved flat records repair their address
without a descriptor walk. On a contended host (median of three paired runs,
1,000,000 elements, sort time only) list[(i64, i64)].sort() went from about
725 ms to 265 ms and list[Point].sort_by_key with an i64 key from about
3,090 ms to 330 ms, against 74 ms for list[i64].sort(). Option 1 below can
therefore reuse that row path.
Dispatch
The hook is resolved statically: a < b lowers to a direct call of the
class’s mangled method, as with lowers to Op::Call of __enter__ in
src/frontend/contexts.rs. There is no vtable or runtime lookup. Plenty does
not inline calls. Cranelift 0.131 offers Context::inline with an
embedder-supplied Inline policy, but the compiler does not use it, so every
user comparison costs one call. A probe returning a three-variant nullary enum
compiled to a register return with no runtime call; a less wrapper around it
was 39 bytes. Deriving < from __cmp__ therefore costs one integer compare
on the returned tag, versus a direct __lt__: no measurable difference, and
lexicographic composition saves a second call per tied component.
Compared with built-ins: integer < is one instruction; str and tuple <
already call the runtime (TextCompare through plenty_collection, and
OrderCompare through the narrow plenty_order_compare). A user __cmp__ over integer
fields is a direct call with field loads, and is expected to be no slower than
tuple <; measure it (below).
Sort path
Two lowerings are possible for list.sort() on user elements:
- Runtime sort with a descriptor hook. Runtime type descriptors already
carry one callback into generated code: the class
__del__adapter (Type::dropinplenty-runtime/src/aggregates.rs, emitted bysrc/codegen/metadata.rs). Acompareslot of the same kind lets the runtime’scomparecall__cmp__for a record that has one. This makessort, tuple operators with user components, andOrderComparekeys work with one runtime change, keeps the runtime’s row exchange, and adds one indirect call per comparison. It grows every descriptor from 64 to 72 bytes of static data, unless the slot replaces padding in a reorganized layout. - Generated sort, specialized per element type as
sort_by_keywas before issue #331. It calls the hook directly, but the probe shows such a sort is 1.7 times slower than the runtime tuple sort because each step reaches elements through runtime calls, and it adds about 3 KB per element type.
Recommend option 1 for sort, sorted, tuple components, and keys, and
direct generated calls for operators in ordinary and generic code.
Compile time
Detection is one method-name lookup per class or enum and one signature check,
done once per type; operator typing gains one branch in is_orderable. Each
instantiation of a generic body using < on a new type is already a separate
specialization, so ordered user types add no new instantiation mechanism. The
probe programs checked in about 2 ms each, too small to separate the cost.
Measurements required before shipping
- Extend
examples/collection_bench.pywith sorts oflist[(i64, i64)],list[Point]by__cmp__(both lowerings above),list[Point]bysort_by_key, and astr-field class, at 100,000 and 1,000,000 elements, recording binary size per variant. - A
Heap[Task]push/pop benchmark againstHeapBywith the same order. compile_bench--workload checkand--workload aot(see profiling) before and after, with a generated workload of many ordered classes compared and sorted, to confirm detection and specialization cost stays in the noise.- Object-size check of the descriptor change across the test suite’s executables.
Naming
std.atomic already exports Ordering for memory ordering. It is not in the
prelude, but from std.atomic import Ordering would collide with a prelude
Ordering. Use Order for comparisons: short, distinct, and read naturally
as Order.Less. The hook name __cmp__ follows Python 2 and matches the
three-way shape; __order__ is the alternative if the Python 2 association is
unwanted.
Order provides reverse(self) -> Order for descending orders and may provide
then(self, next: Order) -> Order for short compositions. then evaluates its
argument eagerly, so early returns stay the idiom for expensive fields.
Rejected
- Six rich-comparison dunders: allows inconsistent operators and doubles calls in lexicographic composition.
PartialOrd/Ordsplit: two concepts for learners; Plenty already chose one total order for sorting.- Automatic structural ordering: makes field order semantic and gives order to types where it is meaningless.
- Keeping structural
==beside__cmp__: two definitions of equality that can disagree, breaking sorting and hashing invariants. - Fallible comparisons (
-> Result[Order, E]): every operator and sort would become fallible; a failing comparison has no sound place in<. - A runtime callback for every comparison in generic code: the static call is cheaper and already available.
Open questions
- Should
__cmp__also be forbidden from allocating, beyond rule 7? - Should built-in orderable types satisfy a protocol that requires
__cmp__? That needs methods oni64and tuples visible to protocol matching. - Should
Orderandcomparebe prelude names, or imported from a small module? The hook’s signature mentionsOrder, which argues for the prelude. - Does
thenearn its place, or should composition use early returns only? - Should nullary enums get a declared-order shortcut, or keep writing
__cmp__over a rank method? - Coordination with the hashing proposal: one
equality hook, and the required agreement between hash and
__cmp__.
Staged implementation
- Hook and operators. Recognize
__cmp__beside__init__/__del__(src/frontend/classes.rs,src/frontend/enums.rs), validate its signature, and reject direct calls. AddOrderto the prelude (src/frontend/enums.rs,src/frontend.rs) andcomparefor built-in orderable types. ExtendTy::is_orderable(src/op.rs) and binary operator typing (src/frontend.rs, near “operator{op}does not accept”) so comparisons on such types lower to the hook call and a tag test; route==/!=there too instead of structural equality (lower_equalityinsrc/codegen.rs). Enforce rule 7 throughsrc/effects.rs. Generic users (std.math,std.test,Heap[T]) then work unchanged. - Runtime ordering. Add the descriptor
compareslot (plenty-runtime/src/aggregates.rs,src/codegen/metadata.rs) with an adapter like the__del__hook, sosort,sorted,OrderComparekeys, and tuple components with user types use it (src/frontend/sorting.rs,int_cmpinsrc/codegen.rs). - Documentation and lessons. Update the class, enum, tuple, list, and
std.collectionsreference pages,std.time(which can then defineDurationordering), the priority-queue lesson, and B37.
Each stage ships with diagnostics tests for invalid signatures, direct calls, effectful bodies, and inconsistent orders that stay memory safe, plus the measurements above.
Timsort for stable, adaptive sorting
Status: Not implemented. This proposes an algorithm and an API direction, not an approved migration. List sorting and the key-sorting tutorial define today’s contracts. No implementation or performance improvement ships with this proposal. Scheduling belongs to B37. Issue #331’s sorting performance fix shipped allocation-free introsort with a heapsort fallback; it did not implement stability or Timsort.
Why consider it
Stable sorting preserves the input order of equal keys. Records already ordered by time can then be sorted by category without losing their time order within each category. This should be easy to request without adding an original-position tie breaker or constructing a separate decorated list. Exploiting existing order also suits logs, grouped records, and incrementally updated data.
Today sort() is infallible and allocation-free. The runtime uses Rust’s
unstable slice sort for rows without inline payloads and three-way introsort (with a heapsort
fallback) for wider inline rows
(plenty-runtime/src/entries.rs). sort_by_key uses the same introsort for every
row width, calling a generated comparator (src/codegen/sorting.rs) that
recomputes both keys per comparison. sorted
explicitly copies, then invokes the same natural-order sort. Keys cannot return
references or Result, and user-defined natural order is still a
separate proposal.
Issue #331 identified runtime element access, row movement, and descriptor comparison costs; the runtime-driven key sort and per-sort tuple and relocation plans removed most of them. Changing algorithms alone does not remove such costs. Timsort is a candidate for stable sorting and locality improvements, not evidence that it will beat the current scalar path or fix that issue by itself.
Algorithm recommendation
Prototype Timsort: detect existing ordered runs, extend short runs with stable binary insertion, and merge adjacent runs using bounded temporary storage. Galloping searches can reduce comparisons when one run repeatedly wins. CPython’s versioned design notes describe these mechanisms and their memory tradeoff.
Specify the merge scheduler separately. Classic Timsort uses run-length stack invariants; modern CPython uses Powersort scheduling within its adaptive sort. Compare a corrected classic policy with Powersort before selecting Plenty’s implementation. Munro and Wild’s paper gives Powersort a nearly optimal merge-cost bound relative to the existing runs. That bound concerns merge work, not guaranteed wall-clock optimality. If selected, describe Plenty’s implementation as Timsort-style run handling with Powersort scheduling, rather than claiming it uses the classic Timsort policy.
The required contract is stable ordering, O(n log n) worst-case comparisons, O(n) comparisons for a single ordered run, and documented workspace bounds. Constants and insertion/galloping thresholds are implementation details. A simple first version should reverse only strictly descending runs: reversing equal elements would destroy stability. For descending output, reverse the comparison direction, not the final ascending array. Forward merges take from the left on equality; backward merges take from the right.
Surface and compatibility
The recommended eventual teaching surface is items.sort()? and
items.sort_by_key(key)?: stable order with explicit allocation failure. Keep
infallible, allocation-free operations under explicit sort_unstable and
sort_unstable_by_key names. These are proposed spellings. Stability is a useful
default, but it does not make an allocating algorithm optimal for every purpose.
Changing existing return types is a source compatibility change. Algorithm
adoption does not authorize it. A compatible first release could introduce
sort_stable and sort_stable_by_key, each returning Result[(), AllocError],
while retaining the existing methods unchanged. A later naming decision must
include migration diagnostics, reference/tutorial updates, and call-site changes.
sorted already returns AllocError, so a stable implementation can preserve
its signature and source-preservation guarantee, while documenting extra memory.
Other choices have different costs:
| Choice | Consequence |
|---|---|
| Keep stable methods permanently explicit | Preserves compatibility and makes scratch allocation visible; ordinary key sorting still loses ties unless the caller chooses stability. |
| Try scratch, fall back to allocation-free unstable sorting on allocation failure | Can preserve an infallible return type, but introduces hidden allocation and cannot promise stability. This does not meet the proposed stable contract. |
| Fall back to a stable in-place merge | Can preserve stability and avoid returning allocation failure, but simple rotation-based merges can raise worst-case work to O(n log² n); sophisticated bounded-space algorithms add substantial complexity. Attempting allocation still changes today’s allocation-free promise. |
| Caller-owned reusable workspace | Makes repeated allocation avoidable; useful later, but capacity/type/lifetime management should not be required for the ordinary call. |
Prefer a fallible stable operation over either fallback for the first implementation. Preserve an explicit allocation-free route, and decide the default names separately after measurements and compatibility review.
Storage, failure, and callbacks
For a straightforward row-merging implementation, reserve at most floor(n/2)
whole rows before any mutation. Count bytes using the runtime row stride,
including inline payloads, not Python’s pointer-sized estimates. Check all size
arithmetic. Use a run stack whose capacity is justified for the chosen policy
and supported length range. Allocation or size failure returns AllocError
with the original list unchanged. Empty/singleton lists require no workspace;
a read-only single-run check can avoid reserving for already ordered input.
Pre-reservation deliberately trades potentially unused memory for a simple
failure guarantee; adaptive buffer growth would need a weaker guarantee or
rollback and is not the initial recommendation.
Moving a row transfers ownership; it must not clone, drop, or retain each element on every merge step. Relocate inline addresses before any comparison observes the row, including rows held in scratch storage. Every original value must have exactly one live owner and remain available for destruction exactly once. Existing swap-based safety does not automatically establish merge safety. A narrow audited runtime operation is preferable to exposing unchecked moves to Plenty programs. Keep safe orchestration in Plenty where it is efficient; measure the boundary costs identified in issue #331.
Initially retain current key rules: shared &T input, reusable shared-call
callback, direct orderable result, consistent keys, and no concurrent access to
the exclusively borrowed list. Recompute keys; do not promise a call count or
evaluation order. Preserve signed-zero, NaN, string, and tuple total ordering.
Key callbacks remain responsible for explicit allocations inside themselves.
Caching keys once is a separate API decision with additional owned storage and
observable callback semantics; fallible keys need their own cleanup/error design.
Inconsistent keys must never cause memory unsafety, ownership loss, or an unbounded loop; sortedness is only guaranteed for a consistent order. Internal index/stack checks must not rely on comparator consistency. Plenty fatal traps do not unwind, so this proposal promises neither rollback nor recovery after a callback trap. Ordinary allocation failure occurs before rearrangement.
Evidence required for adoption
The verification work by de Gouw and de Boer found errors in widely used Timsort implementations. Review stack invariants, capacity proofs, overflow checks, and merge termination explicitly; copying a familiar implementation is insufficient. Test adversarial run lengths as well as random permutations. Include equal-key identity tags in both directions, insertion and galloping boundaries, empty/singleton inputs, NaNs/signed zeros, wide inline records, affine owners/destructor counts, and injected allocation failures that verify unchanged input. Check callback loan rejection and inconsistent-callback memory safety. Differential tests must compare both sortedness and stable identity order, not just values.
Benchmark scalar, string, tuple, and keyed-record inputs: random, sorted, reverse, duplicate-heavy, almost sorted, and uneven runs. Record comparison and key-call counts, moved bytes, peak scratch, allocations, and elapsed time; retain the current sort as the baseline. Recreate issue #331’s workloads and use the collection benchmarks without treating old measurements as new evidence. Compare classic Timsort, Powersort scheduling, a simple stable mergesort, and the existing allocation-free implementation. A different unstable runtime sort could independently address #331; record-layout work in issue #330 is separate. Publish regressions as well as wins before choosing defaults. Any implementation PR must update contracts and executable lessons together; this proposal alone changes neither.
User-defined dictionary keys and set members
Status: Structural-key policy implemented. Named enums whose payloads are supported key
types now work as dictionary keys and set members, including nested enums.
Classes whose fields are supported keys, including move-only classes and enums
containing them, and the KeyType constraint are also implemented. Anonymous sums with supported key members are implemented. Custom-cleanup
keys are admitted for borrowing and ownership transfer; copying operations reject them.
Named enum and tuple sets with supported finite domains through 256 values now use
bitsets, including
boolean and nested enum/tuple payloads. The
collection reference owns the
current contract. The design and historical analysis below describe the broader
proposal; enum/class admission and the explicit generic constraint have shipped.
Generic bodies remain checked per concrete instance: unconstrained parameters
are not rejected solely for appearing in a collection signature. The original
eager-rejection analysis below is historical, not current behavior.
Problem
Before structural key admission, a program that wanted dict[Point, V],
set[Color] or dict[UserId, V] had to flatten keys into tuples, losing their
nominal types. Eligible classes and enums now work directly. Generic code can
state its key requirement explicitly:
def count[K: KeyType](xs: &list[K]) -> Result[dict[K, i64], AllocError]:
...
The bound rejects unsupported arguments at instantiation. Unconstrained generic functions continue to work when their concrete keys are valid; requiring a bound on every collection signature is not part of the implemented contract.
The suggested direction is a __hash__ dunder and a protocol requiring it as
the key bound. This page assesses that idea against the current implementation,
then recommends a narrower design.
How dictionaries and sets work today
All dictionary and set operations enter one runtime function,
plenty_collection(op, args, descriptor, site)
(plenty-runtime/src/aggregates.rs:1423), with an opcode and the collection’s
static type descriptor. Generated code never hashes or probes.
| Aspect | Current behavior |
|---|---|
| Descriptor | Type (aggregates.rs:36), 64 bytes, asserted layout: kind byte, affine, reflexive, inline payload size, key/value descriptors, name, variants (fields), and a drop function pointer at byte 56. Emitted by src/codegen/metadata.rs. |
| Hash | aggregates/hashing.rs feeds the complete descriptor-guided key into SipHash-2-4: scalar words, length-framed strings, and kind B fields framed by variant tag and field count. Classes now have a field-count-framed structural branch as well. |
| Seeding | One secret 128-bit OS key per process, initialized on the first hash and shared by workers. Entropy failure traps; no fixed-seed production mode. |
| Table | Vec<usize> of entry indices plus one, linear probing, power-of-two size, load at most one half (try_reserve, aggregates.rs:546). No hash is cached; rehashing and removal (aggregates.rs:699, backward-shift repair) recompute the home bucket of every key they visit. |
| Entries | Dense, type-sized rows (plenty-runtime/src/entries.rs). Inline payloads such as multi-field tuples live in the row. Iteration walks rows, not the bucket table, so dictionary order is insertion order regardless of hash values. |
| Equality | equal(a, b, ty) (aggregates.rs:742), structural, also used by == on classes, enums and collections (src/codegen.rs:2481 lowers it to collection op 8). Floats compare by IEEE rules. |
| Ownership | try_insert retains copyable operands and adopts affine ones. Lookups take a value or reference. Borrowed iteration yields shared &K; mutable items() lends &mut V with shared &K (iterables). No operation lends &mut K. |
| Admission | Ty::hashable (src/collection.rs:189): integers, bool, str, byte strings, and tuples of these. Checked where collection types resolve (src/frontend.rs:885) and where literals infer (src/frontend/collections.rs:891, :3138). |
| Callbacks | The runtime already calls generated code: a class’s __del__ adapter through the descriptor’s drop slot, and executor/thread adapters with a three-pointer signature (src/codegen/executors.rs). Sorting has no user comparator; compare (aggregates.rs:808) is descriptor-interpreted like hash. |
The runtime relies on two properties of every legal key, recorded in
try_reserve: hashing never allocates and never calls user code. Table
rebuilding computes hashes alone, in the middle of a reservation, with no error
channel.
Issue #329 addressed two hash-quality defects independently of this proposal:
- The former FNV-1a path had no finalizer. Multiplication carries bits only upward, so the low
k bits of a string’s hash depend only on the low k bits of each byte. A
16-bucket table (the minimum) puts
"a","q","A","Q","1"and"!"in one bucket. Tables below 256 buckets are affected. - It had no seed: colliding inputs could be precomputed.
std.tomlindexes parsed keys indict[(i64, str), i64](stdlib/std/toml.plenty:245), so untrusted TOML can force quadratic parsing. The dictionary reference already says adversarial collisions can make lookup linear.
The replacement keys the entire structured input, rather than applying a secret finalizer to an unkeyed prehash. It covers TOML’s index and duplicate-key set, and JSON’s duplicate-member set. HTTP and CSV currently use no dictionary/set tables in their implementations.
Historical cost of a composite key before keyed hashing
A probe on 2026-10-19 (release build of this branch, Linux x86-64, one pinned core; the machine had load average about 18, so treat these as indicative) ran 20 million membership tests, half hits, minimum of seven runs. These results predate issue #329’s keyed hashing and do not describe its current cost:
| Entries | Empty loop | dict[i64, i64] | dict[(i64, i64), i64] |
|---|---|---|---|
| 1,024 | 0.03 s | 0.39 s (about 18 ns/test) | 0.92 s (about 45 ns/test) |
| 65,536 | 0.04 s | 0.70 s (about 33 ns/test) | 1.53 s (about 75 ns/test) |
A two-field tuple key costs about 2.4 times an integer key, including building the probe tuple. Any user type admitted through the same descriptor path should cost about the same as the tuple with the same fields. This must be confirmed by the benchmark in Measurements.
Precedents
Python. __hash__ returns an integer and must agree with __eq__:
a == b implies hash(a) == hash(b). Nothing enforces it. Defining __eq__
without __hash__ makes a class unhashable; mutable built-ins are unhashable by
convention. A key mutated after insertion silently becomes unreachable.
Dataclasses derive both from fields (frozen=True, eq=True).
Rust. Hash and Eq are separate traits, usually derived together. A type
does not return a hash; it feeds its fields to a Hasher
(fn hash<H: Hasher>(&self, state: &mut H)). The collection chooses the hasher,
so seeding (SipHash with a random key in std), algorithm changes and mixing
quality stay out of user code. Consistency with Eq is a documented logic
requirement, not checked; violations cause wrong results but no memory
unsafety. Interior mutability makes key mutation possible (Cell in a key),
and is documented as a logic error.
| Shape | Distribution and seeding | User can break a == b ⇒ hash(a) == hash(b) |
|---|---|---|
__hash__(self) -> u64 | User mixes bits; no seed; poor combinations (x ^ y) are easy | Yes, by hashing anything not determined by equality |
__hash__(self, h: &mut Hasher) | Collection owns mixing and seed | Only by feeding non-structural data or by custom equality |
| Derived from fields | Runtime owns everything | No |
Assessing the __hash__ protocol idea
The idea needs several things Plenty lacks, and most of them work against the project’s goals.
- A protocol cannot express the bound. Protocols match ordinary methods
(protocols). Integers,
bool, andstrhave no methods, sodict[i64, V]would not satisfy a method protocol unless the compiler special-cased it. Every user key type would also need a hand-written method even when the derived answer is the only correct one. A bound that admits built-ins and derived types is a built-in constraint likeIntType, not a protocol. - Custom hash without custom equality is nearly useless. With structural,
non-overridable equality, any pure function of the fields is a consistent
hash. The only gain from a custom one is speed (hash an
idfield and skip the rest), at the price of a callback per hash. The cases people actually write__hash__for (case-insensitive names, ignoring a cache field) change equality. - Custom equality is a larger change than hashing.
==would stop being structural for that type,equalin the runtime would need a callback, and collection equality,in, set algebra and dictionary comparison would all run user code. The classes reference promises equality “neither invokes user-defined magic methods”. Allowing__hash__and__eq__separately also imports Python’s unenforced consistency rule, which is exactly the kind of soft rule this language avoids. - Callbacks break runtime invariants. A user hash would run inside
try_reserve’s rebuild, inside removal’s cluster repair (once per displaced entry), and during set algebra. It would need a no-allocation, no-trap-on-error contract, an effect restriction, and a new descriptor slot. See Where hashing runs.
The idea is right that keys need a named, generic bound. It does not need a dunder.
Where hashing runs
| A. Descriptor-interpreted (today) | B. Runtime calls user hook | C. Generated code passes the hash in | D. Monomorphized table | |
|---|---|---|---|---|
| Mechanism | Extend hash/equal to classes | Descriptor gains hash (and equality) function pointers, called like drop | Each call site computes the probe hash and passes it to plenty_collection | Dict/set code generated per key type, in Plenty or codegen |
| Per lookup | Same path as tuples | Plus one indirect call and an adapter per hash and per probe equality | Saves descriptor walk for the probe key only | Inlined hash and equality, no dispatch |
| Rehash, removal | Recompute by interpretation | One callback per visited entry | Needs cached per-entry hashes, since the runtime cannot hash stored keys | Inlined |
| Memory per entry | Unchanged | Unchanged | Plus 8 bytes per row or per bucket | Unchanged or less |
| Code size, compile time | One descriptor per key type, already emitted | One adapter function per key type | Hash code at every call site or one helper per type | Whole table per instantiation; large |
| Runtime invariants | Kept: no allocation, no user code | Broken; needs effect and allocation rules | Kept for hashing; equality still interpreted | Must re-implement reservation, failure, destruction-resume and removal semantics in generated code |
Consistency with == | By construction: one equal | Only if equality is also the hook’s | By construction if derived | By construction if derived |
Measured context: in-cache integer lookups cost about 18 ns, almost all of it
call, dispatch, and probing rather than hashing an integer. A callback adds an
indirect call, an adapter that unpacks the slot, and possibly a Hasher object;
expect a few nanoseconds per hash and per equality probe, on top of the
tuple-equivalent cost. That estimate needs the measurement below before B is
considered.
D gives the best per-operation speed and the worst compile time, code size, and
correctness surface. It would duplicate aggregates.rs’s allocation-failure
and destruction-order guarantees per type. It is the right tool only if the
dispatcher itself, not hashing, dominates. Issue #18 gave list reads narrow
runtime helpers for that reason; the same approach could apply to hashed
lookups for all key types at once.
Recommendation
Derived structural keys, a built-in KeyType constraint, and no user hash
or equality hooks. The rule has no exceptions and cannot be violated:
A key type is an integer,
bool,str, byte string, or a tuple, class, or enum whose every field or payload is a key type. Recursive types are not key types. Dictionary keys, set members, andKeyTypearguments must be key types. Hashing and equality are structural, soa == balways implies equal hashes.
class Point:
x: i64
y: i64
enum Shape:
Dot(Point)
Segment(Point, Point)
def count[K: KeyType](xs: &list[K]) -> Result[dict[K, i64], AllocError]:
...
seen: set[Shape] = set[Shape]()? # accepted
weights: dict[Point, f64] = {}? # accepted: values may be floats
bad: set[Sample] = set[Sample]()? # rejected if Sample has an f64 field
Rationale
- Consistency is impossible to break. One structural
equalserves==and lookup; one structuralhashreads the same fields. There is no__hash__to forget, no__eq__to disagree with it. - No new runtime invariant. Hashing still never allocates or calls user
code, so
try_reserve, removal repair, and worker-thread eligibility need no new rules. Effects areComputeonly by construction (src/effects.rs). - Small change. The runtime already hashes kind
Bstructurally; it needs a class branch. The frontend needs one predicate, one constraint, and better diagnostics. Existing keys keep identical code paths. - Custom identity stays explicit. A case-insensitive key is a key built
from the normalized text (
Name(text.lower()?)), not a hidden equality. The allocation is visible at construction, not repeated per probe.
Key immutability
Keys cannot change while stored, by existing rules plus the key-type rule:
- Insertion moves an affine key in or copies a copyable one.
- Every read of a stored key is a shared
&K; no API lends&mut K, including mutableitems()views andreplace(d[k], v), which targets values. - Key types contain no interior mutability. Excluded by the rule, without a
separate list: floats (
NaN != NaN, and0.0 == -0.0with different bits), references,Boxand recursive data, collections, closures and callables, generators, atomics (std.atomic),std.sync.Mutex, channels,CancellationToken, tasks, files, sockets and other handles. A class holding any of these is not a key type. - Strings share immutable storage, so retaining a
strkey is safe.
Affine keys
Every class not declared @copy is affine
(moves and copies),
so class keys are the first affine keys. Operations that today retain keys into
new storage need the same treatment affine values already have:
| Operation | Affine key rule |
|---|---|
insert, add, update | Already adopt affine operands |
get, [], in, pop, discard | Accept &K; an owned temporary is consumed |
Borrowed iteration, items() | Already yield &K |
keys() | Shared KeyView[K], iterating &K, without copying |
keys_snapshot(), into_keys() | Transfer affine keys from an owned temporary; use copy(d)?.keys_snapshot() to preserve the source |
union, intersection, difference, symmetric_difference, key-view to_set() | Copy members into independent result storage; reject keys containing custom cleanup; sources remain unchanged on success or failure |
| Set and dict comprehensions | Move keys in as for insert |
Copyable enums (payloads of copyable key types) avoid this table entirely, which motivates the staging below.
Generic bound
KeyType is a built-in constraint beside IntType and FloatType
(src/frontend/generics.rs). It accepts exactly the current collection key
predicate, on functions, methods, classes and enums. An unsuitable argument
reports TYPE does not satisfy KeyType, even if the generic body does not use
hashing. Unconstrained generic bodies remain checked per concrete instance;
dict[K, V] and set[K] do not independently require an explicit bound.
Diagnostics now name the first unsupported component in declaration order, including class field names, enum variants and payload positions, and tuple element positions. The path ends with the rejected type or a specific reason such as an unsupported anonymous-sum member; see key diagnostics.
DoS resistance and seeding
This runtime concern was implemented independently of user key admission in issue #329. SipHash-2-4 consumes all structural bytes with one OS-generated key per process. Adding a finalizer alone would improve low-bit distribution but would not prevent precomputed full-hash collisions. The key is initialized lazily and atomically; interrupted entropy reads retry, short reads accumulate, and other failures trap rather than weaken the key. Constructing empty collections does not require entropy. See the collection contract.
Dictionary iteration stays deterministic through its entry order, so tests need no public seed override. Set order remains unspecified. Keyed hashing does not remove the table’s theoretical linear-probe worst case.
The pinned siphasher 1.0.4 backend uses the original SipHash-2-4 schedule:
two compression rounds and four finalization rounds. SipHash-1-3 uses fewer
rounds to reduce hashing cost; this change chooses the original schedule as
a conservative security margin. The collection benchmarks quantify the cost:
short-key lookup is more expensive, especially for integers. Hashing remains
a runtime policy, so callers need no new API or manual mixing.
Superseded 2026-11-05 (issue #747): the runtime now uses SipHash-1-3, the schedule of Rust’s
HashMap, CPython and Swift, and hashes top-levelstr,bytes, integer andboolkeys directly without the descriptor walk. Nested keys keep the framed encoding. See the collection contract.
Costs
| Cost | Recommended design |
|---|---|
| Lookup, user key | Same descriptor path as a tuple with the same fields; about 45 ns versus 18 ns for i64 in the probe above |
| Existing keys | SipHash-2-4 adds keyed mixing cost; see the collection benchmark for measured deltas |
| Per field | One recursive structural hash/equality walk. Record::slot recomputes each offset from the start, so hashing or comparing an n-field record is O(n²) descriptor reads; walking offsets incrementally fixes it for tuples and classes alike |
| Memory per entry | Unchanged: 8-byte bucket at load at most one half, plus a row sized from the key’s inline payload, as for tuples |
| Code size, compile time | One static descriptor per key type, which collection types already emit; no per-type functions |
| Rehash | Every resize recomputes all hashes by interpretation; composite keys pay more than integers |
Caching hashes is not part of the recommendation. It would cut rehash and removal cost and give a cheap pre-equality filter, for 8 bytes per bucket (doubling the table) or per row. Measure before adding it.
Measurements
Before implementation and before any follow-up optimization:
- Extend
examples/collection_bench.py(method inexamples/collection-bench.md) with insert, hit, miss and remove workloads at 1,024 and 65,536 entries fordict[i64, i64],dict[(i64, i64), i64],dict[Point, i64](twoi64fields), a copyable enum key, anddict[str, i64]with short keys. Pin to physical cores as in profiling. - Rerun against keyed hashing; report per-operation deltas for existing key types.
- To evaluate design B at all: prototype a descriptor hook that returns the structural hash through an adapter and compare against A on the same workloads.
- Compile time:
compile_bench -- 1000 10 --workload aotbefore and after, per profiling; expected neutral.
Rejected alternatives
__hash__(self) -> u64. User mixing, no seeding, and consistency is unenforced. Adds a callback for no correctness gain under structural equality.__hash__(self, h: &mut Hasher). Fixes distribution and seeding, but the only benefit without custom equality is skipping fields, paid for with a callback per hash and new runtime invariants.__eq__plus__hash__. Makes==non-structural, puts user code inside every collection algorithm, and reintroduces a consistency rule the compiler cannot check.- A method protocol as the bound. Cannot admit built-ins without a special case and forces boilerplate on every key type.
- Monomorphized tables (D). Compile time, code size, and duplicated failure semantics, against fast compilation. Revisit only if dispatch, not hashing, dominates measured cost, and then prefer narrow runtime helpers.
- Hashing floats by bits. Breaks
a == b ⇒ hash(a) == hash(b)for0.0/-0.0and makesNaNkeys unreachable. A user who needs float keys can store a canonical integer encoding explicitly.
Possible later extension: key projection
If custom identity proves necessary, the consistent form is one method that defines both equality and hashing by returning a key-type value:
class Name:
text: str
def __key__(self) -> str:
... # normalized text
==, hash, in and lookup would all compare __key__() results, so they
cannot disagree. It still needs a runtime callback (design B), a
no-allocation rule on the projection (normalizing text usually allocates, which
defeats the motivating example), and a change to the structural-equality
promise. Explicit key construction covers the same cases today; this is not
recommended now.
Ordering and sorted maps
Out of scope. User-defined ordering faces the same
choice for < and sort: compare (aggregates.rs:808) is descriptor-
interpreted like hash, and the same derived, structural rule keeps ordering
consistent with equality. A sorted map would need both bounds; if ordering is
derived structurally, every key type with no float fields is also totally
ordered, and the two constraints could share one admission walk.
That proposal recommends a __cmp__ hook that also defines ==. A type that
defines __cmp__ is therefore not a key type: its equality is user code, so a
structural hash would not agree with it. Such values are keyed by a projection
of the fields their __cmp__ compares.
Open questions
- Name:
KeyType,Hashable, orKey.KeyTypematchesIntType. - Resolved: classes with
__del__, including nested cleanup, are keys when their fields are keys. Hashability does not imply duplicability. Non-mutating set algebra and key-viewto_set()require explicit-copy eligibility, while insertion, borrowed views, relationships, in-place updates and owned extraction work without copying. Equal incoming keys are destroyed; existing key owners remain in the collection. No user-defined clone hook is added. - Private fields: structural hashing reads them, as equality already does. Confirm that is acceptable for public key types across modules.
- Resolved: anonymous sums are key types when every member is.
- Whether to cache hashes, after measurement.
Implementation sketch
- Copyable enums. Extend
Ty::hashable(src/collection.rs:189) to user and standard enums whose payloads are key types, rejecting recursion throughtype_facts(src/type_facts.rs). The runtime path already works for kindB. Update the three diagnostics (src/frontend.rs:885,src/frontend/collections.rs:891,:3138) to name the offending field. Tests intests/test_collections.rs,test_dictionary_lookup.rsandtest_set_operations.rs; reference updates tocollections.md,enums.md,sets.md(which still lists only integers, booleans, and strings) andtutorial/04-collections/dictionaries.md. KeyTypeconstraint. Add it to the built-in constraint table (src/frontend/generics.rs:19, diagnostic at:1001); letdict[K, V]resolve in generic signatures whenK: KeyType. Update generics.- Classes and affine enums. Add a class branch to
write_key(plenty-runtime/src/aggregates/hashing.rs) overRecordfields, plus a unit test besideaggregates/hash_tests.rsthat equal classes in different storage hash equally. Audit each operation in Affine keys insrc/frontend/collections.rsand the runtime’s retain/release paths (try_insert,Collection::at, set algebra). Use the runtime’s running-offset record iterator; field-offset traversal is already linear per pass (issue #330). Hash quality and per-process seeding are already implemented independently (issue #329); this sketch does not change their policy.
Derived serialization for classes and enums
Status: Not implemented. Nothing shipped; this page is the design record
for issue #471. std.json and
std.toml remain authoritative: neither derives
codecs, and a typed codec is written by hand over JSON events or a TOML tree.
No backlog entry exists yet; B51 lists derived typed codecs as
a candidate under B38. The design questions are decided; see
Decisions.
Problem
Reading a configuration file into a class means writing a codec by hand:
pub class Server:
pub host: str
pub port: u16
pub tags: list[str]
pub timeout: Option[i64]
A JSON reader for Server drives json.Decoder.next(), matches every
Event, accepts members in any order, rejects duplicates and unknown members,
reports which members are missing, checks that the port fits u16, and loops
over the tag array. A TOML reader repeats the work over Toml.Table with
find. Writers repeat it again in both directions. The code is long, easy to
get subtly wrong (a forgotten duplicate check, a missing range check), and has
to change with every field.
Rust solves this with serde derives; Python with runtime reflection
(dataclasses, pydantic). Plenty has neither user macros nor runtime
reflection, and its goals rule out hidden allocation and slow compilation.
This page chooses how the per-type code is produced, how a type opts in, how
each kind of type maps to formats, and how errors report where they happened.
What exists today
Probes in this section ran on local main at 7923a7c0 with a debug compiler.
| Fact | Consequence for a design |
|---|---|
A method may declare its own type parameters, including a bound that names another one: def encode[E, S: Sink[E]](self, out: &mut S) -> Result[(), E] compiles, both on a class and as a protocol requirement | A format-neutral protocol can carry each format’s own error type without associated types |
Protocols are structural; std.json.Encoder satisfied a user protocol with its existing null, integer, text, key, start_object and end_object methods, producing {"x":3,"label":"a"} | The JSON encoder needs no adapter if the data model uses its method names |
Unbounded generic bodies are checked per instance; std.fmt.display uses this to call a user method without a named bound | Entry points can accept any type and report an unserializable one at the use site |
A decode shape written by hand (an Option local per field, a str match on member names, construction after the loop) compiled and decoded members in either order | Derived decoding needs no new runtime machinery |
Methods require a self receiver; a receiver-less def decode(n: i64) -> Point is rejected | Decoding, which has no value to call a method on, needs a type-directed hook |
Protocols constrain only classes, enums and bytes; i64, str and list[T] have no methods | A protocol bound cannot admit built-in field types, as the hashing proposal also found |
Generic code reaches another module’s methods only when the type and method are pub; automatic destruction ignores __del__ privacy | Compiler-dispatched hooks can follow the __del__ rule instead |
The runtime prints Point(x=3, y=4) by walking a static descriptor with field names (plenty-runtime/src/aggregates.rs) | A descriptor-driven serializer is possible, at the cost discussed below |
std.json has a resumable event decoder; std.toml has only a tree, because TOML tables can be extended anywhere in a file | The data model must work over both an event stream and a tree |
A closed set of declaration annotations exists with one member, @copy on classes; protocol method bodies agreed @final and @protocol.Name, not yet implemented | An opt-in can extend that closed set rather than introduce user decorators |
Precedents
| System | Where per-type code comes from | Formats | Field options |
|---|---|---|---|
Rust serde | Proc-macro derive generates Serialize/Deserialize against a 29-type data model; formats implement Serializer/Deserializer | Any, via one data model | #[serde(rename, default, skip, flatten, tag)] |
Swift Codable | Compiler synthesizes encode(to:) and init(from:) when a type declares conformance; CodingKeys enum names fields | Any, via Encoder/Decoder protocols with keyed containers | Rename by writing CodingKeys |
Kotlin kotlinx.serialization | Compiler plugin generates a serializer and a static descriptor per @Serializable class | Any, via format-neutral encoders | @SerialName, defaults from constructor |
Zig std.json | Comptime field iteration (inline for (std.meta.fields(T))) in the format library | Each format walks types itself | Per-type override functions |
Go encoding/json | Runtime reflection over struct metadata | One package per format | Struct tags (json:"name,omitempty") |
OCaml ppx_deriving_yojson, C# System.Text.Json source generation | Generated code for one specific format | One deriver per format | Attributes |
Swift and Kotlin are the closest fit: the compiler generates ordinary code against a format-neutral protocol, and the user can write the same methods by hand. Serde does the same with a user macro. Zig and Go push the work into each format library.
Where the per-type code comes from
| A. Compile-time field iteration | B. Derived per-type code, format-neutral protocol | C. Generated per-format code | D. Runtime descriptor walk | |
|---|---|---|---|---|
| Mechanism | A library loop such as for field in fields[T]() unrolled at compile time, with type-shape matching | The compiler writes __encode__/__decode__ for an opted-in type, calling a small std.serial protocol; each format implements the protocol once | The compiler writes JSON code and TOML code per type | The runtime walks the static type descriptor, as printing does |
| New language | Compile-time loops, field values typed per iteration, type-level match, and a partial-construction builder for decoding | One annotation, two hooks, two compiler-expanded std functions | One annotation; format knowledge in the compiler | None visible; a runtime codec per format |
| Adding a format | Write a type walker | Implement one protocol in Plenty | Change the compiler | Change the runtime (Rust) |
| Runtime cost | Direct calls | Direct calls after monomorphization: the JSON instance calls json.Encoder methods, as C would | Direct calls | Descriptor interpretation per field, as == and printing do |
| Compile time, code size | One unrolled walk per type and format | One body per type and format instance, linear in fields | Same as B | None per type |
| Hand-written override | Per-type escape hatch needed | Write the hook; derived code for outer types calls it | Write per-format code | Needs a descriptor callback slot, as hashing design B |
| Visibility | A library reads every type’s fields, including private ones | Code is generated in the type’s own module | Same as B | Runtime reads everything |
| Hidden allocation | Depends on the library | None in derived code (see costs) | None | Rust codec allocates internally |
| Errors | Library’s choice | One path-carrying wrapper | Per format | Runtime error type |
A gives library authors the most power and would make @serial itself
library code. It is also the largest language change: decoding needs a way to
build a value field by field from compile-time-typed parts, and a library that
iterates fields bypasses module privacy. It is a general metaprogramming
feature, not a serialization feature; it should be justified on its own.
C has no runtime advantage over B. Generic instances are specialized per
concrete type, so B’s JSON instance is direct calls on json.Encoder with no
dispatch, which is the code C would write. C only moves format knowledge into
the compiler.
D costs no compile time but conflicts with the “no runtime reflection” goal, cannot call a hand-written codec for a nested type without a new callback slot, and would put format code in Rust instead of checked Plenty.
Recommendation: B. The compiler knows one small data model, formats stay ordinary Plenty, and a derived method is the same code a user could write.
Recommendation
Opting in
A type opts in with one closed annotation on its declaration:
@serial
pub class Server:
pub host: str
pub port: u16
pub tags: list[str]
pub timeout: Option[i64]
@serial
pub enum Mode:
Off
Fixed(i64)
Range(i64, i64)
@serial extends the closed annotation set agreed for
protocol method bodies
to class and enum declarations. The parser already reads class annotations
for @copy; @serial adds a member and accepts annotations on enum. It derives each of two hooks the type does not
already define:
def __encode__[E, S: Encoder[E]](self, out: &mut S) -> Result[(), SerialError[E]]
def __decode__[E, D: Decoder[E]](source: &mut D) -> Result[Self, SerialError[E]]
Self stands for the declaring type. __decode__ is receiver-less, like
__init__ is a construction hook rather than a method; it is the only
receiver-less hook and does not introduce static methods. A user may write
either hook by hand, with or without @serial; derived code for an enclosing
type calls whichever exists. The hooks are invoked only through
std.serial.encode and std.serial.decode, as __init__ and __del__ are
invoked only by construction and destruction, so there is one way to call them.
Without an opt-in, nothing changes: field names do not silently become a wire format, and renaming a field of an unrelated class never breaks a file format.
The data model
std.serial defines two protocols. The model has JSON’s shape, which TOML,
YAML, CBOR and MessagePack also have, and uses std.json.Encoder’s method
names so that the JSON encoder satisfies Encoder[JsonError] without an
adapter.
pub protocol Encoder[E]:
def null(self: &mut Encoder) -> Result[(), E]
def boolean(self: &mut Encoder, value: bool) -> Result[(), E]
def integer(self: &mut Encoder, value: i64) -> Result[(), E]
def unsigned(self: &mut Encoder, value: u64) -> Result[(), E]
def decimal(self: &mut Encoder, value: f64) -> Result[(), E]
def text(self: &mut Encoder, value: &str) -> Result[(), E]
def start_array(self: &mut Encoder) -> Result[(), E]
def end_array(self: &mut Encoder) -> Result[(), E]
def start_object(self: &mut Encoder) -> Result[(), E]
def key(self: &mut Encoder, name: &str) -> Result[(), E]
def end_object(self: &mut Encoder) -> Result[(), E]
pub protocol Decoder[E]:
def null(self: &mut Decoder) -> Result[bool, E] # consumes a null if next
def boolean(self: &mut Decoder) -> Result[bool, E]
def integer(self: &mut Decoder) -> Result[i64, E]
def unsigned(self: &mut Decoder) -> Result[u64, E]
def decimal(self: &mut Decoder) -> Result[f64, E]
def text(self: &mut Decoder) -> Result[str, E]
def start_array(self: &mut Decoder) -> Result[(), E]
def next_element(self: &mut Decoder) -> Result[bool, E] # False consumes the end
def start_object(self: &mut Decoder) -> Result[(), E]
def next_key(self: &mut Decoder) -> Result[Option[str], E] # Nothing consumes the end
def unknown_key(self: &mut Decoder, name: &str) -> Result[(), E]
def reject(self: &Decoder, fault: Fault) -> E
The decoder is pull-based and typed: the derived code asks for the type it
expects, so the format parses JSON number text directly into i64, u64 or
f64, and TOML returns its stored Int or Float without a text round trip.
reject builds the format’s located error from a model-level fault, as
json.Decoder.reject does today. unknown_key applies the format’s policy for
a member the type does not declare: skip its value, or return
reject(Fault.UnknownField). Rejecting is the default; a per-call decode
option skips unknown members instead. The policy is never a per-type setting.
Fault | Raised by derived code when |
|---|---|
UnexpectedKind | The next value is not the requested kind |
MissingField(str) | A required field is absent at the end of an object |
DuplicateField(str) | A field appears twice, whatever the format’s duplicate-key policy |
UnknownField(str) | Raised by the format’s unknown_key under the reject policy |
UnknownVariant(str) | An enum tag names no variant |
OutOfRange | A number does not fit the field’s type |
WrongLength(i64, i64) | A tuple or variant payload array has the wrong length |
DuplicateElement | A set element or dictionary key repeats |
Unrepresentable | Raised by an encoder for a value its format cannot hold, such as null in TOML |
Formats supply implementations in their own modules:
| Format | Encoder | Decoder |
|---|---|---|
| JSON | json.Encoder directly | An adapter over json.Decoder, fed from a buffer or through read_event |
| JSON tree | A builder producing Json | A cursor over &Json |
| TOML | A builder producing Toml, then the existing writer, which orders plain entries before sections | A cursor over &Toml, after parse |
How types map
| Plenty type | Model | JSON | TOML |
|---|---|---|---|
bool | boolean | true | true |
i8..i64 | integer; decoding checks the range | number | integer |
u8..u64 | unsigned; decoding checks the range | number | integer up to i64.MAX, else Unrepresentable |
f32, f64 | decimal; an integer is accepted when decoding | number; non-finite is the encoder’s existing NonFiniteNumber | float, including inf and nan |
str | text | string | string |
Bytes, bytes | text, standard base64 with padding | string | string |
list[T], set[T] | array | array | array, or array of tables |
(A, B, ...) | array of fixed length | array | array |
dict[str, V] | object, insertion order | object | table |
Box[T] | T | ||
| Class | object, fields in declaration order | object | table |
Option[T] field | null when Nothing; absent or null decodes as Nothing | null | omitted |
Option[T] elsewhere | null or T | null | Unrepresentable for Nothing |
| Nullary variant | its name as text | "Off" | "Off" |
| Variant with one payload | object with one key | {"Fixed": 5} | { Fixed = 5 } |
| Variant with several payloads | object with one key, array value | {"Range": [1, 9]} | { Range = [1, 9] } |
Rejected at the use site with the path to the offending field: references
(already excluded from fields), closures, generators, handles and
synchronization types, anonymous sums (their members have no tag names),
Result, Option[Option[T]] (its two Nothing levels would encode alike),
dict with non-str keys (including integer keys), and any class or enum
with neither @serial nor hand-written hooks:
error: `Config` cannot be decoded
note: field `Config.server` has type `Server`, which has no `__decode__`
help: add `@serial` to `Server`, or write `__decode__`
Recursive types through Box, lists or dictionaries are accepted, unlike
automatic == and printing: the derived code recurses explicitly, and every
format’s start_array and start_object enforce its max_depth in both
directions, so a hostile document or a deep value built by hand stops
at the limit instead of overflowing the stack.
Generic types derive per instance: Page[T] with items: list[T] is
serializable exactly when T is, checked where it is used, with the usual
instantiation note.
Field names and optional fields
Member names are the declared field names, unchanged. Encoding writes fields in
declaration order; decoding accepts any order. Missing required fields,
duplicates and (under the default policy) unknown members are errors. A member
is optional when its field is Option[T], or when it corresponds to an
__init__ parameter with a default (see
Invariants and visibility). Plenty has no field
defaults, so a class without __init__ has no other optional members.
A Nothing field is written as null in JSON, so every declared member
appears in the document. TOML has no null, so its encoder omits the member
instead. Both formats decode an absent member as Nothing, so a document
written by one shape reads back in either.
Bytes and bytes are written as standard base64 text with padding. This
suits JSON and TOML, which have no byte string; a future binary format can add
a byte-string method to the model with a default that falls back to base64.
Enums
Enums use the external tag shown above: the variant name, alone for a nullary
variant, otherwise as the only key of an object. This needs no reserved member
name, works in every self-describing format, and lets a decoder read the tag
before the payload, which an internally tagged form ({"type": "Fixed", ...}) cannot guarantee when members arrive in any order. Payloads are
positional, so they are written as a value or an array, never as named
members.
Entry points
std.serial provides two compiler-expanded generic functions:
def encode[T, E, S: Encoder[E]](value: &T, out: &mut S) -> Result[(), SerialError[E]]
def decode[T, E, D: Decoder[E]](source: &mut D) -> Result[T, SerialError[E]]
For a built-in type the compiler expands the mapping above; for a class or
enum it calls the type’s hook. This is how a derived __encode__ handles a
list[Server] field, and how a top-level array decodes without a wrapper
class. Like copy and ==, they are defined for the types the rule admits
rather than through a protocol bound, because no protocol can admit i64.
Formats add typed entry points named dump and load beside their existing
ones. options carries the format’s limits and the unknown-member policy:
json.dump(&value, &options) -> Result[Bytes, SerialError[JsonError]]
json.load[Config](&source, &options) -> Result[Config, SerialError[JsonError]]
json.load_from[Config, R: Reader](&mut reader, &options) # event-driven, bounded window
toml.dump(&value, &options) -> Result[str, SerialError[TomlError]]
toml.load[Config](&text, &options) -> Result[Config, SerialError[TomlError]]
json.load_from decodes straight from events, so memory is the result plus
the decoder’s window. TOML parses to a tree first and writes from one, as its
module already requires. The reader variant’s name follows dump/load; it
is settled when the entry points are implemented.
Errors and field paths
pub class SerialError[E]:
pub cause: E
pub path: list[Step]
pub complete: bool
pub enum Step:
Field(str)
Index(i64)
Key(str)
Variant(str)
The cause is the format’s own error, so JSON keeps its byte offset, line and
column; TOML, decoding from a tree with no positions, relies on the path. A
derived hook adds its step only while an error propagates out of it, so a
successful encode or decode never touches the path. Field and variant names
are string literals; a Key step moves the already decoded key. Adding a step
is a fallible list operation: if memory is refused, complete becomes False
and the cause is kept, so an out-of-memory condition never replaces the
original error. SerialError displays as
servers[2].port: number out of range at line 7, column 12.
Invariants and visibility
-
Construction. For a class without
__init__, derived decoding uses the generated field constructor. An explicit__init__may enforce invariants that field-wise construction would bypass, so decoding never bypasses it. When the__init__parameters afterselfmatch the serialized fields exactly, by name and type,@serialderives__decode__by decoding each member and calling__init__with named arguments. A parameter with a default makes its member optional; when the member is absent, the call omits that argument and__init__applies the default. An error from__init__propagates with the class’s path. Otherwise,@serialderives only__encode__, and the class must define__decode__by hand:error: `@serial` cannot derive `__decode__` for `Server` note: `Server.__init__` has no parameter for field `tags` help: match the `__init__` parameters to the fields, or write `__decode__` -
Members. The serialized fields are all declared fields; names are the declared names, with no renaming.
-
Private fields. Included. The author opted in, the derived code lives in the type’s module, and structural printing and equality already include them.
-
Hook privacy. Hooks follow the
__del__rule: the compiler invokes them regardless ofpub, so a module-private configuration class works withjson.load. Without this, every serializable type would need to be public, since generic code instd.jsonreaches only public methods. -
Destructors. A failed decode drops the fields already decoded, as a failed initializer does; the destructor of the incomplete value never runs.
Costs
| Operation | Allocates | Time |
|---|---|---|
Derived __encode__ | Nothing; the format grows its own output | Linear in the value |
Derived __decode__ | The value’s own strings, lists and dictionaries; field slots are stack locals | Linear in the input; one str comparison chain per member name |
| Error propagation | One path step per enclosing hook, fallibly | Linear in the depth |
| JSON events | One str per key and string event, as today | As json.Decoder.next |
| TOML | The tree, as parse and encode already do | As today |
| Compilation | One encode and one decode body per type and format instance | Linear in fields; measure before adopting |
Member-name dispatch is a linear match over string literals. A type with
many fields could use a length-then-bytes switch; measure first.
Rejected alternatives
- Automatic serialization of every class, as printing works. Field renames would silently change file formats, and private data would be exported without a decision.
- A named
Serializeprotocol bound. Cannot admiti64,strorlist[T]without a special case, and cannot express receiver-less decoding. - Compile-time field iteration (A) as the first step. Larger than
serialization needs; revisit only with independent uses. If it is ever added,
@serialcan become library code without changing the hooks. - Per-format generated code (C). No runtime gain over B after specialization; every new format would be a compiler change.
- Runtime descriptor serialization (D). Runtime reflection in all but name, no path to hand-written nested codecs, and format logic outside checked Plenty.
- A richer data model now (serde’s records, tuple structs and newtypes as separate kinds). It serves positional binary formats; JSON and TOML do not need it. It can be added as new protocol methods with defaults once protocol method bodies ship.
- Internally or adjacently tagged enums as the default. Internal tags need a reserved member name and buffering when the tag arrives late.
Decisions
Each records the choice and the alternatives considered.
- Opt-in spelling:
@serial. Rejected: separate@encodeand@decode, and aserialclause in the class header. A split pair helps types that only ever write;@serialwith a hand-written__decode__covers that. - Renaming members: declared names only, for now. Deferred: a closed
field annotation such as
@serial.name("userId"), which would be the first annotation on fields, and a per-call naming policy (snake to camel case). - Unknown members: reject by default, with a per-call option to skip them. This matches the strictness of both parsers. Rejected: skipping by default, as serde and Swift do, which hides typos in configuration files.
Nothingfields:nullin JSON. Every declared member appears in a JSON document; TOML, which has nonull, omits the member. Rejected: omitting in both formats for a uniform shape, which makes JSON output less explicit about the schema.- Private fields: allowed and included. The author opted in, and printing
and equality already include them. Rejected: excluding them, and rejecting
@serialon a class with private fields. - Hook privacy: the
__del__rule. Generated and hand-written hooks are invoked regardless ofpub, so private types work. Rejected: requiringpubtypes and hooks, asstd.fmtdisplay methods do. - Classes with
__init__: decode through__init__when its parameters exactly match the serialized fields; otherwise require a hand-written__decode__. Parameters with defaults make their members optional, and an absent member takes the default. Rejected: deriving encode only for every class with__init__, which forces hand-written decoders for the common case of a validating constructor over the fields; and field-wise construction that bypasses__init__. - Entry points:
dumpandloadinstd.jsonandstd.toml. Rejected:to_bytes/from_bytesandto_text/from_text, which differ between the two modules, andencode_typed/decode_typed, which collide with theEncoder/Decodervocabulary. Bytes: base64 text. Rejected for now: rejectingBytesfields. Binary formats can add a byte-string model method later.dictwith integer keys: rejected for now. Deferred: writing keys as decimal text, asserde_jsondoes.
Implementation sketch
Each stage is usable on its own.
std.serialin Plenty.Encoder,Decoder,Fault,Step,SerialError, and the JSON decoder adapter. Verify thatjson.EncodersatisfiesEncoder[JsonError]. Hand-written__encode__/__decode__can be tested as ordinary methods before any compiler work.- Hooks and entry points. Accept a receiver-less
__decode__and an__encode__with the fixed signatures; expandstd.serial.encodeanddecodeper instance for built-ins and hooks; diagnostics naming the offending field path. Addjson.dump,loadand the reader variant. @serial. Parse the annotation on classes and enums and generate hook bodies as ordinary checked operations, so ownership, effects and diagnostics need no special path; decoding through a matching__init__is an ordinary call with named arguments.--dumpshows the generated bodies.- TOML. Tree builder and tree cursor;
toml.dumpandload. - Docs. Reference pages for
std.serial, thejsonandtomladditions, classes and enums; a tutorial lesson that replaces a hand-written codec.
Measurements
Before stage 3 is accepted:
- Compile time and object size for 200 classes of 10 fields, each encoded and
decoded through JSON and TOML, against the same program with no derives
(
compile_benchmethod in profiling). - Decode and encode throughput for the 8 MB document in the
std.jsoncosts, comparing derived code withjson.parseto a tree and with a hand-written event codec. Derived code should match the hand-written codec. - Member-name dispatch cost for 4, 16 and 64 fields.
Value generic parameters
Status: Partly implemented. Value parameters
and Bounded[T, lo, hi] are implemented.
The references own their current contracts; the array and general per-instance
assertion sections below remain accepted design. B83/B96 track
remaining scope. The original investigation used local main at 1b5b96b1
on 2026-10-27. Module-scope assertions from local PR #628 are implemented.
The implementation sequence was revised to serve bounded integers first. Const
parameters can name a preceding T: IntType, so bounds retain its exact width
and signedness. This extends the original concrete-integer-only parameter design.
Problem
Plenty cannot write a type whose size is a compile-time number. The language
has inline aggregates (classes, tuples and enums store their fields inline since
B35), but every inline aggregate has a fixed field list written out by hand.
Code that wants “N values of T, stored inline” must either allocate a list
or spell out N fields.
std.simd shows the cost. Its six types (i32x4, u32x4, f32x4, f64x2,
mask32x4, mask64x2) take 614 lines of stdlib/std/simd.plenty. Each stores
its lanes as enum payload fields, Lanes(i32, i32, i32, i32), so every method
destructures both operands by name, and lane(index) is an if chain over the
fields because inline storage cannot be indexed:
pub def lane(self, index: i64) -> Option[i32]:
match self:
case i32x4.Lanes(a0, a1, a2, a3):
if index == 0:
return Some(*a0)
...
The memory proposal
names Array[T, N], BoundedList[T, N] and SmallList[T, N] as later types and
defers them “until inline aggregates and basic constant parameters exist”.
Inline aggregates now exist. This page designs the constant parameters and
the array, spelled array[T, N] to match list and tuple.
Baseline at design acceptance
| Aspect | Current behavior | Source |
|---|---|---|
| Parameters | [T], [T: IntType], [T: FloatType], protocol and Callable bounds. Each is a type. | type_parameters in src/frontend.rs |
| Substitution | A name-to-TypeRef list; a body is substituted per instance | Substitution, substitute in src/frontend/generics.rs |
| Checking | Bodies are checked per used instance, not at declaration; signatures and bounds at declaration | Generics |
| Inference | All arguments explicit, or all inferred structurally from inputs; output-only parameters need an explicit list; no number widening | Inference |
| Identity | Different arguments are different nominal types, even for phantom parameters | Generic data |
| Limits | 16,384 instances per generic function or data type; 262,144 function and 65,536 data instances per compilation; dependency depth 256 plus the number of generic declarations | Emission, src/frontend/generics.rs, src/frontend/data.rs |
| Brackets in expressions | base[...] that parses both ways is resolved by name: a local is indexed, a declaration takes type arguments | Expression::Ambiguous, src/frontend/modules.rs |
| Module constants | const name: T = value, integer or float, literal or numeric type constant initializer; each read becomes an immediate | Modules, call-site substitution |
| Fixed small groups | Tuples are inline; t[0] needs a literal index | Tuples |
Phantom type parameters already give nominal tags: Quantity[Meters] and
Quantity[Seconds] are distinct types. Value parameters add two things phantom
types cannot: storage sized by the value, and the value itself in the body
without a runtime field.
Where value parameters would and would not help
| Code | Today | With value parameters |
|---|---|---|
std.simd lane types | Hand-written payload fields per shape; native lowering matches six names (src/codegen/portable_simd.rs) | An internal Lanes[T, const N: i64] over array[T, N] could share bodies, with the public names kept as aliases. Lowering would match instance identities. The SIMD API keeps a closed public set; that decision stands. |
| Bounded containers (B13) | Not expressible without allocation | BoundedList[T, const N: i64] over inline storage reports capacity exhaustion without allocating |
std.archive 512-byte tar blocks (block_size) | Heap Bytes | array[u8, block_size] once reads can target a borrowed byte view (B11/B39) |
std.random state() -> (u64, u64, u64, u64) | Tuple | No real gain: tuples cover small fixed groups in non-generic code |
std.sketch HyperLogLog precision | Runtime field, validated, checked on merge, read from serialized data | Worse: from_bytes reads the precision from data, so it must stay a runtime value |
std.crypto digest size | Runtime Algorithm; finish_into writes into a caller buffer without allocating | No gain: the algorithm is a runtime choice and the allocation-free path exists |
clist chunks | Physical storage kept private so chunk size can change (chunks) | Not wanted: a public chunk-size parameter would fix what the design keeps private |
The array-oriented consumers above need indexable inline storage. B96 supplied
a separate first consumer: bounds carried in an integer type. Value parameters
and Bounded therefore ship before arrays; arrays retain their accepted scope.
Precedents
| Language | Parameter types | Argument expressions | Inference | When checked |
|---|---|---|---|---|
Rust (stable, min_const_generics, 1.51) | Integers, bool, char | Literal, bare const parameter, or a { } block independent of generic parameters | From [T; N] in argument types | Generically, before monomorphization |
Rust (generic_const_exprs, unstable) | Same | N + 1 and other expressions over parameters | Not through expressions | Needs where [(); N + 1]: evaluatability bounds |
| C++ non-type template parameters | Integers, enums, pointers; floats and structural classes since C++20 | Any constant expression | Deduced only from bare N; N + 1 is a non-deduced context | At instantiation |
| D template value parameters | Any type usable at compile time | Any CTFE-evaluable expression | From argument types | At instantiation; if (n > 0) constraints |
Zig comptime parameters | Any type, including type | Arbitrary compile-time evaluation, bounded by an eval-branch quota | Explicit arguments; types are values | Lazily, per instantiation |
| Mojo parameters | Any compile-time type | Arbitrary parameter expressions | From arguments | Per instantiation; SIMD[DType.float32, 4] is the stdlib vector |
Lessons:
- Rust’s stable subset has served for years; the hard part is expressions. With
generic_const_exprs,N + 1and1 + Ndo not unify, well-formedness needs evaluatability bounds, and the feature has stayed incomplete. The Rust project has since pursued narrower argument forms. Plenty should not start there. - Before const generics, Rust implemented array traits only for lengths 0 to 32.
That is the duplication
std.simdshows today. - C++ and Zig show that arbitrary compile-time evaluation produces late, long diagnostics and needs evaluation budgets.
- Plenty differs from Rust in one useful way: bodies are already checked per
instance. Inside a body
Nis a known number, so arithmetic on it is ordinary arithmetic on a constant. Only type positions need restriction.
Options
| Option | Benefit | Cost |
|---|---|---|
| A. Status quo: concrete types, hand-written or generated | No language change | Duplication (std.simd); no bounded containers without allocation |
| B. Runtime capacity fields | Works today (BufReader, HyperLogLog) | Heap storage; checks at run time; no inline arrays |
C. Minimal value parameters: integer and bool values, bare arguments only | Covers arrays, bounded containers and fixed lanes; no expression unification; finite instance set | No array[T, N + 1] in signatures |
| D. C++-style: expressions in non-inferred type positions, evaluated per instance | concat and push can name result sizes | Errors appear at the use, inside the callee’s signature; more checking machinery |
| E. Zig-style compile-time evaluation | Most expressive | A compile-time interpreter, evaluation budgets, and generic code that is hard to read; against the small-language goal |
Recommendation
Adopt option C, initially with Bounded[T, lo, hi], followed by the accepted
array[T, N] consumer. Defer option D until code shows a need. Reject E. The decisions record
what is settled.
Declaring
class Ring[T, const N: i64]:
items: array[Option[T], N]
head: i64
count: i64
def filled[T, const N: i64](value: T) -> array[T, N]:
array[T, N].filled(value)
const N: TypeName declares a value parameter. The keyword is already reserved
and already means “compile-time value” at module scope. Without it, [N: i64]
would read like a bound, and T: IntType (a type) beside N: i64 (a value)
would differ only by what the right-hand name is. One marker keeps the rule
mechanical.
Value parameter types: the eight integer types and bool; u8 and bool
cost little once integers work. No floats (equality
and NaN make identity awkward), no str, no user types. Value and type
parameters may appear in any order. Functions, methods, classes and enums
accept them, as they accept type parameters; protocols do not in the first
version.
Arguments
A value argument is one of:
| Form | Example | Notes |
|---|---|---|
Integer or bool literal, optionally negative | array[u8, 64] | Typed by the parameter’s declared type, as a nongeneric parameter types its literal |
Module constant, including an imported pub const | array[u8, block_size] | Must have exactly the parameter’s type; no widening |
| A value parameter in scope, bare | array[T, N] | Forwarding only |
Nothing else: no N + 1, no calls, no type constants such as u8.MAX
(name a module constant instead). A derived size is named once as a module
constant, which composes with the parallel work on constant initializers:
whatever arithmetic a const initializer accepts is evaluated in the
declaration, not in the brackets. Inside a generic, a derived size cannot be
named; that is option D.
This rule also bounds expansion. A body can request only values that already appear in the program: literals, constants, and its own parameters. Value arguments cannot grow without end, so the existing type-driven limits remain the only expansion guard needed.
In bodies
N is an immutable value of its declared type. Each instance substitutes the
number, so range(N), N * 2, if N > 0: and comparisons compile to
immediates, as module constant reads do. A local cannot shadow a value
parameter, matching the existing rule that a type parameter name cannot also
name a value binding.
Since bodies are checked per instance, a body rejected for one value, such as
array[T, N] with a negative N, is reported as an error of that instance,
at the use that required it, through the existing
instance diagnostics.
Type identity and checking
Two instances are the same type when their declarations match and their
arguments are equal after resolving aliases and constants: with
const width: i64 = 4, array[i64, width] is array[i64, 4]. Instances print
with values: array[i64, 4], Flags[True]. A type alias can name a complete
instance (type Block = array[u8, 512]), since aliases already name complete
generic instances.
At declaration, the checker verifies kinds: a type where a type parameter is
expected, a value of the right type where a value parameter is expected. Value
constraints such as N > 0 are not part of the signature. A data type that
needs one states it with a per-instance assertion;
array rejects a negative length the same way, built in.
Per-instance assertions
The module-scope assert COND, "message" from local PR #628 is also accepted
in class bodies and generic function bodies:
class Ring[T, const N: i64]:
assert N > 0, "Ring needs a positive capacity"
items: array[Option[T], N]
head: i64
count: i64
def best_window[const N: i64, const W: i64](xs: &array[i64, N]) -> i64:
assert 0 < W and W <= N, "window must fit the array"
...
- Position. Assertions come before any other item or statement in the
body. There
assertbegins the static form, as at module scope; elsewhere in a function body it stays an ordinary name, as PR #628 leaves it. - Operands. The condition and message may use constants and the declaration’s value parameters only: no locals, fields, calls or type parameters. The checker rejects anything else at declaration, once, not per instance.
- When it runs. Once per instance, after substitution, whenever the type or function is instantiated. That includes a type named only in a field, binding or signature with no method ever called. This matters because methods are checked only when used; an assertion attached to a method would never fire for an instance that only stores data.
- Failure. Reported at the use that required the instance, with a note at the assertion, like the instance diagnostics added for issue #518:
main.plenty:9:11: cannot instantiate `Ring[i64, 0]`: static assertion failed: Ring needs a positive capacity
ring.plenty:2:5: note: in instantiation `Ring[i64, 0]`: assert N > 0
In a nongeneric class or function the assertion is accepted and runs once, like a module assertion; it is mainly useful in generic code.
Inference
Value parameters are inferred by the existing structural match: array[T, N]
against array[i64, 4] learns T = i64 and N = 4. A literal passed where
the parameter type is an array also supplies N from its element count; see
literals at generic parameters. Nothing else
supplies evidence: not a list’s length, not an ordinary argument’s value. Output-only
value parameters need the explicit list, as type parameters do:
filled[i64, 4](0). Partial explicit lists stay unsupported (B16).
In expressions, filled[i64, 4](0) keeps today’s bracket rule: brackets after a
local index it, brackets after a declaration are generic arguments. The type
argument grammar gains literals, so xs[0] on a local still indexes.
The first consumer: array[T, N]
A built-in fixed-length inline sequence. This page fixes only what value parameters need; its complete API needs its own review.
| Property | Contract |
|---|---|
| Storage | N elements inline in the containing value, no header, no allocation |
| Copy and move | Copyable when T is copyable, otherwise moves, as tuples do |
| Construction | Complete: filled(value) for copyable T, a literal with an expected array type, or a callback per index. Never partly initialized. |
| Access | a[i] with a runtime index, bounds-checked like list, trapping out of range; &mut a[i] under ordinary loan rules; len(a) is N |
| Iteration | Borrowed iteration yields references, as for lists |
| Equality, printing | Structural, as for tuples |
| Size | Element size times N; large frames are bounded by the existing frame-size diagnostic (issue #441) |
BoundedList[T, N] and a std.simd rewrite are library code over this type.
Array literals
A list literal [1, 2, 3] is typed array[T, N] when its expected type is an
array instance, after resolving aliases. It then needs no ?:
lanes: array[i32, 4] = [1, 2, 3, 4]
values: list[i32] = [1, 2, 3, 4]?
This is sound because an array literal allocates nothing. Its storage is the
destination itself: the local’s frame slot, the field, or the argument or
return slot, as for a tuple. A list literal returns Result only because it
allocates a heap buffer that can fail. Element expressions keep their own
failures, so [f()?, 2] still propagates f’s error, as tuple components do.
The element count is known when parsing, so a count other than N is a
compile-time error, and T gives each element its expected type, as with
add[u8](1, 2).
The expected type comes from the places that already supply one:
| Source | Example |
|---|---|
| Annotated binding | xs: array[i64, 3] = [1, 2, 3] |
| Concrete parameter type, including constructor fields and explicit generic argument lists | checksum([1, 2, 3]) for checksum(block: array[u8, 3]); sum_all[i64, 3]([1, 2, 3]) |
| Generic parameter whose declared type is an array | sum_all([1, 2, 3]) for sum_all[T, const N: i64](xs: array[T, N]); see below |
| Declared return type | return [0, 0] in a function returning array[i64, 2] |
Without one, [...] stays a list. Comprehensions and empty [] are not array
literals: a comprehension’s length is not static, and array[T, 0] is built
with filled.
Literals at generic parameters
A generic parameter still supplies no expected type, since T and N are
unknown until inference finishes. It does supply an expected shape. Parameter
types are known from the signature before any argument is typed, so the
checker can see that xs: array[T, N] is an array before it knows T or N:
- For each argument, the checker reads the parameter’s declared type after
resolving aliases. If it is an
arrayinstance, a[...]literal there is an array literal. The shape is also visible through&, tuple components and the element type of an enclosing array, so[[1, 2], [3, 4]]forarray[array[T, N], M]is two levels of array. - The literal’s element count is evidence for the length argument:
N = 3. - Elements are typed as any generic argument is today, without an expected
type: unsuffixed integers default to
i64. Their common type is evidence forT.sum_all[u8, 3]([1, 2, 3])or suffixed elements choose another type. - Evidence from all arguments is merged and must agree, as now. Arguments still evaluate once, in source order.
The shape stops at type parameters and other nominal types. A parameter typed
as bare T gets a list from [1, 2, 3], which needs ? as usual; so does
Option[array[T, N]], whose argument is a constructor call, not a literal.
One call can mix shapes. For
def pair[T, const N: i64](names: list[T], codes: array[T, N]),
pair([1, 2]?, [3, 4, 5]) builds a list and a three-element array, and both
give evidence for T. pair([1u8]?, [3, 4]) fails as add(1u8, 2) does:
T is u8 from one argument and i64 from the other.
Instances, limits and compile cost
Each distinct value is a distinct instance with its own code. array[u8, 16]
and array[u8, 32] duplicate every method they use. Value instances count
against the same per-template (16,384) and per-compilation limits as type
instances; no separate budget is needed, since arguments cannot grow.
Code size is the real cost. Libraries should keep length-independent work in
functions that take a borrowed view, so only thin wrappers depend on N. That
needs borrowed sub-range views (B11/B12); until then, the guidance belongs in
library documentation rather than the compiler.
Interactions
| Area | Effect |
|---|---|
| Ownership | Value arguments are copied numbers; no loans. array[T, N] follows tuple ownership. |
--check-all-methods | Unchanged: it checks every method of each instance the program creates; a value-generic type with no instance stays unchecked |
| Compile cache | Instances are ordinary functions with N folded to immediates; content keys need no change (cache) |
| Runtime metadata | array needs a type descriptor carrying element descriptor and count, for drop, equality and printing |
std.simd | Public shapes stay closed and named; any generic implementation is internal and must keep the native lowering’s matches |
| Module constants | Constants become usable as arguments; the constant work’s arithmetic and static assertions provide derived sizes and checks without bracket expressions |
| C exports | Public instances already need public concrete arguments; a value argument is a number, so it adds no visibility requirement |
Diagnostics
main.plenty:4:24: value arguments must be a literal, a module constant, or a value parameter; found `N + 1`
note: name a derived size with a module constant: `const padded: i64 = ...`
main.plenty:6:11: `array` expects a type for `T`, found the value `4`
main.plenty:7:11: value argument for `N` must be `i64`, found `u8` constant `width`
main.plenty:9:5: cannot infer `N` for `filled`; supply every argument: `filled[i64, 4](0)`
main.plenty:12:9: `array[i64, -1]`: an array length cannot be negative
main.plenty:14:12: conflicting evidence for `N` in `zip_add`: 4 from argument 1 (`array[i64, 4]`), 3 from the array literal in argument 2
main.plenty:15:10: conflicting evidence for `T` in `pair`: `u8` from argument 1, `i64` from the array literal in argument 2
note: give the literal's elements a type, such as `3u8`, or supply every argument: `pair[u8, 2](...)`
main.plenty:16:14: `?` needs a `Result`, found `array[i64, 3]`
note: a literal passed as `array[T, N]` is an array, which allocates nothing and cannot fail
main.plenty:17:9: cannot infer `T` for `sum_all` from an empty literal; `[]` is not an array literal
note: build an empty array with `filled`, or supply every argument
main.plenty:18:9: `sum_all` expects `array[T, N]` for `xs`, found `list[i64]`
note: only a literal becomes an array here; a list does not convert
Rejected alternatives
| Alternative | Reason |
|---|---|
[N: i64] without const | Confuses bounds with value types; learners need one visible marker |
Expressions in brackets (Rust generic_const_exprs) | Unification of expressions, evaluatability bounds, and unbounded instance growth |
| Zig-style compile-time functions | A second evaluation model; against small-language and fast-compile goals |
Float, str or user-type values | Identity and printing questions for no current use; str can be reconsidered after string constants land |
Inferring N from runtime values or list lengths | Hidden specialization per call site and no static guarantee |
Lane-count-generic public std.simd | Already rejected by the SIMD API |
| Value parameters without a consumer | No motivating use; Bounded now supplies the first consumer |
Decisions
-
Spelling:
[T, const N: i64]. Rejected:[T, N: i64], which reads like a bound. -
Value types: the eight integer types and
bool. Rejected:i64only, and floats,strand user types. -
Name and literal: lowercase
array[T, N], matchinglistandtuple. A literal with an expectedarraytype builds one inline without?; see array literals. -
Value parameters first ship with
Bounded. This supersedes the original decision to requirearrayin the same change. Bounded integers provide a concrete use for values in type identity without indexable storage. -
Derived sizes (option D): deferred. Add expressions such as
array[T, N + 1]in signatures only with evidence fromBoundedListorstd.simdcode. -
Per-instance checks:
assertin class and generic function bodies. The module-scopeassertfrom PR #628 runs once per instance, from constants and value parameters, whenever the type or function is instantiated, with failures reported at the instantiation site; see per-instance assertions. Rejected: each type validating its own arguments, which only built-in types such asarraycan do at compile time; a user type would have to check in its constructor and fail at run time. -
Array literals at generic parameters. A literal where the parameter’s declared type is an array builds an array and supplies
N, sosum_all([1, 2, 3])needs no argument list; see literals at generic parameters. Rejected: requiringsum_all[i64, 3](...), which makes the common call verbose for no gain in clarity.
Implementation sketch
- Parameters. Parse
const N: Tin parameter lists and literals in type argument lists. Represent a generic argument as a type or a value; a separate argument kind is cleaner than aTyleaf that every type position must reject, but touches every place that stores instance arguments asVec<Ty>. ExtendSubstitutionwith value entries, substituteNin bodies as a literal, extend instance names and structural inference. Tests: kinds, constants as arguments, identity across aliases and constants, inference, rejected expressions, phantom value tags. - Per-instance assertions. Accept
assertat the start of class and function bodies, check at declaration that operands are constants and value parameters, and evaluate after substitution whenever an instance is created, including data-only instances whose methods are never checked. Failures use the instance diagnostic chain. Tests: failing and passing instances, a type used only as a field, rejected operands,assertas an ordinary name later in a body. array[T, N]. Inline layout, descriptor, construction including literals with an expected type or at array-typed generic parameters, checked indexing, borrowed iteration, equality, printing, drop of affine elements. Reference page, tutorial lesson in the generics chapter, cheat sheet entry.- Library use.
BoundedList[T, N]under B13. Measure an internalstd.simdrewrite against the current code: native lowering must still recognize the shapes, and object code must not regress. - Deferred: derived sizes. Only if stage 4 code shows a need: expressions in non-inferred output positions, evaluated after substitution.
std.plot: a ggplot2-style plotting module
Status: Partially implemented. Design accepted. The core/SVG reference describes the implemented subset: tables and record accessors, layer-specific data and inherited mappings, nine geoms, grouped counts/binning and bar positions, continuous/discrete position scales with transforms and explicit breaks/labels, named manual palettes and RGB gradients, fixed/free wrapped and grid facets, supplied interval/ribbon geoms, basic themes/legends, wrapped text boxes with layout preflight, and streaming SVG. This does not complete milestone 1: the remaining scales, smoothing, measured layout and the full example gallery remain future work, tracked under B80 in the backlog. The decisions remain the target design. Code fences here show proposed API and are not compiled; use the reference and tutorial for current API.
Problem
Plenty programs can compute summaries of data in lists, records, and
std.dataframe frames, but cannot draw them.
Today a user writes CSV and opens R or Python to plot it.
The plotting library most data analysts know well is R’s ggplot2, built on Wickham’s layered grammar of graphics (2010). A plot is data, plus aesthetic mappings from variables to visual properties, plus layers (a geom, a stat, and a position adjustment), plus scales, a coordinate system, facets, and a theme. Each part has defaults, so a short specification gives a complete plot with axes and legends.
This proposal designs std.plot: the same grammar, with ggplot2’s names and
defaults, expressed in ordinary Plenty. A ggplot2 user should be able to read a
std.plot program without documentation and write one after reading the
correspondence table.
Goals
- The grammar, not only the drawing. Mappings, layers with default stats, trained scales with automatic legends, facets, coordinates, and themes.
- ggplot2 names and defaults.
ggplot,aes,geom_point,scale_colour_brewer,facet_wrap,theme_minimal,labs. Default theme, palette, bin count, units (points for text, millimetres for sizes) and layout match ggplot2 so values transfer and plots look familiar. - Plenty’s builtin data. Lists of records, dictionaries of columns, lists and tuples are first-class input. This is required, not an add-on.
std.dataframelater, with the same plot API.- Plenty conventions. Errors are
Resultvalues with precise causes, allocation is explicit and fallible, ownership is ordinary, and the output is byte-for-byte deterministic. - Written in Plenty, following the standard library policy, with one justified native dependency already in the toolchain.
Non-goals
- Interactive or on-screen display. Output is files or
std.iostreams. - R’s non-standard evaluation: arbitrary expressions inside
aes(). - ggplot2’s extension system (
ggproto) and its ecosystem (ggrepel,patchwork,gganimate). - Complex-script text shaping (Arabic, Indic scripts) in the first releases.
- Pixel-identical output to ggplot2. Layout and defaults match; renderers, fonts and smoothing details differ.
Precedents
ggplot2 composes with +: ggplot(mpg, aes(displ, hwy)) + geom_point().
+ dispatches on the right operand (ggplot_add()), so layers, scales,
themes and even lists of components can be added. aes() captures unevaluated
expressions (quosures) and evaluates them against the data frame at build time,
so aes(colour = factor(cyl)) works without quoting. after_stat(),
after_scale() and stage() select when a mapping is evaluated. Version
4.0.0 (CRAN, 2025-09-11) moved to S7 classes and added theme-level
ink/paper colours and palette settings; the grammar is unchanged.
plotnine (0.15.8) ports ggplot2 to Python.
Python has operator overloading but no lazy evaluation, so plotnine keeps +
(ggplot.__add__) and replaces unevaluated expressions with strings: in
aes("wt", "mpg", color="factor(cyl)") each string names a column or is
evaluated as an expression against the dataframe, with columns taking
precedence over variables. It also accepts arrays and scalars as mapping
values. Theme elements use legend_position for legend.position. Rendering
goes through matplotlib. plotnine shows that strings are an acceptable
replacement for NSE for ggplot2 users; its expression evaluation is the part
that relies on Python’s dynamic eval.
plotters (0.3.7) is a Rust
drawing library for charts. It is not related to the grammar of graphics: there
are no aesthetic mappings, scale training, stats, automatic legends from
mappings, or facets. A program sizes label areas in pixels, builds a Cartesian
coordinate range by hand, configures a mesh, and draws series (LineSeries,
point series, histograms) with explicit styles. It sits at the level of
ggplot2’s grid graphics layer, roughly the bottom fifth of the work. Its
backends are SVG and a bitmap backend that pulls in image, and fonts through
font-kit (system font discovery) or ab_glyph. Rendering
explains why it is not recommended as a backend.
Language probes
Probes ran on local main at dbcdf39a with a debug compiler.
Since those probes, arithmetic operator hooks
have been implemented (issue #512). The historical rejection below no longer
applies; the composition recommendation still accounts for fallible allocation.
| Feature | Result | Consequence |
|---|---|---|
Operator + on a class with __add__ | Rejected: operator `+` does not accept V | No + composition without a language change |
Method with owned receiver self: P | Rejected: receivers are & or &mut | Builder methods cannot consume the plot |
| Free function consuming and returning a plot | Works | add(add(p, a)?, b)? is possible but reads inside out |
Method borrowing self and returning a new value, chained on temporaries | Works: base.plus(a)?.plus(b)? | Composition uses this |
Anonymous sums with defaults: x: str | AfterStat | Unmapped = Unmapped.Unset | Works; "wt" and after_stat("count") inject | aes(x="wt", y=after_stat("count")) needs no wrapper |
size: f64 | Unset given size=2 | Rejected: expected Unset | f64, got i64 | Numeric settings accept f64 | i64 |
| Named arguments and defaults on functions and methods | Works | geom_point(size=2, colour="red") |
| Multi-line list display as an argument | Works | One component per line |
Generic accessor num("wt", def(c: &Car) -> f64: c.wt) infers T | Works, via BoxedCallable | Records enter through typed accessors |
Car.wt as a value | Rejected; the selector exists only for clist.column | No typed column handles yet |
copy() of components holding closures | Not supported | The plot itself holds no closures |
Shared[T] field with share() | Works; handles have no copy() | Plots share their table without copying it |
Also relevant: references cannot be stored in classes, so a plot cannot
borrow caller data; wildcard imports and re-exports do not exist; a function
may have at most 255 parameters; statements nest to 64 levels. The
derived serialization proposal (B77, accepted,
not implemented) gives classes an Encoder hook that a plot table can consume.
std.compress provides zlib compression and
CRC-32, which is all PNG needs beyond pixels.
Correspondence with ggplot2
| ggplot2 | plotnine | std.plot |
|---|---|---|
ggplot(df, aes(x, y)) | ggplot(df, aes("x", "y")) | ggplot(table, aes("x", "y"))? |
p + geom_point() + labs(...) | same | ggplot(table, mapping, [geom_point(), labs(...)]?)? |
base + geom_line() (base unchanged) | same | base.plus([geom_line()]?)? |
aes(colour = cyl) | aes(color="cyl") | aes(colour="cyl") (color= also accepted) |
aes(colour = factor(cyl)) | aes(color="factor(cyl)") | aes(colour=factor("cyl")) |
aes(y = after_stat(density)) | aes(y=after_stat("density")) | aes(y=after_stat("density")) |
aes(y = 2 * hp) | aes(y="2*hp") | Compute the column when building the table |
geom_point(colour = "red") (setting) | same | geom_point(colour="red") |
geom_bar(position = "dodge") | same | geom_bar(position=position_dodge()) |
geom_smooth(method = "lm") | same | geom_smooth(method=Smooth.Lm) |
geom_histogram(binwidth = 500) | same | geom_histogram(binwidth=500) |
scale_x_log10() | same | scale_x_log10() |
scale_colour_brewer(palette = "Set1") | same | scale_colour_brewer(palette=Brewer.Set1) |
scale_y_continuous(limits = c(0, 50)) | limits=(0, 50) | scale_y_continuous(limits=(0, 50)) |
coord_flip(), coord_cartesian(xlim = c(1, 5)) | same | coord_flip(), coord_cartesian(xlim=(1, 5)) |
facet_wrap(~class, ncol = 3) | facet_wrap("class", ncol=3) | facet_wrap("class", ncol=3) |
facet_grid(drv ~ cyl) | facet_grid("drv ~ cyl") | facet_grid(rows="drv", cols="cyl") |
theme_minimal(base_size = 12) | same | theme_minimal(base_size=12) |
theme(legend.position = "bottom") | theme(legend_position="bottom") | theme(legend_position=Legend.Bottom) |
element_text(face = "bold"), element_blank() | same | element_text(face=Face.Bold), element_blank() |
labs(title =, x =, colour =) | same | labs(title=, x=, colour=) |
ggsave("p.png", width = 6, height = 4, dpi = 300) | p.save(...) | p.save("p.png", width=6, height=4, dpi=300)? |
Message: `stat_bin()` using `bins = 30` | same | A note in the render result |
Error: object x not found (at print) | at draw | PlotError.UnknownColumn from ggplot(...)? |
The examples page shows six classic plots side by side
in ggplot2, plotnine and std.plot, and a complete program.
Design
Data: from Plenty values to a plot table
ggplot2 plots a data frame; anything else first goes through fortify().
std.plot keeps that split. Typing happens at the boundary, where Plenty knows
the record type. The grammar then works on a Table of named columns, exactly
as ggplot2 works on a data frame.
pub class Table # immutable after construction; columns behind Shared
pub def share(self) -> Table # another owner of the same columns; no allocation
pub def height(self) -> i64
pub def names(self) -> list[str] ... # (Result-returning in the real signature)
enum Column # what a table column holds
Num(list[f64]) # continuous
Int(list[i64]) # continuous unless wrapped in factor()
Text(list[str]) # discrete
Bool(list[bool]) # discrete
MaybeNum(list[Option[f64]]) # missing values allowed
MaybeText(list[Option[str]])
Ways in, in the order they become available:
-
Columns.
table([("wt", Column.Num(wts)), ("cyl", Column.Int(cyls))]?)?moves owned lists in, without copying.columns(&values)copies adict[str, list[f64]]; the dictionary’s key order becomes column order. Unequal lengths and duplicate names fail withPlotError.ShapeMismatchandPlotError.DuplicateColumn. -
Records with accessors.
rows(&cars, fields)evaluates one typed accessor per column and copies the results:cars_table = rows(&cars, [ num("wt", def(c: &Car) -> f64: c.wt)?, num("mpg", def(c: &Car) -> f64: c.mpg)?, integer("cyl", def(c: &Car) -> i64: c.cyl)?, text("model", def(c: &Car) -> str: c.model)?, ]?)?num,integer,textandflagbox their callback and inferTfrom it (probed). The accessor list lives only for the call; the table holds plain columns. Derived values go here too, replacing ggplot2’saes(y = 2 * hp):num("kpl", def(c: &Car) -> f64: c.mpg * 0.425)?. -
Records by field name, after B77:
records(&cars)?drives each row’s derived__encode__into a column builder that implementsstd.serial.Encoder[PlotError]. Scalar fields become columns named after the fields, with no intermediate tree. Nested records, lists and enums with payloads are rejected with the field path. Afields=["wt", "mpg"]argument limits the copy. This is the closest Plenty gets toggplot(mtcars, ...)and should become the recommended path. -
std.dataframe, through a bridge module that only dataframe users import, sostd.plotnever links the Polars engine.ggplot_frame(&frame, mapping, components)asks the core for the columns the specification uses (required_columns), checks them againstframe.schema(), and extracts only those withvalues_f64,values_str_nullableand the other typed readers. Aesthetics may also be dataframeExprvalues, evaluated by oneselectbefore extraction. That is the dataframe counterpart of ggplot2’s expressions inaes()and of plotnine’s string expressions. A later version can pushstat_countandstat_bininto Polarsgroup_by.
Lists and tuples enter through columns or table. A list[(f64, f64)] of
points goes through rows with two accessors.
Aesthetic mappings
pub enum Unmapped:
Unset
pub type Aes = str | Factor | AfterStat | Stage | Unmapped
pub def aes(x: Aes = Unmapped.Unset, y: Aes = Unmapped.Unset,
colour: Aes = Unmapped.Unset, color: Aes = Unmapped.Unset,
fill: Aes = ..., alpha: Aes = ..., size: Aes = ..., shape: Aes = ...,
linetype: Aes = ..., linewidth: Aes = ..., group: Aes = ...,
label: Aes = ..., weight: Aes = ..., xmin: Aes = ..., xmax: Aes = ...,
ymin: Aes = ..., ymax: Aes = ..., xend: Aes = ..., yend: Aes = ...) -> Mapping
- A
strnames a column.aes("wt", "mpg")works positionally becausexandycome first, as in ggplot2. factor("cyl")treats a numeric column as discrete. Levels are the sorted distinct values;factor("class", levels=[...]?)orders them explicitly.after_stat("count")names a variable computed by the layer’s stat. Each stat documents its variables (count,density,ncount,ndensity,prop,width), as in ggplot2.stage(start="x", after_stat="count")arrives withafter_scalein a later milestone.- There is no expression language. Computed values belong in the data step, where Plenty type-checks them; stat-computed values are limited to the named variables above, which ggplot2 provides to cover the common expressions.
Mapping and setting stay distinct, as in ggplot2: aes(colour="cyl") maps a
column; geom_point(colour="red") sets a constant. ggplot2’s classic mistake,
aes(colour = "blue"), maps a constant and produces a legend labelled “blue”.
In std.plot it fails with UnknownColumn("blue"), and the message suggests
geom_point(colour="blue").
colour and color are both accepted, in aes, geoms and scales; giving
both is PlotError.Conflict. ggplot2 and plotnine accept both spellings, and
users type whichever they learned.
Checking and errors
Column names in strings cannot be checked at compile time. They are checked
eagerly instead: ggplot(...) and plus(...) validate every mapping against
the table, every required aesthetic (geom_point needs x and y), and every
setting (colour names, palette sizes), before anything is rendered. The error
arises at the statement that introduced the mistake. ggplot2 and plotnine
report the same mistakes only when the plot is printed or drawn.
pub enum PlotError:
UnknownColumn(ColumnFault) # name, aesthetic, layer, available names, nearest match
MissingAesthetic(AesFault) # "geom_point requires y"
WrongKind(AesFault) # discrete column for a continuous-only scale, etc.
ShapeMismatch(ShapeFault)
DuplicateColumn(str)
Conflict(str) # colour and color both given; two coords
InvalidSetting(SettingFault) # unknown colour name, negative size, bins = 0
Font(FontError)
Io(IoError)
Allocation(AllocError)
Faults name the layer by position and geom (layer 2 (geom_smooth)), the
aesthetic, and the column, and print through
std.report like other std errors:
error: layer 1 (geom_point): aesthetic `colour` maps unknown column "cly"
available columns: wt, mpg, cyl, gear, model
did you mean "cyl"?
Statically typed handles would move these checks to compile time. Plenty’s
clist already has a compile-time selector, Car.wt, accepted only by
column. Generalising it into a typed Field[Car, f64] value would allow
aes(x=Car.wt, colour=factor(Car.cyl)), checked by the compiler. That is a
language change; decision 6 defers it
after records ships, rather than designing it now.
Composition
Plenty has operator hooks and explicit consuming method receivers. For this API,
+ would be a poor fit: every step can allocate, and
Plenty makes that visible, so a chain would read
((ggplot(t, m)? + geom_point())? + labs(title="x"))?, as str already
requires ("a" + b)?.
std.plot therefore composes in two ways that exist today:
p = ggplot(cars, aes("wt", "mpg", colour=factor("cyl")), [
geom_point(size=2),
geom_smooth(method=Smooth.Lm),
labs(title="Fuel economy", colour="Cylinders"),
theme_minimal(),
]?)?
with_line = p.plus([geom_line()]?)? # p is unchanged
ggplot(data, mapping, components)takes the components as a list, in order, one per line. It reads top to bottom like a+chain. Later components override earlier ones by the same rules as ggplot2: a secondxscale replaces the first, a complete theme replaces the theme, andtheme(...)modifies it.p.plus(components)borrowspand returns a new plot that sharesp’s table and copies its component list (probed, including chaining on temporaries:base.plus(a)?.plus(b)?). This is ggplot2’sbase <- ggplot(...); base + geom_point(); base + geom_line()idiom, and the way to add components conditionally:if smooth: p = p.plus(...)?.
Every component constructor returns one enum, Component, so a single list
holds layers, scales, coords, facets, labels and themes. This mirrors adding a
list() of components in ggplot2. There is no p.add(...) mutator: a plot is
immutable once built, so validation happens once, and sharing a plot or its
table between threads needs no locks.
Ownership and allocation
ggplottakes itsTableby value. A table holds its columns in aSharedowner;table.share()gives another plot the same columns without copying. Tables are immutable, so sharing has value semantics.- Components are plain data: enums, numbers, strings and lists. They hold no
closures, so a component list can be copied with
copyand a plot can be rebuilt or shared freely. - Allocation happens in four visible places, each fallible: building the table
(copies columns, except
tablewhich moves them), building the plot (the component list), rendering (per-layer computed data, scale breaks, text, and for PNG a pixel buffer ofwidth × height × 4bytes), and writing. Rendering to an SVG stream writes directly to astd.iowriter, so the document is never held in memory. Nothing allocates behind a non-Resultcall. - Rendering borrows the plot; one plot can be saved in several formats.
The build pipeline
Rendering follows ggplot2’s ggplot_build() then ggplot_gtable():
- Resolve mappings per layer: inherit the plot mapping unless the layer
sets
inherit_aes=False; read columns. - Facet: assign each row to a panel.
- Transform scales (
log10,sqrt,reverse) before statistics, as ggplot2 does. - Group by the interaction of discrete mappings and
group. - Compute stats per panel and group (
identity,count,bin,smooth,boxplot,density); resolveafter_statmappings. - Adjust positions:
identity,stack,fill,dodge,jitter. - Train and map scales: ranges and levels, breaks, labels, palettes; merge legends that share a title and levels.
- Coordinates: Cartesian limits (zoom without dropping data), flip, fixed aspect ratio; polar later.
- Lay out: measure text, size axes, strips, legends and titles, place panels.
- Draw primitives to a canvas: rectangles, polylines, polygons, circles, text with anchor and rotation, clip regions, groups.
Each stage is a Plenty function over columns of f64 and level indices.
Stages 1–9 are independent of the output format.
Layers, stats and positions
| Milestone | Geoms (default stat) | Stats | Positions |
|---|---|---|---|
| 1 | point, line, path, col, bar (count), histogram (bin), smooth (smooth, Lm only), hline, vline, abline, segment, text | identity, count, bin, smooth | identity, stack, fill, dodge |
| 2 | area, ribbon, boxplot, violin, density, freqpoly, errorbar, pointrange, linerange, tile, raster, rect, label, step, jitter | boxplot, density, ydensity, summary, smooth with Loess | jitter (seeded), dodge2 |
| 3 | contour, density_2d, bin_2d, hex, polygon, quantile | contour, bin_2d, ecdf, qq | nudge |
geom_smooth draws ggplot2’s 95% confidence ribbon (se=True). Linear
regression standard errors need the Student t quantile, implemented in Plenty
from the regularised incomplete beta function. Loess follows R’s
stats::loess defaults (span 0.75, degree 2, tricube weights). ggplot2 picks
loess below 1,000 rows and mgcv GAMs above; GAMs are not planned (see
decision 5). std.stat already provides means, variances,
quantiles and normal, uniform and exponential distributions (Normal has
cdf and inverse_cdf); std.plot should use it and add the Student t
distribution there rather than in the plot module.
Jitter uses a seeded std.random.Rng (default seed fixed, settable with
position_jitter(seed=...)), so output stays deterministic.
Scales and guides
| Milestone | Scales |
|---|---|
| 1 | scale_{x,y}_continuous, _discrete, _log10, _sqrt, _reverse; scale_{colour,fill}_hue (default discrete), _gradient (default continuous), _gradient2, _viridis_d, _viridis_c, _manual, _grey; scale_size, scale_alpha, scale_shape; xlim, ylim, lims |
| 2 | _brewer, _distiller, _gradientn, _identity, scale_linetype, date and datetime scales over std.time and dataframe dates, guides(), guide_legend, guide_colourbar options |
| 3 | Binned scales, sec_axis, custom transforms |
Arguments follow ggplot2: name, breaks, labels, limits, expand,
na_value, palette. Breaks use Wilkinson’s extended algorithm (as R’s
scales::extended_breaks does). Labels for continuous scales format numbers
with the same rules as scales::label_number defaults; labels= accepts a
list of strings. A function-valued labels= would put a closure in the plot;
instead a small LabelFormat enum covers the common cases (Percent,
Comma, Dollar, Scientific, Number(accuracy)).
Colours are names (R’s 657 colour names, grey0–grey100) or hex
"#RRGGBB" and "#RRGGBBAA". The default discrete palette is ggplot2’s HCL
hue palette (hue 15–375, chroma 100, luminance 65); continuous is the
#132B43–#56B1F7 gradient. ColorBrewer and viridis tables ship as generated
Plenty data. Legends are generated from every non-position scale; scales with
the same title and levels merge into one legend, as in ggplot2.
Coordinates and facets
- Milestone 1:
coord_cartesian(xlim=, ylim=, expand=),coord_flip(),coord_fixed(ratio=);facet_wrap(facets, nrow=, ncol=, scales=)andfacet_grid(rows=, cols=, scales=, space=)withScales.Fixed,Free,FreeX,FreeY. Facet variables are column names or lists of names. - Milestone 2:
labellerchoices (LabelBoth,LabelValue),margins. - Milestone 3:
coord_polar,coord_radial,coord_trans.
Formula strings ("drv ~ cyl") are not accepted; rows= and cols= are the
single spelling, matching ggplot2’s vars() interface. A string containing
~ fails with a hint.
Themes and labels
The default is theme_grey (grey panel, white grid), the look ggplot2 users
recognise. theme_bw, theme_minimal, theme_classic, theme_light,
theme_dark, theme_void and theme_linedraw take base_size,
base_family, ink and paper.
theme(...) takes ggplot2’s element names with dots replaced by underscores,
as plotnine does: legend_position, axis_text_x, panel_grid_minor,
plot_title. Each parameter defaults to “inherit”, so a theme call changes
only what it names. Element inheritance (axis_text_x inherits axis_text,
which inherits text) follows ggplot2’s element tree. ggplot2 has about 130
theme elements; the full set fits within the 255-parameter limit, and
milestone 1 ships the ~40 most used. Element constructors are
element_text(size=, face=, colour=, angle=, hjust=, vjust=, margin=),
element_line, element_rect and element_blank(). Enumerated settings are
enums (Legend.Bottom, Face.Bold) rather than strings, so typos fail to
compile.
labs(title=, subtitle=, caption=, tag=, x=, y=, colour=, fill=, ...),
xlab(), ylab() and ggtitle() set labels. Without them, axis and legend
titles are the mapped column names, as in ggplot2.
Notes instead of messages
ggplot2 prints messages and warnings: the bin count chosen by stat_bin, the
smoothing method chosen by geom_smooth, rows removed for missing values.
Plenty has no warning channel, and a library must not write to standard error
on its own. Rendering returns a Rendered value whose notes list carries the
same information; print(rendered.notes)? shows it. save is the convenience
that writes the file and returns the notes. na_rm=True on a layer silences
the missing-value note, as in ggplot2.
Rendering and output
Formats
- SVG first (milestone 1). It is text, so output is easy to test, diff
and view, and needs no rasteriser.
p.save("out.svg", ...)orp.write_svg(&mut writer, ...)for anystd.iowriter. - PNG (milestone 2): an anti-aliased scanline rasteriser in Plenty
(signed-area coverage accumulation, the technique used by
font-rsandstb_truetype), then PNG chunks assembled in Plenty withstd.compress.compress(Format.Zlib, ...)forIDATandstd.compress.crc32for chunk checksums. - PDF later, if requested; it shares the SVG canvas model.
save chooses the format from the extension, with width, height, units
(Units.In, Cm, Mm, Px; inches by default) and dpi (default 300), as
ggsave does.
Text and fonts
Layout needs text measurement even for SVG: axis label widths decide panel
size, and legend widths decide plot width. ggplot2 asks the graphics device.
std.plot uses bundled font data:
- Metrics (advance widths and kerning pairs) for a default sans-serif font
are generated into the module, so measurement needs no file system access and
gives the same layout on every machine. The recommended font is Liberation
Sans (SIL Open Font License), which is metric-compatible with Arial and
Helvetica. An SVG names
font-family="Liberation Sans, Arial, Helvetica, sans-serif", so viewers without the bundled font substitute one with the same widths and the measured layout still holds. - Outlines for PNG and for SVG with
text=Text.Paths(exact rendering without viewer fonts) come from a subset of the same font, embedded as ab"..."byte literal and parsed in Plenty (cmap,hmtx,kern,glyfquadratic outlines). The subset covers Latin-1, Latin Extended-A, Greek, and plotting symbols (minus sign, multiplication sign, degree, micro, en dash). Emission removes it from programs that never draw outlines. font_file(path)loads another TrueType font at run time;base_familythen names it.- Missing glyphs render as the font’s
.notdefand add a note. - Shaping is left-to-right with kerning only. Complex scripts are out of scope; if they are needed later, shaping is the one piece where wrapping an established engine is justified, subject to the allocation rule below.
Deterministic output
Given the same plot and toolchain, output bytes are identical:
- coordinates are written with fixed precision (two decimals in SVG user units), elements in layer order, clip and gradient identifiers numbered sequentially;
- no timestamps, random identifiers or host font lookup;
- jitter uses a fixed default seed;
- the rasteriser is Plenty floating-point code with no target-dependent fast
paths, and PNG compression uses a fixed level of the pinned
zlib-rs.
Tests compare SVG snapshots, like ggplot2’s vdiffr (which also renders
through an SVG device with bundled Liberation fonts), and compare decoded PNG
pixels rather than compressed bytes.
Write in Plenty or wrap
| Part | Choice | Reason |
|---|---|---|
| Grammar: mappings, stats, scales, guides, facets, layout | Plenty | No crate implements it; plotters does not. This is most of the work in any case. |
Statistics (bins, lm, loess, density, quantiles) | Plenty | Formulas over lists; builds on std.stat |
| Colour (HCL conversion, palettes) | Plenty | Formulas and tables |
| SVG writer | Plenty | Text generation, a few hundred lines |
| Font parsing and text measurement | Plenty | Table parsing over bytes; ab_glyph and font-kit allocate infallibly, and font-kit adds host font discovery |
| Rasteriser | Plenty | Short, numeric code the compiler should handle well; tiny-skia allocates infallibly. Measure before milestone 2 closes |
| zlib compression and CRC-32 | Existing native std.compress (zlib-rs) | Already approved, reports allocation failure |
| Complex-script shaping | Not planned; the only candidate to wrap | A large established engine |
plotters is not recommended as a backend. It would replace only the SVG
writer and rasteriser, the smallest parts of the work. Its layout model (label
areas sized in pixels by the caller) does not match ggplot2’s, which sizes
from measured text, so std.plot would compute layout anyway and pass
primitives across a foreign boundary. It allocates through Rust’s global
allocator, which aborts on failure; the
native-wrapper rule rejects that
for new wrappers. Its bitmap backend brings image and font-kit, with
system font lookup that breaks deterministic output. Its role here is a
reference for primitive drawing and a comparison for rasteriser tests.
If the Plenty rasteriser cannot reach acceptable speed (target: a 2,400 ×
1,800 PNG with 100,000 points in under 0.5 s on the reference laptop), the
response is codegen work or std.simd, per the
Plenty-first preference, before
any wrapper.
Costs
- Source size. Milestone 1 is likely 10–15k lines of Plenty, comparable
to
std.markdownplusstd.regex. Importing a large std module costs front-end time; measure the compile time of a one-plot program at each milestone and keep it within a budget agreed before milestone 1 (proposed: under 150 ms over hello world with a warm cache). - Executable size. Emission drops unused std functions, so a program that only writes SVG carries neither the rasteriser nor the font outlines. Add a size check like the existing executable-size budgets.
- Runtime. Each stage is linear in rows except sorting for lines, quantiles and loess (O(n²) per fit before interpolation). Rendering a million-point scatter to SVG produces a file of tens of megabytes, as in ggplot2; PNG is the format for large data.
Phased plan
- Core and SVG.
Table,table,columns,rows;aes,factor,after_stat; milestone-1 geoms, stats, positions, scales, coords, facets and themes; layout with bundled metrics; SVG output;PlotError; notes; reference page, tutorial lesson, SVG snapshot tests. Exit: the six examples render and read correctly next to ggplot2’s output. - Breadth and PNG. Milestone-2 geoms and scales, loess, jitter, dates,
glyph outlines, rasteriser and PNG,
text=Text.Paths. - Records by name.
records()over B77’sEncoderonce B77 lands. - Dataframes. The bridge module with
Expraesthetics and required-column extraction; later, stat push-down into Polars. - Milestone-3 geoms and coords, as demand shows.
Stages 3 and 4 depend on other work and can move earlier if it lands first.
Decisions
The review accepted these choices. Each records the alternative considered.
- ggplot2 names verbatim, not Plenty-style names such as
plot.point(). Recognition is the goal, and plotnine shows the names work outside R. Users import withfrom std.plot import ggplot, aes, geom_pointorimport std.plot as gg; wildcard imports are not proposed. - Both
colourandcolorare accepted; a call that gives both isPlotError.Conflict. This is two spellings for one thing, against the preference for single rules, but ggplot2 users write both and the cost is one parameter per colour aesthetic. - Composition uses the component list and
plus. No operator hooks are added for this: the?after every step would make+worse than the list. - ggplot2’s implicit defaults stay (
bins=30, the smoothing method), each reported in the notes as ggplot2’s messages do. The plot shows the effect immediately, and requiring them would break the “write it as in ggplot2” goal. geom_smoothuses loess at every size, where ggplot2 switches to anmgcvGAM above 1,000 rows. Large inputs are evaluated on an interpolation grid as R’sloess(surface = "interpolate")does, and the note says so. Until loess lands in milestone 2, the default isSmooth.Lm, also noted.- No typed column handles (
aes(x=Car.wt)) for now. Strings checked at construction match ggplot2’s behaviour. Revisit after B77 lands: the per-column record accessor (num("wt", def(c: &Car) -> f64: c.wt)?) is verbose, and ifrecords()does not remove most of that boilerplate, or mistyped column names prove common, typed handles become worth a language change. - Default font: Liberation Sans, for Arial/Helvetica metric compatibility and the OFL licence. DejaVu Sans has wider coverage but different metrics from common viewer fonts.
- PNG uses a Plenty rasteriser, not a
tiny-skiawrapper, with the speed target in Write in Plenty or wrap as the acceptance test. - Missing values: rows missing any aesthetic a layer uses are dropped and
counted in the notes, as ggplot2 does;
na_rm=Truesilences the note. std.plotbelongs instd, besidestd.dataframe, not in a first-party package. The scope aims at data tools without third-party packages, and emission removes what a program does not use.- The dataframe bridge is a separate submodule under
std.plot’s namespace (asstd.httphas submodules), imported only by dataframe users, so neither module forces the other’s dependencies.
Risks
- Scope. ggplot2 has about 50 geoms, 30 stats and over 100 scales. The milestones cut this to the plots most people make; the risk is users arriving with the missing tenth.
- Fidelity expectations. Users will compare against ggplot2 output.
Differences in fonts, break placement or smoothing will be reported as bugs.
Matching defaults and testing against R reference values (breaks,
lmstandard errors, loess fits, histogram bins) limits this. - Text. Without shaping, labels in complex scripts render incorrectly.
SVG viewers without metric-compatible fonts can misplace text slightly;
text=Text.Pathsis the exact fallback. - Runtime checking of names. Mapping errors appear when the program runs, not when it compiles. Eager validation at construction and suggestions limit the cost.
- Compile time and binary size of a large std module; the budgets above make regressions visible.
- Numerical accuracy of statistics written from scratch. Test against R.
- Dependencies on unimplemented work:
records()waits for B77; the dataframe bridge depends on the dataframe module’s typed readers staying stable.
References
- Hadley Wickham, “A Layered Grammar of Graphics”, Journal of Computational and Graphical Statistics 19(1), 2010.
- ggplot2 reference index and changelog (4.0.3).
- plotnine reference, in particular
aesandggplot(0.15.8). - plotters documentation (0.3.7).
- Derived serialization, Polars dataframes, standard library scope.
std.plot examples beside ggplot2 and plotnine
Status: Not implemented. These examples illustrate the
std.plot proposal. The Plenty fences show proposed API and
are not compiled. Each example assumes a table with the columns of the R dataset
named (mtcars, diamonds, mpg) and these imports:
from std.plot import ggplot, aes, factor, after_stat, labs, theme, theme_minimal
from std.plot import geom_point, geom_smooth, geom_histogram, geom_bar
from std.plot import facet_wrap, facet_grid, position_dodge, scale_colour_brewer
from std.plot import Smooth, Brewer, Legend
Scatter plot with a colour mapping
ggplot(mtcars, aes(wt, mpg, colour = factor(cyl))) +
geom_point(size = 2)
(ggplot(mtcars, aes("wt", "mpg", color="factor(cyl)"))
+ geom_point(size=2))
p = ggplot(mtcars, aes("wt", "mpg", colour=factor("cyl")), [
geom_point(size=2),
]?)?
p.save("cars.svg", width=6, height=4)?
factor("cyl") makes the integer column discrete, so the legend shows three
levels instead of a gradient. size=2 is accepted because numeric settings take
i64 as well as f64.
Points with a fitted line
ggplot(mtcars, aes(wt, mpg)) +
geom_point() +
geom_smooth(method = "lm")
(ggplot(mtcars, aes("wt", "mpg"))
+ geom_point()
+ geom_smooth(method="lm"))
p = ggplot(mtcars, aes("wt", "mpg"), [
geom_point(),
geom_smooth(method=Smooth.Lm),
]?)?
The line carries a 95% confidence band by default (se=True), as in ggplot2.
Histogram
ggplot(diamonds, aes(price)) +
geom_histogram(binwidth = 500)
ggplot(diamonds, aes("price")) + geom_histogram(binwidth=500)
p = ggplot(diamonds, aes("price"), [
geom_histogram(binwidth=500),
]?)?
Without binwidth or bins, the stat uses 30 bins and records a note, the
counterpart of ggplot2’s `stat_bin()` using `bins = 30` message. Density
instead of counts: aes("price", after_stat("density")).
Bars with fill and dodge
ggplot(diamonds, aes(cut, fill = clarity)) +
geom_bar(position = "dodge")
(ggplot(diamonds, aes("cut", fill="clarity"))
+ geom_bar(position="dodge"))
p = ggplot(diamonds, aes("cut", fill="clarity"), [
geom_bar(position=position_dodge()),
]?)?
geom_bar counts rows per cut and clarity, so y is not mapped.
position_dodge(width=0.8) takes the same arguments as in ggplot2.
Facets
ggplot(mpg, aes(displ, hwy)) +
geom_point() +
facet_wrap(~class, ncol = 3)
ggplot(mpg, aes(displ, hwy)) +
geom_point() +
facet_grid(drv ~ cyl)
ggplot(mpg, aes("displ", "hwy")) + geom_point() + facet_wrap("class", ncol=3)
ggplot(mpg, aes("displ", "hwy")) + geom_point() + facet_grid("drv ~ cyl")
base = ggplot(mpg, aes("displ", "hwy"), [geom_point(),]?)?
wrapped = base.plus([facet_wrap("class", ncol=3)]?)?
grid = base.plus([facet_grid(rows="drv", cols="cyl")]?)?
base stays usable after each plus; both plots share its table.
Labels, scales and a theme
ggplot(mtcars, aes(wt, mpg, colour = factor(gear))) +
geom_point(size = 3) +
scale_colour_brewer(palette = "Set1") +
labs(title = "Fuel economy by weight",
x = "Weight (1000 lb)", y = "Miles per gallon", colour = "Gears") +
theme_minimal(base_size = 12) +
theme(legend.position = "bottom")
(ggplot(mtcars, aes("wt", "mpg", color="factor(gear)"))
+ geom_point(size=3)
+ scale_color_brewer(type="qual", palette="Set1")
+ labs(title="Fuel economy by weight",
x="Weight (1000 lb)", y="Miles per gallon", color="Gears")
+ theme_minimal(base_size=12)
+ theme(legend_position="bottom"))
p = ggplot(mtcars, aes("wt", "mpg", colour=factor("gear")), [
geom_point(size=3),
scale_colour_brewer(palette=Brewer.Set1),
labs(title="Fuel economy by weight",
x="Weight (1000 lb)", y="Miles per gallon", colour="Gears"),
theme_minimal(base_size=12),
theme(legend_position=Legend.Bottom),
]?)?
Brewer.Set1 and Legend.Bottom are enum values, so a misspelt palette or
position fails to compile.
A complete program from records
from std.plot import ggplot, aes, factor, geom_point, geom_smooth, labs
from std.plot import rows, num, integer, text, Smooth, PlotError
class Car:
model: str
wt: f64
mpg: f64
cyl: i64
def main() -> Result[(), PlotError | AllocError | IoError]:
cars = [
Car("Mazda RX4", 2.620, 21.0, 6),
Car("Datsun 710", 2.320, 22.8, 4),
Car("Hornet Sportabout", 3.440, 18.7, 8),
]?
table = rows(&cars, [
text("model", def(c: &Car) -> str: c.model)?,
num("wt", def(c: &Car) -> f64: c.wt)?,
num("mpg", def(c: &Car) -> f64: c.mpg)?,
integer("cyl", def(c: &Car) -> i64: c.cyl)?,
]?)?
p = ggplot(table, aes("wt", "mpg", colour=factor("cyl")), [
geom_point(size=2),
geom_smooth(method=Smooth.Lm, colour="grey30"),
labs(x="Weight (1000 lb)", y="Miles per gallon", colour="Cylinders"),
]?)?
notes = p.save("cars.svg", width=6, height=4)?
print(notes)?
Ok(())
geom_smooth(colour="grey30") is a setting: one fixed colour, so the line
does not split by the plot’s colour mapping. In ggplot2 the same setting also
overrides the inherited mapping for that layer.
After derived serialization lands, the rows
call can become records(&cars)? for a class marked @serial.
A mistake and its error
p = ggplot(table, aes("wt", "mpg", colour="cly"), [geom_point(),]?)?
error: plot mapping: aesthetic `colour` maps unknown column "cly"
available columns: model, wt, mpg, cyl
did you mean "cyl"?
The error comes from ggplot(...), before rendering. ggplot2 reports
object 'cly' not found when the plot is printed.
Exact decimal arithmetic (std.decimal)
Status: Not implemented. Nothing shipped; no reference page exists. Researched against local
mainat70e85758on 2026-10-31. Dataframe decimal columns wait on this design (B76). No backlog entry exists yet. Revised twice on 2026-10-31 after review; every decision is now settled.
Recommendation
Add std.decimal with one type, Decimal: a fixed-size, exact decimal of up
to 38 significant digits and 0–38 fractional digits, written in Plenty as an
allocation-free copy class.
It follows CPython’s decimal wherever that does not conflict with the rules
below, since CPython’s semantics are the most widely used and tested:
- The scale is kept, as in Python:
Decimal.parse("1.50")displays as1.50, and1.5 * 2.00is3.000. Equality and ordering are numeric across scales (1.0 == 1.00), and equal values hash equal, as in Python. +,-and*are exact. A result that does not fit traps as integer overflow does, and inside achecked:block it becomesErr(OverflowError[Decimal]), as fori64. There are no decimal-specific checked methods, contexts or signals.- Division takes a
NonZero[Decimal]divisor, so division by zero cannot be written. - Nothing rounds unless the call names places and a rounding mode, including
conversion from
f64. Division therefore has no/operator; it isa.div(divisor, places, mode). There is no context, no precision setting, and no silent rounding. - There is no NaN, infinity, or negative zero.
- The digit limit matches Arrow/Polars
Decimal128and SQLDECIMAL(38, s), so dataframe and database values convert without loss.
Do not wrap rust_decimal, libmpdec, or decNumber for the default type.
Leave arbitrary precision to a separate, later design if a workload needs it.
The type needs language work first (staged plan below): a 64×64→128-bit
integer multiply; @arithmetic methods and hooks, so library types join the
integer trap and checked: rules; the __cmp__ hook from the
ordering proposal, so == is numeric; and a wider
NonZero bound. Dictionary keys and decimal format specs follow later.
What financial code needs
| Need | Why | This design |
|---|---|---|
| Exact representation of decimal fractions | 0.1 + 0.2 must equal 0.3; amounts must round-trip through text | Coefficient × 10-scale, scale kept |
| Exact addition, subtraction and multiplication | Totals, quantities × prices, rates × amounts | Exact or overflow; never rounded |
| Explicit rounding with named modes | Tax, interest and currency rules specify both the place and the rule (banker’s vs commercial) | quantize(places, mode); seven modes |
| Division to stated places | Unit prices, rates, averages | div(divisor, places, mode) with a NonZero[Decimal] divisor |
| Overflow detection | A wrong total is worse than a crash | The integer trap, or Err inside checked:; never wrap |
| No surprise from binary floats | 0.1 as f64 is not 0.1 | from_f64 requires places and a mode |
| Parsing and formatting without loss | CSV, JSON, SQL, user input | parse keeps every digit or fails; display keeps the scale |
| Splitting an amount without losing units | Instalments, pro-rata allocation | allocate (stage 2) |
| Interchange with columnar and SQL decimals | Polars, Parquet, databases | Same 38-digit range; scale kept like a column’s |
Not needed for the common case, and excluded from the default: transcendental
functions (ln, exp, sqrt), NaN payloads, signalling NaNs, subnormals,
precision contexts, and numbers beyond 38 digits.
Candidates
The user’s phrase “the Rust decimal crate” most likely means
rust_decimal (168 M downloads, the
de facto Rust choice). The crate actually named
decimal is an older binding to IBM’s
decNumber decimal128 (last release 2.1.0, 2021). Both are covered.
| Candidate | Model | Size and allocation | Rounding and errors | Licence, form |
|---|---|---|---|---|
CPython decimal / libmpdec | General Decimal Arithmetic (GDA) spec, arbitrary precision; context with precision (default 28), rounding (default half-even), traps and sticky flags; NaN, ±Inf, −0; trailing zeros significant | Heap coefficient per value. Allocation is fallible: mpd_switch_to_dyn sets MPD_Malloc_error and returns NaN (mpalloc.c) | Every operation rounds to context precision; traps raise, otherwise flags are set. Python keeps the context in a contextvars variable: hidden per-thread state | BSD-2-Clause C, about 18,600 lines. CPython removed its bundled copy in May 2026 (9eb3b146686) and now requires a system libmpdec ≥ 2.5 |
decNumber via decimal or dec | GDA; fixed decimal128 (34 digits, IEEE 754-2008) plus arbitrary decNumber | decimal128 is 16 bytes and allocation-free; decNumber may malloc | Context and status flags, as libmpdec | ICU licence C (about 16,800 lines), Apache-2.0 bindings; decimal unmaintained since 2021 |
rust_decimal 1.43 | 96-bit coefficient, scale 0–28, 16 bytes; at most 28–29 digits | No allocation for arithmetic | Rounds silently: a product whose scale exceeds 28 is rounded half-even, and one too small becomes zero (src/ops/mul.rs); division rounds to 28 digits. Operators panic on overflow; checked_* return Option. Eq/Hash are numeric (normalized) | MIT Rust, about 12,300 lines |
fastnum 0.7 | D128: 128-bit coefficient plus a 64-bit control block (scale, sign, flags), 24 bytes; NaN, ±Inf, ±0 | No allocation | GDA-like context and signals. Default traps (division by zero, invalid, overflow) panic only in debug builds (context/signal_traps.rs); release builds return special values | MIT/Apache-2.0 Rust, about 25,700 lines; MSRV 1.94 |
bigdecimal | num-bigint coefficient + i64 scale | Vec per value; Rust’s global allocator aborts on failure | Explicit precision on division | MIT/Apache-2.0 Rust |
Polars Decimal(p, s) (polars-compute 0.55.2 decimal.rs) | i128 unscaled values, precision ≤ 38, scale fixed per column | Column storage | Multiplication and division round half-even back to the column scale; overflow returns None | MIT Rust; reference algorithms, not an API |
Plenty-native Decimal (recommended) | Two base-1019 u64 limbs + scale + sign, as libmpdec stores words; 38 digits, scale 0–38, scale kept | Four inline fields; never allocates | Exact or trap; rounding only on request | Plenty source in stdlib/std/decimal.plenty |
Why not wrap
Project policy (AGENTS.md, engine criteria)
requires a concrete reason to wrap, such as substantial complexity or
specialist security work. A fixed 128-bit decimal has neither:
- The core is integer arithmetic on two 64-bit limbs plus scaling by powers of
ten. Polars implements the equivalent
i128operations, including correctly rounded 256-by-128-bit division, in one 1,171-line file. - Every wrapped candidate carries semantics the recommended rules reject:
rust_decimalrounds silently and stops at 28 digits, below the 38 that Polars and SQL use;fastnumreturns NaN in release builds; libmpdec and decNumber need a context on every call and C builds for four targets, including MSVC. - A native call per addition costs more than the addition. A Plenty implementation inlines through ordinary Cranelift code.
- Plenty code links per module, so a program that does not import
std.decimalpays nothing either way (per-module linking, checked bytests/test_executable_size.rs).
Wrapping libmpdec remains the right choice if users later need the full GDA
operation set (ln, exp, sqrt, pow with fractional exponents) at
arbitrary precision. Its allocation failure is already reported as a status
flag, so a wrapper can map it to AllocError without patching.
Semantic choices
Fixed 38 digits, not arbitrary precision
Fixed size is what makes the type a copy class: no allocation, no Result
on every operation, no ownership bookkeeping. 38 digits holds any amount below
1020 currency units with 18 fractional digits (enough for
Ethereum’s 10-18), and is the limit of Arrow Decimal128, Polars,
Parquet, SQL Server, BigQuery NUMERIC and Snowflake NUMBER. IEEE
decimal128 has only 34 digits and a binary-incompatible encoding (BID/DPD);
rust_decimal has 28.
Arbitrary precision moves allocation into every arithmetic expression,
(a * b)? everywhere, and needs a rule for how large results may grow. If a
workload needs it, design it separately; this proposal does not depend on
one.
Representation
Store the coefficient in two u64 words in base 1019, as libmpdec
does on 64-bit targets (MPD_RADIX is 1019, MPD_RDIGITS 19, in
mpdecimal.h). Two words hold exactly 38 digits, so the digit limit is “high
word below 1019”; aligning scales, rounding at a digit, and parsing
and printing split at word boundaries with powers of ten that fit one word.
libmpdec multiplies words with a double-word product (_mpd_mul_words in
typearith.h: __uint128_t or x86-64 mulq); u64.widening_mul is the
Plenty equivalent. Reducing a product by the constant 1019 is a
multiply by a precomputed reciprocal, so the same primitive serves; the
prototype should confirm that before choosing between base 1019 and
binary words. This is an implementation choice, not part of the API.
Scale, equality and hashing
The scale is kept, with Python’s ideal exponents (Lib/_pydecimal.py):
| Operation | Result scale | Example |
|---|---|---|
parse | As written | "1.50" → 1.50 |
a + b, a - b | max(scale_a, scale_b) | 1.5 + 2.25 → 3.75; 1.50 + 1.50 → 3.00 |
a * b | scale_a + scale_b | 1.5 * 2.00 → 3.000 |
quantize(places, mode), div(d, places, mode) | places | 1.5.quantize(2, …) → 1.50 |
normalize() | Smallest exact scale | 3.000 → 3 |
A result needing more than 38 digits, counting trailing zeros, or more than 38
fractional digits overflows. Python would round it to the context precision
instead; here that is a trap, or Err in checked:.
Python compares numerically: Decimal("1.0") == Decimal("1.00") is True
and hash(Decimal("1.0")) == hash(Decimal("1.00")) (__eq__ uses _cmp;
__hash__ hashes coefficient × 10-exp modulo the hash prime).
Decimal matches both. This needs three things from Plenty:
- Equality and ordering:
__cmp__. Plenty’s==on classes is structural and cannot be overridden (operator boundaries), so with the scale kept, structural==would make1.0 == 1.00false. The ordering proposal (B37) already gives the fix: defining__cmp__(self, other: &T) -> Orderderives all six comparisons, and replaces structural==and!=for the type (its rule 3), including inside collections.Decimal.__cmp__aligns scales and compares coefficients; it allocates nothing.Decimalcannot ship before this, because its==would be wrong. A separate__eq__is not needed and the ordering proposal rejects one; Python’s own six methods all reduce to_cmp. - Keys: a key projection. Under the hashing proposal
(B74), keys hash structurally, so a type defining
__cmp__is not a key type. Its key projection extension is the consistent route:Decimal.__key__()returns the normalized parts (coefficient without trailing zeros, scale, sign), a structural key value, so equal decimals hash and look up equally. That proposal’s main objection, that projections usually allocate, does not apply: normalizing is integer division by ten on two words. Until it lands,Decimalis not a key; code can key ond.normalize().parts(). - Consistency.
__key__equality and__cmp__equality must agree, which the compiler cannot check. Restrict both hooks tostdat first, as for@arithmetic, and test the agreement with differential tests against CPython.
No context, no special values
A GDA context is hidden mutable state (Python stores it in a contextvars
variable), and its default precision of 28 silently rounds results that do
not fit. Instead, an operation that can be inexact takes its places and
rounding mode as arguments, and the others are exact. With no context there
are no sticky flags to check, and with errors as values there is no need for
NaN or infinity. Negative zero carries no financial meaning: parse("-0.00")
is 0.00, where Python keeps -0.00 (it still compares equal to zero).
Floats
Python’s Decimal(0.1) and Decimal.from_float(0.1) convert exactly, giving
0.1000000000000000055511151231257827021181583404541015625; mixing floats
into the constructor and comparisons is allowed unless the FloatOperation
signal is trapped. That exact expansion is the confusion this module exists
to avoid, and at 55 digits it does not fit anyway.
Decimal has no implicit or exact float conversion:
Decimal.from_f64(value, places, mode)rounds the float’s exact binary value toplaceswithmode:from_f64(0.1, 2, Rounding.HalfEven)is0.10. NaN, infinity, or a result over 38 digits isErr(RangeError.OutOfRange).- Operators and comparisons never accept
f64; mixing is a compile error, as mixingi64andf64is. parseis the way to write a decimal constant:Decimal.parse("0.1").to_f64()returns the nearestf64.
Overflow: the integer trap and checked:
+, -, * and unary - on Decimal follow the
integer rules:
- Outside a
checked:block, an unrepresentable result is an arithmetic trap at the caller’s position, with the standard-library caller note. Actors and test harnesses see it asTrapKind.Arithmetic, like integer overflow. - Written inside a
checked:block, the same expression makes the blockErr(OverflowError[Decimal]). A block mixing integer and decimal arithmetic gets both errors in its anonymous sum, for exampleOverflowError[i64] | OverflowError[Decimal].
The same rule covers the methods that can overflow (quantize, div,
mul_quantize, div_rem), as it covers pow on integers.
OverflowError[T]’s wrapped field is the exact result modulo
2N for integers. For Decimal it is the decimal analogue: the
exact result truncated toward zero to 38 fractional digits, with its
coefficient reduced modulo 1038. It is deterministic and, like the
integer field, mainly diagnostic.
Gap: library types in the trap rules
Today the rules do not reach library code. A checked: block checks only
builtin integer operators, pow, and operators that @newtype classes
forward; an operator hook such as Complex.__add__ is an ordinary call and
keeps trapping inside a block. Standard-library code also has no way to raise
the arithmetic trap for a condition that is not an integer overflow, such as a
39-digit decimal result.
The proposed fix is one general mechanism, not a decimal one. An
@arithmetic decorator on a method or operator hook whose body returns
Result[T, OverflowError[T]] (optionally | ZeroDivisionError) changes how
callers see it:
- Callers see the method as returning
T. Outsidechecked:, the compiler turns anErrinto the arithmetic trap at the call site, so the caller note andTrapKind.Arithmeticwork as for integers. - A call or operator written lexically in a
checked:block adds the declared errors to the block’s error type and exits the block onErr, as integer operators do. - The lexical rule is unchanged: integer arithmetic inside the hook body is ordinary code, and the hook decides what overflow means for its type.
- Users never call a separate checked form.
checked:is the only way to receive the error.
Closing the gap needs checker work (validate the decorator’s signature, type
the call as T, contribute errors to enclosing blocks, and restrict the
decorator to std) and code generation (the Err branch to the existing
trap and the block’s exit edge). Other library types could adopt
it later; std.time.Duration, which has checked_* methods returning
RangeError, is one candidate, out of scope here.
Rounding modes
Seven modes, CPython’s set without ROUND_05UP, named by direction rather
than by Python’s ambiguous ROUND_UP and ROUND_DOWN:
| Plenty | Python / GDA | Example at 0 places: 2.5, −2.5, 2.4 |
|---|---|---|
HalfEven | ROUND_HALF_EVEN | 2, −2, 2 |
HalfAwayFromZero | ROUND_HALF_UP | 3, −3, 2 |
HalfTowardZero | ROUND_HALF_DOWN | 2, −2, 2 |
TowardZero | ROUND_DOWN | 2, −2, 2 |
AwayFromZero | ROUND_UP | 3, −3, 3 |
Floor | ROUND_FLOOR | 2, −3, 2 |
Ceiling | ROUND_CEILING | 3, −2, 3 |
GDA’s ROUND_05UP is omitted. Every rounding call names its mode; there is no
default.
Division: a non-zero divisor
Division by zero is ruled out by the divisor’s type, following the decision
behind NonZero[T] (issue #584): check once where the value is created, not
at every use. The divisor is NonZero[Decimal], the same builtin type the
integers use, because widening NonZero to library types later lets users
define non-zero forms of their own numeric types.
Widening the NonZero bound
NonZero[T: IntType] becomes NonZero[T: ZeroTest]. ZeroTest is a built-in
constraint, like IntType and the proposed KeyType, because a protocol cannot
admit integers (they have no methods). It is satisfied by:
- every integer type, unchanged: literals convert at compile time,
0is a compile error, and operators forward to the builtin integer operators; - a
@copyclass that definesdef is_zero(self) -> bool, computation only (no I/O, clock, entropy or process effects, as__cmp__requires).
For such a class T, NonZero[T] provides what it provides for integers,
built from T’s methods:
NonZero[i64] | NonZero[Decimal] |
|---|---|
Literal converts at compile time; 0 is a compile error | No decimal literals. Decimal.nonzero(units: NonZero[i64], scale) and Decimal.nonzero_int(value: NonZero[i64]) take integer units, so Decimal.nonzero(0, 2) is a compile error |
NonZero.try_from(value) -> Result[NonZero[i64], RangeError] | Same; OutOfRange when is_zero(), so 0.00 is rejected |
.get() -> i64 | .get() -> Decimal |
* and unary - keep the type; + and - return T | Same, through Decimal’s hooks |
Mixed NonZero[T] and T operands are a compile error suggesting .get() | Same |
Copy, ==, ordering, printing, dictionary key | Copy; == and ordering from Decimal.__cmp__; printing; keys when Decimal is one |
Decimal must provide @copy, is_zero (the coefficient is zero, at any
scale), and * and unary - hooks that never turn non-zero operands into
zero. Both hold: products are exact or overflow, and a product needing more
than 38 fractional digits overflows instead of rounding to zero. Inside
checked:, NonZero[Decimal] products report OverflowError[Decimal], as
NonZero[i64] products report OverflowError[i64].
That last requirement is a contract the compiler cannot check. A user type
whose * can produce zero (a matrix, a modular integer) would make
NonZero[T] lie. So ZeroTest for classes is std-only at first, like
@arithmetic; opening it to user types needs a rule for that contract.
The compiler work is in src/frontend/nonzero.rs: the bound, try_from
through is_zero, forwarding *, unary -, +, - and comparisons through
the class’s hooks rather than the integer operators, and the compile-time
0 check staying integer-only.
There is no flow narrowing (if d != 0: does not change d’s type), as for
integers. A runtime count converts once: Decimal.nonzero_int(NonZero.try_from(n)?).
Division API
a.div(divisor: &NonZero[Decimal], places: i64, mode: Rounding) -> Decimal
a.div_rem(divisor: &NonZero[Decimal], mode: Rounding) -> (Decimal, Decimal)
divreturns the quotient correctly rounded toplacesfractional digits (0–38; others trap likeDecimal.of) withmode. A quotient over 38 digits overflows under thechecked:rules above; it has no other failure.div_remrounds the quotient to an integer withmodeand returns the exact remaindera - q * divisor.TowardZerogives Python’sDecimal//and%(-7 // 2is-3, remainder-1);Floorgives Plenty’s integer//and%(-4, remainder1). The two disagree, so the mode is required rather than picking one.
Why still no /: a non-zero divisor removes the zero case but not rounding.
1 / 3 has no exact result, so / would need fixed places and a fixed mode.
Python rounds to the context precision, Polars to the column scale and
rust_decimal to 28 digits; each is the silent rounding this design excludes,
and a trap on every inexact quotient would make / fail for most inputs.
Integers differ today: i64 // NonZero[i64] is a compile error (the operand
must be i64), and integer // and % still trap on zero. Issue #584 left
NonZero divisors for // and % as separate work; std.decimal starts with
the rule that work would adopt.
Parsing and formatting
Decimal.parse(text) follows the numeric parsing contract:
it allocates nothing, trims white space, and accepts an optional sign, digits,
an optional fraction, and an optional exponent (1.5e3). The scale is the
written one, adjusted by the exponent and never below 0: "1.50" has scale 2,
"1.5e3" is 1500, "1E+2" is 100 (Python keeps exponent 2 and prints
1E+2). More than 38 digits or 38 fractional digits is ParseError.OutOfRange;
it never rounds. inf, nan, grouping and locale forms are
ParseError.Invalid.
display writes the plain form with the stored scale (-1234.50), never an
exponent; Python switches to exponent form for small values (1E-7). To show
a fixed number of places, quantize first: total.quantize(2, Rounding.HalfEven).
Today an f-string spec on a class applies to its text: probing with
std.complex, f"{Complex(1.25, 2.0):.3}" prints 1.2. For a decimal,
{total:.2} would therefore cut 64.92 to 64. Until format specs reach
library types (stage 3), the reference must warn against specs on Decimal.
Naming against Python
| Python | Plenty | Reason |
|---|---|---|
quantize(Decimal("0.01"), rounding) | quantize(2, mode) | Same operation and name; places as a count, since scale is never negative |
round(d, n) | Not provided | Python’s uses the context’s rounding, and round(d) returns int |
normalize() | normalize() | Same, but exact: no rounding to a precision |
compare, <, == | compare, <, == from __cmp__ | Same |
int(d) | to_i64(mode) | int truncates silently |
as_tuple() | parts() | Coefficient words, scale, sign |
ROUND_HALF_UP etc. | Rounding.HalfAwayFromZero etc. | Python’s UP/DOWN names read as directions they are not |
Fit with the language today
| Need | Today | Consequence |
|---|---|---|
| 128-bit coefficient | No u128/i128; u64 has wrapping and checked operations but no high product | Two u64 words; add u64.widening_mul (Cranelift umulhi, available on x86-64 and AArch64 in the pinned 0.137) |
| Arithmetic operators | Operator hooks as in std.complex | + - * and unary - |
checked: and traps | Integer operators, pow, @newtype forwarding; hooks keep trapping; std cannot raise the arithmetic trap | @arithmetic methods and hooks, std only |
Numeric == and < | Structural == only; no ordering hooks (B37) | __cmp__ from the ordering proposal; a prerequisite |
| Dictionary keys | Not available for classes (B74) | __key__ projection after B74 |
| Non-zero divisor | NonZero[T] for integer T only | ZeroTest bound, std classes only |
| Literals and constants | const accepts only scalars, str, bytes | Decimal.of(1999, 2) is allocation-free and infallible; parse for text |
| Format specs | Applied to display text | quantize then display; later a spec hook |
Decimal needs widening_mul on its own account: every product of two
coefficients, and the 256-by-128-bit division behind div and quantize, is
built from 64×64→128-bit products.
API sketch
from std.decimal import Decimal, Rounding
pub enum Rounding:
HalfEven
HalfAwayFromZero
HalfTowardZero
TowardZero
AwayFromZero
Floor
Ceiling
@copy
pub class Decimal:
# private: _high: u64, _low: u64 (base 10**19), _scale: u8, _negative: bool
# Construction: allocation-free.
Decimal.ZERO, Decimal.ONE
Decimal.of(units: i64, scale: i64) -> Decimal # of(1999, 2) is 19.99; traps outside 0..38
Decimal.from_int(value: i64) -> Decimal
Decimal.nonzero(units: NonZero[i64], scale: i64) -> NonZero[Decimal]
Decimal.nonzero_int(value: NonZero[i64]) -> NonZero[Decimal]
Decimal.parse(text: str) -> Result[Decimal, ParseError]
Decimal.from_f64(value: f64, places: i64, mode: Rounding) -> Result[Decimal, RangeError]
Decimal.from_parts(high: u64, low: u64, scale: i64, negative: bool) -> Result[Decimal, RangeError]
# Exact arithmetic. @arithmetic: traps on overflow; Err(OverflowError[Decimal]) inside checked:.
a + b, a - b, a * b, -a
a.abs() -> Decimal
# Rounding is always explicit. Also @arithmetic.
a.quantize(places: i64, mode: Rounding) -> Decimal
a.div(divisor: &NonZero[Decimal], places: i64, mode: Rounding) -> Decimal
a.mul_quantize(&b, places: i64, mode: Rounding) -> Decimal # exact product, then quantize
a.div_rem(divisor: &NonZero[Decimal], mode: Rounding) -> (Decimal, Decimal)
# Comparison: __cmp__ gives == != < <= > >= numerically across scales.
a.compare(&b) -> Order
a.is_zero() -> bool # ZeroTest
a.normalize() -> Decimal # strip trailing fractional zeros; exact
# Inspection and conversion.
a.sign() -> i64
a.scale() -> i64 # stored scale
a.is_integer() -> bool
a.parts() -> (u64, u64, i64, bool)
a.to_i64(mode: Rounding) -> Result[i64, RangeError]
a.to_f64() -> f64 # nearest f64
a.to_units(scale: i64) -> Result[(u64, u64, bool), RangeError] # unscaled value at a fixed scale; fails if not exact
# Text.
a.display(out) # plain form with the stored scale
Conversions report RangeError.OutOfRange, as integer try_from does; there
is no DecimalError. No arithmetic reports division by zero, since no
operation accepts a zero divisor.
Example, in the proposed API:
from std.decimal import Decimal, Rounding
from std.fmt import FormatError
def unit_price(total: Decimal, count: i64) -> Result[Decimal, RangeError]:
per = Decimal.nonzero_int(NonZero.try_from(count)?)
Ok(total.div(&per, 2, Rounding.HalfEven))
def main() -> Result[(), IoError | ParseError | RangeError | FormatError]:
price = Decimal.parse("19.99")?
rate = Decimal.parse("0.0825")?
subtotal = price * Decimal.from_int(3) # 59.97
tax = (subtotal * rate).quantize(2, Rounding.HalfEven) # 4.947525 -> 4.95
total = subtotal + tax
print(f"{total}"?)? # 64.92
print(f"{unit_price(total, 7)?}"?)? # 9.27
half = Decimal.parse("1.50")?
print(f"{half}"?)? # 1.50
print(half == Decimal.of(15, 1))? # True
Ok(())
# Errors as values instead of a trap, with the integer rule.
def grand_total(lines: &list[Decimal]) -> Result[Decimal, OverflowError[Decimal]]:
checked:
total = Decimal.ZERO
for line in lines:
total += line
total
Interchange
- Polars/Parquet (B76): enable Polars’
dtype-decimalfeature inplenty-dataframeand measure the dataframe executable’s growth. A columnDecimal(p, s)reads asDecimalvalues with scales, since the scale is kept; writing to a column needs each value at scalesexactly (to_units(s)fails withRangeErrorotherwise), so the caller quantizes first. Column arithmetic stays in Polars, with its own half-even rounding to the column scale, and the reference must say so. - JSON:
Event.Numberalready exposes number text (std.json); add a decimal reader and writer without going throughf64. - CSV, SQLite: parse fields and bind as text. SQLite has no exact decimal
storage class; binding through
REALwould lose digits.
Licences, size and platforms
A Plenty implementation ships no third-party code. If it ports algorithms from
Polars’ decimal.rs (MIT), credit Polars in the module header, the reference
page’s acknowledgements and scripts/release/third_party_notices.py. Tests
can use the GDA decTest vectors and CPython’s decimal as a differential
oracle (configured to 38 digits with traps for Inexact), without shipping
either. Code size reaches only programs that import the module. The
implementation uses only integer operations, so it behaves identically on
Linux x86-64 and ARM64, macOS ARM64 and Windows MSVC; C candidates would need
per-target builds and MSVC configuration.
Staged plan
- Prerequisites.
u64.widening_mul(other) -> (u64, u64)(high, low), lowered toumulhiandimul. Check the generated code before timing.@arithmeticmethods and operator hooks,stdonly (gap), with tests on a small test-only class: trap position and caller note,TrapKind.Arithmetic,checked:error types, and the lexical rule.__cmp__from the ordering proposal, at least forstdclasses, soDecimal’s==is numeric.- The
ZeroTestbound forNonZero,stdclasses only.
- Core module.
DecimalandRounding; parsing and display with the stored scale; exact+ - *;quantize,div,mul_quantize,div_rem,normalize, conversions;allocate(parts)for splitting an amount into parts that sum exactly. Reference page, a tutorial lesson replacingf64money arithmetic, differential tests against CPython (38 digits, traps forInexactandRounded), allocation tests proving no allocation. - Language integration. Dictionary keys through
__key__once B74 covers classes; a format-spec hook so{total:.2f}quantizes (half-even, as Python’sformatdoes) instead of truncating text. - Interchange. Polars decimal columns (B76), JSON number text, CSV helpers.
- Only on evidence. A standalone arbitrary-precision decimal design, or a
fallible libmpdec wrapper for the full GDA operation set; a decimal literal
suffix (
19.99d) ifparseandofprove too noisy;@arithmeticandZeroTestfor user types.
Decisions
Decided in review (2026-10-31):
- Scale. Kept, as in Python. Equality and ordering are numeric across
scales through
__cmp__, and keys hash the normalized value through__key__. - Digit limit. 38 digits.
- Rounding mode. Always explicit, including
from_f64anddiv_rem. - Multiply primitive.
u64.widening_mul, as libmpdec uses a double-word multiply on base-1019 words. - Non-zero divisor.
NonZero[Decimal], through aZeroTestbound;stdclasses now, user types later. wrapped. Truncated to 38 fractional digits, coefficient modulo 1038.@arithmetic.stdonly for now.- Overflow traps like integers and is
Err(OverflowError[Decimal])insidechecked:; nochecked_*methods, contexts or signals. - Division is
div(divisor, places, mode); no/. - No coupling to other numeric proposals. Arbitrary precision is a separate later design.
Still open, none blocking stage 1:
- Rounding names. Direction names (recommended) or CPython’s
ROUND_HALF_UPstyle. - User types. The rule for opening
@arithmetic,ZeroTestand__key__to user code: each rests on a contract the compiler cannot check (non-zero-preserving*,__key__agreeing with__cmp__). - Word base. Base 1019 (recommended, as libmpdec) or binary words, settled by the prototype; not user-visible.
A numerics core for scientific work
Status: Not implemented. This page scopes a numerical library for Plenty
and recommends what to build first. It ships no module or API. It replaces
issues #492 (optimization) and #493 (evaluating numeris). Scheduling belongs
to B38. Shipped neighbours are std.linalg,
std.stat, std.math,
std.complex, std.random,
std.dataframe and std.plot.
Companion pages sketch the API and record the
crate evaluation.
Goal
A graduate student or researcher in a STEM field should be able to load data,
fit a model, solve an equation, integrate a function or an ODE, and plot the
result using only std. In Python this needs NumPy and SciPy. Plenty should
cover the parts of them that most research code uses, and stop there. Most of
SciPy is not needed to reach that goal.
Recommendation
- One family,
std.numeric, in Plenty. New modules live understd.numeric:optimize,integrate,differentiate,interpolate,specialandfft, plus array helpers instd.numericitself. The algorithms are written in Plenty. Dense linear algebra inside them callsstd.linalg, which keeps faer. - Adopt neither
basinnornumerisas a dependency. Use them, with SciPy, QUADPACK, MINPACK and Cephes, as algorithm and test-vector references. The crate evaluation gives the reasons: objective functions are Plenty callbacks, neither crate reports allocation failure, and both are months old with frequent breaking releases. - Keep the existing modules where they are.
std.linalgandstd.statgain what the new modules need (least squares, more distributions); they are not moved understd.numeric. A reference index page lists the whole numerics core. - Exchange data as
list[f64]andla.Matrix. These are whatstd.dataframereturns fromvalues_f64, whatstd.plottakes inColumn.Num, and whatstd.linalgalready uses. No new array type. - Deliver in three milestones (below), each useful on its own, and add further scope only for a stated use case.
How large is SciPy, and what does Plenty take?
SciPy is a collection of subpackages, each a library in its own right, many wrapping decades of Fortran and C. Matching it would be a multi-year project and is not the goal. The table decides each subpackage. Core is in the milestones below. Later needs a use case and its own scope before work starts. Out is not planned.
| SciPy / NumPy area | Decision | Plenty scope |
|---|---|---|
numpy.ndarray, broadcasting | Out for now | list[f64], la.Matrix, iterators and std.simd. An N-dimensional array needs indexing overloads, views and a broadcasting design; it is a separate proposal if ever. |
| NumPy helpers | Core | linspace, logspace, cumsum, diff, dot, norm, accurate fsum, polynomial polyval/polyfit/roots |
numpy.random | Shipped | std.random; new distributions sample through std.stat |
scipy.optimize | Core subset | Scalar roots and minimization; Nelder–Mead, BFGS, L-BFGS-B; nonlinear least squares and curve_fit; root finding for systems; dense linear programming |
| Later | Differential evolution, SLSQP or other constrained NLP, mixed-integer LP, basin hopping | |
scipy.integrate | Core subset | Adaptive Gauss–Kronrod quad, including infinite ranges; trapezoid/Simpson over samples; solve_ivp with Dormand–Prince 5(4) and one stiff method |
| Later | Events, dblquad/nquad, boundary-value problems | |
scipy.differentiate | Core | Central differences with Richardson extrapolation; gradient, Jacobian and Hessian by finite differences |
scipy.interpolate | Core subset | Linear, cubic spline (natural, clamped, not-a-knot), PCHIP |
| Later | Smoothing splines, 2-D regular grids, B-spline objects | |
| Out | Scattered-data RBF and triangulation | |
scipy.special | Core subset | Gamma, log-gamma, digamma, beta, regularized incomplete gamma and beta with inverses, erf/erfc/erfinv |
| Later | Bessel functions, elliptic integrals, orthogonal-polynomial evaluation | |
| Out | Hypergeometric and the long tail | |
scipy.fft | Core subset | Complex FFT of any length (mixed radix plus Bluestein), rfft/irfft, fftfreq |
| Later | 2-D FFT, DCT | |
scipy.linalg | Shipped, extend | std.linalg plus lstsq and det, which least squares and fitting need |
| Later | Matrix functions (expm), condition estimates, pseudoinverse | |
scipy.stats | Shipped, extend | std.stat gains Student’s t, χ², F, gamma, beta, binomial and Poisson once special exists |
| Later | Hypothesis tests, linear regression summaries, kernel density estimation | |
scipy.signal | Later | Convolution, filtering, peak finding, Welch spectra (std.signal already names Unix signals) |
scipy.spatial | Later | k-d tree and distance functions |
scipy.sparse, sparse.linalg | Out | A sparse format and iterative solvers need their own proposal |
ndimage, io, odr, cluster, datasets, constants | Out | std.image covers images; physical constants may be a small later table |
The core is a few dozen algorithms, each documented in the literature with a reference implementation and published test problems. That is the size this proposal commits to.
Why std.numeric rather than top-level modules
std.linalg and std.stat are top-level, so std.optimize would match them.
Top-level names were rejected:
- Several SciPy names are already taken or ambiguous at the top level:
std.signalis Unix signal handling, andstd.specialsays nothing about numerics. - One prefix makes the family easy to find in the signature index and keeps shared types (tolerances, result records) in one parent module.
stdalready has this shape:std.cache.lruandstd.sketch.xorsit under parent modules.
Moving std.linalg and std.stat would break shipped programs for no gain;
they stay, and the reference index links them.
Why Plenty and not a wrapped crate
The standard library policy prefers Plenty unless a native library is clearly better. Here Plenty is better for three reasons:
- The user’s function is the inner loop. Root finding, minimization,
quadrature and ODE solving call a Plenty function thousands of times. A
callback bound by
F: Callable[...]is specialized at compile time and can be inlined. A native solver would call back across the engine’s C boundary on every evaluation and would have to carry the callback’s error out again. - Errors propagate unchanged. A probe with the release compiler built from
mainon 2026-11-02 compiledbisect[E, F: Callable[[f64], Result[f64, E]]](...) -> Result[f64, RootError | E]with a closure returningmath.MathError, and a finite-difference gradient takingCallable[[&list[f64]], Result[f64, E]]and writing into&mut list[f64]. Both ran correctly. A failing objective, such aslogof a negative number, leaves the solver with its own error, not a sentinel NaN. - Allocation failure. Plenty code reports
AllocErrorby construction. Neither candidate crate does (evaluation).
The dense kernels that justify native code (GEMM, LU, QR, SVD, eigenvalues)
are already native in std.linalg. Solvers needing them (Newton steps,
Levenberg–Marquardt, trust regions, interior-point LP) call std.linalg.
An FFT written in Plenty may be slower than a SIMD library. If measurement
shows that, the bigint pattern applies: an
allocation-free native kernel writing into caller-allocated buffers, added
only when it is measured to pay.
Shared conventions
The API sketch has signatures. The rules:
f64only. As instd.linalgandstd.stat.f32and generic float code wait until there is demand and a numeric protocol to bound them.- Fallible callbacks. Objectives return
Result[f64, E], and solvers returnResult[T, NumericError | E]. One form, not paired fallible and infallible variants; an infallible objective writesOk(...). - No allocation per evaluation. Vector objectives borrow
&list[f64]and write gradients into&mut list[f64]and Jacobians into&mut la.Matrixsupplied by the solver. Workspace is allocated once per solve. - Non-convergence is an error that keeps the result. Reaching an
iteration or evaluation limit returns
Err(NumericError.Unconverged(report)). The report holds the last iterate, function value and counts. SciPy returnssuccess=Falsein an ordinary result, which is easy to ignore. In Plenty the caller has to handle it, and still gets the last iterate. - Options are keyword arguments with documented defaults:
xtol,rtol,atol,max_iterations,max_evaluations. No options dictionaries and no method-name strings: each method is its own function, so a misspelt method fails to compile. - Explicit randomness. Stochastic methods take
&mut random.Rngfromstd.random; there is no global seed. - Nonfinite values. Inputs are checked; a NaN or infinity from a callback
is
NumericError.NonFinite, reported at the evaluation that produced it. - Documented accuracy. Each function’s reference page states its method, tolerance semantics and measured accuracy over stated input ranges.
Working with linalg, dataframes and plots
The exchange types already connect the three shipped modules:
| From | To | Today |
|---|---|---|
std.dataframe | numerics | frame.values_f64("x") returns list[f64] |
| numerics | std.dataframe | df.column_f64("fit", &values) |
| numerics | std.plot | Column.Num(values) takes the list without copying |
| numerics | std.linalg | la.vector(&values) copies; matrix.to_list() copies back |
The last row is the weak one. An optimizer converting between a list and a
Matrix every iteration would copy every time. Milestone 2 therefore adds
consuming conversions to std.linalg, such as la.vector_from(values),
which takes ownership of a list without copying, and matrix.into_list().
Solvers can then keep state in lists and call linalg without copying.
A complete fit, sketched in the API page:
read a CSV into a dataframe, take two columns as lists, fit a model with
curve_fit, evaluate it on a linspace grid, and plot data and fit with
geom_point and geom_line. Every step uses shipped or core APIs; the
tutorial lesson for milestone 2 is this program.
Milestones
Each milestone ships reference pages, a tutorial lesson and its tests.
- Scalar problems and special functions. The
std.numerichelpers;special(gamma family, incomplete gamma/beta,erffamily); scalar root finding (Brent, Chandrupatla, Newton, secant); bounded scalar minimization (Brent);quad;derivative.std.statgains t, χ², F, gamma, beta, binomial and Poisson distributions, which need incomplete gamma and beta. This also covers B38’s special functions and accurate summation. - Fitting and multivariate problems. Nelder–Mead, BFGS and L-BFGS-B;
least_squares(Levenberg–Marquardt and trust-region reflective for bounds) andcurve_fit; systems of equations (Powell hybrid); finite-difference gradients, Jacobians and Hessians;interpolate;la.lstsq,la.detand the consuming conversions. - ODEs, spectra and linear programming.
solve_ivpwith Dormand–Prince 5(4) and dense output, plus one stiff method (Radau IIA or a Rosenbrock method; chosen by measurement on the Robertson and Van der Pol problems);fft; denselinprog(bounded revised simplex first; an interior-point method if needed for larger problems).
After milestone 3, the Later rows above are candidates. Each needs a use case and its own scope under B38.
Testing and acceptance
- Reference values. Expected values come from SciPy and mpmath, generated offline by a checked-in script and stored as fixtures. Tests do not need Python.
- Standard problems. Moré–Garbow–Hillstrom functions (Rosenbrock, Powell
singular, Brown badly scaled) for optimization; NIST StRD nonlinear
regression datasets with certified values for least squares; QUADPACK’s
test integrals, including endpoint singularities, for
quad; Hairer’s nonstiff and stiff problems forsolve_ivp; Netlib LP problems small enough to keep in the repository forlinprog. - Accuracy. Special functions report maximum relative error in ULPs over stated ranges. A change that increases it fails the test.
- Allocation and errors. Allocation-denial tests cover every solver’s workspace. Each solver has a test where the callback fails, showing that its error reaches the caller.
- Cost. Each solver reports evaluation counts on the standard problems. Counts are compared with SciPy’s. Wall-clock comparisons are optional. They follow the profiling guide and do not promise a speedup.
Risks and open questions
- Scope creep. Requests will arrive for SciPy features not listed here. The Later and Out rows are the answer until a use case is written down.
- Numerical maintenance. Special functions and stiff ODE solvers are hard to get right. Ports keep the source algorithm’s structure and credit, so upstream fixes can be followed. SciPy and Cephes are BSD-3 and QUADPACK is public domain, so ports are allowed with attribution.
- Fixed-size problems. Small, fixed-dimension problems (3-vectors, quaternions, Kalman filters) would avoid heap allocation with value generics (B83). The list-based API does not preclude adding fixed-size variants later.
- FFT speed. A Plenty FFT may need
std.simdor a native kernel to be competitive. Measure first. - Sparse problems. Large finite-element and network problems need sparse matrices, which this proposal leaves out. Researchers with those problems will still need another library.
Numerics API sketch
Companion to the numerics core proposal. Every signature
here is hypothetical and may change before implementation. Fences are text
because nothing on this page compiles today. Shipped calls, such as
scan_csv, values_f64 and gg.table, use their current signatures.
Shared types (std.numeric)
enum NumericError:
Domain # invalid argument: empty bracket, negative tolerance
NoBracket # f(a) and f(b) have the same sign
NonFinite # an input or callback value was NaN or infinite
Unconverged(Report)
Linalg(la.LinalgError)
Allocation(AllocError)
class Report:
x: list[f64] # last iterate; one element for scalar problems
fun: f64 # objective value or residual norm at x
iterations: i64
evaluations: i64
A successful solve returns a result type for its problem, such as Root,
Minimum, Fit, Integral or Solution. Each has the fields of Report
plus fields specific to the problem, such as the error estimate for
Integral and the covariance for Fit.
Array helpers:
def linspace(start: f64, stop: f64, n: Positive[i64]) -> Result[list[f64], AllocError]
def logspace(start: f64, stop: f64, n: Positive[i64], base: f64 = 10.0) -> Result[list[f64], AllocError]
def cumsum(values: &list[f64]) -> Result[list[f64], AllocError]
def diff(values: &list[f64]) -> Result[list[f64], AllocError]
def dot(a: &list[f64], b: &list[f64]) -> Result[f64, NumericError]
def norm(values: &list[f64]) -> f64 # scaled, no spurious overflow
def fsum(values: &list[f64]) -> f64 # correctly rounded sum
def polyval(coefficients: &list[f64], x: f64) -> f64
def polyfit(x: &list[f64], y: &list[f64], degree: i64) -> Result[list[f64], NumericError]
def roots(coefficients: &list[f64]) -> Result[(list[f64], list[f64]), NumericError] # real, imaginary parts
roots uses companion-matrix eigenvalues from la.eig, so it needs the linalg
engine; polyfit uses la.lstsq.
Scalar problems (milestone 1)
import std.numeric.optimize as opt
import std.numeric.integrate as integ
import std.numeric.differentiate as d
import std.math as math
from std.math import MathError
def brentq[E, F: Callable[[f64], Result[f64, E]]](
f: &F, a: f64, b: f64, xtol: f64 = 2e-12, rtol: f64 = 8.9e-16,
max_iterations: i64 = 100,
) -> Result[opt.Root, NumericError | E]
def newton[E, F: Callable[[f64], Result[(f64, f64), E]]](
f_and_derivative: &F, x0: f64, ...
) -> Result[opt.Root, NumericError | E]
def minimize_scalar[E, F: Callable[[f64], Result[f64, E]]](
f: &F, a: f64, b: f64, xtol: f64 = 1.48e-8, max_iterations: i64 = 500,
) -> Result[opt.Minimum, NumericError | E]
def quad[E, F: Callable[[f64], Result[f64, E]]](
f: &F, a: f64, b: f64, atol: f64 = 1.49e-8, rtol: f64 = 1.49e-8,
max_subintervals: i64 = 50,
) -> Result[integ.Integral, NumericError | E]
def derivative[E, F: Callable[[f64], Result[f64, E]]](
f: &F, x: f64, order: i64 = 1, ...
) -> Result[d.Derivative, NumericError | E]
quad accepts infinite limits (f64.inf, -f64.inf) and maps them to a
finite interval, as QUADPACK’s QAGI does. Its result includes an error
estimate and the number of subintervals used.
def main() -> Result[(), AllocError | IoError | NumericError | MathError]:
# Root of cos(x) = x
root = opt.brentq(&(def(x: f64) -> Result[f64, MathError]: Ok(math.cos(x)? - x)), 0.0, 1.0)?
print(root.x[0])?
# Gaussian integral over the real line
g = integ.quad(&(def(x: f64) -> Result[f64, MathError]: math.exp(-x * x)), -f64.inf, f64.inf)?
print(g.value, g.error)?
Ok(())
Multivariate problems (milestone 2)
def minimize_bfgs[E, F: Callable[[&list[f64], &mut list[f64]], Result[f64, E]]](
f_and_gradient: &F, x0: &list[f64], gtol: f64 = 1e-5, max_iterations: i64 = 0,
) -> Result[opt.Minimum, NumericError | E]
def minimize_lbfgsb[E, F: Callable[[&list[f64], &mut list[f64]], Result[f64, E]]](
f_and_gradient: &F, x0: &list[f64], lower: &list[f64], upper: &list[f64], ...
) -> Result[opt.Minimum, NumericError | E]
def minimize_nelder_mead[E, F: Callable[[&list[f64]], Result[f64, E]]](
f: &F, x0: &list[f64], ...
) -> Result[opt.Minimum, NumericError | E]
def least_squares[E, F: Callable[[&list[f64], &mut list[f64]], Result[(), E]]](
residuals: &F, x0: &list[f64], m: i64, ...
) -> Result[opt.Fit, NumericError | E]
def curve_fit[E, F: Callable[[f64, &list[f64]], Result[f64, E]]](
model: &F, x: &list[f64], y: &list[f64], p0: &list[f64], ...
) -> Result[opt.Fit, NumericError | E]
def root[E, F: Callable[[&list[f64], &mut list[f64]], Result[(), E]]](
f: &F, x0: &list[f64], ...
) -> Result[opt.Root, NumericError | E]
The solver owns the gradient and residual buffers and passes them as
&mut list[f64], so an evaluation allocates nothing. If no gradient is
available, d.gradient(&f, &x, &mut g) gives a finite-difference one. Wrap
that in a closure to pass it as f_and_gradient. Separate gradient-free
variants of every solver are not provided.
Fit adds covariance: la.Matrix, computed from the Jacobian at the solution
as curve_fit does, and residual_norm.
Interpolation constructs an object once and evaluates it repeatedly:
import std.numeric.interpolate as ip
spline = ip.cubic_spline(&x, &y, boundary=ip.Boundary.NotAKnot)?
value = spline.at(2.5)?
values = spline.at_all(&grid)? # list[f64]
slope = spline.derivative(2.5, 1)?
ODEs, spectra and linear programming (milestone 3)
def solve_ivp[E, F: Callable[[f64, &list[f64], &mut list[f64]], Result[(), E]]](
rhs: &F, t0: f64, t1: f64, y0: &list[f64],
method: integ.Method = integ.Method.DormandPrince54,
rtol: f64 = 1e-3, atol: f64 = 1e-6, t_eval: Option[list[f64]] = Nothing,
) -> Result[integ.Solution, NumericError | E]
Method is an enum, not a string. A stiff method also takes a Jacobian
callback; without one, it uses a finite-difference Jacobian. Solution holds
t: list[f64] and y: list[list[f64]], one list per component, so each
component can go straight into a plot Column.Num.
import std.numeric.fft as fft
spectrum = fft.rfft(&samples)? # list[Complex]
frequencies = fft.rfftfreq(len(samples), 1.0 / rate)?
back = fft.irfft(&spectrum, len(samples))?
FFT outputs are list[Complex] from std.complex,
a @copy class, so a list stores the values inline.
solution = opt.linprog(&c, a_ub=&a, b_ub=&b, lower=&lo, upper=&hi)?
linprog minimizes cᵀx subject to A_ub x ≤ b_ub, A_eq x = b_eq and
bounds. The constraint matrices are la.Matrix. An infeasible or unbounded
problem is an error variant, not a status code.
An end-to-end fit
This program shows the milestone 2 workflow: load data, fit an exponential
decay, and plot data and fit. scan_csv, values_f64, gg.table and
Column.Num exist today. The std.numeric calls are proposed.
import std.dataframe as df
import std.math as math
import std.numeric as num
import std.numeric.optimize as opt
import std.plot as gg
from std.math import MathError
from std.plot import Column, PlotError
def decay(t: f64, p: &list[f64]) -> Result[f64, MathError]:
Ok(p[0] * math.exp(-p[1] * t)? + p[2])
def main() -> Result[(), AllocError | IoError | df.DataFrameError | NumericError | MathError | PlotError]:
frame = df.scan_csv("decay.csv")?.collect()?
t = frame.values_f64("t")?
y = frame.values_f64("signal")?
fit = opt.curve_fit(&decay, &t, &y, &[1.0, 0.1, 0.0]?)?
print(fit.x, fit.residual_norm)?
grid = num.linspace(t[0], t[len(t) - 1], 200)?
mut curve = list[f64].with_capacity(len(grid))?
for s in grid.iter():
curve.append(decay(s, &fit.x)?)?
points = gg.table([("t", Column.Num(t)), ("signal", Column.Num(y))]?)?
line = gg.table([("t", Column.Num(grid)), ("signal", Column.Num(curve))]?)?
chart = gg.ggplot(points, gg.aes("t", "signal"), [
gg.geom_point(),
gg.geom_line(data=line, colour="firebrick"),
]?)?
chart.save("decay.svg", width=6.0, height=4.0)?
Ok(())
The data passes from the dataframe to the solver to the plot as list[f64].
The only copies are values_f64 out of Polars storage and the model’s own
evaluation.
Numerics crate evaluation
Companion to the numerics core proposal. It answers issue
#493, which asked whether numeris should back linear algebra, statistics or
optimization, and the native-library half of issue #492, which proposed
optimization through basin. Facts are from the crates’ docs.rs pages read on
2026-11-03. Neither crate was built or measured.
Decision
Neither crate is a dependency. std.linalg keeps faer. The numerics modules
are written in Plenty and may use either crate as a reference for algorithms
and tests, with attribution.
The candidates
basin 3.2.0 | numeris 0.6.0 | faer 0.24.4 (shipped) | |
|---|---|---|---|
| Scope | Optimization only: first-order, derivative-free, least squares, global/stochastic, constrained; Brent root finding | Broad and SciPy-like: fixed and dynamic matrices, linalg, ODE, optim, quadrature, interpolation, FFT, special functions, statistics, control, estimation, image processing | Dense linear algebra |
| License | MIT OR Apache-2.0; PRIMA-derived solvers and the L-BFGS-B port are BSD-3 | MIT | MIT |
| First release | 0.0.1 on 2026-04-30 | 0.1.0 on 2026-02-28 | Released since 2021 |
| Churn | About 40 versions; majors 1.0 (June), 2.0 and 3.0 (19–20 October), 3.2 on 2026-11-02 | About 37 versions, pre-1.0, single maintainer | Pinned through a fork |
| Required dependencies | num-traits, rand, rand_chacha, rand_distr, web-time | num-traits | Built only into the separate linalg engine |
| Linear algebra backend | Generic over Vec<f64>, nalgebra, ndarray or faer | Its own kernels; optional nalgebra conversion | — |
| Fallible allocation | Not documented | Not documented; dynamic types use Vec behind alloc | Low-level APIs take caller scratch; Plenty’s fork makes GEMM workspaces fallible |
Reasons
Callbacks. An optimizer or ODE solver spends its time calling the user’s
function. With a wrapped crate, every evaluation would cross from Rust into
Plenty through the engine’s C interface. The callback’s environment and its
typed error would have to cross as well, and the error would have to be
carried back out through the crate’s own return path, which has no slot for
it. A Plenty solver specializes the callback at compile time and propagates
its error with ?. The proposal
records a probe of this on the current compiler.
Allocation failure. The wrapper rules
require a wrapped crate to report allocation failure as AllocError.
Neither crate documents fallible allocation, and both allocate through
ordinary Vec, which aborts on failure. faer needed a
fork to meet this
rule for GEMM alone. Repeating that for a second, broader crate would mean
maintaining a second fork.
Maturity. Both crates are under a year old. basin made three major
releases in a month. numeris has one maintainer and is pre-1.0. A
standard library promises stability that its dependencies need to share.
Pinning would work around churn but not supply fixes.
Replacing faer. numeris reports faster small fixed-size decompositions
than faer on Apple Silicon. That is its stack-allocated Matrix<T, M, N>,
which needs const generics. Plenty’s matrices are dynamically sized, and
std.linalg’s faer integration already passes allocation-denial,
concurrency and non-AVX tests. There is no measured reason to switch.
What each crate is still useful for
basin: the choice and structure of solvers (L-BFGS-B, trust-region reflective least squares, the PRIMA derivative-free family) and its test problems. Its separation of problem, solver, termination and driver is a reasonable model for keeping Plenty’s solvers small.numeris: compact reference implementations for adaptive Runge–Kutta pairs, RODAS4, Bluestein FFT, natural splines and the special functions in milestone 1, with tests that can be checked against SciPy and mpmath.
Ported code keeps its license notice. BSD-3 portions (PRIMA, L-BFGS-B, SciPy, Cephes) need attribution in the source and the release notices.
When to revisit
Reopen the choice for a specific algorithm, not a whole crate, if:
- a Plenty implementation is measured to be too slow and the work can be done without callbacks into Plenty, as in an FFT kernel writing into a caller-allocated buffer; and
- the native code allocates nothing or reports allocation failure, and adds its own runtime entry symbol under the wrapper rules.
Units of measure
Status: Partly implemented. This page proposes compile-time dimensional
analysis for Plenty: quantities whose physical dimension is part of the type,
checked by the compiler and erased before code generation. Milestone 1, the
language, shipped through issue #701 (local PRs #885, #887–#891); see
dimensions,
quantities and
where the implementation differs. Milestone 2’s
std.units shipped, including wrap_list and
strip_list. Milestone 3’s
shared instance bodies
shipped; plot axis labels remain under B97. Companion
pages record the precedents and the
spike evidence. Researched against local
main at bd407f05 on 2026-11-03; reconciled with eb772202.
Goal
A program that adds a length to a time, or passes an acceleration where a
velocity is expected, should fail to compile. A program that passes these
checks should run exactly as fast as the same program written with f64, and
allocate nothing more.
This example needs std.units (milestone 2). Quantity is built in and
needs no import.
import std.units as u
from std.units import Length, Time, Velocity
def speed(d: Quantity[f64, Length], t: Quantity[f64, Time]) -> Quantity[f64, Velocity]:
d / t
def main() -> Result[(), IoError]:
d = 3.2 * u.km
t = 41.0 * u.minute
v = speed(d, t)
print(v, v.to(u.km / u.hour))? # 1.3008130081300813 m/s 4.682926829268292
print(d + t)? # compile-time error: Length + Time
Ok(())
Recommendation
- Dimensions are a compiler-known kind of type. A module declares base
dimensions (
dimension Length("m")) and names derived ones (dimension Velocity = Length / Time). The compiler normalizes every dimension expression to a product of base dimensions with integer exponents, soLength / Time * TimeisLength. Quantity[T, D]is a built-in numeric wrapper, likeNonZero[T]. It hasT’s representation. The compiler defines its operators:+,-and comparisons need equal dimensions;*and/combine them.- The type carries the dimension, not the unit. A value is stored in the
coherent SI unit of its dimension (metres, seconds, kilograms).
km,minuteandpoundare constants, andq.to(unit)reads a value back in any unit. This is theuomand GNAT model. - Storage types compose.
Tmay be any number type, includingPositive[i64],NonNegative[i64]andNonZero[i64]. An operation’s storage result is whateverT’s own operator returns; its dimension comes from the dimension algebra. Units never drop out of a result. std.unitsdeclares SI in Plenty: the seven base dimensions, named derived dimensions, prefixed and common units, CODATA constants, and dimension-awaresqrt,hypotand friends.
The recommendation needs no value generics, no type-level integers and no generic operator hooks, which Plenty lacks; see what the language could do before.
What the language could do before
The spikes ran on the release compiler built
from main on 2026-11-02, before issue #701. Their library workarounds still
behave as shown; the built-in Quantity[T, D] now covers the rows marked
No and Partly.
| Need | Today | Evidence |
|---|---|---|
| Distinct types per dimension | Phantom type parameters: Q[f64, Length] and Q[f64, Time] are different types | Spike 1 |
Same-dimension +, -, < | @newtype forwarding over a phantom-tagged class; mixing dimensions is rejected | Spikes 1, 2 |
| Code generic over one dimension | def total[T, D](xs: &list[Q[T, D]]) -> Q[T, D] | Spike 7 |
| Zero cost | One-field classes are represented as their field; operator hooks on a one-f64 class compile to the same instructions as a.x + b.x (operator lowering) | Reference |
Length * Length -> Area, generically | No. Operator hooks cannot have type parameters, a class has one __mul__ signature, and there are no associated types or type-level arithmetic | Spike 4 |
| A domain inside a quantity | Partly. @newtype forwards Positive + Positive as Positive, but Positive - Positive returns a bare i64: the unit is lost | Spike 3 |
Positive[f64] | No. Integer domains take only signed integers | Spike 5 |
So a phantom-tag library can check additive code today, but multiplication
and division, the core of dimensional analysis, need a language change.
Hand-written classes per dimension pair (spike 6) work for a handful of
quantities and do not scale: each class can declare only one __mul__.
Options
| Option | Benefit | Cost |
|---|---|---|
A. Phantom-tag library with @newtype (works today) | No language change | Additive checks only; products need a hand-written function per pair |
B. std library over value generics: Quantity[T, const L: i8, const M: i8, const T: i8, ...] | Reuses B83 | Needs B83, its deferred derived-size expressions (L + L2), generic operator hooks with value parameters, and a second __mul__ for scalars. Seven fixed base dimensions, no user dimensions. Types print as Quantity[f64, 1, 0, -1, 0, 0, 0, 0]; a generic function over one dimension takes seven parameters |
C. Compiler-known dimensions and built-in Quantity (recommended) | Readable types and diagnostics; user dimensions; one parameter per generic dimension; no new operator-hook machinery | A new kind of type and dimension expressions in the checker |
| D. C plus F#-style inference through dimension expressions in parameter types | def sqrt[D](x: Quantity[T, D * D]) -> Quantity[T, D] infers D | Unification over abelian groups in the checker; later, if code needs it |
E. Runtime units (Pint, astropy, Measurement in Swift) | No type-system work | Run-time checks and storage per value; errors appear only on executed paths |
F. Units in the type (mp-units, Unitful): km and m are distinct types | Exact integer storage in any unit | Conversion and promotion rules between units of one dimension; many more types |
Rust is a useful warning about option B. uom encodes exponents with
typenum type-level integers because stable const generics cannot compute
L1 + L2 in a signature; that needs the unstable generic_const_exprs. The
const-generic ports (tiny-uom, const_unit_poc) require nightly features
and stopped in 2020. Option B in Plenty would hit the same wall at the
derived sizes that B83 deliberately defers.
C++ took the library route too: Boost.Units with template metaprogramming, then mp-units with C++20 class-type template parameters, now proposed for C++29 as P3045. Both are large libraries and rely on language features Plenty does not have and should not add for this. Ada GNAT and F# build dimensions into the compiler, as option C does. See the precedents.
Design
Declaring dimensions
dimension Length("m")
dimension Mass("kg")
dimension Time("s")
dimension Velocity = Length / Time
dimension Force = Mass * Length / Pow[Time, 2]
- A base dimension has a name and the symbol of its coherent unit. Its
identity is nominal, like a class:
pkg.Lengthandstd.units.Lengthdiffer. - A derived dimension is a transparent alias for a dimension expression,
like a type alias.
VelocityandLength / Timeare the same type. Oneis the built-in dimensionless dimension, the empty product.dimensiondeclarations are module-level and follow ordinary visibility (pub dimension Length("m")).dimensionbecomes a keyword; it is a declaration form, so this is the one source change existing programs could meet.
A dimension is a type with no values. It may appear only as a dimension
argument, so Quantity[f64, Length] is valid and x: Length is not.
Dimension expressions
Wherever a dimension is expected, the compiler accepts:
| Form | Meaning |
|---|---|
Length, Velocity, D | A base dimension, alias or dimension parameter |
A * B, A / B | Product and quotient, left-associative, with parentheses |
Pow[A, n] | Integer power; n is an integer literal, possibly negative |
Root[A, n] | nth root; every exponent of A must be divisible by n; n is nonzero, and a negative n takes the reciprocal |
One | Dimensionless |
The normal form is a list of (base dimension, nonzero i8 exponent) pairs
sorted by base identity. Two dimensions are equal exactly when their normal
forms are equal. An exponent outside i8 is a compile-time error, including
in an intermediate result that a later factor would cancel.
Exponents are integers. Root rejects a dimension it cannot divide exactly:
the square root of Pow[Length, 3] is an error, not Length^(3/2). This
excludes units such as V/√Hz from noise spectral density. F# added
rational exponents in its fourth version for these cases; the normal form
could store rationals later without changing any spelling.
Generic dimensions
D: Dimension declares a dimension parameter. Dimension is a built-in
constraint, like IntType; it admits exactly the dimension types.
def mean[T: FloatType, D: Dimension](xs: &list[Quantity[T, D]]) -> Quantity[T, D]
def kinetic[M: Dimension, V: Dimension](m: Quantity[f64, M], v: Quantity[f64, V]) -> Quantity[f64, M * V * V]
def sqrt[T: FloatType, D: Dimension](x: Quantity[T, D]) -> Result[Quantity[T, Root[D, 2]], MathError]
One rule keeps inference structural, as it is today: in parameter types, a
dimension argument is either concrete or a bare parameter. Return types,
annotations and bodies may use any expression. Since Plenty checks bodies per
instance, M * V * V and Root[D, 2] are evaluated after substitution, where
every operand is known. This is the form B83 calls “expressions in
non-inferred output positions”, applied to dimensions instead of integers.
A Root that does not divide is reported as an instance error at the call
that required it, through the existing
instance diagnostics:
main.plenty:8:9: cannot instantiate `units.sqrt[f64, Pow[Length, 3]]`: `Root[Pow[Length, 3], 2]` is not a whole dimension
note: `Length` has exponent 3, which 2 does not divide
Option D, inferring D from x: Quantity[T, D * D], stays out until code
needs it. Option C programs remain valid under it.
Quantities
Quantity[T, D] holds one T measured in the coherent unit of D. T is a
number type: an integer, a float, or an integer domain type. D is a
dimension.
| Operation | Result |
|---|---|
Quantity[f64, Length](3.0) | 3 m. The constructor takes the coherent-unit value; prefer 3.0 * u.metre in application code. |
q.get() | The stored T, in coherent units |
q.to(unit) | (q / unit).get(), for float storage: unit is a quantity of the same dimension, such as u.km. Mismatched dimensions are a compile-time error. |
a + b, a - b | Both Quantity[T, D]; result Quantity[R, D], where R is the type T + T or T - T returns |
a * b, a / b | Quantity[R, A * B] and Quantity[R, A / B]; / needs float storage |
k * q, q * k, q / k | A scalar T scales without changing the dimension |
k / q | Quantity[R, One / D] |
a // b, a % b | Integer storage: // divides dimensions, % keeps a’s and needs equal dimensions |
-q, +q | As for T, dimension unchanged |
<, <=, ==, … | Equal dimensions; bool |
+=, -= | Equal dimensions. *= and /= take a scalar T only, since the dimension of a variable cannot change. |
print(q), f"{q}" | The value, then the coherent unit: 1.3 m/s, 9.81 m/s^2. A format spec applies to the number: f"{q:.2f}" is 1.30 m/s. |
debug | Quantity[f64, Length * Pow[Time, -1]](1.3), the normal form |
Rules with no exceptions:
- Scalars only scale. A plain
T, including a literal, may multiply or divide a quantity, never be added to it or compared with it.q > 0.0is an error; writeq > 0.0 * u.metre. F#’s0.0<_>wildcard is the exception this rule avoids. - Dimensionless is still a quantity.
d / disQuantity[f64, One]..get()returns thef64. There is no implicit conversion. - Storage types never mix.
Quantity[f32, D] + Quantity[f64, D]is an error, asf32 + f64is.
Overflow, division by zero and checked: behave as they do for T. A
quantity operation is T’s operation, so a checked: block reports
OverflowError[T] or ZeroDivisionError.
Unit constants and conversions
A module constant may have a quantity type, with an initializer built from literals and other quantity constants:
pub const metre: Quantity[f64, Length] = Quantity[f64, Length](1.0)
pub const km: Quantity[f64, Length] = 1000.0 * metre
pub const hour: Quantity[f64, Time] = 3600.0 * second
pub const newton: Quantity[f64, Force] = kg * metre / (second * second)
Construction multiplies by the unit, and to divides by it. Both are one
floating-point operation, folded at compile time when the operands are
constants. Converting through the coherent unit can round: 0.1 * u.foot
then .to(u.foot) need not return exactly 0.1. Programs that need exact
values in a non-coherent unit should declare their own dimension or keep that
unit’s integer count, as std.time.Duration keeps nanoseconds.
Unit constants are f64. f32 quantities convert with
Quantity[f32, D].from(q), which narrows the stored value as f32(x) does;
Quantity[f64, D].from(q) widens.
Domain storage
The user-facing question was whether Positive, NonNegative and NonZero
can carry units. They can, by nesting the domain inside the quantity and
taking the storage result from the domain’s own rules:
Expression, with p: Quantity[Positive[i64], Length] | Result |
|---|---|
p + p, p * p | Quantity[Positive[i64], Length], Quantity[Positive[i64], Pow[Length, 2]] |
p - p, -p | Quantity[i64, Length]: the domain drops, the unit stays |
n // p, with n: Quantity[i64, Area] | Quantity[i64, Length], without the zero or MIN // -1 check |
Quantity[Positive[i64], Length](0) | Compile-time error, as for Positive[i64] |
Quantity[Positive[i64], Length].try_from(k) | Result[..., RangeError] for a computed k |
Positive[Quantity[i64, Length]] | Rejected: one nesting order, the domain inside |
Today’s @newtype forwarding does the opposite for -: it returns the bare
i64 (spike 3). The built-in rule rewraps every result.
Integer storage counts whole coherent units, so Quantity[i64, Length] holds
whole metres. That suits counts such as pixels, bytes or items, declared as
user dimensions, more than SI quantities.
Float domains do not exist. Mass, absolute temperature and the argument of a
logarithm would benefit from a Positive[f64]. Its rules differ from the
integer domain: it must exclude NaN, and * cannot preserve it because a
product of two small positive values can underflow to zero. That belongs in a
separate proposal. The storage rule above means a later float domain works
inside quantities with no further units design.
std.units
Written in Plenty, as declarations and small functions:
| Part | Contents |
|---|---|
| Base dimensions | Length (m), Mass (kg), Time (s), Current (A), Temperature (K), Amount (mol), Luminosity (cd) |
| Derived dimensions | Area, Volume, Velocity, Acceleration, Force, Pressure, Energy, Power, Charge, Voltage, Resistance, Frequency, Density, and the other named SI derived quantities |
| Units | SI prefixes on the base and named derived units (km, ms, kN, MPa, kWh), plus minute, hour, day, litre, tonne, bar, atm, eV, and the international foot, inch, mile, pound and pound-force |
| Constants | The exact SI defining constants (c, h, e, k_B, N_A) and CODATA values such as G, typed |
| Functions | sqrt, cbrt, abs, hypot, min, max, clamp, mean, sum; exp, log and trigonometry take Quantity[f64, One] |
| Temperature | from_celsius, to_celsius, from_fahrenheit, to_fahrenheit (affine; see out of scope) |
| Time | from_duration(&Duration) -> Quantity[f64, Time] and a fallible to_duration |
| Lists | wrap_list[D](xs: list[T]) -> list[Quantity[T, D]] and strip_list, consuming the list without copying, since the element layouts are identical |
Angles are dimensionless, as in SI: u.degree is Quantity[f64, One]
holding π/180. A program that wants angles checked declares
dimension Angle("rad") and converts at trigonometric calls.
Working with the numeric libraries
| Library | Boundary |
|---|---|
std.dataframe | units.wrap_list[Length](frame.values_f64("x")?) gives typed values with no copy beyond values_f64 |
std.plot | Column.Num(units.strip_list(qs)), with a hand-written label for now. Labels from units are open design. |
std.numeric (proposal) | Solvers keep f64 callbacks. A callback strips and rewraps at its boundary. Quantity-generic solvers wait for evidence. |
std.linalg | Matrix stays f64. Elements of one matrix often have different dimensions, which a single D cannot express; out of scope |
Plot axis labels
Open design, deferred from milestone 3 (B97). std cannot yet give a unit
symbol as text, and a unit constant is only a value: u.km is
Quantity[f64, Length] holding 1000.0, with no name. Candidate pieces:
symbol()onQuantity[T, D]. Returns the coherent-unit symbol that printing uses ("m/s") as a static string, without allocating;""for a dimensionless quantity. This also serves hand-written labels:f"speed ({Quantity[f64, Velocity].symbol()})".- A quantity column in
std.plot. For exampleplot.quantity("speed", &xs)stores the values throughstrip_list, with no copy, and records the dimension. With no explicit label, the axis title isspeed (m/s). An explicit label wins. - A display unit that fits the data. Values of 2.5e6 m read better as
2500 km. The column would pick a unit whose scaled values are easy to read,
for example so the largest magnitude lies in [1, 1000), and label the axis
with that unit. Questions:
- Where candidate units come from. Unit constants carry no names, so this
needs a per-dimension table of (symbol, factor) pairs, from SI prefixes
on base symbols (
mm,m,km), from named units instd.units, or both. Named-unit display (kN,J) is related later work. - Prefixes on compound symbols:
km/sorkN, and which factor of a product takes the prefix. - Scaling without a copy: divide at tick and label time rather than rewriting the column.
- Overrides: the caller fixes the display unit and its symbol, for example
plot.quantity_in("distance", &xs, u.km, "km"), since the constant has no name to show. - Shared axes: series on one axis must share one display unit.
- Where candidate units come from. Unit constants carry no names, so this
needs a per-dimension table of (symbol, factor) pairs, from SI prefixes
on base symbols (
Cost
- Run time: none.
Quantity[T, D]hasT’s representation, operators lower toT’s instructions, and unit constants fold. Onlyprintand formatting read the dimension, as a static symbol string. - Code size. Generic instances that differ only in dimension arguments have identical bodies unless they format a quantity. The implementation should emit one body for them. Rust and C++ rely on the linker’s identical code folding for this; Plenty’s per-instance checking makes it an explicit step after checking.
- Compile time. A normal form is a short sorted list, interned with other types. Each dimension expression is normalized once per instance.
Implementation
Milestone 1 shipped as designed, with these differences. The reference pages are authoritative.
| Topic | This design | Implemented |
|---|---|---|
| Integer division | a // b where T supports it | / is float-only and // integer-only for quantities; integer storage has no to, so use // and .get() |
Root degree | Positive n | Nonzero n; negative takes the reciprocal |
Pow, Root | Reserved forms | Fallback names that a local declaration or import shadows |
| Exponents | i8 normal form | Literal n within i64; every intermediate exponent must fit i8 |
| Debug output | Quantity[f64, Length / Time](1.3) | Normal form: Quantity[f64, Length * Pow[Time, -1]](1.3); alias notes appear only in diagnostics |
checked: | OverflowError[T] | Also ZeroDivisionError, as for the underlying arithmetic |
| Constants | Literals and quantity constants | Also casts of extracted values, .get() and float .to; no user function calls |
| Zero-cost evidence | Release instructions match f64 | Matched for f32 and f64 +, * and .to on x86-64; not measured on other targets |
| Instance sharing | Milestone 1 if cheap | Milestone 3, for every function: identical placed code is folded into one body. Instances that format a value or build a collection keep separate bodies, since their type descriptors name the dimension; functions used as values keep their own addresses |
std.units lists | wrap_list[D](xs: list[T]) and strip_list | f64 storage only, like the other std.units functions. A std-only compiler intrinsic, __list_retag, consumes the list and stores the target list type’s descriptor in its header; it accepts only a quantity and its own storage type as element types |
std.units storage | Unstated | Units, constants and the std.math-based functions use f64; abs and clamp accept any storage |
min, max | In std.units | Not repeated: the generic std.math.min and max already accept quantities |
mean | Unstated | Returns Option, Nothing for an empty list, as std.math.min_all does |
| Symbol order | SI order, kg*m/s^2 | Declaration order within a module, modules by name; std.units declares SI order |
Out of scope
| Feature | Reason |
|---|---|
| Units in the type (option F) | Conversion and promotion rules for one dimension; the dimension-only model covers the checks |
| Affine quantities (°C, °F, positions, points in time) | from_celsius converts at the boundary. mp-units models these as quantity_point; add one only with use cases. std.time already separates Instant from Duration. |
Quantity kinds: torque vs energy, Hz vs Bq, rad vs sr | SI gives them one dimension. A user dimension separates them where it matters. |
| Rational exponents | Integer exponents first; see dimension expressions |
| Logarithmic units (dB, neper) | Not multiplicative; ordinary functions |
| Matrices and dataframes with per-column units | Needs a dimension per column or element, as UnitfulTensors.jl does; later |
| Dimension expressions in parameter types (option D) | Inference by abelian unification; later if needed |
Milestones
- Language. Delivered (issue #701).
dimensiondeclarations, theDimensionconstraint, dimension expressions withPowandRoot, normal forms and identity, the parameter-type rule, built-inQuantity[T, D]for every number and integer domain storage type, operators,get,to,try_from, printing, quantity module constants and diagnostics. Reference page, tutorial lesson and cheat-sheet entry. std.units. Delivered. SI dimensions, units, constants and functions; temperature andDurationconversions; zero-copy list conversions.- Measured follow-ups. Shared bodies for instances differing only in dimension (delivered); plot axis labels from units (deferred, design open); then any later item with a use case.
Testing and acceptance
- Algebra: normalization is commutative and associative, cancels exactly,
and
Rootrejects every non-divisible exponent. Property tests over random expressions compare normal forms against a reference evaluator. - Operators: every operator for every storage type, including each domain
and
checked:blocks, with expected storage and dimension results. - Rejections: each rule above has a diagnostic test, including additive
scalars, mixed storage,
Positive[Quantity[...]], dimensions in value positions and expressions in parameter types. - Zero cost: the operator-lowering probe extended to quantities:
a + b,a * bandq.to(u.km)onQuantity[f64, D]must emit the same release instructions as thef64program. Allocation-denial tests cover quantity arithmetic andwrap_list. - Library:
std.unitsconstants checked against the SI Brochure (9th edition) and CODATA 2022; conversions round-trip within one ulp where the factor is exact.
Risks and open questions
- Diagnostics through aliases. Resolved: diagnostics show the normal form with a note naming the first visible matching alias in declaration or import order, using the importing module’s spelling.
- Symbol spelling. Resolved: ASCII base symbols in declaration order
within a module, with modules ordered by name.
std.unitsdeclares SI order, so a newton printskg*m/s^2. Debug spellings keep the normal form, sorted by qualified base name. Named-unit display (J) is later work. - The
dimensionkeyword may conflict with existing identifiers outside the repository.PowandRootare not keywords: a local declaration or import with either name keeps its ordinary meaning. - Learner cost. Units add a concept. The tutorial lesson should come after generics and teach the rules above without the algebra’s internals.
Units of measure: precedents
Research for the units of measure proposal, gathered on 2026-11-03. Each entry records the model, what it costs at run time, and what it means for Plenty. It contains no Plenty examples.
Summary
| System | Where dimensions live | Run-time cost | Unit in type? | Generic over dimension |
|---|---|---|---|---|
Rust uom | Library; typenum type-level integers | None; values normalized to base units | No | Yes, with trait bounds on typenum arithmetic |
| Rust const-generic ports | Library; const exponents | None | No | Needs nightly generic_const_exprs |
| C++ Boost.Units | Library; template metaprogramming | None | Unit system in type | Yes, verbose |
| C++ mp-units (P3045) | Library; C++20 class-type template parameters | None | Yes | Yes, with concepts |
| F# units of measure | Compiler; abelian-group unification | None; erased | Units, not dimensions | Yes, inferred |
| Ada (GNAT) | Compiler aspects on subtypes | None | No | Limited |
| Julia Unitful.jl | Library; type parameters, compiler specializes | None for homogeneous arrays | Yes | Yes, through dispatch |
| Python Pint, astropy | Run time | Per operation and per value | — | — |
Rust
uom 0.38. Quantities are generic over a dimension type whose seven
exponents are typenum integers, the storage type and a unit system.
Values are normalized to the system’s base units at construction, so
Length::new::<kilometer>(3.2) stores 3200 m; arithmetic is then plain
storage arithmetic. Results of * and / come from trait impls that add or
subtract exponents with typenum operators. Errors name typenum types such
as PInt<UInt<UTerm, B1>>. Plenty takes the normalization and the
dimension-only type; it avoids the type-level integer encoding.
Source: https://docs.rs/crate/uom/latest.
tiny-uom 0.1 (2020) and similar experiments replace typenum with
const generics. Computing L1 + L2 in a signature needs the unstable
generic_const_exprs (then const_evaluatable_checked), so they require
nightly Rust and have not reached stable use. This is the problem B83 avoids by
deferring derived sizes.
Source: https://docs.rs/crate/tiny-uom/latest/source/README.md.
C++
Barton and Nackman (1994) showed dimensional analysis through template parameters. Boost.Units generalized it with template metaprogramming: zero cost, long compile times and diagnostics.
mp-units uses C++20 class types as template parameters, so a quantity is
quantity<si::metre / si::second, double> with readable type names. It
distinguishes quantity kinds (isq::height vs isq::width), separates
points from differences (quantity_point, needed for °C), and keeps units in
the type, converting on assignment where that is exact. P3045 proposes it for
the C++ standard library; R7 is dated February 2026. The maintainer’s reports
state that LEWG gave unanimous support at the Croydon meeting (March 2026)
for the core, quantity kinds, the affine space and text output, and that the
Brno meeting (June 2026) saw a new paper on non-negative absolute
quantities. These are the maintainer’s reports, not committee minutes.
Sources:
https://www.open-std.org/jtc1/sc22/wg21/docs/papers/2026/p3045r7,
https://github.com/mpusz/mp-units,
https://mpusz.github.io/mp-units/HEAD/blog/2026/03/28/report-from-the-croydon-2026-iso-c-committee-meeting,
https://mpusz.github.io/mp-units/HEAD/blog/2026/06/13/report-from-the-brno-2026-iso-c-committee-meeting/.
The non-negative paper is the closest precedent for this proposal’s domain storage: mp-units is adding range constraints to quantities rather than wrapping quantities in a separate range type.
F#
Andrew Kennedy’s units of measure are part of the F# compiler. float<m/s>
is a float with a unit; units form an abelian group, and inference solves
equations over it, so let sqr x = x * x gets the type
float<'u> -> float<'u ^ 2>. Units are erased after checking: a
float<m> is a float at run time. F# 4.0 added rational exponents
(cm Hz^(1/2) / W). F# writes literal zero as 0.0<_> to unify with any
unit; this proposal rejects that exception and requires a typed zero.
F# puts units, not dimensions, in the type: float<km> and float<m> are
different, converted by explicit multiplication. Option F in the proposal
records why Plenty does not.
Sources:
https://learn.microsoft.com/en-us/archive/blogs/andrewkennedy/units-of-measure-in-f-part-three-generic-units,
https://fsharp.github.io/fslang-spec/units-of-measure,
https://www.infoq.com/news/2015/04/FSharp-4/.
Ada (GNAT)
GNAT adds Dimension_System and Dimension aspects. A type declares the
base dimensions; subtypes declare exponents; System.Dim.Mks defines SI.
The compiler checks dimensions statically and generates no extra code.
It is the closest precedent for option C: a small compiler extension rather
than a type-level library, with dimension-only types.
Sources:
https://gcc.gnu.org/onlinedocs/gnat_ugn/Performing-Dimensionality-Analysis-in-GNAT.html,
https://www.adacore.com/uploads/blog/dc.pdf.
Julia
Unitful.jl encodes units and dimensions as type parameters, and Julia’s specializing compiler removes them for concrete types. Arrays of one quantity type are as fast as plain arrays. Users report long type names, slow arrays mixing units, and the need for UnitfulTensors.jl for matrices with per-element units. Plenty’s proposal leaves matrices with units out of scope for the same reason. Sources: https://discourse.julialang.org/t/physical-units-in-julia/76825, https://discourse.julialang.org/t/performance-of-unitful-arrays/52635, https://discourse.julialang.org/t/ann-unitfultensors-jl-efficient-arrays-with-units/114028.
Python
Pint and astropy.units attach a unit object to each value and check at run time. They are flexible and cost storage and time per operation; errors appear only on executed paths. They are option E in the proposal.
Motivating failure
The Mars Climate Orbiter was lost in 1999 after ground software produced
impulse in pound-force seconds where the navigation software expected newton
seconds. Both are impulse. A dimension-only type system would not have caught
this, since the values had the same dimension; normalizing every value to a
coherent unit at construction would have. This is why the proposal stores
coherent-unit values and makes 3.0 * u.lbf * u.second the only way to write a
pound-force second.
Units of measure: spike evidence
Spikes 1–8 ran on the release compiler built from main on 2026-11-02, before
issue #701, and were rerun on eb772202 (after issue #701) with identical
output except spike 5. They support the
units of measure proposal. The
built-in quantity section shows the same
cases with Quantity[T, D]. Outputs are exact except that error paths are
shortened to the file name.
Spike 1: Same-dimension arithmetic
Phantom tags give distinct types; @newtype forwards +, -, < and unary -.
# Phantom-tagged generic newtype: same-dimension arithmetic, distinct types.
class Length:
pass
class Time:
pass
@copy
@newtype
class Q[T, D]:
value: T
def main() -> Result[(), IoError]:
a = Q[f64, Length](3.0)
b = Q[f64, Length](4.5)
print(a + b, a - b, a < b, -a)?
t = Q[f64, Time](2.0)
print(t)?
Ok(())
Output:
Q[f64, Length](value=7.5) Q[f64, Length](value=-1.5) True Q[f64, Length](value=-3.0)
Q[f64, Time](value=2.0)
Spike 2: Mixed dimensions are rejected
The forwarding rule already rejects Length + Time.
class Length:
pass
class Time:
pass
@copy
@newtype
class Q[T, D]:
value: T
def main() -> Result[(), IoError]:
a = Q[f64, Length](3.0)
t = Q[f64, Time](2.0)
print(a + t)?
Ok(())
Output:
error: s2.plenty:12:11: `@newtype` class `Q[f64, Length]` forwards `+` only between two `Q[f64, Length]` values, not `Q[f64, Length]` and `Q[f64, Time]`; read the field, or declare an operator hook for these operands
Spike 3: Domain storage loses the unit on -
Positive - Positive returns i64, so d is a bare i64; the annotation n: i64 = d compiles.
# Domain storage through a newtype: does Positive survive + and -?
class Length:
pass
@copy
@newtype
class Q[T, D]:
value: T
def main() -> Result[(), IoError]:
a = Q[Positive[i64], Length](3)
b = Q[Positive[i64], Length](4)
s = a + b
d = a - b
print(s, d)?
n: i64 = d
Ok(())
Output:
Q[Positive[i64], Length](value=7) -1
Spike 4: A generic operator hook is accepted when unused
The restriction is checked only at a use.
# Generic operator hook: can __mul__ take a different dimension?
class Q[D]:
value: f64
def __mul__[E](self, other: &Q[E]) -> f64:
self.value * other.value
def main() -> () :
()
Output:
(compiles; no output)
Spike 4b: A generic operator hook is rejected when used
Length * Time cannot be written with one generic __mul__.
class L:
pass
class T:
pass
class Q[D]:
value: f64
def __mul__[E](self, other: &Q[E]) -> f64:
self.value * other.value
def main() -> Result[(), IoError]:
a = Q[L](2.0)
b = Q[T](3.0)
print(a * b)?
Ok(())
Output:
error: s4b.plenty:7:5: operator hooks cannot have method type parameters, defaults, or yield
Spike 5: No float domain
Positive[f64] is rejected. On 2026-11-02 the error also suggested NonZero[f64], itself rejected; issue #700 removed that hint (local PR #883).
# Positive over floats?
def f(x: Positive[f64]) -> f64:
x.get()
def main() -> ():
()
Output:
error: s5.plenty:2:10: `Positive` requires a signed integer type
Spike 5b: NonZero is integer-only
def f(x: NonZero[f64]) -> f64:
x.get()
def main() -> Result[(), IoError]:
print(f(2.0))?
Ok(())
Output:
error: s5b.plenty:1:10: f64 does not satisfy IntType
Spike 6: Hand-written classes per dimension
Works, but each class has one __mul__ signature, so Metres * Metres and Metres * f64 cannot both use it; the scalar goes through __rmul__.
# One hook signature: scalar via __rmul__, quantity*quantity via __mul__.
@copy
class Metres:
value: f64
def __mul__(self, other: &Metres) -> SquareMetres:
SquareMetres(self.value * other.value)
def __rmul__(self, k: f64) -> Metres:
Metres(k * self.value)
def __truediv__(self, other: &Seconds) -> MetresPerSecond:
MetresPerSecond(self.value / other.value)
@copy
class Seconds:
value: f64
@copy
class SquareMetres:
value: f64
@copy
class MetresPerSecond:
value: f64
def main() -> Result[(), IoError]:
m = Metres(3.0)
print(m * m, 2.0 * m, m / Seconds(1.5))?
Ok(())
Output:
SquareMetres(value=9.0) Metres(value=6.0) MetresPerSecond(value=2.0)
Spike 7: Code generic over one dimension
Additive unit-polymorphic code works today.
# Unit-polymorphic code over a phantom dimension, today.
class Length:
pass
@copy
@newtype
class Q[T, D]:
value: T
def total[T, D](xs: &list[Q[T, D]]) -> Q[T, D]:
mut sum = xs[0]
for i in range(1, len(xs)):
sum = sum + xs[i]
sum
def main() -> Result[(), AllocError | IoError]:
xs = [Q[f64, Length](1.5), Q[f64, Length](2.0)]?
print(total(&xs))?
Ok(())
Output:
Q[f64, Length](value=3.5)
Spike 8: NonZero storage
* keeps NonZero; + returns a bare i64 and drops the unit.
# Positive survives + inside a newtype; NonZero divisor through a newtype.
class Count:
pass
@copy
@newtype
class Q[T, D]:
value: T
def main() -> Result[(), IoError]:
a = Q[NonZero[i64], Count](3)
b = Q[NonZero[i64], Count](-2)
print(a * b, a + b)?
Ok(())
Output:
Q[NonZero[i64], Count](value=-6) 1
Built-in quantity after issue #701
Run on eb772202. Spike 4b’s product and spike 3’s subtraction now work, and
the proposal’s opening example, with local unit constants in place of
std.units, prints the output the proposal gives.
dimension Length("m")
dimension Time("s")
dimension Velocity = Length / Time
const metre: Quantity[f64, Length] = Quantity[f64, Length](1.0)
const second: Quantity[f64, Time] = Quantity[f64, Time](1.0)
const km: Quantity[f64, Length] = 1000.0 * metre
const minute: Quantity[f64, Time] = 60.0 * second
const hour: Quantity[f64, Time] = 3600.0 * second
def speed(d: Quantity[f64, Length], t: Quantity[f64, Time]) -> Quantity[f64, Velocity]:
d / t
def main() -> Result[(), IoError]:
d = 3.2 * km
t = 41.0 * minute
v = speed(d, t)
print(v, v.to(km / hour))?
a = Quantity[f64, Length](2.0)
b = Quantity[f64, Time](3.0)
print(a * b)?
p = Quantity[Positive[i64], Length](5)
print(p - p, p * p)?
Ok(())
Output:
1.3008130081300813 m/s 4.682926829268292
6.0 m*s
0 m 25 m^2
Mixed dimensions:
dimension Length("m")
dimension Time("s")
def main() -> Result[(), IoError]:
d = Quantity[f64, Length](1.0)
t = Quantity[f64, Time](1.0)
print(d + t)?
Ok(())
error: q2.plenty:6:11: quantity `+` requires equal dimensions, not `Length` and `Time`
p - p keeps the unit and drops the domain:
dimension Length("m")
def main() -> Result[(), IoError]:
p = Quantity[Positive[i64], Length](5)
q: Quantity[Positive[i64], Length] = p - p
print(q)?
Ok(())
error: q3.plenty:4:5: expected Quantity[Positive[i64], Length], got Quantity[i64, Length]
Six rules for a consistent core
Status: Accepted; partially implemented. The backlog tracks remaining scope. The evidence below records research against local
mainatb4580c2don 2026-10-31, with probes on the debug compiler built that day. The evidence is the 2026-10-31 language consistency survey, recorded as issues #612–#637. Accepted and scheduled as backlog rows B88–B93 (staged plan); the review decisions settle five questions. A seventh rule from the same survey, a written convention page for the standard library, is separate documentation work and is only referenced here.
Summary
The survey found a coherent core with local exceptions. Each exception has its own explanation, and a learner has to memorise them one at a time. This proposal replaces about twenty such exceptions with six rules:
- One copy rule. Tuples, built-in types and function values copy when
their parts copy. A declared class or enum copies only when it says
@copy. The rule is stated in the same words in the reference, the cheat sheet and the agent skill. - One formatting path.
print(x)writes whatf"{x}"writes, andstr.repr(x)writes whatf"{x!r}"writes. Both use a type’sdisplayanddebugmethods, at the top level and inside aggregates. - An owner converts to its view. Where a view type (
&[T],&substr) is expected, a reference to the owner (&xs,&s,b"...") is accepted without[:]. Every byte owner and byte view is a byte source. - Adapters belong to iterators.
iter(source)accepts everyforsource and gives the same adapters and terminals. Only iterators (pipelines and generators) have adapter methods; collections, ranges, views and the free sources do not. - An error type names only what can fail. Allocation-only failure is
AllocError. A domain error type carries allocation asAllocation(AllocError). Invalid arguments follow one four-part policy. - Every callable form is a value wherever a callback is accepted. Named functions, closures with or without captures, stored or literal, variant constructors and class constructors all pass to the same parameter, and a callback behaves the same however it is written.
Each rule is a change to behaviour or teaching, so each has its own backlog
row and can ship independently. The rules do not depend on one another,
except that rule 5 decides the error type that rule 2’s print reports.
How to read this page
Each rule has the same sections: what happens today, the proposed rule,
alternatives, cost, what a learner no longer has to learn, and phasing.
“Today” claims were checked against the reference pages linked in the text
and, where marked probe, with plenty --check or a run on the 2026-10-31
debug build.
Earlier changes removed features rather than deprecating them (std.iter was removed in favour of iter(); see
fluent iterator pipelines). This proposal follows that
precedent: a breaking change ships with a diagnostic that states the new
spelling, and the same change updates std, tests, the tutorial and the
reference. Package language editions are not
implemented and are not required.
Rule 1: one copy rule
Today
The bindings, moves and copies
page says scalars, strings, ranges, named function values and “aggregates of
copyable components” copy, and that classes are affine unless declared
@copy. “Aggregates”
includes tuples and enums, but the enums
page never mentions copying. The agent-facing pages each give a different
list (issue #636):
| Source | Says copies |
|---|---|
| Reference | Scalars, str, ranges, function values, copyable aggregates, @copy classes |
| Cheat sheet | Scalars, str, copyable tuples, @copy classes; also says a @copy class “cannot take type parameters”, which classes contradicts (IntType/FloatType bounds are allowed) |
plenty agent skill | Scalars, str, @copy classes |
None of the agent-facing pages mentions enums, and the rule itself has an exception: a class must opt in, an enum copies implicitly.
That exception has a cost. Issue #535, which introduced @copy, made the
annotation “part of the type’s public contract: removing it is a breaking
change”. An enum’s copyability is the same contract, but it is implied by its
payloads. Probe: enum E: A(i64) | B copies, so take(e); take(e) checks.
Changing the payload to A(list[i64]) makes E affine, and the error appears
at the second take(e) in every caller (use of moved binding e), not at the
declaration that changed. @copy enum is rejected today
(`@copy` applies only to class declarations).
Proposed rule
A value copies when its type is a scalar,
str,bytes, a range, a function value, a tuple or built-in generic (Option,Result) whose parts copy, or a class or enum declared@copy. Every other value moves.@copyis checked at the declaration: every field or payload must copy, and the type has no__del__.
The rationale, stated with the rule: anonymous and built-in types copy by structure, because they have no author who could intend otherwise. A declared type copies only when its author says so, because copying is part of its public contract and must not change when a field changes.
The same sentence goes in the reference, the cheat sheet and the skill. The cheat sheet’s type-parameter claim is corrected. A test compares the three copies so they cannot drift again.
Alternatives
- State today’s rule without changing it (classes opt in, enums by structure). This fixes the documentation but keeps the exception and the distant-caller breakage. Rejected as the end state; it is phase 1.
- Make classes structural too, and remove
@copy. One rule with no annotation, but a class that holds a token, an index or an invariant would silently copy, and adding a field would flip the contract of every class. This reverses issue #535. Rejected. - Allow
@copyon enums but keep structural copying as the default. The annotation would only add a declaration-site check. Two ways to get the same behaviour, and the default still breaks distant callers. Rejected.
Cost and compatibility
- Phase 1 is documentation and one test.
- Phase 2 (
@copyaccepted on enums) is additive. - Phase 3 makes undeclared enums affine. std has 158 enum declarations;
tests, examples and book pages have about 460. A migration script adds
@copyto every enum whose payloads all copy, which preserves today’s behaviour exactly; reviewers then remove it where an enum should move. Generic enums whose copying depends on a type argument need a copyable bound, which is B84’s open work for generic@copyclasses. Phase 3 waits for that bound (decided 2026-10-31), so B84 moves earlier: it now gates this rule as well as generic copy classes. - Copy-dependent behaviour elsewhere (dictionary-key rules in user type hashing, by-value receivers) keys off “copies”, so it follows the new rule without its own change.
- The use-after-move diagnostic for an affine enum names the declaration
and suggests
@copywhen its payloads copy.
What a learner no longer has to learn
Which kinds of declaration copy implicitly, and why an enum change broke a caller. The question “does this copy?” has a two-line answer.
Phasing
- One statement in all three places; fix the cheat sheet (issue #636).
- Accept and check
@copyon enums. - Undeclared enums move; migrate std, tests and the book.
Rule 2: one formatting path
Today
The formatting page says
print and str.repr never call user formatting methods. The
format strings page
says a field without a conversion uses a public display method, !r uses
debug, and also that “a field is written as print writes it”, which is
false for such types (issue #615). Inside aggregates, neither path calls user
methods. Probe, for a public class P whose display writes <3> and
debug writes P#3:
| Expression | Output |
|---|---|
print(p) | P(x=3) |
str.repr(p)? | P(x=3) |
f"{p}"? | <3> |
f"{p!r}"? | P#3 |
print(ps), f"{ps}"? for ps = [p] | [P(x=3)] |
A non-pub class with display cannot appear in an f-string in its own
module; the diagnostic names __plenty_fstring_… (issue #615). main’s
error report originally used a third hook, std.report.Formattable.format,
over a different Formatter type, and otherwise the str.repr spelling.
Issue #634 subsequently unified the formatter, error, and reporting protocol in
std.fmt; choosing display for entrypoint reports remains separate work. See
entrypoint reporting.
Proposed rule
A value has two spellings.
displayis the default:print(x),f"{x}"andmain’s error report write it.debugis the precise one:str.repr(x),f"{x!r}"and the elements of a formatted aggregate write it. A type withoutdisplayuses itsdebugspelling; a type withoutdebuguses its structure.
So print(p) writes <3>, print(ps) writes [P#3], as Python’s
print([p]) uses each element’s repr. Strings stay raw at the top level
of print and quoted inside aggregates, which is today’s behaviour and is
the same rule (display of str is its text, debug is quoted).
A type’s own methods are found from its defining module whatever its visibility, so a private class formats in its module. Internal names never appear in diagnostics.
Alternatives
- Make f-strings ignore
display, asprintdoes. Consistent, but removes the only user formatting hook from core syntax and breaks std types (std.path,std.complex,std.errorand others) that definedisplay. Rejected. - Keep the paths separate and fix the sentence in format-strings.md.
Documents the difference instead of removing it;
print(p)andf"{p}"still disagree. Rejected. - Use
displayinside aggregates too. Ambiguous for strings and for types whose display text contains separators; Python’sstr/reprsplit exists for this reason. Rejected.
Cost and compatibility
printandstr.reprstart running user code. That code can fail withFormatError, so their result types follow rule 5: unchanged for values with built-in spellings, widened withFormatErrorwhen a user method can run. Programs that print such types and returnResult[(), IoError]need the wider error; the diagnostic names the type whose method can fail.- Output changes for every printed value of a type with
displayordebug. Expected-output fences in the tutorial and reference that print std types with these methods change; the test suite finds each one. main’s error report usesdisplay.std.fmt.Formattablebecomes a transitional fallback while the entrypoint hook migrates. Issue #634 delivered the shared formatter; the meaning of a method namedformatbelongs to the std conventions page (rule 7). This rule fixes which hook core formatting calls.- Automatic formatting still rejects recursive types. Each user method an
aggregate element calls gets its own
std.fmtformatter with a fresh depth of 32 and no byte budget of its own; the methods do not share one depth count across the aggregate. Theprintorstr.reprbuffer grows with checked allocation as before.
What a learner no longer has to learn
Which of three output paths calls which hook, and why print and an f-string
disagree. One rule: display for people, debug for precision.
Phasing
- Fix the false sentence and the private-class lookup; hide internal names (issue #615).
- Route
print,str.reprand aggregate elements throughdisplay/debug. main’s report usesdisplay; its shared formatter dependency is delivered by issue #634.
Rule 3: an owner converts to its view
Today
Update 2026-11-01: phases 1 and 3 shipped. Owners convert to views at view types, byte-reading std APIs take
&[u8], andByteSourceis removed; naming it is a compile error pointing to&[u8]. The remaining&Bytes, start, stopinputs are listed under std conventions. The probes below record the state before phase 1.
Slice views give
&[T] for a list, a Bytes buffer and a bytes literal, so “one &[u8]
parameter reads any of them”. But “a list is not converted to a view
implicitly; pass the whole list as &xs[:]”. Probes:
| Call | Result |
|---|---|
total(&xs) for total(values: &[i64]) | Rejected: pass &xs[:] |
n(b"lit") for n(t: &[u8]) | Rejected: pass &xs[:] |
n(&s) or n("lit") for n(t: &substr) | Rejected: pass &s[:] |
s.startswith(w) for w: &substr | Rejected: expected str, got &substr |
w.upper() for w: &substr | Rejected; the diagnostic lists the dozen methods a &substr has |
n(&xs) for S: ByteSource and xs: list[u8] | Rejected: protocols require classes, enums or bytes |
n(&xs[:]) for the same | Rejected, and the diagnostic names Result[list[u8], AllocError], the copying slice’s type |
ByteSource has two
implementations, Bytes and bytes; a &[u8] view and a list[u8] are not
among them, although &[u8] already unifies all three owners. Byte APIs
therefore take source: &S, start, stop triples that a view expresses
directly.
Proposed rule
Where a view type is expected, a reference to an owner of that view is accepted:
&xsfor&[T],&mut xsfor&mut [T],&buforb"..."for&[u8],&sor a literal for&substr. The conversion is free: it creates the same view as[:], with no copy and no allocation. Every operation that reads text accepts&substr; every operation that reads bytes accepts&[u8].
&xs[:] keeps working. Expected-type positions are parameters, annotated
locals and returns; a view still cannot be stored (B11).
Alternatives
- Keep
[:]explicit. The current rationale is that conversions should be visible. That matters for costs; this conversion has none, and the design goal is “no hidden allocation”, not “no hidden pointer arithmetic”. Rust performs the same coercion from&Vec<T>to&[T]. Rejected. - Accept views where owners are expected (a
&substrargument for astrparameter). This would need an allocation or a second representation forstr. Rejected; text-reading parameters change to&substrinstead, which is B12’s planned scope. - Make
&[u8]implementByteSourceand keep the protocol. Smaller change, but leaves two spellings for “bytes I read”. Accepted as the first phase only; std then moves to&[u8]parameters.
Cost and compatibility
- Phase 1 is additive: code with
[:]keeps compiling. - Phase 2 changes std signatures. Text APIs taking
strto read change to&substr; callers passingstrkeep working through the conversion. Byte APIs taking&S, start, stopchange to&[u8], sow.write(&buf, a, b)becomesw.write(&buf[a:b]). Thestd.ioWriterprotocol’swritesignature changes, which breaks user writers; the std.io page and its examples change with it. Range-error results on those methods (BytesError.OutOfRange) disappear, because view bounds clamp. ByteSourceis removed once nothing in std takes it.- Per-type specialisation of byte generics goes away: one function body serves every byte owner. Codegen cost is a pointer and a length either way.
- The misleading
Result[list[u8], AllocError]diagnostic is fixed in phase 1.
What a learner no longer has to learn
When to write [:], which of &substr and str a text method wants, and
the ByteSource protocol with its start/stop convention. One rule: pass the
owner or a view of part of it.
Phasing
- Owner-to-view conversion at expected view types;
&[u8]satisfiesByteSource; fix the diagnostic. - Text-reading std APIs accept
&substr(part of B12). - Byte-reading std APIs take
&[u8]; removeByteSource.
Box already converts to &T, while Shared does not (survey section 2).
&Shared[T] converts to &T as &Box[T] does (decided 2026-10-31). The
content is immutable while shared, so the view is sound; the borrow keeps
the handle, and so its count, alive for the view’s lifetime. This joins
phase 1.
Rule 4: adapters belong to iterators
Today
Pipelines start with
iter(source). Probe: iter(...) accepts every for source already:
lists, sets, strings, bytes, ranges, reversed, enumerate, zip,
keys(), values(), items(), generators and pipelines. Some sources also
take adapters directly and others do not (issue #622):
| Source | Direct .count() today |
|---|---|
Generator, enumerate(&xs), zip(&xs, &ys), d.items() | Accepted |
range(3) | Rejected: expected str, got range |
xs (list) | xs.map(...): unsupported method map; xs.count(): list count needs an element |
reversed(&xs) | Rejected: reversed is supported in loops only |
d.keys() | Rejected; suggests the allocating keys_snapshot() |
The three dictionary views differ in kind
(dictionaries, issue #624):
keys() is a bindable KeyView with len and set algebra, values() cannot
be bound or measured, and items() starts chains but cannot be bound.
Terminals differ by item type: sum, min and max read copyable reference
items through, while find and reduce reject them (issue #621). len(x)
and in reject std containers such as Deque and Bytes that define
.len() (issue #623).
Proposed rule
A collection, range or view is a source. An iterator is a single-use stream: a pipeline (
Iter[T]) or a generator.iter(source)makes an iterator from anyforsource, anditer(it)on an iterator returns it. Adapters and terminals are methods of iterators only, and every adapter and terminal treats items the same way whatever the source.
This matches Python’s iterable/iterator split and Rust’s Vec/Iterator
split. It is the only choice with no name clashes: a list already has
count(e) and find, a string has count and find, a set has its own
algebra, and none of them is consumed by iterating.
Under the rule:
enumerate(...),zip(...)andd.items()stayforsources anditer(...)arguments, but lose direct adapter methods; the iterator methods.enumerate()and.zip()already exist, soenumerate(&xs).count()becomesiter(&xs).enumerate().count().keys(),values()anditems()are the same kind of thing: bindable borrowed views withlen, usable infor,in(keys) anditer().KeyViewis accepted wherever a borrowed set is read.find,reduceand every other terminal copy copyable reference items asminandmaxdo.- Diagnostics for an adapter on a source suggest
iter(source).
Alternatives
- Give every source the adapter methods directly. Clashes with existing
collection methods (
list.count,list.find,str.find), and makes it unclear whether a call consumes. Rejected. - Remove direct methods from generators too, so only
Iterhas them. A generator is single-use and supportsnext, exactly like a pipeline; denying it methods would be a distinction without a reason. Rejected. - Fix only the diagnostics. Issue #622’s minimum. Keeps two rules. Rejected as the end state; it is phase 1.
Cost and compatibility
- Direct adapters on
enumerate,zipanditems()are rare: two uses in Plenty code and pages, about thirty intests/*.rs(mostlytest_pipelines.rs). Each becomesiter(...)or the iterator method. - Dictionary-view changes are additive.
find/reduceon copyable references is additive.lenmethod dispatch is implemented for classes and enums (issue #623). Membership on std containers still depends on B49’s protocol work; unsupported operations name their actual type.
What a learner no longer has to learn
Which sources take methods directly, three kinds of dictionary view, and
which terminals read through references. One rule: iter() first, and
everything that works on one iterator works on all.
Phasing
- Diagnostics suggest
iter(source)(issue #622);find/reducecopy copyable items (issue #621). - Uniform dictionary views (issue #624).
- Remove direct adapters from
enumerate,zipanditems(). lenmethod dispatch delivered (issue #623);inon std containers remains with B49.
Rule 5: an error type names only what can fail
Today
Allocation failure has several shapes (issue #632). The table and the probe
describe the state before phasing items 2 and 3. Item 2 made f-strings without
user formatting fail with AllocError; item 3 changed the FormatError,
std.bytes and union rows and std.bytes.with_capacity(-1):
| Operation | Allocation failure reported as |
|---|---|
[1, 2], a + b, copy(x) | AllocError |
| An f-string | std.fmt.FormatError.Bytes(BytesError.Allocation(...)) |
print | IoError.Data(DataError.Allocation(...)) (I/O errors) |
std.bytes.empty(), from_text | BytesError.Allocation |
clist.append(row) | AppendError[T], which returns the row |
| sketch, linalg, mime, graph | A union E | AllocError |
Switching a + b to f"{a}{b}" changes the function’s error type, and
FormatError needs an import although every f-string produces it. Probe:
-> Result[str, FormatError] gives unknown type FormatError, and
-> Result[str, AllocError] returning f"n={n}" is rejected. Tuple slices
return Result although tuples never allocate (issue #625).
Invalid arguments follow four policies (issue #629):
| Argument | Today |
|---|---|
range step | Compile error unless NonZero (ranges) |
split(""), even a literal | Run-time trap (text splitting) |
repeat(-1) | Empty result (text building) |
list.with_capacity(-1) | Err(CapacityOverflow) (construction) |
std.bytes.with_capacity(-1) | Err(OutOfRange) |
.par(&pool, chunks=0) | Trap with the unrelated integer cast out of range |
Proposed rule
An operation’s error type names only failures it can produce. If allocation is its only failure, the type is
AllocError. A domain error type that can also fail to allocate carries it as one variant,Allocation(AllocError). An operation that cannot fail returns noResult.
For f-strings: Result[str, AllocError] when every field has a built-in
spelling and no spec is computed at run time; Result[str, AllocError | FormatError] when a field runs a user display/debug or a computed spec
can be invalid. FormatError becomes a prelude name. print and
str.repr follow the same rule (see rule 2). Tuple slices return the tuple.
Invalid arguments.
- A size passed to allocation that cannot be allocated, including a negative one, is
AllocError.CapacityOverflow.- Slice bounds and repeat counts clamp, as in Python. This is the operation’s defined result, not an invalid argument.
- Any other argument outside its documented domain traps, with a message naming the operation and the argument. A compile-time constant outside the domain is a compile error. Where a type can state the domain (
NonZerofor a range step), the parameter uses it.Erris for failures the caller cannot rule out by checking its arguments: allocation, I/O, and external data.
So split("") with a literal is a compile error and with a variable traps
(as now); std.bytes.with_capacity(-1) is CapacityOverflow; chunks=0
traps naming chunks. repeat(-1) and slice clamping are unchanged.
Alternatives
- Always a union, never a variant (
IoError | AllocErroreverywhere). Uniform for matching, butprintwould needAllocErrorin about 390mainsignatures that now returnResult[(), IoError], and the survey’s agreed wording keeps domain errors for APIs that have them. Rejected. About 42 std error types already use anAllocationvariant and four use a union; the four change. - Keep
FormatErroron every f-string and add it to the prelude. Fixes the import but not the signature change from+to an f-string, which is the more common edit. Rejected; the prelude addition is kept for the user-method case. - Make every invalid argument an
Err. Puts a?on calls whose failure is a bug the caller could rule out, which teaches ignoring it. Rejected. - Trap on negative
repeatcounts. Breaks the padding idiom" ".repeat(width - len(s))that Python habits rely on. Rejected.
Cost and compatibility
- f-string signatures: functions returning
FormatErrorfor f-strings that only use built-in spellings must change toAllocError(or keep a union that contains it). This is the largest edit; it is mechanical, and the diagnostic states the new type. Formatting throughout.format(f"...")indisplaymethods keepsFormatError. FormatError.Bytes(BytesError.Allocation)becomesFormatError.Allocation(AllocError); code matching the nested shape changes.std.bytesconstructors returnAllocError; their callers’ signatures narrow.- The four union-style std modules move to an
Allocationvariant. This is shared with the std conventions page (rule 7). AppendError[T]remains: it returns the moved row, a separate convention (also issue #630 forspawn) that the std conventions page owns.- Tuple slices:
t[0:2]?becomes an error with the slice-view wording (issue #625).
What a learner no longer has to learn
Five shapes of allocation failure, an import for f-strings, and four invalid-argument policies. One rule for each: the type names what can fail, and an argument bug traps.
Phasing
FormatErrorin the prelude (issue #632); infallible tuple slices (issue #625); literalsplit("")rejected and thechunksmessage (issue #629).- f-strings,
printandstr.reprreportAllocErrorwhen no user method runs. Done:printandstr.reprwith B89 phase 2 (issue #615), f-strings with B92 phase 2. A:cspec counts as computed, since an out-of-range scalar isInvalidSpec. - std error types:
Allocation(AllocError)variants,std.bytesconstructors, negative capacities. Done with B92 phase 3:FormatError.Allocation,std.bytesconstructors returnAllocError, andstd.graph,std.sketch,std.linalg,std.shlexandstd.mime(newMimeError) carry the variant;std.actor.stagereturnsStageError.Bytesmethods that fail only to allocate, and sketchto_bytes, returnAllocError. Out-of-domain arguments that depend on run-time state trap through the std-only__trapstatement, with a message naming the operation and argument:Bytesranges and indices (includingread_fileandread_stdin), andRng.index,sample_indicesandfillcounts and ranges. Their results narrow:Bytes.getreturnsu8,sliceResult[Bytes, AllocError], the scans plain values, andRandomErrorlosesOutOfRangeandAllocation. An empty random population staysEmptyRange, since data can produce it. Encoding offsets, UUIDwrite_bytes, cryptofinish_into, and CSV field positions now trap on caller bounds errors (issue #667). Encoding pack/unpack, UUID writes, and CSV positions return plain values. Streamreadranges and TLS session transfers follow the same trap rule (issue #666). Unreachable range variants and byte-error adapters are removed (issue #668). Statically checkable domains (positive capacities and limits) wait for a type that expresses them;BytesError.OutOfRangeremains forstd.actor.netconnection limits.
Rule 6: every callable form is a value wherever a callback is accepted
Today
Function values and closures give four callable types, two constraints and three passing modes. Which forms pass depends on the callee (issues #617–#620). Probes:
| Form | Pipeline filter/map | sort_by_key | f: Callable[...] | f: &mut F |
|---|---|---|---|---|
| Named function | Yes | Yes | Yes | No: named function declarations are immutable |
| Closure literal | Yes | No: sort_by_key requires a named function or reusable closure | Yes, without captures | Needs a binding, as any borrow does |
| Stored closure, no captures | No: pipeline callbacks require a named function or reusable closure | Yes | Yes | Yes, in a mut binding |
| Stored closure with captures | Yes | Yes | No (code address only) | Yes, in a mut binding |
Variant constructor E.A | No | No | Yes | Not probed |
Some, Ok, Err | Not probed | Not probed | No: Some is not a function value | Not probed |
Class constructor Point | Not probed | Not probed | No: unknown binding Point | Not probed |
A closure with a mut capture passed as &F fails at instantiation with
cannot borrow shared reference as mutable, without naming the capture
(issue #619). Behaviour can differ too: inside checked:, callback overflow
is Err for a closure literal and a trap for a named function (issue #614).
Proposed rule
Named functions, closures with or without captures (stored or literal), variant constructors and class constructors are callable values. Each passes wherever a callback with its signature is accepted, and it behaves the same whichever form it has.
Concretely:
- Pipelines,
sort_by_keyand every other builtin callback parameter accept every form. Each currently accepts a different subset: pipelines reject a stored capture-free closure,sort_by_keyrejects a literal. &mut fis accepted for a named function, which has no state to mutate, so a&mut Fparameter accepts every reusable callable.&Fremains the default for read-only callbacks; its diagnostic for amutcapture names the capture and suggests&mut F.- A class constructor
Pointis a function value with the constructor’s positional signature. A fallible initializer gives aResult[Point, AllocError]result, as a call does. - Generic variants, including
Some,OkandErr, follow the rule generic functions already follow: an explicit instance (Option[i64].Some) or a concrete expectedCallablespecialises them. Probe: a genericident[T]passed tof: Callable[[i64], i64]checks today, whileOption[i64].Someis rejected. - Inside
checked:, whether overflow isErrdepends on where the code is written, not on the callback’s form (decided 2026-10-31, issue #614). Arithmetic whose text is in the block, including the bodies of closures defined in the block (literal or stored), reportsErr. Any other function, a named function or a closure defined elsewhere, behaves as its signature says: an infallible signature keeps trapping, and a function that can fail returnsResult, which the block handles withmatchor?like any other call. Builtin reductions such assumare calls, so theiri64result traps on overflow, as the pipelines page already says; to sum with checking, writefoldwith a closure literal in the block. Today a lowered closure literal reportsErrwhile the numbers page says closures defined in a block keep trapping; both change to this rule.
Cost and compatibility
All of rule 6 is additive except two items. The checked: fix (issue #614)
changes which programs trap: a closure defined in a block, stored or
lowered, now reports Err, while sum inside a block, which reports Err
today because of pipeline lowering, traps as documented. A closure defined
in a block may not outlive it, since a later overflow would have no block to
exit (decision 5). Constructor values make a class name usable as a value
expression; how that interacts with local bindings of the same name needs a
check against the name-resolution rules in
names. Worker-thread effect
certification originally rejected thin function values, including constructor
values (local PR #681). Issue #661 adds conservative certification of every
matching address in the closed program; see
thin callable targets.
This is an effect rule independent of callable forms.
What a learner no longer has to learn
Which callback positions accept which forms, why adding an unused capture
fixes an error, and why &mut rejects a named function. One rule: anything
you can call can be passed.
Phasing
- Pipelines and
sort_by_keyaccept every closure form (issue #617);&mutnamed functions and the&Fdiagnostic (issue #619); lead the reference with the constraint form (issue #620). - Variant constructors in pipelines; class constructors as values; generic variants with an expected type (issue #618).
- Inside
checked:, code written in the block and closures defined in it reportErr; other functions behave as their signatures say (issue #614). Delivered as lexical checked callbacks; the reference documents scope and callable-erasure limits.
What the six rules remove
| Rule | Exceptions removed | Issues |
|---|---|---|
| 1 | Three copy lists; implicit enum copying | #636 |
| 2 | print/f-string disagreement; private-class f-string failure; third report hook | #615, part of #634 |
| 3 | [:] at view parameters; &substr rejected by text APIs; &[u8] not a byte source | B12 scope |
| 4 | Five sources without methods, three kinds of dictionary view, uneven terminals, len/in | #621–#624 |
| 5 | Five allocation shapes; f-string import; four invalid-argument policies; fallible tuple slices | #625, #629, #632 |
| 6 | Seven callable forms with different acceptance; form-dependent checked: | #614, #617–#620 |
Not covered: the defects in survey section 1 that are bugs, not rules (issues #612, #613, #616), diagnostics for Python habits (issue #627), reference diagnostics (issue #628), and std conventions (rule 7; issues #630, #631, #633–#635). Survey items with existing backlog homes (B16 contextual literals, B36 builtin argument names, B11 borrowed returns) stay there.
Decisions
Recorded 2026-10-31 after review:
- Rule 1: phase 3 waits for B84’s copyable bound, and B84 is brought forward.
- Rule 3:
&Shared[T]converts to&T, asBoxdoes. - Rule 6:
Some,OkandErrfollow the generic-function rule; issue #556’s exclusion is lifted unless implementation shows the fallback is needed. - Rule 6:
checked:covers arithmetic written in the block and in closures defined in it, whatever their form. Every other function, including a named function passed as a callback and builtins such assum, behaves as its signature says: it traps, or it returns aResultthat the block handles or propagates with?, which also leaves the block. A function’s possible failures are part of its signature, not of the context that calls it. - Rule 6: a closure defined in a
checked:block may not leave it. It leaves by assignment to a binding declared outside the block, as the block’s value or a return value, by being stored in a collection, field,Sharedor channel, or by being passed by value to a function that keeps it (spawn,pool.submit). Passing it as&for&mut fto a function called in the block is allowed. The checker can treat the closure as borrowing a block-scoped value, so existing borrow checking rejects every escape; the diagnostic nameschecked:and suggests defining the closure outside the block. Rejection can be relaxed later without breaking programs; trapping when called outside the block would reintroduce behaviour that depends on where a closure is called.
Staged plan and suggested backlog wording
Each rule’s phase 1 is small and independent; the issues listed are the
work items. Later phases change behaviour and need the user’s agreement per
row. The “Proposal” links below point into this page; in the backlog they
become proposals/consistency-rules.md#....
| ID | Area | Suggested wording |
|---|---|---|
| B88 | Copy rule | One copy rule in the reference, cheat sheet and skill, checked by a test; @copy on enums; undeclared enums affine after a migration that annotates every currently copyable enum. Generic enums wait for B84’s copyable bound, so B84 moves earlier. Proposal; issue #636. |
| B89 | Formatting path | print/f"{x}"/main’s report use display; str.repr/!r/aggregate elements use debug; private types format in their module. Report hook migration uses the shared formatter delivered by issue #634. Proposal; issue #615. |
| B90 | Owner-to-view conversion | &xs, &s, b"..." accepted at view types; &[u8] satisfies ByteSource; &Shared[T] converts to &T like Box; then byte-reading std APIs take &[u8] and ByteSource is removed. Text APIs accepting &substr stay in B12. Proposal. |
| B91 | Iteration sources | Adapters only on iterators; iter() for every source; uniform dictionary views; uniform terminals over copyable references. Membership on std containers stays B49; len method dispatch is delivered (issue #623). Proposal; issues #621, #622, #624. |
| B92 | Error types and invalid arguments | Allocation-only failure is AllocError, including f-strings without user formatting; domain errors carry Allocation(AllocError); FormatError in the prelude; infallible tuple slices; the four-part invalid-argument policy. Proposal; issues #625, #629, #632. |
| B93 | Callable forms | Every callable form accepted at every callback parameter; &mut named functions; class constructors and generic variants, including Some/Ok/Err, as function values; inside checked:, closures defined in the block report Err and other functions follow their signatures. Callable names stay (issue #24). Proposal; issues #614, #617–#620. |
Completed proposals
These proposals have shipped within their agreed scope, or completed the experiment or investigation identified in their entry. They preserve design rationale and alternatives; the reference and tutorial describe current behavior. Further extensions listed below do not make the completed scope pending again. The backlog remains the only work queue.
See the proposal index for partially implemented and unimplemented designs and the policy for moving completed proposals.
Positive and NonNegative integer domain types
Status: Implemented. The language types are recorded in
B95
and the std migration in
B95 std limits.
The numbers reference
defines the types and the
std conventions the
parameter rule. What shipped differs from the tables below as listed in
As shipped; Bounded
now has its own implementation under B96, and a compile-time check of fixed
ceilings is issue #686.
Problem
A std parameter with a fixed domain, such as a byte limit that must not be
negative or a buffer capacity that must be at least one, is checked inside the
function and reported as an Err:
pub def parse(text: &str, max_bytes: i64) -> Result[Url, UrlError] # InvalidLimit for max_bytes < 0
pub def buffered_reader[R](inner: R, capacity: i64) -> Result[BufReader[R], StreamError] # OutOfRange for capacity <= 0
The failure is a caller bug that the caller could rule out, but every call
pays for it: a ?, an error variant to match, and a check on each call. Almost
every call passes a literal or a default, so the check can never fail at those
sites. The consistency rules
(rule 5.3) already say “where a type can state the domain, the parameter uses
it”, and NonZero[i64] does this for range and slice steps. No type states
“at least zero” or “at least one”.
The migration list covers the public std parameters and
Limits fields whose lower bound is 0 or 1 and is checked at run time
today.
Summary
| Question | Decision |
|---|---|
| New types | Positive[T] (value > 0) and NonNegative[T] (value >= 0) |
T | Signed integer types only: i8, i16, i32, i64 |
| Construction | Integer literal where one is expected (checked at compile time), or try_from |
try_from error | RangeError.OutOfRange, as NonZero.try_from and T.try_from |
| Reading back | .get() returns T; no implicit conversion to T |
| Arithmetic | + and * keep the type; - returns T; other operators are compile errors |
Relation to NonZero | Separate nominal types; NonZero.from(p) and NonNegative.from(p) widen a Positive |
| Copy, format, hash | As NonZero: copies, prints as its integer, valid dictionary key |
len() | Unchanged, returns i64; no implicit conversion |
| Sizes | Stay i64 (integer types); only the wrapper is new |
The types
Element type
Positive[T] and NonNegative[T] take a signed integer type. An unsigned
type is a compile error that names the existing spelling, because the domain
already has one:
Positive[u64] # error: every nonzero `u64` is positive; use `NonZero[u64]`
NonNegative[u32] # error: every `u32` is non-negative; use `u32`
One domain has one spelling. Std quantities are i64 by convention, so
Positive[i64] and NonNegative[i64] are the forms std uses; the narrower
signed widths exist for user data and cost nothing extra, since the
implementation is generic over T.
Positive and NonNegative are builtin names and cannot be redefined, like
NonZero.
Construction
The rules are those of NonZero, with a different domain check.
def per_page(items: i64, size: Positive[i64]) -> i64:
items // size.get()
def main() -> Result[(), IoError | RangeError]:
print(per_page(100, 20))? # literal converts at compile time
print(per_page(100, 0))? # error: `0` is not a `Positive[i64]`
configured = 25
size = Positive.try_from(configured)?
print(per_page(100, size))?
limit: NonNegative[i64] = 0 # 0 is in the domain
bad: NonNegative[i64] = -1 # error: `-1` is not a `NonNegative[i64]`
Ok(())
- An unsuffixed integer literal, possibly negated, converts where a
Positive[T]orNonNegative[T]is expected: an argument, an annotated binding, a return value, a class constructor argument, a field assignment, a default parameter value, or insideSome(...). A literal outside the domain is a compile error. These are the positions where aNonZeroliteral converts today; a probe with the current compiler confirmed defaults, returns, constructor arguments, field assignment andSome(3)for anOption[NonZero[i64]]parameter. Positive.try_from(value)returnsResult[Positive[T], RangeError], withTtaken fromvalue;Positive[T].try_from(value)namesTand requires aT.NonNegative.try_fromis the same. Out of domain isErr(RangeError.OutOfRange).- The constructor is private:
Positive(3)is a compile error that names the two conversions.
RangeError is the error every integer narrowing already uses
(checked conversion),
so one error type covers i32.try_from, NonZero.try_from,
Positive.try_from and NonNegative.try_from, and a function that does
several returns Result[_, RangeError] without a union. It has one variant,
so an exhaustive match needs one arm.
Reading back and mixing with T
.get() returns the T. There is no implicit conversion to T: an operation
that mixes Positive[i64] with i64 is a type error whose hint shows
.get() or try_from, as for NonZero. Implicit unwrapping would let a
domain value silently become an i64 and come back only through try_from,
which hides where the check happens.
Arithmetic and comparison
Operands must have the same type. An integer literal beside a domain operand
takes that operand’s type, so n + 1 is a NonNegative[i64] and size * 0
is a compile error for a Positive.
| Operation | Positive[T] | NonNegative[T] | Why |
|---|---|---|---|
a + b, a += b | Positive[T] | NonNegative[T] | Sum stays in the domain; overflow traps |
a * b, a *= b | Positive[T] | NonNegative[T] | Product stays in the domain; overflow traps |
a - b | T | T | Can leave the domain |
unary + | Positive[T] | NonNegative[T] | Identity |
unary - | T | T | Leaves the domain |
//, %, bitwise, shifts | Compile error, suggests .get() | Same | As NonZero; a domain divisor is a later addition (decision 4) |
==, !=, <, <=, >, >= | bool | bool | Same-type operands or a literal |
Overflow in + and * traps, or is an Err in a
checked: block, so a
sum cannot wrap to a negative value. A comparison with a literal outside the
domain, such as size > 0 for a Positive, is a compile error, as
step == 0 is for NonZero: the literal is not a Positive. n >= 0 for a
NonNegative compiles, since 0 is in the domain.
Relation to NonZero
A Positive[T] is nonzero, but it is not a NonZero[T]: Plenty has no
subtyping, and each is a nominal @newtype class. Code that holds a
Positive and needs a NonZero (a range step) or a NonNegative (a limit)
widens it explicitly:
def chunks(total: i64, size: Positive[i64]) -> Result[(), IoError]:
for start in range(0, total, NonZero.from(size)):
print(start)?
Ok(())
NonZero.from(p) and NonNegative.from(p) take a Positive[T], cannot fail,
allocate nothing and compile to nothing. A from that cannot fail beside a
try_from that can matches the pair the reader already sees in T.try_from.
Widening is explicit (decision 3); see
Alternatives.
There is no conversion from NonNegative to NonZero or Positive other
than try_from(n.get()), since zero is in its domain.
Copy, format, hash and layout
As NonZero[T]:
- the value has
T’s representation and allocates nothing; - it copies, is a valid dictionary key and set element, and is usable as a
field, collection element and
@copycomponent; print, f-strings anddebugshow the integer, and format specs apply as toT.
Both are declared in the builtin arithmetic-errors module beside NonZero
(src/frontend/modules.rs), as private @newtype @copy classes with a
get() method.
len() and sizes
len() returns i64 and stays so. Making it return NonNegative[i64] would
break i < len(xs) with i: i64, range(len(xs)), and len(xs) - 1 used
as an index, which are the most common uses of len(). A rule that converts a
len() call where a NonNegative[i64] is expected, like a literal, is
possible but is not needed now:
- Allocation sizes are out of scope.
list.with_capacity,Bytes.reserve,resize,LruCache(capacity)andstd.actor.stagereport a negative size asCapacityOverflow(rule 5.1, local PR #804), sowith_capacity(len(x))is unchanged. - No
Positiveparameter can takelen(), which can be zero. - A search of
stdlib/, the tutorial, the reference,examples/andtests/found two calls that pass a length expression to a migrated parameter, both inside std with arithmetic (text.plentybuilder(self.max_bytes - used_bytes + ...)andregex.plentyself.scan(2 * len(self._names)), which is private). Alen()-only rule would cover neither.
Where a length must become a limit, the call is explicit:
mut out = std.text.builder(NonNegative.try_from(2 * len(text))?)?
Inside std, public functions call .get() once and pass i64 to private
helpers, so internal arithmetic is unchanged. A len() conversion rule is
not part of this design (decision 5).
Migration list
A parameter migrates when its whole static domain is >= 0 or >= 1. The
lists below are the public std and builtin parameters with that lower bound
that return Err or trap today. P is Positive[i64] and N is
NonNegative[i64]; each parameter is i64 today unless stated. Result types
are unchanged except where noted, and follow PR #804 where it changes them.
Group A: the invalid-argument failure leaves the error type
| Function or class | Parameters after | Removed today |
|---|---|---|
std.url.parse(text, max_bytes) | max_bytes: N | UrlError.InvalidLimit (its only use) |
std.db.kv.Store.scan(..., max_rows, ...), scan_all(..., max_rows, ...) | max_rows: N | KvError.InvalidLimit (its only uses) |
std.markdown.Limits | max_input, max_depth, max_output: N | MarkdownFault.InvalidLimits |
std.http.body.FormLimits | max_bytes, max_fields: N | BodyError.InvalidLimit, with the two rows below |
std.http.body.MultipartLimits | max_parts, max_head, max_field, max_file, max_body: N | Same |
std.http.body.read_json(exchange, limits) | via std.json.Limits.max_input: N | Same |
std.actor.net.connection(stream, limits) | std.actor.net.Limits input, output: P | BytesError.OutOfRange; the result can narrow to AllocError |
builtin channel[T](capacity) | capacity: P (from u64) | ChannelError.InvalidCapacity |
builtin ThreadPoolExecutor(max_workers, queue_capacity) | both P (from u64) | PoolError.InvalidSize |
.par(&pool, chunks=N) | chunks: P | The par chunks must be positive trap |
The two builtins move from u64 to Positive[i64] rather than to
NonZero[u64], which removes their listed deviation from the
integer-type convention in
the same break (decision 2, issue #665).
Group B: the call loses this failure; the error type keeps the variant for other causes
| Function or class | Parameters after | Variant kept for |
|---|---|---|
std.io.buffered_reader(inner, capacity), buffered_writer(inner, capacity) | capacity: P | StreamError.OutOfRange: stream ranges |
std.io.typed.buffered_reader(inner, capacity) | capacity: P | Same |
std.io.BufReader.read_until(delimiter, out, limit) | limit: P | Same |
std.io.read_up_to(reader, out, limit) | limit: N | Same |
std.process.Child.communicate(input, limit), output(limit) | limit: N | Same |
std.csv.reader(inner, dialect, capacity) | capacity: P | CsvError.OutOfRange: field indexes |
std.log.text_sink(writer, max_line) | max_line: P | LogError.InvalidSpec: specs |
std.log.memory_sink(capacity) | capacity: N | Same |
std.fmt.Formatter.floating(value, width), integer(value, width), unsigned(value, base, width) | width: N | FormatError.InvalidSpec: base, specs |
std.text.builder(max_bytes), join(..., max_bytes) | max_bytes: N | TextError.InvalidLimit: wrapper geometry, below |
std.text.hard_wrap(text, width, max_bytes) | width: P, max_bytes: N | Same |
std.text.wrap, fill, shorten, text_wrapper | width: P, max_bytes, max_input_bytes: N | Same |
std.text.dedent, indent, indent_with | max_bytes, max_input_bytes: N | Same |
std.text.TextWrapper fields | width: P, tabsize: N, max_lines: Option[P], max_bytes, max_input_bytes: N | An indent wider than width, a placeholder too long |
std.fs.copy_from(..., max_bytes, ...), copy_file(..., max_bytes, ...) | max_bytes: N | FsError.InvalidInput: names, ceilings |
std.fs.glob(parent, pattern, max_entries, max_matches, max_steps) | all three N | Same: pattern length |
std.system.System.block_devices, disk_io | max_devices: N | SystemError.InvalidInput: max_bytes, pid |
std.system.System.process_ids(max_entries) | max_entries: N | Same |
std.archive.zip_reader(inner, max_entries) | max_entries: N | ArchiveError.InvalidInput: names, modes |
std.compress.inflater(format, limit), decompress(format, data, limit) | limit: N | CompressError.OutOfRange: crc32 seed |
std.http.encode_chunk_size(length) | length: N | StreamError.OutOfRange in HttpError |
std.http.typed.fixed_body(inner, length) | length: N | FramingError.OutOfRange: read ranges |
std.http.Limits, decoder and client | max_headers, max_body: N | HttpError.InvalidLimit: max_line, max_head >= 2 |
std.http.server limits | max_line, max_head, max_headers: P, max_body: N | Other Config fields with other bounds |
std.http.websocket.Config | max_frame, max_message: P | WebSocketError.InvalidConfig: one other mapping |
std.actor.http.server config | max_requests: P, limits.max_line, max_head, max_headers: P, max_body: N | ServerError.InvalidConfig: capacity, heap, buffer minimums |
std.actor.net.listener(..., capacity, heap, ...) | capacity, heap: P | NetError: transport failures |
std.actor.net.Connection.reserve(wanted) | wanted: N | NetError: closed connection |
std.actor.tls.over(net, session, input) | input: P | TlsError.Net: transport failures |
std.image.Limits | max_width, max_height, max_bytes: N | ImageError.OutOfRange: dimensions |
std.stat.quantiles(values, n) | n: P | StatError.Domain: float parameters |
std.math.factorial(n) | n: N | MathError.Domain; Overflow above 20 stays |
Group C: the lower bound is typed; a fixed ceiling remains
| Function or class | Parameters after | Ceiling |
|---|---|---|
std.fmt.formatter, stderr_formatter, string_formatter | max_bytes: N, max_depth: N | max_depth <= 64; also scratch.len() >= 32 |
std.json.Limits, std.toml.Limits | max_depth, max_input, max_output: N | max_depth ceiling |
std.xml.Limits | max_token: P, others N | max_depth <= 1024 |
std.regex.Limits | all seven N | max_nesting ceiling |
std.template.Limits | max_context_nodes, max_depth: P, others N | max_depth <= 64 |
std.unicode.nfd, nfc, nfkd, nfkc | max_scalars: N | i64.MAX // 4 |
std.sketch.quantiles(capacity), quantiles_from_bytes(data, max_capacity) | P | 1048576 |
std.sketch.frequent_items(capacity), frequent_items_from_bytes(data, max_capacity) | P | 65536 |
std.sketch.xor.build, from_bytes | max_keys: N | key_limit |
std.sketch.cuckoo.cuckoo, from_bytes | max_buckets: P | bucket_limit; initial_buckets must be a power of two |
std.fs.TempDir.cleanup_tree, std.fs.remove_tree | max_entries, max_depth: P | max_depth <= 64 |
std.fs.temporary_dir(parent, prefix, attempts) | attempts: P | 1024 |
std.tempfile.directory(..., max_entries, max_depth) | max_entries, max_depth: P | max_depth <= 64 |
std.jwt decoding max_size | P | max_token_size |
std.crypto.hkdf_sha256(..., size) | size: N | 8160 |
std.image.new, from_pixels | width, height: P | 2147483647 |
std.actor.Context.spawn, spawn_isolated, Stage.start | limit: P | max_heap_limit |
std.system.System.process(pid, max_bytes) and the max_bytes of load_average, uptime, cpu_count, memory | max_bytes: P | != i64.MAX; pid != 0 |
A ceiling keeps an Err variant, so the migration removes the ? from these
calls only if the ceiling stops being an Err. Rule 5.3 says an argument
outside its documented domain traps and a constant outside it is a compile
error; under that rule these ceilings trap, as the run-time domains in PR #804
do, and the variants leave the error types. This design adopts that rule
(decision 1): a ceiling traps, and a literal above a constant
ceiling is a compile error.
Not migrated
| Parameter | Reason |
|---|---|
with_capacity, Bytes.reserve/resize, Heap.reserve, graph reserve, LruCache(capacity), std.actor.stage counts, std.secrets.token_bytes/token_hex/token_urlsafe | Allocation sizes: CapacityOverflow (rule 5.1, PR #804) |
str.repeat(count), slice bounds | Clamp (rule 5.2) |
std.ready.selector(capacity) | Clamps to 1..most_events today; a deviation to fix separately |
regex replace limit | A negative value means unlimited; a sentinel, not a domain (Option would state it) |
Bytes range methods, std.actor.net.Connection.consume/commit/forward, Context.restart, std.image.Image.crop, std.random.Rng.index/sample_indices, csv/sqlite/watch indexes | The upper bound depends on the receiver or another argument; one trap covers both ends (PR #804) |
std.http max_line, max_head (>= 2), std.sketch.hyperloglog precision (4..18), encode_png level (0..9), actor server capacity, heap_limit, buffer minimums | Lower bound other than 0 or 1; needs Bounded |
std.compress.crc32 seed (0..u32 max), archive modes (0..4095) | The parameter should be a narrower type (u32, u16); separate fix |
std.regex.find_all/split max_matches/max_pieces, std.http.cookie.cookies limit, std.db max_length | Not checked today; their negative behaviour needs its own review first |
As shipped
Differences from the migration list, recorded after the migration (2026-11-02):
- Shared HTTP limits.
std.http.Limitsserves the client, decoder and both servers, and allowsmax_headers = 0. Itsmax_headersandmax_bodyareNonNegative[i64];max_lineandmax_head(>= 2) stayi64with theirInvalidLimitcheck, so the server rows’Ptyping waits forBounded(B96). - Removed beyond the tables.
std.actor.StartError.Limit,SystemError.InvalidInput,CsvError.OutOfRange(field indexes already trapped after PR #804) andBytesError.OutOfRange(its last cause was the actor networking limits). - Narrowed results.
std.actor.net.connectionreturnsAllocError, soListener.acceptaddsAllocErrortoNetError.listenerkeepsResult[Listener, NetError]. - Also migrated.
std.actor.tlsserverandclientinput sizes (they forward toover),std.systemblock_devices/disk_iomax_bytes,std.compress.decompressor(limit),std.fmt.to_string/to_debugandstd.report.to_stringlimits. - Ceilings that trap. As decision 1, plus
std.actor.net.listenerheap abovemax_heap_limit.quantiles_from_bytesandfrequent_items_from_byteshad no ceiling and gained none. The compile-time rejection of a literal or module constant above a ceiling is implemented with std-only declaration metadata (issue #686); dynamic inputs retain the runtime guards. - Constants. A module constant could not have a domain type, so std
passed constant limits through private getters such as
def max_url() -> NonNegative[i64]:. Issue #685 later added domain constants (const max_url: NonNegative[i64] = 8192) and replaced the getters. - Internal lengths. Two std callers of
encode_chunk_sizeconvert alen()result withNonNegative.try_from(n).unwrap(), which cannot fail (decision 5).
Example
Before:
def lines(path: &Path) -> Result[list[str], AppError]:
file = open_file(path)?
mut reader = std.io.buffered_reader(file, 65536).map_error(AppError.Stream)?
mut out = std.text.builder(1048576).map_error(AppError.Text)?
...
After, with literals: the two map_error adapters for impossible failures
go, and the builder’s remaining failure is allocation.
def lines(path: &Path) -> Result[list[str], AppError]:
file = open_file(path)?
mut reader = std.io.buffered_reader(file, 65536)
mut out = std.text.builder(1048576)?
...
buffered_reader can still fail to allocate its buffer, so it keeps a
Result; whether that result narrows to AllocError follows rule 5. With a
configured value, the check moves to where the value is read:
def configured(settings: &Settings) -> Result[Positive[i64], AppError]:
Positive.try_from(settings.buffer_size).map_error(AppError.Config)
Alternatives
Bounded[T, lo, hi]. States every domain in the tables, including ceilings and>= 2. Initially deferred in favor of two fixed domains; bounded integers now implement this as a separate type using B83 value parameters. The arithmetic rules would depend on the bounds:Bounded[i64, 1, 64] + 1has no simple result type.Boundedtherefore uses explicit.get()for arithmetic; the two fixed domains keep their existing operations.- Unsigned parameters.
max_bytes: u64rules out negatives without a new type, but breaks the i64 convention for sizes, forces casts at everylen()and index, andu64cannot state>= 1. Rejected. - Plain
stdclasses withtry_fromonly. No compiler change, but no literal conversion, so every call writesPositive.try_from(64)?, which is the?this proposal removes. Rejected. - Implicit widening to
NonZeroandNonNegative. SavesNonZero.fromat range steps. It adds a third way for a domain value to appear where one is expected (literal,try_from, widening), and is subtyping in all but name. Rejected (decision 3). - Implicit unwrapping to
T.size + offsetwithoffset: i64would work, but the domain value disappears silently and comes back only throughtry_from. Rejected, as forNonZero. - Keep run-time checks, document them. Leaves the
?and the variant on every call. Rejected by rule 5.
Cost and compatibility
- Compiler.
NonZerois 247 lines insrc/frontend/nonzero.rs, plus hooks in operators, f-strings, metadata and diagnostics (116 mentions in 22 files undersrc/). Generalizing it to a domain enum (NonZero,Positive,NonNegative) with a per-domain literal check, operator table and diagnostic is the expected shape, not three copies. The signed-only bound is a new check:IntTypeadmits all eight integer types. Codegen needs no change, since the representation isT. - Breaking changes. Every migrated parameter rejects a plain
i64variable; literals and defaults keep compiling. A call passing a variable addsPositive.try_from(x)?orNonNegative.try_from(x)?, and the diagnostic shows that form. Code that matches a removed variant (UrlError.InvalidLimit,KvError.InvalidLimit,MarkdownFault.InvalidLimits,BodyError.InvalidLimit,ChannelError.InvalidCapacity,PoolError.InvalidSize) gets an unknown variant error. Code that constructs or mutates aLimitswith a variable converts it; code that reads a field calls.get(). - Std internals. Public functions call
.get()and passi64to private helpers, so most of the change is at signatures and invalidfunctions, which shrink or disappear. - Docs. The numbers reference gains the two types next to
NonZero; each module page in the tables updates its signature, error and limit rows;conventions.md“Invalid arguments” and “Limits” state that a fixed lower bound is a type; the cheat sheet, tutorial lessons that pass computed limits, theplentyskill andsignatures.mdfollow. - Performance. No change to representation or allocation; each migrated call drops one comparison and branch.
Phasing
- Language:
Positive[T]andNonNegative[T], literal conversion,try_from,.get(), operators,NonZero.from/NonNegative.from, formatting and hashing, with the numbers reference and cheat sheet. - Group A, which removes variants and needs the largest doc change, with the
builtins
channel,ThreadPoolExecutorandpar(chunks). - Group B, module by module; each module’s page and tests change together.
- Group C, whose ceilings change from
Errto traps in the same step, so each module changes once. - Delivered in issue #664:
//and%with a plain integer dividend and aNonZeroorPositivedivisor, without the zero check. See nonzero arithmetic.
PR #804 merges first; this work changes the same std error types.
Decisions
Decided 2026-11-01, on review of local PR #812.
- Group C ceilings trap under rule 5.3. A literal above a constant
ceiling is a compile error, and the ceiling’s
Errvariant leaves the error type. channelandThreadPoolExecutortakePositive[i64], notNonZero[u64], which removes theiru64deviation from the integer-type convention. Tracked as issue #665, in phase 2.- Widening is explicit:
NonZero.from(p)andNonNegative.from(p). There is no implicit widening. - Division by a domain divisor is delivered (issue #664).
T // NonZero[T],T % NonZero[T],T // Positive[T]andT % Positive[T]returnTwithout a zero check, including augmented assignment. A domain dividend still requires.get();Positivealso rules out the signedMIN // -1overflow case. len()does not convert. It staysi64, and a length becomes a limit throughNonNegative.try_from. This can be reopened if call sites show friction.- Unsigned forms are rejected.
Positive[u64]is a compile error that points toNonZero[u64], andNonNegative[u32]one that points tou32.
Single-file scripts with embedded manifests
Status: Implemented. Delivered under B17:
plenty script, optional embedded manifests, and reusable script builds. The script reference describes current contracts and platform limits; this page records the design rationale. Ordinary single-file compilation and package resolution already exist. This proposal preserves native compilation and explicitmain. Examples involving external dependencies are illustrative rather than standalone lessons.
Motivation and precedent
The separate language editions proposal considers an optional package edition that could also appear in this header. Neither proposal depends on implementing the other.
A small utility should be one file that can be copied, inspected, marked executable, and run. Requiring a directory, manifest and wrapper just to use a dependency discourages that workflow. The same file should remain ordinary Plenty, with static checking and the normal ownership and error rules.
The motivating example is the local ~/recipes/wtnew.rs: a worktree utility
with a cargo +nightly -Zscript shebang and a ---cargo TOML dependency header.
Its Git operations are application logic, not build steps; Plenty does not
need a special command-execution language for this workflow.
Rust’s experimental RFC 3424
motivates optional manifests, reuse of package workflows, independence from
surrounding workspaces, and diagnostics in the original file. Its examples use
doc-comment fences. The Cargo Book’s experimental script documentation
instead shows ---cargo frontmatter, as does the local example. These are
design precedents, not a settled syntax specification for Plenty.
The user-facing shape
The explicit command is plenty script. A Unix executable script starts with:
#!/usr/bin/env -S plenty script
def main() -> Result[(), IoError]:
print("hello from a script")?
Ok(())
chmod +x hello.plenty
./hello.plenty
plenty script hello.plenty
No manifest is needed for the embedded standard library. The shebang is optional
when invoking the command explicitly. env -S passes the subcommand; where it
is unavailable, invoke plenty script directly. The compiler and supported
native linker environment must still be installed.
For package metadata and dependencies, add a header after the optional shebang:
#!/usr/bin/env -S plenty script
---plenty
manifest-version = 1
[package]
name = "worktree_tool"
version = "0.1.0"
description = "A small repository helper"
[dependencies]
util = { path = "../util", version = "^0.1.0" }
---
import util
def main() -> Result[(), IoError]:
print("helper loaded")?
Ok(())
Here ../util is an existing Plenty library package with its own plenty.toml,
not a bundled module. Git dependencies use the existing git plus tag or
rev forms. This introduces neither a registry nor version-only lookup.
Invocation and arguments
plenty script [OPTIONS] [--] FILE [ARGS...]
plenty script --offline tool.plenty --help
plenty script --check tool.plenty
plenty script --compile -o ./tool tool.plenty
Runner options precede FILE. Everything after it goes unchanged to the
program, including --help, --release, and a literal --. The optional --
before FILE permits filenames beginning with -. Thus ./tool.plenty --help
asks the program for help; plenty script --help asks the runner. Preserve
native argument bytes, matching args_bytes.
Default compile-and-run, --check, and --compile -o OUT share resolution and
diagnostics. Check and compile modes reject program arguments and never run
the application. --release, --offline, --locked, and --verbose retain
their existing meanings where applicable. The default profile is debug.
Build status goes to stderr; a fresh cached run is quiet.
Inherit the caller’s working directory, environment and streams. Expose the invoked script path as argument zero rather than the cached executable; retained binaries use normal executable argument zero. Preserve existing exit and signal-status behavior. Do not change directories to resolve imports or interpret the shebang as shell code.
Manifest and source rules
Reuse the strict TOML subset and dependency schema. A script adapter supplies
defaults without weakening ordinary plenty.toml validation. An absent header
means an empty manifest; absent manifest-version means 1. [package] is
optional: default version 0.0.0, and derive a lowercase name from the file
stem, replacing invalid characters with _ and prefixing script_ when needed
for a valid nonreserved identifier. Explicit names follow package validation.
Support description, authors, license and minimum plenty-version; do not
automatically install or switch compilers.
There is exactly one executable target: this file. Reject [workspace],
[lib], [[bin]], unsupported tables and unknown keys. Do not infer neighboring
targets or join an enclosing workspace. Keep plenty FILE and package commands
unchanged; plentyc remains a compiler, without dependency fetching.
Recognize one ---plenty opener and one --- closer before any declarations,
allowing blank lines before the header. Unterminated, duplicate or misplaced
headers are errors. Preserve original filenames, byte spans and line numbers
for both TOML and Plenty diagnostics, including CRLF files. Never silently
ignore a malformed header. Imported modules cannot carry script manifests;
package metadata belongs to the selected root.
Resolve the script’s physical path once. Its parent is the module root and
base for relative dependencies; a symlink launcher refers to resources beside
its target. Permit sibling-module imports, but do not discover dependencies
from the caller’s directory or a nearby plenty.toml. Normal alias-conflict
and cycle rules apply. Sibling modules and path dependencies make a script
convenient locally but no longer self-contained when copied elsewhere.
Storage, locking, and reproducibility
Keep generated files out of the script directory, including when it is read-only. Use private per-user storage with a namespace derived from the resolved script path; equal basenames must not share locks or outputs. This needs explicit storage roots in the package layer, rather than a generated project that loses the original source paths.
Recommend a cached dependency lock, reused across source edits. Dependency
changes reconcile it using normal resolution rules. --locked refuses to
create or change it, before fetching. --offline uses only cached sources.
With no dependencies, an empty graph needs no lockfile and works with both
flags on its first run.
A cached lock is not a portable reproducibility guarantee: deleting it or
copying/moving the script can resolve a moved tag differently. Recommend an
explicit --lockfile PATH for a durable, shareable lock. The option path is
relative to the caller, while paths recorded inside the lock are relative to
the script directory. The same locked/offline rules apply. Full Git revisions
alone do not pin mutable tags in transitive dependencies; path dependencies
remain live trees. Durable locks must not be removed by cache cleanup.
Executable reuse validates script/import contents, dependency graph,
compiler/runtime identity, target, profile and linker inputs. Object/function
caches help but do not themselves remove repeated linking. Plain
plenty FILE already has executable reuse
on supported Linux toolchains; embedded-dependency scripts must extend that
contract. Concurrent launches
need per-entry locking and atomic publication: never execute partial or failed
builds, or replace a running executable in place. Include script outputs in
cache inspection and cleanup.
Boundaries and alternatives
This is native compile-and-run convenience, not an interpreter, REPL, implicit top-level execution, new error policy or sandbox. Dependencies carry the same trust implications as ordinary package builds. No expression mode, stdin source, executable header hooks, library publication, or automatic header rewriting is included. Running a script never edits its source.
A #-commented TOML block avoids new source framing, but older compilers could
silently ignore it. Raw frontmatter makes metadata explicit and older compilers
reject it; the cost is a shared header reader and editor support. Implicitly
enhancing plenty FILE saves a word but mixes the existing compiler shortcut
with dependency resolution and persistent artifacts. Always placing locks
beside scripts improves discoverability but breaks read-only directories.
To grow into a package, move the body to src/main.plenty and the header to
plenty.toml, filling required fields and adjusting relative dependency paths.
No language rewrite is needed. Existing module and entry-point rationale is
recorded in Explicit applications and modules.
Original implementation assessment and acceptance
This is a moderate toolchain change, not a borrow-checker or code-generation
project. A shebang-only prototype is small: at daf964d1, a real
#!/usr/bin/env plenty program compiled and printed through plenty FILE;
adding --help after its filename was rejected. The full proposal needs
manifest adaptation, argument forwarding, isolated resolution/storage, source
mapping and executable reuse, rather than just a CLI wrapper.
Likely boundaries are src/main.rs, src/compiler_cli.rs, src/project.rs,
src/project/{manifest,toml,git,fingerprint}.rs, and the frontend source
loader/LSP. Reuse package resolution behind explicit source/storage inputs.
An unresolved header in plentyc should direct users to plenty script, not
fetch dependencies.
Acceptance requires real executable-shebang tests, argument fidelity (spaces, empty and non-UTF-8 arguments), status propagation, original-source diagnostics, stdlib/path/Git dependencies, offline/locked failures, read-only directories, symlink launchers, surrounding-workspace independence, concurrent launches, and invalidation after source/dependency/compiler changes. Verify that warm runs skip compilation and linking, and failed builds never run an old binary.
The implementation settles storage under the existing Plenty cache, uses the
standalone header delimiters shown here, and includes durable --lockfile support. A single-file test command needs
its own entry-point/discovery design: existing package tests reject a test
file mixing main and test_ functions. It is not implicitly solved here.
Remaining toolchain/platform scope stays in B17/B18.
Library-owned operations and backend hooks
Status: Investigation complete. Issue #145 recommends an API design using existing composition, with capability parameters where needed; new protocol visibility is deferred. No new language feature or stdlib conversion is approved. The current contracts remain protocols, visibility, and classes. Any future access extension belongs to B29.
The later protocol-body proposal records agreed default/final methods and member-specific private-hook grants. That design is not implemented; the wrappers and capabilities here remain the existing-language solution. Its short programs contrast the proposed programming model with this assessment’s composition approach.
Introduction
A library sometimes needs to let other packages customize one step of an operation while keeping the surrounding algorithm under the library’s control. That algorithm may validate inputs, check state, acquire resources, and record success or failure. The customizable step is an implementation hook: it does the backend-specific work, but is not intended to be an independent application entry point.
For example, a library author provides Transport.send(message), which
checks the message-size limit and connection state, asks a backend to send the
message, and updates accounting after success. A backend author supplies
the TCP-specific send_impl(message) hook, or an in-memory version for tests.
An application author selects a backend and calls transport.send(message).
These may be three different people working in three different packages.
Two mistakes should be prevented through that library interface. Application
code should not accidentally call send_impl directly and skip validation or
accounting. A backend’s customization should not replace the surrounding
send algorithm and omit those checks either. Supplying the hook and calling
the hook are therefore different permissions. A similar situation arises when
Store.save(record) must validate a record and manage a transaction around a
backend-specific write: application code should use save, and each backend
should not have to reproduce the transaction algorithm.
C++ addresses this with the non-virtual interface (NVI) pattern: a public non-virtual method owns the algorithm and calls a private virtual hook. A backend class can override that hook, while application code using the base interface cannot call it directly. The public algorithm is not overridden through that interface. This is about separating customization from access, rather than sealing an interface against external implementations.
Rust traits do not directly offer that combination: individual trait methods cannot have independent private/public visibility, and a default method body can be replaced by an implementation. Rust can express the intent using an owning wrapper with a private backend field; a capability argument can further restrict calls to an otherwise public hook. The precedents explain and test these workarounds. The question for Plenty is whether existing composition provides clear enough guarantees, or whether protocols need a new way to permit external hook implementations without exposing ordinary calls to those hooks.
The aim is to prevent accidental bypass through the library abstraction, not to prove that a backend performs its work honestly. A backend can still expose its own separate API, and validation, partial failure and transaction recovery need explicit contracts regardless of method visibility.
Decision and roles
Use an ordinary owning wrapper for the public algorithm. Keep its backend field private. Use a public backend protocol when external implementations are required. If normal application code must also be unable to invoke a hook on a separately obtained backend, require a library-issued capability argument. Defer new protocol access or implementation syntax: the tested examples do not establish a missing guarantee that warrants that complexity.
This is the intent of C++’s non-virtual interface (NVI) pattern, not sealing. Precedents and probes explain the distinction and the Rust workarounds.
| Role | Transport responsibility | Permitted customization |
|---|---|---|
| Library author | Owns Transport[B].send, validation, connection state and successful-send accounting | Changes the algorithm and hook contract |
| Backend author | Supplies Memory.send_impl or TCP-specific work, possibly in another package | Implements the hook; does not replace Transport[B].send |
| Application author | Constructs the selected backend/wrapper and calls transport.send(message) | Selects/configures a backend; cannot access the wrapper’s private field |
One person may fill all three roles; their modules still have different access. The guarantees concern use through this abstraction. A backend author can expose another API on their concrete type or intentionally violate the hook contract. Neither NVI nor a token proves honest transmission, storage, or accounting.
Existing-language baseline
The complete three-module example
executes with today’s compiler. The library author moves an externally defined
public backend into Transport[B], keeps _backend, _limit, _open and
_sent private, and implements one public inherent send method. A private
bounded helper checks the backend protocol; class declarations do not yet
support protocol bounds themselves.
send checks the open state and UTF-8 byte limit before invoking the backend.
Only Ok increments the successful-message count. Err leaves the count
unchanged and closes the wrapper: the backend may already have emitted bytes,
and a message boundary is no longer assumed trustworthy. The toy error does
not report a resumable offset. A production transport must either specify an
all-or-nothing enqueue contract or preserve partial progress and recovery state,
as std.io does. Hidden hooks do not resolve this choice.
There is no inheritance or protocol default body here. send_impl is absent
from Transport[B], and the application cannot reach _backend. A backend
method also named send cannot replace the wrapper’s method. No fixed or
final annotation is needed. Public functions accepting a structural Sendable
instead of the nominal wrapper would allow different algorithms; choose that
contract only when such substitution is intended.
The remaining difference from private virtual hooks is precise: an application
can construct a separate Memory and call its public send_impl. It cannot
use that value to access the stored backend owner. Separate handles can still
alias the same socket, database or other external resource; field privacy is
not proof of exclusive access to that resource. The wrapper’s contract must
account for permitted sharing. A library that only promises wrapper-local
invariants can stop here.
Capability variant
The example also shows a public Permit type with a private constructor,
passed as &Permit to the public hook. The library creates it only after
validation. An external backend author can name Permit and implement the
protocol; the application author cannot manufacture it through either constructor
spelling. This separates the ability to supply the operation from possession
of the authority needed to invoke it. The hook remains publicly nameable and
callable by code that already has a permit; it is not a private method.
A borrowed argument avoids transferring ownership of the permit. Current
Plenty disallows stored reference fields, and the hook’s result contains no
returned borrow. However, ordinary classes support explicit copy, including
their private fields. The tested variant adds a no-op private __del__ so
copy(permit) is rejected under today’s custom-destruction rule. That is
workaround boilerplate, not a proposed long-term spelling for noncopyable types.
The compiler’s borrow rules also forbid escaping a captured borrowed permit;
there is no permission to store a reference merely because it is in a closure.
A permit gates possession during the operation. It does not bind authority to one receiver or message, enforce exactly one nested call, or prevent deliberate delegation to another backend while the borrow is live. The library must not publish a permit factory/accessor, accept an application callback with a permit, or return a permit in an error if it wants this boundary. Token-bearing hooks also become part of the public implementor contract even when omitted from application documentation.
For transaction-sensitive hooks, use a transaction-specific authority rather than treating this generic send permit as proof of an active transaction. Represent the connection/transaction identity and allowed operations explicitly; a dummy token proves only possession.
Storage and transaction ownership
For Store.save(record), the library author owns validation and the transaction
algorithm, a SQLite backend author supplies storage-specific operations, and
the application author calls save. An in-memory backend follows the same
contract. A plausible algorithm is:
validate record
acquire transaction (failure: no hook, no commit)
call storage hook with transaction context
on hook success: explicitly commit
on hook failure: explicitly roll back and retain both errors if rollback fails
on commit failure: follow the documented retry/rollback policy
This is an algorithm sketch, not new Plenty syntax or a working generic SQLite
adapter. Beginning, committing and rolling back are themselves backend-specific;
a single hidden write_impl does not make them backend-independent. A concrete
transaction owner or backend protocol must supply those steps too.
The real SQLite transaction remains open after a busy commit so it can be retried, and rolls back when dropped unfinished. Choose explicitly whether a library wrapper retains that owner for retry or rolls back before returning an error. Never report commit success merely because the write hook succeeded. A borrowed transaction context can restrict writing to the transaction lifetime, but must also carry meaningful backend authority.
Resource scopes supply
cleanup on ordinary exits; __exit__ returns unit and has no success/failure
argument. It cannot replace explicit fallible commit/rollback handling or infer
that a body succeeded. Traps do not unwind it. Coarse distinct transaction/state
owners may enforce sequencing better than method hiding; see
typestate. Ownership transfers, retryable
failure and extraction of affine fields still need an API design.
What the stdlib actually needs
| Current source | Algorithm and extension boundary | Assessment |
|---|---|---|
stdlib/std/io.plenty: Reader, Writer, read_exact, write_all | Public one-attempt operations; free helpers add retry, bounds/progress validation and partial-failure policy | Partial operations are useful application APIs. Do not make every stream operation a hidden hook. |
stdlib/std/http.plenty: BodyReader.read_body, private step | Wrapper checks ranges, decodes framing and poisons its state on failure; external streams supply Reader.read | Composition already protects this algorithm. step is a library-owned shared helper, not an external hook. |
stdlib/std/io.plenty: BufReader.inner | Public underlying owner intentionally permits closing/replacement, with a warning about bypassing buffered bytes | Choosing public escape access is an API decision. An access feature cannot repair a deliberately exposed field. |
stdlib/std/db/sqlite.plenty: Transaction | Concrete transaction owner with commit/rollback lifecycle | Strong motivation for explicit transaction authority and failure semantics, not evidence for private protocol hooks. |
These sources were inspected at 5346f85. No stdlib conversion follows from
this investigation.
Alternative: explicit protocol access grants
If later evidence requires lexically hidden external hooks, the narrowest candidate would keep the ordinary wrapper and add an explicit mapping from a backend’s private method to a public protocol requirement. Only the backend type’s defining module could issue that grant, for a named protocol identity and concrete arguments; the library could not grant itself access by declaring a matching signature. This is a semantic candidate, not proposed syntax.
A coherent version must confine invocation to the protocol’s owning module, not every generic function declaring the bound. Otherwise an application could write its own bounded trampoline and regain access. The access is to the listed method only, never other private members or the whole backend module; aliases, imports and specialization must not broaden it. Ordinary accessible methods would keep structural matching. A private requirement would need the explicit grant, and the same method call in an unauthorized module must fail even if conformance is known. Hidden type names in public signatures remain prohibited; do not silently admit private concrete backend types as a side effect.
This creates two conformance/access paths and provenance that specialization
currently erases. It also couples a backend to a library protocol identity and
requires coherent duplicate/missing/signature-mismatched grant diagnostics.
Inference must check authority before using a private signature as evidence;
it must never search private methods or guess an implementation. A useful error
must identify the requiring call, protocol, candidate method and absent grant,
without simply suggesting pub where hidden access was intended.
No inheritance, runtime protocol object, default method body or override system is needed. Their addition would increase scope without fixing access. Generic specialization can preserve today’s static dispatch, borrowing and allocation costs, but new access provenance would affect checking, caching and diagnostics. That cost is not justified by the current examples.
Tradeoffs and remaining evidence
The plain wrapper adds one owner and a small bounded delegation helper, keeps normal structural inference and requires no runtime allocation. A permit adds one public type and hook parameter, private construction and the current copy barrier; it makes capability errors visible at constructor/arity checks. Both retain ordinary exclusive backend borrows and Result-based cleanup.
Protocol grants could remove token plumbing and keep hook names off application call surfaces, but introduce a new access relation and implementor obligations. Neither approach proves semantic hook correctness. Choose a token when possession or transaction context is useful in its own right; use plain composition when the wrapper owns the only backend application code needs.
The decision against a new feature can be revisited with an external-backend API whose real invariant fails under the baseline, or measured token/delegation friction that exceeds the cost of grants. Identity-scoped transaction capabilities, noncopyable declarations, diagnostic ergonomics and compile-time measurements remain unanswered design questions, not scheduled implementation work.
Transport wrapper and capability examples
Status: Executable design example; not a stdlib API. These experiments use existing Plenty, supporting the assessment. They do not add stdlib types or change the language.
Baseline: three authors, three modules
Save the following files beside each other and run plentyc main.plenty.
The library author owns library.plenty; the backend author owns
backend.plenty; the application author owns main.plenty. Memory stands in
for an external backend with success and failure outcomes. A TCP backend would
need the partial-progress/error contract described in the assessment.
library.plenty:
pub enum TransportError:
Closed
TooLarge
Backend
pub protocol Backend:
def send_impl(self: &mut Backend, message: &str) -> Result[(), TransportError]:
pass
pub class Transport[B]:
_backend: B
_limit: i64
_open: bool
_sent: i64
pub def send(self: &mut Transport[B], message: &str) -> Result[(), TransportError]:
if not self._open:
return Err(TransportError.Closed)
if message.byte_len() > self._limit:
return Err(TransportError.TooLarge)
match deliver(&mut self._backend, message):
case Ok(_):
pass
case Err(cause):
self._open = False
return Err(cause)
self._sent += 1
Ok(())
pub def sent(self) -> i64:
self._sent
pub def close(self: &mut Transport[B]) -> ():
self._open = False
def deliver[B: Backend](backend: &mut B, message: &str) -> Result[(), TransportError]:
backend.send_impl(message)
pub def transport[B: Backend](backend: B, limit: i64) -> Transport[B]:
Transport(backend, limit, True, 0)
backend.plenty:
from library import TransportError
pub class Memory:
pub def send_impl(self: &mut Memory, message: &str) -> Result[(), TransportError]:
if message == "fail":
return Err(TransportError.Backend)
Ok(())
main.plenty:
from library import transport
from backend import Memory
def main() -> Result[(), Failure]:
mut t = transport(Memory(), 4)
print(str.repr(t.send("ok"))?)?
print(str.repr(t.send("large"))?)?
print(str.repr(t.send("fail"))?)?
print(t.sent())?
t.close()
print(str.repr(t.send("ok"))?)?
Ok(())
The calls produce success, TooLarge, Backend, count 1, and Closed.
The application deliberately uses str.repr to show Result wrappers. The
rejected oversized message never reaches the hook; a backend error poisons the
transport and never increments the count. close is also an explicit operation.
transport[B: Backend] validates every requirement when constructed. The
private deliver[B: Backend] helper supplies the bound for delegation from
an otherwise unconstrained generic class method. This is existing Plenty;
class Transport[B: Backend] is not available today.
An application cannot call t.send_impl (no such method) or t._backend
(private field). It can construct a separate Memory and call its public
send_impl directly. That separate value is not the owner moved into t, but
separate handles could still alias the same external resource. Private wrapper
fields do not establish exclusive access to the underlying socket or database.
Capability variant: authorize calls as well as implementations
Keep all three files above. In library.plenty, add this type before the
protocol:
pub class Permit:
_issued: bool
def __del__(self: &mut Permit) -> ():
pass
The generated field constructor is private because _issued is private.
The destructor is deliberately empty: custom destruction disables structural
copy. It is an awkward copy barrier, not cleanup logic or a proposed language
idiom. The flag’s value is irrelevant; nominal type identity and private
construction carry the authority.
Replace the protocol and private helper on the same page with:
pub protocol Backend:
def send_impl(self: &mut Backend, message: &str, permit: &Permit) -> Result[(), TransportError]:
pass
def deliver[B: Backend](backend: &mut B, message: &str) -> Result[(), TransportError]:
permit = Permit(True)
backend.send_impl(message, &permit)
In backend.plenty, import Permit alongside TransportError and add
permit: &Permit after message: &str in Memory.send_impl. Its body can stay
unchanged. The application file stays unchanged, and produces the same outcomes.
All companion code for these replacements is the baseline on this page.
The external backend author can name the public type without constructing it.
Ordinary application calls without a permit fail arity checking; both
Permit(True) and Permit.new(True) fail private-constructor checking.
Attempting saved = copy(permit).unwrap() in the backend fails with
this resource cannot be copied. Removing the destructor makes that explicit
copy legal, despite the private flag. This matters if the backend should be
unable to retain an issued capability accidentally.
A hook can still invoke another hook with the borrowed permit during its call. The example does not prove exactly-once invocation, receiver/message identity, transaction sequencing, or correct external effects. Application code that is explicitly handed a permit also gains call authority. The library must choose which functions receive it.
Probe results
The compiler was freshly built from 5346f85 with
cargo build --locked --offline --bin plentyc in the issue worktree, using a
separate target directory. Both variants compiled and ran natively. These were
focused manual probes, not additions to the tutorial’s executed-fence suite.
Proposal fences intentionally use text.
| Change to the example | Observed result |
|---|---|
Call t.send_impl("ok") | Rejected: unknown method on Transport |
Access t._backend | Rejected: private member |
Make Memory.send_impl private | Rejected at construction requiring the protocol; a matching name does not grant access |
Make Memory private | Rejected importing the class; a public method does not export its class |
| Call a separately constructed baseline backend directly | Accepted |
| Omit capability argument on a separate backend | Rejected: hook expects two arguments |
| Construct capability through either constructor spelling | Rejected: private Permit.__new__ |
| Explicitly copy borrowed capability | Rejected with custom destruction; accepted without it |
The source also confirms lexical access in
src/frontend/protocols.rs (check calls check_member_at) and in protocol
inference in src/frontend/generics.rs. Existing tests/test_modules.rs covers
private classes/methods, transitive generic calls and definition-module access.
The generic caller does not lend its access to the library’s helper.
C++ and Rust precedents
Status: investigation evidence. Supports the library-owned operations assessment, not a change to Plenty’s protocol semantics. Sources and probes checked 2026-10-17.
C++: override and call access are separate
The library author can write:
class Transport {
public:
void send() { validate(); send_impl(); record_success(); }
virtual ~Transport() = default;
private:
virtual void send_impl() = 0;
void validate();
void record_success();
};
The backend author overrides send_impl privately in a derived class. An
application author calls send through Transport&. The wrapper statically
selects the library’s algorithm, which dynamically invokes the backend hook.
Access does not participate in deciding whether one virtual function overrides
another; call accessibility is checked separately using the static expression
type. C++ working draft: virtual functions,
access to virtual functions.
A derived class can declare another send that hides the base name, or expose
another public operation. Calling through the derived type can therefore select
another API; NVI does not prevent that. The guarantee concerns calls through the
base abstraction. In the focused probe, counters showed that Transport&.send
used the wrapper and the private override, while Memory.send selected the
derived hiding method. Direct calls to the private hook through either type were
rejected. The probe compiled and ran with GCC 16.2.1, C++20 and
-Wall -Wextra -Werror.
The issue’s Standard C++ FAQ link returned HTTP 403 during this investigation; the accessible working draft and compiled probe supplied the evidence instead. Exceptions/partial external effects still need a library-specific contract; this minimal example omits them.
Rust: composition supplies the fixed operation
The Rust Reference on traits
specifies replaceable defaults and rejects visibility annotations on individual
trait items. A public trait with default send and required send_impl therefore
does not express either hidden hook access or a fixed algorithm. This is a
constraint on that spelling, not on Rust’s ability to express the intent.
A library author can instead write this public surface, with private fields:
#![allow(unused)]
fn main() {
pub struct Permit { _private: () }
pub trait Backend {
fn send_impl(&mut self, message: &str, permit: &Permit) -> Result<(), ()>;
}
pub struct Transport<B> { backend: B, sent: usize }
impl<B: Backend> Transport<B> {
pub fn send(&mut self, message: &str) -> Result<(), ()> {
if message.len() > 4 { return Err(()); }
self.backend.send_impl(message, &Permit { _private: () })?;
self.sent += 1;
Ok(())
}
}
}
A public constructor implemented in the library moves B into the wrapper.
An external backend author implements Backend and names Permit, without
creating it. An application author obtains Transport<B> and calls its inherent
send. This excerpt isolates the mechanism; the complete Plenty example also
specifies connection state and poisoning after backend failure.
Without the token, a separate backend value’s trait hook is callable. With it,
the hook still has public visibility, but callers need a value they cannot
construct. Permit exposes no constructor, Default, Copy or Clone.
The borrowed permit cannot be retained as an owned token by safe code; storing
its reference requires a compatible lifetime that a callback-local permit does
not provide. This does not restrict external trait implementations. Rust’s
privacy rules
make public trait items accessible but preserve private struct fields.
This adds a wrapper, inherent implementation/bound, delegation and one public token type/parameter. Separate crates need the extension trait and its argument types exposed as public contracts. A private implementation trait is simpler when all backends belong to the library, but removes external extensibility. Documentation hiding alone supplies no call restriction.
Sealing does not hide callable hooks
The Rust API Guidelines
describe sealing to restrict downstream implementations. That is a different
permission from calling a hook. In a separate-crate probe, the library put
pub trait Hook in a private module, made public Api inherit it, and implemented
both for Concrete. An application still compiled this generic call:
#![allow(unused)]
fn main() {
fn bypass<T: Api>(value: &T) -> i32 { value.hook() }
}
The application did not need to name the hidden supertrait. Bounds provide access to supertrait associated items, as the Rust Reference explains. Thus neither a private module path nor sealing is proof of hidden call access.
Rust probe outcomes
Separate library, backend and application crates compiled with rustc 1.99.0,
edition 2021 and -D warnings; the application verified one successful send,
size rejection, backend error and a count of one. No unsafe code was used.
| Focused probe | Result |
|---|---|
| External backend implements token-bearing trait | Accepted |
| Application constructs private-field token | E0451 |
| Application calls hook without token | E0061 |
Backend tries to clone &Permit into owned Permit | E0308: reference cloned; token does not implement Clone |
| Visibility annotation on trait method | E0449 |
Implementor replaces a default send | Accepted; executed replacement returned the alternative value |
| Application’s generic bound calls private-module supertrait hook | Accepted and ran |
These small probes validate the language mechanisms, not transport reliability or database transaction semantics. They support using composition as the baseline and capability possession as a separate design choice.
Fluent iterator pipelines
Status: Implemented. All four agreed stages shipped.
Stages 1 and 2 cover the serial core, pair callbacks, dictionary collection,
generator fluent methods, explicit owned .generator() conversion, chain,
flat_map, closure-literal splicing and comprehensions lowered through the
plan. std.iter was removed rather than deprecated. Stage 3 covers the sum,
min and max terminals, resumable adapters, next(p), and lending through
&mut Iter[T], including pipelines of reference items. Bounded native
substitution of scalar leaf callbacks also covers named functions and immutable
scalar captures; it addresses the original performance gap (issue #399).
Stage 4 covers .par(&pool) with the terminals below, plus sum, min,
max and chunks=N.
The pipeline reference
lists the supported adapters, terminals, ownership rules and current limits.
Current iteration is defined by
iteration and generators,
closures
and ordered parallel mapping.
Related backlog entries are B12 (iterator/view protocols, borrowed adapters),
B49 (collections; its std.iter part was removed), B22 (broader parallel mapping) and B24 (automatic
parallelization). B75 tracks remaining extensions, not the delivered stages.
This page preserves the original design; examples may differ from the current reference.
Summary
Add iter(source), which starts a pipeline: a compiler-known, lazily
evaluated chain of adapter methods (map, filter, take, …) ending in a
loop, a comprehension, or a terminal method (fold, count, collect, …).
total = iter(&payments)
.filter(def(p: &Payment) -> bool: p.name == "bob")
.map(def(p: &Payment) -> i64: p.cents)
.fold(0, def(acc: i64, cents: i64) -> i64: acc + cents)
A pipeline is not a generator and not a library object. The compiler lowers the
whole chain at its consumer into one loop, the same loop a hand-written for
or comprehension produces, with each callback called directly. Serial pipeline
machinery allocates only when a terminal builds a collection, returning Result.
A .par(&pool) step
moves the rest of the pipeline onto an explicit thread pool, reusing the
existing ordered map and fixed-tree reduce contracts.
The surface follows excitertools: one
wrapper, chained methods, terminal “sinks”. The cost model follows Rust’s
Iterator, but fusion is done by the compiler rather than by monomorphized
adapter structs and an optimizer’s inliner, which Plenty does not have.
Starting point before B75
The original design started with iteration but no fluent interface or fused composition.
There are no builtin top-level map or filter functions; the reusable
adapters are the importable std.iter module.
| Facility | Form | Cost |
|---|---|---|
for over list, set, dict, str, bytes, range | Statement | Direct range source is a counted loop; collections use the loop plan in op.rs |
enumerate, zip, reversed | Loop/comprehension sources only | Fused into the loop plan; no iterator object. Rejected elsewhere |
any, all | Builtin boolean reductions | Short-circuit, no allocation |
dictionary.items() | Loop intrinsic, two bindings | No item tuples |
| Comprehensions | [e for x in s if c] | One fused loop; output growth returns Result[_, AllocError] |
| Generators, generator expressions | Generator[T], inline frame | No heap allocation; each advance is a runtime call (below) |
std.iter | map, filter, take, fold, try_fold | Generic generator functions; one frame per adapter |
std.math.sum_iter, prod_iter | Borrowed-iterable folds | Plain loop |
pool.map, pool.map_result, pool.reduce_tree | Explicit parallel operations | Owned list or range input; eager, ordered |
The original std.iter usage was nested calls, generator sources only:
# current
from std.iter import map, filter, fold
total = fold(map(filter(numbers(10), even), square), 0, add)
Gaps
- No method chaining.
numbers(3).map(sq)reports “unsupported methodmap”. Built-in types have no user-extensible methods, and protocols deliberately add no methods on import. - Generator sources only. A list must be moved into a generator expression,
(n for n in values). There is no adapter over a borrowed collection, since a frame cannot hold a reference (B12/B49 wait on B11 stored references). - No borrowed captures. Callbacks must own their captures;
def [&mut seen]is rejected with “pass a borrowing closure by reference”, because the frame that stores the callback can outlive the loan. - Per-item overhead. Every generator advance, including each adapter step,
goes through
collection_call(24)into the runtime dispatcher, thenplenty_generator_resume(reentrancy guard), then an indirect call to the frame’s resume function, which dispatches its state through a linear compare chain and keeps every local in frame memory. Nothing is inlined: the compiler has no function inliner, and Cranelift sees each resume function separately. - No storage. Frames cannot enter class fields, tuples or collections (“generators cannot be stored in generic data”), so a Rust-style adapter struct holding its source cannot be written in Plenty today.
- Missing operations. No
chain,skip,take_while,flat_map,count,find,collect, or parallel chaining.
Measured cost
Release compiler at 6153b46f, x86-64 Linux, summing (n * n) % 1000 over the
even numbers below 100,000,000:
| Program | Time | vs. loop |
|---|---|---|
for n in range(N): if n % 2 == 0: total += ... | 0.31 s | 1x |
Named generator with the filter inside its body, consumed by for | 0.62 s | 2x |
Named generator yielding every n, filtered in the loop | 0.91 s | 3x |
fold(map(filter(numbers(N), even), sq), 0, add) with std.iter | 2.74 s | 9x |
for v in ((n * n) % 1000 for n in range(N) if n % 2 == 0) | 5.15 s | 17x |
The generator expression is the slowest because its outer range is stored in
the frame and then iterated through the runtime, like any stored range
(issue #371). A stored range alone, r = range(N) then
for n in r, takes 4.71 s. Issue #371 has since been fixed: a stored range now
runs as fast as a direct range, and the generator expression drops to about
0.86 s, the cost of one frame resume per element. These numbers set the target: a pipeline should be
indistinguishable from the first row.
Design
Pipelines
iter(source) accepts every current for source and follows the same
ownership rules:
| Source | Item |
|---|---|
iter(values) (owned list, set, dict) | Owned element / key; consumes the collection |
iter(&values) | &T; borrows for the pipeline’s lifetime |
iter(&mut values) | &mut T (lists and dict values only, as today) |
iter(range(...)) | Integer of the range’s type, counted loop |
iter(text), iter(&buffer) | str scalar, u8 |
iter(generator) | Yielded value; consumes the frame |
dictionary.items() | Key/value reference pair |
enumerate(...), zip(...) and reversed(...) become pipeline starts with
the same meaning as iter(...).enumerate() and so on. They keep their current
loop behavior, and gain chaining: enumerate(&rows).filter(...). This mirrors
excitertools, whose module-level range, zip and friends also return Iter.
A pipeline’s type is Iter[T] for item type T, but like a generator it keeps
a concrete identity (its source, adapters and closures), so two pipelines with
equal T do not unify. Iter[T] describes the item contract only.
Where a pipeline may appear
One hard rule: an unfinished pipeline never leaves the scope that created it. It may be:
- a
forsource or comprehension source; - the receiver of a terminal method;
- bound to a local and used later; it is affine, and consuming it ends it;
- lent to a function as
&mut Iter[T](next section); the caller keeps ownership and can continue or collect it afterwards; - turned into an owned generator with explicit
.generator()when items must cross a function boundary, subject to the ownership restrictions below. That conversion is explicit and costs what generators cost today.
It may not be returned, passed by value, stored, captured, or held across
yield. Because a pipeline never outlives its scope, it can hold borrowed
sources and closures with borrowed captures. This removes gaps 2 and 3
without waiting for stored references (B11).
Generator conversion is an explicit lowering boundary, not permission to
store an Iter value. The compiler consumes the plan and transfers its owned
source, callback environments and progress into a generator frame. Conversion
is allowed only when all of that state meets the existing
generator capture, storage and yield rules.
Borrowed sources, borrowed callback captures and sub-pipelines lending another
pipeline are rejected, even if their yielded items are owned. No loan may cross
yield, and no implicit copying makes a borrowed pipeline eligible. The
resulting generator may leave the scope; the unfinished pipeline itself may not.
Generators themselves expose the same fluent methods: source.map(f) builds
a pipeline equivalent to iter(source).map(f), with no frame per adapter.
.generator() creates one frame around the complete fused plan. This spelling
replaces the originally proposed generator-expression bridge. There is no
implicit conversion from an expected Generator[T] type.
Lending a pipeline to a function
Implemented in stage 3; the reference states the final rules and limits. This section keeps the design rationale.
A function can take a pipeline by exclusive reference, consume some of it, and return. The caller then continues from where the callee stopped:
def skip_header(lines: &mut Iter[str]) -> Result[(), CsvError]:
match next(lines):
case Some(header):
check_header(&header)
case Nothing:
Err(CsvError.Empty)
def main() -> Result[(), CsvError | AllocError | IoError]:
mut rows = iter(read_lines(path)?).filter(def(l: &str) -> bool: l != "")
skip_header(&mut rows)?
records: list[Record] = rows.map(parse_row).try_collect()?
...
Generators already work this way today: &mut Generator[i64] parameters
specialize to the caller’s actual frame, and the caller can keep consuming
after the call (verified with the current compiler). A pipeline reuses that
model.
What the callee may do through &mut Iter[T]:
next(p) -> Option[T]takes one item;for x in p:consumes items until the loop ends; afterbreak, the rest stays in the pipeline;- a terminal (
p.count(), orp.collect()with an expected type) consumes everything that remains; - an adapter (
p.take(3),p.filter(...)) makes a sub-pipeline that borrowspand must be consumed inside the callee. Its own closures may borrow the callee’s locals, because it ends in the callee.
What it may not do: move, store, or return the pipeline, or add steps that persist after it returns. A reference parameter cannot be stored (B11 is not implemented), so the rule needs no new check: the pipeline stays owned by its creating scope by construction. Loans held by the pipeline’s own closures stay live through the call, as loans held by any lent value do.
Type. The parameter type is &mut Iter[T]. Like &mut Generator[T], it
names the item contract and specializes to the concrete pipeline at each call;
it is not a generic constraint. &Iter[T] (nothing can be advanced through a
shared reference) and by-value Iter[T] parameters and returns are rejected.
Iterable[T] was considered and rejected as the name: in Python and Rust
“iterable” means something you can start iterating, such as a list, repeatedly.
A pipeline is single-pass state, an iterator, and Iter matches both iter()
and excitertools’ Iter class. Iterable[T] stays free for a possible later
constraint meaning “any source iter() accepts”.
Passing by value and returning it (the alternative) would let a helper add
steps: def evens(p: Iter[i64]) -> Iter[i64]: p.filter(even). It is feasible,
since generator factories already infer a concrete returned identity, but it
needs a return-derivation rule (a function may only return a pipeline derived
from a pipeline parameter, never one it created), a rule for loans in the
steps it adds, and it weakens the hard rule to “may leave if it comes back”.
It is not part of this design; reusable steps are written as functions that
return closures, e.g. .filter(longer_than(3)).
Complexity, stated honestly.
- Resumable state. A pipeline consumed in one place keeps its progress in
registers inside one fused loop. A lent pipeline can be consumed in several
places, so its progress lives in slots: the source cursor, counters,
chain’s phase,flat_map’s inner source, andzip’s pending left item. Each fused loop loads that state on entry and stores it on exit. The compiler sees every lend in the creating function, so pipelines that are never lent skip the store-back. The extra cost is per loop entry and exit, not per item;next(p)expands one step inline, still with no runtime dispatch. - Monomorphization. The callee is instantiated once per distinct pipeline
type passed to it, as with generators today. Pipeline types include closure
identities, so two call sites with different closure literals produce two
instances. That is code size and compile time, bounded by the existing
specialization limits, and must be measured in
compile_bench. - Diagnostics. Errors inside an instance name the pipeline by its creation site, as closures are named today.
- Every adapter must be resumable. Each new adapter specifies its slot state; this is a cost on every future addition, and the reason lending is staged after the serial core.
Implementation decisions. Lending follows option A: the lender passes its
state and the borrower rewrites the plan’s operations to use it, writing state
back only for lent pipelines, with next(p) inline. Where a choice affected
borrow checking, the conservative option was taken:
- State is passed per slot, not as one record. Each owned state slot is a
hidden
&mutargument after the declared ones; each reference held as state is a hidden by-value argument. This is equivalent to one record pointer without a new aggregate layout, and it reuses the existing frame specialization, call verification and borrow tracking for reference arguments. The borrower copies owned state into locals at entry and writes it back on every exit, so the lender never observes a half-updated slot. - Reference state is read-only. A reference the pipeline holds (a borrowed source, a closure literal’s borrowed capture) is passed by value and may not be reassigned in the borrower; the compiler reports any write. The callee cannot extend or shorten the loan behind it.
- Caller loans stay with the caller. The plan’s loans are not reconstructed
in the borrower, which has no names for them. The lender keeps each loan live
across the call and uses it after the call returns, so a conflicting
mutation before the pipeline ends is the same borrow error it is without
lending. In the borrower,
pitself is borrowed exclusively by every loop,next, terminal and sub-pipeline over it, so overlapping uses ofpthere are ordinary borrow conflicts. - Reference items borrow for the call only. A pipeline whose items are
references, such as
iter(&names).filter_value("ada"), can be lent. In the borrower each reference item has a loan rooted at one of the plan’s hidden parameters, which the borrower only reads and cannot name. An item is therefore usable anywhere in the call, including by local adapters, closures and further lends, but cannot leave it: returning an item, or a reference derived from one, is rejected with “an item of a lent pipeline cannot be returned”. The other routes out already reject references:Optionand tuple payloads, class fields, generators, and escaping borrowing closures. Rooting items at the declared&mut Iterparameter instead would let a borrower return an item tied only to that parameter, which can outlive the pipeline’s loan of its source once the lender consumes the pipeline. - Limits. A lent pipeline may have at most 64 state slots, which keeps the borrower within the 256-slot frame limit. A pipeline is rejected when its state holds a reference inside another value (a bound closure with borrowed captures), or when it holds a reference into its own owned state: either would let the borrower’s copy alias the lender’s slot.
One borrower instance is compiled per lent plan, shared by recursive and
repeated lends. In a probe lending three different pipelines to a one-line
skip_one and a recursive drain, the compiler emitted six borrower instances
of 360 to 910 bytes each. The compile_bench workload has no pipelines; its
AOT median was unchanged (223.7 ms against 225.9 ms and 223.0 ms against
223.9 ms on main, alternating runs).
Adapters (core set)
Each adapter is lazy and consumes the pipeline it is called on.
| Adapter | Callback | Item out |
|---|---|---|
.map(f) | (Item) -> U | U |
.filter(keep) | (&Elem) -> bool | Item |
.enumerate(start=0) | none | (i64, Item) |
.zip(other) | none; other is any pipeline source | (Item, Other) |
.take(n) | none | Item, at most n |
.skip(n) | none | Item after the first n |
.take_while(keep) | (&Elem) -> bool | Item until the first false |
.filter_value(v=True) | none; v: Elem or, for borrowed items, &Elem | Item equal to v |
.chain(other) | none; same item type | Item from self, then other |
.flat_map(f) | (Item) -> source | Items of each returned source |
Rules that keep callbacks predictable:
- Predicates always receive a shared reference to the element. For an owned
item
Tthat is&T; for a borrowed item&Tit is the same&T, never&&T; for&mut Tit is&T.mapreceives the item itself. - Pair items are passed as two arguments. See callbacks over pairs.
zipstops at the shorter input and tests the left side first, as today.chainrequires both sides to have exactly the same item type.take_whiledrops the first rejected item; there is no peeking.flat_map’s callback may return any owned iterable: a list, set, dictionary,str,bytes, range, or generator, with the items aforloop over it would yield. There are no per-type exceptions. Returning a nestedIteris rejected by the same no-return rule as any other pipeline. A callback may use an eligible explicit generator conversion, with the frame cost and ownership restrictions described above.
Later additions, each needing a use case: step_by, map_while, scan,
inspect, chunks(n) and windows(n) over borrowed lists (as borrowed
sub-range views, which depend on B11/B39), dedup, rev for reversible
pipelines, zip of more than two sources, peekable, group_by, and
cycle. excitertools’ larger surface (map_reduce, send, I/O sources such as
Iter.open) stays out of the language; such sources belong in std modules
returning generators.
Callbacks over pairs
Some pipelines produce two values per item:
for name, score in scores.items(): # name: &str, score: &i64
for i, row in enumerate(&rows): # i: i64, row: &Row
for a, b in zip(&left, &right): # a: &T, b: &T
A loop binds the two values to two names, which works today. A callback needs
a parameter list, and the question is what it looks like. The obvious answer is
one tuple parameter, def(pair: (&str, &i64)) -> bool, but Plenty cannot form
that type: tuples cannot hold references (“references cannot be stored in
tuples”), because a tuple is a stored value and stored references are B11.
That is also why borrowed zip/enumerate loops today require two unpacked
bindings.
Options:
- Two parameters.
def(name: &str, score: &i64) -> bool. Works with the current type system; matches the loop’s two bindings. - One tuple parameter. Needs tuples of references (B11), and every use
then indexes
pair[0],pair[1]. - A destructuring parameter.
def((name, score): (&str, &i64)) -> bool. Reads like the loop, but adds parameter-pattern syntax to the language and still names a reference tuple type. - Two parameters for borrowed pairs, a tuple for owned pairs. What the first draft proposed; it makes the callback shape depend on whether the source was borrowed.
Recommendation: option 1, for every pair item, owned or borrowed. The hard rule is: a callback over a pair item takes two parameters. No exception for owned pairs, and no new syntax or types:
top: list[str] = scores.items()
.filter(def(name: &str, score: &i64) -> bool: *score >= 90)
.map(def(name: &str, score: &i64) -> str: copy(name))
.collect()?
flagged: list[i64] = enumerate(&rows)
.filter(def(i: &i64, row: &Row) -> bool: row.bad)
.map(def(i: i64, row: &Row) -> i64: i)
.collect()?
Only the top level is spread. In zip(enumerate(a), b) over owned a, the
inner (i64, T) is an owned tuple passed as one argument. An inner pair that
holds references cannot be formed and is rejected until B11; loops cannot
unpack nested pairs today either. A map that returns an owned tuple
produces a pair item, so the next callback takes two parameters. collect()
into a dict[K, V] target consumes owned pairs directly. If B11 later adds tuples of references, option 1
stays valid and nothing has to change.
The accumulator is separate from the item spreading rule: pair fold and
try_fold callbacks take (accumulator, left, right), even when the accumulator
is a tuple. Pair reduce similarly takes (accumulator_pair, left, right).
Predicates borrow each component; owned components therefore become &T and
borrowed &T / &mut T components become &T for the predicate call.
Closure parameter types
Callbacks keep the current rule: parameter and result types are always written. This proposal does not add inference.
The cost is verbosity: def(n: i64) -> i64: n * n where Rust writes |n| n * n.
The benefit is that each step of a chain states its item type where it is
read. In a long chain, the item type changes at every map, and without
annotations a reader must trace the chain from the source to know what n is
at step five. Explicit types also make errors local: a wrong callback is
reported at that callback, not as a mismatch several steps later. This
matches the project’s preference for mandatory rules, and the rule is already
taught. Relaxing it later (inferring parameter types from the adapter when the
item type is known from the same chain, under B16) remains an
open question; it can be added without breaking code that
writes the types.
The closure and inference design history also constrains this choice: captures remain explicit, their loans stay live while callbacks may use them, and whether lowering inlines a callback must not change which captures are legal. Parameter inference would not imply capture ownership inference.
Terminals
| Terminal | Result | Notes |
|---|---|---|
.fold(initial, f) | A | Left fold; f: (A, Item) -> A |
.reduce(f) | Option[Item] | Left fold without an identity; Nothing when empty |
.count() | i64 | Checked; traps past i64.MAX like enumerate |
.any(keep), .all(keep) | bool | Short-circuit; predicate rule above |
.find(keep) | Option[Item] | First match; drops the rest |
.for_each(f) | () | f: (Item) -> (); prefer a for loop when the body has statements |
.last() | Option[Item] | |
.collect() | Result[C, AllocError] | C is the expected list[T], set[T] or dict[K, V]; see below |
.try_fold(initial, f) | Result[A, E] | f: (A, Item) -> Result[A, E]; stops at the first callback Err |
.try_for_each(f) | Result[(), E] | f: (Item) -> Result[(), E]; stops at the first callback Err |
.try_collect() | Result[C, AllocError | E] | Over Result[T, E] items; same targets; first Err wins |
.sum() | Item | Numeric items; zero when empty |
.min(), .max() | Option[Item] | Numeric or str items; NaN propagates |
For serial pipeline machinery, collect and try_collect are the only
terminals that allocate. The try_ prefix means stopping at the first error,
but the error source differs: try_collect consumes Result items, whereas
try_fold and try_for_each consume ordinary items and call a fallible callback.
This preserves the contract of the former std.iter.try_fold; these callback terminals do not implicitly unwrap Result items.
Early-exit terminals drop the remaining state when consuming an owned pipeline.
Through &mut Iter[T], they leave the unvisited remainder with the caller,
just as a borrowed consuming loop does. An item already passed to a callback
or tested by a predicate has been consumed, including the item causing an error.
Agreed for stage 3: .sum(), .min() and .max() are terminals. Each
compiles only when the item type supports the operation: sum needs numeric
items, min and max need ordered items. Implemented for numeric items
and, for min/max, str; borrowed items are read as *item. Tuples are
rejected for now: a tuple containing a float has no NaN rule yet, so they use
fold or reduce.
sum()returnsItem; an empty pipeline gives zero. Integer overflow is a runtime error, as for+.min()andmax()returnOption[Item],Nothingwhen empty. Ties keep the first item.- NaN propagates. For
sumthis is ordinary float addition. Forminandmaxit is an explicit rule, IEEE 754-2019minimum/maximum: any NaN item makes the resultSome(NaN), and-0.0orders below0.0. A plain<loop would instead drop or keep a NaN depending on its position.std.math.min,max,min_allandmax_allfollow the same rule. - There is no parameter to ignore NaN. Filter first; the choice is then visible at the call. Separately named NaN-ignoring terminals can be added if filtering proves too common or slow.
NumPy, Julia, Java and IEEE 754-2019 also propagate by default; where both
behaviors exist, most libraries use separate names rather than a flag.
median is not planned: an exact median stores every item, and a streaming
estimate is a different contract.
Choosing the collection
There is one collecting terminal. Its target comes from the expected type, as empty displays already do.
Hard rule: collect() and try_collect() build the collection named by
their expected type. Without an expected collection type, they are a compile
error. There is no default.
names: list[str] = iter(&people).map(def(p: &Person) -> str: copy(&p.name)).collect()?
tags: set[str] = iter(&posts).flat_map(post_tags).collect()?
index(iter(&people).map(def(p: &Person) -> (str, i64): (copy(&p.name), p.id)).collect()?) # def index(by_name: dict[str, i64])
n = iter(&xs).collect()?.len()
# error: collect() needs an expected collection type; annotate the binding,
# as in `name: list[i64] = ...collect()?`
Expected-type sites. The same sites that type an empty display today (verified with the current compiler):
- an annotated binding,
x: set[u64] = ....collect()?; - assignment to a binding or field whose type is known,
xs = ....collect()?; - a non-generic parameter,
take(....collect()?); - a function’s declared return type, through its
Result; - a class field in a constructor call,
Bag(items=....collect()?).
The expected type passes through ? and through the Result of the return
type, exactly as name: list[i64] = []? does now. Sites that give no
expected type: an unannotated binding, a generic parameter position (as
type argument inference already
states), a method receiver (....collect()?.len()), and later uses of a
binding. The checker does not look ahead.
Precedent. The checker already types expressions from an expected type:
xs: list[u8] = [1, 2]?types the literals asu8; parameters do the same for numeric literals.- Empty displays take their element types from the expected
Resulttype, ands = []?with no expected type is an error: “empty collection needs a type; annotate the binding, as inname: list[i64] = []?, or use a typed constructor”. Atomic(value)takesTfrom the expected type, and expected types allow anonymous-sum injection and widening.
collect() extends this by letting the expected type choose the collection
kind, not only its element type. Displays never needed that, because their
syntax picks the kind. The checker change is to treat collect/try_collect
as a contextually typed terminal in the same expected-type pass.
Target rules.
list[T],set[T]anddict[K, V]are accepted. Any other expected type, is an error naming the accepted kinds; aliases resolve first.- The element type must equal the item type exactly. Items are already typed
by the chain (callback types are explicit), so nothing flows backwards into
the pipeline and there is no conversion:
x: list[u8]overi64items is a mismatch, not a cast. set[T]needs hashable items and keeps the first occurrence, asset[T].fromdoes.dict[K, V]needs owned(K, V)pair items, and a later duplicate key overwrites, as dict displays do.- The
Resultwraps the target:Result[C, AllocError], and fortry_collectoverResult[T, E]items,Result[C, AllocError | E]. The expected type may be the bareC(with?) or the wholeResult.
Why an error, not a list default. Numeric literals default to i64, but
the closer precedent is the empty display, which errors without context. A
default would make xs = ...collect()? silently a list and change meaning
when someone later adds an annotation elsewhere, and readers would need to
know the default to read unannotated code. The error costs one annotation,
which also documents the collection kind at the call site, where readers need
it. It is the rule with no exception: every collect() has a visible or
declared target type.
Alternatives considered.
- Passing the container type,
.collect(set), as excitertools does. Python has no static types, so the container must be a runtime argument there. In Plenty the element type is already known and the target is almost always declared nearby; a type-valued argument would be a new kind of argument (types are not values) and would duplicate the annotation. It also would not remove the need to check it against an expected type. - Separate terminals (
collect_set(),collect_dict()). Explicit at the call, but three names for one operation, and an annotation would still be checked against them. - A type argument,
collect[set[str]](). Turbofish; rejected.
list[T].from(pipeline) is not accepted; its diagnostic points to
collect(), so there is one spelling.
Errors
Closure bodies never return from the enclosing function. A ? inside a callback
returns from the callback, as it does today, so a fallible step produces
Result items. Handle them with try_collect, which stops at the first
Err and drops the rest of an owned pipeline, or with a for loop whose body uses
?:
parsed: list[i64] = iter(&lines).map(def(l: &str) -> Result[i64, ParseError]: parse_int(l)).try_collect()?
try_fold and try_for_each instead propagate their callback’s first error;
handling a Result item inside that callback must be explicit. try_collect
widens its item error with AllocError exactly as anonymous sums widen today.
The callback terminals add no allocation error of their own. There is no
implicit error collection and no hidden recovery.
Allocation
A serial pipeline allocates nothing of its own: no iterator object, no frame, no item
buffer, no closure environment beyond the inline one. Only the collect
family allocates, and it returns Result. Callback bodies and source
expressions keep their own contracts. collect reserves capacity
when the pipeline’s length is known exactly (a range or collection source with
only map, enumerate, zip or take), and otherwise grows like a
comprehension.
These guarantees follow the reference’s allocation boundaries: user callbacks, source expressions and destructors retain their own allocation behavior. Explicit generator conversion introduces an inline frame. Parallel execution is a separate boundary: job cells, output buffers and reduction scratch storage can allocate even for a non-collecting terminal, and failures must be reported through its parallel error type.
Ownership and cleanup
- The pipeline owns its owned source, its closures and adapter state (counters). Dropping an unconsumed pipeline drops them in reverse creation order without running any callback or advancing any generator.
- A borrowed source’s loan lasts until the pipeline is consumed or dropped, and
protects the collection exactly like a
forloop’s loan. - Closures may capture by value,
mut,&or&mut; the captures’ loans last as long as the pipeline. Two adapters may not both hold&mutto the same binding, by the ordinary aliasing rule. break,returnand?in a consuming loop drop the remaining source, pending items and closures when the loop owns the pipeline, as loops over owned generators do today. A consumer borrowing&mut Iter[T]instead releases its local state and loan, leaving the owner’s remaining pipeline available; early termination does not destroy the caller’s source or closures.def onceclosures are rejected as adapter callbacks, since adapters call their callbacks repeatedly. A terminal that calls its callback at most once (none in the core set) could accept them later.
Efficiency model
Fusion at the consumer
A pipeline value is a compile-time plan: a source descriptor plus an ordered list of adapter steps, each with its inline closure environment and any counter slots. Nothing about the plan exists at run time except those slots.
When the plan reaches a consumer, the lowering in op.rs expands it into the
same loop structure it already produces for for and comprehensions:
- the source becomes its existing loop plan: a counted loop for ranges, an indexed loop for lists, a hash-table walk for dicts and sets, a resume call for generators;
filterandtake_whilebecome a conditionalcontinueorbreak;mapbecomes a call whose result replaces the item;enumerate,takeandskipbecome counter slots with checked arithmetic;zipadvances a second source plan in the same loop, as it does today;chainbecomes two loops sharing one outlined body, so the body is not duplicated;flat_mapbecomes a nested loop;- the terminal or loop body is the innermost block.
This is how comprehensions already work, so the work is mainly generalizing an existing plan, not writing a new optimizer. Comprehensions become a spelling of a pipeline (see below).
Callback calls
A named function is called directly. A closure literal written inside the chain is known at the call, so its one-expression body is lowered in place, exactly like a comprehension’s element expression, with its captures resolved to the pipeline’s environment slots. A closure held in a local is called directly through its known producer with its environment address, with no indirect call. This removes all runtime dispatch from the hot loop; what remains is at most one direct call per callback per item, and none for closure literals.
A general function inliner is out of scope for this proposal. If measurement shows direct calls to small named functions matter, a size-bounded inliner at the operation level can follow as a separate B19 item.
Compile-time cost
Fusion adds no generic instances: an adapter is not a generic function, and a chain is not a nested type that must be instantiated and then optimized away. Lowering cost is linear in the number of steps at each consumer. Rust’s model creates one monomorphized type per adapter per chain and relies on LLVM to inline them away; Plenty would pay that as extra Cranelift functions and still not get the inlining. A pipeline that is not lent is lowered once, at its single consumer, because it is affine; a lent pipeline is lowered at each consuming site and in each callee instance. Apply the existing generator nesting limit (64 steps) to chain length.
Verification
Acceptance for the implementation, not optional extras:
- Operation equivalence. Snapshot tests compare the typed operations of a
pipeline with the hand-written loop it replaces, for each adapter. Use the
per-function dump being added in issue #360 (
--dumpops/CLIF/asm). - No runtime dispatch. A test inspects the CLIF of fused pipelines over
range, list and borrowed-list sources and asserts there is no call to the
collection dispatcher and no
call_indirectinside the loop. - No allocation. Allocation-checks tests (the
allocation-checksfeature) run every non-collecting terminal with allocation disabled. - Benchmarks. Add the table above as a release benchmark. A pipeline over a range or list must be within 10% of the hand-written loop; report the generator-source rows separately, since their cost is the generator’s.
- Compile time. Add a pipeline-heavy workload to
compile_benchand check that cold builds do not regress against the equivalent comprehension code. - Semantics. Ordering of callback side effects, cleanup on early exit,
loan errors for borrowed sources, partial consumption through
&mut Iter[T]followed by collection in the caller, and every rejected placement (return, pass by value, store, capture,yield) with specific diagnostics. Cover owned generator conversion and rejection of borrowed conversion,flat_mapreturningIter, and the distinct callback-error and item-error terminal contracts. For parallel stages, verify tree shape across worker counts and chunk boundaries, earliest-input error selection, and draining accepted work on early termination and infrastructure failure.
For loops, comprehensions and generator expressions
for x in pipeline:is the main consumer.for x in iter(values):is valid but redundant; preferfor x in values:. A lint can suggest the shorter form.- A list comprehension
[f(x) for x in s if p(x)]lowers to the same plan asiter(s).filter(p).map(f).collect()into alisttarget. Comprehensions stay: they are the Python-shaped way to build a collection, and the tutorial teaches them first. The two forms must produce the same operations; that is a test, not a goal. Implemented for oneforclause with anyifclauses, for lists, sets and dictionaries; severalforclauses keep the nested loop. .generator()converts an eligible prepared pipeline into one frame. Ordinary generator expressions keep their existing syntax and owned-source rules; an unconverted pipeline is not a valid generator-expression source. Stored range sources already use counted iteration (issue #371).- There are no free
any(xs)/all(xs)builtins, as there are no freesum,minormax..any(keep)/.all(keep)take a predicate;.any()/.all()testboolor&boolitems directly, asfilter_value()does, and compile to the same loop the former builtins did. Over a generator expression the former builtins advanced a frame per item;iter(xs).any(keep)fuses the test into the source loop instead.
Parallel pipelines
.par(&pool) marks the boundary: steps before it run serially on the calling
thread to produce input items; steps after it run on the pool’s workers. The
pool is the existing ThreadPoolExecutor; there is no global pool, so thread
creation stays explicit and fallible, as the
parallelism proposal
requires.
with ThreadPoolExecutor(8, 64)? as pool:
lengths: list[i64] = iter(paths).par(&pool).map(file_size).collect()?
longest = iter(range(1, n)).par(&pool).map(score).reduce(best)?
Stage 4 contract, extending current pool.map/pool.reduce_tree:
- Sources. Owned lists and integer ranges, as
pool.mapaccepts today. Borrowed lists follow once scoped pool jobs can borrow (B22). (Issue #455 added borrowed lists without user-facing scoped jobs: chunks read the rows in place because every terminal drains its jobs before returning.) - Allowed steps after
par.map,filterandfilter_value(over copyable items);enumeratemust be applied beforepar. Order-dependent steps (take,skip,take_while,chain,zip,flat_map) are rejected afterparwith a diagnostic naming the step. - Callbacks. Statically named functions or reusable immutable closures that
wholly own their captures and pass the existing worker eligibility and effect
checks.
mut,&mutand one-shot captures are rejected; shared borrows wait for scoped jobs. Generator sources are rejected, as frames are not worker-eligible. - Terminals.
collectandtry_collectinto alisttarget (lowest-index error wins, asmap_result),count,any/all,sum,min/max, andreduce.sum,minandmaxwork per chunk, then combine chunk results in input order. Integersumtraps exactly when the serial sum would;min/maxequal the serial results (first NaN wins,-0.0 < 0.0); a floating-pointsumregroups additions by chunk. A parallelreduceworks likesum: each chunk reduces its items left to right, then the chunk results are reduced in input order, skipping empty chunks. It is deterministic for a given input length and chunk count, but differs from the serial left fold for non-associative operations such as floating-point addition.pool.reduce_treekeeps one fixed grouping for code that needs it. There is no parallelfold: it would need an identity and associativity the type system cannot check.forloops andfor_eachare rejected afterpar, since their bodies would run concurrently. - Reduction grouping. A parallel
reducegroups by chunk, assumdoes, so no list of all items is collected; empty input yieldsNothing. An earlier version of this proposal used the fixed-tree reduction contract ofreduce_tree, so worker count and chunk boundaries could not change the grouping. That required collecting every item before reducing, which madereducewith a cheap function several times slower than serial. Code that needs the fixed tree callspool.reduce_treedirectly. - Early termination and effects. Follow the
fallible mapping contract:
try_collectselects the lowest original input index’s worker error, including across chunks. An earlier accepted input’s error takes precedence over a later submission failure. Accepted work is drained before returning, and partial outputs are dropped exactly once. Forany/all, a decisive result may stop further submission, but already accepted chunks may still run callbacks; their effects are not undone. These terminals do not promise the serial callback count. An infrastructure failure already encountered is still reported, rather than hidden by a decisive boolean result. - Result type. Parallel terminals return
Result[_, PoolMapError]orParallelError[E], because submission can fail and the pool can be shut down. - Granularity. Each job runs the fused
map/filterbody over a contiguous chunk of input, not one item. The default number of chunks is the pool’s worker count (max_workers), read at run time, never the machine’s core count..par(&pool, chunks=N)fixes the count;Nis a positivei64. An input shorter than the count gets one chunk per item, with no empty chunks. Chunk boundaries depend only on input length and the count, never on scheduling, so the same program and pool size give the same results on any machine. Filtered chunks are concatenated in input order. Further tuning (chunk sizes,par_chunks, adaptive splitting) is deferred until there are benchmarks.
Compile-time checks the type system must enforce: worker eligibility of every
callback and item type (existing closed-world check), no shared mutable
borrows, no order-dependent step after par, and reduce only (not fold)
for parallel combination. Associativity itself cannot be checked; the
documentation states the grouping: per chunk, then chunk results in input
order.
Automatic parallelization
Pipelines give B24 a natural unit: a fused loop body whose callbacks are pure
by effect analysis, over a sized source. B24 remains deferred, and this proposal
does not change that: a pipeline without .par is always serial. Explicit
.par gives the evidence the parallelism proposal asked for: which workloads
benefit and whether effect checks are sufficient.
Migration
Status: complete for stage 2. std.iter was removed outright instead of
deprecated: it had no users outside the repository’s tests, and there is no
deprecation-warning mechanism. Its tests now exercise fluent generator
pipelines and std.error.map_error. The tutorial lesson moved to
Transform a stream of values, after the
borrowed-collection pipeline lesson. The items below record the original plan.
std.iter. Markmap,filter,take,foldandtry_folddeprecated in a deliberate compatibility change after migration review. Fluent methods plus.generator()now cover their cross-function use cases; keep existing functions working until removal is separately approved. Internal uses are limited totests/test_std_collections.rsandtests/test_trap_locations.rs.- Tutorial. Rewrite
Transform a stream of values (then in the modules chapter)
around pipelines (implemented, including explicit returning conversion).
A later organization pass can move it out of the modules chapter, since
iterneeds no import; it belongs after comprehensions and closures in the iteration lessons. Review lessons that build lists by looping only to fold them, and the comprehension lessons for the equivalence note. - Reference. Add a pipeline page under
iteration; update
adapters (enumerate/zip
become pipeline starts), the
std.iterpage,std.math(its list helpers stay; pipelines use the stage 3 terminals), parallel mapping, the cheat sheet andsupport.md. - Compatibility. No existing program changes meaning:
iteris a new name (a user function namediterkeeps priority, as withenumerate), andenumerate/zip/reversedloops lower as before.
Alternatives considered
Method syntax for std.iter (extension methods or uniform call syntax).
Lets numbers(3).map(sq) call std.iter.map. Cheap to add, but keeps the 9x
per-item cost and the generator-only, owned-capture limits, and an import that
adds methods contradicts the protocol rule that imports never add methods.
The related protocol method bodies proposal distinguishes provided methods selected through a bound or protocol qualification from automatic method injection. Its agreed, unimplemented design preserves that import rule. Provided bodies could share adapter implementation logic, but do not by themselves add fluent methods to built-in generators or remove their frame and dispatch costs.
Rust’s model: generic adapter structs plus inlining. Requires storing generator frames and closures in generic class fields (rejected today), user-implementable iterator protocols with associated types (unsupported), and a function inliner in the Cranelift pipeline. It multiplies generic instances per chain and raises compile time, against a design goal, for performance the compiler would still have to recover.
Faster generators only. Inline the runtime dispatch, call resume directly when the producer is known, and use a jump table for state. Worth doing for named generators regardless (a separate B19 item), but a resume per adapter per item, with frame-backed locals, cannot match a fused loop, and it does not lift the borrowing limits.
Comprehensions only. Comprehensions already fuse, but cannot express
take, take_while, chain, early-exit reductions, or par, and nested
clauses become hard to read beyond two levels. The user request is precisely
for the chaining form.
First-class storable pipelines from the start. Convenient, but storing a borrowed pipeline needs stored references (B11), and storing an owned one needs frame and closure storage in generic data. Keeping pipelines in their creating scope, with lending for helpers, ships the useful part now.
std.math helpers over lent pipelines. sum_iter(&mut pipeline, 0) and
fold(0, math.max) were considered for stage 3 and deferred. They need module
functions as values and generic function values without explicit type
arguments, and they cannot reject unsuitable item types at the call. The
terminals above replace them for pipelines.
Open questions
- User sources. A user iterator protocol (
next(self: &mut S) -> Option[T]) could make classes pipeline sources once B12 settles; until then, a method returning a generator works. - Closure parameter inference. Should callbacks later omit parameter types when the adapter knows the item type (B16)? This would shorten chains at the cost of the readability described in closure parameter types. Decide after stage 1 with real code to compare.
- Passing pipelines by value. Whether helpers that add steps
(
def evens(p: Iter[i64]) -> Iter[i64]) are worth weakening the never-leaves-its-scope rule. Not planned; revisit only with use cases that closure-returning helpers do not cover.
Staged plan and suggested backlog wording
- Stage 1 — serial core.
iter,Iter[T], local-only placement,map/filter/enumerate/zip/take/skip/take_while, two-parameter pair callbacks, and terminalsfold/reduce/count/any/all/find/for_each/collect/try_fold/try_for_each/try_collect(target from the expected type), fused at the consumer, with the verification suite above. Issue #371, fixed, removed the stored-range slow path that would have misled the benchmarks. Small scalar leaf callback substitution and SSA scalar captures are also implemented; measurements cover the original workload. Generator fluent entry and the explicit owned.generator()bridge also landed independently of composition. - Stage 2 — composition.
chain,flat_map, closure-literal splicing, comprehensions lowered through the pipeline plan, tutorial rewrite andstd.iterdeprecation.chainandflat_mapare implemented. Closure literals written in a chain are lowered in place for every adapter and callback terminal; literals with?,match, several statements or a nested closure, and closures held in locals, keep the call. Single-forcomprehensions lower through the plan and produce the same operations as the equivalentfilter/map/collectchain; multi-forcomprehensions and generator expressions keep their own lowering. Complete: the tutorial teaches pipelines, andstd.iterwas removed rather than deprecated (see Migration). - Stage 3 — lending.
&mut Iter[T]parameters specialized per pipeline type,next(p), resumable slot state for every adapter, store-back only for lent pipelines, and partial consumption followed by collection in the caller. Measure instance counts incompile_bench. Thesum,minandmaxterminals described under Terminals are implemented. Every adapter is resumable, andnext(p)works in the creating scope: a later loop or terminal continues after the items taken. In that scope the plan’s state already lives in the creator’s slots, so resuming needs no store-back;take_whileandziprecord that they have ended so a later consumer does not pull past the end. Lending is implemented: the callee receives the pipeline’s state through hidden parameters, copied in on entry and written back on every exit. Pipelines of reference items can be lent; their items cannot be returned. A pipeline holding a stored closure with borrowed captures cannot be lent; write the closure literal inside the adapter instead. The reference states the rules and limits. - Stage 4 — parallel (implemented).
.par(&pool)with chunked fused jobs (one chunk per pool worker by default, orchunks=N), the allowed-step rules, andcollect/try_collect/reduce/count/any/all/sum/min/max. The reference lists the implemented contract.try_collectreportsParallelError[E | AllocError], since a chunk’s output list can fail to allocate on the worker;any/alldo not stop early.
Suggested backlog row: B75 — Fluent iterator pipelines. “iter(source)
pipelines with chained adapters and terminals, fused at the consumer into one
loop with no adapter dispatch or serial machinery allocation outside collecting terminals.
Stages: serial core, composition and std.iter deprecation, lending as
&mut Iter[T], explicit .par(&pool). Depends on issue #371. Relates to B12,
B22, B24, B49. Design: proposals/fluent-iterators.md.”
Issues found
- Issue #371: generator expressions over
range(...)and stored ranges iterate through the runtime, about 15x slower than a direct range loop. Fixed by local PR #436: they now compile to counted loops.
Owned response experiment and consuming receivers
Status: Experiment complete; consuming methods implemented. The source is
in experiments/typestate/. The original model used consuming free functions
against baseline 344dfdbd; it now exercises explicit owned receivers. This
refreshes the typestate assessment.
The HTTP server is unchanged. Individual field moves out of owned bindings were
implemented later (issue #534), and the model now uses them.
Real HTTP integration remains unexplored and is tracked in B72.
What the experiment establishes
Two public classes, Unanswered and Writer, hide their resource fields and
constructors. Consuming methods implement these transitions:
Unanswered.begin(self: Unanswered, length) -> Result[Writer, StartError]
Writer.write(self: Writer, count) -> Result[Writer, WriteError]
Writer.finish(self: Writer) -> Result[(), FinishError]
The wire is a deterministic counter with an injected failure position, not a network transport. Negative lengths, oversized writes and incomplete finishes return the still-valid owner in an error payload for retry. A partial write is terminal: its error reports bytes accepted in that call and in total, and the connection drops. An error must never offer a retry owner after effects that make replaying the operation unsafe.
A freshly built debug compiler compiled and ran nine resource lifetimes. The
runner checked exactly one destructor invocation per resource, including legal
retries, cancellation, abandonment, an owner carried through ? in a retry
error, and cleanup when a later function argument propagates an error. The
partial-write case accepted two bytes, then one of the requested three; it
reported (1, 3) and destroyed the connection before returning the error.
Twelve compile-fail probes cover writing/finishing before begin, repeated begin/finish, writing after finish, reusing the owner after a fallible write, forging either state, reading private storage, copying a resource-bearing state, consuming a borrowed writer, and using an owner as a whole after moving a field out of it. Compiler integration tests separately exercise owned receiver lowering, generics, protocols, generators and cleanup. These are reproducible assertions, not only a successful smoke check.
The cost is visible in the API: two public states and three error enums,
new writer bindings after successful writes, and explicit matching to recover
retry owners. begin moves the wire field out of the consumed Unanswered
owner into a new Writer. Before field moves, Unanswered held an
Option[Wire] and begin used replace(..., Nothing); neither version needs
an allocation or a dummy connection. Constructor privacy maintains the invariant that a live
public state contains a resource. No throughput, code-size, or allocation-count
benchmark was performed; the cleanup trace is not a performance measurement.
Why this is not yet an HTTP server conversion
The actual std.http.server.Handler.handle receives &mut Exchange[T]. The
server retains the exchange for request draining, reports and connection reuse.
A handler cannot consume that borrowed owner. Fields such as request-body
state, cancellation tokens and the connection also belong to the same exchange.
An owned response API therefore needs a deliberate split of responsibilities,
or a different handler contract, not just renamed methods.
The model’s recoverable incomplete finish is also deliberately different:
today Exchange.end marks a short sized response broken, and a transport error
during write leaves the response broken. The model does not prove that either
real failure can be retried. Real integration must preserve partial-progress
reporting, HEAD/chunked framing, cancellation and keep-alive behavior.
Keep ordinary writes borrowed if returning a writer for every write is too verbose; then terminal transport failure still needs a runtime broken-state check. Typestate is most promising at coarse begin/finish boundaries. Discarding an unfinished writer remains legal: deterministic resource cleanup is not a guarantee that a response completed. That stronger requirement belongs to linear obligations.
Consuming methods use existing ownership
Explicit self: Class reuses the owned-argument path: it transfers an entire
value, rejects reuse and conflicting borrows, and transfers cleanup
responsibility. value.finish() behaves like finish(value), with static
dispatch and no new allocation required by the call itself. A compiler test
compares optimized assembly for equivalent method and free-function calls.
src/frontend/classes.rs, src/frontend/enums.rs and
src/frontend/protocols.rs admit owned receivers. Ordinary and generic method
calls forward them as values; named/default-argument calls use the existing
wrappers. Temporary borrowed views cannot be turned into owners by dispatch.
Bare self remains a shared borrow and self: &mut Class an exclusive borrow.
Lifecycle methods and operator hooks keep their existing contracts. Owned
parameters remain immutable bindings; a body can move self into a mutable
local. Protocol matching distinguishes owned and borrowed requirements. An owned
receiver cannot silently move through an affine reference, class field or indexed
element, or return a reference to its dying storage. Copyable enums keep ordinary
copy semantics rather than becoming affine solely because a method takes a value.
Tests cover enums and generic classes/methods, temporary and named receivers, protocol calls, named/default arguments, live loans, failure in a later argument, conditional moves and exactly-once destruction. Receiver evaluation precedes later arguments; mutable borrowing’s two-phase reservation does not allow reading the moved owner again. These are ordinary owned-function-call rules.
The first stdlib adoption is std.text.Builder.finish(). It consumes the builder
on success or allocation failure; to_text() remains a borrowed snapshot for
reuse and retry. std.text.join and wrap now finish their builders explicitly.
Both finalization paths still copy output bytes; no zero-copy improvement is
claimed. SQLite and HTTP lifecycle contracts are not converted.
Individual field moves
Implemented (issue #534). This section originally deferred field moves:
they needed field-level availability and cleanup rules across branches, loops,
reassignment and early returns, plus a rule for custom destructors that might
inspect an extracted field. The implemented rules track each moved field like a
binding, reject whole use of the owner until the field is assigned again, clean
up only initialized fields, and reject moves out of a class with __del__. See
field moves. Whole-object destructuring is still not included.
Flat class layout
Status: Stage 1 (one-field classes as newtypes), stage 2 (flat nested class records), stage 3 (fewer runtime walks) and stage 4 (flat tuple and variant payloads) implemented. Issue #547 asked why a class value is an address into its own storage, and whether classes could instead be laid out flat inside their owner. Values and memory own the shipped contract; this page records the findings and the design.
Why a class value holds an address
Every runtime slot is 128 bits: a payload word and a tag word. A class keeps its fields in inline storage that follows its slot in the owner (a local, a record field, a list element, a box), and the payload word is the address of that storage. The address exists because the payload word has to be something uniform:
- Runtime operations (
copy_payload,release,retain, hashing, equality, formatting) receive oneu128and a type descriptor and walk the record through the descriptor. They never receive the slot address. - References are
(address, shift)pairs.&self.fieldis computed from the address of the record, which is read out of the class value word; a reference to a class value therefore projects fields with one load and an add, not by knowing where the value’s slot lives. - Sum payloads are packed into the same 128 bits as their tag; a variant holding a class holds its address word and leaves the tags free.
- A zero payload word means “empty slot” for every type, which lets moved-out locals, fields, and partially initialized records release for free.
The cost is that every move of a class value is a copy of its inline bytes
plus a fix of each nested class, range, closure, or inline-sum address
(inline.rs::fix_record, the runtime’s Payload::relocate); issue #544
measured that as the remaining overhead of nested wrappers.
What depended on the address before stage 2
| Dependent | How it uses the word |
|---|---|
lower_class (src/codegen/enums.rs) | Field loads a slot at word + offset; FieldRef loads the word from the reference’s address and adds the offset |
copy_inline_to, copy_record, fix_*, release_record (src/codegen/inline.rs) | memcpy from the word, then rewrite nested words at base + offset + 16 |
runtime Type::record (plenty-runtime/src/aggregates.rs) | value as u64 as *mut u128 is the record base for walks, drops, copies, hashing, equality, formatting |
references (src/codegen/references.rs) | write_reference copies inline payloads to address + 16 and stores the word at address |
sum payloads (src/codegen/enums.rs, runtime Type::payload) | a variant’s single class field is the class’s 128 bits; tags stack in the high word |
| generators and closures | class captures and frame slots are ordinary slots with following inline bytes |
Box[T], Shared[T], actors | the box holds one slot plus inline bytes; copies between actors use copy_payload |
C export (src/codegen/exports.rs, src/exports/handles.rs) | a handle is a Box holding the instance; a borrowed &Class is the slot address inside the handle; an owned argument is copied out with copy_payload using the class descriptor |
columnar lists (plenty-runtime/src/aggregates/clist.rs) | the row class descriptor’s variants are the column schema |
Stage 1: one-field classes are newtypes
Rule. A class with exactly one field and no __del__ is represented
exactly as that field: same 128-bit slot, same inline bytes, same descriptor
layout. Constructing it, reading or borrowing its field, and moving the field
out are all identity operations in the generated code. No annotation; the rule
is total.
Classes with __del__ are excluded because their record carries a state word
(initialization completed, observer count) that has no home in the field’s
own slot. Field-less classes already lower to a zero word and keep doing so.
Compiler
ClassType::newtype_field(src/record.rs) names the rule once;Ty::newtype_fieldandTy::newtype_baseexpose it.- Layout queries delegate to the field:
has_inline_storage,flat_record,inline_bytes, and the generator frame measurement. lower_class:Newis a zero word (an empty slot, which the field’s__init__write fills);ArmDropis a no-op;FieldandFieldRefreturn their operand unchanged. The finalunpackto the field’s native type is the only code emitted.retain,release, andcopy_inline_tounpack to the field type and recurse.fix_slot,fix_variant,slot_fixes,plain_variantsandrelease_fieldslook through newtypes, so a record or sum holding a newtype copies and releases as if it held the field directly.- Columnar-list row access reads a one-field row as its scalar.
- A newtype has no inline storage, so a function returning one built by a constructor call ends in a tail call. The constructor inlining from issue #548 now also lowers at that site: the caller’s locals are released first, as a real tail call would, and the shared exit then finds empty slots.
Runtime descriptor
A newtype’s descriptor carries the field’s layout (kind, affinity, inline
bytes, links, variants, destructor hook) so every runtime walk treats the
value as the field. The class name and a new wrapper link (byte 64; the
descriptor grows from 64 to 72 bytes) record the source-level shape for
formatting, which prints Wrap(x=1), and for the columnar-list schema.
FFI layout contract
C code never sees a class layout: a class crosses the boundary as an opaque
handle (a Box holding one slot plus the descriptor’s inline bytes). The rule
changes only what that slot holds for one-field classes, and every handle
operation goes through the descriptor, so exported libraries, foreign pointers,
and dynamic exports keep working without a header change. The contract FFI
relies on is therefore:
- A handle points at a
Boxwhose value slot is at offset 32. - A borrowed class argument is the address of that slot.
- An owned class argument or result is copied in or out with the class
descriptor’s
copy_payload.
These three hold for every class, newtype or not.
Costs and limits
Typegrows by 8 bytes, so every program’s read-only data changes; machine code for programs without one-field classes is unchanged.- A one-field class with
__del__keeps the record layout. - Formatting a newtype walks one extra descriptor link.
Measurements
Retired instructions (perf stat -e instructions:u) against main at c78365ad,
which includes constructor inlining, compiled with --release:
| Program | Before | After |
|---|---|---|
Issue #544 reproduction, per cycle (file wrapped twice, 2000 iterations) | 486 | 444 |
| Issue #544 reproduction, 200 iterations, whole program | 340,245 | 331,699 |
200 loopback connect/accept/close pairs through std.net | 2,194,678 | 2,082,256 |
| Nested two-field records copied through a loop (no one-field class) | 283,245 | 283,175 |
The remaining per-cycle cost is caller-site bookkeeping around releases of slots the compiler could prove empty, not record copies. Programs without a one-field class produce identical machine code; only read-only data differs.
Stage 2: flat nested records
Goal. Store a class-typed field at a fixed offset inside the outermost owner without a word of its own, so a move is a byte copy with no address fixes.
Findings. After __init__, a nested class field’s word is redundant: it
is always slot + 16 or zero (moved out). Dropping the word (“rule B”) saves
16 bytes and one load, test and store per nested class per move. It needs:
- a presence marker for nested classes with
__del__(their state word can serve;FieldMovemust clear it) and for partially moved records; record_field,fix_*,release_record, and the runtimeRecord::slotswalkers to compute nested addresses from the owner instead of loading them;- references to nested fields to stay
(address, shift)pairs, which already works becauseFieldRefadds offsets to an address.
What it does not remove: the top-level word stays an address, because runtime
operations and sum payloads still receive one u128. Flattening below the top
level removes fixes but not copies; a move still copies the inline bytes.
First assessment. Stage 2 was first deferred: after stage 1 the remaining
per-move cost of a nested record looked bounded by RECORD_FIX_LIMIT. Measuring
it showed otherwise: in a loop that moves a three-level record through a
function, the slot fixes were about 90 of the 146 instructions the callee
compiled per call (6 fixes of 5 instructions per record copy, 3 copies), and
the nested words added 96 of the 272 bytes copied per record.
Decision. Adopted. A field of a class record whose type is a class with a record is stored flat. The cases that keep a word followed by a record are:
- top-level class values, whose word is the uniform 128-bit value runtime operations and sum payloads receive;
- class payloads of enum variants and tuple components: only class records flatten their fields, so variant payload packing and tuple layout are unchanged (stage 4 later flattened tuples and multi-field variants);
- field-less classes without
__del__, which have no bytes to place.
Contract
- Presence. An all-zero flat record is an empty field. Moving a field out
or releasing it zeroes its bytes; a class with
__del__has a nonzero state word once constructed, and every other record with a record has at least one nonzero word, so empty and live never coincide. - References. A reference to a class value names the 16 bytes before its
record. Readers use
address + 16without loading the word there; writers copy the record bytes, and writing an empty value zeroes them. Top-level values keep a real word at that address, flat fields do not, and no reader depends on which. Sort-key adapters, which receive runtime slots holding a row’s record address, convert to that form before calling compiled code. - Destructors. The runtime passes
record - 16to the__del__hook, which must not read the word. - Newtype construction. A one-field class wrapping a class with a record
is filled through a reference to the record, so
Newallocates the record’s storage as the wrapped class would.
Compiler and runtime
Ty::flat_fieldnames the rule;Ty::field_bytesandClassType::field_offsetgive a field’s width and offset.lower_classprojects flat fields by offset (class_field,referenced_record);read_typed_referenceandwrite_recordimplement the reference contract.fix_record,release_recordandrecord_fixes(src/codegen/inline.rs) walk flat fields in place and emit no fix for them.- The runtime
Field { at, flat }iterator (Record::slots) reads, writes and clears a field by kind. Copy, relocate, release, hashing, equality, ordering and formatting go through it. The resumable release walk clears a finished flat record, andField::is_clearlets a resumed pass skip it.abandon_copyreleases the copied prefix without allocating.
FFI
No change. A handle is still a Box holding one top-level slot, whose word
addresses the record; borrowed and owned arguments use the same descriptor
paths. Generated headers and .plentyi interfaces are byte-identical for a
library exporting nested classes.
Measurements
Retired instructions (perf stat -e instructions:u, --release, mean of 5,
pinned to 8 cores). Base is main at 5e6e98a4, before stage 2; main is
0ec41c06, which adds the empty-slot release removal (issue #549).
| Program | Base | Main | Stage 2 |
|---|---|---|---|
Three-level Point/Segment/Path workload | 5,033,105,263 | 5,033,105,267 | 3,700,365,927 (−26.5%) |
| Its move loop, per iteration | 1,840 | 1,840 | 1,302 (−29.2%) |
| Its list rows alone | 1,325,304,563 | 1,325,304,542 | 1,096,565,424 (−17.3%) |
| Nested two-field records copied through a loop | 282,667 | 281,066 | 280,637 |
| Constructor fixture (wrapped file) | 331,192 | 262,145 | 262,143 |
| Copies fixture, 5M wrapper cycles and 300k shelves | 5,612,022,549 | 3,868,322,585 | 3,570,485,254 (−7.7% vs main) |
| Constructor fixture, scaled | 2,225,242,405 | 505,241,987 | 505,241,982 |
| 300k shelves of nested records appended and destroyed | 3,387,022,898 | 3,363,322,395 | 3,065,484,601 (−8.9% vs main) |
200 loopback connect/accept/close pairs through std.net | 1,864,648 | 1,833,413 | 1,832,631 |
Code size for the three-level workload fell from 130,321 to 121,521 bytes of
.text. compile_bench (aot, 1000 × 10, pinned to 8 cores) retires 0.4%
more instructions (12.28 G to 12.33 G for 3 repetitions); its median times
(244–254 ms on main, 243–260 ms with stage 2) are within noise.
The first version of stage 2 made the shelves workload 7% slower than base. The runtime release walk restarts at an owner’s first field after each heap child, and with flat fields each restart re-entered nested records and scanned finished ones. Two runtime changes removed that cost:
- The walk frees a string in place instead of queueing it and restarting;
a string owns nothing and runs no user code, so the order is unobservable.
A
Shelfwith three strings is now walked once instead of four times. - Record traversal decides a field’s kind once per field, and walks into a
flat record build its
Recorddirectly instead of re-deriving it throughType::record.
Per shelf (callgrind, 20k shelves), the whole program fell from 9,716 instructions on base to 8,658; the release walk itself fell from 1,760 to 1,210, and relocation is level with base (685 and 663). Compiled moves no longer fix flat fields but still copy their bytes, so the per-move saving is the fixes, not the copy.
Stage 3: fewer runtime walks (issue #550)
Stage 2 left flat records copied by memmove and released and relocated by
the runtime’s generic field walk. Profiles of the stage 2 workloads
(callgrind, 20k rows of three-level Path records) put the release walk at
27% of the program, list appends’ copy_payload relocation at 16%, and the
memmove calls for 176-byte record copies at 9%. Each change below targets one
of those shares.
- Inline copies. Copies and clears of 33 to 192 bytes are 16-byte vector
loads and stores instead of
memmove/memsetcalls. All loads precede the stores, so overlapping copies stay correct. Larger records, such as the 272-byteShelf, still callmemmove. - In-place moves. A local moved into a call argument, with only operand
pushes between the move and the call, is passed at its own address: the
callee copies arguments on entry, so the snapshot copy was redundant.
Inlined constructors, generator producers and portable SIMD operations keep
argument addresses and still receive a copy (
inline::Moves). - Descriptor plans. A class descriptor with a record grows from 72 to 80
bytes; byte 72 links a
Planwith the word offsets of the strings the record owns (through flat fields), whether they are all it owns, and whether its bytes contain addresses into themselves. The runtime frees such a record’s strings in one pass instead of walking its fields, skips relocation for a record without internal addresses, and skips per-row release for a list of records that own nothing. Records with__del__at any depth, or any other managed field, keep the walk, so destructor order is unchanged.
The plan is data rather than a compiled hook. Descriptors are defined lazily while relocations are resolved, after the parallel function build; a compiled release function would need every eligible class declared before that build. For records owning only strings the data plan reaches the same result: no per-field dispatch, one load and release per string.
Measurements
Retired instructions (perf stat -e instructions:u, --release, mean of 5,
pinned to 8 cores). Main is 240cdd71.
| Program | Main | Stage 3 |
|---|---|---|
Three-level Point/Segment/Path workload | 2,508,593,168 | 1,312,945,806 (−47.7%) |
2M moves of a Path through keep | 466,240,999 | 376,240,717 (−19.3%) |
300k Path rows appended and destroyed | 2,035,990,651 | 932,943,557 (−54.2%) |
| 300k shelves of nested records appended and destroyed | 2,380,500,554 | 2,108,859,497 (−11.4%) |
| Copies fixture, scaled | 3,024,103,704 | 2,756,512,591 (−8.8%) |
| Constructor fixture, scaled | 1,205,241,633 | 1,205,241,629 |
| Nested two-field records copied through a loop | 280,638 | 277,878 |
| Constructor fixture (wrapped file) | 262,142 | 262,144 |
200 loopback connect/accept/close pairs through std.net | 2,045,036 | 2,009,922 (−1.7%) |
Wall-clock medians (15 runs): the three-level workload 254 to 140 ms, Path
rows 187 to 116 ms, moves 68.7 to 27.5 ms, shelves 246 to 222 ms, copies 313 to
291 ms. Per Path row (callgrind, 20k rows) the program fell from 5,819 to
3,016 instructions: destroying the list, string frees included, from 1,611 to
110, and the append’s copy_payload from 920 to 204. Per shelf, destroying
the list fell from 1,252 to 700 instructions, because its Label fields own
only strings.
Inline copies cost code size: .text grew 0.2% to 1.7% across examples
(classes 81,121 to 81,313 bytes, http_server_bench 556,705 to 566,257,
semantic_search 1,768,878 to 1,773,166, actor_echo 252,641 to 256,193) and
1.3% to 3.2% for the record workloads. A 256-byte limit sped up none of the
workloads and grew actor_echo by a further 2.5%; a 128-byte limit cost the
moves workload 8%. compile_bench (aot, 1000 × 10) retires 0.18% more
instructions (80.53 G to 80.68 G); its median times (332–340 ms on main,
331–336 ms with stage 3) are within noise.
Remaining
Shelves keep part of their walk cost: a Shelf holds ranges and sums, whose
payload addresses depend on the active variant, so it is neither string-only
nor free of internal addresses. Per shelf, relocation still costs about 930
instructions and the 272-byte record copy still calls memcpy. A plan entry
per internal address, tested against the sum’s tag, would cover relocation.
Stage 4: flat tuple and variant payloads (issue #551)
Stages 2 and 3 flattened class fields of class records only. A tuple or enum
variant with a class component still stored that component as a slot word
followed by its record, so every move rewrote the word: copying a
(Point, Point) through a function copied 96 bytes and then ran two address
fixes, and a three-field variant carried one fix per class field.
Decision. A tuple component or a field of a variant with two or more
fields is laid out as a class field is: a class with a record is stored flat
at a fixed offset, with no word. Matching, unpacking, &mut t[0], field
moves and constructors use the offsets class records use, and an all-zero
flat record marks a moved-out component. A tuple or variant whose components
are all flat records without internal addresses copies as plain bytes, so
(Point, Point) is a 64-byte copy.
Kept: single-field variants. The payload of a one-field variant, such as
Some(point), Ok(point) or Err(point), keeps the slot word followed by
the record. Its 128-bit value is the variant’s payload, and sum tags stack in
its high word, so the payload must be a full value as top-level values are;
dropping the word would change the Option/Result contract that runtime
operations, generic code and C headers use. Moving such a sum costs one
address fix, the same as moving a bare class value. A workload of 2M
Option[Segment] and Result[Segment, Problem] returns, matches and ?
propagations retired the same instructions
under callgrind before and after this stage (2,041,257,371 and
2,041,261,463), so flattening there has no measured gain to set against the
contract change.
Runtime. Record::slots marks a variant or tuple field flat by the same
flat_field rule as a class field, so release, retain, copy, relocate,
hashing, equality, ordering and formatting walk tuples and variants without
change. ranges::flat_record treats a multi-field tuple of flat records
without internal addresses as plain bytes, as the compiler does.
Measurements
Retired instructions (perf stat -e instructions:u, --release, mean of 5,
pinned to 8 cores). Main is d2078d82.
| Program | Main | Stage 4 |
|---|---|---|
2M swaps and moves of (Point, Point), 300k tuple rows | 1,758,757,596 | 1,374,894,550 (−21.8%) |
2M moves and matches of a Shape with Seg(Segment), Pair(Point, Point) and Tagged(Segment, i64), 300k rows | 1,820,986,482 | 1,667,438,878 (−8.4%) |
2M Option[Segment]/Result[Segment, Problem] returns and matches, 300k rows | 2,112,425,588 | 2,114,518,618 |
Three-level Point/Segment/Path workload | 4,327,731,012 | 4,325,030,979 |
2M moves of a Path through keep | 3,572,239,283 | 3,572,239,260 |
300k Path rows appended and destroyed | 755,730,005 | 753,030,049 |
| Copies fixture, scaled | 2,756,512,628 | 2,715,363,538 (−1.5%) |
| 300k shelves of nested records appended and destroyed | 2,257,209,573 | 2,218,460,498 (−1.7%) |
| Constructor fixture, scaled | 505,241,977 | 505,241,981 |
200 loopback connect/accept/close pairs through std.net | 2,010,559 | 2,010,513 |
The Option workload’s 0.1% difference is not in its code: its assembly is
unchanged and callgrind counts match. Compiled per call, the tuple keep
fell from 103 to 53 instructions and swap from 296 to 224; the variant
keep from 218 to 155 and its consuming match from 687 to 542.
Runtime allocation audit
Status: Implemented. Reconciled with the compiler on 2026-10-10. Allocation and traps owns the contract; this page is the per-path audit behind it.
Remaining: public allocator selection, allocator lifetimes, and routing of container buffers (B13).
The memory and SIMD assessment retains the public allocator and buffer design alternatives. This audit records the allocation contract that shipped, not those future APIs.
Allocation is fallible by default in the current language. This audit covers
generated programs, not compiler memory. Public allocating operations return
Result; there are no try_ alternatives or prefix try displays. Postfix ?
propagates failure, match handles it, and .unwrap() explicitly terminates on
failure. Nonallocating operations keep their ordinary return types.
| Path | Current contract |
|---|---|
| Collections and reservation | list[T](), new, with_capacity, and reserve return checked results |
| Growth and duplication | append, add, insert, extend, update, and copy use checked reservations and partial-owner guards |
| Class storage | Fields are stored inline; Class(...) and Class.new(...) allocate nothing and return Class. A fallible __init__ makes both return Result[Class, AllocError] |
| Enum storage | Payloads are stored inline; variant constructors allocate nothing, for user enums and builtin sums alike |
| Boxes | Box(value) returns Result[Box[T], AllocError]; it is the heap boundary for recursive types |
| Generator frames | A call to a yielding function returns its concrete inline frame directly and allocates nothing |
| Ranges | range(...) returns a copyable inline value and allocates nothing |
| Iterator collection | List/set construction and from check output growth; iterator bodies handle their own operations |
| Literals/comprehensions | Construction returns Result; failed growth stops evaluation and cleans partial contents |
| Tuples | Tuples are stored inline; a tuple display allocates nothing |
| Strings | Literals are immortal and strings of at most seven bytes are stored inline. + and builders return results. Indexing and iteration yield str and allocate nothing |
| Console/files | I/O results include allocation errors; partial reads/writes retain their documented effects |
| Formatting | str.repr and print use checked buffers. Printing an existing string needs no formatting allocation |
| Equality | Structural equality allocates nothing. It is rejected at compile time for recursive data (B33) |
| Indexed replacement | List/dictionary replacement allocates nothing. Missing dictionary keys trap; adding keys uses insert |
| Destruction | An intrusive queue needs no allocated work list. User destructor bodies handle their own operation results |
| Allocator provenance | Objects and container buffers use the global allocator; the object allocator prefix was removed (issue #814) until allocator selection needs provenance |
Class arguments evaluate left to right and move into the call. Storage cannot fail, because the instance is inline. Initializer failure drops initialized fields; the instance’s custom destructor becomes active only after successful initialization.
Collection display owners are allocated before entries or iterables are evaluated.
The first failed growth skips later expressions and releases the initialized
prefix. A ? inside an entry or filter returns from the enclosing function; the
outer display is not an exception handler. Earlier side effects are not rolled back.
Result and Option wrappers, builtin error payloads, propagation, and native
entrypoint status conversion need no allocation. A main that returns Err
reports it on standard error
and exits with status one. If formatting the report fails to allocate, it falls
back to a fixed allocation-free message.
Failure sweeps exercise construction, growth, duplication, formatting buffers, final string storage, and stopped comprehensions. Native allocation accounting checks reclamation; Miri checks runtime layouts and pointer access. Destructors used in allocation-failure tests either avoid allocation or restore the test budget before explicitly allocating output.
Explicit unwrapping, invalid operations such as out-of-bounds indexing, and fatal runtime invariants can still terminate without unwinding. Public allocator selection, lifetimes, and routing of container buffers remain future work.
References, ownership, and explicit copying
Status: Implemented. Model A was accepted and shipped: mutable values move, borrows are checked, and copying is explicit. The ownership reference owns the contract. This superseded the original ownership proposal, which copied collections on update; that page was removed. Model B below is the rejected alternative.
Shipped beyond this text: returned references, field and element projections, reference rebinding, borrowed matching, user destructors (
__del__), andBox[T]as a public owner type. An&Tparameter borrows an owned argument implicitly. Allocating operations, includingcopy, returnResult.Remaining: stored references, relationships among several reference parameters, and disjoint-index loans (B11).
Motivation and current costs
Before this migration, assignment retained a collection handle. Mutation then copied the collection’s entries and retains nested values before changing the copy. The old append and indexed-update paths did this even with only one language-visible owner, making repeated appends quadratic. Updates now happen in place; explicit copy recursively duplicates mutable contents. Assignment, arguments, indexing, and enum payload projection all need review together; changing assignment alone is insufficient.
We want collection duplication to be explicit, while retaining fast compilation,
typed function contracts, automatic reclamation, and one public str type.
Python assignment binds a name to an object; it does not copy integers or strings either. Immutability makes sharing those objects safe. Plenty can preserve that observable behavior while storing fixed-width scalars directly and sharing immutable string storage. String rebinding must remain distinct from mutation of the string’s bytes. An exclusive reference to a string binding could replace that binding without making the string object mutable.
Two coherent models
A. Transfer mutable values; borrow explicitly
- Fixed-width scalars copy cheaply. Immutable
strstorage can be shared without copying its bytes or introducing another public string type. - Assignment and owned arguments transfer collections, generators, and aggregates that contain owned mutable values. Reading a transferred binding is an error.
&Tlends read access, and&mut Tlends exclusive write access. The owner survives a borrow but conflicting access is prohibited until its last use.copy(value)explicitly produces an independent value. For nested mutable collections this recursively duplicates owned mutable contents; immutable strings may share storage. This differs from Python’s shallowcopy.- Resource types such as generators have no general copy operation.
For example:
def append_three(values: &mut list[i64]) -> Result[(), AllocError]:
values.append(3)?
return Ok(())
def main() -> Result[(), AllocError]:
mut original = [1, 2]?
mut changed = copy(original)?
changed.append(3)? # original remains [1, 2]
append_three(&mut original)? # changes original without copying it
mut destination = original # transfers ownership
# print(original) # error: moved binding
return Ok(())
The copy builtin above takes a temporary shared borrow. Ordinary functions expose
their borrowing or ownership contracts in signatures; the compiler does not infer
cross-function contracts by examining implementations. mut permits writing
through an owner or exclusive access path, not creating another unrestricted
writer. References to whole bindings and references to their collection elements
are different places and must be checked accordingly.
This is the accepted direction for static checking and explicit costs. Its
main usability cost is that collection assignment does not preserve both names.
It needs no lifetime syntax. Box[T] later became the one public owning pointer,
for recursive data.
B. Shared mutable objects, like Python
- Collection assignment and arguments retain handles to the same mutable object. Both names remain usable; mutations are visible through every alias.
copy(value)can naturally be shallow, like Python; nested mutable objects remain shared. Full independence then needs a separate deep-copy operation.- Binding rebinding permissions, permissions to mutate an object through a handle, and guarantees that an object cannot change must be distinguished. An immutable binding does not by itself make a shared object immutable.
- Reference counting handles object lifetime, but does not establish exclusive access or prevent references into relocated buffers from dangling.
For public references into shared mutable collections, choose an explicit policy: conservatively reject conflicting accesses through all possible aliases, check borrows at runtime, or use stable handles that resolve accesses without exposing raw element addresses. Stable handles still require rules for removed/replaced elements and do not themselves provide exclusive access. A handle must not lower to a native exclusive reference without establishing exclusivity.
Runtime borrow guards are a practical implementation for unrestricted shared owners plus temporary direct references. Aliases may coexist; access that conflicts with a live guard fails at runtime. This adds checks and potential runtime errors, including in single-threaded code with nested function calls. It is not a general compile-time borrow-checking guarantee. A static alias analysis can remove some checks or reject more programs, but should not require whole-program analysis to understand a function’s contract.
This model is viable if preserving Python’s mutable aliasing is the priority. Future recursive structs would also require a policy for ownership cycles; reference counting alone would not reclaim them.
Migration requirements
- Define operations on source places in a typed control-flow graph before native lowering. Record reads, writes, moves, borrows, reborrows, and destruction.
- Separate lifetime ownership from permission to access. Internal retained temporaries and reference counts are not proof that a borrow is exclusive.
- Specify argument evaluation and temporary loans, branch joins, loops, and early exits. Use last-use liveness rather than ending every loan at block end.
- Define indexing and match projection without silently copying mutable payloads. Initially reject unsupported moves out of indexed storage; add explicit removal operations separately. Match may borrow a scrutinee or consume it, with field cleanup determined by that choice.
- Define iteration under mutation. Model A can borrow the collection during a loop; model B needs guarded iteration, invalidation detection, or documented mutation behavior. Do not preserve hidden snapshots by copying implicitly.
- Keep public references out of stored fields, returned values, and suspended generators in the first implementation. Add each only with an explicit lifetime contract; reject local references that escape or cross an unsupported yield. (Returned references have since shipped under a single-reference-parameter rule.)
- Test conflicting alias access, collection resizing with live element borrows, reference reassignment, reborrows, branch-dependent moves, early exits, and cleanup. A source check must run independently of Cranelift verification.
These requirements are met: the borrow checker runs on a source control-flow graph with last-use loans, before native lowering.
Sources
The selected model borrows specific ideas rather than adopting either language’s complete ownership model.
One immutable str with explicit lengths
Status: Implemented. One immutable
strwith byte and scalar lengths shipped. Text owns the source contract and value representation the layout.Shipped differently: a string of at most seven bytes is stored inline in its value word, so a
stris not always a pointer.+returnsResult[str, AllocError]. Indexing and iteration allocate nothing. Counts are atomic. The runtime is Rust (plenty-runtime/src/strings.rs), so the C files and helper signatures below are historical. Line input isinput() -> Result[Option[str], IoError]. Slicing,replace,strip,join, andstr.reprwere added later.Shipped since this proposal: std.bytes provides owned binary buffers with explicit fallible allocation, and byte literals provide immutable
bytesvalues. Remaining: broader C text/buffer representations (B27). Language-level slice syntax shipped with issue #579.
Language contract
There is one public string type, str. It is an immutable sequence of Unicode
scalar values, stored as valid UTF-8. Assignment and passing arguments preserve
independent value semantics while sharing immutable storage internally. mut
permits replacing a binding; it does not make the string’s bytes mutable. No
public owning-string/view-string distinction, implicit byte array coercion,
nullable string, or string-specific lifetime parameters are needed.
Preserve the existing surface: literals, concatenation, equality, len, integer
indexing, membership, iteration, typed arguments/results, and collection keys.
len counts Unicode scalar values, not bytes or grapheme clusters. Indexing
returns a one-scalar str, supports negative indices, and reports an out-of-range
index at runtime. A combining mark is its own scalar. Neither equality nor
hashing performs Unicode normalization. Surrogate code points are invalid.
Embedded U+0000 is ordinary content. Add the \0 literal escape and allow a
literal NUL character inside a quoted string. Default storage has no trailing
NUL byte, and no operation searches for a terminator. This requirement applies
to static literals, empty strings, concatenation, indexing, and input alike.
Additional escape forms and methods can be added independently; this change
need not introduce slicing, formatting, case conversion, or a public bytes
type.
Representation and ABI
Keep Ty::Str represented by one non-null machine-word pointer in Cranelift and
collection slots. The pointer addresses an immutable allocation containing a
common reference-counted object header, two unsigned 64-bit lengths, and inline
UTF-8 bytes:
PlentyObject header # {u64 refs, void (*destroy)(void *)}
u64 byte_len
u64 scalar_len
u8 data[byte_len] # exact logical payload; no terminator
Use the ownership proposal’s common header verbatim: {u64 refs, void (*destroy)(void *)}. Offsets are byte_len=16, scalar_len=24, data=32.
The callback lets the base runtime release objects without linking against
optional collection or enum runtime code. Do not put a
second reference count in the string header. Runtime C owns layout details;
compiler literals must use the same documented offsets and 8-byte alignment.
Use a shared compiler-side layout definition and C static assertions to catch
drift. Current targets are 64-bit; reject unsupported targets instead of silently
serializing a mismatched header.
Heap strings start with one owned reference. Static literal headers use the
common immortal count sentinel (UINT64_MAX) and a null callback;
retain/release must inspect the sentinel
before attempting a write because literal data is read-only. Scalar count for
literals is computed once by the compiler. The literal’s header and inline data
can occupy one read-only object symbol, requiring no data-pointer relocation.
The empty literal is a complete aligned header with both lengths zero.
The representation deliberately preserves PTR_TY, clif_type(Ty::Str), source
function signature widths, local SSA variables, return values, and tail-call
signature compatibility. A two-register (pointer, length) source ABI would
force a much broader compiler rewrite without giving users an additional
capability here. No monomorphization or Unicode library is needed.
Check byte-count, scalar-count, and allocation-size addition before allocating.
Keep logical lengths at most i64::MAX, matching len and index types. Refcount
overflow must trap; it must never wrap into zero or the immortal sentinel.
Runtime interface and ownership
Use ordinary C helper signatures taking const PlentyStr * and returning
PlentyStr * where appropriate. All observing arguments below are borrowed for
the duration of a call. Returned strings carry one owned reference. Generated
code releases its owned input temporaries after an observing helper returns;
helpers do not unexpectedly consume their inputs.
| Helper | Behavior |
|---|---|
plenty_concat(a, b) -> str | Allocate exact combined byte length, copy bytes, add cached scalar counts |
plenty_str_eq(a, b) -> i8 | Compare byte lengths, then memcmp; pointer equality is a fast path |
plenty_contains(haystack, needle) -> i8 | Bounded byte-substring search; empty needle is always present |
plenty_println(s) | fwrite(data, 1, byte_len, stdout) then newline |
| internal string length helper | Return cached scalar count |
| internal scalar index helper | Normalize signed index without overflowing; scan UTF-8 boundaries; return a fresh one-scalar string |
| internal string hash helper | Hash exactly byte_len bytes, including NUL, using existing hash algorithm |
Runtime helpers may return a retained existing string for empty concatenation or other safe optimizations later; callers must already treat results as owned. Strings have no child references, so releasing their last reference frees the allocation directly. Collection insertion retains stored string keys/values; projection retains the returned string; collection copying retains all copied children; collection destruction releases them. Enum string payloads follow the same contract. A string stored in a suspended generator is just another owned frame field, and yielded strings obtain an independent owned reference.
String equality and hashing must be changed together. Otherwise embedded NUL would corrupt dictionary/set key equivalence. Keep descriptor C strings as compiler-internal NUL-terminated metadata: those are not Plenty string values.
Strict validation rejects overlong encodings, truncated sequences, stray continuations, UTF-16 surrogates, and values above U+10FFFF. Compiler literals are already valid Rust UTF-8; concatenation and scalar extraction preserve the invariant. Validation is needed at external input boundaries only.
Iteration cost and cursor interface
Existing CollectionOp::IterGet(Ty::Str) uses a scalar index and rescans from the
start. Simply replacing terminator checks with byte bounds fixes correctness
but leaves full iteration quadratic. Do not conceal this cost in the design.
For linear string iteration, use a compiler-private byte cursor. The loop keeps
an owned snapshot of the source, a byte offset, and the current one-scalar
string. Its condition compares byte offset with byte length; scalar extraction
copies the one-to-four bytes at that boundary. Its step adds the current
scalar’s byte width. All three iteration consumers need this path: source
for, comprehension clauses, and list/set construction from iterables.
continue must execute the same cursor step before jumping to the condition.
The byte cursor is an implementation detail; ordinary s[i] remains indexed
by scalar number.
Suggested private operations are TextByteLen(str) -> i64 and
TextAtByte(str, i64) -> str; they fit the existing fixed collection runtime
ABI. Existing scalar Get(str) remains separate. Every internal byte access
must enforce bounds and a UTF-8 boundary, even though well-formed generated
loops satisfy both. An alternative unified iterator object is appropriate if
the generator implementation already supplies one; it should own the string
snapshot and byte cursor, with exactly the same semantics.
For the smallest coherent first patch, retain scalar-index iteration, document its cost, and land cursor iteration immediately afterward. This is a performance staging choice, never a reason to retain NUL-terminated storage.
Exact integration points
runtime/plenty_runtime.c: define the common-header string layout, bounded helpers, strict input validation, and allocation/destruction integration. Removestrlen,strcmp,strstr, andfputsfrom all language-string operations.fputsremains fine for literal diagnostics.runtime/collections.c: replace string casts and scans incollection_hash,collection_value_equal,collection_print_value,collection_text_length,collection_text_at, and the descriptor-sdispatch. Delegate to shared string helpers, avoiding two inconsistent UTF-8 implementations. The existing runtime concatenation build places the base runtime before collections, so shared helpers need no separate link step.src/codegen.rs: changedeclare_str_dataanddeclare_eof_empty_strto serialize complete aligned headers plus exact bytes in native target byte order. Update runtime comments and read-line fallback assumptions. Retain existing helper signatures at the CLIF level. String-pattern comparisons automatically use the new equality helper.src/frontend.rs: decode\0and remove the in-quote literal NUL rejection. Do not loosen identifier or unquoted source token rules. Alias/type rules remain unchanged.src/frontend/collections.rs,src/collection.rs, andsrc/codegen/collections.rs: add the optional byte-cursor lowering described above. Ordinary scalar indexing signatures remain unchanged.- The ARC integration pass of the original ownership proposal, since removed, inserts retain/drop at local loads, overwrites, temporaries, calls, returns, and control-flow exits. Do not ship partial retain/drop insertion that can prematurely free strings still reachable through a collection, enum, frame, or local binding.
- Update the language reference to describe the string representation; teach
one immutable
str, scalar indexing, and embedded NUL in the tutorial with runnable examples. Update existing NUL-rejection diagnostic tests.
Validation gates
- Static, empty, concatenated, returned, mutually tail-called, and nested collection strings preserve their contents through native compilation.
len("a\0b") == 3; indexing1returns"\0"; raw printing preserves the NUL byte; substring search can find text after or spanning it."a\0b" != "a\0c"; dictionary/set keys distinguish those values and distinguish"a"from"a\0".- Empty-needle search, empty haystack, longer needle, final-byte matches, and equal-prefix unequal-length comparisons never read past payload bounds.
- One-, two-, three-, and four-byte scalars; combining characters; negative
indexing; both index boundaries; empty iteration; nested comprehension
iteration;
continueandbreakall work. - Input accepts valid UTF-8 including NUL and rejects each malformed encoding class, while preserving existing EOF and CRLF behavior.
- ARC tests cover strings surviving source-binding replacement, collection projection, enum matching, yield/resume, and early return; repeated string concatenation under AddressSanitizer/LeakSanitizer catches ownership leaks and use-after-free. No writes to immortal read-only headers.
- C runtime builds with warnings denied; formatting, Clippy, full tests, and executable tutorial examples pass. ASan/UBSan cover bounded operations using payloads deliberately lacking any accessible terminating byte.
Foreign interfaces
FFI must explicitly distinguish (pointer, byte_length) APIs from APIs requiring
terminated C strings. Exporting to the latter creates a temporary terminated
buffer and rejects embedded NUL, with lifetime tied to the call unless ownership
is explicitly transferred. Importing C text validates UTF-8 and copies into the
normal representation. FFI declarations and library loading must not dictate
the default language string layout or add another ordinary string type.
Foreign text borrows shipped
the exporting half as &str as utf8 and &str as c_string. Importing C text
is B27.
Sum types and enums
Status: Implemented. Enums, matching, and sums and propagation own the contract.
Shipped differently: enums are inline values and construction allocates nothing; this page proposed heap tagged records. Bare
Some,Nothing,Ok, andErrtake their type from context, and?exists. Unit payloads such asResult[(), E]are accepted. Recursive enums are accepted throughBox[T]or a collection. An ownedmatchmoves an affine scrutinee, and borrowed matching exists. Generic enums andVariant.new(...)were added. Patterns nest, combine with|, and take no type arguments. Integer,str, andboolliterals are patterns, andmatchaccepts those types as scrutinees; see matching. The examples’ module-scope statements are no longer accepted.Omitting a user-defined enum’s qualifier was declined: a bare name is always a binding and a qualified name is always a variant.
Shipped since this proposal: enum methods and protocol satisfaction, human-readable reports, and general anonymous sums, and bounded large dense integer/tag jump tables (issue #395; measurements).
Remaining: deep operations on recursive enums (B33). Guards, enums as dictionary keys or set elements, and
matchas an expression are unscheduled.
Contract
Add nominal concrete enums, compiler-known Option[T] and Result[T, E], and
exhaustive match suites. Preserve one native value per Plenty value and existing
independent-value semantics. This does not require structs, user generics,
traits, dynamic dispatch, a new optimizer, or a borrow checker.
enum Reading:
Missing
Value(i64)
Invalid(str)
def describe(reading: Reading) -> str:
match reading:
case Reading.Missing:
"missing"
case Reading.Value(value):
if value < 0:
"negative"
else:
"available"
case Reading.Invalid(reason):
reason
reading = Reading.Value(42)
print(describe(reading))
An enum declaration is module-scoped and contains one or more variants. Variants
have either no payload or a fixed sequence of statically typed positional
fields. Construction always names the enum and variant. Nullary variants are
values (Reading.Missing); reject Reading.Missing() with a useful diagnostic.
Payload variants require exactly the declared argument count and types. Arguments
evaluate left to right. No implicit integer widening is introduced.
Reserve enum/type names in the same declaration namespace as aliases and built-in type names. Variant names need only be unique within their enum; they do not introduce globals. A binding must not silently change a qualified type constructor into a method call: resolving a type-qualified expression uses the type namespace and rejects ambiguous shadowing, consistently with existing type-constructor rules. Enums do not have methods in this slice.
Type identity and resolution
Two enum declarations with identical variants remain different types. A type alias
is transparent: type Input = Reading names the same enum and may qualify its
constructors and patterns. Forward references to aliases and nonrecursive enum
types are accepted, following the existing module-wide alias behavior.
Collect declaration names before resolving any field types. Resolve a dependency graph containing aliases and enum fields, with explicit visiting/resolved states. Reject recursive enum declarations for the first slice, including recursion through collections. Give the dependency path in the error. This intentionally avoids accidentally promising recursive destruction, cyclic metadata, or mutable cyclic values; recursive sum types can follow with separately bounded implementation.
Ty::Enum(Rc<EnumDef>) is a small integration option for the current compiler.
EnumDef contains a compilation-local nominal ID, display name, and resolved
variants with field types. Its equality and hash use the ID, not structural field
equality. Built-in instantiations are interned by constructor and concrete type
arguments, so independently resolving Option[i64] produces the same identity.
The resolver owns the intern table; code generation only consumes immutable
definitions. A later global type-ID table can replace these references without
changing source semantics.
Option and Result
Use compiler-known type constructors with ordinary enum behavior:
def find_positive(value: i64) -> Option[i64]:
if value > 0:
Option[i64].Some(value)
else:
Option[i64].Nothing
def checked_divide(left: i64, right: i64) -> Result[i64, str]:
if right == 0:
Result[i64, str].Err("division by zero")
else:
Result[i64, str].Ok(left // right)
The variants are Some(T) / Nothing and Ok(T) / Err(E). Nothing is a
variant of one concrete option type, not a universal null value or a None type.
Writing type Lookup = Option[i64] permits Lookup.Some(3) and
Lookup.Nothing, reducing repetition without inference rules.
Require explicit type arguments or an alias initially. In particular, do not
guess the absent side of Result from a constructor argument. Expected-type
inference for abbreviated constructors is an ergonomic follow-up; it is not
needed for a usable type-safe first implementation. No ?, implicit unwrapping,
exception conversion, or truthiness is introduced.
The present frontend represents () as the absence of a stack result rather
than a Ty. Keep that restriction explicit in the first implementation: payload
fields and Option/Result arguments require stored value types, just as collection
elements do today. Thus Result[(), E] initially diagnoses an unsupported unit
payload. A concrete enum Outcome: Success; Failure(E) (written as indented
variants) already expresses that result. Before claiming a complete standard
Result facility, add unit as a first-class value type or erased field metadata;
this must be a deliberate follow-up, not an accidental parser limitation.
Matching and control flow
Introduce Statement::Match { scrutinee, arms } and a frontend pattern AST.
Patterns are qualified variants with bare immutable bindings or _ for each
payload position, plus an optional whole-value wildcard _. The initial slice
does not include guards, OR patterns, nested destructuring, literal payload
patterns, or implicit capture patterns. These restrictions make finite coverage
checking exact and linear in the number of variants and arms.
def unwrap_or(value: Option[i64], fallback: i64) -> i64:
match value:
case Option[i64].Some(number):
number
case Option[i64].Nothing:
fallback
Evaluate the scrutinee exactly once. Each case gets a fresh local-name scope; bindings may shadow outer locals, but duplicate names within one pattern are an error. Restore the outer name scope after each arm and after the match. A binding copies an independent value; matching never partially moves its scrutinee in the first slice. A payload containing a collection therefore retains the existing independent-update behavior.
A variant arm covers that entire variant. A repeated variant is unreachable. A wildcard covers all remaining variants and must be last; diagnose a redundant wildcard after explicit complete coverage. Missing coverage lists missing qualified variants. Constructors and patterns from a different nominal enum are errors, even if their variant names and payload types happen to match.
Use the existing BlockResult::Exits rules: continuing arms must agree in result
type; arms ending in return, break, or continue contribute no join value.
If every arm exits, the match exits and subsequent statements are unreachable.
A match in a function’s tail position yields its continuing arm’s final expression,
exactly as if does today. Elsewhere the match result is discarded. Arbitrary
inline x = match ... is deferred with the same limitation as multiline if;
do not introduce a second expression-block grammar just for this feature.
Loop exits inside a case target the enclosing source loop. A continue in a
for case runs the existing iterator step exactly once before jumping. Preserve
tail-call marking for calls that are final expressions or explicit returns inside
case arms.
The backend already has Op::Match, scalar patterns, an independent match
checker, and Cranelift dispatch/join lowering. Reuse
them rather than synthesizing nested source ifs. Scalar patterns may be exposed
in the frontend at the same time: booleans require both values or _, whereas
integers and strings require _, matching the existing checker. Do not expand
scalar coverage algorithms as part of enum support.
Native representation and operation checking
Initially all enum values use a pointer-sized immutable tagged record. A concrete
record carries a u64 tag followed by one u64 slot per active payload field.
Small integers and booleans use the same pack/unpack rules as collection elements;
strings, collections, and nested enums occupy pointer slots. Tags are assigned in
declaration order and are private to the compiler/runtime ABI. Do not promise a
C ABI, stable cross-version layout, unboxed layout, or niche optimization.
This representation fits current clif_type, function parameters/results, local
variables, branch block parameters, collection storage, and tail calls. It avoids
flattening aggregates throughout the operation stack or adding hidden result
pointers. Allocation is a runtime cost accepted to keep the first implementation
small and compilation fast. Shared immutable records mean assignment does not
copy the payload or permit mutation through another binding.
Add a checked constructor operation carrying enum identity and variant index. It pops the declared fields in order and pushes the enum type. Extend match arms with payload destination slots, or equivalently a checked binding prologue whose field types are derived from the selected variant. Prefer arm-local binding metadata: it prevents an unchecked payload projection from being emitted outside a successful variant test. The independent operation checker verifies identity, tag bounds, arity, destination types, coverage, and each arm’s stack result.
A smaller integration alternative, preferred for the first implementation, stores
the scrutinee in a hidden typed local, lowers a checked enum-tag operation to an
integer, and reuses existing Op::Match with integer patterns. Each selected arm
loads its payload from the hidden local with checked enum/variant/field metadata.
Frontend coverage checking must run before this erasure; ordinary integer coverage
does not know the finite enum domain. Emit an explicit invalid-tag trap fallback
rather than treating an arbitrary integer tag as the final enum variant. Document
the independent checker’s boundary: it verifies projection identity/index/type,
while the structured frontend guarantees the projection follows its variant test.
Native dispatch loads the tag once, selects the arm, and only loads payload fields inside the selected case. A verified exhaustive match still traps on an invalid runtime tag to expose compiler/runtime bugs. Keep a compare chain for small or sparse sets. Sets of at least 256 distinct contiguous integer/tag values use a bounded jump table (issue #395). The B19 measurements support that conservative policy; smaller matches can regress with tables.
Allocation and enum type metadata should live behind a small runtime interface.
An implementation can use generated read-only metadata or per-type compiler
generated callbacks for equality, printing, and destruction. Do not bake display
names into the nominal type identity or require source parsing at runtime.
The shared runtime decision is a 16-byte managed header containing u64 refs and
void (*destroy)(void *), followed by the enum tag and active payload slots.
Immortal objects use UINT64_MAX references and a null callback. Generic retain/
release calls the object’s destructor at its last release; each enum destructor
uses active-variant field metadata. This callback boundary also prevents scalar-
only programs from acquiring link dependencies on every aggregate runtime module.
Ownership/strings implementation owns this header contract; enum support reuses
it. Until reclamation actually lands, explicitly document process-lifetime
allocation as for current collections; tagging a record does not itself implement
ownership.
Collections, strings, and ownership interfaces
Permit enums in list elements and dictionary values, including enum fields that contain collections. Continue to reject enums as set elements and dictionary keys initially, following the current closed hashable-type set. This avoids accidentally committing to public hash/eq traits or deriving hashability.
Current collection descriptors are recursive prefix strings and runtime operations
perform generic equality and printing. Merely adding an enum pointer to Ty is
not enough: list[Reading] must print and compare its active payload correctly.
Extend descriptors with an enum identifier mapped to static variant metadata, or
replace descriptors with shared static type metadata in one coordinated change.
The enum descriptor must encode a concrete identity and field descriptor table;
a bare opaque-pointer descriptor would silently give pointer equality, which is
incorrect for independent values. Do not enable collection-containing enums until
these runtime paths have tests. Equality requires the same nominal type, same tag,
and pairwise equal active fields. No ordering operators are derived.
Printing uses qualified variant names, for example Reading.Missing and
Reading.Value(42), with existing collection element/string formatting conventions.
String payloads reference the single str runtime representation; enums do not
create another string type. Their equality must compare the complete counted
string, including embedded zero bytes, once the string runtime is updated.
All currently admitted enum fields have value semantics. Ownership refinement: an enum remains value-copyable only if all its fields are value-copyable; a future resource-owning or generator payload makes the enum move-only. Initially exclude those payload categories. Shared enum storage owns its fields, so final release destroys only the active variant. Match bindings retain/copy independently; no borrowed pattern bindings, references in enums, or partial moves are promised.
Generator next can return a concrete Option[T], and exhaustion is Nothing.
The enum type/constructor ABI must therefore be usable from compiler-generated
resume functions, without source-level generic functions. Generator state is not
an enum language value and does not need to use this allocated record layout.
Implementation sequence and acceptance checks
- Add enum declaration parsing, name collection, type resolution and interning; diagnose duplicate names, bad type arguments and cycles. Keep this logic in a focused frontend/type module rather than growing the existing expression match.
- Add qualified constructors and typed operation checking; implement immutable record construction, printing, and equality in the runtime/native lowering. Verify enum arguments, results, aliases, and nominal type mismatches.
- Add frontend
matchsuites, payload slots, exhaustive coverage and backend projection. Test all combinations of normal results and early exits. - Instantiate Option/Result through the same enum metadata path. Verify nested options/results, aliases, explicit type argument errors and empty variant values.
- Connect collection descriptors and managed-object ownership hooks. Add nested enum/collection tests before advertising those combinations.
- Update the reference’s implemented/proposed boundary and the tutorial with runnable enum, Option and Result lessons. Its existing executable-guide test checks them.
Required regression cases include multiple fields with different integer widths;
zero-field and string payload variants; constructors with side effects; repeated
matches of the same value; zero bytes in string payloads; nested concrete enums;
independent updates after extracting a collection; list equality/printing with
distinct-but-equal enum allocations; wildcard coverage and arm-local shadowing;
non-exhaustive/repeated/wrong-type variants; missing and excess payload arguments;
wrong payload binding arity; all-exiting and partially exiting matches; break and
continue in nested loops; direct and mutual tail calls inside case arms.
Run frontend diagnostics, native executable tests, the executable tutorial, the full existing suite, formatting and Clippy. Add a large concrete enum/alias input to catch accidental exponential resolution or repeated specialization; do not introduce a specialization optimizer or broad benchmark framework for this slice.
Generics, structural protocols, context managers, and error propagation
Status: Implemented. Proposed 2026-10-05; all four parts shipped. Generics, protocols, resource scopes, and sums and propagation own the contract.
Shipped differently: a protocol requirement ends in
:with apassbody, and the protocol’s own name stands for the implementer; there is noSelf. Generic bodies are checked for each concrete instance, not symbolically against their bounds.?supports lossless anonymous-sum widening and may erase an error toFailureas a deliberate last resort.Fileis a builtin context manager.with &mut existing, returned references, and aResult-returningmainshipped.Remaining: combined bounds, protocol composition, and bounds on data (B29); stored references (B11). Erased
dynprotocol values, fallible__exit__,yieldinsidewith, and?in generators are unscheduled.
The protocol-body proposal records the later agreed default/final method and private-hook design; none of it is implemented by this completed proposal. Error composition connects the shipped sum, mapping, and reporting extensions.
Starting point
At proposal time Plenty had concrete classes and inherent methods, Option[T]
and Result[T, E], checked local/field borrowing, and deterministic cleanup.
__del__ could not fail recoverably. Cleanup already ran on return, break,
and continue. There were no user-defined generics, protocols, returned
references, stored references, context managers, or ?. Built-in parameterized
types were compiler-known special cases, not evidence of a general generic
system.
The recommendations are:
- Add
?directly forResultandOption; it needs no trait system. - Use explicit generic parameters with structural protocol constraints.
- Resolve protocol methods to the concrete type’s inherent methods. Imports identify names; they never activate extension methods or implementations.
- Implement concrete context managers independently of generics. A returned
reference from
__enter__requires additional borrowing rules, however. - Begin with infallible scope exit and explicit fallible resource operations. Defer automatic fallible exit until error composition has a complete design.
Structural protocols with explicit generic parameters
Python protocols permit structural conformance without explicit inheritance. Go interfaces likewise describe sets of implementing types through methods. Both provide useful precedents; Plenty can adopt this idea without inheriting Python’s dynamic model or Go’s runtime interface values. Python typing specification, Go interface specification.
Proposed declarations:
protocol Writer:
def write(self: &mut Self, data: &str) -> Result[(), WriteError]
def write_heading[W: Writer](writer: &mut W, heading: &str) -> Result[(), WriteError]:
writer.write(heading)?
return Ok(())
Self means the concrete implementing type and is a compiler-defined name
inside protocols. WriteError must be declared or explicitly imported.
protocol Writer: follows Plenty’s existing declaration shape. Protocol method
declarations have signatures and no bodies; punctuation remains a parser decision.
A concrete class with an accessible matching write method satisfies Writer.
No registration, inheritance, global implementation search, or implementation
import is needed. A caller need not import Writer merely to call
write_heading; the generic function’s signature already names its constraint.
Calling a concrete object’s write method never depends on whether the caller
imported this protocol. Across module boundaries, required concrete methods must
be pub. Protocol conformance must not expose private methods through a public
generic function.
Begin with required methods only. Defer protocol fields, default methods, inheritance/composition syntax, associated types, generic protocol methods, operator overloading, and recursive protocol constraints. A method-only protocol avoids exposing field representation and mutable-field variance questions. Python supports substantially more, including mutable attributes and generic protocols; its variance rules illustrate the additional complexity. Python generic protocols.
Type checking and ownership
The declaration’s constraints determine every operation permitted in a generic body. Do not accept arbitrary duck typing and discover missing methods only by expanding the first call site. Diagnose a missing constraint at the generic definition; diagnose a failed concrete conformance at the instantiation.
Initially require exact method parameter and return types after substituting
Self and explicit type arguments. Include all ownership-relevant properties:
&Self,&mut Self, and an eventual owned receiver are distinct contracts.T,&T, and&mut Tparameters are distinct. Matching names alone is insufficient, and mutable access cannot substitute for shared access.- Returned references, when supported, include their source/escape contract.
- Fallibility is represented by the exact
Resultreturn type. - Future thread-transfer and thread-sharing guarantees are separate checked capabilities; having appropriately named methods does not establish them.
An unknown T must be treated conservatively as an owned value. It can move,
borrow, and drop; it cannot be silently duplicated because one instantiation
happens to be an integer. An unconstrained generic identity function is useful:
def identity[T](value: T) -> T:
return value
To duplicate a generic value, require an explicit supported copy capability or
a user-provided operation. Allocation failure makes the exact copy API part of
the fallible-allocation design. Do not decide it accidentally in the generic
checker. In particular, a structural copy method is an explicit method call;
it does not silently grant the compiler permission to copy values.
Invariant generic types and exact signature matching are a deliberately small
first subset. More permissive variance or receiver adaptation can be evaluated
later against actual examples. Protocol satisfaction checks shape and types,
not behavioral laws: a method called write might still have incorrect behavior.
Use nominal wrapper types when a domain requires explicit opt-in or distinction.
Compilation and representation
Type-check generic bodies against symbolic constraints, then instantiate only reachable concrete uses. Lower those instances to ordinary typed Plenty IR and Cranelift functions with direct calls. Cache by declaration identity plus concrete type arguments; keep the cache independent of import spelling. Report both the generic definition and instantiation chain in diagnostics.
Monomorphization is simple at runtime but can increase compilation time and binary size. Measure instance counts and time, reuse repeated instances, and diagnose recursively expanding instantiations. A documented implementation limit is preferable to compiler hangs. Ordinary recursion at the same type arguments does not need a new instance. Keep type-level evaluation, specialization, and overload search out of the first implementation.
Initially Writer is a constraint, not a storable runtime type. Write
def f[T: Writer](x: &mut T) explicitly. Defer def f(x: &mut Writer) rather
than letting it ambiguously mean either an implicit generic or dynamic dispatch.
Likewise, a heterogeneous list[Writer] is not available in this first model.
Generic functions can precede generic classes/enums; the latter require explicit
layout, drop, and recursive-type rules.
An erased interface value could later use explicit syntax such as dyn Writer,
with a data pointer and method table. It needs its own ownership/lifetime rules,
representation, and restrictions on methods using Self and generics. It is
not necessary to introduce those constraints for static generic calls. Rust’s
dyn-compatibility rules are a useful checklist for this later feature, not a
requirement to copy its surface design. Rust dyn
compatibility.
? is a small language operation
For Result[T, E], value? evaluates value once, yields its Ok payload, or
returns Err(e) from the enclosing function. That function must return
Result[U, E] with the same error type after alias resolution. For Option[T],
it yields the Some payload or returns Nothing from a function returning
Option[U]. A unit success remains useful: write(data)? is an expression
statement yielding ().
Use postfix precedence so open(path)?.read()? has a predictable parse. Preserve
left-to-right evaluation and full-expression temporary cleanup. Move owned
payloads once; do not copy the enclosing sum or its payload. On propagation,
clean up exactly the scopes that an explicit return would cross, including
partially evaluated expression temporaries and later context-manager exits.
Propagation must not allocate merely to rebuild Err or Nothing. This is an
architectural requirement for recoverable allocation failure, not a property of
today’s heap-backed enum representation. The implementation needs an
allocation-free error carrier or an equivalent verified lowering; reusing an
existing container is insufficient if its type metadata or ownership contract
changes. Error mapping performed explicitly by user code may itself be fallible,
but the primitive propagation path and OOM error construction must work without
allocating. See memory, parallelism, and SIMD.
Rust supplies a familiar precedent but permits conversion on Result error
propagation. Plenty should initially require exact error types and explicit
conversion using match or a later named mapping operation. Do not implicitly
convert Option into Result, infer a new union error type, or add an extensible
Try protocol in this step. Rust try-propagation
expression.
Reject ? in __del__, in unit-returning __init__, and in generators until
generator completion/error semantics are designed. Fatal traps remain aborts;
? is ordinary typed control flow, not exception unwinding. A function entrypoint
returning Result would require a separate startup/status-reporting convention.
Context managers without a trait dependency
Python’s with calls __enter__, binds its result, and arranges __exit__ on
leaving the body. Python also supplies exception information and supports
suppression. Plenty has no Python exception model to reproduce. Python context
manager protocol,
Python with statement.
The compiler can recognize concrete class methods directly, just as it already
recognizes __del__. There is no generics prerequisite. A later structural
protocol can describe this behavior for generic functions without changing
concrete lookup.
Recommended first contract:
# Proposed signatures on a concrete class, not implemented today:
def __enter__(self: &mut File) -> &mut File
def __exit__(self: &mut File) -> ()
# Acquisition reports failure before the context has been entered:
with open_file(path)? as file:
file.write(data)?
file.flush()?
open_file, File, and its methods are illustrative APIs, not present library
features. In this shape, fallible acquisition happens before with, entry is
infallible, and exit performs infallible resource release. Durable writes and
other operations whose failure matters remain explicit Result operations.
Do not pretend that dropping a buffered file proves its data was persisted.
Keep Python’s meaning of as: it binds the result of __enter__, not the manager
by an unrelated special alias rule. Owned entry results and unit entry are also
possible. The common &mut self result is the important missing capability:
the compiler must prove its origin is the hidden manager and prevent it escaping
the scope. It must keep that manager alive, prevent moves/replacement while the
loan exists, make the hidden owner inaccessible while the exclusive borrow is
active, and end the loan before calling __exit__. This should reuse a
general, narrow returned-reference rule rather than invent an unchecked exception
for with. A smaller first slice can support entry methods returning owned
values or () without returned-reference support.
Evaluate the manager expression once and move an owned result into a hidden
mutable local. Supporting with &mut existing can follow after specifying
reborrowing; do not implicitly consume and secretly restore an existing owner.
For every successfully entered context:
- Run the body. Evaluate and preserve any pending return value before cleanup.
- Destroy body-local owned values in normal reverse binding order, and finish
borrows of the manager. Any owned
asbinding is body-scoped too. - Call
__exit__exactly once, then destroy the hidden manager normally. - Continue the pending fallthrough,
return,?,break, orcontinue.
Nested contexts exit in reverse order. Failed acquisition does not enter the
context. If entry later becomes fallible, failed entry must not call __exit__;
normal manager destruction still cleans partially acquired state. __del__
provides fallback release and must recognize an already closed manager so that
exit plus destruction does not release its resource twice. Fatal process aborts
are outside this guarantee, consistently with ordinary destruction.
The compiler’s exit bookkeeping and release path must not require allocation. An infallible exit signature alone cannot prove arbitrary user cleanup avoids allocating: the eventual allocation-failure rules must also constrain cleanup implementations and ensure they can release resources under OOM. In particular, collecting exit errors into an allocating list would defeat recoverability.
Reject a returned/yielded borrow of the hidden manager. Initially reject yield
inside with: retaining manager state and an exit obligation across suspension,
and exiting abandoned generators, need explicit frame support. A later version
can support suspension with owned managers once its loan and cleanup rules are
verified. No async context managers are planned.
Fallible exit is a separate feature
If a context’s exit fails while its body is already returning an error, there
are two errors. Silently overwriting either is a poor default; changing break
or a successful return into a failure also needs an explicit enclosing type.
Python exception suppression does not solve this typed-control-flow problem.
Start with __exit__ -> () and fallible explicit operations such as flush,
commit, or finish inside the block. That gives an honest small feature now.
For a later transactional context, a scoped expression could return an explicitly
composed error enum with Body(E), Exit(F), and Both(E, F) cases. Its shape,
allocation behavior, pending return values, nested exits, and handling of loop
control must be specified before introducing fallible __exit__. Composing
errors must itself work under allocation failure. A context must not implicitly
swallow or transform a caller’s error without visible syntax.
An exit-reason enum distinguishing normal completion, return, and error may be
useful later. For the first contract, transactional success is explicit through
commit()?, and exit rolls back uncommitted state. This avoids assuming that
leaving a block normally is sufficient evidence for a successful transaction.
Milestones and verification
- Add
?for the existing sums. Test nested expressions, unit success, owned payload movement, incompatible errors, and exactly-once cleanup on propagation. - Settle module visibility and the representation of generic declarations. Add unconstrained generic functions, then required-method structural protocols with exact signatures. Measure compile time and emitted instance counts.
- Add the narrow returned-reference contract and its negative borrow tests.
Add concrete
withusing the same exit machinery as existing control flow. Test nested contexts, acquisition failure, return/break/continue/?, escaping borrows, and repeated-close prevention. - Add generic classes/enums and parameterized protocols when practical library examples require them. Keep erased interfaces and fallible context exits as explicit later decisions.
Protocol signature punctuation, returned-reference syntax, and escaping owned entry results are settled. Open decisions are an explicit conformance assertion for library authors and the eventual fallible-exit syntax. None requires an import-sensitive trait lookup system.
Native owned generators
Status: Implemented. Generators own the source contract and environments the frame and resume ABI.
Shipped differently: frames live inline in their owner’s storage and nothing is heap allocated; see allocation-free generator frames. Each producer has its own frame type, and
Generator[T]names only the yield contract.yieldmoves its operand.OptionandResultmay own frames. There is no separate generator control-flow graph; continuation blocks are built during lowering.next(&mut g)was added. A generator body cannot use?, take reference parameters, oryieldinsidewith. The first example’s module-scope loop is no longer accepted.Remaining (unscheduled): frames in collections, class fields, tuples, and user enums;
yield from. Owned generator expressions are now implemented; see the reference.
Source contract
Use the compiler-known concrete type Generator[T] and statement-only yield:
def countdown(start: i64) -> Generator[i64]:
mut remaining = start
while remaining > 0:
yield remaining
remaining = remaining - 1
for number in countdown(3):
print(number)
Calling a generator evaluates its arguments immediately, transfers those values
into a new frame, and returns without executing its body. The first resume starts
the body. Each resume runs until the next yield, bare return, or fallthrough.
Each yield evaluates its operand exactly once and produces one owned value of
the declared element type. There is no implicit yield from a tail expression.
A function containing yield is a generator function and must declare
Generator[T]; ordinary functions may return that type by returning another
generator. Determine whether a function contains a yield syntactically before
checking its body, including yields in unreachable branches. No nested function
syntax currently complicates this scan. An empty generator can contain
if False: yield value (written as the usual indented suite). Do not turn every
ordinary factory returning Generator[T] into a suspended function.
yield requires an expression of exactly T. A generator’s return accepts no
expression and means completion; return value is an error even if value is
itself a generator. Falling off the body also completes it. The initial stored
type restriction excludes Generator[()], consistently with the enum proposal’s
unit-payload restriction. Reject yielding at module scope, yielding in a function
with a different declared result type, and nested generator element types.
Only value-copyable types may be yielded initially: scalar values, the one str
type, current collections, and concrete enums whose fields are value-copyable.
Yielding a managed local retains an independent value; it does not consume that
local. A generator may own another generator as a parameter or local, but may
not yield it. yield from, send, exception injection, generator expressions,
closures, async/await, and public references are outside this slice.
Affine ownership and explicit advancement
Generator[T] is an affine owner of one advancing execution state. Assignment,
argument passing, and return move it. Reading the old binding afterward is a
compile-time error. mut controls replacing or explicitly advancing a binding;
it does not make a generator copyable. Ordinary strings and collections retain
their existing independent-value behavior.
mut numbers = countdown(2)
match next(numbers):
case Option[i64].Some(number):
print(number)
case Option[i64].Nothing:
print("finished")
next is a compiler-known operation on a named mutable Generator[T] binding.
It temporarily borrows that owner exclusively and returns Option[T]. The
borrow ends before subsequent source expressions execute and cannot escape into
a value. This restricted operation needs no public reference syntax, lifetime
parameters, user traits, or general borrow inference. Initially reject
next(countdown(2)), next on an immutable binding, and non-generator operands
with specific diagnostics. Users can name a mutable binding or use a for loop.
Exhaustion is stable: every later next returns Nothing, without rerunning the
body or double-dropping locals. A generator that has completed remains an
available owner until moved or dropped. Dropping an unstarted or suspended
generator releases its captures and locals without executing any remaining user
statements. No implicit user finalizer or exception-unwinding behavior is added.
Reject generators recursively inside list/dict/set elements, enum payloads, and
Option/Result arguments in this slice. Those types currently copy projections,
so admitting an affine field would require aggregate moves and consuming match
semantics. Aliases must not bypass this restriction. No equality, ordering,
hashing, collection indexing, or len is defined on a generator. Printing a
generator should diagnose an unsupported printable type rather than expose its
address or consume it.
Use the ownership proposal’s Available/Moved/MaybeMoved analysis on stable
local IDs. Join only continuing paths. Every reachable loop backedge, including
continue, must preserve availability of outer owners available on entry;
move-then-reinitialize is valid, an unreinitialized move is rejected. Break paths
join the zero-iteration path after the loop. This is deliberately conservative
and should be described as such in diagnostics, not sold as a complete borrow
checker.
Iteration consumers
for item in generator evaluates the iterable once and moves it into a hidden
owner whose lifetime is the loop. Each condition check resumes once. A yield
initializes the loop binding and enters the body; exhaustion exits. continue
goes to that resume condition, and break destroys the hidden owner immediately.
Returning from the consumer also destroys it. Consuming a named generator leaves
the original binding moved even when it yields no values or the loop breaks on
its first item. for does not require the source binding to be mutable because
ownership is transferred to the compiler’s iterator binding.
Comprehensions and currently supported iterable constructors use the same
adapter. For example, [n * n for n in countdown(4) if n > 1] remains eager as a
list construction but obtains its input lazily. Preserve existing left-to-right
clause evaluation and filter behavior. Creating an inner generator expression by
calling a generator function happens each time that clause is entered; trying
to repeatedly move the same outer generator binding across an outer loop’s
backedge is rejected. Do not introduce Python’s tuple-producing dictionary
iterators or a generic iterable protocol as a prerequisite.
Refactor the current collection iteration helper into a private iteration plan:
setup operations, condition, item-binding prefix, step, and cleanup. Existing
collections/ranges keep their snapshot and index step; strings can use the byte
cursor proposed in strings.md; generators have an empty step because resume
belongs in their condition. This removes the current assumption that every
iterator has an integer index to increment. Every consumer must emit the same
step for normal progress and continue and the same cleanup for exhaustion,
break, return, and nested scope exits.
The simplest initial generator adapter calls the typed next helper into a
hidden Option[T] local. Its condition tests Some; its body extracts an owned
payload and drops the option owner before running user statements. Its exit
cleans the final Nothing value and generator frame. This reuses checked enum
operations and avoids inventing a special two-result source operation. A later
optimization can use the native ready flag and output slot directly, without
allocating one option record per iteration.
Native frame and callback ABI
Preserve one pointer-sized native value for Generator[T]. The frame begins
with the shared managed header and stores a concrete resume callback:
typedef uint8_t (*PlentyResume)(void *frame, uint64_t *out);
typedef struct PlentyGenerator {
PlentyObject object; /* { uint64_t refs; void (*destroy)(void *); } */
PlentyResume resume;
uint64_t state;
uint64_t running;
/* Concrete, aligned parameter and local slots follow. */
} PlentyGenerator;
uint8_t plenty_generator_resume(PlentyGenerator *frame, uint64_t *out);
The compiler emits constructor, resume, and destructor functions for each generator function, once per concrete source signature. Constructor and resume callbacks must use signatures/calling conventions compatible with the C runtime; the callback ABI is an internal implementation contract, not an FFI promise. The constructor allocates one concrete frame, zeroes its managed ownership slots, moves parameters into their slots, sets entry state, and returns an owned frame. It executes no body operations. Affine frames are never retained by source copy operations even though they share the common destruction header.
resume borrows the frame exclusively. Returning 1 writes one owned value to
out; returning 0 does not initialize out. Pack integers and pointers with
the same rules as collection slots. The caller must never read or release out
on the completed path. A helper checks the completed state and a running flag,
calls the concrete resume callback, and clears the flag. The guard diagnoses an
invalid reentrant resume rather than corrupting the frame; public callbacks or
concurrency are not introduced by this guard.
A generated typed wrapper converts this protocol into the existing concrete
Option[T] constructors for source next and the initial iteration adapter.
On a yielded path, construction borrows the owned output, retains a stored
managed payload, and then releases the temporary output owner. On the completed
path, it constructs Nothing. Keep this wrapper with enum-aware lowering so
the base runtime does not acquire an unconditional dependency on enum allocation
or metadata. The wrapper borrows, rather than consumes, the generator owner.
Assign dense continuation states in source control-flow order; reserve entry
and completed states. Each yield writes the successor state, transfers the
yielded owner to out, and returns 1. Completion releases live frame owners,
clears their slots, records completed state, and returns 0. The generated
destructor releases remaining non-null owner slots in reverse scope/declaration
order and frees the allocation. The same destructor handles unstarted,
suspended, and already completed frames without needing a distinct callback per
suspension point. Internal null ownership slots are not nullable source values.
Typed control flow without a coroutine runtime
Do not suspend the C stack, keep Cranelift SSA values across native calls, use
ucontext/threads, interpret the body, or collect all yields eagerly. Flatten the
typed structured body into explicit basic blocks for the generator resume body.
Keep the existing structured operation path for ordinary functions initially.
Introduce a small typed generator CFG with source spans, local IDs/types,
ordinary operations, and terminators for jump, branch, match, yield, and complete.
Flatten Op::Loop into header/body/exit, preserving the existing explicit step
operations emitted before continue. Translate Break/Continue using an
innermost-loop target stack, and translate Op::Match arms and joins directly.
This also gives ownership checking and later borrow analysis a reusable place
to record edge cleanup, without requiring a new optimizer or immediate migration
of every non-generator function.
Require yield to leave an otherwise empty expression operand stack. As a
statement, its argument is completely evaluated and consumed before suspension.
The resume CFG can then split at yields without spilling partial expression
evaluation or callee stack frames. Source calls remain ordinary native calls;
only a call to a generator constructor returns a suspended frame.
For the first implementation, give all generator locals frame-backed slots, including hidden iterators, builders, parameters, and shadowed locals. This avoids liveness analysis, SSA reconstruction across resumes, and error-prone copy-in/copy-out ownership. A managed local load retains an expression owner; a move clears its slot; a store releases the old value and transfers the new owner. Scalars use the same frame storage without reference operations. Control-flow definite-initialization checking still rejects source reads from uninitialized locals. Managed slots start internally null for safe cleanup on paths where their declaration never executes.
Emit a state dispatch at resume entry to the entry or saved continuation block. Emit all frame loads/stores in that invocation normally; local expression SSA values need survive only until the current yield. Scope-exit cleanup happens on normal edges, breaks, continues, and returns before their destination. A yield is not scope exit: active owners remain in the frame. Locals dead across a yield may remain in fixed frame slots initially; frame-size/liveness optimization is a later improvement. This is finite per-generator storage, not accumulating one native stack frame per yielded item.
The independent checker must validate yield type, empty residual operand stack, frame slot type/ownership, valid continuation IDs, block-entry operand shapes, and initialized local use. It must reject affine duplication and a yield/complete terminator outside a generator body. It should validate all edges, including resume entry edges, rather than assuming structured source alone proves safety. Generated scope cleanup and definite-move facts must survive CFG conversion.
Calls from a generator before a yield are not tail calls: its frame must remain available for later resume. Ordinary functions forwarding a newly constructed generator can retain existing tail-call behavior. Do not disable ordinary direct or mutual tail calls merely because generators exist in the same module.
Implementation sequence
- Finish concrete Option/enums, common managed headers, complete value ARC, and deterministic scope cleanup. These are correctness dependencies for frames, not requirements for Rust-like references or a trait system.
- Add
Generator[T], recursive placement validation, syntactic generator classification, typed yield/bare-return diagnostics, and affine place analysis. - Add the typed resume CFG and independent checker. Test flattening of nested loops, match exits, and source locations before connecting native frames.
- Add frame layout, constructor/resume/destructor emission, runtime dispatch,
and typed
nextwrapping. Keep frame locals in memory initially. - Refactor the private iteration plan and wire generator
for, comprehensions, and supported collection constructors through it. Preserve existing snapshot iteration, string behavior, and loop control tests. - Add runnable tutorial lessons and mark only delivered behavior implemented in the reference. Record lazy effects, affine consumption, stable exhaustion, and the deliberately unsupported features explicitly.
Acceptance tests
- Construction performs argument side effects once but no body side effects; first and subsequent resumes perform only effects before the corresponding yield. Drop of an unstarted or suspended frame never executes later prints.
- Zero yields, conditional yields, multiple sequential yields, yields in nested
loops/match arms, early completion, and repeated exhausted
nextall behave correctly. A long generator uses bounded native stack depth. - Iteration evaluates its source once; nested generators, filtered and nested comprehensions, break/continue, and consumer return resume the right frame and destroy only owners leaving scope.
- Strings with embedded NUL, nested collections, and enum payloads survive yield and subsequent frame mutation/destruction. Updating a yielded collection does not change the frame’s retained independent value.
- Diagnose use after assignment/call/return/iteration moves, one-branch moves,
repeated moves across backedges, immutable
next, unknown element types, generator-valued yields, forbidden aggregate placement through aliases, and valued returns from generator bodies. Accept move followed by reinitialization and exclude returning branches from joins. - Test frame destruction before first resume, at every suspension state, after
full exhaustion, after
break, and after consumer return. Include a frame owning another frame and shadowed managed locals on mutually exclusive paths. - Test dynamic allocation/drop counts plus ASan/UBSan where available; correct printed output alone does not catch leaks, duplicated ownership, invalid out slot reads, or double release after exhaustion. Static literals/metadata are excluded from dynamic counts.
- Keep native compilation, existing non-generator tail calls, all executable tutorial examples, formatting, Clippy, and the full suite passing. Track compilation cost on a function with many yield points; generated code/frame metadata must grow linearly with that concrete body’s size.
Allocation-free generator frames and ranges
Status: Implemented (B25, B26). Generators and ranges own the contract, and environments the frame ABI.
Settled since: the “original design questions” below are answered. A generator parameter or annotation carries a hidden concrete producer type. A factory must resolve to one producer; no sum or boxed form joins different ones. Limits are 64 nesting levels, 256 consumer specializations, and 32-bit frame offsets.
Remaining: generators in collections or class fields (unscheduled); a public iterator protocol (B12).
Objective and boundary
Creating a generator should require no heap allocation for its execution frame. The compiler already emits a resumable native state machine; replacing that lowering is unnecessary. The change is how its state is represented, owned, passed, and returned.
This guarantee does not cover allocations performed while evaluating arguments
or running the body. A generator may own a list or file, yield an allocated
value, or execute fallible operations. Those operations retain their existing
error contracts. Collecting a generator into a list still returns Result.
Proposed construction returns the generator value directly, without an
AllocError wrapper. Body execution remains lazy. next returns Option[T],
and exhaustion, consuming iteration, moves, and early cleanup retain their
current semantics. This is a representation guarantee across supported uses,
not an optional optimization limited to immediately consumed generators.
Concrete types and source annotations
The previous Generator[T] identified only the yielded type. A pointer to a heap
frame let unrelated bodies, with different frame sizes, use that one type.
Inline storage needs to preserve more information.
The recommended design gives each generator body and concrete specialization
its own compiler-inferred type and finite layout. Users would still declare
-> Generator[T] on a yielding function; this describes its yield contract,
while the function identity determines the concrete returned frame. No user
needs to spell a generated type name.
The concrete-type design is implemented. The alternative was keeping a uniform, size-erased owned type with fallible allocation; that cannot provide the same general inline-storage guarantee.
The original design questions were:
- A factory returning
Generator[T]should resolve to one concrete frame shape for each specialization. Returning different shapes on different branches requires a finite explicit sum or a future boxed form, with no hidden boxing. - Parameters written as
Generator[T]could introduce a hidden concrete type parameter, specializing consumers by frame identity. Alternatively, require an explicit generic iterator constraint. Choose one rule consistently for values, references, aliases, and nestedOption/Resultannotations. OptionandResultcontaining a generator must preserve its concrete layout. The existing scalar-sized wrapper representation cannot simply retain a pointer to a callee’s expired stack frame.- Distinct generator bodies are distinct types even when they yield the same element type. Assignment, branch joins, and return diagnostics must explain this without exposing compiler-generated names as required source syntax.
Storage, moves, and destruction
Use caller-owned result storage for returned frames and a compiler-computed size/alignment for each concrete layout. Frames live inline in their owning locals, arguments, wrappers, or enclosing generator frames. The initial scope need not lift existing restrictions on generators in user collection/class storage.
Nested frame layouts form a dependency graph. Detect recursive inline layouts and explain the need for indirection; do not reserve a global maximum frame size or quietly allocate. Cache layout and specialization work to keep compilation predictable.
Moves transfer ownership exactly once. A moved frame cannot retain absolute pointers into its previous storage. Continue rejecting references across suspension; compute accesses relative to the active frame. Revisit tail-call eligibility whenever an argument points into storage that the tail call removes.
Keep initialized-state information sufficient to drop captures and suspended
locals on exhaustion, break, return, propagation, or explicit drop. Destroy
inline nested frames without freeing their storage as heap objects. Resuming
after a move must use the new owner, and repeated exhaustion must stay stable.
How ranges fit
An allocation-free generator can implement range traversal, but a generator
itself is single-use and has no length, indexing, or membership operations.
Preserve these existing range capabilities with a small inline range[T] value
holding its bounds and signed step. Each traversal creates a concrete generator
from copied scalar fields, with no captured reference to the range.
Range construction and the existing indexed traversal are allocation-free without generator changes. Sharing a future generator implementation is an optional follow-up, not a prerequisite.
Using such a generator would require an iteration dispatch rule shared by for, comprehensions, and
collection constructors. A public iterator protocol is a separate API decision;
do not require one merely to remove the existing builtin allocations.
Typed range traversal must preserve unsigned bounds, descending signed steps,
termination near integer limits, and once-only argument evaluation. A naive
current += step generator is insufficient: even the final unused increment
can overflow. Source-level implementation may need checked arithmetic helpers.
Zero-step and length-limit policy are separate from allocation failure; today’s
runtime rejects both. Do not silently change those rules with this refactor.
Evidence required
Disable heap allocation and exercise scalar-only generators through creation,
resume, moves, factories, consumers, nested generators, and Option/Result.
Verify cleanup exactly once before first resume, after suspension, on exhaustion,
and on all early exits. Include compile-fail cases for incompatible frame types
and recursive layouts, plus native memory checks for stack lifetime errors.
For ranges, cover repeated and nested traversal, indexing/membership/length, all integer widths, both step signs, maximum bounds, empty ranges, and unchanged invalid-input diagnostics. Check frame sizes and compilation cost as well as runtime allocation counts. Update the reference and runnable lessons only when the corresponding behavior is implemented.
Callable surface review
Status: Implemented (B31). Function values, closures, and consuming and stored closures own the contract.
Shipped since this review: explicit fallible shared-call erasure supports heterogeneous callback storage. Mutable/consuming erased modes and generator-containing stored environments remain B32; stored borrows remain B11.
Plenty needs callable values for callbacks, small local functions, and owned resource handoff. This review separates representation from ownership and avoids making every capability a prerequisite for writing an ordinary closure.
Decisions
| Surface | Concrete need | Decision |
|---|---|---|
def(...) | A multiline function expression | Keep; the same typed function body as a named function |
def [captures](../...) | State without implicit sharing or allocation | Keep explicit captures; infer the environment type locally |
value, mut value, &value, &mut value | Own or borrow state, with ordinary mutability | Teach as the existing ownership and mutability rules composed together |
Callable[[A], B] as a value type | Store different named functions in one registry | Keep the code-address representation; it must not silently box captured state |
F: Callable[[A], B] | One reusable callback API for named and captured functions | Prefer this for generic consumers; infer F at calls |
Closure[[A], B] | Describe a factory’s concrete environment result | Keep for factories and specialized interfaces; remove it from the default consumer lesson |
def once [captures] | Transfer captured owners out of a callback | Keep the explicit consuming contract; do not infer it from body changes |
OnceClosure[[A], B] | Describe a factory returning a consuming environment | Keep as an advanced factory annotation |
F: OnceCallable[[A], B] | A task API that accepts all call modes and consumes its callback | Keep as an advanced constraint, never a stored value type |
There are two representations: a code address and a concrete environment. There are two invocation contracts: borrowing an environment for reuse and consuming it. The public annotations describe these differences; they do not introduce four interchangeable forms of dynamically dispatched object.
Collapsing every value into Callable would require either a uniform erased
environment with explicit storage, or a type spelling that sometimes denotes a
code address and sometimes a producer-specific layout. Neither makes storage
costs easier to predict. Keep the distinction visible at interfaces while letting
ordinary local bindings infer it. OnceCallable could not be an ordinary
method-only protocol today: function-call syntax and capture transfer remain
compiler-checked operations. Ordinary protocols can now require a by-value
self, but that alone does not describe callable invocation and captures.
Inferring once from moving a capture out would make a body edit change whether
callers can invoke it again. An explicit word gives a local ownership contract,
like the difference between an owned parameter and a reference. A resource
handoff needs that distinction; an ordinary read-only callback does not.
What belongs in libraries
A registry of named callbacks needs only a collection of Callable values.
A callback plus explicit state can already be a generic class with ordinary
methods. Filtering, dispatching, and adapting these callbacks are library code;
they need no new invocation syntax or builtin registry type.
Concrete owned environments now use ordinary
typed owner slots. Heterogeneous erasure still needs a separate representation and
lifetime contract before library code can build an arbitrary closure registry.
The teaching path is named function values, then ordinary inferred closures, then a reusable generic callback API. Factory annotations, borrowed captures, and consuming callbacks are optional extensions for concrete use cases. The functions chapter marks that shorter path. Existing advanced lessons remain referenceable, so simplifying the entry path loses no examples.
This review does not claim that the current names must be permanent. It records why each retained distinction is necessary today, and replaces unnecessary consumer-specific annotations with the common generic interface.
Stored closure environments
Status: Implemented. This records B32’s storage decision. Consuming and stored closures owns the contract and environments the layout.
Shipped differently: classes, tuples, and enums are inline, so only a collection owner allocates. Environments that contain generators cannot enter heap storage.
Shared-call erasure implemented: explicit fallible BoxedCallable addresses heterogeneous shared-call storage (issue #25).
Remaining: mutable/consuming erased modes and generator-containing environments in heap owners (B32); escaping borrowed captures (B11).
A registry needs an owner for callback state, a call signature, and a cleanup contract. The existing generic class mechanism can express that owner when each instance has one known environment type. Closure bodies already have concrete identities and layouts. Preserve those identities when inferring a generic field or collection element, and place the environment directly in that owner’s slot. There is no separate environment allocation and no dynamic dispatch. Growing a containing collection retains its ordinary fallible contract.
Two calls to the same closure factory produce compatible environments. Different
closure expressions remain different types, even with identical signatures. A
finite family of environments can use an ordinary enum. A registry whose members
must have arbitrary unrelated environments needs a separate erased representation;
silently changing Callable would hide allocation and alter its thin-function ABI.
Owned boxing would give heterogeneous registries a fixed-size element but require
fallible allocation, a call/drop table, and an explicit ownership-transfer contract
on failed construction. Bounded inline erasure avoids that allocation but exposes
a size/alignment budget, rejects oversized environments, and reserves that budget
for every entry. Neither cost is justified for a homogeneous registry. Issue #25’s
heterogeneous handler registry and branch-return factory supply the use case for
explicit boxing. BoxedCallable leaves concrete storage allocation-free and gives
heterogeneous shared-call owners one fallible allocation, a generated adapter,
and the existing concrete drop descriptor. Failed construction drops the consumed
callback. Broader invocation modes and bounded inline erasure remain separate
contracts; erased captures and effects are conservatively ineligible for workers.
Stored environments must own their captures. Borrowed captures cannot escape into heap storage until stored-reference relationships are specified. Invocation follows the existing reusable or consuming contract: reusable mutable callbacks require an exclusive loan; one-shot callbacks must be extracted before invocation. Moving a container relocates inline environment addresses without cloning captured owners. Extraction and destruction must work with allocation disabled. Copy, equality, and hashing remain unavailable for environments, including through containing values.
The native implementation uses type-sized extraction storage, because the
historical range-sized scratch area could not hold arbitrary environments. It
reuses a removed row in the collection’s existing buffer instead of allocating
during pop or remove.
Nested environments must relocate after buffer growth, shifting, reversal, and
transfer. Worker eligibility must inspect captures and callable bodies through
stored environments, including their destructors; storage does not certify thread
safety.
Native concurrency contract
Status: Implemented (B22). Scoped threads, bounded channels, executors, futures, parallel map and reduction, cancellation, and timed selection shipped. The concurrency reference owns the source contract and concurrency architecture the runtime. Later additions include atomics and mutexes and atomically counted Shared owners. These do not implement the linear execution obligations proposed separately.
Shipped differently: generator frames and executor or future handles cannot enter workers. Native files and network handles now cross threads under ordinary ownership and loan rules; the compiler certifies native effects without a stdlib exemption. Foreign calls and opaque foreign resources remain unsupported in workers. The audit below records the earlier restrictions. A failed fallible map returns the earliest failed input’s error as
ParallelError[E].Worker(E), not an index. Start failure returnsThreadError.System(code), or aSpawnError[F]that carries the unstarted job. Executors, map, and reduction are compiler built-ins over a Rust runtime, not a Plenty library.Remaining: transfer of custom-allocator values (B13); automatic parallelization (B24); callbacks on foreign threads (B28). B22’s deferred extensions are richer effects, rendezvous channels, streaming and broader mapping, and broader selection.
Runtime audit
| Component | Current evidence | Consequence |
|---|---|---|
| Type metadata | src/codegen/metadata.rs emits immutable Type/Variant graphs; aggregates retain static pointers | No Rc<Type> conversion is needed; loaded code and descriptors must stay resident |
| Shared storage | plenty-runtime/src/memory.rs now uses atomic counts, including immutable text | Count handoff is tested concurrently; mutable payload access still needs compiler eligibility checks |
| Destruction | The final release reuses the dead count as a thread-local queue link | Only the final releaser may enqueue; synchronize before repurposing the count |
| Inline frames | Generator cleanup uses the same final-release primitive; closures visit captured slots | Inline frames finish synchronously rather than entering the heap destruction queue |
| Allocators | Allocations retain an allocator identity; the built-in one uses the system allocator | Custom allocators need explicit transfer/deallocation permissions and lifetime guarantees |
| Files and foreign resources | File methods mutate state; foreign interfaces do not certify thread affinity | Exclude from initial cross-thread eligibility unless a contract proves transfer and drop are permitted |
| Runtime loading | Library leases release lookup ownership while code stays resident | Residency prevents dangling code, but grants no concurrent-call or affinity permission |
| Instrumentation | Allocation failure budgets are thread-local; live byte counters are atomic | Concurrent tests must keep failure injection local and check leak baselines after joining |
Use one atomic header contract initially, retaining the 16-byte ABI layout and immortal-literal sentinel. Retain requires an existing live owner; it does not publish payload writes. Release decrements with release ordering, and the final releaser acquires before destruction. Reusing the count as a queue link is safe only after that handoff, when no other owner remains. Keep queue operations thread-local and preserve synchronous nested drops from user destructors. The ordering follows the ownership-count argument in the Rustonomicon’s clone and drop chapters. Atomic counts do not make mutable payload access concurrent-safe.
Compiler facts, not user-defined marker methods
Compute transfer and immutable-sharing eligibility structurally. Scalars and immutable text are eligible once runtime sharing is safe. A moved collection requires eligible contents and allocator; a shared collection additionally requires that all reachable observations are safe under shared access. An exclusive borrow transfers only within a scope that joins before that borrow ends. Captures and suspended generator locals participate in the same check.
Do not infer a destructor’s thread safety merely from its fields. A destructor or named callback can call a foreign function with affinity requirements even when it captures nothing. Eligibility needs a call-effect summary, an explicit trusted foreign contract, or a conservative rejection. Recursive data analysis must compute a bounded fixed point over nominal declarations, not recurse indefinitely. Imports never change eligibility implicitly.
Ownership at task boundaries
A scoped task can borrow its parent’s data. Its scope joins every started task
before locals are reclaimed, including on return, ?, and loop exits. Start
failure must preserve the unstarted job. The initial API borrows owners and
reusable closures, leaving the job with the caller on failure; a consuming
submission API would need to return unstarted ownership with its error.
Join transfers one result
to one consumer. A task returning Result[T, E] keeps that application error
distinct from failure to create a thread or allocate task storage.
Bounded channels own their queue storage and transfer values. A full queue may block under an explicit operation; a nonblocking send reports full/closed with the unsent value intact. Closing wakes waiters. Queued values drop exactly once when no receiver remains. Queue allocation is fallible; send into reserved capacity must not add an unreported allocation.
Familiar executor behavior
Use Python’s explicit executor construction, submit, result retrieval, ordered
map, and context-managed shutdown as the interface model. Python also offers a
buffer limit for mapped work and warns about workers waiting on work in the same
exhausted pool. Python executor documentation.
Plenty should require bounded submission storage and report submission failure with ownership of the unqueued job. Exiting the executor joins running jobs and reclaims unconsumed results; shutdown may cancel pending jobs but cannot kill running ones. Worker code must not assume that waiting on another job in its own pool makes progress. Thread count and queue capacity are explicit settings. Result handles here are blocking task handles, not language async futures.
Parallel map returns results in input order; a deterministic fallible variant returns the error of the earliest failed input after earlier work finishes. Cancellation is cooperative. Dropping partial outputs and joining tasks cannot undo external effects. Parallel reduction needs an explicit reassociation policy, especially for floating point. Ordinary loops and comprehensions retain serial semantics.
Executor scheduling, adapters, and traversal belong in libraries where possible. The compiler provides checked ownership/effect facts; the runtime provides fallible thread creation, synchronization, and deterministic cleanup primitives.
Bounded channel implementation decision
Status: Implemented. Channels own the source contract and concurrency architecture the runtime.
Shipped differently: construction makes one allocation, and the endpoint pair is an inline tuple. Timed selection later added waiter nodes to the core.
Remaining: zero-capacity rendezvous, and send-side or larger selection (deferred B22 extensions).
Plenty channels use a fixed-capacity MPMC ring, explicit endpoint sharing, and fallible construction. Both blocking and nonblocking operations preserve unsent ownership. This decision builds on established contracts while preserving Plenty’s recoverable allocation policy.
Contracts borrowed from existing implementations
Rust’s bounded standard channel supplies backpressure and sender cloning. Its receive side is single-consumer. Crossbeam supports multiple competing receivers and distinguishes an empty/full queue from a disconnected peer. Buffered messages remain receivable after senders disconnect; an unsuccessful send returns the message. Plenty adopts these useful behaviors, without implying broadcast or fairness among waiting threads.
Crossbeam’s public bounded constructor returns handles directly. The inspected
0.5.17 implementation allocates its array and shared counter internally, without
a recoverable allocation result. Wrapping it would not satisfy Plenty’s allocation
contract. No Crossbeam source is copied into the runtime.
Synchronization choice
One mutex protects queue positions and disconnection state. Separate condition variables park blocked readers and writers. Predicate loops follow the standard condition-variable contract, including spurious wakeups. Normal transfers notify one peer; disconnection wakes all affected peers. User destructors never execute under this lock.
parking_lot offers compact synchronization and useful wakeup policies, including requeueing waiters during notification. Plenty’s currently supported Linux target already has inline, futex-backed std mutexes and condition variables. Source inspection and allocation-disabled native tests establish the needed allocation behavior on that target, so this implementation adds no dependency. This is a target property to revalidate when porting, not a cross-platform std API guarantee.
Ownership and costs
Construction makes one allocation, for the core and ring. The endpoint pair is an inline tuple. Both endpoint headers live inside the pinned core. Sharing increments the appropriate atomic count without allocating another endpoint object. Last-drop disconnects that family; a separate lifetime count keeps the core alive through queued-message destruction and both endpoint callbacks.
Fixed storage bounds queue memory. Each message is embedded using its native typed slot size; inline closure captures do not need boxing. Heap-owned messages transfer their ownership pointers. Receive copies inline payload bytes into caller storage before freeing a ring slot. These are representation moves, with one logical owner throughout.
Receiver ownership cycles are rejected by a finite compiler type-graph check: queued messages must not retain the receiver whose last-drop would drain them. Ordinary recursive data and sender-based reply paths remain valid. Endpoint context managers reuse lexical cleanup; placing the receiving guard after its producer task closes that handle before joining on early exit.
The mutex serializes transfers and provides a straightforward publication point. There is no lock-free or strict fairness claim. Positive capacities cover the initial contract; zero-capacity rendezvous is a distinct handshake rather than a disguised one-slot queue. Scheduling and subsequent extensions remain in the backlog.
Named arguments and fresh defaults
Status: implemented for named functions, methods and constructors. See the
current call rules
for supported defaults and deliberate restrictions. This is the narrow revision of
B36, separating named arguments/defaults from the older
unpacking proposal. It does not approve
splat unpacking, variadics, /, or * signature markers. The discussion below
records the design rationale; named calls and fresh expression defaults now
use compiler-generated wrappers with ordinary checked ownership. Callable
values and builtins remain positional. Protocol-constrained generic bodies
currently require full positional method calls conservatively.
Construction should communicate structure
At baseline bef41b1, stdlib/std/json.plenty’s decoder (line 701) builds
Decoder with 28 positional fields. Many have identical types. Names would
help review, but 28 independently supplied values remain a poor abstraction.
Prefer an explicit Decoder.__init__ that accepts meaningful inputs, initializes
internal state by field name, and establishes invariants. An instance cannot
be created uninitialized and then escape while a caller fills its fields:
current initialization rules require
all fields on every successful initializer exit. This is desirable.
Use a configuration class for coherent caller choices; std.json.Limits
already does this. Group cursor/location, token-decoding, and container state
only where the grouping has a real invariant. Passing a new class containing
the same 28 unexplained arguments merely relocates the problem.
A builder is useful when construction is incremental, validation spans several
settings, or a sequence of required choices matters. Its build returns a valid
object or a typed error. It is not necessary for every record. Current explicit
initializers allow only unit or allocation failure, so factories remain useful
for domain validation errors. Typestate
can strengthen a builder when required stages justify extra types.
Other review sites: json.encoder, http.server.config, http.client.config,
and http.body.multipart_limits. Names help callers distinguish timeouts,
capacities, and limits; structure and constructors keep internal state private.
Named call arguments without splats
Example call syntax:
def connect(host: &str, port: u16, timeout_ns: u64 = 1000000u64) -> Result[Connection, NetError]:
...
connect("example", timeout_ns=5000000u64, port=443u16)
- Positional arguments must precede every named argument. Names may then appear in any order. No positional argument may follow a named argument.
- Bind names statically; diagnose unknown, repeated, missing, and multiply supplied parameters, including a positional value plus the same name.
- Evaluate the callee/receiver once, then supplied argument expressions once in written order, regardless of parameter order. Only placement into parameter slots is reordered. Type-check each expression against its resolved parameter.
- Preserve move/borrow checking and early-propagation cleanup of evaluated arguments. A method receiver reservation must span the same evaluation steps; reordering must not relax alias restrictions.
- Defaults follow required parameters in a declaration. Naming a required
parameter does not make it optional. Explicitly passing
Nothingis a value, not an instruction to evaluate a default. - Cover named functions, statically resolved methods (excluding
selfas a supplied argument), and generated/explicit constructors. Constructor visibility and private-field restrictions remain unchanged.
Parameter names become source API commitments. A rename can break callers. Callable values currently have positional structural signatures without names; keep their calls positional. Protocol parameter names currently need not match implementations. Initially keep calls through protocol constraints positional too; do not infer a keyword contract from whichever concrete class specializes the body. Adding protocol-level names later would be an explicit compatibility decision. Builtins require a deliberate inventory of stable parameter names.
This needs argument-binding and lowering work, diagnostics, imported signature metadata, and ownership tests. No runtime argument dictionary is required, but calling it a trivial parser change would underestimate the work.
Defaults: two stages with one freshness rule
Start with the older proposal’s allocation-free literal defaults. Then consider fresh expressions, using the same omitted-argument semantics. Defaults are evaluated once per omitted argument per invocation, never stored as a shared mutable object. Python’s documented definition-time evaluation is explicitly not the intended model. Python default arguments
For fresh expressions, evaluate supplied arguments first in written order,
then omitted defaults in parameter declaration order, before entering the body.
Resolve default names in the function’s defining module. Initially prohibit
references to parameters or self, avoiding reordered/dependent initialization.
Reject dependency cycles rather than inventing a dependency scheduler. Callable
values still require full positional arity; omission is not part of their type.
A default expression must produce the parameter’s value using ordinary typing. Construct a fresh owner for an omitted owned parameter. For an omitted borrowed parameter, a proposed extension materializes a fresh temporary owner and lends it to that parameter for the call. Use Plenty’s existing annotation placement:
def work(data: &mut list[i64] = []?,
lookups: &mut dict[str, f64] = {}?) -> Result[Option[bool], AllocError]:
...
The caller supplying either argument must still supply a legal exclusive loan. An omitted argument gets separate storage for every call. Mutation of an omitted scratch buffer is discarded at call completion; callers needing that output must pass storage or receive an owned result. Empty-container expected types come from the parameter. References into these temporary owners must not escape in returned references or borrowed closures. Reject such calls when the return provenance could include an omitted temporary; never extend its lifetime implicitly or store it in an aggregate.
Allocation failure cannot disappear into an Option
In Plenty, [] and {} produce Result because allocation is fallible. A bare
default =[] therefore does not currently type-check as a list, and an
Option[bool] return has no place for AllocError. Freshness alone does not
solve this mismatch.
Recommended rule: explicit ? inside a default propagates through the declared
callee result contract, not the lexical caller’s return type. A call with an
omitted fallible default still has exactly the declared return type. The example
therefore returns Result[Option[bool], AllocError]. On default failure the body
does not run, evaluated owned arguments/defaults are cleaned up, and that call
produces Err. Supplying all arguments does not change its static return type.
Error conversions must be explicit as elsewhere (or follow a separately accepted
error-set rule); no hidden Failure conversion or trap is introduced.
Lower this as a compiler-generated default wrapper or equivalent call-site
control flow with callee-local propagation. Simply pasting a ? expression
into the caller would give incorrect semantics. Diagnostics should identify
the declaration/default and the call that omitted it. Keep this stage separate
from allocation-free defaults because its cleanup and effect semantics are
substantially more involved.
An alternative is to accept an optional owned buffer and construct it inside a fallible function today. That is more ceremony but uses existing rules. Do not make omission alone change a function’s return type or silently abort on OOM.
Evaluation criteria
Use named arguments on HTTP limits and timeout configuration, and compare a structured JSON initializer against the current 28-value constructor. Test out-of-order keywords with observable side effects; positional/name collisions; imports, generics, methods, constructors and visibility; cleanup after argument or default failure; repeated mutable-default calls; explicit arguments skipping defaults; nonescaping default loans; and callable/protocol restrictions.
Field defaults are a separate construction feature, not implied by parameter
defaults. Avoid combining this work with splats or a general builder framework.
The reference and runnable tutorial describe the implemented behavior. The
stdlib’s http.body.multipart_limits now uses named fields as a small adoption
example; broad constructor redesign remains a separate refactor.
Human-readable formatting and error reports
The formatter families were subsequently unified for issue #634. Current
reports use format[W](self, output: &mut std.fmt.Formatter[W]) and the shared
std.fmt.FormatError; see the report reference.
The concrete destination design below records the original implementation.
Status: Implemented. Enum methods, ordinary enum protocol satisfaction, the concrete
std.reportformatting core, dispatch over anonymous sums, entrypoint selection, and typed context formatting have shipped. Entry reports, formatting, andstd.errordefine current behavior.Floating-point report formatting and dynamic cause-chain interfaces are extensions, not prerequisites for this completed core. The anonymous-sum decision retains builtin
Failureas a last resort after removing the cases that forced erasure. The composition overview explains how mapping, sums, reporting, and typed streams fit together; remaining work stays in B71 of the backlog.
Separate data, presentation, and reporting
An error stays ordinary typed data. Neither Result[T, E] nor ? requires a
reporting protocol. Finite sums compose possible causes without deciding their
human-readable wording or recovery policy.
Structural representation remains available for debugging and fallback reports.
A general human-readable formatting protocol lets classes and enums supply
messages. It is useful beyond errors; a nominal error marker is not required.
Context[E] and domain wrappers retain operation, path, and cause information.
Application reporting chooses destination, prefix, fallback, and exit status.
Formatting a value does not itself print or terminate the process.
Entrypoint reports select Formattable when the concrete type satisfies it,
otherwise use structural rendering with special explanations for some builtins.
Context.describe() still uses structural cause text, while
describe_with() accepts caller-provided wording. std.fmt.Formattable
provides borrowed formatting into a concrete destination; entrypoint reports and
eligible Context values use it. Classes and enums can also supply
the existing std.fmt display/debug methods. Those methods use a generic writer
and allocated scratch and remain useful for richer application formatting.
Borrowed formatting into a destination
The protocol borrows its receiver and writes text through a mutable formatter. An enum implementation matches its variants and formats their details. It does not consume the error or require constructing an owned string. A convenience API may collect the same output into a fallible allocated string.
The core formatter must support an allocation-free reporting path for fixed messages and scalar payloads. This is not a promise that every user-defined implementation allocates nothing: custom computation and string collection keep their ordinary fallibility. Formatting errors must be explicit, must not replace the original application error, and must terminate reporting rather than recurse into reporting the formatting error itself.
The original implemented contract was Formattable.format(self, output: &mut Formatter) -> Result[(), FormatError] in std.report. Formatter stores a private inline
destination enum: stderr or an owned byte collector, plus byte/depth counters.
The stderr variant creates no scratch buffer or heap owner. Text, signed/unsigned
integers and booleans need no allocation; collection and finish explicitly
return storage failures. Floating-point spelling and rich format specifications
remain with std.fmt in this first core slice.
render borrows the value, checks its protocol and restores the depth counter on
success or error. Each native stderr attempt reports its exact byte progress;
interruption is retried, and other output errors make that formatter terminal.
No boxed protocol value, generic writer parameter, prefix, newline, exit policy,
or automatic fallback is hidden in this core. See its reference for byte limits,
partial output, and collection guarantees.
Enum protocols and sum formatting
Ordinary protocol satisfaction now includes enums, with enum methods subject to
the same visibility, receiver, ownership, and effect rules as class methods.
Methods use ordinary indented def declarations alongside variants; shared and
mutable receivers borrow the enum. No special error-only implementation
mechanism is needed.
For the formatting protocol, A | B satisfies the protocol exactly when every
normalized member satisfies it. Generated dispatch examines the tag and borrows
the active payload; it adds no allocation or hidden existential container.
Aliases and singleton normalization give the same formatting behavior.
This rule does not automatically forward arbitrary protocols through sums.
Methods involving Self, mutation, or different result types require a separate
general contract. The first version grants only the specified formatting behavior.
Reporting at the application boundary
The entrypoint policy chooses human-readable formatting when the concrete error type satisfies the protocol, otherwise structural representation. For a sum with an unformattable member, use structural reporting for the sum as a whole; do not introduce a hidden per-member relaxation of the protocol bound.
If the selected rendering fails or is unavailable, emit a fixed minimal report on a best-effort basis. The original error is destroyed once and the failing entrypoint still returns status one. Intermediate propagation remains silent. Output failures, including a closed stderr pipe, must not trigger another report. Preserve the existing entrypoint treatment of stderr failure.
Streaming can fail after emitting a prefix or part of a message. Normal output
has a 16,384-byte budget and depth limit 64. A formatting failure appends the
fixed suffix [report unavailable]\n outside that budget; a terminal transport
failure stops output instead. Reports are not atomic, and the reporter never
retries a user formatter after partial output. Ordinary traps retain their existing
non-unwinding semantics; a best-effort reporter cannot catch arbitrary traps.
Context formatting presents the operation/path and then the typed cause.
Explicit caller-provided descriptions remain useful. This avoids duplicating the cause
in both a wrapper’s message and an automatically appended chain. A heterogeneous
source() interface, dynamic error objects, and general cause-chain traversal
are deferred; typed context composition does not depend on them.
Acceptance criteria
Check class and enum protocol satisfaction, missing-method diagnostics, borrowed
payload preservation, sum dispatch for every member, singleton/alias behavior,
and rejection of a formatting bound when any member lacks it. Verify that plain
unformattable data remains legal in Err.
Exercise fixed messages under forced allocation failure, scalar formatting, string collection failure, nested contexts, custom formatter failure before and after output, unsupported structural rendering, closed stderr, exactly-once payload destruction, and identical compile-and-run/native executable reports.
The existing context description/report convenience APIs preserve their behavior.
Conditional Context[E] formatting is a narrow compiler-supported rule for the
trusted standard type, rather than a general conditional-method feature. The
formatter method, result and destination API are defined in the std.report
reference; the entrypoint reference defines reporting limits and fallback.
Explicit error composition
Status: Implemented through HTTP client adoption. Library mapping, general
anonymous sums, human-readable reporting, typed streams and HTTP framing have
shipped. The HTTP client’s head, write and body paths preserve concrete causes
without internal stored-error retrieval. Installed CLI/data and JSON-service
checks cover application errors, reports, recovery and cleanup.
? supports lossless error-set widening without user-defined conversions.
Builtin Failure remains a deliberate last resort; issue #342 is closed.
The legacy response Reader bridge remains available beside typed response
reads and partial body-collection progress.
The agreed B71 acceptance batch is complete. Floating-point report formatting,
dynamic error objects and other additive extensions remain in B71;
the benchmark-template migration stays deferred during performance comparisons.
The companion error sets page records the sum contract and
the last-resort role of builtin Failure; formatting
records the shipped enum-capable reporting design.
Typestate addresses lifecycle invariants.
Reading this family
| Question | Design and current contract |
|---|---|
| How do I deliberately translate an error? | Mapping below; std.error |
| How do I retain several possible error types? | Anonymous-sum design; current sums |
| How do I present an error to a person? | Completed reporting design; std.report |
| How do stream adapters preserve a transport cause? | Experiment and production evidence; std.io.typed, std.http.typed |
Problem and recommendation
At the original investigation baseline, ? propagated an identical error type,
except for deliberate erasure to Failure. Library adapters repeated success forwarding just to
translate failures. A second, independent problem is that std.io.Reader
fixes its error type to StreamError: HTTP adapters store richer causes in a
separate field and return a generic error through the stream interface.
Explicit result mapping and an isolated parameterized-stream experiment
established the baseline. General anonymous sums and human-readable formatting
for enums and classes subsequently shipped. Issue #342 removed the export and
retained-job cases that forced Failure; the builtin stays as a last resort.
Production typed-stream adoption then removed the HTTP client’s internal error
side channels. Keep named domain errors where they encode useful policy:
neither mapping nor sums alone can recover causes discarded by a stream API.
Evidence in the stdlib
Locations below describe the inspected baseline, commit bef41b1; function
names are the durable anchors when line numbers move.
| Location | Existing burden | Candidate improvement |
|---|---|---|
stdlib/std/http/body.plenty:79–140, allocated, http_allocated, lifted, http, json, stored | Repeated Ok forwarding around error translation | Result mapping; retain classifiers for deliberate allocation normalization |
stdlib/std/http/client.plenty:333–387, alloc, bytes_of, framed, netted, parsed | Several near-identical wrappers | Map only the failure payload |
stdlib/std/json.plenty:121–147, faulted, reserved, unplaced | Nested adaptation before parsing can proceed | Shared mapping operation with named classifiers |
stdlib/std/http/client.plenty:599–640, Wire.noted, fault, written | Error stored in _failure, generic stream error returned, cause retrieved later | Preserve the transport error as part of the returned type |
stdlib/std/http/client.plenty:690–707, Response.read, take_failure | Caller must retrieve richer error separately | Typed stream error path with optional compatibility adapter |
stdlib/std/http.plenty:633–647, BodyReader.read | Framing faults become Io(InvalidInput) plus side storage | Parameterized framing/stream error |
stdlib/std/error.plenty:62–85, context, path_context | Already preserves a typed cause and handles failed context allocation | Reuse this foundation; do not replace it with unconditional boxing |
These are opportunities, not claims that all conversions can disappear.
BytesError.InvalidUtf8 becoming BodyError.InvalidUtf8, for example, is
intentional public API policy.
Explicit mapping
Possible future method spelling (not current executable Plenty):
body = std.json.encode(value, limits).map_err(body_json_error)?
def body_json_error(cause: JsonError) -> BodyError:
BodyError.Json(cause)
The operation consumes Result[T, E] and a mapper constrained by
F: OnceCallable[[E], G], returning Result[T, G].
The implemented std.error.map_error(result, mapper) provides these semantics
as an ordinary library function. Method syntax remains an ergonomic choice,
not required semantics. See the reference
and the tutorial.
- Evaluate the result and mapper expressions once, left to right.
- Invoke the mapper exactly once on
Err, never onOk. - Move the original error into the mapper; forward success without copying.
- Drop an unused mapper and its captures on success. Ownership and early-exit cleanup apply normally, including failure while evaluating the mapper.
- Mapping itself allocates nothing. Callback construction and execution retain their ordinary effects; do not silently box captured callbacks.
- A mapper returns
G, not a secretly flattenedResult[G, X]. Fallible context enrichment needs an explicit API and policy for preserving the cause. ?still performs no user-defined conversion. The call site names the policy.
This removes repetitive success branches while leaving semantic conversions reviewable. A helper can still be worthwhile when it communicates a domain decision. Do not mechanically inline all classifiers or introduce complex closures solely to reduce line count.
Rust’s thiserror generates error implementations and selected conversions;
it reduces declaration boilerplate while retaining a named error type.
Plenty could later consider declared conversions, but implicit conversion at
every ? would need ambiguity and coherence rules. Explicit mapping is the
smaller initial contract. thiserror documentation
Does an error protocol solve it?
The idea is coherent, but has three distinct meanings:
| Meaning | What it provides | Fit today |
|---|---|---|
E: Error generic constraint | Operations available on one statically known E | Classes and enums can satisfy protocols; a reporting bound is optional |
Result[T, Error] holding arbitrary implementations | An owned existential value with hidden concrete type | Not supported; protocols are not runtime value types |
| `Result[T, A | B]` | One of a finite, statically declared set |
A generic bound does not allow one specialization to return unrelated concrete types on different branches. It still needs an enum, union, or erased container. Nor does a common reporting interface establish how to classify interruption, retryability, partial progress, or a framing failure.
An existential error container must specify storage, destruction, dispatch, thread transfer, reporting, and whether downcasting is supported. Arbitrarily large owned payloads cannot all fit a fixed-size inline object; boxing or an explicit capacity/fallback policy is needed. Allocation while reporting an allocation failure is a particular problem for Plenty. Borrowed errors cannot generally escape the failing function’s locals.
Rust trait objects make runtime dispatch explicit. anyhow offers an erased
error with contextual reporting and downcasting, trading away an exhaustive
signature of possible causes. That can suit application boundaries, but it is
not equivalent to a finite typed sum. Rust trait objects,
anyhow documentation
Do not make a reporting protocol mandatory for error payloads: Plenty currently
allows ordinary typed data in Err. Reporting and recoverability are separate.
The implemented formatting protocol provides
presentation without an existential error container. Enums can satisfy protocols,
and anonymous sums forward this formatting protocol when all members qualify.
Builtin Failure stays as a last resort, for when no precise error can be
written (issue #342); an erased error container and heterogeneous cause-chain interface remain deferred.
Removing the stream error side channel
Current parameterized protocols
already substitute type arguments in result positions. A minimal Source[E]
and generic reader using an enum error type also passed the local debug compiler
smoke check. Prototype the following
shape in an isolated library module, before changing std.io:
protocol Reader[E]:
def read(self: &mut Reader[E], buffer: &mut Bytes,
start: i64, stop: i64) -> Result[i64, E]:
pass
enum ReadFailure[E]:
Source(E)
UnexpectedEof
InvalidProgress
OutOfRange
Allocation(AllocError)
This is a proposed interface shape, not a claim that changing Reader alone
is sufficient. Helpers generate their own errors, and retry logic needs a way
to recognize interruption. Keep that policy explicit: use a classifier callback
or a protocol parameter for a classifier class, or require a normalized stream
envelope that distinguishes Interrupted from Source(E). Compare these
options against concrete adapters before selecting one.
read_exact[E, R: Reader[E]] would return progress plus ReadFailure[E];
copy_all must preserve independent reader/writer error types and the failing
side. HTTP framing would similarly need HttpError[E] (or a distinct generic
framing error), separating transport causes from protocol failures. Existing
StreamError is still useful for OS adapters and compatibility facades.
Avoid recursive type equations such as making ClientError contain
HttpError[ClientError]. Define a transport-only error underneath framing and
an exchange-level error above it. Separate EOF, malformed progress, cancellation,
and protocol faults; arbitrary E cannot be interrogated as if it were
StreamError. Parameter threading increases signatures and specialization;
measure whether eliminating side storage pays for that complexity.
Retain old APIs during a migration, or make a deliberate coordinated breaking
change. Remove _failure/take_failure only when every relevant adapter and
caller receives the full cause directly. Mapping alone cannot achieve this.
Acceptance criteria
The evidence companion records the mapper comparisons, prototype tests and production adoption. The HTTP qualification record records the installed CLI/data and JSON-service acceptance. The criteria below preserve the checks that motivated those changes.
Compare before/after HTTP body adapters, a JSON classifier, and one complete HTTP-over-stream path. Count adapter functions and side-channel fields, not just lines. Verify exact cause preservation, once-only mapper invocation, success without mapping allocation, destruction of affine payloads/captures, partial progress, interruption, cancellation, and allocation failure. Check diagnostic quality for incompatible error types and protocol inference. Tutorial/reference changes belong with an implementation, not this proposal.
Prototype findings
The experiment record records mapper adoption, specialization limits, and the typed-stream prototype. It distinguishes the isolated examples from the public library APIs.
Additive production slice
The production slice provides typed I/O helpers and fixed-length/chunked/close-delimited HTTP body readers, validated-head integration and full client adoption. Internal transport side channels are removed; the legacy response Reader bridge remains for compatibility. The evidence page records the delivered scope and limits.
Error-composition experiments and typed streams
Status: Implemented through HTTP client adoption. This is the evidence companion to the
composition overview, not a separate proposal or
work queue. It accompanies the completed B71 acceptance batch. Mapping, typed
streams and framing, and the HTTP client’s head,
write and body paths have shipped. Current contracts belong to
std.error,
std.io.typed, and
std.http.typed.
Prototype findings
The first implementation needs no compiler or runtime changes. map_error
uses OnceCallable, a mutable local callback binding, and an ordinary match.
The mutable binding accepts stateful reusable callbacks as well as named
functions and consuming closures. A callback is owned even when unused, so
success drops its captures. Error values need no reporting protocol.
The initial representative stdlib changes covered three result adapters:
std/http/body.plenty removes json and maps directly in read_json and
send_json; std/http/client.plenty uses mapping in framed; and
std/json.plenty uses it in faulted. Three explicit success-forwarding
branches disappear. Two convenience adapters remain and one non-generic
bytes_fault classifier exposes the JSON policy. The number of named
functions is therefore unchanged, while one adapter disappears completely.
Other conversion helpers initially remained available for comparison.
A wider initial conversion of eleven adapters exceeded the compiler’s global
256 generic-function specialization cap in the offline HTTP JSON service test.
Keeping a generic convenience adapter around map_error adds another instance
for each success/callback type. The follow-up for issue #141 separates a
256-level dependency-depth guard from a 16,384-instance total resource budget;
independent specializations no longer consume the depth guard. It restores the
wider eleven-adapter adoption: body allocation, byte classification, HTTP and
storage conversions; client allocation; and JSON allocation and location-free
errors now also use mapping. Their conversion policies are unchanged. This
addresses the observed scaling constraint without removing finite compiler
limits or introducing allocation. See specialization limits.
Worker integration exposed a real ergonomic restriction: thin function values
lose their target identity inside the generic helper, so thread effect checking
rejects their indirect calls. The stdlib adopters use concrete consuming
closures (def once (...)) instead; their bodies remain inspectable and allocate
nothing. Named classifiers are called directly from these closures. This keeps
worker checks intact but adds a wrapper at those sites. Mapping does not solve
that existing callable-design limitation.
The isolated modules in examples/error_composition/ explore a normalized
StepError[E] envelope: Interrupted requests retry; Source(E) preserves
everything else, including cancellation and allocation failure. Reader[E]
and Writer[E] feed read_exact, write_all, and copy_all; helper-generated
failures have their own variants. Copy errors distinguish reader and writer
types and count bytes delivered to the writer, including partial writes.
Caller-provided scratch storage avoids allocation in the helpers.
FixedBody[E, R] demonstrates Content-Length body framing over a typed
transport. Its failures distinguish Transport(E), truncation, invalid
progress, invalid range, and reuse after failure. Interruption leaves it
usable; terminal framing/transport failures mark it broken. The full body-copy
path returns the original transport cause through FramingError[E] and
CopyFailure[FramingError[E], W]. It contains zero error-storage fields and
zero retrieval methods. At that prototype stage, production _failure fields
and take_failure methods were retained: the experiment alone did not justify
their removal.
This is one complete body-transfer path, not a complete HTTP client. It assumes a previously validated Content-Length and response semantics. It does not parse heads, implement chunked/close-delimited framing, perform TLS or network I/O, or manage connection reuse. Scripted sources exercise rich TLS-like errors and cancellation as typed data, not real handshakes or cancellation tokens. Those boundaries must be covered before a production migration.
Protocol-bound inference recovers the error types from concrete methods, even
through the framing adapter: fixed_body(source, length) and
copy_all(&mut body, &mut writer, &mut scratch) need no explicit type arguments.
The generic definitions and nested error signatures are still long. A normalized
envelope keeps retry policy simple but
requires adapters to wrap their failures. A classifier callback would leave
transport signatures alone, at the cost of threading retry policy through
helpers; it remains an alternative, not implemented here. Arbitrary errors
must not be retried by guessing their meaning.
Focused tests live in tests/test_error_composition.rs and
tests/test_typed_stream_prototype.rs. They cover mapper ownership, allocation
failure, nested results, evaluation order, progress, retry, EOF, cancellation,
failure-side attribution, and typed transport preservation. These findings
supported the additive production framing below; mapping remains independently
usable.
Additive production slice
std.io.typed now makes the prototype’s typed
protocols and progress helpers available without copying example modules.
Its owned legacy reader/writer bridges preserve the returned StreamError
and classify only explicit interruption as retryable. A consuming free function
recovers each bridge’s stream for explicit flushing, closing or seeking.
The private optional slot supports extraction under existing field-move rules;
it allocates no storage.
std.http.typed initially provided fixed-length and
close-delimited body readers. Close-delimited EOF is sticky, empty reads do not
consume input, and its connection is never reusable. Both return
FramingError[E] through the typed protocol without error-storage fields or
retrieval methods. The production tests run the prototype’s cause/progress
cases against both implementations and exercise real std.io.Cursor
bridges through both framing modes. The standalone standard.plenty example
is the public-module counterpart to main.plenty.
The subsequent chunked reader preserves typed transport errors and payload
progress through size lines, data CRLF and bounded trailers. It retains parsing
state across interruption and does not read into the next message. Framing and
allocation failures use FramingError.Protocol(HttpError); the concrete
transport cause stays in Transport(E). Its scripted regression tests inject
interruptions and transport errors throughout metadata and payload, and check
limits, malformed input and terminal-state behavior. The original isolated
prototype remains a fixed-body experiment rather than a duplicate chunk parser.
Production now also includes parsed-head framing validation, bounded typed head readers and writers, typed buffering, and guarded recovery of completed persistent connections. Shared grammar/serialization helpers keep legacy and typed HTTP validation aligned. Head failures count consumed wire bytes; writes count delivered wire bytes; body-copy and collection counts refer to payload.
The HTTP client uses these paths for CONNECT, request heads, in-memory and
streamed uploads, response heads and all body modes. Its existing TLS/network
transports, deadlines, cancellation, redirect policy and pool remain in charge
of connection lifetime. Internal Wire/Dialled error storage and retrieval
are removed. Response.read/take_failure remains a compatibility bridge;
typed_response carries the full ClientError through typed stream helpers,
and read_all_progress retains bytes appended before failure. Existing
send/fetch signatures still return causes without per-head/write counts;
those are available from the lower-level typed operations. No builtin Failure,
general-sum, or reporting change was needed for this migration.
On 2026-10-18, a local unoptimized compiler comparison of main.plenty
against standard.plenty used five --compile runs each with
PLENTY_CACHE=off. Median wall time was 0.196 s versus 0.216 s (ranges
0.163–0.209 s and 0.181–0.225 s); linked executable sizes were 3,286,648
and 3,290,816 bytes. Other validation builds were running concurrently, so
these timings are a bounded smoke measurement, not a performance guarantee.
The public fixed path adds roughly 4 KiB in this probe; it imports the legacy
stream module to define bridges. Both copy signatures still carry two error
types and two stream types, while fixed framing carries one error and one
reader type. Call sites infer these parameters. No specialization-count
instrumentation was added, and this small case does not establish scaling for
a complete HTTP client.
Finite error sets as anonymous sums
Status: Implemented within the agreed initial contract. General anonymous
sums have shipped, as described in the
anonymous-sum reference.
It adopts case cause: Type: at the case position, grouped types and inline
normalization/injection/widening, including Result widening and borrowed matching.
Human-readable reporting has also shipped.
Builtin Failure stays as a last resort, for when no precise error can be
written; issue #342 closed after removing the cases that forced erasure. The current
propagation reference
remains authoritative. Case guards, residual-set inference and empty sums are
deferred extensions in B14/B71 of the backlog, not missing
requirements of this completed initial contract. Human-readable presentation is specified separately
in formatting and error reports. The
composition overview separates this language feature
from explicit mapping and typed stream APIs.
Motivation and research
The proposed signature
Result[(), IoError | AllocError | SomeError] describes one error drawn from
a finite set. It does not mean simultaneous errors, a dynamically typed value,
or automatic recovery. This could remove wrapper-only enum declarations while
retaining payloads and compile-time coverage checks.
Dillon McMahon’s article argues for composing precise error sets, widening them when combining operations, and narrowing them after recovery. It also separates recovery type information from operational context. Those are useful design questions, not proof that one implementation is best for Plenty. The Missing Piece in Rust Error Handling
The eros documentation describes typed and erased errors, widening and
recovery. Crucially, version 0.8.1 documents boxed storage for ErrorUnion.
Its ergonomics therefore do not demonstrate allocation-free inline error
composition. Plenty should evaluate its own representation rather than copy
that storage choice. eros documentation
The reported mixed Reddit reaction is not used as evidence here: no particular thread was supplied or independently assessed. The following objections are design analysis, not attributed community consensus.
Agreed initial contract
The following notation describes the implemented composition contract:
def load() -> Result[Config, IoError | ParseError]:
source = read_source()?
parse_source(source)
A | B is a general type expression, available in ordinary value positions
as well as Result errors. Error composition supplies the first acceptance cases;
the type system must not restrict sums to the second argument of Result.
- Treat
A | Bas a tagged, owned sum of whole nominal types, not a C union and not an untagged overlap of values. No boxing is implicit. Storage must accommodate the largest payload and its tag; existing recursive-data rules still apply. Large payloads can enlarge every enclosingResult. - Normalize parentheses, order, duplicates, and resolved aliases:
A | BandB | Ahave the same identity;A | AisA. Preserve distinct nominal wrappers. Publish no stable tag-number/FFI-layout guarantee by accident. - Permit lossless injection of
Einto a destination containingE, and widening of a source set into a superset. Do not invoke arbitrary user conversions. Apply these rules in expected-type checking for bindings, assignments, arguments,Err, direct returns, and?. Also permitResult[T, E]to widen toResult[T, F]when E is a member/subset of F: success moves unchanged and failure is retagged without allocation. This does not establish covariance for arbitrary generic containers or references, and does not change the success type. - Reject propagation when any source alternative is absent. Diagnostics name
the missing alternatives and source operation. The separately specified
builtin
Failureerasure at?remains a deliberate last resort; it is not lossless sum widening. Issue #342 superseded the original removal plan. - Provide exhaustive, unambiguous matching by member type using
case cause: MemberType:. Existing enum constructor patterns cannot identify every class payload; existing pattern|keeps its separate meaning. - Do not implicitly flatten named enum variants.
HttpError | AllocErrormay still containHttpError.Allocation; deduplicating that semantic overlap requires an explicit classifier. Otherwise API abstraction is broken. - A first version can match the entire set without automatic residual-type
inference. Recovery that subtracts a member type is a further feature, not
free syntax sugar. An uninhabited empty set and conversion to infallible
Talso need a decision; do not use inhabited unit as an empty error type.
Function return types remain declared. Do not infer a public error set from the body or silently widen a declared binding after a later assignment. Expected types guide injection; inference must not guess between distinct generic substitutions that satisfy membership. Normalize again after concrete generic substitution, so a generic set can collapse to a single member.
Injection and widening evaluate the source once and preserve the active payload. Owned noncopyable payloads move; inactive alternatives are never initialized or dropped. Early exits and destruction follow the existing sum ownership rules. Copyability and thread-transfer eligibility require every member to qualify. Existing restrictions on sum payloads and recursive storage continue to apply; general sums do not independently introduce stored references.
Matching and remaining surface decisions
Matching must select a whole member type and bind its payload, including class
payloads. The implemented spelling is case cause: IoError: at the case
position. It does not extend existing pattern |. An enum member can then
be matched by its constructors normally.
The initial contract requires exhaustive coverage of normalized members, or a wildcard. Case guards remain unsupported; a future guarded arm alone cannot establish coverage. Reject duplicate unguarded member arms, including members that coincide after generic substitution. Specify those diagnostics on concrete specializations rather than pretending generic alternatives always remain distinct.
Owned matching follows ordinary move rules; matching through & or &mut
borrows the active payload with the corresponding permission and cannot let it
outlive the owner. A wildcard does not infer a smaller residual sum in this
version. Empty sums, automatic error-set subtraction, and automatic conversion
to infallible values are deferred.
The implementation puts | below references and type applications, uses (T)
for grouping and (T,) for singleton tuples, and allows typed-member binding
when normalization leaves an ordinary type. Callable brackets delimit their
argument/result types. Diagnostics cover these cases alongside generic inference
ambiguity; ambiguous constructors require an explicitly typed intermediate.
Replacing Failure
Replace uses of builtin Failure with precise errors or finite sums, optionally
named by ordinary type aliases. Keep domain enums where they express policy;
map_error remains the explicit way to classify or wrap a cause. Do not replace
erasure with an implicit catch-all member or mandatory reporting bound.
The original alternative was to remove the prelude builtin and its special
checking/code generation entirely, then reject old erasure uses. That alternative
was superseded by the issue #342 decision: retain Failure as a last resort and
remove the cases that force it. C exports now accept payload-free user error
enums; explicit retained-job errors and drop_job() preserve start/submission
causes without mandatory erasure. Examples, templates and diagnostics use precise
errors, except the benchmark template deliberately deferred during performance
comparisons. Tests of Failure itself remain intentional.
The mutual-TLS lesson and installed CLI/data and JSON-service workflows are acceptance cases for precise errors and reporting; see the qualification record. Retaining erasure is a documented policy, not unfinished builtin removal.
What sets do not solve
An I/O failure while reading and the same I/O type while writing become the
same set member. std.io.CopyError.Read versus .Write carries useful meaning
that a bare union discards. Use named operation wrappers or structured context.
Likewise, JSON offsets and partial byte counts must remain attached to causes.
A public set exposes its member types and changes when another alternative is added. Exhaustive callers then need updating. Named domain enums can provide a more stable abstraction boundary, though adding their variants also affects exhaustive matching. An erased error hides alternatives but sacrifices static coverage. Choose based on the caller’s recovery contract.
Unions do not automatically fix Reader returning StreamError: the stream
interface must still become generic or adopt a different envelope. Nor do they
replace allocation-aware context construction in std.error.Context[E].
Concrete evaluation targets
Try sets on internal composition near http/body.plenty’s read_json and
send_json, and the HTTP client’s framed/netted boundaries. Compare with
explicit mapping into the current named types. Preserve BodyError.status
classification, allocation normalization, JSON locations, and retry policy.
Do not start by replacing CopyError, Partial, JsonError, or every public
HTTP error. They carry semantics beyond membership. Require comparisons of
signature readability, diagnostics, representation size, compile time, and
client recovery code. Tests must cover aliases, duplicate types, permutation,
nested sets, ambiguous generic members, missing-member diagnostics, payload
destruction, exhaustive matching, and no implicit allocation.
Explicit mapping is the implemented baseline. Add ordinary non-error sum values to the acceptance cases, plus Result widening on both success and failure, generic normalization, borrowed/mutable matching, and entrypoint reporting of each active member. Builtin Failure and its erasure stay as a last resort, for when no precise error can be written (issue #342). Production typed-stream adoption subsequently shipped under the separate composition proposal: sums alone cannot restore causes already discarded by an adapter.
Bundled HTTP acceptance
Status: Implemented and qualified for x86-64 Linux GNU. The HTTP framing layer, client, server, TLS, and deployment helpers ship over checked native I/O effects and bounded workers. The server parks request heads and TLS handshakes, supports signal-driven draining, and passes nginx/Caddy reverse-proxy tests, which CI runs. The qualification record records every B66 gate passing, including coverage-guided fuzzing, retained corpora, resource-failure tests and installed offline application checks. Issue #103 is closed. The linked references own the production-readiness claim, plain and HTTPS baselines, operational limits and maintenance process. HTTP/2, HTTP/3 and additional platforms are outside this completed scope; the requirements below preserve the original design rationale.
The library scope supplies the broader context. Typed stream composition has also shipped through HTTP client adoption; its API acceptance is distinct from these server qualification gates. Platform qualification also remains distinct from HTTP behavior on the current supported host.
The bundled server should remain usable in production without requiring a third-party replacement for a demonstration server. Go’s combined HTTP client/server surface motivated this goal in issue #49; no Go compatibility or claim about its protective defaults is implied.
Execution and ownership
Define trusted native I/O effects and transferable/thread-safe handle ownership before scheduling socket/TLS calls in workers. Current worker restrictions do not permit arbitrary native calls or files. Preserve ordinary checks for std; never exempt the whole namespace. A measured bounded blocking model may be sufficient initially; polling is an alternative for many idle connections. Unbounded threads per connection are not an acceptable default. Async language syntax is not a prerequisite.
Handlers and shared state must obey ownership/thread-safety rules. Basic routing must work without heterogeneous erased closures. Shutdown deadlines must not reclaim borrowed storage still in use. Fatal traps currently do not unwind: expected handler failures use Result, while arbitrary fatal traps remain process-fatal. Per-request recovery cannot be promised by a catch-all mechanism.
Client and wire contracts
Use HTTP/1.1 as the initial conformance boundary. Bound header count/bytes, body/buffer sizes and parser work. Reject malformed/conflicting length or transfer encoding, truncated bodies and invalid headers with protocol errors, not unbounded allocation or accidental traps. Define reuse/framing policy and streamed request/response bodies with partial-progress I/O.
The reusable client needs pooling, DNS/connect/TLS/request deadlines, cancellation, redirect/proxy policy and bounded streaming. Certificate/hostname verification is enabled by default. Avoid automatic retries once request effects may have happened. Maintain backend/trust-root versions and updates; tests use local certificates/services.
Server and deployment contracts
Provide method/path routing, composable handlers, JSON/form helpers, cookies and bounded multipart parsing/storage. Bound connections and queued work with backpressure and documented overload behavior. Specify header/idle/handshake and body/write deadlines plus explicit controls for long-lived streams. Cancellation/failure releases handles and multipart temporary storage.
Support direct TLS and documented reverse-proxy deployment. Forwarded identity or addresses require an explicit trusted-proxy policy. If static-file helpers are adopted, enforce filesystem-root/symlink rules with filesystem-aware checks; string cleanup alone is insufficient.
Graceful shutdown stops accepting, drains or cooperatively cancels active work under a documented deadline policy, and releases resources after safe joining. Provide structured access/error events and metrics hooks with caller-selected sinks and no mandatory telemetry vendor.
Qualification gates
Production qualification is a separate issue from a core server implementation. Require parser fuzzing and retained regression corpora, protocol conformance/ interoperability, slow-client/cancellation/allocation-failure tests, sustained load and resource-leak tests, and repeatable throughput/latency/memory baselines. Publish workloads, hardware, versions, supported platforms and configured limits. Exercise overload and graceful shutdown while work is active.
Run the motivating CLI and JSON service integration examples with an installed, relocated toolchain offline, including verified TLS and proxy fixtures. Check unused-module/backend linking. Document expected failures, fatal boundaries, operational guidance and an ongoing security-fix/maintenance process before calling the server production-ready. HTTP/2 and HTTP/3 remain separate decisions.
Explicit applications, modules, and executable lessons
Status: Implemented. The baseline below is
b690afe, when the tutorial was one file,TUTORIAL.md.
- Entry point: implemented. A parameterless
mainreturns(),i32,Result[(), E], orResult[i32, E]; see entry point and entry reports.- Modules: implemented. Absolute imports, aliases,
pubvisibility, and library checking; see modules and visibility. The source root flag is--module-root DIR.- Executable lessons: implemented as executable Markdown; literate sources not adopted.
tests/test_tutorial.rsruns everyplentyandplenty-errorfence in the tutorial anddesign/examples/pages, inSUMMARY.mdorder; see runnable documentation.Literate lesson sources (B21): not adopted. Reviewed on 2026-10-24 against the tutorial at
161e1c63: 213 tutorial pages plusdesign/examples/, with 420 run-pass and 57 compile-fail examples. The Markdown harness already meets the goals the literate design below sets out:
- Every executed fence is a complete program, compiled as written. No lesson hides setup, so every displayed example is copyable and runnable.
- Expectations are reviewed, hand-written
outputanderrorfences. Nothing regenerates them, so a compiler regression fails instead of rewriting the book.- Each runnable example is checked as an AOT binary; compile-fail examples must produce the expected diagnostic without running.
- Multi-file examples are first-class:
plenty-filecompanion modules,sh plenty-buildlibrary builds, andproject-filepackages run withsh plenty-project. The harness rejects orphan expectations and companions on another page.- A failure names the page and line of the Markdown the author edits.
Literate sources would add a manifest, a renderer, a
--checkcomparison of generated pages, and a migration of every page, and authors would edit one file while reviewers read another. The one clear gain, editor and language server diagnostics while drafting, does not require generating the book: a draft can be written in a scratch.plentyfile and pasted into its fence. The question of comment blocks versus sidecar expectation files is therefore closed.Shipped since this proposal: packages provide manifests, locked path/Git dependencies, workspaces, package commands, and per-target build reuse. Import cycles are rejected. Remaining (B17): re-exports and finer module caching/reuse. Registry resolution remains separate package scope.
Current behavior and the proposed entry point
At the research baseline (b690afe), frontend::compile collected top-level
statements into __plenty_entry; their bindings were startup locals rather
than globals. The implementation now calls the declared main instead. The
native runtime still supplies the platform entry point separately.
Recommend requiring an entry-module function for every binary:
def main() -> ():
print("Hello, Plenty!")
Start with zero parameters and return () or i32. Unit means successful exit;
i32 supplies the process status, subject to the host’s exit-status conventions.
main need not be pub, and imported functions named main are ordinary
functions. A library has no entry-point requirement. Checking a library module
must not produce a missing-main error; checking/building a binary must.
Once error propagation and allocation-free error reporting are ready, consider
main() -> Result[(), E]. Do not silently add an implicit formatting/protocol
requirement on arbitrary E now. Applications can initially use a small i32
main that explicitly matches a fallible application function’s result. (main
now accepts both Result forms.)
Allow only declarations and imports at module scope initially. Defer module-level
constants until their permitted constant expressions are specified. Reject
executable statements with a diagnostic suggesting moving them into main.
There should be one application rule for compile-and-run and binary output, not
an implicit script fallback when main is absent. This is an intentional breaking
change on the language-development branch, accompanied by migrated examples.
Python makes top-level execution meaningful both for scripts and imports, and
documents the __name__ == '__main__' convention. Plenty can adopt familiar
spelling without importing executable-module initialization semantics.
See Python’s entry-module documentation.
Imports name source modules; they do not execute them
Proposed source layout, with src selected explicitly as the module root:
src/
app/
main.plenty
geometry.plenty
storage/
files.plenty
Use absolute paths and familiar name binding:
import app.geometry
import app.storage.files as files
from app.geometry import Point
from app.geometry import distance as measure
import app.geometry makes app.geometry reachable. An alias binds exactly the
alias; a from import binds the requested declaration. Importing a module does
not inject its members into the caller’s unqualified scope. A file’s symbols must
come from its declarations, lexical bindings, documented built-ins, or explicit
imports. Importing a protocol must never silently add methods to unrelated types.
Directories initially serve only as namespaces, without executable __init__
files or implicit package initialization. Resolve a.b to <root>/a/b.plenty;
reject ambiguous file/directory module collisions. No relative imports, wildcard
imports, import hooks, conditional imports, or automatic transitive re-exports.
Importing app.geometry must not grant access to a sibling module simply because
some other file imported that sibling.
Python separates finding a module from binding its name and has extensive dynamic loading rules. We need its visible syntax, not that entire runtime system. See Python import statements and the import system.
Initially use one source root: an explicit build/CLI option, defaulting to the
entry file’s directory for standalone programs. A nested entry file that imports
app.geometry therefore needs the root above app, not an implicit search through
parent directories. Resolve against the configured root, never ambient Python
paths or arbitrary environment search paths. A project manifest can eventually
make the root and dependency mapping durable; no package manager is necessary yet.
Load each reachable module once by canonical module identity; include paths in all diagnostics. Initially reject import cycles with the actual cycle path. Collect public declarations before checking bodies in dependency order. Later, declaration cycles may be supported independently of recursive value layouts.
Private by default, with explicit public API
Use pub for top-level functions, types, class fields, and methods:
pub class Point:
pub x: f64
pub y: f64
pub def length_squared(self) -> f64:
self.x * self.x + self.y * self.y
Recommend module privacy initially: private names and members are accessible
throughout their defining module, including sibling helper functions, but not
from other modules. Directory ancestry grants no extra access. This borrows
Rust’s explicit pub vocabulary with a smaller privacy model; Rust itself has
additional descendant and restricted-visibility rules.
See Rust’s visibility reference.
Rules needed alongside parsing the keyword:
- A public class does not automatically expose its fields or ordinary methods.
- Public signatures cannot leak private nominal types; transparent aliases must not bypass this check. Resolve actual type identity before enforcing visibility.
- A public enum exposes its variants as part of its exhaustively matchable API in the first version. Hiding variants requires a separate non-exhaustive design.
- A generated field constructor is externally callable only when the class and
all fields are public. Otherwise require a public
__init__or public factory to establish invariants. An explicit__init__has its own visibility. __del__remains a compiler-invoked lifecycle hook, never a public callable destructor. Access restrictions must not prevent automatic cleanup.- Borrowing private fields, generated structural printing/equality, and generic protocol checks must respect encapsulation. Initially keep compiler-generated printing/equality behavior documented as structural; do not mistake privacy for data secrecy. Decide whether opaque public classes should later opt out.
- Imports are private bindings. Defer
pub importre-exports rather than letting ordinary imports accidentally widen a module’s API. pubcontrols Plenty visibility, not C ABI exports. Foreign exports require their own explicit declaration and ABI validation.
The main implementation work is resolved symbol identity and access checking,
not accepting a new keyword. Give every nominal type/function a ModuleId plus
local identity instead of using an unqualified string as global identity. Local
aliases preserve identity; two unrelated modules may each define Point.
Compiler changes that keep later features possible
Introduce a path-aware compilation session owning sources, module IDs, an import graph, public signature tables, and diagnostics. Keep source-string APIs for isolated checks/tests, with explicit virtual-module identity and an optional resolver; do not make them silently read unrelated filesystem modules.
Resolve imported type/member syntax before lowering runtime expressions. A qualified symbol path and a field access may look similar but are not the same operation. Likewise, importing a module is not loading a DLL; both may be needed to use a foreign library, and should remain separate concepts.
Initially lower the reachable source graph into one object using the existing pipeline. Stable internal symbol IDs should allow later per-module object caching and generic instantiation caches without promising separate compilation now. Public signatures include ownership, borrowing, and future fallibility contracts; callers must not inspect callee bodies to infer them. Module visibility checks must run before both ordinary and generic calls are emitted.
The tutorial is already executable, but its source can improve
At this snapshot, TUTORIAL.md contained 43 successful programs and 11 expected
compile errors; the lessons are now the pages under book/src/tutorial/. tests/test_tutorial.rs extracts them directly. It invokes both
compile-and-run and explicit AOT compilation; successful binaries must produce
the documented output, and invalid programs must produce the expected diagnostic
without effects. Prose is not semantically checked. The Markdown is currently the
source of truth; the lack of standalone files is an authoring/reuse limitation.
The proposed literate-source approach is workable. Recommend one independently
compilable .plenty file per example, with ordered lesson metadata and optional
named display regions:
#| ## Choose the width of an integer
#|
#| Suffixes make the width explicit.
# docs:begin example
def main() -> ():
count: u8 = 42u8
print(count)
# docs:end example
The #| lines become Markdown prose; region markers remain inert comments to
the compiler. Use a small manifest to order files and identify run-pass versus
compile-fail examples, expected stdout/diagnostics, fixture stdin, and any module
root. A chapter may comprise multiple example files. Prose-only sections can
also be listed. Do not concatenate independent programs into one execution unit.
The first lesson shows the full required main. Later lessons may show named
regions, with a link to the complete runnable source and an explicit statement
that the displayed region is an excerpt. An example with hidden setup is no
longer honestly described as independently copyable from the rendered snippet.
Negative examples are separately labeled intentionally invalid source files.
Proposed build flow:
- Parse the manifest, prose, and regions; reject duplicate IDs, unmatched markers, missing files, invalid paths, and orphan expectations.
- Compile/run each complete source with isolated temporary outputs. Validate expected stdout, exit status, and stderr/diagnostics. Use bounded execution time and deterministic fixtures so an accidental loop cannot hang CI.
- Render validated actual outputs and code regions into the tutorial pages. Preserve source locations in failure reports and links to complete examples.
- A
--checkmode renders to memory and fails on differences from the checked-in tutorial. A generation command writes it deliberately. CI never updates it.
Keep reviewed expectations, not only whatever output happened to be generated. Otherwise a compiler regression would rewrite the book and appear successful. Snapshot acceptance must be an explicit author action. Preserve the existing two-command validation when moving it into a shared harness. Once migrated, edit prose/code in the source lessons and treat the tutorial pages as generated; do not maintain duplicate editable examples in both places.
TinyTemplate can handle ordering and presentation, but it does not execute or validate programs. Its variables, loops, and template calls are sufficient for precomputed lesson records; region markers should be parsed by the doc builder, not treated as executable template commands. A simple renderer may need fewer moving parts initially. If using TinyTemplate, disable HTML escaping for trusted Markdown/code fragments, and choose fences that safely contain backticks. See TinyTemplate’s overview and formatter configuration.
Recommended staging and decisions
- Clarify the current tutorial entry behavior immediately without claiming a language change. This is documentation of implemented behavior.
- Implement required binary
mainand migrate executable examples together. - Add modules, stable identities, and privacy in one coherent change; then expose standard-library operations through explicit imports.
- Move lessons to runnable literate files with output snapshots and generated Markdown. This can happen alongside step 2 if the migration is easier once.
Steps 1-3 are complete, except that builtins stay in a prelude that needs no
import. Step 4 was not adopted; the status block gives the reasons. Declarations
are private to their module, main accepts (), i32, and both Result
forms, and the source root flag is --module-root.
Plenty’s next language phase
Status: Investigation complete. This design review consolidated the entry-point and module investigation and three research proposals against baseline
b690afe. The review is complete: ten of its eleven recommendations shipped at least in part, several differently, and the one that did not, allocators, is B13. Every remaining extension in the table below has a backlog entry. Outcomes were checked against the reference at161e1c63on 2026-10-24. The text below the outcome table describes the baseline and its recommendations, not the current language.
Outcome
| Topic | Outcome | Reference or backlog |
|---|---|---|
| Entry point | Implemented; main also returns Result[(), E] or Result[i32, E] | Entry point |
| Modules and visibility | Implemented | Modules, visibility; B17 |
| C interoperation | Implemented: .plentyi interfaces, adapters, exports, runtime loading | Interop; B27, B28 |
| Context managers | Implemented, with with &mut existing | Resource scopes |
| Generics | Implemented; bodies are checked per instance, not against bounds | Generics; B29 |
| Allocation | Implemented: allocating operations return Result. Class, tuple, enum, generator, and range values are inline and construct without allocation; a fallible class initializer returns Result | Allocation and traps |
| Allocators | Not implemented; objects only record an internal allocator | B13 |
| Error propagation | Implemented, including anonymous-sum widening; Failure erasure remains a deliberate last resort | Error composition, sums and propagation; B14/B71 |
| Parallelism | Implemented: scoped threads, bounded channels, ordered pool.map and fixed-tree reduction; automatic parallelization deferred | Concurrency; B24 |
| SIMD | Partly: optimized builds vectorize one constant-comparison filter and count() on i32/u32 clist columns; portable std.simd remains a proposal | Automatic SIMD counts; B23 |
| Runnable lessons | Implemented as executable Markdown; literate lesson sources not adopted | Runnable documentation, decision |
Recommendations at the baseline
The proposed direction fits Plenty: explicit names, structural interfaces, native ownership and cleanup, and predictable ahead-of-time compilation. The biggest changes to make early are module identities, an allocation-failure representation that does not allocate, and a clear boundary between Plenty’s private runtime representation and the C ABI. Context managers need no general trait system. Threads will need more runtime work than their surface syntax suggests.
| Topic | Recommendation | Main dependency or limitation |
|---|---|---|
| Binary entrypoint | Require def main() -> () or -> i32 in the entry module | Replace current top-level execution and migrate examples |
| Modules and visibility | Absolute imports; private declarations by default; explicit pub | Stable declaration identities and path-aware diagnostics |
| C interoperation | Reusable trusted interface files, checked adapters, ordinary C ABI | Explicit ownership, lifetime, layout, and error contracts |
| Context managers | Concrete __enter__/__exit__ methods and deterministic with cleanup | Returned-borrow rules for managers yielding references |
| Generics | Explicit type parameters constrained by structural protocol Name: declarations | Static constraint checking and cached specialization |
| Allocation | Every allocation-capable operation exposes failure; allocation-free error construction | Replace hidden runtime allocations and current boxed error values |
| Allocators | Startup default plus an allocator handle retained by each allocation owner | Provenance, allocator lifetime, and failure-atomic operations |
| Error propagation | Postfix ?, initially with the same Result error type or within Option | Correct ownership transfer and cleanup on early return |
| Parallelism | Scoped threads, bounded channels, then explicit parallel collection operations | Thread-safe shared storage and compiler-checked transfer rules |
| SIMD | Explicit vector operations and contiguous buffers, starting with a narrow target contract | Backend support, scalar semantics, bounds, and data layout |
| Tutorial | Runnable literate lesson sources with checked outputs generate Markdown | A small extraction/validation tool; existing tests provide the baseline |
Entrypoints, imports, and names
The suspicion about startup was correct at the research baseline. The compiler wrapped top-level
statements in a generated entry function. A user function named main was just
another function; it was not called automatically. Top-level bindings were startup
locals, not globals visible to other functions. That behavior has now been replaced
by explicit main, and the tutorial teaches the new contract.
A required binary main makes that contract clearer. Recommend a parameterless
main returning unit or an i32 process status. It need not be pub. A library
has no required entrypoint, and an imported function named main has no special
meaning. Module top levels should contain declarations and imports, with globals
and constant initialization designed separately. Fallible application code can
initially be called from a small main that explicitly handles its result; a
Result-returning entrypoint can follow once reporting and exit status are defined.
Use familiar absolute import package.module and
from package.module import Name, including explicit aliases. Imported names and
builtins account for everything a file can reference beyond its own declarations.
Start with one explicit source root, no ambient path search, no executable package
initializers, no wildcard imports, and no import cycles. These restrictions keep
resolution and initialization easy to understand and compilation inexpensive.
pub should govern access to functions, types, fields, and methods. Public classes
do not automatically expose all fields or methods. An automatically generated
field constructor should be public only when the class and all its fields are
public; otherwise construction needs an accessible initializer or factory.
Compiler-invoked destruction must still work for private destructors.
For a simple initial rule, privacy can be module-based, including class members. Class-private access could be added deliberately if wanted. Privacy controls source access; today’s structural printing and equality mean it is not a promise that representation details are secret. Public signatures also need checking for accidental exposure of private types.
Detailed resolution, visibility, and migration rules are in Entrypoints, modules, and executable tutorials.
Structural protocols without import-sensitive behavior
Structural protocols are a good fit. Python’s Protocol supplies useful precedent
for matching declared capabilities without requiring explicit inheritance; Go’s
interfaces offer another precedent. Plenty can make that matching entirely static
for generic calls. Neither requires importing an unrelated trait to make a method
appear. See the Python protocol specification
and Go interface types.
Recommend protocol Writer: with typed method requirements, and an explicitly
generic function such as def save[W: Writer](writer: &mut W) -> Result[(), IoError].
This is proposed syntax. A class satisfies the constraint when its accessible
methods match the required receiver, parameters, returns, borrowing, and failure
behavior. Imports only provide names; they never activate methods. An adapter
class can give an external type a different interface when necessary.
Start with protocols as generic constraints, not runtime values. Accepting a protocol-typed object through a vtable, or storing different implementations in one list, is a separate feature. This avoids silently introducing allocation or dynamic dispatch. Unknown generic values remain owned values; generic code cannot assume copying is free.
Check a generic body against its declared constraints, then specialize reachable concrete instantiations and cache them. Detect expanding recursive instantiation and measure generated-code growth. This costs some compile time, but remains a tractable first design without specialization, blanket implementations, or a large trait-resolution engine.
The protocols, generics, and context managers proposal contains the proposed matching and specialization rules.
Context managers and ?
Concrete context managers can precede generics. The compiler already knows a
concrete class’s methods and can check a with operation directly. Returning a
reference from __enter__ does require extending today’s borrowing rules: a hidden
owner must remain alive, the yielded borrow cannot escape it, and the borrow must
end before mutable exit/cleanup access.
The essential behavior is once-only cleanup on normal completion, return,
break, continue, and propagated failure. After successful entry, body locals
are cleaned up, __exit__ runs, then the hidden manager is dropped. An entry
failure must not call __exit__ as though entry had succeeded. Defer yielding from
inside a context until suspended-generator cleanup is specified.
Start with infallible __exit__ -> (), and make operations that can fail visibly
explicit: acquisition before entry, and flush, commit, or finish before exit.
The destructor remains a cleanup fallback. Fallible exit is possible later, but
needs an answer for simultaneous body and exit errors; silently discarding one
would be a poor default. An infallible signature also does not by itself prove
that arbitrary user cleanup never allocates.
Postfix ? is worthwhile, especially with fallible allocation. Start with direct
propagation: Result[T, E] into a function returning Result[U, E], and Option[T]
into Option[U]. Avoid implicit error conversions or cross-family propagation.
Evaluate the operand once, move its payload, and perform the same cleanup as an
explicit early return. This can be implemented against today’s sums first, but it
must not be described as OOM-safe until error propagation itself is allocation-free.
C libraries and trusted interfaces
One correction to the premise: Rust does support C shared libraries, exporting
cdylib libraries, and runtime loading. Dynamic linking is possible; the hard part
is establishing the lifetime, aliasing, ownership, and callback contracts that the
C ABI does not express. Rust documents its library linkage forms,
and libloading supports runtime loading.
Plenty can make that work more convenient with reusable interface files. Separate
the raw native signature from a normal Plenty-facing API, and generate or write
adapters between them. The interface is a trusted contract maintained once by a
binding author. Callers can then use ordinary ownership, borrowing, and Result
without repeating low-level boundary work at each call.
Those contracts need more than const: buffer lengths, nullability, retained
aliases, borrowed lifetimes, ownership transfer on success and failure, the exact
release function, error reporting, callback retention, and thread restrictions.
Annotations cannot prove that arbitrary native code honors its promises; relaxing
the rest of Plenty’s ownership rules would not solve that problem.
Cython’s declaration files and Zig’s explicit C ABI/layout facilities are useful models. C-header translation could eventually be a separate tool, with compiler flags and target information, but cannot infer all semantic contracts. It should not be a prerequisite for a small, useful initial FFI.
Keep Plenty’s str, class, collection, and enum representations private. Begin
with C scalars, opaque handles, and explicit pointer/length or C-string adapters.
Retain the matching allocator and release function for owned foreign resources.
Opt-in C-layout records can follow. pub visibility must remain separate from a
stable C export declaration.
Ordinary shared-library linking and runtime loading should be separate stages. Known signatures can be called indirectly after runtime loading with AOT code; this does not require reviving the JIT. Initially retain loaded libraries for the process lifetime, and defer retained or foreign-thread callbacks until their ownership and synchronization rules exist.
The FFI and dynamic libraries proposal documents the contract fields, current backend assumptions, and staged delivery plan.
Recoverable allocation is an architectural commitment
Rust already has fallible allocation-related operations such as
Vec::try_reserve.
Plenty’s opportunity is to apply a consistent recoverable policy across its own
language and runtime operations.
Today’s runtime cannot make that promise. Result and Option are managed,
heap-allocated sum values. Runtime type metadata, collection rebuilding, strings,
generators, and some structural equality operations also allocate. Even string
indexing and range construction need auditing. An allocation failure cannot
reliably be reported by attempting another allocation to construct Err.
First establish allocation-free finite sum/error lowering and static type
metadata. Keep that lowering contract internal and evolvable. Then audit every
allocation-capable operation, including cleanup and propagation. A small
AllocError must be representable without allocating or formatting a message.
Fallibility belongs to an operation’s static interface even when a particular
call happens to fit existing capacity. Literals, comprehensions, concatenation,
copying, class construction, and generator creation all need an explicit policy,
not just append. Whether their syntax yields Result directly or uses an
explicit allocation effect remains an open design choice; hidden automatic ?
would undermine the desired visibility of failure.
Failed mutations should preserve existing logical contents, and failed builders must release partially constructed results. Decide how failed insertion handles an owned input: consuming and dropping it is simple; returning it with the error needs a deliberate recovery API. Test every allocation position with fault injection, including attempts to report and clean up after failure.
Allocator selection is feasible without exposing a different list type for
every allocator. Recommend a default chosen at startup and an explicit constructor
argument for custom allocation. Each owner retains allocator provenance and keeps
the allocator alive until deallocation. Nested values retain their own allocation
provenance; inserting them does not magically relocate them. Resettable arenas
need additional lifetime constraints and should come later.
Fixed-capacity containers and inline-small containers are both useful but have different guarantees: the latter can still allocate after spilling. Neither needs to complicate the initial ordinary collection type. Budgeted allocators would make predictable service-level refusal practical, although operating-system overcommit means no language can guarantee recovery from every physical-memory exhaustion event.
The memory, parallelism, and SIMD proposal gives the allocation audit, failure guarantees, allocator design, and sources.
Parallelism and SIMD
Owned values make concurrency easier to reason about, but moving a collection does not prove all its storage is exclusive. Shared immutable strings and runtime metadata now use atomic counts and immutable descriptors respectively. Allocators, destructors, and foreign handles can also have thread affinity. These need compiler-recognized transfer/share rules and runtime changes before safe thread APIs.
Start with scoped threads that join on every exit, bounded channels with defined failure ownership, and explicit parallel map/collect/reduce operations using a configured pool. Define result order, error selection, cancellation, and cleanup before implementation. A thread-count setting should initially affect explicitly parallel work only.
Automatic parallelization is a later optimization requiring proof that observable behavior is preserved, including allocation failure, side effects, destruction, and floating-point reduction order. Structural method matching alone cannot prove those properties.
SIMD should have its own narrow, testable contract. Explicit small vector values and contiguous numeric buffers are a reasonable first step; current generic list storage is not automatically a contiguous native numeric array. Begin with operations the selected Cranelift targets can implement, define scalar-equivalent fallbacks and tail handling, and leave CPU dispatch and wider target-specific vectors for later. Do not promise that all backends support identical native instructions or that floating-point reassociation is invisible.
Runnable lessons and generated documentation
The proposed documentation workflow is workable. The tutorial already has direct executable coverage: 43 successful examples and 11 rejected examples. Its test checks output through both compile-and-run and explicitly compiled executables, and checks expected diagnostics for rejected examples. That test passed after the startup clarification in this change.
Standalone .plenty lessons would improve editing, discovery, and reuse. Give
each lesson a complete runnable program, prose in ordinary comments, and inert
markers selecting displayed code regions. An ordered manifest selects lessons;
sidecars can hold expected output or diagnostics. Show the complete main in the
first example and label later shortened snippets as excerpts with links to the
full source.
The documentation tool should run and validate the whole program against reviewed expectations before generating Markdown. It must not silently turn changed output into a new expected result. CI should check that the committed document matches the generated version. Compile-fail lessons and future multi-file module examples need first-class support too.
TinyTemplate can render the final document, but extraction, execution, validation, and ordering are separate work. Its default HTML escaping needs disabling for trusted Markdown. A simple renderer may be sufficient; the important design is a single authored source and independently checked behavior. Details and TinyTemplate references are in the tutorial workflow proposal.
Work tracking
The original cross-feature implementation sequence has been retired. The backlog is the single maintained list of remaining work and proposed order; the implementation status records the result. The sections above preserve the design reasoning and open questions from this review, not a current task queue.
Checked arithmetic blocks
Status: Implemented (issue #557). The numbers reference defines current behavior; this page records the decisions and rationale.
Problem
Integer overflow and division by zero trap. That is the right default when
they mean a bug, but code that takes sizes, counts or prices from input needs
to report them instead. Before this change the only recoverable options were
try_from after widening, or std.math helpers, which do not compose across
an expression.
Rejected: a Result from every operator
Local PR #678 made every integer + - * // % return Result and changed
existing code to .unwrap() or ? each one. It was abandoned unmerged:
- ordinary math became noisy, and most call sites only unwrapped;
- every operation carried a payload and a branch on the result, which was slower than the existing inline trap check;
- the tutorial had to teach
Resultbefore the first addition.
Its error types, float div_checked, T.MIN % -1 == 0, the overflow code
generation and constant-divisor branch elimination were kept.
Decision: an expression block
r = checked:
a * b + c
checked: opens an indented suite whose last expression is the value. The
block’s type is Result[T, E], where E is the
anonymous sum of
OverflowError[T] per operand width and ZeroDivisionError that its
operators can produce. The first failure exits the block with Err.
Outside any block, arithmetic compiles exactly as before, so the cost is paid only where it is asked for.
Lexical scope
Only arithmetic written in the block’s suite is checked. Calls, closures (including closures defined in the block) and other functions keep their own behavior. A dynamic rule, where a callee’s arithmetic depends on its caller, would need either two compilations of every function or a run-time mode flag on every operation, and would make a function’s failure behavior depend on code it cannot see. Comprehensions and compound assignment are lowered inline, so they are checked when written in the block.
Control flow
?, return, break and continue keep their meaning: the block catches
only arithmetic failures. A failure exits like ?: it releases the locals the
block declared and the temporaries on the operand stack, then produces Err.
There is no rollback. An outer variable assigned before the failure keeps the
new value, and a compound assignment whose operation failed does not write.
Rollback would need copies of every mutated value and would hide allocation.
Settled questions
| Question | Decision | Reason |
|---|---|---|
| Where a block may appear | Binding or assignment value, after return, or a function’s or case’s final expression | The positions where a match block produces a value; no new expression grammar |
checked as a keyword | Contextual: checked followed by : and a newline | Existing names such as checked: i64 = 0 keep working |
| Nesting | The innermost block catches | Each block’s error type stays local and predictable |
| A block with no failing operation | Compile-time error | Result with an empty error set would be misleading, and the block is likely misplaced |
| Every path exits | Compile-time error | The block has no value |
| Discarded block | Rejected as an unused Result | Same rule as any Result |
| Reference result | Rejected | Locals the block owns are released on failure |
x += 1, items[i] += 1 | Checked | Lexically arithmetic; the earlier rejection of compound assignment is not needed |
@newtype operators (issue #555) | Forwarded integer operators are checked and report OverflowError of the field type | Forwarding is inline lowering of the field’s operator; user-defined hooks are calls and keep trapping |
| Constant operands | Evaluated at run time inside a block; module constants still reject overflow at compile time | A block’s purpose is to produce the Err value |
| Constant divisor | Error type unchanged; impossible branches omitted | Types follow operators, not values, so editing a constant does not change a signature |
| Shifts and casts | Unchanged | Out of scope; try_from covers casts |
Negative pow exponent | Still traps | It has no integer result to wrap |
| Declared wider error | Kept when it accepts every possible failure | Lets a block be a function body returning a larger error set |
Code generation
Op::CheckedBlock holds the body and the result type; each checked operator
is Op::CheckedArith, carrying the operation, operand type and cleanup for
the block’s locals. Code generation computes the wrapped result and an
overflow flag with Cranelift’s *_overflow instructions, tests the divisor
for zero and the signed MIN // -1 case, and branches to a cold block that
builds the error, releases stack values above the block’s base, runs the
cleanup and jumps to the block’s exit. The success path wraps the value in
Ok. A flag known to be zero, such as the zero test for a nonzero constant
divisor, emits no branch.
Integer pow in a block calls plenty_int_pow_checked, which returns the
wrapped result and an overflow flag. Float div_checked is a one-operation
block.
Programs without checked: produce the same objects as before; the
comparison is recorded in the PR for issue #557.
Follow-up
- Class-destructuring patterns for
OverflowErrorlanded with issue #558:case Err(OverflowError(wrapped)); see class patterns. - Checked shifts and casts, if a use case needs them.
Rust compiler boundaries and measured performance
Status: Completed. Steps 1–6 are implemented (issues #691–#696), and the legacy stack syntax is removed (issue #698). Replayable module lowering remains a design. No speedup is claimed. The frontend reference and cache reference describe current behavior. Remaining scope and priority belong to B17/B19 in the backlog.
Assessment and scope
The compiler’s strongest properties are an independent typed-operation checker, deterministic parallel emission and diagnostics, explicit ownership checks, cycle-safe nominal metadata, and profiling-backed allocation decisions. Preserve them; the aim is code that is easier to change, not a different architecture.
The main concern is coupling: parsing, generic expansion, local inference, ownership provenance, editor recording and operation construction meet in a large frontend with shared mutable tables, and diagnostics travel between phases as rendered text. Improvements should make those dependencies explicit and independently testable.
The source map was rechecked against local main 6477751e (0.8.7). Step 6’s
profile is recorded in measured attribution;
no correctness audit was performed, and historical numbers below retain their
original scope. The runtime crate,
optional dataframe/linalg engines and standard-library adapters were not
audited. This is not an unsafe-code or security review. There is no proposed
learner-visible language change.
Design map
| Question | Detailed design |
|---|---|
| How should lowering, syntax and ambient state be separated? | Frontend and session boundaries |
| How should errors reach CLI, workers and editors? | Structured diagnostics |
| Which hot representations merit experiments? | Identity, type facts and dataflow |
| What must exist before caching lowered imports? | Replayable module results |
| How do we decide whether a change is worthwhile? | Validation and adoption |
These pages describe design dependencies and alternatives, not a second schedule. The backlog owns priority, and the shared local issues register owns defects, claims and PR state. Existing closed investigations remain closed unless new evidence justifies a separately scoped follow-up.
Implementation order
Each step is one local PR that preserves behavior unless it says otherwise.
- Structured diagnostics. The editor renders errors to text and reparses that text for positions; the parallel checker flattens errors to strings. Replace both with one sendable diagnostic value. This fixes a representation, not an IR.
- Scoped ambient state. Install the editor recorder through a guard that restores the previous state on return and unwind, and make teardown skipping a per-call option instead of a process-global flag.
- Split
src/op.rsby responsibility. It holds the IR definition, the historical stack-syntax compiler and the independent checker. Separate files give the checker an explicit boundary and keep the legacy compiler out of the IR module. - Separate the syntax layer from lowering. Tokens, the lexer, the syntax
tree and the core parser make up about a third of
src/frontend.rsand do not use lowering state. Moving them into child modules makes that independence checked by visibility rather than convention. - Group lowering state. Start with passive pipeline state on
Lower, then borrow/provenance state if the first extraction shows a clear gain. The pipeline grouping did not show one, so borrow/provenance state stays onLower. - Attribution before optimization. Function-name and nominal-type lookups are candidates only. Profile them and record the result; change the representation only on measured evidence. The measured attribution found neither material, so no representation changed.
Steps 1 and 2 both edit src/frontend/ide.rs; steps 1, 4 and 5 edit
src/frontend.rs. Land them in order rather than concurrently.
Replayable module lowering is a larger architectural opportunity, not a quick cache patch. Imported source dependencies, generic instances and inferred returned-reference facts must be reproducible before persistence is safe. It stays a design until steps 1–5 land and a std-heavy application shows the need.
Related plans
Several unimplemented plans meet these boundaries:
- B17 editor work (find references, document symbols, signature help and incremental module re-checking) consumes structured diagnostics and needs a recorder whose lifetime is explicit.
- B19 module caching needs the replay contract in replayable module results.
- B20’s GCC and Rust-source backends consume checked operations; a separate checker module and, later, function IDs in the IR affect what a second backend must reimplement.
- Language editions would need per-module options to flow through lowering, which explicit lowering state makes easier.
Existing work not to redo
The performance delivery record documents:
- issue #302: shared token/type children, callable signatures and FxHash tables;
- issue #303: borrowed loan-graph operations and reusable liveness buffers;
- issue #305: large-workload slowdown attributed to cache/TLB misses rather than increasing instruction work per unit;
- issue #315: batched codegen and mimalloc addressing allocator contention;
- issue #321: selective CLI teardown skipping; bump allocation was rejected;
- issue #317: syntax-tree and separately compiled module experiments did not justify adopting those cache designs.
Avoid blanket clone removal, replacing Arc with Rc, a global arena
rewrite, more threads without contention measurements, or replacing Cranelift
without matched workload evidence. These are not prerequisites for the proposed
boundaries. Compiler latency, generated-program performance and Rust build time
are distinct outcomes; report them separately.
Frontend and session boundaries
Parent proposal. The syntax/IR split, pipeline state and scoped ambient state are implemented (issues #692–#695); the session and further lowering-state options remain design sketches.
Source map
The table records the layout these steps started from, at 6477751e.
| File / symbol | Responsibility and coupling |
|---|---|
src/frontend.rs: Token, FileId, Lexer | Tokens, file interning and streaming item tokenization, about 400 lines at the root of the frontend. |
src/frontend.rs: Function, TypeRef, Stmt, Expr | The syntax tree, plus TypeRef::resolve, which turns written types into Ty against aliases and so belongs to resolution rather than syntax. |
src/frontend.rs: Parser | Core statement/expression parsing, about 1,650 lines. Feature modules (classes.rs, enums.rs, patterns.rs, collections.rs, fstrings.rs, protocols.rs, annotations.rs) add their own impl Parser blocks beside their impl Lower blocks. |
src/frontend.rs: Lower | Holds function names/slots, mutable function and generic tables, heap, immutable aliases/access, loans, captures, pipeline state, scopes and source provenance: about 50 fields. |
lower, lower_function, lower_body | Coordinate declaration expansion, registration, on-demand specialization, body lowering and finalization. |
src/frontend/functions.rs: Functions, Symbol | Indexed function table: signatures, declarations and returned-field facts. complete releases eligible syntax bodies. |
src/frontend/generics.rs: Engine | Instance preparation and pending/compiled work interact with ordinary lowering. |
src/frontend/references.rs, provenance.rs, retention.rs | Borrow origins and lifetime-related facts used during expression lowering. |
src/frontend/pipelines.rs and its children | Pipeline plans, callbacks, generator adaptation and specialized lowering. |
src/frontend/ide.rs | Ambient per-thread recording and error recovery alter analysis behavior. |
src/op.rs | The operation IR (about 1,350 lines), the historical stack-syntax compiler (about 600 lines, with src/lexer.rs) and the independent checker (about 1,250 lines). |
At 6477751e, src/frontend.rs was about 7,650 lines. After the moves it is
about 4,950, with frontend/lexer.rs, syntax.rs and parser.rs beside it;
src/op.rs keeps the IR (about 1,500 lines), with op/check.rs. The legacy
stack-syntax compiler, split into op/legacy.rs, was later removed with its
--legacy flag (issue #698). File length is evidence
of navigation cost, not a reason by itself to rewrite the frontend. A move is
worthwhile where it turns an existing independence into one the compiler
enforces; adding files without changing dependencies would not address the main
concern.
Syntax and IR files
The core parser does not read Lower, the heap, the generic engine or module
tables. Move tokens and the lexer, the syntax tree, and the core parser into
child modules of the frontend (for example frontend/lexer.rs,
frontend/syntax.rs and frontend/parser.rs). Keep the per-feature
impl Parser blocks where they are: grouping a feature’s parsing and lowering
together is deliberate. TypeRef::resolve and the builtin-type tables stay
with resolution. Visibility then shows which syntax items lowering uses.
Split src/op.rs into the IR (op.rs), the checker (op/check.rs) and the
legacy stack-syntax compiler (op/legacy.rs, with src/lexer.rs). The checker
then depends on the IR’s public surface only, which is the contract a second
backend would also consume. Removing the legacy syntax entirely (the CLI’s
--legacy flag and the backend tests written in it) was a separate product
decision, not part of this split; issue #698 made it.
These are pure moves: no behavior change, and the diff should review as one.
Lowering state
Separate state by lifetime and responsibility, not by matching every syntax variant with a trait:
- Compilation environment: aliases, access policy, source identity and shared function/type tables. Distinguish read-only inputs from mutable specialization/string services. Do not place every service behind a mutex just to make the context shareable.
- Function state: current function identity, parameters, local slots,
bindings, return inference and scope stacks. Preserve the different meanings
of module scope, unit return and a value return;
Option<Type>currently encodes those distinctions. - Borrow/provenance state: loans, reference-local associations, return origins, value origins and slot names. Expose operations that update related facts together rather than public mutable field collections.
- Pipeline state: plans, resumed/end flags, lent parameters, callback attribution and consumption state. Give this state explicit entry/exit operations before moving pipeline lowering methods.
The first extraction, frontend::pipelines::State, is passive pipeline state. Its nine fields are read and
written almost only by src/frontend/pipelines/ (plus three sites in
references.rs, generators.rs and the generator bridge), and every function
starts with them empty. Group them in one struct owned by the pipelines module,
with a Default initializer, while retaining methods on Lower. The struct’s
documentation then says what the pipeline code owns, and Lower’s
construction loses nine lines. Move operations only when their actual
dependencies are understood. Borrow/provenance state is used throughout
expression lowering and is more correctness-sensitive; extract it only if the
pipeline step shows a clear reduction in coupling. Grouping pipeline state
documented ownership but did not change who calls what, so borrow/provenance
state was not extracted.
The goal is that a reviewer can identify which service a lowering operation may mutate. Avoid a giant context with unrestricted mutable access, cosmetic substructures whose callers still mutate every field, or a visitor/trait framework without multiple real implementations.
Keep independent op::check validation. A future validated-program wrapper
could make backend entry requirements explicit, but must account for passes
that transform operations after checking. A type name alone does not prove
that transformed IR remains valid.
Ambient state and lifetime policy
Three mechanisms deserved separate treatment; the recommended smallest change below is implemented (issue #692). This list records the starting state:
src/frontend.rs(nowfrontend/lexer.rs): process-globalFILES: RwLock<Files>maps paths to compactFileIds. It intentionally avoids atomic path clones per token. It retains each distinct path for the process lifetime.src/frontend/ide.rs: thread-localACTIVEandRECORDER;analyzeenables recording, catches unwinds, disables recording and takes the result. A nestedanalyzeon the same thread would overwrite the outer recorder and then switch recording off for the rest of the outer analysis.src/lib.rs: publicskip_teardown()sets a process-global atomic.src/compiler_cli.rscalls it for one-shot checks; the language server,plenty build,plenty testand library callers must not.
These are design constraints, not reproduced leaks or crashes.
Options
Smallest change (recommended): retain the file interner. Install the
recorder through a guard that saves the previous ACTIVE/RECORDER and
restores them on drop, so nesting and unwinding are both correct. Replace
skip_teardown() with a CheckOptions field set by the CLI; a library caller
then cannot change another caller’s policy. This changes a public function, so
the PR must say so; the CLI is its only caller in this repository.
Broader change: a compilation session owns source identities and analysis services, with per-invocation options. Default library APIs create a normal, fully reclaimed session; the LSP can intentionally reuse one. Token IDs remain compact and every escaped analysis/artifact retains or translates its source identity.
Prefer the smallest change unless bounded source retention or session reuse has a measured consumer. A session is not automatically a cache, a scheduler, or a requirement to make lowering concurrent.
Required invariants
No file ID may resolve through the wrong session or outlive its source table. Do not recycle IDs while retained diagnostics, ASTs or operations refer to them. Keep cached artifact identities independent of process-local file numbers. Distinguish source-path interning from retaining source contents.
Teardown skipping must remain an explicit one-shot CLI optimization, never a default for embedding.
Test nested analysis and unwind restoration of the recorder. Measure retained paths/allocations separately from allocator RSS, which may remain high after correct destruction.
Structured compiler diagnostics
Parent proposal. The diagnostic contract here is proposed; existing CLI spelling remains the compatibility baseline.
Current conversion chain
src/frontend.rs defines Located with a location string, message and
optional ownership Demand. Attributed marks a generic diagnostic already
attributed to its proper instance, but holds only rendered text. Token::error
constructs located errors. sendable preserves those two concrete types across
worker boundaries and renders other boxed errors to strings.
src/op.rs::check_with converts function-check errors with to_string()
to collect parallel results, then selects the first error in program order.
Ownership checking builds richer messages from origins and demand information
in src/ownership.rs, but returns them as text.
In src/frontend/ide.rs, recover stores deduplicated strings in
Recorder.diagnostics. During finish, diagnostic searches the first
line for a source:line:column: prefix and parses it back into a public IDE
Diagnostic, which already exposes file, line, column and message.
src/lsp.rs::check converts these into client diagnostics.
The problem is loss of structure between producers and consumers, not the
absence of a public location structure. Human-readable text is functioning as
an internal interchange format, and the reparse can misplace a message whose
text happens to contain name:1:2: .
Recommended representation
Introduce one internal diagnostic type, outside the IDE module, carrying:
- the source path, line and column, each optional: an error with no source position (I/O, linking, a legacy-syntax error) has none;
- the message text;
- whether generic attribution has already happened (today’s
Attributed); - the ownership
Demand, where callers currently inspect it.
Do not record an end range: tokens record a start position, not an expression’s extent, and a fabricated span is worse than none. Related locations, diagnostic codes and fix suggestions wait for a consumer.
Display stays path:line:column: message, so CLI text and existing test
snapshots are unchanged. Keep source positions in the compiler’s coordinate
convention and convert to negotiated UTF-8/UTF-16/UTF-32 units only in the LSP
adapter. Ty, and therefore Demand, is already Send + Sync, so the type
can cross worker threads without conversion.
Error-type alternatives
Preferred incremental option: the concrete diagnostic boxed inside the
existing Box<dyn Error> results, recovered by downcast at the boundaries
that need structure (worker join, editor recording). External failures stay
ordinary boxed errors without positions.
Alternative: a compiler-error enum distinguishing diagnostics, I/O and
backend/link failures. It gives exhaustive handling but means migrating every
Result signature in the compiler for no current consumer. Adopt only if
callers need those distinctions.
Do not add a diagnostic framework dependency to remove these string conversions.
Migration
- Replace
LocatedandAttributedwith the new type;Token::erroranddemand_errorconstruct it. Keep rendered messages identical. - Make the worker boundary in
check_withandsendablekeep the value instead of flattening it. Preserve program-order error selection. - Store structured diagnostics in the editor recorder and convert them to the
public
Diagnosticdirectly. An error that is not a compiler diagnostic has no position; never guess one from its text.
Deduplication needs a deliberate key: position and message. Two different source sites with the same wording must not collapse; the same error recovered twice must. Test both.
Acceptance
Use tests/test_lsp.rs, test_python_habit_diagnostics.rs,
test_callable_diagnostics.rs, test_storage_diagnostics.rs,
test_recursive_tail_errors.rs, and generic/borrowing regressions.
Add focused tests for:
- a message whose text contains a
path:line:column:lookalike; - colons and spaces in paths;
- identical wording at distinct sites and true duplicate recovery;
- generic definition/use attribution and ownership demand inspection;
- the same diagnostic at one and many workers.
The refactor succeeds when internal text reparsing is removed for compiler diagnostics while CLI text, attribution and diagnostic order remain unchanged. No speedup is claimed.
Identity, type facts and dataflow
Parent proposal. These are experiments to evaluate, not claims that every inspected lookup is a bottleneck.
Function identity across phase boundaries
src/frontend/functions.rs already has Symbol(u32), a name-to-symbol map
and indexed entries for signatures/declarations. Returned-field facts are keyed
by symbol. Do not propose implementing this table again.
The remaining boundary is visible in src/op.rs: DefineFn(String, ...),
Call(String) and TailCall(String) retain function names. check_with
collects Sigs = HashMap<&str, &FnSig>. This table borrows names and
signatures; it does not copy every signature. src/codegen/symbols.rs
also builds a name map when snapshotting declared Cranelift functions, while
its function declarations are already indexed by Cranelift FuncId.
Options to compare
- Keep the current maps as control. Measure signature collection, call lookup and whole-program checking, rather than counting map sites.
- Add a checker-local indexed view, retaining name-bearing operations. This isolates table construction/layout experiments, but calls still require name resolution unless translated once.
- Carry a compiler-owned function ID through modern IR and use indexed signatures/call targets. Retain names for display, exports and external linkage.
- Compare interning all identifiers only if function IDs leave substantial measured name cost. This is a larger parser/AST/environment change, not a natural consequence of option 3.
Prefer a bounded checker experiment before changing the entire IR. For ID-based
IR, use a neutral identity module rather than exposing a frontend-private
Symbol as the backend contract.
IDs must identify the owning program/table. Never serialize a transient index as a stable cache identity without deterministic remapping. Preserve generic instance identity, function ordering, dump names, exported symbols and cache-on/off object identity. Avoid a refactor that saves lookups but adds a second full copy of every body.
Nominal type access and fact caching
src/nominal.rs::Types owns mutex-protected class/enum slot vectors and a
name-keyed facts map. Nominal distinguishes local/stored definitions and
strong/weak table links. get returns an Arc; try_get upgrades a
slot and clones the finalized definition handle. Escaping stored fields promote
weak edges so recursive declarations do not create ownership cycles.
src/type_facts.rs::Ty::facts already handles simple types directly,
combines collection/box content facts, and caches complete class/enum facts.
Nominal::define clears table facts as definitions become available.
Incomplete recursive graphs must not acquire permanent final facts.
Explore these progressively:
- Attribute atomic clone/upgrade traffic, facts-map hashing and locking with sampling/counters. A lock in source is not evidence of contention.
- Compare indexed per-definition fact storage against the shared name map. Specify how finalizing any dependency invalidates prior results.
- Compare a build-then-freeze definition table, with borrowed read-only access during checking/emission. First identify on-demand specialization boundaries: freezing too early prevents legitimate instance construction.
- Consider broader type IDs only after these smaller experiments. Structural
types, local builtin sums and escaped recursive definitions make a wholesale
type arena substantially more complex than replacing an
Arc<Ty>.
An epoch scheme could preserve invalidation with indexed caches; a finalized
immutable graph might permit one-time facts. Neither design is safe merely
because the field is called OnceLock. Prove that no definition consumed by
the cached fact can change within that phase.
Retain cycle release, weak-edge promotion, stable display names and complete
graph checks. Use src/nominal/clone_tests.rs, nominal unit tests,
tests/test_recursive_data.rs, test_type_cycles.rs,
test_generic_data_safety.rs and concurrent compiler checks.
Measured attribution (issue #696)
Sampled on 2026-11-02 at the head of the step 1–5 branches, with a release
build using frame pointers, perf record -e cpu-clock on eight physical cores
and the compiled-function cache off. Shares are of all samples, by self time.
| Family | Generated check | Std-heavy check |
|---|---|---|
collect_sigs (building the name-keyed map) | 1.5% | 0.7% |
| Lookups in that map | 0.1% | <0.1% |
Nominal handle clone, get and drop | 5.6% | 5.6% |
| Type facts, including cached nominal facts | 0.1% | 0.2% |
Mutex/futex/Once | 0.2% | 0.1% |
Ty clone and drop | 4.1% | 4.3% |
| mimalloc | 9.3% | 13.2% |
The generated check is compile_bench 1000 20 --workload check --no-warmup
(39,003 lines; median 138 ms per check with frame pointers). The std-heavy check
is 40 plenty --check runs of a program importing std.http.server,
std.http.client, std.db.sqlite, std.json, std.csv, std.toml,
std.markdown, std.xml, std.url, std.regex, std.time and std.text
(about 100 ms each).
Conclusions:
- Function identity: the signature map costs at most about 1.7% of a check, almost
all of it the walk that collects signatures rather than hashing or lookup.
Function IDs in the IR (options 2–4) would not pay for their migration.
Revisit only if a workload attributes a material share to
collect_sigsor call lookup. - Nominal facts and locking: fact computation and locks are below 0.3%. Indexed fact storage, epochs and a frozen table are not justified by cost.
- Nominal handles: clone/
get/drop traffic is about 5.6%, spread across many callers:Ty::clonein lowering, generator measurement and validation. Together withTyclone/drop and allocation it is a cost of the type representation, not of the table. A change here is the broader type-ID or clone-reduction work this page already defers, and it needs its own design and matched timings.
The memcmp, memmove and allocator shares are larger than any family above.
memcmp is mostly under ownership::check (1.7% of the std-heavy run); it was
not investigated further here.
Ownership dataflow: conditional follow-up only
Checker::loans in src/ownership.rs skips loan-free bodies, borrows operations
in its CFG and reuses sorted Live buffers. Its liveness fixed point scans
reachable nodes until no set changes. Issue #303 already removed operation
cloning and per-round tree allocation; do not undo those improvements.
Only if loan-heavy workloads attribute significant time here, compare:
- a predecessor worklist that revisits nodes affected by changed successors;
- dense bitsets for functions with many dense loan IDs, against sorted sparse buffers for small/sparse sets;
- buffer reuse across functions without retaining excessive peak storage.
Measure nodes, edges, loans, rounds, set sizes and allocations. Full scans can beat worklists on small functions; dense bitsets can waste memory. Any hybrid threshold must come from workload evidence.
A traversal helper may reduce duplicated nested-block enumeration
(in_body, has_loans, codegen walkers), but passes differ in cleanup,
nested-function and control-flow semantics. Share only a precisely specified
traversal with tests for each operation family, not one universal visitor that
silently changes the graph.
Replayable module-lowering results
Parent proposal. No lowered-module cache is implemented by this design. The module-cache investigation is the evidence baseline; this page describes the boundary it requires.
Evidence and limits
The recorded investigation at 00d32a13, on an AMD Ryzen 7 7840HS with
seven repetitions and pinned physical cores, measured a synthetic ten-module
application importing std.http.server:
| Object-build mode | Median |
|---|---|
| Cache disabled | 45.50 ms |
| Warm function cache | 44.54 ms |
| Unchanged program cache | 3.99 ms |
| Ordinary body edit | 44.00 ms |
These exclude native linking and are not current-main benchmark results. The body-edit profile recorded parsing 8.35 ms, resolution 3.32 ms and program lowering 19.28 ms; lower-function time is contained in lowering. Do not sum nested rows or promise the sum as achievable savings.
A scalar separate-compilation spike was slower than the normal compiler. There is no evidence that independently packaging current modules as native libraries solves frontend reuse. Existing whole-program and compiled-function caches remain the default.
Relevant implementation boundaries
src/frontend/modules.rsand its children load/resolve imports and produceResolved; source identities and access maps are part of that environment.src/frontend.rs::lowerconsumesResolvedbut mutates shared string, function and generic state while producing oneProgram.src/frontend/generics.rsandfunctions.rsown pending instances and body-derived facts.src/frontend/references.rsinfers returned-field projections that callers use for borrow checking.src/cache.rs,src/cache/program.rs,src/cache/compiled.rsandsrc/cache/executable.rscover existing cache layers; a module artifact must not silently reuse incompatible keys or change cache-off behavior.
Proposed replay contract
A result needs explicit inputs, outputs and identity relocation, not simply a
serialized Vec<Op>.
| Part | Required information |
|---|---|
| Inputs | Canonical module identity/text, import resolution, compiler/settings identity and imported semantic dependencies |
| Declarations | Signatures, aliases, nominal layouts, visibility, protocols and generated methods |
| Definitions | Generic templates and instantiated bodies, including their dependency provenance |
| Derived interface | Returned-reference projections and any inferred capture/effect facts consumed by other bodies |
| Lowered content | Typed operations, referenced strings/types/functions and native substitutions |
| Attribution | Source identities, original declaration positions, instantiation origins and deterministic diagnostic order |
Replay must remap local string/type/function/source identities into the new environment in a deterministic order. Reject unsupported combinations rather than quietly sharing mutable tables between saved and replayed compilations. Handle cyclic nominal graphs without recursive serialization loops.
First extract an in-memory result for a restricted, explicitly documented module class. Replay it against a fresh imported environment and compare both accepted and rejected programs with normal lowering. Keeping all original global tables alive and returning pointers into them does not demonstrate independent replay.
Conservative full imported-source digests are acceptable initially. Precise interface invalidation is a separate refinement: a body can change semantics that callers need without changing its written signature.
Critical invalidation probe
The existing experiments/module-cache/check_dependencies.py changes an
imported borrow-returning function from lending pair.x to lending
pair.y, keeping select(pair: &mut Pair) -> &mut i64 unchanged.
An unchanged caller mutating pair.y must change from accepted to rejected.
Retain this probe, generic-body edits, imported layout/visibility changes, import-target replacement, missing/deleted files and native substitution changes. Compare diagnostic attribution as well as executable output.
Alternatives and adoption gate
A conservative lowered-import result is the primary experiment. Separately compare lowering only reachable standard-library functions, checking whether on-demand instantiation and diagnostics remain consistent. Streaming functions can reduce peak memory but does not by itself create a module cache; mutually dependent inference/effects must still complete before consumers need them.
Only add persistence after fresh-environment replay is correct and faster on a real std-heavy application. Account for hashing, validation, decoding, relocation and merge allocations. A large theoretical reusable frontend share does not establish net benefit. Corrupt/version-mismatched artifacts must miss safely; invalidation must not suppress errors. No remote cache follows without a separately agreed trust contract.
Validation and adoption
Parent proposal. These are acceptance criteria for possible implementations, not checks claimed to have run for this assessment.
Behavior-preserving changes
Each implementation should have a bounded responsibility, its own local PR and a clear comparison against its base. Do not combine diagnostic migration, function IDs, type storage and module persistence in one refactor.
For Rust changes, run formatting, strict workspace/all-target Clippy and the full workspace/all-feature suite following the development workflow. Targeted tests help development but do not replace the full handoff checks.
| Boundary | Existing coverage to retain / extend |
|---|---|
| Diagnostics and editor | tests/test_lsp.rs, diagnostic-specific files, generic instantiation and borrowing diagnostics |
| Function IDs / IR | src/op.rs unit tests, tests/test_generics.rs, test_recursive_tail_errors.rs, test_compile_cache.rs, test_compiler_parallelism.rs |
| Nominal facts | Nominal clone/cycle unit tests, test_recursive_data.rs, test_type_cycles.rs, test_generic_data_safety.rs |
| Ownership worklists | src/ownership/graph_tests.rs, test_borrowing.rs, test_call_borrowing.rs, test_borrowed_match.rs |
| Module replay | test_modules.rs, test_compile_cache.rs, module-edit harness and returned-projection dependency probe |
| Session state | Repeated library checks, editor recovery, independent concurrent sessions and unwind/nesting policy |
| File moves (syntax, IR, checker) | Full suite; no behavior change; review with git diff --color-moved so only visibility and use lines read as edits |
Add regression tests for the actual new boundary rather than only testing through the CLI. Preserve error order across workers, object identity where the existing contract requires it, source/trap attribution, runtime cleanup, generic limits and invalid-program rejection.
Measurement matrix
Follow profiling and module-edit measurements. Use an isolated target directory/cache directory so profiling builds do not replace another worker’s compiler or contaminate warm-cache comparisons.
Measure at least:
- small checks and larger generated programs across multiple sizes;
- ordinary and generic-heavy checking;
- std-heavy imports and a representative application;
- cache-off object emission, warm-function emission, unchanged-program hits, body edits, interface edits and generic edits;
- one worker/core and the intended multicore configuration.
Use release, feature-disabled compilers for final latency comparisons.
Use phase profiling for attribution, profile-alloc for allocation traffic,
and a sampling profiler for uninstrumented leaf costs. Inclusive rows cannot
be added as independent phase costs. Integration tests use a different
allocator path from compiler executables; do not interpret their wall time as
the CLI’s allocation performance.
Record exact compiler commits, Rust/Cranelift versions, CPU/affinity, allocator, flags, source size, cache state, warm-up policy and every timed sample. Alternate baseline/candidate runs where possible. Report median, dispersion, user/system CPU, peak RSS, allocation bytes/counts, instructions and cache/TLB counters when available. Explain unavailable counters. Keep linking separate from object emission.
Decision rules
- A structural refactor can be adopted without a speedup if it measurably simplifies ownership/dependencies and preserves behavior without a material performance or memory regression. Review the resulting API, not only file size or line count.
- A performance change needs a repeatable end-to-end benefit beyond observed noise, not merely a smaller microbenchmark. Identify any small-program, multicore, memory, artifact-size or generated-code tradeoff explicitly.
- A module replay design needs correct invalidation on rejected as well as accepted inputs, plus net savings after replay costs.
- A session change needs defined lifetimes and ordinary destruction for embedding. Correct destruction does not require the allocator to return every freed page to the operating system immediately.
- A failed experiment is useful evidence. Keep its method/results and reject the complexity rather than weakening checks or moving the benchmark target.
No fixed percentage is promised. Agree on workload-specific acceptance and acceptable regressions before implementing an invasive performance design.
Documentation and handoff
Keep reference pages describing shipped behavior until implementation lands. Then update the frontend/cache/library API references actually affected. Update learner material only for observable changes; do not manufacture tutorial changes for an internal representation refactor.
Run all three docs CI checks for book changes: source links, Python helper tests and the complete versioned-site builder with the pinned mdBook. In each PR, state what ran and what did not. Record delivered scope in completed-work records and narrow the backlog, without marking a design proposal as a shipped compiler optimization.
Python slice syntax
Status: Implemented (issue #579). Copying slices of lists, text, Bytes,
bytes, tuples and clists shipped as one change rather than the planned
sequence; the slices reference
owns the contract. Shipped differently: tuple slices of affine components and
slices of temporary clists are rejected; Python-style copy() of such
components is not implemented. Element types containing Shared fields copy
with their handles shared since issue #582. Remaining: borrowed views &xs[a:b] (issue #581, B39/B11),
negative tuple indices (issue #583), and first-class slice(...) values. The
original text below was researched against local main at f681a0d5 on
2026-10-30; the implementation and documentation plan
is a companion page.
Problem
Slicing is among Python’s most used features, and Plenty’s Python-shaped syntax
makes people and coding agents reach for it first. B39 left it out because
buffer[a:b] would hide a copy. This proposal keeps Python’s syntax and
results, and makes the copy visible the way list literals do: a slice returns
Result[_, AllocError] and needs ?.
def main() -> Result[(), AllocError | IoError]:
xs = [0, 1, 2, 3, 4]?
print(xs[1:3]?, xs[-2:]?, xs[::-1]?)? # [1, 2] [3, 4] [4, 3, 2, 1, 0]
print("Aé🙂Z"[1:3]?)? # é🙂
Ok(())
Decisions
These were settled with the user on the issue and in review of local PR #694:
- A slice that produces a new value is fallible:
ys = xs[1:3]?, typedResult[_, AllocError]. - Elements are duplicated with
copy()semantics: deep and fallible in the sameAllocError.Shared[T]elements are duplicated withshare(). Element types thatcopy()rejects are a compile error naming the type. - Copying slices come first. Borrowed views will be spelled
&xs[1:3]; the syntax reserves that form. - Out-of-range bounds clamp exactly as in Python, including negative indices and negative steps.
- First-class
slice(...)values are deferred. - Receivers:
list,str,Bytes,bytes, tuples andclist. - Every slice returns a
Result, tuples included, for consistency. The compiler may remove work it can prove unnecessary. - A
bytesslice returns an ownedBytes, with no import needed. copy()will duplicate nestedSharedhandles withshare()(issue #582). Slices of elements that containSharedfields depend on that change.- Tuple indexing will accept negative literals, as in Python:
t[-1](issue #583). strslices always copy.&and&mutapplied to a slice are rejected until borrowed views exist (issue #581).str.slice()stays until slice syntax lands; whether to deprecate it is decided then, by its remaining use.- The step has a nonzero integer type, as
rangesteps will (issue #584), so a zero step cannot be written or computed. See the step.
Plenty has no shared mutable aliasing, so a deep copy differs from Python’s
shallow copy only where a Python program mutates an element through one list
and reads it through the other. Plenty can express that only through
Shared[Mutex[T]] elements, and slices share() those, which gives Python’s
reference behavior.
Syntax
A subscript whose brackets contain a top-level : is a slice:
subscript = primary "[" expression "]" # index, unchanged
slice = primary "[" [start] ":" [stop] [":" [step]] "]"
xs[:], xs[::], xs[a:], xs[:b], xs[::c] and xs[a:b:c] are all
slices. A comma inside slice brackets (m[1:2, 3]) is rejected; there are no
multi-dimensional subscripts. The parser builds a new Slice expression
rather than reusing Index, so later passes cannot confuse the two.
A slice is a postfix operation with the same precedence as indexing, calls,
member access and ?. These parse as written:
| Source | Reading |
|---|---|
xs[1:3]? | Copy, then propagate allocation failure |
xs[1:3]?[0] | Index the copy |
len(xs[1:3]?) | Length of the copy |
s[1:]?.upper()? | Method on the copied text |
rows[0][1:]? | Slice an element of a nested list |
&xs[1:3] | Reserved for a borrowed view; rejected for now |
No conflict with generic brackets
NAME[...] is ambiguous in today’s parser: brackets after a name or member path
first parse as type arguments, and module resolution later picks the generic or
index reading. Issue #562 is a case where it picks wrongly: TABLE[n] on a
module constant is read as a generic call. A type-argument list can never
contain :, so type_arguments fails at the colon and the parser falls back
to the subscript path. No Ambiguous node is built, and TABLE[n:] works even
while TABLE[n] does not. Issue #562 remains a separate fix.
The only other expressions with a colon are anonymous functions,
def(x: i64) -> i64: x + 1, whose parser consumes its own colons, and dict
literals, which are inside braces. Neither can end a bound early.
Bounds and normalization
Bounds are i64 expressions, or references to i64, as indices are.
Unsuffixed literals infer i64; other integer types need an explicit cast such
as i64(n). The step is a nonzero i64, described in the step;
an omitted step is 1. Omitted bounds depend on the sign of the step, as in
Python. Positions count elements, Unicode scalars for str, bytes
for Bytes and bytes, and rows for clist.
Normalization follows CPython’s PySlice_Unpack and PySlice_AdjustIndices.
This algorithm is normative:
positions(length, start: Option[i64], stop: Option[i64], step: NonZero[i64]):
# length >= 0. Arithmetic is exact: compute in i128, or first clamp a step
# of i64.MIN to -i64.MAX as CPython does. Both give the same positions,
# since a step of magnitude >= length selects at most one element.
# step != 0 by its type; there is no zero case.
if step > 0:
low, high = 0, length # clamp targets
first, last = 0, length # omitted start, omitted stop
else:
low, high = -1, length - 1
first, last = length - 1, -1 # -1 here means "before 0",
# not the last element
adjust(bound):
if bound < 0:
bound = bound + length
if bound < 0:
bound = low
elif bound >= length:
bound = high
return bound
s = adjust(start) if start is present else first
e = adjust(stop) if stop is present else last
if step > 0:
count = (e - s - 1) // step + 1 if s < e else 0
else:
count = (s - e - 1) // (-step) + 1 if e < s else 0
return s, s + step, s + 2 * step, ... (count positions)
The existing runtime helper slice_bounds in plenty-runtime/src/aggregates.rs
is the step-1 case of this algorithm; the implementation generalizes it.
These are the issue’s examples with a = [0, 1, 2, 3, 4]. Each result matches
CPython 3, checked with slice.indices while writing this page.
| Expression | Meaning | Normalized (s, e, step) | Result |
|---|---|---|---|
a[1:3] | start through stop - 1 | (1, 3, 1) | [1, 2] |
a[1:] | start through the end | (1, 5, 1) | [1, 2, 3, 4] |
a[:3] | Beginning through stop - 1 | (0, 3, 1) | [0, 1, 2] |
a[:] | A copy of the whole sequence | (0, 5, 1) | [0, 1, 2, 3, 4] |
a[1:4:2] | start, not past stop, by step | (1, 4, 2) | [1, 3] |
a[::2] | Every second item | (0, 5, 2) | [0, 2, 4] |
a[-1] | Last item: an index, not a slice | — | 4 |
a[-2:] | Last two items | (3, 5, 1) | [3, 4] |
a[:-2] | All except the last two | (0, 3, 1) | [0, 1, 2] |
a[::-1] | All items, reversed | (4, -1, -1) | [4, 3, 2, 1, 0] |
a[1::-1] | First two items, reversed | (1, -1, -1) | [1, 0] |
a[:-3:-1] | Last two items, reversed | (4, 2, -1) | [4, 3] |
a[-3::-1] | All except the last two, reversed | (2, -1, -1) | [2, 1, 0] |
a[4:1:-1] | Down from 4, not reaching 1 | (4, 1, -1) | [4, 3, 2] |
a[10:20] | Past the end: clamps | (5, 5, 1) | [] |
a[-100:100] | Both sides clamp | (0, 5, 1) | [0, 1, 2, 3, 4] |
a[3:1] | Empty, not reversed | (3, 1, 1) | [] |
a[::0] | Zero step | — | compile error |
Why clamp
Indexing traps on a missing position, and the Bytes methods return
OutOfRange. Slices clamp instead because that is Python’s documented rule,
which users already rely on: line[:80] truncates, and a[-3:] takes at most
three. A trapping slice would turn correct Python habits into crashes. Clamping
is total, so clamping alone never causes an error. The existing slice methods
of lists and text already clamp.
The step
A zero step selects no well-defined positions. Python raises ValueError, and
Plenty’s range traps today. Neither suits Plenty’s reliability goal: a zero
computed elsewhere, such as n // k, must not stop the program where a slice
uses it. So the step’s type cannot hold zero. It is the nonzero integer type of
issue #584, called NonZero[i64] here; its exact construction API is decided
in that issue. range steps use the same type.
- An integer literal step converts at compile time:
xs[::2],xs[::-1]andxs[1:10:3]need no conversion. - A literal
0, or a constant that evaluates to zero, is a compile error. - A step that is a run-time
i64needs the explicit, fallible conversion that issue #584 defines, and the program handles the zero case where the value is created. ANonZero[i64]value can be passed directly. - An omitted step is
1.
The check happens once, where the value is created, not at each slice. A slice
has no zero-step trap and no zero-step error, and its type stays
Result[_, AllocError]. Slice syntax with a written step depends on issue
#584; slices without a step do not.
Results by receiver
The receiver may be an owned value, a named place, a field, an element, or a
& or &mut reference. A slice only observes it: the source is unchanged, and
a mutable receiver is borrowed shared for the duration of the copy.
| Receiver | x[a:b:c] type | Selected values are duplicated by |
|---|---|---|
list[T] | Result[list[T], AllocError] | copy() semantics; share() for Shared[T] |
str | Result[str, AllocError] | Copying the selected scalars’ UTF-8 |
std.bytes.Bytes | Result[Bytes, AllocError] | Copying bytes |
bytes | Result[Bytes, AllocError] | Copying bytes out of read-only data |
tuple[...] | Result[tuple[...selected], AllocError] | copy() semantics per component |
clist[T] | Result[clist[T], AllocError] | Copying column values |
Every slice returns a Result, including empty slices, tuple slices of
copyable components such as (1, 2, 3)[1:], and short strings that happen not
to allocate. One rule is easier to learn and to keep in mind than a list of
exceptions, and code does not change type when a bound changes. The compiler
may remove work it can prove unnecessary, such as the error path of a tuple
slice that cannot allocate.
Other receivers are rejected: dict, set, deques, ranges, generators,
Shared[list[T]] handles (slice &*handle once views exist, or index it), and
user classes. There is no __getitem__-style hook for slices yet.
Lists
Each selected element is duplicated by its type’s copy:
- Scalars,
@copyclasses and copyable aggregates are copied inline. strelements share their immutable storage, ascopy()does.- Affine elements that
copy()accepts, such as nested lists, dictionaries, sets, tuples of those, and classes without__del__, are deep-copied. This is what makesrows[1:]?work for the common nested data. Shared[T]elements are duplicated withshare(): a count increment, no allocation. The handles in both lists name the same values.- Aggregates that contain
Sharedfields, such as(str, Shared[Log]), are copied with their handles shared oncecopy()does that (issue #582). Until then they are rejected, ascopy()rejects them. - Any other element type is a compile error naming the type. These include
Bytesand other resources, generators, closures with captures, types with__del__, and aggregates that contain any of them.
For an owned temporary receiver, as in load()[1:]?, the implementation may
move the selected elements instead of copying them and drop the rest. Types
whose destruction is observable are rejected anyway, so the only difference is
fewer allocations. The existing .slice(start, stop) method keeps that
transfer as its contract and stays the way to take affine, non-copyable
elements out of a temporary.
Generic code is checked per specialization, so xs[1:]? in a function over
&list[T] reports a non-copyable T at the instantiation, naming the concrete
type.
Strings
Positions are Unicode scalars, as for len(text) and text[i]; they are not
bytes and not grapheme clusters. "Aé🙂Z"[::-1] is "Z🙂éA", and a combining
mark is reversed separately from its base.
A step-1 string slice copies. It does not share the source’s storage:
- A
stris one word pointing at a header followed by its UTF-8 bytes (see value representations). A substring that shares storage needs an offset and length, either a second word or a new header kind. Every string operation, hash and comparison would pay to check for it. - A shared substring keeps its whole source alive. A ten-character field from a
100 MB file would hold the file. Java changed
String.substringto copy in JDK 7u6 for this reason, and Go substrings have the same retention cost. That conflicts with low memory use. - The string design chose one public
strwith no owning/view distinction. A view belongs to&s[a:b], a separate type that borrows.
So the rule is the same for every str slice: it returns
Result[str, AllocError]. Two cases allocate nothing as an optimization, not
as a contract: results of at most seven UTF-8 bytes are stored inline (the
runtime’s try_build already does this), and a whole-string step-1 slice may
retain the source’s immutable storage instead of copying it.
Bytes and bytes
buffer[a:b:c] on a Bytes returns a new Bytes with exact capacity, like
the slice method. The syntax clamps; the methods keep the
range rule, under which an
out-of-range bound returns BytesError.OutOfRange. Index syntax already
follows list rules rather than method rules, so this is consistent within
Bytes. Because clamping removes OutOfRange, the error is AllocError, not
BytesError. Code whose error type is only BytesError keeps using
slice(start, stop) or widens its error type.
A bytes value refers to read-only data with its length in a header, so a
sub-range cannot be another bytes without a new representation, and new
read-only data cannot be created at run time. A bytes slice therefore
returns an owned Bytes: b"GET /"[0:3]? is the buffer GET. This is the
same copy as std.bytes.from_bytes(value)? followed by a slice, done in one
allocation. Using the result does not need import std.bytes; naming its type
does, as f-strings produce a std.fmt.FormatError without an import.
Tuples
The result type depends on the bounds, so tuple slice bounds and step must be
integer literals, optionally negated (t[-2:]), or omitted. Any other bound is
rejected: tuple slice bounds must be integer literals; the result type depends on them. The positions are computed at compile time with the arity as the
length:
Source, t: (i64, str, list[i64]) | Type |
|---|---|
t[1:] | Result[(str, list[i64]), AllocError] |
t[:1] | Result[(i64,), AllocError] |
t[::-1] | Result[(list[i64], str, i64), AllocError] |
t[3:] | Result[(), AllocError] |
An empty selection has the unit type (), which is also how Python spells the
empty tuple. Components are duplicated with copy() semantics, so the
list[i64] component is deep-copied and the source tuple is unchanged.
Tuple indexing originally accepted only nonnegative literals; issue #583 made
it accept negative literals too, so indices and slice bounds follow the same
rule.
Columnar lists
items[a:b:c] on a clist[T] returns a new clist[T] holding the selected
rows. Fields are scalars, so no element copy can fail; only the new column
storage can. For a step other than one, each column is gathered with a stride.
Whole-clist copy is unsupported today, so items[:]? becomes the way to
duplicate one.
Evaluation, borrowing and &
A slice evaluates the receiver place, then start, stop and step, left to right,
each once. It then normalizes the bounds and copies under a
shared loan of the receiver. Bounds may read the receiver, as in
xs[len(xs) - 2:]?, because the bounds are evaluated before the copy and only
shared loans are involved.
& and &mut directly applied to a slice are rejected, with or without ?
and with or without parentheses:
&xs[1:3] # error: borrowed slice views are not implemented (B11/B39);
&xs[1:3]? # bind the copy first: `part = xs[1:3]?`, then `&part`
&mut xs[1:] # error: likewise
&xs[1:3] is the planned view. Accepting &xs[1:3]? now as a borrow of a
temporary copy would leave two spellings one character apart, one copying and
one not, once views land. Rejecting both now keeps the view syntax free and the
cost of every accepted form plain. These rejections are phase-guard tests that
name issue #581, B11 and B39.
Costs and allocation failure
For k selected elements, a slice does O(1) bound normalization and O(k)
copying, plus the cost of each element’s copy:
| Receiver | Work | Allocations |
|---|---|---|
list[T], copyable T | O(k) row copies or retains | List header, plus rows when k > 0; as slice() |
list[T], affine T | O(k) plus each element’s deep copy | As above, plus each copy’s allocations |
str, ASCII | O(k) | One, or none when the result fits inline |
str, other text | One scan over scalars to the farthest selected position; O(length) at worst, no auxiliary storage | One, or none when the result fits inline |
Bytes, bytes | O(k) | As Bytes.slice() |
| Tuples | Each selected component’s copy | Only those of the components’ copies |
clist[T] | O(k) per column | As clist[T].with_capacity(k) |
The count k is known before copying, so the result is reserved once with its
exact size; no slice grows its result. A str slice of non-ASCII text needs the
selected scalars’ byte length before allocating, so it scans the selected
region twice.
If an allocation fails, the slice returns Err(AllocError.OutOfMemory). Every
value already copied into the partial result is dropped, last copied first,
and the partial result is freed. The source is never changed, so retrying or
recovering needs no repair. k never exceeds the source length, so a slice
never reports CapacityOverflow.
Copying a slice only to iterate it wastes the copy. Lessons should teach
for i in range(1, len(xs)): or a pipeline
over the source for that, until a view can be iterated with for x in &xs[1:].
Diagnostics
| Source | Message |
|---|---|
ys: list[i64] = xs[1:3] | slice returns Result[list[i64], AllocError]; use ? to propagate allocation failure, matching the collection-literal message |
buffers[1:]?, buffers: list[Bytes] | slicing list[Bytes] copies each element, and Bytes cannot be copied; hint: transfer from an owned temporary with .slice(start, stop), or build a new list from explicit copies |
xs[::0] | slice step cannot be zero |
xs[::k], k: i64 | slice step must be NonZero[i64]; hint: the conversion from issue #584 |
xs[a:b] = ys, xs[a:b] += ys | slice assignment is not supported; hint: assign elements by index, or build a new list |
&xs[1:3], &mut xs[1:3] | borrowed slice views are not implemented yet; hint: bind the copy first |
t[i:], t: tuple | tuple slice bounds must be integer literals; the result type depends on them |
m[1:2, 3] | a slice cannot have several dimensions |
d[1:2], d: dict | dict does not support slicing |
xs[1.5:] | The existing index-type diagnostic, naming i64 |
The current “this resource cannot be copied” message does not name the type. The slice diagnostic names the element type and the receiver type, since the element is not written in the slice expression.
Out of scope
| Feature | What it would need |
|---|---|
Slice assignment a[1:3] = values | Python lets a step-1 assignment change the length, which can allocate. An assignment statement has no value for ?, so this needs a fallible method such as xs.splice(start, stop, values)? or a statement-level failure rule. A stepped assignment requires equal lengths. The right side’s elements would move in. |
Deletion del a[1:3] | Plenty has no del. A remove_range(start, stop) method needs no allocation and can be added independently. |
slice(start, stop, step) values | A prelude slice type with Option[i64] fields, and subscripts that dispatch on the operand’s type; today they accept integers and dictionary keys. A range has required, typed bounds and is a sequence of positions; a slice is unresolved until applied to a length. s.indices(length) would return the normalized range. |
Borrowed views &xs[a:b], &mut xs[a:b] (issue #581) | A view type (address, length, stride) under the existing loan rules. Local-only views can follow the dictionary key-view precedent (issue #396); storing and returning them needs B11 stored references. &mut xs[a:b] would allow in-place sorts and fills of a sub-range. String views need byte offsets; Bytes sub-range views are B39’s remaining item. |
| Ranges, deques, generators, user classes | Slicing a range could return a range without allocation; user classes need a hook design. Each needs its own use case. |
Prior art
- Python copies for
list,str,bytesandtupleslices, clamps, and allows negative steps.memoryviewand NumPy slices are views. This proposal takes Python’s results and adds the visible?. - Rust spells views
&v[a..b]: no copy, no negative positions, no step in the syntax (.step_byon iterators), and out-of-range bounds panic.v.get(a..b)returns anOption, and.to_vec()copies. Plenty’s planned&xs[a:b]follows Rust’s choice that a borrow is spelled with&. - Go slices
s[a:b]share the backing array, soappendthrough one slice can overwrite elements visible through another. Out-of-range bounds panic, and there are no negative positions or steps. Its three-index forms[a:b:c]sets capacity, not a step. Plenty rejects shared mutable aliasing, so it cannot copy this design. - Mojo keeps Python’s
a[start:stop:step]syntax through__getitem__with aSliceargument. ItsListreturns a new list, whileSpanandStringSliceborrow. - Nim writes inclusive
a[1..3]and counts from the end witha[^2..^1]. It copies into a newseqand raises on out-of-range bounds;toOpenArrayborrows without copying.
Slice syntax implementation plan
Status: Implemented in one change rather than the sequence below. This page planned the
slice syntax proposal (issue #579). Its code fences show
proposed syntax and are not compiled. File names refer to local main at
f681a0d5.
Reviewable slices
Each slice is one local PR. Each updates the reference pages, cheat sheet rows and tutorial lessons for what it ships, following the development workflow.
- Parsing, diagnostics and copyable lists. Add
Expression::Slice { base, start, stop, step }insrc/frontend.rs; the postfix loop that buildsExpression::Indexrecognizes a top-level:. Reject slice assignment targets (Statement::SetIndexand augmented assignment), commas,&and&mutapplied to a slice, and unsupported receivers, with the messages in the proposal. Generalize the runtime’sslice_bounds(plenty-runtime/src/aggregates.rs) to the stepped algorithm, and extendtry_list_sliceto copy strided rows of copyable elements, includingstrretains. The step uses the nonzero integer type, so S1 depends on issue #584 for written steps; slices without a step can land first if #584 is not ready. - Lists of affine elements. Reuse the runtime’s
try_copyfor each selected element, andshare()forShared[T]. Elements with nestedSharedfields work once issue #582 lands. Check copyability per specialization, with a diagnostic that names the element type. Owned temporaries may transfer selected elements instead of copying them. - Strings. Extend
try_sliceinplenty-runtime/src/strings.rsto steps: byte offsets for ASCII text, one bounded scan for other text, an exact-size result. Retain the source for a whole-string step-1 slice. Bytesandbytes. LowerBytesslices to the native storage copy used byBytes.slice, without its range rule. Makebytesslices produce aBytes, loadingstd.byteswhen the program does not import it.- Tuples and columnar lists. Compute tuple positions in the checker from
literal bounds and build the result tuple type. Add a strided column copy
for
clist[T].
The first slice alone covers most uses (xs[1:3]? on lists of numbers or
strings). Slices 2 to 5 depend on the first but not on each other.
Tests
Extend tests/test_slicing.rs, which holds the existing slice method tests:
- Normalization model. A Rust model of
PySlice_AdjustIndicesinside the test, compared against compiled programs for list lengths 0 to 6, every bound from -8 to 8 and omitted, steps -3 to 3 except zero, and the extremesi64.MINandi64.MAXfor bounds and steps. The table of examples in the proposal is a separaterstestcase list, with results checked against CPython when it was written. - Runtime unit tests in
plenty-runtimefor the generalizedslice_bounds, including a step ofi64.MINand lengths 0 and 1. - Element duplication. Nested lists, dicts and tuples of lists are
independent after the slice; mutating the copy leaves the source unchanged.
Shared[Mutex[i64]]elements are shared: a change through one list’s handle is visible through the other. - Allocation failure under the
runtime-checksfeature, followingevery_list_slice_allocation_failure_preserves_borrowed_sources: for every allocation budget of a nested-list slice, the result isErr(AllocError.OutOfMemory)or the full slice, and the source prints unchanged. Under--all-featuresthe allocation-checks runtime confirms at exit that partial results were freed. - Evaluation order. Bounds with printing side effects run once, left to right, after the receiver; bounds may read the receiver.
- Parsing.
TABLE[n:]on a module constant works while issue #562 is open; conditional and call expressions as bounds;xs[1:3]?[0]andlen(xs[1:3]?). - Diagnostics. One
plenty-errorstyle case per row of the proposal’s diagnostics table. The&xs[a:b]rejection is a phase-guard test with a comment naming issue #581, B11 and B39, to be replaced when views land. - Step type. Literal steps convert at compile time; literal and constant
0are compile errors; a plaini64step is a compile error whose hint shows the issue #584 conversion; a converted run-time step slices correctly. - Strings. ASCII and mixed text, negative steps, combining marks, results of seven and eight UTF-8 bytes (inline versus heap), and the whole-string case.
- Tutorial and reference fences, run by
test_tutorial::and the design example tests.
Documentation and tutorial
A new reference page, design/language/04-expressions/slices.md, holds the
syntax, the normalization algorithm, the examples table, the step type and
the reserved & form. Receiver pages link to it and state their result types
and costs:
| Page | Change |
|---|---|
design/facilities/01-builtins/lists.md#slices | Lead with xs[a:b:c]?; keep slice() as the transfer from an owned temporary |
design/facilities/01-builtins/text-building.md | Add text[a:b:c]? to the table; remove “Slice syntax and steps are unsupported” |
design/stdlib/bytes.md | Replace “There is no slice syntax” with the clamping syntax and its AllocError; keep the method range rule |
design/language/03-types/bytes.md | Replace “no slicing” with the Bytes result |
design/language/03-types/tuples.md | Literal-bound slices and their types |
design/facilities/01-builtins/clists.md | Row slices; items[:]? as the duplicate |
design/examples/cheatsheet.md | Use name[1:3]? and items[1:3]? in the examples; replace “There is no slice syntax”; change the Python row to xs[1:3] → xs[1:3]? (“allocates a copy”) |
design/examples/collections/list-slices.md, design/examples/text-files/text-23-slice-text-by-unicode-scalar-position.md | Show the syntax first, then the methods |
design/toolchain/02-architecture/support.md | List slice syntax |
Tutorial:
- New lesson “Take part of a list or text” in
tutorial/04-collections/, after Transfer or copy ownership, because a slice is a copy. It teachesxs[a:b]?, omitted bounds, negative positions,[::-1], clamping, and the cost of copying; it shows nested lists being deep-copied, and recommends arangeloop or pipeline over the source when the copy is only iterated. - Build text with recoverable allocation
mentions
text[a:b]?as another allocating text operation. - Byte buffers keeps
frame.slice(1, 3)?where the function’s error type isBytesError, and notes the syntax.
Review other lessons and stdlib/std/*.plenty for .slice( calls and manual
sub-range loops that the syntax states more clearly, following the
tutorial guidelines. The standard library has 45
.slice( calls; migrate only where the result is clearer, as a separate PR.
Status updates when the work lands: this proposal’s status blocks, the string design note “slice syntax remains a B12 candidate”, and the proposals index.
Backlog note
The backlog is edited only after merge. Proposed B39 wording, replacing “Slice syntax is deliberately omitted: it would hide a copy.”:
Slice syntax (
buffer[a:b:c]?, issue #579) copies with Python’s clamping and returnsResult[Bytes, AllocError];bytesslices produce aBytes. See[slices](../../slice-syntax/design/language/04-expressions/slices.md). Remaining: borrowed sub-range views&buffer[a:b](issue #581; needs B11 stored references).
B12 lists “borrowed sub-range views (B11/B39)”; add that their spelling is
&xs[a:b].
Development workflow
The backlog is the sole list of future work and priorities.
Issues, claims, and local PR reviews live in the shared issues.db; the repository’s
AGENTS.md and CLAUDE.md describe the worktree and review workflow.
Keep implementation, reference contracts, and executable lessons aligned in the same change. Reference pages describe implemented rules and current limits; proposals preserve rationale and alternatives. Update a backlog entry when its scope changes or completes.
Native tests should check observable output, diagnostics, evaluation order, mutation, ownership, borrowing, and cleanup. Allocating operations need failure and partial-state cleanup checks. Use runtime allocation checks, sanitizers, or Miri where they exercise the changed boundary.
Build a complete local installation
Install just, a stable Rust toolchain
with its rust-docs component, Python 3, and the native C compiler/linker and
binutils (readelf, nm). From the checkout, run:
just # same as just build
just compiler # compiler only
just engine dataframe # one engine and its redistribution notices
just engine linalg
just test # all-features workspace suite, with a timing report
just docs # all three published-site checks
just ci # every CI check this Linux host can run
just release-check # just ci plus the Docker release build and smoke tests
just --list
The default builds release plenty and plentyc, then both optional engines,
sequentially. The first engine builds are substantial; later builds reuse Cargo
artifacts. target/release/ contains both executables, both shared libraries,
and dataframe-notices/ and linalg-notices/ with checksummed metadata and
licenses for --dist. Missing Rust license files can be installed with
rustup component add rust-docs. Cargo may need network access for uncached
registry and pinned Git dependencies.
Add the checkout’s absolute target/release directory to PATH. For Fish:
fish_add_path /path/to/plenty/target/release
Rebuilding updates this installation; moving the checkout or cleaning its
artifacts removes it. The compiler recipe explicitly selects target/, even
if CARGO_TARGET_DIR is set. Engine caches stay under
target/dataframe-engine/ and target/linalg-engine/. Ordinary
cargo build --release --locked remains compiler-only. Existing Python engine
build commands remain available for custom output paths and job counts.
These are local host builds, not portable release archives. Local packaging
retains architecture, SONAME, export, dependency and RPATH checks, but records
the engine’s actual required glibc symbol version rather than enforcing 2.28.
It also records build_kind: local and whether the checkout is dirty (excluding
the shared issues register); this is provenance information, not an attestation.
The compiler and generated applications can require newer system libraries too.
The explicit --local-install packaging mode does not produce release archives
or enforce the portable GLIBC/GCC symbol baseline; the release workflow never
selects it. Portable releases keep the strict baseline audit.
Do not run builds against the same installation concurrently. Recipes stop on
the first failure; the complete installation is not a transaction. Notices are
staged before replacement, and --dist rejects a library whose checksum no
longer matches its notices after an interrupted update. Re-run the recipe to
repair that installation. Existing notices directories are generated output and
are replaced, not a location for application-owned files.
Check timing reports
just test runs the full all-features workspace suite, including doctests,
with compilation timed separately from test execution. It saves Markdown and
JSON reports under a unique target/check-timings/suite-*/ directory, and
retains Cargo’s compilation HTML under target/cargo-timings/. Output stays
on the terminal and the command returns failure if compilation or tests fail.
The execution phase includes Cargo’s short freshness check and doctest builds;
this is command-level attribution, not a compiler or per-test profiler.
python3 -B scripts/check_timings.py suite --profile ci
python3 -B scripts/check_timings.py report target/check-timings/suite-CURRENT/timings.json --baseline target/check-timings/suite-EARLIER/timings.json
just ci also saves each invocation’s step logs, steps.tsv, timings.json
and timings.md under target/local-ci/run-*/, including failed steps. It
continues to later checks after a failure. These timings use Bash’s integer
seconds; the suite wrapper uses a monotonic clock. Both add negligible work
at command boundaries and retain separate runs for comparison.
Direct cargo test remains available but does not save this report. Reports
record the revision and host; note profile, cache warmth, competing builds,
Cargo lock waits and environment overrides when comparing runs. JSON command
rows record the suite’s actual Cargo arguments. Measurements observe ordinary
work; they do not reserve the host or claim controlled benchmark results.
For CI job/step reports and historical comparisons, see CI timings.
Dependency versions
The compiler executable and runtime share a tracked workspace Cargo.lock.
Fresh checkouts and worktrees use its dependency versions; CI builds and tests
with --locked so a stale or missing lockfile fails instead of resolving a new
dependency graph. Use --locked for local validation too.
Update dependencies deliberately with cargo update, and review the resulting
lockfile changes with the code that needs them. Changes to Cranelift or regalloc2
require rerunning object-identity checks and affected compiler performance
comparisons. Record the dependency versions with benchmark results so results
from different dependency graphs are not compared as if only Plenty changed.
Build profiles and CI
The workspace uses Rust edition 2024. Development and release builds disable incremental compilation; workspace code keeps line tables for backtraces, while dependencies have no debug information. See CI and repository configuration for the build profiles, jobs, caches, timings, and source highlighting.
For the local CI-profile full suite, run:
cargo nextest run --locked --workspace --all-features --cargo-profile ci
Running tests
cargo test --locked --workspace --all-features runs the full suite: the
tests CI runs in shards under nextest, plus the doctests. When several workers
share a machine, add -j 3 -- --test-threads=3.
CI’s lint job also gates cargo fmt --all -- --check,
cargo clippy --locked --workspace --all-targets -- -D warnings, and
python3 -B scripts/check_links.py book/src on stable Rust. Run them before
committing.
The integration tests in tests/test_*.rs are modules of one test binary,
tests/integration.rs, so all of them share one pool of test threads and the
binary is linked once. Select one file’s tests by module, or one test by path:
cargo test --locked --test integration test_std_url::
cargo test --locked --test integration -- --exact test_std_url::rfc_3986_examples_split_into_components
Cargo builds separately for each set of --workspace and feature flags, so
add the full suite’s flags to reuse its build. List a new test file in
tests/integration.rs; every_test_file_is_listed fails until it is. Editing
any test file recompiles the whole binary.
Most tests compile a Plenty program and link it with cc. The workspace’s
.cargo/config.toml sets COMPILER_PATH for the processes Cargo starts, so
where mold is installed as /usr/libexec/mold/ld, cc links these programs
with mold, several times faster than GNU ld. Elsewhere cc uses its default
linker. CI installs mold for this. The setting does not change the compiler’s default cc driver or the
Rust linker; COMPILER_PATH= cargo test ... links with the system default.
Tests must not depend on one linker’s output, such as the exact symbol table.
The file also sets PLENTY_LINKER_THREADS=1, so each test’s link runs mold
on one thread
(Native linking): many links
run at once, and mold’s default thread per core made them compete for the
cores. PLENTY_LINKER_THREADS= cargo test ... restores mold’s default.
The same file points PLENTY_CACHE_DIR at target/plenty-cache, so every
test run of a worktree reuses the
compiled-function cache from
the runs before it and never writes to ~/.cache. A fresh cache must not
change any result: PLENTY_CACHE=off cargo test ... runs without it, and a
test that exercises the cache uses a directory of its own.
cargo-nextest is optional locally; CI uses it.
cargo nextest run --locked --workspace --all-features runs each test in its
own process and does not run doctests (cargo test --locked --workspace --all-features --doc does). Its -j sets test threads, not build jobs: when
sharing a machine, use --build-jobs 3 --test-threads 3. Because each test is
its own process, a static lock in a test file no longer keeps two tests
apart: tests that must not overlap belong to a test group in
.config/nextest.toml, as the timing tests of
tests/test_compile_scaling.rs do. --profile ci is CI’s profile: it runs
every test after a failure and writes target/nextest/ci/junit.xml.
Runnable documentation
tests/test_tutorial.rs reads tutorial and design/ pages in SUMMARY.md
order. A plenty fence has an output fence and runs both through
compile-and-run and an explicitly compiled binary. A plenty fragment fence is
an explanatory excerpt, such as a signature shape, and is not run. A plenty-error fence has an
error substring and must fail without executing effects. A plenty-file
companion module remains on the same page as its example. Every book page must
appear in SUMMARY.md. The examples run as 16 tests in parallel, each taking
every 16th example in reading order; a failure names the example’s page and
line.
book/theme/plenty-highlight.js registers the highlight.js grammars for these
fences: Plenty source for plenty, plenty fragment, plenty-error and plenty-file, compiler
diagnostics with highlighted `code` for error, and plain text for
output. book/theme/plenty-highlight.css colours the classes the bundled
themes leave unstyled. Its keyword, literal, type and built-in lists follow the
compiler; tests/test_book_highlighting.rs fails when a reserved word or
lower-case prelude name is missing from them, or when they name one the compiler
no longer defines.
Proposal fences are illustrative and are not executed by the tutorial suite.
Use text fences for proposed syntax; check implemented behavior with a complete
program or an executable lesson.
Before handing off changes to book content, navigation, assets, configuration, or docs build/check scripts, run the same checks as documentation CI:
python3 -B scripts/check_links.py book/src
python3 -B -m unittest discover -s scripts -p 'test_*.py'
python3 -B scripts/build_docs_site.py
Use the mdBook version pinned in book/releases.json; see Documentation site for installation. Record the complete-site result in the PR.
If a check is blocked, state
which one and why rather than reporting documentation validation as complete.
The assembled-site check includes generated chapter and print links and version
navigation; a plain mdbook build is only a preview, not a substitute.
For moved or changed executable lessons, also run
cargo test --locked --test integration test_tutorial::. Prose-only changes
need the listed docs checks, not a compiler rebuild.
The link checker uses Python’s standard library. It checks local file targets in
inline links, images, and reference definitions, ignoring code and comments.
It does not check remote URLs, HTML links, or heading anchors. To check only a
changed area, pass a directory or Markdown file relative to book/src, for
example python3 -B scripts/check_links.py book/src proposals.
Documentation site
Install the pinned tools and preview the book locally:
rustup toolchain install 1.99.0 --profile minimal
cargo +1.99.0 install mdbook --version 0.5.4 --locked
mdbook serve book --open
python3 -B scripts/build_book.py builds the checkout into book/book;
python3 -B scripts/build_docs_site.py builds and checks the complete versioned
site in book/site. Do not use mdbook test for Plenty
fences; the tutorial integration tests execute them.
See documentation deployment for site previews, publication, permissions and recovery, and documentation versions for release admission and retention.
Reverse-proxy tests
Run the nginx and Caddy integration tests when needed; they require Docker and curl:
PLENTY_DOCKER_TESTS=1 cargo test --locked --test integration test_std_http_server_proxy::
Without the variable, these tests report a skip and pass. With it, a missing Docker daemon or curl fails them. See reverse-proxy tests for container cleanup and CI image maintenance.
Quick reproductions
The compiler expects a complete program with a main entry point. Put probe
statements inside it; for allocation or printing, propagate failures:
def main() -> Result[(), AllocError | IoError]:
xs = [1, 2, 3]?
print(xs)?
Ok(())
An infallible body can use def main() -> ():. The fragment wrapper in
tests/support/mod.rs is test fixture construction. For repeated probes, build
once with cargo build --release, then use target/release/plenty FILE from
that worktree; rebuild when compiler or runtime sources change.
Measure compilation cost when changing checking or
lowering; the architecture guide
describes reproducible performance evidence. The opt-in
compiler profiler subdivides lowering, ownership checks and
native compilation. Use compile_bench --workload check|scale|aot to isolate a
workload, and --no-warmup when the report should contain only the requested
repetitions. Normal tests never start a profiling session automatically.
CI and repository configuration
For contributor commands, see the development workflow.
Rust build profiles
Both workspace crates (plenty and plenty-runtime) use Rust edition 2024.
The embedded runtime archives built directly by build.rs use the same edition;
vendored third-party crates retain their upstream editions.
Development and release builds disable incremental compilation. Workspace
sources (plenty and plenty-runtime) retain line tables for backtraces with
file names and line numbers; third-party dependencies emit no debug information.
Tests inherit the development settings, and benchmarks inherit the release
settings. These profiles are configured in the workspace’s .cargo/config.toml
so they take precedence over user-wide Cargo profile settings.
CI build caches and timings
The Cargo workflow (.github/workflows/ci.yml) runs on stable Rust for every
push to main and every PR. It runs the all-feature suite, doctests and a
default-feature build on both ubuntu-24.04 and ubuntu-24.04-arm; function
and build cache keys include runner architecture. Optional engines are built
and exercised independently. Its jobs run in parallel:
test: four shards of the full all-features suite under cargo-nextest, split with--partition hash:K/4so a test stays in the same shard from run to run. The Docker proxy tests are excluded; they run inproxy_tests. Each shard uploads its JUnit report (junit-shard-K) with per-test durations.doctests: builds default features once, then runs the doctests, which nextest does not run.lint: format, clippy and book links.proxy_tests: the reverse-proxy tests.
Optional-engine jobs use runtime-checks for both module tests and tutorial
examples, reusing the compiler, embedded runtime and integration test binary.
Cargo also builds the CLI for the integration target, so the sales-report
application uses that binary without a separate compiler build.
just ci dataframe linalg follows the same rule locally. The
doctests job still verifies a default-feature compiler build separately.
The shards build with the ci Cargo profile, defined in .cargo/config.toml:
the dev profile with the plenty crate at opt-level = 1 and dependencies
at opt-level = 2, keeping debug assertions and overflow checks. Nearly every
test compiles a Plenty program, so suite time is mostly the compiler’s own
run time, and an optimized compiler shortens it by about a third. Building
takes longer, so the local default stays dev, which rebuilds faster after an
edit. For a local full-suite run, cargo nextest run --locked --workspace --all-features --cargo-profile ci uses CI’s build; its artifacts live in
target/ci/. The doctests and proxy_tests jobs run few tests per build
and stay on dev. The nightly workflow also stays on dev, the profile
cargo test uses locally, so the full suite still runs weekly with an
unoptimized compiler.
Each shard compiles the tests itself. A shared build job with
cargo nextest archive would compile once, but the shards would then wait for
that job’s setup, compilation and upload, so it saves runner time, not wall
time.
Jobs that run tests install mold from Ubuntu’s package and fail unless cc
links with /usr/libexec/mold/ld (scripts/ci_check_mold.sh); see
Running tests for why. A weekly workflow,
.github/workflows/nightly.yml, runs the full all-features suite with
cargo test on nightly Rust, also on demand through workflow_dispatch.
Dependency cache
Third-party Rust dependencies are cached separately for each job. The key
includes the compiler, lockfile, manifests, Cargo configuration and compiler
environment. Only successful main runs save caches, and only the first shard
saves the shards’ shared dependency cache; PRs restore the base branch’s caches
without filling storage with per-PR copies. Workspace artifacts and incremental
state are not retained. The workspace profiles also keep compilation outputs small. The
nightly workflow keeps no cache: each nightly update invalidates compiled
dependencies.
Optional-engine jobs keep compiler dependencies in target/ and engine
dependencies in plenty-dataframe/target/ or plenty-linalg/target/. These
cache paths must not overlap: the cache action cleans each target with its
workspace’s dependency list, so cleaning a parent target would discard nested
engine-only dependencies. The engine build receives the matching explicit
--target-dir; its installed library still goes to target/engine/. The
baseline-isolated-v2 engine cache key starts a new cache population; the
first run is cold, and hosted timings must confirm reuse on later runs.
Embedded-runtime Rust dependencies
The test shards also retain the 27 vendored Rust rlibs that build.rs
compiles for both embedded runtimes. They are independent of runtime source,
Cargo profile and allocation-check mode. A separate cache outside target/
survives workspace-artifact cleanup. Only a successful main run’s first
shard saves it; both architectures have separate entries.
The build script validates a SHA-256 key over the vendored source tree,
build logic, lockfile, target, Rust compiler version and target standard-library
contents. It checks each restored rlib’s content before reuse and publishes
complete entries atomically. Native C archives and the two runtime archives
still rebuild: native headers/SDKs and runtime source need different inputs.
Custom PLENTY_RUNTIME_RUSTFLAGS bypass reuse because flags may name extra
files that the key cannot infer. Unreadable or symlinked key inputs also bypass
reuse; they do not prevent an otherwise valid compiler build.
Locally, entries live under the Cargo profile’s
plenty-runtime-vendor-cache/ directory and survive runtime-source and
feature-only rebuilds. PLENTY_RUNTIME_VENDOR_CACHE=/absolute/path shares
entries across profiles or isolated targets; PLENTY_RUNTIME_VENDOR_CACHE=off
disables reuse. Delete the directory to reclaim entries after a dependency or
toolchain change. Cache status and dependency-stage duration appear in verbose
Cargo build output; saved Cargo timings still report the whole build script.
A local 2026-11-06 probe reduced this dependency stage from 18.95 seconds to 0.46 seconds, removing all 27 vendored rustc invocations on a hit. Whole build-script samples were 82.16 seconds before caching, 82.23 seconds with a cold cache and 108.21 seconds with a warm cache: native/runtime work varied on the shared machine. Those samples establish a dependency-stage saving, not an end-to-end speedup. Hosted restore/save overhead and CI savings remain to be measured. The native and runtime archive stages remain outside this cache.
Compiled-function cache
Each shard also keeps the function entries of the
compiled-function cache,
target/plenty-cache/fn, in its own GitHub Actions cache. The dependency
cache deletes that directory, so it has its own restore and save steps. Their
keys depend on the compiler and Cranelift versions and the compiled IR, so
entries stay valid across commits. Program entries, target/plenty-cache/program,
key on the compiler binary’s modification time, never hit in a new CI build,
and are not kept.
The cache needs no maintenance. A cache hit only reads its entry, so
scripts/ci_fn_cache.sh mark gives every restored entry an old access time,
and the next read updates it. After a successful main run,
scripts/ci_fn_cache.sh prune deletes the entries the run did not read or
write. The saved cache is therefore the shard’s working set for that commit:
it grows only with the suite, and entries for replaced code generation leave
it at the next save. The cache key ends with a digest of the remaining entry
names, so an unchanged working set is not saved again. A save deletes the
cache it replaces, so storage holds about one cache per shard. If reads do not
update access times on the runner, mark warns and nothing is pruned or
saved. GitHub removes a cache unused for seven days; the next run is then
cold, as is the first run after the shard count changes.
Timings
The Cargo workflow ends with a Check timing report job, also after failed
checks. Its job summary shows jobs ranked by duration and a table of step names
aggregated across shards and architectures, with counts, seconds, status and
share of measured step time. The check-timings artifact retains the Markdown,
normalized JSON and raw Actions timestamps for 30 days. The reporting job itself
is excluded. Cancellation can prevent a report from being produced.
The same small helper can report on an existing run of any workflow. It reads
job/step timestamps through an authenticated gh CLI, including all pages and
only the latest run attempt; it does not download logs or rerun checks:
python3 -B scripts/check_timings.py ci --repo OWNER/REPO --run RUN_ID
python3 -B scripts/check_timings.py report target/check-timings/ci-CURRENT/timings.json --baseline target/check-timings/ci-EARLIER/timings.json
Use the output directories printed by the commands. Comparisons show the
absolute delta for every step name; a changed number of steps is marked rather
than treated as a slowdown. Job and step status remains visible when checks
fail. A report collected during a run is partial; unfinished jobs have no
completed duration. Raw timestamps remain in jobs.json and run metadata in
run.json for further inspection.
Elapsed span (first job start to last job finish) and summed runner-step time are separate: jobs run concurrently, and runner time cannot be summed into workflow latency. Initial queue time is outside the span. Step timestamps have second resolution and do not attribute scheduling gaps between steps. Compare the same workflow, architectures, shard counts and cache conditions; cache restore/save and setup/post-action costs are included as their own steps. These reports locate growing process steps; existing JUnit and Cargo artifacts provide more detail when needed.
Test compilation is a separate step from execution. The cargo-timings-*
artifacts retain Cargo’s HTML compilation reports for seven days, including
reports available after a failed run: cargo-timings-tests from the first
shard, cargo-timings-build from the default-feature build,
cargo-timings-proxy and cargo-timings-nightly.
Documentation caches only the pinned mdBook executable, keyed by its version,
Rust version, runner OS, architecture and Ubuntu image. Every run checks its
version before building. Cache hits skip both Rust setup and cargo install.
Superseded PR runs are cancelled; main documentation deployments are not
interrupted. Release packaging keeps its isolated, pinned build environment.
The first run for a compiler or cache key remains cold. Compare the restore, compile, test and cache-save step durations on hosted runs before attributing a speedup; cached dependencies do not eliminate compiling Plenty itself.
GitHub source highlighting
Plenty is its own language. Until GitHub supports it through Linguist, the root
.gitattributes selects Python highlighting for every .plenty file, including
nested modules. This is approximate coloring: Plenty-specific syntax can be
miscolored, and a file viewer may label the file as Python. The
linguist-detectable=false attribute excludes these files from repository
language statistics so they do not count as Python; genuine Python files keep
their normal classification. The files remain visible in diffs.
This override applies only to repositories containing the rule. It does not
provide editor integration or guarantee GitHub code navigation or search
classification. Keep Markdown’s plenty fences and the book’s custom highlighter;
they are independent of GitHub’s source viewer. Once native Plenty support
reaches GitHub, remove both temporary attributes and recheck source highlighting
and language statistics.
Reverse-proxy tests
tests/test_std_http_server_proxy.rs runs std.http.server behind nginx and
Caddy (see reverse proxies).
They need Docker and curl, so they run only when asked:
PLENTY_DOCKER_TESTS=1 cargo test --locked --test integration test_std_http_server_proxy::
Without the variable each test prints that it is skipped and passes, so the
ordinary suite needs no Docker and does not slow down. With it, a missing
Docker daemon or curl fails the tests. The first run pulls the two pinned images
(about 70 MB each). Each test starts its own containers with --network host
on free loopback ports, names them plenty-proxy-test-<pid>-<n> with the label
plenty-test=proxy, and removes them when it ends, also on failure; nothing
else is stopped or pruned. A test process killed outright leaves its
containers running; list them with
docker ps --filter label=plenty-test=proxy.
CI runs them in the proxy_tests job of .github/workflows/ci.yml on
GitHub’s ubuntu-latest, which has Docker and curl. The job pulls
nginx:1.30.6-alpine3.24 and caddy:2.11.7-alpine in an earlier step, so
pulls are not timed with the tests; when the tests change an image tag,
change the job’s pull step to match.
Documentation site deployment
The public book defaults to development until the
first published release is admitted in book/releases.json. After that, the
root serves the selected stable release. Development documentation
always follows main; admitted releases live under /plenty/VERSION/.
Every page identifies its version and offers a version menu. Matching chapters
are preserved when switching versions; missing chapters fall back to that
version’s homepage. The links work without JavaScript; JavaScript also preserves
the current query and heading fragment when the chapter exists.
.github/workflows/docs.yml builds every push to main, PR targeting main
(including forks), and manual dispatch. Only a main push or main dispatch can
deploy. It does not publish packages or push branches. Release admission and
retention are described in versioned documentation.
See the development workflow for pinned tools and local preview commands.
For only the current checkout, python3 -B scripts/build_book.py writes
book/book. The complete site helper writes book/site. Both show Development
and its source commit, explicitly marking local changes. Each version’s
docs-version.json records the full revision and Cargo package version;
the package version alone does not identify a development build. Labels are
static HTML. Raw mdbook serve previews content without deployment metadata.
To preview the complete site, mount book/site at /plenty/ on a static HTTP
server; configure unknown paths to serve its root 404.html as GitHub does.
The build needs no Plenty compiler. Do not use mdbook test for Plenty fences;
the tutorial integration tests execute those examples. The output checker
verifies rendered local links stay inside the artifact and reach existing files,
and checks search, print, 404 and custom highlighting assets. Source checking
alone cannot catch a malformed SUMMARY.md that leaves a chapter unrendered.
Neither checker validates external URLs or heading anchors.
Book links must target published pages or assets. A relative link to a repository
file outside book/src is rejected by the source checker even when it exists,
because mdBook does not publish that file. Refer to such files with a repository-relative
code path (for example, experiments/rust-crates/README.md), or include the needed
content in a book page. Run the complete site helper after adding links; a
successful mdbook build alone does not validate generated links, including
those in print.html.
The Pages artifact is only book/site, which remains ignored. It contains all
admitted versions together, so development updates preserve older books. Each
build receives its own site-url prefix for nested 404 assets. Check navigation,
anchors, search, print, version switching and Plenty highlighting under each
prefix; repeat those checks on the public site after deployment.
First publication and recovery
After review, merge and authorization to publish, an administrator must:
- Check account/repository visibility and organization policies allow Pages and the pinned actions. Choose Settings > Pages > Build and deployment > Source: GitHub Actions. The workflow does not enable Pages automatically.
- Set the
github-pagesenvironment’s deployment branch policy tomainonly, preserving any required reviewers. - Run Documentation from current
main, approve its environment if required, and inspect the deployed URL and source commit in the run summary.
Build jobs have only contents: read; PRs do not configure/upload Pages or use
the deployment environment. The deploy job has pages: write, id-token: write
and contents: read for a main-tip API check. It uses the built-in token, with
no PAT. Actions are pinned by SHA; the upload action also pins its nested action.
Deployment jobs share one concurrency group and do not cancel a running deployment. After acquiring that slot and any environment approval, a run checks the live main SHA and skips if superseded. Queue ordering is not guaranteed: rerun current main if its pending deployment was replaced by an older run that then skipped. A commit arriving after the check can briefly leave the previous revision live until the next successful run; serialization prevents an older deployment finishing after a newer deployment. API failures fail closed.
Failed checks or deployments leave the last successful site available. Rerun current main for transient failures; fix or revert bad content through review and deploy the resulting main commit. Do not roll back by dispatching an old branch or deleting the site. Tool/action upgrades require another complete book build and theme/navigation check. See the official Pages workflow guide and mdBook HTML configuration.
Linux AArch64 development
The source tree implements native aarch64-unknown-linux-gnu alongside
x86_64-unknown-linux-gnu, and releases publish its compiler archive, with both
engines, built and tested on native ARM runners as described in
releasing. This page describes the emulation harness used to
develop the port and the evidence it recorded. That evidence does not establish
minimum kernel versions, page sizes other than 4 KiB, or weak-memory behavior on
other ARM hardware.
Platform support describes the
implemented contract; future work remains in B18.
Backend and C ABI probe
Install QEMU user-mode tools and Python 3, with Cargo’s locked dependencies
available locally. scripts/aarch64_probe.py provisions a checksum-pinned Bootlin
GCC/glibc toolchain, generates ARM ELF with the pinned Cranelift backend, links a
C fixture and executes it on an emulated Cortex-A53:
ARM_TOOLS="$PWD/target/aarch64-toolchain"
python3 scripts/aarch64_probe.py --provision "$ARM_TOOLS"
cargo fetch --locked --manifest-path scripts/aarch64-probe/Cargo.toml
CARGO_TARGET_DIR="$PWD/target/aarch64-probe-build" \
python3 scripts/aarch64_probe.py --toolchain "$ARM_TOOLS" \
--out "$PWD/target/aarch64-probe-output"
Provisioning refuses to replace an existing directory. The archive is
aarch64--glibc--stable-2025.08-1.tar.xz, SHA-256
dfb47eee874eef9e8a7fc042eee4e0a183f444b6bcde6a82fef8f009918389c9.
The probe prints tool versions and ELF relocations. It exercises native C calls,
i64 values, u128 arguments/results, direct and indirect Tail calls
with spilled arguments, a large inline-probed frame, data addresses and a
recorded UDF trap. The expected trap runs in a child process.
Keep the tool/version output with qualification results; the toolchain’s libc
is a development input, not evidence for the release’s glibc 2.28 floor.
This Bootlin GCC also injects 4096-byte maximum/common ELF page sizes through
its linker specs. Its linked binaries qualify the 4 KiB test environment only.
Use native release tools and test the resulting ELF layout before claiming
support for larger kernel page sizes; do not carry those overrides into releases.
Build the ARM compiler on x86
Cross-building the Rust compiler is a development facility. It does not add a user-facing Plenty cross-compilation mode. Install the matching Rust target and supply both Rust and C build tools:
rustup target add aarch64-unknown-linux-gnu
export CC_aarch64_unknown_linux_gnu="$ARM_TOOLS/bin/aarch64-buildroot-linux-gnu-gcc"
export AR_aarch64_unknown_linux_gnu="$ARM_TOOLS/bin/aarch64-buildroot-linux-gnu-ar"
export CARGO_TARGET_AARCH64_UNKNOWN_LINUX_GNU_LINKER="$CC_aarch64_unknown_linux_gnu"
cargo build --locked --target aarch64-unknown-linux-gnu --bins
qemu-aarch64 -cpu cortex-a53 \
-L "$ARM_TOOLS/aarch64-buildroot-linux-gnu/sysroot" \
target/aarch64-unknown-linux-gnu/debug/plenty --print-target
Native build helpers resolve CC_TARGET, CC_target_with_underscores,
TARGET_CC, then CC (and the equivalent AR variables). Without an override,
cross builds choose the architecture-prefixed GNU tools, never host cc.
Both embedded runtime archives, native dependencies and test/fuzz helper objects
must match the selected Cargo target. Use a disk-backed CARGO_TARGET_DIR with
adequate space; the compiler and runtime are large enough to exhaust a small
tmpfs. Limit CARGO_BUILD_JOBS when sharing a machine with other builds.
A Cargo runner wraps the Rust test process only. Plenty integration tests also
spawn a compiler, linker and generated programs, so setting a QEMU runner alone
is insufficient. Use an explicit subprocess harness or run inside an ARM guest.
Keep task scratch files outside the sysroot. In this harness, a task-owned
sysroot/tmp directory interfered with QEMU path handling: three SQLite/KV
cases failed before removing it and passed afterward with unchanged binaries.
Preserve the original results and test artifacts before correcting harness inputs.
Full ARM Linux guest
A full-system guest exercises an ARM kernel’s signal contexts, filesystem flags
and stack behavior. The initial environment used QEMU 10.2.2, Cortex-A53 with TCG,
two virtual CPUs and 2 GiB RAM, and Debian 13 genericcloud ARM64, kernel
6.12.113 with 4 KiB pages, glibc 2.41 and native GCC 14.2. Its base image was
downloaded from
Debian cloud images
and verified against the official sibling SHA512SUMS:
debian-13-genericcloud-arm64.qcow2
71aa5d5dbcb3d9ecfe96bc3f6dd41b6805a0a41c852204a32747d230b64b95912297fb453b11a6031fd54d84a0a0a430e137d07e6986dc9ebc9d41483be003c4
The latest URL can change: preserve the verified base image and checksum,
or obtain the same image from a dated archive. A different checksum is a new
qualification environment, not permission to bypass verification. Create a
16 GiB qcow2 overlay with qemu-img create -f qcow2 -F qcow2 -b BASE work.qcow2 16G. Create meta-data containing instance-id: plenty-arm and
local-hostname: plenty-arm on separate lines. Create this user-data, replacing
the key placeholder with your SSH public key:
#cloud-config
users:
- name: plenty
groups: sudo
shell: /bin/bash
sudo: "ALL=(ALL) NOPASSWD:ALL"
ssh_authorized_keys:
- ssh-ed25519 REPLACE_WITH_YOUR_PUBLIC_KEY
package_update: true
packages: [gcc, libc6-dev, binutils, make, pkg-config, curl, git, python3]
Build the seed with genisoimage -output seed.iso -volid cidata -joliet -rock user-data meta-data. Keep private keys and guest credentials outside the repository.
With AAVMF firmware installed, boot from the directory containing those files:
qemu-system-aarch64 -machine virt -cpu cortex-a53 -accel tcg -smp 2 -m 2048 \
-bios /usr/share/AAVMF/AAVMF_CODE.fd \
-drive file=work.qcow2,if=virtio,format=qcow2 \
-drive file=seed.iso,if=virtio,format=raw,readonly=on \
-netdev user,id=net0,hostfwd=tcp:127.0.0.1:22264-:22 \
-device virtio-net-pci,netdev=net0 -display none \
-serial file:console.log -monitor none -daemonize
ssh -p 22264 plenty@127.0.0.1
Copy the exact source revision and matching ARM binaries/test executables into
the guest, preserving executable permissions. Tests that compile C fixtures need
the guest’s native cc, and tests that launch plenty need the matching compiler
at the path captured when the test executable was built. Record uname -a,
getconf PAGESIZE, ldd --version, cc --version, Rust/QEMU versions, compiler
revision and feature flags with the results. A native build inside the guest
avoids captured host paths; it needs Rust and sufficient disk/RAM as well.
Qualification boundaries
The isolated C/backend probe and all-feature runtime suites passed under QEMU. Compiler workflow qualification combines a broad all-feature user-mode run with exact tests in the full ARM guest. Guest execution covers checked traps and stack overflow, signal-host coexistence, actor ownership, native process and environment contracts, relocated/offline toolchains, persistent SQLite/KV and required fuzz tests. The guest uses native tools where user-mode subprocesses would otherwise find x86 tools or lose their ARM loader settings.
Raw user-mode failures are retained rather than reported as a clean run. Exact
guest reruns preserve the assertions, including missing-program errors and fatal
signal output. A tutorial allocation imbalance observed in user mode did not
reproduce in the complete unchanged guest shard; its cause remains unproven.
An earlier integration harness itself exited on SIGILL without a core, before
several qualification fixes. Focused reruns did not reproduce it, and local
issue #592 records the missing original binary and diagnostic evidence without
attributing a cause. Use the ARM fault capture procedure
to preserve a recurrence before rebuilding. Local PR #706 records revisions,
totals and reconciled results; its retained artifacts are now under the primary
worktree’s ignored target/portability-pr706/worker-qualification/ and
target/portability-pr706/aarch64-vm/evidence/ directories.
Seven tests are explicitly ignored: the large specialization-budget resource
case, the external SQLite-vec extension trial, the systemd/playpen memory-limit
case, and four runtime timing/scaling benchmarks. Additionally, 13 dataframe
real-engine/tutorial cases and eight Docker proxy cases return early without
their opt-ins. Their libtest ok entries do not establish execution coverage.
Dataframe all-target compilation passed; full dataframe execution remains unrun
locally. FFI tests ran without optional Valgrind instrumentation.
The required nine libFuzzer tests passed in the guest with native clang and its matching runtime/interceptor archives. The supported engine-build helper’s SONAME-bearing linalg package passed all six enabled integration tests and the optional tutorial test. That local package requires GLIBC 2.39; it does not qualify the release floor. The release configuration retains glibc 2.28 with GNU ld for ARM, avoiding the pinned ARM mold binary’s glibc 2.30 requirement.
The emulated Cortex-A53 advertises AES, PMULL, SHA1, SHA2 and CRC32, but no LSE.
Known-answer crypto and compression checks also passed with a temporary
getauxval interposer reporting only mandatory FP/ASIMD features. This exercises
runtime fallback dispatch; it does not emulate the physical absence of crypto
instructions or change the compiler’s generic ARMv8-A baseline.
User-mode QEMU cannot establish kernel signal coexistence or ARM’s real
weak-memory concurrency behavior by itself. Full-system TCG adds kernel coverage
but is still emulation. Native hardware stress, additional page sizes and
release-baseline clean-container execution must each be recorded before claiming
them as supported. Native ARM CI is configured for the compiler/runtime suite,
both optional engines and the Docker reverse-proxy tests. Run 41210234842
on 2026-10-31 passed all of those jobs at revision 4c07ea01, including all four
ARM integration shards, default-feature build/doctests and strict lint checks.
These native results supplement the emulation evidence above; they do not
establish minimum deployment versions, additional page sizes or release-package
compatibility.
For the complete developer gate, run the locked workspace suite with all features, default-feature tests, formatting, clippy and tutorial/reference examples, as described in development workflow. Run cold, warm and cache-off compilation, C/library/Tail tests, checked traps and stack overflow, actors, filesystem no-follow cases, crypto/compression known answers, packages, runtime extraction, LSP and relocated distributions. Record missing services, libFuzzer/sanitizer tools and ignored tests explicitly. Release packaging and native runner gates are described in releasing.
ARM fault diagnostics
Use this procedure when the ARM integration harness itself exits on a signal.
The Linux AArch64 guide describes the execution environment
and qualification limits. Local issue #592 records a historical SIGILL whose
exact executable and core were not retained; a successful later run cannot
identify its cause. Do not infer the faulting test from the last printed ok:
parallel tests can still be running when that line appears.
Preserve the run before rebuilding
Stop other writers to the task’s build and cache directories. Create a fresh, absolute evidence directory outside the sysroot and build output. Preserve:
- The exact integration ELF, matching
plenty/plentyc, runtime archives, optional engines, and split debug files (.dwo). Copy bytes, not hard links to artifacts a build can overwrite. Record SHA-256 hashes and ELF build IDs. - The source commit,
Cargo.lock, feature flags, build command, tracked changes (git diff --binary HEAD) and any untracked source inputs. A revision alone does not describe a dirty build. Keep the source checkout available: tests can embed its absolute paths and read fixtures from it. - The complete toolchain/sysroot and tool-wrapper scripts, including symlink targets, plus QEMU and debugger versions. Retain the shared libraries reported by the debugger. A hash without the corresponding file is not a preserved artifact.
- Cache contents before execution, and a separate working copy for the rerun. Keep the post-run working copy too. Do not clear or reuse the original cache.
- The exact command, working directory, CPU affinity, test threads, environment overrides, inherited core limits and raw stdout/stderr. Record opt-ins and skipped tests; a passing count alone does not establish coverage.
Before a rerun, verify the ELF’s --list output, captured source/compiler paths,
subprocess execution and linker setup. An explicit QEMU command wraps only the
initial process: it does not make nested ARM executables runnable. Preserve
existing binfmt or guest configuration as part of the run description. If that
configuration is gone, restoring it is a new experiment, not an exact replay.
Capture the guest fault with GDB
This method captures guest registers and memory without changing host-wide
kernel.core_pattern or relying on the shell’s core limit. It was checked with
QEMU 10.2.2 and an AArch64-capable GDB 18.1. The GDB stub changes timing, so retain
ordinary-run evidence as well. Use a short, task-owned Unix socket path; local
sandbox policies may require permission to create/connect to that socket.
Set these absolute paths for the preserved run (the values are examples):
ARM_EVIDENCE=/tmp/plenty-arm-fault-001
ARM_ELF="$ARM_EVIDENCE/integration"
ARM_SYSROOT="$ARM_EVIDENCE/toolchain/aarch64-buildroot-linux-gnu/sysroot"
From the original test working directory, launch the saved ELF with the original
environment, affinity and test arguments, adding QEMU’s -g option. For a
four-thread Cortex-A53 run the command has this form:
qemu-aarch64 -cpu cortex-a53 -L "$ARM_SYSROOT" \
-g "$ARM_EVIDENCE/gdb.sock" "$ARM_ELF" \
--test-threads=4 --nocapture > "$ARM_EVIDENCE/run.log" 2>&1
It waits for a debugger. In another terminal, open the same ELF with an
AArch64-capable gdb (some distributions call it gdb-multiarch):
gdb -q -iex 'set debuginfod enabled off' "$ARM_ELF"
Enter these GDB commands, substituting the absolute evidence paths. GDB command
files do not expand shell variables such as $ARM_EVIDENCE.
set pagination off
set sysroot /tmp/plenty-arm-fault-001/toolchain/aarch64-buildroot-linux-gnu/sysroot
set logging file /tmp/plenty-arm-fault-001/gdb.txt
set logging overwrite off
set logging enabled on
target remote /tmp/plenty-arm-fault-001/gdb.sock
handle SIGILL stop print pass
continue
When the debugger reports SIGILL, capture before continuing:
info registers
x/8i $pc
x/32bx $pc
thread apply all bt full
info sharedlibrary
info proc mappings
generate-core-file /tmp/plenty-arm-fault-001/guest.core
Some QEMU/GDB combinations do not expose process mappings or core generation. Retain their errors along with the register, instruction and backtrace output; do not claim a core was captured unless it exists and can be reopened. Even a readable core can lack mappings or debugging information. For a generated core, check it independently against the saved ELF:
gdb -q -iex 'set debuginfod enabled off' "$ARM_ELF" "$ARM_EVIDENCE/guest.core"
Compare info registers pc sp and x/i $pc with the live capture. Preserve the
loaded objects and sysroot alongside the core. After capture, continue
delivers the stopped signal normally; Plenty uses intentional illegal
instructions for checked traps, so a SIGILL stop alone does not prove a defect.
Record whether it is handled or terminates the process. The stub observes the
attached process; a subprocess’s failure needs its own capture.
Interpret and minimize
Identify the instruction and containing ELF/object before attributing a cause. Compare with the compiler’s registered trap sites and the runtime signal handler. Keep the first captured failure, then reduce test filters and thread counts in separate runs with copied inputs. Compare cache-off, fresh-cache and preserved cache runs explicitly. Debugger timing or a serial pass does not disprove a race.
Reconcile the failing revision with intervening compiler/runtime fixes. Compare the minimized case in user-mode QEMU, the ARM Linux guest and native ARM when available, preserving the same assertions. Record non-reproduction and missing environments honestly; diagnostic improvements do not resolve an unexplained historical crash.
Documentation versions
The documentation site is assembled from an explicit
manifest, book/releases.json, and the current checkout. There are no published
versions in the initial manifest: a Git tag alone does not establish a release.
| Path | Documentation |
|---|---|
/plenty/ | Selected stable release, or development while stable is null |
/plenty/dev/ | Current main |
/plenty/VERSION/ | Admitted release from its pinned source commit |
Development pages show Development, their source commit, and any local
changes. Released pages show Plenty VERSION. Each book includes a version
menu and docs-version.json; versions.json at the site root lists the entire
catalog. Selecting another version retains the chapter if it exists there,
otherwise opens the homepage. A heading may not exist in an older chapter;
the browser still opens that chapter even when the fragment has no target.
Redirect pages declared in [output.html.redirect] in book/book.toml are
excluded from version-banner decoration; their destination pages carry the
version label. Content pages must still have the expected mdBook layout.
Admit a published release
After a stable GitHub Release is published successfully:
- Verify the published tag and record
git rev-parse 'vX.Y.Z^{commit}'. Do not admit drafts, prereleases, or a tag whose release job failed. - Add a manifest entry with
version,tag, and the fullcommithash. For example, the structure is{ "version": "0.2.0", "tag": "v0.2.0", "commit": "FULL_40_CHARACTER_COMMIT_HASH" }. Setstableto the version that should be the default; never infer it from main’s Cargo version. - Keep all previously admitted entries. Archive URLs are permanent; do not rewrite tag/commit pins. The helper rejects moved tags, version mismatches, duplicates and unsafe version names. Removing an entry would remove its deployed archive, so retention changes require an explicit review decision.
- Run
python3 -B scripts/build_docs_site.py. It uses the manifest’s exact mdBook version, currently 0.5.4. When changing that pin, also change the workflow installation pin and validate every archive with the new renderer. - Check version switching, search, custom highlighting, print, nested 404
handling, and existing root links. Submit the manifest change through the
normal local PR workflow. Its eventual push to
mainrebuilds the site; publishing a release by itself does not change the documentation default.
Historical books use their own prose, configuration, highlighting and included files from the pinned tree. The renderer and navigation layer come from the current site builder. Thus source revisions stay fixed, while presentation can receive fixes. No old compiler or repository script runs during extraction; mdBook may execute preprocessors declared in a historical book configuration, so review such additions when admitting releases.
The v0.1.1 tag predates a table-of-contents indentation fix. If that version is
eventually published and admitted, its entry needs summary_indent_repairs
with tutorial/11-concurrency/stop-on-signal.md and
tutorial/11-concurrency/bounded-io.md. This explicitly recorded, rendering-only
repair restores omitted chapters; it does not replace historical prose. The
build fails if the repair no longer matches. Metadata records applied repairs.
Deployment and existing links
The helper builds all books in temporary directories and checks their HTML
links and assets before replacing book/site. Includes can refer outside
book/, so releases are extracted from their complete tracked tree. Only
generated books enter the artifact; no source trees or issues database are
uploaded. Logged mdBook errors are fatal even if the renderer exits successfully.
The first stable admission deliberately changes the root’s content from
development to stable. Existing chapter paths found in stable continue to work.
Development-only paths get small forwarding pages to /plenty/dev/, preserving
queries and fragments with JavaScript and offering an ordinary link without it.
The builder creates these for development pages present at build time; it is
not a historical redirect registry for subsequently deleted or renamed chapters.
GitHub supplies the site’s root 404 page for missing URLs in any version. JavaScript forwards known version prefixes to their corresponding 404 page; without JavaScript the root 404 and version menu remain usable. No server-side rewrite rules, new domain settings, token, or extra Pages environment are needed.
Local admission experiments can use --manifest /tmp/releases.json and
--output /tmp/docs-preview without changing the reviewed manifest. The output
directory must not exist, or must contain a previous builder’s versions.json.
Follow the deployment recovery procedure
for failures. No archive is retained solely in short-lived Actions artifacts:
each deployment rebuilds the complete admitted catalog from Git.
Local semantic search trial
examples/semantic_search.plenty is a native Plenty CLI for a separate SQLite
search database. It uses sqlite-vec for exact cosine-distance search and
built-in FTS5 for keyword search, combining their top 30 candidates with
reciprocal rank fusion and returning up to ten chunks. All parsing, validation,
hashing, database operations, transactions and ranking are driven by Plenty.
Only model inference runs in Python, through scripts/search_embed.py and
FastEmbed. Plenty owns the CLI, chunking and document lifecycle.
For the repository corpus, use the setup and synchronization wrapper, which scans documentation, source docs and the local issue register and removes deleted documents. This page covers the underlying interface and individual document operations. No generated database or model is committed.
Setup
Use the compiler built from this checkout and the pinned sqlite-vec 0.1.9
shared library for x86-64 Linux. One way to obtain the extension is its Python
wheel. Install FastEmbed in the same environment:
uv venv /tmp/plenty-search-env
uv pip install --python /tmp/plenty-search-env/bin/python sqlite-vec==0.1.9 fastembed==0.9.0
export PLENTY_SQLITE_VEC="$(/tmp/plenty-search-env/bin/python -c 'import sqlite_vec; print(sqlite_vec.loadable_path())')"
cargo run --locked --bin plenty -- --compile examples/semantic_search.plenty -o /tmp/plenty-search
The executable loads that library through
Connection.load_extension.
Use only a trusted extension. SQLite’s SQL loading function remains disabled.
Storage-only commands do not need a Python process.
Search document text
Run these commands from the checkout root. Setup downloads the pinned
BAAI/bge-small-en-v1.5 quantized ONNX model (about 67 MB); inference thereafter
uses only the local cache and CPU. The helper pins the model artifact revision,
uses 384-dimensional vectors, and refuses input beyond its 512-token limit.
export SEARCH_PYTHON=/tmp/plenty-search-env/bin/python
export SEARCH_CACHE=/tmp/plenty-search-model-cache
"$SEARCH_PYTHON" scripts/search_embed.py --setup "$SEARCH_CACHE"
export SEARCH_MODEL="$("$SEARCH_PYTHON" scripts/search_embed.py --identity)"
/tmp/plenty-search "$PLENTY_SQLITE_VEC" /tmp/search.db "$SEARCH_MODEL" init 384
/tmp/plenty-search "$PLENTY_SQLITE_VEC" /tmp/search.db "$SEARCH_MODEL" index docs:sqlite SQLite book/src/design/stdlib/sqlite.md "$SEARCH_PYTHON" scripts/search_embed.py "$SEARCH_CACHE"
/tmp/plenty-search "$PLENTY_SQLITE_VEC" /tmp/search.db "$SEARCH_MODEL" find "How do I undo changes when a transaction fails?" "$SEARCH_PYTHON" scripts/search_embed.py "$SEARCH_CACHE"
index ID TITLE TEXT_FILE PYTHON SCRIPT CACHE reads one UTF-8 text document.
Plenty hashes its title and text before inference; unchanged input returns
immediately. Changed documents are split into 400-character windows with
64-character overlap, then embedded in one subprocess invocation. This simple
trial chunker can split sentences and Markdown structures; corpus-aware
chunking remains follow-on work. Empty files keep a document with zero chunks.
All replacement happens in one transaction after successful inference; a failed
subprocess, invalid response or wrong vector dimension leaves the old document.
find QUERY PYTHON SCRIPT CACHE embeds the query and runs hybrid search. The
process receives JSON text and returns JSON vectors; it never opens a database
or reads source files. There is no shell interpolation of paths or queries.
The helper reports errors on stderr, which Plenty preserves in its error.
Each indexing/query invocation loads the model anew, trading latency for a
simple setup without a background service.
Use show ID, status and delete ID with the same extension, database and
model arguments. Reindex after editing a file; explicitly delete its stable
ID after deleting it. Nothing modifies the original source or issues.db.
Vector input and document operations
Every invocation starts with EXTENSION DB MODEL. Use a new database path,
separate from the authoritative issues.db. MODEL identifies the exact model,
revision, normalization and chunking configuration; changing it requires a new
database. The CLI rejects a different identity or initialization dimension.
The following example uses synthetic two-dimensional vectors to test storage; they are not embeddings of the example text.
/tmp/plenty-search "$PLENTY_SQLITE_VEC" /tmp/search-trial.db fixture-v1 init 2
Save this JSON as /tmp/search-chunks.json:
[
{"text": "Owned resources are released on scope exit.", "embedding": [1.0, 0.0]},
{"text": "Borrows cannot outlive their owners.", "embedding": [0.9, 0.1]}
]
/tmp/plenty-search "$PLENTY_SQLITE_VEC" /tmp/search-trial.db fixture-v1 put docs:ownership "Ownership" /tmp/search-chunks.json
/tmp/plenty-search "$PLENTY_SQLITE_VEC" /tmp/search-trial.db fixture-v1 show docs:ownership
/tmp/plenty-search "$PLENTY_SQLITE_VEC" /tmp/search-trial.db fixture-v1 status
put replaces all chunks belonging to that document in one transaction. Its
hash covers the title, chunk text and supplied vectors. Identical input prints
unchanged. Parsing happens before the transaction; if insertion fails (for
example, a vector has the wrong dimension), the previous document remains.
An empty chunk array keeps the document with no searchable chunks.
Save [1.0, 0.0] as /tmp/search-query.json, then query:
/tmp/plenty-search "$PLENTY_SQLITE_VEC" /tmp/search-trial.db fixture-v1 query /tmp/search-query.json "resource cleanup"
/tmp/plenty-search "$PLENTY_SQLITE_VEC" /tmp/search-trial.db fixture-v1 delete docs:ownership
Query words are quoted individually and joined with OR for FTS5. Empty words give vector-only results. Results show document ID/title, zero-based chunk number, fused score and text. Vectors must come from the same model and preprocessing as the stored documents; the CLI cannot infer this from numbers. Supply finite, nonzero vectors suitable for cosine similarity.
delete removes the document, its chunks and embedded vectors through a foreign
key cascade; triggers remove the associated FTS5 entries. It never deletes a
source file. The trial uses ordinary SQLite BLOB columns and sqlite-vec
distance functions, so vector ownership follows normal relational deletion.
Validation
The deterministic integration test uses the actual extension and fake inference and checks insertion, unchanged input, search, rollback on malformed dimensions, replacement, deletion, Unicode chunking, skipped inference and subprocess failure preserving the previous document:
PLENTY_SQLITE_VEC="$PLENTY_SQLITE_VEC" cargo test --locked --test integration test_std_sqlite::sqlite_vec_search_trial -- --ignored
The test is explicitly ignored in the default suite because the extension is an optional external dependency. The ordinary SQLite tests exercise extension loading with a locally compiled fixture and check FTS5 without any download.
Search the repository
The local semantic index covers tracked Markdown documentation, source
documentation, and all local issue/PR titles and descriptions. It is a separate,
rebuildable database; issues.db remains authoritative. The CLI, extraction,
chunking, invalidation and queries run in Plenty. Python/FastEmbed performs only
embedding inference. See the search interface for the
model identity, storage schema and individual-document commands.
Setup and use
From a checkout, install the pinned dependencies/model and build the executable:
bash scripts/search.sh setup
bash scripts/search.sh sync
bash scripts/search.sh find 'How do borrowed values interact with worker threads?'
bash scripts/search.sh status
bash scripts/search.sh check
bash scripts/search.sh show repo:doc:book/src/design/stdlib/sqlite.md
Setup needs uv, Cargo and a C linker, plus network access for the initial
packages and model. Subsequent indexing/search uses the local model. build
rebuilds just the executable after changing the Plenty source. The wrapper uses
.search/ in the primary worktree for search.db, semantic-search, venv/
and models/. These generated files are ignored by Git. Model weights and the
environment are never part of an index snapshot.
The default corpus is the primary checkout, even when running the wrapper from
a task worktree. Set PLENTY_SEARCH_ROOT to index another checkout, or
PLENTY_SEARCH_HOME to use a separate index/environment. The issues register
must always be the primary worktree’s absolute issues.db path; the Plenty
program validates it against git worktree list.
Coverage and lifecycle
check is a deterministic freshness audit. It recomputes SHA-256 hashes of each
document’s title and extracted text, compares them with the hashes stored in
documents, and lists added, changed and deleted IDs. It opens the search
database read-only, never invokes inference and does not modify embeddings.
Exit status is zero when fresh and nonzero when stale or on an operational
error (the diagnostic distinguishes these). This lets scripts/agents check
freshness without relying on a model’s judgment. Only indexed text matters:
changing implementation code without changing extracted documentation need
not invalidate its embeddings. A rename is a deletion plus an addition.
repo:doc:PATH: tracked Markdown, including the book and root/project docs. Vendor trees and the duplicateBACKLOG.mdalias are excluded.repo:source:PATH: Rust//////!documentation with declaration context; Python and Plenty module/function/class triple-quoted docstrings; and Plenty module headers and comment blocks preceding declarations. Source is grouped by file so its embeddings can be replaced/deleted together. The lightweight lexical extractor skips strings/test fixtures; it is not a general AST parser. Rust block doc comments and#[doc = ...]attributes are not extracted.repo:issue:Nandrepo:pr:N: titles and descriptions from a read-only SQLite snapshot, across all statuses. Discussion comments and PR code/diffs are not read. Code snippets already contained in descriptions remain part of that text.
sync scans Git’s tracked paths (including staged new files and working-copy
edits), skips missing tracked files, and reads issues/PRs in one transaction.
Untracked files are excluded. It compares each document’s extracted title/text
hash before inference, groups changed documents, and embeds at most 128 chunks
per subprocess call. Each document replacement is transactional. Successful
batches survive an interruption and are reused on the next run.
Only after the whole scan and all embeddings succeed does synchronization
remove missing repo: documents. Explicitly indexed IDs outside that prefix
are preserved. It then compacts the database. Run one sync at a time. A failed
run does not prune missing documents, and any document whose replacement fails
retains its previous chunks. sync_info records the source checkout, its HEAD
and the last successful completion time; working-copy edits are also indexed,
so HEAD alone is not a content fingerprint.
Run sync after document edits/deletions or issue/PR updates. Search results can
be stale between runs. The current 400-character chunker can split paragraphs
and Markdown structures; search is a discovery aid, so read the original source
before relying on a result. A different model/chunker identity requires rebuilding
the database.
Tests
After setup/build, run the extraction/synchronization tests against the real SQLite extension and a deterministic fake inference process:
PLENTY_SEARCH_BIN="$PWD/.search/semantic-search" \
PLENTY_SQLITE_VEC="$(.search/venv/bin/python -c 'import sqlite_vec; print(sqlite_vec.loadable_path())')" \
python3 -B scripts/test_search_index.py
The tests cover extraction, skipped inference, changed docs/issues, deletion, failed inference, stale-register rejection, manual document preservation, read-only freshness auditing, foreign keys and FTS cleanup. They need no model download and skip unless both environment variables are set.
Analyzing emitted code
Use this procedure to make generated programs faster: find the loop that dominates a benchmark, compare its machine code with C’s, and account for every extra instruction before changing the compiler. Profiling the compiler itself is covered in Profiling the compiler.
The worked example is local PR #917, whose kernels, scripts and results are in
experiments/kernels/ (see its README.md). It took fannkuch-redux from
3.16x to 1.69x C’s instruction count.
Ground rules
- Measure optimized builds. Builds are optimized by default; never time or
count a
--fast-compilebuild, which runs Cranelift atopt_level=noneand skips call-site inlining. - Count instructions, not time.
perf stat -e instructions:uand per-pass counts read from assembly do not depend on machine load. Wall-clock numbers on a shared machine are noise; take them last, alternating builds in one session, and compare only within that session. - Pin parallel programs to physical cores (
taskset -c 0-7on the reference laptop), so a busy machine does not change thread placement. - Check outputs. Every build in a comparison prints identical output.
- Prove, never assume. A bounds, overflow or index-normalization check can be removed only when the compiler proves at compile time that it cannot fail. The safety requirement is that no out-of-bounds access or unreported overflow can happen, not that every access runs a check. Checks that can fail keep their trap kind, position and message.
Procedure
1. Find the hot loop
Count the whole program, then attribute by function:
plenty --no-strip --compile app.plenty -o /tmp/app
perf stat -e instructions:u taskset -c 0-7 /tmp/app ARGS
valgrind --tool=callgrind --callgrind-out-file=/tmp/cg.out /tmp/app SMALL_ARGS
callgrind_annotate /tmp/cg.out | head -30
perf record followed by perf annotate shows the hot instructions inside a
function. --no-strip keeps the symbols.
2. Isolate kernels
Write each hot loop as a standalone Plenty function and a C function that
does the same work with the same types, as experiments/kernels/kernels.plenty
and kernels.c do. Keep one loop per function, pass data as views or
pointers plus lengths, and give the C version no information Plenty’s lacks
(for example, no restrict unless Plenty’s borrows prove the same thing).
3. Get both disassemblies
gcc -O2 -fno-tree-loop-distribute-patterns -c kernels.c -o /tmp/kc.o
objdump -d --no-show-raw-insn /tmp/kc.o > /tmp/c.asm
plenty --dump ops,clif,clif-opt,asm kernels.plenty -o /tmp/k
-fno-tree-loop-distribute-patterns stops gcc replacing copy loops with
memmove, which would hide the loop. Use --dump-filter NAME to narrow the
dump. See Inspecting generated code
for the stages; clif-opt and asm are what runs.
A dump keeps every function body, while a compiled program drops the standalone bodies of functions inlined at every call site. Compare the inlined code in its caller when that matters.
4. Count instructions per pass
experiments/kernels/loops.py reads either disassembly and prints, for each
innermost loop, the instructions one pass runs on its hot path. Its README
shows how to normalize --dump asm output for it. Record the counts in a
table: one row per kernel, one column per build.
5. Attribute every extra instruction
Line up the two hot paths and give each extra Plenty instruction one cause:
| Cause | Typical shape (x86-64) | Usual fix lives in |
|---|---|---|
| Bounds check | cmp + jae to a cold block | Plenty: prove the index in range, reuse a dominating check, hoist |
| Negative-index join | lea of index + len and a second cmp | Plenty: prove the index nonnegative |
| Overflow check | add/seto/test/jne, or jo | Plenty: prove no overflow (loop counters, range facts) |
| Header or length reload | mov from the list owner each pass | Plenty: row caching across the loop |
| Register move | mov reg, reg with no other effect | Usually regalloc2; check whether a value passed to a cold call causes it |
| Spill or reload | mov to or from the stack | regalloc2, or too many live values from Plenty’s IR |
| Loop layout | unconditional jmp back to the header | Plenty: loop rotation |
| Call overhead | call of a small helper | Plenty: inlining |
| View or address construction | isplit, base + offset each pass | Plenty: build once outside the loop |
| Useful work | Present in C too | Nothing |
Sum the causes per kernel. A cause that accounts for many instructions across kernels is the next target.
6. Test IR shapes before changing the compiler
Copy the kernel’s clif-opt into a .clif file and change one shape per
variant: rotation, check placement, trap form (trapz against a cold call),
what a cold block receives, address arithmetic, block parameters. Compile each
with Plenty’s release flags and count it:
cargo build --release --manifest-path experiments/kernels/harness/Cargo.toml \
--target-dir target/kernel-harness
target/kernel-harness/release/plenty-kernel-harness FILE.clif \
| python3 experiments/kernels/loops.py
Record a variant table (letter, change, count). The best variant shows what
Cranelift can reach; the gap between it and C is the backend’s part. The
Cranelift IR reference
and the ISLE reference
describe the IR and the lowering rules. For the pinned version, read the
mid-end rules (opts/*.isle) and x64 lowering (lower.isle) in the
cranelift-codegen source under ~/.cargo/registry/src/.
7. Decide where the fix lives
- Plenty’s frontend or lowering: facts that remove checks, IR shapes that Cranelift lowers better. Implement these.
- Cranelift mid-end or lowering rules, regalloc2: write the finding up with the variant that demonstrates it, for an upstream report. Do not patch pinned dependencies in a Plenty PR.
8. Implement and verify
- Add tests that fail on the old code: CLIF or asm dump tests asserting the
check is gone or the shape changed, plus behavior tests for the cases that
must still trap (negative, out of range, near
i64.MAX). - Recount the kernel table, then the whole program, then compile time:
compile_bench 1000 10 --workload aotwithPLENTY_CACHE=offand a few larger programs compiled with--compile, counted withperf stat -e instructions:u. Report run-to-run spread. - Run the full suite.
9. Record
Put the kernel table, the attribution, the variant table and the
whole-program counts in the PR description. Keep kernels and scripts in
experiments/ so the comparison can be rerun. File an issue for each
remaining cause you did not fix.
Report template
## Kernel: NAME (source: FILE:LINE)
| Build | Instructions per pass |
|---|---|
| C (gcc VERSION -O2) | |
| Plenty before (COMMIT) | |
| Plenty after (COMMIT) | |
| Cause | Before | After | Notes |
|---|---|---|---|
| Bounds checks | | | |
| Negative-index joins | | | |
| Overflow checks | | | |
| Header reloads | | | |
| Register moves | | | |
| Spills | | | |
| Loop layout | | | |
| Calls | | | |
| Useful work (also in C) | | | |
| Variant | IR change | Count |
|---|---|---|
| a | as dumped | |
Whole program (instructions:u): before, after, C.
Compile time (instructions:u): compile_bench aot, other programs; spread.
Fixed here: ... Upstream: ... Remaining issues: #...
Pitfalls
- An unoptimized build makes every comparison wrong; check the dump index’s
optimizeline. - Cranelift GVN merges identical pure instructions, such as
iconstandstack_addr, and hoists them to a dominator, so a variant that duplicates them may compile to the same code. - A hot value passed to a cold call’s arguments can add moves on the hot path, because regalloc2 keeps the value in the call’s fixed argument register. PR #917 passes list-index failure operands through a stack slot for this reason.
- Iterating a collection directly (
for x in xs) has no bounds check; indexing in a loop has one unless the compiler proves it unnecessary.
Benchmarks Game suite
benchmarks/game/ holds ten programs from the
Computer Language Benchmarks Game
with a Plenty version of each, so that changes to generated code can be
measured against fixed programs and a recorded baseline. It is a regression
reference: it records where Plenty stands, and a slower run after a compiler
change is a finding to explain. To make a program faster, follow
Analyzing emitted code.
Layout
Each benchmark directory contains:
| Path | Contents |
|---|---|
benchmark.json | Provenance, site build and command lines, local build command, thread count, sizes, Plenty variants with any compiler flags, and status (ready or blocked with blocked_reason) |
reference/ | The reference program, copied verbatim from its site page |
expected/ | Output at the small check size, from the site’s downloadable output files |
plenty/ | The Plenty programs: the idiomatic one, and any parallel, data-layout or unchecked variant |
The reference programs are distributed under the Benchmarks Game’s revised
BSD license, reproduced in benchmarks/game/LICENSE-benchmarks-game.txt.
Each manifest names the program’s page and contributors.
| Benchmark | Reference | Threads | Plenty variants | Status |
|---|---|---|---|---|
| fannkuch-redux | C #1 | 1 | idiomatic; parallel (examples/fannkuch_redux.plenty, 8 workers) | ready |
| n-body | Rust #3 | 1 | idiomatic; clist (bodies in a clist); unchecked (--unchecked assume) | ready |
| spectral-norm | C #8 | 1 | idiomatic | ready |
| mandelbrot | C #8 | 1 | idiomatic | ready |
| fasta | Rust #3 | 1 | idiomatic | ready |
| k-nucleotide | Rust #1 | 7 | idiomatic (1 thread); parallel (7 pool workers) | ready |
| reverse-complement | C #5 | 1 | idiomatic | ready |
| binary-trees | C #1 | 1 | idiomatic | ready |
| pidigits | C #1 (GMP) | 1 | none | blocked: no arbitrary-precision integers (backlog B38, issue #481) |
| regex-redux | Rust #1 | 9 | idiomatic, with std.regex (1 thread) | ready |
Each reference is one C or Rust program from the site. None is marked with an
asterisk, the site’s mark for a program that uses a different algorithm, so
each benchmark has one idiomatic Plenty program that follows the reference’s
algorithm. Where the reference uses
threads and the Plenty program does not, the table says so; compare CPU time
(user_s) as well as wall time for those.
Running
The runner, benchmarks/game/bench.py, uses only the Python standard
library. It needs the release compiler (cargo build --locked --release),
clang, rustc and cargo for the references, and optionally perf,
taskset and GNU time. regex-redux’s reference is a Cargo package that
uses the regex version pinned in the workspace lockfile; it builds offline
from the local registry.
python3 -B benchmarks/game/bench.py list
python3 -B benchmarks/game/bench.py check # build all, verify check sizes
python3 -B benchmarks/game/bench.py run --mode quick n-body # one benchmark, small size
python3 -B benchmarks/game/bench.py run --mode full --perf --output results.json \
--compare benchmarks/game/baseline.json
python3 -B benchmarks/game/bench.py compare benchmarks/game/baseline.json results.json --metric instructions
Builds, generated inputs and outputs go under target/bench-game/. run:
- builds the references with the commands in their manifests, which match the
site’s flags except
-march=native(-C target-cpu=native) instead of the site’sivybridge; - builds each Plenty variant with
plenty --compileand the variant’sflags; - checks every program’s output at the check size against
expected/; - generates the fasta inputs for k-nucleotide, reverse-complement and regex-redux with the fasta reference;
- runs each program
--repeatstimes (default 3), pinned withtaskset -c 0,2,4,6,8,10,12,14(one thread per physical core on the reference laptop), recording wall time, CPU time and peak RSS, and reports the median and minimum; - compares each Plenty program’s output at the timed size with the reference’s output from the same run;
- with
--perf, adds oneperf statrun forinstructions:uandcycles:u; - writes JSON with the commit, tool versions, build commands and machine.
n-body and spectral-norm outputs compare within 1e-8; the others must be byte-identical.
compare (or run --compare) reports each Plenty program whose metric grew
by more than --threshold (default 0.10) and exits 1 if any did. Metrics are
time (median wall time), ratio (median relative to the reference in the
same run, which transfers better between machines) and instructions
(the least noisy on one machine). Benchmarks run at different sizes are
skipped.
Sizes and baseline
quick sizes finish in about half a minute in total. full sizes are the
site’s except reverse-complement, which uses the 25,000,000 fasta input shared
with k-nucleotide (about 250 MB) instead of the site’s 100,000,000 (about
1 GB). benchmarks/game/baseline.json holds a full run on the reference
laptop (AMD Ryzen 7 7840HS, performance governor):
| Benchmark | Size | Reference | Plenty | Ratio |
|---|---|---|---|---|
| binary-trees | 21 | 11.11 s | 50.38 s | 4.54 |
| fannkuch-redux | 12 | 24.64 s | 29.21 s | 1.19 |
| fannkuch-redux (parallel, 8 threads) | 12 | 24.64 s | 3.30 s | 0.13 |
| fasta | 25,000,000 | 1.49 s | 4.17 s | 2.81 |
| k-nucleotide | 25,000,000 | 11.50 s (7 threads) | 60.63 s | 5.27 |
| mandelbrot | 16,000 | 12.49 s | 12.65 s | 1.01 |
| n-body | 50,000,000 | 1.27 s | 42.89 s | 33.67 |
| pidigits | 10,000 | blocked | blocked | |
| regex-redux | 5,000,000 | 0.74 s (9 threads) | 59.70 s | 80.66 |
| reverse-complement | 25,000,000 | 0.36 s | 6.39 s | 17.91 |
| spectral-norm | 5,500 | 1.26 s | 3.78 s | 3.00 |
Times are median wall-clock seconds. The k-nucleotide and regex-redux
references are multithreaded while the idiomatic Plenty programs run one
thread; user_s in the JSON gives CPU time (36.2 s for the k-nucleotide
reference, 0.65 s for regex-redux).
k-nucleotide’s parallel variant postdates this baseline and has no entry in it
yet; record it with the next full run. It counts the seven frame lengths on
seven pool workers, as the reference does, and is the suite’s threaded workload
for reference-count changes (issue #751): each job owns a Shared handle to the
sequence, and its dictionary, with heap-string keys for the 12- and 18-byte
frames, returns through a future and is released on the main thread.
n-body’s clist variant also postdates the baseline. It stores the bodies in a
clist[Body], one column per field, instead of a list[Body], and tracks the
cost of indexed clist field access in loops (issue #759): at 1,000,000 steps it
runs 5.66 billion instructions against 2.95 billion for the list[Body]
program and 0.23 billion for the reference.
n-body’s unchecked variant also postdates the baseline. It is the idiomatic
program with the index arithmetic of energy and advance in
unchecked: suites,
built with --unchecked assume, so it measures what integer overflow checks
cost. List bounds checks remain, since no build setting removes them. At
1,000,000 steps it runs 2.85 billion instructions against 2.95 billion for the
idiomatic program.
Blocked benchmarks
A benchmark whose Plenty program needs a missing feature keeps its reference,
expected output and sizes, with "status": "blocked" and the reason. The
runner prints the reason instead of timing it, and
tests/test_benchmarks_game.rs asserts it stays blocked with a phase-guard
comment naming the backlog item. pidigits waits on std.bigint (backlog B38,
issue #481). Its reference also needs GMP’s gmp.h and -lgmp.
Investigating a regression
dump writes the Plenty program’s optimized Cranelift IR and assembly, and
the reference’s disassembly, under target/bench-game/dump/NAME/:
python3 -B benchmarks/game/bench.py dump n-body --filter advance
From there, follow Analyzing emitted code and its
emitted-code agent skill (.agents/skills/emitted-code/SKILL.md): isolate
the hot loop as a kernel next to an equivalent C kernel, as in
experiments/kernels/, and attribute each extra instruction before changing
the compiler. run --perf --metric instructions confirms the result on the
whole program.
Tests
tests/test_benchmarks_game.rs compiles every Plenty variant without
optimization and checks it at the check size, so a compiler change that breaks
a benchmark fails cargo test. It does not time anything or build the
references. benchmarks/game/test_bench.py tests the runner:
cargo test --locked --test integration test_benchmarks_game::
python3 -B -m unittest discover -s benchmarks/game -p 'test_*.py'
Adding a benchmark
- Copy the reference program verbatim from its site page into
reference/and its site output intoexpected/. - Write
benchmark.jsonfollowing an existing one, then the Plenty program. - Add the benchmark to
ready_benchmarks!(orBLOCKED) intests/test_benchmarks_game.rsand to the table above. - Run
bench.py check, then a full run, and updatebaseline.json.
Profiling the compiler
For fresh-process cold, unchanged and edited builds, see
Measuring module edits. That workload complements the fixed
compile_bench baseline and validates cached objects against uncached ones.
The profile Cargo feature reports, per compiler phase, the call count, time
percentiles, total time and, with profile-alloc, the bytes and number of
allocations. It uses hotpath, pinned to
one version because its report formats change between releases. The feature
is off by default; without it no hotpath code is compiled and the compiler
binary is unchanged.
Running a profile
The profile is recorded only when PLENTY_PROFILE is set, and is printed when
the process exits:
cargo build --release --features profile-alloc --example compile_bench
PLENTY_CACHE=off PLENTY_PROFILE=1 taskset -c 0,2,4,6,8,10,12,14 \
target/release/examples/compile_bench 1000 10 --aot
The same works for the plenty binary, for example
PLENTY_PROFILE=1 target/release/plenty --emit-object FILE -o OUT. Build with
--features profile for time alone. Building with either feature replaces the
release build of the compiler in target/; pass --target-dir to keep both.
| Variable | Effect |
|---|---|
PLENTY_PROFILE | Record and print a profile. Unset, the instrumented functions record nothing. |
HOTPATH_OUTPUT_PATH | Write the report to this file instead of stdout. |
HOTPATH_OUTPUT_FORMAT | table (default), json, or json-pretty. |
HOTPATH_ALLOC_CUMULATIVE | true counts a function’s allocations with those of the functions it calls. |
PLENTY_CACHE=off | Compile every function; otherwise cached functions skip Cranelift. |
The compiler turns hotpath’s live-metrics HTTP server off unless
HOTPATH_METRICS_SERVER_OFF is set.
To see what the compiler generated rather than how long it took, use
plenty --dump: it writes each
function’s operations, Cranelift IR before and after optimization, and
assembly, with an index sorted by code size. To make generated programs
faster, follow Analyzing emitted code.
Selecting a benchmark workload
For generated-program dispatch rather than compiler phase attribution, see match dispatch measurements.
The first two compile_bench arguments are generated units (not lines) and
repetitions. A unit contains a protocol, class, enum and functions. Choose one
workload per invocation to avoid mixing source sizes and checking with AOT:
cargo build --locked --release --features profile --example compile_bench
PLENTY_CACHE=off PLENTY_PROFILE=1 HOTPATH_OUTPUT_PATH=check.txt \
target/release/examples/compile_bench 1000 10 --workload check --no-warmup
PLENTY_CACHE=off PLENTY_PROFILE=1 HOTPATH_OUTPUT_PATH=scale.txt \
target/release/examples/compile_bench 1000 10 --workload scale --no-warmup
PLENTY_CACHE=off PLENTY_PROFILE=1 HOTPATH_OUTPUT_PATH=aot.txt \
target/release/examples/compile_bench 1000 10 --workload aot --no-warmup
| Selection | Work performed |
|---|---|
--workload check | Check the requested unit count only. |
--workload scale | Check four times the requested unit count only; the metadata shows requested_units and the actual units checked. No ratio is printed without a base check. |
--workload aot | Compile and link the requested unit count only, including its frontend checks. |
--workload check-and-scale (default) | Check both sizes and print the scaling ratio. |
--workload all or legacy --aot | Check both sizes and also compile/link the base size. |
Without --no-warmup, each selected workload runs once before its timed
repetitions. That warm-up is excluded from benchmark medians but included in
the profiler report. With --no-warmup, ten repetitions mean ten calls of the
selected workload. Source generation and benchmark setup are still within the
outer main measurement. workload and warmup appear in the printed metadata;
--help lists the options. Conflicting selections and invalid counts fail
before profiling starts.
All workloads also work without profiling. Disable the cache deliberately for cold-build measurements; for warm-cache
comparisons use an isolated PLENTY_CACHE_DIR, retain it between runs, and omit
PLENTY_CACHE=off. Cache hits skip Cranelift but still build IR and assemble the
object. compile_bench never uses program objects;
pass --no-program-cache to plenty for the same warm-function measurement,
since an unchanged program otherwise skips the frontend and code generation. These modes do not collect a whole-test-suite profile.
Reading the report
Each row is one measured function: the coarse phases of the
compiler pipeline, such as
modules::parse, frontend::lower_function, ownership::check,
op::check, codegen::build_user_function, output::compile,
codegen::emit_object and toolchain::link.
Additional rows subdivide the larger phases:
| Parent | Subphase rows and boundaries |
|---|---|
output::compile | output::cranelift_compile: Cranelift compilation or cache lookup/compilation, including cache bookkeeping; output::collect_code: copy machine code and convert relocations after success; output::clear_context: clear reusable compiler state, including after a compilation error. These do not expose individual Cranelift passes. |
ownership::check | ownership::prepare_futures, ownership::initialize_state, ownership::check_moves (outer sequence and final release), and ownership::check_loans. |
ownership::check_loans | For bodies with loans: ownership::loan_graph (CFG, reachability and loan tables), ownership::loan_liveness (fixed point), and ownership::validate_loans (conflict checks and diagnostics). The no-loan scan remains in the parent. |
frontend::lower_function | frontend::setup_function (initial state, parameters and parameter loans; for a foreign function, its whole lowering), frontend::lower_function_body (the outer body-lowering call), and frontend::finish_function (return inference and final operation-body construction). |
frontend::lower | Also reports generics::expand_data and tail_calls::validate; both can return early when there is no relevant work. |
Recursive ownership walks and nested expression/block lowering are not annotated on every visit. A loan-bearing function produces one graph/liveness/validation measurement per ownership check, regardless of CFG nesting. Function lowering can recursively specialize or re-lower a function during inference; those nested function-level measurements remain inclusive.
- Time is inclusive:
frontend::lowercontains everylower_functioncall. % Totalis relative to the whole run. Functions that run on every core, such asownership::check(insideop::check),build_user_functionandoutput::compile, sum the time of all threads and can exceed 100%.- Allocation rows are exclusive by default: bytes allocated by the function itself and unmeasured functions it calls, not by measured functions below it. Adding subphase rows moves allocation attribution from parents to children; compare sums with the same instrumentation, or deliberately use cumulative allocation mode. Parent time still includes its children: never sum all rows as independent costs.
- A function measured per call costs about 40 ns, so only phase-level
functions are measured. To find hot functions inside a phase, use a
sampling profiler such as
perf recordon a build without the feature. - Allocation counting adds work to every allocation: in one measurement
it added about 2% to Cranelift compilation. Compare times from a
profilebuild. - The compiler executables and
compile_benchallocate with mimalloc, andprofile-alloccounts allocations on top of it. Integration tests call the compiler in-process with the system allocator, so their timings do not show the executables’ allocator behavior. - For thread contention, compare system time and voluntary context switches
(
/usr/bin/time -f '%e %U %S %w %c %M') as well as wall time. Those counters include child processes:compile_bench --workload aotand--compilecount the linker’s threads too, whileplenty --emit-objectcounts only the compiler. - hotpath’s CPU sampling mode (
hotpath-cpu) is not enabled: it needs samply and relaxed kernel settings, andperfalready samples the compiler without a feature.
On 2026-10-18 (AMD Ryzen 7 7840HS, eight physical cores, PLENTY_CACHE=off),
compile_bench -- 1000 10 --aot gave the same medians with the feature off,
built but not recording, recording time, and recording allocations: the
fastest of three runs was 646, 648, 644 and 662 ms for the AOT build. Noise
from other load was larger than these differences. A clean release build of
the compiler with the feature compiled 27 more crates and took 10 s longer
(91 s against 81 s with -j 3). Those measurements predate the finer subphase
annotations; measure overhead again when evaluating short-function costs and
confirm optimizations on a feature-disabled build. Use sampling to identify
Cranelift passes and hot leaf functions rather than adding per-operation rows.
cargo test --all-features builds the feature too. The instrumented functions
then record nothing, because no test sets PLENTY_PROFILE except
tests/test_profile.rs, which checks the report.
Other benchmarks and historical results
- Function storage and scalar list rows: allocation and generated-code measurements.
- Match dispatch: dense integer, enum and string dispatch experiments.
- Serial pipelines and generators: callback and generator benchmark commands and results.
- Parallel pipelines: the parallel benchmark command and results.
Function storage and list-row measurements
Historical optimization measurements; see Profiling the compiler for the profiling guide.
Reusing function storage
The B19 follow-up measured on 2026-10-22 reuses the cleared Cranelift Function
already held by each emission worker. Ordinary user functions take its buffers
before building IR and return them through the existing compilation path.
Cache hits and misses both clear function state. Generated adapters and
generator bodies still start with fresh IR; there is no additional pool or
Cranelift patch. Larger buffers can remain until that worker finishes.
On the AMD Ryzen 7 7840HS, pinned to eight physical cores
(taskset -c 0,2,4,6,8,10,12,14), a release profile-alloc build running
compile_bench 1000 1 --workload aot --no-warmup reported these exclusive
allocations (the profiler’s KB/MB units):
| Allocation | Before | Reused storage |
|---|---|---|
build_user_function, cache disabled, per call | 20.3 KB | 4.9 KB |
| Same phase, all 7,001 calls | 138.5 MB | 33.4 MB |
| Same phase, warm function cache | 138.5 MB | 33.5 MB |
| Whole run, warm function cache | 642.4 MB | 535.5 MB |
Function construction allocated about 76% less; total warm-cache allocation
fell about 17%. These replace the older 34.7 KB/function observation in B19;
other compiler improvements had already reduced that baseline. The whole-program
cache is disabled by compile_bench, so a warm run still builds each function’s
IR. Baseline and changed 1,000-unit program objects were byte-identical.
Feature-disabled release binaries were then run in three alternating pairs
on the same cores, with no other agent builds or tests running. The default
Plenty compilation mode was used; times include checking, emission, runtime
extraction and linking. Disabled-cache and warm-cache processes used five
timed repetitions after one warm-up; each cold-cache process used a new empty
directory and one repetition without warm-up. Values below are medians of
the three process results; RSS is peak resident memory from /usr/bin/time.
| Units | Function cache | Before, ms | Reuse, ms | Before, MiB | Reuse, MiB |
|---|---|---|---|---|---|
| 1,000 | Disabled | 438 | 402 | 116.3 | 117.1 |
| 1,000 | Cold | 527 | 534 | 94.9 | 97.1 |
| 1,000 | Warm | 307 | 313 | 120.1 | 120.0 |
| 5,000 | Disabled | 2,317 | 2,261 | 404.1 | 405.8 |
| 5,000 | Cold | 2,263 | 2,385 | 384.6 | 384.4 |
| 5,000 | Warm | 1,620 | 1,799 | 402.5 | 402.6 |
Latency differences changed sign between paired runs. For example, the three 5,000-unit warm-cache pairs changed by +1.6%, +24.2% and -0.4%; these noisy runs establish no consistent speedup. Peak memory stayed similar on these many-function workloads. The reason to retain this small change is its allocation reduction without an extra pool or dependency.
A separate warm-cache perf stat -e instructions,cycles comparison pinned to
one core (taskset -c 0, 1,000 units, three repetitions plus warm-up) counted
11.811 billion user-space instructions before and 11.532 billion after
(-2.4%); cycles were 9.190 and 9.151 billion. This single pair supports lower
compiler work, while adding no reliable latency claim.
Inline scalar list rows
Issue #509 found std.diff’s Myers loop spending most of its time in runtime
calls: on 3,000 dissimilar lines, perf attributed 30% of cycles to
plenty_list_scalar_get, 27% to the Entries::get it called and 19% to
plenty_list_scalar_set, against 21% in the std.diff functions. The access
forms were borrowed &list[i64] parameters indexed in while loops and list
fields of a &mut record read and written in the same loop. Only for loops
over an unchanged borrowed parameter hoisted a row view, so these forms called
the runtime per element.
Codegen now reads and writes borrowed scalar list rows inline through the
fixed offsets in plenty-runtime/src/list_layout.rs (collections).
Screening came first: --dump asm of std.diff.common_prefix shows each read
as three loads from the owner, the index normalization, an unsigned compare
and the row load, with a cold call to plenty_trap_list_index; afterwards
perf attributes 94% of the cycles to std.diff functions and none to list
helpers. User-space instructions and cycles (perf stat, one run, pinned to
the eight physical cores):
| Benchmark | Instructions before / after | Cycles before / after |
|---|---|---|
dis.plenty (3k dissimilar lines, Myers) | 33.31 / 12.85 billion | 7.79 / 3.49 billion |
bench.plenty (all cases) | 81.10 / 38.73 billion | 17.90 / 9.24 billion |
large.plenty (50k lines, 150 edits) | 7.51 / 7.18 billion | 2.59 / 2.37 billion |
Best of three interleaved runs of bench.plenty’s five-repetition minimums
(load average 0.5): dissimilar 3k Myers 265 to 100 ms, patience 265 to 100 ms;
repetitive 3k 7.8 to 3.9 ms for both; similar 3k Myers 2.03 to 1.94 ms;
large 50k Myers 38.4 to 36.2 ms, patience unchanged at 54 ms. Edit counts are
identical. The large case is dominated by splitting and interning lines. The
benchmark programs are outside the repository; issue #509 names them.
Compile cost is small: --compile of bench.plenty rose from 418.9 to 423.0
million instructions (1.0%), its executable’s .text grew 644 bytes, and
compile_bench 1000 10 --workload aot stayed at 76.2 billion instructions.
std.diff.middle_snake grew from 3,320 to 4,168 bytes of machine code.
Match dispatch measurements
Historical dispatch experiments; see Profiling the compiler for the profiling guide and the emission reference for current behavior.
Dense-match dispatch
The 2026-10-22 B19 experiment at 9c994af0 measures dense integer and enum
dispatch with 4, 16, 64 and 256 arms against the then-production comparison chains.
A temporary Cranelift Switch patch produces br_table; the patch is preserved
as the original experiment. The production policy addresses
issue #395. Reproduction, samples and generated-fixture methodology
are in examples/dense_match/.
On a Ryzen 7 7840HS, x86-64 Linux, Cranelift 0.131.1, pinned to one CPU, median wall times over seven shuffled rounds for two million dispatches were:
| Arms | Integer uniform: chain / switch | Enum uniform: chain / switch | Integer 90%-last: chain / switch |
|---|---|---|---|
| 4 | 13.1 / 15.7 ms | 23.0 / 26.9 ms | 8.5 / 9.9 ms |
| 16 | 16.7 / 18.4 ms | 27.2 / 29.2 ms | 11.3 / 10.5 ms |
| 64 | 23.1 / 18.7 ms | 33.9 / 30.8 ms | 22.6 / 10.0 ms |
| 256 | 48.1 / 19.4 ms | 84.0 / 31.2 ms | 68.0 / 11.0 ms |
The same-session baseline and prototype runs interleave. Times include process
startup, all-arm correctness checks, the loop and output; enums also include
borrowed list indexing. They are not pure dispatch latency or integer-versus-enum
ABI comparisons. A Python oracle checks every sequence’s checksum; integer
fallbacks include the first out-of-range value and u64.MAX.
The 256-arm enum 90%-last case fell from 111.4 to 14.6 ms. First-arm-heavy cases show why distribution matters: switches regressed at 4/16/64 arms for both integers and enums; at 256 they improved from 11.7 to 9.2 ms and 18.2 to 14.6 ms. A balanced Plenty integer decision tree was slower than chains for uniform 4/16/64-arm inputs (14.7/22.9/30.2 ms), and slower than the switch at 256 (37.9 ms). Neither balanced trees nor switches should replace every chain.
At 256 arms, isolated dispatch size fell from 5,515 to 3,656 bytes for integers and 7,763 to 3,688 for enums, including table data/padding. At four arms it grew from 91 to 128 and 131 to 152 bytes. Optimized CLIF instruction counts at 256 fell from 1,284 to 521 and 1,549 to 534. Cold compile/link medians for the full uniform fixtures were 69/65 ms (integer chain/switch) and 95/99 ms (enum); these include validation/setup code and linking, with only three samples in separate batches. They establish no reliable compile-time gain.
Production policy
The production lowerer now selects a jump table for at least 256 distinct contiguous integer/tag values, in any source order (issue #395). Smaller and sparse sets retain chains. The threshold deliberately leaves the mixed 64-arm results alone; it is not claimed to be optimal for every CPU or distribution. Regression checks cover all eight integer widths, negative/offset ranges, signed extrema, unsigned maxima, distant 64-bit fallbacks, reversed cases, alternatives, nested payload ordering, owned/borrowed enum payloads and tail exits. Exhaustive enum matching retains the invalid-tag trap. The emission contract describes index normalization and bounded table storage.
The harness also offers --kinds work: each arm computes a different wrapping
multiply/add over a runtime state argument. This avoids drawing the entire
conclusion from constant-return arms, while remaining a synthetic workload.
It does not establish a benefit for any existing application. No source syntax
or tutorial idiom changes follow from this optimization.
A follow-up paired run against the chain compiler at 70aa0a3e validated the
production policy on the same host and CPU affinity, using two million
dispatches and seven shuffled rounds (672 checked timed executions):
| 256-case workload | Uniform chain / table | 90%-first chain / table | 90%-last chain / table |
|---|---|---|---|
| Integer | 55.6 / 21.0 ms | 12.2 / 10.5 ms | 79.7 / 12.1 ms |
| Enum | 100.1 / 35.1 ms | 19.7 / 15.6 ms | 127.7 / 17.8 ms |
| Runtime multiply/add | 57.6 / 22.1 ms | 13.5 / 10.5 ms | 82.1 / 11.8 ms |
Isolated dispatch bytes, including table data/padding, fell from 5,515 to 3,646
for integers, 7,763 to 3,674 for enums and 7,623 to 5,754 for runtime arithmetic.
All 26 optimized-IR/assembly dump files for the smaller matches and unchanged
tree controls are byte-identical between compilers.
Full samples, compiler/source hashes and dump indexes are preserved in
production-results.json.
Cold build samples were collected in separate batches with other builds active
during the baseline batch; they establish no compile-time gain. The machine
was not reserved or frequency-locked, so small timing differences in unchanged
controls are noise rather than evidence for tuning smaller matches. This run
does not measure allocation or peak memory, nor justify a broader sparse-table
policy.
String match dispatch
A match with at least 16 distinct str literals before its fallback calls
plenty_str_match once and branches through a br_table (issue #565). The
runtime binary-searches a read-only table of the literals, ordered by byte
length and then bytes; each entry maps to the first source arm with that
literal. Smaller matches keep the plenty_str_eq chain. The table stores
32-bit offsets followed by the literal bytes, so it needs no load-time
relocations in a PIE.
The 2026-10-30 measurements at 1756617d used a Ryzen 7 7840HS, x86-64 Linux
and Cranelift 0.137, pinned to the eight physical cores. Each program matches
the first n sorted IANA zone names (a seeded sample below 597) and four
misses: a longer string, a shorter prefix, the empty string and an unknown
name. Inputs are split from one literal at run time, so no lookup compares
literal addresses. Times are medians and ranges of five shuffled rounds of
two million lookups, including process start:
| Arms | Uniform: chain / table | First arm only: chain / table | find bytes: chain / table |
|---|---|---|---|
| 5 | 32.5 / 32.9 ms (chain kept) | 18.3 / 17.1 ms | 283 / 283 |
| 15 | 58.5 / 59.6 ms (chain kept) | 17.8 / 17.7 ms | 693 / 693 |
| 16 | 59.7 [59.4–60.5] / 49.2 [48.6–52.1] ms | 18.2 / 28.0 ms | 734 / 357 |
| 50 | 123.8 [123.0–125.0] / 65.1 [63.5–66.2] ms | 17.7 / 30.9 ms | 2,128 / 831 |
| 597 | 1,078.8 [1,077.8–1,087.6] / 110.6 [109.4–112.3] ms | 18.7 / 46.3 ms | 24,555 / 8,491 |
At 597 arms a uniform lookup falls from about 530 to 46 ns. The table data adds 4(n + 1) bytes plus the literal bytes. Optimized CLIF instruction counts fall from 115 to 55 at 16 arms and from 3,601 to 1,217 at 597. Cold compile times were within noise of each other (66–71 ms at 597 arms).
A run that forced the table for every size found the chain faster for uniform
inputs below about 12 arms (5 arms: 33.8 / 38.1 ms), so 16 is where the table
clearly pays. When the first arm matches every input, the chain always wins,
by about 5 ns per lookup at 16 arms and 14 ns at 597: the chain stops after
one comparison, while the search always takes about log2(n) steps. Order arms
by frequency only in a small match; a large match no longer depends on order.
A same-length comparison by 8-byte big-endian words was slower than the
byte-slice comparison, which calls memcmp, and was not kept.
Serial pipeline measurements
Historical measurements; see Profiling the compiler for the profiling guide.
Serial pipeline callbacks
scripts/bench_pipelines.py compares an inline range/filter/map/fold loop with
fused pipelines using named callbacks, closure literals, and immutable scalar
captures. All variants do the same checked arithmetic: filter even values,
square each accepted value modulo 1,000, and add the results. A second pair
compares a loop with a closure-literal map whose body is a conditional
expression, against its own inline loop. Run:
python3 -B scripts/bench_pipelines.py target/debug/plenty --items 100000000 --runs 7
The compiler executable can be a debug build: the harness requests optimized
native output with --release, disables caches, checks every result, interleaves
executables and reverses their order on alternate rounds. Compilation and cold
checking of 100 separate functions are measured separately. This is a repeatable
measurement tool, not a timing-sensitive unit test.
Issue #399 recorded the original named-callback pipeline at 1.906 times its inline-loop baseline. A fresh pre-change run on this host measured 0.160906 s for the inline loop and 0.283777 s for the pipeline (1.764 times, medians of seven executions over 100 million items). The direct callback calls survived fusion. Bounded scalar leaf substitution removes those calls, while SSA mirrors of immutable scalar captures expose constant divisors to native optimization.
Only straight-line scalar leaf bodies with at most 24 meaningful operations qualify. Capture mirrors follow local assignments and joins; exclusive borrowing disables them for the affected local. Checked arithmetic keeps the original callee source location. Debug code and larger or effectful callbacks retain their existing behavior. Broader callback substitution needs separate evidence.
On the same host, a quiet post-change run over 100 million items (seven interleaved executions per variant) measured:
| Variant | Native median | Relative to inline loop | Cold check, 100 functions | Single-program compile |
|---|---|---|---|---|
| Inline loop | 0.156573 s | 1.000 | 0.011199 s | 0.046002 s |
| Named callbacks | 0.155743 s | 0.995 | 0.011768 s | 0.043120 s |
| Closure literals | 0.169209 s | 1.081 | 0.023402 s | 0.042156 s |
| Scalar captures | 0.154756 s | 0.988 | 0.023753 s | 0.044352 s |
All executions returned 24500000000; all three pipeline forms meet the
proposal’s within-10% runtime target for this workload. Closure literals still
cost about 8% here, and checking 100 closure-heavy functions costs roughly twice
the named-function variant. Single-program compilation is one observation per
variant, not evidence of a compilation speedup. The machine was not frequency-
locked or CPU-pinned; small runtime differences around parity are noise. These
results do not promise the same ratios for larger, branching or mutable callbacks.
Closure-literal splicing
Closure literals written in a chain are now lowered in the consumer’s loop
rather than as hidden functions (B75 stage 2). Before timing, --dump ops showed
no ClosureNew or ClosureCall in the fused functions, and the optimized asm of
the filter/map/fold closure variant has no call inside the loop. The branching
variant’s asm has 107 instructions against 108 for its inline loop.
Both compilers were measured on this host back to back, pinned with
taskset -c 0,2,4,6,8,10,12,14, over 100 million items with seven interleaved
executions per variant:
| Variant | Before: median | Before: ratio | After: median | After: ratio |
|---|---|---|---|---|
| Inline loop | 0.155813 s | 1.000 | 0.157660 s | 1.000 |
| Named callbacks | 0.157610 s | 1.012 | 0.156564 s | 0.993 |
| Closure literals | 0.159477 s | 1.024 | 0.156964 s | 0.996 |
| Scalar captures | 0.159048 s | 1.021 | 0.156939 s | 0.995 |
| Branching loop | 0.223247 s | 1.000 | 0.222619 s | 1.000 |
| Branching closure | 0.234639 s | 1.051 | 0.225732 s | 1.014 |
Cold checks of 100 functions, medians of seven runs:
| Variant | Before | After |
|---|---|---|
| Inline loop | 0.003607 s | 0.004034 s |
| Named callbacks | 0.003654 s | 0.003728 s |
| Closure literals | 0.005867 s | 0.005057 s |
| Scalar captures | 0.007005 s | 0.004893 s |
| Branching loop | 0.004116 s | 0.003629 s |
| Branching closure | 0.005514 s | 0.004041 s |
All runs returned 24500000000 and 16850000993. Closure literals now run
within 1.5% of their inline loops. Checking closure-heavy functions fell from
about 1.6 to 1.4 times the named variant; the remainder is per-literal capture
and splice lowering. Cold-check numbers are one series per compiler and vary by
roughly 10% between runs.
Fluent generators and explicit conversion
scripts/bench_pipeline_generators.py measures the same filter/map/fold workload
with range and generator sources, local fused consumption, explicit .generator()
conversion. The compiler executable requests
optimized native output, disables caches, checks equal results, and alternates
execution order as in the callback benchmark:
python3 -B scripts/bench_pipeline_generators.py target/debug/plenty --items 5000000 --runs 7
A quiet run on this host measured the following medians. Every execution returned
1225000000.
| Variant | Native median |
|---|---|
| Inline range loop | 0.010011 s |
| Local fused range pipeline | 0.009781 s |
| Inline generator loop | 0.042060 s |
Generator source through iter(...) | 0.044602 s |
| Direct fluent generator source | 0.044454 s |
Range pipeline converted with .generator() | 0.027202 s |
Generator-source pipeline converted with .generator() | 0.065896 s |
std.iter chain with the same generator source (removed module; see the discussion after the table) | 0.105389 s |
Direct fluent syntax has the same cost as wrapping the source in iter(...)
(0.997 times here), and costs 1.057 times the inline generator loop. Explicit
conversion adds a resumable frame: the converted range pipeline costs 2.781 times
local consumption. With the same generator source, the converted pipeline took
0.625 times the runtime of a nested std.iter adapter chain. That row was
measured at commit 57982e0f; std.iter has since been removed, and the script
no longer runs it. Internal filter/map adapters are fused
into one frame; resuming the source generator and the converted frame still has
a cost. Keep local consumers fused when the stream does not need to escape.
The existing 100-million-item callback benchmark was also rerun with seven interleaved executions. Named callbacks, closure literals and scalar captures measured 0.999, 1.066 and 1.000 times the inline loop respectively, retaining the within-10% target. Compiler tests separately check a single converted frame with inlined scalar callbacks, and zero tracked allocations while constructing, moving, resuming and dropping an owned scalar pipeline. These timings are observations on a machine without frequency locking or CPU pinning; they do not establish universal ratios or allocation behavior for heap-owning source items.
Parallel pipeline measurements
Historical measurements; see Profiling the compiler for the profiling guide.
Parallel pipelines
scripts/bench_par_pipelines.py compares a serial pipeline, the equivalent
pool.map/reduce_tree calls, and the same pipeline after .par(&pool) (B75
stage 4). Cases are map+collect, filter+count, sum over i64 lists,
i64 ranges and f64 lists, and reduce. The cheap callback is x * 3 + 1;
the expensive one runs 100 rounds of a linear congruential step. The pool
column for filter+count maps the predicate to 0/1 and sums serially;
for the sums and reduce it is reduce_tree over pool.map. Run:
taskset -c 0,2,4,6,8,10,12,14 python3 -B scripts/bench_par_pipelines.py target/release/plenty
The taskset mask keeps the eight-worker pool on the eight physical cores of
this laptop; more than ten busy threads drop its clock to about 1.1 GHz. Each
program builds its input outside the timed region, times one terminal with the
monotonic clock per repetition, discards the first repetition and prints a
checksum; every variant of a case must agree. The script interleaves the
variants over three rounds of four timed repetitions and reports medians in
milliseconds. The machine was not otherwise idle (load average 2 to 5 from
other work), so differences under about 10% are noise.
Screening. Before timing, --dump asm of the chunk workers showed the
same per-item calls as the serial loop (plenty_list_scalar_get and, when
collecting, plenty_collection for the append) for map, filter and
count. The signed-sum worker folded into a seven-field tuple and called
signed_step per item, with two memmove calls for the tuple state; a serial
microbenchmark of that fold over ten million items took 400 ms against 20 ms
for the same arithmetic in local variables and 6 ms for sum. The worker now
runs the chunk in a loop over locals, tracking the exact running total as a
wrapping i64 plus a wrap count. Combining collected chunks now appends each
chunk’s storage with extend instead of moving items one at a time.
The first round of measurements found two losses: splitting a list moved each
item on the calling thread with a list append (about 37 ns per item), and
.par reduce collected every item, then ran reduce_tree, which submitted
one job per pair. Three changes followed:
- A list source was split by an internal list operation that copied each chunk’s contiguous rows in one bulk copy. Issue #455 replaced this with borrowed chunks, described in this report.
pool.reduce_treegives one job an aligned run of adjacent inputs, a power of two chosen so there is about one run per worker, and the job reduces it serially through the same rounds. The remaining run results are combined by pair jobs. Because each round maps indexitoi / 2, an aligned run is exactly a subtree of the fixed tree, so the grouping is unchanged; a test compares it with a serial level-by-level evaluation for lengths 0 to 70 and larger odd and even lengths, with 1, 2, 3 and 8 workers..parreducenow groups likesum: each chunk reduces its items left to right in its worker, and the chunk results are reduced in input order. No list of all items is collected. The worker’s loop makes the same direct calls per item as the serial pipeline (plenty_list_scalar_get, the callback andf).
Borrowed chunks (issue #455). A list chunk is now a span of the source’s
rows: the job receives the first row’s address, the row size and the item
count, and reads (or, for owned affine items, takes) items in place. The
calling thread copies nothing; it allocates one small list of chunk bounds.
This relies on every terminal draining its jobs before it returns. --dump asm
of the filter+count worker showed the per-item loop with no calls (an
imul and a load per row, then the inlined predicate); before, it called
plenty_list_scalar_get per item, and the split called the runtime copy.
The first measurement after that change regressed filter+count and the
sums at 100,000 items (0.46 to 1.24 times serial): the owned source was now
dropped on the calling thread, and dropping a list walked and cleared every
row even when the items own nothing (about 3.4 ns per row). A list of items
that own no heap storage now frees its buffer without the walk. That also
makes the serial pipelines, which consume the list, faster.
Results on an AMD Ryzen 7 7840HS, eight workers, with borrowed chunks and the drop change (load average 5 to 6 from other work):
| Case | Callback | Items | Serial | pool | par | par/serial | par/pool |
|---|---|---|---|---|---|---|---|
| map+collect | cheap | 1,000 | 0.041 | 1.082 | 0.042 | 1.03 | 0.04 |
| map+collect | cheap | 100,000 | 5.213 | 88.015 | 3.313 | 0.64 | 0.04 |
| map+collect | cheap | 10,000,000 | 567.259 | 9362.107 | 328.201 | 0.58 | 0.04 |
| map+collect | heavy | 1,000 | 0.340 | 1.142 | 0.129 | 0.38 | 0.11 |
| map+collect | heavy | 100,000 | 34.433 | 97.323 | 9.094 | 0.26 | 0.09 |
| map+collect | heavy | 1,000,000 | 331.989 | 982.120 | 74.592 | 0.22 | 0.08 |
| filter+count | cheap | 1,000 | 0.006 | 1.035 | 0.013 | 2.19 | 0.01 |
| filter+count | cheap | 100,000 | 0.637 | 104.234 | 0.082 | 0.13 | 0.00 |
| filter+count | cheap | 10,000,000 | 84.430 | 9071.530 | 31.855 | 0.38 | 0.00 |
| filter+count | heavy | 1,000 | 0.271 | 0.717 | 0.080 | 0.30 | 0.11 |
| filter+count | heavy | 100,000 | 27.919 | 90.143 | 3.902 | 0.14 | 0.04 |
| filter+count | heavy | 1,000,000 | 274.661 | 883.825 | 34.612 | 0.13 | 0.04 |
| sum i64 | cheap | 1,000 | 0.005 | 0.809 | 0.012 | 2.25 | 0.01 |
| sum i64 | cheap | 100,000 | 0.544 | 89.318 | 0.089 | 0.16 | 0.00 |
| sum i64 | cheap | 10,000,000 | 80.116 | 10037.634 | 36.010 | 0.45 | 0.00 |
| sum i64 | heavy | 1,000 | 0.282 | 0.812 | 0.089 | 0.32 | 0.11 |
| sum i64 | heavy | 100,000 | 28.724 | 108.087 | 6.420 | 0.22 | 0.06 |
| sum i64 | heavy | 1,000,000 | 282.463 | 969.470 | 40.324 | 0.14 | 0.04 |
| sum i64 range | cheap | 1,000 | 0.001 | 0.810 | 0.016 | 23.21 | 0.02 |
| sum i64 range | cheap | 100,000 | 0.068 | 90.465 | 0.079 | 1.16 | 0.00 |
| sum i64 range | cheap | 10,000,000 | 6.585 | 9071.521 | 5.721 | 0.87 | 0.00 |
| sum i64 range | heavy | 1,000 | 0.308 | 0.892 | 0.100 | 0.33 | 0.11 |
| sum i64 range | heavy | 100,000 | 27.919 | 91.461 | 4.882 | 0.17 | 0.05 |
| sum i64 range | heavy | 1,000,000 | 276.006 | 929.201 | 38.852 | 0.14 | 0.04 |
| sum f64 | cheap | 1,000 | 0.005 | 0.836 | 0.010 | 1.82 | 0.01 |
| sum f64 | cheap | 100,000 | 0.547 | 95.449 | 0.046 | 0.08 | 0.00 |
| sum f64 | cheap | 10,000,000 | 75.071 | 9511.754 | 30.247 | 0.40 | 0.00 |
| sum f64 | heavy | 1,000 | 0.103 | 0.827 | 0.034 | 0.33 | 0.04 |
| sum f64 | heavy | 100,000 | 10.583 | 92.945 | 1.673 | 0.16 | 0.02 |
| sum f64 | heavy | 1,000,000 | 104.262 | 919.342 | 13.321 | 0.13 | 0.01 |
| reduce | cheap | 1,000 | 0.005 | 0.850 | 0.011 | 2.09 | 0.01 |
| reduce | cheap | 100,000 | 0.596 | 90.544 | 0.064 | 0.11 | 0.00 |
| reduce | cheap | 10,000,000 | 76.897 | 9112.809 | 29.458 | 0.38 | 0.00 |
| reduce | heavy | 1,000 | 0.270 | 0.772 | 0.076 | 0.28 | 0.10 |
| reduce | heavy | 100,000 | 27.307 | 88.855 | 3.867 | 0.14 | 0.04 |
| reduce | heavy | 1,000,000 | 272.071 | 897.812 | 34.352 | 0.13 | 0.04 |
Before, at main 9e3970d5 with the bulk-copy split, measured just before in
the same conditions (load average 4 to 6):
| Case | Callback | Items | Serial | pool | par | par/serial | par/pool |
|---|---|---|---|---|---|---|---|
| map+collect | cheap | 1,000 | 0.051 | 0.763 | 0.044 | 0.86 | 0.06 |
| map+collect | cheap | 100,000 | 6.016 | 98.522 | 2.838 | 0.47 | 0.03 |
| map+collect | cheap | 10,000,000 | 641.972 | 11471.189 | 540.434 | 0.84 | 0.05 |
| map+collect | heavy | 1,000 | 0.348 | 0.668 | 0.137 | 0.39 | 0.20 |
| map+collect | heavy | 100,000 | 35.873 | 98.538 | 6.967 | 0.19 | 0.07 |
| map+collect | heavy | 1,000,000 | 358.055 | 1089.048 | 71.763 | 0.20 | 0.07 |
| filter+count | cheap | 1,000 | 0.010 | 0.812 | 0.026 | 2.62 | 0.03 |
| filter+count | cheap | 100,000 | 1.115 | 134.447 | 0.738 | 0.66 | 0.01 |
| filter+count | cheap | 10,000,000 | 136.069 | 15679.961 | 262.102 | 1.93 | 0.02 |
| filter+count | heavy | 1,000 | 0.322 | 1.182 | 0.105 | 0.33 | 0.09 |
| filter+count | heavy | 100,000 | 31.387 | 145.978 | 8.457 | 0.27 | 0.06 |
| filter+count | heavy | 1,000,000 | 312.911 | 1371.325 | 63.929 | 0.20 | 0.05 |
| sum i64 | cheap | 1,000 | 0.010 | 0.921 | 0.033 | 3.31 | 0.04 |
| sum i64 | cheap | 100,000 | 0.980 | 132.005 | 0.821 | 0.84 | 0.01 |
| sum i64 | cheap | 10,000,000 | 129.799 | 12216.945 | 248.014 | 1.91 | 0.02 |
| sum i64 | heavy | 1,000 | 0.318 | 0.776 | 0.090 | 0.28 | 0.12 |
| sum i64 | heavy | 100,000 | 29.457 | 114.188 | 6.868 | 0.23 | 0.06 |
| sum i64 | heavy | 1,000,000 | 317.642 | 1362.148 | 69.637 | 0.22 | 0.05 |
| sum i64 range | cheap | 1,000 | 0.002 | 2.140 | 0.054 | 27.75 | 0.03 |
| sum i64 range | cheap | 100,000 | 0.169 | 156.296 | 0.164 | 0.97 | 0.00 |
| sum i64 range | cheap | 10,000,000 | 9.379 | 11315.101 | 5.753 | 0.61 | 0.00 |
| sum i64 range | heavy | 1,000 | 0.265 | 0.810 | 0.083 | 0.31 | 0.10 |
| sum i64 range | heavy | 100,000 | 26.573 | 92.517 | 5.245 | 0.20 | 0.06 |
| sum i64 range | heavy | 1,000,000 | 268.563 | 895.409 | 37.141 | 0.14 | 0.04 |
| sum f64 | cheap | 1,000 | 0.009 | 0.769 | 0.017 | 1.89 | 0.02 |
| sum f64 | cheap | 100,000 | 0.922 | 92.612 | 0.466 | 0.51 | 0.01 |
| sum f64 | cheap | 10,000,000 | 113.042 | 9285.398 | 208.312 | 1.84 | 0.02 |
| sum f64 | heavy | 1,000 | 0.121 | 0.892 | 0.047 | 0.39 | 0.05 |
| sum f64 | heavy | 100,000 | 11.289 | 100.961 | 3.131 | 0.28 | 0.03 |
| sum f64 | heavy | 1,000,000 | 114.782 | 950.894 | 19.875 | 0.17 | 0.02 |
| reduce | cheap | 1,000 | 0.009 | 0.777 | 0.021 | 2.24 | 0.03 |
| reduce | cheap | 100,000 | 1.021 | 105.988 | 0.694 | 0.68 | 0.01 |
| reduce | cheap | 10,000,000 | 119.967 | 9456.671 | 202.308 | 1.69 | 0.02 |
| reduce | heavy | 1,000 | 0.313 | 0.834 | 0.083 | 0.27 | 0.10 |
| reduce | heavy | 100,000 | 28.515 | 91.681 | 4.281 | 0.15 | 0.05 |
| reduce | heavy | 1,000,000 | 279.319 | 911.116 | 39.372 | 0.14 | 0.04 |
Findings.
- With cheap callbacks over ten million list items,
.parnow takes 0.38 to 0.58 times the serial time (1.7 to 1.9 before). At 100,000 items it takes 0.08 to 0.16 times serial forfilter+count, the sums andreduce, and 0.64 formap+collect, which still appends each result. - With expensive callbacks,
.parruns at 0.13 to 0.26 times the serial time from 100,000 items, as before. - At 1,000 cheap items the job overhead still exceeds the work (1.8 to 2.3 times serial). A range source over cheap items is unchanged.
reducekeeps per-chunk grouping;pool.reduce_treekeeps aligned runs.
Remaining work. Early stop for parallel any/all; user-facing scoped
pool jobs (a deferred B22 extension) could reuse the drain-before-return
guarantee.
Warm script startup measurements
Issue #699 investigated fresh-process launches of an unchanged no-op script. The expensive work was extracting the bundled runtime before an executable-cache lookup that did not need it. Deferring that extraction until a miss reduced the measured shebang median from 16.35 ms to 6.56 ms, without bypassing source, toolchain or native-input validation. These are local measurements, not startup guarantees for other machines or toolchains.
Workload and environment
Measured on 2026-11-03: AMD Ryzen 7 7840HS, 16 logical CPUs, x86-64 Fedora Linux
7.2.11-200.fc44, glibc 2.43, GCC 16.2.1 and mold 2.40.4. Both compiler binaries
were Plenty 0.8.8, built with Rust 1.99.1 using
cargo build --locked --release --bin plenty, without profiling features. The
baseline was bd407f0525ab8a4da277de543ba60da3b3001b54; the changed compiler
used the same sources with deferred runtime extraction. The script was:
#!/usr/bin/env -S plenty script
def main() -> Result[(), IoError]:
Ok(())
The benchmark used a temporary working directory, an isolated private cache,
tmpfs temporary outputs, and a fixed environment containing only HOME,
PATH, PLENTY_CACHE_DIR and TMPDIR. PATH contained a benchmark-local
plenty symlink and /usr/bin:/bin. Redirection/injection variables were absent.
Both program-object and executable hits were confirmed before and after timing;
the linker was not rerun in the timed launches. This establishes the cause of
the reported high system time, but does not reconstruct the original fish
session’s compiler build, full environment or exact timing.
Repeated untraced results
Each workload had ten untimed warm-ups and 100 fresh-process samples per
compiler. Workload order was shuffled within each repetition. Wall time includes
Python’s subprocess launch/wait and output collection; child user/system times
come from getrusage(RUSAGE_CHILDREN). Diagnostics, compilation, cache population
and tracing were separate. No fish timing overhead is included.
| Workload | Before median (p10–p90), ms | After median (p10–p90), ms |
|---|---|---|
plenty script hello.plenty | 15.984 (15.237–17.161) | 6.189 (5.686–6.705) |
| Executable env/shebang script | 16.346 (15.597–17.111) | 6.562 (6.097–7.075) |
| Equivalent AOT executable | 0.679 (0.524–0.781) | 0.687 (0.526–0.783) |
plenty --version | 1.152 (0.933–1.319) | 1.118 (0.900–1.285) |
/usr/bin/true | 0.543 (0.440–0.616) | 0.521 (0.425–0.599) |
For direct invocation, median child user/system CPU fell from 2.615/13.114 ms to 2.657/3.324 ms. For shebang invocation it fell from 2.747/13.330 ms to 2.628/3.741 ms. Removing runtime extraction saved about 9.8 ms per launch; shebang/PATH handling was a small part of the original delay.
A reverse-order repeat on the final tree (another 100 samples per workload) measured 17.010 versus 6.650 ms for shebang execution and 16.516 versus 6.179 ms for direct invocation. The improvement persisted when the changed compiler ran first. Raw reports include compiler SHA-256 identities.
Attribution and remaining work in a hit
Separate strace -f -c and strace -f -tt -T runs confirmed the mechanism:
- Before: write a 17,401,052-byte runtime archive into the temporary directory, then remove it after restoring a cached executable. One detailed trace measured 7.78 ms for the write and 2.10 ms for its removal. These traced syscall times are illustrative; the untraced before/after difference above measures the overall saving.
- After: neither open/write nor removal of that archive occurs on a hit.
The object (10,048 bytes) and restored private executable (121,104 bytes)
are still materialized. The native-validation trace retained 447
statxand 222readlinkcalls; it was not replaced with a stale-path shortcut. - Both versions invoke GCC for
-print-prog-name=ld.moldand-print-search-dirs, then launch/wait for the application. Independent untraced probe measurements (100 samples after ten warm-ups) had medians of 0.782 and 0.854 ms. Those probes remain necessary under the current toolchain eligibility contract. - Compiler startup/dynamic loading has the approximately 1.1 ms
--versioncontrol; the AOT child has the approximately 0.7 ms control. The rest of a roughly 6.2 ms direct hit includes runner/header/project setup, private locks, source/native dependency checks, object/executable restoration and cleanup. These independent medians form a rough cost budget, not additive phase instrumentation. Syscall attribution was sufficient to select this change; no compiler-phase profiler was needed.
Cold misses still extract and link the runtime. Wrappers, redirected toolchains, explicit outputs, hooks/fuzz builds and other unsupported executable-cache configurations retain their existing linking behavior. The compiler identity already includes the embedded runtime; deferring its extraction changes no cache key or manifest. Existing shared-library runtime files remain explicit inputs. The regression makes the runtime extraction path a directory: a real warm hit must succeed without writing it, while an edited-object miss must fail extraction and leave no executable to run. Cache invalidation, private executable restore, argument/exit handling and locking retain their existing tests and contracts.
Reproduction
Preserve a baseline release compiler before rebuilding the changed compiler, then run the helper from this checkout on an otherwise quiet machine:
python3 -B scripts/measure_script_startup.py /absolute/before/plenty /absolute/after/plenty \
--runs 100 --warmup 10 --output /tmp/script-startup.json
The helper creates the script, AOT control, isolated cache and fixed environment, checks hits outside timing, prints distributions and writes raw wall/CPU samples to JSON. It refuses to benchmark a toolchain that cannot confirm executable hits. It does not change CPU frequency policy or pin a core. Repeat in reverse compiler order when checking for drift. Do not run compilation or tracing alongside the timed comparison.
For syscall diagnostics, create a separate working directory containing the
same hello.plenty, warm it twice with the exact same fixed environment, then
use strace -f -c or strace -f -tt -T. Keep trace output outside that working
directory, since its identity is a watched linker input. Never report traced
wall time as an ordinary launch sample. For a single fixed compiler, for example:
env -i HOME="$HOME" PATH=/usr/bin:/bin PLENTY_CACHE_DIR=/tmp/script-trace-cache \
/absolute/plenty script --cache-stats hello.plenty
# Repeat to confirm hits, then trace with the same environment:
env -i HOME="$HOME" PATH=/usr/bin:/bin PLENTY_CACHE_DIR=/tmp/script-trace-cache \
strace -f -c -o /tmp/script-syscalls.txt /absolute/plenty script hello.plenty
Measuring module edits
scripts/bench_module_edits.py compares complete builds in fresh compiler
processes. It generates an application with user modules, an edited dependency
and a generic function; by default it also imports std.http.server, using only
a small part of it. It does not change the historical compile_bench template.
cargo build --locked --release --features profile --bin plenty
taskset -c 0,2,4,6,8,10,12,14 python3 scripts/bench_module_edits.py \
/path/to/baseline/plenty target/release/plenty \
--modules 10 --functions 100 --runs 7 --profile --json /tmp/module-edits.json
The compiler paths can name separate worktree builds or saved copies. Choose CPUs available on the measurement machine and pause concurrent builds and tests. Each repetition uses a fresh cache directory per compiler, alternates compiler order, and keeps source paths fixed throughout the comparison.
By default, timings include frontend checking, object emission and process startup, but not linking. The script also compiles and runs each source variant once per compiler to check observable results. Options include:
--release: measure optimized executable builds, including linking, and compare executable bytes. This covers cross-module scalar substitution; the CLI does not expose optimized object-only emission.--import-std '': run a control without the broad standard import.--profile: run separate, untimed profiles after the first repetition. Aprofilebuild records timing rows;profile-allocalso records allocations. Do not mix instrumented reports with ordinary latency samples.
| Case | Work and correctness check |
|---|---|
disabled | Cache disabled, including the whole-program cache. |
populate | First build with an empty cache. The program entry is initially only a marker. |
function_warm | Second build: function cache entries exist; this build stores the program object. |
unchanged | Third build: requires a whole-program hit and the same object bytes as disabled. |
body | Change one ordinary function body; requires a program miss and byte identity with a fresh uncached build. |
interface | Add a parameter and update its caller; requires the same invalidation and identity checks. |
generic | Change a generic body without changing its signature; requires invalidation of its instance and the same identity check. |
Each edit starts from the base program with a current cached base object. The
function cache retains earlier compiled variants, as in normal editing. JSON
records every sample, cache statistics, artifact size and SHA-256, compiler
binary hashes, CPU affinity, and GNU time peak RSS in KiB. Timings use a
monotonic clock.
Interpreting results
Native emission omits unreachable standard definitions only after checking; this is not incremental frontend checking. Applications using most of an imported library may benefit less. See native emission and the separate per-module cache design.
In a 2026 measurement, the emission change improved the broad-import workload but showed no benefit for a control without standard imports. These historical results are workload-specific, not a general performance guarantee.
Module-edit emission measurements
These historical measurements assess issue #400’s unreachable-standard-definition emission pass, not current compiler performance. See the module-edit benchmark for the workload and how to reproduce a comparison, and native emission for the pass’s contract. It runs after checking and retains user definitions, exports, destructors and generators, following explicit calls and callback adapters. Conservative roots may retain functions execution does not need.
Measurements
2026-10-23, AMD Ryzen 7 7840HS, eight physical cores pinned as in the benchmark
example. Baseline 42a4a67c and the reachability change used release compiler
builds with the profile feature inactive during timings. Seven alternating
samples per compiler, with agent builds and tests paused, ten user modules of
100 functions, and the default broad std.http.server import:
| Object build | Before, median ms | After, median ms | Before / after peak RSS, KiB |
|---|---|---|---|
| Cache disabled | 73.93 | 54.73 | 61,228 / 37,360 |
| Empty cache | 77.94 | 54.58 | 62,952 / 38,176 |
| Warm functions | 60.36 | 51.81 | 46,008 / 35,836 |
| Unchanged program | 5.84 | 4.48 | 9,636 / 8,616 |
| One body edit | 60.88 | 50.68 | 45,448 / 35,860 |
| Interface and caller edit | 60.90 | 52.41 | 47,192 / 37,556 |
| Generic body edit | 61.40 | 51.97 | 47,316 / 37,500 |
The base object shrank from 1,368,960 to 408,744 bytes. A separate timing profile counted 1,558 native user-function builds before and 1,004 after; cold backend time fell from 30.7 to 7.9 ms. Lowering still took about 21 ms, which explains why this change does not eliminate the edited-build frontend cost. The isolated pass took about 0.7 ms, including release of discarded operations.
Separate profile-alloc builds of the same workload reported total allocated
bytes of 285.8 → 133.7 MB with caching disabled and 125.1 → 89.2 MB after a body
edit (hotpath’s MB units). These are allocation traffic, not peak RSS. Parsing
and body-lowering allocation remained 21.4 and 19.7 MB respectively; most of the
reduction came from avoiding native construction and compilation. Allocation
instrumentation timings are excluded from the latency table.
Repeating the matrix with --release (optimized executables, linking included)
gave 106.12 → 87.87 ms cold and 95.11 → 82.18 ms after a body edit. Unchanged
linked builds were 36.23 → 35.32 ms: linking dominates this path. The executable
size stayed 3,283,000 bytes because the linker already discarded unused code.
Every measured cached object or executable matched its uncached counterpart;
body, interface and generic-body edits all missed the program cache and produced
the expected changed result.
The same 1,000-function workload with --import-std '' produced identical object
sizes before and after. Its cold median was 19.19 → 19.90 ms and body-edit median
18.93 → 19.02 ms; these short runs establish no benefit for that control. The
pass exits after its candidate scan when there are no standard definitions.
These are workload-specific observations, not a general compilation-speed
guarantee or a substitute for measuring an application that uses most of a
library.
Per-module cache investigation
The 2026-10-24 bounded B19/B17 investigation supports keeping the current function/program caches. A scalar-only separate-compilation spike proved reuse and invalidation at an explicit boundary, but was slower than compiling the same computation normally. It is not a production cache implementation.
Frontend reuse remains promising for imports with substantial bodies, especially the standard library. It requires a real lowering boundary; independently packaging today’s modules as C libraries is not a useful shortcut. This report records evidence and constraints. Scheduling remains in the backlog.
Current compiler measurements
Local main at 00d32a13, AMD Ryzen 7 7840HS, Linux x86-64, Rust 1.99.0,
release compiler with profile compiled in but inactive during timings.
Compiler SHA-256:
61875f0816906d4c5d2577da75bb7f918ad9a26115029bc772d188176d6afc07.
Seven repetitions, physical cores 0,2,4,6,8,10,12,14, other agent builds paused.
The existing module-edit harness generates ten modules with
100 independent functions each, a caller and an entry point. Its broad-import
variant imports std.http.server; the scalar control omits that import.
| Object build | Broad import, median ms | Scalar control, median ms | Broad import peak RSS, KiB |
|---|---|---|---|
| Cache disabled | 45.50 | 17.14 | 35,348 |
| Populate empty cache | 47.73 | 17.66 | 36,264 |
| Warm functions | 44.54 | 15.56 | 33,788 |
| Unchanged program | 3.99 | 4.06 | 8,096 |
| Ordinary body edit | 44.00 | 16.17 | 34,044 |
| Interface and caller edit | 46.79 | 16.35 | 35,804 |
| Generic body edit | 44.67 | 16.66 | 35,588 |
These timings include compiler startup and object writing, but exclude native linking. Every cached artifact matched its uncached counterpart byte for byte, and every source variant executed with its expected output. No-change builds hit the program cache; all three edits missed it. This is a synthetic workload, not a forecast for arbitrary applications.
Separate timing profiles after a body edit recorded:
| Phase | Broad import | Scalar control |
|---|---|---|
| Parse all modules | 8.35 ms / 25 modules | 2.20 ms / 12 modules |
| Resolve names | 3.32 ms | 1.16 ms |
| Lower program | 19.28 ms | 2.66 ms |
| Lower functions, inside program lowering | 10.91 ms / 1,629 functions | 1.42 ms / 1,004 functions |
| Independent operation check | 2.21 ms | 0.49 ms |
Nested profile rows are not additive. The broad-import frontend contains much more reusable work than the scalar control. Avoiding all this work is only an optimistic ceiling: loading, dependency validation, merging and changed-module checking still cost time.
A separate profile-alloc build reported 119.8 MB allocated with caching
disabled and 74.5 MB after a body edit (hotpath’s displayed MB units). Parsing
allocated 10.4 MB, name resolution 4.8 MB and function-body lowering 19.6 MB
after that edit. These are exclusive allocation traffic counters, not retained
memory or RSS. Allocation-instrumented latency is excluded from all timing
tables. This leaves meaningful allocation work for a frontend cache to avoid,
but artifact decoding would introduce allocations of its own.
Five additional broad-import --release repetitions measured optimized
executable builds, including linking: 79.77 ms uncached, 33.77 ms unchanged,
73.49 ms body edit, 71.78 ms interface edit and 77.73 ms generic edit. Cached
executables again matched uncached executables. Reusing frontend artifacts alone
does not eliminate the link floor.
Actual reusable-artifact spike
experiments/module-cache/run.py generates ten scalar modules, each with a
single exported i64 function and 100 independent private functions. A separate
caller uses the first module. It caches existing static-library outputs using
compiler-binary, source, module-name and imported-interface hashes. Integrity
checks reject damaged artifacts. A second mode extracts only each archive’s
plenty_exports.o, leaving the final application to supply the runtime.
This intentionally uses existing C export/import support rather than modifying the compiler. Interfaces are generated from the fixed fixture, not inferred by a new parser. There are no generic imports, classes, borrowed values, strings, closures or cross-module substitution at the artifact boundary. It proves that an ordinary scalar body edit can reuse callers; it does not prove a general Plenty interface is just its written signatures.
Five repetitions per mode, same compiler and CPU affinity. Each repetition has
an empty artifact cache. Existing Plenty caches are disabled to isolate artifact
reuse. The ordinary comparator builds the same scalar computation and independent
functions through normal imports, also with caching disabled. All rows below
include native linking and use default code optimization, unlike --release
above. Source generation and execution checks are outside the timings.
| Build | Full archive cache, ms | Extracted object cache, ms | Ordinary compilation, ms* |
|---|---|---|---|
| Cold | 1,222.61 | 1,236.97 | 48.89 |
| Unchanged | 191.49 | 96.52 | — |
| Ordinary body edit | 279.73 | 197.80 | 55.36 |
| Interface and caller edit | 379.55 | 307.34 | 50.31 |
*Ordinary comparator from the extracted-object run. The independently repeated archive-run comparator was 48.71 / 50.37 / 50.33 ms respectively. These modes were run in separate batches; small differences are not an optimization claim.
An ordinary body edit rebuilt only module m0; its caller and the nine other
modules were hits. A signature edit rebuilt m0 and the caller. Rebuilding all
artifacts afresh produced the same executable bytes as reuse for each edit.
The program produced 42, 43 and 44 in the base, body and interface cases.
Corrupting every cached artifact forced rebuilding and restored correct output.
Full archives occupied 174,083,980 bytes for the base program; extracted objects occupied 335,712 bytes. The former repeats the library runtime for each module. Hashing those archives on hits is part of the measurement. Extraction removes that storage/read cost but still creates archives on misses. The extracted mode’s peak compiler-child RSS was about 55 MiB cold/edited and 32 MiB unchanged, versus about 37 MiB for ordinary compilation. This excludes the Python driver’s RSS and is not a total allocation count. Object and archive modes have different linker retention, so executable byte identity is checked within a mode, not between modes or against ordinary compilation.
Repeated compiler processes, export adapters, archive construction and linking outweigh avoided checking in this small workload. Do not adopt this mechanism as the module cache. The experiment does not estimate the speed of a proper in-process lowered-artifact cache, which avoids these costs.
Required boundary and invalidation
The implementation still lowers a whole resolved program. frontend::lower
creates aliases, signatures, generic instances and pending work; lowering shares
the string heap and records information used by subsequent functions. The module
loader preserves dependency order, import resolution and source identities.
Persisting a function’s operations while ignoring these outputs is insufficient.
The small check_dependencies.py probe changes only &mut pair.x to
&mut pair.y in an imported function. Its declaration stays
select(pair: &mut Pair) -> &mut i64, yet an unchanged caller that mutates
pair.y changes from accepted to a conflicting-borrow diagnostic. Reverting
the body restores acceptance. This directly exercises
frontend::references::returned_fields: inferred projections are interface facts.
| Dependency | What must invalidate or be reconstructed |
|---|---|
| Source/import resolution | Module text, canonical package/module identity, import target changes, compiler identity/settings |
| Public declarations | Signatures, nominal layouts, aliases, visibility, protocols and generated methods |
| Body-derived semantics | Returned-reference projections and any effects/capture facts consumed elsewhere |
| Generic instantiation | Defining body, type arguments, constraints and transitive instantiated dependencies |
| Native substitution | Any body embedded in another function, even if its signature is unchanged |
| Stored operations | Deterministic string/type/instance identities, source locations and ownership diagnostic origins |
A conservative first cache may depend on complete imported source digests; this sacrifices reuse but avoids claiming interface precision prematurely. Precision can follow after an explicit module result can be built and replayed against fresh imported environments. Generic definitions/instances and embedded native bodies need separate dependencies from ordinary declaration hashes. Corrupt/version-mismatched artifacts must miss, and invalidated diagnostics must remain in program order. Existing cache-on/off identity and import-resolution tests remain acceptance gates.
The smallest credible compiler change is therefore to isolate and test a module lowering result without persistence first. Measure replay against normal lowering on an actual std-heavy application before adding serialization. The earlier syntax-tree/copy investigation already argues against inserting a parsed-tree cache instead. This investigation does not authorize a broad redesign or promise a speedup.
Reproduction
Build plenty with cargo build --locked --release --features profile --bin plenty.
Run scripts/bench_module_edits.py with --runs 7 --profile, then repeat with
--import-std ''; use --runs 5 --release for the optimized executable check.
Each command requires a --json PATH. Run the experiment with:
python3 experiments/module-cache/run.py target/release/plenty --runs 5 --json /tmp/archives.json
python3 experiments/module-cache/run.py target/release/plenty --runs 5 --objects --json /tmp/objects.json
python3 experiments/module-cache/check_dependencies.py target/release/plenty
Pin CPUs with taskset as above and avoid concurrent compilation. JSON records
every sample, compiler hash, affinity, outputs, reuse decisions and artifact
sizes. The existing module-edit harness additionally records separate phase
profiles and cache statistics. The spike is single-process orchestration in a
temporary directory; it is not a hardened shared cache or a general-purpose
module parser.
For allocation counts, rebuild with --features profile-alloc and run the
module-edit harness with --runs 1 --profile; keep those results separate from
timing runs. No production Rust code changed, so validation here is the executed
native experiments, edit matrix, Python syntax checks and Markdown link checks.
The Plenty agent skill
The plenty skill teaches coding agents (Claude Code, Codex, Cursor and
others that read the Agent Skills format) to write Plenty. It is for people
writing Plenty programs; the
repository search skill is for work on this
repository. This page covers where the skill lives, how it is published, and
how to trial it.
Layout
| Path | Contents |
|---|---|
.agents/skills/plenty/ | The skill: SKILL.md and hand-written references/ |
.claude/skills/plenty | Symlink to .agents/skills/plenty, so Claude Code loads it here |
scripts/agent_skill_repo/ | Files that exist only in the published repository: plugin manifests and README |
scripts/sync_agent_skill.py | Assembles the published repository |
scripts/skill_trial.py | Runs fresh agents against the published skill |
Agents working in this checkout use the skill directly. In the checkout, its links point at book pages such as the syntax cheat sheet and the signature index, so it never holds a copy that could fall behind the book. Copies are made only when publishing.
tests/test_tutorial.rs runs every executable fence in the skill with the
same fence kinds as the book, so a language change that breaks a skill example
fails the suite. scripts/test_sync_agent_skill.py checks that the skill’s
links resolve and that the published tree is self-contained.
When a skill passage works around a local issue, an HTML comment names it, for
example <!-- issue #575 -->. Remove the workaround when the issue is closed.
The published copy drops these comments.
Keeping the guidance current
When a language or toolchain change lands, review SKILL.md and its handwritten
references alongside the cheat sheet. Correct obsolete advice and remove
workarounds for resolved issues. Keep common error-prevention rules in the
entrypoint; link to reference contracts for specialized features instead of
copying their full specification. The skill’s references/discovery.md guide
routes migration tasks to existing modules and asks for versioned minimal
reproductions before declaring a capability absent. Applied ownership examples
distinguish ordinary copying, explicit class copying and shared content;
package examples teach module visibility and target coverage.
Check both the checkout skill and the assembled tree: a working book link can lose its destination when publishing if that page is not bundled. The language reference is bundled so agents can follow arithmetic, ownership, generic and protocol contracts without this checkout. Run the sync tests and documentation checks; run the executable skill examples when changing code fences. These checks validate examples and packaging; the trial runs separately measure how agents use the guidance.
Publishing
The published repository, plentylang/skills, has one tag per Plenty release,
vX.Y.Z, matching the compiler. To assemble it from a release checkout:
just skill-sync ../skills # or: python3 -B scripts/sync_agent_skill.py ../skills
The script refuses to replace a directory that is not a skills tree, keeps
.git in a clone, and replaces everything else. It:
- copies
.agents/skills/plenty/toskills/plenty/, naming the compiler version fromCargo.toml(--versionoverrides it); - copies the cheat sheet, language reference, package and script pages, every standard library page, and
the builtins, I/O, concurrency and editor-setup pages into
skills/plenty/references/; - rewrites links between copied pages, and reduces links to other pages to their text, until the book has a published URL;
- drops issue notes and contributor-only passages, then recalculates signature-index module line ranges against the published text;
- adds the README and the
.claude-plugin/marketplace and plugin manifests, with the version filled in.
Commit the result in the published repository and tag it vX.Y.Z.
Contributor-only passages
Some book passages address compiler maintainers, such as how to update a vendored crate or regenerate the signature index. Enclose them in markers so the published skill omits them:
<!-- contributors-only -->
To update one, replace its directory with the new release, run
`cargo update -p CRATE --precise VERSION`, and rerun the tests.
<!-- /contributors-only -->
Each marker is a line of its own. The markers are invisible in the book. Unbalanced markers stop the sync.
Trial runs
A trial gives fresh agents only the published skill and the compiler, asks them to write three small programs, and checks the results:
just skill-trial # all tasks, the agent's default model
just skill-trial wordfreq --model sonnet # one task, a chosen model
The recipe builds the release compiler first. Trials need the claude CLI,
signed in, and spend API credit (at most --max-budget-usd per task, default
3). They are not run in CI. python3 -B scripts/skill_trial.py --help lists
every option.
| Task | Asks for | Checks |
|---|---|---|
wordfreq | A CLI counting words in a file, with error and usage exit statuses | Output, exit status 1 naming a missing file, exit status 2 for usage |
inventory | Classes, mutation through &mut self, and an error enum | Exact output |
packages | A library with tests and an executable depending on it, using std.json | plenty test passes; exact output of plenty run |
Each task runs claude -p in a new directory under the system temporary
directory, never inside a checkout, because Claude Code reads CLAUDE.md from
parent directories. The agent gets:
- the published tree as a plugin (
--plugin-dir), as/plugin installwould provide it; plentyonPATH, as a symlink totarget/release/plenty;- no user or project settings,
CLAUDE.md, MCP servers or session history; - the file tools,
BashandSkill, without permission prompts.
Because the agent can run shell commands, it could still read files elsewhere. The prompt tells it not to, and the report lists any tool call that mentions this repository.
Reading the results
Each run writes target/skill-trial/RUN/:
| File | Contents |
|---|---|
report.md | Summary table, then one section per task |
report.json | The same data, for comparing runs |
tasks/NAME/prompt.md | The prompt sent |
tasks/NAME/transcript.jsonl | The agent’s full stream-json transcript |
tasks/NAME/files/ | The sources the agent left |
skill/ | The published tree the agents used |
The summary table records:
- Result: whether every check passed.
- Skill used: whether the agent loaded the skill or read its files. A
nomeans the skill’s description did not trigger. plentyruns: compiler invocations the agent made.- Failed runs: invocations that exited nonzero, including command-line misuse.
- Diagnostics: each compiler error the agent hit.
- Leaks: tool calls that reached outside the task and the skill.
Each task’s section lists failed checks with expected and actual output, every diagnostic, and the failed commands with their last lines of output.
To improve the skill, read the diagnostics and failed commands first. Each one the skill could have prevented points at a missing rule, a misleading example, or a diagnostic worth improving. Then search the transcript for guesses: places where the agent searched the references and did not find an answer. Results vary between runs and models, so compare several runs before changing the skill for a single failure.
Releasing Plenty
A release is one vX.Y.Z tag on main. The maintainer creates and pushes it
with cargo-release; the tag push
starts .github/workflows/release.yml, which builds, tests, and publishes the
GitHub Release. Users install it as described in
installing a release.
Only the maintainer releases, after the changes to ship have been reviewed and
merged under the local PR policy. Agents and PR branches never
run cargo release --execute, create release tags, or push.
Versions
| Package | Version policy |
|---|---|
plenty | The language release version. cargo release bumps it in Cargo.toml and Cargo.lock |
plenty-runtime | Private and embedded in the compiler. Keeps its own version; plenty-runtime/release.toml excludes it, so it is never bumped or tagged |
plenty-dataframe-engine | Separately built workspace; its crate version is internal. Release archives bundle it under the Plenty version on Linux, macOS ARM64 and Windows x86-64, with the ABI generation in the shared-library filename |
plenty-linalg-engine | Separately built faer workspace with the same release-version and ABI-generation policy |
plenty --version reports the plenty package version. Tags are vX.Y.Z, or
vX.Y.Z-PRE for a pre-release; asset names use the same version.
The root Cargo.toml is the source for the current compiler version. The CLI,
LSP server information, source-cache paths, compilation fingerprints, and the
minimum compiler version written by plenty new/plenty init use Cargo’s
CARGO_PKG_VERSION at build time. Release scripts read the same manifest to
name archives and fill the packaged README; packaging rejects a compiler whose
reported version disagrees. The Python engine packager reads [package].version
directly with tomllib, without invoking Bash with a native Windows path;
check-version.sh remains the shell interface for archive/tag checks.
Rebuild executables after bumping the manifest.
Do not globally replace older version strings. The private runtime has an
independent version; new user packages start at 0.1.0; example manifests,
minimum-version requirements, test fixtures, dependency pins, and historical
documentation references describe their own versions. Compiler-output examples
use X.Y.Z placeholders so they do not require edits on every release.
The documentation release catalog is separate from a version bump. Admit a version only after its release is published, preserving earlier archive pins. Development labels identify their source revision automatically; archived labels come from the selected release.
Tools
release.toml was tested with cargo-release 1.1.6:
cargo install cargo-release --version 1.1.6 --locked
The release build itself needs no local tools: it runs on GitHub Actions. To rehearse it, Docker is enough (see checking locally).
Prepare a clean release checkout
cargo-release refuses a dirty tree and commits with git commit -a. The
primary checkout often has a modified issues.db, the shared issue register
described in CLAUDE.md, which must not be reset, stashed, or committed to
make a release. Release from a separate clone of main instead, whose
origin is the primary checkout’s origin:
PRIMARY=/path/to/plenty # the primary checkout
git clone --branch main "$PRIMARY" ~/plenty-release
cd ~/plenty-release
git remote set-url origin "$(git -C "$PRIMARY" remote get-url origin)"
git fetch origin
git status --short --branch # must be clean and level with origin/main
Bring main level with origin/main first, through the usual review flow, so
the release push is a fast-forward. Before releasing, run the full checks on
that commit:
just release-check
This runs every Linux CI step, both optional engines, the Docker reverse-proxy
tests, license notices for every release target, the documentation checks, and
scripts/release/local-check.sh --test. It continues past a failing step,
writes each step’s log to target/local-ci/, and prints a summary.
scripts/local_ci.sh --help lists the steps; just ci STEP... reruns selected
ones. The macOS and Windows release jobs, and the host’s other Linux
architecture, still run only in CI.
Release
Preview, then execute:
cargo release patch # dry run: prints the bump, commit, tag, push
cargo release patch --execute # bump, commit "Release Plenty X.Y.Z", tag vX.Y.Z, push
| Level | Example from 0.3.0 |
|---|---|
patch | 0.3.1 |
minor | 0.4.0 |
major | 1.0.0 |
rc, beta, alpha | 0.3.1-rc.1, then 0.3.1-rc.2 |
release | Drop the pre-release: 0.3.1-rc.2 becomes 0.3.1 |
X.Y.Z | Exactly that version |
The dry run changes nothing.
The configuration rejects branches other than main, never publishes to
crates.io, and pushes the commit and tag together (git push --atomic).
Afterwards, bring the release commit into the primary checkout without touching
issues.db:
cd "$PRIMARY"
git fetch ~/plenty-release main --tags
git merge --ff-only FETCH_HEAD
Then publish the matching coding-agent skill to
plentylang/skills, tagged with the
same version:
git clone git@github.com:plentylang/skills.git ~/plenty-skills # once
cd "$PRIMARY"
just skill-sync ~/plenty-skills
cd ~/plenty-skills
git add -A
git commit -m "Plenty X.Y.Z"
git tag vX.Y.Z
git push --atomic origin main vX.Y.Z
The v0.8.9 glibc verification records downloaded-artifact symbol audits and glibc 2.28 execution evidence for both Linux architectures.
What the workflow does
| Job | Work |
|---|---|
| Verify tag | Requires the tagged commit to be on main and the tag to equal v plus the Cargo.toml version (scripts/release/check-version.sh) |
| Test | The full test suite and release-script tests on native Ubuntu x86-64 and ARM64 runners, with the pinned Rust |
| Build and test | For each architecture in its pinned glibc 2.28 image: installs the pinned Rust, builds both executables with --locked, runs the test suite, audits the compiler, then builds/audits both engines and packages them in the compiler archive |
| Smoke test | For each target and its pinned distribution images: installs only the documented prerequisites, disconnects the network, and exercises all archives, including app-only engine relocation with the compiler/original engine removed (smoke-in-container.sh) |
| Build and test (experimental) | On macos-15 and windows-2025 with the pinned Rust: builds both executables, runs the platform-boundary and compiler/runtime unit tests, audits the executables (check-native.py), and packages the archive (build-native.sh), including the validated dataframe dylib on macOS or DLL on Windows |
| Smoke test (experimental) | On a fresh runner of the same kind, with Rust, Cargo and PLENTY* variables removed from the environment: checks the archive, then type-checks, compiles and runs a program from paths with spaces, before and after moving the installation (smoke-native.py) |
| Publish | Tag pushes only. Creates a draft release with every target’s archives and checksums, then publishes it (publish.sh) |
Native Ubuntu test runners and the x86-64 baseline build install the
SHA-256-verified mold version pinned in scripts/release/toolchain.env, without
requiring root. They check a linked program’s .comment section before running
the suite, so a missing linker fails early rather than silently changing
executable sizes. The ARM glibc 2.28 build and its local rehearsal use the GNU
linker already pinned by the build image: upstream’s ARM mold 3.0.0 binary
imports gettid@GLIBC_2.30 and cannot run in that image. This preserves the
existing glibc floor. Size budgets measured with mold require care when comparing
GNU ld outputs, which can retain additional unused runtime code and panic strings.
The clean distribution smoke tests also exercise system linkers; mold is not a
release-installation dependency.
Builds use the tagged source, not moving main, a host build, or a cache from
another environment. Publication waits for every job to pass. Pre-release tags
(vX.Y.Z-PRE) publish GitHub pre-releases that are not marked latest. Runs for the same tag
are serialized. Only the publish job can write repository contents, through
the workflow token; no crates.io or other credential is used.
scripts/release/target.sh defines each target and scripts/release/matrix.py
turns the list into job matrices:
| Target | Runner | Archives |
|---|---|---|
x86_64-unknown-linux-gnu | ubuntu-24.04 | compiler with dataframe and linalg engines (.tar.gz) |
aarch64-unknown-linux-gnu | ubuntu-24.04-arm | compiler with dataframe and linalg engines (.tar.gz) |
aarch64-apple-darwin (experimental) | macos-15 | compiler with dataframe dylib (.tar.gz), macOS 11 or later |
x86_64-pc-windows-msvc (experimental) | windows-2025 | compiler with dataframe DLL (.zip), Windows 10 or later |
Each target publishes one asset, plenty-VERSION-TRIPLE.EXT, and its checksum.
release_engines in target.sh lists the engines bundled for a target. Linux bundles both ELF engines; Windows bundles the dataframe DLL, built and
validated natively by build-native.sh. macOS bundles the dataframe dylib,
also built and validated natively, with a macOS 11 deployment minimum,
@rpath/libplenty_dataframe_v1.dylib identity, system-only dependencies and
verified ad-hoc signature. Native archive modes distinguish the two compiler
executables (0755) from engine libraries and notices (0644).
Windows packaging audits x86-64 PE headers, Windows/MSVC dependencies and all
facade exports. The native clean smoke tests check a Parquet query, compiler relocation,
--dist application relocation and failure without the engine. Native archives are
not notarized or signed with a distribution identity; macOS executables and
the dataframe dylib carry linker-generated ad-hoc signatures, and the Windows build links the dynamic MSVC runtime that the
documented Build Tools prerequisite installs. archive.py writes those archives
with sorted entries, fixed owners and modes, and SOURCE_DATE_EPOCH timestamps.
The release notes mark them experimental. The ARM harness
records the emulated evidence gathered before native ARM runners.
A Linux archive contains bin/libplenty_NAME_v1.so beside the compiler and
bin/NAME-notices/ with licenses, dependency notices and engine.json
identifying the release/source, target and library checksum. build.sh builds
each engine and package.sh --engine NAME=LIBRARY installs it through
package-engine.py; the archive README covers using and shipping the engines.
package-engine.py NAME rejects an incorrect SONAME, embedded search paths,
unsupported native dependencies/symbol versions, and missing symbols required
by that checkout’s stdlib facade. This catches stale engines with the same
ABI-generation name. The workflow builds all components from the tagged
source; it never substitutes an engine downloaded from another release.
Normal compiler builds remain independent of Polars and faer, and the compiler links neither; programs load an engine only when they reach it. The release engine is built once for reuse when compiling and running many applications. The engine’s native dependency graph supplies its own notices; it is not a copy of the compiler’s dependency inventory.
The release build environment
scripts/release/toolchain.env pins every input that is not in the lockfile:
BUILD_IMAGEandBUILD_IMAGE_AARCH64: architecture-matchedquay.io/pypa/manylinux_2_28_x86_64andmanylinux_2_28_aarch64images by digest. They provide AlmaLinux 8 with glibc 2.28 and gcc-toolset, so C/Rust objects are built against that baseline. ELF symbol auditing and execution in the oldest smoke environment must confirm the result for each target.RUST_VERSION, architecture-specific checksummedrustup-init, and mold.SMOKE_IMAGESandSMOKE_IMAGES_AARCH64: separate pinned inventories. x86 retains its six distributions; ARM uses AlmaLinux 8 and Debian 13. A distribution without a matching ARM image is not silently treated as tested.
Update these deliberately: change the pin, run scripts/release/local-check.sh,
and update the tested distribution list in scripts/release/README.dist.md
and the installation page when it changes. check-elf.sh fails the build if a
change raises the glibc requirement.
Recovering from a failed release
Never move or delete a pushed release tag, and never change a published release.
- A job failed before publishing, for example a flaky test or an unavailable image: rerun the failed jobs. No release exists yet, or only a draft.
- Publishing was interrupted: rerun the publish job.
publish.shfinds the draft, replaces its assets, and publishes it, without creating a second release. - A rerun after publishing: succeeds without changes when both published checksums match, and fails otherwise, including a missing engine checksum.
- The tagged source is wrong: fix it on
mainthrough review and release the next version. Delete an unpublished draft for the bad tag in the GitHub interface if one remains; the tag stays.
Dry runs
Run the workflow manually to test it before tagging. A manual run builds and
smoke-tests the selected targets from the chosen branch and never publishes;
the publish job runs only for tag pushes. It skips the tag checks and reads the
version from Cargo.toml.
gh workflow run release.yml --ref BRANCH -f targets=all # or linux, native
gh run watch
Or use Actions → Release → Run workflow in the GitHub interface. The built archives remain downloadable from the run’s artifacts for 30 days.
Checking locally
scripts/release/local-check.sh runs the build and smoke jobs with Docker,
without tagging, uploading, or publishing. It builds in the pinned image into
target/release-container-ARCH/, separate from host builds, and smoke-tests the
archives in that target’s pinned distributions. By default it selects the host
architecture; PLENTY_RELEASE_TARGET explicitly selects one of the two triples.
The native release build refuses a mismatched host rather than quietly relying
on emulation. For example, on an ARM host:
PLENTY_RELEASE_TARGET=aarch64-unknown-linux-gnu scripts/release/local-check.sh --test
The engine builder accepts --target TRIPLE and --validate; a successful
cross-build alone does not validate engine execution. Options:
| Option | Effect |
|---|---|
--test | Also run the full test suite in the build image |
--skip-smoke | Only build, audit, and package |
scripts/release/smoke-test.sh ARCHIVE runs the archive checks on the host.
bash scripts/release/smoke-engine.sh ARCHIVE dataframe checks the bundled
engine’s files and modes, optional linkage, discovery beside the compiler, a
Parquet query, --dist relocation without the compiler, and the missing runtime
library diagnostic. smoke-engine.sh ARCHIVE linalg runs the same checks with
a linear solve. smoke-in-container.sh runs both after smoke-test.sh.
Containers are labelled org.plenty.release-build and
org.plenty.release-smoke and removed when the check ends.
Licenses in the archive
See release archive licenses for inventories and maintenance requirements.
Linux release glibc verification
Issue #730 asked whether the Linux release baseline was glibc 2.28 or 2.30. The v0.8.9 published archives support glibc 2.28 on both x86-64 and AArch64. The release text’s 2.28 requirement needs no correction. This audit was performed on 2026-11-05; it does not qualify unpublished builds.
Published artifact evidence
The downloaded v0.8.9 archives matched their published SHA-256 files:
| Archive | SHA-256 |
|---|---|
plenty-0.8.9-x86_64-unknown-linux-gnu.tar.gz | 249526b6536b503227e9b9e176398e74a0becddb8eb3db14a3d6e3b53134015a |
plenty-0.8.9-aarch64-unknown-linux-gnu.tar.gz | 370bb7541ab3ebd2026ca0cbeb036814231bf32728296634c5f9153812c4c2da |
For each architecture, scripts/release/check-elf.sh passed for bin/plenty,
bin/plentyc, bin/libplenty_dataframe_v1.so and
bin/libplenty_linalg_v1.so. All eight files have a highest referenced GLIBC
version of 2.28 and GCC symbol version of 4.2.0. The 2.28 reference is
statx, a weak undefined symbol. No file references GLIBC_2.30 or a later
version. The audit also checked ELF architecture, loader, allowed dynamic
libraries, absence of RPATH/RUNPATH and engine SONAMEs. Symbol inspection alone
is not an execution test or proof that earlier glibc versions are supported.
Baseline execution
The published archives were tested with the existing
scripts/release/smoke-in-container.sh harness, using the pinned AlmaLinux 8
images from scripts/release/toolchain.env:
| Architecture | Image manifest digest | Execution |
|---|---|---|
| x86-64 | sha256:8b469a3a78515e8a18ea8fc727e6a3679e1d0c5ba6f58d5b30be3d0d9b86cffe | Native x86-64 container |
| AArch64 | sha256:29fdedeed8a6707558c9e03b5d3e8f09eac5307cdd0811218d7d55a2cc9ae522 | ARM64 container under QEMU user emulation |
Both containers reported AlmaLinux 8.10 and glibc 2.28. The harness installed
only the documented gcc tar gzip prerequisites, disconnected the network,
and ran the extracted archives without Rust or a source checkout. It checked
both compiler version commands, compiled and executed a program using SQLite
and SHA-256, ran a generated project, and repeated the checks after relocating
the installation and installing a second copy. The dataframe and linalg
smokes additionally exercised their native engines, optional linkage and
app-only --dist relocation. All three unmodified smoke scripts passed on
x86-64.
The stock ARM compiler smoke first passed compilation, execution and the
project check, then failed its missing-linker diagnostic assertion. Under
QEMU 10.2.2, invoking absent cc reported
linker driver `cc` failed (exit status: 127) rather than the expected
installation hint. A temporary copy of smoke-test.sh accepted either diagnostic for that
one negative test; the nonzero-exit requirement and all other checks stayed
unchanged. The repeated compiler smoke and both unmodified engine smokes
passed. This adaptation is not a change to the release harness or evidence
that the original ARM diagnostic assertion passed.
These are libc/userspace checks on host Linux 7.2.11-200.fc44.x86_64. They do not qualify the
oldest supported kernel, native ARM concurrency or other ARM page sizes.
They establish the supported 2.28 baseline for these artifacts; they do not
claim that every older version fails or every library operation was exercised.
Current source and generated programs
The source inspected was main commit
fefe4bae (the audit worktree base); the published tag was v0.8.9, locally
resolved to f65fdda439f8af2e15289c441f5aa626ab59c808.
Current build.sh requires glibc 2.28, both release build images are pinned
manylinux_2_28 images, and check-elf.sh rejects release binaries above
2.28. This audit did not rebuild current main and does not substitute for its
release qualification. The Linux build, ELF audit and smoke scripts used here
are unchanged between that tag and the audit base.
The 2.30 reference in the release workflow explanation
concerns the upstream ARM mold 3.0.0 build tool: its binary imports
gettid@GLIBC_2.30. The ARM baseline build uses its image’s GNU linker instead.
Mold is not shipped in the Plenty archive and is not an installation dependency.
This tool requirement does not raise the compiler’s glibc baseline.
Programs link against the compiling machine’s C library and startup objects. A program built on a newer distribution may require a newer glibc even when its compiler runs on 2.28. Build on the oldest supported deployment system, and audit and execute the resulting program there. Programs using dataframe or linalg also need the matching engine; the shipped engines passed this 2.28 audit. Locally rebuilt or substituted engines have their own requirements. See installation requirements and native linking.
Repeating the check
Download both archives and their checksum files for the release being checked.
From the download directory, verify each checksum, extract the archive, then
change to the source checkout and run its audit for each target. Set target
to x86_64-unknown-linux-gnu or aarch64-unknown-linux-gnu, and root to the
absolute path of the extracted archive directory:
export PLENTY_RELEASE_TARGET="$target"
bash scripts/release/check-elf.sh "$root/bin/plenty" "$root/bin/plentyc"
for engine in dataframe linalg; do
library="libplenty_${engine}_v1.so"
bash scripts/release/check-elf.sh --engine "$library" "$root/bin/$library"
done
Run smoke-in-container.sh IMAGE ARCHIVE with each pinned baseline image and
its matching archive. On an x86-64 host with ARM binfmt/QEMU already configured,
set DOCKER_DEFAULT_PLATFORM=linux/arm64 for the ARM invocation. Keep the
checksum file beside the archive. Record the release identity, hashes,
image digests, audit output and smoke exit statuses; do not infer artifact
compatibility solely from the source configuration.
Release archive licenses
Plenty is licensed MIT OR Apache-2.0 (license in both manifests). The
archive ships LICENSE-MIT and LICENSE-APACHE from the repository root and
THIRD-PARTY-NOTICES.md, generated by scripts/release/third_party_notices.py
from cargo metadata. It lists every crate linked into the compiler or its
embedded runtime (normal dependencies only; build scripts and procedural macros
do not ship) with each crate’s own license files, plus the vendored SQLite
amalgamation and Brotli codec, and the third-party code and data that standard
library modules and the runtime embed. VENDORED_SOURCES lists source copied
from a published package with its license files kept beside the copy:
plenty-runtime/src/unicode/vendor holds unicode-normalization 0.1.25
(MIT OR Apache-2.0). EMBEDDED lists the rest: std.markdown derives from
commonmark.js 0.31.2 (BSD-2-Clause) and carries tables generated from the
Unicode Character Database 16.0.0 and the WHATWG HTML named character
references (BSD-3-Clause for portions in source code), and the
unicode-normalization tables derive from the Unicode Character Database 17.0.0
(Unicode License v3). std.diff derives its Myers and patience
algorithms from the similar crate 3.2.1 (Apache-2.0). Their reviewed license texts are bundled in
scripts/release/licenses/; update the entry and its text together when the
upstream version changes. The compiler notices fail when a directory named
vendor, or a subdirectory of plenty-runtime/native, has no notice: a
child of a vendor directory named after a shipped crate (NAME-VERSION, as in
plenty-runtime/vendor) is covered by that crate’s entry, and any other vendored
directory needs a VENDORED_SOURCES or native-library entry. The Rust standard library’s notices come from the build
toolchain’s share/doc/rust. The script fails when a shipped crate has no
license file or reviewed fallback, so a new dependency’s license is reviewed
before it ships. For iced-x86 1.21.0, whose published crate omits its license
file, the generator uses a bundled copy from the upstream commit recorded in
the crate’s .cargo_vcs_info.json. The fallback matches the package name,
version, and declared license and includes the source URL in the notices.
yaxpeax-arm 0.5.0, the A64 assembly decoder, declares 0BSD but omits a
license file; its pinned fallback supplies the standard SPDX 0BSD terms without
inventing a copyright notice. Both fallbacks identify their package version and
source URL. Dependency upgrades require a fresh review if they still omit the license file;
notice generation does not download fallback text during the release build.
For each engine’s NAME-notices/, third_party_notices.py --component NAME
walks its own manifest/lockfile. The faer fork and its backend are Git dependencies
pinned in the linalg lockfile, with their upstream notices; missing workspace-root
licenses in the dependency family have version-pinned, reviewed fallbacks.
Offline builds need the pinned Git checkout as well as Cargo’s registry cache.
For dataframe, alloc-stdlib shares its repository’s
BSD license with alloc-no-stdlib. polars-parquet-format 0.1.0 declares
MIT AND Apache-2.0 but omits top-level license files; a version/license-pinned
fallback preserves its upstream parquet-format-safe license texts and the
published package’s author attribution. Its Thrift sources retain Apache
Software Foundation license headers. The bundled fallback identifies its
upstream revision, and dependency upgrades require review again.