Flat class layout

Status: Stage 1 (one-field classes as newtypes), stage 2 (flat nested class records), stage 3 (fewer runtime walks) and stage 4 (flat tuple and variant payloads) implemented. Issue #547 asked why a class value is an address into its own storage, and whether classes could instead be laid out flat inside their owner. Values and memory own the shipped contract; this page records the findings and the design.

Why a class value holds an address

Every runtime slot is 128 bits: a payload word and a tag word. A class keeps its fields in inline storage that follows its slot in the owner (a local, a record field, a list element, a box), and the payload word is the address of that storage. The address exists because the payload word has to be something uniform:

• Runtime operations (copy_payload, release, retain, hashing, equality, formatting) receive one u128 and a type descriptor and walk the record through the descriptor. They never receive the slot address.
• References are (address, shift) pairs. &self.field is computed from the address of the record, which is read out of the class value word; a reference to a class value therefore projects fields with one load and an add, not by knowing where the value’s slot lives.
• Sum payloads are packed into the same 128 bits as their tag; a variant holding a class holds its address word and leaves the tags free.
• A zero payload word means “empty slot” for every type, which lets moved-out locals, fields, and partially initialized records release for free.

The cost is that every move of a class value is a copy of its inline bytes plus a fix of each nested class, range, closure, or inline-sum address (inline.rs::fix_record, the runtime’s Payload::relocate); issue #544 measured that as the remaining overhead of nested wrappers.

What depended on the address before stage 2

Stage 1: one-field classes are newtypes

Rule. A class with exactly one field and no __del__ is represented exactly as that field: same 128-bit slot, same inline bytes, same descriptor layout. Constructing it, reading or borrowing its field, and moving the field out are all identity operations in the generated code. No annotation; the rule is total.

Classes with __del__ are excluded because their record carries a state word (initialization completed, observer count) that has no home in the field’s own slot. Field-less classes already lower to a zero word and keep doing so.

Compiler

• ClassType::newtype_field (src/record.rs) names the rule once; Ty::newtype_field and Ty::newtype_base expose it.
• Layout queries delegate to the field: has_inline_storage, flat_record, inline_bytes, and the generator frame measurement.
• lower_class: New is a zero word (an empty slot, which the field’s __init__ write fills); ArmDrop is a no-op; Field and FieldRef return their operand unchanged. The final unpack to the field’s native type is the only code emitted.
• retain, release, and copy_inline_to unpack to the field type and recurse. fix_slot, fix_variant, slot_fixes, plain_variants and release_fields look through newtypes, so a record or sum holding a newtype copies and releases as if it held the field directly.
• Columnar-list row access reads a one-field row as its scalar.
• A newtype has no inline storage, so a function returning one built by a constructor call ends in a tail call. The constructor inlining from issue #548 now also lowers at that site: the caller’s locals are released first, as a real tail call would, and the shared exit then finds empty slots.

Runtime descriptor

A newtype’s descriptor carries the field’s layout (kind, affinity, inline bytes, links, variants, destructor hook) so every runtime walk treats the value as the field. The class name and a new wrapper link (byte 64; the descriptor grows from 64 to 72 bytes) record the source-level shape for formatting, which prints Wrap(x=1), and for the columnar-list schema.

FFI layout contract

C code never sees a class layout: a class crosses the boundary as an opaque handle (a Box holding one slot plus the descriptor’s inline bytes). The rule changes only what that slot holds for one-field classes, and every handle operation goes through the descriptor, so exported libraries, foreign pointers, and dynamic exports keep working without a header change. The contract FFI relies on is therefore:

1. A handle points at a Box whose value slot is at offset 32.
2. A borrowed class argument is the address of that slot.
3. An owned class argument or result is copied in or out with the class descriptor’s copy_payload.

These three hold for every class, newtype or not.

Costs and limits

• Type grows by 8 bytes, so every program’s read-only data changes; machine code for programs without one-field classes is unchanged.
• A one-field class with __del__ keeps the record layout.
• Formatting a newtype walks one extra descriptor link.

Measurements

Retired instructions (perf stat -e instructions:u) against main at c78365ad, which includes constructor inlining, compiled with --release:

The remaining per-cycle cost is caller-site bookkeeping around releases of slots the compiler could prove empty, not record copies. Programs without a one-field class produce identical machine code; only read-only data differs.

Stage 2: flat nested records

Goal. Store a class-typed field at a fixed offset inside the outermost owner without a word of its own, so a move is a byte copy with no address fixes.

Findings. After __init__, a nested class field’s word is redundant: it is always slot + 16 or zero (moved out). Dropping the word (“rule B”) saves 16 bytes and one load, test and store per nested class per move. It needs:

• a presence marker for nested classes with __del__ (their state word can serve; FieldMove must clear it) and for partially moved records;
• record_field, fix_*, release_record, and the runtime Record::slots walkers to compute nested addresses from the owner instead of loading them;
• references to nested fields to stay (address, shift) pairs, which already works because FieldRef adds offsets to an address.

What it does not remove: the top-level word stays an address, because runtime operations and sum payloads still receive one u128. Flattening below the top level removes fixes but not copies; a move still copies the inline bytes.

First assessment. Stage 2 was first deferred: after stage 1 the remaining per-move cost of a nested record looked bounded by RECORD_FIX_LIMIT. Measuring it showed otherwise: in a loop that moves a three-level record through a function, the slot fixes were about 90 of the 146 instructions the callee compiled per call (6 fixes of 5 instructions per record copy, 3 copies), and the nested words added 96 of the 272 bytes copied per record.

Decision. Adopted. A field of a class record whose type is a class with a record is stored flat. The cases that keep a word followed by a record are:

• top-level class values, whose word is the uniform 128-bit value runtime operations and sum payloads receive;
• class payloads of enum variants and tuple components: only class records flatten their fields, so variant payload packing and tuple layout are unchanged (stage 4 later flattened tuples and multi-field variants);
• field-less classes without __del__, which have no bytes to place.

Contract

• Presence. An all-zero flat record is an empty field. Moving a field out or releasing it zeroes its bytes; a class with __del__ has a nonzero state word once constructed, and every other record with a record has at least one nonzero word, so empty and live never coincide.
• References. A reference to a class value names the 16 bytes before its record. Readers use address + 16 without loading the word there; writers copy the record bytes, and writing an empty value zeroes them. Top-level values keep a real word at that address, flat fields do not, and no reader depends on which. Sort-key adapters, which receive runtime slots holding a row’s record address, convert to that form before calling compiled code.
• Destructors. The runtime passes record - 16 to the __del__ hook, which must not read the word.
• Newtype construction. A one-field class wrapping a class with a record is filled through a reference to the record, so New allocates the record’s storage as the wrapped class would.

Compiler and runtime

• Ty::flat_field names the rule; Ty::field_bytes and ClassType::field_offset give a field’s width and offset.
• lower_class projects flat fields by offset (class_field, referenced_record); read_typed_reference and write_record implement the reference contract.
• fix_record, release_record and record_fixes (src/codegen/inline.rs) walk flat fields in place and emit no fix for them.
• The runtime Field { at, flat } iterator (Record::slots) reads, writes and clears a field by kind. Copy, relocate, release, hashing, equality, ordering and formatting go through it. The resumable release walk clears a finished flat record, and Field::is_clear lets a resumed pass skip it. abandon_copy releases the copied prefix without allocating.

FFI

No change. A handle is still a Box holding one top-level slot, whose word addresses the record; borrowed and owned arguments use the same descriptor paths. Generated headers and .plentyi interfaces are byte-identical for a library exporting nested classes.

Measurements

Retired instructions (perf stat -e instructions:u, --release, mean of 5, pinned to 8 cores). Base is main at 5e6e98a4, before stage 2; main is 0ec41c06, which adds the empty-slot release removal (issue #549).

Code size for the three-level workload fell from 130,321 to 121,521 bytes of .text. compile_bench (aot, 1000 × 10, pinned to 8 cores) retires 0.4% more instructions (12.28 G to 12.33 G for 3 repetitions); its median times (244–254 ms on main, 243–260 ms with stage 2) are within noise.

The first version of stage 2 made the shelves workload 7% slower than base. The runtime release walk restarts at an owner’s first field after each heap child, and with flat fields each restart re-entered nested records and scanned finished ones. Two runtime changes removed that cost:

• The walk frees a string in place instead of queueing it and restarting; a string owns nothing and runs no user code, so the order is unobservable. A Shelf with three strings is now walked once instead of four times.
• Record traversal decides a field’s kind once per field, and walks into a flat record build its Record directly instead of re-deriving it through Type::record.

Per shelf (callgrind, 20k shelves), the whole program fell from 9,716 instructions on base to 8,658; the release walk itself fell from 1,760 to 1,210, and relocation is level with base (685 and 663). Compiled moves no longer fix flat fields but still copy their bytes, so the per-move saving is the fixes, not the copy.

Stage 3: fewer runtime walks (issue #550)

Stage 2 left flat records copied by memmove and released and relocated by the runtime’s generic field walk. Profiles of the stage 2 workloads (callgrind, 20k rows of three-level Path records) put the release walk at 27% of the program, list appends’ copy_payload relocation at 16%, and the memmove calls for 176-byte record copies at 9%. Each change below targets one of those shares.

• Inline copies. Copies and clears of 33 to 192 bytes are 16-byte vector loads and stores instead of memmove/memset calls. All loads precede the stores, so overlapping copies stay correct. Larger records, such as the 272-byte Shelf, still call memmove.
• In-place moves. A local moved into a call argument, with only operand pushes between the move and the call, is passed at its own address: the callee copies arguments on entry, so the snapshot copy was redundant. Inlined constructors, generator producers and portable SIMD operations keep argument addresses and still receive a copy (inline::Moves).
• Descriptor plans. A class descriptor with a record grows from 72 to 80 bytes; byte 72 links a Plan with the word offsets of the strings the record owns (through flat fields), whether they are all it owns, and whether its bytes contain addresses into themselves. The runtime frees such a record’s strings in one pass instead of walking its fields, skips relocation for a record without internal addresses, and skips per-row release for a list of records that own nothing. Records with __del__ at any depth, or any other managed field, keep the walk, so destructor order is unchanged.

The plan is data rather than a compiled hook. Descriptors are defined lazily while relocations are resolved, after the parallel function build; a compiled release function would need every eligible class declared before that build. For records owning only strings the data plan reaches the same result: no per-field dispatch, one load and release per string.

Measurements

Retired instructions (perf stat -e instructions:u, --release, mean of 5, pinned to 8 cores). Main is 240cdd71.

Wall-clock medians (15 runs): the three-level workload 254 to 140 ms, Path rows 187 to 116 ms, moves 68.7 to 27.5 ms, shelves 246 to 222 ms, copies 313 to 291 ms. Per Path row (callgrind, 20k rows) the program fell from 5,819 to 3,016 instructions: destroying the list, string frees included, from 1,611 to 110, and the append’s copy_payload from 920 to 204. Per shelf, destroying the list fell from 1,252 to 700 instructions, because its Label fields own only strings.

Inline copies cost code size: .text grew 0.2% to 1.7% across examples (classes 81,121 to 81,313 bytes, http_server_bench 556,705 to 566,257, semantic_search 1,768,878 to 1,773,166, actor_echo 252,641 to 256,193) and 1.3% to 3.2% for the record workloads. A 256-byte limit sped up none of the workloads and grew actor_echo by a further 2.5%; a 128-byte limit cost the moves workload 8%. compile_bench (aot, 1000 × 10) retires 0.18% more instructions (80.53 G to 80.68 G); its median times (332–340 ms on main, 331–336 ms with stage 3) are within noise.

Remaining

Shelves keep part of their walk cost: a Shelf holds ranges and sums, whose payload addresses depend on the active variant, so it is neither string-only nor free of internal addresses. Per shelf, relocation still costs about 930 instructions and the 272-byte record copy still calls memcpy. A plan entry per internal address, tested against the sum’s tag, would cover relocation.

Stage 4: flat tuple and variant payloads (issue #551)

Stages 2 and 3 flattened class fields of class records only. A tuple or enum variant with a class component still stored that component as a slot word followed by its record, so every move rewrote the word: copying a (Point, Point) through a function copied 96 bytes and then ran two address fixes, and a three-field variant carried one fix per class field.

Decision. A tuple component or a field of a variant with two or more fields is laid out as a class field is: a class with a record is stored flat at a fixed offset, with no word. Matching, unpacking, &mut t[0], field moves and constructors use the offsets class records use, and an all-zero flat record marks a moved-out component. A tuple or variant whose components are all flat records without internal addresses copies as plain bytes, so (Point, Point) is a 64-byte copy.

Kept: single-field variants. The payload of a one-field variant, such as Some(point), Ok(point) or Err(point), keeps the slot word followed by the record. Its 128-bit value is the variant’s payload, and sum tags stack in its high word, so the payload must be a full value as top-level values are; dropping the word would change the Option/Result contract that runtime operations, generic code and C headers use. Moving such a sum costs one address fix, the same as moving a bare class value. A workload of 2M Option[Segment] and Result[Segment, Problem] returns, matches and ? propagations retired the same instructions under callgrind before and after this stage (2,041,257,371 and 2,041,261,463), so flattening there has no measured gain to set against the contract change.

Runtime. Record::slots marks a variant or tuple field flat by the same flat_field rule as a class field, so release, retain, copy, relocate, hashing, equality, ordering and formatting walk tuples and variants without change. ranges::flat_record treats a multi-field tuple of flat records without internal addresses as plain bytes, as the compiler does.

Measurements

Retired instructions (perf stat -e instructions:u, --release, mean of 5, pinned to 8 cores). Main is d2078d82.

The Option workload’s 0.1% difference is not in its code: its assembly is unchanged and callgrind counts match. Compiled per call, the tuple keep fell from 103 to 53 instructions and swap from 296 to 224; the variant keep from 218 to 155 and its consuming match from 687 to 542.