Part I: From Source Text to a Compiler Mental Model#

This part builds a map of rustc before asking you to explore its machinery. It assumes basic Rust, but no compiler theory, systems background, or LLVM knowledge. The conceptual target is Rust 1.97.1; internal names and arrangements remain version-sensitive. Always compare this text with the source revision you are actually changing.

1. A Compiler, Built Up from First Principles#

A computer processor does not directly understand fn, ownership, traits, or iterators. It executes instructions encoded for a particular instruction set and operating environment. A human, meanwhile, benefits from names, modules, types, functions, and helpful restrictions. A compiler bridges those two worlds by translating one representation into another.

The source language is the language accepted as input: here, the Rust language. The implementation language is the language used to write the compiler: rustc is written mostly in Rust. The target machine is the abstract platform whose code and conventions rustc must produce. That platform combines facts such as CPU architecture, operating system, object format, and ABI.

An ABI, or application binary interface, governs low-level cooperation between separately compiled code. It includes calling conventions, symbol representation, data layout, and other binary agreements. The target is not necessarily the computer currently running the compiler. Compiling ARM code on an x86-64 laptop is ordinary cross-compilation.

Translation is not merely replacing each Rust word with one machine instruction. One source construct can become many operations, no operation, or target-specific library calls. Conversely, one machine instruction can embody work represented by several source operations. The compiler must preserve the program's defined meaning while changing its form radically.

Consider a compilable Rust function:

fn twice(x: i32) -> i32 {
	x + x
}

fn main() {
	println!("{}", twice(21));
}

The braces and type spelling matter to parsing and diagnostics. They need not survive as bytes in the final executable. The addition matters semantically, although an optimizer might replace it with another equivalent operation. The name twice may remain only in debug information or a symbol table.

Invariant: for every behavior the Rust specification defines, translation must preserve that behavior. Undefined or unspecified areas require more careful wording; they are not permission for arbitrary compiler bugs.

Prediction checkpoint: if comments change but token meanings do not, which later representations should change? Usually machine behavior should not change, though spans, diagnostics, debug data, and incremental fingerprints might.

Counterfactual: a source-to-source formatter also translates Rust, but it is not rustc's whole job. Its output remains Rust text; rustc must validate semantics and eventually support executable artifacts.

2. Compilation Is Constrained Translation, Not Magic#

It is tempting to imagine a compiler as a black box containing cleverness. That image makes unfamiliar failures feel impossible to locate. A more durable model is evidence-preserving translation through several specialized representations.

Source spelling supplies evidence about syntax and programmer intent. Resolved names supply evidence about which declaration each use denotes. Types supply evidence that operations are meaningful for their operands. Borrow checking supplies evidence that references obey Rust's aliasing and lifetime rules. Code generation must consume this evidence without silently violating its conclusions.

The compiler is also a staged proof in an engineering, not formal-theorem-prover, sense. Each successful analysis establishes conditions that later code is entitled to assume. The parser establishes structural well-formedness sufficiently to continue. Type checking establishes typed relationships, while borrow checking establishes memory-safety obligations.

This proof metaphor has limits and should not be overstated. rustc contains recovery paths, delayed diagnostics, unsafe implementation code, and backend assumptions. Some properties belong to the language specification rather than explicit proof objects in memory. Nevertheless, asking “what evidence justified this assumption?” is an excellent debugging technique.

A stage can preserve information, normalize equivalent forms, add derived facts, or discard facts. Preserve means retain evidence needed later, often including a source span. Normalize means choose a more uniform representation for several surface spellings. Add means compute facts such as a resolved definition or inferred type. Discard means deliberately stop carrying details that no downstream consumer needs.

Discarding too early causes poor diagnostics or makes an analysis impossible. Preserving everything forever consumes memory and complicates every representation. Compiler architecture is therefore partly the art of choosing information boundaries.

Bug-location question: did the compiler misunderstand evidence, lose it, or mistranslate it? That question often narrows a report before you know any relevant source file.

3. Competing Definitions of “A Good Compiler”#

Correctness means accepted programs behave according to Rust's defined semantics. A wrong-code bug violates correctness when compilation succeeds but execution is wrong. A compiler crash is also serious, but differs diagnostically from silently generated wrong code.

Soundness is narrower and especially important for Rust's safety promise. Informally, safe Rust must not enable undefined behavior without an unsafe boundary being responsible. A type-system or borrow-checker hole can therefore be a soundness bug even in obscure code. Rejecting a valid safe program is frustrating, but it is generally not itself unsoundness.

Diagnostics include errors, warnings, labels, notes, suggestions, and their source locations. A diagnostic must be accurate before it is friendly. An attractive machine-applicable suggestion that changes meaning is worse than a plain correct message. Recovery should reveal useful independent errors without flooding the user with consequences of one mistake.

Compile time measures latency and resource use while building. More analysis may improve errors or generated code while making compilation slower. Incremental compilation can reuse unaffected work, but dependency tracking and caches have costs. Memory use matters independently because large crates and parallel code generation can pressure machines.

Runtime performance concerns the output program's speed, size, and resource behavior. Aggressive optimization can increase compile time and occasionally worsen debugging experience or code size. Monomorphization enables specialization for concrete generic arguments but may duplicate machine code.

Compatibility asks existing Rust code to keep compiling and behaving as promised. Language evolution, platform changes, diagnostics, and implementation cleanup all interact with compatibility. Stable behavior cannot casually be “fixed” merely because a new design looks cleaner. Edition mechanisms help evolve syntax while making migration explicit.

These objectives compete rather than forming one score. A borrow-check improvement might accept more valid programs but complicate diagnostics. A new optimization may speed benchmarks while increasing binary size and exposing backend bugs. A compatibility workaround may add enduring complexity to a seemingly unrelated analysis.

Invariant: optimization may change implementation strategy, never defined observable meaning. Counterfactual: if compile speed were the only objective, rustc could skip safety checks. That would produce output sooner, but it would cease to implement Rust's central contract.

When reviewing a change, ask which objective improves, which might regress, and how each is measured. That habit turns architecture from folklore into explicit tradeoff reasoning.

4. rustc and the Tools Around It#

rustc is the Rust compiler executable and the compiler code behind that executable. It accepts one crate root per invocation, options, and references to dependencies. It can emit metadata, libraries, object code, assembly, LLVM IR, or linked products.

Cargo is primarily a package manager and build orchestrator. It reads manifests, resolves packages, builds a dependency graph, runs build scripts, and invokes rustc repeatedly. Cargo does not replace rustc's parser, type checker, or borrow checker. Use cargo build --verbose to observe the rustc commands Cargo constructs.

rustup installs and selects toolchains, targets, and components. It can choose stable, beta, nightly, or pinned toolchains per command or directory. It does not compile your crate merely by selecting a compiler.

The standard library is Rust code and runtime support distributed with toolchains. Its APIs are not hard-coded synonyms for compiler internals, although some features need compiler cooperation. Different targets may provide different standard-library availability. core, alloc, and std occupy different layers and capabilities.

A linker combines object files and libraries, resolves symbols, and lays out a final product. rustc commonly drives a system linker or another selected linker near compilation's end. A missing native symbol can therefore be a linking problem, not a name-resolution problem. Rust source names may have resolved perfectly before binary symbols fail to resolve.

Procedural macros are separately compiled Rust libraries built for the host and loaded by rustc. rustc invokes them through a defined token-stream interface. Their execution is not ordinary parsing: it is compiled macro code running with the compiler process's privileges. A proc macro can panic or emit malformed tokens, creating a distinct diagnostic boundary.

LLVM is the principal code-generation backend used by standard rustc distributions. rustc lowers Rust-oriented MIR into backend IR; LLVM optimizes and emits target machine code. LLVM does not understand Rust ownership in the source-language sense. rustc must encode valid low-level operations and relevant attributes before handing work over.

Prediction checkpoint: Cargo reports that a build script failed before any rustc command. The first investigation belongs in orchestration or the script, not MIR borrow checking.

5. Hosts, Targets, and the Meaning of “Runs Here”#

The host is the platform on which the compiler process runs. The target is the platform for which generated code is intended. Native compilation uses matching host and target, which can hide the distinction.

Imagine rustc running on x86-64 Linux and producing AArch64 Linux code. The parser and type checker execute as x86-64 host programs. The resulting executable contains AArch64 target instructions and cannot run natively on that host. Target data layout affects pointer sizes, alignment, ABI choices, and available features.

Build scripts and procedural macros complicate cross-compilation because they execute during the build. They normally must be compiled for and run on the host. The final application and target dependencies are compiled for the target. Confusing these worlds produces errors that look mysterious until the host/target axis is explicit.

host x86_64-unknown-linux-gnu
    |
    +-- runs Cargo, rustc, build scripts, and host-built proc-macro libraries
    |
    +-- rustc reads target specification for aarch64-unknown-linux-gnu
            |
            +-- emits AArch64 objects and target metadata
                    |
                    +-- target linker creates AArch64 product

A target triple is a conventional string summarizing architecture and environment. Its components are historical conventions, not a complete formal platform description. rustc's target specification carries richer details.

Invariant: code executed during compilation must be executable by its execution host. Counterfactual: if every dependency were blindly built for the final target, cross-built proc macros could not run.

6. Crates Are Compilation Units#

A crate is Rust's unit of compilation and linking-level metadata exchange. It may produce a binary, Rust library, static library, dynamic library, or other crate type. A package is Cargo's manifest-level unit and may contain several crate targets. A module is a namespace and organization construct inside a crate, not a separate rustc invocation.

The official rustc book emphasizes passing the crate root, not every module file. mod declarations tell the compiler how source files join the crate's module structure. This differs from build models where every source file is independently compiled.

// Conceptual contents of src/main.rs; this example expects a sibling tools.rs.
mod tools;

fn main() {
	tools::greet();
}

Cargo invokes rustc separately for dependency crates in an order permitted by the crate graph. Downstream compilation reads exported metadata rather than reparsing every dependency's source. This boundary supports separate compilation and constrains which facts must be serialized.

Crate identity matters because two similarly named declarations in different crates are distinct. Features, versions, and dependency renaming can make source spelling alone insufficient for identity. Compiler identifiers therefore carry crate-relative or globally contextual information.

Prediction checkpoint: editing a private helper in one dependency may avoid rebuilding unrelated crates. Whether downstream crates rebuild depends on fingerprints and externally relevant metadata, not textual proximity.

7. The End-to-End Map#

Here is the complete orientation map promised for this part. Arrows show dominant information flow, not a rigid schedule of independent passes. Queries and whole-crate obligations introduce demand-driven edges and forced work.

Rust source files + command-line options + dependency metadata
                            |
                            v
              driver / interface / Session
                            |
                            v
                  lexing: text -> tokens
                            |
                            v
                   parsing: tokens -> AST
                            |
              +-------------+-------------+
              |                           |
              v                           v
       macro expansion <----------> name resolution
              |             repeated cooperation
              v
      validation / resolved, expanded AST
                            |
                            v
               AST lowering: AST -> HIR
                            |
                            v
         type inference / type checking / trait solving
                            |
                            v
              typed body view: HIR -> THIR
                            |
              +-------------+-------------+
              |                           |
              v                           v
       pattern analysis               MIR building
                                          |
                                          v
                           MIR borrow checking + analyses
                                          |
                                          v
                           MIR transforms / optimization
                                          |
                                          v
                         monomorphization collection
                                          |
                                          v
                         codegen units / backend lowering
                                          |
                                          v
                           LLVM IR -> object machine code
                                          |
                dependency objects -------+------ native libraries
                                          |
                                          v
                                        linker
                                          |
                                          v
                              executable or library artifact

Demand model layered across the middle and later map:
  a requested query -> dependencies requested -> cached/in-memory result when valid
  forced correctness work -> checks bodies even when no machine code needs them
  not every result is persisted; not every early phase is a tiny per-item query

Do not read the picture as thirteen isolated programs that each finish permanently. It names conceptual transformations and analyses useful for reasoning. The custom rustc query engine drives substantial later work by demand and dependency. rustc does not use Salsa as its compiler query engine.

Lexing, parsing, macro expansion, and name resolution have broad crate-level interactions. They are not all tiny per-item queries analogous to asking for one function's optimized MIR. Later work is more query-shaped, but some checks are deliberately forced across bodies. Some query values are cached only in memory; selected incremental information can persist to disk.

Invariant: demand-driven scheduling changes when work occurs, not which invalid programs must be rejected.

8. Invocation, Driver, Interface, and Session#

Compilation starts with arguments and inputs, not with the lexer alone. Options select crate type, edition, target, optimization, diagnostics, emitted artifacts, and unstable behavior. Environment and dependency paths can also affect the compilation world.

rustc_driver provides high-level compiler driving machinery. rustc_interface exposes configuration and coordinates compiler execution for internal tool integrations. rustc_session contains session-oriented configuration and diagnostic state. A Session represents facts and services associated with one compilation session.

“Driver” means the code arranging major compiler activity and backend completion. It should not be confused with an operating-system device driver. Callbacks and interface APIs are internal integration surfaces, not stable embedding guarantees.

Invocation bugs often reproduce only with exact flags. Record the full verbose command, toolchain version, host, target, and environment-sensitive inputs. A report that says only “Cargo failed” may erase the distinction needed to reproduce rustc.

rustc --version --verbose
cargo build --verbose
rustc --print target-list

The first command identifies version and host details. The second reveals orchestrated compiler invocations. The third lists known targets, not necessarily installed target libraries.

Bug-location rule: if changing a flag changes behavior before source is read, inspect configuration plumbing first.

9. Lexing: Turning Characters into Tokens#

A lexer groups source characters into tokens, atomic categories useful to parsing. Examples include identifiers, literals, punctuation, delimiters, and comments or whitespace handling. Lexing answers “what pieces are present?” before parsing answers “how are pieces structured?”

For let total = 1 + 2;, a conceptual token sequence is:

keyword-or-identifier(let)
identifier(total)
punctuation(=)
integer-literal(1)
punctuation(+)
integer-literal(2)
punctuation(;)

This is a conceptual sketch, not rustc debug output or an exact token data type. Real token handling preserves details needed for macro token streams and diagnostics. Unicode, raw identifiers, literal suffixes, and joint punctuation create important edge cases.

rustc_lexer supplies low-level lexical analysis with a deliberately small interface. Higher-level lexer work in rustc_parse performs validation and symbol interning. Keeping low-level lexing less coupled makes it reusable and limits diagnostic dependencies.

A malformed numeric literal is likely discovered around lexing or parsing. An unknown variable with perfectly valid spelling survives lexing and parsing. That distinction is the first practical use of staged bug location.

Counterfactual: parsing raw characters directly could avoid a token structure. It would repeatedly solve lexical boundaries and complicate macro fidelity and diagnostics.

10. Parsing and the Abstract Syntax Tree#

Parsing recognizes grammatical structure among tokens. rustc's parser uses recursive descent, where code for language constructs calls code for subconstructs. The result is an abstract syntax tree, abbreviated AST. A tree node represents constructs such as items, expressions, patterns, statements, and types.

“Abstract” means the tree emphasizes syntactic structure rather than being a character-for-character copy. It still remains close enough to source spelling for early transformations and useful diagnostics. Nodes carry spans so the compiler can connect structure back to source text.

For x + 1, a conceptual AST sketch is:

BinaryExpression
  operator: Add
  left: PathExpression("x")
  right: IntegerLiteral("1")

This sketch is explicitly conceptual, not a promise about enum variants in Rust 1.97.1. Exact definitions live in rustc_ast and evolve.

The parser often recovers after malformed input to issue more than one useful error. Recovery may construct error-marked or approximate structure. Downstream code must respect recovery invariants rather than assuming every node came from valid syntax.

Prediction checkpoint: fn f( {} cannot establish a valid parameter list. Expect parser diagnostics before type inference has meaningful operand types to inspect.

11. Expansion and Name Resolution Cooperate#

Macro expansion replaces macro invocations with generated token-based syntax. Declarative macros match token patterns; procedural macros execute separately and return token streams. Expansion can create names, modules, imports, implementations, expressions, and more macros.

Name resolution determines what a path or identifier denotes. For example, Vec might denote an imported standard-library type, a local type, or nothing. Scopes, namespaces, imports, hygiene, editions, and visibility all influence the answer.

Expansion and resolution cannot be pictured as fully isolated one-shot stages. Resolving a macro name may be necessary before expanding it. Expansion may then introduce imports or macro invocations that require further resolution. The compiler coordinates them iteratively while maintaining macro hygiene.

Macro hygiene tracks syntactic context so generated names do not accidentally capture unrelated names. Textual substitution alone would make macros fragile under innocent renaming. Spans participate in this context as well as source location.

macro_rules! make_answer {
	() => {
		const ANSWER: i32 = 42;
	};
}

make_answer!();

fn main() {
	println!("{ANSWER}");
}

The parser initially sees a macro invocation, not simply the final constant item. After expansion, later representations can reason about generated constructs.

Bug-location rule: an unresolved source path suggests resolution; a path emitted incorrectly by a macro may originate in expansion.

12. HIR: A More Uniform, Resolved View#

HIR means High-level Intermediate Representation. Lowering converts the expanded AST into a form better suited to semantic analysis. HIR remains recognizably related to Rust constructs but removes some surface variety.

Desugaring translates convenient syntax into more explicit or uniform internal constructs. Loops, operators, closures, and asynchronous syntax may receive compiler-oriented treatment across lowering stages. Do not assume every sugar is eliminated in exactly one boundary. The useful question is which downstream consumer needs which explicit facts.

HIR associates uses with resolution results and organizes bodies for later queries. It represents what the user wrote more directly than MIR does. Type checking uses HIR alongside compiler type representations and inferred facts.

At the AST-to-HIR boundary:

• Preserved: source relationships, spans, item/body structure, and semantic distinctions still needed.
• Normalized: multiple surface forms become fewer compiler-oriented patterns.
• Added: stable-enough local identities and resolution-informed structure.
• Discarded: syntax details with no remaining semantic or diagnostic purpose.

The exact list changes with implementation details; it is a reasoning framework, not a serialization contract.

Counterfactual: type checking the raw AST would retain convenient spelling but repeatedly handle expansion-era complexity. HIR centralizes normalization so analyses can share assumptions.

13. Types, Inference, and Traits#

A type classifies values and permits or rejects operations. bool, i32, references, tuples, function types, and user-defined structures are examples. Type inference computes omitted types from constraints instead of requiring every type to be written.

fn choose(flag: bool) -> i64 {
	let number = 7;
	if flag { number } else { 9 }
}

The annotation -> i64 constrains both branches and therefore number. Inference is not guessing; it solves relationships generated from program structure. Ambiguity remains an error when constraints do not select an allowed answer.

Traits state shared behavior and relationships. Trait solving asks whether obligations such as “this type implements this trait” can be established. Method lookup, operators, conversions, associated types, and generic bounds all involve trait-related reasoning.

The compiler distinguishes a syntax-level type written in HIR from its semantic internal type representation. rustc_middle::ty::Ty<'tcx> is a central, interned handle for semantic types. The lifetime indicates data tied to the compiler context's allocation lifetime.

Type errors can depend on earlier resolution. If x.foo() resolves to the wrong trait candidate set, a later-looking mismatch may have an earlier cause. Trace facts backward rather than patching the first function that emits an error.

Invariant: accepted operations must have justified types and satisfied required obligations.

14. THIR: A Typed Bridge#

THIR means Typed High-level Intermediate Representation. It bridges HIR-oriented type checking and MIR construction. THIR is fully typed and more explicit about operations such as adjustments and method calls.

An adjustment is an implicit semantic operation, such as an automatic borrow or dereference. Source ergonomics permit the user to omit some mechanical steps. MIR construction needs those steps represented precisely enough to build control flow and places.

THIR is also used by pattern and exhaustiveness analysis. Exhaustiveness asks whether patterns cover every value required by the matched type. Usefulness analysis identifies unreachable or redundant pattern cases.

At the HIR-to-THIR boundary:

• Preserved: body meaning, links to source locations, and type-checked expression relationships.
• Normalized: overloaded and implicit operations become more explicit.
• Added: resolved semantic types and adjustments.
• Discarded: some HIR organization irrelevant to body-level MIR construction.

THIR is often temporary and body-focused. Do not infer that every internal representation must persist in the incremental cache. Its value lies in a clean handoff, not permanence.

15. MIR: Explicit Control Flow and Operations#

MIR means Mid-level Intermediate Representation. It represents a function body as basic blocks connected by control-flow edges. A basic block is a sequence entered at the top and exited through one terminator. A terminator chooses what happens next: return, branch, call, unwind, or another destination.

MIR uses places to describe locations and operands to describe values used by operations. Assignments, moves, copies, references, storage, and drops become explicit enough for dataflow analysis. Control flow is no longer hidden inside nested expression syntax.

Conceptual MIR for a branch might look like this:

bb0:
  switch flag -> [true: bb1, false: bb2]
bb1:
  result = 10
  goto bb3
bb2:
  result = 20
  goto bb3
bb3:
  return result

This is pedagogical pseudocode, not exact -Z dump-mir output. Real MIR includes locals, types, source scopes, unwind behavior, and exact statement/terminator forms.

MIR supports borrow checking, initialization analysis, constant evaluation machinery, optimizations, and code generation. Its Rust-aware semantics permit analyses that would be harder after lowering to LLVM IR. Its uniform control flow supports reusable dataflow frameworks.

Counterfactual: borrow checking nested AST expressions directly would obscure branch joins and move paths. MIR makes relevant effects and paths explicit without yet committing to target instructions.

16. Borrow Checking as Dataflow Reasoning#

Borrow checking verifies Rust's ownership, borrowing, and lifetime constraints over MIR. It reasons about where values move, where references are live, and which accesses conflict. The modern checker is often called the MIR borrow checker.

fn first_length(words: &mut Vec<String>) -> usize {
	let first = &words[0];
	first.len()
}

This compiles because the shared borrow used by first does not conflict with mutation here. Adding words.push(String::new()) before first.len() would create a conflict. The vector mutation may reallocate, invalidating a reference into its storage.

The borrow checker's message uses source spans, but its core reasoning happens over MIR facts. An incorrect error can originate in MIR construction, type adjustments, region inference, or diagnostics. Do not equate the visible message's module with the root cause.

Dataflow means computing facts as they propagate along control-flow edges until required relationships stabilize. Branches merge information; loops may require repeated propagation. The graph representation makes these joins explicit.

Invariant: safe accepted MIR must not contain an access forbidden by established borrow relationships. Prediction checkpoint: unreachable source bodies still require checking for language correctness. Codegen reachability alone must not decide whether borrow errors are reported.

17. Monomorphization, Code Generation, and Linking#

Generic Rust code describes behavior parameterized by types or constants. Monomorphization creates concrete code instances needed for actual uses. For identity::<u32> and identity::<String>, backend code may need distinct instances.

fn identity<T>(value: T) -> T {
	value
}

fn main() {
	let a = identity(3_u32);
	let b = identity(String::from("three"));
	println!("{a} {b}");
}

Monomorphization collection determines reachable concrete items needed for code generation. The work is partitioned into codegen units, which balance parallelism, optimization opportunities, and incremental reuse. More units can permit parallel work but reduce cross-unit optimization unless additional mechanisms compensate.

The LLVM backend lowers monomorphized MIR semantics to LLVM IR. LLVM performs target-independent and target-specific optimization and emits object machine code. Alternative backend experiments illustrate that LLVM is a major component, not Rust's language definition.

The linker combines generated objects, upstream libraries, runtime support, and native dependencies. Duplicate symbols, missing symbols, relocation failures, and linker command failures belong near this boundary. Earlier rustc stages may have succeeded completely.

Invariant: each generated instance must obey the already-checked generic semantics under its concrete substitutions. Tradeoff: monomorphization improves specialization but increases compilation work and potential code size.

18. Why So Many Intermediate Representations?#

No single representation is ideal for every task. Raw text preserves spelling but makes control-flow analysis painful. MIR clarifies control flow but is poor for explaining a missing comma. LLVM IR models low-level execution but no longer naturally expresses Rust trait obligations.

Multiple IRs separate concerns and establish explicit contracts between analyses. They also impose lowering code, memory costs, conversion bugs, and contributor learning overhead. Adding an IR is justified only when its uniformity serves enough consumers.

Tokens preserve lexical fidelity and delimit macro-oriented pieces. AST adds grammatical nesting and source-oriented constructs. HIR adds resolution-friendly identity and desugared semantic organization. Type results add inferred meaning and proven obligations. THIR makes typed operations and adjustments explicit. MIR adds control-flow, place, move, drop, and branch precision. LLVM IR adds target-lowerable operations and optimization-oriented annotations. Machine objects commit to architecture, ABI, relocations, and object-format conventions.

At every boundary, ask four questions:

1. Which facts must remain for correctness?
2. Which surface alternatives can be normalized now?
3. Which derived facts become available only now?
4. Which details can be discarded without harming diagnostics or later analysis?

Counterfactual: keeping source text as the universal truth would force every analysis to repeat parsing and resolution. Counterfactual: lowering immediately to machine code would erase information needed for Rust-specific safety checks.

19. A Function Traced Through Every Representation#

The following function is compilable Rust and will anchor a conceptual trace.

fn clamp_add(x: i32, add: i32, limit: i32) -> i32 {
	let sum = x + add;
	if sum > limit { limit } else { sum }
}

Everything after this source block in the section is a conceptual sketch. Exact debug output, enum names, optimizations, and ordering can differ in Rust 1.97.1. The purpose is to follow information, not imitate unstable printer syntax.

Invocation and source loading#

The driver receives a crate root, edition, target, crate type, and output choices. The session records configuration and arranges diagnostics and source access. The source map assigns file positions so later facts can point back here.

Tokens#

The lexer recognizes fn, identifier clamp_add, delimiters, parameter names, type names, operators, and literals. Whitespace separates some tokens but does not become an arithmetic operation. The > character is punctuation whose grammatical role is decided during parsing.

fn clamp_add ( x : i32 , add : i32 , limit : i32 ) -> i32 {
  let sum = x + add ;
  if sum > limit { limit } else { sum }
}

AST#

The parser constructs a function item with three typed parameters and one return type. Its body contains a let statement followed by a tail if expression. The additions and comparison are binary-expression nodes with source spans.

FunctionItem clamp_add
  Parameters: (x: i32, add: i32, limit: i32)
  ReturnType: i32
  Body
    Let sum = Binary(Add, Path(x), Path(add))
    Tail If Binary(Greater, Path(sum), Path(limit))
      Then Path(limit)
      Else Path(sum)

Expansion and resolution#

There are no macro invocations in this function, so expansion changes little locally. Resolution links each i32 spelling to the primitive type and each local use to its binding. The two sum uses resolve to the local introduced by let sum. The function name receives an identity distinct from every same-spelled item elsewhere.

HIR#

HIR stores a lowered, compiler-oriented function body with identities and preserved spans. The if remains semantically recognizable, while incidental syntax is normalized. Operator expressions are ready to participate in type-dependent checking.

Type checking and trait-related meaning#

Parameter annotations establish all inputs as i32. The addition must produce an i32, so sum is inferred as i32. The comparison produces bool; both if arms produce i32. The tail expression therefore matches the declared i32 return type.

Although primitive operators receive special compiler handling, operator meaning is still type-directed. The important evidence is that selected operations are valid for these operands. No unresolved inference variable remains for this simple body.

THIR#

THIR presents every expression with its computed type. Implicit adjustments, if any, would be represented explicitly enough for MIR building. Here the arithmetic, comparison, local accesses, and branch values are straightforward i32 operations.

TypedLet sum:i32 = Add<i32>(x:i32, add:i32)
TypedIf
  condition: Greater<i32> -> bool (sum:i32, limit:i32)
  then: limit:i32
  else: sum:i32
result type: i32

MIR construction#

MIR introduces locals and basic blocks. The if becomes a branch terminator, and both paths assign a return place. Overflow checks may appear depending on compilation settings and exact MIR phase.

bb0:
  sum = Add(x, add)
  condition = Greater(sum, limit)
  switch condition -> [true: bb1, false: bb2]
bb1:
  return_place = limit
  goto bb3
bb2:
  return_place = sum
  goto bb3
bb3:
  return

Borrow checking and MIR analyses#

The body owns no references, so borrow relationships are trivial. Initialization analysis still verifies that sum is assigned before either use. Every control-flow path reaching return assigns the return place.

MIR optimization#

Transforms may simplify temporaries, propagate values, or reshape control flow. They must preserve overflow and comparison semantics appropriate to the chosen mode. The exact optimized MIR is not a stable interface.

Monomorphization#

This function has no generic parameters, so its concrete identity needs no type specialization. Reachability still determines whether codegen requires an emitted instance. An unused function must be semantically checked even if not emitted into a final binary.

Backend and linking#

Backend lowering maps operations and branch selection into LLVM IR. LLVM may choose a conditional-select instruction rather than an explicit branch when profitable. Target machine code follows the target's register, instruction, ABI, and object conventions. The linker includes the symbol if required and combines it with the crate's other products.

Prediction checkpoint: changing > to >= first changes one token and AST operator. That semantic difference propagates through typed operations, MIR comparison, and machine behavior.

Prediction checkpoint: renaming sum consistently should preserve runtime behavior. It changes source evidence and identifiers but should not alter the arithmetic meaning.

20. Spans and the Source Map#

A span identifies a region of source-related input plus contextual information. Compiler nodes and derived facts carry spans so diagnostics can label relevant text. Spans also participate in macro hygiene, where “where did this token come from?” is richer than byte offsets.

The SourceMap manages loaded source files and translates compiler positions into file, line, and column views. Line and column are presentation concepts derived from underlying positions and file data. Generated code and macro expansions may involve call-site and definition-site relationships.

A good diagnostic often needs several spans:

• the primary expression where the error becomes visible;
• the declaration establishing an incompatible requirement;
• a previous move or borrow that explains current invalidity;
• the macro invocation responsible for generated syntax.

Span correctness is user-facing correctness. An analysis can reject the right program for the right reason yet point to the wrong token. That becomes a diagnostics bug rather than a semantic acceptance bug.

Invariant: retaining a span must not be mistaken for retaining the complete source construct. A span is a bridge back to text, not a substitute for semantic representation.

Bug-location rule: correct rejection with a shifted underline points toward span construction or diagnostic selection.

21. Identity: NodeId, DefId, and HirId#

Names are poor internal identities because scopes can contain repeated spellings. Compiler processing therefore assigns numeric or structured identifiers. These IDs are meaningful only within their documented domains and phases.

A NodeId conceptually identifies an AST node during earlier processing. It belongs to an expansion-era, crate-local world and should not be treated as permanent global identity. Exact usage evolves as compiler architecture changes.

A DefId conceptually identifies a definition, such as an item, across crate boundaries. It includes enough crate context to distinguish upstream and local definitions. A LocalDefId expresses that the definition belongs to the crate currently compiled. Compiler queries commonly take definition IDs as keys.

A HirId conceptually identifies a HIR node within an owning definition. Thinking “owner plus local position” is more useful than memorizing field layouts. HIR ownership helps stable traversal and per-owner processing.

IDs are not source addresses, human names, or guaranteed stable database keys across versions. Converting between ID kinds requires proof that the referenced entity exists in both domains. Unwrap-heavy conversion code can reveal an unstated invariant worth documenting or testing.

Prediction checkpoint: two functions named parse in different modules need different DefId values. Their source spelling alone cannot key semantic facts.

22. Interning, Arenas, and Cheap Shared Data#

Compilers create enormous numbers of repeated values: types, symbols, lists, and structured facts. Copying and deeply comparing them would consume time and memory. rustc therefore uses interning and arena allocation extensively.

Interning stores a canonical shared instance for an equal value. Users receive a compact handle or reference to that instance. Equality can often become cheap identity comparison after canonicalization. This is why semantic Ty values are lightweight handles despite describing rich types.

An arena allocates many values with a shared lifetime and frees them together. It avoids per-object deallocation bookkeeping and supports stable references. The tradeoff is that individual values usually cannot be reclaimed early. Peak memory can therefore matter even when allocation is fast.

Interning and arenas are related but distinct. An arena answers where and how long storage lives. Interning answers whether equal values share one canonical stored instance. An arena can store non-interned values; an interner needs some storage strategy.

Invariant: interned identity comparisons are valid only if canonicalization's equality contract is correct. Counterfactual: cloning full recursive type trees into every expression would simplify local ownership but badly duplicate data.

23. TyCtxt and the 'tcx Lifetime#

TyCtxt, commonly stored in a variable named tcx, is the central compiler context handle. Its historic name “type context” understates its reach. It provides access to queries, interned data, definitions, metadata, and broad compilation state.

Many query methods are invoked through tcx. The custom query engine records dependencies when one query requests another. In-memory query caches are connected to this context. This does not mean every method call is a query or every returned value is serialized.

The lifetime 'tcx marks data tied to the context's arena-backed lifetime. A Ty<'tcx> cannot safely outlive the storage that owns its interned representation. Passing TyCtxt<'tcx> by copy passes a small handle, not a duplicate compiler universe.

This lifetime design makes bulk-owned compiler data ergonomic to reference. It also creates architectural constraints: short-lived temporary data should not be smuggled into 'tcx storage casually. Interning everything would increase memory and blur ownership boundaries.

When a lifetime error appears inside rustc, ask which owner should truly retain the data. Do not automatically stretch the lifetime through allocation. The borrow checker may be exposing a mistaken phase or caching boundary.

24. Queries: Demand, Dependencies, and Caches#

A compiler query is conceptually a named function from a stable key to a result. Examples include asking for type-checking results or optimized MIR for a definition. During execution, a query can request other queries, forming a dependency graph.

If an input changes, dependency information helps determine which derived results may be invalid. This supports incremental compilation without rerunning every possible computation. Memoization within one session also avoids repeating identical requests.

rustc's query system is custom infrastructure implemented in the compiler. It is not Salsa, although Salsa is another Rust ecosystem framework with related ideas. Using the correct name matters because implementation details, cycles, keys, and persistence differ.

The pipeline metaphor remains useful for representation order but fails as a schedule. Requesting code generation for reachable items can demand optimized MIR, which demands earlier body facts. Separately, compiler coordination forces checks for bodies whose errors must be reported even when unreachable.

Not all query results persist between compiler processes. Some are cached only in memory, some participate through fingerprints or serialized forms, and some recompute. Disk caching everything would make cache loading, storage, validation, and compatibility too expensive.

Early compilation is not uniformly decomposed into tiny per-item queries. Lexing, parsing, expansion, and resolution include broad crate-level work and iterative interaction. Never claim “every rustc stage is one query per item.”

Invariant: query providers should behave like deterministic computations of declared inputs and dependencies. Hidden inputs can cause stale reuse and are a classic incremental-compilation bug source.

Bug-location rule: a clean build works but an incremental build fails or miscompiles. Suspect missing dependency edges, unstable fingerprints, serialization, or invalid cache assumptions.

25. Crate Graphs and Metadata Boundaries#

The crate graph is a directed graph whose nodes are crates and edges are dependencies. It is related to, but not identical with, Cargo's package-resolution graph. One package can produce multiple crates, and host-built units can differ from target-built units.

Upstream crates export metadata needed by downstream rustc invocations. Metadata can describe public definitions, types, trait implementations, generics, and codegen-relevant material. The exact format is internal and version-sensitive. Downstream rustc need not retain upstream ASTs as if all source formed one giant crate.

Metadata is an architectural boundary with two competing pressures. Export too little and downstream compilation cannot type-check or instantiate generic code. Export too much and artifacts, decoding cost, privacy risk, and compatibility burden increase.

Cross-crate inlining and generic monomorphization require selected implementation information beyond signatures. Private details can still affect generated downstream code through controlled metadata mechanisms. That does not make every private source node globally addressable.

Prediction checkpoint: a corrupted dependency artifact can fail during metadata loading before local HIR type checking. Invariant: a DefId for an upstream item must be interpreted with the correct crate metadata context.

26. Frontend, Middle End, and Backend#

Compiler discussions often divide work into frontend, middle end, and backend. The frontend usually means source-language parsing and semantic analysis. The middle end usually means language-informed IR analysis and optimization. The backend usually means lowering to target code and emitting objects.

For rustc, a rough classification places parsing, expansion, resolution, HIR, and type checking toward the frontend. MIR building, borrow checking, MIR analysis, and optimizations occupy a Rust-specific middle. LLVM lowering, LLVM optimization, object emission, and linking sit toward the backend.

These labels are imprecise rather than authoritative module boundaries. Borrow checking is a semantic language check but runs on “middle” IR. Monomorphization is both Rust-semantic and codegen-oriented. Diagnostics and query infrastructure cut across all three.

Use the labels to orient a conversation, then name the actual representation or subsystem. “Frontend bug” is less actionable than “resolution assigned the wrong DefId after expansion.”

Counterfactual: calling everything before LLVM “frontend” hides most of rustc's architecture. Calling MIR purely backend hides why borrow checking and const evaluation use it.

27. The rust-lang/rust Repository Map#

The main repository contains far more than the compiler crates. Top-level areas include compiler implementation, standard libraries, tools, tests, bootstrap machinery, and documentation. Paths move, so learn search strategies and conceptual ownership rather than memorizing a snapshot.

The compiler/ directory contains many crates prefixed rustc_. The library/ directory contains standard-library layers and related support. The tests/ hierarchy contains compiler tests organized by purpose and harness. The src/tools/ area contains project tools, though exact arrangements evolve. Bootstrap code builds a compiler using an earlier compiler.

Important source-facing compiler crates include:

• rustc_driver and rustc_interface: invocation and compiler coordination;
• rustc_session: per-compilation options, diagnostics, and session state;
• rustc_span: source positions, spans, symbols, and source mapping;
• rustc_lexer, rustc_parse, and rustc_ast: tokens, parsing, and AST definitions;
• rustc_expand and rustc_resolve: macro expansion and name resolution;
• rustc_ast_lowering and rustc_hir: HIR construction and representation;
• rustc_hir_typeck, rustc_infer, and trait-solving crates: semantic type reasoning;
• rustc_middle: central type and MIR definitions plus shared semantic machinery;
• rustc_mir_build: THIR-related lowering and MIR construction;
• rustc_borrowck: MIR borrow checking;
• rustc_mir_dataflow and rustc_mir_transform: reusable analysis and MIR transformations;
• rustc_monomorphize: collecting concrete codegen work;
• rustc_codegen_ssa and rustc_codegen_llvm: backend abstraction and LLVM backend;
• rustc_metadata: encoding and decoding cross-crate information;
• rustc_query_impl and rustc_incremental: query providers and incremental machinery;
• rustc_errors: diagnostic construction and emission support;
• rustc_target: target descriptions and target-specific configuration.

This list is an orientation aid, not a claim that every responsibility fits one crate. Nightly API crate listings are the best current inventory for a particular toolchain snapshot. Search call sites before assuming a crate name completely defines ownership.

28. Stability Boundaries and Version Sensitivity#

Rust 1.97.1 is this text's conceptual target. Language behavior and stable rustc command-line options carry compatibility expectations appropriate to stable Rust. The Rust Reference describes language rules, while the rustc book documents compiler options.

Compiler internals are not a stable public API. Crate layouts, query names, IR fields, debug printers, and internal invariants can change between nightlies. rustc_private crates require unstable mechanisms and are intended for compiler/tool development, not stable application dependencies.

“Available in nightly rustdoc” does not mean “stable to depend upon.” Nightly API documentation is invaluable for reading the matching source snapshot. It is descriptive evidence, not a semver promise.

The stable CLI also contains output whose exact formatting may not be a guaranteed machine interface. Check each option's documented stability and output contract before building tools around it. Prefer explicit structured formats where the compiler documents them for tooling.

When teaching publicly, state the version and separate durable architecture from exact names. Say “conceptually keyed by a definition” before projecting a current query signature. That keeps a talk truthful when implementation details move.

Invariant: never infer language semantics solely from one unstable IR dump. Use tests, the Reference, design documents, and source behavior together.

29. From Symptom to Likely Stage#

Bug triage starts with the earliest fact that appears wrong. The visible crash site can be downstream of the actual corruption. Minimize the example, classify the symptom, and inspect representation boundaries.

• Token splitting, literal spelling, or Unicode boundary wrong: inspect lexer layers.
• Unexpected grammar error or recovery cascade: inspect parsing and AST recovery.
• Macro-generated syntax, hygiene, or expansion order wrong: inspect expansion and spans.
• An identifier denotes the wrong declaration: inspect name resolution and identity conversion.
• Desugared structure loses source meaning: inspect AST-to-HIR lowering.
• Inference, method lookup, or trait obligation wrong: inspect type checking and trait solving.
• Pattern coverage or typed adjustment wrong: inspect THIR and pattern analysis.
• Move, drop, branch, or temporary represented wrongly: inspect MIR construction.
• Valid borrow rejected or invalid borrow accepted: inspect MIR, borrow checking, and region facts.
• Correct checked program executes incorrectly: inspect MIR transforms, codegen, LLVM interaction, or linking.
• Only cross-crate builds fail: inspect metadata encoding, decoding, and crate identity.
• Only incremental builds fail: inspect query dependencies, fingerprints, and persisted state.
• Underline or suggestion wrong but rejection correct: inspect spans and diagnostics.
• Native symbol missing: inspect attributes, codegen symbol handling, native libraries, and linker command.
• Cross-compilation-only failure: separate host tools from target artifacts and target ABI assumptions.

The map gives hypotheses, not verdicts. A parser crash can be caused by tokens emitted from expansion. Wrong code can begin with an invalid type-system assumption and surface in LLVM. Confirm by dumping or instrumenting the representation immediately before and after the suspected boundary.

A disciplined narrowing loop#

First, preserve the exact command and toolchain revision. Second, reduce source while retaining the behavior. Third, decide whether the failure is acceptance, rejection, diagnostics, crash, performance, or wrong code. Fourth, identify the earliest representation where expected and actual facts differ. Fifth, find the producer of that fact, not merely its final consumer. Sixth, add a regression test at the narrowest public behavior boundary.

Prediction checkpoint: MIR is already wrong before optimization. Disabling LLVM optimizations cannot fix the root cause, though it might hide the symptom.

30. Initial Exercises for Building Source Fluency#

These exercises require observation rather than immediate compiler modification. Use a matching nightly when an unstable -Z option is needed. Exact printer output is unstable, so record the full compiler version.

Exercise 1: Observe orchestration#

Create a tiny Cargo package and run cargo build --verbose. Identify each rustc invocation, its crate name, crate type, dependency arguments, host, and target. Predict which command changes after editing only the binary crate. Explain why Cargo's graph is visible as multiple crate compilations.

Exercise 2: Classify failures#

Produce one parse error, one unresolved name, one type mismatch, one borrow error, and one linker error. For each, write the earliest likely representation containing enough information to diagnose it. Then explain one plausible earlier root cause that could imitate the same symptom.

Exercise 3: Compare representations#

Use nightly compiler debugging options documented for your revision to inspect HIR or MIR. Choose a for loop, a method call with autoref, and an if let expression. Mark which source conveniences remain and which become explicit. Do not treat printer formatting as stable API.

Exercise 4: Follow one definition#

Find a small function in compiler/ and search for callers by symbol rather than browsing directories. Identify its input ID type, returned data's owner, and whether it invokes a query. Write down every hidden assumption you discover at conversions or unwraps.

Exercise 5: Reason about a counterfactual#

Suppose rustc emitted code only for reachable functions and checked only emitted functions. Construct an unreachable function containing a type or borrow error. Explain why accepting the crate would violate Rust's compilation model. Relate this to forced query work versus codegen-driven demand.

Exercise 6: Trace evidence#

For the clamp_add function, annotate source spans needed for a hypothetical type mismatch. State when binding identity, expression type, control flow, and target instruction choice become known. Identify one fact discarded at each transition.

Exercise 7: Host versus target#

Describe a cross-build containing a build script and procedural macro. Draw which binaries run on the host and which artifacts target the destination. Predict the failure if a proc macro is built only for the target.

Exercise 8: Prepare a five-minute talk#

Explain rustc using the evidence-preserving translation frame and the end-to-end diagram. Include one tradeoff, one query caveat, and one example of symptom localization. Avoid saying “the compiler just knows” or “every stage runs in sequence.”

31. Reading Live Source Without Memorizing Paths#

Begin from an observable artifact: a diagnostic phrase, query name, type name, flag, or debug label. Search exact symbols with repository search tools. Then read definitions and callers together, because names alone rarely reveal lifecycle and invariants.

Use the nightly API index matching your checkout to discover crates and public internal items. Open source links from rustdoc when possible. Read the rustc-dev-guide chapter for intent, then verify claims against current code. Guide prose may lag architecture, while code alone may omit motivation.

Follow data in both directions. Ask who constructs a value, who consumes it, where its IDs originate, and how long it lives. For a query, find its key, provider, dependencies, forcing points, and result ownership. For a diagnostic, follow semantic detection separately from message rendering and span selection.

Tests are executable architecture notes. Search existing regression tests for flags, error codes, revisions, and expected output. A test directory name is helpful context, not proof of subsystem ownership.

Prefer conceptual landmarks over fixed paths:

• representation definition;
• lowering producer;
• analysis consumer;
• query provider;
• diagnostic emission;
• regression test.

When a path in an article is stale, search the symbol in the repository. When the symbol is gone, search its old callers, associated diagnostic text, or git history. This method survives refactors better than memorizing directory trees.

Invariant: source is authoritative for implementation, but public language behavior is constrained by specifications and stability policy.

32. Glossary, Sources, and the Mental Model to Keep#

ABI: binary-level rules for calls, data layout, symbols, and cooperation between artifacts.

Arena: an allocator that stores many values and releases them together at a shared lifetime boundary.

AST: Abstract Syntax Tree, a source-oriented grammatical structure produced by parsing.

Backend: imprecise term for target-oriented lowering, optimization, and machine-code emission.

Basic block: straight-line MIR operations ending in a control-flow terminator.

Borrow checking: analysis enforcing ownership, reference, and lifetime access rules.

Cargo: Rust package manager and build orchestrator that invokes rustc for crates.

Codegen unit: a partition of monomorphized work compiled by a backend.

Compiler: a program that validates and translates a source language into another representation.

Crate: Rust compilation unit and principal metadata boundary.

Crate graph: dependency graph among separately compiled crates.

DefId: contextual identity for a definition, including distinctions across crates.

Desugaring: replacing convenient surface forms with more uniform internal forms.

Diagnostic: compiler communication such as an error, warning, label, note, or suggestion.

Driver: machinery configuring and coordinating a compiler invocation.

HIR: High-level Intermediate Representation used for source-near semantic analysis.

HirId: identity for a HIR node understood relative to an owning definition.

Host: platform on which the compiler or compile-time tool executes.

Hygiene: macro context tracking that prevents accidental name capture and preserves origin relationships.

Incremental compilation: reusing valid work from earlier compilations after inputs change.

Inference: deriving omitted facts, especially types, from constraints.

Interner: storage that canonicalizes equal values and returns cheap shared handles.

IR: Intermediate Representation, a form chosen to support particular analyses or transformations.

Lexer: component that groups source characters into tokens.

Linker: tool combining object files and libraries into a linked artifact.

LLVM IR: typed low-level representation accepted by LLVM's optimization and code-generation machinery.

Metadata: encoded cross-crate facts consumed by later rustc invocations.

MIR: Mid-level Intermediate Representation with explicit typed control flow and places.

Monomorphization: producing concrete code instances for generic arguments that are used.

NodeId: conceptually, an identity for an AST-era node within early crate processing.

Parser: component recognizing grammatical structures in a token stream.

Place: MIR description of a memory location, possibly with projections such as fields.

Proc macro: separately compiled macro program executed during compilation to transform token streams.

Query: tracked computation keyed by compiler identity or input, able to request dependencies.

Resolution: determining which declaration or entity a source name denotes.

Session: state and services associated with one compiler invocation.

Soundness: here, the guarantee that safe accepted Rust cannot itself enable undefined behavior.

Source language: language accepted as compiler input.

SourceMap: compiler facility mapping internal positions to loaded files and human locations.

Span: source region plus context used for origin tracking, hygiene, and diagnostics.

Target: platform and environment for which output code is generated.

THIR: Typed High-level Intermediate Representation bridging typed HIR bodies and MIR construction.

Token: lexical unit such as an identifier, literal, delimiter, or punctuation.

Trait solving: establishing trait obligations and associated relationships required by typed code.

TyCtxt: central rustc context handle providing queries, interners, and compilation-wide semantic access.

Authoritative references#

The rustc-dev-guide overview supplies the maintained architectural tour. Its chapters on queries, the HIR, MIR, and code generation deepen this map.

The nightly rustc API index inventories current internal crates and links their source. Consult a dated toolchain's generated documentation when exact signatures matter.

The Rust Reference is the primary reference for Rust language rules. Its crate and source-file chapter describes crate-level language organization.

The official rustc book documents rustc as a command-line compiler. Its opening chapter explains direct invocation and crates as translation units. The platform support chapter documents target support expectations.

These sources have different authority domains. Use the Reference for language rules, the rustc book for supported compiler usage, nightly rustdoc for a snapshot of internals, and the development guide for contributor explanations.

The final durable picture#

rustc is not a magic Rust-to-binary dictionary. It is a versioned, query-driven system that repeatedly translates and enriches evidence. Text becomes tokens; tokens gain grammar; names gain identities; expressions gain types. Typed bodies gain explicit control flow; borrow analysis establishes safety conditions; monomorphization selects concrete work. Backend lowering commits that checked meaning to a target, and linking joins binary worlds.

The representations form a semantic direction even when execution is not a simple sequential pipeline. The query engine schedules much later work by demand, tracks dependencies, and supports reuse. Whole-crate and forced checks coexist with that demand model. Early expansion and resolution have interactions too broad for a fairy tale of tiny independent item queries.

When something fails, locate the first missing, incorrect, or prematurely discarded piece of evidence. When proposing a design, identify its correctness, soundness, diagnostics, compile-time, runtime, and compatibility tradeoffs. When reading source, follow producers, consumers, identities, owners, and queries rather than memorizing paths.

That is the foundation for contributing and speaking accurately about rustc: the compiler is an evidence-preserving translation system and a staged engineering proof, built from explicit compromises, whose internals move while its public promises remain the reason the machinery exists.

Part I-B: The 73-Subsystem Mastery Map#

This map turns the rest of the book into a deliberate curriculum.

The number 73 is not a compiler law.

It is a useful decomposition of rustc into bounded responsibilities grouped under 20 themes.

Real source ownership crosses these boundaries, and one compiler crate can serve several subsystems.

The purpose of the map is to prevent accidental breadth from masquerading as mastery.

For every subsystem, the reader should learn five things:

1. the problem that forced the subsystem to exist;
2. the representation and invariant it owns;
3. its inputs, outputs, and neighboring contracts;
4. its production failure modes and debugging evidence;
5. how to implement a bounded educational version and recognize its omissions.

33. How to use this map#

Do not read the 73 rows as 73 independent passes.

Rustc is demand-driven in important areas, macro expansion and resolution cooperate, and semantic facts can cross several representations.

Use each row as a competency checkpoint.

The primary study column identifies the deepest treatment in this guide.

The mastery proof column names something the reader should be able to do without reciting prose.

overview mental model
        |
        v
subsystem mechanism --> bounded implementation
        |                       |
        v                       v
production internals <-- compare omissions
        |
        v
debugging + contribution + teaching

The progression is intentionally cyclic.

After implementing a toy mechanism, reread the production chapter.

The omissions will become concrete rather than abstract warnings.

After reading production source, return to the toy and add one feature.

That exercise reveals which complexity belongs to Rust and which belongs to any production compiler.

34. Source management#

Source management preserves the connection between compiler facts and the bytes users can edit.

It is the foundation of every later diagnostic.

The shared invariant is provenance.

A later phase may simplify syntax, but it must either preserve the source relationship needed by consumers or state that the relationship is gone.

The philosophical lesson is that a compiler error is not just text.

It is a claim backed by semantic evidence and a mapping back to the user's program.

35. Parsing layer#

The parsing layer separates recognition, structure, and policy so malformed input cannot trap the compiler or poison later phases.

Lexing, parsing, and validation are not interchangeable names for “the front end.”

The lexer knows character boundaries but little grammar.

The parser knows local grammar but not every semantic condition.

Validation owns selected structural promises before lowering trusts them.

36. Macro system#

Macros make compilation a staged feedback process rather than a one-way pipeline.

The central philosophical idea is staged responsibility.

Macros move work across time.

They do not delete the need for parsing, validation, resolution, provenance, security, or resource policy.

37. Module and name resolution#

Resolution gives identity to names while respecting namespaces, scopes, modules, imports, hygiene, and crate boundaries.

Resolution teaches that identity is contextual.

Text equality is neither definition identity nor accessibility.

Separating lookup from privacy permits better recovery and diagnostics while preserving the policy boundary.

38. Type-system core#

The type-system core turns partially known source programs into consistent semantic terms and recorded implicit operations.

The core art is disciplined uncertainty.

Inference variables are not errors and are not permission to guess.

They are owned unknowns whose eventual assignments must satisfy every recorded relation.

39. Trait system#

Trait solving is recursive proof search under assumptions, open-world coherence rules, and incomplete information.

The philosophy is calibrated proof.

“Maybe,” overflow, and ambiguity preserve facts about uncertainty.

Turning them into convenient Boolean answers can create order dependence, incompatibility, or unsoundness.

40. Advanced type features#

Advanced features combine quantified variables, associated families, symbolic values, and hidden identities.

These mechanisms preserve who is allowed to choose a value or reveal a type.

Many advanced-type soundness bugs are authority bugs: an inner placeholder escapes, a caller loses choice, or a hidden type is revealed outside its defining boundary.

41. HIR layer#

HIR is a resolved, owner-organized representation that keeps enough source structure for semantic analysis while removing selected surface complexity.

HIR demonstrates purposeful forgetting.

Lowering should erase only distinctions that later consumers no longer need or preserve their provenance elsewhere.

42. MIR construction#

MIR makes execution order, control flow, places, effects, cleanup, and destruction explicit.

MIR's philosophy is explicit effect ownership.

What source syntax leaves implicit must become a typed operation or control-flow edge before analyses can reason uniformly.

43. Borrow checking#

Borrow checking proves that every reachable access is compatible with ownership, initialization, and loans that can matter there.

Lexical checking, NLL, and Polonius refine representation without changing the underlying safety obligation.

This is a central compiler-design pattern: preserve the invariant while improving the proof.

44. Async and coroutines#

Async compilation turns resumable source computations into explicit state, resume control flow, and state-specific destruction.

The important mental shift is that a suspended stack frame becomes stored data.

Liveness affects layout, layout affects pinning, and cancellation requires destruction correct for each reachable state.

45. Constant evaluation#

Compile-time interpretation and optimization-time constant propagation answer different questions under different policies.

CTFE decides whether a compile-time execution is permitted and what it produces.

Constant propagation proves selected values to optimize runtime MIR.

Sharing an interpreter mechanism does not make their policies identical.

46. MIR optimizations#

MIR optimization is a sequence of semantics-preserving rewrites under phase and resource contracts.

Optimization is not “make the code look simpler.”

It preserves meaning under an explicit observation model while trading compiler time and memory for runtime or code-size gains.

47. Monomorphization#

Monomorphization turns generic semantic instances into a finite set of concrete code-generation work.

The representation changes from “works for every permitted argument” to “this concrete instance is required.”

That change makes machine layout and dispatch cheap while increasing code size and collection obligations.

48. Backend#

The backend translates concrete MIR obligations into target artifacts without inventing semantics.

The backend boundary is a contract.

An LLVM assertion can expose malformed compiler input, invalid target assumptions, backend misuse, or an LLVM defect.

“The backend crashed” is a symptom, not a diagnosis.

49. Metadata#

Metadata carries selected semantic facts across compiler processes and crate boundaries.

Serialization is not clerical work.

It is a compatibility and security boundary whose decoded values can influence accepted programs and generated code.

50. Incremental compilation#

Incremental compilation reuses work only when reuse is observationally equivalent to a clean build.

Caching creates proof obligations.

Every omitted semantic input and every partially committed value is a possible wrong program, not merely a missed optimization.

51. Static analysis#

Static analyses prove bounded facts or enforce explicit policy before later stages rely on stronger assumptions.

Analysis precision and language acceptance are connected but not identical.

A conservative false positive may reject safe code; an unsound false negative may invalidate downstream assumptions.

State which direction an approximation permits.

52. Platform layer#

Platform support turns abstract Rust operations into layouts and calling contracts for a concrete target.

The host executes the compiler; the target executes generated code.

Confusing their widths, ABIs, features, or object conventions creates failures that can remain hidden on native builds.

53. Compiler runtime infrastructure#

Runtime infrastructure owns allocation lifetime, canonical shared values, invocation policy, and concurrent publication.

The philosophical lesson is lifetime and publication ownership.

Fast shared access is safe only when construction, immutability, cancellation, and destruction have explicit owners.

54. Dependency order for learning#

A useful first pass follows representation dependencies:

source + spans
      |
      v
lexer -> parser -> AST -> expansion <-> resolution
                              |
                              v
                             HIR
                              |
                              v
type inference <-> trait solving
                              |
                              v
                         THIR / MIR
                              |
              +---------------+---------------+
              |               |               |
              v               v               v
       borrow checking    CTFE/analysis    optimization
              \               |               /
               +--------------+--------------+
                              |
                              v
                 monomorphization + backend
                              |
                              v
                    objects + linker artifacts

queries, metadata, diagnostics, sessions, and targets cross the whole graph

This order is pedagogical, not execution order.

After the first pass, study one vertical concern across the compiler:

• identity from token to linker symbol;
• provenance from source byte to diagnostic;
• uncertainty from parser recovery to trait ambiguity;
• target policy from session option to object file;
• caching from query input to clean-build equivalence.

Vertical study prevents subsystem expertise from becoming tunnel vision.

55. A proof ladder for subsystem mastery#

Use the same ladder for each of the 73 rows.

Level one: explain the problem.

Give a concrete program that the subsystem handles and show why the previous representation cannot answer the question cheaply or correctly.

Level two: trace the mechanism.

Name the inputs, every important state change, output, and invariant.

Do not hide unknowns behind “the compiler figures it out.”

Level three: implement a bounded model.

Use stable Rust and the standard library where practical.

State exactly which language forms, failure modes, and production requirements are omitted.

Level four: harden it.

Add malformed input, resource bounds, determinism, cancellation, observability, cross-platform behavior, and clean-equivalence tests as relevant.

Level five: read current rustc.

Pin a commit, trace one real example, and distinguish current source facts from language guarantees and historical design.

Level six: debug.

Seed or find a defect and locate the earliest broken invariant rather than patching the final symptom.

Level seven: contribute and teach.

Write a focused regression, justify the owning layer, measure risk, and teach the mechanism with one trace and one counterexample.

Reading establishes vocabulary.

The proof ladder turns that vocabulary into engineering judgment.

56. The philosophy shared by all 73 subsystems#

Representations determine cheap questions.

Tokens make lexical boundaries cheap.

HIR makes owner-oriented semantic traversal cheap.

MIR makes control flow and effects cheap.

Canonical goals make reusable solver evaluation cheap.

No representation makes every question cheap.

Abstractions move work.

Macros move construction to compilation.

Generics move concrete choice to instantiation.

Trait objects move concrete identity into runtime metadata.

Queries move scheduling to demand.

The cost reappears in another owner.

Uncertainty must remain explicit.

Error nodes, inference variables, ambiguous goals, provisional cache entries, and unavailable metadata all record that a conclusion is not yet justified.

Guessing creates distant and misleading failures.

Caching is a correctness claim.

A cache entry says that old work is valid under a new observation.

The key, dependencies, compatibility boundary, commit protocol, and corruption behavior form part of that proof.

Identity is not location.

A byte offset, AST index, HirId, DefId, inference-variable number, allocation ID, and linker symbol answer different identity questions.

Converting among them requires an owned mapping.

The first visible failure is often late.

A linker error may begin in crate identity.

A borrow error may begin in binder shifting.

An incremental miscompile may begin with an untracked session input.

Debug backward through representations until expected and actual meaning first diverge.

57. Maintaining the map as rustc evolves#

This map targets the rustc 1.97.1 perspective used throughout the book.

Crate boundaries, default solver modes, query names, pass order, experimental integrations, and internal APIs evolve.

When auditing a newer revision:

1. pin the compiler commit and generated rustdoc;
2. check the Reference for language guarantees;
3. find the current query or driver entry point;
4. trace callers and consumers rather than trusting old paths;
5. read nearby tests and recent changes;
6. update implementation claims without rewriting durable mental models as accidental history.

A subsystem should split when it acquires a distinct invariant, representation, failure surface, and contribution path.

Two subsystems should merge in the map only when separating them no longer teaches a useful ownership boundary.

The goal is not to defend the number 73.

The goal is to ensure no important compiler responsibility disappears behind a broad label such as “front end,” “type checker,” or “backend.”

Part II: Lexing, Parsing, Macros, and the Expanded AST#

This part follows source text from a file to rustc's expanded abstract syntax tree. It targets the architecture around rustc 1.97.1, but compiler-internal crates have no stability promise and evolve continuously. Names of private types, fields, queries, processes, and debugging flags may therefore differ in another checkout. The durable goal is to understand the contracts between stages, not to memorize today's spelling.

1. The front end's job and vocabulary#

A compiler translates a program from one representation into another. Its front end is the portion that recognizes source language structure and reports early mistakes. For Rust, that journey is not a straight lexer-then-parser pipeline because macros can manufacture more Rust syntax. The broad path is source bytes, lexical tokens, token trees, an initial abstract syntax tree, repeated macro expansion and integration, and a fully expanded tree ready for later lowering.

Lexical analysis, usually shortened to lexing, groups characters into tokens such as an identifier, integer, or plus sign. Parsing discovers grammatical relationships among tokens. An abstract syntax tree, or AST, is an in-memory tree that records meaningful syntax while omitting some surface detail. Expansion replaces macro invocations and attribute-driven transformations with generated syntax. Lowering is a later translation into a representation better suited to semantic analysis; it is outside this part except where the boundary matters.

An invariant is a fact that must remain true at a particular boundary. For example, a lexer must always advance or terminate, and every source-backed token span must lie within its source file. A failure mode is a way that an invariant can be broken. A bug-location clue is evidence identifying the stage likely responsible. If punctuation is colored incorrectly before parsing, suspect lexing. If tokens are right but multiplication associates incorrectly, suspect parsing. If only generated names fail to resolve, suspect expansion or hygiene.

Do not imagine that all ordinary name resolution finishes before expansion. Macro name resolution, import resolution, expansion, and AST integration interact. Expansion can reveal imports and macro definitions; resolving a macro can be necessary before expansion can continue. Later semantic name resolution has distinct responsibilities, but the front-end process is deliberately iterative.

2. Bytes, UTF-8, and characters#

A byte is an eight-bit integer, normally in the range 0 through 255. A source file on disk is a sequence of bytes, not an array of human-visible letters. Rust source is interpreted as UTF-8, a variable-width encoding of Unicode. Unicode is a standard assigning abstract characters, called Unicode scalar values in Rust's model, to numeric values. UTF-8 represents one scalar value with one to four bytes.

The ASCII character a occupies one byte. The scalar é occupies two UTF-8 bytes, while many emoji occupy four. A grapheme cluster is what a reader may perceive as one character; it can contain several scalar values, such as a base letter followed by a combining accent. Lexers generally reason about scalar values and bytes, not grapheme clusters.

A byte offset is a count of bytes from a chosen beginning. It is not a Unicode scalar index and certainly not a screen column. This distinction is essential because rustc spans are fundamentally byte-position based. For aé, byte offsets are 0 for a, 1 for é, and 3 for the end. Offset 2 lies inside the encoding of é and is not a valid Rust string boundary.

Rust's str type guarantees valid UTF-8. Slicing a str at a non-boundary panics, so a lexer must obtain boundaries from UTF-8-aware iteration or explicitly validate them. Byte scanning is still valuable for ASCII punctuation because every ASCII byte is self-delimiting and cannot appear as a continuation byte inside another scalar. Production lexers combine fast byte checks with careful Unicode decoding.

Invalid UTF-8 must be diagnosed while loading or decoding source, before ordinary token logic assumes &str validity. If a diagnostic points one byte into a multibyte scalar, investigate offset arithmetic and source decoding before grammar code. If columns drift only after non-ASCII text, look for code confusing bytes, scalar values, UTF-16 code units used by some editors, and rendered columns affected by tabs.

3. Unicode identifiers and normalization#

An identifier names something, as in a variable named total. Rust permits many Unicode characters in identifiers according to language-defined Unicode identifier classes. The first character and continuation characters have different rules; digits, for example, may continue many identifiers but cannot normally begin one. The underscore has special language treatment.

Normalization converts equivalent Unicode spellings into a chosen form. Some visible text can be represented as one precomposed scalar or as multiple combining scalars. The Rust language's identifier rules include normalization behavior; a home-grown lexer should not infer it from visual appearance. Use the Unicode tables and normalization policy required by the language version being implemented.

Keywords such as fn are token-shaped like identifiers but have grammatical significance. Raw identifiers, written with an r# prefix such as r#type, allow many keyword spellings to be used as names. They are not raw strings. An edition is a language compatibility mode selected for a crate. Some words become keywords only in particular editions, so the low-level lexer can preserve neutral spelling while a higher layer interprets it using the active edition.

That separation is a design tradeoff. Classifying every keyword immediately can simplify a tiny parser, but couples reusable tokenization to edition policy. Deferring interpretation preserves source fidelity and lets macro tokens cross edition boundaries with contextual rules, at the cost of later checks.

Confusable characters are distinct Unicode characters that look alike. The compiler may lint suspicious names, but a lexer must not silently merge arbitrary lookalikes. If two visibly similar names unexpectedly differ, inspect scalar values, normalization, and hygiene context. If every non-ASCII name is rejected, inspect identifier classification. If only an edition keyword fails, inspect high-level keyword and edition handling rather than UTF-8 decoding.

4. Source files, SourceMap, and Span#

A source file is rustc's record of source text plus metadata such as its name, location range, and line starts. A SourceMap manages source files and translates compiler byte positions into user-facing file, line, and column information. It also supports snippets when source text is available. Generated input and remapped paths complicate that apparently simple task.

A Span describes a half-open source range: a low position is included and a high position is excluded. Half-open ranges compose well: adjacent ranges [a, b) and [b, c) neither overlap nor leave a gap. In rustc a span carries more than two offsets; hygiene information connects generated syntax to expansion history and name-resolution context.

Compiler positions are commonly global within the source map, while offsets in a single file are relative to that file's start. Never add two global positions or subtract positions from unrelated files without checking the contract. Overflow, ordering, and file containment are basic invariants. A zero-width span can identify an insertion point, such as where a missing semicolon belongs.

The source map can map a span to source text, but not every span has recoverable original text. Macro-generated tokens may have synthetic or inherited locations. Source may have been unavailable, remapped, or composed from multiple origins. Diagnostics should degrade gracefully instead of unwrapping a missing snippet.

Line and column are presentation products, not the canonical identity of a span. Tabs, multibyte text, and editor protocols make columns policy-sensitive. Store byte ranges internally and translate at the edge. If a label highlights the following token, check whether the high endpoint was accidentally treated as inclusive. If diagnostics jump files after concatenated loading, check global-to-local conversion. If generated code resolves names incorrectly despite plausible highlighting, inspect the span's syntax context rather than only its coordinates.

5. Lexical analysis in rustc_lexer#

rustc_lexer is rustc's low-level, hand-written lexical layer. It accepts text and breaks it into chunks. Its token result fundamentally says what kind of chunk was seen and how many bytes it occupied; the caller can recover the spelling by slicing the input at accumulated boundaries. This keeps the crate relatively independent of the compiler's source map, symbol interner, and diagnostic machinery.

A token kind is an enumeration variant classifying a token. Representative categories include whitespace, line and block comments, identifiers and raw identifiers, lifetimes, literals, delimiters, individual punctuation characters, unknown input, and end of input. Exact variants and attached fields are version-sensitive. The nightly API documentation should be checked against the checkout under study.

The cursor is mutable scanning state containing the remaining input and enough lookahead to classify the next token. Lookahead examines future input without committing to consumption. A hand-written lexer makes contextual rules such as nested block comments and raw-string terminators explicit and can be tuned for common ASCII input. A generated finite-state machine is an alternative: it can derive efficient recognizers from regular descriptions, but custom diagnostics and Rust's literal edge cases still require surrounding logic.

The critical progress invariant is that every non-end token consumes at least one byte. Unknown input must therefore become an error-bearing token that advances, rather than causing an infinite loop. Token lengths must end on UTF-8 boundaries. The sum of lengths must equal the input length unless an explicitly documented preprocessing step removed bytes.

Low-level error data is intentionally structural rather than a polished diagnostic. For example, a token can record that a block comment was unterminated or a raw string lacked a terminator. The high-level layer knows spans, editions, symbols, and diagnostic style, so it turns that data into a useful message. This division makes the scanner reusable and testable while avoiding a dependency cycle.

6. Trivia, comments, and source fidelity#

Trivia is lexical material that usually does not affect grammar, principally whitespace and comments. Discarding trivia makes parsing simpler, but retaining its boundaries matters for formatters, IDEs, documentation comments, diagnostics, and faithful token reconstruction. Rustc's layers choose what to preserve and what to transform according to downstream needs.

A line comment begins with two slashes and runs to a line ending or end of input. A block comment begins with slash-star and ends with star-slash. Rust block comments nest, so a depth counter is required. Increment depth at each opener, decrement at each closer, and finish at zero. End of file with positive depth is an unterminated-comment error whose span should include the opening site.

Documentation comments are comments with language meaning. The high-level front end can convert them into attribute-shaped syntax so later stages share attribute machinery. That conversion must preserve enough span information for diagnostics and documentation tools. Treating every comment as disposable would lose program meaning.

Whitespace includes more than a literal space. Line ending handling affects line maps, and a byte-order mark or shebang-like beginning may receive special policy. Do not copy such rules from memory; inspect the targeted lexer and Reference.

Source fidelity means retaining distinctions needed to reproduce or correctly interpret input. Macros heighten this requirement because their matchers can care about token boundaries and delimiters even when the ordinary parser would discard surface detail. On the other hand, preserving every byte in every AST node would waste memory and complicate transformations. Token streams provide a fidelity-oriented representation while AST nodes provide a grammar-oriented representation.

When doc comments disappear, inspect trivia-to-attribute conversion. When nested comments stop early, inspect depth handling and overlapping two-byte lookahead. When a formatter changes macro behavior, inspect token spacing, delimiter, and trivia preservation rather than expression parsing.

7. Literals and raw strings#

A literal is source syntax that directly denotes a value-like piece of data, such as an integer, character, byte, or string. Lexing generally finds the literal's extent and coarse kind; later stages validate detailed meaning, suffixes, escapes, and type constraints. Splitting responsibilities avoids forcing the low-level scanner to understand the type system.

Numeric literals include bases, separators, fractions, exponents, and optional suffixes. A dot creates an instructive ambiguity: in 1.foo, the parser may need an integer followed by member-access punctuation rather than a floating literal. The lexer follows precise lexical rules and records sufficient structure; parser and literal validation complete interpretation.

An escape is a sequence beginning with a backslash that denotes a character difficult to write directly. Quoted strings must account for escapes so an escaped quote does not terminate the token. Character and lifetime syntax both begin with an apostrophe, requiring careful classification and recoverable errors.

A raw string begins with an r, zero or more hash signs, and a quote. It ends at a quote followed by exactly the required number of hash signs. Its body does not process ordinary backslash escapes. The scanner should count opening hashes, search candidate quotes, and compare following hashes without quadratic rescanning. An unterminated raw string should carry data such as the expected hash count or a useful candidate mismatch when the current API provides it.

Raw byte strings add another prefix and semantic restrictions. Prefix recognition is edition-sensitive in some cases because reserving new literal prefixes can change how old source tokenizes or diagnoses. The low-level layer can report an unknown prefix while the high-level layer selects an edition-specific explanation.

If a following token is swallowed after a string, inspect escape and terminator consumption. If performance collapses on many quotes and hashes, inspect repeated searches. If a valid spelling changes behavior by edition, inspect prefix reservation and high-level policy.

8. Punctuation, delimiters, and joint spacing#

Punctuation consists of symbolic tokens such as plus, minus, equals, colon, and greater-than. The low-level lexer commonly emits individual punctuation characters. The high-level token layer can combine or interpret sequences such as ==, =>, ::, ..=, and >> according to parser needs.

Spacing is token metadata describing whether punctuation is joint with the following token. Joint punctuation has no separating trivia and may form a multi-character operator. Alone punctuation does not join. This distinction matters to macros because - > and -> are not interchangeable token spellings, even if a naive token list contains minus and greater-than in both cases.

A delimiter is a matched parenthesis, bracket, or brace. Delimiters impose nested structure and are represented specially in token trees. Unmatched delimiters deserve targeted diagnostics because blindly continuing at the same nesting level causes cascades. A cascade is a chain of secondary errors caused by one primary mistake.

Why not lex every operator as one token? Doing so can simplify expression parsing but makes angle brackets and shift-like sequences harder in generic syntax, and it can lose the faithful punctuation model exposed to macros. Single-character tokens plus jointness allow later consumers to split or glue under explicit rules. The tradeoff is more complex cursor logic in the parser.

An invariant is that spacing reflects the original or intentionally synthesized adjacency. Macro transcription that manufactures punctuation must assign spacing deliberately. If macro_rules! matches a symbol sequence in handwritten code but not generated code, compare token kinds and spacing. If nested generic closers are consumed as a shift operator, inspect parser token splitting and context. If a delimiter error points far away, inspect delimiter-stack recovery first.

9. From rustc_lexer to rustc_parse#

The high-level lexer in rustc_parse integrates low-level chunks with compiler services. It accumulates byte positions into Span values, interns identifier spellings, converts token categories into rustc_ast token forms, handles comments and diagnostics, and applies policy requiring session or edition information.

Interning stores one shared immutable representation for each distinct string and refers to it with a compact symbol. It speeds equality comparisons and reduces repeated allocation for common names. Interning does not make two hygienically different identifiers equivalent: spelling and syntax context are separate dimensions.

A parse session is shared context for parsing, including diagnostics and the source map. Binding lexer and parser lifetimes to this session lets them borrow shared data instead of copying it. The architectural cost is lifetime complexity and reduced isolation; the benefit is memory and consistency.

The adapter computes each token's low and high position from the current offset and low-level length. It must check conversion widths and preserve UTF-8 boundaries. Low-level error flags become labels, notes, and suggestions with source context. Edition-sensitive reserved prefixes and keyword behavior also belong at this policy-aware boundary or parser layer.

Do not treat rustc_lexer::TokenKind and rustc_ast::token::TokenKind as interchangeable merely because their names resemble one another. They serve different abstraction levels. Likewise, a low-level lexer success does not imply a valid Rust literal; it may only mean the scanner found its end.

If token lengths are right but all labels are shifted by a constant, suspect source-file base arithmetic in the adapter. If only repeated identifiers behave strangely, inspect interning and symbol slicing. If the diagnostic lacks a useful span despite low-level error data, inspect conversion into the high-level diagnostic rather than the scanner recognizer.

10. A small production-minded lexer: model#

The following stable-Rust lexer is intentionally a language subset, not a clone of rustc_lexer. It recognizes identifiers, decimal integers, selected punctuation, whitespace, line comments, nested block comments, and simple quoted strings. Its production-minded properties are explicit byte spans, guaranteed progress, UTF-8-safe decoding, errors as data, and recovery after bad input.

#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub struct Span {
    pub lo: usize,
    pub hi: usize,
}

#[derive(Clone, Debug, PartialEq, Eq)]
pub enum Kind {
    Ident,
    Integer,
    String,
    Whitespace,
    LineComment,
    BlockComment,
    Punct(char),
    Error(ErrorKind),
    Eof,
}

#[derive(Clone, Debug, PartialEq, Eq)]
pub enum ErrorKind {
    Unexpected(char),
    UnterminatedString,
    UnterminatedBlockComment,
}

#[derive(Clone, Debug, PartialEq, Eq)]
pub struct Token {
    pub kind: Kind,
    pub span: Span,
}

pub struct Lexer<'a> {
    src: &'a str,
    pos: usize,
}

The span uses byte offsets. usize is convenient for slicing one in-memory string, whereas rustc uses compiler-specific compact position types and a source map. The upper endpoint is exclusive. Errors are token kinds so callers can keep parsing and can display diagnostics separately.

The source is a valid &str, so decoding starts at valid boundaries if every update comes from decoded character lengths or ASCII bytes known to be boundaries. The cursor pos always names the next unread byte. The central invariant is pos <= src.len() and src.is_char_boundary(pos).

This example approximates Unicode identifiers with alphabetic and alphanumeric standard-library predicates. That is not the complete Rust identifier specification and is labeled as such. A real implementation should use language-approved Unicode XID tables and normalization. The point is safe mechanics, not standards compliance.

11. A small production-minded lexer: cursor and scanning#

impl<'a> Lexer<'a> {
    pub fn new(src: &'a str) -> Self {
        Self { src, pos: 0 }
    }

    fn rest(&self) -> &'a str {
        &self.src[self.pos..]
    }

    fn peek(&self) -> Option<char> {
        self.rest().chars().next()
    }

    fn starts_with(&self, text: &str) -> bool {
        self.rest().starts_with(text)
    }

    fn bump(&mut self) -> Option<char> {
        let ch = self.peek()?;
        self.pos += ch.len_utf8();
        Some(ch)
    }

    fn take_while(&mut self, mut predicate: impl FnMut(char) -> bool) {
        while let Some(ch) = self.peek() {
            if !predicate(ch) {
                break;
            }
            self.bump();
        }
    }

    fn token(&self, start: usize, kind: Kind) -> Token {
        Token { kind, span: Span { lo: start, hi: self.pos } }
    }

    pub fn next_token(&mut self) -> Token {
        let start = self.pos;
        let Some(first) = self.bump() else {
            return self.token(start, Kind::Eof);
        };

        if first.is_whitespace() {
            self.take_while(char::is_whitespace);
            return self.token(start, Kind::Whitespace);
        }

        if first == '/' && self.starts_with("/") {
            self.bump();
            self.take_while(|ch| ch != '\n' && ch != '\r');
            return self.token(start, Kind::LineComment);
        }

        if first == '/' && self.starts_with("*") {
            self.bump();
            return self.block_comment(start);
        }

        if first == '"' {
            return self.string(start);
        }

        if first == '_' || first.is_alphabetic() {
            self.take_while(|ch| ch == '_' || ch.is_alphanumeric());
            return self.token(start, Kind::Ident);
        }

        if first.is_ascii_digit() {
            self.take_while(|ch| ch.is_ascii_digit() || ch == '_');
            return self.token(start, Kind::Integer);
        }

        if "+-*/(){}[],;=".contains(first) {
            return self.token(start, Kind::Punct(first));
        }

        self.token(start, Kind::Error(ErrorKind::Unexpected(first)))
    }
}

Lookahead never changes pos; bump commits one scalar. The slash was already consumed when comment lookahead occurs, so starts_with("/") tests for the second slash. Every non-end branch has consumed first, guaranteeing progress even on an unexpected character.

The integer rule deliberately does not validate separator placement. That illustrates staged validation: lex a broad candidate, then issue a precise literal error later. The punctuation list emits one character at a time; a separate adapter could calculate joint spacing.

12. A small production-minded lexer: errors and tests#

impl Lexer<'_> {
    fn block_comment(&mut self, start: usize) -> Token {
        let mut depth = 1usize;
        while self.peek().is_some() {
            if self.starts_with("/*") {
                self.pos += 2;
                depth += 1;
            } else if self.starts_with("*/") {
                self.pos += 2;
                depth -= 1;
                if depth == 0 {
                    return self.token(start, Kind::BlockComment);
                }
            } else {
                self.bump();
            }
        }
        self.token(start, Kind::Error(ErrorKind::UnterminatedBlockComment))
    }

    fn string(&mut self, start: usize) -> Token {
        let mut escaped = false;
        while let Some(ch) = self.bump() {
            if escaped {
                escaped = false;
            } else if ch == '\\' {
                escaped = true;
            } else if ch == '"' {
                return self.token(start, Kind::String);
            } else if ch == '\n' || ch == '\r' {
                return self.token(start, Kind::Error(ErrorKind::UnterminatedString));
            }
        }
        self.token(start, Kind::Error(ErrorKind::UnterminatedString))
    }
}

#[cfg(test)]
mod lexer_tests {
    use super::*;

    fn all(src: &str) -> Vec<Token> {
        let mut lexer = Lexer::new(src);
        let mut result = Vec::new();
        loop {
            let token = lexer.next_token();
            let done = token.kind == Kind::Eof;
            result.push(token);
            if done { break; }
        }
        result
    }

    #[test]
    fn unicode_spans_are_bytes() {
        let tokens = all("λ + 2");
        assert_eq!(tokens[0].span, Span { lo: 0, hi: 2 });
        assert_eq!(&"λ + 2"[tokens[0].span.lo..tokens[0].span.hi], "λ");
        assert_eq!(tokens.last().unwrap().span.hi, "λ + 2".len());
    }

    #[test]
    fn nested_comment_and_recovery() {
        let tokens = all("/* a /* b */ c */x");
        assert_eq!(tokens[0].kind, Kind::BlockComment);
        assert_eq!(tokens[1].kind, Kind::Ident);
    }

    #[test]
    fn bad_character_advances() {
        let tokens = all("🙂x");
        assert!(matches!(tokens[0].kind, Kind::Error(ErrorKind::Unexpected('🙂'))));
        assert_eq!(tokens[0].span, Span { lo: 0, hi: 4 });
        assert_eq!(tokens[1].kind, Kind::Ident);
    }

    #[test]
    fn reports_unterminated_constructs() {
        assert!(matches!(all("/*")[0].kind,
            Kind::Error(ErrorKind::UnterminatedBlockComment)));
        assert!(matches!(all("\"abc")[0].kind,
            Kind::Error(ErrorKind::UnterminatedString)));
    }
}

Production tests should also fuzz arbitrary valid UTF-8, assert monotonic nonoverlapping spans, assert exact source coverage, and impose time and memory limits. A fuzzer is a tool that generates many unusual inputs to discover crashes or invariant violations. Property tests should ensure no panic and eventual EOF. Differential tests can compare intended shared behavior with rustc, while allowing documented subset differences.

13. Token streams, token trees, and delimiters#

A token stream is an ordered persistent-style collection of tokens and grouped token trees. A token tree is either a token or a delimited subtree. For f!(a + (b)), the invocation body has tokens a and +, then a parenthesis-delimited subtree containing b. The tree makes balanced nesting explicit.

TokenStream
├── Ident("a")
├── Punct('+', Alone)
└── Delimited(Parenthesis)
    └── Ident("b")

Delimiter metadata includes the kind and spans for opening and closing boundaries. An invisible or synthetic delimiter may appear in internal representations; exact variants are internal and version-sensitive. Consumers must not assume every group came from typed punctuation.

Macros need token fidelity because they operate before or instead of ordinary AST interpretation. A matcher can distinguish an identifier fragment from arbitrary token trees and repetition separators from body tokens. Procedural macros receive token streams through the stable proc_macro API, not rustc AST nodes. If the compiler eagerly converted input only to AST, it would erase punctuation spacing, delimiters, and spellings that macro authors can observe.

Token trees also limit accidental delimiter confusion. A macro matcher operating inside one group sees a bounded stream rather than manually counting every parenthesis. Malformed delimiters are diagnosed while constructing the grouped stream, enabling recovery before parsing expressions.

AST nodes may retain token data where later macro or attribute processing needs it, but retaining all forms has memory cost. Rustc uses internal sharing and lazy strategies that change over time. The conceptual invariant is that syntax promised to macro consumers can be reconstructed with correct token semantics.

14. Recursive descent and parser state#

Recursive-descent parsing represents grammar rules with mutually recursive functions. A function parsing an item calls one parsing a type, which may call one parsing a path, and so forth. It is top-down because it begins with a broad expected construct and asks what subconstruct comes next. Rustc's normal parser in rustc_parse follows this style.

The Parser owns a cursor over tokens, a current token, lookahead facilities, edition and restriction state, and access to the parse session. Lookahead answers questions about future tokens without permanently consuming them. bump-like operations advance. eat-like operations conditionally consume. expect-like operations consume an expected token or emit a diagnostic. Exact methods and fields are nightly internals.

The cursor invariant is that consumed input never silently reappears. Speculative parsing must either commit or restore all relevant state, including diagnostics and side tables. Unlimited backtracking is expensive and can make malformed input pathological. Rust's parser prefers bounded lookahead, grammar knowledge, and targeted speculation.

Conceptual parse entry points accept a source file or token stream, create a parser tied to a ParseSess, and invoke a parse_* routine for the expected category. Crate and module entry points build a Crate root or module contents. Fragment entry points parse expressions, types, patterns, statements, items, or other macro fragments. Parsing a fragment has different termination rules from parsing a complete file.

If the parser loops, inspect whether an error branch advances. If it skips a valid construct after recovery, inspect synchronization boundaries. If behavior changes depending on an unrelated failed speculation, inspect restoration of cursor and diagnostic state.

15. Precedence, associativity, and Pratt ideas#

Precedence decides which operator binds more tightly. Multiplication has higher precedence than addition, so 1 + 2 * 3 means 1 + (2 * 3). Associativity chooses grouping among operators at the same level. Left associativity makes a - b - c mean (a - b) - c.

A grammar can encode each precedence level with a separate recursive-descent function. That is readable but repetitive. A Pratt parser is an expression-parsing technique that assigns operators binding powers and repeatedly consumes operators strong enough for the current context. Precedence climbing is a closely related formulation. Rustc's exact expression machinery should be read in the targeted source, but these concepts explain its operator handling.

Prefix operators occur before an operand, postfix operators after it, and infix operators between two operands. Calls and indexing behave like high-precedence postfix constructs. Ranges and assignments have special associativity and admissibility rules. Some tokens that resemble operators participate in type or generic syntax, requiring parser context.

The key invariant is that each operator and operand is consumed exactly once into one intended subtree. An off-by-one binding power changes association without necessarily producing an error. Tests must inspect AST shape, not merely parse success.

Alternatives include parser generators and generalized parsers. They can make formal ambiguity handling clearer, but hand-written recursive descent gives rustc precise recovery, edition checks, and diagnostics near language-specific decisions. The tradeoff is that grammar structure is distributed through code and must be maintained carefully.

16. AST nodes and NodeId#

An AST node is a typed record for a syntactic construct. An expression node may contain a literal, binary operation, call, block, or many other variants. Items represent declarations such as functions and structs. Patterns describe how values are matched or bound. Each node carries a span connecting it to source or generated origin.

NodeId is a compiler-internal numeric identity unique for an AST node within a crate. Macro expansion and early name-resolution work use these identities. They are not stable source identities: inserting one early node can renumber many later nodes. Consequently they are unsuitable as durable incremental-compilation keys or external references.

During initial parsing and expansion, nodes may temporarily have dummy or placeholder IDs. Integration assigns IDs as required by the current expansion pipeline. The precise visitation and assignment APIs evolve. The invariant is that stages requiring assigned IDs do not receive accidental duplicates or unresolved placeholders except where explicitly allowed.

AST is called abstract because it records semantic syntax structure rather than every concrete formatting choice. Token streams remain available for syntax requiring fidelity. This duality explains why a compiler front end cannot be understood as one universal tree.

An error node is an AST node marking syntax already diagnosed as invalid. It lets parents remain structurally complete and prevents consumers from treating malformed pieces as valid. Later stages should recognize the error marker and avoid redundant messages. If an AST dump has the wrong operator tree, parsing is implicated. If its shape is right but IDs collide after expansion, inspect integration and ID assignment. If a later pass crashes on malformed input, inspect whether it respected error nodes.

17. A small expression parser: data and diagnostics#

The next conceptual implementation parses integers, names, parentheses, prefix minus, and four binary operators. It is complete for that subset, not for Rust. Calling it a Rust parser would be misleading because Rust expressions include blocks, paths, calls, control flow, macros, and many contextual restrictions.

#[derive(Clone, Debug, PartialEq, Eq)]
pub enum Expr {
    Integer { text: String, span: Span },
    Name { text: String, span: Span },
    Prefix { op: char, rhs: Box<Expr>, span: Span },
    Binary { lhs: Box<Expr>, op: char, rhs: Box<Expr>, span: Span },
    Error { span: Span },
}

impl Expr {
    fn span(&self) -> Span {
        match self {
            Expr::Integer { span, .. }
            | Expr::Name { span, .. }
            | Expr::Prefix { span, .. }
            | Expr::Binary { span, .. }
            | Expr::Error { span } => *span,
        }
    }
}

#[derive(Clone, Debug, PartialEq, Eq)]
pub struct Diagnostic {
    pub message: String,
    pub primary: Span,
    pub label: String,
}

pub struct Parser<'a> {
    src: &'a str,
    tokens: Vec<Token>,
    cursor: usize,
    diagnostics: Vec<Diagnostic>,
}

A diagnostic has a message, primary byte span, and label explaining that exact region. In a real compiler it may include secondary labels, notes, error codes, and machine-applicable suggestions. A label fragment such as “expected an expression” describes a selected span; a complete diagnostic should read naturally with its main message. Do not concatenate sentence fragments into grammar accidents.

The parser filters trivia before constructing this token vector. Keeping trivia and teaching lookahead to skip it is an alternative that better preserves positions but complicates every query. Because each token already has a source span, filtering does not change offsets.

18. A small expression parser: precedence and recovery#

impl<'a> Parser<'a> {
    pub fn from_tokens(src: &'a str, tokens: Vec<Token>) -> Self {
        let tokens = tokens.into_iter()
            .filter(|t| !matches!(t.kind,
                Kind::Whitespace | Kind::LineComment | Kind::BlockComment))
            .collect();
        Self { src, tokens, cursor: 0, diagnostics: Vec::new() }
    }

    fn current(&self) -> &Token {
        &self.tokens[self.cursor.min(self.tokens.len() - 1)]
    }

    fn bump(&mut self) -> Token {
        let token = self.current().clone();
        if token.kind != Kind::Eof { self.cursor += 1; }
        token
    }

    fn punct(&self, wanted: char) -> bool {
        self.current().kind == Kind::Punct(wanted)
    }

    fn binding_power(op: char) -> Option<(u8, u8)> {
        match op {
            '+' | '-' => Some((1, 2)),
            '*' | '/' => Some((3, 4)),
            _ => None,
        }
    }

    pub fn parse_complete(mut self) -> (Expr, Vec<Diagnostic>) {
        let expression = self.parse_bp(0);
        if self.current().kind != Kind::Eof {
            self.diagnostics.push(Diagnostic {
                message: "unexpected input after expression".into(),
                primary: self.current().span,
                label: "this token does not continue the expression".into(),
            });
            while self.current().kind != Kind::Eof { self.bump(); }
        }
        (expression, self.diagnostics)
    }

    fn parse_bp(&mut self, minimum: u8) -> Expr {
        let mut lhs = self.parse_prefix();
        loop {
            let Kind::Punct(op) = self.current().kind else { break };
            let Some((left, right)) = Self::binding_power(op) else { break };
            if left < minimum { break; }
            self.bump();
            let rhs = self.parse_bp(right);
            let span = Span { lo: lhs.span().lo, hi: rhs.span().hi };
            lhs = Expr::Binary { lhs: Box::new(lhs), op,
                rhs: Box::new(rhs), span };
        }
        lhs
    }
}

Different left and right binding powers produce left association. Recovery must still make progress when an operand is missing; parse_prefix handles that next. parse_complete distinguishes a complete-code entry point from a fragment parser that may legally stop before a caller-owned separator.

19. A small expression parser: atoms and tests#

impl Parser<'_> {
    fn parse_prefix(&mut self) -> Expr {
        if self.punct('-') {
            let start = self.bump().span.lo;
            let rhs = self.parse_bp(5);
            let span = Span { lo: start, hi: rhs.span().hi };
            return Expr::Prefix { op: '-', rhs: Box::new(rhs), span };
        }

        let token = self.bump();
        match token.kind {
            Kind::Integer => Expr::Integer {
                text: self.src[token.span.lo..token.span.hi].into(),
                span: token.span,
            },
            Kind::Ident => Expr::Name {
                text: self.src[token.span.lo..token.span.hi].into(),
                span: token.span,
            },
            Kind::Punct('(') => {
                let inner = self.parse_bp(0);
                if self.punct(')') {
                    self.bump();
                } else {
                    self.diagnostics.push(Diagnostic {
                        message: "unclosed parenthesized expression".into(),
                        primary: token.span,
                        label: "opening parenthesis has no closing parenthesis".into(),
                    });
                }
                inner
            }
            _ => {
                self.diagnostics.push(Diagnostic {
                    message: "expected an expression".into(),
                    primary: token.span,
                    label: "an integer, name, prefix minus, or parenthesis was expected".into(),
                });
                Expr::Error { span: token.span }
            }
        }
    }
}

#[cfg(test)]
mod parser_tests {
    use super::*;

    fn parse(src: &str) -> (Expr, Vec<Diagnostic>) {
        let mut lexer = Lexer::new(src);
        let mut tokens = Vec::new();
        loop {
            let token = lexer.next_token();
            let end = token.kind == Kind::Eof;
            tokens.push(token);
            if end { break; }
        }
        Parser::from_tokens(src, tokens).parse_complete()
    }

    #[test]
    fn multiplication_binds_tighter() {
        let (tree, errors) = parse("1 + 2 * 3");
        assert!(errors.is_empty());
        let Expr::Binary { op: '+', rhs, .. } = tree else { panic!() };
        assert!(matches!(*rhs, Expr::Binary { op: '*', .. }));
    }

    #[test]
    fn subtraction_associates_left() {
        let (tree, _) = parse("a-b-c");
        let Expr::Binary { lhs, op: '-', .. } = tree else { panic!() };
        assert!(matches!(*lhs, Expr::Binary { op: '-', .. }));
    }

    #[test]
    fn missing_operand_is_one_local_error() {
        let (tree, errors) = parse("1 + )");
        assert!(matches!(tree, Expr::Binary { .. }));
        assert_eq!(errors.len(), 1);
        assert_eq!(errors[0].primary, Span { lo: 4, hi: 5 });
    }
}

This recovery is intentionally modest. For lists and statements, synchronize at commas, semicolons, or a delimiter at the current nesting depth. Never consume a caller-owned closing delimiter casually. Record one error node and suppress derivative complaints until a trustworthy boundary.

20. Grammar ambiguity and editions#

A grammar is a set of rules describing valid token arrangements. An ambiguity occurs when the same input admits multiple plausible structures or when bounded lookahead cannot immediately distinguish them. Rust has contextual ambiguities involving angle brackets, paths, closures, ranges, statements, patterns, and generic arguments.

The parser resolves many cases with context and precedence rather than global backtracking. For example, a token sequence legal after an expression may differ from one legal in a type. Restrictions carried in parser state can forbid a construct temporarily to disambiguate a larger rule. Such flags are powerful but dangerous: failure to restore one after nested parsing creates distant bugs.

An edition allows Rust to make selected compatibility changes without breaking every existing crate simultaneously. The active edition affects keywords and some parsing behavior. Tokens originating in macros can carry edition implications from their defining or call-site context according to language rules; simply using the final crate's edition everywhere is not a safe model.

Parsing a superset means accepting some structures that are not ultimately legal Rust, building a useful tree, and issuing a targeted error in validation. This often produces better messages than forcing an awkward grammar rejection. The tradeoff is a crucial contract: validation must reject every accepted-but-illegal form before later stages assume validity.

When a syntax error appears only inside a macro, compare fragment context and token edition. When a parser accepts invalid syntax without any error, search both parser and AST validation. When a distant construct changes after parsing a closure or generic, inspect contextual restriction restoration and punctuation splitting.

21. Error recovery, diagnostics, and cascade control#

Error recovery is the strategy for continuing after malformed input. The objective is not to guess the programmer's entire intent; it is to preserve enough structure to report independent mistakes without inventing dozens of consequences.

Insertion recovery pretends a likely missing token existed, often using a zero-width span. Deletion recovery consumes an unexpected token. Synchronization skips until a token likely to begin or end a construct. Delimiter-aware recovery tracks nesting so a comma inside an inner call does not synchronize an outer item list.

A diagnostic should identify the primary error, label relevant source, and offer a suggestion only when the replacement is reliable. Machine applicability communicates whether tooling may safely apply a suggestion. Generated spans and unavailable snippets require conservative wording.

Error nodes and an “already emitted” marker prevent cascades. Once an operand is erroneous, type-like or precedence diagnostics derived solely from it should usually be suppressed. But suppression must not hide a later independent error after synchronization. This balance is tested with malformed multi-error fixtures.

Never recover without advancing unless the caller demonstrably consumes an inserted token state. Never skip across an unmatched closing delimiter owned by a parent parser. Never turn malformed text into a valid-looking node with no attached error.

Bug clues are characteristic. A hang means progress failure. Hundreds of errors after one missing brace indicate poor synchronization. A missing second independent diagnostic means suppression lasted too long. A suggestion replacing half of a Unicode scalar means broken byte boundaries. A panic only on erroneous source means a later pass trusted an invariant that recovery did not restore.

22. AST validation and feature gates#

AST validation checks structural rules that are inconvenient or undesirable to encode directly in parsing. Examples vary by compiler version but include restrictions on where particular syntax forms may occur and relationships among fields of a declaration. Validation happens with broader context than a token-level parser function may possess.

A feature gate controls access to an unstable language feature. The parser may recognize gated syntax so users receive an explicit “unstable feature” diagnostic rather than a misleading syntax error. This is another form of parsing a superset. Gate checking records use sites and consults enabled features under the compiler's stability policy.

Recognition, validation, and gating answer different questions. Recognition asks what structure the tokens resemble. Validation asks whether that structure is legal in its location. Gating asks whether this compilation is permitted to use a known but unstable structure. Mixing them can produce incorrect recovery or accidentally stabilize syntax.

Macros complicate timing because expansion creates new AST nodes and attributes. Checks must run at a point where their required context exists and on generated syntax where appropriate, while respecting allowances intended for compiler-generated constructs. The exact pass ordering is internal and should be verified in the 1.97.1 source.

The invariant at the handoff to lowering is stronger than “the parser returned.” Illegal superset forms must have diagnostics or error markers; gated features must be checked; expanded syntax must satisfy structural assumptions. If invalid handwritten and generated forms differ, inspect validation coverage after integration. If stable code reports a feature gate after an edition change, inspect gate-site spans and edition policy.

23. macro_rules matching from first principles#

macro_rules! defines declarative macros: the author states token patterns and corresponding output templates rather than writing an arbitrary Rust function. Macros By Example, abbreviated MBE, is rustc's common name for this system.

A metavariable is a named capture beginning with a dollar sign. A fragment specifier after a colon states the grammatical category to capture, such as ident, expr, ty, pat, item, or tt. The exact set and edition behavior are defined by the Reference. An ident captures identifier-like input; an expr asks the ordinary Rust parser to recognize an expression fragment; a tt captures one token tree.

macro_rules! pair_sum {
    ($left:expr, $right:expr) => {
        ($left) + ($right)
    };
}

The matcher is the left side and the transcriber is the right side. Matching binds left and right to captured token sequences. Transcription copies template tokens and substitutes captures. The parentheses in output protect expression grouping.

The macro parser is not merely textual search. At a fragment metavariable it invokes normal parser logic appropriate to that fragment. At literal matcher tokens it compares token structure. Rules are tried in declaration order, and the first matching rule is used. Local ambiguity while matching one rule is rejected rather than resolved by arbitrary distant guessing.

A small matcher can honestly support literal tokens plus ident and one-token-tree captures. It would maintain a set of candidate (pattern_position, input_position, bindings) states, advance matching literals, and branch at repetitions. It must reject conflicting bindings and require full input consumption. That sketch does not implement MBE: complete fragment parsing, repetition nesting, follow-set restrictions, edition details, and diagnostics are substantial.

24. Repetition, transcription, and ambiguity#

Repetition syntax applies a matcher or transcriber sequence zero or more times with *, one or more times with +, or optionally with ? where allowed. A separator such as a comma may appear between repetitions.

macro_rules! make_tuple {
    ($($value:expr),* $(,)?) => {
        ($($value),*)
    };
}

The first repetition captures a sequence of expression fragments separated by commas. The second accepts an optional trailing comma. Transcription repeats each captured value at the corresponding nesting depth.

Repetition nesting creates a shape, not a flat map. A metavariable captured inside two nested repetitions has two indices. Transcription must use it at a compatible depth, and metavariables driving the same repetition must have compatible lengths. Otherwise there is no unambiguous number of output copies.

Local ambiguity arises when the matcher cannot decide whether a token belongs to the current repetition or what follows without forbidden lookahead. For example, repeating identifiers followed by a final identifier has no marker identifying where repetition stops. Adding punctuation or a keyword sentinel can make the boundary explicit.

The implementation is often explained using nondeterministic finite automaton ideas. Nondeterministic means multiple candidate matching states may be considered at once; finite automaton means states advance through pattern positions based on input. Fragment captures call back into the Rust parser, making MBE richer than regular-expression matching.

Failure clues include “no rules expected this token,” local ambiguity, a fragment parse error, and repetition-depth mismatch. Inspect the matcher before generated output for the first three. Inspect transcription nesting for the fourth. Do not “fix” ambiguity by silently selecting the first path; that makes macro meaning unstable under innocuous rule changes.

25. Iterative macro expansion and AST integration#

Initial parsing encounters macro definitions, invocations, and attributes but cannot replace every invocation immediately. Definitions may be introduced by earlier expansions, imports may bring macros into scope, and generated output may contain more macros. Rustc therefore expands iteratively.

Conceptually, the initial AST contains placeholders where expansion results will belong. A placeholder is a temporary node carrying an expansion identity rather than final syntax. The expander maintains a queue or worklist of unresolved invocations. It repeatedly chooses work, resolves the macro when possible, runs its expander, parses the output into the expected fragment, and integrates resulting nodes at the placeholder.

Integration is not plain text replacement. It assigns or updates node identities, applies expansion hygiene, records expansion relationships, visits generated nodes, and discovers nested invocations for the queue. The expected fragment matters: output replacing an expression must parse as an expression, while an item-position macro can produce items.

Macro name resolution and import resolution interleave with this process. Expansion can create a use, module, or macro definition affecting later resolution. Resolution can make a queued macro expandable. The system may make partial progress and retry unresolved work. It is inaccurate to say that all ordinary name resolution happens before macros or that the complete crate is resolved first.

An eager built-in macro is an exception to ordinary lazy argument handling. Selected compiler built-ins expand some arguments before expanding the outer invocation, commonly because they need resulting values or diagnostics. Eagerness is not a general property of macro_rules! or procedural macro arguments and should not be inferred from one built-in.

Termination requires progress or a definitive unresolved error. Recursive expansion has limits to prevent unbounded resource use. If nested generated macros remain as placeholders, inspect queue discovery and integration. If a macro is spuriously unresolved until an import is reordered, inspect the expansion-resolution fixed point.

26. Hygiene, syntax contexts, and expansion IDs#

Hygiene prevents names introduced by a macro from accidentally capturing, or being captured by, unrelated names merely because their text matches. Consider a macro introducing a temporary named x into a call site that already has an x. Pure textual substitution would confuse the two.

The call site is where a macro is invoked. The definition site is where it is defined. Different names in output can resolve relative to different contexts according to macro kind and token origin. Declarative macros have mixed-site hygiene rules documented by the Reference; a slogan such as “everything uses definition site” is false.

A syntax context is metadata attached to a span that records hygiene marks or transformations. An expansion ID identifies one expansion and links it to its parent expansion, call-site span, macro definition, and related metadata. Two identifier tokens with the same interned spelling but different syntax contexts may resolve differently.

Conceptually, expansion applies a new mark to tokens according to whether they came from the definition, invocation capture, or newly produced output. Captured call-site tokens retain relevant context. Template tokens carry definition-related context. Procedural macros can construct and adjust spans through the stable API's available span operations, though stable controls are deliberately limited and evolve.

A macro backtrace is the chain of expansions leading to generated syntax. Rustc derives it from expansion and hygiene data in rustc_span. Diagnostics can show the immediate generated location and successive invocation sites. Coordinates alone cannot supply this history.

If a generated identifier resolves to the wrong same-named binding, inspect syntax context provenance. If a macro backtrace skips a nested invocation, inspect parent expansion IDs. If moving a macro definition changes call-site captures unexpectedly, compare whether tokens came from the transcriber or metavariable substitution.

27. Attributes, derives, and built-in macros#

An attribute is metadata attached to a crate or syntax node, written in outer form such as #[name] or inner form such as #![name] where grammar permits. Attributes can configure compilation, drive linting, or request macro expansion. Their paths and token arguments are preserved for interpretation.

An attribute procedural macro transforms the item it annotates. It receives two stable proc_macro::TokenStream values: attribute arguments and the annotated item. It returns tokens replacing or augmenting syntax according to the API contract. It does not receive rustc's AST.

A custom derive is requested through #[derive(TraitName)] and generates items associated with implementing behavior for the annotated type. The derive macro receives the annotated item as a token stream. Helper attributes can be declared by the derive macro so its input may carry derive-specific metadata.

Function-like procedural macros use invocation syntax resembling name!(...) and transform one input token stream into one output token stream. Declarative macro_rules! macros use a compiler matcher and transcriber instead of arbitrary host Rust code. Built-in macros are implemented by the compiler and can have privileged behavior impossible to express with stable macro APIs.

Built-in attributes likewise may be active, inert, or handled by specialized compiler phases. An inert attribute remains attached as metadata rather than transforming the item immediately. Classification and ordering details are version-sensitive.

When an attribute is reported unknown, inspect attribute path resolution and registration. When derive output is malformed, inspect emitted token streams before blaming the original struct parser. When behavior differs between a built-in and a visually similar user macro, check for compiler privileges or eager expansion rather than assuming a general language rule.

28. Procedural macro architecture and trust boundary#

A procedural macro crate is compiled for the host, meaning the machine running the compiler, because its code executes during compilation. This differs from the target, meaning the machine or environment for which the final program is being built. During cross-compilation, host and target can have different architectures and operating systems.

The stable proc_macro API exposes token streams, token trees, identifiers, punctuation, literals, groups, and spans. Rustc bridges between its internal token representation and this stable facade. The bridge protects API stability while compiler internals change. Procedural macros receive token streams, never rustc AST objects through the stable API.

The exact process boundary is an implementation detail and version-sensitive. Depending on rustc's architecture and mode, compiler and proc-macro execution may communicate through an internal bridge and a process arrangement that has changed over time. Do not build correctness or security assumptions on a specific process topology without checking that toolchain's source.

Most importantly, procedural macros are not sandboxed by the Rust language. They execute build-time code with the user's permissions and can access files, environment variables, network facilities, and other operating-system resources unless an external sandbox restricts them. A panic is caught and reported as macro failure where possible, but aborts, resource exhaustion, or unsafe behavior can still damage the build process.

Reading environment variables and files creates untracked or imperfectly tracked inputs unless the toolchain and build integration explicitly records them. Such hidden dependencies harm reproducibility, which means obtaining the same output from the same declared inputs. Macro authors should minimize external reads, expose explicit configuration, emit deterministic token order, and use supported tracking APIs when available.

Consumers should audit proc-macro dependencies as executable build dependencies. CI can apply operating-system sandboxing, network restrictions, resource limits, and locked dependency resolution. If clean and incremental builds differ, inspect environmental and filesystem inputs. If only cross-compilation fails, inspect host dependencies and accidental target assumptions.

29. Detailed trace: source to expanded AST#

Consider this conceptual crate.

#[derive(Debug, MakeAnswer)]
#[answer(label = "life")]
struct Reply;

macro_rules! wrap {
    ($value:expr) => {{
        let local = $value;
        Some(local)
    }};
}

#[trace_calls]
fn answer() -> Option<u32> {
    wrap!(Reply::answer())
}

Assume MakeAnswer is a custom derive procedural macro and trace_calls is an attribute procedural macro imported from dependencies. This trace is conceptual; their output is invented for explanation.

First, source loading validates UTF-8, creates a source file, records line starts, and assigns its global position interval. The low-level lexer emits chunk kinds and byte lengths, including comment or whitespace trivia, #, brackets, identifiers, literals, and delimiters. The high-level lexer interns names, creates spans, interprets punctuation spacing, and groups balanced delimiters into token trees.

The parser builds items for Reply, the macro_rules! definition, and answer. It records attributes with token arguments. It parses the wrap! invocation as a macro node or placeholder-bearing form rather than pretending its body is already an ordinary expression. AST nodes receive spans and IDs according to current pipeline rules.

Expansion collects invocations and assigns expansion IDs. Resolving MakeAnswer and trace_calls may require imports and macro namespace resolution. The macro_rules! wrap definition is made available under its scope rules. Work becomes expandable as resolution progresses; there is no prerequisite that all names in ordinary expressions are fully resolved.

The custom derive receives a token stream representing the Reply item, including relevant helper attributes. It might return conceptually:

impl Reply {
    fn answer() -> u32 { 42 }
}

Those output tokens cross the proc-macro bridge, receive expansion-related spans, are parsed as items, assigned integration metadata, and inserted near the derived item under derive expansion rules. The original struct remains; derive output supplements it. Malformed output would be diagnosed as macro-generated syntax with a backtrace to #[derive(...)].

The attribute macro receives its attribute arguments and the token stream for answer. It might return:

fn answer() -> Option<u32> {
    eprintln!("enter answer");
    wrap!(Reply::answer())
}

That output is parsed and integrated in place of the annotated item. Integration discovers the newly emitted eprintln! built-in invocation and the copied wrap! invocation, adding work to the queue. Their expansion IDs name the attribute expansion as parent, preserving a backtrace.

The MBE matcher sees Reply::answer() as an expr fragment by calling normal expression parsing on the invocation token stream. It binds value, transcribes the nested block, and substitutes captured tokens. The transcriber-origin identifier local receives hygiene appropriate to the macro definition, while captured Reply::answer() retains its relevant call-site context. The output parses as an expression because wrap! occupied expression position.

Integration replaces the wrap! placeholder with a block expression containing a local statement and Some call. The Some token from the definition resolves under MBE hygiene rules, not by raw spelling alone. Generated output may reveal yet more macros; eprintln! itself expands through compiler built-in machinery and can generate lower-level formatting syntax.

The queue repeats resolution, expansion, parsing, and integration until no expandable invocation remains or errors prevent completion. AST validation and feature checking inspect the resulting structures at their required points. The resulting expanded AST conceptually contains Reply, its generated implementation, and a transformed answer body with macro-produced syntax. It no longer contains successfully expanded derive, trace_calls, wrap!, or eprintln! invocations as pending placeholders.

If Reply::answer later fails semantic resolution, inspect derive output and its integration before type checking. If local collides with a caller variable, inspect hygiene. If wrap! is unresolved only after attribute output, inspect queue discovery and macro/import resolution interaction. If parsing fails at Some, inspect whether attribute output preserved delimiters and whether the MBE result was parsed under the expected expression fragment.

30. Nightly debugging views and traces#

Rustc has internal pretty-printing and tracing facilities useful for front-end investigation. Options beginning with -Z, including forms of -Zunpretty, are nightly-only, unstable, and version-sensitive. Their accepted values and output formats can change or disappear. Always run the targeted nightly compiler's rustc -Z help rather than copying a stale command.

Conceptually, unpretty views can show parsed or expanded representations. Comparing an earlier syntax-oriented view with an expansion-oriented view helps localize whether a construct was parsed incorrectly or transformed incorrectly. Pretty output is not a lossless serialization: formatting, synthetic nodes, hygiene, and token spacing may be hidden. It is evidence, not a contract.

: "Concept only: inspect this toolchain's actual unstable options first."
rustc +nightly -Z help
rustc +nightly -Zunpretty=expanded example.rs

The second spelling has commonly existed, but must still be treated as unstable and checked for the installed toolchain. Do not put it in stable user workflows.

Rustc uses structured tracing in many internal crates. Which tracing targets, environment filters, and compiler build settings produce useful events is checkout-specific. A debug-built compiler may expose expansion queue, parser, or resolver events that a distributed build omits or logs differently. Search the relevant source for tracing calls and enable narrowly scoped targets; unrestricted logs can perturb timing and overwhelm useful evidence.

Macro backtraces exposed in diagnostics are generally more durable conceptually than private trace event names. Minimize a failing source, retain exact edition and feature flags, and compare token, parsed, and expanded boundaries. If a debug dump “fixes” a bug, suspect nondeterminism, untracked proc-macro input, or timing rather than trusting the dump.

31. Stage-specific testing strategy#

Front-end tests should fail at the narrowest responsible stage. Low-level lexer unit tests feed strings and assert token kinds, byte lengths, error data, and complete coverage. Include empty input, every punctuation, nested and unterminated comments, raw strings with varying hashes, malformed literals, Unicode boundaries, and edition-neutral prefixes.

High-level lexer tests assert spans, interned spellings, doc-comment conversion, spacing, edition diagnostics, and source-map translation. Delimiter tests assert grouping and recovery. Property tests assert that concatenated token slices cover the expected bytes and that no token crosses a UTF-8 boundary.

Parser tests inspect AST shape, not only acceptance. For each precedence pair, test both orders and parenthesized overrides. Test complete entry points and fragment entry points separately. Malformed tests should assert the primary message, labels, suggestions, and bounded number of follow-on diagnostics.

Macro matcher tests isolate matching, repetition depth, separators, fragment follow sets, and ambiguity. Expansion tests combine resolver progress, placeholders, nested generated invocations, imports, and hygiene. Procedural macro fixtures test token bridge fidelity, panics, malformed output, helper attributes, host compilation, and deterministic clean rebuilds.

UI tests are compiler tests that compile source and compare rendered diagnostics with expected output. They catch spans and wording but can be brittle under intentional diagnostic improvements. AST or token assertions offer more structural precision. Use both where each contract matters.

Regression tests should be minimized while retaining the trigger. Record edition, target, features, macro crate boundaries, and whether incremental compilation is enabled. A test that fails only in the full suite may expose global interning, source-map leakage, nondeterministic work ordering, or environmental proc-macro input.

32. Common front-end bug patterns and triage#

Offset bugs appear as shifted labels, slicing panics, or failures only after Unicode. Check byte-versus-scalar arithmetic, half-open endpoints, source-file bases, and overflow conversions. Token-boundary bugs appear as glued punctuation, a swallowed suffix, or macro mismatch while ordinary parsing seems plausible. Check lookahead commitment and joint spacing.

Progress bugs hang on malformed input. Instrument cursor positions and assert that recovery iterations consume input or return. Delimiter bugs produce diagnostics at end of file or skip whole modules. Log delimiter stack transitions and distinguish caller-owned closing tokens.

Precedence bugs produce valid but wrong ASTs. Print the minimal tree and inspect binding powers and associativity. Context leaks make later syntax depend on an earlier construct. Audit every temporary parser restriction and speculative checkpoint for restoration on success and error paths.

Expansion scheduling bugs leave placeholders, report order-dependent unresolved macros, or miss generated imports. Trace the worklist, expansion IDs, integration visits, and resolver progress. Hygiene bugs affect only same-spelled names across macro boundaries. Compare syntax contexts and token provenance, not just symbol text.

Proc-macro nondeterminism appears across machines, clean builds, or environment changes. Capture host toolchain, dependency lockfile, environment, filesystem reads, locale, and output ordering. Remember that no language sandbox prevents external effects.

Cascade bugs report many symptoms after one typo. Identify the earliest diagnostic, inspect the recovery boundary, and ensure error nodes suppress dependent checks only. Feature-gate bugs either reject stable syntax or silently accept unstable syntax. Trace recognition, gate recording, and validation as separate concerns.

33. Exercises from beginner to contributor#

1. Write byte offsets under a, é, and an emoji in one string.

Explain why visual columns differ from byte positions.

1. Extend the sample lexer with raw strings.

Bound its running time on input containing thousands of quote-and-hash candidates. Return structured unterminated-string data.

1. Replace the sample identifier approximation with a crate implementing the required Unicode XID classes and normalization.

Document how its Unicode version relates to the language toolchain.

1. Add punctuation spacing to the lexer adapter.

Test -> against - >, ..= against . .=, and punctuation adjacent to comments.

1. Add calls and comma-separated arguments to the expression parser.

Recover from a missing argument without consuming the closing parenthesis owned by the call parser.

1. Add assignment as a right-associative operator.

Prove its AST shape with a = b = c and contrast subtraction.

1. Design a diagnostic for a missing closing parenthesis after Unicode input.

Provide a zero-width insertion span and explain suggestion applicability.

1. Implement the explicitly small MBE matcher described earlier: literals, ident, tt, and one unnested comma repetition.

Reject ambiguity rather than choosing the first candidate. List the missing features that prevent calling it macro_rules! compatible.

1. Draw the expansion worklist for a macro that emits a macro definition, an import, and another invocation.

Mark where resolver progress is required.

1. Construct a hygiene experiment where a macro-defined local and call-site local have identical spelling.

Predict resolution from token provenance, then confirm with a pinned toolchain.

1. Create a procedural macro that intentionally panics and one that emits malformed tokens.

Compare diagnostic spans and backtraces without assuming process isolation.

1. Audit a proc macro for undeclared environment and file reads.

Propose explicit inputs and reproducibility tests.

1. Minimize an actual rustc parser diagnostic test.

Classify each assertion as token, AST, recovery, or presentation behavior.

1. Use a matching nightly's help to find available unpretty modes.

Compare parsed and expanded views, then list fidelity omitted by both.

34. Contribution map, version guardrails, and sources#

Start lexer investigations in compiler/rustc_lexer for chunk recognition and error data. Move to compiler/rustc_parse/src/lexer when spans, symbols, diagnostics, comments, or edition policy are involved. Parser grammar and recovery live primarily under compiler/rustc_parse/src/parser. AST token, token-stream, node, and NodeId definitions live under compiler/rustc_ast.

Macro expansion and MBE machinery live primarily under compiler/rustc_expand, with built-ins under compiler built-in macro code and hygiene under rustc_span. Macro and import resolution interactions involve resolver code as well as expansion. Procedural macro bridging spans compiler and stable-library-facing components. Paths and ownership move: search symbols in the exact 1.97.1 checkout before editing.

A good first contribution changes one stage, adds a minimized regression test at that stage, and runs focused tests before broad suites. Read nearby comments and history because apparently redundant state may support recovery or diagnostics. Preserve token fidelity, progress, byte-boundary, and no-cascade invariants. When changing internal APIs, update all producers and consumers rather than converting away meaningful error data.

The following official sources anchor this chapter. Nightly API pages describe unstable internals and may be newer than rustc 1.97.1; use the tagged source for exact correspondence.

• Rust Compiler Development Guide: Lexing and parsing
• Rust Compiler Development Guide: Macro expansion
• Rust Reference: Macros by example
• Rust Reference: Procedural macros
• Rust Reference: Identifiers
• Rust Reference: Tokens
• Nightly rustc API: rustc_lexer
• Nightly rustc API: rustc_parse
• Nightly rustc API: rustc_ast token streams
• Nightly rustc API: rustc_expand
• Nightly rustc API: rustc_span hygiene
• Stable proc_macro API

When documentation and a pinned checkout disagree about a private field or algorithm detail, the checkout controls that compiler build. When stable macro behavior is at issue, the Reference and stable API contract control rather than a private rustc representation. Keep those two levels separate: internal architecture explains implementation, while language and library specifications define promises to users.

Part II-B: Front-End Internals as Executable Contracts#

This continuation assumes Chapters 1–34 and goes beneath their pipeline overview. It follows the architecture near rustc 1.97.1, not a stable compiler API. Private names and ownership boundaries below are source-reading landmarks, not promises; confirm them in the exact checkout before changing code.

The organizing question is: which representation owns each fact, and which invariant lets the next stage trust it? That question is more useful than memorizing functions because rustc's implementation moves while its obligations change more slowly.

35. The front end is a feedback system, not a conveyor belt#

A conveyor-belt picture—bytes, tokens, AST, expanded AST—is useful only for the first minute. Real Rust syntax can define macros that emit imports, macro definitions, attributes, and more invocations. The front end therefore alternates discovery, resolution, expansion, parsing, and integration until no expandable invocation remains.

source bytes
    │ UTF-8 text + file identity
    ▼
low-level lexer ── chunks + lengths + lexical flags
    ▼
high-level lexer ── tokens + spans + symbols + spacing
    ▼
token-tree reader ── balanced groups + delimiter spans
    ▼
parser ── initial AST + recovery markers + retained tokens
    ▼
┌──────────────── expansion fixed-point loop ────────────────┐
│ collect invocations → resolve what is resolvable           │
│        ↑                         ↓                          │
│ integrate generated AST ← parse output ← expand token input│
└────────────────────── unresolved work may remain ──────────┘
    ▼
expanded AST → validation / lowering

Each arrow carries data, not merely control. The first arrow carries byte lengths; the token-tree arrow carries nesting; integration carries identities and expansion provenance. The diagram simplifies import resolution and attribute handling, which interleave with the loop.

Separate four concerns:

Putting policy in the lowest scanner makes that scanner less reusable. Putting structural recognition too late loses precise boundaries. Putting presentation everywhere produces duplicate diagnostics. The useful boundary is not “early is good”; it is “decide at the first stage that has all required information.”

Core invariants

1. Every scanner or parser loop advances, returns, or records explicitly deferred work.
2. Every source-backed range is ordered, file-contained, and on UTF-8 boundaries.
3. Balanced token trees never expose a closing delimiter as an ordinary child by accident.
4. Recovery produces structurally valid placeholders and records that an error was already emitted.
5. Expansion output is parsed in the promised fragment kind.
6. Generated identifiers retain both spelling and hygiene context.
7. Expansion reaches a fixed point or stops under an explicit error/resource policy.

Prediction exercise. A macro invocation is lexically valid but resolves only after another expansion emits an import. Should parsing reject it? No. Parsing records a syntactic invocation; expansion queues it and retries resolution. Rejecting at parse time would move name-resolution knowledge into the grammar and make valid programs order-sensitive.

36. Lexing under a microscope: cursor contracts and cost#

The low-level lexer receives &str, so UTF-8 validation has already happened. It can exploit ASCII's encoding property: an ASCII byte never occurs as part of a multibyte scalar. That permits cheap byte tests for common punctuation while Unicode-aware iteration handles identifiers.

Conceptually, one token step is:

precondition: cursor is at a UTF-8 boundary and at or before end
inspect bounded lookahead
choose one lexical rule
consume a nonempty byte range, unless emitting EOF
return coarse kind, byte length, and rule-specific status
postcondition: consumed range ends at a UTF-8 boundary

The scanner need not allocate token spelling. If the start is p and returned length is n, the caller can recover &source[p..p+n]. This representation makes scanning and tests cheap, but deliberately forgets file identity and diagnostic context. The high-level adapter restores those from its source-file state.

Longest match is not a universal rule#

“Always take the longest token” is an attractive but false simplification. Rust's lexical specification and later grammar jointly govern dots, apostrophes, literal suffixes, reserved prefixes, and punctuation jointness. A low-level scanner may identify a broad literal candidate, then literal validation reports malformed digits. Conversely, greedily swallowing punctuation can erase distinctions the parser needs.

Counterexample: treating 1..2 as a malformed float consumes the range operator's first dot. Counterexample: treating 'a unconditionally as a character literal ignores lifetime syntax. The remedy is not arbitrary backtracking; it is a documented decision table with enough lookahead.

Nested comments#

For /* A /* B */ C */, trace (offset, depth):

The invariant is that depth equals openers minus closers in the consumed comment prefix. Use a checked or sufficiently bounded counter: input length bounds nesting, but accidental arithmetic overflow must not become undefined policy. At EOF with positive depth, report the outer construct and useful nested context rather than repeatedly diagnosing every opener.

Raw-string search and adversarial cost#

For r###"body"###, store the opening hash count h, then scan candidate quote bytes. At each quote compare at most h following hashes. A naive implementation can approach O(nh) on many near-closing quotes. Production code should be benchmarked with ordinary strings and adversarial repeated quote/hash patterns. Possible strategies include fast searching for quotes, then bounded comparison; changing the algorithm is valid only if token extent and error metadata remain identical.

The observation model for a lexer optimization includes token kinds, lengths, attached flags, and termination behavior—not only acceptance. A “faster” scanner that shifts an error endpoint changes diagnostics and can break snapshot tests.

Failure map#

Experiment. Generate all strings up to six bytes over /, *, and a. Assert eventual EOF, contiguous token coverage, and no panic. Then retain every failing seed as a regression test.

37. The high-level token layer: symbols, spacing, and editions#

The adapter in and around rustc_parse turns source-relative lengths into compiler positions, interns names, translates low-level kinds to rustc_ast::token forms, handles comments, and has enough session context for edition-sensitive diagnostics.

Interning answers “are these spellings equal?” cheaply. Hygiene answers a different question: “do these occurrences inhabit compatible name-resolution contexts?” Never use symbol equality as identifier identity.

Punctuation carries spacing such as Joint versus Alone. Given a+=b, plus is joint to equals; given a + = b, it is not. Token trees expose punctuation as individual tokens, so jointness preserves the ability to form compound operators. Synthetic transcription must choose spacing rather than inheriting accidental host-language formatting.

bytes:       '+'  '='
              │ no trivia
              ▼
tokens:      Punct('+', Joint)  Punct('=', Alone)
              │ parser gluing policy
              ▼
operator:    PlusEq

This representation makes token-level macro behavior faithful and generic closers splittable. It makes parser consumption more complicated than a lexer that emits one PlusEq token. The alternative moves complexity; it does not remove it.

Edition is policy context. The same character sequence can produce a reserved-prefix diagnostic or keyword interpretation depending on edition. Cross-edition macro behavior also depends on where a token originated, so “use the consuming crate's edition everywhere” is not a safe model. Check the Reference and current span/identifier APIs for the exact rule.

Prediction. Two identifiers have spelling x, same byte coordinates in synthetic text, but different syntax contexts. Can the interner distinguish them? No. Their symbols compare equal; hygiene-aware resolution distinguishes them.

38. Token trees: lossless enough, intentionally not concrete syntax#

A token tree is either a leaf token or a delimited stream. Grouping converts a flat delimiter problem into recursive structure. For { a([b]) }, a consumer of the brace group cannot accidentally consume the file's later closing brace.

Delimited Brace
├── Ident a
└── Delimited Parenthesis
    └── Delimited Bracket
        └── Ident b

Open and close delimiter spans are separately useful: a mismatch can label both sites. Internal invisible delimiters may preserve grouping without claiming that the user typed punctuation. Code that prints or compares streams must define whether invisible groups are observable.

Token trees preserve:

• token kind and spelling-bearing symbols;
• nesting and delimiter kind;
• punctuation spacing;
• spans and therefore hygiene/provenance.

They do not inherently preserve every whitespace byte or comment as ordinary macro input. They are therefore “lossless enough for promised token semantics,” not a full concrete syntax tree. A formatter requiring exact trivia needs a different representation or side channel.

Delimiter recovery#

On f!([a}), a reader can diagnose expected ], label [, and decide how to recover around }. Policies include synthesizing a closer, treating the mismatched closer as ending an outer group, or skipping to a synchronization point. Each policy changes cascade quality. The mechanism is a stack; policy decides which stack frame a closer may satisfy.

Invariant: after recovery, every emitted group has a coherent delimiter record, even if one endpoint is synthetic. Downstream code must be able to ask whether a span is synthetic instead of guessing from coordinates.

Counterexample. Flattening all groups and re-pairing delimiters after macro matching is not equivalent. Matchers can capture one tt as a whole group; flattening changes what one repetition iteration consumes.

39. Parser internals: snapshots, restrictions, and fragment boundaries#

Rustc's hand-written recursive-descent parser carries more state than a token index. Besides current/lookahead tokens, it has edition and restriction state, diagnostics, expected-token information, and context controlling ambiguous grammar decisions. Exact fields around 1.97.1 must be checked in compiler/rustc_parse/src/parser/.

A parser snapshot used for speculation must account for all observable state. Restoring only the cursor while retaining speculative diagnostics yields ghost errors. Restoring diagnostics but not contextual restrictions can parse the same token differently on retry. Where possible, rustc uses bounded lookahead predicates rather than general rollback.

Fragment contracts#

A fragment parser promises both a node category and where parsing stops. Parsing an expression from macro output is not “parse anything and take the first expression.” Trailing tokens must satisfy the fragment's contract, and follow-set restrictions protect future grammar evolution for declarative macros.

The parser may retain tokens on AST nodes that need later macro/attribute processing. This trades memory for reconstructability. Eagerly dropping tokens makes ordinary AST traversal cheaper, but reparsing source is impossible for generated syntax and can lose hygiene.

Ambiguity as deferred commitment#

Angle brackets may begin generic arguments or participate in comparisons. Braces may be blocks or delimit structures in context. The parser uses grammar position, bounded lookahead, and recovery heuristics. Deferring every choice to semantic analysis would preserve ambiguity but burden all later phases with forests of alternatives. Committing too early gives poor errors. Rustc chooses local commitment with language-specific diagnostics.

Debugging workshop. A valid expression parses only after an unrelated comma is inserted. Dump tokens first. If unchanged up to the failure, instrument parser entry/exit, restrictions, and consumed range. If a speculative path emitted the final diagnostic, test whether snapshot restoration includes diagnostic state.

40. AST construction: ownership, identities, and retained uncertainty#

The AST converts grammar relationships into typed nodes. It makes “visit every item” cheap, but forgets much formatting and may normalize several surface spellings into one form. Spans and retained tokens preserve selected provenance.

A node commonly contains a kind plus span and identity-related metadata. NodeId is an in-compilation identity, not a stable source key. During parse/expansion, dummy IDs may exist until integration assigns real ones. The invariant is phase-indexed: dummy IDs are legal only before consumers requiring uniqueness.

Construct only after ownership is clear#

For a + b * c, precedence parsing first owns b * c, then uses it as the right child of +. Constructing a + b eagerly and rotating later complicates spans, attributes, and recovery. Representations should follow the grammar decision that established ownership.

token range [0, 9)
└── Binary + span [0, 9)
    ├── Name a span [0, 1)
    └── Binary * span [4, 9)
        ├── Name b span [4, 5)
        └── Name c span [8, 9)

The parent span usually covers children and intervening syntax, but synthetic/recovered nodes complicate containment. Do not assert simple numeric containment across macro files or expansion provenance without using span APIs.

Error nodes are represented uncertainty#

Suppose let x = 1 + ;. Recovery can create an error expression as the missing right operand. The enclosing binary and statement remain traversable. An emitted-error guarantee prevents each later visitor from rediscovering the same syntax failure.

Bad alternative: invent literal 0. It makes the tree type-correct-looking while discarding uncertainty, allowing constant evaluation or lints to report nonsense. An explicit error node says “structure continues, meaning is unavailable.”

Bad alternative: return no statement. That reduces bogus semantics but can hide subsequent names and cause resolution cascades. Recovery policy balances structural continuity against false interpretation.

41. Recovery engineering: find the earliest broken invariant#

Recovery has three operations:

1. Diagnose the primary mismatch once.
2. Repair the local representation by insertion, deletion, replacement, or error node.
3. Synchronize at a boundary where ordinary parsing can safely resume.

Candidate synchronization tokens include semicolons, commas, closing delimiters, and item starters, but the correct set depends on nesting and expected fragment. Skipping to the next semicolon without respecting groups can discard an entire function.

A concrete trace#

Input:

fn f() {
    let x = call(1, 2;
    let y = 3;
}

The parser sees ( with no matching ) before ;. A useful repair synthesizes ) at the semicolon, keeps the semicolon as statement terminator, and resumes at let y. The synthetic close carries an insertion-point span and the diagnostic labels the opener.

before: call ( 1 , 2 ; let y ...
             └─ unmatched
repair: call ( 1 , 2 [synthetic )] ;
resume:                              let y ...

If recovery consumes ;, the statement parser may then complain that a semicolon is missing. That second message reveals the repair violated token ownership. The first visible failure is often later than the first broken invariant.

Evaluate recovery quality#

Do not optimize solely for diagnostic count. Measure:

• whether the primary message identifies the real edit;
• whether later independent errors still appear;
• whether suggestions produce parsable code;
• whether malformed input terminates with bounded time and memory;
• whether a change destabilizes UI snapshots across unrelated cases.

Use tests/ui/parser/ and neighboring suites in the exact rustc checkout; directories evolve. A regression test should be the smallest source preserving the bad recovery path, with expected stderr reviewed rather than blindly blessed.

Prediction exercise. On a missing comma in a list, should recovery always insert one? No. If the next token cannot begin another element, insertion may create a bogus element error. Compare insertion and deletion hypotheses using local grammar evidence.

42. Expansion as a work queue and fixed-point computation#

An invocation is syntax requesting expansion: function-like macro call, attribute, derive, or compiler-built-in form. Initial collection replaces or marks expansion sites and places work in a queue. Resolution may succeed, fail finally, or remain indeterminate because another expansion can reveal a binding.

Conceptual pseudocode (not rustc source):

queue := invocations collected from current AST
while queue is not empty:
    invocation := choose according to expansion order
    if resolvable(invocation):
        output_tokens := run_expander(invocation)
        fragment_ast := parse_as(output_tokens, invocation.expected_fragment)
        assign expansion provenance and required identities
        integrate fragment_ast at invocation site
        collect newly revealed invocations into queue
    else if progress elsewhere can make it resolvable:
        defer invocation
    else:
        emit resolution error and integrate error placeholder
    require progress, bounded deferral, or explicit termination

The actual rustc loop around rustc_expand and resolver interfaces is more nuanced. The durable model is monotonic discovery plus controlled deferral, not a single pre-order tree rewrite.

Why iteration is necessary#

macro_rules! make { () => { macro_rules! later { () => { 7 } } } }
make!();
const N: i32 = later!();

Expanding make! reveals later!'s definition. A one-shot resolver that freezes the namespace before expansion rejects valid structure. Conversely, eagerly executing every syntactic macro before resolution cannot know which expander a name denotes.

Progress and cycles#

A recursive macro can emit itself forever. “Queue nonempty” is not evidence of useful progress. Expansion depth and recursion limits provide policy boundaries; exact defaults and diagnostics are version-sensitive and must be verified rather than copied from memory. Token growth can also be enormous without deep recursion, so production defenses need more than a depth counter.

Expansion order affects diagnostics and visibility details, so it is observable policy. Parallel expansion is not automatically safe: procedural macros execute code, resolution state evolves, and deterministic diagnostic order matters. Parallelism would require explicit dependency and observation models.

43. macro_rules! matching: the nondeterministic-looking machine#

A declarative macro contains ordered rules. Each rule has a matcher and transcriber. The matcher consumes token trees using literal tokens, metavariables such as $e:expr, and repetitions. The first successfully matching rule is transcribed; later rules are not “more specific” winners.

Fragment specifiers ask the Rust parser to recognize a category. A $t:tt capture consumes one token tree; $i:ident consumes an identifier-shaped fragment; $e:expr invokes expression-fragment machinery under the applicable edition semantics.

Matcher positions#

Repetition makes matching resemble a nondeterministic automaton: several matcher positions may be viable after one input token. rustc tracks candidate positions and advances them, invoking fragment parsing where needed. It must reject local ambiguity rather than guess based on future semantic meaning.

Trace this matcher:

macro_rules! names {
    ($( $name:ident ),* $(,)?) => { /* ... */ };
}
names!(a, b,);

The implementation must preserve capture nesting: names is not merely [a,b]; it is data indexed by repetition depth. Nested repetitions require a tree or sequence-of-sequences shape.

Follow sets and future compatibility#

Fragment captures cannot be followed by arbitrary tokens even if today's parser could find an endpoint. Follow-set restrictions prevent a macro definition from depending on grammar accidents that a future Rust edition might change. This is language policy, documented in the Reference, not merely an implementation limitation.

No arbitrary lookahead rescue#

Counterexample:

macro_rules! ambiguous {
    ($($i:ident)* $j:ident) => {};
}

At an identifier, the matcher cannot know whether it belongs to the repetition or final capture without guessing. Rust reports local ambiguity rather than backtracking until one interpretation works. That keeps matching predictable and avoids pathological search.

44. Transcription: repetition shape is a type system#

Transcription copies literal template tokens and substitutes captured fragments. Repetitions in output must be driven by metavariables captured at a compatible repetition depth. This is analogous to a shape type: a scalar capture cannot determine list length, and two unrelated lists cannot silently zip with mismatched lengths.

macro_rules! fields {
    ($( $name:ident : $ty:ty ),* $(,)?) => {
        struct Generated { $( $name: $ty ),* }
    };
}
fields!(x: i32, label: &'static str);

Trace:

1. Matcher records two iterations, each with name and ty.
2. Transcriber enters one repetition level.
3. Iteration 0 substitutes x and i32.
4. Separator comma is emitted.
5. Iteration 1 substitutes label and &'static str.
6. Generated item tokens are parsed as an item and integrated.

Invariant: all metavariables used to determine one transcriber repetition agree on iteration count at that nesting level. Failure should point to the macro definition, while useful notes may connect to invocation captures.

Token provenance during transcription#

Literal tokens written in the macro definition carry definition-side provenance. Substituted input retains call-side provenance in important respects. New grouping and synthesized separators need deliberate spans and spacing. Assigning the invocation's one span to all output would make every diagnostic highlight the call and destroy useful source distinctions.

Prediction exercise#

Can a transcriber repeat $(,)* with no metavariable inside to choose a count? No: there is no captured sequence driving repetition length. This is not a parser failure; it is a transcription-shape error.

45. Hygiene: identity includes a history of expansion#

Hygiene prevents generated names from accidentally capturing, or being captured by, names that merely share spelling. A useful mental model is an identifier (symbol, syntax_context). The syntax context is derived from expansion marks/history, not a lexical-scope number and not just a source span.

Consider a macro that introduces a temporary named tmp around a caller expression that also mentions tmp. Textual substitution would capture one with the other. Hygienic expansion keeps contexts distinct where the macro system's hygiene rules require it.

definition token `tmp`
    │ apply expansion mark E
    ▼
(`tmp`, def-context + E) ──resolution──► macro-introduced binding

captured caller token `tmp`
    │ preserve call context through substitution
    ▼
(`tmp`, call-context) ─────resolution──► caller's binding

The diagram simplifies Rust's mixed-site behavior. macro_rules! does not apply one uniform def-site rule to every name category; labels, local variables, and $crate have specific semantics. Consult the Reference for normative behavior and rustc source for implementation.

$crate#

$crate lets an exported declarative macro refer to its defining crate despite invocation elsewhere or crate renaming. It addresses crate identity, not visibility: referenced items still need suitable visibility. Replacing $crate with a textual crate name breaks dependency renaming and can resolve to the wrong crate.

Context adjustment#

Resolution crosses boundaries between definition-origin and call-origin tokens. rustc applies context transformations appropriate to macro kind and lookup. Do not infer correctness by printing text: pretty-printed identifiers can be identical while their contexts differ.

Experiment. Write two macros differing only in whether an identifier is literal template syntax or a captured $ident. Use shadowed locals at the call site. Compare compiler behavior and expansion traces; never conclude hygiene from pretty text alone.

46. Span provenance: diagnostics, hygiene, and trust#

A rustc span conceptually combines a byte range with syntax context and provenance through expansion data. The implementation uses compact encodings and interning strategies that are private and version-sensitive. Treat public operations—not presumed field layout—as the contract even inside source-reading notes.

Useful origins include:

• call site: where the invocation appeared;
• definition site: where macro template code was written;
• mixed site: behavior chosen for compatibility/hygiene semantics;
• synthetic site: generated punctuation or recovery insertion.

Span constructors and transformations influence both name resolution and diagnostics. Choosing a span merely because it highlights attractive text can alter hygiene. Choosing a context merely for resolution can make diagnostics unactionable. Span design is a semantic decision, not cosmetic metadata.

Provenance walk#

For outer!(inner!()), an error in inner output may have a chain:

generated failing token
    │ produced by inner expansion
    ▼
inner invocation span
    │ itself contained in outer output
    ▼
outer invocation span
    │ source map lookup
    ▼
user file and line

A diagnostic can present the most actionable site and add expansion backtrace notes. Flattening this chain to the outer call saves metadata but destroys causal explanation. Keeping unbounded verbose chains can overwhelm users, so presentation may summarize without deleting internal provenance.

Security consequence#

Diagnostic snippets and paths can reveal generated or remapped source information. Build systems may remap paths for reproducibility/privacy. Procedural macros can choose spans but cannot be assumed trustworthy. Tools consuming diagnostics should treat paths and snippets as untrusted data and avoid terminal-control injection or unsafe HTML rendering.

47. Built-ins, attributes, derives, and procedural macros#

“Macro” names several mechanisms sharing token-oriented syntax but differing in resolver namespace, input/output contract, execution location, and trust.

Exact categories and implementation registries around 1.97.1 should be read in rustc_builtin_macros, rustc_expand, rustc_resolve, and proc-macro bridge/server code in that checkout.

Attributes need ordering policy#

Attributes can configure whether syntax exists, request derives, or transform an item. Ordering affects which attributes remain visible to later transformations. An attribute macro generally receives an attribute's arguments and annotated item token stream through the stable proc-macro interface, but rustc's internal staging around configuration and derives is nuanced and version-sensitive. Never document a total ordering from one experiment as a language guarantee unless the Reference specifies it.

Derive helpers#

Derive macros can declare helper attributes. These are recognized for the derived item according to the proc-macro contract; they are not arbitrary globally available attributes. If a helper is reported unknown, inspect derive registration and placement before parser syntax.

Built-ins are not ordinary crates#

Compiler built-ins may need source inclusion, environment data, format-string parsing, or AST-aware behavior. Their implementation can bypass costs or contracts of external proc macros, but this creates compiler maintenance and bootstrap obligations. A feature being “macro-like” does not imply it should become a built-in; libraries offer release independence and reduce compiler trust surface.

48. Procedural macros as a compiler trust boundary#

A procedural macro is compiled for and executed by the host, even when the target program is for another architecture. It receives stable proc_macro token streams rather than rustc's private AST. The bridge translates representations across an implementation boundary that may involve a separate process architecture in rustc's design. Details are version-sensitive; inspect the exact toolchain source.

Host versus target#

If compiling on x86-64 Linux for WebAssembly, the proc macro still runs as host code. It can observe host environment behavior unless the build system restricts it. Generated tokens are later compiled for the target. Confusing these worlds causes portability bugs: a macro should not assume target pointer width from its own usize.

Capabilities and threats#

Proc macros are dependencies that execute code during compilation. Depending on environment and sandboxing, they may read files, environment variables, consume CPU/memory, and attempt other host operations. Rust's language-level proc-macro API is not itself a complete security sandbox.

Threats include:

• malicious dependency code and supply-chain compromise;
• accidental secret reads embedded into generated output or diagnostics;
• non-determinism from time, random state, filesystem order, or network;
• hangs, crashes, stack exhaustion, and output explosions;
• confusing spans that blame unrelated user code.

Mitigations belong at several layers:

1. Pin, review, and audit dependencies; minimize macro dependencies.
2. Run builds in OS/container sandboxes with least privilege and no unnecessary secrets/network.
3. Apply wall-clock, memory, process, and output limits in CI/build orchestration.
4. Prefer deterministic inputs and sorted filesystem traversal in macro implementations.
5. Cache only under keys covering compiler, macro binary, target/configuration, environment inputs, and source inputs.

A timeout inside rustc alone cannot revoke all OS capabilities. A container alone does not guarantee deterministic output. Abstractions move responsibility rather than deleting it.

Error channels#

A proc macro can emit compile_error!-shaped output, use diagnostic facilities available on its toolchain/API, or panic. A panic should become a controlled compiler diagnostic, not corrupt compiler state. Invalid output is then lexed/parsed as the promised fragment and may generate ordinary syntax errors. Distinguish “macro panicked,” “macro returned malformed tokens,” and “valid output failed later semantics.”

49. Resource limits, cancellation, and deterministic expansion#

Front-end input is adversarial whenever a compiler runs on untrusted repositories. Correctness includes termination and bounded resource policy, not only accepting valid programs.

Cost dimensions#

Depth limits stop one axis. A macro that doubles tokens each shallow round grows exponentially. Use several budgets: expansion depth, total produced tokens/bytes, diagnostics, time, and memory where infrastructure permits. Changing a limit is policy and compatibility work: too low rejects legitimate generated code; too high fails to protect services.

Cancellation#

Long loops should reach cancellation checkpoints at safe boundaries, but cancellation cannot leave shared interners, resolver tables, or output caches in a state later treated as complete. Transactional integration—build a fragment, validate required invariants, then publish it—makes cancellation safer than mutating global state token by token.

Determinism#

Determinism means the chosen observation is stable for identical declared inputs. Define whether that includes binary bytes, diagnostics ordering, paths, and timings. Hash-map iteration, parallel scheduling, environment variables, and proc-macro behavior can perturb results. Sort only where order is semantically irrelevant; sorting expansion work indiscriminately can change language-observable resolution or diagnostics.

Cache obligations#

Caching macro output creates correctness obligations. A key omitting edition, feature configuration, macro binary version, environment dependency, or span mapping can return syntactically plausible but wrong output. Cached spans may refer to obsolete source positions. Prefer recomputable structured output with provenance rebasing only when its semantics are rigorously specified.

50. From-scratch progression I: balanced token trees in stable Rust#

The following is a complete, stable, dependency-free educational program. It groups only ASCII delimiters and treats every other non-whitespace character as a leaf. It is not a Rust lexer: identifiers become separate character leaves, comments and strings are not recognized, and byte spans are meaningful only for its ASCII-delimiter subset.

#[derive(Debug, PartialEq, Eq)]
enum Tree {
    Leaf { ch: char, lo: usize, hi: usize },
    Group { open: char, close: char, lo: usize, hi: usize, children: Vec<Tree> },
}

#[derive(Debug, PartialEq, Eq)]
struct Error {
    at: usize,
    message: String,
}

fn closing(open: char) -> Option<char> {
    match open {
        '(' => Some(')'),
        '[' => Some(']'),
        '{' => Some('}'),
        _ => None,
    }
}

fn parse_level(
    src: &str,
    chars: &mut std::iter::Peekable<std::str::CharIndices<'_>>,
    expected: Option<(char, char, usize)>,
) -> Result<Vec<Tree>, Error> {
    let mut out = Vec::new();
    while let Some(&(at, ch)) = chars.peek() {
        if let Some((open, close, start)) = expected {
            if ch == close {
                chars.next();
                return Ok(out);
            }
            if ")]}`".contains(ch) {
                return Err(Error {
                    at,
                    message: format!("expected {close:?} for {open:?} at {start}, found {ch:?}"),
                });
            }
        } else if ")]}`".contains(ch) {
            return Err(Error { at, message: format!("unexpected closing delimiter {ch:?}") });
        }

        chars.next();
        if let Some(close) = closing(ch) {
            let children = parse_level(src, chars, Some((ch, close, at)))?;
            let hi = chars.peek().map_or(src.len(), |(next, _)| *next);
            out.push(Tree::Group { open: ch, close, lo: at, hi, children });
        } else if !ch.is_whitespace() {
            out.push(Tree::Leaf { ch, lo: at, hi: at + ch.len_utf8() });
        }
    }

    if let Some((open, close, start)) = expected {
        Err(Error {
            at: src.len(),
            message: format!("unclosed {open:?} at {start}; expected {close:?}"),
        })
    } else {
        Ok(out)
    }
}

fn group(src: &str) -> Result<Vec<Tree>, Error> {
    parse_level(src, &mut src.char_indices().peekable(), None)
}

fn main() {
    let trees = group("a([b])").expect("balanced input");
    assert_eq!(trees.len(), 2);
    assert!(matches!(trees[1], Tree::Group { open: '(', close: ')', .. }));

    let error = group("([)]").expect_err("mismatch must fail");
    assert!(error.message.contains("expected ']'"));
    println!("{trees:#?}");
}

The recursive call owns one delimiter level. Its progress invariant is simple: each iteration consumes one scalar, enters a child that consumes its opener's interior, or returns after consuming the expected closer. The group hi is the next byte position after the consumed closer.

Hardening milestones#

1. Replace character leaves with tokens from the Chapter 10–12 lexer.
2. Store separate open and close spans, including synthetic-close state.
3. Recover from mismatches instead of returning the first error.
4. Add maximum nesting and total-token budgets.
5. Preserve punctuation jointness based on trivia.
6. Fuzz arbitrary UTF-8 and assert each byte belongs to one token/trivia range.

Prediction: this program mishandles "(". It creates a group because it has no string lexer. That failure demonstrates layering: delimiter grouping is correct only over tokens, not raw characters.

51. From-scratch progression II: a bounded declarative matcher#

This complete stable-Rust educational program matches comma-separated identifiers and transcribes them into a bracketed list. It models one repetition and deliberately omits Rust tokens, fragment parsers, nested repetitions, hygiene, and spans.

#[derive(Debug, Clone, PartialEq, Eq)]
enum Tok {
    Ident(String),
    Comma,
}

#[derive(Debug, PartialEq, Eq)]
struct Captures {
    names: Vec<String>,
}

fn match_names(input: &[Tok], max_items: usize) -> Result<Captures, String> {
    let mut at = 0usize;
    let mut names = Vec::new();
    while at < input.len() {
        if names.len() == max_items {
            return Err(format!("item limit {max_items} exceeded"));
        }
        match &input[at] {
            Tok::Ident(name) => names.push(name.clone()),
            Tok::Comma => return Err(format!("expected identifier at token {at}")),
        }
        at += 1;
        if at == input.len() {
            break;
        }
        if input[at] != Tok::Comma {
            return Err(format!("expected comma at token {at}"));
        }
        at += 1;
        if at == input.len() {
            break; // policy: one trailing comma is accepted
        }
    }
    Ok(Captures { names })
}

fn transcribe(captures: &Captures) -> String {
    let body = captures.names.join(", ");
    format!("[{body}]")
}

fn main() {
    let input = vec![
        Tok::Ident("alpha".into()),
        Tok::Comma,
        Tok::Ident("beta".into()),
        Tok::Comma,
    ];
    let captures = match_names(&input, 100).expect("valid list");
    assert_eq!(captures.names, ["alpha", "beta"]);
    assert_eq!(transcribe(&captures), "[alpha, beta]");
    assert!(match_names(&[Tok::Comma], 100).is_err());
}

The representation makes one repetition's shape explicit as Vec<String>. It forgets token identity and origin, which is why string transcription is unsuitable for real macros. The next milestone is to output token trees with inherited spans rather than text.

Further milestones:

1. Represent matcher instructions Literal, Capture, and Repeat.
2. Track a set of matcher positions instead of one imperative loop.
3. Store captures as nested sequences indexed by repetition depth.
4. Add a fragment-parser callback with explicit consumed range.
5. Reject local ambiguity instead of choosing the longest successful path.
6. Transcribe tokens while preserving capture contexts and definition-token contexts.
7. Apply total-state and output-token budgets to hostile matchers.

Counterexample exercise. Add two lists, names and types, then intentionally give unequal lengths. Make transcription reject the shape mismatch rather than truncate via zip.

52. From-scratch progression III: an iterative hygienic expander model#

The next complete stable-Rust educational program demonstrates queue-based expansion and context-bearing identifiers. Its language has only literal words and calls written as nodes; macro bodies are prebuilt nodes. It is not a Rust macro implementation and has no resolver scopes or parser.

use std::collections::{HashMap, VecDeque};

#[derive(Clone, Debug, PartialEq, Eq)]
struct Ident {
    text: String,
    context: u32,
}

#[derive(Clone, Debug, PartialEq, Eq)]
enum Node {
    Word(Ident),
    Call(String),
    Define { name: String, body: Vec<Node> },
}

fn expand(mut nodes: Vec<Node>, limit: usize) -> Result<Vec<Node>, String> {
    let mut definitions: HashMap<String, Vec<Node>> = HashMap::new();
    let mut queue: VecDeque<Node> = nodes.drain(..).collect();
    let mut output = Vec::new();
    let mut steps = 0usize;
    let mut next_context = 1u32;

    while let Some(node) = queue.pop_front() {
        steps += 1;
        if steps > limit {
            return Err(format!("expansion step limit {limit} exceeded"));
        }
        match node {
            Node::Word(word) => output.push(Node::Word(word)),
            Node::Define { name, body } => {
                definitions.insert(name, body);
            }
            Node::Call(name) => {
                let body = definitions
                    .get(&name)
                    .ok_or_else(|| format!("unresolved macro {name}"))?
                    .clone();
                let context = next_context;
                next_context = next_context.checked_add(1)
                    .ok_or_else(|| "context id overflow".to_string())?;
                for generated in body.into_iter().rev() {
                    let marked = match generated {
                        Node::Word(mut word) => {
                            word.context = context;
                            Node::Word(word)
                        }
                        other => other,
                    };
                    queue.push_front(marked);
                }
            }
        }
    }
    Ok(output)
}

fn main() {
    let program = vec![
        Node::Define {
            name: "make".into(),
            body: vec![Node::Word(Ident { text: "tmp".into(), context: 0 })],
        },
        Node::Word(Ident { text: "tmp".into(), context: 0 }),
        Node::Call("make".into()),
    ];
    let output = expand(program, 100).expect("bounded expansion");
    let Node::Word(caller) = &output[0] else { panic!("word") };
    let Node::Word(generated) = &output[1] else { panic!("word") };
    assert_eq!(caller.text, generated.text);
    assert_ne!(caller.context, generated.context);
}

This model marks every generated word, unlike Rust's mixed-site and capture-preserving hygiene. It resolves definitions only when encountered, so a call before its definition fails rather than being deferred. Those omissions are productive exercises:

1. Add a deferred queue and terminate when one full pass makes no progress.
2. Distinguish template words from captured caller words and mark only the former.
3. Detect recursive output through step and output-node limits.
4. Make publication transactional: a failed body contributes no partial output.
5. Record parent expansion IDs and print a backtrace on failure.

The key lesson is that hygiene context is identity data carried with a token, not a renaming pass after expansion.

53. Testing the real front end: layers and oracles#

Tests should fail at the narrowest responsible boundary. A parser UI test is too broad for a scanner progress bug; a lexer unit test cannot verify a recovery suggestion.

Property catalogue#

• Token lengths cover input according to documented trivia policy.
• Non-EOF tokens have positive length.
• Token endpoints are UTF-8 boundaries.
• Grouping never changes leaf order.
• Parsing malformed input terminates and preserves all independent trailing items possible.
• Pretty-print/reparse tests compare a defined structure, not unstable IDs/spans.
• Macro matching is deterministic for accepted definitions.
• Transcriber repetition dimensions agree.
• Expansion either shrinks pending work/makes defined progress or reaches a diagnosed bound.
• No generated NodeId duplicates another where uniqueness is required.
• Diagnostic suggestions marked machine-applicable actually parse after application.

Differential testing cautions#

Current rustc is not an oracle for a proposed behavior change, and compiler internals can intentionally differ by edition. Record toolchain revision, flags, target, environment, and expected divergence. Comparing only success status misses AST-shape and diagnostic regressions.

Benchmark discipline#

Measure release builds, warm-up behavior, input bytes, nesting, token count, and machine details. Include ordinary crates and crafted worst cases. Report medians and tails rather than one best run. A raw-string optimization should be benchmarked against long ordinary source too; optimizing adversarial input while slowing all identifiers is a poor trade unless security policy requires it.

54. Debugging and current rustc source-reading around 1.97.1#

Start from a minimized reproducer and identify the earliest representation that is wrong. Do not begin by reading the entire compiler.

Source-reading route#

In a rust-lang/rust checkout corresponding to the target toolchain, inspect:

1. compiler/rustc_lexer/ for low-level scanning and token kinds.
2. compiler/rustc_parse/src/lexer/ and parser modules for adaptation and grammar.
3. compiler/rustc_ast/ for tokens, token streams, AST, spans attached to nodes.
4. compiler/rustc_expand/ for expansion, declarative macro machinery, and integration.
5. compiler/rustc_builtin_macros/ for compiler-provided expanders.
6. compiler/rustc_resolve/ for macro resolution interactions.
7. compiler/rustc_span/ for source maps, spans, symbols, and hygiene data.
8. proc-macro bridge/server crates and library/proc_macro/ for stable API boundaries.
9. tests/ui/parser, macro-related UI suites, and neighboring tests for executable intent.

Paths are landmarks, not guaranteed layout. Use symbol search from public entry points and follow actual calls. Read tests before changing a diagnostic: they expose recovery policy that type signatures omit.

Reproducible investigation#

rustc --version --verbose
rustc failing.rs

On a suitable nightly/compiler-development environment, -Z debugging options can expose token, AST, macro backtrace, or expansion information, but names and output are unstable. Run rustc -Z help for the exact binary rather than relying on remembered flags. Environment logging targets are likewise internal; search the current source for tracing instrumentation.

Useful stable-facing tools include cargo expand as a third-party convenience, proc-macro unit tests over helper logic, and minimized rustc invocations. Pretty expansion is diagnostic evidence, not a faithful dump of syntax contexts or all token spacing.

Triage ladder#

1. Confirm exact toolchain, edition, target, and command.
2. Minimize while preserving the failure and macro origin.
3. Inspect source bytes for Unicode/line-ending surprises.
4. Inspect token kinds, boundaries, spacing, and groups.
5. Inspect parser entry kind, restrictions, recovery, and AST shape.
6. Inspect expansion queue, resolution state, and generated fragment.
7. Inspect symbol plus syntax context, not spelling alone.
8. Walk expansion provenance for diagnostics.
9. Only then investigate later resolution/type checking.

55. Contribution workflow: changes that reviewers can trust#

A good front-end patch proves which contract changed and which stayed fixed.

1. Minimize and classify the bug by stage.
2. Find the owning crate and nearest tests in the exact checkout.
3. Write a regression test that fails for the intended reason.
4. Instrument locally; remove noisy instrumentation before submission.
5. Change the smallest mechanism or policy layer with sufficient information.
6. Test adjacent editions, delimiters, macro origins, and malformed variants.
7. Run formatter and the rustc test commands prescribed by the repository's current contributor guide.
8. Explain diagnostic changes and performance/security implications in the PR.

Likely review questions:

• Does every error path advance or terminate?
• Are spans source-correct and hygiene-correct?
• Is this edition behavior normative or accidental?
• Does recovery steal tokens owned by an outer parser?
• Can generated output violate an identity assignment assumption?
• Does matcher work become superlinear on hostile input?
• Is a proc-macro failure contained?
• Are test snapshots asserting useful behavior rather than implementation noise?

Contribution exercises#

Lexer exercise. Add a corpus test for nested comments with every prefix truncated. Predict which truncations are complete tokens and which carry unterminated status before running it.

Parser exercise. Find one recovery path that inserts punctuation. Draw token ownership before and after repair; then add a trailing independent item and verify it still parses.

Macro exercise. Locate declarative matcher state in the target source. Annotate where literal tokens, fragment captures, and repetition transitions advance. Construct one locally ambiguous matcher and trace candidate states.

Hygiene exercise. Follow a $crate token from matcher/transcriber representation through resolution. Record which claims come from the Reference and which are implementation details.

Security exercise. Create a harmless proc macro that emits geometrically increasing tokens under a small input parameter. Run only in a resource-limited environment. Graph output tokens and elapsed time, then propose separate depth/output budgets.

Capstone. Implement a bounded toy expander by combining Chapters 50–52: real subset tokens, grouped trees, matcher instructions, nested captures, context-bearing output, deferred calls, expansion backtraces, deterministic diagnostics, and fuzzed resource limits. Document omissions: no Rust fragment grammar, mixed-site hygiene, attributes, proc macros, or compatibility guarantee.

56. Derived philosophy and mastery map#

The mechanisms above support several conclusions.

Representations determine cheap questions#

Byte-length tokens make scanning cheap but cannot answer file/line questions. Token trees make delimiter containment and tt capture cheap but not exact whitespace reproduction. AST makes grammar traversal cheap but erases surface distinctions. Syntax contexts make hygienic identity answerable but increase copying, serialization, and debugging obligations. No representation is universally “lower” or “better”; each chooses affordable questions and forgotten facts.

Abstractions move work#

Single-character punctuation simplifies lexing and macro fidelity while moving gluing work to parsers. Fragment specifiers let macro authors reuse Rust grammar while coupling matching to edition-aware parser contracts. Proc macros stabilize a token API while moving security and reproducibility responsibility to the build boundary. Error nodes simplify later traversal while requiring every consumer to propagate uncertainty.

Identity is not text or location#

Two identifiers can share spelling and coordinates yet differ hygienically. One logical generated token can have useful call-site and definition-site ancestry. NodeId is temporary identity, not durable source identity. Whenever a bug report says “these are the same,” ask: same under which observation—bytes, symbol, context, node, or provenance?

Optimization needs an observation model#

A lexer rewrite preserving acceptance but changing token boundaries is not equivalent. A parallel expander preserving final AST but reordering diagnostics may not be equivalent for users. A cache preserving text but dropping spans is incorrect for diagnostics and perhaps hygiene. State what must remain observable before measuring speed.

Uncertainty must not be guessed away#

Parser error nodes, unresolved expansion work, and synthetic delimiters represent uncertainty explicitly. Replacing them with plausible valid syntax makes local code simpler but creates distant false failures. The earliest broken invariant is usually more diagnostic than the first crash.

Mastery checks#

A reader is ready to contribute when they can:

• trace r#name, punctuation spacing, and nested groups from bytes to tokens;
• explain why token trees and AST coexist;
• predict precedence tree shape and recovery ownership;
• simulate matcher candidate states and transcriber repetition dimensions;
• explain iterative expansion without claiming all names resolve first;
• distinguish spelling, span coordinates, syntax context, and expansion provenance;
• threat-model a proc macro as host code;
• design tests that isolate one boundary and budgets that cover more than recursion depth;
• locate current source and separate Reference guarantees from rustc 1.97.1 details.

Talk outline#

1. One failing macro invocation traced through every representation.
2. Token boundaries and why joint punctuation survives.
3. Group trees versus grammar trees.
4. Expansion's resolver feedback loop.
5. Repetition shape and local ambiguity.
6. Hygiene as context-bearing identity.
7. Span provenance as semantic and diagnostic data.
8. Proc macros as a build security boundary.
9. Earliest-invariant debugging and contribution strategy.

Authoritative reading map#

• The Rust Reference: tokens — normative lexical categories and literals.
• The Rust Reference: macros by example — matching, transcription, follow sets, hygiene behavior including $crate.
• The Rust Reference: procedural macros — stable language-facing proc-macro contracts and span behavior.
• The Rust Reference: attributes — attribute syntax and categories.
• Standard proc_macro documentation — stable API surface for token streams, groups, punctuation, identifiers, literals, and spans.
• rustc development guide: parsing — implementation orientation; verify against current source.
• rustc development guide: macro expansion — expansion algorithm and integration concepts; historical details may lag source.
• rustc development guide: macro hygiene — hygiene implementation orientation.
• rust-lang/rust source — authoritative for the selected compiler revision's implementation.
• rustc contributor guide and tests — current test workflow and suite conventions.
• Rust security policy — reporting compiler security issues; it is not a proc-macro sandbox specification.

Read guarantees in the Reference first, current source and tests second, and the development guide as an architectural map. When they differ, do not silently merge them: label normative language behavior, current implementation, historical explanation, and proposed change separately.

Part II-C: Attributes, Procedural Macros, and the Stable Token Boundary#

This continuation closes the gap between “an attribute exists” and being able to change rustc's expansion machinery safely. It assumes Chapters 1–56 and basic Rust, but no compiler implementation knowledge. Stable language and proc_macro contracts are distinguished from implementation landmarks observed around rustc 1.97.1. Private crate names, types, pass boundaries, and queue details can move; verify them in the exact checkout.

The recurring question is not merely “when does this run?” It is: which stage has enough information to decide, which facts must survive that decision, and who diagnoses failure?

57. Five different jobs hidden behind #[...]#

An attribute is syntax attached to another syntactic object. The outer form #[x] applies to the thing following it. The inner form #![x] applies to the enclosing crate, module, function body, or other grammar-permitted container. The Reference defines where each form is legal; the parser should not infer semantics from punctuation alone.

Five mechanisms share this spelling:

“Built-in” and “active” are different axes. An inert attribute remains attached as metadata for a later consumer and does not itself replace the node during expansion. An active attribute participates in expansion or configuration and is consumed as an operation. Some active attributes are compiler built-ins; others resolve to procedural macros. Some built-ins are inert. Classify by behavior, not spelling or namespace folklore.

The smallest useful mental model is:

parsed node + ordered attributes
              │
              ▼
      classify next attribute
       ┌──────┼────────┐
       │      │        │
 configuration   expansion   retained metadata
       │      │        │
  remove/keep  replace/add   later validation/lowering

The diagram omits resolution retries and feature gates. It establishes one invariant: an active attribute is neither silently retained as inert metadata nor executed twice.

Recognition exercise. Classify cfg, allow, derive, repr, and a user attribute macro along both axes: compiler/user-defined and active/inert. Then identify which classifications require name resolution rather than spelling alone.

58. AST validation is distributed ownership, not one cleanup pass#

The parser recognizes grammatical shape with local context. It intentionally accepts a useful superset when later context can produce a better error. AST validation rejects structurally impossible combinations before consumers rely on stronger assumptions. Attribute checking validates attribute form, placement, duplicates, and arguments where the compiler owns that policy. Feature gating decides whether recognized unstable constructs are permitted for this crate. Expansion creates syntax that may require the same structural scrutiny as handwritten syntax.

No universal rule says “all validation runs after all expansion.” Checks have different information requirements:

Derive ownership by asking four questions:

1. What exact fact is required to decide?
2. At what stage does that fact first exist?
3. Will a transformation later erase or alter it?
4. Which downstream stage first assumes the resulting invariant?

The decision belongs after answer 2 but before answers 3 or 4 make delay unsafe. This is a partial order, not a single numbered pass.

Ordering example#

Consider:

#[cfg(any())]
#[repr(unknown)]
struct Hidden;

Configuration removes the item. A diagnostic about repr(unknown) would inspect dead configuration and could make portable conditional code unusable. Therefore configuration must suppress many later checks on removed nodes. By contrast, malformed tokenization or an unclosed delimiter must still be diagnosed because parsing needs enough structure to locate the item's extent.

Now consider an attribute macro that emits an illegal AST combination. Validating only the initial AST misses it. Validating only before invocation cannot inspect output. Integration must either validate generated nodes or guarantee a later comprehensive validator covers them before lowering.

Handoff invariant. Before lowering trusts an expanded node:

• its grammar-level structure is well formed or explicitly error-marked;
• active attributes intended for this phase have been consumed;
• required gates have been checked at the correct provenance;
• retained attributes are legal enough for their next owner;
• generated identities and spans are installed;
• no removed cfg branch leaks semantic obligations.

59. Attribute syntax, paths, payloads, and built-in policy#

An attribute carries a path and optional input. Common source forms resemble a bare word, a delimited token sequence, or a name-value form. Do not reduce all payloads to strings: delimiters, punctuation spacing, literals, and spans matter.

#[test]
#[allow(dead_code, unused_variables)]
#[doc = "public API"]
#[cfg(all(target_os = "linux", target_pointer_width = "64"))]

The compiler has a registry or equivalent knowledge of built-in attributes around the selected revision. That knowledge may include:

• accepted syntactic template;
• duplicate policy;
• allowed targets;
• whether it is crate-level;
• whether it is active;
• feature-gate requirements;
• whether later phases encode or consume it;
• whether it is allowed on generated or internal constructs.

A table-driven registry makes policy searchable and consistency-testable. It can also become too coarse: context-sensitive rules still need dedicated validators. Encoding every exception in parser branches makes discovery and audit harder.

Duplicate policy is semantic#

Two allow attributes can combine. Two incompatible repr hints may conflict. Repeated doc fragments can intentionally concatenate in some contexts. A singleton attribute may require an error with both spans. “Keep last” discards provenance and silently makes source order semantic.

Diagnostics should point to the conflicting uses and name the owning rule. If configuration removes one use, duplication is assessed on the configured program according to the relevant contract.

Unknown attributes#

An unrecognized attribute might be a misspelled built-in, a tool attribute, a derive helper attribute, or an attribute macro awaiting resolution. Diagnosing it before those namespaces are known creates false positives. Delaying forever loses typo diagnostics. The implementation therefore needs a state richer than known: bool:

known built-in | active macro | registered helper | permitted tool | unresolved | definitively unknown

That state also explains why attribute checking and macro resolution interact.

60. cfg and cfg_attr: syntax controls which program exists#

cfg evaluates a configuration predicate. If false, the attributed construct does not participate in the configured crate. cfg_attr(predicate, attrs...) conditionally contributes attributes. This is not a normal runtime branch and not merely lint suppression.

raw ordered attributes
       │ parse predicate using configured key/value facts
       ▼
expand cfg_attr into zero or more attributes
       │ repeat if generated attributes include cfg_attr
       ▼
evaluate cfg on owning node
       ├── false → remove node and its expansion work
       └── true  → continue attribute classification

The exact rustc 1.97.1 implementation may combine or iterate these operations differently. The semantic obligations are stable enough to audit:

1. predicate evaluation is deterministic for one compilation configuration;
2. false nodes do not register imports, derives, macro invocations, or semantic items;
3. cfg_attr preserves the written order of attributes it contributes;
4. malformed predicates receive source-local diagnostics;
5. expansion cannot loop forever through recursively produced configuration attributes;
6. configuration facts used for incremental work are tracked as dependencies.

Concrete trace#

#[cfg_attr(feature = "wire", derive(Clone, Debug))]
#[cfg(any(unix, target_os = "wasi"))]
struct Packet(u32);

With feature="wire" and a Unix target:

1. Parsing records both attributes and the item.
2. cfg_attr's predicate is true, so an ordered derive(Clone, Debug) request appears.
3. The item's cfg predicate is true, so the item survives.
4. Derive scheduling records Clone then Debug under the derive-order contract.
5. Each expansion sees the original derive input plus allowed helper attributes.
6. Generated items are parsed, integrated, and scanned for further work.

With neither Unix nor WASI, the item is removed and no derive executes. If the compiler loads and runs the derive anyway, configuration happened too late.

Configuration is not textual preprocessing#

Rust must still lex and parse enough syntax to delimit a configured-away item. Therefore #[cfg(FALSE)] cannot hide arbitrary unmatched braces. Conversely, later name and type errors in a well-formed removed item generally should not occur. The boundary is structural recognition before removal, semantic participation after survival.

61. Feature gating: recognition is not permission#

A feature gate lets the compiler recognize unstable syntax or behavior without promising it on stable Rust. The parser may record a feature-use site while constructing an AST node. A gate checker later consults crate features and release-channel policy. Attribute syntax may itself be stable while a particular built-in attribute, argument, or placement is gated.

Keep three decisions separate:

Rejecting gated syntax as “unexpected token” destroys a precise stability diagnostic. Checking only handwritten syntax allows a macro to bypass the gate. Blindly gating compiler-synthesized internal syntax may reject implementation details users never requested. The gate boundary therefore needs provenance and an explicit policy for generated forms.

Around rustc 1.97.1, inspect feature registration, AST gate visiting, attribute gate metadata, expansion marks, and tests rather than relying on a remembered function order. The durable invariant is that stable users cannot acquire unstable language semantics merely by moving syntax through a macro.

Audit questions#

• Is the use site recorded even after parser recovery?
• Does cfg removal occur before irrelevant gate diagnostics?
• Are generated spans attributed to a useful call site?
• Can a proc macro manufacture the internal form without passing the public gate?
• Is the gate tested on stable, nightly without feature, nightly with feature, and configured-away code?
• Does an edition alter recognition without silently altering stability?

62. Expansion scheduling with attributes in the work queue#

Expansion is a fixed-point computation coupled to macro resolution. Function-like invocations, derives, and attribute macros have different input and integration rules, but all can emit more expandable syntax.

A conceptual scheduler maintains:

pending invocation: identity, kind, path, input, expected fragment, parent expansion, source order
resolution state: ready | blocked on names/imports | failed
integration site: placeholder or owner node
budgets: depth, invocations, input/output tokens, diagnostics, wall/cancellation policy

One safe scheduling outline is:

1. Configure newly visible AST.
2. Collect active attributes and macro invocations in deterministic order.
3. Resolve work that can currently resolve.
4. Expand one unit under a fresh expansion identity.
5. Parse output in its required fragment context.
6. Integrate transactionally.
7. Configure and validate newly generated syntax.
8. Collect nested work and update resolution.
9. Stop at a fixed point or diagnose blocked work.

This is conceptual, not a claim that rustc 1.97.1 has one loop with these exact numbered calls. Resolver callbacks and eager built-ins complicate the implementation.

Attribute order is observable#

An attribute macro receives the attributed item after rules about which attribute is selected and which remaining attributes are included in the item token stream. Reordering active attributes can change output. Derives also have an ordering contract for generated expansion and helper visibility. Optimization may parallelize only if it preserves the language's relevant order, resolution effects, diagnostics policy, and deterministic integration.

Transactional integration#

Do not publish half an expansion. Parse output into temporary structures, validate required local invariants, allocate identities according to compiler policy, then replace the placeholder. On failure, install an explicit error result and retain provenance. Partial publication produces duplicate definitions, orphaned NodeIds, and cascades.

63. Derive macros: additive generation with shared input#

A custom derive is declared in a proc-macro crate and named in #[derive(Path)]. Its stable input is a proc_macro::TokenStream representing the item being derived, including information exposed by the language contract. Its output is tokens for items to add; it does not replace the original item.

That additive contract distinguishes derive from an attribute macro:

original item ───────────────────────────────┐
      │ tokenized input                     │ retained
      ▼                                     │
derive macro → generated item tokens → parse/integrate
      │                                     │
      └──────── output appended conceptually┴→ expanded module

The diagram says “appended conceptually,” not “string-concatenated at EOF.” Rustc integrates output at the correct expansion site with identities, spans, hygiene, and nested-work discovery.

Derive helper attributes#

A derive declaration can register helper attribute names. Those attributes are inert with respect to ordinary expansion but meaningful to the registered derive macros. For example, a serialization derive might accept #[wire(rename = "id")].

The helper namespace creates ordering obligations:

• helper recognition depends on derives declared for the item;
• a helper should not be rejected as unknown before derive registrations are known;
• relevant derives can inspect helper tokens;
• the helper does not become a globally available attribute macro;
• duplicate and argument policy belongs primarily to the derive, unless the language/compiler reserves part of it.

Helper names from multiple derives can overlap. Do not assume one global owner without consulting the stable contract and current diagnostics policy. The macro should reject malformed helper input with a useful span rather than silently defaulting.

Derive ordering and output integration#

For #[derive(A, B, C)], preserve the defined source ordering of requests and a deterministic order of resulting items. Each derive's logical input is the original attributed item under the derive contract, not the prior derive's generated output as a rewritten replacement. Generated output can nevertheless affect later resolution once integrated.

Trace #[derive(A, B)] struct S;:

1. Configuration retains S.
2. Derive paths and helper declarations are resolved.
3. A receives S's derive input and returns item tokens.
4. Output A is parsed as items, marked with expansion A, and integrated.
5. Nested work from A is queued.
6. B receives the contractually appropriate input for S, not arbitrary output A tokens.
7. Output B is similarly integrated.
8. Duplicate implementation errors, if any, emerge from ordinary later semantics with macro provenance.

Do not overstate exact interleaving of steps 4–7 without checking rustc 1.97.1 source. The observable requirements are source-consistent scheduling, correct inputs, additive retention, and deterministic integration.

64. Attribute macros: replacement contracts and recursive responsibility#

An attribute procedural macro function receives two stable token streams:

// Interface shape only; this declaration requires a proc-macro crate.
// #[proc_macro_attribute]
// pub fn route(args: proc_macro::TokenStream,
//              item: proc_macro::TokenStream) -> proc_macro::TokenStream;

args represents tokens inside the attribute's delimiter, excluding the attribute path itself. item represents the attributed syntax according to the procedural-macro contract. The returned stream replaces the attributed item. Returning an empty stream deletes it. Returning the input preserves it. Returning the input plus generated tokens is the common “decorate” pattern.

The macro is responsible for preserving anything it wants retained. If it parses an item into a third-party AST and prints it again, it may alter token spelling, delimiter choices, or spans while preserving broad semantics. That is library policy, not a rustc guarantee.

Outer-attribute trace#

#[audit(level = "full")]
#[inline]
fn send(x: Packet) { transmit(x) }

A useful investigation records:

1. which active attribute is selected first;
2. what exact tokens appear in args;
3. whether #[inline] appears in item input;
4. which spans each token exposes through stable APIs;
5. whether returned #[inline] is later retained and validated;
6. how nested macro calls in the returned body enter the queue.

Do not infer these details from a pretty-printed expansion. Write an integration proc macro that serializes token kinds to generated compile_error! text or test helper output, then pin the exact toolchain.

Output contract#

Rustc treats output as Rust tokens, not a trusted AST. It must parse in the expected context. An attribute attached to an item must produce syntax acceptable for that integration site. Malformed output is diagnosed with expansion provenance. Generated active attributes are scheduled recursively rather than treated as inert text.

Invariant: once replacement commits, the original node is no longer independently semantically visible unless the macro returned an equivalent copy.

65. Stable tokens are not rustc_ast tokens#

The stable proc_macro API deliberately exposes a small token model. Its central types include TokenStream and TokenTree variants for Group, Ident, Punct, and Literal. Groups carry a delimiter and stream. Punctuation carries a character, spacing, and span. Identifiers and literals carry spelling-like information and spans.

Rustc internally uses richer and revision-specific token, AST, symbol, span, hygiene, and diagnostic structures. The bridge translates between them. Never expose an internal enum discriminant or serialized layout as if it were stable.

TokenStream::to_string() is display-oriented and not a lossless serialization promise. Parsing its output may be useful but is not identity-preserving for spans and may not preserve every surface distinction. Use token trees for transformations.

Punctuation decomposition#

The stable model represents operators such as += as punctuation trees with spacing, rather than one arbitrary operator enum. The bridge must split internal compound representations when needed and reconstruct parser meaning later without inventing whitespace. Negative literals deserve special care: a minus can be punctuation adjacent to a positive literal rather than part of the literal token.

Span boundary#

A stable Span is an opaque capability with documented operations, not byte offsets plus a public hygiene stack. Methods available on stable, nightly, or under feature gates evolve. For rustc 1.97.1 claims, read that toolchain's standard-library documentation and library/proc_macro source. Internally, rustc must retain enough identity to map bridge handles back to source and hygiene information while preventing stale or forged handles.

66. Host and target are different machines#

A procedural macro executes during compilation. Therefore it is compiled for and loaded by the host running rustc, even when the user's ordinary crate is compiled for another target.

proc-macro source ── compile for HOST ──► host dynamic artifact
                                               │ load/execute
                                               ▼
target crate source ── target cfg/options ──► rustc ──► TARGET artifact

The arrows simplify build scripts and cross-compilation, but expose the key invariant: host executable code must not be linked as target application code merely because it is a dependency in Cargo's graph.

Directly relevant consequences:

• proc-macro dependencies are built in the host universe;
• their own ordinary dependencies must be host-compatible;
• target-only native libraries cannot automatically satisfy host macro loading;
• a crate graph may contain host and target builds of the same package with different configurations;
• environment and filesystem observations occur on the host;
• generated tokens are then compiled under the target crate's language/target context.

Crate loading resolves proc-macro exports from a compiler-compatible artifact and registers callable expanders. ABI and metadata details are internal, toolchain-specific contracts, not stable plugin APIs. A proc-macro artifact built for an incompatible compiler or host should fail cleanly rather than be interpreted as arbitrary bytes.

Do not broaden this chapter into Cargo's entire feature resolver. For an expansion bug, record host triple, target triple, proc-macro artifact identity, rustc revision, dependency features, and loader error.

67. Client, bridge, and server architecture#

The stable library code used by a proc macro cannot directly manipulate rustc's private in-memory types. A client/server bridge mediates operations. “Server” here means the compiler-side implementation of stable operations; it does not necessarily imply a network service or separate process.

macro crate
  proc_macro::TokenStream handles
         │ client calls: iterate, construct, span operation
         ▼
bridge protocol / dispatch
         │ validated operation + encoded values
         ▼
rustc server
  internal token streams, spans, symbols, diagnostics

The boundary may use handles, callbacks, encoded buffers, thread-local context, or a mixture around rustc 1.97.1. Read the current proc_macro bridge and server crates before naming a concrete transport.

Why a bridge exists#

It provides:

1. a stable conceptual API over unstable representations;
2. control over which span and diagnostic operations macros may perform;
3. conversion of owned stable values to compiler-managed data;
4. compatibility checks at macro loading;
5. one place to validate lengths, tags, and handles;
6. containment of macro panics at an expansion boundary.

It does not by itself provide sandboxing. If the macro executes in rustc's process with ordinary host privileges, it can use filesystem, network, environment, threads, and unsafe native dependencies to the extent the operating system allows.

Serialization obligations#

Even an in-process encoded protocol must treat decoding as a trust boundary:

• reject unknown variant tags;
• bounds-check lengths before allocation;
• limit recursive group depth;
• preserve token order and punctuation spacing;
• validate span handles and invocation lifetime;
• avoid platform-dependent integer assumptions;
• never turn malformed data into undefined behavior;
• associate every request with the correct expansion context;
• clean up server-owned handles on success, panic, or cancellation.

Serialization can forget internal details only if stable observations cannot reveal them. Forgetting hygiene context while retaining coordinates violates name resolution, even if diagnostics still look plausible.

68. A bounded stable-Rust token bridge, stage I: values and codec#

The following program is a complete stable-Rust educational model. It uses only the standard library. It models groups, identifiers, literals, punctuation spacing, opaque span handles, deterministic binary encoding, decoding limits, and an attribute dispatcher. It does not implement Rust lexing, parsing, hygiene, dynamic loading, proc_macro, or rustc's actual protocol.

First establish representation and decoding invariants:

• every decoded tag is known;
• every length fits both the input and configured budget;
• recursion depth is bounded;
• every span handle was issued by this server;
• punctuation is ASCII and explicitly carries jointness;
• complete decoding consumes the complete message.

use std::collections::{BTreeMap, BTreeSet};

#[derive(Clone, Copy, Debug, PartialEq, Eq)]
struct Span(u32);

#[derive(Clone, Copy, Debug, PartialEq, Eq)]
enum Delimiter { Parenthesis, Brace, Bracket, None }

#[derive(Clone, Copy, Debug, PartialEq, Eq)]
enum Spacing { Alone, Joint }

#[derive(Clone, Debug, PartialEq, Eq)]
enum Tree {
	Group { delimiter: Delimiter, stream: Vec<Tree>, span: Span },
	Ident { text: String, span: Span },
	Punct { ch: char, spacing: Spacing, span: Span },
	Literal { text: String, span: Span },
}

#[derive(Clone, Copy)]
struct Limits {
	max_depth: usize,
	max_trees: usize,
	max_string_bytes: usize,
	max_message_bytes: usize,
}

#[derive(Debug, PartialEq, Eq)]
enum Error {
	Cancelled,
	DuplicateAttribute(String),
	Invalid(String),
	Limit(&'static str),
	Macro(String),
	TrailingBytes,
	Truncated,
	UnknownAttribute(String),
}

fn put_u32(out: &mut Vec<u8>, n: u32) { out.extend_from_slice(&n.to_le_bytes()); }

fn put_string(out: &mut Vec<u8>, text: &str) -> Result<(), Error> {
	let n = u32::try_from(text.len()).map_err(|_| Error::Limit("string encoding"))?;
	put_u32(out, n);
	out.extend_from_slice(text.as_bytes());
	Ok(())
}

fn encode_tree(tree: &Tree, out: &mut Vec<u8>, depth: usize, limits: Limits) -> Result<(), Error> {
	if depth > limits.max_depth { return Err(Error::Limit("encode depth")); }
	match tree {
		Tree::Group { delimiter, stream, span } => {
			out.push(0);
			out.push(match delimiter {
				Delimiter::Parenthesis => 0, Delimiter::Brace => 1,
				Delimiter::Bracket => 2, Delimiter::None => 3,
			});
			put_u32(out, span.0);
			put_u32(out, u32::try_from(stream.len()).map_err(|_| Error::Limit("group length"))?);
			for child in stream { encode_tree(child, out, depth + 1, limits)?; }
		}
		Tree::Ident { text, span } => {
			out.push(1); put_u32(out, span.0); put_string(out, text)?;
		}
		Tree::Punct { ch, spacing, span } => {
			if !ch.is_ascii_punctuation() { return Err(Error::Invalid("non-ASCII punctuation".into())); }
			out.push(2); put_u32(out, span.0); put_u32(out, *ch as u32);
			out.push(match spacing { Spacing::Alone => 0, Spacing::Joint => 1 });
		}
		Tree::Literal { text, span } => {
			out.push(3); put_u32(out, span.0); put_string(out, text)?;
		}
	}
	if out.len() > limits.max_message_bytes { return Err(Error::Limit("encoded message")); }
	Ok(())
}

struct Decoder<'a> {
	bytes: &'a [u8],
	at: usize,
	count: usize,
	limits: Limits,
	valid_spans: &'a BTreeSet<u32>,
}

impl Decoder<'_> {
	fn byte(&mut self) -> Result<u8, Error> {
		let value = *self.bytes.get(self.at).ok_or(Error::Truncated)?;
		self.at += 1;
		Ok(value)
	}

	fn u32(&mut self) -> Result<u32, Error> {
		let end = self.at.checked_add(4).ok_or(Error::Truncated)?;
		let bytes: [u8; 4] = self.bytes.get(self.at..end)
			.ok_or(Error::Truncated)?.try_into().map_err(|_| Error::Truncated)?;
		self.at = end;
		Ok(u32::from_le_bytes(bytes))
	}

	fn span(&mut self) -> Result<Span, Error> {
		let raw = self.u32()?;
		if !self.valid_spans.contains(&raw) {
			return Err(Error::Invalid(format!("unknown span handle {raw}")));
		}
		Ok(Span(raw))
	}

	fn string(&mut self) -> Result<String, Error> {
		let n = usize::try_from(self.u32()?).map_err(|_| Error::Limit("string length"))?;
		if n > self.limits.max_string_bytes { return Err(Error::Limit("string bytes")); }
		let end = self.at.checked_add(n).ok_or(Error::Truncated)?;
		let raw = self.bytes.get(self.at..end).ok_or(Error::Truncated)?;
		self.at = end;
		String::from_utf8(raw.to_vec()).map_err(|_| Error::Invalid("UTF-8".into()))
	}

	fn tree(&mut self, depth: usize) -> Result<Tree, Error> {
		if depth > self.limits.max_depth { return Err(Error::Limit("decode depth")); }
		self.count += 1;
		if self.count > self.limits.max_trees { return Err(Error::Limit("tree count")); }
		match self.byte()? {
			0 => {
				let delimiter = match self.byte()? {
					0 => Delimiter::Parenthesis, 1 => Delimiter::Brace,
					2 => Delimiter::Bracket, 3 => Delimiter::None,
					other => return Err(Error::Invalid(format!("delimiter tag {other}"))),
				};
				let span = self.span()?;
				let n = usize::try_from(self.u32()?).map_err(|_| Error::Limit("group length"))?;
				if n > self.limits.max_trees.saturating_sub(self.count) {
					return Err(Error::Limit("declared group length"));
				}
				let mut stream = Vec::with_capacity(n);
				for _ in 0..n { stream.push(self.tree(depth + 1)?); }
				Ok(Tree::Group { delimiter, stream, span })
			}
			1 => { let span = self.span()?; let text = self.string()?; Ok(Tree::Ident { text, span }) }
			2 => {
				let span = self.span()?;
				let ch = char::from_u32(self.u32()?).ok_or_else(|| Error::Invalid("punct scalar".into()))?;
				if !ch.is_ascii_punctuation() { return Err(Error::Invalid("punct class".into())); }
				let spacing = match self.byte()? {
					0 => Spacing::Alone, 1 => Spacing::Joint,
					other => return Err(Error::Invalid(format!("spacing tag {other}"))),
				};
				Ok(Tree::Punct { ch, spacing, span })
			}
			3 => { let span = self.span()?; let text = self.string()?; Ok(Tree::Literal { text, span }) }
			other => Err(Error::Invalid(format!("tree tag {other}"))),
		}
	}
}

fn decode(bytes: &[u8], limits: Limits, spans: &BTreeSet<u32>) -> Result<Vec<Tree>, Error> {
	if bytes.len() > limits.max_message_bytes { return Err(Error::Limit("input message")); }
	let mut d = Decoder { bytes, at: 0, count: 0, limits, valid_spans: spans };
	let n = usize::try_from(d.u32()?).map_err(|_| Error::Limit("stream length"))?;
	if n > limits.max_trees { return Err(Error::Limit("stream count")); }
	let mut stream = Vec::with_capacity(n);
	for _ in 0..n { stream.push(d.tree(0)?); }
	if d.at != bytes.len() { return Err(Error::TrailingBytes); }
	Ok(stream)
}

fn encode(stream: &[Tree], limits: Limits) -> Result<Vec<u8>, Error> {
	let mut out = Vec::new();
	put_u32(&mut out, u32::try_from(stream.len()).map_err(|_| Error::Limit("stream length"))?);
	for tree in stream { encode_tree(tree, &mut out, 0, limits)?; }
	Ok(out)
}

type MacroFn = fn(&[Tree], Vec<Tree>, &mut Context) -> Result<Vec<Tree>, Error>;

struct Context {
	cancelled: bool,
	remaining_output: usize,
	diagnostics: Vec<String>,
}

struct Dispatcher {
	macros: BTreeMap<String, MacroFn>,
}

impl Dispatcher {
	fn new() -> Self { Self { macros: BTreeMap::new() } }

	fn register(&mut self, name: &str, function: MacroFn) -> Result<(), Error> {
		if self.macros.insert(name.into(), function).is_some() {
			return Err(Error::DuplicateAttribute(name.into()));
		}
		Ok(())
	}

	fn expand(&self, name: &str, args: &[Tree], item: Vec<Tree>, cx: &mut Context)
		-> Result<Vec<Tree>, Error>
	{
		if cx.cancelled { return Err(Error::Cancelled); }
		let function = self.macros.get(name)
			.ok_or_else(|| Error::UnknownAttribute(name.into()))?;
		let output = function(args, item, cx)?;
		if output.len() > cx.remaining_output { return Err(Error::Limit("macro output")); }
		cx.remaining_output -= output.len();
		Ok(output)
	}
}

fn preserve(args: &[Tree], item: Vec<Tree>, cx: &mut Context) -> Result<Vec<Tree>, Error> {
	if !args.is_empty() {
		cx.diagnostics.push("preserve ignores arguments".into());
	}
	Ok(item)
}

fn require_public(_args: &[Tree], item: Vec<Tree>, _cx: &mut Context) -> Result<Vec<Tree>, Error> {
	match item.first() {
		Some(Tree::Ident { text, .. }) if text == "pub" => Ok(item),
		_ => Err(Error::Macro("expected item beginning with pub".into())),
	}
}

fn main() {
	let limits = Limits { max_depth: 8, max_trees: 100, max_string_bytes: 64, max_message_bytes: 4096 };
	let span = Span(7);
	let stream = vec![
		Tree::Ident { text: "pub".into(), span },
		Tree::Punct { ch: ':', spacing: Spacing::Joint, span },
		Tree::Punct { ch: ':', spacing: Spacing::Alone, span },
		Tree::Group { delimiter: Delimiter::Brace, stream: vec![], span },
	];
	let encoded = encode(&stream, limits).expect("encode valid stream");
	let spans = BTreeSet::from([7]);
	assert_eq!(decode(&encoded, limits, &spans), Ok(stream.clone()));

	let mut dispatcher = Dispatcher::new();
	dispatcher.register("preserve", preserve).expect("unique registration");
	dispatcher.register("require_public", require_public).expect("unique registration");
	assert!(matches!(dispatcher.register("preserve", preserve), Err(Error::DuplicateAttribute(_))));

	let mut cx = Context { cancelled: false, remaining_output: 20, diagnostics: vec![] };
	let output = dispatcher.expand("require_public", &[], stream.clone(), &mut cx)
		.expect("public item");
	assert_eq!(output, stream);
	assert_eq!(cx.remaining_output, 16);

	let mut bad = encoded.clone();
	bad.push(99);
	assert_eq!(decode(&bad, limits, &spans), Err(Error::TrailingBytes));
	let wrong_spans = BTreeSet::new();
	assert!(matches!(decode(&encoded, limits, &wrong_spans), Err(Error::Invalid(_))));

	let mut cancelled = Context { cancelled: true, remaining_output: 20, diagnostics: vec![] };
	assert_eq!(dispatcher.expand("preserve", &[], vec![], &mut cancelled), Err(Error::Cancelled));
}

Compile it as one file; fragments earlier in the chapter are not independent programs. The use of BTreeMap makes registry iteration deterministic, though this example never relies on iteration order. The codec uses explicit little-endian integers, validates before allocation, and rejects trailing bytes.

69. Stage II: trace the model and expose its omissions#

Trace the complete program's first tree:

The model deliberately omits:

1. real Rust tokenization and fragment parsing;
2. lifetimes for server handles beyond one invocation;
3. hygiene contexts and expansion ancestry;
4. span operations such as call-site/def-site transformations;
5. proc-macro dynamic loading and ABI negotiation;
6. panic catching and thread cleanup;
7. nested output token counting—its budget counts only roots;
8. byte budgets for in-memory strings after decoding;
9. diagnostics with levels, labels, suggestions, and spans;
10. subprocess isolation, timeouts, and operating-system sandboxing;
11. derive registration/helper attributes;
12. cfg, feature gates, parser recovery, and AST integration.

These omissions prevent the model from being called a procedural-macro implementation. They also define a progression:

• Replace root counting with iterative total-tree counting using checked addition.
• Give each invocation a nonce and store (nonce, local_handle) for spans.
• Add a Cancelled check in long encode/decode and output walks.
• Catch panics around only the macro call, then discard the transaction.
• Parse output with a tiny declared item grammar before commit.
• Add derive mode whose output is additive and attribute mode whose output replaces.
• Record a parent expansion ID and print a deterministic trace.

Tests to add before features#

Use table tests for every tag and delimiter. Truncate every valid encoded message at every byte offset; each prefix must return an error, never panic. Mutate each tag to 255. Declare lengths near u32::MAX with tiny input. Generate nesting exactly at and one beyond the limit. Generate invalid UTF-8. Use valid and invalid span handles. Check that a failed expansion does not decrement the committed budget or publish output; the current model needs a policy decision here because it charges only successful returned output.

Property test without a crate: enumerate trees from a tiny alphabet and assert decode(encode(x)) == x under limits. Do not assert encode(decode(bytes)) == bytes for arbitrary bytes unless canonical encoding is a declared contract.

70. Panic, diagnostics, cancellation, and resource behavior#

A proc macro is executable build-time code. It can panic, loop, allocate until failure, spawn threads, abort the process, call native code, inspect secrets, or emit enormous output. The stable API contract and rustc's current containment are not equivalent to a security sandbox.

Panic#

The compiler can catch an ordinary unwind crossing the macro invocation boundary and convert it to a diagnostic. It should associate failure with the invocation and avoid integrating partial output. But abort, process termination, memory corruption, and some foreign-code failures cannot be repaired by catch_unwind. Never claim “proc-macro panics are safe” without naming the panic strategy and failure class.

Diagnostics#

A macro can encode compile_error! in output on stable Rust. Additional diagnostic APIs have historically had varying stability; verify 1.97.1 documentation. Compiler diagnostics should preserve macro backtraces and avoid duplicate “failed to expand” cascades.

Diagnostic budgets matter. A malicious macro can emit thousands of errors even with tiny token output. Cap or coalesce repeated diagnostics while retaining the first useful provenance. Deterministic ordering is part of reproducible user experience even when exact diagnostic text is not a language guarantee.

Cancellation#

Cancellation is cooperative inside an in-process macro unless execution is isolated strongly enough to terminate a worker. Rustc can check before and after bridge calls, but arbitrary macro computation between calls may ignore cancellation. Killing a thread safely is not generally available. A separate process offers a stronger termination boundary but introduces protocol, startup, filesystem, and compatibility costs.

Resource dimensions#

One recursion limit is insufficient:

Limits are policy, not merely mechanism. Too-low limits reject legitimate code generation; unbounded limits expose denial of service. Report which budget was exceeded, its configured value, invocation provenance, and a way to reduce or intentionally adjust work where supported.

71. Security and nondeterminism are build-system properties#

Procedural macros commonly run with the invoking build user's authority. Treat adding one as adding a build-time executable dependency. Review transitive dependencies, native code, release provenance, and update policy.

Threats include:

• reading environment variables containing credentials;
• reading source files outside the intended crate;
• writing generated or persistent state;
• network access and dependency exfiltration;
• command execution;
• exploiting compiler bridge or loader bugs;
• output designed to trigger compiler worst cases;
• nondeterministic output that poisons caches or reproducibility.

The compiler cannot make arbitrary in-process code safe merely through a narrow token API. Meaningful isolation needs operating-system enforcement: process identity, filesystem view, network policy, syscall controls, memory/CPU quotas, and a kill boundary. That architecture costs portability, startup time, debugging convenience, and protocol maintenance.

Sources of nondeterminism#

Wall clock, random numbers, hash-map iteration, directory order, network responses, process IDs, locale, host paths, and environment variables can all alter generated tokens or diagnostics. Spans can also embed host-specific paths in diagnostics.

A reproducible macro should:

1. derive output only from declared token input and explicitly tracked files/environment;
2. sort unordered external data;
3. avoid time/randomness or accept an explicit seed;
4. normalize paths only under a documented policy;
5. produce deterministic diagnostics;
6. expose all relevant observations to the build system's dependency tracking.

Caching expansion by input token text alone is unsound if the macro reads files, environment, compiler version, host state, or spans. A sound cache key needs every observable dependency plus stable semantics for span rebasing and diagnostics. If those facts cannot be enumerated, do not guess a cache hit.

72. Failure map: find the earliest broken invariant#

Triage in this order:

1. Pin rustc revision, host, target, edition, flags, and crate graph.
2. Minimize while preserving attribute order and macro provenance.
3. Classify every attribute as configured away, built-in inert, active macro, derive, helper, or unresolved.
4. Trace scheduling and expansion IDs.
5. Capture stable token-tree kinds, delimiters, spacing, and available spans.
6. Trace bridge conversion and handle lifetime.
7. Inspect output parsing and transactional integration.
8. Verify post-expansion validation and gates.
9. Only then investigate resolution, types, or code generation.

The first visible type error can be caused by an attribute macro dropping one token much earlier. Later diagnostics are evidence; they are not proof of ownership.

73. Production hardening and test matrix#

Test each boundary with the narrowest oracle.

Attribute matrix#

For a changed attribute rule, vary:

• outer and inner forms;
• crate, module, item, field, statement, expression, and unsupported positions;
• handwritten and macro-generated use;
• retained and false-cfg use;
• single, duplicate, and conflicting uses;
• malformed empty, token, list, and name-value payloads;
• stable/nightly feature states where relevant;
• adjacent editions;
• ordinary, derive-helper, tool, and unresolved namespaces.

Proc-macro matrix#

Test empty input/output, very deep groups, very wide streams, every delimiter, joint punctuation chains, raw identifiers, unusual literals, generated attributes, nested derives, panic, compile_error!, malformed output, and cancellation. Use a cross-compilation job where host differs from target. Run sanitizer or platform-appropriate memory checks on unsafe bridge/loader code. Fuzz decoders independently of real dynamic libraries.

Performance model#

Let I be invocations, T total transferred token trees, B total token bytes, and H live handles. A basic bridge should aim for work near O(T + B) and memory bounded by retained streams plus H. Repeatedly flattening or cloning a stream at each nested group can become quadratic. Benchmark ordinary derives and adversarial depth/width separately; report host, build mode, revision, medians, tails, allocations, and output size.

Never optimize by dropping span provenance unless the observation model explicitly proves no semantic, diagnostic, or hygiene change.

74. Source navigation around rustc 1.97.1#

Begin from the exact rust-lang/rust revision used to build the compiler. Paths below are landmarks and may move.

1. compiler/rustc_ast/ — AST attributes, tokens, token streams, node forms.
2. compiler/rustc_parse/ — attribute parsing and token conversion.
3. compiler/rustc_expand/ — invocation collection, scheduling, derive/attribute expansion, integration, AST validation landmarks.
4. compiler/rustc_builtin_macros/ — active compiler-provided expanders.
5. compiler/rustc_attr_parsing/ and neighboring attribute crates — structured built-in parsing if present in the selected revision.
6. compiler/rustc_feature/ and session/configuration code — feature declarations and gating policy.
7. compiler/rustc_resolve/ — macro and import resolution feedback.
8. compiler/rustc_metadata/ and loader-related code — proc-macro crate metadata/loading landmarks.
9. proc-macro bridge/server crates under compiler/ — compiler-side stable API implementation.
10. library/proc_macro/ — stable-facing client API and bridge bindings.
11. compiler/rustc_span/ — spans, hygiene, expansion provenance, symbols.
12. tests/ui/attributes, proc-macro, derive, cfg, feature-gate, and incremental suites — executable policy.

Search by public concepts and types rather than assuming filenames:

rg "proc_macro_attribute|proc_macro_derive" compiler library tests
rg "cfg_attr|derive.*helper|builtin.*attr" compiler tests
rg "TokenStream|TokenTree|Spacing" library/proc_macro compiler
rg "catch_unwind|panic" compiler/rustc_expand compiler/*proc_macro*

For each claim, label its source:

• normative: Reference or stable standard-library documentation;
• current implementation: source/tests at the pinned 1.97.1 revision;
• historical architecture: development-guide text or design discussion;
• proposal: issue/PR not merged into that revision.

Do not use cargo expand output to prove hygiene, hidden attributes, scheduling internals, or exact stable token input. It is a useful presentation aid, not a complete front-end trace.

75. Contribution workshops#

Workshop A: validation ownership#

Choose one AST validation diagnostic. Minimize it, then list every fact needed to decide it. Find where each fact first exists and where it is first discarded. Draw the valid scheduling interval. Add handwritten, generated, recovered, and false-configured tests. The patch is incomplete if it merely moves the diagnostic without proving downstream invariants.

Workshop B: helper attribute typo#

Create a derive with one registered helper. Test correct use, misspelling, use without derive, two derives registering the same helper, duplicate helper input, and false configuration. Trace whether rustc or macro code owns each message. Improve only the earliest owner with sufficient information.

Workshop C: bridge decoder#

Take the Chapter 68 model. Add a total decoded-byte budget and invocation nonce to span handles. Generate every truncated prefix and random unknown tag. Prove no allocation happens from an unvalidated length. Benchmark nested singleton groups versus one wide group. Explain why a recursion limit alone is insufficient.

Workshop D: transactional integration#

Instrument a local compiler so a test expander returns two items but the second fails parsing. Observe whether the first becomes visible. Identify temporary storage, identity assignment, and placeholder replacement. Write a regression asserting one primary expansion error and no leaked definition.

Workshop E: host/target failure#

Cross-compile a tiny crate using a tiny derive. Record which artifacts use host and target triples. Introduce a host-incompatible native dependency in a controlled branch. Classify whether failure belongs to dependency selection, compilation, metadata, loading, or execution. Do not “fix” it by linking host code into the target.

Workshop F: deterministic scheduling#

Construct two attribute macros that each generate a named item and diagnostic. Repeat clean builds and record invocation, integration, and diagnostic order. Replace internal unordered collections experimentally. Specify which observations must remain stable before proposing parallelism.

Contributor patch checklist#

• Pin and state revision.
• Add the narrow regression first.
• Explain ownership and ordering, not only changed function names.
• Preserve attribute source order and expansion provenance.
• Test generated and configured-away forms.
• Audit panic and cancellation paths for partial state.
• Audit lengths, recursion, allocation, and handle lifetime at bridge boundaries.
• Run neighboring UI, proc-macro, cross-target, and incremental tests.
• Measure representative and adversarial performance when touching traversal or serialization.
• Update current implementation documentation without turning internals into guarantees.

76. Capstone: evolve the bounded dispatcher honestly#

Build a mini front end with this bounded scope:

1. Reuse Chapter 68's token values and hardened codec.
2. Define Item { attrs, body, origin, state } where state is pending, staged, committed, removed, or error.
3. Implement cfg(false) and a simple boolean cfg_attr without pretending to parse Rust predicates.
4. Register inert built-ins, attribute macros, derives, and derive helpers in separate namespaces.
5. Process ordered active attributes through a deterministic queue.
6. Make attribute macros replace and derives add.
7. Parse output with a tiny item grammar such as item IDENT ;.
8. Stage all output and commit only after complete parse/validation.
9. Assign expansion IDs with parent links and nonce-bound span handles.
10. Enforce depth, jobs, trees, bytes, diagnostics, and live-handle limits.
11. Catch ordinary unwinding at the dispatcher boundary.
12. Add cancellation checks to every bounded traversal.
13. Produce a trace suitable for differential tests.

Required properties:

• every loop advances, blocks with a reason, or terminates;
• configured-away items create no work;
• one active attribute executes at most once per owner state;
• failed output publishes nothing;
• original derive input remains committed;
• attribute output replaces the original exactly once;
• helper attributes are visible only under registered derive scope;
• all decoded handles belong to the current invocation;
• output order is deterministic;
• every diagnosed generated token has an expansion ancestry.

Required negative tests include malformed tags, huge declared lengths, stale spans, duplicate registrations, unknown attributes, recursive output, panic after staged output, cancellation during nested decode, invalid generated grammar, and derive-generated active attributes.

Explicitly state omissions in the final README: no Rust grammar, real cfg vocabulary, stable proc_macro ABI, dynamic loading, mixed-site hygiene, source mapping, compiler feature gates, incremental compilation, OS sandbox, or compatibility claim with rustc. Omitting these honestly is a stronger engineering result than implementing misleading approximations.

77. Derived philosophy#

Attributes are delayed decisions#

Attribute syntax preserves a request until the stage with enough information can act. Making every attribute immediate simplifies parsing but prevents resolution, configuration, and later semantic consumers. Making every attribute late preserves flexibility but risks invalid state reaching consumers. Correct placement is an information interval.

Configuration changes existence#

cfg is not a boolean annotation on an otherwise ordinary item. It determines which syntax contributes to the configured program. That power creates an ordering obligation: enough parsing must happen to recover structure, while removed nodes must not leak expansion or semantic work.

Stable abstraction moves translation work#

The small proc_macro token model protects macro authors from rustc's internal churn. It does not delete complexity. The bridge must preserve every stable observation while adapting richer tokens, spans, hygiene, diagnostics, ownership, and lifetimes. Local simplicity in the public API creates a permanent compiler-side proof obligation.

Execution creates authority#

Calling a dependency “a macro” does not reduce its host privileges. Once arbitrary code executes, token API type safety is not process isolation. Security claims must name the operating-system boundary, resource controls, and observations allowed.

Caching creates correctness obligations#

Macro caching is correct only under an explicit observation model. Token input, spans, host files, environment, compiler revision, configuration, diagnostics, and nondeterministic state may all matter. An unrepresentable dependency is not permission to omit it from a cache key.

Earliest failures retain the best provenance#

A dropped punctuation spacing bit may surface as a parser error. A stale span may surface as wrong name resolution. A late cfg may surface as an irrelevant gate error. Debugging should walk backward to the first violated invariant, not patch the loudest downstream symptom.

78. Mastery checks, talk plans, and reading map#

A reader is ready to work in this area when they can:

• derive where an AST check belongs from its required facts and downstream assumptions;
• classify attributes independently as built-in/user and inert/active;
• trace cfg_attr, cfg, gating, collection, expansion, integration, and validation;
• explain helper-attribute scope without calling helpers attribute macros;
• distinguish additive derive output from replacing attribute output;
• preserve derive request and generated-item order without inventing an implementation guarantee;
• translate stable token-tree concepts without conflating them with rustc_ast enums;
• explain why stable spans are opaque handles and why hygiene still must survive;
• diagnose host/target proc-macro loading failures;
• threat-model panic, abort, hangs, output floods, stale handles, and nondeterminism separately;
• design transactional integration and multidimensional budgets;
• navigate the pinned compiler source and classify evidence correctly.

30-minute talk: one attribute through the compiler#

1. Parse #[cfg_attr(...)] as retained syntax.
2. Configuration determines the program.
3. Attribute classification requires namespaces and resolution.
4. Derive and attribute macros have different contracts.
5. Stable trees cross an unstable internal boundary.
6. Host execution generates target syntax.
7. Integration restores AST invariants.
8. One failure map demonstrates earliest-invariant debugging.

60-minute compiler talk#

1. The five attribute jobs.
2. Validation ownership as an information partial order.
3. Configuration and feature gates.
4. Expansion fixed point and deterministic scheduling.
5. Derive helper registration and additive integration.
6. Attribute replacement and recursive expansion.
7. Stable/internal token comparison.
8. Client/bridge/server protocol and span handles.
9. Host/target loading.
10. Panic, cancellation, security, and reproducibility.
11. Bounded bridge demo and adversarial tests.
12. Contribution workflow at a pinned revision.

Authoritative reading map#

• Rust Reference: attributes — syntax, placement, categories, and built-in attribute links.
• Rust Reference: conditional compilation — normative cfg and cfg_attr behavior.
• Rust Reference: procedural macros — macro kinds, token contracts, hygiene, and failure behavior.
• Standard proc_macro documentation — stable API for the selected toolchain.
• Cargo Reference: build scripts and dependencies — host-executed build context; combine with Cargo dependency documentation for the exact graph question.
• rustc development guide: macro expansion — architecture map, not a substitute for current source.
• rustc development guide: stability — implementation orientation for feature stability.
• rust-lang/rust source — authority for rustc 1.97.1 implementation claims.
• Neighboring rust-lang/rust UI and proc-macro tests — executable behavior and diagnostic policy at the pinned revision.

Read the Reference and stable library docs for guarantees. Read pinned source and tests for current mechanics. Use the development guide to form search hypotheses. Treat issues and old design documents as history or proposals until the selected source confirms them.

79. A subsystem audit worksheet#

A broad compiler audit becomes useful only when every box names an invariant, evidence, and owner. The following worksheet turns “check procedural macros” into bounded investigations. It is intentionally redundant at boundaries: disagreements between adjacent owners reveal gaps.

Syntax and validation#

1. Lexing: attribute punctuation and delimiters retain correct spans and spacing.
2. Token-tree grouping: malformed delimiters terminate with one owned recovery path.
3. Attribute parsing: path, style, and payload survive without premature stringification.
4. Attachment: outer and inner attributes bind to the grammar-defined owner.
5. Recovery: synthesized owners carry an error marker and cannot trigger unsafe assumptions.
6. Initial validation: accepted-superset forms receive a diagnostic before a trusting consumer.
7. Generated validation: macro output receives equivalent structural coverage.
8. Built-in registry: template, target, duplication, gate, and activity metadata agree.
9. Unknown names: tool, helper, macro, and built-in namespaces are considered before error.
10. Duplicate policy: diagnostics retain both source locations and respect configuration.
11. Feature use recording: use-site provenance survives parsing and expansion.
12. Gate checking: generated user-visible syntax cannot bypass stability policy.

Configuration and scheduling#

1. Predicate parsing: malformed cfg input cannot become a guessed boolean.
2. Predicate evaluation: key/value facts come from the compilation configuration.
3. cfg_attr rewriting: contributed attributes preserve source order.
4. Node removal: false nodes create no imports, macros, derives, or semantic definitions.
5. Dependency tracking: configuration observations invalidate reused compilation work.
6. Invocation collection: every active form is collected once with an owner identity.
7. Macro resolution: blocked work is distinguishable from definitively unresolved work.
8. Resolver feedback: generated imports and macro definitions can make progress visible.
9. Queue order: relevant source order remains deterministic.
10. Fixed-point termination: no-progress and resource exhaustion have distinct diagnostics.
11. Expansion ancestry: every job has a fresh ID and correct parent.
12. Placeholder ownership: one integration site is replaced exactly once.

Derive and attribute contracts#

1. Derive path resolution: aliases and imports follow macro namespace rules.
2. Derive input: each macro receives the contractually complete original item.
3. Derive retention: original syntax remains semantically present.
4. Derive output: output parses as additive items in the correct context.
5. Derive ordering: requests and generated results obey defined observable order.
6. Helper registration: helper names are scoped to declared derives.
7. Helper visibility: all relevant derive invocations can inspect helper tokens.
8. Helper diagnostics: rustc does not steal macro-owned payload policy prematurely.
9. Attribute arguments: the first stream excludes the active attribute path as specified.
10. Attributed input: remaining syntax and attributes cross the stable contract correctly.
11. Replacement: empty, identity, and decorating outputs have distinct expected effects.
12. Recursive output: generated active attributes re-enter scheduling rather than inert storage.

Stable token boundary#

1. Stream order: iteration and reconstruction preserve token-tree order.
2. Groups: delimiter kind, child stream, and span survive conversion.
3. Identifiers: spelling, rawness behavior, and span operations follow stable docs.
4. Punctuation: character and Alone/Joint spacing round-trip.
5. Literals: bridge conversion does not assume every display string is source spelling.
6. Negative values: minus punctuation is not accidentally fused into a changed contract.
7. Opaque spans: no internal address or unchecked index escapes as authority.
8. Hygiene: stable span observations map to correct internal syntax contexts.
9. Display: to_string is never used as a lossless protocol silently.
10. Ownership: cloned/dropped streams cannot invalidate live server state.
11. Compatibility: loader and bridge versions reject incompatible artifacts cleanly.
12. Unknown protocol data: tags, lengths, and handles fail closed without panic.

Host execution and loading#

1. Dependency graph: proc-macro code and its executable dependencies build for host.
2. Target separation: generated code is interpreted under the target crate context.
3. Artifact selection: loader chooses the matching host/compiler artifact.
4. Metadata: exported macro kind, name, and helper declarations decode consistently.
5. Dynamic loading: missing symbols and ABI mismatch become bounded errors.
6. Invocation context: bridge calls cannot leak into another concurrent expansion.
7. Thread context: macro-created threads do not inherit invalid bridge authority accidentally.
8. Cleanup: unload/drop paths reclaim invocation-local handles.

Failure containment#

1. Ordinary unwind: panic becomes one expansion failure with provenance.
2. Abort: documentation does not promise recovery that the process cannot provide.
3. Partial output: no staged AST is visible after any macro failure.
4. Cancellation: checks exist at compiler-controlled long operations.
5. Uncooperative loops: architecture states whether a worker can be terminated.
6. Depth budget: recursive ancestry is bounded.
7. Width budget: broad output and invocation fan-out are bounded.
8. Byte budget: literals and encoded messages are charged, not merely tree roots.
9. Allocation budget: declared lengths are validated before reserve/allocation.
10. Diagnostic budget: floods are limited without losing the primary cause.
11. Handle budget: tables cannot grow without an invocation-scoped ceiling.

Security, determinism, and maintenance#

1. Host authority: documentation treats macros as executable dependencies, not sandboxes.
2. External observations: files, environment, network, and time are threat-modeled.
3. Reproducibility: unordered inputs are sorted and undeclared inputs are exposed.
4. Cache keys: every permitted observation is represented or caching is disabled.
5. Observability: traces identify macro, expansion parent, token counts, and elapsed work safely.
6. Regression ownership: the narrowest unit, UI, integration, fuzz, or cross-target test pins the fix.

For each row, record:

An audit is complete only when cross-row contradictions are resolved. For example, if row 16 says removed nodes create no jobs but row 18 collects before configuration, the implementation needs either an explicit filtered collection contract or an ordering correction. If row 43 validates handles but row 55 lets child threads retain them after invocation teardown, integer validation alone does not establish lifetime safety.

Final prediction exercise#

A derive on a false-configured item registers a helper attribute, reads an environment variable, emits joint punctuation with a call-site span, and then panics after producing one item. Predict the correct observations before testing:

• configuration prevents the derive from executing, so no helper registration is needed for that removed owner;
• no environment observation should enter dependency tracking from an invocation that never occurred;
• no output token, punctuation spacing, or span reaches integration;
• no panic occurs because macro code was never called;
• later stages see neither the original configured-away item nor partial derive output.

If any observation differs, begin with configuration-versus-collection ordering. Do not start by changing panic handling or token serialization: they are downstream and should have been unreachable.

Part III: Names, Definitions, Lowering, and HIR#

This part follows source text from meaningful names to rustc's High-level Intermediate Representation. It targets the concepts exposed by rustc 1.97.1, dated July 2026. Compiler internals are not a stable API, so details explicitly called version-sensitive may move or change. The goal is not merely to list data structures. We will derive why they exist, which promises they make, and how failures look.

1. Parsing Gives Shape, Not Meaning#

A parser answers grammatical questions. Given f(x), it can recognize a call expression whose callee is a path named f. It cannot generally decide which declaration introduced f. That question depends on source outside the expression, imported crates, and generated macro output.

Consider a tiny program.

fn watts() -> u32 { 60 }

fn main() {
	let watts = 5;
	println!("{}", watts);
}

The first watts declares a function. The second declares a local variable and shadows the function in the rest of its scope. The third is a use and denotes the local variable. All three identifiers have the same spelling, and the parser represents their surrounding syntax. Resolution connects each use to the declaration that gives it meaning.

A declaration introduces a binding: an association between a name and an entity. A use asks for a binding in a particular context. A scope is the region where a binding is eligible to answer that request. Shadowing means a nearer eligible binding hides a farther binding with the same name. These definitions are deliberately independent of implementation terminology.

Why not resolve while parsing? Rust permits many items to be used before their textual declaration. Imports can form chains, glob imports can reveal unknown sets of names, and macros can create declarations. Parsing one token at a time therefore lacks the required future and generated context. An eager parser would need backtracking and repeated mutation, coupling grammar to semantic policy.

fn first() -> Packet { second() }

struct Packet;

fn second() -> Packet { Packet }

When first is parsed, both Packet and second occur later. Rust intentionally treats item scope differently from sequential local binding scope. By contrast, the following first use is invalid.

fn main() {
	println!("{answer}"); // no local `answer` yet
	let answer = 42;
}

Separating parsing and resolution also improves recovery. The parser can construct an error-marked syntax tree even when a name is unknown. The resolver can then report an unresolved name at its use span and suggest nearby spellings. The key invariant is that every successfully resolved use identifies an appropriate declaration, not merely another token with equal text.

A parser bug tends to produce “expected expression” or a malformed tree. A resolver bug instead produces “cannot find” for valid syntax, selects a shadowed declaration, or attaches a diagnostic to a syntactically valid but semantically wrong location. Keeping those symptom families separate is the first useful debugging skill.

2. Bindings, Scope, and Shadowing from First Principles#

Imagine resolving locals with a stack of dictionaries. Entering a block pushes a dictionary; leaving it pops that dictionary. A declaration inserts into the current dictionary. A use searches from the newest dictionary toward the oldest. This simple model explains lexical scope and ordinary shadowing.

fn main() {
	let color = "blue";
	{
		let color = 17;
		assert_eq!(color, 17);
	}
	assert_eq!(color, "blue");
}

The inner binding does not destroy or modify the outer binding. It merely wins lookup while its scope is active. This matters because later analyses assign the two bindings unrelated types and storage.

Rust's real scopes are not one uniform stack. Items in a block are visible throughout that block, including before their declarations. Most let bindings become visible only after their pattern has been initialized. Generic parameters are visible in selected parts of an item. Loop labels have their own lookup rules. Macro hygiene can distinguish equal textual names by syntax context.

rustc models many late lexical scopes with stacks of structures called ribs. A rib is an implementation abstraction for a point where available bindings or crossing rules change. Separate stacks serve different namespaces. Ribs can also record boundaries that lookup may not transparently cross. For example, a nested function item cannot capture an enclosing function's local.

fn outer() {
	let local = 3;
	let closure = || local + 1; // closure capture is allowed
	fn nested() -> i32 {
		// local + 1 would be an error: a nested item is not a closure
		1
	}
	assert_eq!(closure() + nested(), 5);
}

The boundary rule is as important as the map contents. Blindly searching every outer dictionary would incorrectly resolve local inside nested. A resolver must ask both “does this rib bind the name?” and “may lookup cross this rib?”

Duplicate declarations are context dependent. Two same-named functions in one value namespace and module are an error. Two sequential let bindings are intentional shadowing. The resolver therefore cannot implement duplicates as a universal hash-map collision rule.

Useful invariants are that stack pushes and pops are balanced, the innermost legal declaration wins, and crossing a boundary applies the namespace-specific policy. A leaked rib can make names from one function visible in the next. An early pop creates spurious unresolved-name errors. A wrong boundary kind often appears as an illegal capture being accepted or a legal closure capture rejected.

3. Rust Has Several Namespaces#

A namespace is a category of names looked up independently. It is not a module or filesystem directory. Rust chiefly distinguishes type, value, macro, and lifetime namespaces, with labels handled by a related dedicated mechanism.

The type namespace includes structs, enums, unions, traits, type aliases, modules, and type parameters in contexts where those entities can appear. The value namespace includes local variables, constants, statics, functions, and constructors that can be used as values. The macro namespace contains macro names. The lifetime namespace contains named lifetimes and lifetime parameters.

type meter = u32;

fn demo() {
	let meter: meter = 4;
	let doubled: meter = meter * 2;
	assert_eq!(doubled, 8);
}

Here meter legally names a type and a local value simultaneously. Syntactic position tells resolution which namespace to query. In let meter: meter, the pattern introduces a value while the annotation uses a type.

A unit-like struct demonstrates that one declaration may create bindings in multiple namespaces.

struct Marker;

fn takes_type(_: Marker) {}

fn main() {
	let value = Marker;
	takes_type(value);
}

The spelling Marker in the parameter type uses the type namespace. The expression Marker uses the value constructor binding. This is not one namespace magically changing meaning; collection records appropriate bindings.

Macros can coexist with ordinary items.

macro_rules! report {
	() => { "macro" };
}

fn report() -> &'static str { "function" }

fn main() {
	assert_eq!(report!(), "macro");
	assert_eq!(report(), "function");
}

The exclamation mark makes the macro context clear. Not every path position is so decisive. An import may potentially import bindings from more than one namespace, and ambiguity can remain until rustc has enough context or must report it. Glob imports are especially capable of introducing competing candidates.

mod a { pub struct Thing; }
mod b { pub struct Thing; }

use a::*;
use b::*;

// fn choose(_: Thing) {} // ambiguous: both globs supply a type candidate

Separating namespaces permits expressive reuse but costs implementation complexity. A single namespace would simplify lookup but reject familiar code such as constructors and same-named locals. Too many namespaces would surprise users and make imports difficult to explain. The invariant is that a use queries the language-defined namespace set for its syntactic role. Querying values for a type path causes false unresolved errors; merging macro and value candidates causes false duplicate or ambiguity errors.

4. Modules Form the Crate's Global Skeleton#

A crate has a root module, and nested mod items form a module tree. This is a semantic tree, although source may be split among several files. File layout helps the module loader find text; it is not itself name resolution.

crate root
├── api
│   ├── request
│   └── response
└── storage
    └── disk

An item path identifies movement through this tree and then a binding in a module. Modules provide long-lived item scopes, visibility boundaries, and import destinations. Blocks provide shorter lexical scopes and may themselves contain items and imports.

mod outer {
	pub mod inner {
		pub fn run() {}
	}

	pub fn call() {
		inner::run();
	}
}

fn main() {
	outer::call();
}

Within a path, self denotes the current module, super denotes its parent, and crate denotes the current crate root. These are semantic path roots, not ordinary user bindings.

mod network {
	pub mod client {
		pub fn connect() {
			super::log();
			crate::audit();
		}
	}

	fn log() {}
}

fn audit() {}

At a module root, self::client is explicit about starting locally. Repeated super::super climbs ancestors and fails if it climbs beyond the crate root. Absolute paths and edition rules must be interpreted deliberately rather than as string prefixes.

A block scope complicates the tree model because lexical scopes nest inside module scopes. An item declared in a block is not a child users can generally address through a public module path, yet it has a definition and is visible according to block item rules. The resolver therefore combines a global module graph with lexical rib stacks.

The module tree invariant is that every module has a coherent parent except the crate root. Incorrect parent links manifest as super reaching the wrong module, privacy checks using the wrong ancestry, or stable definition paths changing unexpectedly. Conflating files with modules manifests when inline and out-of-line modules behave differently.

5. Imports, Aliases, and Globs#

An import makes an existing binding available under a name in another scope. It does not copy the item and does not create a second function body. An alias changes the local spelling used to reach the same target.

mod geometry {
	pub struct Rectangle;
	pub fn area() -> u32 { 12 }
}

use geometry::Rectangle as Rect;
use geometry::area;

fn main() {
	let _shape = Rect;
	assert_eq!(area(), 12);
}

Explicit imports name a path and usually provide strong evidence for diagnostics. A glob import, written with *, requests all eligible names from a module. Globs reduce boilerplate but defer knowing the imported set and can become ambiguous after upstream changes.

mod warm { pub const COLOR: &str = "red"; }
mod cool { pub const COLOR: &str = "blue"; }

use warm::*;
use cool::*;

// const PICK: &str = COLOR; // ambiguous value imported by two globs

Import resolution can require a fixed point. Resolving one import can reveal a module or reexport needed by another. Globs may receive additional names as their source module's imports settle. The resolver repeats productive work until no relevant bindings change, rather than assuming source order is dependency order.

Explicit aliases are also conflict-management tools.

use std::fmt::Result as FmtResult;
use std::io::Result as IoResult;

fn format_it() -> FmtResult { Ok(()) }
fn read_it() -> IoResult<()> { Ok(()) }

Imports can participate in multiple namespaces according to what their target provides. An import's success means that its path found candidate definitions. It does not alone prove the imported definition is accessible at every eventual use. Privacy and effective visibility add separate questions later in this part.

Good duplicate diagnostics retain both the new import span and the earlier binding span. Good ambiguity diagnostics name competing sources, especially competing globs. A resolver that chooses “first hash-map entry” is nondeterministic and unsound as a language implementation. The invariant is that ordering accidents never silently decide a semantically ambiguous program.

6. The Extern Prelude, Standard Prelude, and Editions#

Some names appear usable without an explicit local use. Two mechanisms commonly explain that experience: the extern prelude and the standard prelude. They are distinct and should not be modeled as magical global text insertion.

The extern prelude supplies crate names available for paths, based on compiler and Cargo inputs. In modern editions, a dependency such as serde can usually begin a path without extern crate serde;. The current crate can be named with crate, while self and super remain module-relative roots.

The standard prelude is a set of commonly used names automatically brought into ordinary modules. It includes familiar traits and types such as Option, Result, and Vec. The selected prelude is edition-aware because language evolution sometimes changes its contents. Attributes such as #![no_std] and #![no_implicit_prelude] alter defaults.

fn main() {
	let numbers: Vec<i32> = vec![1, 2, 3];
	let answer: Option<i32> = numbers.first().copied();
	assert_eq!(answer, Some(1));
}

Neither Vec nor Option is declared locally here. Resolution considers the applicable prelude after stronger lexical and explicit candidates according to Rust's rules. Prelude names are shadowable, which is essential for local control and occasionally confusing.

Editions are per-crate language configurations, not compiler release numbers. Two dependency crates compiled in different editions can interoperate. Edition affects path interpretation, prelude composition, keyword status, and macro-related behavior. It does not mean rustc maintains unrelated module systems.

When debugging a path, record the crate's edition and whether implicit preludes are disabled. A minimal example copied into a default new project can behave differently if the original crate is 2015 edition. Tests should set the intended edition explicitly rather than inherit a harness default.

An implementation alternative would eagerly synthesize ordinary use items for all preludes. That seems simple but loses origin information and makes precedence and diagnostics harder. Representing prelude candidates with their origin allows rustc to say what is implicit, apply edition rules, and avoid treating them exactly like user-written explicit imports.

Symptoms of errors here include a dependency crate unresolved only in one edition, an unexpected ambiguity after a prelude gains a trait, or a no_implicit_prelude test accidentally passing. The invariant is that implicit candidates are deterministic functions of crate configuration and language edition.

7. Why Macro Expansion and Resolution Interleave#

A macro invocation names a macro, so rustc must resolve that name before expanding it. The expansion may then produce modules, imports, macro definitions, and ordinary items. Those products can change how later invocations resolve. Therefore expansion is not a clean, independent phase completed before all resolution.

macro_rules! make_helper {
	() => {
		fn helper() -> u32 { 9 }
	};
}

make_helper!();

fn main() {
	assert_eq!(helper(), 9);
}

The parser records an invocation rather than the final helper function. Early resolution identifies make_helper in the macro namespace. Expansion creates helper. Collection must then make that generated declaration available to later full resolution.

Now imagine an expansion that defines another macro and invokes it later. A rigid “resolve every macro first” pass cannot see the generated definition. A rigid “expand everything first” pass cannot know which macro implementation to run. The circular dependency is broken by iteration: resolve what is currently resolvable, expand it, integrate output, and continue.

rustc documentation often describes two broad phases. Early resolution operates during expansion and handles imports and macro names needed to make progress. Late or full resolution traverses the expanded AST and resolves the remaining names once no new expansion can add declarations. This is a conceptual division, not permission to regard expansion as resolver-free.

Macro hygiene adds identity beyond spelling. Tokens carry syntax context describing expansion history and where a name should be looked up. A macro-introduced temporary should not accidentally capture a caller's same-spelled local. Conversely, metavariables substituted from the call site often retain call-site relationships.

The progress invariant for iterative expansion is that each successful step consumes an invocation or adds useful information. Failure to enforce progress can loop on unresolved imports or macros. Resolving too early can bind a macro to the wrong scope; resolving too late can leave the expander unable to proceed. Hygiene bugs characteristically change behavior when an unrelated local is renamed.

8. Early Resolution and the Reduced Graph#

Before resolving every expression, rustc needs a coarse view of globally relevant declarations. Compiler discussions call this a reduced graph: a module-centered representation containing enough bindings to resolve imports and support expansion without modeling every expression and statement.

“Reduced” means irrelevant detail is omitted. A function body may contain thousands of arithmetic expressions, but import resolution primarily needs modules, items, imports, and macro-related declarations. Skipping expression internals makes early work cheaper and avoids pretending all names can already be finalized.

Conceptually, collection performs steps like these.

visit crate root
  create module node for `api`
  record item binding `start` in value namespace
  record import directive `use api::Config`
visit module `api`
  record `Config` in type and constructor/value roles
  record child module `detail`

Definition collection assigns compiler identities to entities that count as definitions. Name binding and identity allocation are related but not identical. One definition may introduce bindings in multiple namespaces, and one alias binding may point to an already existing definition.

Items are collected before ordinary body lookup because item declarations are not sequential like locals. Within a block, rustc can pre-scan item declarations so an earlier expression may call a later local item. Imports then resolve against module bindings, potentially iterating because aliases and globs depend on each other.

The reduced graph is not the final HIR and should not accumulate type-checker decisions. It answers questions such as “which module binding could this path segment denote?” It should not answer “which trait implementation makes this method call legal?” Maintaining that boundary prevents circular dependencies with type inference.

Alternative designs could repeatedly scan the entire expanded AST for each import. That would be simpler in a toy compiler but scales poorly and makes ambiguity state difficult to maintain. An explicit graph centralizes module candidates, import provenance, and fixed-point status.

Useful invariants are that collected definitions are not accidentally allocated twice, every import directive retains its source span, and graph updates invalidate dependent unresolved work. A duplicate allocation produces inconsistent IDs for one item. A missing update produces an unresolved import that vanishes when source order changes.

9. Late Resolution and Ribs#

After expansion has produced the crate's complete AST, late resolution walks it and resolves ordinary paths, locals, labels, lifetimes, and other names that did not need final answers during expansion. No future macro expansion should add a declaration at this point. Thus an unresolved required name can become a firm diagnostic rather than a temporary state.

Late resolution uses namespace-specific rib stacks. Entering a function establishes ribs for parameters, generic parameters, labels, and relevant item scopes. Entering blocks and patterns changes those stacks according to exact language rules. Lookup walks outward while respecting boundary kinds.

fn convert<T: Default>(input: T) -> T {
	let saved = input;
	{
		let saved = T::default();
		saved
	}
}

The T uses search the type namespace and find a generic parameter. The first input use searches values and finds the parameter. The returned saved is the outer binding because the inner rib has been popped.

Patterns require care because one pattern may introduce several bindings, and alternatives must bind compatible names. Initializers generally do not see the binding being introduced by their own let.

fn f() {
	let x = 1;
	let x = x + 1; // right-hand `x` is the old binding
	assert_eq!(x, 2);
}

Resolving each identifier once in this phase is an important simplification. Results can be stored for lowering and diagnostics. Still, recovery may install error resolutions so later phases can continue without cascades. An error marker is not a valid declaration; it is controlled damage containment.

Late resolution may also record partial information used by later analyses. It should preserve ambiguity when the language requires type information to choose, rather than guessing based on lexical order.

Common bugs include forgetting to prepopulate block items, introducing a pattern binding before its initializer, or searching the wrong namespace rib stack. Small tests should vary declaration order and add same-spelled bindings in other namespaces.

10. Paths Are Structured Questions#

A path such as crate::service::Config is a sequence of segments with a starting rule. It is not best understood as one dotted string in a global table. Resolution finds a root, resolves intermediate module-like segments, and determines how far lexical resolution can proceed.

mod service {
	pub struct Config;
}

fn make() -> crate::service::Config {
	crate::service::Config
}

The first segment crate fixes the current crate root. service denotes a child module. Config denotes a type in the return position and a constructor value in the expression position. The shared spelling does not erase namespace context.

Relative single-segment paths search applicable lexical scopes. Multi-segment paths combine lexical lookup for their start with member lookup for subsequent segments. Edition rules affect what an unqualified leading segment may mean, especially around external crate names and crate-root interpretation.

Imports ask path questions too, but import paths have specialized rules and ambiguity handling. Macros and attributes may carry paths with their own resolution timing. Using one generic “split and lookup” routine without context tends to mishandle these distinctions.

Qualified paths can explicitly provide a type and trait context.

trait Render {
	fn render() -> &'static str;
}

struct Page;

impl Render for Page {
	fn render() -> &'static str { "page" }
}

fn main() {
	let text = <Page as Render>::render();
	assert_eq!(text, "page");
}

Lexical resolution can identify Page and Render. The associated render selection involves trait and type semantics. That separation becomes the subject of the next section.

Path diagnostics should report the segment at which progress failed. Pointing only at the entire path hides whether the root was absent or a child was inaccessible. The invariant is that each resolved prefix has an explicit semantic target and provenance.

11. The Associated-Item Boundary#

Ordinary lexical resolution cannot fully decide every associated item. In receiver.draw(), the method may be inherent, supplied by an in-scope trait, or selected after autoderef and autoref adjustments. Those choices depend on the receiver's inferred type and trait solving.

trait Draw {
	fn draw(&self) -> &'static str;
}

struct Icon;

impl Draw for Icon {
	fn draw(&self) -> &'static str { "icon" }
}

fn paint<T: Draw>(value: T) -> &'static str {
	value.draw()
}

Name resolution identifies the local value, the trait Draw, and declarations in paths. Type inference determines that value has generic type T constrained by Draw. Method lookup and trait selection determine which draw is applicable. Calling all of this “name resolution” obscures vital phase boundaries.

Similarly, Type::item can be ambiguous between associated constants, functions, or trait-provided items until type and trait context is available. The resolver may resolve a path prefix and leave an associated segment for type checking. It must not choose a same-spelled lexical free function merely because that is easy.

struct Counter;

impl Counter {
	fn new() -> Self { Counter }
}

fn new() -> u32 { 0 }

fn main() {
	let _counter = Counter::new();
	let _number = new();
}

The two calls use distinct mechanisms after their initial names are understood. new() finds a free value binding lexically. Counter::new() starts from the resolved type and performs associated-item lookup.

Why maintain this boundary? Type inference is constraint-based and may learn types from later expressions. Forcing late type facts into lexical resolution creates cycles and order dependence. Conversely, making type checking rediscover all lexical scopes duplicates policy and degrades diagnostics.

The invariant is that resolver output contains enough partial resolution for type checking, without claiming a final associated item where type-dependent lookup is required. Bug symptoms include method calls changing target when an unrelated free function is imported, trait methods reported missing despite a valid bound, or lexical ambiguity emitted where type information should disambiguate.

12. Lifetimes, Labels, and Generic Parameters#

Lifetimes name relationships between borrows; labels name control-flow destinations. Both use apostrophe syntax, but context distinguishes them and their scopes differ. They should not be thrown into the ordinary value namespace.

fn first<'a>(left: &'a str, _right: &str) -> &'a str {
	left
}

fn search() {
	'outer: loop {
		loop {
			break 'outer;
		}
	}
}

'a is a declared lifetime parameter used in reference types and return type. 'outer is a loop label used by break. Equal textual apostrophe forms do not imply one namespace or one entity kind.

Generic type, const, and lifetime parameters have item-specific scope rules.

fn repeat<'a, T, const N: usize>(value: &'a T) -> [&'a T; N] {
	[value; N]
}

T resolves in the type namespace, N in value-like const contexts, and 'a in the lifetime namespace. Their bounds and defaults have ordering and visibility rules that rustc must encode accurately.

Labels are searched through label ribs. A break 'name must reach an eligible labeled loop or block. Crossing a function, closure, or async boundary may be forbidden even if spelling is visible textually. Diagnostics should distinguish an undeclared label from a label attached to an invalid construct.

Lifetime resolution also handles special forms such as inferred lifetimes and 'static. Not every omitted lifetime corresponds to a user declaration. Elision rules synthesize relationships based on language-defined contexts, and later type checking reasons about those relationships.

An implementation might unify all apostrophe names into one map for convenience. That risks allowing break 'a to target a lifetime parameter or a reference to use a loop label. Separate semantic contexts make impossible states harder to represent.

Boundary mistakes produce striking symptoms: labels escaping closures, generic parameters captured by nested items that should not see them, or a lifetime diagnostic naming an unrelated loop. Tests should use identical spellings for a lifetime and label to prove contextual separation.

13. Closures, Captures, and Upvars#

A closure is an executable expression that may use locals from an enclosing function or closure. Such a referenced outer local is often called an upvar, short for “upward variable.” Resolution first identifies the outer binding; later analysis decides how the closure captures it.

fn main() {
	let greeting = String::from("hello");
	let length = || greeting.len();
	assert_eq!(length(), 5);
}

The use of greeting searches outward across a closure boundary and resolves to the local. That crossing is recorded as relevant to capture analysis. Type checking and borrow analysis determine whether capture is by shared borrow, mutable borrow, or value, potentially at a field-level precision.

The resolver must distinguish a closure boundary from a nested function-item boundary.

fn outer() {
	let x = 10;
	let allowed = || x;

	fn separate() -> i32 {
		// `x` is not available here.
		10
	}

	assert_eq!(allowed(), separate());
}

A move closure changes capture mode expectations but not which lexical declaration x names. Resolution should not decide ownership transfer simply from the keyword. It supplies identity; later semantic analysis supplies capture kind.

Nested closures can capture through intermediate closure contexts. The compiler must attribute uses and ownership to the correct closure owners. Otherwise diagnostics may blame an outer closure for a borrow created only by an inner closure.

fn nested() {
	let text = String::from("abc");
	let outer = || {
		let inner = || text.len();
		inner()
	};
	assert_eq!(outer(), 3);
}

The central invariant is that every captured use still resolves to the original local definition, while each closure boundary crossed is represented consistently for downstream capture computation. A resolver bug may report x unresolved only inside closures. A capture-analysis bug more often reports an incorrect move or borrow after resolution succeeded. That distinction narrows triage quickly.

14. Hygiene: Same Text, Different Origin#

Macro hygiene prevents generated names from accidentally colliding with names at the invocation site. Text alone is insufficient: rustc identifiers conceptually combine a symbol with syntax context. The context records expansion history and controls where lookup occurs.

macro_rules! with_temp {
	($body:expr) => {{
		let temp = 1;
		($body, temp)
	}};
}

fn main() {
	let temp = 99;
	let pair = with_temp!(temp);
	assert_eq!(pair, (99, 1));
}

The temp supplied as $body originates at the call site and denotes the caller's binding. The temp written in the macro definition belongs to the macro's generated context. Treating both as plain strings could produce (1, 1), an accidental capture.

Declarative macros, procedural macros, built-in macros, and edition transitions have nuanced hygiene behavior. Exact internal context adjustment APIs are version-sensitive. The durable concept is that lookup receives both spelling and provenance, and expansion boundaries can alter which scopes are considered.

Hygiene also affects $crate, which lets an exported declarative macro refer reliably to its defining crate. Using ordinary crate at a call site could instead point at the caller's crate. This distinction supports macros that work when renamed dependencies or downstream modules invoke them.

#[macro_export]
macro_rules! make_default_vec {
	() => {
		$crate::support::new_vec()
	};
}

pub mod support {
	pub fn new_vec() -> Vec<u8> { Vec::new() }
}

Diagnostics complicate hygiene because users need source-facing locations. An error in generated code may have a definition-site span, call-site span, and a chain of expansion backtraces. Choosing the wrong one yields a technically correct but unusable message.

Hygiene bugs often disappear when a local is renamed, appear only when a macro is exported, or resolve to a private helper in the caller rather than the defining crate. Always minimize with both macro definition and invocation retained. The invariant is capture behavior prescribed by token origin, not accidental textual coincidence.

15. Resolution Is Not Accessibility#

Finding a definition and being allowed to use it are separate propositions. Resolution answers “which entity does this path denote?” Privacy answers “may code from this location access that entity?” Conflating them loses precise errors and makes reexports difficult to reason about.

mod vault {
	fn secret() {}

	pub fn open() {
		secret();
	}
}

fn main() {
	// vault::secret(); // resolves conceptually, but is private here
	vault::open();
}

The path can identify vault::secret even though access is rejected. A good diagnostic says the function is private, not that no function exists. That requires preserving resolution information through privacy checking.

Rust visibility includes private defaults, pub, and restricted forms such as pub(crate) or pub(in crate::some_module) where permitted by ancestry rules. Visibility is interpreted relative to the module tree. A child module can access certain ancestor-private items because privacy boundaries are module based.

Reexports create public routes to existing definitions.

mod implementation {
	pub struct Token;
}

pub use implementation::Token;

fn main() {
	let _ = Token;
}

The reexport does not allocate a new Token type. It exposes the same definition through another binding and path. The source module's own visibility and the reexport's visibility must combine correctly.

Effective visibility describes how exposed an item actually is after accounting for containing modules, restricted visibilities, and reexport chains. Writing pub inside a private module does not necessarily make an item reachable from another crate. Effective visibility supports reachability decisions, lints, metadata, and diagnostics.

mod hidden {
	pub struct PubliclySpelled;
}

// Another crate cannot name `hidden::PubliclySpelled` because `hidden` is private.

The invariant is that every access check has both a resolved target and an access origin. Privacy bugs show valid target suggestions followed by incorrect “private” errors, or allow external code through a path whose ancestor is hidden. Resolution bugs instead fail to identify the target at all.

16. DefId and Its Compact Components#

rustc needs a cheap handle for definitions used throughout one compilation session. DefId serves that role for local and external definitions. Conceptually it combines a CrateNum and a DefIndex.

DefId {
  crate: CrateNum,
  index: DefIndex,
}

CrateNum identifies a crate within the current compilation session's crate graph. It is not a permanent package registry number. DefIndex identifies a definition within that crate's loaded definition table. Together they make compact indexing and comparison efficient.

A LocalDefId is a type-safe handle known to belong to the current crate. Code that needs local HIR should request LocalDefId, because dependencies do not have local HIR bodies. An external DefId may have metadata describing signatures and attributes, but its source HIR is not simply part of the current crate's HIR map.

Definitions include familiar items such as functions, modules, structs, and traits, plus compiler-relevant entities that beginners may not think of as standalone declarations. The precise set and allocation details are version-sensitive. Never infer definition kind solely from an integer index.

Crucially, a DefId is not stable across arbitrary builds. Crate numbering can change when dependency order changes, and definition indexing is an in-session compact representation. Persisting raw DefId numbers in a long-lived cache would confuse unrelated entities.

// Conceptual only: rustc-private types are not stable Rust APIs.
// let id = DefId { krate: CrateNum::from_u32(2), index: DefIndex::from_u32(7) };

The useful invariant is uniqueness within the active compiler context, not historical permanence. APIs should distinguish local from potentially external IDs in their types. Bug symptoms include trying to fetch HIR for a dependency definition, mixing IDs from different compiler contexts, or cache hits that point at the wrong item.

17. Definition Paths and Stable Hash Concepts#

Incremental compilation and cross-crate metadata need identity more durable than compact session numbering. rustc describes a definition through a DefPath: a parent-linked semantic path whose components represent definition kinds, names when available, and disambiguators for collisions.

Not every definition has a pleasant source path. Closures, anonymous constants, and generated entities may need synthetic path components. Two same-named items in distinguishable contexts require disambiguation. A plain string such as crate::module::name is therefore insufficient.

A DefPathHash is a stable-hash concept derived from definition-path information and crate identity. It is designed for stable identification needs such as metadata and incremental systems. “Stable” here means stable under the compiler's specified hashing scheme and suitable changes, not immortal across arbitrary source edits, compiler changes, crate identity changes, or all builds.

source definition
      |
      v
parent DefPath + component + disambiguator
      |
      v
DefPathHash for stable-oriented lookup
      |
      +---- map to current session's compact DefId

Cross-crate metadata serializes semantic information using identities that can be translated when a dependency is loaded into a new session with new CrateNum values. Consumers then operate with current-session DefIds. The translation boundary prevents raw local indices from pretending to be global addresses.

Path construction must be deterministic with respect to semantically relevant ordering. Changing unrelated code should avoid renumbering all downstream definitions where the design permits. At the same time, identity must change when a definition is genuinely replaced or moved in a meaningful way. This is a tradeoff between reuse and correctness; false cache reuse is worse than recomputation.

Incremental identity bugs often appear only after a second build. A clean build succeeds, while an edited incremental build crashes, misdiagnoses, or reuses stale results. Tests should compare clean and incremental outcomes after inserting same-named items nearby.

The invariant is explicit conversion between durable identity concepts and session-local handles. Do not log only numeric DefIds when investigating persistence; include definition paths or source-facing descriptions so two sessions can be compared meaningfully.

18. Why NodeId Is Not Enough#

The expanded AST uses node identifiers commonly called NodeIds to distinguish syntax nodes during compilation. They are convenient while operating on that AST, but their allocation is tied to syntax traversal and expansion activity. Inserting syntax can renumber many later nodes.

A NodeId also does not by itself encode crate ownership, stable definition ancestry, or HIR owner-local structure. It can identify nodes that are not definitions and therefore cannot replace DefId. It belongs to a particular compiler run and representation stage.

AST NodeId: temporary identity for expanded syntax work
DefId: compact semantic definition identity in a session
DefPathHash: stable-oriented definition identity concept
HirId: owner plus item-local identity for local HIR nodes

Lowering needs mappings from relevant AST identities and resolver results into HIR identities. Once lowering is complete, later analyses should use HIR and definition identifiers appropriate to their query. Holding a NodeId and assuming it indexes HIR is a stage error.

Consider adding a macro invocation near the top of a crate. Expansion can allocate many new AST nodes and shift later NodeIds. If incremental caches were keyed persistently by those numbers, unchanged later functions could be mistaken for generated nodes from the new expansion.

The alternative is not one universal perfect ID. Different stages need different granularity and performance. AST processing benefits from cheap temporary IDs; definition queries benefit from DefId; fine-grained HIR traversal benefits from HirId. Typed wrappers prevent accidental interchange.

Bug symptoms include map misses after lowering, diagnostics pointing at a different AST node after macro expansion, or code converting integers between ID types to “make it compile.” The invariant is that every ID is interpreted only in its declared domain and compiler context.

19. Lowering AST to HIR#

Lowering transforms the expanded, resolved AST into HIR, the High-level Intermediate Representation consumed by many later rustc analyses. HIR resembles Rust source but is more regular and omits or desugars selected surface constructs.

HIR is not simply a typed AST. It is largely untyped: type checking consumes HIR and computes type-dependent results. Some paths already carry resolution information, but expression types and final method selection are generally later concerns.

Why introduce another representation? The AST must faithfully support parsing, macro expansion, token-oriented diagnostics, and language surface variety. Every later pass would be burdened if it independently normalized for, ?, and async syntax. Lowering centralizes that normalization while preserving source provenance.

tokens
  -> parsed AST
  -> expansion interleaved with early resolution
  -> expanded AST plus full resolution results
  -> AST lowering
  -> HIR
  -> type checking and later IR construction

Lowering is not arbitrary optimization. It must preserve program meaning, maintain mappings needed by queries and diagnostics, and construct valid HIR ownership. Generated HIR nodes often carry spans associated with source syntax or marked desugaring origins.

Some syntax disappears because it is irrelevant to semantic analysis. Parentheses used only for grouping need not remain as dedicated semantic nodes. Other syntax expands into explicit control flow so later passes handle fewer cases. Exact HIR enum variants and lowering helper names are version-sensitive in rustc 1.97.1 internals.

An alternative would keep every surface form until MIR construction. That preserves syntax but forces type checking, linting, and other HIR consumers to understand every sugar. Another alternative would desugar in the parser, but that loses clean source structure too early and complicates macro input and diagnostics.

The lowering invariant is semantic equivalence plus valid provenance, not textual resemblance. Failures appear as compiler panics on valid source, wrong control-flow behavior after compilation, or diagnostics pointing into synthetic code users never wrote.

20. HIR Owners and HirId#

HIR is organized around owners, usually item-like definitions such as functions, trait items, and impl items. An OwnerId identifies an owner through its local definition identity. Within an owner, HIR nodes receive dense ItemLocalIds. A HirId combines owner and local ID.

HirId
├── owner: OwnerId / owning LocalDefId
└── local_id: ItemLocalId

The pair means local expression numbering in one function does not collide with numbering in another. Dense owner-local IDs support vector-backed maps and compact per-owner data. Owner boundaries also align with incremental query dependencies: reading one owner need not imply reading every nested owner's body.

fn outer() {
	let closure = |x: i32| x + 1;
	assert_eq!(closure(2), 3);
}

The function is an owner, and closure-like bodies can have their own ownership relationships. Exact owner categories and representation details are version-sensitive, so inspect the 1.97.1 API rather than memorizing an older diagram.

Lowering a node must allocate its HirId under the correct current owner. When lowering creates parts belonging to another owner, rustc uses owner-switching machinery rather than continuing the old local counter. Otherwise two owners' maps and parent relationships become corrupt.

Not every HirId has a separate DefId. Fine-grained expressions, patterns, and statements need HIR identity but are not all standalone definitions. Conversely, DefId can refer to external definitions that have no local HirId.

The HIR map provides source-like traversal and lookup by these IDs, usually accessed through compiler query context APIs rather than global mutable tables. Query-mediated access records dependencies important to incremental compilation.

Invariants include uniqueness within an owner, dense local-ID allocation where promised, correct parent links, and owner-local lookup isolation. Wrong-owner bugs often panic while building the HIR map or produce an ICE saying a HirId was not found.

21. Desugaring a for Loop#

A source for loop expresses iteration without exposing iterator protocol plumbing. HIR lowering replaces it with more primitive control flow and calls connected to language items. The following trace is conceptual IR, not exact rustc 1.97.1 pretty output.

for value in values {
	consume(value);
}

Conceptually, evaluation proceeds like this.

CONCEPTUAL IR
match IntoIterator::into_iter(values) {
    mut iterator => loop {
        match Iterator::next(&mut iterator) {
            Some(value) => { consume(value); }
            None => break,
        }
    }
}

The outer match-like structure controls temporary lifetime and evaluates values once. That “once” property is an invariant: duplicating the iterable expression could duplicate side effects. The iterator remains mutable because next advances it. Pattern matching extracts each yielded item and terminates on None.

Real lowering uses compiler-known diagnostic and language-item machinery, not necessarily paths resolved as though the user typed every name above. This prevents a local type named Iterator from changing for semantics. It also lets diagnostics point at the for source rather than synthetic method tokens.

Labels and break behavior must be preserved.

'scan: for row in rows {
	for cell in row {
		if cell == target {
			break 'scan;
		}
	}
}

Lowering must attach 'scan to the semantically corresponding loop destination. A mistaken synthetic wrapper can cause break to exit the wrong construct.

Type checking later verifies the iterable and item types. Lowering does not decide a concrete iterator implementation by lexical name lookup alone. It creates the normalized form and records relevant semantic hooks.

Bug symptoms include the iterable evaluated twice, temporary values dropped too early, incorrect break targets, or errors referring to invisible next code. Always compare runtime behavior and diagnostic spans when changing this lowering.

22. Desugaring ? and Residual Flow#

The ? operator means “continue with the success value, or return compatible residual control flow.” It is broader than a textual early return of Err, because Rust's Try and residual machinery supports applicable types and conversions. The exact internal form is version-sensitive.

fn load() -> Result<u32, std::io::Error> {
	let text = std::fs::read_to_string("number.txt")?;
	Ok(text.trim().parse().unwrap_or(0))
}

A useful conceptual trace is:

CONCEPTUAL IR
match Try::branch(read_to_string("number.txt")) {
    Continue(text) => text,
    Break(residual) => return FromResidual::from_residual(residual),
}

This trace explains several invariants. The operand is evaluated exactly once. The continuing branch yields the expression's value. The breaking branch converts a residual to the enclosing return context. The enclosing function, closure, or try-like context determines where propagation goes.

Older explanations often translate Result<T, E>? directly to matching Ok and Err. That is useful intuition but too narrow as an internal contract. It hides residual conversion and can mislead contributors working on generalized try semantics.

The generated match needs a span tied to the question mark and operand. If conversion fails, the diagnostic should explain incompatibility at the user's ?, not claim an internal FromResidual call was handwritten. Desugaring-kind metadata helps diagnostics recognize synthetic structure.

fn wrong() -> u32 {
	// std::fs::read_to_string("x")?;
	0
}

The desired error explains that ? cannot be used in this return context. It should not become an unresolved name for branch or an opaque type mismatch in generated code.

Lowering leaves trait selection and type compatibility to later analysis. It must preserve control-flow destination and source provenance. Failures include propagation to the wrong owner, duplicate operand evaluation, and suggestions inserting edits at synthetic spans.

23. Desugaring async fn and .await#

An async fn presents a function whose call returns a future. Its body executes through generated state-machine behavior when polled, not simply as an ordinary function body completed at call time. HIR lowering represents the async nature and prepares normalized constructs for later transformations.

async fn fetch(id: u32) -> Result<String, Error> {
	let record = database_lookup(id).await?;
	Ok(record.name)
}

At the language level, the signature is roughly understood as returning an opaque future whose output is Result<String, Error>. The body is enclosed in async-generated structure. This is conceptual IR and omits captures, lifetimes, and internal nodes.

CONCEPTUAL IR
fn fetch(id: u32) -> impl Future<Output = Result<String, Error>> {
    async move-like body {
        let record = await database_lookup(id);
        propagate `?` if needed;
        Ok(record.name)
    }
}

The “move-like” phrase is intentionally not an exact source rewrite. Parameter capture, edition behavior, and generated owner details require current implementation inspection. Exact rustc 1.97.1 HIR variants and desugaring order are version-sensitive.

An .await is not ordinary lexical method lookup for a user method named await. It suspends the future when pending and resumes later, with semantic machinery ultimately related to polling and context. Conceptually it repeatedly polls a pinned future and yields on Pending.

CONCEPTUAL CONTROL FLOW
loop {
    match poll(pinned_future, current_context) {
        Ready(value) => break value,
        Pending => suspend this async state and later resume,
    }
}

Do not treat that as exact HIR or safe source code. Pinning, context access, suspension points, and generator/coroutine representation are compiler-sensitive. The trace explains why locals live across await points and why borrow checking needs suspension information.

Owner boundaries are crucial because generated async bodies and closures participate in query ownership. Spans must connect generated state behavior to async, .await, and captured source. Bugs appear as wrong capture diagnostics, ICEs on owner lookup, or errors mentioning synthetic generators instead of the async function.

24. if let, while let, and Let Chains#

Pattern condition syntax combines matching with control flow. An if let tests a pattern and makes successful bindings available only in the then branch. A useful conceptual lowering is a match.

if let Some(value) = maybe {
	consume(value);
} else {
	recover();
}

CONCEPTUAL IR
match maybe {
    Some(value) => consume(value),
    _ => recover(),
}

The scrutinee is evaluated once. value exists only in the successful arm. The wildcard arm preserves the else behavior. Real HIR may preserve or encode details differently in 1.97.1, so this is a semantic trace, not an asserted enum dump.

A while let repeats matching and must reevaluate its expression each iteration.

while let Some(job) = queue.pop() {
	run(job);
}

Conceptually it is a loop containing a match, where a successful arm runs the body and a failed arm breaks. Unlike the for iterable, queue.pop() intentionally runs on every iteration. Getting that evaluation frequency wrong changes observable behavior.

Let chains combine boolean conditions and pattern bindings with left-to-right short-circuiting.

if let Some(user) = lookup()
	&& user.enabled
	&& let Some(email) = user.email()
{
	send(email);
}

The binding user is available to conditions to its right and the body, but not when its pattern fails or outside the whole conditional. email begins only after its own successful match. Lowering must preserve short-circuit order, drops, and scope boundaries.

An eager translation that evaluates all conditions first is wrong because later conditions may have side effects or rely on earlier bindings. Diagnostics need original condition spans rather than synthetic nested-match arms. Symptoms include bindings leaking into else, operands evaluated despite short circuit, or temporary lifetime differences between chained and nested source.

25. Diagnostic Provenance During Lowering#

Desugaring creates nodes with no literal token in user source. Nevertheless, every error and lint must communicate through source locations users recognize. A span records source range and expansion context; generated nodes also carry desugaring provenance.

Suppose a for loop's iterator conversion fails. The type checker may be examining a generated call, but the diagnostic should underline the iterable expression and explain that it is not an iterator. Showing “cannot call hidden lang item” would leak implementation rather than teach the language rule.

fn example() {
	for item in 12 {
		println!("{item}");
	}
}

There are competing goals. A broad span over the whole sugar gives context but imprecise underlining. A tiny generated span may point nowhere useful. rustc often preserves both a source-relevant span and metadata identifying the desugaring, allowing diagnostics to select an explanation.

Macro expansion adds another provenance dimension. A desugared construct may itself come from a macro, so its span has both expansion history and surface-sugar meaning. Diagnostic code may walk toward a call site or show an expansion note. Exact APIs for span adjustment are version-sensitive.

Suggestions impose a stronger requirement than labels. An automatic edit must target editable user source and remain syntactically valid. A suggestion anchored inside generated HIR can corrupt code or be impossible to apply. When no reliable source range exists, a note is safer than a machine-applicable suggestion.

Provenance invariants are that generated nodes retain a meaningful origin, desugaring kind is not lost during cloning or owner transfer, and emitted edits never target synthetic text. Wrong-span bugs are often regressions even when the primary error code remains correct. Tests should assert labels and suggestions, not only that compilation fails.

26. HIR Validation and Lint Timing#

Validation checks representation and language constraints at stages where enough information exists. AST validation can reject structurally invalid combinations before lowering. HIR validation checks assumptions expected by HIR consumers after normalization. These checks are not substitutes for type checking.

An invariant checker should fail close to the producer of malformed HIR. Otherwise a bad owner or impossible expression variant may crash much later in borrow checking, making the true cause difficult to locate. Compiler errors in recovered user programs must still avoid uncontrolled cascades.

Lints are diagnostics for suspicious or discouraged code that is often otherwise valid. Early lints operate on AST-oriented information and can see surface syntax before it disappears. Late lints operate on HIR and can use resolved definitions and more normalized structure. Some later lint work can also consume type-checking results.

fn demo() {
	let unused_name = 3;
	if true { println!("always"); }
}

Where a lint runs depends on what it needs. A purely token or syntax-style concern belongs early enough to preserve spelling. An unused binding requires identity and use information. A lint about calling a particular API should compare resolved DefIds, not textual paths that aliases can change.

Lint levels attach to scopes through attributes such as allow, warn, deny, and forbid. HIR owners and nodes provide attachment points for scoped lint processing. Generated code and external macros may have special reporting policy, so source provenance remains relevant.

#[allow(unused_variables)]
fn quiet() {
	let intentionally_unused = 1;
}

Running a lint too early can confuse same-spelled definitions or miss desugared semantics. Running it too late can lose source syntax needed for a precise suggestion. Duplicating it in both stages risks duplicate diagnostics.

The invariant is one authoritative stage with all required facts and usable source mapping. Bug symptoms include a lint ignoring item-level attributes, firing inside external macro output unexpectedly, or changing solely because an import alias changed spelling.

27. Build a Small Resolver in Stable Rust#

We now implement a deliberately small lexical resolver. It supports nested scopes, declarations, shadowing, duplicate diagnostics, unresolved diagnostics, and stable-oriented definition identities. It is not a replacement for rustc's modules, namespaces, imports, or hygiene.

The symbol table stores a stack of maps from names to definition records. Definition identity hashes a crate identity, parent identity, semantic kind, name, and disambiguator. We use a deterministic FNV-1a-style hash rather than DefaultHasher, whose persistence contract is unsuitable.

use std::collections::HashMap;

#[derive(Clone, Copy, Debug, Eq, Hash, PartialEq)]
struct StableDefId(u64);

#[derive(Clone, Debug, Eq, PartialEq)]
struct Definition {
	name: String,
	kind: &'static str,
	id: StableDefId,
	line: usize,
}

#[derive(Clone, Debug, Eq, PartialEq)]
enum Diagnostic {
	Duplicate {
		name: String,
		first_line: usize,
		second_line: usize,
	},
	Unresolved { name: String, line: usize },
}

fn stable_hash(parts: &[&str]) -> StableDefId {
	let mut hash = 0xcbf2_9ce4_8422_2325_u64;
	for part in parts {
		for byte in part.as_bytes().iter().copied().chain([0xff]) {
			hash ^= u64::from(byte);
			hash = hash.wrapping_mul(0x0000_0100_0000_01b3);
		}
	}
	StableDefId(hash)
}

struct Resolver {
	crate_identity: String,
	scopes: Vec<HashMap<String, Definition>>,
	parents: Vec<StableDefId>,
	diagnostics: Vec<Diagnostic>,
}

impl Resolver {
	fn new(crate_identity: impl Into<String>) -> Self {
		let crate_identity = crate_identity.into();
		let root = stable_hash(&[&crate_identity, "crate"]);
		Self {
			crate_identity,
			scopes: vec![HashMap::new()],
			parents: vec![root],
			diagnostics: Vec::new(),
		}
	}

	fn enter_scope(&mut self, label: &str) {
		let parent = self.parents.last().expect("root scope exists");
		let id = stable_hash(&[
			&self.crate_identity,
			&format!("{:016x}", parent.0),
			"scope",
			label,
		]);
		self.scopes.push(HashMap::new());
		self.parents.push(id);
	}

	fn leave_scope(&mut self) {
		assert!(self.scopes.len() > 1, "cannot leave root scope");
		self.scopes.pop();
		self.parents.pop();
	}

	fn declare(
		&mut self,
		name: &str,
		kind: &'static str,
		disambiguator: u32,
		line: usize,
	) -> Option<StableDefId> {
		let scope = self.scopes.last_mut().expect("root scope exists");
		if let Some(first) = scope.get(name) {
			self.diagnostics.push(Diagnostic::Duplicate {
				name: name.to_owned(),
				first_line: first.line,
				second_line: line,
			});
			return None;
		}
		let parent = self.parents.last().expect("root scope exists");
		let id = stable_hash(&[
			&self.crate_identity,
			&format!("{:016x}", parent.0),
			kind,
			name,
			&disambiguator.to_string(),
		]);
		scope.insert(name.to_owned(), Definition {
			name: name.to_owned(), kind, id, line,
		});
		Some(id)
	}

	fn resolve(&mut self, name: &str, line: usize) -> Option<StableDefId> {
		for scope in self.scopes.iter().rev() {
			if let Some(definition) = scope.get(name) {
				return Some(definition.id);
			}
		}
		self.diagnostics.push(Diagnostic::Unresolved {
			name: name.to_owned(), line,
		});
		None
	}
}

The separator byte prevents ambiguous concatenations such as ab plus c from matching a plus bc. The parent identity makes equal names in different scopes distinct. The disambiguator handles equal semantic components when the language permits them. Our labels must themselves be deterministic; source byte offsets would make identities fragile under edits.

Tests exercise shadowing, restoration, duplicates, unresolved uses, and identity repeatability.

#[cfg(test)]
mod tests {
	use super::*;

	#[test]
	fn nearest_binding_wins_then_outer_returns() {
		let mut resolver = Resolver::new("demo@1");
		let outer = resolver.declare("x", "local", 0, 1).unwrap();
		resolver.enter_scope("main/block-0");
		let inner = resolver.declare("x", "local", 0, 3).unwrap();
		assert_ne!(outer, inner);
		assert_eq!(resolver.resolve("x", 4), Some(inner));
		resolver.leave_scope();
		assert_eq!(resolver.resolve("x", 6), Some(outer));
	}

	#[test]
	fn duplicate_in_one_scope_reports_both_lines() {
		let mut resolver = Resolver::new("demo@1");
		assert!(resolver.declare("run", "function", 0, 2).is_some());
		assert!(resolver.declare("run", "function", 0, 8).is_none());
		assert_eq!(resolver.diagnostics, vec![Diagnostic::Duplicate {
			name: "run".into(), first_line: 2, second_line: 8,
		}]);
	}

	#[test]
	fn unresolved_use_is_recorded() {
		let mut resolver = Resolver::new("demo@1");
		assert_eq!(resolver.resolve("missing", 11), None);
		assert_eq!(resolver.diagnostics, vec![Diagnostic::Unresolved {
			name: "missing".into(), line: 11,
		}]);
	}

	#[test]
	fn stable_identity_repeats_for_same_semantic_path() {
		fn build() -> StableDefId {
			let mut resolver = Resolver::new("demo@1");
			resolver.enter_scope("module-api");
			resolver.declare("serve", "function", 0, 20).unwrap()
		}
		assert_eq!(build(), build());
	}

	#[test]
	fn crate_identity_separates_equal_paths() {
		let mut left = Resolver::new("left@1");
		let mut right = Resolver::new("right@1");
		let a = left.declare("run", "function", 0, 1).unwrap();
		let b = right.declare("run", "function", 0, 1).unwrap();
		assert_ne!(a, b);
	}
}

This identity is educational, not collision-proof compiler metadata. A production design stores structured paths, handles hash collisions, defines crate identity rigorously, and separates namespaces. It does demonstrate the crucial distinction between a stable-oriented key and a compact table index.

28. Trace a Multi-Module Crate End to End#

Consider this crate, shown inline so file loading does not distract from semantic structure.

mod model {
	pub struct User {
		pub name: String,
	}

	impl User {
		pub fn label(&self) -> &str { &self.name }
	}
}

mod service {
	use super::model::User as Account;

	pub fn greet(account: &Account) -> String {
		let prefix = "hello";
		format!("{prefix} {}", account.label())
	}
}

pub use model::User;
use service::greet;

fn main() {
	let user = User { name: "Ada".into() };
	println!("{}", greet(&user));
}

Parsing creates an AST with modules, items, imports, paths, expressions, and macro invocations. It knows that account.label() is a method-call expression, but it does not yet choose the User::label implementation. It records format! and println! as macro invocations awaiting expansion.

Early collection creates module nodes for the root, model, and service. It records User in relevant type and constructor roles, the inherent impl and method definitions, greet, main, and import directives. Definition collection assigns local definition identities with semantic parent paths.

Import resolution interprets super in service as the crate root, finds root child model, then its public User, and binds the local alias Account to that same definition. The root reexport binds User publicly to model::User. The root greet import points to service::greet.

Macro resolution and expansion identify format! and println! through applicable macro scope and prelude mechanisms. Their generated AST is integrated before full resolution completes. Hygiene keeps generated temporary names separate from prefix and user.

Late resolution enters greet's owner and ribs. The annotation Account resolves in the type namespace through the alias to model::User. The parameter account and local prefix become value bindings. Uses inside the formatted expression resolve to those locals. The method segment label remains subject to type-dependent associated method lookup.

In main, User resolves through the root reexport to the original struct definition. The field name is interpreted in the struct-literal context. greet resolves to service::greet, and user resolves to the local binding. Privacy checks approve the public routes while still recognizing that module ancestry matters.

Compact LocalDefIds identify local definitions during this session. Their conceptual paths resemble crate-root/model/User and crate-root/service/greet, with structured components and disambiguators rather than those literal strings. DefPathHash-style identities support metadata and incremental lookup; raw indices are not promised to survive another build.

Lowering creates HIR owners for item-like definitions and owner-local IDs for expressions and patterns. Macro-produced syntax is lowered with expansion provenance. The method call remains HIR suitable for type checking; HIR is not pre-filled with every final expression type.

Type checking infers account as &model::User, performs method lookup for label, checks formatting arguments, and verifies main's calls. This trace illustrates the handoff: lexical resolution supplies identities, while type and trait machinery completes associated selection.

29. Bug Triage by Symptom#

Begin triage by preserving the smallest source that still fails, the exact compiler version, edition, flags, and whether the build was incremental. Then classify the first wrong fact rather than the last panic.

For an unresolved import, inspect each path segment and namespace. Check module ancestry, aliases, edition, extern-prelude inputs, visibility, and whether a macro should generate the item. If changing source order fixes it, suspect collection or import fixed-point progress. If an explicit import works but a glob does not, inspect glob propagation and ambiguity state.

For an ambiguous import, list every candidate and its provenance. Determine whether candidates came from explicit imports, globs, preludes, or multiple namespaces. A valid ambiguity should be deterministic and explain both sources. An arbitrary winner indicates iteration-order dependence.

For hygiene failures, retain both macro definition and invocation. Rename call-site locals and move the macro across crate boundaries. Inspect syntax contexts and expansion backtraces, not only symbol text. If $crate reaches the caller, suspect context adjustment or defining-crate tracking.

For wrong spans, ask which stage created the diagnosed node. Compare the node's source span, call site, definition site, and desugaring kind. Check whether a suggestion is marked machine-applicable despite touching generated text. A correct error with an unusable underline is still a compiler bug.

For privacy issues, first prove resolution found the intended DefId. Then trace module ancestry, declared visibility, reexport chain, and effective visibility. If the error says “not found” for a known private item, resolution may be hiding privacy information too early.

For HIR lowering failures, print AST and HIR when possible and isolate the smallest sugar involved. Check owner switches, local-ID allocation, parent links, evaluation count, drop order, and provenance. Compare sugared code with a manually expanded semantic equivalent, while remembering the manual form may not exactly reproduce lang-item behavior.

For incremental identity problems, run a clean build, then an unchanged rebuild, then a targeted edit and rebuild. Compare against another clean build of the edited source. Log definition paths and hashes rather than only session-local IDs. Failures existing only after edits strongly implicate dependency tracking or identity translation.

symptom                         first subsystem to inspect
unresolved local               late ribs and declaration timing
unresolved generated item      expansion, early collection, hygiene
ambiguous glob                 import fixed point and provenance
private but clearly found      privacy/effective visibility
wrong method selected          type checking and trait/method lookup
HIR node missing               lowering owner and HirId allocation
second-build-only corruption   stable identity and incremental dependencies

30. Inspecting rustc Without Memorizing Paths#

Compiler source moves, modules split, and APIs change. Learn to navigate by concepts and public entry points rather than memorizing one checkout's filenames. Start from the official rustc-dev-guide, then follow links into the matching nightly API source.

The Name resolution chapter explains early and late resolution, namespaces, ribs, and overall strategy. The Macro expansion chapter explains iterative expansion and its resolver interaction. The HIR chapter introduces HIR, owners, identifiers, bodies, and map access. The AST lowering chapter describes conversion and owner-sensitive ID allocation. The HIR debugging page documents source-facing inspection techniques.

For the requested compiler line, the official nightly API currently identifies rustc_hir 1.97.1. Use the crate pages for rustc_resolve, rustc_ast_lowering, and rustc_hir. Versioned API docs are snapshots of unstable internals; verify the displayed version before relying on fields.

Search source for semantic types such as DefId, DefPathHash, HirId, OwnerId, Rib, and for diagnostic text from a failing test. Follow “Source” links from rustdoc to current definitions and callers. Search query providers to learn which stage computes a result, then follow the input types backward and consumers forward.

On a compiler built with nightly debugging options, useful commands include:

cargo +nightly rustc -- -Zunpretty=expanded
cargo +nightly rustc -- -Zunpretty=hir
cargo +nightly rustc -- -Zunpretty=hir-tree
rustc +nightly -Zunpretty=hir-tree example.rs

Unstable flag spelling and output are version-sensitive and not machine-stable interfaces. Use them to form hypotheses, not snapshot internal formatting forever. Macro backtraces, compiler logging, and targeted UI tests can reveal resolution provenance and spans.

When reading rustc_resolve, identify collection, import handling, early macro interaction, late visitor logic, and diagnostics by symbol search. In rustc_ast_lowering, start from the lowering entry point and search for the surface construct. In rustc_hir, inspect data definitions and map/query access separately. This navigation method survives directory rearrangements better than a memorized path list.

31. Exercises from Beginner to Contributor#

1. Draw the scope stack after every statement in the following function.

Mark which x each use denotes and explain why the initializer sees the old binding.

fn scope() {
	let x = 1;
	{
		let x = x + 1;
		println!("{x}");
	}
	println!("{x}");
}

1. Write one program where the same spelling legally names a type, value, macro, lifetime, and label.

For every occurrence, annotate its syntactic context and namespace. Then create a genuine ambiguity using two glob imports and explain why source order must not select a winner.

1. Build a three-module crate with one private module, one pub(crate) item,

and a public reexport. List all paths that resolve, then separately list which are accessible from a sibling module and another crate. Explain effective visibility for each route.

1. Extend the small resolver with a Namespace enum and one map per namespace.

Make a type and value coexist, but reject two values in one non-shadowing declaration scope. Add tests proving a value lookup never returns a type.

1. Add explicit imports and aliases to the toy resolver.

Represent imports as pending directives and iterate to a fixed point. Create an alias chain that needs multiple iterations, then detect a cycle with no productive resolution and issue one focused diagnostic.

1. Add a boundary kind distinguishing closures from nested functions.

Permit lookup across closure boundaries while recording captures. Reject local lookup across nested function boundaries. Test nested closures and identical names in inner scopes.

1. For for, ?, .await, and a let chain,

write conceptual IR that records evaluation order and source span for every generated operation. Identify which decisions need type checking rather than lexical resolution. Do not claim your drawing is exact rustc HIR.

1. Use -Zunpretty=hir-tree on a tiny program with one construct at a time.

Compare output from the compiler version you are studying with this chapter's conceptual traces. Record version-sensitive differences in owner structure, spans, and desugaring nodes.

1. Design an incremental identity edit matrix.

Try adding a statement inside one function, adding a same-named closure in a sibling function, renaming a module, and changing crate identity. Predict which semantic identities should remain reusable and which should change.

1. Find a rustc UI test for an unresolved import or privacy error.

Read its stderr expectations, locate the diagnostic implementation by searching its wording, and follow the resolved identity into the check. Propose one additional edge case with a clear expected message.

1. Investigate a method call supplied by a trait.

Document what lexical resolution decides, what type inference learns, and what method or trait lookup decides. Repeat with an inherent method of the same name and avoid saying the choice is purely lexical.

1. Create a macro hygiene test where definition-site and call-site locals share a spelling.

Add a $crate path and invoke the exported macro from another crate. Explain every token's intended lookup context and expected diagnostic span.

32. Contribution Pathways and Final Invariants#

Start contributing with a narrow, observable issue. Good first tasks include improving a diagnostic span, adding a regression test, documenting a version-sensitive owner rule, or reducing a known ICE. Avoid beginning by redesigning the entire resolver.

Reproduce with the exact toolchain and minimize source while preserving the failure. Search existing issues and tests for the diagnostic code or message. Read the compiler team's contribution instructions and the current rustc-dev-guide workflow. Ask focused design questions on the issue or compiler development channels, showing the reduced example and the stage you believe is first wrong.

A resolution change should test namespaces, shadowing, declaration order, imports, editions, macro expansion, hygiene, and privacy interactions relevant to the patch. A lowering change should test semantic behavior, HIR shape where appropriate, owner integrity, spans, suggestions, and recovered-error behavior. An identity change should include incremental and cross-crate scenarios, not only clean compilation.

Prefer source-facing UI tests for diagnostics. Use targeted internal tests when an invariant cannot be observed reliably from output. Do not freeze incidental debug formatting unless that formatting is the interface under test. Label conceptual explanations as conceptual when exact internals may evolve.

Keep these final invariants in view:

• Parsing determines grammatical structure; resolution connects uses to declarations.
• Namespace and syntactic context constrain lookup before spelling comparison can decide anything.
• Expansion and early resolution cooperate iteratively; expansion is not an independent completed pre-phase.
• Late resolution may finalize lexical names only after generated declarations are known.
• Successful resolution does not imply privacy accessibility.
• Associated method selection is not purely lexical; type inference and trait lookup participate.
• DefId is a compact in-session identity, not a promise across arbitrary builds.
• DefPathHash is a stable-oriented concept with defined limits, not eternal identity.
• NodeId, DefId, and HirId serve different stages and granularities.
• HIR is a largely untyped compiler IR consumed by type checking, not merely a typed AST.
• Lowering may desugar syntax but must preserve meaning, evaluation order, ownership, and diagnostic provenance.
• Every diagnostic should identify the first violated language rule at source the user can recognize.

The deepest lesson is architectural. Names acquire meaning through several cooperating systems rather than one dictionary lookup. Module collection supplies the global skeleton; ribs represent lexical change; hygiene protects token origin; privacy evaluates access; type checking completes type-dependent lookup; and lowering records the result in an owner-structured IR suitable for further reasoning. Understanding the boundaries is what turns a compiler reader into an effective compiler contributor.

Authoritative references used for this part:

• Rust Compiler Development Guide: Name resolution
• Rust Compiler Development Guide: Macro expansion
• Rust Compiler Development Guide: The HIR
• Rust Compiler Development Guide: AST lowering
• Rust Compiler Development Guide: HIR debugging
• Nightly rustc API: rustc_resolve
• Nightly rustc API: rustc_ast_lowering
• Nightly rustc API: rustc_hir 1.97.1

Part III-B: Resolver and HIR Internals in Practice#

This continuation begins where Chapter 32 stopped. It targets rustc 1.97.1 (July 2026), while treating compiler internals as version-sensitive. Names below describe that revision unless a language guarantee is explicitly identified. The stable Rust model later in this part is educational, not code copied from rustc.

33. One Lookup Is Several Questions#

“What does x mean?” is too small a specification for Rust. A resolver must know the syntactic role, namespace, lexical position, module, edition, and syntax context. It must also know whether lookup crossed a closure, item, macro, or module boundary.

Use this lookup key as a first mental model:

(symbol, namespace, lexical point, parent module, syntax context)

The answer is not merely an address. It records a definition or local identity, and often provenance used by diagnostics. Accessibility is a later question: a path can resolve to a private item and still be rejected.

Rust's principal namespaces are types, values, macros, and lifetimes. Labels use specialized lexical lookup. A tuple or unit struct can introduce both a type binding and a value constructor binding. The same spelling can therefore answer different queries without ambiguity.

struct S;

macro_rules! S { () => { 4 } }

fn main() {
	let S: S = S;
	let n = S!();
	assert_eq!(n, 4);
}

Prediction: which S is a pattern, type, expression, and macro? The pattern names the value constructor, the annotation names the type, the expression constructs the value, and S! names the macro.

This yields the first invariant:

A successful lookup must come from the namespace and hygiene context required by the use.

Equal text is insufficient. Combining all namespaces makes legal programs ambiguous. Splitting every syntactic category into a namespace would make imports and user expectations unmanageable. Rust's division is language policy, not a generic symbol-table optimization.

34. Ribs, Scope Entry, and Boundary Policy#

rustc uses stacks commonly called ribs for late lexical resolution. A rib represents a change in available bindings or in the rules for crossing a boundary. Think of one stack per relevant namespace, not one universal stack.

lookup `x`
    |
    v
[block rib: x -> local 9] -- found --> local 9
    |
[closure rib: capture allowed]
    |
[function rib: crossing locals forbidden]
    |
[outer block: x -> local 2]

The arrows carry a lookup request plus accumulated boundary information. The diagram omits hygiene and modules.

Scopes do not all begin at the opening brace. An item's binding is generally available throughout its item scope. A let binding is not available in its own initializer.

fn main() {
	let x = 1;
	let x = x + 1;
	assert_eq!(x, 2);
}

Trace:

1. Resolve the second initializer before installing its pattern bindings.
2. Its x finds the first local.
3. Install the second x after the initializer.
4. The assertion finds the second local.

Installing the pattern first would silently produce self-reference. Discarding the old binding would make scope exit and diagnostics harder.

Closures may cross a closure rib and record a capture. Nested function items may not capture an enclosing local.

fn outer() {
	let answer = 42;
	let closure = || answer;
	fn item() -> i32 {
		// `answer` is unavailable here.
		0
	}
	assert_eq!(closure() + item(), 42);
}

Resolution identifies that the closure body uses the outer local. Later capture analysis determines capture mode and representation. Do not make lexical resolution decide whether capture is by shared borrow, mutable borrow, or move.

Labels form another stack-like environment. They are not values and cannot be imported.

fn main() {
	'outer: loop {
		loop {
			break 'outer;
		}
	}
}

A useful resolver records push/pop operations in a debug trace. If unrelated functions see each other's locals, suspect an unbalanced pop. If only nested items fail, inspect boundary kinds before changing ordinary shadowing.

35. Modules, Imports, Globs, and Visibility#

Module resolution is graph construction over a tree-shaped module skeleton. Imports add named or glob edges; reexports can create public routes to definitions. An import is not textual inclusion.

mod engine {
	pub(crate) struct Core;
	pub mod public {
		pub struct Handle;
	}
}

use engine::public::Handle as H;

fn build(_: H) {}

The alias adds H to the importing scope. It does not rename Handle at its definition. The Core path can resolve inside the crate but is not externally accessible.

Explicit imports usually outrank glob candidates. Two globs can leave a name ambiguous. Source order must not select a winner.

mod left { pub struct Token; }
mod right { pub struct Token; }
use left::*;
use right::*;

// fn consume(_: Token) {} // ambiguous

Imports can depend on other imports, so collection cannot always finish in one textual pass. A conceptual fixed-point algorithm is:

collect direct module declarations
repeat
    resolve every directive whose prefix is known
    add newly determined bindings and glob contributions
until no binding changes
diagnose unresolved, cyclic, or ambiguous directives

Real rustc is demand-driven and optimized; this is a semantic teaching model. A fixed point must be monotone: candidates are discovered, not silently replaced by traversal order. Termination requires finite directives and candidate sets.

Visibility answers who may name an item through a route. Privacy checks use resolved identities and their module ancestry. pub on an item does not make a private containing module externally reachable. pub(in path), pub(super), and pub(crate) restrict the visibility domain.

Prediction:

mod hidden {
	pub struct Public;
}
pub use hidden::Public;

External code cannot traverse hidden, but can use the public reexport from the crate root. The definition and exported route are distinct facts.

Keep three records separate:

This separation enables diagnostics such as “private item imported here” rather than “not found.” Collapsing inaccessible into unresolved loses useful evidence and can produce wrong suggestions.

36. Expansion, Early Resolution, and Hygiene#

Macro expansion and name resolution interleave. The compiler must resolve enough of a macro invocation to choose an expander. Expansion can then create modules, imports, macros, and items that future lookup needs.

collect invocations
       |
       v
resolve macro names -- unresolved --> wait / diagnose at fixed point
       |
       v
expand one invocation
       |
       v
integrate generated syntax and bindings
       |
       +-----------------------------> repeat

This diagram carries token trees and newly discovered bindings. It omits proc-macro process boundaries and error placeholders.

Hygiene prevents equal-looking identifiers from being confused solely because expansion pasted them together. Tokens carry a syntax context reflecting expansion history. Informally, definition-site context protects names written by a macro definition, while call-site context allows metavariables to refer to the caller's names. The exact context-adjustment algorithms are rustc internals.

macro_rules! use_argument {
	($name:ident) => {{
		let temporary = 10;
		$name + temporary
	}};
}

fn main() {
	let temporary = 100;
	assert_eq!(use_argument!(temporary), 110);
}

The $name token originates at the call site and resolves to the caller's temporary. The literal temporary in the macro body has definition-associated context and denotes the generated local. Text-only lookup would be capture-prone.

$crate gives an exported declarative macro a hygienic route to its defining crate. It is not equivalent to spelling the current caller's crate name. Edition changes can affect path-root interpretation, so tests should include the relevant editions.

Procedural macros return tokens with spans and contexts through stable interfaces, but rustc's internal representation and context transformations are not stable APIs. Never infer a language guarantee from one -Z dump.

For a hygiene bug, minimize to one generated identifier and record:

• where the token was authored;
• which expansion produced it;
• its span and syntax context;
• which namespace was queried;
• what context adjustment occurred;
• which candidate won.

The first broken invariant often precedes the eventual “cannot find value” diagnostic.

37. Paths Stop Being Purely Lexical#

A path has segments, and not every segment is selected by the lexical resolver. The first segment may establish a module, type, local, or language-defined root. Later segments can cross into associated-item lookup, which needs type information.

trait Make {
	fn make() -> Self;
}

struct Widget;

impl Make for Widget {
	fn make() -> Self { Widget }
}

fn main() {
	let _ = <Widget as Make>::make();
}

Lexical resolution can identify Widget and Make. Selecting and validating make depends on trait and type checking machinery. Likewise, receiver.method() is not resolved by searching a value rib for method.

`crate::m::Type::item`
  crate ---- module root
  m -------- module child lookup
  Type ----- type-namespace binding
  item ----- possible associated-item boundary

The exact boundary depends on path form and semantic context. Do not hard-code “the third segment” as associated. Qualified paths can state both a type and trait. Imports of traits can affect method availability without importing each method as a lexical value.

Counterexample: a module and a type can have similar-looking paths. Assuming every :: means module traversal rejects valid associated constants. Assuming every suffix is associated lookup breaks ordinary nested modules.

Resolution should preserve partial information for later phases. It must not guess an associated item merely to eliminate uncertainty. Representing “prefix resolved; suffix deferred” is more correct than inventing certainty.

38. Generics, Lifetimes, Labels, and Captures#

Generic type and const parameters inhabit different namespaces. Lifetime parameters have dedicated resolution and scope rules. Higher-ranked binders introduce lifetimes only inside their binder.

fn copy_n<'a, T, const N: usize>(input: &'a [T; N]) -> &'a [T; N] {
	input
}

T is a type parameter, N is a const parameter used in a value-like const position, and 'a is a lifetime parameter. Their common angle-bracket syntax does not imply one symbol table.

fn accepts<F>(f: F)
where
	F: for<'a> Fn(&'a str) -> &'a str,
{
	let _ = f;
}

The binder owns 'a; it must not leak outside the bound. Late-bound versus early-bound lifetime classification has consequences beyond spelling lookup, and current rustc code may perform related work across resolution and later analysis. Check the 1.97.1 source before changing that boundary.

Labels are lexically resolved and can shadow labels. A label use must not resolve to a lifetime with equal text. The apostrophe is shared syntax, not shared semantic identity.

Closures add a second result to local resolution: capture evidence.

resolved use: local L7
crossed boundaries: [block, closure C2, closure C1]
capture evidence: C2 and C1 refer to an outer local

Later analyses refine this evidence into place projections and capture kinds. A resolver that prematurely stores “capture by value” mixes lexical mechanism with ownership policy.

Prediction: does move || x.len() prove the captured representation immediately? No. move constrains capture semantics, but detailed capture analysis and type information still matter.

39. Identity: From Syntax Nodes to Definitions and HIR#

Compiler identities answer different questions and have different lifetimes. Treating them as interchangeable creates incremental and diagnostic bugs.

A DefId is conceptually (CrateNum, DefIndex). CrateNum allocation can differ between sessions. Do not persist raw DefId values as durable cache keys.

A definition path describes semantic ancestry and includes disambiguation where names alone do not suffice. Hashing it with crate identity yields DefPathHash. Changes to crate identity or relevant definition ancestry can change the hash. Collisions are engineered to be extraordinarily unlikely and guarded where appropriate, but a hash is not a mathematical proof of global identity forever.

crate name / disambiguator
          |
          v
    StableCrateId
          |
definition path: module `m` -> function `f` -> closure #0
          |
          v
      DefPathHash

Numbering anonymous definitions by traversal position creates edit sensitivity. rustc's precise disambiguation and stable-hashing behavior are version-sensitive. Test sibling insertion, movement, and macro-generated definitions rather than promising perfect stability.

HIR uses owner-local indexing. Conceptually, HirId = (OwnerId, LocalItemId). Changing one body can then mostly perturb local indices within that owner instead of renumbering the entire crate. The owner boundary is an incremental-compilation and query boundary as well as an organizational device.

Invariant:

Every lowered HIR node belongs to exactly one owner, and its owner-local index is valid in that owner's node table.

Symptoms of violation include ICEs during parent lookup, wrong spans, unstable fingerprints, or a node attributed to a sibling body.

40. AST-to-HIR Lowering and Desugaring#

AST preserves source-oriented grammar and expansion products. HIR presents a more uniform, resolved, owner-organized representation to later analyses. HIR is not simply “AST with types”; most type checking follows lowering.

Lowering consumes resolution results rather than redoing lookup by spelling. It maps identities, establishes owners, assigns owner-local HIR ids, records spans, and translates source constructs into forms expected by later phases.

expanded AST + resolution maps
             |
             v
      owner preallocation
             |
             v
 lower item signatures and bodies
             |
             v
 assign owner-local HirIds + provenance
             |
             v
             HIR

Preallocating owner identities avoids depending on accidental recursive visitation order. The exact rustc lowering entry points and query arrangement can change.

Desugaring replaces convenient syntax with a smaller semantic vocabulary. It must preserve evaluation order, control flow, drop behavior, and source provenance. Conceptual examples include:

A conceptual for trace is:

evaluate iterable expression once
convert it to an iterator
loop:
    ask for next element
    if Some(value): match value against pattern and execute body
    if None: break

This is not exact rustc HIR. It intentionally omits lang items, adjustment details, and precise temporary scopes.

Prediction: may lowering duplicate iterable() because the conceptual expansion mentions it near a loop? No. That would change side effects and violate evaluate-once semantics.

Generated nodes need provenance. A diagnostic should point to the user's ?, .await, pattern, or loop expression where possible, not an invisible synthetic match arm. rustc spans can carry expansion information and desugaring reasons. Exact enums and APIs in 1.97.1 must be confirmed in source.

41. A Stable-Rust Resolver Model#

The following complete program is deliberately small. Its purpose is to make namespaces, ribs, boundaries, shadowing, and capture evidence executable. It uses stable Rust and the standard library. It omits modules, imports, hygiene, labels, lifetimes, associated items, privacy, and recovery.

use std::collections::HashMap;

#[derive(Clone, Copy, Debug, Eq, Hash, PartialEq)]
enum Namespace {
	Type,
	Value,
	Macro,
}

#[derive(Clone, Copy, Debug, Eq, PartialEq)]
enum Boundary {
	Block,
	Closure(u32),
	Function,
}

#[derive(Clone, Copy, Debug, Eq, PartialEq)]
struct Binding(u32);

#[derive(Debug, Eq, PartialEq)]
struct Found {
	binding: Binding,
	captured_by: Vec<u32>,
}

#[derive(Debug)]
struct Rib {
	boundary: Boundary,
	bindings: HashMap<(Namespace, String), Binding>,
}

#[derive(Debug)]
struct Resolver {
	ribs: Vec<Rib>,
}

impl Resolver {
	fn new() -> Self {
		Self { ribs: Vec::new() }
	}

	fn push(&mut self, boundary: Boundary) {
		self.ribs.push(Rib {
			boundary,
			bindings: HashMap::new(),
		});
	}

	fn pop(&mut self) {
		assert!(self.ribs.pop().is_some(), "unbalanced rib pop");
	}

	fn bind(&mut self, ns: Namespace, name: &str, binding: Binding) {
		let rib = self.ribs.last_mut().expect("binding without a rib");
		let old = rib.bindings.insert((ns, name.to_owned()), binding);
		assert!(old.is_none(), "duplicate binding in one rib");
	}

	fn resolve(&self, ns: Namespace, name: &str) -> Result<Found, String> {
		let mut closures = Vec::new();
		for rib in self.ribs.iter().rev() {
			if let Some(&binding) = rib.bindings.get(&(ns, name.to_owned())) {
				return Ok(Found {
					binding,
					captured_by: closures,
				});
			}
			match rib.boundary {
				Boundary::Block => {}
				Boundary::Closure(id) => closures.push(id),
				Boundary::Function => {
					return Err(format!("`{name}` cannot cross a function boundary"));
				}
			}
		}
		Err(format!("unresolved {ns:?} name `{name}`"))
	}
}

fn main() {
	let mut r = Resolver::new();
	r.push(Boundary::Block);
	r.bind(Namespace::Type, "Item", Binding(1));
	r.bind(Namespace::Value, "x", Binding(2));
	r.bind(Namespace::Macro, "show", Binding(3));

	r.push(Boundary::Closure(7));
	assert_eq!(
		r.resolve(Namespace::Value, "x"),
		Ok(Found { binding: Binding(2), captured_by: vec![7] })
	);
	r.push(Boundary::Block);
	r.bind(Namespace::Value, "x", Binding(4));
	assert_eq!(
		r.resolve(Namespace::Value, "x"),
		Ok(Found { binding: Binding(4), captured_by: vec![] })
	);
	assert!(r.resolve(Namespace::Value, "Item").is_err());
	r.pop();
	r.pop();

	r.push(Boundary::Function);
	assert!(r.resolve(Namespace::Value, "x").is_err());
	r.pop();
	r.pop();
}

Expected behavior: all assertions pass and the program prints nothing. The inner x requires no capture because it is found before crossing the closure rib. Item cannot answer a value query. The nested function cannot reach the outer x.

The use of name.to_owned() during every probe is intentionally simple, not efficient. A production implementation interns symbols and avoids allocation in lookup. Optimization must preserve namespace and context equality.

Tests to add before extending the model:

1. Two namespaces can bind the same spelling.
2. Two bindings in one namespace and rib are rejected.
3. An inner block shadows and popping restores the outer binding.
4. Nested closures record both closure ids in inner-to-outer order.
5. A function boundary rejects an outer local.
6. An unresolved spelling produces an error without mutating state.

42. Add Owner-Based Lowering Without Pretending It Is rustc#

Now derive a tiny lowering representation. The mechanism allocates definitions for owners and local indices for nodes. The policy chooses which syntax creates an owner. The presentation layer prints source spans. Keeping these separate mirrors useful rustc boundaries.

The complete stable-Rust program below lowers named functions containing integer addition. It omits parsing, types, macros, modules, patterns, real spans, recovery, and incremental persistence.

use std::collections::HashMap;

#[derive(Clone, Copy, Debug, Eq, Hash, PartialEq)]
struct DefId(u32);

#[derive(Clone, Copy, Debug, Eq, PartialEq)]
struct HirId {
	owner: DefId,
	local: u32,
}

#[derive(Clone, Debug)]
enum AstExpr {
	Int(i64, u32),
	Add(Box<AstExpr>, Box<AstExpr>, u32),
}

#[derive(Clone, Debug)]
struct AstFn {
	name: String,
	body: AstExpr,
	span: u32,
}

#[derive(Debug)]
enum HirExprKind {
	Int(i64),
	Add(Box<HirExpr>, Box<HirExpr>),
}

#[derive(Debug)]
struct HirExpr {
	id: HirId,
	kind: HirExprKind,
	span: u32,
}

#[derive(Debug)]
struct HirOwner {
	def: DefId,
	name: String,
	span: u32,
	body: HirExpr,
}

struct Lowerer {
	next_def: u32,
	owners: HashMap<String, DefId>,
	local: u32,
}

impl Lowerer {
	fn new() -> Self {
		Self { next_def: 0, owners: HashMap::new(), local: 0 }
	}

	fn preallocate(&mut self, functions: &[AstFn]) {
		for function in functions {
			let def = DefId(self.next_def);
			self.next_def += 1;
			assert!(self.owners.insert(function.name.clone(), def).is_none());
		}
	}

	fn lower_function(&mut self, function: &AstFn) -> HirOwner {
		let def = self.owners[&function.name];
		self.local = 0;
		let body = self.lower_expr(def, &function.body);
		HirOwner { def, name: function.name.clone(), span: function.span, body }
	}

	fn lower_expr(&mut self, owner: DefId, expression: &AstExpr) -> HirExpr {
		let id = HirId { owner, local: self.local };
		self.local += 1;
		match expression {
			AstExpr::Int(value, span) => HirExpr {
				id,
				kind: HirExprKind::Int(*value),
				span: *span,
			},
			AstExpr::Add(left, right, span) => {
				let left = self.lower_expr(owner, left);
				let right = self.lower_expr(owner, right);
				HirExpr {
					id,
					kind: HirExprKind::Add(Box::new(left), Box::new(right)),
					span: *span,
				}
			}
		}
	}
}

fn eval(expression: &HirExpr) -> i64 {
	match &expression.kind {
		HirExprKind::Int(value) => *value,
		HirExprKind::Add(left, right) => eval(left) + eval(right),
	}
}

fn count_and_check(expression: &HirExpr, owner: DefId, seen: &mut Vec<u32>) {
	assert_eq!(expression.id.owner, owner);
	seen.push(expression.id.local);
	match &expression.kind {
		HirExprKind::Int(_) => {}
		HirExprKind::Add(left, right) => {
			count_and_check(left, owner, seen);
			count_and_check(right, owner, seen);
		}
	}
}

fn main() {
	let functions = vec![
		AstFn {
			name: "one".into(),
			body: AstExpr::Int(1, 11),
			span: 10,
		},
		AstFn {
			name: "sum".into(),
			body: AstExpr::Add(
				Box::new(AstExpr::Int(20, 21)),
				Box::new(AstExpr::Int(22, 23)),
				20,
			),
			span: 19,
		},
	];

	let mut lowerer = Lowerer::new();
	lowerer.preallocate(&functions);
	let owners: Vec<_> = functions.iter().map(|f| lowerer.lower_function(f)).collect();
	assert_eq!(eval(&owners[0].body), 1);
	assert_eq!(eval(&owners[1].body), 42);
	assert_ne!(owners[0].def, owners[1].def);
	assert_eq!(owners[0].body.id.local, 0);
	assert_eq!(owners[1].body.id.local, 0);

	let mut seen = Vec::new();
	count_and_check(&owners[1].body, owners[1].def, &mut seen);
	seen.sort_unstable();
	assert_eq!(seen, vec![0, 1, 2]);
	assert_eq!(owners[1].name, "sum");
	assert_eq!(owners[1].span, 19);
	assert_eq!(owners[1].body.span, 20);
}

Expected behavior: all assertions pass and the program prints nothing. Each function restarts local indexing at zero because the owner disambiguates HirId. Preallocation means a function body can refer to an already identified sibling in a future extension.

The model's numeric DefId is deliberately session-local. To explore stable identity, derive a definition path from crate identity, parent path, name, and disambiguator, then hash a canonical encoding with a specified algorithm. Do not use DefaultHasher as a persistence format: its stability is not promised.

Milestone extensions:

1. Add variable AST nodes whose lowering consumes resolver identities, never raw text.
2. Add a synthetic node and retain both generated span and source-cause span.
3. Add nested closures as owners and prove owner-local ids never cross.
4. Fingerprint each owner from semantic inputs and compare clean versus edited runs.
5. Add error nodes and prove lowering terminates after one unresolved variable.

Explicit omission: this model does not reproduce rustc query keys, arena allocation, interning, exact owner rules, DefPathData, span encoding, or stable-hash infrastructure. Its value is the invariants it makes testable.

43. Stable Hashing, Incremental Boundaries, and Provenance#

Incremental compilation asks whether a result can be reused under an explicit observation model. It does not mean every numeric id stays unchanged. Queries need stable-oriented keys and fingerprints of relevant inputs.

Owner boundaries localize work. If a function body changes, unrelated owners should often remain reusable. Changing a module name can alter descendant definition paths and invalidate more identities. Changing crate disambiguation can alter the stable crate identity and therefore every descendant path hash.

source / metadata inputs
        |
        v
query key (stable-oriented identity)
        |
        v
dependency edges ---- changed input? ---- yes --> recompute
        |                                   
        no
        |
        v
reuse cached result after validation

A cache creates correctness obligations. Omitting edition, features, target-sensitive facts, hygiene, or resolved identity from a fingerprint can reuse a wrong result. Hashing incidental addresses or map iteration order creates needless misses. Deterministic traversal is part of reproducibility.

Provenance is also an incremental input when it affects diagnostics. Two semantically equivalent generated nodes can still require different source spans. If diagnostic output is observed, span changes cannot always be ignored. Separate semantic hashing from diagnostic hashing only when the query architecture explicitly supports it.

An edit experiment matrix:

These are predictions, not guarantees for every 1.97.1 query. Measure with that compiler's incremental diagnostics rather than reasoning from names alone.

44. Recovery and Diagnostics Preserve What Went Wrong#

A production resolver cannot stop at the first unknown name. It creates error-marked results where safe, suppresses cascades, and preserves enough structure for lowering and later diagnostics. Recovery must not turn uncertainty into a valid-looking arbitrary binding.

For an unresolved name, useful evidence includes:

• spelling and namespace;
• use span and expansion ancestry;
• lexical and module scopes searched;
• inaccessible candidates;
• candidates from wrong namespaces;
• import and glob provenance;
• edit-distance suggestions;
• boundary that blocked a local capture.

An ambiguity diagnostic should show competing routes. A privacy diagnostic should identify the item and restrictive boundary. A hygiene diagnostic should prefer source the user wrote, while retaining expansion backtraces when useful.

Counterexample: changing “private” into “not found” may seem simpler. It discards proof that lookup succeeded and often suggests importing the exact inaccessible item again.

Recovery invariants:

1. Emit the primary error at the earliest source-recognizable broken rule.
2. Mark the failed result so downstream code does not repeat the same complaint.
3. Continue only with structurally valid placeholders.
4. Never expose a private or ambiguous candidate as uniquely valid.
5. Preserve spans and parentage for later diagnostics.
6. Avoid ICEs when an owner contains error nodes.

Prediction: if a generated path fails, should the message always point inside macro output? No. The best primary span may be the invocation or a metavariable supplied by the caller. Expansion provenance determines which location is actionable.

45. Reading rustc 1.97.1 and Finding the First Broken Invariant#

Start from behavior, not a guessed data structure. Pin the exact toolchain, minimize the program, and identify whether parsing, expansion, early resolution, late resolution, privacy, lowering, type checking, or diagnostics first diverges.

Useful source areas in the 1.97.1 tree include:

• compiler/rustc_resolve/ for module, import, early, and late resolution;
• compiler/rustc_ast_lowering/ for AST-to-HIR lowering;
• compiler/rustc_hir/ for HIR structures and maps;
• compiler/rustc_span/ for symbols, spans, hygiene, and stable-oriented identifiers;
• compiler/rustc_middle/ for query integration and later compiler context;
• tests/ui/resolve/, macro, privacy, and HIR-related tests for observable contracts.

Directory names are more durable than individual function names, but still verify them in the tagged source. Nightly API documentation reflects the generated docs for a particular build, not a stable compiler API.

On a matching development toolchain, useful investigations may include:

rustc +nightly -Zunpretty=hir-tree example.rs
rustc +nightly -Zunpretty=expanded example.rs
rustc +nightly -Ztrace-macros example.rs
RUSTC_LOG=rustc_resolve=debug rustc +nightly example.rs

Flags and tracing targets are unstable and may differ or require a compiler built with suitable tracing. Use rustc -Z help and inspect current source before relying on them. Never place unstable flags in a user's normal build merely to fix a compiler bug.

A disciplined debugging trace is:

1. Is parsed syntax what the source intended?
2. Did expansion produce the expected tokens, spans, and contexts?
3. Was the declaration collected in the correct namespace and module?
4. Did import fixed-point processing expose the intended candidate?
5. Did rib lookup stop or capture at the correct boundary?
6. Did privacy classify the resolved route correctly?
7. Did lowering consume the recorded identity and assign the correct owner?
8. Did a desugaring preserve order, scope, and source provenance?
9. Did later type-dependent lookup incorrectly receive blame for an earlier failure?

Bisect dumps carefully. Pretty-printed output can omit identity and hygiene details. Debug HIR is version-sensitive and can change without a language change. Prefer assertions on semantic behavior and diagnostics over snapshots of incidental field order.

46. Contributor Workshops and Mastery Tests#

Workshop A: an import ambiguity#

Create two modules exporting the same type through globs. Confirm the ambiguity appears only when the name is used. Add an explicit import and predict which candidate wins. Then inspect the UI test suite for the expected diagnostic style.

Deliverables:

• minimized source;
• candidate/provenance table;
• edition and compiler revision;
• explanation of why source order is not a tie-breaker;
• one regression test that does not freeze unrelated wording.

Workshop B: closure versus nested item#

Put the same local use in a closure and nested function. Record the rib boundaries crossed. Explain why one becomes capture evidence and one errors. Extend the Chapter 41 model with two nested closures and test capture order.

Counterexample to investigate: a move closure does not make a nested fn capture legal.

Workshop C: hygiene provenance#

Write a macro with a definition-authored temporary, a call-site identifier metavariable, and $crate. Invoke it from another crate. For every identifier, record origin, context expectation, namespace, and intended definition. Force one failure and decide whether invocation, argument, or macro definition is the actionable span.

Workshop D: associated-item boundary#

Compare module::item, Type::CONST, <Type as Trait>::item, and value.method(). Mark the lexical prefix and type-dependent suffix. Import and remove the trait to observe method availability. Do not describe method selection as ordinary rib lookup.

Workshop E: owner integrity#

Extend the lowering model with closures as owners. Write a validator that checks unique local ids, correct owner ids, and valid parent edges. Inject a bug that fails to reset local numbering. Explain why globally unique local numbers might hide the bug rather than establish the intended invariant.

Workshop F: incremental identity#

Build a crate twice with a small edit under a matching rustc development environment. Compare query reuse and stable-oriented identities where tools permit. Distinguish expected invalidation from accidental instability. Document toolchain flags because these interfaces are unstable.

Production review checklist#

Before proposing a resolver change, ask:

1. Which namespaces are affected?
2. Which editions alter path behavior?
3. Can macros generate either side of the case?
4. Are syntax contexts compared or adjusted correctly?
5. Do explicit and glob imports behave deterministically?
6. Is successful lookup separated from accessibility?
7. Are closure and item boundaries distinct?
8. Are labels and lifetimes kept separate?
9. Is associated lookup deferred to the phase with type information?
10. Does recovery preserve uncertainty without cascading?

Before proposing a lowering change, ask:

1. Which construct owns each new node?
2. Are owner-local ids allocated exactly once?
3. Was resolution consumed rather than repeated from text?
4. Is evaluation order unchanged?
5. Are temporary and drop scopes unchanged?
6. Are generated spans tied to actionable source?
7. Are error placeholders accepted safely?
8. Which query fingerprints change?
9. Is debug output being mistaken for a stable contract?
10. Is a UI, MIR, HIR, incremental, or unit test the narrowest durable test?

Final prediction set#

Predict before running each experiment:

• Can a type and local value share a spelling? Yes, because namespaces differ.
• Can two glob candidates silently use the later import? No, ambiguity is not source-order shadowing.
• Can a resolved private item still fail? Yes, accessibility is separate.
• Can a closure capture an outer local? Yes, and resolution records evidence.
• Can a nested function item do the same? No.
• Does a DefId persist safely in an arbitrary external cache? No.
• Does a HirId identify a node without its owner? No.
• Is HIR fully typed? No.
• May lowering duplicate a side-effecting expression during desugaring? No.
• Does pretty-printed expanded code fully reveal hygiene? No.
• Is every path segment lexically resolved? No, associated lookup can require types.
• Does stable hashing remove the need to define inputs? No; it makes that obligation sharper.

Contribution exercises#

1. Find one 1.97.1 resolver diagnostic and trace it from UI test to emission site.
2. Add a wrong-namespace candidate and evaluate whether the suggestion improves.
3. Locate the current rib boundary variants and map each to a language example.
4. Locate import resolution's handling of glob ambiguity and write a state trace.
5. Trace a public reexport through resolution and privacy checking.
6. Find where a local use becomes closure capture evidence; identify later refinement.
7. Dump HIR for for, ?, .await, and a let chain separately.
8. Compare each dump with Chapter 40's conceptual account and list omissions.
9. Locate current HirId and OwnerId definitions and verify the owner-local invariant.
10. Locate DefPathHash construction and list all crate-identity inputs in that revision.
11. Design an edit that should preserve a sibling owner's fingerprint and measure it.
12. Improve one span without changing semantic resolution, then add a focused UI test.
13. Add malformed macro output and prove recovery does not ICE during lowering.
14. Review a recent resolver or lowering pull request and identify its invariant and test matrix.

Mastery means being able to locate the earliest broken invariant, not memorizing struct fields. Representations make some questions cheap and others expensive. Ribs make lexical shadowing cheap but require explicit boundary policy. Module graphs make item lookup and imports tractable but require fixed-point and ambiguity handling. Hygiene preserves token origin but makes text-only debugging inadequate. Owner-local HIR makes incremental isolation practical but creates ownership integrity obligations. Stable hashing enables reuse only after inputs and observation boundaries are defined. Desugaring simplifies later analyses only by moving responsibility for semantic preservation into lowering.

47. Version Caveats and Authoritative Reading Map#

The Rust Reference defines language behavior; it does not promise rustc's internal staging or type names. The rustc-dev-guide explains architecture but can lead or lag a particular release. Generated nightly rustc API docs describe one compiler build and remain unstable. Tagged source and tests are the final evidence for claims about rustc 1.97.1 internals.

In particular, verify against the 1.97.1 source before relying on:

• exact early/late resolver division;
• rib enums and boundary variants;
• import fixed-point work queues;
• hygiene context-adjustment helper names;
• which AST nodes receive NodeIds;
• OwnerId wrappers and owner-node rules;
• exact HIR desugarings;
• stable-hash field inputs;
• query boundaries and logging targets;
• unstable -Z output formats.

Authoritative starting points:

• The Rust Reference: Names and scopes
• The Rust Reference: Namespaces
• The Rust Reference: Paths
• The Rust Reference: Visibility and privacy
• The Rust Reference: Macros by example hygiene
• Rust Compiler Development Guide: Name resolution
• Rust Compiler Development Guide: Macro expansion
• Rust Compiler Development Guide: The HIR
• Rust Compiler Development Guide: AST lowering
• Rust Compiler Development Guide: HIR debugging
• Rust Compiler Development Guide: Incremental compilation
• rustc 1.97.1 source tree
• Generated rustc internal API documentation

Read in this order: Reference for guarantees, dev guide for a map, tagged source for implementation, and tests for user-observable behavior. When they disagree, state the revision and classify the claim rather than averaging the sources.

Part IV: Type Inference, Traits, Coherence, and the Solver#

1. The question rustc is really answering#

From labels to evidence#

A beginner may imagine type checking as a table lookup.

The declaration says that x is an integer, so the compiler looks up x and checks the next operation.

That picture is useful for one minute, but real Rust immediately outgrows it.

fn choose<T>(left: T, right: T, first: bool) -> T {
	if first { left } else { right }
}

let answer = choose(1, 2, true);

No declaration spells out the concrete type argument supplied for T.

The literals are initially compatible with several integer types.

The call, the function signature, and later uses of answer jointly restrict the answer.

A restriction such as “these two types must be equal” is a constraint.

Type inference collects and solves constraints.

Type checking asks whether the resulting assignments obey every language rule.

Trait checking adds propositions such as “this type implements Clone.”

A proposition is a statement that can be true or false under the language rules.

An obligation is a proposition rustc still has to justify.

The justification is informally called a proof.

This is not proof in the sense of storing a beautiful mathematical derivation for every program.

It means rustc follows trusted rules from known premises to an acceptable conclusion.

Why constraints and proofs matter#

Consider xs.iter().map(|x| x + 1).collect().

Method lookup needs the receiver type.

Closure checking needs the iterator item type.

Addition introduces an Add obligation and an associated output type.

Collection introduces a FromIterator obligation whose target may be inferred from context.

These questions are mutually connected, so a one-way lookup is insufficient.

Rustc must tolerate temporary ignorance while refusing contradictions.

Its central invariant is that accepted code has a consistent assignment of inferred values and discharged required obligations.

“Discharged” means proved, deferred to a valid generic assumption, or recorded for a later phase that is responsible for it.

The architecture therefore separates local guesses, reusable logical queries, and final checks.

When debugging, ask which constraint was created, where it was registered, and when it was forced.

That sequence is often more useful than asking merely “what type did rustc choose?”

2. A small vocabulary of types#

Types classify values#

A type describes which values and operations are allowed.

bool, u32, &str, and Vec<String> are types.

Vec alone is a type constructor in ordinary explanatory language: it needs an element argument before describing ordinary vector values.

A generic parameter is a named blank such as T in Vec<T>.

A generic argument fills a blank, as u8 fills T in Vec<u8>.

Rust generic arguments are not only types.

They can be lifetimes, which rustc generally calls regions, and compile-time values, which it calls consts.

struct Window<'a, T, const N: usize> {
	items: &'a [T; N],
}

Here the arguments have three different kinds: region 'a, type T, and const N.

“Kind” in this sentence means the category of an argument, not TyKind yet.

Keeping categories distinct prevents nonsense such as using a lifetime where an array length belongs.

Equality is not the only relation#

Type equality says two types denote the same type after permitted normalization.

Subtyping says a value of one type may be used where another is required without an ordinary user-defined conversion.

Coercion is a compiler-directed conversion at designated sites.

Trait implementation says a type satisfies an interface and its semantic contract.

Well-formedness says a type or predicate obeys the prerequisites needed merely to make sense.

Outlives says one region or type remains valid for at least as long as another region.

These relations interact, but combining them into one vague “matches” operation would lose essential rules and diagnostics.

Open-world caution#

Generic checking does not know all future callers.

Inside fn f<T: Display>(x: T), rustc normally proves displayability from the parameter bound.

It does not select one concrete Display impl for unknown T.

This distinction is crucial: trait selection during type checking may succeed through an assumption, may remain ambiguous because inference variables are unknown, or may identify an impl candidate.

Selection does not always pick a concrete impl in generic code.

3. From source syntax to HIR types#

The surface-shaped representation#

After parsing, macro expansion, and name resolution, rustc lowers source into HIR, the High-level Intermediate Representation.

HIR is compiler data that remains recognizably close to Rust source while removing some syntactic variation.

An rustc_hir::Ty represents a type as written in HIR.

It retains source-oriented structure and identifiers useful for precise diagnostics.

For example, a path, tuple, reference, inferred _, or trait object has an HIR form connected to a span.

A span identifies a region of source text.

HIR types are not the final currency of type reasoning.

The compiler must resolve paths, instantiate omitted arguments, create inference variables, elaborate shorthand, and attach semantic identities.

That produces internal types.

source text
    |
parser and expansion
    |
resolved, lowered HIR: rustc_hir::Ty
    |
type lowering and inference
    |
interned semantic type: Ty<'tcx>

Why preserve both worlds#

The internal type Vec<u8> knows the definition identity of Vec, not merely its spelling.

This avoids confusion between same-named items from different modules.

The HIR node remembers how the user expressed the type and where.

Diagnostics often need both facts.

If well-formedness fails inside Outer<Inner<T>>, walking an HIR type can point at Inner<T> rather than underlining the entire item.

If a bug prints a correct semantic type but highlights the wrong token, inspect HIR-to-type correspondence and obligation causes before blaming inference.

Lowering is contextual#

The meaning of a type can depend on scope, generic parameters, elided lifetimes, and the parameter environment.

Self means different semantic types in different owners.

_ requests inference rather than naming a stable type.

Associated type syntax may lower to a projection that still needs normalization.

Therefore lowering is not a purely textual conversion.

Its invariant is that internal terms refer to resolved compiler identities and represent unresolved facts explicitly rather than inventing answers.

4. Ty<'tcx>, TyKind, and generic arguments#

Internal semantic types#

In compiler code, Ty<'tcx> is the common handle for an interned type.

Conceptually, examining it yields a TyKind, an enum-like classification such as primitive, tuple, reference, function, parameter, inference variable, alias, or algebraic data type.

Exact variants and helper APIs evolve.

Always consult the nightly rustdoc matching the checkout instead of memorizing a historical list.

The lifetime 'tcx ties the handle to a compiler context.

It is not a lifetime written by the program being compiled.

That difference prevents a frequent beginner mistake: Rust compiler implementation lifetimes and user-program regions solve different bookkeeping problems.

Arguments are one ordered list#

An instantiated definition carries generic arguments in compiler-defined order.

Each argument is conceptually a tagged choice among type, region, and const.

For Window<'x, String, 8>, the semantic application includes all three.

Associated items may combine parent arguments with their own arguments.

Code that slices, rebases, or substitutes argument lists is especially vulnerable to off-by-owner errors.

When an associated type unexpectedly contains the wrong parameter, inspect argument construction before investigating the solver.

Aliases are not always immediately reduced#

<T as Iterator>::Item is a projection: selecting an associated type through a trait.

impl Trait introduces an opaque alias with a hidden type determined under specific rules.

Type aliases and weak aliases have their own normalization behavior.

An internal alias term is not automatically equal to its final revealed form in every context.

Normalization asks the type system to replace an alias with a justified result where permitted.

Keeping aliases explicit supports lazy work, cycle handling, and better reasoning about generic code.

The tradeoff is that callers must know whether they require normalized, deeply normalized, or still-symbolic terms.

A classic bug location is a consumer assuming normalization already happened.

5. TyCtxt, interning, and arenas#

The compiler context#

TyCtxt<'tcx> is the central, copyable handle through which compiler code accesses interned data, definitions, queries, and global compilation state.

It is pronounced “type context.”

It does not mean that every operation is globally mutable.

Rustc's query system and disciplined APIs control computation and reuse.

An interner stores one shared representative for structurally equal data.

If many expressions use Option<String>, their type handles can refer to shared type data instead of duplicating the whole tree.

An arena allocates many values that are freed together near the end of a compilation context.

This makes allocation and lifetime management practical for compiler workloads.

Why compact handles help#

Type equality checks are extremely frequent.

Shared, pointer-like handles can make common operations cheap and keep enclosing structures smaller than recursively embedding full type trees.

Interning also makes immutable semantic data naturally shareable.

However, this textbook deliberately makes no promise about Ty<'tcx> size, bit layout, address stability beyond API guarantees, or niche encoding.

Those are implementation details, may differ by target or revision, and must not become correctness assumptions in compiler code.

Say “compact pointer-like interned handle” as motivation, not “it is exactly one pointer forever.”

Invariants and tradeoffs#

Interned data must be immutable in the semantic sense expected by its users.

Keys must include every property relevant to equality.

The context lifetime must outlive handles derived from it.

Interning costs a lookup and retains data for the arena's lifetime, so it is not automatically ideal for every temporary object.

Inference variables, mutable tables, and short-lived probe state belong elsewhere.

If identical-looking types fail equality, check for unnormalized aliases or distinct definition identities before suspecting the interner.

If memory grows, distinguish expected arena retention from accidental creation of many unique terms.

6. ParamEnv: assumptions travel with questions#

The local logical environment#

A ParamEnv is the parameter environment: assumptions and mode information in force while checking a generic item.

For a function with T: Read and T::Error: Send, those predicates become available premises.

The same type-level question can have different answers under different environments.

T: Read is provable inside that function but not in an unrelated function with unconstrained T.

Therefore caches and query keys must not silently omit the relevant environment.

fn consume<T>(value: T)
where
	T: IntoIterator,
	T::Item: Send,
{
	// Both bounds are assumptions available while checking this body.
}

Environment is not evidence of everything#

Rust does not generally infer arbitrary bounds from operations in a function body and add them to the public contract.

The signature defines what callers promise.

The body must be checked using those promises.

This supports separate compilation and comprehensible APIs.

Some facts are elaborated from assumptions, such as supertraits.

If Sub: Super, a T: Sub premise can support a T: Super goal.

That is rule-driven elaboration, not unrestricted theorem discovery.

Reveal and normalization context#

Parameter environments also participate in decisions about revealing opaque types and normalization behavior.

Details are version-sensitive and represented through rustc APIs rather than prose slogans.

The engineering lesson is stable: do not solve a semantic query with a bare type when its answer depends on surrounding assumptions or reveal mode.

An apparent cache corruption that crosses item boundaries often indicates an incomplete key.

An unexpected “trait not implemented” inside a generic body may indicate that a predicate was not lowered, elaborated, or placed into the environment expected by the query.

7. Binders, bound variables, and de Bruijn indices#

Names that do not matter#

Compare for<'a> fn(&'a u8) and for<'b> fn(&'b u8).

They mean the same thing because renaming a locally bound lifetime changes nothing.

The for<...> introduces variables whose scope is the following type or predicate.

A Binder<T> conceptually packages a value T with variables bound at its outer edge.

Bound variables can include regions and, in internal formulations, other supported variable categories.

They are different from free inference variables.

An inference variable is an unknown to solve.

A bound variable stands for any choice allowed by its quantifier.

Counting binders instead of trusting names#

Rustc uses de Bruijn indices to identify which enclosing binder owns a bound variable.

A de Bruijn index is a distance measured through binders.

The innermost binder is reached with the smallest depth; crossing another binder increments the distance to an outer variable.

for<'a> fn(for<'b> fn(&'a u8, &'b u8))
             inner use: 'b belongs to nearest binder
             outer use: 'a crosses the inner binder

This avoids dependence on user-chosen names and makes alpha-equivalent forms equal.

The cost is bookkeeping complexity whenever code enters, exits, shifts, substitutes, or instantiates binders.

An escaping bound variable is one referenced outside the binder depth where it is valid.

Compiler helpers detect and transform such terms.

A safe mental procedure#

Draw every binder as a box.

At a variable occurrence, count outward to its owner.

When moving a term beneath a new box, shift references that point outside the moved term.

When instantiating, replace only variables owned by the binder being opened.

Never strip a Binder merely because current test data has no bound variables.

Nested HRTBs will expose the mistake.

Mis-shifting usually appears as a late mismatch, leak-check failure, or ICE far from the transformation.

8. Inference variables: disciplined unknowns#

Several unknown domains#

Rustc uses inference variables for unknown types, integer-literal types, floating-literal types, regions, and consts.

These domains have different legal solutions.

A general type variable might become Vec<u8>.

An integer variable is restricted to integer types and carries literal-specific behavior.

A floating variable is restricted to floating types.

A region variable participates in outlives constraints rather than becoming an arbitrary type.

A const variable must become a const term of the required const type.

Keeping domains explicit preserves invariants and allows better fallback messages.

let count = 0;          // initially an integer inference variable
let ratio = 1.0;        // initially a float inference variable
let values = Vec::new(); // element type remains unknown until context helps

Variables are local state#

Inference variables belong to an inference context.

They are not globally meaningful identifiers.

Variable number 7 in one body has no relationship to variable number 7 in another.

This is why raw inference terms must not leak into cross-context caches.

The context stores equivalence relationships, known assignments, and region or const constraints.

Reading a variable may “shallow resolve” one assignment link or “fully resolve” through nested structure, depending on the API.

Unknown is not error#

An unresolved variable during checking often means “not enough information yet,” not invalid code.

The compiler delays some decisions and revisits obligations after more equations arrive.

Conversely, accepting every unknown as success would be unsound.

At designated completion points, rustc performs fallback, reports ambiguity, or demands full resolution.

The invariant is temporal: uncertainty is permitted only while an owner still has a responsible later step.

When debugging, identify that owner and completion point.

9. Unification and occurs-like concerns#

Making two terms agree#

Unification finds assignments that make two terms equal.

Unifying unknown ?T with u32 assigns ?T = u32.

Unifying Vec<?T> with Vec<String> recursively yields ?T = String.

Unifying Vec<u8> with Option<u8> fails because their outer constructors differ.

Unification may also emit obligations or constraints rather than finishing every semantic question immediately.

For example, aliases can require normalization, and regions require relation constraints.

Infinite-type danger#

A naïve unifier must reject assigning ?T = Vec<?T>.

Otherwise expanding the assignment produces Vec<Vec<Vec<...>>> forever.

The traditional prevention is an occurs check: verify that a variable does not occur inside its proposed value.

Rustc's real inference representations and cycles are more nuanced than the textbook algorithm.

Use “occurs-like concerns” when reasoning broadly rather than asserting every rustc relation runs one simple recursive occurs-check function.

Aliases, binders, canonical variables, and recursive trait goals introduce distinct cycle questions.

Direction and diagnostics#

Equality is symmetric mathematically, but implementation direction can affect which variable is assigned, which span is primary, and how an error is phrased.

Expected-versus-found ordering should reflect user intent.

A mismatch should preserve the obligation cause that explains why equality was requested.

If rustc reports reversed types, the solver may be correct while diagnostic orientation is wrong.

If an infinite recursion or stack overflow occurs, determine whether the cycle is in type structure, alias normalization, trait goals, or diagnostic formatting.

Those require different guards.

10. Snapshots, probes, and rollback#

Trying without committing#

Candidate search often needs speculation.

A method candidate may unify with the receiver but fail a where-clause.

Another candidate may succeed.

Changes made while testing the first candidate must not contaminate the second.

An inference snapshot records a point to which mutable inference state can roll back.

A probe runs an operation speculatively, then rolls its changes back unless the API explicitly commits an accepted result.

state S0
  snapshot
  try candidate A -> assign ?T = u8 -> nested goal fails
  rollback to S0
  try candidate B -> assign ?T = String -> succeeds
  commit selected consequences

What must roll back#

Every speculative side effect matters: variable assignments, generated obligations, region constraints, const relations, and diagnostic artifacts.

Rustc APIs are designed to manage these sets, but additions to inference state can create rollback bugs if not integrated correctly.

External mutable collections captured by a probe deserve scrutiny.

The invariant is observational: after a failed probe, later computation should behave as though the probe never happened, except for intentionally retained debugging telemetry.

Alternatives and costs#

Purely immutable inference state would make rollback conceptually simple but could allocate heavily and complicate hot paths.

Cloning all tables for every candidate is straightforward but expensive.

Undo logs and snapshots are efficient but require disciplined mutation.

Rustc chooses specialized mechanisms according to subsystem needs.

If candidate ordering changes compilation results, suspect leaked probe state, premature commitment, or an ambiguity policy bug.

Candidate order may affect diagnostics and performance, but it must not silently alter sound language meaning where order is not specified.

11. Expected types and fallback#

Information flows inward and outward#

Checking let x: Vec<u8> = collect_expression; provides an expected type to the initializer.

That expectation can guide method obligations and closure results before they would otherwise be known.

Checking also synthesizes an actual type from an expression.

Bidirectional checking is the practical combination: expectations flow inward, synthesized results flow outward, and relations reconcile them.

An expected type is guidance plus a requirement at the appropriate coercion site.

It must not justify assuming an impossible candidate.

let bytes: Vec<u8> = (0..4).collect();
// Vec<u8> helps choose the FromIterator target and integer type.

Fallback is a final policy#

Some unconstrained literal variables receive defaults, commonly integer and float defaults according to language rules.

Fallback is not the main inference algorithm.

It is a late policy for particular unresolved domains.

General unknown element types do not all become a universal default.

Vec::new() with no constraining use may require annotation.

Fallback can unlock obligations, so ordering matters: apply the permitted defaults, reprocess affected work, and then diagnose remaining ambiguity.

Diagnostic reasoning#

An annotation suggestion should be placed where it supplies useful information with minimal burden.

If inference fails only after an unrelated edit, inspect whether that edit removed an expected type or changed a coercion site.

If rustc picks i32 too early and rejects evidence for another integer type, suspect premature fallback.

If an error says “type annotations needed” although a clear annotation exists, trace whether expectation propagation stopped at a closure, ?, method call, or alias boundary.

12. Subtyping and variance from first principles#

Safe replacement#

Type A is a subtype of type B when an A value can safely stand where B is expected under Rust's rules.

Rust subtyping is deliberately limited.

Lifetimes provide the most visible examples: a reference valid for longer can often be used where a shorter validity is required.

It is not class inheritance.

How constructors transmit relations#

Variance describes how a type constructor transforms subtyping relationships among its arguments.

Covariant means the direction is preserved.

Contravariant means the direction reverses.

Invariant means neither substitution is generally accepted.

Bivariant means the argument imposes no relation in that position, a specialized category not to casually assume.

Shared references are generally covariant in their lifetime and referent where safe.

Mutable references require stricter treatment of their referent because mutation could store a value valid for too short a time.

Function inputs have contravariant behavior; outputs have covariant behavior.

fn shorten<'long, 'short>(x: &'long str) -> &'short str
where
	'long: 'short,
{
	x
}

'long: 'short reads “'long outlives 'short.”

Compiler boundary#

Variance is computed for definitions and used by relation code.

It affects which region or type constraints are generated while relating applications.

It does not mean the compiler physically converts the referenced bytes.

A variance bug can be a soundness bug, so fixes demand minimized tests in positive and negative directions.

When a lifetime error seems backwards, draw producer and consumer positions through every constructor before changing relation code.

13. Coercions, adjustments, and unsizing#

Directed conversions at special sites#

A coercion converts an expression toward an expected type at language-defined coercion sites.

Examples include reborrowing references, turning a noncapturing closure into a function pointer where allowed, or converting &[T; N] to &[T].

Coercion is directed: source toward target.

It is not the same as symmetric unification.

An adjustment is rustc's recorded operation describing how the expression's computed type is adapted for later lowering.

Type checking records adjustments; MIR/THIR construction must honor them.

Losing an adjustment can make type checking succeed but later phases miscompile or ICE.

Autoderef and autoref#

Method lookup may repeatedly dereference a receiver to find methods.

This is autoderef.

It may then borrow the resulting place as &self or &mut self; this is autoref.

The compiler records the chosen receiver adjustments.

User-defined Deref can introduce trait obligations and potential cycles.

Candidate search must cap or detect runaway dereference chains and produce useful overflow diagnostics.

Unsizing#

Unsizing changes a sized representation behind an appropriate pointer into a dynamically sized target, such as array to slice or concrete type to dyn Trait when requirements hold.

Smart-pointer unsizing involves language traits and strict structural rules.

It is not arbitrary conversion between containers.

The conceptual invariant is representation-safe adaptation authorized by compiler rules and relevant trait obligations.

If a call type-checks but receives the wrong receiver in MIR, inspect adjustment recording and application.

If method lookup cannot see a method through a wrapper, inspect autoderef obligations and probe rollback before adding ad hoc candidates.

14. Inference is not checking#

Two related jobs#

Inference determines unknown values from constraints.

Checking verifies that a proposed or inferred relationship is legal.

They frequently run together, but neither contains the other.

A fully annotated program still needs checking for invalid calls, unsatisfied bounds, privacy-related validity, and lifetime relations.

A compiler can infer ?T = String and then reject String + String because the required operation is unavailable.

Conversely, checking a generic operation may leave a variable unknown until later context arrives.

Why architecture keeps boundaries#

Relation code handles equality and subtyping structure.

Inference tables store unknown assignments.

Trait engines track obligations.

Normalization reduces aliases under rules.

Body checking determines expression-specific expectations and causes.

Clear boundaries allow each part to maintain its invariants.

They also make uncertainty explicit: “relation succeeded but emitted obligations” differs from “all obligations are fulfilled.”

A debugging checklist#

First ask whether the wrong internal type was created.

Then ask whether the right relation was requested.

Then ask whether relation consequences were registered.

Then ask whether fulfillment ran to completion.

Finally ask whether diagnostics selected the correct cause and span.

A patch that makes inference eagerly choose an answer can hide missing obligation registration.

That may improve one error while introducing acceptance bugs.

Prefer restoring the phase contract over forcing the observed output.

15. Checking an expression body#

The body-local context#

rustc_hir_typeck checks function, closure, and constant-like bodies using a body-local function context backed by an inference context.

It walks expressions and patterns, supplies expected types, synthesizes actual types, relates them, registers obligations, and records results.

The exact type and function names evolve; consult matching nightly API docs and source.

fn total(xs: &[u32]) -> u32 {
	xs.iter().copied().sum()
}

The body checker resolves xs, obtains its reference type, searches iter, determines iterator items, searches copied, and constrains sum using the declared return type.

Trait goals are interleaved with inference because each can reveal facts needed by the other.

Results and tables#

Body type-check results map HIR expressions and patterns to semantic types.

They also contain adjustments, method or associated-item resolutions, field indices, user type information, and other facts needed downstream.

Think of these results as a checked semantic annotation of the HIR body.

Downstream phases should not redo source-level method lookup with different assumptions.

Completion#

After the main walk, rustc performs deferred checks and fulfillment, applies fallback where permitted, resolves variables as required, and reports errors.

The ordering is carefully engineered and version-sensitive.

An ICE about an unresolved type in MIR often means body checking failed to force or poison a result after an earlier error.

A wrong method in downstream code may mean the type-dependent definition table recorded a candidate before its obligations were confirmed.

Trace from the HIR node through typeck results rather than beginning in code generation.

16. Method lookup and candidate search#

Building receiver possibilities#

For value.method(args), rustc does more than search the inherent methods textually named method.

It constructs receiver candidates through autoderef and borrowing possibilities.

It considers inherent impls and trait methods visible or otherwise relevant under Rust's lookup rules.

Each candidate has a receiver type, generic arguments, predicates, and a definition identity.

Search and confirmation are distinct.

A candidate may look structurally applicable but fail after substitutions and obligations are checked.

Ambiguity is information#

Two applicable traits can produce a genuine ambiguity requiring fully qualified syntax.

Unknown inference variables can also prevent choosing yet.

The compiler should delay where later information may resolve uncertainty, but it must eventually diagnose unresolved ambiguity.

trait A { fn label(&self) -> &'static str; }
trait B { fn label(&self) -> &'static str; }

// If both traits are implemented and in scope:
// A::label(&value) disambiguates explicitly.

Candidate diagnostics#

Good errors explain nearby methods, missing trait imports, receiver mismatches, and unsatisfied bounds without claiming a candidate was selected when it was only considered.

Candidate probing must roll back inference effects.

Autoderef steps must retain causes for overflows and failed obligations.

If adding an unrelated trait import changes a selected inherent method, inspect candidate ordering and ambiguity handling.

If the compiler suggests importing a trait whose where-clause cannot hold, inspect the diagnostic candidate filter, not necessarily the core solver.

17. Well-formedness and implied bounds#

Meaningful before usable#

A term is well formed when its components satisfy the prerequisites required for that term to be meaningful under the current environment.

For a reference &'a T, relevant rules connect T and 'a.

For a trait application, declaration bounds may create requirements on arguments.

Well-formedness checks occur for item signatures and at use sites through generated predicates.

They are not equivalent to proving every operation in a body.

Implied bounds#

Some bounds are implied by the well-formedness of types appearing in a function signature.

The important common family concerns lifetimes.

If a function accepts &'a T, well-formedness supports an appropriate T: 'a fact inside the function.

Rust does not imply all trait bounds merely because a nested generic type's definition mentions them in every imaginable way.

Exact implied-bound rules are language details, so test examples rather than extending intuition.

fn use_ref<'a, T>(x: &'a T) {
	// The reference's well-formedness carries relevant outlives information.
	let _ = x;
}

Outlives obligations#

T: 'a means values contained by T do not contain borrowed data invalid before 'a, according to Rust's outlives rules.

'a: 'b means 'a lasts at least as long as 'b.

Type checking can generate these obligations; region checking and borrow checking process related constraints at their proper boundaries.

If a lifetime error appears only after normalization, inspect whether associated-type outlives obligations were emitted and preserved.

If well-formedness reports at an entire signature, improve HIR WellFormedLoc-style source mapping rather than weakening the rule.

18. Associated types, projection, and normalization#

Naming an output of an implementation#

A trait can define an associated type.

trait Stream {
	type Item;
	fn next(&mut self) -> Option<Self::Item>;
}

For a particular S: Stream, <S as Stream>::Item is a projection.

Normalization attempts to determine the type selected by that projection.

If S is concrete and one impl applies, the answer may be concrete.

If S is a generic parameter, an environment equality bound may answer it.

If information is missing, normalization can remain ambiguous rather than fabricate a type.

Obligations during normalization#

Applying an impl's associated type definition requires proving the impl applies.

Its where-clauses become nested goals.

Normalization can therefore invoke trait solving, which can encounter more projections.

This mutual recursion is a major source of cycles and overflow.

Lazy normalization keeps aliases symbolic until a relation actually needs their shape.

Eager normalization simplifies consumers but can perform unnecessary work and expose cycles sooner.

Rustc contains APIs with different normalization contracts; choose deliberately.

Projection equality#

A goal may say <T as Iterator>::Item = u8.

This is not merely a string comparison.

The solver must justify T: Iterator, identify applicable evidence or premises, and relate the associated result to u8.

If an error displays an unreduced projection, that can reflect honest ambiguity, a reveal restriction, solver overflow, or missed normalization.

Check which case before “prettifying” the diagnostic, because forcing normalization can change semantics or performance.

19. Traits, implementations, and supertraits#

Interfaces plus logical rules#

A trait declares associated items and constraints.

An impl supplies those items for a self type and trait arguments under optional where-clauses.

trait Encode {
	fn encode(&self) -> Vec<u8>;
}

impl<T: Encode> Encode for Vec<T> {
	fn encode(&self) -> Vec<u8> {
		self.iter().flat_map(Encode::encode).collect()
	}
}

Logically, the impl resembles a rule: if T: Encode, then Vec<T>: Encode.

Operationally, rustc must instantiate impl parameters, relate the impl header to a goal, and create nested obligations from its predicates.

Supertraits#

trait Ordered: Eq says implementing Ordered also requires Eq.

A goal for a supertrait may be proven by elaborating an available subtrait premise.

Supertraits also influence dyn compatibility and object vtable shape conceptually.

They are not inheritance of fields or implementation bodies.

Defaults and specialization pressure#

Traits may provide default methods.

Impls can override them.

Unstable specialization mechanisms permit controlled overlap and more-specific behavior, creating difficult ordering and soundness questions.

Even without specialization, trait defaults require substitutions using the selected impl's arguments.

When the wrong associated item instance appears, inspect both trait-item-to-impl-item mapping and substitutions.

When a supertrait method is unavailable, check predicate elaboration and method lookup before blaming coherence.

20. Dyn compatibility and vtables#

Erasing a concrete type#

dyn Trait is a dynamically dispatched trait object type.

The concrete implementing type is hidden behind a pointer-like owner or reference.

Operations use runtime metadata to reach appropriate method implementations.

The modern rustc term is dyn compatibility; older material often says “object safety.”

A trait must satisfy language restrictions to be used as dyn Trait.

Restrictions ensure callable methods can be represented and invoked without knowledge that was erased.

Methods with problematic uses of Self, unsupported generic dispatch requirements, or other forbidden forms can make a trait non-dyn-compatible, subject to exact current rules.

Conceptual vtable boundary#

A vtable is runtime metadata containing function pointers and other information needed for dynamic operations.

This chapter stops at the conceptual boundary.

It does not promise vtable field order, binary ABI, pointer count, or layout stability.

Compiler internals and target ABIs may evolve.

Type checking proves the coercion to a trait object is legal and records adjustments.

Later lowering and code generation construct and use metadata.

Diagnostic separation#

“Trait is not dyn compatible” differs from “type does not implement trait.”

The former rejects forming or using the erased interface.

The latter is a trait goal failure for a type.

Upcasting a dyn trait to a supertrait introduces another set of rules and metadata concerns.

If an error lists irrelevant methods, inspect dyn-compatibility violation collection and spans.

If type checking accepts an object coercion but codegen fails, verify that the expected adjustment and principal trait information reached downstream phases.

21. Obligations, goals, and candidates#

Turning language questions into solver input#

An old-style obligation and a next-solver goal both package a predicate to establish with context such as a parameter environment and diagnostic cause.

Terminology and concrete structs differ across architectures.

Typical predicates include trait implementation, projection equality, well-formedness, subtype or equality relations, and const evaluatability-related statements.

A candidate is one possible route to proving a goal.

Candidates can arise from impls, environment assumptions, built-in rules, aliases, object bounds, and other language-defined sources.

Nested goals#

For Vec<T>: Clone, an impl candidate may require T: Clone.

That requirement is a nested goal.

A candidate is not truly applicable merely because its header unifies.

Its nested requirements must reach an acceptable result under the solver's certainty rules.

goal: Vec<?T>: Clone
candidate: impl<U: Clone> Clone for Vec<U>
unify: ?T = U
nested goal: U: Clone

Three broad outcomes#

A goal may be proved, disproved with an error, or remain ambiguous.

Ambiguous means available information does not justify a unique or definite conclusion now.

Unknown inference variables commonly cause ambiguity during type checking.

Overflow means evaluation exceeded a recursion or complexity limit; it is not automatically logical falsehood.

Cycles require policy: some are errors, some are ambiguous, and restricted coinductive reasoning may accept certain cycles.

Never turn ambiguity into success merely to make a test compile.

The surrounding query decides whether ambiguity may be deferred or must be reported.

22. Fulfillment and delayed work#

A queue of promises#

Body checking generates obligations before all variables are known.

A fulfillment context stores pending obligations and repeatedly selects those that can make progress.

Successful selection can add nested obligations.

Ambiguous obligations remain pending while other checks constrain variables.

At a completion point, remaining ambiguity becomes an inference or trait error unless a specific API contract permits otherwise.

This fixed-point behavior is why trait errors can surface after the expression that originated them.

The obligation cause preserves the route back to useful source.

Selection is not fulfillment#

Selection examines one obligation and proposes evidence or a result.

Fulfillment manages a set until all transitive requirements are handled.

Choosing an impl whose where-clause fails is not fulfillment.

Likewise, proving a top-level predicate while dropping a nested normalization obligation violates the central invariant.

Error prioritization#

One root mismatch can generate many downstream failures.

Diagnostics try to suppress cascades and point to actionable causes.

Error types and tainted inference state help compilation continue safely enough to report more independent errors.

This recovery must not be confused with accepting the program.

If a nested obligation disappears, inspect registration and drain paths.

If the same error repeats, inspect deduplication keys and whether normalization creates syntactically fresh but equivalent obligations.

If an ambiguity is reported too early, inspect whether fulfillment had a later retry after expectation propagation and fallback.

23. Canonicalization: portable questions#

The cache contamination problem#

Suppose one inference context asks whether Vec<?T>: Clone.

Its ?T has a local identity and mutable assignment state.

Caching that raw goal globally would be meaningless in another context, where the same variable number denotes something else.

Worse, a cached answer could retain assumptions from a speculative probe.

Canonicalization replaces free inference variables and suitable placeholders with numbered canonical variables plus metadata describing their categories and universes.

The resulting canonical goal is independent of local variable identities.

local goal A: Vec<?type_17>: Clone
local goal B: Vec<?type_4>: Clone
canonical form: for canonical ?0, Vec<?0>: Clone

Canonical responses#

The solver returns a canonical response describing certainty, substitutions for canonical variables, and constraints that the caller must instantiate back into its local inference context.

Applying a response is a checked operation.

The caller maps canonical variables to original values and registers returned region, type, or const constraints.

A response is not permission to copy internal variable IDs across contexts.

Cache invariants#

Canonical keys must include all semantically relevant input, including environment and solver mode.

Responses must not mention unbound local inference state.

Probe-local effects must be reflected only through sanctioned canonical constraints.

Canonicalization can reduce cache fragmentation but costs conversion and may intentionally erase distinctions irrelevant to solving.

If a bug appears only with incremental or query caching, compare canonical keys and response instantiation.

If disabling cache changes correctness, treat it as a serious invariant violation, not a performance quirk.

24. The old/current-style solver architecture#

Version-sensitive status#

At the time of writing, around the rustc 1.97.1 timeframe, official documentation remains deliberately cautious.

The rustc-dev-guide chapter titled “Trait resolution (old-style)” says it describes how the solver currently works and points to a new design.

The in-tree next-generation solver documentation calls that solver work in progress.

The next-generation solver has already been adopted in some domains, including coherence on stable toolchains, while other call sites can still use older paths or configurable modes.

Migration changes quickly by nightly and query.

Do not summarize the situation as “rustc has switched completely” or “the new solver is unused.”

Check the exact revision, flags, and call site.

Selection and fulfillment#

The old architecture is organized around selection and fulfillment.

Selection assembles candidates for a trait obligation, winnows possibilities, confirms a candidate by relating arguments, and emits nested obligations.

Environment bounds can prove generic obligations without choosing a concrete impl.

Fulfillment tracks pending obligations and retries ambiguity as inference advances.

Projection normalization has old-solver-specific caches and machinery.

Codegen can perform selection again when concrete monomorphized arguments are available.

Evaluation versus commitment#

Candidate evaluation may answer whether a route could apply without committing all inference changes.

Selection uses probes to avoid pollution.

Ambiguity conservatively preserves possibilities.

Overflow handling limits recursive exploration and reports or propagates uncertainty according to context.

For current details, start at rustc_trait_selection::traits, its select, fulfill, project, and normalize areas, then follow the call from the relevant type checker.

Names and module visibility in nightly rustdoc can differ from source organization.

25. The in-tree next-generation solver#

A goal-oriented design#

The next-generation solver lives in rustc, principally exposed through rustc_next_trait_solver and integration layers.

It recasts type-system operations as evaluation of canonical goals.

A goal combines a parameter environment with a predicate.

Evaluation assembles all relevant candidate sources, evaluates each under controlled inference state, and combines their results according to solver rules.

This architecture aims for clearer logical behavior, robust caching, inspectable proof trees, and maintainability across normalization and trait solving.

It is not merely the old selector with renamed functions.

Candidate assembly and nested recursion#

Candidate assembly considers impls, environment clauses, built-in rules, alias-related rules, and other predicate-specific candidates.

Each candidate can introduce nested goals.

The evaluator recursively solves those goals, canonicalizing at query boundaries as required.

Candidate results include a certainty rather than only Boolean success.

Conceptually, certainty distinguishes a definite answer from a maybe answer caused by ambiguity or overflow.

Exact enums and combination behavior are API details to inspect in 1.97.1 source.

Canonical constraints and caches#

Successful evaluation returns canonical constraints needed by the caller.

The cache stores context-independent evaluations.

Cycle handling recognizes a repeated canonical goal on the evaluation stack and applies the policy appropriate to that goal and cycle kind.

Some cycles support coinductive reasoning; blindly treating every recursive goal as true would be unsound.

The next solver's design supports proof-tree inspection for diagnostics without requiring every successful compilation to retain a heavyweight permanent tree.

Official docs may still label areas WIP.

Treat implementation comments and tests at the pinned revision as authoritative for fine behavior.

26. Cycles, certainty, and overflow#

Why recursion is normal#

Recursive traits and types naturally create repeated goals.

An impl for a wrapper may require the same auto trait for its field, whose type refers back through another wrapper.

Associated type normalization can revisit an earlier projection.

The presence of a cycle does not alone decide truth.

Inductive reasoning requires finite evidence built from base facts.

Coinductive reasoning can accept certain self-supporting structures as a greatest fixed point, traditionally relevant to auto-trait-style reasoning.

Rust's actual permitted coinduction is restricted and evolving.

Cache states and stacks#

An evaluator can record goals currently in progress.

Encountering one again reveals a cycle.

The response may depend on whether the cycle is inductive, coinductive, mixed, or encountered while normalizing an alias.

Provisional cache entries help avoid infinite recursion, but their validation must account for dependencies.

A cached “maybe” may be safer than a false proof.

Overflow is resource uncertainty#

Deep acyclic goals and nonterminating recursive patterns can exceed recursion limits.

The solver returns or reports overflow according to its API.

Overflow should not be silently equated with “trait not implemented.”

Diagnostics should show a useful obligation chain and mention raising a recursion limit only when appropriate.

If changing candidate order removes overflow, investigate repeated equivalent goals, canonicalization quality, and cacheability.

If a cycle becomes accepted after a refactor, verify the coinductive classification and add a negative soundness-oriented test.

27. Chalk: influential history, not today's engine#

Logic-programming philosophy#

Chalk explored expressing Rust's trait system as logic clauses.

An impl like impl<T: Clone> Clone for Vec<T> lowers conceptually to “for every T, if T: Clone, then Vec<T>: Clone.”

A query asks whether a goal follows from the program clauses and environment assumptions.

This separates declarative language rules from a general solving strategy.

Chalk's book explains goals, clauses, universes, associated types, and lowering in a Prolog-inspired vocabulary.

SLG and recursive ideas#

Chalk experimented with an SLG-style solver using tabling and with recursive solving approaches.

Tabling stores intermediate answers to repeated logical subgoals, helping with cycles and duplicate work.

SLG refers to a family of logic-programming evaluation techniques; knowing the expansion of the acronym is less useful here than understanding delayed, tabled answers.

Recursive solvers more directly follow candidate rules through nested goals and can be simpler to integrate with existing inference and diagnostics.

Rustc learned from both the declarative formulation and practical limitations.

Sunset status#

The Chalk repository README marks the project as sunset.

Current rustc does not simply call Chalk as an embedded library for ordinary trait solving.

The next-generation solver is an in-tree rustc implementation informed by Chalk's work, rustc's old solver, and subsequent design.

Use Chalk documentation for historical concepts, not as a precise specification of rustc 1.97.1 behavior.

This distinction prevents stale claims about integration, clause sets, or solver algorithms.

28. Higher-ranked bounds, placeholders, and universes#

“For every lifetime”#

A higher-ranked trait bound, or HRTB, can state that an implementation works for every lifetime chosen by a caller.

fn call_with_any<F>(f: F)
where
	F: for<'a> Fn(&'a str),
{
	let local = String::from("hello");
	f(&local);
}

for<'a> does not ask inference to find one convenient lifetime.

It requires validity for all suitable 'a.

To test such a claim, rustc replaces the bound variable with a fresh placeholder.

A placeholder is a rigid representative that cannot be unified with arbitrary older local choices.

Universes as visibility levels#

A universe records which placeholders existed when an inference variable was created.

An older inference variable must not be assigned a value containing a newer placeholder it could not have known about.

Imagine nested locked rooms: a variable created outside cannot smuggle out a name introduced only inside.

This prevents proving “there exists one value” when the requirement was “for every value.”

Leak checking#

A leak check ensures placeholders introduced for a higher-ranked test do not escape into inference state or results that outlive their binder.

Canonicalization records universe information so cached queries preserve this restriction.

A missing leak check can be a soundness bug.

An overly strict check rejects valid higher-ranked code.

When debugging, minimize nested binders, label creation universes, and inspect response instantiation.

Do not “fix” the failure by treating placeholders as ordinary inference variables.

29. Coherence, orphan rules, and overlap#

One compatible global meaning#

Coherence ensures trait implementations interact predictably across crates.

For ordinary dispatch, rustc must not face two equally applicable impls whose coexistence the language forbids.

The orphan rules restrict which crate may implement a trait for a type.

Informally, an impl needs an appropriate local trait or local type anchor, with detailed rules for uncovered parameters and fundamental wrappers.

Consult the reference and compiler tests for exact legality.

Semver motivation#

Without orphan restrictions, crate A and crate B could independently add the same impl for foreign trait and foreign type.

A downstream crate using both would break even though each dependency made an apparently local addition.

Coherence turns many such ecosystem conflicts into decisions owned by a crate that controls a relevant trait or type.

This supports semantic versioning, though it cannot eliminate all ecosystem evolution hazards.

Overlap checking#

Two impls overlap if there exists a substitution under which both could apply, considering relevant assumptions and conservative future possibilities.

Overlap is an existential question: can a common case exist?

Unknown or downstream types make negative reasoning difficult.

The solver must not assume “no impl today” when another crate could legally add one tomorrow.

If overlap results differ between solver modes, reduce the headers and inspect intercrate ambiguity, orphan knowledge, and normalization.

Coherence bugs can create either needless rejection or dispatch unsoundness, so changes require cross-crate tests.

30. Specialization, negative reasoning, and fundamental types#

Controlled overlap#

Specialization aims to permit a general impl and a more specific impl while defining which wins.

It remains an advanced, largely unstable area with difficult soundness conditions.

The compiler builds a specialization relationship rather than applying source order.

Associated items can be inherited or overridden along that relationship.

“More specific” must be justified under trait rules, not guessed from textual complexity.

Knowing absence#

Negative reasoning concludes that an impl does not exist.

In an open crate ecosystem, absence from the current crate graph is often not proof of permanent impossibility.

Orphan rules can sometimes establish that no downstream crate may add a conflicting impl.

Explicit negative impls provide stronger information in supported contexts.

The distinction matters for overlap, auto traits, and semver.

Closed-world reasoning is tempting and frequently wrong.

Fundamental types#

Certain compiler-recognized fundamental type constructors receive special coherence treatment.

References and selected library types can allow locality to pass through in carefully specified ways.

This helps crates implement foreign traits for references or wrappers involving their local types without opening broad conflicts.

“Fundamental” is a language/coherence designation, not a claim that the type is philosophically basic or layout-transparent.

Do not infer the rule from examples; use current reference and rustc_hir_analysis::coherence implementation/tests.

If a proposed coherence fix treats every smart pointer as fundamental, it is almost certainly too broad.

31. Opaque types, RPITIT, and GATs#

Hidden but fixed#

impl Trait in return position describes an opaque type.

Callers know the listed bounds but not the concrete hidden type.

Within the defining scope, rustc infers and constrains a hidden type according to opaque-type rules.

“Opaque” does not mean a different concrete type may be chosen at every return expression.

The defining uses must agree under the feature's identity and capture rules.

fn numbers() -> impl Iterator<Item = u8> {
	0u8..4
}

RPITIT#

Return-position impl Trait in traits, abbreviated RPITIT, gives trait methods opaque return types whose identities and associated representations require careful lowering.

Each impl must satisfy the trait method's promised bounds while preserving abstraction.

Normalization must respect whether the hidden type may be revealed in the current context.

Method lookup, associated item matching, and capture of generic parameters all interact.

GATs#

A generic associated type, or GAT, is an associated type with its own generic parameters.

trait Lending {
	type Item<'a>
	where
		Self: 'a;
	fn get<'a>(&'a self) -> Self::Item<'a>;
}

Normalizing T::Item<'x> must align parent trait arguments, GAT arguments, binders, and where-clauses.

RPITIT and GAT failures often expose substitution or binder-depth bugs rather than a simple missing impl.

When minimizing, vary captures and nested lifetimes independently.

Avoid forcing global reveal of opaque types to solve one local diagnostic.

32. Const generics and evaluatability#

Values inside types#

Const generics let compile-time values parameterize types.

[T; N] depends on type T and const N.

Const inference variables and generic const expressions participate in type relations, but equality of const expressions is not unrestricted symbolic algebra.

The compiler normalizes and evaluates consts only under defined rules and feature boundaries.

fn array_len<T, const N: usize>(xs: &[T; N]) -> usize {
	let _ = xs;
	N
}

The point is that N is a value and also part of the array type.

Evaluatability boundary#

A const appearing where a concrete value is required must be evaluatable under the current generic environment.

Proving that an expression is well typed is not always enough to prove it can be evaluated for every allowed substitution.

Const-evaluatability obligations bridge trait/type reasoning and compile-time evaluation.

They must not execute arbitrary unknown generic computations as though concrete.

Conservative engineering#

Equivalent-looking expressions may not normalize to syntactic equality under stable rules.

Feature-gated generic const expressions extend what can be represented and proven.

Never patch a const mismatch by comparing pretty-printed strings.

Track typed const terms, substitutions, and environments.

If a trait goal depends on a const comparison, ambiguity may be the honest result until substitution.

If an evaluatability error points at the wrong expression, trace obligation causes across type lowering and const evaluation rather than weakening the check.

33. Educational unifier in stable Rust#

Conceptual code#

The following standalone model demonstrates variables, structural unification, rollback by cloning, and an occurs check.

It is educational code, not rustc code and not a sound model of Rust's full type system.

use std::collections::BTreeMap;

#[derive(Clone, Debug, PartialEq, Eq)]
enum Ty {
	Var(u32),
	Named(&'static str),
	App(&'static str, Vec<Ty>),
}

#[derive(Clone, Debug, Default, PartialEq, Eq)]
struct Unifier {
	solutions: BTreeMap<u32, Ty>,
}

impl Unifier {
	fn resolve(&self, ty: &Ty) -> Ty {
		match ty {
			Ty::Var(v) => match self.solutions.get(v) {
				Some(next) => self.resolve(next),
				None => ty.clone(),
			},
			Ty::App(name, args) => Ty::App(
				name,
				args.iter().map(|arg| self.resolve(arg)).collect(),
			),
			Ty::Named(_) => ty.clone(),
		}
	}

	fn occurs(&self, needle: u32, ty: &Ty) -> bool {
		match self.resolve(ty) {
			Ty::Var(v) => v == needle,
			Ty::App(_, args) => args.iter().any(|arg| self.occurs(needle, arg)),
			Ty::Named(_) => false,
		}
	}

	fn bind(&mut self, var: u32, ty: Ty) -> Result<(), String> {
		let ty = self.resolve(&ty);
		if ty == Ty::Var(var) {
			return Ok(());
		}
		if self.occurs(var, &ty) {
			return Err(format!("infinite type for ?{var}: {ty:?}"));
		}
		self.solutions.insert(var, ty);
		Ok(())
	}

	fn unify(&mut self, left: &Ty, right: &Ty) -> Result<(), String> {
		let left = self.resolve(left);
		let right = self.resolve(right);
		match (left, right) {
			(Ty::Var(v), ty) | (ty, Ty::Var(v)) => self.bind(v, ty),
			(Ty::Named(a), Ty::Named(b)) if a == b => Ok(()),
			(Ty::App(a, xs), Ty::App(b, ys))
				if a == b && xs.len() == ys.len() =>
			{
				for (x, y) in xs.iter().zip(&ys) {
					self.unify(x, y)?;
				}
				Ok(())
			}
			(a, b) => Err(format!("cannot unify {a:?} with {b:?}")),
		}
	}

	fn probe<T>(
		&mut self,
		action: impl FnOnce(&mut Self) -> Result<T, String>,
	) -> Result<T, String> {
		let saved = self.clone();
		match action(self) {
			Ok(value) => Ok(value),
			Err(error) => {
				*self = saved;
				Err(error)
			}
		}
	}
}

Tests#

#[cfg(test)]
mod unifier_tests {
	use super::*;

	fn vec_of(ty: Ty) -> Ty {
		Ty::App("Vec", vec![ty])
	}

	#[test]
	fn infers_an_element_type() {
		let mut u = Unifier::default();
		u.unify(&vec_of(Ty::Var(0)), &vec_of(Ty::Named("u8")))
			.unwrap();
		assert_eq!(u.resolve(&Ty::Var(0)), Ty::Named("u8"));
	}

	#[test]
	fn rejects_an_infinite_type() {
		let mut u = Unifier::default();
		let error = u.unify(&Ty::Var(0), &vec_of(Ty::Var(0))).unwrap_err();
		assert!(error.contains("infinite type"));
	}

	#[test]
	fn failed_probe_rolls_back() {
		let mut u = Unifier::default();
		let result = u.probe(|trial| {
			trial.unify(&Ty::Var(0), &Ty::Named("u8"))?;
			trial.unify(&Ty::Named("u8"), &Ty::Named("str"))
		});
		assert!(result.is_err());
		assert_eq!(u.resolve(&Ty::Var(0)), Ty::Var(0));
	}
}

This model lacks regions, binders, aliases, subtyping, variance, consts, error recovery, efficient union-find, and rustc's snapshot machinery.

Its clone-based probe commits on success, unlike APIs that always roll back and separately apply a canonical response.

It demonstrates mechanics only and makes no soundness claim about Rust.

34. A miniature trait goal solver#

Conceptual code#

This extension treats impls as rules and recursively proves ground goals.

It uses the preceding Ty and Unifier definitions.

#[derive(Clone, Debug)]
struct Goal {
	trait_name: &'static str,
	self_ty: Ty,
}

#[derive(Clone, Debug)]
struct Rule {
	trait_name: &'static str,
	self_ty: Ty,
	conditions: Vec<Goal>,
}

#[derive(Clone, Copy, Debug, PartialEq, Eq)]
enum Answer {
	Unique,
	NoSolution,
	Ambiguous,
}

struct MiniSolver {
	rules: Vec<Rule>,
	depth_limit: usize,
}

impl MiniSolver {
	fn solve(&self, goal: &Goal) -> Answer {
		self.solve_at(goal, &Unifier::default(), 0).0
	}

	fn solve_at(&self, goal: &Goal, input: &Unifier, depth: usize) -> (Answer, Unifier) {
		if depth >= self.depth_limit {
			return (Answer::Ambiguous, input.clone());
		}

		let mut successes = Vec::new();
		for rule in &self.rules {
			if rule.trait_name != goal.trait_name {
				continue;
			}
			let mut trial = input.clone();
			if trial.unify(&rule.self_ty, &goal.self_ty).is_err() {
				continue;
			}

			let mut candidate = Answer::Unique;
			for condition in &rule.conditions {
				let instantiated = Goal {
					trait_name: condition.trait_name,
					self_ty: trial.resolve(&condition.self_ty),
				};
				let (answer, next) = self.solve_at(&instantiated, &trial, depth + 1);
				trial = next;
				match answer {
					Answer::NoSolution => {
						candidate = Answer::NoSolution;
						break;
					}
					Answer::Ambiguous => candidate = Answer::Ambiguous,
					Answer::Unique => {}
				}
			}
			if candidate != Answer::NoSolution {
				successes.push((candidate, trial));
			}
		}

		match successes.len() {
			0 => (Answer::NoSolution, input.clone()),
			1 => successes.pop().unwrap(),
			_ => (Answer::Ambiguous, input.clone()),
		}
	}
}

Tests and limitations#

#[cfg(test)]
mod solver_tests {
	use super::*;

	fn list(ty: Ty) -> Ty {
		Ty::App("List", vec![ty])
	}

	#[test]
	fn proves_a_nested_goal() {
		let solver = MiniSolver {
			rules: vec![
				Rule {
					trait_name: "Show",
					self_ty: Ty::Named("u8"),
					conditions: vec![],
				},
				Rule {
					trait_name: "Show",
					self_ty: list(Ty::Var(10)),
					conditions: vec![Goal {
						trait_name: "Show",
						self_ty: Ty::Var(10),
					}],
				},
			],
			depth_limit: 16,
		};

		assert_eq!(
			solver.solve(&Goal {
				trait_name: "Show",
				self_ty: list(Ty::Named("u8")),
			}),
			Answer::Unique,
		);
	}

	#[test]
	fn reports_missing_evidence() {
		let solver = MiniSolver { rules: vec![], depth_limit: 8 };
		assert_eq!(
			solver.solve(&Goal {
				trait_name: "Show",
				self_ty: Ty::Named("Secret"),
			}),
			Answer::NoSolution,
		);
	}
}

Rule variables are accidentally global by number; a real solver freshens each candidate.

The model has no parameter environment, canonicalization, negative impls, specialization, associated types, universes, cycles, coinduction, consts, regions, coherence, or diagnostic proof trees.

Its “multiple successes means ambiguous” rule is much simpler than Rust's candidate preference and overlap rules.

Depth overflow is labeled ambiguous only as a teaching convenience.

Consequently this code must never be used to decide Rust safety or compatibility.

35. A detailed type-check trace#

Program under examination#

fn render<T>(items: Vec<T>) -> String
where
	T: ToString,
{
	items.into_iter().map(|x| x.to_string()).collect()
}

Name resolution has already linked Vec, String, ToString, methods, and local names where possible.

HIR lowering records the generic parameter and bound.

Type lowering creates semantic Vec<T> and String terms and a parameter environment containing T: ToString.

Expression walk#

The path items synthesizes Vec<T>.

Method lookup for into_iter considers receiver candidates and finds an applicable IntoIterator route.

Confirmation relates the receiver and produces the iterator type plus obligations.

The body checker records method resolution and receiver adjustments.

For map, lookup learns the iterator's item type is T after normalization.

The closure receives expected input type T, so pattern x is assigned T.

Looking up to_string creates a goal equivalent to T: ToString.

The parameter environment proves it; no concrete impl for unknown T is selected in the generic body.

The method result is String, making the mapped iterator item String.

The declared function return supplies expected type String to collect().

That introduces a FromIterator<String> for String-shaped goal.

Candidate evaluation checks the relevant implementation and nested requirements.

Completion and failure variants#

Fulfillment drains pending obligations after inference progress.

Fallback handles only eligible literal domains; none is central here.

Typeck results retain expression types, method instances, and adjustments for downstream lowering.

Remove T: ToString, and method lookup may still find a named candidate for diagnostics, but fulfillment cannot prove applicability.

Change the return to Vec<String>, and expected type guides collect toward a different implementation.

Remove the return annotation in a context with no expectation, and collection target inference may remain ambiguous.

This trace shows why type checking is a constraint-and-proof process, not declaration lookup.

36. Compiler bugs, ICEs, and wrong diagnostics#

Classify before patching#

An ICE is an internal compiler error: rustc violated an internal expectation instead of issuing a normal diagnostic.

It does not identify the subsystem at fault.

The crashing assertion may only be the first consumer to notice poisoned data.

Classify failures as acceptance of invalid code, rejection of valid code, compiler crash, nontermination, performance regression, or wrong diagnostic.

Acceptance and miscompilation deserve immediate soundness scrutiny.

Common fault patterns#

Leaked inference assignments make results depend on candidate order.

Missing obligation registration accepts a top-level relation while dropping a where-clause.

Premature fallback rejects code that later context could infer.

Unnormalized aliases produce false mismatches.

Over-normalization reveals opaque types where abstraction should hold.

Incorrect binder shifting captures or leaks higher-ranked variables.

Incomplete canonical cache keys reuse answers under the wrong environment.

Cycle policy mistakes either overflow valid recursive code or prove unsupported cycles.

Incorrect coherence negative reasoning assumes downstream crates cannot add legal impls.

Lost adjustments cause later MIR failures despite successful type checking.

Wrong diagnostics are separate bugs#

Core solving may correctly reject code while diagnostics report the wrong candidate, reverse expected and found types, underline generated syntax, or suggest an impossible bound.

Preserve obligation causes and proof-tree information.

Test emitted messages without coupling every test to unstable debug formatting.

To locate a regression, minimize macros, replace aliases with explicit forms, toggle generic versus concrete types, and compare solver modes only as a diagnostic experiment.

Never conclude that the alternate solver defines correct language behavior merely because it accepts the testcase.

37. Source map for contributors#

Body checking and inference#

Start in compiler/rustc_hir_typeck for expression and body type checking, expectations, coercions, method lookup integration, and typeck result recording.

Look in compiler/rustc_infer for inference contexts, type relations, snapshots, canonical query plumbing, region constraints, and diagnostics tied to inference.

The exact directory layout changes, so search by current symbols from a backtrace rather than relying only on this map.

Nightly rustdoc for rustc_hir_typeck and rustc_infer provides public and internal API orientation.

Solvers and coherence#

compiler/rustc_trait_selection contains old-style selection, fulfillment, projection/normalization, dyn compatibility utilities, and integration logic.

compiler/rustc_next_trait_solver contains the in-tree next-generation solver's canonical goal evaluation, candidate machinery, search graph, and related data.

Coherence orchestration is associated with rustc_hir_analysis::coherence, while supporting overlap and specialization machinery also appears in trait-selection code.

Search for the query provider invoked by the failing test.

Practical contribution route#

Build a stage compiler, add a minimized UI test, and run the narrow test suite with ./x test tests/ui/path/to/test.rs according to current bootstrap guidance.

Use RUSTC_LOG targets selectively; tracing every trait event can become enormous.

Compare query stacks and obligation causes.

Read nearby tests before changing semantics.

Ask the compiler types team when a change touches solver certainty, coherence, HRTBs, or opaque normalization.

A good first contribution improves a diagnostic span or adds a regression test around understood behavior.

38. Exercises, talks, and further reading#

Exercises#

1. Trace constraints for let x: Vec<_> = (0..3).map(|n| n + 1).collect(); and identify expected-type flow.

1. Draw binders and de Bruijn distances for two nested for<'a> function types.

1. Extend the educational unifier with tuples and prove the occurs check still traverses every child.

1. Freshen rule variables in the miniature solver so separate candidates cannot share variable numbers.

1. Construct a method-call ambiguity involving two traits and explain search, confirmation, and diagnostics separately.

1. Explain why raw inference variables make unsafe cache keys.

1. Compare an inductive recursive trait rule with an auto-trait-like coinductive cycle without claiming either is accepted by current Rust.

1. Design cross-crate impl headers that would conflict without orphan rules.

1. Minimize a projection error by replacing a GAT with a non-generic associated type, then interpret the difference.

1. Find the exact 1.97.1 source path that evaluates a trait goal from method lookup and record every context boundary.

Talk-ready explanations#

In thirty seconds: rustc type checking is cooperative constraint solving.

Expressions contribute equations, expectations guide unknowns, trait operations create proof goals, and fulfillment revisits uncertain goals as knowledge grows.

In two minutes: source-shaped HIR types lower to interned semantic types.

A body-local inference context owns mutable unknowns.

Relations constrain them, method lookup probes candidates, normalization handles associated types, and a trait solver proves goals from impls and assumptions.

Canonicalization removes local variable identities before caching.

Coherence ensures separate crates cannot create forbidden overlapping meanings.

In ten minutes: add binders and universes, explain rollback, contrast old selection/fulfillment with canonical next-solver goal evaluation, and end with diagnostics as reconstruction of why proof search failed.

Versioned sources#

• Rust Compiler Development Guide: type inference
• Rust Compiler Development Guide: HIR type checking
• Rust Compiler Development Guide: old-style trait resolution
• Rust Compiler Development Guide: next solver overview
• Rust Compiler Development Guide: canonical queries
• Nightly rustdoc: rustc_middle::ty
• Nightly rustdoc: rustc_infer
• Nightly rustdoc: rustc_hir_typeck
• Nightly rustdoc: rustc_trait_selection
• Nightly rustdoc: rustc_next_trait_solver
• Chalk sunset README
• Chalk book, for historical logic-programming ideas
• Rust Reference: implementations and coherence
• Rust Reference: trait objects

Read docs built from the same commit as the compiler whenever exact APIs matter.

The architecture is moving, uncertainty is an explicit solver result, and careful version labels are part of technical correctness.

Part IV-B: Type-System and Inference Internals#

39. The boundary this continuation studies#

Type checking turns source-shaped type syntax into justified facts about a body.

This continuation follows that transformation from HIR to the products consumed by later passes.

It assumes only basic Rust, but ends at the level needed to review a compiler patch.

The concrete problem is uncertainty.

In let x = Vec::new(), neither the parser nor name resolver knows the element type.

In f(&mut value), several legal conversions may connect the argument to the parameter.

In x.method(), the receiver type, dereferencing, trait scope, and generic arguments interact.

Rustc must preserve uncertainty long enough to learn from context without accepting contradictions.

The small mental model is a notebook.

Each expression writes constraints into a body-local notebook.

Inference variables are blank cells.

Relations connect cells and known terms.

Probes tentatively write and erase alternatives.

Finalization rejects unresolved cells that have no specified fallback.

HIR body --lower types--> semantic terms
    |                         |
    | expressions             | known signatures
    v                         v
type-checking context --constraints--> inference tables
    |                                  |
    | obligations (solver boundary)    | resolved terms
    v                                  v
fulfillment/normalization           typeck results
                                          |
                           adjustments, node types, user types

The arrows carry facts, not values at runtime.

This model stops being sufficient at trait proof search.

Detailed candidate assembly, search graphs, cycles, and solver algorithms belong in the solver continuation.

Here a trait query is treated as a service with inputs, certainty, constraints, and errors.

That boundary matters: inference owns local unknowns; the solver must not retain their accidental identities.

Core invariant: every accepted body has internally consistent resolved types, justified required predicates, and recorded implicit conversions.

The first visible error may be much later than the first broken invariant.

Therefore debugging follows fact creation, not only the final diagnostic.

40. Three representations, three jobs#

The source text &'a mut Vec<T> is spelling.

An HIR type is a resolved, source-oriented compiler node.

A semantic type is a term used for equality, substitution, inference, and queries.

These are not redundant copies.

HIR can distinguish explicit syntax from omission.

Semantic terms can distinguish two same-named definitions by DefId.

Keeping only HIR would make every comparison repeatedly resolve names and elaborate sugar.

Keeping only semantic types would make it difficult to underline the user's _ or omitted lifetime.

Lowering therefore moves cost.

It pays resolution and elaboration once so later semantic operations are cheaper.

It must retain side information so diagnostics can recover source intent.

Consider:

type Bytes = Vec<u8>;

fn consume(_: Bytes) {}

The alias spelling is valuable to a diagnostic.

Many semantic questions need the expanded meaning.

Those observations are different, so “equal after normalization” does not imply “print identically.”

HIR lowering is contextual.

Self depends on the owner.

Elided regions depend on the syntactic position and elision rules.

_ creates an inference request where permitted.

An associated-type path can become an alias term rather than an immediately known type.

The lowering invariant is that every resolved identity is unambiguous and every unknown remains explicit.

Never replace a failed lowering with a plausible type merely to continue.

Use a designated error term and preserve the emitted error guarantee.

That prevents one error from manufacturing false certainty.

Debugging experiment: print HIR and semantic type at the earliest surprising node.

If HIR is wrong, investigate parsing, expansion, or resolution.

If HIR is right and Ty is wrong, inspect lowering, generic argument construction, and owner context.

If both are right but the message is wrong, inspect diagnostic provenance and pretty-printing.

41. Ty, interning, and flags#

At the rustc 1.97.1 source line, Ty<'tcx> should be understood as a cheap handle to an interned TyKind<'tcx>.

Exact aliases, fields, and variant names are internal APIs and must be checked against that revision's generated rustdoc.

Interning stores one canonical allocation for structurally equal immutable terms within a compiler context.

Pointer-sized handles then make cloning cheap.

Identity comparison can often be fast because equal interned values share storage.

Interning is not a language guarantee and does not mean addresses are stable across sessions.

construct Tuple([u8, bool])
          |
          v
      interner lookup ---- hit ----> existing handle
          |
         miss
          v
 immutable arena allocation --> new handle

The key invariant is immutability after interning.

Mutating an interned node would silently change every user and corrupt hash tables.

Interning moves work from repeated traversal to construction and memory retention.

It improves repeated equality-heavy workloads when sharing is high.

It can lose when most terms are unique or hashing dominates.

Measure representative crates rather than counting allocations in a toy.

Useful measurements include intern-table hit rate, bytes retained, type-fold visits, and query time.

Semantic types carry or expose cached type flags.

Flags summarize properties such as containing inference variables, parameters, placeholders, aliases, errors, regions requiring work, or escaping bound variables.

The precise set changes with rustc.

A flag answers “might this subtree contain X?” without recursively walking it.

It is a conservative summary: consumers must understand whether a flag means definitely or possibly.

Cached flags establish an invariant between a node and all descendants.

Forgetting a child's flag can skip substitution or binder shifting and become unsound.

Adding an unnecessary flag usually costs performance but preserves correctness.

Thus flag maintenance deserves tests even when the field looks like an optimization.

An educational implementation illustrates the contract.

#[derive(Clone, Debug, PartialEq, Eq, Hash)]
enum Ty {
	Bool,
	Param(u32),
	Infer(u32),
	Tuple(Vec<Ty>),
}

const HAS_PARAM: u8 = 1;
const HAS_INFER: u8 = 2;

fn flags(ty: &Ty) -> u8 {
	match ty {
		Ty::Bool => 0,
		Ty::Param(_) => HAS_PARAM,
		Ty::Infer(_) => HAS_INFER,
		Ty::Tuple(xs) => xs.iter().fold(0, |all, x| all | flags(x)),
	}
}

fn main() {
	let ty = Ty::Tuple(vec![Ty::Bool, Ty::Infer(0)]);
	assert_eq!(flags(&ty), HAS_INFER);
}

This is a complete stable-Rust program.

It does not intern values; it isolates the summary invariant.

Production rustc uses richer compact representations and compiler-owned arenas.

42. Generic arguments and substitution#

A generic definition has parameters.

An instantiated use has arguments.

Rust has three argument domains: regions, types, and consts.

Treating them as one untagged array would permit category mistakes.

Conceptually rustc uses a tagged generic argument and an interned ordered argument list.

struct SliceRef<'a, T, const N: usize>(...)

parameters: ['a: region, T: type, N: const]
arguments:  ['x,          u16,     4]

Order is compiler-defined by the definition's generics, including parent generics.

Associated items are the common trap.

Their argument list may include the trait or impl's parent arguments before item-local arguments.

A substitution maps each parameter identity to an argument of the same domain.

It is not textual replacement.

It must walk semantic structure, cross binders correctly, and preserve interned sharing.

Invariant: substitution never captures a bound variable and never changes argument kind.

Suppose Pair<T, U> contains (T, Vec<U>).

Applying [T := u8, U := bool] gives (u8, Vec<bool>).

Applying the list in reverse can still produce a valid-looking but wrong type.

This is why generic argument bugs often survive until a later mismatch.

Regions inside types include early-bound parameters, late-bound variables, placeholders, inference variables, erased regions, and special forms used by the compiler.

Const arguments can be values, parameters, unevaluated expressions, inference variables, bound variables, placeholders, or errors depending on phase and revision.

Do not assume a const in a type has already evaluated to bits.

[T; N + 1] may need generic const reasoning and evaluatability checks.

Substitution can expose new work.

Replacing a parameter by an alias can make normalization necessary.

Replacing a const parameter can make evaluation possible.

Replacing a region can create outlives constraints.

Doing all that eagerly would simplify consumers but duplicate work and amplify cycles.

Doing nothing would force every consumer to rediscover prerequisites.

Rustc uses phase-specific operations; callers must choose substitution, normalization, and evaluation deliberately.

Counterexample: two argument lists of equal length are not necessarily compatible.

['a, T] and [u32, 3] have matching lengths but mismatched domains.

Validate against parameter definitions, not shape alone.

43. Binders, de Bruijn indices, and universes#

for<'a> fn(&'a u8) -> &'a u8 means one function value works for every permitted 'a.

The lifetime is bound rather than free.

A binder packages a value with variables introduced at its boundary.

Bound variables need identity that remains correct when syntax is moved or nested.

Names are a poor internal identity because alpha-renaming changes names without changing meaning.

Rustc uses de Bruijn-style indexing for bound variables.

The index records how many binder boundaries outward one must travel to find the binder.

binder A: for<'a>
  binder B: for<'b>
    (&'a u8, &'b u8)
       ^ depth 1  ^ depth 0

Entering a binder increases the distance to variables bound outside it.

Leaving decreases that distance where legal.

Substituting under a binder must shift inserted bound variables.

Failing to shift causes capture: a variable accidentally refers to the inner binder.

Shifting too far causes a variable to escape its intended binder.

An “escaping bound variable” flag is therefore both a fast check and an invariant alarm.

A universe models which placeholders an inference variable is allowed to name.

When checking a higher-ranked requirement, rustc can instantiate its bound variable with a fresh placeholder in a new universe.

An older inference variable must not be solved with that newer placeholder if doing so would let a local choice escape.

U0: ?R created
 |
 +-- enter higher-ranked binder --> U1: placeholder !P

forbidden solution: ?R := !P
reason: !P is not nameable from U0

Invariant: a solution mentions only placeholders visible from the variable's universe.

This is the type-theoretic analogue of preventing a local reference from escaping its scope.

Binders and universes solve related but different problems.

Binders represent lexical binding in terms.

Universes constrain inference while reasoning about fresh placeholders.

Replacing both with globally unique IDs would simplify identity but not enforce visibility.

The visibility relation would have to be rebuilt elsewhere.

Prediction exercise: in three nested binders, what happens to an outer variable's index when one new binder is inserted around its occurrence?

Answer: its de Bruijn distance increases by one; the variable's binder does not change.

Debug higher-ranked failures by printing binder depth, variable kind, universe, and the operation that instantiated the binder.

Pretty-printed lifetime names alone conceal the critical information.

44. Inference variables and their domains#

An inference variable is a mutable unknown owned by an inference context.

It is not a generic parameter.

A parameter means the checked code must work for every caller-chosen argument satisfying bounds.

An inference variable means rustc may choose one answer consistent with constraints.

Rustc keeps distinct variable domains.

General type variables can stand for types.

Integer variables range over integer types and participate in integer fallback.

Floating variables range over floating types and participate in float fallback.

Region variables stand for inferred regions and accumulate region constraints.

Const variables stand for compile-time arguments of an expected const type.

Effect-like or specialized variables may exist at particular revisions; consult 1.97.1 source rather than extrapolating.

Domain separation prevents solving an integer literal variable with bool.

It also permits compact, specialized tables and fallback policy.

Inference tables often use union-find-like structures for equality classes.

A root stores either unresolved metadata or a known value.

Path compression and union by rank make repeated lookup nearly constant amortized time.

Rollback complicates ordinary destructive path compression.

An implementation can log every mutation, use rollback-friendly union-find, or clone state.

Cloning is simplest but expensive for large bodies and frequent probes.

Mutation logs add implementation complexity to preserve fast speculation.

Invariant: resolving any member of an equivalence class observes the root's current value.

Invariant: assigning a root cannot contradict an existing assignment.

Invariant: variables never migrate between inference contexts.

Raw variable indices are local identities.

They are unsafe query-cache keys outside their context.

Canonicalization replaces them before crossing that boundary.

Region inference differs from ordinary type equality.

It often records outlives constraints for later solving rather than selecting one lexical lifetime immediately.

Const inference must track the const's type as well as its eventual value or expression.

An unresolved _ array length is not interchangeable with an unresolved element type.

Fallback is policy, not unification.

An unconstrained integer literal commonly falls back according to language/compiler rules.

An unconstrained general type variable is usually an error.

Moving fallback earlier reduces ambiguity temporarily but can reject programs whose later context would decide differently.

Moving it later improves information but may worsen diagnostic locality.

45. A progressive educational unifier#

The following complete stable-Rust program implements type variables, constructors, an occurs check, snapshots by cloning, and rollback.

It intentionally omits regions, subtyping, binders, aliases, consts, and diagnostics.

#[derive(Clone, Debug, PartialEq, Eq)]
enum Ty {
	Var(usize),
	Bool,
	Int,
	List(Box<Ty>),
	Pair(Box<Ty>, Box<Ty>),
}

#[derive(Clone, Debug)]
struct Infer {
	values: Vec<Option<Ty>>,
}

impl Infer {
	fn new_var(&mut self) -> Ty {
		let id = self.values.len();
		self.values.push(None);
		Ty::Var(id)
	}

	fn shallow(&self, ty: Ty) -> Ty {
		match ty {
			Ty::Var(v) => match &self.values[v] {
				Some(value) => self.shallow(value.clone()),
				None => Ty::Var(v),
			},
			other => other,
		}
	}

	fn occurs(&self, needle: usize, ty: Ty) -> bool {
		match self.shallow(ty) {
			Ty::Var(v) => v == needle,
			Ty::List(x) => self.occurs(needle, *x),
			Ty::Pair(a, b) => self.occurs(needle, *a) || self.occurs(needle, *b),
			Ty::Bool | Ty::Int => false,
		}
	}

	fn unify(&mut self, left: Ty, right: Ty) -> Result<(), String> {
		let left = self.shallow(left);
		let right = self.shallow(right);
		match (left, right) {
			(Ty::Var(a), Ty::Var(b)) if a == b => Ok(()),
			(Ty::Var(v), ty) | (ty, Ty::Var(v)) => {
				if self.occurs(v, ty.clone()) {
					Err("infinite type".into())
				} else {
					self.values[v] = Some(ty);
					Ok(())
				}
			}
			(Ty::Bool, Ty::Bool) | (Ty::Int, Ty::Int) => Ok(()),
			(Ty::List(a), Ty::List(b)) => self.unify(*a, *b),
			(Ty::Pair(a1, b1), Ty::Pair(a2, b2)) => {
				self.unify(*a1, *a2)?;
				self.unify(*b1, *b2)
			}
			(a, b) => Err(format!("cannot unify {a:?} with {b:?}")),
		}
	}

	fn probe<T>(&mut self, f: impl FnOnce(&mut Self) -> T) -> T {
		let saved = self.values.clone();
		let result = f(self);
		self.values = saved;
		result
	}
}

fn main() {
	let mut inf = Infer { values: vec![] };
	let x = inf.new_var();
	inf.unify(Ty::List(Box::new(x.clone())), Ty::List(Box::new(Ty::Int))).unwrap();
	assert_eq!(inf.shallow(x), Ty::Int);

	let y = inf.new_var();
	let failed = inf.probe(|i| i.unify(y.clone(), Ty::Bool).and_then(|_| i.unify(y, Ty::Int)));
	assert!(failed.is_err());
	assert_eq!(inf.values[1], None);

	let z = inf.new_var();
	let recursive = Ty::List(Box::new(z.clone()));
	assert!(inf.unify(z, recursive).is_err());

	let pair = Ty::Pair(Box::new(Ty::Bool), Box::new(Ty::Int));
	assert!(inf.unify(pair.clone(), pair).is_ok());
}

The occurs check prevents the finite representation ?0 = List<?0> from pretending to be an infinite tree.

Rustc's exact occurs and recursion handling depends on term kinds; do not transplant this toy rule blindly.

The probe restores state even after a failed alternative.

The toy's clone costs O(number of variables) per probe.

A mutation log costs roughly O(number of writes in the probe), plus rollback bookkeeping.

Milestone exercises:

1. Replace recursive shallow with a cycle-safe resolver.

1. Add function types and report which parameter differs.

1. Replace cloning with an undo log.

1. Property-test that a failed probe leaves byte-for-byte equivalent tables.

1. Fuzz malformed variable indices and return errors rather than panicking.

46. Relations: equality, subtyping, and variance#

Unification is one relation, not the entire type checker.

Equality requires both terms to denote the same type under the operation's normalization policy.

Subtyping permits a value of one type where another is expected.

Coercion performs designated implicit conversion and can record an adjustment.

These must not be collapsed into “compatible.”

A type relation recursively compares structure while accumulating constraints.

The relation carries cause, parameter environment, variance, and normalization choices as needed.

Lifetimes motivate subtyping.

A reference valid for longer can often be used where a shorter validity is required, subject to mutability and type rules.

Function parameters reverse direction.

A function able to accept a broader set of inputs can stand in for one requiring a narrower set.

This reversal is contravariance.

Results preserve direction and are covariant.

Invariant positions permit neither direction.

Bivariant positions impose no relation, though they are uncommon and must be justified by representation use.

&'a T is covariant in appropriate components when T is shared.

&'a mut T is invariant in T because writing a B through storage actually containing only A would be unsafe.

Raw pointers and interior mutability require their exact language rules; intuition is not enough.

Variance is inferred for definitions from how parameters occur, then used by relations.

Inferring it once makes later substitution relations cheap.

The moved cost is maintaining a correct variance analysis and metadata across crates.

Counterexample: equality in every generic argument is safe but rejects valid lifetime subtyping.

Declaring every argument covariant accepts invalid writes.

The correct policy follows observable operations of the constructor.

Debug a relation failure by recording direction.

“Expected A, found B” loses whether rustc attempted B <: A, equality, or coercion.

Swapping expected and actual can invert region constraints and diagnostics.

47. Snapshots, probes, and transactional inference#

Method lookup and coercion frequently test alternatives.

Testing can unify variables, add obligations, and register region constraints.

Rejected alternatives must leave no residue.

A snapshot marks inference state.

A probe runs speculative work and rolls back.

A committed snapshot preserves accepted mutations.

state S0 --snapshot--> candidate A writes w1,w2
                         |
                       reject
                         |
                     rollback --> exactly S0

state S0 --snapshot--> candidate B writes w3
                         |
                       accept
                         |
                      commit --> S1

Transactional invariant: rollback restores every observable inference component, not only type-variable values.

That includes region constraints, const variables, universe counters, delayed obligations, and side tables covered by the transaction.

If one table escapes rollback, candidate order can change compilation results.

That is a powerful regression test: permute candidate enumeration and check semantic stability.

Some diagnostics should also be buffered during probes.

Emitting an error from a rejected candidate creates a ghost diagnostic.

Instead return structured failure and report only after selection policy decides relevance.

Snapshots are not ordinary parallel transactions.

The body-local inference context is generally used under compiler-controlled sequencing.

Adding locks would not repair logical leakage and would increase cost.

Use narrow probes.

Long probes enlarge undo logs and make the winning path hard to diagnose.

Nested probes require stack discipline.

Rolling back an outer snapshot while retaining an inner commit is usually invalid unless explicitly supported.

Failure symptom map:

48. Expected types and bidirectional information#

Pure bottom-up checking would infer every child before considering its parent.

Rust needs information to flow downward as well.

An expected type is context supplied by the expression's destination.

In let x: Vec<u8> = collect();, the annotation guides the call result.

In return None, the function result can guide None's hidden type argument.

Expected types are hints with different strengths, not permission to force an incorrect answer.

The checker may use an expectation to instantiate variables, select a coercion target, or check a closure parameter.

It must still relate the computed type back to the expectation.

parent expected type
      |
      v
check child ----> child inferred type
      |                   |
      +---- relate/coerce-+
                |
          adjustment or error

Expected types improve inference locality and error quality.

They can also bias diagnostics if propagated too aggressively.

For example, forcing a whole block to the expected result before checking divergent branches can hide the branch that introduced the mismatch.

Rustc distinguishes forms of expectation in revision-specific APIs.

Read the 1.97.1 rustc_hir_typeck definitions before changing propagation.

The mechanism is downward information.

The policy decides where expectations are useful and how hard they constrain.

The presentation layer decides which side is labeled expected.

Keeping these separate prevents a wording fix from changing inference.

Prediction: what should happen to let x: u64 = 1;?

The annotation guides the unsuffixed integer variable to u64; no ordinary runtime cast is inserted.

Counterexample: an explicit 1i32 cannot become u64 merely because of expectation.

Its suffix is a committed type fact.

49. Coercions, adjustments, and coercion sites#

A coercion is an implicit compiler-directed conversion allowed at designated sites.

Examples include reborrowing, dereference-related conversions, function item to function pointer, noncapturing closure to function pointer, and certain unsizings.

The exact legal set is specified by the language and implementation; it is not arbitrary conversion search.

A coercion site has a target type supplied by context.

Common sites include let with an annotation, function arguments, returns, struct fields, and branch joins under their detailed rules.

Rustc records adjustments describing how an expression reaches its final type.

Later lowering needs them to build correct executable semantics.

Type checking that merely returns the target type but drops adjustments is incomplete.

expression type: &mut [i32; 3]
       |
       | reborrow + unsize metadata
       v
coerced type: &[i32]
       |
       v
recorded adjustment sequence consumed by THIR/MIR lowering

An adjustment sequence is ordered.

Changing order can change mutability, method dispatch, or generated MIR.

Autoderef adjustments and autoref must correspond to the receiver form selected during method confirmation.

Unsizing changes a sized pointer-like pointee to an unsized form while preserving a representation-specific pointer wrapper.

Array-to-slice and concrete-to-trait-object are familiar cases.

Struct-tail unsizing and smart-pointer participation depend on language traits and compiler rules.

Do not describe unsizing as copying an array into a slice.

It changes pointer metadata and view semantics, not the underlying allocation in the usual reference case.

Coercion is not transitive arbitrary search.

Allowing an unbounded chain of “reasonable conversions” would make inference unpredictable and method resolution unstable.

Rust instead defines bounded, contextual conversions.

At branch joins, rustc may need a common target and may revisit earlier expressions.

This motivates a coercion accumulator rather than a single left-to-right equality check.

Diverging expressions such as return or panic!() interact through the never type and its coercion behavior.

Version-sensitive edge cases should be tested, not inferred from the slogan “never coerces to anything.”

50. Autoderef and the method-lookup boundary#

For receiver.method(args), type inference and method lookup cooperate.

The checker starts from a receiver type.

Autoderef produces a bounded sequence of candidate receiver types.

Lookup considers inherent and in-scope trait methods under language rules.

Candidate probing may create temporary inference constraints.

Confirmation selects substitutions and records receiver adjustments.

Detailed trait candidate solving is outside this file.

The important boundary is the query contract.

Method lookup may ask whether predicates could hold under the current parameter environment.

It must distinguish impossible, ambiguous due to unknowns, and viable-enough-to-continue outcomes.

Treating ambiguity as failure rejects inferable calls.

Treating ambiguity as success without later confirmation accepts unresolved calls.

receiver expression
  |
  v
infer receiver Ty --> autoderef steps --> assemble method candidates
                                             |
                                  probe relations/obligations
                                             |
                                   select and confirm
                                             |
                      substitutions + autoref/autoderef adjustments

Autoderef is bounded to prevent runaway recursion and pathological compile time.

Failures should report an overflow or lookup error rather than hang.

Built-in dereference and trait-based dereference must preserve their distinction in diagnostics and obligations.

Autoref chooses a reference form required by the method receiver.

Two-phase borrowing may affect later borrow checking; type checking records the appropriate adjustment information rather than proving all borrow validity itself.

Method generic arguments combine receiver-owner and method-local parameters.

Argument rebasing mistakes can select the right method with wrong substitutions.

Debugging checklist:

1. Print the unadjusted receiver type.

1. Print each autoderef step and obligations it introduced.

1. Separate candidate assembly from viability probing.

1. Print selected item DefId and instantiated signature.

1. Inspect final receiver adjustment order.

1. Only then inspect solver internals if the boundary answer is suspect.

51. Function signatures, function items, and ABI types#

A function declaration has a signature: inputs, output, safety, ABI, and variadic status where applicable.

Bound lifetimes can make the signature higher-ranked.

A function item type identifies one specific function definition and its generic arguments.

A function pointer type describes callable pointer values with a signature.

These are distinct semantic types.

Function items are zero-sized identities in the language model and can coerce to compatible function pointers.

fn add_one(x: i32) -> i32 { x + 1 }

fn main() {
	let item = add_one;
	let pointer: fn(i32) -> i32 = item;
	assert_eq!(pointer(4), 5);
}

This complete stable program demonstrates a coercion, not type identity.

Closure types are also unique and are not function item types.

A noncapturing closure can coerce to an appropriate function pointer under language rules.

An ABI is a calling convention contract.

It affects how arguments and results cross a call boundary, symbol-level interoperability, unwind behavior, and target-specific lowering.

ABI spelling in a Rust function type is semantic, not decoration.

extern "C" fn(i32) and Rust-ABI fn(i32) must not be unified merely because inputs match.

Variadic signatures have additional restrictions.

Unsafe function pointers and safe function pointers participate in their specified relation; never infer safety variance from ordinary argument variance.

Rust-level types are not yet final machine ABI layouts.

Later layout and ABI classification depend on the target, repr attributes, scalar validity, aggregate layout, and backend contract.

Type checking validates the language-level signature and required feature or FFI conditions.

It does not choose every register.

Moving target ABI classification into type inference would contaminate portable semantic queries with target-lowering details.

Moving language ABI compatibility entirely to codegen would detect errors too late and risk inconsistent monomorphizations.

The architecture keeps a deliberate boundary.

52. Well-formedness and the parameter environment#

A type can be syntactically valid yet not well formed under current assumptions.

Well-formedness asks whether prerequisites of a type or predicate hold.

For a generic application, that may require bounds declared by the referenced definition.

For references and projections, region and trait-related conditions can arise.

The parameter environment packages assumptions available while checking an item.

Inside fn clone_it<T: Clone>(x: T), T: Clone is an assumption, not a globally proven concrete implementation.

Queries that omit the parameter environment can reject generic code or prove facts under assumptions that were never in scope.

item generics + where clauses + reveal/typing mode
                    |
                    v
               ParamEnv
                    |
        +-----------+-----------+
        |                       |
 relation/normalization      WF/trait query

The exact constituents and typing modes are revision-sensitive around rustc 1.97.1.

Use the matching source definition as normative for implementation work.

Well-formedness is not the same as proving every operation used in a body.

It establishes that declarations and instantiated terms have required foundations.

Obligation causes should identify why WF was requested.

Otherwise a missing bound may point at an unrelated use site.

Implied bounds are facts made available from certain well-formed input types according to language rules.

Do not generalize “mentioned in a signature” into “all nested trait bounds are implied.”

Rust's lifetime implied bounds and trait-bound behavior differ.

The Reference and rustc-dev-guide should settle the exact rule.

Early WF checking improves locality but may lack inferred substitutions.

Late WF checking has more information but worse spans and can permit invalid terms to contaminate inference.

Rustc checks at multiple architectural points with explicit responsibility.

Contributor task: trace one generic struct field from HIR lowering through its WF obligations and identify which query owns each emitted diagnostic.

53. The normalization boundary#

Normalization replaces an alias-like term with an equivalent form when justified.

Examples include associated type projections and opaque aliases under allowed reveal modes.

Normalization is not string expansion.

It can require predicates, produce obligations, remain ambiguous, encounter cycles, or be forbidden by abstraction boundaries.

This file deliberately does not derive the solver algorithm that answers those predicates.

Instead, treat normalization as a boundary operation:

input: term + ParamEnv + inference context/typing mode + cause
output: normalized term + constraints/obligations, or ambiguity/error

Shallow normalization exposes only enough outer structure for the caller.

Deep normalization recursively visits nested terms.

Eager deep normalization simplifies consumers but repeats work and destroys useful alias provenance.

Lazy normalization preserves structure and controls cycles but requires callers to request it before structural assumptions.

Invariant: a consumer that pattern-matches on a required outer constructor has established that aliases cannot hide a different outer form.

Violating it causes “impossible” matches or missed coercions.

Normalization can introduce new inference constraints.

Those constraints must be applied transactionally during probes.

It can also introduce obligations that fulfillment handles later.

Dropping returned obligations turns “could normalize if P holds” into “normalized unconditionally.”

That is a correctness bug.

Reveal policy matters for opaque types.

Code outside the defining scope must observe the abstraction, not freely substitute the hidden type.

Caching normalization therefore requires all semantically relevant environment and mode inputs in the key.

Failure map:

54. Opaque types and impl Trait#

impl Trait presents an abstract type constrained by bounds.

The hidden concrete type is inferred or defined under rules tied to the opaque definition and its defining use sites.

The abstraction is semantic, not merely a pretty-printing alias.

Return-position impl Trait lets a function expose capabilities without exposing the concrete return type.

Argument-position impl Trait is largely surface syntax for a generic parameter, subject to its precise language rules.

They should not be given one implementation story merely because spelling matches.

fn numbers() -> impl Iterator<Item = u8> {
	[1, 2, 3].into_iter()
}

fn main() {
	assert_eq!(numbers().sum::<u8>(), 6);
}

This complete stable program uses return-position opaque type abstraction.

All defining returns for one opaque must agree on one hidden type after permitted inference and normalization.

“Any type implementing the bounds per return” would describe existential choice per branch and is not the rule.

The compiler must track which generic parameters and lifetimes the hidden type may capture.

Capture rules have evolved, including edition-sensitive behavior; verify Rust 1.97.1 and the item's edition.

An opaque's identity includes its definition, not only its stated bounds.

Two functions returning impl Iterator<Item=u8> generally expose distinct opaque types.

Within allowed defining scope, rustc relates defining uses to infer the hidden type.

Outside, normalization must respect reveal mode.

Cycle and leakage checks prevent an opaque from defining itself illegally or exposing unnameable local facts.

Error recovery should poison the opaque once rather than report every later use as a separate mismatch.

Useful diagnostics compare defining sites and explain conflicting hidden types while printing the abstraction appropriately elsewhere.

55. GATs, RPITIT, and associated opaque types#

A generic associated type is an associated type with its own generic parameters.

trait Lending {
	type Item<'a> where Self: 'a;
	fn get<'a>(&'a self) -> Self::Item<'a>;
}

Self::Item<'x> is a projection with parent arguments and item-local arguments.

Substitution must preserve their ordering and binder depth.

Required where clauses express conditions needed for maximal useful implementations under current language rules.

Those rules have changed historically; consult the 1.97.1 Reference and tests.

GATs stress normalization because the projected result depends on lifetime or const arguments introduced at use sites.

They stress higher-ranked reasoning when a bound quantifies over all arguments.

They do not justify treating every projection as eagerly reducible.

Return-position impl Trait in traits, commonly RPITIT, exposes an opaque result associated with a trait method.

Rustc models this using internal associated-item and opaque machinery whose exact lowering is an implementation detail.

Do not state that source authors literally declared the synthesized items.

Trait declaration, impl method, and call site need identities and arguments aligned across owners.

Impl Trait in associated types and related features may have stability constraints distinct from RPITIT.

Always separate stable language behavior from internal representation and nightly features.

Counterexample: replacing an RPITIT result by Box<dyn Trait> changes allocation, dynamic dispatch, auto traits, lifetimes, and identity.

It may be a useful API alternative but is not equivalent implementation lowering.

Debug trace:

1. Find the source method and synthesized associated item identities.

1. Print parent and method generic arguments.

1. Inspect capture and binder lists.

1. Observe the projection or opaque term before normalization.

1. Check reveal mode and defining scope.

1. Only then inspect the trait solver response.

56. Const generics inside the type system#

Const generics let compile-time values participate in type identity.

[u8; 3] and [u8; 4] are different types.

A const term has a type, such as usize for ordinary array lengths.

It may be a concrete value, parameter, inference variable, bound variable, placeholder, unevaluated expression, or error term.

The exact ConstKind representation is internal and revision-sensitive.

Equality of const terms is not always simple syntax equality.

Evaluation may reveal equal values.

Generic expressions may remain symbolic.

The compiler must avoid claiming algebraic equalities it cannot justify under current language rules.

fn first<const N: usize>(xs: [u8; N]) -> Option<u8> {
	xs.first().copied()
}

fn main() {
	assert_eq!(first([9, 8]), Some(9));
}

This complete stable program infers N = 2 from the argument type.

Const inference relates a variable of known const type to an argument.

Evaluatability asks whether a const expression can be evaluated where a concrete value is required.

Validity asks whether the resulting value is legal for its use, such as representable layout constraints.

These checks belong to different stages and should produce different diagnostics.

An unevaluated const must retain its definition identity and generic arguments.

Otherwise two same-spelled expressions in different scopes could be confused.

Caching evaluation requires target and environment inputs where semantics depend on them.

Resource limits matter because compile-time evaluation can consume time and memory.

Cancellation and deterministic error behavior are production concerns, not optional polish.

Do not reduce const generics to “run arbitrary Rust during inference.”

Inference manipulates symbolic const terms; const evaluation is a service with its own restrictions and query boundaries.

57. Error recovery and diagnostic provenance#

A compiler should continue after many user errors to report useful independent problems.

Continuation requires an explicit error type or error-tainted term.

It must not pretend the erroneous expression had a normal arbitrary type.

Rustc tracks an emitted-error guarantee so recovery values are associated with a diagnostic already issued.

Exact token and API names should be checked in 1.97.1 source.

An error type often relates permissively to prevent cascades.

That permissiveness is recovery policy, not language subtyping.

No accepted program gains capabilities from it because compilation already failed.

Invariant: introducing an error term requires an existing or deliberately emitted primary error.

Invariant: secondary checks recognize taint and avoid redundant messages without suppressing independent errors.

Diagnostics need provenance:

• the HIR node and span;
• expected-type origin;
• relation direction;
• obligation cause;
• generic argument position;
• coercion or method candidate attempted;
• normalization and opaque context.

Discarding provenance makes semantic data smaller but forces diagnostics to guess later.

That is a moved cost, usually toward worse messages.

Error reporting should resolve variables as far as safely possible.

Printing raw ?17t may help compiler developers but not users.

Over-resolving through a failed normalization may hide the meaningful alias.

Maintain separate debug and user-facing printers.

Delay choosing the primary mismatch until enough context exists, but do not let failed probes emit.

Regression tests should cover message, span, labels, suggestions, and absence of cascades.

Suggestions must be machine-applicable only when syntax and semantics are sufficiently certain.

58. Type-checking query products#

Body type checking produces more than “success.”

Later stages need a map from HIR expressions and patterns to semantic types.

They need substitutions for resolved paths and methods.

They need adjustment sequences for implicit operations.

They may need user-provided type annotations, liberated function signatures, closure information, field indices, and coercion results.

Rustc groups these in typeck result structures whose precise fields evolve.

Inspect rustc_middle::ty::TypeckResults at the 1.97.1 revision before writing code against it.

HIR owner body
    |
rustc_hir_typeck body query
    |
    +-- node types
    +-- node generic args
    +-- adjustments
    +-- method/field resolutions
    +-- user type annotations
    +-- closure/coroutine facts
    `-- error taint and auxiliary maps
             |
             v
        THIR/MIR construction and later analyses

Products are generally owner-scoped.

Indexing with a node from another owner violates assumptions even if numeric IDs happen to match.

Every expression consumed downstream must have the expected type and adjustments recorded.

Missing entries should fail near the producer in debug validation rather than panic deep in MIR lowering.

Query results must not expose mutable inference variables.

Resolve or erase body-local uncertainty according to the product contract before caching.

Regions may be represented in phase-appropriate forms; “resolved” does not always mean final borrow-checker regions.

Type checking establishes type and method facts.

Borrow checking later proves use and lifetime constraints on MIR.

Crossing those responsibilities causes either duplicated reasoning or missing constraints.

Incremental compilation hashes query inputs and outputs.

Nondeterministic iteration order in typeck products can cause unstable fingerprints or diagnostics.

Use stable ordering where output observation requires it.

59. Source architecture at rustc 1.97.1#

This map is orientation, not a promise of permanent paths.

Pin the Rust 1.97.1 source commit or release tag and use generated nightly-style rustdoc from that checkout.

compiler/rustc_hir_typeck owns much body expression checking, expectations, coercions, method lookup integration, and result recording.

compiler/rustc_infer owns inference contexts, variable tables, relations, snapshots, region constraints, and interfaces to canonical reasoning.

compiler/rustc_middle/src/ty defines central semantic terms, contexts, binders, generic arguments, flags, and typeck products.

compiler/rustc_hir_analysis covers item-level checking and well-formedness/coherence orchestration in revision-specific modules.

compiler/rustc_trait_selection and compiler/rustc_next_trait_solver provide trait and normalization services.

Their detailed solving algorithms are intentionally delegated to the separate solver chapter.

compiler/rustc_query_impl and provider registration connect query declarations to implementations.

THIR and MIR construction consume typeck products in later compiler crates.

Search symbols rather than relying on remembered file names:

rg "struct TypeckResults" compiler
rg "struct InferCtxt" compiler
rg "enum TyKind" compiler/rustc_middle
rg "fn check_expr" compiler/rustc_hir_typeck
rg "adjustments" compiler/rustc_hir_typeck compiler/rustc_middle

These are source-reading commands, not claims that every symbol spelling remains unchanged.

Use -Z query-dep-graph, query logging, or RUSTC_LOG only with the matching compiler's help and tracing targets.

Unfiltered logs can be enormous and perturb timings.

A productive path starts from a minimized UI test.

Find the first query whose output differs from expectation.

Then find the earliest incorrect term, constraint, or adjustment inside that query.

Do not start by editing the final diagnostic snapshot.

60. End-to-end trace: array, closure, and collection#

Trace this body conceptually:

fn make() -> Vec<u16> {
	[1, 2, 3].into_iter().map(|x| x + 1).collect()
}

The return position supplies expected Vec<u16> to the tail expression.

Array literals begin with an element integer inference variable, constrained consistently across elements.

The length is the concrete const 3 of the required array-length type.

Method lookup starts from [?I; 3] and explores the receiver's valid autoderef and trait/inherent candidates.

Confirmation records the selected into_iter method and generic arguments.

Its result type contains an iterator item related to ?I.

The map method supplies the item type as expectation for closure parameter x.

The unsuffixed 1 introduces another integer inference variable.

x + 1 requires operator typing and a trait obligation at the solver boundary.

The resulting mapped item remains related to integer variables and associated output normalization.

collect creates a result type variable and a collection obligation.

The return expectation constrains that result to Vec<u16>.

Collection's element relationship then pushes information toward the mapped item.

Ultimately integer constraints settle on u16 if all obligations and relations permit it.

Final typeck results record types for each HIR expression, selected methods and arguments, and any adjustments.

Fulfillment ensures required obligations are discharged or diagnosed.

Fallback should not override the explicit Vec<u16> evidence.

If the result incorrectly becomes Vec<i32>, inspect expected-type propagation before integer fallback.

If map cannot find a method, inspect the into_iter receiver and selected edition-sensitive behavior before solver search.

If type checking succeeds but MIR calls the wrong receiver form, inspect adjustments.

If only a generic wrapper fails, inspect ParamEnv, substitutions, and normalization.

This trace demonstrates cooperative inference rather than one-way deduction.

61. Production hardening and performance#

Correctness tests should isolate each invariant.

UI tests cover accepted and rejected programs with diagnostics.

Run-pass tests cover executable adjustment semantics.

Incremental tests cover stable query dependencies.

Cross-crate tests cover metadata, opaque boundaries, and generic argument identity.

Target tests cover ABI and layout-sensitive interfaces.

Property tests suit educational relations and rollback tables.

Differential tests may compare compiler revisions, but changed output is evidence, not proof of a regression.

Fuzz malformed syntax, extreme nesting, huge tuples, deep autoderef, and many ambiguous variables.

Set recursion and resource limits so adversarial code yields controlled diagnostics.

Avoid algorithms quadratic in expression count where a work list or interned sharing suffices.

Performance must be measured under an observation model.

Track wall time, peak memory, instruction counts, query hits, allocations, and diagnostic changes on representative crates.

An optimization is valid only if accepted programs, errors, and recorded adjustments preserve intended meaning.

Interning cost model:

cost ~= construction hashing + table lookup + retained unique terms
saving ~= avoided allocations + cheap clones + shared traversal/cache hits

Probe cost model:

clone snapshots: probes * total table size
undo log: writes inside probes + rollback traversal

Normalization cost depends on alias frequency, cache quality, environment diversity, and cycle handling.

Measure generic-heavy and associated-type-heavy crates, not only arithmetic examples.

Cancellation must not publish partial query products.

Panics and ICEs must avoid turning malformed user code into memory unsafety.

Compiler unsafe internals carry proof obligations about arena lifetime, interning uniqueness, and index validity.

Document caller guarantees and test boundary assertions.

62. Debugging workshops and failure maps#

Workshop A: candidate order changes a method result#

Minimize imports and candidate traits.

Reverse candidate iteration in a local compiler only as an experiment.

Log snapshot depth and inference writes.

If the result changes, locate state not rolled back after a rejected probe.

Do not “fix” the test by sorting candidates unless language policy specifies priority.

Workshop B: opaque hidden type leaks#

Compare diagnostics inside and outside the defining module.

Print opaque identity, reveal mode, and normalization caller.

Determine whether semantic normalization was unauthorized or only user printing exposed detail.

The fixes belong to different layers.

Workshop C: GAT mismatch mentions wrong lifetime#

Replace the GAT with a nongeneric associated type.

If the error disappears, print parent/item argument partition and binder depth.

Replace named lifetimes with fresh simple examples.

Inspect substitution shifting before changing solver logic.

Workshop D: MIR builder lacks an adjustment#

Find the HIR expression ID used by the consumer.

Inspect typeck results immediately after body checking.

If absent there, trace coercion or method confirmation.

If present there but absent later, inspect owner mapping and product consumption.

The diagnostic location is evidence about observation, not necessarily origin.

Always seek the earliest term that differs between good and bad builds.

63. Bounded compiler lab#

Build a small expression checker in milestones.

Milestone 1: define source expressions for booleans, integers, variables, pairs, lambdas, calls, and annotations.

Milestone 2: define semantic types with variables, primitives, pairs, and functions.

Milestone 3: implement equality unification with occurs checking.

Milestone 4: add lexical environments and instantiate generic schemes.

Milestone 5: pass expected types downward for annotations, calls, and lambdas.

Milestone 6: add snapshots and one speculative conversion.

Milestone 7: record adjustment-like evidence rather than silently changing types.

Milestone 8: add source spans and structured mismatch causes.

Explicit omissions:

• no Rust trait system;
• no region inference or borrow checking;
• no subtyping beyond a deliberately documented function rule;
• no opaque types, GATs, or const evaluation;
• no unsafe code;
• no incremental cache;
• no claim of Rust compatibility.

These omissions keep the proof surface bounded.

The lab still teaches representation, transactions, expected-type flow, and provenance.

Required invariants:

1. Every variable ID belongs to one checker instance.

1. Occurs checking visits every recursive child.

1. Failed probes restore all tables and buffered diagnostics.

1. Final output contains no unresolved non-fallback variable.

1. Every inserted conversion has a recorded reason and source span.

1. Error terms are created only after one primary diagnostic.

Test with unit cases, generated terms, and differential comparison against a simple substitution-based reference implementation.

Benchmark deeply nested pairs, many repeated variables, and many failed probes separately.

Do not optimize until each workload's cost is explained.

64. Contribution tasks#

Begin with source reading, not architectural redesign.

1. On the Rust 1.97.1 checkout, locate Ty, TyKind, GenericArg, Binder, and type flags.

1. Draw ownership and lifetime relationships among TyCtxt, interned terms, and InferCtxt.

1. Trace one _ annotation from HIR to a fresh type variable and final resolution.

1. Find snapshot creation in method lookup and list every table covered by rollback.

1. Trace an array-to-slice coercion through adjustment recording into MIR construction.

1. Trace a function-item-to-pointer coercion and verify ABI and safety checks.

1. Locate WF obligation creation for a generic field and its cause span.

1. Trace a projection normalization call only to the solver interface; stop before candidate algorithms.

1. Map one RPITIT source method to internal item identities and arguments.

1. Find const inference fallback or unresolved-const diagnostics and add a narrow UI test.

1. Add a debug assertion that catches a cross-owner typeck-result lookup.

1. Improve one cascade suppression case without suppressing an independent error.

For each task, submit a minimized regression test before or with code.

Run the narrow suite first, then relevant compiler tests according to current bootstrap documentation.

Use ./x test tests/ui/... only after confirming command syntax for that checkout.

Review concerns include cross-crate behavior, editions, feature gates, diagnostic snapshots, incremental fingerprints, and compile-time regressions.

A semantic change needs language-team or types-team context where policy is unsettled.

A refactor must prove query products and diagnostics remain equivalent under the intended observation model.

65. Derived philosophy#

Representations determine cheap questions.

HIR makes “where did the user write this?” cheap.

Interned semantic types make repeated identity and decomposition cheap.

Inference tables make “what is known now?” cheap.

No one representation makes all three cheap.

Abstractions move responsibility rather than deleting it.

Opaque types simplify an API consumer's view while moving capture, reveal, and hidden-type consistency work into the compiler and defining scope.

Interning removes repeated allocation but creates immutability, hashing, and retention obligations.

Lazy normalization avoids eager cycles but requires explicit caller discipline.

Identity is not spelling, address, or local index.

A DefId distinguishes definitions.

An interned address is session-local storage identity.

An inference index is context-local uncertainty.

Confusing them creates cache and cross-owner bugs.

Uncertainty must be represented rather than guessed away.

Inference variables, ambiguous query responses, and unevaluated consts let later evidence decide.

Fallback is an explicit policy for the cases the language chooses to default.

Optimization preserves meaning under stated observations.

Changing candidate order is harmless only if results, diagnostics where promised, and adjustments remain stable.

Caching creates correctness obligations because keys must include every semantically relevant environment and mode.

Diagnostics depend on preserved provenance.

Once expectation origin or obligation cause is discarded, no clever printer can reconstruct it reliably.

The first visible failure can occur after the first broken invariant.

A wrong argument order may surface during normalization.

A dropped adjustment may surface in MIR.

A leaked probe constraint may surface at fallback.

Contributor skill is the discipline of walking backward to the earliest divergence.

66. Mastery assessment and authoritative reading#

Prediction and implementation exercises#

1. Predict all inference domains created by let x = [0; _]; in a context where the syntax is accepted, then verify against HIR and inference logs.

1. Draw de Bruijn indices for three nested higher-ranked function signatures and perform one capture-avoiding substitution.

1. Explain why a general type variable cannot use integer fallback.

1. Design a rollback test that detects leaked region constraints.

1. Compare equality, subtyping, and coercion for &mut [u8; 4] to &[u8].

1. Explain every adjustment in a method call on Box<[T; N]> without assuming every candidate succeeds.

1. Trace function item identity through coercion to an extern "C" pointer and identify where incompatibility must be rejected.

1. Find a type that is source-valid but requires a WF obligation under a generic environment.

1. Explain why normalization results need reveal mode in their context.

1. Compare two distinct return-position opaque identities with identical bounds.

1. Map parent and local arguments of a GAT projection.

1. Separate const inference, evaluation, evaluatability, and validity for an array type.

1. Minimize a cascading diagnostic and identify the first error guarantee.

1. Enumerate the typeck products needed to lower one overloaded method call.

1. Produce a performance hypothesis with a measurable baseline before changing interning or snapshots.

Source-reading practicum#

Choose one accepted body and one minimally different rejected body.

Build the same rustc 1.97.1 revision for both.

Record the HIR owner and every relevant node ID before examining semantic output.

At type lowering, record source form, semantic term, span, and generic argument owner.

At inference-variable creation, record domain, universe, origin, and owning inference context.

At each relation, record expected term, actual term, direction, variance, and cause.

At each snapshot, record depth and the lengths of all rollback logs.

At coercion, record the source type, target type, and ordered adjustments.

At method confirmation, record the selected DefId, instantiated signature, and receiver transformation.

At normalization, record only request and response boundary data; leave candidate search to the solver study.

At finalization, list fallback decisions and unresolved variables separately.

At query publication, prove no body-local inference identity escaped.

Compare the two traces and mark the first divergence.

Do not begin from the final error wording.

Classify the divergence as representation, mechanism, policy, optimization, or presentation.

If it is representation, state which required fact was unavailable or malformed.

If it is mechanism, state the invariant violated by the operation.

If it is policy, cite the language rule or explicitly identify an unsettled choice.

If it is optimization, specify the observation under which meaning should be preserved.

If it is presentation, prove semantic products are unchanged before editing diagnostics.

Repeat with a generic wrapper around the body.

That variation tests parameter-environment and substitution boundaries.

Repeat across a crate boundary.

That variation tests stable identity, metadata, and query inputs.

Repeat after changing only source aliases.

That variation distinguishes semantic equality from diagnostic provenance.

Repeat with one additional higher-ranked binder.

That variation stresses shifting and universe visibility.

Repeat with an explicit annotation replacing inference.

That variation isolates expected-type flow and fallback.

The practicum is complete when every observed change has an owning phase and an explicit invariant.

Talk outline#

Start with source uncertainty and the notebook model.

Contrast HIR types with semantic interned terms.

Introduce three generic argument domains.

Use nested binders to explain capture and universes.

Implement unification and transactional probes.

Separate equality, subtyping, and coercion.

Trace expectations and adjustments through one method call.

Show normalization as a solver boundary without teaching solver search.

Finish with opaque abstraction, query products, and earliest-broken-invariant debugging.

Versioned authoritative sources#

Implementation claims in this chapter target the Rust 1.97.1 source line available in August 2026.

Internal modules and APIs are not stable; use the exact release source and generated rustdoc whenever details matter.

• Rust 1.97.1 source release
• rustc-dev-guide: type inference
• rustc-dev-guide: HIR type checking
• rustc-dev-guide: method lookup
• rustc-dev-guide: variance
• rustc-dev-guide: opaque types
• rustc-dev-guide: normalization
• nightly rustdoc: rustc_middle::ty
• nightly rustdoc: rustc_infer
• nightly rustdoc: rustc_hir_typeck
• Rust Reference: type coercions
• Rust Reference: subtyping and variance
• Rust Reference: function pointer types
• Rust Reference: impl Trait
• Rust Reference: generic parameters
• Rust Reference: trait associated items
• Rust Reference: dynamically sized types
• Rust Compiler Tests Guide

The Reference is the first stop for language behavior.

The matching rustc source and rustdoc are the source for current implementation structure.

The rustc-dev-guide explains architecture but can lag a moving checkout.

Historical RFCs explain motivation, not necessarily current implementation or complete current rules.

Mastery means being able to state which category supports each claim.

Part IV-C: Trait Solving as Recursive, Cached Proof Search#

Revision note. This chapter describes the in-tree next-generation solver near Rust 1.97.1. Compiler internals, query names, source locations, and migration switches are revision-sensitive. Language guarantees come from the Reference; implementation claims should be checked against the matching rustc checkout and nightly rustdoc.

67. From trait bounds to logic#

The concrete problem#

Rust permits code to state requirements without naming the implementation that will satisfy them.

fn duplicate<T: Clone>(x: T) -> (T, T) {
	(x.clone(), x)
}

While checking this body, T is unknown. The compiler must justify that T: Clone from the function's assumptions, resolve clone, and determine its output. At a call with String, it may instead use the concrete Clone for String implementation.

A goal is a proposition the solver is asked to establish. Examples include T: Clone, <I as Iterator>::Item == u8, and “this type is well formed.” A clause is a rule or fact that can help prove a goal. The environment is the set of assumptions available at the point of checking.

The bound above gives an environment clause:

for every T, while checking duplicate<T>:
    GIVEN T: Clone

An implementation gives a conditional rule:

impl<T: Clone> Clone for Box<T>

becomes, conceptually:
    if T: Clone, then Box<T>: Clone

The premise T: Clone is a nested goal. Trait solving is therefore recursive proof search rather than a flat implementation lookup.

One useful model#

Think of evaluation as a function:

(environment, goal) -> response

The response contains constraints discovered while proving the goal and a certainty. Yes means the goal follows under those constraints. Maybe means evaluation cannot currently choose or finish safely. Failure means there is no acceptable response for that branch.

The model is deliberately smaller than rustc. Real inputs also carry typing mode and query context; responses can contain region and opaque-type constraints; diagnostics preserve causes outside the pure logical key.

source bound / operation
        | lowers to
        v
   goal in ParamEnv
        | canonicalize unknowns
        v
 canonical query input -----> global evaluation cache
        | cache miss
        v
 assemble candidates -> evaluate nested goals -> merge response
        |                                      |
        +------------- proof tree -------------+

The key invariant is: a successful response is valid for every substitution represented by its canonical input, subject to the returned constraints. That is why query caching cannot key directly on one inference context's variable numbers.

Mechanism, policy, presentation#

Keep three layers separate.

Changing an error message must not silently change proof policy. Changing cycle policy must not be disguised as a cache optimization.

First trace#

Given impl<T: Copy> Copy for Pair<T>, prove Pair<u8>: Copy.

G0: Pair<u8>: Copy
  candidate: impl<T: Copy> Copy for Pair<T>
  unify Pair<?T> with Pair<u8>       => ?T = u8
  nested G1: u8: Copy
    candidate: built-in/library impl => Yes
  all nested goals Yes               => G0 Yes

If ?X replaces u8, matching can infer ?X only when that inference is justified by the surrounding relation. The solver must return the assignment; mutating a hidden global inference table would make caching unsound.

Prediction exercise#

Suppose both impl<T> Marker for Vec<T> and an environment assumption Vec<X>: Marker apply. Should candidate assembly stop after seeing the environment candidate?

It should not in general. Assembly finds plausible proof routes; evaluation and candidate merging decide whether their results agree. Prematurely choosing by discovery order makes behavior depend on indexing and refactoring.

68. Terms, predicates, binders, and environments#

Terms and predicates#

The solver relates types, regions, and consts, collectively called generic arguments or terms in this chapter. A predicate wraps a proposition over terms. Important predicate families include:

• trait predicates such as T: Send;
• projection or alias relations involving <T as Trait>::Assoc;
• type, region, and const outlives predicates;
• well-formedness predicates;
• subtype/coercion-related goals at integration boundaries;
• const evaluatability and host-effect predicates in relevant modes.

The exact internal predicate and goal enums change. Do not infer a stable language taxonomy from one rustc_middle::ty enum.

Clauses are implications#

An impl is not merely a database row.

trait Render {
	type Error;
	fn render(&self) -> Result<(), Self::Error>;
}

impl<T> Render for Vec<T>
where
	T: Render<Error = std::io::Error>,
{
	type Error = std::io::Error;
	fn render(&self) -> Result<(), Self::Error> { Ok(()) }
}

Conceptually it contributes both a trait rule and associated-type information:

T: Render
<T as Render>::Error = io::Error
--------------------------------
Vec<T>: Render
<Vec<T> as Render>::Error = io::Error

This notation is explanatory, not rustc's stored layout. Elaboration may derive consequences from supertraits and associated-type bounds. The environment must preserve which assumptions are actually in scope; globally assuming every syntactically nearby bound would accept invalid programs.

Parameter environments#

A ParamEnv packages caller bounds and mode information relevant to generic checking. Inside fn f<T: Ord>(...), T: Ord is usable even though no concrete impl can be selected. Supertrait elaboration can make T: Eq available because Ord: Eq.

Environment candidates differ from impl candidates:

environment candidate: the caller promised this proposition
impl candidate: a globally coherent implementation establishes it

Confusing them harms both diagnostics and specialization.

Binders#

for<'a> Fn(&'a str) does not mean “there exists some convenient lifetime.” It means the property holds for every lifetime selected by the caller. A binder records variables quantified by for<...>.

To test a universally quantified requirement, rustc replaces each bound variable with a fresh placeholder in a fresh universe. An inference variable created outside that universe cannot later contain the placeholder. This is leak prevention.

outer universe U0: inference variable ?T
        |
        | enter for<'a>
        v
inner universe U1: rigid placeholder '!a

forbidden result: ?T := &'!a u8
reason: a value escaping U1 would mention a lifetime chosen only inside U1

A leak counterexample#

If placeholders behaved like ordinary inference variables, a solver could “prove” a higher-ranked requirement using one specially chosen lifetime.

fn needs_any<F>(_: F)
where
	F: for<'a> Fn(&'a u8),
{}

The caller may invoke F with short, unrelated lifetimes. Proving only Fn(&'static u8) is insufficient. Universe checks prevent existential choice from masquerading as universal validity.

Bound-variable hygiene#

Bound variables are usually represented by position under binders, not source names. Substitution must shift indices when crossing binders. Canonicalization must retain universe relationships. Pretty-printing may invent friendly names, but names are presentation, not identity.

Earliest broken invariant: if a placeholder appears in a response visible to an older universe, the bug occurred during instantiation, relation, or response canonicalization—not when a later region error happens to expose it.

69. Inference and canonical query boundaries#

Why inference cannot be the cache key#

An inference context contains mutable variables such as ?T = unknown. Variable index 7 in one function has no relationship to index 7 in another. Caching Goal<?7> globally would conflate unrelated states.

Canonicalization replaces free inference variables with numbered canonical variables and records each variable's kind, universe, and constraints needed by the query.

local input:  (?17, Vec<?23>): Relate
canonical:    (^0, Vec<^1>): Relate
variables:    [^0: type in U0, ^1: type in U0]

Two alpha-equivalent local inputs then share a cache key. “Alpha-equivalent” means they differ only in irrelevant variable names.

Input and response#

A canonical input contains the canonical variable metadata plus the canonicalized goal and environment. Evaluation occurs with fresh local inference variables instantiated from those canonical variables. The canonical response describes what was learned in terms of the canonical input variables and any response variables.

input:  exists ^0. Vec<^0>: Iterator<Item = u8>
result: ^0 = some constrained term, certainty = Yes

The exact rustc response structure is richer than this notation. It can carry external constraints which the caller must instantiate and apply to its own inference context.

Applying a response#

The caller must:

1. map canonical input variables back to the original local variables;
2. create local variables for response-only unknowns;
3. relate returned values to the caller's values;
4. register returned region, opaque, and nested obligations as required;
5. reject universe-invalid assignments;
6. preserve certainty instead of treating ambiguity as proof.

Application can fail because the caller changed since canonicalization or because returned constraints conflict. A cached response is not permission to overwrite inference state.

Snapshots and probes#

Candidate evaluation is speculative. Trying one impl may unify variables before a nested goal fails. An inference snapshot permits rollback.

snapshot
  instantiate candidate
  relate headers
  evaluate nested goals
  if candidate fails: rollback
  if retained: package constraints, then rollback local speculation

The response is later applied through the query boundary. Without rollback, candidate order changes the answer.

Freshening versus canonicalization#

Freshening erases unstable local inference identities into stable placeholders useful for comparison or a local cache. Canonicalization additionally creates a quantified query interface with enough metadata to instantiate a reusable response. They solve related but different problems.

A freshened key may say “some unknown type occurs twice” while intentionally forgetting its mutable assignment. A canonical response must still explain constraints on that repeated unknown. Production rustc's exact uses of freshening have evolved; inspect the current inference and solver caches before assuming old documentation still applies.

Prediction exercise#

Can canonicalization replace a placeholder from universe U2 with an ordinary canonical variable in U0?

No. That forgets rigidity and visibility. The cache might return a solution that lets an outer variable capture an inner placeholder.

70. Build a minimal stable-Rust solver#

The first program intentionally handles only ground unary traits and Horn-like impls. It compiles on stable Rust without dependencies. Its purpose is to make recursion visible before adding inference and cycles.

use std::collections::HashSet;

#[derive(Clone, Debug, PartialEq, Eq, Hash)]
struct Goal {
	trait_name: &'static str,
	type_name: &'static str,
}

#[derive(Clone, Debug)]
struct Rule {
	head: Goal,
	premises: Vec<Goal>,
}

fn prove(goal: &Goal, facts: &HashSet<Goal>, rules: &[Rule]) -> bool {
	if facts.contains(goal) {
		return true;
	}
	rules.iter().filter(|r| &r.head == goal).any(|rule| {
		rule.premises.iter().all(|premise| prove(premise, facts, rules))
	})
}

fn main() {
	let copy_u8 = Goal { trait_name: "Copy", type_name: "u8" };
	let copy_pair = Goal { trait_name: "Copy", type_name: "Pair<u8>" };
	let facts = HashSet::from([copy_u8.clone()]);
	let rules = vec![Rule {
		head: copy_pair.clone(),
		premises: vec![copy_u8],
	}];
	assert!(prove(&copy_pair, &facts, &rules));
}

Run it with rustc solver0.rs && ./solver0.

What it teaches#

Facts model environment or unconditional impl candidates. Rules model candidates with nested goals. all is conjunction: every where-clause must hold. any is alternative proof search: one applicable rule can suffice.

What it dangerously omits#

• Variables and unification.
• Candidate overlap and equivalent responses.
• Associated types and normalization.
• Binders, universes, regions, and consts.
• Recursion detection: A if A overflows the host stack.
• Ambiguity: false conflates contradiction, no candidate, and “not known yet.”
• Coherence and negative reasoning.
• Resource limits and diagnostics.

The omissions matter more than the code. An educational solver becomes misleading when it silently labels “not proved by my tiny search” as “false in Rust.”

Exercise: expose the cycle#

Add the rule Loop(X) :- Loop(X) as a ground rule. Predict the result before running. The program recurses forever or stack-overflows because it has no search graph. Adding a visiting set prevents overflow, but returning false on every repeated goal is also wrong for restricted coinductive goals such as auto traits.

71. A richer educational solver#

We now design, rather than dump, a solver with variables, environments, canonical keys, and tabling. Its syntax is intentionally smaller than Rust.

Type  := Atom(name) | Apply(name, Type...) | Var(id) | Placeholder(universe, id)
Goal  := Implemented(Type, Trait) | Normalize(Alias, Type) | Equal(Type, Type)
Rule  := forall variables. Goal :- Goal...
Env   := Goal...

Unification#

Unification seeks a substitution making two terms equal.

unify Vec<?0> with Vec<u8>
  decompose equal constructors
  unify ?0 with u8
  substitution {?0 -> u8}

An occurs check rejects ?0 = Vec<?0> because applying that substitution would create an infinite type. A universe check rejects assigning an outer variable a term containing an invisible placeholder.

unify(a, b):
    a = shallow_resolve(a)
    b = shallow_resolve(b)
    if identical: succeed
    if a is variable: bind_after_occurs_and_universe_checks(a, b)
    if b is variable: bind_after_occurs_and_universe_checks(b, a)
    if matching constructors: unify corresponding children
    otherwise: fail

This is pseudocode. Rustc relations are mode-sensitive and include subtyping, variance, regions, consts, aliases, and delayed obligations.

Freshening#

For a local table key, walk the resolved goal and replace each unbound inference variable consistently:

Foo<?17, ?9, ?17> -> Foo<F0, F1, F0>

The repeated F0 preserves equality information. Do not replace each occurrence independently. Resolved variables are replaced by their values before freshening. Placeholders retain universe identity.

Canonicalization#

Canonicalization similarly numbers unknowns but stores variable metadata and creates a response protocol.

canonicalize(goal, env):
    resolve what inference already knows
    replace each free infer variable by one canonical variable
    record kind and universe
    canonicalize relevant environment assumptions
    return Canonical { variables, value: (env, goal) }

The environment belongs in the key. T: Clone is provable in one generic function and not another.

Evaluation context#

An EvalCtxt-style evaluator owns one recursive proof attempt:

evaluate(goal):
    check depth/fuel
    canonicalize input
    consult search graph/table
    assemble candidates by predicate kind
    evaluate each candidate in a probe
    recursively evaluate candidate nested goals
    merge compatible candidate responses
    canonicalize response

Production names and ownership boundaries vary, but this shape is central. The context coordinates inference, recursion, candidate evaluation, certainty, and proof-tree recording.

Candidate assembly by predicate#

There is no single “scan all impls” algorithm.

Assembly should cheaply reject impossible candidates but must not perform irreversible inference. Predicate-specific assembly preserves rules that a generic Horn-clause engine would obscure.

Nested-goal discipline#

A candidate is only as strong as its premises. Evaluate nested goals under the candidate's instantiated substitution and environment. Combine certainties monotonically: one ambiguous required premise makes the candidate ambiguous unless another route yields an equivalent certain result. Do not discard an ambiguous branch merely because a currently failing branch looks more concrete.

72. Candidate merging and certainty#

More than true and false#

The solver must distinguish:

Internal names and overflow propagation evolve. The semantic distinction matters: ambiguity is not negative proof, and overflow is not permission to accept.

Why ambiguity is useful#

For ?T: Iterator, many impls might apply. Rejecting immediately would make inference order brittle. The caller can add constraints, then retry or finalize obligations later.

For Vec<?T>: Clone, a unique generic impl may reduce the question to ?T: Clone without choosing ?T. Unknown input does not automatically imply ambiguity; what matters is whether all valid proof routes induce one compatible response.

Merging#

Suppose two candidates both prove a goal but return substitutions.

candidate A: ?T = u8, Yes
candidate B: ?T = u8, Yes
merge:       ?T = u8, Yes

candidate A: ?T = u8, Yes
candidate B: ?T = u16, Yes
merge:       ambiguous unless language policy orders/excludes one

Candidate identity alone is not the answer. Equivalent responses may be merged even if obtained by different routes. Specialization and coherence can permit ordering in carefully defined contexts; ordinary evaluation must not invent “first impl wins.”

Overflow#

Depth and graph limits protect compiler resources and bound pathological recursion. An overflow-tainted answer must remain distinguishable from ordinary ambiguity where diagnostics and callers care. Increasing a recursion limit may hide a compiler performance defect or merely move a stack failure.

Useful overflow diagnostics include the repeating goal shape, depth, cache behavior, and whether terms grow:

T: Trait
Vec<T>: Trait
Vec<Vec<T>>: Trait
...

That is not a simple cycle because each goal is larger. Exact-key cycle detection cannot terminate it; resource limits are still necessary.

Prediction exercise#

If one candidate succeeds and another overflows, may the solver always return the success?

No universal rule is safe. If the overflowing branch could yield a conflicting response, ignoring it is unsound. Candidate merging and language policy determine when a success dominates; the evaluator must preserve enough uncertainty to decide conservatively.

73. Search graphs, tables, and cycles#

From recursion stack to graph#

Recursive evaluation revisits canonical goals. The active stack plus completed and provisional table entries forms a search graph.

G0: A: Send
 | field b
 v
G1: B: Send
 | field a
 +--------------------+
                      |
          back-edge to G0

A cache of completed answers saves repeated work. A provisional entry represents an active evaluation whose final answer is not yet known. Encountering it is a cycle, not an ordinary cache hit.

Why provisional results need care#

If a provisional Yes escapes before its strongly connected component is validated, a self-supporting proof can prove anything shaped like its cycle. If every cycle returns No, valid recursive auto-trait reasoning fails.

Cycle policy is logical semantics, not just termination machinery.

Induction and coinduction#

Ordinary inductive reasoning requires a finite proof grounded in facts.

trait Bad {}
impl<T: Bad> Bad for T {}

u8: Bad must not become true merely because proving it asks for itself.

Coinductive reasoning can accept a cycle as a provisional hypothesis for designated predicates. Auto traits use restricted coinduction because recursive data structures should be Send when all reachable components satisfy Send and no negative fact blocks them.

struct Node {
	next: Option<Box<Node>>,
}

Proving Node: Send structurally reaches Node: Send again through Box and Option. The cycle is acceptable only within the solver's designated coinductive domain and only if every noncyclic obligation succeeds.

Mixed cycles#

A strongly connected component may contain both coinductive and inductive goals. It is unsafe to bless the whole component because one node is an auto trait. Restricted coinduction tracks the cycle participants and rejects or delays cycles crossing disallowed predicates.

Fixpoint iteration#

Recursive tables can begin with a conservative provisional answer, evaluate dependencies, and iterate until answers stop changing. That stable state is a fixpoint.

round 0: assume provisional answer according to cycle policy
round 1: evaluate candidates using current table
round 2: re-evaluate dependents whose answer changed
stop:    no answer changes, or resource limit reached

The answer ordering must be monotone for this argument. If evaluation oscillates between incompatible substitutions, the abstraction or merge policy is wrong. Rustc's concrete search-graph algorithm should be read in current source; this fixpoint account is a learning model, not a claim about every data structure.

Cache invariants#

• Keys include every proof-relevant mode and environment component.
• Responses contain no local inference identities from the producing context.
• Provisional entries do not leak as final global answers.
• Cycle participants are invalidated or recomputed when assumptions change.
• An overflow answer is cached only at a scope where its depth/fuel meaning remains valid.
• Diagnostic recording does not alter logical results.

Debugging workshop#

Symptom: adding an unrelated bound changes whether a recursive goal succeeds.

1. Print canonical keys and environment fingerprints.
2. Check whether two distinct environments collide.
3. Record provisional-entry creation and completion.
4. Identify the strongly connected component.
5. Verify every edge's predicate kind and coinductive eligibility.
6. Disable only the suspect cache locally; if behavior changes, inspect key completeness before blaming candidate assembly.

74. Aliases, projection, and normalization#

Why aliases remain symbolic#

I::Item cannot always be reduced when I is unknown. Keeping <I as Iterator>::Item as an alias preserves the question until an impl or environment equality supplies an answer.

Normalization relates an alias to its underlying term where the current environment and reveal policy justify doing so.

given impl Iterator for Bytes { type Item = u8; }

Normalize(<Bytes as Iterator>::Item -> ?X)
  select Iterator for Bytes
  read associated type value u8
  relate ?X with u8
  response ?X = u8

Projection goals#

Older descriptions often speak of “projection equality.” The next solver commonly reasons through alias terms, normalization, and alias-relate goals. Exact goal variants are revision-sensitive. The conceptual obligations remain:

1. establish which trait implementation or assumption controls the associated item;
2. instantiate its generic arguments correctly;
3. satisfy associated item and impl bounds;
4. relate the normalized value to the expected term;
5. avoid revealing opaque identities in forbidden contexts.

Alias-relate#

Eagerly normalizing both sides can loop or lose useful directionality. An alias-relate goal says two terms must denote compatible values while allowing the solver to choose a justified normalization route.

Relate(<T as A>::X, <T as B>::Y)

Possible routes include normalizing the left, normalizing the right, or using an environment equality. If multiple routes produce incompatible responses, the result is ambiguous or erroneous; source order is not policy.

Normalization cycles#

trait Loop { type X; }
// A hypothetical set of aliases can recursively demand normalization.

Projection cycles are not automatically coinductive. Accepting <T as Loop>::X = <T as Loop>::X does not discover a concrete type needed by layout or method checking. The caller's observation matters: proving a reflexive relation can require less information than producing a normalized type.

Associated type bounds#

trait Streaming { type Item: Clone; } contributes a requirement on the normalized associated type. When an impl chooses type Item = X, well-formedness checks must establish X: Clone in the proper environment. Using the bound can also generate nested goals after projection.

GATs#

A generic associated type has its own binders:

trait Lending {
	type Item<'a>
	where
		Self: 'a;

	fn item<'a>(&'a self) -> Self::Item<'a>;
}

Normalization must instantiate the associated item's lifetime arguments, preserve the Self: 'a requirement, and respect higher-ranked use sites. Treating Item<'a> as a non-generic slot loses both binder scope and required bounds.

RPITIT and opaques#

Return-position impl Trait in traits is represented through compiler-generated opaque/associated machinery whose details evolve. The hidden type is constrained in a defining context and abstract elsewhere. Solver reveal modes determine when an opaque may be related to its hidden type.

An opaque alias is not “just its hidden type with privacy.” Its identity supports abstraction, borrow checking, auto-trait computation, and incremental compilation. Revealing it too early can accept dependencies users are not allowed to rely on; never revealing it can prevent legitimate defining-use checks.

75. Coherence and global consistency#

Selection is not coherence#

Selection asks whether a goal has an applicable proof in one environment. Coherence asks whether the global impl set remains unambiguous under Rust's compatibility rules. The latter must account for types and impls downstream crates may legally add.

Orphan rules#

Orphan rules constrain which crate may implement which trait for which type. Their purpose is global ownership of implementation choices: two unrelated downstream crates must not both be able to add conflicting impls that a third crate combines.

The full rule includes local traits, local types, fundamental type constructors, and ordering/coverage of type parameters. Use the Reference and current coherence source for exact behavior; slogans such as “trait or type must be local” omit important details.

Overlap#

Two impls overlap if there exists a substitution under which both can apply, considering satisfiable where-clauses and permitted assumptions.

trait Mark {}
impl<T> Mark for Vec<T> {}
// impl<T: Copy> Mark for Vec<T> {} // overlaps the first

The second header is a subset of the first. Without an accepted specialization relationship, allowing both would make selection policy-dependent.

Overlap checking uses solver-like reasoning but under a special environment and negative-reasoning policy. It must not assume a downstream trait impl is impossible merely because none exists today.

Open-world negative reasoning#

Failure to prove T: Trait is not proof of T: !Trait. Another crate may add an impl when orphan rules permit it, or T may remain an inference variable.

Negative reasoning is valid only when justified by an explicit negative impl, a closed-world property established by coherence, or another language-defined rule in the current mode. This is one of the most important soundness boundaries in trait solving.

Specialization#

Specialization permits certain overlapping impls with an ordering from less specific to more specific. It remains unstable and has subtle soundness constraints. The solver may need to know that one candidate specializes another, but ordinary candidate merging must not recreate specialization accidentally.

Associated items complicate the ordering: a selected trait impl and the impl providing one specialized associated item may involve inheritance/defaulting rules. Always inspect current specialization graph code and feature restrictions rather than extrapolating from a toy example.

Counterexample#

Bad rule: “if no positive impl candidate is assembled, answer false.”

For a foreign generic T, downstream code or caller assumptions may establish the trait. The correct result can be ambiguity or no current proof, not a stable negative fact. Conflating these would make adding a legal impl a breaking semantic contradiction in already compiled generic code.

76. Auto traits and structural reasoning#

Auto traits such as Send and Sync can be inferred structurally when no explicit rule overrides that inference. The solver examines constituent types and creates nested auto-trait goals.

struct Envelope<T> {
	value: T,
}

Conceptually, Envelope<T>: Send requires T: Send. Real structural computation must account for fields, closures, generators/coroutines, raw pointers, references, phantom data, explicit impls, and compiler-defined types.

Negative impls#

An explicit negative auto-trait impl can block structural derivation. Negative impl support and stability differ by trait and feature; do not generalize from Send/Sync to every user trait.

Recursive types#

Recursive type definitions motivate coinduction, but indirection and well-formedness still matter. The solver is not proving that an infinitely sized value can exist. Layout separately rejects unboxed size recursion.

mechanism: structural auto-trait candidate follows fields
policy: designated auto-trait cycles may be coinductive
separate check: layout requires finite representable size

Unsafe significance#

Incorrectly proving Send or Sync can make safe code trigger data races through unsafe library implementations that rely on those contracts. Auto-trait solver bugs are therefore potential soundness bugs, not merely type-checking inconveniences. Tests should include explicit negative constituents, recursive structures, higher-ranked fields, opaques, and dyn objects.

77. Trait objects and dyn#

dyn Trait is a dynamically sized type carrying a data pointer and metadata used for dynamic dispatch. Only traits satisfying dyn-compatibility rules can form trait objects. The Reference uses dyn compatibility; older material often says “object safety.”

Object candidates#

A dyn Principal + Auto + 'a type carries bounds that can justify some trait goals. Supertraits of the principal trait may also be available. The solver must distinguish what follows from the object's existential erased type from what is callable through its vtable.

dyn Draw + Send
  | existential hidden implementer X
  | evidence X: Draw + Send
  v
object candidate for Draw / supertraits / listed auto traits

Associated types on objects#

dyn Iterator<Item = u8> fixes an associated type and can support projection normalization. Leaving required associated types unspecified may make the object type invalid or unusable for the intended operation. GATs and methods with generic parameters introduce additional dyn-compatibility restrictions.

Upcasting and unsizing#

Conversions from a concrete type to dyn Trait, and trait-object upcasts to supertraits, involve unsizing/coercion logic plus trait obligations. Do not reduce them to a normal user impl lookup. Metadata and vtable construction are later code-generation concerns, while the solver establishes permitted relationships.

Prediction exercise#

Does knowing dyn Draw: Draw reveal the concrete implementation type?

No. The object deliberately erases it. Representation makes dispatch cheap enough while making concrete-type-dependent questions unavailable without another mechanism such as Any and a runtime check.

78. Proof trees and diagnostics#

Logical answers are too compressed#

A canonical response may say only “ambiguous.” A useful diagnostic needs to know which candidates were tried, which nested goal first failed, and where the obligation originated. A proof tree records evaluation structure for inspection without making source spans part of the reusable logical cache key.

Goal: Wrapper<X>: Display
  ImplCandidate: impl<T: Debug> Display for Wrapper<T>
    Nested: X: Debug
      No candidates
  Result: NoSolution

Obligation causes#

The solver's goal says what must hold. The type checker tracks why it must hold: method call, cast, item bound, derived impl, return type, and so on. Diagnostics join the logical proof failure with this source-level cause chain.

If spans were embedded in global cache keys, identical logic at different source locations would miss the cache. If causes were discarded entirely, errors would point only at generic internals. The architecture preserves provenance beside, not inside, the pure proposition.

Inspecting without perturbing#

Diagnostic evaluation must avoid changing inference results by rerunning candidates in a different mode. Proof-tree recording should be observational: enabling it may cost time and memory but should not change candidate ordering or certainty. Where rustc performs a diagnostic rerun, compare its environment and mode carefully.

Find the earliest broken invariant#

A concrete trace with ambiguity#

source: values.collect::<C>()
goal: C: FromIterator<Item>
environment: caller bounds

candidate assembly:
  impl FromIterator<char> for String       (possible if Item = char, C = String)
  impl<T> FromIterator<T> for Vec<T>        (possible if C = Vec<Item>)
response merge:
  incompatible assignments for unknown C
result:
  Maybe(Ambiguity)
final obligation:
  request a type annotation for C

The diagnostic should explain the unconstrained collection target, not claim either impl is broken.

79. Old solver, next solver, and migration#

Rustc historically used a selection-oriented trait solver often called the old or legacy solver. The in-tree next-generation solver centralizes canonical goal evaluation, candidate merging, alias relations, and search-graph handling more systematically.

This is not simply “old algorithm slow, new algorithm fast.” The migration changes architecture and sometimes exposes old accidental behavior, incompleteness, or diagnostics differences.

Broad comparison#

This table is architectural, not an API contract. Both implementations contain exceptions and shared infrastructure.

Migration discipline#

During migration, compare observable language behavior, inference constraints, and diagnostics—not merely whether both return “success.” Useful differential cases include:

• success with the same inferred substitutions;
• ambiguity versus hard failure;
• normalization result and reveal mode;
• overflow classification;
• coherence overlap conclusions;
• higher-ranked leak behavior;
• proof-tree cause quality.

Compiler flags and default solver modes change across nightlies. For 1.97.1-era work, discover the supported options with that compiler's rustc -Z help rather than copying a stale -Ztrait-solver=... command. -Z options require nightly and are not stable user interfaces.

Compatibility is asymmetric#

Code accepted only because of an old solver bug may need a future-compatibility path. Code rejected due to old incompleteness may begin compiling. Neither difference should be waved away as implementation detail without language-team and compatibility review.

80. Chalk: influence, not dependency folklore#

Chalk explored modeling Rust traits as logic, with program clauses, canonical goals, tabling, universes, and an SLG-style solver. It gave the Rust project a vocabulary and an executable laboratory for ideas that influenced rustc's solver design.

Rustc's current next solver is in-tree and specialized for rustc integration. It is not accurate to say modern rustc simply lowers every goal to the Chalk crate and asks Chalk for an answer. The standalone Chalk project was sunset/archived as active solver integration moved into rustc; historical Chalk books and repositories remain valuable design evidence, not current normative documentation.

What transferred conceptually#

• Canonical queries separate inference contexts from reusable solving.
• Universes model higher-ranked visibility.
• Program clauses expose implication structure.
• Tabling makes recursion and repeated subgoals explicit.
• Answers may carry substitutions and ambiguity rather than booleans.

What cannot be copied blindly#

Rustc has legacy compatibility, diagnostics, region inference, opaque types, coherence modes, incremental queries, and performance constraints tied to its own type representation. A mathematically elegant general solver may allocate too much, erase source provenance, or fail to match Rust's evolving policy.

Historical documents should be labeled by date and role. Use them to understand motivation; use current rustc source and official language documentation to claim behavior.

81. Production performance and soundness#

A cost model#

Let C be assembled candidates, D recursive depth, G distinct canonical goals, and R average response size. Naive backtracking can approach exponential work in D when branches repeat. Tabling aims to make repeated exact goals closer to reuse over G, but unification, environment size, growing terms, and multiple answers still dominate real workloads.

Canonicalization costs a walk over the input. Caching pays when evaluation saved exceeds hashing, allocation, and response application. Large environments can make keys expensive and reduce hit rates. Measure rather than asserting that “more caching is faster.”

Representative measurement#

Benchmark:

1. clean check and incremental check;
2. successful, ambiguous, and failing crates;
3. deeply generic iterator/future code;
4. projection-heavy GAT and async code;
5. auto-trait recursion;
6. coherence workloads with many impls;
7. pathological generated goals under limits.

Record candidate counts, canonicalization time, cache hit rate, cycle count, maximum depth, proof-tree overhead, and allocations. Wall-clock time without these counters cannot distinguish fewer goals from cheaper goals.

Indexing#

Impl indexing should reject impossible self-type heads cheaply. An index keyed on Vec<_> can avoid scanning impls for unrelated primitive heads. Unknown self types necessarily broaden search. Indexing is an optimization and must never omit a candidate that semantic assembly would accept.

Determinism#

Hash iteration order must not select a winner among ambiguous candidates. Responses should merge independently of discovery order. Deterministic proof trees and stable diagnostics are valuable even when logical results match.

Resource controls#

Production evaluation needs recursion/fuel limits, cancellation checks, bounded diagnostic recording, and care against adversarial macro-generated terms. Compiler denial-of-service is a practical security concern. An overflow should terminate predictably and preserve enough context for diagnosis.

Soundness boundaries#

High-risk areas include:

• universe leaks in higher-ranked goals;
• unsound negative reasoning under the open world;
• accepting inductive cycles;
• cache keys missing reveal or coherence mode;
• opaque hidden types escaping defining scope;
• candidate probes leaking inference mutations;
• auto-trait mistakes affecting unsafe code assumptions;
• specialization overlap accepted without a valid ordering.

Every optimization needs an observation model: which answers, constraints, certainties, and diagnostics must remain unchanged? “Same boolean result” is too weak.

82. Testing the solver#

Layers of tests#

Properties worth checking#

• Renaming inference variables does not change a canonical result.
• Applying a valid response then reevaluating does not contradict it.
• Candidate permutation does not change the merged logical answer.
• Entering a universe never lets older variables capture its placeholders.
• Disabling a sound completed-result cache changes performance, not answers.
• Adding an environment premise cannot invalidate a proof solely using unchanged premises, except where explicit language modes make the context different.
• A coinductive success becomes failure if a reachable non-coinductive or negative obligation fails.

Regression minimization#

Keep the logical shape while deleting syntax. Reduce unrelated methods, modules, and lifetimes last if they affect binder depth. Record solver mode, toolchain commit, crate edition, feature gates, and recursion limit. A five-line test without the original mode is not a reproducible regression.

Compile-checking examples#

Complete examples in this chapter use stable syntax and should be extracted by the book's fence checker or copied to temporary .rs files and compiled with the matching stable toolchain. Fragments labeled conceptual, hypothetical, or pseudocode must not be treated as independent programs. Negative examples should be checked for the intended earliest error, not merely any nonzero exit status.

83. Reading current rustc source#

Paths below are signposts near the 1.97.1 development era, not stable module contracts. Search symbols in the exact checkout before editing.

• compiler/rustc_next_trait_solver/ — core in-tree next-solver implementation, including evaluation, candidates, search graph, and solve/delegate layers.
• compiler/rustc_next_trait_solver/src/solve/ — goal evaluation and predicate-specific candidate logic in contemporary layouts.
• compiler/rustc_infer/ — inference contexts, canonicalization/application support, relations, snapshots, and region interactions.
• compiler/rustc_middle/src/ty/ — interned types, predicates, clauses, canonical data, ParamEnv, and query-facing representations.
• compiler/rustc_trait_selection/ — integration, legacy solver, normalization/selection consumers, coherence-related support, and migration seams.
• compiler/rustc_hir_analysis/ — type checking and coherence entry points that create obligations.
• compiler/rustc_type_ir/ — solver-generic type IR traits and shared abstractions used to decouple core logic from one interner.
• tests/ui/traits/, tests/ui/associated-types/, and solver-specific revisions — behavior and diagnostics.
• tests/crashes/ — minimized historical ICE/overflow cases, useful but not a complete specification.

Directory ownership moves. Use rg 'struct EvalCtxt|enum Certainty|SearchGraph|assemble.*candidate|AliasRelate' compiler/ rather than relying only on paths.

A source-reading route#

1. Find the public solve_goal-like entry point.
2. Identify canonical input construction and response application.
3. Follow one simple trait goal to candidate assembly.
4. Follow one impl candidate through header relation and nested goals.
5. Find response merging and certainty combination.
6. Locate search-graph entry, provisional cache handling, and cycle policy.
7. Trace a projection/alias-relate goal separately.
8. Find proof-tree recording and its diagnostic consumer.
9. Read tests changed by the same commit.

Draw the call graph yourself. Function names are easier to remember after you know which invariant each boundary owns.

Current-source verification#

For exact 1.97.1 behavior, pin the rustc commit represented by that release/toolchain, read its submodule/compiler source, and use generated rustdoc from that revision. The stable Reference describes language rules but intentionally omits many solver internals. Nightly rustdoc describes current implementation APIs but is not a stability promise.

84. Contributor workflow#

Start from one observable discrepancy#

Write a minimized UI test before changing the solver. State expected certainty, substitutions, and candidate route. If comparing engines, capture both outputs and confirm the difference is not only diagnostics ordering.

Instrument the proof#

Use rustc's current tracing facilities and solver-specific dump flags discovered from the checkout. Limit logs to a crate, item, or goal where possible. Capture:

canonical input
candidate list and rejection reasons
nested-goal edges
inference snapshot boundaries
canonical response
cache/provisional transitions
certainty and overflow source

Avoid committing broad debug prints. Tracing in hot recursion can distort performance enough to hide races in cancellation or expose different limits.

Make the smallest invariant repair#

If an impl is absent, fix candidate indexing/assembly rather than weakening merge policy. If a placeholder leaks, fix universe preservation rather than rejecting the final type by string pattern. If a cache collides, add the omitted semantic input rather than disabling all caching.

Validate broadly#

Run the focused UI test, relevant solver test suite, tidy/format checks, and compiler tests required by rustc's contributor guide. For semantic migration changes, run legacy/next revisions and request crater where appropriate. For hot-path changes, collect perf results with query counters and representative benchmarks.

Review checklist#

• Which invariant changes?
• Is behavior normative, compatibility policy, or implementation strategy?
• Does candidate order matter after the patch?
• Are canonical cache keys complete?
• Can a response mention a forbidden universe?
• What happens on a cycle and on growing recursion?
• Does ambiguity remain distinct from no solution?
• Are coherence and normal evaluation modes both tested?
• Is proof-tree output still observational?
• What ecosystem code could rely on old behavior?

85. A bounded solver capstone#

Build a small solver in milestones rather than copying rustc. Each milestone establishes one testable invariant.

Milestone 1: ground rules#

Implement the program from Chapter 70. Add an explicit depth limit and an outcome enum Yes | No | Overflow. Invariant: evaluation always terminates within the configured resource bound.

Milestone 2: terms and unification#

Add Atom, Apply, and inference Var. Implement shallow resolution, occurs checks, snapshots, and rollback. Invariant: a failed probe leaves the substitution exactly unchanged.

Test Pair<?X> against Pair<u8> and reject ?X = Vec<?X>.

Milestone 3: environments#

Add environment facts to every evaluation input. Include a canonicalized environment fingerprint in table keys. Invariant: no proof uses a premise outside its environment.

Regression: evaluate T: Clone under two environments and ensure cached success does not cross into the empty one.

Milestone 4: certainty#

Replace boolean failure with Yes, Maybe(Ambiguous), Maybe(Overflow), and NoSolution at branch boundaries. Define combination tables for conjunction and alternative candidates. Invariant: insufficient information never becomes a negative fact.

Milestone 5: freshening and canonicalization#

Freshen local table keys consistently. Then implement canonical input metadata and response substitutions. Property-test alpha-renaming of variables. Invariant: cached responses contain no producer-local variable IDs.

Milestone 6: tabled search graph#

Store active provisional and completed entries. Detect back-edges and compute strongly connected participants or an equivalent dependency structure. Begin with inductive cycle rejection. Invariant: P :- P cannot prove P.

Milestone 7: restricted coinduction#

Mark only an Auto predicate family coinductive. Allow a pure auto-goal cycle provisionally, then validate all outgoing noncyclic premises. Invariant: introducing any inductive edge or explicit negative fact prevents unsupported cyclic success.

Milestone 8: aliases#

Add Alias(name, args) and Normalize(alias, target). Store associated values on rules. Detect normalization cycles and distinguish “related symbolically” from “produced a concrete normal form.” Invariant: normalization never invents an associated value without a controlling candidate or environment equality.

Milestone 9: universes#

Add placeholders and universe indices. Instantiate forall goals in a fresh universe. Reject bindings whose term contains an invisible placeholder. Invariant: values cannot escape the binder that introduced their rigid placeholders.

Milestone 10: proof trees#

Record candidate and nested-goal events behind an optional observer. Run all semantic tests with recording on and off. Invariant: observation does not change the answer.

Explicit production omissions#

The capstone is not a Rust trait solver. It omits regions and borrow checking, variance, const generics, dyn compatibility, unsizing, opaque reveal modes, specialization, orphan rules, incremental compilation, interning, cancellation, parallel queries, stable diagnostics, and decades of compatibility behavior.

Do not use it to decide whether real Rust code is sound. Its value is that every omitted mechanism now has a named reason to exist.

86. Experiments and debugging practice#

Experiment A: cache effectiveness#

Generate a diamond-shaped rule graph where many paths reach the same canonical goal. Measure evaluations with no cache, completed cache, and tabled provisional entries. Predict which graph width causes caching overhead to pay for itself. Report distinct goals, candidate probes, allocations, and time.

Experiment B: environment collision#

Deliberately omit the environment from the cache key. First prove X: Show with that assumption, then query under an empty environment. Observe the false cache hit. Restore the key and write a regression test.

Experiment C: universe leak#

Implement higher-ranked instantiation but temporarily disable the visibility check. Construct a response assigning an outer variable a type containing an inner placeholder. Explain why the apparent solution depends on a caller-chosen lifetime and cannot escape.

Experiment D: mixed cycle#

Create:

Auto(A) :- Ordinary(A)
Ordinary(A) :- Auto(A)

Predict the result under “any auto node blesses the cycle” and under restricted coinduction. The first policy unsoundly accepts; the second demands inductive grounding.

Experiment E: alias growth#

Create a normalization rule whose output nests the input alias. Exact-key cycle detection will miss each larger goal. Add a resource limit and collect term-size growth. Discuss why semantic termination checks are difficult and why production limits remain necessary.

Debugging exercise: order dependence#

Randomize candidate order across 1,000 runs. If answers differ, log snapshots and response merges. Common causes are leaked substitutions, first-success policy, unstable ambiguity handling, or an incomplete canonical key.

Contribution exercise#

Choose one recent solver pull request from the rust-lang/rust repository. Identify:

1. the earliest broken invariant;
2. the goal kind involved;
3. whether the fix changes mechanism or policy;
4. the regression test and why it distinguishes the bug;
5. compatibility and performance risks reviewers discussed.

Do not summarize only the final patch. Trace one failing goal through the old and fixed proof trees.

87. Derived philosophy#

Representations decide cheap questions#

An impl index makes “which headers might match this known constructor?” cheap. A canonical key makes “have we solved this alpha-equivalent goal?” cheap. A proof tree makes “where did this obligation fail?” cheap. No one representation makes all three cheap; duplicating representations creates synchronization obligations.

Abstractions move responsibility#

Canonicalization removes local variable identity from queries, but response application must reconstruct local constraints. Trait objects remove concrete type identity from callers, but vtables and dyn-compatibility rules absorb the cost. Opaque types hide implementation details, but reveal modes and defining-use checks become necessary.

Uncertainty must be represented#

Ambiguity is not solver weakness to erase with a guess. It records that the current information does not justify one stable conclusion. Guessing turns inference-order accidents into language behavior.

Caching creates proof obligations#

A cache is sound only if its key captures all semantic inputs and its value is portable across callers represented by that key. Provisional caching adds a temporal obligation: no unfinished assumption may escape as final evidence.

Identity is not an index#

Inference variable ?7, bound variable 0, placeholder !1, and canonical variable ^0 may all print like numbered unknowns. Their identities and scopes differ. Treating numbers as globally meaningful is a recurring source of leaks and cache bugs.

The first visible failure is late#

A region error may originate in wrong binder instantiation. An ambiguity may originate in an omitted response constraint. An overflow may originate in normalization choosing the wrong direction. Debug by walking backward to the earliest invariant violation, not by patching the final symptom.

Local simplicity can create global complexity#

Eagerly normalizing every alias simplifies one consumer but may create cycles, erase abstraction, and duplicate work globally. Choosing the first candidate simplifies evaluation but exports nondeterminism and breaks coherence assumptions. Good compiler architecture accounts for costs moved to other phases.

Optimization preserves meaning under an observation model#

For the solver, observable meaning includes substitutions, external constraints, certainty, overflow behavior, and accepted programs. For developer tooling it may also include deterministic diagnostics and proof traces. State that model before approving caching or pruning.

88. Mastery route, talk outlines, and reading map#

Mastery checks#

You should now be able to:

• derive a goal, clauses, and environment from a generic Rust fragment;
• explain why canonical input and response boundaries enable caching;
• trace inference snapshots through candidate evaluation;
• distinguish freshening, canonicalization, and placeholder instantiation;
• explain nested goals, candidate merging, ambiguity, and overflow;
• draw a search graph and classify inductive versus restricted coinductive cycles;
• trace associated-type normalization and identify reveal-policy risks;
• explain why coherence needs open-world reasoning;
• locate source modules and design a focused rustc regression test.

Prediction-based oral exam#

1. Why can Vec<?T>: Clone progress while ?T: Iterator often remains ambiguous?
2. Which metadata must survive canonicalizing a variable from universe U2?
3. Why is P :- P different from recursive Send through a linked structure?
4. When do two different candidates still produce one unambiguous answer?
5. Why does failure to find an impl not prove a negative proposition?
6. How can eager projection normalization make termination worse?
7. Which facts belong in a cache key, and which source facts should remain diagnostic side data?
8. What test shows that a candidate probe leaked inference state?

Answer each with an invariant and a counterexample, not only a definition.

Thirty-minute talk: the mental model#

1. Five minutes: bounds become goals; impls become conditional clauses.
2. Five minutes: canonical queries isolate mutable inference.
3. Five minutes: candidates produce nested goals and constrained responses.
4. Five minutes: search graphs, cycles, and why auto traits need restricted coinduction.
5. Five minutes: aliases, coherence, and open-world uncertainty.
6. Five minutes: one proof trace and one earliest-invariant debugging story.

Sixty-minute contributor talk#

1. Derive a trait and projection goal from source.
2. Walk canonicalization and response application.
3. Live-trace an impl candidate in EvalCtxt-style evaluation.
4. Compare inductive, coinductive, and growing recursion.
5. Show coherence and specialization as policy layers.
6. Compare legacy and next-solver architecture without caricature.
7. Navigate current source and tests.
8. End with a real regression's proof tree, performance counters, and review checklist.

Capstone assessment#

Extend the educational solver with one of:

• associated-type bounds under binders;
• a deterministic multi-answer merge;
• proof-tree minimization that preserves the earliest failure;
• a coherence mode with an explicitly bounded closed world.

Submit the invariant, counterexample to the simpler design, implementation, property tests, representative benchmark, explicit omissions, and a comparison to current rustc source.

Authoritative reading map#

• The Rust Reference: traits, implementations, coherence, trait objects, and impl Trait chapters for normative language rules.
• The rustc-dev-guide trait solving chapters for maintained compiler orientation; verify pages against the checkout because architecture migrates.
• rust-lang/rust source for current implementation facts; pin a commit for citations.
• Matching nightly rustdoc for rustc_next_trait_solver, rustc_type_ir, rustc_infer, and rustc_middle APIs.
• Chalk repository and book for historical design and terminology, clearly labeled as historical.
• Rust compiler performance infrastructure and rustc contributor guide for benchmarking, UI tests, and review workflow.

Final perspective#

The trait solver does not search for a convenient impl and return a boolean. It evaluates canonical propositions under explicit assumptions, interacts transactionally with inference, recursively justifies nested requirements, and returns constraints with calibrated certainty. Its graph and caches are part of logical correctness because cycles can be proofs only under restricted policy. Its binder machinery protects universal claims from existential shortcuts. Its coherence modes protect separate compilation from conclusions that are true only in today's crate graph.

When reading or changing it, ask four questions in order:

1. What exact proposition and environment reached the solver?
2. What information did canonicalization preserve or forget?
3. Which candidate and nested-goal edge first violated an invariant?
4. Is the proposed fix mechanism, policy, optimization, or presentation?

Those questions turn an intimidating recursive compiler subsystem into a sequence of checkable proof obligations.

Part IV-D: Higher-Ranked, Generic, Const, and Opaque Types#

This continuation gives four advanced type families their own end-to-end treatment.

It targets the Rust 1.97.1 source revision.

Language guarantees come from the Reference; implementation names and paths are revision-sensitive.

The trait solver remains the subject of Part IV-C.

Here we construct the terms sent to that solver, preserve their scopes, interpret its answers, and follow consequences into type and borrow checking.

89. Why these features belong together#

The concrete pressure#

Ordinary generics say that a definition works for caller-selected arguments.

Advanced types add four harder questions.

1. Can a value work for every lifetime, including one selected after the value was created?
2. Can an associated result itself be generic?
3. Can a compile-time value participate in type identity before that value is known?
4. Can a definition expose capabilities while hiding one fixed concrete type?

Higher-ranked trait bounds, GATs, const generics, and opaque types answer those questions.

They share one engineering problem: rustc must represent unavailable information without guessing it away.

source declaration
    |
    | lower binders, owners, arguments, and bounds
    v
semantic term + ParamEnv + typing/reveal mode
    |
    +--> relation and inference
    +--> solver boundary: goals and canonical responses
    +--> const-evaluation boundary
    `--> borrow-check constraints
             |
             v
typeck products, diagnostics, and metadata

Each arrow carries typed terms, not source strings.

The diagram omits query caching and error recovery.

One mental model#

Think of a type term as a sealed form with numbered holes and doors.

A binder owns holes that may be mentioned underneath it.

A universe records which rigid names an inference variable was allowed to see when created.

A projection is a suspended request for an associated result.

An unevaluated const is a suspended computation with an owner and substitutions.

An opaque alias is a public seal with one hidden type chosen at defining uses.

Opening the wrong door early creates a leak.

Never opening a permitted door causes false ambiguity.

Mechanism, policy, optimization, presentation#

Mechanism includes shifting bound variables, partitioning generic arguments, and recording hidden-type equations.

Policy decides required GAT bounds, legal const parameter types, opaque capture, and where a hidden type may be revealed.

Optimization includes interning, normalization caches, and delayed evaluation.

Presentation chooses source spans and whether diagnostics print an alias or a revealed term.

Changing presentation must not silently change reveal policy.

Changing a cache must preserve universes, owners, environments, and modes.

Four invariants#

Scope invariant: a bound variable is interpreted only relative to its binder.

argument invariant: every generic argument occupies the slot declared by its owner, after parent arguments.

const invariant: type equality uses justified const equality, never textual resemblance.

opaque invariant: one opaque identity has one compatible hidden type for each permitted substitution, revealed only where policy allows.

The earliest broken invariant may surface much later.

A wrong binder shift can appear as a borrow error.

A swapped GAT argument can appear as failed normalization.

A missing evaluatability obligation can appear at monomorphization.

A wrong reveal mode can expose a private type in another crate.

90. Quantifiers before HRTB syntax#

“Some” and “every” are different programs#

An existential statement says there exists an argument making a proposition true.

exists 'a. F: Fn(&'a str)

A universal statement says the proposition holds for every argument the caller may choose.

forall 'a. F: Fn(&'a str)

Rust's for<'a> expresses the second shape.

Inference normally solves existential unknowns.

It may choose a convenient value for ?a.

That is insufficient for a universal claim because the implementation, not the caller, would control the choice.

A stable end-to-end example#

fn twice<F>(f: F) -> usize
where
	F: for<'a> Fn(&'a str) -> usize,
{
	let first = String::from("one");
	let second = String::from("three");
	f(&first) + f(&second)
}

fn length(value: &str) -> usize {
	value.len()
}

fn main() {
	assert_eq!(twice(length), 8);
}

This is a complete stable program.

The caller of twice cannot predict the block lifetimes of first and second.

The function item length works for either.

A near miss#

fn accepts_one<'x, F>(f: F, value: &'x str) -> usize
where
	F: Fn(&'x str) -> usize,
{
	f(value)
}

This fragment requires one caller-selected 'x.

It does not establish that F accepts every lifetime.

Moving the quantifier outward changes who chooses.

for<'a> (F: Fn(&'a str))    caller may choose a fresh 'a for each proof
(F: Fn(&'a str))            one 'a is supplied by surrounding generics

Prediction#

Suppose a closure captures a reference and accepts another reference.

Does capture alone prevent it from satisfying an HRTB?

No.

The relevant question is whether its input signature works for every required input lifetime and whether the captured environment itself remains valid where the closure is used.

Input quantification and environment lifetime are related by constraints, not by a blanket ban.

When HRTBs are appropriate#

Use an HRTB when the callee needs to create short-lived borrows and pass them to caller-provided behavior.

Common patterns include callbacks, deserialization-like families, lending operations, and trait bounds over references.

Do not add for<'a> merely to silence a named-lifetime error.

It strengthens a contract.

That strength can reject a value that works for one particular lifetime.

91. Binders and de Bruijn indices#

Names are for people#

Alpha-equivalent types differ only in bound names.

for<'a> fn(&'a u8)
for<'z> fn(&'z u8)

They should have the same semantic shape.

Rustc therefore represents a bound variable by position and binder depth, not by source spelling.

A de Bruijn index counts binders outward from the variable use.

INNERMOST means the nearest enclosing binder.

Crossing another binder increments the distance.

Exact rustc newtypes and formatting should be checked at the 1.97.1 revision.

Concrete trace#

Consider this conceptual nested type:

for<'a> fn(for<'b> fn(&'a u8, &'b u8))

Inside the inner function:

• 'b refers to bound variable 0 at de Bruijn depth 0;
• 'a refers to bound variable 0 at de Bruijn depth 1.

The variable slot and binder distance are separate numbers.

Binder A: ['a = slot 0]
  |
  +-- function
      |
      `-- Binder B: ['b = slot 0]
          |
          `-- (&'a [depth 1, slot 0], &'b [depth 0, slot 0])

The diagram omits late-bound region kinds and source identities.

Entering and leaving binders#

When a free term moves underneath a new binder, references to surrounding bound variables must be shifted.

When replacing a binder's variables with arguments, the substitution must avoid capturing variables already inside those arguments.

shift_in(term, 1): increase indices that cross the insertion point
instantiate(binder, args): replace innermost slots, then adjust survivors
rebind(term): verify no replaced variable remains free

These are operations on scoped syntax.

They are not integer arithmetic performed without a traversal context.

Counterexample: capture by omission#

Suppose a type containing an outer depth-0 variable is inserted under a new binder without shifting.

The new binder now appears to own that variable.

The compiler has changed which lifetime the type means.

This can accept invalid borrowing or reject valid code depending on relation direction.

Bound, free, placeholder, and inference variables#

Printing all five as anonymous lifetimes would make debugging impossible.

Treating any two as interchangeable is a correctness bug.

Diagnostic provenance#

De Bruijn indices are good for capture-safe operations and poor user diagnostics.

Rustc must preserve or recover source binder spans and names separately.

A message should say which for<'a> introduced the lifetime, not only “bound region 0.”

This is a representation tradeoff: compact semantic identity moves presentation work into provenance tables.

92. Placeholders, universes, and leak checks#

Skolem-like testing#

To test for<'a> P('a), rustc enters a fresh universe and replaces 'a with a fresh rigid placeholder.

If it can prove P(!a) without relying on the identity of !a, it has evidence for the universal claim.

The placeholder is sometimes explained using Skolemization.

The useful operational rule is simpler: it is fresh, rigid, and not visible to older inference variables.

Universe U0: ?T created here
    |
    | enter higher-ranked binder
    v
Universe U1: placeholder !a created here
    |
    +-- legal: relate !a to itself
    `-- illegal: assign old ?T = &'!a u8

Visibility#

An inference variable records a universe at creation.

It may be assigned only a term whose placeholders are visible from that universe under the relation's rules.

An older variable cannot contain a newer placeholder.

A variable created inside the newer universe may mention it, but exporting that variable's value still requires the enclosing operation's response and leak discipline.

The classic leak#

Imagine proving:

exists ?T. forall 'a. ?T = &'a u8

After replacing 'a by !a, naive unification sets ?T = &'!a u8.

The result appears to choose one type that works for all lifetimes.

It actually captured the one placeholder used to test the claim.

When !a leaves scope, ?T contains a name that cannot exist there.

The universe check rejects this assignment.

Leak checking beyond one assignment#

Leaks can travel indirectly through variable equivalence classes, region constraints, canonical responses, and normalized aliases.

A robust check asks whether any result observable outside the binder depends on its placeholders except through an allowed re-abstraction.

The old and next solver architectures may perform this accounting at different boundaries.

Part IV-C explains their proof search.

This chapter's boundary contract is:

input: binder instantiated with fresh placeholders
output: constraints whose variables and universes are valid for caller
forbidden: response that embeds an unnameable placeholder in older state

A progressive stable-Rust checker#

The following complete educational program models only universe visibility.

#[derive(Clone, Debug, PartialEq, Eq)]
enum Ty {
	Infer { id: usize, universe: u32 },
	Placeholder { id: usize, universe: u32 },
	Ref(Box<Ty>),
	U8,
}

fn max_placeholder_universe(ty: &Ty) -> Option<u32> {
	match ty {
		Ty::Placeholder { universe, .. } => Some(*universe),
		Ty::Ref(inner) => max_placeholder_universe(inner),
		Ty::Infer { .. } | Ty::U8 => None,
	}
}

fn may_assign(variable_universe: u32, value: &Ty) -> bool {
	max_placeholder_universe(value)
		.map(|placeholder| placeholder <= variable_universe)
		.unwrap_or(true)
}

fn main() {
	let outer = Ty::Infer { id: 0, universe: 0 };
	let inner_name = Ty::Placeholder { id: 0, universe: 1 };
	let captured = Ty::Ref(Box::new(inner_name));

	let Ty::Infer { universe, .. } = outer else { unreachable!() };
	assert!(!may_assign(universe, &captured));
	assert!(may_assign(universe, &Ty::Ref(Box::new(Ty::U8))));
}

The program is stable and dependency-free.

It intentionally treats universe numbers as a total visibility order.

It omits region constraints, variable unification, binders, canonicalization, higher-ranked subtyping, snapshots, and re-abstraction.

It is not a Rust type checker.

Tests to add#

1. A same-universe placeholder is visible.
2. A placeholder nested three constructors deep is still found.
3. A value with no placeholder is accepted.
4. Equating two inference variables uses the stricter resulting visibility.
5. A failed speculative assignment rolls back both value and universe metadata.

The fourth test cannot be implemented correctly by the tiny program without adding an inference table.

That omission is the next milestone, not permission to fake the result.

Failure symptoms#

93. Higher-ranked relations and solver boundaries#

Relating two quantified types#

Equality and subtyping between higher-ranked types are not implemented by erasing for.

Direction determines which side is instantiated with placeholders and which may use inference variables.

The goal is to preserve who chooses each lifetime.

For a universal obligation, placeholders test an arbitrary choice.

For an existential side, fresh inference variables permit a choice subject to universe visibility.

Higher-ranked subtyping also interacts with function contravariance.

Do not derive the algorithm from one covariance slogan.

One trace#

Conceptually compare:

expected: for<'a> fn(&'a u8) -> &'a u8
found:    for<'b> fn(&'b u8) -> &'b u8

Enter one binder with a fresh placeholder !x.

Instantiate both corresponding bound slots consistently.

Relate inputs in the function relation's parameter direction.

Relate outputs in its result direction.

No placeholder escapes.

The different source names are irrelevant.

Now compare a function returning &'static u8.

Whether it is usable depends on the exact relation and output position, not on textual binder matching.

Write a compile test rather than guessing from names.

Trait solver boundary#

An HRTB often reaches the solver as a predicate under a binder.

The caller owns:

• lowering the correct binder and bound-variable kinds;
• supplying ParamEnv and typing mode;
• canonicalizing local inference state;
• applying response constraints transactionally;
• retaining obligation cause and source span.

The solver owns candidate assembly, nested proof search, certainty, and cycle handling described in Part IV-C.

Neither side may silently erase universes.

Borrow-check boundary#

Type checking can establish a higher-ranked callable signature and emit region constraints.

Borrow checking later tests use of concrete places over MIR control flow.

An HRTB is not proof that every captured reference lives forever.

Conversely, a borrow checker should not rediscover trait candidate selection.

When an HRTB error appears as “not general enough,” compare the impl's actual late-bound regions with the required binder before changing borrow checking.

Performance#

Fresh placeholders reduce accidental sharing across binder instantiations.

That improves soundness but can fragment caches if canonicalization fails to alpha-normalize equivalent goals.

Measure:

• binder instantiations;
• canonical goal count;
• cache hit rate by universe shape;
• relation probes and rollback writes;
• proof-tree recording cost.

Never merge placeholders from distinct universes solely to improve hit rate.

Contribution exercise#

Find a 1.97.1 UI test containing “not general enough.”

Trace the HIR binder, semantic Binder, placeholder creation, canonical input, and final obligation cause.

Add one nested binder and predict every de Bruijn shift before compiling.

Classify any discrepancy as lowering, relation, solver response, borrow constraint, or diagnostic presentation.

94. GATs from declaration to semantic projection#

Why ordinary associated types stop short#

An ordinary associated type selects one result for an impl.

trait IteratorLike {
	type Item;
}

A lending interface needs a family indexed by the borrow lifetime.

trait Lending {
	type Item<'a>
	where
		Self: 'a;

	fn get<'a>(&'a self) -> Self::Item<'a>;
}

Item<'a> is not one type containing a free lifetime.

It is an associated type constructor applied to 'a.

Definition and lowering#

Source lowering records an associated item owned by the trait, with parent generics and item-local generics.

Conceptually:

trait owner arguments: [Self, trait parameters...]
GAT local arguments:   ['a, local types..., local consts...]
projection arguments:  [parent arguments..., local arguments...]

Rustc stores generic arguments in declaration order determined across the owner chain.

The familiar category slogan “lifetimes, then types, then consts” is not a safe replacement for querying the definition's actual generic parameter indices.

Synthetic and parent parameters make hand-built vectors especially dangerous.

Argument ordering trace#

trait Family<T, const N: usize> {
	type Out<'a, U>
	where
		Self: 'a;
}

A conceptual projection <S as Family<u8, 4>>::Out<'x, bool> needs:

[Self = S, T = u8, N = 4, 'a = 'x, U = bool]

The exact encoded order follows Generics indices in the pinned compiler.

Do not reorder by the spelling visible in only the GAT declaration.

Stable implementation example#

trait Lending {
	type Item<'a>
	where
		Self: 'a;

	fn get<'a>(&'a self) -> Self::Item<'a>;
}

struct Cell(u32);

impl Lending for Cell {
	type Item<'a> = &'a u32 where Self: 'a;

	fn get<'a>(&'a self) -> Self::Item<'a> {
		&self.0
	}
}

fn read<L: Lending>(value: &L) -> L::Item<'_> {
	value.get()
}

fn main() {
	let cell = Cell(7);
	assert_eq!(*read(&cell), 7);
}

This is a complete stable program.

It demonstrates a lifetime-indexed associated family.

It does not demonstrate every higher-ranked use of GATs.

Representation tradeoff#

Keeping a projection symbolic preserves its trait and associated-item identity, arguments, and diagnostic provenance.

It makes structural consumers request normalization.

Eagerly replacing every projection simplifies those consumers but may duplicate proof work, trigger cycles, and lose the source alias users recognize.

Rustc therefore has context-specific normalization APIs rather than one unconditional expansion pass.

95. GAT well-formedness and required bounds#

Well-formedness is local proof work#

A GAT declaration must be meaningful for its permitted arguments.

An impl's chosen value must satisfy the declaration and impl environment.

A projection use must satisfy bounds required to form that application.

These are related but distinct checks with distinct useful spans.

For the lending pattern, where Self: 'a says the projected family is used where Self outlives the borrowed result's index.

It prevents implementations from being required to produce impossible borrowed forms for arbitrary 'a longer than Self.

Required bounds#

Rust has rules requiring certain bounds on GAT declarations so impls retain maximal flexibility and methods using the GAT are soundly expressible.

These rules were refined during GAT stabilization.

Do not infer the exact 1.97.1 rule from one example or an old RFC.

Use the Reference, the matching compiler source, and tests/ui/generic-associated-types at the release revision.

The compiler conceptually intersects constraints implied at uses to determine requirements, but current implementation details and diagnostics are not a language-level algorithm promise.

Counterexample to omitting the bound#

trait BadLending {
    type Item<'a>;
    fn get<'a>(&'a self) -> Self::Item<'a>;
}

This is a conceptual negative example; acceptance and suggested bounds must be checked against 1.97.1.

The method use connects 'a to Self.

Without expressing the necessary relationship on the GAT, implementations may face an impossible “for all 'a” obligation detached from Self's validity.

WF pipeline#

GAT declaration bounds
      |
      v
item-level WF predicates ----+
                             |
impl value + impl ParamEnv --+--> normalize/relate --> obligations
                             |
projection use arguments ----+

The solver proves trait and outlives-related goals at its boundary.

The item checker decides which WF goals to create and where to report them.

Borrow checking later validates actual loans.

Diagnostics#

Good diagnostics distinguish:

• missing required bound on the GAT declaration;
• impl associated type failing a declared bound;
• projection used without satisfying its where-clause;
• higher-ranked limitation or ambiguity;
• actual borrow conflict in a method body.

One generic “lifetime may not live long enough” message discards the phase distinction.

Decision table#

Rust chooses explicit declaration requirements in important cases because the bound is part of the associated family contract.

96. GAT projection, normalization, and borrowing#

Projection is a question#

<T as Lending>::Item<'x> means “the Item selected by the applicable Lending implementation for T, instantiated at 'x.”

It is not a field lookup.

Normalization needs evidence for T: Lending, correct parent and local arguments, applicable bounds, and a permitted normalization mode.

The solver algorithms for selecting and merging evidence remain in Part IV-C.

Normalization trace#

For concrete Cell from Chapter 94:

input alias: <Cell as Lending>::Item<'x>
candidate: impl Lending for Cell
associated value: &'a u32
substitute local 'a := 'x
check where clause Cell: 'x
result: &'x u32 plus any obligations/constraints

For generic L: Lending, no impl need be selected.

An environment projection equality or bound may provide information.

Otherwise the alias honestly remains symbolic.

Higher-ranked GAT bounds#

fn require_debug<L>()
where
	L: Lending,
	for<'a> L::Item<'a>: core::fmt::Debug,
{
}

This stable fragment states that every member of the associated family implements Debug.

It does not select one convenient 'a.

The binder encloses the projection argument.

Instantiation therefore combines GAT substitution with placeholder and universe rules.

Borrow-check interaction#

GATs make types depend directly on borrow lifetimes.

Type checking normalizes or relates a projection enough to know method signatures and emits region constraints.

MIR borrow checking determines whether the actual loan can remain live over control flow.

Polonius or NLL architecture does not remove the need for correct GAT binder lowering.

A bogus 'static requirement may originate in higher-ranked trait solving or normalization before the borrow checker sees MIR.

Famous limitation class#

Some higher-ranked GAT patterns have historically produced implied-'static limitations or diagnostics.

Their exact status evolves.

For 1.97.1, compile the minimized program and inspect current release notes and tests rather than repeating an older blog post as current behavior.

Label observed rejection as implementation limitation only when authoritative current sources do.

Debugging workshop#

Symptom: for<'a> L::Item<'a>: Debug reports that a local borrow must be 'static.

1. Replace the HRTB with one named lifetime from the function.
2. Replace the GAT with an ordinary associated type.
3. Print the projection's parent/local argument partition.
4. Print binder depth before and after canonicalization.
5. Inspect placeholder universe in the solver response.
6. Inspect region constraints passed onward.
7. Only then classify the borrow-check result.

The two reductions distinguish GAT substitution from higher-ranked handling.

Educational projection checker#

Build a table keyed by (trait_name, self_type, associated_name).

Store an associated body with numbered local parameters.

On projection, first split parent and local arguments using declaration metadata.

Then substitute local parameters capture-avoidantly.

Finally return the body plus declared obligations.

Required tests:

1. Two GAT lifetime applications normalize differently.
2. Parent and local type arguments cannot swap.
3. Missing local argument reports arity, not a solver failure.
4. A where-clause is returned, never discarded.
5. Recursive alias expansion terminates with a cycle result.

Explicit omissions: impl selection, overlap, regions, variance, HRTBs, const arguments, reveal modes, and real Rust syntax.

97. Const generics: values as type arguments#

The basic mechanism#

Const generics allow a compile-time value to occupy a generic argument slot.

fn repeat<const N: usize>(byte: u8) -> [u8; N] {
	[byte; N]
}

fn main() {
	let bytes = repeat::<3>(9);
	assert_eq!(bytes, [9, 9, 9]);
}

This complete stable program uses N in type identity and value construction.

[u8; 3] and [u8; 4] are distinct types.

The const argument has its own type, here usize.

Structural const parameters#

Not every Rust value may be a const generic parameter.

Parameter values need a stable, structural notion of equality suitable for type identity, matching, metadata, and coherence.

The stable set has historically centered on integers, bool, and char, with feature-gated expansion under the structural-match/adt-const-parameter design space.

For exact Rust 1.97.1 support, consult the Reference and feature gates in that checkout.

Do not claim that every ConstParamTy-like implementation or every const-evaluable type is stable as a const parameter.

Evaluation ability and admissibility as type-level identity are different policies.

Why floating point is difficult#

Floating-point equality includes NaN and signed-zero behavior that is unsuitable for a simple structural type-identity relation.

User-defined values add concerns about private fields, derived structural equality, future field changes, and cross-crate metadata.

The compiler needs equality that is deterministic and compatible with pattern-like structural comparison, not arbitrary user code.

Running PartialEq during type checking would make type identity effectful, potentially nonterminating, and semver-sensitive.

Internal const forms#

A semantic const may be:

• a concrete value;
• a generic parameter;
• an inference variable;
• a bound variable;
• a placeholder;
• an unevaluated definition plus generic arguments;
• an expression-like abstract const;
• an error/recovery term.

Exact ConstKind variants and the split between ty::Const, valtrees, and expression representations are revision-sensitive.

The invariant is that identity and substitutions survive until justified evaluation or normalization.

Inference#

In fn take<const N: usize>(x: [u8; N]), passing [u8; 7] constrains N = 7.

Const inference variables belong to a separate domain from type and region variables.

They carry a const type.

Unifying a usize const variable with a bool const is malformed even before comparing values.

Fallback rules for integer literals are not a general const-variable default.

Counterexample to textual equality#

N + 1 in one definition and M + 1 in another are not equal because their printed strings resemble each other.

Their owners and substitutions differ.

Conversely, two differently written concrete computations may evaluate to the same value.

The correct operation depends on whether the language permits evaluation and normalization at that point.

98. Abstract consts and evaluatability#

Why symbolic expressions are needed#

Minimal const generics can use a const parameter directly in supported positions.

Generic const expressions need to represent computations such as N + 1 before N is known.

An AbstractConst-style representation captures the expression's structure over generic parameters so rustc can reason conservatively without executing an unknown instantiation.

The exact implementation and feature status at 1.97.1 must be checked in source.

The concept is not a promise that rustc performs general symbolic algebra.

Evaluatability#

An expression is well typed if its operations accept their operand types.

It is evaluatable in a generic context only if rustc can justify that every permitted substitution can produce the required const without forbidden failure.

Overflow, panic, division by zero, and unsupported generic operations matter.

well typed: operands and result have valid const types
evaluatable: computation is justified under current generic assumptions
valid value: result is allowed at its use site
layout usable: target/layout constraints can consume it

These checks can occur at different phases.

The unconstrained generic constant pattern#

A type containing a generic expression may require a corresponding evaluatability bound or witness under feature-gated syntax.

Compiler diagnostics often suggest a where [(); EXPR]:-shaped bound in applicable eras.

Treat that spelling as version-sensitive and feature-sensitive.

The semantic purpose is to make evaluatability part of the item's contract rather than discovering failure only at one monomorphization.

Normalization#

Const normalization can:

1. substitute known generic arguments;
2. evaluate a fully concrete const;
3. simplify through compiler-supported abstract-const rules;
4. leave the term symbolic;
5. emit an error or ambiguity with cause.

It must not rewrite by unproven algebraic identities.

For machine integers, overflow semantics and evaluation mode make school algebra an unsafe guide.

N + 1 - 1 = N is not universally valid if the intermediate addition can overflow.

Solver boundary#

Const evaluatability and equality can appear as obligations or goals.

Part IV-C owns the recursive goal algorithm.

The const/type layer owns constructing typed const terms, carrying their environment, invoking CTFE only through valid query boundaries, and applying answers without losing unevaluated identity.

CTFE itself is treated in Part V.

Educational evaluator#

The following complete stable program implements a deliberately tiny symbolic const language.

use std::collections::BTreeMap;

#[derive(Clone, Debug, PartialEq, Eq)]
enum Expr {
	Value(u64),
	Param(&'static str),
	Add(Box<Expr>, Box<Expr>),
}

fn evaluate(expr: &Expr, args: &BTreeMap<&str, u64>) -> Result<u64, String> {
	match expr {
		Expr::Value(value) => Ok(*value),
		Expr::Param(name) => args
			.get(name)
			.copied()
			.ok_or_else(|| format!("unknown const parameter {name}")),
		Expr::Add(left, right) => evaluate(left, args)?
			.checked_add(evaluate(right, args)?)
			.ok_or_else(|| "const addition overflowed".to_owned()),
	}
}

fn main() {
	let expr = Expr::Add(
		Box::new(Expr::Param("N")),
		Box::new(Expr::Value(1)),
	);
	let args = BTreeMap::from([("N", 4)]);
	assert_eq!(evaluate(&expr, &args).unwrap(), 5);
	assert!(evaluate(&expr, &BTreeMap::new()).is_err());

	let overflow = BTreeMap::from([("N", u64::MAX)]);
	assert!(evaluate(&expr, &overflow).is_err());
}

The model distinguishes unknown from failed evaluation.

It uses u64, not Rust's target-dependent usize semantics.

It omits owners, substitutions by index, typed nodes, multiplication, casts, branches, abstract-const unification, CTFE, target layout, inference, and query caching.

Progressive tests#

Add tests in this order:

1. Concrete addition evaluates.
2. Substitution of a parameter evaluates.
3. Missing substitution remains explicitly unknown.
4. Overflow is not simplified away.
5. Alpha-renamed parameters compare only after owner-aware substitution.
6. Evaluation cache keys include expression identity and arguments.
7. Cancellation never publishes a partial value.

The fifth test requires replacing string names with owner-and-index identities.

That is an intentional representation milestone.

99. Const-generic limits, diagnostics, and production costs#

Current limits are layered#

Ask four separate questions.

1. Is this const parameter type permitted?
2. Is this expression legal in a const generic position?
3. Can this generic expression be proven evaluatable?
4. Can this concrete result be used for layout or matching?

The answer can differ at each layer.

Feature gates may expand one layer without stabilizing the others.

Never summarize a nightly experiment as stable Rust 1.97.1 behavior.

Inference limits#

Rustc can infer a const from structural type equality when enough information exists.

It does not solve arbitrary Diophantine equations to infer N from 2 * N + 1 = 9.

Generic const expression inference remains intentionally conservative.

Ambiguity is better than choosing one mathematical solution that does not follow language rules.

Diagnostics map#

A final layout failure does not justify moving all evaluation into type lowering.

Earlier generic evaluation may lack target and substitutions and can duplicate work.

Soundness concerns#

Type identity must be deterministic across crates and compiler sessions.

Const values in metadata must preserve semantic identity and supported encoding.

Evaluation must honor overflow and target behavior.

Private representation must not leak through structural parameter encoding.

Malformed or adversarial const expressions must terminate under resource limits.

Unsafe code may rely on array length and layout facts, so accepting a false const equality can become memory unsafety.

Cost model#

Let E be expression size, S substitutions, and Q repeated query count.

Re-evaluating costs approximately Q * walk(E, S) plus CTFE work.

Caching can reduce repeated work but key construction includes expression identity, substitutions, environment, target-sensitive mode, and feature/reveal context where relevant.

Large abstract expressions can grow during substitution and normalization.

Hash-consing saves repeated structure but retains memory.

Measure peak term size, cache hit rate, CTFE steps, allocations, and diagnostic latency.

Adversarial tests#

• deeply nested arithmetic expressions;
• many alpha-equivalent expressions under different owners;
• recursive associated const dependencies;
• overflow near integer boundaries;
• huge array lengths that exceed layout limits;
• cancellation during evaluation;
• cross-crate metadata round trips;
• incremental recompilation after changing one const argument.

Contribution exercise#

Locate one 1.97.1 const-evaluatability diagnostic.

Trace from HIR expression to semantic const, ParamEnv, evaluatability obligation, solver boundary, and CTFE query.

Record whether failure means unsupported symbolic reasoning, proven evaluation failure, or invalid final value.

Improve only the span or cause if semantics are already correct.

100. Opaque types: one hidden identity#

Existential interface, fixed implementation#

Return-position impl Trait exposes bounds while hiding a concrete type.

fn numbers() -> impl Iterator<Item = u8> {
	0u8..4
}

fn main() {
	assert_eq!(numbers().sum::<u8>(), 6);
}

This complete stable program has one opaque identity associated with numbers' return.

Callers can use Iterator<Item = u8> behavior.

They cannot name or rely on the concrete range type through the API.

Identity is definition-based#

Two functions with identical impl Iterator<Item = u8> bounds generally define distinct opaque types.

fn left() -> impl Iterator<Item = u8> { 0u8..1 }
fn right() -> impl Iterator<Item = u8> { 0u8..1 }

The hidden concrete types happen to match.

The opaque identities do not thereby become interchangeable nominal aliases.

Identity includes the defining opaque item and its captured generic arguments, not the text of its bounds or hidden type.

Defining uses#

Within a permitted defining scope, return expressions constrain the hidden type.

All defining uses for one opaque instantiation must agree under the relevant relation and normalization policy.

opaque O<T>
return site 1 gives HiddenA<T>
return site 2 gives HiddenB<T>
require compatible single hidden type for O<T>

An if returning two different iterator adapter types does not mean “choose either implementation of Iterator.”

The branches need one concrete hidden type, perhaps obtained by changing the implementation or boxing a trait object.

Bounds flow both ways#

The declaration promises bounds to callers.

The inferred hidden type must satisfy those bounds.

Callers may use only promised properties, even if the hidden type implements more traits.

Auto traits and lifetime capture have detailed exposure rules; do not assume every hidden implementation property is part of the public contract in every opaque form.

Capture#

An opaque hidden type may capture permitted generic type, lifetime, and const parameters.

Capture rules depend on opaque form and edition, and have evolved through precise-capturing syntax.

For Rust 1.97.1, consult the Reference for use<...> bounds and edition-specific automatic capture.

An implementation claim must state the crate edition.

Over-capture can impose unexpected borrow restrictions or semver commitments.

Under-capture makes a hidden type unrepresentable.

Reveal modes#

Outside the defining context, normalization preserves the opaque alias.

Inside an authorized context, rustc may reveal or relate it to its inferred hidden type.

ParamEnv/typing mode and query context carry this policy in revision-specific ways.

Reveal is semantic access, not merely pretty-printing.

public caller: O<args> --bounds only--> operations promised by API

defining scope: O<args> --authorized reveal/relate--> Hidden<args>
                                            |
                                            `--> verify bounds and consistency

Cache keys for normalization must distinguish modes that can observe different forms.

Cross-crate behavior#

Metadata exports opaque identity, generics, bounds, and other required semantic information.

It does not turn the hidden implementation into a public type alias.

Downstream type checking reasons from bounds and identity.

Upstream code generation may know concrete monomorphized hidden types as needed while preserving language abstraction.

Changing a hidden type can be source-compatible when promised bounds and capture behavior remain compatible, but layout, auto traits, lifetime captures, and semver details require care.

Do not promise ABI stability from impl Trait alone.

101. RPIT, async return, RPITIT, and associated opaques#

RPIT#

Return-position impl Trait, RPIT, belongs to a function or method return.

Its defining uses are the return expressions in the permitted defining body/scope.

The opaque captures allowed generics and is abstract to ordinary callers.

Argument-position impl Trait is instead generic-parameter-like surface syntax and must not be assigned RPIT's hidden-type inference model.

Async functions#

An async fn returns an anonymous future-like type.

Conceptually, it resembles a function returning impl Future<Output = T>, but this is an explanatory model, not a source-to-source identity promise for every capture and diagnostic rule.

The hidden state machine captures values that live across suspension points.

Type checking establishes output and capture-related facts; coroutine lowering and borrow checking handle state-machine details in later phases.

async fn answer() -> u32 {
	42
}

fn assert_future<F: core::future::Future<Output = u32>>(_: F) {}

fn main() {
	assert_future(answer());
}

This complete stable program type-checks the async return bound without requiring an executor.

It does not inspect the hidden future type.

RPITIT#

Return-position impl Trait in traits gives each trait method an opaque result contract.

trait Source {
	fn values(&self) -> impl Iterator<Item = u8>;
}

struct Bytes;

impl Source for Bytes {
	fn values(&self) -> impl Iterator<Item = u8> {
		[1, 2, 3].into_iter()
	}
}

fn total(source: &impl Source) -> u8 {
	source.values().sum()
}

fn main() {
	assert_eq!(total(&Bytes), 6);
}

This complete stable program demonstrates RPITIT.

Rustc lowers RPITIT through synthesized associated/opaque machinery in a revision-sensitive implementation.

The source trait did not literally declare that synthetic item.

Parent trait arguments, method generics, captures, impl method arguments, and opaque identity must all align.

Associated opaque types#

“Associated opaque type” can describe internal synthesized items used for RPITIT or source features that place impl Trait in an associated type value.

Do not collapse them.

Their stability, defining scopes, and identity rules can differ.

Impl Trait in associated types is commonly discussed under associated type impl Trait and remains feature-sensitive; verify 1.97.1 before showing it as stable source.

Object and dispatch tradeoffs#

RPITIT avoids requiring callers to name the concrete return and can preserve static dispatch.

It may affect dyn compatibility because a trait object cannot freely call methods whose opaque associated result cannot be represented under current rules.

Box<dyn Iterator<Item = u8>> is an alternative with allocation, dynamic dispatch, object lifetime, and auto-trait differences.

A named GAT is another alternative when callers need to write bounds over the return family.

No option is universally superior.

Debug trace#

1. Identify source trait method and impl method DefIds.
2. Locate any synthesized associated and opaque item identities.
3. Partition trait-parent and method-local generic arguments.
4. Record captures and binder depths.
5. Compare trait promised bounds with impl hidden type obligations.
6. Observe alias before normalization and current reveal mode.
7. Stop at solver candidate evaluation; use Part IV-C for that algorithm.
8. Follow returned constraints into body and borrow checking.

102. Type alias impl trait in depth#

What TAIT tries to provide#

Type alias impl Trait, TAIT, gives a named opaque identity whose hidden type is inferred from authorized defining uses.

Conceptual feature-gated syntax is:

#![feature(type_alias_impl_trait)]

type Numbers = impl Iterator<Item = u8>;

#[define_opaque(Numbers)]
fn numbers() -> Numbers {
	0u8..4
}

This is a nightly, version-sensitive illustration, not a stable complete program.

The exact defining-use attribute and restrictions must be verified on Rust 1.97.1.

TAIT has undergone design and implementation changes; old examples are especially unreliable.

Identity#

The alias definition creates the opaque identity.

Uses of the alias refer to that identity instantiated with its captured/generic arguments.

Two TAIT declarations with equal bounds and equal inferred hidden types remain distinct opaque identities.

Renaming the alias does not change semantic identity within a compilation; moving to another definition does.

Cross-crate identity is carried by stable definition identity and metadata, not an interned pointer or source text.

Defining uses are authorized#

Not every occurrence of a TAIT may choose or reveal its hidden type.

Defining-use rules identify items allowed to constrain the hidden type.

This prevents an arbitrary downstream or unrelated use from changing the alias's meaning.

Conceptually:

definition: opaque O<P...>: Bounds
authorized use f: expression type H<P...>  => constraint hidden(O<P...>) = H<P...>
ordinary use g: O<P...>                    => abstraction only

The compiler must check that generic arguments at a defining use map to the opaque's parameters in the permitted pattern.

Otherwise one use could accidentally define only a special instantiation or make hidden type identity depend on inference accidents.

Hidden-type inference#

Rustc gathers hidden-type candidates from defining uses and relates them.

Each candidate retains source span, defining item, substitutions, and obligations.

The result must be one coherent hidden type family over the alias generics.

Hidden-type inference is not trait-object candidate selection.

The bounds constrain the one inferred type; they do not authorize any implementer at each use.

Generic family trace#

Suppose conceptual TAIT Opaque<T>: Trait is defined by Wrapper<T>.

At defining use with T = ?X, rustc must infer the family Wrapper<T>, not freeze one probe-local Wrapper<?X> containing an inference ID.

The hidden representation must be expressed in terms of the opaque's generic parameters after reverse mapping the defining substitution.

Failure to remap causes cross-item and incremental bugs.

defining body local type: Wrapper<local ?T resolved from alias arg P0>
              |
              | remap from use args to opaque parameters
              v
stored hidden family: Wrapper<OpaqueParam0>
              |
              `-- instantiate later with each legal alias argument

Conflicting defining uses#

If one authorized function yields Range<u8> and another yields Once<u8>, equal Iterator bounds do not reconcile them.

The diagnostic should point to both defining sites and explain that the concrete hidden types differ.

It should not suggest adding another trait bound.

The broken invariant is single hidden identity, not missing capability.

Capture and generics#

TAIT may be generic over lifetimes, types, and consts under feature-specific rules.

The hidden type may use only permitted captures.

An unused alias parameter and a hidden type that depends on it create identity and variance questions.

The compiler's capture analysis, generic argument mapping, and variance/well-formedness checks must agree.

Precise capture behavior is unstable territory; pin every claim to 1.97.1 source and feature documentation.

Bounds#

Declared bounds are the public facts consumers receive.

The hidden type must prove those bounds for all valid alias arguments under the declaration's environment.

Associated type bindings in bounds may require normalization.

Lifetime bounds constrain captured references.

Auto-trait behavior needs explicit compatibility review because downstream code may infer capabilities from an opaque under language rules.

Reveal modes#

Authorized defining checks need enough reveal to compare alias and hidden candidate.

Ordinary users must retain opacity.

Reveal-all is not a debugging shortcut safe to leave in place.

It can make downstream code rely on operations absent from declared bounds, expose private types in diagnostics, and contaminate caches.

Reveal-never is also wrong inside defining checks because the hidden-type equation could never be established.

The correct policy is context-sensitive and part of query input.

Cross-crate behavior#

A downstream crate sees the TAIT identity and exported bounds according to visibility and feature rules.

It must not contribute defining uses or recover the hidden type merely from metadata implementation details.

The defining crate may change the hidden concrete type subject to compatibility constraints.

Incremental metadata hashes must change when semantically relevant hidden-type facts used by code generation change, while diagnostics and public type identity remain opaque.

Separate compilation therefore needs both abstraction-facing metadata and compiler-internal information at sanctioned consumers.

TAIT versus a normal alias#

type Plain = Concrete;        transparent: users normalize to Concrete
type Hidden = impl Trait;     opaque: users reason from bounds and identity

A normal alias does not create a new nominal type and exposes its expansion.

A TAIT creates opaque identity and hidden-type obligations.

Replacing one with the other changes abstraction and possibly semver.

TAIT versus RPIT#

RPIT's identity is tied to one return position and its defining body.

TAIT names an opaque identity that can be shared among authorized defining and use sites.

That sharing makes defining-use discovery, generic remapping, and coherence of hidden candidates substantially harder.

RPIT is stable in broad use; TAIT remains feature-sensitive at the target revision.

Educational hidden-type checker#

Represent an opaque as:

OpaqueDef {
    id,
    params,
    bounds,
    authorized_definers
}

HiddenCandidate {
    opaque_id,
    definer_id,
    use_arguments,
    local_type,
    span
}

Milestone 1 checks authorization.

Milestone 2 verifies argument arity and kind.

Milestone 3 remaps local use arguments back to opaque parameters.

Milestone 4 structurally relates all remapped hidden candidates.

Milestone 5 verifies declared bounds through a mocked boundary returning Yes, No, or Ambiguous.

Milestone 6 implements User and Defining reveal modes.

Milestone 7 serializes public identity/bounds separately from private hidden data.

Required tests#

1. Two equal hidden candidates succeed.
2. Different candidates report both spans.
3. An unauthorized definer is rejected before inference.
4. Local inference IDs never enter stored hidden families.
5. Swapped generic arguments are rejected or remapped correctly.
6. User mode cannot reveal.
7. Defining mode reveals only its authorized opaque.
8. Distinct opaque IDs stay distinct with equal hidden types.
9. A hidden self-cycle is rejected.
10. Metadata round trip preserves opaque identity and bounds.

Explicit omissions#

The educational checker omits real trait solving, regions, variance, associated types, const evaluation, HIR ownership, incremental dependency graphs, privacy, coherence, code generation, and feature gating.

Its mocked bound checker must never be described as proving Rust traits.

Its structural type equality is insufficient for aliases and subtyping.

Its value is isolating authorization, remapping, consistency, and reveal invariants.

103. Unified debugging, testing, soundness, and source map#

Find the earliest broken invariant#

Start at the source term and walk forward until the first semantic representation differs.

Do not begin by editing the final message.

Source map for 1.97.1#

Paths are signposts, not stable APIs.

• compiler/rustc_middle/src/ty/ defines binders, regions, generic arguments, aliases, opaques, and const terms.
• compiler/rustc_type_ir/ contains solver-generic IR abstractions, binder and universe-related interfaces in contemporary layouts.
• compiler/rustc_infer/ owns inference variables, higher-ranked relations, canonicalization integration, snapshots, and region constraints.
• compiler/rustc_hir_analysis/ performs item lowering/checking, well-formedness, associated item checks, and opaque-related orchestration in revision-specific modules.
• compiler/rustc_hir_typeck/ checks defining bodies, return expressions, method calls, and type-dependent products.
• compiler/rustc_trait_selection/ contains legacy normalization/selection integration and opaque/type-system support.
• compiler/rustc_next_trait_solver/ owns next-solver goal evaluation described in Part IV-C.
• compiler/rustc_const_eval/ and compiler/rustc_middle query interfaces connect const terms to evaluation.
• compiler/rustc_borrowck/ consumes region constraints and MIR; it is not the owner of GAT argument lowering.
• compiler/rustc_metadata/ serializes cross-crate identities, bounds, and sanctioned opaque/const facts.
• tests/ui/higher-ranked/, tests/ui/generic-associated-types/, tests/ui/const-generics/, and tests/ui/impl-trait/ contain behavior tests; exact directories vary.

Search current symbols rather than assuming file names:

rg "DebruijnIndex|BoundVar|Placeholder|UniverseIndex" compiler/
rg "type_of.*opaque|opaque.*hidden|define_opaque" compiler/ tests/ui/
rg "generic_const_exprs|AbstractConst|ConstEvaluatable" compiler/
rg "required.*bound|generic-associated" compiler/ tests/ui/
rg "RPITIT|return_position_impl_trait_in_trait" compiler/ tests/ui/

These are source-reading commands, not guaranteed symbol spellings.

Focused test matrix#

For every semantic change, vary:

• accepted and rejected cases;
• one and two nested binders;
• inferred and explicit generic arguments;
• local and cross-crate use;
• old and next solver only where supported by that compiler;
• defining and user reveal contexts;
• clean and incremental builds;
• stable syntax and explicitly feature-gated nightly syntax;
• ordinary, ambiguous, overflow, and error-recovery paths.

Record edition and feature gates in every reproducer.

Compile-check protocol#

Complete examples in this file are marked complete and use stable syntax.

Extract each complete fence to its own file and compile it with the intended toolchain.

Fragments and conceptual/nightly examples must not be treated as complete stable programs.

For negative tests, assert the intended diagnostic class and span, not merely nonzero exit.

Run rustc UI tests with the command documented by the pinned checkout.

Do not copy a current bootstrap command into a historical revision without checking it.

Soundness review#

Treat these as high risk:

• an older inference variable captures a newer placeholder;
• binder shifting changes variable ownership;
• a GAT projection drops a where-clause;
• const equality is accepted without justified normalization;
• evaluatability is deferred past a boundary that assumes success;
• an opaque hidden type is revealed to an unauthorized caller;
• different defining uses select different hidden types;
• a cache key omits environment, universe, owner, target, or reveal mode;
• cross-crate metadata confuses two opaque or const identities.

Any can lead beyond a wrong diagnostic.

Libraries and unsafe code may rely on trait, lifetime, and layout facts established here.

Performance review#

Benchmark generic-heavy async code, lending/GAT libraries, const-expression stress tests, and many opaque defining uses.

Capture:

• number and size of instantiated binders;
• canonical goals by universe shape;
• projection normalization requests and cache hits;
• abstract-const node count and CTFE steps;
• opaque hidden candidates and remapping work;
• peak memory and retained interned terms;
• incremental reuse and metadata size;
• diagnostic proof/provenance overhead.

An optimization must preserve accepted programs, inferred constraints, reveal boundaries, deterministic errors, and resource-limit behavior under the stated observation model.

Debugging workshops#

Workshop A: universe leak.

Add one nested for binder to a passing case.

Log creation universe for every inference variable and placeholder.

Find the first assignment containing an invisible placeholder.

Verify rollback removes it after a failed probe.

Workshop B: GAT argument swap.

Use different unmistakable types for every parent and local slot.

Print declaration parameter indices and projection arguments.

Check the error before normalization.

If already wrong, the solver is downstream of the defect.

Workshop C: const mismatch.

Replace a generic expression with a concrete value.

If the concrete case works, separate inability to prove evaluatability from actual unequal values.

Inspect owner-aware unevaluated const identity before CTFE.

Workshop D: opaque leak.

Compare debug output and user diagnostics inside and outside the defining crate.

If only the printer leaks, preserve semantic mode and repair presentation.

If ordinary code can use hidden-only methods, inspect normalization mode and cache key immediately.

Workshop E: TAIT disagreement.

Reduce to two authorized defining uses.

Replace generic arguments by distinct marker types.

Print local candidate, remapped family, and spans.

Determine whether disagreement is real or caused by failed reverse substitution.

Contribution ladder#

1. Add a focused regression reproducing one current diagnostic.
2. Improve binder or generic-argument debug formatting without semantic changes.
3. Add an assertion that no local inference ID enters an opaque hidden family.
4. Improve a GAT required-bound span.
5. Add a const-evaluatability cause through one query boundary.
6. Add a cross-crate opaque reveal regression.
7. Property-test alpha-renaming under nested binders.
8. Measure and reduce one repeated normalization workload.
9. Review a TAIT change for authorization and generic remapping.
10. Change semantic policy only with language/types-team guidance and compatibility evidence.

Every contribution should name its invariant, first failing representation, solver boundary, tests, and performance risk.

104. Capstone, philosophy, talks, and authoritative reading#

Bounded capstone#

Build one dependency-free “advanced term laboratory” in stable Rust.

Milestone 1 defines owner-indexed type, lifetime, and const parameters.

Invariant: equal numeric indices from different owners never compare equal.

Milestone 2 adds binders and de Bruijn indices.

Invariant: shift-then-instantiate preserves free-variable ownership.

Milestone 3 adds universes, placeholders, and inference variables.

Invariant: older variables never contain newer placeholders.

Milestone 4 adds a generic associated projection with parent/local argument partitioning.

Invariant: all arguments follow declaration metadata and all where-clauses are returned.

Milestone 5 adds symbolic const Value, Param, and checked Add.

Invariant: unknown, overflow, and value are distinct outcomes.

Milestone 6 adds opaque definitions, authorized defining uses, and hidden candidates.

Invariant: stored hidden families contain opaque parameters, not body-local inference IDs.

Milestone 7 adds User and Defining reveal modes.

Invariant: mode changes observability only at authorized boundaries and is included in cache keys.

Milestone 8 adds structured provenance and errors.

Invariant: every rejection names the earliest failed operation and source-like location.

Milestone 9 adds property tests.

Check alpha-renaming, shift round trips, candidate permutation, owner separation, cache-on/cache-off equivalence, and serialization round trips.

Milestone 10 adds representative benchmarks.

Measure deep binders, many GAT applications, large const expressions, and many opaque defining sites independently.

Explicit capstone omissions#

The laboratory is not rustc.

It omits trait candidate search, coherence, specialization, variance, subtyping, region inference, MIR borrow checking, CTFE, macros, HIR, interning, incremental compilation, metadata compatibility, feature gates, cancellation, parallel queries, and production diagnostics.

Mock trait answers must stay visibly mocked.

Const arithmetic must not claim target usize behavior unless target information is modeled.

Opaque bounds must not be called verified unless a real proof service exists.

These omissions bound the proof, rather than hiding missing work.

Mastery exercises#

1. Translate three HRTBs into explicit universal statements and identify who chooses each lifetime.
2. Draw de Bruijn indices for four nested binders with references to every outer binder.
3. Implement capture-avoiding shifting and property-test alpha-renaming.
4. Construct an indirect universe leak through two inference variables.
5. Explain why canonical variable, placeholder, and bound variable 0 are different identities.
6. Lower a GAT with trait type and const parameters plus local lifetime and type parameters.
7. Derive every WF obligation from one lending trait and implementation.
8. Trace a higher-ranked GAT projection to the solver boundary and back.
9. Distinguish a true borrow conflict from an implied-'static limitation.
10. Design a structural const parameter policy and attack it with NaN, privacy, and semver cases.
11. Explain why abstract const normalization cannot use unrestricted algebra.
12. Separate const well-typedness, evaluatability, validity, and layout.
13. Compare RPIT identity for two functions with equal hidden types.
14. Explain async return opacity without claiming source-level desugaring identity.
15. Map RPITIT trait, impl, synthetic associated, and opaque arguments.
16. Design a TAIT defining-use authorization checker.
17. Reverse-map a generic hidden candidate into opaque parameters.
18. Create a cross-crate test proving user mode cannot reveal a hidden type.
19. State every semantic input needed by a normalization cache key.
20. Locate the earliest invariant in a real rustc regression and defend the classification.

Derived philosophy#

Quantifier placement assigns power.

Moving for<'a> changes who chooses a lifetime.

Syntax that looks like punctuation can reverse the proof obligation.

Representations determine cheap questions.

De Bruijn indices make alpha-equivalence and capture-safe traversal practical.

They make source explanation expensive unless provenance survives.

Abstract consts make symbolic dependency visible.

They do not make arbitrary theorem proving cheap.

Abstraction moves responsibility.

An opaque return simplifies callers while creating capture, hidden-type consistency, reveal, metadata, and diagnostic obligations for the compiler.

TAIT broadens where one identity can be used and therefore makes authorization and generic remapping harder than RPIT.

Identity is not spelling or representation.

Equal hidden concrete types do not merge opaque identities.

Equal parameter numbers do not merge owners.

Equal printed const expressions do not prove semantic equality.

Uncertainty must remain explicit.

A symbolic projection, unevaluated const, inference variable, or ambiguous solver result is honest state.

Forcing a convenient answer converts phase order into accidental language policy.

Caching creates correctness obligations.

Universe, environment, owner, substitutions, target mode, and reveal policy can all affect answers.

Omitting one input turns a speed optimization into cross-context proof forgery.

Local simplicity can create global complexity.

Eager normalization helps one pattern match while increasing cycles, work, and abstraction leaks.

Silent GAT bound inference shortens one declaration while hiding a public contract.

Running user equality for const parameters simplifies admissibility wording while making type identity effectful and unstable.

The first visible failure is late.

A borrow error may begin as a binder shift.

A trait ambiguity may begin as a GAT argument swap.

A layout failure may begin as a dropped evaluatability obligation.

An opaque method leak may begin as a cache hit under the wrong reveal mode.

Debugging means walking backward to the earliest broken invariant.

Thirty-minute talk#

1. Four minutes: existential versus universal choice.
2. Four minutes: binders, de Bruijn indices, placeholders, and universes.
3. Five minutes: GATs as associated type families and projection questions.
4. Five minutes: const terms, abstract consts, and evaluatability.
5. Five minutes: opaque identity, hidden type, capture, and reveal.
6. Four minutes: TAIT defining uses and cross-crate abstraction.
7. Three minutes: one earliest-invariant debugging trace.

Sixty-minute contributor talk#

Begin by lowering one HRTB and tracing its leak check.

Implement the tiny universe checker and demonstrate its omissions.

Lower a mixed-parameter GAT and partition arguments from metadata.

Trace normalization only to and from the Part IV-C solver boundary.

Build a symbolic const expression and separate unknown from overflow.

Compare RPIT, async return, RPITIT, associated opaque forms, and TAIT identities.

Walk TAIT hidden-family reverse mapping and authorization.

End with source paths, a cross-crate regression, performance counters, and review questions.

Authoritative reading map#

Implementation statements here target Rust 1.97.1.

Internal modules are unstable, and nightly behavior can differ before and after that release.

• Rust 1.97.1 source tree
• Rust Reference: higher-ranked trait bounds
• Rust Reference: generic parameters
• Rust Reference: associated items
• Rust Reference: impl Trait
• Rust Reference: async functions
• rustc-dev-guide: higher-ranked trait bounds
• rustc-dev-guide: type inference
• rustc-dev-guide: opaque types
• rustc-dev-guide: normalization
• rustc-dev-guide: const evaluation
• rustc-dev-guide: next solver
• nightly rustdoc: rustc_middle::ty
• nightly rustdoc: rustc_infer
• nightly rustdoc: rustc_next_trait_solver
• RFC 1598: generic associated types, as historical design context
• RFC 2000: const generics, as historical design context
• RFC 2515: type alias impl trait, as historical design context, not current stability documentation
• Rust compiler tests guide

The Reference is the first source for stable language behavior.

Feature documentation and the 1.97.1 tests establish the status of TAIT, generic const expressions, const parameter types, and precise capture at that revision.

The matching rustc source establishes implementation structure.

The dev guide explains architecture but may lag a moving implementation.

RFCs explain motivation and historical decisions; they do not override stabilized behavior or current feature restrictions.

Final checklist#

Before changing this subsystem, answer:

1. Which definition owns every generic argument and bound variable?
2. Who chooses each quantified lifetime?
3. Which placeholders can each inference variable see?
4. Which GAT bounds make the projection well formed?
5. Is a const unknown, unevaluatable, invalid, or merely unnormalized?
6. Which opaque identity is involved, and is this an authorized defining use?
7. What reveal mode and environment entered the query?
8. Which work belongs to Part IV-C's solver rather than this caller?
9. What source provenance must survive for the diagnostic?
10. What cross-crate, incremental, soundness, and performance test distinguishes the repair?

Advanced types become manageable when these questions remain separate.

Their complexity is not ornamental.

It is the cost of preserving caller choice, associated families, symbolic values, and abstraction without sacrificing soundness or separate compilation.

Part V: THIR, MIR, Dataflow, Patterns, and Constant Evaluation#

1. Why another representation exists#

Rust source is pleasant for humans but awkward for flow-sensitive reasoning. The compiler therefore translates one representation into another as questions become more specific. HIR, the High-level Intermediate Representation, records resolved, desugared Rust items and bodies. It remains close enough to source structure to support name resolution, type checking, and diagnostics. That source-shaped structure is not the ideal final form for every later analysis.

Consider xs[i] += f()?. Its meaning depends on overloaded indexing, coercions, autodereferencing, method lookup, and Try desugaring. Type checking decides which implementations and types those operations use. It also records adjustments that are not obvious in the HIR node itself. An adjustment is a compiler-inserted conversion, such as borrowing, dereferencing, or coercing. Later consumers must see those decisions consistently rather than repeat type inference.

Control-flow questions also want a graph, not a nested syntax tree. “Was this field initialized on every path?” requires joining facts from predecessor paths. “Can this borrow overlap that mutation?” requires precise program points and successor edges. “What runs if this call unwinds?” requires explicit exceptional control flow. Nested HIR can encode all this indirectly, but every analysis would rebuild it differently.

The broad pipeline is therefore:

source -> AST -> HIR -> type checking results -> THIR -> built MIR
       -> analysis and transformation MIR forms -> optimized MIR -> code generation
                              \-> MIR interpretation for constants

This is a map, not a promise that every body visits one globally fixed sequence. Queries can request particular products lazily. Compiler-generated shims and promoted bodies have special origins. The exact query names, phases, and pass ordering change between compiler versions.

This chapter targets rustc 1.97.1 concepts. Names from compiler internals are explicitly version-sensitive unless part of stable Rust. The important design lesson is stable: representations should expose the distinctions their consumers need.

Bug clue: if a later pass appears to infer source sugar again, inspect the type-check adjustments and lowering boundary. Duplicated semantic reconstruction often causes disagreements on coercions, overloaded operations, and temporary lifetimes.

2. From type-check results to adjusted expressions#

Type checking does more than attach one type to each expression. It resolves methods, operators, fields, associated items, coercions, and inference variables. It records whether an expression is used as a value or as a place. A place denotes storage that can be read, written, borrowed, or moved from. A value is the result obtained from computation.

For example, s.len() may involve autodereferencing s, then autoref-borrowing the receiver. The source has one method-call expression. The checked meaning resembles a selected function plus an explicit receiver conversion. Similarly, x + y can mean a built-in integer operation or an Add::add call. MIR must not guess which one was selected.

An adjusted expression is the original expression interpreted through its recorded adjustment chain. Typical adjustments include dereference, borrow, pointer coercion, and unsizing. Their ordering matters. Borrowing and then coercing is not generally interchangeable with coercing and then borrowing.

fn takes_slice(_: &[i32]) {}

fn example(a: &[i32; 3]) {
    takes_slice(a);
}

The call argument's checked interpretation includes conversion from an array reference to a slice reference. The exact internal adjustment sequence is version-sensitive. The semantic invariant is that lowering preserves the type checker's chosen conversion and evaluation order.

HIR remains valuable because it retains item-level and source-oriented information. Replacing HIR globally with a low-level graph would make many diagnostics and language checks harder. Conversely, annotating HIR forever with every later detail would create a large, tangled representation. rustc instead builds body-oriented forms suitable for narrower jobs.

Alternative: every analysis could walk HIR with side tables. That saves one lowering but multiplies complexity across consumers. Alternative: lower directly from HIR to a control-flow graph. That can work, but a typed structured intermediate form gives pattern and unsafety analyses a shared explicit view.

Bug clue: an incorrect implicit borrow usually starts before MIR optimization. Compare inferred types and adjustments, then inspect THIR and built MIR. Do not begin by blaming LLVM when the wrong callee or receiver appears in built MIR.

3. THIR: a typed, explicit body view#

THIR means Typed High-level Intermediate Representation. It is generated after type checking and represents executable bodies, not an entire crate's item hierarchy. Function bodies, constant initializers, and other body owners can receive THIR. Struct and trait declarations are not themselves THIR bodies.

THIR is more explicit than HIR. Types are available on expressions and patterns. Automatic references and dereferences can be represented explicitly. Method calls and overloaded operators can become ordinary calls selected by type checking. Destruction scopes carry information needed to arrange temporary and local destruction.

Yet THIR is still structured around expressions, statements, blocks, arms, and patterns. That makes it suitable for operations which benefit from typed source-like structure. Current documented users include MIR construction, exhaustiveness checking, and unsafety checking. The exact set is an implementation detail and may change.

THIR is not simply “typed HIR stored forever.” It has additional lowering, a body-only scope, and a different storage strategy. Current rustc allocates a body's THIR temporarily and can discard it after use. This reduces peak memory compared with retaining all THIR bodies for the whole compilation. Its indexed arenas may hold expressions, blocks, statements, arms, and parameters separately.

The temporary/query nature affects compiler engineering. A consumer should request the appropriate query product rather than retain references beyond their arena lifetime. Recomputing and query ownership rules matter for memory and correctness. Internal APIs around thir_body, arenas, and returned ownership are version-sensitive.

THIR is also deliberately unstable as a public format. Expression variants, fields, and dumps evolve with language implementation needs. Tools should not parse its debug output as a stable protocol. For robust tooling, prefer stable compiler interfaces where available or pin an exact nightly toolchain.

Invariant: every THIR expression used by lowering has a coherent type after substitutions and adjustments. Invariant: source scopes and destruction scopes preserve language lifetime behavior. Bug clue: a missing scope wrapper can become a temporary dropped too early or too late. Bug clue: an explicit call with the wrong substitutions points toward type-dependent lowering.

4. Why patterns need their own analysis#

A pattern is not merely a boolean test. It can test a constructor, destructure fields, bind names, borrow subplaces, and move values. Several patterns in order define both coverage and control flow. The compiler must detect unreachable arms and non-exhaustive matches.

enum Signal { Stop, Data(u8), Pair(bool, bool) }

fn classify(s: Signal) -> u8 {
    match s {
        Signal::Stop => 0,
        Signal::Data(0) => 1,
        Signal::Data(_) => 2,
        Signal::Pair(false, false) => 3,
        Signal::Pair(_, _) => 4,
    }
}

Usefulness asks whether a new pattern matches any value not already matched by previous patterns. Exhaustiveness asks whether a wildcard pattern would still be useful after all arms. If wildcard remains useful, its uncovered values can become diagnostic witnesses. A witness is an example pattern describing a missing region of the value space.

Naively enumerating values is impossible for integers, strings, slices, and recursive types. The analysis instead reasons symbolically about constructors. A constructor describes one top-level way to make or partition values of a type. For an enum, variants are constructors. For a tuple, the tuple shape is one constructor with fields. For booleans, false and true are constructors. Integer ranges use interval-like constructor partitions rather than every integer.

The analysis is often described using a pattern matrix. Rows are earlier patterns or pattern vectors. Columns correspond to scrutinee components. Specialization chooses a constructor and rewrites rows to its fields. This reduces a top-level question to smaller questions about subpatterns.

This abstraction derives directly from the question being answered. Coverage depends on sets of possible values. Constructors partition that set, while specialization recursively explores relevant partitions. The implementation's exact types and algorithm refinements are version-sensitive.

5. Constructors, specialization, and witnesses#

Suppose the type is Option<bool> and previous rows are Some(true) and None. The top-level constructors are Some and None. Specializing on None consumes no fields and finds it covered. Specializing on Some exposes one boolean field. The previous Some(true) row becomes true; candidate wildcard becomes wildcard. The boolean constructor false remains uncovered. Reconstructing outward yields witness Some(false).

matrix                  specialize Some       specialize boolean
Some(true)              true                  true covered
None                    removed               false uncovered

witness inside: false
reconstructed witness: Some(false)

A wildcard can stand for constructors not explicitly mentioned. The algorithm must compute which constructor set is relevant for the type and matrix. For large scalar domains, it partitions around boundaries appearing in patterns. This avoids iterating across all u128 values.

Witnesses serve humans, not just proofs. The compiler may compress multiple missing values into a readable pattern. Diagnostic formatting is separate from the logical fact of non-exhaustiveness. A poor witness can be confusing even when the coverage answer is correct.

Invariants include constructor arity matching the number of specialized fields. Type normalization and constructor selection must agree. Bindings do not reduce coverage; the subpattern beneath a binding does. An @ pattern combines binding with its constrained subpattern.

Alternative algorithms include decision trees, automata, or SAT-like encodings. Pattern matrices naturally support recursive algebraic data types and useful witnesses. Their hard cases include blowups from nested alternatives and subtle type inhabitedness. Implementations use memoization, constructor grouping, and careful complexity controls.

Bug clue: a wrong missing-pattern suggestion can indicate witness reconstruction rather than coverage failure. Bug clue: only nested tuple cases failing suggests specialization arity or column handling. Bug clue: exponential compile time often involves deeply nested or-patterns or many overlapping ranges.

6. Guards and or-patterns#

A match guard is evaluated after its pattern matches. Because an arbitrary guard may return false, a guarded arm generally cannot prove later values unreachable. The guard may read bindings and run code, so coverage cannot treat it as a pure constructor constraint.

fn sign(x: i32) -> &'static str {
    match x {
        n if n < 0 => "negative",
        0 => "zero",
        _ => "positive or guarded-negative",
    }
}

The final wildcard remains necessary because the first guard can fail from the coverage algorithm's perspective. Even if a human sees n < 0, rustc does not generally prove arbitrary guard predicates. A constant-true guard may receive special handling in some versions, but do not build a model around incidental behavior.

An or-pattern A | B denotes the union of alternatives. Conceptually it can expand into multiple matrix rows. Real implementations avoid indiscriminate full expansion because nested alternatives can grow exponentially. All alternatives must bind compatible names with compatible binding modes and types.

fn bit(x: Option<u8>) -> bool {
    match x {
        Some(0 | 1) => false,
        Some(2..=255) => true,
        None => false,
    }
}

For usefulness, alternatives contribute coverage as a union. For lowering, they converge into a common arm body with bindings initialized consistently. These are related but distinct jobs. Coverage may reject a redundant alternative even though code generation could execute it correctly.

Invariant: a guard executes only after all required pattern tests and bindings succeed. Invariant: failed alternatives do not leave partially initialized user bindings visible. Bug clue: a binding available on only one alternative signals an earlier type-checking error, not a MIR join solution.

7. Slices, ranges, and uninhabited types#

Slice patterns combine fixed prefixes, fixed suffixes, and an optional variable-length middle. [first, middle @ .., last] matches lengths at least two. Usefulness must reason about length constructors and element subpatterns. It cannot enumerate every possible slice length.

fn shape(xs: &[u8]) -> &'static str {
    match xs {
        [] => "empty",
        [_] => "one",
        [0, ..] => "many starting zero",
        [_, ..] => "many",
    }
}

The last two arms collectively cover lengths at least two after the earlier singleton case. Arrays have a statically known length, while slices do not. Pattern analysis uses that type distinction.

Range patterns represent contiguous scalar regions. The algorithm partitions at endpoints so overlaps become finite symbolic intervals. Inclusive and half-open syntax must be interpreted according to current language rules. Character validity introduces a domain distinction from arbitrary integers. Floating-point patterns have separate restrictions and should not be extrapolated from integer ranges.

An uninhabited type has no valid values, with ! as the clearest example. Inhabitedness is subtle in generic, visibility, non-exhaustive, and cross-crate contexts. A compiler cannot always use private knowledge from another crate to declare a public match exhaustive. Generic parameters may be instantiated with inhabited types. References and unsafe code also require care when reasoning about impossible values.

Therefore, avoid the slogan “empty enum means every match is automatically exhaustive” without context. The answer depends on type normalization, module visibility, feature semantics, and compiler version. The usefulness implementation has explicit inhabitedness rules rather than one universal syntactic test.

Bug clue: cross-crate-only exhaustiveness differences point to visibility or non_exhaustive handling. Bug clue: array versus slice discrepancies point to length constructor logic. Bug clue: boundary failures at minimum or maximum scalar values suggest interval splitting or overflow.

8. MIR as a typed control-flow graph#

MIR means Mid-level Intermediate Representation. For a function-like body, MIR is a typed control-flow graph, abbreviated CFG. A graph contains basic blocks connected by control-flow edges. A basic block executes sequentially and ends in exactly one terminator. The terminator chooses successors, returns, unwinds, or otherwise transfers control.

MIR deliberately has fewer constructs than Rust source. Nested if, match, and loops become blocks and edges. Method calls are no longer a special call category. Many implicit operations become explicit assignments, borrows, projections, assertions, and calls.

This shape directly supports its consumers. Borrow checking needs exact places and program points. Definite-initialization analysis needs assignments, moves, and joins. CTFE needs an executable instruction-like form. Optimization needs small operations with explicit dependencies and control flow.

MIR is typed, unlike a raw assembly listing. Every local declaration has a Rust type. Places and rvalues must obey typing rules for the body's current MIR phase. Source information remains attached for diagnostics and debugging.

Do not imagine one immutable “the MIR.” There are built, analyzed, transformed, promoted, borrow-check-related, and optimized forms. Passes establish and consume phase-specific invariants. Some constructs are legal only before or after particular transformations. Not every pass may consume every phase.

The pretty-printed MIR syntax is a debugging format, not a stable language. Diagrams in this chapter are conceptual sketches. They omit details and may not match rustc 1.97.1 dumps character for character.

9. Body, locals, blocks, and source information#

Body is the container for one MIR body. Conceptually it owns basic blocks, local declarations, argument metadata, source scopes, debug information, and phase metadata. Exact fields vary with compiler version.

A Local is a compact index such as _0, _1, or _2. _0 conventionally names the return place. Arguments follow, then user variables and compiler-created temporaries. A LocalDecl describes a local's type, mutability-related metadata, source information, and other classification details.

fn add_one(x: i32) -> i32

_0: i32    return place
_1: i32    argument x
_2: i32    temporary for checked addition result, if needed

A BasicBlock is an index into block data. Its block data contains zero or more statements and a terminator. There is one distinguished start block. Only graph-reachable blocks matter for execution, though transient unreachable blocks can exist between passes.

SourceInfo associates MIR entities with a source span and source scope. Spans identify source ranges. Source scopes model lexical/debugging context and inlining ancestry. Optimizations must decide how to preserve useful locations when combining or deleting operations.

Debug information and semantic execution information overlap but are not identical. Removing a dead assignment may be semantically harmless yet degrade variable inspection. Compiler options and optimization levels influence this tradeoff.

Invariant: each local use is type-compatible with its declaration in the current phase. Invariant: each block has one valid terminator once construction is complete. Bug clue: diagnostics pointing to nonsensical source often indicate lost or incorrectly propagated SourceInfo, not wrong execution.

10. Places and projections#

A MIR Place identifies storage. It begins with a local or another permitted base and follows zero or more projections. A projection selects a component or changes how storage is reached. Common conceptual projections include dereference, field, index, constant index, subslice, and downcast.

_1                         local place
(_1.0)                     tuple or struct field zero
(*_2)                      dereference
((*_2).field)              dereference then field
_3[_4]                     runtime index
((_5 as Some).0)           enum downcast then variant field

The exact pretty syntax and projection variants are version-sensitive. A downcast does not perform a runtime branch by itself. It records that control flow has established the active enum variant before accessing fields.

Places make partial moves expressible. Moving pair.0 need not move pair.1. They also make alias-sensitive operations precise: borrowing one field may differ from borrowing an entire aggregate. Packed fields, unions, raw pointers, and dynamic indices introduce additional safety complications.

Projection typing is sequential. Starting from the base type, each projection computes the next type. Dereferencing requires a pointer-like type appropriate to the operation. Field selection requires the correct aggregate and variant context. Indexing requires an index local and an indexable base.

Alternative: flatten every field into a separate local. That works poorly for references, dynamic indexing, unsized values, and aggregates passed as wholes. Alternative: use arbitrary expression trees as lvalues. That obscures side effects and evaluation order. MIR places occupy a useful middle ground.

Bug clue: a move error on the whole object instead of one field may come from an overly broad place. Bug clue: wrong enum field access often means a missing discriminant test or downcast.

11. Operand, Rvalue, Statement, and Terminator#

An Operand supplies an already available input to an operation. Conceptually it is a copy from a place, a move from a place, or a constant. Copying leaves the source initialized. Moving can make the source or move path uninitialized. Whether a type is Copy constrains which form is valid.

An Rvalue computes a value without itself naming a destination. Examples include using an operand, binary operations, references, casts, aggregates, lengths, and discriminants. An assignment combines them: destination = Rvalue. Some variants and semantics differ by MIR phase.

A Statement performs an operation that continues to the next statement in the same block. Assignments and storage markers are familiar examples. Statements do not select a CFG successor. Instrumentation and analysis-only statement kinds can exist in particular dialects.

A Terminator ends a block. Examples include Goto, SwitchInt, Call, Drop, Assert, Return, Unreachable, and coroutine-related control transfer. Calls are terminators because normal return and possible unwind create control-flow outcomes. Exact unwind edge representation depends on compiler version and panic strategy.

bb0:
    StorageLive(_2)
    _2 = copy _1
    _0 = Add(copy _2, const 1_i32)
    StorageDead(_2)
    Return

This sketch ignores overflow checks. Debug builds may represent checked arithmetic and an assertion. Optimizations may fold or remove locals and markers.

Invariant: statements precede the sole terminator. Invariant: operand move/copy choice respects ownership semantics. Invariant: successor argument and destination behavior agrees with the terminator. Bug clue: use-after-move diagnostics require checking operand kind and move path, not merely textual variable names.

12. Lowering arithmetic and evaluation order#

Rust specifies important evaluation-order behavior, and MIR makes the chosen sequence explicit. Subexpressions are lowered into temporaries before a final operation when necessary. This prevents an optimizer or later analysis from accidentally re-evaluating side effects.

fn f() -> i32 { 10 }
fn g() -> i32 { 20 }
fn sum() -> i32 { f() + g() }

Conceptual built MIR:

bb0:
    Call f() -> _1, return bb1, unwind cleanup?
bb1:
    Call g() -> _2, return bb2, unwind cleanup?
bb2:
    _3 = CheckedAdd(copy _1, copy _2)
    Assert(no_overflow(_3)) -> bb3, unwind/panic edge?
bb3:
    _0 = value(_3)
    Return

This is a conceptual sketch; actual checked-operation representation and cleanup edges vary. The key fact is that f runs before g, and each runs once. At an optimization level permitting it, pure constant expressions may collapse. Calls with observable effects cannot simply be reordered.

Built-in operations and overloaded operations lower differently. An overloaded a + b becomes a selected trait method call after type checking. A built-in integer addition can become an MIR binary operation, with overflow policy represented appropriately.

Compound assignment must evaluate its destination carefully. For a[index()] += rhs(), the base and index cannot be recomputed arbitrarily. The lowering creates places and temporaries so side effects occur in language order.

Bug clue: duplicated output from a side-effecting index expression suggests erroneous place reconstruction. Bug clue: release-only arithmetic differences may involve overflow-check configuration or optimization, not ordering.

13. Lowering if and match#

An if becomes a condition operand and a branch terminator. Both arms eventually join if control continues.

fn choose(flag: bool, a: i32, b: i32) -> i32 {
    if flag { a } else { b }
}

bb0: SwitchInt(copy _1) -> false: bb2, otherwise: bb1
bb1: _4 = copy _2; Goto bb3
bb2: _4 = copy _3; Goto bb3
bb3: _0 = copy _4; Return

The arm-result temporary makes the expression's single resulting value explicit. Optimization can copy-propagate _4 or restructure blocks later.

A match first evaluates its scrutinee according to source semantics. Tests form a decision tree or graph. Enum matching typically reads the discriminant and branches with SwitchInt. Within a proven variant, projected fields are tested or bound. Pattern order and guards constrain decision-tree sharing.

fn unwrap_or(x: Option<i32>, d: i32) -> i32 {
    match x {
        Some(v) if v > 0 => v,
        Some(_) => d,
        None => 0,
    }
}

bb0: _3 = Discriminant(_1); SwitchInt(_3) -> None: bb4, Some: bb1
bb1: _4 = copy ((_1 as Some).0); _5 = Gt(_4, 0); SwitchInt(_5) -> false: bb3, true: bb2
bb2: _0 = copy _4; Goto bb5
bb3: _0 = copy _2; Goto bb5
bb4: _0 = const 0; Goto bb5
bb5: Return

Real lowering also handles binding modes, moves, drops, scopes, and false-unwind details. Usefulness checking happens conceptually before code generation of an exhaustive decision graph. It does not itself replace runtime tests.

14. Lowering loops, calls, indexing, and borrows#

A loop is naturally a back edge. break targets an exit block, while continue targets a loop continuation point.

fn count(mut n: u32) -> u32 {
    let mut x = 0;
    while n != 0 { x += 1; n -= 1; }
    x
}

bb0: initialize x; Goto bb1
bb1: test n != 0; SwitchInt -> false: bb3, true: bb2
bb2: checked increment x; checked decrement n; Goto bb1
bb3: _0 = copy x; Return

A call evaluates callee and arguments, then terminates its block. Normal return writes the destination and enters a successor. If unwinding is possible and enabled, an unwind action leads toward cleanup. Targets using abort panic strategy may not have executable unwind cleanup edges. Foreign ABIs and nounwind knowledge further affect behavior.

Indexing computes the index once, obtains the length, and checks bounds before projecting.

_idx = call index()
_len = Len(_slice)
_ok = Lt(_idx, _len)
Assert(_ok, bounds-check message) -> in_bounds
in_bounds: _value = copy _slice[_idx]

A borrow creates a reference rvalue to a place. Shared, mutable, raw, and two-phase-borrow-related details differ. The borrow checker reasons about permission and lifetime using MIR and inferred regions.

fn first(xs: &mut [i32]) -> &mut i32 {
    &mut xs[0]
}

The conceptual graph dereferences xs, checks length against zero, then creates a mutable reference to the indexed place. The check must precede creation of a valid element reference.

15. Temporaries, storage, and drop scopes#

Temporaries hold intermediate values so evaluation occurs exactly once and at defined points. Their destruction time is a language-semantic issue, not merely a stack-allocation optimization. THIR destruction scopes help MIR construction place drops correctly.

StorageLive(_n) and StorageDead(_n) describe when storage for a local is live. They are not the same as initializing and dropping the local's value. Storage can be live while its value is uninitialized. A value can be dropped before its storage becomes dead.

StorageLive(_1)
_1 = make_string()
Drop(_1) -> next
next:
StorageDead(_1)

This distinction supports stack-slot reuse and validates accesses. Optimization may remove storage markers when they are no longer useful. Their exact downstream effect is backend- and phase-dependent.

A drop scope is the source-language region at whose exit a value must be destroyed, if initialized. Exits include normal fallthrough, break, return, and unwind where applicable. Lowering must schedule destruction in reverse ownership order where required. Temporary lifetime extension rules can make a temporary outlive the immediate expression.

Never infer drop order solely from local numbers in a dump. Inspect explicit drops and cleanup paths in the relevant MIR phase. Before drop elaboration, drops may still be conditional in a higher-level sense. After elaboration, drop flags and branches make conditional destruction explicit.

Bug clue: double destruction suggests incorrect initialization/drop-flag transitions. Bug clue: a destructor omitted on break suggests a scope-exit edge bypassing cleanup. Bug clue: stack-use-after-scope may involve storage lifetime, but semantic borrow validity is a separate question.

16. Cleanup, unwind, and analysis-only edges#

Panics and some calls may unwind, transferring control through cleanup blocks that destroy live values. Cleanup blocks are marked so validation and code generation understand their role. An unwind edge is not universally present or executable. Its existence depends on target capabilities, ABI, operation properties, and panic strategy.

With panic=abort, a panic terminates rather than unwinds through ordinary Rust cleanup. With unwinding enabled, cleanup paths must avoid illegal transitions back into normal control flow. Double panic during cleanup may abort. Exact MIR unwind actions and cleanup representations are version-sensitive.

Older and current MIR discussions mention false edges, false unwind edges, and fake reads. These are analysis-oriented concepts used at certain phases to expose conservative control-flow or reads. For example, a fake read can make a match scrutinee's use visible to an analysis without being a machine load. A false edge can influence reachability reasoning without becoming ordinary runtime behavior.

Do not assume these constructs exist unchanged in rustc 1.97.1 or survive to optimized MIR. They are useful concepts: an analysis sometimes needs edges or effects beyond executable machine control flow. The current enum definitions and phase validators are authoritative for a pinned toolchain.

Invariant: executable unwind paths destroy exactly the values requiring cleanup and never treat uninitialized values as initialized. Invariant: analysis-only constructs are removed before consumers that cannot interpret them. Bug clue: a backend ICE mentioning a fake or false construct usually means a phase transition failed to eliminate it.

17. MIR phases, dialects, and query products#

MIR evolves through dialects. A dialect is a family of allowed constructs and invariants, not merely an optimization level. The body's phase records enough information for passes and validation to reject illegal combinations. Names and subdivisions have changed across rustc releases.

Built MIR is close to direct THIR lowering. It contains scopes, temporaries, and constructs useful to early checks. Promotion identifies eligible expressions and creates promoted MIR bodies. Borrow checking consumes a suitable analyzed form with ownership and region information. Drop elaboration and other transformations prepare later forms. Optimized MIR is prepared for code generation and has undergone configured optimization passes.

“Borrowck MIR” is often conversational shorthand, not necessarily one stable public query type. Likewise, “promoted MIR” can refer to a body extracted for a promoted constant and to a stage containing promotion results. Read exact query/API documentation for the target revision.

Crucially, never write a pass that accepts arbitrary Body and assumes every variant is meaningful. A pass should declare or enforce the phase it consumes and the phase/invariants it produces. Some optimization passes require drops already elaborated. Some early analyses require constructs later removed. Codegen expects a constrained final dialect.

Alternative: one enormous MIR enum valid at all times with every pass defensive. That weakens invariants and makes latent bugs survive longer. Phase validation turns accidental misuse into an earlier internal compiler error.

Bug clue: an ICE only under -Zmir-opt-level changes may identify a pass-order or phase precondition. Dump immediately before and after the first divergent pass.

18. Move paths and partial initialization#

Ownership analysis needs more precision than one initialized bit per local. After moving x.0, x.1 may remain usable. A move path models a place and relevant descendants that can be tracked independently.

fn partial(pair: (String, String)) -> String {
    let left = pair.0;
    println!("{}", pair.1);
    left
}

Conceptually, move paths form a tree:

pair
|- pair.0
`- pair.1

Moving the parent makes descendants unavailable. Moving one child may make the parent as a whole unavailable while preserving a disjoint sibling. Assignments can reinitialize a path. Dynamic indices generally cannot be distinguished as neatly as fixed fields because aliases between indices are unknown.

Definite initialization asks whether a place is initialized on every path reaching a program point. Maybe initialization asks whether it is initialized on at least one path. Different checks use differently oriented facts. A move records deinitialization; an assignment records initialization; a drop requires appropriate initialization.

fn maybe(flag: bool) -> String {
    let x: String;
    if flag { x = String::from("yes"); }
    x
}

At the join, one predecessor has initialized x and one has not. Therefore x is not definitely initialized, and the final use is rejected.

Move paths must account for destructors and types whose fields cannot be moved independently. Moving out of a type implementing Drop is restricted because its destructor expects a coherent value. Unions and dereferences have additional aliasing constraints.

Bug clue: a false positive after reassigning a moved field suggests missing reinitialization propagation. Bug clue: a false negative at a join suggests union was used where intersection was required for definite facts.

19. Dataflow from first principles#

Dataflow analysis computes facts at program points by repeatedly propagating information along CFG edges. First choose a domain: the set of possible abstract states. For initialized locals, a state might be a set of local indices. The state abstracts many concrete executions into one finite description.

A partial order says when one abstract state contains no more information than another. A join combines facts arriving from multiple predecessors. For “possibly initialized,” set union is a natural join. For “definitely initialized,” set intersection can represent the meet-like combination, often encoded by reversing the order. Terminology varies, so define the ordering and operation rather than relying on “top” and “bottom” alone.

A transfer function describes how a statement changes state. Assignment generates initialization. Move kills initialization. Other statements preserve the set in this tiny model. Real rustc analyses distinguish effects before and after statements and terminators.

A fixpoint is reached when another full propagation changes no states. Loops require iteration because a back edge can add facts to an earlier block. If the domain has finite height and transfer/join functions are monotone, ascending iteration terminates. Monotone means providing a larger input cannot produce a smaller output under the chosen order.

A worklist avoids rescanning every block after every change. When a block's output changes, enqueue its successors. Processing order affects speed but not the least fixpoint under the usual conditions. Reverse postorder is often effective for forward analyses, though correctness must not depend on it.

Bug clue: nontermination indicates an infinite-height domain, non-monotone transfer, or equality instability. Bug clue: predecessor-order-dependent answers indicate a non-associative or mutating join.

20. A stable-Rust forward dataflow engine#

The following complete example uses only stable Rust library APIs. It computes locals that may be initialized on entry to and exit from each block. The tiny CFG has block-local Gen and Kill effects. Union is the join, so a fact means “initialized on at least one reaching path.”

use std::collections::{BTreeSet, VecDeque};

type Fact = BTreeSet<usize>;

#[derive(Clone, Debug)]
enum Effect {
    Gen(usize),
    Kill(usize),
}

#[derive(Clone, Debug)]
struct Block {
    effects: Vec<Effect>,
    successors: Vec<usize>,
}

#[derive(Debug, PartialEq, Eq)]
struct Results {
    entry: Vec<Fact>,
    exit: Vec<Fact>,
}

fn transfer(mut state: Fact, block: &Block) -> Fact {
    for effect in &block.effects {
        match *effect {
            Effect::Gen(local) => {
                state.insert(local);
            }
            Effect::Kill(local) => {
                state.remove(&local);
            }
        }
    }
    state
}

fn analyze(blocks: &[Block], start: usize, boundary: Fact) -> Results {
    assert!(start < blocks.len());
    for block in blocks {
        assert!(block.successors.iter().all(|&s| s < blocks.len()));
    }

    let mut entry = vec![Fact::new(); blocks.len()];
    let mut exit = vec![Fact::new(); blocks.len()];
    entry[start] = boundary;

    let mut queued = vec![false; blocks.len()];
    let mut work = VecDeque::new();
    work.push_back(start);
    queued[start] = true;

    while let Some(block) = work.pop_front() {
        queued[block] = false;
        let new_exit = transfer(entry[block].clone(), &blocks[block]);
        if new_exit == exit[block] {
            continue;
        }
        exit[block] = new_exit;

        for &successor in &blocks[block].successors {
            let old_len = entry[successor].len();
            entry[successor].extend(exit[block].iter().copied());
            if entry[successor].len() != old_len && !queued[successor] {
                work.push_back(successor);
                queued[successor] = true;
            }
        }
    }

    Results { entry, exit }
}

#[cfg(test)]
mod tests {
    use super::*;

    fn set(xs: &[usize]) -> Fact {
        xs.iter().copied().collect()
    }

    #[test]
    fn joins_diamond_paths() {
        let cfg = vec![
            Block { effects: vec![], successors: vec![1, 2] },
            Block { effects: vec![Effect::Gen(1)], successors: vec![3] },
            Block { effects: vec![Effect::Gen(2)], successors: vec![3] },
            Block { effects: vec![], successors: vec![] },
        ];
        let result = analyze(&cfg, 0, set(&[0]));
        assert_eq!(result.entry[3], set(&[0, 1, 2]));
    }

    #[test]
    fn converges_around_loop_and_applies_kill() {
        let cfg = vec![
            Block { effects: vec![Effect::Gen(1)], successors: vec![1] },
            Block { effects: vec![Effect::Gen(2)], successors: vec![1, 2] },
            Block { effects: vec![Effect::Kill(1)], successors: vec![] },
        ];
        let result = analyze(&cfg, 0, Fact::new());
        assert_eq!(result.entry[1], set(&[1, 2]));
        assert_eq!(result.exit[2], set(&[2]));
    }

    #[test]
    fn unreachable_blocks_stay_empty() {
        let cfg = vec![
            Block { effects: vec![Effect::Gen(0)], successors: vec![] },
            Block { effects: vec![Effect::Gen(9)], successors: vec![] },
        ];
        let result = analyze(&cfg, 0, Fact::new());
        assert!(result.entry[1].is_empty());
        assert!(result.exit[1].is_empty());
    }
}

Each set can grow at most once per local per block entry. Consequently, propagation performs finitely many fact insertions. Kills affect block exits but do not make the union join shrink an entry. This establishes convergence for a finite CFG and finite local universe.

The equality shortcut requires care at the start block. If a transfer of an empty entry produces empty exit, successors need no new facts. That is correct for this may-analysis but may not be correct for a framework tracking reachability separately. Production frameworks distinguish unreachable from reachable-with-empty-facts.

To compute definite initialization, initialize reachable non-start entries appropriately and intersect predecessor outputs. You must separately handle reachability; intersecting zero predecessors is otherwise ambiguous. rustc's framework supplies richer direction, cursor, visitor, and lattice machinery. This teaching engine is not a replacement for compiler infrastructure.

21. Dataflow precision and edge effects#

Block entry and exit facts are sometimes too coarse. Diagnostics need facts immediately before or after one statement. An analysis can replay transfer functions through a block from its fixpoint entry state. rustc dataflow cursors support location-oriented inspection in current APIs.

Terminators can have successor-specific effects. A successful call initializes its destination on the normal-return edge, not on an unwind edge. A branch can establish facts unique to one target. Applying such effects indiscriminately before the terminator is unsound.

before call: destination uninitialized
normal edge: destination initialized
unwind edge: destination still uninitialized; cleanup runs

Analysis direction also matters. Initialization naturally flows forward. Liveness often flows backward: a local is live before a statement if it is used there or live afterward without being overwritten. The same CFG supports both, with reversed propagation and appropriately defined transfer.

Precision costs memory and time. One bit per move path is cheap and compositional. Tracking relationships between arbitrary places can be much more expensive. Compiler analyses choose abstractions adequate to prove required safety without modeling every runtime detail.

A conservative safety analysis may reject a safe program if the language allows that conservatism. It must never accept an execution violating the rule it enforces. Optimization analyses have a dual obligation: insufficient precision loses speed, while unsound precision miscompiles.

Bug clue: cleanup-only use errors suggest normal and unwind edge effects were merged. Bug clue: a diagnostic one statement late suggests before/after-location confusion.

22. MIR validation, invariants, and Steal#

MIR validation checks structural and phase-specific invariants. Examples include valid block targets, well-typed assignments, legal projection chains, cleanup-edge rules, and allowed statement variants. Validation is especially valuable after transformations. It turns silent corruption into an internal compiler error near the responsible pass.

A pass should document its preconditions and postconditions. If it removes analysis-only statements, later validators can reject their presence. If it assumes critical edges split or drops elaborated, scheduling must guarantee that state. Validation does not prove full semantic equivalence, but it catches many local violations.

Steal<T> is an internal ownership concept used in rustc query products. It permits a value to be borrowed until one owner takes, or “steals,” it for mutation or transformation. After stealing, attempts to access the old product should fail rather than observe stale state. This helps express phase ownership without eagerly cloning large MIR bodies.

Do not recommend adding new Steal uses casually. Its query interactions, diagnostics, and ownership discipline are subtle and version-sensitive. Prefer established query patterns and consult compiler-team reviewers when changing phase ownership. The lesson is broader: large mutable IR needs one clear owner at each transformation stage.

Alternative: clone a body for every query consumer. That simplifies ownership but costs memory and risks consumers accidentally comparing unrelated copies. Alternative: globally mutable shared MIR. That undermines incremental queries and makes ordering bugs pervasive.

Bug clue: a “stolen” access indicates a query requested an earlier MIR product after ownership moved. Fix dependency ordering or request the proper later product; do not bypass the assertion.

23. Unsafety checking versus borrow checking#

Unsafety checking asks whether operations requiring an unsafe context occur where permitted. Examples include dereferencing raw pointers, accessing union fields, calling unsafe functions, and using mutable statics under applicable rules. THIR's typed explicit operations make these sites easier to identify than source sugar alone.

Borrow checking asks different questions. It enforces ownership, initialization, aliasing permissions, and reference lifetime constraints. Safe code can fail borrow checking. Code inside an unsafe block still undergoes borrow checking. unsafe does not disable the ownership system.

unsafe fn read(p: *const i32) -> i32 {
    unsafe { *p }
}

fn aliasing() {
    let mut x = 0;
    let a = &mut x;
    // let b = &mut x; // borrow-check error despite no unsafe operation
    *a += 1;
}

An unsafe block is a programmer assertion that additional safety obligations are upheld. The compiler verifies that syntactically unsafe operations are enclosed as required. It does not prove every raw-pointer precondition or diagnose all undefined behavior.

Unsafety diagnostics also care about source scopes and editions or lints such as unsafe operations inside unsafe functions. That makes source-oriented typed structure useful. Borrow checking benefits more directly from MIR's places and CFG.

Bug triage should separate the systems. A missing “unsafe required” diagnostic points toward unsafety checking or lowering. An aliasing rejection points toward borrow checking. A raw-pointer misuse accepted by both may still be undefined behavior at runtime; acceptance is not proof of soundness.

24. Drop elaboration#

Before drop elaboration, MIR can express “drop this place if it is initialized” at a relatively high level. Control flow may initialize only parts of a value. Drop elaboration turns that conditional obligation into explicit CFG decisions and drop flags.

A drop flag is conceptual state recording whether a value or tracked part currently requires destruction. Initialization sets relevant flags. Moves and completed drops clear them. At a scope exit, branches test flags before invoking destructor glue. The actual representation and optimizations around flags are version-sensitive.

if condition:
    x = make_string()
    flag_x = true
...
scope exit:
    if flag_x -> drop x -> continue
    else -> continue

For aggregates without a custom destructor, fields may be dropped separately after partial moves. For a type implementing Drop, moving fields out is generally prohibited in safe code. Destructor glue recursively drops fields after invoking custom behavior as specified.

Unwind complicates elaboration. If dropping one local unwinds, cleanup must continue or abort according to the active rules. The pass must avoid dropping already moved or already dropped values. It must also preserve source-specified destruction order.

Optimization can remove provably constant drop flags and unreachable drop paths. Correctness comes first: deleting a destructor call can change observable behavior. Even an apparently empty destructor can affect borrowing or future compilation assumptions.

Bug clue: behavior differs only for partial moves and panics, so inspect elaborated cleanup MIR and flag transitions.

25. Generators, coroutines, and async state machines#

An async fn returns a future whose polling resumes suspended computation. Conceptually, the compiler transforms the body into a state machine. Values live across an .await must be stored in the future object rather than ordinary stack temporaries that disappear between polls.

async fn fetch_twice() -> u32 {
    let first = fetch().await;
    let second = fetch().await;
    first + second
}

Conceptual states:

Start: create/poll first future
WaitingFirst: retain first future until ready
WaitingSecond: retain first result and second future
Done: retain completion state; polling rules apply

Coroutine lowering identifies locals live across suspension points. It lays them out as fields of a generated state-machine type. The resume discriminant selects code for the current state. Suspension stores state and yields pending or a yielded value. Completion stores a terminal state and returns ready.

Self-references and pinning make this transformation delicate. Once polling begins, moving the future may invalidate references into its own fields. The Pin API expresses the caller's obligation. Compiler lowering and borrow checking cooperate to enforce language rules; neither makes arbitrary self-references safe automatically.

Drop glue for a suspended coroutine must destroy exactly fields initialized in its current state. This resembles drop elaboration over a state-dependent aggregate. Panic/unwind behavior remains target- and strategy-sensitive.

Internal terminology has shifted from generators toward coroutines in APIs. Variant names, passes, and ordering in rustc 1.97.1 should be checked directly. The stable conceptual model is resumable CFG plus captured state.

Bug clue: async-only miscompilations often depend on a local being live across one suspension boundary.

26. MIR optimization passes#

MIR optimization improves code before backend optimization and can expose Rust-specific facts. Pass scheduling depends on optimization level, body kind, and compiler version. No single list should be treated as permanent.

Simplify-CFG transformations remove unreachable blocks, merge compatible blocks, and redirect trivial gotos. They must preserve cleanup classifications and source behavior. CFG simplification often enables every subsequent analysis by reducing graph noise.

Constant propagation evaluates operations whose inputs are known constants. It can replace a conditional branch with one target and fold arithmetic under the correct overflow semantics. It must not evaluate forbidden effects or assume a panic cannot occur when it can.

Dead-store elimination removes writes whose values are overwritten or never observed. “Observed” includes drops, volatile operations, inline assembly, aliasing accesses, and debug requirements as applicable. A store through a pointer cannot be discarded merely because no local read is obvious.

Inlining replaces a call with a transformed copy of the callee body. It can expose optimization opportunities and remove call overhead. It increases code size, compile time, and debug stepping complexity. Recursive and cross-crate decisions need cost models and availability rules.

Copy propagation replaces uses of a copied temporary with its source when legal. Moves, aliasing, mutation, and source-lifetime distinctions limit it. After propagation, dead locals and assignments can disappear.

Correctness means preserving all observable behavior allowed by Rust's abstract machine. Undefined behavior permits some assumptions, but compiler analyses must establish their preconditions soundly. Debug information creates a quality tradeoff: optimized code may not map variables one-to-one to source.

27. CTFE as MIR interpretation#

CTFE means compile-time function evaluation. rustc evaluates constants by interpreting suitable MIR rather than generating host machine code and running it blindly. The interpreter executes statements and terminators while maintaining call frames, locals, memory, and control flow.

const fn triangular(n: u32) -> u32 {
    let mut i = 0;
    let mut total = 0;
    while i <= n {
        total += i;
        i += 1;
    }
    total
}

const TEN: u32 = triangular(4);

The evaluator executes a deterministic compile-time world. It cannot simply read the compiler process's clock, random state, or arbitrary files. Operations allowed in const contexts are restricted by language rules and implementation support. Heap-like allocations may exist in interpreter memory without corresponding to ordinary runtime allocation calls.

Interpretation supports precise diagnostics for many invalid constant executions. Examples include out-of-bounds access, division by zero, invalid scalar values, and certain provenance violations. It does not imply that all undefined behavior in all Rust programs can be diagnosed. Runtime paths not evaluated at compile time remain outside CTFE. Some validity and aliasing properties are undecidable or intentionally not completely checked.

Older compilers enforced a const-evaluation step limit. Current compilers instead warn about long-running constant evaluation and continue, so a nonterminating constant can hang compilation. Recursion-depth and other resource errors still apply, and these controls remain version-sensitive. No resource limit can make a semantically forbidden operation legal.

CTFE and optimization constant propagation can share interpreter machinery or concepts but have different contracts. An optimization may decline to fold an expression that CTFE could evaluate. A required constant must either evaluate successfully or produce an error.

28. Interpreter memory, provenance, and machines#

The MIR interpreter models allocations as abstract memory objects. An allocation has bytes, alignment, mutability, initialization state, and provenance-related metadata. Pointers are not adequately modeled as host integer addresses. They identify abstract allocations and offsets under rules that evolve with Rust's memory model.

Provenance records information about where a pointer came from and what memory it may designate. This helps reject operations that fabricate invalid references or access outside an allocation. Pointer-to-integer and integer-to-pointer behavior in constants is deliberately constrained and version-sensitive. Never infer language guarantees from one debug print of an interpreter pointer.

Uninitialized bytes are not automatically zero. Reading them as a value can be invalid. Padding bytes require care because copying storage and interpreting a typed value are different operations. Alignment and bounds are checked according to the interpreted access.

The interpreter is parameterized by a “machine” abstraction. The machine supplies policy for operations that differ between CTFE, validation, or other interpreter clients. This separates core execution mechanics from client-specific permissions and diagnostics. Internal machine traits and hooks are unstable.

Miri uses rustc interpreter infrastructure to dynamically check many behaviors while running Rust programs. Miri is not simply “rustc CTFE,” and the two are not identical products. They use different machine policies and execution purposes. Miri can execute runtime-oriented programs with extra checking; CTFE enforces compile-time restrictions and produces constants. Neither claim implies detection of all undefined behavior.

Bug clue: host-dependent constant results violate the deterministic model and should be minimized urgently. Bug clue: failures only after pointer casts often concern provenance or const permission, not arithmetic folding.

29. Forbidden operations, errors, and deterministic limits#

Compile-time evaluation admits only operations compatible with a deterministic, isolated result. Arbitrary system calls, networking, environment mutation, and thread scheduling are not ordinary const operations. Some language operations are forbidden because their semantics or cleanup cannot be represented safely as a constant. The permitted set expands through language design; always consult the targeted compiler and Reference.

A constant error may be unconditional or encountered only after monomorphization. Generic code can contain an unevaluated constant whose validity depends on substitutions. Diagnostics must retain spans and a stack trace-like chain of const calls where useful.

Arithmetic uses Rust's typed semantics, not the compiler host's accidental behavior. Overflow in a required constant is diagnosed unless an explicitly wrapping operation is used. Shifts, division, casts, and layout calculations require target-specific widths. Cross-compilation evaluates for the target model, not the host CPU's layout.

Resource limits serve compiler availability. A terminating mathematical function can still exceed configured evaluation effort. That is not proof the function diverges. Conversely, allowing unbounded CTFE would let a crate hang compilation.

Caching evaluated constants can improve compilation, but cache keys must include relevant substitutions and target/compiler context. Results containing allocations need canonicalization and stable handling for incremental compilation. Implementation details are especially version-sensitive.

Bug triage separates categories:

wrong accepted value     interpreter semantics or validation
wrong rejection          const qualification or machine permission
hang/resource blowup     loop detection, limits, caching, or query cycle
host/target difference   layout, endianness, pointer width, or nondeterminism
poor diagnostic          span/frame propagation rather than evaluator result

30. Promotion, qualification, and const generics#

Promotion extracts certain runtime-looking expressions into compiler-generated constants. A familiar effect is allowing a reference to a promoted literal or aggregate to have a longer lifetime. Promotion is deliberately restricted so extraction does not change observable side effects or destruction behavior.

fn promoted() -> &'static [i32; 3] {
    &[1, 2, 3]
}

Conceptually, the array expression can become a promoted body evaluated as immutable constant data. Not every temporary is promotable. Interior mutability, destructor behavior, calls, and evaluation effects matter under current rules. Promotion rules and implementation analyses are version-sensitive.

Const qualification determines which operations a const context or promotable expression may perform. It is not identical to successfully interpreting one particular execution path. Qualification may reject constructs structurally to uphold language guarantees. Later CTFE executes accepted MIR and can still encounter errors such as division by zero.

Const generics put constants into types, such as [T; N]. Type-system constants may remain symbolic while a generic item is checked. MIR constants used as operands and type-system constants used in types have related but distinct representations. After substitution, normalization and evaluation may produce concrete values.

fn head<const N: usize>(xs: [u8; N]) -> Option<u8> {
    if N == 0 { None } else { Some(xs[0]) }
}

The boundary is subtle: a branch can protect a runtime index, while type-level reasoning about N follows const-generic normalization rules. Do not assume CTFE proves arbitrary equations between symbolic const expressions. Generic const expression support is feature- and version-sensitive.

Bug clue: only generic uses failing suggests substitution, normalization, or abstract-const handling rather than basic MIR interpretation.

31. A tiny MIR and interpreter in stable Rust#

This teaching IR has integer locals, straight-line assignments, conditional jumps, and returns. All arithmetic uses checked signed 64-bit operations. Overflow is an explicit Error::Overflow; reading an unset local is Error::Uninitialized. Step exhaustion is Error::StepLimit, making nontermination policy visible.

#[derive(Clone, Debug)]
enum Operand {
    Local(usize),
    Const(i64),
}

#[derive(Clone, Copy, Debug)]
enum BinOp {
    Add,
    Sub,
    Mul,
    Eq,
}

#[derive(Clone, Debug)]
enum Rvalue {
    Use(Operand),
    Binary(BinOp, Operand, Operand),
}

#[derive(Clone, Debug)]
enum Statement {
    Assign(usize, Rvalue),
}

#[derive(Clone, Debug)]
enum Terminator {
    Goto(usize),
    If { condition: Operand, yes: usize, no: usize },
    Return(Operand),
}

#[derive(Clone, Debug)]
struct Block {
    statements: Vec<Statement>,
    terminator: Terminator,
}

#[derive(Clone, Debug)]
struct Body {
    locals: usize,
    blocks: Vec<Block>,
}

#[derive(Debug, PartialEq, Eq)]
enum Error {
    BadBlock(usize),
    BadLocal(usize),
    Uninitialized(usize),
    Overflow,
    StepLimit,
}

fn operand(op: &Operand, locals: &[Option<i64>]) -> Result<i64, Error> {
    match *op {
        Operand::Const(value) => Ok(value),
        Operand::Local(index) => locals
            .get(index)
            .ok_or(Error::BadLocal(index))?
            .ok_or(Error::Uninitialized(index)),
    }
}

fn rvalue(rv: &Rvalue, locals: &[Option<i64>]) -> Result<i64, Error> {
    match rv {
        Rvalue::Use(op) => operand(op, locals),
        Rvalue::Binary(kind, left, right) => {
            let left = operand(left, locals)?;
            let right = operand(right, locals)?;
            match kind {
                BinOp::Add => left.checked_add(right).ok_or(Error::Overflow),
                BinOp::Sub => left.checked_sub(right).ok_or(Error::Overflow),
                BinOp::Mul => left.checked_mul(right).ok_or(Error::Overflow),
                BinOp::Eq => Ok(i64::from(left == right)),
            }
        }
    }
}

fn interpret(body: &Body, inputs: &[(usize, i64)], limit: usize) -> Result<i64, Error> {
    let mut locals = vec![None; body.locals];
    for &(index, value) in inputs {
        let slot = locals.get_mut(index).ok_or(Error::BadLocal(index))?;
        *slot = Some(value);
    }

    let mut block = 0;
    let mut steps = 0;
    loop {
        if steps >= limit {
            return Err(Error::StepLimit);
        }
        steps += 1;

        let data = body.blocks.get(block).ok_or(Error::BadBlock(block))?;
        for statement in &data.statements {
            match statement {
                Statement::Assign(destination, value) => {
                    let value = rvalue(value, &locals)?;
                    let slot = locals
                        .get_mut(*destination)
                        .ok_or(Error::BadLocal(*destination))?;
                    *slot = Some(value);
                }
            }
        }

        match &data.terminator {
            Terminator::Goto(target) => block = *target,
            Terminator::If { condition, yes, no } => {
                block = if operand(condition, &locals)? != 0 { *yes } else { *no };
            }
            Terminator::Return(value) => return operand(value, &locals),
        }
    }
}

#[cfg(test)]
mod tests {
    use super::*;

    #[test]
    fn executes_branch_and_arithmetic() {
        let body = Body {
            locals: 2,
            blocks: vec![
                Block {
                    statements: vec![],
                    terminator: Terminator::If {
                        condition: Operand::Local(0),
                        yes: 1,
                        no: 2,
                    },
                },
                Block {
                    statements: vec![Statement::Assign(
                        1,
                        Rvalue::Binary(BinOp::Mul, Operand::Const(6), Operand::Const(7)),
                    )],
                    terminator: Terminator::Return(Operand::Local(1)),
                },
                Block {
                    statements: vec![],
                    terminator: Terminator::Return(Operand::Const(0)),
                },
            ],
        };
        assert_eq!(interpret(&body, &[(0, 1)], 10), Ok(42));
        assert_eq!(interpret(&body, &[(0, 0)], 10), Ok(0));
    }

    #[test]
    fn reports_overflow() {
        let body = Body {
            locals: 1,
            blocks: vec![Block {
                statements: vec![Statement::Assign(
                    0,
                    Rvalue::Binary(BinOp::Add, Operand::Const(i64::MAX), Operand::Const(1)),
                )],
                terminator: Terminator::Return(Operand::Local(0)),
            }],
        };
        assert_eq!(interpret(&body, &[], 2), Err(Error::Overflow));
    }

    #[test]
    fn reports_uninitialized_read() {
        let body = Body {
            locals: 1,
            blocks: vec![Block {
                statements: vec![],
                terminator: Terminator::Return(Operand::Local(0)),
            }],
        };
        assert_eq!(interpret(&body, &[], 2), Err(Error::Uninitialized(0)));
    }

    #[test]
    fn limits_infinite_loop() {
        let body = Body {
            locals: 0,
            blocks: vec![Block {
                statements: vec![],
                terminator: Terminator::Goto(0),
            }],
        };
        assert_eq!(interpret(&body, &[], 5), Err(Error::StepLimit));
    }
}

Unlike rustc, this interpreter has no types beyond i64, no memory, pointers, calls, unwinding, or source spans. Its condition treats zero as false and every other value as true; that policy is explicit rather than inherited from Rust bool. The step count charges once per entered block, not per statement. A production limit needs a more carefully specified cost model.

Validation could reject bad targets and locals before execution. This interpreter instead reports them dynamically. That tradeoff keeps the example small but repeats checks on every execution. rustc combines up-front MIR validation with interpreter checks for runtime-dependent conditions.

32. End-to-end source, THIR, MIR, and optimization trace#

Trace this function:

fn answer(flag: bool) -> i32 {
    let base = if flag { 40 } else { 10 };
    base + 2
}

The HIR remains source-shaped: a function item, a block, a let, an if, and an addition. Type checking determines that literals default to i32, flag is bool, both if arms agree, and addition is built-in integer addition.

A conceptual THIR view is typed and explicit:

Block : i32
  Let pattern base : i32
    initializer If : i32
      condition Var(flag) : bool
      then Literal(40) : i32
      else Literal(10) : i32
  tail Binary(Add, Var(base), Literal(2)) : i32
  destruction and lexical scopes attached

This is not actual dump syntax. It demonstrates that all expression types and the selected built-in operation are known.

Conceptual built MIR with overflow checking:

bb0:
    StorageLive(_2)                 // base
    SwitchInt(copy _1) -> false: bb2, otherwise: bb1
bb1:
    _2 = const 40_i32
    Goto bb3
bb2:
    _2 = const 10_i32
    Goto bb3
bb3:
    _3 = CheckedAdd(copy _2, const 2_i32)
    Assert(no_overflow(_3)) -> bb4
bb4:
    _0 = value(_3)
    StorageDead(_2)
    Return

An optimized form may simplify temporaries and combine return assignments:

bb0: SwitchInt(copy _1) -> false: bb2, otherwise: bb1
bb1: _0 = const 42_i32; Return
bb2: _0 = const 12_i32; Return

Constant propagation proves each arm's addition and absence of overflow. CFG simplification removes the join if duplicating a tiny return is profitable. Storage markers and dead temporaries disappear. Another compiler revision may choose a different but equivalent optimized graph.

The trace suggests a triage method. Wrong type in THIR points before MIR construction. Correct THIR but wrong built MIR points to construction. Correct built MIR but wrong optimized MIR identifies a transform. Correct optimized MIR but wrong machine behavior points toward codegen or runtime assumptions.

33. Inspecting MIR and THIR safely#

Nightly rustc offers unstable -Z debugging flags. Their names, accepted values, and output formats can change. Use a pinned nightly close to rustc 1.97.1 and begin with rustc -Z help. Stable rustc rejects -Z flags by design.

Common version-sensitive commands include:

rustup run nightly rustc -Zunpretty=thir-flat example.rs
rustup run nightly rustc --emit=mir example.rs
rustup run nightly rustc -Zmir-opt-level=0 --emit=mir example.rs
rustup run nightly rustc -Zdump-mir=all example.rs
rustup run nightly rustc -Zdump-mir=SomePass example.rs

-Zunpretty=thir-flat is documented by the rustc-dev-guide at the time of writing. --emit=mir produces human-oriented MIR. -Zmir-opt-level=0 helps expose less-optimized structure, but does not promise pristine built MIR. -Zdump-mir can create per-pass files, commonly under a dump directory configurable by another unstable option. Check -Z help for exact spelling and defaults.

Compile with the same edition, target, panic strategy, optimization, and features as the bug. Otherwise the graph can differ for legitimate reasons. Minimize source while retaining the first divergent dump. Compare pass-before and pass-after files rather than only final output.

Never write a long-lived parser for pretty MIR without pinning and tests. The output explicitly serves humans and changes without compatibility guarantees. For compiler tests, use repository-supported normalization and expected-output mechanisms.

Useful official API entry points include rustc_middle::mir, rustc_middle::thir, rustc_mir_build, rustc_mir_transform, rustc_mir_dataflow, and rustc_const_eval. Nightly rustdoc reflects tip, not necessarily rustc 1.97.1. Select the matching toolchain source when exact fields matter.

34. MIR tests and a bug-triage map#

rustc has multiple relevant test styles. UI tests assert diagnostics and compile outcomes. MIR-opt tests compare dumps around selected transformations. Codegen tests inspect backend patterns when MIR alone cannot capture the regression. Run-pass tests execute behavior. Miri tests exercise interpreter checking under Miri's machine policy. Const-eval tests cover accepted and rejected compile-time execution.

Choose the narrowest test that captures the contract. A CFG simplification regression belongs in a MIR-opt test plus execution coverage if semantics changed. A bad borrow diagnostic belongs in UI tests. A CTFE-only provenance rejection belongs with const-eval tests, while a Miri runtime case may need a distinct test. Do not assume passing Miri proves CTFE behavior or vice versa.

Triage map:

wrong constructor coverage        usefulness/pattern analysis
wrong binding or test order       match lowering / MIR construction
wrong implicit call or coercion   type-check adjustments / THIR lowering
use after move accepted/rejected  move paths / borrowck dataflow
double or missing destructor      drop scopes / drop elaboration / cleanup
async state corruption            coroutine transform / liveness / layout
wrong folded branch               const propagation / interpreter semantics
const-only failure                qualification / CTFE machine / validation
optimized-only miscompile         first differing MIR pass, then codegen
panic-strategy-only failure       unwind actions / cleanup / target policy
bad source location               SourceInfo / scope propagation
ICE about illegal MIR variant     phase scheduling / validator invariant

Always record the exact rustc commit or verbose version. “Nightly” alone is not reproducible. Bisect when possible, and attach compact source plus relevant before/after dumps. Remove giant logs that do not narrow the first divergence.

An internal compiler error is a compiler bug even if the input also has user errors. Unsound acceptance and miscompilation deserve especially careful reduction and responsible reporting.

35. Exercises and contribution map#

1. Draw conceptual MIR for short-circuit a() && b().

Mark exactly where b is skipped and where results join.

1. Extend the dataflow engine with explicit reachability.

Create a test distinguishing unreachable from reachable with an empty fact set.

1. Implement definite initialization using intersection.

Explain the boundary state and why zero-predecessor blocks need special treatment.

1. Add predecessor-specific edge effects to the tiny engine.

Model a call destination initialized only on normal return.

1. Construct a usefulness matrix for (Option<bool>, bool).

Derive one missing witness by constructor specialization.

1. Explain why guarded wildcard arms do not generally make following arms unreachable.

Give a guard with observable side effects.

1. Partition the u8 domain for ranges 0..=9, 5..=20, and 200..=255.

Identify covered and uncovered symbolic intervals without enumerating 256 values.

1. Extend the tiny interpreter with checked division.

Use separate errors for division by zero and signed overflow at MIN / -1.

1. Add a validation pass to the tiny interpreter.

Reject bad block targets, bad locals, and a body with no blocks before execution.

1. Add function calls and a bounded call stack.

Define whether the step limit charges calls, blocks, or instructions.

1. Compare nightly dumps at MIR optimization levels zero and three.

List only semantic-preserving differences you can justify.

1. Write examples where a field is moved and then reinitialized.

Predict move-path states before checking compiler diagnostics.

1. Trace destruction on normal return and panic for three String locals.

Repeat under abort and unwind panic strategies, noting target support.

1. Find an async function with one value live across the first await but not the second.

Sketch minimum conceptual state-machine fields per state.

1. Explain why StorageDead is not equivalent to Drop.

Provide a Copy local and a destructor-bearing local as contrasting cases.

For a first rustc contribution, documentation and diagnostic tests are approachable. Pattern work lives around usefulness and match checking. MIR construction work lives in rustc_mir_build and should include lowering tests. Analysis and optimization work spans rustc_mir_dataflow and rustc_mir_transform. CTFE and interpreter work centers on rustc_const_eval, with coordination from the const-eval and Miri communities.

Before editing, build the compiler revision, read directory guidance, and locate the owning team. Add a regression test first when practical. Use MIR validation and focused dumps to identify the earliest broken invariant. Avoid broad refactors in the same patch as a semantic fix.

36. Sources and durable conclusions#

The following sources were checked for this chapter. They describe current internals, so pin revisions when exact API behavior matters.

• The THIR, rustc-dev-guide
• The MIR, rustc-dev-guide
• MIR construction, rustc-dev-guide
• MIR optimizations, rustc-dev-guide
• MIR dataflow, rustc-dev-guide
• Pattern and exhaustiveness checking, rustc-dev-guide
• Constant evaluation, rustc-dev-guide
• MIR interpretation, rustc-dev-guide
• Nightly rustc_middle::thir API
• Nightly rustc_middle::mir API
• Nightly rustc_mir_build API
• Nightly rustc_mir_transform API
• Nightly rustc_mir_dataflow framework API
• Nightly rustc_const_eval API
• The Rust Reference: patterns
• The Rust Reference: constant evaluation
• The Miri project

Three conclusions should survive internal renaming. First, typed structured lowering centralizes decisions made by type checking before graph construction. Second, MIR's explicit places, effects, and edges enable borrow checking, dataflow, CTFE, optimization, and code generation to share semantics. Third, every transformation is a proof obligation: preserve language behavior while establishing stronger invariants for a narrower set of consumers.

When debugging, move from source toward later forms and stop at the first wrong representation. When designing an analysis, derive the representation from the facts and joins it needs. When changing a pass, state its accepted dialect, preserve source and unwind semantics, validate its output, and test the smallest contract.

Part V-B: MIR and Constant-Evaluation Internals#

This continuation assumes basic Rust and the representation pipeline developed in Part V. It targets the rustc 1.97.1 development line. Internal names, query boundaries, pass order, and dump syntax are observations about that revision, not language guarantees. The Rust Reference is normative where it specifies behavior; compiler source describes one implementation.

37. THIR ownership: who may keep what#

THIR is a typed, body-local bridge between type checking and consumers that still benefit from source-shaped structure. The first ownership lesson is negative: a THIR expression ID is not a durable identity for tooling. It indexes an arena associated with one constructed body. Keeping it after that body's storage is released would be like retaining an index into a destroyed vector.

Conceptually, construction has four owners:

HIR body owner
  | asks type checking for decisions
  v
typeck results -- adjustments, resolutions, inferred types --> THIR builder
                                                          |
                                                          | owns temporary arenas
                                                          v
                                  pattern checks / unsafety / MIR builder
                                                          |
                                                          v
                                                  arenas may be released

The arrows carry semantic decisions, not references with a stable public lifetime. Query caching and arena placement are implementation details. The durable invariant is that every consumer sees one coherent set of type-checking results for the body.

An adjustment is an implicit operation selected during type checking. Examples include dereferencing, borrowing, pointer conversion, and unsizing. The order is semantic. For a receiver, autoderef can discover a method and autoref can then create the receiver expected by that method. Reversing those steps could select a different implementation or produce a different place.

fn need_slice(_: &[u8]) {}

fn adjusted(a: &[u8; 4]) {
	need_slice(a);
}

The source argument is one expression. Its checked meaning includes the array-to-slice unsizing coercion. THIR lowering must consume the recorded decision rather than rerun inference.

Separate four concerns when investigating this boundary:

Moving adjustment application later would make every THIR consumer understand implicit coercions. Moving it earlier into HIR would force an inference-dependent meaning into a representation also used while inference is occurring. The intermediate boundary pays one lowering cost to avoid both forms of coupling.

Ownership experiment#

Choose an expression with autoref, unsizing, and overloaded indexing. Dump the typed form and built MIR on the pinned compiler. Then simplify one feature at a time. If removing unsizing repairs the MIR, inspect the adjustment chain before optimization. If the typed operation is already wrong, the MIR builder is downstream of the first broken invariant.

Contributor rule: never use debug indices as cross-body keys. Use stable source identities only where rustc explicitly supplies them, and keep body-local IDs body-local. Do not make THIR permanent merely to simplify one consumer; measure peak memory and query recomputation first.

38. Usefulness, exhaustiveness, and useful witnesses#

Pattern checking asks a set question before MIR construction asks an execution question. A pattern denotes a set of values. A row is useful when it contains at least one value not denoted by earlier unguarded rows. A match is exhaustive when a wildcard row is not useful after all covering rows.

The constructor abstraction avoids enumerating values. Enum variants, tuple shapes, references, scalar intervals, and slice-length classes partition the relevant space. Specialization selects a constructor and replaces its column by fields of that constructor. The recursion ends at a constructor with no fields or a matrix with no columns.

Trace Option<bool> with rows None and Some(true):

candidate _
  | specialize top constructor None
  |   matrix contains None -> covered
  |
  ` specialize top constructor Some(field)
      matrix field is true
      | specialize true  -> covered
      ` specialize false -> uncovered
                              |
                              ` rebuild Some(false)

Witness construction runs the proof backward. An uncovered leaf contributes a field witness. Each returning recursion wraps fields in the constructor that led to that branch. Formatting may later merge or abbreviate witnesses, but it must not invent coverage.

Important invariants are:

1. specialization preserves exactly the values represented by the chosen constructor;
2. constructor arity equals the number of introduced columns;
3. wildcard handling accounts for every constructor relevant to the type;
4. witness reconstruction is the inverse shape operation of specialization;
5. guards do not normally contribute unconditional coverage;
6. inhabitedness assumptions agree with the type environment and feature rules.

Ranges require endpoint partitioning. Given 0..=4 and 3..=8, useful scalar classes include 0..=2, 3..=4, 5..=8, and the outside regions. Each class has uniform membership in the source ranges. This is cheaper than enumerating u128, while preserving the answer.

Slices need length constructors. [a, b] covers length two. [head, middle @ .., tail] covers lengths at least two. The fields exposed during specialization depend on fixed prefix, fixed suffix, and a variable middle.

Or-patterns denote union. Eagerly expanding every nested alternative is simple but may grow exponentially. A production checker delays or shares expansion and imposes complexity controls. That optimization must preserve usefulness of each alternative, including diagnostics for redundant alternatives.

Witness debugging workshop#

Use a tiny enum with two nested booleans. First verify the yes/no exhaustiveness answer. Then verify witness shape independently. A correct rejection with an impossible printed witness isolates reconstruction or formatting. A wrong answer only with nested tuples suggests column/arity bookkeeping. A timeout only after adding nested or-patterns suggests expansion rather than constructor semantics.

Alternative: compile patterns directly into a decision tree and infer coverage from that tree. This unifies execution and coverage but makes diagnostic witnesses and guarded-arm reasoning harder. rustc pays for a separate symbolic analysis because diagnostic quality and language checking need information execution lowering may discard.

Security concern: patterns are adversarial compiler input. Bound recursion, avoid repeated matrix clones, test deeply nested alternatives, and ensure a complexity error does not become unsound acceptance.

39. MIR bodies, dialects, and phase contracts#

MIR is a typed control-flow graph. A body contains local declarations, basic blocks, source scopes, promoted references, and metadata. A basic block contains statements followed by exactly one terminator. Statements continue within a block; the terminator chooses successors or ends execution.

MIR is not one immutable language. Construction, analysis, runtime lowering, optimization, and code generation accept different subsets and invariants. The phase marker is a contract between passes. It says what may still occur and what has already been made explicit.

built MIR
  | type- and shape-valid, source lowering constructs may remain
  v
analysis MIR
  | borrow/move analyses observe required structure
  v
runtime-lowered MIR
  | analysis-only constructs removed; drops/coroutines elaborated as required
  v
optimized MIR
  | semantics preserved under runtime observation
  v
codegen

This diagram suppresses query forks and exact 1.97.1 pass names. Do not infer a stable public pipeline from it.

A validator should check at least:

• every block has a terminator;
• every target indexes an existing block;
• local and projection types compose;
• assignment destination and rvalue types agree;
• cleanup edges enter blocks valid for cleanup;
• constructs forbidden in the current phase are absent;
• unwind actions satisfy the current panic and phase policy;
• call destinations are initialized only on normal return;
• source scopes and promoted references point to valid entries.

Validation is a bug detector, not a semantic proof. It can prove that an index is in range but not that replacing signed division with multiplication preserved behavior. Run it after the pass that could first violate a structural contract.

The dialect model moves complexity. One universal MIR would simplify scheduling but force every consumer to handle every historical form. Many unrelated IRs would simplify each pass but multiply conversion and provenance loss. Phased MIR shares structure while narrowing legal forms over time.

Version-sensitive investigation should record rustc -vV, the exact dump flags, and the queried MIR product. Comparing “MIR” without identifying phase can produce a false bug report.

40. Precise lowering and evaluation order#

MIR construction linearizes nested expressions without changing Rust evaluation order. The safe technique is to assign each observable subexpression to a temporary at the point where source semantics evaluate it. Later operations consume those temporaries.

For left() + right(), a conceptual lowering is:

StorageLive(_l)
_l = Call left() -> bb_left_done, unwind bb_cleanup
bb_left_done:
StorageLive(_r)
_r = Call right() -> bb_right_done, unwind bb_cleanup
bb_right_done:
_out = Add(move _l, move _r)

The exact built MIR can differ. The invariant is that left completes before right begins and each is evaluated once.

Compound assignment is especially revealing. For an indexed destination, rustc must evaluate the destination components and right-hand side according to language semantics, preserve a place for the write, and avoid duplicating overloaded operations. Lowering a[index()] += rhs() as a[index()] = a[index()] + rhs() is wrong because it can evaluate index() twice.

Short-circuit operators require branches:

evaluate a
  | false ----------------------> result = false
  | true
  v
evaluate b ---------------------> result = b
                                   |
                                   v
                                  join

The branch carries control, not a speculative value. Turning this into an eager bitwise operation would expose effects and panics from b.

Calls terminate blocks because they can return normally, unwind, diverge, or transfer through special ABI behavior. The destination becomes initialized only on the normal-return edge. An analysis that applies destination initialization before the call incorrectly exposes a value in cleanup code.

Match lowering separates tests from bindings and body entry. Tests may inspect discriminants, lengths, constants, and ranges. Bindings occur only along a successful path. A guard runs after provisional matching and must not leak partially initialized bindings when false.

Evaluation-order experiment#

Write functions that append labels to a Vec<&'static str> and optionally panic. Use them in call arguments, binary expressions, indexing, if, and match guards. Compare unoptimized and optimized binaries. If labels differ only after one MIR pass, dump immediately before and after that pass. The earliest reordered side effect is stronger evidence than final assembly.

Optimization may reorder only when the language observation model permits it. Potential panic, destructor timing, volatile access, aliasing, and unwinding all constrain movement.

41. Places, projections, operands, and rvalues#

A place identifies storage. It begins at a local or static-like base and follows projections. Common conceptual projections are dereference, field, index, constant index, subslice, and downcast.

_1                         local place
_1.0                      field projection
(*_2).1                   dereference then field
_3[_4]                    dynamic index
((_5 as Variant1).0)      downcast then field

A projection is typed incrementally. If _1: (u32, bool), _1.0: u32. If _2: &mut Pair, *_2: Pair before selecting its field. A downcast does not execute a runtime conversion; it selects the known variant view after control flow established the discriminant.

An operand supplies a value to an rvalue or terminator. It can copy from a place, move from a place, or embed a constant. Copy leaves the source initialized. Move can make the source move path uninitialized for later checking and drop elaboration.

An rvalue computes a value for assignment. Examples include using an operand, taking a reference, aggregate construction, arithmetic, casts, discriminant extraction, length, and pointer operations. Rvalues do not themselves choose CFG successors. Operations that may require explicit normal and unwind destinations are represented with terminators or lowered forms appropriate to the phase.

Statements describe effects that continue to the next statement. Assignments, storage markers, fake reads, deinitialization, retags, and other variants are phase-sensitive. Do not teach every nightly variant as permanent syntax.

Terminators include conceptual goto, switch, call, return, drop, assert, unwind continuation, unreachable, and coroutine-related control transfers. Their successor lists are part of dataflow semantics.

Representation tradeoff:

Earliest-invariant debugging matters. A malformed projection type is a builder or transform bug. A valid projection with incorrect alias assumptions is an analysis bug. A valid optimized place producing wrong machine addressing may be codegen.

42. Temporaries, scopes, drops, cleanup, and unwind#

Source expressions create values whose storage and destruction times need not match lexical braces exactly. THIR destruction scopes preserve the type checker's lifetime decisions. MIR construction turns those decisions into temporaries, storage markers, and drop control flow.

StorageLive and StorageDead describe a local's storage-liveness model. They are not destructor calls. A Copy integer can have storage markers without a destructor. A String requires a drop when initialized, even if its eventual storage marker is optimized away.

Three states must remain distinct:

1. storage exists;
2. the place is initialized;
3. an initialized value requires destruction.

Conflating them causes use-after-move, double drop, or skipped drop.

Consider:

fn scoped(flag: bool) {
	let outer = String::from("outer");
	if flag {
		let inner = String::from("inner");
		consume(inner);
	}
	drop(outer);
}

fn consume(_: String) {}

Moving inner means the scope exit must not drop its old value. The move path becomes uninitialized. Normal and unwind exits must each consult the correct initialization state after elaboration.

Cleanup blocks run destruction required while unwinding. They are not ordinary alternate success paths. An unwind edge from a call observes only effects completed before the call and effects explicitly guaranteed on that edge. The call destination does not exist there.

before call: x initialized, destination uninitialized
       |
       v
      call
     /    \
normal    unwind
  |          |
dest init    dest remains uninit
  |          |
continue     cleanup x

Panic strategy and target support can remove runtime unwinding, but transforms must consume the correct policy rather than silently reinterpret cleanup MIR. Abort and unwind builds are separate test dimensions.

Failure experiments:

• panic before a local initializes: its destructor must not run;
• panic after initialization: its destructor runs on supported unwind paths;
• move one field from a struct: only still-initialized fields are dropped;
• reinitialize that field: it becomes droppable again;
• panic inside a destructor: observe the platform/panic policy rather than assuming recovery.

Moving drop insertion into the parser would be impossible because types and moves are unknown. Moving all drop decisions to codegen would duplicate language analysis per backend. MIR's explicit control and initialization information is the useful middle boundary.

43. Dataflow equations, worklists, and edge effects#

A forward dataflow analysis associates an abstract fact with each program point. The fact is an approximation chosen for one question. The domain needs a finite-height ordering or another termination argument.

For block b, a conventional forward equation is:

IN[b]  = boundary(b) join JOIN { EDGE[p -> b](OUT[p]) | p is a predecessor }
OUT[b] = transfer_block(b, IN[b])

join may be union for “may” facts or intersection for “must” facts. That choice changes the question. Union initialization means “initialized on at least one path,” which cannot justify a read requiring initialization on every path.

A backward analysis reverses flow:

OUT[b] = boundary(b) join JOIN { EDGE[b -> s](IN[s]) | s is a successor }
IN[b]  = transfer_block_backward(b, OUT[b])

Liveness is a classic backward may analysis. Its transfer resembles (live_after - defs) union uses.

The worklist algorithm is simple:

1. initialize facts and enqueue blocks whose inputs can change;
2. pop a block;
3. compute transfer in statement order;
4. apply terminator and per-edge effects;
5. join into each successor;
6. enqueue a successor only when its input changes;
7. stop when no fact changes.

Reachability must not be encoded as an empty bitset. “No facts on a reachable path” differs from “no path reached this block.” Represent it separately or use a lattice with an explicit unreachable element.

Call edges demonstrate edge effects. Initialization of the destination occurs on the normal edge only. Switch branches may carry branch-refined facts. Coroutine yields and unwind edges also have semantics that a block-only transfer cannot express precisely.

A complete stable-Rust CFG and may-dataflow model#

This program models generated facts with bitsets, explicit reachability, and normal-edge generation. It intentionally omits kills, statement points, cleanup kinds, backward flow, and malformed-graph diagnostics beyond assertions.

use std::collections::VecDeque;

type Block = usize;
type Bits = u64;

#[derive(Clone, Debug)]
struct Edge {
	target: Block,
	generate: Bits,
}

#[derive(Clone, Debug)]
struct BasicBlock {
	generate: Bits,
	edges: Vec<Edge>,
}

#[derive(Clone, Copy, Debug, PartialEq, Eq)]
enum State {
	Unreachable,
	Reachable(Bits),
}

fn join(old: State, incoming: State) -> State {
	match (old, incoming) {
		(State::Unreachable, x) => x,
		(x, State::Unreachable) => x,
		(State::Reachable(a), State::Reachable(b)) => State::Reachable(a | b),
	}
}

fn analyze(cfg: &[BasicBlock], entry: Block, boundary: Bits) -> Vec<State> {
	assert!(entry < cfg.len());
	for block in cfg {
		for edge in &block.edges {
			assert!(edge.target < cfg.len());
		}
	}
	let mut inputs = vec![State::Unreachable; cfg.len()];
	inputs[entry] = State::Reachable(boundary);
	let mut work = VecDeque::from([entry]);
	let mut queued = vec![false; cfg.len()];
	queued[entry] = true;

	while let Some(block) = work.pop_front() {
		queued[block] = false;
		let State::Reachable(input) = inputs[block] else {
			continue;
		};
		let output = input | cfg[block].generate;
		for edge in &cfg[block].edges {
			let candidate = State::Reachable(output | edge.generate);
			let merged = join(inputs[edge.target], candidate);
			if merged != inputs[edge.target] {
				inputs[edge.target] = merged;
				if !queued[edge.target] {
					work.push_back(edge.target);
					queued[edge.target] = true;
				}
			}
		}
	}
	inputs
}

fn main() {
	// 0 branches. The normal edge generates fact 1; unwind does not.
	let cfg = vec![
		BasicBlock {
			generate: 0b001,
			edges: vec![
				Edge { target: 1, generate: 0b010 },
				Edge { target: 2, generate: 0 },
			],
		},
		BasicBlock { generate: 0b100, edges: vec![Edge { target: 3, generate: 0 }] },
		BasicBlock { generate: 0, edges: vec![Edge { target: 3, generate: 0 }] },
		BasicBlock { generate: 0, edges: vec![] },
		BasicBlock { generate: 0, edges: vec![] }, // unreachable, even though empty facts exist
	];
	let got = analyze(&cfg, 0, 0);
	assert_eq!(got[1], State::Reachable(0b011));
	assert_eq!(got[2], State::Reachable(0b001));
	assert_eq!(got[3], State::Reachable(0b111));
	assert_eq!(got[4], State::Unreachable);
	println!("{got:?}");
}

Expected output contains reachable bitsets for blocks zero through three and Unreachable for block four. Compile with stable Rust using rustc cfg.rs && ./cfg.

To implement definite initialization, use intersection and define the top/boundary states carefully. An all-zero initial fact is not a neutral element for intersection. Reachability prevents zero-predecessor blocks from contaminating the result.

Production hardening adds compact bitsets, statement-level transfer, direction abstraction, cached predecessor/successor traversal, loop-order heuristics, edge-specific callbacks, and cancellation checks. Benchmark dense and sparse facts separately. A faster worklist with the wrong edge semantics is not an optimization.

44. Move paths, initialization, and drop elaboration#

A local is too coarse for partial moves. Move analysis therefore tracks a tree-like family of move paths for places that can be reasoned about separately. A root might represent _x; descendants represent _x.0 and _x.1.

_x
 |-- _x.0
 |    `-- _x.0.field
 `-- _x.1

Moving _x.0 deinitializes that path and relevant descendants. It also prevents treating the complete parent _x as initialized. It need not deinitialize disjoint _x.1. Assigning _x.0 can restore parent completeness when every required child is initialized.

Not every arbitrary runtime-indexed place receives an independent static path. Aliasing and dynamic indexing limit precision. The representation chooses which initialization questions are cheap and conservatively merges the rest.

Drop elaboration turns conditional destruction obligations into explicit control flow and flags. Before elaboration, a conceptual drop means “drop this place if initialized according to analysis.” After elaboration, control flow tests the required drop flags or follows statically known paths.

flag_x = false
x = construct()       // on success: flag_x = true
...
move x                // flag_x = false
...
if flag_x { Drop(x) }

Real rustc can avoid flags where dataflow proves a fixed state. Flags cost storage and branches but make path-dependent obligations executable.

Core invariants:

• no destructor observes an uninitialized place;
• every initialized, still-owned destructor-bearing value is dropped exactly once on each relevant exit;
• moving a parent updates obligations for descendants;
• assigning a child updates parent completeness;
• normal and cleanup paths use edge-correct initialization facts;
• elaborated control flow remains valid for the next MIR phase.

Workshop: reduce a suspected double drop to a two-field struct whose fields print in Drop. Move one field conditionally, then panic. Compare normal and unwind traces. Dump before and after drop elaboration. If the pre-elaboration initialization fact is wrong, fix move/dataflow logic. If it is right but generated flag updates are wrong, fix elaboration.

Alternative: attach an optional runtime initialized bit to every value. That simplifies elaboration but bloats all programs and still needs partial-place structure. Static analysis removes most runtime cost while paying compiler complexity.

45. Coroutines and async lowering#

An async fn returns a future whose polling resumes suspended computation. Conceptually, lowering creates a state machine containing values live across suspension points. Values dead before an await need not become stored fields.

source async body
  | lower await into suspension-capable control flow
  v
MIR with yield/resume structure
  | compute locals live across each suspension
  v
coroutine layout: discriminant + saved fields
  | transform control and drops
  v
poll-like state dispatch and state-specific cleanup

This is educational, not the stable ABI of futures. Coroutine layout and transform types are internal and version-sensitive.

At each suspension point, the transform must preserve:

1. where execution resumes;
2. which locals remain initialized;
3. which borrows and pinned relationships remain valid;
4. what must be dropped if the future is cancelled in that state;
5. what state marks completion or poisoning where applicable.

Self-references make pinning relevant. Once a future that relies on stable internal addresses is pinned, moving its storage can violate assumptions. The transform does not make arbitrary self-referential movement safe; generated future types and polling APIs participate in the contract.

State-specific drop is mandatory. Cancelling before the first await differs from cancelling after a String was saved. A single unconditional destructor list would either leak or touch uninitialized fields.

Optimization can share storage between locals never live in the same state. This is an interference/layout problem. It saves memory but complicates debuginfo, provenance, and validation. Measure future size and poll performance; do not optimize field count without representative workloads.

Failure experiment:

• create values with logging destructors before and between two awaits;
• poll to each suspension and then drop the future;
• verify exactly the values initialized in that state are destroyed;
• repeat with a panic during polling;
• inspect transformed MIR on the pinned compiler.

An async-only wrong drop often begins in liveness, state layout, or coroutine drop elaboration. An error already visible before the coroutine transform belongs earlier.

46. MIR optimization and validation#

Every optimization is a proof obligation under Rust's observation model. It must preserve returned values, permitted side effects, destructor behavior, panic behavior where observable, alias/provenance constraints, and unwind semantics required by the compilation mode.

Major families include:

Constant propagation commonly uses interpreter machinery so arithmetic, layout, and validity semantics are shared. That reuse reduces duplication but makes interpreter correctness part of optimizer correctness. Failure must usually mean “cannot fold,” not “reject a runtime-valid program.”

Pass ordering matters. Inlining exposes constants and dead stores but increases body size. Simplifying CFG before dataflow can reduce cost; simplifying too early can erase structure required by diagnostics or borrow checking. No ordering is free: it moves work and information.

Validation strategy:

1. validate the input phase in debug/testing configurations;
2. run one pass;
3. validate its declared output phase;
4. compare execution or codegen with optimization disabled;
5. use pass-by-pass dumps to find the first divergence.

Differential tests should include panic and drop traces, not only return values. Property tests can generate small typed CFGs, apply a pass, and compare bounded interpreter outcomes. The generator must produce valid MIR; otherwise it mostly fuzzes validation.

Performance work records compile time, peak memory, optimized code size, and runtime separately. A pass that saves 0.1% runtime while adding 20% compile time may be wrong policy for default builds. Use rustc-perf-style representative workloads and retain before/after profiles.

47. Qualification, promotion, and const contexts#

Const qualification asks whether an expression is permitted or requires special handling in a const context. CTFE asks what happens when eligible MIR is interpreted. They are separate stages with different errors.

Promotion extracts certain rvalues into hidden constant-like allocations or bodies so a reference can have a sufficiently long lifetime. Not every expression that can be evaluated at compile time is promotable. Promotion must account for mutability, interior mutability, destruction, and contextual rules.

const ANSWER: usize = 40 + 2;

fn promoted() -> &'static [i32; 3] {
	&[1, 2, 3]
}

These examples are stable source behavior, but the exact promoted MIR body and pass names are internal.

Keep these questions separate:

• Is the operation allowed in this const context?
• Does it need promotion to satisfy lifetime behavior?
• Can the interpreter evaluate this concrete instance?
• Is the resulting value valid for its type?
• Can the value be serialized into compiler metadata or codegen form?

Moving all qualification into the interpreter gives concrete execution good precision but can make acceptance depend on dead branches and incidental evaluation. Purely syntactic qualification is predictable but rejects semantically harmless cases. rustc's division of checks evolves as const capabilities grow. Pin the version before claiming a particular operation is accepted.

Tests should distinguish const, static, array-length/const-generic, inline const, and promoted runtime contexts. An operation accepted by Miri at runtime is not automatically legal in a const item.

48. Interpreter architecture: values, memory, and machines#

The MIR interpreter is an abstract machine. It executes MIR statements and terminators, tracks stack frames and memory, and delegates policy to a machine configuration. CTFE and Miri-style execution share mechanisms but enforce different permitted operations and goals.

MIR stepper
 | reads/writes
 v
operand/place evaluation ----> layout/type queries
 |                             |
 v                             v
allocation memory <------ target data layout
 |
 v
machine hooks: calls, provenance, validity, externs, limits, diagnostics

An allocation is not merely a host byte vector. It has bytes, initialization state, alignment, mutability, provenance-related information, and identity. Target pointers can differ in width and endianness from host pointers. The interpreter must use target layout, never usize assumptions from the machine running rustc.

Provenance records information about where pointer authority came from. The exact model is actively developed and differs by machine/configuration. The durable warning is that an integer address alone does not capture all rules for dereference.

Validity is type-dependent. An arbitrary byte may be invalid as bool. A reference has stronger requirements than a raw pointer. Reading uninitialized bytes as a typed value can be invalid even though copying them as opaque storage may be modeled differently.

Machine hooks separate mechanism from policy. One machine may reject host interaction for deterministic CTFE. Another may model operating-system calls for Miri under explicit controls. Sharing the stepper does not imply identical accepted programs.

Proof obligations at unsafe interpreter boundaries include:

• checked allocation bounds and offset arithmetic;
• alignment before typed access;
• initialization before semantic reads;
• provenance permission for access;
• target-sized arithmetic;
• validity at required observation points;
• no host pointer dereference for target memory.

An interpreter bug can become a compiler security issue because untrusted crates are input to rustc. Avoid unchecked host allocation growth, integer overflow in sizes, and diagnostics that recursively print attacker-sized structures.

49. A bounded stable-Rust educational interpreter#

The next complete program implements a typed-enough integer CFG, locals, checked arithmetic, branches, calls to no external functions, step limits, and validation. It deliberately omits memory, references, drops, unwind, target layouts, provenance, aggregates, and Rust validity. It is a teaching model, not a Rust interpreter.

#[derive(Clone, Debug)]
enum Operand {
	Const(i64),
	Local(usize),
}

#[derive(Clone, Debug)]
enum Rvalue {
	Use(Operand),
	Add(Operand, Operand),
	Sub(Operand, Operand),
	Mul(Operand, Operand),
	Div(Operand, Operand),
	Eq(Operand, Operand),
}

#[derive(Clone, Debug)]
enum Statement {
	Assign(usize, Rvalue),
}

#[derive(Clone, Debug)]
enum Terminator {
	Goto(usize),
	Switch { cond: Operand, if_true: usize, if_false: usize },
	Return(Operand),
}

#[derive(Clone, Debug)]
struct Block {
	statements: Vec<Statement>,
	terminator: Terminator,
}

#[derive(Clone, Debug)]
struct Body {
	local_count: usize,
	blocks: Vec<Block>,
}

#[derive(Debug, PartialEq, Eq)]
enum Error {
	NoBlocks,
	BadTarget(usize),
	BadLocal(usize),
	Uninitialized(usize),
	Overflow,
	DivisionByZero,
	StepLimit,
}

fn check_operand(op: &Operand, locals: usize) -> Result<(), Error> {
	if let Operand::Local(local) = op {
		if *local >= locals {
			return Err(Error::BadLocal(*local));
		}
	}
	Ok(())
}

fn validate(body: &Body) -> Result<(), Error> {
	if body.blocks.is_empty() {
		return Err(Error::NoBlocks);
	}
	let target = |target: usize| {
		if target < body.blocks.len() { Ok(()) } else { Err(Error::BadTarget(target)) }
	};
	for block in &body.blocks {
		for statement in &block.statements {
			let Statement::Assign(destination, value) = statement;
			if *destination >= body.local_count {
				return Err(Error::BadLocal(*destination));
			}
			let operands: &[&Operand] = match value {
				Rvalue::Use(a) => &[a],
				Rvalue::Add(a, b)
				| Rvalue::Sub(a, b)
				| Rvalue::Mul(a, b)
				| Rvalue::Div(a, b)
				| Rvalue::Eq(a, b) => &[a, b],
			};
			for operand in operands {
				check_operand(operand, body.local_count)?;
			}
		}
		match &block.terminator {
			Terminator::Goto(next) => target(*next)?,
			Terminator::Switch { cond, if_true, if_false } => {
				check_operand(cond, body.local_count)?;
				target(*if_true)?;
				target(*if_false)?;
			}
			Terminator::Return(value) => check_operand(value, body.local_count)?,
		}
	}
	Ok(())
}

fn read(op: &Operand, locals: &[Option<i64>]) -> Result<i64, Error> {
	match op {
		Operand::Const(value) => Ok(*value),
		Operand::Local(local) => locals[*local].ok_or(Error::Uninitialized(*local)),
	}
}

fn eval(value: &Rvalue, locals: &[Option<i64>]) -> Result<i64, Error> {
	let pair = |a: &Operand, b: &Operand| Ok((read(a, locals)?, read(b, locals)?));
	match value {
		Rvalue::Use(a) => read(a, locals),
		Rvalue::Add(a, b) => {
			let (a, b) = pair(a, b)?;
			a.checked_add(b).ok_or(Error::Overflow)
		}
		Rvalue::Sub(a, b) => {
			let (a, b) = pair(a, b)?;
			a.checked_sub(b).ok_or(Error::Overflow)
		}
		Rvalue::Mul(a, b) => {
			let (a, b) = pair(a, b)?;
			a.checked_mul(b).ok_or(Error::Overflow)
		}
		Rvalue::Div(a, b) => {
			let (a, b) = pair(a, b)?;
			if b == 0 {
				Err(Error::DivisionByZero)
			} else {
				a.checked_div(b).ok_or(Error::Overflow)
			}
		}
		Rvalue::Eq(a, b) => {
			let (a, b) = pair(a, b)?;
			Ok(i64::from(a == b))
		}
	}
}

fn run(body: &Body, limit: usize) -> Result<i64, Error> {
	validate(body)?;
	let mut locals = vec![None; body.local_count];
	let mut current = 0;
	let mut steps = 0;
	loop {
		let block = &body.blocks[current];
		for statement in &block.statements {
			if steps == limit {
				return Err(Error::StepLimit);
			}
			steps += 1;
			let Statement::Assign(destination, value) = statement;
			let result = eval(value, &locals)?;
			locals[*destination] = Some(result);
		}
		if steps == limit {
			return Err(Error::StepLimit);
		}
		steps += 1;
		match &block.terminator {
			Terminator::Goto(next) => current = *next,
			Terminator::Switch { cond, if_true, if_false } => {
				current = if read(cond, &locals)? != 0 { *if_true } else { *if_false };
			}
			Terminator::Return(value) => return read(value, &locals),
		}
	}
}

fn main() {
	// Sum 1 + 2 + 3 using locals: sum, i, condition.
	let body = Body {
		local_count: 3,
		blocks: vec![
			Block {
				statements: vec![
					Statement::Assign(0, Rvalue::Use(Operand::Const(0))),
					Statement::Assign(1, Rvalue::Use(Operand::Const(1))),
				],
				terminator: Terminator::Goto(1),
			},
			Block {
				statements: vec![Statement::Assign(
					2,
					Rvalue::Eq(Operand::Local(1), Operand::Const(4)),
				)],
				terminator: Terminator::Switch {
					cond: Operand::Local(2),
					if_true: 3,
					if_false: 2,
				},
			},
			Block {
				statements: vec![
					Statement::Assign(0, Rvalue::Add(Operand::Local(0), Operand::Local(1))),
					Statement::Assign(1, Rvalue::Add(Operand::Local(1), Operand::Const(1))),
				],
				terminator: Terminator::Goto(1),
			},
			Block { statements: vec![], terminator: Terminator::Return(Operand::Local(0)) },
		],
	};
	assert_eq!(run(&body, 100), Ok(6));
	assert_eq!(run(&body, 3), Err(Error::StepLimit));

	let bad = Body {
		local_count: 0,
		blocks: vec![Block { statements: vec![], terminator: Terminator::Goto(9) }],
	};
	assert_eq!(run(&bad, 10), Err(Error::BadTarget(9)));
	println!("educational interpreter checks passed");
}

The evaluator reads both operands before writing a destination. That preserves a simple assignment invariant when destination also appears as an operand. Checked operations make overflow policy explicit. The step counter charges each statement and terminator, so a self-loop cannot evade the limit.

Milestone extensions, in order:

1. add source spans to every instruction and include them in errors;
2. add per-local integer widths and signedness;
3. add stack frames and bounded internal calls;
4. add allocation IDs and checked byte ranges;
5. add initialized-byte masks;
6. add target endianness and pointer width;
7. add pointers with explicit provenance tokens;
8. add type validity checks;
9. add machine hooks rather than hard-coding host access;
10. differential-test only the supported subset.

At each milestone, state omissions again. A toy with allocation IDs but no provenance rules must not claim to detect Rust undefined behavior.

50. Determinism, limits, undefined behavior, and diagnostics#

Compile-time evaluation must be reproducible enough for compiler outputs and incremental behavior. Host time, random state, uncontrolled filesystem access, and host pointer layout cannot silently determine a constant. The CTFE machine rejects or controls such effects.

Resource limits are semantic policy around an implementation mechanism. Possible limits include interpreter steps, recursion depth, allocation size, total memory, diagnostic size, and compiler cancellation. A deterministic limit should count target-level work in a documented-enough manner, not wall-clock milliseconds. Wall time varies across machines and load.

Exhaustion is not proof of an infinite loop. Report that evaluation exceeded a bound and preserve the most useful call/span context. Do not relabel it as undefined behavior.

Undefined behavior detection needs the earliest violated invariant. Examples include out-of-bounds access, invalid alignment, use of uninitialized data, invalid typed values, and provenance violations under the selected model. The exact set and timing differ between CTFE and Miri configurations.

A strong diagnostic answers:

• what operation failed;
• at which source span;
• what allocation/place was involved in readable terms;
• what invariant was required;
• which call or const-eval stack led there;
• whether the cause is language-invalid, const-forbidden, or a resource limit.

Avoid leaking unstable implementation trivia unless it materially helps. Allocation IDs can correlate notes, but source-level names and spans are usually better primary labels. Cap recursive value printing and repeated frames to resist diagnostic denial of service.

Failure workshop:

1. trigger division by zero and verify the arithmetic span;
2. trigger an out-of-bounds read and verify allocation creation context;
3. exceed a step limit with a loop and distinguish it from UB;
4. make a deeply recursive constant and check stack-note truncation;
5. compare Miri and rustc CTFE; classify policy differences instead of assuming one is wrong.

51. Source navigation and version-pinned archaeology#

Start from the owning query or pass registration, not from a remembered filename. In a rustc 1.97.1 checkout, use symbol search because modules move.

rg "struct .*Builder|fn thir|mir_built" compiler/
rg "AnalysisDomain|ResultsCursor" compiler/rustc_mir_dataflow
rg "MovePath|drop.*elabor" compiler/
rg "run_pass|MirPass" compiler/rustc_mir_transform
rg "InterpCx|Machine" compiler/rustc_const_eval
rg "usefulness|Witness" compiler/rustc_pattern_analysis

Names shown are search seeds, not a promise that each exact spelling exists unchanged. Read call sites, tests, and module documentation together. Generated API docs reveal types but rarely explain phase assumptions fully.

Navigation procedure:

1. record git rev-parse HEAD and rustc -vV;
2. reproduce with a minimal crate;
3. identify the earliest wrong representation;
4. find the query producing it;
5. find pass scheduling and the phase accepted by the consumer;
6. read nearby tests for established conventions;
7. write a regression test before broad changes;
8. validate and compare dumps around one boundary.

Do not parse debug MIR as a stable external format. For a tool tied to internals, pin a toolchain and fail clearly on mismatch.

52. Testing, performance, and security hardening#

Different contracts require different tests.

Compile complete educational programs independently. For rustc changes, use the repository's bootstrap test commands for the exact suite rather than assuming ordinary Cargo commands cover compiler tests.

Optimization testing needs at least three axes:

• semantic equivalence with the pass disabled;
• structural expectation immediately after the pass;
• compile-time and generated-code impact on representative crates.

Interpreter fuzzing should generate well-typed, phase-valid MIR where testing semantics. Mutation of arbitrary bytes is still useful for deserializers and validators, but it answers a different question.

Security review treats source code and metadata as attacker-controlled. Audit:

• matrix and CFG complexity explosions;
• allocation size arithmetic overflow;
• recursion in types, witnesses, and diagnostics;
• unbounded interpreter work;
• invalid indices crossing unsafe boundaries;
• target/host size confusion;
• cache keys missing semantic configuration;
• nondeterministic iteration affecting diagnostics or artifacts.

Caching creates correctness obligations. A cached interpretation result must include every input that affects semantics: instance, substitutions, target, features, machine mode, and relevant compiler options. An incomplete key can turn a performance feature into a cross-context miscompile.

53. Contributor workshops and implementation capstones#

Workshop A: edge-correct definite initialization#

Extend the Chapter 43 engine with kills and intersection. Add a call-like terminator whose destination is generated only on one edge. Predict every block input before running tests. Include an unreachable predecessor. The capstone passes only if reachable-empty and unreachable remain distinct.

Workshop B: pattern witness construction#

Implement constructors for bool, Option<T>, and pairs. Represent rows as vectors. Specialize one column recursively and rebuild witnesses on return. Add Option<(bool, bool)> and verify all four missing combinations. Explicitly omit ranges, guards, uninhabitedness, and or-pattern sharing.

Workshop C: move paths and drops#

Model a two-field aggregate. Support move, assign, and scope exit. Derive drop actions from definite initialization. Then add a branch and runtime drop flags. Test move-then-reinitialize on normal and simulated unwind exits.

Workshop D: interpreter memory#

Extend the toy interpreter with allocation IDs and byte arrays. Reject out-of-bounds ranges using checked arithmetic. Track initialized bytes separately. Do not add host pointers. Explain why this still lacks alignment, layouts, provenance, and typed validity.

Workshop E: optimizer differential harness#

Generate tiny arithmetic CFGs without undefined behavior. Interpret before and after constant folding and CFG simplification. Retain the seed and print the smallest failing body. Add checked overflow and a step limit so the oracle itself is deterministic.

Workshop F: first rustc patch#

Find one existing FIXME or diagnostic issue in the owning module. Confirm with the current issue tracker and exact revision. Add the narrow regression first. Use focused x.py test suites, formatting, tidy checks, and compiler-team review guidance. Avoid combining a phase refactor with a semantic fix.

Likely bug locations by first broken invariant:

wrong adjustment             type checking / THIR conversion
wrong coverage answer        pattern usefulness
right answer, bad witness    witness reconstruction/formatting
wrong source evaluation      MIR construction
illegal phase construct      pass scheduling or transform
wrong join fact              dataflow domain/transfer
call destination on unwind   edge effect
partial move confusion       move paths/init analysis
double or missing drop       drop elaboration/cleanup
bad saved async field        coroutine liveness/layout
optimized-only mismatch      first differing MIR pass
const-only policy rejection  qualification/CTFE machine
bad interpreted bytes        allocation/layout/validity
host-dependent result        machine isolation/configuration

54. Derived philosophy, talk plans, and reading map#

The mechanisms above support conclusions more precise than “compilers use intermediate representations.”

Representations determine cheap questions. THIR makes typed source structure cheap. MIR makes predecessor, successor, place, and effect questions cheap. Move paths make partial initialization cheap for selected projections. Allocation identities make interpreter memory ownership cheap to state. Each representation forgets something, so diagnostics need preserved source provenance.

Abstractions move responsibility rather than deleting it. Implicit adjustments simplify source but type checking and THIR must record them. Drop flags simplify dynamic destruction decisions but elaboration must update them exactly. Machine hooks share interpretation mechanism but every machine must define policy.

Optimization preserves meaning under an observation model. Removing a store is valid only if no legal observation sees it. Panic, drop, alias, provenance, and unwind behavior define observations. Calling a transformation “obvious” does not discharge that proof obligation.

Identity is not an address, index, or source location. A THIR index is arena-local. A MIR local can be rewritten. An allocation ID is not necessarily a numeric machine address. Confusing these identities produces stale references and provenance mistakes.

Uncertainty must be represented. Unreachable is not an empty fact set. Maybe initialized is not definitely initialized. Interpreter exhaustion is not nontermination. An opaque pointer is not permission to guess provenance.

Caching creates semantic debt. Every omitted input from a cache key is a possible cross-context wrong answer. Performance improves only while invalidation remains correct.

The visible failure is downstream of the first broken invariant. A machine-code crash can begin with a wrong adjustment. A double destructor can begin with a missing edge effect. Debug forward through representations and stop at the first divergence.

A 30-minute talk#

1. Show one adjusted expression and its MIR places.
2. Trace one branch through dataflow.
3. Move one struct field and explain a drop flag.
4. Execute one constant through an abstract allocation.
5. End with the earliest-invariant debugging map.

A 60-minute contributor talk#

1. Establish THIR and MIR phase contracts.
2. Derive usefulness using Option<bool>.
3. Lower short-circuiting and a may-unwind call.
4. Derive union and intersection analyses.
5. Trace move paths into drop elaboration.
6. Explain coroutine state-specific cleanup.
7. Separate interpreter mechanism from machine policy.
8. Demonstrate pass-by-pass bug isolation and testing.

Mastery checks#

You are ready to contribute when you can:

• explain why adjustment order is not reconstructible from syntax alone;
• derive a missing witness without enumerating values;
• state a pass's input and output phase invariants;
• lower effects without duplicating or reordering them;
• distinguish places, values, storage, initialization, and destruction;
• derive forward and backward equations with edge effects;
• predict partial-move and drop-flag states;
• explain cancellation drops for each async suspension state;
• review an optimization against panic, alias, and unwind observations;
• separate qualification, promotion, interpretation, and validity;
• explain why target memory is not host memory;
• design deterministic limits and useful UB diagnostics;
• locate the earliest wrong query product and write its narrow regression.

Authoritative reading map#

• rustc-dev-guide: THIR
• rustc-dev-guide: pattern and exhaustiveness checking
• rustc-dev-guide: MIR
• rustc-dev-guide: MIR construction
• rustc-dev-guide: MIR dataflow
• rustc-dev-guide: MIR optimizations
• rustc-dev-guide: constant evaluation
• rustc-dev-guide: MIR interpretation
• nightly rustc internal API: rustc_middle::thir
• nightly rustc internal API: rustc_middle::mir
• nightly rustc internal API: rustc_mir_dataflow
• nightly rustc internal API: rustc_const_eval
• The Rust Reference: patterns
• The Rust Reference: constant evaluation
• The Rust Reference: behavior considered undefined
• The Rust Reference: destructors
• Miri source and documentation
• Rust compiler source

For version-sensitive statements, read these at the commit corresponding to rustc 1.97.1. Nightly API pages follow the current nightly and may already differ. Historical design notes explain motivation but do not override current source or the Reference.

The final working rule is operational: preserve typed decisions into THIR, preserve execution and destruction semantics into MIR, preserve phase invariants through every transform, and preserve target-machine meaning through interpretation.

Part V-C: Async Lowering, MIR Analyses, and Optimization#

This continuation closes the operational gaps between checked source and optimized MIR. It assumes basic Rust and Parts V and V-B, but restates every invariant needed here. Unless a statement cites the language specification, rustc names and ordering describe the 1.97.1 development line and may change. The educational IR below is not rustc MIR.

55. The checking boundary: patterns, unsafe operations, and MIR#

Three checks near MIR construction answer different questions.

None is borrow checking. Borrow checking asks whether references and moves obey temporal rules. Unsafe checking asks whether the programmer acknowledged an operation whose contract the compiler cannot prove. An unsafe block does not relax ownership, lifetime, type, initialization, or alias checks.

Consider:

unsafe fn read(p: *const i32) -> i32 {
	unsafe { *p }
}

The raw dereference is an unsafe operation. The inner block supplies an unsafe context, including under unsafe_op_in_unsafe_fn discipline. It does not prove that p is aligned, initialized, live, or points to an i32. Those are the caller and implementation proof obligations.

The useful integration rule is: resolve meaning before classifying danger. Method resolution, adjustment insertion, and overloaded operators determine which operation exists. Pattern checking determines whether lowering has all cases. Only then can MIR construction make control flow explicit without inventing a default case.

parsed unsafe syntax --scope nesting-------------------+
                                                       |
typed expressions --resolved operation + adjustments--+--> unsafety check
                                                       |      |
typed patterns --constructor analysis--> coverage -----+      ` diagnostic span
                                |
                                ` exhaustive match --> MIR decision control flow

Arrows carry decisions, not merely syntax. The simplification is that exact query boundaries are omitted.

Pattern integration trace#

For match x: Option<bool> { None => 0, Some(true) => 1 }, usefulness specializes constructors.

space                    remaining after arm
{None, Some(false),      initial
 Some(true)}
{Some(false),Some(true)} after None
{Some(false)}            after Some(true)

The witness is Some(false). MIR must not silently add unreachable for that valid value. After adding Some(false), lowering may emit a variant switch and then a boolean switch. Guards do not normally provide unconditional coverage because a guard can evaluate false.

Invariants:

1. every operation classified unsafe is the operation selected by type checking;
2. every diagnostic points to a source scope that encloses that operation;
3. constructor specialization preserves exactly the represented value set;
4. exhaustiveness is established before ordinary execution lowering assumes full coverage;
5. an internal complexity limit fails conservatively, never by accepting unsound code.

Counterfactual: checking raw syntax is simpler, but misses implicit dereferences and misclassifies overloaded operations. Checking only optimized MIR is smaller, but optimization may erase the source operation and its useful span. The compiler pays to preserve typed source structure because moving the check later loses evidence.

Security review must include maliciously nested or-patterns, huge ranges, macros with misleading spans, and unsafe operations introduced by adjustments. The first debugging question is whether the typed operation is right, not whether the final machine instruction looks dangerous.

56. From async syntax to a coroutine#

An async function does not execute its body when called. It constructs a future. Polling that future resumes computation until it returns Ready or reaches an await whose child future returns Pending.

async fn sum(a: i32) -> i32 {
	let b = a + 1;
	ready(b).await + 2
}

async fn ready(x: i32) -> i32 { x }

At the language level, this is a future-producing function. Inside rustc, async syntax is progressively desugared and its coroutine body is transformed. Do not treat the exact intermediate syntax or pass name as stable API.

One useful mental model is a tagged record plus a resume function:

future value
+------------------------------+
| discriminant: start/s1/done  |
| argument a                    |
| child future, when suspended |
| locals live across await     |
+------------------------------+
              |
              | Pin<&mut Self>, Context
              v
        resume/poll dispatch
        start -> s1 -> done

The discriminant identifies the legal interpretation of fields. Only locals live across a suspension need saved fields. Ordinary temporaries dead before suspension can remain stack-like within one resume call.

Concrete state trace#

Conceptual pre-transform MIR:

bb0: b = Add(a, 1)
     child = ready(b)
     goto bb1
bb1: p = poll(Pin(&mut child), cx)
     switch p -> Pending: bb2, Ready(v): bb3
bb2: yield Pending, resume bb1, drop bb_drop
bb3: out = Add(v, 2)
     return Ready(out)

Conceptual post-transform behavior:

poll(self, cx):
  switch self.state
    Start: self.b = self.a + 1; self.child = ready(self.b); self.state = S1
    S1:    continue
    Done:  panic("polled after completion") [policy/detail is implementation-sensitive]
  p = poll(Pin(&mut self.child), cx)
  if p == Pending { self.state = S1; return Pending }
  self.state = Done
  return Ready(p.value + 2)

This is explanatory pseudocode, not promised MIR syntax. The actual body also carries storage, drop, unwind, source, and layout details.

The transform needs four analyses:

• suspension points and their resume/drop successors;
• local liveness at each suspension;
• initialization state, because a live field may not yet be initialized;
• borrowing/layout constraints for references into saved state.

Core invariants:

1. resume dispatch enters only a state compatible with the discriminant;
2. every value needed after suspension is saved before returning Pending;
3. no value dead at suspension is accidentally retained merely by the model;
4. each saved field is read only in states where it is initialized;
5. completion cannot run the completed path twice;
6. cancellation drops exactly the initialized fields for the current state;
7. source evaluation order and unwind behavior are preserved.

Saving every local would simplify liveness but increase future size and may extend destruction or auto-trait consequences. Saving too few is a use-after-return bug. The representation is therefore a soundness boundary, not merely compression.

57. Suspension, pinning, resume, and cancellation#

Pinning addresses a specific problem: a suspended coroutine can contain a reference into itself. Moving the enclosing future would then leave the saved reference pointing to the old address. Pin<&mut F> prevents safe code from moving an F that may rely on address stability. It does not pin every field independently, make arbitrary projection safe, or promise that the value was never moved before pinning.

before first poll                after suspension
future at A                      future at A
  buf [saved]                      buf [saved]
  ref ----+                        ref ----+
          `--> A.buf                       `--> A.buf

illegal move to B would leave ref --------> A.buf

The arrow is a stored reference. The diagram omits provenance and layout details.

Borrow checking and coroutine transformation meet at a delicate boundary. Borrow checking must reason about loans crossing a suspension. The transform later turns relevant locals into fields and makes states explicit. A compiler cannot simply run ordinary stack-local borrow checking and assume lowering preserves every fact. Its phase contracts must ensure the checked representation and transformed layout agree.

Cancellation is destruction while suspended. Suppose x is initialized before the first await and y after it.

State Start: initialized {}
State S1:    initialized {x, child1}
State S2:    initialized {x, y, child2}
State Done:  initialized {}

Conceptual drop dispatch:

switch state
  Start -> drop nothing
  S1    -> drop child1; drop x
  S2    -> drop child2; drop y; drop x
  Done  -> drop nothing

Order shown is illustrative; actual order follows Rust destruction semantics and lowering details. Unwinding during a destructor follows cleanup policy and must not cause another initialized field to be forgotten or dropped twice.

Earliest-invariant debugging#

State updates must occur at a point that makes cancellation and panic coherent. Updating too early can advertise initialized fields that are not ready. Updating too late can make a constructed child invisible to cleanup. Review each potentially panicking operation between initialization and discriminant update.

Cost model for a future with S suspension states and L locals:

• naive liveness storage is O(SL) bits;
• sparse sets help when few locals cross each await;
• dispatch is usually a switch over S states;
• layout is approximately the tag plus the maximum/union of simultaneously saved state, subject to alignment and implementation choices;
• drop code can grow with states and initialized-field combinations.

Measure future size, poll latency, code size, and compile time separately. Shrinking fields can increase dispatch or recomputation.

58. MIR dataflow as a reusable framework#

A dataflow analysis maps each program point to an abstract fact. The fact is not the concrete machine state; it retains only information needed by a question.

For a forward may-analysis:

IN[b]  = union OUT[p] for predecessors p of b
OUT[b] = transfer_block(b, IN[b])

For a backward may-analysis:

OUT[b] = union IN[s] for successors s of b
IN[b]  = transfer_block_backward(b, OUT[b])

A finite-height lattice and monotone transfer functions guarantee worklist convergence. Bitsets are common because locals, move paths, or expressions form a finite indexed universe.

Framework concerns must remain separate from analysis policy:

Unreachable is not always represented by the empty fact. In a must-analysis, empty can mean “nothing definitely holds,” while an unreachable block has no execution evidence at all. Conflating them corrupts joins.

Forward trace: maybe initialized#

bb0: x = 1; switch c -> bb1, bb2
bb1: y = x; goto bb3
bb2: x = 2; goto bb3
bb3: use x

With facts as maybe-initialized locals:

OUT[bb0] = {x,c}
OUT[bb1] = {x,c,y}
OUT[bb2] = {x,c}
IN [bb3] = union = {x,c,y}

For definitely initialized, the join is intersection and y is absent at bb3. Changing only the join changes the question.

Edge effects#

A call destination is initialized on normal return, not on unwind.

bb0 Call f() -> bb_ok
       |
       ` unwind -> bb_cleanup

bb_ok input:      destination initialized
bb_cleanup input: destination unchanged

Applying the effect to the terminator before choosing an edge is wrong. The framework needs edge-specific transfer or equivalent modeling.

Complexity for B blocks, E edges, W machine words per fact, and lattice height H is roughly worklist-dependent but bounded by repeated edge propagation, often discussed as O(E * H * W). Scheduling and sparse representations change constants dramatically. Profile joins, cloning, and cursor reconstruction before redesigning the domain.

59. A stable-Rust mini pass manager and two analyses#

The following complete dependency-free program defines a tiny IR, validates it, computes forward reachability and backward liveness, then runs simple optimizations. It is deliberately smaller than rustc: integers only, no places, aliases, drops, unwind, calls, overflow modes, provenance, or source scopes.

use std::collections::{BTreeSet, VecDeque};

type Local = usize;
type Block = usize;

#[derive(Clone, Debug, PartialEq, Eq)]
enum Value { Local(Local), Const(i64) }

#[derive(Clone, Debug, PartialEq, Eq)]
enum Inst {
	Set(Local, Value),
	Add(Local, Value, Value),
	Print(Value),
	Nop,
}

#[derive(Clone, Debug, PartialEq, Eq)]
enum Term {
	Goto(Block),
	If(Value, Block, Block),
	Return(Value),
}

#[derive(Clone, Debug, PartialEq, Eq)]
struct BasicBlock { insts: Vec<Inst>, term: Term }

#[derive(Clone, Debug, PartialEq, Eq)]
struct Body { locals: usize, blocks: Vec<BasicBlock>, phase: Phase }

#[derive(Clone, Copy, Debug, PartialEq, Eq, PartialOrd, Ord)]
enum Phase { Built, Simplified, Optimized }

fn uses(v: &Value, out: &mut BTreeSet<Local>) {
	if let Value::Local(l) = v { out.insert(*l); }
}

fn successors(t: &Term) -> Vec<Block> {
	match *t {
		Term::Goto(b) => vec![b],
		Term::If(_, a, b) if a == b => vec![a],
		Term::If(_, a, b) => vec![a, b],
		Term::Return(_) => vec![],
	}
}

fn validate(body: &Body) -> Result<(), String> {
	if body.blocks.is_empty() { return Err("body has no entry".into()); }
	let local = |l: Local| {
		if l < body.locals { Ok(()) } else { Err(format!("bad local {l}")) }
	};
	let value = |v: &Value| match v { Value::Local(l) => local(*l), Value::Const(_) => Ok(()) };
	for (bb, data) in body.blocks.iter().enumerate() {
		for i in &data.insts {
			match i {
				Inst::Set(d, v) => { local(*d)?; value(v)?; }
				Inst::Add(d, a, b) => { local(*d)?; value(a)?; value(b)?; }
				Inst::Print(v) => value(v)?,
				Inst::Nop => {}
			}
		}
		match &data.term {
			Term::Goto(t) => if *t >= body.blocks.len() { return Err(format!("bb{bb}: bad target")); },
			Term::If(v, a, b) => {
				value(v)?;
				if *a >= body.blocks.len() || *b >= body.blocks.len() {
					return Err(format!("bb{bb}: bad branch target"));
				}
			}
			Term::Return(v) => value(v)?,
		}
	}
	Ok(())
}

fn reachable(body: &Body) -> BTreeSet<Block> {
	let mut seen = BTreeSet::new();
	let mut work = VecDeque::from([0]);
	while let Some(bb) = work.pop_front() {
		if !seen.insert(bb) { continue; }
		work.extend(successors(&body.blocks[bb].term));
	}
	seen
}

fn live_in(body: &Body) -> Vec<BTreeSet<Local>> {
	let mut input = vec![BTreeSet::new(); body.blocks.len()];
	let mut changed = true;
	while changed {
		changed = false;
		for bb in (0..body.blocks.len()).rev() {
			let data = &body.blocks[bb];
			let mut state = BTreeSet::new();
			for s in successors(&data.term) { state.extend(input[s].iter().copied()); }
			match &data.term {
				Term::If(v, _, _) | Term::Return(v) => uses(v, &mut state),
				Term::Goto(_) => {}
			}
			for i in data.insts.iter().rev() {
				match i {
					Inst::Set(d, v) => { state.remove(d); uses(v, &mut state); }
					Inst::Add(d, a, b) => { state.remove(d); uses(a, &mut state); uses(b, &mut state); }
					Inst::Print(v) => uses(v, &mut state),
					Inst::Nop => {}
				}
			}
			if state != input[bb] { input[bb] = state; changed = true; }
		}
	}
	input
}

trait Pass {
	fn name(&self) -> &'static str;
	fn min_phase(&self) -> Phase;
	fn output_phase(&self) -> Phase;
	fn run(&self, body: &mut Body) -> bool;
}

struct Manager { validate_each: bool, passes: Vec<Box<dyn Pass>> }

impl Manager {
	fn run(&self, body: &mut Body) -> Result<Vec<String>, String> {
		validate(body)?;
		let mut log = Vec::new();
		for pass in &self.passes {
			if body.phase < pass.min_phase() {
				return Err(format!("{} cannot consume {:?}", pass.name(), body.phase));
			}
			let changed = pass.run(body);
			body.phase = pass.output_phase();
			if self.validate_each { validate(body)?; }
			log.push(format!("{}: changed={changed}", pass.name()));
		}
		Ok(log)
	}
}

struct SimplifyCfg;
impl Pass for SimplifyCfg {
	fn name(&self) -> &'static str { "simplify-cfg" }
	fn min_phase(&self) -> Phase { Phase::Built }
	fn output_phase(&self) -> Phase { Phase::Simplified }
	fn run(&self, body: &mut Body) -> bool {
		let before = body.blocks.len();
		let keep = reachable(body);
		let mut map = vec![usize::MAX; before];
		let mut blocks = Vec::new();
		for old in keep { map[old] = blocks.len(); blocks.push(body.blocks[old].clone()); }
		for data in &mut blocks {
			match &mut data.term {
				Term::Goto(a) => *a = map[*a],
				Term::If(_, a, b) => { *a = map[*a]; *b = map[*b]; }
				Term::Return(_) => {}
			}
		}
		body.blocks = blocks;
		before != body.blocks.len()
	}
}

struct ConstFold;
impl Pass for ConstFold {
	fn name(&self) -> &'static str { "const-fold" }
	fn min_phase(&self) -> Phase { Phase::Simplified }
	fn output_phase(&self) -> Phase { Phase::Optimized }
	fn run(&self, body: &mut Body) -> bool {
		let mut changed = false;
		for data in &mut body.blocks {
			for i in &mut data.insts {
				if let Inst::Add(d, Value::Const(a), Value::Const(b)) = i.clone() {
					if let Some(n) = a.checked_add(b) {
						*i = Inst::Set(d, Value::Const(n)); changed = true;
					}
				}
			}
		}
		changed
	}
}

struct DeadStores;
impl Pass for DeadStores {
	fn name(&self) -> &'static str { "dead-stores" }
	fn min_phase(&self) -> Phase { Phase::Optimized }
	fn output_phase(&self) -> Phase { Phase::Optimized }
	fn run(&self, body: &mut Body) -> bool {
		let input = live_in(body);
		let mut changed = false;
		for bb in (0..body.blocks.len()).rev() {
			let succ = successors(&body.blocks[bb].term);
			let mut live = BTreeSet::new();
			for s in succ { live.extend(input[s].iter().copied()); }
			match &body.blocks[bb].term {
				Term::If(v, _, _) | Term::Return(v) => uses(v, &mut live),
				Term::Goto(_) => {}
			}
			for i in body.blocks[bb].insts.iter_mut().rev() {
				match i.clone() {
					Inst::Set(d, v) if !live.contains(&d) => { *i = Inst::Nop; changed = true; drop(v); }
					Inst::Add(d, a, b) if !live.contains(&d) => { *i = Inst::Nop; changed = true; drop((a,b)); }
					Inst::Set(d, v) => { live.remove(&d); uses(&v, &mut live); }
					Inst::Add(d, a, b) => { live.remove(&d); uses(&a, &mut live); uses(&b, &mut live); }
					Inst::Print(v) => uses(&v, &mut live),
					Inst::Nop => {}
				}
			}
		}
		changed
	}
}

fn eval(body: &Body) -> (i64, Vec<i64>) {
	let mut locals = vec![0; body.locals];
	let mut output = Vec::new();
	let get = |v: &Value, ls: &[i64]| match v { Value::Local(l) => ls[*l], Value::Const(n) => *n };
	let mut bb = 0;
	for _fuel in 0..1000 {
		let data = &body.blocks[bb];
		for i in &data.insts {
			match i {
				Inst::Set(d, v) => locals[*d] = get(v, &locals),
				Inst::Add(d, a, b) => locals[*d] = get(a, &locals).checked_add(get(b, &locals)).unwrap(),
				Inst::Print(v) => output.push(get(v, &locals)),
				Inst::Nop => {}
			}
		}
		match &data.term {
			Term::Goto(t) => bb = *t,
			Term::If(v, a, b) => bb = if get(v, &locals) != 0 { *a } else { *b },
			Term::Return(v) => return (get(v, &locals), output),
		}
	}
	panic!("educational interpreter exhausted fuel")
}

fn sample() -> Body {
	Body { locals: 3, phase: Phase::Built, blocks: vec![
		BasicBlock { insts: vec![Inst::Add(0, Value::Const(20), Value::Const(22)),
			Inst::Set(2, Value::Const(99))], term: Term::If(Value::Const(1), 1, 2) },
		BasicBlock { insts: vec![Inst::Print(Value::Local(0))], term: Term::Return(Value::Local(0)) },
		BasicBlock { insts: vec![], term: Term::Return(Value::Const(-1)) },
		BasicBlock { insts: vec![], term: Term::Goto(3) },
	] }
}

fn main() {
	let original = sample();
	let expected = eval(&original);
	let mut optimized = original.clone();
	let manager = Manager { validate_each: true, passes: vec![
		Box::new(SimplifyCfg), Box::new(ConstFold), Box::new(DeadStores),
	] };
	let log = manager.run(&mut optimized).unwrap();
	assert_eq!(eval(&optimized), expected);
	assert_eq!(expected, (42, vec![42]));
	assert_eq!(optimized.blocks.len(), 3);
	assert!(log.iter().all(|line| line.contains("changed=true")));
	println!("{log:#?}");
}

#[cfg(test)]
mod tests {
	use super::*;

	#[test]
	fn forward_and_backward_facts() {
		let b = sample();
		assert_eq!(reachable(&b), BTreeSet::from([0, 1, 2]));
		let live = live_in(&b);
		assert!(live[1].contains(&0));
		assert!(!live[1].contains(&2));
	}

	#[test]
	fn optimization_is_differentially_equivalent() {
		for n in -100..=100 {
			let mut before = sample();
			before.blocks[0].insts[0] = Inst::Add(0, Value::Const(n), Value::Const(7));
			let expected = eval(&before);
			let mut after = before.clone();
			Manager { validate_each: true, passes: vec![
				Box::new(SimplifyCfg), Box::new(ConstFold), Box::new(DeadStores),
			] }.run(&mut after).unwrap();
			assert_eq!(eval(&after), expected, "input {n}");
		}
	}

	#[test]
	fn validation_rejects_bad_target() {
		let mut b = sample();
		b.blocks[0].term = Term::Goto(99);
		assert!(validate(&b).unwrap_err().contains("bad target"));
	}
}

Compile and test it by saving the block as mini.rs outside the repository:

rustc --edition=2021 mini.rs
./mini
rustc --edition=2021 --test mini.rs -o mini-test
./mini-test

The forward analysis is reachability. The backward analysis is liveness. The differential oracle compares interpreter-visible return and output before and after optimization. The loop over inputs is a bounded property test without an external crate.

Explicit omissions matter. Dead-store removal here is valid only because Set and Add have no side effects except checked overflow in the interpreter, and the pass refuses to fold overflowing constants. In real Rust, an apparently unused assignment may evaluate a panic, volatile access, destructor, allocation, or provenance-sensitive operation. This pass must not be transplanted into rustc.

60. Coroutine transform model in stable Rust#

The next complete program models state, saved locals, pending, completion, and cancellation drops. It does not implement Future, self-references, pin projection, unwind, or real wakers. Its purpose is to make the state invariant executable.

#[derive(Debug, Clone, Copy, PartialEq, Eq)]
enum Poll<T> { Pending, Ready(T) }

#[derive(Debug, Clone, Copy, PartialEq, Eq)]
enum State { Start, Waiting, Done }

#[derive(Debug)]
struct Child { remaining: u8, value: i32 }

impl Child {
	fn poll(&mut self) -> Poll<i32> {
		if self.remaining == 0 { Poll::Ready(self.value) }
		else { self.remaining -= 1; Poll::Pending }
	}
}

#[derive(Debug)]
struct Machine {
	state: State,
	argument: i32,
	saved_b: Option<i32>,
	child: Option<Child>,
	drop_log: Vec<&'static str>,
}

impl Machine {
	fn new(argument: i32, delays: u8) -> Self {
		Self {
			state: State::Start,
			argument,
			saved_b: None,
			child: Some(Child { remaining: delays, value: argument + 1 }),
			drop_log: vec![],
		}
	}

	fn resume(&mut self) -> Poll<i32> {
		match self.state {
			State::Start => {
				self.saved_b = Some(self.argument + 1);
				self.state = State::Waiting;
			}
			State::Waiting => {}
			State::Done => panic!("resumed completed machine"),
		}
		let child = self.child.as_mut().expect("waiting state owns child");
		match child.poll() {
			Poll::Pending => Poll::Pending,
			Poll::Ready(v) => {
				assert_eq!(Some(v), self.saved_b);
				self.child = None;
				self.saved_b = None;
				self.state = State::Done;
				Poll::Ready(v + 2)
			}
		}
	}

	fn cancel(&mut self) {
		match self.state {
			State::Start => {}
			State::Waiting => {
				if self.child.take().is_some() { self.drop_log.push("child"); }
				if self.saved_b.take().is_some() { self.drop_log.push("b"); }
			}
			State::Done => {}
		}
		self.state = State::Done;
	}
}

fn main() {
	let mut m = Machine::new(39, 1);
	assert_eq!(m.resume(), Poll::Pending);
	assert_eq!(m.resume(), Poll::Ready(42));
	assert_eq!(m.state, State::Done);

	let mut cancelled = Machine::new(5, 3);
	assert_eq!(cancelled.resume(), Poll::Pending);
	cancelled.cancel();
	assert_eq!(cancelled.drop_log, vec!["child", "b"]);
}

#[cfg(test)]
mod tests {
	use super::*;

	#[test]
	fn every_delay_finishes_once() {
		for delay in 0..20 {
			let mut m = Machine::new(7, delay);
			let mut pending = 0;
			loop {
				match m.resume() {
					Poll::Pending => pending += 1,
					Poll::Ready(v) => { assert_eq!(v, 10); break; }
				}
			}
			assert_eq!(pending, delay as usize);
			assert_eq!(m.state, State::Done);
		}
	}

	#[test]
	fn cancellation_is_idempotent_in_this_model() {
		let mut m = Machine::new(1, 2);
		assert_eq!(m.resume(), Poll::Pending);
		m.cancel();
		m.cancel();
		assert_eq!(m.drop_log, vec!["child", "b"]);
	}
}

The Option fields are educational drop flags. Production layout does not have to use these exact Rust types. The state says which interpretation is legal; field presence independently records initialization in this model.

Transform exercise:

1. add a second suspension after the child completes;
2. derive the live locals for both suspension points before adding fields;
3. add a WaitingSecond state;
4. write cancellation expectations before code;
5. inject cancellation at every pending return;
6. verify each logical resource appears exactly once in the log.

Counterexample: set Done before extracting the ready value. If extraction could panic, cleanup sees Done and may skip initialized fields. Counterexample: clear child after returning Ready. There is no later execution in which to clear it. The sequencing is part of correctness.

61. CTFE is not MIR constant propagation#

Both systems compute constants, but they have different contracts.

CTFE must reject an operation forbidden in a const context even if runtime execution would be valid. Const propagation can simply leave an expression unchanged if it cannot prove a replacement. An optimization timeout must not become a program error.

source: const N: usize = f(3);
CTFE:   interpret f under const machine policy -> required value or diagnostic

source: let n = f(3);
ConstProp: infer n=... only if analysis proves it cheaply and safely
otherwise: retain runtime call

A useful scalar lattice is:

          Unknown
         /   |   \
       C(0) C(1) C(n)
         \   |   /
       Unreachable

Here Unreachable means no execution reaches the point and Unknown means executions may disagree. The exact ordering notation varies; state it before implementing joins.

Constant propagation trace:

before:
  _1 = const 20
  _2 = Add(_1, const 22)
  switchInt(const true) -> [1: bb_yes, otherwise: bb_no]

facts:
  after stmt 1: _1 = 20
  after stmt 2: _2 = 42

after:
  _1 = const 20
  _2 = const 42
  goto bb_yes

Removing _1 and its store is a later dead-store decision. Combining all changes in one pass makes a wrong transformation harder to isolate.

Overflow is part of the observation model. Folding must match the operation's checked, wrapping, unchecked, or target-width semantics. Host usize is not target usize during cross-compilation. Floating-point, pointer provenance, validity, and relocations require still more care.

Security rules:

• never execute target code as host code to “speed up” CTFE;
• meter interpreter steps and allocations deterministically;
• do not let optimization infer a value from undefined behavior;
• preserve panic and side-effect behavior;
• treat malformed internal constants as compiler bugs, not trusted input.

62. Simplification, inlining, dead code, and constant propagation#

“Simplification” is a family, not a proof. Each pass needs a stated observation model and precondition.

CFG simplification#

Typical rewrites include removing unreachable blocks, threading trivial gotos, merging compatible blocks, and simplifying known switches.

before: bb0 -> bb1; bb1 -> bb2; bb_dead: return 9
after:  bb0 -> bb2

Preserve cleanup classification, source coverage needs, unwind edges, and phase legality. Repeated jump threading can be quadratic if every rewrite rescans all predecessors. Maintain predecessor data or batch rewrites.

MIR inlining#

Inlining copies a callee body into a call site, substitutes arguments and return destination, remaps locals and blocks, and reconnects normal and unwind exits.

caller before:
  _3 = Call add1(copy _2) -> bb1, unwind bb9

callee:
  _0 = Add(copy _1, const 1)
  return

caller after, conceptual:
  _inl_arg = copy _2
  _inl_ret = Add(copy _inl_arg, const 1)
  _3 = move _inl_ret
  goto bb1

Inlining preserves behavior but changes opportunities, code size, compile time, stack traces, and diagnostics. It must respect ABI-relevant semantics, recursion controls, optimization attributes, and unwind behavior. The 1.97.1 heuristics and thresholds are implementation details.

A cost model can score callee statements, calls, loops, and caller frequency estimates. Let benefit estimate B, size cost C, and budget K. Inline only when policy judges B - C favorable and cumulative cost stays below K. This is policy, not correctness. Correctness still requires exact remapping.

Dead-store elimination#

A store is dead when its written value cannot be observed before being overwritten or storage ends.

before: _1 = 4; _1 = 5; print(_1)
after:           _1 = 5; print(_1)

The right-hand side must also be unobservable before deleting the whole statement. _1 = may_panic() is not equivalent to nothing merely because _1 is dead. Aliased memory, volatile operations, raw pointers, drops, and unwind make local reasoning insufficient.

Dead-code elimination#

Unreachable blocks are dead control flow. Unused pure computations are dead value flow. These are related but not identical. A block reachable only through an unwind edge is not unreachable. A statement producing no used value can still panic.

Constant propagation#

Propagation substitutes proven constants. Folding computes operations on them. Branch simplification uses a constant condition. Dead-code elimination removes the now-unreachable branch. Dead-store elimination removes obsolete producers. Scheduling them in a cleanup loop can expose more opportunities:

ConstProp -> simplify switch -> remove unreachable blocks
    ^                                  |
    |                                  v
    `------ expose constants <- remove dead stores

Do not iterate without a bound or convergence metric. A pair of passes that alternately canonicalize forms can loop forever.

Optimization proof checklist#

For every rewrite state:

1. matched input form and phase;
2. replacement type and phase;
3. normal result equivalence;
4. panic equivalence;
5. unwind and cleanup equivalence;
6. drop and storage equivalence;
7. alias, provenance, and validity assumptions;
8. target-width and overflow semantics;
9. source/coverage/diagnostic consequences;
10. a differential or targeted regression test.

63. Pass management, dialects, validation, and diagnostics#

A pass manager turns individual correct transforms into a pipeline with explicit contracts. Its mechanism includes ordering, enabling, validation, dumps, timing, and change reporting. Optimization policy decides which optional passes and thresholds to use.

Built phase
  | validate Built
  | required lowering
  v
Analysis phase -- borrow/move checks consume this contract
  | validate Analysis
  | coroutine/drop/runtime lowering
  v
Runtime phase
  | validate Runtime after each risky pass
  | optional optimizations under budgets
  v
Optimized phase -- codegen contract

The exact rustc 1.97.1 phases, dialect names, query forks, and pass lists must be read at the pinned source revision. This diagram is a durable conceptual contract only.

Validation should reject:

• nonexistent block/local targets;
• a missing terminator;
• type-invalid assignments or projections;
• constructs forbidden in the current dialect;
• malformed unwind or cleanup edges;
• impossible coroutine state/layout references;
• call destinations initialized on the wrong edge;
• stale source-scope and debug-place references;
• broken storage or drop invariants required by the phase.

Validation cannot prove optimization equivalence. Well-typed wrong MIR remains wrong. Differential tests, interpreter comparison, codegen tests, and semantic reasoning cover that gap.

Diagnostic preservation#

Optimization often merges or removes locations. Keep enough provenance to answer where a panic, lint, coverage region, or compiler bug originated. Copying a source span mechanically can also lie: a synthesized operation may have several origins. Use explicit source-info policy for each transform.

A useful pass log records:

compiler revision and target
body identity and queried MIR product
input/output phase
pass name and enabled reason
changed/no-change
block/local/statement counts before and after
validation result
time and optional dump path

Do not print user secrets or enormous constants by default. Diagnostics and dumps process adversarial source. Bound rendering and preserve deterministic ordering.

Counterfactual: one giant optimization pass avoids intermediate traversals. It also entangles proofs, prevents bisection, and hides which rewrite first diverged. Many small passes cost traversal time but improve testing and attribution. The right granularity follows invariants, not a universal pass-size rule.

Counterfactual: validate only at codegen. This lowers compiler time but reports corruption far downstream. Validation immediately after risky transforms localizes bugs; release policy can tune frequency without deleting tests.

64. Testing, benchmarks, and adversarial hardening#

Optimization tests need independent oracles. Snapshotting only optimized MIR proves stability, not correctness.

Use a layered matrix:

Coroutine cancellation matrix#

For every suspension state test:

• immediate pending then drop;
• several pending polls then drop;
• ready transition;
• panic before state update;
• panic after field initialization;
• nested future cancellation;
• partially moved or conditionally initialized saved values;
• Send/Sync expectations when captures cross await;
• future size where layout is the regression.

Property design#

Generate only valid tiny IR first. Keep block and local counts bounded. Interpret before and after with a fuel limit. Compare return, output, panic class, and relevant drop log. When a mismatch appears, shrink statements, then edges, then locals while preserving validity.

Undefined behavior is not an oracle value. Exclude UB programs or compare only executions defined under the same model. Likewise, do not compare host overflow with a different target width.

Representative benchmarks#

Measure independently:

• total compile wall time;
• time per analysis/pass;
• peak memory and fact-set bytes;
• join count and changed-bit count;
• MIR blocks, statements, and locals;
• generated code size;
• runtime on representative workloads;
• future size and poll cost for async cases.

Synthetic diamond graphs stress joins. Long chains stress traversal. Dense loops stress fixed points. Huge matches stress usefulness. Many awaits stress saved-local and drop-state construction. Real crates reveal distributions absent from synthetic tests.

A performance claim requires baseline, target, compiler revision, flags, warmup policy, variance, and representative corpus. An optimization that saves runtime but doubles compile memory is a trade, not a free win.

Security and soundness review#

Treat source as hostile. Bound recursion, state explosion, inlining growth, fixed-point iterations, dump size, and interpreter resources. Avoid integer overflow in compiler-side cost arithmetic. Ensure a resource limit causes a conservative diagnostic or skipped optimization, never fabricated proof. Fuzz validators as well as passes. Run passes on malformed IR only in dedicated robustness harnesses; ordinary passes may rely on validated preconditions.

65. Source navigation and contribution workshops#

Pin the exact repository commit corresponding to the compiler under investigation. Record rustc -vV; “nightly” is insufficient. Current online nightly API documentation may not match 1.97.1.

Start from concepts, then confirm names with source search:

rg "trait.*Analysis" compiler/rustc_mir_dataflow
rg "Coroutine|coroutine" compiler/rustc_mir_transform compiler/rustc_middle
rg "ConstProp|constant propagation" compiler/rustc_mir_transform
rg "SimplifyCfg|Inline" compiler/rustc_mir_transform
rg "unsafety" compiler/rustc_hir_analysis compiler/rustc_mir_build
rg "usefulness|exhaust" compiler/rustc_pattern_analysis

Paths are orientation for the 1.97.1-era tree, not stable module guarantees. Follow the query provider to learn which MIR product is consumed. Then follow pass registration, implementation, validator, and tests. Do not infer ordering from filenames.

Useful source neighborhoods include:

• compiler/rustc_pattern_analysis for usefulness machinery;
• compiler/rustc_mir_build for checked typed-body to built MIR concerns;
• compiler/rustc_mir_dataflow for framework and analyses;
• compiler/rustc_mir_transform for lowering and optimization passes;
• compiler/rustc_middle/src/mir for MIR representation;
• compiler/rustc_const_eval for interpretation and const evaluation;
• tests/ui, MIR optimization tests, and codegen tests for contracts in practice.

Verify these names against the pinned checkout because internal crates move.

Workshop A: edge effect bug#

1. Build a call with normal and unwind successors.
2. Track destination initialization.
3. Predict each edge state before running the compiler.
4. Add an intentionally wrong universal terminator effect in the mini model.
5. Observe cleanup claiming an uninitialized destination is initialized.
6. Fix it with edge-specific transfer.
7. Add the smallest regression graph.

Contribution target: framework tests or one analysis's terminator effect. Reviewer concern: other analyses sharing the cursor must not inherit a policy accidentally.

Workshop B: saved-local regression#

1. Write an async function with a large value used only before await.
2. Measure future size on the pinned toolchain.
3. Dump relevant MIR products.
4. Locate the liveness result and transformed layout.
5. Add a use after await and confirm the field becomes necessary.
6. Reduce any unexpected retention to one local and one suspension.

Contribution target: coroutine liveness/layout or regression test. Reviewer concern: drops and auto traits can make “unused” semantically relevant.

Workshop C: constant propagation#

1. Add one scalar operation to the mini lattice.
2. Define overflow and target-width semantics first.
3. Implement transfer without rewriting.
4. inspect point facts on a diamond and loop.
5. Add rewrite only after facts stabilize.
6. Differentially test all i8 inputs using an i8 interpreter.
7. Add panic-equivalence cases.

Contribution target: a narrowly justified fold. Reviewer concern: CTFE success does not automatically authorize runtime replacement.

Workshop D: inliner remapping#

1. Model a one-block callee.
2. Allocate fresh caller locals.
3. remap arguments, return place, blocks, and normal exit.
4. add an unwind edge before supporting multi-block bodies.
5. compare traces before and after.
6. add recursion and budget guards.

Contribution target: diagnostics, cost model, or a remapping bug. Reviewer concern: source scopes, coverage, cleanup, and debug info are semantic tooling outputs.

Workshop E: unsafe diagnostic#

1. Find one operation introduced after adjustment.
2. identify its unsafe classification and enclosing scope.
3. test macro and non-macro spans.
4. compare unsafe fn with an explicit unsafe block under the lint.
5. ensure borrow checking still rejects an independent ownership violation.

Contribution target: classification or diagnostic quality. Reviewer concern: never imply that acknowledgment proves the unsafe contract.

Capstone#

Extend the mini IR with calls, explicit normal/unwind edges, and observable panic. Implement edge-sensitive maybe-initialized analysis. Implement inlining for one-block callees. Run validation after each pass. Differentially compare return, prints, panic, and drop log. Document unsupported aliasing and reject it in validation rather than assuming it away.

The capstone is complete only when a seeded wrong edge effect is caught by a test and localized to the first pass.

66. Derived philosophy, mastery, talks, and sources#

The mechanisms justify several reusable conclusions.

Representations make selected questions cheap. Typed patterns make constructor coverage cheap. CFG MIR makes predecessor joins cheap. Coroutine state fields make resumption cheap. Each choice forgets source structure, so provenance must be carried deliberately.

A state machine is a proof artifact. Its tag is not merely dispatch optimization. It states which fields are initialized, which resume path is legal, and what cancellation must destroy. A wrong discriminant transition violates all three obligations at once.

Analyses represent uncertainty rather than guessing it away. Unknown, uninitialized, and unreachable have different meanings. A join is a semantic decision about executions, not a bitset convenience.

Optimization preserves observations under an explicit model. Return values alone are insufficient. Panic, unwind, drops, volatile effects, provenance, target arithmetic, and diagnostics can constrain a rewrite. The model may intentionally exclude some tooling observations, but that exclusion must be policy, not accident.

Abstractions move work. Inlining removes call overhead but creates remapping and code-size work. Saving fewer coroutine locals reduces layout but demands precise liveness. Small passes improve attribution but increase traversals. No layer deletes complexity globally.

Pinning moves responsibility to an API boundary. It allows state containing address-sensitive relationships while requiring projection and unsafe code to preserve immobility guarantees. It is not a general certificate of memory safety.

CTFE and optimization share mechanism without sharing policy. An interpreter can help both, but required compile-time evaluation may diagnose where an optimizer must abstain. Conflating them turns missed optimization into language rejection.

The earliest broken invariant is the best debugging coordinate. A cancellation double drop may first appear in transformed cleanup, but begin in liveness or state timing. An optimized crash may begin in a wrong constant fact. Compare query products and pass outputs until the first divergence.

Mastery checklist#

A reader is ready for contributor work when they can:

• distinguish unsafe acknowledgment from borrow and validity proofs;
• derive a usefulness witness and explain guard treatment;
• lower one await into suspension, resume, and drop paths;
• derive saved locals from liveness rather than intuition;
• explain why pinning is needed for self-reference;
• state forward and backward equations and edge effects;
• implement and test a finite worklist analysis;
• separate CTFE failure from optimization abstention;
• review dead-store removal for panic and alias observations;
• describe every remapping obligation of inlining;
• state phase preconditions and validator limits;
• construct a differential oracle with defined-input restrictions;
• measure compile time, memory, code size, and runtime independently;
• navigate from query provider through pass registration to tests;
• reduce a failure to the first changed MIR body.

Prediction exercises#

1. A local owns a guard object used only before await. Must it be saved? Predict destruction timing, then inspect MIR.
2. A call writes its destination only on success. What fact reaches cleanup? Draw edges before answering.
3. A dead assignment calls an indexing operation. Can it be deleted? Account for bounds panic.
4. A constant branch removes the only normal predecessor of a block, but an unwind predecessor remains. Is it dead?
5. An inlined callee has a cleanup return. Where must it reconnect?
6. CTFE exhausts fuel. Does that prove runtime nontermination?
7. A future is moved before first poll and pinned afterward. Which guarantee is violated?
8. A nested or-pattern causes exponential work. Which conservative outcomes preserve soundness?

Talk plans#

A focused 30-minute talk:

1. execute one async source example to first Pending;
2. draw saved fields and discriminant;
3. cancel it and trace drops;
4. derive one backward liveness equation;
5. show a wrong dead-store rewrite and its missing observation;
6. finish with earliest-invariant bisection.

A 60-minute contributor talk:

1. place pattern and unsafe checks at the typed boundary;
2. trace async syntax through coroutine transformation;
3. explain pinning and borrow-check interaction;
4. derive forward and backward dataflow with edge effects;
5. run the mini pass manager;
6. separate CTFE from const propagation;
7. review CFG simplification, inlining, and dead stores;
8. validate phases and preserve diagnostics;
9. design differential and adversarial tests;
10. navigate the pinned rustc source tree.

Authoritative reading map#

• The Rust Reference: unsafe blocks
• The Rust Reference: patterns
• The Rust Reference: async blocks
• The Rust Reference: constant evaluation
• The Rust standard library: Future
• The Rust standard library: Pin
• rustc-dev-guide: MIR
• rustc-dev-guide: MIR construction
• rustc-dev-guide: MIR dataflow
• rustc-dev-guide: MIR optimizations
• rustc-dev-guide: constant evaluation
• rustc-dev-guide: pattern checking
• nightly rustc API: rustc_middle::mir
• nightly rustc API: rustc_mir_dataflow
• nightly rustc API: rustc_mir_transform
• Rust compiler source

The Reference and standard-library contracts are authoritative for specified language and API behavior. The dev guide explains architecture but can lag source. Nightly internal API pages describe their generated nightly, not necessarily 1.97.1. For claims about pass names, order, coroutine layout, thresholds, or query ownership, use the source commit matching rustc 1.97.1 and record it with the experiment.

The operational rule is concise: preserve checked meaning into MIR, preserve live and initialized state across suspension, represent uncertainty in analyses, optimize only under a stated observation model, and validate at every phase boundary where the legal language changes.

67. Generator and coroutine lowering, state by state#

“Generator” appears in older discussions and internal history; current rustc code increasingly uses “coroutine.” Rust's stable user-facing async constructs are built on coroutine-like compiler machinery. Do not infer that every internal coroutine facility is a stable source-language feature.

A coroutine has four logically distinct interfaces:

Async adapts this mechanism to Future::poll. An await polls a child future. Pending corresponds to suspension and returning Poll::Pending. Ready(v) supplies the await result and continues the body. The task context and wake protocol are part of Future, not generic facts about all coroutine forms.

Deriving a two-suspension layout#

Start with conceptual source:

async fn pair(a: String) -> usize {
	let n = a.len();
	first().await;
	let b = vec![0_u8; n];
	second().await;
	b.len()
}

async fn first() {}
async fn second() {}

Ask about uses after each suspension.

local       after await 1?   after await 2?   save requirement
a           no               no               may die before suspension 1
n           yes              no*              save in state 1
b           not created      yes              save in state 2
child1      being polled      no               save in state 1
child2      not created       being polled     save in state 2

The asterisk matters. Although n is not textually used after the second await, its value determined b before suspension. The state machine need not retain causal history after its effect has been materialized.

A conceptual variant layout is:

Start { a }
WaitingFirst { n, child1 }
WaitingSecond { b, child2 }
Done
Poisoned [if an implementation uses such a state]

Actual rustc layout may overlap fields from mutually exclusive states and includes implementation metadata. Source-level field order and stable discriminant values are not promised.

Concrete transitions:

Construct --first poll--> Start body
                           | child1 Pending; save n + child1
                           v
                     WaitingFirst
                           | child1 Ready; create b; poll child2
                           | child2 Pending; save b + child2
                           v
                     WaitingSecond
                           | child2 Ready; compute b.len()
                           v
                         Done

Every downward arrow carries both a control state and an initialization set. The diagram omits unwind edges.

Adding unwind paths#

Allocation for b can panic. Polling user code can panic. Destructors can panic. Therefore a complete lowering review annotates every potentially unwinding operation.

WaitingFirst resume:
  extract child1 result
  drop child1
  allocate b --------------------unwind--> cleanup state-1 resources
  construct child2 --------------unwind--> drop b, then remaining resources
  publish WaitingSecond state
  poll child2 --------------------unwind--> cleanup state-2 resources

“Publish” means make the discriminant and initialization metadata agree with fields. There may not be one literal instruction with that name.

An update protocol can use an intermediate state, drop flags, or carefully ordered writes. Whichever mechanism is chosen must establish this invariant at every unwind point:

Cleanup dispatch describes every resource initialized so far, and no resource not initialized so far.

If state publication moves earlier, cleanup may read absent fields. If it moves later, cleanup may leak present fields. An intermediate representation can temporarily violate the final invariant only when no unwind, yield, or observing operation can occur in that window and the pass contract says so.

Saved-local selection is more than lexical liveness#

Storage liveness asks whether storage is live. Value liveness asks whether the current value may be read later. Drop liveness asks whether destruction may observe a value even when no ordinary read occurs. Borrow liveness asks whether a loan remains relevant. Coroutine layout must consume the appropriate facts rather than treating one bitset as all four.

Examples:

• a Copy integer overwritten immediately after resume may not need its old value;
• a guard with a destructor can matter even without a later read;
• a reference can make the referent's address relevant;
• a local mentioned only in debug information is not necessarily semantically saved;
• a child future is needed while its poll is pending even if source has no explicit later name.

Counterfactual: determine fields from textual names used after .await. This misses temporaries, destructors, desugared child futures, and control-dependent initialization. It can also retain values whose effects were already materialized.

Discriminants and invalid resumes#

The discriminant is internal state, not an enum API visible to ordinary Rust code. The resume shim switches on it. Start, suspended states, completed, and possibly poisoned/unresumable cases need defined internal handling.

Polling a future after it returned Ready must not cause undefined behavior through the safe Future API. The Future trait documentation says such a call may panic, block forever, or otherwise misbehave, but must not cause undefined behavior. An implementation's exact post-completion action is not a stable rustc guarantee.

The resume argument also matters for general coroutine models. Async polling receives a task context through the future interface. Do not bake “resume argument is always unit” into a generic transform abstraction merely because one source form uses a restricted path.

Pin projection proof obligations#

Transform-generated field access can be equivalent to unsafe pin projection. The compiler must know which fields may be moved and which are structurally pinned. Safe user code cannot use Pin::get_mut for a !Unpin future. Compiler-generated code can access fields because the lowering owns and proves the representation invariants.

The proof obligations are:

1. once pinned, an address-sensitive field is not moved before drop;
2. replacing or taking a field is allowed only when its pinning/destruction contract permits it;
3. drop executes in place for pinned state;
4. references into state never outlive that state;
5. layout overlap never makes two simultaneously live fields share storage.

Unsafe code implementing a custom projection has analogous obligations but cannot rely on rustc's private layout. Use established projection APIs or carefully reviewed unsafe code rather than guessing generated-future fields.

Auto traits and captured state#

Whether a future is Send can depend on values held across suspension. A non-Send value created and destroyed entirely before await need not make the future non-Send for that reason. Retaining it as a saved field can alter trait results or size. This makes liveness precision visible to users even though field layout is private.

A regression workflow is:

1. state which value crosses which suspension;
2. assert Send or non-Send with a helper bound;
3. inspect compiler diagnostics identifying the capture;
4. reduce scopes to test whether earlier destruction changes the result;
5. inspect pinned-version MIR only after the source-level expectation is clear.

Coroutine transform review table#

Implementation milestone#

Extend the chapter 60 model without pretending to model pinning:

1. introduce WaitingFirst and WaitingSecond;
2. represent each state's legal fields with separate structs instead of many options;
3. wrap them in an enum so illegal combinations are unrepresentable;
4. implement resume(self_state) as a state replacement;
5. explain why moving enum payloads is unsuitable for a self-referential pinned model;
6. add a cancellation log for every variant;
7. generate delays from zero through ten for both children;
8. assert one completion and exact drop multiplicity;
9. inject a modeled panic during transition;
10. state the missing proof needed for real Pin<&mut Self>.

This milestone teaches a key counterfactual. An enum gives an excellent safe educational representation because it rules out mixed states. A production self-referential future cannot casually move enum payloads during variant replacement. Local representational elegance can transfer difficulty to pin-preserving mutation.

68. Pass-by-pass traces and review drills#

This chapter turns subsystem knowledge into repeatable review practice. Use one trace sheet per body and do not skip unchanged passes: “unchanged” is evidence about scheduling.

Trace one: branch simplification pipeline#

Input MIR-like body:

bb0:
  _1 = const 2
  _2 = Mul(copy _1, const 3)
  _3 = Eq(copy _2, const 6)
  switchInt(move _3) -> [1: bb1, otherwise: bb2]
bb1:
  _0 = const 10
  goto bb3
bb2:
  _0 = const 20
  goto bb3
bb3:
  return

After scalar constant propagation:

bb0:
  _1 = const 2
  _2 = const 6
  _3 = const true
  switchInt(const true) -> [1: bb1, otherwise: bb2]

Other blocks remain for now. The pass proves values but need not own CFG cleanup.

After switch simplification:

bb0:
  _1 = const 2
  _2 = const 6
  _3 = const true
  goto bb1

After unreachable-block removal:

bb0 -> bb1 -> bb3
bb2 removed

After dead-store elimination:

bb0:
  goto bb1
bb1:
  _0 = const 10
  goto bb3
bb3:
  return

Possible block merging then produces:

bb0:
  _0 = const 10
  return

At each stage ask whether _0 has special return-place rules, whether any removed operation can panic, and whether source coverage needs retained regions. The arithmetic here is pure only under an explicitly matching overflow mode.

Trace two: dead store with a panic#

_1 = Index(a, i)
_1 = const 0
return

The first value is overwritten. Its evaluation performs a bounds check. Deleting the entire first assignment removes a possible panic. A sound transform could, depending on MIR forms and phase, preserve the check while dropping only the unused result. It cannot infer that “dead destination” means “dead statement.”

Review drill:

1. identify destination liveness;
2. classify rvalue effects;
3. include unwind edges;
4. decide whether only the store, only the value, or neither can disappear;
5. write one in-bounds and one out-of-bounds differential test.

Trace three: inlining with unwind#

Caller:

bb0:
  _4 = Call parse(copy _1) -> bb1, unwind bb_cleanup
bb1:
  use(_4)
  return
bb_cleanup [cleanup]:
  drop(_2)
  resume

Callee:

cb0:
  assert(valid(_1)) -> cb1, unwind cb_cleanup
cb1:
  _0 = compute(_1)
  return
cb_cleanup [cleanup]:
  drop(_temp)
  resume

Conceptual remapping:

caller arg _1 -> fresh callee arg _10
callee return _0 -> fresh _11, then caller destination _4
cb0/cb1/cb_cleanup -> fresh caller blocks
callee normal return -> assign _4, goto bb1
callee unwind resume -> continue to caller bb_cleanup after local cleanup

Simply redirecting every callee unwind edge to caller cleanup skips callee-owned temporaries. Simply retaining a resume can bypass caller cleanup. Inlining must compose cleanup, not choose one side.

Inlining review drill:

• Are all locals fresh?
• Are promoted/constants instantiated in the caller context correctly?
• Are source scopes and debug records remapped?
• Do normal returns assign the call destination once?
• Do divergent paths avoid assigning it?
• Do unwind paths run callee then caller cleanup?
• Are recursion and growth bounded?
• Are required dialect lowerings already complete?

Trace four: analysis loop convergence#

bb0: x = 0; goto bb1
bb1: use x; switch c -> bb2, bb3
bb2: x = x + 1; goto bb1
bb3: return x

For reachable constants, the first visit may infer x=0 at bb1. The backedge introduces x=1, then more values. A finite constant lattice must join disagreement to Unknown rather than enumerate integers forever.

iteration 1 IN[bb1] = C(0)
iteration 2 IN[bb1] = join(C(0), C(1)) = Unknown
iteration 3 IN[bb1] = join(C(0), Unknown) = Unknown [stable]

Widening is unnecessary for this finite scalar lattice. An interval domain over unbounded mathematical integers may need widening to guarantee convergence. Choosing a richer domain changes both proof power and compile-time risk.

Pass-order counterfactuals#

Run dead-store elimination before constant propagation. It may remove obvious overwritten locals but cannot expose a constant branch. Run constant propagation first. It may expose unreachable blocks that improve liveness and then dead-store results.

Inline before constant propagation. Caller constants become visible inside the copied callee, but code growth increases analysis cost. Propagate before inline. The call boundary may hide facts, but cheaply proves some sites unprofitable or unreachable.

Simplify CFG aggressively before diagnostics. Analyses get fewer blocks, but source correspondence may degrade. Delay simplification. Diagnostics retain structure, but all consumers pay for noise.

No order dominates universally. State required dependencies, then measure optimization opportunity and compiler resources.

Pass-manager hardening checklist#

Configuration:

• deterministic pass order;
• target and optimization level included in decisions;
• feature gates and unstable flags recorded;
• no accidental dependence on hash iteration;
• pass disabling has a documented testing mode.

Contracts:

• input dialect checked;
• output dialect declared;
• required predecessor analyses available;
• invalidated analysis caches discarded;
• stolen or consumed bodies not reused;
• body ownership visible in query boundaries.

Observability:

• before/after dumps use stable enough labels for debugging;
• timing overhead can be disabled;
• pass names are unique in logs or include occurrence numbers;
• no-change is distinguished from skipped;
• validator failure names the producer pass;
• source snippets are bounded and sanitized.

Correctness:

• mandatory lowerings cannot be disabled by optimization policy;
• optional failure falls back safely;
• resource exhaustion does not fabricate facts;
• panic strategy and target layout are inputs;
• cleanup edges count as reachability;
• repeated pipelines terminate.

Performance:

• analysis results are cached only with complete keys;
• invalidation is narrower only when proved correct;
• bitsets use dense indices;
• sparse forms are justified by measured density;
• inlining budgets are cumulative;
• validation frequency is appropriate to build mode while tests remain strict.

Diagnostic workshop#

Create a pass that folds division by a known nonzero denominator. Preserve the numerator's source span for the resulting computation only if that span honestly describes it. Keep the division operation's span for diagnostics that still concern division. If the operation disappears, ensure coverage mapping does not claim an impossible execution region.

Then seed three bugs:

1. fold when denominator is zero;
2. use host width for target-width minimum divided by minus one;
3. attach a synthetic span from another macro expansion.

The first is semantic. The second is portability and potentially semantic. The third is diagnostic provenance. Structural validation may catch none. This demonstrates why “validator passes” is not the end of optimization review.

Final contribution protocol#

Before opening a compiler change:

1. pin revision and reproduce with the smallest source;
2. identify the first wrong query product;
3. identify the producing pass or analysis;
4. write the invariant in one sentence;
5. add a regression that fails at that boundary;
6. make the smallest mechanism change;
7. validate all supported phases;
8. run focused UI/MIR tests;
9. run broader compiler tests required by the touched area;
10. benchmark compile time and code quality when policy changes;
11. document version-sensitive behavior without promoting it to a language promise;
12. ask reviewers specifically about unwind, drop, alias, target, and diagnostics.

A high-quality optimization contribution is often small code surrounded by strong evidence. The proof is distributed across representation invariants, analysis equations, transform reasoning, tests, and measurements.

Part VI: Ownership, NLL, Borrow Checking, and Polonius#

1. Why memory needs rules#

Imagine a program as a workshop full of numbered boxes. Values occupy boxes, and names tell us which box to use. A pointer is a note containing another box's address. That note is useful only while the target box still exists. Reading an expired address is a use-after-free. Writing through an unexpected alias can invalidate another reader's assumptions. Freeing one allocation twice can corrupt the allocator itself. These are not merely untidy outcomes: they can expose secrets, miscompile code, or crash far from their cause.

An alias is a second route to the same storage. Aliasing is often convenient. Mutation is often necessary. The dangerous combination is unrestricted aliasing plus unrestricted mutation. If one route changes a vector while another route points into its old buffer, the second route may dangle. If two threads mutate the same integer without synchronization, a data race occurs. Rust therefore makes a practical bargain: shared access permits observation, while exclusive access permits mutation. The slogan is “shared XOR mutable,” but the real rules account for nested places, reborrows, interior mutability, raw pointers, and control flow.

Ownership answers who is responsible for eventually destroying a value. Normally one local place owns a value. Moving transfers that responsibility and makes the old place unavailable. Borrowing temporarily grants access without transferring destruction responsibility. These rules let ordinary safe Rust avoid a tracing garbage collector while preventing broad classes of memory errors.

The compiler proves a deliberately limited proposition about safe code. It checks types, moves, lifetimes, and access conflicts under Rust's language model. It does not prove that every algorithm is correct. It does not prove that all unsafe code is sound. An unsafe block is a boundary where humans promise additional contracts that static checking cannot establish. The surrounding safe API must uphold those contracts for every safe caller. Miri, sanitizers, fuzzing, review, and formal verification complement borrow checking; none is replaced by it.

This part develops the user model first and the compiler model second. The recurring picture will be a subway map. MIR control-flow points are stations, edges are possible journeys, regions mark stations where a reference must be valid, and loans are permissions traveling through that map. This is a compile-time model. A lifetime is not a stopwatch attached to a runtime object.

2. Ownership as destruction responsibility#

Every value has a type and resides somewhere. For an owned String, a small header resides in its local variable while bytes reside in a heap allocation. Dropping the header runs String's destructor, which frees those bytes. Only one ordinary owner should perform that destruction.

fn consume(s: String) {
    println!("{s}");
}

fn main() {
    let first = String::from("map");
    let second = first;
    consume(second);
    // println!("{first}"); // error: first was moved
}

The assignment transfers the String value. Rust calls that transfer a move. It is usually only bookkeeping: the compiler may copy machine words, but semantically the old place is uninitialized. “Move” therefore does not promise a physical relocation.

Some types implement Copy. Reading a Copy place duplicates its value and leaves the source initialized. Integers and shared references are common examples. Types with destructors cannot be Copy, because silently duplicating destruction responsibility would be ambiguous.

let x: u32 = 7;
let y = x;
assert_eq!(x + y, 14);

Ownership is compositional. A tuple owns its fields; a struct owns its fields; a vector owns its elements and allocation. Destruction recursively drops owned components, normally in a specified structural order. Library types may encode different ownership arrangements: Rc<T> counts shared owners, Arc<T> counts them atomically, and arenas may destroy many values together. The borrow checker does not insist on one physical owner in every implementation. It checks the contracts expressed by types.

An invariant is a property that must always hold at an interface boundary. For Vec<T>, length must not exceed capacity and initialized elements must occupy the prefix described by length. Safe methods preserve those facts. Unsafe implementation code may manipulate raw parts, but it must restore all required invariants before safe code can observe them.

Ownership gives deterministic destruction. That improves resource management for files, locks, and sockets as well as memory. Its cost is that programmers must make sharing and transfer explicit. Rust chooses that cost to make resource behavior local and optimizable.

3. Places, values, and projections#

Borrow checking speaks about places rather than merely variable names. A place denotes storage that can be read from or written to. The local x is a place. So are x.field, x.0, array[i], and *pointer. A value is the data obtained by evaluating an expression.

In MIR, a place begins with a local and may have projections. A projection selects a field, dereferences a pointer, indexes an aggregate, or downcasts an enum variant. Think of a postal address followed by directions inside the building. This vocabulary allows precise questions: does writing pair.left conflict with a loan of pair.right? For known disjoint fields, usually not. Does indexing slice[i] conflict with slice[j]? The checker generally cannot prove arbitrary indices differ, so it conservatively treats them as potentially overlapping.

struct Pair { left: String, right: String }

let mut p = Pair {
    left: "L".into(),
    right: "R".into(),
};
let l = &mut p.left;
p.right.push('!'); // disjoint field
l.push('?');

An access is an operation on a place. Reads inspect a value. Writes replace or mutate it. Moves read while also deinitializing the source. Borrows create a reference and an associated loan. Drops run destruction and therefore require suitable access to the dropped place.

Place overlap is central to conflict checking. Equal places overlap. A parent generally overlaps each child: assigning all of p affects p.left. Different struct fields can be disjoint. Dereferences and indices require type-specific conservative reasoning. Interior-mutability types such as Cell<T> deliberately move some mutation checking from compile time into their safe API and runtime implementation.

The distinction between expression and place also explains assignment. The left side identifies storage. The right side computes a value. Evaluation order, temporary creation, and implicit borrows are made explicit during lowering so later analyses do not need to understand every surface syntax form.

4. Moves, copies, and move paths#

The compiler's move analysis tracks initialization at useful granularities. A move path is a node in a tree describing a place and move-relevant descendants. For a struct local, the root may have one child per field. Moving one field deinitializes that child without necessarily deinitializing its siblings.

struct Record { name: String, count: u32 }

let r = Record { name: "Ada".into(), count: 3 };
let name = r.name;
println!("{}", r.count); // remaining Copy field is usable
// drop(r);              // whole value is partially moved
drop(name);

This is a partial move. The compiler must reject accesses that require the complete parent while allowing initialized siblings. It computes dataflow facts such as “maybe initialized” and “maybe uninitialized” at each CFG point. At a merge after an if, a place may be initialized on one incoming path and uninitialized on another. A read is legal only when the required place is definitely initialized.

let mut s = String::from("start");
if condition() {
    drop(s);
} else {
    s.push('!');
}
// println!("{s}"); // not initialized on every path

Copy classification is type directed. The MIR operation may be represented as a copy or move based on the inferred type and context. Changing a generic bound from T to T: Copy can therefore change legal use after assignment.

Move paths are not a byte-level alias analysis. They approximate semantic ownership units the language exposes. That keeps analysis finite and diagnostics understandable. The tradeoff is conservative rejection when disjointness depends on values, such as two runtime indices. Safe abstractions like split_at_mut use checked arithmetic plus an unsafe implementation to expose disjoint slices in a form the type system can understand.

Move errors and borrow errors interact but are distinct. A use may fail because its source was deinitialized even if no reference exists. Conversely, an initialized place may be inaccessible because an active loan reserves it. Good triage asks which analysis established the prohibition before changing lifetime annotations.

5. Borrowing creates loans#

Evaluating &place or &mut place creates a reference value. The borrow checker also models a loan: permission tied to the borrowed place and borrow kind. The reference may be copied, moved, stored, or reborrowed; the loan constrains conflicting accesses while it can still matter.

A shared borrow permits shared reads through aliases but normally forbids mutation of overlapping data. A mutable borrow permits mutation through its exclusive route and normally forbids other overlapping accesses. “Mutable reference” is shorthand; exclusivity is the deeper property. The reference need not actually mutate.

let mut n = 10;
let shared = &n;
println!("{shared}");
n += 1; // allowed after shared is no longer needed

let unique = &mut n;
*unique += 1;

Borrow kind is not the only input. The checker considers the access kind attempted, place overlap, loan phase, and location. A read can conflict with an active mutable loan. A write or move can conflict with either a shared or mutable loan. Creating another loan is itself an access with compatibility rules.

The term origin is often used in Polonius discussions for the abstract provenance slot associated with references. Historically, lifetime and region terminology overlaps. We will use region for a set of CFG points in classic NLL inference and origin for a Polonius set of loans. Neither is a runtime allocation.

Borrow checking is conservative. It must approve only programs justified by its model. Rejecting a safe program is an expressiveness limitation; accepting a program that permits undefined behavior is a soundness bug. Compiler engineering prioritizes avoiding the second while steadily reducing the first.

6. Access kinds and conflict geometry#

Visualize every loan as a colored translucent sheet covering a place subtree. A shared sheet permits read rays but blocks write, move, and exclusive-borrow rays. An active mutable sheet blocks all competing rays. Two sheets conflict when both their time ranges and place footprints overlap incompatibly.

Important access categories include read, write, move, shared borrow, mutable borrow, and drop. Compiler internals refine these categories for shallow versus deep access and special cases. A shallow access may touch a container without recursively accessing data behind pointers. That distinction matters for assignments, discriminants, and destructor behavior.

let mut pair = (String::from("a"), String::from("b"));
let first = &pair.0;
pair.1.push('!');       // disjoint
// pair = ("x".into(), "y".into()); // overlaps pair.0
println!("{first}");

Conflict is symmetric as an incompatibility, but diagnostics have direction. One operation creates a loan; a later operation invalidates it; a still-later use may explain why the loan remained relevant. Reporting all three locations produces a narrative rather than a bare prohibition.

Unsafe code can use raw pointers to perform operations not statically tracked as ordinary reference accesses. That does not suspend Rust's semantic rules. Creating a raw pointer may be allowed, but dereferencing it is unsafe and must respect validity, alignment, initialization, aliasing, and provenance requirements. The exact unsafe aliasing model continues to be specified and researched. Do not infer that “borrow checker accepted the unsafe block” proves its pointer manipulation valid.

Compiler conflict logic is necessarily aligned with language semantics and diagnostics. Changing overlap rules may accept more code, but it can also invalidate optimization assumptions or unsafe-library contracts. Such changes need tests and language-level review, not merely an appealing example.

7. Reborrowing and nested permissions#

A reborrow creates a new reference through an existing reference. It does not take ownership of the referent. Instead, it temporarily narrows or shares the permission already available.

fn inspect(x: &i32) { println!("{x}"); }

let mut value = 4;
let outer = &mut value;
inspect(&*outer);       // shared reborrow
*outer += 1;            // outer usable again

For the call, &*outer lends shared access derived from the exclusive reference. While that shared reborrow matters, mutation through outer would conflict. After its last control-flow-relevant use, the parent permission can resume. This nesting resembles lending a key while retaining ownership of the key ring: the parent cannot exercise incompatible authority during the subloan.

Passing &mut T to a function often performs an implicit reborrow rather than moving the reference permanently. That is why a mutable reference can be passed repeatedly in sequence. Assignments and generic contexts can alter whether a move or reborrow is inserted, so surface similarity is not always semantic identity. Inspect MIR when investigating subtle compiler behavior.

Reborrows establish outlives relationships. The source reference must remain valid for every point where the derived reference can be used. In region language, the source region must include the derived region. In loan language, permissions flow into the derived origin under constraints.

Variance affects whether one reference type can be shortened or related to another. Shared references are covariant in their referent lifetime and suitable type parameters. Mutable references are covariant in their own lifetime but invariant in the pointee type parameter, because they can both read and write that type. This boundary prevents storing a short-lived reference through a slot promised to hold a longer-lived one. Variance is type-level permission geometry, not a runtime conversion.

8. From lexical AST checking to MIR NLL#

Early Rust borrow checking was closely tied to lexical scopes and AST-shaped program structure. A lexical scope is a source-code block delimited by syntax. That was simple to explain but often too coarse. A borrow assigned to a local could be treated as lasting until the closing brace even when no later path used it.

let mut names = vec!["Ada".to_string()];
let first = &names[0];
println!("{first}");
names.push("Grace".to_string());

A block-wide approximation rejects the push because vector growth may relocate elements while first points into one. Control-flow reasoning sees that no path uses first after the print, so the loan need not constrain the push.

NLL means non-lexical lifetimes. More precisely, relevant regions are derived from MIR control flow and constraints rather than being forced to equal source lexical blocks. It does not mean all lifetimes disappear. It does not mean a reference is valid forever. It does not mean every borrow ends at the last textual use. Loops, branches, drops, destructors, unwind edges, and values carried through aggregates can require points that are not obvious from textual order.

MIR is a better substrate because complex Rust syntax is desugared into a smaller set of statements and terminators. Method calls, matches, temporaries, and control flow become explicit enough for uniform dataflow. The CFG captures possible execution paths, including branches and loops.

NLL shipped after migration machinery compared old and new checking and managed diagnostics across editions. That history matters: replacing a soundness-critical analysis requires ecosystem testing and diagnostic work, not only a new algorithm. MIR NLL is now the conceptual baseline, but source and version should be consulted before making claims about exact internal organization.

9. CFG points and MIR locations#

A control-flow graph, or CFG, has basic blocks as nodes and possible transfers as edges. A basic block is a straight-line sequence with one terminator such as goto, switch, call, or return. Borrow analysis often needs points before and after statements, finer than whole blocks. A MIR location commonly identifies a block and statement index.

Consider this schematic program.

let mut x = 0;          // P0
let r = &x;             // P1: loan L issued
if choose() {           // P2
    println!("{r}");   // P3
}                       // P4
x = 1;                  // P5

The CFG has edges P0→P1→P2, P2→P3→P4, P2→P4, and P4→P5. The reference must be valid on the path reaching P3. There is no need to keep its loan active at P5 solely because the variable's lexical scope continues.

Dataflow associates facts with points. A forward analysis propagates facts along execution direction. A backward liveness analysis starts from uses and travels toward definitions. At a branch join, union is common for “may” facts: a fact possible on either predecessor is possible at the join. Intersection is common for “must” facts: a property guaranteed at a join must hold on every predecessor.

Loops require iteration. Facts propagated around a back edge may enlarge the input of an earlier block. The solver repeats transfer and merge operations until no set changes. That stable state is a fixed point. Finite domains and monotone growth guarantee termination for these formulations.

The station-map model must include edge-sensitive subtleties. A call has normal and unwind successors. A false edge or cleanup block can affect analysis even though source code gives it little visual space. When a diagnostic seems impossible, dump MIR and inspect actual successors before assuming source order is the CFG.

10. Entering the MIR borrow checker#

The rustc-dev-guide identifies the mir_borrowck query in the rustc_borrowck crate as the main entry point. The query consumes the MIR body associated with a definition and produces borrow-check results and diagnostics used by later compilation. Exact function boundaries evolve, so treat this as an architectural map rather than a frozen call graph.

First rustc obtains a local copy of MIR suitable for borrow-check preparation. Preparation normalizes or renumbers details needed by analysis and avoids mutating a shared query result. Borrow checking works after high-level type checking and MIR construction, but before code generation.

The broad pipeline is:

1. prepare and locally transform MIR;
2. gather move paths and initialization dataflow;
3. identify universal regions and replace body regions with fresh inference variables;
4. MIR-type-check statements and terminators;
5. collect outlives, liveness, and type-test constraints;
6. solve region variables to CFG-point sets;
7. generate loans and determine where they are in scope;
8. walk MIR accesses and test conflicts, moves, and initialization;
9. build source-oriented diagnostics and recovery results.

This ordering is conceptually useful, not an assertion that every implementation pass is cleanly isolated. Move results feed region computation. Type checking creates constraints. Region values determine loan scope. The final walk combines several analyses. Diagnostics may retain auxiliary facts throughout.

Why re-type-check MIR? The earlier HIR type checker solved surface-language typing and produced adjustments. The MIR checker sees explicit operations and fresh region variables. Its job is narrower: verify MIR typing relationships and emit region constraints for this body. It is not needlessly repeating all source type inference.

The official overview and region-inference chapters are the starting references: https://rustc-dev-guide.rust-lang.org/borrow-check.html and https://rustc-dev-guide.rust-lang.org/borrow-check/region-inference.html.

11. Universal, free, and local regions#

A region is an abstract compile-time set used to express where a reference must be valid. In classic NLL inference, region values contain CFG points and special end elements. Do not picture elapsed runtime seconds.

Universal regions arise from a body's externally visible assumptions. For fn get<'a>(x: &'a str) -> &'a str, 'a is chosen by the caller subject to the signature. Inside the function, the checker must be correct for every allowed caller choice. Such regions have also been called free regions because they appear free with respect to the body. 'static is represented too.

Local region variables describe lifetimes inferred within the body. For let r = &x, the source omitted a lifetime name, but analysis introduces a fresh variable for the relevant reference type. Liveness and type relations determine its minimum valid point set.

Region mapping connects source-level or type-system regions to these internal variables. replace_regions_in_mir finds universal regions and replaces body regions with fresh inference variables. This prevents obsolete lexical approximations embedded in input MIR from dictating NLL results.

fn choose<'a, 'b>(flag: bool, a: &'a str, b: &'b str) -> &str {
    if flag { a } else { b }
}

This signature is incomplete because the output cannot be related unambiguously to inputs. An explicit common output region requires bounds showing both candidates can be shortened to it. The error is about caller-visible guarantees, not merely where braces end.

Universal regions have known relations from declarations, implied bounds, and where clauses. If inference would require an additional universal outlives fact not promised by the signature, checking fails. That protects callers from an implementation silently demanding a stronger lifetime relationship.

12. Placeholders and higher-ranked binders#

A higher-ranked trait bound, or HRTB, says a property holds for every lifetime selected under a binder. For example, for<'a> Fn(&'a str) requires callability with any suitable 'a, not one particular inferred lifetime.

fn apply<F>(f: F)
where
    F: for<'a> Fn(&'a str),
{
    let local = String::from("temporary");
    f(&local);
}

To test such obligations, compiler reasoning may replace bound lifetimes with placeholders. A placeholder is a fresh, rigid representative for an arbitrary choice. It must not accidentally unify with a local region in a way that lets local data escape. A leak check detects forbidden dependence.

Keep the scope precise. Ordinary local NLL variables are flexible sets solved from constraints. Universals represent caller-chosen external regions. Placeholders model rigid choices introduced while reasoning under higher-ranked binders. They interact, but they are not interchangeable names for “a lifetime.”

Higher-ranked outlives constraints sit near the boundary between trait solving, type checking, and borrow checking. The 2023 Inside Rust Polonius update discussed factoring higher-ranked concerns and eventually having the next trait solver solve more of them. That post is a historical roadmap, not a promise of dates or a statement of current completion. Compiler architecture and enablement must be checked against the exact source revision.

When debugging an HRTB lifetime failure, first identify the binder. Then ask which region was universally chosen, which local value allegedly escaped, and which outlives edge would justify the conversion. Adding 'static often masks the design by demanding too much. It is not a universal lifetime-error repair.

13. Outlives constraints as subset constraints#

The notation 'a: 'b is read “'a outlives 'b.” In a point-set model, every point required by 'b must also belong to 'a. Thus points('b) ⊆ points('a). The colon direction and subset direction can feel reversed; saying both aloud prevents mistakes.

Suppose reference long: &'a T is used where &'b T is expected. That coercion is safe if 'a covers everything demanded by 'b. The MIR type checker records the corresponding relation. Assignments, calls, returns, casts, and aggregate construction can generate constraints.

Start with these liveness seeds:

R1 = {P2}
R2 = {P5}
constraint: R1 includes R2

Propagation adds P5 to R1. If R2 later gains P8, R1 gains P8 too. Repeated propagation reaches a fixed point. Strongly connected components can compress mutually including regions, reducing repeated work.

Universal ends need special treatment. The dev guide describes abstract end('a) elements. If solving makes one universal region contain another's end, the declared universal relations must justify it. Otherwise the function body demands an outlives promise absent from its interface.

Outlives does not claim one runtime object is created earlier or dropped later on every execution. It is a validity implication within type checking. Likewise, subset here is an order on inferred facts, not necessarily a source-level subtyping relation in every context.

Type tests such as T: 'a require that components of a type validly outlive a region. They are checked alongside inferred region values. Variance influences how nested types produce these requirements. The solver must preserve all such invariants, not merely stop loans at appealing source lines.

14. Liveness and fixed-point solving#

A reference region must include points where the reference's value may be used. Liveness analysis finds those demands. For a local, “live” roughly means its current value may be read on some path before being overwritten, but compiler details distinguish regular use, drop use, and type liveness.

Consider:

let mut x = 1;
let r = &x;
if coin_flip() {
    println!("{r}");
}
x = 2;

Backward liveness begins at the print. It travels to the branch and then to the definition of r along the path that can reach the use. At the assignment to x, r is not live merely because its variable remains lexically in scope.

A standard block equation is:

live_out[B] = union(live_in[S] for each successor S)
live_in[B]  = use[B] union (live_out[B] minus def[B])

Initialize sets to empty, place blocks on a worklist, recompute, and enqueue predecessors when input changes. Because sets only gain finitely many locals, iteration terminates. Statement-level states are obtained by walking each block backward.

Region solving combines liveness seeds with outlives propagation. One analysis says where a particular value is needed. The other says requirements on one region imply requirements on another. Together they compute least sets satisfying all constraints. Choosing least solutions avoids extending loans without reason while retaining soundness under the model.

“Last use” is a helpful beginner intuition but not the algorithm. A use in a loop may make a reference live across a back edge. A destructor may inspect fields when a value is dropped. A reference stored in another live value can propagate requirements. Unwind paths can add successors. Always translate difficult cases into CFG reachability and constraints.

15. Loan generation and scope#

When MIR performs a borrow, rustc records a loan with its borrowed place, kind, issuance location, and associated region. After region inference, a traditional NLL view considers that loan in scope at points contained in the relevant region, subject to analysis details. Conflict checking asks whether an access invalidates any loan in scope.

let mut data = vec![1, 2];
let element = &data[0]; // shared loan of indexed content
data[0] = 9;            // conflicting write if element remains needed
println!("{element}");

The print seeds liveness for element. The associated region reaches the write. The write overlaps the borrowed element conservatively. The checker reports the borrow, conflicting assignment, and later use.

Issuance and activation are not always identical. Two-phase borrowing, discussed next, can reserve an exclusive loan before activating it. Loan state therefore may include phase as well as location.

Kills remove a loan when its borrowed storage is overwritten in formulations that explicitly track loan flow. Care is required: overwriting while a reference is live is itself often illegal. A kill is useful along paths where no protected use remains, especially in location-sensitive reasoning.

Loan scope is not the runtime lifetime of an object. The object may exist before and after the loan. The region represents where validity and access restrictions are required for this reference relation. Storage liveness, value initialization, destructor execution, and reference validity are related analyses with different domains.

16. Two-phase borrows#

Some expressions need an exclusive borrow and another access during argument evaluation. The classic example is v.push(v.len()). The receiver needs &mut v, while v.len() needs shared access to v. If the mutable loan became fully active before arguments were evaluated, the shared read would conflict.

Two-phase borrows split selected implicit mutable borrows into reservation and activation. During reservation, the future exclusive permission is recorded but certain shared accesses remain allowed. At the call, the loan activates and behaves as an ordinary mutable loan.

let mut v = vec![10, 20];
v.push(v.len());

Conceptually:

P0 reserve mutable loan of v
P1 read v for len()
P2 activate mutable loan
P3 call push

This feature is deliberately limited. It is not a general promise that every explicit &mut waits until first mutation. Compiler eligibility depends on how the borrow was introduced, notably implicit autoref contexts. The reservation itself can conflict with other mutable loans, and activation checks whether incompatible loans survived.

Autoref is the compiler-inserted borrowing used to match a method receiver. Method lookup may select a method requiring &mut self, then insert the borrow. Index assignment can also involve implicit borrowing and evaluation-order subtleties. For a[a.len() - 1] = value, receiver and index evaluation may expose patterns different from an equivalent helper call. Do not generalize from one syntax without examining adjustments and MIR.

Two-phase logic preserves an invariant: once the exclusive operation actually begins, competing access is gone. It improves ergonomics without weakening the active aliasing rule. Diagnostics should distinguish reservation, activation, and conflict when those locations differ.

17. Method calls, indexing, and implicit borrows#

Surface Rust omits many references. Calling text.push('!') asks method lookup to autoderef receiver candidates and autoref the selected one. An Index or IndexMut operation similarly lowers through trait methods and dereferences the result. Borrow behavior follows lowered operations, not punctuation alone.

fn append_len(v: &mut Vec<usize>) {
    v.push(v.len());
}

The mutable receiver can use a two-phase autoref. The nested len shared borrow is evaluated before activation. By contrast, explicitly storing let r = &mut *v; may activate ordinary exclusivity immediately under relevant rules.

Indexing has uncertain disjointness. slice[i] and slice[j] may denote the same element, even if a human knows current values differ. General integer inequality proof is outside borrow checking. Use APIs that establish structural separation.

let (left, right) = slice.split_at_mut(mid);
let a = &mut left[i];
let b = &mut right[j];
use_both(a, b);

split_at_mut validates the boundary and returns separate slice references. Its unsafe implementation carries the proof the compiler cannot derive from arbitrary indexing. This illustrates Rust's philosophy: concentrate hard proofs inside small audited abstractions, then expose safe types that ordinary checking can compose.

Evaluation order matters for reservations. Receiver evaluation, argument evaluation, activation, normal return, and unwind edges all appear in MIR. A bug report should include the minimal source and, when useful, -Zunpretty=mir output from the exact nightly. Nightly flags are unstable and may be renamed or removed, so record rustc -Vv.

18. Partial moves, drop flags, and destruction#

Destruction cannot blindly run on a value that was moved. rustc elaborates drops using initialization information and drop flags. A drop flag is conceptual or materialized state indicating whether a place currently needs destruction. Conditional control flow may set and clear it as values are initialized, moved, and reassigned.

let mut slot: Option<String> = Some("hello".into());
if take_it() {
    let owned = slot.take();
    consume(owned);
}
// slot remains a valid Option and is dropped once

For direct partial moves, a type implementing Drop receives special restrictions. Its destructor may inspect the entire value, so moving out an individual field could leave the destructor observing uninitialized data. Wrappers such as ManuallyDrop, Option::take, and replacement operations express controlled patterns, but unsafe use must preserve invariants.

Drop check, often called dropck, determines which generic data must remain valid when a destructor can run. A destructor can access fields even if ordinary code no longer uses them. Therefore drop liveness can extend requirements beyond the apparent last source read. This is one reason NLL is not simply “end every borrow at last textual use.”

The internal pipeline uses move paths and maybe-initialized analyses to elaborate legal drops. Borrow checking also treats dropping as an access that can conflict with outstanding loans. Panic unwind paths may run cleanup drops, adding CFG edges relevant to liveness.

The invariant is exactly-once destruction for initialized owned values, except where language constructs intentionally suppress or transfer it. Leaks are generally memory-safe in Rust, though they can be resource bugs and can invalidate unsafe APIs that incorrectly assume destructors always run. Unsafe code must not use destructor execution as a universal safety guarantee.

19. Drop check, may_dangle, and variance#

Generic destructors raise a subtle question: must every generic argument remain fully valid until the destructor runs? Conservatively yes, because drop could read it. But some library destructors only deallocate storage and never inspect elements already known to be irrelevant.

The unsafe may_dangle concept allows a destructor implementation to promise that a parameter need not remain valid at drop time in specified ways. It is an expert contract used in standard-library internals, not a casual escape hatch. The implementation must still account for fields whose ownership requires dropping and for all actual accesses made by the destructor.

struct Inspector<'a>(&'a str);

impl Drop for Inspector<'_> {
    fn drop(&mut self) {
        println!("{}", self.0);
    }
}

Here the referent clearly must be valid when Inspector drops because the destructor reads it. Declaration order and explicit drop can therefore affect acceptance.

Variance decides how lifetime and type arguments may be substituted. Outlives constraints state validity inclusion. Dropck adds requirements at destruction boundaries. These mechanisms cooperate but answer different questions. Treating all three as “the lifetime system” hides useful debugging distinctions.

PhantomData<T> can tell the type system that a type logically owns or relates to T despite storing no T bytes. Its shape affects variance, auto traits, and drop checking. Unsafe container authors choose it to encode real invariants, not to silence errors experimentally.

At this boundary, consult the Rustonomicon and current compiler source. The stable surface language intentionally hides implementation complexity. A change that appears to relax dropck can create dangling reads in a destructor or invalidate #[may_dangle] assumptions. Such issues deserve a minimized compiler test and review by types and unsafe-code experts.

20. Closures and capture inference#

A closure is lowered to an anonymous environment type plus a call implementation. The environment stores captured places. Capture inference decides whether each capture is by shared borrow, mutable borrow, or value, and can capture a precise projection rather than an entire variable.

let mut count = 0;
let label = String::from("hits");
let mut bump = || {
    count += 1;
    println!("{label}: {count}");
};
bump();

Mutating count requires a mutable capture. Reading label can use a shared capture unless move or later operations require ownership. The resulting closure's call traits—Fn, FnMut, or FnOnce—follow what calling does to captures.

Borrow checking the closure body needs relationships between captured references and the outer body that constructs the closure. Constraints may be propagated outward or summarized as closure requirements. Universal-region handling has closure-specific details, as the dev guide notes.

A move closure moves captures into the environment according to capture analysis. It does not magically make borrowed data 'static. If a captured value is itself a reference, moving that reference preserves its validity requirements. Thread spawning often requires 'static because the new thread may outlive the caller, not because all move closures inherently have that bound.

Diagnostics can point to closure creation, captured variable use, and escape through a return or spawned task. Minimization should retain the closure's inferred trait and capture mode; replacing it with a function can erase the bug.

21. Async, suspension, and pinning#

An async fn or async block is lowered to a coroutine-like state machine. Locals that remain needed across .await become fields in that state machine. Each suspension point can return control while preserving state for a later poll.

async fn show_after_wait(text: String) {
    let view = &text;
    ready().await;
    println!("{view}");
}

The borrow crosses suspension because view is used after the await. The generated future may contain both text and a reference logically pointing into its state. Moving such a self-referential state after establishing the pointer could invalidate it.

Pinning addresses that boundary. Pin<P> restricts movement of a pointee when the type's guarantees rely on a stable address. Compiler-generated futures are commonly !Unpin when necessary, and polling APIs use pinned receivers. Pinning does not make every reference valid and does not replace borrow checking. Unsafe projection and pin APIs carry contracts about movement and destruction.

Borrow analysis sees MIR for coroutine bodies, suspension points, saved locals, and resumes. Liveness across a yield can enlarge regions dramatically compared with straight-line intuition. Send errors on spawned futures often reveal that a non-Send value remains live across await. Moving a statement before await, narrowing a scope, or dropping a guard can change saved state, but explicit drop behavior and compiler precision should be verified on the target version.

Async diagnostics span multiple transformations. A primary type error may mention a future, while the causal borrow begins in source before await. Good reports include the executor bound, await point, captured value, and full diagnostic. Do not reduce away the suspension if it is the relevant CFG edge.

22. Building a useful diagnostic#

A borrow error is a proof failure translated back into source language. The analysis works on MIR locations and inferred facts; users need names, spans, and an actionable causal story. That translation is a separate engineering problem.

Three spans often matter. The origin span shows where a borrow or move began. The conflict span shows the incompatible access. The use span shows why the earlier permission remained live.

let mut s = String::from("x");
let r = &s;       // origin
s.push('!');      // conflict
println!("{r}"); // later use explaining liveness

An explanation path connects these points through the CFG and constraints. For return errors it may explain that a value is required to live for an output lifetime. For closures it may point to capture and escape. For loops it may explain a borrow from a previous iteration.

Correct rejection can still produce a bad diagnostic. Desugaring may attach broad spans. Several equivalent paths may exist, and choosing the least confusing one is nontrivial. An earlier type error may inject error types and distort later facts. Macro expansion can obscure source ownership. Diagnostics therefore have dedicated tests independent of semantic acceptance.

Error recovery lets compilation continue after a failure to report additional useful errors. It may suppress cascades, taint inference, or use placeholders. A strange secondary borrow error can vanish once the first type error is fixed. Triage should reproduce with the full output and identify the earliest relevant diagnostic.

23. Reading common errors as failed invariants#

“Cannot borrow as mutable because it is also borrowed as immutable” means a shared loan overlaps a requested exclusive access at that CFG point. Find its later use, not merely its lexical declaration. Possible repairs include reordering the last needed read, borrowing disjoint fields, or changing the API to return owned data.

“Use of moved value” means initialization analysis found a path where the place was deinitialized. Borrow annotations are irrelevant unless a borrow caused the move choice. Clone only when duplication is semantically correct and affordable. Often passing &T, restructuring ownership, or returning the value is better.

“Borrowed value does not live long enough” means a required region includes a point beyond the referent's validity. Identify who requires the longer validity: a return type, stored field, closure, trait object default, spawned task, or destructor. Do not reflexively add 'static; that demands validity for the program's entire relevant duration and is frequently impossible for locals.

“Cannot move out because it is borrowed” combines initialization transfer with a live loan. End actual uses of the reference, redesign ownership, or extract through an API that updates state safely.

Error codes and wording change. The stable debugging method is invariant based:

1. name the place;
2. classify the attempted access;
3. locate the loan or prior move;
4. find the use or drop that keeps it relevant;
5. trace all CFG paths connecting them;
6. inspect implicit operations.

This method scales from beginner code to rustc bugs because it mirrors the analysis without requiring internal type names.

24. An educational liveness and loan checker#

The following stable Rust program implements a tiny CFG analysis. It teaches fixed points, backward liveness, forward loan propagation, and access conflicts. It is intentionally not Rust-complete. Places are flat strings, there are no projections, region variables, moves, drops, unwind edges, two-phase loans, reborrows, or unsafe semantics. Each borrow is permanently associated with one reference local and dies when that local is not live.

use std::collections::{BTreeMap, BTreeSet, VecDeque};

type Node = usize;

#[derive(Clone, Copy, Debug, Eq, PartialEq)]
enum BorrowKind {
    Shared,
    Mutable,
}

#[derive(Clone, Copy, Debug, Eq, PartialEq)]
enum AccessKind {
    Read,
    Write,
}

#[derive(Clone, Debug, Eq, PartialEq)]
enum Op {
    Borrow {
        reference: &'static str,
        place: &'static str,
        kind: BorrowKind,
    },
    UseRef(&'static str),
    Access {
        place: &'static str,
        kind: AccessKind,
    },
    Define(&'static str),
    Nop,
}

#[derive(Clone, Debug)]
struct Program {
    ops: Vec<Op>,
    successors: Vec<Vec<Node>>,
}

#[derive(Clone, Debug, Eq, Ord, PartialEq, PartialOrd)]
struct Loan {
    reference: &'static str,
    place: &'static str,
    kind: LoanKind,
    issued_at: Node,
}

#[derive(Clone, Copy, Debug, Eq, Ord, PartialEq, PartialOrd)]
enum LoanKind {
    Shared,
    Mutable,
}

impl From<BorrowKind> for LoanKind {
    fn from(value: BorrowKind) -> Self {
        match value {
            BorrowKind::Shared => Self::Shared,
            BorrowKind::Mutable => Self::Mutable,
        }
    }
}

fn predecessors(program: &Program) -> Vec<Vec<Node>> {
    let mut result = vec![Vec::new(); program.ops.len()];
    for (from, nexts) in program.successors.iter().enumerate() {
        for &to in nexts {
            result[to].push(from);
        }
    }
    result
}

fn reference_use(op: &Op) -> Option<&'static str> {
    match op {
        Op::UseRef(reference) => Some(*reference),
        _ => None,
    }
}

fn reference_definition(op: &Op) -> Option<&'static str> {
    match op {
        Op::Borrow { reference, .. } | Op::Define(reference) => Some(*reference),
        _ => None,
    }
}

fn liveness(program: &Program) -> (Vec<BTreeSet<&'static str>>, Vec<BTreeSet<&'static str>>) {
    let count = program.ops.len();
    let preds = predecessors(program);
    let mut live_in = vec![BTreeSet::new(); count];
    let mut live_out = vec![BTreeSet::new(); count];
    let mut work: VecDeque<Node> = (0..count).collect();

    while let Some(node) = work.pop_front() {
        let mut new_out = BTreeSet::new();
        for &successor in &program.successors[node] {
            new_out.extend(live_in[successor].iter().copied());
        }

        let mut new_in = new_out.clone();
        if let Some(definition) = reference_definition(&program.ops[node]) {
            new_in.remove(definition);
        }
        if let Some(used) = reference_use(&program.ops[node]) {
            new_in.insert(used);
        }

        if new_in != live_in[node] || new_out != live_out[node] {
            live_in[node] = new_in;
            live_out[node] = new_out;
            work.extend(preds[node].iter().copied());
        }
    }

    (live_in, live_out)
}

fn issued_loans(program: &Program) -> BTreeMap<&'static str, Loan> {
    program
        .ops
        .iter()
        .enumerate()
        .filter_map(|(node, op)| match op {
            Op::Borrow { reference, place, kind } => Some((
                *reference,
                Loan {
                    reference,
                    place,
                    kind: (*kind).into(),
                    issued_at: node,
                },
            )),
            _ => None,
        })
        .collect()
}

fn active_loans(
    program: &Program,
    live_in: &[BTreeSet<&'static str>],
    loans: &BTreeMap<&'static str, Loan>,
) -> Vec<BTreeSet<Loan>> {
    let mut active = vec![BTreeSet::new(); program.ops.len()];
    for node in 0..program.ops.len() {
        for reference in &live_in[node] {
            if let Some(loan) = loans.get(reference) {
                if loan.issued_at < node {
                    active[node].insert(loan.clone());
                }
            }
        }
    }
    active
}

fn conflicts(access: AccessKind, loan: LoanKind) -> bool {
    match (access, loan) {
        (AccessKind::Read, LoanKind::Shared) => false,
        (AccessKind::Read, LoanKind::Mutable) => true,
        (AccessKind::Write, _) => true,
    }
}

#[derive(Debug, Eq, PartialEq)]
struct Error {
    at: Node,
    loan_from: Node,
    place: &'static str,
}

fn check(program: &Program) -> Vec<Error> {
    assert_eq!(program.ops.len(), program.successors.len());
    let (live_in, _) = liveness(program);
    let loans = issued_loans(program);
    let active = active_loans(program, &live_in, &loans);
    let mut errors = Vec::new();

    for (node, op) in program.ops.iter().enumerate() {
        let Op::Access { place, kind } = op else { continue };
        for loan in &active[node] {
            if loan.place == *place && conflicts(*kind, loan.kind) {
                errors.push(Error {
                    at: node,
                    loan_from: loan.issued_at,
                    place,
                });
            }
        }
    }
    errors
}

fn main() {
    let program = Program {
        ops: vec![
            Op::Borrow { reference: "r", place: "x", kind: BorrowKind::Shared },
            Op::Access { place: "x", kind: AccessKind::Write },
            Op::UseRef("r"),
        ],
        successors: vec![vec![1], vec![2], vec![]],
    };
    println!("{:?}", check(&program));
}

The checker computes liveness before checking accesses. At node 1, r is live because node 2 uses it. The shared loan of x is therefore active, and writing x conflicts. If node 2 becomes Nop, r is not live at node 1 and the write passes.

25. Testing and extending the educational checker#

Place these tests below the program in a Cargo binary or library. They use only stable standard-library APIs.

#[cfg(test)]
mod tests {
    use super::*;

    #[test]
    fn later_use_keeps_shared_loan_live() {
        let p = Program {
            ops: vec![
                Op::Borrow { reference: "r", place: "x", kind: BorrowKind::Shared },
                Op::Access { place: "x", kind: AccessKind::Write },
                Op::UseRef("r"),
            ],
            successors: vec![vec![1], vec![2], vec![]],
        };
        assert_eq!(check(&p), vec![Error { at: 1, loan_from: 0, place: "x" }]);
    }

    #[test]
    fn dead_reference_releases_loan() {
        let p = Program {
            ops: vec![
                Op::Borrow { reference: "r", place: "x", kind: BorrowKind::Shared },
                Op::Access { place: "x", kind: AccessKind::Write },
                Op::Nop,
            ],
            successors: vec![vec![1], vec![2], vec![]],
        };
        assert!(check(&p).is_empty());
    }

    #[test]
    fn branch_use_keeps_loan_live_before_branch() {
        let p = Program {
            ops: vec![
                Op::Borrow { reference: "r", place: "x", kind: BorrowKind::Shared },
                Op::Access { place: "x", kind: AccessKind::Write },
                Op::UseRef("r"),
                Op::Nop,
            ],
            successors: vec![vec![1], vec![2, 3], vec![3], vec![]],
        };
        assert_eq!(check(&p).len(), 1);
    }

    #[test]
    fn shared_read_is_compatible() {
        let p = Program {
            ops: vec![
                Op::Borrow { reference: "r", place: "x", kind: BorrowKind::Shared },
                Op::Access { place: "x", kind: AccessKind::Read },
                Op::UseRef("r"),
            ],
            successors: vec![vec![1], vec![2], vec![]],
        };
        assert!(check(&p).is_empty());
    }

    #[test]
    fn mutable_loan_blocks_a_read() {
        let p = Program {
            ops: vec![
                Op::Borrow { reference: "r", place: "x", kind: BorrowKind::Mutable },
                Op::Access { place: "x", kind: AccessKind::Read },
                Op::UseRef("r"),
            ],
            successors: vec![vec![1], vec![2], vec![]],
        };
        assert_eq!(check(&p).len(), 1);
    }
}

Several simplifications are pedagogically dangerous unless stated. Reference liveness alone is not full region inference. The implementation assumes one static loan per reference name, orders issuance by node number, and does not truly propagate through loops. Real CFG reachability cannot use numeric comparison as dominance. Real rustc handles assignments between references, nested types, projections, kills, calls, drops, universals, and constraints.

A valuable exercise is to replace issued_at < node with forward reachability from issuance. Then add explicit Kill(place) operations and ensure a kill cannot erase a still-needed incompatible loan without first reporting the access. Next represent places as a root plus field projections and implement conservative overlap. Each extension should add a failing test before code.

The lesson is the shape of analysis, not a substitute Rust checker. A toy can accidentally accept unsafe patterns because omitted features correspond to real soundness obligations.

26. What current NLL still rejects#

NLL greatly improved precision, but its region model can conflate loans flowing through different control-flow paths. The canonical “problem case #3” involves a function that conditionally returns a borrow; on the nonreturning path, code wants another mutable borrow. Traditional NLL may keep the first loan relevant too broadly.

fn get_or_insert<'a>(map: &'a mut Vec<String>) -> &'a mut String {
    if let Some(value) = map.get_mut(0) {
        return value;
    }
    map.push(String::new());
    &mut map[0]
}

Depending on the exact compiler version and formulation, this family has historically produced conflicts because the returned loan and continuation path are not distinguished precisely enough. Do not use this snippet as a permanent acceptance test without recording rustc -Vv; Polonius migration changes which cases pass.

Other limitations arise from place disjointness. The checker cannot generally prove i != j for two indices. It does not solve arbitrary predicates to establish alias separation. Control-flow precision and value-based alias precision are different axes; Polonius primarily addresses the former formulation, not every theorem about indices.

Loop-carried borrows, lending iterators, and conditional returns can expose path conflation. Destructor liveness and trait constraints can also retain requirements users do not expect. Some rejected programs should be expressed through a safe abstraction rather than accepted directly.

Classify a limitation before attributing it to NLL: Is it region path sensitivity, place overlap, trait solving, dropck, closure capture, or a deliberate language rule? That classification points to the correct subsystem and avoids claiming Polonius will solve unrelated cases.

27. Polonius changes the central question#

Classic NLL asks which CFG points belong to each region, then derives which loans are in scope. Polonius turns the relationship around. An origin is modeled as a set of loans, and subset relationships describe how loans can flow between origins. Those relationships can vary by CFG location.

Suppose loan L0 is issued into origin Oa at P1. An assignment of one reference to another creates Oa ⊆ Ob at the relevant point. Then L0 belongs to Ob where that relationship applies. If control flow splits, propagation can follow only edges and subset facts that actually occur on each path. This avoids some global conflation inherent in a single location-insensitive relation.

Polonius is a formulation, not simply a faster region solver. Its philosophy is to track the permission facts directly and locally enough to separate conditional flows. Illegal access checking remains recognizable: an access invalidates a live conflicting loan.

Origins are not runtime pointer provenance and not source variable scopes. They are analysis entities associated with reference-bearing types. Sets of loans can flow through types and assignments under constraints. Loan liveness still depends on CFG paths and kills.

The name also refers historically to a standalone project that encoded analyses in Datalog-like relations. That prototype was invaluable for experimentation and differential comparison. It must not be confused with every current in-rustc implementation bearing the Polonius name.

28. Location-insensitive and location-sensitive forms#

A location-insensitive form computes subset relationships without distinguishing every CFG location. Other aspects may remain flow sensitive, but the origin subset closure is global. This can reproduce existing NLL acceptance while changing the internal representation from point sets to loan sets. It is a migration foundation and a useful comparison mode.

A location-sensitive form indexes subset relationships by location. subset(O1, O2, P) can hold at P without holding everywhere. Facts flow along CFG edges, are transformed by assignments, and interact with origin liveness. That path sensitivity can accept conditional-borrow patterns safely.

Think of colored water in pipes. Location-insensitive analysis says two reservoirs are connected somewhere, so dye can be treated as globally shared. Location-sensitive analysis records which valve is open at each station. It can show that dye reaches the return branch but not the mutation branch.

Greater precision costs memory and time. The fact domain may multiply origins, loans, locations, and edges. Efficient implementations compress graphs, exploit sparse liveness, cache reachability, and avoid materializing huge cross-products. Diagnostic explanations also need provenance for why a loan reached a point.

“Location-sensitive” does not mean path-perfect. Merges still combine facts according to the chosen abstraction. Place overlap remains conservative. Trait and drop rules remain independent constraints. No formulation promises acceptance of every memory-safe program.

29. Reachability inside current rustc Polonius support#

The rustc 1.97.1 nightly API documentation contains rustc_borrowck::polonius. Its module description says flow-sensitive borrow-check concerns are modeled as a graph containing region and control-flow information. Loan propagation is treated as a reachability problem with subtleties.

Type-checking constraints let loans flow from region to region at the same CFG point. Liveness constraints let loans flow between points through regions live at those points. Invariant relationships can create bidirectional edges. The documentation notes that loans can flow “back in time” to traverse constraints arising earlier in the CFG. Kills are incorporated when deciding which reaching loans remain live.

After this reachability phase, the documented implementation feeds live loans into familiar NLL dataflow, combining them with frontiers where loans cease propagating to produce active loans. Illegal accesses are still found by checking loan invalidation. This hybrid description is more accurate for that source version than saying rustc simply runs the historical Datalog rules.

The exact 1.97.1 API source is: https://doc.rust-lang.org/stable/nightly-rustc/rustc_borrowck/polonius/index.html. Nightly API documentation is unstable compiler-internal documentation. Its existence proves support code exists, not by itself which mode is default for every channel or edition.

Reachability offers an implementation vocabulary familiar from graph algorithms. Nodes encode origin-at-point states. Edges encode same-point subset transfer or CFG liveness transfer. A loan introduced at one node reaches another if a permitted path exists without a relevant kill. Optimizations can answer many such queries without eagerly listing every path.

30. Prototype history and version-sensitive status#

The standalone rust-lang/polonius repository historically implemented the formulation using Datalog-style input facts and engines such as datafrog. It enabled rapid rule experimentation, fact dumps from rustc, and comparison among algorithm variants. It was research infrastructure, not evidence that an external Datalog process is the default production borrow checker.

The official October 2023 Inside Rust update explicitly said the plan did not use that Datalog-based implementation and instead reimplemented lessons within rustc. It proposed milestones including location-insensitive loans, full testing, location sensitivity, modeling, and eventual stabilization. Those milestones and Rust 2024 aspiration are historical roadmap material. They were forecasts with unknowns, not commitments. Source: https://blog.rust-lang.org/inside-rust/2023/10/06/polonius-update.html.

As of the specified rustc 1.97.1 documentation, an in-tree Polonius module and reachability implementation are visible. Exact enablement, default status, accepted examples, flags, and migration completeness remain version sensitive. Verify the release source, compiler options, project-goal updates, and current test configuration before stating them. Do not extrapolate from a newer nightly post to 1.97.1 or from an older roadmap to current behavior.

This textbook deliberately does not promise an acceptance or stabilization timeline. Soundness review, performance, diagnostics, ecosystem testing, and unresolved interactions can alter plans. “Polonius” may also label several staged modes with different precision. Always name the mode and revision.

The prototype remains educational. Datalog relations make inputs and derived facts explicit, which is excellent for understanding fixed points. The production engineering question is broader: memory use, incremental compilation, diagnostics, integration with rustc types, and maintenance all matter.

31. Trait solving and higher-ranked outlives#

Borrow checking does not operate after trait solving has erased every uncertainty. Trait obligations determine method choices, associated types, normalization, variance-relevant structure, and outlives facts. The borrow checker can receive constraints whose justification involves trait reasoning.

Higher-ranked constraints are especially delicate because they quantify over arbitrary lifetimes. A naive subset closure must not instantiate a universally bound region with a convenient local region. Placeholder and leak-check machinery protects that boundary.

The next trait solver aims to provide a more systematic logical foundation for trait goals. Official Polonius planning has discussed moving or factoring higher-ranked outlives processing so ordinary Polonius input can be simpler. That architectural direction should not be converted into a date, guaranteed division of responsibility, or claim that all HRTB issues are solved. Consult current rustc source and types-team updates.

When a test changes under trait-solver flags, isolate whether the changed fact is type normalization, implied bounds, universe handling, or loan propagation. A borrow diagnostic can be downstream of a trait answer. Differential testing should compare semantic outcomes and ensure neither mode silently accepts an unsound program.

The clean conceptual boundary is: trait solving establishes type-level propositions; MIR type checking turns relevant propositions into body constraints; region or origin analysis propagates them over control flow; conflict checking validates accesses. Real implementation boundaries may overlap for performance and legacy reasons.

32. Performance, facts, and soundness engineering#

Borrow checking runs on every body, so average overhead matters. Large generated functions, deeply nested types, many regions, dense CFGs, and abundant constraints stress worst cases. A theoretically polynomial algorithm can still allocate too many sets or traverse too many nearly identical edges.

Common tools include strongly connected component compression, bitsets, sparse sets, worklists, dominance information, and memoized reachability. Location sensitivity can multiply state, so liveness pruning is valuable: a loan need not flow through an origin that cannot carry a relevant reference there. Measurements should include wall time, peak memory, instruction counts, and representative crates.

Fact dumps serialize analysis inputs or derived relations for inspection. They allow replay outside rustc and comparison with reference formulations. Because internal flags and formats are unstable, commands must be taken from the exact compiler revision. Fact dumps can be enormous and may contain source paths; sanitize them before sharing.

Differential testing runs old and new modes over the same corpus. An acceptance difference is not automatically a new-mode win. New acceptance requires a soundness argument; new rejection may be a regression; diagnostic-only differences need review. Crater tests published crates, while compiler suites target known corner cases. Fuzzers generate MIR or source patterns that humans may overlook.

A soundness regression is a compiler change that accepts code capable of violating Rust's safety guarantees in safe code or invalidates established unsafe contracts. Treat suspected regressions as high priority and minimize without publishing unnecessary exploit detail. Check stable releases to find the regression range. Do not “fix” performance by dropping a constraint unless its redundancy is proved.

33. Where to look when something breaks#

Start from the symptom, not a favorite subsystem. For move-after-use or partial initialization, inspect move paths, initialization dataflow, and drop elaboration. For obviously overlapping accesses, inspect place conflict and access-depth classification. For a loan lasting too long, inspect liveness, region constraints, kills, and active-loan computation.

For conditional-return precision, inspect Polonius constraints, reachability, and location-sensitive propagation. For method-call surprises, inspect type-check adjustments, autoref, two-phase reservation, and activation. For closure or async failures, inspect capture inference, coroutine saved locals, and propagated closure requirements. For destructor-related cases, inspect dropck and drop liveness. For HRTBs, inspect universes, placeholders, leak checking, and trait-solver output.

In the rust repository, likely areas include the rustc_borrowck crate, MIR dataflow crates, MIR type checking, diagnostic modules, and UI tests. Paths and module names evolve. Use the failing diagnostic's code references and rg exact message fragments against the checked-out revision rather than relying on an old blog post.

For diagnostics that reject correctly but point badly, preserve the semantic test and add or adjust a UI diagnostic test. For acceptance bugs, add a compile-fail test that reaches the forbidden operation. For erroneous rejection, add a check-pass test and explain why aliases are separated.

A useful triage note records compiler version, host, target, edition, command, flags, expected behavior, actual output, earliest known affected release, and whether unsafe code is present. That information turns “borrow checker weird” into an actionable report.

34. Minimizing and reporting lifetime bugs#

Begin with a copy of the reproducer. Remove dependencies, macros, generic parameters, branches, and fields one at a time while checking that the same root behavior remains. Replace library types with tiny local types only if that preserves relevant Drop, variance, and trait behavior.

Do not minimize away a later use that keeps the loan live. Do not remove an await, closure capture, return, or destructor if it supplies the crucial edge. Compare full diagnostics, not merely error codes. For suspected unsoundness, demonstrate the invalid safe behavior with the smallest responsible unsafe abstraction clearly identified.

Nightly -Z options can dump MIR, region facts, or select experimental checking modes, but names and semantics change. Run rustc -Z help on the exact nightly and include rustc -Vv. Never recommend a nightly flag as a stable production guarantee. If a mode difference disappears on current nightly, bisect revisions when practical.

Search existing Rust issues using the error text and conceptual labels such as two-phase, dropck, lending iterator, or Polonius. If filing, include a standalone file and one command. State whether the report is wrong acceptance, wrong rejection, performance, ICE, or diagnostics. An internal compiler error, or ICE, is a compiler crash and should include the query stack if available.

Avoid lifetime annotation cargo cult. Minimization should reveal the actual required relation. An explicit bound can legitimately repair an underspecified API, but an arbitrary 'static can hide the intended caller contract and make the example less useful.

35. Compiler tests and contribution workflow#

Build the compiler following the current rustc-dev-guide rather than old personal scripts. Use the repository's bootstrap tool and a stage appropriate to the changed crate. Run the narrowest relevant test first, then broader suites requested by reviewers or CI.

Borrow-check tests commonly live in UI suites. A source file is compiled with annotations or companion expected stderr. Check-pass tests establish acceptance. Compile-fail tests establish rejection and diagnostic shape. Known-bug tests may document unresolved behavior without pretending it is correct. Exact directories and directives evolve, so copy a nearby current test.

A strong semantic test is minimal but adversarial. Include the operation that would become invalid if the checker were wrong. For a two-phase case, distinguish reservation and activation. For path sensitivity, include both branches. For dropck, ensure a destructor could observe the reference. For diagnostics, assert origin, conflict, and use labels where stable enough.

Before changing code, reproduce with the repository compiler. Read blame and linked issues for invariants. Add the failing test, make the smallest implementation change, run formatting, and execute targeted tests. For performance-sensitive changes, collect before-and-after measurements. For semantic changes, explain why newly accepted code is safe under the language model.

Review is part of correctness. Borrow checking touches unsafe-code expectations and optimization assumptions. A locally green test cannot cover the state space. Be explicit about uncertainty, request types-team input when needed, and do not combine a semantic change with broad refactoring unless necessary.

36. A stage-talk outline#

Open with two boxes, one address note, and a use-after-free. Ask the audience what fact would prevent the read. Introduce ownership as destruction responsibility and borrowing as temporary permission. Avoid lifetime syntax for the first ten minutes.

Next draw a struct as a tree of places. Move one field, copy another, and show initialization lights beside each node. Overlay shared and exclusive loan sheets. Use split_at_mut to show how APIs expose proofs.

Then draw a five-station CFG. Place a borrow at station one, a branch at two, a use on one arm, and a mutation after the join. Compute backward liveness with the audience. Emphasize that NLL means control-flow-derived regions, not vanished lifetimes or simplistic textual last use.

Walk through the rustc pipeline on one slide: MIR preparation, move paths, region replacement, MIR type check, constraints, fixed point, loans, conflicts, diagnostics. On another slide, contrast regions-as-point-sets with origins-as-loan-sets. Animate location-insensitive global flow and location-sensitive valves.

Show v.push(v.len()) as reservation then activation. Show a partial move and a destructor. Show an async borrow crossing await and identify the pinning boundary without diving into executor details.

Finish with a diagnostic containing origin, conflict, and use spans. Present the toy checker as a lab, prominently list its omissions, and close with a version-status slide linking exact sources. Never put a stabilization date on that slide.

37. Exercises from apprentice to compiler contributor#

1. Draw the place tree for a nested struct and classify five accesses as overlapping or disjoint.

Explain where runtime indexing defeats the proof.

1. For a branch and loop, write live_in and live_out sets until a fixed point.

Mark which edge causes a second iteration.

1. Take an NLL example accepted after a final use.

Add a destructor or loop use that legitimately extends the requirement. Explain why “last textual use” fails.

1. Modify the educational checker to compute true CFG reachability from loan issuance.

Add a loop test where node numbering is misleading.

1. Add projected places such as p.left.

Define overlap rules for parents and distinct fields. Document why dereferences and indices remain conservative.

1. Model two-phase loans with Reserved and Active states.

Test a shared read during reservation, a mutable conflict during reservation, and a conflict at activation.

1. Minimize a closure lifetime diagnostic while preserving capture mode.

Label origin, escape, and required universal region.

1. Compare a historical Polonius Datalog rule description with the rustc 1.97.1 reachability module documentation.

List vocabulary that maps cleanly and implementation claims that do not.

1. Design a differential test matrix covering acceptance, rejection, diagnostics, compile time, and memory.

Explain how you would investigate a newly accepted case.

1. Draft a rustc issue for an imaginary wrong rejection.

Include version, standalone code, command, expected invariant, and suspected subsystem without claiming certainty.

1. Explain why borrow checking cannot prove an unsafe linked-list implementation sound.

List additional contracts and tools needed.

1. Read current source for one -Z borrow-check option and report its exact revision.

Explain why the answer should not be copied into timeless user documentation.

38. Predictions, boundaries, and source ledger#

Prediction one: future borrow checking will become more path precise while preserving a conservative place model. Evidence is the origin-and-loan formulation and in-rustc reachability work. This is a technical forecast, not a schedule.

Prediction two: trait solving and borrow checking will exchange cleaner constraint interfaces. Higher-ranked reasoning motivates that separation, but difficult universe and diagnostic questions remain. No roadmap date follows from the architectural appeal.

Prediction three: performance work will increasingly avoid materializing all location-by-origin-by-loan facts. Sparse liveness, graph compression, and demand-driven reachability are natural tools. Actual choices must be benchmarked on rustc workloads.

Prediction four: diagnostics will remain a first-class migration constraint. A checker can be semantically better and still be unsuitable as a default if explanations regress badly. Origin, conflict, use, and explanation-path provenance should shape analysis APIs.

The durable boundaries are equally important. Lifetimes are compile-time validity relationships, not runtime durations. NLL derives regions from control flow but retains lifetime reasoning. Polonius does not solve arbitrary index inequality, all trait ambiguity, or unsafe-code correctness. No borrow checker proves every unsafe implementation sound.

Primary sources verified for this chapter:

• rustc-dev-guide, “The borrow checker”: https://rustc-dev-guide.rust-lang.org/borrow-check.html
• rustc-dev-guide, “Region inference (NLL)”: https://rustc-dev-guide.rust-lang.org/borrow-check/region-inference.html
• rustc 1.97.1 nightly API, rustc_borrowck::polonius: https://doc.rust-lang.org/stable/nightly-rustc/rustc_borrowck/polonius/index.html
• Inside Rust, “Polonius update,” 2023-10-06: https://blog.rust-lang.org/inside-rust/2023/10/06/polonius-update.html
• NLL RFC 2094: https://rust-lang.github.io/rfcs/2094-nll.html
• historical standalone prototype: https://github.com/rust-lang/polonius

The 2023 post is marked historical because its milestones describe intent at that date. The 1.97.1 module documents code present in that compiler source line, not a timeless default-status guarantee. Readers investigating later versions should re-open all version-sensitive sources.

The expert habit is not memorizing one checker implementation. It is preserving invariants while translating among source code, places, MIR CFGs, constraints, loans, and diagnostics. With that map, both accepted programs and compiler bugs become explainable rather than mystical.

Part VI-B: Lifetimes, NLL, and Polonius from First Principles#

39. The proof that every borrow checker must make#

This continuation develops one proof three times. The historical lexical checker approximated the proof with source scopes. MIR NLL computes it over control-flow points. Polonius recasts its hardest part as propagation of loans through origins.

The safety question is concrete. Whenever code uses a reference, its referent must still be alive and suitably accessible. Whenever code accesses a place directly, no incompatible live loan may cover that place. Whenever destruction runs, the values inspected by that destruction must remain valid.

Memory is storage with an address and a period during which its contents are initialized. Ownership assigns responsibility for ending that period. A move transfers a value and deinitializes its old place. A place is a local plus projections such as .field, [index], or *deref. An access reads, writes, moves, borrows, or drops a place. A loan records permission created by a borrow. A region describes where a reference may need to be valid.

These nouns describe different things. The reference is a runtime value. The place is storage. The loan is a compile-time permission. The region is a compile-time set or relation. The lifetime syntax is a programmer-visible name for some region relationships.

The core invariant is:

At every reachable program point, each performed access is compatible with every loan that can matter there.

Compatibility has a spatial half and a temporal half. Spatial reasoning asks whether two places overlap. Temporal reasoning asks whether the loan can still matter at that point. Borrow checking joins those answers.

source expression
    │ lowers to
    ▼
MIR place + access ──overlap──► borrowed place
    │                              │
    │ at CFG point                 │ issued as loan
    ▼                              ▼
access compatibility ◄──── loan live/reachable here
    │
    ├── compatible: continue
    └── conflict: explain issuance, access, and later need

The diagram suppresses moves, drops, trait obligations, and unwind edges. Those omissions are educational, not semantic claims.

Consider a field borrow.

struct Pair {
	left: String,
	right: String,
}

fn main() {
	let mut pair = Pair { left: "L".into(), right: "R".into() };
	let left = &pair.left;
	pair.right.push('!');
	println!("{left}");
}

The shared loan covers pair.left. The mutation covers pair.right. Known fields are disjoint, so the accesses coexist. Replacing pair.right.push with pair.left.push makes their footprints overlap. The later println! keeps the shared loan relevant.

Prediction: if the print is removed, should mutation of pair.left be accepted? Lexical checking historically often said no because left remained in scope. NLL can say yes because no later use requires the reference. That contrast begins the three-stage arc.

An accepted safe program is not evidence that arbitrary raw-pointer variants are safe. Raw pointers can bypass this static permission proof. Unsafe code must establish validity, alignment, initialization, aliasing, and provenance obligations itself. A borrow-checker soundness bug can therefore become a security bug: safe code might construct dangling or conflicting references and trigger undefined behavior.

40. Stage one: lexical regions and AST-era reasoning#

Early Rust borrow checking operated much closer to the abstract syntax tree. Its natural vocabulary was nested lexical scopes. A lexical scope is a source interval introduced by a block, statement, or expression. The checker assigned a borrow a scope broad enough to contain uses of the resulting reference.

This design solved an important problem cheaply. Scopes already existed. They nested like a tree. Containment was easy to compute and explain. If reference r could be used throughout block B, keeping its referent borrowed throughout B was conservative.

outer block B0
┌──────────────────────────────────────┐
│ let mut x = 0;                       │
│ inner block B1                       │
│ ┌──────────────────────────────────┐ │
│ │ let r = &x;                      │ │
│ │ print(r);                        │ │
│ │ x = 1;  lexical model rejects   │ │
│ └──────────────────────────────────┘ │
└──────────────────────────────────────┘

The lexical approximation remembers scope nesting. It forgets that r is dead after print(r). That forgotten fact causes a false conflict.

fn main() {
	let mut text = String::from("old");
	let view = &text;
	println!("{view}");
	text.clear();
}

Modern stable Rust accepts this program. An educational lexical checker can model the old rejection by extending each loan to the end of the block containing its binding.

The AST-era approach was not merely “bad NLL.” It reflected the compiler representation and language of its time. Before MIR supplied a normalized control-flow graph, expression-specific checking had to understand loops, matches, overloaded operators, and temporaries directly. Lexical scopes made that complexity tractable.

The approximation breaks visibly around conditionals.

fn choose(flag: bool) {
	let mut value = String::from("v");
	if flag {
		let r = &value;
		println!("{r}");
	}
	value.push('!');
}

Scope nesting can handle this particular block. More difficult cases store a reference in a variable assigned on only one path or use it in one match arm. The required validity is a shape in a control-flow graph, not necessarily one contiguous source interval.

Loops expose another mismatch. Text appears once, but execution can revisit it. break, continue, and early return create edges that scope intervals do not display. An AST visitor must reconstruct these possibilities or remain coarse.

Lexical checking established useful enduring ideas:

1. References carry validity requirements.
2. Outlives relationships connect those requirements.
3. Borrowing creates restrictions on access.
4. Place overlap determines whether restrictions apply.
5. Errors need source spans even if the proof uses another representation.

It also exposed a policy choice. The compiler can reject programs it cannot prove safe. That preserves soundness but imposes ergonomic cost. NLL improved the proof rather than weakening the invariant.

Counterexample: shrinking every borrow to its final textual use is unsound.

fn branch(flag: bool) {
	let mut x = 0;
	let r = &x;
	if flag {
		println!("{r}");
	}
	x += 1;
}

The print is textually before the mutation, but path reasoning still matters in general. With loops, a later CFG iteration may use a reference at an earlier source line. “Last line number” is not a region algorithm.

Prediction: move the mutation into the else arm. Can it coexist with the loan? Yes, because no execution performs both the conflicting mutation and the use requiring the loan. A path-sensitive control-flow model can represent that distinction.

Historical claims should be read as architecture, not exact archaeology. Compiler internals changed repeatedly before and during RFC 2094's implementation. The normative lesson is only that pre-NLL checking was lexically scoped and less control-flow precise. Exact behavior belongs to a pinned historical compiler and its tests.

41. A complete lexical checker in stable Rust#

The first checker deliberately handles a tiny language. Programs are one linear block. Places are whole local names. Statements can borrow, read, write, or end a scope. A borrow lasts through the numeric end of its lexical scope.

This representation makes one policy obvious. Borrow { end } receives an end point from parsing rather than deriving one from uses. It cannot express branches, projections, moves, or reborrows. That weakness is the lesson.

The following is a complete stable-Rust program.

use std::collections::HashMap;

#[derive(Clone, Copy, Debug, PartialEq, Eq)]
enum Kind {
	Shared,
	Unique,
}

#[derive(Debug)]
enum Stmt<'a> {
	Borrow { place: &'a str, kind: Kind, end: usize },
	Read(&'a str),
	Write(&'a str),
}

#[derive(Clone, Copy, Debug)]
struct Loan {
	kind: Kind,
	issued: usize,
	end: usize,
}

fn conflicts(kind: Kind, write: bool) -> bool {
	write || kind == Kind::Unique
}

fn check(program: &[Stmt<'_>]) -> Vec<String> {
	let mut loans: HashMap<&str, Vec<Loan>> = HashMap::new();
	let mut errors = Vec::new();

	for (point, stmt) in program.iter().enumerate() {
		loans.values_mut().for_each(|xs| xs.retain(|loan| point <= loan.end));

		match *stmt {
			Stmt::Borrow { place, kind, end } => {
				let incompatible = loans
					.get(place)
					.into_iter()
					.flatten()
					.any(|loan| kind == Kind::Unique || loan.kind == Kind::Unique);
				if incompatible {
					errors.push(format!("point {point}: incompatible borrow of {place}"));
				} else {
					loans.entry(place).or_default().push(Loan { kind, issued: point, end });
				}
			}
			Stmt::Read(place) | Stmt::Write(place) => {
				let write = matches!(stmt, Stmt::Write(_));
				for loan in loans.get(place).into_iter().flatten() {
					if conflicts(loan.kind, write) {
						errors.push(format!(
						"point {point}: access to {place} conflicts with {:?} loan from {}",
						loan.kind, loan.issued
					));
					}
				}
			}
		}
	}
	errors
}

fn main() {
	let rejected = [
		Stmt::Borrow { place: "x", kind: Kind::Shared, end: 2 },
		Stmt::Read("x"),
		Stmt::Write("x"),
	];
	assert_eq!(check(&rejected).len(), 1);

	let accepted = [
		Stmt::Borrow { place: "x", kind: Kind::Shared, end: 1 },
		Stmt::Read("x"),
		Stmt::Write("x"),
	];
	assert!(check(&accepted).is_empty());
}

The checker separates mechanism from policy. The active-loan table and compatibility test are mechanism. Supplying end from a lexical scope is policy. NLL will replace that policy without discarding every concept.

Walk the rejected program. At point zero, loan L0 is inserted for x through point two. At point one, reading x is compatible with a shared loan. At point two, writing overlaps x and conflicts with L0.

The accepted array manually shortens the scope to point one. A real lexical parser would obtain that boundary from syntax. The test demonstrates the only fact the checker needs from that parser.

Compile it with:

rustc --edition=2021 lexical.rs
./lexical

Exercises:

1. Add Move and make it conflict like a write.
2. Add EndLoan(id) and compare explicit ends with lexical ends.
3. Add pair.left and pair.right as segmented places.
4. Make parent places overlap descendants.
5. Write a test showing why string-prefix overlap mishandles similarly named locals.

The complete program omits reference variables. It therefore cannot ask whether the reference is later used. It omits CFG edges, so it cannot model loops or branches. It omits initialization, so moving and reading moved data are outside its proof. It omits types, destructors, implicit borrows, unsafe code, and unwind. It is an executable scope-conflict model, not a Rust checker.

42. Stage two: MIR turns lifetime checking into dataflow#

MIR gives borrow checking a control-flow graph after much surface syntax is normalized. A basic block is a straight-line sequence ending in a terminator. A location identifies a statement boundary or terminator. Edges represent possible continuation, including branches and loops.

NLL treats a region approximately as a set of MIR points. The compiler generates requirements saying which points or other regions it must contain. Inference grows sets until every requirement holds. The resulting region need not resemble a lexical source interval.

bb0:
  p0: r = &x       loan L issued
  p1: switch flag
       ├─true──► bb1:p2 use(r) ──┐
       └─false─► bb2:p3          │
                                 ▼
                              bb3:p4 x += 1

If only the true arm uses r, liveness requires its region on the path to p2. It need not include the false arm merely because that arm lies between source braces. At the join, whether p4 is legal depends on whether any continuation still needs r.

MIR does not make the checker perfectly path sensitive. Dataflow joins information conservatively. Values and constraints may lose correlations between conditions. The advance is that control flow is explicit and fixed-point analysis can respect edges.

The broad borrow-checking pipeline is:

1. Build and prepare MIR.
2. Gather move paths and initialization dataflow.
3. Replace inferred regions with inference variables.
4. Type-check MIR and collect region constraints.
5. Add liveness and universal-region requirements.
6. Solve region relationships.
7. Gather borrows and their activation information.
8. Walk MIR, checking accesses against loans.
9. Convert failed invariants back to source diagnostics.

This ordering is conceptual. The exact query and pass organization is revision-sensitive. Do not infer an API guarantee from it.

Moving work from AST to MIR has costs. Source constructs have been desugared, so diagnostics need preserved spans and semantic provenance. One source expression can produce several MIR operations. Generated temporaries can obscure the user's intent. Normalization simplifies analysis while moving explanatory work into diagnostics.

NLL shipped for ordinary Rust code after an extended migration period. The old checker was retained temporarily for comparison and migration diagnostics. That history demonstrates a production rule: semantic replacement needs compatibility data, crater-style testing, performance measurement, and diagnostic review.

43. Region inference: liveness, outlives, and universes#

A local reference type contains a region inference variable. Write it as '?a when the programmer did not name it. Inference does not guess a source span. It accumulates constraints and computes a least solution sufficient for validity.

Liveness says a region must contain a point because a value carrying that region is live there. If local r: &'?a T may be used at point P, then '?a must include P. This connects ordinary variable liveness to reference validity.

Outlives says one region must include another. 'a: 'b means values valid for 'a remain valid for every point required by 'b. Under the point-set model, this is inclusion: points('b) ⊆ points('a).

Direction is easy to reverse. The longer-lived region is the superset. Read 'a: 'b as “'a outlives 'b,” not “'a is inside 'b.”

liveness seed:      P ∈ '?r
outlives edge:      '?long : '?r
set consequence:    points('?r) ⊆ points('?long)
fixed point:        propagate P into '?long

Universal regions represent lifetimes supplied by a caller, including named lifetime parameters and function-body boundaries. They are not inferred as arbitrary local point sets in the same sense. The body must be valid for every caller choice satisfying declared bounds.

fn first<'a, 'b>(x: &'a str, _y: &'b str) -> &'a str {
	x
}

Returning x is valid without relating 'b to 'a. Returning _y would require 'b: 'a. The callee cannot silently invent that promise because a caller may choose 'b shorter than 'a.

Higher-ranked bounds quantify over lifetimes.

fn call_any<F>(f: F)
where
	F: for<'a> Fn(&'a str),
{
	let local = String::from("local");
	f(&local);
}

for<'a> requires the implementation to work for every suitable 'a chosen at each call. Checking such a binder introduces a fresh placeholder in a fresh universe. A universe records which inference variables were created early enough to name which placeholders.

The anti-cheating invariant is:

An inference variable cannot be solved using a placeholder from a universe it cannot name.

Otherwise an existential choice could depend on a later universally quantified lifetime. That would turn “there exists one choice working for all lifetimes” into the weaker “for each lifetime, choose a convenient answer.”

A placeholder leak occurs when constraints let a placeholder escape into an older inference variable or observable result. Leak checking rejects that illicit dependency. This is analogous to skolemization checks in type systems, but rustc's exact representation and solver interaction are implementation details.

Prediction: is for<'a> fn(&'a T) equivalent to fn(&'static T)? No. The first accepts any caller-chosen borrow, including short locals. The second accepts only a reference valid for 'static. Universal capability is not one maximally long concrete lifetime.

Region constraints can be conditional on types and trait solving. Normalization may reveal references hidden in associated types. Errors that appear to be borrow-check failures can begin as unresolved or overly strong type obligations. Contributor debugging must inspect the earliest failed invariant, not just the final label.

44. Loans, move paths, and access checking in NLL#

Region inference answers where references must be valid. Borrow data answers which place was borrowed, how, and where the loan was issued. Move analysis answers whether places are initialized. Access checking combines these analyses.

A move path tree tracks move-relevant places. For record.field, the compiler can track the field separately from its siblings. Moving a non-Copy field deinitializes that path. Reading its sibling can remain valid. Using the whole parent generally cannot.

struct Record {
	name: String,
	count: u32,
}

fn main() {
	let r = Record { name: "Ada".into(), count: 1 };
	let name = r.name;
	println!("{}", r.count);
	drop(name);
}

Initialization dataflow has its own lattice. At a CFG merge, a path may be initialized on one predecessor and moved on another. A read requiring definite initialization must be valid on every incoming path. This is not a lifetime error even though diagnostics can mention moves and borrows together.

Place conflict is structural. Equal places overlap. A parent overlaps its field. Distinct fields of a known struct can be disjoint. Arbitrary indices are conservatively treated as potentially equal. Dereferences depend on pointer kind and alias assumptions.

fn bad(xs: &mut [i32], i: usize, j: usize) {
	let a = &mut xs[i];
	let b = &mut xs[j];
	*a += *b;
}

Even if a caller promises i != j, the signature does not encode that proof. split_at_mut performs a bounds-checked split behind a safe abstraction and returns disjoint mutable slices. Borrow checking does not solve arbitrary integer inequalities.

Loans and reference values are not interchangeable. Copying a shared reference creates another value but does not need an independent permission to the referent. Moving a mutable reference moves the reference value while the underlying exclusivity continues. Reborrowing creates a derived loan with constraints connecting it to the parent access.

fn reborrow(x: &mut i32) {
	let short = &mut *x;
	*short += 1;
	*x += 1;
}

The first unique access is temporarily used through short. After short is no longer needed, the parent route is usable again. Treating reborrow as permanently moving x would reject common code. Treating both routes as simultaneously active would violate exclusivity.

Shared reborrows can shorten a mutable permission for observation.

fn observe_then_write(x: &mut String) {
	let view: &str = &*x;
	println!("{view}");
	x.push('!');
}

The shared loan ends when no later use needs view. Then the unique parent route resumes.

45. Two-phase borrows and implicit receiver choreography#

Ordinary mutable borrows become exclusive when created. That rule would reject useful method calls whose receiver is implicitly borrowed before another argument is evaluated.

fn main() {
	let mut values = vec![1, 2];
	values.push(values.len());
}

Evaluation needs a mutable receiver for push and a shared receiver for len. If the mutable loan activated immediately, the shared access would conflict. Two-phase borrowing splits selected implicit mutable borrows into reservation and activation.

During reservation, compatible shared reads may occur. At method-call activation, the loan becomes fully exclusive. Competing accesses then conflict normally.

p0 reserve mutable loan L for values
 │ reservation permits selected shared access
p1 evaluate values.len()
 │ argument value produced
p2 activate L at call
 │ exclusive from here while needed
p3 execute push

This is not a general promise that every explicit &mut is delayed. Eligibility is intentionally constrained by MIR construction and borrow kind. Generalizing it carelessly could accept aliasing patterns the model cannot justify.

Prediction: should values.push(values.pop().unwrap()) be accepted by the same argument? No simple “arguments before receiver” slogan decides it. The nested pop itself needs mutable access during the outer reservation and can conflict. Inspect actual compiler behavior for the pinned version rather than extrapolating.

Activation is a point in the CFG. Dominance and reachability matter when calls occur conditionally. Diagnostics should identify both reservation and activation when that distinction explains the conflict.

An educational model needs at least three states:

“Sometimes” is deliberate. Real compatibility depends on access depth, place overlap, and borrow classification. A toy checker that permits every shared action during reservation overstates Rust.

46. Variance, drop checking, and destruction-time validity#

Variance determines how subtyping of lifetime arguments flows through a type constructor. Shared references are covariant in their lifetime and referent type under the relevant conditions. Mutable references are covariant in their lifetime but invariant in their referent type. Interior mutation is the reason broad covariance in T would be dangerous.

If 'long: 'short, an &'long T can often be used where &'short T is required. Shortening observation is safe. For &mut T, changing T covariantly could permit writing a shorter-lived reference into storage expected to hold a longer-lived one.

fn shorten<'long, 'short, T>(x: &'long T) -> &'short T
where
	'long: 'short,
{
	x
}

The bound supplies the subset relationship. The function does not alter runtime data.

Drop checking asks what must remain valid when a destructor runs. A destructor can inspect fields even if ordinary code never reads them again. Therefore “last explicit use” is not sufficient around values with Drop.

struct Inspector<'a>(&'a str);

impl Drop for Inspector<'_> {
	fn drop(&mut self) {
		println!("dropping {}", self.0);
	}
}

An Inspector's referent must remain valid when its destructor executes. Drop order becomes part of the proof. Locals and fields have defined destruction rules, but relying on an incorrect remembered order can create confusing errors or unsafe designs.

#[may_dangle] is an unsafe, specialized promise used in library internals. It tells drop checking that a parameter need not be accessed in a forbidden way during destruction, subject to important ownership caveats. It is not a routine escape hatch. Its implementation must uphold the promise for every instantiation.

Variance and dropck interact. PhantomData can express ownership and variance relationships even when no runtime field directly stores a T. Choosing the wrong phantom form can misstate auto-trait, drop, or variance properties. Unsafe abstractions must model what they logically own, not merely what bits they contain.

Debugging checklist:

1. Does the type implement Drop directly or transitively?
2. Which generic parameters may its destructor observe?
3. What is the actual local and field drop order?
4. Did variance permit or forbid a lifetime shortening?
5. Does a phantom field encode the intended ownership?
6. Is an unsafe opt-out making a promise stronger than its body supports?

47. Closures, generators, async suspension, and pinning#

A closure can borrow, mutate, or move captured places. Capture inference chooses enough access for the closure body. The resulting environment is a generated struct-like value whose fields carry those captures.

fn main() {
	let text = String::from("captured");
	let show = || println!("{text}");
	show();
}

The closure environment contains a shared capture related to text. Returning show from a scope where text is local would let that borrow escape and must be rejected. Adding move transfers captured values according to capture semantics; it does not magically make references owned.

Capture precision matters. Capturing one field can avoid borrowing an entire aggregate. Packed layout and dereference boundaries can force coarser captures because forming references to unaligned fields or assuming pointer ownership would be unsafe. Exact capture rules are language-version-sensitive and belong to the Reference and compiler tests.

Async functions and blocks lower to state machines. Values live across .await become fields stored in the suspended future. A borrow crossing suspension therefore relates fields inside that state machine across calls to poll.

async fn length_after_wait(text: String) -> usize {
	let view = &text;
	std::future::ready(()).await;
	view.len()
}

The owned text and view both survive suspension. Conceptually the future may become self-referential after polling: one field can refer to another field in the same future allocation. Moving such a value afterward could invalidate the internal reference.

Pin provides a library-level contract that the pinned value will not be moved in ways forbidden by its pinning guarantees. The compiler-generated future and executor APIs rely on that contract. Pinning does not extend the lifetime of external data. It does not make every field immovable by itself. Projection from pinned containers requires careful guarantees.

before first poll:
Future { text, state = Start }

poll reaches await:
Future { text, view ──► text, state = Suspended }
                   internal relationship

next poll:
Pin<&mut Future> preserves required address stability

The diagram is conceptual. Rust does not guarantee this exact generated layout.

An async closure may capture references whose validity must cover every suspension where they are retained. Send requirements add another dimension: an executor may move the future between threads when it is unpinned and require captured values to be Send. Borrow checking, auto traits, and generator layout diagnostics can therefore meet in one error.

Prediction: does an .await always lengthen every local borrow to the end of the future? No. Only values live across that suspension need storage across it. Dropping or ceasing to use a reference before .await can avoid the cross-suspension relationship. Temporary lifetime and lowering details still matter.

48. A stable-Rust CFG and NLL checker#

The second educational checker replaces lexical ends with backwards liveness. Its language has reference names, whole local places, explicit CFG successors, and four operations. It derives where each reference is live, then considers its loan active from issuance along reachable paths that can reach a live use.

The implementation is complete and dependency free.

use std::collections::{BTreeMap, BTreeSet, VecDeque};

type Point = usize;

#[derive(Clone, Debug)]
enum Op<'a> {
	Borrow { reference: &'a str, place: &'a str, unique: bool },
	UseRef(&'a str),
	Read(&'a str),
	Write(&'a str),
	Nop,
}

#[derive(Clone, Debug)]
struct Node<'a> {
	op: Op<'a>,
	succ: Vec<Point>,
}

#[derive(Clone, Debug)]
struct Loan<'a> {
	reference: &'a str,
	place: &'a str,
	unique: bool,
	issued: Point,
}

fn predecessors(nodes: &[Node<'_>]) -> Vec<Vec<Point>> {
	let mut pred = vec![Vec::new(); nodes.len()];
	for (from, node) in nodes.iter().enumerate() {
		for &to in &node.succ {
			pred[to].push(from);
		}
	}
	pred
}

fn reference_liveness(nodes: &[Node<'_>]) -> Vec<BTreeSet<String>> {
	let mut live_in = vec![BTreeSet::new(); nodes.len()];
	let mut live_out = vec![BTreeSet::new(); nodes.len()];
	loop {
		let old = live_in.clone();
		for point in (0..nodes.len()).rev() {
			live_out[point] = nodes[point]
				.succ
				.iter()
				.flat_map(|&s| live_in[s].iter().cloned())
				.collect();
			let mut now = live_out[point].clone();
			if let Op::Borrow { reference, .. } = nodes[point].op {
				now.remove(reference);
			}
			if let Op::UseRef(reference) = nodes[point].op {
				now.insert(reference.to_owned());
			}
			live_in[point] = now;
		}
		if live_in == old {
			return live_in;
		}
	}
}

fn points_for_loan(nodes: &[Node<'_>], live: &[BTreeSet<String>], loan: &Loan<'_>) -> BTreeSet<Point> {
	let pred = predecessors(nodes);
	let targets: BTreeSet<Point> = (0..nodes.len())
		.filter(|&p| live[p].contains(loan.reference) || matches!(nodes[p].op, Op::UseRef(r) if r == loan.reference))
		.collect();
	let mut can_reach_use = targets.clone();
	let mut work: VecDeque<Point> = targets.into_iter().collect();
	while let Some(point) = work.pop_front() {
		for &before in &pred[point] {
			if can_reach_use.insert(before) {
				work.push_back(before);
			}
		}
	}

	let mut active = BTreeSet::new();
	let mut work = VecDeque::from([loan.issued]);
	while let Some(point) = work.pop_front() {
		if !can_reach_use.contains(&point) || !active.insert(point) {
			continue;
		}
		work.extend(nodes[point].succ.iter().copied());
	}
	active
}

fn check(nodes: &[Node<'_>]) -> Vec<String> {
	let live = reference_liveness(nodes);
	let loans: Vec<Loan<'_>> = nodes
		.iter()
		.enumerate()
		.filter_map(|(issued, node)| match node.op {
			Op::Borrow { reference, place, unique } => Some(Loan { reference, place, unique, issued }),
			_ => None,
		})
		.collect();
	let active: BTreeMap<Point, Vec<&Loan<'_>>> = loans
		.iter()
		.flat_map(|loan| points_for_loan(nodes, &live, loan).into_iter().map(move |p| (p, loan)))
		.fold(BTreeMap::new(), |mut map, (p, loan)| {
			map.entry(p).or_default().push(loan);
			map
		});
	let mut errors = Vec::new();
	for (point, node) in nodes.iter().enumerate() {
		let access = match node.op {
			Op::Read(place) => Some((place, false)),
			Op::Write(place) => Some((place, true)),
			_ => None,
		};
		if let Some((place, write)) = access {
			for loan in active.get(&point).into_iter().flatten() {
				if loan.place == place && (write || loan.unique) {
					errors.push(format!("point {point}: {place} conflicts with loan at {}", loan.issued));
				}
			}
		}
	}
	errors
}

fn main() {
	let branch = vec![
		Node { op: Op::Borrow { reference: "r", place: "x", unique: false }, succ: vec![1] },
		Node { op: Op::Nop, succ: vec![2, 3] },
		Node { op: Op::UseRef("r"), succ: vec![4] },
		Node { op: Op::Write("x"), succ: vec![4] },
		Node { op: Op::Nop, succ: vec![] },
	];
	assert!(check(&branch).is_empty());

	let conflict = vec![
		Node { op: Op::Borrow { reference: "r", place: "x", unique: false }, succ: vec![1] },
		Node { op: Op::Write("x"), succ: vec![2] },
		Node { op: Op::UseRef("r"), succ: vec![] },
	];
	assert_eq!(check(&conflict).len(), 1);

	let unique_read = vec![
		Node { op: Op::Borrow { reference: "r", place: "x", unique: true }, succ: vec![1] },
		Node { op: Op::Read("x"), succ: vec![2] },
		Node { op: Op::UseRef("r"), succ: vec![] },
	];
	assert_eq!(check(&unique_read).len(), 1);
}

The backwards phase marks points from which a use is reachable. The forward phase restricts that set to points reachable after issuance. Their intersection approximates the useful lifetime of the loan.

Trace the branch test. The true arm reaches UseRef("r"), so its points remain relevant. The false-arm write cannot reach that use without first joining after it. It is excluded and accepted.

The checker treats names as exact whole places. It does not kill old reference values on reassignment. It assumes every graph index is valid. It does not distinguish a use before versus after a statement. It ignores borrow conflicts at issuance. It lacks unwind edges, call effects, types, moves, drops, universes, reborrows, and two-phase activation.

Most importantly, it binds one loan directly to one reference name. Assignments such as r2 = r1 and control-flow merges need loans to flow through regions or origins. That limitation motivates Polonius.

49. Stage three: Polonius changes the propagated object#

Classic NLL can be explained as computing points contained in regions and then deciding which loans are in scope. Polonius starts from explicit facts and reasons about loans contained in origins at locations. An origin is an abstract set associated with reference provenance. It can contain zero, one, or several loans.

This shift helps represent reference assignment and control-flow-sensitive subset relationships. Instead of globally saying origin 'a is a subset of 'b, a relation can hold at a location and propagate along selected CFG edges.

Use the following bounded vocabulary:

• loan_issued_at(origin, loan, point) creates a loan in an origin.
• subset_base(origin1, origin2, point) states a local subset requirement.
• origin_live_on_entry(origin, point) says the origin matters at the point.
• cfg_edge(point1, point2) exposes control flow.
• loan_killed_at(loan, point) stops appropriate propagation.
• loan_invalidated_at(loan, point) records an incompatible access.

Names have varied across prototype eras and implementations. Treat this list as pedagogical vocabulary, not a guaranteed rustc API.

A simplified rule for subset transitivity is:

subset(O1, O3, P) :- subset(O1, O2, P), subset(O2, O3, P).

A simplified rule for loan membership is:

contains(O, L, P) :- loan_issued_at(O, L, P).
contains(O2, L, P) :- contains(O1, L, P), subset(O1, O2, P).

Propagation across an edge can be sketched as:

contains(O, L, Q) :-
    contains(O, L, P),
    cfg_edge(P, Q),
    origin_live_on_entry(O, Q),
    not loan_killed_at(L, P).

An error rule is then conceptually small:

error(L, P) :- loan_invalidated_at(L, P), live_loan(L, P).

Real algorithms need careful point conventions, kills, subset propagation, universals, placeholders, and initialization interactions. Negation and joins also affect evaluation strategy. The rules are a derivation aid, not a verbatim current implementation.

MIR + type checking
      │ emits base facts
      ▼
issued loans ─► origin membership ─► subset closure
      │                 │                 │
      └──── CFG edges + liveness + kills ─┘
                              │
                              ▼
                   live loan at each point
                              │ joins
                              ▼
                      invalidations/errors

The representation preserves provenance better than one undifferentiated region point set. It also risks creating a large product of points, origins, and loans. Production integration is therefore as much a sparse-graph and diagnostics problem as a semantics problem.

50. Deriving facts on a concrete branch#

Consider this abstract program:

P0: Oa receives loan Lx for x
P1: branch
P2: Ob = Oa
P3: write x
P4: use reference in Ob
edges: P0→P1, P1→P2, P1→P3, P2→P4, P3→exit

Base facts include:

loan_issued_at(Oa, Lx, P0)
subset_base(Oa, Ob, P2)
origin_live_on_entry(Ob, P4)
loan_invalidated_at(Lx, P3)
cfg_edge(P0, P1)
cfg_edge(P1, P2)
cfg_edge(P1, P3)
cfg_edge(P2, P4)

At P0, Oa contains Lx. Along the edge to P1, membership propagates while relevant. Along the P2 arm, subset transfers Lx into Ob. Ob is live at P4, so that arm retains the loan. The P3 arm does not flow into P4. Its invalidation need not conflict merely because another branch uses the loan.

This is the precision goal. A global approximation that unions both arms could report Lx live at P3 and reject. Location-sensitive propagation distinguishes them.

Now add edge P3→P4. The write occurs on a path that reaches the later use. The loan remains relevant at P3 and the invalidation becomes an error.

Prediction: add a kill at P2 before copying Oa to Ob. Should the assignment transfer Lx? No, if the kill semantically ends that loan before the subset event. Point ordering is critical: “on entry,” “midpoint,” and “after statement” conventions cannot be mixed.

Fact generation is part of the trusted semantic boundary. A perfect solver over missing invalidation facts is unsound. A perfect generator with a wrong propagation rule is also unsound. Differential testing should isolate both layers.

A useful audit table is:

51. Location-insensitive prepass and location-sensitive closure#

A location-insensitive analysis intentionally forgets where subset relations hold. It unions relationships over the body and computes a cheaper global closure. Because it forgets control-flow distinctions, it can overapproximate which loans may reach which origins.

That loss can be useful. If even the coarse graph proves a loan cannot reach an invalidation, the expensive analysis need not examine that pair. If the coarse graph finds a possible conflict, location-sensitive reasoning decides whether one realizable CFG flow witnesses it.

base origin/loan graph
       │ erase locations
       ▼
location-insensitive closure
       │ candidate loan-error pairs
       ▼
location-sensitive reachability on CFG
       │
       ├── witness path exists: report
       └── no witness: coarse false positive discarded

The prepass must be conservative. It may retain too many candidates. It must not discard a real error. That establishes a one-sided invariant suitable for optimization.

Location-sensitive reasoning can be viewed as reachability in a graph whose nodes combine origin state and program location. Subset edges transfer loan membership. CFG edges advance execution. Liveness gates propagation that cannot affect observations. Kills stop particular loan flows.

Materializing every combined node is expensive. Implementations can use sparse adjacency, strongly connected components, bitsets, interval compression, or demand-driven searches. Each optimization must preserve the same observation model: which invalidations can meet which live loans.

Demand-driven search starts from a suspected invalidation and asks whether the invalidated loan can reach it. Forward saturation starts from issuances and computes all reachable memberships. The first can save work when errors are rare and candidate sets are small. The second can share work among many queries. Real workload measurements decide.

Performance metrics should include:

• wall time on clean and erroring crates;
• peak memory;
• number of origins and loans;
• base and derived edge counts;
• largest strongly connected component;
• candidate pairs after the prepass;
• reachability queries and cache hit rate;
• diagnostic path reconstruction cost.

Microbenchmarks with acyclic toy graphs can hide loop-heavy worst cases. Representative crates, generated stress tests, and compiler-wide suites serve different purposes.

52. A bounded Polonius-style engine in stable Rust#

The final educational engine computes location-sensitive origin/loan membership. It supports issuance, same-point subset edges, CFG propagation, origin liveness, kills, and invalidations. Facts are integer identifiers to keep joins explicit.

The engine uses a work queue rather than a Datalog library. Each newly discovered tuple is processed once. This is semi-naive in spirit but not a general relational evaluator.

use std::collections::{BTreeMap, BTreeSet, VecDeque};

type Origin = u32;
type Loan = u32;
type Point = usize;

#[derive(Default)]
struct Facts {
	issued: Vec<(Origin, Loan, Point)>,
	subset: Vec<(Origin, Origin, Point)>,
	edges: Vec<(Point, Point)>,
	live: BTreeSet<(Origin, Point)>,
	killed: BTreeSet<(Loan, Point)>,
	invalidated: Vec<(Loan, Point)>,
}

#[derive(Debug, PartialEq, Eq)]
struct Error {
	loan: Loan,
	point: Point,
}

fn solve(facts: &Facts) -> (BTreeSet<(Origin, Loan, Point)>, Vec<Error>) {
	let mut subsets_at: BTreeMap<Point, Vec<(Origin, Origin)>> = BTreeMap::new();
	for &(from, to, point) in &facts.subset {
		subsets_at.entry(point).or_default().push((from, to));
	}
	let mut succ: BTreeMap<Point, Vec<Point>> = BTreeMap::new();
	for &(from, to) in &facts.edges {
		succ.entry(from).or_default().push(to);
	}

	let mut contains = BTreeSet::new();
	let mut queue = VecDeque::new();
	for &tuple in &facts.issued {
		if contains.insert(tuple) {
			queue.push_back(tuple);
		}
	}

	while let Some((origin, loan, point)) = queue.pop_front() {
		for &(from, to) in subsets_at.get(&point).into_iter().flatten() {
			if from == origin {
				let tuple = (to, loan, point);
				if contains.insert(tuple) {
					queue.push_back(tuple);
				}
			}
		}

		if !facts.killed.contains(&(loan, point)) {
			for &next in succ.get(&point).into_iter().flatten() {
				if facts.live.contains(&(origin, next)) {
					let tuple = (origin, loan, next);
					if contains.insert(tuple) {
						queue.push_back(tuple);
					}
				}
			}
		}
	}

	let live_loans: BTreeSet<(Loan, Point)> = contains
		.iter()
		.filter(|&&(origin, _, point)| facts.live.contains(&(origin, point)))
		.map(|&(_, loan, point)| (loan, point))
		.collect();
	let errors = facts
		.invalidated
		.iter()
		.filter(|&&(loan, point)| live_loans.contains(&(loan, point)))
		.map(|&(loan, point)| Error { loan, point })
		.collect();
	(contains, errors)
}

fn main() {
	let mut conflict = Facts::default();
	conflict.issued.push((0, 10, 0));
	conflict.edges.extend([(0, 1), (1, 2)]);
	conflict.live.extend([(0, 0), (0, 1), (0, 2)]);
	conflict.invalidated.push((10, 1));
	let (_, errors) = solve(&conflict);
	assert_eq!(errors, vec![Error { loan: 10, point: 1 }]);

	let mut transfer = Facts::default();
	transfer.issued.push((0, 20, 0));
	transfer.subset.push((0, 1, 0));
	transfer.edges.push((0, 1));
	transfer.live.extend([(0, 0), (1, 0), (1, 1)]);
	transfer.invalidated.push((20, 1));
	let (contains, errors) = solve(&transfer);
	assert!(contains.contains(&(1, 20, 0)));
	assert!(contains.contains(&(1, 20, 1)));
	assert_eq!(errors.len(), 1);

	let mut killed = Facts::default();
	killed.issued.push((0, 30, 0));
	killed.edges.push((0, 1));
	killed.live.extend([(0, 0), (0, 1)]);
	killed.killed.insert((30, 0));
	killed.invalidated.push((30, 1));
	let (_, errors) = solve(&killed);
	assert!(errors.is_empty());
}

Trace transfer. Issuance inserts (O0, L20, P0). The subset edge derives (O1, L20, P0). Because O1 is live at P1, the CFG edge derives (O1, L20, P1). Invalidation (L20, P1) meets a live membership and reports one error.

The liveness facts are supplied rather than inferred. Subset edges do not themselves propagate across CFG locations. There are no universal regions or placeholder loans. Kills use a simplified “after this point” convention. There is no place model; invalidations are assumed correct. There are no move paths, two-phase states, reborrows, drops, or diagnostics spans. The algorithm may revisit scans and is not tuned for large input.

These omissions define its bounded claim:

Given correct finite base facts in this toy convention, it computes the least membership closure generated by its two propagation rules and reports matching invalidations.

It is not a model of all Polonius rules and must not be used to decide Rust soundness.

Extension milestones:

1. Compute transitive subset closure at each point.
2. Propagate subset relationships along CFG edges while origins are live.
3. Record predecessor tuples for an explanation path.
4. Add location-insensitive candidate filtering.
5. Compare full saturation with demand-driven reverse search.
6. Generate random finite fact sets and compare both solvers.
7. Add point phases so kills and invalidations have unambiguous order.

53. Datafrog history and current in-tree integration#

The standalone Polonius project historically expressed analyses in Datalog-like rules and evaluated them with Datafrog, a lightweight dataflow engine. That work was invaluable for specifying relations, testing variants, and exploring precision. It should not be mistaken for a guarantee that current rustc simply invokes the historical repository unchanged.

The project vocabulary evolved. Older material often says “region” where newer explanations prefer “origin.” Rule names, fact schemas, algorithm variants, and command-line switches changed. When reading an issue, first identify its date, commit, and vocabulary.

For the rustc 1.97.1 source line, the generated nightly rustc documentation for rustc_borrowck::polonius is the appropriate revision-qualified implementation source. That module documents in-tree Polonius support organized around reachability rather than proving that the historical Datafrog engine is the default production path. Exact submodule names and flags must be checked against that release's source.

The safe status statement is:

In rustc 1.97.1's documented source line, Polonius-related implementation exists in rustc_borrowck; its presence does not by itself establish that every stable compilation uses a complete replacement checker.

Do not convert experimental integration into a stabilization date. Do not infer default status from one -Z option. Nightly flags are not stable interfaces and can be renamed or removed.

The 2023 Inside Rust Polonius update is a historical roadmap snapshot. It described a location-insensitive analysis and future integration work at that date. It is evidence of direction, not proof of 1.97.1 status. Current source and tests outrank that post for current implementation claims.

The rustc-dev-guide explains the established MIR borrow checker and NLL region inference. It may lag very recent implementation details. Use it for architecture, then inspect pinned source for exact data structures and call paths.

Migration has at least five dimensions:

1. Soundness: newly accepted programs must preserve Rust's safety invariants.
2. Compatibility: newly rejected programs need classification and transition policy.
3. Performance: time and memory must work on real crates and worst cases.
4. Diagnostics: errors need stable, comprehensible source explanations.
5. Maintenance: one source of semantic truth is preferable to indefinitely duplicated checkers.

A shadow or compare mode can run analyses and record mismatches without changing user-visible acceptance. Such modes need resource limits so comparison itself does not destabilize builds. Mismatch triage must distinguish intended precision gains, bugs, fact-generation differences, and diagnostic-only differences.

54. Diagnostics as proof reconstruction#

A useful borrow diagnostic reconstructs a short causal chain. It identifies where permission began. It identifies the incompatible access. It identifies why the permission still mattered. For escaping references, it identifies the boundary requiring a longer lifetime.

The first visible conflict is not always the first broken invariant. A later use can keep a loan live across an earlier mutation. The mutation receives the primary error, but the later use explains duration.

fn example() {
	let mut s = String::from("x");
	let r = &s;          // loan issued
	s.push('!');          // incompatible write
	println!("{r}");     // reason loan remains relevant
}

A three-span explanation follows the proof. Removing only the later use changes liveness. Changing the first borrow to a clone changes ownership. Moving the write changes control flow. The diagnostic should not reflexively recommend 'static.

For returned references, labels should distinguish input relationships.

fn wrong<'a, 'b>(x: &'a str, y: &'b str, flag: bool) -> &'a str {
	if flag { x } else { y }
}

The y arm needs 'b: 'a. That may be too strong for callers, so alternatives include returning a shorter common lifetime, owning the result, or changing API semantics. Adding a bound is not merely syntax repair; it exports a caller obligation.

Diagnostic provenance has memory and engineering costs. If a solver compresses a graph, it may discard the exact edge chain needed for an explanation. Possible strategies include retaining parent edges, replaying a bounded search, or recording only candidate witnesses. The choice is part of analysis design, not decoration after the fact.

For higher-ranked errors, identify binder boundaries and placeholder escape without drowning users in universe indices. For contributors, debug output can expose universe and constraint IDs. User diagnostics should translate those IDs into “must work for any lifetime” and “borrow escapes here.”

For async errors, mark the borrow, suspension point, and later use. For closure errors, mark capture, escape boundary, and required call lifetime. For dropck errors, mark where destruction may occur. For two-phase errors, distinguish reservation and activation when relevant.

55. Debugging workshops: find the earliest broken invariant#

Workshop A: mutation after observation#

Start with:

fn demo(v: &mut Vec<i32>) {
	let first = &v[0];
	v.push(2);
	println!("{first}");
}

Prediction: does spare capacity make this safe? No language guarantee ties push to preserving element addresses merely because capacity happens to be sufficient. The mutable access also conflicts with the shared borrow under the reference model.

Earliest invariant: a live shared loan of an indexed place overlaps the mutable receiver access. Experiment one: print before push; the loan can end earlier. Experiment two: copy the integer if it is Copy; no reference remains. Experiment three: store an index and re-index after mutation; bounds and element identity become algorithmic responsibilities.

Workshop B: partial move plus drop#

Start with a struct containing two String fields. Move one field and attempt to pass the entire struct to drop. The error is initialization, not primarily region inference.

Experiment one: use only the untouched sibling. Experiment two: destructure both fields. Experiment three: add a Drop implementation and observe that moving fields out becomes more restricted because destruction expects the whole value.

Earliest invariant: a move path required by whole-value use is not definitely initialized.

Workshop C: branch false positive#

Minimize to one issuance, one branch, one invalidation, and one use. Draw CFG edges rather than relying on indentation. Ask whether one path contains both invalidation and later use.

Dump or instrument base facts in a pinned compiler build. If the expected CFG edge is absent, investigate lowering. If invalidation is attached to the wrong loan, investigate place conflict or fact generation. If facts are right but closure is wrong, investigate reachability.

Workshop D: higher-ranked closure#

Compare a closure that accepts one particular borrowed value with a function required to accept for<'a> &'a T. Replace the closure with a free function to separate capture inference from binder reasoning. Make lifetime quantification explicit in a helper trait bound.

Earliest invariant may be placeholder containment, not local liveness. Inspect universes and normalization obligations before modifying MIR borrow checking.

Workshop E: async suspension#

Move the final reference use before .await. If the error disappears, cross-suspension liveness is central. Replace borrowed data with owned data to separate external validity from self-reference. Check Send errors independently from lifetime errors.

Earliest invariant could be referent lifetime, pin projection, or auto-trait eligibility. Do not collapse all three into “async borrow checker limitation.”

Workshop F: diagnostic regression#

Acceptance and rejection match, but labels move to generated code. The semantic solver may be correct. Trace source spans through MIR construction and constraint provenance. Compare primary span, secondary spans, suggestions, and emitted notes in UI tests.

Earliest broken invariant: explanatory provenance was discarded or mapped incorrectly.

56. Testing, soundness, security, and performance#

Borrow-checker testing needs both positive and negative cases. A compile-pass test prevents accidental rejection. A compile-fail test prevents accidental acceptance and checks diagnostics. Every regression should minimize syntax while preserving the relevant CFG, type, and place shape.

Unit tests suit graph closure and place overlap. UI tests suit source diagnostics. MIR-opt-style snapshots can expose lowering changes. Run-pass tests validate behavior but cannot prove aliasing soundness. Miri can detect many dynamic undefined-behavior manifestations in executed paths.

Differential testing can compare NLL and a Polonius mode. Classify each mismatch:

• intentionally newly accepted;
• intentionally newly rejected;
• likely fact-generation bug;
• likely solver bug;
• diagnostic-only change;
• timeout or memory regression;
• dependent on an unstable feature.

Never treat “new checker accepts more” as automatically correct. Construct an unsafe-code witness when possible, reason from the aliasing model, and request language-team review for semantic boundaries. Safe-code acceptance bugs can expose use-after-free or mutable-aliasing undefined behavior.

Adversarial graphs include long chains, dense subset cliques, deeply nested loops, many origins carrying one loan, and one origin carrying many loans. Generated tests should cap resources and record their seed. Sparse normal workloads and dense worst cases need separate baselines.

Caching reachability creates correctness obligations. The cache key must include every input affecting the answer. Incremental compilation must invalidate results when MIR, types, features, or relevant options change. A stale “no path” result is potentially unsound. A stale “path exists” result causes a false rejection.

Cancellation matters inside an interactive compiler or language server. Long closure computations should reach cancellation checks without leaving reusable partial state mislabeled as complete. Out-of-memory behavior should not silently fall back to a less sound analysis.

Observability should expose counts and timings without requiring enormous tuple dumps. Sensitive source paths and code may appear in debug facts, so telemetry and bug attachments need privacy review.

57. Source map for rustc 1.97.1 investigations#

Version qualification is mandatory here. Paths and APIs below describe where to begin for the rustc 1.97.1 source line; later nightlies may move them.

Start at the borrow-checking entry points in the rustc_borrowck compiler crate. Follow construction of borrow-check results and MIR traversal. Locate region inference code for constraint solving and universal-region handling. Locate borrow-set construction for issuance, borrow kind, and two-phase activation. Locate place-conflict code for overlap decisions. Locate move-data construction and initialization analyses in the relevant MIR dataflow crates.

For Polonius integration, begin with the documented rustc_borrowck::polonius module for that revision. Read its module-level documentation before following reachability types and call sites. Search callers rather than assuming an experimental path is active by default. Read tests adjacent to flags and modes to establish supported behavior.

For higher-ranked regions, follow universal-region construction, placeholder handling, and constraint checks. Trait solver code may generate or normalize obligations that feed borrow checking. Do not assign every universe failure to rustc_borrowck without tracing the obligation.

For closures and async, inspect capture analysis, coroutine transformation, and generated MIR before the borrow-check query. For drop checking and variance, inspect type-system modules and language definitions as well as borrow checking. These concerns cross crate boundaries.

Useful official and primary sources:

• rustc-dev-guide borrow checker: https://rustc-dev-guide.rust-lang.org/borrow-check.html
• rustc-dev-guide NLL region inference: https://rustc-dev-guide.rust-lang.org/borrow-check/region-inference.html
• RFC 2094, NLL: https://rust-lang.github.io/rfcs/2094-nll.html
• RFC 2025, nested method calls and two-phase borrowing context: https://rust-lang.github.io/rfcs/2025-nested-method-calls.html
• Rust Reference lifetime bounds: https://doc.rust-lang.org/reference/trait-bounds.html#lifetime-bounds
• Rust Reference higher-ranked bounds: https://doc.rust-lang.org/reference/trait-bounds.html#higher-ranked-trait-bounds
• Rust Reference closure types: https://doc.rust-lang.org/reference/types/closure.html
• standard-library Pin: https://doc.rust-lang.org/std/pin/
• rustc 1.97.1 nightly API Polonius module: https://doc.rust-lang.org/1.97.1/nightly-rustc/rustc_borrowck/polonius/index.html
• historical Polonius repository: https://github.com/rust-lang/polonius
• historical 2023 update: https://blog.rust-lang.org/inside-rust/2023/10/06/polonius-update.html

The nightly API URL is implementation documentation, not a stable user API. The RFCs record accepted designs and motivation; subsequent implementation details can differ. The Inside Rust post records status in October 2023. The standalone repository records prototype history.

A source-reading session should record:

1. exact rustc --version --verbose output;
2. repository commit;
3. enabled feature and -Z options;
4. call path proving which algorithm ran;
5. relevant test names;
6. any discrepancy with the dev guide.

Without those items, a claim about “current rustc” ages badly.

58. Contributor tasks and bounded implementation projects#

Task one: add projections to the NLL toy checker. Represent places as a local plus typed projection elements. Define overlap for equal fields, distinct fields, parents, dereferences, and indices. State where the model remains conservative.

Task two: add reference assignment. Show why binding one loan to one reference name fails at a branch merge. Translate assignments into subset facts and compare with direct copying.

Task three: implement two-phase state. Record reservation and activation points. Write tests for shared access during reservation, unique access during reservation, and access after activation. Document which real-rustc eligibility rules are omitted.

Task four: infer liveness facts for the Polonius toy engine. Compute uses and kills over a CFG. Separate origin liveness from local-variable liveness. Add a loop requiring several fixed-point iterations.

Task five: build a location-insensitive prepass. Erase points from subset relations. Prove that filtering is conservative. Property-test that every location-sensitive error remains a prepass candidate.

Task six: preserve explanation paths. For every derived membership, store one predecessor rule and tuple. Reconstruct a path from issuance to invalidation. Measure memory overhead on sparse and dense generated graphs.

Task seven: differential solver. Implement forward saturation and reverse demand-driven reachability. Generate random bounded facts. Assert equal error sets and minimize failures.

Task eight: rustc UI contribution. Find a labeled issue, pin the toolchain, and minimize the reproducer. Add the smallest test in the established suite. Run the targeted test before broad suites. Avoid mixing semantic changes with diagnostic rewrites.

Task nine: performance contribution. Add counters behind existing compiler instrumentation. Benchmark representative crates and an adversarial generator. Report wall time, peak memory, and graph sizes. Explain variance and retain raw commands.

Task ten: source archaeology. Compare one historical Datafrog rule with the 1.97.1 in-tree reachability implementation. Map concepts, not just names. Identify which rule is fused, precomputed, demanded lazily, or absent.

Review questions for every semantic patch:

• Which invariant changes?
• Is the change an acceptance gain or a safety restriction?
• Which MIR forms exercise it?
• Are unwind and cleanup edges covered?
• Are universal regions and placeholders affected?
• Does diagnostic provenance survive?
• What is the complexity bound?
• Which tests would fail if one relation direction were reversed?

59. Counterexamples that calibrate judgment#

“The reference is not used again” is insufficient if destruction observes it. Inspect drop behavior.

“The indices differ” is insufficient if the checker has no proof of inequality. Use an API that exposes disjointness.

“move owns everything” is false when a moved capture is itself a reference. Ownership of the pointer does not own its referent.

“Pin makes data live forever” is false. Pinning constrains movement, not deallocation or external lifetime validity.

“NLL ends borrows at the last source line” is false. It reasons over CFG points, including loops and generated operations.

“Polonius is just more path sensitivity” is incomplete. Its origin-and-loan formulation changes which relationships are propagated and can improve reasoning about subset relations.

“Datalog means slow” is not a useful conclusion. Evaluation strategy, sparsity, indexing, and demanded outputs determine cost.

“The prototype repository is current rustc” is false. Historical experiments and in-tree integration must be distinguished.

“A new acceptance is ergonomic only” is dangerous. Borrow checking guards memory-safety assumptions used by unsafe code and optimization.

“A compiler error proves the program is unsafe” is false. Conservative analyses reject some safe programs they cannot establish.

“Unsafe can fix the borrow checker” reverses responsibility. Unsafe code replaces a compiler proof with a human proof and should sit behind a safe abstraction with documented invariants.

Prediction exercise: two disjoint fields are mutably borrowed. Usually accepted because place projections prove disjointness. Replace fields with runtime indices. Usually rejected because equality is unknown. Replace the slice with two outputs from split_at_mut. Accepted because the API encodes a checked partition.

Prediction exercise: a shared reborrow is printed, then the parent mutable reference is used. Accepted when the shared reborrow is no longer live. Move the print after the parent use. Rejected because both permissions matter across the conflict.

Prediction exercise: a reference is used before .await only. It need not cross suspension. Use it after .await and its referent must satisfy the cross-suspension validity requirement.

60. Philosophy derived from the three stages#

Representations determine which questions are cheap. Lexical scope trees make containment cheap but path shapes expensive. MIR point sets make CFG liveness natural but can blur loan provenance. Origins containing loans make provenance and subset flow explicit but enlarge the relational state space.

Abstractions move work rather than deleting it. MIR simplifies checker cases by moving source reconstruction into diagnostics. split_at_mut simplifies caller proofs by moving arithmetic and unsafe reasoning into one reviewed implementation. Pinning simplifies users of self-referential futures by imposing contracts on projection and movement.

Precision is preserved information. Lexical checking forgets intra-scope death. Location-insensitive Polonius forgets where relations hold. Each forgotten distinction can create a false positive. Recovering precision costs memory, time, or implementation complexity.

Optimization requires an observation model. A borrow solver need not preserve every intermediate tuple. It must preserve acceptance, rejection, and enough provenance for required diagnostics under the intended semantics. A location-insensitive prepass is valid only because its one-sided approximation cannot hide real candidates.

Identity is not an address or source line. A loan has semantic identity even if reference values move. A MIR point can correspond to generated operations sharing one source span. A pinned value's semantic immovability is governed by contracts, not merely its current numeric address.

Caching creates proof obligations. Memoized reachability is correct only for the exact graph, liveness, and kill inputs in its key. Incremental reuse must honor dependency changes. Performance state can become semantic state when stale negatives suppress errors.

Diagnostics depend on provenance. A compact closure answers yes or no. A teacher-quality error needs why. Discarding witness edges locally can create global explanatory complexity.

The first visible failure may be late. A mutation error can originate in later liveness. A borrow error can originate in trait normalization. An async error can originate in capture layout. Debugging improves when it seeks the earliest broken invariant rather than the loudest span.

Uncertainty must be represented, not guessed away. Unknown index equality becomes conservative overlap. Unknown implementation status becomes a revision-qualified statement. Unknown semantic impact becomes a differential test and review request.

61. Three stage-talk outlines#

Talk one: lexical lifetimes and the problem to solve#

Open with one owner, one place, one shared loan, and one write. Draw spatial overlap before mentioning lifetime syntax. Ask the audience which facts make the write unsafe.

Introduce lexical blocks as the first finite approximation. Run the complete lexical checker. Show one accepted conflict-free trace and one false rejection after final use. End with a branch whose valid region is not one source interval.

Audience prediction: move the mutation to an arm with no reference use. Collect votes before revealing the CFG.

Closing invariant: compatible accesses require both place and time reasoning.

Talk two: MIR NLL as a fixed-point proof#

Begin with the branch CFG from talk one. Define points, successors, uses, and kills. Compute live_in and live_out on paper.

Add an outlives edge and propagate one point between region sets. Introduce universal regions only after local inference works. Use a higher-ranked function to motivate universes and placeholder leaks.

Overlay move paths and loans. Animate a reborrow. Animate reservation and activation for v.push(v.len()). Show how drop and .await create less obvious later requirements.

Run the stable-Rust CFG checker. End by assigning one reference to another and exposing its missing representation.

Talk three: Polonius and production integration#

Begin with origins as boxes containing loan IDs. Derive issuance, subset, CFG propagation, kill, and error rules from one trace. Do not begin with Datalog syntax.

Erase locations to construct the prepass. Ask what information was lost and why the result remains useful. Restore locations with a reachability query.

Run the bounded engine. Show one membership tuple's derivation path. List omissions before comparing it with rustc.

Separate three status slides: historical Datafrog prototype; October 2023 roadmap snapshot; rustc 1.97.1 in-tree documented support. Never announce an inferred stabilization schedule.

Finish with performance counters, diagnostic provenance, and a contributor-sized task.

62. Mastery path and final exercises#

Week one: ownership, places, and accesses. Draw move-path trees for structs, tuples, and enums. Classify shared, unique, move, write, and drop conflicts. Implement projected-place overlap in the lexical checker.

Week two: CFG dataflow. Compute liveness on branches and loops by hand. Implement a work-list solver instead of whole-vector rescans. Prove termination for finite sets.

Week three: region constraints. Translate 'a: 'b into set inclusion without reversing it. Trace local and universal regions. Explain one higher-ranked placeholder leak in plain English.

Week four: production features. Trace reborrows and two-phase activation. Build examples involving Drop, closure capture, and .await. For each, identify the point that keeps a loan relevant.

Week five: Polonius facts. Derive base facts from ten-line abstract MIR. Compute subset and membership closure manually. Implement explanation predecessors in the bounded engine.

Week six: source reading. Pin rustc 1.97.1 or a recorded commit. Trace one test from source through MIR and borrow checking. Record module paths and distinguish documentation from observed implementation.

Week seven: performance and diagnostics. Generate sparse and dense graphs. Compare saturation and demand-driven reachability. Reconstruct a user-facing three-span explanation from a witness path.

Capstone A: add point phases, subset propagation, two-phase loans, and projected invalidations to the toy engine. State all omissions. Differentially test two independent solvers.

Capstone B: minimize a real borrow-checker issue. Draw its MIR-level control flow. Identify whether the likely layer is lowering, facts, closure, conflict, or diagnostics. Submit a test-only patch before proposing semantics.

Capstone C: teach the three talks to different audiences. For beginners, omit universes but preserve truth. For Rust users, emphasize repair choices and API design. For compiler contributors, show constraints, facts, and source paths.

Final prediction questions:

1. Which analysis rejects reading a partially moved parent?
2. Which fact keeps a shared loan relevant after issuance?
3. Why can a destructor extend a validity requirement?
4. Why is textual last use wrong in a loop?
5. What does a universe prevent an inference variable from naming?
6. Why can an origin contain several loans?
7. What information does a location-insensitive closure forget?
8. Why must its candidate filter overapproximate?
9. How does pinning differ from lifetime extension?
10. What evidence establishes a rustc algorithm's default status?

Answers:

1. Move/init dataflow establishes whether the required move path is initialized.
2. Reference liveness and propagated origin membership connect the loan to later use.
3. Drop::drop may inspect fields when destruction runs.
4. A later execution iteration can revisit an earlier source line.
5. A placeholder introduced in a universe the variable cannot access.
6. Assignments and subset flow can merge provenance from different loans.
7. The CFG locations at which subset and membership relationships hold.
8. Dropping a real candidate could suppress a real error and become unsound.
9. Pin constrains movement; a lifetime constrains validity.
10. Pinned source, call paths, options, and tests—not a roadmap or module's existence alone.

The durable mental model is now complete. Ownership controls destruction responsibility. Places locate accesses. Move paths track initialization. Loans encode permissions. NLL derives temporal requirements over MIR control flow. Universes protect quantifier boundaries. Polonius propagates loan provenance through origins and locations. Production rustc must preserve all of that while remaining fast, diagnosable, and migratable.

63. Full traces: from source place to failed constraint#

The fastest route to contributor fluency is tracing one tiny body through every representation. Begin with a shared borrow that survives a mutation.

fn trace(input: &mut String, flag: bool) {
	let view = &*input;
	if flag {
		input.push('!');
	}
	println!("{view}");
}

At source level, input is a mutable reference value. Dereferencing it produces a place for the caller-owned String. &*input performs a shared reborrow. The local view carries the reborrow's inferred region.

At MIR level, the exact statements are revision-dependent. Conceptually there is an assignment creating a shared reference, a conditional branch, a mutable receiver borrow for push, a call, a join, and a use of view. Calls also have unwind behavior unless proven otherwise by the relevant stage.

The move analysis establishes that the reference locals remain initialized. No owned String moves out through input. This removes one competing explanation for the failure.

Liveness seeds the region carried by view at the final formatting use. Backward propagation reaches the join and both incoming paths. The shared loan therefore matters on the true arm before the join.

Borrow gathering records a shared loan for *input. The call to push requires mutable access to overlapping data. The place comparison cannot call the shared and mutable targets disjoint: both route through input to the same String.

The compatibility matrix rejects mutable access while the shared loan matters. The primary diagnostic belongs on the attempted mutable access. The issuance span explains where the restriction began. The print span explains why it remained live.

Move println! before the branch. The liveness seed no longer propagates across the branch. The place and access facts are otherwise similar. The changed temporal fact removes the conflict.

Now trace a returned reference.

fn select<'a, 'b>(left: &'a str, right: &'b str, pick_left: bool) -> &'a str {
	if pick_left { left } else { right }
}

The left arm satisfies the declared output directly. The right arm attempts to coerce &'b str to &'a str. Covariance permits that coercion only when 'b: 'a. No body-local liveness choice can manufacture this caller-facing relationship.

The failed invariant is a universal-region outlives requirement. Possible repairs express different policies. Add 'b: 'a to constrain callers. Return a reference with a lifetime bounded by both inputs. Return owned data to remove the borrowing relationship. Remove the branch if the API should always return left.

Now trace a loop-carried reference.

fn loop_trace(values: &mut Vec<i32>) {
	let first = &values[0];
	for _ in 0..2 {
		println!("{first}");
	}
	values.clear();
}

The CFG has a back edge from the loop body toward its header. Liveness reaches a fixed point only after information traverses that edge. The loan remains relevant throughout loop execution. After the loop exits, no later use requires first, so clear can proceed.

Move clear into the loop before the print. One iteration contains both invalidation and use. The conflict is real even though the source has only one textual print. This is the canonical refutation of line-number lifetimes.

Finally trace a destructor.

struct ShowOnDrop<'a>(&'a str);

impl Drop for ShowOnDrop<'_> {
	fn drop(&mut self) {
		println!("{}", self.0);
	}
}

fn drop_trace() {
	let owner = String::from("owned");
	let guard = ShowOnDrop(&owner);
	drop(guard);
	drop(owner);
}

The explicit drop(guard) is a use that moves guard into drop and runs its destructor. The destructor may read the borrowed string during that call. Only after it completes can owner be destroyed. Reversing the two explicit drops would violate the required validity and is rejected.

The trace habit is always the same:

1. Write the source-level ownership story.
2. Draw places and projections.
3. Draw CFG edges, including loops and cleanup when relevant.
4. Separate initialization from region requirements.
5. Locate liveness seeds and outlives edges.
6. Locate loan issuance, activation, kills, and invalidations.
7. Find the first incompatible join.
8. Reconstruct source spans only after the proof is understood.

64. Fact-generation lab: assignment, reborrow, and merge#

Origin reasoning becomes useful when reference provenance moves. Consider three source references flowing into one destination.

P0: ra = &x            issue Lx into Oa
P1: rb = &y            issue Ly into Ob
P2: branch flag
P3: out = ra           subset Oa → Oout at P3
P4: out = rb           subset Ob → Oout at P4
P5: use out            Oout live

At P5, Oout may contain Lx or Ly depending on the path. A path-insensitive union says it can contain both. That is safe but can be imprecise for invalidations confined to one arm. Location-sensitive propagation preserves which transfer occurred on which edge.

Add write x on the rb arm before P4. That write should not conflict with Lx merely because the other arm transfers ra. There is no execution where out receives ra and the rb-arm write executes.

Add a join assignment after the branch that always copies ra. Now both incoming paths reach a later use carrying Lx. The rb-arm write to x conflicts because that path subsequently transfers and uses ra.

This distinction requires more than origin membership at the final point. It requires a witness path connecting issuance, transfer, invalidation, and use in a compatible order.

Reborrowing introduces a parent-child relationship.

P0: parent = &mut x     issue unique Lparent into Oparent
P1: child = &mut *parent
P2: use child
P3: use parent

The child access must be authorized by the parent reference. While the child unique reborrow is active, competing use through the parent is restricted. After the child is dead, use through the parent resumes.

A naive fact generator might issue unrelated unique loans for x at P0 and P1 and report an immediate conflict. A correct model recognizes the second as a reborrow through the existing permission. It still prevents independent aliases from using the parent route concurrently.

Shared reborrows change compatibility but not referent ownership.

P0: parent = &mut x
P1: child = &*parent
P2: read child
P3: write parent

At P2, shared observation is authorized. At P3, the shared child must no longer matter before unique mutation resumes. If another edge reaches a later child use, that edge can retain the restriction.

Two-phase facts add an activation event.

reserve(L, Preserve)
activate(L, Pcall)

Before Pcall, the reserved loan follows reservation compatibility. At and after Pcall, active unique compatibility applies. If control bypasses the call, activation should not be imagined on that path.

Kills need semantic care. Overwriting a reference local can kill provenance carried only by that local. It does not necessarily kill the same loan in another origin reached by an earlier copy. Killing a loan globally because one carrier died can be unsound.

P0: r1 = &x
P1: r2 = r1
P2: overwrite r1
P3: use r2

Lx remains observable through r2. The model must distinguish local carrier death from elimination of every path carrying the loan.

Fact-generation review questions:

• Is each access attached to the correct MIR point phase?
• Is place overlap decided before invalidation facts are emitted?
• Does a reference assignment create the subset direction intended?
• Are subset facts restricted to realizable control-flow edges?
• Is a kill local to one carrier or global to a loan?
• Does reborrow preserve parent authorization?
• Is two-phase activation path correct?
• Are universal and placeholder origins distinguishable?
• Are unwind destinations included where values remain observable?

Property tests can mutate one fact at a time. Reverse a subset edge and require a known test to fail. Remove a CFG edge and require reachability to shrink. Add a kill and require membership after it to disappear only where specified. Duplicate a tuple and require the fixed point to remain unchanged. Permute input fact order and require deterministic errors.

65. Production migration plan without semantic wishful thinking#

A checker migration begins with a written equivalence boundary. List behavior intended to remain identical. List known precision gains. List unresolved cases. List unstable features outside the initial boundary.

Phase one validates base facts. Run fact generation without changing acceptance. Count loans, origins, subset edges, liveness facts, kills, and invalidations. Investigate impossible counts and pathological growth.

Phase two runs a location-insensitive prepass in shadow mode. Assert that established errors remain candidates. Measure candidate reduction and memory. Do not report prepass false positives to users.

Phase three runs location-sensitive reasoning for selected bodies. Use deterministic sampling or explicit options. Record semantic mismatches separately from timeouts. Bound memory and preserve cancellation.

Phase four compares diagnostics. Acceptance equality is insufficient. Compare primary spans, labels, notes, suggestions, and delayed bugs. Retain witness paths for mismatches.

Phase five broadens ecosystem testing. Compiler suites cover designed cases. Crater-like runs cover accidental language patterns. Large internal codebases expose performance shapes. Fuzzers discover small relational corner cases.

Phase six considers default changes only with explicit governance. Language and types teams evaluate semantic differences. Compiler teams evaluate maintainability and performance. Diagnostics owners evaluate user impact. Release processes evaluate rollback.

Rollback must preserve soundness. A resource failure cannot quietly select a known-less-sound answer. Fallback policy must be designed and tested before rollout. Telemetry should distinguish fallback from successful completion.

Performance budgets should be distributions, not one average. Track median, high percentiles, and worst observed bodies. Separate clean compilation, incremental compilation, and error-heavy editing. Track peak resident memory as well as allocated tuples.

Migration risks include:

No benchmark proves soundness. No proof sketch proves acceptable compile time. No test suite proves diagnostic quality for every user. Production readiness is the intersection of these evidence streams.

Revision-qualified release notes should say exactly which paths changed. They should not describe experimental internals as language guarantees. They should provide minimized examples of newly accepted or rejected code. They should state how to report regressions with version output.

66. Audit checklist and reading map#

Audit the document itself before trusting its model. There must be exactly one H1. H2 numbering must continue from the preceding part. Every fence must be balanced and use a renderer-supported tag. Every complete program must compile on stable Rust. Every intentionally rejected Rust fragment must be described as rejected.

Audit the lexical checker. It must reject a write during a shared lexical loan. It must reject a read during a unique lexical loan. It must allow a write after the supplied lexical end. Its omissions must prevent readers from treating it as Rust semantics.

Audit the NLL checker. Its graph must include all successor indices. Its liveness iteration must terminate on finite sets. Its branch test must accept an invalidation isolated from the use arm. Its linear conflict test must reject mutation before a later use. Its unique-loan test must reject a direct shared read.

Audit the Polonius-style engine. Issuance must seed membership. Subset must transfer a loan in the documented direction. CFG propagation must require destination-origin liveness. A kill must stop the simplified edge propagation. Invalidation must report only a live matching loan. The omissions must explicitly deny equivalence with rustc.

Audit technical language. A reference is not a loan. A region is not runtime duration. An origin is not a memory allocation. Outlives direction matches set inclusion. Pinning is not lifetime extension. Move paths are not byte-level alias analysis. Location-insensitive means conservative loss of location information, not random imprecision.

Audit version claims. Every rustc 1.97.1 implementation claim names that source line. Nightly API documentation is labeled unstable implementation documentation. The October 2023 post is labeled historical. The standalone Datafrog repository is labeled prototype history. No stabilization date is inferred.

Reading order for a new Rust programmer:

1. Chapters 39–41 for memory, ownership, places, accesses, and lexical checking.
2. Chapters 42–48 for CFGs, NLL, regions, loans, and the NLL checker.
3. Chapters 46–47 again for advanced user-facing cases.
4. Chapter 54 for interpreting diagnostics.
5. Chapter 59 for counterexamples.

Reading order for a compiler student:

1. Chapters 42–44 for the MIR proof pipeline.
2. Chapter 43 for constraints, universes, and placeholders.
3. Chapters 48–52 for executable progression.
4. Chapters 50–51 and 64 for relational traces.
5. Chapters 55–56 for debugging and validation.

Reading order for a rustc contributor:

1. Chapter 57 with a pinned checkout.
2. Chapters 63–64 while tracing one UI test.
3. Chapter 53 for historical/current status separation.
4. Chapters 56 and 65 for evidence and rollout.
5. Chapter 58 for bounded first contributions.

The final self-test is explanatory, not terminological. Can you explain why one path needs a loan and another does not? Can you state which information each analysis forgets? Can you distinguish a move error from a loan conflict? Can you show the direction of an outlives constraint? Can you explain why a placeholder cannot leak? Can you derive an origin-membership tuple from base facts? Can you locate the earliest broken invariant in rustc source? Can you qualify implementation claims by revision?

If those answers are precise, the three stages form one continuous idea. Lexical checking used source scopes as time. NLL uses MIR control-flow points as time. Polonius uses location-sensitive origin-and-loan reachability to preserve more provenance through that time. The safety invariant remains stable even as its representation improves.

Part VII: Monomorphization, Code Generation, LLVM, and Linking#

1. The backend boundary#

The compiler backend is the part that turns a checked, mostly language-level program into target-specific artifacts. An artifact is an output file or in-memory product, such as object code, LLVM bitcode, assembly, or metadata. By this point rustc has already established that ordinary type and ownership rules hold. The backend must preserve those conclusions while choosing concrete representations and machine operations.

There is no perfectly sharp boundary mandated by the Rust language. In current rustc, monomorphization, layout computation, ABI classification, MIR preparation, LLVM IR construction, optimization, and object emission span that boundary. These internals are version-sensitive: names, ordering, and ownership of individual steps can change between compiler versions. The useful stable picture is a pipeline of progressively more concrete commitments.

checked generic MIR
        |
        v
reachable concrete work
        |
        v
prepared monomorphic MIR + layouts + ABI
        |
        v
codegen units
        |
        v
LLVM IR -> optimized machine code -> object files
        |
        v
linker -> executable or library

“Checked” does not mean “ready to execute.” A generic function may still mention T, while a processor needs instructions for a concrete representation. A call may name a trait method, while emitted code needs a direct address or a virtual-table lookup. An enum may be logically one of several variants, while memory needs a byte layout.

The backend therefore answers three broad questions. First, which concrete functions, statics, and compiler-generated helpers can the final artifact need? Second, how are their values and calls represented for the selected target? Third, how is that work divided and handed to lower-level tools?

The target is the execution environment described by a target specification. It includes an architecture, operating-system conventions, pointer width, object format, and calling conventions. Compiling the same crate for x86-64 Linux and WebAssembly can produce different layouts, ABIs, and instructions without changing Rust source semantics.

Language guarantees and implementation choices must remain distinct. Rust guarantees, for example, that a reference points to a suitably aligned valid value under its contract. Rust generally does not guarantee the field offsets of an ordinary repr(Rust) struct. Current rustc chooses an offset arrangement, but unsafe code may not treat that choice as a portable promise.

Backend bugs often masquerade as source bugs because their products run later. A wrong type error usually belongs earlier than this chapter's scope. A program accepted but misbehaving only with optimization, only on one target, or only across a foreign call points strongly toward lowering, layout, ABI, or LLVM interaction. The backend's core invariant is semantic preservation: every concrete artifact must implement the behavior allowed by the checked program and its unsafe-code contracts.

2. From checked MIR to artifacts#

MIR, the Mid-level Intermediate Representation, is rustc's control-flow-oriented representation of function bodies. Generic MIR can describe one source function for many substitutions. The backend usually consumes a prepared body associated with a concrete Instance, which will be defined precisely later.

Consider a generic identity function.

fn identity<T>(value: T) -> T {
    value
}

fn main() {
    let a = identity(10_u32);
    let b = identity(String::from("backend"));
    println!("{a} {b}");
}

The source contains one identity, but the uses require behavior for u32 and String. Those types differ in size, move behavior, and drop requirements. Current rustc normally creates concrete work for each used substitution, although optimization may later merge or erase equivalent code.

The journey is not simply “translate every MIR statement.” The compiler first discovers required mono items. It resolves generic operations into concrete callees and generated helpers. It computes layouts and function ABIs. It prepares MIR so code generation sees explicit, legal operations. It partitions work into codegen units and emits backend IR.

Several output modes stop at different points.

Requested output     Typical meaning
metadata             Rust information needed by dependent crates
llvm-bc              LLVM bitcode
llvm-ir              textual LLVM intermediate representation
asm                  target assembly
obj                  relocatable object file
link                  final executable, static library, or shared library

A relocatable object contains machine code and data whose final addresses may be unknown. A relocation is a record asking the linker to fill in an address or offset later. A symbol is a linker-visible name associated with code or data. Symbol naming must distinguish concrete instantiations and avoid accidental collisions.

Not every checked body produces a standalone symbol. Inlining can place a callee's operations inside callers. Some zero-sized operations need no instructions. Unused generic definitions may produce no machine code at all. Conversely, one source expression can require extra symbols for drop glue, closures, or vtables.

The important invariant is demand-driven concreteness. Every emitted operation must have enough type, layout, and ABI information for the target, and every operation reachable at runtime must have an implementation. Emitting too little causes missing symbols or unimplemented paths. Emitting too much wastes compile time, memory, and binary size.

When diagnosing a missing artifact, ask where the item disappeared. Was it absent from reachability roots, omitted during mono-item discovery, assigned incorrectly during partitioning, declared with the wrong linkage, or removed by optimization? This staged question is more productive than blaming “LLVM” immediately.

3. Reachability begins with roots#

Reachability asks which program entities might be needed from a chosen starting set. A root is an entity treated as needed without first discovering a caller inside the same analysis. The collector walks dependencies outward from roots rather than generating every possible generic combination.

An executable's entry path is an obvious source of roots. Exported functions can also be roots because unknown code outside the crate may call them. Public visibility alone does not always imply final binary reachability; crate type, linkage, attributes, and downstream use matter. Statics retained for external or platform reasons may likewise seed collection.

pub fn library_api(x: u32) -> u32 {
    helper(x) + 1
}

fn helper(x: u32) -> u32 {
    x * 2
}

fn never_called() {
    println!("possibly no code emitted");
}

If library_api must be available, its call makes helper reachable. Nothing in this fragment requires never_called. This is a graph problem: items are nodes and “requires” relationships are directed edges.

Function pointers complicate the graph. Taking a function's address requires an addressable body even if no direct call appears. Dynamic dispatch complicates it further because the exact method is selected through a vtable at runtime. The vtable itself names concrete method implementations, so constructing that vtable exposes those dependencies.

Inline assembly, externally imposed entry points, and special runtime hooks can create edges not visible as ordinary MIR calls. Compiler attributes may request retention or export. Such mechanisms are dangerous areas for reachability bugs because an omitted edge can survive checking and fail only during linking or execution.

Whole-program reachability is limited by compilation boundaries. A library is compiled without knowing every future consumer. Rust metadata and generic MIR permit downstream crates to instantiate generic definitions. Non-generic externally usable code may need an exported body even when the current crate has no local caller.

The conservative alternative is to retain everything. That is simple but increases compile time and binary size, and it is impossible for unrestricted generics because there are infinitely many conceivable type arguments. The aggressive alternative is to retain only provably called code. That risks breaking dynamic or external uses if the model misses an edge. Rustc balances these concerns with crate-aware roots and explicit dependency discovery.

A sound collection is allowed to over-approximate. It may include an item that execution never reaches. It must not under-approximate by excluding an item that execution can require. Later dead-code elimination can remove excess work, but it cannot invent a body that was never generated.

For an undefined-symbol failure, inspect whether the missing symbol should have been a root or a dependency. For suspicious binary growth, inspect unexpectedly broad roots, address-taken functions, and generated vtables before studying instruction selection.

4. Mono items: units of concrete work#

A mono item is a concrete entity selected for code generation after generic parameters have been fixed. The term is short for monomorphized item, though some mono items were never generic in source. Current rustc broadly deals with concrete function instances, statics, and global assembly as collection units.

Source entity                     Possible concrete work
fn parse<T>(T)                    parse::<u8>, parse::<String>
ordinary fn start()               start
closure expression                a concrete closure call body
static TABLE: [u8; 4]             storage and initialization data
trait default method              an instance for a selected Self type
drop of Vec<String>               generated drop glue

Collection traverses the prepared meaning of reachable bodies. A direct call contributes its resolved callee. Creating a constant may require an allocation or static. Coercing a concrete reference to a trait object may require a vtable. Dropping a value may require type-specific glue.

The collector cannot merely search source syntax. One generic call expression can resolve differently after substitution. Compiler-inserted operations have no direct source function name. Constants can contain pointers to functions or allocations. Collection therefore uses typed, concrete information.

fn call_twice<T, F: Fn(&T)>(value: &T, f: F) {
    f(value);
    f(value);
}

fn use_it() {
    let n = 3_u64;
    call_twice(&n, |x| println!("{x}"));
}

This use can require an instance of call_twice specialized for u64 and the unique closure type. It also requires a closure call implementation and dependencies reached through formatting and printing. The graph quickly becomes larger than the visible user functions.

Recursive functions need cycle handling. When collection sees an item already being visited or already collected, it must not recursively expand forever. Memoizing discovered nodes turns recursive calls into graph cycles rather than compiler recursion disasters.

Polymorphic recursion is a harder pattern in which recursion changes type arguments each time. Unbounded forms imply infinitely many instantiations and cannot yield finite machine code through ordinary monomorphization. Rust's type system and compiler checks reject or constrain problematic constructions, but compiler developers must still guard collection against explosive instance growth.

Collection order should not determine semantics. Deterministic ordering is nevertheless valuable for reproducible builds, stable diagnostics, and incremental reuse. Hash iteration or unstable symbol construction can create output differences even when behavior remains correct.

If collection crashes on a generic program, reduce it to the edge that introduces a new concrete item. If only one substitution fails, inspect substituted types and instance resolution. If all substitutions fail, inspect the shared MIR operation or collection rule.

5. Why monomorphize#

Monomorphization creates a specialized implementation of generic code for concrete type arguments. “Mono” means one shape: inside that implementation, sizes, alignments, methods, and drop behavior are known. This strategy makes Rust's zero-cost abstractions practical.

fn maximum<T: Ord + Copy>(a: T, b: T) -> T {
    if a > b { a } else { b }
}

let x = maximum(2_i32, 9_i32);
let y = maximum('a', 'z');

The i32 version can compare integer registers using the selected Ord implementation. The char version uses char's representation and implementation. No universal runtime box or type tag is required merely because the source was generic.

Specialization enables optimization. The backend can inline a concrete method, remove a drop that is known to do nothing, propagate a constant size, or vectorize operations over a known element type. The cost is duplication. Many type combinations increase compile time and executable size, a phenomenon called monomorphization bloat.

One alternative is type erasure. Code can operate through pointers and runtime descriptors, as trait objects do for selected interfaces. This shares machine code but introduces indirection, restricts available operations, and may inhibit optimization. Another alternative is boxing all values into a uniform representation. That simplifies calling conventions but adds allocation or tagging costs and abandons Rust's direct value model.

A third alternative is dictionary passing. A dictionary is a runtime record of operations required by generic constraints. The generic function receives that record and uses indirect calls. Some languages compile generics this way. Rust uses related machinery for dynamic trait objects, but ordinary statically dispatched generics are generally monomorphized.

Hybrid strategies are possible. A compiler can share code between types with identical representations, outline common operations, or merge equivalent functions after emission. Current optimizer decisions are not language guarantees. Program correctness cannot depend on two instantiations having equal or unequal addresses unless an API explicitly promises something relevant.

Code-size-sensitive users can move non-generic work into ordinary helper functions. They can choose trait objects at deliberate abstraction boundaries. They can inspect symbols and size reports to find multiplying instantiations. The correct tradeoff depends on hot-path speed, build latency, deployment limits, and API design.

The monomorphization invariant is substitution completeness. No unresolved type or const parameter may reach an operation that needs concrete layout or behavior. A compiler error mentioning “polymorphization,” an unexpected generic parameter, or inability to normalize a type near codegen suggests that an earlier resolution step left an illegal abstraction behind.

6. GenericArgs: filling generic parameters#

GenericArgs is current rustc terminology for the ordered arguments applied to a generic definition. An argument can represent a lifetime, a type, or a compile-time constant. The exact internal APIs are version-sensitive, but substitution as a concept is fundamental.

struct Buffer<'a, T, const N: usize> {
    slice: &'a [T; N],
}

fn first<'a, T, const N: usize>(b: Buffer<'a, T, N>) -> &'a T {
    &b.slice[0]
}

An application might provide a particular region, u16, and 8. Lifetimes are normally erased before machine code because they guide static validity rather than runtime representation. They still occupy places in generic argument structures used while types are transformed. Types and constants can directly affect layout and generated operations.

Arguments are positional relative to a definition's generic parameter list, including parameters inherited from an enclosing item. An associated method may need arguments for its trait, its implementing type, and the method itself. Confusing these layers can substitute a plausible but wrong type, making such bugs subtle.

Substitution replaces parameters throughout a type or body. [T; N] becomes [u16; 8]. &'a T becomes a reference carrying the selected compile-time region and pointee type. Associated types may then require normalization, meaning reduction to the concrete type chosen by a trait implementation.

Definition:  fn f<T, U, const N: usize>(x: T) -> [U; N]
Arguments:              String, u8,             4
Resulting type: fn(String) -> [u8; 4]

Not every parameter affects generated code. A lifetime commonly does not. A type used only in a marker such as PhantomData<T> may affect static reasoning without changing bits. Rustc can sometimes avoid multiplying code by irrelevant parameters, but this is an optimization detail and must preserve symbol and semantic requirements.

Const arguments deserve care. They may need evaluation before layout, and two syntactically different constants may evaluate to the same value. The compiler needs a canonical enough identity for caching and symbol naming while respecting cases that cannot yet be evaluated.

Key invariants are correct arity, correct parameter order, well-formed substituted types, and sufficient normalization before concrete use. An internal mismatch in argument count points toward construction of the GenericArgs list. A layout failure containing a bare T points toward incomplete substitution. A wrong associated type points toward trait selection or normalization rather than raw instruction emission.

7. Instance resolution#

An Instance identifies executable behavior for a definition under concrete generic arguments. It is more precise than “function plus types” because the executable body may be a compiler-generated shim or a selected trait implementation. Think of it as the answer to “what exactly runs for this callable thing here?”

trait Render {
    fn render(&self) -> String;
}

impl Render for u32 {
    fn render(&self) -> String {
        self.to_string()
    }
}

fn show<T: Render>(x: &T) -> String {
    x.render()
}

For show::<u32>, resolution can identify the u32 implementation of Render::render. The resulting instance records enough information to find or synthesize its body and determine its symbol. The generic declaration alone is not a callable machine-code destination.

Instances also represent intrinsic operations, drop glue, closure adapters, function-pointer shims, and other special forms in current rustc. An intrinsic is an operation known specially to the compiler rather than implemented as an ordinary Rust body. Some instances lower directly to backend operations; others use MIR bodies.

Resolution must happen under an environment of proven assumptions. A generic definition may be checked assuming T: Render. At a concrete use, rustc selects the implementation justified by u32: Render. Selection must agree with type checking; code generation must not silently choose a different method.

An instance may be impossible to instantiate. For example, a default trait method can contain operations whose additional requirements are not met for a supposed concrete use. Normally earlier checking and reachability prevent invalid instances from arriving at codegen. If one does arrive, an internal compiler error is preferable to emitting guessed behavior.

Resolution and symbol identity must agree across crates. If one crate declares a concrete instance under one symbol and another emits it under another, linking fails. If distinct instances collide, the linker may choose the wrong body. Symbol mangling encodes identity into linker-compatible names, often with hashes to distinguish contexts.

Do not infer semantic identity from readable symbol fragments alone. Demangling is a diagnostic aid, and mangling formats can evolve. Use compiler-level instance information when debugging rustc itself.

A missing call target before LLVM IR suggests failed instance resolution or collection. A declared-but-undefined target suggests collection, linkage, or cross-crate placement. A call reaching the wrong implementation suggests substitution, trait selection, or symbol identity. This separation sharply narrows the search.

8. Trait methods at code generation time#

Traits support static dispatch and dynamic dispatch. Static dispatch chooses a concrete implementation during compilation. Dynamic dispatch chooses through a runtime table because the concrete pointee type is hidden behind a trait object. The two routes must implement the same trait contract but use different call machinery.

trait Speak {
    fn speak(&self) -> &'static str;
}

impl Speak for Dog {
    fn speak(&self) -> &'static str { "woof" }
}

fn static_call<T: Speak>(x: &T) -> &'static str {
    x.speak()
}

fn dynamic_call(x: &dyn Speak) -> &'static str {
    x.speak()
}

static_call::<Dog> can resolve to the Dog method instance and often inline it. dynamic_call receives a fat pointer containing data and vtable components. Its method call loads a function pointer from a defined vtable slot and calls it with the proper ABI.

Default methods do not remove the need for resolution. An implementation may override a default, and the selected body may still mention Self or associated types. The backend needs the concrete resolved instance, not merely the trait item's source identifier.

Receiver adjustment matters. A method written with &self, &mut self, Box<Self>, or another permitted receiver form expects a particular representation. Autoref and autoderef decisions were established earlier, while codegen must honor the resulting MIR and ABI. For virtual calls, object safety and vtable construction ensure a callable erased receiver form.

Trait object behavior is a Rust language feature, but exact vtable layout is generally an implementation detail unless documented by a stable ABI contract. Unsafe code should not hard-code rustc's current slot indices. Even when current vtables commonly contain drop, size, alignment, and methods, that observation is version-sensitive.

Supertraits and trait upcasting can require vtable relationships or adjustments. Auto traits can affect which trait-object type is legal without necessarily adding callable slots. Associated constants and types are resolved compile-time entities, not universally entries in runtime vtables.

Static dispatch trades code duplication for direct optimization. Dynamic dispatch shares caller code and supports heterogeneous values but adds pointer metadata and an indirect call. Neither is categorically faster: cache behavior and inlining opportunities depend on the program.

If only dyn Trait calls fail, inspect vtable creation, slot selection, receiver adjustment, and indirect-call ABI. If generic calls choose the wrong override, inspect instance selection. If both fail for one method, inspect the implementation body or shared ABI assumptions.

9. Closures and callable shims#

A closure is an anonymous callable value that may capture values from its environment. Each closure expression has a unique compiler-defined type. Its stored fields correspond conceptually to captures, though exact layout remains an implementation choice.

fn make_adder(base: i32) -> impl Fn(i32) -> i32 {
    move |value| base + value
}

let add_ten = make_adder(10);
assert_eq!(add_ten(7), 17);

The returned value stores base because move captures it by value. Calling the value invokes a generated closure body with the closure environment and explicit argument. The compiler knows the environment's concrete type during monomorphization.

The Fn, FnMut, and FnOnce traits describe callable receiver modes. Fn can call through a shared reference, FnMut through mutable access, and FnOnce consumes the closure. A closure's capture use determines which of these it implements. Adapters may be needed when one callable representation is used where another ABI shape is expected.

A shim is a small compiler-generated adapter function. It changes calling shape or performs required glue while preserving meaning. For example, a shim may adapt a Rust callable to a function pointer form, invoke a method with an adjusted receiver, or bridge a virtual call.

caller arguments
      |
      v
adapter shim ---- adjusts receiver/tuple/ABI ----> real body

Noncapturing closures can often coerce to ordinary function pointers because no environment data is needed. Capturing closures cannot generally become plain function pointers: a code address alone has nowhere to store captures. A trait object or generic parameter can carry the environment.

Async blocks and generators use related compiler-generated state representations, but their detailed lowering is beyond this half's scope. The shared lesson is that source-level callable syntax may become a data object plus one or more generated call bodies.

Shims avoid burdening every caller with special cases. They centralize adaptation and provide an addressable target when runtime tables need one. Their cost is extra symbols and sometimes an extra call, although inlining can erase that boundary.

Closure bugs are often representation mismatches. Wrong captured values suggest capture layout or projection errors. Failures only through FnOnce suggest receiver consumption or shim selection. Failures only after coercion suggest the adapter ABI. An unresolved closure symbol suggests mono-item collection failed to follow the callable edge.

10. Drop glue and destruction#

Drop glue is compiler-generated code that destroys a value according to its concrete type. It may call a user-written Drop::drop, recursively destroy fields, or do nothing for types needing no destruction. “Glue” means automatically inserted support connecting language semantics to executable operations.

struct Packet {
    label: String,
    bytes: Vec<u8>,
}

impl Drop for Packet {
    fn drop(&mut self) {
        eprintln!("dropping {}", self.label);
    }
}

Destroying a Packet calls its Drop implementation and then destroys its fields in Rust's specified destruction sequence. The concrete glue knows that String and Vec<u8> themselves require destruction. For (u32, bool), glue can be trivial because neither component owns a resource.

Generic code cannot always know whether T needs drop until substitution. After monomorphization, rustc can ask whether the concrete type has nontrivial destruction and collect the appropriate glue instance. This avoids requiring every type to carry a runtime destructor pointer.

Partial initialization makes destruction subtle. If constructing a struct panics after only some fields initialize, cleanup must drop exactly those initialized fields. Dropping an uninitialized field is invalid; leaking an initialized owner is usually undesirable and may violate expected cleanup. MIR uses control flow and drop-related state to express these paths before final lowering.

Unwinding is propagation of a panic through stack frames where the target and panic strategy support it. Cleanup edges run destructors during unwinding. With an abort strategy, panic terminates instead, allowing different generated control flow. The Rust source semantics and compilation options jointly determine which path is required.

ManuallyDrop<T> suppresses automatic destruction through its wrapper contract. MaybeUninit<T> represents possibly uninitialized storage and must not be treated as an initialized T prematurely. Unsafe code using these types carries the obligation to invoke destruction exactly when appropriate.

Drop glue can be invoked indirectly from a trait-object vtable because the hidden concrete type must be destroyed correctly. That makes drop glue both a collection dependency and an ABI-sensitive callable target.

Double drops, drops of uninitialized memory, and missing cleanup often originate in MIR drop elaboration or state tracking. A crash only for trait objects may instead implicate the vtable's drop entry. A linker error naming drop glue suggests instance collection or symbol placement. Do not begin with field layout unless evidence shows the destructor receives the wrong address.

11. Vtables and erased values#

A vtable is a runtime table associated with a concrete type's implementation of a trait-object interface. It lets code operate on a value whose concrete type has been erased. Erasure hides static identity while retaining metadata needed for permitted operations.

trait Area {
    fn area(&self) -> u64;
}

fn sum(values: &[&dyn Area]) -> u64 {
    values.iter().map(|value| value.area()).sum()
}

Each &dyn Area conceptually carries a pointer to data and a pointer to an appropriate vtable. Different elements can point to different concrete types and therefore different vtables. The shared sum machine code uses the metadata rather than being monomorphized per element type.

A vtable must provide compatible method targets for its trait-object type. It also needs enough information for operations such as size, alignment, and destruction when those operations apply. Exact entries and ordering are current-rustc details, not a general stable Rust ABI.

Vtable creation induces reachability. If code coerces &Circle to &dyn Area, the compiler must make the relevant Circle method implementation reachable even without a direct call. It may also need drop glue and supertrait-related metadata. Missing this dependency produces an incomplete runtime table or undefined symbol.

The data pointer may need adjustment for some coercions and receivers. The method function pointer must use the ABI expected by the virtual caller. The caller and table producer must agree on slot meaning, receiver representation, return mode, and unwind behavior.

Why not store type tags and use a giant switch? A closed switch cannot support independently defined implementations cleanly and grows with every type. Vtables package behavior per concrete implementation and permit separate compilation. The cost is one metadata pointer, table storage, and indirect dispatch.

Vtable identity should not be used casually as stable type identity. Optimizers and codegen arrangements may duplicate or merge constants, and language guarantees do not generally promise one immortal unique address per conceptual vtable. Use supported reflection facilities such as TypeId where their contracts fit.

When a virtual method calls the wrong function, compare trait-object type, vtable construction, and slot lookup. When destruction alone fails, inspect the drop entry and data pointer. When sizes are wrong through allocation or deallocation paths, inspect vtable metadata and target layout. These symptoms distinguish table contents from the method body's own logic.

12. Target data layout and ABI#

Data layout describes how target memory represents values: pointer sizes, integer alignments, endianness, and aggregate rules. ABI means Application Binary Interface, the machine-level agreement for calls, symbol linkage, register use, stack layout, and unwinding. Two modules can link successfully yet misbehave if they disagree on ABI.

Question                         Answered mainly by
How many bytes is this value?    type layout for the target
At what boundary may it start?   alignment rules
Which register carries arg 1?    calling convention and ABI classification
How is a symbol named/exported?  linkage and object-format conventions
Which byte comes first?          target endianness

Size is the byte extent occupied by a value in its representation. Alignment is a power-of-two-style placement requirement: an address aligned to 8 is divisible by 8. Padding is unused space inserted so fields or following values meet alignment requirements.

#[repr(C)]
struct Header {
    tag: u8,
    length: u32,
}

On many targets this has padding after tag, but portable code should obtain size and alignment through Rust facilities rather than assume host figures. repr(C) requests C-compatible layout under the target's C ABI rules; it does not make every contained Rust type automatically safe for foreign code.

Ordinary repr(Rust) gives rustc freedom to arrange fields subject to language guarantees. repr(transparent) supplies specific layout and ABI promises for qualifying wrappers. Packed representations reduce alignment and padding but can make references to fields invalid due to misalignment. Unsafe code must respect each representation's documented contract.

Function ABI classification decides whether an argument goes in registers, memory, or an indirect pointer, and how returns work. Large aggregates may be passed indirectly. Small aggregates may be split into register classes. Rules vary across targets and calling conventions.

Rust's default Rust ABI is not generally promised stable across compiler versions. Foreign interfaces should use an explicit supported ABI such as extern "C" and only FFI-safe types. Even then, both sides must agree on target, declarations, variadic rules, ownership, and unwinding policy.

Cross-compilation exposes hidden assumptions. Host pointer width must not leak into target layout. The compiler process might be 64-bit while emitting 32-bit code. Layout arithmetic, constants, and debug information must all use target properties.

Failures at every call to one foreign function strongly suggest a declaration or ABI mismatch. Failures only for one aggregate argument suggest classification or layout. Failures only on a different target suggest target data layout, calling convention, or unsupported assumptions before generic logic.

13. Enums, niches, scalar pairs, and fat pointers#

An enum represents one of several variants. A straightforward representation stores a discriminant, which identifies the variant, plus space for the largest payload. Rustc may choose more compact representations when language layout freedom permits.

enum Message {
    Empty,
    Number(u64),
    Pair(u32, u32),
}

A niche is a bit pattern that a type's valid values never use. For example, a valid reference cannot be null. An implementation can use the null pattern to encode None in an Option<&T> without adding a separate discriminant. This is called niche optimization.

Some layout facts are guaranteed for specified standard patterns, but developers must consult the language and library documentation rather than generalize from experiments. The fact that current rustc optimizes one enum does not promise identical treatment for every similar user-defined enum. Version-sensitive layout choices must not become undocumented unsafe-code contracts.

A scalar is a primitive machine-like value in rustc's layout model, such as an integer or pointer with a valid range. A scalar pair is a representation consisting of two such components. Many fat pointers naturally fit this model. This terminology is an internal modeling choice, not a new Rust source type category.

A fat pointer is a pointer-like value with metadata in addition to a data address. A slice reference &[T] carries a data pointer and length. A trait-object reference &dyn Trait carries a data pointer and vtable pointer. The metadata explains how to interpret an unsized pointee.

&[T]       = [ data address | element count ]
&dyn Trait = [ data address | vtable address ]
&SizedT    = [ data address ]

An unsized type lacks a compile-time constant size for values behind it, such as [T] or dyn Trait. It is ordinarily accessed behind a pointer carrying sufficient metadata. The pointer itself still has a known size for the target.

Layout feeds ABI classification. A scalar pair might travel in two registers on one target and indirectly on another. Codegen must extract, pass, reconstruct, compare, and store components consistently. A mismatch can corrupt only calls while local memory operations appear correct.

Niche bugs often appear as wrong variant discrimination after optimization or transmutation. Fat-pointer bugs split naturally: correct address but wrong length indicates metadata construction; correct vtable but wrong data indicates coercion or adjustment; correct local value but wrong callee value indicates ABI classification. These observations guide debugging toward representation boundaries.

14. Preparing MIR for code generation#

Codegen MIR is MIR transformed into a form suitable for concrete lowering. Preparation makes implicit language obligations explicit, removes forms the backend should not see, and applies substitutions or simplifications needed for emission. The exact pass sequence and names are current-rustc details and therefore version-sensitive.

Earlier MIR serves several consumers and may preserve abstractions useful for analysis. The backend benefits from a narrower contract. It wants explicit control flow, concrete types, resolved constants, elaborated drops, and operations with known target representations. Separating preparation from LLVM construction keeps language reasoning out of instruction-by-instruction lowering.

Before preparation                  Backend-oriented result
generic type parameter T            concrete substituted type
high-level drop obligation          explicit drop/cleanup control flow
constant expression                 evaluated or lowerable constant
aggregate operation                 representation-aware construction
assertion with unwind behavior      explicit success/failure edges

Preparation may simplify code, but it must not depend on LLVM to repair invalid semantics. It can remove unreachable blocks, propagate simple facts, or normalize operations when permitted. Aggressive machine optimization belongs later, where target costs and LLVM analyses are available.

Drop elaboration is especially important. It turns conditional destruction obligations into control flow and state that ensures exactly initialized values are dropped. Unwind edges must agree with panic strategy and callable ABI.

Constants can embed allocations, pointers, or values whose validity depends on layout. Lowering them requires the same target representation discipline as runtime values. A constant that printed correctly during earlier evaluation can still be emitted incorrectly if relocation or alignment handling is wrong.

Preparation also faces unreachable code. An unreachable block can contain types or operations that are impossible under valid execution. The compiler must neither generate unsound code nor crash merely because a legal program retains dead structure. Precisely which impossible forms may remain is part of the internal pass contract.

Validation between stages is valuable. Internal MIR checks can catch malformed places, inconsistent types, or illegal statements near the pass that introduced them. Without validation, the eventual LLVM error may be far removed from the cause. Assertions in compiler internals document invariants as well as detecting violations.

If an error appears in unoptimized and optimized builds alike before LLVM, inspect preparation or resolution. If dumping MIR shows the last good stage and first bad stage, focus on that transformation. If prepared MIR is correct but LLVM IR is wrong, move downstream to lowering. This “last good representation” method is one of the strongest compiler-debugging tools.

15. Codegen units and partitioning#

A codegen unit, commonly abbreviated CGU, is a group of mono items compiled together into one backend module. Partitioning assigns collected work to these groups. It is primarily a compile-time and optimization engineering choice, not a Rust language concept.

One giant unit gives the optimizer broad visibility. It can inline and combine code across the whole crate, but it consumes more memory, limits parallel compilation, and causes large rebuilds after small changes. Many small units improve parallelism and reuse but hide bodies across module boundaries and add coordination overhead.

collected mono items
   | partition by stable policy
   +-------> CGU 0 ---> LLVM module ---> object 0
   +-------> CGU 1 ---> LLVM module ---> object 1
   +-------> CGU 2 ---> LLVM module ---> object 2
                                      |
                                      v
                                    linker

Partitioning tries to balance locality, stable incremental reuse, duplicate availability for inlining, and symbol ownership. An externally visible function needs a clear home that emits its definition. Other units may receive declarations or internal copies under carefully chosen linkage. The details change as rustc's incremental and optimization strategies evolve.

Linkage describes how a definition participates in combination across object files. Some symbols are local to one object. Some are externally visible. Some may be emitted in multiple places under rules allowing the linker to select or merge equivalent definitions. Incorrect linkage creates duplicate-symbol errors, undefined symbols, or silently selected wrong bodies.

Inlining across CGUs can be enabled by exporting suitable IR summaries or duplicate candidate bodies, especially with link-time optimization. Link-time optimization, or LTO, optimizes across module or crate boundaries later in the pipeline. LTO can improve runtime performance and reduce duplication, but it increases link time and memory use.

Incremental compilation rewards stable partitioning. If a small source edit randomly moves hundreds of mono items between units, cached backend results become useless. Stable naming and deterministic item assignment are therefore performance invariants even though they do not affect language meaning.

The number of CGUs is a tuning knob with workload-dependent effects. More units do not always mean faster builds, because tiny modules incur setup cost and can reduce optimization quality. Fewer units do not always mean faster programs, because later LTO and profile behavior complicate the result. Measure representative clean builds, incremental builds, binary size, and runtime separately.

When one CGU lacks a symbol defined in another, inspect declaration creation, linkage, and partition ownership. When changing an unrelated file recompiles most backend modules, inspect partition stability and mono-item identity. When performance changes with the CGU count, inspect lost cross-unit inlining before assuming instruction selection regressed.

At this handoff, rustc has selected reachable concrete work, resolved instances and generated helpers, committed to layouts and ABIs, prepared MIR, and assigned mono items to backend modules. The next stage lowers each module into LLVM IR: typed operations, basic blocks, calls, memory accesses, attributes, and metadata that LLVM can verify and optimize before producing machine code.

16. The backend boundary#

The backend turns compiler meaning into artifacts that a target machine and its linker understand. In rustc, much of the shared machinery lives under rustc_codegen_ssa. SSA means static single assignment, an intermediate form in which each computed value is assigned once. The name is historical as well as descriptive: shared codegen also coordinates partitioning, symbols, linking, and archives. The concrete LLVM implementation lives in rustc_codegen_llvm. This split is an abstraction boundary, not a promise that every backend behaves identically internally.

The input is chiefly optimized MIR plus type layouts, ABI decisions, crate metadata, and compilation options. MIR is control-flow-oriented Rust compiler IR. The output may be LLVM modules, object files, metadata, archives, shared libraries, or an executable. Codegen units, abbreviated CGUs, partition one crate into separately optimized modules. More CGUs can improve parallelism and incremental reuse. Fewer CGUs can expose more code to optimization and reduce duplicated work.

An essential invariant is that lowering preserves Rust's defined behavior. For unsafe code, preservation applies only while the program obeys every relevant safety and validity requirement. Rust specifies language semantics; rustc chooses a current implementation; LLVM transforms IR under LLVM's rules. The linker resolves and lays out symbols; the target ABI defines calling and data conventions. Confusing those authorities produces misleading explanations and bad bug reports.

rustc_codegen_ssa offers traits and shared routines so rustc is not wholly fused to LLVM. The abstraction covers concepts such as builders, values, types, constants, modules, and backend execution. It is deliberately shaped by rustc's needs and has historically reflected LLVM concepts. It is not a stable public Rust API. Exact trait names, methods, and ownership boundaries can change between nightly compilers.

Conceptually, the path is:

optimized MIR + layouts + ABI
             |
             v
  rustc_codegen_ssa coordination
       /          |          \
  LLVM backend  alternative  shared link/archive work
       |             |
       +------ object files ------+
                     |
                  linker

Monomorphization supplies concrete instances of generic functions before or during collection for codegen. An instance is a callable body paired with substitutions and dispatch choices. Backend lowering must use the already-decided layout and ABI rather than inventing a second type system. If layout says a field is at an offset, loads and debug descriptions must agree with that offset. If the ABI passes a value indirectly, caller and callee must agree on the hidden pointer.

A counterfactual backend that lowered source text directly would duplicate type checking and layout decisions. It could silently disagree with the rest of rustc. The shared boundary exists to make disagreement harder and alternative implementations practical. Symptoms of boundary mistakes include wrong-code only on one target, linker symbol mismatches, and backend crashes. Begin investigation by recording the compiler version, target triple, flags, and smallest MIR-producing example.

17. LLVM IR and static single assignment#

LLVM IR is a typed, control-flow intermediate representation consumed by LLVM optimizers and code generators. It is lower level than MIR but higher level than machine instructions. A function contains basic blocks, and each basic block has ordinary instructions followed by a terminator. A terminator transfers control by branching, returning, switching, or declaring that execution cannot continue.

SSA values are immutable names for computation results. Memory is not magically immutable: a pointer can address storage that receives multiple stores. SSA therefore makes data dependencies explicit while memory dependencies require separate analysis. A phi node selects an incoming SSA value according to the predecessor block. Modern LLVM frontends may build equivalent control flow and let LLVM form or simplify phis.

entry:
  branch condition, then, else
then:
  a = 10
  branch join
else:
  b = 20
  branch join
join:
  result = phi [a, then], [b, else]

This is explanatory pseudocode, not guaranteed output syntax. Real IR includes exact integer widths, pointer forms, data layout, attributes, and metadata. LLVM's opaque-pointer transition also changed printed pointer types across LLVM versions. Never make a test depend on incidental spelling when the semantic property can be checked.

LLVM IR has stronger rules than “whatever the hardware does.” Undefined behavior, or UB, permits LLVM to assume the forbidden event never occurs. Poison is a deferred invalid result that can contaminate later operations and become UB at sensitive uses. undef, poison, and uninitialized Rust memory are related debugging topics but are not interchangeable terms. Rust validity requirements and LLVM IR rules meet at lowering, making this boundary especially delicate.

Attributes communicate facts such as calling convention, alignment, aliasing, unwinding, and valid ranges. An attribute that is too weak can lose optimization. An attribute that is too strong can enable miscompilation. For example, claiming pointers never alias is a semantic assertion, not a performance hint. Metadata can guide optimization or debugging, but some metadata also carries correctness-sensitive assumptions.

The target data layout states pointer sizes, alignments, integer rules, and address spaces as LLVM understands them. It must match rustc's target description and the eventual object environment. A mismatch can make apparently valid offset arithmetic access the wrong bytes. Endianness determines byte order; it does not alter Rust source-level arithmetic.

You can ask a nightly compiler for intermediate output, but unstable flags change.

rustup run nightly rustc -Z help
rustup run nightly rustc --emit=llvm-ir -C opt-level=0 example.rs
rustup run nightly rustc --emit=llvm-ir -C opt-level=3 example.rs

Stable rustc supports --emit=llvm-ir, although the textual IR is not a stable compatibility interface. Compare unoptimized and optimized output to learn which layer removed an operation. Do not infer a Rust guarantee merely because one compiler version emitted a particular instruction.

18. Lowering MIR places and rvalues#

A MIR place describes a storage location rather than necessarily a loaded value. It starts from a local and may project through fields, dereferences, indexes, or downcasts. An rvalue describes a computation whose result can be assigned to a place. Examples include constants, references, casts, aggregates, unary operations, and binary operations.

Lowering first asks how the Rust value is represented. Some values are immediate scalars in registers. Some are scalar pairs, such as many fat pointers. Others live indirectly in memory because their size or ABI treatment requires an address. Zero-sized types may require no data bytes while still participating in control flow and drop semantics.

The backend computes addresses from layout, not from source field order alone. Rust generally does not guarantee the layout of the default repr(Rust) representation. repr(C), repr(transparent), and integer representation attributes provide specific additional contracts. Even then, the target ABI contributes size and alignment rules.

A place projection can become pointer arithmetic plus a load or store. Bounds checks are explicit control flow before an indexed access unless proven unnecessary. Alignment must be correct for an aligned operation; packed fields may require unaligned access. Creating an invalid reference is not repaired merely because the hardware tolerates an unaligned load.

#[repr(C)]
struct Pair {
    left: u32,
    right: u16,
}

fn sum(pair: &Pair) -> u32 {
    pair.left + u32::from(pair.right)
}

Rust guarantees the repr(C) field-order and C-compatible layout rules applicable here. Rust does not guarantee that sum contains two literal machine loads after optimization. rustc may combine, eliminate, or rearrange accesses while preserving observable behavior. LLVM performs many such transformations; the target decides available instruction forms.

Integer arithmetic requires careful semantic selection. Debug overflow checks may branch to panic where a release configuration wraps for ordinary integer operators. Certain operations, such as division by zero, still require defined Rust behavior independent of optimization level. Shifts, casts, discriminants, and pointer operations each have distinct rules.

Aggregates can be assembled in SSA values or written field by field into a destination. Destination-passing avoids large temporary copies and supports indirect returns. Enum lowering uses layout-selected discriminants, niches, and payload placement. A niche is an otherwise invalid bit pattern reused to encode an enum variant. The familiar null-pointer optimization is an example, not a blanket layout guarantee for every enum.

If place lowering is wrong, symptoms often depend on optimization, alignment, or enum variant. Inspect MIR to confirm the intended operation, then IR to locate representation drift. Compare targets because a 32-bit pointer width or stricter alignment can expose assumptions hidden on x86-64.

19. Calls, switches, and drops#

A call combines a function identity, calling convention, argument ABI, return ABI, and unwind behavior. Rust-level type agreement is necessary but not sufficient for binary compatibility. The backend may pass scalars directly, split aggregates, or use hidden pointers for returns and arguments. Calling through an incompatible function pointer is therefore dangerous even if register contents look plausible.

Rust ABI strings state language-level intent, while rustc maps them to target conventions. extern "C" requests the platform C ABI where supported. It does not make arbitrary Rust types safe or portable across an FFI boundary. Use explicitly compatible representations and document ownership and unwinding constraints.

#[unsafe(no_mangle)]
pub extern "C" fn add(left: i32, right: i32) -> i32 {
    left + right
}

The exact syntax accepted for unsafe attributes depends on edition and compiler version. The exported name can be stable by explicit choice, but its surrounding object format remains target-specific. Whether a linker retains the symbol also depends on artifact kind, visibility, and dead stripping.

MIR SwitchInt selects successors from an integer-like value. LLVM may represent it as a switch, comparisons, or branches. Later instruction selection may choose a jump table, binary decision tree, or linear chain. Density, profile information, code-size goals, and target support influence that choice. Rust guarantees branch behavior, not a jump-table implementation.

Drops run destructors for initialized values along required paths. Drop flags track whether partially moved or conditionally initialized values still need destruction. The backend lowers cleanup paths and invokes drop glue, which is compiler-generated destruction code. It must not double-drop moved data or skip a live destructor.

Unwinding adds exceptional edges from calls that may panic or throw through an allowed ABI boundary. Landing pads or target-specific equivalents enter cleanup regions. With panic abort, many Rust panic cleanup edges disappear, but foreign exceptions and platform details still require care. An optimizer may remove a drop only when its observable effects are proven absent or unreachable.

Virtual calls load a method address from a vtable and pass the data pointer according to the chosen ABI. A vtable is compiler-produced read-only data containing method pointers and layout-related entries. Its exact layout is a rustc implementation detail unless an explicit stable contract says otherwise. Trait-object internals should not be treated as a cross-version C ABI.

Tail calls, inlining, and devirtualization are optimizations rather than ordinary Rust promises. Failure to inline is usually a performance observation, not a correctness bug. A wrong cleanup edge, by contrast, can appear as a leak, duplicate destructor output, or crash only during panic. Test normal return and panic paths separately when reducing such a bug.

20. LLVM optimization and machine-code generation#

Optimization is a sequence of analyses and transformations, commonly called passes. Rustc performs MIR optimization before LLVM receives IR. LLVM then performs target-independent and target-aware IR optimization. Finally it lowers IR toward machine instructions, schedules them, and emits object code.

Inlining substitutes a callee body into a caller. It can expose constants and remove call overhead, but increases code size and compile time. Dead-code elimination removes computations with no observable effect. Loop transformations may unroll, vectorize, or simplify loops when legality and profitability analyses agree. Alias analysis estimates whether memory accesses can refer to the same location.

Instruction selection maps IR operations to target machine patterns. One IR addition may fold into an addressing mode rather than become a standalone add instruction. Legalization rewrites operations unsupported at a given width or type. Instruction scheduling orders operations to respect dependencies and use a processor pipeline effectively.

Register allocation maps an unbounded set of temporary values to finite physical registers. When registers are insufficient, values spill to stack slots and later reload. Spills can be expensive, but a visible stack access does not automatically prove allocation failure. Calling conventions, debug quality, stack protection, and addressing constraints also force memory traffic.

rustc -C opt-level=0 --emit=asm example.rs
rustc -C opt-level=3 --emit=asm example.rs
rustc -C opt-level=s --emit=asm example.rs

Optimization levels are policy bundles, not formal promises about a specific pass list. opt-level=s favors size while retaining optimization; z is more aggressively size-oriented. Compiler releases can alter pipelines and cost models. Benchmark the generated program rather than counting instructions in one small function alone.

Backend transforms rely on semantic facts encoded in IR. If lowering omits a fact, optimization may be weaker. If lowering lies, optimization may produce wrong code far from the original mistake. This explains why a failure appearing only at -O is not necessarily an optimizer bug. The optimizer may merely expose invalid input IR generated by rustc.

Machine code belongs to a specific target and feature set. The same LLVM IR can select different instructions for AArch64, RISC-V, WebAssembly, or x86-64. Assemblers encode instructions; object writers package code and relocation requests. The linker does not normally redo high-level SSA optimization unless LTO supplies suitable IR.

For a performance regression, compare MIR, LLVM IR, assembly, object size, and runtime measurements. The first layer that diverges narrows ownership. Keep compiler version, flags, CPU affinity, workload, and measurement method fixed. Counterfactual “shorter assembly must be faster” reasoning ignores latency, throughput, branches, and cache behavior.

21. Target features and CPU dispatch#

A target triple identifies an architecture, vendor, operating system, and environment by convention. It selects an ABI and broad platform model, not one exact physical processor. The target CPU and target features refine which instructions codegen may emit. Examples include SIMD extensions, atomics, floating-point facilities, and security-related instructions.

-C target-cpu=native asks rustc to optimize for the build machine's detected CPU. That can improve local performance and make the binary fail with an illegal instruction elsewhere. Portable release artifacts should choose an explicit deployment baseline. Cross compilation makes native particularly suspect because build and execution machines differ.

Feature enablement is an unsafe compatibility promise at execution time. If a function is compiled with AVX2, calling it on a CPU without AVX2 can fault. Runtime feature detection permits dispatch among specialized implementations.

#[cfg(target_arch = "x86_64")]
fn compute(data: &[u8]) -> u64 {
    if std::is_x86_feature_detected!("avx2") {
        // A real program may call an AVX2-specialized unsafe function here.
        scalar(data)
    } else {
        scalar(data)
    }
}

fn scalar(data: &[u8]) -> u64 {
    data.iter().map(|&byte| u64::from(byte)).sum()
}

The example stays safe and stable while illustrating the dispatch decision. A specialized function commonly uses #[target_feature(enable = "avx2")] and requires an unsafe call. The caller must establish the feature precondition every time. Inlining across different feature sets is constrained because it could leak unsupported instructions.

Compile-time cfg(target_feature = ...) answers what this compilation enables globally at that location. Runtime detection answers what the current processor reports. They solve different problems. The operating system may also need to enable register-state support for a CPU extension.

Multiversioning creates several implementations and a selector. It improves reach and peak speed but increases binary size and test combinations. Dispatch overhead matters for tiny operations, so cache a choice or dispatch around a substantial workload. Libraries should not silently raise the caller's minimum CPU without documenting it.

LLVM decides instruction selection under enabled features. rustc decides how options and attributes communicate those features. The target specification establishes defaults and compatibility constraints. Hardware determines what actually executes. An illegal-instruction crash usually suggests a baseline or dispatch error before it suggests memory corruption.

22. Object files, sections, symbols, and relocations#

An object file is a relocatable container, not yet a complete executable. Common formats are ELF, Mach-O, COFF, and WebAssembly object conventions. Each has different commands, section names, symbol rules, and relocation encodings. Do not universalize an ELF observation to every target.

Sections group bytes with related purposes. Typical conceptual groups hold executable code, read-only constants, writable data, zero-initialized data, unwind data, and debug records. A section has alignment and permission expectations. The final linker combines, discards, splits, or renames input sections according to platform policy and scripts.

A symbol gives a name and attributes to an addressable definition or reference. It can be local or externally visible, defined or undefined, weak or strong. A relocation asks a later tool to patch bytes once a referenced address or offset is known. Position-independent code uses relocation forms suitable for loading at varying addresses.

producer.o: code calls helper + relocation(helper)
library.o: symbol helper defined in text section
                         |
                         v
linker lays out sections, resolves helper, applies relocation
                         |
                         v
                 executable or shared object

Relocations have width, signedness, addends, range, and addressing-mode constraints. “Relocation truncated to fit” often means code or data lies outside the representable range. It can also indicate the wrong code model, visibility, or relocation kind. The error belongs to the object/link interface, though its cause may begin in compiler flags.

COMDAT is a mechanism, especially associated with COFF and supported analogously elsewhere, for grouping duplicate definitions. The linker retains an allowed representative and discards equivalent duplicates as a unit. LLVM linkonce-style linkage similarly permits deduplication under defined rules. These facilities matter for instantiated generics, inline functions, vtables, and compiler-generated glue.

Weak or deduplicable does not mean “all implementations are semantically interchangeable.” The compiler must assign compatible identity and contents. Incorrect grouping can retain references to discarded companions or merge definitions that should differ. Symptoms include duplicate symbols, missing symbols, or behavior dependent on link order.

Useful tools include llvm-readobj, llvm-objdump, platform readelf, nm, and otool. Tool availability and flags vary. Inspect headers before assuming an object format, then inspect sections, symbols, and relocations separately. Preserve the original object when debugging because the final link erases useful provenance.

23. Symbol names, visibility, and linkage#

Source names are not always globally unique. Modules, generic arguments, crate identity, namespaces, and compiler-generated instances all contribute identity. Symbol mangling encodes enough of that identity into names accepted by object tools. Rust's current mangling schemes are rustc behavior, with selected details documented for tooling, not a general C ABI promise.

Demangling turns an encoded symbol into a human-readable approximation. It aids profiles and backtraces but does not recover source code. Two long demangled names may still refer to different monomorphizations or crate versions. Always retain the raw name when diagnosing linker resolution.

Visibility controls which symbols can be referenced or preempted outside a linkage unit. Language pub controls Rust accessibility and is not identical to dynamic-library export visibility. A public generic often contributes no single exported callable symbol because downstream crates instantiate it. Conversely, runtime support symbols may be exported without being part of a Rust source API.

Linkage describes how definitions combine: internal, external, weak, deduplicable, or declarations only. Object formats and linkers map these concepts imperfectly. Hidden visibility can improve optimization and avoid symbol interposition, but prevents external lookup. Default visibility may support a plugin ABI while increasing exported surface and relocation work.

Stable cross-language entry points require deliberate names and ABIs.

#[unsafe(no_mangle)]
pub extern "C" fn api_version() -> u32 {
    1
}

This controls the function's symbol name and C calling convention on a compatible target. It does not freeze Rust's standard-library ABI, panic behavior, struct layouts, or allocator ownership. An FFI API should expose C-compatible types and define who allocates, frees, and handles errors.

Symbol collisions are possible when mangling is suppressed. Two crates exporting api_version into one linkage namespace conflict unless platform rules choose one. That selection is usually not a sound versioning mechanism. Prefix names or use symbol-versioning facilities designed for the platform.

Dead stripping removes unreferenced sections or symbols. An apparent export can disappear if no root or export list retains it. Dynamic lookup, registration tables, and embedded assembly may need explicit retention mechanisms. Those mechanisms are linker- and format-specific, so test the final artifact's export table.

24. Linking and native libraries#

Linking combines Rust-produced objects, native objects, archives, runtime components, and shared-library references. Rustc usually drives a system linker or a linker-compatible program with target-specific arguments. The linker resolves symbols, chooses archive members, lays out output sections, and applies relocations. It may also perform dead stripping, identical-code folding, and platform signing-related preparation.

A static archive is an indexed collection of object files. Traditional archive extraction is demand-driven: members are pulled when they satisfy unresolved symbols. Library order can therefore matter on linkers that process inputs from left to right. Groups or repeated libraries can resolve circular archive dependencies, at a cost in link work.

A shared library is loaded as a separately mapped image. Its consumers record imports rather than copying all implementation bytes. Runtime search paths, loader configuration, install names, and versioned sonames are platform concerns. Successful linking does not guarantee the loader can find the library at runtime.

Build scripts can emit Cargo linking directives, but rustc also accepts native-library specifications directly. Static versus dynamic preference, bundle behavior, whole-archive behavior, and modifiers are version-sensitive. Consult the rustc command-line reference for the exact toolchain rather than copying folklore.

undefined reference at link time
  -> inspect raw missing symbol
  -> identify object expected to define it
  -> inspect that object's symbol table
  -> check ABI/name decoration and visibility
  -> check archive extraction and library order

library not found at program start
  -> inspect dynamic dependencies
  -> inspect loader search policy
  -> distinguish build-time path from runtime path

Native libraries can impose licensing, deployment, initialization, threading, and ABI constraints. A C header is not itself proof that every target implements an identical ABI. Widths of long, structure padding, calling conventions, and exception models vary. Generate or validate bindings against the actual target.

Linker scripts provide exact layout control on some targets, especially embedded systems. They can place vectors, firmware headers, and memory regions. A wrong script can overlap sections or discard required metadata even when compilation succeeds. Map files are invaluable because they show placement and symbol provenance.

Rust guarantees behavior of valid Rust programs within documented platform support. rustc selects libraries and flags. The linker implements resolution and layout. The operating-system loader implements runtime mapping. Keep those stages separate when assigning a failure.

25. Artifact kinds and their contracts#

An artifact is an output consumed by another compiler invocation, linker, loader, or user. The common names describe different contracts rather than mere filename extensions. Exact contents and naming conventions depend on target and compiler version.

rmeta contains Rust crate metadata used for downstream compilation. It supports understanding exported Rust items without carrying ordinary native code for final linkage. Its serialized format is private to rustc and not stable across compiler versions. It is particularly useful for checking workflows that avoid full code generation.

An rlib is a Rust static-library artifact for rustc-to-rustc consumption. It commonly packages metadata and native object material in an archive-like container. It is not simply a C static library and should not be handed to arbitrary native consumers as an ABI contract. Rustc knows how to select and combine its contents.

A Rust dylib is a dynamic Rust library intended for Rust linkage under compatible compiler conditions. Its Rust ABI and metadata coupling make long-term binary compatibility unsuitable as a default assumption. A staticlib packages a crate and statically needed Rust dependencies for linking from non-Rust build systems. A cdylib creates a dynamic library aimed at a foreign-language ABI surface.

Neither staticlib nor cdylib automatically makes every public Rust item callable from C. Export explicit extern "C" functions using FFI-safe representations. Prevent unwinding across boundaries that do not permit it. Provide headers and lifecycle rules as part of the real interface.

A binary artifact has an entry point and is linked for execution on the target environment. It may still depend on shared system or Rust libraries according to options and platform defaults. “Standalone” must be verified by inspecting dependencies and testing on a clean target system.

rustc --crate-type=rlib lib.rs
rustc --crate-type=staticlib lib.rs
rustc --crate-type=cdylib lib.rs
rustc main.rs

These stable commands demonstrate categories, not portable output filenames. Cargo usually orchestrates artifact selection and dependency compilation more conveniently. Mixing multiple compiler versions can reject metadata or, worse for unsupported native mixing, violate ABI assumptions.

Choose by consumer. Use rmeta for compiler analysis workflows, rlib for ordinary Rust static dependency flow, and Rust dylib for compatible Rust dynamic flow. Use staticlib or cdylib behind a designed foreign ABI. Use a binary when the operating system or runtime is the direct consumer.

26. LTO, ThinLTO, and incremental object reuse#

Ordinary separate compilation limits optimization to each codegen unit plus available summaries. Link-time optimization, LTO, carries optimizer-readable program information later so calls across units can be analyzed. This can improve inlining, constant propagation, dead-code removal, and devirtualization. It increases compile time, memory use, and coupling between units.

Fat LTO considers broad combined IR and can maximize global opportunity at substantial cost. ThinLTO uses compact summaries to plan cross-module importing and optimization in parallel. ThinLTO is “thin” in orchestration, not necessarily small in every workload. Results depend on LLVM version, CGU partitioning, visibility, and profile information.

LTO does not transcend semantics. An opaque external call remains constrained by its declared attributes and visibility. Dynamic symbol interposition may prevent assuming a local definition is final. Foreign objects without compatible embedded IR generally participate as machine code only.

Incremental compilation tries to reuse work products when inputs have not changed materially. At the backend level, an unchanged CGU object can be reused rather than regenerated. Partition stability therefore matters: a tiny edit that moves many items among CGUs destroys reuse. Inlining dependencies can also invalidate a caller even when its source text is untouched.

Reuse has a strict correctness invariant. A reused object must match current source semantics, compiler version, target, options, dependency identities, and relevant environment. If any correctness-affecting input changes, reuse must be rejected or recomputed. Reusing more is a performance goal; reusing stale code is a compiler bug.

LTO and incremental reuse pull policy in different directions. Global optimization benefits from seeing and reconsidering more modules. Fast edits benefit from stable, independent reusable units. rustc and Cargo expose profiles so users can choose development latency or final performance.

Symptoms of poor reuse are unexpectedly broad recompilation and increased edit-build time. Symptoms of incorrect reuse are output differing between incremental and clean builds. Always confirm the latter with identical commands and a cleared target/incremental directory. Do not erase the failing cache before preserving a reproducer if compiler developers may need it.

ThinLTO caches are another implementation detail with version-sensitive diagnostics. Use timing and compiler self-profile tools to distinguish optimization cost, object emission, and linking. A large final link is not proof that incremental compilation failed. The linker may repeat work even when rustc reused every eligible object.

27. Debug information, unwinding, and panic strategy#

Debug information maps machine addresses and locations back to source concepts. Common formats include DWARF on many Unix-like targets and CodeView/PDB on Windows. It describes files, lines, functions, lexical scopes, types, and variable locations. Optimization makes this mapping approximate because variables can be merged, split, moved, or removed.

Rustc creates debug metadata; LLVM tracks it through transforms and emits target records. The linker relocates, combines, strips, or separates those records. The debugger interprets them. A missing local variable can therefore originate in any of several stages and is not automatically a debugger defect.

Line tables are enough for basic symbolic stacks but not full variable inspection. Higher debug levels increase artifact size and sometimes optimization constraints. Split debuginfo stores substantial records outside the primary binary on supported platforms. Stripping symbols and stripping debug sections are related but distinct operations.

Unwind tables describe how to recover caller state from a program counter and stack state. They support exceptions or panic unwinding, but also profilers and backtraces. A panic strategy selects whether Rust panics unwind or abort the process. panic=unwind runs cleanup while searching for a handler where the target supports it. panic=abort terminates without Rust stack unwinding and can reduce code size.

struct Notice;

impl Drop for Notice {
    fn drop(&mut self) {
        eprintln!("dropping");
    }
}

fn fail() {
    let _notice = Notice;
    panic!("stop");
}

Under unwinding, Notice::drop ordinarily runs while leaving fail. Under abort, the process terminates and that cleanup is not performed. Programs must not rely on destructors for persistence guarantees during process abort or power loss.

FFI unwind behavior depends on the declared ABI and both languages' exception models. Unwinding through a boundary that forbids it is not made safe by observing it work once. Use an unwind-permitting ABI where documented, or catch and translate failure before crossing. Some panic conditions cannot be caught because they abort directly.

An “unable to unwind” crash may mean missing/corrupt tables, crossing a forbidden frame, or double panic during cleanup. Capture the panic strategy, target, backtrace, binary stripping steps, and native frames. Compare at low optimization before blaming optimized line mappings.

28. Reproducibility and cross compilation#

A reproducible build produces byte-identical artifacts from declared identical inputs under a controlled environment. Deterministic compiler behavior is necessary but not sufficient. Paths, timestamps, archive member order, random seeds, linker build IDs, environment variables, and native tools can alter bytes. Debug information commonly embeds paths unless remapped.

First define the claim precisely. Rebuilding on one host is weaker than rebuilding across hosts. Semantic equivalence is weaker than byte identity. Identical final binaries do not imply identical intermediate metadata. A trustworthy report names compiler, linker, target, flags, source tree, dependency lockfile, and environment.

Path remapping can replace build-machine prefixes with stable virtual paths. It improves privacy and byte stability but can make local source lookup less direct. Stable ordering prevents hash-map or filesystem enumeration from leaking nondeterminism. Archives and platform signing may need their own deterministic modes.

Cross compilation means the host running rustc differs from the target running the output. Rustc must have target support, target libraries, and a suitable linker or linker driver. Native dependencies need target-built objects, not host objects. Build scripts execute on the host even when they help produce target artifacts.

build machine: x86_64 Linux
  host tool and build script: x86_64 Linux executable
  rustc backend target: aarch64 unknown Linux
  native dependency: AArch64 object
  linker/sysroot: understands AArch64 target ABI
  result: runs on compatible AArch64 Linux, not on the host

The target triple alone does not provide a C sysroot, system libraries, or device firmware layout. Some pure-Rust targets can link with bundled or self-contained components; others require an external SDK. Cross-link errors often reveal a host archive accidentally entering the target link. Inspect every suspicious object's machine header.

Target behavior includes pointer width, endianness, atomics, TLS, unwind model, and relocation limits. Portable Rust source can still depend on unavailable OS facilities through libraries. Test on the target or a faithful emulator; successful code generation proves only part of compatibility.

To diagnose nondeterminism, build twice in fresh directories and compare hashes. Then compare section tables, symbols, and byte ranges rather than treating the whole file as opaque. Vary one suspected input at a time. Do not promise reproducibility from --remap-path-prefix alone.

29. Alternative backends and backend debugging#

LLVM is rustc's mature default backend for most supported targets. Alternative backends explore compile-time, target, ecosystem, and architectural tradeoffs. They consume rustc's backend-facing abstractions but need not implement LLVM internally. Backend availability, completeness, and supported platforms are version-sensitive.

The Cranelift-based rustc backend prioritizes fast code generation and is valuable for development builds and experimentation. Its optimization level, target coverage, debuginfo, and platform integration may differ from LLVM's. Some toolchains or projects distribute it separately or require nightly/bootstrap steps. Consult the backend project's current documentation for installation and support status.

The GCC codegen backend targets GCC's code-generation infrastructure. It can open access to GCC-supported targets and provide an independent implementation for comparison. Its build requirements, feature completeness, and upstream integration continue to evolve. Do not state that either alternative is universally drop-in or production-equivalent without a dated support matrix.

Differential testing compiles the same valid program with two backends and compares observable results. A disagreement proves that at least one path is wrong or that the test relied on unspecified behavior. It does not by itself identify the guilty backend. Unsafe undefined behavior, races, floating-point variation, and environment dependence must be excluded.

A disciplined backend triage ladder is:

1. Reduce to a small, valid Rust program.
2. Record rustc version, backend, target, and all flags.
3. Compare debug/release and incremental/clean builds.
4. Inspect optimized MIR for the intended operation.
5. Inspect backend IR before and after optimization.
6. Inspect assembly and object relocations.
7. Inspect linker command and final image.
8. Compare another backend or compiler revision when available.

Use rustc --emit=mir,llvm-ir,asm,obj where supported for stable output categories. Nightly -Z flags can expose more detail but are intentionally unstable. LLVM tools can verify, disassemble, and analyze IR or objects when versions match. Mismatched LLVM tool versions may reject valid newer bitcode or print misleading differences.

An LLVM verifier failure suggests malformed IR and is often actionable near lowering. An LLVM assertion may indicate malformed input or an LLVM defect. Wrong machine code with valid IR points later, but only after the IR's semantic claims are checked. A linker crash needs the exact input objects and command, not merely the Rust source.

Backend tests should pin semantic patterns while allowing harmless optimizer variation. Use run-pass tests for behavior, codegen tests for essential IR properties, and assembly checks for target-specific contracts. Add regression tests at the lowest layer that reliably catches the bug. Avoid checking every temporary name or instruction order.

30. Exercises, contribution paths, official sources, and synthesis#

Begin with observation before modification. Write a stable Rust function containing a branch, a loop, an enum match, and one destructor. Emit MIR, LLVM IR, assembly, and an object at optimization levels zero and three. Annotate which operations disappear at each boundary and which remain because they are observable.

Next, inspect the object with a format-appropriate tool. Find code, constants, symbols, and relocations. Explain one undefined symbol and identify the library or object expected to satisfy it. Then link a binary and use a map or symbol table to confirm the resolution.

Build a tiny cdylib with one extern "C" integer function. Inspect its export table and call it from a small C program if a C toolchain is available. Add a Rust String parameter only as a written counterexample: explain why it is not a C-compatible ABI. Design a buffer-and-length replacement with explicit ownership rules.

Measure CPU dispatch rather than assuming it helps. Implement scalar and target-feature-specialized paths on a supported architecture. Verify runtime detection, test both paths directly, and benchmark enough work to amortize dispatch. Run the baseline artifact on the oldest supported CPU class or emulator.

For an optimization exercise, vary CGU count and LTO policy in a nontrivial program. Record clean build time, edit build time, binary size, and benchmark performance. Explain why no single setting wins every column. Repeat measurements to separate noise from a real tradeoff.

For reproducibility, make two fresh builds in different absolute directories. Compare hashes, then inspect differing sections. Apply path remapping and deterministic native-tool options one at a time. Report byte identity separately from behavioral equivalence.

Good first contributions improve a reduced test, target documentation, diagnostics, or tooling around existing behavior. Backend correctness patches require evidence about the violated invariant and tests across relevant optimization levels. Performance patches require compile-time and runtime measurements plus code-size effects. Target changes need reviewers familiar with that ABI and hardware.

Read current official sources before relying on internal names:

• The rustc development guide: https://rustc-dev-guide.rust-lang.org/backend/codegen.html and adjacent backend chapters.
• The rustc book: https://doc.rust-lang.org/rustc/ for codegen, linking, targets, and unstable flags.
• The Rust Reference: https://doc.rust-lang.org/reference/ for language, ABI, attributes, and behavior contracts.
• The Rust platform support pages: https://doc.rust-lang.org/rustc/platform-support.html.
• LLVM language reference: https://llvm.org/docs/LangRef.html.
• LLVM code generator documentation: https://llvm.org/docs/CodeGenerator.html.
• Cargo reference: https://doc.rust-lang.org/cargo/reference/ for profiles, build scripts, and linkage orchestration.
• Cranelift codegen backend repository: https://github.com/rust-lang/rustc_codegen_cranelift.
• GCC codegen backend repository: https://github.com/rust-lang/rustc_codegen_gcc.

For a talk, begin with the contract stack rather than a tour of directories. Rust defines source behavior; rustc lowers chosen representations; LLVM optimizes under declared IR facts. The target ABI constrains calls and layout; object formats carry sections, symbols, and relocations. The linker resolves and lays out; the loader maps dynamic dependencies; hardware executes enabled instructions.

Then show one function crossing every layer. Use its place projection to explain layout, its branch to explain SSA, and its call to explain ABI. Use its destructor to contrast abort and unwind. Use its symbol and relocation to connect object inspection to final linking.

Close with the central engineering tradeoff. Global visibility enables stronger optimization, while partitioning enables parallelism and reuse. Rich debug and unwind data improve diagnosis, while increasing artifact and pipeline costs. Aggressive CPU features improve peak performance, while narrowing deployability. Alternative backends broaden design space, while multiplying conformance work.

The expert habit is attribution. Ask which layer owns the promise, which layer merely implements a current choice, and what evidence isolates the first divergence. Preserve exact artifacts and commands, reduce without introducing undefined behavior, and test counterfactual configurations. That method turns mysterious wrong code, link errors, and performance surprises into tractable backend engineering.

Part VII-C: Building and Debugging rustc's Backend#

31. Scope, version, and the concrete problem#

This continuation follows one question: how does checked Rust become a trustworthy target artifact? Its implementation claims are qualified against rustc 1.97.1. Internal module names, query boundaries, MIR pass order, vtable details, and backend traits are not stable APIs. The Rust Reference states language contracts; rustc source describes this release; LLVM documentation states LLVM IR contracts.

The pre-backend program still lacks answers a processor requires. A generic copy<T> has no fixed width. A trait call may have no direct address. An enum has variants but not necessarily a tag byte. A call has Rust types but not assigned registers. An external symbol has a name but no final address.

The smallest useful model is a chain of commitments.

roots
  | require concrete behavior
  v
instances + mono items
  | require representation
  v
layout + FnAbi + prepared MIR
  | partitioned work
  v
backend IR modules
  | optimization and instruction selection
  v
objects: sections + symbols + relocations
  | resolution and placement
  v
linked artifact

Every arrow forgets something and establishes an invariant. Monomorphization forgets type-parametric sharing but establishes concrete operations. Layout forgets source-level abstraction but establishes byte-level shape. LLVM lowering forgets much Rust provenance but establishes valid LLVM IR. Linking forgets most object boundaries but establishes resolved addresses.

The master invariant is not “LLVM accepts the module.” It is: every execution allowed by Rust and the program's unsafe contracts remains correctly represented. Malformed LLVM IR is an obvious failure. Valid IR containing a false noalias, alignment, or range claim is more dangerous because optimization can exploit the lie.

Use four categories throughout:

A failure should be assigned to the earliest broken invariant, not its latest symptom. An undefined symbol appears at link time but may begin as a missed monomorphization edge. Wrong code appears at runtime but may begin as an incorrect layout. An optimizer assertion may begin with invalid IR emitted by rustc.

32. Collection: roots, edges, and finite concrete work#

Monomorphization collection computes a conservative graph of concrete work. A node is broadly a function instance, static, or global assembly item. An edge means that one node requires another. Roots are needed without an incoming edge discovered inside the same analysis.

Executable entry machinery, externally retained definitions, reachable statics, and crate-type obligations can supply roots. The precise root rules are rustc policy and vary with linkage, visibility, attributes, and crate type. pub is not by itself synonymous with “one exported native symbol.”

Collection is demand driven because all generic instantiations cannot be enumerated. There are infinitely many possible array lengths and recursively composed types. The compiler emits combinations demanded by actual uses and cross-crate obligations.

fn wrap<T>(value: T) -> Option<T> {
	Some(value)
}

fn root() {
	let _ = wrap(3_u8);
	let _ = wrap(String::from("three"));
}

Conceptually, root creates edges to wrap::<u8> and wrap::<String>. The latter can expose destruction work even if wrap itself merely moves its argument. Optimization may later inline, merge, or remove symbols; collection still had to make behavior available.

Address-taking is an edge. A direct call is an edge. A constant containing a function pointer is an edge. A coercion to dyn Trait can create a vtable edge, and the vtable creates method and drop-glue edges. Inline assembly and runtime hooks need explicit treatment because ordinary MIR calls may not reveal them.

The soundness rule is asymmetric. Over-collection costs compile time and size. Under-collection can make a legal execution impossible. Dead-code elimination can repair excess, but no optimizer can recover a body never emitted.

Collection requires a visited set. Ordinary recursion then forms a graph cycle instead of unbounded compiler recursion. It also needs resource awareness: adversarial generic nesting can create a huge finite graph. Diagnostics should identify the expansion chain rather than merely exhaust memory.

An approximate collection cost model is:

time   = O(nodes + dependency edges + body scanning)
memory = O(nodes + edges + cached resolution)
output = sum(emitted body sizes) - later deduplication and elimination

Body scanning is not uniform; constant evaluation, normalization, and vtable construction can dominate an edge. Counting source functions therefore predicts neither collection cost nor binary size.

Full trace: collection.

trait Read {
	fn read(&self) -> u8;
}

struct Port(u8);

impl Read for Port {
	fn read(&self) -> u8 {
		self.0
	}
}

fn twice<T: Read>(x: &T) -> u16 {
	u16::from(x.read()) * 2
}

fn root() -> (u16, Box<dyn Read>) {
	let p = Port(7);
	(twice(&p), Box::new(Port(9)))
}

1. root is selected by its caller or export obligation.
2. Its static call requires twice::<Port>.
3. Resolving T::read requires the Read for Port method instance.
4. Box::new introduces allocator and ownership dependencies after normal resolution.
5. Coercion from Box<Port> to Box<dyn Read> requires a Port-as-Read vtable.
6. That vtable references the virtual method target, size, alignment, and destruction behavior in current rustc's scheme.
7. Destroying the returned box eventually uses erased-type destruction through metadata.
8. Partitioning later chooses where definitions live; it does not alter this semantic graph.

Do not hard-code this numbered list as rustc's query order. It is a dependency explanation, not a scheduling guarantee.

Experiment. Compile variants that remove the trait-object result, the static call, and both. Emit LLVM IR or assembly and compare demangled symbols. Predict which method remains before looking. Repeat at optimization level zero because optimized absence may mean elimination rather than missed collection.

33. Instances, substitutions, and dispatch identity#

A definition identifies source-level code. GenericArgs supplies ordered lifetime, type, and const arguments. An Instance identifies executable behavior after dispatch and adaptation decisions. The exact rustc 1.97.1 types and variants must be read in source; the conceptual distinction is durable.

Lifetimes normally erase before runtime representation. Types determine operations and often layout. Const arguments can determine layout, control flow, and symbol identity. Arguments inherited from an enclosing trait or impl make ordering easy to get wrong.

definition: Matrix<T, const R: usize, const C: usize>
arguments:             u16,             2,             3
result:    Matrix<u16, 2, 3>
layout need: size_of::<u16>() * 2 * 3, plus representation rules

Substitution must be complete before any operation needs concrete representation. Associated types may require normalization after substitution. Const expressions may require evaluation or canonical identity. A bare parameter reaching layout is evidence that the boundary contract broke upstream.

Instance resolution answers more than “which function?” It may select an impl method, intrinsic, closure body, drop glue, virtual-call adapter, or other shim. Checking proved that a call is valid; resolution must choose behavior consistent with that proof.

Four identities must not be conflated:

Optimization can merge equal bodies, so distinct instances need not have distinct final instruction bytes. Code placement can duplicate eligible definitions, so conceptual identity is not a promise of one address. Use documented APIs, not function or vtable pointer coincidence, for semantic identity.

Counterfactual dictionary passing would compile one generic body and pass operation tables at runtime. That reduces duplication but adds indirect operations and a uniform representation problem. Rust uses erasure selectively through trait objects while ordinary generic dispatch usually specializes. Neither strategy deletes cost; each moves it between compile time, code size, calls, and representation.

34. Shims, drop glue, and vtables#

A shim is an adapter with an addressable callable shape. It can adjust a receiver, bridge call conventions, adapt callable traits, or route erased dispatch. Centralizing adaptation prevents every caller from reproducing subtle ABI logic. Inlining may erase its runtime boundary, but collection and symbol reasoning still see it.

Closure environments are concrete aggregate values. Fn, FnMut, and FnOnce differ in receiver mode. A noncapturing closure can coerce to a function pointer because no environment is required. A capturing closure needs both behavior and environment; a plain address cannot carry the latter.

Drop glue destroys a concrete value correctly. It may call Drop::drop, then destroy fields in the required order, or be trivial. Generic code discovers whether destruction is nontrivial only after substitution.

The key drop invariant is exact initialization. Every initialized owner that remains live is dropped once on each cleanup path. No uninitialized or moved value is dropped. Drop flags and elaborated control flow represent uncertainty instead of guessing it away.

construct field a ---- success ----> construct field b ---- success ----> complete
       |                                  |
       | panic                            | panic
       v                                  v
drop nothing                         drop a only

With panic-unwind, cleanup paths execute while unwinding where supported. With panic-abort, process termination can remove those cleanup obligations. ManuallyDrop and MaybeUninit deliberately move responsibility to unsafe code; they do not permit treating arbitrary bits as a valid T.

A trait-object pointer carries data and metadata. For dyn Trait, metadata points to implementation data commonly called a vtable. Current rustc vtables support dispatch and layout/destruction needs, but their exact entries and order are not a stable foreign ABI.

&dyn Draw
  data ---------------------------> concrete object bytes
  metadata ---> vtable
                 | drop behavior
                 | size/alignment information
                 + method slots ----> concrete method instances/shims

Vtable construction creates collection edges. The virtual caller and producer must agree on slot, receiver shape, return ABI, and unwind contract. A wrong method with a correct data pointer suggests slot or table construction. A correct method receiving nonsense suggests data adjustment or ABI. A destruction-only crash suggests drop entry or ownership.

Security follows directly from these invariants. A forged vtable function pointer is control-flow corruption. A wrong size or alignment can corrupt allocation and deallocation. Calling through an ABI-incompatible slot can expose stale registers or overwrite stack state. Unsafe code must not fabricate trait objects from guessed layouts.

35. Layout: bytes, validity, niches, and unsized values#

Layout answers size, alignment, field offsets, ABI representation, and valid-value facts for a target. It is not only a packing algorithm. Validity facts influence enum encoding and optimization assumptions.

repr(Rust) permits implementation freedom subject to documented guarantees. repr(C) applies the target C-layout rules to qualifying aggregates. repr(transparent) establishes specified wrapper guarantees. repr(packed) lowers alignment but can make creating references to fields invalid.

A niche is a bit pattern excluded by a type's valid values and reused to encode another state. A non-null pointer can offer a null niche for an option-like enum where guarantees permit. Do not extrapolate one observed niche optimization into a stable layout promise.

straight enum: [ explicit tag | payload storage ]
niche enum:    [ payload whose otherwise-invalid pattern denotes empty ]

An unsized pointee has no statically fixed byte size. [T], str, and dyn Trait are important examples. Pointers to them carry metadata: a slice length or trait-object metadata. Such a pointer is often called fat, but exact ABI treatment remains target dependent.

&u32       = data pointer
&[u32]     = data pointer + element count
&str       = data pointer + byte count
&dyn Trait = data pointer + trait metadata

A scalar pair is an internal representation category often suitable for two-component pointers. It does not guarantee two machine registers. ABI classification may split it, pass it indirectly, or place components according to target convention.

Layout and validity must agree everywhere:

1. constants encode the same bytes as runtime stores;
2. field projections use the selected offsets;
3. discriminant reads match aggregate construction;
4. caller and callee classify the same value identically;
5. debuginfo describes the emitted representation;
6. drop glue projects the same fields;
7. allocation uses compatible size and alignment.

Prediction workshop. If local enum matching works but passing the enum to an extern "C" function fails, inspect FFI safety and ABI classification before local discriminant lowering. If only one optimized variant fails, inspect niche assumptions and invalid values. If only 32-bit fails, inspect pointer-width arithmetic and aggregate classification.

36. Function ABI and calling conventions#

A function ABI (FnAbi in current rustc terminology) is the concrete call contract. It classifies arguments and returns as ignored, direct, split, cast, or indirect forms as appropriate to the target and rustc model. Exact internal variants are version-sensitive.

Caller and callee must agree on:

• calling convention;
• register and stack locations;
• extension of narrow integers;
• aggregate decomposition;
• hidden return storage;
• alignment and dereferenceability claims;
• unwind permission;
• enabled target features where relevant.

Rust's default ABI is not a stable cross-version C ABI. extern "C" requests a target C convention, but only FFI-safe types and a complete ownership/error contract form a sound interface. String, Rust trait objects, and default-layout enums are not made portable by changing the ABI string.

Calling convention is target policy, not merely syntax. The same source signature can use different registers on x86-64 System V, Windows x64, AArch64, and WebAssembly. The host compiler process must never use host pointer size when classifying target calls.

Counterfactually, passing every value through a pointer simplifies classification. It increases memory traffic, inhibits scalar optimization, and requires storage ownership rules. Passing everything in registers fails for large values and finite register files. Real ABIs compromise for compatibility and machine cost.

37. MIR preparation and the codegen contract#

Backend lowering should not rediscover Rust semantics from source syntax. Prepared monomorphic MIR makes control flow, concrete types, assertions, drops, and cleanup explicit enough for codegen. The exact rustc 1.97.1 pass sequence must be checked in source rather than memorized from this guide.

Preparation separates language mechanisms from backend mechanisms. Drop elaboration decides what must be cleaned up. Codegen decides how branches and calls represent that decision. Constant evaluation decides a value where allowed. Codegen decides bytes and relocations.

A useful pass contract records:

Validation close to each transform reduces diagnostic distance. Without it, malformed MIR can survive until an LLVM verifier or machine crash. The earliest broken dump is stronger evidence than the final stack trace.

38. Codegen units, partitioning, and stable ownership#

A codegen unit (CGU) groups mono items into one backend module. Partitioning controls parallelism, optimization visibility, memory, and incremental reuse. It must also assign coherent symbol ownership and linkage.

mono-item graph
  | deterministic partition key
  +--> CGU A: definitions + declarations --> object A
  +--> CGU B: definitions + declarations --> object B
  +--> CGU C: definitions + declarations --> object C

One giant CGU maximizes local visibility but serializes expensive work and broadens invalidation. Many tiny CGUs increase parallel setup and can block inlining. LTO can restore some cross-unit visibility at later cost.

A practical cost model is:

clean build ≈ collection + max(parallel CGU costs) + coordination + link
edit build  ≈ invalidated CGUs + downstream effects + link
runtime     ≈ local optimization + imported/global optimization - code/cache costs

Stable partitioning is a performance invariant. An unrelated edit should not randomly move most items and invalidate cached objects. Deterministic hashes, ordering, symbol names, and option fingerprints support that goal.

Linkage mistakes produce three characteristic symptoms. No owner yields undefined symbols. Multiple strong owners yield duplicate definitions. Incorrectly mergeable owners can silently select incompatible contents.

39. LLVM IR: SSA, memory, poison, and provenance#

LLVM IR is typed control-flow IR, not portable assembly. SSA names each computed value once. Memory remains mutable and is ordered through loads, stores, calls, atomics, and LLVM's memory model.

entry:
  branch cond, left, right
left:
  branch join with value 10
right:
  branch join with value 20
join:
  value = phi selected by predecessor

This is explanatory pseudocode. Real syntax and opaque-pointer details depend on rustc's bundled LLVM.

Undefined behavior allows the optimizer to assume a forbidden event does not occur. Poison is a deferred invalid result propagated by many operations and dangerous at specified uses. undef, poison, frozen values, and uninitialized Rust storage are distinct LLVM concepts. Do not explain all optimizer surprises as “reading garbage.”

Attributes are proofs supplied to LLVM. Alignment, ranges, nonnull, aliasing, dereferenceability, and unwind properties can enable transforms. A weak claim loses performance. A false strong claim permits wrong code.

Pointer provenance concerns which allocation or authority a pointer is derived from, not just its numeric address. Rust's unsafe-code rules and LLVM's evolving pointer model meet here. Do not treat integer-pointer round trips or arbitrary address arithmetic as guaranteed merely because one target accepts them. Use current Rust unsafe-code guidance and LLVM LangRef for the exact operation involved.

The observation model defines optimization correctness. An optimizer may remove dead stores but not externally observable volatile operations. It may reorder ordinary independent accesses but must honor atomic ordering and synchronization. It may exploit UB but must preserve every defined execution.

40. Lowering places, expressions, calls, and unwind#

A MIR place denotes storage and projections such as field, dereference, index, or downcast. An operand may become an immediate scalar, scalar pair, or memory-backed value. An rvalue computes a value for a destination.

Place lowering uses layout offsets and alignment. Packed fields may require unaligned operations; creating an aligned reference to a misaligned field is not repaired by tolerant hardware. Index lowering includes bounds behavior unless an earlier proof removes it.

Integer lowering must preserve overflow policy, division checks, shift semantics, and cast rules. Enum lowering must construct and inspect the same niche or explicit-tag representation. Destination-passing can avoid large temporary aggregates and implement indirect returns.

A call combines instance identity and FnAbi. Direct calls name symbols or local values. Virtual calls load a slot from metadata. Indirect calls require a compatible function-pointer type and convention. Calls that may unwind need exceptional successors or target-specific equivalents when the panic strategy permits.

MIR call
  | classify args/return
  | select direct, indirect, virtual, intrinsic, or shim target
  | attach unwind contract
  v
LLVM call/invoke plus normal and cleanup blocks

SwitchInt may lower to an LLVM switch or branches. Machine lowering can later choose jump tables or comparison trees. Rust promises selected behavior, not a specific dispatch instruction.

Full trace: Option<&u8> call.

1. Collection selects the concrete caller and callee instances.
2. Layout decides whether the option uses the pointer's null niche for this contract.
3. FnAbi classifies the argument for the target.
4. MIR construction of Some(p) becomes the chosen valid pointer representation.
5. The call transports that representation according to the ABI.
6. The callee's discriminant test interprets the same representation.
7. A false nonnull claim on the whole option would be wrong if null denotes None.
8. LLVM may simplify the test only from valid facts.
9. Object emission records a relocation if the call target is not locally fixed.
10. Linking resolves the call address without changing option semantics.

41. Atomics, SIMD, and target-sensitive lowering#

Atomics combine an operation, width, alignment, and ordering. Ordering describes constraints among threads, not merely whether one instruction is indivisible. LLVM atomic orderings must faithfully represent Rust's atomic API contract.

Unsupported atomic widths may require library calls, locks, or rejection according to target support and operation. Silently lowering to a non-atomic load/store violates concurrency semantics. Misalignment can be invalid even when ordinary loads happen to work.

Volatile is not atomic synchronization. It preserves specified accesses for device or externally observed memory but does not create inter-thread happens-before relationships. Likewise, compiler fences and hardware fences have related but different roles.

SIMD lowering maps lane operations, masks, shuffles, and reductions to backend vector operations where available. Vector shape does not guarantee one machine instruction. Legalization may split wide vectors, scalarize them, or call helpers. Out-of-range shuffle indices and poison-sensitive operations require exact semantics.

Target features are execution preconditions. Compiling a function for AVX2 and calling it on a CPU without AVX2 can fault. Runtime detection and feature-specialized functions permit multiversioning, but all paths need tests. Inlining must not leak specialized instructions into a baseline caller.

Security review should ask whether untrusted input controls indexes, masks, lengths, alignment, or atomic addresses. Backend correctness does not replace source bounds checks; source validity does not excuse an incorrect lowering.

42. Optimization, target features, and honest performance work#

rustc optimizes MIR; LLVM optimizes IR; the target backend selects and schedules instructions. Inlining, scalar replacement, dead-code elimination, vectorization, alias analysis, legalization, and register allocation occur at different layers. A failure at -O can expose a lowering lie rather than an LLVM defect.

Optimization level is a policy bundle whose pass details can change. Target CPU and features define legal instructions and cost assumptions. target-cpu=native is unsuitable for broadly deployed binaries unless the deployment fleet matches the build host.

Performance has several currencies:

Shorter assembly is not necessarily faster. One instruction can have high latency; more code can avoid branches; inlining can improve computation while harming instruction-cache locality. Use representative data, warmup, repetitions, confidence intervals, and fixed toolchains.

For a regression, compare in order: optimized MIR, pre/post-optimization LLVM IR, assembly, object size, link map, runtime profile. Stop at the first meaningful divergence. Record CPU governor, affinity, target features, CGUs, LTO, panic strategy, debuginfo, and incremental state.

43. A minimal stable-Rust bytecode backend, progressively#

The following complete program is an educational compiler pipeline. It verifies a stack language, lowers it to bytecode, encodes an object-like container, decodes it, and executes it. It is not LLVM, not native code, not SSA, and not a rustc backend. Its purpose is to expose invariants without hiding them behind crates.

Instruction set policy is deliberately tiny: constants, arithmetic, print, and halt. The verifier establishes stack safety. Encoding establishes deterministic bytes. Loading validates the container before execution.

use std::fmt;

const MAGIC: &[u8; 4] = b"MOBJ";
const VERSION: u8 = 1;

#[derive(Clone, Copy, Debug, PartialEq, Eq)]
enum Op {
	Const(i32),
	Add,
	Mul,
	Print,
	Halt,
}

#[derive(Debug, PartialEq, Eq)]
enum Error {
	StackUnderflow { pc: usize, needed: usize, have: usize },
	MissingHalt,
	BadMagic,
	BadVersion(u8),
	Truncated,
	BadOpcode(u8),
	TrailingBytes,
}

impl fmt::Display for Error {
	fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
		write!(f, "{self:?}")
	}
}

fn verify(ops: &[Op]) -> Result<usize, Error> {
	let mut depth = 0usize;
	let mut maximum = 0usize;
	let mut halted = false;
	for (pc, op) in ops.iter().enumerate() {
		let (pop, push) = match op {
			Op::Const(_) => (0, 1),
			Op::Add | Op::Mul => (2, 1),
			Op::Print => (1, 0),
			Op::Halt => {
				halted = true;
				(0, 0)
			}
		};
		if depth < pop {
			return Err(Error::StackUnderflow { pc, needed: pop, have: depth });
		}
		depth = depth - pop + push;
		maximum = maximum.max(depth);
		if halted && pc + 1 != ops.len() {
			return Err(Error::TrailingBytes);
		}
	}
	if !halted {
		return Err(Error::MissingHalt);
	}
	Ok(maximum)
}

fn emit(ops: &[Op]) -> Result<Vec<u8>, Error> {
	verify(ops)?;
	let mut out = Vec::from(MAGIC.as_slice());
	out.push(VERSION);
	out.extend_from_slice(&(ops.len() as u32).to_le_bytes());
	for op in ops {
		match op {
			Op::Const(value) => {
				out.push(1);
				out.extend_from_slice(&value.to_le_bytes());
			}
			Op::Add => out.push(2),
			Op::Mul => out.push(3),
			Op::Print => out.push(4),
			Op::Halt => out.push(255),
		}
	}
	Ok(out)
}

fn take<const N: usize>(bytes: &[u8], at: &mut usize) -> Result<[u8; N], Error> {
	let end = at.checked_add(N).ok_or(Error::Truncated)?;
	let src = bytes.get(*at..end).ok_or(Error::Truncated)?;
	let mut result = [0; N];
	result.copy_from_slice(src);
	*at = end;
	Ok(result)
}

fn load(bytes: &[u8]) -> Result<Vec<Op>, Error> {
	let mut at = 0usize;
	if take::<4>(bytes, &mut at)? != *MAGIC {
		return Err(Error::BadMagic);
	}
	let version = take::<1>(bytes, &mut at)?[0];
	if version != VERSION {
		return Err(Error::BadVersion(version));
	}
	let count = u32::from_le_bytes(take::<4>(bytes, &mut at)?) as usize;
	let mut ops = Vec::with_capacity(count.min(1_000_000));
	for _ in 0..count {
		let opcode = take::<1>(bytes, &mut at)?[0];
		ops.push(match opcode {
			1 => Op::Const(i32::from_le_bytes(take::<4>(bytes, &mut at)?)),
			2 => Op::Add,
			3 => Op::Mul,
			4 => Op::Print,
			255 => Op::Halt,
			other => return Err(Error::BadOpcode(other)),
		});
	}
	if at != bytes.len() {
		return Err(Error::TrailingBytes);
	}
	verify(&ops)?;
	Ok(ops)
}

fn run(ops: &[Op]) -> Result<Vec<i32>, Error> {
	let capacity = verify(ops)?;
	let mut stack = Vec::<i32>::with_capacity(capacity);
	let mut output = Vec::new();
	for (pc, op) in ops.iter().enumerate() {
		match *op {
			Op::Const(value) => stack.push(value),
			Op::Add | Op::Mul => {
				let right = stack.pop().ok_or(Error::StackUnderflow {
					pc, needed: 2, have: stack.len(),
				})?;
				let left = stack.pop().ok_or(Error::StackUnderflow {
					pc, needed: 2, have: stack.len(),
				})?;
				stack.push(if *op == Op::Add {
					left.wrapping_add(right)
				} else {
					left.wrapping_mul(right)
				});
			}
			Op::Print => output.push(stack.pop().ok_or(Error::StackUnderflow {
				pc, needed: 1, have: 0,
			})?),
			Op::Halt => break,
		}
	}
	Ok(output)
}

fn main() -> Result<(), Error> {
	let source = [
		Op::Const(6), Op::Const(7), Op::Mul,
		Op::Const(1), Op::Add, Op::Print, Op::Halt,
	];
	let object = emit(&source)?;
	let loaded = load(&object)?;
	assert_eq!(run(&loaded)?, vec![43]);
	assert!(matches!(
		verify(&[Op::Add, Op::Halt]),
		Err(Error::StackUnderflow { pc: 0, .. })
	));
	println!("{} bytes, output 43", object.len());
	Ok(())
}

The format has a header and code payload but no symbols, sections, or relocations. The u32 instruction count is a format limit; production code must reject source lengths exceeding it rather than use as as above. The loader caps initial allocation but still accepts a million instructions and should impose byte, time, stack, and output limits for hostile input.

Progressive milestones:

1. Add named functions and direct-call instructions.
2. Collect reachable functions from an entry root.
3. Assign each function one stable symbol index.
4. Encode calls with relocation records rather than final indices.
5. Split code and read-only constants into sections.
6. Resolve relocations in a linker stage.
7. Add basic blocks and verify branch targets.
8. Convert stack values to SSA names and insert block parameters.
9. Add a target-independent optimizer with constant folding.
10. Preserve source spans for diagnostics.

At milestone four, a relocation should contain section, offset, kind, symbol, and addend. The linker must range-check patched values and reject duplicate strong definitions. At milestone seven, verification becomes dataflow: every predecessor must agree on stack shape at a block entry. At milestone eight, phi-like block parameters make joins explicit.

This model intentionally omits memory, pointers, provenance, concurrency, exceptions, debug formats, native ABI, and instruction selection. Calling it “a small LLVM” would be false. Its value is showing why validation and explicit contracts precede optimization.

44. Objects, symbols, sections, and relocations#

Native objects such as ELF, Mach-O, and COFF package bytes and unresolved relationships. WebAssembly has related but distinct conventions. Always identify the format before interpreting names or relocation kinds.

Sections group code, constants, mutable data, zero-filled storage, TLS, unwind records, and debug data. Symbols name definitions or references and carry binding and visibility. Relocations request a format-specific patch once placement is known.

.text caller: call [unknown displacement]
                   |
                   + relocation(kind, helper, addend)
.text helper: symbol helper at offset 0
                   |
                   v
linker places sections, computes displacement, checks range, patches call

Relocations are not all pointer-sized absolute addresses. They may be PC-relative, GOT-relative, TLS-specific, signed, narrow, paired, or architecture-specific. “Truncated to fit” can indicate range, code model, visibility, or a wrong relocation selection.

COMDAT-like grouping and weak/deduplicable linkage support repeated eligible definitions. The compiler must ensure grouped contents are compatible. Accidentally merging different instantiations is wrong even if their readable names resemble each other.

Preserve intermediate objects when debugging. Inspect headers, sections, symbols, then relocations with matching LLVM or platform tools. A final executable often hides which input supplied a definition.

45. Linking, native libraries, and artifact contracts#

The linker resolves symbols, extracts archive members, lays out sections, applies relocations, and emits the final image. It may dead-strip, fold, or synthesize platform structures. The loader later maps shared dependencies; successful linking does not prove runtime discovery.

Traditional static archives are demand-extracted. Order can matter when a linker processes unresolved references left to right. Whole-archive modes retain more but increase size and collision risk. Link maps reveal which member supplied each symbol.

Native libraries add ABI, allocator, exception, initialization, licensing, and deployment obligations. Build scripts execute on the host during cross compilation; libraries they produce for linking must target the target. Inspect object machine headers when host contamination is suspected.

Artifact choice follows the consumer:

None grants a stable Rust ABI automatically. Foreign boundaries need explicit ABI, FFI-safe layout, ownership, errors, threading, versioning, and unwind policy.

46. LTO, ThinLTO, and incremental reuse#

LTO carries optimizer-readable information across ordinary module boundaries. Fat LTO offers broad visibility at high memory and latency cost. ThinLTO exchanges summaries, plans imports, and preserves more parallelism. Exact rustc/LLVM pipelines and defaults are profile- and release-sensitive.

LTO cannot infer through incorrect declarations or unavailable foreign IR. Visibility and symbol interposition constrain assumptions. Inlining imports create dependencies that affect incremental invalidation.

Incremental reuse is a cache with a correctness obligation. A reusable CGU must match source meaning, compiler identity, target, options, dependency identities, and all relevant tracked inputs. False misses cost time. False hits produce stale wrong code.

Global optimization and incremental reuse pull in opposite directions. Global visibility couples modules; reuse rewards independence. Measure clean build, one-line edit build, link time, memory, output size, and runtime separately.

To investigate an incremental mismatch, preserve the failing cache, then compare with a clean build under identical commands. Reduce without deleting evidence. Do not attribute a long linker run to failed rustc reuse without timing both stages.

47. Debuginfo, unwind data, and panic#

Debug information maps optimized machine state back to source approximations. DWARF and CodeView/PDB are prominent formats. Optimization can eliminate, merge, split, and move variables, so perfect source stepping is impossible in general.

rustc creates metadata, LLVM transforms and emits it, the linker combines or strips it, and a debugger interprets it. A missing variable can originate at any boundary. Compare line tables, variable-location records, optimized IR metadata, and linker stripping policy.

Unwind information reconstructs caller state for panic, exceptions where permitted, profilers, and backtraces. panic=unwind needs cleanup and catch behavior on supported targets. panic=abort terminates and can omit Rust cleanup paths, though some unwind tables may remain for other purposes.

FFI boundaries must explicitly permit unwinding if unwinding can cross them. Otherwise catch and translate failure before crossing. A destructor panic during existing unwinding can abort, so “cleanup exists” does not guarantee graceful recovery.

Debug and unwind data cost object size, link work, storage, and sometimes optimization freedom. Removing them improves one metric while reducing diagnostics and profiling quality. That is moved cost, not free simplification.

48. Cross compilation and reproducibility#

Cross compilation separates build host, rustc host, and execution target. Build scripts run on the host. Target Rust libraries, native objects, sysroot, linker, and SDK must agree with the target ABI. A target triple alone cannot supply an operating-system SDK.

Test pointer width, endianness, alignment, atomics, TLS, unwind support, relocation range, and CPU baseline. Code generation success is weaker than execution compatibility. Use target hardware or a faithful emulator and inspect all linked objects.

Reproducibility requires a defined claim. Byte identity across directories is stronger than behavior equality on one host. Inputs include source, lockfile, compiler, linker, flags, environment, paths, timestamps, archive order, build IDs, and signing.

Build twice in fresh absolute paths. Compare hashes, then section tables and differing ranges. Apply path remapping and deterministic native-tool options one at a time. Stable iteration order and partitioning matter even when semantics do not.

Reproducibility is also a supply-chain control. It helps independent parties verify released bytes, but only if source provenance and complete build inputs are authenticated. Deterministic malware remains malware; reproducibility does not prove safety.

49. Alternative backends and rustc's backend API#

rustc_codegen_ssa contains shared coordination and traits; rustc_codegen_llvm implements the mature default backend. The interface covers values, types, builders, modules, constants, ABI lowering, codegen, and shared artifact work. It is an internal, evolving API shaped partly by LLVM concepts, not a stable plugin interface.

The Cranelift backend emphasizes fast development code generation and independent implementation. The GCC backend connects rustc to GCC infrastructure and may expand target opportunities. For rustc 1.97.1, exact platform support, feature completeness, bootstrap steps, and distribution status must be taken from each project's dated documentation. Do not claim drop-in parity without testing the required target and features.

An alternative backend must preserve rustc's layout and ABI decisions, MIR semantics, atomics, unwind behavior, debuginfo expectations, symbols, and artifacts. It need not copy LLVM's internal IR. Forcing every backend to imitate LLVM exactly would defeat design diversity; making the API too abstract can hide capabilities and impose inefficient lowest-common-denominator designs.

Differential testing compares observable results across backends. A difference means at least one implementation is wrong, the test invokes undefined behavior, or the observation is unspecified. Use safe reduced programs first; control floating-point, races, environment, and target features.

Backend API work requires bootstrap awareness. Read call sites and implementations, not only trait declarations. Changing a shared method may affect LLVM, Cranelift, GCC, tests, and bootstrap stages. Prefer a semantic operation over exposing one backend's incidental instruction when the abstraction can express it honestly.

50. Testing, diagnostics, security, and contribution practice#

Use the narrowest test that captures a durable contract. Run-pass tests establish behavior. Codegen tests establish essential IR properties. Assembly tests establish target-specific instruction contracts. UI tests establish diagnostics. Incremental tests establish dependency and reuse behavior.

Avoid checks for temporary SSA names, harmless instruction ordering, or exact optimizer spelling. A pattern test should fail when the invariant fails and survive equivalent transformations. Add tests at more than one optimization level when poison, attributes, aliasing, or cleanup is involved.

Useful test dimensions include:

• debug and optimized;
• panic abort and unwind;
• one and many CGUs;
• LTO off, thin, and fat where relevant;
• incremental and clean;
• 32-bit and 64-bit;
• little- and big-endian where available;
• baseline and feature-enabled CPU;
• LLVM and an alternative backend;
• static and dynamic linkage;
• malformed and resource-exhausting inputs.

Codegen diffs need semantic normalization. Compare control-flow shape, attributes, calls, loads/stores, and target operations rather than line count. Pair diffs with runtime tests because prettier IR can be wrong. Pair benchmarks with correctness tests because faster UB is not an improvement.

Compiler security includes adversarial compile-time behavior. Bound recursion, allocation, graph expansion, constant sizes, symbol lengths, and diagnostic output. Treat object parsers, native linkers, build scripts, and proc macros as additional trust boundaries. Never execute a produced test binary when the contribution protocol only requires compilation and the source is untrusted.

Symptom map:

A high-quality bug report preserves source, exact command, rustc -vV, target spec, linker version, environment, failing artifacts, and expected behavior. Bisect only after obtaining a deterministic reproducer. Reduce unsafe code carefully; deleting the operation that violates a contract can make a backend symptom disappear for the wrong reason.

Contribution route:

1. Reproduce on the intended rustc revision.
2. Locate the last good and first bad representation.
3. Read the owning module and nearby tests.
4. State the violated invariant before editing.
5. Implement the smallest semantic fix.
6. Add a durable regression test.
7. Run targeted tests, formatting, and relevant bootstrap checks.
8. Measure compile time, runtime, and size if policy or optimization changes.
9. Request target/backend experts when ABI or LLVM contracts are involved.

Likely source-reading areas in the rustc tree include mono collection and partitioning, rustc_middle instance/layout types, rustc_ty_utils layout-related code, MIR transforms, rustc_codegen_ssa, rustc_codegen_llvm, target specifications, and compiletest suites. Directories move; search by semantic type and query on the pinned revision.

51. Laboratories and mastery exercises#

Laboratory 1: concrete reachability. Write one generic function used by two types, one unused function, one function pointer, and one trait-object coercion. Predict mono items. Compare low-optimization symbols. Explain every extra shim or glue symbol you can justify and mark uncertain ones as implementation details.

Laboratory 2: layout. Measure size_of, align_of, and field addresses for carefully selected representations on two targets. Separate guaranteed facts from observations. Add Option, slices, trait objects, repr(C), and a packed counterexample. Never create a misaligned reference.

Laboratory 3: ABI. Export integer and repr(C) aggregate functions to C. Inspect both declarations and assembly. Test normal return and errors. Design buffer ownership and prohibit unwinding across the boundary.

Laboratory 4: poison-aware optimization. Find one LLVM LangRef example involving poison and freeze. Predict legal transformations before running tools. Explain why it cannot be copied directly into a claim about arbitrary Rust source.

Laboratory 5: object anatomy. Emit one object with a call and static string. Identify format, architecture, sections, defined and undefined symbols, and relocations. Link it and locate each input in a map.

Laboratory 6: panic paths. Use destructors with visible logging and a deliberate panic. Compare abort and unwind artifacts and behavior. Do not treat output buffering as destruction evidence without flushing or direct synchronization.

Laboratory 7: partition cost. Vary CGUs and LTO on a nontrivial crate. Measure clean time, edit time, peak memory, size, and representative runtime. Explain why the winner differs by column.

Laboratory 8: backend differential. Run a safe integer/control-flow corpus with LLVM and an available alternative backend. Compare results, not assembly identity. Reduce one difference and determine whether it is correctness, unspecified output, or unsupported functionality.

Laboratory 9: harden the bytecode pipeline. Reject instruction counts above a fixed limit before allocation. Add branch targets and dataflow verification. Add source offsets to every error. Fuzz arbitrary byte arrays through load and require no panic.

Capstone. Add functions, symbols, code/data sections, relocations, a linker, and an SSA-like block IR to the educational pipeline. Specify each binary field and limit. Test duplicate symbols, undefined symbols, overflow, malformed branches, deterministic output, and incremental function reuse. Document what remains unlike a native backend.

Debugging workshop: a trait-object program fails only with LTO on AArch64. First validate source and unsafe contracts. Compare prepared MIR. Compare vtable constants and call ABI before LTO. Compare imported IR after ThinLTO. Inspect target features and relocation results. The first changed incorrect representation identifies the owner more reliably than the final crash.

52. Derived philosophy, teaching plan, and authoritative reading#

Representations determine cheap questions. Mono-item graphs make reachability cheap but duplicate generic bodies. SSA makes value dependence cheap but pushes mutable state into memory analysis. Objects make separate production cheap but defer addresses to relocations and linking.

Abstractions move responsibility. Trait objects move code duplication into metadata and indirect dispatch. Many CGUs move global optimization into LTO. repr(C) moves layout choice toward a platform contract but does not solve ownership. Caches move computation into invalidation proofs.

Optimization preserves meaning only under an observation model. The backend must encode that model honestly through calls, atomics, volatile accesses, unwind edges, validity, and attributes. An optimization bug may be the first visible failure long after lowering broke the first invariant.

Identity is layered. A definition, instance, symbol, section offset, and runtime address answer different questions. Confusing them causes fragile vtable tricks, symbol collisions, and incorrect deduplication assumptions.

Diagnostics depend on provenance retained earlier. Source spans, instance names, section ownership, relocations, and incremental fingerprints cost space and engineering effort. Discarding them simplifies local structures while making later failures harder to attribute.

A technically honest 45-minute talk can use this sequence:

1. Five minutes: one generic trait call and the missing concrete decisions.
2. Seven minutes: roots, instances, shims, drop glue, and vtables.
3. Seven minutes: layout and ABI with one fat pointer trace.
4. Seven minutes: prepared MIR to SSA, including poison and attributes.
5. Five minutes: CGUs, optimization, and cost model.
6. Five minutes: objects, relocations, and linking.
7. Four minutes: LTO, incremental reuse, and reproducibility.
8. Five minutes: alternative backends and the earliest-broken-invariant method.

State that directory and trait details are rustc 1.97.1 observations, not Rust promises. Show one full trace rather than dozens of disconnected flags. End with a real object inspection and a semantic diff, not an assembly beauty contest.

Authoritative reading map, to be pinned to the rustc 1.97.1 source revision where implementation details matter:

• Rust Reference, representations, behavior, ABI, attributes, and inline assembly: https://doc.rust-lang.org/reference/.
• Standard library primitive and pointer documentation: https://doc.rust-lang.org/std/.
• rustc development guide, monomorphization: https://rustc-dev-guide.rust-lang.org/backend/monomorph.html.
• rustc development guide, code generation: https://rustc-dev-guide.rust-lang.org/backend/codegen.html.
• rustc development guide, backend and LLVM chapters: https://rustc-dev-guide.rust-lang.org/backend/.
• rustc command-line book, codegen and linking options: https://doc.rust-lang.org/rustc/codegen-options/.
• rustc platform support: https://doc.rust-lang.org/rustc/platform-support.html.
• rustc 1.97.1 source tag/tree for normative implementation evidence: https://github.com/rust-lang/rust/tree/1.97.1/.
• LLVM Language Reference matching rustc's bundled LLVM: https://llvm.org/docs/LangRef.html.
• LLVM atomics guide: https://llvm.org/docs/Atomics.html.
• LLVM code generator documentation: https://llvm.org/docs/CodeGenerator.html.
• System V ABI and target processor supplements for relevant ELF targets: https://gitlab.com/x86-psABIs/x86-64-ABI and architecture-specific authorities.
• Microsoft PE/COFF specification: https://learn.microsoft.com/windows/win32/debug/pe-format.
• Apple Mach-O ABI reference and LLVM object tooling documentation for Mach-O inspection.
• Cargo profiles and build scripts: https://doc.rust-lang.org/cargo/reference/profiles.html and https://doc.rust-lang.org/cargo/reference/build-scripts.html.
• Cranelift backend project: https://github.com/rust-lang/rustc_codegen_cranelift.
• GCC backend project: https://github.com/rust-lang/rustc_codegen_gcc.
• Rust unstable book for explicitly unstable flags and features: https://doc.rust-lang.org/beta/unstable-book/.

Read guarantees before source, source before folklore, and target ABI before interpreting assembly. Record the exact revision because LLVM, rustc internals, alternative backends, and target support evolve independently. The durable expert skill is not memorizing one pipeline diagram; it is proving where meaning first diverged.

Part VIII: Queries, Incremental Compilation, Metadata, and Diagnostics#

This part targets rustc 1.97.1 internals. Compiler internals are not a stable API, so names and boundaries can move even when the ideas remain useful. Every command shown with -Z requires a matching nightly toolchain and should be checked with rustc -Z help. The goal is not merely to list structures, but to explain why the compiler is built this way and how to investigate it safely.

1. The compiler as a system#

An absolute beginner often imagines a compiler as a conveyor belt. Text enters parsing, then name resolution, type checking, optimization, and machine-code generation in one fixed order. That picture is useful, but incomplete for rustc. Rustc combines an eagerly orchestrated pipeline with a demand-driven database of computations called queries. “Eager” means a coordinator deliberately starts work. “Demand-driven” means a result is calculated when a caller asks for it, rather than at one globally fixed moment.

The broad data progression still matters:

command line and files
        |
        v
tokens and AST --> resolved HIR --> types --> MIR --> codegen units --> artifacts
        |              |            |        |             |
        +--------------+------------+--------+-------------+
                       query dependencies

AST is the abstract syntax tree, close to what the programmer wrote. HIR is the high-level intermediate representation, a compiler-friendly form after important lowering and resolution work. MIR is the mid-level intermediate representation, a control-flow form used for borrow checking and optimization. A codegen unit, or CGU, is a partition handed toward the backend.

The arrows do not mean that every node in one layer is completed before any request reaches the next. An orchestrator may request crate analysis; analysis requests the type of one definition; that query requests predicates and another definition's type. The resulting execution order is the order demanded by actual dependencies. Other activities remain explicit phases because they perform broad coordination, mutate phase-owned state, emit files, or predate query conversion.

This hybrid is intentional. Queries make dependencies observable, cache repeated computations, support incremental validation, and offer a natural boundary for selected concurrency. Eager roots make policy visible: whether this invocation checks only, emits metadata, produces an executable, or stops after an error. Do not claim that rustc is “just a query engine,” that every front-end operation is a query, or that all compiler work runs fully in parallel by default.

The central invariant is semantic equivalence. For the same compiler, target, options, source, dependencies, and relevant environment, cached execution must produce the same accepted program, errors, and artifacts as a clean execution. Performance is negotiable; correctness is not.

2. Entering through rustc_driver#

rustc_driver is the outer executable-facing layer. It handles process-level concerns such as command-line invocation, compiler setup, callbacks, and the final exit status. Tools that embed rustc internals commonly enter near this layer rather than calling a random analysis function. This is an unstable, nightly-only integration surface.

The conceptual call chain is:

rustc main or embedding tool
    -> rustc_driver
    -> build rustc_interface::Config
    -> run_compiler(config, callbacks)
    -> create global compiler state and Session
    -> enter compiler context
    -> choose analysis and emission roots

Exact signatures vary by nightly. Treat the following as shape, not copy-paste API:

struct MyCallbacks;

// Conceptual only: consult the 1.97.1 nightly API for exact methods.
impl Callbacks for MyCallbacks {
    fn config(&mut self, config: &mut Config) {
        // Adjust approved configuration before the compiler is created.
    }

    fn after_analysis(&mut self, compiler: &Compiler, tcx: TyCtxt<'_>) -> Compilation {
        // Inspect analysis under the compiler's lifetime rules.
        Compilation::Continue
    }
}

fn main_like(config: Config) {
    let mut callbacks = MyCallbacks;
    run_compiler(config, &mut callbacks);
}

Config describes an invocation: input, output choices, parsed options, diagnostics setup, file loading, and other hooks. Callbacks gives a driver client controlled interception points. run_compiler conceptually establishes compiler-owned lifetimes and runs the invocation. It is not a promise that arbitrary callbacks may retain TyCtxt or arena references after the compiler context ends.

Why use callbacks instead of forking the whole driver? They let tools observe or stop at supported checkpoints while the driver still enforces initialization and teardown. Why are callbacks not a stable plugin API? Compiler representations and timing change together; freezing them would constrain compiler evolution.

An embedding bug often comes from treating callback timing as mere convenience. Reading analysis before it exists, keeping session-local IDs across runs, or emitting diagnostics after their context has gone away violates ownership and phase assumptions. Start from a known driver pattern in the same toolchain revision and pin the toolchain.

3. Session, options, target, and diagnostic context#

The Session represents one compilation session. It gathers invocation-wide facts and services rather than the semantic answer for one function. Examples include compiler options, target information, source mapping, diagnostic machinery, crate configuration, and output-related policy. The exact fields are internal and should not be memorized as an API.

Options are not cosmetic. Optimization level can change generated work products. Edition can change parsing and name resolution. Feature gates can change accepted syntax. Lint caps can change whether a warning becomes an error. The target specification changes pointer width, ABI, data layout, available target features, and linking behavior. Therefore relevant options and target facts must participate in invalidation or in construction of a fresh incremental cache namespace.

The diagnostic context is the service through which structured diagnostics become terminal text, JSON, counts, and error-state evidence. It knows emission policy, but a diagnostic normally carries the actual message, labels, notes, and suggestions. The SourceMap translates internal byte positions into human-facing file, line, and column locations.

Keep three identities separate:

1. The process may invoke the compiler more than once.
2. Each invocation has a Session and session-local allocations.
3. Incremental data may describe an older session using stable identities.

A DefId or byte position useful now is not automatically meaningful in the next invocation. Serialization must convert unstable identity into an explicitly stable representation or reconstruct it against current state.

Session initialization establishes invariants before semantic work begins. The target must be known before target-sensitive layout. The source map must own source files before spans can be rendered. Diagnostic policy must be coherent before errors are emitted. Incremental state must be loaded or rejected before query reuse.

When debugging an option-dependent stale result, ask two questions. Was the option tracked as an input to the relevant dependency node? Was the old cache rejected at a broader compatibility boundary? Either strategy can be sound; silently doing neither cannot.

4. Compile, check, and emit roots#

Demand-driven does not mean “nothing happens until a user calls an arbitrary query.” The driver and interface select roots based on the requested operation. A root is a top-level demand or orchestration step whose dependencies pull in lower-level work.

cargo check asks rustc for enough front-end and analysis work, plus appropriate metadata, to validate a crate without producing ordinary final machine code for it. A normal build asks for code generation and linking outputs as required by crate type and --emit settings. --emit=metadata emphasizes metadata output. Other emit kinds can request MIR, LLVM IR, assembly, object files, dependency information, or linked output, subject to current compiler support.

Conceptually:

check root
  -> parse/lower/resolve coordination
  -> type checking and required analyses
  -> linting and diagnostics
  -> metadata needed by dependents

build root
  -> everything required for check
  -> monomorphization collection
  -> MIR optimization and backend translation
  -> codegen work products
  -> link when requested

This is not an exact call graph. Some work can be shared, reordered, conditionally skipped, or represented by queries at different granularity. Metadata may need information that sounds “late,” and codegen can request type-system facts on demand.

The root design expresses policy and gives a place to check global error state. After fatal front-end errors, requesting codegen would waste time and might encounter invalid assumptions. After recoverable errors, analysis can continue far enough to discover useful independent diagnostics. The driver decides when that continued work remains safe.

A common tracing mistake is to run cargo check and conclude that a codegen query is dead because it did not appear. The requested roots differ. Always record crate type, emit settings, incremental settings, and whether Cargo reused an artifact before comparing traces.

5. Why queries and passes coexist#

A pass usually walks a broad representation and performs a planned activity. A query names a computation by a key and returns a result. For example, “type-check this body” is naturally keyable by the body's definition identity, while “coordinate all outputs requested by this invocation” is naturally orchestration.

Queries offer memoization. Memoization stores the answer to a function call so a repeated call with the same key can reuse it. They also record dynamic dependencies: the engine learns that query A used query B because A actually called B. This is more precise than declaring that all type checking depends on all HIR.

Passes remain valuable when work is inherently batch-shaped, when ordering itself is part of policy, when a subsystem has controlled mutation, or when conversion cost exceeds likely benefit. The backend illustrates the nuance. Codegen uses query-produced facts and integrates with dependency tracking, but LLVM is not simply a forest of ordinary rustc queries. CGU work products have manual incremental integration.

The architecture is therefore two graphs superimposed:

orchestration: prepare -> analyze -> codegen -> link
                         |       |
query demands:       A -> B -> C +-> D
                         \-> E

An eager phase can call queries. A query provider can invoke other queries. An eager coordinator can iterate definitions and demand a per-definition query to ensure diagnostics are produced. Laziness alone would otherwise leave an erroneous but never-demanded body unchecked.

The key tradeoff is granularity. One crate-wide query has little overhead but invalidates too much. One query per tiny expression may create huge bookkeeping, hashing, and synchronization costs. Rustc chooses boundaries where results are reusable and dependencies meaningful, then adjusts hot spots empirically.

An architectural bug is often not “passes are bad” or “queries are slow.” It is a boundary mismatch: mutable hidden state inside a supposedly deterministic query, an eager walk that bypasses tracking, or a giant result that makes every small edit appear global.

6. Rustc's custom query engine#

Rustc has its own query engine. It is explicitly not Salsa, even though Salsa is a Rust library built around related incremental, demand-driven ideas. Using “Salsa” as a generic nickname obscures important implementation and API differences. For target 1.97.1, study the current rustc tree and nightly docs, not an obsolete story centered on a standalone rustc_query_system crate.

Current declarations and generated plumbing live chiefly around rustc_middle::query and macros supplied from rustc_macros, with dependency-graph implementation in rustc's current compiler crates. These locations can move. The important structure is stable enough to learn:

query declaration
   + key type
   + result type
   + modifiers and policies
        |
        v
generated TyCtxt methods, caches, descriptors, dispatch
        |
        v
provider table entry -> provider function

TyCtxt, pronounced “type context,” is the principal handle used to ask semantic questions. A call shaped like tcx.some_query(key) enters generated query plumbing rather than directly calling an arbitrary helper. The plumbing checks cache state, records dependency edges, coordinates jobs, and invokes a provider if calculation is required.

The query key identifies one invocation. It might be a definition ID, crate number, type, or compound key. The result must obey its query's lifetime, hashing, serialization, and recovery rules. Many results are arena-allocated or interned so returning them is cheap and identity is controlled.

Macros reduce enormous repetition. They do not make the system magical. When debugging, expand the mental model: method call, key normalization, cache lookup, active-job handling, dependency task, provider dispatch, result hashing, and cache insertion.

The provider is semantic implementation; generated plumbing is execution policy. Calling a provider directly can bypass caching and dependency tracking and is generally wrong unless a very specific internal pattern requires it.

7. Providers: local and external knowledge#

A provider is the function that computes a query on a cache miss or forced recomputation. Compiler crates populate provider tables with implementations from type checking, traits, MIR, metadata, and other subsystems. This registration avoids one giant module knowing every implementation.

Many definition-keyed queries distinguish local definitions from external definitions. A local definition belongs to the crate currently being compiled. Its provider can inspect local HIR, resolution results, and local bodies. An external definition belongs to an upstream crate. Its answer is usually decoded through crate metadata rather than by reading the upstream source or rerunning its compiler front end.

tcx.type_of(local DefId)
    -> local provider
    -> local lowered representation

tcx.type_of(extern DefId)
    -> extern provider
    -> crate store
    -> decode upstream metadata

External providers are mainly metadata-backed. That boundary is central to separate compilation. The downstream compiler knows exported signatures, predicates, selected attributes, trait information, and other encoded facts needed to compile uses. It intentionally does not receive a complete replayable copy of every upstream compiler state.

Some queries use one provider because the key or implementation already handles locality. Others have generated support for separate external provision. Modifiers and exact table organization are version-sensitive. Do not infer an API rule from one query declaration.

Provider registration has a completeness invariant. Every reachable query/key category must have a valid implementation or an intentional diagnostic path. A missing provider is a compiler construction error and commonly becomes an internal compiler error, or ICE. An incorrect external provider can masquerade as a metadata decode failure or return a semantically stale answer.

When adding a query, inspect analogous current queries. Check where providers are registered, whether external keys are legal, whether results are hashable, whether disk caching is justified, how cycles recover, and whether diagnostics are replayable.

8. Dynamic dependency recording#

Dependency recording is dynamic because edges reflect calls made during this execution. Suppose optimized_mir(f) asks for mir_built(f), typing_env(f), and the layout of one used type. The engine records those reads while the provider runs. If another function uses different types, its dependency set differs.

optimized_mir(f)
  |-- mir_built(f)
  |-- typing_env(f)
  `-- layout_of(Vec<u8>)

The active query stack provides context. Entering a query creates a query job or task. Calls made beneath it can add dependency edges from the parent node to child nodes. Leaving finalizes the node's result and fingerprint according to policy.

Dynamic recording has a profound requirement: all semantic inputs must be tracked. If a provider reads a file, environment variable, global atomic, wall clock, random value, mutable table, or command option outside tracked mechanisms, the dependency graph cannot know that the input changed. That is an untracked input. It can produce a stale green result.

Not every incidental read should become a dependency. Profiling timestamps must not alter semantic fingerprints. Logging configuration should control observation, not accepted programs. The discipline is to separate semantic inputs from instrumentation and to mark special computations appropriately.

Dependencies are often ordered in the saved graph. Order matters during green validation because a changed branch condition may mean later old dependencies would not be read now. Validating an old later edge before the branch-defining edge could execute work that is no longer legal or reachable.

The stack also enables cycle diagnostics. If A demands B and B demands A in the same dependency chain, the engine can report the cycle with meaningful frames. This is preferable to stack overflow, but recovery must still return a result consistent with that query's type and error policy.

9. Caches, purity, and side effects#

The in-memory query cache maps a query key to completed state. The first caller computes; later callers reuse. This assumes query-like determinism: relevant inputs and dependencies determine the answer. Rustc documentation often calls queries pure, but production reality requires nuance.

Some query computations produce diagnostics as side effects. If a cached value is reused without rerunning the provider, those diagnostics must not silently disappear. Rustc can capture, store, and replay diagnostic side effects for suitable queries. Other effects may force always-evaluate behavior or remain outside ordinary query caching. “Diagnostics can be replayed” does not license arbitrary file writes or mutation in a provider.

Useful purity questions are:

• Would two executions with identical tracked inputs return equivalent results?
• Would they produce the same structured diagnostics?
• Is every semantic read represented by a dependency or explicit input?
• Is every externally visible effect repeated, replayed, or deliberately orchestrated elsewhere?
• Can execution order change without changing semantics?

Cache insertion must not expose a half-built result. On panic or fatal failure, a query job cannot be marked successfully complete. Waiters must wake into a defined failure path rather than read uninitialized state. This is one meaning of poisoning: failed computation prevents unsafe reuse of its slot. Rustc's exact panic and fatal-error machinery is internal, but the invariant is general.

Error recovery complicates “same result.” A sentinel error value can let independent analysis continue, yet caching it is safe only if it depends on the original error-producing inputs and carries appropriate error evidence. Returning a normal-looking fabricated type without an error token can suppress later required errors.

Side-effect bugs often appear as duplicated diagnostics on recomputation or missing diagnostics on cache hits. Investigate whether emission happened directly, was captured for replay, and was also emitted by an eager caller.

10. DepGraph, DepNode, and DepKind#

The dependency graph, DepGraph, is the incremental system's record of which tracked computations depend on which others. A DepNode identifies a dependency-graph node. Its DepKind says what category of computation or input it represents. Its remaining identity is a stable fingerprint of the key or unit represented by that kind.

Conceptually:

struct DepNode {
    kind: DepKind,
    key_fingerprint: Fingerprint,
}

The real types and encodings are optimized and version-sensitive. Do not confuse a dependency node with a cached result. The graph can know that a node existed, what it read, and what result fingerprint it had even when the full result was never persisted.

Edges point according to a defined dependency convention. When reading diagrams or APIs, verify whether “edge A to B” means A reads B or B invalidates A. Natural-language sources sometimes reverse the drawing while preserving the same relationship. State the convention before reasoning about traversal.

The graph from the previous session is immutable evidence. The current session builds a new graph as work is validated or executed. Green nodes are promoted into the new session's history with the dependencies needed for the next run. This two-graph view prevents a common misunderstanding: rustc is not mutating yesterday's graph in place until it becomes today's graph.

Dep kinds encode policies such as whether a key can be reconstructed, whether a result is hashable, or whether special handling applies. Generated query plumbing connects query definitions to dep kinds. Input dep nodes represent facts that do not come from ordinary query providers, such as tracked source structure or configuration.

A graph with too many edges loses reuse but can remain correct. A graph missing one real edge can be fast and wrong. Optimization should first establish soundness, then remove false dependencies through smaller results, projection queries, or better boundaries.

11. Stable identity and stable hashing#

In-session identity is optimized for one run. DefId contains a crate-local indexing scheme suitable for quick lookup. CrateNum assignments can vary. Interned type pointers and arena addresses are process-local. Raw BytePos values depend on how files were loaded into the source map. None should be blindly written to disk as permanent meaning.

Cross-session identity uses stable descriptions. A DefPath describes how a definition is reached through compiler-assigned path components. A DefPathHash is a stable hash of that identity, including crate identity as appropriate. On decoding, rustc maps stable identity back to a current-session DefId.

Stable hashing recursively converts data into a session-independent digest. The result is commonly represented by a 128-bit Fingerprint. Hashing a definition uses stable identity, not its current numeric index. Hashing an unordered semantic set must not depend on hash-table iteration order. Hashing a span requires deliberate provenance policy rather than an address.

current DefId --stable hashing--> DefPathHash/Fingerprint
                                           |
next session DefId <--reconstruction--------+

Stable does not mean standardized forever. It means stable enough across compatible incremental sessions under the compiler's cache protocol. Compiler version changes can invalidate the whole cache. Metadata formats and stable-hash implementations are not public compatibility guarantees.

Hash collisions are theoretically possible. Rustc uses high-quality wide fingerprints so accidental collision risk is negligible, while accepting hashing cost. The more practical danger is an incomplete or nondeterministic hash implementation. Forgetting a semantic field causes false green. Including allocation order causes false red and unstable cache behavior.

Moving an item should ideally preserve identities of unaffected definitions, but path construction has difficult cases such as anonymous or duplicate-shaped nodes. Incremental tests and stable-hash assertions guard these cases. Never “fix” unstable hashes by sorting with an unstable session-local key.

12. The red-green algorithm#

Red and green describe a particular query invocation in the current comparison with the previous session. Green means its result is known equivalent to the previous result. Red means it changed, is new, or must conservatively be treated as changed. Color is not attached permanently to the query definition. type_of(A) can be green while type_of(B) is red.

The basic algorithm asks whether a previous node can be marked green. This operation is commonly called try-mark-green.

try_mark_green(Q):
    find yesterday's node corresponding to Q
    if absent: fail; Q must execute
    validate yesterday's dependencies in recorded order
    if all are green: mark Q green without executing Q
    otherwise: fail; execute Q

execute Q:
    record today's dependencies
    fingerprint today's result
    compare with yesterday's result fingerprint
    equal  -> Q is green (early cutoff)
    unequal -> Q is red

The striking step is early cutoff. An input may change, forcing Q to execute, but Q's output may remain identical. Then dependents of Q need not execute. For example, editing a private function body may change its MIR while leaving its exported type and metadata projection unchanged.

If all old dependencies validate green, deterministic execution implies Q's result is unchanged. The compiler can mark Q green without loading the full old result. If a caller needs the value, rustc either loads a persisted result or recomputes according to that query's cache policy. Color and value availability are distinct.

Recorded dependency order is significant. Stop validation at the first red dependency because the provider might now choose a different branch and never read later old dependencies.

Red-green correctness rests on complete dependencies, deterministic providers, stable fingerprints, compatible compiler state, and correct reconstruction. Breaking any premise turns an optimization into miscompilation or missed diagnostics.

13. Force and reconstructing keys#

Try-mark-green starts from a current query key, so rustc can compute the corresponding stable dep-node and find yesterday's node. Recursive validation is harder. Yesterday's graph contains a child dep-node, but validating it may require executing that child's query. Execution needs the typed query key, not merely an opaque fingerprint.

For suitable dep kinds, rustc can reconstruct or recover the current-session key from stable identity. It can then force the query: invoke the machinery needed to calculate or validate that dep node even though no ordinary semantic caller directly requested it. Definition-keyed nodes are often reconstructible through stable definition identity. Not every key has a practical inverse from fingerprint to rich typed value.

old DepNode(kind, key fingerprint)
             |
             v
recover current key, if supported
             |
             v
force query provider / validation

This explains query policies such as “no force.” If a node cannot safely or cheaply reconstruct its key, recursive green marking must take a conservative path. The exact modifier names and implementation are version-sensitive, but the conceptual constraint is unavoidable.

Forcing is not arbitrary eager compilation. It is validation work justified by an old dependency edge. It must preserve query stack and cycle behavior, and it must not perform forbidden side effects merely because an old graph mentioned a node.

A reconstruction bug can map yesterday's node to the wrong current definition. Symptoms include stale results, surprising cycles, or an ICE while validating code that appears unrelated to the edit. Log both stable dep-node identity and reconstructed current key. Do not compare only numeric DefId debug output across sessions.

The best query key is therefore not only convenient for callers. It should have clear semantic identity, stable hashing, appropriate reconstruction policy, and manageable cache size.

14. On-disk incremental state#

In-memory memoization helps within one compiler run. Incremental compilation adds persistence between compatible runs. The incremental directory stores forms of the previous dependency graph, query-result cache data, and indexes for reusable work products. The current rustc_incremental crate supports loading and saving this state; core dep-graph types and query integration live elsewhere in current rustc.

Not every query result is serialized. Serialization has CPU cost, disk cost, schema complexity, and stable-identity requirements. A cheap query may be faster to recompute. A huge result may cost more to decode than it saves. A result containing unsuitable session-local references may not be safely persistent. Rustc can still save its result fingerprint and dependency edges without saving the value.

For selected queries, a disk-cache policy permits encoding the result. On a later run, if the dep node validates green, rustc may decode that result rather than execute the provider. Cache promotion ensures useful green results do not accidentally vanish merely because they were validated without being loaded during the current run.

Work products are backend artifacts such as codegen-unit outputs. They are tracked separately from ordinary typed query values. If a CGU's dependencies remain green and the artifact is present, rustc can copy or reuse it rather than regenerate it. An index connects dep nodes to files and validates availability.

Incremental session publication is transactional in spirit. Incomplete or errorful sessions must not become trusted future state. Temporary session data is finalized only when safe; invalid sessions are discarded.

Compatibility includes compiler version, relevant options, target, crate identity, and cache format. Broad invalidation is preferable to decoding incompatible bytes. Deleting the incremental directory must always be a safe recovery action. It may lose speed, never semantics.

15. Invalidation, crate identity, and options#

An incremental cache belongs to a compilation configuration, not merely a source directory. Changing the target from x86-64 to ARM changes layout and code generation. Changing panic strategy can change generated paths. Changing enabled features can alter parsed and configured items. Changing compiler revision can alter every internal representation.

Rustc and Cargo partition or invalidate caches using identity and compatibility data. A stable crate identity, influenced by the crate name and metadata configuration, distinguishes crates that share a textual name but arise in different compilation contexts. A crate hash summarizes identity-relevant information and supports upstream dependency tracking. Exact formulas are internal and evolve.

Upstream invalidation must propagate. If crate a changes metadata relevant to crate b, b cannot reuse answers based on old a. But a private implementation edit in a may leave its exported metadata fingerprint unchanged, allowing portions of b to remain green. This is the cross-crate form of early cutoff.

a source edit
   -> a local query changes
   -> exported metadata projection unchanged?
          yes: many b dependencies stay green
          no:  dependent b nodes validate or recompute

Option tracking has two levels. Coarse compatibility rejects the whole old cache. Fine-grained input nodes let only dependent computations turn red. The right level depends on implementation cost and expected reuse.

An environment variable read by a build script normally affects generated source through Cargo's dependency protocol, outside rustc's internal query graph. An environment variable read directly by a rustc provider requires explicit tracking or conservative evaluation. Do not assume the build system will rescue hidden compiler inputs.

When comparing incremental behavior, preserve the exact command line and environment. The crate hash changing is evidence, not a diagnosis. Find which identity component changed and whether that scope is intended.

16. Soundness and clean-build equivalence#

Incremental compilation is sound when reuse cannot change observable compiler semantics relative to a clean build. The practical oracle is clean-build equivalence:

incremental(old source -> edited source) == clean(edited source)

Equality includes success or failure, diagnostics that policy promises, metadata, and generated program behavior. Binary bytes may differ for legitimate nondeterministic or path-related reasons, so tests often compare selected artifacts, hashes, or execution results.

The most dangerous bug is a missing edge. If query A reads semantic input X without recording it, X can change while A remains green. False extra edges merely cause unnecessary recomputation. This asymmetry justifies conservative tracking.

Typical untracked inputs include direct filesystem reads, mutable global state, hash-map order leaking into output, option fields omitted from hashing, source text accessed outside tracked source structures, and external tool output. Side effects introduce a dual problem: an answer may be correct while a required diagnostic or file is absent.

Incremental test suites compile revisions in sequence and assert which nodes should be clean or dirty. Rustc has internal attributes and dep-graph assertion support specifically for compiler tests; syntax is internal and should be copied from current tests. A robust regression also compiles the final revision from a fresh directory and compares behavior.

Useful manual protocol:

1. Reproduce with a fixed nightly and exact command.
2. Save the incremental directory before experimentation.
3. Confirm deleting it fixes the symptom.
4. Compare incremental and clean diagnostics or artifacts.
5. Narrow the edit and identify the first incorrectly green node.
6. Audit all provider reads and stable hashing for that node.
7. Add a multi-revision regression test.

Never fix a stale result solely by marking a vast query always-red unless correctness urgently requires a conservative stopgap. Find the missing dependency and document any temporary breadth.

17. Concurrency, query jobs, and waiters#

The query cache is also a coordination point. If two workers request the same uncached key, rustc should compute it once. The first request owns a query job. Other requests become waiters and sleep or cooperate until the job completes. Completion publishes the result and wakes waiters.

worker 1: request Q -> owns job -> computes -> publishes -> wake
worker 2: request Q -> waits ---------------------------> reads

Synchronization must preserve dependency recording for each caller. The waiter depends on Q even though it did not run Q's provider. Failure must also wake waiters. A lost wakeup hangs the compiler; premature publication exposes incomplete data; duplicate ownership wastes work and can duplicate side effects.

Cross-thread cycles are harder than recursive cycles. Worker one can own A and wait for B while worker two owns B and waits for A. The engine tracks active jobs and waiting relationships so it can detect deadlock-shaped query cycles and apply query-specific cycle reporting or recovery. Cycle handling is semantic, not just locking. A trait-cycle recovery value differs from a layout cycle's policy.

Parallel query capability does not mean rustc's entire front end is query-driven or that normal compiler builds execute every query in parallel. Some compiler parallelism is conditional, some data structures impose serial regions, some orchestration remains eager, and backend parallelism has its own mechanisms. Toolchain build configuration and current feature policy matter. Avoid promises such as “rustc compiles all functions simultaneously by default.”

Concurrency exposes hidden impurity. A provider depending on execution order may pass serial tests and fail under parallel scheduling. Diagnostics need deterministic ordering or explicit sorting where promised. Interning and arenas need synchronized or partitioned access.

When debugging a hang, collect query job ownership, query stacks for workers, and waiter edges. A CPU profile alone may show sleeping threads but not the semantic wait cycle.

18. Cycles, poisoning, and recovery#

A query cycle occurs when a demanded answer depends, directly or indirectly, on itself before completion. Some cycles correspond to invalid Rust, such as infinitely recursive type relationships. Others reveal a rustc dependency-design bug. The query stack gives a path that can be transformed into a user diagnostic or ICE report.

type_of(A)
  -> predicates_of(B)
      -> type_of(A)   cycle closes

Not all cycles are treated identically. A query declaration can specify cycle recovery suited to its result. Recovery may emit an error and return an error-marked sentinel so analysis can continue. Other cycles are fatal because no sound placeholder exists. The recovery result must carry evidence that an error was already reported.

Poisoning addresses abnormal termination of a query job. If a provider panics or raises a fatal compiler condition, its cache entry cannot become a normal completed answer. Waiters must not continue as if a valid result exists. Cleanup must unwind active-stack and dependency-task state without publishing corrupt graph data.

Error recovery seeks more useful diagnostics without cascades. A cascade is a flood of secondary messages caused by one primary error. An “error type” can unify permissively and suppress conclusions that would only restate the original problem. But excessive suppression misses independent errors. Recovery boundaries are therefore semantic design choices.

Cycle bugs can be intermittent if stable identity or parallel waiting is wrong. Classify first:

• same-thread query stack repeats: direct semantic cycle;
• cross-thread owners wait on each other: deadlock cycle;
• only incremental run cycles: stale edge, reconstruction, or changed branch validation;
• only malformed input cycles: likely missing recovery path;
• valid code cycles after refactor: provider dependency regression.

Always preserve the smallest query stack and source reproducer in a regression test.

19. Crate metadata and rmeta#

Separate compilation requires a downstream crate to understand an upstream crate without reparsing and rechecking all upstream source. Rustc encodes a compiler-private crate metadata format. An .rmeta artifact is metadata-focused output used especially by checking and dependency pipelines. It is analogous to a rich compiled interface, not a stable language-level header format.

Metadata can include exported item identities and visibility, types and generic predicates, trait and implementation information, selected attributes, symbol and linkage facts, stability information, and other data needed by downstream compilation. Exact contents are demand- and version-sensitive. Some information is encoded lazily or through tables so the decoder need not materialize everything.

What downstream intentionally does not know is equally important. It does not receive arbitrary upstream session state, query caches, source ownership, or every private body as a general promise. Privacy and optimization needs determine exposure. The format is not intended for third-party long-term archival or cross-version interoperability.

Generic code complicates the boundary. Downstream monomorphization may need an upstream generic function's MIR or related optimized representation because machine code depends on downstream type arguments. Selected MIR is therefore encoded where cross-crate use requires it. This is not equivalent to shipping all source or all MIR for every item. Inline and const evaluation needs can also influence encoded bodies.

upstream source
  -> local queries
  -> metadata encoder
  -> .rmeta
  -> downstream metadata decoder
  -> external query providers

Metadata is both an artifact and a query boundary. An external provider makes decoded data look like a TyCtxt query answer, allowing downstream code to use a common semantic interface for local and external definitions.

20. Crate store, decoding, and external providers#

The crate store, often discussed as the cstore, tracks loaded external crates and exposes decoded metadata to compiler queries. It maps crate identities, resolves stable encoded references, and supports lazy access to tables or blobs. Exact modules and trait boundaries change; follow current rustc_metadata and nightly documentation.

Decoding must translate identities. An encoded reference to an upstream definition uses metadata-level crate and definition identity. The downstream session assigns its own CrateNum and DefId values. The decoder maps between them while preserving which crate and definition were meant. Spans may map to imported source-file information with provenance retained.

Lazy decoding saves memory and startup work. If downstream never asks for a private-like detail that was encoded for another purpose, it need not decode it. Queries provide a convenient demand point. Caching then avoids decoding the same answer repeatedly.

Metadata decode failures have several families:

• incompatible compiler or metadata version;
• truncated or corrupt artifact;
• crate identity mismatch;
• invalid table offset or malformed encoding;
• failure mapping stable definition or source identity;
• provider requesting data not encoded under its contract.

User-facing rustc normally prevents incompatible metadata from reaching deep decoding by version and crate-hash checks. An ICE in a decoder can still indicate cache corruption or a compiler bug. First retry after a clean build; then preserve the offending artifact and exact producer/consumer versions.

External query answers must behave like local answers semantically, but their implementation paths differ. A bug that affects only downstream crates often belongs in metadata encoding, decoding, or external provision rather than the local analysis provider. A focused test should use at least two crates so it actually crosses the boundary.

21. rlib, dylib, and proc-macro boundaries#

An rlib is a Rust static-library artifact used by rustc. It packages crate metadata with native object code and, depending on settings, other backend material such as bitcode. An .rmeta contains metadata without the ordinary full library code payload. A Rust dylib is a dynamically linked Rust library and also carries metadata needed by rustc consumers. These formats and the Rust ABI are compiler-private and version-sensitive.

Do not equate an rlib with a stable C archive. For a stable foreign interface, projects usually expose a C ABI through staticlib or cdylib as appropriate, with explicit ABI-compatible types. Rust-to-Rust artifacts are expected to be rebuilt with compatible toolchains and settings.

Crate hash and disambiguator information prevents accidental identity collision and feeds symbol naming and dependency validation. Two packages can both declare a crate called util; their compilation contexts must remain distinct. Changing an upstream crate can alter the downstream dependency identity even if filenames happen to match.

Proc macros create a stronger process and compilation boundary. A proc-macro crate is compiled for the host because its code executes during compilation. It exchanges token streams through the proc-macro interface rather than exposing its internal TyCtxt or query graph to the consuming crate. Generated tokens enter the consumer's parsing and expansion provenance.

Consequences include:

• cross-compiling may involve host artifacts for proc macros and target artifacts for normal crates;
• proc-macro behavior is an input to the expanded consumer source;
• crashes and diagnostics can cross a tool boundary;
• source spans from generated tokens need call-site or definition-site provenance;
• arbitrary proc-macro side effects are not ordinary rustc query dependencies.

Cargo's build graph and fingerprinting complement rustc's intra-crate dep graph. Do not expect rustc's query engine to see every dependency of an external proc-macro process or build script.

22. SourceMap and source files#

Diagnostics begin with bytes, but users think in filenames, lines, columns, and macro calls. SourceMap is rustc's mapping service between those worlds. It owns or references SourceFile records containing a file's name, source text when available, line starts, and provenance-related data.

BytePos is a position in rustc's global source-coordinate space for the session. A source file occupies a byte interval. Adding a relative byte offset to that file's start yields a global position. This makes spans compact and lookup efficient, but positions are session-local.

global byte space
0 ........ file A ........ 140 | 141 .... generated file B .... 260
             ^ BytePos(42)              ^ BytePos(190)

A Span roughly combines a low byte position, a high byte position, and a SyntaxContext. The high endpoint is conventionally exclusive. SyntaxContext records macro hygiene context: which expansion marks affect how identifiers resolve and where source came from. The real compact representation and APIs are internal.

Line and column display requires lookup in the containing SourceFile. Columns are subtle because bytes, Unicode scalar values, grapheme clusters, and terminal display width differ. Rustc's renderer owns the current policy; tools should use source-map APIs rather than inventing byte_offset + 1 columns.

Source may be remapped for reproducible paths. Some imported metadata spans refer to upstream files whose full text is unavailable. Synthetic spans may not correspond cleanly to user text. Diagnostic code must tolerate these cases and choose useful fallbacks.

The source-map invariant is containment: a renderable span's positions must map coherently to source files, or be recognized as dummy/synthetic. Combining low and high positions from unrelated files creates broken labels and often an ICE.

23. Spans, hygiene, and macro provenance#

A span answers two distinct questions. Where are the bytes? Under what expansion context should this syntax be understood? Ignoring the second question breaks hygienic macros and produces misleading diagnostics.

Macro expansion creates provenance chains. A token may originate in a macro definition, be substituted from an argument at the call site, and pass through nested expansions. Rustc can walk expansion data to produce a macro backtrace. The diagnostic renderer may show the immediate generated location and notes identifying invocations that produced it.

user call span
   -> expansion of outer!
       -> argument token or definition token
           -> expansion of inner!
               -> reported span

Call-site span usually points to the invocation context. Definition-site span points toward macro definition context. Mixed-site behavior reflects hygiene rules rather than a simple filename choice. Procedural macros can assign spans available through their token API, but cannot forge arbitrary access to rustc internals.

When constructing a suggestion, use a span whose source text the user can edit. A machine-applicable replacement aimed at generated or unavailable text is dishonest. Walk to a suitable call site or reduce applicability. Macro backtraces explain provenance, but too much expansion detail can overwhelm beginners; rendering policy balances precision and noise.

Incremental hashing of spans is delicate. Absolute byte positions shift when unrelated earlier files change. Stable hashing must use source-file identity and relative positions or an appropriate span hashing policy. Including volatile spans in a large semantic query result can create false red changes. Omitting semantically important hygiene can create false green changes.

Span bugs often look cosmetic, but hygiene influences name resolution and therefore semantics. Separate “wrong underline” from “wrong SyntaxContext caused wrong binding,” and test both expansion behavior and rendered location.

24. Diagnostics are structured products#

A diagnostic is not a preformatted string. It is structured data with a level, primary message, primary span, labels, notes, helps, suggestions, diagnostic code, and child messages. Rustc's diagnostic builder, often seen through Diag-related APIs, accumulates this structure and then emits it through the diagnostic context. Exact type names and lifetimes evolve.

error[E....]: primary message
  --> src/lib.rs:3:9
   |
 3 |     bad + value
   |     ^^^ label explaining this expression
   |
   = note: supporting fact
   = help: an action the user can take

Labels tie explanations to spans. Notes provide facts that may not suggest action. Helps provide guidance. Suggestions pair one or more edits with explanatory text and an applicability rating. Applicability tells tools how confidently an edit can be applied: machine-applicable requires exact syntax-preserving confidence and no hidden placeholders.

Builder-style construction reduces premature emission. Code can add context discovered along an error path and emit once. Failing to emit may trigger safeguards because a diagnostic builder was abandoned. Emitting twice duplicates output and error counts.

Diagnostics are semantic outputs of queries when their presence depends on that query's analysis. Caching must replay captured diagnostics when needed. However, presentation can vary by output mode, color, terminal width, and localization without changing the semantic analysis result.

Good diagnostic invariants are:

• one primary explanation for one root problem;
• spans point to relevant editable source when possible;
• child messages add information rather than repeat the title;
• suggestions parse after application under stated assumptions;
• emission updates error state exactly once;
• cached execution neither loses nor duplicates the diagnostic.

25. Error guarantees, delayed bugs, and recovery#

Rustc uses error-guarantee tokens to represent proof that an error has been emitted. Instead of returning an unadorned failure or fabricated normal value, a function can return a token such as ErrorGuaranteed or embed it in a result. The exact APIs are internal, but the type-level idea is important.

An error token prevents code from claiming “this impossible state has been handled” when no diagnostic was actually issued. It also lets later phases propagate failure without printing the same root error repeatedly. Sentinel types and values can carry or be associated with this evidence.

A delayed bug records an internal inconsistency that may be a consequence of already-invalid user input. Rustc delays turning it into an ICE until it knows compilation did not already fail for an expected user error. If no ordinary error explains the inconsistency, the delayed bug surfaces as a compiler bug. Delayed bugs must not become a trash bin for violated invariants.

Recovery has three goals in tension:

1. continue far enough to report independent mistakes;
2. avoid cascades derived from nonsense;
3. never proceed into code generation with invalid assumptions.

For example, after an unresolved type, an error type can flow through unification. Operations involving it avoid asserting additional mismatches that are merely consequences. An unrelated borrow error in another body can still be found. The driver later checks emitted-error state and prevents unsafe output.

Incremental reuse adds a guarantee invariant. If a cached result contains an error sentinel, replay must restore the corresponding emitted diagnostic or error state. Otherwise a later phase sees “error occurred” with no user explanation, or worse, believes compilation succeeded.

When debugging a missed error, trace both value flow and guarantee flow. When debugging an ICE after a user error, locate where code discarded or failed to test the guarantee before relying on a normal invariant.

26. Fluent, JSON, and stable presentation boundaries#

Rustc supports message localization infrastructure based on Fluent concepts. Messages can be identified separately from source code, with arguments substituted into localized templates. Not every diagnostic is equally migrated at every revision, and internal APIs change. The design separates semantic construction from human-language rendering.

A Fluent message should not force translators to reconstruct Rust syntax from fragmented clauses. Arguments need meaningful names and types. Labels and suggestions may have separate message identifiers. Localization changes wording, not spans, diagnostic level, or applicability truth.

JSON diagnostics expose structured output for Cargo, IDEs, and tools. The JSON includes rendered text plus structured spans and children according to current schema. Consumers should use documented stable command-line JSON contracts where available, not deserialize rustc-private Diag structures. Terminal layout is for humans and can change.

analysis fact
   -> structured diagnostic
       |-> Fluent-selected human messages
       |-> terminal emitter
       `-> JSON emitter for tools

Source snippets in JSON require source availability and valid mappings. Macro expansion information may appear through expansion fields or child diagnostics depending on current schema. Tools must tolerate optional fields and evolving non-stable details.

Snapshot tests are useful for presentation but can overconstrain harmless wording. Pair them with semantic assertions: diagnostic code, primary span, suggestion edits, and applicability. For compiler UI tests, use current normalization conventions so paths and unstable hashes do not make output flaky.

The stability boundary matters. Error codes and documented JSON options have user-facing expectations. Nightly internal constructors and Fluent identifiers are compiler implementation details. An external tool should invoke rustc and consume supported output rather than link to internal diagnostics crates unless it deliberately pins nightly.

27. Lints, levels, caps, and expectations#

A lint is a diagnostic check with configurable severity. Common levels are allow, warn, deny, and forbid. Allow suppresses emission, warn emits a warning, deny promotes to an error, and forbid prevents lowering the level in a nested scope. Exact interactions include command-line and attribute precedence rules documented by rustc.

Lint caps place an upper bound on effective severity. Cargo uses caps in dependency contexts so a dependency's warnings do not usually break a downstream build. The cap changes diagnostic policy and must be represented in invocation state. It should not silently alter the underlying semantic fact found by the lint.

Lint expectations, written with #[expect(...)], say that a lint is expected to trigger. The compiler tracks whether an expectation was fulfilled and can diagnose unfulfilled expectations. This requires identity and bookkeeping beyond simply suppressing a warning. Incremental reuse must preserve or replay fulfillment correctly; otherwise an edit can produce a false unfulfilled-expectation warning.

Future-compatibility lints warn about code accepted today that is expected to become rejected or change meaning later. Cargo can summarize such warnings from dependencies. They allow ecosystem migration before a language change becomes a hard error. Do not describe them as all ordinary warnings: their reporting and downstream summaries have special policy.

Lint execution spans several points in the compiler because different facts become available at different representations. Early lints can inspect syntax; late lints can inspect typed HIR; MIR-related checks may use still later facts. Not all linting is one query or one pass.

A duplicated lint can come from visiting an owner twice, replaying cached diagnostics in addition to eager emission, or giving one logical expectation multiple unstable identities. A missed lint can come from an undemanded owner, wrong level resolution, stale query result, or recovery suppression. Debug effective level and execution separately.

28. A miniature typed query engine#

The following safe Rust program demonstrates typed keys, dependency recording, caching, recursive validation, and early cutoff. It intentionally supports one query family, runs on one thread, stores values in memory, and uses integers instead of 128-bit stable fingerprints. It is a teaching model, not rustc code and not Salsa.

use std::collections::{BTreeMap, BTreeSet};

#[derive(Clone, Copy, Debug, Eq, Ord, PartialEq, PartialOrd)]
struct Key(u32);

#[derive(Clone, Debug)]
struct OldEntry {
    value: i64,
    fingerprint: u64,
    deps: Vec<Key>,
}

#[derive(Clone, Debug)]
struct Entry {
    value: i64,
    fingerprint: u64,
    deps: Vec<Key>,
}

struct Db {
    inputs: BTreeMap<Key, i64>,
    old_inputs: BTreeMap<Key, i64>,
    old: BTreeMap<Key, OldEntry>,
    now: BTreeMap<Key, Entry>,
    stack: Vec<(Key, Vec<Key>)>,
    validating: BTreeSet<Key>,
    executions: BTreeMap<Key, usize>,
}

impl Db {
    fn fingerprint(value: i64) -> u64 {
        // Not collision-resistant; rustc uses stable, wide fingerprints.
        value as u64 ^ 0x9e37_79b9_7f4a_7c15
    }

    fn record_read(&mut self, key: Key) {
        if let Some((parent, deps)) = self.stack.last_mut() {
            if *parent != key && !deps.contains(&key) {
                deps.push(key);
            }
        }
    }

    fn input_changed(&self, key: Key) -> bool {
        self.inputs.get(&key) != self.old_inputs.get(&key)
    }

    fn try_mark_green(&mut self, key: Key) -> bool {
        if self.now.contains_key(&key) {
            return true;
        }
        let Some(old) = self.old.get(&key).cloned() else {
            return false;
        };
        if !self.validating.insert(key) {
            panic!("query cycle while validating {key:?}");
        }
        if self.input_changed(key) {
            self.validating.remove(&key);
            return false;
        }
        for dependency in old.deps.iter().copied() {
            if !self.try_mark_green(dependency) {
                self.validating.remove(&key);
                return false;
            }
        }
        self.validating.remove(&key);
        self.now.insert(
            key,
            Entry {
                value: old.value,
                fingerprint: old.fingerprint,
                deps: old.deps,
            },
        );
        true
    }

    fn get(&mut self, key: Key) -> i64 {
        self.record_read(key);
        if let Some(entry) = self.now.get(&key) {
            return entry.value;
        }
        if self.try_mark_green(key) {
            return self.now[&key].value;
        }
        self.execute(key)
    }

    fn execute(&mut self, key: Key) -> i64 {
        if self.stack.iter().any(|(active, _)| *active == key) {
            panic!("query cycle while executing {key:?}");
        }
        *self.executions.entry(key).or_default() += 1;
        self.stack.push((key, Vec::new()));

        // Keys 0 and 1 are inputs. Other keys sum their two predecessors.
        let value = if key.0 < 2 {
            self.inputs[&key]
        } else {
            self.get(Key(key.0 - 1)) + self.get(Key(key.0 - 2))
        };

        let (finished, deps) = self.stack.pop().unwrap();
        assert_eq!(finished, key);
        let fingerprint = Self::fingerprint(value);
        self.now.insert(key, Entry { value, fingerprint, deps });
        value
    }
}

#[test]
fn unchanged_graph_is_reused() {
    let old = BTreeMap::from([
        (Key(0), OldEntry { value: 2, fingerprint: Db::fingerprint(2), deps: vec![] }),
        (Key(1), OldEntry { value: 3, fingerprint: Db::fingerprint(3), deps: vec![] }),
        (Key(2), OldEntry { value: 5, fingerprint: Db::fingerprint(5), deps: vec![Key(1), Key(0)] }),
    ]);
    let inputs = BTreeMap::from([(Key(0), 2), (Key(1), 3)]);
    let mut db = Db {
        inputs: inputs.clone(), old_inputs: inputs, old, now: BTreeMap::new(),
        stack: vec![], validating: BTreeSet::new(), executions: BTreeMap::new(),
    };
    assert_eq!(db.get(Key(2)), 5);
    assert!(db.executions.is_empty());
}

#[test]
fn changed_input_recomputes_dependents() {
    let old = BTreeMap::from([
        (Key(0), OldEntry { value: 2, fingerprint: Db::fingerprint(2), deps: vec![] }),
        (Key(1), OldEntry { value: 3, fingerprint: Db::fingerprint(3), deps: vec![] }),
        (Key(2), OldEntry { value: 5, fingerprint: Db::fingerprint(5), deps: vec![Key(1), Key(0)] }),
    ]);
    let mut db = Db {
        inputs: BTreeMap::from([(Key(0), 2), (Key(1), 4)]),
        old_inputs: BTreeMap::from([(Key(0), 2), (Key(1), 3)]),
        old, now: BTreeMap::new(), stack: vec![], validating: BTreeSet::new(),
        executions: BTreeMap::new(),
    };
    assert_eq!(db.get(Key(2)), 6);
    assert_eq!(db.executions[&Key(1)], 1);
    assert_eq!(db.executions[&Key(2)], 1);
}

The model's validation rejects a node immediately when its same-key input changed. Real rustc has many dep kinds, stable key reconstruction, disk serialization, diagnostics, work products, and concurrency. The sample computes fingerprints but does not yet use equality for early cutoff propagation after execution. The next section adds that focused mechanism.

29. Adding early cutoff to the model#

After executing a dirty query, compare its new fingerprint with yesterday's fingerprint. If equal, the node is semantically green even though it had a red dependency. The miniature engine can record a color separately from value storage.

#[derive(Clone, Copy, Debug, Eq, PartialEq)]
enum Color { Green, Red }

fn classify_after_execution(
    key: Key,
    new_fingerprint: u64,
    old: &BTreeMap<Key, OldEntry>,
) -> Color {
    match old.get(&key) {
        Some(previous) if previous.fingerprint == new_fingerprint => Color::Green,
        _ => Color::Red,
    }
}

#[test]
fn equal_projection_cuts_off_change() {
    // Imagine this query returns only whether an input is nonzero.
    fn projected_fingerprint(input: i64) -> u64 {
        Db::fingerprint((input != 0) as i64)
    }

    let old = BTreeMap::from([(
        Key(10),
        OldEntry {
            value: 1,
            fingerprint: projected_fingerprint(7),
            deps: vec![Key(0)],
        },
    )]);

    // The semantic input changed from 7 to 9, but the projection stayed true.
    assert_eq!(
        classify_after_execution(Key(10), projected_fingerprint(9), &old),
        Color::Green,
    );
}

To integrate this fully, execute would store the color and try_mark_green would accept a dependency that executed but classified green. It would stop at the first dependency classified red. That distinction is why a Boolean “was executed” is not enough.

The sample uses the result value as its simplistic fingerprint. Production stable hashing must include every semantic field, normalize session-local identity, and avoid nondeterministic iteration. The sample clones old entries freely; rustc carefully avoids expensive decoding. The sample's old input map is an explicit input oracle; rustc represents many inputs as dep nodes and compatibility state.

Safe Rust prevents memory races here, but does not guarantee incremental correctness. Removing one dependency from deps compiles safely and returns stale values. Type safety and dependency-graph soundness are separate properties.

An instructive extension is a Parity(Key) query whose output stays equal when an input changes from 7 to 9, plus a dependent formatting query. Assert that parity executes once, becomes green, and formatting does not execute. Then deliberately omit the input edge and observe the false reuse. This experiment makes early cutoff and missing-edge unsoundness concrete without rustc's scale.

30. Rendering a simple span diagnostic#

This standalone example renders one source line with a byte-range underline. It uses only safe standard-library Rust and deliberately restricts input to a single ASCII line. That restriction avoids pretending byte offsets equal display columns for Unicode or tabs.

use std::ops::Range;

#[derive(Debug)]
struct SimpleDiag<'a> {
    level: &'a str,
    message: &'a str,
    file: &'a str,
    line: usize,
    span: Range<usize>,
    label: &'a str,
}

fn render(source: &str, diag: &SimpleDiag<'_>) -> Result<String, &'static str> {
    if !source.is_ascii() {
        return Err("this teaching renderer accepts ASCII only");
    }
    if source.contains('\n') {
        return Err("this teaching renderer accepts one line only");
    }
    if diag.span.start >= diag.span.end || diag.span.end > source.len() {
        return Err("invalid byte span");
    }

    let spaces = " ".repeat(diag.span.start);
    let carets = "^".repeat(diag.span.len());
    Ok(format!(
        "{level}: {message}\n --> {file}:{line}:{column}\n  |\n{line:>2} | {source}\n  | {spaces}{carets} {label}\n",
        level = diag.level,
        message = diag.message,
        file = diag.file,
        line = diag.line,
        column = diag.span.start + 1,
        source = source,
        spaces = spaces,
        carets = carets,
        label = diag.label,
    ))
}

#[test]
fn renders_primary_label() {
    let source = "let answer = nope;";
    let diag = SimpleDiag {
        level: "error",
        message: "cannot find value",
        file: "demo.rs",
        line: 1,
        span: 13..17,
        label: "not found in this scope",
    };
    let text = render(source, &diag).unwrap();
    assert!(!text.contains("13 | nope")); // Column is shown in the header, not here.
    assert!(text.contains("^^^^ not found in this scope"));
    assert!(text.contains("demo.rs:1:14"));
}

The assertions check both the human label and the one-based column calculation. Examples should be compiled rather than merely admired, especially when they teach position arithmetic.

Rustc's renderer handles multiline spans, overlapping labels, Unicode display widths, tabs, color, elision, source remapping, unavailable source, macro provenance, suggestions, and JSON. Its Span uses global positions plus syntax context, not a range relative to an arbitrary string. The miniature renderer demonstrates validation and structure, not a replacement.

31. Detailed edit trace through the compiler#

Consider an existing crate with a generic function and a call:

pub fn choose<T: Clone>(x: T) -> T {
    x.clone()
}

fn use_it() {
    let n = choose(21_u32);
    println!("{n}");
}

Now edit 21_u32 to 22_u32. Lexing and parsing for the affected source owner observe changed bytes. Lowered HIR for use_it changes because the literal value changes. HIR for choose and its exported signature can remain stable. Owner-granular hashing prevents the entire crate from becoming one red blob.

Name resolution for choose at the call may execute or validate. Its output still resolves to the same DefId, so its fingerprint can be green after early cutoff. Type checking use_it may be forced by changed HIR. The inferred type remains u32, the selected Clone obligations remain equivalent, and projections exposing only those facts can become green. Type checking choose itself should remain reusable if no dependency changed.

Built MIR for use_it changes because the constant operand is now 22. Borrow-check results may execute but remain semantically unchanged: ownership behavior did not change. Optimized MIR for the instantiated path changes where it retains the constant. Monomorphization collection can produce the same set of instances because the type argument is still u32.

The CGU containing use_it becomes dirty and is regenerated. A CGU containing unrelated functions may reuse its work product if partitioning and dependencies remain stable. Linking runs when a final linked artifact was requested because an object changed. Metadata for choose remains identical. Whether the private use_it body contributes encoded MIR depends on metadata needs and attributes; do not assume all MIR is exported.

No diagnostic should be emitted for either literal. Cached diagnostic side effects for unchanged queries remain empty. Now instead edit the call to choose("x") while declaring let n: u32. HIR changes, type checking produces a mismatch diagnostic, and an error guarantee propagates. MIR/codegen roots are withheld after analysis errors. Unrelated body diagnostics may still be reused or discovered.

Finally edit only whitespace before choose. Absolute byte positions after the edit can shift. Semantic HIR may hash green while span-sensitive diagnostic products or debuginfo locations change according to their policies. This distinction explains why spans in broad query results can reduce reuse and why source provenance cannot simply be omitted.

32. A bug-location matrix#

Use symptoms to choose the first subsystem to inspect, then verify rather than assuming.

One symptom can have several causes. A missed error may be stale semantic data, but it may also be a correct cached error value whose diagnostic was not replayed. A metadata ICE may originate from a bad stable identity emitted upstream. A cycle seen during incremental validation may come from forcing an old edge in the wrong order.

Collect a timeline:

requested root
-> query/key requested
-> old node found or absent
-> dependencies validated
-> provider executed or cache loaded
-> result fingerprint and color
-> diagnostics replayed/emitted
-> artifact produced/reused

Stop at the first divergence between clean and incremental runs. Later differences are consequences. Prefer stable key descriptions over pointer addresses in reports.

33. Observability and profiling#

Rustc has tracing, self-profiling, incremental diagnostics, and unstable debugging options, but exact flags change. For 1.97.1, begin with rustc +nightly -Z help from the pinned toolchain and current rustc-dev-guide profiling pages. Do not copy a -Z command from an old blog post and treat absent output as a compiler fact.

The self-profiler can record query activities, generic activities, and timing data for tools such as the current profiling viewer workflow. Useful questions include which query kinds dominate time, how many invocations hit memory or disk caches, where blocked time occurs, and which CGUs were reused. Tracing can expose specific query keys but may generate enormous logs.

Incremental information options can report reused versus rebuilt work products or reasons for cache rejection, depending on current nightly. Dep-graph debugging can dump graphs or enable test assertions in compiler test builds. These facilities can materially slow compilation and perturb parallel scheduling.

Use controlled comparisons:

1. Pin rustc commit or exact version.
2. Disable Cargo-level ambiguity by noting whether rustc actually ran.
3. Warm once, then make one recorded edit.
4. Keep target, options, environment, and incremental directory fixed.
5. Capture both clean and incremental profiles.
6. Compare counts and dependency behavior before interpreting wall time.

Query logs can contain source paths, item names, and enormous generated keys. Redact responsibly in bug reports without removing stable identity needed to diagnose. Attach the exact -Z help output or command line because unstable flag semantics are version-sensitive.

Profiling itself does not prove correctness. A suspiciously perfect cache hit rate may indicate missing dependencies. A slower but correct clean build is the reference, not a performance failure to “fix” by trusting more old nodes.

34. Source map laboratory#

Build a small source-map model before editing rustc. Take three virtual files, assign each a non-overlapping global byte interval, and implement lookup from global position to (file, relative offset). Use binary search over file starts. Reject positions in padding or beyond a file's end.

Then add line starts. For source a\nβ\n, record byte offsets, not character counts. Observe that β occupies two UTF-8 bytes. Implement byte-to-line lookup, but delegate display-column policy to a separate function. Add tabs and combining characters to prove that bytes are not terminal columns.

Next model a span as (lo, hi, context). Enforce lo <= hi and same-file containment for ordinary labels. Represent a dummy span explicitly rather than with a plausible numeric zero. Add a provenance chain:

Context 0: root source
Context 1: generated by call span 12..20 in Context 0
Context 2: generated by call span 40..48 in Context 1

Render a short macro backtrace by walking parent contexts. Test a token copied from a macro argument separately from a token authored in the macro definition. The expected “best primary location” can differ.

Finally simulate source remapping. Store a physical filename and a displayed filename. Stable hashing should use the intended remapped identity policy, while file loading still uses the physical path. Verify that diagnostics do not leak the physical build directory when remapping is enabled.

Contribution exercise: locate SourceMap, SourceFile, Span, and SyntaxContext in the 1.97.1 nightly docs. Write down which crate owns each type, then find one call site that converts a span to a source snippet and one that walks expansion provenance. Do not submit a patch yet. First explain unavailable-source behavior and Unicode columns in a short design note.

35. Query and metadata contribution exercises#

Exercise one is a read-only query census. Choose a small TyCtxt method used by type checking. Find its declaration in current query plumbing, its key and result types, local provider registration, and any external provider. Record modifiers without assuming their names mean what an old guide says; read current modifier documentation and implementation.

Exercise two is dependency tracing. Starting from that provider, list every tcx query it invokes on one simple program. Separate dynamic calls from helper calls and untracked reads. Predict the dep graph for two definitions, then compare with available compiler tracing. Explain any extra edges before proposing removal.

Exercise three is stable identity. Create a compiler test with two revisions that inserts an unrelated item before a target item. Determine which target dep nodes should remain clean. Inspect current incremental test annotations and compiletest conventions. Add a clean-build final revision to guard semantics.

Exercise four crosses crates. Make upstream crate api export a generic function and downstream crate use_api instantiate it. Change only the generic body, then only its bound, then only a private non-generic body. Observe .rmeta, downstream checking, and downstream codegen behavior. Explain when MIR must cross metadata and when machine code can be reused.

Exercise five examines corruption handling. Build an rmeta-producing crate, copy the artifact, and inspect strings or container structure without treating it as a stable format. Never feed deliberately corrupted metadata into a production build directory. In a temporary directory, truncate the copy and record whether rustc rejects it gracefully or ICEs; report an ICE with exact versions and artifact reproduction.

Exercise six designs a query. Propose key, result, provider, dep kind, cycle policy, hashing, disk-cache policy, external behavior, and diagnostics for a hypothetical “public API summary per module” query. Compare one module-wide result with per-item projection queries. Estimate invalidation and storage costs.

36. Diagnostic contribution exercises#

Begin with a real, small diagnostic in current rustc. Trace from the semantic condition to builder creation, message selection, labels, suggestions, emission, and resulting error guarantee. Find its terminal UI test and any JSON or suggestion test. Identify whether the diagnostic is produced inside a query and whether side effects can be replayed.

Improve a diagnostic only after answering:

• What is the root mistake?
• Which span is primary and editable?
• Is the proposed replacement always syntactically valid?
• Does macro expansion make the span generated?
• What applicability is honest?
• How does recovery avoid a second diagnostic?
• Does localization need a new Fluent message or argument?

Create cases for ASCII, Unicode before the span, multiline expressions, nested macros, external macro definitions, remapped paths, and JSON output. Do not mark a suggestion machine-applicable if it contains prose placeholders or requires imports not included in multipart edits.

Exercise two targets delayed bugs. Find a delayed-bug call and trace which earlier user error is expected to justify it. Imagine the earlier diagnostic becomes allowed by a lint level or is skipped by a cache bug. Would the delayed bug correctly become an ICE? Document the invariant rather than weakening the delayed bug.

Exercise three targets lints. Write a two-revision incremental test with #[expect(lint_name)]. Revision one fulfills it; revision two removes the lint trigger but leaves the expectation. The second run must report the unfulfilled expectation exactly as a clean build does. Then reverse the revisions and check diagnostic replay.

Exercise four targets macro provenance. Make a declarative macro generate an invalid expression from both a definition token and an argument token. Compare primary spans and macro backtraces. Propose a rendering improvement only if it helps both terminal and JSON consumers without lying about editable source.

37. A disciplined debugging playbook#

First classify the boundary: driver configuration, root orchestration, local query, incremental validation, metadata, source mapping, diagnostic emission, or backend work product. Then minimize while preserving that boundary. A one-crate reproducer cannot preserve an external metadata-provider bug. A clean-only reproducer cannot preserve incremental state.

For stale behavior, save three things: old source, edited source, and old incremental directory. Record exact rustc version verbose output, command line, target, environment inputs, and Cargo invocation. Confirm clean-build equivalence fails. Binary-search the edit and the first incorrectly green query.

For an ICE, keep the full query stack and macro backtrace. The top panic site states where an invariant was noticed, not necessarily where it was broken. Look earlier for an error guarantee discarded, a metadata identity mistranslated, or an untracked input.

For duplicated diagnostics, count provider executions and emission events separately. One execution can emit twice; two executions can each emit once; one cached replay plus one eager emission can look identical at the terminal. Inspect diagnostic IDs, primary spans, and query keys.

For nondeterminism, run identical clean compilations in separate directories and compare stable fingerprints or normalized output. Vary thread scheduling only after establishing a baseline. Audit unordered maps, allocation-derived ordering, race-dependent diagnostic accumulation, and source path remapping.

For performance, prove the dependency is unnecessary before removing it. Use a projection query when many consumers need a small stable fact from a volatile large result. Measure fingerprinting and serialization as well as provider execution. Disk caching a query can make a benchmark slower.

Every fix needs a regression at the narrowest faithful layer, plus a clean-build comparison for incremental bugs. Comments should state the invariant and failure mode, not narrate syntax.

38. Further reading and version audit#

The authoritative starting point for query concepts is the current rustc-dev-guide query chapter: https://rustc-dev-guide.rust-lang.org/query.html. Its incremental overview explains red-green and try-mark-green: https://rustc-dev-guide.rust-lang.org/queries/incremental-compilation.html. The detailed chapter covers stable hashing, two dep graphs, persistence, work products, and query modifiers: https://rustc-dev-guide.rust-lang.org/queries/incremental-compilation-in-detail.html.

For driver and interface architecture, consult the current compiler guide overview and nightly APIs: https://rustc-dev-guide.rust-lang.org/rustc-driver/intro.html, https://doc.rust-lang.org/nightly/nightly-rustc/rustc_driver/, and https://doc.rust-lang.org/nightly/nightly-rustc/rustc_interface/. For TyCtxt and current query modules, use https://doc.rust-lang.org/nightly/nightly-rustc/rustc_middle/ty/struct.TyCtxt.html and https://doc.rust-lang.org/nightly/nightly-rustc/rustc_middle/query/.

For persistence implementation boundaries, consult https://doc.rust-lang.org/nightly/nightly-rustc/rustc_incremental/ and https://doc.rust-lang.org/nightly/nightly-rustc/rustc_middle/dep_graph/. For crate metadata and backend artifacts, start with https://rustc-dev-guide.rust-lang.org/backend/libs-and-metadata.html and https://doc.rust-lang.org/nightly/nightly-rustc/rustc_metadata/.

For diagnostics and source locations, read https://rustc-dev-guide.rust-lang.org/diagnostics.html, https://rustc-dev-guide.rust-lang.org/diagnostics/diagnostic-structs.html, https://rustc-dev-guide.rust-lang.org/diagnostics/macros.html, and current nightly docs for https://doc.rust-lang.org/nightly/nightly-rustc/rustc_errors/ and https://doc.rust-lang.org/nightly/nightly-rustc/rustc_span/. For supported command-line behavior, prefer the rustc book over internal APIs: https://doc.rust-lang.org/rustc/.

Audit any statement against the pinned 1.97.1 source revision before contributing. Search current declarations rather than relying on brittle line numbers. Verify that a named modifier still exists and read its implementation. Verify that a -Z flag appears in that toolchain's help. Verify metadata claims with a two-crate test. Verify incremental claims against a clean build. Verify diagnostics in terminal and structured output where relevant.

The durable philosophy is simple even though the machinery is not. Make semantic dependencies explicit. Use stable identity only where crossing sessions demands it. Cache selectively. Treat diagnostics as structured, replayable products rather than print statements. Preserve source provenance. Let broad orchestration and demand-driven computation coexist. Above all, require every optimization to remain observationally equivalent to compiling cleanly.

Part VIII-B: Rustc as a Production System#

This continuation targets the rustc 1.97.1 source tree and nightly documentation. Internal crate boundaries, query modifiers, diagnostics APIs, and -Z flags are not stable interfaces. Normative language behavior comes from the Reference; implementation statements below are revision-qualified. The aim is to turn the concepts in Part VIII into operational judgment: follow one edit, locate the earliest broken invariant, build a useful model, and prepare a narrowly justified contribution.

39. One invocation, from process policy to target policy#

The compiler is not merely a function from text to bytes. It is a process which must interpret command-line policy, acquire inputs, select a target, expose semantic services, report failures, and commit artifacts without confusing one invocation with another.

For 1.97.1, use this ownership model:

rustc main / embedding tool
  | argv, environment, callbacks
  v
rustc_driver                 process orchestration and exit policy
  | rustc_interface::Config
  v
rustc_interface              create and enter compiler context
  | Session + arena-owned global context
  v
TyCtxt and query system      demand semantic facts
  | roots selected by requested outputs
  v
metadata / codegen / linker  commit externally visible artifacts

rustc_driver owns top-level sequencing, callback checkpoints, and conversion of compiler outcomes into process outcomes. rustc_interface turns a configuration into a compiler context with tightly bounded lifetimes. Session contains invocation-wide facilities and policy: parsed options, target information, source mapping, diagnostics, crate configuration, and output choices. TyCtxt is the handle through which compiler code reaches interned types, definitions, and query results. These descriptions are roles, not promises about public fields.

Config is an assembly boundary. An embedding tool can choose input, file loading, diagnostic output, lint registration, and callbacks, but it must finish configuration before dependent services are built. Changing a target after layouts have been computed would invalidate meanings already cached. Keeping TyCtxt<'tcx> after the context exits would retain references into arenas whose lifetime ended.

The target specification is semantic input, not backend decoration. It determines pointer width, integer layouts, ABI, data layout, target features, atomics, panic strategy constraints, linker flavor, and object conventions. A target-sensitive query must either read tracked target input or live under a cache compatibility boundary that changes with it.

Configuration has four useful classes:

The categories overlap. -D warnings is diagnostic policy but changes success status. Path remapping affects diagnostics, metadata, and reproducibility. Do not classify an option by where it is parsed; classify every observation it can change.

Prediction. Two runs have identical source but different --target values. May a cached usize layout be reused because its query key is the same DefId?

No. The key identifies the requested item, not all inputs to its answer. Either target facts appear in its dynamic dependency closure or the old cache must be rejected more broadly.

40. Query declarations become a protocol#

A query declaration establishes more than a function signature. At the 1.97.1 boundary, generated plumbing associates a query name with a key type, value type, cache behavior, dependency-node policy, provider dispatch, cycle handling, description, and sometimes persistence or evaluation modifiers. Read current declarations in compiler/rustc_middle/src/query and macro generation in compiler/rustc_macros; never infer current modifiers from an old article.

Conceptually:

query K -> V
  key identity and stable identity
  provider: local crate or external crate
  in-memory memo slot
  active job state and waiters
  dependency-node identity
  value fingerprint / early-cutoff policy
  disk-load and force policy, where supported
  cycle reporting or recovery policy

The provider computes a miss. Provider tables are populated by compiler crates so the engine need not hard-code every semantic subsystem. The key selects both the fact and often the provider domain. A local DefId can route to local analysis; an external DefId can route to decoded crate metadata. The values may look alike while their acquisition is radically different.

A query job is one active evaluation of (query kind, key). It records its parent job, dependency reads, waiters, and completion state. The engine pushes the job before calling the provider and pops it on every completion path. That stack makes dynamic dependency recording possible and makes a same-thread recursive demand visible as a cycle.

The core invariants are:

1. A completed memo entry corresponds to exactly one successful semantic result.
2. Every tracked query read while a provider runs becomes an edge from the active job.
3. A waiter observes completion, failure, or cancellation; it cannot wait forever on an abandoned owner.
4. Values crossing sessions are identified and fingerprinted without session-local addresses.
5. Side effects are absent, explicitly replayed, or represented in the result.

Dynamic recording is precise with respect to execution, not omniscient. If provider A takes a hidden global shortcut instead of calling tracked input B, no edge appears. The graph faithfully records the wrong program.

push A(k)
  provider A
    read B(x) ---- record A(k) -> B(x)
      push B(x)
      provider B
      fingerprint and memoize B(x)
      pop B(x)
    read C(y) ---- record A(k) -> C(y)
  fingerprint and memoize A(k)
pop A(k)

Counterexample. A provider reads std::env::var("SDK") directly. Nothing in (kind, key) or its recorded edges changes when SDK changes. The provider is deterministic only by accident within one process. Fix this by turning the environment fact into declared invocation input, disabling reuse, or making the compatibility boundary include it.

41. Red-green validation and the two meanings of unchanged#

Incremental compilation asks whether an old result remains semantically usable after an edit. The previous dependency graph and fingerprints are loaded read-only. The current graph is built by this invocation. An old node starts unknown, not green.

To try to mark node A green:

1. Find its old dependency node by stable identity.
2. Recursively validate every old dependency.
3. If each dependency is green, A can often be marked green without running its provider.
4. If a dependency is red, force A when possible.
5. Fingerprint the new value and compare it with the old fingerprint.
6. Equal fingerprints make A green by early cutoff; unequal fingerprints make it red.

There are therefore two useful meanings of unchanged:

• not re-executed: old dependencies proved unchanged;
• re-executed but equal: an input changed, yet the provider's observable result fingerprint did not.

The second is early cutoff. It prevents harmless upstream edits from propagating indefinitely.

old: parse(F) -> names(M) -> type(Fn) -> codegen(CGU)
       red          green       ?             ?

new parse differs
  names reruns, same exported-name fingerprint => green
  type dependencies all green                  => green without rerun
  codegen remains green                         => reuse work product

Fingerprint equality is not value equality unless the hash and serialization contracts correctly represent all observable meaning. A collision is theoretically possible, so production hashing chooses a sufficiently strong stable fingerprint and treats stable-hash implementation as correctness-sensitive. Pointers, hash-table iteration order, transient indices, and absolute paths must not leak into stable hashes unless deliberately normalized.

Stable hashing must answer: “equal under which observation?” If diagnostics include a span, a fingerprint that omits source provenance may incorrectly suppress changed output. If downstream consumers only observe a type's normalized semantic shape, including an allocation address creates false invalidation.

Force is the operation that recomputes a dependency node during validation. It needs to reconstruct the query key from a stable dependency-node identity. Not every key is reconstructible or every node forceable. Those restrictions shape which queries can be loaded from disk and how validation falls back. Treat force_from_dep_node and try_load_from_disk behavior as query-specific current implementation, not universal language machinery.

42. Persistence is a compatibility protocol#

An incremental directory is untrusted optimization state, not semantic authority. The compiler persists enough previous graph, fingerprints, serialized query results, and backend work-product records to attempt reuse. It must reject incompatible data and recover safely from missing or corrupt cache state.

A disk record needs at least this conceptual information:

format/compiler compatibility identity
crate and invocation identity
stable dep-node identity
result fingerprint
old dependency identities
optional encoded value or work-product location
integrity framing

The compiler must not serialize a raw DefId, arena pointer, Span byte position, or interned index and assume it means the same thing next run. It uses stable crate and definition identities, stable hashing contexts, and decoding tables to reconnect persisted descriptions to current data. What is stable is purpose-specific; it is not a general permanent rustc object ID.

Disk reuse has a cost model:

T_incremental = T_load + T_validate + T_forced + T_hash + T_serialize + T_commit
T_clean       = T_compute_all + T_hash_needed_for_outputs
benefit       = T_clean - T_incremental

Fine granularity reduces forced computation but increases graph nodes, edges, locks, hashes, and serialization. Persisting a cheap value can cost more than recomputing it. A useful optimization report measures all terms rather than reporting only provider time.

Security boundaries matter. Treat cache bytes, dependency metadata, object inputs, source files, proc-macro output, and linker messages as potentially malformed. Bounds-check lengths and indices, avoid unchecked allocation from encoded sizes, reject incompatible formats, and make corruption trigger clean recomputation rather than unsound acceptance. Proc macros and build scripts execute code with the user's privileges; incremental validation does not sandbox them.

Reproducibility requires deterministic semantic and artifact observations despite different scheduling or directories. Normalize configured paths, sort unordered output, make symbol and metadata identity stable, avoid timestamps unless specified, and distinguish deterministic bytes from merely equivalent behavior. Compare clean builds in fresh remapped directories when auditing.

43. Cycles, parallelism, cancellation, and side effects#

A recursive query demand can reveal either a user-level semantic cycle or a compiler architecture bug. The active job stack gives a chain such as A(k) -> B(x) -> A(k). Cycle recovery is query-specific: some cycles produce a diagnostic and a designated recovery value; some are fatal because no honest value exists. Never invent a plausible value merely to continue.

Parallel execution adds a wait-for graph. If worker 1 owns A and waits for B, while worker 2 owns B and waits for A, stack-local cycle detection is insufficient. The runtime must inspect cross-thread waits, report a deterministic useful cycle, release waiters, and avoid publishing partial values.

worker 1: owns A ----waits-for----> B :owned by worker 2
    ^                                      |
    +---------------waits-for--------------+

Parallelism is safe only when providers obey a stronger contract than “Rust prevents data races.” They must avoid schedule-dependent semantic output, globally ordered mutation, duplicate irreversible effects, and locks held across unknown query calls. Concurrent maps need deterministic meaning even if insertion order varies. Diagnostics should be buffered, sorted by stable source/provenance criteria where required, then emitted under explicit policy.

Cancellation must be contagious but controlled. When fatal failure, user interruption, or worker failure cancels an invocation, active jobs must stop at safe points, wake waiters, avoid committing incomplete artifacts, and leave no memo entry that looks successful. RAII guards should pop active jobs and release locks during unwinding. Temporary files should be written and atomically renamed only after successful completion.

Query side effects create three failures:

• a green query skips an effect that the current run requires;
• a forced query repeats an effect;
• parallel scheduling changes effect order.

Prefer returning structured data. If a query discovers diagnostics, model them as replayable products associated with the cached result or ensure a deliberate eager consumer emits them once. File writes, metrics, interning, and diagnostic counts each need an explicit ownership rule.

Prediction. A query logs “entered provider” and the log disappears on an incremental run. Is the query engine wrong?

No. Provider execution is not observable language behavior and green reuse intentionally skips it. If the log is required observability, instrument cache hits and validation separately rather than hiding telemetry in providers.

44. Crate metadata is a semantic API between compiler processes#

An upstream crate is usually not re-analyzed from source while compiling a downstream crate. Rustc encodes selected semantic facts into crate metadata and decodes them lazily or on demand in later compilations. The exact 1.97.1 format is compiler-private and tied to compiler compatibility; it is not a stable interchange format.

Metadata can describe exported definitions, paths, visibility, generics, predicates, types, trait information, MIR needed for cross-crate activities, attributes, spans/provenance, lang items, and other downstream-required facts. Not every internal fact is encoded. Encoding policy is an API design decision: omit too much and downstream compilation cannot answer queries; include too much and metadata becomes expensive and tightly coupled.

crate A local provider
  semantic result keyed by local DefId
       |
       | encode stable crate/definition references
       v
     A.rmeta inside artifact boundary
       |
       | validate + decode + intern into crate B session
       v
crate B external provider keyed by external DefId

The crate store maps external crate identities to loaded metadata. Cross-crate query providers decode facts rather than running A's local HIR analysis in B. Decoded types and predicates are interned into B's current context, so raw encoded indices must be interpreted through decoder tables. Span decoding must preserve enough source and expansion provenance for useful diagnostics while respecting path remapping and source availability.

A metadata bug often presents far away:

Construct a two-crate reproducer before editing metadata code. Change one public fact in crate A, rebuild A and B, inspect whether B's external provider was invalidated, and compare with a clean B build. An in-crate unit test cannot exercise this boundary.

45. Diagnostics preserve provenance, not just coordinates#

A SourceMap owns source files and translates byte positions into file, line, and column displays. A Span is more than (line, column): it identifies a byte range and syntax context, and can carry macro-expansion provenance. Hygiene distinguishes identifiers that spell the same text but originate in different expansion contexts.

token bytes in SourceFile
  -> BytePos range
  -> Span + SyntaxContext
  -> expansion chain / call site / definition site
  -> rendered terminal location or structured JSON span

Unicode makes byte offsets and displayed columns differ. Tabs, wide characters, combining marks, remapped paths, virtual files, and macro-generated text defeat hand-written line arithmetic. Use source-map APIs and test the actual emitter.

A structured diagnostic has a primary message, code where applicable, primary span, labels, notes, helps, and suggestions with applicability. Fluent resources separate localizable text from semantic construction. Arguments passed to messages must remain structured; assembling an English sentence in the provider defeats localization. JSON emission is an interface consumed by tools, but not every internal rendering detail is a forever-stable schema promise; use the documented rustc JSON output contract for the target toolchain.

An error guarantee is evidence that an error was emitted. Passing it through a recovery value prevents code from pretending that an invalid semantic object is ordinary success. Delayed bugs encode the invariant “an earlier user error must exist; otherwise this is a compiler bug.” Weakening a delayed bug to silence an ICE can hide the true lost guarantee.

Recovery must maximize independent useful errors while preserving invariants. Use explicit error-tainted values, stop before operations requiring valid layout or codegen, suppress direct cascades, and never let recovery output enter metadata as if valid.

Suggestions are edits, not decoration. Machine-applicable means tooling can apply the complete replacement without human judgment. Macro provenance, missing imports, placeholders, and multipart dependencies can require a weaker applicability. Test terminal output, JSON spans, Unicode prefixes, and macro call/definition sites.

46. Lints, stability, and feature gates are policy systems#

A lint separates detection from level policy. The producer identifies a condition; lint levels derive from defaults, attributes, command-line settings, groups, caps, and expectations. The final level determines allow, warning, denial, or expectation bookkeeping. Do not emit a normal error directly when the condition is intended to remain configurable.

Lint timing matters. Early lints can inspect syntax before lowering destroys details. Late lints can use HIR and type information. A lint query must track every semantic input and reproduce diagnostics or expectations under incremental reuse. #[expect(...)] adds an observation: whether the expected lint was fulfilled. Removing the triggering code must invalidate that bookkeeping.

Stability checking protects use of unstable standard-library/compiler features according to staged API metadata and channel policy. Language feature gates protect syntax or semantics not stabilized for the active channel/edition. These are related but not interchangeable. An internal compiler feature, a library #[unstable] item, and a #![feature(...)] language gate can have distinct tracking and diagnostics.

Gate checking must occur where enough information exists to distinguish the feature, but early enough to prevent unsupported constructs from flowing as accepted input. Recovery may create placeholder nodes after reporting a gate error; later phases must retain the error guarantee. Bootstrap and compiler-internal allowances are implementation policy, not user-visible stabilization.

47. A materially capable typed educational query engine#

The following complete stable Rust program implements two typed query families:

• Parse(File) returns normalized words;
• Exports(Module) follows use NAME references and returns sorted public names.

It records dynamic dependencies, memoizes values, detects cycles, invalidates inputs, and performs early cutoff. It models disk persistence by exporting stable fingerprints and dependency identities, then seeding a fresh process with those records. It intentionally uses deterministic 64-bit FNV-1a for readable fingerprints; production rustc uses stronger, implementation-specific stable hashing.

Save as query_lab.rs; compile with rustc --edition=2021 query_lab.rs.

use std::cell::RefCell;
use std::collections::{BTreeMap, BTreeSet};
use std::fmt;
use std::sync::{Arc, RwLock};

#[derive(Clone, Debug, Eq, PartialEq, Ord, PartialOrd)]
struct File(String);

#[derive(Clone, Debug, Eq, PartialEq, Ord, PartialOrd)]
struct Module(String);

#[derive(Clone, Debug, Eq, PartialEq)]
struct Words(Vec<String>);

#[derive(Clone, Debug, Eq, PartialEq)]
struct Names(Vec<String>);

#[derive(Clone, Debug, Eq, PartialEq, Ord, PartialOrd)]
enum Key {
	Input(File),
	Parse(File),
	Exports(Module),
}

#[derive(Clone, Debug, Eq, PartialEq)]
enum Value {
	Text(String),
	Words(Words),
	Names(Names),
}

impl Value {
	fn fingerprint(&self) -> u64 {
		let mut bytes = Vec::new();
		match self {
			Value::Text(s) => {
				bytes.push(0);
				put_str(&mut bytes, s);
			}
			Value::Words(Words(xs)) => {
				bytes.push(1);
				for x in xs { put_str(&mut bytes, x); }
			}
			Value::Names(Names(xs)) => {
				bytes.push(2);
				for x in xs { put_str(&mut bytes, x); }
			}
		}
		fnv1a(&bytes)
	}
}

fn put_str(out: &mut Vec<u8>, value: &str) {
	out.extend_from_slice(&(value.len() as u64).to_le_bytes());
	out.extend_from_slice(value.as_bytes());
}

fn fnv1a(bytes: &[u8]) -> u64 {
	let mut h = 0xcbf29ce484222325_u64;
	for b in bytes {
		h ^= u64::from(*b);
		h = h.wrapping_mul(0x100000001b3);
	}
	h
}

#[derive(Clone, Debug)]
struct Memo {
	value: Value,
	fingerprint: u64,
	deps: BTreeMap<Key, u64>,
	dirty: bool,
}

#[derive(Clone, Debug, Eq, PartialEq)]
struct DiskRecord {
	fingerprint: u64,
	deps: BTreeMap<Key, u64>,
}

#[derive(Default, Clone, Debug, Eq, PartialEq)]
struct Stats {
	providers: BTreeMap<Key, usize>,
	hits: usize,
	early_cutoffs: usize,
}

#[derive(Clone, Debug, Eq, PartialEq)]
enum Error {
	MissingFile(File),
	Cycle(Vec<Key>),
	WrongType { key: Key, expected: &'static str },
}

impl fmt::Display for Error {
	fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
		write!(f, "{self:?}")
	}
}

thread_local! {
	static STACK: RefCell<Vec<Frame>> = const { RefCell::new(Vec::new()) };
}

#[derive(Debug)]
struct Frame {
	key: Key,
	deps: BTreeMap<Key, u64>,
}

struct Engine {
	inputs: RwLock<BTreeMap<File, String>>,
	memo: RwLock<BTreeMap<Key, Memo>>,
	old: RwLock<BTreeMap<Key, DiskRecord>>,
	stats: RwLock<Stats>,
}

impl Engine {
	fn new(files: impl IntoIterator<Item = (File, String)>) -> Self {
		Self {
			inputs: RwLock::new(files.into_iter().collect()),
			memo: RwLock::new(BTreeMap::new()),
			old: RwLock::new(BTreeMap::new()),
			stats: RwLock::new(Stats::default()),
		}
	}

	fn with_disk(
		files: impl IntoIterator<Item = (File, String)>,
		old: BTreeMap<Key, DiskRecord>,
	) -> Self {
		let engine = Self::new(files);
		*engine.old.write().unwrap() = old;
		engine
	}

	fn set_file(&self, file: File, text: String) {
		self.inputs.write().unwrap().insert(file.clone(), text);
		let changed = Key::Input(file);
		let mut memo = self.memo.write().unwrap();
		for entry in memo.values_mut() {
			if entry.deps.contains_key(&changed) { entry.dirty = true; }
		}
		// Transitive users are validated lazily through stored fingerprints.
	}

	fn parse(&self, file: File) -> Result<Words, Error> {
		let key = Key::Parse(file);
		match self.get(key.clone())? {
			Value::Words(v) => Ok(v),
			_ => Err(Error::WrongType { key, expected: "Words" }),
		}
	}

	fn exports(&self, module: Module) -> Result<Names, Error> {
		let key = Key::Exports(module);
		match self.get(key.clone())? {
			Value::Names(v) => Ok(v),
			_ => Err(Error::WrongType { key, expected: "Names" }),
		}
	}

	fn get(&self, key: Key) -> Result<Value, Error> {
		if let Some(value) = self.try_valid_memo(&key)? {
			self.stats.write().unwrap().hits += 1;
			self.record_read(key, value.fingerprint());
			return Ok(value);
		}

		let cycle = STACK.with(|stack| {
			let stack = stack.borrow();
			stack.iter().position(|f| f.key == key).map(|at| {
				let mut path: Vec<_> = stack[at..].iter().map(|f| f.key.clone()).collect();
				path.push(key.clone());
				path
			})
		});
		if let Some(path) = cycle { return Err(Error::Cycle(path)); }

		STACK.with(|s| s.borrow_mut().push(Frame { key: key.clone(), deps: BTreeMap::new() }));
		let computed = self.compute(&key);
		let frame = STACK.with(|s| s.borrow_mut().pop().unwrap());
		let value = computed?;
		let fingerprint = value.fingerprint();
		let old_fp = self.memo.read().unwrap().get(&key).map(|m| m.fingerprint)
			.or_else(|| self.old.read().unwrap().get(&key).map(|m| m.fingerprint));
		if old_fp == Some(fingerprint) {
			self.stats.write().unwrap().early_cutoffs += 1;
		}
		self.memo.write().unwrap().insert(key.clone(), Memo {
			value: value.clone(), fingerprint, deps: frame.deps, dirty: false,
		});
		self.record_read(key, fingerprint);
		Ok(value)
	}

	fn try_valid_memo(&self, key: &Key) -> Result<Option<Value>, Error> {
		let snapshot = self.memo.read().unwrap().get(key).cloned();
		let Some(entry) = snapshot else { return Ok(None) };
		if !entry.dirty {
			for (dep, old_fp) in &entry.deps {
				let now = self.read_dependency(dep.clone())?.fingerprint();
				if now != *old_fp { return Ok(None); }
			}
			return Ok(Some(entry.value));
		}
		Ok(None)
	}

	fn read_dependency(&self, key: Key) -> Result<Value, Error> {
		match key {
			Key::Input(file) => self.input(file),
			other => self.get(other),
		}
	}

	fn compute(&self, key: &Key) -> Result<Value, Error> {
		*self.stats.write().unwrap().providers.entry(key.clone()).or_default() += 1;
		match key {
			Key::Input(file) => self.input(file.clone()),
			Key::Parse(file) => {
				let Value::Text(text) = self.input(file.clone())? else { unreachable!() };
				let words = text.split_whitespace().map(str::to_owned).collect();
				Ok(Value::Words(Words(words)))
			}
			Key::Exports(module) => {
				let words = self.parse(File(format!("{}.toy", module.0)))?;
				let mut names = BTreeSet::new();
				let mut it = words.0.iter();
				while let Some(word) = it.next() {
					match word.as_str() {
						"pub" => if let Some(name) = it.next() { names.insert(name.clone()); },
						"use" => if let Some(name) = it.next() {
							let child = self.exports(Module(name.clone()))?;
							names.extend(child.0);
						},
						_ => {}
					}
				}
				Ok(Value::Names(Names(names.into_iter().collect())))
			}
		}
	}

	fn input(&self, file: File) -> Result<Value, Error> {
		let text = self.inputs.read().unwrap().get(&file).cloned()
			.ok_or_else(|| Error::MissingFile(file.clone()))?;
		let value = Value::Text(text);
		self.record_read(Key::Input(file), value.fingerprint());
		Ok(value)
	}

	fn record_read(&self, key: Key, fingerprint: u64) {
		STACK.with(|s| {
			if let Some(parent) = s.borrow_mut().last_mut() {
				if parent.key != key { parent.deps.insert(key, fingerprint); }
			}
		});
	}

	fn disk_model(&self) -> BTreeMap<Key, DiskRecord> {
		self.memo.read().unwrap().iter().map(|(key, memo)| (key.clone(), DiskRecord {
			fingerprint: memo.fingerprint,
			deps: memo.deps.clone(),
		})).collect()
	}

	fn stats(&self) -> Stats { self.stats.read().unwrap().clone() }
}

fn files(a: &str, b: &str) -> Vec<(File, String)> {
	vec![
		(File("a.toy".into()), a.into()),
		(File("b.toy".into()), b.into()),
	]
}

fn main() {
	let db = Arc::new(Engine::new(files("pub apple use b", "pub berry")));
	assert_eq!(db.exports(Module("a".into())).unwrap(), Names(vec!["apple".into(), "berry".into()]));
	let disk = db.disk_model();

	// A whitespace edit changes Parse(b), but not Exports(b): early cutoff.
	db.set_file(File("b.toy".into()), "pub   berry".into());
	assert_eq!(db.exports(Module("a".into())).unwrap().0, ["apple", "berry"]);
	assert!(db.stats().early_cutoffs >= 1);

	// The persisted model contains fingerprints and edges, never session pointers.
	let fresh = Engine::with_disk(files("pub apple use b", "pub berry pub blue"), disk);
	assert_eq!(fresh.exports(Module("a".into())).unwrap().0, ["apple", "berry", "blue"]);

	// Cross-query cycle detection reports the whole repeated path.
	let cyclic = Engine::new(files("use b", "use a"));
	assert!(matches!(cyclic.exports(Module("a".into())), Err(Error::Cycle(_))));

	// Providers are thread-safe; independent roots may be demanded concurrently.
	let parallel = Arc::new(Engine::new(files("pub apple", "pub berry")));
	let p1 = Arc::clone(&parallel);
	let p2 = Arc::clone(&parallel);
	let t1 = std::thread::spawn(move || p1.exports(Module("a".into())).unwrap());
	let t2 = std::thread::spawn(move || p2.exports(Module("b".into())).unwrap());
	assert_eq!(t1.join().unwrap().0, ["apple"]);
	assert_eq!(t2.join().unwrap().0, ["berry"]);
}

The typed public methods prevent a caller from interpreting an Exports value as Words. Internally, the enum permits one cache while retaining checked variants. BTreeMap and sorted exports make fingerprints independent of randomized hash-map order. Each frame collects the exact reads made by its provider.

The engine's persisted records are deliberately only a disk-fingerprint model. They demonstrate stable identities, old fingerprints, and dependencies, but do not deserialize values or mark old nodes green without execution. A real loader would validate old dependencies recursively, reconstruct keys, decode values, and enforce a format/compiler namespace.

The parallel example permits independent work, but duplicate simultaneous demand of the same key can compute twice because this model has no per-key job ownership or waiter condition variable. Production design needs a state machine such as Vacant -> Running(owner, waiters) -> Complete(value) plus cross-thread cycle detection and cancellation wakeups. Never hold the global memo lock while invoking a provider.

Explicit omissions include collision-resistant hashing, stable identity across file renames, serialized values, corruption handling, provider panic poisoning, query-specific cycle recovery, diagnostic replay, work products, memory reclamation, priority scheduling, deadlock detection, cancellation tokens, and bounded resource use. Those omissions are boundaries to investigate, not minor polish.

48. Clean-build equivalence as executable properties#

Incremental correctness is observational:

observe(incremental(old_inputs -> new_inputs))
    == observe(clean(new_inputs))

Choose observe deliberately. For this engine it is Result<Names, Error> for each root. For rustc it can include acceptance, diagnostics after normalization, metadata meaning, executable behavior, and artifact bytes where reproducibility is promised.

Append these tests to the program or place them under #[cfg(test)] in a library version:

#[cfg(test)]
mod tests {
	use super::*;

	fn clean(a: &str, b: &str) -> Result<Names, Error> {
		Engine::new(files(a, b)).exports(Module("a".into()))
	}

	#[test]
	fn every_small_edit_matches_clean() {
		let variants = [
			("pub apple use b", "pub berry"),
			("pub apricot use b", "pub berry"),
			("pub apricot use b", "pub berry pub blue"),
			("use b", "pub blue"),
		];
		let db = Engine::new(files(variants[0].0, variants[0].1));
		let _ = db.exports(Module("a".into())).unwrap();
		for (a, b) in variants.into_iter().skip(1) {
			db.set_file(File("a.toy".into()), a.into());
			db.set_file(File("b.toy".into()), b.into());
			assert_eq!(db.exports(Module("a".into())), clean(a, b));
		}
	}

	#[test]
	fn whitespace_stops_at_semantic_boundary() {
		let db = Engine::new(files("pub apple use b", "pub berry"));
		let before = db.exports(Module("a".into())).unwrap();
		db.set_file(File("b.toy".into()), "\n pub    berry \n".into());
		let after = db.exports(Module("a".into())).unwrap();
		assert_eq!(before, after);
		assert!(db.stats().early_cutoffs > 0);
	}

	#[test]
	fn disk_records_are_deterministic() {
		let left = Engine::new(files("pub apple use b", "pub berry"));
		let right = Engine::new(files("pub apple use b", "pub berry"));
		left.exports(Module("a".into())).unwrap();
		right.exports(Module("a".into())).unwrap();
		assert_eq!(left.disk_model(), right.disk_model());
	}
}

Property testing should generate valid and malformed module texts, edit sequences, query-root orders, and thread schedules. After every edit, compare all roots against a fresh engine. Metamorphic properties include whitespace invariance, declaration-order invariance where the language says order is irrelevant, and idempotence of repeated demand. Inject hash collisions in a test hasher to prove collision assumptions are explicit. Inject provider panic and cancellation to prove no partial memo becomes complete.

49. Concrete edit traces#

Consider two crates:

// api/src/lib.rs
pub fn answer() -> u32 { 42 }

// app/src/main.rs
fn main() { println!("{}", api::answer()); }

Trace A: comment-only edit in the function body#

1. File input fingerprint changes.
2. Parsing/lowering-related nodes affected by source text are reconsidered.
3. A semantic representation may fingerprint equal if comments are discarded.
4. Signature and exported metadata remain green.
5. Depending on body representation and debuginfo observations, body/codegen work may remain green or be recomputed.
6. Downstream app should not semantically re-type-check because the public contract did not change.

Do not assert exact green nodes without running the 1.97.1 instrumentation; granularity changes.

Trace B: u32 becomes u64#

1. The item's signature fingerprint changes.
2. Metadata for api changes.
3. app's external provider decodes a new function type.
4. Calls and formatting obligations are revalidated.
5. MIR/codegen dependencies affected by layout become red.
6. Incremental and clean builds must accept/reject identically.

Trace C: body returns 43, signature unchanged#

The local optimized MIR or codegen work product changes. Ordinary downstream type checking may remain green because the callable contract is unchanged. Cross-crate inlining can expose body changes to downstream optimization, so the encoded/reachable MIR and downstream codegen dependencies must account for that observation.

Trace D: rename private helper only#

Definition identity and spans may change even if public behavior does not. Local name resolution, HIR owners, diagnostics, and codegen symbols can be affected. The ideal cutoff is not “nothing changes”; it is “changes stop at the narrowest honest semantic boundary.”

For each trace, record the exact toolchain, command, target, incremental directory, Cargo freshness, emitted artifact kinds, and thread settings. Keep old and new source plus the cache directory. Then identify the earliest node whose color contradicts the expected observation.

50. Failure maps and distinguishing experiments#

The panic site is the last detected invariant violation, not necessarily the first broken one. A decoder panic can originate in an encoder omission. A codegen ICE can originate in recovery that discarded an error guarantee. A wrong diagnostic can originate in lowering that discarded provenance.

51. Profiling and observability without changing meaning#

Observability must distinguish provider execution, memo hits, disk loads, try-mark-green validation, forced recomputation, fingerprinting, serialization, blocked time, and backend reuse. One aggregate “query time” number cannot tell whether a change reduced work or moved it into hashing.

Useful measurements are:

• invocation wall and CPU time;
• query counts and self/blocked time;
• dep-node and edge counts;
• red/green/unknown outcomes;
• bytes loaded and written;
• metadata encode/decode time;
• peak resident memory;
• codegen-unit reuse;
• diagnostic construction and emission counts.

Use current rustc -Z help to discover available profiling flags in 1.97.1; -Z interfaces change. For compiler performance work, follow rustc-perf's current collector and benchmark procedure rather than presenting one local wall-clock run. Warm filesystem caches, Cargo freshness, background load, and bootstrap stages can dominate small differences.

A representative benchmark matrix includes clean check, clean build, no-op incremental, tiny body edit, public signature edit, large generated crate, two-crate metadata boundary, multiple targets, and low/high thread counts. State the baseline commit, machine, build configuration, repetitions, variance, and expected user workload.

Telemetry itself must not become a hidden dependency. Counters use synchronization or thread-local aggregation, event timestamps should not enter semantic hashes, and trace ordering should not be mistaken for deterministic query ordering.

52. Production hardening checklist#

Correctness

• State each query's key, complete inputs, result observation, and side effects.
• Test incremental versus clean after edit sequences.
• Test local and cross-crate providers.
• Make stable hashing deterministic and context-aware.
• Preserve error guarantees and reject invalid recovery values at hard boundaries.

Concurrency and cancellation

• Give each active key one owner and wake every waiter on all exits.
• Detect cross-thread wait cycles.
• Never hold subsystem locks across arbitrary query calls without a proven lock order.
• Check cancellation in bounded expensive loops.
• Publish memo entries and output files only after complete success.

Persistence and security

• Version and namespace persisted formats.
• Bounds-check indices, lengths, recursion, and allocations.
• Treat malformed cache as a recoverable cache miss where feasible.
• Use atomic commit and clean abandoned temporary files.
• Fuzz metadata and incremental decoders at their actual entry points.
• Do not promise that compilation safely executes hostile proc macros or build scripts.

Diagnostics

• Keep messages structured and localizable.
• Preserve primary and macro provenance spans.
• Test terminal, JSON, suggestions, Unicode, and path remapping.
• Ensure cached and clean diagnostic sets agree after normalization.
• Sort only where policy requires; do not conceal distinct errors as duplicates.

Reproducibility and maintenance

• Eliminate accidental map, address, thread, directory, and timestamp dependence.
• Record relevant target, compiler, options, and environment identity.
• Add tracing that can identify the first changed dependency.
• Document invariants beside unsafe decoding or manual incremental integration.
• Prefer a narrow regression over a giant snapshot that fails for unrelated wording.

53. Source navigation for a 1.97.1 contribution#

Begin from behavior, then pin the exact source revision corresponding to rustc 1.97.1. Directory names below are navigation anchors and can move:

Use symbol search rather than memorized line numbers. For a query, find its declaration, key and value definitions, provider assignment, provider body, call sites, dep-kind/modifiers, encode/decode support, and tests. For a diagnostic, find the semantic condition, structured diagnostic definition, Fluent key, emission point, guarantee flow, UI stderr, JSON or suggestion coverage, and incremental replay behavior.

Read the current rustc-dev-guide first, but verify implementation claims in the pinned tree. Guide prose may lag a refactor; source alone may hide design intent. Use both, and mention uncertainty in review descriptions.

54. Contribution laboratories#

Laboratory A: an over-invalidating query#

Find a query whose consumers need only a small projection of a large volatile result. Measure clean, no-op, and representative edits. Draw current dependency edges. Propose a projection query only if provider, hashing, graph, and memory overhead are lower than saved recomputation. Add clean-equivalence and performance evidence.

Laboratory B: metadata round trip#

Create upstream/downstream crates exercising one generic item, associated type, macro span, and inline candidate. Locate encoder and decoder paths. Change one field, predict affected downstream queries, and verify clean equivalence. Then truncate metadata in a controlled test and verify a bounded diagnostic rather than unchecked memory use or panic where the boundary promises handling.

Laboratory C: query cycle report#

Construct the smallest legal source that reaches a user-reportable cycle. Map semantic cycle to query-job cycle. Check determinism under multiple thread schedules. Improve the report only if the primary cycle and provenance become clearer without relying on incidental job order.

Laboratory D: diagnostic provenance#

Trigger one diagnostic directly, through a declarative macro argument, through a definition token, and through nested expansion. Capture terminal and JSON forms. Classify editable spans and suggestion applicability. Trace every transformation where syntax context could be lost.

Laboratory E: lint expectation across revisions#

Revision one contains #[expect(...)] and fulfills it. Revision two removes the trigger. The incremental second run must match a clean second run, including unfulfilled-expectation behavior. Reverse the transition and vary command-line caps.

Laboratory F: cancellation fault injection#

In an educational engine, cancel after owner registration, after one dependency edge, after provider return, and before output rename. Assert that waiters wake, no incomplete memo is visible, retry works, and temporary output is absent. Translate each assertion into the corresponding rustc subsystem before proposing production code.

Laboratory G: reproducibility#

Build equivalent source trees in different absolute directories with path remapping. Compare normalized diagnostics, metadata, and selected artifacts. Repeat with different thread counts and insertion orders. When bytes differ, locate the first differing structured field before patching output wholesale.

Contributor review questions#

1. What earliest invariant was broken?
2. Which observation changes and which must remain equal?
3. Is identity stable only in-session or across sessions/crates?
4. Does the change add or remove a dynamic dependency?
5. What happens on a cache hit, force, cycle, panic, and cancellation?
6. Are local and external providers symmetric where required?
7. Can malformed inputs cause unbounded allocation or recursion?
8. Does parallel order affect diagnostics or hashes?
9. Which clean-build equivalence test fails before the fix?
10. What measured workload justifies added complexity?

55. Derived philosophy, mastery path, and talk designs#

The mechanisms above support conclusions more precise than “incremental compilation is caching.”

Representations determine cheap questions. A crate-wide result makes “did any item change?” cheap and “which signature changed?” expensive. Per-item dep nodes reverse some of that cost while adding identity, graph, and synchronization overhead.

Abstractions move responsibility. Dynamic tracking removes hand-written dependency lists only when all semantic reads pass through tracked services. It moves responsibility to provider discipline and input modeling.

Optimization preserves meaning under an observation model. Early cutoff is valid only because a fingerprint claims to encode everything downstream may observe. Changing that observation requires changing hashing and invalidation.

Identity is contextual. An arena address is identity for neither another session nor metadata. A stable definition path is useful only under its crate/disambiguation and revision rules. Demand the narrowest identity lifetime required.

Caching creates obligations. Every cache adds compatibility, invalidation, corruption, memory, cancellation, and observability work. The fastest safe cache may be no cache for cheap values.

Diagnostics depend on preserved provenance. Once lowering or metadata erases expansion context or stable source identity, a later emitter cannot reconstruct an honest editable span. Presentation quality begins upstream.

The first visible failure is late. A linker error may begin with target configuration, a decoder panic with encoding, and stale codegen with an absent query edge. Debug backward to the earliest violated invariant.

Mastery sequence#

1. Draw one invocation's driver, interface, session, target, and roots.
2. Trace one local query from declaration through provider and callers.
3. Trace the same semantic fact through an external metadata provider.
4. Run a two-revision edit and explain each observed red/green boundary.
5. Compile and extend the educational engine with serialized values.
6. Add per-key owners, waiters, cancellation, and a cross-thread cycle detector.
7. Write generated edit-sequence clean-equivalence tests.
8. Trace a structured diagnostic through Fluent, terminal, and JSON.
9. Reproduce and classify a real issue before reading its fix.
10. Submit a narrow test or documentation correction, then a measured implementation change.

Talk one: “A query is a protocol, not a function”#

Start with K -> V. Add provider dispatch, dynamic edges, active jobs, fingerprints, persistence, force, cycles, and side effects one failure at a time. End by deriving clean-build equivalence and showing a concrete edit trace.

Talk two: “Why the wrong error points to the right architecture”#

Follow source bytes through span, hygiene, lowering, metadata, query reuse, Fluent, and JSON. Use one macro suggestion to show that diagnostics are a cross-layer correctness product, not terminal decoration.

Talk three: “The cost of green”#

Derive the full incremental cost model. Compare no-op, body, signature, and cross-crate edits. Show early cutoff, over-invalidation, hash cost, metadata decoding, waiter time, and backend work-product reuse. Finish with the evidence required for a performance contribution.

Prediction exercises#

• If a provider stops reading a tracked option but keeps reading an equivalent global, what graph change appears? None; that absence is the bug.
• If an upstream result reruns and fingerprints equal, can its dependent stay green? Yes, under the declared observation.
• If two external definitions decode to equal types, may their identities be merged? Not generally; semantic equality is not definition identity.
• If terminal diagnostics match but JSON spans differ from clean, is equivalence satisfied? Not for JSON consumers.
• If more threads reduce provider CPU but increase wall time, where should measurement look? Waiters, lock contention, duplicate jobs, memory pressure, and serialization.
• If corrupt cache bytes cause a successful but wrong build, is that only robustness? No; optimization state became semantic authority.

Authoritative reading map#

For the pinned revision, begin with the rustc-dev-guide chapters on the driver, queries, incremental compilation, metadata, diagnostics, parallel compiler, profiling, and stability:

• https://rustc-dev-guide.rust-lang.org/rustc-driver/intro.html
• https://rustc-dev-guide.rust-lang.org/query.html
• https://rustc-dev-guide.rust-lang.org/queries/incremental-compilation.html
• https://rustc-dev-guide.rust-lang.org/queries/incremental-compilation-in-detail.html
• https://rustc-dev-guide.rust-lang.org/backend/libs-and-metadata.html
• https://rustc-dev-guide.rust-lang.org/diagnostics.html
• https://rustc-dev-guide.rust-lang.org/parallel-rustc.html
• https://rustc-dev-guide.rust-lang.org/profiling.html
• https://rustc-dev-guide.rust-lang.org/stability.html

Cross-check current nightly API/source navigation at:

• https://doc.rust-lang.org/nightly/nightly-rustc/rustc_driver/
• https://doc.rust-lang.org/nightly/nightly-rustc/rustc_interface/
• https://doc.rust-lang.org/nightly/nightly-rustc/rustc_session/
• https://doc.rust-lang.org/nightly/nightly-rustc/rustc_target/
• https://doc.rust-lang.org/nightly/nightly-rustc/rustc_middle/query/
• https://doc.rust-lang.org/nightly/nightly-rustc/rustc_middle/dep_graph/
• https://doc.rust-lang.org/nightly/nightly-rustc/rustc_incremental/
• https://doc.rust-lang.org/nightly/nightly-rustc/rustc_metadata/
• https://doc.rust-lang.org/nightly/nightly-rustc/rustc_span/
• https://doc.rust-lang.org/nightly/nightly-rustc/rustc_errors/

Use the rustc book for supported command-line behavior, the Reference for language rules, and the pinned compiler source for implementation claims: https://doc.rust-lang.org/rustc/, https://doc.rust-lang.org/reference/, and https://github.com/rust-lang/rust/tree/1.97.1/compiler.

The final standard is not that an incremental run “looks plausible.” It is that every retained result has an explainable identity, dependency closure, observation, and compatibility boundary; every failure preserves enough provenance to diagnose; and every optimized execution remains equivalent to the clean execution it replaces.

Part VIII-C: Diagnostics, Lints, and Stability Policy#

This continuation starts at the point where analysis knows something is wrong. Its subject is how rustc turns that fact into an honest, reproducible, actionable report, and how closely related policy machinery decides whether accepted Rust is warned about, rejected, migrated, or kept behind a feature boundary.

The implementation descriptions are revision-qualified for rustc 1.97.1. Names and module boundaries are compiler-private unless a language or command-line reference says otherwise. The stable Rust model later in this part teaches contracts, not rustc APIs.

56. The problem before a diagnostic system#

A compiler can stop at the first bad byte and print invalid program. That is sufficient for rejection and nearly useless for repair. A practical compiler must preserve answers to several different questions:

• What semantic invariant failed?
• Which source text best demonstrates the failure?
• Did that text come from the user, a macro argument, or a macro definition?
• Which related locations establish the explanation?
• Is there a complete edit a tool may apply safely?
• Is the condition an unconditional error or a configurable lint?
• Is the construct unavailable because of language, library, const, or edition policy?
• Can analysis continue without inventing a valid program?
• Can an incremental cache replay exactly the required observations?

Printing inside every analysis branch fails these requirements. The branch knows a fact, but often not terminal width, locale, output format, the final lint level, neighboring labels, or whether an earlier error already explains the failure. Immediate printing also turns query execution order into output order.

The smallest useful mental model is a pipeline:

source bytes + provenance
          |
          v
semantic detection ---- policy lookup
          |                 |
          +----> structured diagnostic
                         |
             normalize, sort, deduplicate
                  /                  \
          terminal renderer       JSON emitter

The arrows carry data, not formatted English. The model deliberately separates four concerns:

The first invariant is therefore detection is not presentation. A detector produces structured evidence. Policy may alter level or suppress the report. An emitter turns the surviving structure into an output contract.

A concrete failure trace#

Suppose a macro call passes a string where a u32 is required:

macro_rules! pass { ($value:expr) => { takes_u32($value) } }
fn takes_u32(_: u32) {}
fn main() { pass!("twelve"); }

The type mismatch may be detected on expanded syntax. The most editable text is the argument "twelve", not necessarily the generated call. The report needs the expanded expression's semantic type, the argument token's call-site provenance, the expected type from takes_u32, and an expansion note if the immediate location is otherwise confusing.

If provenance was discarded during expansion or lowering, the emitter cannot reconstruct it from line numbers. If the type checker prints immediately, JSON and terminal consumers may receive inconsistent structures. If recovery forgets that an error was emitted, code generation may later treat an error type as a real layout.

The earliest broken invariant can therefore appear much later:

lost syntax context
  -> plausible but wrong Span
  -> suggestion targets macro definition
  -> rustfix edits dependency source or refuses edit
  -> user sees an “applicability” failure

Do not begin debugging at the last symptom. Trace the first point at which required information was discarded or policy was bypassed.

57. Source files, byte positions, and spans#

Users name files, lines, and displayed columns. The compiler starts with byte sequences. In rustc 1.97.1, rustc_span is the principal source-location and hygiene crate, and SourceMap, SourceFile, BytePos, and Span are useful navigation symbols. Their exact fields and compact encodings are internal.

A source file records enough information to map relative byte offsets to lines. Conceptually it has:

SourceFile
  displayed name
  original/physical naming information
  optional source text
  global start BytePos
  byte length
  line-start offsets
  multibyte/non-narrow character information
  provenance and remapping information

BytePos belongs to a session-global coordinate space. Files occupy non-overlapping intervals in that space.

0       app/src/main.rs       184 185   generated expansion   247
|-------------------------------| |-----------------------------|
       ^ lo=93       ^ hi=101            ^ another span

The useful arithmetic is:

global position = source_file.start_pos + relative byte offset
relative offset = global position - source_file.start_pos

It is not:

column = global position
column = UTF-8 byte offset + 1
column = Unicode scalar count + 1

Tabs, combining characters, wide glyphs, and rendering policy break those equations. Byte offsets remain appropriate for slicing UTF-8 only when both endpoints are character boundaries. Terminal columns are a presentation calculation.

A conceptual span is:

Span = [lo BytePos, hi BytePos) + SyntaxContext

The half-open range permits adjacent spans and zero-width insertion points. SyntaxContext carries hygiene and expansion history. A span is therefore not merely a pair of coordinates.

Span invariants#

For an ordinary source-backed label:

1. lo <= hi.
2. Endpoints map coherently to the intended source file.
3. Endpoints obey source encoding boundaries when text is sliced.
4. The context corresponds to the token's expansion history.
5. The chosen rendered location is useful under the diagnostic's policy.

Not every valid internal span is directly renderable. Dummy spans represent absence rather than file zero. Imported spans may identify upstream source whose snippet is unavailable. Synthetic source can have a name and range without editable disk text. A robust diagnostic handles each state explicitly.

Multispans#

A type mismatch can involve a use, a declaration, and an inferred constraint. A multispan collects a primary span and related labeled spans. It is presentation input, not permission to underline every fact.

fn choose(flag: bool) -> u32 {
    if flag { 1 } else { "no" }
              ^ expected because of return type
                         ^^^^^ found string here
}

Primary means “the location around which this report is organized.” Secondary does not mean unimportant. Too many primaries make ordering and terminal focus ambiguous. Prefer one root location and labels that establish the causal relationship.

A location debugging table#

Prediction: inserting a comment at the top of a file shifts every later BytePos. Does every semantic query need to become red? No. Semantic identity can remain unchanged while source-sensitive diagnostic products need updated locations. The representation and fingerprint must state which observation it preserves.

58. Hygiene provenance: call site, definition site, and mixed site#

Macro hygiene prevents generated identifiers from accidentally capturing or being captured by names that merely share spelling. Location provenance and name-resolution context travel together through SyntaxContext and expansion data. This is why “just replace the span with the invocation span” can repair an underline while breaking a hygiene-sensitive operation elsewhere.

Three terms guide reasoning:

• call site points toward the context where a macro was invoked;
• definition site points toward the macro's definition context;
• mixed site follows the mixed hygiene behavior used for relevant declarative-macro tokens.

These are semantic provenance choices, not aliases for three filenames. The precise APIs and hygiene behavior depend on macro kind and compiler revision. For 1.97.1 contributions, inspect current rustc_span::hygiene and call sites rather than relying on old method names.

Consider:

macro_rules! add_one {
	($input:expr) => {{
		let helper = 1;
		$input + helper
	}};
}

fn main() {
	let helper = "outer";
	let _ = add_one!(helper);
}

The argument token helper originates at the invocation. The generated binding and generated use of helper originate in the definition's expansion context. Spelling alone cannot decide which declaration each use denotes. The type error's best primary location is likely the argument supplied by the caller, while a supporting expansion label may concern generated syntax.

Source callsite#

Nested expansions require more than one step toward a caller. An immediate call site can itself be generated by another macro. A source-callsite operation walks outward toward source-originating invocation context. The exact method behavior must be verified in pinned source, especially around desugaring and proc-macro spans.

user token
  -> outer!(argument) call-site span
      -> generated inner!(argument) call-site span
          -> inner definition token span
              -> diagnostic detection span

source-callsite walk: detection -> inner call -> outer call -> user source

Walking all the way outward is not always best. If an inner macro is authored by the user and contains the actual bug, collapsing to the outermost call hides the repair location. Location selection is policy over preserved provenance.

Expansion backtraces#

An expansion backtrace explains how generated syntax came to exist. Each frame can identify an invocation, macro kind/name, and definition location where available. Backtraces are valuable when the primary span alone looks surprising. They are noisy when every standard macro frame is printed unconditionally.

Renderer policy may compress or omit frames. JSON consumers need structured provenance where the supported schema exposes it, not a parser for human notes. Tests should distinguish “provenance preserved” from “every frame always displayed.”

Provenance experiments#

Use four fixtures:

1. invalid direct syntax;
2. the same token passed as a declarative macro argument;
3. invalid syntax authored in a declarative macro body;
4. nested declarative and procedural expansion.

For each, record:

• raw detection span;
• immediate call site;
• source call site;
• definition-oriented location;
• expansion frames;
• whether source text is available;
• whether an edit is safe;
• terminal and JSON observations.

Counterexample: a suggestion has correct replacement text but targets generated definition-site bytes. It is not machine-applicable merely because the replacement parses. Editability and provenance are part of applicability truth.

59. Structured diagnostics and DiagCtxt#

At the 1.97.1 implementation boundary, begin source navigation in rustc_errors. DiagCtxt is the diagnostic context name to search for, but builder types, ownership, emission methods, derives, and lifetime details can change. Do not expose them as stable APIs.

A structured diagnostic usually contains some combination of:

level: error, warning, note, help, fatal, bug, ...
message: localizable message identity plus arguments
code: optional public-facing diagnostic/lint code
primary span or multispan
span labels
child notes and helps
one-part or multipart suggestions
suggestion applicability
emission metadata

Construction and emission are different events. A builder can accumulate evidence along a branch and then emit once. Emission updates diagnostic state and, for errors, can produce evidence that an error was issued.

detect mismatch
  -> create pending Diag
  -> set primary span
  -> add expected/found labels
  -> add obligation note
  -> add suggestion if honest
  -> emit through DiagCtxt exactly once
  -> receive/propagate error evidence

Primary and secondary labels#

The title should state the root problem without requiring the snippet. The primary label says why the focused bytes matter. Secondary labels establish constraints, origins, or contrasts.

Bad structure:

error: mismatched types
  label A: mismatched types
  label B: mismatched types
  note: mismatched types

Better structure:

error: `if` and `else` have incompatible types
  primary on else: expected `u32`, found `&str`
  secondary on then: this branch has type `u32`

Every child should add a fact, an implication, or an action. Repeated prose consumes attention and makes localization harder.

Suggestions and applicability#

A suggestion is an edit protocol:

message + [(span_1, replacement_1), ...] + applicability

Multipart suggestions are atomic recommendations. Applying only one part may leave invalid syntax or alter meaning. The edits must not overlap and should be ordered deterministically for serialization.

Applicability categories in supported rustc output distinguish confidence. The commonly encountered conceptual meanings are:

Verify exact enum names and serialization in 1.97.1 source and rustc output docs.

Machine-applicable requires more than grammatical output:

• every affected byte range is editable and source-backed;
• replacements are complete;
• no hidden import or rename is required;
• macro projection does not target generated text;
• edits do not conflict;
• the promised transformation is valid for all represented cases;
• escaping and indentation are correct.

If a proposed edit says use SOME_TRAIT;, it has a placeholder. If two traits with the required method may be in scope, confidence may be lower. If the exact import and insertion position are known, a multipart edit may be machine-applicable.

Builder failure modes#

The diagnostic context is also a synchronization and policy boundary. Do not infer that calling an emission method from any parallel provider gives deterministic order. Understand current buffering and sorting behavior before changing concurrency.

60. Error guarantees, taint, delayed bugs, and recovery#

After an error, analysis often must return something. Returning a normal-looking type or node allows later code to forget that an invariant failed. Rustc uses error evidence, including ErrorGuaranteed in current compiler vocabulary, to make “an error was emitted” explicit in types and recovery paths.

Conceptually:

#[derive(Clone, Copy, Debug, Eq, PartialEq)]
struct ErrorReported(());

enum Checked<T> {
	Valid(T),
	Tainted(ErrorReported),
}

This fragment is illustrative, not rustc API. The token does not prove which message was printed or that recovery is semantically valid. It proves only the contract represented by its constructor.

Taint#

A context is tainted when prior errors make some later conclusions unreliable. Taint is not permission to suppress every subsequent error. An unrelated body can still contain an independently actionable mistake.

Useful recovery policy asks:

1. Does this check depend on the erroneous fact?
2. Would the report add an independent constraint?
3. Can the check run without assuming layout, validity, or resolved identity?
4. Can a sentinel preserve the guarantee through this layer?
5. Is this the last safe boundary before artifact production?

Error types often unify permissively to suppress direct type cascades. They must not reach code generation as ordinary types. The driver/session's error state blocks successful artifact completion.

Delayed bugs#

A delayed bug means:

this internal invariant is broken;
it may be downstream of an already reported user error;
if no such error exists, expose an internal compiler error.

This is a diagnostic assertion, not a quieter warning. Replacing a delayed bug with a default value can turn an exposed compiler bug into wrong code or a later crash. Adding an unrelated ordinary error merely to hide it is equally wrong.

Trace a delayed-bug failure backward:

expected user error
  -> should emit and create guarantee
  -> recovery sentinel
  -> later invariant cannot hold
  -> delayed bug registered
  -> finalization checks emitted-error state

missing first emission => delayed bug becomes ICE (correct alarm)

The fix belongs at the earliest lost guarantee or invalid recovery assumption.

Cascade suppression#

A cascade is not “more than one diagnostic.” It is a later report whose only cause is an already explained invalid premise.

Example:

unknown type `Meterss`
  -> recovery creates error type
  -> method lookup cannot find `display`
  -> trait solver sees no formatter implementation

The latter two may be consequences and should often be suppressed. But an unused variable elsewhere is independent.

Suppression strategies have costs:

Recovery failure workshop#

Given an ICE after an unresolved name:

1. Preserve the original program and full diagnostic sequence.
2. Find the first error guarantee creation.
3. Follow the sentinel through lowering, type checking, and query results.
4. Locate the first branch that converts it to a normal value.
5. Ask which invariant the ICE assumes.
6. Restore explicit propagation rather than weakening that invariant.
7. Add a UI regression that checks the user error and absence of ICE.
8. Add a focused internal assertion if the guarantee can be lost again.

61. Localization, terminal output, and JSON#

Structured messages allow presentation to vary without rerunning semantic analysis. Rustc's localization infrastructure uses Fluent resources in relevant diagnostic paths. Migration coverage and derive APIs are revision-sensitive. Inspect the 1.97.1 source's locale resources and the subsystem's diagnostic definitions.

A localizable message should carry named arguments:

typeck-incompatible-types = expected {$expected}, found {$found}

The example is schematic, not an asserted rustc message key. Arguments should represent semantic values, not English sentence fragments such as "because it was". Translators need freedom to reorder clauses and inflect words.

Do not split Rust syntax into translatable punctuation. Code snippets and identifiers need consistent isolation/formatting policy. Pluralization should be represented as a count, not preselected English.

Terminal rendering#

The terminal emitter may choose:

• color according to configuration and output capability;
• line elision for large spans;
• label lanes for overlaps;
• Unicode-aware display widths;
• macro expansion notes;
• wrapping according to available width;
• short or human-readable output variants.

Terminal text is optimized for a reader. Tools should not parse carets, ANSI escapes, or English labels.

JSON rendering#

The documented rustc JSON diagnostic mode is the tool-facing boundary. It can include a rendered human string alongside structured level, message, code, spans, children, suggestion replacements, and expansion information. Consumers must tolerate optional fields according to the documented schema. They must not deserialize compiler-private rustc_errors structures.

Diag structure
  |-- localized message selection
  |-- SourceMap lookup and snippet extraction
  |-- terminal layout -> stderr bytes
  `-- JSON serialization -> one machine-readable record

Terminal equality does not imply JSON equality. Two reports can render identically while one has the wrong primary flag, missing replacement, or different expansion provenance. JSON tests must assert structure.

Determinism#

Parallel analysis can discover independent diagnostics in either order. Stable presentation requires an explicit ordering policy, often based on source/provenance and stable tie breakers rather than worker arrival. Never use pointer values, randomized map order, or thread IDs as semantic ties.

Determinism has a cost. Buffering all diagnostics uses memory and delays feedback. Sorting needs stable keys even for source-less reports. Streaming improves latency but exposes schedule order. The compiler must choose an observation and test it; contributors should not promise more than current behavior.

Deduplication#

Text equality is a poor deduplication key. Two distinct errors can share wording and span. The same root error can acquire slightly different notes along two paths.

A conceptual identity may include:

diagnostic kind/code
semantic owner or root cause
primary source provenance
relevant substitutions
emission phase

Over-deduplication hides errors. Under-deduplication creates noise. Prefer preventing duplicate production through ownership and guarantees; use emitter deduplication only under a documented identity.

Output security#

Diagnostics can leak:

• absolute workspace and home paths;
• source snippets containing secrets;
• environment-derived linker arguments;
• proc-macro output;
• dependency paths;
• control characters that manipulate terminals or logs.

Path remapping supports reproducibility and privacy but is not a complete redaction system. JSON preserves data that terminal layouts may hide. CI artifact policies must treat diagnostics as potentially sensitive. Render untrusted names safely and avoid interpreting source-controlled escape sequences as terminal control.

62. Diagnostic side effects and incremental boundaries#

A query is memoized as though it were a function from tracked inputs to an observable result. Direct diagnostic emission is a side effect. If a green query does not execute, its direct print does not happen. If a cached diagnostic is replayed and an eager caller also emits it, it happens twice.

The semantic contract is:

diagnostics(clean(inputs)) == diagnostics(incremental(inputs))

Equality needs a normalization policy for paths, unstable hashes, and intentionally unordered details. It still includes codes, levels, spans, suggestions, and expectation fulfillment where tools observe them.

Three designs are possible:

No design removes responsibility. It moves construction, storage, policy application, and replay boundaries.

Diagnostic-only options complicate reuse. Color and width should not usually invalidate semantic type checking. Lint levels can change whether the invocation succeeds. Locale can change rendered words without changing spans. Path remapping changes displayed filenames and potentially stable artifact observations.

Separate:

semantic fact: an unnecessary allocation exists
policy fact: lint level is deny here
presentation fact: localized wording and terminal width

Caching the fully rendered string couples all three. Caching only the semantic fact requires reapplying policy correctly.

Incremental failure traces#

Missing warning

revision 1 query executes -> warning printed directly
revision 2 unrelated edit -> query green -> provider skipped
                         -> warning absent

Duplicate error

cached query diagnostic replayed
              + eager crate visitor emits same fact
              -> two records, error count incremented twice

Wrong level

old run #[allow(lint)] caches “suppressed” result
new run -D lint changes policy
semantic query reused without level reevaluation
warning incorrectly remains absent

The earliest invariant in the third trace is that policy became hidden inside a cache whose dependency closure omitted policy input.

Side-effect audit#

For every diagnostic-producing query, answer:

1. What exact key owns the logical diagnostic?
2. Is the report in the result, captured, or emitted elsewhere?
3. What happens on a memory hit?
4. What happens on a disk hit?
5. What happens when the provider reruns but fingerprints equal?
6. Which options alter detection, level, structure, and rendering?
7. How is error state restored?
8. How are expectation fulfillments restored?
9. Can parallel demand duplicate ownership?
10. Which clean-versus-incremental test proves the contract?

63. Lint passes, levels, groups, caps, and expectations#

A lint is a named, configurable diagnostic policy point. Detection says a condition exists. Level resolution says whether it is allowed, warned, denied, or forbidden. At rustc 1.97.1, source navigation commonly begins in rustc_lint, with level and registration machinery spread across compiler crates. Use current declarations to locate early, late, and other pass interfaces.

Pass timing#

Different representations preserve different evidence:

Running later is not automatically more powerful. Lowering can erase parentheses, exact tokens, or syntactic distinctions a style lint needs. Running earlier means type-driven exceptions are unavailable. Choose the earliest representation containing all required facts and preserving useful spans.

Level resolution#

Conceptually, a lint level comes from:

lint default
  overridden by command line and scoped attributes
  constrained by forbid semantics
  affected by groups/renames
  capped by invocation policy
  integrated with expectation bookkeeping
  -> effective level at a particular node

The precise precedence and edge cases are normative user behavior. Use the rustc lint-level documentation for 1.97.1 rather than deriving rules from this diagram.

allow suppresses ordinary emission. warn emits a warning. deny promotes the lint to an error. forbid prevents inner lowering of that lint level under documented rules. A cap limits maximum effective severity, notably for dependency compilation workflows.

A lint group names several lints. Groups improve policy ergonomics but create migration obligations: adding a lint to a widely denied group can break builds. Renamed and removed lints need diagnostics and compatibility handling. Group expansion must be deterministic and detect cycles if extension mechanisms can create them.

Expectations#

#[expect(lint_name)] is not merely allow with different spelling. It records an obligation that a matching lint should occur. When the condition is found, the expectation is fulfilled and ordinary output is controlled accordingly. If no matching lint occurs, the unfulfilled expectation itself can be diagnosed.

parse expectation attribute
  -> assign stable-enough expectation identity and scope
  -> lint detector reports condition
  -> level lookup finds expectation
  -> mark fulfillment
  -> final expectation check reports any unfulfilled entries

Incremental correctness requires fulfillment bookkeeping to survive edits honestly. If the triggering expression is removed while the attribute remains, the next run must match a clean build and report the unfulfilled expectation as policy requires.

Tool lints#

Names such as tool::lint_name reserve a tool namespace under documented rules. This lets tools such as Clippy integrate attribute-level policy without pretending every lint is built into rustc. Unknown-tool and unknown-lint behavior depends on tool registration and invocation context. Do not silently reinterpret a tool lint as a built-in lint with the same suffix.

Tool lints cross a compatibility boundary:

• rustc parses and scopes attributes;
• the tool owns detection and often defaults/groups;
• command-line and cap policy must compose;
• users need useful behavior when the tool is absent;
• incremental keys must include tool-relevant configuration.

Lint detector checklist#

Before adding a lint:

1. State the harmful or suspicious pattern.
2. Give valid counterexamples that must not warn.
3. Choose pass timing based on required evidence.
4. Identify the primary editable span.
5. Choose a conservative default level.
6. Define future-compatibility status if applicable.
7. Decide whether a suggestion is complete and semantics-preserving.
8. Test scoped allow/warn/deny/forbid.
9. Test group and cap interaction.
10. Test expectations and incremental revisions.
11. Test macro-generated and external-macro code.
12. Test JSON and rustfix behavior when suggestions exist.

64. Feature gates and staged stability#

Feature gates and stability attributes both control evolution, but they guard different boundaries.

Language feature gating asks whether syntax or semantics may be used on this channel/toolchain, often with a crate-level #![feature(...)] opt-in available only where policy permits. Library stability asks whether an API item is stable for use and since which release, or unstable under a tracking issue/feature name.

Do not infer one from the other. A stable parser can recognize syntax that is still gated so it can issue a targeted error. A syntactically ordinary call can refer to an unstable standard-library item.

Gate placement#

A gate check needs enough context to distinguish the intended feature. Checking too early can reject syntax that has a stable interpretation. Checking too late allows unsupported meaning to influence analysis and produces cascades.

lex/parse construct
  -> preserve feature marker and span
  -> expansion/lowering as required
  -> gate check at discriminating boundary
  -> emit feature diagnostic + tracking/help information
  -> recovery node carrying error evidence

Gating must account for macro expansion provenance. Compiler- or library-internal expansions sometimes require allowances so stable public macros can expand to implementation details. Such allowances are narrow trust mechanisms, not general user opt-outs.

Stability attributes#

Staged API stability conceptually records:

stable: feature identity + stabilization version
unstable: feature identity + reason + tracking metadata
deprecated: since + note/suggestion policy
const-stable/const-unstable: const-context availability
promotion or indirect stability metadata where required

Exact attribute spellings, accepted fields, and internal enforcement are revision-sensitive. They are compiler/bootstrap infrastructure, not ordinary stable user attributes. Inspect the pinned standard library and rustc_passes/stability-related code paths for 1.97.1.

Stability is checked at use sites, not only definitions. Reexports, trait items, associated items, fields, implementations, and generated references can require special handling. The path the user wrote and the definition ultimately resolved can provide different diagnostic context.

Const stability#

An API being stable at runtime does not imply it is stable in const evaluation. Const stability allows a function or operation to become callable in const contexts on a separate schedule.

const VALUE: usize = some_api();

The call may be valid in ordinary code and gated or unstable in this context. The checker needs both resolved-item stability and const-context information. Diagnostics should say that const use is unavailable rather than claiming the whole API is unstable.

Const stability creates compatibility obligations. Once stable const evaluation accepts behavior, changes to the const evaluator and API implementation must preserve the promised observation. Runtime and compile-time execution can expose different panics, resource costs, and target assumptions.

allow_internal mechanics#

Internal attributes commonly discussed as allow_internal_unstable and allow_internal_unsafe support trusted expansion boundaries in compiler/standard-library infrastructure. Their exact validation and effects must be read from 1.97.1 source.

The core motivation is counterfactual:

• A stable macro may need to expand to an unstable implementation detail.
• Without a narrow allowance, every user would need the internal feature.
• With a broad allowance, the macro could accidentally expose arbitrary unstable behavior.

Therefore the allowance must be attached to a trusted definition boundary, limited to named features or a precise safety behavior, and propagated through expansion provenance correctly. It does not stabilize the internal item for direct user use.

allow_internal_unsafe concerns where unsafe operations generated by trusted macros are attributed under relevant lint/safety policy. It must not be described as making an unsafe operation safe. The safety proof obligation still belongs to the macro implementation and its contract.

Stability failure trace#

stable user crate
  -> invokes stable standard macro
  -> expansion references unstable internal intrinsic
  -> provenance identifies trusted macro definition
  -> named internal allowance applies only to expansion
  -> direct user reference remains rejected

If the provenance mark is lost, the stable macro may spuriously fail. If the allowance leaks past its expansion, direct unstable use may be incorrectly accepted. Test both directions.

65. Edition migrations as lint-driven policy#

An edition changes selected language behavior while preserving ecosystem migration paths. The edition is a crate-level input, not a global compiler mode for every dependency. Crates of different editions interoperate.

Migration commonly uses lints before or while changing edition interpretation:

old edition source accepted
  -> compatibility lint identifies future ambiguity/change
  -> suggestion rewrites source while preserving old meaning
  -> cargo fix applies machine-applicable edits
  -> crate opts into new edition
  -> new parser/resolver semantics accept rewritten source

A migration lint has a stronger contract than ordinary style advice. Its edit should preserve behavior under the old edition and produce the intended behavior under the new edition. That is a cross-version semantic claim.

Migration hazards#

• macro-generated syntax may not be editable at the use site;
• a macro definition can be in a crate with a different edition;
• tokenization changes can alter how an edit is parsed;
• name-resolution changes can make a simple rename capture another binding;
• multipart edits can overlap across lint instances;
• cfg-disabled code may escape the current compilation;
• build scripts and generated files may need separate ownership;
• rustfmt can alter snapshots after migration without changing semantics.

Edition-specific parsing and expansion need the edition associated with the relevant span/context. Using only the final crate's edition can misinterpret cross-edition macro tokens.

Designing a migration lint#

1. State old and new semantics normatively.
2. Enumerate syntax forms affected.
3. Identify cases whose meaning already differs for unrelated reasons.
4. Preserve token and expansion provenance through detection.
5. Construct complete non-overlapping edits.
6. Downgrade applicability for generated or ambiguous text.
7. Test both editions before and after applying suggestions.
8. Test macros defined and invoked across edition combinations.
9. Test cargo fix --edition-style machine consumption where applicable.
10. Verify repeated migration is idempotent.

Edition test matrix#

Do not assert one blanket answer for all macro constructs. Use the Reference's edition rule and expansion-specific implementation tests.

66. A stable-Rust structured diagnostic and lint model#

The following complete program builds the educational model progressively in one file. It supports UTF-8 byte spans, source provenance, primary/secondary labels, structured suggestions, lint levels, groups, caps, expectations, deterministic sorting, deduplication by explicit identity, taint, terminal rendering, and deterministic JSON-like output.

It uses only stable Rust and the standard library. Save it as diag_lab.rs and compile with:

rustc --edition=2021 --test diag_lab.rs -o diag_lab_tests
./diag_lab_tests

use std::cmp::Ordering;
use std::collections::{BTreeMap, BTreeSet};
use std::fmt::Write;
use std::ops::Range;

#[derive(Clone, Debug, Eq, PartialEq, Ord, PartialOrd)]
struct FileId(String);

#[derive(Clone, Debug, Eq, PartialEq)]
struct SourceFile {
	id: FileId,
	display_name: String,
	text: String,
	line_starts: Vec<usize>,
}

impl SourceFile {
	fn new(id: &str, display_name: &str, text: &str) -> Self {
		let mut line_starts = vec![0];
		for (index, byte) in text.bytes().enumerate() {
			if byte == b'\n' { line_starts.push(index + 1); }
		}
		Self {
			id: FileId(id.into()),
			display_name: display_name.into(),
			text: text.into(),
			line_starts,
		}
	}

	fn validate(&self, range: &Range<usize>) -> Result<(), String> {
		if range.start > range.end || range.end > self.text.len() {
			return Err("span is outside its source file".into());
		}
		if !self.text.is_char_boundary(range.start)
			|| !self.text.is_char_boundary(range.end)
		{
			return Err("span splits a UTF-8 code point".into());
		}
		Ok(())
	}

	fn line_col(&self, byte: usize) -> Result<(usize, usize), String> {
		self.validate(&(byte..byte))?;
		let line_index = self.line_starts.partition_point(|start| *start <= byte) - 1;
		let line_start = self.line_starts[line_index];
		let scalar_column = self.text[line_start..byte].chars().count() + 1;
		Ok((line_index + 1, scalar_column))
	}

	fn line_text(&self, one_based_line: usize) -> Option<&str> {
		let start = *self.line_starts.get(one_based_line.checked_sub(1)?)?;
		let end = self.line_starts.get(one_based_line).copied()
			.unwrap_or(self.text.len());
		Some(self.text[start..end].trim_end_matches('\n'))
	}
}

#[derive(Clone, Debug, Eq, PartialEq)]
enum Origin {
	Source,
	Expansion {
		macro_name: String,
		call_site: Box<Span>,
		definition_site: Box<Span>,
	},
}

#[derive(Clone, Debug, Eq, PartialEq)]
struct Span {
	file: FileId,
	bytes: Range<usize>,
	origin: Origin,
}

impl Span {
	fn source(file: &str, bytes: Range<usize>) -> Self {
		Self { file: FileId(file.into()), bytes, origin: Origin::Source }
	}

	fn source_callsite(&self) -> &Span {
		match &self.origin {
			Origin::Source => self,
			Origin::Expansion { call_site, .. } => call_site.source_callsite(),
		}
	}

	fn expansion_names(&self, out: &mut Vec<String>) {
		if let Origin::Expansion { macro_name, call_site, .. } = &self.origin {
			out.push(macro_name.clone());
			call_site.expansion_names(out);
		}
	}
}

#[derive(Clone, Copy, Debug, Eq, PartialEq, Ord, PartialOrd)]
enum Level { Warning, Error }

#[derive(Clone, Copy, Debug, Eq, PartialEq, Ord, PartialOrd)]
enum Applicability {
	MachineApplicable,
	MaybeIncorrect,
	HasPlaceholders,
	Unspecified,
}

#[derive(Clone, Debug, Eq, PartialEq)]
struct Label {
	span: Span,
	message: String,
	primary: bool,
}

#[derive(Clone, Debug, Eq, PartialEq)]
struct Edit {
	span: Span,
	replacement: String,
}

#[derive(Clone, Debug, Eq, PartialEq)]
struct Suggestion {
	message: String,
	edits: Vec<Edit>,
	applicability: Applicability,
}

#[derive(Clone, Debug, Eq, PartialEq, Ord, PartialOrd)]
struct DiagId {
	kind: String,
	owner: String,
	root_file: FileId,
	root_byte: usize,
}

#[derive(Clone, Debug, Eq, PartialEq)]
struct Diagnostic {
	id: DiagId,
	level: Level,
	code: Option<String>,
	message: String,
	labels: Vec<Label>,
	notes: Vec<String>,
	suggestions: Vec<Suggestion>,
}

impl Diagnostic {
	fn primary(&self) -> Option<&Span> {
		self.labels.iter().find(|label| label.primary).map(|label| &label.span)
	}

	fn sort_key(&self) -> (&FileId, usize, &str, &str) {
		(&self.id.root_file, self.id.root_byte, &self.id.kind, &self.id.owner)
	}

	fn validate(&self, files: &BTreeMap<FileId, SourceFile>) -> Result<(), String> {
		if self.labels.iter().filter(|label| label.primary).count() != 1 {
			return Err("a diagnostic must have exactly one primary label".into());
		}
		for label in &self.labels {
			let file = files.get(&label.span.file)
				.ok_or_else(|| "label refers to an unknown file".to_string())?;
			file.validate(&label.span.bytes)?;
		}
		for suggestion in &self.suggestions {
			let mut prior: Option<(&FileId, usize)> = None;
			for edit in &suggestion.edits {
				let file = files.get(&edit.span.file)
					.ok_or_else(|| "edit refers to an unknown file".to_string())?;
				file.validate(&edit.span.bytes)?;
				if suggestion.applicability == Applicability::MachineApplicable
					&& !matches!(edit.span.origin, Origin::Source)
				{
					return Err("machine edit targets generated source".into());
				}
				if let Some((old_file, old_end)) = prior {
					if old_file == &edit.span.file && edit.span.bytes.start < old_end {
						return Err("suggestion edits overlap or are unsorted".into());
					}
				}
				prior = Some((&edit.span.file, edit.span.bytes.end));
			}
		}
		Ok(())
	}
}

#[derive(Clone, Copy, Debug, Eq, PartialEq, Ord, PartialOrd)]
enum LintLevel { Allow, Warn, Deny, Forbid }

impl LintLevel {
	fn severity(self) -> u8 {
		match self {
			Self::Allow => 0,
			Self::Warn => 1,
			Self::Deny => 2,
			Self::Forbid => 3,
		}
	}

	fn capped(self, cap: Self) -> Self {
		if self.severity() <= cap.severity() { self } else { cap }
	}
}

#[derive(Clone, Debug)]
struct Expectation {
	id: String,
	lint: String,
	span: Span,
	fulfilled: bool,
}

#[derive(Clone, Debug)]
struct LintPolicy {
	defaults: BTreeMap<String, LintLevel>,
	overrides: BTreeMap<String, LintLevel>,
	groups: BTreeMap<String, BTreeSet<String>>,
	cap: LintLevel,
	expectations: Vec<Expectation>,
}

impl LintPolicy {
	fn expand(&self, name: &str, seen: &mut BTreeSet<String>) -> Result<BTreeSet<String>, String> {
		if !seen.insert(name.into()) { return Err(format!("lint-group cycle at {name}")); }
		let mut result = BTreeSet::new();
		if let Some(members) = self.groups.get(name) {
			for member in members {
				result.extend(self.expand(member, seen)?);
			}
		} else {
			result.insert(name.into());
		}
		seen.remove(name);
		Ok(result)
	}

	fn level(&self, lint: &str) -> LintLevel {
		let mut chosen = self.defaults.get(lint).copied().unwrap_or(LintLevel::Warn);
		for (name, level) in &self.overrides {
			let contains = self.expand(name, &mut BTreeSet::new())
				.map(|set| set.contains(lint)).unwrap_or(false);
			if contains && level.severity() >= chosen.severity() { chosen = *level; }
		}
		chosen.capped(self.cap)
	}

	fn fulfill(&mut self, lint: &str) -> bool {
		if let Some(expectation) = self.expectations.iter_mut()
			.find(|expectation| expectation.lint == lint && !expectation.fulfilled)
		{
			expectation.fulfilled = true;
			true
		} else {
			false
		}
	}
}

#[derive(Clone, Copy, Debug, Eq, PartialEq)]
struct ErrorReported(());

#[derive(Default)]
struct DiagCtxt {
	files: BTreeMap<FileId, SourceFile>,
	diagnostics: Vec<Diagnostic>,
	seen: BTreeSet<DiagId>,
	tainted: bool,
}

impl DiagCtxt {
	fn add_file(&mut self, file: SourceFile) { self.files.insert(file.id.clone(), file); }

	fn emit(&mut self, diagnostic: Diagnostic) -> Result<Option<ErrorReported>, String> {
		diagnostic.validate(&self.files)?;
		if !self.seen.insert(diagnostic.id.clone()) { return Ok(None); }
		let is_error = diagnostic.level == Level::Error;
		self.diagnostics.push(diagnostic);
		if is_error {
			self.tainted = true;
			Ok(Some(ErrorReported(())))
		} else {
			Ok(None)
		}
	}

	fn emit_lint(
		&mut self,
		policy: &mut LintPolicy,
		lint: &str,
		mut diagnostic: Diagnostic,
	) -> Result<Option<ErrorReported>, String> {
		if policy.fulfill(lint) { return Ok(None); }
		match policy.level(lint) {
			LintLevel::Allow => Ok(None),
			LintLevel::Warn => { diagnostic.level = Level::Warning; self.emit(diagnostic) }
			LintLevel::Deny | LintLevel::Forbid => {
				diagnostic.level = Level::Error;
				self.emit(diagnostic)
			}
		}
	}

	fn finish_expectations(&mut self, policy: &LintPolicy) -> Result<(), String> {
		for expectation in policy.expectations.iter().filter(|item| !item.fulfilled) {
			let diagnostic = Diagnostic {
				id: DiagId {
					kind: "unfulfilled_expectation".into(),
					owner: expectation.id.clone(),
					root_file: expectation.span.file.clone(),
					root_byte: expectation.span.bytes.start,
				},
				level: Level::Warning,
				code: Some("unfulfilled_lint_expectations".into()),
				message: format!("expected lint `{}` did not occur", expectation.lint),
				labels: vec![Label {
					span: expectation.span.clone(),
					message: "expectation declared here".into(),
					primary: true,
				}],
				notes: vec![], suggestions: vec![],
			};
			self.emit(diagnostic)?;
		}
		Ok(())
	}

	fn sorted(&self) -> Vec<&Diagnostic> {
		let mut diagnostics: Vec<_> = self.diagnostics.iter().collect();
		diagnostics.sort_by(|left, right| {
			let order = left.sort_key().cmp(&right.sort_key());
			if order == Ordering::Equal { left.message.cmp(&right.message) } else { order }
		});
		diagnostics
	}

	fn render_terminal(&self) -> Result<String, String> {
		let mut output = String::new();
		for diagnostic in self.sorted() {
			let level = match diagnostic.level { Level::Warning => "warning", Level::Error => "error" };
			writeln!(output, "{level}: {}", diagnostic.message).unwrap();
			for label in &diagnostic.labels {
				let file = &self.files[&label.span.file];
				let (line, column) = file.line_col(label.span.bytes.start)?;
				let marker = if label.primary { "-->" } else { ":::" };
				writeln!(output, " {marker} {}:{line}:{column}", file.display_name).unwrap();
				writeln!(output, "  | {}", file.line_text(line).unwrap_or("<source unavailable>")).unwrap();
				writeln!(output, "  = {}", label.message).unwrap();
			}
			for note in &diagnostic.notes { writeln!(output, "  = note: {note}").unwrap(); }
			for suggestion in &diagnostic.suggestions {
				writeln!(output, "  = help: {} [{:?}]", suggestion.message, suggestion.applicability).unwrap();
			}
		}
		Ok(output)
	}

	fn render_json_lines(&self) -> String {
		fn escaped(text: &str) -> String {
			let mut out = String::new();
			for ch in text.chars() {
				match ch {
					'"' => out.push_str("\\\""), '\\' => out.push_str("\\\\"),
					'\n' => out.push_str("\\n"), '\r' => out.push_str("\\r"),
					'\t' => out.push_str("\\t"), c if c.is_control() => {
						write!(out, "\\u{:04x}", c as u32).unwrap();
					}, c => out.push(c),
				}
			}
			out
		}
		let mut output = String::new();
		for diagnostic in self.sorted() {
			let level = match diagnostic.level { Level::Warning => "warning", Level::Error => "error" };
			let primary = diagnostic.primary().unwrap();
			writeln!(output,
				"{{\"level\":\"{}\",\"message\":\"{}\",\"file\":\"{}\",\"start\":{}}}",
				level, escaped(&diagnostic.message), escaped(&primary.file.0), primary.bytes.start
			).unwrap();
		}
		output
	}
}

fn sample_diagnostic() -> Diagnostic {
	let span = Span::source("main", 4..9);
	Diagnostic {
		id: DiagId { kind: "need_type".into(), owner: "main".into(), root_file: span.file.clone(), root_byte: 4 },
		level: Level::Error,
		code: Some("EDEMO".into()),
		message: "expected an integer".into(),
		labels: vec![Label { span: span.clone(), message: "this is text".into(), primary: true }],
		notes: vec!["the teaching language requires an integer here".into()],
		suggestions: vec![Suggestion {
			message: "replace it with a number".into(),
			edits: vec![Edit { span, replacement: "12".into() }],
			applicability: Applicability::MachineApplicable,
		}],
	}
}

fn main() {
	let mut dcx = DiagCtxt::default();
	dcx.add_file(SourceFile::new("main", "/workspace/main.toy", "let text = nope;\n"));
	let guarantee = dcx.emit(sample_diagnostic()).unwrap();
	assert_eq!(guarantee, Some(ErrorReported(())));
	print!("{}", dcx.render_terminal().unwrap());
}

#[cfg(test)]
mod tests {
	use super::*;

	fn context() -> DiagCtxt {
		let mut dcx = DiagCtxt::default();
		dcx.add_file(SourceFile::new("main", "src/main.toy", "let nope = text;\n"));
		dcx
	}

	fn policy() -> LintPolicy {
		LintPolicy {
			defaults: BTreeMap::from([("demo".into(), LintLevel::Warn)]),
			overrides: BTreeMap::new(), groups: BTreeMap::new(),
			cap: LintLevel::Forbid, expectations: vec![],
		}
	}

	#[test]
	fn byte_positions_do_not_claim_byte_columns() {
		let file = SourceFile::new("u", "unicode.toy", "αx\n");
		assert_eq!(file.line_col(2).unwrap(), (1, 2));
		assert!(file.validate(&(1..2)).is_err());
	}

	#[test]
	fn source_callsite_walks_nested_expansions() {
		let source = Span::source("main", 4..8);
		let generated = Span {
			file: FileId("macro".into()), bytes: 0..2,
			origin: Origin::Expansion {
				macro_name: "outer".into(), call_site: Box::new(source.clone()),
				definition_site: Box::new(Span::source("macro", 0..2)),
			},
		};
		assert_eq!(generated.source_callsite(), &source);
		let mut names = vec![];
		generated.expansion_names(&mut names);
		assert_eq!(names, ["outer"]);
	}

	#[test]
	fn emission_taints_once_and_deduplicates() {
		let mut dcx = context();
		assert_eq!(dcx.emit(sample_diagnostic()).unwrap(), Some(ErrorReported(())));
		assert_eq!(dcx.emit(sample_diagnostic()).unwrap(), None);
		assert!(dcx.tainted);
		assert_eq!(dcx.diagnostics.len(), 1);
	}

	#[test]
	fn machine_edit_must_target_source() {
		let mut dcx = context();
		let call = Span::source("main", 4..8);
		let generated = Span {
			file: FileId("main".into()), bytes: 4..8,
			origin: Origin::Expansion {
				macro_name: "m".into(), call_site: Box::new(call.clone()),
				definition_site: Box::new(call),
			},
		};
		let mut diagnostic = sample_diagnostic();
		diagnostic.suggestions[0].edits[0].span = generated;
		assert!(dcx.emit(diagnostic).unwrap_err().contains("generated"));
	}

	#[test]
	fn deny_turns_lint_into_error() {
		let mut dcx = context();
		let mut policy = policy();
		policy.overrides.insert("demo".into(), LintLevel::Deny);
		let result = dcx.emit_lint(&mut policy, "demo", sample_diagnostic()).unwrap();
		assert_eq!(result, Some(ErrorReported(())));
	}

	#[test]
	fn cap_limits_a_denied_lint() {
		let mut dcx = context();
		let mut policy = policy();
		policy.overrides.insert("demo".into(), LintLevel::Deny);
		policy.cap = LintLevel::Warn;
		assert_eq!(dcx.emit_lint(&mut policy, "demo", sample_diagnostic()).unwrap(), None);
		assert_eq!(dcx.diagnostics[0].level, Level::Warning);
	}

	#[test]
	fn expectation_is_fulfilled_without_emission() {
		let mut dcx = context();
		let mut policy = policy();
		policy.expectations.push(Expectation {
			id: "exp-1".into(), lint: "demo".into(),
			span: Span::source("main", 0..3), fulfilled: false,
		});
		dcx.emit_lint(&mut policy, "demo", sample_diagnostic()).unwrap();
		dcx.finish_expectations(&policy).unwrap();
		assert!(dcx.diagnostics.is_empty());
	}

	#[test]
	fn missing_expected_lint_is_reported() {
		let mut dcx = context();
		let mut policy = policy();
		policy.expectations.push(Expectation {
			id: "exp-2".into(), lint: "demo".into(),
			span: Span::source("main", 0..3), fulfilled: false,
		});
		dcx.finish_expectations(&policy).unwrap();
		assert_eq!(dcx.diagnostics[0].code.as_deref(), Some("unfulfilled_lint_expectations"));
	}

	#[test]
	fn output_is_deterministic_and_json_is_escaped() {
		let mut left = context();
		let mut a = sample_diagnostic();
		a.message = "quote: \"x\"".into();
		a.id.owner = "z".into();
		let mut b = sample_diagnostic();
		b.id.root_byte = 0;
		b.id.owner = "a".into();
		b.labels[0].span = Span::source("main", 0..3);
		left.emit(a.clone()).unwrap();
		left.emit(b.clone()).unwrap();

		let mut right = context();
		right.emit(b).unwrap();
		right.emit(a).unwrap();
		assert_eq!(left.render_json_lines(), right.render_json_lines());
		assert!(left.render_json_lines().contains("\\\"x\\\""));
	}

	#[test]
	fn lint_group_cycles_are_rejected() {
		let mut policy = policy();
		policy.groups.insert("a".into(), BTreeSet::from(["b".into()]));
		policy.groups.insert("b".into(), BTreeSet::from(["a".into()]));
		assert!(policy.expand("a", &mut BTreeSet::new()).is_err());
	}
}

What the model establishes#

The source map validates UTF-8 byte boundaries. It reports scalar columns but explicitly does not solve grapheme width, tabs, or terminal width. The span stores expansion provenance and can walk to a source call site. The diagnostic has one primary label, structured edits, applicability, and explicit identity.

The context emits once per identity and returns error evidence. Sorting makes output independent of insertion order. JSON escaping prevents source-controlled quotes and controls from corrupting records. The lint policy separates detection from effective level and expectation fulfillment.

Explicit omissions#

This is not a small rustc replacement. It omits:

• a session-global BytePos address space;
• line-table compression and external source loading;
• tabs, grapheme clusters, wide glyphs, and multiline labels;
• true hygiene and mixed-site name resolution;
• proc-macro server boundaries;
• Fluent resource loading and plural selection;
• the complete rustc JSON schema;
• nested scope precedence and true forbid semantics;
• lint renames, aliases, future-incompatibility reports, and tool registration;
• query capture, disk replay, and dep-graph integration;
• thread-safe emission and streaming;
• delayed-bug finalization;
• feature and API stability tables;
• edition-aware tokenization;
• rustfix conflict resolution;
• source secrecy policy and full path remapping;
• bounded memory, cancellation, and malformed metadata handling.

Each omission is a design boundary. Adding it requires tests and an observation contract, not just another field.

67. Hardening the model for production reasoning#

The teaching model accepts all source in memory. Rustc sees virtual files, imported metadata, generated source, very large files, malformed proc-macro output, and paths from different hosts.

Resource limits#

Bound:

• number of labels per diagnostic;
• total snippet bytes retained;
• expansion backtrace depth;
• child diagnostics and suggestion edits;
• decoded JSON/source-map lengths;
• recursion in provenance and lint groups;
• queued diagnostics before sorting;
• duplicate-identity storage.

Truncation must be explicit. Silently dropping the primary span is worse than rendering a source-less diagnostic with a note. A malicious macro should not force quadratic label layout or unbounded expansion notes.

Concurrency#

A production context needs an ownership protocol:

detector threads
  -> construct thread-local/buffered diagnostics
  -> validate immutable spans and messages
  -> transfer ownership to collector
  -> stable ordering/dedup policy
  -> single or coordinated emitter

Avoid locks held while source providers, localization, or arbitrary query calls execute. Cancellation must release pending builders and wake query waiters. An interrupted invocation must not serialize a partial “diagnostics complete” cache entry.

Deterministic identities#

The model's string owner is convenient and expensive. Production identity may use stable definition identity and diagnostic kind, but cannot assume every parse error has a definition. Source-less errors need deterministic tie breakers. Macro expansions need stable provenance rather than expansion allocation order.

Hashing a fully localized message is wrong for semantic identity. Changing locale would evade deduplication. Hashing only error code and byte position is too coarse.

Security and privacy#

Treat these as distinct controls:

Rust compilation normally executes build scripts and procedural macros in the build environment's security context. Diagnostics infrastructure is not a sandbox. Do not compile hostile projects merely to inspect their errors without an external isolation strategy.

Suggestion hardening#

Before assigning machine applicability, run the edit in a temporary copy and reparse. For migration tools, compile under both relevant editions when practical. Check that multipart ranges do not overlap after path normalization. Reject edits to read-only dependency source. Preserve newline style and avoid splitting UTF-8.

Reparsing is necessary but not sufficient. x + 1 and x - 1 both parse. Semantic preservation needs a stronger argument or a weaker applicability.

68. Testing diagnostics and policy#

Rustc's UI tests compile fixtures and compare normalized output with checked-in expectations. At the 1.97.1 boundary, inspect current tests/ui conventions and compiletest documentation/source. Directives, normalization syntax, bless workflows, and revision handling evolve.

What a UI snapshot proves#

A .stderr snapshot can prove the normalized human output for one invocation. It does not alone prove:

• JSON structure;
• suggestion applicability;
• edit validity;
• clean/incremental equivalence;
• another target's behavior;
• all locale behavior;
• absence of hidden path leaks before normalization.

Use the narrowest complementary test for each contract.

Revisions#

Revisions let one fixture run under multiple configurations, such as editions, feature flags, lint levels, or incremental stages. Shared source reduces drift but can obscure which revision establishes which invariant. Name revisions semantically and keep directives local where possible.

Conceptual fixture:

//@ revisions: warn deny
//@[warn] compile-flags: -W demo_lint
//@[deny] compile-flags: -D demo_lint

fn main() { trigger(); }

This syntax is illustrative. Verify exact 1.97.1 compiletest directives before committing.

Normalization#

Normalize environmental noise:

• temporary roots;
• platform separators where intended;
• nondeterministic hashes explicitly known to be irrelevant;
• compiler build paths;
• target-specific addresses when not under test.

Do not normalize the behavior being tested. Replacing every line number with $LINE cannot catch a span regression. Replacing every path cannot catch a remapping leak. Overbroad regexes can make empty output pass.

Suggestion testing#

Test at least:

1. displayed help and replacement;
2. machine applicability in structured output;
3. applying all multipart edits;
4. parsing/compiling the result under promised assumptions;
5. no overlapping edits;
6. macro and Unicode cases;
7. rustfix behavior where that tool is the consumer.

An intentionally non-applicable suggestion needs a test too. Otherwise a later refactor may accidentally upgrade its promise.

JSON and machine-readable output#

Capture rustc's JSON mode and parse records as JSON. Assert fields rather than substring order inside objects. Check:

• level and diagnostic code;
• primary span flag;
• byte and line/column positions;
• suggested replacement and applicability;
• children and expansion provenance where contractually exposed;
• rendered field only if human rendering is under test;
• absence or remapping of physical paths.

Do not use the teaching model's four-field JSON as rustc's schema. Consult the rustc book's JSON output chapter for the target toolchain.

Incremental diagnostics tests#

Use at least two revisions and a clean comparison:

rev 1: trigger lint, fulfill expectation, populate cache
rev 2: remove trigger, retain expectation
  incremental output ---- compare ---- clean rev 2 output

reverse transition as a second test

Also vary policy without semantic source change:

• warn to deny;
• changed cap;
• changed group selection;
• expectation added or removed;
• edition changed under isolated cache compatibility;
• path remapping changed.

Test review checklist#

• Does the fixture cross the suspected boundary?
• Is the primary span asserted, not merely the title?
• Are independent diagnostics retained after recovery?
• Are cascades absent for the intended reason?
• Are terminal and JSON tests separated?
• Are paths normalized narrowly?
• Does a clean build serve as oracle for incremental behavior?
• Is a two-crate fixture used for metadata stability/provenance?
• Does applying the suggestion produce expected source?
• Does the regression fail before the fix?

69. Source map and contributor workshops#

These workshops progress from observation to a patch-sized contribution. Use the exact rustc 1.97.1 source revision and record any divergence from current nightly documentation.

Workshop A: map one byte#

Create a file containing ASCII, β, a tab, and a combining character. Record UTF-8 bytes with a hex viewer. Use compiler output to compare byte offsets, scalar columns, and displayed columns. Find the source-map routine responsible for each conversion.

Deliverable:

input bytes
line-start table
queried BytePos
containing SourceFile
relative byte
reported line/column
terminal display width decision

Do not patch until you can explain every difference.

Workshop B: trace a span#

Select one type error. Find where its primary Span is obtained. Trace it backward through HIR lowering and expansion. Record every use of call-site, source-callsite, or hygiene transformation. Then repeat through a macro argument and a macro definition token.

Contribution candidate: a focused regression for a wrong macro label, not a broad rewrite of span selection.

Workshop C: expansion backtrace#

Nest three macros, with one from another crate. Capture terminal and JSON output. Identify which frame is immediate, which is source-level, and what source is unavailable downstream. Vary the macro backtrace option only after confirming it exists in rustc -Z help for the pinned toolchain.

Workshop D: diagnostic lifecycle#

Choose a structured type-checking diagnostic. Find:

1. semantic predicate;
2. diagnostic structure or builder creation;
3. Fluent message definition;
4. arguments and labels;
5. suggestion construction;
6. emission call;
7. error guarantee return/use;
8. UI snapshot;
9. JSON or rustfix test;
10. query boundary around detection.

Draw ownership from construction to emission. Look for paths that can abandon or emit twice.

Workshop E: recovery and delayed bug#

Find one delayed bug in pinned source. Identify the earlier error expected to taint compilation. Construct malformed input reaching that route. Temporarily reason about removing the earlier emission: should the delayed bug surface? If not, the delayed assertion may encode a different invariant than assumed.

Never submit a change that merely suppresses the delayed bug. Submit the smallest reproducer and an explanation of the guarantee flow first.

Workshop F: lint registration#

Choose one simple built-in lint. Trace declaration, default level, group membership, pass registration, detector, level lookup, expectation fulfillment, diagnostic structure, and tests. Run cases under allow/warn/deny/forbid and a cap. Record macro behavior.

Then design a counterexample where the lint would be wrong. If the implementation handles it, explain how. If not, minimize before proposing a patch.

Workshop G: tool lint namespace#

Compile the same attribute with and without the corresponding tool driver. Observe unknown-tool and unknown-lint behavior under current documented policy. Test command-line override spelling and scoped attributes. Do not assume rustc runs the tool's lint pass when invoked directly.

Workshop H: stability use-site#

Choose an unstable library API available in the pinned source. Trace its stability attribute through metadata encoding and downstream use-site checking. Use two crates so decoding is exercised. Compare direct use, reexport, trait-associated use, and macro-generated use.

Record feature identity, span selection, and diagnostic source. Do not generalize from one item category.

Workshop I: const stability#

Find an API whose ordinary stability and const stability differ at the chosen revision. Call it at runtime and in a const. Trace why only one use is rejected. Locate the const-context check and stability metadata. Verify the message does not falsely call runtime use unstable.

Workshop J: internal allowance containment#

Find a stable standard macro that relies on a named internal unstable feature, if one exists in the pinned revision. Confirm macro invocation works in a stable crate. Confirm direct use of the internal feature fails. Trace the expansion provenance that confines the allowance.

This is read-only unless you have a dedicated regression. The mechanism is security- and soundness-adjacent policy infrastructure.

Workshop K: edition migration#

Pick one documented edition migration lint. Build old-definition/new-invocation and new-definition/old-invocation macro cases. Apply suggestions in a temporary tree. Compile before and after under both relevant editions. Classify every non-machine-applicable case.

Workshop L: path privacy#

Build equivalent crates in /tmp/private-one and /tmp/private-two. Use supported path remapping. Capture terminal, JSON, metadata-related observations, and dep-info as relevant. Search outputs for both physical roots. Explain each remaining occurrence before filing a bug.

70. Failure maps and production debugging#

Start with the observed boundary, then design one experiment that distinguishes causes.

Debugging protocol#

1. Pin rustc -vV and source commit.
2. Save complete source, dependencies, flags, target, and environment facts.
3. Classify direct, macro, cross-crate, incremental, edition, and output-mode boundaries.
4. Capture terminal and JSON separately.
5. Reduce while preserving the classified boundary.
6. Identify the first wrong structured field, not the first ugly rendered line.
7. Trace provenance, policy, guarantee, and query ownership independently.
8. Compare with a clean build and a neighboring valid program.
9. Add the narrowest failing test before changing implementation.
10. Re-run adjacent suites and inspect normalized diffs manually.

Counterfactual diagnosis#

Ask what moving work would cost.

If a lint moves earlier, it regains tokens but loses types. If it moves later, it gains semantics but may lose surface syntax. If rendering moves into a query, caching captures locale and width. If emission moves out, callers need complete diagnostic facts. If every macro span becomes source-callsite, definition bugs become harder to fix. If every generated suggestion is suppressed, useful macro-argument repairs disappear. If every error taints globally, independent diagnostics vanish. If recovery never taints, later phases operate on fiction.

There is no cost-free layer. The design question is which layer has enough information to establish and maintain the invariant.

Production review questions#

• What user-observable policy changes?
• Is the named API stable or compiler-private?
• Which source provenance is preserved and which is discarded?
• Can the primary span be unavailable or generated?
• Does localization receive semantic arguments?
• Is the suggestion complete under macro and Unicode cases?
• Does effective level account for scope, groups, caps, and expectations?
• What happens on a green query?
• What happens after an earlier error?
• Can output order depend on scheduling?
• Can a path or snippet disclose private data?
• Which clean/incremental and terminal/JSON tests fail before the fix?

71. Derived philosophy, mastery exercises, and reading map#

Diagnostics are compiler correctness products. A wrong message is frustrating; a wrong span can make an automated edit destructive; a lost error guarantee can allow invalid state into code generation; a stale lint level can change whether a build succeeds; a leaked stability allowance can accept unsupported language.

Representations determine which repairs are possible#

A byte range can underline text but cannot explain macro origin. A source-callsite can identify editable source but may forget where generated syntax was authored. A formatted string can be printed but cannot reliably drive an IDE edit. A structured suggestion preserves edits and applicability but costs validation and schema maintenance.

Preserve information until the layer that owns the decision. Do not preserve everything forever: large provenance and snippets increase memory, metadata, hashing, and privacy costs.

Policy is executable compatibility#

Lint defaults, feature gates, stability metadata, const stability, and editions are not labels pasted onto analysis. They encode staged ecosystem transitions. Changing a group, cap interaction, or migration applicability can break real builds and tools.

The mechanism detects a fact. Policy assigns meaning at a release and scope. Presentation explains the result. Keeping these separable allows policy to evolve without duplicating semantic checks.

Recovery spends an uncertainty budget#

After an error, some facts remain trustworthy and others do not. An error sentinel represents uncertainty instead of guessing it away. Every recovery step should state which invariants remain available. The budget ends before layout, code generation, or metadata publication requires fully valid meaning.

Caching creates replay obligations#

A cached value is not enough when executing the query would have emitted diagnostics, fulfilled expectations, updated error state, or registered delayed bugs. Either those observations become cache products or ownership moves outside the query. Optimization cannot erase policy effects.

Provenance is a security and usability boundary#

Provenance answers who owns editable bytes. It also controls which physical paths and snippets are disclosed. Source remapping, macro hygiene, metadata spans, and JSON serialization converge here. Once provenance is discarded, neither honesty nor privacy can be reconstructed reliably downstream.

Mastery exercises#

1. Implement tab-aware display columns in the model with an explicit tab width.
2. Add multiline labels without quadratic rendering.
3. Add nested expansion backtrace truncation with a visible omitted-frame count.
4. Add multipart edit application in reverse byte order per file.
5. Property-test that valid edits never split UTF-8.
6. Generate diagnostic insertion permutations and prove deterministic output.
7. Replace dedup identity with text equality and produce a hidden-error counterexample.
8. Add scoped lint levels and model forbid as a non-lowerable floor.
9. Add group overrides with explicit precedence rather than “highest severity wins.”
10. Persist expectation identities across an edit and compare with a clean run.
11. Model a language feature and separate runtime API stability from const stability.
12. Add edition-tagged spans and a cross-edition macro experiment.
13. Add a locale catalog while keeping semantic diagnostic identity unchanged.
14. Add source redaction and test terminal plus JSON leakage.
15. Fuzz malformed ranges, cyclic provenance, and cyclic lint groups.
16. Bound every collection and define truncation diagnostics.
17. Add cancellation between validation and emission and prove no partial replay record commits.
18. Write a two-crate test plan for stability metadata round trips.
19. Review a real rustc diagnostic PR and identify mechanism, policy, provenance, and presentation changes.
20. Teach one failure trace without using internal type names.

Capstone#

Extend the stable model into a two-revision engine. Revision one detects two lints, fulfills one expectation, and emits one denied lint. Revision two changes only source offsets and lint policy. Cache semantic findings by stable owner, not raw byte position. Reproject current spans, reevaluate levels, replay expectation fulfillment, and compare terminal plus JSON output with a clean revision-two run.

Required injected failures:

• stale raw span reused;
• old lint level reused;
• expectation fulfillment omitted;
• diagnostic emitted both eagerly and on replay;
• randomized insertion order;
• physical path leaked despite remapping;
• generated edit marked machine-applicable;
• error guarantee omitted on cached denial.

For each, report the earliest broken invariant and the later visible symptom.

Talk designs#

Talk one: “A span is not a line number.”

Begin with UTF-8 bytes. Add source-file intervals, line tables, display columns, hygiene context, call/definition/mixed provenance, nested expansions, metadata, and path remapping. End with one macro suggestion whose correctness depends on every layer.

Talk two: “An error is evidence.”

Start with immediate printing. Derive structured diagnostics, a diagnostic context, error guarantees, taint, delayed bugs, cascade suppression, query replay, and hard phase cutoffs. Use one ICE trace to find the earliest discarded guarantee.

Talk three: “Warnings are language policy.”

Start with one detector. Add scoped levels, forbid, groups, caps, expectations, tool namespaces, future compatibility, editions, feature gates, library stability, and const stability. End with the compatibility review required to change a default.

Talk four: “Human output and machine output are different products.”

Construct one diagnostic. Render it through Fluent and terminal layout, then serialize JSON. Demonstrate how terminal snapshots can pass while applicability or primary-span fields are wrong. Finish with deterministic and privacy-aware testing.

Prediction exercises#

• A byte offset is stable but a tab width changes. Which layer changes? Presentation columns, not semantic span bytes.
• A token came from a macro argument. Is definition site the best edit location? Usually not; inspect provenance and ownership.
• A denied lint is capped at warn. Does the detector stop finding it? No; policy changes effective emission.
• An expected lint is found. Is that equivalent to allow? No; fulfillment is an additional observation.
• A normal API is stable but const use is unstable. Can runtime calls compile? Yes, subject to ordinary checks.
• A stable macro uses an allowed internal feature. May users invoke the feature directly? No.
• Incremental terminal text matches clean but JSON applicability differs. Is compilation observationally equivalent for tools? No.
• Two diagnostics have equal English text and span. May one be dropped? Not without stronger root identity.
• A delayed bug surfaces after a refactor. Should it be deleted? No; find the missing prior error or newly broken invariant.
• Remapped paths hide the workspace root. Are diagnostics now secret-safe? No; snippets and names can still disclose data.

Authoritative reading map#

Use the pinned 1.97.1 source for implementation claims:

• https://github.com/rust-lang/rust/tree/1.97.1/compiler/rustc_span
• https://github.com/rust-lang/rust/tree/1.97.1/compiler/rustc_errors
• https://github.com/rust-lang/rust/tree/1.97.1/compiler/rustc_lint
• https://github.com/rust-lang/rust/tree/1.97.1/compiler/rustc_feature
• https://github.com/rust-lang/rust/tree/1.97.1/compiler/rustc_passes
• https://github.com/rust-lang/rust/tree/1.97.1/tests/ui

Read the rustc-dev-guide for architecture and contribution workflows:

• https://rustc-dev-guide.rust-lang.org/diagnostics.html
• https://rustc-dev-guide.rust-lang.org/diagnostics/diagnostic-structs.html
• https://rustc-dev-guide.rust-lang.org/diagnostics/macros.html
• https://rustc-dev-guide.rust-lang.org/diagnostics/translation.html
• https://rustc-dev-guide.rust-lang.org/diagnostics/implementing_new_lints.html
• https://rustc-dev-guide.rust-lang.org/stability.html
• https://rustc-dev-guide.rust-lang.org/tests/intro.html
• https://rustc-dev-guide.rust-lang.org/tests/ui.html
• https://rustc-dev-guide.rust-lang.org/queries/incremental-compilation.html

Use user-facing normative or supported documentation for contracts:

• https://doc.rust-lang.org/reference/attributes/diagnostics.html
• https://doc.rust-lang.org/reference/conditional-compilation.html
• https://doc.rust-lang.org/reference/names/preludes.html#tool-prelude
• https://doc.rust-lang.org/rustc/lints/levels.html
• https://doc.rust-lang.org/rustc/lints/groups.html
• https://doc.rust-lang.org/rustc/json.html
• https://doc.rust-lang.org/edition-guide/editions/index.html
• https://doc.rust-lang.org/edition-guide/rust-2024/index.html

Use current nightly API docs only as navigation and verify them against the pinned revision:

• https://doc.rust-lang.org/nightly/nightly-rustc/rustc_span/
• https://doc.rust-lang.org/nightly/nightly-rustc/rustc_errors/
• https://doc.rust-lang.org/nightly/nightly-rustc/rustc_lint/
• https://doc.rust-lang.org/nightly/nightly-rustc/rustc_feature/

The practical standard is not “the compiler printed something helpful.” It is that every report has an owned semantic cause, preserved and privacy-aware provenance, honest policy and applicability, deterministic machine-readable structure, and recovery evidence that survives queries and incremental reuse.

Part VIII-D: Metadata and Compiler Runtime Infrastructure#

72. The hidden systems that make a compiler invocation coherent#

This chapter targets rustc 1.97.1. Names and layouts below are compiler-private and must be checked against that release's source. The durable subject is the contracts between loading, identity, metadata, allocation, interning, queries, and configuration.

A compiler cannot begin with “parse the file” and end with “write an object” in isolation. It must decide which bytes are the file, which crate a dependency denotes, and whether an old answer belongs to this invocation. It must keep millions of small semantic values alive without giving each one an allocator header. It must let several workers demand the same fact without publishing half an answer.

The smallest useful model is one ownership spine:

Compiler invocation
  | owns configuration and orchestration
  v
Session ---- target/options/diagnostics/source map
  | creates global semantic context
  v
TyCtxt ---- arenas + interners + providers + query runtime
  | asks for external facts
  v
CrateStore ---- validated metadata blobs + translation tables
  | obtains bytes from
  v
FileLoader / crate locator / filesystem or virtual host

The arrows carry capabilities, not merely data. FileLoader grants access to named bytes. The crate store grants access to validated external semantics. TyCtxt grants access to interned semantic values and queries during its lifetime. Session grants access to invocation policy and diagnostics.

Five invariants organize the chapter:

1. Every byte has an explicit origin and displayed identity.
2. Every cross-session reference is translated before use.
3. Every shared value has one publication and lifetime protocol.
4. Every query waiter observes success, failure, or cancellation.
5. Reuse under current inputs is observationally equal to a clean build.

These systems are easy to overlook because success makes them quiet. Their failures surface late: a linker collision, a decoder panic, a stale incremental answer, or a deadlocked worker. Debug toward the earliest broken invariant.

73. Loading source: names are not bytes#

A beginner sees rustc main.rs and imagines one read_to_string call. Production compilation also sees module files, macro-generated virtual files, standard input, IDE overlays, imported source descriptions, and remapped paths. Loading therefore separates a requested name from byte acquisition and from the name shown to users.

Conceptually a FileLoader answers operations such as “does this path exist?” and “read this file.” The exact trait methods in rustc_span::source_map are version-sensitive. The abstraction permits rustc's normal filesystem loader and hosts that supply virtual contents.

requested module path
  -> FileLoader existence/read policy
  -> bytes + physical origin
  -> UTF-8/source validation
  -> SourceFile with line starts
  -> displayed/remapped name
  -> SourceMap global byte interval

Do not merge these identities:

Path remapping changes selected observations; it does not redirect ordinary file I/O by itself. Applying remapping before loading would try to open a synthetic path. Applying it only at terminal rendering can leak physical paths into metadata, dep-info, hashes, or JSON. The owner of each output must use the intended path form.

A virtual loader must define snapshot semantics. If an IDE overlay changes halfway through compilation, two reads of one path cannot silently return different texts. Freeze a revision, version each read, or model changes as cancellation and restart. “The editor usually does not change that fast” is not an invariant.

Source loading should bound file size before allocating, report I/O errors with the requested context, and reject invalid assumptions without panicking. Rust source is UTF-8, but loaders may also encounter unreadable bytes, races, symlink changes, and permission errors. Canonicalizing every path can alter symlink semantics and reveal private roots; it is policy, not a free normalization.

A concrete virtual-source trace#

An IDE asks to check src/lib.rs while an unsaved buffer differs from disk.

1. Module lookup requests src/lib.rs.
2. The overlay loader finds snapshot revision 41.
3. It returns the in-memory bytes and records the virtual origin.
4. SourceMap assigns an interval and computes line starts.
5. Diagnostics display src/lib.rs, not an overlay cache path.
6. Dependency tracking associates the invocation with revision 41's content.
7. An edit to revision 42 cancels or schedules another invocation.

The clean reference for revision 41 must use exactly those bytes. Comparing against the on-disk file would test a different input.

Loading failure map#

74. External crates: discovery is not identity#

extern crate, an extern-prelude name, or a resolved dependency tells rustc it needs another crate. The loader still has to locate a compatible artifact among search paths and crate types. Filenames are hints, never complete semantic identity.

Discovery considers compiler-provided extern mappings, dependency search paths, target conventions, artifact flavors, metadata headers, and compatibility data. An rlib, rmeta, Rust dylib, and proc-macro artifact serve different roles. Host and target differ when a proc macro runs on the host while ordinary dependencies target another platform.

logical request: crate name + dependency context
  -> candidate paths
  -> artifact/header validation
  -> compiler/target/crate-kind compatibility
  -> stable crate identity and dependency edges
  -> local CrateNum assignment
  -> loaded crate-data handle

CrateNum is a compact identity allocated in one compilation session. It is not stable across invocations and must not be serialized as permanent identity. A crate graph edge records which loaded crate depends on which other loaded crate, including identity translation needed by metadata.

Two crates can share the source-level name util. Package resolution, disambiguation, compilation inputs, and dependency context keep them distinct. Conversely, two paths to the same compatible artifact should not accidentally load two semantic copies when the graph expects one.

StableCrateId supplies a stable, hash-based crate identity for compiler mechanisms that cross local numbering boundaries. Its exact construction and collision handling are implementation details. It is not a package-manager UUID and not a forever-stable public identifier. Its domain includes enough crate/disambiguation context to prevent ordinary same-name collisions.

Graph identity must be validated before decoding references. If metadata says its crate slot 2 means stable crate X, the consumer maps X to its own CrateNum; it must not assume slot 2 locally means X.

Discovery alternatives#

Rustc and Cargo divide responsibility. Cargo builds the package/unit graph and supplies rustc invocations. Rustc validates and builds its semantic crate graph for that invocation. Neither should trust an artifact merely because its filename came from the other.

75. Metadata as a compiler-private database#

Crate metadata is a serialized semantic interface from one rustc process to another compatible process. It is not Rust's language ABI, not a stable schema for third-party tools, and not guaranteed readable by another compiler release. For 1.97.1, inspect compiler/rustc_metadata and its rmeta encoder/decoder modules.

The encoder starts after local analyses have produced facts downstream may demand. It writes a header and root structure, tables indexed by definition-local identities, shared or lazy payloads, source/provenance data, crate dependencies, and compatibility information. Exact fields are demand- and revision-sensitive.

local TyCtxt facts
  -> encode crate/definition references in metadata domain
  -> deduplicate/shared encode where schema permits
  -> tables of lazy positions and optional values
  -> compressed/framed metadata bytes
  -> artifact container (.rmeta or library)

consumer
  -> validate framing/version
  -> locate root
  -> map crate identities
  -> index requested DefIndex
  -> seek lazy payload
  -> decode and intern into consumer TyCtxt

Schema decisions#

A schema answers four questions for every fact:

1. Is it available cross-crate?
2. What key locates it?
3. Is it inline, tabled, shared, or lazy?
4. Which identities need translation while encoding and decoding?

Omitting private implementation detail reduces size and coupling. But privacy alone cannot decide: downstream monomorphization, const evaluation, inlining, trait solving, or diagnostics may need selected non-public bodies or provenance. Encoding everything makes local refactors format changes and increases I/O.

Tables make per-definition access cheap. A dense table gives approximately constant-time indexing but pays for absent entries. A sparse map saves empty slots but costs keys and lookup. Specialized table encodings can represent optional or fixed-width values compactly. The correct choice depends on cardinality and access frequency, not elegance.

Lazy positions and indexes#

A lazy reference is conceptually an offset into the metadata blob plus knowledge of the encoded type. It avoids decoding unrelated material. Offsets must be validated against blob bounds and section framing before arithmetic or allocation. An index points to a lazy record; it does not prove the record is valid.

Relative positions can compress well because nearby records have small deltas. Variable-length integer coding saves bytes for small values but requires bounded decoding and overflow checks. Shared tables avoid repeatedly encoding common structures but introduce indirection and cycle/order constraints.

The decoder must never execute this policy:

read attacker-controlled length N
allocate N elements
then discover only three bytes remain

It should validate maximums, remaining bytes, multiplication overflow, recursion depth, and element-specific constraints before committing resources.

Compression#

Compression trades producer CPU and consumer CPU for artifact and I/O reduction. It may apply to the metadata payload or selected forms according to current implementation. Do not claim a particular codec or threshold without checking 1.97.1 source.

The cost model is:

T_read = bytes_on_disk / storage_bandwidth
T_use  = T_read + T_decompress + T_index + sum(T_decode demanded records)
M_peak = compressed buffer + decompression state + decoded live values

Whole-blob compression improves ratio but can defeat random lazy access or require a decompressed resident blob. Block compression preserves bounded random access but adds an index and loses some ratio. No compression favors CPU and simplicity but increases transfer and cache pressure.

Measure check builds, full builds, cold and warm storage, tiny and huge crates, and downstream demand patterns. Metadata bytes alone are not the user cost.

Versioning and compatibility#

The format has a compatibility boundary tied to compiler behavior. A header or version marker rejects obvious mismatch before deep decoding. Crate hash/disambiguation and dependency identities prevent using semantically wrong artifacts that happen to parse.

A schema change must update producer and consumer together. Backward compatibility is usually less valuable than a clear rejection because Rust-to-Rust artifacts are rebuilt with compatible toolchains. Incremental and build caches must treat incompatibility as a miss, not reinterpret bytes optimistically.

Version checks are necessary but insufficient. A correctly versioned file can be truncated, corrupted, maliciously modified, or produced by a buggy encoder. Every unsafe or unchecked decode optimization carries a proof obligation established by prior validation.

76. A bounded stable-Rust metadata round trip#

The following complete program models a header, version, crate identity, string pool, fixed-width item index, lazy item records, and checked decoding. It uses stable Rust and the standard library. It deliberately has no compression so offsets remain inspectable.

Save as metadata_lab.rs, then run rustc --edition=2021 --test metadata_lab.rs && ./metadata_lab.

use std::collections::BTreeMap;

const MAGIC: &[u8; 4] = b"RMD0";
const VERSION: u16 = 1;
const MAX_ITEMS: usize = 1_000;
const MAX_STRING: usize = 16 * 1024;

#[derive(Clone, Debug, Eq, PartialEq)]
struct Item {
	id: u32,
	name: String,
	public: bool,
	ty: String,
}

#[derive(Clone, Debug, Eq, PartialEq)]
struct CrateMeta {
	stable_crate_id: u64,
	items: Vec<Item>,
}

#[derive(Debug, Eq, PartialEq)]
enum Error {
	Truncated,
	BadMagic,
	WrongVersion(u16),
	Limit(&'static str),
	Overflow,
	BadOffset,
	BadUtf8,
	DuplicateItem(u32),
	TrailingRecord,
}

fn put_u16(out: &mut Vec<u8>, value: u16) { out.extend_from_slice(&value.to_le_bytes()); }
fn put_u32(out: &mut Vec<u8>, value: u32) { out.extend_from_slice(&value.to_le_bytes()); }
fn put_u64(out: &mut Vec<u8>, value: u64) { out.extend_from_slice(&value.to_le_bytes()); }

fn put_str(out: &mut Vec<u8>, value: &str) -> Result<(), Error> {
	if value.len() > MAX_STRING { return Err(Error::Limit("string")); }
	let len = u32::try_from(value.len()).map_err(|_| Error::Overflow)?;
	put_u32(out, len);
	out.extend_from_slice(value.as_bytes());
	Ok(())
}

fn encode(meta: &CrateMeta) -> Result<Vec<u8>, Error> {
	if meta.items.len() > MAX_ITEMS { return Err(Error::Limit("items")); }
	let mut records = Vec::new();
	let mut index = Vec::with_capacity(meta.items.len());
	for item in &meta.items {
		let offset = u32::try_from(records.len()).map_err(|_| Error::Overflow)?;
		index.push((item.id, offset));
		put_u32(&mut records, item.id);
		records.push(u8::from(item.public));
		put_str(&mut records, &item.name)?;
		put_str(&mut records, &item.ty)?;
	}
	let count = u32::try_from(index.len()).map_err(|_| Error::Overflow)?;
	let header_len = 4usize + 2 + 8 + 4 + 4;
	let index_len = index.len().checked_mul(8).ok_or(Error::Overflow)?;
	let records_start = header_len.checked_add(index_len).ok_or(Error::Overflow)?;
	let records_start = u32::try_from(records_start).map_err(|_| Error::Overflow)?;

	let mut out = Vec::new();
	out.extend_from_slice(MAGIC);
	put_u16(&mut out, VERSION);
	put_u64(&mut out, meta.stable_crate_id);
	put_u32(&mut out, count);
	put_u32(&mut out, records_start);
	for (id, relative) in index {
		put_u32(&mut out, id);
		put_u32(&mut out, relative);
	}
	out.extend_from_slice(&records);
	Ok(out)
}

struct Reader<'a> { bytes: &'a [u8], at: usize }

impl<'a> Reader<'a> {
	fn new(bytes: &'a [u8], at: usize) -> Result<Self, Error> {
		if at > bytes.len() { return Err(Error::BadOffset); }
		Ok(Self { bytes, at })
	}
	fn take(&mut self, n: usize) -> Result<&'a [u8], Error> {
		let end = self.at.checked_add(n).ok_or(Error::Overflow)?;
		let value = self.bytes.get(self.at..end).ok_or(Error::Truncated)?;
		self.at = end;
		Ok(value)
	}
	fn u8(&mut self) -> Result<u8, Error> { Ok(self.take(1)?[0]) }
	fn u16(&mut self) -> Result<u16, Error> {
		Ok(u16::from_le_bytes(self.take(2)?.try_into().unwrap()))
	}
	fn u32(&mut self) -> Result<u32, Error> {
		Ok(u32::from_le_bytes(self.take(4)?.try_into().unwrap()))
	}
	fn u64(&mut self) -> Result<u64, Error> {
		Ok(u64::from_le_bytes(self.take(8)?.try_into().unwrap()))
	}
	fn string(&mut self) -> Result<String, Error> {
		let len = usize::try_from(self.u32()?).map_err(|_| Error::Overflow)?;
		if len > MAX_STRING { return Err(Error::Limit("string")); }
		let text = std::str::from_utf8(self.take(len)?).map_err(|_| Error::BadUtf8)?;
		Ok(text.to_owned())
	}
}

fn decode(bytes: &[u8]) -> Result<CrateMeta, Error> {
	let mut r = Reader::new(bytes, 0)?;
	if r.take(4)? != MAGIC { return Err(Error::BadMagic); }
	let version = r.u16()?;
	if version != VERSION { return Err(Error::WrongVersion(version)); }
	let stable_crate_id = r.u64()?;
	let count = usize::try_from(r.u32()?).map_err(|_| Error::Overflow)?;
	if count > MAX_ITEMS { return Err(Error::Limit("items")); }
	let records_start = usize::try_from(r.u32()?).map_err(|_| Error::Overflow)?;
	let expected_start = r.at.checked_add(count.checked_mul(8).ok_or(Error::Overflow)?)
		.ok_or(Error::Overflow)?;
	if records_start != expected_start || records_start > bytes.len() {
		return Err(Error::BadOffset);
	}
	let mut index = Vec::with_capacity(count);
	for _ in 0..count { index.push((r.u32()?, r.u32()?)); }

	let mut items = BTreeMap::new();
	for position in 0..index.len() {
		let (expected_id, relative) = index[position];
		let start = records_start.checked_add(relative as usize).ok_or(Error::Overflow)?;
		let end = if let Some((_, next)) = index.get(position + 1) {
			records_start.checked_add(*next as usize).ok_or(Error::Overflow)?
		} else { bytes.len() };
		if start > end || end > bytes.len() { return Err(Error::BadOffset); }
		let mut item_reader = Reader::new(&bytes[..end], start)?;
		let id = item_reader.u32()?;
		if id != expected_id { return Err(Error::BadOffset); }
		let public = match item_reader.u8()? { 0 => false, 1 => true, _ => return Err(Error::BadOffset) };
		let name = item_reader.string()?;
		let ty = item_reader.string()?;
		if item_reader.at != end { return Err(Error::TrailingRecord); }
		let item = Item { id, name, public, ty };
		if items.insert(id, item).is_some() { return Err(Error::DuplicateItem(id)); }
	}
	Ok(CrateMeta { stable_crate_id, items: items.into_values().collect() })
}

fn sample() -> CrateMeta {
	CrateMeta {
		stable_crate_id: 0x1234,
		items: vec![
			Item { id: 2, name: "answer".into(), public: true, ty: "fn() -> u32".into() },
			Item { id: 9, name: "Hidden".into(), public: false, ty: "struct".into() },
		],
	}
}

fn main() {
	let original = sample();
	let bytes = encode(&original).unwrap();
	assert_eq!(decode(&bytes).unwrap(), original);
}

#[cfg(test)]
mod tests {
	use super::*;

	#[test]
	fn round_trip() {
		let value = sample();
		assert_eq!(decode(&encode(&value).unwrap()).unwrap(), value);
	}

	#[test]
	fn rejects_wrong_version_before_records() {
		let mut bytes = encode(&sample()).unwrap();
		bytes[4..6].copy_from_slice(&99_u16.to_le_bytes());
		assert_eq!(decode(&bytes), Err(Error::WrongVersion(99)));
	}

	#[test]
	fn every_truncation_is_an_error() {
		let bytes = encode(&sample()).unwrap();
		for end in 0..bytes.len() { assert!(decode(&bytes[..end]).is_err()); }
	}

	#[test]
	fn rejects_forged_large_count_without_allocating_it() {
		let mut bytes = encode(&sample()).unwrap();
		bytes[14..18].copy_from_slice(&u32::MAX.to_le_bytes());
		assert_eq!(decode(&bytes), Err(Error::Limit("items")));
	}

	#[test]
	fn output_is_deterministic() {
		assert_eq!(encode(&sample()).unwrap(), encode(&sample()).unwrap());
	}
}

The model validates framing before following offsets. It bounds counts and strings before allocation. Each index entry redundantly carries the item ID so a misdirected offset is detected. Tests exercise round-trip symmetry, deterministic bytes, truncation, version rejection, and an allocation attack.

The model omits real rustc's schema, specialized tables, lazy type machinery, crate-reference translation, spans, shared encodings, compression, hashing, artifact containers, and zero-copy optimizations. Its monotonically ordered record assumption is stricter than a general lazy store. It clones strings rather than interning them. Those omissions keep the proof inspectable; they are not claims about rustc.

Extensions should first add a malformed-input property: decoding any byte string below a chosen size must terminate within a resource budget and never panic. Then add a string pool, rejecting out-of-range indexes. Only then add compression with an explicit decompressed-byte limit.

77. CrateStore and external query providers#

Downstream compiler code should not contain a branch at every use saying “if local, analyze HIR; if external, decode bytes.” Query provider dispatch hides that acquisition difference while preserving semantic type.

The crate store owns or reaches loaded external crate data. In current rustc this area is commonly called cstore and lives around rustc_metadata interfaces. It supports crate graph questions, metadata lookup, source information, exported symbols, and provider-backed semantic facts. Exact trait boundaries and provider tables must be verified in 1.97.1.

tcx.type_of(DefId)
  -> inspect DefId.krate
     local    -> local provider computes from local representations
     external -> external provider asks CrateStore/metadata decoder
                    -> table lookup
                    -> decode references
                    -> intern type in current TyCtxt
  -> same semantic result type

Same result type does not imply same dependencies or failure modes. The local provider may demand HIR and type checking. The external provider depends on validated metadata and crate mapping. A regression test for external behavior needs at least two crates.

Provider completeness is a schema obligation. If downstream can ask predicates_of(external_def), metadata must encode enough to answer it or the query protocol must forbid that key. Returning an empty predicate list because no field was found converts corruption or encoder omission into miscompilation.

Lazy decoding should cache at an appropriate level. Caching raw bytes avoids I/O but repeats parsing. Caching fully decoded values saves CPU but retains memory. Interning into TyCtxt canonicalizes values and ties them to this invocation's lifetime. The demand distribution determines the right boundary.

78. Translating definition identity#

DefId is a pair in spirit: a crate-local session identity and an index within that crate's definition space. It is compact and excellent inside one invocation. It is invalid as a raw cross-session or permanent identity.

DefPathHash identifies a definition through a stable hash of its crate-relative definition path under rustc's rules. StableCrateId identifies the crate domain. Together they support stable hashing, incremental reconstruction, and metadata mapping where current contracts require it. They are compiler mechanisms, not stable public IDs.

producer DefId = (producer CrateNum 0, DefIndex 17)
  -> producer StableCrateId + definition reference/path identity
  -> metadata crate-number table + encoded DefIndex/DefPathHash
  -> consumer maps StableCrateId to consumer CrateNum 4
  -> consumer DefId = (4, translated index)

Never translate by crate name alone. Never preserve a producer CrateNum numerically. Never infer definition identity from source line, address, iteration position, or symbol text.

Definition paths need disambiguators because two same-named definitions can occupy one namespace context. Anonymous and generated definitions need deterministic path components. Changing sibling insertion should not unnecessarily rename an unrelated definition, but exact stability is constrained by the current DefPath construction.

Hash collision is a correctness concern even when astronomically unlikely. Code must not accidentally truncate a stable hash into an unchecked table key. Where uniqueness is assumed, document the domain and collision strategy.

Translation trace#

Crate app decodes fn api::make() -> api::Token.

1. Metadata identifies api through its dependency table.
2. The decoder maps that entry's stable crate identity to app's local CrateNum for api.
3. It translates the function definition reference.
4. It decodes the return type's ADT definition reference through the same mapping.
5. It interns the resulting type in app's TyCtxt.
6. Queries return consumer-session DefIds only.

If step 2 chooses another same-name crate, type text may look plausible while trait identity and layout are wrong. That is why names are presentation, not authority.

79. Symbols, mangling, and linkage boundaries#

The word symbol is overloaded. An interned Symbol in rustc often denotes text such as an identifier. A linker symbol denotes a named object or function in an object-file namespace. A Rust definition and a concrete monomorphized instance are different identities again.

Rust symbol mangling encodes enough path, instance, type/const, and disambiguation information to distinguish required object identities under the selected mangling scheme. Exact legacy and v0 forms are documented/versioned separately; inspect rustc's symbol-mangling modules for 1.97.1. Demangling is presentation and does not reverse every compiler identity decision.

Linkage controls visibility, coalescing, duplication, and resolution across codegen units and objects. pub language visibility is not object external linkage. Inlining, internalization, COMDAT-like deduplication, LTO, #[no_mangle]-style attributes, export names, and crate type all affect the boundary.

An explicit export name transfers collision responsibility toward the programmer and platform namespace. The compiler must diagnose applicable duplicate definitions or allow the linker to report them according to contract; it must not silently bind two unrelated semantics. Foreign ABI names obey external conventions rather than Rust's full identity encoding.

Counterexample: using the demangled path as an incremental key. Two monomorphizations can display similarly while differing in hidden disambiguation or arguments. Presentation changes could invalidate caches despite unchanged semantics. Use semantic identity for semantic caches and mangled identity only at the linkage boundary.

80. Arenas: bulk ownership, not immortal memory#

Compilers allocate many small nodes and keep most until a phase or invocation ends. Calling the general allocator and freeing each node separately pays bookkeeping, fragmentation, and traversal costs. An arena allocates chunks, places objects sequentially, and releases them as a group.

arena owns chunks
chunk A: [Ty][Ty][List header][padding][Const]...
chunk B: [Ty][Predicate][Predicate]...

allocation -> bump cursor
individual free -> unavailable
arena drop -> run required destructors / release chunks by arena policy

Arena advantages are cheap allocation, locality, and stable addresses. Costs are coarse reclamation, retained peak memory, alignment waste, and lifetime coupling. An arena is unsuitable for large temporary objects whose lifetimes are much shorter than the arena unless separated into a short-lived arena.

Strategies include:

“Dropless” is a proof obligation. It is valid only when skipping individual destructors cannot leak required non-arena resources or violate semantics. Memory owned entirely by the arena can be reclaimed wholesale, but a value owning a file descriptor or independent heap buffer needs proper destruction or another ownership design.

Rustc's arena declarations and generated allocation plumbing are version-sensitive. 'tcx commonly denotes data valid for the lifetime of a particular type context, not process-global eternity. References with 'tcx must not outlive the invocation or be smuggled into static storage.

The Compiler orchestration scopes the invocation. Within it, global compiler context construction makes arenas and interners available while a closure executes with TyCtxt. This closure/lifetime shape prevents safe code from returning context-borrowed data after teardown.

Arena cost model#

T_arena = chunk_allocations * allocator_cost + objects * bump_cost
M_live  = sum(chunk capacities retained until arena drop)
waste   = unused chunk tails + alignment padding + dead-but-retained objects

Compare against individual allocation including allocator metadata, pointer chasing, destructor traversal, and cache misses. Measure peak RSS as well as allocation count. An arena optimization that makes a long-lived context retain a one-gigabyte temporary is a regression.

81. Interning and canonicalization#

Interning maps structurally equal immutable values to one canonical representative in a domain. Instead of copying Vec<Ty> into every function type, rustc can store one interned list and compare compact references where the interner contract permits.

candidate TyKind / Const / List
  -> stable structural hash for current in-memory table
  -> equality against bucket candidates
  -> existing canonical reference OR arena-allocate and insert
  -> immutable `'tcx` reference

Interned families include identifier symbols, types, constants, generic arguments, predicates, and immutable lists, with exact APIs varying by revision. Not every rustc value is interned and not every interner has the same lifetime or synchronization strategy.

Interning establishes:

1. Canonical representatives are immutable.
2. Equal candidates in one interner domain yield the same representative under that contract.
3. A representative never outlives its backing storage.
4. Hash and equality include every structural field.
5. Publication is synchronized before other threads observe it.

Pointer equality can be an optimization after canonicalization, not a universal semantic definition. Values from different TyCtxts cannot be compared by address. Serialization must encode structure or stable identity, never an arena pointer.

Cardinality and memory#

Interning helps when repetitions are common and values are expensive. It hurts when almost every candidate is unique: the arena stores all values and the hash table adds buckets, hashes, and synchronization.

without = occurrences * average_value_bytes
with    = unique_values * value_bytes
        + occurrences * reference_bytes
        + hash_table_capacity * bucket_overhead
        + alignment/arena waste

Let U be unique values and N occurrences. The deduplication ratio is N / U. Measure distributions by family; one global ratio hides a pathological constant interner behind highly repetitive symbols.

Lists deserve special attention. Interning every prefix can create quadratic candidate hashing. Hashing a long list for every lookup costs O(length) even when equality later becomes pointer-fast. Empty and common singleton lists may warrant specialized representations, but each specialization increases code and invariant surface.

Canonicalization is not normalization in every semantic sense. Two type aliases may normalize to the same type under one environment but retain distinct provenance or delayed obligations elsewhere. Intern only under the equality relation consumers are allowed to observe.

82. A bounded arena and interner model#

This complete stable-Rust test program uses a safe typed arena backed by Box<T> and a string interner keyed by owned strings. It demonstrates lifetime ownership and canonical IDs, not bump allocation performance.

use std::collections::HashMap;

#[derive(Clone, Copy, Debug, Eq, Hash, PartialEq)]
struct Symbol(u32);

#[derive(Default)]
struct Arena<T> { values: Vec<Box<T>> }

impl<T> Arena<T> {
	fn alloc(&mut self, value: T) -> &T {
		self.values.push(Box::new(value));
		self.values.last().unwrap()
	}
	fn len(&self) -> usize { self.values.len() }
}

#[derive(Default)]
struct Interner {
	map: HashMap<String, Symbol>,
	strings: Vec<String>,
	bytes: usize,
}

impl Interner {
	fn intern(&mut self, text: &str) -> Result<Symbol, &'static str> {
		if text.len() > 1024 { return Err("symbol too long"); }
		if let Some(symbol) = self.map.get(text) { return Ok(*symbol); }
		if self.strings.len() >= 10_000 { return Err("too many symbols"); }
		let id = u32::try_from(self.strings.len()).map_err(|_| "symbol overflow")?;
		let owned = text.to_owned();
		let symbol = Symbol(id);
		self.bytes += owned.len();
		self.strings.push(owned.clone());
		self.map.insert(owned, symbol);
		Ok(symbol)
	}
	fn get(&self, symbol: Symbol) -> Option<&str> {
		self.strings.get(symbol.0 as usize).map(String::as_str)
	}
	fn unique(&self) -> usize { self.strings.len() }
}

fn main() {
	let mut arena = Arena::default();
	let number = arena.alloc(41_u32);
	assert_eq!(*number, 41);

	let mut symbols = Interner::default();
	let a = symbols.intern("metadata").unwrap();
	let b = symbols.intern("metadata").unwrap();
	assert_eq!(a, b);
	assert_eq!(symbols.get(a), Some("metadata"));
}

#[cfg(test)]
mod tests {
	use super::*;

	#[test]
	fn arena_owns_until_drop() {
		let mut arena = Arena::default();
		assert_eq!(*arena.alloc(String::from("owned")), "owned");
		assert_eq!(arena.len(), 1);
	}

	#[test]
	fn repeated_text_has_one_identity() {
		let mut i = Interner::default();
		let first = i.intern("Ty").unwrap();
		for _ in 0..100 { assert_eq!(i.intern("Ty").unwrap(), first); }
		assert_eq!(i.unique(), 1);
		assert_eq!(i.bytes, 2);
	}

	#[test]
	fn distinct_text_does_not_alias() {
		let mut i = Interner::default();
		assert_ne!(i.intern("T").unwrap(), i.intern("U").unwrap());
	}

	#[test]
	fn resource_limit_precedes_copy() {
		let mut i = Interner::default();
		let huge = "x".repeat(1025);
		assert_eq!(i.intern(&huge), Err("symbol too long"));
		assert_eq!(i.bytes, 0);
	}
}

The arena uses one general allocation per value, so it intentionally does not prove bump-allocation speed. It does prove that returned references are borrowed from the arena and cannot safely survive it. The interner's numeric ID is local to that instance and cannot be serialized as stable identity.

The interner duplicates each new string in its vector and hash-map key. A production design can avoid that cost with arena-backed keys or raw-entry techniques, but unsafe self-reference requires a precise address/lifetime proof. The model omits concurrency, list interning, structural types, hash-flood resistance, reclamation, and deterministic ID assignment across schedules.

83. Session, Compiler, and TyCtxt#

These names are related but not interchangeable. Exact fields and construction APIs evolve; reason from responsibility.

Session holds invocation-wide policy and services: parsed options, target information, source mapping, diagnostics, lint/configuration state, and other compilation inputs. It answers “under what invocation are we compiling?”

Compiler in the interface layer orchestrates an invocation and exposes controlled phase operations or callbacks. It owns the path from configured inputs through global-context creation and output handling. It answers “which work is this invocation performing and when?”

TyCtxt<'tcx> is a cheap, copyable handle into the global semantic context for lifetime 'tcx. It provides interners, arenas, query access, language items, crate information, and semantic services. It answers “what semantic facts and canonical values are available in this context?”

Config + callbacks
  -> interface creates Compiler
      -> Compiler owns/uses Session
      -> enter global context closure
          -> TyCtxt<'tcx>
              -> query providers
              -> arenas/interners
              -> crate store
      -> closure ends; `'tcx` values cannot escape safely
  -> finalize diagnostics and outputs

Do not put all state into TyCtxt merely because it is widely available. Operational state such as a temporary output transaction may belong to orchestration. Do not put semantic query results into Session merely because its lifetime is long enough. That bypasses dependency tracking and canonical ownership.

Browser-like compiler embeddings must not retain a previous TyCtxt pointer and reuse it for a new invocation. Stable descriptions must be re-resolved in the new context. The lifetime boundary makes this misuse hard in safe Rust; unsafe extensions must restore the proof manually.

84. Parallel query jobs: one key, one publication#

A memoized query must coordinate simultaneous demand. The unit is a query job for (query kind, key). One worker owns computation; others become waiters or perform unrelated work.

Vacant
  --claim(worker 3)--> Running { owner: 3, waiters: [] }
Running
  --demand(worker 7)--> Running { owner: 3, waiters: [7] }
Running
  --success(value)--> Complete(value) --wake all waiters-->
Running
  --panic/cancel--> Poisoned/Cancelled --wake all waiters-->

Publication ordering matters. The owner must finish the value, dependency edges, fingerprint, and replayable side effects before marking complete. The synchronization primitive must establish that waiters see initialized data after observing completion. Safe Rust prevents many data races, but a wrong state transition can still return stale or duplicate semantics.

Never hold a global query-map lock while running a provider. Providers call other queries and can take arbitrary subsystem locks. Holding the map lock serializes the compiler and invites deadlock. Claim under a short lock, release it, compute, then publish under the job's protocol.

Owners and waiters#

The owner records its active parent and dependency reads. A waiter still records a dependency on the demanded query even though it does not execute the provider. Failure must wake every waiter. A dropped owner guard should transition away from Running during unwinding.

Condition variables require predicates in loops. Wakeups can be spurious, and a notification can occur before a waiter sleeps unless state and sleeping use one correct lock protocol. The condition is “job is no longer running,” not “I received one notification.”

Work stealing#

A blocked worker need not remain idle if the scheduler has independent ready work. Work stealing lets it execute jobs from another worker's queue. This improves utilization when query trees are irregular.

Stealing does not allow the waiter to compute the same owned key independently. It executes other work while preserving one publication for that key. Priority inversion, cache locality, queue contention, and deep dependency chains affect benefit.

useful_parallelism <= min(ready independent jobs, workers)
wall time >= critical dependency path
overhead = scheduling + synchronization + cache misses + blocked time

More workers can increase wall time when jobs are tiny, interners contend, memory bandwidth saturates, or locality collapses. Measure provider self-time and blocked/scheduler time separately.

85. Cross-thread cycles, cancellation, and poisoning#

Stack recursion detects A -> B -> A on one worker. Parallel execution can distribute the cycle:

worker 1 owns A --waits for--> B owned by worker 2
worker 2 owns B --waits for--> C owned by worker 3
worker 3 owns C --waits for--> A owned by worker 1

The runtime needs a wait-for graph or equivalent traversal across owner jobs. Before blocking, it can follow owners and waiting edges to determine whether the new edge closes a cycle. Cycle reporting should choose a deterministic semantic path where possible, not depend on which worker happened to notice first.

Recovery is query-specific. Some semantic cycles can emit a user error and return an error-tainted sentinel. Others cannot produce an honest value and must abort the path. The scheduler detects the cycle; query policy decides its meaning.

Cancellation can come from a fatal diagnostic, explicit interrupt, failed worker, or embedding host. It must be checked in bounded expensive loops and query boundaries. Cancellation is not success and must never publish a normal memo value or atomically commit an output.

Poisoning distinguishes abnormal provider exit from a valid error-valued query result. A provider panic may indicate an internal compiler bug. Waiters must wake and propagate the failure policy; silently recomputing can duplicate effects and obscure the first panic. Invocation-level abort may make retry irrelevant, but cleanup still must release locks and temporary resources.

Determinism under scheduling#

Semantic results, stable hashes, metadata, and promised diagnostics must not depend on job completion order. Use canonical ordering at observation boundaries. Do not assign stable identity from “next global counter” raced among workers. Do not hash randomized map iteration. Do not choose the first arriving diagnostic as primary without a semantic tie-breaker.

Determinism does not require identical trace timestamps or worker IDs. Define the observation precisely. Debug traces may expose scheduling, while artifact bytes may promise reproducibility under controlled inputs.

86. A stable-Rust query job state model#

This bounded complete model coordinates one computation per key, waiters, cancellation, and panic poisoning. It does not implement dependency recording, work stealing, or cross-thread cycle detection.

use std::collections::BTreeMap;
use std::panic::{catch_unwind, AssertUnwindSafe};
use std::sync::{Arc, Condvar, Mutex};

#[derive(Clone, Copy, Debug, Eq, Ord, PartialEq, PartialOrd)]
struct Key(u32);

#[derive(Clone, Debug, Eq, PartialEq)]
enum Failure { Cancelled, Poisoned }

#[derive(Debug)]
enum State {
	Vacant,
	Running { owner: std::thread::ThreadId, waiters: usize },
	Complete(Result<u64, Failure>),
}

struct Job { state: Mutex<State>, changed: Condvar }

impl Job {
	fn new() -> Self { Self { state: Mutex::new(State::Vacant), changed: Condvar::new() } }
}

struct Engine {
	jobs: Mutex<BTreeMap<Key, Arc<Job>>>,
	cancelled: Mutex<bool>,
	executions: Mutex<BTreeMap<Key, usize>>,
}

impl Engine {
	fn new() -> Self {
		Self {
			jobs: Mutex::new(BTreeMap::new()),
			cancelled: Mutex::new(false),
			executions: Mutex::new(BTreeMap::new()),
		}
	}
	fn cancel(&self) {
		*self.cancelled.lock().unwrap() = true;
		let jobs: Vec<_> = self.jobs.lock().unwrap().values().cloned().collect();
		for job in jobs { job.changed.notify_all(); }
	}
	fn get<F>(&self, key: Key, provider: F) -> Result<u64, Failure>
	where F: FnOnce() -> u64 {
		let job = self.jobs.lock().unwrap().entry(key)
			.or_insert_with(|| Arc::new(Job::new())).clone();
		let mut provider = Some(provider);
		loop {
			if *self.cancelled.lock().unwrap() { return Err(Failure::Cancelled); }
			let mut state = job.state.lock().unwrap();
			match &mut *state {
				State::Complete(result) => return result.clone(),
				State::Vacant => {
					*state = State::Running {
						owner: std::thread::current().id(), waiters: 0,
					};
					drop(state);
					*self.executions.lock().unwrap().entry(key).or_default() += 1;
					let result = catch_unwind(AssertUnwindSafe(provider.take().unwrap()))
						.map_err(|_| Failure::Poisoned);
					let result = if *self.cancelled.lock().unwrap() {
						Err(Failure::Cancelled)
					} else { result };
					*job.state.lock().unwrap() = State::Complete(result.clone());
					job.changed.notify_all();
					return result;
				}
				State::Running { owner, waiters } => {
					if *owner == std::thread::current().id() {
						// A real engine invokes query-specific cycle policy here.
						return Err(Failure::Poisoned);
					}
					*waiters += 1;
					state = job.changed.wait(state).unwrap();
					if let State::Running { waiters, .. } = &mut *state {
						*waiters = waiters.saturating_sub(1);
					}
					drop(state);
				}
			}
		}
	}
	fn executions(&self, key: Key) -> usize {
		self.executions.lock().unwrap().get(&key).copied().unwrap_or(0)
	}
}

fn main() {
	let engine = Arc::new(Engine::new());
	let mut threads = Vec::new();
	for _ in 0..8 {
		let e = Arc::clone(&engine);
		threads.push(std::thread::spawn(move || e.get(Key(1), || 42)));
	}
	for thread in threads { assert_eq!(thread.join().unwrap(), Ok(42)); }
	assert_eq!(engine.executions(Key(1)), 1);
}

#[cfg(test)]
mod tests {
	use super::*;
	use std::sync::Barrier;

	#[test]
	fn one_owner_many_waiters() {
		let e = Arc::new(Engine::new());
		let gate = Arc::new(Barrier::new(6));
		let mut ts = vec![];
		for _ in 0..6 {
			let (e, gate) = (Arc::clone(&e), Arc::clone(&gate));
			ts.push(std::thread::spawn(move || {
				gate.wait();
				e.get(Key(7), || 99)
			}));
		}
		for t in ts { assert_eq!(t.join().unwrap(), Ok(99)); }
		assert_eq!(e.executions(Key(7)), 1);
	}

	#[test]
	fn panic_is_published_as_poison() {
		let e = Engine::new();
		assert_eq!(e.get(Key(2), || panic!("boom")), Err(Failure::Poisoned));
		assert_eq!(e.get(Key(2), || 3), Err(Failure::Poisoned));
		assert_eq!(e.executions(Key(2)), 1);
	}

	#[test]
	fn cancellation_prevents_new_work() {
		let e = Engine::new();
		e.cancel();
		assert_eq!(e.get(Key(3), || 8), Err(Failure::Cancelled));
		assert_eq!(e.executions(Key(3)), 0);
	}
}

The state predicate prevents duplicate providers. Result publication and waiter wakeup occur after computation. catch_unwind is a teaching boundary, not a claim that rustc recovers from every panic.

The cancellation test covers cancellation before ownership, not every race. Cancellation immediately after provider return is resolved by checking before publication. The model can still leave a Running job if the thread aborts rather than unwinds. It also catches a same-thread re-demand only after reaching the same job, and labels it poison rather than constructing a semantic cycle.

Explicit omissions are dependency edges, parent stacks, priorities, work stealing, wait-for traversal, deterministic cycle selection, diagnostic replay, incremental persistence, retry policy, lock poisoning recovery, resource quotas, and asynchronous host cancellation. A production model must fault-inject at every state transition.

87. Target and session inputs#

A query key rarely contains every input to its answer. layout_of(T) is keyed by a type, but pointer width and ABI come from target/session configuration. Correct dependency tracking or cache namespacing must account for those inputs.

Important classes include:

Operational inputs must not accidentally become semantic. Changing worker count may alter traces, but not definition identities, metadata meaning, diagnostics promised as deterministic, or executable behavior.

Target specifications require validation. A self-consistent-looking pointer width combined with an incompatible data layout can produce wrong layout. The session should derive related facts from one validated target contract rather than accept conflicting independent globals.

Host and target must remain distinct. Proc macros and build-time tools execute for the host. The compiled crate obeys the target. Loading a target proc-macro artifact into the host process or using host usize layout for target MIR are boundary errors.

Path remapping, current directory, environment variables consumed by macros, and explicitly passed --cfg values can influence observations. Any direct ambient read inside a provider risks an invisible dependency. Capture it as invocation input, prohibit it, or reject reuse across changes.

88. Clean-build equivalence#

Every reuse optimization is justified by one equation:

observe(incremental(previous inputs, current inputs))
    == observe(clean(current inputs))

Choose observe broadly enough:

• acceptance or rejection;
• normalized diagnostics and applicability;
• crate metadata meaning;
• linker-visible symbols and ABI;
• executable behavior;
• artifact bytes where reproducibility is promised;
• termination rather than deadlock under bounded resources.

Clean-build equivalence does not require equal query schedules, addresses, cache statistics, or trace timestamps. Those are operational unless a supported interface promises them.

Input matrix#

Test changes one coordinate at a time and in combinations:

1. virtual source text with unchanged physical file;
2. physical path changed under stable remapping;
3. same-name external crate changed to another identity;
4. upstream private body, public signature, and generic body edits;
5. metadata producer/consumer compatibility mismatch;
6. target pointer width or feature change in isolated caches;
7. lint cap and edition changes;
8. thread count and query-root order permutations;
9. cancellation followed by fresh retry;
10. truncated or corrupted optimization state.

Corrupt incremental state should cause safe rejection/recomputation where feasible. Corrupt dependency artifacts may require a clear hard error because the producer artifact itself is an input, not merely an optimization. Do not blur those policies.

Differential harness#

For each generated edit sequence:

1. Capture all virtual and filesystem bytes.
2. Run incrementally from the previous state.
3. Run clean in a fresh directory with identical captured inputs.
4. Normalize only declared environmental noise.
5. Compare every selected observation.
6. On mismatch, identify the first divergent query/input/decoded field.

Metamorphic properties add power. Vary thread count while keeping semantic inputs fixed. Move a remapped tree between physical directories. Permute declarations where the language makes order irrelevant. Demand external queries in different orders. All should preserve their declared observations.

89. Security, corruption, and resource limits#

Compiler inputs include source, metadata, libraries, proc-macro output, target data, caches, and linker messages. Treat byte-oriented decoders as hostile-input boundaries even when normal build tools produce the bytes. Compiler execution of proc macros and build scripts is a separate code-execution risk; decoder hardening is not sandboxing.

Required limits#

Bound metadata blob size, decompressed size, table count, string length, nesting depth, list cardinality, source-file size, crate graph size, query recursion, active jobs, waiters, diagnostics, and temporary output bytes. Use checked addition and multiplication before slicing or allocating. Reject cyclic structures where the schema requires a DAG. Represent truncation explicitly rather than silently dropping semantic data.

Commit protocol#

Write metadata and caches to temporary files in the destination filesystem. Flush as policy requires, validate completion, then atomically rename. Do not leave a valid header pointing to a partial body. Clean abandoned temporaries conservatively.

Integrity hashes detect accidental corruption but do not authenticate an untrusted producer unless keyed or signed under a trust model. Version tags identify format, not trust. Compression libraries and parsers need fuzzing at actual rustc entry points.

Unsafe decoder proof obligation#

If implementation uses unchecked indexing or typed views for speed, state:

• caller guarantee: complete validation established bounds, alignment, and format;
• implementation assumption: bytes cannot mutate during the view;
• lifetime rule: decoded references do not outlive backing storage;
• consequence of violation: memory unsafety, not merely a bad diagnostic;
• tests/fuzzing: every validation path precedes unsafe construction.

Prefer safe decoding until profiling proves a hotspot. Moving checks out of a loop can help, but only when one dominating check proves every iteration.

90. Performance engineering across the infrastructure#

Optimizing one component can move cost rather than remove it. Smaller metadata may require more decoder CPU. More interning may reduce values but increase lock contention. Finer query granularity may improve reuse but enlarge job and dependency tables.

An end-to-end model is:

T_total = T_load_source
        + T_discover_and_validate_crates
        + T_metadata_io_decompress_decode
        + T_query_compute
        + T_query_wait_schedule
        + T_hash_and_intern
        + T_codegen_link
        + T_persist

M_peak = source maps + metadata blobs + decoded values
       + arenas + interner tables + query jobs/caches + backend state

Measure clean check, no-op incremental check, tiny body edit, public API edit, full build, large generated crate, deep dependency graph, cold metadata cache, and constrained-memory environments. Report CPU time, wall time, blocked time, peak RSS, bytes read/written, decode counts, unique/total interning cardinality, and cache hit rates.

A high cache hit rate can be a correctness bug if inputs are missing. A low decode count can hide eager whole-blob decompression. Lower allocation count can accompany higher retained memory. Use clean equivalence before celebrating performance.

Representative experiments#

• Disable one metadata cache and compare decode CPU plus RSS.
• Record demanded table entries versus encoded entries by family.
• Measure interner N, U, candidate bytes, table capacity, and lock wait.
• Compare thread counts while graph roots and outputs remain fixed.
• Plot source loading bytes against wall time on cold and warm storage.
• Perturb hash-map insertion order and compare stable artifacts.

Never report invented benchmark numbers. Publish commands, commit, machine, compiler configuration, repetitions, variance, and raw observations.

91. Failure maps and earliest invariants#

Debugging protocol#

1. Pin rustc -vV, source revision, host, target, and all flags.
2. Preserve input bytes, virtual revisions, dependency artifacts, and cache state.
3. Classify source, crate discovery, metadata, identity, interning, query, or session boundary.
4. Build the smallest reproducer that preserves that boundary.
5. Compare clean and optimized executions.
6. Record stable descriptions, not pointer addresses alone.
7. Find the first wrong structured value or state transition.
8. Add fault injection or corruption only in a disposable directory.
9. Write a focused regression that fails before the fix.
10. Measure overhead if the fix adds checks, locks, or retained data.

The panic site reports where an assumption was noticed. A mangling collision can begin in crate identity. A decoder panic can begin in encoding. A query deadlock can begin in a provider holding an unrelated lock. Trace backward.

92. Source navigation for rustc 1.97.1#

Pin the tag/commit corresponding to 1.97.1 before trusting names. Use current source as normative evidence for implementation and the rustc-dev-guide as architectural explanation.

For one external query, follow this exact route:

1. Find the query declaration and key/result types.
2. Find local and external provider registration.
3. Find the external provider body.
4. Find the table accessor and lazy decode site.
5. Find every identity translation invoked by the decoder.
6. Find where the decoded value is interned.
7. Find the two-crate test exercising it.

For metadata source archaeology, use symbol search before filenames. Then use git log -S for a schema field or magic/version identifier. Read the pull request that introduced it to understand compatibility and performance intent. Line numbers and private type names are not durable documentation.

93. Contributor workshops#

Workshop A: virtual loader consistency#

Implement a loader whose file_exists and read_file consult one immutable map. Inject an overlay revision change during a compile request. Prove the invocation either sees the old snapshot completely or cancels. Deliver a test for no mixed digest and no physical-path leak.

Workshop B: crate identity collision thought experiment#

Build two dependencies with the same crate name but distinct disambiguation contexts. Trace Cargo extern arguments, rustc candidate validation, local CrateNums, StableCrateIds, and mangled symbols. Predict the failure if any layer uses name alone. Do not force an actual stable-ID hash collision.

Workshop C: one metadata field#

Choose a small per-definition fact used externally. Locate its local query, encoder table, decoder accessor, external provider, and tests. State whether absent means None, default, not encoded, or corruption. Add a two-crate round-trip test before changing schema.

Workshop D: lazy access profile#

Instrument one large dependency build. Count encoded entries, indexed entries, and decoded entries by table. Measure cold/warm time and peak RSS. Propose eager or lazier behavior only from demand evidence.

Workshop E: corruption matrix#

In disposable artifacts, mutate magic, version, root position, table offset, count, string length, UTF-8, and compression framing. Classify clear incompatibility error, corruption error, safe cache miss, and ICE. Never place corrupted bytes in a shared build cache.

Workshop F: definition translation#

Trace one external trait method and one anonymous definition. Record producer DefId, encoded crate slot, stable crate identity, path hash where used, consumer CrateNum, and final DefId. Insert an unrelated sibling and observe which stable descriptions change.

Workshop G: symbol boundary#

Compile generic functions in two same-name crates. Collect concrete instances, mangled names, linkage, object ownership, and demangled presentation. Repeat with an explicit export name and explain the changed collision responsibility.

Workshop H: arena retention#

Instrument allocations by arena family during check and build. Find a large value that dies semantically long before its arena. Estimate benefit and complexity of a phase-local arena. Verify no reference escapes before proposing lifetime changes.

Workshop I: interner census#

Measure candidates, uniques, bytes, average list length, lookup probes, and lock wait for one interner. Create a repeated and a high-cardinality workload. Do not optimize from the repeated workload alone.

Workshop J: query waiter fault injection#

Pause after claim, after dependency reads, before publication, after publication, and during wakeup. Inject panic and cancellation at each point. Assert all waiters terminate and no incomplete value is visible.

Workshop K: cross-thread cycle#

Extend the educational runtime with parent jobs and wait-for edges. Construct three workers owning A, B, and C in a cycle. Report the same normalized cycle under schedule permutations. Separate detection from semantic recovery.

Workshop L: target isolation#

Compile a layout-sensitive crate for two targets using intentionally isolated and then intentionally shared experimental cache locations. Verify rustc's compatibility boundaries reject invalid reuse. Compare with clean outputs and inspect the earliest target-dependent query.

Workshop M: clean-equivalence capstone#

Combine a virtual source edit, upstream metadata change, thread-count change, and cancellation/retry. Capture normalized diagnostics, rmeta meaning, symbols, and executable output. Compare incremental/retried and clean invocations. Explain every allowed operational difference.

94. Design reviews: alternatives and counterfactuals#

If source paths are canonicalized at first contact, duplicate detection becomes easier but symlink presentation and privacy become harder. If canonicalization is delayed, every identity consumer must choose a path form. The complexity moved; it did not vanish.

If metadata eagerly decodes everything, provider code simplifies but startup and memory scale with all encoded facts. If decoding is lazy, each accessor needs bounds, lifetime, synchronization, and caching discipline.

If definitions serialize raw indices, files are compact but session ordering becomes authority. Stable translation adds hashing and maps so local numbering may remain cheap and free to change.

If all semantic values are arena allocated without interning, insertion is cheap but repetition consumes memory and equality stays structural. Interning moves work to hashing, synchronization, and a canonical lifetime.

If every query key has a single global mutex, correctness is easier to inspect but independent work serializes. Fine-grained jobs improve parallelism while adding waiter, cycle, cancellation, and memory obligations.

If diagnostics emit directly from workers, latency can improve but ordering and duplicate suppression become schedule-sensitive. Buffering enables deterministic policy but retains memory and delays presentation.

If target options are copied into every key, dependencies are explicit but keys become large and pervasive. Tracked singleton inputs or cache namespaces centralize compatibility but can over-invalidate. Choose the smallest boundary that proves every affected observation changes.

Prediction laboratory: source and crate inputs#

Predict each result before reading the explanation.

An overlay loader returns new bytes while file_exists still consults disk. Module discovery can report absence even though a read would succeed. The two operations violated one-snapshot consistency.

A remapping rule changes but physical bytes do not. Semantic type checking may remain equal. Diagnostics, dep-info, metadata provenance, debuginfo, and reproducible bytes may change. The affected cache boundary follows observations, not file content alone.

Two paths contain byte-identical rmeta files. May rustc always merge their crate nodes? No: graph role and expected crate identity still require validation. Byte equality is evidence, not a complete dependency-graph policy.

Two crates have equal names and exported APIs. May their DefIds compare equal? No: definition identity includes the crate domain. Structural API equivalence does not erase origin.

One external provider is never demanded. Must its whole table be decoded? Not under a genuinely lazy design. Header, graph, or index validation can still occur eagerly.

Prediction laboratory: metadata#

The metadata version matches but an offset points one byte past the blob. Decoding must reject it before constructing a slice. Version compatibility never proves integrity.

A length is below the configured element limit. May allocation proceed immediately? Not until length * element_size, remaining input, and representation constraints are checked.

Compression expands one kilobyte into ten gigabytes. A compressed-size limit alone is ineffective. The decompressor needs an output budget and bounded nesting/framing.

An unknown optional field appears in a hypothetically extensible schema. Skipping is safe only if framing identifies its extent and compatibility policy permits unknown fields. Rustc's private format need not provide that forward-compatibility promise.

Two records decode to structurally equal types. May their definition IDs be merged? No: canonicalizing type structure does not merge nominal definition identity.

A decoder caches an interned Ty<'tcx> beside a process-global artifact cache. That reference would outlive its owning context. Cache stable bytes or descriptions globally and re-intern per context.

Prediction laboratory: arenas and interning#

An arena allocation becomes unreachable after one query. Is its memory reclaimed immediately? Normally no; the owning arena's lifetime controls reclamation.

A dropless arena stores a String and skips its destructor. The string's separate heap buffer leaks even if the arena's chunks are released. Dropless eligibility must include transitive ownership behavior.

Every candidate type is unique. Interning can consume more memory than direct arena allocation. The canonical table adds capacity and hashing without deduplication benefit.

Two threads race to intern equal lists. Both may prepare candidates, but publication must select one canonical representative. The losing candidate needs safe disposal or arena-retention policy.

Pointer equality says two interned values differ. Can structural equality be skipped? Only if both references are known to belong to one canonical interner domain. Cross-context pointers provide no such guarantee.

A list interner reports excellent deduplication but compilation slows. Candidate hashing, equality probes, lock wait, and cache locality may exceed bytes saved. Measure all terms.

Prediction laboratory: query jobs#

Worker A owns a key and worker B demands it. May B omit a dependency edge because it did not execute the provider? No: B's caller still semantically depends on the result.

An owner panics before notification. A waiter must not sleep forever. An unwind guard or invocation abort protocol must change state and wake it.

A waiter wakes while state is still Running. It must recheck the predicate and sleep again or observe cancellation. Notifications are not values.

A worker waiting for key X steals independent key Y. Has ownership of X changed? No: stealing scheduler work is separate from query publication ownership.

Three workers close a wait cycle at nearly the same time. The reported semantic cycle should not rely on the winner's thread number. Normalize through stable query/key descriptions and query-specific policy.

Cancellation arrives after a provider computes but before publication. Publishing success would make interruption timing semantic. The runtime must resolve cancellation policy before committing completion.

A poisoned key is demanded again in the same invocation. Blind recomputation can duplicate side effects and hide the original failure. Propagation or invocation abort is safer unless retry is explicitly designed.

Production review ledger#

For source loading, record the requested, physical, remapped, and display identities. Record whether every read belongs to one immutable snapshot. Record the maximum accepted source size and error behavior. Record symlink and canonicalization policy. Record which source identity enters stable hashing.

For crate discovery, record who supplies candidates. Record how host and target artifacts are distinguished. Record which header facts reject incompatibility. Record same-name and duplicate-artifact behavior. Record local crate-number assignment and stable mapping.

For every metadata field, record its semantic owner. Record why downstream requires it. Record absent, optional, and malformed meanings separately. Record key cardinality and expected demand fraction. Record position, size, and nesting bounds. Record every translated identity nested inside it. Record whether decoding interns or clones. Record format compatibility impact of changing it.

For every arena family, record destruction policy. Record the longest-lived borrower. Record peak retained bytes by phase. Record whether values own non-arena resources. Record thread access and publication rules. Record why a shorter lifetime is impractical.

For every interner, record its equality relation. Record candidate count N and unique count U. Record candidate and canonical byte sizes. Record table capacity and collision/probe behavior. Record synchronization cost. Record whether IDs are schedule-dependent and where that is allowed. Record whether values can contain context-local references.

For every parallel query, record the key's stable description. Record owner registration and publication linearization points. Record dependency behavior for owners and waiters. Record same-thread and cross-thread cycle policies. Record cancellation safe points. Record panic/poison behavior. Record side effects and replay ownership. Record deterministic output policy.

For every target-sensitive query, list target facts read dynamically. List compatibility facts supplied by a broader cache namespace. Explain why changing each relevant option invalidates enough work. Explain why operational options do not alter semantics. Test host/target confusion explicitly.

Negative examples worth preserving#

A one-crate test is negative evidence for an external-provider fix. It can pass without encoding or decoding anything.

A round-trip test using only valid bytes is negative evidence for decoder hardening. It does not exercise bounds, overflow, or allocation attacks.

A serial query test is negative evidence for waiter correctness. It never enters the running-owned-by-another-worker state.

A pointer-equality unit test in one context is negative evidence for stable identity. It says nothing about another session.

A metadata size benchmark is negative evidence for compile-time improvement. It omits compression CPU, demand, and retained decoded memory.

A clean-build-only test is negative evidence for incremental input tracking. No old answer exists to misuse.

An incremental-only expected-output test is negative evidence for correctness. Without the clean oracle, both expectation and cache can be wrong.

A many-thread speedup on one wide graph is negative evidence for universal parallel benefit. Critical paths, memory bandwidth, and tiny jobs differ across workloads.

A deterministic terminal snapshot is negative evidence for deterministic metadata. Different observations can have independent ordering leaks.

A successful corrupted-cache fallback is negative evidence for corrupted dependency handling. Optimization state and required upstream artifacts have different recovery policies.

Maintainer handoff template#

State the user-visible or compiler-engineering problem. State the exact 1.97.1 commit investigated. State the earliest invariant believed broken. State the producer and consumer of the suspect representation. State every identity domain crossed. State every lifetime and owner involved. State whether the path is local, external, incremental, or clean. State host and target. State thread count and cancellation conditions. State the smallest reproducer preserving those boundaries. State the clean-build oracle. State corruption and resource-limit behavior. State measured time, memory, and artifact effects. State explicit omissions and unresolved uncertainty. State the focused tests that fail before the proposed fix.

95. Derived philosophy#

Identity has a lifetime. A CrateNum, DefId, symbol ID, arena pointer, DefPathHash, and linker name solve different identity problems. Using a longer-lived identity everywhere adds cost; using a shorter-lived one across a boundary is wrong.

Representations make selected questions cheap. A metadata table makes “fact for definition 17” cheap and whole-schema evolution costly. An interner makes equality cheap after insertion and makes insertion, retention, and synchronization costly.

Abstractions move authority. FileLoader moves byte authority from direct filesystem calls into a host policy. External providers move acquisition behind queries but do not remove schema obligations. Arenas move deallocation from each value to an owner lifetime.

Caching creates a protocol. A cached answer needs identity, complete inputs, publication, compatibility, corruption handling, cancellation, and observability. The value alone is the easy part.

Canonicalization must name its equality. Equal spelling, equal type structure, equal normalized semantics, equal definitions, and equal object symbols are not interchangeable. Intern only after choosing the observation that equality preserves.

Parallelism reveals hidden impurity. Scheduling variation exposes identities derived from allocation order, diagnostics emitted by arrival order, and providers reading ambient mutable state. The race is often semantic even when memory access is data-race-free.

Provenance must be preserved before it is needed. A decoder cannot reconstruct a source origin or definition domain that the encoder discarded. A diagnostic cannot remap a physical path if an earlier stage serialized only presentation text.

The first visible failure is late. Wrong code may begin in target input tracking. A linker collision may begin in crate identity. A deadlock may begin in lock ownership before the wait cycle closes. Find the earliest broken invariant.

96. Mastery path, talk designs, and authoritative reading#

Mastery checks#

1. Explain why remapped path and physical path must coexist.
2. Design snapshot semantics for a virtual loader.
3. Distinguish crate discovery, compatibility, and identity.
4. Explain why raw CrateNum cannot cross metadata.
5. Derive a lazy-table corruption checklist.
6. Extend the metadata model with a bounded string pool.
7. Trace one external query from DefId to interned result.
8. Distinguish DefId, StableCrateId, and DefPathHash domains.
9. Explain why mangled name and semantic definition differ.
10. Select typed, dropless, phase, or context arena for four workloads.
11. Compute interning break-even from measured N, U, and overhead.
12. Explain 'tcx without calling it static.
13. Draw job ownership and waiter transitions for panic and cancellation.
14. Detect a cross-thread cycle without holding provider locks.
15. Define deterministic observations under work stealing.
16. Enumerate target/session inputs for one layout query.
17. Design clean-equivalence tests across two crates and two targets.
18. Review an unsafe decoder as a proof obligation.
19. Locate the earliest invariant in three failure-map cases.
20. Teach the entire path without relying on private type names.

Talk one: “A crate is not a filename”#

Start with a search path. Add artifact validation, crate graph edges, same-name crates, stable crate identity, local numbering, definition translation, and symbols. End with one collision traced from discovery to linkage.

Talk two: “Metadata is a database under hostile conditions”#

Derive schema, indexes, lazy offsets, sharing, compression, versioning, bounds, and external providers. Demonstrate the stable-Rust round trip and mutate every framing field. Finish with demand-driven cost measurements.

Talk three: “Why rustc keeps so much memory”#

Begin with millions of small values. Compare individual allocation, typed arenas, dropless arenas, and interning. Derive cardinality and retention costs. Finish at the 'tcx ownership boundary.

Talk four: “One query, eight workers, one answer”#

Derive claim, owner, waiter, publication, work stealing, cross-thread cycles, cancellation, poisoning, and deterministic output. Use fault injection to show that every transition is a correctness boundary.

Talk five: “Clean build is the oracle”#

Classify source, crate, target, diagnostic, and operational inputs. Define observations. Run edit sequences across metadata and thread schedules. End with why a perfect cache hit rate can be a bug.

Authoritative reading map#

Use pinned 1.97.1 source for all implementation claims:

• https://github.com/rust-lang/rust/tree/1.97.1/compiler/rustc_span
• https://github.com/rust-lang/rust/tree/1.97.1/compiler/rustc_session
• https://github.com/rust-lang/rust/tree/1.97.1/compiler/rustc_target
• https://github.com/rust-lang/rust/tree/1.97.1/compiler/rustc_interface
• https://github.com/rust-lang/rust/tree/1.97.1/compiler/rustc_middle
• https://github.com/rust-lang/rust/tree/1.97.1/compiler/rustc_query_system
• https://github.com/rust-lang/rust/tree/1.97.1/compiler/rustc_metadata
• https://github.com/rust-lang/rust/tree/1.97.1/compiler/rustc_symbol_mangling

Architecture and contributor context:

• https://rustc-dev-guide.rust-lang.org/rustc-driver/intro.html
• https://rustc-dev-guide.rust-lang.org/query.html
• https://rustc-dev-guide.rust-lang.org/parallel-rustc.html
• https://rustc-dev-guide.rust-lang.org/backend/libs-and-metadata.html
• https://rustc-dev-guide.rust-lang.org/memory.html
• https://rustc-dev-guide.rust-lang.org/queries/incremental-compilation.html
• https://rustc-dev-guide.rust-lang.org/queries/incremental-compilation-in-detail.html

Current nightly API docs are navigation aids, not evidence for the pinned revision:

• https://doc.rust-lang.org/nightly/nightly-rustc/rustc_span/source_map/trait.FileLoader.html
• https://doc.rust-lang.org/nightly/nightly-rustc/rustc_span/def_id/
• https://doc.rust-lang.org/nightly/nightly-rustc/rustc_metadata/
• https://doc.rust-lang.org/nightly/nightly-rustc/rustc_middle/ty/struct.TyCtxt.html
• https://doc.rust-lang.org/nightly/nightly-rustc/rustc_query_system/query/
• https://doc.rust-lang.org/nightly/nightly-rustc/rustc_interface/interface/struct.Compiler.html

Use the rustc book for supported command-line contracts and the Reference for language semantics. Record the exact commit because 1.97.1 may differ from current nightly documentation and development-guide prose.

The final test of mastery is not recalling an internal field name. It is being able to state which authority owns bytes, identity, lifetime, publication, and configuration at every boundary; to build a bounded model; and to prove every optimization against the clean execution it replaces.

Part IX: Building, Testing, Debugging, and Contributing to rustc#

1. The production-contributor mindset#

This part turns compiler knowledge into a safe working practice. The goal is not merely to make one local test pass. The goal is to make a change that millions of builds can depend on. Production contribution therefore combines technical skill, evidence, communication, and restraint.

An absolute beginner can contribute without understanding all of rustc. Nobody understands every subsystem in equal depth. A reliable contributor learns one path through the system, states uncertainty, and asks the owners of adjacent paths for review. Reliability means that reviewers can reproduce the claim, understand the risk, and maintain the result after the author leaves.

The central loop is simple: reproduce, reduce, locate, test, change, measure, explain, and respond. Each verb protects against a common failure. Reproduction prevents solving an imagined bug. Reduction removes irrelevant code. Location avoids scattering speculative edits. A test preserves the behavior. A small change limits risk. Measurement challenges intuition. Explanation transfers knowledge. Response turns review into collaboration rather than combat.

rustc is a self-hosted production compiler. Self-hosted means that the compiler is largely written in the language it compiles. Production means stable users expect compatibility, diagnostics, performance, and target support. Those facts explain the unusually elaborate bootstrap and test systems described below.

Commands and labels in this chapter are guardrails for the Rust 1.97.1 era, not eternal interfaces. Bootstrap flags, suite names, bots, team structure, and repository paths evolve. Before acting, compare this chapter with the live rustc development guide at https://rustc-dev-guide.rust-lang.org/ and its current command help. Use ./x --help and ./x test --help in the checkout you actually modify. The checkout is more authoritative than a blog post, including this textbook.

Do not make RUSTC_BOOTSTRAP part of an ordinary production crate's build. That variable bypasses stability checks for compiler-development purposes and can silently tie users to internals. Use the repository's supported bootstrap machinery instead.

2. Governance, authority, and review#

The rust-lang/rust repository is shared infrastructure, not a personal project with one maintainer. The Rust project is organized into teams with delegated responsibilities. The compiler team stewards rustc implementation, compiler contributors, reviews, regressions, and technical direction. Other teams own the language design, libraries, release operations, infrastructure, documentation, and specialized areas. Current membership and charters live in the team repository and Rust Forge, not in a static name list here.

Governance answers who may decide, while review answers whether a particular patch is ready. A reviewer approval is technical and social accountability. It does not mean the reviewer wrote every line or can prove the patch perfect. It means the evidence and risk are acceptable under project policy.

Ownership is often overlapping. A parser change can affect diagnostics, language semantics, rustfmt, proc macros, and edition behavior. A standard-library change can require library-api judgment even when compiler CI builds it. Ask rather than assuming that a compiler reviewer owns every consequence.

Start with the official contribution procedures at https://rustc-dev-guide.rust-lang.org/contributing.html. Consult the compiler team material at https://forge.rust-lang.org/compiler/ and the project governance pages linked there. The Rust Code of Conduct at https://www.rust-lang.org/policies/code-of-conduct applies to issues, pull requests, chat, meetings, and private interactions connected to the project. It is a baseline for respectful participation, not a weapon for winning technical disputes.

Most implementation fixes do not require an RFC. An RFC, or Request for Comments, is a formal design proposal used when project-level design and consensus are needed. A tracking issue records implementation and stabilization work for an accepted feature or other multi-step effort. It is not itself the design authority. Small bug fixes, diagnostic improvements, refactors, and many performance changes normally proceed through issues and pull requests. When semantic policy is unclear, ask whether a language-team decision, compiler-team proposal, library API process, or RFC is appropriate.

3. Communication, Zulip, and asking for help#

Rust uses Zulip for much project discussion at https://rust-lang.zulipchat.com/. Zulip groups messages into streams and topics. A stream is a broad room; a topic is a focused thread inside it. The compiler help stream is a good place for concrete rustc-development questions. Choose a descriptive topic such as “building stage1 after parser change,” not “help” or “urgent.”

Good help requests are compressed investigations. State the desired behavior, actual behavior, host platform, checkout commit, exact command, and smallest relevant error. Link an issue or branch when public. Say what you already tried and what evidence changed your mind. Put long logs in a paste or code block and quote the decisive lines.

Do not privately message a stranger merely because they touched nearby code. Public questions distribute knowledge and let another expert answer. Do not repeatedly ping people across GitHub and Zulip. Maintainers may be volunteers, asleep, on leave, or focused on releases. A patient follow-up after meaningful new evidence is useful; repeated “any update?” messages are not.

Before asking, search the development guide, issue tracker, merged pull requests, and Zulip history. Searching is not a prerequisite to deserving help. It is a way to make the conversation more productive. If terminology is blocking you, say so plainly.

Disagreement should target claims and tradeoffs. Write “this test does not cover cross-crate metadata” rather than “you ignored correctness.” Summarize decisions after a long thread. If guidance conflicts, ask the assigned reviewer to resolve scope rather than assembling whichever answer you prefer.

Never paste embargoed security details, private user code, tokens, or personally identifying crash data into public chat. The responsible-disclosure boundary is discussed later. Code of Conduct concerns should use the current reporting route on the policy page, not an improvised public trial.

4. Issues, labels, assignment, and design records#

An issue is a durable report or work item. Search open and closed issues before filing because an old closed report may explain intent. Useful reports contain a reproducer, actual output, expected outcome, version details from rustc --version --verbose, and regression status. “Regression” means behavior became worse compared with an earlier compiler.

Labels are metadata used for triage, not verdicts about the reporter. Common families identify area, kind, priority, regression status, effort, and teams. Their exact spellings and meanings change. Read the repository's current label descriptions and triage documentation rather than copying an old label recipe. A priority label expresses project impact, not how much an individual wants a fix.

Triagebot automates repository chores. Depending on current configuration, commands can claim an issue, assign a reviewer, request labels, or notify a team. The bot's command grammar is version-sensitive. Consult https://forge.rust-lang.org/triagebot/ and the repository's triagebot configuration before using it. If a command fails, do not spam variants; read the bot response or ask in the appropriate Zulip topic.

Reviewer assignment is routing, not guaranteed immediate service. The bot may choose from expertise and workload data. An author can explain why a specialist is needed, but should avoid reviewer shopping after receiving difficult feedback. When an assigned reviewer is unavailable for a long period, ask publicly for reassignment with context.

An RFC describes an agreed direction and rationale. A tracking issue coordinates unresolved questions, implementation steps, tests, documentation, and stabilization. A pull request changes artifacts. Link all three when relevant, but do not collapse them into one concept. Read the latest RFC text and subsequent team decisions because tracking-issue checklists can evolve.

Claiming an issue prevents duplicated effort but does not confer ownership forever. Post a short plan, expected first step, and availability. If you stop, unassign yourself or leave a handoff note. That small act makes open source kinder and more efficient.

5. A map of the repository#

The root is an integration workspace containing the compiler, standard library, tests, tools, bootstrap system, and release machinery. Learn boundaries before searching by filename. The live repository tree and https://rustc-dev-guide.rust-lang.org/getting-started.html are the final references.

compiler contains crates whose names usually begin with rustc_. A crate is Rust's compilation unit and dependency boundary. Examples cover parsing, expansion, type representation, trait solving, borrow checking, middle-level IR, code generation, metadata, sessions, errors, and the driver. Names can move as architecture changes, so follow Cargo.toml dependencies and API documentation.

library contains core, alloc, std, test, proc_macro, and related library implementation. core supplies language-adjacent facilities without operating-system assumptions. alloc adds allocation-based types. std adds platform integration and a broad stable API. Changing library source is not the same as changing compiler source even though bootstrap builds both.

tests contains repository-wide suites and fixtures. tests/ui is the large diagnostics-oriented compiletest suite. Other directories cover code generation, assembly, MIR optimization, incremental compilation, rustdoc, run-make projects, and platform behavior. Always inspect neighboring tests and suite headers before inventing syntax.

src/tools contains tools shipped with Rust or used to build and test it, such as rustdoc-related support, compiletest, tidy, and bootstrap-adjacent programs. Tools may have their own owners and test conventions. src/bootstrap and the root x entry point implement the build orchestrator. CI configuration describes remote builders, while tidy enforces repository policies cheaply and consistently.

Ownership boundaries reduce accidental coupling. A change to compiler diagnostics should not casually alter bootstrap. A build-system convenience should not smuggle a language semantic change. When a patch crosses boundaries, explain why each crossing is necessary and seek the relevant reviewers.

6. Source navigation and finding expertise#

Start navigation from a behavior, not from a guess at a crate name. Search the exact diagnostic fragment with ripgrep, then search its error code, fluent message key, or test annotation. Search syntax tokens in parser tests and semantic terms in the development guide. Use rust-analyzer for definitions, references, type information, and call hierarchy, while remembering generated code and query indirection may confuse it.

Nightly compiler API documentation at https://doc.rust-lang.org/nightly/nightly-rustc/ exposes internal crate items. These APIs are explicitly unstable. The docs help answer “what type is this?” but source and callers explain invariants better. Generate local documentation only when needed because it costs time and disk.

Git history records why code reached its current shape. Use git log -- path to find changes around a file. Use git log -S followed by a distinctive string to find when that string's occurrence count changed. Use git log -G with a regular expression for changed lines matching a pattern. Use git blame on a narrow range, then open the associated pull request and issue.

Blame identifies the last textual edit, not the owner or culprit. Formatting and moves obscure ancestry. Use blame as an index into history, never as a public accusation. Recent authors and reviewers of several related patches are useful potential experts. Ask publicly and give them an easy way to decline.

Read horizontally as well as vertically. Horizontally means following one value through parser, AST, lowering, type checking, MIR, and diagnostics. Vertically means understanding one subsystem's data structures, invariants, and tests in depth. Beginners often read entire files without a question and retain little. Write down a concrete question before following each symbol.

Maintain a small investigation notebook with commands, commit hashes, observations, and disproved hypotheses. It prevents circular debugging and produces a strong pull-request explanation later.

7. Prerequisites, machine cost, and planning#

A rustc checkout is much heavier than an ordinary crate. You need Git, Python, a C/C++ toolchain, platform development libraries, enough memory, and substantial free storage. Some targets require linkers, SDKs, emulators, or system packages. The current platform-specific prerequisites are listed in the getting-started and build chapters of the development guide.

Bootstrap downloads a stage0 toolchain and may download LLVM or a previous rustc artifact. Build outputs, incremental caches, LLVM, standard libraries, and test artifacts can consume tens of gigabytes. Exact costs depend on profile, host, targets, debug information, and time. Do not promise a fixed number to a learner; inspect free space and the build directory.

CPU parallelism speeds many steps but raises peak memory. If the operating system kills compiler processes, reduce jobs before diagnosing a compiler bug. Thermal throttling, antivirus scanning, network filesystems, and low disk space can create misleading timing or linker failures. Keep the repository on a fast local filesystem when possible.

The fastest onboarding strategy is not always the smallest download. A recommended bootstrap profile and downloaded LLVM may consume more network and disk but save hours of compilation. A full debug-information build helps a debugger but slows linking and enlarges artifacts. Choose for the task.

Record the host platform and bootstrap configuration when reporting failures. Do not publish private paths or credentials from logs. If working in a container, remember that debugger permissions, available RAM, and mounted-filesystem performance differ from the host.

Budget attention as well as compute. Start a large build before a break, but use fast checks while editing. Do not repeatedly launch full CI-equivalent suites because uncertainty makes you anxious. Match verification cost to the changed boundary and expand it once the patch stabilizes.

8. Clone strategies and history tradeoffs#

A normal clone downloads reachable Git history and gives the best archaeology experience. It supports log searches, blame, bisecting, and examining old code without later network fetches. The cost is initial download size and time.

A shallow clone limits commit depth. It is attractive on constrained networks but weakens blame, merge-base calculations, bisecting, and investigation of older regressions. You can deepen it later, but that may happen at the least convenient moment. A contributor planning compiler archaeology should prefer adequate history.

A partial clone can omit file contents until Git needs them. This reduces initial transfer while preserving more graph information than a shallow clone. The tradeoff is network-dependent operations, server support, and occasional tool surprises. Sparse checkout hides paths from the working tree but does not make rustc's build graph sparse automatically.

A typical full start is intentionally boring.

git clone https://github.com/rust-lang/rust.git
cd rust
git remote add your-fork https://github.com/YOUR-NAME/rust.git
git status

Replace the fork URL with a real fork; the uppercase form is explanatory, not a command to copy blindly. Keep an upstream remote if your initial clone points at a fork. Fetch before starting work and branch from the current default branch according to project guidance.

Avoid mixing generated build output into source control. Check git status before and after every broad formatting or blessing operation. If a command modifies hundreds of unexpected tests, stop and inspect instead of committing noise.

Git worktrees can maintain separate working directories sharing object history. They are useful for a clean benchmark baseline and an experimental patch. Separate build directories still consume storage, and accidentally benchmarking two active builds creates resource contention.

9. Why a self-hosted compiler needs bootstrap#

Suppose source version N of rustc is written in Rust N. To compile it, a rustc executable must already exist. The project solves this apparent circle by using a known earlier compiler capable of compiling the new source. That trusted starting toolchain is stage0.

Stage0 builds the current source to produce a stage1 compiler. Stage1 is therefore the first compiler containing your source changes. The process can then use stage1 to rebuild the compiler and libraries, producing stage2 artifacts. Rebuilding checks self-hosting and moves toward release-like output.

“Stage” is not simply an optimization level or stability channel. It describes ancestry in the bootstrap sequence. Different bootstrap steps may reuse or copy artifacts, so the conceptual model is clearer than guessing paths from stage names.

Why not compile rustc with any stable compiler? The compiler source uses internal features and follows a controlled beta-based bootstrap policy. Stage0's exact version is pinned by the repository. This maintains reproducibility and supports the language's release cadence. The exact stage0 contents and arrangement of compiler and library artifacts have changed over time, so verify them against the current bootstrap documentation.

Bootstrapping creates subtle failures. A change may compile under stage1 but fail when stage1 compiles the same source for stage2. A library can be built by one stage and linked with assumptions from another. An environment can accidentally invoke rustup's default rustc rather than the local compiler. Always identify the executable and sysroot being used.

Bootstrap is implemented by the x tool and src/bootstrap. Read https://rustc-dev-guide.rust-lang.org/building/bootstrapping/intro.html for the current detailed model. Do not hand-assemble stage commands as a substitute for understanding x. The orchestrator carries target flags, snapshots, library dependencies, and policy that raw cargo commands omit.

10. Build, host, and target triples#

A target triple is a structured platform name such as x86_64-unknown-linux-gnu. Despite the historical word “triple,” names may encode architecture, vendor, operating system, and environment with varying components. Rust uses them to select data layout, ABI, linker behavior, conditional compilation, and libraries.

The build triple is the platform on which bootstrap itself runs. The host triple is a platform on which the produced compiler runs. The target triple is the platform for which that compiler emits user programs. In an ordinary native build all three are equal, which hides the distinction.

In cross compilation, a compiler running on Linux x86-64 might emit code for ARM Linux or WebAssembly. That target needs an appropriate standard library and often an external linker or SDK. Running target binaries may require hardware, an emulator, or a runner. A compile-only success is not a run-time success.

A cross-host compiler is harder: bootstrap running on one machine creates rustc intended to run on another. Do not volunteer cross-host claims from a target-only test. Use the project's current supported builders and ask target maintainers about validation.

Paths often contain the host triple. Instead of copying a path from another person's machine, inspect build output or ask x for a command that runs the right artifact. On Windows, executable suffixes, path quoting, debuggers, and C toolchains differ. On macOS, SDK and code-signing details can matter.

Target-dependent tests should declare requirements or ignore conditions using the suite's current directives. Do not make an assertion vague merely to pass every target. Separate portable semantic assertions from architecture-specific bytes, symbols, or messages. Check neighboring tests and compiletest documentation for accepted directives.

11. The x bootstrap interface#

The root x program is the supported human interface to bootstrap. On Unix-like shells it is commonly invoked as ./x; platform forms differ. It dispatches check, build, test, run, doc, fmt, and other tasks through the repository's dependency graph.

The current how-to-build-and-run chapter is https://rustc-dev-guide.rust-lang.org/building/how-to-build-and-run.html. Read it before copying commands because selectors and defaults evolve. Use help locally.

./x --help
./x check --help
./x test --help

A selector narrows a command to a path, crate, or suite understood by bootstrap. Exact accepted selectors are discoverable from help and errors. The safest practical habit is to begin with a documented broad command, then copy the narrower selector shown in current docs or neighboring contributor notes.

./x check performs fast type checking without producing all final machine-code artifacts. It is valuable for edit loops but cannot reveal linker, code generation, run-time, or stage2 failures. ./x build produces requested artifacts and dependencies. ./x test invokes the appropriate harness rather than treating all tests as Cargo tests.

./x run runs repository tools through bootstrap with the right build context. It can be preferable to invoking a tool binary by a guessed path. The available tools and argument separator behavior are version-sensitive; inspect ./x run --help.

Bootstrap prints the commands and paths it chooses. Read those lines. When reporting a failure, include the top-level x invocation and relevant configuration, not only a deep cargo command that bootstrap happened to execute.

Avoid scripts that depend on internal build-directory layout unless the task specifically maintains that interface. Paths are implementation details more often than x commands are.

12. bootstrap.toml and profiles#

bootstrap.toml is the local build configuration file recognized by x. It controls choices such as build profile, targets, LLVM strategy, debug information, assertions, and artifact reuse. The exact schema belongs to the checkout's bootstrap documentation and example configuration.

A profile is a named bundle of sensible settings for a use case. Compiler development, library development, tools, and distribution builds have different costs. Use a current recommended profile as a base rather than accumulating copied options from old posts. Comment unusual choices so future you understands why a build differs from CI.

Configuration changes can invalidate caches or alter semantics. Enabling assertions may catch invariant violations but slow the compiler. Adding debuginfo helps LLDB or GDB resolve source lines but increases build and link cost. Changing optimization can make a bug disappear or a benchmark incomparable.

download-rustc is a bootstrap facility that can reuse a previously built compiler artifact rather than locally compiling all compiler prerequisites. It can make many contribution loops dramatically faster. It also introduces an ancestry constraint: local changes that affect an artifact assumed downloadable may require rebuilding rather than reuse. Follow the current guide's compatibility and setup instructions at https://rustc-dev-guide.rust-lang.org/building/how-to-build-and-run.html.

Downloaded LLVM similarly avoids building the large LLVM backend locally when the available artifact matches. LLVM is the code-generation infrastructure used by the primary backend. If working inside LLVM integration, target support, or certain codegen paths, a local LLVM build may be necessary.

Keep experimental configurations separate or recorded. When comparing behavior, verify both worktrees use equivalent bootstrap.toml files. Never commit machine-local paths or accidental configuration unless the project explicitly requests a configuration change.

If x reports an unknown key, remove the stale advice rather than searching for a way to force it. Bootstrap's current configuration parser is the authority.

13. Stage0, stage1, and stage2 in practice#

Most compiler implementation testing uses stage1 because it contains the local compiler changes without paying for another full self-build. Stage0 does not contain those changes. Stage2 offers stronger release-like self-hosting confidence at higher cost.

For a local file, confirm the compiler identity.

./build/HOST/stage1/bin/rustc --version --verbose
./build/HOST/stage1/bin/rustc sample.rs

HOST is conceptual here. Use the actual host directory printed by bootstrap, not the literal word. Direct paths are useful for debugging, but current x run facilities may be more robust.

A compiler executable needs a sysroot: the root directory containing its standard libraries and related target artifacts. Mixing a stage1 rustc with an unrelated sysroot causes missing-crate, metadata, or ABI errors. Prefer bootstrap-provided invocation or explicitly inspect --print sysroot.

path/to/stage1/bin/rustc --print sysroot
path/to/stage1/bin/rustc -vV

rustup toolchain link can register a local built sysroot under a name, making cargo +name convenient. Follow the current dev-guide instructions because the directory to link and completeness requirements can change. A linked toolchain is a pointer to local artifacts; rebuilding or deleting them changes it.

rustup toolchain link rustc-stage1 path/to/stage1-sysroot
rustc +rustc-stage1 --version --verbose

Do not distribute such a local toolchain as though it were an official release. Do not infer stage2 correctness from stage1 alone. Conversely, do not rebuild stage2 after every keystroke. Use the cheapest stage that can falsify the current hypothesis, then increase confidence before review.

14. Fast compiler and library loops#

The fastest loop is selected by the changed dependency boundary. For compiler Rust code, begin with a narrow ./x check selector documented for the affected crate or compiler library group. Then build the compiler and run a focused test. For standard-library code, build or test the affected library without rebuilding unrelated compiler crates when bootstrap permits it.

Representative intent, whose exact selectors must be checked locally, is:

./x check compiler/rustc_parse
./x build compiler/rustc
./x build library
./x test tests/ui/parser

These commands illustrate narrowing by repository path. If 1.97-era help chooses a different selector, use the checkout's spelling. Do not preserve a brittle command merely because it appears in a textbook.

A parser implementation edit requires a compiler rebuild but usually not a new LLVM build. A change only to a UI test requires no compiler source rebuild if a suitable compiler already exists. A core or std implementation edit requires rebuilding the affected library and dependents, but not necessarily rustc itself. A compiler data-structure change can fan out through many rustc crates.

Run the produced stage1 compiler on a tiny scratch program when interactive exploration is faster than compiletest. Convert any lasting claim into a repository test afterward. Scratch commands are evidence for you; checked tests are evidence for everyone.

Keep LLVM cached unless LLVM source, configuration, or incompatible bootstrap settings require rebuilding it. Repeated LLVM builds are a classic waste of contributor time. Likewise, preserve Cargo and bootstrap caches when diagnosing ordinary Rust code.

Do not use ./x clean routinely. Cleaning discards valuable artifacts, lengthens the next cycle, and can hide an invalidation bug. Remove the narrow artifact only after evidence points to corruption or configuration mismatch. Before deletion, save logs needed to report a bootstrap cache bug.

15. Choosing the right test layer#

A test should fail for the intended reason before the patch and pass afterward. This “test first” sequence proves that the test observes the bug. It does not require strict test-driven development for every refactor, but regressions need this before-and-after evidence.

Choose the narrowest suite representing the contract. Parser recovery and diagnostics usually belong in UI tests. Generated IR belongs in MIR-opt or codegen tests. Linker integration belongs in run-make. Stable library behavior may belong in library unit or documentation tests. Compiler process crashes need a regression test even when exact diagnostics are not the main point.

Tests have costs. An exact stderr snapshot catches accidental wording and span changes but needs maintenance. A run-pass test proves behavior but may hide diagnostic quality. An assembly pattern can detect optimization loss but varies by architecture and LLVM version. A broad integration test catches boundaries but is slower and harder to diagnose.

Avoid testing implementation trivia unless it is an intentional invariant. Do not weaken expected output until a wrong compiler passes. Do not overspecify unstable ordering, temporary paths, pointer values, or timing. Normalize nondeterministic text through supported harness features rather than shell post-processing.

Read several neighboring tests, including recent history. Their directives encode suite conventions that prose may lag. Run the single new test, its directory when feasible, and a broader relevant suite before review.

Cross-platform behavior requires explicit thought. If a test fundamentally needs Unix signals or a particular architecture, state that through current test directives. If the semantic behavior is portable, structure it so platform noise does not become the assertion.

16. UI compiletest fundamentals#

Compiletest is Rust's compiler test harness for source files and expected outcomes. The UI suite focuses on the user interface of compilation: diagnostics, acceptance, rejection, spans, suggestions, and crash freedom. “UI” here means compiler-facing textual behavior, not a graphical interface.

A typical test pairs name.rs with name.stderr. The Rust file triggers behavior. The stderr file records normalized expected diagnostics. Some tests use inline annotations that point at expected errors or warnings. Exact conventions and directive syntax are documented at https://rustc-dev-guide.rust-lang.org/tests/compiletest.html and in nearby tests.

Place a regression test in the most specific established directory. Give it a behavior-oriented name rather than only an issue number. Add a comment explaining nonobvious syntax or why a crash once occurred. Keep source minimal while retaining the semantic trigger.

Run a focused UI test through x using the current selector.

./x test tests/ui/some-area/example.rs

If the test fails, compare actual and expected output. An unexpected diagnostic may expose a real ordering or recovery issue, not merely a snapshot to update. Inspect every changed line before blessing.

Parser tests should assert recovery and avoid diagnostic cascades where possible. Type-checking tests should isolate the relevant mismatch. Trait tests should make solver assumptions visible. Borrow-check tests should distinguish primary errors from later consequences. Diagnostic fixes should test message, span, labels, and suggestions appropriate to the bug.

A passing UI test does not prove semantic correctness for accepted programs. Pair it with run-pass, codegen, or another suite when the contract includes execution or generated output.

17. Blessing, normalization, and revisions#

Blessing means updating expected output to match the compiler's current output. It is a convenience, not approval of the output. The current x test interface exposes a bless mode; check help for exact placement of flags.

./x test tests/ui/some-area/example.rs --bless
git diff -- tests/ui/some-area/example.stderr

Never bless a broad suite and commit blindly. Review wording, line numbers, carets, labels, notes, suggestions, and ordering. Confirm unrelated tests did not change because of environment or stale artifacts.

Normalization replaces unstable details with stable forms. Compiletest already normalizes many paths and platform differences. Suite directives can normalize additional output when justified. Over-normalization is dangerous: replacing every number could hide a wrong line, type size, or error count. Normalize only the irrelevant source of variation.

Revisions run one source file under multiple named configurations. They can compare editions, feature flags, solver modes, or compile flags without duplicating most source. Revision-specific expected stderr may be needed. Use revisions when cases share one conceptual test; separate files when combined directives become harder to understand than duplication.

Known-bug tests document currently accepted failures or unsound behavior under the harness's current convention. Crash tests ensure malformed or edge-case input does not cause an internal compiler error. Do not turn an ordinary wrong diagnostic into a permanent known-bug excuse. Link the issue and make the expected status clear.

Expected output changes across targets need supported target conditions or separate files. Do not edit stderr on one platform and assume every builder agrees. CI diversity is a feature: it reveals assumptions unavailable on a developer laptop.

18. Run-pass, run-fail, and run-make#

A run-pass test compiles a program and expects its execution to succeed. It validates behavior that compile-only testing cannot see. Assertions inside the program should make wrong results fail clearly. Avoid network access, wall-clock assumptions, and uncontrolled randomness.

A run-fail concept expects execution to fail in a specified way, such as a panic. Repository suite organization changes over time, and some historical modes have migrated into UI arrangements. Look for the current neighboring pattern rather than assuming an old directory still exists. Distinguish an expected program failure from a compiler crash.

run-make tests are small project-like integration tests driven by compiletest. Each current test is an rmake.rs Rust program that uses the run_make_support library. Older Makefile-based run-make infrastructure has been retired. They are appropriate when a test needs multiple compilation steps, custom linker behavior, object inspection, environment setup, foreign-language code, or rustc invocations that one .rs file cannot express. Their power carries maintenance cost.

A run-make test should be deterministic and narrowly scoped. Use harness helpers instead of hard-coded compiler paths. Declare target tools and conditions. Avoid shell-specific assumptions; use run_make_support helpers unless a test deliberately targets shell behavior.

Ask whether a UI auxiliary crate is enough before choosing run-make. Auxiliary crates can test cross-crate behavior with less machinery. Choose run-make only when process-level orchestration is part of the contract.

Execution tests cannot run normally for every cross target. Some builders only compile them; others use runners or emulators. Mark requirements through current compiletest directives and never interpret “not run” as “passed at run time.”

When execution crashes, capture exit status, signal where meaningful, and normalized output. Avoid asserting an operating system's exact crash prose unless that prose is the feature under test.

19. Codegen, assembly, and MIR-opt tests#

Codegen tests inspect LLVM IR or other generated representations for patterns. They verify properties such as attributes, calling conventions, vectorization opportunities, bounds-check elimination, and symbol linkage. LLVM IR is an intermediate representation consumed by the primary backend.

Assembly tests inspect final machine instructions. They can catch missed optimizations or wrong lowering that IR checks cannot. Assembly is architecture-, feature-, and backend-sensitive. Assertions should establish the property without freezing irrelevant register allocation or instruction scheduling.

MIR is rustc's Mid-level Intermediate Representation. MIR-opt tests snapshot selected MIR before or after passes. They are useful for transformation correctness and optimization regressions. Generated diffs can be large; review control-flow, locals, and pass phase rather than blessing mechanically.

Each of these suites can become brittle. A CHECK pattern that merely matches one favored instruction may fail after an equivalent LLVM improvement. A loose pattern may pass despite extra expensive work. Explain the performance or correctness invariant in comments and use the suite's current matching syntax.

Code generation tests usually need optimization and target settings chosen deliberately. A result at opt-level zero says little about optimized production code. CPU feature flags can change instruction selection. Do not generalize one x86-64 result to all architectures.

When changing a backend-facing subsystem, test semantic execution separately if feasible. Pattern tests show shape, while run tests show outcome. Neither alone proves absence of undefined behavior.

Inspect current chapters under https://rustc-dev-guide.rust-lang.org/tests/ for precise commands, bless behavior, and target directives. These suites evolve alongside compiler internals.

20. Incremental, rustdoc, unit, and tidy tests#

Incremental compilation reuses results from an earlier compilation. Its dependency graph decides what must be recomputed after a source change. An incorrect “green” result can reuse stale data; an overly conservative result is correct but slow. Incremental tests exercise both correctness and invalidation behavior across revisions.

Design an incremental test around an edit sequence. State which item changes, which queries should rerun, and what externally observable result must change. Do not infer correctness only from a cache hit count. The development guide's incremental debugging material explains dependency-graph tools and version-sensitive internal attributes.

Rustdoc tests cover generated documentation, links, search, doctests, and rendering behavior. Rustdoc is a tool with compiler integration but distinct ownership. A parser or type-system change may affect doctest compilation even if HTML stays unchanged. Use rustdoc's established suites and avoid hand-comparing giant generated pages.

Unit tests exercise functions or data structures in a crate through Rust's test harness. They are fast and precise when the logic has a stable local boundary. Many compiler behaviors are query- and session-dependent, making a compiletest regression more representative than elaborate mocking. Use both when local algorithm correctness and end-to-end behavior matter.

Tidy is a repository policy checker. It enforces formatting-adjacent rules, licensing and dependency policies, test conventions, forbidden patterns, and generated-file consistency. Its checks change as policy changes. Run the focused tidy or recommended pre-PR command from current contribution docs.

A tidy failure is not “just CI.” Read its explanation and repair the underlying policy violation. Do not add an exception merely to silence it. If the policy is wrong for the case, discuss changing policy explicitly with owners.

21. Regression tests and a minimal complete example#

A regression test prevents a specific old failure from returning. It should preserve the essential trigger and omit accidental details. MCVE means minimal, complete, verifiable example: minimal enough to understand, complete enough to run, and verifiable with exact versions and commands.

Start from the reporter's crate but remove dependencies and modules cautiously. Replace external types with small local definitions. Delete functions, bounds, fields, and expressions one at a time. After each deletion, verify the same failure remains, not merely any failure. For diagnostics, compare the primary message and span. For an ICE, compare the panic location or query stack when available.

Reduction has traps. Removing code can move failure to parsing before the type checker. Inlining can eliminate a cross-crate metadata bug. Changing names can alter macro hygiene. Removing optimization can hide codegen failure. Record the defining characteristic of “same bug” before reducing.

creduce is an automated reducer originally designed around C-like text but adaptable with an interestingness script. Perses is a syntax-aware program reducer. An interestingness script returns success only when the candidate still exhibits the target behavior. Reducers save labor but can produce bizarre examples; manually clean and understand the result.

A strong regression test often includes a short comment linking the issue and stating the invariant. Do not include proprietary original code when a synthetic reproducer suffices. Obtain permission before publishing source from a private report.

Test the minimized case against the known good and bad compilers. If both fail differently, refine the oracle. An oracle is the rule deciding whether a candidate reproduces the bug.

22. Bisecting versions and identifying the responsible PR#

Bisecting repeatedly divides a version range to find the first bad version. First establish the newest known good and oldest known bad compiler. Nightly archives make dates useful, but a date boundary may contain multiple merged pull requests.

cargo-bisect-rustc automates downloading compiler artifacts and running a test command. Its tutorial and current limitations are linked from https://rustc-dev-guide.rust-lang.org/compiler-debugging.html. Provide a deterministic command whose exit status distinguishes good from bad. Pin dependencies so ecosystem updates do not move during the search.

cargo bisect-rustc --start GOOD-DATE --end BAD-DATE --script ./is-regressed.sh

GOOD-DATE and BAD-DATE describe values to supply after reading current tool help, not literal dates. The command-line interface is version-sensitive. Never report a culprit without opening the resulting PR and validating nearby artifacts when possible.

Artifacts may be unavailable, broken for unrelated reasons, or affected by host changes. A non-monotonic bug can disappear and return, violating binary search assumptions. Flaky tests produce false boundaries. Run candidates repeatedly and inspect output, not only exit status.

After finding a likely PR, explain whether it caused, exposed, or merely coincided with the bug. A soundness check can expose invalid existing code without introducing the underlying unsoundness. A diagnostics refactor can reveal a latent panic. Words matter because reverting the exposing change may be the wrong fix.

Link the bisect result in the issue and notify relevant experts politely. Do not assign blame to the author. Regressions are a system outcome, and the original patch may have been reasonable under available tests.

23. Internal compiler errors and query stacks#

An internal compiler error, commonly ICE, means rustc violated an internal expectation and aborted rather than issuing a normal user diagnostic. ICE output often contains the compiler version, panic message, location, query stack, and instructions for reporting. Preserve this information while removing private paths and source.

First reproduce with the same toolchain and flags. Then try the latest nightly because the ICE may already be fixed. Check whether incremental artifacts are involved by reproducing in a fresh temporary target directory, but do not globally clean the rust checkout. Confirm whether the source is accepted Rust, rejected Rust, or uses unstable features; rustc should generally avoid ICEs in all cases.

A query is a memoized compiler computation such as obtaining an item's type or MIR. The query stack shows computations active near the panic. It is a lead, not a complete causal trace. The top query may simply request data corrupted earlier.

Enable backtraces for local reproduction.

RUST_BACKTRACE=1 path/to/stage1/rustc reduced.rs
RUST_BACKTRACE=full path/to/stage1/rustc reduced.rs

Backtrace quality depends on debug information and optimization. “Full” can be noisy. Look for the first relevant rustc frame around the panic and follow invariants backward.

A robust ICE fix handles the invalid or unusual state at the correct abstraction boundary. Replacing unwrap with an arbitrary default may turn a visible crash into silent miscompilation. Determine whether the state is impossible, user-caused, delayed error recovery, or a previously valid new case. Add a minimized crash regression and, when appropriate, assert the corrected diagnostic or behavior.

24. Tracing with RUSTC_LOG#

rustc uses structured tracing instrumentation in many subsystems. A trace event is a diagnostic record emitted by compiler implementation code, not a user-facing compiler diagnostic. RUSTC_LOG selects modules and verbosity through a filter.

RUSTC_LOG=rustc_parse=debug path/to/stage1/rustc reduced.rs
RUSTC_LOG=rustc_trait_selection=trace path/to/stage1/rustc reduced.rs

Module paths and available levels change. Consult https://rustc-dev-guide.rust-lang.org/tracing.html and source instrumentation in the checkout. If the build's maximum tracing level excludes verbose events, changing the environment alone cannot restore them; follow current build guidance.

Start narrow. Global trace output can be enormous, slow compilation, and change timing. Filter by crate module, item, or span using supported facilities where available. Capture output to a file, then search around the first divergence between good and bad cases.

Tracing is strongest when comparing two nearly identical compilations. Label logs with compiler commit and command. Normalize nondeterministic identifiers cautiously. Do not conclude that the last event before a panic caused it; buffered output and unwinding can reorder perception.

Add temporary tracing locally when existing events do not answer the question. Do not commit noisy logging without considering cost and usefulness to future debugging. Structured fields are easier to filter than a large formatted debug dump.

Logs may contain source fragments, paths, crate names, and macro expansions. Review them before sharing. For secret code, create a synthetic reproducer instead of publishing a raw trace.

25. Dumps and unstable diagnostic flags#

The -Z namespace contains unstable compiler options available on nightly or development compilers. They are implementation tools, not stable interfaces. Run rustc -Z help on the exact compiler and consult the nightly rustc book. Names, accepted values, and output formats can change in the 1.97.1 era and beyond.

-Zunpretty can print selected representations, including expanded or lowered forms depending on current options. It helps answer whether a bug arose before or after macro expansion or lowering. -Zdump-mir emits MIR around selected passes. -Ztrace-macros can expose macro expansion activity, though modern macro diagnostics and expansion tools may be more useful for some cases.

path/to/stage1/rustc -Zunpretty=expanded reduced.rs
path/to/stage1/rustc -Zdump-mir=all reduced.rs
path/to/stage1/rustc -Ztrace-macros reduced.rs

Treat these as illustrative and check -Z help first. Dumping all MIR creates many files and can overwhelm an investigation. Select an item or pass when current syntax permits. Store dumps outside tracked source paths or remove them before committing.

A dump is a snapshot of internal state. It tells what happened, not necessarily why. Compare a good and bad build at successive compiler phases to locate the earliest meaningful divergence. Earlier divergence often narrows ownership better than the final wrong assembly.

Do not tell production crate users to set RUSTC_BOOTSTRAP simply to access these flags. Use a nightly toolchain for voluntary experimentation or a compiler-development build inside rustc's supported workflow. Depending on unstable output in a production build script creates fragile and unsupported coupling.

26. Native debuggers and crash investigation#

GDB and LLDB are native debuggers that can pause a process, set breakpoints, inspect stack frames, and examine variables. They are useful when tracing cannot reveal control flow or a native crash crosses Rust, LLVM, linker, or operating-system boundaries.

Build relevant artifacts with enough debug information according to current bootstrap documentation. Higher debuginfo improves symbol and line resolution but increases disk use and link time. Compiler optimization can inline or remove variables, so debugger views may not match source literally.

Launch the actual stage1 rustc and pass a minimized source file.

lldb -- path/to/stage1/rustc reduced.rs
gdb --args path/to/stage1/rustc reduced.rs

These are debugger invocations, not universal platform instructions. On macOS or containers, permissions and code-signing can block attachment. On Windows, use the supported debugger and symbol format described by current guide material.

Break at the panic location or a narrow subsystem function. Inspect the call stack and arguments. Conditional breakpoints prevent stopping thousands of times, but evaluating them can be slow. Watchpoints detect memory changes but are scarce and platform-limited.

Rust values may display imperfectly through native debuggers. Use source-level invariants and strategic logging together with debugger state. If the crash occurs in LLVM, determine whether rustc supplied invalid IR or LLVM mishandled valid IR before assigning ownership.

Record a textual backtrace and exact binary commit for reports. Core dumps can contain source and environment secrets; handle them as sensitive artifacts.

27. Self-profile and measureme concepts#

-Zself-profile records rustc's own activity as timestamped events. This is different from profiling the program produced by rustc. It can expose time spent in queries, code generation, optimization, and other labeled work.

path/to/nightly-or-stage1/rustc -Zself-profile=profile-data crate.rs

The flag and tools are nightly and version-sensitive. Read https://rustc-dev-guide.rust-lang.org/profiling.html and current -Z help. Use a release-like compiler when studying realistic compile time; debug compilers distort costs.

measureme is the ecosystem of libraries and tools underlying rustc self-profile data. It records events efficiently and supports summaries and timeline conversion. crox converts suitable profile data into a format viewable in Chromium's tracing interface. Tool names and invocation details may evolve, so follow the measureme repository and current guide rather than memorizing one command.

Query time needs interpretation. A query can be expensive because its own algorithm is slow, because it triggers child queries, or because it reruns too often. Inclusive time includes descendants; self time excludes them. Invocation count and cache-hit behavior can matter more than one call's duration.

Profiles add overhead and generate data. Compare like with like: same compiler settings, crate revision, incremental state, machine load, and number of runs. Use profiles to form hypotheses, then validate with stable benchmark measurements.

A beautiful trace does not establish ecosystem impact. It explains one workload. Keep the original command and source available so reviewers can reproduce the interpretation.

28. Hardware counters, wall time, and memory#

Wall time is elapsed human-visible time. It includes CPU work, scheduling, filesystem activity, page faults, thermal effects, and contention. One faster run is weak evidence. Warm up, repeat, alternate baseline and patch, and report variability.

Instructions retired approximate executed machine instructions. CPU cycles measure processor work but vary with stalls and frequency behavior. Instructions per cycle can indicate utilization, though different work can change the ratio. Cache misses suggest data was not found in a fast cache, but hardware events and sampling support differ by CPU and operating system.

Linux perf and analogous platform tools can sample stacks and count hardware events. Symbols require suitable debuginfo. Virtual machines and containers may restrict counters. Sampling attributes cost statistically; it can miss short functions or misrepresent inlined frames.

Memory profiling asks several questions. Peak resident set size measures the maximum physical memory mapped into the process approximately. Allocation profilers show allocation volume and live objects. Heap, mappings, LLVM allocations, and child processes require different tools. A lower allocation count does not guarantee a lower peak.

Measure memory with a realistic crate and fixed settings. Incremental and parallel compilation change peaks. Report whether numbers include linker and proc-macro processes. Avoid running unrelated jobs on the machine.

Interpret counter changes through an algorithmic story. Fewer instructions with unchanged wall time may be hidden by I/O. Lower cache misses with more total work may still regress. Never promise a compiler-wide performance win from one microbenchmark.

29. rustc-perf and benchmark discipline#

rustc-perf is the official compiler performance benchmark infrastructure at https://github.com/rust-lang/rustc-perf. Hosted results appear at https://perf.rust-lang.org/. It measures a corpus of real crates under configurations such as clean and incremental builds. The current test documentation is https://rustc-dev-guide.rust-lang.org/tests/perf.html.

A benchmark comparison needs a baseline commit and patch commit built comparably. The service reports statistics such as instruction counts and may provide profiles. Bot commands, permissions, and queue procedures evolve. Ask the assigned reviewer or compiler performance channel for a run rather than guessing commands from an old PR.

Compile-time benchmark discipline includes four rules. First, define the hypothesis before examining dozens of metrics. Second, use workloads representative of the affected path. Third, preserve correctness and diagnostic quality. Fourth, inspect regressions as carefully as improvements.

Microbenchmarks remain useful for isolating algorithmic scaling. Vary input size to distinguish constant overhead from quadratic growth. Keep generators and raw results available. Then test broader rustc-perf workloads because real crates exercise caching, metadata, macros, and parallel activity differently.

Never add a crate-name check, syntax fingerprint, or benchmark-specific fast path. That is a test-specific hack: it optimizes the scoreboard rather than the compiler's general algorithm. Generalize the invariant and add correctness coverage.

Small movements can be noise. Large aggregate wins can hide a serious regression in one important scenario. Read per-benchmark and per-profile results. Explain expected winners and losers in the pull request. Performance evidence supports review; it does not replace design review.

30. Crater, compatibility, and soundness#

Crater compiles and tests a large selection of ecosystem crates with two compiler versions. It estimates real-world breakage from a change. The official guide is https://rustc-dev-guide.rust-lang.org/tests/crater.html, with infrastructure details linked from Rust project resources.

A crater run produces regressions, improvements, and failures unrelated to the patch. Regressions require triage. Group duplicate root causes, inspect logs, reproduce locally where possible, and identify crates relying on intended or accidental behavior. Network failures, native dependencies, flaky tests, and platform assumptions create noise.

Compatibility is not a single yes-or-no rule. Stable source compatibility, inference changes, diagnostics, linker behavior, proc-macro token behavior, and performance can affect users differently. A bug fix can break code that relied on a compiler bug. Language and compiler teams evaluate whether breakage is acceptable and whether migration, linting, or a transition period is needed.

Soundness means safe Rust cannot cause undefined behavior through the claimed guarantee. Fixing unsoundness may justify compatibility breakage, but the response still needs impact analysis. Crater can estimate visible source breakage; it cannot prove soundness or find every private codebase.

Request crater when the change has plausible ecosystem impact, guided by reviewers and current procedure. Runs consume shared infrastructure, so formulate the experiment and avoid redundant requests. Do not interpret zero crater regressions as proof that a semantic change is harmless.

Document categorized results in the tracking issue or pull request. For affected maintained crates, communicate respectfully and avoid describing valid historical code as careless.

31. The complete contribution lifecycle#

Reproduce the report on the stated version and current default branch. Search issues, pull requests, Zulip, and history. Ask a focused question when intent or ownership remains uncertain. Claim the issue if current project procedure supports claiming.

Write a short design note before editing. State the invariant, likely subsystem, alternatives, and risks. For a small fix this may be an issue comment; for broad semantics it may need team process. Do not assume every pull request needs an RFC.

Add or prepare the smallest regression test and show it fails correctly. Make the smallest coherent patch. Avoid drive-by renames, formatting, dependency updates, and speculative refactors. Run focused checks after each meaningful change.

Format only relevant files through the repository's current x formatting command. Run focused tests, tidy, and broader checks proportionate to impact. Review git diff and git status from top to bottom. Remove debug prints and generated dumps.

The pull-request description should include problem, root cause, solution, alternatives, tests, and risk. Link the issue with the repository's accepted closing syntax only if the PR fully resolves it. Include performance or crater evidence when relevant. Say what you could not test.

Reviewer feedback is part of the work. Answer questions directly, push focused revisions, and avoid rewriting history in a way that makes active review unnecessarily difficult unless requested. When disagreeing, provide evidence and restate the tradeoff.

After approval, bors or its current successor tests the merge candidate through CI. Rollups combine eligible approved PRs to amortize long CI cycles; they trade throughput against harder failure attribution. Read current Rust Forge merge and rollup policy at https://forge.rust-lang.org/release/rollups.html. Do not promise merge timing.

After landing, watch automation, performance results, and issue reports. Submit a focused follow-up for discovered work rather than hiding scope in the merged claim.

32. Diagnostics and parser fixes#

For a diagnostics contribution, begin with a real confusing case. Identify the primary error, user intent, best span, and whether a machine-applicable suggestion is always correct. Applicability is the compiler's confidence that applying a suggestion mechanically is safe. Overconfident suggestions damage tools and trust.

Find the diagnostic definition by searching message text or localization key. Trace where spans and type information originate. Prefer structured diagnostic data over reparsing rendered strings. Test positive and negative neighbors so the new heuristic does not trigger on unrelated code.

For an ICE fix, minimize the crash, inspect panic and query stack, and identify the violated invariant. Do not merely replace a panic with “delay bug” or a dummy type without understanding downstream effects. Choose a normal diagnostic, graceful recovery, or restored invariant as appropriate. Test crash freedom and meaningful output.

For a parser fix, determine tokenization, grammar, recovery, and edition interactions. The lexer turns characters into tokens; the parser forms syntax structures; recovery continues after malformed input to produce useful diagnostics. A recovery change can create cascades or reinterpret valid macro tokens.

Add tests for valid syntax, the malformed reproducer, nearby ambiguous forms, macro context when relevant, and editions affected. If language grammar changes rather than implementation matching established grammar, seek language-team direction. A parser patch is not automatically “just diagnostics.”

Review suggestion spans at character boundaries and with Unicode input. Test multipart suggestions when edits touch separate spans. Avoid embedding unstable debug text in user-facing messages.

A good diagnostic teaches the correction without guessing intent recklessly. Conciseness, precision, and consistency with neighboring diagnostics matter more than clever wording.

33. Type, trait, borrow, lint, and performance fixes#

A type-checking fix starts by locating the first incorrect type relation, not the final error emission. Record expected and inferred types at a narrow boundary. Check coercions, inference variables, normalization, and error-recovery placeholders. A local diagnostic patch can hide a semantic bug if the relation itself is wrong.

A trait-system fix needs explicit assumptions about goals, candidates, coherence, normalization, and solver mode. A goal is a proposition the solver tries to prove, such as a type implementing a trait. Test ambiguous, proven, and disproven neighbors. Cross-crate and associated-type cases often expose caching or canonicalization mistakes.

A borrow-check fix should follow ownership facts through MIR. Distinguish move analysis, region inference, place overlap, drop checking, and diagnostic explanation. Test accepted and rejected cases. If acceptance changes, consider whether the program is actually sound and whether edition or feature gates matter.

A new lint is a policy feature, not merely a pattern match. Define what it detects, default level, false positives, future compatibility role, macro behavior, and fix applicability. Stable lint naming and groups are user-facing API considerations. Follow current compiler and language lint process; broader lints may require design approval.

A performance fix starts with a profile and complexity hypothesis. Change a general algorithm or data representation, preserve invariants, add correctness tests, and measure representative workloads. Check compile time, memory, incremental behavior, and output quality as relevant. Request rustc-perf evidence for plausible broad impact.

For every category, inspect owners and recent experts, keep the patch narrow, and document why the chosen layer is correct. Expertise is demonstrated by boundaries and evidence, not by diff size.

34. Unsafe code, soundness, and security boundaries#

Unsafe Rust permits operations the compiler cannot fully verify. It does not disable all rules. The author must uphold documented safety invariants so safe callers cannot trigger undefined behavior. Compiler and library unsafe code deserves especially explicit review because impact is broad.

Review each unsafe block by stating its proof obligation. Check pointer provenance, alignment, initialization, aliasing, lifetimes, thread safety, panic paths, layout assumptions, and drop behavior. Minimize the unsafe surface and expose a safe abstraction only when its contract can be enforced. Tests can find bugs but cannot prove absence of undefined behavior.

For standard-library changes, run relevant interpreter or sanitizer tooling when current project guidance recommends it. Miri can detect classes of undefined behavior by interpreting Rust's intermediate representation, but it is not a complete proof and does not model every platform effect.

A soundness issue is not automatically an embargoed security vulnerability, and a security vulnerability is not merely a public soundness bug with a scarier label. Potentially exploitable, confidential, supply-chain, or release-critical reports must follow Rust's current security policy at https://www.rust-lang.org/policies/security. Do not open a public issue containing a working exploit before the security team assesses it.

Responsible disclosure means sharing enough detail through the designated private channel, preserving confidentiality, and coordinating publication and fixes. Do not promise bounty, severity, timeline, or CVE assignment on the project's behalf. If uncertain whether a boundary is crossed, use the private security route and let the team redirect it.

After public resolution, write tests and rationale without preserving unnecessary weaponized detail. Respect embargoes even when a fix appears in a branch.

35. A sample investigation from report to pull request#

Imagine a public report: a current nightly ICEs when an invalid associated type appears inside an async function, while stable emits a normal trait error. This is a teaching investigation, not a claim about a live rustc bug and not a pretend patch.

First capture rustc --version --verbose, the command, complete ICE text, and whether incremental compilation is enabled. Reproduce on the reported nightly and latest nightly in a temporary directory. Search the exact panic text, associated query name, and syntax shape in issues and merged pull requests.

Reduce the program. Remove Cargo dependencies, executor code, function bodies, generic bounds, and unrelated associated items one at a time. Keep async only if removing it changes the panic into an ordinary error. The final MCVE might contain one trait, one malformed projection, and one async function. Verify the panic location remains the same.

Use cargo-bisect-rustc with a deterministic script to find a likely nightly boundary. Open the candidate PR and inspect whether it changed trait normalization, async lowering, or error recovery. Describe it as “likely exposed by” until causality is established.

Search the panic location in compiler source. Follow callers and query stack through rustc_trait_selection or the currently responsible crates. Compare RUSTC_LOG output for a synchronous non-crashing variant and async crashing variant. Inspect the earliest point where an error placeholder reaches code assuming a normalized projection.

Before editing, add a focused UI test under the neighboring trait or async diagnostics directory. The .rs file should be readable and the .stderr should expect a normal error, not an ICE. Run it against the unmodified stage1 and confirm failure by crash.

Discuss the invariant with the assigned expert. Possible fixes include preventing an invalid goal, propagating an error result, or making a recovery path tolerate the placeholder. Reject an arbitrary fallback type because it could mask later errors or miscompile accepted code.

Implement only the agreed boundary fix. Run the new test, nearby trait tests, relevant async UI tests, ./x check for affected crates, and tidy guidance. If solver modes or editions differ, add revisions.

Open a PR explaining reproduction, reduced case, first invalid state, restored invariant, tests, bisect evidence, and untested platforms. Respond to review and let current bors/CI procedure validate integration. No live patch is invented here because real code and ownership must be inspected at the current commit.

36. Exercises for deliberate practice#

Exercise one is repository orientation. Choose a diagnostic from tests/ui, find its message definition, emitting function, owning crate, three recent related PRs, and current likely reviewers. Write a one-page map without changing code.

Exercise two is bootstrap literacy. For your machine, record build, host, and one cross target. Explain which compiler builds stage1, which artifacts form its sysroot, and why stage2 costs more. Verify claims using x output and --version --verbose.

Exercise three is test selection. Classify ten existing tests among UI, run-pass concept, run-make, codegen, assembly, MIR-opt, incremental, rustdoc, unit, and tidy. For each, explain why a cheaper test would miss the contract and why a broader test might be wasteful.

Exercise four is reduction. Take an already public fixed ICE from history, check out an affected compiler artifact, and reduce its public reproducer. Write an interestingness script. Compare your result with the merged regression test without criticizing differences that preserve clarity.

Exercise five is diagnostic design. Find an issue labeled for diagnostics and draft expected stderr, including spans and suggestion applicability. List three false-positive guards before touching implementation.

Exercise six is performance analysis. Profile a small public crate with self-profile, identify one expensive query family, and formulate a hypothesis. Do not claim a win. Describe the rustc-perf evidence needed to validate it.

Exercise seven is review practice. Read a merged unsafe standard-library PR and reconstruct each safety obligation, test limitation, and reviewer concern. Compare final rationale with the first revision.

Exercise eight is communication. Draft a Zulip help request with commit, platform, command, minimal error, attempted steps, and one precise question. Then cut its length by one third without deleting evidence.

37. A 30, 60, and 90 day plan#

During days one through thirty, optimize for orientation and finished small work. Build a recommended profile, run stage1, learn one UI suite, and read contribution procedures. Triage or reproduce several public issues. Submit documentation or test improvements where evidence is clear. Join Zulip, observe topic etiquette, and ask one well-formed question.

Keep a learning log of subsystem vocabulary and commands verified against the checkout. Read two merged PRs each week from one area. Do not measure progress by lines changed. Measure whether another person can reproduce your observations.

During days thirty-one through sixty, own one bounded bug lifecycle. Claim an issue, reduce it, bisect if appropriate, discuss the invariant, submit a tested fix, and respond to review. Learn tracing and one representation dump. Review small tests or documentation when invited, clearly stating your review scope.

During days sixty-one through ninety, deepen one subsystem and broaden one boundary. For example, deepen trait diagnostics while learning how incremental tests cover query caching. Attempt a measured performance investigation or crater-result triage with mentorship. Write a short technical explanation that links source, tests, and design history.

The plan is not a promotion ladder. Health, employment, time zones, and review latency vary. Consistency and honest handoffs matter more than calendar speed. If a task stalls, post findings and release the claim.

By day ninety, aim to be predictable: small diffs, clear evidence, prompt responses, respectful disagreement, and no hidden uncertainty. That is the foundation of trusted production contribution.

38. Portfolio evidence, talks, and long-term reliability#

A strong portfolio shows decisions and outcomes, not a raw pull-request count. For each public contribution, record the report, reduced test, root-cause explanation, review changes, verification, and landed result. Link only public material and credit co-investigators and reviewers.

Resume evidence should be precise. “Minimized and fixed a parser recovery ICE with cross-edition UI coverage” is stronger than “worked on Rust compiler.” For performance work, report the official measured range and workloads, including regressions or uncertainty. Never inflate a microbenchmark into a compiler-wide percentage.

A technical talk needs a reproducible narrative. Begin with user impact, explain just enough compiler pipeline, show the minimal example, trace the investigation, and state alternatives rejected. Use a pinned commit and prerecorded backup for live demos. Remove private paths and secrets from screenshots.

Explain project process as part of engineering. Review assignment, CI, rollups, crater, and perf infrastructure are not bureaucracy surrounding the “real” code. They are mechanisms that make risky global infrastructure changeable by a distributed community.

Long-term reliability means returning after merge. Watch regressions, answer maintenance questions, update documentation when workflow changes, and help newcomers discover the evidence you once needed. Admit when an area is outside your expertise and route it to current owners.

Re-read official sources regularly: the rustc development guide, Rust Forge, nightly rustc API docs, rustc-perf documentation, current issue-label descriptions, triagebot instructions, Code of Conduct, and security policy. Links are version-sensitive, and commands in a 1.97.1-era chapter will age.

Expert status is not knowing every flag. It is the ability to find current truth, design a falsifiable investigation, protect compatibility and soundness, communicate tradeoffs, and leave the repository easier to maintain than you found it.

Part IX-B: Bootstrap, Validation, and the Path to Independent rustc Contribution#

39. Self-hosting without circular reasoning#

rustc is mostly written in Rust, so a fresh checkout cannot compile itself from source alone. Bootstrap breaks that apparent circle with a trusted, already-built compiler called stage0. For the Rust 1.97.1 development era, exact snapshot versions and commands remain checkout data. Inspect src/stage0 and ./x --help; do not infer them from this guide.

The smallest useful model has three compiler generations. Stage0 is downloaded and is never produced by the checkout being tested. Stage1 is source from the checkout compiled by stage0. Stage2 is the same checkout source compiled by stage1. The source may be identical while the compiler doing the compilation differs.

published snapshot --downloads--> stage0 compiler + stage0 std
stage0 rustc --compiles checkout--> stage1 rustc
stage1 rustc --compiles checkout--> stage2 rustc
stageN rustc --compiles library--> stageN library artifacts
(simplification: tools and cross-target libraries are omitted)

The central invariant is provenance: every artifact has a known compiler, source revision, host, target, profile, configuration, and dependency set. “It is stage1” is insufficient provenance. A stage1 standard library for one target is not interchangeable with stage1 rustc for another host.

Stage1 is normally the productive compiler-development loop. It contains the local compiler change and is much cheaper than rebuilding stage2. Stage2 checks that the locally built compiler can rebuild the system in the intended final shape. It is not automatically “more correct”; it answers a different question.

Bootstrapping does not prove compiler correctness. Two generations may reproduce the same bug. A successful stage2 build establishes a self-hosting and integration fact under one configuration. Semantic tests, differential checks, target builders, and review establish other facts.

Prediction: a parser edit is made, stage1 builds, and stage2 fails compiling rustc. What broke first? Possibilities include accepting syntax in stage0 but not stage1, a new warning becoming denied, metadata incompatibility, or latent code exercised only by the second generation. The stage number localizes provenance; it does not diagnose the cause.

Do not set RUSTC_BOOTSTRAP=1 globally to “fix” ordinary builds. Bootstrap deliberately controls unstable features for in-tree components. A global override erases the boundary being validated and contaminates unrelated Cargo builds.

40. Build, host, and target are three independent coordinates#

Build is the machine on which bootstrap itself executes. Host is a platform on which a produced compiler executable runs. Target is a platform for which that compiler emits programs or libraries. In a native build all three names may match, hiding the distinction.

build machine B
  | runs bootstrap and a compiler hosted on B
  +-- produces rustc(host=H) --------runs on--------> H
  |                                      |
  |                                      +-- emits code for target T
  +-- produces std(target=T) ------------links with emitted code
(each edge carries a platform requirement, not merely a filename)

The matrix can be read as a sentence: “On build platform B, construct a compiler that runs on H and produces code for T.” A Canadian cross build has distinct B, H, and T. It requires more than passing --target to Cargo.

Compiler artifacts are host artifacts because rustc must execute. Standard-library artifacts are target artifacts because user programs link them. Proc macros and build scripts execute on a host even while their enclosing crate targets T. That split is a frequent source of wrong-linker and wrong-architecture failures.

Tools divide by behavior. Cargo, rustdoc, rustfmt, Clippy, and linker wrappers run on a host. Their tested outputs can target another platform. Some tools are distributed only on supported hosts; bootstrap policy decides the set.

Write an artifact label before debugging cross compilation:

kind=std
stage=1
compiler-host=x86_64-unknown-linux-gnu
artifact-target=aarch64-unknown-linux-gnu
source=<commit>
config=<config digest>

If an executable reports Exec format error, suspect that a target artifact was run as a host tool. If linking reports incompatible object format, inspect each input's target. If core cannot be found, ask whether std/core was built for T, not whether rustc exists for H. If a proc macro cannot load, inspect its host ABI and dynamic-library search path.

The invariant is that execution edges use host-compatible artifacts and link edges use target-compatible artifacts. Moving compilation earlier may save work but requires preserving this coordinate information. Discarding it makes cache hits cheap and wrong.

41. What bootstrap actually constructs#

Bootstrap is a build orchestrator in src/bootstrap, entered through the repository's x command. It schedules Rust libraries, compiler crates, LLVM or another backend, documentation, tools, distribution components, and tests. It is not merely a wrapper around one Cargo invocation.

The standard-library family includes core, alloc, std, test, and target-dependent support. Dependency order matters: alloc relies on core, and std builds atop lower layers. Compiler crates form another graph ending in the driver and executable. Tools may consume both compiler-private APIs and libraries.

stage compiler
  +--> core(target) --> alloc(target) --> std(target) --> test(target)
  +--> rustc_* crates(host) --> rustc_driver(host) --> rustc(host)
  +--> rustdoc(host) --uses compiler crates and target libraries
  +--> compiletest(host) --drives compilers, programs, and comparison
(arrows mean “must exist before,” not necessarily Cargo dependencies)

A step is bootstrap's unit of requested work. Steps declare dependencies and produce artifacts or validation results. Selecting a path lets bootstrap infer a suitable step, but path-to-step mapping is policy. Use ./x build --help, ./x test --help, and verbose/dry-run facilities supported by the checkout.

Profiles provide coherent configuration defaults for use cases such as compiler development, library development, tools, or distribution. config.toml overrides details: downloaded components, hosts, targets, optimization, debug assertions, LLVM strategy, and more. Names and defaults are version-sensitive; start from config.example.toml or ./x setup.

Configuration is policy, not identity. Two textual configurations can lead to equivalent artifacts, and one configuration can produce many distinct artifacts. The build graph plus actual inputs defines identity.

Bootstrap caches expensive outputs under the configured build directory. Stamps summarize whether a step's prerequisite state is still valid. Cargo fingerprints and incremental state add lower-level caches. A stamp is evidence used by the scheduler, not proof that an artifact is semantically correct.

When source, environment, or config changes outside a tracked input, stale reuse is possible. Before reporting a compiler bug, rerun the narrow step verbosely. Then remove only the implicated output if needed. Deleting all build output is a diagnostic experiment of last resort, not a daily ritual.

42. Downloads, trust, cache hygiene, and reproducibility#

Bootstrap can download stage0 rustc and std, LLVM, rustfmt, and other snapshots selected by checkout metadata and configuration. The authoritative URLs, checksums, and component versions are in the checkout and Rust build infrastructure, not in a contributor's shell history.

A secure download pipeline needs authenticated transport, expected cryptographic digest, atomic installation, and a clear cache key. Transport encryption alone does not prove that bytes match the version selected by the source. An interrupted archive must never masquerade as a completed component.

checkout metadata --selects--> URL + expected digest
network --delivers--> temporary archive --verify--> content-addressed cache
verified archive --atomically installs--> stage0/sysroot
(trust ultimately includes repository metadata and release infrastructure)

Corporate proxies, mirrors, and offline caches alter the path but not the invariant: the consumed bytes must be the selected component. Do not disable certificate or checksum checks to get past a transient failure. Capture the selected URL, proxy behavior, digest error, and filesystem state.

Reproducibility has levels. Functional reproducibility means independently built compilers behave equivalently for an observation set. Bit-for-bit reproducibility means bytes match. Self-hosting means a compiler can build the checkout. None implies all the others.

Timestamps, paths, archive order, linker versions, backend versions, debug information, randomization, and environment can affect bytes. Before claiming non-reproducibility, define which outputs, configurations, and observations are compared. Use clean directories and record all tool versions.

Bootstrap tests often compare stage outputs to catch unexpected differences. Interpret such checks according to their documented normalization and platform scope. A byte difference can be harmless; a byte match can preserve a semantic bug.

For air-gapped work, provision verified artifacts and dependencies in advance. Record the source commit and cache manifest. Do not silently fall back to whatever toolchain happens to be installed globally.

43. A stable-Rust model of build-graph scheduling#

The following complete educational program models dependency ordering and cycle detection. It deliberately omits parallel execution, artifact identities, timestamps, failures, and caching. Its purpose is to make one invariant executable: every dependency precedes its consumer.

use std::collections::{BTreeMap, BTreeSet};

#[derive(Clone, Copy, Debug, Eq, Ord, PartialEq, PartialOrd)]
enum Mark {
	Visiting,
	Done,
}

fn visit<'a>(
	node: &'a str,
	graph: &'a BTreeMap<&'a str, Vec<&'a str>>,
	marks: &mut BTreeMap<&'a str, Mark>,
	order: &mut Vec<&'a str>,
) -> Result<(), String> {
	match marks.get(node) {
		Some(Mark::Done) => return Ok(()),
		Some(Mark::Visiting) => return Err(format!("dependency cycle at {node}")),
		None => {}
	}
	marks.insert(node, Mark::Visiting);
	for dependency in graph.get(node).into_iter().flatten() {
		if !graph.contains_key(dependency) {
			return Err(format!("unknown dependency {dependency} of {node}"));
		}
		visit(dependency, graph, marks, order)?;
	}
	marks.insert(node, Mark::Done);
	order.push(node);
	Ok(())
}

fn schedule<'a>(
	requested: &[&'a str],
	graph: &'a BTreeMap<&'a str, Vec<&'a str>>,
) -> Result<Vec<&'a str>, String> {
	let mut marks = BTreeMap::new();
	let mut order = Vec::new();
	for node in requested {
		if !graph.contains_key(node) {
			return Err(format!("unknown requested step {node}"));
		}
		visit(node, graph, &mut marks, &mut order)?;
	}
	let unique: BTreeSet<_> = order.iter().copied().collect();
	assert_eq!(unique.len(), order.len());
	Ok(order)
}

fn main() {
	let graph = BTreeMap::from([
		("stage0", vec![]),
		("std1", vec!["stage0"]),
		("rustc1", vec!["stage0", "std1"]),
		("ui", vec!["rustc1", "std1"]),
	]);
	assert_eq!(
		schedule(&["ui"], &graph).unwrap(),
		vec!["stage0", "std1", "rustc1", "ui"]
	);
	let cycle = BTreeMap::from([("a", vec!["b"]), ("b", vec!["a"])]);
	assert!(schedule(&["a"], &cycle).unwrap_err().contains("cycle"));
	println!("build graph invariants hold");
}

Compile-check it with the stable toolchain available on the workstation:

rustc build_graph.rs --edition=2021
./build_graph

A production scheduler also needs keys containing source and configuration identity, cross-process locking, atomic publication, cancellation, bounded concurrency, structured logs, and recovery after process death. Its cycle report should print the whole cycle, not merely one node. It must distinguish “dependency completed” from “artifact durably published.”

Exercise: extend the model with an Artifact { host, target, stage } key. Reject an execution edge whose host does not match the current machine. Add tests for duplicate requests, diamonds, missing dependencies, and a self-cycle. Then explain why a topological order does not choose optimal parallelism.

44. Choosing the shortest honest development loop#

Compiler work usually starts with ./x check or a narrow build and a stage1 test. Library work often uses library-focused profiles and tests to avoid rebuilding compiler crates. Tool work should select that tool and its tests. Exact selectors vary, so confirm them with local help and neighboring CI commands.

What changed?
  compiler Rust code --> x check compiler/... --> x test tests/<narrow-suite>
  std/library code --> x test library/<crate> or relevant tests
  bootstrap code --> bootstrap unit tests + dry/verbose selected step
  rustdoc/tool code --> tool unit tests + tool-specific integration suite
  backend/codegen --> narrow codegen tests --> broader target/perf validation

./x check catches type and many lint errors without constructing every final artifact. It cannot establish runtime behavior, generated code, diagnostics, or self-hosting. A stage1 compiler test exercises local compiler source. A stage2 build is justified when the change touches bootstrap boundaries, compiler self-use, metadata compatibility, distribution shape, or when CI requires it.

Command decision tree:

Need only type feedback?                 -> ./x check <path>
Need changed compiler behavior?          -> ./x test tests/ui/<area>
Need to run local compiler manually?     -> ./x build compiler/rustc
Changed one library crate?               -> test that crate and affected UI/run-make cases
Changed bootstrap scheduling/config?     -> bootstrap tests, dry run, then selected real step
Changed code generation?                 -> codegen/assembly test plus representative execution
Changed broad semantics?                 -> targeted suite, then expanded suites and CI plan
Unsure what selector means?              -> ./x <verb> --help and inspect verbose plan

Keep one known-good command in the investigation notebook. Change one factor at a time: source, stage, target, flags, or environment. If all factors move together, a success teaches little.

Incremental compiler builds can be fast but consume disk and preserve state. If a failure disappears after cleaning, report that fact and preserve the pre-clean log. The bug may be invalidation rather than “random cache corruption.”

Resource policy:

45. Cross compilation as an evidence ladder#

Cross compilation separates front-end correctness, target libraries, backend support, linking, and execution. Treat them as separate gates. A successful compile without linking says nothing about target runtime behavior.

parse/type/MIR on host
  -> target code generation
  -> target assembler/object
  -> target linker + target native libraries
  -> target loader
  -> execution on hardware/emulator
(each gate can fail independently)

First confirm that the target is recognized and its target specification is appropriate. Then build or obtain core/std for that target. Then identify the linker, sysroot, CRT objects, libc or equivalent, and SDK. Finally identify how tests execute: native runner, emulator, remote device, or compile-only.

A target can support core but not std. It can support compilation but not upstream test execution. Tier policy documents guarantees and CI coverage; it is not merely a popularity ranking. Read the current target-tier policy and platform support pages.

Never interpret skipped execution as passing runtime tests. Report “compile-only validation for T” explicitly. For emulator runs, record emulator version and CPU model. For remote runners, preserve exit status, stdout, stderr, timeout, and transport failure distinctly.

Cross-test failure map:

The production invariant is that every supported target's declared guarantees have a builder or documented validation path. Absence of a builder is uncertainty and must remain visible.

46. compiletest: executor, oracle, and test protocol#

compiletest is rustc's specialized integration-test driver. It discovers fixtures, parses directives, invokes compilers or tools, captures results, normalizes unstable text, compares expectations, and reports mismatches. The fixture plus suite mode defines a protocol, not just a Rust program.

test source + directives
  -> compiletest configuration
  -> command construction
  -> rustc/rustdoc/program execution
  -> stdout/stderr/artifacts
  -> normalization
  -> expected comparison and annotations

An oracle is the mechanism deciding pass or fail. For a UI test, expected diagnostics are an oracle. For run-pass, process success and sometimes output are oracles. For codegen, FileCheck-like patterns inspect IR. Choose the oracle nearest the invariant.

Overly broad snapshots preserve irrelevant spelling and create churn. Overly weak substring checks allow regressions. Normalization removes environmental noise but can also erase real distinctions. Every normalization rule therefore carries a proof obligation: the removed difference is outside the test's observation model.

Read adjacent tests because directive syntax and suite defaults evolve. Use compiletest's current help and source for 1.97.1-era behavior. Do not copy an old directive from a blog without confirming that it is accepted.

A test has four conceptual phases:

1. arrange source, auxiliaries, target, flags, environment, and revision;
2. act by compiling, running, documenting, or inspecting;
3. normalize platform- or allocation-dependent noise;
4. assert status, diagnostics, output, or artifacts.

Failures in arrange can resemble compiler failures. Always inspect the exact command and selected expectation file. The earliest broken invariant might be “wrong revision ran,” not “diagnostic changed.”

47. UI tests, revisions, blessing, and auxiliary crates#

UI tests are appropriate when user-visible diagnostics, acceptance, rejection, or compiler exits are the primary contract. The source often includes annotations tying messages to source spans. Expected stderr/stdout files preserve fuller output when required by suite convention.

Make a test minimal but semantic. Keep the language construct that triggers the bug and guards against a nearby false positive. Delete unrelated dependencies and feature gates. Retain edition and target directives when they are causal.

Revisions run one fixture under multiple named configurations. They are useful for editions, solver modes, feature states, or compare modes sharing one source. Revision-specific directives and expectation files must make each case explicit. Avoid revisions when independent files would be clearer.

single source
  +-- revision old: flags A --> expected old.stderr
  +-- revision new: flags B --> expected new.stderr
  +-- shared directives and annotations

Blessing updates expected output to actual output. It is a recording operation, not an approval operation. Before blessing, read every changed diagnostic, span, suggestion, exit status, and normalization. Then inspect the diff for unrelated churn.

Never bless a large suite merely to make red disappear. If output changed unexpectedly, first ask whether the implementation, diagnostic policy, normalizer, or expectation is wrong. Bless only the intended observation.

Auxiliary builds create helper crates used by the primary fixture. They model cross-crate metadata, macros, dylibs, and dependency behavior. An auxiliary crate has its own host/target role; proc-macro auxiliaries execute on the host. Name and structure them according to neighboring current tests.

Normalization commonly handles paths, line endings, addresses, hashes, timing, or platform text. Prefer the narrowest pattern. After adding one, construct two outputs that must remain distinguishable and prove they do.

Suggestion tests should check applicability and replacement text where relevant. A syntactically plausible suggestion can still change meaning. Add a case where the suggestion must not be emitted.

48. Executable, project, and backend-facing suites#

Run-pass tests compile and execute a program expected to succeed. Run-fail variants, where supported by current suite organization, validate controlled failure. They require an executable target and suitable runner. Do not use execution merely to test a compile-time diagnostic.

Run-make tests are small projects driven by a make-like harness. They cover multi-step behavior that one source fixture cannot express: separate compilation, linking, archive shape, environment, native objects, incremental rebuilds, and tool interaction. Their power makes them slower and more platform-sensitive.

Codegen tests inspect backend IR or related generated representation. They establish optimization or lowering properties without depending on final assembly syntax. Patterns should encode semantics, not incidental temporary names or instruction order.

Assembly tests inspect emitted machine assembly when the instruction sequence or ABI is the contract. Pin target and CPU features deliberately. Assembly differs by backend version and target, so explain why each checked instruction matters.

MIR-opt tests inspect MIR before or after named transformations. Expected diffs make changes reviewable but can be large. Use them when the invariant belongs to MIR; use runtime or UI tests for user-observable behavior. An optimization test should also protect semantic equivalence where practical.

Incremental tests perturb source across sessions and check invalidation or reuse. Their essential invariant is not simply “second compilation succeeds.” It is that changed dependencies are recomputed and unchanged eligible work may be reused. False reuse is a correctness bug; false invalidation is usually a performance bug.

rustdoc tests cover rendered documentation, search indices, JSON output, doctests, and UI. HTML string tests are vulnerable to presentation churn. Prefer semantic selectors and the tool's established harness conventions. Remember that doctests compile synthetic crates under rustdoc-selected settings.

Unit tests belong near pure components when public integration behavior is unnecessary. They are fast and diagnostic, but can overfit internals. Tidy is different: it checks repository-wide hygiene, generated-file consistency, licenses, features, formatting conventions, and policy cheaply before expensive builders.

49. Designing tests from invariants rather than directories#

Start with a sentence: “When condition C holds, property P must remain true under observations O.” Then choose the cheapest suite capable of observing P. Directory familiarity is not a test strategy.

For a bug fix, include the smallest positive case, a near miss, and a boundary case when each guards a plausible recurrence. Do not multiply cases without distinct failure mechanisms. A regression test should fail before the fix for the intended reason.

Run the test against the parent revision or temporarily revert the fix. If it already passes, the oracle is wrong, the reproducer is incomplete, or another change fixed it. Record the expected pre-fix failure.

Test layers answer different questions:

unit: local mechanism
  -> integration fixture: subsystem contract
  -> suite: neighboring interactions
  -> CI matrix: platforms and configurations
  -> Crater/perf: ecosystem and workload effects
(increasing breadth, cost, and diagnosis distance)

Flaky tests violate repeatability. Classify nondeterminism: scheduling, time, random seed, resource pressure, filesystem order, network, address, or data race. Do not hide it with a large timeout until the cause is known.

Timeouts are resource limits, not semantic assertions. Set them from representative slow builders with margin and preserve a timeout-specific result. Infinite hangs require process-tree cleanup.

Production test hardening includes hermetic temporary directories, explicit environment, bounded output, deterministic ordering, safe cleanup, and actionable failure logs. Secrets and private paths must not enter snapshots or uploaded artifacts.

Exercise: for an ICE involving an auxiliary proc macro, design three tests. One preserves the crash regression. One checks a diagnostic without loading the macro. One checks host/target separation under cross compilation. Explain why a single UI snapshot cannot establish all three.

50. A minimal stable-Rust test-harness model#

This complete program demonstrates explicit outcomes, normalization, and a timeout-free command runner. It intentionally does not provide production process-tree cancellation or sandboxing.

use std::process::Command;

#[derive(Debug, PartialEq)]
struct Outcome {
	status: Option<i32>,
	stdout: String,
	stderr: String,
}

fn normalize(text: &[u8], root: &str) -> String {
	String::from_utf8_lossy(text)
		.replace("\r\n", "\n")
		.replace(root, "$DIR")
}

fn run(program: &str, args: &[&str], root: &str) -> std::io::Result<Outcome> {
	let output = Command::new(program).args(args).output()?;
	Ok(Outcome {
		status: output.status.code(),
		stdout: normalize(&output.stdout, root),
		stderr: normalize(&output.stderr, root),
	})
}

fn main() -> std::io::Result<()> {
	let outcome = run("rustc", &["--version"], "/impossible/private/root")?;
	assert_eq!(outcome.status, Some(0));
	assert!(outcome.stdout.starts_with("rustc "));
	assert_eq!(normalize(b"a\r\nb /work/x\n", "/work"), "a\nb $DIR/x\n");
	println!("harness model passed");
	Ok(())
}

The model preserves exit status separately from output. That matters because identical diagnostics with a crash status are not equivalent. Lossy UTF-8 is an explicit policy and would be unacceptable for byte-exact tests. Root replacement is dangerously broad if root is empty or appears inside unrelated text.

A hardened harness needs deadlines, process-group termination, output-size limits, signal results, environment allowlists, temporary-directory isolation, platform quoting, and concurrent pipe reads. It needs structured records containing command, cwd, revision, target, duration, and artifact paths. It must not execute untrusted tests without an appropriate sandbox.

Add tests before extending it:

1. nonzero status remains nonzero;
2. stdout and stderr are not merged;
3. CRLF normalization does not remove ordinary carriage returns unexpectedly;
4. path replacement respects component boundaries;
5. invalid UTF-8 follows documented policy;
6. an output cap reports truncation rather than silently matching.

Omissions are part of technical honesty. This model teaches the observation pipeline, not compiletest compatibility. Do not grow it into a parallel testing framework inside a compiler patch.

51. Reading failures by the earliest broken invariant#

The first red line is often downstream of the cause. A linker failure may begin with the wrong target library selection. A snapshot mismatch may begin with an unintended compiler flag. Work backward through provenance and phases.

Failure decision tree
  command never started?
    -> executable path, permissions, host architecture
  compiler exited unexpectedly?
    -> signal/ICE/status, stderr, resource limits
  compile succeeded but comparison failed?
    -> selected revision, normalization, expectation, intended output
  link failed?
    -> object targets, linker command, native dependencies, sysroot
  run failed?
    -> runner, loader, environment, program semantics
  only CI fails?
    -> builder config, platform, clean state, sharding, resources

Preserve the first failing command from verbose logs. Rerun it in the environment bootstrap established when possible. If manual execution succeeds, compare cwd, environment, sysroot, dynamic library path, and flags.

Reduce along one axis at a time. Reduce source while keeping compiler and flags fixed. Then reduce flags. Then compare revisions or hosts. This makes each observation interpretable.

CI logs may omit context or interleave jobs. Download the relevant artifact only from trusted project infrastructure. Do not execute arbitrary failure attachments. Quote decisive lines in the PR and link the complete log.

52. Regression triage and disciplined bisection#

A regression is a behavior that is worse in a newer revision under a defined expectation. First establish one known-good and one known-bad compiler with the same reproducer, flags, target, environment, and dependency inputs. Nightly dates are useful only after toolchain identity is confirmed.

Classify the regression: accepted-to-rejected, rejected-to-accepted, wrong code, ICE, diagnostic degradation, performance, bootstrap, or target support. The category controls urgency, bisection oracle, and team routing.

A bisection requires a monotonic interval for the chosen predicate. If revisions alternate good and bad because of flakiness or dependency changes, binary search lies. Repeat boundary observations and pin external inputs.

known good G ------------------------------ known bad B
                    test midpoint M
predicate(M)=bad -> search [G,M]
predicate(M)=good -> search [M,B]
predicate(M)=unknown -> repair oracle; do not guess

cargo-bisect-rustc can search published compiler artifacts and help identify a nightly or commit range. Its exact options and available artifacts are version-sensitive. Read its current documentation and preserve the generated command. Missing artifacts and build failures are “unknown,” not automatically bad.

For source bisects, automate one bounded interestingness script. Exit distinctly for good, bad, and skip if the tool supports it. Set timeouts and capture the compiler identity. Do not use a diagnostic substring if the actual regression is process status.

After finding a candidate commit, read the diff, issue, rollup context, and later fixes. The first bad commit is evidence, not blame. Rollups can contain several PRs; a merge conflict resolution can matter. Notify relevant teams through normal triage routes without assigning motives.

Performance regressions need representative repeated measurements and noise analysis. A single wall-clock result on a busy laptop is not a reliable predicate. Use rustc-perf infrastructure when the impact is compiler-wide.

53. Debuggers, tracing, logs, and representation dumps#

Choose instrumentation by the question. A debugger answers control flow, state, crashes, and native stack questions. Tracing answers event-order and query questions. Representation dumps answer “what did this phase produce?” Profiling answers where resources went.

Build enough debug information for the component under investigation. Too little yields opaque frames; maximum debuginfo everywhere can make linking impractical. Record optimization and assertion settings because they change behavior and inspectability.

Use gdb, lldb, or platform debuggers according to host support. Launch the exact stage compiler with the same sysroot and arguments as the failing test. For child rustc processes, attach or configure follow-fork behavior deliberately. Debugger commands are platform-version-sensitive.

RUSTC_LOG and tracing targets can expose internal events in builds where the relevant maximum level is enabled. Start with a narrow module target and modest verbosity. Unbounded logs perturb timing, fill disks, and hide causality. Never assume an absent trace proves an event did not occur without checking compile-time filters.

Compiler flags can dump HIR, MIR, query, macro expansion, LLVM IR, assembly, or other internal representations. Many are unstable internal interfaces. Use the local compiler's -Z help, record its commit, and avoid presenting dump syntax as stable.

Compare dumps at the earliest diverging phase. If HIR matches but MIR differs, stop searching parser code first. If MIR matches but LLVM IR differs, inspect codegen configuration and backend. If IR matches but binaries differ, inspect linker and post-processing.

Dump files can contain source, paths, symbol names, and proprietary data. Sanitize before sharing, but preserve a private original hash so sanitization is auditable. Security-sensitive dumps must not enter public issues.

54. Self-profile, measureme, rustc-perf, and Crater#

Self-profile records compiler events and resource attribution using the measureme ecosystem. It helps answer which queries or activities consumed time and how counts changed. It does not by itself explain why an event is expensive or whether a change helps users.

Choose a representative workload, warm/cold policy, compiler configuration, and repeated-run plan. Control CPU contention and thermal behavior. Record wall time alongside event data. Compare the same metric and units.

rustc instrumentation -> measureme event data
  -> analysis tools -> query time/count/incremental summaries
  -> hypothesis -> code change -> repeated measurement
(measurement supports causality only with controlled comparison)

rustc-perf maintains benchmark workloads and infrastructure for compiler performance tracking. Its collector and site evolve; consult https://github.com/rust-lang/rustc-perf and current compiler team procedures. Report wins and regressions across check/debug/opt/incremental scenarios as relevant.

Benchmark representativeness is a policy choice. One tiny crate may expose fixed overhead while missing monomorphization costs. One giant crate may hide interactive latency. State which workload population supports the claim.

Crater builds a large sample of ecosystem crates with two toolchains to estimate compatibility impact. It is broad differential evidence, not proof that all Rust code is safe. Failures include genuine regressions, network breakage, nondeterminism, environment assumptions, and crates already broken in the baseline.

Crater triage compares baseline and candidate logs, groups root causes, minimizes representative cases, and routes findings. Respect crate authors: do not characterize a crate as broken without checking baseline and context. Do not expose private runs or embargoed changes.

Performance and ecosystem experiments consume shared resources. Request them according to team policy with a clear hypothesis and expected decision. Cancel obsolete runs and summarize results for future contributors.

55. Soundness and security response#

Soundness means safe Rust cannot cause undefined behavior under the language and library contract. A soundness bug is not merely an ICE or incorrect diagnostic. Wrong code in safe code, invalid layout assumptions, aliasing violations, and unsafe library APIs with insufficient preconditions can be soundness issues.

Security severity depends on exploitability, affected channels, deployment context, and exposure. Do not independently publish a sensational severity score. Use the Rust security policy at https://www.rust-lang.org/policies/security for potentially embargoed reports.

If a reproducer may disclose a vulnerability, stop public triage. Send the minimal report through the current private channel. Include compiler versions, target, optimization, code, observed behavior, and why safe code is involved. Do not paste it into Zulip, a public issue, Crater request, or public CI.

potential vulnerability
  -> private intake
  -> reproduce and classify
  -> identify affected versions/targets
  -> design fix + regression test under embargo
  -> coordinated release/advisory
  -> public postmortem when authorized

Minimization can accidentally make exploit details easier to weaponize. Keep artifacts access-controlled and remove secrets. Use approved infrastructure for collaboration. Follow the response team's disclosure timeline even if a patch appears obvious.

The fix must restore an explicit invariant. For unsafe code, document caller guarantees, implementation assumptions, and consequence of violation. Test the safe boundary and nearby optimizations. Consider old release branches, bootstrap snapshots, standard-library distribution, and targets.

Security hardening also applies to bootstrap and tests: verify downloads, avoid shell injection, bound decompression and output, isolate untrusted code, and protect credentials in CI. A test suite routinely compiles and executes repository content; trust boundaries must be stated.

After disclosure, derive lessons from mechanism rather than blaming the discoverer or author. Preserve enough public regression coverage to prevent recurrence without publishing unnecessary weaponization details.

56. RFCs, MCPs, teams, and review authority#

An RFC establishes broad project design when the governance process requires community-level consensus. A compiler MCP, or Major Change Proposal, is a compiler-team process for significant compiler changes that need early visibility and team decision without necessarily requiring an RFC. Exact process and templates evolve; use current compiler-team documentation.

Small implementation fixes usually need neither. Ask these questions:

Does this change stable language or library meaning?
  yes -> consult owning team; likely design/stabilization process
Does it substantially alter compiler architecture or contributor workflow?
  yes -> ask whether an MCP is expected
Is it a bounded bug fix within accepted behavior?
  usually -> issue/PR and normal review
Does it cross team authority?
  yes -> obtain each required decision; one approval cannot substitute for another

Teams delegate authority by domain. Compiler implementation, language semantics, library API, release, infrastructure, and security can overlap in one patch. Current team rosters and charters, not historical names, identify decision makers.

Review is risk analysis and knowledge transfer. A good PR tells the reviewer the invariant, root cause, alternatives, test evidence, performance impact, compatibility impact, and unresolved uncertainty. Small diffs reduce review surface but must not hide necessary generated changes.

Respond to review by resolving the underlying concern, not merely the comment text. When disagreeing, restate the shared requirement and provide evidence. Mark conversations resolved according to repository etiquette. Do not force-push away evidence reviewers are using unless current policy requests it.

Rollups increase throughput by testing several approved PRs together. A rollup failure may be interaction or unrelated flakiness. Approval is not landing, and landing is not release. Track what actually merged and whether it was reverted.

57. The complete contribution lifecycle#

1. Search issues, merged PRs, the development guide, and team discussions.
2. Reproduce on an exact current compiler and record --version --verbose.
3. Reduce while preserving the failure category and flags.
4. Establish known-good and known-bad behavior; bisect when useful.
5. Identify the earliest broken invariant and owning subsystem.
6. Discuss design early when semantics, architecture, or security require it.
7. Claim or coordinate the issue under current triagebot practice.
8. Select a development profile and shortest honest bootstrap loop.
9. Add a regression test and prove it fails before the fix.
10. Implement the smallest complete fix with comments on non-obvious invariants.
11. Run formatting, tidy/unit checks, focused suites, and expanded risk-selected tests.
12. Measure performance when the cost model predicts impact.
13. Audit target, stage, incremental, diagnostic, and compatibility boundaries.
14. Write a PR description that lets another person reproduce the claim.
15. Respond to review, rebase when required, and rerun invalidated evidence.
16. Observe CI, distinguish infrastructure failures, and avoid blind retries.
17. Confirm landing; watch rollup, perf, Crater, and regression signals.
18. Help with backport, release note, stabilization, or docs if requested.
19. Return for regressions and leave a handoff if unavailable.

The PR description should include:

Problem and user impact
Root cause and earliest broken invariant
Chosen mechanism and rejected alternatives
Test: pre-fix failure and post-fix result
Commands, stage, host, target, and configuration
Performance/compatibility/security analysis
Known omissions and follow-up work
Issue/design links

Generated output belongs in the same logical change when policy requires it. Do not hand-edit generated files unless their generator instructs that workflow. Do not combine opportunistic refactors with a regression fix.

Before each expensive test, ask what result would change your decision. Before omitting a test, state which risk remains. That discipline converts compute spending into evidence.

After merge, distinguish source landing, nightly availability, beta backport, and stable release. Users asking “is it fixed?” need a channel and version, not merely a merged commit.

58. Resource planning and production hardening#

Plan work as a budget across CPU, memory, disk, network, specialized runners, and reviewer attention. A maximal local build can consume all six while adding little evidence. Start narrow, then expand along changed boundaries.

Keep at least enough free disk for temporary linker outputs and duplicate artifacts. Monitor actual build-directory growth rather than relying on a universal estimate. Limit jobs when memory pressure causes swapping or OOM kills.

CI production hardening requires pinned inputs, least-privilege credentials, verified artifacts, bounded retries, cancellation, logs, retention policy, and reproducible builder descriptions. Retries are appropriate for classified transient infrastructure failures, not deterministic tests.

Caches need namespaced keys, integrity checks, atomic writes, eviction, and observability. Treat cache poisoning as a correctness and security threat. A cache hit should log enough provenance to audit why it was accepted.

Tests executing target programs need process isolation and cleanup after timeout. Never rely on dropping one parent process if descendants survive. Network access should be absent unless a test's contract explicitly requires controlled service.

Observability should answer: which step ran, why it ran, what it consumed, which artifact it produced, why a cache hit occurred, and where failure began. Logs must redact tokens while preserving compiler arguments and provenance needed for diagnosis.

Disaster exercise: assume a shared compiler cache served one wrong-target std artifact. Design detection, quarantine, key correction, cache invalidation, builder recovery, and an incident record. Explain why rerunning only failed tests is insufficient.

59. Guided practice from newcomer to independent contributor#

Milestone one: map a native build. Use a dry or verbose supported bootstrap command. Draw nodes for stage0, stage1 std, stage1 rustc, and one UI test. Annotate compiler host and artifact target.

Milestone two: perturb one UI test. Predict the mismatch before running it. Bless into a temporary diff, inspect normalization, then restore it. Explain why blessing did not prove correctness.

Milestone three: build the graph program in Chapter 43. Add full cycle-path diagnostics and artifact keys. Property-test mentally that every edge points earlier in the returned order. Create a counterexample for a cache keyed only by step name.

Milestone four: classify ten neighboring tests. For each, name its oracle and one behavior it cannot observe. Move none until you can explain the lost information.

Milestone five: perform a historical regression investigation. Use a public fixed issue. Reconstruct good/bad boundaries, candidate commit, test, and review discussion. Do not rerun untrusted historical binaries without isolation.

Milestone six: inspect one self-profile capture. Predict the expensive query family first. Compare counts and time, then propose an experiment that could falsify your explanation. Do not optimize yet.

Milestone seven: cross-compile a trivial no-std artifact for a supported target available to you. Stop separately after code generation, object inspection, linking, and execution if possible. Label unperformed gates as unknown.

Milestone eight: triage a public Crater or CI failure. Separate baseline failure, candidate-only failure, flaky infrastructure, and duplicate root cause. Write a five-line handoff with evidence.

Milestone nine: review a merged unsafe fix. List caller guarantees, implementation assumptions, violated invariant, test boundary, and release response. Compare your list with reviewer discussion.

Capstone: choose one fixed rustc regression and reproduce its lifecycle. Produce a reducer, bisection script, subsystem trace, pre-fix test, minimal patch sketch, verification matrix, PR narrative, and post-merge monitoring plan. The capstone succeeds when another contributor can challenge every claim from your evidence.

Independence does not mean never asking questions. It means knowing which authority to consult, presenting bounded uncertainty, and leaving a reproducible trail.

60. Derived philosophy, talk outlines, and authoritative reading map#

Bootstrap demonstrates that abstractions move trust rather than delete it. Stage0 moves trust to a published snapshot and its supply chain. Stage2 moves the self-hosting question one generation forward. Neither removes the need for semantic validation.

The build/host/target matrix demonstrates that identity is not a filename. An artifact's usability depends on where it runs, what it targets, who built it, and under which configuration. Caches become correctness obligations because they claim two such identities are equivalent.

compiletest demonstrates that expected behavior depends on an observation model. Normalization intentionally forgets information. Blessing accepts a new observation but supplies no judgment. Good tests make preserved and discarded information explicit.

Bisection demonstrates that uncertainty must be represented. A missing artifact or flaky result is neither good nor bad. Forcing it into a Boolean can produce a precise but false culprit.

Profiles and narrow steps demonstrate that local speed and global assurance are different goals. Moving validation later improves iteration but increases the distance between a mistake and its discovery. The contribution lifecycle places cheap, diagnostic checks early and broad checks before release.

The earliest broken invariant may precede the visible failure by several systems. Wrong provenance appears as a linker error. Wrong invalidation appears as wrong code. Wrong normalization appears as a harmless pass. Debugging is the work of tracing backward without guessing away uncertainty.

Twenty-minute talk: “Why rustc builds itself more than once.”

1. the circularity problem and stage0;
2. stage1 versus stage2 with one artifact trace;
3. build/host/target on a cross target;
4. what a successful bootstrap proves and does not prove;
5. one cache-provenance failure;
6. contributor command decision tree.

Forty-five-minute talk: “A diagnostic's journey through rustc validation.”

1. reduce a public regression;
2. choose a UI oracle;
3. trace compiletest arrange/act/normalize/assert;
4. revisions, auxiliary crates, and blessing;
5. stage1 local loop and CI expansion;
6. bisection and review evidence;
7. post-merge nightly and release path.

Ninety-minute workshop: “Proving a compiler change.”

1. participants label a bootstrap graph;
2. run and break the educational scheduler;
3. design tests for one invariant across three suites;
4. diagnose a staged failure map;
5. construct a monotonic bisection predicate;
6. interpret a self-profile summary;
7. draft an evidence-complete PR description;
8. peer-review omissions and remaining risk.

Authoritative reading map, current as a 1.97.1-era orientation:

• rustc development guide: https://rustc-dev-guide.rust-lang.org/
• bootstrap chapters and local source: src/bootstrap, config.example.toml, src/stage0
• testing guide and compiletest source: development guide, src/tools/compiletest, tests/
• nightly internal API docs: https://doc.rust-lang.org/nightly/nightly-rustc/
• Rust Forge and platform tiers: https://forge.rust-lang.org/
• compiler team and MCP process: https://forge.rust-lang.org/compiler/
• RFC repository: https://github.com/rust-lang/rfcs
• rustc-perf source and docs: https://github.com/rust-lang/rustc-perf
• Crater source and documentation: https://github.com/rust-lang/crater
• project security policy: https://www.rust-lang.org/policies/security
• Code of Conduct: https://www.rust-lang.org/policies/code-of-conduct

Normative language and library behavior comes from accepted specifications and team decisions. The development guide explains contributor practice. Checkout source and command help define current implementation details. Forge records current infrastructure and policy. Old design documents explain history, not necessarily present guarantees.

Before teaching from these outlines, pin a checkout and rerun every command. Replace screenshots with text where possible, disclose compile-only targets, and distinguish measured data from illustrative numbers. The mark of an independent contributor is not command memorization; it is preserving provenance, choosing an honest oracle, spending validation resources deliberately, and communicating exactly what remains unknown.

Part X: Build a Rust-Like Compiler from First Principles#

1. The workshop contract#

This part builds FerrisC, a deliberately small Rust-like language, in stable Rust. The goal is not to produce a toy that merely prints plausible assembly. The goal is to understand why each compiler representation exists, which facts each stage establishes, and which later stages are allowed to trust.

A compiler is a sequence of contracts. The lexer promises that token boundaries respect UTF-8. The parser promises that every input produces an AST, even after errors. The resolver promises that resolved names identify one stable definition. The type checker promises that typed expressions obey its stated rules. The MIR builder promises explicit control flow. The back end promises to preserve MIR behavior.

FerrisC accepts files like this:

fn max<T: Ord>(a: T, b: T) -> T {
    if a < b { b } else { a }
}

fn main() -> i32 {
    let mut n: i32 = 3;
    while 0 < n {
        n = n - 1;
    }
    max(10, n)
}

The core language has integer and Boolean literals, local bindings, assignment, blocks, conditionals, loops, functions, calls, and a small reference layer. Later we add one educational generic-and-trait layer. That layer is intentionally finite and explicit.

FerrisC excludes modules loaded from disk, closures, async code, pattern matching, structs, enums, unsafe code, method lookup, associated types, lifetimes written by users, and procedural macros. It also excludes integer overflow modes, unwinding, dynamic dispatch, platform ABIs, and most coercions. Every integer is a signed 64-bit value at run time, although we retain the spelling i32 as a teaching type.

Those exclusions are not claims that the omitted work is unimportant. They make every state transition small enough to inspect. Real rustc has many more representations, compatibility obligations, target details, and diagnostics refined over years.

Our implementation rule is simple: never continue because “this case probably cannot happen.” Either an invariant proves the case impossible, or the program returns a structured diagnostic. Compiler bugs may still use an internal error, but malformed source must not panic.

Prediction question: if parsing fails halfway through one function, should valid later functions disappear? Our answer is no. Recovery nodes preserve the file shape so later stages can report more errors.

Counterfactual design: we could combine lexing, parsing, and type checking in one recursive routine. That can work for a calculator, but it hides stage contracts and makes recovery, caching, and source-oriented diagnostics much harder. FerrisC keeps boundaries because rustc contribution work depends on them.

2. The language grammar and static promises#

A grammar describes legal syntax, not meaning. We use an extended Backus–Naur form: quoted words are literal tokens, parentheses group alternatives, an asterisk means repetition, and a question mark means optional.

file       = item* EOF
item       = fn_item | impl_item
fn_item    = "fn" IDENT generics? "(" params? ")" ret? block
generics   = "<" IDENT (":" IDENT)? ">"
params     = param ("," param)* ","?
param      = IDENT ":" type
ret        = "->" type
type       = "i32" | "bool" | "()" | "&" "mut"? type | IDENT
impl_item  = "impl" IDENT "for" type ";"
block      = "{" stmt* expr? "}"
stmt       = let_stmt | while_stmt | expr ";"
let_stmt   = "let" "mut"? IDENT (":" type)? "=" expr ";"
while_stmt = "while" expr block
expr       = assignment
assignment = logic_or ("=" assignment)?
logic_or   = logic_and ("||" logic_and)*
logic_and  = equality ("&&" equality)*
equality   = comparison (("==" | "!=") comparison)*
comparison = sum (("<" | "<=") sum)*
sum        = product (("+" | "-") product)*
product    = unary (("*" | "/") unary)*
unary      = ("!" | "-" | "&" | "&mut" | "*") unary | call
call       = primary ("(" arguments? ")")*
primary    = INT | "true" | "false" | IDENT | block
           | "if" expr block "else" block | "(" expr ")"
arguments  = expr ("," expr)* ","?

The final expression in a block has no semicolon and supplies its value. An expression followed by a semicolon is evaluated for effects and becomes unit. This distinction explains a famous Rust beginner error: adding a semicolon can change a function result from a value to ().

Static promises are separate from grammar. Assignment requires a mutable local place. if conditions require bool. Both branches must agree on a type. while has type unit. Calls must match parameter counts and types. Every local must be initialized before use.

References are intentionally restricted. &x and &mut x may borrow only a local variable, not an arbitrary projection or temporary. A reference may not escape its function, be returned, or be stored inside another value. These rules reject valid Rust programs, but make our educational checker easier to state.

Our accepted subset is not automatically sound just because it is smaller. Soundness means accepted programs cannot violate the language safety model. Later we identify an aliasing hole and close it conservatively. We will never call the result “Rust's borrow checker.”

Prediction question: does if true { 1 } else { false } parse? Yes, because syntax permits it. Type checking rejects it later. Keeping that distinction makes each diagnostic come from the stage that actually understands the violated rule.

3. Source files, byte positions, and spans#

Diagnostics need to point back to source. Rust strings are UTF-8, so byte offsets are efficient and unambiguous, but they are not character counts. A span is a half-open byte range [start, end) in one file. Half-open ranges compose cleanly: adjacent spans can meet at one byte without overlap, and length is simply end - start.

#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)]
pub struct FileId(pub u32);

#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub struct Span {
	pub file: FileId,
	pub start: u32,
	pub end: u32,
}

impl Span {
	pub fn new(file: FileId, start: usize, end: usize) -> Option<Self> {
		if start > end || end > u32::MAX as usize {
			return None;
		}
		Some(Self { file, start: start as u32, end: end as u32 })
	}

	pub fn join(self, other: Span) -> Option<Span> {
		(self.file == other.file).then_some(Span {
			file: self.file,
			start: self.start.min(other.start),
			end: self.end.max(other.end),
		})
	}
}

pub struct SourceFile {
	pub id: FileId,
	pub name: String,
	pub text: String,
	line_starts: Vec<u32>,
}

impl SourceFile {
	pub fn new(id: FileId, name: String, text: String) -> Self {
		let mut line_starts = vec![0];
		for (byte, ch) in text.char_indices() {
			if ch == '\n' && byte + 1 <= u32::MAX as usize {
				line_starts.push((byte + 1) as u32);
			}
		}
		Self { id, name, text, line_starts }
	}

	pub fn line_col(&self, byte: u32) -> Option<(usize, usize)> {
		let b = byte as usize;
		if b > self.text.len() || !self.text.is_char_boundary(b) {
			return None;
		}
		let line = self.line_starts.partition_point(|&s| s <= byte) - 1;
		let start = self.line_starts[line] as usize;
		let column = self.text[start..b].chars().count();
		Some((line + 1, column + 1))
	}
}

The span invariant is stronger than the constructor currently checks: both endpoints should be UTF-8 character boundaries in the referenced file. The lexer naturally creates such boundaries from char_indices. Synthetic spans created by later stages must preserve the same property.

Why not store line and column directly? Editing or macro expansion can make those values expensive to update, and slicing still needs byte positions. rustc likewise uses compact source positions and a source map, though its model handles expansions and many more details.

Columns above count Unicode scalar values, not displayed terminal cells. A tab, combining mark, or wide East Asian glyph breaks that approximation. A production renderer should compute display columns consistently. FerrisC states the limitation instead of claiming perfect highlighting.

4. Structured diagnostics, not printed strings#

A compiler stage should describe an error, not decide colors, terminal widths, or JSON formatting. We therefore store diagnostics as data.

#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub enum Severity { Error, Warning, Note }

#[derive(Clone, Debug, PartialEq, Eq)]
pub struct Label {
	pub span: Span,
	pub message: String,
	pub primary: bool,
}

#[derive(Clone, Debug, PartialEq, Eq)]
pub struct Diagnostic {
	pub code: &'static str,
	pub severity: Severity,
	pub message: String,
	pub labels: Vec<Label>,
	pub notes: Vec<String>,
}

impl Diagnostic {
	pub fn error(code: &'static str, span: Span, message: impl Into<String>) -> Self {
		Self {
			code,
			severity: Severity::Error,
			message: message.into(),
			labels: vec![Label { span, message: String::new(), primary: true }],
			notes: Vec::new(),
		}
	}
}

Codes such as E-PARSE-EXPECTED remain stable even if wording improves. Tests and editor clients can rely on codes rather than whole sentences. Labels explain relationships: one points to a duplicate definition, and another points to the original. Notes offer context without claiming another source location.

The diagnostic invariant says every primary source error has a valid primary span. An error caused only by command-line configuration may instead use a driver location. Do not invent byte zero as a fake source position; model non-source diagnostics explicitly in a larger implementation.

Rendering sorts by file, start position, severity, then code. This deterministic order matters for tests and incremental builds. Collection order can vary when queries become parallel.

Counterfactual design: printing immediately from the lexer looks convenient. It prevents callers from choosing JSON, makes duplicate suppression difficult, and tangles tests with terminal output. rustc's diagnostic machinery is much richer, but the architectural lesson is identical: preserve structure until presentation.

5. Tokens and UTF-8-safe lexing#

Lexing groups characters into tokens. Tokens carry kind and span; they do not yet know whether an identifier names a variable or function.

#[derive(Clone, Debug, PartialEq, Eq)]
pub enum TokenKind {
	Ident(String),
	Int(String),
	Fn, Let, Mut, If, Else, While, True, False, Impl, For,
	LParen, RParen, LBrace, RBrace, Lt, Le, Gt,
	Colon, Semi, Comma, Arrow,
	Eq, EqEq, Bang, BangEq, Plus, Minus, Star, Slash,
	Amp, AmpAmp, PipePipe,
	Error(char),
	Eof,
}

#[derive(Clone, Debug, PartialEq, Eq)]
pub struct Token {
	pub kind: TokenKind,
	pub span: Span,
}

FerrisC identifiers start with an ASCII letter or underscore and continue with ASCII letters, digits, or underscores. The source may contain arbitrary UTF-8, but non-ASCII identifier characters receive a precise error token. Real Rust follows Unicode identifier rules. Using ASCII here is a language choice, not a UTF-8 bug.

The lexer advances only at char_indices boundaries. It recognizes // comments and ASCII whitespace. Integer text remains unparsed so overflow diagnostics can point to the literal.

pub fn lex(file: FileId, src: &str) -> (Vec<Token>, Vec<Diagnostic>) {
	let mut out = Vec::new();
	let mut errors = Vec::new();
	let mut it = src.char_indices().peekable();

	while let Some((start, ch)) = it.next() {
		if ch.is_whitespace() { continue; }
		let one = |end| Span::new(file, start, end).unwrap();

		if ch == '/' && it.peek().is_some_and(|&(_, c)| c == '/') {
			it.next();
			while let Some(&(_, c)) = it.peek() {
				if c == '\n' { break; }
				it.next();
			}
			continue;
		}

		if ch.is_ascii_alphabetic() || ch == '_' {
			let mut end = start + ch.len_utf8();
			while let Some(&(i, c)) = it.peek() {
				if !(c.is_ascii_alphanumeric() || c == '_') { break; }
				it.next();
				end = i + c.len_utf8();
			}
			let word = &src[start..end];
			let kind = match word {
				"fn" => TokenKind::Fn, "let" => TokenKind::Let,
				"mut" => TokenKind::Mut, "if" => TokenKind::If,
				"else" => TokenKind::Else, "while" => TokenKind::While,
				"true" => TokenKind::True, "false" => TokenKind::False,
				"impl" => TokenKind::Impl, "for" => TokenKind::For,
				_ => TokenKind::Ident(word.to_owned()),
			};
			out.push(Token { kind, span: one(end) });
			continue;
		}

		if ch.is_ascii_digit() {
			let mut end = start + 1;
			while let Some(&(i, c)) = it.peek() {
				if !c.is_ascii_digit() { break; }
				it.next();
				end = i + 1;
			}
			out.push(Token { kind: TokenKind::Int(src[start..end].into()), span: one(end) });
			continue;
		}

		let next_is = |it: &mut std::iter::Peekable<_>, wanted| {
			if it.peek().is_some_and(|&&(_, c)| c == wanted) { it.next(); true } else { false }
		};
		let (kind, end) = match ch {
			'(' => (TokenKind::LParen, start + 1), ')' => (TokenKind::RParen, start + 1),
			'{' => (TokenKind::LBrace, start + 1), '}' => (TokenKind::RBrace, start + 1),
			':' => (TokenKind::Colon, start + 1), ';' => (TokenKind::Semi, start + 1),
			',' => (TokenKind::Comma, start + 1), '+' => (TokenKind::Plus, start + 1),
			'*' => (TokenKind::Star, start + 1),
			'-' if next_is(&mut it, '>') => (TokenKind::Arrow, start + 2),
			'-' => (TokenKind::Minus, start + 1),
			'/' => (TokenKind::Slash, start + 1),
			'=' if next_is(&mut it, '=') => (TokenKind::EqEq, start + 2),
			'=' => (TokenKind::Eq, start + 1),
			'!' if next_is(&mut it, '=') => (TokenKind::BangEq, start + 2),
			'!' => (TokenKind::Bang, start + 1),
			'<' if next_is(&mut it, '=') => (TokenKind::Le, start + 2),
			'<' => (TokenKind::Lt, start + 1), '>' => (TokenKind::Gt, start + 1),
			'&' if next_is(&mut it, '&') => (TokenKind::AmpAmp, start + 2),
			'&' => (TokenKind::Amp, start + 1),
			'|' if next_is(&mut it, '|') => (TokenKind::PipePipe, start + 2),
			other => (TokenKind::Error(other), start + other.len_utf8()),
		};
		let span = one(end);
		if matches!(kind, TokenKind::Error(_)) {
			errors.push(Diagnostic::error("E-LEX-CHAR", span, "character is not valid here"));
		}
		out.push(Token { kind, span });
	}

	let eof = Span::new(file, src.len(), src.len()).unwrap();
	out.push(Token { kind: TokenKind::Eof, span: eof });
	(out, errors)
}

The closure using an inferred iterator type may need an explicit helper function when assembled under a particular compiler version. That snippet teaches the loop and recovery design; the chapter's module version should give helpers concrete types. We do not falsely claim every isolated excerpt compiles unchanged.

On an unknown character, the lexer emits one error token and advances once. That is recovery: progress is guaranteed. Without progress, one bad character could create an infinite diagnostic loop.

6. Lexer tests and hostile inputs#

Tests should assert boundaries, not only token names. UTF-8 bugs often hide behind correct-looking token sequences.

#[test]
fn unicode_error_consumes_one_scalar_value() {
	let (tokens, errors) = lex(FileId(0), "let x = 🦀; let y = 1;");
	assert_eq!(errors.len(), 1);
	let crab = tokens.iter().find(|t| matches!(t.kind, TokenKind::Error('🦀'))).unwrap();
	assert_eq!(crab.span.end - crab.span.start, 4);
	assert!(tokens.iter().any(|t| matches!(&t.kind, TokenKind::Ident(s) if s == "y")));
}

#[test]
fn all_token_spans_are_boundaries() {
	let source = "fn café() {}";
	let (tokens, _) = lex(FileId(0), source);
	for token in tokens {
		assert!(source.is_char_boundary(token.span.start as usize));
		assert!(source.is_char_boundary(token.span.end as usize));
	}
}

The second input intentionally rejects é under our ASCII identifier rule. The property under test is still UTF-8 safety.

Test empty input, comments ending at EOF, every two-character operator, very long digit runs, a lone pipe, embedded null characters, and every byte boundary around multibyte characters. Integer parsing belongs later, so a million-digit literal must not overflow or panic in the lexer. A production compiler should impose a token-length limit and issue one resource-limit diagnostic.

Prediction question: what happens to |||? The lexer produces ||, then an error for the final |. It does not silently reinterpret the sequence.

7. Deriving the abstract syntax tree#

An abstract syntax tree records grammatical meaning while dropping punctuation that later stages do not need. We retain spans on every node because every later error needs provenance.

Recursive types need indirection. Without Box, an expression containing another expression would have infinite size.

pub type NodeId = u32;

#[derive(Clone, Debug)]
pub struct Expr {
	pub id: NodeId,
	pub span: Span,
	pub kind: ExprKind,
}

#[derive(Clone, Debug)]
pub enum ExprKind {
	Int(String),
	Bool(bool),
	Name(String),
	Unary(UnOp, Box<Expr>),
	Binary(BinOp, Box<Expr>, Box<Expr>),
	Assign(Box<Expr>, Box<Expr>),
	Call(Box<Expr>, Vec<Expr>),
	Block(Block),
	If { condition: Box<Expr>, then_block: Block, else_block: Block },
	Error,
}

#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub enum UnOp { Not, Neg, SharedBorrow, MutBorrow, Deref }

#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub enum BinOp { Add, Sub, Mul, Div, Eq, Ne, Lt, Le, And, Or }

#[derive(Clone, Debug)]
pub struct Block {
	pub span: Span,
	pub statements: Vec<Stmt>,
	pub tail: Option<Box<Expr>>,
}

#[derive(Clone, Debug)]
pub enum Stmt {
	Let { span: Span, mutable: bool, name: String, annotation: Option<TypeAst>, init: Expr },
	While { span: Span, condition: Expr, body: Block },
	Semi(Expr),
	Error(Span),
}

#[derive(Clone, Debug)]
pub enum TypeAst {
	Named { name: String, span: Span },
	Ref { mutable: bool, inner: Box<TypeAst>, span: Span },
	Error(Span),
}

Node IDs are assigned monotonically in source traversal order. They identify nodes only within one parsed file revision. They are not persistent definition IDs. Confusing those roles causes unstable caches later.

ExprKind::Error is not a user language construct. It is a recovery witness: the parser has already emitted a diagnostic, and later stages should avoid cascading complaints. rustc also uses error recovery values, though its concrete AST and error-guarantee machinery are much more sophisticated.

Counterfactual design: we could preserve every token in the AST. That is useful for formatters and IDE syntax trees, but noisier for semantic passes. FerrisC keeps the token vector separately. A production frontend often has both a concrete lossless tree and semantic AST.

8. Recursive descent and precedence climbing#

Recursive descent gives one function to each grammar level. Binary precedence can be implemented without ten nearly identical functions using precedence climbing. Higher binding power means an operator grips its operands more tightly.

fn infix(kind: &TokenKind) -> Option<(u8, u8, BinOp)> {
	Some(match kind {
		TokenKind::PipePipe => (1, 2, BinOp::Or),
		TokenKind::AmpAmp => (3, 4, BinOp::And),
		TokenKind::EqEq => (5, 6, BinOp::Eq),
		TokenKind::BangEq => (5, 6, BinOp::Ne),
		TokenKind::Lt => (7, 8, BinOp::Lt),
		TokenKind::Le => (7, 8, BinOp::Le),
		TokenKind::Plus => (9, 10, BinOp::Add),
		TokenKind::Minus => (9, 10, BinOp::Sub),
		TokenKind::Star => (11, 12, BinOp::Mul),
		TokenKind::Slash => (11, 12, BinOp::Div),
		_ => return None,
	})
}

impl Parser {
	fn parse_expr_bp(&mut self, min_bp: u8) -> Expr {
		let mut left = self.parse_prefix();
		loop {
			if self.at(&TokenKind::LParen) {
				left = self.parse_call(left);
				continue;
			}
			if self.at(&TokenKind::Eq) && min_bp == 0 {
				self.bump();
				let right = self.parse_expr_bp(0);
				left = self.expr_join(&left, &right, ExprKind::Assign(Box::new(left), Box::new(right)));
				continue;
			}
			let Some((left_bp, right_bp, op)) = infix(&self.current().kind) else { break };
			if left_bp < min_bp { break; }
			self.bump();
			let right = self.parse_expr_bp(right_bp);
			left = self.expr_join(&left, &right, ExprKind::Binary(op, Box::new(left), Box::new(right)));
		}
		left
	}
}

This is an excerpt: Parser, parse_prefix, parse_call, and expr_join are introduced around it in the assembled module. The ownership expression passed to expr_join should compute the joined span before moving left and right in concrete code. Showing that detail explicitly avoids pretending the pedagogical sketch compiles as-is.

The pair (9, 10) makes addition left associative. Parsing a - b - c first forms a - b, then subtracts c. Assignment uses equal recursive power and becomes right associative: a = b = c means a = (b = c). Type checking may still reject assignment as a value in contexts we disallow.

Prediction question: how does 1 + 2 * 3 group? Multiplication's left binding power exceeds the minimum established by addition, so the result is 1 + (2 * 3).

9. Parser mechanics and guaranteed progress#

The parser owns tokens, a cursor, diagnostics, and a node counter. It never indexes beyond EOF because the lexer appends exactly one EOF token.

pub struct Parser {
	tokens: Vec<Token>,
	pos: usize,
	next_node: NodeId,
	pub diagnostics: Vec<Diagnostic>,
}

impl Parser {
	fn current(&self) -> &Token { &self.tokens[self.pos] }

	fn bump(&mut self) -> Token {
		let token = self.tokens[self.pos].clone();
		if !matches!(token.kind, TokenKind::Eof) { self.pos += 1; }
		token
	}

	fn at(&self, expected: &TokenKind) -> bool {
		std::mem::discriminant(&self.current().kind) == std::mem::discriminant(expected)
	}

	fn expect(&mut self, expected: TokenKind, description: &'static str) -> Option<Token> {
		if self.at(&expected) { return Some(self.bump()); }
		self.diagnostics.push(Diagnostic::error(
			"E-PARSE-EXPECTED", self.current().span,
			format!("expected {description}"),
		));
		None
	}

	fn fresh_id(&mut self) -> NodeId {
		let id = self.next_node;
		self.next_node = self.next_node.checked_add(1).expect("per-file node limit");
		id
	}
}

Discriminant comparison deliberately ignores payloads in Ident(String). It is suitable for punctuation tests, but code extracting an identifier must pattern-match and clone its text. An explicit TokenTag enum is clearer in a hardened implementation.

The expect function does not consume an unexpected token. That permits an enclosing parser to decide recovery, but creates a danger: a loop may call expect forever. Every parsing loop records its starting cursor. If an iteration made no progress, it consumes one token into an error node unless at EOF.

fn recover_statement(&mut self) -> Span {
	let start = self.current().span;
	let mut end = start;
	while !matches!(self.current().kind, TokenKind::Semi | TokenKind::RBrace | TokenKind::Eof) {
		end = self.bump().span;
	}
	if matches!(self.current().kind, TokenKind::Semi) { end = self.bump().span; }
	start.join(end).unwrap_or(start)
}

Semicolon and right brace are synchronization tokens: they often mark a boundary after which parsing can resume. Consuming the right brace would steal it from the block parser, so recovery stops before it.

10. Parsing blocks, tails, and error nodes#

Blocks expose the semicolon distinction directly. After parsing an expression, the next token decides whether it is a statement or tail.

fn parse_block(&mut self) -> Block {
	let open = self.expect(TokenKind::LBrace, "`{`")
		.map(|t| t.span).unwrap_or(self.current().span);
	let mut statements = Vec::new();
	let mut tail = None;

	while !matches!(self.current().kind, TokenKind::RBrace | TokenKind::Eof) {
		let before = self.pos;
		if self.at(&TokenKind::Let) {
			statements.push(self.parse_let());
		} else if self.at(&TokenKind::While) {
			statements.push(self.parse_while());
		} else {
			let expr = self.parse_expr_bp(0);
			if self.at(&TokenKind::Semi) {
				self.bump();
				statements.push(Stmt::Semi(expr));
			} else {
				tail = Some(Box::new(expr));
				break;
			}
		}
		if self.pos == before {
			let span = self.recover_statement();
			statements.push(Stmt::Error(span));
		}
	}

	let close = self.expect(TokenKind::RBrace, "`}`")
		.map(|t| t.span).unwrap_or(self.current().span);
	Block { span: open.join(close).unwrap_or(open), statements, tail }
}

If a tail expression is followed by another token before }, the block parser should report “expected semicolon or closing brace” and synchronize. That check is omitted from the short excerpt but required in assembly.

An integer literal parser uses text.parse::<i64>(). On overflow it emits E-LIT-RANGE and returns ExprKind::Error. It must not substitute zero, because zero could create misleading constant-evaluation behavior.

Parser tests should snapshot a compact tree shape, not Rust's debug formatting, which changes whenever fields are added. For example, serialize 1 + 2 * 3 as (+ 1 (* 2 3)).

11. Token trees and bounded hygienic substitution#

Rust macros operate on token trees and have nuanced hygiene, fragment parsing, repetition, and edition behavior. FerrisC implements only a built-in twice!(expression) expander to teach phase ordering and fresh names. It is not macro_rules! compatibility.

The source form twice!(e) expands conceptually to:

{
    let __fresh = e;
    __fresh + __fresh
}

Evaluating e once matters. Naively substituting e + e duplicates side effects. The generated local must not capture a user's local with the same spelling.

Before normal parsing, we group balanced delimiters into token trees.

#[derive(Clone, Debug)]
pub enum TokenTree {
	Leaf(Token),
	Delimited { open: Span, close: Span, children: Vec<TokenTree> },
}

#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)]
pub struct SyntaxContext(pub u32);

#[derive(Clone, Debug, PartialEq, Eq, Hash)]
pub struct HygienicName {
	pub text: String,
	pub context: SyntaxContext,
}

Source identifiers use context zero. Each macro invocation receives a fresh nonzero context. Resolution compares both text and context for generated locals. References to the fresh local carry that same context. Tokens copied from the argument retain their original contexts.

This tiny rule prevents one form of accidental capture. It does not model Rust's definition-site versus call-site lookup, nested macro scopes, $crate, labels, or imported macros.

Expansion has hard bounds: maximum nesting 64, maximum output 100,000 tokens, and only the known macro name is accepted. The expander records an expansion backtrace mapping generated spans to the invocation span. Diagnostics can then say both where generated syntax failed and which invocation caused it.

Counterfactual design: text substitution before lexing loses token boundaries and spans. Replacing text inside comments or identifiers becomes possible. Token trees retain structure and make delimiter errors recoverable.

Prediction question: why does expansion happen before ordinary expression parsing? Because the parser needs to see the generated block as normal syntax. rustc's expansion pipeline is substantially more involved, but generated syntax likewise enters later semantic stages.

12. Items, scopes, and two namespaces#

FerrisC has a value namespace and a type namespace. Functions and locals live in values. primitive types, type parameters, and trait names live in types. The same text may therefore denote a type and a function without conflict.

A scope is a parent-linked map. Looking up a local starts in the innermost block and walks outward. Items are collected before bodies are resolved, so a function may call another function written later.

#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash, PartialOrd, Ord)]
pub struct DefId(pub u64);

#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub enum DefKind { Function, Local, TypeParam, Trait }

#[derive(Clone, Debug)]
pub struct Definition {
	pub id: DefId,
	pub kind: DefKind,
	pub name: HygienicName,
	pub span: Span,
}

#[derive(Default)]
pub struct Scope {
	pub parent: Option<usize>,
	pub values: std::collections::BTreeMap<HygienicName, DefId>,
	pub types: std::collections::BTreeMap<HygienicName, DefId>,
}

We use BTreeMap for deterministic iteration. Hash maps are fine for lookup, but randomized iteration can perturb diagnostics and generated symbol order.

Stable item definition IDs are derived from a crate identity, item kind, hygienic name, and deterministic disambiguator among same-name errors. Local IDs are derived from the containing function ID and lexical declaration index. Never derive IDs from raw byte offsets alone: inserting a comment would invalidate everything after it.

For this workshop, a fixed-key 64-bit FNV-1a hash is easy to implement, and collisions are detected by storing the full identity beside the hash. On collision, compilation reports an internal deterministic-ID error. Production rustc uses its own stable hashing and definition identity machinery.

fn stable_hash(parts: &[&[u8]]) -> u64 {
	let mut h = 0xcbf29ce484222325u64;
	for part in parts {
		for &byte in *part {
			h ^= byte as u64;
			h = h.wrapping_mul(0x100000001b3);
		}
		h ^= 0xff;
		h = h.wrapping_mul(0x100000001b3);
	}
	h
}

This hash is deterministic, not cryptographically secure. Do not use it for untrusted hash-table collision resistance.

Resolution stores NodeId -> DefId for every name expression. An unresolved name gets E-RESOLVE-NAME and an error resolution marker. A duplicate declaration gets one primary label on the duplicate and a secondary label on the first declaration. The first definition remains in the scope, preventing arbitrary “last error wins” behavior.

13. Resolving bodies without losing errors#

Resolution is a tree walk with explicit scope entry and exit. Each block allocates a child scope. A let initializer is resolved before its new name is inserted. Therefore let x = x; refers to an outer x, if one exists, rather than recursively referring to the uninitialized new binding. This is a deliberate rule matching the intuition of Rust local bindings.

#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub enum Res {
	Def(DefId),
	Error,
}

pub struct Resolution {
	pub names: std::collections::BTreeMap<NodeId, Res>,
	pub definitions: std::collections::BTreeMap<DefId, Definition>,
	pub diagnostics: Vec<Diagnostic>,
}

fn lookup_value(scopes: &[Scope], mut scope: usize, name: &HygienicName) -> Option<DefId> {
	loop {
		if let Some(&id) = scopes[scope].values.get(name) { return Some(id); }
		match scopes[scope].parent {
			Some(parent) => scope = parent,
			None => return None,
		}
	}
}

For a name expression, the resolver records either a definition or Res::Error. The error marker prevents type checking from emitting “cannot infer the type” for the same unknown name. One root cause should not become a wall of secondary errors.

Shadowing in nested scopes is accepted. Duplicates in the same namespace and same scope are rejected. A value fn Meter() and type parameter Meter can coexist because lookup always states which namespace it needs.

Prediction question: inside let x = 1; { let x = x + 1; x }, which definition does the initializer use? It uses the outer binding; the tail uses the inner binding.

rustc's resolver handles modules, imports, macros, labels, visibility, editions, and partial resolutions. FerrisC's two maps teach the essential distinction without claiming to reproduce that system.

14. Lowering AST into HIR#

The AST mirrors written syntax. High-level intermediate representation, or HIR, removes syntax distinctions that semantic analysis need not revisit. Lowering is a one-way translation, not mutation of the parser's tree.

FerrisC HIR resolves names to IDs, parses literal values, represents every block with an explicit tail, and desugars while into a loop plus conditional branch.

#[derive(Clone, Debug)]
pub struct HirExpr {
	pub id: NodeId,
	pub span: Span,
	pub kind: HirExprKind,
}

#[derive(Clone, Debug)]
pub enum HirExprKind {
	Int(i64),
	Bool(bool),
	Local(DefId),
	Function(DefId),
	Unary(UnOp, Box<HirExpr>),
	Binary(BinOp, Box<HirExpr>, Box<HirExpr>),
	Assign(DefId, Box<HirExpr>),
	Call(DefId, Vec<HirExpr>),
	Block(HirBlock),
	If { condition: Box<HirExpr>, yes: HirBlock, no: HirBlock },
	Loop { body: HirBlock },
	Break,
	Error,
}

#[derive(Clone, Debug)]
pub struct HirBlock {
	pub statements: Vec<HirStmt>,
	pub tail: Box<HirExpr>,
	pub span: Span,
}

An absent AST tail becomes a synthetic unit expression. FerrisC can represent unit as an empty block or a dedicated Unit variant; the dedicated variant is clearer and should be added in the assembled enum. This illustrates representation design: if many consumers ask whether a block is empty, make the semantic value explicit.

The source loop:

while condition { body }

lowers to this conceptual HIR:

loop {
    if condition { body } else { break; }
}

The generated nodes keep the original while span and carry a synthetic-origin flag in a complete implementation. That lets diagnostics prefer source syntax over invented syntax.

Why desugar here rather than in the parser? The parser should report syntax in the user's vocabulary. Why not wait for bytecode? Type checking and CFG construction benefit from fewer constructs. rustc similarly lowers AST to HIR and desugars constructs, although its HIR and lowering rules are far richer.

The HIR invariant says every non-error name is resolved, every literal fits its declared storage, and all implicit unit values are explicit. It does not yet say expressions are well typed.

15. Types, variables, and error absorption#

Types classify values and permitted operations. Our representation distinguishes known types, generic parameters, inference variables, references, function signatures, and an absorbing error type.

#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)]
pub struct TyVar(pub u32);

#[derive(Clone, Debug, PartialEq, Eq, Hash)]
pub enum Ty {
	I32,
	Bool,
	Unit,
	Param(DefId),
	Infer(TyVar),
	Ref { mutable: bool, inner: Box<Ty> },
	Fn { params: Vec<Ty>, result: Box<Ty> },
	Error,
}

Ty::Error unifies with every type without another diagnostic. It is evidence that an earlier error already poisoned this expression. It must never reach code generation as though it were a real value. The driver stops after semantic errors.

A type variable means “one type not known yet.” Repeated occurrences of the same variable must receive the same solution. For let x = if c { 1 } else { y };, the expected type of x can constrain both branches.

Function types are not first-class values in FerrisC. We still represent signatures with Ty::Fn because call checking needs a uniform shape. Name resolution ensures a call target is a function definition; arbitrary closure-like calls are excluded.

Reference types omit explicit regions, also called lifetimes. That omission is safe only because our borrow checker uses local CFG liveness and forbids escaping references. It is a major simplification from rustc's region inference.

Counterfactual design: using strings such as "&mut i32" for types seems easy. It makes substitution, structural comparison, and unification fragile. A recursive enum makes every legal form explicit.

16. Unification derived step by step#

Unification solves equality constraints between types. If variable ?0 must equal i32, we bind ?0 to i32. If &?0 must equal &i32, we recursively solve the inner equality.

The inference context stores optional variable values. Resolving a type follows variable links until reaching an unbound variable or a concrete type.

#[derive(Default)]
pub struct InferCtx {
	values: Vec<Option<Ty>>,
}

impl InferCtx {
	pub fn fresh(&mut self) -> Ty {
		let id = TyVar(self.values.len() as u32);
		self.values.push(None);
		Ty::Infer(id)
	}

	pub fn shallow_resolve(&self, mut ty: Ty) -> Ty {
		while let Ty::Infer(v) = ty {
			match self.values[v.0 as usize].clone() {
				Some(next) => ty = next,
				None => return Ty::Infer(v),
			}
		}
		ty
	}

	fn occurs(&self, needle: TyVar, ty: &Ty) -> bool {
		match self.shallow_resolve(ty.clone()) {
			Ty::Infer(v) => v == needle,
			Ty::Ref { inner, .. } => self.occurs(needle, &inner),
			Ty::Fn { params, result } => {
				params.iter().any(|p| self.occurs(needle, p)) || self.occurs(needle, &result)
			}
			_ => false,
		}
	}
}

The occurs check rejects an infinite type such as ?0 = &?0. FerrisC syntax rarely constructs that constraint, but the unifier should be correct independently of current callers.

impl InferCtx {
	pub fn unify(&mut self, left: Ty, right: Ty) -> Result<Ty, (Ty, Ty)> {
		let left = self.shallow_resolve(left);
		let right = self.shallow_resolve(right);
		if left == Ty::Error || right == Ty::Error { return Ok(Ty::Error); }
		match (left, right) {
			(Ty::Infer(a), Ty::Infer(b)) if a == b => Ok(Ty::Infer(a)),
			(Ty::Infer(v), ty) | (ty, Ty::Infer(v)) => {
				if self.occurs(v, &ty) { return Err((Ty::Infer(v), ty)); }
				self.values[v.0 as usize] = Some(ty.clone());
				Ok(ty)
			}
			(Ty::Ref { mutable: am, inner: a }, Ty::Ref { mutable: bm, inner: b })
				if am == bm => {
				let inner = self.unify(*a, *b)?;
				Ok(Ty::Ref { mutable: am, inner: Box::new(inner) })
			}
			(Ty::Fn { params: ap, result: ar }, Ty::Fn { params: bp, result: br })
				if ap.len() == bp.len() => {
				let params = ap.into_iter().zip(bp).map(|(a, b)| self.unify(a, b))
					.collect::<Result<Vec<_>, _>>()?;
				let result = self.unify(*ar, *br)?;
				Ok(Ty::Fn { params, result: Box::new(result) })
			}
			(a, b) if a == b => Ok(a),
			(a, b) => Err((a, b)),
		}
	}
}

Mutability must match exactly in unification. A separate coercion relation may convert &mut T to &T. Mixing equality and coercion would allow conversions in unintended places.

17. Bidirectional expression checking#

Inference works better when information flows both upward and downward. Synthesis asks an expression for its type. Checking supplies an expected type and verifies compatibility. Together they form bidirectional type checking.

pub struct Typeck {
	pub infer: InferCtx,
	pub node_types: std::collections::BTreeMap<NodeId, Ty>,
	pub diagnostics: Vec<Diagnostic>,
	pub locals: std::collections::BTreeMap<DefId, Ty>,
	pub signatures: std::collections::BTreeMap<DefId, Ty>,
}

impl Typeck {
	fn check(&mut self, expr: &HirExpr, expected: Ty) -> Ty {
		let actual = self.synth(expr);
		match self.coerce(actual.clone(), expected.clone()) {
			Ok(ty) => { self.node_types.insert(expr.id, ty.clone()); ty }
			Err(()) => {
				self.diagnostics.push(Diagnostic::error(
					"E-TYPE-MISMATCH", expr.span,
					format!("expected {expected:?}, found {actual:?}"),
				));
				self.node_types.insert(expr.id, Ty::Error);
				Ty::Error
			}
		}
	}
}

The debug formatting in this excerpt should be replaced by a stable type printer before diagnostics become a public contract.

Literal synthesis returns i32. Boolean synthesis returns bool. A local returns the type recorded at its definition. Arithmetic checks both operands against i32 and returns i32. Logical operations check bool and return bool. Comparisons require compatible operands and return bool.

For if, check the condition against bool. Create a fresh result variable. Check each branch against that variable. The resolved variable becomes the conditional's type. An absent branch is impossible after grammar lowering, because FerrisC requires else for expression conditionals.

A let with an annotation checks its initializer against that annotation. Without an annotation it synthesizes and records the initializer type. An unconstrained inference variable at function end receives E-TYPE-ANNOTATION, not an arbitrary default. FerrisC does not implement Rust's integer fallback rules.

Only one coercion exists: &mut T may become &T when an expected shared reference is present. There is no numeric widening, dereference coercion, never type, unsizing, or function-item coercion. Explicit limits make behavior teachable.

Prediction question: should if c { &mut x } else { &x } infer &x? With an expected shared reference, both branches can check. Without one, our simple synthesis may reject the mismatch. Real rustc computes richer coercion sites and least upper bounds.

18. Calls, assignments, and place categories#

Not every expression denotes a storage location. A place is something assignable or borrowable, such as a local variable. A value is the result obtained by reading a place or computing an operation. rustc MIR has a detailed Place with projections. FerrisC starts with local-only places.

#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub enum PlaceKind {
	Local(DefId),
	NotAPlace,
}

fn classify_place(expr: &HirExpr) -> PlaceKind {
	match expr.kind {
		HirExprKind::Local(id) => PlaceKind::Local(id),
		_ => PlaceKind::NotAPlace,
	}
}

Assignment first classifies its left side. If it is not a local place, emit E-ASSIGN-PLACE. If the local was not declared mutable, emit E-ASSIGN-IMMUTABLE with a secondary label on its declaration. Then check the right side against the local type. Assignment itself has type unit.

Borrowing also requires a local place. &mut x additionally requires mutable declaration. Type checking establishes shape and mutability permission; the later borrow checker establishes temporal non-aliasing. Those are distinct questions.

Call checking obtains the resolved function signature, checks argument count, then checks each argument against its corresponding parameter. Extra arguments are still type-checked for independent errors. Missing arguments have no expression span, so the closing parenthesis is the primary label.

This error-preserving behavior matters. Returning immediately after count mismatch would hide a useful type error in an extra argument. Conversely, generating fake missing expressions would produce nonsense spans.

19. A finite trait goal solver#

Traits express capabilities shared by types. FerrisC defines two built-in trait IDs, Addable and Ord, and reads finite impl declarations such as impl Ord for i32;. There are no user-defined trait bodies or methods. Operators generate goals.

#[derive(Clone, Debug, PartialEq, Eq, Hash)]
pub struct Goal {
	pub trait_id: DefId,
	pub ty: Ty,
}

#[derive(Clone, Debug)]
pub struct ImplEntry {
	pub trait_id: DefId,
	pub for_ty: Ty,
	pub where_goals: Vec<Goal>,
	pub span: Span,
}

#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub enum Certainty { Proven, NoSolution, Ambiguous }

The solver substitutes current inference values, finds impl heads that unify with the goal, and recursively proves their where-goals. Zero candidates means no solution. One successful candidate means proven. Multiple successful candidates mean ambiguous.

Unification during candidate probing must be transactional. A failed candidate may not leave bindings behind. The simplest implementation clones InferCtx for each candidate and commits only one uniquely successful clone. That is slower than a snapshot-and-rollback system, but obviously correct for a tiny database.

fn solve(
	goal: Goal,
	impls: &[ImplEntry],
	infer: &InferCtx,
	stack: &mut Vec<Goal>,
	depth: usize,
) -> (Certainty, Option<InferCtx>) {
	if depth > 32 { return (Certainty::Ambiguous, None); }
	if stack.contains(&goal) { return (Certainty::Ambiguous, None); }
	stack.push(goal.clone());
	let mut successes = Vec::new();
	for entry in impls.iter().filter(|i| i.trait_id == goal.trait_id) {
		let mut trial = infer.clone();
		if trial.unify(goal.ty.clone(), entry.for_ty.clone()).is_err() { continue; }
		let all = entry.where_goals.iter().all(|nested| {
			matches!(solve(nested.clone(), impls, &trial, stack, depth + 1).0, Certainty::Proven)
		});
		if all { successes.push(trial); }
	}
	stack.pop();
	match successes.len() {
		0 => (Certainty::NoSolution, None),
		1 => (Certainty::Proven, successes.pop()),
		_ => (Certainty::Ambiguous, None),
	}
}

The sketch needs Clone on InferCtx and substitution of variables inside nested goals in the assembled implementation. Its important contracts are candidate isolation, finite depth, and three-valued results.

An ambiguity is not silently accepted. At the end of type checking, unresolved obligations become diagnostics. A goal involving an unbound variable may become solvable later, so obligations are queued and retried after more unification.

Major omissions include coherence, orphan rules, specialization, negative impls, associated items, higher-ranked binders, auto traits, canonicalization, and coinduction. rustc's trait solver addresses fundamentally harder problems. FerrisC teaches goals, candidates, ambiguity, and cycle guards only.

20. Generic functions as substitution#

A generic function has a type parameter represented by Ty::Param(def_id). Calling it creates a fresh inference type for that parameter, substitutes the fresh type through parameters and result, and adds trait obligations from bounds.

For this signature:

fn min<T: Ord>(a: T, b: T) -> T

the call min(1, 2) creates ?0, checks both arguments against ?0, thereby binds ?0 = i32, and asks the solver to prove i32: Ord.

fn substitute(ty: &Ty, param: DefId, replacement: &Ty) -> Ty {
	match ty {
		Ty::Param(id) if *id == param => replacement.clone(),
		Ty::Ref { mutable, inner } => Ty::Ref {
			mutable: *mutable,
			inner: Box::new(substitute(inner, param, replacement)),
		},
		Ty::Fn { params, result } => Ty::Fn {
			params: params.iter().map(|p| substitute(p, param, replacement)).collect(),
			result: Box::new(substitute(result, param, replacement)),
		},
		other => other.clone(),
	}
}

FerrisC permits at most one type parameter per function. It has no turbofish syntax, const generics, generic impls, or lifetime parameters. The restriction keeps substitution visible.

Prediction question: what happens for min(1, true)? The first argument binds the fresh type to i32. The second requires bool = i32 and receives a mismatch. Trait solving is not the first failure.

21. From typed HIR to control-flow MIR#

Trees hide evaluation order and joins. A control-flow graph, or CFG, contains basic blocks connected by explicit edges. A basic block is a straight-line statement sequence ending in exactly one terminator.

FerrisC MIR uses numbered local slots. Slot zero is the return place, parameter slots follow, then source locals and compiler temporaries.

pub type Local = u32;
pub type BasicBlock = u32;

#[derive(Clone, Debug, PartialEq, Eq)]
pub enum Operand {
	Copy(Local),
	Const(Value),
}

#[derive(Clone, Debug, PartialEq, Eq)]
pub enum Rvalue {
	Use(Operand),
	Unary(UnOp, Operand),
	Binary(BinOp, Operand, Operand),
	Ref { mutable: bool, local: Local },
}

#[derive(Clone, Debug, PartialEq, Eq)]
pub enum MirStatement {
	Assign(Local, Rvalue),
	StorageLive(Local),
	StorageDead(Local),
}

#[derive(Clone, Debug, PartialEq, Eq)]
pub enum Terminator {
	Goto(BasicBlock),
	Switch { condition: Operand, yes: BasicBlock, no: BasicBlock },
	Call { function: DefId, args: Vec<Operand>, destination: Local, next: BasicBlock },
	Return,
	Unreachable,
}

#[derive(Clone, Debug)]
pub struct MirBody {
	pub locals: Vec<Ty>,
	pub blocks: Vec<MirBlock>,
	pub spans: Vec<Span>,
}

#[derive(Clone, Debug)]
pub struct MirBlock {
	pub statements: Vec<MirStatement>,
	pub terminator: Terminator,
}

Every operand is atomic. Complex HIR expressions are evaluated into temporaries from left to right. That ordering is part of FerrisC semantics. Real Rust has carefully specified evaluation behavior and rustc MIR has places, projections, constants, unwind edges, and more.

MIR construction must only run after successful type checking. Therefore local types are concrete, trait obligations are proven, and HIR error nodes are absent. Asserting those facts here is reasonable because the driver established them.

22. Building branches, loops, and calls#

The MIR builder tracks the current block and appends statements until setting its terminator. After termination, no statement may be appended. Creating blocks first and filling terminators later requires a temporary “unterminated” state; the final validator rejects any remaining one.

For if c { a } else { b }, create three blocks: then, else, and join. Evaluate c in the current block, terminate with a switch, evaluate each branch into the same destination local, then jump both branches to join.

entry: evaluate c
       switch c -> then, else

then:  destination = evaluate a
       goto join

else:  destination = evaluate b
       goto join

join:  continue with destination

A loop has header, body, and exit blocks. The lowered Break jumps to exit. The body back-edge jumps to header. This explicit back-edge is what dataflow algorithms need.

A call is a terminator rather than a statement. It transfers control to another body and resumes at next. FerrisC has no unwinding edge; division by zero and interpreter limits are run-time traps. Real MIR can model cleanup and unwinding.

Validation checks block indices, local indices, one terminator per block, operand types, destination types, and Boolean switch conditions. Run the validator after construction and after every optimization. An optimizer should not be trusted merely because its input was valid.

23. Dataflow as equations over a graph#

Dataflow computes facts at every program point. Each block has an input fact, a transfer function describing statements, and an output fact. Edges combine facts with a join operation. We iterate until no fact changes.

For definite initialization, the fact is a bit set of locals initialized on every path. The entry set contains parameters but not ordinary locals. Assignment inserts a local. StorageDead removes it. At a merge, intersection is correct: a local is definitely initialized only if initialized on all predecessors.

#[derive(Clone, Debug, PartialEq, Eq)]
pub struct Bits(Vec<bool>);

impl Bits {
	fn intersect_with(&mut self, other: &Bits) -> bool {
		let before = self.clone();
		for (a, b) in self.0.iter_mut().zip(&other.0) { *a &= *b; }
		*self != before
	}

	fn union_with(&mut self, other: &Bits) -> bool {
		let before = self.clone();
		for (a, b) in self.0.iter_mut().zip(&other.0) { *a |= *b; }
		*self != before
	}
}

Initialization is a forward analysis. Reading Copy(x) requires x in the current set. Check reads before applying the statement's write. For x = x + 1, the old x must already be initialized.

The initial value for non-entry blocks is the universal set, because intersection's identity is “everything.” Starting with empty would incorrectly lock results to empty.

Prediction question: after if c { x = 1; } else { }, is x definitely initialized? No. The join intersects {x} with {}.

Worklists make iteration efficient. When one block's output changes, enqueue only its successors. Because bit sets have finite height and transfer is monotone, iteration terminates.

24. Backward liveness#

A local is live at a point if its current value may be read along some future path before being overwritten. Liveness runs backward. The output of a block is the union of successor inputs. Union is appropriate because use on any path matters.

For each statement in reverse: remove its assigned destination from the live set, then add every operand it reads. For a call terminator, the destination is killed after the call, and arguments are generated before it.

The equations are:

live_out(block) = union(live_in(successor))
live_in(block)  = uses(block) union (live_out(block) minus defs(block))

Block summaries are enough for iteration, but borrow checking needs facts at each statement. After block-level convergence, walk each block backward once more and record the live set before every location.

Why calculate liveness if bytecode can run without it? It determines when a reference is no longer used, enables dead-store elimination, and provides excellent practice with rustc's many MIR dataflow analyses.

Counterfactual design: ending every borrow at lexical block end is simpler. It rejects code where a reference's last use occurs early. Liveness permits a modest non-lexical behavior, while still falling far short of Rust's region model.

25. An educational CFG borrow checker#

FerrisC's checker tracks loans created by Rvalue::Ref. A loan contains borrowed local, shared or mutable kind, and destination reference local. The loan is active wherever that destination is live.

#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub enum LoanKind { Shared, Mutable }

#[derive(Clone, Debug)]
pub struct Loan {
	pub borrowed: Local,
	pub reference: Local,
	pub kind: LoanKind,
	pub issued_at: Location,
}

#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)]
pub struct Location {
	pub block: BasicBlock,
	pub statement: u32,
}

At each program point, collect loans whose reference locals are live. Creating a shared loan conflicts with an active mutable loan of the same local. Creating a mutable loan conflicts with any active loan of the same local. Writing a borrowed local conflicts with any active loan. Reading a local conflicts with an active mutable loan, except through the reference operation modeled by dereference.

FerrisC presently supports dereference reads but not dereference writes. References cannot be copied into another reference local, passed to a function, returned, or selected across an if join. These restrictions make loan identity equal to one MIR destination slot.

Accepted example:

let mut x = 1;
let r = &x;
let y = *r;
x = 2;

The loan ends after r's last use, so assignment to x is accepted.

Rejected example:

let mut x = 1;
let r = &x;
x = 2;
let y = *r;

The future dereference keeps r live at the assignment. The checker reports the borrow creation, conflicting assignment, and later use that extends the loan.

Rejected but safe Rust subset: borrows returned from helper functions, disjoint field borrows, reference reborrowing, borrows carried through loops, and many branch-sensitive patterns. These are false positives caused by simplification.

Potential unsoundness boundary: if references could be copied or stored, ending a loan when only its original destination dies would be wrong. An alias could remain live. We close that hole by rejecting reference copies and escapes. If those features are later added, loan provenance must flow through assignments and calls.

Another boundary is calls that mutate globals or borrowed arguments. FerrisC has no globals and rejects reference arguments, so its call effects are isolated. Adding either feature without extending effect analysis would be unsound.

This checker is not Polonius, not rustc's borrow checker, and not a general proof of Rust lifetimes. It is a conservative checker for the precisely stated FerrisC subset. Its value is teaching how CFG locations, liveness, loans, and diagnostics interact.

26. Interpreting MIR and evaluating constants#

An interpreter gives MIR executable meaning before code generation exists. It is also a reference implementation for optimization and bytecode tests. The machine state contains a call stack. Each frame contains a body, current block, statement index, and local values.

#[derive(Clone, Debug, PartialEq, Eq)]
pub enum Value {
	I32(i64),
	Bool(bool),
	Unit,
	Ref { frame: usize, local: Local, mutable: bool },
	Uninit,
}

pub struct Frame {
	pub function: DefId,
	pub block: BasicBlock,
	pub statement: usize,
	pub locals: Vec<Value>,
	pub return_to: Option<(usize, Local, BasicBlock)>,
}

pub struct Machine<'a> {
	pub bodies: &'a std::collections::BTreeMap<DefId, MirBody>,
	pub stack: Vec<Frame>,
	pub fuel: u64,
}

Uninit should be unreachable after successful initialization analysis, but the interpreter checks it anyway. Independent validation catches compiler bugs close to their source.

Each machine step consumes one fuel unit. Fuel bounds infinite loops and malicious constant evaluation. When fuel reaches zero, the machine returns EvalError::Limit with the current MIR span. It does not guess a value.

#[derive(Clone, Debug, PartialEq, Eq)]
pub enum EvalError {
	Uninitialized(Local),
	DivisionByZero,
	Overflow,
	BadOperand,
	Limit,
	MissingBody(DefId),
}

fn eval_binary(op: BinOp, a: Value, b: Value) -> Result<Value, EvalError> {
	match (op, a, b) {
		(BinOp::Add, Value::I32(a), Value::I32(b)) =>
			a.checked_add(b).map(Value::I32).ok_or(EvalError::Overflow),
		(BinOp::Sub, Value::I32(a), Value::I32(b)) =>
			a.checked_sub(b).map(Value::I32).ok_or(EvalError::Overflow),
		(BinOp::Mul, Value::I32(a), Value::I32(b)) =>
			a.checked_mul(b).map(Value::I32).ok_or(EvalError::Overflow),
		(BinOp::Div, Value::I32(_), Value::I32(0)) => Err(EvalError::DivisionByZero),
		(BinOp::Div, Value::I32(a), Value::I32(b)) =>
			a.checked_div(b).map(Value::I32).ok_or(EvalError::Overflow),
		(BinOp::Eq, a, b) => Ok(Value::Bool(a == b)),
		(BinOp::Lt, Value::I32(a), Value::I32(b)) => Ok(Value::Bool(a < b)),
		(BinOp::And, Value::Bool(a), Value::Bool(b)) => Ok(Value::Bool(a && b)),
		(BinOp::Or, Value::Bool(a), Value::Bool(b)) => Ok(Value::Bool(a || b)),
		_ => Err(EvalError::BadOperand),
	}
}

Use checked arithmetic so host debug and release modes agree. FerrisC defines overflow as a trap. Real Rust's overflow behavior depends on operation and compilation settings; we choose one rule and enforce it consistently.

Constant evaluation reuses the same interpreter with stricter entry rules: no mutable references, no calls except functions marked constant by the driver, smaller stack and fuel limits, and no unresolved input. Reusing semantics reduces divergence, but mode checks remain explicit. rustc's const evaluator is vastly more capable and precise.

27. Optimizations as semantics-preserving rewrites#

Optimization changes representation without changing observable behavior. That sentence is a proof obligation, not a slogan. Our observable behavior is return value or exact trap category, not timing or allocation.

Constant folding replaces operations on constant operands. It must preserve traps. 1 / 0 cannot become an arbitrary constant, and overflow cannot wrap.

fn fold_rvalue(rv: &Rvalue) -> Rvalue {
	match rv {
		Rvalue::Binary(op, Operand::Const(a), Operand::Const(b)) => {
			match eval_binary(*op, a.clone(), b.clone()) {
				Ok(value) => Rvalue::Use(Operand::Const(value)),
				Err(_) => rv.clone(),
			}
		}
		_ => rv.clone(),
	}
}

Branch simplification turns a switch on constant true into Goto(yes) and constant false into Goto(no). Unreachable-block removal traverses from entry, retains visited blocks, renumbers them deterministically, and rewrites every edge.

Dead-store elimination uses liveness. An assignment to a dead local may be removed only when its rvalue cannot trap. Removing dead = 1 / x could remove division by zero, so arithmetic division is not considered pure-and-nontrapping. Constants, copies, and Boolean negation are safe under our semantics.

Counterfactual design: “the destination is unused, therefore remove the statement” confuses value usefulness with effectlessness. This exact distinction matters throughout real optimization work.

After each pass, validate MIR and run equivalence tests. For generated small inputs, interpret original and optimized bodies under the same fuel. Compare values and traps. Fuel exhaustion is inconclusive because optimization can change step counts; discard that case rather than declare inequality.

#[test]
fn folding_preserves_division_trap() {
	let rv = Rvalue::Binary(
		BinOp::Div,
		Operand::Const(Value::I32(1)),
		Operand::Const(Value::I32(0)),
	);
	assert_eq!(fold_rvalue(&rv), rv);
}

28. Monomorphizing one generic function#

Monomorphization turns a generic body into a body for concrete type arguments. FerrisC supports one type parameter, so an instance key is a function definition plus one concrete type.

#[derive(Clone, Debug, PartialEq, Eq, PartialOrd, Ord)]
pub struct Instance {
	pub function: DefId,
	pub argument: Option<Ty>,
}

pub fn mangle(instance: &Instance) -> String {
	let suffix = match &instance.argument {
		None => "mono".to_owned(),
		Some(Ty::I32) => "i32".to_owned(),
		Some(Ty::Bool) => "bool".to_owned(),
		Some(other) => format!("h{:016x}", stable_type_hash(other)),
	};
	format!("_FC_{:016x}_{suffix}", instance.function.0)
}

Starting from main, walk calls in deterministic block order. For each generic call, read its concrete type from type checking, create an instance, and enqueue it if unseen. Clone its MIR while substituting the parameter type in local declarations. Repeat until the worklist is empty.

The reachable-instance set must be finite. FerrisC rejects polymorphic recursion, where a generic function recursively calls itself with a structurally new type. Our type grammar lacks containers, so this mostly acts as a future-proof rule. A hard instance count limit also protects the compiler.

Trait bounds have already been proved for each call. Because traits have no methods, there is no dictionary or vtable to generate. The monomorphization layer exists to teach instance discovery and substitution. rustc monomorphization includes much more, including codegen units, drop glue, vtables, and cross-crate concerns.

Prediction question: if min<i32> is called ten times, how many bodies are generated? One, because the Instance key deduplicates it.

29. Stack bytecode generation#

Our executable target is stack bytecode. Locals remain indexed slots, while expression operands are pushed onto an operand stack. Labels become instruction offsets after assembly.

#[derive(Clone, Debug, PartialEq, Eq)]
pub enum Instr {
	Push(Value),
	Load(Local),
	Store(Local),
	Add, Sub, Mul, Div, Eq, Lt, Not,
	Jump(u32),
	JumpIfFalse(u32),
	Call(u32, u16),
	Return,
	Trap(&'static str),
}

#[derive(Clone, Debug)]
pub struct ByteFunction {
	pub name: String,
	pub local_count: u32,
	pub code: Vec<Instr>,
}

To compile destination = a + b, emit code loading a, then code loading b, then Add, then Store(destination). The bytecode validator tracks stack height through control flow. At every join, incoming heights must agree. No instruction may pop below zero.

MIR blocks first receive symbolic labels. Emission records each label's instruction index. A second pass replaces symbolic jump targets with checked u32 offsets. This is assembly in miniature.

Calls refer initially to Instance symbols. Module assembly sorts instances by mangled name, assigns function-table indices, and patches call operands. An unknown symbol becomes a linker diagnostic. Duplicate symbols are rejected even if bodies happen to match.

A traditional native linker combines object files, resolves symbols, lays out sections, and applies relocations. FerrisC's function-table patching is a small analogue. It does not implement object formats, machine ABIs, or relocation types.

Pseudo assembly is useful for traces:

function _FC_main_mono locals=3
L0:
    push 40
    push 2
    add
    store 0
    return

30. A query database and memoization#

A query is a pure-looking function from a stable key to a value. Examples are tokens(file), ast(file), hir(function), typeck(function), and mir(instance). Memoization stores a query result so repeated requests avoid repeated work.

The database owns source inputs and caches. Values use Arc so callers share immutable results.

use std::collections::{BTreeMap, BTreeSet};
use std::sync::Arc;

#[derive(Clone, Debug, PartialEq, Eq, PartialOrd, Ord)]
pub enum QueryKey {
	Tokens(FileId),
	Ast(FileId),
	Hir(DefId),
	Typeck(DefId),
	Mir(Instance),
}

#[derive(Default)]
pub struct Database {
	pub sources: BTreeMap<FileId, Arc<SourceFile>>,
	pub cache: BTreeMap<QueryKey, Arc<QueryValue>>,
	pub dependencies: BTreeMap<QueryKey, BTreeSet<QueryKey>>,
	pub active: Vec<QueryKey>,
}

QueryValue is an enum holding typed result variants. A typed wrapper checks that Tokens receives a token value, so callers do not downcast arbitrary objects. In a larger Rust design, separate typed cache fields can avoid this enum.

When query A requests query B, record edge A -> B. If B is already active, the request forms a query cycle. Return a cycle diagnostic listing the active suffix. Do not recurse until stack overflow.

Memoization is safe only if all relevant inputs appear in the key or are dependencies whose revisions are checked. Reading an environment variable behind the database's back makes the cache unsound. Target configuration, compiler options, and language version therefore become explicit input keys.

rustc's query system supports sophisticated dependency tracking, parallelism, red-green evaluation, and on-disk caching. FerrisC implements the smallest model that makes invalidation visible.

31. Incremental invalidation and clean-build equivalence#

When a source file changes, mark its token query dirty. Use reverse dependency edges to recursively invalidate dependents. Unaffected cached queries remain. Recomputing a query first clears its old outgoing edges, then records dependencies from the new execution.

The simplest safe fallback is coarse invalidation: changing any item header invalidates resolution and every body, while changing a function body invalidates only that function's HIR onward. If the parser cannot reliably distinguish header from body after errors, choose coarse invalidation. Extra work is preferable to stale answers.

Every cache entry stores: query schema version, compiler build version, target configuration hash, input fingerprint, and serialized result checksum. Unknown versions are cache misses, not attempts at best-effort decoding.

Our first implementation keeps cache only in memory. Persistence is a later hardening exercise because serialization creates compatibility and corruption concerns.

Clean-build equivalence is the central test:

1. Compile revision A in a fresh database.
2. Edit A into revision B and incrementally compile.
3. Compile B in another fresh database.
4. Compare diagnostics, MIR, bytecode, and runtime result.

Ignore explicitly nondeterministic trace timings, but compare stable IDs and ordering exactly. Generate edits at token boundaries: rename a local, change a literal, insert whitespace, alter a function signature, and introduce then repair syntax errors.

Prediction question: should inserting a comment change bytecode? No. Spans may move, but stable item IDs and semantic output should remain.

32. Stage traces and observability#

Learning and debugging improve when stages can explain themselves. FerrisC supports deterministic text traces for tokens, AST, resolution, HIR, types, trait goals, MIR, dataflow, instances, and bytecode.

A trace event is data before formatting.

#[derive(Clone, Debug)]
pub struct TraceEvent {
	pub stage: &'static str,
	pub key: String,
	pub message: String,
	pub span: Option<Span>,
}

pub trait TraceSink {
	fn event(&mut self, event: TraceEvent);
}

pub struct NoTrace;
impl TraceSink for NoTrace {
	fn event(&mut self, _: TraceEvent) {}
}

Passing a sink makes tracing explicit and testable. Sensitive source text is not included by default. A production server may compile private code, so logs should contain IDs and counts unless users opt in.

Metrics include stage duration, cache hits and misses, token count, MIR block count, solver candidate count, and limit failures. Durations are useful operationally but excluded from golden files.

Diagnostic rendering and traces serve different audiences. A user diagnostic says how to fix source. A trace explains compiler decisions. Do not turn internal dumps into normal error messages.

33. Driver, CLI, and compilation modes#

The driver orders stages and enforces stop conditions. It is the only layer that reads files, interprets command-line options, and chooses output formatting.

#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub enum Mode {
	Check,
	Run,
	EmitMir,
	EmitBytecode,
	ConstEval,
}

pub struct Options {
	pub mode: Mode,
	pub input: std::path::PathBuf,
	pub trace: bool,
	pub fuel: u64,
}

pub fn compile(options: &Options) -> Result<Compilation, Vec<Diagnostic>> {
	// Each named call below is a query boundary in the assembled compiler.
	let source = read_source(&options.input)?;
	let tokens = tokens(&source);
	let expanded = expand(tokens)?;
	let ast = parse(expanded)?;
	let resolved = resolve(&ast)?;
	let hir = lower(&ast, &resolved)?;
	let typed = type_check(&hir)?;
	let mir = build_and_check_mir(&typed)?;
	finish_mode(options, mir)
}

This orchestration excerpt intentionally omits concrete result adapters and does not compile in isolation. Its lesson is stage ordering and explicit error gates. The central structures and algorithms around it are implementable Rust, not one opaque source dump.

check stops after borrow checking. run monomorphizes and interprets or executes bytecode. emit-mir prints validated MIR even without optimization unless requested. emit-bytecode writes deterministic assembly text. const-eval evaluates a named zero-argument function under stricter rules.

ferrisc check program.fc
ferrisc run program.fc --fuel 1000000
ferrisc emit-mir program.fc
ferrisc emit-bytecode program.fc -o program.fbc

Exit status zero means successful requested action. Source diagnostics use status one. Invalid CLI usage uses status two. An internal compiler failure uses a distinct status and prints a request for a bug report without exposing private source.

Argument parsing can use std::env::args_os. Reject unknown flags and missing values. Use OsString for paths so non-UTF-8 Unix paths are not mangled. FerrisC otherwise stays standard-library-only.

34. Testing from examples to fuzzing#

Unit tests isolate local contracts. Lexer tests assert kinds and UTF-8 boundaries. Parser tests assert precedence and recovery progress. Unifier tests assert symmetry and occurs checks. Dataflow tests use hand-written diamonds and loops. Interpreter tests exercise every trap.

Golden tests compare stable human-readable outputs in checked-in files. Use them for diagnostics, HIR, MIR, and bytecode. A deliberate update command rewrites goldens; ordinary test runs never do. Normalize path separators but not semantic ordering.

Property tests can be written without dependencies. Use a deterministic linear congruential generator and print the seed on failure.

#[derive(Clone)]
struct Rng(u64);

impl Rng {
	fn next(&mut self) -> u64 {
		self.0 = self.0.wrapping_mul(6364136223846793005).wrapping_add(1);
		self.0
	}

	fn below(&mut self, n: u64) -> u64 {
		assert!(n > 0);
		self.next() % n
	}
}

Generate bounded well-typed expression trees. Pretty-print them, parse them, interpret unoptimized MIR, optimize, interpret again, and compare outcomes. The generator tracks requested type, so Boolean contexts receive Boolean expressions.

Model tests compare dataflow against exhaustive path enumeration on tiny acyclic CFGs. The exhaustive model is slow but obviously direct. Agreement builds confidence in the worklist algorithm.

Differential tests compare two independent implementations. For the arithmetic subset, compare the HIR evaluator, MIR interpreter, and bytecode VM. Do not compare FerrisC directly with rustc unless syntax and overflow semantics genuinely overlap. Differences in language contract are not compiler bugs.

Fuzzing means feeding many generated or mutated byte sequences and asserting no panic, hang, or excessive allocation. A dependency-free smoke fuzzer can mutate a corpus under strict sizes. Run lexer and parser on arbitrary String::from_utf8_lossy data. For semantic stages, fuzz valid AST constructors and validate preconditions.

Shrinking makes failures understandable. Try deleting token ranges, replacing expressions with literals, and removing CFG blocks while the failure persists. Store the minimal seed as a regression test.

35. Architecture and ownership boundaries#

A practical first layout keeps one crate and splits concepts into modules:

src/
  main.rs          CLI and exit statuses
  driver.rs        pipeline and modes
  source.rs        files, spans, source map
  diagnostic.rs    structured diagnostics and rendering
  lexer.rs         tokens and UTF-8 lexing
  token_tree.rs    delimiter grouping and expansion
  ast.rs           syntax representation
  parser.rs        recursive descent and recovery
  resolve.rs       namespaces, scopes, definition IDs
  hir.rs           lowered representation
  typeck.rs        inference, coercion, obligations
  traits.rs        finite goal solver
  mir.rs           CFG structures and validator
  dataflow.rs      initialization and liveness
  borrowck.rs      restricted loan checker
  interpret.rs     MIR execution and constant mode
  optimize.rs      MIR transformations
  mono.rs          instance collection
  bytecode.rs      emission, assembly, VM
  query.rs         cache and dependency graph
  trace.rs         structured stage events

The source map owns source strings. Syntax nodes own compact strings initially; an interner can later reduce duplication. HIR owns semantic nodes and references definitions by IDs, never by borrowing AST nodes. MIR owns its locals and blocks. Queries share immutable results through Arc.

Avoid one global mutable compiler context. It obscures dependencies and prevents parallel queries. Pass narrow read-only contexts, return owned results, and keep mutable inference state local to one function body.

Split crates only after module dependencies stabilize. A sensible future split is ferrisc_syntax, ferrisc_middle, ferrisc_codegen, and ferrisc_driver. Crate boundaries speed conceptual enforcement but add compile time and public API maintenance.

rustc itself is a large workspace of crates with explicit query-driven layers. Names and exact boundaries differ, yet ownership questions transfer directly: who interns data, who may emit diagnostics, which representation crosses a crate, and which IDs remain stable?

36. Production hardening and hostile environments#

Correctness on small friendly files is only a beginning. Every untrusted dimension needs a limit: source bytes, tokens, delimiter depth, AST nodes, scope depth, type recursion, trait candidates, solver depth, MIR blocks, monomorphized instances, call stack, interpreter fuel, diagnostic count, and output bytes.

Limits produce structured errors and stop expensive dependent work. After the diagnostic cap, emit one “further errors omitted” note. Do not allocate one diagnostic per character in a gigabyte input.

Cancellation is a shared atomic flag checked at bounded intervals. Long lexer loops, solver candidate loops, dataflow worklists, optimization passes, and interpretation all cooperate. Cancellation returns a distinct outcome, not a source error and not a cached query result.

Determinism requires sorted iteration, stable hashes, explicit locale-independent formatting, fixed optimization order, and no timestamps in artifacts. Build twice under different hash seeds and compare bytes.

Persistent caches need versioned schemas, checksums, atomic write-then-rename, file locking, size budgets, and corruption fallback. Treat cache bytes as untrusted. Never deserialize unchecked lengths into allocations. A cache failure should trigger recomputation, not change program meaning.

Security includes path traversal in output names, symlink races, terminal escape characters in source snippets, denial-of-service recursion, and secret leakage in logs. Escape control characters in diagnostics. Write only beneath an explicitly allowed output directory. Run generated programs in a sandbox if bytecode gains I/O.

Observability should answer which stage consumed resources without recording private source. Use request IDs, stage counters, limit names, cache statistics, and sampled durations. Keep trace volume bounded.

Internal invariants use validators and controlled assertions. Panics at trust boundaries become bug reports, but source-triggered malformed states return diagnostics. Fuzz each boundary specifically to verify that distinction.

37. Capstone: add the remainder operator end to end#

The capstone adds integer remainder written %. Before editing, predict every affected stage. This discipline prevents “parser accepts it but backend crashes” changes.

Language contract: remainder has the same precedence and associativity as multiplication. Both operands are i32. Zero divisor traps. The signed result follows Rust's checked_rem behavior, including overflow for the minimum integer remainder by negative one according to the chosen checked operation.

Lexer: add Percent and recognize one ASCII byte. Test its span beside a multibyte source character.

Parser: add BinOp::Rem and map Percent to binding powers (11, 12). Predict 8 % 3 * 2 as (8 % 3) * 2 because equal-precedence operators associate left.

AST and HIR: the shared BinOp flows through unchanged. If separate operator enums are used, both conversion matches need new arms. Exhaustive Rust matches help locate omissions.

Type checking: check both operands as i32 and return i32. If arithmetic is trait-driven, decide whether Rem requires a new trait. For this capstone it is primitive-only, so no trait database change is needed.

MIR: the existing binary rvalue carries the operator. MIR validation permits Rem only on integer operands.

Interpreter and constant evaluator:

(BinOp::Rem, Value::I32(_), Value::I32(0)) => Err(EvalError::DivisionByZero),
(BinOp::Rem, Value::I32(a), Value::I32(b)) =>
	a.checked_rem(b).map(Value::I32).ok_or(EvalError::Overflow),

Optimization: constant folding reuses eval_binary, so successful constants fold and traps remain. Dead-store elimination classifies remainder as potentially trapping.

Bytecode: add Instr::Rem, emission mapping, stack validation effect “pop two, push one,” VM execution, text assembly spelling, and decoder version bump if binary persistence exists.

Queries: changing compiler code invalidates persistent caches through build version. No query key shape changes. Golden traces will change only for programs using %.

Tests: add lexing, precedence, type mismatch, zero trap, minimum-value edge, constant folding, MIR versus VM differential cases, and incremental clean-build equivalence.

Diagnostics: ensure % no longer receives E-LEX-CHAR. The zero divisor runtime span should cover the operator or whole expression, according to one documented convention.

This list is the real capstone skill. Compiler contribution means tracing one semantic choice through representations, validators, execution, caching, and tests.

38. Exercises and review answers#

Exercise A: spans#

Explain why byte offsets and character columns are both needed. Then modify SourceFile to reject files larger than u32::MAX bytes before any span is built.

Answer: byte offsets slice UTF-8 strings and compactly identify boundaries. Character columns support human-facing locations. The source-map constructor should return Result<SourceFile, Diagnostic> after checking byte length, then build line starts with checked conversion.

Exercise B: recovery#

Predict the tokens and parser progress for let x = @@@; fn good() { 1 }.

Answer: the lexer emits three error tokens and advances over each @. The parser creates an error initializer, synchronizes at the semicolon, and still reaches fn good. Diagnostic deduplication may combine adjacent invalid characters, but must not skip the later item.

Exercise C: associativity#

Change subtraction's binding powers from (9, 10) to (9, 9). What tree does 10 - 3 - 2 produce?

Answer: equal right binding power permits the next subtraction inside the right operand, producing 10 - (3 - 2). That changes the result from five to nine, showing that precedence metadata defines semantics.

Exercise D: namespace resolution#

Should a local named Ord hide the trait Ord in a bound?

Answer: no. The local occupies the value namespace, while a bound performs type-namespace lookup. Two explicit maps prevent accidental cross-namespace shadowing.

Exercise E: expected types#

Why can checking a branch against an expected type succeed when synthesizing both branches first might fail?

Answer: the expected type supplies information before unconstrained choices are finalized. Our &mut T to &T coercion occurs only at a checking site. Without context, two independently synthesized reference mutabilities need not unify.

Exercise F: unification#

Construct a direct unit test for the occurs check.

Answer: create fresh ?0, then ask to unify it with &?0. Expect an error and verify the variable remains unbound. If failed unification mutates state, transactional callers will receive contaminated inference.

Exercise G: trait ambiguity#

Insert two identical impl Ord for i32; entries. Should the solver choose the first?

Answer: no. Choosing by source order hides incoherence and makes editing reorder semantics. The finite solver returns ambiguous, while item collection should preferably diagnose duplicate impl heads earlier.

Exercise H: initialization joins#

Why is intersection used for definite initialization while union is used for liveness?

Answer: definite means true on every incoming path, so one missing path removes the fact. Live means a future read may occur on any outgoing path, so one using path adds the fact. The words “every” and “any” reveal the join.

Exercise I: borrow checking boundary#

What becomes unsound if let s = r; copies a reference but loans are tied only to r's liveness?

Answer: r may die while s remains usable. The checker would end the loan early and permit mutation of the borrowed local before dereferencing s. Either reject the copy, as FerrisC does, or propagate loan provenance to s.

Exercise J: optimization#

Can x = 1 / 0 be removed when x is dead?

Answer: no. The expression traps, and traps are observable FerrisC behavior. Dead destination does not imply effect-free computation.

Exercise K: monomorphization#

Why does the instance key include concrete type, not only function definition?

Answer: different substitutions can produce different local layouts, operators, and callees. Deduplicating only by function would incorrectly share incompatible bodies.

Exercise L: incremental safety#

An optimization option is read from an environment variable but absent from query keys. What failure can occur?

Answer: a cached optimized MIR result may be reused after the variable changes. Incremental output then differs from a clean build. Make the option an explicit database input or clear all affected caches when it changes.

Exercise M: cancellation#

Where should a solver check cancellation?

Answer: before expanding each goal, inside large candidate loops, and before recursive descent. Checking only at query entry leaves one pathological query uninterruptible. The check interval should be bounded but not dominate normal cost.

Exercise N: rustc transfer#

When reading an unfamiliar rustc pass, which questions should you ask first?

Answer: identify its input representation, output representation, query key, declared dependencies, invariants assumed, invariants established, error recovery value, and tests that observe behavior. Then locate where spans and stable IDs originate. These questions scale better than memorizing module names.

Final review#

You began with bytes and ended with executable instructions. At each boundary, the representation discarded irrelevant detail and made a new invariant explicit. Tokens made UTF-8 boundaries safe. AST made grammar structure explicit. Resolution replaced names with identities. HIR removed surface sugar. Type checking proved local semantic constraints. MIR exposed evaluation order and control flow. Dataflow summarized paths. The restricted borrow checker enforced its stated alias rules. The interpreter defined behavior. Optimization preserved that behavior. Monomorphization selected concrete bodies. Bytecode made execution mechanical. Queries made dependencies observable.

The most important expert habit is not knowing every rustc type by memory. It is asking what is guaranteed here, who established that guarantee, how malformed input is represented, and which test would catch a broken assumption. FerrisC is intentionally much smaller than Rust, but that habit transfers directly to serious compiler work.

Assembly checklist#

When turning the progressive excerpts into one crate, begin with source, diagnostics, lexer, and parser. Compile and test that vertical slice before adding semantic analysis. Next add item collection and resolution, then HIR lowering and type checking. At that milestone, the check mode can already provide substantial value. Add MIR construction and its validator together; never allow invalid MIR to become the interpreter's problem. Initialization, liveness, and restricted borrow checking come next. Only then add execution, optimization, monomorphization, and bytecode.

Resolve excerpt-level differences deliberately. Add the explicit HIR unit variant noted during lowering. Give lexer helper functions concrete iterator types. Compute joined spans before moving boxed expressions. Derive Clone for inference state used by trait candidate probing. Substitute inference values into nested trait goals. Replace debug type output with a stable printer. Define the omitted container types such as function items, HIR statements, query values, and compilation results according to the invariants established in their sections. These are small integration choices, not permission to skip a stage.

After each module arrives, run its focused unit tests and one malformed-input test. Once MIR exists, run the validator after construction and after every pass. Once two execution engines exist, start differential testing immediately. Once caching exists, run clean-build equivalence for every edit scenario.

Keep a stage contract document beside the code. For each representation, record legal variants, whether error nodes are allowed, who owns spans, which IDs are stable, and whether values may contain inference variables. If a new feature weakens a contract, update every consumer before accepting syntax for that feature.

A completed educational compiler still has stated boundaries. It does not become rustc because all FerrisC modes work. Its success criterion is stronger understanding: you can locate a violated invariant, explain why a representation carries each field, predict downstream effects of a language change, and distinguish a safe conservative rejection from an unsound acceptance.

Part XI: The Art, Philosophy, and Mastery of rustc#

1. Compilation as evidence-preserving translation#

A compiler is often pictured as a machine that turns text into machine code. That picture names the endpoints but misses the intellectual work between them. rustc repeatedly changes the form of a program while preserving evidence about what it means. Each phase establishes facts, records enough justification for later phases, and rejects claims it cannot support. Translation is therefore closer to a chain of audited arguments than to changing a file format.

source claim
    | syntax evidence
    v
resolved and typed claim
    | control-flow and ownership evidence
    v
optimized executable behavior

Suppose x += f() is lowered too early into an ordinary assignment without preserving evaluation rules. The output might evaluate x or its projections a different number of times. The representation changed successfully, yet the translation failed as an argument about behavior.

Normative language guarantee: accepted safe Rust must obey the language's safety and behavioral rules. Current implementation: rustc distributes the supporting evidence among syntax structures, HIR, types, THIR, MIR, query results, and metadata. Version note: names, ownership, and exact contents of those structures are implementation details, especially around rustc 1.97.1.

Ask of every pass: what proposition does its input already justify? What new proposition does the pass establish? What evidence may it erase without making a future answer a guess? Those questions are the beginning of compiler mastery.

2. Representations make questions cheap#

A representation is useful when it makes the next important questions inexpensive and hard to answer incorrectly. Source text makes “what did the programmer type here?” cheap. It makes “which declaration does this name denote?” expensive. A resolved identifier reverses that trade: identity becomes cheap, spelling and textual context become secondary.

Imagine keeping only one magnificent universal tree. Every consumer would skip irrelevant cases, rediscover invariants, and defend against states left by unrelated phases. Its apparent convenience would charge interest through the entire compiler. A smaller representation can encode an invariant by making invalid alternatives unrepresentable.

enum ParsedCallee {
	Name(String),
	Field(Box<ParsedExpr>, String),
	Error,
}

struct CheckedCall {
	callee: DefId,
	args: Vec<TypedOperand>,
	result: Ty,
}

The first form is appropriate while syntax is uncertain. The second is appropriate once lookup and typing have succeeded. Asking code generation to interpret ParsedCallee would duplicate front-end reasoning. Asking error recovery to manufacture a valid CheckedCall would lie about established facts.

Representation design is thus complexity placement. Do we pay once while constructing a control-flow graph, or repeatedly while every analysis reconstructs successors from nested syntax? Do we canonicalize types once, or repeatedly compare trees modulo aliases and inference variables?

Socratic test: if a pass contains many searches through unrelated nodes, is its algorithm poor, or is its input representation making the desired question expensive? Another test: if every consumer checks the same condition, should construction establish it once? Mastery means noticing when algorithmic difficulty is really representational debt.

3. Many IRs and the discipline of forgetting#

Rust needs multiple intermediate representations because no single view serves all questions equally well. AST-like syntax retains punctuation, ordering, and incomplete constructs. HIR expresses a resolved, desugared view suited to much semantic analysis. THIR exposes typed expression structure useful for lowering and checks. MIR presents explicit places, operands, statements, terminators, and control flow. Back-end IRs expose machine-oriented operations and target constraints.

rich surface detail                         machine detail
        AST -> HIR -> THIR -> MIR -> monomorphized code -> backend IR
          \       semantic facts       /               target facts

This is not a ladder from “bad” to “good.” Each representation is authoritative for different questions. MIR is better for dataflow, but worse for reproducing the programmer's exact nesting. Tokens are ideal for macro input, but poor for deciding whether a trait obligation holds.

Deliberate forgetting is a design skill. Desugaring a for loop removes a surface construct and reveals iteration operations. That simplification lets later analyses avoid special-casing every loop spelling. Yet spans and expansion information must retain enough provenance to avoid blaming mysterious generated code.

A counterexample clarifies the limit. If lowering erases whether an operation came from unsafe source context before safety checking uses that fact, later code cannot recover the distinction by inspecting machine behavior. The compiler has forgotten evidence too soon.

Current implementation near 1.97.1: exact lowering paths and data types should be checked in that revision's source and rustc-dev-guide. Do not memorize a diagram as an eternal API. Memorize the reason for the boundaries: expose facts when needed, then retire distinctions whose obligations are discharged.

4. Provenance is part of diagnostic truth#

Correct rejection is not sufficient for a humane compiler. rustc must connect a failed proof to the source decisions that caused it. That connection is provenance: where syntax came from, which expansion produced it, which obligation caused another obligation, and which inference choice constrained a value.

fn consume(_: String) {}

fn example(name: String) {
	consume(name);
	println!("{name}");
}

The essential fact is not only “use after move.” A useful diagnostic identifies the consuming call, the later use, and perhaps a repair such as borrowing when valid. If lowering keeps only an unlocated move bit, the checker can reject correctly but explain poorly. Diagnostic quality was lost upstream, not in the final wording routine.

Provenance resembles a chain of custody. An investigator does not merely possess an object; they show how it reached the evidence room. Likewise, a compiler should distinguish source-written code from macro-generated code, primary causes from downstream consequences, and confident suggestions from speculative ones.

Normative guarantee: diagnostics are generally not the semantic definition of Rust, and exact wording is not usually a stable language contract. Quality obligation: regressions in spans, labels, and suggestions still matter deeply to users and tooling. Implementation fact: expansion contexts, spans, and cause codes evolve; inspect rustc 1.97.1 rather than assuming old field names.

Ask before deleting metadata: could a later phase need this to identify responsibility? Ask before presenting metadata: does this fact explain the user's mistake, or merely the compiler's route through it? Good diagnostics require both preserved evidence and editorial judgment.

5. Explicitness, convenience, and hidden work#

Compiler APIs often trade explicitness for convenience. A helper that accepts a compiler context and returns a type may hide interning, query execution, normalization, allocation, caching, diagnostics, or even a dependency edge. Convenience is valuable, but hidden work makes cost and authority harder to reason about.

let ty = tcx.type_of(def_id).instantiate_identity();

This compact line is not conceptually a field read. Depending on context, obtaining the value may invoke query machinery and require earlier facts. The syntax should not seduce a contributor into assuming constant cost or absence of side effects in the architectural sense.

At the opposite extreme, passing every arena, table, cache, diagnostic sink, and configuration option separately would bury intent under plumbing. rustc contexts bundle capabilities because coherent access is useful. The danger is treating capability as permission to reach across phase boundaries.

Counterexample: a formatting method that unexpectedly normalizes a type and triggers substantial solving. Calling it inside a hot diagnostic loop could multiply work and create cycles. A more explicit API or naming convention might expose that cost.

Socratic questions improve reviews. Does this helper merely project stored evidence, or establish new evidence? Can it emit an error? Can it execute another query? Is its result valid in every phase where the type permits calling it? Would an explicit parameter reveal an accidental dependency?

6. Local reasoning under global language rules#

Rust code feels local: a function body, an impl, a module. Language rules are often global: coherence spans crates, name resolution spans modules, and trait selection may depend on impls visible through dependencies. The compiler must provide local units of work without pretending global constraints disappear.

trait Render { fn render(&self); }

impl<T: Render> Render for Vec<T> {
	fn render(&self) { /* ... */ }
}

Whether another implementation overlaps cannot be decided from this body alone. Yet checking the method body should not scan every crate repeatedly. rustc separates globally established environments and identities from local analyses that consume them.

The orphan and coherence rules are examples of global restrictions that preserve ecosystem reasoning. They constrain who may add implementations so separate crates do not silently make method selection ambiguous. The exact rule is normative where specified; the indexing strategy used to enforce it is implementation.

Local reasoning is therefore earned through architecture. A query key can name one item, but its result may depend on crate-wide collections. A type-checking routine can focus on one body because resolution, signatures, and environments arrive as trusted inputs. Locality is a contract about prepared context, not proof that the language rule itself is local.

Counterexample: checking each impl independently and accepting it if it looks internally valid. Two individually valid impls may overlap together. No amount of cleaner local code repairs the missing global comparison.

Ask: what is the smallest unit whose result can be valid independently? Which global facts must be frozen before that unit is checked? Could adding an unrelated item change the answer? If yes, the dependency architecture must represent that possibility.

The goal is not to eliminate global reasoning. It is to isolate it, summarize its conclusions, and make consumers depend on those conclusions honestly.

7. Five obligations, no single scoreboard#

rustc serves at least five competing obligations: soundness, compatibility, diagnostics, compile time, and generated-code quality. They cannot be reduced to one score without hiding value judgments. A change can improve one while harming another.

Soundness is asymmetric. Accepting an unsound safe program can invalidate every safe abstraction above it. Rejecting a sound program is also a defect, but usually not a memory-safety breach. This asymmetry justifies conservative choices under genuine uncertainty.

Compatibility includes more than old source continuing to parse. It includes behavior users rely on, target support, metadata interactions, and carefully managed language evolution. An optimization that changes observable semantics is not “better code generation.”

Diagnostics consume compile time and compiler complexity. Computing a perfect suggestion might require speculative solving or repeated analysis. Conversely, a very fast rejection that points nowhere externalizes cost onto every user. Engineering chooses a sensible frontier rather than declaring one axis absolute.

proposal: retain extra provenance through MIR
benefit:  clearer source labels
cost:     memory and pass complexity
risk:     new stale-provenance bugs
question: can a smaller cause summary provide most of the value?

Generated-code quality itself has dimensions: run time, size, latency, debugability, and predictability across targets. Inlining may speed one workload while increasing code size and compile time.

When reviewing a patch, enumerate obligations explicitly. Which safety invariant changes? Which accepted programs or outputs change? What diagnostic evidence improves or degrades? What work moves onto a common path? What happens to code size and performance?

Maturity is refusing the slogan “correctness first” as a substitute for analysis. Correctness establishes a floor; the compiler still owes users a balanced, measurable design.

8. Uncertainty and conservative decisions#

Compilers regularly know that a proposition is true, false, or not yet decidable. Collapsing “unknown” into either Boolean answer is a common source of bugs. Inference variables, opaque types, trait obligations, and incomplete error recovery all create uncertainty.

Proven     the available environment entails the goal
Refuted    the environment rules the goal out
Ambiguous  more information or a future choice may decide it

Suppose an optimizer cannot prove two references are disjoint. Treating lack of proof as proof of aliasing is safe but may miss an optimization. Treating lack of proof as disjointness can miscompile the program. The conservative action depends on the consumer's obligation.

In diagnostics, the polarity can differ. If a suggestion is only possibly valid, rustc should avoid marking it machine-applicable. It may still explain the possibility in qualified language. Uncertainty must survive long enough to control confidence.

fn choose<T>(x: T) {
	// An obligation involving T may remain undecidable
	// until bounds or inference provide more information.
	drop(x);
}

Conservatism is not synonymous with rejection. A phase may defer a decision, register an obligation, or use a symbolic value. Premature rejection loses programs that later evidence could validate. Premature acceptance spends evidence the compiler does not possess.

Error recovery adds another category: a value may be tainted by an earlier error. Propagating a designated error type can suppress cascades, but it must not accidentally certify unsafe code or poison unrelated analysis.

Ask: unknown to whom, and until when? What future event could resolve the uncertainty? Which action is safe if it never resolves? Is ambiguity a language-level possibility or an implementation's temporary lack of information?

Precise uncertainty is knowledge. It tells the architecture where decisions may occur and prevents wishful reasoning from masquerading as proof.

9. Phase boundaries are trust boundaries#

A phase boundary should state what is accepted, what is produced, and which invariants become available. Without that contract, defensive checks spread everywhere or, worse, disappear everywhere. The boundary is where one subsystem earns another subsystem's trust.

unresolved syntax --[resolution contract]--> stable definition identities
typed body       --[MIR lowering contract]--> explicit control flow and places
validated MIR    --[codegen contract]------> target operations preserving semantics

Trust does not mean inputs are eternally perfect. Malformed user programs travel through recovery paths. Internal assertions should distinguish impossible states from expected error states. An ICE for ordinary invalid source is evidence that a boundary contract was misstated or violated.

Boundary placement controls architecture. If name lookup leaks into code generation, codegen must understand scopes, imports, and hygiene. If target layout decisions leak into parsing, the front end ceases to be target-independent. Good boundaries narrow vocabulary as well as data access.

A useful counterexample is a pass that accepts either pre-normalized or normalized types “for convenience.” Every branch must then ask which world it inhabits. Requiring one form at entry can centralize normalization and make downstream assumptions reviewable.

Version-sensitive warning: rustc's actual pass ordering and query boundaries are not fixed by Rust's language contract. Around 1.97.1, verify comments and callers in the source before relying on remembered diagrams. Documentation from an older compiler may express the philosophy while naming obsolete boundaries.

Socratic review asks: who validates this invariant once? Can a caller bypass validation? How does error-tainted input cross the boundary? What source provenance must accompany the lowered fact? Would moving this operation earlier make the needed evidence unavailable?

Strong boundaries do not merely organize files. They organize justified belief.

10. Every fact needs an owner#

When several components can mutate or independently derive the same fact, disagreement becomes possible. Compiler architecture works better when each fact has an owner: the subsystem authorized to establish it, invalidate it, and define its meaning.

A parser owns syntactic structure, not the final identity of a path. Resolution owns that identity, while type checking owns inferred expression types. A later phase may cache or summarize those facts, but should not silently create a competing interpretation.

fact:       expression E has type T
authority:  type checking under parameter environment P
key:        body identity plus relevant compiler inputs
consumers:  MIR lowering, diagnostics, later analyses
not owner:  pretty-printer guessing from syntax

Ownership also applies to invalidation. If a command-line option changes target layout, all results depending on layout must be recomputed or keyed accordingly. A cache that stores the fact without its authority's relevant inputs invents stale truth.

Duplicated facts are sometimes necessary for performance. The discipline is to mark one canonical and others derived. Derived copies need construction and consistency rules. “These fields should normally match” is not an invariant; it is a future bug report.

Consider source locations after macro expansion. Different views may hold call-site and definition-site spans. These are not competing owners if their meanings differ explicitly. The bug appears when both fields claim to answer the same question without a selection rule.

Ask of any table: what semantic sentence does an entry assert? Which inputs qualify that sentence? Who may insert it? Can recovery create a placeholder, and how is that placeholder distinguished from proof?

Fact ownership turns a sprawling compiler into a society of accountable authorities. Without it, debugging becomes diplomacy between contradictory caches.

11. Identities are not locations#

A source offset tells where text appeared; it does not provide a durable semantic identity. Moving an item by one line changes its location without changing its role in the program. Conversely, two macro expansions may point near the same text while denoting distinct generated entities.

rustc uses several kinds of identifiers because identity has several domains. A definition identity, a HIR node identity, a local MIR index, and a source span answer different questions. Converting among them should be explicit and justified.

fn first() {}
fn second() {}

If a tool identifies second only as “the function beginning at byte 14,” adding a comment before it destroys the association. A semantic identifier can remain meaningful within the compilation even as positions shift. Cross-session stability requires still stronger design and cannot be assumed from an in-memory index.

Pointer identity is also seductive. Two equal interned values may share an address today, but persistence, serialization, and deterministic output require semantic keys rather than accidental allocation locations. Addresses are locations in one process, not portable names.

Implementation fact: rustc IDs often have carefully limited scopes and conversion APIs. Not a guarantee: exact numeric values or debug formatting need not remain stable between compiler runs or versions. Version note: consult 1.97.1 definitions before assuming whether an ID is local, stable, indexed, or hash-derived.

An identity should encode the equality relation consumers need. Should two uses of the same generic definition compare equal before substitution? Should two monomorphized instances compare equal afterward? There is no universal ID that answers both without context.

Mastery begins when “where is it?” and “what is it?” stop sounding like the same question.

12. Interning and arenas: lifetime as architecture#

Interning stores one canonical representative for repeatedly used equal values. Arenas allocate many values with a shared lifetime and reclaim them together. Both strategies reduce allocation overhead, but their deeper value is architectural: they express sharing and lifetime assumptions in the shape of data.

Types recur throughout a crate. If every occurrence owns a separately allocated tree, equality and cloning become expensive. Interned forms permit compact references and cheap reuse, provided construction enforces canonicalization and the interner outlives every reference.

construct type --> canonicalize --> interner table
                                      | shared handle
                         +------------+------------+
                         v                         v
                    expression A                obligation B

An arena suits phase data that dies together. Rather than negotiate individual object lifetimes and frees, the compiler ties many nodes to a context lifetime. This is not merely “faster malloc”; it makes illegal escape visible to Rust's type system where possible.

The costs deserve equal attention. Interners retain values for their owning lifetime, so accidental cardinality growth consumes memory. Canonical handles can tempt code to rely on pointer equality where semantic equality was intended. Arena allocation makes individual reclamation difficult and can hide unexpectedly long retention.

Counterexample: interning every transient diagnostic string. Most strings may never repeat, yet all survive until the interner dies. The strategy adds lookup cost and retention without useful sharing.

Ask whether equality is structural, canonical, or identity-based. Ask whether values truly share one lifetime. Ask how many distinct values adversarial source can create. Ask whether serialization needs a stable encoding independent of allocation order.

Around rustc 1.97.1, concrete arena and interned type APIs remain implementation details. The enduring lesson is to treat allocation strategy as a statement about ownership, equality, and phase lifetime.

13. Queries are dependency architecture#

rustc's query system is not just a memoization utility. It names computations by keys, records dependencies, detects certain cycles, and provides a vocabulary for deciding which facts can be requested independently. The query graph is an executable account of compiler causality.

type_of(ItemA) ----> predicates_of(ItemA)
       |                     |
       v                     v
type_check(BodyA) ---> trait obligations

A query boundary can improve locality and incremental reuse. It can also conceal poor granularity. One crate-wide query invalidated by a tiny body edit may be correct but expensive. Thousands of tiny queries may spend more on coordination and hashing than they save.

Important correction: current rustc has its own custom query engine. It is not simply an application built by embedding Salsa. Salsa is useful comparative background for incremental computation, but substituting its APIs or terminology for rustc's actual engine produces false explanations.

Current implementation near 1.97.1: query providers, keys, feeding, serialization, and incremental behavior must be verified against that compiler's source. Language guarantee: none of this query organization is promised to Rust programs. Architectural principle: dependency edges must reflect every input that can change a result.

Counterexample: a query reads an ambient mutable flag without declaring it as tracked input. The cache may return a result computed under another flag value. Memoization has converted an invisible dependency into incorrect evidence.

Socratic questions: is the key sufficient to identify the proposition? Is the result deterministic from tracked inputs? Can requesting it recurse to itself through another route? Does failure become a cached value, a diagnostic, or a poisoned computation? What granularity matches likely edits?

Think of queries as constitutional boundaries among facts, not fashionable wrappers around functions.

14. Laziness meets whole-crate duties#

Queries encourage demand-driven work: compute a fact only when a consumer requests it. Laziness avoids spending time on irrelevant paths and naturally supports memoization. But a compiler also has whole-crate duties that no local consumer may happen to request.

An unused function can still contain an error that Rust requires rustc to report. An impl can violate coherence even if no method call selects it. Metadata and exported symbols may need collection regardless of local demand. Therefore “nobody asked” cannot mean “the language obligation vanished.”

fn unused() {
	let x: u32 = "not a number";
}

fn main() {}

A purely demand-driven compiler starting at main might never inspect unused. rustc must arrange eager roots or crate-wide traversals so required checking occurs. The internal computations may remain lazy while a driver deliberately demands all mandatory results.

The balance resembles a library inspection. Readers request particular books lazily, but the building still performs a complete fire-safety inspection. Demand determines optional service, not institutional obligation.

Over-eagerness has costs too. Forcing code generation for generic items never instantiated wastes work. Performing expensive optimization before determining reachability can inflate compile time. The right root set depends on the semantic duty of each phase.

Ask: if no query requests this check, can an invalid crate be accepted? If yes, establish an explicit forcing edge. If no, can the work remain lazy? Does “unused” mean semantically irrelevant, or merely unreachable at run time?

Laziness is an evaluation strategy, not a philosophy of neglect.

15. Caching is a proof obligation#

A cache asserts that two requests may reuse one answer. That assertion requires proof that all answer-affecting inputs are equal under the cache key. Without such proof, a cache is a source of plausible stale lies.

result = compile(item, target, features, options, upstream metadata)

unsafe key: item
safer key: semantic identity plus fingerprints of every relevant input

The challenge is not merely listing obvious parameters. Ambient compiler options, target data layout, environment values deliberately read by compilation, macro inputs, and upstream crate metadata may all matter. A hidden dependency that bypasses tracking can survive ordinary testing because clean builds remain correct.

Incremental correctness should be tested by comparison with a clean rebuild. Change one input, reuse what the system believes reusable, and ask whether diagnostics and artifacts match the clean result. Performance measurements matter only after semantic equivalence is credible.

Caching failures have several forms. Under-invalidation reuses an obsolete answer. Over-invalidation is correct but loses performance. Unstable fingerprints make unchanged work appear changed. Non-deterministic serialization can alternate between all three symptoms.

Error results require policy. Can a failure be reused? Did it emit a diagnostic that must not be duplicated or lost? Was it caused by an input represented in the key? Caching only successes may be simpler but can repeat expensive doomed work.

Implementation fact near 1.97.1: rustc's on-disk incremental format and fingerprint details are private and version-sensitive. Engineering principle: cache validity is part of correctness, not an optimization afterthought.

Before adding a cache, write the semantic sentence that makes reuse legal. If that sentence cannot be stated precisely, the key is not ready.

16. Determinism is controlled causality#

Given equivalent inputs and configuration, compiler output should not depend accidentally on thread scheduling, hash-table iteration, allocator addresses, or filesystem discovery order. Determinism improves reproducible builds, debugging, caching, tests, and user trust.

use std::collections::HashMap;

// Iteration order is not a semantic ordering contract.
let mut symbols = HashMap::new();
symbols.insert("beta", 2);
symbols.insert("alpha", 1);

If emission walks this map directly, output order may vary. Sorting at a deliberate boundary or using an order defined by semantic identity makes causality explicit. The point is not that every map must be ordered; only externally significant choices require stable order.

Parallel compilation creates another trap. The first worker to report an error may vary, so diagnostic order can become scheduling order. Buffering, stable sorting, or carefully designed emission points can recover predictability, though each choice costs memory or latency.

Determinism does not require identical binaries across different rustc versions or targets. Those are different inputs. Nor does it mean all addresses at run time are fixed. It means accidental implementation noise should not influence outputs that claim reproducibility.

Counterexample: assigning persistent IDs from arena insertion order while constructing nodes in parallel. Even if each run is semantically correct, fingerprints churn and caches miss. Using stable semantic ingredients breaks the dependence on race order.

Ask which outputs are observed: bytes, diagnostics, symbol names, metadata, fingerprints, timings. Which variations are intentional? Which input explains each intentional variation? Can a randomized stress run expose an unexplained one?

Determinism is the art of ensuring that every visible difference has a visible cause.

17. Bootstrap and the self-hosting mirror#

rustc is written largely in Rust, so building it confronts a circular story: the compiler is needed to compile the compiler. Bootstrap breaks the circle with stages built by an existing compiler and then rebuilt by newer artifacts.

downloaded stage0 compiler
          |
          v
build newer compiler and libraries
          |
          v
rebuild or validate with the newly built toolchain

Self-hosting is impressive, but it is not proof of correctness. A compiler bug can compile itself successfully. The compiler exercises only some language behavior, and matching outputs may reproduce a shared mistake. Tests, specifications, differential checks, and reasoning remain necessary.

Bootstrap introduces practical constraints. The source cannot immediately require arbitrary features unsupported by the designated stage0 toolchain. Build scripts must distinguish host, build, and target platforms. Libraries and compiler artifacts must come from compatible stages.

The three-platform vocabulary matters in cross compilation. The build machine runs the build process. The host platform runs the compiler being produced. The target platform runs code that compiler generates. They may coincide on a laptop and diverge in serious toolchain work.

./x.py check compiler
./x.py test tests/ui

These commands illustrate workflow, not a timeless exact recipe. Bootstrap flags, stage defaults, and repository scripts around rustc 1.97.1 should be read from that checkout. Do not infer stage semantics from an old blog post.

Socratic question: when two stages differ, is the cause compiler semantics, library contents, build configuration, target behavior, or nondeterminism? Bootstrap mastery means tracing artifact ancestry rather than merely rerunning the build.

18. Macros are staged programming and a trust boundary#

Macros execute part of the programmer's intent before ordinary type checking of the expanded program. They are staged programming: one program constructs fragments that become input to later compiler stages. This changes provenance, name lookup, diagnostics, and the security posture of the build.

macro_rules! make_answer {
	($name:ident) => {
		fn $name() -> u32 { 42 }
	};
}

make_answer!(answer);

Declarative macros transform token structures under matching and hygiene rules. Procedural macros are compiled programs invoked during compilation. Their execution can consume resources and, under the surrounding process environment, interact with capabilities available to them. Using an untrusted procedural macro is therefore not analogous to importing inert type declarations.

Hygiene prevents simplistic textual substitution from accidentally capturing every nearby name. Yet hygiene is not invisibility: generated definitions and uses still participate in later resolution under defined contexts. Provenance must connect failures inside expansions to useful call-site and definition-site information.

Counterexample: report only the span of a generated token deep in a derive expansion. The location may be technically accurate but useless to the user. Blaming only the macro invocation can also hide an actionable mistake in macro input. Good diagnostics navigate both stages.

Normative territory: macro matching, expansion behavior, and hygiene include language-defined rules. Implementation territory: expansion data structures, scheduling, and span encoding evolve. Security reality: compiler acceptance does not certify that a procedural macro is safe to execute.

Ask which stage owns an error. Was malformed input supplied to the macro, did expansion generate invalid Rust, or did valid generated Rust fail a later semantic rule? Staged programming demands staged accountability.

19. Trait solving as reasoning under uncertainty#

Trait solving asks whether goals follow from implementations, bounds, normalization rules, and the current environment. It is not a dictionary lookup. Generic parameters, associated types, recursion, coherence, and inference turn selection into logical search under uncertainty.

trait Measure { type Unit; }

fn compare<T>(a: T, b: T)
where
	T: Measure + PartialEq,
{
	let _ = a == b;
}

The solver may prove obligations from explicit bounds without knowing concrete T. Other goals remain ambiguous until inference chooses types. Recursive impl patterns require cycle and overflow policies. A “no answer yet” must not be confused with a proof that no implementation exists.

Canonicalization can abstract inference details so logically equivalent goals share work. Environments delimit assumptions. Candidate assembly gathers possible routes; confirmation tests whether one route satisfies nested obligations. Those concepts endure even as rustc's concrete solver changes.

Historical precision matters. Chalk strongly influenced Rust trait-system modeling and solver design, but it should be described as a historical and now-sunset influence, not as a library simply embedded by current rustc. Current rustc around 1.97.1 has its own evolving implementation and transition history. Consult that revision before naming default modes, modules, or unsupported corners.

Counterexample: cache T: Trait without including the parameter environment. The same textual goal may be provable inside one function's bounds and ambiguous outside it. The missing environment turns context-sensitive reasoning into a false global fact.

Ask whether a result is unique, merely possible, or forced by all valid future substitutions. Ask whether failure is definitive under Rust's openness and coherence rules. Ask which diagnostics can honestly be derived from an overflow or ambiguity.

Trait solving teaches a broad lesson: sophisticated compilers preserve the shape of doubt until the language authorizes commitment.

20. Mastery is the ability to follow evidence#

Compiler mastery is not memorizing every rustc module. The repository changes, names migrate, solvers evolve, and optimizations are replaced. Mastery is the ability to reconstruct why a fact exists, who owns it, which representation makes it available, and which consumers may trust it.

For work near rustc 1.97.1, begin with version-matched source and documentation. Label a claim as a normative Rust guarantee, a current implementation fact, or history. When uncertain, design an experiment or trace a caller rather than promoting memory into authority. Dump an IR, minimize a program, compare clean and incremental builds, and read the relevant query edge.

surface symptom
    -> locate the deciding phase
    -> identify the representation and fact owner
    -> state the invariant or uncertainty
    -> trace provenance backward and consumers forward
    -> test the smallest counterexample

The most useful habit is to ask paired questions. What does this representation reveal, and what has it forgotten? What does this helper simplify, and what work does it hide? What can be checked locally, and which global fact licenses that locality? What does the cache save, and what equivalence makes reuse sound?

Borrow checking is the natural next proving ground. It combines identities with locations, types with control flow, local body analysis with language-wide aliasing rules, and conservative decisions with demanding diagnostics. MIR makes places, moves, and branches cheap to ask about, while provenance carries failures back toward source expressions.

The handoff is therefore not to a disconnected subsystem. It is to a concentrated example of everything developed here: evidence-preserving translation, deliberate representations, explicit uncertainty, phase contracts, owned facts, dependency-aware computation, and humane explanation.

Follow a borrow error from source through lowered control flow and back to its diagnostic. Then follow an accepted borrow as a proof whose evidence eventually permits code generation. Doing that carefully is more than learning one checker. It is practicing the broader art of understanding rustc without confusing today's machinery for tomorrow's language.

21. Borrow checking is a useful conservative argument#

Rust promises memory safety for accepted safe programs, not acceptance of every safe program. That distinction explains both the power and frustration of borrow checking. The checker proves properties from a finite representation of code. It must reject when its proof is insufficient, even if execution would happen safely. A rejection can therefore reveal danger, imprecision, or missing expressiveness. These explanations should not be collapsed into “the checker is confused.”

Consider an index known to the programmer but opaque to the analysis.

fn choose(xs: &mut [i32], i: usize, j: usize) -> (&mut i32, &mut i32) {
    assert!(i != j);
    // (&mut xs[i], &mut xs[j]) // two indexing operations appear to overlap
    let (a, b) = xs.split_at_mut(j.max(i));
    if i < j { (&mut a[i], &mut b[0]) } else { (&mut b[0], &mut a[j]) }
}

The library operation carries a proof boundary the checker understands. Its unsafe implementation establishes disjointness once; callers use a safe abstraction. The lesson is not “use unsafe whenever analysis loses.” The lesson is to package dynamic facts behind reviewed, narrow interfaces.

Ask three questions about every rejection. What aliasing or lifetime fact must hold? Where is that fact represented in the compiler's model? Could accepting this case make an actually dangerous neighbor pass?

Counterfactually, an omniscient checker could simulate every input and schedule. It would solve undecidable problems and would not terminate reliably. A practical checker chooses predictable approximation, useful errors, and compilation speed. Conservatism is therefore an engineering budget, not a defect to eliminate absolutely.

22. NLL and Polonius are different ways to model facts#

Non-lexical lifetimes made borrow extent follow use rather than block shape. That phrase names user-visible progress, but not one eternal implementation algorithm. Current rustc lowers relevant expressions to MIR and performs region reasoning there. The old lexical contrast remains pedagogically useful, while details have evolved.

Polonius frames major parts of borrow checking as relations and derived facts. It can express loans, origins, control-flow points, and subset relationships explicitly. This differs from merely shortening a lifetime at its last textual use. The models can agree on common programs while differing on difficult control flow.

MIR facts ──> region/loan analysis ──> errors
   │                │
   │                └── implementation strategy can change
   └── program points, uses, outlives relations, moves

Around rustc 1.97.1, integration status is version-sensitive. Do not claim that full Polonius is universally the default borrow checker. Some Polonius-informed work may be integrated or tested without replacing every path. Consult the current rustc development guide, tracking issues, and checkout flags. Distinguish “the research model can derive this” from “stable rustc accepts this today.”

A useful comparison table prevents slogan-driven explanations.

Socratic prompt: if two engines reject the same program, are their explanations equivalent? No; provenance, diagnostic spans, performance, and future extensibility can still differ.

23. Optimization must preserve observable meaning#

Optimization is not permission to produce a faster nearby program. Each transformation must preserve meaning under Rust's language and machine contracts. The hard phrase is “observable meaning,” especially around unsafe code and concurrency.

pub fn sum(xs: &[u32]) -> u32 {
    xs.iter().copied().fold(0, u32::wrapping_add)
}

Vectorization may change instruction count and grouping. It may not silently replace wrapping arithmetic with overflow assumptions. Likewise, dead-store removal depends on whether a write is volatile, atomic, or observable.

Think of optimization as a chain of proof obligations.

Rust source meaning
      │ lowering preserves contract
      v
MIR ── transform ──> optimized MIR
      │ encode valid assumptions
      v
backend IR ── transform ──> machine code

Case study: a bounds check disappears after proving an index is in range. Hypothesis A says MIR range information justified removal. Hypothesis B says LLVM inferred the condition after inlining. Hypothesis C says the check remains but assembly inspection missed another path. Compare MIR, backend IR, and complete control flow before assigning credit.

“It produced correct output once” is weak optimization evidence. Use differential tests, sanitizer configurations where supported, codegen tests, and benchmarks. For unsafe-sensitive changes, seek the relevant aliasing and validity model. Optimization quality includes compile time and code size, not runtime alone.

24. The backend boundary is a contract, not a blame line#

rustc performs parsing, type checking, MIR construction, and Rust-specific analysis. It then communicates code and assumptions to a codegen backend, commonly LLVM. LLVM selects and optimizes machine-level operations for many targets. The boundary moves over time and alternative backends expose its real shape.

When generated code is wrong, “LLVM bug” is only a hypothesis. rustc may have emitted invalid IR, wrong attributes, or unsound alias information. LLVM may miscompile valid IR. A target specification, linker, assembler, runtime library, or test may be wrong.

Reduce while retaining the failing layer. Source reduction is useful until it destroys the emitted pattern. IR reduction is stronger evidence for a backend report. Assembly comparison can identify the first divergent instruction, not its cause.

Counterfactual design: rustc owns every target optimization itself. That offers control but duplicates decades of backend work and target maintenance. Another design emits portable C, gaining portability while losing precise semantics and diagnostics. Architecture is tradeoff allocation, not proof that one component is smarter.

25. Diagnostics are empathetic technical interfaces#

A compiler error meets a person whose intended model just failed. Empathy means reducing uncertainty without pretending to know their intent. It is compatible with rigor: name the violated rule, show evidence, offer honest next steps.

A useful diagnostic answers four different questions. Where did the relevant events occur? What relationship made them incompatible? Why does the rule protect the program? Which changes might establish a valid relationship?

Span selection is interface design. The primary span should identify the decisive operation. Secondary labels should narrate causality in reading order. Notes explain persistent constraints, including destructor or closure behavior. Long type names belong in supporting output when they obscure the conflict.

Case study: a borrow appears to outlive a loop iteration. One hypothesis is genuine retention in a collection. Another is a temporary whose drop scope extends unexpectedly. A third is analysis imprecision around control flow. The error should expose observed constraints, not announce guessed motivation.

Test diagnostics as communication, not snapshots alone. Read them without the patch context. Check narrow terminals, macro expansions, repeated errors, and translated prose assumptions. An error that is technically correct but sends users toward unsafe code is harmful.

26. Recovery and suggestions must tell the truth#

Parser and type-error recovery let compilation continue after an earlier fault. Recovery builds placeholders or alternate interpretations so later analysis can proceed. Its output is provisional evidence, not a faithful program. Secondary errors should be suppressed when they merely echo poisoned state.

The earliest error is not automatically the root cause. Yet recovery quality improves when causal provenance is tracked. A synthetic node should not masquerade as confident user syntax. Delayed bugs are for violated compiler assumptions, not hiding ordinary user errors.

Suggestions have an applicability contract. Machine-applicable edits require high confidence in syntax and semantics. Placeholders tell tools and users that choices remain. Maybe-incorrect suggestions should explain uncertainty. Help text without an edit is often more honest than a seductive patch.

Bad:  replace `x` with `&x`       [always]
Better: consider borrowing `x` here
        this may require changing the callee's ownership contract

Counterfactual recovery could stop after one error. That maximizes causal purity but creates slow edit-compile cycles. Unlimited speculative recovery produces noise and unstable diagnostics. The practical design balances useful continuation against confidence decay.

27. Stability is a promise maintained by machinery and evidence#

Stable Rust promises more than successful parsing on today's compiler. It includes specified language behavior, stable library APIs, and compatibility expectations. Implementation details, diagnostic wording, and performance have different promise levels. State which level a claim concerns.

Feature gates isolate incomplete or undecided behavior on nightly. A gate is not a quality badge or a shortcut around design review. Tracking issues collect unresolved questions and implementation work. Stabilization requires explicit judgment about semantics, teaching, and maintenance.

Editions allow opt-in incompatible surface changes while preserving ecosystem coexistence. They do not split Rust into unrelated languages. Crates of different editions interoperate through the same broader ecosystem. Migration tooling should make mechanical changes where confidence permits.

Ecosystem testing catches interactions unit tests cannot anticipate. Crater-style experiments compare many real crates across compiler versions. A regression can be intentional tightening, accidental rejection, miscompilation, or tooling breakage. Counts need triage; one severe soundness failure can outweigh many harmless warnings.

28. rustc internals are deliberately not a stable library API#

rustc is decomposed into crates, but crate boundaries do not imply public stability. Internal types encode compiler phases, arenas, sessions, queries, and evolving invariants. Freezing them would make refactoring a compatibility event. Publishing every internal crate would also expose unsupported composition assumptions.

An internal function may require a global context and interned identities. Its type signature cannot express every ordering, lifetime, or thread-local precondition. Calling it outside the driver can compile yet be conceptually invalid.

Counterfactually, imagine stable rustc_typeck::check(expr). What owns source maps, hygiene, crate metadata, diagnostics, and incremental state? Which edition and target define the expression? How are changing language features represented without breaking callers? The apparently convenient function expands into a compiler service protocol.

Consumers should prefer stable surfaces matched to their need. Use rustdoc JSON only according to its documented status. Use Cargo metadata for package graphs. Use stable diagnostics formats and language-server protocols where promised. Tools choosing private compiler APIs must pin versions and budget migration work.

This is not hostility toward tools. It protects compiler evolution and prevents accidental guarantees. A future supported API should be designed around user tasks, not frozen implementation anatomy.

29. Compilation is a security boundary under hostile input#

Compilers routinely process untrusted source, metadata, macros, object files, and paths. A crash is availability loss; arbitrary execution in the compiler is worse. Resource exhaustion matters even when memory safety prevents corruption. CI services and editors make attacker-controlled compilation especially realistic.

Threat-model dimensions include input size, recursion, generated tokens, and dependency graphs. Procedural macros execute code and therefore have a different boundary from declarative parsing. Build scripts are programs, not passive metadata. Sandbox claims must name which processes and capabilities are constrained.

Case study: a tiny source triggers exponential trait solving. Hypothesis A is uncontrolled candidate branching. Hypothesis B is repeated query invalidation. Hypothesis C is diagnostic formatting of an enormous type. Measure phase timing and profiles before imposing a random recursion limit.

Security fixes should minimize disclosure risk without erasing engineering evidence. Add regression tests that cannot become denial-of-service attacks on CI. Use bounded synthetic cases, timeouts at infrastructure boundaries, and complexity reasoning. Audit temporary files, symlink behavior, archive paths, and command construction.

Reproducibility also has security value. Pinned inputs and recorded tool versions make unexpected artifacts investigable. But bit identity alone does not prove an artifact benign.

30. Production operations turn correctness into a sustained service#

A compiler release is a coordinated artifact, not a commit with a tag. Bootstrap stages, host triples, target libraries, signing, distribution, and rollback all matter. Release channels distribute risk and feedback over time.

Testing should mirror the failure surface. UI tests protect diagnostics and accept/reject behavior. Run-pass and codegen tests exercise execution and emitted patterns. Incremental tests target reuse and invalidation. Debuginfo, rustdoc, target, and bootstrap tests cover distinct contracts.

./x test tests/ui/path/to/test.rs
./x test compiler/rustc_some_crate
./x build library

These commands are illustrative around rustc 1.97.1. Check ./x --help and the development guide in the actual checkout. Do not convert a local command into eternal documentation.

Operational triage asks blast radius before elegance. Can the issue miscompile safe code? Which channels and targets contain it? Is rollback safer than a forward fix? Can detection be added while the complete repair is reviewed?

A green suite is evidence bounded by its configurations. Flakiness is production information, not noise to rerun away. Record seeds, host load, timing, and test ordering when investigating intermittence.

31. Critique pipelines by requirements, not by number of boxes#

A smaller compiler pipeline can be excellent for a smaller contract. It may teach parsing clearly, compile a DSL quickly, or target one machine. rustc's complexity follows generics, macros, coherence, diagnostics, targets, and compatibility. Complexity is justified only when traced to such requirements.

tokens -> AST -> typed tree -> machine code

tokens -> AST -> expansion -> HIR -> type system -> THIR -> MIR -> backend
                  ^ queries, metadata, diagnostics, incremental reuse ^

Do not praise the second diagram merely because it resembles rustc. Ask what information each representation makes explicit and what consumers require it. A stage that preserves no useful invariant may be accidental complexity. A merged stage may create coupling that hides semantic boundaries.

When importing an engine, reject cargo-cult arguments. “Project X uses it” is neither compatibility proof nor maintenance plan. Compare semantics, diagnostics, cancellation, determinism, memory, and governance. Prototype the hardest mismatch, not the happy-path example.

Counterfactual: replace rustc's query engine with a fashionable incremental framework. Would keys preserve identity across sessions? Can cycles produce Rust-specific diagnostics? Can disk caches validate target and compiler fingerprints? Migration cost and dual-system debugging belong in the decision table.

32. Chalk is an influence and laboratory, not a simple embedded answer#

Chalk explored trait solving through logic programming and a Rust-like trait model. Its concepts influenced compiler design and made assumptions explicit. That history does not imply current rustc simply embeds Chalk as its universal solver. Around rustc 1.97.1, solver integration and defaults remain version-sensitive.

rustc must handle diagnostics, coherence, normalization, inference, performance, and compatibility. A standalone logical engine can model core judgments without owning all those product constraints. Representations and APIs also evolve on both sides. Embedding a crate does not erase semantic adaptation at the boundary.

Three competing proposals deserve separate evaluation. Embed Chalk unchanged and translate rustc goals at every boundary. Port Chalk-derived ideas into rustc-native infrastructure. Maintain old and new solvers while incrementally comparing behavior. Each trades conceptual purity against integration control and transition risk.

Socratic prompt: if an engine proves more goals, is it automatically better? No; it might accept unsound programs, diverge, or violate compatibility. If it rejects more goals, is it automatically conservative and safe? No; rejection can still break stable code and conceal implementation bugs.

Use current solver documentation, tracking issues, and tests for current facts. Use Chalk books and design notes to understand historical ideas. Never convert lineage into a deployment claim.

33. Locate bugs by finding the earliest broken invariant#

The visible failure is often downstream of the defect. Debugging improves when every stage states what must be true on entry and exit. Find the earliest representation where expected and actual meaning diverge.

Workshop case: an async closure reports a nonsensical lifetime error. Hypothesis A: capture analysis selected the wrong capture mode. Hypothesis B: MIR lowering assigned an incorrect region relation. Hypothesis C: borrow checking is right but diagnostic provenance is wrong. Hypothesis D: recovery after an earlier type error poisoned the body.

Inspect expanded and lowered forms before editing borrow checking. Reduce the source while retaining the wrong capture. Compare a nearby non-async closure and a manually desugared future. Log facts at the boundary where hypotheses predict different output.

source intent
  -> expansion shape correct?
  -> inferred types and captures correct?
  -> MIR places and regions correct?
  -> borrow result correct?
  -> diagnostic mapping correct?

The earliest broken invariant is not necessarily the earliest phase. An invalid cache reuse can inject old correct data into a late query. An optimization can violate an invariant established much earlier. Bisect dataflow as well as chronology.

Finish the workshop by writing a regression at the responsible boundary. An end-to-end test protects behavior; a focused test explains ownership.

34. Read source as a historical argument#

Current code shows the surviving design, not all reasons it survived. Names can preserve abandoned architecture. Comments may describe a transition that later became permanent. Tests often encode compatibility incidents more precisely than prose.

Begin with a behavior and follow its query or data path. Read type definitions before method bodies. Write down invariants and ownership boundaries in your own words. Then use blame and pull-request history to test those interpretations.

History has traps. An old review discussed constraints that no longer exist. A rejected alternative may now be viable after another subsystem changed. A commit message can simplify a contentious decision. Current maintainers and documentation outrank archaeology for present policy.

Use three columns in reading notes.

This separation prevents accidental stabilization by explanation. It also prevents dismissing present code as arbitrary because its original motivation vanished. Historical reading should produce hypotheses you can check, not founder mythology.

35. Teach a nonlinear pipeline without lying#

The familiar pipeline diagram is a map, not an execution trace. rustc uses demand-driven queries, caching, cycles under controlled rules, and parallel opportunities. Macro expansion and name resolution can interact rather than form clean sequential boxes. Monomorphization revisits generic meaning with concrete substitutions late in compilation.

Teach in three passes. First present representations as semantic viewpoints. Then show dependencies between questions. Finally show scheduling, caching, and feedback edges. Students retain a useful scaffold without mistaking it for a conveyor belt.

                ┌──── metadata from dependencies ────┐
parse -> expand <-> resolve -> HIR -> types -> MIR -> codegen
                    │          ^       │       ^
                    └ queries/cache ───┴───────┘

Socratic prompt: when does type checking “start”? Answering requires choosing a crate, body, query, and meaning of start. Prompt: does MIR exist once for a generic function? The answer distinguishes generic MIR from later codegen instances.

Use precise simplifications: “we ignore incremental scheduling for five minutes.” Do not say “the compiler always completes phase A before phase B.” A disclosed simplification is a teaching tool; an undisclosed one becomes a misconception.

36. Design talks around one claim and one transformation#

A twenty-minute talk should establish one durable claim. Use three minutes for stakes, twelve for one case, and five for consequences. For example: “borrow errors are failed proofs, not runtime predictions.” One before-and-after MIR sketch is enough.

A forty-five-minute talk can compare two models. Spend ten minutes on shared vocabulary, twenty on a case, ten on alternatives, and five on sources and limits. NLL and Polonius fit only if integration status is explicitly version-guarded.

A ninety-minute session can become a workshop. Open with a reproducible bug, collect competing hypotheses, inspect stages, take a short break, then locate the earliest broken invariant together. Reserve time to discuss why plausible wrong hypotheses were plausible.

Demo design needs a failure plan. Pin the checkout, prebuild expensive artifacts, enlarge text, and record commands. Keep captured output for network, projector, or bootstrap failure. Never fake liveness by hiding that output is prerecorded.

Evaluate a talk by transfer. Can attendees predict a new example? Can they name which claim was implementation-specific? Can they find an official source when the version changes? Applause measures experience, not necessarily learning.

37. Contribution mastery is disciplined uncertainty#

A strong contributor makes small claims backed by reproducible evidence. They reduce scope before expanding architecture. They know when semantic approval is needed and when implementation review suffices. They leave tests and explanations that survive personal memory.

Start a patch with a failure narrative. Record compiler commit, command, target, expected behavior, and actual behavior. Add the smallest regression that expresses the public or internal contract. Change the owner of the earliest broken invariant, not every downstream symptom.

Review mastery is different from finding faults. Restate the patch's intended invariant. Check semantic risk, diagnostics, performance, tests, and maintainability proportionately. Label blocking issues separately from preferences and questions.

A reviewer should ask, “What evidence would change my mind?” An author should answer criticism with data or a revised claim. Neither should use CI as a substitute for reasoning. Approval transfers responsibility, so uncertainty belongs in the review record.

Case study: a faster query adds a global cache. The speedup hypothesis competes with stale-key and memory-growth hypotheses. Request invalidation tests, peak-memory data, and incremental comparisons. A benchmark win alone does not establish production readiness.

38. A ninety-day path from orientation to independent judgment#

Days one through thirty build vocabulary and reproducibility. Read the development guide's architecture overview alongside one tiny program. Inspect expansion, HIR descriptions, MIR, and emitted backend artifacts using current flags. Build one focused compiler crate and run one UI test locally. Keep a glossary separating promises, implementations, and historical terms.

Days thirty-one through sixty build causal skill. Reduce three existing regressions from different subsystems. For each, list competing hypotheses before reading the fix. Recreate one historical patch and identify its earliest broken invariant. Review a small pull request privately, then compare with public review.

Days sixty-one through ninety build contribution judgment. Choose a labeled issue with a bounded reproduction. Discuss scope publicly, add a regression, implement the narrow repair, and measure it. Write review notes that explain risks rather than merely requesting changes. Present a twenty-minute case study with explicit version guards.

Weekly cadence should remain sustainable. One source-reading session, one experiment, one written synthesis, and one community interaction suffice. Do not measure progress by compiler crates visited. Measure whether predictions become more accurate and uncertainty more precise.

At day ninety, repeat the original tiny-program tour. The goal is not memorized flags. The goal is seeing contracts, evidence, and ownership where boxes once appeared.

39. Durable models survive changing implementations#

Some mental models age better than component names. Compilation progressively establishes and translates invariants. Queries are memoized questions with dependency and invalidation obligations. Borrow checking is conservative proof over a chosen representation. Optimization preserves meaning while changing realization. Diagnostics translate failed internal obligations into human action.

Use official sources according to authority. The Rust Reference describes much stable language behavior, while noting scope. RFCs and stabilization records explain accepted design, not necessarily current internals. The rustc development guide explains implementation and contribution practice. Rust Forge covers project and release operations. The current source, tests, and command help answer checkout-specific questions.

Issue threads, Zulip, talks, and blog posts add context. They are valuable but time-indexed. Record dates, compiler versions, commits, and unresolved disagreement when citing them. Prefer a current primary source over a confident secondary summary.

Use a claim ledger.

Durability comes from knowing how to update knowledge, not refusing details.

40. Mastery means preserving reasons while permitting change#

The mature compiler engineer does not worship complexity or simplicity. They ask which promise requires a mechanism, which invariant it establishes, which evidence supports it, and who bears its operational cost. They can admire an imported idea without assuming integration is free. They can criticize rustc without pretending its constraints are imaginary.

Borrow checking exemplifies this stance. Conservative rejection protects a safety promise while inviting better approximations. NLL and Polonius show that models evolve without making history identical to deployment. Optimization and backend work show that performance remains subordinate to meaning. Diagnostics show that correctness reaches users through language and trust.

Stability turns design into a long relationship with an ecosystem. Security treats compilation inputs and costs as potentially hostile. Operations turn local correctness into reliable releases. Testing maps confidence rather than manufacturing certainty. Review turns individual reasoning into maintained collective knowledge.

When a bug appears, seek the earliest broken invariant. When a source file looks strange, read its history without surrendering present judgment. When teaching, reveal the nonlinear pipeline in layers. When speaking, fit one honest transformation to the available time. When studying, revisit the same path with increasingly precise questions.

The final Socratic prompt is simple: what would change your mind? A master can name the test, source, counterexample, or measurement they need. That answer is more valuable than certainty performed from memory. rustc will change; targets, solvers, queries, and teams will change. The durable craft is preserving meaning, locating responsibility, and updating beliefs with evidence. That is how understanding becomes contribution, and contribution becomes stewardship.