Part I: Foundations — SIMD in Rust: From First Principles to Production Engineering#

Verified: 2026-08-04 Audience: Rust programmers who want a correct mental model before using vector APIs Confidence target: By the end, you should be able to judge simple SIMD candidates with high confidence, explain likely bottlenecks with moderate confidence, and avoid claiming performance until measurement confirms it.

Outcomes#

After this part, you will be able to:

• explain SIMD from bits, registers, and ordinary scalar loops;
• distinguish instruction-level parallelism from data-level parallelism;
• trace lane-wise arithmetic, comparisons, masks, and selection;
• recognize reductions, shuffles, and other cross-lane work;
• write a scalar Rust function that serves as a semantic oracle;
• state integer overflow and floating-point behavior as explicit contracts;
• estimate whether compute, memory bandwidth, or latency limits a loop;
• account for setup, dispatch, alignment, and tail costs;
• use a disciplined checklist before choosing SIMD;
• describe why benchmark evidence, not lane count, establishes speedup.

This part does not teach raw std::arch intrinsics in depth. It builds the reasoning needed to use them safely later.

Prerequisites#

You should know:

• basic Rust syntax;
• arrays, slices, loops, and functions;
• the difference between an integer and a floating-point number;
• how to run a Rust program with Cargo.

You do not need assembly knowledge. You do not need prior performance-engineering experience.

Your first hour: make the scalar meaning executable#

Do not begin by memorizing an intrinsic. Begin with a program whose behavior you can state, run, and test. Create a new Cargo binary, replace src/main.rs with this program, and run cargo run.

fn add_wrapping(a: &[u32], b: &[u32], out: &mut [u32]) {
    assert_eq!(a.len(), b.len());
    assert_eq!(a.len(), out.len());

    for i in 0..a.len() {
        out[i] = a[i].wrapping_add(b[i]);
    }
}

fn main() {
    let a = [10, 20, 30, u32::MAX];
    let b = [1, 2, 3, 1];
    let mut out = [0; 4];

    add_wrapping(&a, &b, &mut out);
    assert_eq!(out, [11, 22, 33, 0]);
    println!("{out:?}");
}

Before reading on, answer these questions aloud:

1. What value does each iteration promise to write?
2. Why is the final result 0 rather than a panic or a larger integer?
3. Which iteration depends on another iteration's result?

The answers establish your first SIMD contract: this is wrapping arithmetic, the output length matches the inputs, and every iteration is independent. SIMD will later change how many values are described at once; it must not change those promises.

If this feels comfortable, read Sections 1 through 3 in order. When you can explain the loop without using the word “SIMD,” continue to Section 4. That is the point where vectorization becomes a consequence of the program's meaning rather than a trick.

1. Begin with bits#

A bit is a binary digit: either 0 or 1. Eight bits usually form a byte. Memory is commonly addressed by byte, which means each memory address identifies one byte.

one byte:  1 0 1 1 0 0 1 0
bit index: 7 6 5 4 3 2 1 0

The displayed byte has eight bit positions. Its interpretation depends on the program. The same bits can represent an unsigned integer, part of a signed integer, text, or raw data. Rust integer names expose both size and signedness. u8 is an unsigned eight-bit integer. i8 is a signed eight-bit integer. u32 and i32 each occupy 32 bits, or four bytes. An unsigned u8 represents values from 0 through 255. Rust defines i8 with two's-complement representation; it represents -128 through 127. The bits do not announce their signedness; the operation gives them meaning.

Prediction checkpoint 1#

Suppose memory contains the byte 1111_1111.

1. What is its value as u8?
2. What is its value as i8 in two's complement?
3. Did the stored bits change between those interpretations?

Answer: It is 255 as u8, -1 as i8, and the stored bits did not change.

2. Memory, registers, and instructions#

Main memory stores program data and code. A processor cannot usually perform arithmetic directly on arbitrary bulk memory. It brings values into small, fast storage locations called registers. An instruction is one operation the processor knows how to execute. Examples include loading data, adding values, comparing values, and storing a result. At a simplified level, scalar addition looks like this:

memory A ----load----> register r1 --\
                                     add --> register r3 ----store----> memory C
memory B ----load----> register r2 --/

Scalar means one logical value at a time. A scalar register may hold one u32, and a scalar add produces one u32 result. Real CPUs are more complex than this picture. The picture remains useful because data must still move and dependencies still matter.

3. A scalar loop#

Consider adding two equally sized arrays. Start with a reference implementation whose meaning is obvious.

fn add_wrapping(a: &[u32], b: &[u32], out: &mut [u32]) {
    assert_eq!(a.len(), b.len());
    assert_eq!(a.len(), out.len());
    for i in 0..a.len() {
        out[i] = a[i].wrapping_add(b[i]);
    }
}

Each iteration loads a[i] and b[i], adds them, and stores out[i]. The explicit wrapping_add defines overflow behavior in every build mode. That detail will matter when we compare implementations. One conceptual trace is:

i | a[i] | b[i] | wrapping result
0 |   10 |    1 |              11
1 |   20 |    2 |              22
2 |   30 |    3 |              33
3 |   40 |    4 |              44

There is no dependency from result 0 to result 1. The iterations are independent if the slices do not overlap in a harmful way. Independent repeated work is the basic opportunity SIMD exploits.

4. Parallelism inside one CPU core#

Modern processors overlap work in several ways. These ways are related, but they are not synonyms.

Pipelining#

A pipeline divides instruction processing into stages. Different instructions can occupy different stages at the same time. It resembles an assembly line: one item finishes while later items are already underway. Pipelining improves the rate of completed work. It does not imply that one instruction handles multiple array elements.

Superscalar execution#

A superscalar processor can begin more than one instruction in a clock cycle when resources and dependencies permit. For example, it may issue a load and an unrelated arithmetic instruction together.

Instruction-level parallelism#

Instruction-level parallelism, abbreviated ILP, is overlap among separate instructions. Independent scalar additions may execute concurrently on multiple arithmetic units.

time --->
add item 0: [execute]
add item 1: [execute]
load item 2: [load   ]

The hardware can discover some ILP dynamically. Compilers also schedule and transform code to expose it.

Data-level parallelism#

Data-level parallelism applies the same operation to several data items. SIMD is a machine-level expression of data-level parallelism.

scalar instructions: add a0,b0   add a1,b1   add a2,b2   add a3,b3
SIMD instruction:    add [a0 a1 a2 a3], [b0 b1 b2 b3]

ILP can overlap several scalar instructions. SIMD can make one instruction describe several lane operations. A CPU may exploit both at once.

Prediction checkpoint 2#

A core starts two independent scalar adds in one cycle. Is that necessarily SIMD? Answer: No. It is superscalar ILP unless each add instruction itself operates on multiple lanes.

5. What SIMD means#

SIMD stands for Single Instruction, Multiple Data. One instruction applies one operation to multiple values packed into a vector register. A lane is one element position inside a vector. All lanes usually have the same element type and width for an operation.

vector A: [ 10 | 20 | 30 | 40 ]
vector B: [  1 |  2 |  3 |  4 ]
             lane-wise add
result:   [ 11 | 22 | 33 | 44 ]
lane:        0    1    2    3

Vector width is the total number of bits in a vector register or operation. Lane count is vector width divided by element width. A 128-bit vector can hold:

• sixteen 8-bit lanes;
• eight 16-bit lanes;
• four 32-bit lanes;
• two 64-bit lanes.

The same physical width supports different lane interpretations through different instructions. Four f32 lanes and four u32 lanes both occupy 128 bits, but use different arithmetic semantics.

6. Manually batching before intrinsics#

Do not begin with special instruction names. First rewrite the mental model in fixed-size batches. For a conceptual batch width of four:

fn add_wrapping_batched(a: &[u32], b: &[u32], out: &mut [u32]) {
    assert_eq!(a.len(), b.len());
    assert_eq!(a.len(), out.len());
    let mut i = 0;
    while i + 4 <= a.len() {
        out[i] = a[i].wrapping_add(b[i]);
        out[i + 1] = a[i + 1].wrapping_add(b[i + 1]);
        out[i + 2] = a[i + 2].wrapping_add(b[i + 2]);
        out[i + 3] = a[i + 3].wrapping_add(b[i + 3]);
        i += 4;
    }
    while i < a.len() {
        out[i] = a[i].wrapping_add(b[i]);
        i += 1;
    }
}

This code is not a promise that SIMD instructions will be generated. It exposes three structural parts:

1. a main loop over full batches;
2. independent work within each batch;
3. a tail, meaning elements left after the final full batch.

For length 11 and width 4, there are two full batches and a tail of 3.

indices: 0 1 2 3 | 4 5 6 7 | 8 9 10
work:      batch 0 | batch 1 | tail

Compilers may auto-vectorize suitable scalar loops. Auto-vectorization is a compiler transformation from scalar operations to vector operations. Manual batching helps reasoning, but clear scalar code can sometimes optimize better than hand-unrolling.

7. The load-compute-store model#

Most basic SIMD kernels repeat three phases:

1. load lanes from memory into vector registers;
2. compute on those registers;
3. store result lanes back to memory.

arrays in memory
      |
      v
[vector load] --> [lane-wise compute] --> [vector store]
                                              |
                                              v
                                       output memory

A vector load does not make memory free. The bytes still travel through the memory hierarchy. Wider computation can expose memory as the limiting resource. Loads are simplest when elements are contiguous, meaning adjacent in memory. Gathering scattered elements is usually more expensive and may need special instructions. A gather loads lanes from multiple noncontiguous addresses. A scatter stores lanes to multiple noncontiguous addresses.

8. Lane-wise operations#

An operation is lane-wise when each result lane depends only on corresponding input lanes. Addition, subtraction, multiplication, bitwise Boolean operations, and many comparisons fit this model.

a:       [ 3,  8,  2,  9]
b:       [ 4,  1,  2, 12]
a + b:   [ 7,  9,  4, 21]
a < b:   [ T,  F,  F,  T]

Lane independence makes these operations easy to map to SIMD. No lane waits for another lane's result. Integer division is a warning. Some instruction sets lack a general packed integer divide instruction. A source expression that looks lane-wise is not necessarily cheap in hardware.

9. Masks and selection#

A mask records a Boolean condition per lane. Each mask lane says whether a condition is true for the corresponding data lane. Suppose we clamp signed values below zero to zero.

x:          [ -3,  5, -1,  8]
x < 0:      [  T,  F,  T,  F]
select 0/x: [  0,  5,  0,  8]

Select chooses between two lane values according to a mask. Conceptually, select(mask, yes, no) chooses yes where true and no where false.

fn clamp_nonnegative(input: &[i32], out: &mut [i32]) {
    assert_eq!(input.len(), out.len());
    for (dst, &x) in out.iter_mut().zip(input) {
        *dst = if x < 0 { 0 } else { x };
    }
}

A compiler may turn this branch-looking scalar expression into a maximum or masked selection. Branchless means choosing results without changing control flow per item. Branchless code is not automatically faster, but uniform lane selection suits SIMD. Masks differ across architectures and APIs. Some represent truth as all one-bits in a full data lane. Others use dedicated mask registers or abstract mask types. Treat a mask as a logical value unless an API contract exposes its bit representation.

Prediction checkpoint 3#

Given mask [F, T, T, F], yes values [1, 2, 3, 4], and no values [9, 8, 7, 6], what does select produce? Answer: [9, 2, 3, 6].

10. Vertical and horizontal work#

Vertical operations combine corresponding lanes from separate vectors. The ordinary lane-wise add is vertical.

A0 [a0 a1 a2 a3]
       vertical add
B0 [b0 b1 b2 b3]
 = [a0+b0 a1+b1 a2+b2 a3+b3]

Horizontal operations combine lanes within one vector. Summing all lanes is horizontal.

[x0 x1 x2 x3]
 \ /     \ /
  s0      s1
    \    /
      sum

A reduction combines many values into fewer values, often one. Sum, minimum, maximum, logical all, and logical any are reductions. Reductions create dependencies. The final answer must collect information from every lane. They often need cross-lane instructions and a scalar finish. Multiple accumulators can expose ILP in a long reduction. For example, accumulate four independent partial sums and combine them at the end. This changes floating-point addition order, so it requires an appropriate numeric contract.

11. Shuffles and cross-lane operations#

A shuffle rearranges lanes according to a pattern. It can duplicate, swap, or select lanes, depending on the instruction.

input:              [A, B, C, D]
reverse shuffle:    [D, C, B, A]
duplicate evens:    [A, A, C, C]
rotate left:        [B, C, D, A]

Cross-lane work is any work where a result lane depends on a different input lane. Shuffles, reductions, prefix sums, and transposes are examples. Cross-lane operations are not forbidden. They can cost more than simple lane-wise arithmetic and may have architecture-specific limits. A prefix sum illustrates dependency:

input:  [2, 5, 1, 3]
output: [2, 7, 8, 11]

Output lane 3 depends on all preceding input lanes. Efficient SIMD prefix sums use staged shifts and adds rather than four independent adds. We will study such patterns later.

12. A high-level architecture map#

An instruction set architecture, or ISA, defines programmer-visible machine instructions and state. Different CPU families expose different SIMD facilities.

x86 and x86-64#

SSE introduced widely used 128-bit vector operations on x86. Later SSE revisions expanded integer, floating-point, comparison, and shuffle support. AVX added 256-bit vector registers and operations, initially strongest for floating point. AVX2 extended 256-bit support broadly to integer operations and added useful gathers and permutations. AVX-512 is a family of extensions using 512-bit vectors and dedicated mask registers. Its subfeatures vary by processor. The name does not guarantee every AVX-512 instruction group is available. Wider x86 vectors can process more lanes per instruction. They may also affect clock frequency, power, instruction count, and transition behavior on some CPUs. Availability and performance must be detected and measured on target hardware.

AArch64 NEON#

AArch64 is the 64-bit Arm architecture. Its baseline SIMD facility is commonly called NEON, also known architecturally as Advanced SIMD. It primarily uses 128-bit vector registers, with operations that may also use narrower portions. NEON supports common integer and floating-point lane operations, permutations, and reductions. A 128-bit width can still deliver excellent performance because width is only one factor.

WebAssembly SIMD128#

WebAssembly, abbreviated wasm, is a portable binary instruction format for sandboxed execution. The SIMD128 extension defines portable 128-bit vector operations. Browsers and runtimes translate wasm operations to host instructions. The portable contract is 128 bits even when the host has wider vectors. Performance still depends on the runtime, host ISA, generated code, and surrounding JavaScript or host calls.

Portability lesson#

Do not equate “SIMD” with one x86 extension. The shared model is lanes, masks, loads, stores, and cross-lane operations. Exact widths, instructions, and costs differ.

13. Why eight lanes are not automatically eight times faster#

Eight lanes describe work represented by an instruction. They do not describe whole-program speedup. Several limits intervene:

• vector instructions may have higher latency or lower throughput than expected;
• loads and stores may exhaust memory bandwidth;
• data may miss caches;
• tails and setup consume scalar work;
• runtime feature dispatch costs time;
• dependencies may serialize a reduction;
• shuffles or conversions may dominate arithmetic;
• the compiler may already auto-vectorize the scalar loop;
• only part of the program may be vectorizable;
• wider vectors can trigger lower CPU frequencies on some processors.

If 50% of runtime becomes eight times faster and the other 50% is unchanged, total speedup is:

new time = 0.50 / 8 + 0.50 = 0.5625
speedup  = 1 / 0.5625 ≈ 1.78x

This is an example of Amdahl's law: unchanged work limits total speedup. The formula is useful, but measurements must provide the fractions.

14. Latency and throughput#

Latency is the time from starting an operation until its result is available to dependent work. Throughput is the sustained rate at which operations can begin or complete. Imagine a multiply with latency four cycles but throughput one per cycle. One dependent chain must wait about four cycles between multiplies. Four independent chains may keep the unit busy by starting one multiply each cycle.

cycle:       0 1 2 3 4 5 6 7
chain A:     M.......M
chain B:       M.......M
chain C:         M.......M
chain D:           M.......M

The diagram is conceptual, not a timing claim for a particular CPU. Instruction timing depends on microarchitecture, meaning a specific processor's internal design. SIMD can increase useful work per instruction. ILP can hide latency by overlapping independent instructions. Good kernels often need both.

Prediction checkpoint 4#

An operation has long latency but excellent throughput. Would one dependency chain use its full throughput? Answer: Usually not. Several independent chains are needed to overlap the latency.

15. The memory hierarchy#

CPU registers are tiny and fast. Caches hold recently or nearby used memory. Main memory is much larger and slower. A simplified hierarchy is:

fast, tiny     registers
                 |
                L1 cache
                 |
                L2 cache
                 |
          shared/larger cache
                 |
slow, large    main memory

A cache line is the fixed-size block moved between levels of the cache hierarchy. Many current CPUs use 64-byte cache lines, but code should not assume that universally. Reading one byte can cause the surrounding cache line to be fetched. Sequential traversal benefits because neighboring elements arrive together. Random access may waste most bytes in each fetched line. Spatial locality means accessing nearby addresses. Temporal locality means reusing data soon while it is still in cache. SIMD works naturally with spatial locality when lanes come from contiguous arrays. It cannot repair a poor access pattern by itself.

16. Bandwidth and memory latency#

Memory bandwidth is the sustained amount of data transferred per unit time. It is often measured in gigabytes per second. Memory latency is the delay before requested data arrives. Many outstanding independent requests can overlap latency and approach bandwidth limits. Pointer chasing often cannot, because each address depends on the previous load. Consider adding two u32 arrays into one output array. Ignoring cache effects and write-allocation details, each element reads 8 bytes and writes 4 bytes. It performs one addition for roughly 12 bytes of explicit traffic. Once memory channels are saturated, wider arithmetic cannot increase useful throughput much. It may reduce instruction overhead, but bytes remain the dominant cost.

17. Arithmetic intensity and roofline intuition#

Arithmetic intensity is useful arithmetic work divided by bytes moved. The exact unit might be operations per byte. Array addition has low arithmetic intensity: one add for substantial data movement. A polynomial evaluated repeatedly on cached values has higher arithmetic intensity. The roofline model gives a simple upper-bound intuition. Performance is limited by the lower of:

• the machine's peak compute rate;
• memory bandwidth multiplied by arithmetic intensity.

attainable performance <= min(
    peak compute performance,
    memory bandwidth * arithmetic intensity
)

This is not a cycle-accurate prediction. It helps ask the right first question: are more arithmetic lanes useful, or are bytes the roof? If bandwidth is 40 GB/s and a kernel performs 0.25 operations per byte, the bandwidth roof is 10 billion operations per second. Even a 100-billion-operation compute engine cannot cross that roof for this traffic pattern. Cache-resident data has a different effective bandwidth roof than main-memory data.

18. Compute-bound, bandwidth-bound, and latency-bound#

A compute-bound kernel spends its limiting resource on arithmetic or execution units. More efficient vector arithmetic can help substantially. A bandwidth-bound kernel is limited by sustained data transfer. Reducing bytes, improving reuse, or changing layout may matter more than wider vectors. A latency-bound kernel waits on dependent operations or unpredictable memory requests. More independent work may help; simply widening a dependency chain may not. These labels describe the dominant limit in a particular context. The same kernel can be cache-compute-bound for small inputs and memory-bandwidth-bound for large inputs.

Prediction checkpoint 5#

A loop doubles each element of a huge array and writes it back. After SIMD, arithmetic instructions fall sharply but runtime barely changes. What is the leading hypothesis? Answer: The loop is probably limited by memory bandwidth or memory-system traffic, not multiplication throughput. Measure before concluding.

19. Fixed costs: setup, tails, and dispatch#

SIMD loops carry overhead outside their ideal steady state. Setup overhead includes preparing constants, calculating loop bounds, and initializing accumulators. For tiny inputs, setup can cost more than saved lane operations. Tail overhead handles elements that do not fill a complete vector. Possible strategies include:

• a scalar cleanup loop;
• a masked final vector operation;
• padding when the data contract safely permits it;
• processing an overlapping final full vector when aliasing and bounds permit.

Each strategy has correctness conditions. Never read beyond a Rust slice merely because hardware might tolerate the address. Dispatch overhead chooses an implementation based on CPU features at runtime. A small branch or indirect call can be negligible for large buffers and significant for tiny calls. Feature detection also affects deployment. One binary may contain a baseline version and several optimized versions. The chosen version must never execute unsupported instructions. For frequent tiny operations, moving dispatch outside the hot call path can help. That design choice comes later, after semantics are stable.

20. Semantic equivalence comes first#

An optimized function is correct when it satisfies the same stated contract as the reference. Call this semantic equivalence. Equivalence is not always “identical source-level operations.” It may mean exactly equal integer bits, exactly equal floating-point bits, or values within an error bound. The contract must say which. A scalar oracle is a simple trusted implementation used to check optimized results. Keep it even after a SIMD implementation is fast. The oracle supports:

• unit tests on hand-picked edge cases;
• randomized differential tests;
• debugging on machines without the target SIMD feature;
• future rewrites for different architectures;
• documentation of intended behavior.

Differential testing runs two implementations on the same inputs and compares outputs. It is especially valuable for tails, overflow, NaNs, and unusual lengths.

fn threshold_oracle(input: &[i16], threshold: i16, out: &mut [u8]) {
    assert_eq!(input.len(), out.len());
    for (dst, &x) in out.iter_mut().zip(input) {
        *dst = if x >= threshold { 255 } else { 0 };
    }
}

This function defines comparison direction, equality behavior, and output encoding. A future SIMD version must preserve all three.

21. Integer signedness#

Signed and unsigned addition have identical low-bit wrapping behavior at a fixed width. Comparisons and some widening operations do not.

bits:       1111_1111
as u8:      255
as i8:       -1
u8 > 1:     true
i8 > 1:     false

A vector comparison must use the intended signed or unsigned instruction semantics. Looking only at register bits is insufficient. Right shift is another distinction. A logical right shift inserts zero bits. An arithmetic right shift replicates the sign bit.

input bits:             1111_0000
logical right by 2:     0011_1100
arithmetic right by 2:  1111_1100

Define the element type before choosing operations.

22. Integer overflow contracts#

Fixed-width integer arithmetic can exceed its representable range. Several valid contracts exist.

Wrapping arithmetic#

Wrapping arithmetic keeps the low bits, behaving modulo 2 raised to the bit width.

u8: 250 + 10 = 4 with wrapping

Use Rust methods such as wrapping_add in the oracle when wrapping is intended.

Saturating arithmetic#

Saturating arithmetic clamps to the nearest representable endpoint.

u8: 250 + 10 = 255 with saturation
i8: -120 - 20 = -128 with saturation

Image and audio processing often use saturation, but never assume it implicitly.

Checked arithmetic#

Checked arithmetic reports overflow, commonly as None or an error condition. SIMD checked arithmetic may require detecting overflow masks and combining them. That extra work can alter the performance model.

Widening arithmetic#

Widening converts inputs to a larger type before or while computing. For example, multiply two u8 values into a u16 result.

200u8 * 2u8
wrapping u8 result: 144
widened u16 result: 400

Widening changes lane count because wider elements consume more vector bits. A 128-bit vector has sixteen u8 lanes but only eight u16 lanes.

Narrowing conversions#

Narrowing converts a wider value to a smaller type. It must define whether high bits are truncated, values saturate, or out-of-range inputs are rejected. Rust's as conversion and a saturating pack do not necessarily mean the same thing. Make narrowing behavior explicit in the scalar oracle and tests.

Prediction checkpoint 6#

What should a SIMD version return for 250u8 + 10u8? Answer: The question is incomplete. It returns 4 under wrapping, 255 under saturation, and an overflow indication under checked semantics.

23. Floating-point foundations#

Rust f32 and f64 generally follow IEEE 754 binary floating-point semantics. A floating-point value contains a sign, an exponent, and a significand. The significand carries precision; the exponent scales the value. Most real numbers cannot be represented exactly in a finite binary format. Each operation usually rounds an exact mathematical result to a representable value. This makes floating-point addition non-associative:

(a + b) + c may differ from a + (b + c)

For a concrete f32 example, let a = 100_000_000.0, b = -100_000_000.0, and c = 1.0. One grouping can produce 1, while another can produce 0 because the small value is rounded away.

24. NaN, infinities, and signed zero#

NaN means “Not a Number.” It represents invalid or indeterminate floating-point results and can carry payload bits. NaN comparisons are unusual. NaN == NaN is false, and ordered comparisons with NaN are generally false. A minimum instruction's NaN behavior may differ from Rust method semantics or another ISA's instruction. Positive and negative infinity represent values beyond finite range and results such as nonzero division by zero under usual IEEE rules. Floating point has both +0.0 and -0.0. They compare equal, but their sign can affect operations such as reciprocal and some minimum/maximum definitions.

1.0 / +0.0 = +infinity
1.0 / -0.0 = -infinity

If bitwise reproducibility matters, NaN payloads and signed zero need explicit treatment. If only numerical tolerance matters, document how exceptional values are compared.

25. Rounding and fused multiply-add#

The usual default rounding mode is round to nearest, ties to even. Halfway cases choose the result whose final significand bit is even. Other rounding modes exist, but many programs assume the default environment. A fused multiply-add, abbreviated FMA, computes a * b + c with one final rounding instead of rounding the multiplication and addition separately.

separate: round(round(a * b) + c)
fused:    round((a * b) + c)

FMA can be faster and more accurate relative to the exact expression. It can also differ bit-for-bit from separate operations. A SIMD contract must state whether contraction into FMA is allowed. Compiler settings and target features influence contraction.

26. Reassociation and reduction order#

Reassociation changes the grouping of arithmetic expressions. For exact integer wrapping addition, regrouping preserves the modular sum. For signed saturating arithmetic and floating point, regrouping can change results. A scalar left-to-right sum is:

fn sum_left_to_right(values: &[f32]) -> f32 {
    let mut sum = 0.0;
    for &x in values {
        sum += x;
    }
    sum
}

A vector reduction commonly forms several partial sums and then combines them. That is a different addition tree. Possible floating-point contracts include:

• bitwise identical to the scalar left-to-right oracle;
• deterministic for a chosen SIMD width and implementation;
• within an absolute or relative error tolerance;
• correctly handle NaNs and infinities under stated rules;
• permit any IEEE-valid reassociation for speed.

“Approximately equal” is incomplete without tolerance, scale, and exceptional-value behavior. An absolute tolerance checks |actual - expected| <= limit. A relative tolerance scales allowed error with result magnitude. Near zero, relative error needs special care, so tests often combine both.

Prediction checkpoint 7#

Can a vectorized floating-point sum be numerically reasonable yet fail bitwise comparison with a scalar sum? Answer: Yes. A different reduction order changes rounding, even when both results satisfy a useful error contract.

27. Trace a conceptual vector kernel#

We will trace thresholding four i16 values at a time. No architecture-specific API is needed.

input slice: [-2, 7, 7, 12, 1, 9]
threshold:   7
batch width: 4

Full batch trace:

stage          lane 0  lane 1  lane 2  lane 3
load input         -2       7       7      12
compare >= 7        F       T       T       T
select 255/0        0     255     255     255
store output        0     255     255     255

Tail trace:

index              4       5
input              1       9
compare >= 7       F       T
output             0     255

The vector path and tail use exactly the oracle's >= comparison. Changing it to > would fail on both middle values equal to 7.

28. Early exercises#

Exercise 1: Find the batches#

For 23 elements and a conceptual width of 8, identify full batches and tail length. Answer: Two full batches cover indices 0–7 and 8–15; the tail has 7 elements at indices 16–22.

Exercise 2: Trace saturation#

Saturating-add these u8 vectors.

a = [250, 10, 100, 255]
b = [ 10, 20, 200,   1]

Answer: [255, 30, 255, 255].

Exercise 3: Identify cross-lane work#

Classify each operation as lane-wise or cross-lane:

1. add corresponding pixels;
2. compare each value with zero;
3. reverse eight lanes;
4. sum all lanes;
5. choose per lane from two vectors.

Answer: 1, 2, and 5 are lane-wise; 3 and 4 are cross-lane.

29. A disciplined SIMD-candidate checklist#

Use this checklist before writing architecture-specific code.

Semantics#

• Is there a small, readable scalar oracle?
• Are input, output, and overlap rules explicit?
• Is element signedness explicit?
• Is integer overflow wrapping, saturating, checked, or widened?
• Are narrowing conversions defined?
• For floats, are NaN, infinity, and signed-zero rules defined?
• Is FMA allowed?
• Is reassociation allowed?
• Is comparison exact, bitwise, absolute-tolerance, or relative-tolerance based?

Dataflow#

• Does the same operation repeat across many independent elements?
• Which operations are lane-wise?
• Which operations cross lanes?
• Is there a loop-carried dependency, meaning one iteration needs the previous result?
• Can several independent accumulators break a latency chain?
• Are masks enough, or does each lane require different control flow?

Data layout#

• Are lane values contiguous in memory?
• Are element types uniformly sized?
• Is data reused while it remains in cache?
• Would a structure-of-arrays layout avoid gathers?
• Are input and output alignment guarantees known rather than assumed?
• Can fewer bytes represent the same information safely?

Work size and boundaries#

• How many full vectors are processed per call?
• What is the tail strategy?
• Are lengths 0, 1, width minus 1, width, and width plus 1 tested?
• Does setup dominate tiny inputs?

Hardware and portability#

• Which x86, AArch64, or wasm features may be present?
• Is runtime dispatch needed?
• Is detection outside or inside the hot path?
• Is a scalar fallback always available?
• Do wider-vector frequency effects matter on target CPUs?

Performance hypothesis#

• Is the kernel likely compute-bound, bandwidth-bound, or latency-bound?
• What is its approximate arithmetic intensity?
• How many bytes are loaded and stored per output?
• Are shuffles, gathers, conversions, or reductions likely dominant?
• What speedup is possible after accounting for unchanged work?
• What observation would disprove the hypothesis?

Verification#

• Are optimized results checked against the oracle?
• Do tests include integer endpoints and overflow cases?
• Do float tests include NaN, infinities, signed zeros, subnormal values, and large magnitude differences when relevant?
• Are all dispatch paths exercised on suitable hardware or emulation?

If several answers are unknown, the next step is investigation, not intrinsics.

30. Transition: from semantics to patterns and layout#

We have derived SIMD without relying on assembly syntax. A CPU moves values through registers, instructions, and a memory hierarchy. SIMD packs equal-width elements into lanes and applies one described operation across them. The ideal case has contiguous data, independent lanes, much repeated work, and a clear numeric contract. Real kernels add masks, tails, reductions, shuffles, dispatch, and hardware-specific costs. The scalar oracle remains the center of the engineering process. It defines meaning while optimized implementations change shape. The next part moves from semantic foundations to recurring SIMD patterns and data layout. We will ask how arrays of structures differ from structures of arrays, how to expose contiguous lanes, and how maps, filters, reductions, and table-driven work map to vectors. Only after those patterns are clear will architecture-specific APIs become useful rather than distracting.

Part II: Recognizing and Reshaping SIMD Work#

Fundamentals taught us what lanes, vectors, and masks are. Part II asks the more useful question: which programs can become lane-wise work? The answer is rarely “replace the scalar type with a vector type.” It is usually “change iteration order, representation, or intermediate precision while preserving a contract.”

1. Memory is part of the algorithm#

Contiguous, strided, and random access#

Contiguous access touches neighboring elements: a[i], a[i + 1], and so on. It gives a vector load useful lanes and gives hardware prefetchers a simple stream. Strided access touches a[base + i * stride]. A small fixed stride may still be predictable, but each cache line can contain mostly unwanted bytes. Random access uses indices or pointers whose next values are not predictable from the current address. Gather instructions can express it; they cannot make distant memory arrive cheaply. A cache line is the block moved between cache levels, commonly 64 bytes. Reading one byte can therefore fetch many neighbors, which helps contiguous loops and wastes bandwidth for sparse ones. A page is a larger virtual-memory block, commonly 4 KiB. The TLB is a small cache of recent virtual-page-to-physical-page translations. Scattered accesses across many pages can miss the TLB even when the total byte count sounds modest. Hardware prefetch predicts regular future addresses; it does not understand an arbitrary index array.

contiguous:  [0 1 2 3 4 5 6 7]  one useful run
stride 4:    [0 . . . 4 . . .]  several lines, few useful values
random:      [page A] [page Q] [page C] [page Z]

Alignment without folklore#

An aligned address is divisible by the vector load's preferred boundary. Modern targets usually support unaligned loads, and an unaligned load wholly inside a cache line is often inexpensive. Crossing a cache-line boundary may require two line accesses. Crossing a page boundary can require two translations and exposes mistakes near unmapped memory. Do not copy data merely to make an already-cheap load aligned without measuring the copy. Allocation alignment and pointer-offset alignment are different. An allocation may begin at 32-byte alignment, while base.add(1) for f32 is no longer 32-byte aligned. Peeling scalar elements can align a hot steady-state loop, but the peel itself has a cost.

AoS, SoA, and blocked AoSoA#

Suppose particles have x, y, z, and mass.

struct Particle { x: f32, y: f32, z: f32, mass: f32 }
let particles: Vec<Particle>;              // Array of Structs
let (xs, ys, zs, masses): (Vec<f32>, Vec<f32>, Vec<f32>, Vec<f32>); // SoA

AoS keeps one complete particle together and is pleasant when every operation consumes every field. But a vector of x values is interleaved with y, z, and mass, requiring deinterleave shuffles or gathers. SoA gives each field a contiguous stream and naturally maps one property across lanes. It costs more streams, more allocations, and sometimes worse locality when code needs one whole object at a time. AoSoA stores small blocks in SoA form, then repeats blocks.

AoS:   x0 y0 z0 m0 | x1 y1 z1 m1 | x2 y2 z2 m2
SoA:   x0 x1 x2 ... | y0 y1 y2 ... | z... | m...
AoSoA: [x0..x7][y0..y7][z0..z7][m0..m7] | next block

AoSoA bounds working-set size and can match a chosen lane block while retaining object-block locality. Its block size should be a storage decision, not hard-coded to one machine's vector width unless deployment is fixed. Converting AoS to SoA is profitable only when enough later work amortizes allocation, copying, and extra memory footprint. Sometimes the right answer is to produce SoA at ingestion and never convert back.

Aliasing, overlap, and Rust#

Two mutable Rust references promise exclusive access to their referents. Safe APIs can exploit that promise, but unsafe vector code must not manufacture overlapping &mut slices. For out[i] = a[i] + b[i], decide whether out may equal either input or partially overlap it. Exact in-place overlap can be legal; offset overlap may require forward or backward traversal like memmove. Vector stores cover several elements, so an offset overlap can overwrite an input needed by a later lane. Express non-overlap in the API when required, or detect overlap before entering the vector path.

2. Element-wise map and zip#

The friendliest pattern applies one independent function to each element.

fn affine(src: &[f32], dst: &mut [f32], scale: f32, bias: f32) {
    assert_eq!(src.len(), dst.len());
    for (d, &x) in dst.iter_mut().zip(src) {
        *d = x * scale + bias;
    }
}

src:    [x0 x1 x2 x3]   scale splat [s s s s]
mul/add: independent lane arrows ↓  ↓  ↓  ↓
dst:    [y0 y1 y2 y3]

Contract: output i equals scalar src[i] * scale + bias under the chosen fused-or-unfused semantics. If SIMD uses fused multiply-add while scalar code rounds twice, bitwise results may differ. The likely limit is memory bandwidth once the function performs only a few operations per element. SIMD helps for long slices already in cache or streamed sequentially. It disappoints for tiny calls, expensive setup, alias checks, or when conversion dominates arithmetic. Handle the remainder with a scalar epilogue; it is simple and usually fast here.

Zip extends the pattern to multiple streams.

fn mix(a: &[f32], b: &[f32], out: &mut [f32], t: f32) {
    assert!(a.len() == b.len() && b.len() == out.len());
    for i in 0..out.len() { out[i] = a[i] * (1.0 - t) + b[i] * t; }
}

Three streams increase bandwidth pressure and require an explicit equal-length policy. Exercise: specify whether mix permits out to be the same allocation as a, then draw the addresses touched by one four-lane iteration.

3. Clamp, threshold, and absolute value#

Scalar clamp often hides semantic traps.

fn clamp_signal(xs: &mut [f32], lo: f32, hi: f32) {
    for x in xs { if *x < lo { *x = lo } else if *x > hi { *x = hi } }
}

x       [ -2.0   NaN   0.4   9.0 ]
x < lo  [ true false false false ]
x > hi  [false false false  true ]
result  [  lo    NaN   0.4    hi ]

Contract details include lo <= hi, NaN propagation, and the sign of zero. Target min/max instructions do not all implement Rust-like NaN behavior; compare-and-select is easier to reason about. Integer absolute value must define the minimum signed value: wrapping, saturating, widening, or rejecting it.

fn abs_i16_wrapping(x: i16) -> i16 { x.wrapping_abs() }
fn threshold(x: u8, cut: u8) -> u8 { if x >= cut { 255 } else { 0 } }

These are compute-light and commonly bandwidth-bound, yet SIMD reduces branch unpredictability and loop overhead. It wins when many values share the same operation and loses when each value triggers substantial distinct work. A masked final vector or scalar tail both fit; never read beyond the slice merely because unwanted lanes are masked later. Exercise: write truth tables for f32::NAN, -0.0, infinities, and reversed bounds before choosing clamp operations.

4. Comparisons are masks, not little booleans#

A vector comparison produces one truth value per lane. Masks can select values, suppress supported operations, or be compressed into bits.

fn count_in_range(xs: &[i32], lo: i32, hi: i32) -> usize {
    xs.iter().filter(|&&x| x >= lo && x < hi).count()
}

values:   [ 3  10  -1   7  12   4   8   0 ]
>= lo:    [ 1   1   0   1   1   1   1   0 ]
< hi:     [ 1   0   1   1   0   1   1   1 ]
and mask: [ 1   0   0   1   0   1   1   0 ] -> popcount 4

Contract: bounds are half-open, every input is counted once, and the count type cannot overflow for valid slice lengths. The efficient reduction may count mask bits rather than expand booleans to full integers. Mask representation is API- and target-specific, so keep the algorithm phrased as compare, combine, count. SIMD is strong when comparisons are uniform; loading cold data remains the bottleneck for one-pass classification. Exercise: alter the contract to inclusive bounds and identify integer edge cases where rewriting as subtraction is unsafe.

5. Search and first match#

Search combines parallel testing with an ordered answer.

fn find_byte(haystack: &[u8], needle: u8) -> Option<usize> {
    for (i, &b) in haystack.iter().enumerate() {
        if b == needle { return Some(i); }
    }
    None
}

chunk base 16: bytes [a b x c x d e f]
equal mask:          [0 0 1 0 1 0 0 0] -> numeric bits 0b0001_0100
first set lane:             2          -> answer 18

Compare all lanes, convert the mask to a bitset, then locate its least significant set position. The chunk loop still exits early, preserving the first-match contract. Be explicit about bit ordering: “lane zero maps to bit zero” is an algorithm assumption to verify at the API boundary.

Search is limited by memory for absent or late needles and by setup overhead for very short strings. It can lose when matches are almost always at byte zero because scalar exits after one load. For a tail, use a partial safe load, a padded local buffer, or scalar search. If padding with the needle, mask out padded lanes before finding the first bit. Exercise: design find_any(h, a, b) with two comparisons and prove that its selected index is the earliest match of either byte.

6. Reductions and independent accumulators#

A reduction merges many values into one, creating an apparent dependency chain.

fn sum(xs: &[f32]) -> f32 {
    let mut total = 0.0;
    for &x in xs { total += x; }
    total
}

strict scalar: (((x0 + x1) + x2) + x3) + ...
lane sums:     [x0+x4+...  x1+x5+...  x2+x6+...  x3+x7+...]
horizontal:             combine four lane totals

The lane version changes addition order. For integers with wrapping addition, reassociation preserves the modular result. For checked integers it can change where overflow is detected. For floating point it changes rounding, NaN payload selection, and possibly signed zero. Thus “approximately equal,” “within an error bound,” and “bitwise scalar order” are different contracts.

Even scalar code benefits from two or four accumulators because each accumulator has a shorter dependency chain. The processor can execute independent additions while previous additions are still in flight. SIMD then makes each accumulator itself a vector. Too many accumulators consume registers and can spill to memory, erasing the gain.

fn min_i32(xs: &[i32]) -> Option<i32> {
    let (&first, rest) = xs.split_first()?;
    Some(rest.iter().fold(first, |m, &x| m.min(x)))
}

Empty-input identity must be settled: Option, a documented identity, or a precondition. The main costs are loads and horizontal finalization; SIMD shines on large arrays and is needless for four values. A scalar tail can feed one scalar accumulator, or remaining values can be folded into a vector lane. Exercise: compare strict-order sum, pairwise tree sum, and four-accumulator sum on adversarial f32 magnitudes.

7. Dot products, widening, and overflow budgets#

fn dot_i16(a: &[i16], b: &[i16]) -> i64 {
    assert_eq!(a.len(), b.len());
    a.iter().zip(b).map(|(&x, &y)| i64::from(x) * i64::from(y)).sum()
}

a i16:       [a0 a1 a2 a3]
b i16:       [b0 b1 b2 b3]
widen:       [ i32 lanes for products ]
accumulate:  [ wider partial sums, periodically promoted if needed ]

Widen before multiplying if the scalar contract multiplies in the wider type. Multiplying in i16 and widening afterward preserves an already-overflowed result, which is different. Determine the maximum product and maximum count before selecting accumulator width. For signed i16, one product can reach 1_073_741_824; an i32 accumulator may overflow after only a few worst-case terms. Some dot-product instructions accumulate grouped narrow products into medium-width lanes, requiring periodic widening.

For f32, fused multiply-add changes rounding relative to separate multiply and add. Dot products are usually friendly: two contiguous reads, reuse-free arithmetic, and no output stream until the end. They fail to scale when inputs are random gathers or the contract demands scalar bit identity. The tail may use scalar products in the same chosen precision. Exercise: compute a safe flush interval for unsigned u8 * u8 products accumulated into u32 lanes.

8. Scans and prefix sums#

A scan emits every partial reduction.

fn prefix_sum(xs: &mut [u32]) {
    let mut carry = 0u32;
    for x in xs { carry = carry.wrapping_add(*x); *x = carry; }
}

The direct loop has a loop-carried dependency: output i is needed for output i + 1. SIMD becomes legal by scanning within a vector using shifted doubling steps, then adding the previous block's carry.

input:       [a b c d]
shift 1/add: [a a+b b+c c+d]
shift 2/add: [a a+b a+b+c a+b+c+d]
add carry k:[k+a k+a+b k+a+b+c k+a+b+c+d]
next carry = final lane

Contract: wrapping u32, in-place output, scalar prefix order modulo 2^32. For floating point, the tree inside each vector does not match left-to-right rounding. Each block needs shuffles and serial carry transfer, so scans gain less than maps or reductions. They help when vectors are wide and the rest of the pipeline consumes scanned blocks. They lose for tiny arrays, strict floating semantics, or expensive cross-lane movement. A scalar tail naturally starts from the final full-vector carry. Exercise: derive the three shift distances needed for eight lanes and count additions per output block.

9. From branches to masks—and back when needed#

fn piecewise(x: f32) -> f32 {
    if x >= 0.0 { x.sqrt() } else { -x }
}

The simple vector form computes a comparison and selects between candidate results.

x:       [ 4  -3   9  -2]
mask:    [ T   F   T   F]
positive candidate and negative candidate -> lane-wise select

But unconditional evaluation of sqrt(x) on negative lanes may raise floating exceptions, create NaNs, or waste work. Predicated arithmetic helps only if the operation truly suppresses inactive-lane effects on the target. A select after two ordinary operations is not suppression; both sides already ran.

Branch-to-mask conversion wins for cheap arms and unpredictable, mixed lane decisions. It loses when one arm is rare and costly, when branches are highly predictable, or when arms perform different memory accesses. If 1% of records need a hash-table lookup, evaluating that lookup for every lane is not clever predication. Alternatives include testing whether any lane is active, partitioning work, or compacting active lanes.

Correctness must include side effects, panics, invalid addresses, and exceptions—not merely the final selected values. Tail handling follows the surrounding data path, but inactive tail lanes must not trigger either arm's unsafe access. Exercise: classify division, table indexing, pure addition, and logging as safe or unsafe to compute on both sides.

10. Fixed-size lookup and shuffle tables#

Small tables can live conceptually inside registers and be indexed by lane shuffles.

const HEX: &[u8; 16] = b"0123456789abcdef";
fn nibble_to_hex(n: u8) -> u8 { HEX[usize::from(n & 15)] }

table: [0 1 2 3 4 5 6 7 8 9 a b c d e f]
index: [3 f 0 9 ...] -> shuffle -> [3 f 0 9 ... as ASCII]

The contract deliberately masks to four bits; without masking, out-of-range shuffle semantics differ by ISA. Tables larger than one register may use high index bits to select subtables and masks to combine candidates. SIMD avoids cache lookup latency and performs many substitutions at once. It loses when setup and table broadcasts exceed a handful of scalar indexed loads, or when tables are large and changing. For a short tail, scalar indexing is clear; a padded temporary also works if only valid output bytes are copied back. Exercise: design a 32-entry ASCII-folding table from two 16-entry tables and state how invalid indices are rejected.

11. Byte and text classification#

Classification usually asks whether each byte belongs to a set.

fn is_json_space(b: u8) -> bool { matches!(b, b' ' | b'\n' | b'\r' | b'\t') }
fn count_digits(s: &[u8]) -> usize {
    s.iter().filter(|&&b| b >= b'0' && b <= b'9').count()
}

bytes:  [ '7' 'x' '\n' '0' 0xff ' ' ]
digit:  [  1   0    0    1   0   0  ]
space:  [  0   0    1    0   0   1  ]

Unsigned comparisons matter for bytes above 0x7f; a signed-byte comparison can misclassify them. Ranges suit compares, tiny arbitrary sets suit equality combinations, and richer ASCII classes can use nibble tables. Compressing a delimiter mask to bits is especially useful because downstream parsing can skip directly between set bits.

This is bandwidth-friendly and branch-free, but UTF-8 semantics are not ASCII semantics. ASCII punctuation can be found byte-wise because continuation bytes never equal ASCII bytes. Unicode character classes require decoding and often tables too large for a register. The last chunk requires a safe partial read; classification of padded zeros must be excluded from the valid-lane mask. Exercise: build a byte-wise mask for JSON structural characters and explain why quoted-string state is a separate problem.

12. Bitsets: masks become durable data#

Bitsets pack predicates into one bit per item.

fn intersection_count(a: &[u64], b: &[u64]) -> u64 {
    assert_eq!(a.len(), b.len());
    a.iter().zip(b).map(|(&x, &y)| (x & y).count_ones() as u64).sum()
}

word A: 10110100
word B: 00111100
A & B:  00110100 -> popcount 3

The natural lanes are machine words, not individual logical elements. Vector bitwise operations are cheap; population count support and horizontal reduction determine the rest. The contract must define bit numbering and whether unused bits in the final word are guaranteed zero. If padding bits are not zero, mask the final word before counting.

SIMD helps for large dense bitsets and combined expressions such as (a & b) & !c. Sparse sets may be better represented as sorted indices, avoiding scans over mostly zero words. Memory bandwidth dominates simple intersections; fusing several logical operations avoids intermediate bitsets. Exercise: compare bytes read for intersecting a million-bit dense set versus two sorted lists containing 100 indices each.

13. Filtering and compaction#

Filtering computes a predicate and writes only selected elements.

fn keep_positive(src: &[i32], dst: &mut Vec<i32>) {
    dst.extend(src.iter().copied().filter(|&x| x > 0));
}

values: [ 5 -2  7  0 -1  8  4  0]
mask:   [ 1  0  1  0  0  1  1  0] -> 10100110 by chosen bit order
compact:[ 5  7  8  4] ; output advances by popcount 4

The hard step is packing selected lanes contiguously while preserving order. Some targets offer compress-store; others use shuffle tables indexed by the mask. At low selectivity, scalar iteration over set mask bits can minimize stores. At high selectivity, copying almost everything with a vector compaction path is attractive. Around middling selectivity, shuffle/compress throughput and output bandwidth decide.

Contract: stable order, enough destination capacity, and defined overlap. In-place stable filtering can be safe when the write cursor never passes the read cursor, but wide stores need careful proof near overlap. If the predicate is expensive or matches in long predictable runs, scalar branching may compete well. The final partial input can be scalar; output has no “tail,” only a variable count. Exercise: make a worksheet for 1%, 50%, and 99% matches: expected loads, stores, branch predictability, and compaction work.

14. Gathers, scatters, and duplicate indices#

fn gather_add(table: &[f32], indices: &[usize], out: &mut [f32], bias: f32) {
    assert_eq!(indices.len(), out.len());
    for i in 0..out.len() { out[i] = table[indices[i]] + bias; }
}

indices lanes: [ 2  90  3  41 ]
addresses:     [near][far][near][farther] -> gather -> arithmetic

Every index must be bounds-checked before an unsafe gather can issue. Masking the result after gathering does not make an invalid address safe. Gathers help when indices land in cache and replace substantial scalar address/instruction overhead. They cannot combine unrelated cache misses into one cheap memory transaction.

Scatter adds a more serious semantic issue.

for i in 0..indices.len() { table[indices[i]] += values[i]; }

If two lanes contain the same index, simultaneous read-modify-write loses an update unless conflict handling is explicit. Even plain assignment needs a policy: scalar order says the later element wins. Solutions include proving uniqueness, detecting lane conflicts, sorting/grouping indices, atomics, or falling back for conflicts. Atomics preserve updates but not necessarily deterministic floating reduction order. Random pages can dominate through cache and TLB misses; restructure to sort indices or process buckets when possible. Use scalar handling for the final few indices and for conflict repair. Exercise: for indices [3, 8, 3, 3], list outputs required by assignment and addition under scalar order.

15. Dense matrices: vectorize a reused dimension#

A naive matrix product is correct but may reload data excessively.

fn matmul(a: &[f32], b: &[f32], c: &mut [f32], m: usize, n: usize, k: usize) {
    for i in 0..m {
        for j in 0..n {
            let mut sum = 0.0;
            for p in 0..k { sum += a[i*k+p] * b[p*n+j]; }
            c[i*n+j] = sum;
        }
    }
}

With row-major arrays, a[i*k+p] is contiguous across p, but b[p*n+j] is strided. Changing loop order to i, p, j broadcasts one a value and walks contiguous rows of b and c.

microkernel computes C rows × columns:
          b[p,j..j+W]
a[i,p] -> [c0 c1 c2 c3] accumulator row 0
a[i+1,p]->[d0 d1 d2 d3] accumulator row 1

A microkernel holds a small rectangular tile of C in vector accumulators across the k loop. Register count limits tile size; cache capacity and reuse determine outer tile sizes. Tiling packs or traverses submatrices so reused A and B data remain in nearby cache. Packing costs a copy but can turn strided accesses into contiguous ones and amortize across many output elements.

Contract choices include dimensions, row-major layout, overlap prohibition, and acceptable floating reassociation/FMA. SIMD helps because each loaded matrix value participates in many operations. It fails for tiny matrices where packing dominates, extremely skinny shapes, or layouts causing constant cache conflict. Edge tiles can use scalar kernels, masked stores, or padded packed panels; established libraries often provide several kernels. Exercise: for a 2 × 4 microkernel, count accumulators and loads per k step, then estimate register pressure.

16. Images: channels, rows, and convolution halos#

RGBA pixels in AoS form interleave channels.

fn brighten_rgba(px: &mut [u8], add: u8) {
    for p in px.chunks_exact_mut(4) {
        for c in &mut p[..3] { *c = c.saturating_add(add); }
    }
}

memory: R0 G0 B0 A0 R1 G1 B1 A1 ...
lanes may map bytes directly; channel mask disables every fourth alpha byte

For uniform byte operations, interleaved data can remain interleaved and use a periodic channel mask. For color transforms mixing R, G, and B, deinterleaving or planar SoA may reduce repeated shuffles. Conversion is worthwhile when a long pipeline reuses planar channels, not for one small adjustment.

A convolution computes neighboring weighted samples.

for x in 1..width-1 {
    out[x] = (u16::from(row[x-1]) + 2*u16::from(row[x]) + u16::from(row[x+1])) / 4;
}

left:   [p0 p1 p2 p3]
center: [p1 p2 p3 p4]
right:  [p2 p3 p4 p5] -> widen, weight, sum, round, narrow

Adjacent output vectors reuse overlapping input; sliding loads or aligned neighboring loads plus shuffles can form windows. Borders require a declared rule: ignore, clamp, reflect, wrap, or supplied halo pixels. Rows may have a stride larger than visible width; never vectorize across padding into the next row by accident. Convolution often benefits from tiling multiple rows so source lines remain cached. Large kernels may be separable, replacing a 2-D kernel with horizontal and vertical passes at the cost of an intermediate image. Tail and border code are naturally separate; padded row buffers can simplify both if copy cost is amortized. Exercise: prove the accumulator width needed for an 8-bit 3×3 kernel whose nonnegative weights sum to 256.

17. Audio and DSP: parallel channels versus recurrence#

Independent samples are easy when an effect has no history.

fn gain(samples: &mut [f32], g: f32) { for x in samples { *x *= g; } }

Interleaved stereo can map lanes to consecutive L,R,L,R samples if both channels use the same gain. Different channel gains use a repeating gain vector or deinterleaving.

An IIR filter is different:

fn one_pole(xs: &[f32], ys: &mut [f32], a: f32, mut prev: f32) {
    for i in 0..xs.len() { prev = a * prev + xs[i]; ys[i] = prev; }
}

time dependency: y0 -> y1 -> y2 -> y3 ; time samples cannot simply occupy independent lanes
channel mapping: [ch0 ch1 ch2 ch3] each lane has its own previous state

Vectorize across independent channels, voices, or filter instances rather than across time. For one channel, algebraic prefix transformations exist, but they alter work, precision, and stability and need a proof. FIR filters depend on input history but not prior outputs; multiple output times can be computed in parallel with overlapping windows.

Audio contracts often require denormal handling, clipping policy, latency bounds, and stable state across callback blocks. SIMD loses if converting between interleaved host buffers and internal SoA consumes the callback budget. Tails correspond to channel counts or frame blocks; preserve per-channel state exactly once. Exercise: map eight lanes for four stereo biquads, and identify where each lane's delay state lives between calls.

18. Quantization, rounding, and narrowing#

fn quantize(x: f32, scale: f32, zero: i32) -> u8 {
    let q = (x / scale).round() as i32 + zero;
    q.clamp(0, 255) as u8
}

float lanes -> divide/multiply reciprocal -> round -> add zero point -> clamp -> narrow bytes

Each arrow hides a contract. Is scale positive and finite? Does round() mean ties away from zero, ties to even, truncation, or current environment mode? What should NaN and infinities produce? Does conversion saturate or return a target-specific indefinite value when out of range?

Replacing division by a precomputed reciprocal can change rounding near bin boundaries. Integer requantization uses widened multiplication, a rounding offset, a shift, and saturation; intermediate overflow analysis is mandatory. Narrowing packs more outputs per register, so lane order after pack operations must be checked.

SIMD excels because the pipeline is identical per element and often compute-heavy enough to hide loads. It fails bit-exact tests when the vector rounding instruction does not match the scalar language operation. A scalar tail is safe only if it uses identical rounding semantics; otherwise use the same conceptual pipeline on padded lanes. Exercise: choose expected outputs for ±0.5, ±1.5, NaN, and values beyond range, then make scalar and SIMD contracts agree.

19. Parsing, UTF-8, and state carry#

Parsing mixes parallel classification with sequential state. The useful split is often “find candidates in parallel, resolve grammar in order.”

fn comma_positions(s: &[u8], out: &mut Vec<usize>) {
    for (i, &b) in s.iter().enumerate() { if b == b',' { out.push(i); } }
}

Finding commas is SIMD search; deciding whether a comma is inside a quoted string requires state. A quote mask can be prefix-XORed so each position knows whether the number of preceding unescaped quotes is odd. Backslash escaping introduces runs whose parity may cross vector boundaries. Carry into each block includes “inside string” and possibly “previous block ended in an odd backslash run.”

block N masks -> local prefix state -> structural bits
      carry in ↑                    ↓ carry out to block N+1

UTF-8 validation similarly combines lane-local byte classes with cross-byte constraints. Leading bytes prescribe continuation counts; continuations can cross a vector boundary. Validation must also reject overlong encodings, surrogates, values above U+10FFFF, and stray continuation bytes. Padding with zeros is not neutral if zero is valid input; always combine with a valid-byte mask.

SIMD helps on long mostly regular text by classifying many bytes and producing compact masks. It helps less when every token triggers branches, allocation, hashing, or user callbacks. Keep block state explicit in a small scalar struct; do not hide it in assumptions about chunk boundaries. For the final block, a padded temporary is attractive because parsers already need validity masks, but only copy the actual bytes. Exercise: process two blocks where the first ends with \\ and the second begins with "; derive escape and quote state for both possible backslash-run parities.

20. Tails are correctness code#

There are three broadly useful strategies.

Scalar remainder: process full vectors, then run the scalar operation from full_len to len. It is easy to audit and ideal when tails are small and scalar semantics match.

Masked partial operation: use an API that guarantees inactive lanes do not access memory. This can avoid branches but must be distinguished from a full load followed by a mask.

Padded temporary: initialize a local vector-sized array, copy remaining inputs into it, compute, then copy valid outputs back. It has copy overhead but is portable and makes every actual load in bounds.

slice ends:       [v0 v1 v2 | end]
unsafe full load: [v0 v1 v2 ??]  mask later cannot undo the read of ??
padded temporary: [v0 v1 v2  0]  all bytes belong to the temporary

A full vector load from an address with only three valid bytes is out of bounds in Rust even if hardware usually tolerates it. At a page end, the extra lane can touch an unmapped page and fault before selection occurs. Tests that allocate large buffers often miss this because neighboring allocation bytes happen to be mapped. Guard-page tests place inaccessible pages around valid data and expose such bugs.

Padding the allocation itself is a fourth strategy only when the allocation contract guarantees readable initialized padding. Spare capacity in a Vec is not initialized readable data, and a slice does not grant access beyond its length. Exercise: test lengths 0..2W+1 with data ending immediately before a protected page; explain which tail designs survive.

21. Legal transformations for dependencies#

A loop-carried dependency is a value or memory effect flowing from one iteration to another. Classify it before attempting to “break” it.

• A true recurrence, such as y[i] = a*y[i-1] + x[i], carries a required value.
• A reduction carries an accumulator but may be reassociated if the operation and contract allow it.
• A scan carries state yet can use an intra-block parallel prefix plus block carry.
• An apparent dependency may vanish after privatizing independent counters or accumulators.
• A memory dependency may be false if alias analysis proves the arrays do not overlap.

Loop interchange changes which dimension occupies lanes. Strip-mining splits an iteration space into vector-width blocks plus inner lanes. Tiling applies strip-mining to retain reused data in cache. Unrolling creates independent operations and can hide latency without changing the data model. Loop fission separates classification from rare work; fusion combines passes to save memory traffic.

Every transformation needs a mapping from old operations to new operations and an ordering argument for dependent pairs. “The compiler probably handles it” is not a proof of overlap safety or floating semantics.

22. A decision tree for a candidate loop#

Are iterations independent?
├─ yes: are useful accesses contiguous or cheaply rearranged?
│  ├─ yes: is there enough trip count or repeated use to amortize setup?
│  │  ├─ yes: vectorize lanes; choose tail; benchmark.
│  │  └─ no: keep scalar or let autovectorization decide.
│  └─ no: can layout, loop order, packing, or batching make them regular?
│     ├─ yes: include conversion cost in the measurement.
│     └─ no: assess gather latency and cache/TLB behavior.
└─ no: what kind of dependency?
   ├─ reduction: choose reassociation and accumulator contract.
   ├─ scan: local prefix plus cross-block carry.
   ├─ independent streams/channels: map streams, not time, to lanes.
   ├─ rare branch work: classify then compact or partition.
   └─ strict recurrence/side effects: retain scalar order unless proven transform exists.

Then ask four veto questions:

1. Can any speculative lane panic, fault, or cause a side effect?
2. Does widening, fusion, rounding, or reassociation change required results?
3. Can input and output overlap in a way wide stores violate?
4. Is memory behavior so irregular that arithmetic width is irrelevant?

23. Cost worksheet: lookup and compaction#

Suppose 64 million records each contain a four-byte key index and only 5% pass a cheap predicate. The passing records trigger a random four-byte table read and emit eight bytes.

Doing table gathers for all records would request twenty times as many lookups as needed. A better pipeline classifies contiguous records, extracts selected lanes, then gathers only selected indices. Whether to materialize a compact index buffer depends on cache locality and whether later stages reuse it. If each selected index touches a separate 64-byte line, useful-byte accounting understates traffic by sixteenfold. If accesses span millions of pages, TLB misses add latency beyond line traffic.

Record observed selectivity, runs of matches, table working-set size, and duplicate-index frequency. Those distributions are algorithm inputs, not benchmark noise.

Part III: Production SIMD on Stable Rust Without Dependencies#

This Rust 1.97.1-era chapter builds stable, dependency-free x86/x86-64 SIMD. Public APIs remain safe and portable; private kernels are checked against scalar oracles. Other architectures use the same boundaries but different intrinsics.

1. Three contracts, not one#

SIMD code combines three contracts that should be reviewed separately.

1. CPU-feature safety: may this process execute this instruction here?
2. Memory safety: are all pointers valid for the complete vector access?
3. Algorithmic correctness: do lanes, masks, tails, and overflow match the oracle?

An AVX2 load from a dangling pointer violates memory safety even on an AVX2 CPU. An AVX2 add over valid memory violates CPU-feature safety on an older CPU. An in-bounds add that skips the last three elements is safe but incorrect.

Keep those arguments separate in each SAFETY comment. “This machine has AVX2” never proves a pointer valid. “The slice is long enough” never proves an AVX2 instruction legal.

2. What std::arch and core::arch provide#

std::arch exposes architecture intrinsics and runtime detection macros. core::arch exposes the architecture intrinsics in no_std code. The x86 modules are selected by the compilation target:

#[cfg(target_arch = "x86")]
use core::arch::x86::*;

#[cfg(target_arch = "x86_64")]
use core::arch::x86_64::*;

Do not import x86_64 merely because development happens on a 64-bit laptop. That module does not exist when compiling for 32-bit x86. Conversely, x86 is the correct module on an x86 target. An alias avoids duplicating every import in larger modules:

#[cfg(target_arch = "x86")]
use core::arch::x86 as arch;

#[cfg(target_arch = "x86_64")]
use core::arch::x86_64 as arch;

Then call arch::_mm_loadu_si128, for example. Put the entire implementation module behind an architecture cfg. Do not scatter unsupported imports through otherwise portable modules.

The principal x86 vector types used here are:

• __m128: four packed f32 lanes in 128 bits;
• __m128i: 128 bits interpreted by integer intrinsics in various lane widths;
• __m256: eight packed f32 lanes in 256 bits;
• __m256i: 256 bits interpreted by integer intrinsics.

Integer intrinsics determine lane interpretation: _mm_add_epi32 sees four 32-bit integers, while _mm_add_epi8 sees sixteen bytes. Intrinsic casts such as _mm_castsi128_ps are bit reinterpretations, not numeric conversions, and normally compile to no instruction.

Intrinsic names encode width and operation:

• _mm_... usually names 128-bit SSE-family operations;
• _mm256_... names 256-bit AVX-family operations;
• ps means packed single-precision floats;
• epi8, epi32, and similar suffixes describe integer lane views;
• loadu and storeu permit unaligned addresses.

Some instructions contain an immediate field in machine code. Stable intrinsics model many such operands as const generics:

// Fragment: compare all eight f32 lanes with ordered, quiet `a > b`.
// Each result lane is all ones or all zeroes; the predicate is an immediate.
let mask = unsafe { _mm256_cmp_ps::<_CMP_GT_OQ>(a, b) };

The argument must be a compile-time constant, not an arbitrary runtime integer. Consult the intrinsic's rustdoc for the precise immediate domain.

3. Compile time, function time, and runtime#

3.1 cfg(target_feature)#

#[cfg(target_feature = "avx2")] asks what the current compilation unit is allowed to assume everywhere in the selected item. It is resolved at compile time. It does not probe the user's CPU.

#[cfg(target_feature = "avx2")]
fn compiled_with_avx2() -> bool {
	true
}

#[cfg(not(target_feature = "avx2"))]
fn compiled_with_avx2() -> bool {
	false
}

This suits a fixed image, not runtime dispatch in one portable binary.

3.2 #[target_feature]#

#[target_feature(enable = "avx2")] permits AVX2 in one function despite a conservative crate baseline. It does not perform runtime detection.

#[target_feature(enable = "avx2")]
unsafe fn avx2_kernel(/* ... */) {
	// AVX2 intrinsics may be used here.
}

On stable Rust, calling such a function is an unsafe operation unless the calling context itself has the required target features enabled. The caller must establish that every required feature is available. In a Rust 2024 crate, an unsafe fn body is not an implicit unsafe block. The unsafe_op_in_unsafe_fn lint warns by default when unsafe operations lack explicit unsafe { ... } blocks; use those blocks to keep each proof visible.

#[target_feature(enable = "avx2")]
unsafe fn explicit_blocks(p: *const __m256i) -> __m256i {
	// SAFETY: caller guarantees `p` is valid to read one full `__m256i`.
	unsafe { _mm256_loadu_si256(p) }
}

That explicit block makes the memory proof visible independently of the feature promise attached to the function.

3.3 Runtime detection#

is_x86_feature_detected! asks whether the running x86 CPU and operating-system state support a named feature. It is available through std, not as a general core facility.

if std::is_x86_feature_detected!("avx2") {
	// SAFETY: this branch established AVX2 support for the current process.
	unsafe { avx2_kernel() }
}

For AVX-family features, detection accounts for the OS support needed to preserve extended vector state; hand-written CPUID checks often get that detail wrong. Detect every feature a kernel requires. AVX2 does not imply FMA in Rust's feature model or in your safety proof. A kernel using _mm256_fmadd_ps must detect both "avx2" and "fma", or be compiled under a boundary that guarantees both.

Dispatch strongest-first:

AVX2 implementation, if all its features are present
then SSE2 implementation, if its features are present
then scalar implementation

On x86-64, SSE2 is part of the architectural baseline, but keeping the explicit branch can make a shared x86/x86-64 module easier to reason about. On 32-bit x86, do not assume SSE2 unless the target or runtime check guarantees it.

4. Discovering and selecting target features#

Ask the installed compiler what the target understands:

$ rustc --print target-features
$ rustc --print target-features --target x86_64-unknown-linux-gnu

The output is target- and compiler-specific. It is a discovery tool, not proof of what the current CPU supports.

Crate-wide flags change the baseline of broad regions of generated code:

$ RUSTFLAGS='-C target-cpu=native' cargo build --release
$ RUSTFLAGS='-C target-feature=+avx2' cargo build --release

These can improve local benchmarks and silently make binaries illegal elsewhere. target-cpu=native bakes assumptions about the build machine into the artifact. CI runners, containers, virtual machines, and customer hosts may differ. The optimizer may emit advanced instructions outside your explicit kernels, including startup or supposedly scalar code, so runtime dispatch cannot rescue a binary whose crate-wide baseline already exceeds the destination CPU.

Prefer a conservative deployment target plus isolated #[target_feature] kernels. Use crate-wide features only when deployment has a documented CPU floor. Record that floor beside release configuration, not merely in benchmark notes.

5. Baseline oracles#

Scalar implementations define semantics and provide universal fallbacks. Keep them plain enough to audit. The optimizer may auto-vectorize them; they remain semantic oracles regardless.

// Complete module: scalar.rs
pub(crate) fn add_scaled_f32(dst: &mut [f32], src: &[f32], scale: f32) {
	assert_eq!(dst.len(), src.len(), "slice lengths differ");
	for (d, &s) in dst.iter_mut().zip(src) {
		*d += s * scale;
	}
}

pub(crate) fn find_byte(haystack: &[u8], needle: u8) -> Option<usize> {
	haystack.iter().position(|&byte| byte == needle)
}

pub(crate) fn clamp_u8(values: &mut [u8], lo: u8, hi: u8) {
	assert!(lo <= hi, "invalid clamp interval");
	for value in values {
		*value = (*value).clamp(lo, hi);
	}
}

pub(crate) fn sum_u8(values: &[u8]) -> u64 {
	values.iter().map(|&value| u64::from(value)).sum()
}

add_scaled_f32 deliberately uses multiplication followed by addition. The SIMD versions below do the same and do not require FMA. Floating-point results can still differ under compiler floating-point choices or because SIMD and scalar evaluation group work differently in other algorithms. Here each output element is independent, so exact bit comparison is reasonable for finite ordinary inputs and the same operations.

sum_u8 returns u64. Its mathematical sum fits only when len * 255 <= u64::MAX. No real Rust slice can approach that bound on current 64-bit targets without running into address-space limits, but state the accumulator contract anyway.

6. A complete x86 kernel module#

The following complete x86.rs module supplies SSE2 and AVX2 kernels. It uses unaligned accesses by default. That is the right default for arbitrary slices and interior subslices.

// Complete module: x86.rs
#![cfg(any(target_arch = "x86", target_arch = "x86_64"))]

#[cfg(target_arch = "x86")]
use core::arch::x86::*;
#[cfg(target_arch = "x86_64")]
use core::arch::x86_64::*;

#[target_feature(enable = "avx2")]
pub(crate) unsafe fn add_scaled_f32_avx2(
	dst: &mut [f32],
	src: &[f32],
	scale: f32,
) {
	debug_assert_eq!(dst.len(), src.len());
	let lanes = 8;
	let vector_end = dst.len() / lanes * lanes;
	let scale_v = _mm256_set1_ps(scale);
	let mut index = 0;

	while index < vector_end {
		// SAFETY: the loop bound proves two full 32-byte reads and one full
		// 32-byte write are in bounds. `loadu`/`storeu` need no alignment.
		unsafe {
			let d = _mm256_loadu_ps(dst.as_ptr().add(index));
			let s = _mm256_loadu_ps(src.as_ptr().add(index));
			let result = _mm256_add_ps(d, _mm256_mul_ps(s, scale_v));
			_mm256_storeu_ps(dst.as_mut_ptr().add(index), result);
		}
		index += lanes;
	}

	for i in vector_end..dst.len() {
		dst[i] += src[i] * scale;
	}
}

#[target_feature(enable = "sse2")]
pub(crate) unsafe fn find_byte_sse2(
	haystack: &[u8],
	needle: u8,
) -> Option<usize> {
	let lanes = 16;
	let vector_end = haystack.len() / lanes * lanes;
	let wanted = _mm_set1_epi8(needle as i8);
	let mut index = 0;

	while index < vector_end {
		// SAFETY: `index + 16 <= vector_end <= len`; the cast changes only
		// pointer type and `_mm_loadu_si128` accepts unaligned addresses.
		let block = unsafe {
			_mm_loadu_si128(haystack.as_ptr().add(index).cast::<__m128i>())
		};
		let equal = _mm_cmpeq_epi8(block, wanted);
		let mask = _mm_movemask_epi8(equal) as u32;
		if mask != 0 {
			return Some(index + mask.trailing_zeros() as usize);
		}
		index += lanes;
	}

	haystack[vector_end..]
		.iter()
		.position(|&byte| byte == needle)
		.map(|tail_index| vector_end + tail_index)
}

#[target_feature(enable = "avx2")]
pub(crate) unsafe fn find_byte_avx2(
	haystack: &[u8],
	needle: u8,
) -> Option<usize> {
	let lanes = 32;
	let vector_end = haystack.len() / lanes * lanes;
	let wanted = _mm256_set1_epi8(needle as i8);
	let mut index = 0;

	while index < vector_end {
		// SAFETY: the rounded-down vector end proves a full 32-byte read.
		let block = unsafe {
			_mm256_loadu_si256(haystack.as_ptr().add(index).cast::<__m256i>())
		};
		let equal = _mm256_cmpeq_epi8(block, wanted);
		let mask = _mm256_movemask_epi8(equal) as u32;
		if mask != 0 {
			return Some(index + mask.trailing_zeros() as usize);
		}
		index += lanes;
	}

	haystack[vector_end..]
		.iter()
		.position(|&byte| byte == needle)
		.map(|tail_index| vector_end + tail_index)
}

#[target_feature(enable = "avx2")]
pub(crate) unsafe fn clamp_u8_avx2(
	values: &mut [u8],
	lo: u8,
	hi: u8,
) {
	debug_assert!(lo <= hi);
	let lanes = 32;
	let vector_end = values.len() / lanes * lanes;
	let lo_v = _mm256_set1_epi8(lo as i8);
	let hi_v = _mm256_set1_epi8(hi as i8);
	let mut index = 0;

	while index < vector_end {
		// SAFETY: a complete 32-byte load and store fit in the mutable slice.
		unsafe {
			let pointer = values.as_mut_ptr().add(index).cast::<__m256i>();
			let block = _mm256_loadu_si256(pointer.cast_const());
			let raised = _mm256_max_epu8(block, lo_v);
			let clamped = _mm256_min_epu8(raised, hi_v);
			_mm256_storeu_si256(pointer, clamped);
		}
		index += lanes;
	}

	for value in &mut values[vector_end..] {
		*value = (*value).clamp(lo, hi);
	}
}

#[target_feature(enable = "sse2")]
pub(crate) unsafe fn sum_u8_sse2(values: &[u8]) -> u64 {
	let lanes = 16;
	let vector_end = values.len() / lanes * lanes;
	let zero = _mm_setzero_si128();
	let mut sums = _mm_setzero_si128();
	let mut index = 0;

	while index < vector_end {
		// SAFETY: the loop condition proves a full unaligned 16-byte read.
		let block = unsafe {
			_mm_loadu_si128(values.as_ptr().add(index).cast::<__m128i>())
		};
		let pair_sums = _mm_sad_epu8(block, zero);
		sums = _mm_add_epi64(sums, pair_sums);
		index += lanes;
	}

	let mut lanes_out = [0_u64; 2];
	// SAFETY: `lanes_out` is exactly 16 writable bytes; storeu permits its
	// natural 8-byte alignment even though `__m128i` may have stricter alignment.
	unsafe {
		_mm_storeu_si128(lanes_out.as_mut_ptr().cast::<__m128i>(), sums);
	}
	let vector_sum = lanes_out[0] + lanes_out[1];
	let tail_sum: u64 = values[vector_end..]
		.iter()
		.map(|&value| u64::from(value))
		.sum();
	vector_sum + tail_sum
}

#[target_feature(enable = "avx2")]
pub(crate) unsafe fn sum_u8_avx2(values: &[u8]) -> u64 {
	let lanes = 32;
	let vector_end = values.len() / lanes * lanes;
	let zero = _mm256_setzero_si256();
	let mut sums = _mm256_setzero_si256();
	let mut index = 0;

	while index < vector_end {
		// SAFETY: the loop condition proves a full unaligned 32-byte read.
		let block = unsafe {
			_mm256_loadu_si256(values.as_ptr().add(index).cast::<__m256i>())
		};
		let partial = _mm256_sad_epu8(block, zero);
		sums = _mm256_add_epi64(sums, partial);
		index += lanes;
	}

	let mut lanes_out = [0_u64; 4];
	// SAFETY: the array provides exactly 32 writable bytes, and storeu has no
	// 32-byte alignment precondition.
	unsafe {
		_mm256_storeu_si256(lanes_out.as_mut_ptr().cast::<__m256i>(), sums);
	}
	let vector_sum: u64 = lanes_out.into_iter().sum();
	let tail_sum: u64 = values[vector_end..]
		.iter()
		.map(|&value| u64::from(value))
		.sum();
	vector_sum + tail_sum
}

6.1 Why the tails are exact#

Each vector end is rounded down:

let vector_end = len / LANES * LANES;

The vector loop touches [0, vector_end); scalar cleanup touches the rest once. There is no padded read, speculative overread, or requirement that an allocator placed accessible bytes after the slice.

Reading a 32-byte vector at len - 1 is undefined behavior even if only one lane is later used. Pointer validity applies to the intrinsic's full access width. Crossing into another live allocation is not repaired by mapped virtual memory.

Masked operations can express some tails on newer instruction sets, but they add complexity and are unnecessary here. For short tails, scalar cleanup is straightforward and usually fast.

6.2 Compare masks and the first byte#

Packed byte equality produces sixteen or thirty-two byte lanes. Each matching lane is all one bits (0xff); each non-match is zero. movemask extracts the high bit of every byte lane into an integer bit mask. Bit zero corresponds to the lowest-addressed byte in these loaded blocks.

trailing_zeros() therefore finds the earliest match in a block. Blocks are visited in increasing address order. The vector search consequently preserves scalar position semantics, including the first index when the needle occurs multiple times. Never call trailing_zeros as an index without first handling the zero mask.

6.3 Unsigned clamp#

SSE2 supplies unsigned byte minimum and maximum, so no sign-bit bias trick is needed for this operation. Casting u8 to i8 when broadcasting preserves the byte's bit pattern. The unsigned min/max intrinsics interpret that pattern as an unsigned lane.

If implementing unsigned comparisons through signed compare instructions, XOR both operands with 0x80 first to bias the ordering. Document that transform; a plain signed compare is wrong above 127.

An ASCII classifier can use the same ingredients. For example, classify lowercase bytes by testing byte >= b'a' and byte <= b'z', combining masks with AND, and applying movemask. Its tail and first-index logic should mirror find_byte, not invent an overread.

6.4 Widening sums and accumulator bounds#

_mm_sad_epu8(block, zero) computes sums of groups of unsigned bytes into wide 64-bit lanes. Despite “sad” in the name, absolute differences from zero are simply byte values. This avoids overflow in byte or 16-bit accumulators.

SSE2 accumulates two u64 lanes and AVX2 accumulates four. Finally, an unaligned store extracts lanes into a scalar array for reduction. This lane extraction is stable, transparent, and easy to verify in assembly.

A narrow accumulator needs a periodic flush whose maximum block count is proven. Do not rely on debug overflow checks inside vector lanes; machine packed adds wrap. For integer dot products, multiply width, signedness, pairwise-add behavior, and accumulator width all need explicit bounds before choosing an intrinsic.

7. Alignment: optional optimization, mandatory contract#

loadu and storeu work with arbitrary valid slice alignment. Modern x86 often handles unaligned accesses efficiently when they stay within a cache line, making them a robust default.

Aligned intrinsics such as _mm256_load_si256 impose a stronger precondition. The pointer must satisfy the required alignment in addition to being valid for the entire width. A normal Vec<u8> does not promise 32-byte alignment.

Use a representation contract when storage itself must be aligned:

#[repr(align(32))]
struct Aligned32<T>(T);

let block = Aligned32([0_u8; 64]);
assert_eq!((block.0.as_ptr() as usize) % 32, 0);

The wrapper aligns its start. An arbitrary subslice such as &block.0[1..] is still misaligned. If an aligned fast path peels elements until alignment, prove that the peel stays in bounds and benchmark whether the extra branch is worthwhile.

Alignment is not pointer validity. A correctly aligned dangling address remains invalid. A valid unaligned address is valid for loadu if the full width is in bounds.

8. Safe dispatch with a cached function pointer#

Public wrappers validate ordinary API preconditions and choose a safe thunk. The thunk is a normal safe function. Only it crosses into the feature-enabled unsafe kernel.

Caching avoids repeating detection in hot call paths. OnceLock is stable in std and can store a plain function pointer.

// Complete module: dispatch.rs
use std::sync::OnceLock;

use crate::scalar;

#[cfg(any(target_arch = "x86", target_arch = "x86_64"))]
use crate::x86;

type AddScaledFn = fn(&mut [f32], &[f32], f32);
type FindByteFn = fn(&[u8], u8) -> Option<usize>;
type ClampFn = fn(&mut [u8], u8, u8);
type SumFn = fn(&[u8]) -> u64;

static ADD_SCALED: OnceLock<AddScaledFn> = OnceLock::new();
static FIND_BYTE: OnceLock<FindByteFn> = OnceLock::new();
static CLAMP: OnceLock<ClampFn> = OnceLock::new();
static SUM: OnceLock<SumFn> = OnceLock::new();

pub fn add_scaled_f32(dst: &mut [f32], src: &[f32], scale: f32) {
	assert_eq!(dst.len(), src.len(), "slice lengths differ");
	let implementation = ADD_SCALED.get_or_init(select_add_scaled);
	implementation(dst, src, scale);
}

pub fn find_byte(haystack: &[u8], needle: u8) -> Option<usize> {
	let implementation = FIND_BYTE.get_or_init(select_find_byte);
	implementation(haystack, needle)
}

pub fn clamp_u8(values: &mut [u8], lo: u8, hi: u8) {
	assert!(lo <= hi, "invalid clamp interval");
	let implementation = CLAMP.get_or_init(select_clamp);
	implementation(values, lo, hi);
}

pub fn sum_u8(values: &[u8]) -> u64 {
	let implementation = SUM.get_or_init(select_sum);
	implementation(values)
}

fn select_add_scaled() -> AddScaledFn {
	#[cfg(any(target_arch = "x86", target_arch = "x86_64"))]
	{
		if std::is_x86_feature_detected!("avx2") {
			return add_scaled_avx2_thunk;
		}
	}
	scalar::add_scaled_f32
}

fn select_find_byte() -> FindByteFn {
	#[cfg(any(target_arch = "x86", target_arch = "x86_64"))]
	{
		if std::is_x86_feature_detected!("avx2") {
			return find_byte_avx2_thunk;
		}
		if std::is_x86_feature_detected!("sse2") {
			return find_byte_sse2_thunk;
		}
	}
	scalar::find_byte
}

fn select_clamp() -> ClampFn {
	#[cfg(any(target_arch = "x86", target_arch = "x86_64"))]
	{
		if std::is_x86_feature_detected!("avx2") {
			return clamp_avx2_thunk;
		}
	}
	scalar::clamp_u8
}

fn select_sum() -> SumFn {
	#[cfg(any(target_arch = "x86", target_arch = "x86_64"))]
	{
		if std::is_x86_feature_detected!("avx2") {
			return sum_avx2_thunk;
		}
		if std::is_x86_feature_detected!("sse2") {
			return sum_sse2_thunk;
		}
	}
	scalar::sum_u8
}

#[cfg(any(target_arch = "x86", target_arch = "x86_64"))]
fn add_scaled_avx2_thunk(dst: &mut [f32], src: &[f32], scale: f32) {
	// SAFETY: this thunk is returned only after successful AVX2 detection.
	unsafe { x86::add_scaled_f32_avx2(dst, src, scale) }
}

#[cfg(any(target_arch = "x86", target_arch = "x86_64"))]
fn find_byte_sse2_thunk(haystack: &[u8], needle: u8) -> Option<usize> {
	// SAFETY: this thunk is returned only after successful SSE2 detection.
	unsafe { x86::find_byte_sse2(haystack, needle) }
}

#[cfg(any(target_arch = "x86", target_arch = "x86_64"))]
fn find_byte_avx2_thunk(haystack: &[u8], needle: u8) -> Option<usize> {
	// SAFETY: this thunk is returned only after successful AVX2 detection.
	unsafe { x86::find_byte_avx2(haystack, needle) }
}

#[cfg(any(target_arch = "x86", target_arch = "x86_64"))]
fn clamp_avx2_thunk(values: &mut [u8], lo: u8, hi: u8) {
	// SAFETY: this thunk is returned only after successful AVX2 detection.
	unsafe { x86::clamp_u8_avx2(values, lo, hi) }
}

#[cfg(any(target_arch = "x86", target_arch = "x86_64"))]
fn sum_sse2_thunk(values: &[u8]) -> u64 {
	// SAFETY: this thunk is returned only after successful SSE2 detection.
	unsafe { x86::sum_u8_sse2(values) }
}

#[cfg(any(target_arch = "x86", target_arch = "x86_64"))]
fn sum_avx2_thunk(values: &[u8]) -> u64 {
	// SAFETY: this thunk is returned only after successful AVX2 detection.
	unsafe { x86::sum_u8_avx2(values) }
}

Detection happens once per operation, not once per vector block. The chosen pointer is immutable, and OnceLock synchronizes racing first calls.

The indirect call can cost more than scalar work on tiny slices. An alternative wrapper checks a small-input threshold before consulting the cached pointer:

pub fn sum_u8(values: &[u8]) -> u64 {
	if values.len() < 64 {
		return scalar::sum_u8(values);
	}
	SUM.get_or_init(select_sum)(values)
}

Thresholds are workload-dependent; benchmark representative distributions rather than copying 64, because setup can move the crossover by several vectors.

One cached pointer per operation is simple but consumes data and initialization code. A cached enum or a single detected capability level can select a family of operations, reducing detection and static storage. That can also couple unrelated kernels and prevent operation-specific choices. Measure before centralizing.

9. Why the public ABI stays scalar#

Expose slices, scalar values, and ordinary return types. Do not expose __m256i in a general public API unless callers are deliberately joining the same target-feature contract.

Vector types create architecture-specific signatures, complicate portability, and can make ABI and inlining behavior part of the compatibility surface. A safe slice API lets ARM builds use the same crate with a scalar fallback. It also gives the implementation freedom to replace AVX2 with another strategy.

Keep feature boundaries out of generic public methods: monomorphization inflates specialized code, while private non-generic kernels are easy to inspect.

10. Inlining across feature boundaries#

#[target_feature] is an optimization boundary as well as a safety boundary. Code requiring AVX2 must not be inlined into a context that lacks AVX2 permission. Rust and LLVM enforce those feature constraints.

Do not put #[inline(always)] on a #[target_feature] function. Rust restricts that combination because “always inline” cannot override feature safety. Ordinary #[inline] is at most a hint and is often unnecessary on a substantial kernel.

The safe thunk may inline into the public wrapper, but the feature-enabled kernel typically remains a call boundary. That is acceptable: vector loops amortize one call. For tiny operations, a scalar threshold avoids paying it.

Mixing legacy SSE and AVX code historically raised transition-penalty concerns. Compilers generally emit vzeroupper where needed when returning from AVX code to code that may use legacy SSE. Do not sprinkle _mm256_zeroupper through loops without inspecting output. Manual insertion can add instructions and obscure compiler boundary handling. Inspect the exact compiler and target used for release when this nuance matters.

11. no_std: compile-time multiversioning without runtime detection#

core::arch works without std. std::is_x86_feature_detected! and std::sync::OnceLock do not. Select one implementation with cfg(target_feature) when the image's CPU is fixed:

pub fn sum_u8(values: &[u8]) -> u64 {
	#[cfg(target_feature = "avx2")]
	// SAFETY: the compile-time target guarantees AVX2 for this image.
	return unsafe { x86::sum_u8_avx2(values) };
	#[cfg(not(target_feature = "avx2"))]
	return scalar::sum_u8(values);
}

Architecture-gate the surrounding module as before. This is one strategy per build, not runtime multiversioning; deployment must enforce the CPU floor. Platform-specific detection must also prove vector-state support. Alternatively, initialization can pass a selected pointer through a context; avoid mutable statics.

12. Differential tests: every boundary matters#

SIMD testing should attack vector boundaries systematically. For a width W, test every remainder 0..W, not just W - 1. Test several complete-vector counts so accumulator and loop increments execute. Use interior slices to force misalignment. Include empty slices and lengths smaller than one vector.

The following complete test module uses deterministic data and no dependencies.

// Complete module fragment: tests.rs, compiled only for tests.
use crate::{dispatch, scalar};

fn bytes(length: usize) -> Vec<u8> {
	(0..length)
		.map(|index| {
			index
				.wrapping_mul(73)
				.wrapping_add(length * 19) as u8
		})
		.collect()
}

fn floats(length: usize) -> Vec<f32> {
	(0..length)
		.map(|index| ((index as i32 % 17) - 8) as f32 * 0.25)
		.collect()
}

#[test]
fn dispatched_add_matches_scalar_for_all_avx2_remainders() {
	for length in 0..(8 * 5 + 8) {
		let src_storage = floats(length + 3);
		let src = &src_storage[1..1 + length];
		let mut expected_storage = floats(length + 5);
		let mut actual_storage = expected_storage.clone();

		scalar::add_scaled_f32(
			&mut expected_storage[2..2 + length],
			src,
			-1.5,
		);
		dispatch::add_scaled_f32(
			&mut actual_storage[2..2 + length],
			src,
			-1.5,
		);

		assert_eq!(actual_storage, expected_storage, "length={length}");
	}
}

#[test]
fn dispatched_find_preserves_first_index() {
	for length in 0..(32 * 3 + 32) {
		let mut storage = vec![7_u8; length + 5];
		let haystack = &mut storage[3..3 + length];
		if length > 0 {
			haystack[length - 1] = 99;
		}
		if length > 35 {
			haystack[35] = 99;
		}
		if length > 2 {
			haystack[2] = 99;
		}
		assert_eq!(
			dispatch::find_byte(haystack, 99),
			scalar::find_byte(haystack, 99),
			"length={length}",
		);
	}
}

#[test]
fn dispatched_clamp_matches_scalar_at_extremes() {
	let intervals = [(0, 0), (0, 255), (1, 254), (127, 128), (255, 255)];
	for length in 0..(32 * 4 + 32) {
		for &(lo, hi) in &intervals {
			let mut expected_storage = bytes(length + 7);
			let mut actual_storage = expected_storage.clone();
			scalar::clamp_u8(&mut expected_storage[1..1 + length], lo, hi);
			dispatch::clamp_u8(&mut actual_storage[1..1 + length], lo, hi);
			assert_eq!(
				actual_storage,
				expected_storage,
				"length={length}, lo={lo}, hi={hi}",
			);
		}
	}
}

#[test]
fn dispatched_sum_matches_scalar_for_misaligned_slices() {
	for offset in 0..32 {
		for length in 0..(32 * 5 + 32) {
			let storage = bytes(offset + length + 1);
			let values = &storage[offset..offset + length];
			assert_eq!(
				dispatch::sum_u8(values),
				scalar::sum_u8(values),
				"offset={offset}, length={length}",
			);
		}
	}
}

The add test compares the whole backing allocation. That catches accidental writes before or after the selected subslice. The clamp test does the same. Offsets vary independently from lengths in the sum test, exercising unaligned addresses and all AVX2 remainders.

Include float extremes deliberately when they belong to the API contract:

#[test]
fn add_special_floats_obeys_elementwise_contract() {
	let src = [0.0, -0.0, f32::INFINITY, f32::NEG_INFINITY, f32::NAN];
	let initial = [1.0, -1.0, 2.0, 3.0, 4.0];
	let mut expected = initial;
	let mut actual = initial;
	scalar::add_scaled_f32(&mut expected, &src, 2.0);
	dispatch::add_scaled_f32(&mut actual, &src, 2.0);
	for (left, right) in actual.into_iter().zip(expected) {
		if right.is_nan() {
			assert!(left.is_nan());
		} else {
			assert_eq!(left.to_bits(), right.to_bits());
		}
	}
}

NaN payload propagation is not a sound cross-implementation equality contract unless explicitly designed and tested at the bit level. The example checks NaN-ness and exact bits for non-NaNs.

13. Direct kernel tests#

Dispatch tests do not prove every compiled kernel ran, so gate direct calls by runtime detection and run the same remainder, offset, and extreme-value matrix:

if std::is_x86_feature_detected!("avx2") {
	// SAFETY: AVX2 was detected and `input` is a valid slice.
	let actual = unsafe { crate::x86::sum_u8_avx2(input) };
	assert_eq!(actual, scalar::sum_u8(input));
}

Never call an AVX2 kernel unconditionally in CI; compile coverage and execution coverage are separate concerns.

14. Making dispatch testable#

Global OnceLock state intentionally cannot be reset safely. That makes tests which mutate fake capabilities awkward. Separate pure selection policy from process-global detection.

#[derive(Clone, Copy, Debug, Eq, PartialEq)]
enum X86Level {
	Scalar,
	Sse2,
	Avx2,
}

fn choose_level(has_sse2: bool, has_avx2: bool) -> X86Level {
	if has_avx2 {
		X86Level::Avx2
	} else if has_sse2 {
		X86Level::Sse2
	} else {
		X86Level::Scalar
	}
}

#[test]
fn policy_is_strongest_first() {
	assert_eq!(choose_level(false, false), X86Level::Scalar);
	assert_eq!(choose_level(true, false), X86Level::Sse2);
	assert_eq!(choose_level(true, true), X86Level::Avx2);
	assert_eq!(choose_level(false, true), X86Level::Avx2);
}

The last case models the policy input, not a claim about real feature implication. Kernel requirements still need independent checks. If an implementation needs AVX2 and FMA, model both booleans and select it only when both are true.

Private helpers may accept a safe implementation pointer so tests can inject the scalar path or a counting stub. Never expose arbitrary pointers to unsafe feature kernels; keep the safety boundary under module control.

15. Production module layout#

A small crate can use this structure:

src/
  lib.rs
  scalar.rs
  x86.rs
  dispatch.rs

lib.rs declares portable and conditional modules, then re-exports safe APIs:

mod dispatch;
mod scalar;

#[cfg(any(target_arch = "x86", target_arch = "x86_64"))]
mod x86;

pub use dispatch::{add_scaled_f32, clamp_u8, find_byte, sum_u8};

scalar.rs owns semantics, x86.rs owns intrinsics, dispatch.rs owns detection and thunks, and lib.rs owns the portable surface. On non-x86 targets the x86 module is inactive and users retain identical signatures. Add a selector hierarchy only when a second architecture makes it useful.

16. Code size and operational choices#

Multiversioning duplicates loop bodies. Four operations times scalar, SSE2, and AVX2 already produce twelve bodies. Generic helpers instantiated inside each feature level can multiply this further.

Measure release binaries with size, nm, or objdump; optional developer tools do not become implementation dependencies.

Function-pointer dispatch trades an indirect branch for low steady-state detection cost and compact call sites. A direct detection branch may be better for a rarely called operation. An enum match can make all selected operations share one capability cache but can duplicate branch logic at each call. There is no universally best dispatch shape.

Small-input scalar thresholds reduce call and setup overhead but add a branch and maintain two paths in the wrapper. Tune thresholds with realistic lengths, hot/cold cache states, and host diversity. Do not infer production crossovers from one target-cpu=native laptop benchmark.

17. Inspecting generated assembly#

Correct intrinsics can still compile into disappointing loops. Inspect optimized output for the actual target configuration.

Ask rustc to emit assembly:

$ cargo rustc --release --lib -- --emit=asm
$ find target/release/deps -maxdepth 1 -name '*.s'

Search for kernel symbols and verify:

• the expected packed loads, arithmetic, compares, and stores appear;
• the vector loop increments by the intended width;
• bounds checks are absent from the hot vector loop;
• scalar cleanup handles only the remainder;
• AVX2 code is confined to the feature-enabled function;
• no accidental FMA appears in an AVX2-only contract;
• reduction does not spill excessively;
• transition handling such as vzeroupper is sensible at boundaries.

Symbol names may be mangled. An inspection harness can use #[unsafe(no_mangle)] in Rust 2024, but do not ship that export accidentally. Compiler Explorer and llvm-objdump are also useful. Inspection complements tests; it proves neither pointer validity nor feature-safe dispatch.

18. Closing design#

Production SIMD is less about writing one packed add than about maintaining clear boundaries. The scalar module defines meaning. The architecture module translates that meaning into lanes while proving full-width pointer validity. The dispatcher proves CPU support and crosses the feature boundary once. The public API remains safe and portable.

Stable Rust supplies all required pieces: core::arch intrinsics, #[target_feature], architecture and feature cfgs, std runtime detection, and OnceLock for cached safe function pointers. No external crate is necessary for these kernels.

Start with exact tails and unaligned accesses. Add wider paths only when measurements justify their code size and call overhead. Test every remainder, every practical alignment, and the semantic extremes. Finally, inspect assembly—not to replace the safety argument, but to confirm that the implementation the CPU receives is the implementation you intended.

Part IV: Portability, Autovectorization, and SIMD Crates#

SIMD portability means expressing parallel work without hiding it from the compiler, selecting instructions a machine may execute, and preserving behavior on every deployment target. Treat those as separate engineering decisions; their separation matters more than any intrinsic.

Examples assume stable Rust unless marked nightly-only; status is current as of August 4, 2026. Check linked compiler and crate documentation before fixing an MSRV or feature policy.

1. Begin with scalar code the optimizer can understand#

Explicit intrinsics are not the first rung of the ladder. A clear scalar loop often gives LLVM enough information to emit SIMD instructions. That result is called autovectorization.

Autovectorization has an important advantage: the source remains ordinary Rust, while each target gets instructions suitable for it. Its equally important disadvantage is that it is an optimization, not a language guarantee.

Consider a slice addition kernel.

pub fn add_into(dst: &mut [f32], a: &[f32], b: &[f32]) {
    assert_eq!(dst.len(), a.len());
    assert_eq!(dst.len(), b.len());

    for i in 0..dst.len() {
        dst[i] = a[i] + b[i];
    }
}

This shape is promising because:

• every iteration does the same operation,
• accesses advance by one element,
• there are no data-dependent exits,
• there are no opaque calls in the body, and
• the borrowed slices communicate useful aliasing facts.

The indexing expressions still contain conceptual bounds checks. LLVM may prove and remove them, but source structure affects how easy that proof is. An iterator spelling makes the common length explicit to Rust's iterator machinery.

pub fn add_into_zip(dst: &mut [f32], a: &[f32], b: &[f32]) {
    assert_eq!(dst.len(), a.len());
    assert_eq!(dst.len(), b.len());

    for ((out, x), y) in dst.iter_mut().zip(a).zip(b) {
        *out = *x + *y;
    }
}

“Iterators are slow” and “iterators always vectorize” are both poor rules.

1.1 The loop vectorizer and the SLP vectorizer#

LLVM commonly reaches SIMD through two related optimizations.

The loop vectorizer combines operations from different iterations. Four scalar additions from four successive iterations may become one four-lane addition. It may also create a scalar remainder loop for a length not divisible by the vector width.

The SLP vectorizer combines independent, similar statements in a basic block. SLP means superword-level parallelism, and it can act even when no loop is present. Four adjacent field multiplications in a small pixel calculation, for example, are candidates. Whether they combine depends on layout, extraction costs, target instructions, and surrounding code. Loop and SLP vectorization can also cooperate inside one function.

Stable rustc exposes switches that are useful for experiments:

-C no-vectorize-loops
-C no-vectorize-slp

Disable one at a time to learn which optimizer produced a speedup. Do not ship the slower form merely because it makes a benchmark story simpler.

1.2 Optimize an optimized build#

Debug builds prioritize compilation speed and debuggability. They retain checks and often miss the transformations needed for useful SIMD. Evaluate autovectorization in release mode.

cargo build --release
cargo test --release

For a small standalone experiment:

rustc -O --crate-type lib kernel.rs --emit asm
rustc -O --crate-type lib kernel.rs --emit llvm-ir

Cargo can pass code-generation flags through RUSTFLAGS.

RUSTFLAGS='-C no-vectorize-loops' cargo bench
RUSTFLAGS='-C no-vectorize-slp' cargo bench

Flags in RUSTFLAGS rebuild affected crates and can change dependencies too. Keep benchmark configurations and build caches clearly separated.

1.3 Aliasing is a question, not a magic word#

A vectorizer must preserve the behavior of overlapping memory accesses. If writing dst[i] could change a future src[i + 1], batching iterations might be illegal. Safe references already rule out many harmful overlaps.

During a call, safe Rust cannot derive overlapping live &mut [u8] and &[u8] borrows. Raw pointers and FFI erase much of that evidence and may force runtime alias checks. Do not switch to pointers merely to “help SIMD”; they can remove optimizer information. When in-place work is intended, say so directly with one &mut [T] argument.

1.4 Bounds checks and loop shape#

Bounds checks are usually removable when all accesses share a proven trip count. Hoist length relations outside the loop. Avoid repeatedly asking unrelated containers for lengths in the body.

A branch such as if i < a.len() in every iteration mixes policy with the hot operation. Prove equal lengths first, or deliberately slice all operands to a common prefix. Those contracts differ: the former panics while the latter truncates. Never change that behavior accidentally while reshaping a loop.

Fixed chunks_exact(4) can expose four independent operations and make tails explicit. It can also inhibit a target that prefers eight or sixteen lanes and duplicate remainder logic. Use this transformation only when inspection shows that it helps the shipping targets.

1.5 Calls, branches, and inlining#

An unknown call in every iteration may block vectorization. Small visible functions can be inlined and then vectorized. Trait objects, function pointers, logging, allocation, and synchronization are usually harder barriers.

#[inline] is a hint, not a vectorization command. It can increase code size and may have no effect.

Branches are not automatically fatal. LLVM may turn a simple condition into lane-wise selection. A loop replacing negative values with zero, for example, can become a vector compare and select. Complicated control flow, early exits, and data-dependent loop bounds are less friendly. Splitting one pass into several branch-free passes may vectorize but increase memory traffic. The supposedly helpful rewrite can therefore lose overall.

1.6 Reductions expose semantic constraints#

A sum carries one iteration's accumulator into the next.

pub fn sum(xs: &[f32]) -> f32 {
    let mut total = 0.0;
    for &x in xs {
        total += x;
    }
    total
}

Integer addition without overflow complications is associative in wrapping arithmetic. Floating-point addition is not associative under IEEE-754 rounding. A parallel reduction changes grouping and may change the low bits, NaN behavior, or signed zero.

Rust does not silently grant broad fast-math permission. Consequently, a float reduction may remain scalar even when a map loop vectorizes. Multiple partial accumulators can expose instruction-level parallelism without promising SIMD.

Four partial accumulators can expose instruction-level parallelism, but their final tree has a different rounding order from the simple sum. Document that contract before using the rewrite. Reproducibility and throughput are design requirements, not compiler trivia.

1.7 Inspect rather than assume#

Timing alone says that something changed, not why. Assembly reveals instructions, unrolling, tails, and calls. LLVM IR can reveal vector operations before final instruction selection.

Use cargo asm, cargo llvm-ir, Compiler Explorer, or rustc's --emit outputs as appropriate. Third-party Cargo subcommands have their own installation and version requirements. Official rustc code-generation option documentation is at https://doc.rust-lang.org/rustc/codegen-options/index.html.

Optimization remarks can explain why LLVM did or did not vectorize a loop. The exact remark flags, formats, and usefulness are LLVM/rustc-version sensitive. Some diagnostics require unstable compiler options. Treat blog-post command lines as starting points, not permanent interfaces.

Inspect with the production optimization level, target, features, panic strategy, LTO, and call context. Inlining can make an isolated function differ from its production form.

-C target-cpu=native lets rustc optimize for the build machine. It is excellent for a binary that stays on that machine or a homogeneous fleet. It is dangerous for a redistributable binary. The result may contain instructions absent on the user's CPU and fail with an illegal instruction.

Benchmark at least these candidates when SIMD matters:

1. the simplest scalar/autovectorized implementation,
2. any source-level rewrite intended to aid autovectorization,
3. the explicit SIMD implementation, and
4. the dispatch plus scalar fallback path.

Measure short, medium, and large inputs. Explicit SIMD setup and dispatch can dominate tiny inputs. Memory bandwidth can erase arithmetic improvements on large inputs.

2. Stable AArch64 NEON without dependencies#

Rust exposes architecture intrinsics through core::arch and std::arch. For AArch64, the module is core::arch::aarch64 or std::arch::aarch64. The intrinsic APIs are the same re-exports; core is suitable for no_std code.

Names follow Arm's intrinsic conventions rather than Rust operator conventions. Representative 128-bit vector types include:

• uint8x16_t for sixteen u8 lanes,
• int16x8_t for eight i16 lanes,
• uint32x4_t for four u32 lanes,
• float32x4_t for four f32 lanes, and
• float64x2_t for two f64 lanes.

The q in many intrinsic names denotes a 128-bit vector form. For example, vaddq_f32 adds four f32 lanes. The suffix describes the element interpretation, not a distinct register file.

vld1q_f32(pointer) loads four contiguous f32 values. vst1q_f32(pointer, value) stores four contiguous f32 values. Their pointer arguments make memory validity an unsafe obligation. They do not check slice bounds.

The definitive API list and per-intrinsic target requirements are in the official docs: https://doc.rust-lang.org/core/arch/aarch64/index.html.

2.1 Kernel one: add two f32 slices#

Keep a scalar implementation both as fallback and executable specification.

pub fn add_f32_scalar(dst: &mut [f32], rhs: &[f32]) {
    assert_eq!(dst.len(), rhs.len());
    for (x, y) in dst.iter_mut().zip(rhs) {
        *x += *y;
    }
}

The NEON body is compiled only for AArch64.

#[cfg(target_arch = "aarch64")]
#[target_feature(enable = "neon")]
unsafe fn add_f32_neon(dst: &mut [f32], rhs: &[f32]) {
    use core::arch::aarch64::{vaddq_f32, vld1q_f32, vst1q_f32};

    let lanes = 4;
    let vector_end = dst.len() / lanes * lanes;
    let mut i = 0;
    while i < vector_end {
        // SAFETY: i..i+4 is in both equally sized slices. The feature contract
        // is supplied by this function and upheld by the dispatching caller.
        unsafe {
            let x = vld1q_f32(dst.as_ptr().add(i));
            let y = vld1q_f32(rhs.as_ptr().add(i));
            vst1q_f32(dst.as_mut_ptr().add(i), vaddq_f32(x, y));
        }
        i += lanes;
    }

    for i in vector_end..dst.len() {
        dst[i] += rhs[i];
    }
}

#[target_feature] changes the function's code-generation contract. Calling such a function is unsafe unless the feature is known to be available. It does not perform runtime detection.

The public dispatcher isolates that contract.

pub fn add_f32(dst: &mut [f32], rhs: &[f32]) {
    assert_eq!(dst.len(), rhs.len());

    #[cfg(target_arch = "aarch64")]
    {
        if std::arch::is_aarch64_feature_detected!("neon") {
            // SAFETY: detection established NEON support.
            unsafe { add_f32_neon(dst, rhs) };
            return;
        }
    }

    add_f32_scalar(dst, rhs);
}

The detection macro is in std::arch, so this exact dispatcher requires std. In no_std, obtain feature knowledge from the platform, build configuration, or caller contract. Do not invent a universal no_std detector: firmware, kernels, and bare metal expose capabilities differently.

Test semantic edges, not only one vector-sized input.

#[test]
fn add_f32_matches_scalar_for_tails() {
    for len in 0..33 {
        let rhs: Vec<f32> = (0..len).map(|i| i as f32 * 0.25).collect();
        let mut expected: Vec<f32> = (0..len).map(|i| i as f32 - 7.0).collect();
        let mut actual = expected.clone();
        add_f32_scalar(&mut expected, &rhs);
        add_f32(&mut actual, &rhs);
        assert_eq!(actual, expected, "length {len}");
    }
}

#[test]
#[should_panic]
fn add_f32_rejects_mismatched_lengths() {
    add_f32(&mut [1.0, 2.0], &[3.0]);
}

2.2 Kernel two: sum absolute byte differences#

Image and media code often needs the sum of absolute differences, or SAD. For each byte pair, compute the absolute difference and sum it into a wide scalar.

pub fn sad_u8_scalar(a: &[u8], b: &[u8]) -> u64 {
    assert_eq!(a.len(), b.len());
    a.iter()
        .zip(b)
        .map(|(&x, &y)| x.abs_diff(y) as u64)
        .sum()
}

NEON provides vabdq_u8 for lane-wise absolute differences. vaddlvq_u8 performs a widening reduction of sixteen bytes to a scalar u16. Reducing every vector prevents a narrow vector accumulator from overflowing.

#[cfg(target_arch = "aarch64")]
#[target_feature(enable = "neon")]
unsafe fn sad_u8_neon(a: &[u8], b: &[u8]) -> u64 {
    use core::arch::aarch64::{vabdq_u8, vaddlvq_u8, vld1q_u8};

    let vector_end = a.len() / 16 * 16;
    let mut total = 0u64;
    let mut i = 0;
    while i < vector_end {
        // SAFETY: i..i+16 is in both equally sized input slices.
        unsafe {
            let x = vld1q_u8(a.as_ptr().add(i));
            let y = vld1q_u8(b.as_ptr().add(i));
            total += vaddlvq_u8(vabdq_u8(x, y)) as u64;
        }
        i += 16;
    }
    total + sad_u8_scalar(&a[vector_end..], &b[vector_end..])
}

pub fn sad_u8(a: &[u8], b: &[u8]) -> u64 {
    assert_eq!(a.len(), b.len());

    #[cfg(target_arch = "aarch64")]
    if std::arch::is_aarch64_feature_detected!("neon") {
        // SAFETY: detection established NEON support.
        return unsafe { sad_u8_neon(a, b) };
    }

    sad_u8_scalar(a, b)
}

#[test]
fn sad_u8_matches_scalar_across_boundaries() {
    for len in 0..65 {
        let a: Vec<u8> = (0..len).map(|i| (i * 17) as u8).collect();
        let b: Vec<u8> = (0..len).map(|i| 255u8.wrapping_sub((i * 11) as u8)).collect();
        assert_eq!(sad_u8(&a, &b), sad_u8_scalar(&a, &b));
    }
}

2.3 Baselines, custom targets, and operating systems#

An architecture name does not completely describe a deployment environment. The target specification chooses a CPU baseline and target features. The operating system, ABI, and hardware determine whether detection is meaningful and available.

On many mainstream AArch64 operating-system targets, Advanced SIMD/NEON is part of the practical baseline. That does not justify assuming the same for every custom target, unusual execution environment, or emulator. Rust's target documentation is the source for built-in target support tiers and caveats: https://doc.rust-lang.org/rustc/platform-support.html.

For a known baseline, compiling a whole artifact with the feature may be appropriate.

RUSTFLAGS='-C target-feature=+neon' cargo build --release --target aarch64-unknown-linux-gnu

That artifact is no longer suitable for a target where NEON is absent. A custom JSON target can encode a product's exact processor contract, but custom targets are outside Rust's normal built-in target guarantees and often need nightly options to use. Treat the target file as versioned platform configuration, not as a casual optimization switch.

Function-level multiversioning preserves a conservative common artifact:

• compile a scalar baseline,
• compile one or more #[target_feature] functions,
• detect once at a boundary, and
• call only a compatible version.

Detection in every tiny call can cost more than the kernel. Cache a selected function pointer or dispatch around a larger batch when profiling supports it.

2.4 NEON is not every AArch64 extension#

Do not place optional operations inside a function enabled only for neon. Dot-product instructions, for example, require the separate dotprod feature. Other extensions likewise have their own names and platform availability.

#[cfg(target_arch = "aarch64")]
pub fn choose_dot_product_path() -> bool {
    std::arch::is_aarch64_feature_detected!("dotprod")
}

A dot-product-specialized body should use #[target_feature(enable = "dotprod")] and be called only after the corresponding check. It may also rely on baseline features implied by that extension and target. Consult each intrinsic's rustdoc rather than inferring requirements from its name.

Keep optional tiers separate:

portable scalar
    -> common NEON kernel
        -> dotprod-specialized kernel
            -> any still-newer, independently detected specialization

More tiers increase code size, tests, dispatch complexity, and maintenance. An extension deserves a tier only when representative benchmarks justify it.

3. WebAssembly simd128#

The wasm32 SIMD extension uses one 128-bit value type, exposed as v128. Operations interpret those bits as lanes such as sixteen i8s, four i32s, or four f32s. There are no Rust source types corresponding to distinct physical vector registers.

Stable intrinsics live in core::arch::wasm32 and std::arch::wasm32. Official rustdoc is at https://doc.rust-lang.org/core/arch/wasm32/index.html.

Loads and stores use byte pointers. Arithmetic names encode lane interpretation, for example f32x4_add and i32x4_mul.

#[cfg(target_arch = "wasm32")]
#[target_feature(enable = "simd128")]
unsafe fn add_f32_simd128(dst: &mut [f32], rhs: &[f32]) {
    use core::arch::wasm32::{f32x4_add, v128_load, v128_store};

    debug_assert_eq!(dst.len(), rhs.len());
    let vector_end = dst.len() / 4 * 4;
    let mut i = 0;
    while i < vector_end {
        // SAFETY: four f32 values occupy 16 valid bytes in each slice.
        unsafe {
            let x = v128_load(dst.as_ptr().add(i).cast());
            let y = v128_load(rhs.as_ptr().add(i).cast());
            v128_store(dst.as_mut_ptr().add(i).cast(), f32x4_add(x, y));
        }
        i += 4;
    }
    for i in vector_end..dst.len() {
        dst[i] += rhs[i];
    }
}

The #[target_feature(enable = "simd128")] contract permits SIMD instructions in that function. Unlike native Rust targets, WebAssembly has no in-module Rust runtime detection macro for simd128. Executing a module is preceded by validation, and a host lacking the extension rejects a module that uses it.

That changes dispatch architecture. Do not put baseline and SIMD functions in one module and expect an in-module detector to rescue validation. Instead, produce two artifacts:

1. a baseline WebAssembly module without simd128, and
2. a SIMD module compiled with simd128 enabled.

The embedding host selects an artifact before instantiation. In a browser, JavaScript can test whether a tiny known SIMD module validates with WebAssembly.validate. Libraries such as wasm-feature-detect package probes, but they are dependencies with their own review needs. Servers and other runtimes can use host-specific capability configuration or attempted validation.

A typical build separates output directories and flags.

cargo build --release --target wasm32-unknown-unknown
RUSTFLAGS='-C target-feature=+simd128' \
  cargo build --release --target wasm32-unknown-unknown --target-dir target/simd128

Ensure the first artifact really lacks SIMD instructions. Global environment flags, cached configuration, and dependencies can otherwise invalidate the baseline plan. Validate both artifacts on representative hosts and inspect them with WebAssembly tooling.

The public Rust wrapper can still have a scalar fallback for non-wasm builds and source-level tests. Inside the SIMD artifact, however, host selection—not a fictional runtime macro—establishes support.

3.1 simd128 versus relaxed SIMD#

WebAssembly fixed-width SIMD (simd128) specifies portable operations with defined semantics. WebAssembly relaxed SIMD is a distinct extension. It permits selected operations to have implementation-dependent choices intended to map efficiently to hardware.

Do not use “relaxed SIMD” as another name for simd128. A host may support fixed SIMD without supporting relaxed SIMD. A module using relaxed instructions needs its own validation and deployment policy. Its allowed result variation may also be unacceptable for deterministic workloads.

The WebAssembly specifications and proposals are indexed at https://webassembly.org/specs/. Rust intrinsic availability and stability can lag or differ from proposal status. Check rustdoc and the target-feature reference for the toolchain being shipped.

4. Portable SIMD in the standard library: still nightly#

As of August 4, 2026, std::simd and core::simd remain nightly-only behind #![feature(portable_simd)]. Track the project at https://github.com/rust-lang/portable-simd and API docs at https://doc.rust-lang.org/std/simd/index.html.

Portable SIMD offers vector and mask abstractions rather than architecture mnemonic wrappers. A vector has an element type and lane count, such as Simd<f32, 4>. Operators act lane-wise. Comparisons produce masks, and masks select lanes or control masked operations.

Conceptually:

• Simd<T, N> stores N lanes of T,
• Mask<Element, N> stores N boolean lane decisions,
• arithmetic operators are lane-wise,
• simd_lt and related methods produce masks,
• mask.select(if_true, if_false) chooses each result lane, and
• reductions collapse lanes to a scalar.

The following example is nightly-only and intentionally uses the unstable feature gate.

#![feature(portable_simd)]

use std::simd::{cmp::SimdPartialOrd, f32x8};

pub fn relu(xs: &mut [f32]) {
    let zero = f32x8::splat(0.0);
    let (chunks, tail) = xs.as_chunks_mut::<8>();

    for chunk in chunks {
        let x = f32x8::from_array(*chunk);
        let positive = x.simd_gt(zero);
        *chunk = positive.select(x, zero).to_array();
    }
    for x in tail {
        if !(*x > 0.0) {
            *x = 0.0;
        }
    }
}

The negated comparison is deliberate: like the vector mask, it maps NaN to zero.

Exact trait imports and APIs may change while the feature is unstable. Pin the nightly toolchain for reproducible builds and revisit code during upgrades.

“Portable” describes semantics and source API, not uniform cost. A lane count that maps to one instruction on one target may require multiple instructions on another. An operation may lower to a short sequence, a scalarized loop, or a library call. Masks can be registers on one ISA and synthesized values on another. Gather, scatter, conversion, and reduction costs vary especially widely.

Portable SIMD therefore removes much source duplication but not benchmarking. Inspect each important target and retain tails and semantic tests.

5. Crates as policy layers#

Crates can provide stable vector types, safer wrappers, dispatch, or all three. They are secondary choices after understanding the required semantics and deployment model. No crate makes hardware differences disappear.

The examples below are deliberately small and conceptual. Consult each crate's current docs for exact feature flags, MSRV, and supported targets. Avoid pinning a textbook to an unsupported “latest version” claim.

5.1 wide: stable portable-looking concrete vectors#

The wide crate provides concrete vector types on stable Rust. Its API is convenient for arithmetic code that wants types such as f32x4 without nightly.

use wide::f32x4;

pub fn four_multiply_add(a: [f32; 4], b: [f32; 4], c: [f32; 4]) -> [f32; 4] {
    let a = f32x4::from(a);
    let b = f32x4::from(b);
    let c = f32x4::from(c);
    (a * b + c).into()
}

Benefits include a compact stable API and implementations across several architectures. Questions include how unsupported operations fall back, which lane widths are available, whether semantics match your NaN and rounding needs, and what code appears on each target.

5.2 safe_arch: typed wrappers around architecture operations#

The safe_arch crate wraps many architecture intrinsics in safer types and functions where the operation itself can be made safe. It reduces direct raw-intrinsic exposure; it does not eliminate CPU feature requirements.

#[cfg(target_arch = "x86_64")]
fn add_four(a: safe_arch::m128, b: safe_arch::m128) -> safe_arch::m128 {
    safe_arch::add_m128(a, b)
}

That tiny x86 example illustrates the layer, not coverage for this chapter's NEON kernel. Feature-gated module availability and compilation target still establish which instructions are legal. Evaluate whether compile-time gating fits your distribution model or whether runtime dispatch is also needed.

5.3 pulp: vectorized closures and runtime dispatch#

The pulp crate focuses on safe, runtime-dispatched SIMD abstractions. Its model can keep feature detection and architecture-specific implementations behind a dispatch API.

use pulp::Arch;

pub fn run_with_best_arch() {
    let arch = Arch::new();
    arch.dispatch(|| {
        // Put work expressed through pulp's supported abstractions here.
    });
}

Real kernels use the crate's SIMD traits and dispatched closure patterns; inspect current examples. The tradeoff is learning a generic abstraction and accepting its supported operation/target matrix. Check binary size and whether dispatch is hoisted far enough from small loops.

5.4 multiversion: generate and select function variants#

The multiversion crate uses attributes to compile function variants for selected CPU feature sets and dispatch among them.

#[multiversion::multiversion(targets = "simd")]
fn square_all(xs: &mut [f32]) {
    for x in xs {
        *x *= *x;
    }
}

This is especially attractive when autovectorizable scalar source merely needs several target contexts. It is less direct when a kernel needs wholly different algorithms per ISA. Generated variants increase build time and code size, and supported target syntax is crate-specific.

5.5 How to evaluate a SIMD dependency#

Evaluate a crate as both code and long-lived policy.

Dependency surface: inspect direct and transitive crates, default features, proc macros, and build scripts. MSRV: confirm the minimum supported Rust version from the crate's maintained documentation and CI. Test it in your own workspace because feature combinations and dependencies can raise the effective MSRV. Security: review unsafe blocks, advisories, release provenance, and response history. Use cargo audit or an equivalent, but do not mistake an empty database query for proof. Maintenance: inspect recent releases, issue handling, bus factor, and compatibility policy. Low release frequency may mean stability or abandonment; read the repository rather than guessing. no_std: verify with the exact feature set and target. Core vector operations may be no_std while runtime detection, allocation, or convenience layers are not. Dispatch: determine whether selection occurs per process, per closure, per function call, or per loop. Confirm thread safety and behavior under emulators, virtual machines, and restricted operating systems. Code generation: inspect final assembly or WebAssembly for every important target. An elegant generic API can still spill registers, duplicate tails, miss inlining, or scalarize. Semantics: test overflow, shifts, NaNs, signed zero, rounding, lane order, and out-of-range conversions. Names that look alike across ISAs do not always imply identical corner cases. Prefer the smallest layer that owns the policy you actually need.

6. Other architectures, honestly scoped#

Rust exposes modules for x86/x86-64, 32-bit Arm, WebAssembly, and other supported targets. The module index is https://doc.rust-lang.org/core/arch/index.html.

On x86, SSE, AVX, FMA, and AVX-512 capabilities form many tiers, and OS extended-state support matters. Use is_x86_feature_detected! plus exact feature contracts rather than CPUID folklore. On 32-bit Arm, NEON baseline assumptions, intrinsics, ABI details, and detection differ from AArch64. RISC-V vectors are scalable rather than fixed 128-bit NEON-style vectors, and target profiles vary. This part does not claim a production recipe for RVV, SVE, SVE2, Power/VSX, or LoongArch vectors.

For any unaddressed ISA, define scalar semantics, learn the target baseline, read official docs, isolate feature-enabled unsafe code, dispatch correctly, test edge cases, and inspect final code.

7. Choose your layer#

No single layer wins every project. Use this matrix as a design prompt, not a ranking.

Choose scalar plus autovectorization first when the loop is regular and performance is adequate. It gives the compiler freedom to choose widths and avoids a feature-dispatch API.

Choose architecture intrinsics when measurements identify a kernel and instruction sequence that autovectorization cannot reliably produce, or when an operation has no good scalar expression. Keep the unsafe region small and preserve a scalar oracle.

Choose nightly portable SIMD when one source-level vector algorithm is valuable enough to justify a pinned nightly compiler and unstable API migration work.

Choose a crate when its policy—types, safe wrappers, or dispatch—matches yours and its audit surface is cheaper than maintaining that machinery internally.

Choose a C library when a mature domain implementation repays FFI and build-system costs. Define ownership, alignment, panic, error, and threading contracts at its boundary. Choose assembly only after inspection proves a durable gap; ABI, unwind, relocation, and audit work remain.

Hybrid designs are normal, but every path needs shared semantic tests and representative benchmarks. Test zero, width boundaries, and nonmultiple tails; record compiler, flags, machine, OS, and inputs. The fastest implementation is not portable if it runs before its feature check, if a baseline Wasm artifact contains SIMD, or if target-cpu=native leaks into redistribution. Portability means explicit machine contracts, correct selection, shared semantics, and evidence.

Part V: Expert SIMD Engineering in Production#

SIMD expertise begins after the first vectorized loop works. The production problem is to make it correct, fast, portable, observable, and maintainable. That requires reasoning across algebra, instruction sets, compilers, memory systems, operating environments, and team boundaries. This part treats a vector kernel as one component of a larger system rather than an isolated trick.

1. A performance model before instructions#

Start with work, traffic, dependencies, and machine resources. Count useful operations per input element. Count bytes read and written at every relevant cache level. Identify loop-carried dependencies before choosing an ISA. Estimate available instruction-level parallelism, not merely lane count. Ask whether the scalar version is compute-bound, bandwidth-bound, latency-bound, or branch-bound.

A vector width of W does not imply a speedup of W. Loads, stores, shuffles, conversion, tails, and dispatch consume time too. Front-end bandwidth may cap a shuffle-heavy loop. Memory bandwidth may make wider arithmetic irrelevant. A dependency chain may leave most execution ports idle.

Use a roofline-like sanity check:

arithmetic intensity = useful operations / bytes transferred
compute ceiling      = vector operations per cycle × clock
memory ceiling       = bytes per second × arithmetic intensity
attainable rate       ≤ min(compute ceiling, memory ceiling)

The model need not predict an exact runtime. Its job is to reject impossible expectations and guide measurement.

Questions that shape a kernel#

• Is the operation associative, commutative, or neither?
• Can independent dependency chains be exposed?
• Does lane order matter to observable semantics?
• Are inputs contiguous, strided, indexed, or pointer-linked?
• Is output dense, sparse, or data-dependent?
• What are the legal aliasing and overlap relationships?
• Which exceptions, saturation rules, and rounding modes are required?
• How large is the realistic working set?
• Does selectivity alter the amount of output traffic?
• Will this kernel run once, repeatedly, or inside a larger pipeline?

2. Reductions: algebra meets dependency depth#

A scalar sum forms one dependency chain:

s0 = (((x0 + x1) + x2) + x3) + ...

Its latency grows linearly even when throughput is high. A balanced tree changes depth from O(n) to O(log n):

t0 = x0 + x1    t1 = x2 + x3
t2 = x4 + x5    t3 = x6 + x7
u0 = t0 + t1    u1 = t2 + t3
sum = u0 + u1

Across a long stream, maintain multiple vector accumulators. If vector-add latency is four cycles, four independent chains can often hide that latency. The exact useful count depends on throughput, ports, unrolling, and register supply. More accumulators eventually increase register pressure and code size.

Deriving an accumulator count#

Suppose an add has latency L cycles and reciprocal throughput T cycles. A rough lower bound for saturating that operation is ceil(L / T) independent chains. This is a starting hypothesis, not a contract. Loads and address generation may become the next bottleneck. The compiler may also schedule chains differently from the source.

For eight vectors, a four-chain stream might be:

a0 += v0    a1 += v1    a2 += v2    a3 += v3
a0 += v4    a1 += v5    a2 += v6    a3 += v7

Combine accumulators with a tree after the loop. Then reduce lanes with the least expensive available sequence. Some ISAs provide horizontal reductions; others favor shuffle-and-add trees. Do not invoke a horizontal instruction every iteration. That recreates a dependency bottleneck and usually adds shuffle work.

Semantics of reductions#

Integer wrapping addition is associative modulo 2^N. Checked integer addition is not freely reorderable because the first reported overflow may change. Signed saturating addition is not associative; unsigned saturating addition is associative because values only move toward the upper bound. Floating-point addition is not associative because each operation rounds. Minimum and maximum require explicit NaN and signed-zero policy.

For floating-point sums, choose a contract:

• Bit-identical to a specified scalar order.
• Deterministic for one chosen vector tree.
• Bounded error independent of thread scheduling.
• Best practical accuracy, perhaps with compensated summation.
• Fast-math behavior allowing reassociation and altered exceptional cases.

Pairwise summation usually has better error growth than a long scalar chain. Multiple accumulators alter rounding but can improve both speed and numerical behavior. Widening f32 inputs into f64 accumulators costs throughput but may greatly reduce error. Compensated algorithms are possible in SIMD, though their extra operations deserve measurement.

Other reduction operators#

Bitwise AND, OR, and XOR are straightforward associative reductions. Population count may reduce per lane only after local byte or word counting. Argmin and argmax carry both value and index through each comparison. Tie-breaking must be encoded, such as choosing the lowest index. Boolean all and any often reduce a comparison mask rather than vector lanes.

3. Scans, transposes, and data motion#

A scan emits every prefix, so a lane scan uses shift-and-add stages at offsets 1, 2, 4, .... This has logarithmic depth but more work than a reduction tree. Long scans add the preceding vector's final value to every lane of the next vector. That carry is a cross-vector dependency; parallel scans instead scan block totals and add block offsets. Exclusive scans shift by the identity, while segmented scans propagate values together with boundary masks.

Transposes bridge array-of-structures and structure-of-arrays layouts through unpack and interleave stages. Count shuffle instructions, dependency latency, port pressure, and extra live registers. Wide registers may retain 128-bit shuffle domains, making cross-lane permutations comparatively expensive. Amortize a layout conversion across reuse; for one touch, scalar field loads may win.

4. Gather, scatter, and locality#

Gather works best when indexed lanes share nearby cache lines. Independent cache misses or page walks erase much of its apparent parallelism. First consider sorting indices, packing hot fields, bucketing dense ranges, or changing producer layout. Validate indices even when lanes are masked; fault and speculation behavior is ISA-specific.

Scatter causes fragmented writes and has ambiguous duplicate destinations unless software defines them. “Last wins” requires order, “sum” requires conflict reduction, and atomic updates require synchronization. A dense temporary plus sequential merge can beat direct scatter.

5. Lookup and nibble techniques#

Byte shuffles can hold 16-entry nibble tables in registers. Split each byte into high and low nibbles, look up partial properties, and combine them. Hex decoding must retain an independent validity mask so invalid bytes cannot alias valid entries. Without byte popcount, table[x & 15] + table[x >> 4] counts each byte. Audit zeroing and out-of-range lookup semantics on every target ISA. Larger tables may require several shuffles and blends, eventually losing to another algorithm.

6. Compress and branch divergence#

Filtering first classifies lanes and then compacts selected values. Compress-store is direct; alternatives use mask-indexed permutations, prefix offsets, or scalar selected stores. Advance output by popcount(mask) and unroll only while masks and vectors fit in registers. Benchmark 0%, 50%, and 100% selectivity plus realistic clustering. Stable filtering preserves order; overlap and in-place guarantees need explicit contracts.

CPU lane divergence becomes predication, computing both alternatives, or partitioning work. Blend when both sides are cheap; compact a rare expensive path; retain predictable scalar branches when they win. Masked memory and exceptional arithmetic still require fault, NaN, and side-channel analysis.

7. Mixed widths: widen early, narrow deliberately#

Many useful kernels read narrow values and accumulate wider results. Examples include u8 × i8 → i32, i16 × i16 → i32, and pixel sums into u32. The width plan determines both overflow behavior and instruction count.

Widening reduces lane count at each step. A 256-bit byte vector contains 32 lanes but only eight i32 lanes. Track how data splits into low and high halves. Missing one half is a classic SIMD correctness bug.

Choose among extension rules explicitly:

• Zero extension for unsigned inputs.
• Sign extension for signed inputs.
• Bias conversion when an instruction expects opposite signedness.
• Pairwise widening operations that combine adjacent lanes.

Intermediate overflow can occur before a final wide accumulator sees the value. Prove bounds for each instruction, not only the mathematical result. For a dot product of K signed bytes, bound products and partial sums. If an instruction saturates an intermediate pair, it may not implement ordinary arithmetic.

Narrowing requires a policy:

Pack instructions often impose a lane ordering that needs a final permutation. Signed and unsigned saturation opcodes are not interchangeable. Always test values around every boundary.

8. Fixed-point, FMA, and approximation#

Fixed-point multiplication doubles fractional bits; rescaling must specify signed rounding and tie behavior. Adding a positive half-unit before an arithmetic shift biases negative ties. Quantization uses real = scale × (integer - zero_point) and often precomputes zero-point corrections. Prove accumulator bounds, distinguish per-channel from per-tensor scales, and validate task-level accuracy. Integer rearrangement can cost more than modern floating-point FMA, so measure both.

FMA rounds a × b + c once and is not bit-identical to separate operations. Specify domain, rounding mode, contraction, NaNs, infinities, subnormals, and signed zero. Absolute error protects values near zero; relative error scales elsewhere; ULP distance measures representable spacing. Derive tolerances from the algorithm and test cancellation rather than widening epsilon until tests pass.

Polynomial vector math combines range reduction, evaluation, and reconstruction. Horner minimizes live values but has a dependency chain; Estrin exposes parallel products but consumes registers. State the approximation domain and metric, fit coefficients reproducibly, and test boundaries and exceptional values.

9. Microkernels, cache tiles, and prefetch#

A microkernel keeps an Mr × Nr output tile in accumulator registers while reusing loaded operands. Count accumulators, operands, masks, pointers, and temporaries before unrolling; inspect assembly for spills. Register, L1, L2, and NUMA blocks serve different reuse distances. Packing pays only when reuse amortizes it, and edge tiles need masks, smaller kernels, or cleanup.

Cache capacity is not fully available: associativity, TLBs, metadata, and competing threads need headroom. Hardware prefetch handles simple streams well. Software prefetch may hide predictable latency, but can waste bandwidth, evict useful lines, and add instructions. Tune distance per machine and workload rather than treating it as universal. Use non-temporal stores only for sufficiently large write-only data unlikely to be reread soon.

10. Case study: delimiter and byte search#

The task is to find a target byte or any byte from a small delimiter set. Load a vector, compare lanes, and extract a bit mask. If the mask is zero, advance by the vector width. Otherwise, trailing-zero count locates the earliest matching lane.

For multiple delimiters, OR equality masks before extraction. For a larger class, a nibble classifier may beat many comparisons. The first-match requirement makes lane order observable.

Correctness details dominate the tail:

• Return the same sentinel as the scalar API.
• Never read beyond the allocation.
• Handle an empty slice without pointer arithmetic tricks.
• Verify matches in the final partial vector.
• Preserve the earliest match when unrolling multiple vectors.

A safe tail can use a scalar loop or a masked load with proven semantics. Copying the tail into a zeroed local buffer is safe but adds overhead. An unaligned full-width load past the logical end is invalid unless the allocation contract explicitly grants readable padding.

Benchmark no-match, early-match, late-match, and periodic-match inputs. Short haystacks may favor scalar code because dispatch and mask extraction are fixed costs.

11. Case study: ASCII validation#

ASCII bytes have the high bit clear. Validation can OR vectors together and test the aggregate high bit once per unrolled block. Alternatively compare bytes against 0x7f using an unsigned strategy.

The OR-reduction reduces mask extraction frequency. It also delays discovery of the first invalid byte. If the API returns only true or false, that is harmless. If it returns an error offset, preserve enough masks to locate the first failure.

Signed byte comparisons are a common trap. Bytes 0x80–0xff look negative under signed interpretation. Sometimes that property is useful; document it rather than relying on intuition.

Validation can fuse with another pass. A parser that already classifies bytes may obtain ASCII validity almost free. But fusion enlarges code, increases live state, and can inhibit reuse. Measure the actual pipeline rather than assuming fewer passes always win.

12. Case study: RGBA image transform#

Consider applying per-channel gain and bias to interleaved 8-bit RGBA pixels. The conceptual operation is simple:

out[c] = clamp(round(in[c] * gain[c] + bias[c]), 0, 255)

The implementation must decide whether to deinterleave channels. Repeated channel-wise work can amortize a transpose. A single affine transform may use widening, patterned constants, and packed arithmetic in place.

Prove intermediate ranges. Choose fixed-point coefficients with an explicit scale and rounding rule. Preserve alpha exactly if that is part of the API. Narrow with unsigned saturation only after rounding.

Image rows may include stride padding. Process each row's logical width and do not blend padding into neighboring rows. Input and output can be identical for a point transform. Partially overlapping buffers require either rejection or a direction-aware strategy.

Test black, white, channel boundaries, maximum gains, negative biases, odd widths, and one-pixel rows. Compare color error under the documented numeric contract, not arbitrary exactness.

13. Case study: audio mixing and FIR#

Mixing several tracks is a reduction over sources for every sample. Use multiple accumulators across sample blocks and FMA for gain application. Clipping behavior belongs at the output conversion, not after each source.

Floating-point mixers must define NaN and denormal policy. Integer mixers need a wide accumulator and a proven source-count bound. Saturating every addition changes the result relative to saturating once.

An FIR filter computes a sliding dot product. Vectorize across output samples to reuse coefficients and load overlapping input vectors. Vectorizing across taps instead requires horizontal reduction for every output. Which orientation wins depends on tap count, output count, and ISA.

For long filters, block algorithms such as FFT convolution may be superior. SIMD does not replace algorithm selection. For short filters, unrolled direct convolution can keep coefficients resident.

Boundary handling must match the signal model: zero padding, reflection, wraparound, or retained history. Real-time code should avoid allocation, locks, page faults, and unpredictable initialization in the callback. Benchmark under callback-sized blocks, not only megabyte buffers.

14. Case study: integer dot product#

Suppose unsigned activations multiply signed weights and accumulate into i32. Some ISAs directly support grouped byte dot products. Others require widening and multiply-add stages.

Inspect the exact intermediate semantics. An instruction may pair products into saturating i16 intermediates. That is incorrect for unrestricted inputs even if the final i32 would fit. Input range constraints can make it valid, but those constraints become part of the proof.

Use several i32 accumulators to break dependency chains. Periodically widen into i64 when reduction length can overflow i32. The safe period follows from the maximum absolute grouped contribution.

Quantized zero points add correction terms. Precompute weight sums when weights are reused. Avoid recomputing input sums for each output channel.

Test worst-case signed combinations, lengths around unroll boundaries, and accumulation limits. Compare against a high-precision independent oracle.

15. Case study: bitset operations#

Bitset AND, OR, XOR, and AND-NOT have high SIMD potential and low arithmetic intensity. Large bitsets are usually memory-bandwidth-bound. Wider vectors may help only until bandwidth is saturated.

Population count after a bitwise operation adds compute. Use native vector popcount when available. Otherwise nibble tables or arithmetic bit counting can be competitive. Reduce byte counts into wider accumulators before narrow lanes overflow.

For intersection cardinality, avoid writing the intermediate bitset. Fuse AND and popcount when the API needs only the count. This halves output traffic and often matters more than instruction cleverness.

Sparse sets may be better represented as sorted integer lists. Roaring-like chunked representations can choose dense or sparse kernels per container. Benchmark density and set-size distributions from production.

The last word may contain unused bits. Mask them according to the logical bit length before counting or comparison.

16. Case study: a matrix microkernel#

Take a row-major A, packed B, and a register tile of C. For each reduction index k, broadcast A scalars and load a vector from B. FMA each pair into independent C accumulators.

Tile dimensions balance reuse against register pressure. A larger Mr reuses each B vector more. A larger Nr reuses each broadcast A value more. Both increase accumulator count.

The surrounding driver must:

• Pack panels in the layout expected by the kernel.
• Handle K tails without reading packing padding unintentionally.
• Scale or initialize C according to alpha and beta semantics.
• Select edge handling for incomplete M and N tiles.
• Partition work to avoid false sharing between threads.

Benchmark square, tall-skinny, short-wide, and tiny matrices. Report packing separately and included. A beautiful steady-state kernel can lose end-to-end on one-shot small inputs.

17. Case study: parser byte classifier#

A JSON-like parser may classify quotes, backslashes, whitespace, and structural characters in parallel. Comparisons or nibble tables produce bit masks for each class. Bit operations then reason over dozens of bytes at once.

Quote state crosses vector boundaries. An escaped quote depends on the parity of preceding backslashes. Prefix XOR can propagate in-string state across a mask. Runs of backslashes require careful carry handling from the previous block.

Classification is not full parsing. Numbers, Unicode escapes, nesting, and error offsets still require semantic work. The SIMD stage should expose a clean contract to the scalar or bit-parallel stage.

Adversarial tests include all quotes, all backslashes, alternating escapes, unterminated strings, and boundaries at every lane. Never overread an untrusted input merely because another mapped page often follows it.

18. Production multiversion architecture#

Keep a scalar implementation as specification and fallback. Add ISA-specific kernels behind one semantic interface. Centralize feature detection and dispatch. Do not scatter runtime checks through inner loops.

Common dispatch choices include:

• Resolve once during initialization and store a function pointer.
• Branch at a high-level call site when calls are coarse.
• Use platform-supported indirect functions where appropriate.
• Build separate binaries for controlled deployment tiers.

Initialization must be thread-safe. The fallback must work before optional initialization if reentrancy is possible. Avoid executing unsupported instructions in feature-detection code itself. On x86, OS support for saving extended vector state matters in addition to CPU feature bits.

A portability matrix#

Examples of tiers might be scalar, 128-bit baseline SIMD, 256-bit SIMD, and a newer dot-product tier. Tier names should describe required features precisely. Avoid using one marketing generation as a proxy for every needed instruction.

19. Baselines, containers, and virtual machines#

A build-machine CPU is not a deployment baseline. Compiler-wide native tuning can accidentally emit unsupported instructions outside guarded kernels. Set a conservative global target and opt specific functions into higher tiers.

Containers do not emulate a CPU. They generally expose host CPU capabilities, subject to runtime and migration constraints. A container image may move between heterogeneous nodes. Dispatch must remain valid on every eligible node.

Virtual machines may hide features, expose a virtual model, or migrate between hosts. Cloud instance families can differ across regions and generations. Record actual feature flags and CPU model with benchmark and incident data.

For fleets, decide whether to:

• Ship one portable binary with runtime dispatch.
• Publish baseline-specific artifacts.
• Constrain scheduling to a declared CPU class.
• Accept lower optimization for operational simplicity.

The right choice is an operational decision, not just a compiler flag.

20. Frequency, instruction mix, and code size#

Some processors may run at different frequencies under sustained use of certain wide or power-dense instructions. The effect varies by microarchitecture, instruction mix, active cores, cooling, firmware, and workload duration. Do not state that “AVX always downclocks” or that it never matters.

Measure the whole application under representative concurrency. A faster kernel can still win despite a frequency change. A tiny wide-vector region can affect neighboring work on some systems. On other systems, no meaningful effect may be visible.

Instruction count alone is insufficient. Consider decoded-uop cache capacity, instruction-cache misses, branch-target resources, and front-end bandwidth. Multiple heavily unrolled ISA variants increase text size. Inlining all variants into callers can multiply that cost.

Prefer out-of-line cold dispatch and compact kernels unless measurement supports expansion. Audit both hot-path instructions and total binary impact.

21. ABI, FFI, and vector-state transitions#

Vector types are often unstable across language or compiler ABIs. Avoid exposing architecture-specific vector values in a public FFI boundary. Pass pointers, lengths, strides, and scalar configuration instead.

Calling conventions define which vector registers are caller-saved and callee-saved. Wider register state may have special preservation rules. Handwritten assembly must emit correct unwind and stack metadata where required.

On relevant x86 transitions, legacy SSE code after AVX code can incur penalties when upper register state remains dirty. vzeroupper is a conventional boundary tool. Modern compilers often insert it, but assembly and unusual FFI boundaries require auditing. Do not sprinkle it inside hot loops.

Foreign callbacks, exceptions, and unwinding complicate assumptions. Keep unsafe vector regions small and avoid allowing unwinding across unsupported assembly boundaries.

22. Safety and error behavior#

Unsafe SIMD code should have a safe contract stronger than its instruction preconditions. Validate lengths, strides, alignment assumptions, and feature support before entering the kernel. Inside the kernel, maintain explicit invariants about processed ranges.

Panic behavior matters. If the scalar API reports an error before modifying output, a vector path must not partially write and then fail unless documented. Checked arithmetic must not silently become wrapping arithmetic. Error offsets and tie-breaking must remain stable when promised.

Overlapping buffers#

Define overlap as one of:

• Forbidden and checked.
• Exactly in-place only.
• Any overlap with memmove-like semantics.
• Safe only in a documented direction.

Vector loads and stores can reorder visible effects relative to scalar iteration. Partial overlap that worked accidentally in scalar code may fail in chunks. Use address-range checks that themselves avoid integer overflow.

No out-of-bounds overread#

Reading past a slice is not justified because loaded lanes are masked later. It can cross an allocation, page boundary, guard page, or memory-mapped file. It can also expose data through speculative or cache side channels.

Safe tail choices include scalar cleanup, proven masked memory operations, or copying into a local padded buffer. Each choice has a cost and a precise validity argument.

23. Constant-time and security caveats#

SIMD does not automatically make code constant-time. Data-dependent table indices, gathers, branches, memory addresses, and early exits can leak information. Even fixed instruction counts can leave cache-visible access patterns.

For secret data, review:

• Control flow dependent on secrets.
• Address calculations dependent on secrets.
• Variable-latency instructions on target CPUs.
• Compiler transformations that reintroduce branches.
• Error paths and observable output length.
• Speculative execution around bounds checks.

Register-resident shuffle tables can avoid secret-dependent cache lookup for small tables. That is one tool, not a complete side-channel proof.

Masked loads after a bounds check may still deserve speculative-execution analysis. Use platform mitigations where the threat model requires them. Do not advertise “constant time” based only on source inspection. Audit machine code and test across supported compiler versions.

24. Benchmark science: find the hotspot first#

Optimization begins with a profile of the real workload. A function consuming 1% of runtime cannot deliver a 2× application speedup. Measure call frequency, input sizes, and surrounding costs.

Collect a realistic corpus rather than one convenient buffer. Preserve length distributions, alignment, value distributions, and selectivity. Include malformed data if production sees it. Separate public benchmark fixtures from confidential traces without erasing important structure.

black_box can discourage obvious constant folding. It cannot make an unrealistic benchmark representative. It does not guarantee a particular optimization barrier on every toolchain. Inspect generated code when benchmark validity depends on it.

25. Cache regimes and tails#

Benchmark at least these regimes:

• Tiny inputs dominated by call and dispatch overhead.
• Hot inputs resident in L1 or L2.
• Last-level-cache-sized inputs.
• Streaming inputs larger than cache.
• Cold starts when initialization or faults matter.

Do not call a buffer “cold” merely because a new iteration began. Eviction requires a defensible method and can introduce its own noise. Report the method.

Test every tail length from zero through at least one full vector width. For unrolled loops, test around the entire unroll quantum. Vary pointer alignment across plausible offsets. Include page-end placements when safety is relevant.

Input distributions affect branches and compress operations. For search, vary match location. For filter, vary selectivity and clustering. For parsers, vary structural density and error location.

26. Controlling the measurement environment#

Record CPU model, microcode, OS, compiler, flags, and binary revision. Pin the benchmark thread when scheduler migration would add noise. Consider isolating a core for sensitive experiments.

Frequency governors, turbo, thermal limits, and background load all affect results. There is no single universally correct frequency policy. Use a policy matching the question and document it. Warm-up can stabilize code, caches, and frequency, but can also heat the CPU.

Run enough independent samples to estimate variation. Randomize variant order when drift could favor one implementation. Watch for throttling over long runs. Avoid comparing results collected under materially different environments.

Criterion-style harnesses provide sampling and statistical summaries. Manual timing is useful for controlled throughput experiments and hardware-counter collection. Both can be wrong if the workload, setup, or synchronization is flawed.

27. Hardware counters and profiles#

Wall-clock time answers whether the program got faster. Counters help explain why. On Linux, perf stat can collect aggregate events. perf record can locate sampled hotspots and annotate instructions.

Useful metrics include:

• Cycles and task-clock.
• Retired instructions.
• Instructions per cycle.
• Branches and branch misses.
• Cache references and misses, interpreted cautiously.
• Front-end and back-end stall indicators when available.
• Memory bandwidth from appropriate uncore or platform tools.

Generic event names can map differently across CPUs. Multiplexed counters reduce precision when too many events are requested. Permissions and virtualization may restrict access. Never present an unsupported counter interpretation as fact.

Lower instructions with unchanged cycles may indicate latency or frequency limits. Higher IPC is not inherently better if total cycles rise. Cache misses without access counts or latency context can mislead.

28. Detecting a memory ceiling#

Symptoms of bandwidth saturation include flat throughput as arithmetic gets wider and scaling that stalls across cores. Measure bytes transferred, not only source-level bytes. Write allocation, packing, and extra passes increase traffic.

Compare achieved bandwidth with a separately measured sustainable bandwidth for a similar access pattern. Do not use a vendor peak number as the practical ceiling. NUMA placement and concurrent traffic can change the ceiling substantially.

Try reducing arithmetic while preserving traffic. If runtime barely changes, memory is likely dominant. Try reducing traffic through fusion or narrower storage. If runtime falls, that supports the diagnosis.

Once bandwidth-bound, SIMD may still reduce core consumption. That can improve fleet capacity even when one-thread latency barely changes. Report the relevant system objective.

29. Reporting benchmark results#

Report distributions or confidence intervals, not only the best run. Include sample count and benchmark setup. State whether numbers are medians, means, or another statistic.

Separate microkernel results from end-to-end results. Include dispatch, packing, allocation, parsing, and output costs when users pay them. A microbenchmark speedup is evidence about a kernel, not an application claim.

Regression thresholds should exceed normal noise. Use historical variance to choose thresholds by benchmark class. Require investigation rather than blindly failing on tiny movement. Track code size and correctness alongside speed.

30. Differential verification#

Every optimized path should be compared with an oracle. Prefer an independent reference, not the same algorithm rewritten with scalar syntax. Shared mistakes can make differential tests agree incorrectly.

For exact operations, compare full outputs, return values, errors, and side effects. For floating point, use the documented exact, ULP, relative, or absolute contract. Check exceptional values separately from ordinary tolerance tests.

Property tests should generate:

• Lengths around every block and tail boundary.
• Alignment offsets.
• Extreme integer values and overflow cases.
• NaNs, infinities, subnormals, and signed zeros.
• Aliasing configurations allowed by the API.
• Structured adversarial patterns, not only uniform random bytes.

Fuzz the dispatcher as well as each kernel directly. Feature overrides can force tiers during testing on capable hardware. Never execute a forced unsupported tier.

31. Memory-safety verification#

Place buffers adjacent to inaccessible guard pages. Exercise every tail length at both page boundaries. This catches full-width overreads that ordinary allocators conceal.

AddressSanitizer can find many out-of-bounds and use-after-free errors. UndefinedBehaviorSanitizer can expose alignment, shift, and arithmetic mistakes in applicable code. MemorySanitizer can detect uninitialized data where supported. Sanitizers do not model every instruction or assembly block perfectly.

Miri is valuable for language-level undefined behavior in supported Rust code. Hardware intrinsics and target-specific execution are limited or unsupported in many Miri scenarios. Use it for safe wrappers and scalar logic, not as proof of intrinsic correctness.

Guard-page tests, sanitizers, fuzzing, and code review are complementary. No single tool closes the safety argument.

32. Cross-architecture confidence#

Compile every supported target in CI. Run tests on real hardware for important tiers. Cross-compilation catches type and intrinsic availability issues but not runtime semantics.

QEMU and similar emulators are useful for functional coverage. They are not reliable performance models of the target CPU. They may differ in obscure exception or instruction behavior. Treat real-hardware testing as necessary for release-critical SIMD paths.

Maintain a matrix of architecture, OS, compiler version, and exercised tier. If a tier lacks routine hardware coverage, state that risk explicitly.

33. Disassembly audits and compiler upgrades#

Intrinsics express operations, not always exact instructions. Compilers may fold, split, spill, or replace them. Audit optimized disassembly for hot kernels.

Look for:

• Unexpected scalarization.
• Stack spills and reloads.
• Redundant moves or broadcasts.
• Calls inside the inner loop.
• Incorrectly placed feature checks.
• Missing transition cleanup at ABI boundaries.
• Unrolled code that overwhelms the front end.

Compiler upgrades can improve or regress lowering. Pin toolchains for reproducibility while testing upgrades intentionally. Retain benchmark baselines and assembly snapshots for critical kernels. Do not require byte-identical assembly when equivalent improvements are possible.

34. Failure-mode table#

35. Incident playbook: illegal instruction#

1. Capture binary hash, CPU model, feature flags, OS, and virtualization details.
2. Identify the faulting instruction and containing function from the program counter.
3. Confirm whether the function was reached through dispatch or emitted into baseline code.
4. Inspect build flags for native tuning, link-time optimization, and inlining across target boundaries.
5. Reproduce on the lowest supported hardware or a matching VM model.
6. Disable the tier through a runtime kill switch if available.
7. Add a regression test that exercises dispatch on that capability set.

Do not assume the user misreported CPU features. The compiler may have emitted an instruction outside the intended guarded region.

36. Incident playbook: correctness mismatch#

1. Preserve the smallest failing input, alignment, tier, and compiler version.
2. Compare scalar and SIMD outputs including status and partial writes.
3. Shrink length while retaining the failure.
4. Move the failing bytes across lane, vector, unroll, and page boundaries.
5. Force individual ISA tiers to localize the defect.
6. Inspect signedness, mask order, lane order, narrowing, and tail logic.
7. Add both the concrete regression and a generalized property.

For floating-point mismatches, classify whether the contract or implementation is wrong. Do not widen tolerance before analyzing the numerical cause.

37. Incident playbook: performance regression#

1. Confirm benchmark environment and statistical significance.
2. Compare end-to-end profiles before examining the kernel.
3. Record cycles, instructions, branches, and cache behavior where available.
4. Diff compiler, flags, CPU, input corpus, and dispatch decisions.
5. Inspect disassembly for spills, scalarization, or code growth.
6. Sweep sizes and distributions to find where the regression begins.
7. Test whether the bottleneck moved to memory or surrounding code.

Rollback is a valid engineering response. A theoretically superior kernel should not remain enabled without production evidence.

38. Review checklist with rationale#

• Semantic contract is written: reviewers need a stable target beyond “matches most inputs.”
• Scalar oracle remains clear: optimized code needs an understandable reference.
• Feature detection is centralized: duplicated checks drift and waste hot-path work.
• Baseline build is conservative: unsupported instructions must not leak globally.
• Every memory access has a range argument: masked consumption does not legalize overread.
• Overlap policy is explicit: chunked stores change accidental scalar behavior.
• Integer intermediates are bounded: final-type width alone proves nothing.
• Float error policy is derived: arbitrary epsilon hides defects.
• All tails and alignments are tested: most lane bugs live at boundaries.
• Exceptional values are covered: normal random floats rarely include them.
• Disassembly was inspected: source-level vector intent may not survive lowering.
• Register pressure was checked: spills can erase arithmetic gains.
• Representative sizes were benchmarked: large hot buffers are not universal.
• Selectivity was varied: data-dependent output and branches change costs.
• Environment is recorded: otherwise results cannot be reproduced.
• End-to-end effect is measured: kernel wins can vanish in integration.
• Security claims have a threat model: constant instruction count is insufficient.
• Fallback and kill switch exist: production recovery should not require a new algorithm.
• Comments explain invariants, not opcodes: intrinsics already reveal operations.
• Maintenance owner is known: dormant ISA code decays across toolchains.

39. Architecture defense questions#

Be prepared to answer these before merging:

1. What profile evidence says this code matters?
2. Why is SIMD preferable to an algorithm or layout change?
3. Which semantic differences from scalar evaluation are allowed?
4. What are the minimum CPU and OS assumptions?
5. How does dispatch remain safe under containers and VM migration?
6. What proves that no access crosses the logical allocation?
7. How are partial overlaps and exact in-place calls handled?
8. What bounds every integer intermediate?
9. What floating-point metric and domain justify the tolerance?
10. Which data distributions make the vector path lose?
11. What is the crossover size for each tier?
12. Does the kernel increase cache traffic or code footprint?
13. Did wider instructions affect whole-application behavior?
14. How were duplicate gather/scatter indices treated?
15. What happens on panic, error, unwind, or callback?
16. Which real machines execute each CI tier?
17. How will compiler upgrades be evaluated?
18. Can the optimized tier be disabled quickly?
19. Who will maintain the unsafe code?
20. What evidence would cause this design to be removed?

40. Open-source integration workflow#

Begin with profile evidence and a concise problem statement. Agree on semantics with maintainers before producing intrinsics. Clarify support policy, minimum toolchain, and acceptable unsafe scope.

Clean up the scalar path first. Separate validation, bulk loop, and tail handling where that improves clarity. Add differential and boundary tests before optimization. This turns the scalar code into a credible oracle.

Introduce one ISA tier at a time. Keep each change reviewable and benchmarkable. Avoid combining a new algorithm, broad refactor, assembly, and several architectures in one patch.

Minimize unsafe code. Put precondition checks in safe wrappers. Comment pointer-range and lane invariants near the operations they justify. Do not use unsafe solely to avoid measuring a safe tail.

Publish reproducible benchmark commands, corpus generation, environment details, and raw summaries. Include cases where the patch is neutral or slower. Maintainers need the decision boundary, not marketing.

Plan rollback. A feature flag, dispatch override, or simple revert path reduces operational risk. Respond to review with evidence and smaller changes, not instruction-count rhetoric.

41. Maintaining a portability ledger#

For every kernel, record:

• Scalar semantic reference.
• Required ISA features.
• Compiler target mechanism.
• Supported operating systems.
• Alignment and aliasing contract.
• Tail strategy.
• Numerical policy.
• Benchmark crossover threshold.
• Real-hardware test coverage.
• Last assembly and performance audit.

This ledger turns tribal knowledge into reviewable maintenance data. Update it when a compiler, baseline, or deployment target changes.

42. Resume credibility#

“Used SIMD” is too vague to demonstrate expertise. A credible project has a measurable workload, correctness evidence, deployment context, and maintained implementation.

Strong evidence includes:

• A profile showing the original hotspot.
• A scalar reference and differential tests.
• Multiple ISA tiers with safe runtime dispatch.
• Benchmarks across size, cache, alignment, and data distributions.
• Hardware-counter analysis tied to a bottleneck model.
• Real-hardware CI or a documented test matrix.
• An upstream review, release, or production deployment.
• Before-and-after end-to-end results with uncertainty.

Honest wording distinguishes scope:

Implemented AVX2 and NEON byte-classification kernels with scalar fallback;
reduced parser-stage median time by 31% on two documented CPU families,
with exhaustive tail tests, guard-page tests, and runtime feature dispatch.

If only a microbenchmark improved, say “microbenchmark,” not “application.” If testing covered one CPU, name it rather than claiming cross-platform speedups. If the work was experimental, call it a prototype. If you contributed part of a team effort, identify your part.

Avoid claims such as “made code 8× faster with 8 lanes” without context. Expertise is visible in tradeoffs, rejected designs, safety proofs, and reproducibility.

43. When not to use SIMD#

SIMD may be the wrong tool when inputs are tiny, control flow is highly irregular, or data follows pointer chains. It may lose when gather locality is poor, output is sparse, or setup dominates. A better algorithm can dwarf instruction-level optimization. A layout change may help more than heroic shuffling.

Maintenance cost matters. An unsafe tier that saves little total CPU may not justify review and CI burden. Library code with unknown deployment CPUs may prefer portable compiler autovectorization. Latency-sensitive code may reject a throughput optimization that increases tail latency.

The expert conclusion is sometimes to keep the scalar loop. That decision should be backed by the same careful evidence as a vector implementation.

44. Final synthesis#

Expert work connects algebra, instructions, memory, deployment, and verification. Keep scalar truth legible, optimize measured bottlenecks, and state safety and numerical contracts. Measure relevant machines and data, then design a fallback before production needs it. Ask not only whether work can be vectorized, but whether the complete system makes it dependable.

Part VI: Building Real SIMD Mastery#

SIMD confidence is the ability to derive behavior and verify it, not to memorize intrinsics. This workbook turns that principle into artifacts you can inspect, test, benchmark, and explain. Keep a lab notebook, a small repository, compiler versions, and raw measurements for every day. In this pinned snapshot, stable dependency-free Rust SIMD means std::arch or core::arch. Portable std::simd remains nightly, so do not present it as a stable portability solution. Maintain a scalar implementation as fallback, readability anchor, and test oracle. Remember throughout: feature checks establish CPU capability; pointer safety is a separate proof.

How to use this workbook#

• Work on one day at a time; do not substitute reading for producing the named artifact.
• Write the expected result before running code, then record where your model was wrong.
• Save commands, compiler flags, target features, CPU identity, and representative inputs.
• Compare optimized code to the scalar oracle with edge cases and randomized inputs.
• Inspect release assembly; debug assembly mostly measures instrumentation and missed optimization.
• Make performance claims only for the workloads, machines, and distributions actually measured.
• Revisit failed verification instead of weakening a test or accepting an unexplained benchmark.

The 30-day mastery plan#

Day 1 — Draw the machine model#

Artifact: A one-page diagram showing registers, lanes, scalar memory, and lane-wise addition. Reasoning goal: Distinguish a vector value from an array and an instruction from an algorithm. Verification: Predict four lane results by hand, then check them with a tiny scalar simulation.

Day 2 — Trace lane semantics#

Artifact: A table for wrapping add, saturating add, minimum, shift, and comparison on u8 lanes. Reasoning goal: State overflow, signedness, and shift behavior without relying on names alone. Verification: Include boundary values 0, 1, 127, 128, 254, and 255 and run assertions.

Day 3 — Make masks concrete#

Artifact: A worked example of compare, mask, select, and bitmask extraction for eight lanes. Reasoning goal: Separate boolean intent from the target's all-zero/all-one mask representation. Verification: Derive the selected output manually and compare every lane with executed code.

Day 4 — Recognize patterns#

Artifact: Annotate scalar loops as map, zip, reduce, scan, lookup, filter, or stateful recurrence. Reasoning goal: Identify easy lane independence and the cross-lane dependencies that resist SIMD. Verification: Have each classification justify where values come from and where they flow next.

Day 5 — Measure data layout#

Artifact: Array-of-structs and struct-of-arrays versions of the same particle update. Reasoning goal: Explain how contiguous fields reduce shuffles and unnecessary memory traffic. Verification: Benchmark both scalar layouts and inspect loads in optimized assembly.

Day 6 — Build a scalar oracle#

Artifact: A clear scalar byte-classification function with exhaustive tests over all byte values. Reasoning goal: Treat reference code as maintained production logic, not disposable scaffolding. Verification: Run tests in debug and release; confirm the oracle handles empty and tiny slices.

Day 7 — Write a first std::arch kernel#

Artifact: An x86/x86-64 SSE2 or AVX2 implementation of the classifier behind a safe wrapper. Reasoning goal: Minimize the unsafe region and document every target-feature and pointer premise. Verification: Differential-test all lengths 0..=128, offsets, and byte values against scalar.

Day 8 — Prove tail handling#

Artifact: Three tail designs: scalar cleanup, padded temporary buffer, and masked conceptual model. Reasoning goal: Show exactly why no full-width load can cross the valid slice boundary. Verification: Run under Miri where applicable and a sanitizer-capable native setup for intrinsics.

Day 9 — Separate dispatch from safety#

Artifact: A dispatch diagram from public safe API to detected kernel and scalar fallback. Reasoning goal: Explain why is_x86_feature_detected!("avx2") does not validate a pointer. Verification: Unit-test forced scalar and forced SIMD paths through an internal test seam.

Day 10 — Read x86 assembly#

Artifact: Commented assembly marking loads, compares, masks, reductions, loop control, and tail. Reasoning goal: Map algorithmic steps to instructions without assuming fewer instructions is faster. Verification: Use cargo asm, emitted assembly, or Compiler Explorer and record exact flags.

Day 11 — Port to NEON#

Artifact: An AArch64 NEON kernel with the same safe contract and scalar oracle. Reasoning goal: Translate operations by semantics rather than searching for name-equivalent intrinsics. Verification: Cross-compile, run on AArch64 hardware or emulation, and differential-test boundaries.

Day 12 — Explore WebAssembly SIMD#

Artifact: A wasm32 SIMD128 variant or a documented feasibility prototype. Reasoning goal: Understand compile-time target features and host/runtime support as different layers. Verification: Inspect Wasm for v128 operations and run tests in at least one SIMD-capable engine.

Day 13 — Challenge the autovectorizer#

Artifact: Three equivalent loops: vectorized, intentionally inhibited, and repaired. Reasoning goal: Recognize aliasing, bounds checks, control flow, and reductions as optimization inputs. Verification: Compare vectorization remarks and assembly, then ensure all forms stay semantically equal.

Day 14 — Evaluate crates responsibly#

Artifact: A decision note comparing scalar, std::arch, nightly std::simd, and relevant crates. Reasoning goal: Weigh stability, maintenance, portability, API ergonomics, and generated code. Verification: Pin versions and verify claims against documentation and actual supported targets.

Day 15 — Implement a reduction#

Artifact: Scalar and SIMD sum or maximum with an explicit overflow and NaN policy. Reasoning goal: Account for reassociation, horizontal operations, accumulators, and determinism. Verification: Test extremes and compare exact or tolerance-based results according to the contract.

Day 16 — Implement byte search#

Artifact: A memchr-style first-match routine using compare plus mask extraction. Reasoning goal: Convert lane matches into the earliest scalar index correctly. Verification: Exhaustively vary match position, alignment offset, absence, and short tails.

Day 17 — Implement ASCII validation#

Artifact: A validator that returns the first non-ASCII index, not merely a boolean. Reasoning goal: Preserve observable error position while processing many bytes at once. Verification: Generate one invalid byte at every position for lengths around vector boundaries.

Day 18 — Implement delimiter scanning#

Artifact: A scanner producing delimiter positions or a compact bitmask stream. Reasoning goal: Connect masks across chunks while retaining stable global indices. Verification: Compare exact position vectors for dense, sparse, empty, and all-delimiter inputs.

Day 19 — Study a shuffle-heavy algorithm#

Artifact: A small transpose, deinterleave, or table-lookup experiment. Reasoning goal: Estimate when shuffle cost and lane topology dominate arithmetic throughput. Verification: Draw each intermediate lane arrangement and reconcile it with assembly.

Day 20 — Design benchmarks#

Artifact: A benchmark matrix by size, alignment, match density, and input distribution. Reasoning goal: Separate latency, throughput, setup cost, cache effects, and dispatch overhead. Verification: Use repeated samples, report variance, prevent dead-code elimination, and save raw data.

Day 21 — Investigate a regression#

Artifact: A short report on one input where SIMD loses to scalar. Reasoning goal: Attribute cost to evidence: tiny input, branch predictability, frequency, memory, or codegen. Verification: Change one factor at a time and reject explanations unsupported by measurements.

Day 22 — Strengthen property tests#

Artifact: A deterministic randomized differential harness with a saved seed corpus. Reasoning goal: Generate cases around semantic and memory boundaries, not just uniform noise. Verification: Inject a deliberate lane or tail bug and confirm the harness finds and minimizes it.

Day 23 — Audit unsafe code#

Artifact: A safety proof beside each unsafe block, listing bounds, alignment, initialization, and features. Reasoning goal: Make every unsafe operation locally reviewable from explicit invariants. Verification: Ask a reviewer to challenge each premise without reading implementation intent into it.

Day 24 — Test dispatch policy#

Artifact: A dispatch table for x86 baseline/AVX2 and AArch64 NEON or Wasm SIMD. Reasoning goal: Distinguish compile-time enablement, runtime detection, and platform baseline guarantees. Verification: Build each target configuration and prove unsupported CPUs retain scalar behavior.

Day 25 — Consider security#

Artifact: A threat note covering out-of-bounds reads, timing leakage, untrusted lengths, and denial of service. Reasoning goal: Avoid equating branchless SIMD with constant-time behavior. Verification: Run malformed inputs and inspect secret-dependent control flow and memory access patterns.

Day 26 — Integrate like an OSS maintainer#

Artifact: A reviewable commit series: oracle/tests, kernel, dispatch, benchmarks, then documentation. Reasoning goal: Reduce reviewer cognitive load and make correctness review independent of speed claims. Verification: Every commit builds; the final diff contains no generated output or unrelated formatting.

Day 27 — Explain the work#

Artifact: A five-minute talk with one lane diagram, one safety proof, and one benchmark chart. Reasoning goal: Communicate tradeoffs without jargon, hype, or unsupported universality. Verification: A listener can restate the contract, fallback, and measurement scope afterward.

Day 28 — Complete a cold review#

Artifact: A written review of unfamiliar SIMD code using the checklist below. Reasoning goal: Detect semantic, safety, portability, testing, and evidence gaps systematically. Verification: Compare findings with maintainers, issue history, or a second independent reviewer.

Day 29 — Run the capstone gate#

Artifact: A release candidate satisfying the capstone specification and scoring rubric. Reasoning goal: Integrate algorithm design, architecture code, dispatch, evidence, and maintenance. Verification: Run the full target matrix from a clean checkout and archive outputs.

Day 30 — Defend and retrospect#

Artifact: A ten-minute defense plus a one-page account of mistakes and next experiments. Reasoning goal: Derive answers under questioning instead of reciting intrinsic names. Verification: Answer adversarial questions below and mark every mastery item with evidence links.

Progressive exercise set#

Each answer sketch is a direction, not a substitute for writing and verifying the code.

1. Lane arithmetic#

Compute wrapping u8 addition of [250, 1, 127, 128] and [10, 2, 1, 255]. Answer sketch: Add per lane modulo 256: [4, 3, 128, 127]; no carry crosses lanes.

2. Saturation versus wrapping#

Repeat Exercise 1 with unsigned saturating addition. Answer sketch: Clamp each sum to 255: [255, 3, 128, 255].

3. Signed interpretation#

Interpret bytes [0x7f, 0x80, 0xff, 0x01] as i8 and compare them with zero. Answer sketch: Values are 127, -128, -1, 1; true lanes are first and fourth.

4. Mask selection#

Given mask [T, F, T, F], select from a=[9,8,7,6] else b=[1,2,3,4]. Answer sketch: Result [9,2,7,4]; verify whether the API selects mask ? a : b.

5. Bitmask extraction#

Encode match lanes 0, 3, and 7 in an eight-lane least-significant-lane bitmask. Answer sketch: (1<<0)|(1<<3)|(1<<7) = 0b10001001; confirm target lane ordering.

6. First match#

Find the first lane from the nonzero bitmask in Exercise 5. Answer sketch: Use trailing-zero count after checking nonzero; result lane 0.

7. Map recognition#

Decide whether uppercasing independent ASCII lowercase bytes is SIMD-friendly. Answer sketch: Yes: range compares form a mask, then subtract 32 in matching lanes.

8. Recurrence recognition#

Explain why out[i] = out[i-1] + input[i] is not a direct lane-wise map. Answer sketch: Each result depends on the previous result; use a scan algorithm or stay scalar.

9. Reduction semantics#

Why can SIMD floating-point sum differ from scalar left-to-right sum? Answer sketch: Parallel partial sums reassociate non-associative floating-point addition.

10. Layout choice#

For updating only particle x, compare {x,y,z} records with separate x, y, z arrays. Answer sketch: Separate x values are contiguous, avoiding gathers/deinterleaving and unused loads.

11. Alignment#

May an unaligned load be safe and still have a performance consequence? Answer sketch: Yes if all loaded bytes are valid and the intrinsic permits it; cache-line crossings may cost.

12. Full-chunk bound#

For width 32 and length n, state a safe loop condition using index i. Answer sketch: i <= n - 32 only after avoiding subtraction underflow, or n - i >= 32 with proven i<=n.

13. Scalar tail#

Process length 70 with 32-byte chunks. Answer sketch: Two full chunks cover 64 bytes; scalar code handles indices 64 through 69.

14. Overread fallacy#

Why is reading 32 bytes from a 20-byte slice invalid even if only 20 results are used? Answer sketch: The load itself accesses outside the allocation; later masking cannot legalize it.

15. Unsafe boundary#

Design a safe public API around an AVX2 kernel. Answer sketch: Safe wrapper detects AVX2, passes a valid slice, and calls a small #[target_feature] unsafe fn.

16. Two independent proofs#

List what feature detection proves and what slice invariants prove. Answer sketch: Detection proves instruction support; slice reasoning proves address validity, length, and initialization.

17. x86 byte equality#

Outline a 32-byte equality search on AVX2. Answer sketch: Broadcast needle, unaligned load, byte compare, movemask, trailing zeros, then scalar tail.

18. NEON first match#

NEON lacks an identical x86 movemask workflow; what is the porting lesson? Answer sketch: Preserve compare-and-first-match semantics, using a target-appropriate reduction or mask extraction.

19. Wasm feature model#

Why is compiling with SIMD128 not the same as runtime dispatch in a native x86 binary? Answer sketch: Wasm validation/runtime must support emitted operations; deployment often selects compatible modules/builds.

20. Autovectorization evidence#

How do you establish that a scalar loop vectorized? Answer sketch: Use compiler optimization remarks and release assembly; timing alone cannot identify code generation.

21. Bounds-check inhibition#

Rewrite parallel indexing of equal-length slices to help optimization. Answer sketch: Assert lengths once, then use iterators/chunks or a structure that exposes one checked range.

22. Crate evaluation#

A crate promises “portable SIMD.” What must you verify? Answer sketch: Rust stability, target coverage, fallback behavior, maintenance, safety, codegen, license, and benchmarks.

23. Horizontal maximum#

How should an integer max reduction combine vector and tail results? Answer sketch: Maintain vector maxima, reduce lanes to scalar, process tail, then combine under empty-input policy.

24. ASCII validation#

Give a vector condition for detecting non-ASCII bytes. Answer sketch: ASCII has high bit clear; test whether any lane intersects 0x80, respecting mask semantics.

25. Delimiter positions#

A chunk begins at index 96 and mask bits 2 and 9 are set. What positions are emitted? Answer sketch: 98 and 105, consuming bits in increasing order for stable output.

26. Shuffle cost#

Why might an arithmetic-light AoS kernel fail to speed up after vectorization? Answer sketch: Shuffles, gathers, and extra loads can exceed the saved arithmetic and pressure execution ports.

27. Benchmark crossover#

Scalar wins below 48 bytes and AVX2 wins above it. What policy follows? Answer sketch: Consider a measured size threshold, but include dispatch/branch costs and workload size distribution.

28. Frequency effects#

Why can wider x86 vectors disappoint in an otherwise compute-heavy benchmark? Answer sketch: Power/frequency behavior, port pressure, and throttling can change whole-workload throughput.

29. Differential generation#

Name high-value lengths for a 32-byte kernel. Answer sketch: 0,1,30,31,32,33,63,64,65 plus larger random lengths and offset variations.

30. Fault injection#

How can you validate that tests cover the tail? Answer sketch: Deliberately skip or corrupt cleanup and ensure boundary-focused tests fail.

31. Constant-time claim#

Does replacing branches with SIMD masks prove constant time? Answer sketch: No; inspect secret-dependent addresses, loop counts, instructions, compiler transformations, and platform behavior.

35. OSS benchmark claim#

Rewrite “AVX2 makes parsing 8× faster.” Answer sketch: “On CPU X, compiler Y, and corpus Z, median throughput rose from A to B for sizes S.”

Debugging playbook: symptom, evidence, action#

Wrong only near vector-width boundaries#

Collect: Failing length, chunk index, remainder, input offset, and first divergent output. Suspect: Off-by-one full-chunk condition, omitted tail, or duplicated tail processing. Act: Trace covered index intervals; add tests at W-1, W, W+1, 2W-1, 2W, 2W+1.

Wrong only for bytes above 127#

Collect: Hex inputs and signed/unsigned lane interpretation at each compare. Suspect: Signed comparison used for unsigned data or accidental sign extension. Act: Derive the intended ordering; use biasing or an unsigned operation appropriate to the target.

First-match index is reversed or shifted#

Collect: A single set lane at every position and the extracted scalar mask. Suspect: Lane-to-bit ordering, base-index arithmetic, or leading versus trailing zeros. Act: Build a lane-order truth table for that exact intrinsic and architecture.

Debug passes but release fails#

Collect: Minimal input, optimization flags, panic behavior, and sanitizer output. Suspect: Undefined behavior, uninitialized lanes, overflow assumptions, or optimizer-sensitive aliasing. Act: Audit every unsafe premise; do not “fix” it by disabling optimization.

Release passes locally but crashes on another x86 CPU#

Collect: CPU flags, selected dispatch branch, binary target features, and illegal-instruction location. Suspect: AVX2 executed without runtime support or feature-enabled code leaked into baseline path. Act: Re-establish the dispatch boundary and inspect baseline assembly for unsupported instructions.

Sanitizer reports an overread despite masked results#

Collect: Load address, allocation extent, width, and remaining byte count. Suspect: A full-width load was performed before masking invalid lanes. Act: Use scalar cleanup, valid masked-load semantics, or copy into an initialized padded temporary.

SIMD is correct but slower for tiny inputs#

Collect: Latency by length including dispatch, setup, and function-call overhead. Suspect: Fixed costs dominate the small amount of useful work. Act: Keep scalar for short inputs if the real size distribution supports a threshold.

SIMD loses only on large inputs#

Collect: Hardware counters if available, bandwidth, cache misses, frequency, and threads. Suspect: Memory bandwidth, downclocking, cache behavior, or contention rather than instruction count. Act: Benchmark the complete workload and reduce traffic before adding arithmetic width.

Benchmark results vary wildly#

Collect: Raw samples, CPU affinity, warmup, power mode, background load, and input generation timing. Suspect: Environmental noise, allocation in timed region, turbo transitions, or insufficient duration. Act: Stabilize setup, lengthen samples, separate data preparation, and report distributions.

Glossary#

Alignment: The divisibility of an address by a byte boundary; an API may permit unaligned access. Autovectorization: A compiler transformation that turns suitable scalar code into vector operations. Baseline target: The minimum instruction set a binary path may assume without extra checks. Bitmask: A scalar integer whose bits summarize boolean lane results. Broadcast: Copying one scalar value into every lane of a vector. Chunk: One vector-width portion of an input processed per loop iteration. Dispatch: Selecting an implementation based on target, runtime features, size, or policy. Gather: Loading lanes from multiple noncontiguous addresses. Horizontal operation: Combining values across lanes, such as reducing lanes to one sum. Intrinsic: A language-level function exposing an operation with target-specific semantics. Lane: One fixed-width element inside a vector value. Mask: Per-lane comparison information used for selection, filtering, or control. Masked load: A load whose architecture/API semantics suppress access for inactive lanes; verify carefully. Multiversioning: Shipping multiple implementations compiled for different feature sets. NEON: Arm's SIMD architecture; Advanced SIMD is baseline in standard AArch64 environments. Oracle: A trusted reference implementation used to judge optimized results. Pack/narrow: Converting wider lanes to narrower lanes, possibly with truncation or saturation. Reduction: Combining many input elements into fewer outputs, often one value. Reassociation: Changing grouping of operations, which can alter floating-point results. Scatter: Storing lanes to multiple noncontiguous addresses. Shuffle: Rearranging values among lanes according to a pattern. SIMD: Single instruction, multiple data: one operation applied across multiple lanes. SoA: Structure of arrays, storing each field contiguously in its own array. AoS: Array of structures, storing all fields of each record together. Tail: Elements remaining after all complete vector-width chunks are processed. Target feature: An instruction-set capability enabled at compile time or checked at runtime. Throughput: Work completed per unit time under sustained processing. Latency: Time required to complete one operation or request. std::arch / core::arch: Stable Rust architecture-specific intrinsic modules without extra dependencies. std::simd: Rust's portable SIMD API, still nightly in this pinned snapshot.

SIMD code review checklist#

Contract and semantics#

• [ ] Is the exact scalar behavior documented for empty, short, and malformed inputs?
• [ ] Are signedness, overflow, saturation, shifts, NaNs, and ordering explicit where relevant?
• [ ] Does first-match or stable-output behavior survive parallel processing?
• [ ] Is the scalar fallback also the maintained oracle rather than a stale alternate path?

Safety#

• [ ] Is the public API safe unless callers truly must uphold an unsafe contract?
• [ ] Does each unsafe block state address bounds, alignment, initialization, and aliasing premises?
• [ ] Are full-width loads and stores proven in bounds before they execute?
• [ ] Is tail handling valid for lengths smaller than one vector?
• [ ] Are feature checks separate from pointer and slice safety arguments?
• [ ] Can arithmetic in loop bounds or pointer offsets overflow?

Features and portability#

• [ ] Is baseline code free of instructions unavailable on baseline CPUs?
• [ ] Is each #[target_feature] function reached only when its feature is guaranteed?
• [ ] Are compile-time feature enablement and runtime detection used intentionally?
• [ ] Does every supported target retain a scalar implementation?
• [ ] Are AArch64, x86, and Wasm assumptions stated rather than implied?
• [ ] If portability relies on a crate or nightly feature, are that cost and version recorded?

Code generation and performance#

• [ ] Was optimized assembly inspected for each important target?
• [ ] Are loads, masks, shuffles, reductions, and tails recognizable in generated code?
• [ ] Do benchmarks include realistic sizes, distributions, alignments, and match densities?
• [ ] Is setup, allocation, dispatch, and input generation included or excluded deliberately?
• [ ] Are raw samples, variance, hardware, compiler, and flags available?
• [ ] Are claims scoped to measured workloads instead of asserted universally?

Testing and maintenance#

• [ ] Do differential tests force scalar and every available SIMD implementation?
• [ ] Are W-1, W, W+1, multiple widths, offsets, and empty inputs covered?
• [ ] Are known-answer tests independent of the shared oracle?
• [ ] Are randomized seeds reproducible and regressions preserved?
• [ ] Do CI configurations compile architecture-specific code even when they cannot execute it?
• [ ] Can maintainers disable or remove the SIMD path without changing public behavior?

Final capstone: production-grade byte analytics#

Build a library that scans a byte slice once and returns ASCII validity, first invalid index, delimiter count, and first occurrence of a caller-provided ASCII delimiter.

Required implementations#

1. A clear scalar implementation that remains the fallback and differential oracle.
2. An AVX2 implementation using stable std::arch/core::arch and a small unsafe kernel.
3. A NEON implementation for AArch64, or a Wasm SIMD128 variant when AArch64 execution is unavailable.
4. A safe public entry point with correct compile-time guards and runtime/platform dispatch.
5. No required SIMD dependency; explain optional crates if you use them for tooling only.

Required correctness evidence#

• Define exact behavior for empty input, non-ASCII delimiter, and first-index ties.
• Test every length 0..=3W+3 for each kernel's width and offsets 0..=W where practical.
• Include all-ASCII, all-invalid, dense delimiter, no delimiter, and boundary-position fixtures.
• Run deterministic randomized differential tests with saved failing seeds.
• Add independent known-answer examples so a shared scalar mistake cannot bless all paths.
• Keep the scalar path tested directly; dispatch success must not make fallback untested.

Required safety evidence#

• Put a # Safety explanation on each target-feature kernel.
• Prove every vector load in bounds and every tail byte initialized before reading.
• Show that runtime feature checks do not stand in for pointer validity.
• Run available sanitizers or equivalent dynamic checks and document platform limitations.

Required code-generation evidence#

• Save release assembly excerpts for the hot loop on every implemented SIMD target.
• Annotate vector loads, comparisons, mask extraction/reduction, loop branch, and tail transition.
• Record compiler version, target triple, target features, optimization level, and LTO setting.
• Explain surprising instructions honestly; do not count instructions as a performance proof.

Required benchmark evidence#

• Measure scalar, direct kernels, and dispatched API separately.
• Cover tiny, crossover, cache-resident, and large streaming sizes.
• Vary invalid-byte and delimiter position/density using realistic workload distributions.
• Report median and spread, raw throughput or latency, CPU/runtime identity, and sample method.
• State where SIMD loses and why the evidence supports only a scoped conclusion.

Required documentation#

• Provide a semantic contract, architecture support table, and fallback behavior.
• Explain why stable std::arch/core::arch is used and std::simd is nightly in this snapshot.
• Include safety invariants, dispatch flow, benchmark reproduction commands, and known limitations.
• Add a maintenance note describing how to update the oracle and all target implementations together.

Capstone scoring rubric — 100 points#

Semantic correctness — 20: Exact contract (5), scalar oracle (5), boundary behavior (5), stable first indices (5). Memory and feature safety — 20: Local unsafe proofs (8), bounded tails (5), correct feature gates (5), tooling evidence (2). Architecture implementations — 15: AVX2 quality (6), NEON or Wasm quality (6), semantic parity (3). Dispatch and fallback — 10: Safe API (3), runtime/platform selection (4), maintained scalar fallback (3). Testing — 12: Boundary matrix (4), randomized differential tests (4), known answers (2), forced paths (2). Assembly analysis — 8: Reproducible output (3), accurate annotation (3), codegen interpretation (2). Benchmarking — 10: Workload matrix (3), sound method (3), raw evidence (2), scoped claims (2). Documentation — 5: Reproduction, limitations, architecture table, and maintenance guidance.

Mastery threshold: 85 points with at least half credit in every category. Automatic revision required: Any unexplained unsafe load, unsupported instruction path, or unmaintained fallback.

Teaching the material#

Beginner talk: “SIMD without magic” — 20 minutes#

1. Show four scalar additions and one four-lane addition.
2. Explain lanes, masks, select, and why carries do not cross lanes.
3. Live-draw a byte-search compare and match mask.
4. Show scalar cleanup for a short tail.
5. Close with oracle-first testing and the principle that derivation beats memorization.

Intermediate talk: “From scalar contract to safe Rust SIMD” — 35 minutes#

1. Start from a tested scalar byte classifier.
2. Classify its map, compare, mask, reduction, and first-index patterns.
3. Introduce stable std::arch, a target-feature kernel, and safe dispatch wrapper.
4. Prove pointer bounds separately from AVX2 detection.
5. Compare AVX2 and NEON semantic translations.
6. Inspect release assembly and benchmark crossover sizes.
7. End with differential tests and maintaining the scalar fallback.

Expert talk: “Evidence-driven multiversioning” — 50 minutes#

1. Define semantic risks: floating reassociation, masks, narrowing, and stable ordering.
2. Compare x86, NEON, and Wasm feature/dispatch models.
3. Audit unsafe memory invariants and compiler target-feature boundaries.
4. Analyze shuffle-heavy and reduction-heavy codegen.
5. Diagnose performance with distributions, counters, frequency, and memory ceilings.
6. Discuss constant-time caveats and untrusted input.
7. Present an OSS integration strategy and deletion criteria for SIMD code.
8. Defend claims with reproducible assembly, tests, and benchmark artifacts.

Live demonstration menu#

Demo 1 — Predict, then execute#

Write four wrapping lane additions on screen, ask for results, then run assertions. Deliberately contrast saturation to expose semantic assumptions before intrinsics appear.

Demo 2 — One-bit-at-a-time byte search#

Use an input with one match in each possible lane. Print compare vectors and scalar masks, then derive the first index with trailing zeros.

Demo 3 — Tail bug caught live#

Temporarily change a loop bound or skip scalar cleanup. Run the boundary matrix, inspect the smallest failure, restore the proof-driven condition.

Adversarial audience and interview questions#

Q: Why not memorize the fastest intrinsic sequence and reuse it? A: Sequences encode target and data assumptions. Deriving from semantics exposes signedness, tails, ordering, and portability; assembly and benchmarks then validate the chosen mapping.

Q: If AVX2 is detected, why is the function still unsafe? A: Detection proves instruction availability only. The function may still require valid addresses, sufficient lengths, initialized bytes, alignment, and aliasing guarantees.

Q: Isn't std::simd the stable portable answer now? A: Not in this pinned snapshot: it remains nightly. Stable dependency-free architecture SIMD is available through std::arch or core::arch, with explicit target implementations.

Q: A test suite has a billion random cases. Is correctness established? A: It is useful evidence, not proof. Boundary structure, independent known answers, unsafe invariants, and semantic review catch classes uniform random data may miss.

Q: Why inspect assembly if benchmarks already improve? A: Assembly confirms which path and operations were generated, catches missing vectorization or feature leakage, and helps explain results; it does not replace workload measurements.

Q: How do you review an intrinsic you have never seen? A: Read its exact types and documented semantics, draw lane inputs/outputs, test boundary values, then map it to the algorithm and inspect generated code.

Resume claims: honest and useful#

Defensible bullet examples#

• Implemented scalar, AVX2, and AArch64 NEON byte scanners with runtime-safe dispatch and differential boundary tests.
• Improved median throughput from X to Y on documented corpus Z and CPU C; preserved scalar fallback for small and unsupported targets.
• Audited std::arch unsafe kernels, proving load bounds and target-feature preconditions; added sanitizer and randomized regression coverage.
• Diagnosed an autovectorization regression using compiler remarks and assembly, restoring vector codegen without changing public behavior.
• Added reproducible benchmarks across input sizes and distributions, identifying a measured SIMD crossover threshold.

Replace every placeholder with measured, reviewable evidence and state your personal contribution.

Claims to avoid#

• “Made the application 8× faster with SIMD” when only a microkernel improved.
• “Expert in AVX2/NEON” based on memorizing intrinsic names or completing tutorials.
• “Guaranteed constant time” because code is branchless.
• “Supports all CPUs” when only the development machine was tested.
• “Memory safe because Rust” around undocumented unsafe pointer operations.
• “Portable SIMD on stable Rust” when relying on nightly std::simd in this snapshot.
• “Zero overhead dispatch” without measuring detection, calls, and small inputs.
• “Fully tested” when SIMD only ran indirectly on one architecture.

Final mastery checklist#

• [ ] I can derive lane results for signed, unsigned, wrapping, saturating, and narrowing operations.
• [ ] I can explain masks as semantics first and target representations second.
• [ ] I can classify loops by map, zip, reduction, scan, lookup, filter, and recurrence patterns.
• [ ] I can choose AoS or SoA from access patterns and verify the effect in generated loads.
• [ ] I can prove full-chunk and tail bounds for empty and short slices.
• [ ] I keep unsafe architecture kernels small and document every premise.
• [ ] I never confuse CPU feature checks with pointer safety.
• [ ] I can build safe scalar/AVX2/NEON or Wasm dispatch without feature leakage.
• [ ] I know stable dependency-free Rust SIMD uses std::arch/core::arch.
• [ ] I state that portable std::simd is nightly in the pinned snapshot.
• [ ] I can inspect x86, NEON, and Wasm output by operation rather than mnemonic trivia.
• [ ] I can use optimization remarks to verify or diagnose autovectorization.
• [ ] I evaluate crates by stability, targets, fallback, safety, maintenance, license, and codegen.
• [ ] I can implement and test search, validation, reduction, delimiter, and shuffle-heavy kernels.
• [ ] I preserve first-index, stable-order, overflow, and floating-point semantics deliberately.
• [ ] I benchmark representative sizes, alignments, distributions, and target machines.
• [ ] I report raw evidence, variance, toolchain, hardware, and scoped conclusions.
• [ ] I test known answers, boundaries, randomized cases, forced paths, and injected failures.
• [ ] I consider overreads, timing leakage, untrusted lengths, and denial-of-service behavior.
• [ ] I can submit SIMD work as reviewable OSS commits with docs and reproducible evidence.
• [ ] I keep the scalar fallback and oracle maintained as the contract evolves.
• [ ] I can explain when scalar or compiler-vectorized code is the better engineering choice.
• [ ] I can defend an optimization under adversarial questioning without exaggerating.
• [ ] I can delete a SIMD path when evidence no longer justifies its cost.

Synthesis#

Real SIMD mastery joins semantics, memory proofs, architecture knowledge, and workload evidence. Start from a maintained scalar oracle, derive the parallel transformation, isolate unsafe operations, dispatch only where features are guaranteed, and verify results with tests, assembly, and benchmarks. The goal is not a catalog of intrinsics; it is repeatable judgment about correctness and value.

Part VII: The Art and Philosophy of SIMD — Designing Parallel Meaning#

Framing: 2026-08-04

Purpose: This part is not another catalogue of intrinsics. It develops the judgment needed to explain, design, criticize, and improve vector systems.

1. The central thesis: one description, many values#

SIMD means single instruction, multiple data. Its central achievement is not merely that several additions happen together. It is that one description of an addition can govern several values. A machine fetches, decodes, schedules, and retires less instruction description per useful data item.

Call this control compression or instruction compression. Control is the information that says what happens next. A scalar loop repeats much of that information: load, add, store, increment, compare, branch. A vector loop shares the description while retaining distinct values in its lanes.

scalar description                     vector description

add a0, b0 -> c0                        add [a0 a1 a2 a3],
add a1, b1 -> c1                            [b0 b1 b2 b3]
add a2, b2 -> c2                         -> [c0 c1 c2 c3]
add a3, b3 -> c3

four descriptions, four results         one description, four results

A lane is one position in a vector operation. Lane 2 in the diagram adds a2 and b2. The lanes share an operation, but not values. This distinction is the seed from which the rest of SIMD design grows.

The compressed description has conditions. The data items must admit the same operation at roughly the same time. Their memory representation must be reachable without spending more on rearrangement than the sharing saves. The program must preserve its promised meaning after grouping operations.

That is why “use all the lanes” is not the governing rule. The governing rule is “share a truthful description at acceptable cost.” Empty lanes, masked lanes, scalar cleanup, or no SIMD at all can follow from that rule.

Art here means cultivated engineering judgment: seeing real regularity,
choosing a representation that exposes it, preserving meaning through the
transformation, and balancing machine, software, and human constraints.

This map has five layers:

1. Meaning: What result, ordering, failures, and side effects are promised?
2. Representation: Which facts are adjacent, and which operations look alike?
3. Mechanism: What vector, mask, memory, and shuffle operations exist?
4. Proof: Why are feature, memory, and semantic preconditions satisfied?
5. Evidence: On which workloads and machines is the choice worthwhile?

Confusing layers produces familiar mistakes. A fast instruction does not prove that it implements the required meaning. A legal transformation does not prove profitability. An attractive representation does not make conversion free. A benchmark win does not expand its own scope.

Thought experiment — what if instruction handling were free? If fetch, decode, scheduling, register naming, and loop control consumed no space, time, or energy, control compression would lose part of its value. SIMD could still save register-file ports and express data parallelism, but manually repeated scalar operations would become much more competitive. SIMD exists in part because descriptions have physical cost.

2. Physical first principles: computation occupies a world#

A computer is finite matter operating with finite energy. Transistors switch; wires charge and discharge; signals take time to cross distance; stored bits must be read and moved. “One operation” in source code is therefore not one uniform physical event.

A transistor is an electronic switch used to build logic and storage. A wire is a physical conductor carrying a signal between structures. Moving a bit over a long on-chip wire can cost more energy and delay than a small local arithmetic operation. Moving it from a distant cache or main memory costs more again. Exact ratios vary by processor and fabrication technology, so the sound lesson is qualitative: placement and movement matter.

CPU designers seek reuse at several levels. Caches reuse fetched data. Pipelines reuse execution structures across cycles. SIMD reuses instruction work across values. A vector adder contains replicated or widened data paths, but the lanes can share decoding and control structures.

instruction bytes -> fetch -> decode -> shared vector control
                                            |  |  |  |
data registers --------------------------> L0 L1 L2 L3 -> result

Sharing is not free. A wider vector register needs storage and ports. Wider execution units occupy area. Cross-lane networks need wires. Loading more bytes raises demand on caches. On some processors, sustained use of particular wide instructions changes clock frequency or power behavior. Wider state can also increase context-management obligations.

A context switch occurs when an operating system stops one task and runs another, preserving architectural state that the first task must later recover. More architectural register state can make this contract more complex, even when implementations use techniques that avoid saving every byte every time.

Consequently, “twice the bits means twice the speed” is not a physical law. Consider a loop that reads 64 bytes and performs one comparison per byte. If memory delivers only 32 bytes in the relevant interval, a 64-byte operation cannot create missing bandwidth. If a 64-byte instruction causes extra down-clocking, it can lose to narrower instructions. If only five items remain, setup and masking may dominate.

possible limiting rates

front end:       descriptions per cycle
load/store:      bytes per cycle
arithmetic:      operations per cycle
cross-lane net:  shuffles per cycle
dependencies:    latency along a required chain
power/thermal:   sustainable activity

Thought experiment — what if memory were free? Suppose any number of bytes could arrive instantly at zero energy cost. Data layout would still influence meaning and lane mapping, but locality would cease to be a performance concern. Arithmetic throughput, dependencies, instruction handling, and rearrangement would dominate. Gather would become cheap only if address generation and result placement were free too. The experiment isolates memory cost; it does not erase all structure.

Counter-choice: A designer could build only narrow scalar units and rely on issuing many scalar instructions. That preserves flexible control but spends more instruction bandwidth and register bookkeeping. A designer could build only very wide units. That compresses regular work but wastes resources on small, irregular, latency-sensitive, or branch-heavy tasks. General CPUs contain multiple mechanisms because workloads disagree.

3. Five counterfactual machines#

Deriving SIMD by comparison is more revealing than treating it as inevitable. Begin with five models and ask what each shares and duplicates.

MIMD means multiple instruction, multiple data: different processing agents may execute different instruction streams on different data. CPU threads are a common example. Threads can perform the same loop, but sameness is then a software choice rather than the defining execution contract.

3.1 Scalar and manually repeated scalar#

A scalar machine says, “describe each operation separately.” It handles a linked-list walk naturally because the next address depends on the current node. It also handles four independent additions, but must encode and manage four additions.

Loop unrolling repeats a loop body several times while sharing loop overhead. It can expose instruction-level parallelism, meaning independent instructions that the processor may overlap. Yet each scalar add remains separately decoded.

// Conceptual four-way unrolling, not a claim about generated assembly.
out[i] = a[i].wrapping_add(b[i]);
out[i + 1] = a[i + 1].wrapping_add(b[i + 1]);
out[i + 2] = a[i + 2].wrapping_add(b[i + 2]);
out[i + 3] = a[i + 3].wrapping_add(b[i + 3]);

Unrolling and SIMD can cooperate: several vector instructions may be unrolled to hide latency or use multiple execution units. They answer different questions. Unrolling shares loop control; SIMD shares operation description.

3.2 SIMD#

SIMD says, “when operations are genuinely alike, name the operation once.” It usually retains one control flow for a vector group. Masks can qualify which lanes participate, but too much qualification reveals that the group was not very alike.

3.3 Threads and GPU execution#

Threads duplicate instruction-stream state, including program counters and register contexts. This costs more than adding lanes to one instruction, but it permits one thread to wait or branch while another follows a different path. Threads also introduce communication, synchronization, and scheduling issues.

GPUs typically execute large numbers of logical threads in groups whose members share instruction issue for periods of time. Terminology and exact grouping differ by vendor. The useful conceptual point is that GPU programming combines a thread-like model with SIMD- or SIMT-like shared execution. SIMT, single instruction multiple threads, presents threads while hardware often groups their execution. Divergent paths may be serialized under masks.

Thought experiment — replace every SIMD loop with one thread per element. Control becomes flexible, but creating, scheduling, and coordinating ordinary CPU threads per element is absurdly expensive. GPU hardware makes many logical threads cheaper, yet still depends on grouping and regularity. Granularity is a design decision, not an ideological preference.

The models coexist because sharing and independence pull in opposite directions. More sharing compresses regular work. More independent state tolerates irregular work. Architecture is the placement of boundaries between those goals.

4. Sameness is a design decision#

Two iterations are “the same” only relative to a chosen description. Adding 1 to every pixel is plainly alike. Clamping each pixel also becomes alike if a comparison produces per-lane choices. Parsing four unrelated programming languages is unlikely to admit an honest shared description.

Semantic independence means that performing one candidate operation does not improperly alter another operation's required result or observable effects. Independence is about the program contract, not visual similarity in source.

// Independent map: each output depends only on the corresponding input.
for i in 0..x.len() {
    y[i] = x[i].wrapping_mul(3).wrapping_add(1);
}

// Recurrence: each output depends on the preceding output.
for i in 1..x.len() {
    x[i] = x[i].wrapping_add(x[i - 1]);
}

The second loop repeats text, but its iterations are not independent. It is a prefix operation in disguise. A parallel scan may compute it, but not by naively placing consecutive iterations in lanes.

Regularity is a compressed description. “Apply f to indices 0 through 999” is shorter than listing a thousand unrelated actions. Good SIMD engineering finds such compression without deleting semantic distinctions.

Sameness stops being honest when lanes require materially different algorithms, when side-effect order matters, when an exception in one lane must prevent later effects, or when data movement needed to align the fiction exceeds useful work. A mask can express exceptions to regularity, but a mask full of exceptions is a diagnostic, not a triumph.

Consider case-folding ASCII letters. The lanes differ in byte value, yet all share: test whether the byte is in A..=Z, then conditionally add the ASCII offset. The conditional is itself regular.

input:       [ 'A'  '?'  'Q'  'z' ]
is uppercase [  T    F    T    F  ]
offset:      [ 32   32   32   32  ]
selected:    [ 'a'  '?'  'q'  'z' ]

By contrast, “decode an image, decompress an archive, resolve DNS, and multiply a matrix” shares only the word “compute.” Putting these jobs in four lanes would hide difference rather than compress sameness.

Adversarial question: If every lane executes the same opcode, is the work necessarily SIMD-friendly? No. Addresses may be scattered, masks may leave most lanes inactive, and cross-lane dependencies may dominate. Opcode sameness is necessary for a lane-wise instruction but insufficient for a good algorithm.

5. Lanes are coordinates, not tiny cores#

A vector register is an architectural container whose bits are interpreted as several lane values by an operation. Lanes are best understood as coordinates: lane 0, lane 1, and so on. They help state correspondence between inputs and outputs. They are not generally independent processors with private program counters, stacks, and arbitrary branches.

This is a useful temporary fiction:

vector register bits
+--------+--------+--------+--------+
| lane 0 | lane 1 | lane 2 | lane 3 |
+--------+--------+--------+--------+
       interpretation: four u32 values

Another instruction may interpret the same total bits as bytes, signed values, or floating-point values, subject to ISA rules. Meaning comes from type and operation, not painted borders inside silicon.

A vector value is not an array merely because both hold multiple values. An array is a memory and language-level aggregate with indexing and layout rules. A vector is an operation-oriented value whose supported indexing may be limited or expensive. Extracting an arbitrary lane can require movement to a scalar register; dynamically indexing lanes may lower to a store and reload on some targets. Treat vectors as vectors while computing and arrays as arrays while storing collections.

Vertical or lane-wise work preserves lane correspondence:

[1 2 3 4] + [10 20 30 40] = [11 22 33 44]

Horizontal or cross-lane work combines or rearranges lanes:

reduce_add [1 2 3 4] -> 10
rotate      [1 2 3 4] -> [2 3 4 1]

Cross-lane networks tend to be more constrained than lane-wise arithmetic. Their exact cost varies. A design with no cross-lane operations would be simple and good for pure maps, but reductions, transposes, scans, and table rearrangements would require stores or multiple instructions. A design with arbitrary cheap permutations would require substantial routing hardware and encoding space.

Thought experiment — give each lane its own program counter. Lanes could branch independently, but the register would begin to resemble a bundle of threads. Fetch, decode, scheduling, and exception state would multiply. The original control-compression benefit would shrink. This does not make the idea invalid; it places it in a different point in the design space.

6. Masks: control represented as data#

A mask is a set of Boolean lane conditions. A predicated operation uses a predicate, usually a mask, to decide which lanes participate. Masks turn a control question—“should this item execute?”—into data that can be computed, combined, and consumed.

Scalar branch trace:

value       3   -2    7   -9
branch      T    F    T    F
execution  add  skip add  skip

Masked vector trace:

values          [ 3  -2   7  -9 ]
values > 0      [ T   F   T   F ]
candidate + 10  [13   8  17   1 ]
merge old/new   [13  -2  17  -9 ]

Some machines provide predicated arithmetic that avoids committing inactive lanes. Others compute candidates and select. These can have the same ordinary result but differ for faults, exceptions, power, and cost. An API must not infer hardware behavior merely from a mask-shaped abstraction.

Branch divergence occurs when members of a shared-execution group require different control-flow paths. A common implementation executes one path for the lanes whose masks select it, then another path for the rest. If both paths are long, control sharing saves less.

all lanes active:       [0 1 2 3]
condition true path:    [0 . 2 .]
condition false path:   [. 1 . 3]
reconverged:            [0 1 2 3]

“Branchless” means source or generated code avoids a branch in some region. It does not automatically mean faster. Computing both candidates may double work. A predictable scalar branch can be cheap. Selection can add dependencies, and masked memory instructions may still have target-specific costs.

Branchless also does not automatically mean constant-time. Constant-time code aims to make observable resource use independent of secrets within a defined threat model. Data-dependent memory addresses, cache effects, variable instruction latency, compiler transformations, and speculative behavior may still leak information. Constant-time is a whole-proof claim, not a spelling.

Thought experiment — what if vectors had no masks? Full vectors of regular data would still work. Tails would need scalar cleanup, padding, or special loads. Conditionals would need bitwise select tricks where possible. Scalable vectors would become awkward because the active element count varies. Masks are not decorative; they let partial regularity remain in the shared description.

Thought experiment — what if every operation trapped per lane? A trap is a synchronous transfer to an exception handler. Precise per-lane traps would need rules for which lanes completed, trap order, restart, and side effects. The complexity would weaken the simple fiction of one operation. Many vector systems therefore restrict, aggregate, suppress, or otherwise define exceptional behavior carefully. API designers must state behavior rather than assume scalar exceptions copied lane by lane.

7. Data layout is ontology and design language#

An ontology is a choice about what kinds of things a representation treats as primary and how they relate. Data layout acts as ontology because it decides which facts are neighbors. The machine cannot load “all x coordinates” cheaply unless the representation makes them adjacent or provides a tolerable gather.

An array of structures, AoS, stores complete records together:

AoS particles:
[x0 y0 z0 mass0][x1 y1 z1 mass1][x2 y2 z2 mass2]

A structure of arrays, SoA, stores each field together:

SoA particles:
x:    [x0 x1 x2 ...]
y:    [y0 y1 y2 ...]
z:    [z0 z1 z2 ...]
mass: [m0 m1 m2 ...]

AoS says a particle is the adjacent unit. That is excellent when code consumes all fields of one particle. SoA says a coordinate field across particles is the adjacent unit. That is excellent for lane-wise updates of many x coordinates. Neither is universally “SIMD layout.” Each embodies an access hypothesis.

Locality is the tendency to access data near recently accessed data in time or address. Think of it as geometry: cache lines bring neighboring bytes, pages group addresses for translation, and vector loads cover contiguous ranges. Representation determines whether logical neighbors align with this physical geometry.

Converting AoS to SoA costs reads, writes, storage, and latency. A transient conversion may pay off for a long compute phase and lose for one small pass. An array-of-structures-of-arrays layout groups a modest block in SoA form, then repeats blocks. This can balance vector access, cache locality, and record-level management, but complicates indexing and APIs.

external AoS -> pack/transpose -> vector-friendly blocks -> compute
      ^                                                |
      +--------------- unpack if required <------------+

Representation can preserve or destroy meaning. Reordering transactions for throughput is invalid if order is observable. Splitting fields may complicate atomic updates that must appear as one record change. Padding can make bounded loads safe only when allocation and initialization contracts truly provide it.

Thought experiment — make all records SoA. Field-wise analytics improve, but object-oriented traversal, insertion, ownership, and whole-record transfer may worsen. The counterfactual shows that layout belongs at an abstraction boundary informed by all important consumers, not inside one benchmark.

A sound layout argument answers:

• Which fields does each hot operation consume?
• In what order and batch size are records visited?
• Who owns conversion, and how often is it paid?
• Does the new layout alter synchronization or serialization contracts?
• Can scalar and non-hot paths remain understandable?

8. Tails and boundaries test philosophical correctness#

Finite data rarely divides perfectly by a machine width. The remainder after full vector groups is the tail. Tail handling exposes whether an API treats lane width as essence, implementation detail, or recoverable optimization.

For 10 elements and four lanes:

full group 0: indices [0 1 2 3]
full group 1: indices [4 5 6 7]
tail:         indices [8 9]
invalid:      indices [10 11]

Common strategies include scalar cleanup, a masked load/store, copying into a temporary padded vector, or maintaining real padding in the allocation. Each strategy has a proof and cost.

Scalar cleanup is simple and often excellent. Masked access expresses the tail in vector form, but hardware semantics and API guarantees must ensure inactive lanes do not access invalid addresses. Temporary copying is portable but adds movement. Permanent padding can simplify kernels but changes allocation, initialization, and ownership contracts.

No-overread rule: never justify an out-of-bounds read by saying the extra
bytes are “probably mapped,” “inside a cache line,” or “masked away later.”

An overread is a memory-safety violation when the language contract does not permit the access. It can cross a page boundary, encounter inaccessible memory, be detected by tooling, expose adjacent secrets, or let the optimizer reason from undefined behavior. Ignoring unwanted lane results does not undo the read.

// Conceptual safe structure: full chunks plus an explicit remainder.
let mut chunks = input.chunks_exact(LANES);
for full in &mut chunks {
    process_full(full); // full.len() == LANES
}
for value in chunks.remainder() {
    process_scalar(*value);
}

A scalable-vector loop often uses a predicate for “indices below length.” A fixed-width loop may do the same if the target provides suitable masked memory operations. Neither removes the need to prove bounds. Predication changes the shape of the proof: only active lanes may access memory, and the instruction's documented semantics must support that statement.

Thought experiment — declare that all slices are padded to 64 bytes. Some kernels simplify, but every producer, allocator, foreign-function boundary, and subslice must preserve padding and initialization. Publicly imposing this contract may cost more than local tail handling. Boundaries reveal who bears optimization complexity.

9. Reductions and scans: parallel shape versus sequential meaning#

A reduction combines many values into one, such as a sum or maximum. A scan produces every prefix result, such as running sums. Both challenge the simple independent-lane picture because values must interact.

An operation is associative if regrouping does not change the result: (a op b) op c == a op (b op c). An identity is a value e such that e op x == x and x op e == x. Integer addition modulo a fixed width is associative with zero as identity. Ordinary mathematical integer addition is also associative, but a finite-language overflow contract may differ.

sequential fold: (((a + b) + c) + d)

balanced tree:       (+)
                    /   \
                  (+)   (+)
                 /  \   /  \
                a    b c    d

A tree shortens the dependency depth, which is why reductions can benefit from parallel grouping. It also changes operation order. For exact associative operations under the chosen contract, this is harmless. For floating-point addition, it can change rounding.

Floating-point numbers approximate real values with finite precision. Addition is generally not associative because each intermediate result may round. NaNs, signed zero, infinities, and target modes further complicate min, max, and sum semantics. “The mathematical sum” is not a complete programming contract.

f32 conceptual example:
(large + -large) + small  -> 0 + small
large + (-large + small)  -> large + approximately -large

The two evaluation trees can differ after rounding.

A deterministic contract may specify a fixed order or fixed tree independent of target width. That supports reproducibility but limits transformations. An approximate contract may permit reassociation within error bounds, enabling faster trees but requiring numerical analysis. A third contract may promise only target-local determinism, which is easier but must be stated honestly.

Scans need communication in stages. A four-lane inclusive sum can shift and add by distances 1 and 2, then carry the preceding vector's total into the next group. This is parallel, but not lane-independent.

start:             [a       b       c       d]
add shift by 1:    [a      a+b     b+c     c+d]
add shift by 2:    [a      a+b   a+b+c  a+b+c+d]

The diagram assumes an identity fills shifted positions and exact associativity for equality with a sequential specification. Different scan algorithms expose different rounding and overflow behavior.

Thought experiment — require bit-identical floating sums across every width. Then a width-dependent tree is unacceptable. The implementation could preserve scalar order, use a specified width-independent tree, or adopt an exact/binned accumulator with extra cost. Reproducibility is purchasable, not free.

10. Width: useful fact, dangerous identity#

Classic mainstream SIMD grew through fixed architectural widths. Fixed width fits register encodings, pipelines, ABI rules, and incremental ISA evolution. Programmers naturally exposed types such as “four f32 lanes” because hardware did. This history creates design pressure: code, calling conventions, and data formats begin to assume a width.

Width leaks through unrolling, shuffle constants, alignment expectations, tail frequency, public type names, serialized layouts, and reduction order. Once a public API accepts a particular vector type, callers may compile that width into their own structures. Replacing it later becomes an API migration rather than a local optimization.

Wider can hurt through underused lanes, higher setup cost, extra register pressure, more expensive shuffles, frequency effects on some processors, or greater memory demand. Register pressure is competition for a finite set of registers; excess live values may be spilled to memory. The relevant comparison is complete code on a target, not nominal lane count.

Arm SVE and the RISC-V Vector extension embody vector-length-agnostic or scalable-vector programming. The same compiled loop can operate with a vector length selected by the implementation within architectural rules. Software asks which lanes correspond to remaining elements, operates under predicates, and advances by the amount handled. It need not encode one concrete lane count into the algorithm.

i = 0
while i < n:
    active = lanes_for_indices_below(i, n)
    x = predicated_load(input + i, active)
    y = predicated_compute(x, active)
    predicated_store(output + i, y, active)
    i += runtime_vector_length

This repeated processing of chunks is strip-mining: transform one large iteration space into strips sized for the vector mechanism. Predication lets the final strip be partial. SVE and RISC-V V differ in architecture and details; the conceptual similarity does not make their instructions or policies interchangeable.

The pseudocode is shaped like SVE's predicate-driven style. RISC-V V typically asks for a current element count, receives the usable vl, and advances by that returned count; its final strip is commonly represented by a shorter active vector length rather than the same predicate idiom.

Scalable vectors reduce width assumptions but do not eliminate cost variation, cross-lane complexity, memory limits, or semantic questions. Algorithms needing a fixed permutation may be easier on fixed vectors. Small fixed-size media data may map naturally to a fixed register. ABI and compiler support also matter.

Thought experiment — what if width were part of every public type? Code would express layouts and exact lane operations clearly, but libraries would multiply APIs for widths, users would face conversions, and scalable targets would fit poorly. Width-specific types are valid at architecture boundaries; making them universal would confuse mechanism with domain meaning.

No model universally wins. Fixed width offers explicit control and stable-sized values. Scalable width offers longevity across implementations and elegant predicated strip-mining. Judgment asks which assumptions belong in this layer.

11. Gather, scatter, and the confession of irregularity#

A gather loads lanes from multiple addresses. A scatter stores lanes to multiple addresses. They let an instruction express irregular memory access. They do not make those addresses physically adjacent.

indices: [ 0  9  2  31 ]
base:    [................................]
gather:    ^        ^ ^                            ^

If addresses hit different cache lines or pages, hardware must perform several memory transactions and translations. Duplicate scatter addresses require a defined conflict policy or a programmer proof that conflicts do not occur. Fault handling and masking are also architectural concerns.

Gather is valuable when arithmetic after each load is substantial, indices have some locality, or restructuring data is impossible or more expensive. It can also be a warning that the algorithm's logical grouping disagrees with memory geometry. An ISA feature can admit a mismatch without repealing it.

Compare three choices for a repeated indexed computation:

1. Gather directly on every pass: no conversion, recurring irregular accesses.
2. Sort or bucket indices: pay organization cost, improve locality, perhaps

alter required output order and need an inverse permutation.

1. Pack selected data contiguously: pay copy cost, then run regular kernels.

The right choice depends on reuse, dataset size, mutation, and ordering. “The ISA has gather” answers only whether expression is possible.

Thought experiment — make gather one instruction with advertised latency 1. If the data miss in cache, the memory system still waits. An implementation might initiate requests efficiently, but an instruction name cannot compress physical distance. Cost tables must state conditions.

12. The ISA is a language and a treaty#

An instruction set architecture, or ISA, is the programmer-visible contract of instructions, registers, memory effects, exceptions, and related state. It is a language because software uses it to describe computation. It is a treaty because hardware, compilers, operating systems, debuggers, ABIs, and programmers must agree on its meaning over long periods.

An ABI, application binary interface, defines binary-level conventions such as argument passing, register preservation, stack layout, and object formats. Adding registers or vector types affects more than an arithmetic unit. The OS must preserve required state across context switches and signals. Debuggers must display it. Compilers must allocate it. Calling conventions must decide whether and how it crosses function boundaries.

Instructions exist when a semantic pattern is common enough and hardware can implement it advantageously under the treaty's constraints. Some instructions expose fundamental operations; others fuse sequences, provide a permutation, or accelerate a domain such as cryptography. Encoding space, verification, decode complexity, and future compatibility constrain additions.

An immediate operand is a constant encoded in an instruction rather than read from a general register. Shuffle controls and rounding modes often use immediates because constants permit compact or simpler hardware selection. The tradeoff is reduced runtime flexibility; dynamic control may require another instruction or sequence.

Feature levels let implementations support extensions incrementally. Software must either target a guaranteed baseline, compile separate variants and dispatch, or require a minimum feature contract. Executing an unsupported instruction is not a performance loss; it is typically illegal execution.

Instruction cost varies across microarchitectures implementing the same ISA. One processor may execute a shuffle as one fast operation; another may split it internally or provide fewer execution resources. Architectural equivalence is semantic, not cost equivalence.

12.1 Judging a proposed instruction#

Ask in order:

1. What exact semantic pattern does it name, including edge cases?
2. Is the pattern frequent across durable workloads or narrowly benchmarked?
3. Can existing instructions express it, and at what measured cost?
4. Can compilers recognize the pattern reliably from higher-level code?
5. What register, mask, exception, and memory state does it affect?
6. Does it compose with predication, widths, and existing types?
7. What encoding, hardware area, verification, OS, ABI, and tool costs follow?
8. How will old binaries and new binaries behave on old and new machines?
9. Could a more general primitive solve this and nearby problems better?
10. What evidence would cause rejection?

Suppose someone proposes VASCII_LOWERCASE. It could accelerate a common task, but its semantics must define non-ASCII bytes, locale, mask behavior, and width. Existing compare/add/select operations already compose the operation. A fused instruction may save issue bandwidth, yet occupy encoding and hardware for one policy. A more general range-check primitive might benefit parsing broadly.

Counterfactual — an ISA with one instruction per library function. Programs would look concise until policies changed. Hardware would contain or emulate rare operations, encoding would become unwieldy, and compatibility would freeze accidental semantics. Good ISAs choose reusable atoms and selected fusions, not either extreme of one universal opcode or every imaginable function.

13. Compiler philosophy: proof before enthusiasm#

A compiler can transform scalar source into vector operations. It commonly passes through an intermediate representation, or IR, a program form used for analysis and transformation. A target-independent vector IR expresses lane operations without committing immediately to one target instruction. Target lowering later selects instructions or sequences for a specific ISA.

Two gates govern vectorization:

• Legality: Does the transformed program preserve required behavior?
• Profitability: Is the transformed program expected to cost less?

Legality must precede profitability. A spectacularly fast wrong program is not an optimization.

Aliasing means two references or pointers may designate overlapping memory. If a store through one pointer changes a later load through another, grouping iterations can alter meaning. A dependence is an ordering relation where one operation needs a value or effect from another. Compilers use types, language rules, bounds, runtime checks, and analysis to prove safe independence.

source loop
   |
   v
canonical loop IR -> alias/dependence proof -> vector IR + scalar remainder
                                                |
                                                v
                                      target cost model/lowering
                                                |
                                                v
                                       machine instructions

A cost model estimates instruction, memory, setup, remainder, and sometimes code-size costs. It is necessarily imperfect because runtime sizes, alignment, data distributions, cache state, and future processors are not fully known. Compiler conservatism may be wisdom rather than weakness.

Loop vectorization groups iterations of a loop. Superword-level parallelism, or SLP, groups similar independent scalar operations already visible together, often within one basic block. Unrolled code can feed SLP; straight-line image or matrix expressions are common candidates.

Autovectorization can fail because dependence is unproven, control is complex, the trip count seems too small, calls have unknown effects, a reduction contract forbids reassociation, or the cost model predicts loss. A programmer can improve structure, provide safe no-alias information through language types, or select an explicit vector API. Forcing vectorization without addressing legality merely hides a question.

Intrinsics are functions closely corresponding to architecture operations. They provide power: precise instructions, types, and feature-specific behavior. They are also an abstraction leak, where lower-level mechanism becomes visible above its natural boundary. Width, instruction quirks, immediates, and feature checks enter source and constrain portability.

Thought experiment — a perfect cost model. Even perfect knowledge of one target cannot know an unknown future workload unless runtime facts are supplied. Dynamic versioning could test size or alignment, but tests cost time and code space. Profitability is a decision under incomplete knowledge.

14. API philosophy: portable meaning, variable cost#

SIMD APIs occupy several layers. Rust architecture intrinsics in std::arch or core::arch expose target-specific mechanisms on stable Rust where supported. In this textbook's pinned 2026 framing, std::simd portable SIMD remains nightly, not stable. Crates may offer portable vectors, target wrappers, dispatch, or domain-specific algorithms. Safe wrappers can contain unsafe operations behind checked contracts.

“Portable” can promise that source compiles on supported targets and operations have consistent documented semantics. It cannot promise identical instructions, lane widths, floating-point environment, throughput, or speedup. Distinguish semantic portability, preserving meaning, from cost portability, having similar performance characteristics. The first is achievable with careful contracts; the second is rarely absolute.

Architecture vector types are often poor public API currency. Exposing them can force callers to enable features, share an ABI assumption, and adopt a width. Keeping them inside a module lets the boundary speak in slices, records, or domain concepts while implementations evolve.

public domain API: normalize_points(&mut [Point])
                    |
safe validation and dispatch
                    |
        +-----------+-----------+
        |                       |
 scalar implementation     target SIMD kernel
        |                       |
        +------ same contract --+

Public vectors are appropriate when vector shape is genuinely part of the domain, as in a fixed four-component transform, or when an explicitly low-level API serves architecture-aware callers. Even then, distinguish a mathematical four-vector from a hardware register type; their operations and ABI needs may differ.

A good abstraction boundary contains volatility. Feature detection, tail strategy, and instruction selection change with targets. Input validity, overflow policy, NaN policy, and output ordering belong to the stable semantic contract. Escape hatches can expose lower layers without making every user pay their complexity.

Thought experiment — one portable vector API promises one-cycle operations. The promise fails as soon as targets implement operations differently or lower one operation to a sequence. A portable API can document complexity tendencies and expose capabilities, but fixed cycle claims belong to measured target contexts.

15. Safety is composition of three proofs#

SIMD safety is not one Boolean property. It composes at least three proofs.

1. Feature safety: the executing processor and context support the issued

instructions.

1. Memory safety: every active access obeys validity, bounds, alignment,

initialization, lifetime, and aliasing requirements.

1. Semantic correctness: outputs, ordering, overflow, floating behavior,

and side effects satisfy the public contract.

Rust's unsafe marks operations whose requirements the compiler cannot fully verify. It should be a focused proof boundary, not a performance decoration. The unsafe block or function should state premises close to each operation.

// Conceptual structure; target names and vector details are intentionally hidden.
pub fn transform(input: &[u32], output: &mut [u32]) {
    assert_eq!(input.len(), output.len());

    if feature_available() && input.len() >= SIMD_THRESHOLD {
        // SAFETY: dispatch proved the feature; the kernel receives equal valid
        // slices and handles only full chunks before a bounded remainder.
        unsafe { transform_simd(input, output) }
    } else {
        transform_scalar(input, output)
    }
}

Feature detection does not prove pointer validity. Bounds checks do not prove instruction availability. Matching test outputs do not prove absence of an overread. Keeping the proofs separate prevents one piece of evidence from being asked to carry another's burden.

The scalar fallback is honesty. It says the operation's meaning is broader than one mechanism. It supports unsupported processors, small inputs, unusual build targets, testing, and future removal of an unprofitable path. It is resilience, not an admission of failure.

Fallback code must remain maintained as an oracle, not rot behind dispatch. Tests should force each path. If the SIMD implementation needs a weaker semantic contract than the scalar implementation, either fix it or expose the difference explicitly; silent weakening is not optimization.

Thought experiment — declare all SIMD functions unsafe and stop documenting them. Callers still need the missing premises, but now every caller must infer them. Unsafety spreads without increasing capability. A narrow unsafe core is valuable because proof obligations become reviewable and reusable.

16. Measurement is epistemology#

Epistemology studies what counts as knowledge and how knowledge is justified. Performance work needs an epistemology because machines are complicated and claims are easy to overgeneralize.

A benchmark establishes a scoped observation: with a stated implementation, compiler, flags, machine, operating conditions, input sizes, distributions, and method, measured outcomes had a distribution. It does not prove universal speed.

claim ladder

“SIMD is faster.”                         unsupported universal claim
“This benchmark was 1.8x faster.”         missing scope
“Kernel K had median 1.8x lower time on
 CPU C for corpus D at sizes 4 KiB–1 MiB,
 with toolchain T and stated variance.”   useful scoped knowledge

Assembly is observation of compiler output. It answers whether vector operations, loads, branches, spills, and dispatch appeared. It does not directly reveal runtime cache misses or establish speed. Compiler remarks explain transformation decisions but may omit runtime effects.

Hardware performance counters count selected events or estimates, such as cycles, instructions, cache misses, and branch misses. They are clues constrained by event definitions, multiplexing, skid, privilege, and processor behavior. Counters support causal hypotheses; they do not automatically prove them.

# Representative workflow; choose tools available on the target system.
cargo build --release
perf stat -r 10 ./target/release/benchmark --size 65536

One machine cannot establish universal truth because ISAs and microarchitectures vary. Even machines with the same ISA may have different execution resources, caches, frequencies, and instruction costs. Diverse machines are not noise to average away; they are part of the question.

Beauty can guide hypotheses. A symmetric loop with few moves is aesthetically promising. Evidence decides whether it wins. Conversely, ugly assembly can be fast for the workload. Engineering art disciplines taste with observation.

Thought experiment — benchmark only huge aligned arrays. The result may describe throughput after warm-up while hiding setup, dispatch, tails, cache fit, misalignment, and common small inputs. A benchmark suite is an argument; its case selection determines what the argument can support.

17. The aesthetics of good SIMD#

Good SIMD code has visible invariants. An invariant is a fact maintained at a defined point, such as “i is within bounds and all indices below i are processed.” A reviewer can connect each load to a bound, each mask to meaning, and each instruction to the scalar contract.

Its aesthetic qualities are practical:

• A simple scalar oracle defines behavior independently of SIMD.
• Regularity is visible in data flow rather than hidden in clever macros.
• The unsafe core is narrow and its premises are local.
• Data movement is minimized, especially avoidable shuffles and conversions.
• Tail and fallback behavior are honest, bounded, and tested.
• Dispatch is explainable and feature requirements do not leak accidentally.
• The optimization is deleteable without redesigning the public API.

Deleteability is a strong test. Hardware and compilers change. If removing a specialized kernel requires breaking callers, mechanism escaped its boundary.

Clever-but-fragile SIMD often reuses one vector as several undocumented types, overreads because “the allocation is probably larger,” depends on one compiler's shuffle recognition, mixes feature detection with pointer arithmetic, and has no scalar oracle. Its density can look expert while making proof expensive.

fragile pipeline:
domain input -> exposed target type -> magic shuffle constants -> assumed ISA
             -> width-dependent result order -> benchmark on one machine

durable pipeline:
domain contract -> scalar oracle -> bounded internal vectors -> checked dispatch
                -> differential tests -> scoped measurements

The aim is not minimum lines. It is minimum accidental complexity for the required result. A few extra scalar tail lines can be more beautiful than a masked trick whose safety depends on undocumented lowering.

18. Design workshop: a tiny conceptual vector machine#

We will design rather than reveal a finished specification. Call the machine ClearVec. Its purpose is to run maps, filters, and modest reductions while remaining teachable. Every inclusion spends encoding, hardware, compiler, and proof budget.

18.1 First decision: fixed or scalable width#

A fixed 128-bit model is simple: four u32 lanes, known shuffle patterns, and ordinary sized registers. It also bakes width into software. A scalable model lets one binary use future implementations, but requires predicates and makes some exact permutations harder to state.

Choose scalable registers with an implementation-defined byte length that is a positive multiple of 16. Programs cannot observe register bits as a fixed-size memory object; they can query the number of lanes for a selected element type. This choice prioritizes strip-mined loops over fixed-format media operations.

Consequence: the ABI will not pass scalable vector values through ordinary public aggregate layouts. They remain inside functions or use a dedicated calling convention. If ClearVec were a tiny embedded machine with permanently fixed hardware, 128-bit fixed width might be the better treaty.

18.2 Lane types#

Include unsigned and signed integers of 8, 16, 32, and 64 bits, plus f32 and f64. Exclude decimal and arbitrary-width lanes. The chosen set covers common systems data and permits one physical register length to hold different lane counts.

Operations interpret lanes by opcode. Reinterpretation preserves bits; numeric conversion changes values and receives explicit widening, narrowing, rounding, and saturation forms. This prevents a cast-like instruction from hiding policy.

18.3 Loads, stores, and predicates#

Add contiguous predicated loads and stores. Every instruction receives a mask and a policy for inactive destination lanes: merge with an old destination or replace with zero. Masked-off memory lanes perform no access. This strong rule supports safe tails but increases architectural and verification obligations.

Do not initially include gather or scatter. ClearVec targets contiguous data; excluding them keeps memory exceptions and duplicate-address policy small. The consequence is poor expression of sparse indexed algorithms. Version 2 may add gather after workloads justify it.

VL32             -> runtime count of 32-bit lanes
WHILE_LT p0,i,n  -> p0 lane k is true when i+k < n
LD32.Z v0,p0,[x] -> active lanes load; inactive lanes become zero
ST32   p0,[y],v0 -> active lanes store; inactive lanes do not access

18.4 Arithmetic and comparisons#

Include wrapping integer add, subtract, multiply, shifts, and bitwise operations. Add explicit saturating add/subtract for narrow integers because media and signal work uses them often. Do not make trapping overflow the default: precise per-lane restart would complicate execution. A compare can detect overflow when the program needs checked arithmetic.

Floating operations follow a documented IEEE 754 profile with explicit modes where feasible. We do not promise that reassociation preserves results. Comparisons produce mask registers, not all-ones integer vectors. Distinct mask types prevent accidental arithmetic on truth values and permit compact hardware.

18.5 Selection, shuffle, and cross-lane work#

Add SELECT mask, true_value, false_value; it makes conditional data flow direct. Add neighboring slide up/down and table permutation within bounded groups of 16 bytes. Why not arbitrary whole-register permutation? Scalable length could make the routing network and index semantics costly. Bounded groups support parsers and transposes while exposing the limit honestly.

This exclusion hurts scans spanning groups. Slides plus scalar carry can still implement them, but not optimally. The design records this as a known pressure, not a fact to conceal in a library.

18.6 Reductions and exceptions#

Include integer bitwise reductions and wrapping sums. Define their grouping as implementation-dependent but their exact modular result as stable, since those operations are associative under the contract. Include floating reductions only with an explicitly relaxed-order opcode. A strict-order floating sum lowers to a sequence preserving its specified order.

Memory faults are precise at the instruction boundary: inactive lanes cannot fault; if any active lane faults, no store from that instruction commits. Loads either complete or trap without an architecturally visible partial destination. Arithmetic does not trap per lane. Floating status flags are aggregated across active lanes.

18.7 The provisional ClearVec core#

The final specification is less important than the trail of consequences. A real proposal would add binary encodings, formal pseudocode, memory ordering, privilege interactions, ABI work, compiler IR mappings, emulator tests, hardware cost studies, and compatibility plans.

Adversarial revision: A cryptography team needs constant-pattern byte table lookups larger than 16 bytes. Expanding arbitrary permutation may help but could create timing differences across implementations. We should define semantic behavior separately from constant-time claims, prototype bounded multi-table operations, and test whether a composable primitive serves parsers too.

19. Compiler workshop: recognizing three shapes#

Use one map, one reduction, and one masked conditional. The point is the chain from source meaning to target instructions, not syntax trivia.

19.1 Map#

for i in 0..n {
    out[i] = a[i].wrapping_mul(3).wrapping_add(b[i]);
}

Recognition finds a canonical induction variable i, contiguous accesses, and the same expression each iteration. Legality asks whether out overlaps a or b in a way that changes future loads, whether bounds are valid, and whether wrapping arithmetic matches vector operations. A runtime alias check may select a vector path only when ranges do not overlap.

Target-independent IR might express vector.load, vector.splat(3), vector.mul.wrap, vector.add.wrap, and vector.store. Fixed-width lowering chooses a profitable width and creates a remainder. ClearVec lowering creates a while-less-than predicate and advances by runtime vector length.

Profitability asks trip count, memory cost, target multiply throughput, alignment, runtime-check overhead, register pressure, and code size. Verification compares against scalar results for boundaries and inspects optimization remarks and assembly to confirm the intended path.

19.2 Reduction#

let mut sum = 0u32;
for &x in input {
    sum = sum.wrapping_add(x);
}

Wrapping u32 addition is associative modulo 2^32, so vector partial sums and a final reduction preserve the result. The compiler may use several vector accumulators to break dependency chains. It needs an identity of zero for inactive tail lanes.

For f32 with strict source-order semantics, the same reassociation may be illegal. A language option or explicit API can grant a relaxed contract, but a cost model cannot grant semantic permission. Target-independent IR must carry reassociation flags rather than losing the distinction.

19.3 Masked conditional#

for i in 0..n {
    out[i] = if x[i] >= 0 { x[i].wrapping_mul(2) } else { 0 };
}

The compiler can compare a vector to zero, compute a candidate multiply, and select. For this exact wrapping example, computing both candidates is valid: the unused multiplication cannot panic or create a side effect. Legality changes if either branch calls a function, traps, accesses memory, or has observable side effects. Speculatively computing both sides is not generally valid. Predicated target IR can preserve conditional execution when the ISA supports it.

Cost asks branch predictability, inactive-lane proportion, multiply cost, and whether masked execution avoids work. For mostly uniform data, versioning or a branch may win. The optimizer has estimates, not prophecy.

19.4 Tail lowering and verification#

Fixed-width lowering can generate a vector loop and scalar epilogue, or masked final iteration if safe and profitable. Scalable lowering naturally uses a predicate each strip. The compiler must ensure masked memory semantics match the target; emulating a masked load by full load plus select would be invalid at an unmapped boundary.

Verification layers include IR checks for operation flags, differential tests for lengths around widths, alias tests, sanitizer runs, target assembly review, and benchmarks. Compiler correctness testing should also mutate passes and use adversarial cases such as zero length, one active lane, NaNs, overflow, and page boundaries.

20. API workshop: a small Rust vector abstraction#

Call the conceptual library PlainVec. We design contracts before methods. It is not copy-and-paste production code and does not claim stable std::simd.

20.1 Lane count and representation#

Use a const generic fixed lane count for sized values:

pub struct VecN<T, const N: usize> {
    lanes: [T; N], // conceptual representation, not a codegen promise
}

pub struct Mask<const N: usize> {
    lanes: [bool; N],
}

The semantic type contains exactly N lanes. The implementation may use one hardware vector, several vectors, or scalar code. This promises semantic portability, not one instruction. Reject N == 0 to simplify identities and lowering. Large N remains legal only if the library is willing to lower it; otherwise supported sizes must be explicit.

A scalable companion would need a different, likely unsized or scope-bound API. Pretending const N models runtime vector length would leak contradictions.

20.2 Masks and selection#

Masks have Boolean semantics and do not expose a bit encoding. Provide &, |, !, any, all, and select. Converting to a bitset is explicit and defines lane 0's bit position. This keeps target all-ones conventions internal.

select(mask, yes, no) promises lane-wise choice. It does not promise lazy evaluation because yes and no are already values. For potentially failing computations, offer a predicated operation with documented inactive-lane behavior or keep control scalar.

20.3 Loads and tails#

Provide a total checked load that returns absence when too short, plus a partial load with fill and mask:

fn load_exact<T: Copy, const N: usize>(s: &[T]) -> Option<VecN<T, N>>;

fn load_partial<T: Copy, const N: usize>(
    s: &[T],
    fill: T,
) -> (VecN<T, N>, Mask<N>);

fn store_partial<T: Copy, const N: usize>(
    v: VecN<T, N>,
    mask: Mask<N>,
    out: &mut [T],
) -> Result<(), LengthError>;

load_partial reads only min(N, s.len()) elements and fills inactive lanes. The returned mask records validity. store_partial rejects a mask selecting an out-of-range lane. An unsafe unchecked load may exist as an escape hatch with a precise s.len() >= N premise, but ordinary tails need no unsafe call.

20.4 Integer and floating semantics#

Do not overload integer + until overflow policy is clear. Provide wrapping_add, saturating_add, and overflowing_add, with the last returning a value plus an overflow mask. An Option-like checked_add can be built by testing whether any overflow lane is set. A trapping vector add is omitted because panic order and partial results are hard to make compositional.

For floats, ordinary arithmetic follows Rust and target contracts as documented, without implicit reassociation. Define minimum_number and maximum_number separately from NaN-propagating minimum and maximum; names alone are not enough, so document NaNs and signed zero with truth tables. Reduction methods distinguish reduce_sum_ordered from reduce_sum_relaxed.

example minimum contract questions

min(NaN, 3.0)  -> NaN or 3.0?
min(+0.0,-0.0) -> which signed zero?
min(NaN, NaN)  -> which NaN payload, if payload is promised?

If payload preservation is not promised, say so. If target instructions differ, the implementation may need fixups. Portable semantics can cost extra.

20.5 Dispatch and escape hatches#

PlainVec's default implementation chooses supported mechanisms internally. A domain algorithm can also expose with_target_features through a checked dispatcher, but should not let feature-specific values escape to baseline code.

Escape hatches include conversion to/from arrays, architecture-specific modules, and an unsafe constructor from target registers where ABI and feature premises are explicit. They are pressure valves, not the main door.

Thought experiment — make VecN<T, N> part of every public algorithm API. Callers must chunk, mask, and dispatch even when they care only about slices. Mechanism complexity spreads outward. Keep domain APIs domain-shaped and use vector APIs where vector composition is genuinely the caller's task.

21. Improving existing SIMD systems responsibly#

Improvement starts with friction, not a favorite instruction. Friction might be unsafe tail boilerplate, inability to express a floating contract, poor compiler recognition, expensive ABI crossings, or unpredictable lowering. Record a small reproducible case and affected users.

Then formulate a hypothesis: “A first-class bounded partial-load operation will reduce unsafe overreads without slowing supported targets.” Separate semantic goals from performance goals. Safety and clarity may justify a change even when speed is neutral; speed claims need measurements.

Prototype at the cheapest faithful layer:

• An emulator tests instruction semantics and exceptions.
• Compiler IR tests recognition and lowering across targets.
• A library tests API usability and fallback semantics.
• Hardware models estimate area, routing, and timing before silicon.

Build adversarial tests: zero and one lane, all masks false, page boundaries, overlap, duplicate scatter addresses, NaNs, signed zeros, overflow, maximum vector length, and unsupported features. Compare against an independent oracle.

Measure diverse machines and workloads, including small sizes and unfavorable data. Consider ecosystem consequences: ABI state, debugger display, compiler maintenance, feature detection, binary size, migration, old hardware, and how code removes the experiment if it fails.

21.1 Plausible improvement: safer partial loads#

A library operation that guarantees no inactive-lane access could remove common unsafe code. It might fail as an improvement if targets lack masked loads and fallback copying is unexpectedly costly, or if its name causes users to assume fault suppression that the implementation cannot guarantee. Semantic clarity must survive every lowering.

21.2 Plausible improvement: width-polymorphic shuffle language#

A higher-level shuffle description based on segments and neighbors could map to fixed and scalable targets. It might improve portability. It might also be too abstract for compilers to optimize, conceal expensive lowering, or fail to express exact algorithms. Prototype representative transposes, parsers, and scans before standardizing it.

21.3 Plausible improvement: explicit reduction contracts#

APIs could distinguish ordered, reproducible-tree, and relaxed reductions. This clarifies floating meaning and compiler permission. It might fail through user confusion, ecosystem fragmentation, or a reproducible algorithm whose cost is unacceptable. Names, documentation, and defaults matter as much as instructions.

21.4 Plausible improvement: conflict-aware scatter#

A scatter returning a mask of conflicting lanes could help histograms. But hardware detection may cost as much as resolving conflicts; ordering semantics could become complex; and privatized bins or sorting may be superior. An ISA feature should not fossilize one workaround before algorithmic alternatives are measured.

Reject seductive changes when they optimize one demonstration, duplicate a composable sequence without durable gain, make semantics target-dependent, expand global unsafe obligations, or burden ABI and tooling disproportionately. Rejection is design work, not lack of ambition.

friction -> precise case -> semantic contract -> hypothesis
    -> prototype -> adversarial correctness -> diverse measurement
    -> ecosystem/ABI/migration review -> adopt, revise, or reject

22. Mental models to keep and discard#

Keep these models:

• SIMD compresses a truthful description across distinct values.
• Lanes are coordinates in an operation, not autonomous cores.
• Masks represent conditional participation as data.
• Layout chooses which facts become physically adjacent.
• Tails are ordinary inputs and proof tests, not embarrassing leftovers.
• Reductions are contracts about algebra and order, not just opcodes.
• Width is a mechanism parameter that may or may not belong in an API.
• Legal, profitable, safe, and portable are separate judgments.
• Scalar fallback defines resilience and can serve as an oracle.
• Performance claims are scoped evidence.

Discard these models:

• “Four lanes means four times faster.” Bottlenecks and overhead disagree.
• “Branchless means fast and constant-time.” Neither follows automatically.
• “A mask erases an invalid load.” Access semantics precede result selection.
• “Vectors are small arrays.” That encourages expensive indexing and ABI leaks.
• “Gather fixes bad locality.” It expresses addresses; it does not move them near.
• “The compiler should always vectorize obvious loops.” Obviousness omits proofs.
• “More ISA features are progress.” Every feature spends permanent treaty budget.
• “Floating addition is mathematical addition.” Finite rounding makes order real.
• “Portable means equal cost.” Semantic portability is the defensible promise.
• “Unsafe makes code fast.” Unsafe carries unverified premises, not speed.

When a mental model predicts an outcome, ask what observation could refute it. A model immune to evidence is not an engineering tool.

23. Philosophical glossary#

Abstraction boundary: The point where one layer exposes a contract while hiding changeable implementation details.

Associativity: The property that regrouping operands preserves a result. It must be judged under the actual finite arithmetic contract.

Control compression: Sharing one description of operation and progression across multiple data items.

Cost portability: Similar performance behavior across targets. It is a much weaker and harder promise than source or semantic portability.

Dependence: A required ordering because one operation consumes or conflicts with another operation's value or effect.

Determinism: Repetition under stated conditions produces the same result. Cross-machine bitwise reproducibility is a stronger, separate contract.

Divergence: Different lanes require different control paths while execution is shared, often causing paths to run under separate masks.

Identity: A neutral element for an operation, such as zero for addition.

Invariant: A fact maintained at a specified program point and usable in a correctness proof.

Lane: One coordinate of values participating in a vector operation.

Legality: Preservation of the language and program's required semantics by a transformation.

Locality: Nearness of accesses in time or address, allowing storage and translation mechanisms to reuse fetched data.

Mask: A vector of Boolean participation conditions.

Ontology: A representation's decision about primary entities and relations; in data layout, the decision about which facts live together.

Predication: Qualifying an operation so only selected lanes participate or commit results according to documented semantics.

Profitability: Expected benefit after complete costs, distinct from legality.

Reduction: Combining many values into fewer values, commonly one.

Scalable vector: A vector whose effective length is selected by an implementation rather than fixed as one source-level lane count.

Scan: Computing prefix combinations, such as every running sum.

Semantic portability: Preservation of documented meaning across supported targets, even if implementation and cost differ.

Strip-mining: Processing an iteration space in repeated chunks sized to an execution mechanism.

Vectorization: Transforming scalar or independently described work into vector operations while preserving the chosen contract.

24. Socratic and adversarial questions#

Q1: Why does SIMD exist if superscalar CPUs already overlap instructions?

Superscalar execution overlaps separately described instructions and is limited by fetch, decode, scheduling, dependencies, and execution resources. SIMD shares description across data. The mechanisms complement each other: a processor may issue multiple vector instructions simultaneously.

Q2: Are masked-off lanes free?

Not necessarily. They may consume issue slots and arithmetic resources; exact behavior depends on the target. Their semantic value is conditional participation, not guaranteed proportional cost reduction.

Q3: If lanes are independent, why are reductions SIMD?

Reduction begins with independent partial accumulation, then deliberately uses cross-lane combination. SIMD is broader than lane-wise maps, but reductions require algebraic and ordering arguments.

Q4: Why not always transpose AoS into SoA?

Transposition costs movement and storage, may harm other consumers, and may alter ownership or synchronization. It pays only when improved repeated access outweighs those costs under the full workload.

Q5: Can a masked load safely point outside the allocation?

Only if every out-of-range lane is inactive and the API and ISA explicitly guarantee inactive lanes make no access, while address formation itself obeys language rules. A full load followed by select cannot provide that guarantee.

Q6: Why can a wider vector be slower?

It may increase register pressure, shuffles, memory demand, setup, and power or frequency cost, while useful work or bandwidth remains fixed. Width is one resource choice among many.

Q7: Is scalable-vector code automatically future-proof?

It avoids one class of fixed-width assumptions. It still depends on operation availability, cost, compiler quality, ABI constraints, and algorithm shape. Future-proof is an aspiration with boundaries, not a binary property.

Q8: Why preserve a scalar fallback after SIMD tests pass?

Unsupported targets, small inputs, regressions, and changing hardware remain. The fallback is also a readable semantic oracle and enables deletion of the specialized path.

Q9: Should a compiler vectorize a legal loop?

Only when its cost model predicts benefit or policy requests size-independent vectorization. Legal merely means allowed. Setup, tails, code size, and target cost can make scalar code better.

Q10: Does one SIMD instruction imply one hardware operation?

No. An implementation may split it into internal operations, and a portable vector operation may lower to several ISA instructions. Architectural words describe semantics, not a universal physical count.

Q11: Can intrinsics be a clean abstraction?

They are clean at a target-specific mechanism boundary. They become leaks when width, features, and instruction quirks unnecessarily enter domain-facing APIs.

Q12: What makes SIMD code art rather than cleverness?

Art is disciplined selection: honest regularity, fitting representation, explicit contracts, local proofs, economical movement, and evidence. Cleverness without those constraints produces fragile novelty.

25. Exercises requiring design arguments#

Exercise 1: No-mask vector machine#

Design tail handling and conditional clamp for a fixed eight-lane machine with no masks. Compare scalar cleanup, padding, and bitwise selection. State memory safety, code-size, and cost consequences. Do not merely choose the fastest; specify workloads under which each choice changes.

Exercise 2: Width in a public image API#

An image library proposes fn filter(block: VecU8x64) -> VecU8x64. Argue for or against it from domain block size, x86 and Arm support, scalable vectors, ABI, callers, and deleteability. Propose a domain-shaped alternative and one justified low-level escape hatch.

Exercise 3: Floating sum contracts#

Specify three f32 sum APIs: source-ordered, cross-target reproducible, and relaxed. Define NaN handling, signed zero if relevant, allowed trees, and error or performance expectations. Explain which compiler transformations each permits.

Exercise 4: Gather or reorganize#

Given repeated reads through one million indices into a 64 MiB table, construct a decision model comparing direct gather, sorting indices, and packing values. Include reuse count, cache behavior, output ordering, conversion cost, and mutation. Identify measurements that could reverse your initial recommendation.

Exercise 5: Per-lane exceptions#

Extend ClearVec with checked division. Decide behavior when lanes divide by zero: aggregate trap, result mask, saturating value, or scalarized execution. Discuss restart, inactive lanes, stores after the operation, compiler IR, and language mapping. Reject at least one option with reasons.

Exercise 6: Alias proof#

Analyze vectorization of out[i + 1] = input[i] + 1 under separate slices and under raw pointers that may overlap. Give a legal runtime range check, explain which overlap direction creates dependence, and describe a scalar fallback.

Exercise 7: Improvement proposal#

Choose one SIMD friction point. Write a one-page proposal containing exact semantics, a counterfactual without the feature, prototype layer, adversarial tests, target matrix, ecosystem cost, migration plan, and rejection criteria. The grade depends more on credible rejection criteria than enthusiasm.

Exercise 8: Beauty challenged by evidence#

Construct two implementations: one visually elegant vector pipeline and one less elegant mixed scalar/vector path. Predict bottlenecks, measure several sizes and alignments on two machines if available, inspect assembly, and write a conclusion whose scope does not exceed observations.

Exercise 9: Representation as ontology#

Design AoS, SoA, and blocked layouts for particles updated by physics, rendered as complete records, and occasionally serialized. Draw byte adjacency, list each consumer's accesses, estimate conversion frequency, and defend one compromise.

Exercise 10: Teach the disagreement#

Prepare opposing arguments for fixed-width and scalable-vector APIs. Each side must acknowledge its strongest weakness. End with criteria for choosing rather than declaring a universal winner.

26. A framework for a compelling technical talk#

A SIMD talk should produce transferable judgment, not applause for large speedup numbers. Build it as an argument with visible boundaries.

26.1 Begin with one meaning#

Show a scalar oracle small enough for the audience to predict. State overflow, floating, ordering, and boundary behavior. Ask which iterations are truly independent. This establishes a contract before machinery appears.

26.2 Reveal compressed description#

Draw scalar operations beside one vector operation. Explain shared control and distinct lane values. Avoid saying the CPU “does magic” or that lanes “want” to execute. Physical mechanisms do not have intentions.

26.3 Introduce a constraint that breaks the simple story#

Use a tail, conditional, reduction, or AoS layout. Trace it visually. Let the audience see why masks, cross-lane operations, or representation changes exist. Then show their cost and proof obligations.

meaning -> regularity -> vector mechanism -> broken edge case
   ^                                          |
   +---- revised representation/proof --------+

26.4 Compare counterfactuals#

Ask what scalar unrolling, threads, a no-mask ISA, and scalable vectors would do. Counterfactuals explain design rather than presenting current hardware as fate. State where each alternative wins.

26.5 Separate four verdicts#

Put legality, safety, profitability, and portability on separate slides. A compiler may prove legality; an unsafe block needs memory and feature proofs; a benchmark supports profitability on named systems; an API defines portability. Do not let one green check mark stand for all four.

26.6 Show evidence without theater#

Include benchmark distributions, machine and toolchain, sizes, assembly excerpts, and at least one result where SIMD loses. Explain counters as clues. A negative result increases credibility when it sharpens the model.

26.7 End with a reusable critique#

Give the audience questions they can apply tomorrow:

1. What meaning is fixed?
2. What description is genuinely shared?
3. What representation exposes that sharing?
4. What movement and control remain?
5. What proofs make the mechanism legal and safe?
6. What evidence establishes value, and only where?

Avoid hype phrases such as “free speed,” “always faster,” and “future-proof.” Replace them with scope, mechanism, and uncertainty. The most compelling talk lets listeners criticize the speaker's own design using the supplied framework.

27. Further thought experiments#

27.1 What if vectors had infinitely many lanes?#

Finite inputs would fit in one description, but register storage, wires, memory delivery, mask representation, and reduction routing would be infinite too. The impossible machine clarifies that width trades control sharing against physical reach. Scalable vectors abstract implementation width; they do not make it infinite.

27.2 What if shuffles were free?#

AoS-to-SoA rearrangement inside registers would become easier, as would scans and transposes. Memory layout would still determine bytes fetched, cache lines, and conversion at boundaries. Free shuffles reduce one movement cost, not all movement or semantic ordering.

27.3 What if branches never mispredicted?#

Predictable control would be cheaper, reducing motivation for branchless select. SIMD would still compress loop and operation description for lanes taking the same path. Divergent lanes would remain difficult because one vector instruction still names one operation.

27.4 What if all operations were associative?#

Reductions and scans could regroup freely, but dependencies and communication would remain. Side effects, memory bounds, and cost would still constrain vectorization. Associativity grants algebraic permission, not bandwidth.

27.5 What if hardware exposed no stable ISA?#

Compilers would need a deployment-time or runtime translation layer, and binary compatibility would move into that layer. Hardware could evolve freely, but tooling, reproducibility, startup, and trust costs would rise. An ISA's rigidity is also the source of ecosystem coordination.

27.6 What if every vector had an implicit validity mask?#

Tails and nullable data could compose naturally. Every operation would need rules for validity propagation, reductions, exceptions, and storage. Extra mask state could cost registers and complicate interoperation. Some domain APIs benefit from this model; imposing it on all vectors may tax dense computation.

27.7 What if compilers could ask the programmer one question?#

The highest-value question is often semantic: “May I reorder these operations?” The answer can unlock reduction trees and memory scheduling. Asking “Do you want SIMD?” is weaker because SIMD is a mechanism whose legality and value depend on the unstated contract.

28. Synthesis: designing parallel meaning#

SIMD begins with meaning, not registers. A program promises values, order, failure behavior, and memory effects. The engineer identifies where one honest description covers many distinct values. That regularity is control compression.

Representation makes the regularity reachable. AoS and SoA choose different neighbors; masks represent conditional participation; lane coordinates preserve correspondence; fixed and scalable widths place different assumptions in code. Every representation illuminates some operations and casts shadows on others.

Mechanism turns representation into execution. Loads, arithmetic, predicates, shuffles, gathers, reductions, compiler IR, and ISA instructions are vocabulary, not purpose. Their existence reflects treaties among physical hardware and a long-lived software ecosystem. Wider vocabulary is useful only when its meaning is coherent and its full cost justified.

Proof keeps compression truthful. Feature support, active memory bounds, alias rules, tail validity, algebraic identities, floating order, and side effects are separate obligations that compose. A narrow unsafe core states the pieces the language cannot prove. A scalar fallback keeps the operation larger than one machine mechanism.

Evidence disciplines taste. Assembly reveals translation; counters suggest bottlenecks; adversarial tests challenge correctness; benchmarks establish scoped performance. No attractive diagram, nominal lane count, or single machine can substitute for this chain.

The art of SIMD is therefore neither mnemonic mastery nor decorative parallelism. It is the cultivated ability to find regularity without inventing it, choose adjacency without forgetting conversion, compress control without erasing exceptions, expose mechanism without trapping an API, and prefer measured truth over elegant expectation.

Meaning asks what must remain. Representation asks what can be made adjacent. Mechanism asks what the machine can express. Proof asks why the transformation is allowed. Evidence asks where it is worthwhile. A durable SIMD design answers all five, and knows which answer to revise when the world disagrees.