Three optimizations, three estimates, zero hits

By Spencer and Claude. Claude did the writing; the material is Spencer's engineering notes, measurements, and repositories. Nothing here has been independently fact-checked, so accuracy is not guaranteed — read it as a lab notebook, not a datasheet.

Three changes to one hot function on an ESP32, each with a predicted range written down before measuring, each measured alone behind its own compile flag from a single source tree. Predicted 20 to 30%, 10 to 20%, 15 to 30%. Measured 19%, 33%, 5.5%. The change the plan led with came in under its band, the one nobody was excited about was three times its own low estimate, and the third missed so badly it shipped switched off.

The function

fillBuffer() encodes pixel data into the bit pattern an I2S peripheral shifts out to six parallel LED strips. It runs inside the refill interrupt, once per DMA buffer, for every frame. At the start of this work it cost 8.99 ms per frame and occupied 62% of the output burst. That is the kind of number that makes everything else on the chip worse, because while it runs, the interrupt is holding off the Ethernet driver.

Three candidate optimizations came out of reading the driver:

1. Skip dead transposes. The driver runs three 8x8 bit transposes per pixel and masks two of them away for anyone driving fewer than 24 parallel outputs, which is nearly every user of it.
2. Hoist the store arithmetic. The inner loop that writes the encoded words recomputes things that do not change.
3. Multi-pixel DMA buffers. Put four pixels in each DMA buffer instead of one, so the interrupt fires a quarter as often.

Estimates first, and why that is a rule here

Our dev notes contract requires the estimate to be written into the file before the measurement is taken. It is a small discipline that costs about ninety seconds and it is the only way the error stays visible, because afterwards you cannot reconstruct what you believed. Everyone remembers having expected roughly what happened.

Recorded up front: 20 to 30% for the transposes, 10 to 20% for the store hoist, 15 to 30% for the multi-pixel buffers. Those bands came out of reading the code, not out of a cycle model: two of three transposes are provably discarded, the store loop recomputes an index 48 times a pixel, four pixels per buffer is a quarter of the interrupts. What the bands are calibrated on is judgment, which is the whole point of writing them down where they can be scored.

The plan led with the transposes. That is the change with the good story, the one you can explain in a sentence, and it is the one the roadmap put first.

Measured, one flag at a time, from one tree

Each one sits behind its own compile flag, and every arm was built from the same source tree with only that flag moved. This part is not optional and it is worth saying why: two builds of behaviorally identical code on this target differ by up to 10.7% purely through code layout shifting cache line alignment, against a same-binary repeat noise floor of 0.4%. Comparing against a number recorded from last week's binary measures layout as much as substance. We know because we did it, published a 6.6% improvement internally, and withdrew it the same day once a clean A/B existed.

One tree and one flag is the discipline, not a control. Flipping a flag changes the code, so it changes the layout too, and each figure above still carries a layout draw of unknown sign inside it. The 10.7% is the worst case we have measured, and it was on idle compute; on this metric, fillBuffer itself, the same identical-code pair differed by 4.4%. That is comfortably below the two wins and it is most of the 5.53%, which is worth remembering before treating that number as an effect size.

The flag's value is also baked into the artifact and read back out of the binary, so an arm cannot claim to be something it is not.

Together the two shipping changes took fillBuffer() from 8.99 ms to 4.45 ms per frame, a 50.5% cut, with interrupt duty falling from 62.3% to 30.8% and no change in show() above the 0.4% same-binary repeat noise floor. That is a fourth build with both flags on, not the two figures above multiplied together: neither on, each alone, both on is a complete square, and the pair turns out to add rather than compound.

Why the boring one won

The store hoist was underestimated by the roadmap and by me, and the mechanism is visible in the disassembly.

The output buffer is a volatile uint32_t *. From the compiler's point of view that pointer may alias anything, including the two globals the loop reads on every iteration. So it cannot cache them. Each of the 48 stores per pixel reloads both globals from memory, performs two multiplies, and re-ANDs a mask that never changes across the loop. None of that is the interesting part of the function, and all of it is inside the hottest loop in the driver.

Hoisting the invariants out is a boring, four line change with no story attached. It was worth about 1.7 times the transpose change.

That is the shape of the finding, and it generalizes further than this function: optimization intuition anchors on whichever part of the code is easiest to describe. A triple bit transpose is photogenic. The address arithmetic wrapped around it is not, and it was nearly twice the cost.

The dud, and why 5.53% was not the reason to drop it

Multi-pixel DMA buffers returned 5.53% for four times the DMA RAM. That alone might still have been worth taking. Two other numbers killed it.

It costs show() an extra 0.09 ms, because the driver prefills every DMA buffer before starting the transfer and each buffer now carries four pixels, so prefill volume rises. That one was pre-registered as an expected regression with a predicted size of about 0.2 ms: right mechanism, right direction, half the magnitude, which is the estimate working as intended.

The real objection is that it raises typical interrupt residency from 19 to 76 microseconds with level 1 interrupts masked throughout, which delays the Ethernet controller. That pair is derived from the buffer geometry rather than timed on the board. On a device whose job is to receive 60 frames a second over the network, spending that to save 5% of a function we had just halved anyway is a bad trade. It ships off.

"Dud" is true of throughput and only of throughput. A later counter-based A/B, run on top of an unrelated flash-exclusion fix, found the bigger buffers absorbing a small residual of interrupt-latency events: matched seven-minute soaks logged 4 late events and 7 replayed pixels at one pixel per buffer against 0 and 0 at four. That is promising as mitigation and it is not shipping evidence, so the flag stays off until its wire-quality gate passes. A change can be worthless on the axis you estimated and useful on one you never priced.

It is also the only one of the three that is not an equivalence transform by construction, which matters for the next section.

Correctness tested rather than asserted

Both shipping changes are equivalence transforms by construction, and the argument for each is a paragraph of code reading: the two skipped transposes write only into bits a mask discards below nine controllers, and the hoist moves quantities that do not change inside the loop. An argument like that is exactly the kind that is right until it is not, so it gets tested. A host harness replicates both encode paths and compares their 480 byte output word for word, across controller counts from 1 to 24, with random stale bytes seeded into unused slots and a controller dropping out on one trial in seven.

One precondition worth naming: the transpose skip is only enabled where the discarded results are provably dead, controller counts of eight or fewer, which covers every PoleFX build; outside that it falls back to the stock path. The 1-to-24 sweep exists to exercise both the optimized branch and the fallback.

1,431,426 comparisons, zero mismatches. That is a randomized sweep over a tested domain rather than an exhaustion of the input space, so what carries the word "equivalent" is the structural argument, which the sweep is there to falsify and did not. The harness spent a week existing only in a session scratchpad, which meant that count was our word rather than a receipt; it has since been recovered from the session record, committed (tools/engine-host-test/fillbuffer_equiv.cpp), and re-run: it is seeded, and it reproduces 1,431,426 comparisons and zero mismatches exactly.

On hardware, each arm was a six-second capture, about 358 frames on each of the three lanes the analyzer was probing at the time. Every arm: zero sparkle flips, zero late frames, zero frames below the 11,520-bit body. Not exactly 11,520 on the wire, note. The bursts run about 11,548, because every frame is followed by the driver's documented stale tail, and reading that overage as corruption has cost us a day before.

The part that stung

While all of this was being measured carefully, the test suite guarding the patterns underneath it had silently stopped compiling the day before.

A previous task had added one #include to a shared header. Three of that header's declarations take a type the host test shim deliberately stubs out, so the cross language equivalence harness, 756 combinations across three independent implementations, failed to build. It had been reviewed, built, measured on hardware, committed and pushed. Nothing said a word, because that day's verification ran a different make target.

It surfaced by luck, and we reproduced it against a pristine archive of the committed revision to be sure it was not an artifact of somebody's working tree.

That was the fourth instance in two days of the same shape: a build stamp dropped by the linker, a guardrail whose key names never matched the metrics it guarded, two mutation testing blind spots, and now a harness that stopped building. Every one of them was written carefully by someone competent, looked correct, and did nothing.

What we do now

The estimate goes in the file before the measurement, as a range, every time. You will be wrong in one direction more often than the other and the direction is the finding. Ninety seconds later it is gone, because by then you remember having expected whatever happened.

The other rule is duller and saved us more. Every change gets its own compile flag, every arm comes from one tree, and the flag's value is readable in the built binary so an arm cannot misreport itself. At these effect sizes the alternative is benchmarking your linker.

All figures measured on the bench board over 2026-07-27, idle runs pinned to one pattern from a cold state, every arm's flag values read back out of build/pfx.bin rather than trusted from the edit. The equivalence harness runs on the host, not on the board.