One interrupt into RAM, and the firmware got 18 KB smaller

By Spencer and Claude. Claude did the writing; the material is Spencer's engineering notes, measurements, and repositories. Nothing here has been independently fact-checked, so accuracy is not guaranteed — read it as a lab notebook, not a datasheet.

Written as if we were about to send it upstream, because that is the only format that forces you to say what you actually proved. A WS281x driver had carried IRAM_ATTR for years, but its refill interrupt still lived in flash. Three edits and a build-time placement gate moved the whole path into internal RAM. The change costs 1,740 bytes of IRAM, returns 18,680 bytes of firmware image, and survives the flash-write storm that makes the shipping driver replay pixels.

The failure

An ESP32 normally executes most code from external SPI flash through a cache. During a flash write or erase that cache is disabled, and ESP-IDF masks every interrupt that was not explicitly registered as safe to run without it. A page program lasts a few hundred microseconds. A sector erase can last tens of milliseconds.

Our LED driver cannot wait that long. FastLED's I2S parallel backend sends from a circular list of four DMA descriptors and refills them from an end-of-frame interrupt. Each descriptor holds one pixel, or about 30 microseconds of wire time, so the whole ring buys roughly 120 microseconds.

If the refill interrupt misses that window, the DMA engine does not stop or report an underrun. It wraps around and transmits descriptors that still hold old pixels. Those pixels are inserted into an otherwise valid frame, everything after them shifts, and the same-length tail falls off. The arithmetic is simple: a gap of n descriptor-times replays max(0, n - 4) descriptors, so the ring absorbs three descriptor-times for free and everything past that reaches the wire. Two extra laps walks through a byte-exact capture of one such splice: four pixels sent three times, everything after them shifted, the rest of the frame byte for byte correct. From across the room it looks like a one-frame spatial jump, not random color noise.

At 60 fps the bitstream occupies about 14.4 of every 16.7 milliseconds (why that is a multiplication). A flash operation starting at a random moment therefore lands inside a transmission about 86% of the time, and one page program is already longer than the ring.

This is the mechanism behind a familiar complaint: the LEDs glitch when Wi-Fi connects, settings are saved, or OTA runs. It is usually filed as noise, or a power problem, or "add a capacitor." It is none of those. It is an interrupt that is not allowed to run.

Why the attribute did nothing

ESP-IDF's intended fix is straightforward: mark the entire handler path with IRAM_ATTR and allocate the interrupt with ESP_INTR_FLAG_IRAM. The upstream driver already carried the attribute. The handler was still in flash, along with the two functions it called.

The handler is a static member of a class template:

template <int DATA_PIN, int T1, int T2, int T3, EOrder RGB_ORDER, ...>
class ClocklessController : public CPixelLEDController<RGB_ORDER> {
    static IRAM_ATTR void interruptHandler(void *arg);   // silently ignored

An implicitly instantiated template member gets vague, or COMDAT, linkage so duplicate copies can be merged across translation units. Its section name is part of the deduplication key, and this toolchain silently drops the request to place it in a different section. There is no warning.

We tried the four plausible exits on the real ESP32 compiler (xtensa-esp32-elf-g++ 8.4.0, -Og, -ffunction-sections):

Specialization works, but our firmware instantiates the controller eight times, once per pin. Eight hand-maintained interrupt handlers would be a poor fix for one duplicated path.

There was a clue directly above the driver's interrupt registration. An upstream comment says enabling the IRAM flag causes a panic even though "everything is in IRAM." The panic was the allocator correctly rejecting a flash-resident handler. The mystery was the evidence.

The change

The patch has three parts.

De-template the interrupt path. The handler, buffer fill, per-pixel fill, and 32-lane transpose are file-scope functions with internal linkage. Their bodies are the old members' bodies. Internal linkage removes the mergeable template section that defeated the placement attribute, and eight copies become one.

Require the IRAM interrupt flag. The driver now calls esp_intr_alloc(..., ESP_INTR_FLAG_IRAM, ...). If a later change puts any of the entry path back in flash, the board aborts at boot instead of quietly replaying pixels. The boot log also prints the resolved address:

I2S: EOF ISR at 0x40083cac (IRAM); flash ops excluded by show: 1
I2S: EOF ISR at 0x400e2e88 (flash, cache-dependent); flash ops excluded by show: 1

Gate the linked artifact. Source annotations had already lied once, so there is a check that reads the ELF instead. tools/iram-eof-check/check.sh requires every reachable function to sit in IRAM or mask ROM, every touched data object to sit in DRAM rather than flash-mapped DROM, and every old template member to be absent. It also resolves Xtensa's far-call sequence, where l32r loads a literal-pool address and callx8 looks like an indirect call through a register.

The gate reads nm, not the linker map. GNU ld maps omit the internal-linkage functions that make this technique work. On the patched branch it finds 30 reachable functions, all in IRAM or ROM, with no escape-hatch allow-list. Run against the shipping binary as a negative control, it fails with 22 problems, including all eight flash-resident handlers.

The ELF check proves placement, not that ESP_INTR_FLAG_IRAM was passed. The allocator's boot-time rejection and the address in the log cover that half. The two halves do not cover each other: the allocator only inspects the handler's own address, so a later change that pushed something deeper in the call graph back into flash would boot fine and replay pixels. That is the case the ELF check exists for, and it is still a script you run against a build rather than a stage in the tier-1 gate. Wiring it in is the follow-up.

The four-arm test

One production pole ran its idle show. Each arm came from its own clean build and ran once, in sequence, with two quiet minutes followed by a 60-second storm of settings writes at roughly one every two seconds.

The second variable was our existing mitigation. Shipping firmware holds ESP-IDF's flash-operation lock across each frame, serializing flash against LED output. Compiling that lock out is not a shipping configuration; it is the sharpest way to expose the mechanism.

The discriminating comparison is the middle pair: same board, same stimulus, lock removed on both. Shipping code replays pixels. The IRAM build does not, and no longer needs that lock to survive this mechanism.

Every arm confirmed 21 to 29 real writes. That matters because NVS skips a write when the value is unchanged; an unconfirmed storm can produce a clean zero without touching flash. We made that mistake on the first attempt.

Worst service gap falls from 8.6 and 10.2 descriptor-times to about 1.6. That is roughly five times more headroom against the four-descriptor boundary. These are since-boot maxima, not windowed storm deltas, so the chart is a margin measurement rather than a causal attribution. It still separates builds that look identical on the lock-on counter window.

Internal RAM is the scarce resource, and the change spends 1,740 bytes of it. Four functions and their literal pools explain 1,737 bytes, so the number is accounted for.

The full image shrinks by 18,680 bytes. The old template path existed once per pin even though only one instance ran, so moving one copy into IRAM removed eight copies from flash. That direction is not in doubt, but unlike the IRAM figure it is an attribution rather than a per-symbol ledger: 20,436 bytes of flash text over eight copies is about 2.5 KB each, which is the right size for this path, and nothing separates it from whatever inlining and dead-stripping also moved when the members became free functions. The same property that defeated the attribute had also quietly inflated the firmware.

The paper trail caught two mistakes

This failure had already appeared in our own firmware. A heartbeat read the OTA partition every ten seconds to report a version line. Nobody noticed, because nothing about "report the version number" suggests touching flash. Over 1,803 seconds it produced 184 near misses and nine late events, the gaps between late events all integer multiples of a 10.0166-second period, and 28 replayed pixels. Latching the version after the first answer took both late events and replayed pixels to zero.

It also contaminated an A/B test. A profiling emitter seemed guilty: 50 late events against the control's four, all six randomized pairs in the same direction, sign test p = 0.031. Both arms carried the version read, and the emitter's five-second tick coincided with every other ten-second read. After the latch landed, the same comparison produced zero late events in both arms. The control's nonzero background was the clue the p-value could not explain.

Our prewritten prediction set made a different mistake. It correctly called all the discriminating arms, but predicted hundreds of replayed pixels because we assumed every late event cost at least one full four-pixel lap. The exposed arm measured 121 pixels across 86 late events, or 1.41 per event. A marginal overrun often repeats one or two pixels, not four. The conclusion survived; the arithmetic did not.

What IRAM does not buy

This test covers cache-disabled flash operations of the length an NVS page program takes, on one board running its idle show, not being streamed to. It does not cover flash-bus contention from cache misses, which needs a busy working set; it did not exercise an OTA or a sector erase; and it does not prove wire geometry, because no logic analyzer was attached. The counters were designed after a prior byte-exact capture had already established the replay mechanism.

More importantly, IRAM cannot rescue an interrupt when the core or a required kernel lock is unavailable. We found that boundary the next day. A bench-only task sampler called uxTaskGetSystemState once a second. That function scanned every task's unused stack under the kernel task lock, holding it for a 3,383 microsecond median against a DMA ring that buffers 120 microseconds. It consumed only 0.357% of core 0 and still damaged a frame every second.

Replacing the call with a snapshot plus per-task queries that skipped the stack scan took the reported median hold to 60 microseconds, with one hold in 119 still reading 306. That 306 was not a lock hold. The sampler stopped its clock one line after taskEXIT_CRITICAL(), which on this IDF releases the spinlock before it restores the interrupt level, so every microsecond of the task being descheduled after the lock was already free got charged to a hold that had already ended. Splitting the window at the release put the time the lock was actually held at a 97 microsecond median and a 146 microsecond maximum, with 7 of 353 ticks over the line: out of budget by tens of microseconds rather than by 2.6x, and the thing over the line was the enumeration itself rather than a stall.

Merging that split with the placement change above closed it. On the bench pole under the live stream, 897 ticks read 18 microseconds median and 25 for the lifetime maximum, none over the ring, which meets the rule with 4.8 times the margin. The rule is the part that outlives the episode: any interrupts-off or required-lock hold longer than the hardware buffer is a failure, regardless of its average CPU percentage.

Residency buys the right to execute while the cache is off. It does not buy a core, a lock, or a deadline.

Why this is not a pull request yet

The color-order blocker is gone. The file-scope path now supports all six FastLED color orders, including mixed orders across lanes, while preserving the optimized all-GRB path.

Upstreaming still needs a decision about whether the binary-placement gate belongs in the library or beside it. The mechanism, the compiler behavior, and the artifact gate transfer; that maintenance boundary still needs an explicit owner.

What transfers

There are two different ways to lose a deadline-critical interrupt.

Residency: can the handler, every transitive call, and every referenced object be reached with the cache off? A placement attribute is a request. If placement is load-bearing, assert it against the linked binary with a tool that can see the relevant symbols.

Blocking: can any task hold a core or required lock longer than the hardware buffers? Symbol tables cannot answer that, and duty-cycle percentages hide burst shape. Measure the worst hold against the hardware deadline.

And when a control arm has a background rate that should be zero, do not call it noise. Ours was explaining the experiment to us.

Counters are the driver's own, read over HTTP and differenced across windows; sizes are from idf.py size with per-symbol figures from nm -S. The four-arm harness, raw results, and prediction scorer are committed in pfx-firmware under docs/DEVNOTES/captures/2026-08-01-iram-smoke/; the design record and placement gate live on the iram-eof/claude branch. The original wire capture is described in Two extra laps. The sampler measurements are recorded in docs/DEVNOTES/2026-08-02-sampler-fix.md and docs/DEVNOTES/2026-08-02-sampler-hold-outlier.md, with the bugs in pfx-proto/bugs/2026-08-02-trace-runtime-sampler-blocks-i2s-eof-isr.md and pfx-proto/bugs/2026-08-02-sampler-hold-measured-past-the-lock-release.md.