Chasing 60 fps down the LED wire with a $15 logic analyzer

Per-pixel decoding of our LED data lines with a knockoff Saleae: a refactored firmware that turned out to deliver 42.5 of the engine's 60 fps, five measured builds to close the gap, and an RGB sparkle bug that predated the whole session. Includes one self-inflicted bug, one disproven hypothesis, and a ~392,000-frame soak to prove the fixes.

Why

Our poles are driven by an ESP32 receiving 60 fps frames over Ethernet from a Raspberry Pi and shifting them out to six channels of 480 SK6813-mini pixels through FastLED-idf's I2S parallel driver (the same hardware measured electrically in the power post; the firmware drives the strips within WS2813 timing spec). We had just rewritten the firmware's frame pipeline: event-driven, triple-buffered, direct task notifications, replacing a polling design with silent packet-drop races. The refactor had never been hardware-tested, and our only acceptance test was "the LEDs look fine."

"Looks fine" is not a measurement. A $15 FX2 logic-analyzer clone clipped to the data lines turns the wire itself into ground truth: every bit of every pixel of every frame, timestamped at 125 ns resolution. So we asked four questions:

1. Does every frame the Pi sends actually reach the LEDs: any stale, repeated, or torn frames?
2. What frame rate hits the wire, and how is it paced?
3. Where does the 16.67 ms frame budget actually go, and can we hit 60 fps?
4. Is the waveform itself in spec, pulse by pulse?

The answers, in order: yes-but, 42.5 fps in bursts, mostly one bad memcpy, and no, and that "no" turned out to be a bug older than the day's work.

The bench

Hardware under test: the "Top Support" bench controller (ESP32, ESP-IDF v4.4.8), fresh refactor branch, driving 6 x 480 px strips. Source: the Pi engine multicasting 240x12 frames at 60 fps over UDP, 6 packets x 1,440 bytes per frame. Instrument: a knockoff "Saleae Logic", a Cypress FX2 clone driven by sigrok-cli/fx2lafw at 8 MHz x 8 channels, probes clipped to the LED data lines:

sigrok-cli --driver fx2lafw --config samplerate=8m --samples 100m -o cap.sr

Decoding is simple enough to do with numpy (the tooling is now committed as pfx-firmware/tools/analyzer/analyze_capture.py): each bit is 1.25 µs, a high longer than 500 ns is a '1', a low longer than 50 µs is a frame gap, and a frame is 480 px x 24 = 11,520 bits per lane. At 8 MHz sampling the legal high widths land in clean 125 ns bins, so anything outside them is a generation glitch, not sampling noise.

Fig 1: real captured data. Top: all lanes start within 125 ns of each other, direct wire proof that one I2S peripheral clocks every strip. Bottom: single bits resolved and measured, 250 ns highs for '0', 750 ns for '1'.

Instrument gotchas worth passing on:

• It buffers in RAM: ~100-128 M samples max, so 12-16 s per capture at 8 MHz. Long soaks are capture loops, not one capture.
• It frequently stops early ("Device only sent 91396608 samples."), which is harmless, and occasionally wedges completely: 0 edges on all 8 channels while the device under test demonstrably streams. Replug USB.
• sigrok uploads its own firmware into the FX2's RAM per session, so the vendor app still works after a replug.

Finding 1: the refactor works, but only 42.5 fps reaches the wire

First capture of the day, stock WS2813 timing (320/320/640 ns), 2 DMA buffers. The event pipeline itself passed cleanly: 170 of 170 frame payloads unique, zero repeated stale frames, zero tearing.

The pacing did not pass:

frame intervals (us): n=169 median=17547.8 mean=23593.1 std=7835.91
equivalent fps (median): 56.99
intervals >1.5x median (likely dropped/late frames): 69   (of 169 = ~41%)

170 frames in a 4 s capture is 42.5 fps average. The interval histogram is bimodal, ~17 ms and ~33 ms, and the 33 ms mode is a missed frame period. The skips came in bursts, which is why the strip still looked smooth-ish.

Meanwhile a raw packet sniff on the Pi's interface showed the engine sending a clean 60.2 fps with zero frame-number skips. The deficit was entirely in the firmware, and the budget math said why: show() for 480 pixels took ~14.85 ms at stock timing, plus ~2.5 ms of loop overhead, against a 16.67 ms budget. There was no budget left, and an unpinned priority-18 lwIP task added jitter on top.

Finding 2: five builds to 60 fps, each measured before the next change

Fig 5: the day's builds. Average fps on the wire (top) and skipped or late frame periods (bottom).

Three of these steps deserve their stories told honestly.

The self-inflicted bug (#1 to #2)

Raising the DMA buffer count from 2 to 4 with an all-prefilled circular link made the end-of-frame semaphore fire while three real buffers were still queued, so i2sStop() cut the tail off every frame. Nothing visibly wrong on the strip; the analyzer caught it because it counts bits:

bits-per-frame histogram bottom: 11480: 2, 11481: 3, 11504: 12, 11505: 5, ...

Frames up to 40 bits (~1.7 pixels) short of 11,520. The fix is a stop countdown (gStopCountdown = NUM_DMA_BUFFERS - 1 before releasing the semaphore); after it, minimum frame transmission duration jumped from 2,054 µs to 14,470 µs, meaning no more truncated tails. A frame shorter than 11,520 bits remains the number one red flag in any future capture.

While here we measured DMA buffer economics: each extra buffer costs ~30 µs per frame in prefill and stop countdown. A later experiment with 8 buffers measured +0.4 ms per frame and 26% late frames. 4 is the sweet spot, and the comment now lives at NUM_DMA_BUFFERS in the driver.

The big one (#4): move work off the render path

Temporary PROFILE_FRAME_TIMING instrumentation printed the budget over serial:

FRAMETIMING n=300 map=1194/1201 show=15175/15670 wait=2062/17115 cycle=18434/33293 (avg/max us)

map+copy 1.19 ms, show() 15.18 ms average and 15.7 ms max: ~0.3 ms of margin against 16.67 ms. That margin is the residual skip bursts. The fix: process_packet now applies a precomputed per-packet pixel map plus gamma as each UDP packet arrives, while the previous frame is still streaming out of the I2S DMA, and renderFrame collapses to a memcpy. Measured after:

FRAMETIMING n=300 map=24/32 show=14891/15623 wait=1862/18145 cycle=16777/33057 (avg/max us)

1,190 µs to 24 µs, a 50x cut, and the wire went to 60.0 fps. (We also asked whether dropping gamma correction would help: no. It rides along in the same per-packet pass, so it is free.)

The build-hygiene incident

One flash of "the fix" changed nothing on the wire. The reason was not firmware: idf.py build 2>&1 | tail had eaten the compiler's exit status, and a stale binary got flashed. The actual failure:

main/leds.cpp:437:48: error: 'void* memcpy(void*, const void*, size_t)'
writing to an object of type 'struct CRGB' with no trivial copy-assignment;
use copy-assignment or copy-initialization instead [-Werror=class-memaccess]

Since then, no build gets piped without checking its exit status.

Fig 4: the pacing story in one picture. Baseline is bimodal with a heavy skipped-frame mode at ~33 ms; the final build is a single spike at the 16.67 ms budget.

Finding 3: the sparkles were always there

Right after the pipeline change, we looked at the physical strip and noticed RGB sparkles that suggested a timing issue.

Forensics on the captures found the signature: pure single-channel, single-bit 0-to-1 pixel flips. One color channel of one pixel jumps by exactly one high bit weight (32, 64, or 128) for exactly one frame. The flipped bit positions concentrated in the top three bits (255, 384, and 398 flips respectively), which is why they read as bright sparkles. A simultaneous source-versus-wire capture put the corruption on the wire side: the Pi's UDP stream was clean.

Then the uncomfortable part. Running the flip detector across every capture of the day:

frames.bin    baseline (WS2813, 2 DMA bufs, morning)   frames= 168 bitflip-sparkles= 369 rate=2.223/frame
frames2.bin   WS2812 + 4 bufs (truncation bug)         frames= 170 bitflip-sparkles= 128 rate=0.762/frame
frames3.bin   + stop countdown                         frames= 198 bitflip-sparkles= 161 rate=0.821/frame
frames4.bin   + lwIP pinned (visually approved)        frames= 223 bitflip-sparkles= 251 rate=1.136/frame
frames6.bin   pipeline in UDP task (4s)                frames= 231 bitflip-sparkles=   6 rate=0.026/frame
frames7.bin   pipeline in UDP task (8s)                frames= 413 bitflip-sparkles=1037 rate=2.523/frame

The bug was in the morning baseline, before any of the day's changes. It was a longstanding fault surfaced by scrutiny, not a regression. And the rate wanders wildly on identical firmware: 0.026 and 2.523 flips per frame twenty minutes apart on the same build. That is exactly why every visual check had ever passed. It was content-independent too: correlation between frame luminance and flips was -0.056, flips were uniform across all 480 pixel positions, and they arrived at a steady 125-165 per second.

On the wire, the corruption looked like this:

Fig 3: HIGH pulse widths over 8 s captures. The before build has a tail of 0.5-2.5 µs stuck-high pulses; legal widths top out at 1 µs. An LED reads a stretched high as extra '1' bits: a sparkle.

The wrong hypothesis, tested and disproven

First theory: ISR latency. If EOF interrupts coalesce, the buffer-fill position races the DMA and replays stale data. We flashed a resync (refill position re-derived from the DMA's own EOF descriptor address) plus 8 DMA buffers for slack, and measured. The corruption barely moved: still 4,097 anomalous pulses, 1.714 flips per frame versus 2.523. Pacing got worse (26% late, the +0.4 ms buffer cost from above). Hypothesis dead. The resync was kept as cheap robustness, the mechanism was not the cause, and we went back to 4 buffers. Measure before believing a mechanism.

The real cause: the I2S driver's DMA arithmetic

The I2S parallel driver quantizes each 1.25 µs bit into time slices of duration GCD(T1, T2, T3), one 32-bit DMA word per slice carrying all six lanes in parallel. WS2812 timing is 250/625/375 ns; its GCD is 125 ns, so every bit is 10 slices, and at 8 M slices per second times 4 bytes that is a sustained 32 MB/s DMA read stream for the whole ~14.4 ms transmission. Six UDP packets arrive mid-transmission every frame, and their Ethernet DMA bursts intermittently won memory-bus arbitration. When the I2S FIFO underruns, the output holds its last level: a '1' bit's high stretches until the DMA catches up. That is the 0.5-2.5 µs stuck-high pulse, and the LED reads it as extra '1' bits.

The fix is arithmetic, not heroics: custom timing 250/500/500 ns (T0H = 250 ns, T1H = 750 ns, still 1.25 µs per bit, within WS2813 datasheet tolerance). The GCD becomes 250 ns, so 5 slices per bit and 16 MB/s, half the bus pressure for a wire waveform the LED cannot tell apart.

Fig 2: the whole fix. Same 1.25 µs bit on the wire; half the DMA words to feed it.

Measured result over an 8 s capture (committed as a3c0109):

anomalous pulses (outside expected 0/1 widths): 9        <- was 4097
frames=478 bitflip-sparkles=0 rate=0.000/frame (was 2.52 / 1.71)
median interval 16.65ms (60.08fps), late intervals 1/477

Out-of-spec pulses 4,097 to 9. Sparkles 2.5 per frame to 0.00. And pacing improved to 60.08 fps median with 1 late interval in 477, because the underruns had been stalling the DMA and eating frame time too. One root cause, two symptoms.

Finding 4: proof by soak

One clean 8 s capture is an anecdote, and this bug had already taught us how it hides. So: probes on all outputs, and captures back to back.

First a sync check across lanes: D1, D2, D3, D5, D6 each carried exactly 5,575,296 edges and 241 frames, with frame starts offset by at most 1 sample (125 ns). One I2S engine really does clock every strip (the sixth strip's probe never made contact all night; given the shared engine, five lanes are representative).

Then the soak: 24 captures x ~11 s over ~18 minutes, roughly 70% wire-time coverage on 5 lanes. Result: 0 sparkle flips in ~392,000 lane-frames, median interval 16.65 ms, ~0.5% late.

One iteration out of 24 showed millisecond-scale garbage, and it is worth recording why we excluded it with confidence: its lanes disagreed (436 flips on D1, 200 on D5, different anomaly spectra). A real fault in the shared I2S engine hits all lanes identically, so disagreeing lanes are a probe-handling artifact, and the two follow-up captures were clean while the device heartbeat never blinked. Same logic caught a fallen-off D1 probe later: continuous hash with edges during the inter-frame dead time, when the engine is provably idle. These probe-artifact signatures are cataloged in pfx-firmware/docs/logic-analyzer.md.

The residual ~0.5% late frames are upstream: a 13-minute watch on the Pi showed the engine itself sends 0.4% of its intervals over 25 ms (188 of 46,800, worst ~49 ms). Wire-side lates match the engine's own hiccups, so the firmware side of the pacing story is closed. (The engine hiccup is a separate hunt, still open, in the Pi codebase.)

Eight days later: the durability check

While assembling this post (2026-07-17) we recaptured raw data on current main-line firmware, several merges after the session: one probe, 8 s, 64 M samples, a busy pattern with 15.6% one-bits and 409 of 480 pixels lit.

D6: frames=479 flips=2 anom=41 [125ns x8 500ns x31 625ns x2] med=16.67ms late=0/478

479 frames, median 16.670 ms (59.99 fps), 0 late of 478 intervals (interval std 177 µs, worst 18.4 ms), and zero out-of-spec pulses inside the real 480 pixels of all 479 frames. The fixes held.

And the flips=2? Detector false positives, and a useful lesson. Both hits are in one frame, on adjacent pixels 354 and 355, red going 3 to 36 to 3, with pixels 353 and 356 also lit at 36: a fast-moving one-frame-wide pattern feature, not corruption. The delta of 33 happens to have two bits set, which satisfied the detector's "≤2 bits changed" rule. A real sparkle is an isolated pixel backed by an out-of-spec pulse on the wire; on busy fast patterns the detector needs that isolation check or wire-level corroboration. The pixel-level detector flagged it, the wire acquitted it.

What we changed

• WS2812-class timing and 4 DMA buffers, with an end-of-frame stop countdown so the tail is never truncated (0e527da).
• lwIP's tcpip task pinned to core 0, away from the LED core (0e527da).
• Pixel mapping and gamma moved into the UDP task, applied per packet while the previous frame streams; renderFrame is a memcpy, 1.19 ms to 24 µs (a3c0109).
• Custom PoleFXController timing 250/500/500 ns: 5 DMA slices per bit instead of 10, halving I2S DMA bandwidth from 32 MB/s to 16 MB/s, which is the sparkle fix (a3c0109).
• The analyzer recipes, decoding rules, and anomaly catalog are committed in pfx-firmware/docs/logic-analyzer.md and tools/analyzer/analyze_capture.py; the running status log is VALIDATION_PLAN.md.

Rules we keep

1. The wire is ground truth. Visual checks passed a 2.2-flips-per-frame bug for the life of the product, because the rate wanders (0.026 to 2.523 per frame on the same build, 20 minutes apart).
2. Count the bits. A frame shorter than 11,520 bits is the number one red flag; that count is what caught our own truncation bug.
3. Measure before believing a mechanism. The ISR-latency theory was plausible, cheap to believe, and wrong; the measurement said so in one capture.
4. Never pipe a build without checking the exit status. A masked compile error means measuring a stale binary and drawing conclusions from it.
5. On I2S parallel, pick LED timing constants with a coarse GCD. The GCD sets the DMA slice rate; finer granularity buys nothing on the wire and doubles memory-bus pressure.
6. Know the probe-artifact signatures. Lanes disagreeing means handling; edges during the inter-frame dead time mean a floating probe. A real shared-engine fault hits every lane identically.
7. A flip detector needs corroboration. Isolated pixel plus out-of-spec pulse is a sparkle; a moving pattern feature can fake the pixel half of that signature.

The full interactive dashboard from the session, with the waveform viewer and every histogram, is preserved here: forensics-dashboard.html.

All numbers in this post are measured: from the 2026-07-09 bench session (13 capture rounds), the recovered forensics dataset, or the 2026-07-17 recapture. The one estimated figure is the price of the logic analyzer.