Chasing 60 fps down the LED wire with a $6 logic analyzer

By Spencer and Claude. Claude did the writing; the material is Spencer's engineering notes, measurements, and repositories. Nothing here has been independently fact-checked, so accuracy is not guaranteed — read it as a lab notebook, not a datasheet.

Per-pixel decoding of our LED data lines with a generic FX2 logic analyzer: a refactored firmware that turned out to deliver 42.5 of the engine's 60 fps, five measured builds to close the gap, and an RGB sparkle bug that predated the whole session. Includes one self-inflicted bug, one disproven hypothesis, and an 18-minute soak, about 15,000 frames per lane, to test the fixes.

Why

Our poles are driven by an ESP32 receiving 60 fps frames over Ethernet from a Raspberry Pi and shifting them out to six channels of 480 SK6813-mini pixels through FastLED-idf's I2S parallel driver (the same hardware measured electrically in the power post; the firmware drives the strips within WS2813 timing spec). We had just rewritten the firmware's frame pipeline: event-driven, triple-buffered, direct task notifications, replacing a polling design with silent packet-drop races. The refactor had never been hardware-tested, and our only acceptance test was "the LEDs look fine."

"Looks fine" is not a measurement. A generic FX2 logic-analyzer clone, about $3 on Alibaba or $6 to $12 on Amazon if you would rather have it this week, clipped to the data lines puts the wire itself on the record: every bit of every pixel of every frame, timestamped at 125 ns resolution. So we asked four questions:

1. Does every frame the Pi sends actually reach the LEDs: any stale, repeated, or torn frames?
2. What frame rate hits the wire, and how is it paced?
3. Where does the 16.67 ms frame budget actually go, and can we hit 60 fps?
4. Is the waveform itself in spec, pulse by pulse?

The answers, in order: yes-but, 42.5 fps in bursts, mostly one bad memcpy, and no, and that "no" turned out to be a bug older than the day's work.

The bench

Hardware under test: the "Top Support" bench controller (ESP32, ESP-IDF v4.4.8), fresh refactor branch, driving 6 x 480 px strips. Source: the Pi engine multicasting 240x12 frames at 60 fps over UDP, 6 packets x 1,440 bytes per frame. Instrument: a generic Cypress FX2 logic analyzer, driven by sigrok-cli/fx2lafw at 8 MHz x 8 channels, probes clipped to the LED data lines:

sigrok-cli --driver fx2lafw --config samplerate=8m --samples 100m -o cap.sr

Decoding is simple enough to do with numpy (the tooling is now committed as pfx-firmware/tools/analyzer/analyze_capture.py): each bit is 1.25 µs, a high longer than 500 ns is a '1', a low longer than 50 µs is a frame gap, and a frame is 480 px x 24 = 11,520 bits per lane. At 8 MHz sampling each legal high lands in one of two adjacent 125 ns bins, 250 or 375 ns for a '0' and 750 or 875 ns for a '1', because the FX2's clock and the ESP32's are free-running against each other and an individual pulse reads one sample long or short. So the test for a generation glitch is not "is it exactly 250 or 750", it is "can one sample of rounding explain this width".

Fig 1: real captured data. Top: all lanes start within 125 ns of each other, direct wire proof that one I2S peripheral clocks every strip. Bottom: single bits resolved and measured, 250 ns highs for '0', 750 ns for '1'.

Instrument gotchas worth passing on:

• It buffers in RAM: ~100-128 M samples max, so 12-16 s per capture at 8 MHz. Long soaks are capture loops, not one capture.
• It frequently stops early ("Device only sent 91396608 samples."), which is harmless, and occasionally wedges completely: 0 edges on all 8 channels while the device under test demonstrably streams. Replug USB.
• sigrok uploads its own firmware into the FX2's RAM per session, so the vendor app still works after a replug.

Finding 1: the refactor works, but only 42.5 fps reaches the wire

First capture of the day, stock WS2813 timing (320/320/640 ns), 2 DMA buffers. The event pipeline itself passed cleanly: 170 of 170 frame payloads unique, zero repeated stale frames, zero tearing.

The pacing did not pass:

frame intervals (us): n=169 median=17547.8 mean=23593.1 std=7835.91
equivalent fps (median): 56.99
intervals >1.5x median (likely dropped/late frames): 69   (of 169 = ~41%)

170 frames in a 4 s capture is 42.5 fps average. The interval histogram is bimodal, ~17 ms and ~33 ms, and the 33 ms mode is a missed frame period. The skips came in bursts, which is why the strip still looked smooth-ish.

Meanwhile a raw packet sniff on the Pi's interface showed the engine sending a clean 60.2 fps with zero frame-number skips. The deficit was entirely in the firmware, and the budget math said why: show() for 480 pixels took ~14.85 ms at stock timing, plus ~2.5 ms of loop overhead, against a 16.67 ms budget. There was no budget left, and an unpinned priority-18 lwIP task added jitter on top.

Finding 2: five builds to 60 fps, each measured before the next change

Fig 5: the day's builds. Average fps on the wire (top) and skipped or late frame periods (bottom).

One caveat on reading that as a controlled sequence: build #1 moved two things at once, the bit timing and the buffer count, so its +9.5 fps cannot be split between them. #2 through #5 are one change each, measured before the next went in.

Three of these steps deserve their stories told honestly.

The self-inflicted bug (#1 to #2)

Raising the DMA buffer count from 2 to 4 with an all-prefilled circular link made the end-of-frame semaphore fire while three real buffers were still queued, so i2sStop() cut the tail off every frame. Nothing visibly wrong on the strip; the analyzer caught it because it counts bits:

bits-per-frame histogram bottom: 11480: 2, 11481: 3, 11504: 12, 11505: 5, ...

Frames up to 40 bits (~1.7 pixels) short of 11,520. The fix is a stop countdown (gStopCountdown = NUM_DMA_BUFFERS - 1 before releasing the semaphore); after it, minimum frame transmission duration jumped from 2,054 µs to 14,470 µs, meaning no more truncated tails. A frame shorter than 11,520 bits remains the number one red flag in any future capture.

While here we measured DMA buffer economics: each extra buffer costs ~30 µs per frame in prefill and stop countdown. A later experiment with 8 buffers measured +0.4 ms per frame and 26% late frames. 4 is the sweet spot, and the comment now lives at NUM_DMA_BUFFERS in the driver.

The big one (#4): move work off the render path

Temporary PROFILE_FRAME_TIMING instrumentation printed the budget over serial:

FRAMETIMING n=300 map=1194/1201 show=15175/15670 wait=2062/17115 cycle=18434/33293 (avg/max us)

map+copy 1.19 ms, show() 15.18 ms average and 15.7 ms max: ~0.3 ms of margin against 16.67 ms. That margin is the residual skip bursts. The fix: process_packet now applies a precomputed per-packet pixel map plus gamma as each UDP packet arrives, while the previous frame is still streaming out of the I2S DMA, and renderFrame collapses to a memcpy. Measured after:

FRAMETIMING n=300 map=24/32 show=14891/15623 wait=1862/18145 cycle=16777/33057 (avg/max us)

1,190 µs to 24 µs, a 50x cut, and the wire went to 60.0 fps. (We also asked whether dropping gamma correction would help: no. It rides along in the same per-packet pass, so it is free.)

The build-hygiene incident

One flash of "the fix" changed nothing on the wire. The reason was not firmware: idf.py build 2>&1 | tail had eaten the compiler's exit status, and a stale binary got flashed. The actual failure:

main/leds.cpp:437:48: error: 'void* memcpy(void*, const void*, size_t)'
writing to an object of type 'struct CRGB' with no trivial copy-assignment;
use copy-assignment or copy-initialization instead [-Werror=class-memaccess]

Since then, no build gets piped without checking its exit status.

Fig 4: the pacing story in one picture. Baseline is bimodal with a heavy skipped-frame mode at ~33 ms; the final build is a single spike at the 16.67 ms budget.

Finding 3: the sparkles were always there

Right after the pipeline change, we looked at the physical strip and noticed RGB sparkles that suggested a timing issue.

Forensics on the captures found the signature: pure single-channel, single-bit 0-to-1 pixel flips. One color channel of one pixel jumps by exactly one high bit weight (32, 64, or 128) for exactly one frame. The flipped bit positions concentrated in the top three bits (255, 384, and 398 flips respectively), which is why they read as bright sparkles. A simultaneous source-versus-wire capture put the corruption on the wire side: the Pi's UDP stream was clean.

Then the uncomfortable part. Running the flip detector across every capture of the day:

frames.bin    baseline (WS2813, 2 DMA bufs, morning)   frames= 168 bitflip-sparkles= 369 rate=2.223/frame
frames2.bin   WS2812 + 4 bufs (truncation bug)         frames= 170 bitflip-sparkles= 128 rate=0.762/frame
frames3.bin   + stop countdown                         frames= 198 bitflip-sparkles= 161 rate=0.821/frame
frames4.bin   + lwIP pinned (visually approved)        frames= 223 bitflip-sparkles= 251 rate=1.136/frame
frames6.bin   pipeline in UDP task (4s)                frames= 231 bitflip-sparkles=   6 rate=0.026/frame
frames7.bin   pipeline in UDP task (8s)                frames= 413 bitflip-sparkles=1037 rate=2.523/frame

The bug was in the morning baseline, before any of the day's changes. It was a longstanding fault surfaced by scrutiny, not a regression. And the rate wanders wildly on identical firmware: 0.026 and 2.523 flips per frame twenty minutes apart on the same build. That is exactly why every visual check had ever passed. It was content-independent too: correlation between frame luminance and flips was -0.056, flips were uniform across all 480 pixel positions, and they arrived at a steady 125-165 per second.

On the wire, the corruption looked like this:

Fig 3: HIGH pulse widths over 8 s captures. The before build has a tail of 0.5-2.5 µs stuck-high pulses; legal widths top out at 1 µs. An LED reads a stretched high as extra '1' bits: a sparkle.

The wrong hypothesis, tested and disproven

First theory: ISR latency. If EOF interrupts coalesce, the buffer-fill position races the DMA and replays stale data. We flashed a resync (refill position re-derived from the DMA's own EOF descriptor address) plus 8 DMA buffers for slack, and measured. The corruption barely moved: 1.714 flips per frame versus 2.523, and 4,097 anomalous pulses against the 3,057 in the build it was meant to improve on, so slightly worse on pulses. Pacing got worse (26% late, the +0.4 ms buffer cost from above). Hypothesis dead. The resync was kept as cheap robustness, the mechanism did not explain the sparkles, and we went back to 4 buffers. Measure before believing a mechanism.

The mechanism we settled on: the I2S driver's DMA arithmetic

The I2S parallel driver quantizes each 1.25 µs bit into time slices of duration GCD(T1, T2, T3), one 32-bit DMA word per slice carrying all six lanes in parallel. WS2812 timing is 250/625/375 ns; its GCD is 125 ns, so every bit is 10 slices, and at 8 M slices per second times 4 bytes that is a sustained 32 MB/s DMA read stream for the whole ~14.4 ms transmission. Six UDP packets arrive mid-transmission every frame, and their Ethernet DMA bursts contend for the same memory bus. Our reading is that the I2S FIFO intermittently lost that fight and underran; when it does, the output holds its last level and a '1' bit's high stretches until the DMA catches up. That is the 0.5-2.5 µs stuck-high pulse, and the LED reads it as extra '1' bits.

We never caught an underrun in the act. The driver exposes no FIFO status we were reading, we ran no Ethernet-off control, and we never swept bus load with the wire timing held fixed. What we have is a mechanism that predicts the exact waveform signature we were seeing and a fix derived from it that removed the signature. That is good evidence and it is not a unique identification.

The fix is arithmetic, not heroics: custom timing 250/500/500 ns (T0H = 250 ns, T1H = 750 ns, still 1.25 µs per bit, within WS2813 datasheet tolerance). The GCD becomes 250 ns, so 5 slices per bit and 16 MB/s, half the bus pressure for a wire waveform the LED cannot tell apart.

The bandwidth story does not explain everything in the census above, and it is worth saying so before the good news. The morning baseline ran WS2813 timing, 320/320/640 ns, whose GCD is 320 ns: 4 slices per bit and 12.5 MB/s, less bus pressure than the 16 MB/s we ended up choosing, and it sparkled at 2.22 flips per frame. It also ran 2 DMA buffers rather than 4, which leaves the refill interrupt about one pixel-time of slack instead of three. So there are at least two candidate ways to starve that FIFO, too little bandwidth headroom and too little time headroom, and we fixed the one our measurements pointed at, on a build that already had 4 buffers. (That 12.5 MB/s is computed from the driver's arithmetic, not measured on the bench; the 2.22 flips per frame is measured.)

Fig 2: the whole fix. Same 1.25 µs bit on the wire; half the DMA words to feed it.

Measured result over an 8 s capture (committed as a3c0109):

anomalous pulses (outside expected 0/1 widths): 9        <- was 4097
frames=478 bitflip-sparkles=0 rate=0.000/frame (was 2.52 / 1.71)
median interval 16.65ms (60.08fps), late intervals 1/477

Read the tool's was 4097 carefully, because that label is the easiest thing in the whole capture to misquote: 4,097 is the count from the 8-buffer resync build it had been comparing against, not from the stock-WS2812 capture plotted in Fig 3. On that capture the part of the damage no decoding rule can argue with is the tail past 1 µs: 607 pulses that no legal width plus a sample of rounding can reach. After the fix that tail is empty, and the nine anomalies left are all within one sample of a legal bin, four at 125 ns and four at 500 ns, plus the usual stray high in the post-frame dead zone. Sparkles 2.5 per frame to 0.00.

Pacing improved too, to 60.08 fps median with 1 late interval in 477. We read that as the same root cause showing a second symptom: the underruns had been stalling the DMA and eating frame time as well as corrupting pixels. There is a duller explanation sitting right next to it, though. Half the slices is also half the bit-expansion work in the refill interrupt, and that interrupt runs on the CPU inside the DMA window. We never separated the two.

Finding 4: the soak

One clean 8 s capture is an anecdote, and this bug had already taught us how it hides. So: probes on all outputs, and captures back to back.

First a sync check across lanes: D1, D2, D3, D5, D6 each carried exactly 5,575,296 edges and 241 frames, with frame starts offset by at most 1 sample (125 ns). One I2S engine really does clock every strip. The sixth strip's probe never made contact all night, so five lanes stand in for six: fine for anything the shared engine does, and no evidence at all about that lane's own pin and wiring. (All six were finally on probes together on 2026-07-31, and they still start on the same sample; that capture is Fig 1 of the companion post.)

Then the soak: 24 captures x ~11 s over ~18 minutes, roughly 70% wire-time coverage on 5 lanes. Result: 0 sparkle flips in ~74,000 lane-frames, about 15,000 frames per lane, median interval 16.65 ms, ~0.5% late.

Count that honestly, because the unit flatters itself. Lane-frames are not independent trials: all five lanes come out of one I2S engine on one shared 32-bit DMA word, so what the soak really watched is ~15,000 transmissions, each observed five times. Five correlated views of fifteen thousand events, not seventy-four thousand events.

One iteration out of 24 showed millisecond-scale garbage. Counted inclusively, the raw census is 1,714 flips in 78,480 lane-frames, and every single one of those flips is inside that one iteration. We excluded it because its lanes disagreed, and not subtly: 58,256 anomalous pulses on D1 against 430 on D3, D1's widths running out past 4 ms, flips of 436 / 344 / 253 / 200 / 481. Two follow-up captures were clean and the device heartbeat never blinked. Two honest caveats on that. The rule was applied after looking rather than declared before, which is the weaker order. And the rule is not "lanes must agree exactly": a shared-engine fault can surface on a subset of lanes when the corrupted pixels happen to carry identical content elsewhere, and probe loading alone biases lanes against each other by a few tenths of a percent. What a shared 32-bit word cannot do is put 58,256 anomalous pulses on one lane and 430 on its neighbor. Same logic caught a fallen-off D1 probe later: continuous hash with edges during the inter-frame dead time, when the engine is provably idle. These probe-artifact signatures are cataloged in pfx-firmware/docs/logic-analyzer.md.

The residual ~0.5% late frames look upstream: a 13-minute watch on the Pi showed the engine itself sends 0.4% of its intervals over 25 ms (188 of 46,800, worst ~49 ms), and the wire's own late threshold, 1.5x a 16.65 ms median, works out to the same 25 ms. That is two aggregate rates agreeing at a matched threshold, not late outputs matched to late sends: the Pi watch ran alongside the captures rather than timestamp-locked to them. Enough to stop us blaming the firmware, not enough to call the pacing story closed. (The engine hiccup is a separate hunt, still open, in the Pi codebase.)

Eight days later: the durability check

While assembling this post (2026-07-17) we recaptured raw data on current main-line firmware, several merges after the session: one probe, 8 s, 64 M samples, a busy pattern with 15.6% one-bits and 409 of 480 pixels lit.

D6: frames=479 flips=2 anom=41 [125ns x8 500ns x31 625ns x2] med=16.67ms late=0/478

479 frames, median 16.670 ms (59.99 fps), 0 late of 478 intervals (interval std 177 µs, worst 18.4 ms), and zero out-of-spec pulses inside the real 480 pixels of all 479 frames. The fixes held.

And the flips=2? Detector false positives, and a useful lesson. Both hits are in one frame, on adjacent pixels 354 and 355, red going 3 to 36 to 3, with pixels 353 and 356 also lit at 36: a fast-moving one-frame-wide pattern feature, not corruption. The delta of 33 happens to have two bits set, which satisfied the detector's "≤2 bits changed" rule. A real sparkle is an isolated pixel backed by an out-of-spec pulse on the wire; on busy fast patterns the detector needs that isolation check or wire-level corroboration. The pixel-level detector flagged it, the wire acquitted it.

What we changed

• WS2812-class timing and 4 DMA buffers, with an end-of-frame stop countdown so the tail is never truncated (0e527da).
• lwIP's tcpip task pinned to core 0, away from the LED core (0e527da).
• Pixel mapping and gamma moved into the UDP task, applied per packet while the previous frame streams; renderFrame is a memcpy, 1.19 ms to 24 µs (a3c0109).
• Custom PoleFXController timing 250/500/500 ns: 5 DMA slices per bit instead of 10, halving I2S DMA bandwidth from 32 MB/s to 16 MB/s, which is what removed the sparkles (a3c0109).
• The analyzer recipes, decoding rules, and anomaly catalog are committed in pfx-firmware/docs/logic-analyzer.md and tools/analyzer/analyze_capture.py; the running status log is VALIDATION_PLAN.md.

Rules we keep

1. The wire outranks the eye. Visual checks passed a 2.2-flips-per-frame bug for the life of the product, because the rate wanders (0.026 to 2.523 per frame on the same build, 20 minutes apart).
2. Count the bits. A frame shorter than 11,520 bits is the number one red flag; that count is what caught our own truncation bug.
3. Measure before believing a mechanism. The ISR-latency theory was plausible, cheap to believe, and wrong; the measurement said so in one capture.
4. Never pipe a build without checking the exit status. A masked compile error means measuring a stale binary and drawing conclusions from it.
5. On I2S parallel, pick LED timing constants with a coarse GCD. The GCD sets the DMA slice rate; finer granularity buys nothing on the wire and doubles memory-bus pressure.
6. Know the probe-artifact signatures. Lanes disagreeing by orders of magnitude means handling; edges during the inter-frame dead time mean a floating probe. A shared-engine fault can still show up on a subset of lanes, so judge by the size of the disagreement, not by exact agreement.
7. A flip detector needs corroboration. Isolated pixel plus out-of-spec pulse is a sparkle; a moving pattern feature can fake the pixel half of that signature.

The full interactive dashboard from the session, with the waveform viewer and every histogram, is preserved here: forensics-dashboard.html.

Numbers in this post are measured unless labeled otherwise: from the 2026-07-09 bench session (13 capture rounds), the recovered forensics dataset, or the 2026-07-17 recapture. The two exceptions are flagged where they appear, the DMA read rates (computed from the driver's own arithmetic) and the analyzer price (a market price: about $3 on Alibaba, $6 to $12 on Amazon).