Two cores is not the same as at the same time

By Spencer and Claude. Claude did the writing; the material is Spencer's engineering notes, measurements, and repositories. Nothing here has been independently fact-checked, so accuracy is not guaranteed — read it as a lab notebook, not a datasheet.

Moving one render loop onto the ESP32's second core took the pole's onboard animations from 56 to 67 fps. Why that worked is the part worth writing down, because we had already built something that looked like the same trick years earlier, on a different path, and it had never overlapped anything. Also here: two GPIO pins that turn a logic analyzer into a CPU profiler, our own tooling insisting that a board paced to 60 fps was running at 58.8, and two builds of identical code benchmarking 10.7% apart.

Why the pole renders anything at all

The last analyzer post put probes on the LED data lines to find out whether a rewritten frame pipeline was really delivering the 60 fps the Pi was feeding it. It was delivering 42.5. This one is about the other path entirely: what the pole does when no frames are arriving at all.

Every pole also carries its own pattern set, generative animations that run on the ESP32 with no controller involved. They take over whenever the Pi is not driving, which happens more often than it sounds:

• while the Pi boots
• between sets, when nobody is at the controls
• during a pattern upload or an engine restart
• when someone unplugs something, which they will
• when the Pi dies mid-show

So "the onboard renderer looks worse than the real thing" is not a footnote. It is the failure mode, and it was running at 52 to 56 fps against a 60 fps target.

The part that is physics

Six parallel strips, 480 pixels each, 2,880 LEDs total. The LED protocol spends 1.25 microseconds per bit, so one frame is 480 x 24 x 1.25us = 14.40 ms of wire time, and no processor makes that smaller. Measured end to end, pushing a frame blocks for 14.825 ms.

A 60 fps frame budget is 16.67 ms. Subtract the wire and you have 1.85 ms left for all the pattern maths. The cheapest pattern in the set cost more than that.

That is the whole problem in one line, and it is not something you optimize your way out of one pattern at a time. Compute and transmission were happening one after the other:

frame = compute + transmit

Two pins that turn a logic analyzer into a profiler

Pointed at the LED data lines, an analyzer shows the output: frame period, wire time, corrupted bits. What it cannot show is what the processor is doing, and the single most expensive thing in the driver is invisible there.

The routine that repacks pixel data for the LED hardware runs inside the transmission window, in the gaps between the hardware asking for more bytes. On the data lines that looks exactly like bits flowing normally, whether the routine is using 10% of the available time or 95%. We had no idea which.

So the firmware drives two otherwise-unused pins high while it works and low when it stops. A pin held high for 3.3 ms is a measurement of 3.3 ms. Note what that is a measurement of: the wall interval from the start of the job to the end of it, with anything that preempted the job in the middle still inside the span. It is not a cycle count.

There were no spare pins, so we borrowed two of the six LED outputs and those two strips go dark. But we never told the firmware it had four strips: it kept doing all six strips of work, and the outputs were detached at the last moment inside the chip's GPIO matrix. Dropping to four would have made the repacking about a third lighter, and we would have carefully measured a machine easier than the one we ship.

The probe also has to be nearly free, which it is: one instruction writing straight to a hardware register, a clock cycle or two, against jobs lasting microseconds to milliseconds.

What we had actually built

Overlap needs two things at once: work split across tasks that do not block each other, and somewhere for the producer to put its output that the consumer is not currently reading. We had each of those, in different places, and never both together.

The streaming path already had the buffers. Frames arriving from the Pi go through a triple buffer: the UDP receiver assembles frame N+1 into one slot while the LED task pushes frame N out of another. That part was built right years ago.

Both of those tasks are pinned to core 1, and that was deliberate. The comment above them still says why: the I2S interrupt is allocated on whichever core first calls show(), so putting the LED task on core 1 lands the interrupt there, away from the network stack on core 0. Core 1 got reserved for LED output and frame reception, core 0 for housekeeping. Given what that interrupt does (it fires once per pixel, 480 times a frame) keeping it away from lwIP is a reasonable thing to want.

The cost of that choice is the part I had not thought through. Two tasks on one core interleave: while the LED task sits blocked waiting on the DMA to drain, the UDP receiver gets the processor. That is genuinely useful, and it is not the same thing as two jobs running at the same time. The buffers were never the limit. The core count was, and the core count was set for an unrelated and good reason.

The onboard path had neither. One task computed the frame and then pushed it, in sequence, on one core. No second buffer to fill, and nothing else to run.

It is a two-by-two, and only one square actually overlaps:

Both squares in the top row shipped, and between them they covered the whole product: the onboard path sequential, the streaming path interleaving. Neither overlaps. The bottom-left square we never built at all, which is just as well, because it is the one that looks most like parallelism from the outside and behaves the least like it.

Moving into the bottom right

The onboard path needed both halves at once: a second core to run on, and a buffer of its own to write into.

frame = max(compute, transmit)      only once nobody is waiting on a buffer

The renderer fills a shadow buffer on core 0. The LED task copies it out in about 30 microseconds, hands the shadow straight back, and only then starts the 14.8 ms transmit, so the renderer is already working on the next frame before the current one reaches the wire.

That does mean putting the renderer on core 0, the core deliberately kept clear of LED work. It is safe for a reason specific to this job: the onboard renderer only runs when the Pi is not sending frames, and when the Pi is not sending frames core 0 has almost nothing to do (a heartbeat twice a second, a version line every ten). The moment frames start arriving the renderer stands down and the streaming path takes over unchanged. The two workloads are mutually exclusive, so they never actually compete.

That copy buys something else worth having: the blocking transmit call stays exactly as it was, so the vendored LED driver needed no fork. An earlier design note had rejected double-buffering here because it would mean making the driver's push return early. True when the producer is on the same core, where the only way to overlap is to stop blocking. It stops being true once the producer is somewhere else and hands over a finished buffer.

Both traces come from the same source tree, one compile flag apart. Green inside blue is overlap. Green in the gap is taking turns.

The zoom panel matters more than it looks. At full-frame scale the interrupt's 480 pulses per frame are smaller than a screen pixel and alias into about thirty fat bars, which reads as "it fires thirty times a frame". Zoomed in they resolve properly, and compute stays high right across them: core 0 working while core 1 pushes bits.

Whether any of this is obvious from reading the source, I genuinely do not know. The information is all in there, in which tasks exist, which cores they are pinned to, and how many buffers sit between them. But every square of that table looks alike in code. All of them are concurrent, and concurrent is what you are checking for when you go looking. The frame rate is one number and it does not tell you which one you built. On the wire there is nothing left to interpret.

The numbers that came with that picture:

Transmit time identical to four decimal places is the load-bearing number. It says the gain came from overlapping work with the wire and not from anything happening to the push itself.

The surprise is the last row. Compute got 8.6% more expensive. The two stages never contend for processor time now, and the mechanism that fits is that they still contend for memory bandwidth, because the compute runs while the DMA engine is streaming. We had modelled the stages as independent once they were on separate cores, and the memory bus does not care about that separation.

Read that as the leading explanation rather than a measured one. The A/B is one tree behind one compile flag, but a compile flag still produces a different binary, and the section below shows two behaviorally identical builds landing 10.7% apart on this same number. 8.6% sits inside that. Pinning it on the memory bus would take a bandwidth counter or a layout-preserving control, and we ran neither. What is not in doubt is the trade: 0.23 ms of compute bought 3.35 ms of frame period. Worth it, but not free, and "free" is what the plan had assumed.

Then we paced it back down, and the tool lied about it

67 fps is past the target, and a declared rate is more useful than a fast one: it leaves headroom, and it gives future expensive patterns something to be measured against. Pacing is a deadline accumulator. Work out when the next frame is due, sleep until then, and add the nominal period to the deadline rather than to now, so error cannot accumulate.

It worked. The measurement tool reported 58.8 fps.

The scheduler tick is 1 millisecond and 16.667 is not a whole number of them, so no single sleep can express the target. The accumulator instead lands the mean on it and lets individual periods quantize to the 16 and 17 ms ticks: 115 intervals in the 16 ms bucket, 223 in the 17 ms bucket, and 21 that landed in neither. Those bucket labels are nominal, so the mean is not something you can reconstruct from the two counts. Measured over the capture it is 16.6695 ms, which is 59.990 fps, about 0.02% off the 60.0 target. (An earlier reduction of this capture said 16.645 ms and 0.13% fast; that mean averaged in the frame the capture window clipped at t=0, whose truncated burst manufactured a spurious 8 ms interval. The raw edge list was re-derived on 2026-08-05 with the artifact excluded, identically on all three LED lanes.) The deviation from target over those 359 real intervals is 0.0028 ms, which says the accumulator is unbiased on this build; six seconds of frames says nothing about oscillator drift or missed frames over a night.

Our tool derived frames per second from the median. The median of a two-valued distribution is one of the two values: 62% of the samples sit in the upper spike, so the median is pinned to 17.0 ms and 58.82 fps no matter how well the pacing works. You could make the accumulator perfect and the median would not move.

"Take the median, it is robust" is good advice for noisy unimodal data and actively wrong for quantised bimodal data. For every unpaced build this bench had ever measured it was also the right call: those distributions are unimodal with a long tail, and a median throws out the occasional stalled frame that would drag a mean around. The advice is sound, for a shape this data no longer has. Nothing was broken except our reading of it, which is the kind of bug that survives a long time because every individual step looks correct.

The fix was to make the harness report the mean for paced output, and to write down why: for quantized data the mean is the physical throughput and the median is an artifact of the tick. That is a claim about one statistic; jitter, phase error, and missed deadlines are separate questions the mean answers none of. There was a tidier looking fix, a sub-tick wake from a hardware one-shot timer, which would have put the median on target too. We turned it down: it injects a spurious notification into the same channel the UDP receive path uses to signal a streamed frame, and buying a prettier statistic with a real interaction on the streaming path is a bad trade. The median is not a physical quantity here. There was nothing to fix.

The case that actually tests the claim

Overlapping only helps while compute fits inside transmission. The expensive case is a crossfade, where two patterns are evaluated for every pixel. Pinning two patterns with the shortest legal dwell forces one every three seconds.

Three clusters, not two. Each pattern's own dwell cost is separately visible, and the fade sits above both. The worst frame of that fade computes in 12.637 ms against a 14.825 ms ceiling, so it fits with 2.19 ms to spare and the frame rate does not move during a fade.

That is one pinned pair, caustics into ripple, over 208 frames. It is the pair our own notes already had down as the expensive one, which is why it was the pair we pinned, but 16 of the 17 patterns were still unmeasured at this point and no sweep had visited every combination. So it is the worst crossfade we have measured, not the worst the set can produce.

32% of frames land in the fade cluster against 33% predicted from one second in every three, which is a free check that the top cluster really is the crossfades and not something else.

Two more things the wire settled

After the real pixel data ends, the hardware keeps shifting out stale bytes until the software stops it. Those bits fall off the end of the strip and are cosmetically harmless, and the tail length had been assumed to be a driver constant. It is 28 bits when idle and 771 while the pole is also receiving network traffic, with traffic the only variable. That makes it contention between two tasks on the same core, not a property of the driver: the LED task and the UDP receiver, the same pair from earlier, taking turns on core 1 in a way that costs real microseconds.

We wrote that prediction down before taking the measurement, which is the only reason it counts for anything.

That same tail metric delivered a sequel two days later. It had always been unimodal, median around 771 bits. After the repack interrupt was made substantially cheaper, its median started flipping between 63 and 530 across consecutive runs of the same binary. An 8x swing on identical code is the signature of a broken rig, and that is what we assumed for an afternoon. It was the optimization working: the distribution had gone bimodal, roughly 30% of samples down at the new floor and 63% still high, and the median was teleporting between the humps depending on which side of the split the middle sample landed. Two metrics, two days apart, both misreported by their medians for different reasons, and in both cases the median looked stable while it lied. A mean would have been wrong in the second case too, so the lesson is not "prefer the mean." A summary statistic assumes a shape; quantized data breaks a median one way, bimodal data another, and neither failure shows up in the number. Both are obvious the moment you plot. Two habits came out of it: print percentiles either side of whatever you report, because p25 and p75 landing far apart is the cheapest tell that one number is hiding a shape, and when a metric swings hard between identical runs, do not reach for the rig first. A property of the code under test cannot vary that way, so either the cause is outside your system or your statistic is the wrong one.

Measure the same binary twice and the driver-side numbers reproduce to about 0.4%; pattern compute, the noisiest row, reproduces to about 1.4% in the controlled repeat. Measure two builds whose behavior is identical and they disagree by up to 10.7%, seven times the worst same-binary floor. Code layout moves cache alignment, and cache alignment moves everything, so a before-and-after against last week's binary is partly measuring where the compiler happened to put things.

That is not an abstract worry. Three apparent findings in one day turned out to be nothing else, and one of them we had already written up: "pipelining makes the repack routine 6.6% cheaper", complete with a mechanism that sounded right, that the second core stops thrashing the first core's instruction cache. Then we built a proper A/B from one tree behind a single compile flag. The effect was 2.6% in idle and 2.8% in streaming, and that near-equality is what killed it: during streaming the onboard renderer does not run at all, in either build, so there was no thrashing for the change to have stopped. Two numbers moving together like that point at the binaries differing by about a kilobyte, and nothing else.

The discipline that fixes it: put the change behind a compile flag so both arms come from one tree, and bake the flag's value into the binary so the artifact itself can tell you which arm it is. We now do the second part with a string the build verifier greps out of the firmware image, per source file.

Where it ended up

The 67.5 fps ceiling is mostly wire time, not processor time. Mostly, not entirely: the protocol's own 14.40 ms works out to 69.4 fps, and the blocking push measures 14.825 ms, so about 0.4 ms of the frame is driver overhead and stale tail rather than physics. That part is software and could in principle move. The 14.40 ms underneath it cannot, short of shorter strips or a faster LED protocol, and both are real options we have declined for now.

Everything underneath that ceiling turned out to be a question of which square we were standing in, and that was the one thing the source would not tell us.

Every figure here is generated from the raw analyzer captures by a committed script, and every number is computed from the same samples rather than transcribed from notes. Captures at 8 MHz across eight channels; the frame-period histograms hold 167, 201, and 359 intervals for the serial, free-running, and paced arms, after excluding intervals led by a frame the capture window clipped. Builds verified from the artifact rather than from the edit. The stale-tail median figures are from the same bench two days later. Measurements are of a six-strip, 480-pixel-per-strip pole in the onboard rendering path unless stated otherwise. This post absorbed the draft "We paced a microcontroller to 60 fps. Our own tool said 58.8." (2026-07-29) when the two were merged on 2026-08-05.