Six sessions were queued for the same bench, and the bench was wrong

By Spencer and Claude. Claude did the writing; the material is Spencer's engineering notes, measurements, and repositories. Nothing here has been independently fact-checked, so accuracy is not guaranteed — read it as a lab notebook, not a datasheet.

Several agent sessions had each independently designed an experiment, and every one of them needed the same physical hardware. Serializing that queue meant checking the instrument first, which turned up 16.79 Mbit/s of unsubscribed multicast on a network where a degraded pole falls back to a 10 Mbit port. That one number retracted a conclusion filed the day before, explained a 60 Hz quantization nobody could place, and turned three carefully built firmware experiments into negative results before they were run.

The setup

Work here happens across a lot of parallel sessions, and by the end of one evening six of them had converged on the same bottleneck. One wanted to A/B a knob that puts the Ethernet MAC interrupt in IRAM. One wanted the assumption-free version of that test, with the LED output disabled instead. One had a full freeze-spray and multimeter plan for a board with a dying PHY. One had stopped a single step short of proving what was flooding the network.

Each plan was reasonable on its own. All of them needed the one bench. The job was to run what could be run, queue what followed, and leave the things that need hands for the morning.

First, the instrument

Every queued test in the latency family reports a round trip time, so the first question is whether this network can currently measure one. It could not: paths that should be sub-millisecond were showing about 95 ms with intermittent loss.

One session had already written the A/B as its next step without running it. Running it took sixty seconds. Five probes, all concurrent inside each arm, thirty seconds per arm, with the gateway pinged inside every arm as a control, since it shares the Mac's port but not the bench crossing.

We observed a factor of about 16 from the Mac and about 275 from the bench Pi, reversible in both directions, with a flat control.

It also corrected the existing record, which read "the whole switch is poisoned" with the healthy pole listed at 120 ms. That came from measuring every host from the Mac. From a host inside the bench segment, that same pole is 0.40 ms and never degrades. The saturation is not the fabric and not the poles, it is the crossing between the bench segment and everything else.

Then the number

A read-only raw socket sniff on the bench segment, counted at the Ethernet frame boundary:

None of it is subscribed, and every port gets it anyway, which is what a switch with no working IGMP snooping does. Worth being exact about the standing of that sentence: nobody has read this switch's configuration, and nobody has had its admin login all night. "Never configured" is an inference from the wire, not something anyone looked up. And that frame boundary matters for the arithmetic that follows: it excludes preamble, start-of-frame delimiter, and interframe gap, so adding those back at 1802 packets per second moves the number up. Every utilization figure below is a floor, not an exact reading.

Hold that number against the other thing this project ships. When a pole's link degrades at 100 Mbit, the firmware withdraws 100BASE-TX from its autonegotiation advertisement and stays up at 10 Mbit rather than going dark. So a pole in fallback sits on a 10 Mbit port, on a network offering 16.79 Mbit/s of background traffic to every port.

The retraction

The day before, a session had concluded that 10 Mbit mode itself imposes a fixed 100 to 170 ms latency floor on everything through the stack. It was careful work. It flashed a healthy pole with a forced-10 build, measured a floor where there had been none, and wrote the cost into the codebase docs. It even checked that the LAN was quiet with a concurrent capture.

That measurement was real. The attribution was wrong, because link speed was confounded with offered load.

This is an interaction, not a main effect. Ten megabit with a quiet network is half a millisecond. The full flood at 100 Mbit costs nothing measurable. Only both together produce a floor, and the arithmetic says why: 16.79 Mbit/s is at least 17% of a 100 Mbit link and at least 168% of a 10 Mbit one. The port cannot drain what it is offered, so its queue sits at whatever depth the switch will hold, the excess is dropped, and every packet that does get in waits behind a backlog that never clears. The 25.6 ms is that depth, not an unbounded queue.

The mechanism is a saturated switch port. It is not the ESP32's Ethernet MAC, not the IP stack, and not the render loop.

In fairness to the retracted session, the first crack showed up in its own instruments before the 2x2 existed, and the way it showed up is instructive. Its logic-analyzer run measured the frame path into the forced-10 pole at 4.9 ms while its pings to the same pole were reading 104, and the discrepancy is not subtle. The reason turned out to be procedural: the analyzer rig stops the show sender for every capture cell as a matter of hygiene, so each cell was, without anyone intending it, a flood-off intervention. The rig's discipline had been quietly running the decisive experiment all along; the ad-hoc pings, taken outside that discipline, were measurements of the flood. When the clean instrument and the casual one disagree, the difference between their procedures is usually the finding.

Two details fall out that had been genuinely puzzling. The floor's jitter was uniform over 16.7 ms, one frame period at 60 fps, and that was read as evidence of coupling to the render tick. But the flood's dominant component is itself sent at 60 fps, so the queue is refilled on exactly that cadence. Both stories predict 16.7 ms, which is why the first one survived. The quiet cells have a render loop running at 60 fps too, and no quantization to see. Separately, the floor's depth had appeared to vary by device profile, which suggested render load. Queue depth on a saturated port depends on how much of the flood that port is offered, which varies by port.

The board that started it

The grid above used a healthy pole forced to 10 Mbit. The board the original 171 ms was measured on is a different one, with a genuinely dying transmit path, and it is sitting at 10 Mbit right now through the fallback firing for real. So it is the honest control: does the retraction survive on the board that produced the number?

It survives, and not narrowly. That board answers in 0.57 ms once the multicast stops. No reflash, nothing touched on it at all, only the traffic changed. Whatever is wrong with its transmit path, and something is, it was never the cause of the floor.

The middle arm is the one I find most convincing. At 4.29 Mbit/s, which is 43% of a 10 Mbit port, it sits at 2.5 ms: neither saturated nor free. That is a queue filling, not a gate opening.

The same table appears to size the other mystery, and here I got it wrong, which is worth showing because the error is a tidy one.

The crossing carrying that 95 ms pedestal is saturated at 12.49 Mbit/s and clean at 4.29, which brackets its capacity between the two. A 10 Mbit link sits exactly in that interval, and this network is already known to produce 10 Mbit links. I wrote that down as the answer. It is a good story and it is the same shape as the finding I had just made, which should have been the warning.

Then I measured the link instead of inferring it. Raw TCP, no cipher in the path, with the network quiet: 934 Mbit/s from the bench Pi and 941 Mbit/s from the desk Pi. Every host reports a gigabit link and the Mac's wired port is gigabit. The crossing is not a 10 Mbit link and never was. At 16.79 Mbit/s the flood is using 1.7% of it.

A first attempt at that measurement went over ssh and read 58 Mbit/s, which is a Raspberry Pi's encryption ceiling rather than the wire. Taking that at face value would have been a second wrong answer pointing the same way as the first.

So the pedestal on crossing paths is a packet-rate or per-host processing effect, 1802 packets per second of unwanted multicast delivered to every port, and not a full pipe. Its mechanism is open. The pole-port result is untouched by any of this, because that one is bandwidth and the arithmetic is not in dispute: a 10 Mbit port offered 16.79 Mbit/s is oversubscribed by at least 68%, and the grid measures what follows. One thing the sniff cannot tell me is what that particular port is handed, since it was captured at another port on the same segment rather than mirrored off the pole's own egress. Unregistered multicast goes everywhere by definition, which is why I am content with it, but a switch quietly rate-limiting one port would not show up in this number.

The thing I want to keep from this is how comfortable the wrong answer was. It fit the bracket, it reused the mechanism I had just proven, and it would have sent someone looking for a bad cable that does not exist.

What can be said about the pedestal is narrower. Measured under flood and again quiet, in both directions on the same wire: toward the bench Pi it adds 49.62 ms, toward the Mac 50.16 ms, and a probe that stays inside the segment adds 0.06 ms. Symmetric to within one percent, so it is the path rather than one host's receive side, and it is avoided entirely by traffic that does not cross.

Then I stopped guessing and generated traffic. Multicast to a group nothing subscribes to, sent from inside the segment with both real senders stopped, at a fixed packet rate and two different packet sizes:

Identical packet rate in the middle two rows. Only the bit rate differs, and only the fast one hurts. So it is not per-packet processing, and a synthetic copy of the real flood reproduces the real effect to within 4 ms, which is a nice check that the rig measures what I think it does.

Which means the bottleneck I withdrew two sections ago exists after all, and I withdrew it for the wrong reason. Both measurements are correct and they are about different paths: unicast between those endpoints is 934 Mbit/s, and flooded multicast saturates at roughly a tenth of that. That is not a contradiction, it means the flood is not being carried the same way the unicast is. The signature, a threshold in Mbit/s with complete indifference to packet rate, affecting multicast while unicast runs at line rate, is what a switch's storm control or unregistered-multicast rate limiter looks like. I cannot confirm that without the switch's admin interface, so it stays a candidate rather than a conclusion.

Three retractions in one night, then, and the last one is a retraction of a retraction. What I would keep is that each was overturned by an intervention rather than by better reasoning about existing numbers: change the speed, change the load, change the packet size. Every time I reasoned instead of intervened, I produced something plausible and wrong.

The same mistake, one level in

While I was writing this up, the session whose conclusion I had just retracted came back with a new one. It measured that dying board from three vantages, subtracted each vantage's own share of the pedestal, found about 170 ms left over from all three, and concluded that this residual was the board itself after all.

Our numbers agreed to under one percent. Bench Pi to the board: their 274, my 274.0. Wired Mac: their 227.6, my 227.3. Wireless Mac: their 178.6, my 177.2. Nobody mismeasured anything.

The arm neither of their vantages included was turning the flood off:

The board was not touched: no reflash, no reboot, only the traffic changed. There is no invariant residual, and the dying board is exonerated for round trip time at 10 Mbit in every traffic state I put it in. That is narrower than it sounds in a good way and in a bad one: it does not speak for the board at 100 Mbit, where it cannot hold a link long enough to measure anything. Its transmit fault is real and unaffected, and it still needs the morning's freeze spray.

What makes this worth writing down is that the arithmetic was clean. Subtracting a vantage's pedestal from its total really does isolate that board's own port queue. And that queue really is invariant across vantages, because it belongs to the board's port and looks the same from everywhere you stand. Invariance across observers is normally exactly how you argue something is intrinsic. Here it was the property of a queue that all three observers were looking at through different windows.

So this is the original confound one level in. First link speed was confounded with offered load. Then the residual-after-subtraction was confounded with offered load. Both times the resolution was the same, and both times it was not more arithmetic: change the load and see whether the number moves.

Why the quiet check missed it

The earlier session verified the LAN was quiet with a concurrent capture and recorded 52 non-mDNS multicast packets in 35 seconds. That check was honest and it was run.

The Mac has two active interfaces on the same subnet: a wired one and the one holding the default route. They traverse different physical links, and they disagree by more than an order of magnitude in opposite directions depending on the target. A capture on one says nothing about the flood on the other, and an unbound ping or capture picks by service order rather than by intent.

So the standing rule now is that a measurement on this bench states which interface it came from, and a quiet-LAN check is taken on the interface the measurement uses, or from a host inside the segment.

Three experiments that became negative results

Three firmware arms had been built for the mechanism that turned out not to exist. They were run anyway, because they were already built and because a null result is worth having on the record.

Putting the Ethernet interrupt in IRAM changes nothing. Disabling LED output entirely changes nothing. Since arms cannot run concurrently when each needs its own flash, the last arm re-flashes the control as a drift check, and it reproduces the first to within 0.015 ms of mean. What that bounds, precisely: 150 pings per arm, standard deviations of 0.14 to 0.17 ms, four means inside a 0.023 ms spread. No equivalence interval was pre-registered, so this rules out an effect the size of the floor these arms were built to explain, not an effect.

The LED arm is worth one note, because the build system caught a mistake in it. The obvious implementation skips controller registration, and that built cleanly and was then refused: skipping registration never instantiates the templated controller, so the entire I2S driver drops out of the image and its build stamp goes missing. That is a much larger change than the experiment intends, and on a codebase with a measured flash-cache placement lottery it would have moved every hot loop around it. The arm would have differed from its control in code layout as well as in LED output. Gating the output calls instead keeps the driver linked and placed, and stops it running.

The lock had a hole

The bench has a careful arbitration story. Its lock holds three things at once, a local lock, a lease visible across hosts, and the testing Pi's own lock, because an earlier bug let two machines each take a local lock and both flash the same board.

It did not save me. I ran a build with an argument I believed selected where the compiler writes. It does not, it selects where the resulting artifact is verified. The build always compiles into one shared directory, and another session's job had staged its own test image there and was mid-cycle. My compile overwrote it.

Nothing broke, and why is the part worth keeping. That job gates every flash on a hash the board itself broadcasts, so flashing the wrong image raises rather than proceeding quietly, and its restore reads from its own backup rather than from the shared directory. Two independent applications of the same house rule, verify from the artifact and never from the build system's claim, contained a collision the lock could not see.

The general version: the lock covered the resource everyone agreed was shared, and not the resource nobody thought of as a resource. Compiling is not a board operation, so nobody takes the lock to compile, so two sessions can queue politely for the board and still corrupt each other through a directory.

There is a smaller sibling. That job takes the lock per flash rather than for its whole run, so the lease reads free between operations while the board is still in use. An orchestrator polling the lease would have walked into the middle of someone's experiment. Watching the process was the correct gate. "Is the lock held" and "is the bench available" are different questions, and only one of them has an API.

And a third hole, found the hard way earlier the same evening: when the lock grew its cross-host teeth a few days ago, it silently broke the one tool that had been careful before careful was mandatory. The latency rig had always taken its own lease as belt and suspenders, then entered the lock. Once the lock started taking a lease of its own, the rig's lease made the lock's acquisition fail, and every run of that tool died at startup against its own reservation. Nobody noticed for four days because nobody ran it. A lock that gains enforcement breaks exactly the callers that were already polite, and it breaks them at the first use after the change, not at the change.

What was left for a human

Three queued items are blocked on hands rather than effort: freeze-spraying a suspect chip while watching an error rate, metering two supply rails against that same ramp, and configuring IGMP snooping on the switch, which would remove the floor, the trunk pedestal, and a pause-frame episode that preceded them, all at once.

One item moved into that category on inspection. A drive-strength sweep was queued as runnable unattended, and it counts symbol errors while walking a pad's output impedance, which is a 100BASE-TX measurement. But the board it targets carries the fallback and drops itself to 10 Mbit within about thirty seconds, where the receive coding is different and the counter no longer measures the thing on the label. Holding it at 100 Mbit needs a cable trick that needs a person. Running it anyway would have produced a clean-looking table answering a different question than the one asked.

Which is the same failure this whole night was about. The forced-10 measurement was clean, repeatable, correctly instrumented, and pointed at the wrong cause, because the thing that mattered was not in the frame.