Engineering Journal

PoleFX Engineering Journal

Notes from the bench

LED power, signal integrity, firmware and related topics.

Latest field notes

10 / RECOMMENDED
13 min read

Post-mortem of a dying PHY

A pole controller that was healthy on July 29 was unreachable by August 4. Three sessions across two nights chased it: the first convicted the wrong components with real measurements, the second acquitted them with a better experiment and found a fault that worsens with 100 Mbit operation and rewinds with a register-level reset, and a third proved that the scariest symptom, a 274 ms ping, was never this board's fault at all. This is the post-mortem: what the fault looks like, where it probably lives (the root cause is not yet proven; the bench session that would prove it is queued), and an itemized list of the investigative flaws, because every one of them is the reusable part.

NetworkingHardware & PCBDebugging
Read the field note →
In a felt-animal workshop, one of three glowing glass network channels is packed with pixel blocks. Its swollen bellows sends a mint feedback line to the shared source while a mouse and an otter trace the unwanted slowdown.
13 min read

The switch said slow down, and the Pi said yes

The player dropped to 26 fps with three outputs and nobody on the Pi was guilty: cool SoC, idle CPU, a near-empty gigabit link. The culprit was a polite Ethernet feature from 1997, arriving 130 times a second from the cheapest box in the rack. This is the diagnosis, in the order it actually happened, and the one-line fix that now ships baked into the image every new unit is built from.

NetworkingPerformanceDebugging
Read the field note →
14 min read

Anatomy of a frame

One 60 fps frame drawn across every owner that touches it: the Pi's render loop, the network, the pole's receive task, its LED task, the strip's own wire protocol, and the instant the pixels change. Three of the six lanes are now captured frames rather than reconstructions. The figure says which is which, and which clock each one came off.

FirmwareInstrumentationPerformance
Read the field note →
14 min read

Two extra laps of a four-pixel ring

A flicker you see twice an hour and cannot reproduce. It turned out to be an LED driver's DMA engine re-sending its own buffer while the interrupt meant to refill it was switched off, because a keyword that was supposed to put that interrupt in fast memory had quietly compiled to nothing.

FirmwareInstrumentationDebugging
Read the field note →
14 min read

A bench that measures photons, watts, and bits

What we built to develop LED poles against: seven current monitors, a light sensor taped over one masked pixel, a logic analyzer on all six data lines, and a relay that can yank power from a Raspberry Pi on command. What each instrument is for, what the bench can answer that we could not ask before, and the numbers it produced in its first week.

Hardware & PCBPowerInstrumentation
Read the field note →
series16 posts154 min total

Building the 3D Club Scene

Retargeting mocap onto arbitrary bodies, an AI-built motion-QA lab, the camera department, and the acceptance process behind it — design and debugging notes from a real-time three.js club stage that is still very much in progress, not a tour of the finished thing.

InstrumentationPerformanceDebuggingTesting & QA3D & motionAI & agentsSoftware & toolingProcess
Read the series →
25 min read

Ten slices to five: what a WS2812 bit really is, and how an ESP32 builds one

Our LED driver's changelog has a one-line entry: "custom 250/500/500 ns timing." Behind that line is a wire protocol with no clock, a peripheral that was designed for audio and cannot vary a pulse width, and a piece of integer arithmetic that decides how much memory bandwidth your LEDs cost. Getting that arithmetic wrong cost us 32 MB/s of DMA reads and, in the worst capture we took, two or three corrupted pixels in every frame. This post explains the protocol from scratch, then shows exactly what the ESP32 does with it, then hands you an interactive model of the thing so you can check any LED chip yourself.

FirmwareHardware & PCBPerformance
Read the field note →
12 min read

Measuring an LED pole until it browns out (on purpose)

Real current, voltage, and light measurements from an SK6813 LED installation: what FastLED's power limiter actually delivers, a per-pixel power model that tracked whole-pole draw within 3%, and a deliberately induced power supply collapse.

Hardware & PCBPowerInstrumentation
Read the field note →
9 min read

Hollow git objects, a browning-out Pi, and a link we never proved

Zero-byte git objects kept appearing on the Raspberry Pi that drives our LED poles. We blamed power loss, rebuilt the whole SD card around a read-only root filesystem, and then the corruption came back while the Pi was plugged in. One `vcgencmd` bitmask gave us a suspect: continuous undervoltage, glitching the SD card mid-write while everything appeared to run normally. This is that investigation, the mechanism that fits, the partition architecture that now ships in every controller, and an honest accounting of why the suspect is still only a suspect.

PowerDebuggingSoftware & tooling
Read the field note →
10 min read

Torturing 100 Mbit Ethernet through a slip ring, six connectors, and two coat hangers

We measured how much a spinning slip ring, two couplers, solder joints, two pairs of aviation connectors, and six feet of untwisted wire degrade an Ethernet link, using iperf UDP tests at the product's real data rate. Answer: almost nothing, to a degree that surprised us. Includes one botched first test round, one lesson about link negotiation, and a closing experiment involving wire coat hangers that we do not endorse but cannot stop being proud of.

NetworkingHardware & PCBInstrumentationTesting & QA
Read the field note →