Engineering Journal

← all posts

9 min read

Hollow git objects, a browning-out Pi, and a link we never proved

PowerDebuggingSoftware & tooling

By Spencer and Claude. Claude did the writing; the material is Spencer's engineering notes, measurements, and repositories. Nothing here has been independently fact-checked, so accuracy is not guaranteed — read it as a lab notebook, not a datasheet.

Zero-byte git objects kept appearing on the Raspberry Pi that drives our LED poles. We blamed power loss, rebuilt the whole SD card around a read-only root filesystem, and then the corruption came back while the Pi was plugged in. One vcgencmd bitmask gave us a suspect: continuous undervoltage, glitching the SD card mid-write while everything appeared to run normally. This is that investigation, the mechanism that fits, the partition architecture that now ships in every controller, and an honest accounting of why the suspect is still only a suspect.

Why

Our LED poles are driven by a Raspberry Pi 4 that renders and streams 60 fps of pixel data over Ethernet. It lives inside a controller box in the field, which means it leads a hard life by computer standards: it gets power when the venue gets power, and it gets unplugged the way stage gear gets unplugged, which is to say without ceremony, mid-write, whenever the night ends.

The failure that started this investigation was quieter than a crash. The app repo on the Pi would intermittently break: object files inside .git/objects/ were present in the directory listing but 0 bytes long, and git operations that touched them failed. Delete the empty objects, re-fetch, fine for a while, then another one. The filesystem itself was never damaged; fsck was happy; only the newest-written files were hollow.

Theory 1: power loss

Files dying young on an SD card plus a product that gets hard-unplugged made for an obvious theory: writes were in flight when the plug came out. The fix for that theory is architectural, and it was worth building regardless of whether the theory was complete (it was not). The design goal, from the December plan:

The Pi can be unplugged at any time without risk of bricking the device.

The plan splits the card into three partitions with three different relationships to danger:

/dev/mmcblk0p1 - boot   (FAT32, effectively read-only)
/dev/mmcblk0p2 - root   (ext4, mounted read-only, overlayroot on top)
/dev/mmcblk0p3 - data   (ext4, writable: the only place writes can land)

The root partition holds the OS and the application and is mounted read-only, with a tmpfs overlay (overlayroot) absorbing every write to /. Those writes live in RAM and evaporate at reboot, which converts "unplug during a write" from a corruption event into a non-event: the SD card was not being written in the first place. The /data partition holds what actually must persist (settings, patterns, uploaded media), reached through symlinks and bind mounts, so the app itself never needs to know the layout changed.

The migration, and what the checklist caught

We migrated a live controller to this layout in phases in early February, rebooting and verifying between each: /data mounted, bind mounts up, symlinks resolving, services running. The phase checklist earned its keep immediately: the reboot audit caught that patterns.json was still a plain file where a symlink to /data/pfx-config/patterns.json should have been (the two copies were byte-identical, so nothing had broken yet; it just would have silently diverged later). Converted, re-verified, 58 patterns readable through the link. The overlayroot install itself tripped once on an initramfs rebuild and needed a second attempt with TMPDIR pointed at real disk space.

The last checklist item was the point of the whole exercise:

5. Power-cycle test (unplug power, verify recovery)
6. Git repos intact after power cycle

The twist

While closing out that checklist, the uncomfortable observation finally got said out loud, verbatim from the bench notes:

i have been experiencing corruption like that and it doesnt seem to have to do with power loss, but that was the inspiration of having this read only partition to begin with. what coudl be causing it

Corruption had happened between power cycles, on a Pi that stayed plugged in. The unplug theory explained the risk, but it did not explain the bodies. Something was glitching writes while the system was, by all appearances, running normally.

The measurement

The next clue took one command:

$ vcgencmd get_throttled
throttled=0x50000

That bitmask decodes as: bit 16, undervoltage has occurred since boot; bit 18, throttling has occurred since boot. And the kernel log showed it was not historical: voltage was actively bouncing, flagging undervoltage every few seconds, on a Pi that looked perfectly healthy from the outside. No reboots, no crashes, no visible symptom at all except, eventually, hollow files.

Here is the mechanism we inferred from that, step by step. None of these steps was observed directly: there is no trace of the card's acknowledgment, its cache state, or the write that got interrupted.

Git creates an object file: directory entry and inode allocated. Git writes the compressed content, and the SD card acknowledges the write. An undervoltage dip hits, and the card's controller browns out internally, losing its write buffer. The ext4 journal has already committed the metadata (that is its job, and in data=ordered mode it did everything right), but the journal can only trust the card's claim that the data blocks were committed. During a sag, that claim is a lie: the card ACKs the write, then loses it. The next reader finds a structurally valid filesystem containing a directory entry, an inode, and zero data: a 0-byte git object.

That fits every symptom: why fsck found nothing wrong (metadata was consistent), why only recently written files died (only in-flight buffers would be lost), and why it kept happening with the plug firmly in the wall.

Fitting the symptoms is not the same as being the cause, and this is the part worth being blunt about, because it is where an investigation like this usually declares victory. Two things here are measured. The Pi was flagging undervoltage every few seconds, and the repo was growing hollow objects. Nothing we did connects them. There was no supply A/B, no rail capture taken during a git write, no injected sag on a known-good card, and no run that changed the power and nothing else. A card failing on its own, or a kernel or application bug, would leave a scene that looks much the same, and the tidiness of the story above is not evidence for it. What the bitmask does buy is narrower: the undervoltage was continuous rather than historical, so unplugging was a red herring and the plug was never the thing to chase.

The architecture stands either way, for one reason the original theory got accidentally correct: a filesystem that is not being written cannot be corrupted by a write, whether the interruption is an unplugging, a sagging rail, or a bug in our own code.

What ships now

Every controller now runs the full layered design, and it is the same system we develop on daily:

Durability check, and what it is worth

Five months later (2026-07-17), on the same controller, after a day of uptime since its last reboot:

$ vcgencmd get_throttled
throttled=0x0
$ git fsck --no-progress; echo $?
0

The throttle bitmask latches since boot, so 0x0 after a day means one full day in which the firmware detected no undervoltage event; the kernel log for the current boot contains zero voltage messages; fsck walks the entire object store and finds nothing hollow. The validation service is enabled, the config snapshots are current, and no 0-byte object has appeared since the migration.

Which is a good outcome and a bad experiment. The migration changed the storage layout and the supply situation in the same move, so a clean five months cannot say which one bought them, and the most obvious candidate is simply that almost nothing gets written to the card any more. A repo that is barely touched is a poor detector of a fault that only appears mid-write.

And that clean bitmask was a snapshot of one boot, not a fixed rail. On 2026-08-04 the same Pi read throttled=0x50000 again, turned up in passing during an unrelated frame-rate investigation, so it had browned out at least once since that boot. Still no hollow objects. The supply was never repaired, the undervoltage is still there, and whatever has been keeping the corruption away, a clean rail is not it. That is the single strongest piece of evidence against the story this post tells, and it is ours.

Rules we keep

  1. vcgencmd get_throttled is the first command on any misbehaving Pi. It is free, it latches everything since boot, and 0x50000 at the top of an investigation would have saved this one months.
  2. 0-byte files plus a clean fsck do not tell you what emptied them. A clean fsck is tempting to read as "the filesystem is fine, so this must be hardware," and it says no such thing. The journal protects metadata, not your data blocks: an SD card's write acknowledgment is only as trustworthy as its supply rail, and fsck checks filesystem invariants rather than the provenance of file contents. Software mints the same hollow file just as easily. An architecture review in July found that this app's own atomic_write fallback truncated a config file to zero bytes when the disk filled, on a perfectly consistent filesystem, which is our own local proof that a passing fsck rules out nothing.
  3. Storage that is not written cannot be corrupted mid-write. Read-only by default, writable on purpose, and everything that must persist lives in one small, snapshotted, validated corner. It is no defense against a card that dies on its own, which is what the off-device copies are for.
  4. Verify migrations with a checklist, then a power cycle. The audit caught a symlink that silently was not one; the unplug test is the only proof that matters.
  5. Fix the suspected fault too, or you will never know. The architecture makes corruption survivable and does nothing about a sagging rail. We still have the sagging rail, which is both a hazard and the reason the causal question is still open: the one experiment that would settle it, a known-good supply on an otherwise unchanged system, is also the fix.

This is the same hardware whose power side we later put on a measurement bench until it collapsed on purpose (Measuring an LED pole until it browns out). There, undervoltage on the LED rail reboots the ESP32 measurably, on demand, with the rail on a scope. Here, undervoltage on the Pi remains the best explanation we have for the eaten files and nothing more than that. In this product a lot of bugs turn out to be power bugs, which is exactly why the ones that merely look like power bugs are worth being careful with.

Reconstructed from the December 2025 deployment plan, the February 2026 migration and diagnosis notes, live checks against the running controller on 2026-07-17, and a get_throttled reading taken on the same machine on 2026-08-04. All command output is real.