Engineering Journal

← all posts

Part of Building the 3D Club Scene

The 727mm Teleport Was the Fix, Not the Bug

Debugging3D & motion

One knee, on one body, for one frame: 727mm in a sixtieth of a second. It took three generations of diagnosis and five failed fixes to work out that the jump was not where the defect lived, it was the leg returning to truth after half a second of confident wrongness. A debugging story about reading a timeline backwards from its loudest frame.


I bought exactly one pole-dance motion-capture take: 30 seconds, one performer, one set of proportions. Classical retargeting copies that performer's joint rotations onto a new skeleton and hopes the limbs land somewhere sensible, which is fine for walking and falls apart when a body is gripping a fixed steel pole. So instead I extract the essence: a contact script (a semantic record of what touches what and when: right hand grips the pole in this height band over this span, left foot is planted, this leg wraps) plus style curves, and each target body re-solves its own motion against that contract. Three bodies carry the project: ch33, ch02, and medea. At the end of the campaign, the thing between me and shipping was one knee, on one body, for one frame.

The frame

The configuration I wanted was the one I'd been calling the 13-stack: honest span-segmented script, whole-foot clearance instead of ankle-only. It measured 13 leg-through-pole crossings across the three bodies (ch33 4, medea 0, ch02 9) against 15 for the configuration that shipped. Medea at zero was the best single result in the campaign.

It never shipped, because medea failed one gate: point continuity, 18.94s, left knee, 23.8 m/s against an 18 m/s cap. On the raw solver scan, past the gate's smoothing, the event is blunter than that. The leftleg joint (the knee) moves 727.0mm in one 60Hz frame at t=18.944s, which is 43.60 m/s. The hip above it drifts 60 to 90mm over the surrounding 0.3s. The ankle below it moves about 5mm. One bone, one frame, most of a meter.

That is a very loud frame. Everything that follows is a story about how long it took me to stop looking at it.

Generation one: the noisy cross product

Limb placement here runs through a two-bone IK solve. Inverse kinematics means you say where the end of the chain should be (the ankle) and the solver works backward for joint angles that put it there. For two bones the law of cosines fixes the interior knee angle from the reach distance, so the only freedom left is which way the knee points, picked with a cross product of two vectors along the limb. Cross products have a well-known failure: as the inputs line up, the result's length goes to zero and its direction becomes meaningless. A nearly straight leg is exactly that case, and my own source file says so in a comment, sitting above a guard that protects the first place this arithmetic appears.

The first diagnosis found a second place. Inside twoBone() there is a "re-flex" correction that cleans up residual distance error after the main aim step, and it computes its own cross product with no confidence gate and no memory. Its confidence value (the sine of the interior angle) bottoms out at 0.009 at t=18.927s on medea, where the leg is at 88 to 95 percent of maximum reach. So: near-collinear vectors, unguarded cross product, direction flips, knee teleports. Clean story. It also killed the previous suspect, a reach-distance acos singularity that sits behind a flag not even set here.

Two fixes followed. Rate-limiting the reach distance left the teleport byte-for-byte identical (727.0mm, 43.60 m/s). Giving the second cross product the same hemisphere-pin memory the first one has did clear the spike, and broke three other ratified gates for the rest of the dwell: planted sole 36.6mm against a 5mm limit, planted foot speed 2.545 m/s against 0.03, floor penetration to -56.8mm. The arithmetic that flips is also what holds the ankle on its plant target, so freezing it stops the teleport and stops the legitimate tracking with it. The window sweep also had a knife edge one frame wide: a guard starting at 18.77s passed continuity and failed all three plant gates, and 18.78s did the exact opposite. That sensitivity is a reliable sign you are patching a symptom.

Generation two: the wrong branch

The next attempt built the guard properly: active everywhere, no window list, no timestamps, thresholds derived from the take's own data. That derivation produced the best forensic number in the campaign. For every pair of adjacent 60Hz frames where this correction fires, across all three bodies, both configurations, every limb chain: does the raw cross-product direction reverse? Answer: 2,002 pairs, exactly one sign flip in the entire corpus. Medea, leftupleg, t=18.944s, confidence 0.009, dot product -0.995. One fingerprint at one crime scene, and it was the frame under investigation.

Calibrated on that, the guard bridges the one degenerate frame and the spike does not move: 727.0mm unguarded, 727.5mm guarded. Widening the band lowers it (23.8 to 22.9 to 21.6 m/s, never under the cap) and kills tracking until the plant gates go red, the same trade again.

That failure was the diagnosis. If bridging the degenerate frame changes nothing, the event is not a one-frame noise excursion that recovers. Logging the cross product's sign against the frame's own resolved style bend plane showed what it actually is: the sign reverses at 18.944s and stays reversed while confidence climbs back to 0.715 by t=19.194s. Every reading after the crossing is confidently on the wrong side.

That is not noise, it is an IK branch ambiguity. The leg passes through its own straight-leg configuration, comes out on the knee side the source is never on, and holds it until the correction being demanded decays to nothing about a quarter second later. A branch cannot be recovered from the ill-conditioned cue that lost it, which is why every "remember the direction" scheme either can't reach the frame or fights the correct answer for the rest of the dwell. Pinning to the style bend plane was worse: the retargeted leg legitimately sits opposite the source's style pose on 74 percent of leg calls (leftupleg 116 agreeing against 188 disagreeing, rightupleg 14 against 174). There is no free branch oracle in the file.

Generation three: the jump is the recovery

The fourth attempt went in to bound the applied rotation, and started by re-measuring what actually moves. Both leg bones are rigid across the event (799.4mm hip to knee, 883.4mm knee to ankle, constant to the digit). The interior knee angle is monotone and smooth straight through the spike frame: 120.22, 121.87, 122.87, on up to 132.45 degrees. And the knee still moves 727.0mm.

Rigid bones plus a continuous interior angle leave the two-bone chain exactly one free parameter: rotation of the bend plane about the hip-to-ankle axis. The knee sits 362.3mm off that axis, so a 180 degree flip predicts 2 x 362.3 = 724.7mm of travel. Measured: 727.0mm, agreement to 0.3 percent, and the 2.3mm residual is about the size of the hip's own drift across one frame. A direct swivel measurement puts the step at 174.31 degrees where its neighbors are 1 to 3 degrees.

Then the inversion. Comparing the 13-stack bake against the configuration that keeps every gate green, medea's left knee sits 637 to 675mm off that track from 18.81s through 18.927s. At 18.944s that gap closes to 120mm, and by 19.01s it is 6 to 8mm.

That control is a differential reference, not ground truth. It is the same script with one flag off, so what it establishes on its own is which of the two configurations moved, not which one is right. What argues the direction is the corpus: across three bodies and both configurations, 2,002 adjacent frame pairs hold exactly one cross-product sign reversal, and it is in this bake at this frame.

The knee was in the wrong place for about half a second. The 727mm jump is it coming home. I had spent three rounds trying to suppress the one frame in the whole window where the solver was doing the right thing.

None of which makes the frame shippable. 43.60 m/s still fails point continuity, and that gate is why the 13-stack is frozen instead of live. What changes is where the fix goes.

Two stacked plots sharing a time axis from 18.3 to 19.2 seconds. The upper plot shows left-knee speed spiking to 43.6 meters per second at 18.944 seconds, far above the 18 meter-per-second continuity cap and flat everywhere else. The lower plot shows the knee's distance from a control bake rising from 6mm at 18.46 seconds to about 700mm, holding there, then collapsing back to 6mm on the same frame as the spike.
The spike and the error, on one time axis. The control differs from this bake by PLANT_SEG_FLOOR_SHIN and nothing else, so the lower trace is what that one flag adds: 0.48 s of the knee sitting two-thirds of a meter off the control track, ending on the frame everyone wanted to delete.

The proof: it migrates

That reading predicts something testable: a fix at the return can only move where the leg jumps back. The fourth attempt's second formulation tested it, supplying the missing sign bit from something that does not degenerate. The knee is a hinge, so the bend plane normal is a constant in the mid bone's local frame, readable through that bone's world orientation however straight the leg gets. It is stable (re-derived at every well-conditioned frame of medea's walk it drifts 6.72 degrees at the median, 27.51 at the worst) and quiet where it should be: it disagrees with the cross product on 0 of 191 calls on ch33 and 13 of 472 on medea, all thirteen on flips an independent tool finds too.

On the target frame it works. 727.0mm becomes 35.3mm and medea's continuity gate goes green. Then I scoped it by conditioning, applying it only where confidence is low, and swept that scope. Worst excursion in the bake, by timestamp:

18.927, 18.910, 18.877, 18.860, 18.810, 18.760, 18.694, 18.644.

It does not shrink. Sizes across those eight runs stay between 582.5mm and 1023.0mm, five of them within 715 to 733mm. Only the clock moves, earlier every time, tracking the boundary of wherever the correction stops applying. Every run in that table is an offline solver bake, not a browser capture: the double-solve determinism check passes on all three bodies and the shipped bakes re-verify byte-identical, so the migration is a property of the fix rather than run-to-run noise. The widest version also takes the campaign total from 13 crossings to 16 (worse than the shipped 15) and breaks ch02, a body that had nothing wrong with it. The bounded-rotation formulation agreed from another angle: its single clamp fired on exactly the right event, and the crossing re-asserted the next frame anyway, flip and all (162.03 degrees).

Five mechanisms, three tasks, all intervening where the error becomes visible. The migration table says that site can only relocate the error, never remove it.

What was actually broken

The leg enters the bad branch around 18.48s, at a confidently conditioned frame. Nothing at the knee decided that. What decided it lives upstream and was already traced most of the way. The extra clearance push applied to a different span perturbs the root and hip path, and that perturbation persists, because solver state crosses span boundaries with neither a principled reset nor a principled blend. Medea's right-leg motor state diverges first at t=0.067s, inside a wrapped plant span. Cross-leg swivel state for the left leg diverges at t=9.188s, nearly nine seconds before medea's own left-leg span is pushed at all. Hip position then differs smoothly between configurations, 5 to 12mm and climbing, no discontinuity anywhere, arriving at 18.48s with a leg near full extension.

That points at root and span ownership rather than at the knee, and it is the one real engineering task left when I stopped. The carrier was followed that far and no further: nobody has removed the upstream perturbation and re-measured, which is the experiment that would actually settle it. The wrapped span at the head of that chain is the same mis-extracted "plant" behind most of the week's other defects, the other half of this story.

What I'd do differently

The loudest frame is the end of the story. It took four rounds of measured negatives to teach me to read the timeline backwards from it, which is three rounds more than that should cost.