Part of Building the 3D Club Scene
We asked three frontier models to attack our design. They contradicted each other on the fix we had already queued.
Before committing four to seven expensive sessions to the riskiest rewrite stage in the project, we wrote the whole problem out cold and sent it to three models from different labs, explicitly asking to be told why we were wrong. The useful output was not a design. It was one flat contradiction on our own next step, three methodology corrections, and a criticism that caught us rebuilding a bug we had just finished writing a lesson about. The responses themselves were not archived, so this is our retrospective account, not a transcript.
The situation
Our offline solver takes one purchased 30-second pole routine and re-performs it on a given body, judged by a harness of fourteen kinematic checks plus a human who has repeatedly caught real defects the metrics missed.
A staged rewrite was replacing per-frame corrector passes with an interval trajectory solver, and the stage just completed had produced an unusually clean diagnosis: all four remaining user-visible defects share one mechanism. Planting a floating foot requires the root about 145 mm lower, and that drives the short-armed character's gripping forearm into the pole — collision frames going 7 → 46. The two kick-through windows are unroutable with the hip frozen by arm mechanisms. The residual shake is the same mechanisms dithering against each other.
The proposed next stage, "S4," was to co-solve pelvis and arms as one variable block: four to seven expensive sessions, the highest-risk stage in the plan, and the one every prior round had been pointing at for months.
Which is exactly the moment to spend an hour trying to get told you are wrong.
How we asked
The prompt was written for a reader with no prior context — architecture, harness, gate definitions, the history of the rewrite, the current numbers, all of it inline — and it named five specific assumptions to sanity-check rather than asking for a review in general. It also said, in the first paragraph:
Please be adversarial where warranted — we are specifically looking for reasons the current thinking is wrong, not validation.
That framing matters more than the model choice. A request for review returns review. A request for reasons this is wrong returns the thing you cannot generate yourself, which is a stranger's objection to your load-bearing assumption.
The two hardest technical questions went out with it: how to model deformable flesh against a rigid pole for the purposes of a detector, and how to keep a stateful per-frame corrector stable across a multi-second hold.
One limitation to state up front, since the rest of this post asks you to take our word for what came back. The prompt is archived verbatim in the repo. The three responses are not, and we did not record which models produced which answer. Everything quoted below is our adjudicated summary, not a transcript anyone can check us against.
They contradicted each other on exactly the queued fix
One model recommended decaying accumulated corrector state toward zero on a timer, presented as a standard stabilization technique. Another explicitly warned against that: a timer decay bleeds away a correction that is legitimately needed during a long hold, and hides the defect rather than fixing it.
"Decay the accumulated state toward zero on a timer" was already our own candidate fix #1 for the retry, one of four redesigns listed when we reverted the offending corrector, and none of the four had been attempted.
The second reading won, and its reasoning checks out against our own telemetry. The failure we were trying to stabilize is a corrector whose state accumulates unbroken across 14+ seconds of continuous grip and then detonates: 187 mm on one run in five, against about 28 mm on the other four. A timer decay makes the detonation rarer and the underlying hold worse.
But the tiebreak was not the panel's to award. We had already run the experiment. Days before this consultation went out, on a different stateful motor, we tried exactly the decay the first model recommended: bleeding the elbow-swivel angle toward zero at its rate cap on every free frame instead of snapping it to zero the instant a hand releases. Measured, it did not close the release spike it was aimed at (still 8,800 to 9,800 deg/s at 8.52 s on two bodies, unchanged) and it regressed a third body from 9 of 14 checks passing to 8. We reverted it and kept the instant reset.
So the design we favor was fixed by measurement rather than by argument: a bounded tracking state, a frame-local optimum stepped toward under both rate and absolute magnitude bounds, with resets at semantic boundaries rather than on a clock. That is the elbow-swivel motor we run today, and it predates the consultation by days.
The corrector that detonated is a different one, and it is not running at all. It margin-checked the rendered pose rather than the pre-blend solve, which worked on its own target, and it did that by adding a second stateful motor. We reverted it. The re-implementation is still queued and has never run, and its brief is explicit about what the replacement may not be: take the idea and the evidence, not the stateful second motor. The queued design is stateless.
So the honest accounting is weaker than "the panel found our bug." The panel did not hand us an answer. It disagreed with itself, in both directions, about a call our own measurements had already made.
The meta-lesson is still the one worth carrying: you do not get consensus from three models for free. You get three answers, and if two of them are incompatible on your critical path, the adjudication is still yours and still has to be done against your own measurements. What the panel bought us was not an answer. It was the knowledge that our answer was contested, which is precisely the thing a single confident recommendation would have concealed.
Where they were confidently useless
Worth reporting honestly, because the failure mode is systematic rather than incidental: outside models reason about a greenfield system.
Several high-confidence recommendations described things already built. Add a collision check — it existed (and was miscalibrated, which is worse, but that is not what they said). Anchor foot plants to intervals — that had been the architecture for rounds; the actual cause of the floor-slide complaint was a drag-follow gain running three to six times the source's own motion.
None of that is the models being bad. It is a structural consequence of a reader who cannot run the code, cannot see the git history, and is working from a prompt that necessarily compresses months of decisions into two pages. The better an unseen system is described, the more it resembles every other system in the training distribution, and the recommendations regress toward the architecture everyone would build.
The practical filter that came out of it: ask outside reviewers about failure modes and methodology, not about what to build. They cannot know your system. They can absolutely know how systems like yours fail.
Where they were worth the hour
Three methodology corrections, all of which changed what we did next:
- Rate limiting does not bound state. Obvious once said, and we had been treating a rate-bounded corrector as a bounded corrector for weeks. A cap on how fast a value may change says nothing about how far it may travel, which is exactly how a slow accumulation over a long hold ends in a 187 mm step. We went and put an absolute clamp on the one stateful channel that had none, and measured its real ceiling first instead of guessing: 3.33 rad. The guess we tried first, 2.2 rad borrowed from a different quantity in the same file, would have silently clipped genuine motion on two of three bodies. The clamp that shipped sits at 6.0 rad and changes nothing today, which is the correct resting state for a guard rail.
- A barely-moving motor plus a sudden large end-effector error is a discontinuity signature, not a reach failure. That reframing pointed the next diagnosis at branch and coordinate flips rather than at IK quality, which is where the bug was.
- Twenty-one runs are needed to see a 1-in-5 defect with 99% confidence. We had been failing and passing rounds off single captures, on a build where one body is stochastically unstable: one of six identical-code captures spiking to 83 mm where the rest sit near 30 mm. That number settled an argument we had been having with ourselves for two days, and recalibrated our acceptance protocol: one capture for continuity, six with max-of-N for the noisy tails. Six is a budget, not the calculated number, and it is worth saying so plainly: six runs see a true 1-in-5 event about 74% of the time, against the 99% that twenty-one buys. The "1-in-5" is itself a single spike in five runs, with a separate single spike in six alongside it, so the rate we are powering against is barely estimated at all.
The best contribution was a criticism
The single most valuable thing any of the three said was not a proposal. It was an objection to our newest fix.
We had just replaced a set of hardcoded, far-too-thin collision radii — the green-light bug that cost us a day — with per-patch compression budgets: how far each body part may compress against the pole before it counts as interpenetration. Flesh is soft, a rigid capsule on a thigh is not physically honest, and the budgets are defensible.
The third model pointed out that they are also a new family of hand-set constants standing between "intersecting geometry" and "pass." The same hazard we had just written a lesson about, moved up one level of abstraction, where it is harder to see and easier to defend.
Its replacement is a calibration loop our existing WebGL harness can run with no new machinery: per camera, render body front-face depth, body back-face depth, and pole depth. A pixel where the pole is the visible surface and its depth lies between the body's two faces is the pole emerging through the body. Count those over a ring of six to eight cameras. A budget is correct when its worst frame produces about zero such pixels from the product cameras. Perceptual, view-dependent, and calibrated against the rendered output rather than against a geometry proxy.
We built that metric and tested it, and as a reference-free absolute threshold it did not work. A bake that penetrates the pole for 52% of its duration peaked at 549 pixels against 484 to 519 for six clean runs, and three separate refinements failed to pull them apart; the whole write-up is its own post. Compared frame-for-frame against a known-good bake the same pixel test does separate cleanly, so what died was the threshold rather than the rendering. The criticism still stands on its own terms. It was right that a new family of hand-set constants had appeared, and it was right about that before anybody had a replacement to offer.
It also caught the thing three humans and two other models missed: the pole is emissive. It is an LED product. Glow bleeding through a thigh is far more visible than the same geometry would be on a matte cylinder, and no depth budget captures that. The pixel metric does help, because the pixels it counts are the lit pole, but only in proportion to area. Bloom bleeds well past the geometry, and that part we still have not measured.
That observation is why the consultation was worth the hour, and it is also why we now route every new constant family past an outside reader before it becomes load-bearing.
What we changed about how we ask
- Send the whole context cold, and say what you have already decided. The contradiction was only visible because the prompt named our queued fix. A prompt that asks "how should we stabilize this?" gets you two plausible answers and no signal that they disagree.
- Ask about the newest thing, not the oldest thing. Old design decisions have been attacked by your own measurements for months. The fresh constant, the fresh abstraction, the fix that landed yesterday — that is where an outside reader has an edge, because nobody has argued with it yet.
- Expect the greenfield bias and budget for it. Roughly half of what came back described the system we already have. Reading past that is cheap; being disappointed by it would have cost us the other half.
- Adjudicate with your own numbers. Three models, one contradiction, and the tiebreak came from an A/B we had already run on our own motor. The panel narrowed the question. It did not answer it.
One rule came out of this exercise and now lives in our acceptance-criteria file: a work stage may change the solver or change the definition of correct, never both. It started as a process observation in a consultation response, which is a fair summary of what an adversarial outside read is actually good for.