DDLS

The short version, for the founder. Updated 2026-11-03. The long version is SYSTEM.md (every system, what it cannot do) and COMMAND.md (every workstream, owner and number).

Goal. Video of trades work goes in. A physics sim with the real hands and objects comes out, proven against held-out frames of the same video. From that: datasets now, a policy on our hands next.

The number. Held-out edge F1 of the framing scene against its video: 0.293 now, 0.97 to ship. It counts only once the scorer's 11 controls pass. 1 passes today.

What is wrong, with the cause

Nothing runs itself. Every fit, score and sim starts from a chat and lands in a folder; the site shows none of it. There is no job table and no runner.
The static scene is already placed; the tail comes from how we fit it. SfM localised all 1063 frames against the structure, and the median member sits 4.2 px off the video. The 21.5 px tail is every fit being time-blind (members drawn before they exist, frames pooled across installs) and actors covering members.
The scorer counts actors and needs instances. Hands, tools and walls score as wood, and member matching waits on per-member tracks that we do not need.
The hand lift is geometric, and gloves have no training data. Gloved fingers come from RTMPose lifted along lens rays, not a learned lift. EgoSuite has no gloves. The retrained EgoSuite lift (8.3 mm held out) scores 81.9% on the gloves, below geometric 84.7%, because 58% of gloved hands have fewer than 6 confident keypoints.
The nailer is never in the hand. The tool and the hand are placed by two trackers apart (71 mm gap at frame 0), so the tool falls from step one. ddls carry never turns the grip on. The handle is about 1.5x too thick for this hand (55 to 58 mm against about 37 mm). The sim hand never holds still (finger bend 0.4-0.5 rad/s where real holds sit at 0). Physics strength and contact settings are not the limit.

How we get it done, in order

#	Step	What it moves	Status
1	The loop: job table, runner, typed questions with owners, `/runs` page	Every job below runs without a chat	Not started. First
J1	Camera frozen and audited: lens into the ledger, per-frame residual by image row and head speed	Proves the camera; decides if rolling shutter needs a parameter	Filed
J2	Lumber masks on every frame, one pod run, no tracking	Lumber on 90% of frames where a member should show	Done: masks frozen; the 100% gate is meaningless (every frame is 34–53% wood); used as lumber confirmation inside other rules
J3	Install times as presence (first seen in place, bounded by first nail)	The pre-install misses, most of the scene-side error	Done with your 17 answers, frozen: 99 of 102 static, 3 install at their nail, 4 caveated; 5 of 10 "first nails" were re-nails on members already in place; joist_002 and joist_016 outline off, handed to J4
J4	One rigid layout fitted on present frames, camera fixed, actors masked	Member p97 21.5 px to 3 px or under	Filed; needs J3
J5	Install times again on the J4 layout, audited by Jev	Shifting installs by 20 s halves F1	Filed; needs J4
J6	Scorer v3.2 without instances: footprint inside the lumber mask, actors out	All 11 controls pass	Filed; needs J2
J7	Track only what moves: hands, nailer, the member being carried	Carried member IoU 0.8	v2: hands pass your gate (coverage 1.0, IoU 0.632) and are frozen as actors-v2; nailer one frame short (24/28 = 0.857, IoU 0.605); audio finds nail shots (test F1 0.80); caulking gun and carry (v3 precision 0.152) still open
G	Grasp proof: seat a wrapped grasp, grip on in `carry`, handle girth measured	The nailer held through the lift, one rollout	Filed; fourth cause found: the sim hand jitters during the hold
H	Hand lift trained on EgoSuite 3D joints (no frames needed)	Hand inside its mask 84.7% to 95%	Done, gate not passed (81.9%), lift off; 2D glove keypoints are the bottleneck; Job B done: fine-tuned 2D model lifts the gate to 88.6% geometric (87.6% learned), still under 95%; part of the gain is which hands are checked; next fix is box recovery between keyframes
P1-P3	pi 0.5 on our hands: export joint vectors, fine-tune, evaluate in the twin	Success per gate for a policy	Not started; needs G
P5	Weight edit: repair one diagnosed failure of that policy	The filed gate passes, no other falls	Needs P3

Everything after step 1 runs as a job inside it. That is why it is first.

What we were wrong about

Per-member masks. Static members do not need tracking. Most "motion" in the masks probe was the camera.
"Physics cannot hold a tool." The hand never started on the handle and the grip was never on.
"SAM 3.1 has hand skeletons." It gives masks, boxes and track ids only. SAM 3D Body has hand pose, but its weights' commercial terms are not stated and Meta says its hands are weak.

How the work runs now

The system does the work: jobs, a runner and questions with owners. Astra via Codex and Opus agents do the figuring out and the building. The orchestrator (Fable) sets direction, writes briefs and judges results; it does not do the work. We scrap what holds us back: the per-member tracking code is retired (list in COMMAND.md), and tests that cost time and prove nothing are culled.

Needs you

Outline-adjust pass, 17 frames (0/17 done): run .venv/bin/python out/presence/run_nails.py serve --adjust and drag each outline onto the wood. It writes adjust.json; export-adjust turns it into per-member pixel offsets that J4 uses to refit the layout and J1 to check the cameras.
J7 confirm page: done (55/56 steps, 579 s). Restarted on http://127.0.0.1:8767 with a working delete key and a notes field; nothing more is needed from it unless you want to correct a step.
The test cull list: approve it when it lands. One candidate so far: tests/test_focal_hypotheses.py (tests experiments/camera/focal_hypotheses.py, which nothing else references).

Today

Done: your caulking-gun sitting (10/10 frames). Caulk truth is now 30 frames, 12 boxes; on it nothing passes: the SAM 3.1 concept path finds 5 of 12 (no false boxes on the 18 none), caulk v3 3 of 12. The hose-linked nailer veto deleted a caulking gun you confirmed. Next: SAM 3.1 with all 12 of your boxes as exemplars. No spend.
Done: nailer round 3 (SAM 3.1 points from Jev's box; pod USD 0.148, deleted; pods USD 3.82 of 50, Jev about USD 0.39 today). Coverage 28/32 = 0.875, IoU 0.509 (round 2: 0.625, 0.458): still fails on IoU and coverage, not frozen. (Corrected: the 0.598 first given as your agreement with Jev was you vs the track; you vs Jev is 0.540.)
Done: nailer box-extent measurement (no spend). You vs Jev: 0.540 over 27 frames, 0.666 on the 19 where both boxed the same tool, about the same as hands (0.648), so the extent is not the problem; the low number is Jev boxing the wrong tool. Your boxes are the looser ones (whole tool down to the air fitting, glove on the grip; hose and forearm out); Jev's and the track's sit inside them at about 0.8 of the area. Measured, the bar would be 0.54; nothing on the nailer passes at 0.54 or at 0.6. Your call: keep 0.6 or adopt 0.54. Either way the work left is finding the tool on the missed frames and boxing it to your extent.
Done: J7 concept round, your direction (SAM 3.1 from text + exemplar boxes, Jev as the agreement check; pod USD 0.807, deleted; pods USD 3.67 of 50). Hands pass (0.979, IoU 0.680) but trail the frozen actors-v2; the nailer (0.625, IoU 0.458) and caulking gun (1 of 2) fail. The text prompt "nail gun" alone lands off the tool (IoU 0.018). Nothing frozen.
Done: J7 confirm round. Hands pass on your merged answers (48/48, IoU 0.733); the nailer is three frames short (26/32 = 0.812); the caulking gun detector, built from written knowledge, finds both of your yes frames with no false boxes on the 18 none. Your boxes were tighter than the proposals, not wider (median area ratio 0.63); no change made.
Done: J7 v2 pod (USD 0.776, deleted; pods USD 2.86 of 50, none running). Hands pass your gate and are frozen; the nailer is one frame short; the caulking gun and carry are open; the confirm page is ready.
Done: your 17 presence answers. Installs frozen with you as the verifier: 99 static, 3 install at their nail, 4 caveated; 5 of the 10 nailed members were already in place (re-nails).
Done: your J7 click sitting (1373 s). Against your boxes the hands track covers 0.878 (gate 0.9) at IoU 0.607, not the 0.975 it scored against itself; the nailer gate fails (0.679, IoU 0.401); carry v1 precision 0.167.
Done: J2 and J7 pods (pods USD 1.08 so far, all deleted; Jev about USD 0.02). J7: hands tracked and frozen with a caveat; nailer and carries fail their gates (details in COMMAND.md, "J7 results").
Done: Job B (hands-2d pod, USD 1.007, deleted): fine-tuned RTMPose-hand raises the hands gate 84.7% → 88.6%, still under 95%. Pod spend today USD 2.09 of 50; no pods running.
Done: EgoSuite hand agent: learned lift 8.3 mm held-out MPJPE, but 81.9% on the framing gloves (gate 95%), so it stays off.
Done: SharpaDex tactile agent (development only): slip-onset detector precision 0.53-0.62 against 0.10 for the rule; CARD and FoLD gave no gain; it found the sim hand never holds still.
Done: your EgoSuite ruling landed (EgoSuite-trained hand weights may label delivered sessions; the data never ships); the two hand-training agents (EgoSuite skeletons, SharpaDex holds, both with CARD and FoLD) are running.
Done: every worktree merged into main; SYSTEM.md on the site; three fact checks (scene, grasp, hands) folded into COMMAND.md and SYSTEM.md.
Done: the merge's test suite passes (477 pass, 8 skipped by environment; the default run takes 5 minutes, the one 22-minute test runs only with -m slow).