arthur/docs/multi-face-representation.md

Multi-face representation

Status (2026-09-29): representation work complete. Reopen it for a concrete requirement, rather than another round of abstract alternatives.

Implemented: each tracked face has a drawing timeline, placed by an ordinary symbol instance. Timelines already provide local node names, independent playback, and persistence. No new kind of scene container is needed.

:timelines
{:main   {:nodes {:root   {:time {:mode :map :expose 2}}
                  :face   {:parent :root :channels <source-to-stage placement>}
                  :face-1 {:kind :symbol :of :face-1 :parent :face :z "a0"}
                  :face-2 {:kind :symbol :of :face-2 :parent :face :z "a1"}}}
 :face-1 {:nodes {:head {...} :mouth {:parent :head ...} ...}}
 :face-2 {:nodes {:head {...} :mouth {:parent :head ...} ...}}}

:features
{:face-1/mouth {:subject :face-1 :timeline :face-1 :area :mouth
                :nodes [:mouth :mouth-in]}
 :face-2/mouth {:subject :face-2 :timeline :face-2 :area :mouth
                :nodes [:mouth :mouth-in]}}

The example omits ordinary ids, frame counts and channel details.

What belongs where

• A subject identifies a source track and supplies shared measurement settings. Its timeline has the same id and contains its measured :head.
• A feature owns nodes in an explicitly named timeline. Its clip-level id is qualified when generated; ownership is read from fields, never parsed from ids.
• A node has a local name. Parents, stencils and pose groups use local names too.
• An instance places and retimes a drawing. Its pose tracks can hold one face's mouth while the other face continues moving.

Subject metadata and drawing timelines remain separate facts. Hand-drawn timelines need no subject. Features retain explicit timeline references, so their locations are not inferred from their labels.

Both filmed faces share one source-to-stage transform. Fitting them independently would stack them at the center. Additional placement uses each instance's ordinary channels. Cross-face draw order is the instances' :z order.

Consequences

Regeneration updates the addressed timeline directly. There is no temporary swap into :main, no special stage regeneration path, and no renaming of parents or stencils. Composing a stage moves the take's root into the library and preserves its child timelines. Settings appear once per tracked object, since all placements read that same drawing.

Presence masks inside a subject use local feature names, matching measurement. Freeze qualifies them when building block descriptors. Head and retained-source blocks explicitly name their subject; otherwise two faces with identical detection masks could produce different bytes under the same key. Analysis addresses also include detection capacity and assignment settings, so old single-face detection results cannot satisfy a new multi-face request.

Validation counts node ownership by [timeline node]. The server requires one complete set of retained source roles per subject, rather than exactly three blocks for the entire analysis.

Nested rectangles retain fractional sizes until rasterization. Rounding inside a face timeline discarded small head-local pupils before the source-to-stage scale was applied. This was a real rendering error missed by the earlier proposal's coordinate-only benchmark; the regression now compares all mark extents as well.

Verification and limits

frontend/test/arthur/flow/multi_face_test.cljs exercises two distinct subjects, source block separation, detection and feature gaps, independent pose cuts and regeneration, nested stage save/load, and equivalence to a flat single-face scene. Existing geometry, raster, source, and regeneration tests cover the same paths. clips/tests/test_api.py checks complete source roles per subject and immutability. The browser suite checks rendering, pupils, playback, save/open, drawing and upload.

Assignment remains a nearest-centroid heuristic, with a version and distance gate recorded in the analysis. Reordered detections, late arrivals and gaps are tested; identity through crossings or long disappearances is not guaranteed. Assignment happens before measurement, so correcting it requires measuring again.

This changes the freeze and retained-source contracts. It does not migrate older flat captures; reanalyze their footage to use the new regeneration path. Existing rendering and leaf codecs still understand their node/channel representation.

Next steps

1. Commit the verified checkpoint: 319 frontend tests, 44 API tests, browser checks, and app/test builds passed. Builds reported no warnings.
2. Exercise real two-person footage, especially crossings, late arrivals and disappearances. Check assignment before treating the resulting geometry as evidence about the representation.
3. Build performance-pose selection and instance-scoped picture rates, reusing the existing held-frame lookup and pose groups.
4. Then build plate-drawing selection and independent tracing references.

The timing handoff owns the detailed next implementation sequence. Older flat captures need reanalysis unless a migration is separately undertaken to preserve their authored edits.