arthur/docs/multi-face-representation.md

# Multi-face representation

Status (2026-09-29): representation work complete. Reopen it for a concrete
requirement, rather than another round of abstract alternatives.

Implemented: each tracked face has a drawing timeline, placed by an ordinary
symbol instance. Timelines already provide local node names, independent playback,
and persistence. No new kind of scene container is needed.

```clojure
:symbols
{:main   {:nodes {:root   {:time {:mode :map :expose 2}}
                  :face   {:parent :root :channels <source-to-stage placement>}
                  :face-1 {:kind :instance :source {:symbol :face-1}
                           :parent :face :z "a0"}
                  :face-2 {:kind :instance :source {:symbol :face-2}
                           :parent :face :z "a1"}}}
 :face-1 {:nodes {:head {...} :mouth {:parent :head ...} ...}}
 :face-2 {:nodes {:head {...} :mouth {:parent :head ...} ...}}}

:features
{:face-1/mouth {:subject :face-1 :timeline :face-1 :area :mouth
                :nodes [:mouth :mouth-in]}
 :face-2/mouth {:subject :face-2 :timeline :face-2 :area :mouth
                :nodes [:mouth :mouth-in]}}
```

The example omits ordinary ids, frame counts and channel details.

## What belongs where

- A **subject** identifies a source track and supplies shared measurement settings.
  Its timeline has the same id and contains its measured `:head`.
- A **feature** owns nodes in an explicitly named timeline. Its clip-level id is
  qualified when generated; ownership is read from fields, never parsed from ids.
- A **node** has a local name. Parents, stencils and pose groups use local names too.
- An **instance** places and retimes a drawing. Its pose tracks can hold one face's
  mouth while the other face continues moving.

Subject metadata and drawing timelines remain separate facts. Hand-drawn timelines
need no subject. Features retain explicit timeline references, so their locations
are not inferred from their labels.

Both filmed faces share one source-to-stage transform. Fitting them independently
would stack them at the center. Additional placement uses each instance's ordinary
channels. Cross-face draw order is the instances' `:z` order.

## Consequences

Regeneration updates the addressed timeline directly. There is no temporary swap
into `:main`, no special stage regeneration path, and no renaming of parents or
stencils. Composing a stage moves the take's root into the library and preserves
its child timelines. Settings appear once per tracked object, since all placements
read that same drawing.

Presence masks inside a subject use local feature names, matching measurement.
Freeze qualifies them when building block descriptors. Head and retained-source
blocks explicitly name their subject; otherwise two faces with identical detection
masks could produce different bytes under the same key. Analysis addresses also
include detection capacity and assignment settings, so old single-face detection
results cannot satisfy a new multi-face request.

Validation counts node ownership by `[timeline node]`. The server requires one
complete set of retained source roles **per subject**, rather than exactly three
blocks for the entire analysis.

Nested rectangles retain fractional sizes until rasterization. Rounding inside a
face timeline discarded small head-local pupils before the source-to-stage scale
was applied. This was a real rendering error missed by the earlier proposal's
coordinate-only benchmark; the regression now compares all mark extents as well.

## Verification and limits

`frontend/test/arthur/flow/multi_face_test.cljs` exercises two distinct subjects,
source block separation, detection and feature gaps, independent pose cuts and
regeneration, nested stage save/load, and equivalence to a flat single-face scene.
Existing geometry, raster, source, and regeneration tests cover the same paths.
`clips/tests/test_api.py` checks complete source roles per subject and immutability.
The browser suite checks rendering, pupils, playback, save/open, drawing and upload.

Assignment remains a nearest-centroid heuristic, with a version and distance gate
recorded in the analysis. Reordered detections, late arrivals and gaps are tested;
identity through crossings or long disappearances is not guaranteed. Assignment
happens before measurement, so correcting it requires measuring again.

This changes the freeze and retained-source contracts. It does not migrate older
flat captures; reanalyze their footage to use the new regeneration path. Existing
rendering and leaf codecs still understand their node/channel representation.

## Next steps

1. Commit the verified checkpoint: 319 frontend tests, 44 API tests, browser
   checks, and app/test builds passed. Builds reported no warnings.
2. Exercise real two-person footage, especially crossings, late arrivals and
   disappearances. Check assignment before treating the resulting geometry as
   evidence about the representation.
3. Build performance-pose selection and instance-scoped picture rates, reusing
   the existing held-frame lookup and pose groups.
4. Then build plate-drawing selection and independent tracing references.

The [timing handoff](timing-handoff.md) owns the detailed next implementation
sequence. Older flat captures need reanalysis unless a migration is separately
undertaken to preserve their authored edits.