101 lines
5.2 KiB
Markdown
101 lines
5.2 KiB
Markdown
# Multi-face representation
|
|
|
|
Status (2026-09-29): representation work complete. Reopen it for a concrete
|
|
requirement, rather than another round of abstract alternatives.
|
|
|
|
Implemented: each tracked face has a drawing timeline, placed by an ordinary
|
|
symbol instance. Timelines already provide local node names, independent playback,
|
|
and persistence. No new kind of scene container is needed.
|
|
|
|
```clojure
|
|
:timelines
|
|
{:main {:nodes {:root {:time {:mode :map :expose 2}}
|
|
:face {:parent :root :channels <source-to-stage placement>}
|
|
:face-1 {:kind :symbol :of :face-1 :parent :face :z "a0"}
|
|
:face-2 {:kind :symbol :of :face-2 :parent :face :z "a1"}}}
|
|
:face-1 {:nodes {:head {...} :mouth {:parent :head ...} ...}}
|
|
:face-2 {:nodes {:head {...} :mouth {:parent :head ...} ...}}}
|
|
|
|
:features
|
|
{:face-1/mouth {:subject :face-1 :timeline :face-1 :area :mouth
|
|
:nodes [:mouth :mouth-in]}
|
|
:face-2/mouth {:subject :face-2 :timeline :face-2 :area :mouth
|
|
:nodes [:mouth :mouth-in]}}
|
|
```
|
|
|
|
The example omits ordinary ids, frame counts and channel details.
|
|
|
|
## What belongs where
|
|
|
|
- A **subject** identifies a source track and supplies shared measurement settings.
|
|
Its timeline has the same id and contains its measured `:head`.
|
|
- A **feature** owns nodes in an explicitly named timeline. Its clip-level id is
|
|
qualified when generated; ownership is read from fields, never parsed from ids.
|
|
- A **node** has a local name. Parents, stencils and pose groups use local names too.
|
|
- An **instance** places and retimes a drawing. Its pose tracks can hold one face's
|
|
mouth while the other face continues moving.
|
|
|
|
Subject metadata and drawing timelines remain separate facts. Hand-drawn timelines
|
|
need no subject. Features retain explicit timeline references, so their locations
|
|
are not inferred from their labels.
|
|
|
|
Both filmed faces share one source-to-stage transform. Fitting them independently
|
|
would stack them at the center. Additional placement uses each instance's ordinary
|
|
channels. Cross-face draw order is the instances' `:z` order.
|
|
|
|
## Consequences
|
|
|
|
Regeneration updates the addressed timeline directly. There is no temporary swap
|
|
into `:main`, no special stage regeneration path, and no renaming of parents or
|
|
stencils. Composing a stage moves the take's root into the library and preserves
|
|
its child timelines. Settings appear once per tracked object, since all placements
|
|
read that same drawing.
|
|
|
|
Presence masks inside a subject use local feature names, matching measurement.
|
|
Freeze qualifies them when building block descriptors. Head and retained-source
|
|
blocks explicitly name their subject; otherwise two faces with identical detection
|
|
masks could produce different bytes under the same key. Analysis addresses also
|
|
include detection capacity and assignment settings, so old single-face detection
|
|
results cannot satisfy a new multi-face request.
|
|
|
|
Validation counts node ownership by `[timeline node]`. The server requires one
|
|
complete set of retained source roles **per subject**, rather than exactly three
|
|
blocks for the entire analysis.
|
|
|
|
Nested rectangles retain fractional sizes until rasterization. Rounding inside a
|
|
face timeline discarded small head-local pupils before the source-to-stage scale
|
|
was applied. This was a real rendering error missed by the earlier proposal's
|
|
coordinate-only benchmark; the regression now compares all mark extents as well.
|
|
|
|
## Verification and limits
|
|
|
|
`frontend/test/arthur/flow/multi_face_test.cljs` exercises two distinct subjects,
|
|
source block separation, detection and feature gaps, independent pose cuts and
|
|
regeneration, nested stage save/load, and equivalence to a flat single-face scene.
|
|
Existing geometry, raster, source, and regeneration tests cover the same paths.
|
|
`clips/tests/test_api.py` checks complete source roles per subject and immutability.
|
|
The browser suite checks rendering, pupils, playback, save/open, drawing and upload.
|
|
|
|
Assignment remains a nearest-centroid heuristic, with a version and distance gate
|
|
recorded in the analysis. Reordered detections, late arrivals and gaps are tested;
|
|
identity through crossings or long disappearances is not guaranteed. Assignment
|
|
happens before measurement, so correcting it requires measuring again.
|
|
|
|
This changes the freeze and retained-source contracts. It does not migrate older
|
|
flat captures; reanalyze their footage to use the new regeneration path. Existing
|
|
rendering and leaf codecs still understand their node/channel representation.
|
|
|
|
## Next steps
|
|
|
|
1. Commit the verified checkpoint: 319 frontend tests, 44 API tests, browser
|
|
checks, and app/test builds passed. Builds reported no warnings.
|
|
2. Exercise real two-person footage, especially crossings, late arrivals and
|
|
disappearances. Check assignment before treating the resulting geometry as
|
|
evidence about the representation.
|
|
3. Build performance-pose selection and instance-scoped picture rates, reusing
|
|
the existing held-frame lookup and pose groups.
|
|
4. Then build plate-drawing selection and independent tracing references.
|
|
|
|
The [timing handoff](timing-handoff.md) owns the detailed next implementation
|
|
sequence. Older flat captures need reanalysis unless a migration is separately
|
|
undertaken to preserve their authored edits.
|