arthur/docs/frame-selection.md

# Frame selection

Two mechanisms. One vocabulary. An earlier draft of this document claimed they
were one component used twice — because `suggestPlateFrames` in the old
`js/pipeline.js` and the never-built "performance poses" of
[timing-handoff](timing-handoff.md) looked like the same function — and that claim
is wrong. They share how a selection is *read* and how the hand overrides one.
They do not share how frames get chosen, because the two are answering questions
of different shapes.

[Time selection](time.md) is the floor both stand on: an output frame reads the
latest native frame at or before its time, and nothing rewrites the dense
measurements. That is a *cadence*: an answer with no opinion about content. It
cannot know that the one frame where the eye is fully closed is worth more than
its neighbours, so at 12fps out of 30 it drops that frame two times in three.
This document is how the picture gets an opinion.

## The one idea

**A selection is a set of frames chosen out of a dense measurement, and read by
holding the latest one at or before now.**

The holding half already exists and is already shared: `pose/held-frame` is called
by `trace/held-frame` and by `pose/source-frame`, which is the two sites agreeing
about reading. The hand half is shared too — see *Three layers* below. Choosing is
what differs.

| | Plate drawings (tracing) | Performance poses |
| --- | --- | --- |
| the question | which frames does an artist have to draw a head on? | which frames does the picture change a shape on? |
| the cost being managed | a person drawing | a pose looking wrong |
| signal | the measured head's motion | — none; a stored cut |
| the baseline it improves on | drawing on 2s | the cadence, or the exposure grid |
| the shape of the answer | a non-uniform set out of dense | the same grid, nudged |
| lives on | the face's `:head` `:trace` | the instance's `:playback :tracks` |
| hand edit today | `trace/toggle-frame` | `pose/put-cut` / `pose/remove-cut` |
| UI today | `params/trace-keys` | **none** |
| proposes today | **nothing** | **nothing** |

## Why they are not one function

A plate selection has to be **non-uniform**, and that is the whole reason it
exists. A head still for sixty frames and then whipping across in ten wants two
drawings for the first stretch and eight for the second. Drawing on 2s gives
thirty-five drawings, most of them identical, and no amount of nudging a uniform
grid will produce the distribution that is wanted — the spacing itself is the
answer. That is what the prototype's walk was for, and it is why a cost knob
(`:tolerance`) belongs on this side: the artist is buying drawings.

A performance selection is **not choosing sparseness at all**. The output rate or
the exposure setting has already chosen it. The question left over is only *which*
native frame each already-decided slot reads, and the failure it fixes is narrow:
a slot landing one or two frames off the closure. Nudging the grid is the right
size of answer, and there is nothing for a tolerance to mean.

There is a second, harder reason, and it is the one that settles it:

**A selection cannot put a frame on screen that the output grid never samples.**
At 12fps out of 30, output frame 5 reads native 12 and output frame 6 reads native
15. A closure at native 13 is *between* them. Protecting frame 13 in a set of
kept frames makes it available and makes it the frame held across 13 and 14 in
native space — and at a 12fps output it still never appears, exactly as
[time.md](time.md) says: an event between output frames cannot create an extra
frame in a 12fps output. Only moving what output frame 6 reads can show it. So
the performance side has to act on the grid, not on a set beside it.

## Three layers, and the middle one is derived

The trap this is designed around is stated in
[timing-handoff](timing-handoff.md) and is worth repeating because it is the
only hard rule here:

> Store manual edits separately from generated proposals so changing the rate or
> tolerance retains hand decisions.

So a selection is:

```clojure
{:policy {:tolerance 0.02}   ; what the proposer was asked for
 :keep   #{47}               ; frames the hand insists on
 :drop   #{30}}              ; frames the hand refuses
```

and the effective set is `(proposed ∪ keep) \ drop`, always containing frame 0.

`:keep` and `:drop` are the document. The proposal is not: it is recomputed from
`:policy` and the dense signal whenever either changes. **Re-suggesting at a new
tolerance must never cost somebody their pinned blink**, and that is the entire
reason the hand decisions are stored as their own two sets rather than as the
resulting frame list.

This layering is the part that really is shared. On the performance side there is
no `:policy` worth storing — the grid is the policy — but `:keep` and `:drop` mean
exactly what they mean on the plate side, and `select/effective` is the one
implementation for both. A preserve mark *is* a keep.

### Materialise the result, do not derive it on the render path

The effective set is written back to where each site already reads it —
`:trace :frames`, or the pose track — so that every existing reader is untouched
and nothing on the per-frame path has to open a dense block. Proposing is a
command, not a subscription. `ch/value-at` allocates per call and says so; that
is fine for a button press over a few hundred frames and would not be fine at
30fps.

This means the stored frame list is redundant with `policy + keep + drop`. That
is deliberate and it is the cheap direction of the trade: a stale list is
recoverable by pressing Suggest again, and a dense read per node per frame is
not recoverable at all.

## The plate selection: a non-uniform chooser

`arthur.domain.select`, built — see *What is revertible* for the one part of it
that is not yet wanted. Pure, no store access, no clip access: the site hands it a
signal it has already read.

```clojure
(defn propose
  "Frames worth keeping out of `n`, given `signal`."
  [n signal {:keys [tolerance protect]}])

(defn effective
  "`proposed` with the hand's decisions applied. Always contains 0."
  [proposed keep drop])
```

`propose` is the prototype's walk, generalised off landmarks:

1. keep frame 0, make it the anchor;
2. settle the protected frames from `:protect` *before* walking;
3. for each later frame, keep it when it is protected, or when
   `distance(anchor, f) > tolerance`. Either way it becomes the anchor.

`distance` is the max absolute difference over components, so a signal of
landmark pairs and a signal of one number both work without the caller saying
which it handed over. Every sample in a signal is the same width, and a signal of
two widths is refused: comparing the prefix two samples happen to share would let
a reader that drops a component read as no movement at all.

**A protected frame anchors the walk like any other kept frame**, which is why
protection is settled first and is not unioned onto the walk's result. The anchor
is what is on screen; once a protected frame is kept the viewer is looking at it,
so measuring the next frame's drift from a frame no longer displayed is wrong.

**An absent measurement is `nil`, and converting to that is the reader's job.** A
frame where the face was not found says nothing about the signal: it cannot move
the anchor and it is not a frame worth keeping. `trace/measured-local` already
returns nil there. A protected frame with no measurement is still kept — a plate
frame is a frame somebody draws on whether or not the detector found a face.

**A non-finite tolerance falls back to nought.** A cleared slider reads as NaN and
every comparison against NaN is false, which taken literally proposes frame 0
alone and collapses the whole take to one drawing. Nought proposes every frame
that changes, which is merely the baseline back again: wrong in a way somebody can
see and undo.

### The signal

The head's measured transform, applied to a fixed reference quad, giving
displacement in stage units — so a tolerance means "the head has moved this far"
and is a number a person can reason about. `trace/measured-local` already builds
that matrix per frame and is private; make it public rather than writing a second
one. Map it over the frames and transform four corners through `node/apply-pt!`.

The quad's size is a real parameter hiding in the word "fixed": it sets how much
rotation registers against translation. Give it a name and a comment rather than
an inline literal.

Do not reach for raw landmarks. The prototype used them because it had them lying
around; the transform is what the drawing actually follows, it is already on the
node, and it is three channels instead of a block.

### The upgrade path, and do not start here

The greedy walk is order-dependent and slightly suboptimal. The optimal version
is a dynamic program — choose `k` frames minimising held-reconstruction error,
which is textbook segmented least squares and is O(n²k), nothing at n≈300 — and
it keeps extrema *for free*, because an extremum is exactly where a zero-order
hold is most wrong.

Build the greedy one first anyway. It is proven, it shipped in the prototype, and
having two implementations to compare is how the DP gets tested. Swap it behind
`propose` afterwards, where the signature already permits it.

Most of `select_test` pins the greedy walk's exact output, deliberately, for that
comparison — so expect to rewrite those expectations when the DP lands, and keep
them as greedy-specific tests rather than deleting them. The assertion that is a
*spec* rather than a pinned vector, and should be written on this side before the
swap, is:

> reading the signal through the selection, held, never differs from the dense
> measurement by more than `tolerance`

That is what makes the tolerance number mean something to a person. It is true of
the greedy walk by construction and it is what the DP optimises, so it survives
the swap untouched.

## The performance selection: a preserve-snap on the grid

Not built. The rule is ten lines; getting the two halves of it into the same place
is the work. An earlier draft of this section said "about thirty lines" and that
was understated — see *Where it goes* below.

A grid slot already picks a native frame — `cadence/frame` for the output rate, or
the exposure fold for a deliberate hold at full rate. Write `d(k)` for the native
frame slot `k` defaults to. Slot `k` is the first slot to cover everything in
`(d(k-1), d(k)]`, and the frames strictly inside that interval are the ones the
grid shows to nobody. So:

> **Slot `k` reads the latest preserved frame in `(d(k-1), d(k)]`, and `d(k)` when
> there is none.**

That is the whole mechanism. It recovers a dropped frame out of the slot's own gap,
it can never read a frame another slot already showed, and it cannot reach past
`d(k)`.

**Snap backward only, never forward**, and note which direction that actually is,
because it is easy to get backwards. The closure at native 13 in a 12-from-30
output is recovered by slot **6** — whose default is 15 — reading 13. It is *not*
recovered by slot 5, whose default is 12, reaching forward to 13: slot 5's instant
is 5/12s = 0.4167s and native 13's is 13/30s = 0.4333s, so that would show the
closure 17ms before the mouth shut. `cadence/frame`'s contract is the latest native
frame at or before the slot's time and `cadence_test` asserts
`selected <= f*native/grid` over every grid and native pair, so reaching forward
breaks a tested invariant as well as the no-lead rule. Reading 13 at slot 6 shows
the closure two native frames late, which is the same lateness every hold already
has.

**The marks come from a cut that is already stored.** `flow/freeze` computes both
closures and keys them as `[:vis]`, with the thresholding and hysteresis already
decided:

- the mouth, from the aperture relative to the take's peak — `[:vis]` on
  `:mouth-in`, provenance `:roto/mouth-aperture` (`freeze.cljs:473`);
- the eyes, from `condition/resolve-blink` with its cut, dwell and hold — `[:vis]`
  keyed per eye part, provenance `:roto/blink` (`freeze.cljs:533`).

So "preserve the frames the cut says shut" reads what the document already holds.
No new signal, no new dense track, no threshold decided twice. Brows have no
closure and get no marks, which is correct: there is no extreme brow position
worth protecting.

**Precompute the mark set when the resolver is built**, the way `pose/prepare` and
`trace/prepare` already do (`symbol.cljs:420`, `:470`, `:536`). A `[:vis]` channel
is keys, not dense, so reading it per node per frame would be cheap — but the snap
also needs the marks sorted for a backward lookup, and building that per frame is
the one thing [animation-model.md](animation-model.md) and the render-path note
above both forbid.

### Where it goes, and why it is not a one-liner

The rule needs two things that currently live at opposite ends of the resolver:

- **the slot interval** `(d(k-1), d(k)]`, which needs the output slot index and the
  fps ratio. Both exist at `clip.cljs:261`, the single place the grid becomes a
  native frame: `(cadence/frame f (or (:grid-fps opts) (:fps clip)) (fps clip sid))`.
- **the marks**, which are per pose group, and so belong where group identity
  exists: `symbol/base-channel-frame` (`symbol.cljs:399`), whose `:pose-sampled?`
  branch already calls `pose/source-frame` with `(js/Math.floor lf)` as the default
  pose. **That default is the snap's seat.** Replacing it leaves an explicit hand
  cut winning over a snap, which is correct — manual precedence is absolute — and
  costs no new plumbing on the pose side, because per-group choices are already
  threaded and already prepared.

By the time control reaches `base-channel-frame` there is only `lf`, a native local
frame that placement and retime have already been through, so the slot interval
cannot be recovered there. It has to be threaded down from `clip.cljs:261`
alongside the frame. That is the actual work of this step: two namespaces' internal
signatures, not a drop-in.

**Do not take the shortcut of snapping at `clip.cljs:261` itself.** It is right
there, it needs no threading, and it is wrong: one native frame per output frame
means the *whole picture* reads 13 instead of 15, so the head goes two frames stale
for one output frame to fix the mouth. At 12fps that is a 67ms hitch on a moving
head, and it fights the trace selection, which has its own opinion about which head
frame to show. The snap is per group because the thing being recovered is one
group's closure.

### Why not an aperture signal

An earlier draft had this side read the group's aperture as a single-component
signal and hand it to `propose` with `:protect :extrema`. It cannot. The aperture
is the separation of landmarks 13 and 14, at positions 5 and 15 of the 20-slot
`LIPS-INNER` ring, and `freeze/rings->flat` subsamples the ring to the `verts`
budget: those two positions survive only when `verts` is a multiple of four. At
`verts` 6, 10, 14 and 18 — all legal, all even — they are not in the stored data
at all. `ring/subsample-slots` used to claim otherwise and has been corrected.

`flow/measure/mouth` does compute the exact scalar and `freeze` does throw it away
after thresholding, so storing it was an option. Reading the cut is strictly less
work and decides nothing twice.

## How the two interact

They are keyed in **the same frame space**: `clip/resolver` hands an instance's
`:playback :tracks` down into the child it places, and `symbol/base-channel-frame`
reads both the trace and the pose choices at `lf`, the node's local frame inside
that face. A trace frame and a pose cut are the same kind of number.

What differs is the owner, and that asymmetry is load-bearing:

- the **trace** is the face's, on its `:head` — every instance of that face shares
  it, because it says how the drawings were made;
- the **pose tracks** are the instance's — two placements of one face can be
  timed differently.

### The hazard, which is already written down

[animation-model.md](animation-model.md) states it for exposure and it is the
same hazard here:

> a head cutting on odd frames against a mouth cutting on even ones reads as two
> performances

**It is not a correctness problem.** The mouth is a child of `:head` and its
geometry is stored head-local, so a mouth from frame 17 composed onto a head held
at frame 12 is exactly lip-sync on a held drawing — the decomposition already
decoupled them and nothing is geometrically wrong. The problem is perceptual, and
perceptual problems want a constraint rather than a repair.

### Nest them, do not couple them

The two are not peers. One is coarse and expensive — the prototype's own comment
says it: *"The cost being managed is an artist drawing a head, which is why the
signal is head pose and not the mouth — the mouth is traced and free."* The other
is fine and cheap.

So the rule is a subset, in one direction only:

**Every kept plate frame is a preserved frame of the performance selection.**

When the drawing changes, the performance changes with it, so the two can never
cut against each other on neighbouring frames. The mouth stays free to change on
frames where the head does not, which is what shooting a held drawing with a live
mouth *is*.

This is why the preserve set is a set and not a closure predicate: plate frames
and shut-mouth frames go into the same pile, and the snap does not care which is
which. It needs no new mechanism on either side.

The reverse is a suggestion and never automatic. A mouth closure is a reasonable
place to want a new drawing, but proposing one spends somebody's afternoon. Offer
it; do not take it.

It also composes with `:origin` for free. A head on `:continuous` has opted out
of its own selection, so there are no plate frames to preserve and the constraint
is vacuous — which is correct, because a continuously moving head cannot cut
against anything.

### Between the kept frames is a third shared field

`:trace :origin` is not a tracing setting. It is the answer to *what happens
between kept frames*, and the plate selection has to answer it:

- `:continuous` — ignore the selection for this purpose and read the frame you
  are on,
- `:keys` — jump to each kept frame and hold it, a hold and not a tween,
- `:start` — hold the first forever.

The performance side does not need the field: a snap picks which frame a slot
reads and the grid does the holding, so `:keys` is the only behaviour there is.
Do not rename `:origin` to match anything: for a head it genuinely means where the
face's origin goes, and a saved field in a shipped UI is not worth churning for a
vocabulary tidy.

## The modes

One setting, two states, and **"manual" is not a third state.**

- **off** — the cadence alone, which is what ships today. The output frame
  reads the latest native frame at or before it, and nothing has an opinion.
- **smart frame picking** — on the plate side the proposal is live, and `:policy`
  holds the tolerance; on the performance side the snap is active.

Hand keeps and drops apply in **both** states, which is why they are not a mode:
turning smart picking off must not throw away the frames somebody pinned, and
pinning a frame with smart picking off is a perfectly reasonable thing to want.
Manual precedence is absolute — a drop beats a proposal, always.

The name on the toggle should be the same word in both sections. "Smart frame
picking" is fine. What it must not be is two different names for the one idea,
which is how these became two features the first time. That the two sections are
now backed by different code is an implementation fact and must not reach the UI.

## The UI

One component rendered twice, in `ui/params.cljs`:

```
smart frame picking   [ off | on ]
tolerance             [ ----•------- ]   0.02      (plate section only)
                      [ Suggest ]
frames   0  12  30  47*  61        (* = kept by hand, strikethrough = dropped)
```

The frame strip already exists in miniature — `params/trace-keys` draws the trace
keys as seek buttons (`params.cljs:386`). Lift it into a shared component and
give it three affordances: click to seek, a modifier to pin, a modifier to drop.
A pinned frame and a proposed frame must be visually distinct, because "will this
survive me moving the slider?" is the question the strip exists to answer.

Render it in two sections:

- **`tracing · <face>`** (`params.cljs:426`), beside the existing origin row,
  with the tolerance slider and Suggest.
- **`performance · <group>`** — new, on an instance's inspector, one per pose
  group the placed symbol has. No tolerance: the strip shows the preserved frames
  and the grid slots that snapped to them.

## Order to build it

1. **`domain/select` with the greedy walk, and its tests.** **Done**, in commit
   `02069e8`. Pure, no store, no clip. Note that the extrema part of it is not
   wanted by anything below — see *What is revertible*.
2. **The preserve-snap**, pulled forward ahead of the plate side because it is the
   half with no UI and nothing proposing today, and because it needs nothing from
   the plate side except a fold that can land last. Three pieces: the mark set from
   the stored `[:vis]` cuts, built in a prepare step beside `pose/prepare`; the
   slot interval threaded from `clip.cljs:261`; the rule itself, seated in
   `base-channel-frame`'s default pose. The test that matters is the same synthetic
   case `select_test` uses — a one-frame closure at native 13 that a 12-from-30
   grid drops — asserted end to end this time: the resolver shows a shut mouth on
   exactly one output frame, and the head's frame does not move while it happens.
   Write it first.
3. **The plate signal reader.** `trace/head-signal`: make `trace/measured-local`
   public, map it over the frames, four corners through `node/apply-pt!`. Test
   that it returns a vector of the right length, that a motionless take proposes
   `[0]`, and that a stretch where the face was not found becomes `nil` rather
   than a pose, a zero or a gap in the vector. Add the held-reconstruction
   invariant from *The upgrade path* here, since this is the first place a real
   signal exists to assert it over.
4. **Storage for the plate selection.** `:policy`/`:keep`/`:drop` beside
   `:trace :frames`. Extend `leaf/leaves` and the key whitelists in the same
   commit — a field without a leaf saves silently and comes back missing, which
   is the one bug persistence must not be able to have. Round-trip test.
5. **Re-suggest preserves hand decisions.** Propose at one tolerance, pin a frame,
   drop a frame, propose at another, assert both survive. Settle here whether the
   hand's `:keep` is also fed to `propose` as `:protect`: under the layering as
   written it is not, so a pinned frame does not re-anchor the walk even though it
   is on screen, which contradicts the anchor rule above. It is the one live
   caller for `propose`'s `:protect` frames.
6. **Fold the plate frames into the mark set**, which is the nesting rule and is
   one line once both sides exist.
7. **The shared UI component**, then its two mountings.

## What is revertible, and where it is

Commit `02069e8` contains one part that **nothing below asks for**: extrema
detection. Specifically `segments`, `turns`, the `:extrema` branch of `propose`,
the multi-component refusal that exists only to guard it, and three tests —
`a-one-frame-closure-survives-only-because-it-is-protected`,
`extrema-are-turns-worth-more-than-the-tolerance` and
`extrema-are-refused-on-a-multi-component-signal`.

It has no caller because plate selections take no extrema by design — displacement
is the whole story for a head — and the performance side reads a stored cut
instead of finding extrema in a signal. It is correct, tested and speculative.

Revert it if the shape above holds. Keep it if either of these turns out to be
wanted: a group whose extreme is not a closure and therefore has no `[:vis]` cut
to read (a mouth at its widest, a head at the top of a nod), or a take whose mouth
never shuts far enough to cross `aperture-cut`, where a peak-relative threshold
marks nothing and an extremum would still find the most closed frame. Neither is
asked for today. The DP in *The upgrade path* gets extrema for free regardless, so
reverting costs nothing that cannot be had again more cheaply.

## Traps

- **Do not let Suggest write `:keep`.** The proposal and the hand are different
  layers; collapsing them is the bug this whole shape exists to avoid, and it
  will look like it works right up until somebody moves the tolerance slider.
- **Do not snap a grid slot forward.** Every hold and every pick is "at or
  before". Showing a closure before the mouth shut is a lead, which is a different
  control for a different reason.
- **Do not quantise a plate selection onto the output grid.** A kept frame is a
  native frame and lands where it lands. [time.md](time.md) is explicit that an
  event between output frames appears on the next one; forcing kept frames onto
  the grid would re-create the problem the selection exists to solve. The snap is
  the opposite operation and is on the other side of the fence: it moves the grid's
  pick, never the kept frame.
- **Do not read the aperture pair off a subsampled ring.** Positions 5 and 15 of
  `LIPS-INNER` survive only at `verts` divisible by four.
- **Do not thin the plate selection with the mouth's.** Two scopes, two owners,
  and the nesting runs one way only: plate frames preserve performance frames,
  never the reverse.
- **Do not give the performance side a tolerance.** The grid has already chosen
  the sparseness. A second knob there would be a control with nothing to control.
- **A skipped frame, a hidden feature and an absent measurement remain three
  different facts.** A selection says nothing about visibility and nothing about
  whether a face was found. Note that the preserve marks are *derived from* a
  visibility cut, which makes this easy to blur: the cut says the interior is
  hidden, the mark says the frame is worth landing on, and one is not the other.

## Not in scope

- Automatic *grouping* of related parts beyond the existing `:pose-group`.
  Related parts must share one selection — a mouth outline, its interior, the
  teeth and the generated visibility reading different frames is the bug that
  grouping prevents — and `:pose-group` is where the grouping already lives.
- A per-instance request for a different tolerance than the symbol's. The
  resolver threads no such option today and should not grow one until something
  needs it.
- Variable frame rate. [time.md](time.md) assumes constant fps and so does this;
  a VFR source needs presentation timestamps before any of this means anything.