noe/arthur
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
arthur/docs/frame-selection.md
Your Name cc42155ffa The plate side chooses frames; the performance side nudges the grid 
Corrects this plan's central claim. It said one component used twice, because
the prototype's `suggestPlateFrames` and timing-handoff's performance poses
looked like the same function. They are not.

A plate selection has to be NON-UNIFORM — that is the entire reason it exists. A
head still for sixty frames and then whipping across in ten wants two drawings
and then eight, and no nudging of a uniform grid produces that distribution,
because the spacing itself is the answer. A tolerance belongs here: the artist is
buying drawings.

A performance selection is not choosing sparseness at all; the output rate or the
exposure setting already did. What is left is which native frame each decided
slot reads, so nudging the grid is the right size of answer and a tolerance would
be a knob with nothing to control.

And the harder reason, which settles it: a selection cannot put a frame on screen
that the output grid never samples. At 12fps out of 30 a closure at native 13
falls between output frames 5 and 6, so keeping 13 in a set makes it available
and never shows it — exactly what time.md says about an event between output
frames. Only moving what output frame 6 reads can show it.

So the performance half is a backward-only snap of the grid's own pick toward a
preserved frame, about thirty lines plus a prepare step, and its marks come from
cuts `flow/freeze` already stores: the mouth's aperture threshold and the eyes'
`resolve-blink` with its hysteresis. No aperture signal, no new dense track, no
threshold decided twice. The nesting survives unchanged, because a kept plate
frame is just another mark in the same pile.

Records why the aperture signal is impossible as drafted — positions 5 and 15 of
LIPS-INNER survive `freeze/rings->flat` only at verts divisible by four — and
names commit 02069e8 as the one holding the now-uncalled extrema detection, with
the two conditions under which it would be wanted again.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-10-01 10:14:54 -04:00

453 lines
23 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

# Frame selection
Two mechanisms. One vocabulary. An earlier draft of this document claimed they
were one component used twice — because `suggestPlateFrames` in the old
`js/pipeline.js` and the never-built "performance poses" of
[timing-handoff](timing-handoff.md) looked like the same function — and that claim
is wrong. They share how a selection is *read* and how the hand overrides one.
They do not share how frames get chosen, because the two are answering questions
of different shapes.
[Time selection](time.md) is the floor both stand on: an output frame reads the
latest native frame at or before its time, and nothing rewrites the dense
measurements. That is a *cadence*: an answer with no opinion about content. It
cannot know that the one frame where the eye is fully closed is worth more than
its neighbours, so at 12fps out of 30 it drops that frame two times in three.
This document is how the picture gets an opinion.
## The one idea
**A selection is a set of frames chosen out of a dense measurement, and read by
holding the latest one at or before now.**
The holding half already exists and is already shared: `pose/held-frame` is called
by `trace/held-frame` and by `pose/source-frame`, which is the two sites agreeing
about reading. The hand half is shared too — see *Three layers* below. Choosing is
what differs.
| | Plate drawings (tracing) | Performance poses |
| --- | --- | --- |
| the question | which frames does an artist have to draw a head on? | which frames does the picture change a shape on? |
| the cost being managed | a person drawing | a pose looking wrong |
| signal | the measured head's motion | — none; a stored cut |
| the baseline it improves on | drawing on 2s | the cadence, or the exposure grid |
| the shape of the answer | a non-uniform set out of dense | the same grid, nudged |
| lives on | the face's `:head` `:trace` | the instance's `:playback :tracks` |
| hand edit today | `trace/toggle-frame` | `pose/put-cut` / `pose/remove-cut` |
| UI today | `params/trace-keys` | **none** |
| proposes today | **nothing** | **nothing** |
## Why they are not one function
A plate selection has to be **non-uniform**, and that is the whole reason it
exists. A head still for sixty frames and then whipping across in ten wants two
drawings for the first stretch and eight for the second. Drawing on 2s gives
thirty-five drawings, most of them identical, and no amount of nudging a uniform
grid will produce the distribution that is wanted — the spacing itself is the
answer. That is what the prototype's walk was for, and it is why a cost knob
(`:tolerance`) belongs on this side: the artist is buying drawings.
A performance selection is **not choosing sparseness at all**. The output rate or
the exposure setting has already chosen it. The question left over is only *which*
native frame each already-decided slot reads, and the failure it fixes is narrow:
a slot landing one or two frames off the closure. Nudging the grid is the right
size of answer, and there is nothing for a tolerance to mean.
There is a second, harder reason, and it is the one that settles it:
**A selection cannot put a frame on screen that the output grid never samples.**
At 12fps out of 30, output frame 5 reads native 12 and output frame 6 reads native
15. A closure at native 13 is *between* them. Protecting frame 13 in a set of
kept frames makes it available and makes it the frame held across 13 and 14 in
native space — and at a 12fps output it still never appears, exactly as
[time.md](time.md) says: an event between output frames cannot create an extra
frame in a 12fps output. Only moving what output frame 6 reads can show it. So
the performance side has to act on the grid, not on a set beside it.
## Three layers, and the middle one is derived
The trap this is designed around is stated in
[timing-handoff](timing-handoff.md) and is worth repeating because it is the
only hard rule here:
> Store manual edits separately from generated proposals so changing the rate or
> tolerance retains hand decisions.
So a selection is:
```clojure
{:policy {:tolerance 0.02} ; what the proposer was asked for
:keep #{47} ; frames the hand insists on
:drop #{30}} ; frames the hand refuses
```
and the effective set is `(proposed ∪ keep) \ drop`, always containing frame 0.
`:keep` and `:drop` are the document. The proposal is not: it is recomputed from
`:policy` and the dense signal whenever either changes. **Re-suggesting at a new
tolerance must never cost somebody their pinned blink**, and that is the entire
reason the hand decisions are stored as their own two sets rather than as the
resulting frame list.
This layering is the part that really is shared. On the performance side there is
no `:policy` worth storing — the grid is the policy — but `:keep` and `:drop` mean
exactly what they mean on the plate side, and `select/effective` is the one
implementation for both. A preserve mark *is* a keep.
### Materialise the result, do not derive it on the render path
The effective set is written back to where each site already reads it —
`:trace :frames`, or the pose track — so that every existing reader is untouched
and nothing on the per-frame path has to open a dense block. Proposing is a
command, not a subscription. `ch/value-at` allocates per call and says so; that
is fine for a button press over a few hundred frames and would not be fine at
30fps.
This means the stored frame list is redundant with `policy + keep + drop`. That
is deliberate and it is the cheap direction of the trade: a stale list is
recoverable by pressing Suggest again, and a dense read per node per frame is
not recoverable at all.
## The plate selection: a non-uniform chooser
`arthur.domain.select`, built — see *What is revertible* for the one part of it
that is not yet wanted. Pure, no store access, no clip access: the site hands it a
signal it has already read.
```clojure
(defn propose
"Frames worth keeping out of `n`, given `signal`."
[n signal {:keys [tolerance protect]}])
(defn effective
"`proposed` with the hand's decisions applied. Always contains 0."
[proposed keep drop])
```
`propose` is the prototype's walk, generalised off landmarks:
1. keep frame 0, make it the anchor;
2. settle the protected frames from `:protect` *before* walking;
3. for each later frame, keep it when it is protected, or when
`distance(anchor, f) > tolerance`. Either way it becomes the anchor.
`distance` is the max absolute difference over components, so a signal of
landmark pairs and a signal of one number both work without the caller saying
which it handed over. Every sample in a signal is the same width, and a signal of
two widths is refused: comparing the prefix two samples happen to share would let
a reader that drops a component read as no movement at all.
**A protected frame anchors the walk like any other kept frame**, which is why
protection is settled first and is not unioned onto the walk's result. The anchor
is what is on screen; once a protected frame is kept the viewer is looking at it,
so measuring the next frame's drift from a frame no longer displayed is wrong.
**An absent measurement is `nil`, and converting to that is the reader's job.** A
frame where the face was not found says nothing about the signal: it cannot move
the anchor and it is not a frame worth keeping. `trace/measured-local` already
returns nil there. A protected frame with no measurement is still kept — a plate
frame is a frame somebody draws on whether or not the detector found a face.
**A non-finite tolerance falls back to nought.** A cleared slider reads as NaN and
every comparison against NaN is false, which taken literally proposes frame 0
alone and collapses the whole take to one drawing. Nought proposes every frame
that changes, which is merely the baseline back again: wrong in a way somebody can
see and undo.
### The signal
The head's measured transform, applied to a fixed reference quad, giving
displacement in stage units — so a tolerance means "the head has moved this far"
and is a number a person can reason about. `trace/measured-local` already builds
that matrix per frame and is private; make it public rather than writing a second
one. Map it over the frames and transform four corners through `node/apply-pt!`.
The quad's size is a real parameter hiding in the word "fixed": it sets how much
rotation registers against translation. Give it a name and a comment rather than
an inline literal.
Do not reach for raw landmarks. The prototype used them because it had them lying
around; the transform is what the drawing actually follows, it is already on the
node, and it is three channels instead of a block.
### The upgrade path, and do not start here
The greedy walk is order-dependent and slightly suboptimal. The optimal version
is a dynamic program — choose `k` frames minimising held-reconstruction error,
which is textbook segmented least squares and is O(n²k), nothing at n≈300 — and
it keeps extrema *for free*, because an extremum is exactly where a zero-order
hold is most wrong.
Build the greedy one first anyway. It is proven, it shipped in the prototype, and
having two implementations to compare is how the DP gets tested. Swap it behind
`propose` afterwards, where the signature already permits it.
Most of `select_test` pins the greedy walk's exact output, deliberately, for that
comparison — so expect to rewrite those expectations when the DP lands, and keep
them as greedy-specific tests rather than deleting them. The assertion that is a
*spec* rather than a pinned vector, and should be written on this side before the
swap, is:
> reading the signal through the selection, held, never differs from the dense
> measurement by more than `tolerance`
That is what makes the tolerance number mean something to a person. It is true of
the greedy walk by construction and it is what the DP optimises, so it survives
the swap untouched.
## The performance selection: a preserve-snap on the grid
Not built. About thirty lines, plus a prepare step.
A grid slot already picks a native frame — `cadence/frame` for the output rate, or
the exposure fold for a deliberate hold at full rate. The change is: when a slot's
pick has a *preserved* frame within the span of native frames that slot is
responsible for, read the preserved frame instead.
**Snap backward only, never forward.** `cadence/frame`'s contract is the latest
native frame at or before the slot's time, and `cadence_test` asserts
`selected <= f*native/grid` over every grid and native pair. Snapping forward
would show a closure before the mouth shut, which is a lead — a different control,
applied for a different reason, after this one. A backward snap shows the closure
up to a frame or two late, which is the same lateness a hold already has.
**The marks come from a cut that is already stored.** `flow/freeze` computes both
closures and keys them as `[:vis]`, with the thresholding and hysteresis already
decided:
- the mouth, from the aperture relative to the take's peak — `[:vis]` on
`:mouth-in`, provenance `:roto/mouth-aperture` (`freeze.cljs:473`);
- the eyes, from `condition/resolve-blink` with its cut, dwell and hold — `[:vis]`
keyed per eye part, provenance `:roto/blink` (`freeze.cljs:533`).
So "preserve the frames the cut says shut" reads what the document already holds.
No new signal, no new dense track, no threshold decided twice. Brows have no
closure and get no marks, which is correct: there is no extreme brow position
worth protecting.
**Precompute the mark set when the resolver is built**, the way `pose/prepare` and
`trace/prepare` already do. A `[:vis]` channel is keys, not dense, so reading it
per node per frame would be cheap — but the snap also needs the marks as a set
with a span lookup, and building that per frame is the one thing
[animation-model.md](animation-model.md) and the render-path note above both
forbid.
### Why not an aperture signal
An earlier draft had this side read the group's aperture as a single-component
signal and hand it to `propose` with `:protect :extrema`. It cannot. The aperture
is the separation of landmarks 13 and 14, at positions 5 and 15 of the 20-slot
`LIPS-INNER` ring, and `freeze/rings->flat` subsamples the ring to the `verts`
budget: those two positions survive only when `verts` is a multiple of four. At
`verts` 6, 10, 14 and 18 — all legal, all even — they are not in the stored data
at all. `ring/subsample-slots` used to claim otherwise and has been corrected.
`flow/measure/mouth` does compute the exact scalar and `freeze` does throw it away
after thresholding, so storing it was an option. Reading the cut is strictly less
work and decides nothing twice.
## How the two interact
They are keyed in **the same frame space**: `clip/resolver` hands an instance's
`:playback :tracks` down into the child it places, and `symbol/base-channel-frame`
reads both the trace and the pose choices at `lf`, the node's local frame inside
that face. A trace frame and a pose cut are the same kind of number.
What differs is the owner, and that asymmetry is load-bearing:
- the **trace** is the face's, on its `:head` — every instance of that face shares
it, because it says how the drawings were made;
- the **pose tracks** are the instance's — two placements of one face can be
timed differently.
### The hazard, which is already written down
[animation-model.md](animation-model.md) states it for exposure and it is the
same hazard here:
> a head cutting on odd frames against a mouth cutting on even ones reads as two
> performances
**It is not a correctness problem.** The mouth is a child of `:head` and its
geometry is stored head-local, so a mouth from frame 17 composed onto a head held
at frame 12 is exactly lip-sync on a held drawing — the decomposition already
decoupled them and nothing is geometrically wrong. The problem is perceptual, and
perceptual problems want a constraint rather than a repair.
### Nest them, do not couple them
The two are not peers. One is coarse and expensive — the prototype's own comment
says it: *"The cost being managed is an artist drawing a head, which is why the
signal is head pose and not the mouth — the mouth is traced and free."* The other
is fine and cheap.
So the rule is a subset, in one direction only:
**Every kept plate frame is a preserved frame of the performance selection.**
When the drawing changes, the performance changes with it, so the two can never
cut against each other on neighbouring frames. The mouth stays free to change on
frames where the head does not, which is what shooting a held drawing with a live
mouth *is*.
This is why the preserve set is a set and not a closure predicate: plate frames
and shut-mouth frames go into the same pile, and the snap does not care which is
which. It needs no new mechanism on either side.
The reverse is a suggestion and never automatic. A mouth closure is a reasonable
place to want a new drawing, but proposing one spends somebody's afternoon. Offer
it; do not take it.
It also composes with `:origin` for free. A head on `:continuous` has opted out
of its own selection, so there are no plate frames to preserve and the constraint
is vacuous — which is correct, because a continuously moving head cannot cut
against anything.
### Between the kept frames is a third shared field
`:trace :origin` is not a tracing setting. It is the answer to *what happens
between kept frames*, and the plate selection has to answer it:
- `:continuous` — ignore the selection for this purpose and read the frame you
are on,
- `:keys` — jump to each kept frame and hold it, a hold and not a tween,
- `:start` — hold the first forever.
The performance side does not need the field: a snap picks which frame a slot
reads and the grid does the holding, so `:keys` is the only behaviour there is.
Do not rename `:origin` to match anything: for a head it genuinely means where the
face's origin goes, and a saved field in a shipped UI is not worth churning for a
vocabulary tidy.
## The modes
One setting, two states, and **"manual" is not a third state.**
- **off** — the cadence alone, which is what ships today. The output frame
reads the latest native frame at or before it, and nothing has an opinion.
- **smart frame picking** — on the plate side the proposal is live, and `:policy`
holds the tolerance; on the performance side the snap is active.
Hand keeps and drops apply in **both** states, which is why they are not a mode:
turning smart picking off must not throw away the frames somebody pinned, and
pinning a frame with smart picking off is a perfectly reasonable thing to want.
Manual precedence is absolute — a drop beats a proposal, always.
The name on the toggle should be the same word in both sections. "Smart frame
picking" is fine. What it must not be is two different names for the one idea,
which is how these became two features the first time. That the two sections are
now backed by different code is an implementation fact and must not reach the UI.
## The UI
One component rendered twice, in `ui/params.cljs`:
```
smart frame picking [ off | on ]
tolerance [ ----•------- ] 0.02 (plate section only)
[ Suggest ]
frames 0 12 30 47* 61 (* = kept by hand, strikethrough = dropped)
```
The frame strip already exists in miniature — `params/trace-keys` draws the trace
keys as seek buttons (`params.cljs:386`). Lift it into a shared component and
give it three affordances: click to seek, a modifier to pin, a modifier to drop.
A pinned frame and a proposed frame must be visually distinct, because "will this
survive me moving the slider?" is the question the strip exists to answer.
Render it in two sections:
- **`tracing · <face>`** (`params.cljs:426`), beside the existing origin row,
with the tolerance slider and Suggest.
- **`performance · <group>`** — new, on an instance's inspector, one per pose
group the placed symbol has. No tolerance: the strip shows the preserved frames
and the grid slots that snapped to them.
## Order to build it
1. **`domain/select` with the greedy walk, and its tests.** **Done**, in commit
`02069e8`. Pure, no store, no clip. Note that the extrema part of it is not
wanted by anything below — see *What is revertible*.
2. **The plate signal reader.** `trace/head-signal`: make `trace/measured-local`
public, map it over the frames, four corners through `node/apply-pt!`. Test
that it returns a vector of the right length, that a motionless take proposes
`[0]`, and that a stretch where the face was not found becomes `nil` rather
than a pose, a zero or a gap in the vector. Add the held-reconstruction
invariant from *The upgrade path* here, since this is the first place a real
signal exists to assert it over.
3. **Storage for the plate selection.** `:policy`/`:keep`/`:drop` beside
`:trace :frames`. Extend `leaf/leaves` and the key whitelists in the same
commit — a field without a leaf saves silently and comes back missing, which
is the one bug persistence must not be able to have. Round-trip test.
4. **Re-suggest preserves hand decisions.** Propose at one tolerance, pin a frame,
drop a frame, propose at another, assert both survive. Settle here whether the
hand's `:keep` is also fed to `propose` as `:protect`: under the layering as
written it is not, so a pinned frame does not re-anchor the walk even though it
is on screen, which contradicts the anchor rule above. It is the one live
caller for `propose`'s `:protect` frames.
5. **The preserve-snap.** The mark set from the stored `[:vis]` cuts, built in a
prepare step; the backward-only snap in the grid pick; plate frames folded into
the same mark set. Independent of steps 2–4 and cheaper than any of them, so it
can be pulled forward if the performance side is wanted sooner — the only thing
it needs from the plate side is the fold, which can land last.
6. **The shared UI component**, then its two mountings.
## What is revertible, and where it is
Commit `02069e8` contains one part that **nothing below asks for**: extrema
detection. Specifically `segments`, `turns`, the `:extrema` branch of `propose`,
the multi-component refusal that exists only to guard it, and three tests —
`a-one-frame-closure-survives-only-because-it-is-protected`,
`extrema-are-turns-worth-more-than-the-tolerance` and
`extrema-are-refused-on-a-multi-component-signal`.
It has no caller because plate selections take no extrema by design — displacement
is the whole story for a head — and the performance side reads a stored cut
instead of finding extrema in a signal. It is correct, tested and speculative.
Revert it if the shape above holds. Keep it if either of these turns out to be
wanted: a group whose extreme is not a closure and therefore has no `[:vis]` cut
to read (a mouth at its widest, a head at the top of a nod), or a take whose mouth
never shuts far enough to cross `aperture-cut`, where a peak-relative threshold
marks nothing and an extremum would still find the most closed frame. Neither is
asked for today. The DP in *The upgrade path* gets extrema for free regardless, so
reverting costs nothing that cannot be had again more cheaply.
## Traps
- **Do not let Suggest write `:keep`.** The proposal and the hand are different
layers; collapsing them is the bug this whole shape exists to avoid, and it
will look like it works right up until somebody moves the tolerance slider.
- **Do not snap a grid slot forward.** Every hold and every pick is "at or
before". Showing a closure before the mouth shut is a lead, which is a different
control for a different reason.
- **Do not quantise a plate selection onto the output grid.** A kept frame is a
native frame and lands where it lands. [time.md](time.md) is explicit that an
event between output frames appears on the next one; forcing kept frames onto
the grid would re-create the problem the selection exists to solve. The snap is
the opposite operation and is on the other side of the fence: it moves the grid's
pick, never the kept frame.
- **Do not read the aperture pair off a subsampled ring.** Positions 5 and 15 of
`LIPS-INNER` survive only at `verts` divisible by four.
- **Do not thin the plate selection with the mouth's.** Two scopes, two owners,
and the nesting runs one way only: plate frames preserve performance frames,
never the reverse.
- **Do not give the performance side a tolerance.** The grid has already chosen
the sparseness. A second knob there would be a control with nothing to control.
- **A skipped frame, a hidden feature and an absent measurement remain three
different facts.** A selection says nothing about visibility and nothing about
whether a face was found. Note that the preserve marks are *derived from* a
visibility cut, which makes this easy to blur: the cut says the interior is
hidden, the mark says the frame is worth landing on, and one is not the other.
## Not in scope
- Automatic *grouping* of related parts beyond the existing `:pose-group`.
Related parts must share one selection — a mouth outline, its interior, the
teeth and the generated visibility reading different frames is the bug that
grouping prevents — and `:pose-group` is where the grouping already lives.
- A per-instance request for a different tolerance than the symbol's. The
resolver threads no such option today and should not grow one until something
needs it.
- Variable frame rate. [time.md](time.md) assumes constant fps and so does this;
a VFR source needs presentation timestamps before any of this means anything.
Reference in a new issue View git blame Copy permalink