Preserve source cadence and sample picture fps after analysis

This commit is contained in:
Olive Vaughn 2026-09-27 19:31:39 -04:00
parent 32683efccf
commit 06cf02db83
20 changed files with 303 additions and 87 deletions

View file

@ -333,11 +333,15 @@ Every node may map the frame it is evaluated at:
```clojure
:time {:mode :inherit} ; the default, and almost always right
:time {:mode :map :expose 2 :offset -1 :rate 1.0 :loop? false}
:time {:mode :map :source-fps 30 :sample-fps 12} ; root: lower picture cadence
```
Three features that look unrelated are this one mechanism:
- **exposure** is `⌊f/n⌋·n`,
- **picture fps** quantises source time to a chosen picture grid, then reads the
latest source pose at or before that time; source analysis and audio keep their
original cadence,
- **mouth lead** is `f + k`,
- **a symbol instance's timing** is `(f - at)·rate + in`, with optional looping.
@ -463,8 +467,11 @@ world transform of the node it rides:
Composed with image-pixels-to-local — **both axes divided by `imgH`**, never by
their own dimension — the photo is registered with the shapes by construction,
and an unregistered underlay is merely decorative. Which frame it shows, the
current one or the held plate frame, is a UI choice and not a stored one.
and an unregistered underlay is merely decorative. The tracing editor chooses
which source frame to show under a cel. That reference choice is independent of
the finished picture fps and does not change the dense analysis track. A cel can
therefore use any useful source frame as its drawing reference, even when that
frame is not one of the displayed picture poses.
A photo that has to sit *between* two drawn layers is the case that would make it
a `:bitmap` node with an op of its own. Nothing wants that yet: a reference is
@ -527,6 +534,7 @@ Proof that it covers what exists, not just what is wanted:
| painted background cel | node per layer, `[:geom :pts]` **framed**, `[:style :color]` framed |
| `mouth lead` | `:time {:offset k}` on performance nodes only |
| `exposure` | `:time {:expose n}` on the clip root, inherited |
| picture fps | `:time {:source-fps s :sample-fps p}` on the clip root, applied after analysis |
| hand correction | an `:over` layer, `:offset` or `:replace` |
The brow row is the one worth looking at twice. `docs/design.md` argues at length

View file

@ -49,6 +49,11 @@ overrides and kept-frame sets are all in clip-frame space, so a clip slides on
the timeline without a single stored number changing. Exposure and lead are
transforms *within* clip space:
Detection retains every source frame. A chosen picture fps samples the frozen
roto in clip time; it changes neither source-frame count nor the audio clock.
The set of source frames an artist uses as cel tracing references is another
selection, independent of the picture fps.
```clojure
(defn pose-frame [clip cf]
(-> cf (expose (:exposure clip)) (shift (:lead clip) (count-frames clip))))
@ -73,7 +78,7 @@ project
│ ├── scene node tree
│ ├── channels node+property -> keyframe stream, in cf
│ ├── overrides (node, property, cf) -> value, applied last
│ └── cels painted vector layers, per exposure slot
│ └── cels painted vector layers, keyed by chosen clip frames
└── sequence[] clip placements: {clip-id, at, in, out}
```
@ -81,6 +86,10 @@ project
it a name is most of the work: `state` in `app.js` is a clip with its analysis
inlined and its palette global.
Cel keys select where drawings begin and how long they hold. The source frames
shown beneath a cel while tracing are chosen independently, and picture fps
only controls which analyzed pose the finished roto displays at a given time.
### Two things called "track"
`docs/design.md` says "dense track" for the landmark stream. A timeline also

View file

@ -4,8 +4,10 @@ Self-contained. You should not need any prior conversation to execute this.
**Implementation status (2026-09-27):** steps 0–6 are in the CLJS frontend.
Step 6 reads extracted footage from the manifest, detects landmarks with local
MediaPipe assets, and runs the same freeze path as the synthetic take. Step 7 is
next: eyes, brows and the pixel-derived mouth interior.
MediaPipe assets at full source cadence, and runs the same freeze path as the
synthetic take. The scene time map can sample the frozen roto at a lower picture
fps without changing source analysis, duration or audio. Step 7 is next: eyes,
brows and the pixel-derived mouth interior.
## What arthur is
@ -302,6 +304,10 @@ MediaPipe interop behind one namespace; real frames, real audio, real fps from
the manifest. **Vendor the wasm** rather than fetching from jsdelivr — it is
currently the only thing in the tool that silently requires a network.
Decode every source frame for analysis. A lower output picture fps is a time map
over frozen channels, not a reduced detection track. Selecting source frames to
trace into cels is independent again and remains outside this port's paint scope.
**Done:** real footage plays back as a rotoscoped mouth.
### 7 — the rest of measure