Port steps 0-1: scaffold, the oracle, and the pure bottom
Scaffolds frontend/ (shadow-cljs, reagent 1.2.0, re-frame 1.4.3) and ports everything below the data model, with the JS kept as a numeric oracle. domain/landmarks index tables, verbatim domain/ring subsample, offset, simplicity domain/geom similarity fit, procrustes, moving average domain/raster indexed scanline fill, stencil, disc, rect domain/palette the ramp, and the no-sampled-RGB rule 58 tests, 166 assertions. Parity with js/ on the identical 72-frame synthetic track: fit-similarity, procrustes-mean, fit-residual, moving-average, smooth-transforms, offset-ring and subsample-slots to 1e-9; the raster pixel-for-pixel over the whole buffer. Three deviations from the JS, each for a reason: - synth.cljs jitters from a SEEDED generator, not Math.random. Parity is only checkable if both sides can be handed the same track, and a failing assertion has to be reproducible. `:rand-fn` takes the generator over, so oracle.mjs stubs js/Math.random and js/ itself stays untouched. - raster/->rgba replaces toImageData. ImageData is a DOM type and domain/ may not touch the DOM; returning plain bytes also lets the no-intermediate-colours assertion run in node. ui/canvas wraps it later. - offset-ring lives in domain/ring, not domain/geom, per architecture.md: it is an operation on an ordered traversal, not on a transform. Step 1's "done" also names the swapped-iris vote, but pairIrises is in pipeline.js and belongs to step 7. The precondition is asserted instead -- `:swap-iris` really does move both blocks -- so the vote will have a track that disagrees with it when it arrives. One finding, recorded in full in the test that measures it: smooth-transforms buys nothing on the synthetic track. Against jitter-free ground truth, radius 1 helps by 17% on one noise realisation and hurts by 0.5% on another, so its benefit is within noise; from radius 2 up the cost is unambiguous, and by radius 5 the filter is below the true motion's own high-frequency energy, i.e. smoothing away performance. The test pins the shape of the knob rather than a preferred value. This may say more about the synth's jitter being unrealistically small (+/-0.001 normalised) than about the knob; step 6 settles it on real footage. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Olive Vaughn
parent
082d8561d2
commit
eb06be005c
25 changed files with 4932 additions and 0 deletions
351
docs/animation-model.md
Normal file
351
docs/animation-model.md
Normal file
|
|
@ -0,0 +1,351 @@
|
|||
# arthur — the animation model
|
||||
|
||||
The data that describes a moving picture: what the primitives are, how they
|
||||
nest, how they change over time, and how rotoscoped and hand-authored work end
|
||||
up being the same thing with one flag between them.
|
||||
|
||||
`docs/design.md` is the aesthetic argument. `docs/architecture.md` is where the
|
||||
code goes. This is the type that both of them are about.
|
||||
|
||||
## What this replaces
|
||||
|
||||
`docs/design.md` has a table of five kinds of part — plate, feature, interior,
|
||||
primitive, scalar — each with its own source, vocabulary and interpolation. That
|
||||
table is a good description of **where data comes from** and a bad description of
|
||||
**what data is**, and the current code follows it too literally: eyes, brows,
|
||||
teeth and mouth each get their own build function, their own key shape and their
|
||||
own path through `take.js`.
|
||||
|
||||
They are all one thing. A part is a **node** with **channels**, and the five
|
||||
kinds collapse into differences of which channels exist and who filled them in.
|
||||
|
||||
## Prior art, and what each one gets right
|
||||
|
||||
| System | The idea worth taking |
|
||||
| --- | --- |
|
||||
| **Flash / SWF** | A **library of symbols** and a timeline of **instances** at depths. "Framed" content that simply exists on a frame, versus tweened content. `DefineMorphShape` requires matching vertex counts — the fixed-topology rule, arrived at from the other direction. |
|
||||
| **Blender** | Animation is **addressed by path into the data** (`location[0]`), not stored as fields on the object. An Action is a bag of F-Curves. That decoupling is what makes the dope sheet, the graph editor and the NLA three views of one dataset. Also: parenting captures a `parent_inverse` so the child does not jump. |
|
||||
| **After Effects** | Every leaf property is animatable, uniformly. Property groups form a tree. Pre-comps nest arbitrarily and a pre-comp is just a layer. |
|
||||
| **Lottie** | The uniform property shape: `{a: 0, k: <value>}` or `{a: 1, k: [<keys>]}`. One representation for static and animated, which is exactly "framed or keyframed". |
|
||||
| **Grease Pencil** | A 2D layer holds frames at frame numbers, and a frame **holds until the next one**. Hold is the default, not a special case. |
|
||||
|
||||
What none of them get right for this project: colour. All four store RGB on the
|
||||
shape. `docs/design.md` forbids that, so colour is a palette index here and it is
|
||||
a channel like any other.
|
||||
|
||||
## The one idea
|
||||
|
||||
**Analysis is a channel generator.** It does not produce a different kind of
|
||||
data; it produces keys, densely, on the same channels a hand would fill in
|
||||
sparsely. So:
|
||||
|
||||
```
|
||||
footage ──▶ analysis ──▶ FREEZE ──▶ channels on nodes ──▶ evaluate ──▶ raster
|
||||
▲
|
||||
hand authoring ──┘
|
||||
```
|
||||
|
||||
Freezing is not a conversion into a second format. There is one format, and
|
||||
freezing fills it in. That is what makes "the only difference is a special flag"
|
||||
literally true: the flag is provenance on a channel, and nothing in the renderer
|
||||
reads it.
|
||||
|
||||
## Node
|
||||
|
||||
A node is an instance in the scene. The tree is stored **flat, with parent
|
||||
pointers** — never as nested maps.
|
||||
|
||||
```clojure
|
||||
{:id :mouth
|
||||
:name "mouth"
|
||||
:kind :poly ; :poly :disc :rect :group :bitmap :symbol
|
||||
:parent :head ; nil at the root
|
||||
:z "a3" ; fractional index, ordered among all siblings
|
||||
:symbol nil ; or :sym/blink — see Symbols
|
||||
:stencil :mouth-in ; colour-key clip; structural, not a channel
|
||||
:span [0 240] ; in/out in the parent's frame space
|
||||
:pinv [1 0 0 1 0 0] ; parent-inverse, captured when parented
|
||||
:channels {...}}
|
||||
```
|
||||
|
||||
Flat with pointers, for four reasons that all point the same way: any node is
|
||||
addressable without a walk; reparenting is a one-field write rather than a
|
||||
subtree move; an edit to a leaf does not change the identity of its ancestors, so
|
||||
re-frame's structural sharing keeps ancestor subs from invalidating; and it is
|
||||
what lets every node be its own sync leaf. Flash, Blender and AE all store it
|
||||
this way.
|
||||
|
||||
`:span` is Lottie's `ip`/`op` and Flash's `PlaceObject`/`RemoveObject`: the range
|
||||
over which the node exists at all. Distinct from a `[:vis]` channel, which
|
||||
blinks an existing node on and off.
|
||||
|
||||
## Channel
|
||||
|
||||
Every animatable property is a channel, and channels are addressed **by path**:
|
||||
|
||||
```clojure
|
||||
:channels
|
||||
{[:xform :pos] {:animated? false :value [0.0 0.0]}
|
||||
[:xform :rot] {:animated? false :value 0.0}
|
||||
[:xform :scale] {:animated? false :value [1.0 1.0]}
|
||||
[:xform :skew] {:animated? false :value [0.0 0.0]}
|
||||
[:xform :anchor]{:animated? false :value [0.0 0.0]}
|
||||
[:geom :pts] {:animated? true :interp :hold :dense {...} :generated {...}}
|
||||
[:style :color] {:animated? false :value :skin-dark}
|
||||
[:vis] {:animated? true :interp :hold :keys {0 true, 37 false}}}
|
||||
```
|
||||
|
||||
A path is a **vector**, not a string — CLJS maps take vectors as keys natively,
|
||||
so Blender's `data_path` idea arrives with no parsing. The set of valid paths for
|
||||
a node follows from its `:kind`, and that is a spec, not a schema migration.
|
||||
|
||||
Three channel shapes, and the uniformity across them is the point:
|
||||
|
||||
```clojure
|
||||
;; FRAMED — one static thing. No animation, no vertex correspondence to worry
|
||||
;; about. A painted background cel is this.
|
||||
{:animated? false :value v}
|
||||
|
||||
;; KEYED — sparse, authored, in the document. Undoable and syncable.
|
||||
{:animated? true :interp :hold :keys {0 v, 4 v, 12 v}}
|
||||
|
||||
;; DENSE — generated, one value per frame, held in tier 2 as a typed array.
|
||||
{:animated? true :interp :hold
|
||||
:dense {:store "sha256:…" :offset 0 :stride 40 :frames 600}
|
||||
:generated {...}}
|
||||
```
|
||||
|
||||
`:interp` defaults to `:hold`, which `docs/design.md` requires of every cut part.
|
||||
A key may carry its own `:interp` to override the channel's, which is how Lottie
|
||||
and Blender both do per-key easing; nothing uses it yet and the door is cheap to
|
||||
leave open.
|
||||
|
||||
### Keys are a map by frame, not a list
|
||||
|
||||
Already argued in `docs/architecture.md` for merge reasons; here it also gives
|
||||
"the most recent key at or before `f`" as a `rsubseq` on a sorted map instead of
|
||||
a scan. **Store a plain map** in the document — transit and JSON both lose
|
||||
sortedness — and build the sorted index in the resolver.
|
||||
|
||||
### The flag lives on the channel, not the node
|
||||
|
||||
```clojure
|
||||
:generated {:by :roto/lips-outer
|
||||
:analysis "sha256:…" ; which analysis artifact
|
||||
:params {:verts 8 :contour-avg 1 :aperture-cut 0.004}}
|
||||
```
|
||||
|
||||
Present means the UI offers a parameter panel and a re-freeze button. Absent
|
||||
means the UI offers the keys directly. **The renderer never reads it.**
|
||||
|
||||
It belongs on the channel rather than the node because a node routinely wants
|
||||
both at once: a mouth whose `[:geom :pts]` is rotoscoped and whose `[:xform :pos]`
|
||||
is hand-animated to sit on a plate. Putting the flag on the node would forbid the
|
||||
most useful thing in the model.
|
||||
|
||||
### Channels are layered
|
||||
|
||||
A channel is a base plus optional override layers, and a layer declares how it
|
||||
combines:
|
||||
|
||||
```clojure
|
||||
{:animated? true :interp :hold
|
||||
:dense {...} :generated {...}
|
||||
:over [{:blend :offset :keys {88 [2 0], 96 [0 0]}}
|
||||
{:blend :replace :keys {104 [[3 7] [4 7] …]}}]}
|
||||
```
|
||||
|
||||
- **`:offset`** adds a delta to the base. "Nudge the mouth two pixels right for
|
||||
ten frames" survives a re-freeze at different parameters, because it was never
|
||||
a position — it was a correction.
|
||||
- **`:replace`** wins outright. For the frame where detection simply failed.
|
||||
|
||||
This is what `docs/design.md` means by an override layer, and it is why
|
||||
re-freezing is safe: the base is regenerated, the layers are untouched. It is
|
||||
Blender's NLA blending and AE's effect stack at one property.
|
||||
|
||||
## Transform: decomposed, never a matrix
|
||||
|
||||
```clojure
|
||||
{:pos [x y] :rot θ :scale [sx sy] :skew [kx ky] :anchor [ax ay]}
|
||||
```
|
||||
|
||||
Stored decomposed for two reasons. Each component has to be independently
|
||||
keyframable, which is the entire point of channels. And interpolating matrix
|
||||
entries is meaningless — a rotation tweened through its matrix shears on the way.
|
||||
|
||||
Composition, per node:
|
||||
|
||||
```
|
||||
local = T(pos) · T(anchor) · R(rot) · K(skew) · S(scale) · T(-anchor)
|
||||
world = world(parent) · pinv · local
|
||||
```
|
||||
|
||||
`:anchor` is Flash's registration point and Blender's origin: rotation and scale
|
||||
happen about it, and getting it wrong is why hand-placed parts swing rather than
|
||||
turn.
|
||||
|
||||
`:pinv` is Blender's `parent_inverse`, captured at the moment of parenting so the
|
||||
child does not jump when it acquires a parent. Small, and its absence is the kind
|
||||
of thing that makes a parenting feature feel broken.
|
||||
|
||||
**The similarity fit already produces a decomposition.** `fitSimilarity` returns
|
||||
`{s θ tx ty}`, which drops straight into `[:xform :scale]`, `[:xform :rot]` and
|
||||
`[:xform :pos]` with no conversion. The analysis output and the animation model
|
||||
meet without an adapter, which is a sign the decomposition is the right one.
|
||||
|
||||
## Time maps — exposure, lead and symbol timing are one thing
|
||||
|
||||
Every node may map the frame it is evaluated at:
|
||||
|
||||
```clojure
|
||||
:time {:mode :inherit} ; the default, and almost always right
|
||||
:time {:mode :map :expose 2 :offset -1 :rate 1.0 :loop? false}
|
||||
```
|
||||
|
||||
Three features that look unrelated are this one mechanism:
|
||||
|
||||
- **exposure** is `⌊f/n⌋·n`,
|
||||
- **mouth lead** is `f + k`,
|
||||
- **a symbol instance's timing** is `(f - at)·rate + in`, with optional looping.
|
||||
|
||||
Composed along the nesting chain, outermost first. Two rules follow, and they are
|
||||
different rules:
|
||||
|
||||
- **Exposure inherits strictly.** `docs/design.md` is emphatic that everything
|
||||
rides one grid, because a head cutting on odd frames against a mouth cutting on
|
||||
even ones reads as two performances. The model permits a per-node grid; the
|
||||
default must be `:inherit`, and setting it lower is a deliberate act the UI
|
||||
should make feel like one.
|
||||
- **Offset is per-node by design.** Mouth lead applies to performance nodes and
|
||||
*not* to the plate, which is the whole point of it — so the offset genuinely
|
||||
belongs at the node, not the clip.
|
||||
|
||||
## Symbols
|
||||
|
||||
The Flash idea, kept:
|
||||
|
||||
```clojure
|
||||
:library
|
||||
{:sym/head {:kind :poly :channels {...}}
|
||||
:sym/blink {:kind :timeline :frames 3 :nodes [...]}}
|
||||
```
|
||||
|
||||
A node with `:symbol :sym/blink` is an **instance**. Its own channels compose
|
||||
*over* the symbol's, so one definition can be placed many times and tinted,
|
||||
offset or retimed at each placement. A `:timeline` symbol has its own frame space,
|
||||
which the instance's `:time` maps into — that is Flash's MovieClip, Lottie's
|
||||
precomp and AE's pre-comp, and it is how a three-frame blink gets reused at
|
||||
frames 40, 88 and 200 without copying it.
|
||||
|
||||
This is also where `docs/design.md`'s "closed vocabulary is right for the head"
|
||||
lands: a plate library is a set of `:sym/head-*` definitions, and the strip
|
||||
chooses which instance is placed on which frame. The take format's `plate=` field
|
||||
becomes an instance reference.
|
||||
|
||||
## Evaluating a frame
|
||||
|
||||
```clojure
|
||||
(defn eval-frame
|
||||
"Scene at clip frame f -> draw ops in z order. Pure."
|
||||
[scene f] ...)
|
||||
```
|
||||
|
||||
1. Walk nodes in **topological order** by parent depth (cached; recompute only
|
||||
when parentage changes).
|
||||
2. Skip nodes outside `:span`.
|
||||
3. Apply the node's time map to get its own local frame `fn`.
|
||||
4. **Sample** each channel at `fn`: a map lookup for framed, a sorted-index
|
||||
lookup for keyed, an array read for dense. Then apply `:over` layers.
|
||||
5. Compose `world` from the parent's.
|
||||
6. Transform geometry into raster space, writing into a **preallocated buffer**
|
||||
owned by the node.
|
||||
7. Emit `{:kind :poly :pts buf :n 20 :color idx :stencil id}`.
|
||||
8. Sort by resolved `z`.
|
||||
|
||||
The op list is the boundary with stage 7 in `docs/architecture.md`: the
|
||||
rasteriser takes ops and knows nothing about nodes, channels or time.
|
||||
|
||||
### Making it fast in CLJS
|
||||
|
||||
Three things, and only these three matter:
|
||||
|
||||
- **Decomposed and persistent for storage; flat and mutable for evaluation.**
|
||||
Composed transforms are 6-element `Float64Array`s, not maps. Every renderer
|
||||
does this; the storage form and the evaluation form are allowed to differ.
|
||||
- **A cursor per channel.** Playback is sequential, so "most recent key at or
|
||||
before `f`" is an advance of a saved index, O(1) amortised. Binary search only
|
||||
on a seek. This is the difference between a `rsubseq` allocation per channel per
|
||||
frame and none.
|
||||
- **Preallocated point buffers per node.** Fixed topology means the size is known
|
||||
at freeze time, so a frame allocates nothing. At 30fps, per-frame allocation is
|
||||
the only thing that will make this stutter.
|
||||
|
||||
### What is in app-db, and what is not
|
||||
|
||||
| In app-db (tier 1) | In tier 2, behind a handle |
|
||||
| --- | --- |
|
||||
| nodes, parentage, z, spans, stencils | dense channel blocks |
|
||||
| channel definitions, `:interp`, `:generated` | analysis artifacts |
|
||||
| **framed** values, **keyed** keys, `:over` layers | preallocated eval buffers |
|
||||
| library / symbol definitions | composed transform scratch |
|
||||
|
||||
The rule: **anything a human placed is in the document; anything a generator
|
||||
produced is a handle.** Which is the same line `docs/architecture.md` draws for
|
||||
sync and baking, arrived at again from the renderer's side.
|
||||
|
||||
## The current parts, in this model
|
||||
|
||||
Proof that it covers what exists, not just what is wanted:
|
||||
|
||||
| Now | Becomes |
|
||||
| --- | --- |
|
||||
| `mouth` outer ring, every frame | node `:mouth`, `[:geom :pts]` dense, `:generated {:by :roto/lips-outer}` |
|
||||
| `mouth_in`, hidden below aperture | node `:mouth-in`, parent `:mouth`, `[:geom :pts]` dense + `[:vis]` dense |
|
||||
| `teeth` from image content | node `:teeth`, stencil `:mouth-in`, `[:geom :pts]` dense, `:generated {:by :interior/teeth}` |
|
||||
| lid rings | nodes `:lid-r/-l`, `[:geom :pts]` dense |
|
||||
| lash line (`offsetRing`) | not data — a stage-6 parameter on the node, `{:grow px}` |
|
||||
| iris disc | node `:iris-r`, `:kind :disc`, parent `:lid-r`, stencil `:sclera-r`, `[:xform :pos]` dense (quantised at freeze), radius framed |
|
||||
| square pupil | node `:pupil-r`, `:kind :rect`, parent `:iris-r`, stencil `:iris-r` |
|
||||
| brow ring + quantised raise | node `:brow-r`, `[:geom :pts]` dense (the traced ring with height removed), `[:xform :pos]` dense (the quantised raise). **The decomposition design.md insists on is two channels.** |
|
||||
| head plate, kept frames | node `:head`, `:symbol` per instance, keys on `[:symbol]` at kept frames |
|
||||
| painted background cel | node per layer, `[:geom :pts]` **framed**, `[:style :color]` framed |
|
||||
| `mouth lead` | `:time {:offset k}` on performance nodes only |
|
||||
| `exposure` | `:time {:expose n}` on the clip root, inherited |
|
||||
| hand correction | an `:over` layer, `:offset` or `:replace` |
|
||||
|
||||
The brow row is the one worth looking at twice. `docs/design.md` argues at length
|
||||
that the traced ring already contains the height, so the quantised raise must be
|
||||
measured *out* and put *back* or the brow moves twice. In this model that is not
|
||||
an argument to remember — it is two channels on one node, and getting it wrong
|
||||
would mean writing the height into both.
|
||||
|
||||
## Format on disk and on the wire
|
||||
|
||||
Tier 1 is EDN/transit: the node tree, channel definitions, framed values, keys,
|
||||
layers, library. Kilobytes, human-readable, diffable, and leaf-addressable for
|
||||
sync.
|
||||
|
||||
Dense blocks are separate content-addressed binaries — `Int16Array` for raster
|
||||
geometry, `Float32Array` for transforms — with a small header naming the channel
|
||||
path, frame count and stride.
|
||||
|
||||
**Not Lottie internally**, despite the property shape being borrowed from it.
|
||||
Lottie has no palette-indexed colour, its shapes are bezier with in/out tangents
|
||||
where these are integer polygons, and its interpolation defaults are the opposite
|
||||
of what is wanted. It is a good **export** target later, next to the `.take`
|
||||
writer, and a bad internal format.
|
||||
|
||||
## Deferred
|
||||
|
||||
- **Per-key easing.** The structure allows it; nothing should use it until a
|
||||
parented transform on a painted cel asks for it.
|
||||
- **More than two channel layers.** The `:over` vector is already a list; a real
|
||||
blend stack with weights is the NLA, and it is not needed to fix a bad frame.
|
||||
- **Skew beyond the field.** `[:xform :skew]` is in the transform and in the
|
||||
composition order from the start, because adding a component to a decomposition
|
||||
later means migrating every stored transform.
|
||||
- **Instance channel overrides on symbols.** Compose-over is specified; only
|
||||
colour and transform need it at first.
|
||||
- **Constraints and drivers.** Blender's other half. A gaze that aims at a null
|
||||
object is the obvious first one, and it is a long way off.
|
||||
869
docs/architecture.md
Normal file
869
docs/architecture.md
Normal file
|
|
@ -0,0 +1,869 @@
|
|||
# arthur — structure
|
||||
|
||||
How the suite is laid out once it is a suite: ClojureScript and re-frame, a
|
||||
clip as the unit of work, keying as its own step, and painting as a peer of
|
||||
rotoscoping rather than a sketch bolted to the side.
|
||||
|
||||
`docs/design.md` says what the thing is and why. This says where the code goes.
|
||||
`docs/animation-model.md` specifies the data both of them are about — nodes,
|
||||
channels, symbols and time maps — and supersedes this document wherever the two
|
||||
describe the same type.
|
||||
Nothing here revises an aesthetic decision; several things here split a
|
||||
decision that is currently made in two places at once.
|
||||
|
||||
## What is actually wrong with the current shape
|
||||
|
||||
The pure modules are fine. `pipeline.js`, `mathutil.js`, `landmarks.js`,
|
||||
`interior.js`, `raster.js` and `take.js` are functions over data with the
|
||||
reasoning written down next to them, and they port nearly verbatim.
|
||||
|
||||
`app.js` is the whole problem, and not because it is long. It is long because
|
||||
six unrelated jobs are braided together in it:
|
||||
|
||||
- reading knob values out of the DOM (`opts()`),
|
||||
- deriving everything from them (`rebuild`, `buildEyes`, `buildBrows`,
|
||||
`resolveTeeth`),
|
||||
- resolving what is on screen at a frame (`plateIndex`, `perfIndex`, `leadIndex`),
|
||||
- rasterising (`renderFrame`, `compositeRender`),
|
||||
- driving the clock (`tick`),
|
||||
- persisting (`saveCels`, `loadCels`, `exportTake`).
|
||||
|
||||
Every one of those is a different layer, and `rebuild` recomputes all of them
|
||||
whenever any knob moves. That is affordable at one clip and 72 frames and is
|
||||
not affordable at a project. The re-frame port is worth doing chiefly because
|
||||
the subscription graph *is* the staged dataflow `docs/design.md` already
|
||||
describes — "three artifacts, not two" is a dependency graph written in prose.
|
||||
|
||||
## Three frame spaces, named
|
||||
|
||||
Most of the future bugs live here, so name them before anything else.
|
||||
|
||||
| Space | Symbol | Meaning |
|
||||
| --- | --- | --- |
|
||||
| **source frame** | `sf` | Index into the footage and the dense track. What detection produced. |
|
||||
| **clip frame** | `cf` | 0-based within a clip. `cf = sf - (:in clip)`. |
|
||||
| **sequence frame** | `qf` | Position on the timeline. `qf = (:at clip) + cf`. |
|
||||
|
||||
**Nothing authored is ever stored in sequence-frame space.** Keys, cels,
|
||||
overrides and kept-frame sets are all in clip-frame space, so a clip slides on
|
||||
the timeline without a single stored number changing. Exposure and lead are
|
||||
transforms *within* clip space:
|
||||
|
||||
```clojure
|
||||
(defn pose-frame [clip cf]
|
||||
(-> cf (expose (:exposure clip)) (shift (:lead clip) (count-frames clip))))
|
||||
```
|
||||
|
||||
which is the composition `app.js` currently performs in `leadIndex` and
|
||||
`perfIndex` with the order implicit. Expose first, then lead: exposure floors
|
||||
onto a grid and lead deliberately reads the future, so leading first and then
|
||||
flooring would quietly discard the lead on most frames.
|
||||
|
||||
## The entity model
|
||||
|
||||
```
|
||||
project
|
||||
├── palette authored ramp; indices, never RGB
|
||||
├── character plate library, per-plate slots
|
||||
├── footage[] frames + audio + manifest (fps, w, h)
|
||||
├── analysis[] dense track, cached, keyed by footage+model+version
|
||||
├── clip[] the unit of work
|
||||
│ ├── source footage id, analysis id, in/out in sf
|
||||
│ ├── exposure, lead timing, in cf
|
||||
│ ├── scene node tree
|
||||
│ ├── channels node+property -> keyframe stream, in cf
|
||||
│ ├── overrides (node, property, cf) -> value, applied last
|
||||
│ └── cels painted vector layers, per exposure slot
|
||||
└── sequence[] clip placements: {clip-id, at, in, out}
|
||||
```
|
||||
|
||||
`clip` is the entity the current app has exactly one of and never names. Giving
|
||||
it a name is most of the work: `state` in `app.js` is a clip with its analysis
|
||||
inlined and its palette global.
|
||||
|
||||
### Two things called "track"
|
||||
|
||||
`docs/design.md` says "dense track" for the landmark stream. A timeline also
|
||||
wants tracks. Pick now: the landmark stream is **analysis**, a keyframe stream
|
||||
is a **channel**, and the word *track* is reserved for a row on the timeline.
|
||||
Renaming this later costs a day.
|
||||
|
||||
## The node, decomposed
|
||||
|
||||
`take.js` today gives a part a `parent` and a `clip`, and `parent` is doing
|
||||
nothing except documenting intent — `mouth_in` is already in the same
|
||||
head-local raster space as `mouth`, so composing its transform would be
|
||||
composing identity. The moment a painted cel is attached to a head plate that
|
||||
moves, transform composition becomes real, and the three ideas currently sharing
|
||||
two fields have to come apart:
|
||||
|
||||
| Field | What it does | Wrong to conflate because |
|
||||
| --- | --- | --- |
|
||||
| `:parent` | Transform composition. Child geometry is in parent's local space. | A node can be drawn over its parent without inheriting its motion. |
|
||||
| `:stencil` | Colour-key clip, the `clip=` of the take format. | The iris is stencilled by the sclera and parented to the lid ring; those are different nodes. |
|
||||
| `:z` | Draw order. | Order is authored per scene, not implied by the tree. |
|
||||
|
||||
A node is then:
|
||||
|
||||
```clojure
|
||||
{:id :eye-r/iris
|
||||
:source {:kind :primitive ...} ; the five kinds from design.md
|
||||
:parent :eye-r/lid
|
||||
:stencil :eye-r/sclera
|
||||
:z 42
|
||||
:color :iris ; palette key, never a hex
|
||||
:interp :hold}
|
||||
```
|
||||
|
||||
`:source` carries the taxonomy `docs/design.md` already has — `:plate`,
|
||||
`:feature`, `:interior`, `:primitive`, `:scalar` — plus `:cel` for painted
|
||||
vectors and `:group` for a pure transform node. That is the decomposition that
|
||||
makes painting a peer rather than an annex: `paint.js` is not a special case,
|
||||
it is a source kind whose channels happen to be authored by hand instead of
|
||||
measured. `docs/design.md` already promises "pluggable sources"; this is the
|
||||
data shape that keeps the promise.
|
||||
|
||||
Each node also gets an optional local transform channel — translate, rotate,
|
||||
scale, squash. That is the thing the current tool cannot express and that the
|
||||
`slot_mouth` / `scale` / `rot` / `squash` fields in `take.js` are reaching for.
|
||||
|
||||
## The flow, in seven stages
|
||||
|
||||
The user-facing flow is `frames -> analysis -> keying -> geometry -> palette`.
|
||||
Analysis is three stages, not one, and the split is the most load-bearing
|
||||
decision in this document, because **exactly one boundary is a cache
|
||||
boundary.**
|
||||
|
||||
| # | Stage | In | Out | Cost |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| 1 | **ingest** | video | footage: frames, audio, manifest | minutes, in-app |
|
||||
| 2 | **detect** | footage | raw landmarks per frame | minutes, **cached** |
|
||||
| 3 | **measure** | landmarks | anchor fit, residual, head-local rings, signals, interior pixels | seconds |
|
||||
| 4 | **condition** | measurements | smoothed transforms and contours | milliseconds |
|
||||
| 5 | **key** | conditioned signals + policy | channels: sparse keys, quantised holds, kept frames | milliseconds |
|
||||
| 6 | **resolve** | channels + scene + exposure + lead + overrides | geometry per output frame | per frame |
|
||||
| 7 | **palette** | geometry | indexed raster | per frame |
|
||||
|
||||
Stage 2 is the only artifact worth persisting and the only one worth a progress
|
||||
bar. Stage 3 reads pixels — teeth extraction is here, and that is why it is
|
||||
here rather than in keying: **after stage 3, nothing downstream may look at a
|
||||
source pixel.** Stage 4 is separated from 3 only because `contour avg` and
|
||||
`anchor avg` are knobs and the rest of stage 3 is not; splitting them means
|
||||
dragging that slider does not re-run the interior extraction.
|
||||
|
||||
### Where the current code lands
|
||||
|
||||
| Now | Stage |
|
||||
| --- | --- |
|
||||
| `stabilize` | 3 measure |
|
||||
| `eyeSignals`, `browSignals`, `pairIrises`, `pairBrows` | 3 measure |
|
||||
| `extractTeeth` | 3 measure — it reads pixels |
|
||||
| `smoothTransforms`, `smoothContours` | 4 condition |
|
||||
| `selectKeys`, `suggestPlateFrames` | 5 key |
|
||||
| `quantizeSnap`, `resolveBlink` | 5 key — a held value *is* a key |
|
||||
| `exposeIndex`, `shiftIndex`, `activeKey`, `heldFrame` | 6 resolve |
|
||||
| `toRasterRing`, iris placement at ring slots, `offsetRing` | 6 resolve |
|
||||
| `IndexedRaster`, `drawCel` | 7 palette |
|
||||
| `drawRegistered`, `posterizeInto` | reference only, off to the side |
|
||||
| `writeTake` | serialization, off the end |
|
||||
|
||||
Three things in `app.js` currently straddle a boundary, and each straddle is a
|
||||
bug waiting for a bigger project:
|
||||
|
||||
- **`resolveTeeth`** extracts from pixels *and* applies contrast/dwell/blob
|
||||
knobs. Extraction is stage 3 and cached with the track; resolution is stage 5.
|
||||
Today, nudging `teeth dwell` re-runs Otsu over every frame.
|
||||
- **`buildEyes`** measures, quantises, *and* places the iris on the subsampled
|
||||
lid ring. Those are stages 3, 5 and 6. The placement argument in
|
||||
`docs/design.md` — that the iris is read off the already-smoothed ring — is a
|
||||
stage-6 fact and stays true; it just needs to be *in* stage 6.
|
||||
- **`buildBrows`** likewise: measure the height out of the ring (3), quantise
|
||||
it (5), put it back (6). The doc's warning about moving the brow twice is
|
||||
precisely a warning about these being one function.
|
||||
|
||||
### What each knob invalidates
|
||||
|
||||
This table is the argument for the split, and it should be derivable from the
|
||||
sub graph rather than maintained by hand.
|
||||
|
||||
| Knob | Invalidates from |
|
||||
| --- | --- |
|
||||
| model / footage | 2 detect |
|
||||
| `teeth contrast`, `cavity erode`, `blob grow`, `tongue reject`, `prefer upper` | 3 measure (mouth crop only) |
|
||||
| `contour avg`, `anchor avg` | 4 condition |
|
||||
| `teeth dwell`, `blink cut/hold/dwell`, `gaze step/dwell`, `brow step/dwell`, `suggest tolerance` | 5 key |
|
||||
| `vertices`, `eye vertices`, `brow vertices`, `teeth vertices` | 5 key |
|
||||
| `exposure`, `mouth lead`, kept-frame edits, overrides | 6 resolve |
|
||||
| `lash line`, `iris size`, `pupil`, `brow weight` | 6 resolve |
|
||||
| palette edits, colour assignment | 7 palette |
|
||||
|
||||
## Namespaces
|
||||
|
||||
```
|
||||
src/arthur/
|
||||
domain/ pure data, specs, ops. No re-frame, no DOM.
|
||||
palette.cljs ramp; index lookup; the no-RGB rule as a spec
|
||||
landmarks.cljs index tables (port verbatim)
|
||||
ring.cljs ordered traversal: subsample, offset, simplicity
|
||||
geom.cljs similarity fit, procrustes, moving average
|
||||
node.cljs scene node: source, parent, stencil, z
|
||||
scene.cljs node tree: topo order, transform composition
|
||||
channel.cljs keyframe stream: active-key-at, hold semantics
|
||||
clip.cljs clip entity; the frame-space conversions
|
||||
cel.cljs painted vector layers
|
||||
timeline.cljs sequence: clip placement, qf <-> cf
|
||||
take.cljs take-file writer (port take.js)
|
||||
flow/
|
||||
ingest.cljs
|
||||
detect.cljs the MediaPipe boundary, and the only one
|
||||
measure/
|
||||
anchor.cljs stabilise: procrustes, similarity, residual
|
||||
mouth.cljs
|
||||
eyes.cljs openness, gaze, iris pairing vote
|
||||
brows.cljs raise/tilt, ring and end pairing votes
|
||||
interior.cljs otsu, morphology, components, radial contour
|
||||
condition.cljs
|
||||
key.cljs velocity minima, dwell, quantise, blink, decimate
|
||||
resolve.cljs scene eval at a frame; exposure, lead, overrides
|
||||
raster.cljs indexed scanline fill, stencil, disc, rect
|
||||
reference.cljs registered underlay, posterise
|
||||
db.cljs app-db schema + spec
|
||||
store.cljs handles for things too big for app-db
|
||||
clock.cljs audio clock, outside app-db
|
||||
events/ one ns per domain area
|
||||
subs/ one ns per stage
|
||||
ui/
|
||||
shell.cljs
|
||||
mode/ one ns per tool mode
|
||||
panel/ strip, worksheet, readouts, palette, layers
|
||||
canvas.cljs the one imperative sink
|
||||
fx/ mediapipe, files, audio, persistence, export
|
||||
```
|
||||
|
||||
Two rules about this tree. `domain/` may not require `flow/`, and neither may
|
||||
require `re-frame`. And `flow/` namespaces take explicit arguments — no
|
||||
namespace below `subs/` ever calls `subscribe`.
|
||||
|
||||
## app-db, and what is not in it
|
||||
|
||||
**app-db holds authored data and ids. Nothing derived, and nothing large.**
|
||||
|
||||
That sounds like ordinary hygiene and it is not: it is the precondition for two
|
||||
features that are otherwise unbuildable.
|
||||
|
||||
- **Spec validation on every event.** Worth having, affordable only over
|
||||
authored data.
|
||||
- **Cheap writes.** Every edit `assoc`es into app-db, and every mounted layer-2
|
||||
sub compares the result. Both are structural-sharing operations and stay cheap
|
||||
only while the map is small.
|
||||
|
||||
Undo is *not* on that list, and an earlier draft said it was. See Collaboration:
|
||||
snapshot-based undo is right single-player and wrong once two people edit.
|
||||
|
||||
So the dense track and the decoded frames live in `arthur.store`, a `defonce`
|
||||
map of id to JS object, and app-db holds `{:analysis/id "sha-..."}`. Landmarks
|
||||
stay as they arrive — arrays of `{x, y, z}`, or better, a flat `Float32Array` —
|
||||
and are *not* converted to Clojure maps. 478 points × 600 frames is 286,800
|
||||
maps, allocated for nothing: no sub ever needs to diff them, and every stage
|
||||
that touches them walks all of them anyway.
|
||||
|
||||
### The playhead, and where per-frame work goes
|
||||
|
||||
A reaction propagates downstream only when its own **output value** changes, not
|
||||
when one of its inputs notifies. So with layer-2 extractors and layer-3
|
||||
computations, a playhead tick costs one cheap extractor run per mounted layer-2
|
||||
sub, each of which returns the same value for every subtree the tick did not
|
||||
touch and therefore notifies nobody. `::measurements`, `::channels` and
|
||||
`::resolver` do not re-run. The `:<-` graph is what buys that, and it buys it
|
||||
whether or not the playhead is in app-db.
|
||||
|
||||
So **the playhead lives in app-db**, like everything else. An earlier draft of
|
||||
this document put it in a standalone ratom to avoid an invalidation storm that
|
||||
does not happen. Two independent reasons it belongs in db:
|
||||
|
||||
- `[:playback/seek ...]` in the event log is how scrubbing becomes inspectable
|
||||
in re-frame-10x.
|
||||
- **A collaborator's playhead is a feature.** Putting it outside app-db puts it
|
||||
outside the machinery that shares it. See Collaboration.
|
||||
|
||||
What is true is narrower, and is about sub authoring and interceptors rather
|
||||
than about the playhead:
|
||||
|
||||
- **Expensive work must not live in a layer-2 sub.** A sub that derefs app-db
|
||||
directly re-runs on every db change whatever changed. That is the actual
|
||||
content of the folklore about re-frame and canvas.
|
||||
- **Do not put a per-frame event behind a global interceptor that walks the
|
||||
whole db** — a spec-validating `after`, or `std-interceptors/debug` and its
|
||||
`clojure.data/diff`. At 30fps that is thirty full-db traversals a second. The
|
||||
transport event carries its own interceptor chain and is excluded from the
|
||||
global ones.
|
||||
- **The render sink is not a Reagent component.** Stage 7 writes bytes into a
|
||||
canvas from an rAF loop; it is not a view that re-renders.
|
||||
|
||||
Stage 6 is still arranged so the per-frame path is a lookup, not a computation:
|
||||
|
||||
```clojure
|
||||
;; recomputes when channels, scene, exposure, lead or overrides change
|
||||
(rf/reg-sub ::resolver ...) ;; => (fn [cf] geometry), or an index into a bake
|
||||
|
||||
;; the rAF loop: reads, blits, dispatches nothing
|
||||
(defn tick []
|
||||
(let [cf (clock/clip-frame)]
|
||||
(raster/paint! (@resolver cf))
|
||||
(canvas/blit!)))
|
||||
```
|
||||
|
||||
The sub produces a *resolver*; the loop applies it. A knob change costs one sub
|
||||
recomputation; a frame costs a lookup and a blit.
|
||||
|
||||
## Events and subs
|
||||
|
||||
Events are named for intent, and carry the clip they apply to:
|
||||
|
||||
```clojure
|
||||
[:clip/set-exposure clip-id 2]
|
||||
[:clip/toggle-kept clip-id cf]
|
||||
[:keying/re-key clip-id] ; not a setter; policy unchanged, redo the work
|
||||
[:keying/suggest clip-id]
|
||||
[:paint/commit-shape clip-id node-id cel-frame pts]
|
||||
[:node/set-parent clip-id node-id parent-id]
|
||||
[:override/set clip-id node-id :pts cf value]
|
||||
[:timeline/move-clip seq-id clip-id qf]
|
||||
```
|
||||
|
||||
Setters are fine where the intent *is* the value — `set-exposure` is honest.
|
||||
Where they differ, name the intent: `:keying/re-key` takes no value and is not
|
||||
`:clip/set-keys`.
|
||||
|
||||
Subs mirror the stages one-to-one, so the graph and the table above are the
|
||||
same object:
|
||||
|
||||
```
|
||||
::footage -> ::landmarks -> ::measurements -> ::conditioned -> ::channels -> ::resolved -> ::raster
|
||||
```
|
||||
|
||||
Each is a layer-3 sub over the previous plus the parameters for its stage only.
|
||||
That is what makes "drag `exposure`" recompute `::resolved` and nothing above it.
|
||||
|
||||
## Tool modes
|
||||
|
||||
A mode is a namespace, not a branch. Each exposes a map:
|
||||
|
||||
```clojure
|
||||
{:id :paint/pen
|
||||
:cursor "crosshair"
|
||||
:keymap {"Enter" [:paint/close-shape] "Escape" [:paint/abort]}
|
||||
:on-pointer (fn [ev ctx] ...)
|
||||
:overlay (fn [g ctx] ...) ; draws handles, never output pixels
|
||||
:enter :exit (fn [ctx] ...)}
|
||||
```
|
||||
|
||||
with two rules worth stating because the current `PaintUI` breaks both and gets
|
||||
away with it at this size:
|
||||
|
||||
- **In-progress gesture state is mode-local**, in a ratom the mode owns — not in
|
||||
app-db. An unclosed polygon must not appear in the undo history, and a drag
|
||||
must produce one undo entry rather than one per pointermove. The mode
|
||||
dispatches a single `:paint/commit-shape` on release.
|
||||
- **Overlays draw to a separate canvas.** Handles, vertex boxes and the green
|
||||
close-target are not indexed pixels and must never be in the raster. The
|
||||
current code composites them into the same context as the output, which is
|
||||
fine for a preview and lies to you the moment you want to judge the look.
|
||||
|
||||
The palette-index constraint and grid snapping stay where they are: they belong
|
||||
to `domain/cel` and `domain/palette`, enforced at commit, so no tool can bypass
|
||||
them.
|
||||
|
||||
## Where things live
|
||||
|
||||
Superseded in detail by Serialization and Baking below; the local picture is:
|
||||
|
||||
| Thing | Where | Keyed by |
|
||||
| --- | --- | --- |
|
||||
| project (tier 1) | server, plus a local autosave copy | project id, then leaf path |
|
||||
| footage frames, audio (tier 3) | on disk / OPFS, untouched | content hash |
|
||||
| analysis artifact, geometry bakes (tier 2) | OPFS, with IndexedDB for the index | content hash over every input |
|
||||
| render output | never persisted | — |
|
||||
|
||||
The cache key on the analysis artifact must include the **detector version**. A
|
||||
model upgrade that silently reuses old landmarks presents as "the tool got worse"
|
||||
with no event to attach it to — which is the argument for content-addressing
|
||||
rather than version-numbering everything derived.
|
||||
|
||||
## Baking
|
||||
|
||||
"No analysis during playback" is `docs/design.md`'s three-artifact rule with a
|
||||
number attached. The expensive stages are 2 (detect), 3 (measure) and, over a
|
||||
long take, 4 (condition). Stages 5–7 are arithmetic over small arrays. So there
|
||||
are two bakes, not one, and they answer different questions.
|
||||
|
||||
### Bake A — the analysis artifact. Mandatory.
|
||||
|
||||
Stages 2–4 for one footage at one set of conditioning parameters. This is what
|
||||
must never run while the transport is moving.
|
||||
|
||||
| Field | Shape |
|
||||
| --- | --- |
|
||||
| landmarks | `Float32Array[n_frames × 478 × 2]` |
|
||||
| transforms | `Float32Array[n_frames × 4]` — s, θ, tx, ty |
|
||||
| residual | `Float32Array[n_frames]` |
|
||||
| head-local rings | `Int16Array` per ring table, isotropic fixed point |
|
||||
| signals | openness, gaze, brow ends, aperture — `Float32Array[n_frames]` each |
|
||||
| **mouth crops** | `Uint8Array`, one small crop per frame |
|
||||
|
||||
The mouth crops are the non-obvious entry, and they are what makes remote work
|
||||
possible at all. `extractTeeth` reads source pixels, so without them a
|
||||
collaborator holding the analysis but not the 600 source PNGs cannot touch a
|
||||
single teeth knob. A 40×30 crop is about 1.2KB; a 600-frame take is under a
|
||||
megabyte against hundreds for the footage.
|
||||
|
||||
### Bake B — resolved geometry. For scale.
|
||||
|
||||
Stage 6 output per node. Not needed for one clip — resolving a frame is a key
|
||||
lookup and a transform — and needed the moment six roto tracks are live at 30fps.
|
||||
|
||||
**Fixed topology is what makes this flat.** Because every key of a part carries
|
||||
the same vertex count with the same vertex meanings, a node's whole bake is a
|
||||
rectangular array with no per-frame header and no indirection:
|
||||
|
||||
```
|
||||
pts Int16Array[n_frames × n_verts × 2] raster space, already grid-snapped
|
||||
state Uint8Array[n_frames] hidden flag + palette index
|
||||
```
|
||||
|
||||
Frame `f` of a node is the subarray at `f * n_verts * 2`. 600 frames × 20 verts
|
||||
is 48KB; a twelve-node clip about 600KB; six roto tracks about 3.5MB. So this is
|
||||
the payoff on the aesthetic constraint rather than a cost of it — a variable
|
||||
vertex count would force a per-frame offset table and a scan.
|
||||
|
||||
### Rasters are not baked
|
||||
|
||||
Stage 7 is the cheapest stage and the one most worth keeping live: palette and
|
||||
colour assignment are the last decisions and the ones you want to change in
|
||||
motion. Baking to pixels would freeze exactly the loop that should be instant.
|
||||
Small raster *thumbnails* for the strip and the timeline are a separate artifact
|
||||
with a separate cache.
|
||||
|
||||
### Unbaking is not an inverse
|
||||
|
||||
The bake never replaces its input. Authored state — channels, params, overrides —
|
||||
is always retained and always authoritative, so "unbake this roto and edit it" is
|
||||
a boolean about which side of the cache the renderer reads, not a computation.
|
||||
Instant by construction. Three rules keep it that way:
|
||||
|
||||
- **No destructive bake.** Nothing is discarded when something is baked.
|
||||
- **Hand corrections never go into the bake.** They are `:overrides`, applied at
|
||||
stage 6 *after* the bake is read, so a correction survives a rebake. This is
|
||||
`docs/design.md`'s "hand corrections must not live in the take", and it is the
|
||||
rule that stops baking from becoming a trap.
|
||||
- **Bakes are content-addressed** by a hash over every input that produced them:
|
||||
footage hash, detector version, and the parameters of each stage at or above
|
||||
the bake line. A stale bake is then unreachable rather than wrong, and a
|
||||
collaborator's bake is fetchable by the same key — see Collaboration.
|
||||
|
||||
### Partial bakes
|
||||
|
||||
Bake per node, per frame range. Scrubbing into an unbaked range resolves on
|
||||
demand and fills the cache; a worker bakes ahead of the playhead. Show it: a
|
||||
bake-state bar under the timeline is an affordance every NLE has already taught
|
||||
people to read, and it is honest about what is ready.
|
||||
|
||||
## Serialization, in three tiers
|
||||
|
||||
Cut by mutability and size. The cut is what makes collaboration and baking both
|
||||
tractable, because it decides what is allowed on the wire.
|
||||
|
||||
| Tier | What | Size | Synced | Undoable |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| 1 **authored** | palette, clips, scene graphs, channels, cels, overrides, sequences | KB | yes — it *is* the document | yes |
|
||||
| 2 **derived** | analysis artifacts, geometry bakes, thumbnails | MB | no — content-addressed blobs, fetched | no |
|
||||
| 3 **source** | frames, audio | MB–GB | no — immutable, by hash | no |
|
||||
|
||||
Tier 3 is produced **by the app**, not by `extract.sh`: wasm ffmpeg decodes the
|
||||
clip to frames and audio in the browser, and they are uploaded and served as
|
||||
content-addressed blobs. That makes tier 3 the same kind of thing as tier 2 — a
|
||||
cache with a hash — and it removes the one step that currently needs a terminal.
|
||||
Two consequences worth planning for: the extraction rate is recorded by the code
|
||||
that chose it rather than by a `manifest.json` a human might edit, and **offline
|
||||
work needs the frames already fetched**, so the local blob cache is what makes a
|
||||
plane usable. Decoding server-side instead is the same data model with a
|
||||
different worker; the format does not care which side runs it.
|
||||
|
||||
**Only tier 1 is the document.** A bake inside the shared document is a system
|
||||
that puts 48KB on the wire per vertex drag, and it is unnecessary, because tier 2
|
||||
is a pure function of tiers 1 and 3.
|
||||
|
||||
Tier 1 stays small enough to read. The only vertex data in it is what a human
|
||||
placed by hand: cel polygons and overrides. Everything traced is tier 2.
|
||||
|
||||
### Keys are a map, not a vector
|
||||
|
||||
```clojure
|
||||
;; not this
|
||||
{:keys [{:f 0 :pts [...]} {:f 4 :pts [...]}]}
|
||||
;; this
|
||||
{:keys {0 {:pts [...]}, 4 {:pts [...]}}}
|
||||
```
|
||||
|
||||
`activeKey` becomes a lookup in a sorted map rather than a scan; a single key
|
||||
becomes addressable as a path; and two people keying different frames of one part
|
||||
merge field-wise with no merge algorithm at all. `selectKeys` returns an array
|
||||
today — change it before anything depends on the order.
|
||||
|
||||
## Collaboration
|
||||
|
||||
`../tl` already has the model, and it is the right one to copy:
|
||||
|
||||
- **An authoritative server, and one broadcast source.** Edits go over HTTP; the
|
||||
server persists and then broadcasts the delta to the room. The socket is
|
||||
read-only for document state. No OT, no CRDT.
|
||||
- **Presence gossiped between peers** on the same socket, with the server doing
|
||||
exactly two things: handing out a connection id, and stamping the sender's
|
||||
server-side identity onto every message so nobody can post as somebody else.
|
||||
- **Parties as a pure function of the roster**, so every peer computes the same
|
||||
answer and the server knows nothing about them.
|
||||
- **Revisions**: one snapshot of the *authored layer* per save, with a user and a
|
||||
summary.
|
||||
|
||||
That maps onto the tiers exactly — the server stores tier 1 and snapshots tier 1,
|
||||
with tiers 2 and 3 as content-addressed blobs beside it.
|
||||
|
||||
### Why this model, and not a CRDT
|
||||
|
||||
The usual reason to reach for Yjs or Automerge is automatic convergence without
|
||||
conflict dialogs. **In this domain the primary contested data type is a vector
|
||||
path, and automatic convergence of a vector path is undesirable** — two people
|
||||
dragging vertices on one polygon merge into a shape neither drew. So the CRDT's
|
||||
headline benefit is neutralised on exactly the data that needs help, and the
|
||||
parts where it would work fine (scalars, maps keyed by frame) are the parts that
|
||||
LWW already handles.
|
||||
|
||||
What a CRDT would additionally cost here is not the library, it is the second
|
||||
source of truth: the canonical document moves into an opaque blob, and every
|
||||
server-side thing that reads the document — revisions, collaborator lists,
|
||||
thumbnail generation, the admin, any query — needs it materialised back out with
|
||||
`y-py` or a Node sidecar. That is real operational weight bought for a feature
|
||||
this domain does not want.
|
||||
|
||||
So: authoritative server, HTTP writes, LWW on addressed leaves, presence on the
|
||||
side. And **writes stay on HTTP rather than moving to the socket**, which is
|
||||
tl's choice and the right one: auth, idempotency, status codes, retries and
|
||||
conditional requests all come for free, and a dropped socket cannot lose a
|
||||
write. Latency is not the reason to move them, because latency is handled on the
|
||||
client (below) and never by waiting for a round trip.
|
||||
|
||||
### The model is standard; only tl's plumbing is hand-rolled
|
||||
|
||||
Worth separating. *Optimistic concurrency control over addressed resources with
|
||||
an entity tag* is RFC 7232 — `ETag`, `If-Match`, `412`/`409` — and it is what
|
||||
databases and HTTP APIs have done for thirty years. It is not a bespoke
|
||||
invention, and every HTTP client implements its half. The hand-rolled part of tl
|
||||
is the plumbing around it, so the move is to keep the model and replace the
|
||||
plumbing with the boring standard wherever one exists.
|
||||
|
||||
### Four things to add
|
||||
|
||||
tl's implementation is missing two pieces that hand-rolled sync usually omits,
|
||||
and arthur needs two more that tl does not because its write rate is low.
|
||||
|
||||
1. **Conditional writes.** `PUT /project/:id/leaf/<path>` with `If-Match: <etag>`,
|
||||
answering `409` on a stale write. tl's PUT replaces, so the loser's work
|
||||
disappears silently. For a knob that is survivable; for a painted cel it is
|
||||
the class of bug that ends trust in a tool. The 409 body carries the current
|
||||
value so the client can offer keep-mine / take-theirs.
|
||||
|
||||
2. **A monotonic project version, and gap detection.** The broadcast is
|
||||
fire-and-forget, so a peer that misses one delta — queue pressure, a
|
||||
reconnect gap — is silently stale forever. Every delta carries `seq`; a client
|
||||
that sees `seq > local + 1` refetches. This is what makes staleness
|
||||
self-healing instead of permanent, and it is about twenty lines.
|
||||
|
||||
3. **Gesture coalescing, client side.** This is the one real load difference
|
||||
between tl and arthur: tl's user annotates a clip every few seconds, arthur's
|
||||
user drags a vertex. A drag is **one** write on release, not one per
|
||||
pointermove. With the gesture held in mode-local state and committed once (see
|
||||
Tool modes), arthur's write rate is comparable to tl's and the model holds
|
||||
unchanged. Without it, no sync model survives.
|
||||
|
||||
4. **An outbox, so nothing ever waits on the network.** "No lag" is satisfied
|
||||
entirely on the client: apply locally, enqueue, reconcile on ack.
|
||||
|
||||
```clojure
|
||||
{:pending {"clip/7/cel/bg/12" {:value … :etag … :tries 0}}}
|
||||
```
|
||||
|
||||
With one rule that is easy to get wrong: **while a leaf has a pending local
|
||||
write, incoming broadcasts for that leaf are ignored.** Otherwise a remote
|
||||
delta lands mid-gesture and the shape snaps back and forth between the two
|
||||
values until the ack arrives.
|
||||
|
||||
### Two operational notes
|
||||
|
||||
- tl's `ROOMS` is process-local, as its own comment says. That is a single-worker
|
||||
ceiling, not a design flaw, and it has to move to Redis before a second worker.
|
||||
- **Revisions need a coarser trigger than every save.** tl snapshots a small
|
||||
annotation layer; arthur's tier 1 contains cel polygons, so a snapshot per save
|
||||
will bloat the table. Snapshot on an explicit "mark version", or time-boxed.
|
||||
|
||||
### When this model is wrong, and what replaces it
|
||||
|
||||
State the tripwire rather than trusting the judgement: **if 409s on cel leaves
|
||||
become common in real use, the merge unit is too coarse.** The answer then is a
|
||||
finer unit, not a CRDT, and there is a planned progression:
|
||||
|
||||
| Step | Merge unit | Gets you |
|
||||
| --- | --- | --- |
|
||||
| now | the cel (`cel/:node/:frame`) | two people on different frames |
|
||||
| next | the layer | two people on different layers of one cel |
|
||||
| last | **the stroke** — an append-only list of immutable stroke records | two people painting one layer at once |
|
||||
|
||||
The third step is worth seeing now because it is the escape hatch that keeps
|
||||
this decision from being a dead end. Once a drawing is a list of immutable
|
||||
records rather than a mutable point array, concurrent painting merges by
|
||||
construction — two people appending different records cannot conflict — and that
|
||||
is a domain-specific version of what a CRDT would have given, without the second
|
||||
source of truth. It is also a plausible thing to want for undo granularity
|
||||
anyway.
|
||||
|
||||
### Make the merge unit small instead of clever
|
||||
|
||||
Last-writer-wins clobbers only when its unit is too big. tl can save a scene;
|
||||
arthur cannot save a project, because one vertex drag would then clobber a
|
||||
collaborator's keying. The fix is addressing, not an algorithm:
|
||||
|
||||
```
|
||||
palette
|
||||
sequence/:sid
|
||||
clip/:cid/timing exposure, lead, kept frames
|
||||
clip/:cid/params/:area teeth | eyes | brows | mouth | plate
|
||||
clip/:cid/node/:nid one node: source, parent, stencil, z, colour
|
||||
clip/:cid/channel/:nid/:prop
|
||||
clip/:cid/cel/:nid/:frame
|
||||
clip/:cid/overrides/:nid/:prop
|
||||
```
|
||||
|
||||
An earlier draft of this list had `params` and `scene` as one leaf each, and both
|
||||
were too coarse: one person tuning teeth while another tunes eyes would have
|
||||
collided on every slider move, and two people adding nodes would have collided
|
||||
always. Split params **by feature area** and give every node its own leaf. With
|
||||
fractional `:z` there is no separate order leaf to contend on, which is the
|
||||
second thing fractional indices buy.
|
||||
|
||||
Each path is a **leaf**: independently addressed, independently versioned, LWW
|
||||
with an `If-Match` on its version. The boundaries are chosen so the things people
|
||||
actually do simultaneously land on different leaves — two people painting
|
||||
different cels, or keying different parts, never meet. Within a channel leaf the
|
||||
server merges **field-wise by frame**, which is the return on keys-as-a-map and
|
||||
about fifteen lines of Python.
|
||||
|
||||
### Where it stays honest about conflict
|
||||
|
||||
| Data | Concurrency | Resolution |
|
||||
| --- | --- | --- |
|
||||
| palette, clip params, knobs | scalar | LWW on the leaf. Two people tuning one clip will fight — that is a real thing to surface, not to merge. |
|
||||
| channels | map by frame | field-wise union, LWW per frame |
|
||||
| kept-frame set | add / remove | store `{frame → bool}`, not a set, so removals merge instead of vanishing |
|
||||
| cel layer order, `:z` | list insert | **fractional indices, not integers** |
|
||||
| cel polygon points | sequence | LWW on the whole layer, plus an advisory lease |
|
||||
| parentage | tree | LWW per node, with a cycle check that rejects the losing move |
|
||||
| playhead, selection, tool | ephemeral | presence only — never in the document, never in history |
|
||||
|
||||
Two of those are cheap now and expensive later.
|
||||
|
||||
**Fractional indices for anything ordered.** Integer `:z` and integer layer
|
||||
positions do not survive concurrent insertion: you get duplicates and gaps. A
|
||||
fractional key between neighbours makes concurrent inserts commute. Adopt it in
|
||||
`domain/node` and `domain/cel` from the first commit.
|
||||
|
||||
**Do not merge one polygon between two people.** Two simultaneous edits to the
|
||||
same path have no meaningful merge, and the result is a shape nobody drew. The
|
||||
layer is the unit, LWW, and presence shows who is on it before the collision.
|
||||
A coarse merge unit that is *visible* beats a fine one that invents geometry.
|
||||
|
||||
### Presence carries more than a cursor
|
||||
|
||||
tl's roster entry, extended with what arthur's views need:
|
||||
|
||||
```clojure
|
||||
{:cid "…" :user "…" :party nil :joinable true
|
||||
:clip cid :frame cf :playing? false :rate 1.0
|
||||
:node :mouth :tool :paint/pen
|
||||
:editing "clip/7/cel/bg/12"} ; advisory lease on a leaf
|
||||
```
|
||||
|
||||
`:editing` is what makes leaf-level LWW liveable — the collision becomes visible
|
||||
before it happens. Advisory only: no server enforcement, so there is no lock to
|
||||
leak when a tab closes.
|
||||
|
||||
### Follow mode, and why frames never go on the wire
|
||||
|
||||
tl's parties become arthur's follow mode nearly unchanged: mirror `:clip`,
|
||||
`:frame`, `:playing?` and the plate mode.
|
||||
|
||||
**Broadcast transport state, never frames.** On play, send
|
||||
`{:clip :frame :playing? :rate}` once; each peer's own audio element then becomes
|
||||
its own clock and the picture follows it exactly as it does locally. Per-frame
|
||||
messages would put thirty packets a second on the wire to reproduce something
|
||||
every peer can compute. A peer whose bake is cold shows a cold indicator and
|
||||
catches up rather than stalling the party.
|
||||
|
||||
### Going offline
|
||||
|
||||
Arthur is offline-capable almost by accident, and it is worth noticing why: tier
|
||||
1 is small enough to hold entirely in app-db, tier 2 is a local content-addressed
|
||||
cache, and tier 3 is a directory of PNGs that `extract.sh` put on your own disk.
|
||||
So when the network drops, **everything needed to scrub, key, paint, resolve and
|
||||
render is already local.** Nothing about playback or editing touches the server.
|
||||
|
||||
What stops is only the three things that are inherently remote: your writes stop
|
||||
acking, others' deltas stop arriving, and presence goes dark.
|
||||
|
||||
Two preconditions, or the above is a lie:
|
||||
|
||||
- **The outbox must be durable.** In memory, a closed tab loses the session. It
|
||||
goes in IndexedDB, written in the same task as the optimistic local apply —
|
||||
a few hundred microseconds, invisible, and the difference between "offline is
|
||||
fine" and "offline is a trap".
|
||||
- **Vendor MediaPipe's wasm.** `README.md` notes that `face_landmarker.task` is
|
||||
local but the wasm bundle is fetched from jsdelivr on first use. That makes
|
||||
stage 2 the one thing that silently requires a network, and it will be
|
||||
discovered on a plane. Vendor it or precache it in a service worker.
|
||||
|
||||
### Reconnecting
|
||||
|
||||
Refetch the document first, then drain the outbox — in that order, so conflicts
|
||||
are detected against current state rather than against a stale etag. Since the
|
||||
document is kilobytes, a full refetch is cheaper than reasoning about a long
|
||||
`seq` gap.
|
||||
|
||||
Then each pending write lands in one of four cases, and the design goal is that
|
||||
only the last one reaches a human:
|
||||
|
||||
| Case | Resolution |
|
||||
| --- | --- |
|
||||
| nobody touched your leaves | etags still match, outbox drains, converged. The common case. |
|
||||
| a **channel** leaf collided | **auto-merges** field-wise by frame — union the frames, LWW per frame. No human. |
|
||||
| a **scalar** leaf collided (knob, timing) | a compact list with one "keep all mine / take all theirs", because a knob is one number and the value is visible in the output anyway |
|
||||
| a **cel** leaf collided | see below — never a dialog |
|
||||
|
||||
The channel row is the third time keys-as-a-map pays for itself, and it is the
|
||||
reason a long offline session usually reconciles with no interaction at all.
|
||||
|
||||
### A colliding cel forks into a layer
|
||||
|
||||
Do not ask an artist to choose between two drawings in a modal. **Keep both:**
|
||||
their cel stays where it is, and yours is added on top as a layer named for the
|
||||
session that made it. Nothing is lost, nothing is silently clobbered, and the
|
||||
conflict is resolved by looking at it and deleting a layer — which is an
|
||||
operation the paint tool already has, and which is what an artist would do
|
||||
anyway.
|
||||
|
||||
This is the general principle for this whole class of conflict: **when the merge
|
||||
unit is coarse, resolve by stacking rather than by choosing.** A stack is
|
||||
reversible and legible; a choice made in a dialog against two thumbnails is
|
||||
neither.
|
||||
|
||||
### The awkward cases, named
|
||||
|
||||
- **A write to a leaf whose parent was deleted** — a node or clip removed while
|
||||
you were away. The server answers `404`; the client parks it in an orphaned-edits
|
||||
tray rather than dropping it. Rare, and the alternative is silent loss.
|
||||
- **A very long offline session against a busy project.** Merging may be the
|
||||
wrong frame entirely. Offer the escape hatch revisions already make possible:
|
||||
save the offline session as a revision, take theirs, and reconcile by hand from
|
||||
two named versions. Better than a hundred-row conflict list.
|
||||
- **Two peers both run detection offline.** Both compute the same analysis
|
||||
artifact and both try to upload it. Content addressing makes that idempotent —
|
||||
same hash, same bytes — so let it race rather than coordinating a claim.
|
||||
|
||||
### Undo is per-user — and this revises an earlier claim
|
||||
|
||||
An earlier section of this document said undo was free if app-db held only
|
||||
authored data, via `day8.re-frame/undo`. That is true single-player and wrong the
|
||||
moment two people edit: a snapshot of app-db reverts their work along with yours.
|
||||
|
||||
So undo is a **per-user stack of inverse leaf writes**, applied as ordinary
|
||||
edits. Your undo writes a leaf back to the value you last saw, and it conflicts
|
||||
with a concurrent editor in the same visible way any other write does. Keeping
|
||||
app-db small is still right — for validation cost and allocation — but it is no
|
||||
longer the undo story.
|
||||
|
||||
## The 30fps budget
|
||||
|
||||
At 320×200 stage 7 is not the problem. An even-odd scanline fill over 64,000
|
||||
pixels with a dozen mostly-small parts is well under 200,000 byte writes a frame:
|
||||
six million a second at 30fps. Six roto tracks off baked geometry adds a handful
|
||||
of subarray reads.
|
||||
|
||||
Two things in the current code *will* miss 30fps, and neither is the rasteriser:
|
||||
|
||||
- **`posterizeInto` does a `getImageData` every frame.** A GPU→CPU readback
|
||||
stalls the pipeline, and it is followed by 64,000 linear scans over the palette
|
||||
to find the nearest colour — around a million distance computations a frame.
|
||||
Both are avoidable: posterise once per source frame into a cached `Uint8Array`
|
||||
(it is a pure function of the frame, the transform and the palette), and
|
||||
replace the nearest-colour scan with a 5-5-5 RGB lookup table — 32KB, built
|
||||
once per palette.
|
||||
- **`drawRegistered` draws a full-resolution photo every frame.** Correct as a
|
||||
drawing reference on a held frame; not something to do thirty times a second.
|
||||
Pre-scale each source frame to raster size once, at load.
|
||||
|
||||
The rule both are instances of: **nothing that reads source pixels may run on the
|
||||
transport path.** That is the same line the bake boundary draws, which is why
|
||||
drawing it once, in stage 3, pays twice.
|
||||
|
||||
## Porting order
|
||||
|
||||
Tests first, and not as discipline — as an oracle.
|
||||
|
||||
1. **Port `synth.js` and `selftest.js` first**, to `cljs.test`. The synthetic
|
||||
track is the only fixture with ground truth, and the 105 assertions encode
|
||||
invariants that are invisible to inspection: the ring-simplicity check, the
|
||||
bowtie, the iris-pairing vote fed a deliberately swapped track.
|
||||
2. **Run both implementations on the same synthetic track and diff numerically**
|
||||
while porting each pure module. `fitSimilarity` and `procrustesMean` should
|
||||
agree to 1e-9; a disagreement is a port bug, not float noise.
|
||||
3. Port `landmarks`, `geom`, `ring`, `raster`, `take` — mechanical, and they are
|
||||
where the invariant comments live. **Carry the comments across verbatim.**
|
||||
They are the most valuable text in the repo and every one of them is a bug
|
||||
that already happened.
|
||||
4. Port `flow/measure/*` by lifting out of `pipeline.js`, then split
|
||||
`condition` and `key` out of it.
|
||||
5. Build `db`, `store`, `clock`, and stage 6 + 7 against a single hardcoded
|
||||
clip. At this point the synthetic take should render, with no UI beyond a
|
||||
canvas and a play button.
|
||||
6. Only then the UI, the modes, the timeline.
|
||||
7. Add the ring-simplicity and no-RGB checks as specs, so they fire on authored
|
||||
data too and not only in tests.
|
||||
|
||||
Three things are cheap in step 3 and expensive after step 6, so they go in early
|
||||
even though nothing needs them yet: **keys as a map** keyed by frame, **fractional
|
||||
indices** on `:z` and cel layer order, and **leaf addressing** for tier 1 — the
|
||||
paths under Collaboration, used as the shape of app-db even while single-player.
|
||||
None of them cost anything on day one and all three are retrofits that touch
|
||||
every namespace.
|
||||
|
||||
The wiring cross-check in `selftest.js` — every `el('id')` must exist in
|
||||
`index.html` — becomes unnecessary and should be deleted rather than ported. It
|
||||
is a test for a failure mode that Reagent does not have.
|
||||
|
||||
## Decisions I would defer
|
||||
|
||||
- **Whether `raster` stays JS.** 320×200 is 64,000 pixels and CLJS over a
|
||||
`Uint8Array` with no seq allocation handles it comfortably; port it and
|
||||
measure. Do not port it into idiomatic `map`/`reduce`.
|
||||
- **Interpolation.** `:hold` is the only interp the aesthetic wants, and the
|
||||
channel model should still carry `:interp` per node, because a parented
|
||||
transform on a painted cel is the one place a tween might be right. Do not
|
||||
implement it until something needs it.
|
||||
- **Multi-character scenes.** `character` is in the entity model as one field
|
||||
and should stay a stub until there are two.
|
||||
- **Audio per clip vs per sequence.** One master track is right until it is not.
|
||||
- **Whether bake B exists at all in v1.** One clip does not need it; the flat
|
||||
typed-array layout should be designed now and built when the second roto track
|
||||
appears.
|
||||
- **Sharing tier 2.** Content addressing means a collaborator *can* fetch a
|
||||
bake instead of recomputing, and also means nothing breaks if they do not.
|
||||
Ship the cache-miss path first and treat blob sharing as an optimisation —
|
||||
the analysis artifact is the only one where it clearly pays, because it is the
|
||||
only one that costs minutes.
|
||||
321
docs/port-plan.md
Normal file
321
docs/port-plan.md
Normal file
|
|
@ -0,0 +1,321 @@
|
|||
# arthur — port plan and handoff
|
||||
|
||||
Self-contained. You should not need any prior conversation to execute this.
|
||||
|
||||
## What arthur is
|
||||
|
||||
A tool that turns live-action video into 2D animation that reads as
|
||||
hand-authored: flat polygons, a tiny indexed palette, hard edges, 320×200, no
|
||||
antialiasing, motion carried by silhouette. It tracks a face out of a clip,
|
||||
reduces the lip contour to a handful of vertices, derives teeth from image
|
||||
content, and renders flat indexed fills.
|
||||
|
||||
It currently works, as vanilla JS ES modules with no build step. `python3
|
||||
serve.py`, open `127.0.0.1:8777`. **Synthetic take** exercises everything below
|
||||
detection with no video needed.
|
||||
|
||||
This plan converts it to ClojureScript + re-frame, restructured around one
|
||||
uniform animation data model, and adds a Django backend for persistence and
|
||||
(later) collaboration.
|
||||
|
||||
## Status of the existing documents
|
||||
|
||||
| File | What it is | Authority |
|
||||
| --- | --- | --- |
|
||||
| `js/**` | the working tool, ~4,800 lines | **authoritative.** The comments encode bugs that actually happened. |
|
||||
| `docs/animation-model.md` | the target data model: nodes, channels, symbols, time maps | build to this |
|
||||
| `docs/architecture.md` | module layout, stages, sync and baking design | build to this; much of it is future scope |
|
||||
| `docs/design.md`, `README.md` | prior synthesis by an earlier agent | useful, **not authoritative**. Revise freely. Do not treat its aesthetic claims as settled requirements. |
|
||||
|
||||
Where a document and the code disagree, the code wins, and the invariant list
|
||||
below is lifted from the code for exactly that reason.
|
||||
|
||||
## Target repo layout
|
||||
|
||||
Both halves live here. Django at the root, because `manage.py` at the root is the
|
||||
convention and keeps every `python manage.py` invocation working with no `cd`.
|
||||
|
||||
```
|
||||
arthur/
|
||||
mise.toml toolchain for both halves
|
||||
manage.py
|
||||
requirements.txt
|
||||
server/ Django project: settings, urls, asgi, wsgi
|
||||
clips/ Django app: models, views, consumers, routing, migrations
|
||||
frontend/ the CLJS app
|
||||
shadow-cljs.edn
|
||||
package.json
|
||||
src/arthur/** namespace root stays arthur.* whatever the dir is called
|
||||
test/arthur/**
|
||||
static/arthur/js/ shadow-cljs output, collected by Django staticfiles
|
||||
docs/
|
||||
js/ index.html serve.py extract.sh the old tool — see "the oracle"
|
||||
```
|
||||
|
||||
`clips` is a naming call, not a constraint — it is the Django app holding
|
||||
Project, Clip, Footage, Analysis, Leaf and Revision. Rename in one line if
|
||||
something fits better.
|
||||
|
||||
Dev runs two processes: Django serves the page, `shadow-cljs watch app` rebuilds
|
||||
into `static/arthur/js`. Set `:output-dir "../static/arthur/js"` in
|
||||
`shadow-cljs.edn`.
|
||||
|
||||
## Toolchain
|
||||
|
||||
`mise install` from the repo root. `mise.toml` pins java 21+, node 20, clojure,
|
||||
python 3.12, and creates `.venv`.
|
||||
|
||||
Verified to resolve cleanly: `reagent 1.2.0`, `re-frame 1.4.3`, current
|
||||
shadow-cljs.
|
||||
|
||||
## Scope
|
||||
|
||||
**In:** analysis → keyframes → playback. The pure numeric core, the animation
|
||||
data model, a player, the measurement stages, and freezing measurements into
|
||||
channels.
|
||||
|
||||
**Out, and do not build it:** paint and cels; `suggest` (it only decides which
|
||||
frames get a hand-drawn cel, so it has no job until drawing exists); the timeline
|
||||
and sequences; symbols and the plate library; multiplayer; the override layer.
|
||||
Each is designed for in `docs/architecture.md` and `docs/animation-model.md`.
|
||||
Leave the `:over` field present and empty; leave `:symbol` out entirely.
|
||||
|
||||
## The data model
|
||||
|
||||
Full specification in `docs/animation-model.md`. The subset to build:
|
||||
|
||||
```clojure
|
||||
;; The scene is a flat map of id -> node. Parent pointers, never nested maps.
|
||||
{:id :mouth :kind :poly :parent :head :z "a3" :stencil nil :span [0 240]
|
||||
:time {:mode :inherit} ; or {:mode :map :expose 2 :offset -1 :rate 1.0}
|
||||
:channels
|
||||
{[:xform :pos] {:animated? false :value [0.0 0.0]}
|
||||
[:xform :rot] {:animated? false :value 0.0}
|
||||
[:xform :scale] {:animated? false :value [1.0 1.0]}
|
||||
[:xform :skew] {:animated? false :value [0.0 0.0]}
|
||||
[:xform :anchor] {:animated? false :value [0.0 0.0]}
|
||||
[:geom :pts] {:animated? true :interp :hold
|
||||
:dense {:store "sha256:…" :offset 0 :stride 40 :frames 600}
|
||||
:generated {:by :roto/lips-outer :analysis "sha256:…"
|
||||
:params {:verts 8 :contour-avg 1}}
|
||||
:over []}
|
||||
[:style :color] {:animated? false :value :skin-dark}
|
||||
[:vis] {:animated? false :value true}}}
|
||||
```
|
||||
|
||||
Three channel shapes, one accessor `(value-at channel f)`:
|
||||
|
||||
- `{:animated? false :value v}` — static. A thing that simply exists.
|
||||
- `{:animated? true :interp :hold :keys {0 v, 4 v}}` — sparse, authored, in the
|
||||
document. **Keys are a map by frame, never a vector.** Store a plain map
|
||||
(transit loses sortedness) and build the sorted index in the resolver.
|
||||
- `{:animated? true :interp :hold :dense {...}}` — generated, one value per
|
||||
frame, in a typed array outside app-db.
|
||||
|
||||
`:generated` is provenance and **the renderer never reads it.** It is what the UI
|
||||
uses to offer a parameter panel instead of raw keys. It lives on the *channel*,
|
||||
not the node, because a node wants a rotoscoped `[:geom :pts]` and a
|
||||
hand-animated `[:xform :pos]` at the same time.
|
||||
|
||||
`:skew`, `:span`, `:anchor` and `:over` stay in the shape even though nothing
|
||||
drives them yet: each is a component of a decomposition or of a composition
|
||||
order, and adding one later migrates every stored transform.
|
||||
|
||||
Transform composition, per node:
|
||||
|
||||
```
|
||||
local = T(pos) · T(anchor) · R(rot) · K(skew) · S(scale) · T(-anchor)
|
||||
world = world(parent) · local
|
||||
```
|
||||
|
||||
## What the prototype knows that you would otherwise rediscover
|
||||
|
||||
**The JS is a prototype.** Its conclusions about what looks right are provisional
|
||||
and you may revisit any of them; several contradict each other already. But a few
|
||||
things in it are not taste — they are facts about MediaPipe, about the maths, or
|
||||
about what an operation means — and those cost real time to rediscover.
|
||||
|
||||
### Mechanical. Getting these wrong produces wrong output, not a different look.
|
||||
|
||||
1. **MediaPipe's normalised space is anisotropic.** It divides x by image *width*
|
||||
and y by *height*, so equal numbers do not mean equal pixels. Multiply x by
|
||||
`aspect = W/H` before any fit, or a "similarity" fitted in that space is not
|
||||
one and head roll comes out subtly wrong. When mapping pixels for an underlay,
|
||||
**both** axes divide by `imgH`.
|
||||
2. **MediaPipe's left/right naming is viewer-relative in some places and
|
||||
subject-relative in others.** Any left/right pairing read off a table is a coin
|
||||
flip, and a swap looks *almost* right — each eye still has an iris roughly
|
||||
where it belongs — so it survives inspection. Resolve it from geometry.
|
||||
3. **Ring tables are ordered traversals**, and slot position is the vertex's
|
||||
identity. That is what makes temporal correspondence possible at all, whatever
|
||||
you decide the shapes should look like. `subsampleSlots` returns ring
|
||||
*positions*, not landmark ids.
|
||||
4. **A wrongly-ordered ring self-intersects, and it is invisible at odd vertex
|
||||
budgets and obvious at even ones.** If you keep ordered rings, assert
|
||||
simplicity in a test; no amount of looking will catch it reliably.
|
||||
5. **Scaling a ring to thicken it collapses when the ring is degenerate** — a shut
|
||||
eyelid scaled by 1.1 is still shut, so the lash line vanishes on exactly the
|
||||
frames where it is the whole drawing. A fixed radial offset does not. Maths,
|
||||
not taste.
|
||||
6. **A fractional centre for a small integer-sized shape changes its size.**
|
||||
Round the origin, not the extents, or a 3px mark is 3px on one frame and 4px on
|
||||
the next.
|
||||
7. **Order of operations on time:** flooring onto a grid and shifting against the
|
||||
clock do not commute. Shift first and the floor discards it on most frames.
|
||||
|
||||
### Choices the prototype made. Revisit freely; here is what each was for.
|
||||
|
||||
| Choice | Its stated reason | How you would learn it was wrong |
|
||||
| --- | --- | --- |
|
||||
| similarity (4 DOF), not affine | extra DOF absorbs out-of-plane head rotation as shear and smears it into the mouth | the residual readout stops responding to head turn |
|
||||
| reference is the Procrustes mean over the shot, not frame 0 | no single frame's idiosyncrasies get baked into every other | one frame's detection error biases the whole take |
|
||||
| smooth the transform, not the contour | sparse keys at velocity minima rejected detector noise for free | it was already broken by a "bounded exception" once keys went dense, so it was never a law |
|
||||
| gaze measured against the eye's corner midpoint | measured against the lid, every blink drags the origin down and fakes a glance at the floor | gaze correlates with blinks |
|
||||
| one gaze shared by both eyes | at this size the per-eye difference is noise, and independent noise reads as wall-eyed | a wink or a real vergence is lost |
|
||||
| hold, never interpolate | a tweened mouth reads as puppet software | motion looks stepped rather than snappy |
|
||||
| palette indices, never sampled RGB | sampling colour produces a pixel-art filter irrecoverably | — |
|
||||
|
||||
These are where to look first if the output is wrong. They are also where to look
|
||||
first if you want to change the look.
|
||||
|
||||
## Conventions
|
||||
|
||||
- `domain/*` may not require `flow/*`; neither may require `re-frame`.
|
||||
- Every flow function is `(f params inputs) -> output`. No state, no db, no atoms.
|
||||
- Nothing below `subs/` calls `subscribe`.
|
||||
- Every analysis function that reads pixels takes a `debug?` flag and returns its
|
||||
intermediate masks alongside its result, the way
|
||||
`interior.js/extractTeeth(..., wantDebug)` already does.
|
||||
- Port the invariant comments across verbatim. They are the most valuable text in
|
||||
the repo.
|
||||
|
||||
## The oracle
|
||||
|
||||
**Keep `js/`, `index.html` and `serve.py` in the tree through step 5.** They cost
|
||||
nothing, `serve.py` still runs the old tool, and they are the numeric oracle:
|
||||
run both implementations on the same synthetic track and diff.
|
||||
`fit-similarity` and `procrustes-mean` should agree to **1e-9**; a larger gap is a
|
||||
port bug, not float noise.
|
||||
|
||||
**Parity proves the port is faithful, not that the answer is right.** The JS is a
|
||||
prototype, so keep the two kinds of test apart: a *parity* test pins behaviour
|
||||
while you move it, and is deleted once the move is done; a *correctness* test
|
||||
asserts something you have decided you want, and stays. Conflating them bakes the
|
||||
prototype's mistakes into the rewrite and makes them permanent. Delete them in one commit once the CLJS player renders
|
||||
the synthetic take correctly.
|
||||
|
||||
**Do not port the debug views** (`drawPanes`, `drawInteriorDebug`,
|
||||
`drawEyeOverlay`, `drawGazeDebug` in `js/app.js`). The knowledge in them is not
|
||||
the canvas calls — it is *which things you must see to tune teeth*: the source
|
||||
crop, the in-region mask, the surviving mask, and the local contour. That contract
|
||||
already exists as `extractTeeth(..., wantDebug)` returning
|
||||
`debugCanvas(src, inReg, mask, pw, ph, local)`. **Port the payload, skip the
|
||||
drawing.** Redrawing it is ten lines whenever it is wanted.
|
||||
|
||||
## Steps
|
||||
|
||||
Each step ends somewhere runnable. Do not proceed past a step whose "done" does
|
||||
not hold.
|
||||
|
||||
### 0 — scaffold and the oracle
|
||||
`mise install`. Create `frontend/` with shadow-cljs, reagent, re-frame. Port
|
||||
`synth.js` (the synthetic landmark generator, including its `swapIris` flag) and
|
||||
the numeric assertions from `selftest.js` to `cljs.test`.
|
||||
|
||||
**Done:** the suite runs and fails informatively.
|
||||
|
||||
### 1 — the pure bottom
|
||||
Port verbatim: `landmarks.js` → `domain/landmarks`, `mathutil.js` → `domain/geom`,
|
||||
ring helpers → `domain/ring`, `raster.js` → `domain/raster`, the palette →
|
||||
`domain/palette`.
|
||||
|
||||
**Done:** tests pass, including ring simplicity and the swapped-iris vote. Numeric
|
||||
agreement with the JS to 1e-9. Nothing renders.
|
||||
|
||||
### 2 — the data model, with no analysis in it
|
||||
`domain/channel` (`value-at` across all three shapes, plus a per-channel cursor),
|
||||
`domain/node` (transform composition), `domain/scene` (topological order by parent
|
||||
depth, `eval-frame` → draw ops in z order).
|
||||
|
||||
Hand-write a scene in EDN — a rectangle parented to a group whose
|
||||
`[:xform :pos]` is keyed on four frames — and render it through `domain/raster`
|
||||
into a canvas.
|
||||
|
||||
This is deliberately before any analysis. **The data model has never been
|
||||
validated; find out here**, with fifty lines to throw away, rather than after
|
||||
porting nine hundred lines of measurement into a shape that does not work.
|
||||
|
||||
**Done:** something moves on screen.
|
||||
|
||||
### 3 — the player
|
||||
`clock` (audio-clocked: `frame = ⌊currentTime · fps⌋`, so a slow loop drops frames
|
||||
instead of drifting; ½× and ¼× come free from `playbackRate`), the rAF loop, a
|
||||
`::resolver` sub, and transport UI.
|
||||
|
||||
The loop reads and blits and **dispatches nothing**. The sub yields a resolver
|
||||
closure; the loop applies it at the playhead. The playhead itself lives in app-db
|
||||
like everything else — with layer-2 extractors and layer-3 computations, a
|
||||
playhead tick does not invalidate the expensive stages.
|
||||
|
||||
**Done:** the hand-written scene plays at 30fps against audio, scrubs, and runs at
|
||||
½× and ¼×.
|
||||
|
||||
### 4 — measure: anchor and mouth
|
||||
Port `stabilize` and the lip rings out of `pipeline.js`. Split **condition**
|
||||
(`smoothTransforms`, `smoothContours`) into its own stage so the two smoothing
|
||||
knobs do not re-run measurement.
|
||||
|
||||
**Done:** measured numbers match the JS on the synthetic track.
|
||||
|
||||
### 5 — freeze
|
||||
The new module, and the heart of this work: measurements → channels. A dense
|
||||
`[:geom :pts]` block per node — `Int16Array[frames × verts × 2]`, raster space,
|
||||
grid-snapped — with `:generated` attached. Fixed topology is what makes this a
|
||||
rectangular array with no per-frame header.
|
||||
|
||||
**Done:** the synthetic take plays back as a moving mouth. Full vertical slice.
|
||||
|
||||
### 6 — detect
|
||||
MediaPipe interop behind one namespace; real frames, real audio, real fps from
|
||||
the manifest. **Vendor the wasm** rather than fetching from jsdelivr — it is
|
||||
currently the only thing in the tool that silently requires a network.
|
||||
|
||||
**Done:** real footage plays back as a rotoscoped mouth.
|
||||
|
||||
### 7 — the rest of measure
|
||||
Eyes (openness, gaze, iris pairing vote, blink resolution with its `hold`), brows
|
||||
(raise and tilt at both ends, both correspondence votes), interior (otsu,
|
||||
morphology, components, radial contour). Each keeps its `debug?` payload.
|
||||
|
||||
Two things fall out of the model instead of being written: the brow's
|
||||
measure-the-height-out-and-put-it-back is `[:geom :pts]` plus `[:xform :pos]`, two
|
||||
channels on one node; and the iris is a `:disc` node parented to the lid ring and
|
||||
stencilled by the sclera.
|
||||
|
||||
**Done:** parity with the JS tool, minus paint.
|
||||
|
||||
### 8 — knobs
|
||||
The parameter UI, as leaf-addressed params in app-db
|
||||
(`clip/:cid/params/:subject/:area` — see below), so the sync layer added later has
|
||||
nothing to retrofit.
|
||||
|
||||
### 9 — backend and take export
|
||||
Django project, the `clips` app, models for Project/Clip/Footage/Analysis/Leaf,
|
||||
and project load/save. Port `take.js` — sixty-five lines, and it is the proof the
|
||||
model serialises.
|
||||
|
||||
## Two things to not foreclose
|
||||
|
||||
The feature controls will later be rethought to handle more than one face,
|
||||
periodic occlusion, stable identity across frames, and feature groups with their
|
||||
own parameters. That design can wait; two decisions here are free now and
|
||||
annoying to reverse:
|
||||
|
||||
- **Presence is not visibility.** An occluded subject has *no value* on a frame,
|
||||
which is different from a part being hidden. Give every dense block a
|
||||
`Uint8Array` state mask per frame and let it mean *absent* as well as hidden.
|
||||
- **Params carry a subject segment.** `clip/:cid/params/:subject/:area`, with one
|
||||
subject today. Adding a path segment later touches every read and write.
|
||||
|
||||
The identity tracker, when it comes, should use the same pattern the iris and brow
|
||||
correspondences already use: vote across every frame rather than trusting one.
|
||||
Loading…
Add table
Add a link
Reference in a new issue