noe/arthur
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Port steps 0-1: scaffold, the oracle, and the pure bottom

Browse source 
Scaffolds frontend/ (shadow-cljs, reagent 1.2.0, re-frame 1.4.3) and ports
everything below the data model, with the JS kept as a numeric oracle.

  domain/landmarks  index tables, verbatim
  domain/ring       subsample, offset, simplicity
  domain/geom       similarity fit, procrustes, moving average
  domain/raster     indexed scanline fill, stencil, disc, rect
  domain/palette    the ramp, and the no-sampled-RGB rule

58 tests, 166 assertions. Parity with js/ on the identical 72-frame synthetic
track: fit-similarity, procrustes-mean, fit-residual, moving-average,
smooth-transforms, offset-ring and subsample-slots to 1e-9; the raster
pixel-for-pixel over the whole buffer.

Three deviations from the JS, each for a reason:

- synth.cljs jitters from a SEEDED generator, not Math.random. Parity is only
  checkable if both sides can be handed the same track, and a failing assertion
  has to be reproducible. `:rand-fn` takes the generator over, so oracle.mjs
  stubs js/Math.random and js/ itself stays untouched.

- raster/->rgba replaces toImageData. ImageData is a DOM type and domain/ may
  not touch the DOM; returning plain bytes also lets the
  no-intermediate-colours assertion run in node. ui/canvas wraps it later.

- offset-ring lives in domain/ring, not domain/geom, per architecture.md: it is
  an operation on an ordered traversal, not on a transform.

Step 1's "done" also names the swapped-iris vote, but pairIrises is in
pipeline.js and belongs to step 7. The precondition is asserted instead --
`:swap-iris` really does move both blocks -- so the vote will have a track that
disagrees with it when it arrives.

One finding, recorded in full in the test that measures it: smooth-transforms
buys nothing on the synthetic track. Against jitter-free ground truth, radius 1
helps by 17% on one noise realisation and hurts by 0.5% on another, so its
benefit is within noise; from radius 2 up the cost is unambiguous, and by radius
5 the filter is below the true motion's own high-frequency energy, i.e.
smoothing away performance. The test pins the shape of the knob rather than a
preferred value. This may say more about the synth's jitter being unrealistically
small (+/-0.001 normalised) than about the knob; step 6 settles it on real
footage.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Olive Vaughn 2026-09-27 14:43:34 -04:00
parent 082d8561d2
commit eb06be005c
25 changed files with 4932 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

351
docs/animation-model.md Normal file
View file

@ -0,0 +1,351 @@
# arthur — the animation model
The data that describes a moving picture: what the primitives are, how they
nest, how they change over time, and how rotoscoped and hand-authored work end
up being the same thing with one flag between them.
`docs/design.md` is the aesthetic argument. `docs/architecture.md` is where the
code goes. This is the type that both of them are about.
## What this replaces
`docs/design.md` has a table of five kinds of part — plate, feature, interior,
primitive, scalar — each with its own source, vocabulary and interpolation. That
table is a good description of **where data comes from** and a bad description of
**what data is**, and the current code follows it too literally: eyes, brows,
teeth and mouth each get their own build function, their own key shape and their
own path through `take.js`.
They are all one thing. A part is a **node** with **channels**, and the five
kinds collapse into differences of which channels exist and who filled them in.
## Prior art, and what each one gets right
| System | The idea worth taking |
| --- | --- |
| **Flash / SWF** | A **library of symbols** and a timeline of **instances** at depths. "Framed" content that simply exists on a frame, versus tweened content. `DefineMorphShape` requires matching vertex counts — the fixed-topology rule, arrived at from the other direction. |
| **Blender** | Animation is **addressed by path into the data** (`location[0]`), not stored as fields on the object. An Action is a bag of F-Curves. That decoupling is what makes the dope sheet, the graph editor and the NLA three views of one dataset. Also: parenting captures a `parent_inverse` so the child does not jump. |
| **After Effects** | Every leaf property is animatable, uniformly. Property groups form a tree. Pre-comps nest arbitrarily and a pre-comp is just a layer. |
| **Lottie** | The uniform property shape: `{a: 0, k: <value>}` or `{a: 1, k: [<keys>]}`. One representation for static and animated, which is exactly "framed or keyframed". |
| **Grease Pencil** | A 2D layer holds frames at frame numbers, and a frame **holds until the next one**. Hold is the default, not a special case. |
What none of them get right for this project: colour. All four store RGB on the
shape. `docs/design.md` forbids that, so colour is a palette index here and it is
a channel like any other.
## The one idea
**Analysis is a channel generator.** It does not produce a different kind of
data; it produces keys, densely, on the same channels a hand would fill in
sparsely. So:
```
footage ──▶ analysis ──▶ FREEZE ──▶ channels on nodes ──▶ evaluate ──▶ raster
▲
hand authoring ──┘
```
Freezing is not a conversion into a second format. There is one format, and
freezing fills it in. That is what makes "the only difference is a special flag"
literally true: the flag is provenance on a channel, and nothing in the renderer
reads it.
## Node
A node is an instance in the scene. The tree is stored **flat, with parent
pointers** — never as nested maps.
```clojure
{:id :mouth
:name "mouth"
:kind :poly ; :poly :disc :rect :group :bitmap :symbol
:parent :head ; nil at the root
:z "a3" ; fractional index, ordered among all siblings
:symbol nil ; or :sym/blink — see Symbols
:stencil :mouth-in ; colour-key clip; structural, not a channel
:span [0 240] ; in/out in the parent's frame space
:pinv [1 0 0 1 0 0] ; parent-inverse, captured when parented
:channels {...}}
```
Flat with pointers, for four reasons that all point the same way: any node is
addressable without a walk; reparenting is a one-field write rather than a
subtree move; an edit to a leaf does not change the identity of its ancestors, so
re-frame's structural sharing keeps ancestor subs from invalidating; and it is
what lets every node be its own sync leaf. Flash, Blender and AE all store it
this way.
`:span` is Lottie's `ip`/`op` and Flash's `PlaceObject`/`RemoveObject`: the range
over which the node exists at all. Distinct from a `[:vis]` channel, which
blinks an existing node on and off.
## Channel
Every animatable property is a channel, and channels are addressed **by path**:
```clojure
:channels
{[:xform :pos] {:animated? false :value [0.0 0.0]}
[:xform :rot] {:animated? false :value 0.0}
[:xform :scale] {:animated? false :value [1.0 1.0]}
[:xform :skew] {:animated? false :value [0.0 0.0]}
[:xform :anchor]{:animated? false :value [0.0 0.0]}
[:geom :pts] {:animated? true :interp :hold :dense {...} :generated {...}}
[:style :color] {:animated? false :value :skin-dark}
[:vis] {:animated? true :interp :hold :keys {0 true, 37 false}}}
```
A path is a **vector**, not a string — CLJS maps take vectors as keys natively,
so Blender's `data_path` idea arrives with no parsing. The set of valid paths for
a node follows from its `:kind`, and that is a spec, not a schema migration.
Three channel shapes, and the uniformity across them is the point:
```clojure
;; FRAMED — one static thing. No animation, no vertex correspondence to worry
;; about. A painted background cel is this.
{:animated? false :value v}
;; KEYED — sparse, authored, in the document. Undoable and syncable.
{:animated? true :interp :hold :keys {0 v, 4 v, 12 v}}
;; DENSE — generated, one value per frame, held in tier 2 as a typed array.
{:animated? true :interp :hold
:dense {:store "sha256:…" :offset 0 :stride 40 :frames 600}
:generated {...}}
```
`:interp` defaults to `:hold`, which `docs/design.md` requires of every cut part.
A key may carry its own `:interp` to override the channel's, which is how Lottie
and Blender both do per-key easing; nothing uses it yet and the door is cheap to
leave open.
### Keys are a map by frame, not a list
Already argued in `docs/architecture.md` for merge reasons; here it also gives
"the most recent key at or before `f`" as a `rsubseq` on a sorted map instead of
a scan. **Store a plain map** in the document — transit and JSON both lose
sortedness — and build the sorted index in the resolver.
### The flag lives on the channel, not the node
```clojure
:generated {:by :roto/lips-outer
:analysis "sha256:…" ; which analysis artifact
:params {:verts 8 :contour-avg 1 :aperture-cut 0.004}}
```
Present means the UI offers a parameter panel and a re-freeze button. Absent
means the UI offers the keys directly. **The renderer never reads it.**
It belongs on the channel rather than the node because a node routinely wants
both at once: a mouth whose `[:geom :pts]` is rotoscoped and whose `[:xform :pos]`
is hand-animated to sit on a plate. Putting the flag on the node would forbid the
most useful thing in the model.
### Channels are layered
A channel is a base plus optional override layers, and a layer declares how it
combines:
```clojure
{:animated? true :interp :hold
:dense {...} :generated {...}
:over [{:blend :offset :keys {88 [2 0], 96 [0 0]}}
{:blend :replace :keys {104 [[3 7] [4 7] …]}}]}
```
- **`:offset`** adds a delta to the base. "Nudge the mouth two pixels right for
ten frames" survives a re-freeze at different parameters, because it was never
a position — it was a correction.
- **`:replace`** wins outright. For the frame where detection simply failed.
This is what `docs/design.md` means by an override layer, and it is why
re-freezing is safe: the base is regenerated, the layers are untouched. It is
Blender's NLA blending and AE's effect stack at one property.
## Transform: decomposed, never a matrix
```clojure
{:pos [x y] :rot θ :scale [sx sy] :skew [kx ky] :anchor [ax ay]}
```
Stored decomposed for two reasons. Each component has to be independently
keyframable, which is the entire point of channels. And interpolating matrix
entries is meaningless — a rotation tweened through its matrix shears on the way.
Composition, per node:
```
local = T(pos) · T(anchor) · R(rot) · K(skew) · S(scale) · T(-anchor)
world = world(parent) · pinv · local
```
`:anchor` is Flash's registration point and Blender's origin: rotation and scale
happen about it, and getting it wrong is why hand-placed parts swing rather than
turn.
`:pinv` is Blender's `parent_inverse`, captured at the moment of parenting so the
child does not jump when it acquires a parent. Small, and its absence is the kind
of thing that makes a parenting feature feel broken.
**The similarity fit already produces a decomposition.** `fitSimilarity` returns
`{s θ tx ty}`, which drops straight into `[:xform :scale]`, `[:xform :rot]` and
`[:xform :pos]` with no conversion. The analysis output and the animation model
meet without an adapter, which is a sign the decomposition is the right one.
## Time maps — exposure, lead and symbol timing are one thing
Every node may map the frame it is evaluated at:
```clojure
:time {:mode :inherit} ; the default, and almost always right
:time {:mode :map :expose 2 :offset -1 :rate 1.0 :loop? false}
```
Three features that look unrelated are this one mechanism:
- **exposure** is `⌊f/n⌋·n`,
- **mouth lead** is `f + k`,
- **a symbol instance's timing** is `(f - at)·rate + in`, with optional looping.
Composed along the nesting chain, outermost first. Two rules follow, and they are
different rules:
- **Exposure inherits strictly.** `docs/design.md` is emphatic that everything
rides one grid, because a head cutting on odd frames against a mouth cutting on
even ones reads as two performances. The model permits a per-node grid; the
default must be `:inherit`, and setting it lower is a deliberate act the UI
should make feel like one.
- **Offset is per-node by design.** Mouth lead applies to performance nodes and
*not* to the plate, which is the whole point of it — so the offset genuinely
belongs at the node, not the clip.
## Symbols
The Flash idea, kept:
```clojure
:library
{:sym/head {:kind :poly :channels {...}}
:sym/blink {:kind :timeline :frames 3 :nodes [...]}}
```
A node with `:symbol :sym/blink` is an **instance**. Its own channels compose
*over* the symbol's, so one definition can be placed many times and tinted,
offset or retimed at each placement. A `:timeline` symbol has its own frame space,
which the instance's `:time` maps into — that is Flash's MovieClip, Lottie's
precomp and AE's pre-comp, and it is how a three-frame blink gets reused at
frames 40, 88 and 200 without copying it.
This is also where `docs/design.md`'s "closed vocabulary is right for the head"
lands: a plate library is a set of `:sym/head-*` definitions, and the strip
chooses which instance is placed on which frame. The take format's `plate=` field
becomes an instance reference.
## Evaluating a frame
```clojure
(defn eval-frame
"Scene at clip frame f -> draw ops in z order. Pure."
[scene f] ...)
```
1. Walk nodes in **topological order** by parent depth (cached; recompute only
when parentage changes).
2. Skip nodes outside `:span`.
3. Apply the node's time map to get its own local frame `fn`.
4. **Sample** each channel at `fn`: a map lookup for framed, a sorted-index
lookup for keyed, an array read for dense. Then apply `:over` layers.
5. Compose `world` from the parent's.
6. Transform geometry into raster space, writing into a **preallocated buffer**
owned by the node.
7. Emit `{:kind :poly :pts buf :n 20 :color idx :stencil id}`.
8. Sort by resolved `z`.
The op list is the boundary with stage 7 in `docs/architecture.md`: the
rasteriser takes ops and knows nothing about nodes, channels or time.
### Making it fast in CLJS
Three things, and only these three matter:
- **Decomposed and persistent for storage; flat and mutable for evaluation.**
Composed transforms are 6-element `Float64Array`s, not maps. Every renderer
does this; the storage form and the evaluation form are allowed to differ.
- **A cursor per channel.** Playback is sequential, so "most recent key at or
before `f`" is an advance of a saved index, O(1) amortised. Binary search only
on a seek. This is the difference between a `rsubseq` allocation per channel per
frame and none.
- **Preallocated point buffers per node.** Fixed topology means the size is known
at freeze time, so a frame allocates nothing. At 30fps, per-frame allocation is
the only thing that will make this stutter.
### What is in app-db, and what is not
| In app-db (tier 1) | In tier 2, behind a handle |
| --- | --- |
| nodes, parentage, z, spans, stencils | dense channel blocks |
| channel definitions, `:interp`, `:generated` | analysis artifacts |
| **framed** values, **keyed** keys, `:over` layers | preallocated eval buffers |
| library / symbol definitions | composed transform scratch |
The rule: **anything a human placed is in the document; anything a generator
produced is a handle.** Which is the same line `docs/architecture.md` draws for
sync and baking, arrived at again from the renderer's side.
## The current parts, in this model
Proof that it covers what exists, not just what is wanted:
| Now | Becomes |
| --- | --- |
| `mouth` outer ring, every frame | node `:mouth`, `[:geom :pts]` dense, `:generated {:by :roto/lips-outer}` |
| `mouth_in`, hidden below aperture | node `:mouth-in`, parent `:mouth`, `[:geom :pts]` dense + `[:vis]` dense |
| `teeth` from image content | node `:teeth`, stencil `:mouth-in`, `[:geom :pts]` dense, `:generated {:by :interior/teeth}` |
| lid rings | nodes `:lid-r/-l`, `[:geom :pts]` dense |
| lash line (`offsetRing`) | not data — a stage-6 parameter on the node, `{:grow px}` |
| iris disc | node `:iris-r`, `:kind :disc`, parent `:lid-r`, stencil `:sclera-r`, `[:xform :pos]` dense (quantised at freeze), radius framed |
| square pupil | node `:pupil-r`, `:kind :rect`, parent `:iris-r`, stencil `:iris-r` |
| brow ring + quantised raise | node `:brow-r`, `[:geom :pts]` dense (the traced ring with height removed), `[:xform :pos]` dense (the quantised raise). **The decomposition design.md insists on is two channels.** |
| head plate, kept frames | node `:head`, `:symbol` per instance, keys on `[:symbol]` at kept frames |
| painted background cel | node per layer, `[:geom :pts]` **framed**, `[:style :color]` framed |
| `mouth lead` | `:time {:offset k}` on performance nodes only |
| `exposure` | `:time {:expose n}` on the clip root, inherited |
| hand correction | an `:over` layer, `:offset` or `:replace` |
The brow row is the one worth looking at twice. `docs/design.md` argues at length
that the traced ring already contains the height, so the quantised raise must be
measured *out* and put *back* or the brow moves twice. In this model that is not
an argument to remember — it is two channels on one node, and getting it wrong
would mean writing the height into both.
## Format on disk and on the wire
Tier 1 is EDN/transit: the node tree, channel definitions, framed values, keys,
layers, library. Kilobytes, human-readable, diffable, and leaf-addressable for
sync.
Dense blocks are separate content-addressed binaries — `Int16Array` for raster
geometry, `Float32Array` for transforms — with a small header naming the channel
path, frame count and stride.
**Not Lottie internally**, despite the property shape being borrowed from it.
Lottie has no palette-indexed colour, its shapes are bezier with in/out tangents
where these are integer polygons, and its interpolation defaults are the opposite
of what is wanted. It is a good **export** target later, next to the `.take`
writer, and a bad internal format.
## Deferred
- **Per-key easing.** The structure allows it; nothing should use it until a
parented transform on a painted cel asks for it.
- **More than two channel layers.** The `:over` vector is already a list; a real
blend stack with weights is the NLA, and it is not needed to fix a bad frame.
- **Skew beyond the field.** `[:xform :skew]` is in the transform and in the
composition order from the start, because adding a component to a decomposition
later means migrating every stored transform.
- **Instance channel overrides on symbols.** Compose-over is specified; only
colour and transform need it at first.
- **Constraints and drivers.** Blender's other half. A gaze that aims at a null
object is the obvious first one, and it is a long way off.