arthur/docs/animation-model.md

# arthur — the animation model

The data that describes a moving picture: what the primitives are, how they
nest, how they change over time, and how rotoscoped and hand-authored work end
up being the same thing with one flag between them.

`docs/design.md` is the aesthetic argument. `docs/architecture.md` is where the
code goes. This is the type that both of them are about.

## What this replaces

`docs/design.md` has a table of five kinds of part — plate, feature, interior,
primitive, scalar — each with its own source, vocabulary and interpolation. That
table is a good description of **where data comes from** and a bad description of
**what data is**, and the current code follows it too literally: eyes, brows,
teeth and mouth each get their own build function, their own key shape and their
own path through `take.js`.

They are all one thing. A part is a **node** with **channels**, and the five
kinds collapse into differences of which channels exist and who filled them in.

## Prior art, and what each one gets right

| System | The idea worth taking |
| --- | --- |
| **Flash / SWF** | A **library of symbols** and a timeline of **instances** at depths. "Framed" content that simply exists on a frame, versus tweened content. `DefineMorphShape` requires matching vertex counts — the fixed-topology rule, arrived at from the other direction. |
| **Blender** | Animation is **addressed by path into the data** (`location[0]`), not stored as fields on the object. An Action is a bag of F-Curves. That decoupling is what makes the dope sheet, the graph editor and the NLA three views of one dataset. Also: parenting captures a `parent_inverse` so the child does not jump. |
| **After Effects** | Every leaf property is animatable, uniformly. Property groups form a tree. Pre-comps nest arbitrarily and a pre-comp is just a layer. |
| **Lottie** | The uniform property shape: `{a: 0, k: <value>}` or `{a: 1, k: [<keys>]}`. One representation for static and animated, which is exactly "framed or keyframed". |
| **Grease Pencil** | A 2D layer holds frames at frame numbers, and a frame **holds until the next one**. Hold is the default, not a special case. |

What none of them get right for this project: colour. All four store RGB on the
shape. `docs/design.md` forbids that, so colour is a palette index here and it is
a channel like any other.

## The one idea

**Analysis is a channel generator.** It does not produce a different kind of
data; it produces keys, densely, on the same channels a hand would fill in
sparsely. So:

```
footage ──▶ analysis ──▶ FREEZE ──▶ channels on nodes ──▶ evaluate ──▶ raster
                                        ▲
                       hand authoring ──┘
```

Freezing is not a conversion into a second format. There is one format, and
freezing fills it in. That is what makes "the only difference is a special flag"
literally true: the flag is provenance on a channel, and nothing in the renderer
reads it.

## Node

A node is an instance in the scene. The tree is stored **flat, with parent
pointers** — never as nested maps.

```clojure
{:id      :mouth
 :name    "mouth"
 :kind    :poly                  ; :poly :disc :rect :group :bitmap :symbol
 :parent  :head                  ; nil at the root
 :z       "a3"                   ; fractional index, ordered among all siblings
 :symbol  nil                    ; or :sym/blink — see Symbols
 :stencil :mouth-in              ; colour-key clip; structural, not a channel
 :span    [0 240]                ; in/out in the parent's frame space
 :pinv    [1 0 0 1 0 0]          ; parent-inverse, captured when parented
 :channels {...}}
```

Flat with pointers, for four reasons that all point the same way: any node is
addressable without a walk; reparenting is a one-field write rather than a
subtree move; an edit to a leaf does not change the identity of its ancestors, so
re-frame's structural sharing keeps ancestor subs from invalidating; and it is
what lets every node be its own sync leaf. Flash, Blender and AE all store it
this way.

`:span` is Lottie's `ip`/`op` and Flash's `PlaceObject`/`RemoveObject`: the range
over which the node exists at all. Distinct from a `[:vis]` channel, which
blinks an existing node on and off.

## Channel

Every animatable property is a channel, and channels are addressed **by path**:

```clojure
:channels
{[:xform :pos]   {:animated? false :value [0.0 0.0]}
 [:xform :rot]   {:animated? false :value 0.0}
 [:xform :scale] {:animated? false :value [1.0 1.0]}
 [:xform :skew]  {:animated? false :value [0.0 0.0]}
 [:xform :anchor]{:animated? false :value [0.0 0.0]}
 [:geom :pts]    {:animated? true :interp :hold :dense {...} :generated {...}}
 [:style :color] {:animated? false :value :skin-dark}
 [:vis]          {:animated? true :interp :hold :keys {0 true, 37 false}}}
```

A path is a **vector**, not a string — CLJS maps take vectors as keys natively,
so Blender's `data_path` idea arrives with no parsing. The set of valid paths for
a node follows from its `:kind`, and that is a spec, not a schema migration.

Three channel shapes, and the uniformity across them is the point:

```clojure
;; FRAMED — one static thing. No animation, no vertex correspondence to worry
;; about. A painted background cel is this.
{:animated? false :value v}

;; KEYED — sparse, authored, in the document. Undoable and syncable.
{:animated? true :interp :hold :keys {0 v, 4 v, 12 v}}

;; DENSE — generated, one value per frame, held in tier 2 as a typed array.
{:animated? true :interp :hold
 :dense {:store "sha256:…" :offset 0 :stride 40 :frames 600}
 :generated {...}}
```

`:interp` defaults to `:hold`, which `docs/design.md` requires of every cut part.
A key may carry its own `:interp` to override the channel's, which is how Lottie
and Blender both do per-key easing; nothing uses it yet and the door is cheap to
leave open.

### Keys are a map by frame, not a list

Already argued in `docs/architecture.md` for merge reasons; here it also gives
"the most recent key at or before `f`" as a `rsubseq` on a sorted map instead of
a scan. **Store a plain map** in the document — transit and JSON both lose
sortedness — and build the sorted index in the resolver.

### The flag lives on the channel, not the node

```clojure
:generated {:by       :roto/lips-outer
            :analysis "sha256:…"          ; which analysis artifact
            :params   {:verts 8 :contour-avg 1 :aperture-cut 0.004}}
```

Present means the UI offers a parameter panel and a re-freeze button. Absent
means the UI offers the keys directly. **The renderer never reads it.**

It belongs on the channel rather than the node because a node routinely wants
both at once: a mouth whose `[:geom :pts]` is rotoscoped and whose `[:xform :pos]`
is hand-animated to sit on a plate. Putting the flag on the node would forbid the
most useful thing in the model.

### Channels are layered

A channel is a base plus optional override layers, and a layer declares how it
combines:

```clojure
{:animated? true :interp :hold
 :dense {...} :generated {...}
 :over  [{:blend :offset  :keys {88 [2 0], 96 [0 0]}}
         {:blend :replace :keys {104 [[3 7] [4 7] …]}}]}
```

- **`:offset`** adds a delta to the base. "Nudge the mouth two pixels right for
  ten frames" survives a re-freeze at different parameters, because it was never
  a position — it was a correction.
- **`:replace`** wins outright. For the frame where detection simply failed.

This is what `docs/design.md` means by an override layer, and it is why
re-freezing is safe: the base is regenerated, the layers are untouched. It is
Blender's NLA blending and AE's effect stack at one property.

## Transform: decomposed, never a matrix

```clojure
{:pos [x y] :rot θ :scale [sx sy] :skew [kx ky] :anchor [ax ay]}
```

Stored decomposed for two reasons. Each component has to be independently
keyframable, which is the entire point of channels. And interpolating matrix
entries is meaningless — a rotation tweened through its matrix shears on the way.

Composition, per node:

```
local = T(pos) · T(anchor) · R(rot) · K(skew) · S(scale) · T(-anchor)
world = world(parent) · pinv · local
```

`:anchor` is Flash's registration point and Blender's origin: rotation and scale
happen about it, and getting it wrong is why hand-placed parts swing rather than
turn.

`:pinv` is Blender's `parent_inverse`, captured at the moment of parenting so the
child does not jump when it acquires a parent. Small, and its absence is the kind
of thing that makes a parenting feature feel broken.

**The similarity fit already produces a decomposition.** `fitSimilarity` returns
`{s θ tx ty}`, which drops straight into `[:xform :scale]`, `[:xform :rot]` and
`[:xform :pos]` with no conversion. The analysis output and the animation model
meet without an adapter, which is a sign the decomposition is the right one.

## Time maps — exposure, lead and symbol timing are one thing

Every node may map the frame it is evaluated at:

```clojure
:time {:mode :inherit}                        ; the default, and almost always right
:time {:mode :map :expose 2 :offset -1 :rate 1.0 :loop? false}
```

Three features that look unrelated are this one mechanism:

- **exposure** is `⌊f/n⌋·n`,
- **mouth lead** is `f + k`,
- **a symbol instance's timing** is `(f - at)·rate + in`, with optional looping.

Composed along the nesting chain, outermost first. Two rules follow, and they are
different rules:

- **Exposure inherits strictly.** `docs/design.md` is emphatic that everything
  rides one grid, because a head cutting on odd frames against a mouth cutting on
  even ones reads as two performances. The model permits a per-node grid; the
  default must be `:inherit`, and setting it lower is a deliberate act the UI
  should make feel like one.
- **Offset is per-node by design.** Mouth lead applies to performance nodes and
  *not* to the plate, which is the whole point of it — so the offset genuinely
  belongs at the node, not the clip.

## Symbols

The Flash idea, kept:

```clojure
:library
{:sym/head        {:kind :poly :channels {...}}
 :sym/blink       {:kind :timeline :frames 3 :nodes [...]}}
```

A node with `:symbol :sym/blink` is an **instance**. Its own channels compose
*over* the symbol's, so one definition can be placed many times and tinted,
offset or retimed at each placement. A `:timeline` symbol has its own frame space,
which the instance's `:time` maps into — that is Flash's MovieClip, Lottie's
precomp and AE's pre-comp, and it is how a three-frame blink gets reused at
frames 40, 88 and 200 without copying it.

This is also where `docs/design.md`'s "closed vocabulary is right for the head"
lands: a plate library is a set of `:sym/head-*` definitions, and the strip
chooses which instance is placed on which frame. The take format's `plate=` field
becomes an instance reference.

## Evaluating a frame

```clojure
(defn eval-frame
  "Scene at clip frame f -> draw ops in z order. Pure."
  [scene f] ...)
```

1. Walk nodes in **topological order** by parent depth (cached; recompute only
   when parentage changes).
2. Skip nodes outside `:span`.
3. Apply the node's time map to get its own local frame `fn`.
4. **Sample** each channel at `fn`: a map lookup for framed, a sorted-index
   lookup for keyed, an array read for dense. Then apply `:over` layers.
5. Compose `world` from the parent's.
6. Transform geometry into raster space, writing into a **preallocated buffer**
   owned by the node.
7. Emit `{:kind :poly :pts buf :n 20 :color idx :stencil id}`.
8. Sort by resolved `z`.

The op list is the boundary with stage 7 in `docs/architecture.md`: the
rasteriser takes ops and knows nothing about nodes, channels or time.

### Making it fast in CLJS

Three things, and only these three matter:

- **Decomposed and persistent for storage; flat and mutable for evaluation.**
  Composed transforms are 6-element `Float64Array`s, not maps. Every renderer
  does this; the storage form and the evaluation form are allowed to differ.
- **A cursor per channel.** Playback is sequential, so "most recent key at or
  before `f`" is an advance of a saved index, O(1) amortised. Binary search only
  on a seek. This is the difference between a `rsubseq` allocation per channel per
  frame and none.
- **Preallocated point buffers per node.** Fixed topology means the size is known
  at freeze time, so a frame allocates nothing. At 30fps, per-frame allocation is
  the only thing that will make this stutter.

### What is in app-db, and what is not

| In app-db (tier 1) | In tier 2, behind a handle |
| --- | --- |
| nodes, parentage, z, spans, stencils | dense channel blocks |
| channel definitions, `:interp`, `:generated` | analysis artifacts |
| **framed** values, **keyed** keys, `:over` layers | preallocated eval buffers |
| library / symbol definitions | composed transform scratch |

The rule: **anything a human placed is in the document; anything a generator
produced is a handle.** Which is the same line `docs/architecture.md` draws for
sync and baking, arrived at again from the renderer's side.

## The current parts, in this model

Proof that it covers what exists, not just what is wanted:

| Now | Becomes |
| --- | --- |
| `mouth` outer ring, every frame | node `:mouth`, `[:geom :pts]` dense, `:generated {:by :roto/lips-outer}` |
| `mouth_in`, hidden below aperture | node `:mouth-in`, parent `:mouth`, `[:geom :pts]` dense + `[:vis]` dense |
| `teeth` from image content | node `:teeth`, stencil `:mouth-in`, `[:geom :pts]` dense, `:generated {:by :interior/teeth}` |
| lid rings | nodes `:lid-r/-l`, `[:geom :pts]` dense |
| lash line (`offsetRing`) | not data — a stage-6 parameter on the node, `{:grow px}` |
| iris disc | node `:iris-r`, `:kind :disc`, parent `:lid-r`, stencil `:sclera-r`, `[:xform :pos]` dense (quantised at freeze), radius framed |
| square pupil | node `:pupil-r`, `:kind :rect`, parent `:iris-r`, stencil `:iris-r` |
| brow ring + quantised raise | node `:brow-r`, `[:geom :pts]` dense (the traced ring with height removed), `[:xform :pos]` dense (the quantised raise). **The decomposition design.md insists on is two channels.** |
| head plate, kept frames | node `:head`, `:symbol` per instance, keys on `[:symbol]` at kept frames |
| painted background cel | node per layer, `[:geom :pts]` **framed**, `[:style :color]` framed |
| `mouth lead` | `:time {:offset k}` on performance nodes only |
| `exposure` | `:time {:expose n}` on the clip root, inherited |
| hand correction | an `:over` layer, `:offset` or `:replace` |

The brow row is the one worth looking at twice. `docs/design.md` argues at length
that the traced ring already contains the height, so the quantised raise must be
measured *out* and put *back* or the brow moves twice. In this model that is not
an argument to remember — it is two channels on one node, and getting it wrong
would mean writing the height into both.

## Format on disk and on the wire

Tier 1 is EDN/transit: the node tree, channel definitions, framed values, keys,
layers, library. Kilobytes, human-readable, diffable, and leaf-addressable for
sync.

Dense blocks are separate content-addressed binaries — `Int16Array` for raster
geometry, `Float32Array` for transforms — with a small header naming the channel
path, frame count and stride.

**Not Lottie internally**, despite the property shape being borrowed from it.
Lottie has no palette-indexed colour, its shapes are bezier with in/out tangents
where these are integer polygons, and its interpolation defaults are the opposite
of what is wanted. It is a good **export** target later, next to the `.take`
writer, and a bad internal format.

## Deferred

- **Per-key easing.** The structure allows it; nothing should use it until a
  parented transform on a painted cel asks for it.
- **More than two channel layers.** The `:over` vector is already a list; a real
  blend stack with weights is the NLA, and it is not needed to fix a bad frame.
- **Skew beyond the field.** `[:xform :skew]` is in the transform and in the
  composition order from the start, because adding a component to a decomposition
  later means migrating every stored transform.
- **Instance channel overrides on symbols.** Compose-over is specified; only
  colour and transform need it at first.
- **Constraints and drivers.** Blender's other half. A gaze that aims at a null
  object is the obvious first one, and it is a long way off.