arthur/docs/port-plan.md

# arthur — port plan and handoff

Self-contained. You should not need any prior conversation to execute this.

**Implementation status (2026-09-27):** steps 0–7 are in the CLJS frontend.
Step 6 reads extracted footage from the manifest, detects landmarks with local
MediaPipe assets at full source cadence, and runs the same freeze path as the
synthetic take. The scene time map can sample the frozen roto at a lower picture
fps without changing source analysis, duration or audio. Step 7 adds dense
eyelids, shared gaze, brows and pixel-derived teeth. Step 8 is next: authored
parameter controls and their scoped recomputation.

## What arthur is

A tool that turns live-action video into 2D animation that reads as
hand-authored: flat polygons, a tiny indexed palette, hard edges, 320×200, no
antialiasing, motion carried by silhouette. It tracks a face out of a clip,
reduces the lip contour to a handful of vertices, derives teeth from image
content, and renders flat indexed fills.

It currently works, as vanilla JS ES modules with no build step. `python3
serve.py`, open `127.0.0.1:8777`. **Synthetic take** exercises everything below
detection with no video needed.

This plan converts it to ClojureScript + re-frame, restructured around one
uniform animation data model, and adds a Django backend for persistence and
(later) collaboration.

## Status of the existing documents

| File | What it is | Authority |
| --- | --- | --- |
| `js/**` | the working tool, ~4,800 lines | **authoritative.** The comments encode bugs that actually happened. |
| `docs/animation-model.md` | the target data model: nodes, channels, symbols, time maps | build to this |
| `docs/architecture.md` | module layout, stages, sync and baking design | build to this; much of it is future scope |
| `docs/design.md`, `README.md` | prior synthesis by an earlier agent | useful, **not authoritative**. Revise freely. Do not treat its aesthetic claims as settled requirements. |

Where a document and the code disagree, the code wins, and the invariant list
below is lifted from the code for exactly that reason.

## Target repo layout

Both halves live here. Django at the root, because `manage.py` at the root is the
convention and keeps every `python manage.py` invocation working with no `cd`.

```
arthur/
  mise.toml               toolchain for both halves
  manage.py
  requirements.txt
  server/                 Django project: settings, urls, asgi, wsgi
  clips/                  Django app: models, views, consumers, routing, migrations
  frontend/               the CLJS app
    shadow-cljs.edn
    package.json
    src/arthur/**         namespace root stays arthur.* whatever the dir is called
    test/arthur/**
  static/arthur/js/       shadow-cljs output, collected by Django staticfiles
  docs/
  js/  index.html  serve.py  extract.sh      the old tool — see "the oracle"
```

`clips` is a naming call, not a constraint — it is the Django app holding
Project, Clip, Footage, Analysis, Leaf and Revision. Rename in one line if
something fits better.

Dev runs two processes: Django serves the page, `shadow-cljs watch app` rebuilds
into `static/arthur/js`. Set `:output-dir "../static/arthur/js"` in
`shadow-cljs.edn`.

## Toolchain

`mise install` from the repo root. `mise.toml` pins java 21+, node 20, clojure,
python 3.12, and creates `.venv`.

Verified to resolve cleanly: `reagent 1.2.0`, `re-frame 1.4.3`, current
shadow-cljs.

## Scope

**In:** analysis → keyframes → playback. The pure numeric core, the animation
data model, a player, the measurement stages, and freezing measurements into
channels.

**Out, and do not build it:** paint and cels; `suggest` (it only decides which
frames get a hand-drawn cel, so it has no job until drawing exists); the timeline
and sequences; symbols and the plate library; multiplayer; the override layer.
Each is designed for in `docs/architecture.md` and `docs/animation-model.md`.
Leave the `:over` field present and empty; leave `:symbol` out entirely.

## The data model

Full specification in `docs/animation-model.md`. The subset to build:

```clojure
;; The scene is a flat map of id -> node. Parent pointers, never nested maps.
{:id :mouth :kind :poly :parent :head :z "a3" :stencil nil :span [0 240]
 :time {:mode :inherit}          ; or {:mode :map :expose 2 :offset -1 :rate 1.0}
 :channels
 {[:xform :pos]    {:animated? false :value [0.0 0.0]}
  [:xform :rot]    {:animated? false :value 0.0}
  [:xform :scale]  {:animated? false :value [1.0 1.0]}
  [:xform :skew]   {:animated? false :value [0.0 0.0]}
  [:xform :anchor] {:animated? false :value [0.0 0.0]}
  [:geom :pts]     {:animated? true :interp :hold
                    :dense {:store "sha256:…" :offset 0 :stride 40 :frames 600}
                    :generated {:by :roto/lips-outer :analysis "sha256:…"
                                :params {:verts 8 :contour-avg 1}}
                    :over []}
  [:style :color]  {:animated? false :value :skin-dark}
  [:vis]           {:animated? false :value true}}}
```

Three channel shapes, one accessor `(value-at channel f)`:

- `{:animated? false :value v}` — static. A thing that simply exists.
- `{:animated? true :interp :hold :keys {0 v, 4 v}}` — sparse, authored, in the
  document. **Keys are a map by frame, never a vector.** Store a plain map
  (transit loses sortedness) and build the sorted index in the resolver.
- `{:animated? true :interp :hold :dense {...}}` — generated, one value per
  frame, in a typed array outside app-db.

`:generated` is provenance and **the renderer never reads it.** It is what the UI
uses to offer a parameter panel instead of raw keys. It lives on the *channel*,
not the node, because a node wants a rotoscoped `[:geom :pts]` and a
hand-animated `[:xform :pos]` at the same time.

`:skew`, `:span`, `:anchor` and `:over` stay in the shape even though nothing
drives them yet: each is a component of a decomposition or of a composition
order, and adding one later migrates every stored transform.

Transform composition, per node:

```
local = T(pos) · T(anchor) · R(rot) · K(skew) · S(scale) · T(-anchor)
world = world(parent) · local
```

## What the prototype knows that you would otherwise rediscover

**The JS is a prototype.** Its conclusions about what looks right are provisional
and you may revisit any of them; several contradict each other already. But a few
things in it are not taste — they are facts about MediaPipe, about the maths, or
about what an operation means — and those cost real time to rediscover.

### Mechanical. Getting these wrong produces wrong output, not a different look.

1. **MediaPipe's normalised space is anisotropic.** It divides x by image *width*
   and y by *height*, so equal numbers do not mean equal pixels. Multiply x by
   `aspect = W/H` before any fit, or a "similarity" fitted in that space is not
   one and head roll comes out subtly wrong. When mapping pixels for an underlay,
   **both** axes divide by `imgH`.
2. **MediaPipe's left/right naming is viewer-relative in some places and
   subject-relative in others.** Any left/right pairing read off a table is a coin
   flip, and a swap looks *almost* right — each eye still has an iris roughly
   where it belongs — so it survives inspection. Resolve it from geometry.
3. **Ring tables are ordered traversals**, and slot position is the vertex's
   identity. That is what makes temporal correspondence possible at all, whatever
   you decide the shapes should look like. `subsampleSlots` returns ring
   *positions*, not landmark ids.
4. **A wrongly-ordered ring self-intersects, and it is invisible at odd vertex
   budgets and obvious at even ones.** If you keep ordered rings, assert
   simplicity in a test; no amount of looking will catch it reliably.
5. **Scaling a ring to thicken it collapses when the ring is degenerate** — a shut
   eyelid scaled by 1.1 is still shut, so the lash line vanishes on exactly the
   frames where it is the whole drawing. A fixed radial offset does not. Maths,
   not taste.
6. **A fractional centre for a small integer-sized shape changes its size.**
   Round the origin, not the extents, or a 3px mark is 3px on one frame and 4px on
   the next.
7. **Order of operations on time:** flooring onto a grid and shifting against the
   clock do not commute. Shift first and the floor discards it on most frames.

### Choices the prototype made. Revisit freely; here is what each was for.

| Choice | Its stated reason | How you would learn it was wrong |
| --- | --- | --- |
| similarity (4 DOF), not affine | extra DOF absorbs out-of-plane head rotation as shear and smears it into the mouth | the residual readout stops responding to head turn |
| reference is the Procrustes mean over the shot, not frame 0 | no single frame's idiosyncrasies get baked into every other | one frame's detection error biases the whole take |
| smooth the transform, not the contour | sparse keys at velocity minima rejected detector noise for free | it was already broken by a "bounded exception" once keys went dense, so it was never a law |
| gaze measured against the eye's corner midpoint | measured against the lid, every blink drags the origin down and fakes a glance at the floor | gaze correlates with blinks |
| one gaze shared by both eyes | at this size the per-eye difference is noise, and independent noise reads as wall-eyed | a wink or a real vergence is lost |
| hold, never interpolate | a tweened mouth reads as puppet software | motion looks stepped rather than snappy |
| palette indices, never sampled RGB | sampling colour produces a pixel-art filter irrecoverably | — |

These are where to look first if the output is wrong. They are also where to look
first if you want to change the look.

## Conventions

- `domain/*` may not require `flow/*`; neither may require `re-frame`.
- Every flow function is `(f params inputs) -> output`. No state, no db, no atoms.
- Nothing below `subs/` calls `subscribe`.
- Every analysis function that reads pixels takes a `debug?` flag and returns its
  intermediate masks alongside its result, the way
  `interior.js/extractTeeth(..., wantDebug)` already does.
- Port the invariant comments across verbatim. They are the most valuable text in
  the repo.

## The oracle

**Keep `js/`, `index.html` and `serve.py` in the tree through step 5.** They cost
nothing, `serve.py` still runs the old tool, and they are the numeric oracle:
run both implementations on the same synthetic track and diff.
`fit-similarity` and `procrustes-mean` should agree to **1e-9**; a larger gap is a
port bug, not float noise.

**Parity proves the port is faithful, not that the answer is right.** The JS is a
prototype, so keep the two kinds of test apart: a *parity* test pins behaviour
while you move it, and is deleted once the move is done; a *correctness* test
asserts something you have decided you want, and stays. Conflating them bakes the
prototype's mistakes into the rewrite and makes them permanent. Delete them in one commit once the CLJS player renders
the synthetic take correctly.

**Do not port the debug views** (`drawPanes`, `drawInteriorDebug`,
`drawEyeOverlay`, `drawGazeDebug` in `js/app.js`). The knowledge in them is not
the canvas calls — it is *which things you must see to tune teeth*: the source
crop, the in-region mask, the surviving mask, and the local contour. That contract
already exists as `extractTeeth(..., wantDebug)` returning
`debugCanvas(src, inReg, mask, pw, ph, local)`. **Port the payload, skip the
drawing.** Redrawing it is ten lines whenever it is wanted.

## Steps

Each step ends somewhere runnable. Do not proceed past a step whose "done" does
not hold.

### 0 — scaffold and the oracle
`mise install`. Create `frontend/` with shadow-cljs, reagent, re-frame. Port
`synth.js` (the synthetic landmark generator, including its `swapIris` flag) and
the numeric assertions from `selftest.js` to `cljs.test`.

**Done:** the suite runs and fails informatively.

### 1 — the pure bottom
Port verbatim: `landmarks.js` → `domain/landmarks`, `mathutil.js` → `domain/geom`,
ring helpers → `domain/ring`, `raster.js` → `domain/raster`, the palette →
`domain/palette`.

**Done:** tests pass, including ring simplicity and the swapped-iris vote. Numeric
agreement with the JS to 1e-9. Nothing renders.

### 2 — the data model, with no analysis in it
`domain/channel` (`value-at` across all three shapes, plus a per-channel cursor),
`domain/node` (transform composition), `domain/scene` (topological order by parent
depth, `eval-frame` → draw ops in z order).

Hand-write a scene in EDN — a rectangle parented to a group whose
`[:xform :pos]` is keyed on four frames — and render it through `domain/raster`
into a canvas.

This is deliberately before any analysis. **The data model has never been
validated; find out here**, with fifty lines to throw away, rather than after
porting nine hundred lines of measurement into a shape that does not work.

**Done:** something moves on screen.

### 3 — the player
`clock` (audio-clocked: `frame = ⌊currentTime · fps⌋`, so a slow loop drops frames
instead of drifting; ½× and ¼× come free from `playbackRate`), the rAF loop, a
`::resolver` sub, and transport UI.

The loop reads and blits and **dispatches nothing**. The sub yields a resolver
closure; the loop applies it at the playhead. The playhead itself lives in app-db
like everything else — with layer-2 extractors and layer-3 computations, a
playhead tick does not invalidate the expensive stages.

**Done:** the hand-written scene plays at 30fps against audio, scrubs, and runs at
½× and ¼×.

### 4 — measure: anchor and mouth
Port `stabilize` and the lip rings out of `pipeline.js`. Split **condition**
(`smoothTransforms`, `smoothContours`) into its own stage so the two smoothing
knobs do not re-run measurement.

**Done:** measured numbers match the JS on the synthetic track. Note that
parity here is on `stabilize`'s output, not on `toRasterRing`'s — the framing
step is being deleted, not ported.

### 5 — freeze
The new module, and the heart of this work: measurements → channels. A dense
`[:geom :pts]` block per node — `Int16Array[frames × verts × 2]` in the node's
own local space, with the block's fixed-point scale in its header — plus
`:generated`. Fixed topology is what makes this a rectangular array with no
per-frame header.

**Do not port `makeXform`.** The prototype bakes the framing into the stored
numbers: it centres on the face oval's bbox and zooms until the face is 80% of
the raster height, so every vertex carries a cropping decision made once from one
frame's landmarks. Dropping it is a deletion. Placement becomes `[:xform :*]` on
an authored `:face` node, the stage clips whatever hangs off, and project
dimensions stop being tied to the footage. See "What space geometry is in" in
`docs/animation-model.md`.

The anchor transform freezes onto `:head`, one level under `:face`, and the
normalise on/off/per-plate toggle is which of the three channel shapes that node
carries. Always measure and always store factored, whatever the toggle says:
smoothing and velocity-minimum key selection both require the split to exist in
storage.

**Done:** the synthetic take plays back as a moving mouth. Full vertical slice.

### 6 — detect
MediaPipe interop behind one namespace; real frames, real audio, real fps from
the manifest. **Vendor the wasm** rather than fetching from jsdelivr — it is
currently the only thing in the tool that silently requires a network.

Decode every source frame for analysis. A lower output picture fps is a time map
over frozen channels, not a reduced detection track. Selecting source frames to
trace into cels is independent again and remains outside this port's paint scope.

**Done:** real footage plays back as a rotoscoped mouth.

### 7 — the rest of measure
Eyes (openness, gaze, iris pairing vote, blink resolution with its `hold`), brows
(raise and tilt at both ends, both correspondence votes), interior (otsu,
morphology, components, radial contour). Each keeps its `debug?` payload.

Two things fall out of the model instead of being written: the brow's
measure-the-height-out-and-put-it-back is `[:geom :pts]` plus `[:xform :pos]`, two
channels on one node; and the iris is a `:disc` node parented to the lid ring and
stencilled by the sclera.

**Done:** the same face parts are measured and rendered through the CLJS scene,
minus paint. The fixed pixel thresholds remain provisional; step 8 exposes their
parameters for tuning without changing the source track or picture timing.

### 8 — knobs
The parameter UI, as leaf-addressed params in app-db
(`clip/:cid/params/:subject/:area` — see below), so the sync layer added later has
nothing to retrofit.

### 9 — backend
Django project, the `clips` app, models for Project/Clip/Footage/Analysis/Leaf,
and project load/save. Round-tripping a project through the server is the proof
the model serialises.

**Output is not in this plan.** The `.take` writer in `js/take.js` was for
driving an Animator Pro render script and it is not where this is going: the
target is encoding video in the browser, and that is a separate piece of design
nobody should pre-empt by porting the old one.

## Two things to not foreclose

The feature controls will later be rethought to handle more than one face,
periodic occlusion, stable identity across frames, and feature groups with their
own parameters. That design can wait; two decisions here are free now and
annoying to reverse:

- **Presence is not visibility.** An occluded subject has *no value* on a frame,
  which is different from a part being hidden. Give every dense block a
  `Uint8Array` state mask per frame and let it mean *absent* as well as hidden.
- **Params carry a subject segment.** `clip/:cid/params/:subject/:area`, with one
  subject today. Adding a path segment later touches every read and write.

The identity tracker, when it comes, should use the same pattern the iris and brow
correspondences already use: vote across every frame rather than trusting one.