arthur/docs/port-plan.md

# arthur — port plan and handoff

Self-contained. You should not need any prior conversation to execute this.

**Implementation status (2026-09-29):** steps 0–9 are in. Step 6 reads extracted
footage, detects landmarks with local MediaPipe assets at full source cadence, and
runs the same freeze path as the synthetic take. The scene time map can sample the
frozen roto at a lower picture fps without changing source analysis, duration or
audio. Step 7 adds dense eyelids, shared gaze, brows and pixel-derived teeth.
Step 8's data model has stable feature identity, feature-level presence, explicit
eye pairs and shared parameter definitions; a manifest can supply known feature
absence intervals through measurement and freeze. Step 9 adds the Django backend,
the three-tier split, content-addressed tier 2 with the detector version inside
every key, leaf addressing for tier 1, and project load/save that round-trips.

Step 8 now has parameter controls and scoped regeneration from retained source.
Multi-face representation is complete: each tracked subject has a drawing
timeline, placed by an ordinary symbol instance. See
[multi-face representation](multi-face-representation.md) for the implemented
model, verification and compatibility limits.

**Next, in order:** commit the verified checkpoint; exercise real two-person
footage, including crossings and disappearances; build performance-pose
Suggest/Keep/Drop and instance-scoped picture rates; then add plate-drawing
selection and independent tracing references. The
[timing handoff](timing-handoff.md) records current code and implementation order.
Reopen the representation only for a concrete requirement it cannot express.

**Still open:** real-footage identity validation, automatic per-feature detection,
the timing and tracing work above, and time-varying parameter settings. Older flat
captures need reanalysis for the new regeneration path; no migration is included.
The step descriptions below retain the original port scope; this status and the
linked handoffs describe subsequent work.

## What arthur is

A tool that turns live-action video into 2D animation that reads as
hand-authored: flat polygons, a tiny indexed palette, hard edges, 320×200, no
antialiasing, motion carried by silhouette. It tracks a face out of a clip,
reduces the lip contour to a handful of vertices, derives teeth from image
content, and renders flat indexed fills.

It currently works, as vanilla JS ES modules with no build step. `python3
serve.py`, open `127.0.0.1:8777`. **Synthetic take** exercises everything below
detection with no video needed.

This plan converts it to ClojureScript + re-frame, restructured around one
uniform animation data model, and adds a Django backend for persistence and
(later) collaboration.

## Status of the existing documents

| File | What it is | Authority |
| --- | --- | --- |
| `js/**` | the working tool, ~4,800 lines | **authoritative.** The comments encode bugs that actually happened. |
| `docs/animation-model.md` | the target data model: nodes, channels, symbols, time maps | build to this |
| `docs/architecture.md` | module layout, stages, sync and baking design | build to this; much of it is future scope |
| `docs/design.md`, `README.md` | prior synthesis by an earlier agent | useful, **not authoritative**. Revise freely. Do not treat its aesthetic claims as settled requirements. |

Where a document and the code disagree, the code wins, and the invariant list
below is lifted from the code for exactly that reason.

## Target repo layout

Both halves live here. Django at the root, because `manage.py` at the root is the
convention and keeps every `python manage.py` invocation working with no `cd`.

```
arthur/
  mise.toml               toolchain for both halves
  manage.py
  requirements.txt
  server/                 Django project: settings, urls, asgi, wsgi
  clips/                  Django app: models, views, consumers, routing, migrations
  frontend/               the CLJS app
    shadow-cljs.edn
    package.json
    src/arthur/**         namespace root stays arthur.* whatever the dir is called
    test/arthur/**
  static/arthur/js/       shadow-cljs output, collected by Django staticfiles
  static/arthur/audio.wav the synthetic take's clock. NOT extract.sh's output —
                          that is tier 3 and lives in the blob store
  var/blobs/              the content-addressed blob store: tiers 2 and 3. Gitignored
  docs/
  js/  index.html  serve.py  extract.sh      the old tool — see "the oracle"
```

The namespaces step 9 added, since the list under **Namespaces** in
`docs/architecture.md` predates them:

```
domain/sha256.cljs    SHA-256, synchronous and pure, byte-compatible with hashlib
domain/canon.cljs     the one canonical text for a descriptor, so hashing it means
                      something
domain/leaf.cljs      leaf addressing: the document as path -> value
domain/wire.cljs      transit for tier 1, base64 for tier 2
domain/project.cljs   clip <-> the document and blocks that travel
flow/address.cljs     tier-2 keys, and the invalidation table they are built from
fx/http.cljs          the only namespace that talks to the server
events/project.cljs   save and open
```

`clips` is a naming call, not a constraint — it is the Django app holding
Project, Clip, Footage, Analysis, Leaf and Revision. Rename in one line if
something fits better.

Dev runs two processes and they do not talk to each other: Django serves the page,
`shadow-cljs watch app` rebuilds into `static/arthur/js`, which is already
`:output-dir` in `shadow-cljs.edn`. `:dev-http` is gone.

```sh
mise exec -- python manage.py runserver 8778        # from the repo root
cd frontend && mise exec -- npx shadow-cljs watch app
```

## Toolchain

`mise install` from the repo root. `mise.toml` pins java 21+, node 20, clojure,
python 3.12, and creates `.venv`.

Verified to resolve cleanly: `reagent 1.2.0`, `re-frame 1.4.3`, current
shadow-cljs.

## Scope

**In:** analysis → keyframes → playback. The pure numeric core, the animation
data model, a player, the measurement stages, and freezing measurements into
channels.

**Out, and do not build it:** paint and cels; `suggest` (it only decides which
frames get a hand-drawn cel, so it has no job until drawing exists); the timeline
and sequences; symbols and the plate library; multiplayer; the override layer.
Each is designed for in `docs/architecture.md` and `docs/animation-model.md`.
Leave the `:over` field present and empty; leave `:symbol` out entirely.

## The data model

Full specification in `docs/animation-model.md`. The subset to build:

```clojure
;; The scene is a flat map of id -> node. Parent pointers, never nested maps.
{:id :mouth :kind :poly :parent :head :z "a3" :stencil nil :span [0 240]
 :time {:mode :inherit}          ; or {:mode :map :expose 2 :offset -1 :rate 1.0}
 :channels
 {[:xform :pos]    {:animated? false :value [0.0 0.0]}
  [:xform :rot]    {:animated? false :value 0.0}
  [:xform :scale]  {:animated? false :value [1.0 1.0]}
  [:xform :skew]   {:animated? false :value [0.0 0.0]}
  [:geom :pts]     {:animated? true :interp :hold
                    :dense {:store "sha256:…" :offset 0 :stride 40 :frames 600}
                    :generated {:by :roto/lips-outer :analysis "sha256:…"
                                :params {:verts 8 :contour-avg 1}}
                    :over []}
  [:style :color]  {:animated? false :value :skin-dark}
  [:vis]           {:animated? false :value true}}}
```

Three channel shapes, one accessor `(value-at channel f)`:

- `{:animated? false :value v}` — static. A thing that simply exists.
- `{:animated? true :interp :hold :keys {0 v, 4 v}}` — sparse, authored, in the
  document. **Keys are a map by frame, never a vector.** Store a plain map
  (transit loses sortedness) and build the sorted index in the resolver.
- `{:animated? true :interp :hold :dense {...}}` — generated, one value per
  frame, in a typed array outside app-db.

`:generated` is provenance and **the renderer never reads it.** It is what the UI
uses to offer a parameter panel instead of raw keys. It lives on the *channel*,
not the node, because a node wants a rotoscoped `[:geom :pts]` and a
hand-animated `[:xform :pos]` at the same time.

`:skew`, `:span` and `:over` stay in the shape even though nothing drives them
yet: each is a component of a decomposition or of a composition order, and adding
one later migrates every stored transform.

`:anchor` was in this list and has since been **deleted**, which is the one place
the reasoning above came out wrong. It is not a component of the decomposition:
`T(a)·M·T(-a)` is `M` conjugated by a translation, and a parent already is a
translated frame, so an anchor is a peg written inline — one that cannot be
selected, keyed, shared, or put above a measured channel. Rotation and scale
happen about the node's own origin; a pivot nobody chose is derived per drag by
`domain/gesture` and a pivot somebody chose is a peg. See
docs/animation-model.md, "There is no `:anchor`, because an anchor is a peg".

Transform composition, per node:

```
local = T(pos) · R(rot) · K(skew) · S(scale)
world = world(parent) · pinv · local
```

## What the prototype knows that you would otherwise rediscover

**The JS is a prototype.** Its conclusions about what looks right are provisional
and you may revisit any of them; several contradict each other already. But a few
things in it are not taste — they are facts about MediaPipe, about the maths, or
about what an operation means — and those cost real time to rediscover.

### Mechanical. Getting these wrong produces wrong output, not a different look.

1. **MediaPipe's normalised space is anisotropic.** It divides x by image *width*
   and y by *height*, so equal numbers do not mean equal pixels. Multiply x by
   `aspect = W/H` before any fit, or a "similarity" fitted in that space is not
   one and head roll comes out subtly wrong. When mapping pixels for an underlay,
   **both** axes divide by `imgH`.
2. **MediaPipe's left/right naming is viewer-relative in some places and
   subject-relative in others.** Any left/right pairing read off a table is a coin
   flip, and a swap looks *almost* right — each eye still has an iris roughly
   where it belongs — so it survives inspection. Resolve it from geometry.
3. **Ring tables are ordered traversals**, and slot position is the vertex's
   identity. That is what makes temporal correspondence possible at all, whatever
   you decide the shapes should look like. `subsampleSlots` returns ring
   *positions*, not landmark ids.
4. **A wrongly-ordered ring self-intersects, and it is invisible at odd vertex
   budgets and obvious at even ones.** If you keep ordered rings, assert
   simplicity in a test; no amount of looking will catch it reliably.
5. **Scaling a ring to thicken it collapses when the ring is degenerate** — a shut
   eyelid scaled by 1.1 is still shut, so the lash line vanishes on exactly the
   frames where it is the whole drawing. A fixed radial offset does not. Maths,
   not taste.
6. **A fractional centre for a small integer-sized shape changes its size.**
   Round the origin, not the extents, or a 3px mark is 3px on one frame and 4px on
   the next.
7. **Order of operations on time:** flooring onto a grid and shifting against the
   clock do not commute. Shift first and the floor discards it on most frames.

### Choices the prototype made. Revisit freely; here is what each was for.

| Choice | Its stated reason | How you would learn it was wrong |
| --- | --- | --- |
| similarity (4 DOF), not affine | extra DOF absorbs out-of-plane head rotation as shear and smears it into the mouth | the residual readout stops responding to head turn |
| reference is the Procrustes mean over the shot, not frame 0 | no single frame's idiosyncrasies get baked into every other | one frame's detection error biases the whole take |
| smooth the transform, not the contour | sparse keys at velocity minima rejected detector noise for free | it was already broken by a "bounded exception" once keys went dense, so it was never a law |
| gaze measured against the eye's corner midpoint | measured against the lid, every blink drags the origin down and fakes a glance at the floor | gaze correlates with blinks |
| one gaze shared by both eyes | at this size the per-eye difference is noise, and independent noise reads as wall-eyed | a wink or a real vergence is lost |
| hold, never interpolate | a tweened mouth reads as puppet software | motion looks stepped rather than snappy |
| palette indices, never sampled RGB | sampling colour produces a pixel-art filter irrecoverably | — |

These are where to look first if the output is wrong. They are also where to look
first if you want to change the look.

## Conventions

- `domain/*` may not require `flow/*`; neither may require `re-frame`.
- Every flow function is `(f params inputs) -> output`. No state, no db, no atoms.
- Nothing below `subs/` calls `subscribe`.
- Every analysis function that reads pixels takes a `debug?` flag and returns its
  intermediate masks alongside its result, the way
  `interior.js/extractTeeth(..., wantDebug)` already does.
- Port the invariant comments across verbatim. They are the most valuable text in
  the repo.

## The oracle

**Keep `js/`, `index.html` and `serve.py` in the tree through step 5.** They cost
nothing, `serve.py` still runs the old tool, and they are the numeric oracle:
run both implementations on the same synthetic track and diff.
`fit-similarity` and `procrustes-mean` should agree to **1e-9**; a larger gap is a
port bug, not float noise.

**Parity proves the port is faithful, not that the answer is right.** The JS is a
prototype, so keep the two kinds of test apart: a *parity* test pins behaviour
while you move it, and is deleted once the move is done; a *correctness* test
asserts something you have decided you want, and stays. Conflating them bakes the
prototype's mistakes into the rewrite and makes them permanent. Delete them in one commit once the CLJS player renders
the synthetic take correctly.

**Do not port the debug views** (`drawPanes`, `drawInteriorDebug`,
`drawEyeOverlay`, `drawGazeDebug` in `js/app.js`). The knowledge in them is not
the canvas calls — it is *which things you must see to tune teeth*: the source
crop, the in-region mask, the surviving mask, and the local contour. That contract
already exists as `extractTeeth(..., wantDebug)` returning
`debugCanvas(src, inReg, mask, pw, ph, local)`. **Port the payload, skip the
drawing.** Redrawing it is ten lines whenever it is wanted.

## Steps

Each step ends somewhere runnable. Do not proceed past a step whose "done" does
not hold.

### 0 — scaffold and the oracle
`mise install`. Create `frontend/` with shadow-cljs, reagent, re-frame. Port
`synth.js` (the synthetic landmark generator, including its `swapIris` flag) and
the numeric assertions from `selftest.js` to `cljs.test`.

**Done:** the suite runs and fails informatively.

### 1 — the pure bottom
Port verbatim: `landmarks.js` → `domain/landmarks`, `mathutil.js` → `domain/geom`,
ring helpers → `domain/ring`, `raster.js` → `domain/raster`, the palette →
`domain/palette`.

**Done:** tests pass, including ring simplicity and the swapped-iris vote. Numeric
agreement with the JS to 1e-9. Nothing renders.

### 2 — the data model, with no analysis in it
`domain/channel` (`value-at` across all three shapes, plus a per-channel cursor),
`domain/node` (transform composition), `domain/scene` (topological order by parent
depth, `eval-frame` → draw ops in z order).

Hand-write a scene in EDN — a rectangle parented to a group whose
`[:xform :pos]` is keyed on four frames — and render it through `domain/raster`
into a canvas.

This is deliberately before any analysis. **The data model has never been
validated; find out here**, with fifty lines to throw away, rather than after
porting nine hundred lines of measurement into a shape that does not work.

**Done:** something moves on screen.

### 3 — the player
`clock` (audio-clocked: `frame = ⌊currentTime · fps⌋`, so a slow loop drops frames
instead of drifting; ½× and ¼× come free from `playbackRate`), the rAF loop, a
`::resolver` sub, and transport UI.

The loop reads and blits and **dispatches nothing**. The sub yields a resolver
closure; the loop applies it at the playhead. The playhead itself lives in app-db
like everything else — with layer-2 extractors and layer-3 computations, a
playhead tick does not invalidate the expensive stages.

**Done:** the hand-written scene plays at 30fps against audio, scrubs, and runs at
½× and ¼×.

### 4 — measure: anchor and mouth
Port `stabilize` and the lip rings out of `pipeline.js`. Split **condition**
(`smoothTransforms`, `smoothContours`) into its own stage so the two smoothing
knobs do not re-run measurement.

**Done:** measured numbers match the JS on the synthetic track. Note that
parity here is on `stabilize`'s output, not on `toRasterRing`'s — the framing
step is being deleted, not ported.

### 5 — freeze
The new module, and the heart of this work: measurements → channels. A dense
`[:geom :pts]` block per node — `Int16Array[frames × verts × 2]` in the node's
own local space, with the block's fixed-point scale in its header — plus
`:generated`. Fixed topology is what makes this a rectangular array with no
per-frame header.

**Do not port `makeXform`.** The prototype bakes the framing into the stored
numbers: it centres on the face oval's bbox and zooms until the face is 80% of
the raster height, so every vertex carries a cropping decision made once from one
frame's landmarks. Dropping it is a deletion. Placement becomes `[:xform :*]` on
an authored `:place` node inside the face, the stage clips whatever hangs off,
and project
dimensions stop being tied to the footage. See "What space geometry is in" in
`docs/animation-model.md`.

The anchor transform freezes onto `:head`, one level under `:place`, and the
normalise on/off/per-plate toggle is which of the three channel shapes that node
carries. Always measure and always store factored, whatever the toggle says:
smoothing and velocity-minimum key selection both require the split to exist in
storage.

**Done:** the synthetic take plays back as a moving mouth. Full vertical slice.

### 6 — detect
MediaPipe interop behind one namespace; real frames, real audio, real fps from
the manifest. **Vendor the wasm** rather than fetching from jsdelivr — it is
currently the only thing in the tool that silently requires a network.

Decode every source frame for analysis. A lower output picture fps is a time map
over frozen channels, not a reduced detection track. Selecting source frames to
trace into cels is independent again and remains outside this port's paint scope.

**Done:** real footage plays back as a rotoscoped mouth.

### 7 — the rest of measure
Eyes (openness, gaze, iris pairing vote, blink resolution with its `hold`), brows
(raise and tilt at both ends, both correspondence votes), interior (otsu,
morphology, components, radial contour). Each keeps its `debug?` payload.

Two things fall out of the model instead of being written: the brow's
measure-the-height-out-and-put-it-back is `[:geom :pts]` plus `[:xform :pos]`, two
channels on one node; and the iris is a `:disc` node parented to the lid ring and
stencilled by the sclera.

**Done:** the same face parts are measured and rendered through the CLJS scene,
minus paint. The fixed pixel thresholds remain provisional; step 8 exposes their
parameters for tuning without changing the source track or picture timing.

### 8 — knobs — DONE for static settings and scoped regeneration
Build the parameter model before its UI. Define each parameter once with its
default, validation, applicable area and regeneration dependencies. Store values
by stable subject and feature ID. Represent an eye pair as one group with one or
two eye member IDs from the same subject; a profile view with one identified eye
needs no invented partner. Each eye may override a pair value. Removing an eye
from a pair materialises its effective values so playback does not change. Keep the
existing frozen channels as renderer input; settings and provenance do not enter
the render path.

Carry feature-level presence through freeze and dense channel state. The same
feature ID covers every observed run across occlusion; a missing measurement
has no channel value on that frame. Full-face detection is the fallback mask
until there is a feature-level detector or authored presence data. A shared gaze
measurement may still feed two independently identified eyes. A small manifest
annotation can supply feature absence intervals now: the loader expands them
before measurement, so invalid eye landmarks are ignored and gaze uses the
visible eye. This is an input format, not a control UI or an automatic detector.

Use leaf-addressable settings under the clip, subject, feature and optional
group. Retain source measurements so a setting change can regenerate affected
channels without re-detecting footage. Static parameter controls and scoped
regeneration are implemented for takes and composed stages. Time-varying
parameter values remain deferred.

### 9 — backend — DONE
Django project, the `clips` app, models for
Project/Clip/Footage/FootageFrame/Analysis/Block/Leaf/Revision/Blob, and project
load/save. Round-tripping a project through the server is the proof the model
serialises.

Django was the easy half; the tier split was the work. What it came to:

**Tier 2 keys are content addresses over every input**, and the detector version
is in every one of them, through the analysis id that each block descriptor names.
`flow/address`'s `block-knobs` is the invalidation table, and it is not trusted:
`address-test` re-freezes the take once per knob and asserts the biconditional —
a block's bytes changed if and only if its key changed. That test found two
things reading the code would not have. `brow-pos` does not depend on
`contour-avg`, because the brow RING is smoothed and the raise is not. And the
first version of the test was itself wrong: a 3% perturbation of `gaze-gain`
moves every sample inside the grid cell `quantize-snap` had already rounded it
into, so the bytes came out identical and the knob looked like an input the block
did not have.

**The server verifies what it is handed.** It recomputes every key from the
descriptor stored beside it and refuses a mismatch, refuses an analysis that does
not declare a detector version, and refuses a document naming blocks it does not
hold. It hashes the descriptor TEXT rather than re-rendering it from parsed
values, because JS prints an integral double as `1` and Python as `1.0` — a
scheme where both sides re-render breaks on the first parameter whose value
happens to be whole.

**Tier 3 is served by hash.** `extract.sh` still decodes; `manage.py
ingest_bundle` hashes the result into the blob store, by hard link. The manifest
the client receives now carries a URL per frame, so the frame layout stopped being
a shared secret between a shell script and a ClojureScript namespace. The
cache-busting `?v=` on every frame URL went with it: a blob's name is the hash of
its bytes, so a stale copy is not a thing that can happen.

**Leaf addressing exists**, with conditional writes and a monotonic project
version, so the sync design has nothing to retrofit. The socket, presence and
broadcasts are still out of scope.

Two loose ends from step 8 closed on the way. `pack` no longer takes a
`(track, frame)` predicate whose call sites each derived a feature from an index —
every track names the feature it follows, which deleted five hand-maintained
mappings and handed `flow/address` the same list for its observation digest. And
the `presence-check` binding in a `let` nobody read is now an ordinary `doseq`.

**Output is not in this plan.** The `.take` writer in `js/take.js` was for
driving an Animator Pro render script and it is not where this is going: the
target is encoding video in the browser, and that is a separate piece of design
nobody should pre-empt by porting the old one.

## Two things to not foreclose

Feature controls now handle more than one face; editing presence remains future work.
The underlying identity, occlusion and group association model begins in step 8:

- **Presence is not visibility.** An occluded feature has *no value* on a frame,
  which is different from a part being hidden. Dense blocks carry a per-track,
  per-frame absence mask; `[:vis]` remains the sole hiding mechanism.
- **Params carry stable identity.** A subject and its features keep their IDs
  across observation gaps. A run of visible frames is not a new identity.

The current identity tracker uses nearest-centroid assignment. Validate it on
real crossings and disappearances before choosing a more elaborate policy; the
iris and brow correspondence code offers whole-take voting as one option.