# arthur — port plan and handoff Self-contained. You should not need any prior conversation to execute this. **Implementation status (2026-09-27):** steps 0–6 are in the CLJS frontend. Step 6 reads extracted footage from the manifest, detects landmarks with local MediaPipe assets, and runs the same freeze path as the synthetic take. Step 7 is next: eyes, brows and the pixel-derived mouth interior. ## What arthur is A tool that turns live-action video into 2D animation that reads as hand-authored: flat polygons, a tiny indexed palette, hard edges, 320×200, no antialiasing, motion carried by silhouette. It tracks a face out of a clip, reduces the lip contour to a handful of vertices, derives teeth from image content, and renders flat indexed fills. It currently works, as vanilla JS ES modules with no build step. `python3 serve.py`, open `127.0.0.1:8777`. **Synthetic take** exercises everything below detection with no video needed. This plan converts it to ClojureScript + re-frame, restructured around one uniform animation data model, and adds a Django backend for persistence and (later) collaboration. ## Status of the existing documents | File | What it is | Authority | | --- | --- | --- | | `js/**` | the working tool, ~4,800 lines | **authoritative.** The comments encode bugs that actually happened. | | `docs/animation-model.md` | the target data model: nodes, channels, symbols, time maps | build to this | | `docs/architecture.md` | module layout, stages, sync and baking design | build to this; much of it is future scope | | `docs/design.md`, `README.md` | prior synthesis by an earlier agent | useful, **not authoritative**. Revise freely. Do not treat its aesthetic claims as settled requirements. | Where a document and the code disagree, the code wins, and the invariant list below is lifted from the code for exactly that reason. ## Target repo layout Both halves live here. Django at the root, because `manage.py` at the root is the convention and keeps every `python manage.py` invocation working with no `cd`. ``` arthur/ mise.toml toolchain for both halves manage.py requirements.txt server/ Django project: settings, urls, asgi, wsgi clips/ Django app: models, views, consumers, routing, migrations frontend/ the CLJS app shadow-cljs.edn package.json src/arthur/** namespace root stays arthur.* whatever the dir is called test/arthur/** static/arthur/js/ shadow-cljs output, collected by Django staticfiles docs/ js/ index.html serve.py extract.sh the old tool — see "the oracle" ``` `clips` is a naming call, not a constraint — it is the Django app holding Project, Clip, Footage, Analysis, Leaf and Revision. Rename in one line if something fits better. Dev runs two processes: Django serves the page, `shadow-cljs watch app` rebuilds into `static/arthur/js`. Set `:output-dir "../static/arthur/js"` in `shadow-cljs.edn`. ## Toolchain `mise install` from the repo root. `mise.toml` pins java 21+, node 20, clojure, python 3.12, and creates `.venv`. Verified to resolve cleanly: `reagent 1.2.0`, `re-frame 1.4.3`, current shadow-cljs. ## Scope **In:** analysis → keyframes → playback. The pure numeric core, the animation data model, a player, the measurement stages, and freezing measurements into channels. **Out, and do not build it:** paint and cels; `suggest` (it only decides which frames get a hand-drawn cel, so it has no job until drawing exists); the timeline and sequences; symbols and the plate library; multiplayer; the override layer. Each is designed for in `docs/architecture.md` and `docs/animation-model.md`. Leave the `:over` field present and empty; leave `:symbol` out entirely. ## The data model Full specification in `docs/animation-model.md`. The subset to build: ```clojure ;; The scene is a flat map of id -> node. Parent pointers, never nested maps. {:id :mouth :kind :poly :parent :head :z "a3" :stencil nil :span [0 240] :time {:mode :inherit} ; or {:mode :map :expose 2 :offset -1 :rate 1.0} :channels {[:xform :pos] {:animated? false :value [0.0 0.0]} [:xform :rot] {:animated? false :value 0.0} [:xform :scale] {:animated? false :value [1.0 1.0]} [:xform :skew] {:animated? false :value [0.0 0.0]} [:xform :anchor] {:animated? false :value [0.0 0.0]} [:geom :pts] {:animated? true :interp :hold :dense {:store "sha256:…" :offset 0 :stride 40 :frames 600} :generated {:by :roto/lips-outer :analysis "sha256:…" :params {:verts 8 :contour-avg 1}} :over []} [:style :color] {:animated? false :value :skin-dark} [:vis] {:animated? false :value true}}} ``` Three channel shapes, one accessor `(value-at channel f)`: - `{:animated? false :value v}` — static. A thing that simply exists. - `{:animated? true :interp :hold :keys {0 v, 4 v}}` — sparse, authored, in the document. **Keys are a map by frame, never a vector.** Store a plain map (transit loses sortedness) and build the sorted index in the resolver. - `{:animated? true :interp :hold :dense {...}}` — generated, one value per frame, in a typed array outside app-db. `:generated` is provenance and **the renderer never reads it.** It is what the UI uses to offer a parameter panel instead of raw keys. It lives on the *channel*, not the node, because a node wants a rotoscoped `[:geom :pts]` and a hand-animated `[:xform :pos]` at the same time. `:skew`, `:span`, `:anchor` and `:over` stay in the shape even though nothing drives them yet: each is a component of a decomposition or of a composition order, and adding one later migrates every stored transform. Transform composition, per node: ``` local = T(pos) · T(anchor) · R(rot) · K(skew) · S(scale) · T(-anchor) world = world(parent) · local ``` ## What the prototype knows that you would otherwise rediscover **The JS is a prototype.** Its conclusions about what looks right are provisional and you may revisit any of them; several contradict each other already. But a few things in it are not taste — they are facts about MediaPipe, about the maths, or about what an operation means — and those cost real time to rediscover. ### Mechanical. Getting these wrong produces wrong output, not a different look. 1. **MediaPipe's normalised space is anisotropic.** It divides x by image *width* and y by *height*, so equal numbers do not mean equal pixels. Multiply x by `aspect = W/H` before any fit, or a "similarity" fitted in that space is not one and head roll comes out subtly wrong. When mapping pixels for an underlay, **both** axes divide by `imgH`. 2. **MediaPipe's left/right naming is viewer-relative in some places and subject-relative in others.** Any left/right pairing read off a table is a coin flip, and a swap looks *almost* right — each eye still has an iris roughly where it belongs — so it survives inspection. Resolve it from geometry. 3. **Ring tables are ordered traversals**, and slot position is the vertex's identity. That is what makes temporal correspondence possible at all, whatever you decide the shapes should look like. `subsampleSlots` returns ring *positions*, not landmark ids. 4. **A wrongly-ordered ring self-intersects, and it is invisible at odd vertex budgets and obvious at even ones.** If you keep ordered rings, assert simplicity in a test; no amount of looking will catch it reliably. 5. **Scaling a ring to thicken it collapses when the ring is degenerate** — a shut eyelid scaled by 1.1 is still shut, so the lash line vanishes on exactly the frames where it is the whole drawing. A fixed radial offset does not. Maths, not taste. 6. **A fractional centre for a small integer-sized shape changes its size.** Round the origin, not the extents, or a 3px mark is 3px on one frame and 4px on the next. 7. **Order of operations on time:** flooring onto a grid and shifting against the clock do not commute. Shift first and the floor discards it on most frames. ### Choices the prototype made. Revisit freely; here is what each was for. | Choice | Its stated reason | How you would learn it was wrong | | --- | --- | --- | | similarity (4 DOF), not affine | extra DOF absorbs out-of-plane head rotation as shear and smears it into the mouth | the residual readout stops responding to head turn | | reference is the Procrustes mean over the shot, not frame 0 | no single frame's idiosyncrasies get baked into every other | one frame's detection error biases the whole take | | smooth the transform, not the contour | sparse keys at velocity minima rejected detector noise for free | it was already broken by a "bounded exception" once keys went dense, so it was never a law | | gaze measured against the eye's corner midpoint | measured against the lid, every blink drags the origin down and fakes a glance at the floor | gaze correlates with blinks | | one gaze shared by both eyes | at this size the per-eye difference is noise, and independent noise reads as wall-eyed | a wink or a real vergence is lost | | hold, never interpolate | a tweened mouth reads as puppet software | motion looks stepped rather than snappy | | palette indices, never sampled RGB | sampling colour produces a pixel-art filter irrecoverably | — | These are where to look first if the output is wrong. They are also where to look first if you want to change the look. ## Conventions - `domain/*` may not require `flow/*`; neither may require `re-frame`. - Every flow function is `(f params inputs) -> output`. No state, no db, no atoms. - Nothing below `subs/` calls `subscribe`. - Every analysis function that reads pixels takes a `debug?` flag and returns its intermediate masks alongside its result, the way `interior.js/extractTeeth(..., wantDebug)` already does. - Port the invariant comments across verbatim. They are the most valuable text in the repo. ## The oracle **Keep `js/`, `index.html` and `serve.py` in the tree through step 5.** They cost nothing, `serve.py` still runs the old tool, and they are the numeric oracle: run both implementations on the same synthetic track and diff. `fit-similarity` and `procrustes-mean` should agree to **1e-9**; a larger gap is a port bug, not float noise. **Parity proves the port is faithful, not that the answer is right.** The JS is a prototype, so keep the two kinds of test apart: a *parity* test pins behaviour while you move it, and is deleted once the move is done; a *correctness* test asserts something you have decided you want, and stays. Conflating them bakes the prototype's mistakes into the rewrite and makes them permanent. Delete them in one commit once the CLJS player renders the synthetic take correctly. **Do not port the debug views** (`drawPanes`, `drawInteriorDebug`, `drawEyeOverlay`, `drawGazeDebug` in `js/app.js`). The knowledge in them is not the canvas calls — it is *which things you must see to tune teeth*: the source crop, the in-region mask, the surviving mask, and the local contour. That contract already exists as `extractTeeth(..., wantDebug)` returning `debugCanvas(src, inReg, mask, pw, ph, local)`. **Port the payload, skip the drawing.** Redrawing it is ten lines whenever it is wanted. ## Steps Each step ends somewhere runnable. Do not proceed past a step whose "done" does not hold. ### 0 — scaffold and the oracle `mise install`. Create `frontend/` with shadow-cljs, reagent, re-frame. Port `synth.js` (the synthetic landmark generator, including its `swapIris` flag) and the numeric assertions from `selftest.js` to `cljs.test`. **Done:** the suite runs and fails informatively. ### 1 — the pure bottom Port verbatim: `landmarks.js` → `domain/landmarks`, `mathutil.js` → `domain/geom`, ring helpers → `domain/ring`, `raster.js` → `domain/raster`, the palette → `domain/palette`. **Done:** tests pass, including ring simplicity and the swapped-iris vote. Numeric agreement with the JS to 1e-9. Nothing renders. ### 2 — the data model, with no analysis in it `domain/channel` (`value-at` across all three shapes, plus a per-channel cursor), `domain/node` (transform composition), `domain/scene` (topological order by parent depth, `eval-frame` → draw ops in z order). Hand-write a scene in EDN — a rectangle parented to a group whose `[:xform :pos]` is keyed on four frames — and render it through `domain/raster` into a canvas. This is deliberately before any analysis. **The data model has never been validated; find out here**, with fifty lines to throw away, rather than after porting nine hundred lines of measurement into a shape that does not work. **Done:** something moves on screen. ### 3 — the player `clock` (audio-clocked: `frame = ⌊currentTime · fps⌋`, so a slow loop drops frames instead of drifting; ½× and ¼× come free from `playbackRate`), the rAF loop, a `::resolver` sub, and transport UI. The loop reads and blits and **dispatches nothing**. The sub yields a resolver closure; the loop applies it at the playhead. The playhead itself lives in app-db like everything else — with layer-2 extractors and layer-3 computations, a playhead tick does not invalidate the expensive stages. **Done:** the hand-written scene plays at 30fps against audio, scrubs, and runs at ½× and ¼×. ### 4 — measure: anchor and mouth Port `stabilize` and the lip rings out of `pipeline.js`. Split **condition** (`smoothTransforms`, `smoothContours`) into its own stage so the two smoothing knobs do not re-run measurement. **Done:** measured numbers match the JS on the synthetic track. Note that parity here is on `stabilize`'s output, not on `toRasterRing`'s — the framing step is being deleted, not ported. ### 5 — freeze The new module, and the heart of this work: measurements → channels. A dense `[:geom :pts]` block per node — `Int16Array[frames × verts × 2]` in the node's own local space, with the block's fixed-point scale in its header — plus `:generated`. Fixed topology is what makes this a rectangular array with no per-frame header. **Do not port `makeXform`.** The prototype bakes the framing into the stored numbers: it centres on the face oval's bbox and zooms until the face is 80% of the raster height, so every vertex carries a cropping decision made once from one frame's landmarks. Dropping it is a deletion. Placement becomes `[:xform :*]` on an authored `:face` node, the stage clips whatever hangs off, and project dimensions stop being tied to the footage. See "What space geometry is in" in `docs/animation-model.md`. The anchor transform freezes onto `:head`, one level under `:face`, and the normalise on/off/per-plate toggle is which of the three channel shapes that node carries. Always measure and always store factored, whatever the toggle says: smoothing and velocity-minimum key selection both require the split to exist in storage. **Done:** the synthetic take plays back as a moving mouth. Full vertical slice. ### 6 — detect MediaPipe interop behind one namespace; real frames, real audio, real fps from the manifest. **Vendor the wasm** rather than fetching from jsdelivr — it is currently the only thing in the tool that silently requires a network. **Done:** real footage plays back as a rotoscoped mouth. ### 7 — the rest of measure Eyes (openness, gaze, iris pairing vote, blink resolution with its `hold`), brows (raise and tilt at both ends, both correspondence votes), interior (otsu, morphology, components, radial contour). Each keeps its `debug?` payload. Two things fall out of the model instead of being written: the brow's measure-the-height-out-and-put-it-back is `[:geom :pts]` plus `[:xform :pos]`, two channels on one node; and the iris is a `:disc` node parented to the lid ring and stencilled by the sclera. **Done:** parity with the JS tool, minus paint. ### 8 — knobs The parameter UI, as leaf-addressed params in app-db (`clip/:cid/params/:subject/:area` — see below), so the sync layer added later has nothing to retrofit. ### 9 — backend Django project, the `clips` app, models for Project/Clip/Footage/Analysis/Leaf, and project load/save. Round-tripping a project through the server is the proof the model serialises. **Output is not in this plan.** The `.take` writer in `js/take.js` was for driving an Animator Pro render script and it is not where this is going: the target is encoding video in the browser, and that is a separate piece of design nobody should pre-empt by porting the old one. ## Two things to not foreclose The feature controls will later be rethought to handle more than one face, periodic occlusion, stable identity across frames, and feature groups with their own parameters. That design can wait; two decisions here are free now and annoying to reverse: - **Presence is not visibility.** An occluded subject has *no value* on a frame, which is different from a part being hidden. Give every dense block a `Uint8Array` state mask per frame and let it mean *absent* as well as hidden. - **Params carry a subject segment.** `clip/:cid/params/:subject/:area`, with one subject today. Adding a path segment later touches every read and write. The identity tracker, when it comes, should use the same pattern the iris and brow correspondences already use: vote across every frame rather than trusting one.