Port steps 0-1: scaffold, the oracle, and the pure bottom
Scaffolds frontend/ (shadow-cljs, reagent 1.2.0, re-frame 1.4.3) and ports everything below the data model, with the JS kept as a numeric oracle. domain/landmarks index tables, verbatim domain/ring subsample, offset, simplicity domain/geom similarity fit, procrustes, moving average domain/raster indexed scanline fill, stencil, disc, rect domain/palette the ramp, and the no-sampled-RGB rule 58 tests, 166 assertions. Parity with js/ on the identical 72-frame synthetic track: fit-similarity, procrustes-mean, fit-residual, moving-average, smooth-transforms, offset-ring and subsample-slots to 1e-9; the raster pixel-for-pixel over the whole buffer. Three deviations from the JS, each for a reason: - synth.cljs jitters from a SEEDED generator, not Math.random. Parity is only checkable if both sides can be handed the same track, and a failing assertion has to be reproducible. `:rand-fn` takes the generator over, so oracle.mjs stubs js/Math.random and js/ itself stays untouched. - raster/->rgba replaces toImageData. ImageData is a DOM type and domain/ may not touch the DOM; returning plain bytes also lets the no-intermediate-colours assertion run in node. ui/canvas wraps it later. - offset-ring lives in domain/ring, not domain/geom, per architecture.md: it is an operation on an ordered traversal, not on a transform. Step 1's "done" also names the swapped-iris vote, but pairIrises is in pipeline.js and belongs to step 7. The precondition is asserted instead -- `:swap-iris` really does move both blocks -- so the vote will have a track that disagrees with it when it arrives. One finding, recorded in full in the test that measures it: smooth-transforms buys nothing on the synthetic track. Against jitter-free ground truth, radius 1 helps by 17% on one noise realisation and hurts by 0.5% on another, so its benefit is within noise; from radius 2 up the cost is unambiguous, and by radius 5 the filter is below the true motion's own high-frequency energy, i.e. smoothing away performance. The test pins the shape of the knob rather than a preferred value. This may say more about the synth's jitter being unrealistically small (+/-0.001 normalised) than about the knob; step 6 settles it on real footage. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Olive Vaughn
parent
082d8561d2
commit
eb06be005c
25 changed files with 4932 additions and 0 deletions
321
docs/port-plan.md
Normal file
321
docs/port-plan.md
Normal file
|
|
@ -0,0 +1,321 @@
|
|||
# arthur — port plan and handoff
|
||||
|
||||
Self-contained. You should not need any prior conversation to execute this.
|
||||
|
||||
## What arthur is
|
||||
|
||||
A tool that turns live-action video into 2D animation that reads as
|
||||
hand-authored: flat polygons, a tiny indexed palette, hard edges, 320×200, no
|
||||
antialiasing, motion carried by silhouette. It tracks a face out of a clip,
|
||||
reduces the lip contour to a handful of vertices, derives teeth from image
|
||||
content, and renders flat indexed fills.
|
||||
|
||||
It currently works, as vanilla JS ES modules with no build step. `python3
|
||||
serve.py`, open `127.0.0.1:8777`. **Synthetic take** exercises everything below
|
||||
detection with no video needed.
|
||||
|
||||
This plan converts it to ClojureScript + re-frame, restructured around one
|
||||
uniform animation data model, and adds a Django backend for persistence and
|
||||
(later) collaboration.
|
||||
|
||||
## Status of the existing documents
|
||||
|
||||
| File | What it is | Authority |
|
||||
| --- | --- | --- |
|
||||
| `js/**` | the working tool, ~4,800 lines | **authoritative.** The comments encode bugs that actually happened. |
|
||||
| `docs/animation-model.md` | the target data model: nodes, channels, symbols, time maps | build to this |
|
||||
| `docs/architecture.md` | module layout, stages, sync and baking design | build to this; much of it is future scope |
|
||||
| `docs/design.md`, `README.md` | prior synthesis by an earlier agent | useful, **not authoritative**. Revise freely. Do not treat its aesthetic claims as settled requirements. |
|
||||
|
||||
Where a document and the code disagree, the code wins, and the invariant list
|
||||
below is lifted from the code for exactly that reason.
|
||||
|
||||
## Target repo layout
|
||||
|
||||
Both halves live here. Django at the root, because `manage.py` at the root is the
|
||||
convention and keeps every `python manage.py` invocation working with no `cd`.
|
||||
|
||||
```
|
||||
arthur/
|
||||
mise.toml toolchain for both halves
|
||||
manage.py
|
||||
requirements.txt
|
||||
server/ Django project: settings, urls, asgi, wsgi
|
||||
clips/ Django app: models, views, consumers, routing, migrations
|
||||
frontend/ the CLJS app
|
||||
shadow-cljs.edn
|
||||
package.json
|
||||
src/arthur/** namespace root stays arthur.* whatever the dir is called
|
||||
test/arthur/**
|
||||
static/arthur/js/ shadow-cljs output, collected by Django staticfiles
|
||||
docs/
|
||||
js/ index.html serve.py extract.sh the old tool — see "the oracle"
|
||||
```
|
||||
|
||||
`clips` is a naming call, not a constraint — it is the Django app holding
|
||||
Project, Clip, Footage, Analysis, Leaf and Revision. Rename in one line if
|
||||
something fits better.
|
||||
|
||||
Dev runs two processes: Django serves the page, `shadow-cljs watch app` rebuilds
|
||||
into `static/arthur/js`. Set `:output-dir "../static/arthur/js"` in
|
||||
`shadow-cljs.edn`.
|
||||
|
||||
## Toolchain
|
||||
|
||||
`mise install` from the repo root. `mise.toml` pins java 21+, node 20, clojure,
|
||||
python 3.12, and creates `.venv`.
|
||||
|
||||
Verified to resolve cleanly: `reagent 1.2.0`, `re-frame 1.4.3`, current
|
||||
shadow-cljs.
|
||||
|
||||
## Scope
|
||||
|
||||
**In:** analysis → keyframes → playback. The pure numeric core, the animation
|
||||
data model, a player, the measurement stages, and freezing measurements into
|
||||
channels.
|
||||
|
||||
**Out, and do not build it:** paint and cels; `suggest` (it only decides which
|
||||
frames get a hand-drawn cel, so it has no job until drawing exists); the timeline
|
||||
and sequences; symbols and the plate library; multiplayer; the override layer.
|
||||
Each is designed for in `docs/architecture.md` and `docs/animation-model.md`.
|
||||
Leave the `:over` field present and empty; leave `:symbol` out entirely.
|
||||
|
||||
## The data model
|
||||
|
||||
Full specification in `docs/animation-model.md`. The subset to build:
|
||||
|
||||
```clojure
|
||||
;; The scene is a flat map of id -> node. Parent pointers, never nested maps.
|
||||
{:id :mouth :kind :poly :parent :head :z "a3" :stencil nil :span [0 240]
|
||||
:time {:mode :inherit} ; or {:mode :map :expose 2 :offset -1 :rate 1.0}
|
||||
:channels
|
||||
{[:xform :pos] {:animated? false :value [0.0 0.0]}
|
||||
[:xform :rot] {:animated? false :value 0.0}
|
||||
[:xform :scale] {:animated? false :value [1.0 1.0]}
|
||||
[:xform :skew] {:animated? false :value [0.0 0.0]}
|
||||
[:xform :anchor] {:animated? false :value [0.0 0.0]}
|
||||
[:geom :pts] {:animated? true :interp :hold
|
||||
:dense {:store "sha256:…" :offset 0 :stride 40 :frames 600}
|
||||
:generated {:by :roto/lips-outer :analysis "sha256:…"
|
||||
:params {:verts 8 :contour-avg 1}}
|
||||
:over []}
|
||||
[:style :color] {:animated? false :value :skin-dark}
|
||||
[:vis] {:animated? false :value true}}}
|
||||
```
|
||||
|
||||
Three channel shapes, one accessor `(value-at channel f)`:
|
||||
|
||||
- `{:animated? false :value v}` — static. A thing that simply exists.
|
||||
- `{:animated? true :interp :hold :keys {0 v, 4 v}}` — sparse, authored, in the
|
||||
document. **Keys are a map by frame, never a vector.** Store a plain map
|
||||
(transit loses sortedness) and build the sorted index in the resolver.
|
||||
- `{:animated? true :interp :hold :dense {...}}` — generated, one value per
|
||||
frame, in a typed array outside app-db.
|
||||
|
||||
`:generated` is provenance and **the renderer never reads it.** It is what the UI
|
||||
uses to offer a parameter panel instead of raw keys. It lives on the *channel*,
|
||||
not the node, because a node wants a rotoscoped `[:geom :pts]` and a
|
||||
hand-animated `[:xform :pos]` at the same time.
|
||||
|
||||
`:skew`, `:span`, `:anchor` and `:over` stay in the shape even though nothing
|
||||
drives them yet: each is a component of a decomposition or of a composition
|
||||
order, and adding one later migrates every stored transform.
|
||||
|
||||
Transform composition, per node:
|
||||
|
||||
```
|
||||
local = T(pos) · T(anchor) · R(rot) · K(skew) · S(scale) · T(-anchor)
|
||||
world = world(parent) · local
|
||||
```
|
||||
|
||||
## What the prototype knows that you would otherwise rediscover
|
||||
|
||||
**The JS is a prototype.** Its conclusions about what looks right are provisional
|
||||
and you may revisit any of them; several contradict each other already. But a few
|
||||
things in it are not taste — they are facts about MediaPipe, about the maths, or
|
||||
about what an operation means — and those cost real time to rediscover.
|
||||
|
||||
### Mechanical. Getting these wrong produces wrong output, not a different look.
|
||||
|
||||
1. **MediaPipe's normalised space is anisotropic.** It divides x by image *width*
|
||||
and y by *height*, so equal numbers do not mean equal pixels. Multiply x by
|
||||
`aspect = W/H` before any fit, or a "similarity" fitted in that space is not
|
||||
one and head roll comes out subtly wrong. When mapping pixels for an underlay,
|
||||
**both** axes divide by `imgH`.
|
||||
2. **MediaPipe's left/right naming is viewer-relative in some places and
|
||||
subject-relative in others.** Any left/right pairing read off a table is a coin
|
||||
flip, and a swap looks *almost* right — each eye still has an iris roughly
|
||||
where it belongs — so it survives inspection. Resolve it from geometry.
|
||||
3. **Ring tables are ordered traversals**, and slot position is the vertex's
|
||||
identity. That is what makes temporal correspondence possible at all, whatever
|
||||
you decide the shapes should look like. `subsampleSlots` returns ring
|
||||
*positions*, not landmark ids.
|
||||
4. **A wrongly-ordered ring self-intersects, and it is invisible at odd vertex
|
||||
budgets and obvious at even ones.** If you keep ordered rings, assert
|
||||
simplicity in a test; no amount of looking will catch it reliably.
|
||||
5. **Scaling a ring to thicken it collapses when the ring is degenerate** — a shut
|
||||
eyelid scaled by 1.1 is still shut, so the lash line vanishes on exactly the
|
||||
frames where it is the whole drawing. A fixed radial offset does not. Maths,
|
||||
not taste.
|
||||
6. **A fractional centre for a small integer-sized shape changes its size.**
|
||||
Round the origin, not the extents, or a 3px mark is 3px on one frame and 4px on
|
||||
the next.
|
||||
7. **Order of operations on time:** flooring onto a grid and shifting against the
|
||||
clock do not commute. Shift first and the floor discards it on most frames.
|
||||
|
||||
### Choices the prototype made. Revisit freely; here is what each was for.
|
||||
|
||||
| Choice | Its stated reason | How you would learn it was wrong |
|
||||
| --- | --- | --- |
|
||||
| similarity (4 DOF), not affine | extra DOF absorbs out-of-plane head rotation as shear and smears it into the mouth | the residual readout stops responding to head turn |
|
||||
| reference is the Procrustes mean over the shot, not frame 0 | no single frame's idiosyncrasies get baked into every other | one frame's detection error biases the whole take |
|
||||
| smooth the transform, not the contour | sparse keys at velocity minima rejected detector noise for free | it was already broken by a "bounded exception" once keys went dense, so it was never a law |
|
||||
| gaze measured against the eye's corner midpoint | measured against the lid, every blink drags the origin down and fakes a glance at the floor | gaze correlates with blinks |
|
||||
| one gaze shared by both eyes | at this size the per-eye difference is noise, and independent noise reads as wall-eyed | a wink or a real vergence is lost |
|
||||
| hold, never interpolate | a tweened mouth reads as puppet software | motion looks stepped rather than snappy |
|
||||
| palette indices, never sampled RGB | sampling colour produces a pixel-art filter irrecoverably | — |
|
||||
|
||||
These are where to look first if the output is wrong. They are also where to look
|
||||
first if you want to change the look.
|
||||
|
||||
## Conventions
|
||||
|
||||
- `domain/*` may not require `flow/*`; neither may require `re-frame`.
|
||||
- Every flow function is `(f params inputs) -> output`. No state, no db, no atoms.
|
||||
- Nothing below `subs/` calls `subscribe`.
|
||||
- Every analysis function that reads pixels takes a `debug?` flag and returns its
|
||||
intermediate masks alongside its result, the way
|
||||
`interior.js/extractTeeth(..., wantDebug)` already does.
|
||||
- Port the invariant comments across verbatim. They are the most valuable text in
|
||||
the repo.
|
||||
|
||||
## The oracle
|
||||
|
||||
**Keep `js/`, `index.html` and `serve.py` in the tree through step 5.** They cost
|
||||
nothing, `serve.py` still runs the old tool, and they are the numeric oracle:
|
||||
run both implementations on the same synthetic track and diff.
|
||||
`fit-similarity` and `procrustes-mean` should agree to **1e-9**; a larger gap is a
|
||||
port bug, not float noise.
|
||||
|
||||
**Parity proves the port is faithful, not that the answer is right.** The JS is a
|
||||
prototype, so keep the two kinds of test apart: a *parity* test pins behaviour
|
||||
while you move it, and is deleted once the move is done; a *correctness* test
|
||||
asserts something you have decided you want, and stays. Conflating them bakes the
|
||||
prototype's mistakes into the rewrite and makes them permanent. Delete them in one commit once the CLJS player renders
|
||||
the synthetic take correctly.
|
||||
|
||||
**Do not port the debug views** (`drawPanes`, `drawInteriorDebug`,
|
||||
`drawEyeOverlay`, `drawGazeDebug` in `js/app.js`). The knowledge in them is not
|
||||
the canvas calls — it is *which things you must see to tune teeth*: the source
|
||||
crop, the in-region mask, the surviving mask, and the local contour. That contract
|
||||
already exists as `extractTeeth(..., wantDebug)` returning
|
||||
`debugCanvas(src, inReg, mask, pw, ph, local)`. **Port the payload, skip the
|
||||
drawing.** Redrawing it is ten lines whenever it is wanted.
|
||||
|
||||
## Steps
|
||||
|
||||
Each step ends somewhere runnable. Do not proceed past a step whose "done" does
|
||||
not hold.
|
||||
|
||||
### 0 — scaffold and the oracle
|
||||
`mise install`. Create `frontend/` with shadow-cljs, reagent, re-frame. Port
|
||||
`synth.js` (the synthetic landmark generator, including its `swapIris` flag) and
|
||||
the numeric assertions from `selftest.js` to `cljs.test`.
|
||||
|
||||
**Done:** the suite runs and fails informatively.
|
||||
|
||||
### 1 — the pure bottom
|
||||
Port verbatim: `landmarks.js` → `domain/landmarks`, `mathutil.js` → `domain/geom`,
|
||||
ring helpers → `domain/ring`, `raster.js` → `domain/raster`, the palette →
|
||||
`domain/palette`.
|
||||
|
||||
**Done:** tests pass, including ring simplicity and the swapped-iris vote. Numeric
|
||||
agreement with the JS to 1e-9. Nothing renders.
|
||||
|
||||
### 2 — the data model, with no analysis in it
|
||||
`domain/channel` (`value-at` across all three shapes, plus a per-channel cursor),
|
||||
`domain/node` (transform composition), `domain/scene` (topological order by parent
|
||||
depth, `eval-frame` → draw ops in z order).
|
||||
|
||||
Hand-write a scene in EDN — a rectangle parented to a group whose
|
||||
`[:xform :pos]` is keyed on four frames — and render it through `domain/raster`
|
||||
into a canvas.
|
||||
|
||||
This is deliberately before any analysis. **The data model has never been
|
||||
validated; find out here**, with fifty lines to throw away, rather than after
|
||||
porting nine hundred lines of measurement into a shape that does not work.
|
||||
|
||||
**Done:** something moves on screen.
|
||||
|
||||
### 3 — the player
|
||||
`clock` (audio-clocked: `frame = ⌊currentTime · fps⌋`, so a slow loop drops frames
|
||||
instead of drifting; ½× and ¼× come free from `playbackRate`), the rAF loop, a
|
||||
`::resolver` sub, and transport UI.
|
||||
|
||||
The loop reads and blits and **dispatches nothing**. The sub yields a resolver
|
||||
closure; the loop applies it at the playhead. The playhead itself lives in app-db
|
||||
like everything else — with layer-2 extractors and layer-3 computations, a
|
||||
playhead tick does not invalidate the expensive stages.
|
||||
|
||||
**Done:** the hand-written scene plays at 30fps against audio, scrubs, and runs at
|
||||
½× and ¼×.
|
||||
|
||||
### 4 — measure: anchor and mouth
|
||||
Port `stabilize` and the lip rings out of `pipeline.js`. Split **condition**
|
||||
(`smoothTransforms`, `smoothContours`) into its own stage so the two smoothing
|
||||
knobs do not re-run measurement.
|
||||
|
||||
**Done:** measured numbers match the JS on the synthetic track.
|
||||
|
||||
### 5 — freeze
|
||||
The new module, and the heart of this work: measurements → channels. A dense
|
||||
`[:geom :pts]` block per node — `Int16Array[frames × verts × 2]`, raster space,
|
||||
grid-snapped — with `:generated` attached. Fixed topology is what makes this a
|
||||
rectangular array with no per-frame header.
|
||||
|
||||
**Done:** the synthetic take plays back as a moving mouth. Full vertical slice.
|
||||
|
||||
### 6 — detect
|
||||
MediaPipe interop behind one namespace; real frames, real audio, real fps from
|
||||
the manifest. **Vendor the wasm** rather than fetching from jsdelivr — it is
|
||||
currently the only thing in the tool that silently requires a network.
|
||||
|
||||
**Done:** real footage plays back as a rotoscoped mouth.
|
||||
|
||||
### 7 — the rest of measure
|
||||
Eyes (openness, gaze, iris pairing vote, blink resolution with its `hold`), brows
|
||||
(raise and tilt at both ends, both correspondence votes), interior (otsu,
|
||||
morphology, components, radial contour). Each keeps its `debug?` payload.
|
||||
|
||||
Two things fall out of the model instead of being written: the brow's
|
||||
measure-the-height-out-and-put-it-back is `[:geom :pts]` plus `[:xform :pos]`, two
|
||||
channels on one node; and the iris is a `:disc` node parented to the lid ring and
|
||||
stencilled by the sclera.
|
||||
|
||||
**Done:** parity with the JS tool, minus paint.
|
||||
|
||||
### 8 — knobs
|
||||
The parameter UI, as leaf-addressed params in app-db
|
||||
(`clip/:cid/params/:subject/:area` — see below), so the sync layer added later has
|
||||
nothing to retrofit.
|
||||
|
||||
### 9 — backend and take export
|
||||
Django project, the `clips` app, models for Project/Clip/Footage/Analysis/Leaf,
|
||||
and project load/save. Port `take.js` — sixty-five lines, and it is the proof the
|
||||
model serialises.
|
||||
|
||||
## Two things to not foreclose
|
||||
|
||||
The feature controls will later be rethought to handle more than one face,
|
||||
periodic occlusion, stable identity across frames, and feature groups with their
|
||||
own parameters. That design can wait; two decisions here are free now and
|
||||
annoying to reverse:
|
||||
|
||||
- **Presence is not visibility.** An occluded subject has *no value* on a frame,
|
||||
which is different from a part being hidden. Give every dense block a
|
||||
`Uint8Array` state mask per frame and let it mean *absent* as well as hidden.
|
||||
- **Params carry a subject segment.** `clip/:cid/params/:subject/:area`, with one
|
||||
subject today. Adding a path segment later touches every read and write.
|
||||
|
||||
The identity tracker, when it comes, should use the same pattern the iris and brow
|
||||
correspondences already use: vote across every frame rather than trusting one.
|
||||
Loading…
Add table
Add a link
Reference in a new issue