arthur/frontend/README.md

# frontend

The ClojureScript half. See `docs/port-plan.md` for what is being built and in
what order; this file is only how to run it.

## Once

```sh
mise install                                  # from the REPO ROOT
pip install -r requirements.txt               # the Django half; one dependency
mise exec -- python manage.py migrate         # the document database
cd frontend && npm install
```

`java` must be 21+. On an older JDK shadow-cljs fails with "CompilerOptions has
been compiled by a more recent version of the Java Runtime", which reads like a
shadow-cljs bug and is not one. `mise install` is what prevents it.

## The tests

```sh
cd frontend && mise exec -- npm test
```

Two things: compile the `:test` build, run it under node.

```
shadow-cljs compile test
node out/node-tests.js
```

Run them separately if a compile error is in the way.

**Run them through `mise`**, or make sure `mise`'s node is first on PATH. `java`
must be 21+ and node 20.19+. On an nvm node 20.11 shadowing the pinned one,
things fail in ways that read like the code being broken and are not.

### And the Django one

```sh
mise exec -- python manage.py test clips      # from the REPO ROOT
```

Tests cover the API: the blob store, key verification, the load/save round trip,
the conditional write, source analysis blocks, and video upload and extraction.
The two groups worth
reading are the ones that make the tier split a property of the system rather than
a convention in ClojureScript — the server recomputes every tier-2 key it is
handed, and refuses a block whose analysis does not declare a detector version.

### And the browser one

Step 5's done-criterion is a PICTURE, and no assertion in `cljs.test` can check
one: a take that resolves to the right numbers and draws nothing would pass every
test in `arthur.flow.freeze-test`. A blank canvas under a perfectly correct
transport is the bug class unit tests miss, and it has happened here once.

Step 9's is a picture too, for a different reason: the ways a document survives a
round trip LOOKING correct are the interesting ones. So the suite now also saves
the take, reopens it, and checks the frames are the same pixels.

It drives a real Chrome over CDP, and needs both processes up:

```sh
mise exec -- python manage.py runserver 8778              # from the REPO ROOT
cd frontend && mise exec -- npx shadow-cljs watch app     # in one shell
cd frontend && mise exec -- npm run browser               # in another
```

`ARTHUR_URL` overrides the page it drives; it defaults to
`http://localhost:8778/index.html`, which since step 9 is Django's.

No dependencies. Playwright is not installed and CDP needs none —
`node --experimental-websocket` has a global `WebSocket` and
`--headless=new --remote-debugging-port=N` is the whole of the other side. It
reads the canvas's own pixels rather than a screenshot, because the CSS scales
the stage up by 2 and a screenshot is four pixels per raster pixel; it writes
PNGs into `test/browser/out/` anyway, so "it drew something" can be checked by
eye as well as by count.

## The app

Two processes, which do not talk to each other:

```sh
mise exec -- python manage.py runserver 8778              # from the REPO ROOT
cd frontend && mise exec -- npx shadow-cljs watch app
```

Then open **<http://localhost:8778/>**. Django serves the page from
`clips/templates/clips/index.html`, its styles from `static/arthur/app.css`, and
the bundle out of `static/arthur/js`, where `shadow-cljs` already writes it — so
nothing copies files between the two.

### The window

One screen, five panes, no scrolling page. `src/arthur/ui/shell.cljs` is the grid
and nothing else; each pane owns its own subscriptions.

```
 top      the document: its name and last status, export, new / open / save
 left     media pool — the open document's symbols, and footage on the server
 centre   the palette strip (16 slots) above the stage
 right    inspector — the clip, the selected node, the tracked objects
 bottom   timeline — transport, ruler, a row per node
```

**It opens on a blank document**, and **new** makes another one. Nothing is
loaded until it is asked for.

**Whole documents live under `open ▾`, not in the media pool**, and the split is
load-bearing rather than tidy. Opening a project REPLACES the stage; everything
in the pool is a thing to put ON it. Listing documents beside the symbols inside
one of them makes them read as two kinds of the same thing. The menu lists the
projects the server holds; the built-in scenes are under their own heading,
italic, and are not projects — they are compiled into the bundle and the server
has never heard of them.

Everything that holds nodes is a **symbol**, and none is special: a new document
has one called `main` because it has to be called something. Which symbol is on
screen is editor state, `[:ui :open]`, not a fact about the document — the stage
draws it, the timeline lists it, the transport plays it and a new shape goes into
it. A document opens on the longest symbol nothing else places.

Selection lives in app-db under `:ui`, as `[:node <symbol> <node>]`,
`[:symbol <id>]` or `[:subject|:feature|:group <id>]` — four panes ask what is
selected, and a ratom private to one of them can only be shared by making the
other three require it.

**Drop a video on the media pool** and it uploads, extracts and goes straight on
into detection. Dragging a symbol out of the pool onto the stage places an
instance of it at the playhead.

The timeline's rows are the open symbol's nodes, front-most first, with a dot per
keyframe and a bar over the frames the node exists on; a dense channel is hatched
rather than ticked, because one value per frame is a solid block that says less
than the bar does. Opening a row shows its channels; opening an **instance** row
shows the symbol it places, with every frame number mapped back into the open
symbol's frame space — see the namespace docstring in `ui/timeline.cljs`, which
is where that mapping is argued.

### Paint sketch

Pick a tone from the palette strip, click **polygon**, place at least three
vertices on the stage, then click **finish**. Select a shape — on the stage, or by
its timeline row — to drag its vertices. Scrub to another frame and click
**drawing key here** in the inspector to copy the visible outline there; the
previous drawing holds until that key. The numbered key buttons jump to editable
keys. The transition control between two drawing keys can switch that gap between
a hold and linear vertex tweening. Other gaps keep their own timing. Tweening works
best when the same vertex
keeps the same meaning in every drawing. Paint shapes use the timeline clock
directly, so the roto exposure grid does not delay a drawing key or step its
tween. Use the project **save** button to persist the drawings.

`/index.html` still works, and that is deliberate: it is the URL the browser suite
has used since step 5, when shadow-cljs's `:dev-http` did no directory-index
resolution and the suite learned to ask for the file.

Four built-in clips, under **built-in examples** in the open menu:

| | |
| --- | --- |
| `take` | the synthetic take, head **as filmed**. Step 5's deliverable: a moving mouth, frozen into dense channels, with no video file anywhere. |
| `locked` | the same freeze, head **locked**. The same blocks — `:head`'s channels are written as framed identity instead of as a dense track, and nothing in tier 2 differs. |
| `demo` | the hand-written scene from step 2. Not a face: the smallest scene that exercises every mechanism the model claims to have, so that each one is visible when it breaks. |
| `swarm` | a hundred and twenty dense nodes. Not useful; it is the load test. |

`take` and `locked` are the pair worth looking at together, because switching
between them is the whole of what "stabilisation is a channel, not a mode" means.

The demo scene itself is `src/arthur/demo/scene.edn`. Both the synthetic take
and real footage use `src/arthur/flow/take.cljs` for the measurement order and
`src/arthur/flow/freeze.cljs` for the landmark-to-channel conversion.

**8625 stage study**, in the open menu, loads the locally saved `IMG_8625.MOV` project and places its
post-processed face symbol twice. The stage layout is
`src/arthur/demo/stage_8625.edn`: the right picture and sound start at frame 48,
and the two pictures overlap slightly in stage space. Audio has its own
nodes, linked to the picture instances but with independent spans and gain
channels. The right sound swells and pans across the stage, then fades out at
frame 260 while its picture continues to
frame 280. The row needs that saved 8625 project in the local server database.

### Projects and the EDN fixtures

The EDN files under `demo/` are authored examples compiled into the frontend.
They seed a clip in memory; the server does not read EDN. Clicking **save** on a
clip without a project id creates a project through `POST /api/projects`, uploads
any missing content-addressed blocks, then writes the clip's addressed leaves
through `PUT /api/projects/<id>`. Each leaf value is Transit JSON inside the
request's ordinary JSON envelope. Python stores those values in JSON columns and
does not need an EDN parser. **open** reads the leaves and blocks and rebuilds
the same ClojureScript clip.

The intended editor creates and changes that in-memory clip directly: a project
browser and **new stage** action, timeline instance placement, node and channel
editors, then the existing save path. EDN remains useful for checked-in examples
and reproducible studies. The UI now has the blank-stage action (**new**), a
project browser (`open ▾`), and placement by dragging a symbol out of the media
pool; node and channel editors are still to come — the inspector reports a
channel's shape but has nowhere to change its values.

### Real footage

Drop a video on the media pool, or use its **+** button. The server probes it, re-encodes it
to an H.264 proxy and a raw stream of the same coded frames, pulls WAV audio and one tracing JPEG per frame,
then makes the resulting footage selectable and runs detection on it. **roto**, in
the pool's header, does the same for footage that is already there. Extraction progress is currently read from `/api/extractions/<key>`; a
future WebSocket can push the same job state. The uploaded bytes, extraction job,
and decoded footage have separate records, so the same uploaded video can be
reopened without decoding it again.

**The proxy is what gets measured, and the stills are not.** `flow/ingest` cuts
its raw H.264 stream into coded frames and decodes them in order with WebCodecs.
The proxy has no B-frames, so decode order matches frame order. `flow/detect`
hands each decoded frame to MediaPipe in **VIDEO** running mode at
`i * 1000 / fps` milliseconds. That timestamp has to increase
strictly and has to be real footage time: video mode is a tracker, it reads the
gap between timestamps as motion, and a repeat leaves the graph in an error state
that every later call re-throws. The JPEGs beside the proxy are reference images
for the tracing editor and nothing measures them, so they are not in the footage
digest.

It is re-encoded even when the upload is already H.264, for two reasons: an
iPhone's HEVC is not decodable in every browser, and the footage's identity is the
proxy's digest — one produced by one ffmpeg invocation, not one that depends on
which branch the source happened to take. Its raw stream is copied from that
proxy without another encode.

The command-line route is also available for an existing extracted bundle:

```sh
./extract.sh /path/to/clip.mov          # decode to frames + audio + manifest
mise exec -- python manage.py ingest_bundle
```

`extract.sh` keeps every source frame and writes `frames/0001.png` onward,
`audio.wav` and `manifest.json`. Variable frame rate sources are rejected until the
manifest and clock carry per-frame timestamps.

`ingest_bundle` then hashes all of it into the content-addressed blob store under
`var/blobs` — by hard link, so 112MB of PNGs is not copied — and registers one
`Footage` row. From then on the frames are the backend's: `GET /api/footage/<id>`
answers with a manifest carrying **a URL per frame**, and the app fetches those.

That replaced a shared secret. Until step 9 the page fetched `/manifest.json` off
the filesystem and built `frames/0001.png` itself, with shadow-cljs serving the
repo root — so the frame layout was agreed between a shell script and a
ClojureScript namespace, and "where are the frames" was answered by a directory
listing. The cache-busting `?v=` that used to hang off every frame URL went with
it: a blob's name is the hash of its bytes, so re-extracting gives a frame a
different URL rather than overwriting one.

To keep several takes, pass a bundle directory; each ingests separately and both
stay selectable in the app.

```sh
./extract.sh /path/to/clip.mov scratch/my-take
mise exec -- python manage.py ingest_bundle scratch/my-take
```

`scratch/` is ignored by Git, as are `frames/`, `audio.wav` and `manifest.json` at
the root — all of it is extraction output, and tier 3 does not belong in the repo.

Loading detects one face per frame, measures the mouth, eyes and brows from
landmarks and the teeth from source pixels, then freezes them into channels,
and opens the footage clip. Detection happens once when you load;
playback only resolves channels and paints. Frames without a detection remain
marked absent even though their neighbouring poses are used to condition the
track. The scene now records stable subject and feature IDs and explicit eye
pairs; dense channels can mark one feature absent while another is observed.
Current MediaPipe loading supplies only the full-face detection mask. The stage
stays 320×200 regardless of the footage dimensions. Real
footage starts at the source picture rate. The **picture** buttons in the inspector sample the
frozen roto at lower rates while the source track, duration and audio clock stay
unchanged. Picking frames to trace into cels is a separate future editing step.
**save** also stores the detection mask, dense landmarks and raw RGBA mouth crops
as three analysis blocks. **open** restores these without running MediaPipe or
loading source PNGs. The frozen shapes remain separate channel blocks.

For known occlusion intervals, an extracted manifest may add
`"feature-absence": {"eye-r": [[10, 14]]}`. Frame numbers are one-based and
inclusive, matching PNG filenames. The eye remains the same feature when it
reappears; the other eye and the mouth continue through the gap. This is an
input annotation, with no UI for editing it yet.

MediaPipe's JS, wasm and model are under `public/mediapipe/`, served by Django's
staticfiles under `/static/mediapipe/`. No CDN is used by this app. See that
directory's README for provenance.

The server reports what it serves at `GET /api/detector`: the package version plus
the **sha256 of the model asset**, and that string goes inside the content address
of every block a detection produces. Asked rather than assumed, because a version
constant in the client is one somebody has to remember to bump — and
`docs/architecture.md` is explicit that a model upgrade silently reusing old
landmarks presents as "the tool got worse", with no event to attach it to.

Port 8778 is deliberately not 8777. `python3 serve.py` from the repo root still
runs the old JS tool on 8777, and the two are meant to run side by side.

## Saving

**new**, **open** and **save** in the top bar. A save has three ordered stages:
is the tier split:

1. the **analysis** record, so every block stored afterwards can name the detector
   version that produced it. The server refuses a block whose analysis it does not
   know.
2. ask which **blocks** are missing, upload the source analysis blocks and frozen
   channel blocks, then link the source blocks to the analysis.
3. the **document** — tier 1, as leaves. The server refuses a clip that names
   blocks it does not hold, so a saved document cannot load into a blank stage
   somewhere else.

The status line says what happened: `saved r3 · 64 leaves · 8 blocks`. Saving an
unchanged document says `0 leaves · 0 blocks`, which is both halves of the
addressing working at once — an unchanged leaf keeps its version, and a
content-addressed block is already there.

Saving `swarm` is deliberately visible as a failure: its blocks have
hand-written names and a document may only name content addresses.

`open ▾` lists every project the server holds, newest first, and shows the first
clip of whichever one is picked — the store holds one clip at a time.

## The oracle, which is finished

`js/` was the numeric oracle through step 4: `test/parity/` ran both
implementations on the same synthetic track and diffed `fit-similarity`,
`procrustes-mean`, the raster and `stabilize` to 1e-9.

**It was deleted at step 5, on purpose.** Parity proves the port is FAITHFUL, not
that the answer is RIGHT. The JS is a prototype and several of its conclusions
contradict each other; a parity test pins behaviour while code moves, and keeping
it afterwards would bake the prototype's mistakes into the rewrite and make them
permanent. `docs/port-plan.md` says to delete it in one commit once the CLJS
player renders the synthetic take, and that is what happened.

`js/` itself stays as the reference for the MediaPipe setup, face measurements
and pixel extraction. Its comments encode bugs that actually happened.

## Layout

```
src/arthur/domain/      pure. No re-frame, no DOM, no flow/.
src/arthur/fx/          the only namespaces that talk to the network
src/arthur/flow/        the stages. `(f params inputs) -> output`, no state.
src/arthur/synth.cljs   the synthetic track. In src/ because the take PLAYS it —
                        it stands in for flow/detect, and a tool that needs a
                        video file before it shows you anything is one you
                        cannot debug.
src/arthur/demo.cljs    the hand-written scene, read from demo/scene.edn
src/arthur/demo/take.cljs  the synthetic source for the shared flow/take path
src/arthur/ui/canvas.cljs  indexed raster blit to the display canvas
test/arthur/support/    machinery shared between suites; not tests itself
test/browser/           drives a real Chrome over CDP. Not run by `npm test`.
public/mediapipe/       vendored wasm and model, served under /static/mediapipe/
```

`public/` holds nothing but those assets now. The host page that used to sit beside
them is `clips/templates/clips/index.html`.

## Two evaluators, on purpose

`domain/symbol` has both `eval-frame` and `resolver`, and they are not
alternatives:

- **`(eval-frame symbol f store)`** is the specification. Allocating, order-free,
  obviously correct. Tests and one-off renders use it.
- **`(resolver symbol store)` -> `(fn [f] ops)`** is what playback uses. It caches
  the topological order and the z paths, holds a cursor per channel and reuses
  one point buffer per node, so a frame allocates the op maps and nothing else.

Both run the same walk, parameterised by how a channel is read and where its
points are written — two independent implementations of frame evaluation would
drift, and the drift would look like a rendering bug rather than like two
functions disagreeing. What differs between them is exactly the part that can be
wrong, and `scene-test` asserts they agree frame for frame in forward, backward
and random order.

Because the resolver reuses its buffers, **ops must be rasterised before the
next frame is asked for.** That is the contract the rAF loop wants anyway: it
reads, blits, and dispatches nothing.