163 lines
7.1 KiB
Markdown
163 lines
7.1 KiB
Markdown
# frontend
|
||
|
||
The ClojureScript half. See `docs/port-plan.md` for what is being built and in
|
||
what order; this file is only how to run it.
|
||
|
||
## Once
|
||
|
||
```sh
|
||
mise install # from the REPO ROOT: java 21+, node 20, clojure, python
|
||
cd frontend && npm install
|
||
```
|
||
|
||
`java` must be 21+. On an older JDK shadow-cljs fails with "CompilerOptions has
|
||
been compiled by a more recent version of the Java Runtime", which reads like a
|
||
shadow-cljs bug and is not one. `mise install` is what prevents it.
|
||
|
||
## The tests
|
||
|
||
```sh
|
||
cd frontend && mise exec -- npm test
|
||
```
|
||
|
||
Two things: compile the `:test` build, run it under node.
|
||
|
||
```
|
||
shadow-cljs compile test
|
||
node out/node-tests.js
|
||
```
|
||
|
||
Run them separately if a compile error is in the way.
|
||
|
||
**Run them through `mise`**, or make sure `mise`'s node is first on PATH. `java`
|
||
must be 21+ and node 20.19+. On an nvm node 20.11 shadowing the pinned one,
|
||
things fail in ways that read like the code being broken and are not.
|
||
|
||
### And the browser one
|
||
|
||
Step 5's done-criterion is a PICTURE, and no assertion in `cljs.test` can check
|
||
one: a take that resolves to the right numbers and draws nothing would pass every
|
||
test in `arthur.flow.freeze-test`. A blank canvas under a perfectly correct
|
||
transport is the bug class unit tests miss, and it has happened here once.
|
||
|
||
So there is a second suite that drives a real Chrome over CDP. It needs the dev
|
||
server up:
|
||
|
||
```sh
|
||
cd frontend && mise exec -- npx shadow-cljs watch app # in one shell
|
||
cd frontend && mise exec -- npm run browser # in another
|
||
```
|
||
|
||
No dependencies. Playwright is not installed and CDP needs none —
|
||
`node --experimental-websocket` has a global `WebSocket` and
|
||
`--headless=new --remote-debugging-port=N` is the whole of the other side. It
|
||
reads the canvas's own pixels rather than a screenshot, because the CSS scales
|
||
the stage up by 2 and a screenshot is four pixels per raster pixel; it writes
|
||
PNGs into `test/browser/out/` anyway, so "it drew something" can be checked by
|
||
eye as well as by count.
|
||
|
||
## The app
|
||
|
||
```sh
|
||
cd frontend && mise exec -- npx shadow-cljs watch app
|
||
```
|
||
|
||
Then open **<http://localhost:8778/index.html>** — with the `/index.html`, not
|
||
bare `/`. This shadow-cljs does no directory-index resolution, so `/` is a 404
|
||
whatever the roots are.
|
||
|
||
Four built-in clips, on buttons in the transport:
|
||
|
||
| | |
|
||
| --- | --- |
|
||
| `take` | the synthetic take, head **as filmed**. Step 5's deliverable: a moving mouth, frozen into dense channels, with no video file anywhere. |
|
||
| `locked` | the same freeze, head **locked**. The same blocks — `:head`'s channels are written as framed identity instead of as a dense track, and nothing in tier 2 differs. |
|
||
| `demo` | the hand-written scene from step 2. Not a face: the smallest scene that exercises every mechanism the model claims to have, so that each one is visible when it breaks. |
|
||
| `swarm` | a hundred and twenty dense nodes. Not useful; it is the load test. |
|
||
|
||
`take` and `locked` are the pair worth looking at together, because switching
|
||
between them is the whole of what "stabilisation is a channel, not a mode" means.
|
||
|
||
The demo scene itself is `src/arthur/demo/scene.edn`. Both the synthetic take
|
||
and real footage use `src/arthur/flow/take.cljs` for the measurement order and
|
||
`src/arthur/flow/freeze.cljs` for the landmark-to-channel conversion.
|
||
|
||
### Real footage (port step 6)
|
||
|
||
From the repo root, extract a clip, then click **load frames** in the CLJS app:
|
||
|
||
```sh
|
||
./extract.sh /path/to/clip.mov 12
|
||
```
|
||
|
||
This writes `frames/0001.png` onward, `audio.wav`, and `manifest.json` at the
|
||
repo root. The manifest supplies the exact frame count, fps and audio path.
|
||
Loading detects one face per frame, freezes the measured mouth into channels,
|
||
and adds a button for the footage clip. Detection happens once when you load;
|
||
playback only resolves channels and paints. Frames without a detection remain
|
||
marked absent even though their neighbouring poses are used to condition the
|
||
track. The stage stays 320×200 regardless of the footage dimensions.
|
||
|
||
MediaPipe's JS, wasm and model are under `public/mediapipe/` and served locally.
|
||
No CDN is used by this app. See that directory's README for provenance.
|
||
|
||
Port 8778 is deliberately not 8777. `python3 serve.py` from the repo root still
|
||
runs the old JS tool on 8777, and the two are meant to run side by side.
|
||
|
||
From step 9 Django serves the page and `:dev-http` goes away.
|
||
|
||
## The oracle, which is finished
|
||
|
||
`js/` was the numeric oracle through step 4: `test/parity/` ran both
|
||
implementations on the same synthetic track and diffed `fit-similarity`,
|
||
`procrustes-mean`, the raster and `stabilize` to 1e-9.
|
||
|
||
**It was deleted at step 5, on purpose.** Parity proves the port is FAITHFUL, not
|
||
that the answer is RIGHT. The JS is a prototype and several of its conclusions
|
||
contradict each other; a parity test pins behaviour while code moves, and keeping
|
||
it afterwards would bake the prototype's mistakes into the rewrite and make them
|
||
permanent. `docs/port-plan.md` says to delete it in one commit once the CLJS
|
||
player renders the synthetic take, and that is what happened.
|
||
|
||
`js/` itself stays. It is not an oracle any more, it is the SOURCE for steps 6
|
||
and 7 — the MediaPipe setup, the eye and brow signals, the interior extraction —
|
||
and its comments encode bugs that actually happened.
|
||
|
||
## Layout
|
||
|
||
```
|
||
src/arthur/domain/ pure. No re-frame, no DOM, no flow/.
|
||
src/arthur/flow/ the stages. `(f params inputs) -> output`, no state.
|
||
src/arthur/synth.cljs the synthetic track. In src/ because the take PLAYS it —
|
||
it stands in for flow/detect, and a tool that needs a
|
||
video file before it shows you anything is one you
|
||
cannot debug.
|
||
src/arthur/demo.cljs the hand-written scene, read from demo/scene.edn
|
||
src/arthur/demo/take.cljs the seven stages composed in order, and the only place
|
||
they are
|
||
src/arthur/ui/canvas.cljs the one imperative sink — the only DOM canvas call
|
||
test/browser/ drives a real Chrome over CDP. Not run by `npm test`.
|
||
public/index.html dev host page. Django replaces it at step 9.
|
||
```
|
||
|
||
## Two evaluators, on purpose
|
||
|
||
`domain/scene` has both `eval-frame` and `resolver`, and they are not
|
||
alternatives:
|
||
|
||
- **`(eval-frame scene f store)`** is the specification. Allocating, order-free,
|
||
obviously correct. Tests and one-off renders use it.
|
||
- **`(resolver scene store)` -> `(fn [f] ops)`** is what playback uses. It caches
|
||
the topological order and the z paths, holds a cursor per channel and reuses
|
||
one point buffer per node, so a frame allocates the op maps and nothing else.
|
||
|
||
Both run the same walk, parameterised by how a channel is read and where its
|
||
points are written — two independent implementations of frame evaluation would
|
||
drift, and the drift would look like a rendering bug rather than like two
|
||
functions disagreeing. What differs between them is exactly the part that can be
|
||
wrong, and `scene-test` asserts they agree frame for frame in forward, backward
|
||
and random order.
|
||
|
||
Because the resolver reuses its buffers, **ops must be rasterised before the
|
||
next frame is asked for.** That is the contract the rAF loop wants anyway: it
|
||
reads, blits, and dispatches nothing.
|