noe/arthur
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Port step 6: detect real footage with local MediaPipe

Browse source
This commit is contained in:
Olive Vaughn 2026-09-27 19:13:20 -04:00
parent 8a06835895
commit 32683efccf
27 changed files with 18258 additions and 54 deletions
Download patch file Download diff file Expand all files Collapse all files

27
frontend/README.md
View file

@ -66,7 +66,7 @@ Then open **<http://localhost:8778/index.html>** — with the `/index.html`, not
bare `/`. This shadow-cljs does no directory-index resolution, so `/` is a 404
whatever the roots are.
Four clips, on buttons in the transport:
Four built-in clips, on buttons in the transport:
| | |
| --- | --- |
@ -78,9 +78,28 @@ Four clips, on buttons in the transport:
`take` and `locked` are the pair worth looking at together, because switching
between them is the whole of what "stabilisation is a channel, not a mode" means.
The demo scene itself is `src/arthur/demo/scene.edn`; the take is built in
`src/arthur/demo/take.cljs`, which is also the only place the seven stages are
composed in order.
The demo scene itself is `src/arthur/demo/scene.edn`. Both the synthetic take
and real footage use `src/arthur/flow/take.cljs` for the measurement order and
`src/arthur/flow/freeze.cljs` for the landmark-to-channel conversion.
### Real footage (port step 6)
From the repo root, extract a clip, then click **load frames** in the CLJS app:
```sh
./extract.sh /path/to/clip.mov 12
```
This writes `frames/0001.png` onward, `audio.wav`, and `manifest.json` at the
repo root. The manifest supplies the exact frame count, fps and audio path.
Loading detects one face per frame, freezes the measured mouth into channels,
and adds a button for the footage clip. Detection happens once when you load;
playback only resolves channels and paints. Frames without a detection remain
marked absent even though their neighbouring poses are used to condition the
track. The stage stays 320×200 regardless of the footage dimensions.
MediaPipe's JS, wasm and model are under `public/mediapipe/` and served locally.
No CDN is used by this app. See that directory's README for provenance.
Port 8778 is deliberately not 8777. `python3 serve.py` from the repo root still
runs the old JS tool on 8777, and the two are meant to run side by side.