noe/arthur
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Measure the video, not a PNG per frame

Browse source 
Detection now walks a browser-seekable H.264 proxy in MediaPipe's VIDEO
running mode. The PNG sequence it replaces was 112MB for 7.6 seconds at
1440x1920 and 1.1GB at the 900-frame limit; the proxy is 6MB, and landmarks
detected off decoded H.264 rather than off the PNGs moved at most 0.0033 of
frame width.

Three things had to be true for video mode to work, and each was measured
against the same footage decoded to PNGs:

/blob/<digest> answers byte ranges. Django's FileResponse does no Range
handling, and a media element handed 200 with no Accept-Ranges reports an
empty `seekable`, no-ops every currentTime write, and detects frame one
ninety times without raising.

A seek aims at the MIDDLE of its frame. Aiming at i/fps sits on a frame
boundary and landed one frame early 31 times in 91; (i + 0.5)/fps was exact
on all 91.

Timestamps are strictly increasing footage milliseconds. Video mode is a
tracker: a repeat leaves the graph in an error state every later call
re-throws, so the landmarker is discarded on failure, and passing the frame
index instead of i*1000/fps moved landmarks six times further from the
per-frame answer.

Frames are verified rather than trusted. requestVideoFrameCallback states
which frame it handed over, the walker discards any other and fails loudly
if the one it asked for never arrives — a stale presentation from the tail
of a previous seek is what produced "asked for frame 1 and it presented
frame 2" on a video whose seeks were in fact exact.

The proxy is re-encoded even when the upload is already H.264: HEVC is not
decodable everywhere, and footage identity is the proxy's digest. The JPEG
stills beside it are tracing references, outside the footage digest because
re-rendering them at another size is not different footage.

Verified end to end in a real browser against real footage: 228/228 frames
detected, a drawn roto face, 37 backend and 234 frontend tests green.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Olive Vaughn 2026-09-28 11:32:01 -04:00
parent 686f897401
commit 83d106bbc5
14 changed files with 748 additions and 142 deletions
Download patch file Download diff file Expand all files Collapse all files

29
frontend/README.md
View file

@ -114,12 +114,29 @@ and real footage use `src/arthur/flow/take.cljs` for the measurement order and
### Real footage
Choose a video in the **footage** file input. The server probes it, extracts one
PNG per source frame and WAV audio, then makes the resulting footage selectable.
Click **load frames** to detect and freeze it. Extraction progress is currently
read from `/api/extractions/<key>`; a future WebSocket can push the same job state.
The uploaded bytes, extraction job, and decoded footage have separate records, so
the same uploaded video can be reopened without decoding it again.
Choose a video in the **footage** file input. The server probes it, re-encodes it
to a browser-seekable H.264 proxy, pulls WAV audio and one tracing JPEG per frame,
then makes the resulting footage selectable. Click **load frames** to detect and
freeze it. Extraction progress is currently read from `/api/extractions/<key>`; a
future WebSocket can push the same job state. The uploaded bytes, extraction job,
and decoded footage have separate records, so the same uploaded video can be
reopened without decoding it again.
**The proxy is what gets measured, and the stills are not.** `flow/ingest` steps
the proxy one frame at a time — seek to `(i + 0.5) / fps`, wait for
`requestVideoFrameCallback`, check the `mediaTime` it reports is the frame that
was asked for — and `flow/detect` hands each frame to MediaPipe in **VIDEO**
running mode at `i * 1000 / fps` milliseconds. That timestamp has to increase
strictly and has to be real footage time: video mode is a tracker, it reads the
gap between timestamps as motion, and a repeat leaves the graph in an error state
that every later call re-throws. The JPEGs beside the proxy are reference images
for the tracing editor and nothing measures them, so they are not in the footage
digest.
It is re-encoded even when the upload is already H.264, for two reasons: an
iPhone's HEVC is not decodable in every browser, and the footage's identity is the
proxy's digest — one produced by one ffmpeg invocation, not one that depends on
which branch the source happened to take.
The command-line route is also available for an existing extracted bundle: