noe/arthur
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Decode uploaded footage in order with WebCodecs

Browse source
This commit is contained in:
Olive Vaughn 2026-09-28 14:18:09 -04:00
parent 131b39bff0
commit 65ad67c129
12 changed files with 454 additions and 364 deletions
Download patch file Download diff file Expand all files Collapse all files

15
frontend/README.md
View file

@ -115,18 +115,18 @@ and real footage use `src/arthur/flow/take.cljs` for the measurement order and
### Real footage
Choose a video in the **footage** file input. The server probes it, re-encodes it
to a browser-seekable H.264 proxy, pulls WAV audio and one tracing JPEG per frame,
to an H.264 proxy and a raw stream of the same coded frames, pulls WAV audio and one tracing JPEG per frame,
then makes the resulting footage selectable. Click **load frames** to detect and
freeze it. Extraction progress is currently read from `/api/extractions/<key>`; a
future WebSocket can push the same job state. The uploaded bytes, extraction job,
and decoded footage have separate records, so the same uploaded video can be
reopened without decoding it again.
**The proxy is what gets measured, and the stills are not.** `flow/ingest` steps
the proxy one frame at a time — seek to `(i + 0.5) / fps`, wait for
`requestVideoFrameCallback`, check the `mediaTime` it reports is the frame that
was asked for — and `flow/detect` hands each frame to MediaPipe in **VIDEO**
running mode at `i * 1000 / fps` milliseconds. That timestamp has to increase
**The proxy is what gets measured, and the stills are not.** `flow/ingest` cuts
its raw H.264 stream into coded frames and decodes them in order with WebCodecs.
The proxy has no B-frames, so decode order matches frame order. `flow/detect`
hands each decoded frame to MediaPipe in **VIDEO** running mode at
`i * 1000 / fps` milliseconds. That timestamp has to increase
strictly and has to be real footage time: video mode is a tracker, it reads the
gap between timestamps as motion, and a repeat leaves the graph in an error state
that every later call re-throws. The JPEGs beside the proxy are reference images
@ -136,7 +136,8 @@ digest.
It is re-encoded even when the upload is already H.264, for two reasons: an
iPhone's HEVC is not decodable in every browser, and the footage's identity is the
proxy's digest — one produced by one ffmpeg invocation, not one that depends on
which branch the source happened to take.
which branch the source happened to take. Its raw stream is copied from that
proxy without another encode.
The command-line route is also available for an existing extracted bundle: