Uploading a clip shot straight from the iPhone camera app failed with
"variable-frame-rate video needs timestamp-aware playback". The file was not
variable: its container reports avg_frame_rate 8670/299 and nb_frames 289 over
a stream whose decoded timestamps are 280 frames exactly 1/30s apart. The
guard compared two pieces of container metadata and rejected CFR video on the
strength of a summary the container had got wrong about its own contents.
The guard was also obsolete. It dates from when the page measured the source's
own frames, where a wandering frame duration really does break
`frame = floor(t * fps)`. Nothing measures the source now — ffmpeg resamples it
onto a constant rate and the proxy is re-probed after it is written — so
variable input is a thing this converts rather than a thing it refuses.
So: probe picks a rate instead of validating one. It takes the nominal rate,
which is the rate every timestamp in the stream can be expressed at and so the
one that keeps every distinct source frame, and carries it as an exact fraction
because 30000/1001 is not a float and a rounded -r is how a long take drifts.
The disagreement is still recorded as `vfr`, just not fatal.
The frame-count cross-check went with it. It compared the proxy against the
source's nb_frames, which is the number this whole bug proves can lie, and a
resample to a constant rate legitimately changes the count. It now checks the
proxy's DURATION against the source's, because what must not drift is how long
the picture lasts against how long the audio lasts.
Verified on the reported file: 280 frames at 30fps, picture 9.3333s against
audio 9.3167s — half a frame — and 280/280 detected in the real app. 41 backend
tests green, including a genuinely variable fixture end to end.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Detection now walks a browser-seekable H.264 proxy in MediaPipe's VIDEO
running mode. The PNG sequence it replaces was 112MB for 7.6 seconds at
1440x1920 and 1.1GB at the 900-frame limit; the proxy is 6MB, and landmarks
detected off decoded H.264 rather than off the PNGs moved at most 0.0033 of
frame width.
Three things had to be true for video mode to work, and each was measured
against the same footage decoded to PNGs:
/blob/<digest> answers byte ranges. Django's FileResponse does no Range
handling, and a media element handed 200 with no Accept-Ranges reports an
empty `seekable`, no-ops every currentTime write, and detects frame one
ninety times without raising.
A seek aims at the MIDDLE of its frame. Aiming at i/fps sits on a frame
boundary and landed one frame early 31 times in 91; (i + 0.5)/fps was exact
on all 91.
Timestamps are strictly increasing footage milliseconds. Video mode is a
tracker: a repeat leaves the graph in an error state every later call
re-throws, so the landmarker is discarded on failure, and passing the frame
index instead of i*1000/fps moved landmarks six times further from the
per-frame answer.
Frames are verified rather than trusted. requestVideoFrameCallback states
which frame it handed over, the walker discards any other and fails loudly
if the one it asked for never arrives — a stale presentation from the tail
of a previous seek is what produced "asked for frame 1 and it presented
frame 2" on a video whose seeks were in fact exact.
The proxy is re-encoded even when the upload is already H.264: HEVC is not
decodable everywhere, and footage identity is the proxy's digest. The JPEG
stills beside it are tracing references, outside the footage digest because
re-rendering them at another size is not different footage.
Verified end to end in a real browser against real footage: 228/228 frames
detected, a drawn roto face, 37 backend and 234 frontend tests green.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
