noe/arthur
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Stop refusing ordinary phone footage as variable-frame-rate

Browse source 
Uploading a clip shot straight from the iPhone camera app failed with
"variable-frame-rate video needs timestamp-aware playback". The file was not
variable: its container reports avg_frame_rate 8670/299 and nb_frames 289 over
a stream whose decoded timestamps are 280 frames exactly 1/30s apart. The
guard compared two pieces of container metadata and rejected CFR video on the
strength of a summary the container had got wrong about its own contents.

The guard was also obsolete. It dates from when the page measured the source's
own frames, where a wandering frame duration really does break
`frame = floor(t * fps)`. Nothing measures the source now — ffmpeg resamples it
onto a constant rate and the proxy is re-probed after it is written — so
variable input is a thing this converts rather than a thing it refuses.

So: probe picks a rate instead of validating one. It takes the nominal rate,
which is the rate every timestamp in the stream can be expressed at and so the
one that keeps every distinct source frame, and carries it as an exact fraction
because 30000/1001 is not a float and a rounded -r is how a long take drifts.
The disagreement is still recorded as `vfr`, just not fatal.

The frame-count cross-check went with it. It compared the proxy against the
source's nb_frames, which is the number this whole bug proves can lie, and a
resample to a constant rate legitimately changes the count. It now checks the
proxy's DURATION against the source's, because what must not drift is how long
the picture lasts against how long the audio lasts.

Verified on the reported file: 280 frames at 30fps, picture 9.3333s against
audio 9.3167s — half a frame — and 280/280 detected in the real app. 41 backend
tests green, including a genuinely variable fixture end to end.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Olive Vaughn 2026-09-28 12:37:21 -04:00
parent 83d106bbc5
commit 44976cbb4b
2 changed files with 132 additions and 16 deletions
Download patch file Download diff file Expand all files Collapse all files

73
clips/extraction.py
View file

@ -101,9 +101,10 @@ def _encode_proxy(job, source_path, proxy_path, facts, root):
_run_with_progress(
job,
["-i", str(source_path), "-an",
# Constant frame rate at the source's own rate. `probe` has already
# refused VFR, so this asserts that rather than resampling.
"-fps_mode", "cfr", "-r", str(facts["fps"]),
# Constant frame rate at the rate `probe` chose. This RESAMPLES rather
# than asserts: the upload is allowed to be variable, and this is the
# step that makes the thing the page measures not be.
"-fps_mode", "cfr", "-r", facts.get("rate") or str(facts["fps"]),
"-c:v", "libx264", "-preset", "veryfast", "-crf", PROXY_CRF,
# yuv420p and an even frame size are what makes this playable everywhere
# rather than only in the browser that happened to be tested.
@ -123,7 +124,33 @@ def _extract_stills(job, proxy_path, frames_dir, frames, root):
root, "stills", frames, (55, 85))
MAX_RATE = 120 # a capture rate; past this the container is describing something else
def probe(path):
"""What the upload is, as far as choosing a proxy rate goes.
IT NO LONGER REFUSES VARIABLE-FRAME-RATE INPUT, and the reason is the proxy.
That refusal was written when the page measured the source's own frames, where
a wandering frame duration really does break `frame = floor(t * fps)`. Nothing
measures the source now: ffmpeg resamples it onto a constant rate, and the
proxy — constant by construction, and re-probed after it is written — is the
only timeline anything downstream sees.
Keeping the check would have been worse than useless, because the thing it
tested is not reliable. Ordinary iPhone footage, shot straight from the camera
app, reports `avg_frame_rate` 8670/299 and `nb_frames` 289 on a stream whose
decoded timestamps are 280 frames exactly 1/30s apart. The container's summary
of itself disagreed with the container's own contents, so the guard rejected
CFR video for being variable.
THE RATE IS THE NOMINAL ONE. `r_frame_rate` is the rate every timestamp in the
stream can be expressed at, which is the rate that keeps every distinct source
frame; resampling to the average would drop some. Duration is preserved either
way — ffmpeg's CFR conversion is driven by timestamps, so the audio stays in
sync at any rate — so this trades a possible duplicated frame against a
certainly lost one.
"""
data = json.loads(_command(["ffprobe", "-v", "error", "-show_streams",
"-show_format", "-of", "json", str(path)]))
video = next((s for s in data.get("streams", []) if s.get("codec_type") == "video"), None)
@ -131,22 +158,29 @@ def probe(path):
raise ValueError("the uploaded file has no video stream")
nominal = Fraction(video.get("r_frame_rate") or "0")
average = Fraction(video.get("avg_frame_rate") or "0")
if nominal <= 0 or average <= 0:
if nominal <= 0 and average <= 0:
raise ValueError("the video's frame rate is unknown")
vfr = abs(float(nominal / average) - 1) > 0.001
if vfr:
raise ValueError("variable-frame-rate video needs timestamp-aware playback")
frames = video.get("nb_frames")
rate = nominal if 0 < nominal <= MAX_RATE else average
if not 0 < rate <= MAX_RATE:
raise ValueError(f"the video reports a frame rate of {float(rate):g}, which is "
"not a rate footage can be measured at")
duration = float(data.get("format", {}).get("duration") or 0)
if ((frames and frames.isdigit() and int(frames) > 900)
or (duration > 0 and duration * float(average) > 901)):
if duration > 0 and duration * float(rate) > 901:
raise ValueError("video is longer than the 900-frame footage limit")
return {"fps": float(average), "nominal_fps": float(nominal),
frames = video.get("nb_frames")
return {"fps": float(rate),
# The exact rate, for ffmpeg. 30000/1001 is not a float, and handing
# `-r` a rounded one is how a long take drifts out of sync.
"rate": f"{rate.numerator}/{rate.denominator}",
"nominal_fps": float(nominal), "average_fps": float(average),
"width": int(video["width"]), "height": int(video["height"]),
"duration": duration,
# KEPT, AND NO LONGER TRUSTED AS A COUNT. See the docstring: this is
# the container's claim about itself, it is wrong on ordinary phone
# footage, and `run` checks the proxy's DURATION instead.
"reported_frames": int(frames) if frames and frames.isdigit() else None,
"has_audio": any(s.get("codec_type") == "audio" for s in data.get("streams", [])),
"vfr": False}
"vfr": nominal != average}
def count_frames(path):
@ -224,11 +258,18 @@ def run(key):
frames = count_frames(proxy_path)
if not 1 <= frames <= 900:
raise ValueError(f"the proxy holds {frames} frames; the limit is 1–900")
expected = facts.get("reported_frames")
if expected and frames != expected:
# CHECKED AS A DURATION, not as a frame count. The page's clock is
# `frame = floor(audio.currentTime * fps)`, so what must not drift is
# how long the picture lasts against how long the audio lasts — and
# the source's own frame count is a number this has already caught
# lying. A resample to a constant rate legitimately changes the count
# and must not change the duration.
drift = abs(frames / proxy_facts["fps"] - facts["duration"])
if facts["duration"] > 0 and drift > 0.5:
raise ValueError(
f"the proxy holds {frames} frames and the upload reports {expected}; "
"refusing footage whose picture and audio would drift")
f"the proxy runs {frames / proxy_facts['fps']:.2f}s and the upload "
f"runs {facts['duration']:.2f}s; refusing footage whose picture and "
"audio would drift")
proxy_facts["frames"] = frames
frames_dir = root / "stills"