noe/arthur
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Calibrate the video's frame origin instead of assuming it is zero

Browse source 
"the video never presented frame 1; it offered 2" was this function waiting
for a frame it had not asked for a second time. The walk discarded a wrong
frame and re-armed the callback WITHOUT seeking again, and nothing further is
ever presented to a paused element that has not been asked to move — so one
wrong answer starved until the timeout and reported it as the browser refusing.

Underneath that, the wrong answer was not wrong. A container can carry an edit
list, and `currentTime` then counts from the start of the edited presentation
while a frame's `mediaTime` counts from the start of the media. The two differ
by a constant, so the frame at currentTime 0 can honestly report a mediaTime
two frames in. Seeking cannot correct for it in the positive direction: source
frame 0 would have to be found before the start of the video, every attempt
clamps at zero, and the walk offers frame 2 forever.

So the constant is measured once and subtracted. `calibrate!` takes whatever
the browser calls the first frame it shows and makes that the origin, which is
the honest definition anyway — it is what a viewer sees at time zero, and the
audio clock this take plays against starts in the same place.

Calibration also leaves the element on frame 0, so the walk starts holding it.
`frame!` returns immediately for a frame already on screen rather than seeking
to where it already is, which presents nothing and would hang.

Residual error still re-seeks, corrected by exactly the measured miss, and
still fails loudly with every frame that was offered and where it was asked
from, so a next failure is diagnosable in one shot rather than four.

The proxy also drops B-frames now. That removes the edit list at the source
rather than only coping with it, and makes decode order presentation order
should this ever be fed to WebCodecs. 14% larger, and extraction refuses a
proxy whose timeline is shifted so it cannot come back silently. Honest note:
this was my first diagnosis and it did NOT reproduce the failure — both
proxies walk correctly here on hardware decode — so it is hardening, not the
fix.

Verified by forcing the fault: a harness that offsets the reported timeline by
-2, -1, 0, +1, +2 and +5 frames failed on every positive offset before and
recovers on all six now, first attempt. Real app in headed Chrome with Metal
hardware decode: 280/280. 41 backend and 234 frontend tests green.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Olive Vaughn 2026-09-28 12:50:11 -04:00
parent 44976cbb4b
commit 131b39bff0
3 changed files with 204 additions and 75 deletions
Download patch file Download diff file Expand all files Collapse all files

4
frontend/src/arthur/events/footage.cljs
View file

@ -35,7 +35,7 @@
total (:frames manifest)]
(set! (.-width canvas) w)
(set! (.-height canvas) h)
(-> (ingest/video! (ingest/video-url manifest) w h)
(-> (ingest/video! (ingest/video-url manifest) fps w h)
(.then
(fn [video]
(js/Promise.
@ -49,7 +49,7 @@
(-> (ingest/frame! video fps i)
(.then
(fn [_]
(.drawImage ctx video 0 0)
(.drawImage ctx (:el video) 0 0)
(let [face (detect/detect! model canvas
(ingest/frame-ms fps i))
ring (when face (mapv #(nth face %) lm/LIPS-INNER))

241
frontend/src/arthur/flow/ingest.cljs
View file

@ -110,51 +110,6 @@
;; ---------------------------------------------------------------------------
;; walking the proxy, one frame at a time
(def ^:private seek-timeout-ms
"How long one frame may take to arrive before the run gives up.
Long, because the first seek of a take also opens the file and fills a buffer,
and short enough that a video the browser cannot decode fails with a sentence
instead of hanging with a spinner."
10000)
(defn video!
"Load the proxy as a decodable, seekable element.
`preload=auto` and nothing else: the element is never added to the document and
never played. It is a decoder with a seek function, and the only reason it is a
DOM element rather than a `VideoDecoder` is that a `VideoDecoder` needs the
container demuxed before it can be handed a single frame, and this does not."
[src width height]
(js/Promise.
(fn [resolve reject]
(let [video (.createElement js/document "video")]
(set! (.-muted video) true)
(set! (.-playsInline video) true)
(set! (.-preload video) "auto")
(set! (.-crossOrigin video) "anonymous")
(set! (.-onerror video)
(fn [_]
(reject (ex-info (str "the browser could not decode this footage's video"
(when-let [e (.-error video)]
(str " (" (.-message e) ")")))
{:src src}))))
(set! (.-onloadeddata video)
(fn [_]
(cond
(not (fn? (.-requestVideoFrameCallback video)))
(reject (ex-info (str "this browser has no requestVideoFrameCallback, so "
"which frame is on screen cannot be established")
{}))
(not= [(.-videoWidth video) (.-videoHeight video)] [width height])
(reject (ex-info "the video's size disagrees with the footage manifest"
{:manifest [width height]
:video [(.-videoWidth video) (.-videoHeight video)]}))
:else (resolve video))))
(set! (.-src video) src)))))
(defn seek-time
"When to ask the video for source frame `i`: the MIDDLE of the frame, not its
start.
@ -187,45 +142,185 @@
[fps i]
(/ (* i 1000) fps))
(def ^:private seek-timeout-ms
"How long one frame may take to arrive before the run gives up.
Long, because the first seek of a take also opens the file and fills a buffer,
and short enough that a video the browser cannot decode fails with a sentence
instead of hanging with a spinner."
10000)
(defn- await-frame!
"One presentation, with its raw index. `nil` if none arrives in time."
[^js video fps at]
(js/Promise.
(fn [resolve _reject]
(let [settled (volatile! false)
give (fn [v] (when-not @settled (vreset! settled true) (resolve v)))]
(.requestVideoFrameCallback
video (fn [_now metadata]
(give (presented-frame fps (.-mediaTime metadata)))))
(js/setTimeout #(give nil) seek-timeout-ms)
(set! (.-currentTime video) at)))))
(defn calibrate!
"What the browser CALLS the first frame it will show us.
Not always zero, and that is not the browser being wrong. A container can carry
an edit list — ffmpeg writes one to absorb an encoder's reordering delay — and
then `currentTime` counts from the start of the edited presentation while the
`mediaTime` on a frame counts from the start of the media. The two differ by a
constant, and the frame at `currentTime` 0 can honestly report a `mediaTime` of
two frames in.
SO THE CONSTANT IS MEASURED ONCE AND SUBTRACTED, rather than corrected for by
seeking. Seeking cannot fix it: when the offset is positive, source frame 0
would have to be found BEFORE the start of the video, every attempt clamps at
zero, and the walk reports `never presented frame 1; it offered 2` forever. The
first frame the element presents IS frame 0 — it is what a viewer sees at time
zero, and the audio clock this take plays against starts in the same place — so
its own label is the origin everything else is counted from."
[^js video fps]
(-> (await-frame! video fps (seek-time fps 0))
(.then (fn [base]
(when (nil? base)
(throw (ex-info "the video presented no frame at all; it cannot be walked"
{})))
base))))
(defn video!
"Load the proxy as a decodable, seekable element, and find its origin.
`preload=auto` and nothing else: the element is never added to the document and
never played. It is a decoder with a seek function, and the only reason it is a
DOM element rather than a `VideoDecoder` is that a `VideoDecoder` needs the
container demuxed before it can be handed a single frame, and this does not."
[src fps width height]
(-> (js/Promise.
(fn [resolve reject]
(let [video (.createElement js/document "video")]
(set! (.-muted video) true)
(set! (.-playsInline video) true)
(set! (.-preload video) "auto")
(set! (.-crossOrigin video) "anonymous")
(set! (.-onerror video)
(fn [_]
(reject (ex-info (str "the browser could not decode this footage's video"
(when-let [e (.-error video)]
(str " (" (.-message e) ")")))
{:src src}))))
(set! (.-onloadeddata video)
(fn [_]
(cond
(not (fn? (.-requestVideoFrameCallback video)))
(reject (ex-info (str "this browser has no requestVideoFrameCallback, so "
"which frame is on screen cannot be established")
{}))
(not= [(.-videoWidth video) (.-videoHeight video)] [width height])
(reject (ex-info "the video's size disagrees with the footage manifest"
{:manifest [width height]
:video [(.-videoWidth video) (.-videoHeight video)]}))
:else (resolve video))))
(set! (.-src video) src))))
(.then (fn [video]
(-> (calibrate! video fps)
;; Calibration left the element ON frame 0, so the walk starts
;; already holding it. `current` is what is on screen now.
(.then (fn [base] {:el video :base base :current (atom 0)})))))))
(def ^:private seek-attempts
"How many times one frame may be asked for before the run gives up.
More than one because the browser is allowed to disagree with us about where a
frame starts, and few because each attempt corrects by the exact size of the
disagreement — so a constant offset is gone on the second try and anything still
wrong on the sixth is not an offset."
6)
(defn frame!
"Seek to source frame `i` and resolve once the browser has PRESENTED it.
IT WAITS FOR THE FRAME IT ASKED FOR, rather than trusting the first callback.
`requestVideoFrameCallback` is a queue of presentations, not an answer to our
seek: a frame presented while the file was still opening, or the tail of the
previous seek, arrives on the next callback we happen to have registered. Taking
it at face value is what produced `asked the video for frame 1 and it presented
frame 2` on a video whose seeks were in fact exact. So a callback whose
`mediaTime` is not this frame's is DISCARDED and the wait re-armed — the
browser states which frame it handed over, and that is the only frame we let
through.
COUNTED FROM THE CALIBRATED ORIGIN. `base` is what the browser called the first
frame it showed (see `calibrate!`), so the frame we want is the one whose raw
index is `base + i`. Subtracting a measured constant is what makes a container
with an edit list walk the same as one without, and it is the half that seeking
cannot do: when the offset is positive, frame 0 lies before the start of the
video and no amount of re-seeking will reach it.
It still fails loudly. A silent one-frame slip between the landmarks and the
audio is not something anyone finds by looking at the result, so a frame that
never arrives inside `seek-timeout-ms` ends the run and says which one."
[^js video fps i]
(js/Promise.
IT STILL CORRECTS TOWARDS THE FRAME IT WANTS, for whatever the constant does not
explain. A wrong frame is not merely an error to report, it is a MEASUREMENT of
how far off the aim was, and the next attempt shifts by exactly that. Re-seeking
rather than only re-listening is load-bearing: nothing further is ever presented
to a paused video that has not been asked to move, so an earlier version that
re-armed the callback without seeking again starved until its timeout.
It fails loudly rather than accepting a near miss. A one-frame slip between the
landmarks and the audio is not something anyone finds by looking at the result,
so exhausting the attempts ends the run and reports every frame that was offered
and where it was asked from."
[{:keys [^js el base current]} fps i]
(if (= @current i)
;; Already on screen. Seeking to where we already are presents NOTHING — a
;; paused element with an unchanged frame fires no callback — so asking again
;; would wait out the timeout. This is frame 0 straight after `calibrate!`,
;; and it is the difference between a walk that starts and one that hangs.
(js/Promise.resolve el)
(js/Promise.
(fn [resolve reject]
(let [settled (volatile! false)
seen (volatile! [])
offered (volatile! [])
finish (fn [f] (when-not @settled (vreset! settled true) (f)))
duration (or (.-duration el) 0)
describe (fn []
(if (seq @offered)
(str/join ", "
(map (fn [[idx at]]
(str (inc idx) " (asked at " (.toFixed at 4) "s)"))
@offered))
"nothing at all"))
timer (js/setTimeout
#(finish
(fn []
(reject (ex-info (str "the video never presented frame " (inc i)
(when (seq @seen)
(str "; it offered "
(str/join ", " (map inc @seen)))))
{:frame i :offered @seen}))))
seek-timeout-ms)]
(letfn [(listen []
"; it offered " (describe))
{:frame i :base base :offered @offered}))))
seek-timeout-ms)
seek! (fn [at]
;; A repeat of the current position is not a seek and presents
;; nothing, so nudge within the frame rather than stall.
(let [at (min (max at 0) (max 0 (- duration 1e-3)))
at (if (= at (.-currentTime el)) (+ at (/ 0.25 fps)) at)]
(set! (.-currentTime el) at)))]
(letfn [(attempt [n at]
(.requestVideoFrameCallback
video
el
(fn [_now metadata]
(when-not @settled
(let [presented (presented-frame fps (.-mediaTime metadata))]
(if (= presented i)
(do (js/clearTimeout timer) (finish #(resolve video)))
(do (vswap! seen conj presented) (listen))))))))]
(listen))
(set! (.-currentTime video) (seek-time fps i))))))
(let [presented (- (presented-frame fps (.-mediaTime metadata)) base)]
(cond
(= presented i)
(do (js/clearTimeout timer)
(reset! current i)
(finish #(resolve el)))
(< n seek-attempts)
(let [next-at (- at (/ (- presented i) fps))]
(vswap! offered conj [presented at])
(attempt (inc n) next-at)
(seek! next-at))
:else
(do (vswap! offered conj [presented at])
(js/clearTimeout timer)
(finish
(fn []
(reject (ex-info
(str "the video kept presenting the wrong frame for "
(inc i) "; it offered " (describe))
{:frame i :base base :offered @offered})))))))))))]
(let [at (seek-time fps i)]
(attempt 1 at)
(seek! at))))))))