Measure the video, not a PNG per frame
Detection now walks a browser-seekable H.264 proxy in MediaPipe's VIDEO running mode. The PNG sequence it replaces was 112MB for 7.6 seconds at 1440x1920 and 1.1GB at the 900-frame limit; the proxy is 6MB, and landmarks detected off decoded H.264 rather than off the PNGs moved at most 0.0033 of frame width. Three things had to be true for video mode to work, and each was measured against the same footage decoded to PNGs: /blob/<digest> answers byte ranges. Django's FileResponse does no Range handling, and a media element handed 200 with no Accept-Ranges reports an empty `seekable`, no-ops every currentTime write, and detects frame one ninety times without raising. A seek aims at the MIDDLE of its frame. Aiming at i/fps sits on a frame boundary and landed one frame early 31 times in 91; (i + 0.5)/fps was exact on all 91. Timestamps are strictly increasing footage milliseconds. Video mode is a tracker: a repeat leaves the graph in an error state every later call re-throws, so the landmarker is discarded on failure, and passing the frame index instead of i*1000/fps moved landmarks six times further from the per-frame answer. Frames are verified rather than trusted. requestVideoFrameCallback states which frame it handed over, the walker discards any other and fails loudly if the one it asked for never arrives — a stale presentation from the tail of a previous seek is what produced "asked for frame 1 and it presented frame 2" on a video whose seeks were in fact exact. The proxy is re-encoded even when the upload is already H.264: HEVC is not decodable everywhere, and footage identity is the proxy's digest. The JPEG stills beside it are tracing references, outside the footage digest because re-rendering them at another size is not different footage. Verified end to end in a real browser against real footage: 228/228 frames detected, a drawn roto face, 37 backend and 234 frontend tests green. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
parent
686f897401
commit
83d106bbc5
14 changed files with 748 additions and 142 deletions
|
|
@ -1,5 +1,5 @@
|
|||
(ns arthur.flow.ingest-test
|
||||
(:require [cljs.test :refer [deftest is]]
|
||||
(:require [cljs.test :refer [deftest is testing]]
|
||||
[arthur.flow.ingest :as ingest]))
|
||||
|
||||
(deftest feature-absence-uses-the-source-frame-numbers
|
||||
|
|
@ -13,3 +13,52 @@
|
|||
{:eye-r [[6 9]]}
|
||||
{:eye-r [[5 3]]}]]
|
||||
(is (thrown? ExceptionInfo (ingest/feature-presence 8 bad)))))
|
||||
|
||||
;; ---------------------------------------------------------------------------
|
||||
;; walking the proxy
|
||||
;;
|
||||
;; These three are arithmetic, and all three were wrong in a way that no error
|
||||
;; reported. A seek that lands one frame early, a timestamp read back with the
|
||||
;; wrong inverse, or a frame time given as an index all produce a take that runs
|
||||
;; to completion and is quietly off — which is why the numbers are asserted here
|
||||
;; rather than left inline where they read as obvious.
|
||||
|
||||
(deftest a-seek-aims-at-the-middle-of-its-frame
|
||||
;; THE BUG THIS ENCODES: seeking to `i/fps` sits exactly on the boundary between
|
||||
;; two frames, and over 91 frames of real footage it produced the PREVIOUS frame
|
||||
;; 31 times. Every seek must land strictly inside its own frame's interval, with
|
||||
;; half a frame of slack on each side.
|
||||
(doseq [fps [12 23.976 24 25 29.97 30 60]]
|
||||
(testing (str fps "fps")
|
||||
(doseq [i [0 1 2 41 90 899]]
|
||||
(let [t (ingest/seek-time fps i)]
|
||||
(is (< (/ i fps) t (/ (inc i) fps))
|
||||
(str "frame " i " at " fps "fps seeks to " t
|
||||
", outside [" (/ i fps) ", " (/ (inc i) fps) ")"))
|
||||
;; Within a rounding error of dead centre. Not `=`: `(/ i fps)` and
|
||||
;; `(/ (+ i 0.5) fps)` are two different divisions, so their difference
|
||||
;; carries the last bit of both and 0.4999999999999982 is a pass.
|
||||
(is (< (abs (- 0.5 (* fps (- t (/ i fps))))) 1e-9)
|
||||
"the aim point drifted off the middle of the frame"))))))
|
||||
|
||||
(deftest a-presented-media-time-reads-back-as-its-own-frame
|
||||
;; The inverse of `i/fps` and NOT of `seek-time`: requestVideoFrameCallback
|
||||
;; reports the frame's START. Rounding the wrong one is a silent off-by-one
|
||||
;; between the landmarks and the audio.
|
||||
(doseq [fps [12 24 29.97 30]]
|
||||
(doseq [i [0 1 2 41 90]]
|
||||
(is (= i (ingest/presented-frame fps (/ i fps)))
|
||||
(str "frame " i " at " fps "fps did not read back as itself")))))
|
||||
|
||||
(deftest a-frame-time-is-footage-milliseconds-and-not-a-frame-number
|
||||
;; MediaPipe's video mode is a tracker that reads the gap between timestamps as
|
||||
;; motion. Handing it `i` still runs, and measurably moved landmarks six times
|
||||
;; further from the per-frame answer. It must be real elapsed time.
|
||||
(is (= 0.0 (ingest/frame-ms 12 0)))
|
||||
(is (= 1000.0 (ingest/frame-ms 12 12)))
|
||||
(is (= (/ 1000 30) (ingest/frame-ms 30 1)))
|
||||
(testing "strictly increasing, which the input stream requires"
|
||||
(doseq [fps [12 23.976 30 60 240]]
|
||||
(let [times (mapv #(ingest/frame-ms fps %) (range 200))]
|
||||
(is (every? pos? (map - (rest times) times))
|
||||
(str "two frames at " fps "fps shared a timestamp"))))))
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue