diff --git a/docs/architecture.md b/docs/architecture.md index a992210..175c736 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -151,6 +151,16 @@ source pixel.** Stage 4 is separated from 3 only because `contour avg` and `anchor avg` are knobs and the rest of stage 3 is not; splitting them means dragging that slider does not re-run the interior extraction. +That last clause is the guarantee, and it is narrower than the table's ordering +looks. Stage 3 is not one pass that finishes before stage 4 begins. The anchor fit +is knob-free; conditioning smooths its four parameters; the head-local rings are +then measured *through* the conditioned transform — so `anchor avg` does re-run +the ring mapping, which is a few hundred frames of twenty points and free. What it +must not re-run is the part that reads a source pixel, and that part takes the +landmarks and the frames and never the transform, so it does not. Built as +`measure/anchor` → `condition/anchor` → `measure/mouth` → `condition/contours`, +and stage 7's eyes, brows and interior follow the same shape. + ### Where the current code lands | Now | Stage | diff --git a/frontend/README.md b/frontend/README.md index c2d6cfd..fc9677d 100644 --- a/frontend/README.md +++ b/frontend/README.md @@ -72,7 +72,12 @@ From step 9 Django serves the page and `:dev-http` goes away. `js/` is the numeric oracle, not dead weight. `test/parity/` runs both implementations on the same synthetic track and diffs them: `fit-similarity` and -`procrustes-mean` agree to 1e-9, the raster pixel-for-pixel. +`procrustes-mean` agree to 1e-9, the raster pixel-for-pixel, and `stabilize`'s +whole output agrees across the three namespaces it was split into. + +The oracle drives `stabilize` at aspect 0.5625 as well as at 1. Aspect 1 makes the +anisotropy correction the identity, so a port that dropped it entirely would pass +— which is the one thing a parity suite on normalised landmarks can be blind to. Both sides get the identical track because `js/synth.js` reads `Math.random` at call time, so `oracle.mjs` stubs it to a constant and the CLJS side passes @@ -86,6 +91,7 @@ the prototype's mistakes into the rewrite and make them permanent. ``` src/arthur/domain/ pure. No re-frame, no DOM, no flow/. +src/arthur/flow/ the stages. `(f params inputs) -> output`, no state. src/arthur/demo.cljs the hand-written scene, read from demo/scene.edn src/arthur/ui/canvas.cljs the one imperative sink — the only DOM canvas call test/arthur/synth.cljs the synthetic track — test infrastructure, not src diff --git a/frontend/src/arthur/flow/condition.cljs b/frontend/src/arthur/flow/condition.cljs new file mode 100644 index 0000000..658c85d --- /dev/null +++ b/frontend/src/arthur/flow/condition.cljs @@ -0,0 +1,50 @@ +(ns arthur.flow.condition + "Stage 4: the two smoothing knobs, and nothing else. + + It is a stage of its own for exactly one reason. `anchor avg` and `contour avg` + are knobs and the rest of measure is not, so dragging either must not re-run the + interior extraction — the one part of measure that reads a source pixel, and the + only part that costs seconds. + + Neither function knows what it is smoothing. `anchor` smooths four transform + parameters and `contours` smooths a ring track per vertex; a face appears + nowhere in here." + (:require [arthur.domain.geom :as geom])) + +(defn anchor + "Smooth the anchor fit's four parameters. `arthur.domain.geom/smooth-transforms` + says why it is the transform and not the contour. + + Returns the measured map with `:transforms` replaced, so the anchor keeps + travelling as one value and nothing downstream has to know whether it has been + conditioned yet. `:residual` is deliberately left alone: it is the residual of + the FIT, and it is not a function of this knob." + [{:keys [anchor-avg]} anchored] + (update anchored :transforms geom/smooth-transforms anchor-avg)) + +(defn contours + "Temporal smoothing of a ring track, per vertex, across time. + + docs/design.md says to smooth the transform and never the contour. That was + correct while keys were sparse: sampling at velocity minima rejected per-frame + detector noise for free. With a key on every frame the noise is visible as a + shimmer along the lip edge, so a bounded exception applies - the window must + stay SHORTER than the shortest articulation worth keeping. At 12fps, mouth + movement spans 3-6 frames and detector noise is per-frame, so a radius of 1 + separates them and a radius of 3 would start eating speech. + + `contour-avg` is in frames either side: 0 off, 1 = 3-frame average, 2 = 5-frame. + It is the same clamped window `geom/moving-average` gives the transform + parameters — reused rather than re-derived, so \"radius 2\" cannot come to mean + two different things at the two knobs." + [{:keys [contour-avg]} rings] + (if (<= contour-avg 0) + (vec rings) + (let [rings (vec rings) + axis (fn [v k] (geom/moving-average (map #(k (nth % v)) rings) contour-avg)) + ;; Transposed once into a per-vertex pair of series, because the + ;; smoothing is along time and the storage is along vertices. + axes (mapv (fn [v] [(axis v :x) (axis v :y)]) + (range (count (first rings))))] + (mapv (fn [t] (mapv (fn [[xs ys]] {:x (nth xs t) :y (nth ys t)}) axes)) + (range (count rings)))))) diff --git a/frontend/src/arthur/flow/measure/anchor.cljs b/frontend/src/arthur/flow/measure/anchor.cljs new file mode 100644 index 0000000..e3a311b --- /dev/null +++ b/frontend/src/arthur/flow/measure/anchor.cljs @@ -0,0 +1,64 @@ +(ns arthur.flow.measure.anchor + "Stage 3, the anchor: the rigid transform per frame, and the space every other + measurement is taken in. + + The fit is knob-free, deliberately. Smoothing its four parameters is stage 4 — + `arthur.flow.condition` — because `anchor avg` is a knob and the rest of measure + is not, and because a residual that moved when a smoothing slider moved would + report the footage as unstabilisable on account of a setting. + + `makeXform` is NOT here, and is not being ported. It centres on the face oval's + bounding box and zooms until the face is 80% of the raster height, so every + vertex it touches carries a cropping decision made once, at analysis time, from + one frame's landmarks. Geometry is stored in the node's own local space and the + framing is a transform on a node; see \"What space geometry is in\" in + docs/animation-model.md. So the face oval is not measured here either — its only + consumers in the prototype were that transform and the placeholder plate + outline, and the plate outline belongs to painting." + (:require [arthur.domain.geom :as geom] + [arthur.domain.landmarks :as lm])) + +(defn pick + "Landmarks `idx` out of one dense `frame`, converted to an ISOTROPIC space. + + MediaPipe normalises x by image WIDTH and y by image HEIGHT, so its normalised + space is anisotropic: for a 1080x1920 frame, one unit of x is 1080px and one + unit of y is 1920px. Treating those as comparable stretches everything + horizontally by H/W, and worse, makes fit-similarity fit a \"rotation\" in a + sheared space, so head roll comes out subtly wrong as well. + + Multiplying x by aspect = W/H converts to an isotropic space whose unit is one + image height, so equal numbers mean equal pixels. Everything downstream - + Procrustes, the similarity fit, the raster transform - depends on that." + [frame idx aspect] + (mapv (fn [i] (let [p (nth frame i)] {:x (* (:x p) aspect) :y (:y p)})) idx)) + +(defn residuals + "RMS misfit per frame, in the isotropic space's units — one image height. + + Taken against the transforms it is HANDED rather than against a fit of its own, + so the same function serves the stage-3 reading and the parity diff. The + prototype took it against the SMOOTHED transforms, which folds the smoothing + error into a number whose whole job is to say whether the footage is + stabilisable at all; `fit` takes it against the raw fit instead." + [ref rigid tfs] + (mapv (fn [rig tf] (geom/fit-residual tf rig ref)) rigid tfs)) + +(defn fit + "Dense landmarks -> the rigid fit of every frame onto the shot's mean pose. + + The reference is the Procrustes MEAN configuration over the shot, not frame + zero, so no single frame's idiosyncrasies get baked into every other frame." + [{:keys [aspect]} {:keys [dense]}] + (let [rigid (mapv #(pick % lm/RIGID aspect) dense) + ref (geom/procrustes-mean rigid) + tfs (mapv #(geom/fit-similarity % ref) rigid)] + {:ref ref + ;; Rigid landmarks in IMAGE space: the head-pose signal. Frame removal is + ;; decided from head motion, not from the mouth, so this has to survive the + ;; fit rather than being consumed by it. + :rigid rigid + :transforms tfs + ;; Residual rises with out-of-plane rotation, which no 2D similarity can + ;; remove. High values mean this section wants a different head plate. + :residual (residuals ref rigid tfs)})) diff --git a/frontend/src/arthur/flow/measure/mouth.cljs b/frontend/src/arthur/flow/measure/mouth.cljs new file mode 100644 index 0000000..0e72c7a --- /dev/null +++ b/frontend/src/arthur/flow/measure/mouth.cljs @@ -0,0 +1,39 @@ +(ns arthur.flow.measure.mouth + "Stage 3, the mouth: the lip rings with the head's motion taken out, and the + aperture that decides whether there is an interior at all. + + Head-local means the anchor's space, so `pick` is read from + `arthur.flow.measure.anchor` rather than written a second time. A ring measured + in a different space from the fit that placed it is not a failure anyone would + see — it is a mouth that is quietly the wrong width. + + The rings keep every slot of their table. A vertex budget is a stage-5 knob, and + subsampling is a per-slot pick while stage 4's contour average is a per-slot + average over time, so the two commute: smooth-then-subsample and + subsample-then-smooth are the same numbers. That is what lets the vertex knob + sit downstream of the smoothing knob instead of alongside it, and mouth-test + asserts it rather than leaving it to look obvious. + + Turning the aperture into `[:vis]` on `:mouth-in` is stage 5. This reports the + measurement, not the decision." + (:require [arthur.domain.geom :as geom] + [arthur.domain.landmarks :as lm] + [arthur.flow.measure.anchor :as anchor])) + +(defn measure + "Dense landmarks plus the anchor's transforms -> head-local lip rings. + + `transforms` is whatever the caller has: the raw fit, or — normally — stage 4's + conditioned one. Which it is belongs to the caller, because \"smooth the + transform, never the contour\" only means anything while the two are separate." + [{:keys [aspect]} {:keys [dense transforms]}] + (let [local (fn [table] + (mapv (fn [tf frame] (geom/apply-sim-all tf (anchor/pick frame table aspect))) + transforms dense))] + {:outer (local lm/LIPS-OUTER) + :inner (local lm/LIPS-INNER) + ;; Not a separate measurement: APERTURE is slots 5 and 15 of LIPS_INNER, so + ;; this is the inner ring's own height read off as a scalar. Writing those + ;; two landmarks a second time is what gave the synthetic mouth a bowtie. + :aperture (mapv (fn [[a b]] (js/Math.hypot (- (:x a) (:x b)) (- (:y a) (:y b)))) + (local lm/APERTURE))})) diff --git a/frontend/test/arthur/flow/condition_test.cljs b/frontend/test/arthur/flow/condition_test.cljs new file mode 100644 index 0000000..d57231d --- /dev/null +++ b/frontend/test/arthur/flow/condition_test.cljs @@ -0,0 +1,73 @@ +(ns arthur.flow.condition-test + (:require [cljs.test :refer [deftest is testing]] + [arthur.domain.ring :as ring] + [arthur.flow.condition :as condition] + [arthur.flow.measure.anchor :as anchor] + [arthur.flow.measure.mouth :as mouth] + [arthur.synth :as synth])) + +;; Jitter ON here, unlike the measure tests: a low-pass with nothing to remove +;; cannot be shown to remove anything. +(def noisy (delay (synth/synth-dense 72))) + +(def fitted (delay (anchor/fit {:aspect 1} {:dense @noisy}))) + +(def rings + (delay (:outer (mouth/measure {:aspect 1} + {:dense @noisy :transforms (:transforms @fitted)})))) + +(defn- hf + "Mean |second difference| per vertex over time: the high-frequency energy a + low-pass is supposed to remove." + [rs] + (let [n (count rs) v (count (first rs)) + at (fn [t i k] (k (nth (nth rs t) i)))] + (/ (reduce + (for [t (range 1 (dec n)) i (range v) k [:x :y]] + (abs (+ (- (at (inc t) i k) (* 2 (at t i k))) (at (dec t) i k))))) + (* (- n 2) v 2)))) + +(deftest radius-zero-is-the-identity + ;; "Off" has to be off. A knob whose zero still rewrote the numbers would make + ;; every parity diff below it untrustworthy. + (is (= (:outer (mouth/measure {:aspect 1} {:dense @noisy :transforms (:transforms @fitted)})) + (condition/contours {:contour-avg 0} @rings))) + (is (vector? (condition/contours {:contour-avg 0} @rings)) + "callers index into this, so it may not come back a lazy seq")) + +(deftest the-anchor-knob-touches-only-the-transforms + ;; `:residual` is the residual of the FIT and is not a function of this knob. + ;; The prototype recomputed it from the smoothed transforms, which made "is this + ;; section stabilisable" move when a smoothing slider moved. + (let [conditioned (condition/anchor {:anchor-avg 5} @fitted)] + (is (= (:ref @fitted) (:ref conditioned))) + (is (= (:rigid @fitted) (:rigid conditioned))) + (is (= (:residual @fitted) (:residual conditioned))) + (is (not= (:transforms @fitted) (:transforms conditioned)) + "...and it did smooth something"))) + +(deftest the-contour-average-is-a-low-pass-along-time + (let [es (mapv #(hf (condition/contours {:contour-avg %} @rings)) [0 1 2 3 5])] + (is (apply >= es) (str "hf energy by radius: " (pr-str es))) + (is (< (nth es 1) (* 0.8 (first es))) + (str "radius 1 removed only " (- 1 (/ (nth es 1) (first es))) " of it")))) + +(deftest a-steady-contour-comes-back-unchanged + (let [r (nth @rings 0) + rs (vec (repeat 9 r))] + (is (< (reduce max (for [f (condition/contours {:contour-avg 3} rs) + [p q] (map vector f r)] + (js/Math.hypot (- (:x p) (:x q)) (- (:y p) (:y q))))) + 1e-15)))) + +(deftest smoothing-cannot-make-a-ring-self-intersect + ;; Averaging across frames is averaging across SHAPES, so in principle it could + ;; fold a ring over itself — and a self-intersecting ring renders as blocks + ;; meeting at corners rather than as an error. Unlikely on a lip contour and + ;; cheap to rule out, which is the same reason domain/ring asserts simplicity on + ;; the raw track. + (doseq [radius [1 2 3 5]] + (let [bad (first (for [[f r] (map-indexed vector (condition/contours {:contour-avg radius} @rings)) + :let [hits (ring/self-intersections r)] + :when (seq hits)] + {:radius radius :frame f :edges (first hits)}))] + (is (nil? bad) (str "self-intersection: " (pr-str bad)))))) diff --git a/frontend/test/arthur/flow/measure/anchor_test.cljs b/frontend/test/arthur/flow/measure/anchor_test.cljs new file mode 100644 index 0000000..955936e --- /dev/null +++ b/frontend/test/arthur/flow/measure/anchor_test.cljs @@ -0,0 +1,82 @@ +(ns arthur.flow.measure.anchor-test + "The anchor is the only stage with genuine ground truth available, and that is + why the synthetic track exists: it moves the head by a KNOWN similarity, so + \"did the fit remove the head motion\" is a number and not an impression. Real + footage gives nothing to compare against. + + One thing cannot be asserted here, and it is worth knowing before trying. The + synth's head is PERFECTLY rigid — the jitter is a whole-head translation, which + a similarity absorbs exactly — so every frame's rigid configuration is congruent + with frame zero's, and the Procrustes mean is therefore frame zero's + configuration to 1e-15 on this track, jitter or no jitter. \"The reference is the + mean and not frame zero\" has to be asserted where the two can differ, which is + arthur.domain.geom-test." + (:require [cljs.test :refer [deftest is testing]] + [arthur.domain.geom :as geom] + [arthur.flow.measure.anchor :as anchor] + [arthur.synth :as synth])) + +;; Jitter of exactly zero, so the synth's head motion IS a similarity and the fit +;; has to recover it exactly. With jitter there is no exact answer to assert. +(def clean (delay (synth/synth-dense 72 {:rand-fn (constantly 0.5)}))) + +(defn- worst [f xs] (reduce max (map f xs))) + +(deftest the-fit-removes-a-known-head-motion + ;; Drift, sway, roll and a slow scale change, all four gone: every frame's + ;; rigid configuration lands on the reference. This is the whole claim of + ;; stage 3 and it is either exact or the fit is wrong. + (let [{:keys [ref rigid transforms residual]} (anchor/fit {:aspect 1} {:dense @clean}) + gap (worst (fn [[tf rig]] + (worst (fn [[m r]] (js/Math.hypot (- (:x m) (:x r)) (- (:y m) (:y r)))) + (map vector (geom/apply-sim-all tf rig) ref))) + (map vector transforms rigid))] + (is (< gap 1e-9) (str "worst stabilised-vs-reference gap " gap)) + (is (< (reduce max residual) 1e-9) + (str "worst residual " (reduce max residual) + " — a pure similarity has to fit at zero")) + ;; And the motion was really there to remove, or the above is vacuous. + (is (> (- (reduce max (map :tx transforms)) (reduce min (map :tx transforms))) 0.05)) + (is (> (- (reduce max (map :theta transforms)) (reduce min (map :theta transforms))) 0.2)))) + +(deftest the-aspect-correction-recovers-roll + ;; MediaPipe divides x by WIDTH and y by HEIGHT, so a 1080x1920 clip arrives + ;; with x stretched by H/W. Simulated here by dividing x back out: `pick` with + ;; the right aspect must undo it exactly, and treating the space as isotropic + ;; must not — a "similarity" fitted in a sheared space is not one, and the error + ;; lands on roll, where it looks like a directed choice rather than a bug. + (let [aspect 0.5625 + squashed (mapv (fn [fr] (mapv #(update % :x / aspect) fr)) @clean) + truth (anchor/fit {:aspect 1} {:dense @clean}) + fixed (anchor/fit {:aspect aspect} {:dense squashed}) + naive (anchor/fit {:aspect 1} {:dense squashed}) + d-theta (fn [a] (worst abs (map - (map :theta (:transforms a)) + (map :theta (:transforms truth)))))] + (is (< (d-theta fixed) 1e-9) + (str "corrected roll is off by " (d-theta fixed))) + (is (> (d-theta naive) 0.01) + (str "uncorrected roll is off by only " (d-theta naive) + " — if this is small the correction is not being exercised")) + ;; The residual notices too, which is what makes a forgotten aspect visible + ;; at the readout rather than only in the output. + (is (< (reduce max (:residual fixed)) 1e-9)) + (is (> (reduce max (:residual naive)) 1e-4)))) + +(deftest out-of-plane-motion-shows-up-in-the-residual + ;; No 2D similarity can remove yaw or pitch, and that is the point of reporting + ;; the residual: high values mean the section wants a different head plate. + ;; A yaw foreshortens x and leaves y alone, which is exactly a non-uniform + ;; scale, so squeezing one frame's x is the cheapest honest stand-in. + (let [yawed (update (vec @clean) 10 + (fn [fr] (mapv #(update % :x * 0.97) fr))) + {:keys [residual]} (anchor/fit {:aspect 1} {:dense yawed})] + (is (> (nth residual 10) 1e-4) + (str "the yawed frame's residual is " (nth residual 10))) + ;; Localised, not smeared. Its neighbours are allowed to be non-zero — the one + ;; bad frame pulls the Procrustes mean, so every frame's reference moves a + ;; little — but by more than an order of magnitude less. Which is the argument + ;; for the mean reference stated as a measurement: one bad detection degrades + ;; the whole shot slightly instead of one frame catastrophically. + (is (< (nth residual 9) (* 0.05 (nth residual 10))) + (str "neighbour residual " (nth residual 9) " is " + (/ (nth residual 9) (nth residual 10)) " of the yawed frame's")))) diff --git a/frontend/test/arthur/flow/measure/mouth_test.cljs b/frontend/test/arthur/flow/measure/mouth_test.cljs new file mode 100644 index 0000000..4fcf7d1 --- /dev/null +++ b/frontend/test/arthur/flow/measure/mouth_test.cljs @@ -0,0 +1,85 @@ +(ns arthur.flow.measure.mouth-test + (:require [cljs.test :refer [deftest is testing]] + [arthur.domain.landmarks :as lm] + [arthur.domain.ring :as ring] + [arthur.flow.condition :as condition] + [arthur.flow.measure.anchor :as anchor] + [arthur.flow.measure.mouth :as mouth] + [arthur.synth :as synth])) + +(def clean (delay (synth/synth-dense 72 {:rand-fn (constantly 0.5)}))) + +(def measured + (delay + (let [fitted (anchor/fit {:aspect 1} {:dense @clean})] + (mouth/measure {:aspect 1} {:dense @clean :transforms (:transforms fitted)})))) + +(defn- spread + "Largest distance between any two of `rings` at the same slot: how much the + shape moved over the frames given." + [rings] + (reduce max (for [a rings b rings [p q] (map vector a b)] + (js/Math.hypot (- (:x p) (:x q)) (- (:y p) (:y q)))))) + +;; The synth holds each mouth pose for nine frames, so frames 0-8 are one pose +;; carried around by a moving, rolling, scaling head. That is the case the whole +;; stage exists for. +(def ^:private hold (range 0 9)) + +(deftest a-held-pose-holds-still-once-the-head-is-taken-out + (let [local (spread (map #(nth (:outer @measured) %) hold)) + image (spread (map (fn [f] (anchor/pick (nth @clean f) lm/LIPS-OUTER 1)) hold))] + (is (< local 1e-9) + (str "head-local outer ring moved " local " over a held pose")) + ;; And it moved plenty in the footage, or the line above is measuring nothing. + (is (> image 0.02) + (str "the ring only moved " image " in image space over the same frames")))) + +(deftest the-aperture-measures-the-mouth-and-not-the-head + (let [ap (:aperture @measured) + beat (fn [b] (map #(nth ap %) (range (* b 9) (* (inc b) 9)))) + ;; open-amt per beat in synth.cljs is [0.004 0.05 0.022 0.0], so the + ;; ordering is known in advance rather than read off the output. + peak (fn [b] (reduce max (beat b)))] + (is (> (peak 1) (peak 2) (peak 0) (peak 3)) + (str "aperture peaks by beat: " (pr-str (mapv peak (range 4))))) + ;; The head's scale swings 6% across the take. A held pose whose aperture + ;; tracked the head would drift with it; head-local, it is one number. + (is (< (- (peak 0) (reduce min (beat 0))) 1e-9) + (str "aperture wandered by " (- (peak 0) (reduce min (beat 0))) + " over a held pose")))) + +(deftest slot-identity-survives-the-stabilisation + ;; Slot position IS the vertex's identity, and that is what makes temporal + ;; correspondence possible at all. On a 20-slot ring the four cardinals land on + ;; the four quarter slots: 0 and 10 are the two corners, 5 and 15 the lip + ;; centres. So the corners are the horizontal extremes and 5 is above 15, with y + ;; growing downward — and a traversal reversed, rotated or transposed anywhere + ;; breaks that. The simplicity check catches the same class; this one says which + ;; slot moved. + ;; + ;; WHICH corner is slot 0 is deliberately not asserted. Left and right in + ;; MediaPipe's own naming are viewer-relative in some places and + ;; subject-relative in others, so a test written from the table would be pinning + ;; a coin flip; the synth puts slot 0 at larger x and that is all this knows. + (doseq [f (range 0 72 7)] + (let [r (nth (:outer @measured) f) + at (fn [s] (nth r s))] + (is (> (:x (at 0)) (:x (at 5)) (:x (at 10))) + (str "frame " f ": corners are not the horizontal extremes")) + (is (< (:y (at 5)) (:y (at 15))) + (str "frame " f ": the top centre is not above the bottom centre"))))) + +(deftest smoothing-and-subsampling-commute + ;; The claim the stage split rests on. `contour avg` is stage 4 and `vertices` is + ;; stage 5, and that ordering is only free because both operations are per-slot: + ;; averaging slot s over time then picking slot s is picking it then averaging. + ;; The prototype subsamples first, so this is also what keeps parity honest. + (doseq [radius [1 2 3] + verts [4 6 8 10]] + (let [slots (ring/subsample-slots (count lm/LIPS-OUTER) verts) + pick (fn [rings] (mapv (fn [r] (mapv #(nth r %) slots)) rings)) + first' (pick (condition/contours {:contour-avg radius} (:outer @measured))) + then (condition/contours {:contour-avg radius} (pick (:outer @measured)))] + (is (= first' then) + (str "radius " radius ", " verts " verts: the two orders disagree"))))) diff --git a/frontend/test/arthur/parity_test.cljs b/frontend/test/arthur/parity_test.cljs index f14a988..8122a43 100644 --- a/frontend/test/arthur/parity_test.cljs +++ b/frontend/test/arthur/parity_test.cljs @@ -21,6 +21,9 @@ [arthur.domain.palette :as pal] [arthur.domain.raster :as raster] [arthur.domain.ring :as ring] + [arthur.flow.condition :as condition] + [arthur.flow.measure.anchor :as anchor] + [arthur.flow.measure.mouth :as mouth] [arthur.synth :as synth])) ;; The port plan's number. A larger gap than this is a port bug, not float noise. @@ -198,3 +201,45 @@ (deftest hex-to-rgb-agrees (agrees? "palette rgb" pal/rgb (:paletteRgb @oracle))) + +;; ---- flow: stage 3 measure + stage 4 condition ---- +;; +;; The prototype's `stabilize` is three things: the anchor fit, the smoothing of +;; its parameters, and the mouth measured through the result. Here they are three +;; functions in two stages, so what is diffed is the COMPOSITION — a split that +;; agreed on every piece and not on the whole would be a split and not a port. + +(deftest stabilize-agrees-across-the-split-stages + (doseq [{:keys [aspect radius ref rigid transforms residual outer inner aperture]} + (:stabilize @oracle)] + (let [label (str "stabilize(aspect " aspect ", radius " radius ")") + fitted (anchor/fit {:aspect aspect} {:dense @track}) + anchd (condition/anchor {:anchor-avg radius} fitted) + got (mouth/measure {:aspect aspect} + {:dense @track :transforms (:transforms anchd)})] + (agrees? (str label " ref") (:ref fitted) ref) + (agrees? (str label " rigid") (:rigid fitted) rigid) + (agrees? (str label " transforms") (:transforms anchd) transforms) + (agrees? (str label " outer") (:outer got) outer) + (agrees? (str label " inner") (:inner got) inner) + (agrees? (str label " aperture") (:aperture got) aperture) + ;; The prototype takes the residual against the SMOOTHED transforms, because + ;; those were the ones in scope. `anchor/fit` takes it against the raw fit, + ;; deliberately: the number's job is to say whether the footage is + ;; stabilisable, and folding the smoothing error into it makes a setting look + ;; like a property of the shot. Parity is on the function, handed what the + ;; prototype handed it — so the divergence is a decision and not a drift. + (agrees? (str label " residual") + (anchor/residuals (:ref fitted) (:rigid fitted) (:transforms anchd)) + residual) + (when (zero? radius) + (agrees? (str label " residual, as fit reports it") (:residual fitted) residual))))) + +(deftest smooth-contours-agrees-at-every-radius + (let [fitted (anchor/fit {:aspect 0.5625} {:dense @track}) + rings (:outer (mouth/measure {:aspect 0.5625} + {:dense @track :transforms (:transforms fitted)}))] + (doseq [{:keys [radius outer]} (:smoothContours @oracle)] + (agrees? (str "condition/contours radius " radius) + (condition/contours {:contour-avg radius} rings) + outer)))) diff --git a/frontend/test/parity/oracle.mjs b/frontend/test/parity/oracle.mjs index 05b42cf..a8ed287 100644 --- a/frontend/test/parity/oracle.mjs +++ b/frontend/test/parity/oracle.mjs @@ -21,6 +21,7 @@ import { RIGID, LIPS_OUTER, EYE_R_RING, BROW_A_RING, FACE_OVAL, subsampleSlots } from '../../../js/landmarks.js'; import { fitSimilarity, applySim, fitResidual, procrustesMean, movingAverage, smoothTransforms, offsetRing } from '../../../js/mathutil.js'; +import { stabilize, smoothContours } from '../../../js/pipeline.js'; import { synthDense } from '../../../js/synth.js'; import { IndexedRaster, hexToRgb } from '../../../js/raster.js'; @@ -34,6 +35,8 @@ const ref = procrustesMean(rigid); const tfs = rigid.map((r) => fitSimilarity(r, ref)); const strip = (p) => ({ x: p.x, y: p.y, z: p.z ?? 0 }); +const xy = (p) => ({ x: p.x, y: p.y }); +const ring = (r) => r.map(xy); const stripTf = (t) => ({ s: t.s, theta: t.theta, tx: t.tx, ty: t.ty }); // A known transform recovered exactly, which is the same case the CLJS unit test @@ -59,6 +62,42 @@ const out = { movingAverage: [0, 1, 2, 3, 7].map((radius) => ({ radius, vals: movingAverage(tfs.map((t) => t.tx), radius), })), + // stabilize(), which the CLJS side reaches as three stages: the anchor fit, + // the conditioning of its parameters, and the mouth measured through the + // result. Diffing the composition is the point — a split that agreed on each + // piece and not on the whole would be a split, not a port. + // + // aspect 1 is in here to isolate the rest, and 0.5625 (a 1080x1920 phone clip) + // because it is the only value that exercises the anisotropy correction at all: + // at aspect 1 `pick` is the identity and a port that dropped it entirely would + // pass. radius 0 and 2 because the split moved the smoothing OUT of the middle + // of this function, so agreeing only at radius 0 would prove nothing about it. + stabilize: [{ aspect: 1, radius: 0 }, + { aspect: 0.5625, radius: 0 }, + { aspect: 0.5625, radius: 2 }].map(({ aspect, radius }) => { + const st = stabilize(track, radius, aspect); + return { + aspect, radius, + ref: st.ref.map(xy), + rigid: st.rigid.map(ring), + transforms: st.transforms.map(stripTf), + residual: st.residual, + outer: st.outer.map(ring), + inner: st.inner.map(ring), + aperture: st.aperture, + }; + }), + // The contour knob, on the ring it is actually dragged for. The rings are the + // full 20 slots and not a subsample, which is where the CLJS side differs in + // arrangement and must not differ in numbers: the prototype subsamples before + // smoothing, the port smooths before subsampling, and the two commute because + // both operations are per-slot. + smoothContours: (() => { + const st = stabilize(track, 0, 0.5625); + return [0, 1, 3].map((radius) => ({ + radius, outer: smoothContours(st.outer, radius).map(ring), + })); + })(), offsetRing: [0, 0.5, 2, -1].map((d) => ({ d, ring: offsetRing(LIPS_OUTER.map((i) => track[0][i]), d).map(strip),