noe/arthur
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Port step 4: measure the anchor and the mouth, condition on its own

Browse source 
`stabilize` is three things wearing one name, and it is now three functions in two
stages: `flow/measure/anchor` fits the rigid transform, `flow/condition` smooths
its parameters, `flow/measure/mouth` measures the lip rings through the result.
Parity is on the COMPOSITION and not on the pieces -- a split that agreed
function by function and not end to end would be a split rather than a port.

The oracle now drives `stabilize` at three configurations and the port agrees to
1e-9 on ref, rigid, transforms, outer, inner and aperture, plus `smoothContours`
at three radii. Two of the three configurations are at aspect 0.5625, a 1080x1920
phone clip, because at aspect 1 `pick` is the identity: a port that dropped the
anisotropy correction outright would pass every other assertion in the suite.
148 tests, up from 134.

Three decisions worth the reading time.

`makeXform` is not ported, and its absence takes the face oval with it. It
centres on the oval's bounding box and zooms until the face is 80% of the raster
height, so every vertex it touched carried a cropping decision made once, at
analysis time, from one frame's landmarks. Geometry belongs in the node's own
local space with the framing as a transform on a node, so this is a deletion. The
oval's only other consumer was the placeholder plate outline, which is painting.

The residual is taken against the RAW fit, and the prototype took it against the
smoothed one. That is the only deliberate numeric divergence here, and parity is
kept by asserting `anchor/residuals` on exactly what the prototype handed it. The
number's job is to say whether a section is stabilisable at all; folding the
smoothing error into it makes a slider look like a property of the footage, and
docs/architecture.md lists the residual under stage 3, which requires it to be
knob-free. `condition/anchor` therefore replaces `:transforms` and leaves
`:residual` alone.

The stage order is not the strict chain the table in docs/architecture.md looks
like, and that document now says so. The fit is knob-free, conditioning smooths
it, and the rings are measured *through* the conditioned transform -- so
`anchor avg` does re-run the ring mapping, which is a few hundred frames of twenty
points. The guarantee was only ever about the part that reads a source pixel, and
that part never sees a transform.

Two things fall out and are asserted rather than assumed. Smoothing and
subsampling commute, because both are per-slot, which is what lets `vertices`
stay a stage-5 knob downstream of a stage-4 one -- and it is also why the port can
smooth the full twenty slots where the prototype smooths eight and still match.
And `condition/contours` is `geom/moving-average` per vertex per axis rather than
its own clamped window, so "radius 2" cannot come to mean two different things at
the two knobs.

One dead end recorded so nobody walks it twice: the synth's head is perfectly
rigid -- its jitter is a whole-head translation, which a similarity absorbs
exactly -- so every frame's rigid configuration is congruent with frame zero's and
the Procrustes mean IS frame zero to 1e-15, jitter or none. "The reference is the
mean and not frame zero" cannot be asserted on this track and is asserted in
geom-test, where the two can differ.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Olive Vaughn 2026-09-27 18:00:11 -04:00
parent 11192d61c6
commit 942e2f38ab
10 changed files with 494 additions and 1 deletions
Download patch file Download diff file Expand all files Collapse all files

10
docs/architecture.md
View file

@ -151,6 +151,16 @@ source pixel.** Stage 4 is separated from 3 only because `contour avg` and
`anchor avg` are knobs and the rest of stage 3 is not; splitting them means `anchor avg` are knobs and the rest of stage 3 is not; splitting them means
dragging that slider does not re-run the interior extraction. dragging that slider does not re-run the interior extraction.
That last clause is the guarantee, and it is narrower than the table's ordering
looks. Stage 3 is not one pass that finishes before stage 4 begins. The anchor fit
is knob-free; conditioning smooths its four parameters; the head-local rings are
then measured *through* the conditioned transform — so `anchor avg` does re-run
the ring mapping, which is a few hundred frames of twenty points and free. What it
must not re-run is the part that reads a source pixel, and that part takes the
landmarks and the frames and never the transform, so it does not. Built as
`measure/anchor` → `condition/anchor` → `measure/mouth` → `condition/contours`,
and stage 7's eyes, brows and interior follow the same shape.
### Where the current code lands ### Where the current code lands
| Now | Stage | | Now | Stage |

8
frontend/README.md
View file

@ -72,7 +72,12 @@ From step 9 Django serves the page and `:dev-http` goes away.
`js/` is the numeric oracle, not dead weight. `test/parity/` runs both `js/` is the numeric oracle, not dead weight. `test/parity/` runs both
implementations on the same synthetic track and diffs them: `fit-similarity` and implementations on the same synthetic track and diffs them: `fit-similarity` and
`procrustes-mean` agree to 1e-9, the raster pixel-for-pixel. `procrustes-mean` agree to 1e-9, the raster pixel-for-pixel, and `stabilize`'s
whole output agrees across the three namespaces it was split into.
The oracle drives `stabilize` at aspect 0.5625 as well as at 1. Aspect 1 makes the
anisotropy correction the identity, so a port that dropped it entirely would pass
— which is the one thing a parity suite on normalised landmarks can be blind to.
Both sides get the identical track because `js/synth.js` reads `Math.random` at Both sides get the identical track because `js/synth.js` reads `Math.random` at
call time, so `oracle.mjs` stubs it to a constant and the CLJS side passes call time, so `oracle.mjs` stubs it to a constant and the CLJS side passes
@ -86,6 +91,7 @@ the prototype's mistakes into the rewrite and make them permanent.
``` ```
src/arthur/domain/ pure. No re-frame, no DOM, no flow/. src/arthur/domain/ pure. No re-frame, no DOM, no flow/.
src/arthur/flow/ the stages. `(f params inputs) -> output`, no state.
src/arthur/demo.cljs the hand-written scene, read from demo/scene.edn src/arthur/demo.cljs the hand-written scene, read from demo/scene.edn
src/arthur/ui/canvas.cljs the one imperative sink — the only DOM canvas call src/arthur/ui/canvas.cljs the one imperative sink — the only DOM canvas call
test/arthur/synth.cljs the synthetic track — test infrastructure, not src test/arthur/synth.cljs the synthetic track — test infrastructure, not src

50
frontend/src/arthur/flow/condition.cljs Normal file
View file

@ -0,0 +1,50 @@
(ns arthur.flow.condition
"Stage 4: the two smoothing knobs, and nothing else.
It is a stage of its own for exactly one reason. `anchor avg` and `contour avg`
are knobs and the rest of measure is not, so dragging either must not re-run the
interior extraction — the one part of measure that reads a source pixel, and the
only part that costs seconds.
Neither function knows what it is smoothing. `anchor` smooths four transform
parameters and `contours` smooths a ring track per vertex; a face appears
nowhere in here."
(:require [arthur.domain.geom :as geom]))
(defn anchor
"Smooth the anchor fit's four parameters. `arthur.domain.geom/smooth-transforms`
says why it is the transform and not the contour.
Returns the measured map with `:transforms` replaced, so the anchor keeps
travelling as one value and nothing downstream has to know whether it has been
conditioned yet. `:residual` is deliberately left alone: it is the residual of
the FIT, and it is not a function of this knob."
[{:keys [anchor-avg]} anchored]
(update anchored :transforms geom/smooth-transforms anchor-avg))
(defn contours
"Temporal smoothing of a ring track, per vertex, across time.
docs/design.md says to smooth the transform and never the contour. That was
correct while keys were sparse: sampling at velocity minima rejected per-frame
detector noise for free. With a key on every frame the noise is visible as a
shimmer along the lip edge, so a bounded exception applies - the window must
stay SHORTER than the shortest articulation worth keeping. At 12fps, mouth
movement spans 3-6 frames and detector noise is per-frame, so a radius of 1
separates them and a radius of 3 would start eating speech.
`contour-avg` is in frames either side: 0 off, 1 = 3-frame average, 2 = 5-frame.
It is the same clamped window `geom/moving-average` gives the transform
parameters — reused rather than re-derived, so \"radius 2\" cannot come to mean
two different things at the two knobs."
[{:keys [contour-avg]} rings]
(if (<= contour-avg 0)
(vec rings)
(let [rings (vec rings)
axis (fn [v k] (geom/moving-average (map #(k (nth % v)) rings) contour-avg))
;; Transposed once into a per-vertex pair of series, because the
;; smoothing is along time and the storage is along vertices.
axes (mapv (fn [v] [(axis v :x) (axis v :y)])
(range (count (first rings))))]
(mapv (fn [t] (mapv (fn [[xs ys]] {:x (nth xs t) :y (nth ys t)}) axes))
(range (count rings))))))

64
frontend/src/arthur/flow/measure/anchor.cljs Normal file
View file

@ -0,0 +1,64 @@
(ns arthur.flow.measure.anchor
"Stage 3, the anchor: the rigid transform per frame, and the space every other
measurement is taken in.
The fit is knob-free, deliberately. Smoothing its four parameters is stage 4 —
`arthur.flow.condition` — because `anchor avg` is a knob and the rest of measure
is not, and because a residual that moved when a smoothing slider moved would
report the footage as unstabilisable on account of a setting.
`makeXform` is NOT here, and is not being ported. It centres on the face oval's
bounding box and zooms until the face is 80% of the raster height, so every
vertex it touches carries a cropping decision made once, at analysis time, from
one frame's landmarks. Geometry is stored in the node's own local space and the
framing is a transform on a node; see \"What space geometry is in\" in
docs/animation-model.md. So the face oval is not measured here either — its only
consumers in the prototype were that transform and the placeholder plate
outline, and the plate outline belongs to painting."
(:require [arthur.domain.geom :as geom]
[arthur.domain.landmarks :as lm]))
(defn pick
"Landmarks `idx` out of one dense `frame`, converted to an ISOTROPIC space.
MediaPipe normalises x by image WIDTH and y by image HEIGHT, so its normalised
space is anisotropic: for a 1080x1920 frame, one unit of x is 1080px and one
unit of y is 1920px. Treating those as comparable stretches everything
horizontally by H/W, and worse, makes fit-similarity fit a \"rotation\" in a
sheared space, so head roll comes out subtly wrong as well.
Multiplying x by aspect = W/H converts to an isotropic space whose unit is one
image height, so equal numbers mean equal pixels. Everything downstream -
Procrustes, the similarity fit, the raster transform - depends on that."
[frame idx aspect]
(mapv (fn [i] (let [p (nth frame i)] {:x (* (:x p) aspect) :y (:y p)})) idx))
(defn residuals
"RMS misfit per frame, in the isotropic space's units — one image height.
Taken against the transforms it is HANDED rather than against a fit of its own,
so the same function serves the stage-3 reading and the parity diff. The
prototype took it against the SMOOTHED transforms, which folds the smoothing
error into a number whose whole job is to say whether the footage is
stabilisable at all; `fit` takes it against the raw fit instead."
[ref rigid tfs]
(mapv (fn [rig tf] (geom/fit-residual tf rig ref)) rigid tfs))
(defn fit
"Dense landmarks -> the rigid fit of every frame onto the shot's mean pose.
The reference is the Procrustes MEAN configuration over the shot, not frame
zero, so no single frame's idiosyncrasies get baked into every other frame."
[{:keys [aspect]} {:keys [dense]}]
(let [rigid (mapv #(pick % lm/RIGID aspect) dense)
ref (geom/procrustes-mean rigid)
tfs (mapv #(geom/fit-similarity % ref) rigid)]
{:ref ref
;; Rigid landmarks in IMAGE space: the head-pose signal. Frame removal is
;; decided from head motion, not from the mouth, so this has to survive the
;; fit rather than being consumed by it.
:rigid rigid
:transforms tfs
;; Residual rises with out-of-plane rotation, which no 2D similarity can
;; remove. High values mean this section wants a different head plate.
:residual (residuals ref rigid tfs)}))

39
frontend/src/arthur/flow/measure/mouth.cljs Normal file
View file

@ -0,0 +1,39 @@
(ns arthur.flow.measure.mouth
"Stage 3, the mouth: the lip rings with the head's motion taken out, and the
aperture that decides whether there is an interior at all.
Head-local means the anchor's space, so `pick` is read from
`arthur.flow.measure.anchor` rather than written a second time. A ring measured
in a different space from the fit that placed it is not a failure anyone would
see — it is a mouth that is quietly the wrong width.
The rings keep every slot of their table. A vertex budget is a stage-5 knob, and
subsampling is a per-slot pick while stage 4's contour average is a per-slot
average over time, so the two commute: smooth-then-subsample and
subsample-then-smooth are the same numbers. That is what lets the vertex knob
sit downstream of the smoothing knob instead of alongside it, and mouth-test
asserts it rather than leaving it to look obvious.
Turning the aperture into `[:vis]` on `:mouth-in` is stage 5. This reports the
measurement, not the decision."
(:require [arthur.domain.geom :as geom]
[arthur.domain.landmarks :as lm]
[arthur.flow.measure.anchor :as anchor]))
(defn measure
"Dense landmarks plus the anchor's transforms -> head-local lip rings.
`transforms` is whatever the caller has: the raw fit, or — normally — stage 4's
conditioned one. Which it is belongs to the caller, because \"smooth the
transform, never the contour\" only means anything while the two are separate."
[{:keys [aspect]} {:keys [dense transforms]}]
(let [local (fn [table]
(mapv (fn [tf frame] (geom/apply-sim-all tf (anchor/pick frame table aspect)))
transforms dense))]
{:outer (local lm/LIPS-OUTER)
:inner (local lm/LIPS-INNER)
;; Not a separate measurement: APERTURE is slots 5 and 15 of LIPS_INNER, so
;; this is the inner ring's own height read off as a scalar. Writing those
;; two landmarks a second time is what gave the synthetic mouth a bowtie.
:aperture (mapv (fn [[a b]] (js/Math.hypot (- (:x a) (:x b)) (- (:y a) (:y b))))
(local lm/APERTURE))}))

73
frontend/test/arthur/flow/condition_test.cljs Normal file
View file

@ -0,0 +1,73 @@
(ns arthur.flow.condition-test
(:require [cljs.test :refer [deftest is testing]]
[arthur.domain.ring :as ring]
[arthur.flow.condition :as condition]
[arthur.flow.measure.anchor :as anchor]
[arthur.flow.measure.mouth :as mouth]
[arthur.synth :as synth]))
;; Jitter ON here, unlike the measure tests: a low-pass with nothing to remove
;; cannot be shown to remove anything.
(def noisy (delay (synth/synth-dense 72)))
(def fitted (delay (anchor/fit {:aspect 1} {:dense @noisy})))
(def rings
(delay (:outer (mouth/measure {:aspect 1}
{:dense @noisy :transforms (:transforms @fitted)}))))
(defn- hf
"Mean |second difference| per vertex over time: the high-frequency energy a
low-pass is supposed to remove."
[rs]
(let [n (count rs) v (count (first rs))
at (fn [t i k] (k (nth (nth rs t) i)))]
(/ (reduce + (for [t (range 1 (dec n)) i (range v) k [:x :y]]
(abs (+ (- (at (inc t) i k) (* 2 (at t i k))) (at (dec t) i k)))))
(* (- n 2) v 2))))
(deftest radius-zero-is-the-identity
;; "Off" has to be off. A knob whose zero still rewrote the numbers would make
;; every parity diff below it untrustworthy.
(is (= (:outer (mouth/measure {:aspect 1} {:dense @noisy :transforms (:transforms @fitted)}))
(condition/contours {:contour-avg 0} @rings)))
(is (vector? (condition/contours {:contour-avg 0} @rings))
"callers index into this, so it may not come back a lazy seq"))
(deftest the-anchor-knob-touches-only-the-transforms
;; `:residual` is the residual of the FIT and is not a function of this knob.
;; The prototype recomputed it from the smoothed transforms, which made "is this
;; section stabilisable" move when a smoothing slider moved.
(let [conditioned (condition/anchor {:anchor-avg 5} @fitted)]
(is (= (:ref @fitted) (:ref conditioned)))
(is (= (:rigid @fitted) (:rigid conditioned)))
(is (= (:residual @fitted) (:residual conditioned)))
(is (not= (:transforms @fitted) (:transforms conditioned))
"...and it did smooth something")))
(deftest the-contour-average-is-a-low-pass-along-time
(let [es (mapv #(hf (condition/contours {:contour-avg %} @rings)) [0 1 2 3 5])]
(is (apply >= es) (str "hf energy by radius: " (pr-str es)))
(is (< (nth es 1) (* 0.8 (first es)))
(str "radius 1 removed only " (- 1 (/ (nth es 1) (first es))) " of it"))))
(deftest a-steady-contour-comes-back-unchanged
(let [r (nth @rings 0)
rs (vec (repeat 9 r))]
(is (< (reduce max (for [f (condition/contours {:contour-avg 3} rs)
[p q] (map vector f r)]
(js/Math.hypot (- (:x p) (:x q)) (- (:y p) (:y q)))))
1e-15))))
(deftest smoothing-cannot-make-a-ring-self-intersect
;; Averaging across frames is averaging across SHAPES, so in principle it could
;; fold a ring over itself — and a self-intersecting ring renders as blocks
;; meeting at corners rather than as an error. Unlikely on a lip contour and
;; cheap to rule out, which is the same reason domain/ring asserts simplicity on
;; the raw track.
(doseq [radius [1 2 3 5]]
(let [bad (first (for [[f r] (map-indexed vector (condition/contours {:contour-avg radius} @rings))
:let [hits (ring/self-intersections r)]
:when (seq hits)]
{:radius radius :frame f :edges (first hits)}))]
(is (nil? bad) (str "self-intersection: " (pr-str bad))))))

82
frontend/test/arthur/flow/measure/anchor_test.cljs Normal file
View file

@ -0,0 +1,82 @@
(ns arthur.flow.measure.anchor-test
"The anchor is the only stage with genuine ground truth available, and that is
why the synthetic track exists: it moves the head by a KNOWN similarity, so
\"did the fit remove the head motion\" is a number and not an impression. Real
footage gives nothing to compare against.
One thing cannot be asserted here, and it is worth knowing before trying. The
synth's head is PERFECTLY rigid — the jitter is a whole-head translation, which
a similarity absorbs exactly — so every frame's rigid configuration is congruent
with frame zero's, and the Procrustes mean is therefore frame zero's
configuration to 1e-15 on this track, jitter or no jitter. \"The reference is the
mean and not frame zero\" has to be asserted where the two can differ, which is
arthur.domain.geom-test."
(:require [cljs.test :refer [deftest is testing]]
[arthur.domain.geom :as geom]
[arthur.flow.measure.anchor :as anchor]
[arthur.synth :as synth]))
;; Jitter of exactly zero, so the synth's head motion IS a similarity and the fit
;; has to recover it exactly. With jitter there is no exact answer to assert.
(def clean (delay (synth/synth-dense 72 {:rand-fn (constantly 0.5)})))
(defn- worst [f xs] (reduce max (map f xs)))
(deftest the-fit-removes-a-known-head-motion
;; Drift, sway, roll and a slow scale change, all four gone: every frame's
;; rigid configuration lands on the reference. This is the whole claim of
;; stage 3 and it is either exact or the fit is wrong.
(let [{:keys [ref rigid transforms residual]} (anchor/fit {:aspect 1} {:dense @clean})
gap (worst (fn [[tf rig]]
(worst (fn [[m r]] (js/Math.hypot (- (:x m) (:x r)) (- (:y m) (:y r))))
(map vector (geom/apply-sim-all tf rig) ref)))
(map vector transforms rigid))]
(is (< gap 1e-9) (str "worst stabilised-vs-reference gap " gap))
(is (< (reduce max residual) 1e-9)
(str "worst residual " (reduce max residual)
" — a pure similarity has to fit at zero"))
;; And the motion was really there to remove, or the above is vacuous.
(is (> (- (reduce max (map :tx transforms)) (reduce min (map :tx transforms))) 0.05))
(is (> (- (reduce max (map :theta transforms)) (reduce min (map :theta transforms))) 0.2))))
(deftest the-aspect-correction-recovers-roll
;; MediaPipe divides x by WIDTH and y by HEIGHT, so a 1080x1920 clip arrives
;; with x stretched by H/W. Simulated here by dividing x back out: `pick` with
;; the right aspect must undo it exactly, and treating the space as isotropic
;; must not — a "similarity" fitted in a sheared space is not one, and the error
;; lands on roll, where it looks like a directed choice rather than a bug.
(let [aspect 0.5625
squashed (mapv (fn [fr] (mapv #(update % :x / aspect) fr)) @clean)
truth (anchor/fit {:aspect 1} {:dense @clean})
fixed (anchor/fit {:aspect aspect} {:dense squashed})
naive (anchor/fit {:aspect 1} {:dense squashed})
d-theta (fn [a] (worst abs (map - (map :theta (:transforms a))
(map :theta (:transforms truth)))))]
(is (< (d-theta fixed) 1e-9)
(str "corrected roll is off by " (d-theta fixed)))
(is (> (d-theta naive) 0.01)
(str "uncorrected roll is off by only " (d-theta naive)
" — if this is small the correction is not being exercised"))
;; The residual notices too, which is what makes a forgotten aspect visible
;; at the readout rather than only in the output.
(is (< (reduce max (:residual fixed)) 1e-9))
(is (> (reduce max (:residual naive)) 1e-4))))
(deftest out-of-plane-motion-shows-up-in-the-residual
;; No 2D similarity can remove yaw or pitch, and that is the point of reporting
;; the residual: high values mean the section wants a different head plate.
;; A yaw foreshortens x and leaves y alone, which is exactly a non-uniform
;; scale, so squeezing one frame's x is the cheapest honest stand-in.
(let [yawed (update (vec @clean) 10
(fn [fr] (mapv #(update % :x * 0.97) fr)))
{:keys [residual]} (anchor/fit {:aspect 1} {:dense yawed})]
(is (> (nth residual 10) 1e-4)
(str "the yawed frame's residual is " (nth residual 10)))
;; Localised, not smeared. Its neighbours are allowed to be non-zero — the one
;; bad frame pulls the Procrustes mean, so every frame's reference moves a
;; little — but by more than an order of magnitude less. Which is the argument
;; for the mean reference stated as a measurement: one bad detection degrades
;; the whole shot slightly instead of one frame catastrophically.
(is (< (nth residual 9) (* 0.05 (nth residual 10)))
(str "neighbour residual " (nth residual 9) " is "
(/ (nth residual 9) (nth residual 10)) " of the yawed frame's"))))

85
frontend/test/arthur/flow/measure/mouth_test.cljs Normal file
View file

@ -0,0 +1,85 @@
(ns arthur.flow.measure.mouth-test
(:require [cljs.test :refer [deftest is testing]]
[arthur.domain.landmarks :as lm]
[arthur.domain.ring :as ring]
[arthur.flow.condition :as condition]
[arthur.flow.measure.anchor :as anchor]
[arthur.flow.measure.mouth :as mouth]
[arthur.synth :as synth]))
(def clean (delay (synth/synth-dense 72 {:rand-fn (constantly 0.5)})))
(def measured
(delay
(let [fitted (anchor/fit {:aspect 1} {:dense @clean})]
(mouth/measure {:aspect 1} {:dense @clean :transforms (:transforms fitted)}))))
(defn- spread
"Largest distance between any two of `rings` at the same slot: how much the
shape moved over the frames given."
[rings]
(reduce max (for [a rings b rings [p q] (map vector a b)]
(js/Math.hypot (- (:x p) (:x q)) (- (:y p) (:y q))))))
;; The synth holds each mouth pose for nine frames, so frames 0-8 are one pose
;; carried around by a moving, rolling, scaling head. That is the case the whole
;; stage exists for.
(def ^:private hold (range 0 9))
(deftest a-held-pose-holds-still-once-the-head-is-taken-out
(let [local (spread (map #(nth (:outer @measured) %) hold))
image (spread (map (fn [f] (anchor/pick (nth @clean f) lm/LIPS-OUTER 1)) hold))]
(is (< local 1e-9)
(str "head-local outer ring moved " local " over a held pose"))
;; And it moved plenty in the footage, or the line above is measuring nothing.
(is (> image 0.02)
(str "the ring only moved " image " in image space over the same frames"))))
(deftest the-aperture-measures-the-mouth-and-not-the-head
(let [ap (:aperture @measured)
beat (fn [b] (map #(nth ap %) (range (* b 9) (* (inc b) 9))))
;; open-amt per beat in synth.cljs is [0.004 0.05 0.022 0.0], so the
;; ordering is known in advance rather than read off the output.
peak (fn [b] (reduce max (beat b)))]
(is (> (peak 1) (peak 2) (peak 0) (peak 3))
(str "aperture peaks by beat: " (pr-str (mapv peak (range 4)))))
;; The head's scale swings 6% across the take. A held pose whose aperture
;; tracked the head would drift with it; head-local, it is one number.
(is (< (- (peak 0) (reduce min (beat 0))) 1e-9)
(str "aperture wandered by " (- (peak 0) (reduce min (beat 0)))
" over a held pose"))))
(deftest slot-identity-survives-the-stabilisation
;; Slot position IS the vertex's identity, and that is what makes temporal
;; correspondence possible at all. On a 20-slot ring the four cardinals land on
;; the four quarter slots: 0 and 10 are the two corners, 5 and 15 the lip
;; centres. So the corners are the horizontal extremes and 5 is above 15, with y
;; growing downward — and a traversal reversed, rotated or transposed anywhere
;; breaks that. The simplicity check catches the same class; this one says which
;; slot moved.
;;
;; WHICH corner is slot 0 is deliberately not asserted. Left and right in
;; MediaPipe's own naming are viewer-relative in some places and
;; subject-relative in others, so a test written from the table would be pinning
;; a coin flip; the synth puts slot 0 at larger x and that is all this knows.
(doseq [f (range 0 72 7)]
(let [r (nth (:outer @measured) f)
at (fn [s] (nth r s))]
(is (> (:x (at 0)) (:x (at 5)) (:x (at 10)))
(str "frame " f ": corners are not the horizontal extremes"))
(is (< (:y (at 5)) (:y (at 15)))
(str "frame " f ": the top centre is not above the bottom centre")))))
(deftest smoothing-and-subsampling-commute
;; The claim the stage split rests on. `contour avg` is stage 4 and `vertices` is
;; stage 5, and that ordering is only free because both operations are per-slot:
;; averaging slot s over time then picking slot s is picking it then averaging.
;; The prototype subsamples first, so this is also what keeps parity honest.
(doseq [radius [1 2 3]
verts [4 6 8 10]]
(let [slots (ring/subsample-slots (count lm/LIPS-OUTER) verts)
pick (fn [rings] (mapv (fn [r] (mapv #(nth r %) slots)) rings))
first' (pick (condition/contours {:contour-avg radius} (:outer @measured)))
then (condition/contours {:contour-avg radius} (pick (:outer @measured)))]
(is (= first' then)
(str "radius " radius ", " verts " verts: the two orders disagree")))))

45
frontend/test/arthur/parity_test.cljs
View file

@ -21,6 +21,9 @@
[arthur.domain.palette :as pal] [arthur.domain.palette :as pal]
[arthur.domain.raster :as raster] [arthur.domain.raster :as raster]
[arthur.domain.ring :as ring] [arthur.domain.ring :as ring]
[arthur.flow.condition :as condition]
[arthur.flow.measure.anchor :as anchor]
[arthur.flow.measure.mouth :as mouth]
[arthur.synth :as synth])) [arthur.synth :as synth]))
;; The port plan's number. A larger gap than this is a port bug, not float noise. ;; The port plan's number. A larger gap than this is a port bug, not float noise.
@ -198,3 +201,45 @@
(deftest hex-to-rgb-agrees (deftest hex-to-rgb-agrees
(agrees? "palette rgb" pal/rgb (:paletteRgb @oracle))) (agrees? "palette rgb" pal/rgb (:paletteRgb @oracle)))
;; ---- flow: stage 3 measure + stage 4 condition ----
;;
;; The prototype's `stabilize` is three things: the anchor fit, the smoothing of
;; its parameters, and the mouth measured through the result. Here they are three
;; functions in two stages, so what is diffed is the COMPOSITION — a split that
;; agreed on every piece and not on the whole would be a split and not a port.
(deftest stabilize-agrees-across-the-split-stages
(doseq [{:keys [aspect radius ref rigid transforms residual outer inner aperture]}
(:stabilize @oracle)]
(let [label (str "stabilize(aspect " aspect ", radius " radius ")")
fitted (anchor/fit {:aspect aspect} {:dense @track})
anchd (condition/anchor {:anchor-avg radius} fitted)
got (mouth/measure {:aspect aspect}
{:dense @track :transforms (:transforms anchd)})]
(agrees? (str label " ref") (:ref fitted) ref)
(agrees? (str label " rigid") (:rigid fitted) rigid)
(agrees? (str label " transforms") (:transforms anchd) transforms)
(agrees? (str label " outer") (:outer got) outer)
(agrees? (str label " inner") (:inner got) inner)
(agrees? (str label " aperture") (:aperture got) aperture)
;; The prototype takes the residual against the SMOOTHED transforms, because
;; those were the ones in scope. `anchor/fit` takes it against the raw fit,
;; deliberately: the number's job is to say whether the footage is
;; stabilisable, and folding the smoothing error into it makes a setting look
;; like a property of the shot. Parity is on the function, handed what the
;; prototype handed it — so the divergence is a decision and not a drift.
(agrees? (str label " residual")
(anchor/residuals (:ref fitted) (:rigid fitted) (:transforms anchd))
residual)
(when (zero? radius)
(agrees? (str label " residual, as fit reports it") (:residual fitted) residual)))))
(deftest smooth-contours-agrees-at-every-radius
(let [fitted (anchor/fit {:aspect 0.5625} {:dense @track})
rings (:outer (mouth/measure {:aspect 0.5625}
{:dense @track :transforms (:transforms fitted)}))]
(doseq [{:keys [radius outer]} (:smoothContours @oracle)]
(agrees? (str "condition/contours radius " radius)
(condition/contours {:contour-avg radius} rings)
outer))))

39
frontend/test/parity/oracle.mjs
View file

@ -21,6 +21,7 @@ import { RIGID, LIPS_OUTER, EYE_R_RING, BROW_A_RING, FACE_OVAL,
subsampleSlots } from '../../../js/landmarks.js'; subsampleSlots } from '../../../js/landmarks.js';
import { fitSimilarity, applySim, fitResidual, procrustesMean, import { fitSimilarity, applySim, fitResidual, procrustesMean,
movingAverage, smoothTransforms, offsetRing } from '../../../js/mathutil.js'; movingAverage, smoothTransforms, offsetRing } from '../../../js/mathutil.js';
import { stabilize, smoothContours } from '../../../js/pipeline.js';
import { synthDense } from '../../../js/synth.js'; import { synthDense } from '../../../js/synth.js';
import { IndexedRaster, hexToRgb } from '../../../js/raster.js'; import { IndexedRaster, hexToRgb } from '../../../js/raster.js';
@ -34,6 +35,8 @@ const ref = procrustesMean(rigid);
const tfs = rigid.map((r) => fitSimilarity(r, ref)); const tfs = rigid.map((r) => fitSimilarity(r, ref));
const strip = (p) => ({ x: p.x, y: p.y, z: p.z ?? 0 }); const strip = (p) => ({ x: p.x, y: p.y, z: p.z ?? 0 });
const xy = (p) => ({ x: p.x, y: p.y });
const ring = (r) => r.map(xy);
const stripTf = (t) => ({ s: t.s, theta: t.theta, tx: t.tx, ty: t.ty }); const stripTf = (t) => ({ s: t.s, theta: t.theta, tx: t.tx, ty: t.ty });
// A known transform recovered exactly, which is the same case the CLJS unit test // A known transform recovered exactly, which is the same case the CLJS unit test
@ -59,6 +62,42 @@ const out = {
movingAverage: [0, 1, 2, 3, 7].map((radius) => ({ movingAverage: [0, 1, 2, 3, 7].map((radius) => ({
radius, vals: movingAverage(tfs.map((t) => t.tx), radius), radius, vals: movingAverage(tfs.map((t) => t.tx), radius),
})), })),
// stabilize(), which the CLJS side reaches as three stages: the anchor fit,
// the conditioning of its parameters, and the mouth measured through the
// result. Diffing the composition is the point — a split that agreed on each
// piece and not on the whole would be a split, not a port.
//
// aspect 1 is in here to isolate the rest, and 0.5625 (a 1080x1920 phone clip)
// because it is the only value that exercises the anisotropy correction at all:
// at aspect 1 `pick` is the identity and a port that dropped it entirely would
// pass. radius 0 and 2 because the split moved the smoothing OUT of the middle
// of this function, so agreeing only at radius 0 would prove nothing about it.
stabilize: [{ aspect: 1, radius: 0 },
{ aspect: 0.5625, radius: 0 },
{ aspect: 0.5625, radius: 2 }].map(({ aspect, radius }) => {
const st = stabilize(track, radius, aspect);
return {
aspect, radius,
ref: st.ref.map(xy),
rigid: st.rigid.map(ring),
transforms: st.transforms.map(stripTf),
residual: st.residual,
outer: st.outer.map(ring),
inner: st.inner.map(ring),
aperture: st.aperture,
};
}),
// The contour knob, on the ring it is actually dragged for. The rings are the
// full 20 slots and not a subsample, which is where the CLJS side differs in
// arrangement and must not differ in numbers: the prototype subsamples before
// smoothing, the port smooths before subsampling, and the two commute because
// both operations are per-slot.
smoothContours: (() => {
const st = stabilize(track, 0, 0.5625);
return [0, 1, 3].map((radius) => ({
radius, outer: smoothContours(st.outer, radius).map(ring),
}));
})(),
offsetRing: [0, 0.5, 2, -1].map((d) => ({ offsetRing: [0, 0.5, 2, -1].map((d) => ({
d, d,
ring: offsetRing(LIPS_OUTER.map((i) => track[0][i]), d).map(strip), ring: offsetRing(LIPS_OUTER.map((i) => track[0][i]), d).map(strip),