Port step 4: measure the anchor and the mouth, condition on its own
`stabilize` is three things wearing one name, and it is now three functions in two stages: `flow/measure/anchor` fits the rigid transform, `flow/condition` smooths its parameters, `flow/measure/mouth` measures the lip rings through the result. Parity is on the COMPOSITION and not on the pieces -- a split that agreed function by function and not end to end would be a split rather than a port. The oracle now drives `stabilize` at three configurations and the port agrees to 1e-9 on ref, rigid, transforms, outer, inner and aperture, plus `smoothContours` at three radii. Two of the three configurations are at aspect 0.5625, a 1080x1920 phone clip, because at aspect 1 `pick` is the identity: a port that dropped the anisotropy correction outright would pass every other assertion in the suite. 148 tests, up from 134. Three decisions worth the reading time. `makeXform` is not ported, and its absence takes the face oval with it. It centres on the oval's bounding box and zooms until the face is 80% of the raster height, so every vertex it touched carried a cropping decision made once, at analysis time, from one frame's landmarks. Geometry belongs in the node's own local space with the framing as a transform on a node, so this is a deletion. The oval's only other consumer was the placeholder plate outline, which is painting. The residual is taken against the RAW fit, and the prototype took it against the smoothed one. That is the only deliberate numeric divergence here, and parity is kept by asserting `anchor/residuals` on exactly what the prototype handed it. The number's job is to say whether a section is stabilisable at all; folding the smoothing error into it makes a slider look like a property of the footage, and docs/architecture.md lists the residual under stage 3, which requires it to be knob-free. `condition/anchor` therefore replaces `:transforms` and leaves `:residual` alone. The stage order is not the strict chain the table in docs/architecture.md looks like, and that document now says so. The fit is knob-free, conditioning smooths it, and the rings are measured *through* the conditioned transform -- so `anchor avg` does re-run the ring mapping, which is a few hundred frames of twenty points. The guarantee was only ever about the part that reads a source pixel, and that part never sees a transform. Two things fall out and are asserted rather than assumed. Smoothing and subsampling commute, because both are per-slot, which is what lets `vertices` stay a stage-5 knob downstream of a stage-4 one -- and it is also why the port can smooth the full twenty slots where the prototype smooths eight and still match. And `condition/contours` is `geom/moving-average` per vertex per axis rather than its own clamped window, so "radius 2" cannot come to mean two different things at the two knobs. One dead end recorded so nobody walks it twice: the synth's head is perfectly rigid -- its jitter is a whole-head translation, which a similarity absorbs exactly -- so every frame's rigid configuration is congruent with frame zero's and the Procrustes mean IS frame zero to 1e-15, jitter or none. "The reference is the mean and not frame zero" cannot be asserted on this track and is asserted in geom-test, where the two can differ. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
parent
11192d61c6
commit
942e2f38ab
10 changed files with 494 additions and 1 deletions
50
frontend/src/arthur/flow/condition.cljs
Normal file
50
frontend/src/arthur/flow/condition.cljs
Normal file
|
|
@ -0,0 +1,50 @@
|
|||
(ns arthur.flow.condition
|
||||
"Stage 4: the two smoothing knobs, and nothing else.
|
||||
|
||||
It is a stage of its own for exactly one reason. `anchor avg` and `contour avg`
|
||||
are knobs and the rest of measure is not, so dragging either must not re-run the
|
||||
interior extraction — the one part of measure that reads a source pixel, and the
|
||||
only part that costs seconds.
|
||||
|
||||
Neither function knows what it is smoothing. `anchor` smooths four transform
|
||||
parameters and `contours` smooths a ring track per vertex; a face appears
|
||||
nowhere in here."
|
||||
(:require [arthur.domain.geom :as geom]))
|
||||
|
||||
(defn anchor
|
||||
"Smooth the anchor fit's four parameters. `arthur.domain.geom/smooth-transforms`
|
||||
says why it is the transform and not the contour.
|
||||
|
||||
Returns the measured map with `:transforms` replaced, so the anchor keeps
|
||||
travelling as one value and nothing downstream has to know whether it has been
|
||||
conditioned yet. `:residual` is deliberately left alone: it is the residual of
|
||||
the FIT, and it is not a function of this knob."
|
||||
[{:keys [anchor-avg]} anchored]
|
||||
(update anchored :transforms geom/smooth-transforms anchor-avg))
|
||||
|
||||
(defn contours
|
||||
"Temporal smoothing of a ring track, per vertex, across time.
|
||||
|
||||
docs/design.md says to smooth the transform and never the contour. That was
|
||||
correct while keys were sparse: sampling at velocity minima rejected per-frame
|
||||
detector noise for free. With a key on every frame the noise is visible as a
|
||||
shimmer along the lip edge, so a bounded exception applies - the window must
|
||||
stay SHORTER than the shortest articulation worth keeping. At 12fps, mouth
|
||||
movement spans 3-6 frames and detector noise is per-frame, so a radius of 1
|
||||
separates them and a radius of 3 would start eating speech.
|
||||
|
||||
`contour-avg` is in frames either side: 0 off, 1 = 3-frame average, 2 = 5-frame.
|
||||
It is the same clamped window `geom/moving-average` gives the transform
|
||||
parameters — reused rather than re-derived, so \"radius 2\" cannot come to mean
|
||||
two different things at the two knobs."
|
||||
[{:keys [contour-avg]} rings]
|
||||
(if (<= contour-avg 0)
|
||||
(vec rings)
|
||||
(let [rings (vec rings)
|
||||
axis (fn [v k] (geom/moving-average (map #(k (nth % v)) rings) contour-avg))
|
||||
;; Transposed once into a per-vertex pair of series, because the
|
||||
;; smoothing is along time and the storage is along vertices.
|
||||
axes (mapv (fn [v] [(axis v :x) (axis v :y)])
|
||||
(range (count (first rings))))]
|
||||
(mapv (fn [t] (mapv (fn [[xs ys]] {:x (nth xs t) :y (nth ys t)}) axes))
|
||||
(range (count rings))))))
|
||||
64
frontend/src/arthur/flow/measure/anchor.cljs
Normal file
64
frontend/src/arthur/flow/measure/anchor.cljs
Normal file
|
|
@ -0,0 +1,64 @@
|
|||
(ns arthur.flow.measure.anchor
|
||||
"Stage 3, the anchor: the rigid transform per frame, and the space every other
|
||||
measurement is taken in.
|
||||
|
||||
The fit is knob-free, deliberately. Smoothing its four parameters is stage 4 —
|
||||
`arthur.flow.condition` — because `anchor avg` is a knob and the rest of measure
|
||||
is not, and because a residual that moved when a smoothing slider moved would
|
||||
report the footage as unstabilisable on account of a setting.
|
||||
|
||||
`makeXform` is NOT here, and is not being ported. It centres on the face oval's
|
||||
bounding box and zooms until the face is 80% of the raster height, so every
|
||||
vertex it touches carries a cropping decision made once, at analysis time, from
|
||||
one frame's landmarks. Geometry is stored in the node's own local space and the
|
||||
framing is a transform on a node; see \"What space geometry is in\" in
|
||||
docs/animation-model.md. So the face oval is not measured here either — its only
|
||||
consumers in the prototype were that transform and the placeholder plate
|
||||
outline, and the plate outline belongs to painting."
|
||||
(:require [arthur.domain.geom :as geom]
|
||||
[arthur.domain.landmarks :as lm]))
|
||||
|
||||
(defn pick
|
||||
"Landmarks `idx` out of one dense `frame`, converted to an ISOTROPIC space.
|
||||
|
||||
MediaPipe normalises x by image WIDTH and y by image HEIGHT, so its normalised
|
||||
space is anisotropic: for a 1080x1920 frame, one unit of x is 1080px and one
|
||||
unit of y is 1920px. Treating those as comparable stretches everything
|
||||
horizontally by H/W, and worse, makes fit-similarity fit a \"rotation\" in a
|
||||
sheared space, so head roll comes out subtly wrong as well.
|
||||
|
||||
Multiplying x by aspect = W/H converts to an isotropic space whose unit is one
|
||||
image height, so equal numbers mean equal pixels. Everything downstream -
|
||||
Procrustes, the similarity fit, the raster transform - depends on that."
|
||||
[frame idx aspect]
|
||||
(mapv (fn [i] (let [p (nth frame i)] {:x (* (:x p) aspect) :y (:y p)})) idx))
|
||||
|
||||
(defn residuals
|
||||
"RMS misfit per frame, in the isotropic space's units — one image height.
|
||||
|
||||
Taken against the transforms it is HANDED rather than against a fit of its own,
|
||||
so the same function serves the stage-3 reading and the parity diff. The
|
||||
prototype took it against the SMOOTHED transforms, which folds the smoothing
|
||||
error into a number whose whole job is to say whether the footage is
|
||||
stabilisable at all; `fit` takes it against the raw fit instead."
|
||||
[ref rigid tfs]
|
||||
(mapv (fn [rig tf] (geom/fit-residual tf rig ref)) rigid tfs))
|
||||
|
||||
(defn fit
|
||||
"Dense landmarks -> the rigid fit of every frame onto the shot's mean pose.
|
||||
|
||||
The reference is the Procrustes MEAN configuration over the shot, not frame
|
||||
zero, so no single frame's idiosyncrasies get baked into every other frame."
|
||||
[{:keys [aspect]} {:keys [dense]}]
|
||||
(let [rigid (mapv #(pick % lm/RIGID aspect) dense)
|
||||
ref (geom/procrustes-mean rigid)
|
||||
tfs (mapv #(geom/fit-similarity % ref) rigid)]
|
||||
{:ref ref
|
||||
;; Rigid landmarks in IMAGE space: the head-pose signal. Frame removal is
|
||||
;; decided from head motion, not from the mouth, so this has to survive the
|
||||
;; fit rather than being consumed by it.
|
||||
:rigid rigid
|
||||
:transforms tfs
|
||||
;; Residual rises with out-of-plane rotation, which no 2D similarity can
|
||||
;; remove. High values mean this section wants a different head plate.
|
||||
:residual (residuals ref rigid tfs)}))
|
||||
39
frontend/src/arthur/flow/measure/mouth.cljs
Normal file
39
frontend/src/arthur/flow/measure/mouth.cljs
Normal file
|
|
@ -0,0 +1,39 @@
|
|||
(ns arthur.flow.measure.mouth
|
||||
"Stage 3, the mouth: the lip rings with the head's motion taken out, and the
|
||||
aperture that decides whether there is an interior at all.
|
||||
|
||||
Head-local means the anchor's space, so `pick` is read from
|
||||
`arthur.flow.measure.anchor` rather than written a second time. A ring measured
|
||||
in a different space from the fit that placed it is not a failure anyone would
|
||||
see — it is a mouth that is quietly the wrong width.
|
||||
|
||||
The rings keep every slot of their table. A vertex budget is a stage-5 knob, and
|
||||
subsampling is a per-slot pick while stage 4's contour average is a per-slot
|
||||
average over time, so the two commute: smooth-then-subsample and
|
||||
subsample-then-smooth are the same numbers. That is what lets the vertex knob
|
||||
sit downstream of the smoothing knob instead of alongside it, and mouth-test
|
||||
asserts it rather than leaving it to look obvious.
|
||||
|
||||
Turning the aperture into `[:vis]` on `:mouth-in` is stage 5. This reports the
|
||||
measurement, not the decision."
|
||||
(:require [arthur.domain.geom :as geom]
|
||||
[arthur.domain.landmarks :as lm]
|
||||
[arthur.flow.measure.anchor :as anchor]))
|
||||
|
||||
(defn measure
|
||||
"Dense landmarks plus the anchor's transforms -> head-local lip rings.
|
||||
|
||||
`transforms` is whatever the caller has: the raw fit, or — normally — stage 4's
|
||||
conditioned one. Which it is belongs to the caller, because \"smooth the
|
||||
transform, never the contour\" only means anything while the two are separate."
|
||||
[{:keys [aspect]} {:keys [dense transforms]}]
|
||||
(let [local (fn [table]
|
||||
(mapv (fn [tf frame] (geom/apply-sim-all tf (anchor/pick frame table aspect)))
|
||||
transforms dense))]
|
||||
{:outer (local lm/LIPS-OUTER)
|
||||
:inner (local lm/LIPS-INNER)
|
||||
;; Not a separate measurement: APERTURE is slots 5 and 15 of LIPS_INNER, so
|
||||
;; this is the inner ring's own height read off as a scalar. Writing those
|
||||
;; two landmarks a second time is what gave the synthetic mouth a bowtie.
|
||||
:aperture (mapv (fn [[a b]] (js/Math.hypot (- (:x a) (:x b)) (- (:y a) (:y b))))
|
||||
(local lm/APERTURE))}))
|
||||
Loading…
Add table
Add a link
Reference in a new issue