noe/arthur
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Give features stable identity, eye pairs and per-feature presence

Browse source 
Step 8's data model, ahead of its controls. Nothing here is a UI.

domain/params holds every knob's definition once — default, applicable area,
value constraints and the areas a change would force to regenerate. flow/take's
literal knob map becomes a view of it, so the take's defaults and the future
parameter panel cannot drift apart.

domain/feature adds subjects, features and groups as document data the renderer
never reads. A feature ID is stable for the whole clip, across occlusion: a run
of visible frames is not a new identity. An eye pair is an explicit group of one
or two eyes of the same subject, so a profile view with one identified eye needs
no invented partner. Settings resolve area -> subject -> group -> feature, and
dropping an eye from a pair materialises its effective values first so playback
does not jump. scene/problems now validates all of it.

Presence becomes per-feature rather than per-subject. freeze's :absent predicate
takes a track as well as a frame, so one occluded eye can be absent while its
partner still has a value; a full-face miss still marks everything absent. A
manifest may annotate known gaps as one-based inclusive intervals, which ingest
expands into observation tracks before measurement. An unobserved eye then gets
no vote in the iris pairing and cannot steer the shared gaze — gaze falls back to
whichever eye is visible. Temporal filters still see a sample on every frame,
held from the last observed one, because the numbers are a rectangular buffer;
the state mask, not the buffer, is what says the frame has no value.

js/app.js gets the same occlusion lesson: leading nulls from a face that starts
occluded used to throw away the whole take, and the neutral frame could be chosen
from a held duplicate pose.

Parameter editing, scoped regeneration and a feature-level detector remain. Until
one exists, footage without annotations falls back to the full-face mask rather
than claiming occlusions it cannot see.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B87NVmiU36qQmN9gmFYnJ9
This commit is contained in:
Olive Vaughn 2026-09-27 22:44:36 -04:00
parent ccca93e233
commit 35ef150b48
18 changed files with 631 additions and 86 deletions
Download patch file Download diff file Expand all files Collapse all files

11
frontend/README.md
View file

@ -110,11 +110,20 @@ landmarks and the teeth from source pixels, then freezes them into channels,
and adds a button for the footage clip. Detection happens once when you load;
playback only resolves channels and paints. Frames without a detection remain
marked absent even though their neighbouring poses are used to condition the
track. The stage stays 320×200 regardless of the footage dimensions. Real
track. The scene now records stable subject and feature IDs and explicit eye
pairs; dense channels can mark one feature absent while another is observed.
Current MediaPipe loading supplies only the full-face detection mask. The stage
stays 320×200 regardless of the footage dimensions. Real
footage starts at the source picture rate. The **picture fps** buttons sample the
frozen roto at lower rates while the source track, duration and audio clock stay
unchanged. Picking frames to trace into cels is a separate future editing step.
For known occlusion intervals, an extracted manifest may add
`"feature-absence": {"eye-r": [[10, 14]]}`. Frame numbers are one-based and
inclusive, matching PNG filenames. The eye remains the same feature when it
reappears; the other eye and the mouth continue through the gap. This is an
input annotation, with no UI for editing it yet.
MediaPipe's JS, wasm and model are under `public/mediapipe/` and served locally.
No CDN is used by this app. See that directory's README for provenance.

85
frontend/src/arthur/domain/feature.cljs Normal file
View file

@ -0,0 +1,85 @@
(ns arthur.domain.feature
"Stable tracked identities and explicit eye-pair settings associations.
These maps are document data. Rendering only reads the nodes and channels."
(:require [arthur.domain.params :as params]))
(defn group-for [scene feature-id]
(first (filter (fn [[_ group]] (some #{feature-id} (:members group)))
(:groups scene))))
(defn effective-params
"Resolve static settings for one feature. A future parameter channel can
replace a scalar at this boundary without changing feature or pair identity."
[scene feature-id]
(let [{:keys [subject area params] :as feature} (get-in scene [:features feature-id])
[_ group] (group-for scene feature-id)]
(when-not feature
(throw (ex-info "unknown feature" {:feature feature-id})))
(merge (params/for-area :subject)
(get-in scene [:subjects subject :params])
(params/for-area area)
(:params group)
params)))
(defn remove-from-pair
"Keep the eye's current settings when its association is removed. Empty pairs
are removed; a one-eye pair remains valid and can acquire a partner later."
[scene feature-id]
(if-let [[group-id group] (group-for scene feature-id)]
(let [area (get-in scene [:features feature-id :area])
values (select-keys (effective-params scene feature-id)
(keys (params/for-area area)))
members (vec (remove #{feature-id} (:members group)))]
(-> scene
(assoc-in [:features feature-id :params] values)
(update :groups (fn [groups]
(if (seq members)
(assoc-in groups [group-id :members] members)
(dissoc groups group-id))))))
scene))
(defn problems
"Check identity references and pair membership before storing a scene."
[scene]
(let [subjects (:subjects scene)
features (:features scene)
groups (:groups scene)
memberships (mapcat (comp :members val) groups)
node-owners (mapcat (comp :nodes val) features)]
(vec
(concat
(for [[id s] subjects :when (not= id (:id s))]
(str "subject " (pr-str id) " has a different :id"))
(for [[id s] subjects
:when (not (params/valid-settings? :subject (or (:params s) {})))]
(str "subject " (pr-str id) " has invalid settings"))
(for [[id f] features :when (not= id (:id f))]
(str "feature " (pr-str id) " has a different :id"))
(for [[id f] features :when (not (contains? subjects (:subject f)))]
(str "feature " (pr-str id) " has no subject"))
(for [[id f] features :when (not (contains? (disj params/areas :subject) (:area f)))]
(str "feature " (pr-str id) " has an unknown area"))
(for [[id f] features
:when (not (params/valid-settings? (:area f) (or (:params f) {})))]
(str "feature " (pr-str id) " has invalid settings for " (pr-str (:area f))))
(for [[id f] features node-id (:nodes f)
:when (not (contains? (:nodes scene) node-id))]
(str "feature " (pr-str id) " refers to missing node " (pr-str node-id)))
(for [[id n] (frequencies node-owners) :when (> n 1)]
(str "node " (pr-str id) " belongs to more than one feature"))
(for [[id g] groups :when (not= id (:id g))]
(str "group " (pr-str id) " has a different :id"))
(for [[id g] groups
:when (not (and (= :eye-pair (:kind g))
(<= 1 (count (:members g)) 2)
(= (count (:members g)) (count (distinct (:members g))))))]
(str "group " (pr-str id) " must be an eye pair of one or two distinct eyes"))
(for [[id g] groups
:when (not (params/valid-settings? :eye (or (:params g) {})))]
(str "group " (pr-str id) " has invalid eye settings"))
(for [[id g] groups member (:members g)
:let [f (get features member)]
:when (not (and f (= :eye (:area f)) (= (:subject g) (:subject f))))]
(str "group " (pr-str id) " has an eye from another subject or an unknown feature"))
(for [[id n] (frequencies memberships) :when (> n 1)]
(str "feature " (pr-str id) " belongs to more than one group"))))))

61
frontend/src/arthur/domain/params.cljs Normal file
View file

@ -0,0 +1,61 @@
(ns arthur.domain.params
"Definitions for generated settings. Defaults, scope and value constraints live
here once; UI metadata and regeneration rules can be added to each entry.")
(def definitions
{:anchor-avg {:area :subject :default 2 :type :integer :min 0
:affects #{:head :mouth :eye :brow :teeth}}
:contour-avg {:area :subject :default 1 :type :integer :min 0
:affects #{:mouth :eye :brow}}
:verts {:area :mouth :default 8 :type :integer :min 4 :even? true}
:aperture-cut {:area :mouth :default 0.12 :type :number :min 0 :max 1}
:eye-verts {:area :eye :default 8 :type :integer :min 4 :even? true}
:blink-cut {:area :eye :default 0.13 :type :number :min 0}
:gaze-gain {:area :eye :default 1 :type :number :min 0}
:gaze-step {:area :eye :default 0.08 :type :number :min 0}
:iris-size {:area :eye :default 0.42 :type :number :min 0}
:lash-weight {:area :eye :default 0.06 :type :number :min 0}
:pupil-size {:area :eye :default 0.15 :type :number :min 0}
:brow-verts {:area :brow :default 6 :type :integer :min 4 :even? true}
:brow-gain {:area :brow :default 1 :type :number :min 0}
:brow-step {:area :brow :default 0.08 :type :number :min 0}
:brow-weight {:area :brow :default 0.05 :type :number :min 0}
:cavity-erode {:area :teeth :default 0.18 :type :number :min 0}
:tongue-reject {:area :teeth :default 0.18 :type :number :min 0}
:blob-grow {:area :teeth :default 0 :type :integer}
:top-bias {:area :teeth :default 0.6 :type :number}
:teeth-verts {:area :teeth :default 10 :type :integer :min 4}
:min-area {:area :teeth :default 12 :type :integer :min 0}
:teeth-on {:area :teeth :default 0.12 :type :number :min 0}
:teeth-smooth {:area :teeth :default 1 :type :integer :min 0}})
(def defaults
(into {} (map (fn [[id spec]] [id (:default spec)])) definitions))
(def areas #{:subject :mouth :eye :brow :teeth})
(defn affected-areas [id]
(when-let [spec (get definitions id)]
(or (:affects spec) #{(:area spec)})))
(defn for-area [wanted-area]
(into {} (keep (fn [[id {:keys [area default]}]]
(when (= wanted-area area) [id default])))
definitions))
(defn valid-value? [id value]
(when-let [{:keys [type min max] must-even? :even?} (get definitions id)]
(and (case type
:integer (integer? value)
:number (number? value)
false)
(or (nil? min) (<= min value))
(or (nil? max) (<= value max))
(or (not must-even?) (and (integer? value) (even? value))))))
(defn valid-settings? [area settings]
(and (map? settings)
(every? (fn [[id value]]
(and (= area (get-in definitions [id :area]))
(valid-value? id value)))
settings)))

3
frontend/src/arthur/domain/scene.cljs
View file

@ -28,6 +28,7 @@
vector of numbers, and they read the same way, which is what makes freezing
fill in the same channel rather than convert into a second format."
(:require [arthur.domain.channel :as ch]
[arthur.domain.feature :as feature]
[arthur.domain.node :as node]
[arthur.domain.palette :as pal]))
@ -451,8 +452,8 @@
(into (for [[id n] nodes
p (node/problems n)]
(str "node " (pr-str id) ": " p)))
(into (feature/problems scene))
(into (try
(doall (map #(depth nodes %) (keys nodes)))
nil
(catch :default e [(ex-message e)])))))))

3
frontend/src/arthur/events/footage.cljs
View file

@ -58,7 +58,8 @@
(defn- build-clip [manifest {:keys [dense detected dimensions interior missing first-real]}]
(let [[w h] dimensions
frozen (take/footage manifest {:dense dense :detected detected
:dimensions dimensions :interior interior})
:dimensions dimensions :interior interior
:presence (:presence manifest)})
scene (:scene frozen)]
(assoc (select-keys scene [:fps :frames :width :height])
:display-fps (:fps scene)

49
frontend/src/arthur/flow/condition/eyes.cljs
View file

@ -13,20 +13,47 @@
:y (/ (+ (:y a) (:y b)) 2)
:w (distance a b)}))
(defn- mean-width [lids]
(/ (reduce + (map (comp :w socket) lids)) (count lids)))
(defn- hold-observed
"Give temporal filters real samples at every index. The freeze mask still
marks the gap absent; these held values are only numeric placeholders."
[values observed fallback]
(let [values (vec values)
seed (or (first (keep-indexed (fn [i v]
(when (and (nth observed i) (some? v)) v))
values))
fallback (first values))]
(loop [f 0 last-value seed out []]
(if (= f (count values))
out
(let [v (nth values f)
next-value (if (and (nth observed f) (some? v)) v last-value)]
(recur (inc f) next-value (conj out next-value)))))))
(defn- mean-width [lids observed]
(let [valid (seq (keep-indexed (fn [i lid] (when (nth observed i) lid)) lids))
rings (or valid lids)]
(/ (reduce + (map (comp :w socket) rings)) (count rings))))
(defn apply-defaults
[{:keys [fps contour-avg blink-cut gaze-gain gaze-step iris-size
lash-weight pupil-size]} {:keys [lid-r lid-l open-r open-l gaze]
lash-weight pupil-size]} {:keys [lid-r lid-l open-r open-l gaze
observed-r observed-l gaze-observed]
:as measured}]
(let [fps (or fps 30)
right (condition/contours {:contour-avg contour-avg} lid-r)
left (condition/contours {:contour-avg contour-avg} lid-l)
w-r (mean-width right)
w-l (mean-width left)
observed-r (or observed-r (vec (repeat (count lid-r) true)))
observed-l (or observed-l (vec (repeat (count lid-l) true)))
gaze-observed (or gaze-observed (vec (repeat (count gaze) true)))
right (condition/contours {:contour-avg contour-avg}
(hold-observed lid-r observed-r nil))
left (condition/contours {:contour-avg contour-avg}
(hold-observed lid-l observed-l nil))
w-r (mean-width right observed-r)
w-l (mean-width left observed-l)
w (/ (+ w-r w-l) 2)
rest (condition/median-point gaze)
valid-gaze (seq (keep-indexed (fn [i point]
(when (nth gaze-observed i) point)) gaze))
gaze (hold-observed gaze gaze-observed {:x 0 :y 0})
rest (if valid-gaze (condition/median-point valid-gaze) {:x 0 :y 0})
px (mapv (fn [{:keys [x y]}]
{:x (* (- x (:x rest)) gaze-gain w)
:y (* (- y (:y rest)) gaze-gain w)}) gaze)
@ -42,8 +69,10 @@
:lid-r right :lid-l left
:lash-r (mapv #(ring/offset-ring % (* lash-weight w-r)) right)
:lash-l (mapv #(ring/offset-ring % (* lash-weight w-l)) left)
:shut-r (condition/resolve-blink open-r blink)
:shut-l (condition/resolve-blink open-l blink)
:shut-r (condition/resolve-blink
(hold-observed open-r observed-r nil) blink)
:shut-l (condition/resolve-blink
(hold-observed open-l observed-l nil) blink)
:iris-r (centres right) :iris-l (centres left)
:radius-r (* 0.5 iris-size w-r)
:radius-l (* 0.5 iris-size w-l)

81
frontend/src/arthur/flow/freeze.cljs
View file

@ -76,10 +76,9 @@
`:ctor` makes the array — a function and not a type, because `new` is not
something a value can carry — `:scale` is the fixed-point scale or nil, and
`:absent` an optional per-frame predicate. Absence is the SUBJECT's, not the
part's, so it is written into every track of the block: the mouth outline and
the mouth interior are one face, and a frame that face was not on has no pose
for either.
`:absent` an optional (track, frame) predicate. The state is per track, so one
occluded eye can be absent while its partner still has a value. A full-face
miss marks every track absent.
Written with `dotimes` and `aset` rather than as a fold, and that is the
exception rather than the rule in this codebase: the destination is a typed
@ -110,7 +109,7 @@
:track i :frame f :component k})))
q)
v))))
(when (and state (absent f))
(when (and state (absent i f))
(aset state (+ (* i nf) f) ch/absent-bit))))))
{:data data :state state :stride stride :frames nf :scale scale
:offsets (mapv #(* % nf stride) (range n))}))
@ -372,16 +371,24 @@
{:by by :analysis analysis
:params (merge {:anchor-avg anchor-avg :contour-avg contour-avg}
extra)})
eye-block (pack {:ctor #(js/Int16Array. %) :scale geom-scale :absent absent}
eye-block (pack {:ctor #(js/Int16Array. %) :scale geom-scale
:absent (when absent (fn [i f]
(absent (if (< i 2) :eye-r :eye-l) f)))}
(mapv #(rings->flat % eye-verts)
[(:lash-r eyes) (:lid-r eyes)
(:lash-l eyes) (:lid-l eyes)]))
iris-block (pack {:ctor #(js/Float32Array. %) :absent absent}
iris-block (pack {:ctor #(js/Float32Array. %)
:absent (when absent (fn [i f]
(absent (if (zero? i) :eye-r :eye-l) f)))}
[(:iris-r eyes) (:iris-l eyes)])
brow-block (pack {:ctor #(js/Int16Array. %) :scale geom-scale :absent absent}
brow-block (pack {:ctor #(js/Int16Array. %) :scale geom-scale
:absent (when absent (fn [i f]
(absent (if (zero? i) :brow-r :brow-l) f)))}
(mapv #(rings->flat % brow-verts)
[(:ring-r brows) (:ring-l brows)]))
brow-pos-block (pack {:ctor #(js/Float32Array. %) :absent absent}
brow-pos-block (pack {:ctor #(js/Float32Array. %)
:absent (when absent (fn [i f]
(absent (if (zero? i) :brow-r :brow-l) f)))}
[(:pos-r brows) (:pos-l brows)])
eye-node (fn [id z track]
{:id id :name (clojure.core/name id) :kind :poly :parent :head :z z
@ -434,11 +441,11 @@
"Freeze the pixel-derived radial contour under the mouth cavity. Missing
contours use the dense block's absence bit; contrast decides editable :vis."
[name {:keys [analysis teeth-verts cavity-erode tongue-reject blob-grow
top-bias teeth-on teeth-smooth]} detected
top-bias teeth-on teeth-smooth]} absent-feature
{:keys [contours shown]}]
(let [key (str name "/teeth")
absent #(or (nil? (nth contours %))
(and detected (not (nth detected %))))
absent (fn [_ f] (or (nil? (nth contours f))
(and absent-feature (absent-feature :teeth f))))
empty-points (vec (repeat (* 2 teeth-verts) 0))
values (mapv (fn [ring]
(if ring
@ -495,7 +502,9 @@
:eyes :brows the CONDITIONED head-local feature measurements
:teeth optional pixel-derived, conditioned radial contour
:aperture head-local aperture per frame
:detected optional per-frame booleans; absent frames get the state mask
:detected optional per-frame face booleans
:presence optional feature-id -> per-frame booleans; false means an
unobserved feature, even if the rest of the face was found
The node tree is the one docs/animation-model.md specifies, and the two groups
are two different things wanting the same transform:
@ -522,21 +531,32 @@
provenance having to be rebuilt."
[{:keys [name fps stage expose verts aperture-cut head kept
analysis anchor-avg contour-avg] :as params}
{:keys [transforms outer inner detected eyes brows teeth] :as inputs}]
{:keys [transforms outer inner detected presence eyes brows teeth] :as inputs}]
(let [nf (count outer)
absent (when detected #(not (nth detected % true)))
_ (doseq [[id track] presence]
(when (not= nf (count track))
(throw (ex-info "feature presence track must match the clip"
{:feature id :frames nf :actual (count track)}))))
absent (when (or detected presence)
(fn [id f]
(or (and detected (not (nth detected f true)))
(and (contains? presence id)
(not (nth (get presence id) f))))))
head-absent (when detected (fn [_ f] (not (nth detected f true))))
geom-k (str name "/geom")
head-k #(str name "/head-" %)
prov (fn [by extra]
{:by by :analysis analysis
:params (merge {:anchor-avg anchor-avg} extra)})
rings (pack {:ctor #(js/Int16Array. %) :scale geom-scale :absent absent}
rings (pack {:ctor #(js/Int16Array. %) :scale geom-scale
:absent (when absent (fn [_ f] (absent :mouth f)))}
[(rings->flat outer verts) (rings->flat inner verts)])
;; The anchor, inverted and split into its three components. Three blocks
;; and not one: they are three channels, they have three strides, and a
;; single block would need a per-component offset table to say so.
inv (mapv invert transforms)
xf (fn [f] (pack {:ctor #(js/Float32Array. %) :absent absent} [(mapv f inv)]))
xf (fn [f] (pack {:ctor #(js/Float32Array. %) :absent head-absent}
[(mapv f inv)]))
pos (xf (fn [t] [(:tx t) (:ty t)]))
rot (xf (fn [t] [(:theta t)]))
scale (xf (fn [t] [(:s t) (:s t)]))
@ -549,10 +569,31 @@
anchor-prov (prov :anchor/similarity nil)
roto (fn [by] (prov by {:verts verts :contour-avg contour-avg}))
features (when (and eyes brows) (feature-parts name absent params inputs))
interior (when teeth (interior-part name params detected teeth))
interior (when teeth (interior-part name params absent teeth))
scene {:name name
:frames nf
:fps fps
;; A gap changes channel state, never these IDs or pair links.
:subjects {:face-1 {:id :face-1 :params {}}}
:features (merge
{:mouth {:id :mouth :subject :face-1 :area :mouth
:nodes [:mouth :mouth-in] :params {}}}
(when features
{:eye-r {:id :eye-r :subject :face-1 :area :eye
:nodes [:eye-r :eye-r-in :iris-r :pupil-r] :params {}}
:eye-l {:id :eye-l :subject :face-1 :area :eye
:nodes [:eye-l :eye-l-in :iris-l :pupil-l] :params {}}
:brow-r {:id :brow-r :subject :face-1 :area :brow
:nodes [:brow-r] :params {}}
:brow-l {:id :brow-l :subject :face-1 :area :brow
:nodes [:brow-l] :params {}}})
(when teeth
{:teeth {:id :teeth :subject :face-1 :area :teeth
:nodes [:teeth] :params {}}}))
:groups (if features
{:eyes-1 {:id :eyes-1 :kind :eye-pair :subject :face-1
:members [:eye-r :eye-l] :params {}}}
{})
;; Stage dimensions, on the clip and not on the footage. See
;; face-placement: nothing below here knows the frame size.
:width (first stage)
@ -591,6 +632,10 @@
(prov :roto/mouth-aperture
{:aperture-cut aperture-cut}))}}}
(:nodes features) (:nodes interior))}
presence-check (doseq [id (keys presence)]
(when-not (contains? (:features scene) id)
(throw (ex-info "presence track names no feature in this scene"
{:feature id :features (keys (:features scene))}))))
;; Tier 2, behind a handle. The keys are descriptive because there is no
;; hashing yet; they become the blocks' sha256 when the backend arrives
;; and nothing above here changes, which is the point of a handle.

29
frontend/src/arthur/flow/ingest.cljs
View file

@ -2,6 +2,32 @@
"Read a pre-extracted take. The manifest owns timing and the exact frame count."
(:require [clojure.string :as str]))
(defn feature-presence
"Expand one-based, inclusive absence intervals from a manifest into boolean
observation tracks. Unknown features are rejected by freeze, where the scene
knows its feature IDs."
[frames absence]
(when (some? absence)
(when-not (map? absence)
(throw (ex-info "feature-absence must be a map of feature IDs to intervals"
{:feature-absence absence})))
(into {}
(map (fn [[id intervals]]
(when-not (and (keyword? id) (sequential? intervals)
(every? (fn [span]
(and (vector? span) (= 2 (count span))
(every? integer? span)
(<= 1 (first span) (second span) frames)))
intervals))
(throw (ex-info "feature-absence intervals must be [first last] source frames"
{:feature id :intervals intervals :frames frames})))
[id (mapv (fn [frame]
(not-any? (fn [[first-frame last-frame]]
(<= first-frame frame last-frame))
intervals))
(range 1 (inc frames)))]))
absence)))
(defn- valid-manifest [m]
(let [fps (js/Number (:fps m))
frames (js/Number (:frames m))]
@ -10,7 +36,8 @@
(string? (:dir m)) (seq (:dir m))
(string? (:audio m)) (seq (:audio m)))
(throw (ex-info "manifest.json needs fps, frames (1–900), dir and audio" {:manifest m})))
(assoc m :fps fps :frames frames)))
(assoc m :fps fps :frames frames
:presence (feature-presence frames (:feature-absence m)))))
(defn manifest!
[path]

48
frontend/src/arthur/flow/measure/eyes.cljs
View file

@ -16,22 +16,38 @@
(geom/apply-sim-all tf (anchor/pick frame table aspect)))
dense transforms))
(defn- iris-pair [iris-a iris-b corners-r]
(defn- observed-track [frames detected presence feature-id]
(let [part (get presence feature-id)]
(mapv (fn [f]
(and (or (nil? detected) (nth detected f))
(or (nil? part) (nth part f))))
(range frames))))
(defn- iris-pair [iris-a iris-b corners-r corners-l observed-r observed-l]
(when iris-a
;; An occluded eye may still get a plausible iris from MediaPipe. It gets no
;; vote. Either eye can establish the pairing when its partner is absent.
(let [votes (reduce +
(map (fn [a b [outer inner]]
(if (< (distance (first a) (midpoint outer inner))
(distance (first b) (midpoint outer inner)))
1 -1))
iris-a iris-b corners-r))]
(for [f (range (count iris-a))
[seen corners sign] [[(nth observed-r f) (nth corners-r f) 1]
[(nth observed-l f) (nth corners-l f) -1]]
:when seen]
(let [a (first (nth iris-a f))
b (first (nth iris-b f))
[outer inner] corners]
(if (< (distance a (midpoint outer inner))
(distance b (midpoint outer inner)))
sign (- sign)))))]
(if (pos? votes) {:right :a :left :b} {:right :b :left :a}))))
(defn measure
"All rings and iris coordinates use the anchor's head-local image-height unit.
Eye openness and gaze use each eye's rigid corner width as their unit. Iris
block identity is voted from proximity over the whole take."
[{:keys [aspect debug?]} {:keys [dense transforms]}]
block identity is voted from observed eyes over the whole take."
[{:keys [aspect debug?]} {:keys [dense transforms detected presence]}]
(let [track (partial local-track dense transforms aspect)
observed-r (observed-track (count dense) detected presence :eye-r)
observed-l (observed-track (count dense) detected presence :eye-l)
lid-r (track lm/EYE-R-RING)
lid-l (track lm/EYE-L-RING)
corners-r (track lm/EYE-R-CORNERS)
@ -41,7 +57,7 @@
has-iris? (every? #(> (count %) (last lm/IRIS-B)) dense)
iris-a (when has-iris? (track lm/IRIS-A))
iris-b (when has-iris? (track lm/IRIS-B))
pairing (iris-pair iris-a iris-b corners-r)
pairing (iris-pair iris-a iris-b corners-r corners-l observed-r observed-l)
iris-for (fn [side f]
(first (nth (if (= (get pairing side) :a) iris-a iris-b) f)))
openness (fn [corners lids]
@ -57,17 +73,25 @@
iris (iris-for side f)]
{:x (/ (- (:x iris) (:x c)) w)
:y (/ (- (:y iris) (:y c)) w)}))
r (one :right corners-r)
l (one :left corners-l)]
(midpoint r l)))
r (when (nth observed-r f) (one :right corners-r))
l (when (nth observed-l f) (one :left corners-l))]
(cond
(and r l) (midpoint r l)
r r
l l
:else nil)))
(range (count dense)))
(vec (repeat (count dense) {:x 0 :y 0})))]
{:lid-r lid-r :lid-l lid-l
:corners-r corners-r :corners-l corners-l
:open-r (openness corners-r lids-r)
:open-l (openness corners-l lids-l)
:observed-r observed-r :observed-l observed-l
:gaze-observed (if pairing (mapv #(or %1 %2) observed-r observed-l)
(vec (repeat (count dense) true)))
:gaze gaze :has-iris? (boolean pairing)
:iris-pair pairing
:debug (when debug? {:open-r (openness corners-r lids-r)
:open-l (openness corners-l lids-l)
:observed-r observed-r :observed-l observed-l
:gaze gaze :iris-pair pairing})}))

31
frontend/src/arthur/flow/take.cljs
View file

@ -9,24 +9,25 @@
[arthur.flow.measure.brows :as brows]
[arthur.flow.measure.eyes :as eyes]
[arthur.flow.measure.interior :as interior]
[arthur.flow.measure.mouth :as mouth]))
[arthur.flow.measure.mouth :as mouth]
[arthur.domain.params :as params]))
(def knobs
{:anchor-avg 2 :contour-avg 1 :verts 8 :aperture-cut 0.12
:eye-verts 8 :blink-cut 0.13 :gaze-gain 1 :gaze-step 0.08
:iris-size 0.42 :lash-weight 0.06 :pupil-size 0.15
:brow-verts 6 :brow-gain 1 :brow-step 0.08 :brow-weight 0.05
:cavity-erode 0.18 :tongue-reject 0.18 :blob-grow 0 :top-bias 0.6
:teeth-verts 10 :min-area 12 :teeth-on 0.12 :teeth-smooth 1})
(def knobs params/defaults)
(defn measure
"Condition the anchor before measuring rings through it."
[{:keys [aspect] :as params} {:keys [dense detected interior]}]
(let [fitted (anchor/fit {:aspect aspect} {:dense dense})
[{:keys [aspect] :as params} {:keys [dense detected interior presence]}]
(let [_ (doseq [[id track] presence]
(when (not= (count dense) (count track))
(throw (ex-info "feature presence track must match the source"
{:feature id :frames (count dense)
:actual (count track)}))))
fitted (anchor/fit {:aspect aspect} {:dense dense})
anchored (condition/anchor params fitted)
rings (mouth/measure {:aspect aspect}
{:dense dense :transforms (:transforms anchored)})
eye-data (eyes/measure params {:dense dense :transforms (:transforms anchored)})
eye-data (eyes/measure params {:dense dense :transforms (:transforms anchored)
:detected detected :presence presence})
brow-data (brows/measure params {:dense dense :transforms (:transforms anchored)})
inner-data (when interior
(condition-interior/apply-defaults
@ -41,7 +42,8 @@
:eyes (condition-eyes/apply-defaults params eye-data)
:brows (condition-brows/apply-defaults params brow-data eye-data)
:teeth inner-data
:detected detected)))
:detected detected
:presence presence)))
(defn build
[params inputs]
@ -51,11 +53,12 @@
"A real manifest and its detected landmarks through the same measurement and
freeze path as the synthetic take. The source cadence stays in :fps; picture
sampling is a root time map applied only after this artifact exists."
[manifest {:keys [dense detected dimensions interior]}]
[manifest {:keys [dense detected dimensions interior presence]}]
(let [[w h] dimensions
params (merge knobs
{:name "footage" :fps (:fps manifest) :aspect (/ w h)
:stage [320 200] :fit-motion? true
:expose 1 :head :as-filmed
:analysis (str "mediapipe:1.0.1/" (:source manifest))})]
(build params {:dense dense :detected detected :interior interior})))
(build params {:dense dense :detected detected :interior interior
:presence presence})))

54
frontend/test/arthur/domain/feature_test.cljs Normal file
View file

@ -0,0 +1,54 @@
(ns arthur.domain.feature-test
(:require [cljs.test :refer [deftest is]]
[arthur.domain.feature :as feature]
[arthur.domain.params :as params]))
(defn- nine-eyes []
(let [people (mapv #(keyword (str "person-" %)) (range 5))
eyes (mapv #(keyword (str "eye-" %)) (range 9))
members [[:eye-0 :eye-1] [:eye-2 :eye-3] [:eye-4 :eye-5]
[:eye-6 :eye-7] [:eye-8]]]
{:subjects (into {} (map (fn [id] [id {:id id :params {}}]) people))
:features (into {} (map-indexed (fn [i id]
[id {:id id :subject (nth people (quot i 2))
:area :eye :nodes [] :params {}}]) eyes))
:groups (into {} (map-indexed (fn [i ids]
(let [id (keyword (str "pair-" i))]
[id {:id id :kind :eye-pair
:subject (nth people i)
:members ids :params {}}])) members))
:nodes {}}))
(deftest five-people-can-have-nine-identified-eyes
(let [scene (nine-eyes)]
(is (empty? (feature/problems scene)))
(is (= [:eye-8] (get-in scene [:groups :pair-4 :members])))
(is (= :person-4 (get-in scene [:features :eye-8 :subject])))))
(deftest parameter-definitions-drive-the-current-take-defaults
(is (= #{:eye} (params/affected-areas :blink-cut)))
(is (= #{:head :mouth :eye :brow :teeth}
(params/affected-areas :anchor-avg)))
(is (params/valid-settings? :eye {:blink-cut 0.2}))
(is (not (params/valid-settings? :eye {:verts 8}))))
(deftest pair-values-are-shared-but-eye-overrides-are-independent
(let [scene (-> (nine-eyes)
(assoc-in [:groups :pair-0 :params :blink-cut] 0.2)
(assoc-in [:features :eye-1 :params :blink-cut] 0.3))]
(is (= 0.2 (:blink-cut (feature/effective-params scene :eye-0))))
(is (= 0.3 (:blink-cut (feature/effective-params scene :eye-1))))
(is (= (:eye-verts params/defaults)
(:eye-verts (feature/effective-params scene :eye-0))))
(let [before (feature/effective-params scene :eye-0)
unpaired (feature/remove-from-pair scene :eye-0)]
(is (= before (feature/effective-params unpaired :eye-0)))
(is (= [:eye-1] (get-in unpaired [:groups :pair-0 :members])))
(is (empty? (feature/problems unpaired))))))
(deftest a-pair-cannot-cross-subjects-or-own-an-eye-twice
(let [scene (nine-eyes)]
(is (seq (feature/problems
(assoc-in scene [:groups :pair-0 :members] [:eye-0 :eye-2]))))
(is (seq (feature/problems
(assoc-in scene [:groups :pair-1 :members] [:eye-0 :eye-3]))))))

67
frontend/test/arthur/flow/eye_occlusion_test.cljs Normal file
View file

@ -0,0 +1,67 @@
(ns arthur.flow.eye-occlusion-test
(:require [cljs.test :refer [deftest is]]
[arthur.demo.take :as take]
[arthur.domain.channel :as ch]
[arthur.domain.landmarks :as lm]
[arthur.domain.palette :as pal]
[arthur.domain.scene :as scene]
[arthur.flow.condition.eyes :as condition-eyes]
[arthur.flow.ingest :as ingest]
[arthur.flow.measure.eyes :as eyes]
[arthur.flow.take :as flow-take]))
(deftest an-unobserved-eye-cannot-steer-the-shared-gaze
(let [frame 11
params (assoc take/params :aspect 1)
presence (ingest/feature-presence take/frames {:eye-r [[10 14]]})
inputs {:dense @take/analysis
:transforms (:transforms @take/measured)
:presence presence}
clean (eyes/measure params inputs)
iris-ids (if (= :a (:right (:iris-pair clean))) lm/IRIS-A lm/IRIS-B)
bad-ids (distinct (concat lm/EYE-R-RING iris-ids))
corrupted (mapv (fn [i source]
(if (<= 9 i 13)
(reduce (fn [points id]
(update-in points [id :x] + 0.25))
source bad-ids)
source))
(range take/frames) @take/analysis)
masked (eyes/measure params (assoc inputs :dense corrupted))
unmasked (eyes/measure params
(dissoc (assoc inputs :dense corrupted) :presence))
drawn-clean (condition-eyes/apply-defaults params clean)
drawn-masked (condition-eyes/apply-defaults params masked)]
(is (= lm/NUM-LANDMARKS (count (nth corrupted frame))))
(is (:has-iris? masked) (str "pair " (:iris-pair masked)))
(is (false? (nth (:observed-r masked) frame)))
(is (true? (nth (:observed-l masked) frame)))
(is (= (nth (:gaze clean) frame) (nth (:gaze masked) frame)))
(is (not= (nth (:gaze clean) frame) (nth (:gaze unmasked) frame)))
(is (= (nth (:lid-r drawn-clean) frame)
(nth (:lid-r drawn-masked) frame)))
(is (= (nth (:iris-l drawn-clean) frame)
(nth (:iris-l drawn-masked) frame)))))
(deftest the-full-take-hides-only-the-annotated-eye-and-returns-to-the-same-id
(let [presence (ingest/feature-presence take/frames {:eye-r [[10 14]]})
{:keys [scene store]} (flow-take/build
(assoc take/params :aspect 1 :name "observed-gap")
{:dense @take/analysis :presence presence})
sample (fn [id frame]
(ch/value-at (get-in scene [:nodes id :channels [:geom :pts]])
frame store))]
(is (empty? (scene/problems scene)))
(is (= [:eye-r :eye-l] (get-in scene [:groups :eyes-1 :members])))
(doseq [f (range 9 14)]
(is (ch/nothing? (sample :eye-r f)))
(is (not (ch/nothing? (sample :eye-l f))))
(is (not (ch/nothing? (sample :mouth f)))))
(let [drawn (into #{} (map :node)
((scene/resolver scene store pal/index-of) 11))]
(is (not (contains? drawn :eye-r)))
(is (not (contains? drawn :iris-r)))
(is (contains? drawn :eye-l))
(is (contains? drawn :mouth)))
(is (not (ch/nothing? (sample :eye-r 8))))
(is (not (ch/nothing? (sample :eye-r 14))))))

32
frontend/test/arthur/flow/freeze_test.cljs
View file

@ -381,6 +381,27 @@
;; ---------------------------------------------------------------------------
;; presence is not visibility
(deftest an-eye-can-disappear-and-return-under-the-same-identity
(let [gap (set (range 40 60))
presence {:eye-r (mapv #(not (contains? gap %)) (range take/frames))}
c (freeze/clip (assoc take/params :name "one-eye-gappy")
(assoc @take/measured :presence presence))
sc (:scene c)
at (fn [id f]
(ch/value-at (get-in sc [:nodes id :channels [:geom :pts]]) f (:store c)))]
(is (= [:eye-r :eye-l] (get-in sc [:groups :eyes-1 :members])))
(is (= :eye-r (get-in sc [:features :eye-r :id])))
(is (not (ch/nothing? (at :eye-r 39))))
(is (ch/nothing? (at :eye-r 50)))
(is (not (ch/nothing? (at :eye-r 60))))
(is (not (ch/nothing? (at :eye-l 50))))
(is (not (ch/nothing? (at :mouth 50))))
(let [drawn-nodes (into #{} (map :node) ((scene/resolver sc (:store c) pal/index-of) 50))]
(is (not (contains? drawn-nodes :eye-r)))
(is (contains? drawn-nodes :eye-l))
(is (contains? drawn-nodes :mouth)))
(is (empty? (scene/problems sc)))))
(deftest an-undetected-frame-has-no-pose-at-all
;; A subject that was not on the frame has NO VALUE, which is different from a
;; part being switched off. The mask lands on every block of the freeze, so an
@ -447,10 +468,15 @@
;; beat are not. This is the numeric half of step 5's done-criterion; the other
;; half is a picture and lives in test/browser/take.mjs.
(let [locked (freeze/head-mode {:mode :locked} @clip)
shot (fn [f] (render locked f))
shot (fn [f]
(let [r (raster/make W H)
mouth (filter #(= :mouth (:node %)) (ops-at locked f))]
(raster/clear! r (get pal/index-of :bg))
(raster/draw-ops! r mouth)
(vec (array-seq (:buf r)))))
differ (fn [a b] (count (remove true? (map = a b))))]
;; Beat 1 is wide open and beat 3 is shut. Rendered with the head LOCKED, so
;; what differs is articulation and not the head wandering across the stage.
;; Beat 1 is wide open and beat 3 is shut. Compare only the mouth outline:
;; eyes and brows added in step 7 have their own motion within the beat.
(is (> (differ (shot 10) (shot 28)) 300)
"the open and the shut mouth rasterise the same")
;; And within a beat, on the exposure grid, it holds.

15
frontend/test/arthur/flow/ingest_test.cljs Normal file
View file

@ -0,0 +1,15 @@
(ns arthur.flow.ingest-test
(:require [cljs.test :refer [deftest is]]
[arthur.flow.ingest :as ingest]))
(deftest feature-absence-uses-the-source-frame-numbers
(let [tracks (ingest/feature-presence
8 {:eye-r [[3 5]] :eye-l [[1 1] [8 8]]})]
(is (= [true true false false false true true true] (:eye-r tracks)))
(is (= [false true true true true true true false] (:eye-l tracks)))))
(deftest malformed-feature-absence-fails-before-analysis
(doseq [bad [{:eye-r [[0 2]]}
{:eye-r [[6 9]]}
{:eye-r [[5 3]]}]]
(is (thrown? ExceptionInfo (ingest/feature-presence 8 bad)))))