noe/arthur
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Give features stable identity, eye pairs and per-feature presence

Browse source 
Step 8's data model, ahead of its controls. Nothing here is a UI.

domain/params holds every knob's definition once — default, applicable area,
value constraints and the areas a change would force to regenerate. flow/take's
literal knob map becomes a view of it, so the take's defaults and the future
parameter panel cannot drift apart.

domain/feature adds subjects, features and groups as document data the renderer
never reads. A feature ID is stable for the whole clip, across occlusion: a run
of visible frames is not a new identity. An eye pair is an explicit group of one
or two eyes of the same subject, so a profile view with one identified eye needs
no invented partner. Settings resolve area -> subject -> group -> feature, and
dropping an eye from a pair materialises its effective values first so playback
does not jump. scene/problems now validates all of it.

Presence becomes per-feature rather than per-subject. freeze's :absent predicate
takes a track as well as a frame, so one occluded eye can be absent while its
partner still has a value; a full-face miss still marks everything absent. A
manifest may annotate known gaps as one-based inclusive intervals, which ingest
expands into observation tracks before measurement. An unobserved eye then gets
no vote in the iris pairing and cannot steer the shared gaze — gaze falls back to
whichever eye is visible. Temporal filters still see a sample on every frame,
held from the last observed one, because the numbers are a rectangular buffer;
the state mask, not the buffer, is what says the frame has no value.

js/app.js gets the same occlusion lesson: leading nulls from a face that starts
occluded used to throw away the whole take, and the neutral frame could be chosen
from a held duplicate pose.

Parameter editing, scoped regeneration and a feature-level detector remain. Until
one exists, footage without annotations falls back to the full-face mask rather
than claiming occlusions it cannot see.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B87NVmiU36qQmN9gmFYnJ9
This commit is contained in:
Olive Vaughn 2026-09-27 22:44:36 -04:00
parent ccca93e233
commit 35ef150b48
18 changed files with 631 additions and 86 deletions
Download patch file Download diff file Expand all files Collapse all files

85
frontend/src/arthur/domain/feature.cljs Normal file
View file

@ -0,0 +1,85 @@
(ns arthur.domain.feature
"Stable tracked identities and explicit eye-pair settings associations.
These maps are document data. Rendering only reads the nodes and channels."
(:require [arthur.domain.params :as params]))
(defn group-for [scene feature-id]
(first (filter (fn [[_ group]] (some #{feature-id} (:members group)))
(:groups scene))))
(defn effective-params
"Resolve static settings for one feature. A future parameter channel can
replace a scalar at this boundary without changing feature or pair identity."
[scene feature-id]
(let [{:keys [subject area params] :as feature} (get-in scene [:features feature-id])
[_ group] (group-for scene feature-id)]
(when-not feature
(throw (ex-info "unknown feature" {:feature feature-id})))
(merge (params/for-area :subject)
(get-in scene [:subjects subject :params])
(params/for-area area)
(:params group)
params)))
(defn remove-from-pair
"Keep the eye's current settings when its association is removed. Empty pairs
are removed; a one-eye pair remains valid and can acquire a partner later."
[scene feature-id]
(if-let [[group-id group] (group-for scene feature-id)]
(let [area (get-in scene [:features feature-id :area])
values (select-keys (effective-params scene feature-id)
(keys (params/for-area area)))
members (vec (remove #{feature-id} (:members group)))]
(-> scene
(assoc-in [:features feature-id :params] values)
(update :groups (fn [groups]
(if (seq members)
(assoc-in groups [group-id :members] members)
(dissoc groups group-id))))))
scene))
(defn problems
"Check identity references and pair membership before storing a scene."
[scene]
(let [subjects (:subjects scene)
features (:features scene)
groups (:groups scene)
memberships (mapcat (comp :members val) groups)
node-owners (mapcat (comp :nodes val) features)]
(vec
(concat
(for [[id s] subjects :when (not= id (:id s))]
(str "subject " (pr-str id) " has a different :id"))
(for [[id s] subjects
:when (not (params/valid-settings? :subject (or (:params s) {})))]
(str "subject " (pr-str id) " has invalid settings"))
(for [[id f] features :when (not= id (:id f))]
(str "feature " (pr-str id) " has a different :id"))
(for [[id f] features :when (not (contains? subjects (:subject f)))]
(str "feature " (pr-str id) " has no subject"))
(for [[id f] features :when (not (contains? (disj params/areas :subject) (:area f)))]
(str "feature " (pr-str id) " has an unknown area"))
(for [[id f] features
:when (not (params/valid-settings? (:area f) (or (:params f) {})))]
(str "feature " (pr-str id) " has invalid settings for " (pr-str (:area f))))
(for [[id f] features node-id (:nodes f)
:when (not (contains? (:nodes scene) node-id))]
(str "feature " (pr-str id) " refers to missing node " (pr-str node-id)))
(for [[id n] (frequencies node-owners) :when (> n 1)]
(str "node " (pr-str id) " belongs to more than one feature"))
(for [[id g] groups :when (not= id (:id g))]
(str "group " (pr-str id) " has a different :id"))
(for [[id g] groups
:when (not (and (= :eye-pair (:kind g))
(<= 1 (count (:members g)) 2)
(= (count (:members g)) (count (distinct (:members g))))))]
(str "group " (pr-str id) " must be an eye pair of one or two distinct eyes"))
(for [[id g] groups
:when (not (params/valid-settings? :eye (or (:params g) {})))]
(str "group " (pr-str id) " has invalid eye settings"))
(for [[id g] groups member (:members g)
:let [f (get features member)]
:when (not (and f (= :eye (:area f)) (= (:subject g) (:subject f))))]
(str "group " (pr-str id) " has an eye from another subject or an unknown feature"))
(for [[id n] (frequencies memberships) :when (> n 1)]
(str "feature " (pr-str id) " belongs to more than one group"))))))

61
frontend/src/arthur/domain/params.cljs Normal file
View file

@ -0,0 +1,61 @@
(ns arthur.domain.params
"Definitions for generated settings. Defaults, scope and value constraints live
here once; UI metadata and regeneration rules can be added to each entry.")
(def definitions
{:anchor-avg {:area :subject :default 2 :type :integer :min 0
:affects #{:head :mouth :eye :brow :teeth}}
:contour-avg {:area :subject :default 1 :type :integer :min 0
:affects #{:mouth :eye :brow}}
:verts {:area :mouth :default 8 :type :integer :min 4 :even? true}
:aperture-cut {:area :mouth :default 0.12 :type :number :min 0 :max 1}
:eye-verts {:area :eye :default 8 :type :integer :min 4 :even? true}
:blink-cut {:area :eye :default 0.13 :type :number :min 0}
:gaze-gain {:area :eye :default 1 :type :number :min 0}
:gaze-step {:area :eye :default 0.08 :type :number :min 0}
:iris-size {:area :eye :default 0.42 :type :number :min 0}
:lash-weight {:area :eye :default 0.06 :type :number :min 0}
:pupil-size {:area :eye :default 0.15 :type :number :min 0}
:brow-verts {:area :brow :default 6 :type :integer :min 4 :even? true}
:brow-gain {:area :brow :default 1 :type :number :min 0}
:brow-step {:area :brow :default 0.08 :type :number :min 0}
:brow-weight {:area :brow :default 0.05 :type :number :min 0}
:cavity-erode {:area :teeth :default 0.18 :type :number :min 0}
:tongue-reject {:area :teeth :default 0.18 :type :number :min 0}
:blob-grow {:area :teeth :default 0 :type :integer}
:top-bias {:area :teeth :default 0.6 :type :number}
:teeth-verts {:area :teeth :default 10 :type :integer :min 4}
:min-area {:area :teeth :default 12 :type :integer :min 0}
:teeth-on {:area :teeth :default 0.12 :type :number :min 0}
:teeth-smooth {:area :teeth :default 1 :type :integer :min 0}})
(def defaults
(into {} (map (fn [[id spec]] [id (:default spec)])) definitions))
(def areas #{:subject :mouth :eye :brow :teeth})
(defn affected-areas [id]
(when-let [spec (get definitions id)]
(or (:affects spec) #{(:area spec)})))
(defn for-area [wanted-area]
(into {} (keep (fn [[id {:keys [area default]}]]
(when (= wanted-area area) [id default])))
definitions))
(defn valid-value? [id value]
(when-let [{:keys [type min max] must-even? :even?} (get definitions id)]
(and (case type
:integer (integer? value)
:number (number? value)
false)
(or (nil? min) (<= min value))
(or (nil? max) (<= value max))
(or (not must-even?) (and (integer? value) (even? value))))))
(defn valid-settings? [area settings]
(and (map? settings)
(every? (fn [[id value]]
(and (= area (get-in definitions [id :area]))
(valid-value? id value)))
settings)))

3
frontend/src/arthur/domain/scene.cljs
View file

@ -28,6 +28,7 @@
vector of numbers, and they read the same way, which is what makes freezing
fill in the same channel rather than convert into a second format."
(:require [arthur.domain.channel :as ch]
[arthur.domain.feature :as feature]
[arthur.domain.node :as node]
[arthur.domain.palette :as pal]))
@ -451,8 +452,8 @@
(into (for [[id n] nodes
p (node/problems n)]
(str "node " (pr-str id) ": " p)))
(into (feature/problems scene))
(into (try
(doall (map #(depth nodes %) (keys nodes)))
nil
(catch :default e [(ex-message e)])))))))

3
frontend/src/arthur/events/footage.cljs
View file

@ -58,7 +58,8 @@
(defn- build-clip [manifest {:keys [dense detected dimensions interior missing first-real]}]
(let [[w h] dimensions
frozen (take/footage manifest {:dense dense :detected detected
:dimensions dimensions :interior interior})
:dimensions dimensions :interior interior
:presence (:presence manifest)})
scene (:scene frozen)]
(assoc (select-keys scene [:fps :frames :width :height])
:display-fps (:fps scene)

49
frontend/src/arthur/flow/condition/eyes.cljs
View file

@ -13,20 +13,47 @@
:y (/ (+ (:y a) (:y b)) 2)
:w (distance a b)}))
(defn- mean-width [lids]
(/ (reduce + (map (comp :w socket) lids)) (count lids)))
(defn- hold-observed
"Give temporal filters real samples at every index. The freeze mask still
marks the gap absent; these held values are only numeric placeholders."
[values observed fallback]
(let [values (vec values)
seed (or (first (keep-indexed (fn [i v]
(when (and (nth observed i) (some? v)) v))
values))
fallback (first values))]
(loop [f 0 last-value seed out []]
(if (= f (count values))
out
(let [v (nth values f)
next-value (if (and (nth observed f) (some? v)) v last-value)]
(recur (inc f) next-value (conj out next-value)))))))
(defn- mean-width [lids observed]
(let [valid (seq (keep-indexed (fn [i lid] (when (nth observed i) lid)) lids))
rings (or valid lids)]
(/ (reduce + (map (comp :w socket) rings)) (count rings))))
(defn apply-defaults
[{:keys [fps contour-avg blink-cut gaze-gain gaze-step iris-size
lash-weight pupil-size]} {:keys [lid-r lid-l open-r open-l gaze]
lash-weight pupil-size]} {:keys [lid-r lid-l open-r open-l gaze
observed-r observed-l gaze-observed]
:as measured}]
(let [fps (or fps 30)
right (condition/contours {:contour-avg contour-avg} lid-r)
left (condition/contours {:contour-avg contour-avg} lid-l)
w-r (mean-width right)
w-l (mean-width left)
observed-r (or observed-r (vec (repeat (count lid-r) true)))
observed-l (or observed-l (vec (repeat (count lid-l) true)))
gaze-observed (or gaze-observed (vec (repeat (count gaze) true)))
right (condition/contours {:contour-avg contour-avg}
(hold-observed lid-r observed-r nil))
left (condition/contours {:contour-avg contour-avg}
(hold-observed lid-l observed-l nil))
w-r (mean-width right observed-r)
w-l (mean-width left observed-l)
w (/ (+ w-r w-l) 2)
rest (condition/median-point gaze)
valid-gaze (seq (keep-indexed (fn [i point]
(when (nth gaze-observed i) point)) gaze))
gaze (hold-observed gaze gaze-observed {:x 0 :y 0})
rest (if valid-gaze (condition/median-point valid-gaze) {:x 0 :y 0})
px (mapv (fn [{:keys [x y]}]
{:x (* (- x (:x rest)) gaze-gain w)
:y (* (- y (:y rest)) gaze-gain w)}) gaze)
@ -42,8 +69,10 @@
:lid-r right :lid-l left
:lash-r (mapv #(ring/offset-ring % (* lash-weight w-r)) right)
:lash-l (mapv #(ring/offset-ring % (* lash-weight w-l)) left)
:shut-r (condition/resolve-blink open-r blink)
:shut-l (condition/resolve-blink open-l blink)
:shut-r (condition/resolve-blink
(hold-observed open-r observed-r nil) blink)
:shut-l (condition/resolve-blink
(hold-observed open-l observed-l nil) blink)
:iris-r (centres right) :iris-l (centres left)
:radius-r (* 0.5 iris-size w-r)
:radius-l (* 0.5 iris-size w-l)

81
frontend/src/arthur/flow/freeze.cljs
View file

@ -76,10 +76,9 @@
`:ctor` makes the array — a function and not a type, because `new` is not
something a value can carry — `:scale` is the fixed-point scale or nil, and
`:absent` an optional per-frame predicate. Absence is the SUBJECT's, not the
part's, so it is written into every track of the block: the mouth outline and
the mouth interior are one face, and a frame that face was not on has no pose
for either.
`:absent` an optional (track, frame) predicate. The state is per track, so one
occluded eye can be absent while its partner still has a value. A full-face
miss marks every track absent.
Written with `dotimes` and `aset` rather than as a fold, and that is the
exception rather than the rule in this codebase: the destination is a typed
@ -110,7 +109,7 @@
:track i :frame f :component k})))
q)
v))))
(when (and state (absent f))
(when (and state (absent i f))
(aset state (+ (* i nf) f) ch/absent-bit))))))
{:data data :state state :stride stride :frames nf :scale scale
:offsets (mapv #(* % nf stride) (range n))}))
@ -372,16 +371,24 @@
{:by by :analysis analysis
:params (merge {:anchor-avg anchor-avg :contour-avg contour-avg}
extra)})
eye-block (pack {:ctor #(js/Int16Array. %) :scale geom-scale :absent absent}
eye-block (pack {:ctor #(js/Int16Array. %) :scale geom-scale
:absent (when absent (fn [i f]
(absent (if (< i 2) :eye-r :eye-l) f)))}
(mapv #(rings->flat % eye-verts)
[(:lash-r eyes) (:lid-r eyes)
(:lash-l eyes) (:lid-l eyes)]))
iris-block (pack {:ctor #(js/Float32Array. %) :absent absent}
iris-block (pack {:ctor #(js/Float32Array. %)
:absent (when absent (fn [i f]
(absent (if (zero? i) :eye-r :eye-l) f)))}
[(:iris-r eyes) (:iris-l eyes)])
brow-block (pack {:ctor #(js/Int16Array. %) :scale geom-scale :absent absent}
brow-block (pack {:ctor #(js/Int16Array. %) :scale geom-scale
:absent (when absent (fn [i f]
(absent (if (zero? i) :brow-r :brow-l) f)))}
(mapv #(rings->flat % brow-verts)
[(:ring-r brows) (:ring-l brows)]))
brow-pos-block (pack {:ctor #(js/Float32Array. %) :absent absent}
brow-pos-block (pack {:ctor #(js/Float32Array. %)
:absent (when absent (fn [i f]
(absent (if (zero? i) :brow-r :brow-l) f)))}
[(:pos-r brows) (:pos-l brows)])
eye-node (fn [id z track]
{:id id :name (clojure.core/name id) :kind :poly :parent :head :z z
@ -434,11 +441,11 @@
"Freeze the pixel-derived radial contour under the mouth cavity. Missing
contours use the dense block's absence bit; contrast decides editable :vis."
[name {:keys [analysis teeth-verts cavity-erode tongue-reject blob-grow
top-bias teeth-on teeth-smooth]} detected
top-bias teeth-on teeth-smooth]} absent-feature
{:keys [contours shown]}]
(let [key (str name "/teeth")
absent #(or (nil? (nth contours %))
(and detected (not (nth detected %))))
absent (fn [_ f] (or (nil? (nth contours f))
(and absent-feature (absent-feature :teeth f))))
empty-points (vec (repeat (* 2 teeth-verts) 0))
values (mapv (fn [ring]
(if ring
@ -495,7 +502,9 @@
:eyes :brows the CONDITIONED head-local feature measurements
:teeth optional pixel-derived, conditioned radial contour
:aperture head-local aperture per frame
:detected optional per-frame booleans; absent frames get the state mask
:detected optional per-frame face booleans
:presence optional feature-id -> per-frame booleans; false means an
unobserved feature, even if the rest of the face was found
The node tree is the one docs/animation-model.md specifies, and the two groups
are two different things wanting the same transform:
@ -522,21 +531,32 @@
provenance having to be rebuilt."
[{:keys [name fps stage expose verts aperture-cut head kept
analysis anchor-avg contour-avg] :as params}
{:keys [transforms outer inner detected eyes brows teeth] :as inputs}]
{:keys [transforms outer inner detected presence eyes brows teeth] :as inputs}]
(let [nf (count outer)
absent (when detected #(not (nth detected % true)))
_ (doseq [[id track] presence]
(when (not= nf (count track))
(throw (ex-info "feature presence track must match the clip"
{:feature id :frames nf :actual (count track)}))))
absent (when (or detected presence)
(fn [id f]
(or (and detected (not (nth detected f true)))
(and (contains? presence id)
(not (nth (get presence id) f))))))
head-absent (when detected (fn [_ f] (not (nth detected f true))))
geom-k (str name "/geom")
head-k #(str name "/head-" %)
prov (fn [by extra]
{:by by :analysis analysis
:params (merge {:anchor-avg anchor-avg} extra)})
rings (pack {:ctor #(js/Int16Array. %) :scale geom-scale :absent absent}
rings (pack {:ctor #(js/Int16Array. %) :scale geom-scale
:absent (when absent (fn [_ f] (absent :mouth f)))}
[(rings->flat outer verts) (rings->flat inner verts)])
;; The anchor, inverted and split into its three components. Three blocks
;; and not one: they are three channels, they have three strides, and a
;; single block would need a per-component offset table to say so.
inv (mapv invert transforms)
xf (fn [f] (pack {:ctor #(js/Float32Array. %) :absent absent} [(mapv f inv)]))
xf (fn [f] (pack {:ctor #(js/Float32Array. %) :absent head-absent}
[(mapv f inv)]))
pos (xf (fn [t] [(:tx t) (:ty t)]))
rot (xf (fn [t] [(:theta t)]))
scale (xf (fn [t] [(:s t) (:s t)]))
@ -549,10 +569,31 @@
anchor-prov (prov :anchor/similarity nil)
roto (fn [by] (prov by {:verts verts :contour-avg contour-avg}))
features (when (and eyes brows) (feature-parts name absent params inputs))
interior (when teeth (interior-part name params detected teeth))
interior (when teeth (interior-part name params absent teeth))
scene {:name name
:frames nf
:fps fps
;; A gap changes channel state, never these IDs or pair links.
:subjects {:face-1 {:id :face-1 :params {}}}
:features (merge
{:mouth {:id :mouth :subject :face-1 :area :mouth
:nodes [:mouth :mouth-in] :params {}}}
(when features
{:eye-r {:id :eye-r :subject :face-1 :area :eye
:nodes [:eye-r :eye-r-in :iris-r :pupil-r] :params {}}
:eye-l {:id :eye-l :subject :face-1 :area :eye
:nodes [:eye-l :eye-l-in :iris-l :pupil-l] :params {}}
:brow-r {:id :brow-r :subject :face-1 :area :brow
:nodes [:brow-r] :params {}}
:brow-l {:id :brow-l :subject :face-1 :area :brow
:nodes [:brow-l] :params {}}})
(when teeth
{:teeth {:id :teeth :subject :face-1 :area :teeth
:nodes [:teeth] :params {}}}))
:groups (if features
{:eyes-1 {:id :eyes-1 :kind :eye-pair :subject :face-1
:members [:eye-r :eye-l] :params {}}}
{})
;; Stage dimensions, on the clip and not on the footage. See
;; face-placement: nothing below here knows the frame size.
:width (first stage)
@ -591,6 +632,10 @@
(prov :roto/mouth-aperture
{:aperture-cut aperture-cut}))}}}
(:nodes features) (:nodes interior))}
presence-check (doseq [id (keys presence)]
(when-not (contains? (:features scene) id)
(throw (ex-info "presence track names no feature in this scene"
{:feature id :features (keys (:features scene))}))))
;; Tier 2, behind a handle. The keys are descriptive because there is no
;; hashing yet; they become the blocks' sha256 when the backend arrives
;; and nothing above here changes, which is the point of a handle.

29
frontend/src/arthur/flow/ingest.cljs
View file

@ -2,6 +2,32 @@
"Read a pre-extracted take. The manifest owns timing and the exact frame count."
(:require [clojure.string :as str]))
(defn feature-presence
"Expand one-based, inclusive absence intervals from a manifest into boolean
observation tracks. Unknown features are rejected by freeze, where the scene
knows its feature IDs."
[frames absence]
(when (some? absence)
(when-not (map? absence)
(throw (ex-info "feature-absence must be a map of feature IDs to intervals"
{:feature-absence absence})))
(into {}
(map (fn [[id intervals]]
(when-not (and (keyword? id) (sequential? intervals)
(every? (fn [span]
(and (vector? span) (= 2 (count span))
(every? integer? span)
(<= 1 (first span) (second span) frames)))
intervals))
(throw (ex-info "feature-absence intervals must be [first last] source frames"
{:feature id :intervals intervals :frames frames})))
[id (mapv (fn [frame]
(not-any? (fn [[first-frame last-frame]]
(<= first-frame frame last-frame))
intervals))
(range 1 (inc frames)))]))
absence)))
(defn- valid-manifest [m]
(let [fps (js/Number (:fps m))
frames (js/Number (:frames m))]
@ -10,7 +36,8 @@
(string? (:dir m)) (seq (:dir m))
(string? (:audio m)) (seq (:audio m)))
(throw (ex-info "manifest.json needs fps, frames (1–900), dir and audio" {:manifest m})))
(assoc m :fps fps :frames frames)))
(assoc m :fps fps :frames frames
:presence (feature-presence frames (:feature-absence m)))))
(defn manifest!
[path]

48
frontend/src/arthur/flow/measure/eyes.cljs
View file

@ -16,22 +16,38 @@
(geom/apply-sim-all tf (anchor/pick frame table aspect)))
dense transforms))
(defn- iris-pair [iris-a iris-b corners-r]
(defn- observed-track [frames detected presence feature-id]
(let [part (get presence feature-id)]
(mapv (fn [f]
(and (or (nil? detected) (nth detected f))
(or (nil? part) (nth part f))))
(range frames))))
(defn- iris-pair [iris-a iris-b corners-r corners-l observed-r observed-l]
(when iris-a
;; An occluded eye may still get a plausible iris from MediaPipe. It gets no
;; vote. Either eye can establish the pairing when its partner is absent.
(let [votes (reduce +
(map (fn [a b [outer inner]]
(if (< (distance (first a) (midpoint outer inner))
(distance (first b) (midpoint outer inner)))
1 -1))
iris-a iris-b corners-r))]
(for [f (range (count iris-a))
[seen corners sign] [[(nth observed-r f) (nth corners-r f) 1]
[(nth observed-l f) (nth corners-l f) -1]]
:when seen]
(let [a (first (nth iris-a f))
b (first (nth iris-b f))
[outer inner] corners]
(if (< (distance a (midpoint outer inner))
(distance b (midpoint outer inner)))
sign (- sign)))))]
(if (pos? votes) {:right :a :left :b} {:right :b :left :a}))))
(defn measure
"All rings and iris coordinates use the anchor's head-local image-height unit.
Eye openness and gaze use each eye's rigid corner width as their unit. Iris
block identity is voted from proximity over the whole take."
[{:keys [aspect debug?]} {:keys [dense transforms]}]
block identity is voted from observed eyes over the whole take."
[{:keys [aspect debug?]} {:keys [dense transforms detected presence]}]
(let [track (partial local-track dense transforms aspect)
observed-r (observed-track (count dense) detected presence :eye-r)
observed-l (observed-track (count dense) detected presence :eye-l)
lid-r (track lm/EYE-R-RING)
lid-l (track lm/EYE-L-RING)
corners-r (track lm/EYE-R-CORNERS)
@ -41,7 +57,7 @@
has-iris? (every? #(> (count %) (last lm/IRIS-B)) dense)
iris-a (when has-iris? (track lm/IRIS-A))
iris-b (when has-iris? (track lm/IRIS-B))
pairing (iris-pair iris-a iris-b corners-r)
pairing (iris-pair iris-a iris-b corners-r corners-l observed-r observed-l)
iris-for (fn [side f]
(first (nth (if (= (get pairing side) :a) iris-a iris-b) f)))
openness (fn [corners lids]
@ -57,17 +73,25 @@
iris (iris-for side f)]
{:x (/ (- (:x iris) (:x c)) w)
:y (/ (- (:y iris) (:y c)) w)}))
r (one :right corners-r)
l (one :left corners-l)]
(midpoint r l)))
r (when (nth observed-r f) (one :right corners-r))
l (when (nth observed-l f) (one :left corners-l))]
(cond
(and r l) (midpoint r l)
r r
l l
:else nil)))
(range (count dense)))
(vec (repeat (count dense) {:x 0 :y 0})))]
{:lid-r lid-r :lid-l lid-l
:corners-r corners-r :corners-l corners-l
:open-r (openness corners-r lids-r)
:open-l (openness corners-l lids-l)
:observed-r observed-r :observed-l observed-l
:gaze-observed (if pairing (mapv #(or %1 %2) observed-r observed-l)
(vec (repeat (count dense) true)))
:gaze gaze :has-iris? (boolean pairing)
:iris-pair pairing
:debug (when debug? {:open-r (openness corners-r lids-r)
:open-l (openness corners-l lids-l)
:observed-r observed-r :observed-l observed-l
:gaze gaze :iris-pair pairing})}))

31
frontend/src/arthur/flow/take.cljs
View file

@ -9,24 +9,25 @@
[arthur.flow.measure.brows :as brows]
[arthur.flow.measure.eyes :as eyes]
[arthur.flow.measure.interior :as interior]
[arthur.flow.measure.mouth :as mouth]))
[arthur.flow.measure.mouth :as mouth]
[arthur.domain.params :as params]))
(def knobs
{:anchor-avg 2 :contour-avg 1 :verts 8 :aperture-cut 0.12
:eye-verts 8 :blink-cut 0.13 :gaze-gain 1 :gaze-step 0.08
:iris-size 0.42 :lash-weight 0.06 :pupil-size 0.15
:brow-verts 6 :brow-gain 1 :brow-step 0.08 :brow-weight 0.05
:cavity-erode 0.18 :tongue-reject 0.18 :blob-grow 0 :top-bias 0.6
:teeth-verts 10 :min-area 12 :teeth-on 0.12 :teeth-smooth 1})
(def knobs params/defaults)
(defn measure
"Condition the anchor before measuring rings through it."
[{:keys [aspect] :as params} {:keys [dense detected interior]}]
(let [fitted (anchor/fit {:aspect aspect} {:dense dense})
[{:keys [aspect] :as params} {:keys [dense detected interior presence]}]
(let [_ (doseq [[id track] presence]
(when (not= (count dense) (count track))
(throw (ex-info "feature presence track must match the source"
{:feature id :frames (count dense)
:actual (count track)}))))
fitted (anchor/fit {:aspect aspect} {:dense dense})
anchored (condition/anchor params fitted)
rings (mouth/measure {:aspect aspect}
{:dense dense :transforms (:transforms anchored)})
eye-data (eyes/measure params {:dense dense :transforms (:transforms anchored)})
eye-data (eyes/measure params {:dense dense :transforms (:transforms anchored)
:detected detected :presence presence})
brow-data (brows/measure params {:dense dense :transforms (:transforms anchored)})
inner-data (when interior
(condition-interior/apply-defaults
@ -41,7 +42,8 @@
:eyes (condition-eyes/apply-defaults params eye-data)
:brows (condition-brows/apply-defaults params brow-data eye-data)
:teeth inner-data
:detected detected)))
:detected detected
:presence presence)))
(defn build
[params inputs]
@ -51,11 +53,12 @@
"A real manifest and its detected landmarks through the same measurement and
freeze path as the synthetic take. The source cadence stays in :fps; picture
sampling is a root time map applied only after this artifact exists."
[manifest {:keys [dense detected dimensions interior]}]
[manifest {:keys [dense detected dimensions interior presence]}]
(let [[w h] dimensions
params (merge knobs
{:name "footage" :fps (:fps manifest) :aspect (/ w h)
:stage [320 200] :fit-motion? true
:expose 1 :head :as-filmed
:analysis (str "mediapipe:1.0.1/" (:source manifest))})]
(build params {:dense dense :detected detected :interior interior})))
(build params {:dense dense :detected detected :interior interior
:presence presence})))