diff --git a/README.md b/README.md index c837083..7ab6dcb 100644 --- a/README.md +++ b/README.md @@ -18,7 +18,8 @@ modern conveniences belong in the workflow, not the output. See The active port has reached [step 7 of the port plan](docs/port-plan.md): it plays the synthetic take and can load extracted real footage with mouth, eyes, brows, -and pixel-derived teeth. +and pixel-derived teeth. The step 8 data model now represents persistent feature +IDs, eye pairs and feature-level observation gaps; its controls are still pending. See [frontend/README.md](frontend/README.md) for setup and the **load frames** workflow. The rest of this README describes the older JS prototype, which still runs separately on port 8777. diff --git a/docs/animation-model.md b/docs/animation-model.md index 1988fdf..ec02118 100644 --- a/docs/animation-model.md +++ b/docs/animation-model.md @@ -79,6 +79,59 @@ this way. over which the node exists at all. Distinct from a `[:vis]` channel, which blinks an existing node on and off. +### Subjects and tracked features + +Scene nodes describe drawings, not tracking identity. A scene may also carry a +flat `:features` map. A feature ID stays stable for the whole clip, including +frames where that feature is occluded and later reappears: + +```clojure +:subjects {:face-1 {:id :face-1}} +:features +{:eye-r {:id :eye-r :subject :face-1 :area :eye + :nodes [:eye-r :eye-r-in :iris-r :pupil-r] :params {}} + :eye-l {:id :eye-l :subject :face-1 :area :eye + :nodes [:eye-l :eye-l-in :iris-l :pupil-l] :params {}} + :mouth {:id :mouth :subject :face-1 :area :mouth + :nodes [:mouth :mouth-in :teeth] :params {}}} +:groups +{:eyes-1 {:id :eyes-1 :kind :eye-pair :subject :face-1 + :members [:eye-r :eye-l] :params {}}} +``` + +An eye pair is an explicit relationship between one or two eyes of the **same +subject**. It may have one member when only one eye has been identified; it does +not invent a second eye. Five subjects with nine identified eyes can have four +two-member pairs and one one-member pair. Each eye still has its own feature ID +and presence track. A group is a settings association, not a scene parent or a +tracking ID. Membership lives only on the group, avoiding a second pointer on +the feature that could disagree with it. + +Each feature resolves settings from its area's definitions, then its group, +then its own `:params`. An eye can therefore inherit a pair setting or override +it without changing its partner. Removing it from a pair copies its effective +values into the feature first, so the result does not jump. Feature identity +and pair membership are clip-wide; a future parameter track can vary values +over time without splitting a feature at an observation gap. + +Parameter definitions live in one registry: key, default, applicable area, +value constraints and affected areas. The registry supplies the take's defaults +today. The parameter UI and regeneration from edited values are later work. + +Dense channel state records whether a measurement exists **for that feature on +that frame**. Occlusion means absent data on that frame, not a false `[:vis]` +value and not the end of the feature's identity. A full-face detection failure +makes all its features absent. A single occluded eye need only make that eye's +channels absent. Footage can carry explicit feature absence intervals in its +manifest, with one-based inclusive source frame numbers, for example +`"feature-absence": {"eye-r": [[10, 14]]}`. The loader expands these into +per-frame observation tracks before measurement. Unobserved landmarks may fill +rectangular numeric buffers, but they cannot contribute to an eye's contour, +blink or shared gaze. When one eye is absent, gaze uses the observed eye. +Until a detector supplies feature-level confidence, footage without annotations +uses the full-face detection mask as the fallback; it must not claim to detect +individual occlusions that it cannot see. + ## Channel Every animatable property is a channel, and channels are addressed **by path**: diff --git a/docs/port-plan.md b/docs/port-plan.md index 52ab76c..5b8935c 100644 --- a/docs/port-plan.md +++ b/docs/port-plan.md @@ -7,8 +7,11 @@ Step 6 reads extracted footage from the manifest, detects landmarks with local MediaPipe assets at full source cadence, and runs the same freeze path as the synthetic take. The scene time map can sample the frozen roto at a lower picture fps without changing source analysis, duration or audio. Step 7 adds dense -eyelids, shared gaze, brows and pixel-derived teeth. Step 8 is next: authored -parameter controls and their scoped recomputation. +eyelids, shared gaze, brows and pixel-derived teeth. Step 8's data model now +has stable feature identity, feature-level presence, explicit eye pairs and +shared parameter definitions. A manifest can now supply known feature absence +intervals through measurement and freeze. Automatic per-feature detection, +parameter editing and scoped regeneration remain. ## What arthur is @@ -326,9 +329,28 @@ minus paint. The fixed pixel thresholds remain provisional; step 8 exposes their parameters for tuning without changing the source track or picture timing. ### 8 — knobs -The parameter UI, as leaf-addressed params in app-db -(`clip/:cid/params/:subject/:area` — see below), so the sync layer added later has -nothing to retrofit. +Build the parameter model before its UI. Define each parameter once with its +default, validation, applicable area and regeneration dependencies. Store values +by stable subject and feature ID. Represent an eye pair as one group with one or +two eye member IDs from the same subject; a profile view with one identified eye +needs no invented partner. Each eye may override a pair value. Removing an eye +from a pair materialises its effective values so playback does not change. Keep the +existing frozen channels as renderer input; settings and provenance do not enter +the render path. + +Carry feature-level presence through freeze and dense channel state. The same +feature ID covers every observed run across occlusion; a missing measurement +has no channel value on that frame. Full-face detection is the fallback mask +until there is a feature-level detector or authored presence data. A shared gaze +measurement may still feed two independently identified eyes. A small manifest +annotation can supply feature absence intervals now: the loader expands them +before measurement, so invalid eye landmarks are ignored and gaze uses the +visible eye. This is an input format, not a control UI or an automatic detector. + +Use leaf-addressable settings under the clip, subject, feature and optional +group. Retain source measurements so a setting change can regenerate affected +channels without re-detecting footage. Time-varying parameter values and all +parameter controls are deferred to the UI pass. ### 9 — backend Django project, the `clips` app, models for Project/Clip/Footage/Analysis/Leaf, @@ -342,16 +364,14 @@ nobody should pre-empt by porting the old one. ## Two things to not foreclose -The feature controls will later be rethought to handle more than one face, -periodic occlusion, stable identity across frames, and feature groups with their -own parameters. That design can wait; two decisions here are free now and -annoying to reverse: +Feature controls will later handle more than one face and editing presence. +The underlying identity, occlusion and group association model begins in step 8: -- **Presence is not visibility.** An occluded subject has *no value* on a frame, - which is different from a part being hidden. Give every dense block a - `Uint8Array` state mask per frame and let it mean *absent* as well as hidden. -- **Params carry a subject segment.** `clip/:cid/params/:subject/:area`, with one - subject today. Adding a path segment later touches every read and write. +- **Presence is not visibility.** An occluded feature has *no value* on a frame, + which is different from a part being hidden. Dense blocks carry a per-track, + per-frame absence mask; `[:vis]` remains the sole hiding mechanism. +- **Params carry stable identity.** A subject and its features keep their IDs + across observation gaps. A run of visible frames is not a new identity. The identity tracker, when it comes, should use the same pattern the iris and brow correspondences already use: vote across every frame rather than trusting one. diff --git a/frontend/README.md b/frontend/README.md index c8aedbd..8fe9c2e 100644 --- a/frontend/README.md +++ b/frontend/README.md @@ -110,11 +110,20 @@ landmarks and the teeth from source pixels, then freezes them into channels, and adds a button for the footage clip. Detection happens once when you load; playback only resolves channels and paints. Frames without a detection remain marked absent even though their neighbouring poses are used to condition the -track. The stage stays 320×200 regardless of the footage dimensions. Real +track. The scene now records stable subject and feature IDs and explicit eye +pairs; dense channels can mark one feature absent while another is observed. +Current MediaPipe loading supplies only the full-face detection mask. The stage +stays 320×200 regardless of the footage dimensions. Real footage starts at the source picture rate. The **picture fps** buttons sample the frozen roto at lower rates while the source track, duration and audio clock stay unchanged. Picking frames to trace into cels is a separate future editing step. +For known occlusion intervals, an extracted manifest may add +`"feature-absence": {"eye-r": [[10, 14]]}`. Frame numbers are one-based and +inclusive, matching PNG filenames. The eye remains the same feature when it +reappears; the other eye and the mouth continue through the gap. This is an +input annotation, with no UI for editing it yet. + MediaPipe's JS, wasm and model are under `public/mediapipe/` and served locally. No CDN is used by this app. See that directory's README for provenance. diff --git a/frontend/src/arthur/domain/feature.cljs b/frontend/src/arthur/domain/feature.cljs new file mode 100644 index 0000000..ecd2b96 --- /dev/null +++ b/frontend/src/arthur/domain/feature.cljs @@ -0,0 +1,85 @@ +(ns arthur.domain.feature + "Stable tracked identities and explicit eye-pair settings associations. + These maps are document data. Rendering only reads the nodes and channels." + (:require [arthur.domain.params :as params])) + +(defn group-for [scene feature-id] + (first (filter (fn [[_ group]] (some #{feature-id} (:members group))) + (:groups scene)))) + +(defn effective-params + "Resolve static settings for one feature. A future parameter channel can + replace a scalar at this boundary without changing feature or pair identity." + [scene feature-id] + (let [{:keys [subject area params] :as feature} (get-in scene [:features feature-id]) + [_ group] (group-for scene feature-id)] + (when-not feature + (throw (ex-info "unknown feature" {:feature feature-id}))) + (merge (params/for-area :subject) + (get-in scene [:subjects subject :params]) + (params/for-area area) + (:params group) + params))) + +(defn remove-from-pair + "Keep the eye's current settings when its association is removed. Empty pairs + are removed; a one-eye pair remains valid and can acquire a partner later." + [scene feature-id] + (if-let [[group-id group] (group-for scene feature-id)] + (let [area (get-in scene [:features feature-id :area]) + values (select-keys (effective-params scene feature-id) + (keys (params/for-area area))) + members (vec (remove #{feature-id} (:members group)))] + (-> scene + (assoc-in [:features feature-id :params] values) + (update :groups (fn [groups] + (if (seq members) + (assoc-in groups [group-id :members] members) + (dissoc groups group-id)))))) + scene)) + +(defn problems + "Check identity references and pair membership before storing a scene." + [scene] + (let [subjects (:subjects scene) + features (:features scene) + groups (:groups scene) + memberships (mapcat (comp :members val) groups) + node-owners (mapcat (comp :nodes val) features)] + (vec + (concat + (for [[id s] subjects :when (not= id (:id s))] + (str "subject " (pr-str id) " has a different :id")) + (for [[id s] subjects + :when (not (params/valid-settings? :subject (or (:params s) {})))] + (str "subject " (pr-str id) " has invalid settings")) + (for [[id f] features :when (not= id (:id f))] + (str "feature " (pr-str id) " has a different :id")) + (for [[id f] features :when (not (contains? subjects (:subject f)))] + (str "feature " (pr-str id) " has no subject")) + (for [[id f] features :when (not (contains? (disj params/areas :subject) (:area f)))] + (str "feature " (pr-str id) " has an unknown area")) + (for [[id f] features + :when (not (params/valid-settings? (:area f) (or (:params f) {})))] + (str "feature " (pr-str id) " has invalid settings for " (pr-str (:area f)))) + (for [[id f] features node-id (:nodes f) + :when (not (contains? (:nodes scene) node-id))] + (str "feature " (pr-str id) " refers to missing node " (pr-str node-id))) + (for [[id n] (frequencies node-owners) :when (> n 1)] + (str "node " (pr-str id) " belongs to more than one feature")) + (for [[id g] groups :when (not= id (:id g))] + (str "group " (pr-str id) " has a different :id")) + (for [[id g] groups + :when (not (and (= :eye-pair (:kind g)) + (<= 1 (count (:members g)) 2) + (= (count (:members g)) (count (distinct (:members g))))))] + (str "group " (pr-str id) " must be an eye pair of one or two distinct eyes")) + (for [[id g] groups + :when (not (params/valid-settings? :eye (or (:params g) {})))] + (str "group " (pr-str id) " has invalid eye settings")) + (for [[id g] groups member (:members g) + :let [f (get features member)] + :when (not (and f (= :eye (:area f)) (= (:subject g) (:subject f))))] + (str "group " (pr-str id) " has an eye from another subject or an unknown feature")) + (for [[id n] (frequencies memberships) :when (> n 1)] + (str "feature " (pr-str id) " belongs to more than one group")))))) diff --git a/frontend/src/arthur/domain/params.cljs b/frontend/src/arthur/domain/params.cljs new file mode 100644 index 0000000..483431e --- /dev/null +++ b/frontend/src/arthur/domain/params.cljs @@ -0,0 +1,61 @@ +(ns arthur.domain.params + "Definitions for generated settings. Defaults, scope and value constraints live + here once; UI metadata and regeneration rules can be added to each entry.") + +(def definitions + {:anchor-avg {:area :subject :default 2 :type :integer :min 0 + :affects #{:head :mouth :eye :brow :teeth}} + :contour-avg {:area :subject :default 1 :type :integer :min 0 + :affects #{:mouth :eye :brow}} + :verts {:area :mouth :default 8 :type :integer :min 4 :even? true} + :aperture-cut {:area :mouth :default 0.12 :type :number :min 0 :max 1} + :eye-verts {:area :eye :default 8 :type :integer :min 4 :even? true} + :blink-cut {:area :eye :default 0.13 :type :number :min 0} + :gaze-gain {:area :eye :default 1 :type :number :min 0} + :gaze-step {:area :eye :default 0.08 :type :number :min 0} + :iris-size {:area :eye :default 0.42 :type :number :min 0} + :lash-weight {:area :eye :default 0.06 :type :number :min 0} + :pupil-size {:area :eye :default 0.15 :type :number :min 0} + :brow-verts {:area :brow :default 6 :type :integer :min 4 :even? true} + :brow-gain {:area :brow :default 1 :type :number :min 0} + :brow-step {:area :brow :default 0.08 :type :number :min 0} + :brow-weight {:area :brow :default 0.05 :type :number :min 0} + :cavity-erode {:area :teeth :default 0.18 :type :number :min 0} + :tongue-reject {:area :teeth :default 0.18 :type :number :min 0} + :blob-grow {:area :teeth :default 0 :type :integer} + :top-bias {:area :teeth :default 0.6 :type :number} + :teeth-verts {:area :teeth :default 10 :type :integer :min 4} + :min-area {:area :teeth :default 12 :type :integer :min 0} + :teeth-on {:area :teeth :default 0.12 :type :number :min 0} + :teeth-smooth {:area :teeth :default 1 :type :integer :min 0}}) + +(def defaults + (into {} (map (fn [[id spec]] [id (:default spec)])) definitions)) + +(def areas #{:subject :mouth :eye :brow :teeth}) + +(defn affected-areas [id] + (when-let [spec (get definitions id)] + (or (:affects spec) #{(:area spec)}))) + +(defn for-area [wanted-area] + (into {} (keep (fn [[id {:keys [area default]}]] + (when (= wanted-area area) [id default]))) + definitions)) + +(defn valid-value? [id value] + (when-let [{:keys [type min max] must-even? :even?} (get definitions id)] + (and (case type + :integer (integer? value) + :number (number? value) + false) + (or (nil? min) (<= min value)) + (or (nil? max) (<= value max)) + (or (not must-even?) (and (integer? value) (even? value)))))) + +(defn valid-settings? [area settings] + (and (map? settings) + (every? (fn [[id value]] + (and (= area (get-in definitions [id :area])) + (valid-value? id value))) + settings))) diff --git a/frontend/src/arthur/domain/scene.cljs b/frontend/src/arthur/domain/scene.cljs index 5df011a..8eaf326 100644 --- a/frontend/src/arthur/domain/scene.cljs +++ b/frontend/src/arthur/domain/scene.cljs @@ -28,6 +28,7 @@ vector of numbers, and they read the same way, which is what makes freezing fill in the same channel rather than convert into a second format." (:require [arthur.domain.channel :as ch] + [arthur.domain.feature :as feature] [arthur.domain.node :as node] [arthur.domain.palette :as pal])) @@ -451,8 +452,8 @@ (into (for [[id n] nodes p (node/problems n)] (str "node " (pr-str id) ": " p))) + (into (feature/problems scene)) (into (try (doall (map #(depth nodes %) (keys nodes))) nil (catch :default e [(ex-message e)]))))))) - diff --git a/frontend/src/arthur/events/footage.cljs b/frontend/src/arthur/events/footage.cljs index 1d08941..f3e6633 100644 --- a/frontend/src/arthur/events/footage.cljs +++ b/frontend/src/arthur/events/footage.cljs @@ -58,7 +58,8 @@ (defn- build-clip [manifest {:keys [dense detected dimensions interior missing first-real]}] (let [[w h] dimensions frozen (take/footage manifest {:dense dense :detected detected - :dimensions dimensions :interior interior}) + :dimensions dimensions :interior interior + :presence (:presence manifest)}) scene (:scene frozen)] (assoc (select-keys scene [:fps :frames :width :height]) :display-fps (:fps scene) diff --git a/frontend/src/arthur/flow/condition/eyes.cljs b/frontend/src/arthur/flow/condition/eyes.cljs index b0aecdb..2ce69be 100644 --- a/frontend/src/arthur/flow/condition/eyes.cljs +++ b/frontend/src/arthur/flow/condition/eyes.cljs @@ -13,20 +13,47 @@ :y (/ (+ (:y a) (:y b)) 2) :w (distance a b)})) -(defn- mean-width [lids] - (/ (reduce + (map (comp :w socket) lids)) (count lids))) +(defn- hold-observed + "Give temporal filters real samples at every index. The freeze mask still + marks the gap absent; these held values are only numeric placeholders." + [values observed fallback] + (let [values (vec values) + seed (or (first (keep-indexed (fn [i v] + (when (and (nth observed i) (some? v)) v)) + values)) + fallback (first values))] + (loop [f 0 last-value seed out []] + (if (= f (count values)) + out + (let [v (nth values f) + next-value (if (and (nth observed f) (some? v)) v last-value)] + (recur (inc f) next-value (conj out next-value))))))) + +(defn- mean-width [lids observed] + (let [valid (seq (keep-indexed (fn [i lid] (when (nth observed i) lid)) lids)) + rings (or valid lids)] + (/ (reduce + (map (comp :w socket) rings)) (count rings)))) (defn apply-defaults [{:keys [fps contour-avg blink-cut gaze-gain gaze-step iris-size - lash-weight pupil-size]} {:keys [lid-r lid-l open-r open-l gaze] + lash-weight pupil-size]} {:keys [lid-r lid-l open-r open-l gaze + observed-r observed-l gaze-observed] :as measured}] (let [fps (or fps 30) - right (condition/contours {:contour-avg contour-avg} lid-r) - left (condition/contours {:contour-avg contour-avg} lid-l) - w-r (mean-width right) - w-l (mean-width left) + observed-r (or observed-r (vec (repeat (count lid-r) true))) + observed-l (or observed-l (vec (repeat (count lid-l) true))) + gaze-observed (or gaze-observed (vec (repeat (count gaze) true))) + right (condition/contours {:contour-avg contour-avg} + (hold-observed lid-r observed-r nil)) + left (condition/contours {:contour-avg contour-avg} + (hold-observed lid-l observed-l nil)) + w-r (mean-width right observed-r) + w-l (mean-width left observed-l) w (/ (+ w-r w-l) 2) - rest (condition/median-point gaze) + valid-gaze (seq (keep-indexed (fn [i point] + (when (nth gaze-observed i) point)) gaze)) + gaze (hold-observed gaze gaze-observed {:x 0 :y 0}) + rest (if valid-gaze (condition/median-point valid-gaze) {:x 0 :y 0}) px (mapv (fn [{:keys [x y]}] {:x (* (- x (:x rest)) gaze-gain w) :y (* (- y (:y rest)) gaze-gain w)}) gaze) @@ -42,8 +69,10 @@ :lid-r right :lid-l left :lash-r (mapv #(ring/offset-ring % (* lash-weight w-r)) right) :lash-l (mapv #(ring/offset-ring % (* lash-weight w-l)) left) - :shut-r (condition/resolve-blink open-r blink) - :shut-l (condition/resolve-blink open-l blink) + :shut-r (condition/resolve-blink + (hold-observed open-r observed-r nil) blink) + :shut-l (condition/resolve-blink + (hold-observed open-l observed-l nil) blink) :iris-r (centres right) :iris-l (centres left) :radius-r (* 0.5 iris-size w-r) :radius-l (* 0.5 iris-size w-l) diff --git a/frontend/src/arthur/flow/freeze.cljs b/frontend/src/arthur/flow/freeze.cljs index 57699ca..cd093a3 100644 --- a/frontend/src/arthur/flow/freeze.cljs +++ b/frontend/src/arthur/flow/freeze.cljs @@ -76,10 +76,9 @@ `:ctor` makes the array — a function and not a type, because `new` is not something a value can carry — `:scale` is the fixed-point scale or nil, and - `:absent` an optional per-frame predicate. Absence is the SUBJECT's, not the - part's, so it is written into every track of the block: the mouth outline and - the mouth interior are one face, and a frame that face was not on has no pose - for either. + `:absent` an optional (track, frame) predicate. The state is per track, so one + occluded eye can be absent while its partner still has a value. A full-face + miss marks every track absent. Written with `dotimes` and `aset` rather than as a fold, and that is the exception rather than the rule in this codebase: the destination is a typed @@ -110,7 +109,7 @@ :track i :frame f :component k}))) q) v)))) - (when (and state (absent f)) + (when (and state (absent i f)) (aset state (+ (* i nf) f) ch/absent-bit)))))) {:data data :state state :stride stride :frames nf :scale scale :offsets (mapv #(* % nf stride) (range n))})) @@ -372,16 +371,24 @@ {:by by :analysis analysis :params (merge {:anchor-avg anchor-avg :contour-avg contour-avg} extra)}) - eye-block (pack {:ctor #(js/Int16Array. %) :scale geom-scale :absent absent} + eye-block (pack {:ctor #(js/Int16Array. %) :scale geom-scale + :absent (when absent (fn [i f] + (absent (if (< i 2) :eye-r :eye-l) f)))} (mapv #(rings->flat % eye-verts) [(:lash-r eyes) (:lid-r eyes) (:lash-l eyes) (:lid-l eyes)])) - iris-block (pack {:ctor #(js/Float32Array. %) :absent absent} + iris-block (pack {:ctor #(js/Float32Array. %) + :absent (when absent (fn [i f] + (absent (if (zero? i) :eye-r :eye-l) f)))} [(:iris-r eyes) (:iris-l eyes)]) - brow-block (pack {:ctor #(js/Int16Array. %) :scale geom-scale :absent absent} + brow-block (pack {:ctor #(js/Int16Array. %) :scale geom-scale + :absent (when absent (fn [i f] + (absent (if (zero? i) :brow-r :brow-l) f)))} (mapv #(rings->flat % brow-verts) [(:ring-r brows) (:ring-l brows)])) - brow-pos-block (pack {:ctor #(js/Float32Array. %) :absent absent} + brow-pos-block (pack {:ctor #(js/Float32Array. %) + :absent (when absent (fn [i f] + (absent (if (zero? i) :brow-r :brow-l) f)))} [(:pos-r brows) (:pos-l brows)]) eye-node (fn [id z track] {:id id :name (clojure.core/name id) :kind :poly :parent :head :z z @@ -434,11 +441,11 @@ "Freeze the pixel-derived radial contour under the mouth cavity. Missing contours use the dense block's absence bit; contrast decides editable :vis." [name {:keys [analysis teeth-verts cavity-erode tongue-reject blob-grow - top-bias teeth-on teeth-smooth]} detected + top-bias teeth-on teeth-smooth]} absent-feature {:keys [contours shown]}] (let [key (str name "/teeth") - absent #(or (nil? (nth contours %)) - (and detected (not (nth detected %)))) + absent (fn [_ f] (or (nil? (nth contours f)) + (and absent-feature (absent-feature :teeth f)))) empty-points (vec (repeat (* 2 teeth-verts) 0)) values (mapv (fn [ring] (if ring @@ -495,7 +502,9 @@ :eyes :brows the CONDITIONED head-local feature measurements :teeth optional pixel-derived, conditioned radial contour :aperture head-local aperture per frame - :detected optional per-frame booleans; absent frames get the state mask + :detected optional per-frame face booleans + :presence optional feature-id -> per-frame booleans; false means an + unobserved feature, even if the rest of the face was found The node tree is the one docs/animation-model.md specifies, and the two groups are two different things wanting the same transform: @@ -522,21 +531,32 @@ provenance having to be rebuilt." [{:keys [name fps stage expose verts aperture-cut head kept analysis anchor-avg contour-avg] :as params} - {:keys [transforms outer inner detected eyes brows teeth] :as inputs}] + {:keys [transforms outer inner detected presence eyes brows teeth] :as inputs}] (let [nf (count outer) - absent (when detected #(not (nth detected % true))) + _ (doseq [[id track] presence] + (when (not= nf (count track)) + (throw (ex-info "feature presence track must match the clip" + {:feature id :frames nf :actual (count track)})))) + absent (when (or detected presence) + (fn [id f] + (or (and detected (not (nth detected f true))) + (and (contains? presence id) + (not (nth (get presence id) f)))))) + head-absent (when detected (fn [_ f] (not (nth detected f true)))) geom-k (str name "/geom") head-k #(str name "/head-" %) prov (fn [by extra] {:by by :analysis analysis :params (merge {:anchor-avg anchor-avg} extra)}) - rings (pack {:ctor #(js/Int16Array. %) :scale geom-scale :absent absent} + rings (pack {:ctor #(js/Int16Array. %) :scale geom-scale + :absent (when absent (fn [_ f] (absent :mouth f)))} [(rings->flat outer verts) (rings->flat inner verts)]) ;; The anchor, inverted and split into its three components. Three blocks ;; and not one: they are three channels, they have three strides, and a ;; single block would need a per-component offset table to say so. inv (mapv invert transforms) - xf (fn [f] (pack {:ctor #(js/Float32Array. %) :absent absent} [(mapv f inv)])) + xf (fn [f] (pack {:ctor #(js/Float32Array. %) :absent head-absent} + [(mapv f inv)])) pos (xf (fn [t] [(:tx t) (:ty t)])) rot (xf (fn [t] [(:theta t)])) scale (xf (fn [t] [(:s t) (:s t)])) @@ -549,10 +569,31 @@ anchor-prov (prov :anchor/similarity nil) roto (fn [by] (prov by {:verts verts :contour-avg contour-avg})) features (when (and eyes brows) (feature-parts name absent params inputs)) - interior (when teeth (interior-part name params detected teeth)) + interior (when teeth (interior-part name params absent teeth)) scene {:name name :frames nf :fps fps + ;; A gap changes channel state, never these IDs or pair links. + :subjects {:face-1 {:id :face-1 :params {}}} + :features (merge + {:mouth {:id :mouth :subject :face-1 :area :mouth + :nodes [:mouth :mouth-in] :params {}}} + (when features + {:eye-r {:id :eye-r :subject :face-1 :area :eye + :nodes [:eye-r :eye-r-in :iris-r :pupil-r] :params {}} + :eye-l {:id :eye-l :subject :face-1 :area :eye + :nodes [:eye-l :eye-l-in :iris-l :pupil-l] :params {}} + :brow-r {:id :brow-r :subject :face-1 :area :brow + :nodes [:brow-r] :params {}} + :brow-l {:id :brow-l :subject :face-1 :area :brow + :nodes [:brow-l] :params {}}}) + (when teeth + {:teeth {:id :teeth :subject :face-1 :area :teeth + :nodes [:teeth] :params {}}})) + :groups (if features + {:eyes-1 {:id :eyes-1 :kind :eye-pair :subject :face-1 + :members [:eye-r :eye-l] :params {}}} + {}) ;; Stage dimensions, on the clip and not on the footage. See ;; face-placement: nothing below here knows the frame size. :width (first stage) @@ -591,6 +632,10 @@ (prov :roto/mouth-aperture {:aperture-cut aperture-cut}))}}} (:nodes features) (:nodes interior))} + presence-check (doseq [id (keys presence)] + (when-not (contains? (:features scene) id) + (throw (ex-info "presence track names no feature in this scene" + {:feature id :features (keys (:features scene))})))) ;; Tier 2, behind a handle. The keys are descriptive because there is no ;; hashing yet; they become the blocks' sha256 when the backend arrives ;; and nothing above here changes, which is the point of a handle. diff --git a/frontend/src/arthur/flow/ingest.cljs b/frontend/src/arthur/flow/ingest.cljs index 1382ecc..c657309 100644 --- a/frontend/src/arthur/flow/ingest.cljs +++ b/frontend/src/arthur/flow/ingest.cljs @@ -2,6 +2,32 @@ "Read a pre-extracted take. The manifest owns timing and the exact frame count." (:require [clojure.string :as str])) +(defn feature-presence + "Expand one-based, inclusive absence intervals from a manifest into boolean + observation tracks. Unknown features are rejected by freeze, where the scene + knows its feature IDs." + [frames absence] + (when (some? absence) + (when-not (map? absence) + (throw (ex-info "feature-absence must be a map of feature IDs to intervals" + {:feature-absence absence}))) + (into {} + (map (fn [[id intervals]] + (when-not (and (keyword? id) (sequential? intervals) + (every? (fn [span] + (and (vector? span) (= 2 (count span)) + (every? integer? span) + (<= 1 (first span) (second span) frames))) + intervals)) + (throw (ex-info "feature-absence intervals must be [first last] source frames" + {:feature id :intervals intervals :frames frames}))) + [id (mapv (fn [frame] + (not-any? (fn [[first-frame last-frame]] + (<= first-frame frame last-frame)) + intervals)) + (range 1 (inc frames)))])) + absence))) + (defn- valid-manifest [m] (let [fps (js/Number (:fps m)) frames (js/Number (:frames m))] @@ -10,7 +36,8 @@ (string? (:dir m)) (seq (:dir m)) (string? (:audio m)) (seq (:audio m))) (throw (ex-info "manifest.json needs fps, frames (1–900), dir and audio" {:manifest m}))) - (assoc m :fps fps :frames frames))) + (assoc m :fps fps :frames frames + :presence (feature-presence frames (:feature-absence m))))) (defn manifest! [path] diff --git a/frontend/src/arthur/flow/measure/eyes.cljs b/frontend/src/arthur/flow/measure/eyes.cljs index 6ae3507..855561b 100644 --- a/frontend/src/arthur/flow/measure/eyes.cljs +++ b/frontend/src/arthur/flow/measure/eyes.cljs @@ -16,22 +16,38 @@ (geom/apply-sim-all tf (anchor/pick frame table aspect))) dense transforms)) -(defn- iris-pair [iris-a iris-b corners-r] +(defn- observed-track [frames detected presence feature-id] + (let [part (get presence feature-id)] + (mapv (fn [f] + (and (or (nil? detected) (nth detected f)) + (or (nil? part) (nth part f)))) + (range frames)))) + +(defn- iris-pair [iris-a iris-b corners-r corners-l observed-r observed-l] (when iris-a + ;; An occluded eye may still get a plausible iris from MediaPipe. It gets no + ;; vote. Either eye can establish the pairing when its partner is absent. (let [votes (reduce + - (map (fn [a b [outer inner]] - (if (< (distance (first a) (midpoint outer inner)) - (distance (first b) (midpoint outer inner))) - 1 -1)) - iris-a iris-b corners-r))] + (for [f (range (count iris-a)) + [seen corners sign] [[(nth observed-r f) (nth corners-r f) 1] + [(nth observed-l f) (nth corners-l f) -1]] + :when seen] + (let [a (first (nth iris-a f)) + b (first (nth iris-b f)) + [outer inner] corners] + (if (< (distance a (midpoint outer inner)) + (distance b (midpoint outer inner))) + sign (- sign)))))] (if (pos? votes) {:right :a :left :b} {:right :b :left :a})))) (defn measure "All rings and iris coordinates use the anchor's head-local image-height unit. Eye openness and gaze use each eye's rigid corner width as their unit. Iris - block identity is voted from proximity over the whole take." - [{:keys [aspect debug?]} {:keys [dense transforms]}] + block identity is voted from observed eyes over the whole take." + [{:keys [aspect debug?]} {:keys [dense transforms detected presence]}] (let [track (partial local-track dense transforms aspect) + observed-r (observed-track (count dense) detected presence :eye-r) + observed-l (observed-track (count dense) detected presence :eye-l) lid-r (track lm/EYE-R-RING) lid-l (track lm/EYE-L-RING) corners-r (track lm/EYE-R-CORNERS) @@ -41,7 +57,7 @@ has-iris? (every? #(> (count %) (last lm/IRIS-B)) dense) iris-a (when has-iris? (track lm/IRIS-A)) iris-b (when has-iris? (track lm/IRIS-B)) - pairing (iris-pair iris-a iris-b corners-r) + pairing (iris-pair iris-a iris-b corners-r corners-l observed-r observed-l) iris-for (fn [side f] (first (nth (if (= (get pairing side) :a) iris-a iris-b) f))) openness (fn [corners lids] @@ -57,17 +73,25 @@ iris (iris-for side f)] {:x (/ (- (:x iris) (:x c)) w) :y (/ (- (:y iris) (:y c)) w)})) - r (one :right corners-r) - l (one :left corners-l)] - (midpoint r l))) + r (when (nth observed-r f) (one :right corners-r)) + l (when (nth observed-l f) (one :left corners-l))] + (cond + (and r l) (midpoint r l) + r r + l l + :else nil))) (range (count dense))) (vec (repeat (count dense) {:x 0 :y 0})))] {:lid-r lid-r :lid-l lid-l :corners-r corners-r :corners-l corners-l :open-r (openness corners-r lids-r) :open-l (openness corners-l lids-l) + :observed-r observed-r :observed-l observed-l + :gaze-observed (if pairing (mapv #(or %1 %2) observed-r observed-l) + (vec (repeat (count dense) true))) :gaze gaze :has-iris? (boolean pairing) :iris-pair pairing :debug (when debug? {:open-r (openness corners-r lids-r) :open-l (openness corners-l lids-l) + :observed-r observed-r :observed-l observed-l :gaze gaze :iris-pair pairing})})) diff --git a/frontend/src/arthur/flow/take.cljs b/frontend/src/arthur/flow/take.cljs index 821835d..8fb75b7 100644 --- a/frontend/src/arthur/flow/take.cljs +++ b/frontend/src/arthur/flow/take.cljs @@ -9,24 +9,25 @@ [arthur.flow.measure.brows :as brows] [arthur.flow.measure.eyes :as eyes] [arthur.flow.measure.interior :as interior] - [arthur.flow.measure.mouth :as mouth])) + [arthur.flow.measure.mouth :as mouth] + [arthur.domain.params :as params])) -(def knobs - {:anchor-avg 2 :contour-avg 1 :verts 8 :aperture-cut 0.12 - :eye-verts 8 :blink-cut 0.13 :gaze-gain 1 :gaze-step 0.08 - :iris-size 0.42 :lash-weight 0.06 :pupil-size 0.15 - :brow-verts 6 :brow-gain 1 :brow-step 0.08 :brow-weight 0.05 - :cavity-erode 0.18 :tongue-reject 0.18 :blob-grow 0 :top-bias 0.6 - :teeth-verts 10 :min-area 12 :teeth-on 0.12 :teeth-smooth 1}) +(def knobs params/defaults) (defn measure "Condition the anchor before measuring rings through it." - [{:keys [aspect] :as params} {:keys [dense detected interior]}] - (let [fitted (anchor/fit {:aspect aspect} {:dense dense}) + [{:keys [aspect] :as params} {:keys [dense detected interior presence]}] + (let [_ (doseq [[id track] presence] + (when (not= (count dense) (count track)) + (throw (ex-info "feature presence track must match the source" + {:feature id :frames (count dense) + :actual (count track)})))) + fitted (anchor/fit {:aspect aspect} {:dense dense}) anchored (condition/anchor params fitted) rings (mouth/measure {:aspect aspect} {:dense dense :transforms (:transforms anchored)}) - eye-data (eyes/measure params {:dense dense :transforms (:transforms anchored)}) + eye-data (eyes/measure params {:dense dense :transforms (:transforms anchored) + :detected detected :presence presence}) brow-data (brows/measure params {:dense dense :transforms (:transforms anchored)}) inner-data (when interior (condition-interior/apply-defaults @@ -41,7 +42,8 @@ :eyes (condition-eyes/apply-defaults params eye-data) :brows (condition-brows/apply-defaults params brow-data eye-data) :teeth inner-data - :detected detected))) + :detected detected + :presence presence))) (defn build [params inputs] @@ -51,11 +53,12 @@ "A real manifest and its detected landmarks through the same measurement and freeze path as the synthetic take. The source cadence stays in :fps; picture sampling is a root time map applied only after this artifact exists." - [manifest {:keys [dense detected dimensions interior]}] + [manifest {:keys [dense detected dimensions interior presence]}] (let [[w h] dimensions params (merge knobs {:name "footage" :fps (:fps manifest) :aspect (/ w h) :stage [320 200] :fit-motion? true :expose 1 :head :as-filmed :analysis (str "mediapipe:1.0.1/" (:source manifest))})] - (build params {:dense dense :detected detected :interior interior}))) + (build params {:dense dense :detected detected :interior interior + :presence presence}))) diff --git a/frontend/test/arthur/domain/feature_test.cljs b/frontend/test/arthur/domain/feature_test.cljs new file mode 100644 index 0000000..1c063f0 --- /dev/null +++ b/frontend/test/arthur/domain/feature_test.cljs @@ -0,0 +1,54 @@ +(ns arthur.domain.feature-test + (:require [cljs.test :refer [deftest is]] + [arthur.domain.feature :as feature] + [arthur.domain.params :as params])) + +(defn- nine-eyes [] + (let [people (mapv #(keyword (str "person-" %)) (range 5)) + eyes (mapv #(keyword (str "eye-" %)) (range 9)) + members [[:eye-0 :eye-1] [:eye-2 :eye-3] [:eye-4 :eye-5] + [:eye-6 :eye-7] [:eye-8]]] + {:subjects (into {} (map (fn [id] [id {:id id :params {}}]) people)) + :features (into {} (map-indexed (fn [i id] + [id {:id id :subject (nth people (quot i 2)) + :area :eye :nodes [] :params {}}]) eyes)) + :groups (into {} (map-indexed (fn [i ids] + (let [id (keyword (str "pair-" i))] + [id {:id id :kind :eye-pair + :subject (nth people i) + :members ids :params {}}])) members)) + :nodes {}})) + +(deftest five-people-can-have-nine-identified-eyes + (let [scene (nine-eyes)] + (is (empty? (feature/problems scene))) + (is (= [:eye-8] (get-in scene [:groups :pair-4 :members]))) + (is (= :person-4 (get-in scene [:features :eye-8 :subject]))))) + +(deftest parameter-definitions-drive-the-current-take-defaults + (is (= #{:eye} (params/affected-areas :blink-cut))) + (is (= #{:head :mouth :eye :brow :teeth} + (params/affected-areas :anchor-avg))) + (is (params/valid-settings? :eye {:blink-cut 0.2})) + (is (not (params/valid-settings? :eye {:verts 8})))) + +(deftest pair-values-are-shared-but-eye-overrides-are-independent + (let [scene (-> (nine-eyes) + (assoc-in [:groups :pair-0 :params :blink-cut] 0.2) + (assoc-in [:features :eye-1 :params :blink-cut] 0.3))] + (is (= 0.2 (:blink-cut (feature/effective-params scene :eye-0)))) + (is (= 0.3 (:blink-cut (feature/effective-params scene :eye-1)))) + (is (= (:eye-verts params/defaults) + (:eye-verts (feature/effective-params scene :eye-0)))) + (let [before (feature/effective-params scene :eye-0) + unpaired (feature/remove-from-pair scene :eye-0)] + (is (= before (feature/effective-params unpaired :eye-0))) + (is (= [:eye-1] (get-in unpaired [:groups :pair-0 :members]))) + (is (empty? (feature/problems unpaired)))))) + +(deftest a-pair-cannot-cross-subjects-or-own-an-eye-twice + (let [scene (nine-eyes)] + (is (seq (feature/problems + (assoc-in scene [:groups :pair-0 :members] [:eye-0 :eye-2])))) + (is (seq (feature/problems + (assoc-in scene [:groups :pair-1 :members] [:eye-0 :eye-3])))))) diff --git a/frontend/test/arthur/flow/eye_occlusion_test.cljs b/frontend/test/arthur/flow/eye_occlusion_test.cljs new file mode 100644 index 0000000..ad20946 --- /dev/null +++ b/frontend/test/arthur/flow/eye_occlusion_test.cljs @@ -0,0 +1,67 @@ +(ns arthur.flow.eye-occlusion-test + (:require [cljs.test :refer [deftest is]] + [arthur.demo.take :as take] + [arthur.domain.channel :as ch] + [arthur.domain.landmarks :as lm] + [arthur.domain.palette :as pal] + [arthur.domain.scene :as scene] + [arthur.flow.condition.eyes :as condition-eyes] + [arthur.flow.ingest :as ingest] + [arthur.flow.measure.eyes :as eyes] + [arthur.flow.take :as flow-take])) + +(deftest an-unobserved-eye-cannot-steer-the-shared-gaze + (let [frame 11 + params (assoc take/params :aspect 1) + presence (ingest/feature-presence take/frames {:eye-r [[10 14]]}) + inputs {:dense @take/analysis + :transforms (:transforms @take/measured) + :presence presence} + clean (eyes/measure params inputs) + iris-ids (if (= :a (:right (:iris-pair clean))) lm/IRIS-A lm/IRIS-B) + bad-ids (distinct (concat lm/EYE-R-RING iris-ids)) + corrupted (mapv (fn [i source] + (if (<= 9 i 13) + (reduce (fn [points id] + (update-in points [id :x] + 0.25)) + source bad-ids) + source)) + (range take/frames) @take/analysis) + masked (eyes/measure params (assoc inputs :dense corrupted)) + unmasked (eyes/measure params + (dissoc (assoc inputs :dense corrupted) :presence)) + drawn-clean (condition-eyes/apply-defaults params clean) + drawn-masked (condition-eyes/apply-defaults params masked)] + (is (= lm/NUM-LANDMARKS (count (nth corrupted frame)))) + (is (:has-iris? masked) (str "pair " (:iris-pair masked))) + (is (false? (nth (:observed-r masked) frame))) + (is (true? (nth (:observed-l masked) frame))) + (is (= (nth (:gaze clean) frame) (nth (:gaze masked) frame))) + (is (not= (nth (:gaze clean) frame) (nth (:gaze unmasked) frame))) + (is (= (nth (:lid-r drawn-clean) frame) + (nth (:lid-r drawn-masked) frame))) + (is (= (nth (:iris-l drawn-clean) frame) + (nth (:iris-l drawn-masked) frame))))) + +(deftest the-full-take-hides-only-the-annotated-eye-and-returns-to-the-same-id + (let [presence (ingest/feature-presence take/frames {:eye-r [[10 14]]}) + {:keys [scene store]} (flow-take/build + (assoc take/params :aspect 1 :name "observed-gap") + {:dense @take/analysis :presence presence}) + sample (fn [id frame] + (ch/value-at (get-in scene [:nodes id :channels [:geom :pts]]) + frame store))] + (is (empty? (scene/problems scene))) + (is (= [:eye-r :eye-l] (get-in scene [:groups :eyes-1 :members]))) + (doseq [f (range 9 14)] + (is (ch/nothing? (sample :eye-r f))) + (is (not (ch/nothing? (sample :eye-l f)))) + (is (not (ch/nothing? (sample :mouth f))))) + (let [drawn (into #{} (map :node) + ((scene/resolver scene store pal/index-of) 11))] + (is (not (contains? drawn :eye-r))) + (is (not (contains? drawn :iris-r))) + (is (contains? drawn :eye-l)) + (is (contains? drawn :mouth))) + (is (not (ch/nothing? (sample :eye-r 8)))) + (is (not (ch/nothing? (sample :eye-r 14)))))) diff --git a/frontend/test/arthur/flow/freeze_test.cljs b/frontend/test/arthur/flow/freeze_test.cljs index e9a5e62..e21aecb 100644 --- a/frontend/test/arthur/flow/freeze_test.cljs +++ b/frontend/test/arthur/flow/freeze_test.cljs @@ -381,6 +381,27 @@ ;; --------------------------------------------------------------------------- ;; presence is not visibility +(deftest an-eye-can-disappear-and-return-under-the-same-identity + (let [gap (set (range 40 60)) + presence {:eye-r (mapv #(not (contains? gap %)) (range take/frames))} + c (freeze/clip (assoc take/params :name "one-eye-gappy") + (assoc @take/measured :presence presence)) + sc (:scene c) + at (fn [id f] + (ch/value-at (get-in sc [:nodes id :channels [:geom :pts]]) f (:store c)))] + (is (= [:eye-r :eye-l] (get-in sc [:groups :eyes-1 :members]))) + (is (= :eye-r (get-in sc [:features :eye-r :id]))) + (is (not (ch/nothing? (at :eye-r 39)))) + (is (ch/nothing? (at :eye-r 50))) + (is (not (ch/nothing? (at :eye-r 60)))) + (is (not (ch/nothing? (at :eye-l 50)))) + (is (not (ch/nothing? (at :mouth 50)))) + (let [drawn-nodes (into #{} (map :node) ((scene/resolver sc (:store c) pal/index-of) 50))] + (is (not (contains? drawn-nodes :eye-r))) + (is (contains? drawn-nodes :eye-l)) + (is (contains? drawn-nodes :mouth))) + (is (empty? (scene/problems sc))))) + (deftest an-undetected-frame-has-no-pose-at-all ;; A subject that was not on the frame has NO VALUE, which is different from a ;; part being switched off. The mask lands on every block of the freeze, so an @@ -447,10 +468,15 @@ ;; beat are not. This is the numeric half of step 5's done-criterion; the other ;; half is a picture and lives in test/browser/take.mjs. (let [locked (freeze/head-mode {:mode :locked} @clip) - shot (fn [f] (render locked f)) + shot (fn [f] + (let [r (raster/make W H) + mouth (filter #(= :mouth (:node %)) (ops-at locked f))] + (raster/clear! r (get pal/index-of :bg)) + (raster/draw-ops! r mouth) + (vec (array-seq (:buf r))))) differ (fn [a b] (count (remove true? (map = a b))))] - ;; Beat 1 is wide open and beat 3 is shut. Rendered with the head LOCKED, so - ;; what differs is articulation and not the head wandering across the stage. + ;; Beat 1 is wide open and beat 3 is shut. Compare only the mouth outline: + ;; eyes and brows added in step 7 have their own motion within the beat. (is (> (differ (shot 10) (shot 28)) 300) "the open and the shut mouth rasterise the same") ;; And within a beat, on the exposure grid, it holds. diff --git a/frontend/test/arthur/flow/ingest_test.cljs b/frontend/test/arthur/flow/ingest_test.cljs new file mode 100644 index 0000000..639ccbb --- /dev/null +++ b/frontend/test/arthur/flow/ingest_test.cljs @@ -0,0 +1,15 @@ +(ns arthur.flow.ingest-test + (:require [cljs.test :refer [deftest is]] + [arthur.flow.ingest :as ingest])) + +(deftest feature-absence-uses-the-source-frame-numbers + (let [tracks (ingest/feature-presence + 8 {:eye-r [[3 5]] :eye-l [[1 1] [8 8]]})] + (is (= [true true false false false true true true] (:eye-r tracks))) + (is (= [false true true true true true true false] (:eye-l tracks))))) + +(deftest malformed-feature-absence-fails-before-analysis + (doseq [bad [{:eye-r [[0 2]]} + {:eye-r [[6 9]]} + {:eye-r [[5 3]]}]] + (is (thrown? ExceptionInfo (ingest/feature-presence 8 bad))))) diff --git a/js/app.js b/js/app.js index 79e9a0e..cf68ee1 100644 --- a/js/app.js +++ b/js/app.js @@ -51,6 +51,7 @@ const state = { lead: 0, // performance-track offset in frames exposure: 1, // 1 = on 1s, 2 = on 2s. Picture holds; audio does not. interior: null, // per-frame teeth measurement from image content + detected: null, // per-frame: did MediaPipe really see a face here? teeth: null, // resolved per-frame {show, t} after knobs eyes: null, // resolved per-frame lid rings, shut flags, iris discs brows: null, // resolved per-frame brow rings after quantised raise @@ -167,18 +168,29 @@ async function initLandmarker() { async function detectAll(images) { const lm = await initLandmarker(); const cv = document.createElement('canvas'); - const dense = [], missing = []; + const dense = [], missing = [], detected = []; for (let i = 0; i < images.length; i++) { const im = images[i]; cv.width = im.naturalWidth; cv.height = im.naturalHeight; cv.getContext('2d').drawImage(im, 0, 0); const out = lm.detect(cv); - if (out.faceLandmarks && out.faceLandmarks.length) dense.push(out.faceLandmarks[0]); - else { missing.push(i); dense.push(dense.length ? dense[dense.length - 1] : null); } + if (out.faceLandmarks && out.faceLandmarks.length) { + dense.push(out.faceLandmarks[0]); detected.push(true); + } else { + missing.push(i); detected.push(false); + dense.push(dense.length ? dense[dense.length - 1] : null); + } if (i % 4 === 0) status(`detecting… ${i + 1}/${images.length}`); } - if (dense[0] === null) throw new Error('no face found in the first frame'); - return { dense, missing }; + // A gap mid-take holds the previous frame, but a gap at the TOP has no + // previous to hold - a face that starts occluded or walks in late left + // leading nulls, and this used to throw and discard the whole take. Back-fill + // from the first real detection: the mirror of hold-previous, and the only + // fill that is a real pose from this take rather than an invention. + const firstReal = dense.findIndex((d) => d !== null); + if (firstReal < 0) throw new Error('no face found in any frame — check framing and light'); + for (let i = 0; i < firstReal; i++) dense[i] = dense[firstReal]; + return { dense, missing, detected, firstReal }; } // Interior measurement is a function of pixels alone, so it runs once with @@ -217,10 +229,19 @@ function rebuild(resetKeep) { state.stab = stabilize(state.dense, o.smoothWin, state.aspect); + // The neutral drives calibration and the placeholder plate, so it has to be a + // frame the camera actually saw: a back-filled or held frame is a duplicate + // pose, and letting one win this contest would calibrate the whole take + // against a landmark set that belongs to some other moment. const ap = state.stab.aperture; + const real = state.detected; + const shut = (a, b) => (b < 0 || ap[a] < ap[b] ? a : b); const head = Math.max(1, Math.floor(N / 4)); - let neutral = 0; - for (let i = 0; i < head; i++) if (ap[i] < ap[neutral]) neutral = i; + let neutral = -1; + for (let i = 0; i < head; i++) if (!real || real[i]) neutral = shut(i, neutral); + // Whole head of the take occluded: widen to any real frame rather than give up. + if (neutral < 0) for (let i = 0; i < N; i++) if (!real || real[i]) neutral = shut(i, neutral); + if (neutral < 0) neutral = 0; state.neutral = neutral; state.xform = makeXform(state.stab, neutral); @@ -1072,8 +1093,8 @@ async function runFrames() { status(`no frames in ${el('framedir').value}/ — run extract.sh first`, 'err'); return; } - const { dense, missing } = await detectAll(images); - state.images = images; state.dense = dense; + const { dense, missing, detected, firstReal } = await detectAll(images); + state.images = images; state.dense = dense; state.detected = detected; state.aspect = images[0].naturalWidth / images[0].naturalHeight; status('measuring mouth interiors…'); state.interior = measureAll(images, dense, opts()); @@ -1086,7 +1107,9 @@ async function runFrames() { status(`${images.length} frames · ${images[0].naturalWidth}x${images[0].naturalHeight} · ` + `${state.fps}fps · ${dur}s` + (state.audio ? ' · audio loaded' : ' · no audio') + - (missing.length ? ` · no face on ${missing.length} (held previous)` : ''), + (missing.length ? ` · no face on ${missing.length}` + + (firstReal ? ` (${firstReal} at the top back-filled from ${firstReal + 1}, rest held)` + : ' (held previous)') : ''), missing.length ? 'warn' : 'ok'); } catch (e) { status(e.message, 'err'); console.error(e); } } @@ -1096,6 +1119,7 @@ function runSynthetic() { attachAudio(null); state.fps = 12; state.aspect = 1; // synthetic landmarks are generated square + state.detected = null; // no detection ran, so every frame counts as real state.lead = 0; state.interior = null; // no pixels, so no teeth