noe/arthur
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Give features stable identity, eye pairs and per-feature presence

Browse source 
Step 8's data model, ahead of its controls. Nothing here is a UI.

domain/params holds every knob's definition once — default, applicable area,
value constraints and the areas a change would force to regenerate. flow/take's
literal knob map becomes a view of it, so the take's defaults and the future
parameter panel cannot drift apart.

domain/feature adds subjects, features and groups as document data the renderer
never reads. A feature ID is stable for the whole clip, across occlusion: a run
of visible frames is not a new identity. An eye pair is an explicit group of one
or two eyes of the same subject, so a profile view with one identified eye needs
no invented partner. Settings resolve area -> subject -> group -> feature, and
dropping an eye from a pair materialises its effective values first so playback
does not jump. scene/problems now validates all of it.

Presence becomes per-feature rather than per-subject. freeze's :absent predicate
takes a track as well as a frame, so one occluded eye can be absent while its
partner still has a value; a full-face miss still marks everything absent. A
manifest may annotate known gaps as one-based inclusive intervals, which ingest
expands into observation tracks before measurement. An unobserved eye then gets
no vote in the iris pairing and cannot steer the shared gaze — gaze falls back to
whichever eye is visible. Temporal filters still see a sample on every frame,
held from the last observed one, because the numbers are a rectangular buffer;
the state mask, not the buffer, is what says the frame has no value.

js/app.js gets the same occlusion lesson: leading nulls from a face that starts
occluded used to throw away the whole take, and the neutral frame could be chosen
from a held duplicate pose.

Parameter editing, scoped regeneration and a feature-level detector remain. Until
one exists, footage without annotations falls back to the full-face mask rather
than claiming occlusions it cannot see.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B87NVmiU36qQmN9gmFYnJ9
This commit is contained in:
Olive Vaughn 2026-09-27 22:44:36 -04:00
parent ccca93e233
commit 35ef150b48
18 changed files with 631 additions and 86 deletions
Download patch file Download diff file Expand all files Collapse all files

3
README.md
View file

@ -18,7 +18,8 @@ modern conveniences belong in the workflow, not the output. See
The active port has reached [step 7 of the port plan](docs/port-plan.md): it plays
the synthetic take and can load extracted real footage with mouth, eyes, brows,
and pixel-derived teeth.
and pixel-derived teeth. The step 8 data model now represents persistent feature
IDs, eye pairs and feature-level observation gaps; its controls are still pending.
See [frontend/README.md](frontend/README.md) for setup and the **load frames**
workflow. The rest of this README describes the older JS prototype, which still
runs separately on port 8777.

53
docs/animation-model.md
View file

@ -79,6 +79,59 @@ this way.
over which the node exists at all. Distinct from a `[:vis]` channel, which
blinks an existing node on and off.
### Subjects and tracked features
Scene nodes describe drawings, not tracking identity. A scene may also carry a
flat `:features` map. A feature ID stays stable for the whole clip, including
frames where that feature is occluded and later reappears:
```clojure
:subjects {:face-1 {:id :face-1}}
:features
{:eye-r {:id :eye-r :subject :face-1 :area :eye
:nodes [:eye-r :eye-r-in :iris-r :pupil-r] :params {}}
:eye-l {:id :eye-l :subject :face-1 :area :eye
:nodes [:eye-l :eye-l-in :iris-l :pupil-l] :params {}}
:mouth {:id :mouth :subject :face-1 :area :mouth
:nodes [:mouth :mouth-in :teeth] :params {}}}
:groups
{:eyes-1 {:id :eyes-1 :kind :eye-pair :subject :face-1
:members [:eye-r :eye-l] :params {}}}
```
An eye pair is an explicit relationship between one or two eyes of the **same
subject**. It may have one member when only one eye has been identified; it does
not invent a second eye. Five subjects with nine identified eyes can have four
two-member pairs and one one-member pair. Each eye still has its own feature ID
and presence track. A group is a settings association, not a scene parent or a
tracking ID. Membership lives only on the group, avoiding a second pointer on
the feature that could disagree with it.
Each feature resolves settings from its area's definitions, then its group,
then its own `:params`. An eye can therefore inherit a pair setting or override
it without changing its partner. Removing it from a pair copies its effective
values into the feature first, so the result does not jump. Feature identity
and pair membership are clip-wide; a future parameter track can vary values
over time without splitting a feature at an observation gap.
Parameter definitions live in one registry: key, default, applicable area,
value constraints and affected areas. The registry supplies the take's defaults
today. The parameter UI and regeneration from edited values are later work.
Dense channel state records whether a measurement exists **for that feature on
that frame**. Occlusion means absent data on that frame, not a false `[:vis]`
value and not the end of the feature's identity. A full-face detection failure
makes all its features absent. A single occluded eye need only make that eye's
channels absent. Footage can carry explicit feature absence intervals in its
manifest, with one-based inclusive source frame numbers, for example
`"feature-absence": {"eye-r": [[10, 14]]}`. The loader expands these into
per-frame observation tracks before measurement. Unobserved landmarks may fill
rectangular numeric buffers, but they cannot contribute to an eye's contour,
blink or shared gaze. When one eye is absent, gaze uses the observed eye.
Until a detector supplies feature-level confidence, footage without annotations
uses the full-face detection mask as the fallback; it must not claim to detect
individual occlusions that it cannot see.
## Channel
Every animatable property is a channel, and channels are addressed **by path**:

48
docs/port-plan.md
View file

@ -7,8 +7,11 @@ Step 6 reads extracted footage from the manifest, detects landmarks with local
MediaPipe assets at full source cadence, and runs the same freeze path as the
synthetic take. The scene time map can sample the frozen roto at a lower picture
fps without changing source analysis, duration or audio. Step 7 adds dense
eyelids, shared gaze, brows and pixel-derived teeth. Step 8 is next: authored
parameter controls and their scoped recomputation.
eyelids, shared gaze, brows and pixel-derived teeth. Step 8's data model now
has stable feature identity, feature-level presence, explicit eye pairs and
shared parameter definitions. A manifest can now supply known feature absence
intervals through measurement and freeze. Automatic per-feature detection,
parameter editing and scoped regeneration remain.
## What arthur is
@ -326,9 +329,28 @@ minus paint. The fixed pixel thresholds remain provisional; step 8 exposes their
parameters for tuning without changing the source track or picture timing.
### 8 — knobs
The parameter UI, as leaf-addressed params in app-db
(`clip/:cid/params/:subject/:area` — see below), so the sync layer added later has
nothing to retrofit.
Build the parameter model before its UI. Define each parameter once with its
default, validation, applicable area and regeneration dependencies. Store values
by stable subject and feature ID. Represent an eye pair as one group with one or
two eye member IDs from the same subject; a profile view with one identified eye
needs no invented partner. Each eye may override a pair value. Removing an eye
from a pair materialises its effective values so playback does not change. Keep the
existing frozen channels as renderer input; settings and provenance do not enter
the render path.
Carry feature-level presence through freeze and dense channel state. The same
feature ID covers every observed run across occlusion; a missing measurement
has no channel value on that frame. Full-face detection is the fallback mask
until there is a feature-level detector or authored presence data. A shared gaze
measurement may still feed two independently identified eyes. A small manifest
annotation can supply feature absence intervals now: the loader expands them
before measurement, so invalid eye landmarks are ignored and gaze uses the
visible eye. This is an input format, not a control UI or an automatic detector.
Use leaf-addressable settings under the clip, subject, feature and optional
group. Retain source measurements so a setting change can regenerate affected
channels without re-detecting footage. Time-varying parameter values and all
parameter controls are deferred to the UI pass.
### 9 — backend
Django project, the `clips` app, models for Project/Clip/Footage/Analysis/Leaf,
@ -342,16 +364,14 @@ nobody should pre-empt by porting the old one.
## Two things to not foreclose
The feature controls will later be rethought to handle more than one face,
periodic occlusion, stable identity across frames, and feature groups with their
own parameters. That design can wait; two decisions here are free now and
annoying to reverse:
Feature controls will later handle more than one face and editing presence.
The underlying identity, occlusion and group association model begins in step 8:
- **Presence is not visibility.** An occluded subject has *no value* on a frame,
which is different from a part being hidden. Give every dense block a
`Uint8Array` state mask per frame and let it mean *absent* as well as hidden.
- **Params carry a subject segment.** `clip/:cid/params/:subject/:area`, with one
subject today. Adding a path segment later touches every read and write.
- **Presence is not visibility.** An occluded feature has *no value* on a frame,
which is different from a part being hidden. Dense blocks carry a per-track,
per-frame absence mask; `[:vis]` remains the sole hiding mechanism.
- **Params carry stable identity.** A subject and its features keep their IDs
across observation gaps. A run of visible frames is not a new identity.
The identity tracker, when it comes, should use the same pattern the iris and brow
correspondences already use: vote across every frame rather than trusting one.

11
frontend/README.md
View file

@ -110,11 +110,20 @@ landmarks and the teeth from source pixels, then freezes them into channels,
and adds a button for the footage clip. Detection happens once when you load;
playback only resolves channels and paints. Frames without a detection remain
marked absent even though their neighbouring poses are used to condition the
track. The stage stays 320×200 regardless of the footage dimensions. Real
track. The scene now records stable subject and feature IDs and explicit eye
pairs; dense channels can mark one feature absent while another is observed.
Current MediaPipe loading supplies only the full-face detection mask. The stage
stays 320×200 regardless of the footage dimensions. Real
footage starts at the source picture rate. The **picture fps** buttons sample the
frozen roto at lower rates while the source track, duration and audio clock stay
unchanged. Picking frames to trace into cels is a separate future editing step.
For known occlusion intervals, an extracted manifest may add
`"feature-absence": {"eye-r": [[10, 14]]}`. Frame numbers are one-based and
inclusive, matching PNG filenames. The eye remains the same feature when it
reappears; the other eye and the mouth continue through the gap. This is an
input annotation, with no UI for editing it yet.
MediaPipe's JS, wasm and model are under `public/mediapipe/` and served locally.
No CDN is used by this app. See that directory's README for provenance.

85
frontend/src/arthur/domain/feature.cljs Normal file
View file

@ -0,0 +1,85 @@
(ns arthur.domain.feature
"Stable tracked identities and explicit eye-pair settings associations.
These maps are document data. Rendering only reads the nodes and channels."
(:require [arthur.domain.params :as params]))
(defn group-for [scene feature-id]
(first (filter (fn [[_ group]] (some #{feature-id} (:members group)))
(:groups scene))))
(defn effective-params
"Resolve static settings for one feature. A future parameter channel can
replace a scalar at this boundary without changing feature or pair identity."
[scene feature-id]
(let [{:keys [subject area params] :as feature} (get-in scene [:features feature-id])
[_ group] (group-for scene feature-id)]
(when-not feature
(throw (ex-info "unknown feature" {:feature feature-id})))
(merge (params/for-area :subject)
(get-in scene [:subjects subject :params])
(params/for-area area)
(:params group)
params)))
(defn remove-from-pair
"Keep the eye's current settings when its association is removed. Empty pairs
are removed; a one-eye pair remains valid and can acquire a partner later."
[scene feature-id]
(if-let [[group-id group] (group-for scene feature-id)]
(let [area (get-in scene [:features feature-id :area])
values (select-keys (effective-params scene feature-id)
(keys (params/for-area area)))
members (vec (remove #{feature-id} (:members group)))]
(-> scene
(assoc-in [:features feature-id :params] values)
(update :groups (fn [groups]
(if (seq members)
(assoc-in groups [group-id :members] members)
(dissoc groups group-id))))))
scene))
(defn problems
"Check identity references and pair membership before storing a scene."
[scene]
(let [subjects (:subjects scene)
features (:features scene)
groups (:groups scene)
memberships (mapcat (comp :members val) groups)
node-owners (mapcat (comp :nodes val) features)]
(vec
(concat
(for [[id s] subjects :when (not= id (:id s))]
(str "subject " (pr-str id) " has a different :id"))
(for [[id s] subjects
:when (not (params/valid-settings? :subject (or (:params s) {})))]
(str "subject " (pr-str id) " has invalid settings"))
(for [[id f] features :when (not= id (:id f))]
(str "feature " (pr-str id) " has a different :id"))
(for [[id f] features :when (not (contains? subjects (:subject f)))]
(str "feature " (pr-str id) " has no subject"))
(for [[id f] features :when (not (contains? (disj params/areas :subject) (:area f)))]
(str "feature " (pr-str id) " has an unknown area"))
(for [[id f] features
:when (not (params/valid-settings? (:area f) (or (:params f) {})))]
(str "feature " (pr-str id) " has invalid settings for " (pr-str (:area f))))
(for [[id f] features node-id (:nodes f)
:when (not (contains? (:nodes scene) node-id))]
(str "feature " (pr-str id) " refers to missing node " (pr-str node-id)))
(for [[id n] (frequencies node-owners) :when (> n 1)]
(str "node " (pr-str id) " belongs to more than one feature"))
(for [[id g] groups :when (not= id (:id g))]
(str "group " (pr-str id) " has a different :id"))
(for [[id g] groups
:when (not (and (= :eye-pair (:kind g))
(<= 1 (count (:members g)) 2)
(= (count (:members g)) (count (distinct (:members g))))))]
(str "group " (pr-str id) " must be an eye pair of one or two distinct eyes"))
(for [[id g] groups
:when (not (params/valid-settings? :eye (or (:params g) {})))]
(str "group " (pr-str id) " has invalid eye settings"))
(for [[id g] groups member (:members g)
:let [f (get features member)]
:when (not (and f (= :eye (:area f)) (= (:subject g) (:subject f))))]
(str "group " (pr-str id) " has an eye from another subject or an unknown feature"))
(for [[id n] (frequencies memberships) :when (> n 1)]
(str "feature " (pr-str id) " belongs to more than one group"))))))

61
frontend/src/arthur/domain/params.cljs Normal file
View file

@ -0,0 +1,61 @@
(ns arthur.domain.params
"Definitions for generated settings. Defaults, scope and value constraints live
here once; UI metadata and regeneration rules can be added to each entry.")
(def definitions
{:anchor-avg {:area :subject :default 2 :type :integer :min 0
:affects #{:head :mouth :eye :brow :teeth}}
:contour-avg {:area :subject :default 1 :type :integer :min 0
:affects #{:mouth :eye :brow}}
:verts {:area :mouth :default 8 :type :integer :min 4 :even? true}
:aperture-cut {:area :mouth :default 0.12 :type :number :min 0 :max 1}
:eye-verts {:area :eye :default 8 :type :integer :min 4 :even? true}
:blink-cut {:area :eye :default 0.13 :type :number :min 0}
:gaze-gain {:area :eye :default 1 :type :number :min 0}
:gaze-step {:area :eye :default 0.08 :type :number :min 0}
:iris-size {:area :eye :default 0.42 :type :number :min 0}
:lash-weight {:area :eye :default 0.06 :type :number :min 0}
:pupil-size {:area :eye :default 0.15 :type :number :min 0}
:brow-verts {:area :brow :default 6 :type :integer :min 4 :even? true}
:brow-gain {:area :brow :default 1 :type :number :min 0}
:brow-step {:area :brow :default 0.08 :type :number :min 0}
:brow-weight {:area :brow :default 0.05 :type :number :min 0}
:cavity-erode {:area :teeth :default 0.18 :type :number :min 0}
:tongue-reject {:area :teeth :default 0.18 :type :number :min 0}
:blob-grow {:area :teeth :default 0 :type :integer}
:top-bias {:area :teeth :default 0.6 :type :number}
:teeth-verts {:area :teeth :default 10 :type :integer :min 4}
:min-area {:area :teeth :default 12 :type :integer :min 0}
:teeth-on {:area :teeth :default 0.12 :type :number :min 0}
:teeth-smooth {:area :teeth :default 1 :type :integer :min 0}})
(def defaults
(into {} (map (fn [[id spec]] [id (:default spec)])) definitions))
(def areas #{:subject :mouth :eye :brow :teeth})
(defn affected-areas [id]
(when-let [spec (get definitions id)]
(or (:affects spec) #{(:area spec)})))
(defn for-area [wanted-area]
(into {} (keep (fn [[id {:keys [area default]}]]
(when (= wanted-area area) [id default])))
definitions))
(defn valid-value? [id value]
(when-let [{:keys [type min max] must-even? :even?} (get definitions id)]
(and (case type
:integer (integer? value)
:number (number? value)
false)
(or (nil? min) (<= min value))
(or (nil? max) (<= value max))
(or (not must-even?) (and (integer? value) (even? value))))))
(defn valid-settings? [area settings]
(and (map? settings)
(every? (fn [[id value]]
(and (= area (get-in definitions [id :area]))
(valid-value? id value)))
settings)))

3
frontend/src/arthur/domain/scene.cljs
View file

@ -28,6 +28,7 @@
vector of numbers, and they read the same way, which is what makes freezing
fill in the same channel rather than convert into a second format."
(:require [arthur.domain.channel :as ch]
[arthur.domain.feature :as feature]
[arthur.domain.node :as node]
[arthur.domain.palette :as pal]))
@ -451,8 +452,8 @@
(into (for [[id n] nodes
p (node/problems n)]
(str "node " (pr-str id) ": " p)))
(into (feature/problems scene))
(into (try
(doall (map #(depth nodes %) (keys nodes)))
nil
(catch :default e [(ex-message e)])))))))

3
frontend/src/arthur/events/footage.cljs
View file

@ -58,7 +58,8 @@
(defn- build-clip [manifest {:keys [dense detected dimensions interior missing first-real]}]
(let [[w h] dimensions
frozen (take/footage manifest {:dense dense :detected detected
:dimensions dimensions :interior interior})
:dimensions dimensions :interior interior
:presence (:presence manifest)})
scene (:scene frozen)]
(assoc (select-keys scene [:fps :frames :width :height])
:display-fps (:fps scene)

49
frontend/src/arthur/flow/condition/eyes.cljs
View file

@ -13,20 +13,47 @@
:y (/ (+ (:y a) (:y b)) 2)
:w (distance a b)}))
(defn- mean-width [lids]
(/ (reduce + (map (comp :w socket) lids)) (count lids)))
(defn- hold-observed
"Give temporal filters real samples at every index. The freeze mask still
marks the gap absent; these held values are only numeric placeholders."
[values observed fallback]
(let [values (vec values)
seed (or (first (keep-indexed (fn [i v]
(when (and (nth observed i) (some? v)) v))
values))
fallback (first values))]
(loop [f 0 last-value seed out []]
(if (= f (count values))
out
(let [v (nth values f)
next-value (if (and (nth observed f) (some? v)) v last-value)]
(recur (inc f) next-value (conj out next-value)))))))
(defn- mean-width [lids observed]
(let [valid (seq (keep-indexed (fn [i lid] (when (nth observed i) lid)) lids))
rings (or valid lids)]
(/ (reduce + (map (comp :w socket) rings)) (count rings))))
(defn apply-defaults
[{:keys [fps contour-avg blink-cut gaze-gain gaze-step iris-size
lash-weight pupil-size]} {:keys [lid-r lid-l open-r open-l gaze]
lash-weight pupil-size]} {:keys [lid-r lid-l open-r open-l gaze
observed-r observed-l gaze-observed]
:as measured}]
(let [fps (or fps 30)
right (condition/contours {:contour-avg contour-avg} lid-r)
left (condition/contours {:contour-avg contour-avg} lid-l)
w-r (mean-width right)
w-l (mean-width left)
observed-r (or observed-r (vec (repeat (count lid-r) true)))
observed-l (or observed-l (vec (repeat (count lid-l) true)))
gaze-observed (or gaze-observed (vec (repeat (count gaze) true)))
right (condition/contours {:contour-avg contour-avg}
(hold-observed lid-r observed-r nil))
left (condition/contours {:contour-avg contour-avg}
(hold-observed lid-l observed-l nil))
w-r (mean-width right observed-r)
w-l (mean-width left observed-l)
w (/ (+ w-r w-l) 2)
rest (condition/median-point gaze)
valid-gaze (seq (keep-indexed (fn [i point]
(when (nth gaze-observed i) point)) gaze))
gaze (hold-observed gaze gaze-observed {:x 0 :y 0})
rest (if valid-gaze (condition/median-point valid-gaze) {:x 0 :y 0})
px (mapv (fn [{:keys [x y]}]
{:x (* (- x (:x rest)) gaze-gain w)
:y (* (- y (:y rest)) gaze-gain w)}) gaze)
@ -42,8 +69,10 @@
:lid-r right :lid-l left
:lash-r (mapv #(ring/offset-ring % (* lash-weight w-r)) right)
:lash-l (mapv #(ring/offset-ring % (* lash-weight w-l)) left)
:shut-r (condition/resolve-blink open-r blink)
:shut-l (condition/resolve-blink open-l blink)
:shut-r (condition/resolve-blink
(hold-observed open-r observed-r nil) blink)
:shut-l (condition/resolve-blink
(hold-observed open-l observed-l nil) blink)
:iris-r (centres right) :iris-l (centres left)
:radius-r (* 0.5 iris-size w-r)
:radius-l (* 0.5 iris-size w-l)

81
frontend/src/arthur/flow/freeze.cljs
View file

@ -76,10 +76,9 @@
`:ctor` makes the array — a function and not a type, because `new` is not
something a value can carry — `:scale` is the fixed-point scale or nil, and
`:absent` an optional per-frame predicate. Absence is the SUBJECT's, not the
part's, so it is written into every track of the block: the mouth outline and
the mouth interior are one face, and a frame that face was not on has no pose
for either.
`:absent` an optional (track, frame) predicate. The state is per track, so one
occluded eye can be absent while its partner still has a value. A full-face
miss marks every track absent.
Written with `dotimes` and `aset` rather than as a fold, and that is the
exception rather than the rule in this codebase: the destination is a typed
@ -110,7 +109,7 @@
:track i :frame f :component k})))
q)
v))))
(when (and state (absent f))
(when (and state (absent i f))
(aset state (+ (* i nf) f) ch/absent-bit))))))
{:data data :state state :stride stride :frames nf :scale scale
:offsets (mapv #(* % nf stride) (range n))}))
@ -372,16 +371,24 @@
{:by by :analysis analysis
:params (merge {:anchor-avg anchor-avg :contour-avg contour-avg}
extra)})
eye-block (pack {:ctor #(js/Int16Array. %) :scale geom-scale :absent absent}
eye-block (pack {:ctor #(js/Int16Array. %) :scale geom-scale
:absent (when absent (fn [i f]
(absent (if (< i 2) :eye-r :eye-l) f)))}
(mapv #(rings->flat % eye-verts)
[(:lash-r eyes) (:lid-r eyes)
(:lash-l eyes) (:lid-l eyes)]))
iris-block (pack {:ctor #(js/Float32Array. %) :absent absent}
iris-block (pack {:ctor #(js/Float32Array. %)
:absent (when absent (fn [i f]
(absent (if (zero? i) :eye-r :eye-l) f)))}
[(:iris-r eyes) (:iris-l eyes)])
brow-block (pack {:ctor #(js/Int16Array. %) :scale geom-scale :absent absent}
brow-block (pack {:ctor #(js/Int16Array. %) :scale geom-scale
:absent (when absent (fn [i f]
(absent (if (zero? i) :brow-r :brow-l) f)))}
(mapv #(rings->flat % brow-verts)
[(:ring-r brows) (:ring-l brows)]))
brow-pos-block (pack {:ctor #(js/Float32Array. %) :absent absent}
brow-pos-block (pack {:ctor #(js/Float32Array. %)
:absent (when absent (fn [i f]
(absent (if (zero? i) :brow-r :brow-l) f)))}
[(:pos-r brows) (:pos-l brows)])
eye-node (fn [id z track]
{:id id :name (clojure.core/name id) :kind :poly :parent :head :z z
@ -434,11 +441,11 @@
"Freeze the pixel-derived radial contour under the mouth cavity. Missing
contours use the dense block's absence bit; contrast decides editable :vis."
[name {:keys [analysis teeth-verts cavity-erode tongue-reject blob-grow
top-bias teeth-on teeth-smooth]} detected
top-bias teeth-on teeth-smooth]} absent-feature
{:keys [contours shown]}]
(let [key (str name "/teeth")
absent #(or (nil? (nth contours %))
(and detected (not (nth detected %))))
absent (fn [_ f] (or (nil? (nth contours f))
(and absent-feature (absent-feature :teeth f))))
empty-points (vec (repeat (* 2 teeth-verts) 0))
values (mapv (fn [ring]
(if ring
@ -495,7 +502,9 @@
:eyes :brows the CONDITIONED head-local feature measurements
:teeth optional pixel-derived, conditioned radial contour
:aperture head-local aperture per frame
:detected optional per-frame booleans; absent frames get the state mask
:detected optional per-frame face booleans
:presence optional feature-id -> per-frame booleans; false means an
unobserved feature, even if the rest of the face was found
The node tree is the one docs/animation-model.md specifies, and the two groups
are two different things wanting the same transform:
@ -522,21 +531,32 @@
provenance having to be rebuilt."
[{:keys [name fps stage expose verts aperture-cut head kept
analysis anchor-avg contour-avg] :as params}
{:keys [transforms outer inner detected eyes brows teeth] :as inputs}]
{:keys [transforms outer inner detected presence eyes brows teeth] :as inputs}]
(let [nf (count outer)
absent (when detected #(not (nth detected % true)))
_ (doseq [[id track] presence]
(when (not= nf (count track))
(throw (ex-info "feature presence track must match the clip"
{:feature id :frames nf :actual (count track)}))))
absent (when (or detected presence)
(fn [id f]
(or (and detected (not (nth detected f true)))
(and (contains? presence id)
(not (nth (get presence id) f))))))
head-absent (when detected (fn [_ f] (not (nth detected f true))))
geom-k (str name "/geom")
head-k #(str name "/head-" %)
prov (fn [by extra]
{:by by :analysis analysis
:params (merge {:anchor-avg anchor-avg} extra)})
rings (pack {:ctor #(js/Int16Array. %) :scale geom-scale :absent absent}
rings (pack {:ctor #(js/Int16Array. %) :scale geom-scale
:absent (when absent (fn [_ f] (absent :mouth f)))}
[(rings->flat outer verts) (rings->flat inner verts)])
;; The anchor, inverted and split into its three components. Three blocks
;; and not one: they are three channels, they have three strides, and a
;; single block would need a per-component offset table to say so.
inv (mapv invert transforms)
xf (fn [f] (pack {:ctor #(js/Float32Array. %) :absent absent} [(mapv f inv)]))
xf (fn [f] (pack {:ctor #(js/Float32Array. %) :absent head-absent}
[(mapv f inv)]))
pos (xf (fn [t] [(:tx t) (:ty t)]))
rot (xf (fn [t] [(:theta t)]))
scale (xf (fn [t] [(:s t) (:s t)]))
@ -549,10 +569,31 @@
anchor-prov (prov :anchor/similarity nil)
roto (fn [by] (prov by {:verts verts :contour-avg contour-avg}))
features (when (and eyes brows) (feature-parts name absent params inputs))
interior (when teeth (interior-part name params detected teeth))
interior (when teeth (interior-part name params absent teeth))
scene {:name name
:frames nf
:fps fps
;; A gap changes channel state, never these IDs or pair links.
:subjects {:face-1 {:id :face-1 :params {}}}
:features (merge
{:mouth {:id :mouth :subject :face-1 :area :mouth
:nodes [:mouth :mouth-in] :params {}}}
(when features
{:eye-r {:id :eye-r :subject :face-1 :area :eye
:nodes [:eye-r :eye-r-in :iris-r :pupil-r] :params {}}
:eye-l {:id :eye-l :subject :face-1 :area :eye
:nodes [:eye-l :eye-l-in :iris-l :pupil-l] :params {}}
:brow-r {:id :brow-r :subject :face-1 :area :brow
:nodes [:brow-r] :params {}}
:brow-l {:id :brow-l :subject :face-1 :area :brow
:nodes [:brow-l] :params {}}})
(when teeth
{:teeth {:id :teeth :subject :face-1 :area :teeth
:nodes [:teeth] :params {}}}))
:groups (if features
{:eyes-1 {:id :eyes-1 :kind :eye-pair :subject :face-1
:members [:eye-r :eye-l] :params {}}}
{})
;; Stage dimensions, on the clip and not on the footage. See
;; face-placement: nothing below here knows the frame size.
:width (first stage)
@ -591,6 +632,10 @@
(prov :roto/mouth-aperture
{:aperture-cut aperture-cut}))}}}
(:nodes features) (:nodes interior))}
presence-check (doseq [id (keys presence)]
(when-not (contains? (:features scene) id)
(throw (ex-info "presence track names no feature in this scene"
{:feature id :features (keys (:features scene))}))))
;; Tier 2, behind a handle. The keys are descriptive because there is no
;; hashing yet; they become the blocks' sha256 when the backend arrives
;; and nothing above here changes, which is the point of a handle.

29
frontend/src/arthur/flow/ingest.cljs
View file

@ -2,6 +2,32 @@
"Read a pre-extracted take. The manifest owns timing and the exact frame count."
(:require [clojure.string :as str]))
(defn feature-presence
"Expand one-based, inclusive absence intervals from a manifest into boolean
observation tracks. Unknown features are rejected by freeze, where the scene
knows its feature IDs."
[frames absence]
(when (some? absence)
(when-not (map? absence)
(throw (ex-info "feature-absence must be a map of feature IDs to intervals"
{:feature-absence absence})))
(into {}
(map (fn [[id intervals]]
(when-not (and (keyword? id) (sequential? intervals)
(every? (fn [span]
(and (vector? span) (= 2 (count span))
(every? integer? span)
(<= 1 (first span) (second span) frames)))
intervals))
(throw (ex-info "feature-absence intervals must be [first last] source frames"
{:feature id :intervals intervals :frames frames})))
[id (mapv (fn [frame]
(not-any? (fn [[first-frame last-frame]]
(<= first-frame frame last-frame))
intervals))
(range 1 (inc frames)))]))
absence)))
(defn- valid-manifest [m]
(let [fps (js/Number (:fps m))
frames (js/Number (:frames m))]
@ -10,7 +36,8 @@
(string? (:dir m)) (seq (:dir m))
(string? (:audio m)) (seq (:audio m)))
(throw (ex-info "manifest.json needs fps, frames (1–900), dir and audio" {:manifest m})))
(assoc m :fps fps :frames frames)))
(assoc m :fps fps :frames frames
:presence (feature-presence frames (:feature-absence m)))))
(defn manifest!
[path]

48
frontend/src/arthur/flow/measure/eyes.cljs
View file

@ -16,22 +16,38 @@
(geom/apply-sim-all tf (anchor/pick frame table aspect)))
dense transforms))
(defn- iris-pair [iris-a iris-b corners-r]
(defn- observed-track [frames detected presence feature-id]
(let [part (get presence feature-id)]
(mapv (fn [f]
(and (or (nil? detected) (nth detected f))
(or (nil? part) (nth part f))))
(range frames))))
(defn- iris-pair [iris-a iris-b corners-r corners-l observed-r observed-l]
(when iris-a
;; An occluded eye may still get a plausible iris from MediaPipe. It gets no
;; vote. Either eye can establish the pairing when its partner is absent.
(let [votes (reduce +
(map (fn [a b [outer inner]]
(if (< (distance (first a) (midpoint outer inner))
(distance (first b) (midpoint outer inner)))
1 -1))
iris-a iris-b corners-r))]
(for [f (range (count iris-a))
[seen corners sign] [[(nth observed-r f) (nth corners-r f) 1]
[(nth observed-l f) (nth corners-l f) -1]]
:when seen]
(let [a (first (nth iris-a f))
b (first (nth iris-b f))
[outer inner] corners]
(if (< (distance a (midpoint outer inner))
(distance b (midpoint outer inner)))
sign (- sign)))))]
(if (pos? votes) {:right :a :left :b} {:right :b :left :a}))))
(defn measure
"All rings and iris coordinates use the anchor's head-local image-height unit.
Eye openness and gaze use each eye's rigid corner width as their unit. Iris
block identity is voted from proximity over the whole take."
[{:keys [aspect debug?]} {:keys [dense transforms]}]
block identity is voted from observed eyes over the whole take."
[{:keys [aspect debug?]} {:keys [dense transforms detected presence]}]
(let [track (partial local-track dense transforms aspect)
observed-r (observed-track (count dense) detected presence :eye-r)
observed-l (observed-track (count dense) detected presence :eye-l)
lid-r (track lm/EYE-R-RING)
lid-l (track lm/EYE-L-RING)
corners-r (track lm/EYE-R-CORNERS)
@ -41,7 +57,7 @@
has-iris? (every? #(> (count %) (last lm/IRIS-B)) dense)
iris-a (when has-iris? (track lm/IRIS-A))
iris-b (when has-iris? (track lm/IRIS-B))
pairing (iris-pair iris-a iris-b corners-r)
pairing (iris-pair iris-a iris-b corners-r corners-l observed-r observed-l)
iris-for (fn [side f]
(first (nth (if (= (get pairing side) :a) iris-a iris-b) f)))
openness (fn [corners lids]
@ -57,17 +73,25 @@
iris (iris-for side f)]
{:x (/ (- (:x iris) (:x c)) w)
:y (/ (- (:y iris) (:y c)) w)}))
r (one :right corners-r)
l (one :left corners-l)]
(midpoint r l)))
r (when (nth observed-r f) (one :right corners-r))
l (when (nth observed-l f) (one :left corners-l))]
(cond
(and r l) (midpoint r l)
r r
l l
:else nil)))
(range (count dense)))
(vec (repeat (count dense) {:x 0 :y 0})))]
{:lid-r lid-r :lid-l lid-l
:corners-r corners-r :corners-l corners-l
:open-r (openness corners-r lids-r)
:open-l (openness corners-l lids-l)
:observed-r observed-r :observed-l observed-l
:gaze-observed (if pairing (mapv #(or %1 %2) observed-r observed-l)
(vec (repeat (count dense) true)))
:gaze gaze :has-iris? (boolean pairing)
:iris-pair pairing
:debug (when debug? {:open-r (openness corners-r lids-r)
:open-l (openness corners-l lids-l)
:observed-r observed-r :observed-l observed-l
:gaze gaze :iris-pair pairing})}))

31
frontend/src/arthur/flow/take.cljs
View file

@ -9,24 +9,25 @@
[arthur.flow.measure.brows :as brows]
[arthur.flow.measure.eyes :as eyes]
[arthur.flow.measure.interior :as interior]
[arthur.flow.measure.mouth :as mouth]))
[arthur.flow.measure.mouth :as mouth]
[arthur.domain.params :as params]))
(def knobs
{:anchor-avg 2 :contour-avg 1 :verts 8 :aperture-cut 0.12
:eye-verts 8 :blink-cut 0.13 :gaze-gain 1 :gaze-step 0.08
:iris-size 0.42 :lash-weight 0.06 :pupil-size 0.15
:brow-verts 6 :brow-gain 1 :brow-step 0.08 :brow-weight 0.05
:cavity-erode 0.18 :tongue-reject 0.18 :blob-grow 0 :top-bias 0.6
:teeth-verts 10 :min-area 12 :teeth-on 0.12 :teeth-smooth 1})
(def knobs params/defaults)
(defn measure
"Condition the anchor before measuring rings through it."
[{:keys [aspect] :as params} {:keys [dense detected interior]}]
(let [fitted (anchor/fit {:aspect aspect} {:dense dense})
[{:keys [aspect] :as params} {:keys [dense detected interior presence]}]
(let [_ (doseq [[id track] presence]
(when (not= (count dense) (count track))
(throw (ex-info "feature presence track must match the source"
{:feature id :frames (count dense)
:actual (count track)}))))
fitted (anchor/fit {:aspect aspect} {:dense dense})
anchored (condition/anchor params fitted)
rings (mouth/measure {:aspect aspect}
{:dense dense :transforms (:transforms anchored)})
eye-data (eyes/measure params {:dense dense :transforms (:transforms anchored)})
eye-data (eyes/measure params {:dense dense :transforms (:transforms anchored)
:detected detected :presence presence})
brow-data (brows/measure params {:dense dense :transforms (:transforms anchored)})
inner-data (when interior
(condition-interior/apply-defaults
@ -41,7 +42,8 @@
:eyes (condition-eyes/apply-defaults params eye-data)
:brows (condition-brows/apply-defaults params brow-data eye-data)
:teeth inner-data
:detected detected)))
:detected detected
:presence presence)))
(defn build
[params inputs]
@ -51,11 +53,12 @@
"A real manifest and its detected landmarks through the same measurement and
freeze path as the synthetic take. The source cadence stays in :fps; picture
sampling is a root time map applied only after this artifact exists."
[manifest {:keys [dense detected dimensions interior]}]
[manifest {:keys [dense detected dimensions interior presence]}]
(let [[w h] dimensions
params (merge knobs
{:name "footage" :fps (:fps manifest) :aspect (/ w h)
:stage [320 200] :fit-motion? true
:expose 1 :head :as-filmed
:analysis (str "mediapipe:1.0.1/" (:source manifest))})]
(build params {:dense dense :detected detected :interior interior})))
(build params {:dense dense :detected detected :interior interior
:presence presence})))

54
frontend/test/arthur/domain/feature_test.cljs Normal file
View file

@ -0,0 +1,54 @@
(ns arthur.domain.feature-test
(:require [cljs.test :refer [deftest is]]
[arthur.domain.feature :as feature]
[arthur.domain.params :as params]))
(defn- nine-eyes []
(let [people (mapv #(keyword (str "person-" %)) (range 5))
eyes (mapv #(keyword (str "eye-" %)) (range 9))
members [[:eye-0 :eye-1] [:eye-2 :eye-3] [:eye-4 :eye-5]
[:eye-6 :eye-7] [:eye-8]]]
{:subjects (into {} (map (fn [id] [id {:id id :params {}}]) people))
:features (into {} (map-indexed (fn [i id]
[id {:id id :subject (nth people (quot i 2))
:area :eye :nodes [] :params {}}]) eyes))
:groups (into {} (map-indexed (fn [i ids]
(let [id (keyword (str "pair-" i))]
[id {:id id :kind :eye-pair
:subject (nth people i)
:members ids :params {}}])) members))
:nodes {}}))
(deftest five-people-can-have-nine-identified-eyes
(let [scene (nine-eyes)]
(is (empty? (feature/problems scene)))
(is (= [:eye-8] (get-in scene [:groups :pair-4 :members])))
(is (= :person-4 (get-in scene [:features :eye-8 :subject])))))
(deftest parameter-definitions-drive-the-current-take-defaults
(is (= #{:eye} (params/affected-areas :blink-cut)))
(is (= #{:head :mouth :eye :brow :teeth}
(params/affected-areas :anchor-avg)))
(is (params/valid-settings? :eye {:blink-cut 0.2}))
(is (not (params/valid-settings? :eye {:verts 8}))))
(deftest pair-values-are-shared-but-eye-overrides-are-independent
(let [scene (-> (nine-eyes)
(assoc-in [:groups :pair-0 :params :blink-cut] 0.2)
(assoc-in [:features :eye-1 :params :blink-cut] 0.3))]
(is (= 0.2 (:blink-cut (feature/effective-params scene :eye-0))))
(is (= 0.3 (:blink-cut (feature/effective-params scene :eye-1))))
(is (= (:eye-verts params/defaults)
(:eye-verts (feature/effective-params scene :eye-0))))
(let [before (feature/effective-params scene :eye-0)
unpaired (feature/remove-from-pair scene :eye-0)]
(is (= before (feature/effective-params unpaired :eye-0)))
(is (= [:eye-1] (get-in unpaired [:groups :pair-0 :members])))
(is (empty? (feature/problems unpaired))))))
(deftest a-pair-cannot-cross-subjects-or-own-an-eye-twice
(let [scene (nine-eyes)]
(is (seq (feature/problems
(assoc-in scene [:groups :pair-0 :members] [:eye-0 :eye-2]))))
(is (seq (feature/problems
(assoc-in scene [:groups :pair-1 :members] [:eye-0 :eye-3]))))))

67
frontend/test/arthur/flow/eye_occlusion_test.cljs Normal file
View file

@ -0,0 +1,67 @@
(ns arthur.flow.eye-occlusion-test
(:require [cljs.test :refer [deftest is]]
[arthur.demo.take :as take]
[arthur.domain.channel :as ch]
[arthur.domain.landmarks :as lm]
[arthur.domain.palette :as pal]
[arthur.domain.scene :as scene]
[arthur.flow.condition.eyes :as condition-eyes]
[arthur.flow.ingest :as ingest]
[arthur.flow.measure.eyes :as eyes]
[arthur.flow.take :as flow-take]))
(deftest an-unobserved-eye-cannot-steer-the-shared-gaze
(let [frame 11
params (assoc take/params :aspect 1)
presence (ingest/feature-presence take/frames {:eye-r [[10 14]]})
inputs {:dense @take/analysis
:transforms (:transforms @take/measured)
:presence presence}
clean (eyes/measure params inputs)
iris-ids (if (= :a (:right (:iris-pair clean))) lm/IRIS-A lm/IRIS-B)
bad-ids (distinct (concat lm/EYE-R-RING iris-ids))
corrupted (mapv (fn [i source]
(if (<= 9 i 13)
(reduce (fn [points id]
(update-in points [id :x] + 0.25))
source bad-ids)
source))
(range take/frames) @take/analysis)
masked (eyes/measure params (assoc inputs :dense corrupted))
unmasked (eyes/measure params
(dissoc (assoc inputs :dense corrupted) :presence))
drawn-clean (condition-eyes/apply-defaults params clean)
drawn-masked (condition-eyes/apply-defaults params masked)]
(is (= lm/NUM-LANDMARKS (count (nth corrupted frame))))
(is (:has-iris? masked) (str "pair " (:iris-pair masked)))
(is (false? (nth (:observed-r masked) frame)))
(is (true? (nth (:observed-l masked) frame)))
(is (= (nth (:gaze clean) frame) (nth (:gaze masked) frame)))
(is (not= (nth (:gaze clean) frame) (nth (:gaze unmasked) frame)))
(is (= (nth (:lid-r drawn-clean) frame)
(nth (:lid-r drawn-masked) frame)))
(is (= (nth (:iris-l drawn-clean) frame)
(nth (:iris-l drawn-masked) frame)))))
(deftest the-full-take-hides-only-the-annotated-eye-and-returns-to-the-same-id
(let [presence (ingest/feature-presence take/frames {:eye-r [[10 14]]})
{:keys [scene store]} (flow-take/build
(assoc take/params :aspect 1 :name "observed-gap")
{:dense @take/analysis :presence presence})
sample (fn [id frame]
(ch/value-at (get-in scene [:nodes id :channels [:geom :pts]])
frame store))]
(is (empty? (scene/problems scene)))
(is (= [:eye-r :eye-l] (get-in scene [:groups :eyes-1 :members])))
(doseq [f (range 9 14)]
(is (ch/nothing? (sample :eye-r f)))
(is (not (ch/nothing? (sample :eye-l f))))
(is (not (ch/nothing? (sample :mouth f)))))
(let [drawn (into #{} (map :node)
((scene/resolver scene store pal/index-of) 11))]
(is (not (contains? drawn :eye-r)))
(is (not (contains? drawn :iris-r)))
(is (contains? drawn :eye-l))
(is (contains? drawn :mouth)))
(is (not (ch/nothing? (sample :eye-r 8))))
(is (not (ch/nothing? (sample :eye-r 14))))))

32
frontend/test/arthur/flow/freeze_test.cljs
View file

@ -381,6 +381,27 @@
;; ---------------------------------------------------------------------------
;; presence is not visibility
(deftest an-eye-can-disappear-and-return-under-the-same-identity
(let [gap (set (range 40 60))
presence {:eye-r (mapv #(not (contains? gap %)) (range take/frames))}
c (freeze/clip (assoc take/params :name "one-eye-gappy")
(assoc @take/measured :presence presence))
sc (:scene c)
at (fn [id f]
(ch/value-at (get-in sc [:nodes id :channels [:geom :pts]]) f (:store c)))]
(is (= [:eye-r :eye-l] (get-in sc [:groups :eyes-1 :members])))
(is (= :eye-r (get-in sc [:features :eye-r :id])))
(is (not (ch/nothing? (at :eye-r 39))))
(is (ch/nothing? (at :eye-r 50)))
(is (not (ch/nothing? (at :eye-r 60))))
(is (not (ch/nothing? (at :eye-l 50))))
(is (not (ch/nothing? (at :mouth 50))))
(let [drawn-nodes (into #{} (map :node) ((scene/resolver sc (:store c) pal/index-of) 50))]
(is (not (contains? drawn-nodes :eye-r)))
(is (contains? drawn-nodes :eye-l))
(is (contains? drawn-nodes :mouth)))
(is (empty? (scene/problems sc)))))
(deftest an-undetected-frame-has-no-pose-at-all
;; A subject that was not on the frame has NO VALUE, which is different from a
;; part being switched off. The mask lands on every block of the freeze, so an
@ -447,10 +468,15 @@
;; beat are not. This is the numeric half of step 5's done-criterion; the other
;; half is a picture and lives in test/browser/take.mjs.
(let [locked (freeze/head-mode {:mode :locked} @clip)
shot (fn [f] (render locked f))
shot (fn [f]
(let [r (raster/make W H)
mouth (filter #(= :mouth (:node %)) (ops-at locked f))]
(raster/clear! r (get pal/index-of :bg))
(raster/draw-ops! r mouth)
(vec (array-seq (:buf r)))))
differ (fn [a b] (count (remove true? (map = a b))))]
;; Beat 1 is wide open and beat 3 is shut. Rendered with the head LOCKED, so
;; what differs is articulation and not the head wandering across the stage.
;; Beat 1 is wide open and beat 3 is shut. Compare only the mouth outline:
;; eyes and brows added in step 7 have their own motion within the beat.
(is (> (differ (shot 10) (shot 28)) 300)
"the open and the shut mouth rasterise the same")
;; And within a beat, on the exposure grid, it holds.

15
frontend/test/arthur/flow/ingest_test.cljs Normal file
View file

@ -0,0 +1,15 @@
(ns arthur.flow.ingest-test
(:require [cljs.test :refer [deftest is]]
[arthur.flow.ingest :as ingest]))
(deftest feature-absence-uses-the-source-frame-numbers
(let [tracks (ingest/feature-presence
8 {:eye-r [[3 5]] :eye-l [[1 1] [8 8]]})]
(is (= [true true false false false true true true] (:eye-r tracks)))
(is (= [false true true true true true true false] (:eye-l tracks)))))
(deftest malformed-feature-absence-fails-before-analysis
(doseq [bad [{:eye-r [[0 2]]}
{:eye-r [[6 9]]}
{:eye-r [[5 3]]}]]
(is (thrown? ExceptionInfo (ingest/feature-presence 8 bad)))))

44
js/app.js
View file

@ -51,6 +51,7 @@ const state = {
lead: 0, // performance-track offset in frames
exposure: 1, // 1 = on 1s, 2 = on 2s. Picture holds; audio does not.
interior: null, // per-frame teeth measurement from image content
detected: null, // per-frame: did MediaPipe really see a face here?
teeth: null, // resolved per-frame {show, t} after knobs
eyes: null, // resolved per-frame lid rings, shut flags, iris discs
brows: null, // resolved per-frame brow rings after quantised raise
@ -167,18 +168,29 @@ async function initLandmarker() {
async function detectAll(images) {
const lm = await initLandmarker();
const cv = document.createElement('canvas');
const dense = [], missing = [];
const dense = [], missing = [], detected = [];
for (let i = 0; i < images.length; i++) {
const im = images[i];
cv.width = im.naturalWidth; cv.height = im.naturalHeight;
cv.getContext('2d').drawImage(im, 0, 0);
const out = lm.detect(cv);
if (out.faceLandmarks && out.faceLandmarks.length) dense.push(out.faceLandmarks[0]);
else { missing.push(i); dense.push(dense.length ? dense[dense.length - 1] : null); }
if (out.faceLandmarks && out.faceLandmarks.length) {
dense.push(out.faceLandmarks[0]); detected.push(true);
} else {
missing.push(i); detected.push(false);
dense.push(dense.length ? dense[dense.length - 1] : null);
}
if (i % 4 === 0) status(`detecting… ${i + 1}/${images.length}`);
}
if (dense[0] === null) throw new Error('no face found in the first frame');
return { dense, missing };
// A gap mid-take holds the previous frame, but a gap at the TOP has no
// previous to hold - a face that starts occluded or walks in late left
// leading nulls, and this used to throw and discard the whole take. Back-fill
// from the first real detection: the mirror of hold-previous, and the only
// fill that is a real pose from this take rather than an invention.
const firstReal = dense.findIndex((d) => d !== null);
if (firstReal < 0) throw new Error('no face found in any frame — check framing and light');
for (let i = 0; i < firstReal; i++) dense[i] = dense[firstReal];
return { dense, missing, detected, firstReal };
}
// Interior measurement is a function of pixels alone, so it runs once with
@ -217,10 +229,19 @@ function rebuild(resetKeep) {
state.stab = stabilize(state.dense, o.smoothWin, state.aspect);
// The neutral drives calibration and the placeholder plate, so it has to be a
// frame the camera actually saw: a back-filled or held frame is a duplicate
// pose, and letting one win this contest would calibrate the whole take
// against a landmark set that belongs to some other moment.
const ap = state.stab.aperture;
const real = state.detected;
const shut = (a, b) => (b < 0 || ap[a] < ap[b] ? a : b);
const head = Math.max(1, Math.floor(N / 4));
let neutral = 0;
for (let i = 0; i < head; i++) if (ap[i] < ap[neutral]) neutral = i;
let neutral = -1;
for (let i = 0; i < head; i++) if (!real || real[i]) neutral = shut(i, neutral);
// Whole head of the take occluded: widen to any real frame rather than give up.
if (neutral < 0) for (let i = 0; i < N; i++) if (!real || real[i]) neutral = shut(i, neutral);
if (neutral < 0) neutral = 0;
state.neutral = neutral;
state.xform = makeXform(state.stab, neutral);
@ -1072,8 +1093,8 @@ async function runFrames() {
status(`no frames in ${el('framedir').value}/ — run extract.sh first`, 'err');
return;
}
const { dense, missing } = await detectAll(images);
state.images = images; state.dense = dense;
const { dense, missing, detected, firstReal } = await detectAll(images);
state.images = images; state.dense = dense; state.detected = detected;
state.aspect = images[0].naturalWidth / images[0].naturalHeight;
status('measuring mouth interiors…');
state.interior = measureAll(images, dense, opts());
@ -1086,7 +1107,9 @@ async function runFrames() {
status(`${images.length} frames · ${images[0].naturalWidth}x${images[0].naturalHeight} · ` +
`${state.fps}fps · ${dur}s` +
(state.audio ? ' · audio loaded' : ' · no audio') +
(missing.length ? ` · no face on ${missing.length} (held previous)` : ''),
(missing.length ? ` · no face on ${missing.length}` +
(firstReal ? ` (${firstReal} at the top back-filled from ${firstReal + 1}, rest held)`
: ' (held previous)') : ''),
missing.length ? 'warn' : 'ok');
} catch (e) { status(e.message, 'err'); console.error(e); }
}
@ -1096,6 +1119,7 @@ function runSynthetic() {
attachAudio(null);
state.fps = 12;
state.aspect = 1; // synthetic landmarks are generated square
state.detected = null; // no detection ran, so every frame counts as real
state.lead = 0;
state.interior = null; // no pixels, so no teeth