797 lines
41 KiB
Clojure
797 lines
41 KiB
Clojure
(ns arthur.flow.freeze
|
|
"Stage 5, the freeze: measurements become CHANNELS.
|
|
|
|
This is the hinge the whole model turns on. Freezing is NOT a conversion into a
|
|
second format — there is one format, and freezing fills it in. That is what
|
|
makes \"the only difference between rotoscoped and hand-authored is a flag\"
|
|
literally true: what comes out of here is the same `:channels` map a hand fills
|
|
in sparsely, and the flag is `:generated`, which nothing in the renderer reads.
|
|
|
|
Three conversions happen here and nowhere else.
|
|
|
|
MAPS BECOME FLAT. `flow/measure/*` speaks {:x :y}, because it is the numeric
|
|
oracle and a faithful port was worth more there than a fast one. A channel
|
|
value is FLAT — [x0 y0 x1 y1 …] — in authored vectors and dense blocks alike.
|
|
`rings->flat` is the only place that crossing is made.
|
|
|
|
FLOATS BECOME FIXED POINT. Geometry goes into an Int16 block with the scale in
|
|
its header; see `geom-scale`.
|
|
|
|
A SIMILARITY BECOMES THREE CHANNELS. `{s θ tx ty}` IS `[:xform :scale]`,
|
|
`[:xform :rot]` and `[:xform :pos]`, so the anchor drops onto `:head` with no
|
|
adapter — which is the sign the decomposition is the right one.
|
|
|
|
`makeXform` IS NOT HERE AND IS NOT COMING. The prototype centres on the face
|
|
oval's bbox and zooms until the face is 80% of the raster height, so every
|
|
stored vertex carries a cropping decision made once, at analysis time, from one
|
|
frame's landmarks. Here the geometry stays in the node's own local space and the
|
|
framing is `[:xform :*]` on each face's own authored `:place` node, which the
|
|
stage clips.
|
|
Project dimensions are therefore independent of the footage — see
|
|
`face-placement`, and \"What space geometry is in\" in docs/animation-model.md.
|
|
|
|
Not `flow/key`. Traced lips, lids and brows keep every source frame; only a
|
|
plate, which a human draws, is worth decimating. Sparse visibility keys capture
|
|
decisions about the mouth cavity, blink and teeth without thinning geometry."
|
|
(:require [arthur.domain.channel :as ch]
|
|
[arthur.domain.feature :as feature]
|
|
[arthur.domain.geom :as geom]
|
|
[arthur.domain.node :as node]
|
|
[arthur.domain.pick :as pick]
|
|
[arthur.domain.palette :as pal]
|
|
[arthur.domain.ring :as ring]
|
|
[arthur.domain.trace :as trace]
|
|
[arthur.flow.address :as address]))
|
|
|
|
;; ---------------------------------------------------------------------------
|
|
;; fixed point
|
|
|
|
(def ^:const geom-scale
|
|
"Q14: the stored integer is the value times 16384.
|
|
|
|
Geometry is head-local and its unit is ONE IMAGE HEIGHT, so 16384 covers ±2
|
|
image heights in an Int16 and quantises to 1/16384 = 6.1e-5 of an image height.
|
|
Against a face placed so that its 0.29-image-height oval fills most of a 200px
|
|
stage — around 850 stage pixels per image height — that is 0.05px, two orders
|
|
below anything the rasteriser can express.
|
|
|
|
A power of two, so the decode is a floating-point exact division and freezing
|
|
the same numbers twice cannot drift.
|
|
|
|
It is a CONSTANT here and a FIELD in the block header, and the difference
|
|
matters: a painted cel's geometry is in stage pixels, where ±2 would be absurd
|
|
and 1/16384 of a pixel is waste. Each block says what its own space needs."
|
|
16384)
|
|
|
|
(def ^:private ^:const int16-max 32767)
|
|
|
|
;; ---------------------------------------------------------------------------
|
|
;; blocks
|
|
|
|
(defn- pack
|
|
"Tracks -> one dense block, NODE-MAJOR and FRAME-MINOR:
|
|
|
|
offset(track i) = i · frames · stride
|
|
value(i, f) = data[offset(i) + f · stride]
|
|
|
|
No per-frame header and no indirection, which FIXED TOPOLOGY is what buys: every
|
|
frame of a part carries the same component count with the same meanings, so a
|
|
frame is a rectangular slice at an arithmetic offset. A variable vertex count
|
|
would force an offset table and a scan per frame, so the aesthetic constraint is
|
|
a performance asset rather than a cost. `demo/swarm` holds the same layout and
|
|
is the load test for it.
|
|
|
|
`:type` names the array — \"int16\" or \"float32\" — rather than handing over a
|
|
constructor, because the type is also a field in the block's descriptor and the
|
|
two must not be able to disagree. `:scale` is the fixed-point scale or nil.
|
|
|
|
ABSENCE IS PER TRACK, and `:features` is what says whose. Each track names the
|
|
feature it follows, so one occluded eye can be absent while its partner still
|
|
has a value; a subject id means the track follows whole-face detection and no
|
|
feature, which is what the head's blocks do. `absent?` is then asked
|
|
`(absent? feature f)` and never about a track index.
|
|
|
|
An earlier shape passed a `(track, frame)` predicate instead, and each call site
|
|
derived a feature from an index — `(if (< i 2) :eye-r :eye-l)` — so the
|
|
predicate and the vector of tracks beside it had to agree BY HAND, in five
|
|
places, with a left/right swap for a failure mode. docs/port-plan.md warns about
|
|
that swap twice: every part is still roughly where it belongs, so it survives
|
|
inspection. Naming the feature per track deletes the derivation, and it hands
|
|
`flow/address` the same list for the block's observation digest, so the key and
|
|
the mask cannot disagree either.
|
|
|
|
`:missing` is an additional per-track predicate for absence that is not a
|
|
feature's: the teeth have no contour on a frame no contour could be extracted
|
|
from, which is a different fact from the teeth being occluded.
|
|
|
|
Written with `dotimes` and `aset` rather than as a fold, and that is the
|
|
exception rather than the rule in this codebase: the destination is a typed
|
|
array, so there is nothing to accumulate into and a collection idiom here would
|
|
allocate a seq per frame to throw away."
|
|
[{:keys [type scale features absent? missing]} tracks]
|
|
(when-not (= (count tracks) (count features))
|
|
(throw (ex-info "every track of a block names the feature it follows"
|
|
{:tracks (count tracks) :features (count features)})))
|
|
(let [n (count tracks)
|
|
nf (count (first tracks))
|
|
stride (count (first (first tracks)))
|
|
ctor (case type
|
|
"int16" #(js/Int16Array. %)
|
|
"float32" #(js/Float32Array. %)
|
|
(throw (ex-info "a block's element type is \"int16\" or \"float32\""
|
|
{:type type})))
|
|
data (ctor (* n nf stride))
|
|
gone? (fn [i f] (or (and absent? (absent? (nth features i) f))
|
|
(and missing (missing i f))))
|
|
state (when (or absent? missing) (js/Uint8Array. (* n nf)))]
|
|
(dotimes [i n]
|
|
(let [track (vec (nth tracks i))
|
|
base (* i nf stride)]
|
|
(dotimes [f nf]
|
|
(let [vs (nth track f)
|
|
o (+ base (* f stride))]
|
|
(dotimes [k stride]
|
|
(let [v (nth vs k)]
|
|
(aset data (+ o k)
|
|
(if scale
|
|
(let [q (js/Math.round (* v scale))]
|
|
;; Saturating would read as articulation flattening off
|
|
;; at the extremes — a bad detection, not a bad scale.
|
|
(when (> (abs q) int16-max)
|
|
(throw (ex-info "value does not fit the block's fixed point"
|
|
{:value v :scale scale :quantised q
|
|
:track i :frame f :component k})))
|
|
q)
|
|
v))))
|
|
(when (and state (gone? i f))
|
|
(aset state (+ (* i nf) f) ch/absent-bit))))))
|
|
{:data data :state state :stride stride :frames nf :scale scale :type type
|
|
:features features
|
|
:offsets (mapv #(* % nf stride) (range n))}))
|
|
|
|
(defn- block
|
|
"Pack the tracks and NAME the result: `pack`'s block plus the `:key` it is
|
|
stored under and the `:descriptor` that key is the hash of.
|
|
|
|
Addressing happens HERE, beside the packing, rather than at the call sites,
|
|
because a block referenced under one key and stored under another is a handle
|
|
into somebody else's array — the failure the key exists to make impossible.
|
|
|
|
`spec` is the block's identity for `flow/address`: its role, the tracks by name,
|
|
and the analysis and settings its bytes came out of. `obs` is the absence data,
|
|
which the descriptor digests down to one line."
|
|
[{:keys [role analysis params tracks] :as spec} opts obs values]
|
|
(let [features (:features opts)
|
|
blk (pack opts values)
|
|
named (address/block
|
|
{:role role :analysis analysis :params params :tracks tracks
|
|
:features features
|
|
:observation (address/observation features obs)
|
|
:layout {:type (:type blk) :scale (:scale blk)
|
|
:stride (:stride blk) :frames (:frames blk)
|
|
:tracks (count tracks)}})]
|
|
(merge blk named)))
|
|
|
|
(defn- stored
|
|
"Blocks -> the tier-2 store they go in: key -> what is kept under it.
|
|
|
|
THE DESCRIPTOR TRAVELS WITH THE BYTES. It is not in the document — tier 1 stays
|
|
the authored layer and a descriptor is derived — and it is not thrown away
|
|
either, because the server verifies `sha256(descriptor) == key` on upload and
|
|
will not take a name on trust. So it rides in tier 2, where a cache entry
|
|
knowing what produced it is the ordinary arrangement. `channel/dense-at` reads
|
|
`:data` and `:state` and ignores the rest."
|
|
[& blocks]
|
|
(into {} (map (juxt :key #(select-keys % [:data :state :descriptor]))) blocks))
|
|
|
|
(defn- dense
|
|
"Track i of a packed block, as a DENSE channel definition.
|
|
|
|
The block names its own key, so a channel cannot be pointed at one block and
|
|
stored under another's address."
|
|
[blk i generated]
|
|
{:animated? true :interp :hold
|
|
:dense (cond-> {:store (:key blk)
|
|
:offset (nth (:offsets blk) i)
|
|
:stride (:stride blk)
|
|
:frames (:frames blk)}
|
|
(:scale blk) (assoc :scale (:scale blk)))
|
|
:generated generated
|
|
:over []})
|
|
|
|
;; ---------------------------------------------------------------------------
|
|
;; geometry
|
|
|
|
(defn rings->flat
|
|
"Ring track -> one flat [x0 y0 x1 y1 …] per frame, at the vertex budget.
|
|
|
|
The vertex budget is a fixed-index SUBSAMPLE, never adaptive decimation: slot k
|
|
means the same anatomy on every frame of the shot, and that is what makes
|
|
temporal correspondence possible at all. It is applied here, after stage 4's
|
|
contour average, and the two commute because both are per-slot — which is the
|
|
whole reason the vertex knob can sit downstream of the smoothing knob instead
|
|
of alongside it. `mouth-test` asserts that rather than leaving it to look
|
|
obvious.
|
|
|
|
An ODD budget is refused. It lands off the cardinal slots — the corners and the
|
|
lip centres — and a wrongly-ordered ring self-intersects INVISIBLY at odd vertex
|
|
counts and obviously at even ones, so an odd budget is the one setting at which
|
|
the simplicity assertion stops protecting anything."
|
|
[rings verts]
|
|
(let [len (count (first rings))]
|
|
(when-not (and (integer? verts) (even? verts) (>= verts 4) (<= verts len))
|
|
(throw (ex-info "vertex budget must be even and between 4 and the ring's slot count"
|
|
{:verts verts :slots len})))
|
|
(let [slots (ring/subsample-slots len verts)]
|
|
(mapv (fn [r]
|
|
(into [] (mapcat (fn [s] (let [p (nth r s)] [(:x p) (:y p)]))) slots))
|
|
rings))))
|
|
|
|
;; ---------------------------------------------------------------------------
|
|
;; the anchor, onto :head
|
|
|
|
(defn invert
|
|
"The inverse of a similarity, STILL FACTORED.
|
|
|
|
The anchor fit maps each frame onto the shot's mean pose, so it is what takes
|
|
the head's motion OUT; geometry is stored in the space it produces. Putting the
|
|
head's motion back — the \"as filmed\" mode — is therefore the fit's inverse, and
|
|
it has to stay a {s θ t} rather than becoming a matrix, because the three
|
|
components land on three independently keyframable channels and interpolating
|
|
matrix entries is meaningless.
|
|
|
|
p ↦ s·R(θ)·p + t inverts to q ↦ (1/s)·R(-θ)·(q - t)"
|
|
[{:keys [s theta tx ty]}]
|
|
(let [s' (/ 1.0 s)
|
|
c (js/Math.cos theta)
|
|
sn (js/Math.sin theta)]
|
|
{:s s'
|
|
:theta (- theta)
|
|
:tx (* (- s') (+ (* c tx) (* sn ty)))
|
|
:ty (* (- s') (+ (* (- sn) tx) (* c ty)))}))
|
|
|
|
(defn head-mode
|
|
"Keep a subject's measured transform dense; optionally trace it.
|
|
|
|
With no trace the head reads measured frame f at frame f (free movement).
|
|
`{:frames [12] :origin :keys}` holds the measured transform of source frame
|
|
12, `{:frames [12 42] :origin :keys}` jumps to 42's at 42, and `{:origin
|
|
:start}` holds frame 0's. See `domain/trace`. The trace selects position,
|
|
rotation and scale together, so the head and registered photo cannot drift
|
|
apart. No analysis block or authored face placement changes.
|
|
|
|
ONE SUBJECT AT A TIME when `:subject` is given, and EVERY subject when it is
|
|
not. Two faces in one shot were filmed together and are posed apart: choosing
|
|
frame 12 for the second face must leave the first one running, and it does,
|
|
because a trace lives on that subject's own head node and `domain/symbol`
|
|
reads it off whatever node carries it."
|
|
[{:keys [subject] t :trace} {:keys [clip]}]
|
|
(when (and subject (not (contains? (:subjects clip) subject)))
|
|
(throw (ex-info "head mode names a subject this clip did not track"
|
|
{:subject subject :subjects (vec (sort-by str (keys (:subjects clip))))})))
|
|
(let [t (some->> t (merge {:frames []}))]
|
|
(reduce
|
|
(fn [c sid]
|
|
(let [frames (get-in c [:symbols sid :frames])]
|
|
(when-let [why (some-> t (trace/problems frames) first)]
|
|
(throw (ex-info (str "a head's trace is not one this take can hold: " why)
|
|
{:subject sid :trace t :frames frames})))
|
|
(update-in c [:symbols sid :nodes :head]
|
|
(fn [n]
|
|
(cond-> (assoc n :channels (:measured n))
|
|
t (assoc :trace t)
|
|
(nil? t) (dissoc :trace))))))
|
|
clip (if subject [subject] (sort-by str (keys (:subjects clip)))))))
|
|
|
|
;; ---------------------------------------------------------------------------
|
|
;; the aperture, onto [:vis] of :mouth-in
|
|
|
|
(defn visibility
|
|
"The aperture track -> `[:vis]` keys on the mouth interior.
|
|
|
|
`flow/measure/mouth` reports the aperture and deliberately does not threshold
|
|
it: the measurement is the inner ring's own height and the threshold is a
|
|
policy, which is stage 5. Relative to the take's PEAK aperture, not absolute, so
|
|
one number works across faces and framings — the prototype's
|
|
`ap.map(v => v / apMax < apertureThresh)`, which lives in its `app.js` and not
|
|
in its `pipeline.js`, and is easy to miss when porting from the latter.
|
|
|
|
KEYED, not dense, although docs/animation-model.md's parts table says dense. A
|
|
threshold crossing is a handful of transitions over a take, hold is the default,
|
|
and keys are what a human can correct — \"this frame's mouth should be shut\" is
|
|
the single most likely hand edit on a lip-sync take, and a dense block in tier 2
|
|
is the one shape that cannot receive it. `:generated` still rides along, which is
|
|
the point of provenance being on the channel rather than implied by its shape.
|
|
|
|
A key on frame 0 always, because the first key is the pose the part starts in."
|
|
[{:keys [aperture-cut]} {:keys [aperture]} generated]
|
|
(let [peak (reduce max aperture)
|
|
;; A take with no mouth at all has no peak to be a fraction of. Present
|
|
;; rather than absent: an all-zero aperture is a shut mouth, and the
|
|
;; interior of a shut mouth is simply not drawn.
|
|
shown (mapv (fn [v] (and (pos? peak) (>= (/ v peak) aperture-cut))) aperture)]
|
|
(assoc (ch/keyed (into {} (keep (fn [f]
|
|
(when (or (zero? f)
|
|
(not= (nth shown f) (nth shown (dec f))))
|
|
[f (nth shown f)])))
|
|
(range (count shown))) :hold)
|
|
:generated generated)))
|
|
|
|
(defn- keyed-visibility [values generated]
|
|
(assoc (ch/keyed (into {} (keep (fn [f]
|
|
(when (or (zero? f)
|
|
(not= (nth values f) (nth values (dec f))))
|
|
[f (nth values f)])))
|
|
(range (count values))) :hold)
|
|
:generated generated))
|
|
|
|
(def ^:private pose-groups
|
|
{:mouth :mouth :mouth-in :mouth :teeth :mouth
|
|
:eye-r :eye-r :eye-r-in :eye-r :iris-r :eye-r :pupil-r :eye-r
|
|
:eye-l :eye-l :eye-l-in :eye-l :iris-l :eye-l :pupil-l :eye-l
|
|
:brow-r :brow-r :brow-l :brow-l})
|
|
|
|
(defn- performance-nodes
|
|
"Mark channels that read the containing instance's pose choices."
|
|
[nodes]
|
|
(into {}
|
|
(map (fn [[id n]]
|
|
[id (if-let [group (get pose-groups id)]
|
|
(-> n
|
|
(assoc :pose-group group)
|
|
(update :channels
|
|
(fn [channels]
|
|
(into {}
|
|
(map (fn [[path ch]]
|
|
[path (cond-> ch
|
|
(and (:generated ch) (:animated? ch))
|
|
(assoc :pose-sampled? true))]))
|
|
channels))))
|
|
n)]))
|
|
nodes))
|
|
|
|
;; ---------------------------------------------------------------------------
|
|
;; the face, onto the stage
|
|
|
|
(defn- motion-points
|
|
"Every source-space point one subject's drawn features visit over the shot,
|
|
including raised brows and a moving mouth."
|
|
[{:keys [rigid transforms outer eyes brows detected]}]
|
|
(mapcat (fn [i]
|
|
(when (or (nil? detected) (nth detected i))
|
|
(let [local (concat (nth outer i)
|
|
(when eyes (concat (nth (:lash-r eyes) i)
|
|
(nth (:lash-l eyes) i)))
|
|
(when brows (concat (nth (:ring-r brows) i)
|
|
(nth (:ring-l brows) i))))]
|
|
(concat (nth rigid i)
|
|
(geom/apply-sim-all (invert (nth transforms i))
|
|
local)))))
|
|
(range (count outer))))
|
|
|
|
(defn- fit-points
|
|
"Source-space points -> the placement that puts their bounding box on stage."
|
|
[[w h] points]
|
|
(let [xs (map :x points)
|
|
ys (map :y points)
|
|
x0 (reduce min xs)
|
|
x1 (reduce max xs)
|
|
y0 (reduce min ys)
|
|
y1 (reduce max ys)
|
|
cx (/ (+ x0 x1) 2)
|
|
cy (/ (+ y0 y1) 2)
|
|
k (min (/ (* 0.8 w) (max 1e-9 (- x1 x0)))
|
|
(/ (* 0.8 h) (max 1e-9 (- y1 y0))))]
|
|
{[:xform :anchor] (ch/framed [cx cy])
|
|
[:xform :scale] (ch/framed [k k])
|
|
[:xform :pos] (ch/framed [(- (/ w 2) cx) (- (/ h 2) cy)])}))
|
|
|
|
(defn face-placement
|
|
"The face's transform on the stage, as FRAMED channels.
|
|
|
|
AUTHORED, and that is the whole difference from `makeXform`. What comes back is
|
|
a DEFAULT — the placement a human would otherwise have to make from scratch on
|
|
first open — and from then on it is an ordinary hand-placed transform on an
|
|
ordinary node. `makeXform` made the same decision and then baked it into every
|
|
vertex, where nothing could ever revise it.
|
|
|
|
The synthetic take uses the reference rigid configuration for its default.
|
|
Real footage can request `:fit-motion?`: its default fits the observed mouth,
|
|
eyes and brows in the stage across the shot.
|
|
Both are ordinary editable transforms on the face's own `:place`, never baked
|
|
into the geometry.
|
|
The face oval is not measured, because its only consumers in the prototype were
|
|
the old baked framing transform and the placeholder plate outline.
|
|
|
|
For the synthetic default, two numbers:
|
|
|
|
SCALE is stage pixels per image height, set so the reference's eye-corner span
|
|
is 40% of the stage width. Landmark-free — it is the rigid configuration's own
|
|
bounding box — and it is a fraction of the STAGE, so a 1440x1920 portrait clip
|
|
composited onto a 320x200 stage is not a problem to solve.
|
|
|
|
ANCHOR is the reference centroid, and this is where `:anchor` earns its place.
|
|
MediaPipe's normalised space has its origin at the image's TOP-LEFT CORNER, so
|
|
head-local geometry is not centred on anything; the registration point of the
|
|
face is the head's own centre, and rotation and scale have to happen about that
|
|
rather than about a corner of the footage. Getting that wrong is why hand-placed
|
|
parts swing rather than turn.
|
|
|
|
POSITION puts the anchor a QUARTER of the way down the stage, because the rigid
|
|
landmarks are eyes and nose — the upper middle of a face — so a quarter down
|
|
leaves the jaw and the mouth on the stage. Whatever hangs off is clipped, which
|
|
is not a feature to add: every fill in `domain/raster` clamps already.
|
|
|
|
ONE MAPPING, COMPUTED OVER EVERY SUBJECT AND WRITTEN INTO EACH FACE. It is
|
|
measured across all of them together — which is what keeps two faces filmed
|
|
side by side in their filmed relation — and then stored on each face's own
|
|
`:place` rather than on a group above them all. Same transform, same subtree,
|
|
one level lower: the composite is identical to the pixel.
|
|
|
|
THE OWNER IS THE POINT. A face carrying its own source-to-stage mapping is a
|
|
face that is the right size wherever it is put — dropped into another symbol,
|
|
or opened in its own tab to be drawn over — and the symbol that places it needs
|
|
to know nothing. On a group above the instances the scale belonged to the take,
|
|
so a face taken out of that take had no size at all and drew at a fraction of a
|
|
pixel."
|
|
[{:keys [stage fit-motion?]} subjects]
|
|
(let [[w h] stage
|
|
inputs (vals subjects)]
|
|
(if fit-motion?
|
|
(fit-points stage (mapcat motion-points inputs))
|
|
(let [ref (mapcat :ref inputs)
|
|
c (geom/centroid ref)
|
|
span (- (reduce max (map :x ref)) (reduce min (map :x ref)))
|
|
k (/ (* 0.4 w) span)]
|
|
{[:xform :anchor] (ch/framed [(:x c) (:y c)])
|
|
[:xform :scale] (ch/framed [k k])
|
|
[:xform :pos] (ch/framed [(- (/ w 2) (:x c))
|
|
(- (* 0.25 h) (:y c))])}))))
|
|
|
|
(defn- mouth-part [subject absent? obs
|
|
{:keys [analysis verts anchor-avg contour-avg aperture-cut] :as params}
|
|
{:keys [outer inner] :as inputs}]
|
|
(let [own (partial feature/owned subject)
|
|
mouth (own :mouth)
|
|
rings (block {:role "geom" :analysis (:id analysis) :params params
|
|
:tracks ["outer" "inner"]}
|
|
{:type "int16" :scale geom-scale
|
|
:features [mouth mouth] :absent? absent?}
|
|
obs [(rings->flat outer verts) (rings->flat inner verts)])
|
|
prov (fn [by]
|
|
{:by by :analysis (:id analysis)
|
|
:params {:anchor-avg anchor-avg :contour-avg contour-avg
|
|
:verts verts}})]
|
|
{:nodes {:mouth
|
|
{:id :mouth :name "mouth" :kind :poly :parent :head :z "a1"
|
|
:channels {[:geom :pts] (dense rings 0 (prov :roto/lips-outer))
|
|
[:style :color] (ch/framed 2)}}
|
|
:mouth-in
|
|
{:id :mouth-in :name "mouth interior" :kind :poly
|
|
:parent :mouth :z "a2"
|
|
:channels {[:geom :pts] (dense rings 1 (prov :roto/lips-inner))
|
|
[:style :color] (ch/framed 3)
|
|
[:vis] (visibility params inputs
|
|
{:by :roto/mouth-aperture
|
|
:analysis (:id analysis)
|
|
:params {:anchor-avg anchor-avg
|
|
:aperture-cut aperture-cut}})}}}
|
|
:store (stored rings)}))
|
|
|
|
(defn- feature-parts
|
|
"Freeze eyes and brows into their own dense blocks and scene nodes. This owns
|
|
only representation: the landmark correspondence, blink and pose choices have
|
|
already been settled by measure and condition."
|
|
[subject absent? obs
|
|
{:keys [eye-verts brow-verts analysis contour-avg anchor-avg] :as params}
|
|
{:keys [eyes brows]}]
|
|
(let [own (partial feature/owned subject)
|
|
provenance (fn [by extra]
|
|
{:by by :analysis (:id analysis)
|
|
:params (merge {:anchor-avg anchor-avg :contour-avg contour-avg}
|
|
extra)})
|
|
named (fn [role tracks features type values]
|
|
(block {:role role :analysis (:id analysis) :params params
|
|
:tracks tracks}
|
|
{:type type :features features :absent? absent?
|
|
:scale (when (= "int16" type) geom-scale)}
|
|
obs values))
|
|
;; Each block's tracks, named, in the order they are packed — and the
|
|
;; feature each one follows, in the same order. The two vectors are read
|
|
;; together on purpose: this is the mapping `pack` cannot check for itself,
|
|
;; and `each-dense-track-follows-its-own-features-presence` is what pins it.
|
|
eye-block (when eyes (named "eyes"
|
|
["lash-r" "lid-r" "lash-l" "lid-l"]
|
|
[(own :eye-r) (own :eye-r) (own :eye-l) (own :eye-l)]
|
|
"int16"
|
|
(mapv #(rings->flat % eye-verts)
|
|
[(:lash-r eyes) (:lid-r eyes)
|
|
(:lash-l eyes) (:lid-l eyes)])))
|
|
iris-block (when eyes
|
|
(named "iris-pos" ["iris-r" "iris-l"]
|
|
[(own :eye-r) (own :eye-l)] "float32"
|
|
[(:iris-r eyes) (:iris-l eyes)]))
|
|
brow-block (when brows
|
|
(named "brows" ["ring-r" "ring-l"]
|
|
[(own :brow-r) (own :brow-l)] "int16"
|
|
(mapv #(rings->flat % brow-verts)
|
|
[(:ring-r brows) (:ring-l brows)])))
|
|
brow-pos-block (when brows
|
|
(named "brow-pos" ["pos-r" "pos-l"]
|
|
[(own :brow-r) (own :brow-l)] "float32"
|
|
[(:pos-r brows) (:pos-l brows)]))
|
|
eye-node (fn [id z track]
|
|
{:id id :name (clojure.core/name id) :kind :poly :parent :head :z z
|
|
:channels {[:geom :pts] (dense eye-block track
|
|
(provenance :roto/eyelid {:verts eye-verts}))
|
|
[:style :color] (ch/framed 2)}})
|
|
inner-node (fn [id parent z track shut]
|
|
{:id id :name (clojure.core/name id) :kind :poly :parent parent :z z
|
|
:channels {[:geom :pts] (dense eye-block track
|
|
(provenance :roto/eye-opening
|
|
{:verts eye-verts}))
|
|
[:style :color] (ch/framed 5)
|
|
[:vis] (keyed-visibility (mapv not shut)
|
|
(provenance :roto/blink nil))}})
|
|
iris-node (fn [id parent track radius]
|
|
{:id id :name (clojure.core/name id) :kind :disc :parent parent :z "a1"
|
|
:stencil parent
|
|
:channels {[:xform :pos] (dense iris-block track
|
|
(provenance :roto/gaze nil))
|
|
[:geom :radius] (assoc (ch/framed radius)
|
|
:generated
|
|
(provenance :roto/iris-size
|
|
{:iris-size (:iris-size params)}))
|
|
[:style :color] (ch/framed 6)}})
|
|
pupil-node (fn [id parent]
|
|
{:id id :name (clojure.core/name id) :kind :rect :parent parent :z "a1"
|
|
:stencil parent
|
|
:channels {[:geom :size] (assoc (ch/framed (:pupil-size eyes))
|
|
:generated
|
|
(provenance :roto/pupil-size
|
|
{:pupil-size (:pupil-size params)}))
|
|
[:style :color] (ch/framed 7)}})
|
|
brow-node (fn [id z track]
|
|
{:id id :name (clojure.core/name id) :kind :poly :parent :head :z z
|
|
:channels {[:geom :pts] (dense brow-block track
|
|
(provenance :roto/brow {:verts brow-verts}))
|
|
[:xform :pos] (dense brow-pos-block track
|
|
(provenance :roto/brow-raise nil))
|
|
[:style :color] (ch/framed 8)}})]
|
|
{:nodes (merge
|
|
(when eyes
|
|
{:eye-r (eye-node :eye-r "a2" 0)
|
|
:eye-r-in (inner-node :eye-r-in :eye-r "a1" 1 (:shut-r eyes))
|
|
:iris-r (iris-node :iris-r :eye-r-in 0 (:radius-r eyes))
|
|
:pupil-r (pupil-node :pupil-r :iris-r)
|
|
:eye-l (eye-node :eye-l "a3" 2)
|
|
:eye-l-in (inner-node :eye-l-in :eye-l "a1" 3 (:shut-l eyes))
|
|
:iris-l (iris-node :iris-l :eye-l-in 1 (:radius-l eyes))
|
|
:pupil-l (pupil-node :pupil-l :iris-l)})
|
|
(when brows
|
|
{:brow-r (brow-node :brow-r "a4" 0)
|
|
:brow-l (brow-node :brow-l "a5" 1)}))
|
|
:store (apply stored (remove nil?
|
|
[eye-block iris-block brow-block brow-pos-block]))}))
|
|
|
|
(defn- interior-part
|
|
"Freeze the pixel-derived radial contour under the mouth cavity. Missing
|
|
contours use the dense block's absence bit; contrast decides editable :vis."
|
|
[subject {:keys [analysis teeth-verts cavity-erode tongue-reject blob-grow
|
|
top-bias teeth-on teeth-smooth] :as params} absent? obs
|
|
{:keys [contours shown]}]
|
|
(let [own (partial feature/owned subject)
|
|
empty-points (vec (repeat (* 2 teeth-verts) 0))
|
|
values (mapv (fn [ring]
|
|
(if ring
|
|
(into [] (mapcat (juxt :x :y)) ring)
|
|
empty-points)) contours)
|
|
blk (block {:role "teeth" :analysis (:id analysis) :params params
|
|
:tracks ["contour"]}
|
|
{:type "int16" :scale geom-scale
|
|
:features [(own :teeth)] :absent? absent?
|
|
;; Not the feature's absence: a frame no contour could be
|
|
;; extracted from has no teeth to draw whether or not the
|
|
;; teeth were occluded, and the two reasons are different facts.
|
|
:missing (fn [_ f] (nil? (nth contours f)))}
|
|
obs [values])
|
|
generated {:by :pixels/teeth :analysis (:id analysis)
|
|
:params {:cavity-erode cavity-erode
|
|
:tongue-reject tongue-reject :blob-grow blob-grow
|
|
:top-bias top-bias :teeth-verts teeth-verts
|
|
:teeth-on teeth-on :teeth-smooth teeth-smooth}}]
|
|
{:nodes {:teeth
|
|
{:id :teeth :name "teeth" :kind :poly
|
|
:parent :mouth-in :z "a1"
|
|
:stencil :mouth-in
|
|
:channels {[:geom :pts] (dense blk 0 generated)
|
|
[:style :color] (ch/framed 4)
|
|
[:vis] (keyed-visibility shown generated)}}}
|
|
:store (stored blk)}))
|
|
|
|
(defn part
|
|
"One feature type: local nodes and blocks addressed by subject and feature."
|
|
[subject area params {:keys [detected presence] :as measured}]
|
|
(let [presence (into {} (map (fn [[role mask]] [(feature/owned subject role) mask])) presence)
|
|
absent? (when (or detected presence)
|
|
(fn [id f]
|
|
(or (and detected (not (nth detected f true)))
|
|
(and (contains? presence id)
|
|
(not (nth (get presence id) f))))))
|
|
obs {:detected detected :presence presence}]
|
|
(update (case area
|
|
:mouth (mouth-part subject absent? obs params measured)
|
|
:eye (feature-parts subject absent? obs params (select-keys measured [:eyes]))
|
|
:brow (feature-parts subject absent? obs params (select-keys measured [:brows]))
|
|
:teeth (interior-part subject params absent? obs (:teeth measured))
|
|
(throw (ex-info "unknown frozen feature type" {:area area})))
|
|
:nodes performance-nodes)))
|
|
|
|
(defn head-part
|
|
"Freeze one subject's measured head transform from its conditioned anchor.
|
|
|
|
THE THREE BLOCKS NAME THE SUBJECT, and that is not decoration. A block's key is
|
|
a hash over its descriptor, and the head follows DETECTION rather than any
|
|
feature's presence — so with `nil` in the feature slot, two faces tracked in one
|
|
analysis, both detected on every frame, produced byte-for-byte different
|
|
transforms under one identical key, and the second freeze's block silently
|
|
replaced the first's. The subject is the feature the head follows."
|
|
[subject {:keys [analysis anchor-avg] :as params} {:keys [transforms detected presence]}]
|
|
(let [absent? (when (or detected presence)
|
|
(fn [_ f] (and detected (not (nth detected f true)))))
|
|
inv (mapv invert transforms)
|
|
xf (fn [role f]
|
|
(block {:role role :analysis (:id analysis) :params params
|
|
:tracks [role]}
|
|
{:type "float32" :features [subject] :absent? absent?}
|
|
{:detected detected}
|
|
[(mapv f inv)]))
|
|
pos (xf "head-pos" (fn [t] [(:tx t) (:ty t)]))
|
|
rot (xf "head-rot" (fn [t] [(:theta t)]))
|
|
scale (xf "head-scale" (fn [t] [(:s t) (:s t)]))
|
|
prov {:by :anchor/similarity :analysis (:id analysis)
|
|
:params {:anchor-avg anchor-avg}}]
|
|
{:measured {[:xform :pos] (dense pos 0 prov)
|
|
[:xform :rot] (dense rot 0 prov)
|
|
[:xform :scale] (dense scale 0 prov)}
|
|
:store (stored pos rot scale)}))
|
|
|
|
;; ---------------------------------------------------------------------------
|
|
;; the clip
|
|
|
|
(defn- place-in
|
|
"Put the source-to-stage mapping on symbol `sym`'s own `:place`, above its head.
|
|
|
|
`:place` is the one authored node a freeze leaves on a face, and `face-placement`
|
|
says why it is a default rather than a measurement."
|
|
[sym channels]
|
|
(-> sym
|
|
(assoc-in [:nodes :place] {:id :place :name "source placement" :kind :group
|
|
:z "a1" :channels channels})
|
|
(assoc-in [:nodes :head :parent] :place)))
|
|
|
|
(defn- subject-part
|
|
"A subject's drawing, metadata and blocks. Node names are symbol-local."
|
|
[params subject {:keys [outer eyes brows teeth] :as inputs}]
|
|
(let [own (partial feature/owned subject)
|
|
areas (cond-> [:mouth] (and eyes brows) (into [:eye :brow]) teeth (conj :teeth))
|
|
parts (mapv #(part subject % params inputs) areas)
|
|
head (head-part subject params inputs)
|
|
features (cond-> {:mouth [:mouth [:mouth :mouth-in]]}
|
|
(and eyes brows)
|
|
(merge {:eye-r [:eye [:eye-r :eye-r-in :iris-r :pupil-r]]
|
|
:eye-l [:eye [:eye-l :eye-l-in :iris-l :pupil-l]]
|
|
:brow-r [:brow [:brow-r]] :brow-l [:brow [:brow-l]]})
|
|
teeth (assoc :teeth [:teeth [:teeth]]))]
|
|
{:symbol {:id subject :frames (count outer)
|
|
:nodes (into {:head {:id :head :name "head" :kind :group :z "a1"
|
|
:measured (:measured head)}}
|
|
(mapcat :nodes) parts)}
|
|
:features (into {} (map (fn [[role [area nodes]]]
|
|
[(own role) {:id (own role) :subject subject
|
|
:symbol subject :area area
|
|
:nodes nodes :params {}}])) features)
|
|
:groups (if (and eyes brows)
|
|
{(own :eyes) {:id (own :eyes) :kind :eye-pair :subject subject
|
|
:members [(own :eye-r) (own :eye-l)] :params {}}}
|
|
{})
|
|
:store (into (:store head) (mapcat :store) parts)}))
|
|
|
|
(defn pivoted
|
|
"Every node a freeze makes that a hand can transform, pivoting about the
|
|
middle of what it draws.
|
|
|
|
THE SAME RULE AS EVERYWHERE ELSE, and this is the one place that used to skip
|
|
it: `clip/place-symbol` writes an instance's anchor, `paint/new-shape` a
|
|
drawing's, `nest/group` a new symbol's, and `face-placement` the source
|
|
placement's — and the traced parts underneath it got none, so each of them
|
|
turned and scaled about ITS OWN ORIGIN, which for head-local geometry is the
|
|
top-left corner of the footage. `freeze_test` already said why that is wrong
|
|
for the face; it is no less wrong for the mouth.
|
|
|
|
WHAT IT SKIPS IS `node/measured?`, the predicate `gesture/refusal` refuses a
|
|
hand edit by — so a node gets a pivot exactly when a hand can use one, which
|
|
is the invariant worth having rather than a list of exceptions. It is also
|
|
what keeps this off `:head`: the head carries the measured similarity, its
|
|
scale is nowhere near 1, and an anchor under a scale does NOT cancel out of
|
|
`node/local!` the way it does at the identity, so writing one there would move
|
|
the whole face. Skipping it because it draws nothing would be true today and
|
|
true by accident.
|
|
|
|
A DEFAULT, written once, never followed: an anchor already on a node is left
|
|
alone, and nothing updates one when the geometry moves later. On everything it
|
|
does write to, rotation and scale are the identity, where the anchor cancels
|
|
out — so this changes where a part pivots and not one pixel of what it draws."
|
|
[clip store]
|
|
(reduce
|
|
(fn [c [sid id]]
|
|
(let [n (get-in c [:symbols sid :nodes id])
|
|
at [:symbols sid :nodes id :channels [:xform :anchor]]]
|
|
(if (or (get-in c at) (node/measured? n))
|
|
c
|
|
(if-let [p (pick/pivot c store sid n (range (get-in c [:symbols sid :frames])))]
|
|
(assoc-in c at (ch/framed p))
|
|
c))))
|
|
clip
|
|
(for [sid (sort-by str (keys (:symbols clip)))
|
|
id (sort-by str (keys (get-in clip [:symbols sid :nodes])))]
|
|
[sid id])))
|
|
|
|
(defn clip
|
|
"Subject-id -> conditioned measurements becomes one symbol per face, and a
|
|
symbol called :main that places them.
|
|
|
|
:main holds exposure and a shared source-to-stage placement. Each subject is
|
|
placed by an ordinary instance, so pose choices and transforms have their
|
|
existing instance scope. Features name local nodes in that subject's symbol;
|
|
block descriptors still name globally distinct features.
|
|
|
|
Subjects share a source frame space. :trace may be overridden per subject;
|
|
all other freeze settings come from params."
|
|
[{:keys [name fps stage expose] :as params} subjects]
|
|
(when-not (and (map? subjects) (seq subjects)
|
|
(every? keyword? (keys subjects))
|
|
(not-any? #{:main :root :place} (keys subjects)))
|
|
(throw (ex-info "a freeze needs subjects with ids distinct from :main, :root and :place" {})))
|
|
(let [ordered (sort-by (comp str key) subjects)
|
|
parts (mapv (fn [[id inputs]] [id (subject-part params id inputs)]) ordered)
|
|
placement (face-placement params subjects)
|
|
lengths (distinct (map #(get-in % [1 :symbol :frames]) parts))
|
|
_ (when-not (and (= 1 (count lengths)) (pos? (first lengths)))
|
|
(throw (ex-info "subjects need the same positive frame count"
|
|
{:frames (vec lengths)})))
|
|
nf (first lengths)
|
|
merged (fn [k] (into {} (mapcat (comp k second)) parts))
|
|
built {:name name :fps fps :analysis (:analysis params)
|
|
:width (first stage) :height (second stage)
|
|
:palettes {pal/default-id pal/default-palette}
|
|
:default-palette pal/default-id
|
|
:subjects (into {} (map (fn [[id _]] [id {:id id :params {}}])) ordered)
|
|
:features (merged :features) :groups (merged :groups)
|
|
:symbols
|
|
(into {:main
|
|
{:id :main :fps fps :frames nf
|
|
:nodes (into {:root {:id :root :name "clip" :kind :group :z "a1"
|
|
:time {:mode :map :expose expose}}}
|
|
(map-indexed
|
|
(fn [i [id _]]
|
|
[id {:id id :kind :instance :parent :root
|
|
:z (str "a" i)
|
|
:source {:symbol id}}]))
|
|
ordered)}}
|
|
(map (fn [[id part]]
|
|
[id (assoc (place-in (:symbol part) placement) :fps fps)]))
|
|
parts)}]
|
|
(doseq [[subject inputs] ordered
|
|
[id track] (:presence inputs)]
|
|
(when-not (and (= nf (count track))
|
|
(= subject (get-in built [:features (feature/owned subject id) :subject])))
|
|
(throw (ex-info "presence must name this subject's feature and span the take"
|
|
{:subject subject :feature id :frames nf :actual (count track)}))))
|
|
(let [store (merged :store)]
|
|
{:store store
|
|
:clip (-> (reduce (fn [c [subject inputs]]
|
|
(head-mode {:subject subject :trace (get inputs :trace (:trace params))}
|
|
{:clip c}))
|
|
built ordered)
|
|
(pivoted store))})))
|