Port step 5: freeze measured mouth into playable channels
This commit is contained in:
Olive Vaughn
parent
942e2f38ab
commit
8a06835895
24 changed files with 1625 additions and 506 deletions
|
|
@ -7,30 +7,50 @@
|
|||
into this map and every mounted layer-2 sub compares the result.
|
||||
|
||||
So the scene is here (it is a document — a human placed every node) and dense
|
||||
channel blocks are not; they live behind a handle in `store`. Today the demo
|
||||
scene has no dense blocks and the store is empty, which is why it is a map and
|
||||
not yet a namespace."
|
||||
channel blocks are not; they live behind a handle in `store`. The hand-written
|
||||
demo scene has no dense blocks and its store is empty; the swarm and the take
|
||||
are entirely dense."
|
||||
(:require [arthur.demo :as demo]
|
||||
[arthur.demo.swarm :as swarm]))
|
||||
[arthur.demo.swarm :as swarm]
|
||||
[arthur.demo.take :as take]))
|
||||
|
||||
(defn- clip
|
||||
"A scene plus the clip-level facts the transport and the stage need.
|
||||
|
||||
Read OFF the scene rather than written again beside it. `:fps`, `:frames` and the
|
||||
stage dimensions belong to the CLIP and not to the timeline — a timeline has a
|
||||
frame space, not a rate and not a size — and they sit on the scene map only
|
||||
because there is one clip per scene today. Copying them by hand into this table
|
||||
is how one of them comes to disagree with the scene it describes."
|
||||
[label scene store]
|
||||
(merge {:label label :scene scene :store store}
|
||||
(select-keys scene [:fps :frames :width :height])))
|
||||
|
||||
(def scenes
|
||||
"Two hand-made clips, selectable from the transport.
|
||||
"The hand-made clips, selectable from the transport.
|
||||
|
||||
`:swarm` holds its geometry in DENSE blocks behind store handles, which is the
|
||||
shape freeze produces at step 5 — so the fun one is also the load test."
|
||||
{:demo {:label "demo" :scene demo/scene :store nil
|
||||
:fps demo/fps :frames demo/frames}
|
||||
:swarm {:label "swarm" :scene @swarm/scene :store @swarm/store
|
||||
:fps swarm/fps :frames swarm/frames}})
|
||||
`:swarm` is the load test: a hundred and twenty nodes, entirely dense. The two
|
||||
takes are step 5's deliverable and they are ONE freeze — the same blocks, with
|
||||
`:head` written as a dense track in one and as framed identity in the other, so
|
||||
the button that switches between them switches a document field and nothing
|
||||
else."
|
||||
{:demo (clip "demo" demo/scene nil)
|
||||
:swarm (clip "swarm" @swarm/scene @swarm/store)
|
||||
:take (clip "take" @take/scene @take/store)
|
||||
:take-locked (clip "locked" @take/locked @take/store)})
|
||||
|
||||
(def default
|
||||
{;; --- the document ---
|
||||
:scene/current :demo
|
||||
:scene/current :take
|
||||
:palette :arthur/default ; a NAME; the ramp itself is project data
|
||||
|
||||
;; --- the clip ---
|
||||
:clip {:fps demo/fps
|
||||
:frames demo/frames}
|
||||
;;
|
||||
;; Including the STAGE DIMENSIONS, which are the project's and not the
|
||||
;; footage's. That is what deleting `makeXform` buys — the framing became a
|
||||
;; transform on a node, so nothing downstream of the freeze knows the frame
|
||||
;; size — and it is why ui/player no longer hardcodes 320x200.
|
||||
:clip (select-keys (:take scenes) [:fps :frames :width :height])
|
||||
|
||||
;; --- transport ---
|
||||
;;
|
||||
|
|
|
|||
|
|
@ -13,9 +13,6 @@
|
|||
|
||||
(def scene (reader/read-string source))
|
||||
|
||||
(def width 320)
|
||||
(def height 200)
|
||||
|
||||
(def fps (:fps scene))
|
||||
(def frames (:frames scene))
|
||||
|
||||
|
|
|
|||
|
|
@ -26,6 +26,12 @@
|
|||
;; frame space, not a rate — and it is here only because there is one clip.
|
||||
:frames 229
|
||||
:fps 30
|
||||
;; The STAGE, in pixels. The project's dimensions, not the footage's — which is
|
||||
;; what makes `makeXform` deletable: placement is a transform on a node and the
|
||||
;; stage clips whatever hangs off. Here everything is authored in stage pixels
|
||||
;; already, because a hand-written scene is a painted one.
|
||||
:width 320
|
||||
:height 200
|
||||
|
||||
:nodes
|
||||
{;; The clip root. EXPOSURE LIVES HERE and is inherited, because
|
||||
|
|
|
|||
|
|
@ -152,6 +152,8 @@
|
|||
{:name "swarm"
|
||||
:frames frames
|
||||
:fps fps
|
||||
:width 320
|
||||
:height 200
|
||||
:nodes
|
||||
(into {:root {:id :root :kind :group :parent nil :z "a1"
|
||||
;; On 2s, like everything else. A hundred and twenty shapes
|
||||
|
|
|
|||
112
frontend/src/arthur/demo/take.cljs
Normal file
112
frontend/src/arthur/demo/take.cljs
Normal file
|
|
@ -0,0 +1,112 @@
|
|||
(ns arthur.demo.take
|
||||
"The synthetic take: the whole vertical slice, with no video file in it.
|
||||
|
||||
This is port-plan step 5's deliverable and it is the first thing in the tree
|
||||
that runs every stage in order —
|
||||
|
||||
synth ──▶ measure/anchor ──▶ condition/anchor
|
||||
│ │
|
||||
└──▶ measure/mouth ◀─┘
|
||||
│
|
||||
condition/contours
|
||||
│
|
||||
FREEZE ──▶ channels on nodes
|
||||
│
|
||||
scene/resolver ──▶ raster
|
||||
|
||||
— and the order of that diagram is the whole argument for the stage split. The
|
||||
anchor fit is knob-free. Conditioning smooths its four parameters. The rings are
|
||||
then measured THROUGH the conditioned transform, so `anchor avg` does re-run the
|
||||
ring mapping — a few hundred frames of twenty points, free — and does not re-run
|
||||
anything that reads a source pixel, because that part takes the landmarks and
|
||||
the frames and never the transform.
|
||||
|
||||
TWO SCENES, ONE STORE. `:take` carries the head as filmed and `:take-locked`
|
||||
carries it locked, and they are the same dense blocks with one node's channels
|
||||
written two ways. That is the claim \"stabilisation is a channel, not a mode\"
|
||||
made checkable by eye: switching between them is a document edit, tier 1, and
|
||||
not one byte of tier 2 differs."
|
||||
(:require [arthur.flow.condition :as condition]
|
||||
[arthur.flow.freeze :as freeze]
|
||||
[arthur.flow.measure.anchor :as anchor]
|
||||
[arthur.flow.measure.mouth :as mouth]
|
||||
[arthur.synth :as synth]))
|
||||
|
||||
(def frames 229)
|
||||
(def fps 30)
|
||||
|
||||
(def ^:private stage
|
||||
;; The project's dimensions, and NOT the footage's. This is what deleting
|
||||
;; `makeXform` buys: the head is placed and scaled on the stage by a transform
|
||||
;; on a node, so a 1440x1920 portrait clip and a 320x200 stage are not a
|
||||
;; conflict to resolve. Whatever hangs off the edge is clipped.
|
||||
[320 200])
|
||||
|
||||
(def ^:private aspect
|
||||
;; The synth writes x and y in the SAME unit, so its normalised space is already
|
||||
;; isotropic and the anisotropy correction is the identity here. Real footage
|
||||
;; passes W/H from the manifest at step 6. Worth knowing while reading anything
|
||||
;; here: aspect 1 is the one setting at which a port that dropped the
|
||||
;; anisotropy correction entirely would still look right, which is why
|
||||
;; anchor-test exercises 0.5625 and this does not.
|
||||
1)
|
||||
|
||||
(def ^:private knobs
|
||||
"The prototype's own defaults, from index.html, so the first thing anyone sees
|
||||
is the thing it was tuned to look like."
|
||||
{:anchor-avg 2 ; smoothWin
|
||||
:contour-avg 1 ; contourSmooth
|
||||
:verts 8 ; vertices
|
||||
:aperture-cut 0.12}) ; apertureThresh 120/1000
|
||||
|
||||
(def analysis
|
||||
"Stage 2's output, synthesised. A seeded generator, so a wrong pose is
|
||||
reproducible rather than something that happened once."
|
||||
(delay (synth/synth-dense frames {:seed 1})))
|
||||
|
||||
(def measured
|
||||
"Stages 3 and 4, in the order the stage split requires."
|
||||
(delay
|
||||
(let [dense @analysis
|
||||
fitted (anchor/fit {:aspect aspect} {:dense dense})
|
||||
anchored (condition/anchor knobs fitted)
|
||||
rings (mouth/measure {:aspect aspect}
|
||||
{:dense dense :transforms (:transforms anchored)})]
|
||||
(assoc anchored
|
||||
:outer (condition/contours knobs (:outer rings))
|
||||
:inner (condition/contours knobs (:inner rings))
|
||||
;; NOT smoothed. The aperture is the inner ring's own height read off
|
||||
;; as a scalar, and `contour avg` is a spatial-correspondence
|
||||
;; smoother over a ring; running it over a scalar would be a second,
|
||||
;; undocumented low-pass on a signal that then decides visibility.
|
||||
:aperture (:aperture rings)))))
|
||||
|
||||
(def params
|
||||
"What the freeze was handed. Public because it is the honest way to re-freeze at
|
||||
other settings — a test that built its own copy would be asserting about a clip
|
||||
nobody looks at."
|
||||
(merge knobs
|
||||
{:name "take"
|
||||
:fps fps
|
||||
:stage stage
|
||||
;; On 2s. docs/design.md is emphatic that everything rides ONE grid: a
|
||||
;; head cutting on odd frames against a mouth cutting on even ones reads
|
||||
;; as two performances, so exposure lives on the clip root and inherits.
|
||||
:expose 2
|
||||
;; The head as filmed. `:take-locked` is the same freeze with this one
|
||||
;; field changed, which is the point.
|
||||
:head :as-filmed
|
||||
;; Provenance. A content hash once the analysis is an artifact the
|
||||
;; backend stores; until then, honest about what it actually is.
|
||||
:analysis "synth:mulberry32/seed-1"}))
|
||||
|
||||
(def frozen
|
||||
(delay (freeze/clip params @measured)))
|
||||
|
||||
(def store (delay (:store @frozen)))
|
||||
|
||||
(def scene (delay (:scene @frozen)))
|
||||
|
||||
(def locked
|
||||
"The same blocks, with `:head` written as framed identity instead."
|
||||
(delay (freeze/head-mode {:mode :locked} @frozen)))
|
||||
|
|
@ -109,11 +109,18 @@
|
|||
;; ---------------------------------------------------------------------------
|
||||
;; dense blocks
|
||||
|
||||
(defn- dense-state
|
||||
[state f]
|
||||
(if (nil? state)
|
||||
present
|
||||
(aget state f)))
|
||||
(defn- absent-at?
|
||||
"Is the subject absent on frame f of this block's slice?
|
||||
|
||||
INDEXED THE WAY THE DATA IS. A block is node-major — offset(node i) =
|
||||
i·frames·stride — so a block holding several nodes holds several mask regions,
|
||||
and the one belonging to this channel starts at offset/stride. Indexing the
|
||||
mask by f alone reads the FIRST node's absence for every node in the block,
|
||||
which is not a subtly wrong pose: it is every part in the block vanishing on
|
||||
the frames where one of them was occluded."
|
||||
[state offset stride f]
|
||||
(and (some? state)
|
||||
(pos? (bit-and (aget state (+ (quot offset stride) f)) absent-bit))))
|
||||
|
||||
(defn dense-at
|
||||
"Read frame f out of a dense block.
|
||||
|
|
@ -130,21 +137,37 @@
|
|||
|
||||
stride 1 yields a number; anything wider yields a SUBARRAY VIEW over the
|
||||
block, not a copy. Fixed topology is what makes that possible — the frame's
|
||||
data is a rectangular slice at a known offset with no per-frame header."
|
||||
[{:keys [store offset stride] nf :frames} f st]
|
||||
(let [{:keys [data state]} (get st store)]
|
||||
(when (nil? data)
|
||||
(throw (ex-info "dense channel's store key is not in the store"
|
||||
{:store store :have (vec (sort (map str (keys st))))})))
|
||||
(let [f (-> f (max 0) (min (dec nf)))
|
||||
sm (dense-state state f)]
|
||||
(cond
|
||||
(pos? (bit-and sm absent-bit)) absent
|
||||
:else
|
||||
(let [o (+ offset (* f stride))]
|
||||
(if (= 1 stride)
|
||||
(aget data o)
|
||||
(.subarray data o (+ o stride))))))))
|
||||
data is a rectangular slice at a known offset with no per-frame header.
|
||||
|
||||
FIXED POINT. `:scale` in the block header means the stored integers are the
|
||||
value times that scale, so a block of geometry in image-height units fills an
|
||||
Int16 usefully and a block of stage pixels — which wants a different scale
|
||||
entirely — fills one too. It is in the header rather than agreed by convention
|
||||
for exactly that reason, and it is why the block in memory is byte for byte the
|
||||
block on the wire: a handle that names a sha256 has to name the bytes you
|
||||
actually hold.
|
||||
|
||||
Decoding costs the view. `out` is a stride-sized destination the caller owns —
|
||||
`cursor` allocates one per channel — because a copy per node per frame is the
|
||||
allocation this whole model is arranged to avoid; passing nil allocates, which
|
||||
is what `value-at`, the specification, does."
|
||||
([blk f st] (dense-at blk f st nil))
|
||||
([{:keys [store offset stride scale] nf :frames} f st out]
|
||||
(let [{:keys [data state]} (get st store)]
|
||||
(when (nil? data)
|
||||
(throw (ex-info "dense channel's store key is not in the store"
|
||||
{:store store :have (vec (sort (map str (keys st))))})))
|
||||
(let [f (-> f (max 0) (min (dec nf)))]
|
||||
(if (absent-at? state offset stride f)
|
||||
absent
|
||||
(let [o (+ offset (* f stride))]
|
||||
(cond
|
||||
(= 1 stride) (let [v (aget data o)] (if scale (/ v scale) v))
|
||||
(nil? scale) (.subarray data o (+ o stride))
|
||||
:else (let [dst (or out (js/Float64Array. stride))]
|
||||
(dotimes [k stride]
|
||||
(aset dst k (/ (aget data (+ o k)) scale)))
|
||||
dst))))))))
|
||||
|
||||
;; ---------------------------------------------------------------------------
|
||||
;; the specification
|
||||
|
|
@ -197,20 +220,28 @@
|
|||
(recur (inc mid) hi mid)
|
||||
(recur lo (dec mid) best))))))
|
||||
|
||||
(deftype Cursor [ch ks store ^:mutable i]
|
||||
(deftype Cursor [ch ks store buf ^:mutable i]
|
||||
Object
|
||||
(toString [_] (str "#Cursor{" (pr-str (if ks :keyed (if (:dense ch) :dense :framed))) " i=" i "}")))
|
||||
|
||||
(defn cursor
|
||||
"A reading head on one channel. Build once per channel per resolver, then
|
||||
`sample!` it per frame. Holds the sorted key index, which is why the index is
|
||||
built here and not in the document."
|
||||
built here and not in the document — and the decode buffer a fixed-point block
|
||||
needs, for the same reason the resolver owns one point buffer per node.
|
||||
|
||||
Only a wide fixed-point block gets a buffer: a stride-1 block decodes to a
|
||||
number and a block with no `:scale` is handed back as a view."
|
||||
([ch] (cursor ch nil))
|
||||
([ch store]
|
||||
(check-unimplemented! ch)
|
||||
(->Cursor ch (when (and (:animated? ch) (not (:dense ch)) (seq (:keys ch)))
|
||||
(frames ch))
|
||||
store 0)))
|
||||
(let [d (:dense ch)]
|
||||
(->Cursor ch
|
||||
(when (and (:animated? ch) (not d) (seq (:keys ch))) (frames ch))
|
||||
store
|
||||
(when (and d (:scale d) (> (:stride d) 1))
|
||||
(js/Float64Array. (:stride d)))
|
||||
0))))
|
||||
|
||||
(defn sample!
|
||||
"Value of the cursor's channel at f. O(1) when f is at or one key past where
|
||||
|
|
@ -222,7 +253,7 @@
|
|||
ks (.-ks cur)]
|
||||
(cond
|
||||
(not (:animated? ch)) (:value ch)
|
||||
(:dense ch) (dense-at (:dense ch) f (.-store cur))
|
||||
(:dense ch) (dense-at (:dense ch) f (.-store cur) (.-buf cur))
|
||||
(nil? ks) absent ; animated with an empty key map
|
||||
:else
|
||||
(let [n (count ks)
|
||||
|
|
@ -281,5 +312,14 @@
|
|||
" requires hold of every cut part and tweening reads as puppet software"))
|
||||
|
||||
(and (map? ch) (seq (:over ch)))
|
||||
(conj ":over layers are not implemented (port-plan step 2 scope)")))
|
||||
(conj ":over layers are not implemented (port-plan step 2 scope)")
|
||||
|
||||
;; A scale of zero divides every value in the block by zero, and a negative
|
||||
;; one mirrors the geometry. Both are authored-data bugs that present as a
|
||||
;; part drawn nowhere or inside out, not as an error.
|
||||
(and (map? ch) (:dense ch) (contains? (:dense ch) :scale)
|
||||
(not (and (number? (:scale (:dense ch))) (pos? (:scale (:dense ch))))))
|
||||
(conj (str ":dense :scale is " (pr-str (:scale (:dense ch)))
|
||||
" — a fixed-point scale is a positive number the stored integers"
|
||||
" were multiplied by"))))
|
||||
|
||||
|
|
|
|||
|
|
@ -190,6 +190,30 @@
|
|||
(ch/nothing? skw) (ch/nothing? anc))
|
||||
[pos rot scl skw anc])))
|
||||
|
||||
(defn- visible?
|
||||
"Is the node switched on this frame?
|
||||
|
||||
`[:vis]` IS A BOOLEAN, and this insists on it rather than testing truthiness,
|
||||
because the two obvious implementations are both wrong about a DENSE `[:vis]`.
|
||||
A dense block yields 0 or 1, and 0 is TRUTHY in CLJS — so `(if v …)` shows a
|
||||
hidden frame, and `(true? v)` hides every frame. Neither reads as an error.
|
||||
|
||||
docs/animation-model.md's parts table says `:mouth-in` carries `[:vis]` dense;
|
||||
flow/freeze writes it KEYED, because a threshold crossing is a handful of
|
||||
transitions and hold is the default, and because a human has to be able to fix
|
||||
one frame of it. When something does want a dense one it will land here loudly
|
||||
instead of blanking the scene.
|
||||
|
||||
Absence is not a boolean and is not an error: a subject that is not on the
|
||||
frame has nothing to show."
|
||||
[id v]
|
||||
(cond
|
||||
(true? v) true
|
||||
(false? v) false
|
||||
(ch/nothing? v) false
|
||||
:else (throw (ex-info "[:vis] must sample to a boolean"
|
||||
{:node id :value v}))))
|
||||
|
||||
(defn- place
|
||||
"Where a node sits this frame, as {:m world :f local-frame :rd reader}, or nil
|
||||
when it is not on the frame at all.
|
||||
|
|
@ -210,7 +234,7 @@
|
|||
chs (node/channels n)
|
||||
lf (node/local-frame n pf)
|
||||
rd (fn [path] (read id path (get chs path) lf))]
|
||||
(when (true? (rd [:vis]))
|
||||
(when (visible? id (rd [:vis]))
|
||||
(when-let [[pos rot scl skw anc] (xform-at rd)]
|
||||
;; dest aliases `local` here, which mul! allows: it reads both
|
||||
;; operands fully before writing either.
|
||||
|
|
|
|||
|
|
@ -92,9 +92,12 @@
|
|||
;; Changing the clip changes the resolver, the frame count and the rate all
|
||||
;; at once, so the playhead goes home rather than being left pointing at a
|
||||
;; frame the new clip may not have.
|
||||
(let [{:keys [fps frames]} (get db/scenes id)]
|
||||
(let [{:keys [fps frames] :as clip} (get db/scenes id)]
|
||||
{:db (-> db
|
||||
(assoc :scene/current id)
|
||||
(assoc :clip {:fps fps :frames frames})
|
||||
;; The stage travels with the clip: two clips may be different
|
||||
;; sizes, and the raster the loop paints into is the clip's, not
|
||||
;; the app's.
|
||||
(assoc :clip (select-keys clip [:fps :frames :width :height]))
|
||||
(assoc-in [:playback :frame] 0))
|
||||
::seek! [fps frames 0]})))
|
||||
|
|
|
|||
440
frontend/src/arthur/flow/freeze.cljs
Normal file
440
frontend/src/arthur/flow/freeze.cljs
Normal file
|
|
@ -0,0 +1,440 @@
|
|||
(ns arthur.flow.freeze
|
||||
"Stage 5, the freeze: measurements become CHANNELS.
|
||||
|
||||
This is the hinge the whole model turns on. Freezing is NOT a conversion into a
|
||||
second format — there is one format, and freezing fills it in. That is what
|
||||
makes \"the only difference between rotoscoped and hand-authored is a flag\"
|
||||
literally true: what comes out of here is the same `:channels` map a hand fills
|
||||
in sparsely, and the flag is `:generated`, which nothing in the renderer reads.
|
||||
|
||||
Three conversions happen here and nowhere else.
|
||||
|
||||
MAPS BECOME FLAT. `flow/measure/*` speaks {:x :y}, because it is the numeric
|
||||
oracle and a faithful port was worth more there than a fast one. A channel
|
||||
value is FLAT — [x0 y0 x1 y1 …] — in authored vectors and dense blocks alike.
|
||||
`rings->flat` is the only place that crossing is made.
|
||||
|
||||
FLOATS BECOME FIXED POINT. Geometry goes into an Int16 block with the scale in
|
||||
its header; see `geom-scale`.
|
||||
|
||||
A SIMILARITY BECOMES THREE CHANNELS. `{s θ tx ty}` IS `[:xform :scale]`,
|
||||
`[:xform :rot]` and `[:xform :pos]`, so the anchor drops onto `:head` with no
|
||||
adapter — which is the sign the decomposition is the right one.
|
||||
|
||||
`makeXform` IS NOT HERE AND IS NOT COMING. The prototype centres on the face
|
||||
oval's bbox and zooms until the face is 80% of the raster height, so every
|
||||
stored vertex carries a cropping decision made once, at analysis time, from one
|
||||
frame's landmarks. Here the geometry stays in the node's own local space and the
|
||||
framing is `[:xform :*]` on an authored `:face` node, which the stage clips.
|
||||
Project dimensions are therefore independent of the footage — see
|
||||
`face-placement`, and \"What space geometry is in\" in docs/animation-model.md.
|
||||
|
||||
Not `flow/key`. A traced mouth costs nothing, so it gets a key on EVERY frame;
|
||||
only a plate, which a human draws, is worth decimating. The one stage-5 policy
|
||||
the mouth does want is the aperture threshold, and that is `visibility` below."
|
||||
(:require [arthur.domain.channel :as ch]
|
||||
[arthur.domain.geom :as geom]
|
||||
[arthur.domain.ring :as ring]))
|
||||
|
||||
;; ---------------------------------------------------------------------------
|
||||
;; fixed point
|
||||
|
||||
(def ^:const geom-scale
|
||||
"Q14: the stored integer is the value times 16384.
|
||||
|
||||
Geometry is head-local and its unit is ONE IMAGE HEIGHT, so 16384 covers ±2
|
||||
image heights in an Int16 and quantises to 1/16384 = 6.1e-5 of an image height.
|
||||
Against a face placed so that its 0.29-image-height oval fills most of a 200px
|
||||
stage — around 850 stage pixels per image height — that is 0.05px, two orders
|
||||
below anything the rasteriser can express.
|
||||
|
||||
A power of two, so the decode is a floating-point exact division and freezing
|
||||
the same numbers twice cannot drift.
|
||||
|
||||
It is a CONSTANT here and a FIELD in the block header, and the difference
|
||||
matters: a painted cel's geometry is in stage pixels, where ±2 would be absurd
|
||||
and 1/16384 of a pixel is waste. Each block says what its own space needs."
|
||||
16384)
|
||||
|
||||
(def ^:private ^:const int16-max 32767)
|
||||
|
||||
;; ---------------------------------------------------------------------------
|
||||
;; blocks
|
||||
|
||||
(defn- pack
|
||||
"Tracks -> one dense block, NODE-MAJOR and FRAME-MINOR:
|
||||
|
||||
offset(track i) = i · frames · stride
|
||||
value(i, f) = data[offset(i) + f · stride]
|
||||
|
||||
No per-frame header and no indirection, which FIXED TOPOLOGY is what buys: every
|
||||
frame of a part carries the same component count with the same meanings, so a
|
||||
frame is a rectangular slice at an arithmetic offset. A variable vertex count
|
||||
would force an offset table and a scan per frame, so the aesthetic constraint is
|
||||
a performance asset rather than a cost. `demo/swarm` holds the same layout and
|
||||
is the load test for it.
|
||||
|
||||
`:ctor` makes the array — a function and not a type, because `new` is not
|
||||
something a value can carry — `:scale` is the fixed-point scale or nil, and
|
||||
`:absent` an optional per-frame predicate. Absence is the SUBJECT's, not the
|
||||
part's, so it is written into every track of the block: the mouth outline and
|
||||
the mouth interior are one face, and a frame that face was not on has no pose
|
||||
for either.
|
||||
|
||||
Written with `dotimes` and `aset` rather than as a fold, and that is the
|
||||
exception rather than the rule in this codebase: the destination is a typed
|
||||
array, so there is nothing to accumulate into and a collection idiom here would
|
||||
allocate a seq per frame to throw away."
|
||||
[{:keys [ctor scale absent]} tracks]
|
||||
(let [n (count tracks)
|
||||
nf (count (first tracks))
|
||||
stride (count (first (first tracks)))
|
||||
data (ctor (* n nf stride))
|
||||
state (when absent (js/Uint8Array. (* n nf)))]
|
||||
(dotimes [i n]
|
||||
(let [track (vec (nth tracks i))
|
||||
base (* i nf stride)]
|
||||
(dotimes [f nf]
|
||||
(let [vs (nth track f)
|
||||
o (+ base (* f stride))]
|
||||
(dotimes [k stride]
|
||||
(let [v (nth vs k)]
|
||||
(aset data (+ o k)
|
||||
(if scale
|
||||
(let [q (js/Math.round (* v scale))]
|
||||
;; Saturating would read as articulation flattening off
|
||||
;; at the extremes — a bad detection, not a bad scale.
|
||||
(when (> (abs q) int16-max)
|
||||
(throw (ex-info "value does not fit the block's fixed point"
|
||||
{:value v :scale scale :quantised q
|
||||
:track i :frame f :component k})))
|
||||
q)
|
||||
v))))
|
||||
(when (and state (absent f))
|
||||
(aset state (+ (* i nf) f) ch/absent-bit))))))
|
||||
{:data data :state state :stride stride :frames nf :scale scale
|
||||
:offsets (mapv #(* % nf stride) (range n))}))
|
||||
|
||||
(defn- dense
|
||||
"Track i of a packed block, as a DENSE channel definition."
|
||||
[store-key blk i generated]
|
||||
{:animated? true :interp :hold
|
||||
:dense (cond-> {:store store-key
|
||||
:offset (nth (:offsets blk) i)
|
||||
:stride (:stride blk)
|
||||
:frames (:frames blk)}
|
||||
(:scale blk) (assoc :scale (:scale blk)))
|
||||
:generated generated
|
||||
:over []})
|
||||
|
||||
;; ---------------------------------------------------------------------------
|
||||
;; geometry
|
||||
|
||||
(defn rings->flat
|
||||
"Ring track -> one flat [x0 y0 x1 y1 …] per frame, at the vertex budget.
|
||||
|
||||
The vertex budget is a fixed-index SUBSAMPLE, never adaptive decimation: slot k
|
||||
means the same anatomy on every frame of the shot, and that is what makes
|
||||
temporal correspondence possible at all. It is applied here, after stage 4's
|
||||
contour average, and the two commute because both are per-slot — which is the
|
||||
whole reason the vertex knob can sit downstream of the smoothing knob instead
|
||||
of alongside it. `mouth-test` asserts that rather than leaving it to look
|
||||
obvious.
|
||||
|
||||
An ODD budget is refused. It lands off the cardinal slots — the corners and the
|
||||
lip centres — and a wrongly-ordered ring self-intersects INVISIBLY at odd vertex
|
||||
counts and obviously at even ones, so an odd budget is the one setting at which
|
||||
the simplicity assertion stops protecting anything."
|
||||
[rings verts]
|
||||
(let [len (count (first rings))]
|
||||
(when-not (and (integer? verts) (even? verts) (>= verts 4) (<= verts len))
|
||||
(throw (ex-info "vertex budget must be even and between 4 and the ring's slot count"
|
||||
{:verts verts :slots len})))
|
||||
(let [slots (ring/subsample-slots len verts)]
|
||||
(mapv (fn [r]
|
||||
(into [] (mapcat (fn [s] (let [p (nth r s)] [(:x p) (:y p)]))) slots))
|
||||
rings))))
|
||||
|
||||
;; ---------------------------------------------------------------------------
|
||||
;; the anchor, onto :head
|
||||
|
||||
(defn invert
|
||||
"The inverse of a similarity, STILL FACTORED.
|
||||
|
||||
The anchor fit maps each frame onto the shot's mean pose, so it is what takes
|
||||
the head's motion OUT; geometry is stored in the space it produces. Putting the
|
||||
head's motion back — the \"as filmed\" mode — is therefore the fit's inverse, and
|
||||
it has to stay a {s θ t} rather than becoming a matrix, because the three
|
||||
components land on three independently keyframable channels and interpolating
|
||||
matrix entries is meaningless.
|
||||
|
||||
p ↦ s·R(θ)·p + t inverts to q ↦ (1/s)·R(-θ)·(q - t)"
|
||||
[{:keys [s theta tx ty]}]
|
||||
(let [s' (/ 1.0 s)
|
||||
c (js/Math.cos theta)
|
||||
sn (js/Math.sin theta)]
|
||||
{:s s'
|
||||
:theta (- theta)
|
||||
:tx (* (- s') (+ (* c tx) (* sn ty)))
|
||||
:ty (* (- s') (+ (* (- sn) tx) (* c ty)))}))
|
||||
|
||||
(def ^:private head-modes #{:locked :as-filmed :per-plate})
|
||||
|
||||
(defn head-mode
|
||||
"Rewrite `:head`'s transform channels into one of the three shapes.
|
||||
|
||||
STABILISATION IS A CHANNEL, NOT A MODE. `{s θ tx ty}` per frame IS
|
||||
`[:xform :scale]`, `[:xform :rot]` and `[:xform :pos]`, so removing the head's
|
||||
motion is not a pipeline setting — it is a question of which shape one node's
|
||||
channels carry:
|
||||
|
||||
:locked framed identity — the head sits still, for tracing and for
|
||||
judging articulation
|
||||
:as-filmed dense — the head moves around the stage
|
||||
:per-plate keyed at the kept frames — the head pose is stable for exactly as
|
||||
long as a drawing is on screen, which is what a plate strip wants
|
||||
|
||||
ALWAYS MEASURE, ALWAYS STORE FACTORED, whatever the toggle says. The blocks are
|
||||
written once and every mode reads them or ignores them; making this an
|
||||
analysis-time switch would lose two things downstream. \"Smooth the transform,
|
||||
never the contour\" only means anything while the two are separate. And a
|
||||
velocity minimum is \"articulation paused\" in head-local space and \"the head
|
||||
happened to be still\" in image space, so key selection needs the split to exist
|
||||
in storage.
|
||||
|
||||
So this is a DOCUMENT EDIT: tier 1, undoable, syncable, instant, and not a
|
||||
reason to re-analyse. It rewrites `:head` and touches nothing else — in
|
||||
particular not `:face`, which a hand placed, and not one byte of any block.
|
||||
|
||||
`:per-plate` samples the dense track at the kept frames and stores plain
|
||||
vectors. It may not store what `value-at` handed it: a wide dense read is a
|
||||
VIEW into the block, and a view into tier 2 sitting in the document is a value
|
||||
that changes when a re-freeze rewrites the array under it."
|
||||
[{:keys [mode kept]} {:keys [scene store]}]
|
||||
(when-not (contains? head-modes mode)
|
||||
(throw (ex-info "head mode is not one of the three channel shapes"
|
||||
{:mode mode :modes head-modes})))
|
||||
(let [base (get-in scene [:nodes :head :measured])
|
||||
at (fn [path f]
|
||||
(let [v (ch/value-at (get base path) f store)
|
||||
n (:stride (:dense (get base path)))]
|
||||
(if (= 1 n) v (mapv #(ch/component v %) (range n)))))
|
||||
keys-of (fn [path]
|
||||
(-> (get base path)
|
||||
(dissoc :dense)
|
||||
(assoc :keys (into {} (map (juxt identity #(at path %))) (sort kept)))))]
|
||||
(when (and (= mode :per-plate) (empty? kept))
|
||||
(throw (ex-info "the per-plate head mode needs a kept-frame set; it is the plate strip's, not measurement's"
|
||||
{:mode mode})))
|
||||
(assoc-in scene [:nodes :head :channels]
|
||||
(case mode
|
||||
;; No `:generated` on the locked shape, and that is not an
|
||||
;; oversight: nothing generated this identity. It is a decision,
|
||||
;; and provenance that claimed otherwise would offer a re-freeze
|
||||
;; button that could only undo it.
|
||||
:locked {[:xform :pos] (ch/framed [0.0 0.0])
|
||||
[:xform :rot] (ch/framed 0.0)
|
||||
[:xform :scale] (ch/framed [1.0 1.0])}
|
||||
:as-filmed base
|
||||
:per-plate (into {} (map (juxt key #(keys-of (key %)))) base)))))
|
||||
|
||||
;; ---------------------------------------------------------------------------
|
||||
;; the aperture, onto [:vis] of :mouth-in
|
||||
|
||||
(defn visibility
|
||||
"The aperture track -> `[:vis]` keys on the mouth interior.
|
||||
|
||||
`flow/measure/mouth` reports the aperture and deliberately does not threshold
|
||||
it: the measurement is the inner ring's own height and the threshold is a
|
||||
policy, which is stage 5. Relative to the take's PEAK aperture, not absolute, so
|
||||
one number works across faces and framings — the prototype's
|
||||
`ap.map(v => v / apMax < apertureThresh)`, which lives in its `app.js` and not
|
||||
in its `pipeline.js`, and is easy to miss when porting from the latter.
|
||||
|
||||
KEYED, not dense, although docs/animation-model.md's parts table says dense. A
|
||||
threshold crossing is a handful of transitions over a take, hold is the default,
|
||||
and keys are what a human can correct — \"this frame's mouth should be shut\" is
|
||||
the single most likely hand edit on a lip-sync take, and a dense block in tier 2
|
||||
is the one shape that cannot receive it. `:generated` still rides along, which is
|
||||
the point of provenance being on the channel rather than implied by its shape.
|
||||
|
||||
A key on frame 0 always, because the first key is the pose the part starts in."
|
||||
[{:keys [aperture-cut]} {:keys [aperture]} generated]
|
||||
(let [peak (reduce max aperture)
|
||||
;; A take with no mouth at all has no peak to be a fraction of. Present
|
||||
;; rather than absent: an all-zero aperture is a shut mouth, and the
|
||||
;; interior of a shut mouth is simply not drawn.
|
||||
shown (mapv (fn [v] (and (pos? peak) (>= (/ v peak) aperture-cut))) aperture)]
|
||||
(assoc (ch/keyed (into {} (keep (fn [f]
|
||||
(when (or (zero? f)
|
||||
(not= (nth shown f) (nth shown (dec f))))
|
||||
[f (nth shown f)])))
|
||||
(range (count shown))))
|
||||
:generated generated)))
|
||||
|
||||
;; ---------------------------------------------------------------------------
|
||||
;; the face, onto the stage
|
||||
|
||||
(defn face-placement
|
||||
"The face's transform on the stage, as FRAMED channels.
|
||||
|
||||
AUTHORED, and that is the whole difference from `makeXform`. What comes back is
|
||||
a DEFAULT — the placement a human would otherwise have to make from scratch on
|
||||
first open — and from then on it is an ordinary hand-placed transform on an
|
||||
ordinary node. `makeXform` made the same decision and then baked it into every
|
||||
vertex, where nothing could ever revise it.
|
||||
|
||||
Derived from the reference rigid configuration, which is what the freeze has:
|
||||
the face oval is not measured, because its only consumers in the prototype were
|
||||
that transform and the placeholder plate outline. Two numbers:
|
||||
|
||||
SCALE is stage pixels per image height, set so the reference's eye-corner span
|
||||
is 40% of the stage width. Landmark-free — it is the rigid configuration's own
|
||||
bounding box — and it is a fraction of the STAGE, so a 1440x1920 portrait clip
|
||||
composited onto a 320x200 stage is not a problem to solve.
|
||||
|
||||
ANCHOR is the reference centroid, and this is where `:anchor` earns its place.
|
||||
MediaPipe's normalised space has its origin at the image's TOP-LEFT CORNER, so
|
||||
head-local geometry is not centred on anything; the registration point of the
|
||||
face is the head's own centre, and rotation and scale have to happen about that
|
||||
rather than about a corner of the footage. Getting that wrong is why hand-placed
|
||||
parts swing rather than turn.
|
||||
|
||||
POSITION puts the anchor a QUARTER of the way down the stage, because the rigid
|
||||
landmarks are eyes and nose — the upper middle of a face — so a quarter down
|
||||
leaves the jaw and the mouth on the stage. Whatever hangs off is clipped, which
|
||||
is not a feature to add: every fill in `domain/raster` clamps already."
|
||||
[{:keys [stage]} {:keys [ref]}]
|
||||
(let [[w h] stage
|
||||
c (geom/centroid ref)
|
||||
span (- (reduce max (map :x ref)) (reduce min (map :x ref)))
|
||||
k (/ (* 0.4 w) span)]
|
||||
{[:xform :anchor] (ch/framed [(:x c) (:y c)])
|
||||
[:xform :scale] (ch/framed [k k])
|
||||
[:xform :pos] (ch/framed [(- (/ w 2) (:x c))
|
||||
(- (* 0.25 h) (:y c))])}))
|
||||
|
||||
;; ---------------------------------------------------------------------------
|
||||
;; the clip
|
||||
|
||||
(defn clip
|
||||
"Conditioned measurements -> a clip: nodes with frozen channels, plus the dense
|
||||
blocks they read. `(f params inputs)`, no state.
|
||||
|
||||
params
|
||||
:name names the clip and its store keys
|
||||
:fps the clip's rate. FRAMES is not a parameter — it is
|
||||
`(count outer)`, because a freeze that could disagree with its
|
||||
own input about the length of the take would.
|
||||
:stage [w h] project dimensions, INDEPENDENT of the footage
|
||||
:expose the clip root's exposure grid, inherited by everything
|
||||
:verts the lip rings' vertex budget
|
||||
:aperture-cut fraction of the take's peak aperture below which the mouth
|
||||
interior is not present
|
||||
:head :locked | :as-filmed | :per-plate
|
||||
:kept frames, for :per-plate only
|
||||
:analysis which analysis artifact these measurements came from
|
||||
:anchor-avg
|
||||
:contour-avg the stage-4 knobs. Freeze does not use them; it RECORDS them,
|
||||
because `:generated` is what lets the UI offer a re-freeze at
|
||||
different parameters instead of raw keys.
|
||||
|
||||
inputs
|
||||
:ref the Procrustes reference configuration
|
||||
:transforms the CONDITIONED anchor transforms
|
||||
:outer :inner the CONDITIONED head-local lip rings
|
||||
:aperture head-local aperture per frame
|
||||
:detected optional per-frame booleans; absent frames get the state mask
|
||||
|
||||
The node tree is the one docs/animation-model.md specifies, and the two groups
|
||||
are two different things wanting the same transform:
|
||||
|
||||
:root the clip. EXPOSURE LIVES HERE and is inherited strictly.
|
||||
:face AUTHORED. where the face sits on the stage, and how big.
|
||||
:head MEASURED. the head's motion, or identity.
|
||||
:mouth
|
||||
:mouth-in
|
||||
|
||||
`:mouth-in`'s parent is `:mouth` and that transform is identity today, so
|
||||
composing it is composing identity. It is documented intent, and it is the one
|
||||
place in the tree where the parent pointer is not yet doing work — the moment a
|
||||
painted cel rides a moving plate it will be.
|
||||
|
||||
`:head` keeps its measured channels under `:measured` as well as in
|
||||
`:channels`, so `head-mode` can switch shapes without the blocks or the
|
||||
provenance having to be rebuilt."
|
||||
[{:keys [name fps stage expose verts aperture-cut head kept
|
||||
analysis anchor-avg contour-avg] :as params}
|
||||
{:keys [transforms outer inner detected] :as inputs}]
|
||||
(let [nf (count outer)
|
||||
absent (when detected #(not (nth detected % true)))
|
||||
geom-k (str name "/geom")
|
||||
head-k #(str name "/head-" %)
|
||||
prov (fn [by extra]
|
||||
{:by by :analysis analysis
|
||||
:params (merge {:anchor-avg anchor-avg} extra)})
|
||||
rings (pack {:ctor #(js/Int16Array. %) :scale geom-scale :absent absent}
|
||||
[(rings->flat outer verts) (rings->flat inner verts)])
|
||||
;; The anchor, inverted and split into its three components. Three blocks
|
||||
;; and not one: they are three channels, they have three strides, and a
|
||||
;; single block would need a per-component offset table to say so.
|
||||
inv (mapv invert transforms)
|
||||
xf (fn [f] (pack {:ctor #(js/Float32Array. %) :absent absent} [(mapv f inv)]))
|
||||
pos (xf (fn [t] [(:tx t) (:ty t)]))
|
||||
rot (xf (fn [t] [(:theta t)]))
|
||||
scale (xf (fn [t] [(:s t) (:s t)]))
|
||||
;; Which knobs each channel records is not decoration, it is the
|
||||
;; invalidation table written down where a re-freeze can read it. The
|
||||
;; anchor depends on `anchor avg` alone. The rings depend on it and on
|
||||
;; `contour avg` and on the vertex budget. The aperture depends on
|
||||
;; `anchor avg` and on its own cut and NOT on `contour avg`, because
|
||||
;; measure reports the inner ring's own height and nothing smooths it.
|
||||
anchor-prov (prov :anchor/similarity nil)
|
||||
roto (fn [by] (prov by {:verts verts :contour-avg contour-avg}))
|
||||
scene {:name name
|
||||
:frames nf
|
||||
:fps fps
|
||||
;; Stage dimensions, on the clip and not on the footage. See
|
||||
;; face-placement: nothing below here knows the frame size.
|
||||
:width (first stage)
|
||||
:height (second stage)
|
||||
:nodes
|
||||
{:root
|
||||
{:id :root :name "clip" :kind :group :parent nil :z "a1"
|
||||
:time {:mode :map :expose expose}}
|
||||
|
||||
:face
|
||||
{:id :face :name "face" :kind :group :parent :root :z "a1"
|
||||
:channels (face-placement params inputs)}
|
||||
|
||||
:head
|
||||
{:id :head :name "head" :kind :group :parent :face :z "a1"
|
||||
:measured {[:xform :pos] (dense (head-k "pos") pos 0 anchor-prov)
|
||||
[:xform :rot] (dense (head-k "rot") rot 0 anchor-prov)
|
||||
[:xform :scale] (dense (head-k "scale") scale 0 anchor-prov)}}
|
||||
|
||||
;; The outer lip ring is the dark band OUTSIDE the interior, and
|
||||
;; that three-layer structure — dark ring, pale interior, teeth
|
||||
;; — is what makes a flat shape read as an opening rather than
|
||||
;; as a blob. So it keeps every frame and is never hidden.
|
||||
:mouth
|
||||
{:id :mouth :name "mouth" :kind :poly :parent :head :z "a1"
|
||||
:channels {[:geom :pts] (dense geom-k rings 0 (roto :roto/lips-outer))
|
||||
[:style :color] (ch/framed :skin-dark)}}
|
||||
|
||||
:mouth-in
|
||||
{:id :mouth-in :name "mouth interior" :kind :poly
|
||||
:parent :mouth :z "a2"
|
||||
:channels {[:geom :pts] (dense geom-k rings 1 (roto :roto/lips-inner))
|
||||
[:style :color] (ch/framed :mouth-dark)
|
||||
[:vis] (visibility params inputs
|
||||
(prov :roto/mouth-aperture
|
||||
{:aperture-cut aperture-cut}))}}}}
|
||||
;; Tier 2, behind a handle. The keys are descriptive because there is no
|
||||
;; hashing yet; they become the blocks' sha256 when the backend arrives
|
||||
;; and nothing above here changes, which is the point of a handle.
|
||||
store (into {} (map (fn [[k blk]] [k (select-keys blk [:data :state])]))
|
||||
{geom-k rings
|
||||
(head-k "pos") pos, (head-k "rot") rot, (head-k "scale") scale})]
|
||||
{:store store
|
||||
:scene (head-mode {:mode head :kept kept} {:scene scene :store store})}))
|
||||
|
|
@ -37,8 +37,9 @@
|
|||
"RMS misfit per frame, in the isotropic space's units — one image height.
|
||||
|
||||
Taken against the transforms it is HANDED rather than against a fit of its own,
|
||||
so the same function serves the stage-3 reading and the parity diff. The
|
||||
prototype took it against the SMOOTHED transforms, which folds the smoothing
|
||||
which is what let the parity diff assert on exactly what the prototype handed
|
||||
it while that diff existed. The prototype took the residual against the
|
||||
SMOOTHED transforms, which folds the smoothing
|
||||
error into a number whose whole job is to say whether the footage is
|
||||
stabilisable at all; `fit` takes it against the raw fit instead."
|
||||
[ref rigid tfs]
|
||||
|
|
|
|||
|
|
@ -11,3 +11,8 @@
|
|||
(rf/reg-sub ::muted? (fn [db _] (get-in db [:playback :muted?])))
|
||||
(rf/reg-sub ::fps (fn [db _] (get-in db [:clip :fps])))
|
||||
(rf/reg-sub ::frames (fn [db _] (get-in db [:clip :frames])))
|
||||
;; The stage, in pixels. On the clip because project dimensions are independent
|
||||
;; of the footage — see flow/freeze/face-placement — so the canvas and the raster
|
||||
;; take their size from the document rather than from a constant.
|
||||
(rf/reg-sub ::width (fn [db _] (get-in db [:clip :width])))
|
||||
(rf/reg-sub ::height (fn [db _] (get-in db [:clip :height])))
|
||||
|
|
|
|||
179
frontend/src/arthur/synth.cljs
Normal file
179
frontend/src/arthur/synth.cljs
Normal file
|
|
@ -0,0 +1,179 @@
|
|||
(ns arthur.synth
|
||||
"Synthetic landmark frames, shaped exactly like FaceLandmarker output.
|
||||
|
||||
Exists so the whole chain downstream of detection - Procrustes, smoothing,
|
||||
stabilisation, key selection, rasterising, take writing - can be exercised and
|
||||
verified without a video file. A synthetic face is also the only way to test
|
||||
stabilisation against a KNOWN head motion, since real footage gives no ground
|
||||
truth to compare against.
|
||||
|
||||
In `src/` and not in `test/`, and that moved at step 5: the synthetic take is
|
||||
what the tool plays before any footage has been detected, so the generator is
|
||||
app code that stands in for `flow/detect` rather than a test fixture. Step 6
|
||||
puts MediaPipe beside it, not in place of it — a tool that needs a video file
|
||||
before it will show you anything is a tool you cannot debug.
|
||||
|
||||
Differs from js/synth.js in exactly one way, deliberately: the jitter comes
|
||||
from a SEEDED generator rather than Math.random. A failing assertion has to be
|
||||
reproducible to be worth anything, and a wrong pose that happened once is not
|
||||
a bug report. `:rand-fn` takes the generator over entirely, which is what let
|
||||
the JS-parity harness hand both implementations the identical track while it
|
||||
existed."
|
||||
(:require [arthur.domain.landmarks :as lm]))
|
||||
|
||||
;; mulberry32. Chosen for being four lines of int32 arithmetic that port
|
||||
;; unambiguously between JS and CLJS, not for its statistics: this is jitter for
|
||||
;; a smoother to remove, not a source of entropy.
|
||||
(defn mulberry32 [seed]
|
||||
(let [a (atom (bit-or seed 0))]
|
||||
(fn []
|
||||
(let [x (swap! a (fn [v] (bit-or (+ v 0x6D2B79F5) 0)))
|
||||
t (js/Math.imul (bit-xor x (unsigned-bit-shift-right x 15)) (bit-or 1 x))
|
||||
t (bit-xor (+ t (js/Math.imul (bit-xor t (unsigned-bit-shift-right t 7))
|
||||
(bit-or 61 t)))
|
||||
t)]
|
||||
(/ (unsigned-bit-shift-right (bit-xor t (unsigned-bit-shift-right t 14)) 0)
|
||||
4294967296)))))
|
||||
|
||||
;; Half the corner separation, and the lid half-height at full open.
|
||||
(def ^:private EYE-RX 0.0235)
|
||||
(def ^:private EYE-RY 0.011)
|
||||
(def ^:private EYE-Y -0.044)
|
||||
|
||||
(defn synth-dense
|
||||
"`n-frames` of dense landmarks.
|
||||
|
||||
`:swap-iris` places the two iris blocks on the opposite eyes. It exists so the
|
||||
pairing resolver can be tested against a track it actually disagrees with:
|
||||
a resolver checked only against the convention it was written for is checking
|
||||
nothing at all."
|
||||
([] (synth-dense 72 {}))
|
||||
([n-frames] (synth-dense n-frames {}))
|
||||
([n-frames {:keys [swap-iris rand-fn seed]
|
||||
:or {swap-iris false, seed 1}}]
|
||||
(let [rnd (or rand-fn (mulberry32 seed))]
|
||||
(vec
|
||||
(for [t (range n-frames)]
|
||||
(let [pts (make-array lm/NUM-LANDMARKS)
|
||||
_ (dotimes [i lm/NUM-LANDMARKS] (aset pts i {:x 0.5 :y 0.5 :z 0}))
|
||||
|
||||
;; Known head motion: drift, sway, roll and a slow scale change, plus a
|
||||
;; little per-frame jitter so transform smoothing has something to remove.
|
||||
ph (/ t n-frames)
|
||||
hx (+ 0.5 (* 0.045 (js/Math.sin (* ph js/Math.PI 2))) (* (- (rnd) 0.5) 0.002))
|
||||
hy (+ 0.5 (* 0.02 (js/Math.cos (* ph js/Math.PI 3))) (* (- (rnd) 0.5) 0.002))
|
||||
roll (* 0.18 (js/Math.sin (* ph js/Math.PI 2.5)))
|
||||
scale (+ 1 (* 0.06 (js/Math.sin (* ph js/Math.PI 1.5))))
|
||||
cr (js/Math.cos roll)
|
||||
sr (js/Math.sin roll)
|
||||
place (fn [i lx ly]
|
||||
(let [sx (* lx scale) sy (* ly scale)]
|
||||
(aset pts i {:x (- (+ hx (* cr sx)) (* sr sy))
|
||||
:y (+ hy (* sr sx) (* cr sy))
|
||||
:z 0})))
|
||||
|
||||
;; Mouth opens in four sustained beats with holds between, so key selection
|
||||
;; has genuine extremes and genuine plateaux to find.
|
||||
beat (mod (js/Math.floor (/ t 9)) 4)
|
||||
open-amt (nth [0.004 0.05 0.022 0.0] beat)
|
||||
wide (+ 0.10 (case beat 1 0.012, 3 -0.008, 0))
|
||||
|
||||
;; A blink is ONE frame, which is the honest hard case: at 12fps that is
|
||||
;; what a real blink costs, and it is exactly the length that reads as a
|
||||
;; dropped frame rather than as a blink unless `hold` extends it.
|
||||
blink (and (> t 5) (zero? (mod t 19)))
|
||||
openness (if blink 0.05 1)
|
||||
|
||||
;; Gaze holds and then jumps, the way gaze actually behaves, with a little
|
||||
;; jitter on top so quantisation has noise to remove and the dwell has
|
||||
;; something to suppress.
|
||||
[gx gy] (nth [[0 0] [0.16 0.0] [-0.16 0.05] [0.0 -0.09]]
|
||||
(mod (js/Math.floor (/ t 11)) 4))
|
||||
jit (fn [] (* (- (rnd) 0.5) 0.012))
|
||||
|
||||
;; Eyes. The corners (RIGID[0..3]) are placed BY the lid rings rather than
|
||||
;; separately, because they are slots 0 and 8 of those rings: writing them
|
||||
;; twice is how the mouth grew a bowtie, and a corner that disagrees with
|
||||
;; its own ring would make the eye self-intersect at some vertex budgets
|
||||
;; and not others.
|
||||
eye (fn [ring cx dir iris]
|
||||
(let [n (count ring)]
|
||||
(dotimes [k n]
|
||||
;; dir flips the traversal so each ring runs the direction its real
|
||||
;; table does: slot 0 outer corner, 4 upper lid, 8 inner, 12 lower.
|
||||
(let [a (if (pos? dir)
|
||||
(+ js/Math.PI (* (/ k n) js/Math.PI 2))
|
||||
(- (* (/ k n) js/Math.PI 2)))]
|
||||
(place (nth ring k)
|
||||
(+ cx (* EYE-RX (js/Math.cos a)))
|
||||
(+ EYE-Y (* EYE-RY openness (js/Math.sin a))))))
|
||||
;; Iris: centre first, then four ring points, as the refined mesh emits.
|
||||
(let [ix (+ cx (* (+ gx (jit)) EYE-RX 2))
|
||||
iy (+ EYE-Y (* (+ gy (jit)) EYE-RX 2))
|
||||
m (count iris)]
|
||||
(place (nth iris 0) ix iy)
|
||||
(doseq [k (range 1 m)]
|
||||
(let [a (* (/ (dec k) (dec m)) js/Math.PI 2)]
|
||||
(place (nth iris k)
|
||||
(+ ix (* 0.008 (js/Math.cos a)))
|
||||
(+ iy (* 0.008 (js/Math.sin a)))))))))
|
||||
|
||||
;; Brows, held in four sustained poses so raise quantisation has genuine
|
||||
;; plateaux to find: rest, surprise (both ends up), worry (inner up only),
|
||||
;; anger (inner down). Commanded in eye widths above the eye centre so the
|
||||
;; measurement can be checked against a number rather than an eyeball.
|
||||
[b-out b-in] (nth [[0.30 0.30] [0.46 0.46] [0.30 0.44] [0.30 0.18]]
|
||||
(mod (js/Math.floor (/ t 13)) 4))
|
||||
EYE-W (* EYE-RX 2)
|
||||
HALF 0.006 ; ring half-thickness
|
||||
brow (fn [ring cx outer-sign]
|
||||
;; Slots 0-4 are one edge outer->inner, 5-9 the other inner->outer, so the
|
||||
;; ends land on {0,9} and {4,5} exactly as the table promises.
|
||||
(let [n (count ring) half (/ n 2)]
|
||||
(dotimes [k n]
|
||||
(let [along (if (< k half)
|
||||
(/ k (dec half))
|
||||
(/ (- n 1 k) (dec half)))
|
||||
rise (+ b-out (* (- b-in b-out) along))]
|
||||
(place (nth ring k)
|
||||
(+ cx (* outer-sign (- EYE-RX (* along EYE-W)) 1.05))
|
||||
(+ (- EYE-Y (* rise EYE-W))
|
||||
(if (< k half) (- HALF) HALF)))))))
|
||||
|
||||
;; Lip rings as ellipse arcs, traversed so ring ORDER matches the tables:
|
||||
;; slot 0 = right corner, 5 = top centre, 10 = left corner, 15 = bottom
|
||||
;; centre, with y growing downward. Getting this convention wrong swaps two
|
||||
;; opposite vertices and the ring self-intersects into a bowtie - see the
|
||||
;; ring-simplicity assertion in domain/ring's tests.
|
||||
ring (fn [table rx ry cy]
|
||||
(let [n (count table)]
|
||||
(dotimes [k n]
|
||||
(let [a (- (* (/ k n) js/Math.PI 2))]
|
||||
(place (nth table k)
|
||||
(* rx (js/Math.cos a))
|
||||
(+ cy (* ry (js/Math.sin a))))))))]
|
||||
|
||||
(place (nth lm/RIGID 4) 0.000 -0.050)
|
||||
(place (nth lm/RIGID 5) 0.000 -0.020)
|
||||
(place (nth lm/RIGID 6) 0.000 0.012)
|
||||
|
||||
(eye lm/EYE-R-RING -0.0515 1 (if swap-iris lm/IRIS-B lm/IRIS-A))
|
||||
(eye lm/EYE-L-RING 0.0515 -1 (if swap-iris lm/IRIS-A lm/IRIS-B))
|
||||
|
||||
(brow lm/BROW-A-RING -0.0515 -1)
|
||||
(brow lm/BROW-B-RING 0.0515 1)
|
||||
|
||||
(ring lm/LIPS-OUTER (/ wide 2) (+ 0.012 (* open-amt 0.6)) 0.075)
|
||||
;; APERTURE (13, 14) are slots 5 and 15 of the inner ring, so the ring itself
|
||||
;; places them at the vertical extremes. Writing them again afterwards is what
|
||||
;; produced the bowtie; the aperture is simply the inner ring's height.
|
||||
(ring lm/LIPS-INNER (/ wide 2.6) (+ 0.001 open-amt) 0.075)
|
||||
|
||||
(let [n (count lm/FACE-OVAL)]
|
||||
(dotimes [k n]
|
||||
(let [a (+ (- (/ js/Math.PI 2)) (* (/ k n) js/Math.PI 2))]
|
||||
(place (nth lm/FACE-OVAL k)
|
||||
(* 0.105 (js/Math.cos a))
|
||||
(+ (* 0.145 (js/Math.sin a)) 0.01)))))
|
||||
|
||||
(vec pts)))))))
|
||||
|
|
@ -107,6 +107,8 @@
|
|||
:ramp @(rf/subscribe [::render/ramp])
|
||||
:fps @(rf/subscribe [::sub/fps])
|
||||
:frames @(rf/subscribe [::sub/frames])
|
||||
:width @(rf/subscribe [::sub/width])
|
||||
:height @(rf/subscribe [::sub/height])
|
||||
:frame @(rf/subscribe [::sub/frame])
|
||||
:playing? @(rf/subscribe [::sub/playing?])})
|
||||
;; A new resolver means a new scene or a new palette, and neither
|
||||
|
|
@ -133,13 +135,16 @@
|
|||
rasterised before the next frame is asked for."
|
||||
[f]
|
||||
(let [{:keys [canvas]} @state
|
||||
{:keys [resolver palette ramp]} @snapshot]
|
||||
(when (and canvas resolver)
|
||||
{:keys [resolver palette ramp width height]} @snapshot]
|
||||
(when (and canvas resolver width height)
|
||||
;; User Timing, so a profile in the DevTools performance panel has named
|
||||
;; spans in the Timings track instead of a wall of anonymous frames. Three
|
||||
;; lines, and the difference between reading a profile and guessing at one.
|
||||
(js/performance.mark "arthur/paint:start")
|
||||
(let [ras (raster-for 320 200)]
|
||||
;; THE STAGE IS THE CLIP'S, not a constant. Project dimensions are
|
||||
;; independent of the footage, so the size the frame is rasterised at comes
|
||||
;; out of the document like everything else.
|
||||
(let [ras (raster-for width height)]
|
||||
(-> ras
|
||||
(raster/clear! (get palette :bg 0))
|
||||
(raster/draw-ops! (resolver f)))
|
||||
|
|
|
|||
|
|
@ -7,7 +7,6 @@
|
|||
why scrubbing at speed does not re-render the page."
|
||||
(:require [arthur.clock :as clock]
|
||||
[arthur.db :as db]
|
||||
[arthur.demo :as demo]
|
||||
[arthur.events.playback :as pb]
|
||||
[arthur.subs.playback :as sub]
|
||||
[arthur.subs.render :as render]
|
||||
|
|
@ -80,14 +79,22 @@
|
|||
(when (and drop (pos? drop))
|
||||
(str " · " (.toFixed drop 2) " frames/paint")))])]]))
|
||||
|
||||
(defn- stage []
|
||||
;; The canvas is the STAGE's size, and the stage is the clip's — not a constant
|
||||
;; and not the footage's. Reactive, so selecting a clip of another size resizes
|
||||
;; it; ui/canvas guards the width assignment, which reallocates the backing
|
||||
;; store, so this being a re-render costs nothing per frame.
|
||||
(let [w @(rf/subscribe [::sub/width])
|
||||
h @(rf/subscribe [::sub/height])]
|
||||
[:canvas.stage
|
||||
{:ref #(player/set-canvas! %)
|
||||
:width w :height h
|
||||
:style {:width (str (* zoom w) "px") :height (str (* zoom h) "px")}}]))
|
||||
|
||||
(defn view []
|
||||
[:main
|
||||
[:h1 "arthur"]
|
||||
[:canvas.stage
|
||||
{:ref #(player/set-canvas! %)
|
||||
:width demo/width :height demo/height
|
||||
:style {:width (str (* zoom demo/width) "px")
|
||||
:height (str (* zoom demo/height) "px")}}]
|
||||
[stage]
|
||||
[audio]
|
||||
[transport]
|
||||
[:p.note
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue