noe/arthur
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

The face owns its placement, not the take that holds it

Browse source 
The source-to-stage mapping moves off :main's :face group and onto each face's
own :place, above its head. `face-placement` computes exactly what it computed
before, over every subject together, so two faces filmed side by side keep
their filmed relation — it is written into each face instead of onto a group
above them all. Same transform, same subtree, one level lower, and the
composite is identical to the pixel: a digest over every op :main emits across
the whole take is unchanged either way.

THE OWNER IS THE POINT. A face carrying its own mapping is the right size
wherever it is put — dropped into another symbol, or opened in its own tab to
be drawn over — and the take that holds it needs to know nothing. On a group
above the instances the scale belonged to the take, so a face taken out of it
had no size at all and drew at a fraction of a pixel.

The pool's thumbnails drop the workaround that knew about this: a symbol is
rendered rooted at itself again, because a face now carries the placement that
makes that honest, so the pool needs to know nothing about where a symbol
happens to be used. `domain/node` and `arthur.export` leave its requires with it.

The tests here were reading the placement off :main. The photo registration
test changes shape rather than location: its premise was that face-1's head is
its own root, so a photo sitting where it was filmed was image pixels over
image height and nothing else. The head still cancels — that is what the test
is about — but it now cancels against the face's own placement, which is why
the photo comes with the face into its own tab instead of sitting at a
fraction of a pixel beside it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Your Name 2026-10-01 01:25:37 -04:00
parent e6d0ededb1
commit a4ce750be2
9 changed files with 106 additions and 85 deletions
Download patch file Download diff file Expand all files Collapse all files

47
frontend/src/arthur/flow/freeze.cljs
View file

@ -25,7 +25,8 @@
oval's bbox and zooms until the face is 80% of the raster height, so every
stored vertex carries a cropping decision made once, at analysis time, from one
frame's landmarks. Here the geometry stays in the node's own local space and the
framing is `[:xform :*]` on an authored `:face` node, which the stage clips.
framing is `[:xform :*]` on each face's own authored `:place` node, which the
stage clips.
Project dimensions are therefore independent of the footage — see
`face-placement`, and \"What space geometry is in\" in docs/animation-model.md.
@ -396,7 +397,8 @@
The synthetic take uses the reference rigid configuration for its default.
Real footage can request `:fit-motion?`: its default fits the observed mouth,
eyes and brows in the stage across the shot.
Both are ordinary editable transforms on :face, never baked into the geometry.
Both are ordinary editable transforms on the face's own `:place`, never baked
into the geometry.
The face oval is not measured, because its only consumers in the prototype were
the old baked framing transform and the placeholder plate outline.
@ -419,8 +421,18 @@
leaves the jaw and the mouth on the stage. Whatever hangs off is clipped, which
is not a feature to add: every fill in `domain/raster` clamps already.
All subjects share one source-to-stage mapping on the :face group. Each
instance can then be placed independently with ordinary transform channels."
ONE MAPPING, COMPUTED OVER EVERY SUBJECT AND WRITTEN INTO EACH FACE. It is
measured across all of them together — which is what keeps two faces filmed
side by side in their filmed relation — and then stored on each face's own
`:place` rather than on a group above them all. Same transform, same subtree,
one level lower: the composite is identical to the pixel.
THE OWNER IS THE POINT. A face carrying its own source-to-stage mapping is a
face that is the right size wherever it is put — dropped into another symbol,
or opened in its own tab to be drawn over — and the symbol that places it needs
to know nothing. On a group above the instances the scale belonged to the take,
so a face taken out of that take had no size at all and drew at a fraction of a
pixel."
[{:keys [stage fit-motion?]} subjects]
(let [[w h] stage
inputs (vals subjects)]
@ -646,6 +658,17 @@
;; ---------------------------------------------------------------------------
;; the clip
(defn- place-in
"Put the source-to-stage mapping on symbol `sym`'s own `:place`, above its head.
`:place` is the one authored node a freeze leaves on a face, and `face-placement`
says why it is a default rather than a measurement."
[sym channels]
(-> sym
(assoc-in [:nodes :place] {:id :place :name "source placement" :kind :group
:z "a1" :channels channels})
(assoc-in [:nodes :head :parent] :place)))
(defn- subject-part
"A subject's drawing, metadata and blocks. Node names are symbol-local."
[params subject {:keys [outer eyes brows teeth] :as inputs}]
@ -727,10 +750,11 @@
[{:keys [name fps stage expose] :as params} subjects]
(when-not (and (map? subjects) (seq subjects)
(every? keyword? (keys subjects))
(not-any? #{:main :root :face} (keys subjects)))
(throw (ex-info "a freeze needs subjects with ids distinct from :main, :root and :face" {})))
(not-any? #{:main :root :place} (keys subjects)))
(throw (ex-info "a freeze needs subjects with ids distinct from :main, :root and :place" {})))
(let [ordered (sort-by (comp str key) subjects)
parts (mapv (fn [[id inputs]] [id (subject-part params id inputs)]) ordered)
placement (face-placement params subjects)
lengths (distinct (map #(get-in % [1 :symbol :frames]) parts))
_ (when-not (and (= 1 (count lengths)) (pos? (first lengths)))
(throw (ex-info "subjects need the same positive frame count"
@ -745,17 +769,16 @@
(into {:main
{:id :main :frames nf
:nodes (into {:root {:id :root :name "clip" :kind :group :z "a1"
:time {:mode :map :expose expose}}
:face {:id :face :name "source placement" :kind :group
:parent :root :z "a1"
:channels (face-placement params subjects)}}
:time {:mode :map :expose expose}}}
(map-indexed
(fn [i [id _]]
[id {:id id :kind :instance :parent :face
[id {:id id :kind :instance :parent :root
:z (str "a" i)
:source {:symbol id}}]))
ordered)}}
(map (fn [[id part]] [id (:symbol part)])) parts)}]
(map (fn [[id part]]
[id (place-in (:symbol part) placement)]))
parts)}]
(doseq [[subject inputs] ordered
[id track] (:presence inputs)]
(when-not (and (= nf (count track))

41
frontend/src/arthur/ui/pool.cljs
View file

@ -45,9 +45,7 @@
footage stay separate records on the server, so dropping the same file twice
does not decode it twice."
(:require [arthur.domain.clip :as clip]
[arthur.domain.node :as node]
[arthur.domain.raster :as raster]
[arthur.export :as export]
[arthur.events.footage :as footage]
[arthur.events.playback :as pb]
[arthur.events.project :as project]
@ -74,35 +72,16 @@
;; cache's.
(atom {:clip nil :urls {}}))
(defn- placed-in
"Where `sid` is placed, as `[host-symbol node-id]`, or nil for one nothing
places. Lowest ids first, so a symbol placed several times gets one picture and
the same one every time."
[document sid]
(first (for [h (sort-by str (keys (:symbols document)))
n (sort-by str (keys (get-in document [:symbols h :nodes])))
:when (contains? (node/sources (get-in document [:symbols h :nodes n])) sid)]
[h n])))
(defn- draw-symbol
"`sid`'s first frame as a PNG data URL, through the resolver and the rasteriser
the stage uses — so the picture in the pool is the picture, not a sketch of it.
THROUGH THE PLACEMENT, WHERE THERE IS ONE, and this is the whole subtlety.
Rooting at a symbol renders its DRAWING, in its own local space; a rotoscoped
face is stored head-local in units of one image height, so its numbers are
around 0..1 and the source-to-stage scale that turns them into stage pixels —
several hundred — lives on the `:face` group of the symbol that PLACES it. See
`freeze/face-placement` and \"What space geometry is in\" in
docs/animation-model.md. Rendered rooted at itself, a face is therefore correct
and under a pixel across, which is a true picture of nothing anybody wants to
look at.
So a placed symbol is drawn where it is placed, with everything else in that
host isolated away — `export/isolate`, whose docstring draws the same
distinction for the same reason. A symbol nothing places has no placement to
borrow and is rendered rooted at itself, which for the main timeline is exactly
right because the stage is its own.
ROOTED AT THE SYMBOL, which is only honest because a symbol carries its own
placement. A face's geometry is head-local in units of one image height, and
the source-to-stage mapping that turns those into stage pixels lives on the
face's own `:place` — see `freeze/face-placement`. So a face drawn by itself is
the size it is on the stage, and this needs to know nothing about where it
happens to be used.
nil rather than a throw when the frame will not resolve — a cycle, a missing
block, an instance naming a symbol that has gone. A pool is a list of names,
@ -116,12 +95,8 @@
[document sid store palette ramp]
(try
(when (clip/symbol document sid)
(let [[host id] (placed-in document sid)
document (cond-> document
host (update-in [:symbols host] export/isolate id))
root (or host sid)
resolve (clip/resolver document root store palette {})
[w h] (clip/stage document root)]
(let [resolve (clip/resolver document sid store palette {})
[w h] (clip/stage document sid)]
(-> (raster/make w h)
(raster/clear! (get palette :bg 0))
(raster/draw-ops! (resolve 0))

28
frontend/test/arthur/domain/trace_test.cljs
View file

@ -4,6 +4,7 @@
[arthur.domain.clip :as clip]
[arthur.domain.leaf :as leaf]
[arthur.domain.nest :as nest]
[arthur.domain.node :as node]
[arthur.domain.palette :as pal]
[arthur.domain.symbol :as symbol]
[arthur.domain.trace :as trace]
@ -58,15 +59,30 @@
(vec (array-seq (trace/photo-matrix (symbol/world-of r :head) h @store
(trace/photo-frame (trace/of h) f) 1000)))))
(defn- filmed
"Where a photo sitting exactly where it was filmed must land: the face's OWN
placement, over image height, and nothing else.
`:place` carries the source-to-stage mapping now, above `:head` — so a face
opened in its own tab draws at the size it is on the stage, and its photo has
to come with it. That is the whole reason the mapping belongs to the face: the
head still cancels out, which is what these assertions are about, but it
cancels against a placement rather than against nothing."
[c f]
(let [r (symbol/resolver (clip/symbol c :face-1) @store pal/index-of nil)]
(r f)
(vec (array-seq (node/mul! (node/mat) (symbol/world-of r :place)
(js/Float64Array. #js [0.001 0 0 0.001 0 0]))))))
(deftest the-photo-registers-to-the-face-it-was-filmed-with
;; Face-1 is its own symbol and its head is its root, so a photo sitting where
;; it was filmed is image pixels over image height and nothing else.
(let [filmed [0.001 0 0 0.001 0 0]]
(is (near? filmed (photo-at (traced {:frames [] :origin :continuous}) 30))
(let [a (traced {:frames [] :origin :continuous})
b (traced {:frames [12] :origin :keys})
d (traced {:frames [12] :origin :continuous})]
(is (near? (filmed a 30) (photo-at a 30))
"a head showing the photo's own frame cancels out")
(is (near? filmed (photo-at (traced {:frames [12] :origin :keys}) 30))
(is (near? (filmed b 30) (photo-at b 30))
"held at the trace frame, face and photo both stand still")
(is (not (near? filmed (photo-at (traced {:frames [12] :origin :continuous}) 30)))
(is (not (near? (filmed d 30) (photo-at d 30)))
"a continuous head carries the held photo along with it")))
(defn- wrapped

54
frontend/test/arthur/flow/freeze_test.cljs
View file

@ -86,13 +86,19 @@
(is (empty? (clip/problems c)) (str spec ": " (pr-str (clip/problems c)))))))
(deftest the-tree-is-the-one-the-model-specifies
(is (= [:face :root] (symbol/lineage (:nodes (clip/symbol @clip* :main)) :face)))
;; :main HOLDS INSTANCES AND NOTHING ELSE. The source-to-stage placement used
;; to be a `:face` group here, above them all; it is each face's own `:place`
;; now, so the take is a container that knows nothing and a face is the right
;; size wherever it is put. Same transform, same subtree, one level lower.
(is (= [:face-1 :root] (symbol/lineage (:nodes (clip/symbol @clip* :main)) :face-1)))
(is (nil? (get-in @clip* [:symbols :main :nodes :face])))
(is (= #{:face-1} (node/sources (get-in @clip* [:symbols :main :nodes :face-1]))))
(is (= [:head] (symbol/lineage (:nodes @sym*) :head)))
(is (= [:mouth :head] (symbol/lineage (:nodes @sym*) :mouth)))
(is (= [:mouth-in :mouth :head] (symbol/lineage (:nodes @sym*) :mouth-in)))
(is (= [:place] (symbol/lineage (:nodes @sym*) :place)))
(is (= [:head :place] (symbol/lineage (:nodes @sym*) :head)))
(is (= [:mouth :head :place] (symbol/lineage (:nodes @sym*) :mouth)))
(is (= [:mouth-in :mouth :head :place] (symbol/lineage (:nodes @sym*) :mouth-in)))
(is (= {:mode :map :expose 2} (:time (node :root))))
(is (every? #(nil? (:time (node %))) [:face :head :mouth :mouth-in])))
(is (every? #(nil? (:time (node %))) [:place :head :mouth :mouth-in])))
(deftest geometry-is-flat-and-dense-and-the-interior-shares-the-outline-s-block
(doseq [id [:mouth :mouth-in]]
@ -121,11 +127,11 @@
(abs (- a b))))]
(is (<= gap (/ q 2))
(str "worst quantisation error " gap " against a quantum of " q))
;; And in stage pixels, which is the unit anyone can judge. `:face`'s scale is
;; And in stage pixels, which is the unit anyone can judge. `:place`'s scale is
;; stage px per image height, so the whole quantum is q·k — a twentieth of a
;; pixel at this placement, two orders below anything the rasteriser can
;; express, which is the argument for Int16 geometry stated as a measurement.
(let [k (first (:value (chan :face [:xform :scale])))]
(let [k (first (:value (chan :place [:xform :scale])))]
(is (< (* q k) 0.1)
(str "the quantum is " (* q k) " stage pixels at " k " px per image height"))
(is (< (* gap k) (* q k))))))
@ -190,15 +196,15 @@
(deftest head-local-geometry-composed-through-head-and-face-lands-on-the-stage
;; The end-to-end claim of the split, asserted against the OPS the resolver
;; actually emits rather than against an intermediate: stored head-local, put
;; back through `:head`, placed by `:face`, the mouth is where the composition
;; back through `:head`, placed by `:place`, the mouth is where the composition
;; of the two says it is. A test that recomputed the chain would only be
;; checking arithmetic against itself; this checks `node/local!`, `node/world!`
;; and `emit` as well.
(let [c (freeze/head-mode {} @frozen)
res (clip/resolver c :main @store pal/index-of nil)
k (first (:value (chan :face [:xform :scale])))
anc (:value (chan :face [:xform :anchor]))
pos (:value (chan :face [:xform :pos]))
k (first (:value (chan :place [:xform :scale])))
anc (:value (chan :place [:xform :anchor]))
pos (:value (chan :place [:xform :pos]))
tfs (:transforms @take/measured)]
(doseq [f (range 0 take/frames 13)]
(let [ops (res f)
@ -217,7 +223,7 @@
;; the fit removed the head's motion, so putting it back is the fit's
;; inverse. Node :head carries exactly this.
im (geom/apply-sim (freeze/invert (nth tfs ef)) g)
;; And where :face puts it: scaled about the anchor, then translated,
;; And where :place puts it: scaled about the anchor, then translated,
;; which is p ↦ k(p - anchor) + anchor + pos.
wx (+ (* k (- (:x im) (nth anc 0))) (nth anc 0) (nth pos 0))
wy (+ (* k (- (:y im) (nth anc 1))) (nth anc 1) (nth pos 1))]
@ -272,8 +278,8 @@
b (freeze/head-mode {:trace {:origin :start}} @frozen)
c (freeze/head-mode {:trace {:origin :keys :frames [0 40]}} @frozen)]
(doseq [x [b c]]
(is (= (get (nodes a) :face) (get (nodes x) :face))
":face moved")
(is (= (get (nodes a) :place) (get (nodes x) :place))
":place moved")
(is (= (dissoc (nodes a) :head) (dissoc (nodes x) :head))
"a node other than :head changed")
;; The measurement does not go away when the head is locked: always measure,
@ -304,7 +310,7 @@
;; transform on a node, which a hand can revise, and it claims no generator that
;; would offer to overwrite it.
(doseq [path [[:xform :pos] [:xform :rot] [:xform :scale] [:xform :anchor]]]
(let [c (get (node/channels (node :face)) path)]
(let [c (get (node/channels (node :place)) path)]
(is (= :framed (ch/describe c)) (str path " is not framed"))
(is (nil? (:generated c)) (str path " claims provenance")))))
@ -324,8 +330,8 @@
(let [c @clip*]
(doseq [sid [:main :face-1]
id (keys (get-in c [:symbols sid :nodes]))
;; `:face` authors its own in `face-placement`, upstream of this.
:when (not= [:main :face] [sid id])]
;; `:place` authors its own in `face-placement`, upstream of this.
:when (not= [:face-1 :place] [sid id])]
(let [n (get-in c [:symbols sid :nodes id])
frames (range (get-in c [:symbols sid :frames]))
anchor (:value (get-in n [:channels [:xform :anchor]]))
@ -364,11 +370,11 @@
;; differently, it has been applied to a node whose transform is not the
;; identity, which is the one way it could go wrong.
(let [c @clip*
;; Every node the freeze makes EXCEPT `:face`, whose anchor
;; Every node the freeze makes EXCEPT `:place`, whose anchor
;; `face-placement` authors — which is the set `pivoted` writes.
every (for [sid [:main :face-1]
id (keys (get-in c [:symbols sid :nodes]))
:when (and (not= [:main :face] [sid id])
:when (and (not= [:face-1 :place] [sid id])
(seq (get-in c [:symbols sid :nodes id :channels])))]
[sid id])
anchors #(into {} (for [[sid id] every]
@ -385,13 +391,13 @@
(is (= (render bare f) (render again f))
(str "frame " f " draws differently once every part has a pivot")))))
(deftest the-face-puts-the-head-s-centre-where-it-says-it-does
(deftest the-place-puts-the-head-s-centre-where-it-says-it-does
;; anchor + pos is where the anchor lands in the parent, which is what makes
;; `:anchor` the registration point: scale and rotation happen about the head's
;; centre rather than about the corner of the footage, where MediaPipe's
;; normalised space has its origin.
(let [anc (:value (chan :face [:xform :anchor]))
pos (:value (chan :face [:xform :pos]))
(let [anc (:value (chan :place [:xform :anchor]))
pos (:value (chan :place [:xform :pos]))
c (geom/centroid (:ref @take/measured))]
(is (< (abs (- (nth anc 0) (:x c))) 1e-12))
(is (< (abs (- (nth anc 1) (:y c))) 1e-12))
@ -418,8 +424,8 @@
(vec (array-seq (:data (get (:store big) (key-of (:clip big)))))))
"the geometry is the same numbers at either stage size")
(is (not= (:value (get-in (nodes (:clip big))
[:face :channels [:xform :scale]]))
(:value (chan :face [:xform :scale]))))))
[:place :channels [:xform :scale]]))
(:value (chan :place [:xform :scale]))))))
;; ---------------------------------------------------------------------------
;; the aperture, as [:vis]

2
frontend/test/arthur/flow/multi_face_test.cljs
View file

@ -177,7 +177,7 @@
flat (assoc-in clip [:symbols :main :nodes]
(merge (dissoc (:nodes (clip/symbol clip :main)) :face-1)
(assoc-in (get-in clip [:symbols :face-1 :nodes])
[:head :parent] :face)))
[:head :parent] :place)))
nested (clip/resolver clip :main store pal/index-of nil)
reference (clip/resolver flat :main store pal/index-of nil)]
(doseq [f [0 1 7 20 39]]

8
frontend/test/arthur/flow/regenerate_test.cljs
View file

@ -46,8 +46,8 @@
(channel after :iris-l [:xform :pos])))
(is (= (channel before :mouth [:geom :pts])
(channel after :mouth [:geom :pts])))
(is (= (get-in before [:clip :symbols :main :nodes :face])
(get-in after [:clip :symbols :main :nodes :face])))
(is (= (get-in before [:clip :symbols :face-1 :nodes :place])
(get-in after [:clip :symbols :face-1 :nodes :place])))
(is (= (channel (full-at {:gaze-gain 2}) :iris-r [:xform :pos])
(channel after :iris-r [:xform :pos])))))
@ -85,8 +85,8 @@
(get-in after [:clip :symbols :face-1 :nodes :head :measured])))
(is (= (get-in full [:clip :symbols :face-1 :nodes :head :measured])
(get-in after [:clip :symbols :face-1 :nodes :head :measured])))
(is (= (get-in before [:clip :symbols :main :nodes :face])
(get-in after [:clip :symbols :main :nodes :face])))))
(is (= (get-in before [:clip :symbols :face-1 :nodes :place])
(get-in after [:clip :symbols :face-1 :nodes :place])))))
(deftest contour-edit-does-not-rebuild-head-or-teeth
(let [before @initial