noe/arthur
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Hold a tracing photo steady, and switch it on where you would look

Browse source 
The photo blinked out for a frame or two and sometimes never arrived, from
three causes that each present as the same bug. The still cache emptied
itself on the frame it filled, so every still on screen had to be fetched
and decoded again — every 48 frames of a scrub, and with two traced faces a
permanent flicker, each one's still evicting the other's; it drops the least
recently used now, a still being touched on every frame it is drawn. A still
cannot decode in the animation frame that asks for it, and a face whose next
one had not arrived drew nothing, so it now keeps the still it was showing
until the new one is there. And nothing was read ahead, so continuous
playback was always a frame behind its own footage; the next few frames'
stills are asked for while the playhead is moving on its own, and only then,
because a scrub asks for a different few at every step.

Whether the footage shows is no longer the document's. It was an :underlay
on an instance, inherited down the row path, nearest wins, and it went
through edit — so showing a reference photo was an undo step that travelled
to collaborators. It is [:ui :trace] now, the faces switched on and one
opacity, like solo, and there is nothing left to inherit: a face is the same
face wherever it is placed, so one switch covers every placement of it. The
paint loop asks for its own redraw when that changes, since the resolver no
longer does it for them.

Opening a face shows its footage, because a symbol has measured footage
behind it only because it was traced from that; a take does not, because a
take is the picture. The switch is on the bar above the stage, with the tone
and the tool, and on each face's timeline row beside solo — not a section
that appeared once the right row had been found. A face open in its own tab
could not show its footage at all before, shown having walked instances, and
that is the one place tracing matters most. The inspector keeps the face's
own keyed facts, its trace keys and its origin, and says why it cannot key a
frame rather than greying out the only button in the section.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Your Name 2026-09-30 10:42:39 -04:00
parent ae03b61dca
commit ed88c5e674
13 changed files with 402 additions and 156 deletions
Download patch file Download diff file Expand all files Collapse all files

2
frontend/src/arthur/db.cljs
View file

@ -12,6 +12,7 @@
are entirely dense."
(:require [arthur.demo :as demo]
[arthur.domain.clip :as domain-clip]
[arthur.domain.trace :as trace]
[arthur.demo.swarm :as swarm]
[arthur.demo.take :as take]))
@ -163,6 +164,7 @@
:tool nil
:draft []
:knobs {}
:trace {:faces #{} :opacity trace/opacity-default}
:expanded #{}}})
(def rates

76
frontend/src/arthur/domain/trace.cljs
View file

@ -9,13 +9,12 @@
:trace {:frames [0 12 30] :origin :keys}
Whether the photo is showing, and how strongly, is a viewing aid for one
placement. It is on an INSTANCE, not keyed, and draws nothing into the picture.
It covers every face at or below that instance, and the nearest instance that
says anything decides, so a take shows its faces' footage and one face inside
it can still be switched off:
:underlay {:on? true :opacity 0.5}
Whether the photo is showing, and how strongly, is not the document's at all.
It is a viewing aid, like soloing a row, so it lives in the editor's own state
— `[:ui :trace]`, a set of face symbols switched on and one opacity — and is
never keyed, saved or exported. A face is the same face wherever it is placed,
so one switch shows it in the take it is placed in AND in its own tab, which is
where it is drawn over; nothing has to be switched on twice or per placement.
THE TRACE IS THE ONE FACT about which measured frame a head reads. `:continuous`
reads the frame it is on, `:keys` jumps to each trace frame's measured head and
@ -29,6 +28,10 @@
(def origins [:continuous :keys :start])
(def ^:const opacity-default
"How strongly a switched-on photo draws until someone moves the slider."
0.5)
(defn of
"The head's trace, a head that has none being one that moves freely."
[head]
@ -117,31 +120,38 @@
(sort-by (comp str key) (get-in clip [:symbols sid :nodes]))))]
(vec (walk sid []))))
(defn underlay-at
"The underlay in force at the instance at row path `path` from symbol `sid`:
the nearest one set on it or above it, `:own?` saying which. Nil when none is."
[clip sid path]
(:u (reduce (fn [{:keys [sid u]} id]
(let [n (get-in clip [:symbols sid :nodes id])]
{:sid (:of n)
:u (if-let [own (:underlay n)]
(assoc own :own? true)
(some-> u (assoc :own? false)))}))
{:sid sid} path)))
(defn traceable-faces
"Every face that can be traced while symbol `sid` is open, each once: `sid`
itself when it is a face — open in its own tab, to be drawn over — and the
faces placed inside it at any depth."
[clip sid]
(into [] (distinct)
(cond->> (map :face (faces clip sid))
(traceable? clip sid) (cons sid))))
(defn showing-for
"The faces showing their footage once `sid` is the open symbol, given the ones
`on` already showing.
OPENING A FACE IS ASKING TO DRAW OVER IT: a symbol has measured footage behind
it only because it was traced from that footage, so its own tab starts with the
footage showing rather than with a switch to be found first. A take or a scene
is the picture itself, and a reference drawn over one would read as part of it,
so nothing is switched on for those. Either way it is switched by hand
afterwards, from the bar above the stage or from a face's own timeline row."
[clip sid on]
(cond-> (set on) (traceable? clip sid) (conj sid)))
(defn shown
"Every face whose footage shows from symbol `sid` down, as `{:path :face
:opacity}`: the row path of the face's instance, the face's symbol, and how
strongly to draw it."
[clip sid]
(letfn [(walk [sid path opacity]
(mapcat (fn [[id n]]
(when (= :instance (:kind n))
(let [path (conj path id)
u (:underlay n)
opacity (if u (when (:on? u) (:opacity u 0.5)) opacity)]
(cond->> (walk (:of n) path opacity)
(and opacity (traceable? clip (:of n)))
(cons {:path path :face (:of n) :opacity opacity})))))
(get-in clip [:symbols sid :nodes])))]
(vec (walk sid [] nil))))
"The faces whose footage is showing while symbol `sid` is open, as `faces` lists
them — the row path of the face's instance from `sid`, and the face — filtered
to the `showing` set.
The open symbol itself is in the list, at the EMPTY path, when it is a face:
that is a face open in its own tab to be drawn over, and the one place tracing
matters most. There is no inheritance to work out and no opacity to carry
because the switch is the face's, not a placement's."
[clip sid showing]
(into [] (filter (comp (set showing) :face))
(cond->> (faces clip sid)
(traceable? clip sid) (cons {:path [] :in sid :face sid}))))

10
frontend/src/arthur/events/playback.cljs
View file

@ -13,6 +13,7 @@
(:require [arthur.audio.mix :as mix]
[arthur.clock :as clock]
[arthur.domain.clip :as clip]
[arthur.domain.trace :as trace]
[arthur.footage.store :as footage]
[re-frame.core :as rf]))
@ -36,7 +37,13 @@
(-> db
(assoc :clip/current id
:clip (select-keys entry [:fps :width :height :audio :display-fps]))
(update :ui merge {:open sid :tabs (if sid [sid] [])})
(update :ui merge
{:open sid :tabs (if sid [sid] [])
;; From scratch, not merged: the faces switched on were another
;; document's, and a face id means nothing in this one.
:trace {:faces (trace/showing-for (:clip entry) sid #{})
:opacity (or (get-in db [:ui :trace :opacity])
trace/opacity-default)}})
(assoc-in [:playback :frame] 0)
(assoc-in [:playback :playing?] false))))
@ -188,6 +195,7 @@
{:db (-> db
(update-in [:ui :tabs] #(if (some #{sid} %) % (conj (vec %) sid)))
(assoc-in [:ui :open] sid)
(update-in [:ui :trace :faces] #(trace/showing-for clip sid %))
(assoc-in [:playback :frame] 0)
(assoc-in [:playback :playing?] false))
::pause! nil

6
frontend/src/arthur/events/project.cljs
View file

@ -630,12 +630,6 @@
(fn [db [_ sid value]]
(edit/edit db #(assoc-in % [:symbols sid :nodes :head :trace] value))))
;; Whether one instance shows its face's footage under it. Not keyed: it is a
;; drawing aid, not part of the picture.
(rf/reg-event-db
::set-underlay
(fn [db [_ sid id underlay]]
(edit/edit db #(assoc-in % [:symbols sid :nodes id :underlay] underlay))))
(rf/reg-event-db
::set-segment-interp

26
frontend/src/arthur/events/ui.cljs
View file

@ -49,6 +49,32 @@
(= on #{path}) #{}
:else #{path})))))
(rf/reg-event-db
::trace-face
;; Showing the footage under a face is a viewing aid, like solo: editor state
;; rather than the document, so it is not an undo step, does not travel to a
;; collaborator and cannot reach an export. Per FACE and not per placement — a
;; face is the same face wherever it is placed, and it is the face being traced
;; — so one switch shows it in the take and in its own tab both.
(fn [db [_ face]]
(update-in db [:ui :trace :faces]
#(if (contains? % face) (disj % face) (conj (set %) face)))))
(rf/reg-event-db
::trace-faces
;; Every face the open symbol has, from the bar above the stage: switched on
;; unless they all already are, which is the one gesture a person wants when
;; there is exactly one face and when there are five.
(fn [db [_ faces]]
(let [faces (set faces)
on (set (get-in db [:ui :trace :faces]))]
(assoc-in db [:ui :trace :faces]
(if (every? on faces) (reduce disj on faces) (into on faces))))))
(rf/reg-event-db
::trace-opacity
(fn [db [_ opacity]] (assoc-in db [:ui :trace :opacity] opacity)))
(defn- where-new-goes
"The row path, from the open symbol down, of the symbol a new thing goes into:
INSIDE the selected instance, or BESIDE any other selected node, or at the top

20
frontend/src/arthur/subs/render.cljs
View file

@ -24,6 +24,7 @@
(rf/reg-sub ::sliding (fn [db _] (get-in db [:ui :sliding])))
(rf/reg-sub ::gesture (fn [db _] (get-in db [:ui :gesture])))
(rf/reg-sub ::solo (fn [db _] (get-in db [:ui :solo (get-in db [:ui :open])])))
(rf/reg-sub ::tracing (fn [db _] (get-in db [:ui :trace])))
(rf/reg-sub
::clip
@ -163,12 +164,19 @@
:<- [::store]
:<- [::open]
:<- [::solo]
(fn [[id document store open solo] _]
;; What `ui/underlay` needs to paint the footage under the faces being
;; traced, besides the resolver that says where they went. A face outside
;; every soloed row is not on stage, so neither is its footage.
:<- [::tracing]
(fn [[id document store open solo {:keys [faces opacity]}] _]
;; What `ui/underlay` needs to paint the footage of the faces being traced,
;; besides the resolver that says where they went. A face outside every soloed
;; row is not on stage, so neither is its footage.
(let [solo (filter #(placed? document open %) solo)]
{:document document :store store
:footage-id (:footage-id (footage/entry id))
:traces (cond->> (when document (trace/shown document open))
(seq solo) (filterv (fn [{:keys [path]}] (some #(= % (take (count %) path)) solo))))})))
:opacity (or opacity trace/opacity-default)
:traces (cond->> (when document (trace/shown document open faces))
(seq solo) (filterv (fn [{:keys [path]}]
;; The open symbol's own face is at the
;; empty path: it is not under any row, so
;; soloing a row cannot hide it.
(or (empty? path)
(some #(= % (take (count %) path)) solo)))))})))

37
frontend/src/arthur/ui/palette.cljs
View file

@ -11,9 +11,18 @@
Slot 0 is the background, which is why it is shown and not selectable: a
polygon filled with index 0 is invisible against a stage cleared to index 0, so
offering it as a fill is offering a shape that vanishes on creation."
offering it as a fill is offering a shape that vanishes on creation.
The footage switch is here too, because it is the same kind of thing as the
tone and the tool: something you set before you draw and leave alone, in one
place whatever is selected. In the inspector it was a section that appeared
only once the right row had been found — so the way to see the footage you are
tracing depended on what you had clicked, which is not a thing anyone can be
expected to learn."
(:require [arthur.domain.palette :as pal]
[arthur.domain.trace :as trace]
[arthur.events.ui :as ui]
[arthur.subs.render :as render]
[arthur.subs.ui :as sub]
[re-frame.core :as rf]))
@ -34,6 +43,31 @@
:disabled (not pick)
:on-click #(rf/dispatch [::ui/set-tone slot-tone])}]))
(defn- tracing
"The footage under the faces the open symbol has, on or off and how strongly.
ONE SWITCH FOR THE FACES THAT ARE HERE. A face's footage is the face's, not a
placement's, so there is nothing to inherit and nothing to set twice; with
several faces in a take the box says how many are showing and switches the rest
on, and one face alone is switched from its own timeline row."
[]
(let [clip @(rf/subscribe [::render/clip])
open @(rf/subscribe [::render/open])
{:keys [faces opacity]} @(rf/subscribe [::render/tracing])
here (trace/traceable-faces clip open)
on (filterv (set faces) here)]
(when (seq here)
[:<>
[:label.dim {:title (str "show the footage these faces were traced from, over the "
"picture · a reference, never exported")}
[:input {:type "checkbox" :checked (= (count on) (count here))
:on-change #(rf/dispatch [::ui/trace-faces here])}]
(str " footage" (when (< 1 (count here)) (str " " (count on) "/" (count here))))]
[:input.trace-opacity
{:type "range" :min 0 :max 1 :step 0.05 :title "how strongly the footage draws"
:value (or opacity trace/opacity-default) :disabled (empty? on)
:on-change #(rf/dispatch [::ui/trace-opacity (js/parseFloat (.. % -target -value))])}]])))
(defn bar []
(let [tone @(rf/subscribe [::sub/tone])
tool @(rf/subscribe [::sub/tool])
@ -42,6 +76,7 @@
[:div.swatches (doall (map #(swatch % tone) (range slots)))]
[:span.dim (name tone)]
[:span {:style {:flex 1}}]
[tracing]
(if (= :polygon tool)
[:<>
[:span.dim (str (quot (count draft) 2) " points")]

102
frontend/src/arthur/ui/params.cljs
View file

@ -236,21 +236,32 @@
;; ---------------------------------------------------------------------------
;; tracing a face
;;
;; Two owners in one section, and the heading says whose each is. Showing the
;; footage and how strongly is this INSTANCE's, a drawing aid that is not keyed,
;; and it covers every face at or below it — so a take shows its faces' footage.
;; The trace keys and the origin are the FACE's — its symbol's head — so they are
;; the same in every placement of it. See `domain/trace`.
;; THE FACE'S OWN FACTS ONLY, and both of them are keyed to the face rather than
;; to a placement of it: which of its frames its drawings were made over, and
;; what its origin does between those frames. See `domain/trace`.
;;
;; Whether the footage is SHOWING is deliberately not here. It is a viewing aid
;; and it belongs with the other things you look through rather than edit, on the
;; bar above the stage — where it is in the same place whatever is selected,
;; instead of appearing in the inspector only once the right row has been found.
(defn- face-trace [face]
(defn- trace-keys
"The face's trace keys and origin. `frame` is the frame of the FACE that the
playhead is over, and `seek!` goes to one of its frames — both of which depend
on whether the face is open in its own tab or placed in what is."
[face frame seek!]
(let [t (trace/of (get-in @(rf/subscribe [::render/clip]) [:symbols face :nodes :head]))
{:keys [frame time]} @(rf/subscribe [::sub/selected-local])
put #(rf/dispatch [::project/set-trace face %])]
put #(rf/dispatch [::project/set-trace face %])
key? (boolean (some #{frame} (:frames t)))]
[:<>
[:div.row {:style {:margin "5px 0"}}
[:button {:disabled (nil? frame)
:on-click #(put (trace/toggle-frame t frame))}
(if (some #{frame} (:frames t)) "remove trace key" "trace key here")]]
;; A DEAD BUTTON WITH NO REASON GIVEN is what reads as the feature not
;; working. There is no frame of this face under the playhead, so say that
;; rather than greying out the one control in the section.
(if (nil? frame)
[:span.dim "move the playhead over this face to key it"]
[:button {:on-click #(put (trace/toggle-frame t frame))}
(if key? (str "remove trace key at " frame) (str "trace key at " frame))])]
(when (seq (:frames t))
[:div.row
[:span.dim "keys"]
@ -258,9 +269,8 @@
(for [f (:frames t)]
^{:key f}
[:button {:class (when (= frame f) "on")
:disabled (nil? time)
:on-click #(rf/dispatch [::pb/seek (js/Math.round
(+ (:at time) (/ f (:rate time))))])}
:disabled (nil? seek!)
:on-click #(seek! f)}
(str f)]))])
[:div.row {:style {:margin-top "5px"}}
[:span.dim "origin"]
@ -268,36 +278,38 @@
(for [[o label] (map vector trace/origins ["continuous" "at keys" "start"])]
^{:key o}
[:button {:class (when (= o (:origin t)) "on")
:title (case o
:continuous "the head reads the frame it is on"
:keys "the head jumps to each trace key and holds it"
:start "the head holds frame 0 forever")
:on-click #(put (assoc t :origin o))}
label]))]]))
(defn- tracing-section [[sid id n] faces]
(defn- tracing-section
"`face` is the face these facts belong to, `faces` the faces placed inside it
to offer as somewhere to go next, and `path` the row path `faces` are under."
[face faces path]
(let [clip @(rf/subscribe [::render/clip])
open @(rf/subscribe [::render/open])
[_ _ _ path] @(rf/subscribe [::sub/selection])
path (or path [id])
{:keys [on? opacity own?] :or {opacity 0.5} :as u} (trace/underlay-at clip open path)
show #(rf/dispatch [::project/set-underlay sid id (merge {:on? (boolean on?) :opacity opacity} %)])]
[section (str "tracing · " (name (:of n)))
[:div.row
[:label.dim [:input {:type "checkbox" :checked (boolean on?)
:on-change #(show {:on? (.. % -target -checked)})}]
" footage"
(when (and u (not own?)) " · as above")]
[:input {:type "range" :min 0 :max 1 :step 0.05 :value opacity
:disabled (not on?)
:on-focus #(rf/dispatch [::history/hold])
:on-blur #(rf/dispatch [::history/settle])
:on-change #(show {:opacity (js/parseFloat (.. % -target -value))})}]]
(when (trace/traceable? clip (:of n)) [face-trace (:of n)])
own? (= face @(rf/subscribe [::render/open]))
;; The face's OWN frame, which is the playhead itself when the face is
;; the open symbol, and the selected placement's local frame when it is
;; placed in it. `time` maps the open symbol's frames to that
;; placement's, so seeking to one of the face's frames is a conversion.
{:keys [frame time]} (when-not own? @(rf/subscribe [::sub/selected-local]))
frame (if own? @(rf/subscribe [::playback/frame]) frame)
seek! (cond own? #(rf/dispatch [::pb/seek %])
time #(rf/dispatch [::pb/seek (js/Math.round
(+ (:at time) (/ % (:rate time))))]))]
[section (str "tracing · " (name face))
(when (trace/traceable? clip face) [trace-keys face frame seek!])
(when (seq faces)
[:div.row {:style {:margin-top "5px"}}
[:span.dim "faces"]
(doall
(for [{p :path in :in face :face} faces]
(for [{p :path in :in f :face} faces]
^{:key (str p)}
[:button {:on-click #(rf/dispatch [::ui/select [:node in (peek p) (into path p)]])}
(name face)]))])]))
(name f)]))])]))
;; ---------------------------------------------------------------------------
;; a symbol
@ -416,17 +428,29 @@
(defn view []
(let [clip @(rf/subscribe [::render/clip])
open @(rf/subscribe [::render/open])
selection @(rf/subscribe [::sub/selection])
node @(rf/subscribe [::sub/selected-node])
tracked? (seq (owners clip))]
tracked? (seq (owners clip))
;; The face the tracing section is about: the SELECTED PLACEMENT's symbol,
;; or, when the selection is not a placement or there is none, the OPEN
;; symbol — which is the face itself when a face is open to be drawn over.
;; That last case had no section at all before, and it is the one where
;; the keys are actually being set.
placed (:of (peek node))
face (or placed (when (trace/traceable? clip open) open))
faces (when face (trace/faces clip face))
;; Where that face sits, as a row path from the open symbol, so the faces
;; inside it can be selected by their own rows. A selection made on the
;; stage has no path and names a node directly in the open symbol; the
;; open symbol itself is at no path at all.
path (if placed (or (nth selection 3 nil) [(second node)]) [])]
[:section.pane.params
[:div.pane-head "inspector"]
[:div {:style {:min-height 0}}
[clip-section]
(when node [node-section node])
(when-let [of (:of (peek node))]
(let [faces (trace/faces clip of)]
(when (or (trace/traceable? clip of) (seq faces))
[tracing-section node faces])))
(when (and face (or (trace/traceable? clip face) (seq faces)))
[tracing-section face faces path])
(when (= :symbol (first selection)) [symbol-section (second selection)])
(when tracked? [tracking-section])]]))

25
frontend/src/arthur/ui/player.cljs
View file

@ -101,8 +101,9 @@
(some-> @tracker ratom/dispose!)
(reset! tracker
(ratom/run!
(let [was (:resolver @snapshot)
now @(rf/subscribe [::render/shown])]
(let [{was :resolver was-u :underlay} @snapshot
now @(rf/subscribe [::render/shown])
u @(rf/subscribe [::render/underlay])]
(reset! snapshot
{:resolver now
:palette @(rf/subscribe [::render/palette])
@ -111,12 +112,23 @@
:frames @(rf/subscribe [::render/frames])
:width @(rf/subscribe [::sub/width])
:height @(rf/subscribe [::sub/height])
:underlay @(rf/subscribe [::render/underlay])
:underlay u
:frame @(rf/subscribe [::sub/frame])
:playing? @(rf/subscribe [::sub/playing?])})
;; A new resolver means a new scene or a new palette, and neither
;; moves the playhead — so nothing else would ask for a redraw.
(when-not (identical? was now) (repaint!))))))
;;
;; Switching a face's footage on wants the same redraw for the same
;; reason, and it needs asking for SEPARATELY: it is a viewing aid
;; in the editor's own state, so it changes what the canvases should
;; show without touching the frame number OR the resolver. The
;; document and the store are left out of the comparison because
;; they are what a new resolver already means.
;; Comparing the whole map is as cheap as picking fields out of it:
;; the document and the store it carries are the same OBJECTS unless
;; the resolver changed too, and that is tested first.
(when-not (and (identical? was now) (= was-u u))
(repaint!))))))
(defn set-canvas! [el]
(swap! state assoc :canvas el)
@ -146,7 +158,7 @@
rasterised before the next frame is asked for."
[f]
(let [{:keys [canvas]} @state
{:keys [resolver palette ramp width height underlay]} @snapshot]
{:keys [resolver palette ramp width height underlay playing?]} @snapshot]
(when (and canvas resolver width height)
;; User Timing, so a profile in the DevTools performance panel has named
;; spans in the Timings track instead of a wall of anonymous frames. Three
@ -163,7 +175,8 @@
(raster/draw-ops! ops)))
(js/performance.mark "arthur/blit:start")
(canvas/blit! canvas ras ramp)
(underlay/paint! (assoc underlay :width width) resolver repaint!))
(underlay/paint! (assoc underlay :width width :playing? playing?)
resolver repaint!))
(js/performance.measure "arthur/resolve+draw" "arthur/paint:start" "arthur/blit:start")
(js/performance.measure "arthur/paint" "arthur/paint:start")
;; User Timing entries otherwise accumulate forever in the browser's

21
frontend/src/arthur/ui/timeline.cljs
View file

@ -22,6 +22,7 @@
happen and not where they happen again."
(:require [clojure.string :as str]
[arthur.domain.node :as node]
[arthur.domain.trace :as trace]
[arthur.events.playback :as pb]
[arthur.events.ui :as ui]
[arthur.subs.playback :as playback]
@ -266,7 +267,7 @@
(memoize (fn [_selection] (fn [el] (some-> el (.scrollIntoView #js {:block "nearest"}))))))
(defn- label-cell [{:keys [path depth label kind node-kind select expandable? expanded? of via]}
selection over solo]
selection over solo tracing]
(let [node? (= :node kind)
[over-path where] @over]
[:div (cond-> {:class (str "tl-label" (when (and select (= select selection)) " on")
@ -323,6 +324,17 @@
(when expandable? (if expanded? "▾" "▸"))]
[:span.name label]
(when node? [:span.kind (if via (str "· in " via) (str "·" (name node-kind)))])
;; A face's row is where its own footage is switched on, next to solo
;; because the two are the same kind of thing: what this row shows, here,
;; now, and nothing the picture keeps. The bar above the stage does all of
;; them at once; this is how one face out of a take is singled out.
(when (contains? (:faces tracing) of)
[:button {:class (str "tl-trace" (when (contains? (:on tracing) of) " on"))
:title "show the footage this face was traced from"
:on-click (fn [^js e]
(.stopPropagation e)
(rf/dispatch [::ui/trace-face of]))}
"T"])
(when (= :instance node-kind)
[:button {:class (str "tl-solo" (when (contains? solo path) " on"))
:title "show only this on the stage (⇧ for more than one)"
@ -402,6 +414,11 @@
drop @(rf/subscribe [::sub/drop])
solo (set @(rf/subscribe [::render/solo]))
open @(rf/subscribe [::render/open])
;; Which rows offer a footage switch, and which of them are switched
;; on. A set rather than a lookup per row: the answer is the same for
;; every placement of a face, because the switch is the face's.
tracing {:faces (set (trace/traceable-faces clip open))
:on (set (:faces @(rf/subscribe [::render/tracing])))}
;; Where a drag out of the pool would land, as a row of its own at the
;; top of its section: its own length, starting on the frame it would
;; start on. The stage's drop shows it too, at the playhead.
@ -441,7 +458,7 @@
(doall (for [row visible]
(with-meta (if (= :section (:kind row))
[:div.tl-label.tl-section (:label row)]
[label-cell row selection over solo])
[label-cell row selection over solo tracing])
{:key (str (:path row))})))]
[:div.tl-tracks
{:on-drag-enter (fn [^js e] (when (drag/accepts?) (.preventDefault e)))

142
frontend/src/arthur/ui/underlay.cljs
View file

@ -2,18 +2,27 @@
"The footage a face is traced over, on its own canvas above the stage.
A REFERENCE, NOT OUTPUT. The tracing still never enters the indexed raster, so
it cannot reach an export, and it is drawn OVER the picture at the instance's
it cannot reach an export, and it is drawn OVER the picture at the face's
opacity rather than under it, because the raster clears to an opaque ground.
The canvas is the stage's size on screen, not the raster's, so a 1280px still
is not squeezed through a 320px stage on its way to being seen.
Painted by `ui/player` straight after each frame, from the same snapshot, so
it moves with the face it registers to — see `domain/trace/photo-matrix`."
it moves with the face it registers to — see `domain/trace/photo-matrix`.
THREE THINGS KEEP IT STEADY, because a still is decoded asynchronously and the
frame it belongs to is already on screen by the time it arrives: the cache drops
the stills used longest ago rather than emptying itself, a face keeps showing
the still it last showed until its next one has decoded, and the stills a little
way ahead of the playhead are asked for before they are needed. Each is a
separate cause of the same symptom — the tracing blinking out for a frame or two
— and none of them covers the others: the hold is what survives a seek, the
reading ahead is what keeps playback from being a frame behind for good."
(:require [arthur.domain.symbol :as symbol]
[arthur.domain.trace :as trace]
[arthur.flow.ingest :as ingest]))
(defonce ^:private state (atom {:canvas nil :urls {} :images {}}))
(defonce ^:private state (atom {:canvas nil :urls {} :images {} :order [] :last {}}))
(defn set-canvas! [el] (swap! state assoc :canvas el))
@ -37,45 +46,134 @@
scrubbing a face with no trace keys asks for every frame of the take."
48)
(def ^:private ^:const ahead
"Frames past the playhead whose stills are asked for while the current one is
drawn. A still cannot be decoded in the animation frame that wants it, so
playing a continuously traced face would otherwise always be a frame or two
behind its own footage."
8)
(defn- decoded
"`img` if the still has arrived and can be drawn."
[^js img]
(when (and (.-complete img) (pos? (.-naturalHeight img))) img))
(defn- touch!
"Hold `img` under `url` as the most recently used still, dropping the ones used
longest ago once there are more than `kept`.
LEAST RECENTLY USED, touched when it arrives and again on every frame it is
DRAWN, which is what makes it safe without a list of exceptions: the still on
screen is the newest thing in the cache and cannot be the one dropped — a still
held at a trace key for a hundred frames included. Read-and-not-drawn does not
touch, because a still read only to be warmed was just inserted anyway, and
reordering the whole list ten times per face per frame in the draw loop is the
kind of allocation `ui/player` exists to avoid. What this replaces emptied the
cache on the frame it filled, so every still on screen had to be fetched and
decoded again: a blink every `kept` frames of a scrub, and with two faces and a
full cache a permanent one."
[url img]
(swap! state
(fn [{:keys [images order] :as s}]
(let [images (assoc images url img)
order (conj (into [] (remove #{url}) order) url)
over (- (count order) kept)]
(if (pos? over)
(assoc s :images (apply dissoc images (subvec order 0 over))
:order (subvec order over))
(assoc s :images images :order order))))))
(defn- image
"The still at `url`, or nil until it has loaded."
"The still at `url`, or nil until it has loaded. Asks for it the first time it
is wanted; `on-ready` paints again once it is there.
A still that fails is remembered as having failed, so a broken URL is one
console line and one request rather than one of each per animation frame."
[url on-ready]
(let [img (or (get-in @state [:images url])
(let [img (js/Image.)]
(when url
(let [held (get-in @state [:images url])]
(cond
(= :failed held) nil
held (decoded held)
:else (let [img (js/Image.)]
(set! (.-onload img) on-ready)
(set! (.-onerror img)
(fn [_]
(swap! state assoc-in [:images url] :failed)
(js/console.error "arthur: a tracing still did not load" url)))
(set! (.-src img) url)
(swap! state update :images
#(assoc (if (< (count %) kept) % {}) url img))
img))]
(when (and (.-complete img) (pos? (.-naturalHeight img))) img)))
(touch! url img)
nil)))))
(defn- still
"The URL of the still showing the face's own frame `p`. The manifest is the
whole footage's and `p` is an index into the analysed range, so the range's
start is where the face's frame 0 was filmed."
[us start p]
(get us (+ start p)))
(defn- warm!
"Ask for the stills the next `ahead` frames will want. Held trace frames
collapse to the one still, so a face traced at keys asks for almost nothing.
WHILE PLAYING ONLY, because that is the only time the next frame is the one
after this one. A scrub asks for a different `ahead` frames at every step, so
reading ahead through one would be `ahead` requests per step for stills the
pointer has already gone past — and a cache thrashed by its own guesses."
[us start t lf on-ready]
(doseq [p (distinct (map #(trace/photo-frame t %) (range (inc lf) (+ lf 1 ahead))))]
(image (still us start p) on-ready)))
(defn paint!
"Draw every switched-on underlay on the frame `resolver` last resolved. It
"Draw every switched-on face's footage on the frame `resolver` last resolved. It
says where each face's head went and which of its frames it was on, so this
reads the frame rather than resolving it again. `on-ready` is called when a
still or a manifest that was missing arrives, to paint again."
[{:keys [document store footage-id traces width]} resolver on-ready]
[{:keys [document store footage-id traces opacity width playing?]} resolver on-ready]
(when-let [^js canvas (:canvas @state)]
(let [ctx (.getContext canvas "2d")
zoom (/ (.-width canvas) width)]
(.setTransform ctx 1 0 0 1 0 0)
(.clearRect ctx 0 0 (.-width canvas) (.-height canvas))
;; What each face is showing, forgotten for the faces switched off. Before
;; the early exits, so switching them all off forgets all of them.
(swap! state update :last select-keys (map :path traces))
(when-let [us (and (seq traces) footage-id (urls footage-id on-ready))]
(let [start (first (get-in document [:analysis :range] [0]))]
(doseq [{:keys [path face opacity]} traces
:let [at (conj path :head)
(let [start (first (get-in document [:analysis :range] [0]))
;; Read once: each face writes only its own entry below, and what it
;; was showing is what it falls back to. A face that has been off
;; stage for a few frames still has the still it went away with.
was (:last @state)]
;; ONE OPACITY for all of them, set once: how strongly the reference
;; draws is a property of looking at the stage, not of a face.
(set! (.-globalAlpha ctx) opacity)
(doseq [{:keys [path face]} traces]
(let [at (conj path :head)
world (symbol/world-of resolver at)
frame (symbol/frame-of resolver at)]
:when (and world (number? frame))
:let [head (get-in document [:symbols face :nodes :head])
p (trace/photo-frame (trace/of head) (js/Math.floor frame))
img (some-> (get us (+ start p)) (image on-ready))
(when (and world (number? frame))
(let [head (get-in document [:symbols face :nodes :head])
t (trace/of head)
lf (js/Math.floor frame)
want (trace/photo-frame t lf)
;; THE STILL IT HAS, not nothing. A frame whose still is
;; still decoding keeps the one before it — a reference a
;; frame stale, registered where that frame's face was,
;; rather than a face with its footage blinking off.
[img url p] (or (let [u (still us start want)]
(when-let [i (image u on-ready)] [i u want]))
(let [{:keys [url p]} (get was path)]
(when-let [i (image url on-ready)] [i url p])))
m (when img
(trace/photo-matrix world head store p (.-naturalHeight img)))]
:when m]
(set! (.-globalAlpha ctx) opacity)
(when m
;; Drawn, so it is the newest still in the cache and what
;; this face falls back to while its next one decodes.
(touch! url img)
(swap! state assoc-in [:last path] {:url url :p p})
(.setTransform ctx
(* zoom (aget m 0)) (* zoom (aget m 1))
(* zoom (aget m 2)) (* zoom (aget m 3))
(* zoom (aget m 4)) (* zoom (aget m 5)))
(.drawImage ctx img 0 0)))))))
(.drawImage ctx img 0 0))
(when playing? (warm! us start t lf on-ready)))))))))))

56
frontend/test/arthur/domain/trace_test.cljs
View file

@ -70,39 +70,47 @@
"a continuous head carries the held photo along with it")))
(defn- wrapped
"Face-1's take placed, moved, inside a symbol :wrap, with `underlays` on the
instances named."
[{:keys [outer inner]}]
(-> @frozen
(assoc-in [:symbols :wrap] {:id :wrap :frames 200
:nodes {:m (cond-> {:id :m :kind :instance :of :main :z "a0"
:channels {[:xform :pos] {:animated? false
:value [30 -10]}}}
outer (assoc :underlay outer))}})
(cond-> inner (assoc-in [:symbols :main :nodes :face-1 :underlay] inner))))
"Face-1's take placed, moved, inside a symbol :wrap."
[]
(assoc-in @frozen [:symbols :wrap]
{:id :wrap :frames 200
:nodes {:m {:id :m :kind :instance :of :main :z "a0"
:channels {[:xform :pos] {:animated? false :value [30 -10]}}}}}))
(deftest an-underlay-covers-the-faces-below-it-and-the-nearest-decides
(is (= [{:path [:m :face-1] :face :face-1 :opacity 0.3}]
(trace/shown (wrapped {:outer {:on? true :opacity 0.3}}) :wrap))
"switched on at the take, its face shows")
(is (= [] (trace/shown (wrapped {:outer {:on? true} :inner {:on? false}}) :wrap))
"and the face can still be switched off inside it")
(is (= [{:path [:m :face-1] :face :face-1 :opacity 0.8}]
(trace/shown (wrapped {:inner {:on? true :opacity 0.8}}) :wrap)))
(let [c (wrapped {:outer {:on? true :opacity 0.3}})]
(is (= {:on? true :opacity 0.3 :own? false} (trace/underlay-at c :wrap [:m :face-1])))
(is (= {:on? true :opacity 0.3 :own? true} (trace/underlay-at c :wrap [:m])))
(is (nil? (trace/underlay-at @frozen :main [:face-1])))))
(deftest a-face-switched-on-shows-wherever-it-is-placed
(let [c (wrapped)]
(is (= [{:path [:m :face-1] :in :main :face :face-1}] (trace/shown c :wrap #{:face-1}))
"a face inside a take inside a symbol, at the row path it is at")
(is (= [] (trace/shown c :wrap #{})) "and nothing when it is switched off")
(is (= [{:path [:face-1] :in :main :face :face-1}] (trace/shown c :main #{:face-1}))
"the same switch, one symbol down")
(is (= [{:path [] :in :face-1 :face :face-1}] (trace/shown c :face-1 #{:face-1}))
"the face open in its own tab is at no path at all — it IS the stage")
(is (= [] (trace/shown c :wrap #{:main}))
"a symbol that is not a face has no footage of its own to show")))
(deftest the-faces-that-can-be-traced-are-listed-once-each
(is (= [:face-1] (trace/traceable-faces (wrapped) :wrap)))
(is (= [:face-1] (trace/traceable-faces @frozen :main)) "the take it was frozen into")
(is (= [:face-1] (trace/traceable-faces @frozen :face-1)) "itself, open to draw over"))
(deftest a-face-opened-to-be-drawn-over-starts-with-its-footage-showing
(let [c (wrapped)]
(is (= #{:face-1} (trace/showing-for c :face-1 #{})) "the face's own tab")
(is (= #{} (trace/showing-for c :main #{}))
"and not the take it is placed in, which is the picture itself")
(is (= #{:face-1} (trace/showing-for c :main #{:face-1}))
"one already switched on stays on wherever you go")))
(deftest a-take-lists-the-faces-in-it
(is (= [{:path [:face-1] :in :main :face :face-1}] (trace/faces @frozen :main)))
(is (= [{:path [:m :face-1] :in :main :face :face-1}] (trace/faces (wrapped {}) :wrap)))
(is (= [{:path [:m :face-1] :in :main :face :face-1}] (trace/faces (wrapped) :wrap)))
(is (= [] (trace/faces @frozen :face-1))))
(deftest the-resolver-says-where-a-nested-head-went-on-its-last-frame
;; The same answer as `nest/placement`, which walks and resolves the path all
;; over again — the resolver has it already, from drawing the frame.
(let [c (wrapped {})
(let [c (wrapped)
r (clip/resolver c @store pal/index-of :wrap)
path [:m :face-1 :head]]
(doseq [f [0 17 60]]

11
static/arthur/app.css
View file

@ -435,6 +435,9 @@ button.share-button:hover, button.share-button.on { filter: brightness(1.1); }
.tab:hover .close, .tab.on .close { visibility: visible; }
.tab .close:hover { background: var(--hair); color: var(--fg); }
.trace-opacity { width: 64px; }
.palette-bar label.dim { display: inline-flex; align-items: center; gap: 2px; white-space: nowrap; }
.palette-bar {
display: flex;
align-items: center;
@ -650,10 +653,10 @@ button.share-button:hover, button.share-button.on { filter: brightness(1.1); }
.tl-twist:disabled { opacity: 0; cursor: default; }
.tl-delete { margin-left: auto; padding: 0 4px; border: 0; background: none; color: var(--dim); }
.tl-delete:hover { color: var(--fg); }
.tl-solo { margin-left: auto; padding: 0 4px; border: 0; background: none; color: var(--dim); font-size: 10px; }
.tl-solo:hover { color: var(--fg); }
.tl-solo.on { color: var(--sel); font-weight: 600; }
.tl-solo + .tl-delete { margin-left: 0; }
.tl-solo, .tl-trace { margin-left: auto; padding: 0 4px; border: 0; background: none; color: var(--dim); font-size: 10px; }
.tl-solo:hover, .tl-trace:hover { color: var(--fg); }
.tl-solo.on, .tl-trace.on { color: var(--sel); font-weight: 600; }
.tl-solo + .tl-delete, .tl-trace + .tl-solo { margin-left: 0; }
/* The ruler and the corner above the labels stay at the top of the body while
the rows scroll under them: the ruler is the scrubber, and scrolling down to