noe/arthur
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Add instanced 8625 stage with independent audio controls

Browse source
This commit is contained in:
Olive Vaughn 2026-09-28 15:05:04 -04:00
parent a611b86c0d
commit ffb95543a3
15 changed files with 507 additions and 48 deletions
Download patch file Download diff file Expand all files Collapse all files

33
docs/animation-model.md
View file

@ -495,6 +495,39 @@ reading head over the same channel. The resolver keys its caches by the instance
path, not by node id — which is a detail of `Making it fast` below, and the one path, not by node id — which is a detail of `Making it fast` below, and the one
place symbol nesting is not free. place symbol nesting is not free.
### Audio placements and controls
Sound is placed on a timeline as a separate `:audio` node. It uses the same
`:span`, `:time`, and channel representation as a drawn node. A `:linked-to` id
records which picture instance it was placed with; it does not force the two
spans or source in-points to match.
```clojure
{:id :voice-right :kind :audio :parent :root :z "a4"
:linked-to :right
:source {:footage "f8cace9e-..."}
:span [48 260]
:time {:mode :map :at 48 :in 0 :rate 1}
:channels {[:audio :gain]
{:animated? true :interp :linear
:keys {48 0.0, 60 1.0, 245 1.0, 259 0.0} :over []}}}
```
`[:audio :gain]`, `[:audio :pan]`, and `[:audio :rate]` are ordinary scalar
channels. They may be framed, keyed, or dense; numeric keyed channels can ramp
linearly. The time map sets the placement's base source rate, and
`[:audio :rate]` multiplies it. Audio is mixed from the referenced immutable
footage when the clip opens. The mix is derived output; the saved document holds
the nodes and channel keys, not another audio file. One audio element plays that
mix and remains the clock for both sound and picture.
This is also the boundary for a future control surface. A control has a stable
target, such as a feature's `:verts` setting or an audio node's
`[:audio :gain]` channel. The UI and a MIDI binding can address both through the
same control interface. Their update costs differ: gain can be keyed over time;
changing the number of lip vertices changes topology and must regenerate its
dense geometry. A topology setting cannot be treated as a per-frame gain curve.
## Evaluating a frame ## Evaluating a frame
```clojure ```clojure

27
frontend/README.md
View file

@ -112,6 +112,33 @@ The demo scene itself is `src/arthur/demo/scene.edn`. Both the synthetic take
and real footage use `src/arthur/flow/take.cljs` for the measurement order and and real footage use `src/arthur/flow/take.cljs` for the measurement order and
`src/arthur/flow/freeze.cljs` for the landmark-to-channel conversion. `src/arthur/flow/freeze.cljs` for the landmark-to-channel conversion.
**stage 8625** loads the locally saved `IMG_8625.MOV` project and places its
post-processed timeline twice. The stage layout is
`src/arthur/demo/stage_8625.edn`: the right picture and sound start at frame 48,
and the two pictures overlap slightly in stage space. Audio has its own timeline
nodes, linked to the picture instances but with independent spans and gain
channels. The right sound swells and pans across the stage, then fades out at
frame 260 while its picture continues to
frame 280. The button needs that saved 8625 project in the local server database.
### Projects and the EDN fixtures
The EDN files under `demo/` are authored examples compiled into the frontend.
They seed a clip in memory; the server does not read EDN. Clicking **save** on a
clip without a project id creates a project through `POST /api/projects`, uploads
any missing content-addressed blocks, then writes the clip's addressed leaves
through `PUT /api/projects/<id>`. Each leaf value is Transit JSON inside the
request's ordinary JSON envelope. Python stores those values in JSON columns and
does not need an EDN parser. **open** reads the leaves and blocks and rebuilds
the same ClojureScript clip.
The intended editor creates and changes that in-memory clip directly: a project
browser and **new stage** action, timeline instance placement, node and channel
editors, then the existing save path. EDN remains useful for checked-in examples
and reproducible studies. The current UI has save and open, but no project
browser, blank-stage action, or authoring controls yet; open chooses the most
recent project.
### Real footage ### Real footage
Choose a video in the **footage** file input. The server probes it, re-encodes it Choose a video in the **footage** file input. The server probes it, re-encodes it

119
frontend/src/arthur/audio/mix.cljs Normal file
View file

@ -0,0 +1,119 @@
(ns arthur.audio.mix
"Render independently placed audio tracks into one stage audio clock.
The mix is derived from saved audio track leaves and immutable footage blobs.
The transport still has one audio element, so seeking, rate changes and looping
stay tied to the same clock the picture reads."
(:require [arthur.domain.channel :as ch]
[arthur.domain.clip :as clip]
[arthur.domain.node :as node]))
(defn- wav [^js buffer]
(let [channels (.-numberOfChannels buffer)
frames (.-length buffer)
rate (.-sampleRate buffer)
bytes (js/ArrayBuffer. (+ 44 (* frames channels 2)))
view (js/DataView. bytes)
samples (mapv #(.getChannelData buffer %) (range channels))
peak (reduce max 0
(for [channel samples i (range frames)]
(js/Math.abs (aget channel i))))
level (if (> peak 0.98) (/ 0.98 peak) 1)]
(doseq [[offset word] [[0 "RIFF"] [8 "WAVE"] [12 "fmt "] [36 "data"]]]
(dotimes [i 4] (.setUint8 view (+ offset i) (.charCodeAt word i))))
(.setUint32 view 4 (- (.-byteLength bytes) 8) true)
(.setUint32 view 16 16 true)
(.setUint16 view 20 1 true)
(.setUint16 view 22 channels true)
(.setUint32 view 24 rate true)
(.setUint32 view 28 (* rate channels 2) true)
(.setUint16 view 32 (* channels 2) true)
(.setUint16 view 34 16 true)
(.setUint32 view 40 (* frames channels 2) true)
(dotimes [i frames]
(dotimes [c channels]
(let [sample (* level (aget (get samples c) i))]
(.setInt16 view (+ 44 (* (+ (* i channels) c) 2))
(js/Math.round (* 32767 (max -1 (min 1 sample)))) true))))
(js/URL.createObjectURL (js/Blob. #js [bytes] #js {:type "audio/wav"}))))
(defn- source! [footage-id]
(-> (js/fetch (str "/api/footage/" footage-id))
(.then (fn [response]
(when-not (.-ok response)
(throw (ex-info "audio track's footage is missing"
{:footage footage-id :status (.-status response)})))
(.json response)))
(.then (fn [^js manifest]
(-> (js/fetch (.-audio manifest))
(.then (fn [response]
(when-not (.-ok response)
(throw (ex-info "audio track's blob is missing"
{:footage footage-id :status (.-status response)})))
(.arrayBuffer response)))
(.then (fn [bytes]
(let [decoder (js/OfflineAudioContext. 1 1 44100)]
(-> (.decodeAudioData decoder bytes)
(.then (fn [buffer]
[footage-id {:buffer buffer
:fps (.-fps manifest)}])))))))))))
(defn- automate! [^js param channel start end fps factor default store]
(let [channel (or channel (ch/framed default))]
(.setValueAtTime param (* factor (ch/value-at channel start store)) (/ start fps))
(cond
(:dense channel)
(doseq [f (range (inc start) end)]
(.setValueAtTime param (* factor (ch/value-at channel f store)) (/ f fps)))
(:animated? channel)
(doseq [[f v] (sort-by key (:keys channel))
:when (and (> f start) (< f end))]
(if (= :linear (:interp channel))
(.linearRampToValueAtTime param (* factor v) (/ f fps))
(.setValueAtTime param (* factor v) (/ f fps)))))))
(defn- render! [document sources store]
(let [fps (:fps document)
frames (clip/frames document)
tracks (filter #(= :audio (:kind %)) (vals (clip/nodes document)))
output (js/OfflineAudioContext.
2 (js/Math.ceil (* (/ frames fps) 44100)) 44100)]
(doseq [track tracks]
(let [[start end] (or (:span track) [0 frames])
start (max 0 start)
end (min frames end)
{:keys [buffer fps]} (get sources (get-in track [:source :footage]))
sound (.createBufferSource output)
gain (.createGain output)
pan (.createStereoPanner output)]
(when (< start end)
(set! (.-buffer sound) buffer)
(set! (.-loop sound) (boolean (get-in track [:time :loop?])))
(automate! (.-playbackRate sound)
(get-in track [:channels [:audio :rate]])
start end (:fps document) (or (get-in track [:time :rate]) 1) 1 store)
(automate! (.-gain gain)
(get-in track [:channels [:audio :gain]])
start end (:fps document) 1 1 store)
(automate! (.-pan pan)
(get-in track [:channels [:audio :pan]])
start end (:fps document) 1 0 store)
(.connect sound gain)
(.connect gain pan)
(.connect pan (.-destination output))
(.start sound (/ start (:fps document)) (/ (node/local-frame track start) fps))
(.stop sound (/ end (:fps document))))))
(-> (.startRendering output) (.then wav))))
(defn mix!
"Promise of a mixed WAV URL, or the original URL for a clip without audio
tracks. Each track can be trimmed and faded independently of its linked picture."
([document fallback-url] (mix! document fallback-url nil))
([document fallback-url store]
(let [tracks (filter #(= :audio (:kind %)) (vals (clip/nodes document)))]
(if (empty? tracks)
(js/Promise.resolve fallback-url)
(-> (js/Promise.all
(into-array (map source! (distinct (map #(get-in % [:source :footage]) tracks)))))
(.then (fn [pairs] (render! document (into {} (array-seq pairs)) store))))))))

32
frontend/src/arthur/demo/stage.cljs Normal file
View file

@ -0,0 +1,32 @@
(ns arthur.demo.stage
"A saved take placed twice on a stage. The EDN is the authored layout."
(:require [cljs.reader :as reader]
[shadow.resource :as rc]))
(def layout (reader/read-string (rc/inline "arthur/demo/stage_8625.edn")))
(defn compose [source]
(let [{:keys [name width height frames symbol instances audio]} layout
original (get-in source [:timelines :main])
nodes (into
{:root {:id :root :name "stage" :kind :group :z "a1"}}
(map (fn [{:keys [id name z span at in pos scale]}]
[id {:id id :name name :kind :symbol :of symbol
:parent :root :z z :span span
:time {:mode :map :at at :in in :rate 1}
:channels {[:xform :pos] {:animated? false :value pos}
[:xform :scale] {:animated? false :value scale}}}])
instances))
nodes (into nodes
(map (fn [{:keys [id linked-to z source span at in gain pan]}]
[id {:id id :kind :audio :parent :root :z z
:linked-to linked-to :source source :span span
:time {:mode :map :at at :in in :rate 1}
:channels (cond-> {[:audio :gain] gain}
pan (assoc [:audio :pan] pan))}])
audio))]
(-> source
(assoc :name name :width width :height height
:timelines {:main {:id :main :frames frames :nodes nodes}
symbol (assoc original :id symbol)})
(dissoc :subjects :features :groups))))

32
frontend/src/arthur/demo/stage_8625.edn Normal file
View file

@ -0,0 +1,32 @@
;; A local stage sketch. The source is the saved, post-processed IMG_8625.MOV
;; clip in the project store; its dense channel blocks remain shared by both
;; instances. Positions and timing are authored here in stage pixels and frames.
{:source-project "4379f900-bdd2-409b-acf6-32081f8ce01f"
:source-cid "f8cace9e-4ad3-4796-973c-c62eeebe3d01"
:symbol :sym/face-8625
:name "8625 stage study"
:width 320 :height 200 :frames 280
;; Audio placements are ordinary timeline nodes with channel parameters.
;; :linked-to is an editorial link; their spans and time maps are independent.
:audio
[{:id :voice-left :linked-to :left :z "a3"
:source {:footage "f8cace9e-4ad3-4796-973c-c62eeebe3d01"}
:span [0 280] :at 0 :in 0
:gain {:animated? false :value 1.0}}
{:id :voice-right :linked-to :right :z "a4"
:source {:footage "f8cace9e-4ad3-4796-973c-c62eeebe3d01"}
:span [48 260] :at 48 :in 0
:gain {:animated? true :interp :linear
:keys {48 0.0, 60 1.0, 90 0.35, 115 0.9, 145 0.45,
170 1.0, 195 0.4, 220 0.85, 245 1.0, 259 0.0}
:over []}
:pan {:animated? true :interp :linear
:keys {48 -0.8, 90 -0.8, 130 0.7, 175 0.7, 220 -0.65, 259 0.65}
:over []}}]
:instances
[{:id :left :name "8625 left" :z "a1"
:span [0 280] :at 0 :in 0
:pos [12 38] :scale [0.62 0.62]}
{:id :right :name "8625 right" :z "a2"
:span [48 280] :at 48 :in 0
:pos [108 38] :scale [0.62 0.62]}]}

38
frontend/src/arthur/domain/channel.cljs
View file

@ -92,12 +92,9 @@
(vec (sort (keys ks))))) (vec (sort (keys ks)))))
(defn- check-unimplemented! (defn- check-unimplemented!
"An override layer or a retimed symbol instance must fail LOUDLY rather than "An override layer must fail LOUDLY rather than be ignored.
be ignored.
Both are specified in docs/animation-model.md and neither is built yet Silently dropping an :over layer would present as a hand
(port-plan scope: \"leave the :over field present and empty; leave :symbol out
entirely\"). Silently dropping an :over layer would present as a hand
correction that did not take — a correction the user made once, watched fail, correction that did not take — a correction the user made once, watched fail,
and has no reason to trust again. Nothing can produce one yet, so this can and has no reason to trust again. Nothing can produce one yet, so this can
only fire on a data shape that has run ahead of the code." only fire on a data shape that has run ahead of the code."
@ -172,6 +169,13 @@
;; --------------------------------------------------------------------------- ;; ---------------------------------------------------------------------------
;; the specification ;; the specification
(defn- interpolate [ch f left right]
(let [a (get (:keys ch) left)]
(if (and (= :linear (:interp ch)) right (>= f left) (> right left))
(let [b (get (:keys ch) right)]
(+ a (* (/ (- f left) (- right left)) (- b a))))
a)))
(defn- keyed-at (defn- keyed-at
"The most recent key at or before f, CLAMPED to the first key below it. "The most recent key at or before f, CLAMPED to the first key below it.
@ -180,11 +184,11 @@
frame before it reads that pose rather than having no value. This is not the frame before it reads that pose rather than having no value. This is not the
same question as presence — a part with no value at all is `absent`, which is same question as presence — a part with no value at all is `absent`, which is
a state bit, not an empty key map." a state bit, not an empty key map."
[ks f] [ch f]
(let [fr (sort (keys ks))] (let [fr (sort (keys (:keys ch)))
(if-let [hit (last (take-while #(<= % f) fr))] left (or (last (take-while #(<= % f) fr)) (first fr))
(get ks hit) right (first (drop-while #(<= % f) fr))]
(get ks (first fr))))) (interpolate ch f left right)))
(defn value-at (defn value-at
"Sample a channel at frame f. THE SPECIFICATION — correct, allocating, and "Sample a channel at frame f. THE SPECIFICATION — correct, allocating, and
@ -196,7 +200,7 @@
(not (:animated? ch)) (:value ch) (not (:animated? ch)) (:value ch)
(:dense ch) (dense-at (:dense ch) f store) (:dense ch) (dense-at (:dense ch) f store)
(:keys ch) (let [ks (:keys ch)] (:keys ch) (let [ks (:keys ch)]
(if (empty? ks) absent (keyed-at ks f))) (if (empty? ks) absent (keyed-at ch f)))
:else :else
(throw (ex-info "animated channel has neither :keys nor :dense" {:channel ch}))))) (throw (ex-info "animated channel has neither :keys nor :dense" {:channel ch})))))
@ -272,7 +276,7 @@
:else (bsearch ks f))] :else (bsearch ks f))]
(set! (.-i cur) i') (set! (.-i cur) i')
(get (:keys ch) (nth ks i')))))) (interpolate ch f (nth ks i') (when (< i' last) (nth ks (inc i'))))))))
;; --------------------------------------------------------------------------- ;; ---------------------------------------------------------------------------
@ -307,9 +311,12 @@
(and (map? ch) (:keys ch) (map? (:keys ch)) (not (every? number? (keys (:keys ch))))) (and (map? ch) (:keys ch) (map? (:keys ch)) (not (every? number? (keys (:keys ch)))))
(conj ":keys has a non-numeric frame") (conj ":keys has a non-numeric frame")
(and (map? ch) (:animated? ch) (not (#{:hold nil} (:interp ch)))) (and (map? ch) (:animated? ch) (not (#{:hold :linear nil} (:interp ch))))
(conj (str ":interp " (:interp ch) " — only :hold is implemented; docs/design.md" (conj (str ":interp " (:interp ch) " is not implemented"))
" requires hold of every cut part and tweening reads as puppet software"))
(and (map? ch) (= :linear (:interp ch))
(or (:dense ch) (not (every? number? (vals (:keys ch))))))
(conj ":linear interpolation needs numeric keyed values")
(and (map? ch) (seq (:over ch))) (and (map? ch) (seq (:over ch)))
(conj ":over layers are not implemented (port-plan step 2 scope)") (conj ":over layers are not implemented (port-plan step 2 scope)")
@ -322,4 +329,3 @@
(conj (str ":dense :scale is " (pr-str (:scale (:dense ch))) (conj (str ":dense :scale is " (pr-str (:scale (:dense ch)))
" — a fixed-point scale is a positive number the stored integers" " — a fixed-point scale is a positive number the stored integers"
" were multiplied by")))) " were multiplied by"))))

82
frontend/src/arthur/domain/clip.cljs
View file

@ -19,9 +19,8 @@
structure with the same `:nodes` key that every walk had to be taught about. structure with the same `:nodes` key that every walk had to be taught about.
Now there is one node-holding type — `arthur.domain.timeline` — and a clip holds Now there is one node-holding type — `arthur.domain.timeline` — and a clip holds
a MAP of them. `:kind :symbol` is still unimplemented and this is the shape it a MAP of them. A `:kind :symbol` instance names a timeline in `:timelines`,
was waiting for: an instance names a timeline in `:timelines`, and the resolver and the clip resolver gives each placement its own reading heads.
recurses into a type it already knows how to evaluate.
THE ROOT TIMELINE HAS A RESERVED ID, `:main`, rather than the clip carrying a THE ROOT TIMELINE HAS A RESERVED ID, `:main`, rather than the clip carrying a
pointer to it. A pointer is a field that can be wrong — it can name a timeline pointer to it. A pointer is a field that can be wrong — it can name a timeline
@ -35,6 +34,8 @@
instance is `:rate` on its `:time` map, which is a factor and not a rate. A instance is `:rate` on its `:time` map, which is a factor and not a rate. A
frame COUNT is a property of a frame space, so every timeline has its own." frame COUNT is a property of a frame space, so every timeline has its own."
(:require [arthur.domain.feature :as feature] (:require [arthur.domain.feature :as feature]
[arthur.domain.node :as node]
[arthur.domain.palette :as pal]
[arthur.domain.timeline :as timeline])) [arthur.domain.timeline :as timeline]))
(def ^:const root-id (def ^:const root-id
@ -81,6 +82,69 @@
[clip] [clip]
(:nodes (root clip))) (:nodes (root clip)))
(defn- transform-op
"Put a symbol's already resolved mark into its instance's parent space."
[op m path]
(let [at (fn [x y] [(+ (* (aget m 0) x) (* (aget m 2) y) (aget m 4))
(+ (* (aget m 1) x) (* (aget m 3) y) (aget m 5))])
scale (node/mean-scale m)
op (assoc op :node (conj path (:node op)))]
(case (:kind op)
:poly (let [out (js/Float64Array. (.-length (:pts op)))]
(dotimes [i (:n op)]
(let [[x y] (at (aget (:pts op) (* 2 i))
(aget (:pts op) (inc (* 2 i))))]
(aset out (* 2 i) x)
(aset out (inc (* 2 i)) y)))
(assoc op :pts out))
:disc (let [[x y] (at (:cx op) (:cy op))]
(assoc op :cx x :cy y :r (* scale (:r op))))
:rect (let [[x y] (at (:cx op) (:cy op))]
(assoc op :cx x :cy y :size (js/Math.round (* scale (:size op)))))
op)))
(defn resolver
"Resolve a clip, including each library timeline placed by a symbol instance.
Each instance owns its own timeline resolver, so two offsets never share a
channel cursor or point buffer. The returned ops must be drawn before the next
frame, as with timeline/resolver."
([clip store] (resolver clip store pal/index-of))
([clip store palette]
(letfn [(build [tid chain]
(when (some #{tid} chain)
(throw (ex-info "symbol timeline cycle" {:chain (conj chain tid)})))
(let [tl (or (timeline clip tid)
(throw (ex-info "symbol names a missing timeline" {:timeline tid})))
nodes (:nodes tl)
rank (timeline/draw-rank nodes (timeline/order nodes))
ids (sort-by rank (keys nodes))
own (timeline/resolver tl store palette)
children (into {}
(for [[id n] nodes :when (= :symbol (:kind n))]
[id (build (:of n) (conj chain tid))]))]
(fn [f]
(let [by-id (into {} (map (juxt :node identity)) (own f))]
(into []
(mapcat
(fn [id]
(let [n (get nodes id)]
(if (= :symbol (:kind n))
(let [m (timeline/world-of own id)
local (timeline/frame-of own id)
target (timeline clip (:of n))
length (:frames target)
frame (when (and m (number? local))
(if (get-in n [:time :loop?])
(mod local length)
local))]
(if (and frame (<= 0 frame) (< frame length))
(map #(transform-op % m [id]) ((get children id) frame))
[]))
(when-let [op (get by-id id)] [op]))))
ids))))))]
(build root-id []))))
(defn problems (defn problems
"Human-readable reasons this clip will not evaluate or will not save. Empty "Human-readable reasons this clip will not evaluate or will not save. Empty
means it will. means it will.
@ -106,5 +170,17 @@
(for [[id tl] (:timelines clip) (for [[id tl] (:timelines clip)
p (timeline/problems tl)] p (timeline/problems tl)]
(str "timeline " (pr-str id) ": " p)) (str "timeline " (pr-str id) ": " p))
(for [[tid tl] (:timelines clip)
[id n] (:nodes tl)
:when (and (= :symbol (:kind n))
(not (contains? (:timelines clip) (:of n))))]
(str "timeline " (pr-str tid) " symbol " (pr-str id)
" names missing timeline " (pr-str (:of n))))
(for [[tid tl] (:timelines clip)
[id n] (:nodes tl)
:when (and (= :audio (:kind n)) (:linked-to n)
(not (contains? (:nodes tl) (:linked-to n))))]
(str "timeline " (pr-str tid) " audio " (pr-str id)
" links to missing node " (pr-str (:linked-to n))))
(when (map? (root clip)) (when (map? (root clip))
(feature/problems clip (nodes clip)))))) (feature/problems clip (nodes clip))))))

28
frontend/src/arthur/domain/node.cljs
View file

@ -18,12 +18,12 @@
(:require [arthur.domain.channel :as ch])) (:require [arthur.domain.channel :as ch]))
(def kinds (def kinds
"`:symbol` and `:bitmap` are in the vocabulary and not implemented; they are "`:bitmap` is in the vocabulary and not implemented; it is
here so that a scene that names one fails as \"not implemented\" rather than as here so that a scene that names one fails as \"not implemented\" rather than as
\"not a kind\"." \"not a kind\"."
#{:poly :disc :rect :group :bitmap :symbol}) #{:poly :disc :rect :group :bitmap :symbol :audio})
(def implemented-kinds #{:poly :disc :rect :group}) (def implemented-kinds #{:poly :disc :rect :group :symbol :audio})
(def xform-paths (def xform-paths
"In composition order, which is also the order they have to be sampled in. "In composition order, which is also the order they have to be sampled in.
@ -45,6 +45,8 @@
change to this spec silently change what gets drawn." change to this spec silently change what gets drawn."
(let [base (into #{[:vis]} xform-paths)] (let [base (into #{[:vis]} xform-paths)]
{:group base {:group base
:symbol base
:audio (into base [[:audio :gain] [:audio :pan] [:audio :rate]])
:poly (into base [[:geom :pts] [:style :color]]) :poly (into base [[:geom :pts] [:style :color]])
;; A disc's radius is framed in practice — iris size is a knob, not a ;; A disc's radius is framed in practice — iris size is a knob, not a
;; performance — but it is a channel like any other so it can be keyed. ;; performance — but it is a channel like any other so it can be keyed.
@ -115,22 +117,20 @@
as two performances; offset is PER-NODE by design, because mouth lead applies as two performances; offset is PER-NODE by design, because mouth lead applies
to performance nodes and not to the plate, which is the entire point of it." to performance nodes and not to the plate, which is the entire point of it."
[n f] [n f]
(let [{:keys [mode offset rate source-fps sample-fps] (let [{:keys [mode offset rate at in source-fps sample-fps]
ex :expose :or {mode :inherit}} (:time n)] ex :expose :or {mode :inherit}} (:time n)]
(if (= mode :inherit) (if (= mode :inherit)
f f
(do (do
;; A symbol instance's (f - at)·rate + in is the third face of this (when (and (not (#{:symbol :audio} (:kind n))) rate (not= rate 1.0) (not= rate 1))
;; mechanism and symbols are out of scope. Loud rather than ignored: a (throw (ex-info "time map :rate belongs to a symbol or audio instance"
;; silently dropped rate is a retimed blink playing at the wrong speed,
;; which looks like a bad blink and not like a missing feature.
(when (and rate (not= rate 1.0) (not= rate 1))
(throw (ex-info "time map :rate is symbol timing and symbols are not built (port-plan step 2 scope)"
{:node (:id n) :time (:time n)}))) {:node (:id n) :time (:time n)})))
(when (and sample-fps (not (and source-fps (pos? source-fps)))) (when (and sample-fps (not (and source-fps (pos? source-fps))))
(throw (ex-info "picture sampling needs a positive source fps" (throw (ex-info "picture sampling needs a positive source fps"
{:node (:id n) :time (:time n)}))) {:node (:id n) :time (:time n)})))
(cond-> f (cond-> (if (#{:symbol :audio} (:kind n))
(+ (or in 0) (* (or rate 1) (- f (or at 0))))
f)
sample-fps (sample-frame source-fps sample-fps) sample-fps (sample-frame source-fps sample-fps)
ex (expose ex) ex (expose ex)
offset (+ offset)))))) offset (+ offset))))))
@ -265,6 +265,12 @@
(not (contains? implemented-kinds k))) (not (contains? implemented-kinds k)))
(conj (str ":kind " k " is in the vocabulary but not implemented")) (conj (str ":kind " k " is in the vocabulary but not implemented"))
(and (= k :symbol) (nil? (:of n))) (conj "a symbol instance needs :of")
(and (= k :audio) (nil? (get-in n [:source :footage])))
(conj "an audio instance needs :source :footage")
(and (#{:symbol :audio} k) (some? (get-in n [:time :rate]))
(not (pos? (get-in n [:time :rate]))))
(conj "an instance's :rate must be positive")
(nil? (:z n)) (conj "no :z — draw order is authored per scene, not implied by the tree") (nil? (:z n)) (conj "no :z — draw order is authored per scene, not implied by the tree")
(and (:span n) (not= 2 (count (:span n)))) (and (:span n) (not= 2 (count (:span n))))
(conj ":span must be [in out]")) (conj ":span must be [in out]"))

18
frontend/src/arthur/domain/timeline.cljs
View file

@ -282,6 +282,8 @@
(let [colour #(colour-index palette (rd [:style :color]))] (let [colour #(colour-index palette (rd [:style :color]))]
(case (:kind n) (case (:kind n)
:group nil :group nil
:symbol nil
:audio nil
:poly :poly
(let [pts (rd [:geom :pts])] (let [pts (rd [:geom :pts])]
@ -356,7 +358,8 @@
(if (and pid (nil? parent)) (if (and pid (nil? parent))
acc acc
(if-let [p (place ctx n parent f)] (if-let [p (place ctx n parent f)]
(let [op (emit ctx n p {:node id :stencil (:stencil n)})] (let [_ (when-let [on-place (:on-place ctx)] (on-place id p))
op (emit ctx n p {:node id :stencil (:stencil n)})]
(cond-> (update acc :placed assoc id p) (cond-> (update acc :placed assoc id p)
op (update :ops conj op))) op (update :ops conj op)))
acc)))) acc))))
@ -418,7 +421,8 @@
A registered photo underlay has to ride the same transform the vectors went A registered photo underlay has to ride the same transform the vectors went
through or it is merely decorative, and it paints immediately after the frame through or it is merely decorative, and it paints immediately after the frame
it belongs to, so \"as of the last frame\" is the only answer that can be it belongs to, so \"as of the last frame\" is the only answer that can be
correct.")) correct.")
(frame-of [this id] "The placed node's local frame on the last resolve."))
(defn resolver (defn resolver
"(fn [f] -> ops). Holds everything that does not change per frame. "(fn [f] -> ops). Holds everything that does not change per frame.
@ -455,21 +459,23 @@
;; it is how the resolver learns which nodes exist this frame without ;; it is how the resolver learns which nodes exist this frame without
;; eval-into having to report it — and it covers groups, which are ;; eval-into having to report it — and it covers groups, which are
;; placed but emit no op, and which are exactly what an underlay rides. ;; placed but emit no op, and which are exactly what an underlay rides.
placed (volatile! #{}) placed (volatile! {})
ctx {:read (fn [id path _c lf] (ch/sample! (get-in cursors [id path]) lf)) ctx {:read (fn [id path _c lf] (ch/sample! (get-in cursors [id path]) lf))
:palette palette :palette palette
:mat-for (fn [id] (vswap! placed conj id) (get mats id)) :mat-for (fn [id] (get mats id))
:on-place (fn [id p] (vswap! placed assoc id p))
:pinv-for (fn [id] (get pinvs id)) :pinv-for (fn [id] (get pinvs id))
:buf-for (fn [id _n] (get bufs id)) :buf-for (fn [id _n] (get bufs id))
:scratch scratch} :scratch scratch}
step (fn [f] step (fn [f]
(vreset! placed #{}) (vreset! placed {})
(eval-into ctx nodes ord rank f))] (eval-into ctx nodes ord rank f))]
(reify (reify
IFn IFn
(-invoke [_ f] (step f)) (-invoke [_ f] (step f))
IResolver IResolver
(world-of [_ id] (when (contains? @placed id) (get mats id))))))) (world-of [_ id] (:m (get @placed id)))
(frame-of [_ id] (:f (get @placed id)))))))
;; --------------------------------------------------------------------------- ;; ---------------------------------------------------------------------------

53
frontend/src/arthur/events/project.cljs
View file

@ -22,6 +22,8 @@
Nothing here touches app-db except through events. The promise chain lives in an Nothing here touches app-db except through events. The promise chain lives in an
fx, which is the only thing in this namespace that is not pure." fx, which is the only thing in this namespace that is not pure."
(:require [arthur.domain.clip :as clip] (:require [arthur.domain.clip :as clip]
[arthur.audio.mix :as mix]
[arthur.demo.stage :as stage]
[arthur.domain.project :as project] [arthur.domain.project :as project]
[arthur.domain.wire :as wire] [arthur.domain.wire :as wire]
[arthur.events.playback :as pb] [arthur.events.playback :as pb]
@ -98,14 +100,16 @@
cid #js {:leaves (.-leaves clip-json) cid #js {:leaves (.-leaves clip-json)
:blocks blocks}) :blocks blocks})
built (:clip loaded)] built (:clip loaded)]
(merge (select-keys built [:fps :width :height]) (let [entry (merge (select-keys built [:fps :width :height])
{:label (str (or (.-name clip-json) cid) " (saved)") {:label (str (or (.-name clip-json) cid) " (saved)")
:cid cid :frames (clip/frames built) :cid cid :frames (clip/frames built)
:display-fps (:fps built) :display-fps (:fps built)
:clip built :store (:store loaded) :clip built :store (:store loaded)
:footage-id footage-id :footage-id footage-id
:audio (if footage (.-audio footage) :audio (if footage (.-audio footage)
"/static/arthur/audio.wav")}))))))) "/static/arthur/audio.wav")})]
(-> (mix/mix! built (:audio entry) (:store entry))
(.then (fn [audio] (assoc entry :audio audio)))))))))))
(rf/reg-fx (rf/reg-fx
::save! ::save!
@ -173,6 +177,29 @@
(js/console.error error) (js/console.error error)
(rf/dispatch [::failed (or (ex-message error) (str error))])))))) (rf/dispatch [::failed (or (ex-message error) (str error))]))))))
(rf/reg-fx
::stage!
(fn [_]
(-> (http/GET (str "/api/projects/" (:source-project stage/layout)))
(.then (fn [^js saved]
(or (first (filter #(= (:source-cid stage/layout) (.-cid ^js %))
(array-seq (.-clips saved))))
(throw (ex-info "the saved 8625 clip is missing" {})))))
(.then opened-entry!)
(.then (fn [entry]
(let [built (stage/compose (:clip entry))
entry (assoc entry :clip built :label (:name built)
:cid "stage-8625" :frames (clip/frames built)
:width (:width built) :height (:height built))]
(-> (mix/mix! built (:audio entry) (:store entry))
(.then (fn [audio]
(rf/dispatch
[::stage-opened
(store/install! (assoc entry :audio audio) "stage")])))))))
(.catch (fn [error]
(js/console.error error)
(rf/dispatch [::failed (or (ex-message error) (str error))]))))))
;; --------------------------------------------------------------------------- ;; ---------------------------------------------------------------------------
;; events ;; events
@ -198,6 +225,28 @@
::pb/pause! nil ::pb/pause! nil
::open! (:id (:project db))}))) ::open! (:id (:project db))})))
(rf/reg-event-fx
::load-stage
(fn [{:keys [db]} _]
(if (:busy? (:project db))
{}
{:db (update db :project merge {:busy? true :status "loading 8625 stage…"})
::pb/pause! nil
::stage! nil})))
(rf/reg-event-fx
::stage-opened
(fn [{:keys [db]} [_ clip-id]]
(let [entry (store/entry clip-id)]
{:db (-> db
(assoc :clip/current clip-id
:clip (select-keys entry [:fps :frames :width :height :audio :display-fps]))
(assoc :project {:id nil :cid nil :name nil :seq nil
:busy? false :status "loaded 8625 stage study"})
(assoc-in [:playback :frame] 0)
(assoc-in [:playback :playing?] false))
::pb/seek! [(:fps entry) (:frames entry) 0]})))
(rf/reg-event-db (rf/reg-event-db
::saved ::saved
(fn [db [_ id cid label seq written uploaded]] (fn [db [_ id cid label seq written uploaded]]

2
frontend/src/arthur/footage/store.cljs
View file

@ -21,6 +21,8 @@
([entry] (install! entry "footage")) ([entry] (install! entry "footage"))
([entry kind] ([entry kind]
(let [id (keyword kind (str (swap! serial inc)))] (let [id (keyword kind (str (swap! serial inc)))]
(when-let [old (:audio @loaded)]
(when (.startsWith old "blob:") (js/URL.revokeObjectURL old)))
(reset! loaded (assoc entry :id id)) (reset! loaded (assoc entry :id id))
id))) id)))

6
frontend/src/arthur/subs/render.cljs
View file

@ -79,8 +79,10 @@
(rf/reg-sub (rf/reg-sub
::resolver ::resolver
:<- [::clip]
:<- [::timeline] :<- [::timeline]
:<- [::store] :<- [::store]
:<- [::palette] :<- [::palette]
(fn [[tl store palette] _] (fn [[document tl store palette] _]
(when tl (timeline/resolver tl store palette)))) (when tl (clip/resolver (assoc-in document [:timelines :main] tl)
store palette))))

2
frontend/src/arthur/ui/shell.cljs
View file

@ -72,6 +72,8 @@
;; the model serialises and a proof nobody can run is not one. ;; the model serialises and a proof nobody can run is not one.
[:button {:disabled busy? :on-click #(rf/dispatch [::project/save])} "save"] [:button {:disabled busy? :on-click #(rf/dispatch [::project/save])} "save"]
[:button {:disabled busy? :on-click #(rf/dispatch [::project/open])} "open"] [:button {:disabled busy? :on-click #(rf/dispatch [::project/open])} "open"]
[:button {:disabled busy? :on-click #(rf/dispatch [::project/load-stage])}
"stage 8625"]
[:span.gap] [:span.gap]
(doall (doall
(for [r db/rates] (for [r db/rates]

14
frontend/test/arthur/domain/channel_test.cljs
View file

@ -222,8 +222,18 @@
(is (seq (ch/problems {:animated? true :keys {0 1} (is (seq (ch/problems {:animated? true :keys {0 1}
:dense {:store "x" :offset 0 :stride 1 :frames 1}})) :dense {:store "x" :offset 0 :stride 1 :frames 1}}))
"one shape at a time") "one shape at a time")
(is (seq (ch/problems {:animated? true :interp :linear :keys {0 1}})) (is (empty? (ch/problems (ch/keyed {0 0.0, 10 1.0} :linear)))
"only :hold is implemented")) "scalar controls can interpolate")
(is (seq (ch/problems (ch/keyed {0 :a, 10 :b} :linear)))
"a cut or tone cannot interpolate"))
(deftest numeric-channels-can-ramp-between-keys
(let [c (ch/keyed {0 0.0, 10 1.0} :linear)
cursor (ch/cursor c)]
(is (= [0.0 0.5 1.0 1.0]
(mapv #(ch/value-at c %) [0 5 10 15])))
(is (= [0.0 0.5 1.0 0.2]
(mapv #(ch/sample! cursor %) [0 5 10 2])))))
(deftest component-reads-vectors-and-typed-arrays-the-same-way (deftest component-reads-vectors-and-typed-arrays-the-same-way
(is (= 3 (ch/component [3 4] 0))) (is (= 3 (ch/component [3 4] 0)))

57
frontend/test/arthur/domain/symbol_test.cljs Normal file
View file

@ -0,0 +1,57 @@
(ns arthur.domain.symbol-test
(:require [cljs.test :refer [deftest is]]
[arthur.demo.stage :as stage]
[arthur.domain.channel :as ch]
[arthur.domain.clip :as clip]
[arthur.domain.leaf :as leaf]))
(def source
{:name "source" :fps 30 :width 320 :height 200
:timelines
{:main {:id :main :frames 4
:nodes {:root {:id :root :kind :group :z "a1"}
:mark {:id :mark :kind :rect :parent :root :z "a1"
:channels {[:xform :pos] (ch/keyed {0 [0 0] 1 [10 0]
2 [20 0] 3 [30 0]})
[:geom :size] (ch/framed 4)
[:style :color] (ch/framed :brow)}}}}}})
(deftest two-instances-own-their-frame-and-placement
(let [document
(-> source
(assoc-in [:timelines :main]
{:id :main :frames 6
:nodes {:root {:id :root :kind :group :z "a1"}
:left {:id :left :kind :symbol :of :sym/test
:parent :root :z "a1" :span [0 4]
:channels {[:xform :pos] (ch/framed [100 50])}}
:right {:id :right :kind :symbol :of :sym/test
:parent :root :z "a2" :span [2 6]
:time {:mode :map :at 2 :in 0 :rate 1}
:channels {[:xform :pos] (ch/framed [120 50])}}}})
(assoc-in [:timelines :sym/test]
(assoc (get-in source [:timelines :main]) :id :sym/test)))
resolve (clip/resolver document nil)
at (fn [f] (mapv (juxt :node :cx) (resolve f)))]
(is (empty? (clip/problems document)))
(is (= [[[:left :mark] 110]] (at 1)))
(is (= [[[:left :mark] 120] [[:right :mark] 120]] (at 2)))
(is (= [[[:right :mark] 150]] (at 5)))
(is (= document (leaf/clip "stage" (leaf/leaves "stage" document))))))
(deftest stage-fixture-keeps-source-as-one-symbol
(let [document (stage/compose source)]
(is (empty? (clip/problems document)))
(is (= #{:main :sym/face-8625} (set (keys (:timelines document)))))
(is (= :sym/face-8625 (get-in document [:timelines :main :nodes :left :of])))
(is (= :sym/face-8625 (get-in document [:timelines :main :nodes :right :of])))
(is (= [48 280] (get-in document [:timelines :main :nodes :right :span])))
(is (= :right (get-in document [:timelines :main :nodes :voice-right :linked-to])))
(is (= [48 260] (get-in document [:timelines :main :nodes :voice-right :span])))
(is (= 0.5 (ch/value-at
(get-in document [:timelines :main :nodes :voice-right
:channels [:audio :gain]]) 54)))
(is (< -0.8 (ch/value-at
(get-in document [:timelines :main :nodes :voice-right
:channels [:audio :pan]]) 110) 0.7))
(is (= document (leaf/clip "stage" (leaf/leaves "stage" document))))))