diff --git a/docs/animation-model.md b/docs/animation-model.md index d7a407e..d8ff025 100644 --- a/docs/animation-model.md +++ b/docs/animation-model.md @@ -495,6 +495,39 @@ reading head over the same channel. The resolver keys its caches by the instance path, not by node id — which is a detail of `Making it fast` below, and the one place symbol nesting is not free. +### Audio placements and controls + +Sound is placed on a timeline as a separate `:audio` node. It uses the same +`:span`, `:time`, and channel representation as a drawn node. A `:linked-to` id +records which picture instance it was placed with; it does not force the two +spans or source in-points to match. + +```clojure +{:id :voice-right :kind :audio :parent :root :z "a4" + :linked-to :right + :source {:footage "f8cace9e-..."} + :span [48 260] + :time {:mode :map :at 48 :in 0 :rate 1} + :channels {[:audio :gain] + {:animated? true :interp :linear + :keys {48 0.0, 60 1.0, 245 1.0, 259 0.0} :over []}}} +``` + +`[:audio :gain]`, `[:audio :pan]`, and `[:audio :rate]` are ordinary scalar +channels. They may be framed, keyed, or dense; numeric keyed channels can ramp +linearly. The time map sets the placement's base source rate, and +`[:audio :rate]` multiplies it. Audio is mixed from the referenced immutable +footage when the clip opens. The mix is derived output; the saved document holds +the nodes and channel keys, not another audio file. One audio element plays that +mix and remains the clock for both sound and picture. + +This is also the boundary for a future control surface. A control has a stable +target, such as a feature's `:verts` setting or an audio node's +`[:audio :gain]` channel. The UI and a MIDI binding can address both through the +same control interface. Their update costs differ: gain can be keyed over time; +changing the number of lip vertices changes topology and must regenerate its +dense geometry. A topology setting cannot be treated as a per-frame gain curve. + ## Evaluating a frame ```clojure diff --git a/frontend/README.md b/frontend/README.md index 507d374..b88ca31 100644 --- a/frontend/README.md +++ b/frontend/README.md @@ -112,6 +112,33 @@ The demo scene itself is `src/arthur/demo/scene.edn`. Both the synthetic take and real footage use `src/arthur/flow/take.cljs` for the measurement order and `src/arthur/flow/freeze.cljs` for the landmark-to-channel conversion. +**stage 8625** loads the locally saved `IMG_8625.MOV` project and places its +post-processed timeline twice. The stage layout is +`src/arthur/demo/stage_8625.edn`: the right picture and sound start at frame 48, +and the two pictures overlap slightly in stage space. Audio has its own timeline +nodes, linked to the picture instances but with independent spans and gain +channels. The right sound swells and pans across the stage, then fades out at +frame 260 while its picture continues to +frame 280. The button needs that saved 8625 project in the local server database. + +### Projects and the EDN fixtures + +The EDN files under `demo/` are authored examples compiled into the frontend. +They seed a clip in memory; the server does not read EDN. Clicking **save** on a +clip without a project id creates a project through `POST /api/projects`, uploads +any missing content-addressed blocks, then writes the clip's addressed leaves +through `PUT /api/projects/`. Each leaf value is Transit JSON inside the +request's ordinary JSON envelope. Python stores those values in JSON columns and +does not need an EDN parser. **open** reads the leaves and blocks and rebuilds +the same ClojureScript clip. + +The intended editor creates and changes that in-memory clip directly: a project +browser and **new stage** action, timeline instance placement, node and channel +editors, then the existing save path. EDN remains useful for checked-in examples +and reproducible studies. The current UI has save and open, but no project +browser, blank-stage action, or authoring controls yet; open chooses the most +recent project. + ### Real footage Choose a video in the **footage** file input. The server probes it, re-encodes it diff --git a/frontend/src/arthur/audio/mix.cljs b/frontend/src/arthur/audio/mix.cljs new file mode 100644 index 0000000..095c476 --- /dev/null +++ b/frontend/src/arthur/audio/mix.cljs @@ -0,0 +1,119 @@ +(ns arthur.audio.mix + "Render independently placed audio tracks into one stage audio clock. + + The mix is derived from saved audio track leaves and immutable footage blobs. + The transport still has one audio element, so seeking, rate changes and looping + stay tied to the same clock the picture reads." + (:require [arthur.domain.channel :as ch] + [arthur.domain.clip :as clip] + [arthur.domain.node :as node])) + +(defn- wav [^js buffer] + (let [channels (.-numberOfChannels buffer) + frames (.-length buffer) + rate (.-sampleRate buffer) + bytes (js/ArrayBuffer. (+ 44 (* frames channels 2))) + view (js/DataView. bytes) + samples (mapv #(.getChannelData buffer %) (range channels)) + peak (reduce max 0 + (for [channel samples i (range frames)] + (js/Math.abs (aget channel i)))) + level (if (> peak 0.98) (/ 0.98 peak) 1)] + (doseq [[offset word] [[0 "RIFF"] [8 "WAVE"] [12 "fmt "] [36 "data"]]] + (dotimes [i 4] (.setUint8 view (+ offset i) (.charCodeAt word i)))) + (.setUint32 view 4 (- (.-byteLength bytes) 8) true) + (.setUint32 view 16 16 true) + (.setUint16 view 20 1 true) + (.setUint16 view 22 channels true) + (.setUint32 view 24 rate true) + (.setUint32 view 28 (* rate channels 2) true) + (.setUint16 view 32 (* channels 2) true) + (.setUint16 view 34 16 true) + (.setUint32 view 40 (* frames channels 2) true) + (dotimes [i frames] + (dotimes [c channels] + (let [sample (* level (aget (get samples c) i))] + (.setInt16 view (+ 44 (* (+ (* i channels) c) 2)) + (js/Math.round (* 32767 (max -1 (min 1 sample)))) true)))) + (js/URL.createObjectURL (js/Blob. #js [bytes] #js {:type "audio/wav"})))) + +(defn- source! [footage-id] + (-> (js/fetch (str "/api/footage/" footage-id)) + (.then (fn [response] + (when-not (.-ok response) + (throw (ex-info "audio track's footage is missing" + {:footage footage-id :status (.-status response)}))) + (.json response))) + (.then (fn [^js manifest] + (-> (js/fetch (.-audio manifest)) + (.then (fn [response] + (when-not (.-ok response) + (throw (ex-info "audio track's blob is missing" + {:footage footage-id :status (.-status response)}))) + (.arrayBuffer response))) + (.then (fn [bytes] + (let [decoder (js/OfflineAudioContext. 1 1 44100)] + (-> (.decodeAudioData decoder bytes) + (.then (fn [buffer] + [footage-id {:buffer buffer + :fps (.-fps manifest)}]))))))))))) + +(defn- automate! [^js param channel start end fps factor default store] + (let [channel (or channel (ch/framed default))] + (.setValueAtTime param (* factor (ch/value-at channel start store)) (/ start fps)) + (cond + (:dense channel) + (doseq [f (range (inc start) end)] + (.setValueAtTime param (* factor (ch/value-at channel f store)) (/ f fps))) + + (:animated? channel) + (doseq [[f v] (sort-by key (:keys channel)) + :when (and (> f start) (< f end))] + (if (= :linear (:interp channel)) + (.linearRampToValueAtTime param (* factor v) (/ f fps)) + (.setValueAtTime param (* factor v) (/ f fps))))))) + +(defn- render! [document sources store] + (let [fps (:fps document) + frames (clip/frames document) + tracks (filter #(= :audio (:kind %)) (vals (clip/nodes document))) + output (js/OfflineAudioContext. + 2 (js/Math.ceil (* (/ frames fps) 44100)) 44100)] + (doseq [track tracks] + (let [[start end] (or (:span track) [0 frames]) + start (max 0 start) + end (min frames end) + {:keys [buffer fps]} (get sources (get-in track [:source :footage])) + sound (.createBufferSource output) + gain (.createGain output) + pan (.createStereoPanner output)] + (when (< start end) + (set! (.-buffer sound) buffer) + (set! (.-loop sound) (boolean (get-in track [:time :loop?]))) + (automate! (.-playbackRate sound) + (get-in track [:channels [:audio :rate]]) + start end (:fps document) (or (get-in track [:time :rate]) 1) 1 store) + (automate! (.-gain gain) + (get-in track [:channels [:audio :gain]]) + start end (:fps document) 1 1 store) + (automate! (.-pan pan) + (get-in track [:channels [:audio :pan]]) + start end (:fps document) 1 0 store) + (.connect sound gain) + (.connect gain pan) + (.connect pan (.-destination output)) + (.start sound (/ start (:fps document)) (/ (node/local-frame track start) fps)) + (.stop sound (/ end (:fps document)))))) + (-> (.startRendering output) (.then wav)))) + +(defn mix! + "Promise of a mixed WAV URL, or the original URL for a clip without audio + tracks. Each track can be trimmed and faded independently of its linked picture." + ([document fallback-url] (mix! document fallback-url nil)) + ([document fallback-url store] + (let [tracks (filter #(= :audio (:kind %)) (vals (clip/nodes document)))] + (if (empty? tracks) + (js/Promise.resolve fallback-url) + (-> (js/Promise.all + (into-array (map source! (distinct (map #(get-in % [:source :footage]) tracks))))) + (.then (fn [pairs] (render! document (into {} (array-seq pairs)) store)))))))) diff --git a/frontend/src/arthur/demo/stage.cljs b/frontend/src/arthur/demo/stage.cljs new file mode 100644 index 0000000..8d87c4f --- /dev/null +++ b/frontend/src/arthur/demo/stage.cljs @@ -0,0 +1,32 @@ +(ns arthur.demo.stage + "A saved take placed twice on a stage. The EDN is the authored layout." + (:require [cljs.reader :as reader] + [shadow.resource :as rc])) + +(def layout (reader/read-string (rc/inline "arthur/demo/stage_8625.edn"))) + +(defn compose [source] + (let [{:keys [name width height frames symbol instances audio]} layout + original (get-in source [:timelines :main]) + nodes (into + {:root {:id :root :name "stage" :kind :group :z "a1"}} + (map (fn [{:keys [id name z span at in pos scale]}] + [id {:id id :name name :kind :symbol :of symbol + :parent :root :z z :span span + :time {:mode :map :at at :in in :rate 1} + :channels {[:xform :pos] {:animated? false :value pos} + [:xform :scale] {:animated? false :value scale}}}]) + instances)) + nodes (into nodes + (map (fn [{:keys [id linked-to z source span at in gain pan]}] + [id {:id id :kind :audio :parent :root :z z + :linked-to linked-to :source source :span span + :time {:mode :map :at at :in in :rate 1} + :channels (cond-> {[:audio :gain] gain} + pan (assoc [:audio :pan] pan))}]) + audio))] + (-> source + (assoc :name name :width width :height height + :timelines {:main {:id :main :frames frames :nodes nodes} + symbol (assoc original :id symbol)}) + (dissoc :subjects :features :groups)))) diff --git a/frontend/src/arthur/demo/stage_8625.edn b/frontend/src/arthur/demo/stage_8625.edn new file mode 100644 index 0000000..3596ec3 --- /dev/null +++ b/frontend/src/arthur/demo/stage_8625.edn @@ -0,0 +1,32 @@ +;; A local stage sketch. The source is the saved, post-processed IMG_8625.MOV +;; clip in the project store; its dense channel blocks remain shared by both +;; instances. Positions and timing are authored here in stage pixels and frames. +{:source-project "4379f900-bdd2-409b-acf6-32081f8ce01f" + :source-cid "f8cace9e-4ad3-4796-973c-c62eeebe3d01" + :symbol :sym/face-8625 + :name "8625 stage study" + :width 320 :height 200 :frames 280 + ;; Audio placements are ordinary timeline nodes with channel parameters. + ;; :linked-to is an editorial link; their spans and time maps are independent. + :audio + [{:id :voice-left :linked-to :left :z "a3" + :source {:footage "f8cace9e-4ad3-4796-973c-c62eeebe3d01"} + :span [0 280] :at 0 :in 0 + :gain {:animated? false :value 1.0}} + {:id :voice-right :linked-to :right :z "a4" + :source {:footage "f8cace9e-4ad3-4796-973c-c62eeebe3d01"} + :span [48 260] :at 48 :in 0 + :gain {:animated? true :interp :linear + :keys {48 0.0, 60 1.0, 90 0.35, 115 0.9, 145 0.45, + 170 1.0, 195 0.4, 220 0.85, 245 1.0, 259 0.0} + :over []} + :pan {:animated? true :interp :linear + :keys {48 -0.8, 90 -0.8, 130 0.7, 175 0.7, 220 -0.65, 259 0.65} + :over []}}] + :instances + [{:id :left :name "8625 left" :z "a1" + :span [0 280] :at 0 :in 0 + :pos [12 38] :scale [0.62 0.62]} + {:id :right :name "8625 right" :z "a2" + :span [48 280] :at 48 :in 0 + :pos [108 38] :scale [0.62 0.62]}]} diff --git a/frontend/src/arthur/domain/channel.cljs b/frontend/src/arthur/domain/channel.cljs index a280449..4933148 100644 --- a/frontend/src/arthur/domain/channel.cljs +++ b/frontend/src/arthur/domain/channel.cljs @@ -92,12 +92,9 @@ (vec (sort (keys ks))))) (defn- check-unimplemented! - "An override layer or a retimed symbol instance must fail LOUDLY rather than - be ignored. + "An override layer must fail LOUDLY rather than be ignored. - Both are specified in docs/animation-model.md and neither is built yet - (port-plan scope: \"leave the :over field present and empty; leave :symbol out - entirely\"). Silently dropping an :over layer would present as a hand + Silently dropping an :over layer would present as a hand correction that did not take — a correction the user made once, watched fail, and has no reason to trust again. Nothing can produce one yet, so this can only fire on a data shape that has run ahead of the code." @@ -172,6 +169,13 @@ ;; --------------------------------------------------------------------------- ;; the specification +(defn- interpolate [ch f left right] + (let [a (get (:keys ch) left)] + (if (and (= :linear (:interp ch)) right (>= f left) (> right left)) + (let [b (get (:keys ch) right)] + (+ a (* (/ (- f left) (- right left)) (- b a)))) + a))) + (defn- keyed-at "The most recent key at or before f, CLAMPED to the first key below it. @@ -180,11 +184,11 @@ frame before it reads that pose rather than having no value. This is not the same question as presence — a part with no value at all is `absent`, which is a state bit, not an empty key map." - [ks f] - (let [fr (sort (keys ks))] - (if-let [hit (last (take-while #(<= % f) fr))] - (get ks hit) - (get ks (first fr))))) + [ch f] + (let [fr (sort (keys (:keys ch))) + left (or (last (take-while #(<= % f) fr)) (first fr)) + right (first (drop-while #(<= % f) fr))] + (interpolate ch f left right))) (defn value-at "Sample a channel at frame f. THE SPECIFICATION — correct, allocating, and @@ -196,7 +200,7 @@ (not (:animated? ch)) (:value ch) (:dense ch) (dense-at (:dense ch) f store) (:keys ch) (let [ks (:keys ch)] - (if (empty? ks) absent (keyed-at ks f))) + (if (empty? ks) absent (keyed-at ch f))) :else (throw (ex-info "animated channel has neither :keys nor :dense" {:channel ch}))))) @@ -272,7 +276,7 @@ :else (bsearch ks f))] (set! (.-i cur) i') - (get (:keys ch) (nth ks i')))))) + (interpolate ch f (nth ks i') (when (< i' last) (nth ks (inc i')))))))) ;; --------------------------------------------------------------------------- @@ -307,9 +311,12 @@ (and (map? ch) (:keys ch) (map? (:keys ch)) (not (every? number? (keys (:keys ch))))) (conj ":keys has a non-numeric frame") - (and (map? ch) (:animated? ch) (not (#{:hold nil} (:interp ch)))) - (conj (str ":interp " (:interp ch) " — only :hold is implemented; docs/design.md" - " requires hold of every cut part and tweening reads as puppet software")) + (and (map? ch) (:animated? ch) (not (#{:hold :linear nil} (:interp ch)))) + (conj (str ":interp " (:interp ch) " is not implemented")) + + (and (map? ch) (= :linear (:interp ch)) + (or (:dense ch) (not (every? number? (vals (:keys ch)))))) + (conj ":linear interpolation needs numeric keyed values") (and (map? ch) (seq (:over ch))) (conj ":over layers are not implemented (port-plan step 2 scope)") @@ -322,4 +329,3 @@ (conj (str ":dense :scale is " (pr-str (:scale (:dense ch))) " — a fixed-point scale is a positive number the stored integers" " were multiplied by")))) - diff --git a/frontend/src/arthur/domain/clip.cljs b/frontend/src/arthur/domain/clip.cljs index 7f88b1d..f02013b 100644 --- a/frontend/src/arthur/domain/clip.cljs +++ b/frontend/src/arthur/domain/clip.cljs @@ -19,9 +19,8 @@ structure with the same `:nodes` key that every walk had to be taught about. Now there is one node-holding type — `arthur.domain.timeline` — and a clip holds - a MAP of them. `:kind :symbol` is still unimplemented and this is the shape it - was waiting for: an instance names a timeline in `:timelines`, and the resolver - recurses into a type it already knows how to evaluate. + a MAP of them. A `:kind :symbol` instance names a timeline in `:timelines`, + and the clip resolver gives each placement its own reading heads. THE ROOT TIMELINE HAS A RESERVED ID, `:main`, rather than the clip carrying a pointer to it. A pointer is a field that can be wrong — it can name a timeline @@ -35,6 +34,8 @@ instance is `:rate` on its `:time` map, which is a factor and not a rate. A frame COUNT is a property of a frame space, so every timeline has its own." (:require [arthur.domain.feature :as feature] + [arthur.domain.node :as node] + [arthur.domain.palette :as pal] [arthur.domain.timeline :as timeline])) (def ^:const root-id @@ -81,6 +82,69 @@ [clip] (:nodes (root clip))) +(defn- transform-op + "Put a symbol's already resolved mark into its instance's parent space." + [op m path] + (let [at (fn [x y] [(+ (* (aget m 0) x) (* (aget m 2) y) (aget m 4)) + (+ (* (aget m 1) x) (* (aget m 3) y) (aget m 5))]) + scale (node/mean-scale m) + op (assoc op :node (conj path (:node op)))] + (case (:kind op) + :poly (let [out (js/Float64Array. (.-length (:pts op)))] + (dotimes [i (:n op)] + (let [[x y] (at (aget (:pts op) (* 2 i)) + (aget (:pts op) (inc (* 2 i))))] + (aset out (* 2 i) x) + (aset out (inc (* 2 i)) y))) + (assoc op :pts out)) + :disc (let [[x y] (at (:cx op) (:cy op))] + (assoc op :cx x :cy y :r (* scale (:r op)))) + :rect (let [[x y] (at (:cx op) (:cy op))] + (assoc op :cx x :cy y :size (js/Math.round (* scale (:size op))))) + op))) + +(defn resolver + "Resolve a clip, including each library timeline placed by a symbol instance. + + Each instance owns its own timeline resolver, so two offsets never share a + channel cursor or point buffer. The returned ops must be drawn before the next + frame, as with timeline/resolver." + ([clip store] (resolver clip store pal/index-of)) + ([clip store palette] + (letfn [(build [tid chain] + (when (some #{tid} chain) + (throw (ex-info "symbol timeline cycle" {:chain (conj chain tid)}))) + (let [tl (or (timeline clip tid) + (throw (ex-info "symbol names a missing timeline" {:timeline tid}))) + nodes (:nodes tl) + rank (timeline/draw-rank nodes (timeline/order nodes)) + ids (sort-by rank (keys nodes)) + own (timeline/resolver tl store palette) + children (into {} + (for [[id n] nodes :when (= :symbol (:kind n))] + [id (build (:of n) (conj chain tid))]))] + (fn [f] + (let [by-id (into {} (map (juxt :node identity)) (own f))] + (into [] + (mapcat + (fn [id] + (let [n (get nodes id)] + (if (= :symbol (:kind n)) + (let [m (timeline/world-of own id) + local (timeline/frame-of own id) + target (timeline clip (:of n)) + length (:frames target) + frame (when (and m (number? local)) + (if (get-in n [:time :loop?]) + (mod local length) + local))] + (if (and frame (<= 0 frame) (< frame length)) + (map #(transform-op % m [id]) ((get children id) frame)) + [])) + (when-let [op (get by-id id)] [op])))) + ids))))))] + (build root-id [])))) + (defn problems "Human-readable reasons this clip will not evaluate or will not save. Empty means it will. @@ -106,5 +170,17 @@ (for [[id tl] (:timelines clip) p (timeline/problems tl)] (str "timeline " (pr-str id) ": " p)) + (for [[tid tl] (:timelines clip) + [id n] (:nodes tl) + :when (and (= :symbol (:kind n)) + (not (contains? (:timelines clip) (:of n))))] + (str "timeline " (pr-str tid) " symbol " (pr-str id) + " names missing timeline " (pr-str (:of n)))) + (for [[tid tl] (:timelines clip) + [id n] (:nodes tl) + :when (and (= :audio (:kind n)) (:linked-to n) + (not (contains? (:nodes tl) (:linked-to n))))] + (str "timeline " (pr-str tid) " audio " (pr-str id) + " links to missing node " (pr-str (:linked-to n)))) (when (map? (root clip)) (feature/problems clip (nodes clip)))))) diff --git a/frontend/src/arthur/domain/node.cljs b/frontend/src/arthur/domain/node.cljs index 248791b..bc2d432 100644 --- a/frontend/src/arthur/domain/node.cljs +++ b/frontend/src/arthur/domain/node.cljs @@ -18,12 +18,12 @@ (:require [arthur.domain.channel :as ch])) (def kinds - "`:symbol` and `:bitmap` are in the vocabulary and not implemented; they are + "`:bitmap` is in the vocabulary and not implemented; it is here so that a scene that names one fails as \"not implemented\" rather than as \"not a kind\"." - #{:poly :disc :rect :group :bitmap :symbol}) + #{:poly :disc :rect :group :bitmap :symbol :audio}) -(def implemented-kinds #{:poly :disc :rect :group}) +(def implemented-kinds #{:poly :disc :rect :group :symbol :audio}) (def xform-paths "In composition order, which is also the order they have to be sampled in. @@ -45,6 +45,8 @@ change to this spec silently change what gets drawn." (let [base (into #{[:vis]} xform-paths)] {:group base + :symbol base + :audio (into base [[:audio :gain] [:audio :pan] [:audio :rate]]) :poly (into base [[:geom :pts] [:style :color]]) ;; A disc's radius is framed in practice — iris size is a knob, not a ;; performance — but it is a channel like any other so it can be keyed. @@ -115,22 +117,20 @@ as two performances; offset is PER-NODE by design, because mouth lead applies to performance nodes and not to the plate, which is the entire point of it." [n f] - (let [{:keys [mode offset rate source-fps sample-fps] + (let [{:keys [mode offset rate at in source-fps sample-fps] ex :expose :or {mode :inherit}} (:time n)] (if (= mode :inherit) f (do - ;; A symbol instance's (f - at)·rate + in is the third face of this - ;; mechanism and symbols are out of scope. Loud rather than ignored: a - ;; silently dropped rate is a retimed blink playing at the wrong speed, - ;; which looks like a bad blink and not like a missing feature. - (when (and rate (not= rate 1.0) (not= rate 1)) - (throw (ex-info "time map :rate is symbol timing and symbols are not built (port-plan step 2 scope)" + (when (and (not (#{:symbol :audio} (:kind n))) rate (not= rate 1.0) (not= rate 1)) + (throw (ex-info "time map :rate belongs to a symbol or audio instance" {:node (:id n) :time (:time n)}))) (when (and sample-fps (not (and source-fps (pos? source-fps)))) (throw (ex-info "picture sampling needs a positive source fps" {:node (:id n) :time (:time n)}))) - (cond-> f + (cond-> (if (#{:symbol :audio} (:kind n)) + (+ (or in 0) (* (or rate 1) (- f (or at 0)))) + f) sample-fps (sample-frame source-fps sample-fps) ex (expose ex) offset (+ offset)))))) @@ -265,6 +265,12 @@ (not (contains? implemented-kinds k))) (conj (str ":kind " k " is in the vocabulary but not implemented")) + (and (= k :symbol) (nil? (:of n))) (conj "a symbol instance needs :of") + (and (= k :audio) (nil? (get-in n [:source :footage]))) + (conj "an audio instance needs :source :footage") + (and (#{:symbol :audio} k) (some? (get-in n [:time :rate])) + (not (pos? (get-in n [:time :rate])))) + (conj "an instance's :rate must be positive") (nil? (:z n)) (conj "no :z — draw order is authored per scene, not implied by the tree") (and (:span n) (not= 2 (count (:span n)))) (conj ":span must be [in out]")) diff --git a/frontend/src/arthur/domain/timeline.cljs b/frontend/src/arthur/domain/timeline.cljs index 1cd47b7..764da10 100644 --- a/frontend/src/arthur/domain/timeline.cljs +++ b/frontend/src/arthur/domain/timeline.cljs @@ -282,6 +282,8 @@ (let [colour #(colour-index palette (rd [:style :color]))] (case (:kind n) :group nil + :symbol nil + :audio nil :poly (let [pts (rd [:geom :pts])] @@ -356,7 +358,8 @@ (if (and pid (nil? parent)) acc (if-let [p (place ctx n parent f)] - (let [op (emit ctx n p {:node id :stencil (:stencil n)})] + (let [_ (when-let [on-place (:on-place ctx)] (on-place id p)) + op (emit ctx n p {:node id :stencil (:stencil n)})] (cond-> (update acc :placed assoc id p) op (update :ops conj op))) acc)))) @@ -418,7 +421,8 @@ A registered photo underlay has to ride the same transform the vectors went through or it is merely decorative, and it paints immediately after the frame it belongs to, so \"as of the last frame\" is the only answer that can be - correct.")) + correct.") + (frame-of [this id] "The placed node's local frame on the last resolve.")) (defn resolver "(fn [f] -> ops). Holds everything that does not change per frame. @@ -455,21 +459,23 @@ ;; it is how the resolver learns which nodes exist this frame without ;; eval-into having to report it — and it covers groups, which are ;; placed but emit no op, and which are exactly what an underlay rides. - placed (volatile! #{}) + placed (volatile! {}) ctx {:read (fn [id path _c lf] (ch/sample! (get-in cursors [id path]) lf)) :palette palette - :mat-for (fn [id] (vswap! placed conj id) (get mats id)) + :mat-for (fn [id] (get mats id)) + :on-place (fn [id p] (vswap! placed assoc id p)) :pinv-for (fn [id] (get pinvs id)) :buf-for (fn [id _n] (get bufs id)) :scratch scratch} step (fn [f] - (vreset! placed #{}) + (vreset! placed {}) (eval-into ctx nodes ord rank f))] (reify IFn (-invoke [_ f] (step f)) IResolver - (world-of [_ id] (when (contains? @placed id) (get mats id))))))) + (world-of [_ id] (:m (get @placed id))) + (frame-of [_ id] (:f (get @placed id))))))) ;; --------------------------------------------------------------------------- diff --git a/frontend/src/arthur/events/project.cljs b/frontend/src/arthur/events/project.cljs index d9e1439..351eadf 100644 --- a/frontend/src/arthur/events/project.cljs +++ b/frontend/src/arthur/events/project.cljs @@ -22,6 +22,8 @@ Nothing here touches app-db except through events. The promise chain lives in an fx, which is the only thing in this namespace that is not pure." (:require [arthur.domain.clip :as clip] + [arthur.audio.mix :as mix] + [arthur.demo.stage :as stage] [arthur.domain.project :as project] [arthur.domain.wire :as wire] [arthur.events.playback :as pb] @@ -98,14 +100,16 @@ cid #js {:leaves (.-leaves clip-json) :blocks blocks}) built (:clip loaded)] - (merge (select-keys built [:fps :width :height]) - {:label (str (or (.-name clip-json) cid) " (saved)") - :cid cid :frames (clip/frames built) - :display-fps (:fps built) - :clip built :store (:store loaded) - :footage-id footage-id - :audio (if footage (.-audio footage) - "/static/arthur/audio.wav")}))))))) + (let [entry (merge (select-keys built [:fps :width :height]) + {:label (str (or (.-name clip-json) cid) " (saved)") + :cid cid :frames (clip/frames built) + :display-fps (:fps built) + :clip built :store (:store loaded) + :footage-id footage-id + :audio (if footage (.-audio footage) + "/static/arthur/audio.wav")})] + (-> (mix/mix! built (:audio entry) (:store entry)) + (.then (fn [audio] (assoc entry :audio audio))))))))))) (rf/reg-fx ::save! @@ -173,6 +177,29 @@ (js/console.error error) (rf/dispatch [::failed (or (ex-message error) (str error))])))))) +(rf/reg-fx + ::stage! + (fn [_] + (-> (http/GET (str "/api/projects/" (:source-project stage/layout))) + (.then (fn [^js saved] + (or (first (filter #(= (:source-cid stage/layout) (.-cid ^js %)) + (array-seq (.-clips saved)))) + (throw (ex-info "the saved 8625 clip is missing" {}))))) + (.then opened-entry!) + (.then (fn [entry] + (let [built (stage/compose (:clip entry)) + entry (assoc entry :clip built :label (:name built) + :cid "stage-8625" :frames (clip/frames built) + :width (:width built) :height (:height built))] + (-> (mix/mix! built (:audio entry) (:store entry)) + (.then (fn [audio] + (rf/dispatch + [::stage-opened + (store/install! (assoc entry :audio audio) "stage")]))))))) + (.catch (fn [error] + (js/console.error error) + (rf/dispatch [::failed (or (ex-message error) (str error))])))))) + ;; --------------------------------------------------------------------------- ;; events @@ -198,6 +225,28 @@ ::pb/pause! nil ::open! (:id (:project db))}))) +(rf/reg-event-fx + ::load-stage + (fn [{:keys [db]} _] + (if (:busy? (:project db)) + {} + {:db (update db :project merge {:busy? true :status "loading 8625 stage…"}) + ::pb/pause! nil + ::stage! nil}))) + +(rf/reg-event-fx + ::stage-opened + (fn [{:keys [db]} [_ clip-id]] + (let [entry (store/entry clip-id)] + {:db (-> db + (assoc :clip/current clip-id + :clip (select-keys entry [:fps :frames :width :height :audio :display-fps])) + (assoc :project {:id nil :cid nil :name nil :seq nil + :busy? false :status "loaded 8625 stage study"}) + (assoc-in [:playback :frame] 0) + (assoc-in [:playback :playing?] false)) + ::pb/seek! [(:fps entry) (:frames entry) 0]}))) + (rf/reg-event-db ::saved (fn [db [_ id cid label seq written uploaded]] diff --git a/frontend/src/arthur/footage/store.cljs b/frontend/src/arthur/footage/store.cljs index f9f7035..08b7c66 100644 --- a/frontend/src/arthur/footage/store.cljs +++ b/frontend/src/arthur/footage/store.cljs @@ -21,6 +21,8 @@ ([entry] (install! entry "footage")) ([entry kind] (let [id (keyword kind (str (swap! serial inc)))] + (when-let [old (:audio @loaded)] + (when (.startsWith old "blob:") (js/URL.revokeObjectURL old))) (reset! loaded (assoc entry :id id)) id))) diff --git a/frontend/src/arthur/subs/render.cljs b/frontend/src/arthur/subs/render.cljs index ba3b0ce..1b9a6ed 100644 --- a/frontend/src/arthur/subs/render.cljs +++ b/frontend/src/arthur/subs/render.cljs @@ -79,8 +79,10 @@ (rf/reg-sub ::resolver + :<- [::clip] :<- [::timeline] :<- [::store] :<- [::palette] - (fn [[tl store palette] _] - (when tl (timeline/resolver tl store palette)))) + (fn [[document tl store palette] _] + (when tl (clip/resolver (assoc-in document [:timelines :main] tl) + store palette)))) diff --git a/frontend/src/arthur/ui/shell.cljs b/frontend/src/arthur/ui/shell.cljs index 8a9e0b1..0a810eb 100644 --- a/frontend/src/arthur/ui/shell.cljs +++ b/frontend/src/arthur/ui/shell.cljs @@ -72,6 +72,8 @@ ;; the model serialises and a proof nobody can run is not one. [:button {:disabled busy? :on-click #(rf/dispatch [::project/save])} "save"] [:button {:disabled busy? :on-click #(rf/dispatch [::project/open])} "open"] + [:button {:disabled busy? :on-click #(rf/dispatch [::project/load-stage])} + "stage 8625"] [:span.gap] (doall (for [r db/rates] diff --git a/frontend/test/arthur/domain/channel_test.cljs b/frontend/test/arthur/domain/channel_test.cljs index 83d4877..b4438d0 100644 --- a/frontend/test/arthur/domain/channel_test.cljs +++ b/frontend/test/arthur/domain/channel_test.cljs @@ -222,8 +222,18 @@ (is (seq (ch/problems {:animated? true :keys {0 1} :dense {:store "x" :offset 0 :stride 1 :frames 1}})) "one shape at a time") - (is (seq (ch/problems {:animated? true :interp :linear :keys {0 1}})) - "only :hold is implemented")) + (is (empty? (ch/problems (ch/keyed {0 0.0, 10 1.0} :linear))) + "scalar controls can interpolate") + (is (seq (ch/problems (ch/keyed {0 :a, 10 :b} :linear))) + "a cut or tone cannot interpolate")) + +(deftest numeric-channels-can-ramp-between-keys + (let [c (ch/keyed {0 0.0, 10 1.0} :linear) + cursor (ch/cursor c)] + (is (= [0.0 0.5 1.0 1.0] + (mapv #(ch/value-at c %) [0 5 10 15]))) + (is (= [0.0 0.5 1.0 0.2] + (mapv #(ch/sample! cursor %) [0 5 10 2]))))) (deftest component-reads-vectors-and-typed-arrays-the-same-way (is (= 3 (ch/component [3 4] 0))) diff --git a/frontend/test/arthur/domain/symbol_test.cljs b/frontend/test/arthur/domain/symbol_test.cljs new file mode 100644 index 0000000..83cec43 --- /dev/null +++ b/frontend/test/arthur/domain/symbol_test.cljs @@ -0,0 +1,57 @@ +(ns arthur.domain.symbol-test + (:require [cljs.test :refer [deftest is]] + [arthur.demo.stage :as stage] + [arthur.domain.channel :as ch] + [arthur.domain.clip :as clip] + [arthur.domain.leaf :as leaf])) + +(def source + {:name "source" :fps 30 :width 320 :height 200 + :timelines + {:main {:id :main :frames 4 + :nodes {:root {:id :root :kind :group :z "a1"} + :mark {:id :mark :kind :rect :parent :root :z "a1" + :channels {[:xform :pos] (ch/keyed {0 [0 0] 1 [10 0] + 2 [20 0] 3 [30 0]}) + [:geom :size] (ch/framed 4) + [:style :color] (ch/framed :brow)}}}}}}) + +(deftest two-instances-own-their-frame-and-placement + (let [document + (-> source + (assoc-in [:timelines :main] + {:id :main :frames 6 + :nodes {:root {:id :root :kind :group :z "a1"} + :left {:id :left :kind :symbol :of :sym/test + :parent :root :z "a1" :span [0 4] + :channels {[:xform :pos] (ch/framed [100 50])}} + :right {:id :right :kind :symbol :of :sym/test + :parent :root :z "a2" :span [2 6] + :time {:mode :map :at 2 :in 0 :rate 1} + :channels {[:xform :pos] (ch/framed [120 50])}}}}) + (assoc-in [:timelines :sym/test] + (assoc (get-in source [:timelines :main]) :id :sym/test))) + resolve (clip/resolver document nil) + at (fn [f] (mapv (juxt :node :cx) (resolve f)))] + (is (empty? (clip/problems document))) + (is (= [[[:left :mark] 110]] (at 1))) + (is (= [[[:left :mark] 120] [[:right :mark] 120]] (at 2))) + (is (= [[[:right :mark] 150]] (at 5))) + (is (= document (leaf/clip "stage" (leaf/leaves "stage" document)))))) + +(deftest stage-fixture-keeps-source-as-one-symbol + (let [document (stage/compose source)] + (is (empty? (clip/problems document))) + (is (= #{:main :sym/face-8625} (set (keys (:timelines document))))) + (is (= :sym/face-8625 (get-in document [:timelines :main :nodes :left :of]))) + (is (= :sym/face-8625 (get-in document [:timelines :main :nodes :right :of]))) + (is (= [48 280] (get-in document [:timelines :main :nodes :right :span]))) + (is (= :right (get-in document [:timelines :main :nodes :voice-right :linked-to]))) + (is (= [48 260] (get-in document [:timelines :main :nodes :voice-right :span]))) + (is (= 0.5 (ch/value-at + (get-in document [:timelines :main :nodes :voice-right + :channels [:audio :gain]]) 54))) + (is (< -0.8 (ch/value-at + (get-in document [:timelines :main :nodes :voice-right + :channels [:audio :pan]]) 110) 0.7)) + (is (= document (leaf/clip "stage" (leaf/leaves "stage" document))))))