noe/arthur
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Stream block uploads and compress cached mouth crops

Browse source
This commit is contained in:
Olive Vaughn 2026-09-28 14:37:09 -04:00
parent 65ad67c129
commit a611b86c0d
9 changed files with 195 additions and 16 deletions
Download patch file Download diff file Expand all files Collapse all files

4
frontend/src/arthur/domain/wire.cljs
View file

@ -15,8 +15,8 @@
a codec that returned a sorted map would work locally and stop working after one
round trip, which is the failure the plain-map rule already prevents.
The bytes are separate and base64, because tier 2 is typed arrays and transit
has nothing to say about them. `channel/dense-at` reads a block as
The bytes are separate from transit: JSON block reads carry base64, while
uploads send binary file parts. `channel/dense-at` reads a block as
`{:data <typed array> :state <Uint8Array>}` and both sides of the wire must hold
byte-for-byte the same array — a handle that names a sha256 has to name the
bytes you actually hold."

27
frontend/src/arthur/events/project.cljs
View file

@ -23,6 +23,7 @@
fx, which is the only thing in this namespace that is not pure."
(:require [arthur.domain.clip :as clip]
[arthur.domain.project :as project]
[arthur.domain.wire :as wire]
[arthur.events.playback :as pb]
[arthur.footage.store :as store]
[arthur.flow.address :as address]
@ -38,6 +39,20 @@
(defn- block-keys [^js doc]
(into-array (map #(.-key %) (array-seq (.-blocks doc)))))
(defn- block-bytes [value]
(if (string? value)
(wire/bytes-of value)
(js/Uint8Array. (.-buffer value) (.-byteOffset value) (.-byteLength value))))
(defn- block-form [^js block]
(let [form (js/FormData.)]
(.append form "key" (.-key block))
(.append form "descriptor" (.-descriptor block))
(.append form "data" (js/Blob. #js [(block-bytes (.-data block))]) "block.bin")
(when-let [state (.-state block)]
(.append form "state" (js/Blob. #js [(block-bytes state)]) "state.bin"))
form))
(defn- upload-missing!
"POST the blocks the server said it does not have, and nothing else.
@ -45,9 +60,9 @@
this and it made sqlite answer \"database is locked\" on a save — which reaches
the page as a 500 with nothing wrong with the request. The backend was fixed too
(WAL, and a busy timeout, in server/settings.py), and this stays sequential
anyway: the uploads are a few kilobytes each, nothing is waiting on them, and a
burst of parallel writes to buy nothing is how the same bug comes back the first
time a take has sixty blocks instead of eleven."
anyway: most uploads are small, and a burst of parallel writes to buy nothing
is how the same bug comes back the first time a take has sixty blocks instead
of eleven."
[^js doc]
(-> (http/POST "/api/blocks/missing" #js {:keys (block-keys doc)})
(.then (fn [^js answer]
@ -55,7 +70,9 @@
todo (filterv #(contains? missing (.-key ^js %))
(array-seq (.-blocks doc)))]
(-> (reduce (fn [chain block]
(.then chain (fn [_] (http/POST "/api/blocks" block))))
(.then chain
(fn [_]
(http/POST-form "/api/blocks" (block-form block)))))
(js/Promise.resolve nil)
todo)
(.then (fn [_] (count todo)))))))))
@ -104,7 +121,7 @@
(.then (fn [_]
(when (seq source-blocks)
(-> (upload-missing!
#js {:blocks (source/wire-blocks source-blocks)})
#js {:blocks (source/upload-blocks source-blocks)})
(.then (fn [_]
(http/PUT
(str "/api/analyses/" (:id analysis))

7
frontend/src/arthur/flow/source.cljs
View file

@ -75,6 +75,13 @@
#js {:key key :descriptor descriptor :data (wire/base64 data)}))
roles)))
(defn upload-blocks [blocks]
(into-array
(map (fn [role]
(let [{:keys [key descriptor data]} (get blocks role)]
#js {:key key :descriptor descriptor :data data}))
roles)))
(defn unpack
"The three block-detail responses -> inputs for measurement and freeze."
[^js responses [width height]]