# arthur — structure How the suite is laid out once it is a suite: ClojureScript and re-frame, a clip as the unit of work, keying as its own step, and painting as a peer of rotoscoping rather than a sketch bolted to the side. `docs/design.md` says what the thing is and why. This says where the code goes. `docs/animation-model.md` specifies the data both of them are about — nodes, channels, symbols and time maps — and supersedes this document wherever the two describe the same type. Nothing here revises an aesthetic decision; several things here split a decision that is currently made in two places at once. ## What is actually wrong with the current shape The pure modules are fine. `pipeline.js`, `mathutil.js`, `landmarks.js`, `interior.js` and `raster.js` are functions over data with the reasoning written down next to them, and they port nearly verbatim. `app.js` is the whole problem, and not because it is long. It is long because six unrelated jobs are braided together in it: - reading knob values out of the DOM (`opts()`), - deriving everything from them (`rebuild`, `buildEyes`, `buildBrows`, `resolveTeeth`), - resolving what is on screen at a frame (`plateIndex`, `perfIndex`, `leadIndex`), - rasterising (`renderFrame`, `compositeRender`), - driving the clock (`tick`), - persisting (`saveCels`, `loadCels`). Every one of those is a different layer, and `rebuild` recomputes all of them whenever any knob moves. That is affordable at one clip and 72 frames and is not affordable at a project. The re-frame port is worth doing chiefly because the subscription graph *is* the staged dataflow `docs/design.md` already describes — "three artifacts, not two" is a dependency graph written in prose. ## Three frame spaces, named Most of the future bugs live here, so name them before anything else. | Space | Symbol | Meaning | | --- | --- | --- | | **source frame** | `sf` | Index into the footage and the dense track. What detection produced. | | **clip frame** | `cf` | 0-based within a clip. `cf = sf - (:in clip)`. | | **sequence frame** | `qf` | Position on the timeline. `qf = (:at clip) + cf`. | **Nothing authored is ever stored in sequence-frame space.** Keys, cels, overrides and kept-frame sets are all in clip-frame space, so a clip slides on the timeline without a single stored number changing. Exposure and lead are transforms *within* clip space: Detection retains every source frame. A chosen picture fps samples the frozen roto in clip time; it changes neither source-frame count nor the audio clock. The set of source frames an artist uses as cel tracing references is another selection, independent of the picture fps. ```clojure (defn pose-frame [clip cf] (-> cf (expose (:exposure clip)) (shift (:lead clip) (count-frames clip)))) ``` which is the composition `app.js` currently performs in `leadIndex` and `perfIndex` with the order implicit. Expose first, then lead: exposure floors onto a grid and lead deliberately reads the future, so leading first and then flooring would quietly discard the lead on most frames. ## The entity model ``` project ├── palette authored ramp; indices, never RGB ├── character plate library, per-plate slots ├── footage[] frames + audio + manifest (fps, w, h) ├── analysis[] dense track, cached, keyed by footage+model+version ├── clip[] the unit of work │ ├── source footage id, analysis id, in/out in sf │ ├── exposure, lead timing, in cf │ ├── scene node tree │ ├── channels node+property -> keyframe stream, in cf │ ├── overrides (node, property, cf) -> value, applied last │ └── cels painted vector layers, keyed by chosen clip frames └── sequence[] clip placements: {clip-id, at, in, out} ``` `clip` is the entity the current app has exactly one of and never names. Giving it a name is most of the work: `state` in `app.js` is a clip with its analysis inlined and its palette global. Cel keys select where drawings begin and how long they hold. The source frames shown beneath a cel while tracing are chosen independently, and picture fps only controls which analyzed pose the finished roto displays at a given time. ### Two things called "track" `docs/design.md` says "dense track" for the landmark stream. A timeline also wants tracks. Pick now: the landmark stream is **analysis**, a keyframe stream is a **channel**, and the word *track* is reserved for a row on the timeline. Renaming this later costs a day. ## The node, decomposed The prototype today gives a part a `parent` and a `clip`, and `parent` is doing nothing except documenting intent — `mouth_in` is already in the same head-local raster space as `mouth`, so composing its transform would be composing identity. The moment a painted cel is attached to a head plate that moves, transform composition becomes real, and the three ideas currently sharing two fields have to come apart: | Field | What it does | Wrong to conflate because | | --- | --- | --- | | `:parent` | Transform composition. Child geometry is in parent's local space. | A node can be drawn over its parent without inheriting its motion. | | `:stencil` | Colour-key clip: write only where the buffer already holds that index. | The iris is stencilled by the sclera and parented to the lid ring; those are different nodes. | | `:z` | Draw order. | Order is authored per scene, not implied by the tree. | A node is then: ```clojure {:id :eye-r/iris :source {:kind :primitive ...} ; the five kinds from design.md :parent :eye-r/lid :stencil :eye-r/sclera :z 42 :color :iris ; palette key, never a hex :interp :hold} ``` `:source` carries the taxonomy `docs/design.md` already has — `:plate`, `:feature`, `:interior`, `:primitive`, `:scalar` — plus `:cel` for painted vectors and `:group` for a pure transform node. That is the decomposition that makes painting a peer rather than an annex: `paint.js` is not a special case, it is a source kind whose channels happen to be authored by hand instead of measured. `docs/design.md` already promises "pluggable sources"; this is the data shape that keeps the promise. Each node also gets an optional local transform channel — translate, rotate, scale, squash. That is the thing the current tool cannot express and that the `slot_mouth` / `scale` / `rot` / `squash` fields in the prototype are reaching for. ## The flow, in seven stages The user-facing flow is `frames -> analysis -> keying -> geometry -> palette`. Analysis is three stages, not one, and the split is the most load-bearing decision in this document, because **exactly one boundary is a cache boundary.** | # | Stage | In | Out | Cost | | --- | --- | --- | --- | --- | | 1 | **ingest** | video | footage: a seekable H.264 proxy, tracing stills, audio, manifest | minutes, in-app | | 2 | **detect** | the proxy, walked one frame at a time | raw landmarks per frame | minutes, **cached** | | 3 | **measure** | landmarks | anchor fit, residual, head-local rings, signals, interior pixels | seconds | | 4 | **condition** | measurements | smoothed transforms and contours | milliseconds | | 5 | **key** | conditioned signals + policy | channels: sparse keys, quantised holds, kept frames | milliseconds | | 6 | **resolve** | channels + scene + exposure + lead + overrides | geometry per output frame | per frame | | 7 | **palette** | geometry | indexed raster | per frame | Stage 2 is the only artifact worth persisting and the only one worth a progress bar. Stage 3 reads pixels — teeth extraction is here, and that is why it is here rather than in keying: **after stage 3, nothing downstream may look at a source pixel.** Stage 4 is separated from 3 only because `contour avg` and `anchor avg` are knobs and the rest of stage 3 is not; splitting them means dragging that slider does not re-run the interior extraction. That last clause is the guarantee, and it is narrower than the table's ordering looks. Stage 3 is not one pass that finishes before stage 4 begins. The anchor fit is knob-free; conditioning smooths its four parameters; the head-local rings are then measured *through* the conditioned transform — so `anchor avg` does re-run the ring mapping, which is a few hundred frames of twenty points and free. What it must not re-run is the part that reads a source pixel, and that part takes the landmarks and the frames and never the transform, so it does not. Built as `measure/anchor` → `condition/anchor` → `measure/mouth` → `condition/contours`, and stage 7's eyes, brows and interior follow the same shape. ### Where the current code lands | Now | Stage | | --- | --- | | `stabilize` | 3 measure | | `eyeSignals`, `browSignals`, `pairIrises`, `pairBrows` | 3 measure | | `extractTeeth` | 3 measure — it reads pixels | | `smoothTransforms`, `smoothContours` | 4 condition | | `selectKeys`, `suggestPlateFrames` | 5 key | | `quantizeSnap`, `resolveBlink` | 5 key — a held value *is* a key | | `exposeIndex`, `shiftIndex`, `activeKey`, `heldFrame` | 6 resolve | | `toRasterRing`, iris placement at ring slots, `offsetRing` | 6 resolve | | `IndexedRaster`, `drawCel` | 7 palette | | `drawRegistered`, `posterizeInto` | reference only, off to the side | Three things in `app.js` currently straddle a boundary, and each straddle is a bug waiting for a bigger project: - **`resolveTeeth`** extracts from pixels *and* applies contrast/dwell/blob knobs. Extraction is stage 3 and cached with the track; resolution is stage 5. Today, nudging `teeth dwell` re-runs Otsu over every frame. - **`buildEyes`** measures, quantises, *and* places the iris on the subsampled lid ring. Those are stages 3, 5 and 6. The placement argument in `docs/design.md` — that the iris is read off the already-smoothed ring — is a stage-6 fact and stays true; it just needs to be *in* stage 6. - **`buildBrows`** likewise: measure the height out of the ring (3), quantise it (5), put it back (6). The doc's warning about moving the brow twice is precisely a warning about these being one function. ### What each knob invalidates This table is the argument for the split, and it should be derivable from the sub graph rather than maintained by hand. As built, half of it is, by a different route. `flow/address/block-knobs` is this table for tier 2 — per block, the settings its bytes depend on — and `address-test` asserts it by biconditional rather than deriving it, which is stronger than a derivation would have been: a derived table is only as right as the graph it reads. What is not written down anywhere is the rest of the column. Stages 3 to 5 are one call chain in `flow/take/measure` rather than a chain of subs, so there is no graph above `::base-scene` to read a dependency off, and the tier-1 half of a re-freeze is therefore whole-clip. An earlier arrangement had a second table in `domain/params`, per feature area, saying which areas a knob affected. It disagreed with the tested one on the first entry where the two granularities part company and it had no caller; see that namespace for why one checked table beats two. | Knob | Invalidates from | | --- | --- | | model / footage | 2 detect | | `teeth contrast`, `cavity erode`, `blob grow`, `tongue reject`, `prefer upper` | 3 measure (mouth crop only) | | `contour avg`, `anchor avg` | 4 condition | | `teeth dwell`, `blink cut/hold/dwell`, `gaze step/dwell`, `brow step/dwell`, `suggest tolerance` | 5 key | | `vertices`, `eye vertices`, `brow vertices`, `teeth vertices` | 5 key | | `exposure`, `mouth lead`, kept-frame edits, overrides | 6 resolve | | `lash line`, `iris size`, `pupil`, `brow weight` | 6 resolve | | palette edits, colour assignment | 7 palette | ## Namespaces ``` src/arthur/ domain/ pure data, specs, ops. No re-frame, no DOM. palette.cljs ramp; index lookup; the no-RGB rule as a spec landmarks.cljs index tables (port verbatim) ring.cljs ordered traversal: subsample, offset, simplicity geom.cljs similarity fit, procrustes, moving average node.cljs scene node: source, parent, stencil, z timeline.cljs node tree: topo order, transform composition channel.cljs keyframe stream: active-key-at, hold semantics clip.cljs clip entity; the frame-space conversions cel.cljs painted vector layers timeline.cljs sequence: clip placement, qf <-> cf flow/ ingest.cljs detect.cljs the MediaPipe boundary, and the only one measure/ anchor.cljs stabilise: procrustes, similarity, residual mouth.cljs eyes.cljs openness, gaze, iris pairing vote brows.cljs raise/tilt, ring and end pairing votes interior.cljs otsu, morphology, components, radial contour condition.cljs key.cljs velocity minima, dwell, quantise, blink, decimate resolve.cljs scene eval at a frame; exposure, lead, overrides raster.cljs indexed scanline fill, stencil, disc, rect reference.cljs registered underlay, posterise db.cljs app-db schema + spec store.cljs handles for things too big for app-db clock.cljs audio clock, outside app-db events/ one ns per domain area subs/ one ns per stage ui/ shell.cljs mode/ one ns per tool mode panel/ strip, worksheet, readouts, palette, layers canvas.cljs the one imperative sink fx/ mediapipe, files, audio, persistence ``` Two rules about this tree. `domain/` may not require `flow/`, and neither may require `re-frame`. And `flow/` namespaces take explicit arguments — no namespace below `subs/` ever calls `subscribe`. ## app-db, and what is not in it **app-db holds authored data and ids. Nothing derived, and nothing large.** That sounds like ordinary hygiene and it is not: it is the precondition for two features that are otherwise unbuildable. - **Spec validation on every event.** Worth having, affordable only over authored data. - **Cheap writes.** Every edit `assoc`es into app-db, and every mounted layer-2 sub compares the result. Both are structural-sharing operations and stay cheap only while the map is small. Undo is *not* on that list, and an earlier draft said it was. See Collaboration: snapshot-based undo is right single-player and wrong once two people edit. So the dense track and the decoded frames live in `arthur.store`, a `defonce` map of id to JS object, and app-db holds `{:analysis/id "sha-..."}`. Landmarks stay as they arrive — arrays of `{x, y, z}`, or better, a flat `Float32Array` — and are *not* converted to Clojure maps. 478 points × 600 frames is 286,800 maps, allocated for nothing: no sub ever needs to diff them, and every stage that touches them walks all of them anyway. ### The playhead, and where per-frame work goes A reaction propagates downstream only when its own **output value** changes, not when one of its inputs notifies. So with layer-2 extractors and layer-3 computations, a playhead tick costs one cheap extractor run per mounted layer-2 sub, each of which returns the same value for every subtree the tick did not touch and therefore notifies nobody. `::measurements`, `::channels` and `::resolver` do not re-run. The `:<-` graph is what buys that, and it buys it whether or not the playhead is in app-db. So **the playhead lives in app-db**, like everything else. An earlier draft of this document put it in a standalone ratom to avoid an invalidation storm that does not happen. Two independent reasons it belongs in db: - `[:playback/seek ...]` in the event log is how scrubbing becomes inspectable in re-frame-10x. - **A collaborator's playhead is a feature.** Putting it outside app-db puts it outside the machinery that shares it. See Collaboration. What is true is narrower, and is about sub authoring and interceptors rather than about the playhead: - **Expensive work must not live in a layer-2 sub.** A sub that derefs app-db directly re-runs on every db change whatever changed. That is the actual content of the folklore about re-frame and canvas. - **Do not put a per-frame event behind a global interceptor that walks the whole db** — a spec-validating `after`, or `std-interceptors/debug` and its `clojure.data/diff`. At 30fps that is thirty full-db traversals a second. The transport event carries its own interceptor chain and is excluded from the global ones. - **The render sink is not a Reagent component.** Stage 7 writes bytes into a canvas from an rAF loop; it is not a view that re-renders. Stage 6 is still arranged so the per-frame path is a lookup, not a computation: ```clojure ;; recomputes when channels, scene, exposure, lead or overrides change (rf/reg-sub ::resolver ...) ;; => (fn [cf] geometry), or an index into a bake ;; the rAF loop: reads, blits, dispatches nothing (defn tick [] (let [cf (clock/clip-frame)] (raster/paint! (@resolver cf)) (canvas/blit!))) ``` The sub produces a *resolver*; the loop applies it. A knob change costs one sub recomputation; a frame costs a lookup and a blit. ## Events and subs Events are named for intent, and carry the clip they apply to: ```clojure [:clip/set-exposure clip-id 2] [:clip/toggle-kept clip-id cf] [:keying/re-key clip-id] ; not a setter; policy unchanged, redo the work [:keying/suggest clip-id] [:paint/commit-shape clip-id node-id cel-frame pts] [:node/set-parent clip-id node-id parent-id] [:override/set clip-id node-id :pts cf value] [:timeline/move-clip seq-id clip-id qf] ``` Setters are fine where the intent *is* the value — `set-exposure` is honest. Where they differ, name the intent: `:keying/re-key` takes no value and is not `:clip/set-keys`. Subs mirror the stages one-to-one, so the graph and the table above are the same object: ``` ::footage -> ::landmarks -> ::measurements -> ::conditioned -> ::channels -> ::resolved -> ::raster ``` Each is a layer-3 sub over the previous plus the parameters for its stage only. That is what makes "drag `exposure`" recompute `::resolved` and nothing above it. ## Tool modes A mode is a namespace, not a branch. Each exposes a map: ```clojure {:id :paint/pen :cursor "crosshair" :keymap {"Enter" [:paint/close-shape] "Escape" [:paint/abort]} :on-pointer (fn [ev ctx] ...) :overlay (fn [g ctx] ...) ; draws handles, never output pixels :enter :exit (fn [ctx] ...)} ``` with two rules worth stating because the current `PaintUI` breaks both and gets away with it at this size: - **In-progress gesture state is mode-local**, in a ratom the mode owns — not in app-db. An unclosed polygon must not appear in the undo history, and a drag must produce one undo entry rather than one per pointermove. The mode dispatches a single `:paint/commit-shape` on release. - **Overlays draw to a separate canvas.** Handles, vertex boxes and the green close-target are not indexed pixels and must never be in the raster. The current code composites them into the same context as the output, which is fine for a preview and lies to you the moment you want to judge the look. The palette-index constraint and grid snapping stay where they are: they belong to `domain/cel` and `domain/palette`, enforced at commit, so no tool can bypass them. ## Where things live Superseded in detail by Serialization and Baking below; the local picture is: | Thing | Where | Keyed by | | --- | --- | --- | | project (tier 1) | server, plus a local autosave copy | project id, then leaf path | | footage frames, audio (tier 3) | on disk / OPFS, untouched | content hash | | analysis artifact, geometry bakes (tier 2) | OPFS, with IndexedDB for the index | content hash over every input | | render output | never persisted | — | The cache key on the analysis artifact must include the **detector version**. A model upgrade that silently reuses old landmarks presents as "the tool got worse" with no event to attach it to — which is the argument for content-addressing rather than version-numbering everything derived. ## Baking "No analysis during playback" is `docs/design.md`'s three-artifact rule with a number attached. The expensive stages are 2 (detect), 3 (measure) and, over a long take, 4 (condition). Stages 5–7 are arithmetic over small arrays. So there are two bakes, not one, and they answer different questions. ### Bake A — the analysis artifact. Mandatory. Stages 2–4 for one footage at one set of conditioning parameters. This is what must never run while the transport is moving. | Field | Shape | | --- | --- | | landmarks | `Float32Array[n_frames × 478 × 2]` | | transforms | `Float32Array[n_frames × 4]` — s, θ, tx, ty | | residual | `Float32Array[n_frames]` | | head-local rings | `Int16Array` per ring table, isotropic fixed point | | signals | openness, gaze, brow ends, aperture — `Float32Array[n_frames]` each | | **mouth crops** | `Uint8Array`, one small crop per frame | The mouth crops are the non-obvious entry, and they are what makes remote work possible at all. `extractTeeth` reads source pixels, so without them a collaborator holding the analysis but not the 600 source PNGs cannot touch a single teeth knob. A 40×30 crop is about 1.2KB; a 600-frame take is under a megabyte against hundreds for the footage. ### Bake B — resolved geometry. For scale. Stage 6 output per node. Not needed for one clip — resolving a frame is a key lookup and a transform — and needed the moment six roto tracks are live at 30fps. **Fixed topology is what makes this flat.** Because every key of a part carries the same vertex count with the same vertex meanings, a node's whole bake is a rectangular array with no per-frame header and no indirection: ``` pts Int16Array[n_frames × n_verts × 2] raster space, already grid-snapped state Uint8Array[n_frames] hidden flag + palette index ``` Frame `f` of a node is the subarray at `f * n_verts * 2`. 600 frames × 20 verts is 48KB; a twelve-node clip about 600KB; six roto tracks about 3.5MB. So this is the payoff on the aesthetic constraint rather than a cost of it — a variable vertex count would force a per-frame offset table and a scan. ### Rasters are not baked Stage 7 is the cheapest stage and the one most worth keeping live: palette and colour assignment are the last decisions and the ones you want to change in motion. Baking to pixels would freeze exactly the loop that should be instant. Small raster *thumbnails* for the strip and the timeline are a separate artifact with a separate cache. ### Unbaking is not an inverse The bake never replaces its input. Authored state — channels, params, overrides — is always retained and always authoritative, so "unbake this roto and edit it" is a boolean about which side of the cache the renderer reads, not a computation. Instant by construction. Three rules keep it that way: - **No destructive bake.** Nothing is discarded when something is baked. - **Hand corrections never go into the bake.** They are `:overrides`, applied at stage 6 *after* the bake is read, so a correction survives a rebake. This is `docs/design.md`'s "hand corrections must not live in the take", and it is the rule that stops baking from becoming a trap. - **Bakes are content-addressed** by a hash over every input that produced them: footage hash, detector version, and the parameters of each stage at or above the bake line. A stale bake is then unreachable rather than wrong, and a collaborator's bake is fetchable by the same key — see Collaboration. ### Partial bakes Bake per node, per frame range. Scrubbing into an unbaked range resolves on demand and fills the cache; a worker bakes ahead of the playhead. Show it: a bake-state bar under the timeline is an affordance every NLE has already taught people to read, and it is honest about what is ready. ## Serialization, in three tiers Cut by mutability and size. The cut is what makes collaboration and baking both tractable, because it decides what is allowed on the wire. | Tier | What | Size | Synced | Undoable | | --- | --- | --- | --- | --- | | 1 **authored** | palette, clips, scene graphs, channels, cels, overrides, sequences | KB | yes — it *is* the document | yes | | 2 **derived** | analysis artifacts, geometry bakes, thumbnails | MB | no — content-addressed blobs, fetched | no | | 3 **source** | frames, audio | MB–GB | no — immutable, by hash | no | Tier 3 is produced **by the app**, not by `extract.sh`: wasm ffmpeg decodes the clip to frames and audio in the browser, and they are uploaded and served as content-addressed blobs. That makes tier 3 the same kind of thing as tier 2 — a cache with a hash — and it removes the one step that currently needs a terminal. Two consequences worth planning for: the extraction rate is recorded by the code that chose it rather than by a `manifest.json` a human might edit, and **offline work needs the frames already fetched**, so the local blob cache is what makes a plane usable. Decoding server-side instead is the same data model with a different worker; the format does not care which side runs it. **Only tier 1 is the document.** A bake inside the shared document is a system that puts 48KB on the wire per vertex drag, and it is unnecessary, because tier 2 is a pure function of tiers 1 and 3. Tier 1 stays small enough to read. The only vertex data in it is what a human placed by hand: cel polygons and overrides. Everything traced is tier 2. ### Keys are a map, not a vector ```clojure ;; not this {:keys [{:f 0 :pts [...]} {:f 4 :pts [...]}]} ;; this {:keys {0 {:pts [...]}, 4 {:pts [...]}}} ``` `activeKey` becomes a lookup in a sorted map rather than a scan; a single key becomes addressable as a path; and two people keying different frames of one part merge field-wise with no merge algorithm at all. `selectKeys` returns an array today — change it before anything depends on the order. ### Serving tiers 2 and 3 Built at step 9. Above this point the tiers are a rule about what is allowed on the wire; this is the shape that enforces it. **One store for both, named by the sha256 of the bytes.** Once tier 3 is decoded by the app rather than by a shell script it becomes the same kind of thing as tier 2 — a cache with a hash — so there is one place that writes bytes, one that reads them, and one URL shape: ``` GET /blob/ raw bytes, Cache-Control: immutable ``` `immutable` is not optimism there, it is the definition: the name IS the hash of the content, so a cached copy cannot be stale. That is what makes serving six hundred frames out of it cheap enough to do on every load. **Two kinds of hash, and they are not the same hash.** A blob is named by the hash of its BYTES, which is what makes an identical frame in two extractions one file. A derived thing — an analysis artifact, a dense block — is named by a hash over its INPUTS, which is what lets a client ask for the block the current settings want *before* anything has computed it, and what makes a stale bake unreachable rather than wrong. So a `Block` row has both: `key` over the inputs, and a foreign key to the blob whose digest is over the bytes. Conflating them would break the half of addressing that answers questions about work not yet done. ``` POST /api/analyses {key, descriptor} idempotent GET /api/analyses/ metadata + source block keys PUT /api/analyses/ link dense landmarks, mask, crops POST /api/blocks/missing {keys} -> {missing} POST /api/blocks {key, descriptor, data, state} GET /api/blocks/ GET /api/footage/ the manifest: the proxy to measure, audio, a URL per tracing still GET /blob/ immutable bytes, and RANGE-capable so a