3d3c1bbca0
A lane's drawings were going to be one instance whose source was a KEYED
channel: frame 0 says `:drawing-a`, frame 4 says `:drawing-b`, and the cels of
a row are that channel's keys. Two things followed from it, and both were
wrong.
The first is that playback meant whichever shape the channel happened to have.
A framed source played its symbol; a keyed source froze the selected frame.
So `node/placed-at` read animation out of storage, and adding an ordinary key
to a still turned it into an animation — the last-key bug, which was not a bug
in the code so much as the rule working as written. But WHICH drawing is used
and HOW time runs inside it are independent questions, and all four combinations
are ordinary: hold one drawing, play one animation, cut between held drawings,
cut between playing ones.
So an occurrence names one symbol in `:source {:symbol ...}` and says how its
source time advances in `:playback {:in :speed :end}` — `source = in + speed *
f`, a hold being speed 0, with `:stop`, `:hold` or `:loop` at the end named
rather than guessed. `node/placed-frame` samples it forwards, which works for
holds too, and `node/source-time` is the separate, invertible edit map, nil
where inversion is meaningless. The two were one function before, and a hold
had to lie about one of them.
The second is that a keyed source only looked necessary because an occurrence
was assumed to need a ROW. It does not. A lane is a group with `:layout
:sequence`, its occurrences are ordinary instances in the same flat node map,
and `timeline/rows` draws them as cel blocks on the lane's own row: twelve
exposures, one row, each cel still separately selectable and addressable. The
vertical growth that justified the keyed source is a presentation question, and
it is answered in the view.
`arthur.domain.sequence` holds the first commands over that shape — add lane,
append drawing, extend hold — each one history step, each refusing rather than
half-applying. Extending a hold leaves the lane's keys at their authored times,
because you are adjusting drawings underneath timed motion; a correction owned
by an occurrence travels with it. Ownership does that work, so no key needs a
flag saying what it follows. Ripple past the symbol's end is refused with the
frame count it would need, and `:extent :grow-symbol` is the caller saying yes.
`clip/blank` no longer carries `:subjects {} :features {} :groups {}`. Empty
maps write no leaf, so a blank document could not survive its own round trip —
`leaf/leaves` promises exactness and was the only honest side of that.
Documents are schema 3. A version 2 document is not read; nothing here converts
one. `docs/lane-model.md` is the design, and says which of its parts are built.
392 tests, 5,525 assertions, and `test/browser/sequence.mjs` drives the editor
through create, hold, explicit overflow and undo.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
1059 lines
53 KiB
Markdown
1059 lines
53 KiB
Markdown
# arthur — structure
|
||
|
||
How the suite is laid out once it is a suite: ClojureScript and re-frame, a
|
||
clip as the unit of work, keying as its own step, and painting as a peer of
|
||
rotoscoping rather than a sketch bolted to the side.
|
||
|
||
`docs/design.md` says what the thing is and why. This says where the code goes.
|
||
`docs/animation-model.md` specifies the data both of them are about — nodes,
|
||
channels, symbols and time maps — and supersedes this document wherever the two
|
||
describe the same type.
|
||
The newer [Lane Model](lane-model.md) takes precedence for occurrence ownership,
|
||
playback semantics, shared editing operations, and multi-view UX. It explicitly
|
||
allows replacing the current format without backward compatibility.
|
||
Nothing here revises an aesthetic decision; several things here split a
|
||
decision that is currently made in two places at once.
|
||
|
||
## What is actually wrong with the current shape
|
||
|
||
The pure modules are fine. `pipeline.js`, `mathutil.js`, `landmarks.js`,
|
||
`interior.js` and `raster.js` are functions over data with the
|
||
reasoning written down next to them, and they port nearly verbatim.
|
||
|
||
`app.js` is the whole problem, and not because it is long. It is long because
|
||
six unrelated jobs are braided together in it:
|
||
|
||
- reading knob values out of the DOM (`opts()`),
|
||
- deriving everything from them (`rebuild`, `buildEyes`, `buildBrows`,
|
||
`resolveTeeth`),
|
||
- resolving what is on screen at a frame (`plateIndex`, `perfIndex`, `leadIndex`),
|
||
- rasterising (`renderFrame`, `compositeRender`),
|
||
- driving the clock (`tick`),
|
||
- persisting (`saveCels`, `loadCels`).
|
||
|
||
Every one of those is a different layer, and `rebuild` recomputes all of them
|
||
whenever any knob moves. That is affordable at one clip and 72 frames and is
|
||
not affordable at a project. The re-frame port is worth doing chiefly because
|
||
the subscription graph *is* the staged dataflow `docs/design.md` already
|
||
describes — "three artifacts, not two" is a dependency graph written in prose.
|
||
|
||
## Three frame spaces, named
|
||
|
||
Most of the future bugs live here, so name them before anything else.
|
||
|
||
| Space | Symbol | Meaning |
|
||
| --- | --- | --- |
|
||
| **source frame** | `sf` | Index into the footage and the dense track. What detection produced. |
|
||
| **clip frame** | `cf` | 0-based within a clip. `cf = sf - (:in clip)`. |
|
||
| **sequence frame** | `qf` | Position on the timeline. `qf = (:at clip) + cf`. |
|
||
|
||
**Nothing authored is ever stored in sequence-frame space.** Keys, cels,
|
||
overrides and kept-frame sets are all in clip-frame space, so a clip slides on
|
||
the timeline without a single stored number changing. Exposure and lead are
|
||
transforms *within* clip space:
|
||
|
||
Detection retains every source frame. A chosen picture fps samples the frozen
|
||
roto in clip time; it changes neither source-frame count nor the audio clock.
|
||
The set of source frames an artist uses as cel tracing references is another
|
||
selection, independent of the picture fps.
|
||
|
||
```clojure
|
||
(defn pose-frame [clip cf]
|
||
(-> cf (expose (:exposure clip)) (shift (:lead clip) (count-frames clip))))
|
||
```
|
||
|
||
which is the composition `app.js` currently performs in `leadIndex` and
|
||
`perfIndex` with the order implicit. Expose first, then lead: exposure floors
|
||
onto a grid and lead deliberately reads the future, so leading first and then
|
||
flooring would quietly discard the lead on most frames.
|
||
|
||
## The entity model
|
||
|
||
```
|
||
project
|
||
├── palette authored ramp; indices, never RGB
|
||
├── character plate library, per-plate slots
|
||
├── footage[] frames + audio + manifest (fps, w, h)
|
||
├── analysis[] dense track, cached, keyed by footage+model+version
|
||
├── clip[] the unit of work
|
||
│ ├── source footage id, analysis id, in/out in sf
|
||
│ ├── exposure, lead timing, in cf
|
||
│ ├── scene node tree
|
||
│ ├── channels node+property -> keyframe stream, in cf
|
||
│ ├── overrides (node, property, cf) -> value, applied last
|
||
│ └── cels painted vector layers, keyed by chosen clip frames
|
||
└── sequence[] clip placements: {clip-id, at, in, out}
|
||
```
|
||
|
||
`clip` is the entity the current app has exactly one of and never names. Giving
|
||
it a name is most of the work: `state` in `app.js` is a clip with its analysis
|
||
inlined and its palette global.
|
||
|
||
Cel keys select where drawings begin and how long they hold. The source frames
|
||
shown beneath a cel while tracing are chosen independently, and picture fps
|
||
only controls which analyzed pose the finished roto displays at a given time.
|
||
|
||
### Two things called "track"
|
||
|
||
`docs/design.md` says "dense track" for the landmark stream. A timeline also
|
||
wants tracks. Pick now: the landmark stream is **analysis**, a keyframe stream
|
||
is a **channel**, and the word *track* is reserved for a row on the timeline.
|
||
Renaming this later costs a day.
|
||
|
||
## The node, decomposed
|
||
|
||
The prototype today gives a part a `parent` and a `clip`, and `parent` is doing
|
||
nothing except documenting intent — `mouth_in` is already in the same
|
||
head-local raster space as `mouth`, so composing its transform would be
|
||
composing identity. The moment a painted cel is attached to a head plate that
|
||
moves, transform composition becomes real, and the three ideas currently sharing
|
||
two fields have to come apart:
|
||
|
||
| Field | What it does | Wrong to conflate because |
|
||
| --- | --- | --- |
|
||
| `:parent` | Transform composition. Child geometry is in parent's local space. | A node can be drawn over its parent without inheriting its motion. |
|
||
| `:stencil` | Colour-key clip: write only where the buffer already holds that index. | The iris is stencilled by the sclera and parented to the lid ring; those are different nodes. |
|
||
| `:z` | Draw order. | Order is authored per scene, not implied by the tree. |
|
||
|
||
A node is then:
|
||
|
||
```clojure
|
||
{:id :eye-r/iris
|
||
:source {:kind :primitive ...} ; the five kinds from design.md
|
||
:parent :eye-r/lid
|
||
:stencil :eye-r/sclera
|
||
:z 42
|
||
:color :iris ; palette key, never a hex
|
||
:interp :hold}
|
||
```
|
||
|
||
`:source` carries the taxonomy `docs/design.md` already has — `:plate`,
|
||
`:feature`, `:interior`, `:primitive`, `:scalar` — plus `:cel` for painted
|
||
vectors and `:group` for a pure transform node. That is the decomposition that
|
||
makes painting a peer rather than an annex: `paint.js` is not a special case,
|
||
it is a source kind whose channels happen to be authored by hand instead of
|
||
measured. `docs/design.md` already promises "pluggable sources"; this is the
|
||
data shape that keeps the promise.
|
||
|
||
Each node also gets an optional local transform channel — translate, rotate,
|
||
scale, squash. That is the thing the current tool cannot express and that the
|
||
`slot_mouth` / `scale` / `rot` / `squash` fields in the prototype are reaching for.
|
||
|
||
## The flow, in seven stages
|
||
|
||
The user-facing flow is `frames -> analysis -> keying -> geometry -> palette`.
|
||
Analysis is three stages, not one, and the split is the most load-bearing
|
||
decision in this document, because **exactly one boundary is a cache
|
||
boundary.**
|
||
|
||
| # | Stage | In | Out | Cost |
|
||
| --- | --- | --- | --- | --- |
|
||
| 1 | **ingest** | video | footage: an H.264 proxy and raw stream, tracing stills, audio, manifest | minutes, in-app |
|
||
| 2 | **detect** | the raw stream, decoded one frame at a time | raw landmarks per frame | minutes, **cached** |
|
||
| 3 | **measure** | landmarks | anchor fit, residual, head-local rings, signals, interior pixels | seconds |
|
||
| 4 | **condition** | measurements | smoothed transforms and contours | milliseconds |
|
||
| 5 | **key** | conditioned signals + policy | channels: sparse keys, quantised holds, kept frames | milliseconds |
|
||
| 6 | **resolve** | channels + scene + exposure + lead + overrides | geometry per output frame | per frame |
|
||
| 7 | **palette** | geometry | indexed raster | per frame |
|
||
|
||
Stage 2 is the only artifact worth persisting and the only one worth a progress
|
||
bar. Stage 3 reads pixels — teeth extraction is here, and that is why it is
|
||
here rather than in keying: **after stage 3, nothing downstream may look at a
|
||
source pixel.** Stage 4 is separated from 3 only because `contour avg` and
|
||
`anchor avg` are knobs and the rest of stage 3 is not; splitting them means
|
||
dragging that slider does not re-run the interior extraction.
|
||
|
||
That last clause is the guarantee, and it is narrower than the table's ordering
|
||
looks. Stage 3 is not one pass that finishes before stage 4 begins. The anchor fit
|
||
is knob-free; conditioning smooths its four parameters; the head-local rings are
|
||
then measured *through* the conditioned transform — so `anchor avg` does re-run
|
||
the ring mapping, which is a few hundred frames of twenty points and free. What it
|
||
must not re-run is the part that reads a source pixel, and that part takes the
|
||
landmarks and the frames and never the transform, so it does not. Built as
|
||
`measure/anchor` → `condition/anchor` → `measure/mouth` → `condition/contours`,
|
||
and stage 7's eyes, brows and interior follow the same shape.
|
||
|
||
### Where the current code lands
|
||
|
||
| Now | Stage |
|
||
| --- | --- |
|
||
| `stabilize` | 3 measure |
|
||
| `eyeSignals`, `browSignals`, `pairIrises`, `pairBrows` | 3 measure |
|
||
| `extractTeeth` | 3 measure — it reads pixels |
|
||
| `smoothTransforms`, `smoothContours` | 4 condition |
|
||
| `selectKeys`, `suggestPlateFrames` | 5 key |
|
||
| `quantizeSnap`, `resolveBlink` | 5 key — a held value *is* a key |
|
||
| `exposeIndex`, `shiftIndex`, `activeKey`, `heldFrame` | 6 resolve |
|
||
| `toRasterRing`, iris placement at ring slots, `offsetRing` | 6 resolve |
|
||
| `IndexedRaster`, `drawCel` | 7 palette |
|
||
| `drawRegistered`, `posterizeInto` | reference only, off to the side |
|
||
|
||
Three things in `app.js` currently straddle a boundary, and each straddle is a
|
||
bug waiting for a bigger project:
|
||
|
||
- **`resolveTeeth`** extracts from pixels *and* applies contrast/dwell/blob
|
||
knobs. Extraction is stage 3 and cached with the track; resolution is stage 5.
|
||
Today, nudging `teeth dwell` re-runs Otsu over every frame.
|
||
- **`buildEyes`** measures, quantises, *and* places the iris on the subsampled
|
||
lid ring. Those are stages 3, 5 and 6. The placement argument in
|
||
`docs/design.md` — that the iris is read off the already-smoothed ring — is a
|
||
stage-6 fact and stays true; it just needs to be *in* stage 6.
|
||
- **`buildBrows`** likewise: measure the height out of the ring (3), quantise
|
||
it (5), put it back (6). The doc's warning about moving the brow twice is
|
||
precisely a warning about these being one function.
|
||
|
||
### What each knob invalidates
|
||
|
||
This table is the argument for the split, and it should be derivable from the
|
||
sub graph rather than maintained by hand.
|
||
|
||
As built, half of it is, by a different route. `flow/address/block-knobs` is this
|
||
table for tier 2 — per block, the settings its bytes depend on — and
|
||
`address-test` asserts it by biconditional rather than deriving it, which is
|
||
stronger than a derivation would have been: a derived table is only as right as
|
||
the graph it reads. What is not written down anywhere is the rest of the column.
|
||
Stages 3 to 5 are one call chain in `flow/take/measure` rather than a chain of
|
||
subs, so there is no graph above `::base-scene` to read a dependency off, and the
|
||
tier-1 half of a re-freeze is therefore whole-clip.
|
||
|
||
An earlier arrangement had a second table in `domain/params`, per feature area,
|
||
saying which areas a knob affected. It disagreed with the tested one on the first
|
||
entry where the two granularities part company and it had no caller; see that
|
||
namespace for why one checked table beats two.
|
||
|
||
| Knob | Invalidates from |
|
||
| --- | --- |
|
||
| model / footage | 2 detect |
|
||
| `teeth contrast`, `cavity erode`, `blob grow`, `tongue reject`, `prefer upper` | 3 measure (mouth crop only) |
|
||
| `contour avg`, `anchor avg` | 4 condition |
|
||
| `teeth dwell`, `blink cut/hold/dwell`, `gaze step/dwell`, `brow step/dwell`, `suggest tolerance` | 5 key |
|
||
| `vertices`, `eye vertices`, `brow vertices`, `teeth vertices` | 5 key |
|
||
| `exposure`, `mouth lead`, kept-frame edits, overrides | 6 resolve |
|
||
| `lash line`, `iris size`, `pupil`, `brow weight` | 6 resolve |
|
||
| palette edits, colour assignment | 7 palette |
|
||
|
||
## Namespaces
|
||
|
||
```
|
||
src/arthur/
|
||
domain/ pure data, specs, ops. No re-frame, no DOM.
|
||
palette.cljs ramp; index lookup; the no-RGB rule as a spec
|
||
landmarks.cljs index tables (port verbatim)
|
||
ring.cljs ordered traversal: subsample, offset, simplicity
|
||
geom.cljs similarity fit, procrustes, moving average
|
||
node.cljs scene node: source, parent, stencil, z
|
||
timeline.cljs node tree: topo order, transform composition
|
||
channel.cljs keyframe stream: active-key-at, hold semantics
|
||
clip.cljs clip entity; the frame-space conversions
|
||
cel.cljs painted vector layers
|
||
timeline.cljs sequence: clip placement, qf <-> cf
|
||
flow/
|
||
ingest.cljs
|
||
detect.cljs the MediaPipe boundary, and the only one
|
||
measure/
|
||
anchor.cljs stabilise: procrustes, similarity, residual
|
||
mouth.cljs
|
||
eyes.cljs openness, gaze, iris pairing vote
|
||
brows.cljs raise/tilt, ring and end pairing votes
|
||
interior.cljs otsu, morphology, components, radial contour
|
||
condition.cljs
|
||
key.cljs velocity minima, dwell, quantise, blink, decimate
|
||
resolve.cljs scene eval at a frame; exposure, lead, overrides
|
||
raster.cljs indexed scanline fill, stencil, disc, rect
|
||
reference.cljs registered underlay, posterise
|
||
db.cljs app-db schema + spec
|
||
store.cljs handles for things too big for app-db
|
||
clock.cljs audio clock, outside app-db
|
||
events/ one ns per domain area
|
||
subs/ one ns per stage
|
||
ui/
|
||
shell.cljs
|
||
mode/ one ns per tool mode
|
||
panel/ strip, worksheet, readouts, palette, layers
|
||
canvas.cljs the one imperative sink
|
||
fx/ mediapipe, files, audio, persistence
|
||
```
|
||
|
||
Two rules about this tree. `domain/` may not require `flow/`, and neither may
|
||
require `re-frame`. And `flow/` namespaces take explicit arguments — no
|
||
namespace below `subs/` ever calls `subscribe`.
|
||
|
||
## app-db, and what is not in it
|
||
|
||
**app-db holds authored data and ids. Nothing derived, and nothing large.**
|
||
|
||
That sounds like ordinary hygiene and it is not: it is the precondition for two
|
||
features that are otherwise unbuildable.
|
||
|
||
- **Spec validation on every event.** Worth having, affordable only over
|
||
authored data.
|
||
- **Cheap writes.** Every edit `assoc`es into app-db, and every mounted layer-2
|
||
sub compares the result. Both are structural-sharing operations and stay cheap
|
||
only while the map is small.
|
||
|
||
Undo is *not* on that list, and an earlier draft said it was. See Collaboration:
|
||
snapshot-based undo is right single-player and wrong once two people edit.
|
||
|
||
So the dense track and the decoded frames live in `arthur.store`, a `defonce`
|
||
map of id to JS object, and app-db holds `{:analysis/id "sha-..."}`. Landmarks
|
||
stay as they arrive — arrays of `{x, y, z}`, or better, a flat `Float32Array` —
|
||
and are *not* converted to Clojure maps. 478 points × 600 frames is 286,800
|
||
maps, allocated for nothing: no sub ever needs to diff them, and every stage
|
||
that touches them walks all of them anyway.
|
||
|
||
### The playhead, and where per-frame work goes
|
||
|
||
A reaction propagates downstream only when its own **output value** changes, not
|
||
when one of its inputs notifies. So with layer-2 extractors and layer-3
|
||
computations, a playhead tick costs one cheap extractor run per mounted layer-2
|
||
sub, each of which returns the same value for every subtree the tick did not
|
||
touch and therefore notifies nobody. `::measurements`, `::channels` and
|
||
`::resolver` do not re-run. The `:<-` graph is what buys that, and it buys it
|
||
whether or not the playhead is in app-db.
|
||
|
||
So **the playhead lives in app-db**, like everything else. An earlier draft of
|
||
this document put it in a standalone ratom to avoid an invalidation storm that
|
||
does not happen. Two independent reasons it belongs in db:
|
||
|
||
- `[:playback/seek ...]` in the event log is how scrubbing becomes inspectable
|
||
in re-frame-10x.
|
||
- **A collaborator's playhead is a feature.** Putting it outside app-db puts it
|
||
outside the machinery that shares it. See Collaboration.
|
||
|
||
What is true is narrower, and is about sub authoring and interceptors rather
|
||
than about the playhead:
|
||
|
||
- **Expensive work must not live in a layer-2 sub.** A sub that derefs app-db
|
||
directly re-runs on every db change whatever changed. That is the actual
|
||
content of the folklore about re-frame and canvas.
|
||
- **Do not put a per-frame event behind a global interceptor that walks the
|
||
whole db** — a spec-validating `after`, or `std-interceptors/debug` and its
|
||
`clojure.data/diff`. At 30fps that is thirty full-db traversals a second. The
|
||
transport event carries its own interceptor chain and is excluded from the
|
||
global ones.
|
||
- **The render sink is not a Reagent component.** Stage 7 writes bytes into a
|
||
canvas from an rAF loop; it is not a view that re-renders.
|
||
|
||
Stage 6 is still arranged so the per-frame path is a lookup, not a computation:
|
||
|
||
```clojure
|
||
;; recomputes when channels, scene, exposure, lead or overrides change
|
||
(rf/reg-sub ::resolver ...) ;; => (fn [cf] geometry), or an index into a bake
|
||
|
||
;; the rAF loop: reads, blits, dispatches nothing
|
||
(defn tick []
|
||
(let [cf (clock/clip-frame)]
|
||
(raster/paint! (@resolver cf))
|
||
(canvas/blit!)))
|
||
```
|
||
|
||
The sub produces a *resolver*; the loop applies it. A knob change costs one sub
|
||
recomputation; a frame costs a lookup and a blit.
|
||
|
||
## Events and subs
|
||
|
||
Events are named for intent, and carry the clip they apply to:
|
||
|
||
```clojure
|
||
[:clip/set-exposure clip-id 2]
|
||
[:clip/toggle-kept clip-id cf]
|
||
[:keying/re-key clip-id] ; not a setter; policy unchanged, redo the work
|
||
[:keying/suggest clip-id]
|
||
[:paint/commit-shape clip-id node-id cel-frame pts]
|
||
[:node/set-parent clip-id node-id parent-id]
|
||
[:override/set clip-id node-id :pts cf value]
|
||
[:timeline/move-clip seq-id clip-id qf]
|
||
```
|
||
|
||
Setters are fine where the intent *is* the value — `set-exposure` is honest.
|
||
Where they differ, name the intent: `:keying/re-key` takes no value and is not
|
||
`:clip/set-keys`.
|
||
|
||
Subs mirror the stages one-to-one, so the graph and the table above are the
|
||
same object:
|
||
|
||
```
|
||
::footage -> ::landmarks -> ::measurements -> ::conditioned -> ::channels -> ::resolved -> ::raster
|
||
```
|
||
|
||
Each is a layer-3 sub over the previous plus the parameters for its stage only.
|
||
That is what makes "drag `exposure`" recompute `::resolved` and nothing above it.
|
||
|
||
## Tool modes
|
||
|
||
A mode is a namespace, not a branch. Each exposes a map:
|
||
|
||
```clojure
|
||
{:id :paint/pen
|
||
:cursor "crosshair"
|
||
:keymap {"Enter" [:paint/close-shape] "Escape" [:paint/abort]}
|
||
:on-pointer (fn [ev ctx] ...)
|
||
:overlay (fn [g ctx] ...) ; draws handles, never output pixels
|
||
:enter :exit (fn [ctx] ...)}
|
||
```
|
||
|
||
with two rules worth stating because the current `PaintUI` breaks both and gets
|
||
away with it at this size:
|
||
|
||
- **In-progress gesture state is mode-local**, in a ratom the mode owns — not in
|
||
app-db. An unclosed polygon must not appear in the undo history, and a drag
|
||
must produce one undo entry rather than one per pointermove. The mode
|
||
dispatches a single `:paint/commit-shape` on release.
|
||
- **Overlays draw to a separate canvas.** Handles, vertex boxes and the green
|
||
close-target are not indexed pixels and must never be in the raster. The
|
||
current code composites them into the same context as the output, which is
|
||
fine for a preview and lies to you the moment you want to judge the look.
|
||
|
||
The palette-index constraint and grid snapping stay where they are: they belong
|
||
to `domain/cel` and `domain/palette`, enforced at commit, so no tool can bypass
|
||
them.
|
||
|
||
## Where things live
|
||
|
||
Superseded in detail by Serialization and Baking below; the local picture is:
|
||
|
||
| Thing | Where | Keyed by |
|
||
| --- | --- | --- |
|
||
| project (tier 1) | server, plus a local autosave copy | project id, then leaf path |
|
||
| footage frames, audio (tier 3) | on disk / OPFS, untouched | content hash |
|
||
| analysis artifact, geometry bakes (tier 2) | OPFS, with IndexedDB for the index | content hash over every input |
|
||
| render output | never persisted | — |
|
||
|
||
The cache key on the analysis artifact must include the **detector version**. A
|
||
model upgrade that silently reuses old landmarks presents as "the tool got worse"
|
||
with no event to attach it to — which is the argument for content-addressing
|
||
rather than version-numbering everything derived.
|
||
|
||
## Baking
|
||
|
||
"No analysis during playback" is `docs/design.md`'s three-artifact rule with a
|
||
number attached. The expensive stages are 2 (detect), 3 (measure) and, over a
|
||
long take, 4 (condition). Stages 5–7 are arithmetic over small arrays. So there
|
||
are two bakes, not one, and they answer different questions.
|
||
|
||
### Bake A — the analysis artifact. Mandatory.
|
||
|
||
Stages 2–4 for one footage at one set of conditioning parameters. This is what
|
||
must never run while the transport is moving.
|
||
|
||
| Field | Shape |
|
||
| --- | --- |
|
||
| landmarks | `Float32Array[n_frames × 478 × 2]` |
|
||
| transforms | `Float32Array[n_frames × 4]` — s, θ, tx, ty |
|
||
| residual | `Float32Array[n_frames]` |
|
||
| head-local rings | `Int16Array` per ring table, isotropic fixed point |
|
||
| signals | openness, gaze, brow ends, aperture — `Float32Array[n_frames]` each |
|
||
| **mouth crops** | `Uint8Array`, one small crop per frame |
|
||
|
||
The mouth crops are the non-obvious entry, and they are what makes remote work
|
||
possible at all. `extractTeeth` reads source pixels, so without them a
|
||
collaborator holding the analysis but not the source video cannot touch a
|
||
single teeth knob without decoding video again. These are RGBA crops: a 40×30
|
||
crop is 4.8KB raw, and current 200-pixel-wide crops can total over 12MB for a
|
||
take. The blob store compresses `source/crops` losslessly with zlib; block reads
|
||
return the original pixels. Run `python manage.py compress_crop_blocks` once to
|
||
convert existing raw crop blobs and remove their unreferenced copies.
|
||
|
||
### Bake B — resolved geometry. For scale.
|
||
|
||
Stage 6 output per node. Not needed for one clip — resolving a frame is a key
|
||
lookup and a transform — and needed the moment six roto tracks are live at 30fps.
|
||
|
||
**Fixed topology is what makes this flat.** Because every key of a part carries
|
||
the same vertex count with the same vertex meanings, a node's whole bake is a
|
||
rectangular array with no per-frame header and no indirection:
|
||
|
||
```
|
||
pts Int16Array[n_frames × n_verts × 2] raster space, already grid-snapped
|
||
state Uint8Array[n_frames] hidden flag + palette index
|
||
```
|
||
|
||
Frame `f` of a node is the subarray at `f * n_verts * 2`. 600 frames × 20 verts
|
||
is 48KB; a twelve-node clip about 600KB; six roto tracks about 3.5MB. So this is
|
||
the payoff on the aesthetic constraint rather than a cost of it — a variable
|
||
vertex count would force a per-frame offset table and a scan.
|
||
|
||
### Rasters are not baked
|
||
|
||
Stage 7 is the cheapest stage and the one most worth keeping live: palette and
|
||
colour assignment are the last decisions and the ones you want to change in
|
||
motion. Baking to pixels would freeze exactly the loop that should be instant.
|
||
Small raster *thumbnails* for the strip and the timeline are a separate artifact
|
||
with a separate cache.
|
||
|
||
### Unbaking is not an inverse
|
||
|
||
The bake never replaces its input. Authored state — channels, params, overrides —
|
||
is always retained and always authoritative, so "unbake this roto and edit it" is
|
||
a boolean about which side of the cache the renderer reads, not a computation.
|
||
Instant by construction. Three rules keep it that way:
|
||
|
||
- **No destructive bake.** Nothing is discarded when something is baked.
|
||
- **Hand corrections never go into the bake.** They are `:overrides`, applied at
|
||
stage 6 *after* the bake is read, so a correction survives a rebake. This is
|
||
`docs/design.md`'s "hand corrections must not live in the take", and it is the
|
||
rule that stops baking from becoming a trap.
|
||
- **Bakes are content-addressed** by a hash over every input that produced them:
|
||
footage hash, detector version, and the parameters of each stage at or above
|
||
the bake line. A stale bake is then unreachable rather than wrong, and a
|
||
collaborator's bake is fetchable by the same key — see Collaboration.
|
||
|
||
### Partial bakes
|
||
|
||
Bake per node, per frame range. Scrubbing into an unbaked range resolves on
|
||
demand and fills the cache; a worker bakes ahead of the playhead. Show it: a
|
||
bake-state bar under the timeline is an affordance every NLE has already taught
|
||
people to read, and it is honest about what is ready.
|
||
|
||
## Serialization, in three tiers
|
||
|
||
Cut by mutability and size. The cut is what makes collaboration and baking both
|
||
tractable, because it decides what is allowed on the wire.
|
||
|
||
| Tier | What | Size | Synced | Undoable |
|
||
| --- | --- | --- | --- | --- |
|
||
| 1 **authored** | palette, clips, scene graphs, channels, cels, overrides, sequences | KB | yes — it *is* the document | yes |
|
||
| 2 **derived** | analysis artifacts, geometry bakes, thumbnails | MB | no — content-addressed blobs, fetched | no |
|
||
| 3 **source** | frames, audio | MB–GB | no — immutable, by hash | no |
|
||
|
||
Tier 3 is produced **by the app**, not by `extract.sh`: wasm ffmpeg decodes the
|
||
clip to frames and audio in the browser, and they are uploaded and served as
|
||
content-addressed blobs. That makes tier 3 the same kind of thing as tier 2 — a
|
||
cache with a hash — and it removes the one step that currently needs a terminal.
|
||
Two consequences worth planning for: the extraction rate is recorded by the code
|
||
that chose it rather than by a `manifest.json` a human might edit, and **offline
|
||
work needs the frames already fetched**, so the local blob cache is what makes a
|
||
plane usable. Decoding server-side instead is the same data model with a
|
||
different worker; the format does not care which side runs it.
|
||
|
||
**Only tier 1 is the document.** A bake inside the shared document is a system
|
||
that puts 48KB on the wire per vertex drag, and it is unnecessary, because tier 2
|
||
is a pure function of tiers 1 and 3.
|
||
|
||
Tier 1 stays small enough to read. The only vertex data in it is what a human
|
||
placed by hand: cel polygons and overrides. Everything traced is tier 2.
|
||
|
||
### Keys are a map, not a vector
|
||
|
||
```clojure
|
||
;; not this
|
||
{:keys [{:f 0 :pts [...]} {:f 4 :pts [...]}]}
|
||
;; this
|
||
{:keys {0 {:pts [...]}, 4 {:pts [...]}}}
|
||
```
|
||
|
||
`activeKey` becomes a lookup in a sorted map rather than a scan; a single key
|
||
becomes addressable as a path; and two people keying different frames of one part
|
||
merge field-wise with no merge algorithm at all. `selectKeys` returns an array
|
||
today — change it before anything depends on the order.
|
||
|
||
### Serving tiers 2 and 3
|
||
|
||
Built at step 9. Above this point the tiers are a rule about what is allowed on
|
||
the wire; this is the shape that enforces it.
|
||
|
||
**One store for both, named by the sha256 of the bytes.** Once tier 3 is decoded
|
||
by the app rather than by a shell script it becomes the same kind of thing as tier
|
||
2 — a cache with a hash — so there is one place that writes bytes, one that reads
|
||
them, and one URL shape:
|
||
|
||
```
|
||
GET /blob/<sha256> raw bytes, Cache-Control: immutable
|
||
```
|
||
|
||
`immutable` is not optimism there, it is the definition: the name IS the hash of
|
||
the content, so a cached copy cannot be stale. That is what makes serving six
|
||
hundred frames out of it cheap enough to do on every load.
|
||
|
||
**Two kinds of hash, and they are not the same hash.** A blob is named by the hash
|
||
of its BYTES, which is what makes an identical frame in two extractions one file.
|
||
A derived thing — an analysis artifact, a dense block — is named by a hash over its
|
||
INPUTS, which is what lets a client ask for the block the current settings want
|
||
*before* anything has computed it, and what makes a stale bake unreachable rather
|
||
than wrong. So a `Block` row has both: `key` over the inputs, and a foreign key to
|
||
the blob whose digest is over the bytes. Conflating them would break the half of
|
||
addressing that answers questions about work not yet done.
|
||
|
||
```
|
||
POST /api/analyses {key, descriptor} idempotent
|
||
GET /api/analyses/<key> metadata + source block keys
|
||
PUT /api/analyses/<key> link dense landmarks, mask, crops
|
||
POST /api/blocks/missing {keys} -> {missing}
|
||
POST /api/blocks multipart: key, descriptor, data file, optional state file (JSON also accepted)
|
||
GET /api/blocks/<key>
|
||
GET /api/footage/<id> the manifest: video and stream URLs, audio, a URL per tracing still
|
||
GET /blob/<digest> immutable bytes, with byte ranges for video playback
|
||
POST /api/sources multipart video upload
|
||
POST /api/extractions idempotent decode job
|
||
GET /api/extractions/<key> job state and footage id
|
||
```
|
||
|
||
**The server verifies, rather than trusting a name it was handed.** It recomputes
|
||
`sha256(descriptor)` for every key and refuses a mismatch; it refuses an analysis
|
||
whose descriptor does not declare a detector and a version; it refuses a block
|
||
whose analysis it does not know; and it refuses a document naming blocks it does
|
||
not hold. The chain from a stored block to the model version that produced it
|
||
therefore cannot be broken by a client that skipped a step — which is what the
|
||
cache-key rule above actually requires, as opposed to recommends.
|
||
|
||
It hashes the descriptor TEXT rather than re-rendering it from parsed values, and
|
||
that is not a shortcut. JS prints an integral double as `1` and Python prints
|
||
`1.0`, so a scheme where both ends re-render the numbers disagrees on the first
|
||
parameter whose value happens to be whole — and the failure is an upload that 409s
|
||
with nothing wrong. The bytes are the contract; the schema on top of them is a
|
||
convention, and the two fields the server reads out of that schema are checked
|
||
separately.
|
||
|
||
**A manifest names frames; it does not locate them.** Until step 9 the client
|
||
fetched `manifest.json` and built `frames/0001.png` itself, which made the frame
|
||
layout a shared secret between a shell script and a ClojureScript namespace. The
|
||
manifest now carries every URL, so the bytes can live in the blob store — or be
|
||
produced by the app's video upload and server-side ffmpeg extraction. The upload
|
||
path needs no `manifest.json` file: the server builds the footage response from
|
||
its own records. The producer changes; the shape does not.
|
||
|
||
**Tier 3 keeps a video, not a frame per file.** Stage 1 used to decode a PNG per
|
||
source frame: 112MB for 7.6 seconds at 1440x1920, and 1.1GB at the 900-frame
|
||
limit, for pixels whose only consumer was a canvas MediaPipe read once. It now
|
||
writes a browser-safe H.264 proxy — 6MB for the same take — and copies its coded
|
||
frames into an Annex-B stream. WebCodecs decodes that stream in order, and the
|
||
page gives each frame to MediaPipe in video running mode. The JPEG stills beside it
|
||
are reference images for tracing; nothing measures them, so they are deliberately
|
||
outside the footage digest and re-rendering them at another size does not
|
||
invalidate an analysis.
|
||
|
||
The proxy is encoded without B-frames, so decode order matches presentation
|
||
order. The client checks that the stream has exactly the manifest's frame count
|
||
before detection. `/blob/<digest>` also answers byte ranges for ordinary video
|
||
playback; Django's `FileResponse` does no Range handling, so this is code we own.
|
||
|
||
**The document stores what a block IS, not what it holds.** A block's element type
|
||
is in its own descriptor, which is the only place it is written down: an
|
||
`Int16Array` and a `Float32Array` over the same bytes are both valid readings, and
|
||
only one of them is the block. That makes the descriptor load-bearing rather than
|
||
documentation, which is the right way round for the thing a key is the hash of.
|
||
|
||
### Leaf paths, as built
|
||
|
||
The list under **Make the merge unit small instead of clever** is the design; this
|
||
is what step 9 implemented, for the subset that exists:
|
||
|
||
```
|
||
clip/<cid>/name clip/<cid>/subject/<sid>
|
||
clip/<cid>/timing clip/<cid>/feature/<fid>
|
||
clip/<cid>/stage clip/<cid>/group/<gid>
|
||
clip/<cid>/source clip/<cid>/symbol/<sid>
|
||
clip/<cid>/symbol/<sid>/node/<nid>
|
||
clip/<cid>/symbol/<sid>/measured/<nid>
|
||
clip/<cid>/symbol/<sid>/channel/<nid>/<prop>
|
||
```
|
||
|
||
Settings live on subject, feature and group leaves. Each feature has one area, so
|
||
these leaves give settings their own address without separating them from the
|
||
identity they describe.
|
||
|
||
`measured/<nid>` holds several channels together. `:head`'s measured channels are
|
||
written and replaced together by a freeze; `head-mode` reads them to write authored
|
||
`:channels`.
|
||
|
||
A leaf path is "/"-delimited and an id is one segment of it, so a namespaced id —
|
||
`:eye-r/iris`, as drawn under **The node, decomposed** — is written `eye-r~iris`,
|
||
and `~` is then refused inside a name. That is the whole of the escaping.
|
||
|
||
## Collaboration
|
||
|
||
`../tl` already has the model, and it is the right one to copy:
|
||
|
||
- **An authoritative server, and one broadcast source.** Edits go over HTTP; the
|
||
server persists and then broadcasts the delta to the room. The socket is
|
||
read-only for document state. No OT, no CRDT.
|
||
- **Presence gossiped between peers** on the same socket, with the server doing
|
||
exactly two things: handing out a connection id, and stamping the sender's
|
||
server-side identity onto every message so nobody can post as somebody else.
|
||
- **Parties as a pure function of the roster**, so every peer computes the same
|
||
answer and the server knows nothing about them.
|
||
- **Revisions**: one snapshot of the *authored layer* per save, with a user and a
|
||
summary.
|
||
|
||
That maps onto the tiers exactly — the server stores tier 1 and snapshots tier 1,
|
||
with tiers 2 and 3 as content-addressed blobs beside it.
|
||
|
||
### As built
|
||
|
||
- **Addresses.** `/` is the index of the projects you own or edit. A project
|
||
is only ever at `/p/<uuid>/<slug>`: the id finds it, the slug is its name and
|
||
follows a rename without a history entry. "new" makes the project on the
|
||
server first and opens it; a built-in example opened in the editor is saved
|
||
at once as a project of its own. There is no bare project. The address, the
|
||
title and the socket follow `[:project :id]` through one global interceptor
|
||
(`events/collab`).
|
||
- **Ownership.** Every project has an owner (`Project.owner`, not nullable) and
|
||
`editors`. Anyone with the link reads; the owner and editors write. Making a
|
||
project needs you signed in. A reader can make a copy of their own.
|
||
`/api/{me,login,signup,logout}`, `/api/projects/<id>/editors[/<username>]`.
|
||
- **Every edit saves.** No save button. The same interceptor sees
|
||
`:paint/revision` move and saves; one request is in flight at a time, and an
|
||
edit made meanwhile goes when it lands — so a drag reaches the room as fast
|
||
as the round trip allows. A save sends only the leaves that differ from what
|
||
was last synced, and a clean document sends nothing. Analyses and blocks
|
||
already put on the server are not asked about again.
|
||
- **Saves are patches.** With `base` (the seq last caught up to) a save names
|
||
only its changed and removed leaves. A named leaf somebody else changed after
|
||
`base`, to something else, fails the whole save with 409.
|
||
- **The first write wins.** A remote write to a leaf with a local change not
|
||
yet sent waits in the entry's `:behind`; the next save, or a 409, puts theirs
|
||
on screen over ours and says so. Ours stays in the undo list.
|
||
- **The socket** (`/ws/projects/<uuid>`, channels + daphne) carries presence,
|
||
the delta each committed write broadcasts, and `access` when the editor list
|
||
changes. A gap in `seq`, a welcome, or a 409 refetches the document.
|
||
- **Undo** is per person, recorded in `events/edit` as leaf befores and afters
|
||
(`domain/history`), and applied as an ordinary edit. A step undoes only if
|
||
every leaf it touched still holds what it left there: somebody else's edit
|
||
since refuses it rather than being undone with it. Edits to the same leaves
|
||
within a second, each starting where the last left off, are one step.
|
||
- **Snapshots** are named revisions (`/api/projects/<id>/revisions`), with each
|
||
clip's block keys. Restoring one is an ordinary write, broadcast like any.
|
||
|
||
Not yet: follow mode, frame/selection in presence, the advisory `:editing`
|
||
lease, the durable outbox.
|
||
|
||
### Why this model, and not a CRDT
|
||
|
||
The usual reason to reach for Yjs or Automerge is automatic convergence without
|
||
conflict dialogs. **In this domain the primary contested data type is a vector
|
||
path, and automatic convergence of a vector path is undesirable** — two people
|
||
dragging vertices on one polygon merge into a shape neither drew. So the CRDT's
|
||
headline benefit is neutralised on exactly the data that needs help, and the
|
||
parts where it would work fine (scalars, maps keyed by frame) are the parts that
|
||
LWW already handles.
|
||
|
||
What a CRDT would additionally cost here is not the library, it is the second
|
||
source of truth: the canonical document moves into an opaque blob, and every
|
||
server-side thing that reads the document — revisions, collaborator lists,
|
||
thumbnail generation, the admin, any query — needs it materialised back out with
|
||
`y-py` or a Node sidecar. That is real operational weight bought for a feature
|
||
this domain does not want.
|
||
|
||
So: authoritative server, HTTP writes, LWW on addressed leaves, presence on the
|
||
side. And **writes stay on HTTP rather than moving to the socket**, which is
|
||
tl's choice and the right one: auth, idempotency, status codes, retries and
|
||
conditional requests all come for free, and a dropped socket cannot lose a
|
||
write. Latency is not the reason to move them, because latency is handled on the
|
||
client (below) and never by waiting for a round trip.
|
||
|
||
### The model is standard; only tl's plumbing is hand-rolled
|
||
|
||
Worth separating. *Optimistic concurrency control over addressed resources with
|
||
an entity tag* is RFC 7232 — `ETag`, `If-Match`, `412`/`409` — and it is what
|
||
databases and HTTP APIs have done for thirty years. It is not a bespoke
|
||
invention, and every HTTP client implements its half. The hand-rolled part of tl
|
||
is the plumbing around it, so the move is to keep the model and replace the
|
||
plumbing with the boring standard wherever one exists.
|
||
|
||
### Four things to add
|
||
|
||
tl's implementation is missing two pieces that hand-rolled sync usually omits,
|
||
and arthur needs two more that tl does not because its write rate is low.
|
||
|
||
1. **Conditional writes.** `PUT /project/:id/leaf/<path>` with `If-Match: <etag>`,
|
||
answering `409` on a stale write. tl's PUT replaces, so the loser's work
|
||
disappears silently. For a knob that is survivable; for a painted cel it is
|
||
the class of bug that ends trust in a tool. The 409 body carries the current
|
||
value so the client can offer keep-mine / take-theirs.
|
||
|
||
2. **A monotonic project version, and gap detection.** The broadcast is
|
||
fire-and-forget, so a peer that misses one delta — queue pressure, a
|
||
reconnect gap — is silently stale forever. Every delta carries `seq`; a client
|
||
that sees `seq > local + 1` refetches. This is what makes staleness
|
||
self-healing instead of permanent, and it is about twenty lines.
|
||
|
||
3. **Gesture coalescing, client side.** This is the one real load difference
|
||
between tl and arthur: tl's user annotates a clip every few seconds, arthur's
|
||
user drags a vertex. A drag is **one** write on release, not one per
|
||
pointermove. With the gesture held in mode-local state and committed once (see
|
||
Tool modes), arthur's write rate is comparable to tl's and the model holds
|
||
unchanged. Without it, no sync model survives.
|
||
|
||
4. **An outbox, so nothing ever waits on the network.** "No lag" is satisfied
|
||
entirely on the client: apply locally, enqueue, reconcile on ack.
|
||
|
||
```clojure
|
||
{:pending {"clip/7/cel/bg/12" {:value … :etag … :tries 0}}}
|
||
```
|
||
|
||
With one rule that is easy to get wrong: **while a leaf has a pending local
|
||
write, incoming broadcasts for that leaf are ignored.** Otherwise a remote
|
||
delta lands mid-gesture and the shape snaps back and forth between the two
|
||
values until the ack arrives.
|
||
|
||
### Two operational notes
|
||
|
||
- tl's `ROOMS` is process-local, as its own comment says. That is a single-worker
|
||
ceiling, not a design flaw, and it has to move to Redis before a second worker.
|
||
- **Revisions need a coarser trigger than every save.** tl snapshots a small
|
||
annotation layer; arthur's tier 1 contains cel polygons, so a snapshot per save
|
||
will bloat the table. Snapshot on an explicit "mark version", or time-boxed.
|
||
|
||
### When this model is wrong, and what replaces it
|
||
|
||
State the tripwire rather than trusting the judgement: **if 409s on cel leaves
|
||
become common in real use, the merge unit is too coarse.** The answer then is a
|
||
finer unit, not a CRDT, and there is a planned progression:
|
||
|
||
| Step | Merge unit | Gets you |
|
||
| --- | --- | --- |
|
||
| now | the cel (`cel/:node/:frame`) | two people on different frames |
|
||
| next | the layer | two people on different layers of one cel |
|
||
| last | **the stroke** — an append-only list of immutable stroke records | two people painting one layer at once |
|
||
|
||
The third step is worth seeing now because it is the escape hatch that keeps
|
||
this decision from being a dead end. Once a drawing is a list of immutable
|
||
records rather than a mutable point array, concurrent painting merges by
|
||
construction — two people appending different records cannot conflict — and that
|
||
is a domain-specific version of what a CRDT would have given, without the second
|
||
source of truth. It is also a plausible thing to want for undo granularity
|
||
anyway.
|
||
|
||
### Make the merge unit small instead of clever
|
||
|
||
Last-writer-wins clobbers only when its unit is too big. tl can save a scene;
|
||
arthur cannot save a project, because one vertex drag would then clobber a
|
||
collaborator's keying. The fix is addressing, not an algorithm:
|
||
|
||
```
|
||
palette
|
||
sequence/:sid
|
||
clip/:cid/timing clip rate
|
||
clip/:cid/subject/:sid tracked subject and settings
|
||
clip/:cid/feature/:fid tracked feature and settings
|
||
clip/:cid/group/:gid shared settings for an eye pair
|
||
clip/:cid/symbol/:sid frame count, palette
|
||
clip/:cid/symbol/:sid/node/:nid one node: parent, stencil, z, time
|
||
clip/:cid/symbol/:sid/channel/:nid/:prop
|
||
clip/:cid/symbol/:sid/measured/:nid
|
||
clip/:cid/symbol/:sid/cel/:nid/:frame
|
||
clip/:cid/symbol/:sid/overrides/:nid/:prop
|
||
```
|
||
|
||
Each feature and node has its own leaf, so tuning separate features and adding
|
||
separate nodes use separate addresses. Fractional `:z` keeps draw order on the
|
||
node leaf.
|
||
|
||
Each path is a **leaf**: independently addressed, independently versioned, LWW
|
||
with an `If-Match` on its version. The boundaries are chosen so the things people
|
||
actually do simultaneously land on different leaves — two people painting
|
||
different cels, or keying different parts, never meet. Within a channel leaf the
|
||
server merges **field-wise by frame**, which is the return on keys-as-a-map and
|
||
about fifteen lines of Python.
|
||
|
||
### Where it stays honest about conflict
|
||
|
||
| Data | Concurrency | Resolution |
|
||
| --- | --- | --- |
|
||
| palette, clip params, knobs | scalar | LWW on the leaf. Two people tuning one clip will fight — that is a real thing to surface, not to merge. |
|
||
| channels | map by frame | field-wise union, LWW per frame |
|
||
| kept-frame set | add / remove | store `{frame → bool}`, not a set, so removals merge instead of vanishing |
|
||
| cel layer order, `:z` | list insert | **fractional indices, not integers** |
|
||
| cel polygon points | sequence | LWW on the whole layer, plus an advisory lease |
|
||
| parentage | tree | LWW per node, with a cycle check that rejects the losing move |
|
||
| playhead, selection, tool | ephemeral | presence only — never in the document, never in history |
|
||
|
||
Two of those are cheap now and expensive later.
|
||
|
||
**Fractional indices for anything ordered.** Integer `:z` and integer layer
|
||
positions do not survive concurrent insertion: you get duplicates and gaps. A
|
||
fractional key between neighbours makes concurrent inserts commute. Adopt it in
|
||
`domain/node` and `domain/cel` from the first commit.
|
||
|
||
**Do not merge one polygon between two people.** Two simultaneous edits to the
|
||
same path have no meaningful merge, and the result is a shape nobody drew. The
|
||
layer is the unit, LWW, and presence shows who is on it before the collision.
|
||
A coarse merge unit that is *visible* beats a fine one that invents geometry.
|
||
|
||
### Presence carries more than a cursor
|
||
|
||
tl's roster entry, extended with what arthur's views need:
|
||
|
||
```clojure
|
||
{:cid "…" :user "…" :party nil :joinable true
|
||
:clip cid :frame cf :playing? false :rate 1.0
|
||
:node :mouth :tool :paint/pen
|
||
:editing "clip/7/cel/bg/12"} ; advisory lease on a leaf
|
||
```
|
||
|
||
`:editing` is what makes leaf-level LWW liveable — the collision becomes visible
|
||
before it happens. Advisory only: no server enforcement, so there is no lock to
|
||
leak when a tab closes.
|
||
|
||
### Follow mode, and why frames never go on the wire
|
||
|
||
tl's parties become arthur's follow mode nearly unchanged: mirror `:clip`,
|
||
`:frame`, `:playing?` and the plate mode.
|
||
|
||
**Broadcast transport state, never frames.** On play, send
|
||
`{:clip :frame :playing? :rate}` once; each peer's own audio element then becomes
|
||
its own clock and the picture follows it exactly as it does locally. Per-frame
|
||
messages would put thirty packets a second on the wire to reproduce something
|
||
every peer can compute. A peer whose bake is cold shows a cold indicator and
|
||
catches up rather than stalling the party.
|
||
|
||
### Going offline
|
||
|
||
Arthur is offline-capable almost by accident, and it is worth noticing why: tier
|
||
1 is small enough to hold entirely in app-db, tier 2 is a local content-addressed
|
||
cache, and tier 3 is a directory of PNGs that `extract.sh` put on your own disk.
|
||
So when the network drops, **everything needed to scrub, key, paint, resolve and
|
||
render is already local.** Nothing about playback or editing touches the server.
|
||
|
||
What stops is only the three things that are inherently remote: your writes stop
|
||
acking, others' deltas stop arriving, and presence goes dark.
|
||
|
||
Two preconditions, or the above is a lie:
|
||
|
||
- **The outbox must be durable.** In memory, a closed tab loses the session. It
|
||
goes in IndexedDB, written in the same task as the optimistic local apply —
|
||
a few hundred microseconds, invisible, and the difference between "offline is
|
||
fine" and "offline is a trap".
|
||
- **Vendor MediaPipe's wasm.** `README.md` notes that `face_landmarker.task` is
|
||
local but the wasm bundle is fetched from jsdelivr on first use. That makes
|
||
stage 2 the one thing that silently requires a network, and it will be
|
||
discovered on a plane. Vendor it or precache it in a service worker.
|
||
|
||
### Reconnecting
|
||
|
||
Refetch the document first, then drain the outbox — in that order, so conflicts
|
||
are detected against current state rather than against a stale etag. Since the
|
||
document is kilobytes, a full refetch is cheaper than reasoning about a long
|
||
`seq` gap.
|
||
|
||
Then each pending write lands in one of four cases, and the design goal is that
|
||
only the last one reaches a human:
|
||
|
||
| Case | Resolution |
|
||
| --- | --- |
|
||
| nobody touched your leaves | etags still match, outbox drains, converged. The common case. |
|
||
| a **channel** leaf collided | **auto-merges** field-wise by frame — union the frames, LWW per frame. No human. |
|
||
| a **scalar** leaf collided (knob, timing) | a compact list with one "keep all mine / take all theirs", because a knob is one number and the value is visible in the output anyway |
|
||
| a **cel** leaf collided | see below — never a dialog |
|
||
|
||
The channel row is the third time keys-as-a-map pays for itself, and it is the
|
||
reason a long offline session usually reconciles with no interaction at all.
|
||
|
||
### A colliding cel forks into a layer
|
||
|
||
Do not ask an artist to choose between two drawings in a modal. **Keep both:**
|
||
their cel stays where it is, and yours is added on top as a layer named for the
|
||
session that made it. Nothing is lost, nothing is silently clobbered, and the
|
||
conflict is resolved by looking at it and deleting a layer — which is an
|
||
operation the paint tool already has, and which is what an artist would do
|
||
anyway.
|
||
|
||
This is the general principle for this whole class of conflict: **when the merge
|
||
unit is coarse, resolve by stacking rather than by choosing.** A stack is
|
||
reversible and legible; a choice made in a dialog against two thumbnails is
|
||
neither.
|
||
|
||
### The awkward cases, named
|
||
|
||
- **A write to a leaf whose parent was deleted** — a node or clip removed while
|
||
you were away. The server answers `404`; the client parks it in an orphaned-edits
|
||
tray rather than dropping it. Rare, and the alternative is silent loss.
|
||
- **A very long offline session against a busy project.** Merging may be the
|
||
wrong frame entirely. Offer the escape hatch revisions already make possible:
|
||
save the offline session as a revision, take theirs, and reconcile by hand from
|
||
two named versions. Better than a hundred-row conflict list.
|
||
- **Two peers both run detection offline.** Both compute the same analysis
|
||
artifact and both try to upload it. Content addressing makes that idempotent —
|
||
same hash, same bytes — so let it race rather than coordinating a claim.
|
||
|
||
### Undo is per-user — and this revises an earlier claim
|
||
|
||
An earlier section of this document said undo was free if app-db held only
|
||
authored data, via `day8.re-frame/undo`. That is true single-player and wrong the
|
||
moment two people edit: a snapshot of app-db reverts their work along with yours.
|
||
|
||
So undo is a **per-user stack of inverse leaf writes**, applied as ordinary
|
||
edits. Your undo writes a leaf back to the value you last saw, and it conflicts
|
||
with a concurrent editor in the same visible way any other write does. Keeping
|
||
app-db small is still right — for validation cost and allocation — but it is no
|
||
longer the undo story.
|
||
|
||
## The 30fps budget
|
||
|
||
At 320×200 stage 7 is not the problem. An even-odd scanline fill over 64,000
|
||
pixels with a dozen mostly-small parts is well under 200,000 byte writes a frame:
|
||
six million a second at 30fps. Six roto tracks off baked geometry adds a handful
|
||
of subarray reads.
|
||
|
||
Two things in the current code *will* miss 30fps, and neither is the rasteriser:
|
||
|
||
- **`posterizeInto` does a `getImageData` every frame.** A GPU→CPU readback
|
||
stalls the pipeline, and it is followed by 64,000 linear scans over the palette
|
||
to find the nearest colour — around a million distance computations a frame.
|
||
Both are avoidable: posterise once per source frame into a cached `Uint8Array`
|
||
(it is a pure function of the frame, the transform and the palette), and
|
||
replace the nearest-colour scan with a 5-5-5 RGB lookup table — 32KB, built
|
||
once per palette.
|
||
- **`drawRegistered` draws a full-resolution photo every frame.** Correct as a
|
||
drawing reference on a held frame; not something to do thirty times a second.
|
||
Pre-scale each source frame to raster size once, at load.
|
||
|
||
The rule both are instances of: **nothing that reads source pixels may run on the
|
||
transport path.** That is the same line the bake boundary draws, which is why
|
||
drawing it once, in stage 3, pays twice.
|
||
|
||
## Porting order
|
||
|
||
Tests first, and not as discipline — as an oracle.
|
||
|
||
1. **Port `synth.js` and `selftest.js` first**, to `cljs.test`. The synthetic
|
||
track is the only fixture with ground truth, and the 105 assertions encode
|
||
invariants that are invisible to inspection: the ring-simplicity check, the
|
||
bowtie, the iris-pairing vote fed a deliberately swapped track.
|
||
2. **Run both implementations on the same synthetic track and diff numerically**
|
||
while porting each pure module. `fitSimilarity` and `procrustesMean` should
|
||
agree to 1e-9; a disagreement is a port bug, not float noise.
|
||
3. Port `landmarks`, `geom`, `ring`, `raster`, `take` — mechanical, and they are
|
||
where the invariant comments live. **Carry the comments across verbatim.**
|
||
They are the most valuable text in the repo and every one of them is a bug
|
||
that already happened.
|
||
4. Port `flow/measure/*` by lifting out of `pipeline.js`, then split
|
||
`condition` and `key` out of it.
|
||
5. Build `db`, `store`, `clock`, and stage 6 + 7 against a single hardcoded
|
||
clip. At this point the synthetic take should render, with no UI beyond a
|
||
canvas and a play button.
|
||
6. Only then the UI, the modes, the timeline.
|
||
7. Add the ring-simplicity and no-RGB checks as specs, so they fire on authored
|
||
data too and not only in tests.
|
||
|
||
Three things are cheap in step 3 and expensive after step 6, so they go in early
|
||
even though nothing needs them yet: **keys as a map** keyed by frame, **fractional
|
||
indices** on `:z` and cel layer order, and **leaf addressing** for tier 1 — the
|
||
paths under Collaboration, used as the shape of app-db even while single-player.
|
||
None of them cost anything on day one and all three are retrofits that touch
|
||
every namespace.
|
||
|
||
The wiring cross-check in `selftest.js` — every `el('id')` must exist in
|
||
`index.html` — becomes unnecessary and should be deleted rather than ported. It
|
||
is a test for a failure mode that Reagent does not have.
|
||
|
||
## Decisions I would defer
|
||
|
||
- **Whether `raster` stays JS.** 320×200 is 64,000 pixels and CLJS over a
|
||
`Uint8Array` with no seq allocation handles it comfortably; port it and
|
||
measure. Do not port it into idiomatic `map`/`reduce`.
|
||
- **Interpolation.** `:hold` is the only interp the aesthetic wants, and the
|
||
channel model should still carry `:interp` per node, because a parented
|
||
transform on a painted cel is the one place a tween might be right. Do not
|
||
implement it until something needs it.
|
||
- **Multi-character scenes.** `character` is in the entity model as one field
|
||
and should stay a stub until there are two.
|
||
- **Audio per clip vs per sequence.** One master track is right until it is not.
|
||
- **Whether bake B exists at all in v1.** One clip does not need it; the flat
|
||
typed-array layout should be designed now and built when the second roto track
|
||
appears.
|
||
- **Sharing tier 2.** Content addressing means a collaborator *can* fetch a
|
||
bake instead of recomputing, and also means nothing breaks if they do not.
|
||
Ship the cache-miss path first and treat blob sharing as an optimisation —
|
||
the analysis artifact is the only one where it clearly pays, because it is the
|
||
only one that costs minutes.
|