A shape's :span stays in its parent's frames, but an instance's or a sound's
is now in its own: dropping a symbol at frame 97 gives it span 0 … length and
:at 97, so moving it along its parent is one write to :at. :time :in is gone
(it was the span's start written twice); node/placed-span maps an own-time
span out to the parent for playback, the mixer and the timeline rows, and
node/problems reports a stale :in.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
'+ symbol' in the timeline makes an empty symbol and places it at the
playhead: inside the selected instance, beside any other selected node, or in
the open symbol. Timeline selections carry their row path so a symbol placed
twice nests into the row that was clicked. Symbols gain a :name.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Everything that holds nodes is a symbol (domain/timeline -> domain/symbol,
:timelines -> :symbols) and a node that places one is :kind :instance. The
reserved :main root is gone: which symbol is on screen is editor state
([:ui :open]), every domain function that needs a symbol is told which, and
a document opens on the longest symbol nothing else places.
Saved projects move to schema 2 through migration 0007, which rewrites leaf
paths, instance kinds and the feature :symbol key; the client refuses a
schema it does not read.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
"the video never presented frame 1; it offered 2" was this function waiting
for a frame it had not asked for a second time. The walk discarded a wrong
frame and re-armed the callback WITHOUT seeking again, and nothing further is
ever presented to a paused element that has not been asked to move — so one
wrong answer starved until the timeout and reported it as the browser refusing.
Underneath that, the wrong answer was not wrong. A container can carry an edit
list, and `currentTime` then counts from the start of the edited presentation
while a frame's `mediaTime` counts from the start of the media. The two differ
by a constant, so the frame at currentTime 0 can honestly report a mediaTime
two frames in. Seeking cannot correct for it in the positive direction: source
frame 0 would have to be found before the start of the video, every attempt
clamps at zero, and the walk offers frame 2 forever.
So the constant is measured once and subtracted. `calibrate!` takes whatever
the browser calls the first frame it shows and makes that the origin, which is
the honest definition anyway — it is what a viewer sees at time zero, and the
audio clock this take plays against starts in the same place.
Calibration also leaves the element on frame 0, so the walk starts holding it.
`frame!` returns immediately for a frame already on screen rather than seeking
to where it already is, which presents nothing and would hang.
Residual error still re-seeks, corrected by exactly the measured miss, and
still fails loudly with every frame that was offered and where it was asked
from, so a next failure is diagnosable in one shot rather than four.
The proxy also drops B-frames now. That removes the edit list at the source
rather than only coping with it, and makes decode order presentation order
should this ever be fed to WebCodecs. 14% larger, and extraction refuses a
proxy whose timeline is shifted so it cannot come back silently. Honest note:
this was my first diagnosis and it did NOT reproduce the failure — both
proxies walk correctly here on hardware decode — so it is hardening, not the
fix.
Verified by forcing the fault: a harness that offsets the reported timeline by
-2, -1, 0, +1, +2 and +5 frames failed on every positive offset before and
recovers on all six now, first attempt. Real app in headed Chrome with Metal
hardware decode: 280/280. 41 backend and 234 frontend tests green.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Uploading a clip shot straight from the iPhone camera app failed with
"variable-frame-rate video needs timestamp-aware playback". The file was not
variable: its container reports avg_frame_rate 8670/299 and nb_frames 289 over
a stream whose decoded timestamps are 280 frames exactly 1/30s apart. The
guard compared two pieces of container metadata and rejected CFR video on the
strength of a summary the container had got wrong about its own contents.
The guard was also obsolete. It dates from when the page measured the source's
own frames, where a wandering frame duration really does break
`frame = floor(t * fps)`. Nothing measures the source now — ffmpeg resamples it
onto a constant rate and the proxy is re-probed after it is written — so
variable input is a thing this converts rather than a thing it refuses.
So: probe picks a rate instead of validating one. It takes the nominal rate,
which is the rate every timestamp in the stream can be expressed at and so the
one that keeps every distinct source frame, and carries it as an exact fraction
because 30000/1001 is not a float and a rounded -r is how a long take drifts.
The disagreement is still recorded as `vfr`, just not fatal.
The frame-count cross-check went with it. It compared the proxy against the
source's nb_frames, which is the number this whole bug proves can lie, and a
resample to a constant rate legitimately changes the count. It now checks the
proxy's DURATION against the source's, because what must not drift is how long
the picture lasts against how long the audio lasts.
Verified on the reported file: 280 frames at 30fps, picture 9.3333s against
audio 9.3167s — half a frame — and 280/280 detected in the real app. 41 backend
tests green, including a genuinely variable fixture end to end.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Detection now walks a browser-seekable H.264 proxy in MediaPipe's VIDEO
running mode. The PNG sequence it replaces was 112MB for 7.6 seconds at
1440x1920 and 1.1GB at the 900-frame limit; the proxy is 6MB, and landmarks
detected off decoded H.264 rather than off the PNGs moved at most 0.0033 of
frame width.
Three things had to be true for video mode to work, and each was measured
against the same footage decoded to PNGs:
/blob/<digest> answers byte ranges. Django's FileResponse does no Range
handling, and a media element handed 200 with no Accept-Ranges reports an
empty `seekable`, no-ops every currentTime write, and detects frame one
ninety times without raising.
A seek aims at the MIDDLE of its frame. Aiming at i/fps sits on a frame
boundary and landed one frame early 31 times in 91; (i + 0.5)/fps was exact
on all 91.
Timestamps are strictly increasing footage milliseconds. Video mode is a
tracker: a repeat leaves the graph in an error state every later call
re-throws, so the landmarker is discarded on failure, and passing the frame
index instead of i*1000/fps moved landmarks six times further from the
per-frame answer.
Frames are verified rather than trusted. requestVideoFrameCallback states
which frame it handed over, the walker discards any other and fails loudly
if the one it asked for never arrives — a stale presentation from the tail
of a previous seek is what produced "asked for frame 1 and it presented
frame 2" on a video whose seeks were in fact exact.
The proxy is re-encoded even when the upload is already H.264: HEVC is not
decodable everywhere, and footage identity is the proxy's digest. The JPEG
stills beside it are tracing references, outside the footage digest because
re-rendering them at another size is not different footage.
Verified end to end in a real browser against real footage: 228/228 frames
detected, a drawn roto face, 37 backend and 234 frontend tests green.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
There were two answers in the tree to "which stored bytes stop being valid when
this knob moves", and only one of them was checked.
`flow/address/block-knobs` is per block and asserted by biconditional —
`address-test` re-freezes the take once per knob and requires that the bytes
changed if and only if the key did. `domain/params`'s `:affects` was per area,
had no caller but a test asserting it returned what it was written as, and was
already wrong in both directions on the one entry where the two granularities
disagree: `:aperture-cut` claimed `#{:mouth}`, where it reaches no block, and
omitted the teeth, whose contour bytes it genuinely moves by gating
`condition/interior`'s smoothing. `:blink-cut` claimed `#{:eye}` and reaches no
block either, because a blink is `[:vis]` keys in tier 1.
So `:affects` and `affected-areas` are gone, and `address/knob-roles` is the
derived inverse of the table that is asserted — which is what a parameter panel
actually wants to ask. A knob absent from it invalidates no block, and that is
an answer rather than a gap.
Two new assertions keep the derivation from rotting at either edge: every role
in the table is reachable from some knob, and every knob a block declares is one
the registry defines. The second closes a real hole — `block-descriptor` checks
only that a knob was PASSED, and the freeze's `merge take/knobs` makes that true
of anything spelled like a keyword, so a typo in `block-knobs` would have named a
setting no slider can move.
228 CLJS tests, green.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Step 9. The tier split was the work; Django was the easy half.
Tier 1 — the authored scene — is the document, and it is addressed as
independently versioned leaves rather than saved whole, so one vertex drag
cannot clobber a collaborator's keying. `domain/leaf` is the document as
path -> value; `domain/wire` puts it on the wire as transit, because JSON
has neither integer map keys nor keywords and a save would quietly turn
`{0 v}` into `{"0" v}`.
Tier 2 — the dense channel blocks — is content-addressed by a hash over
every input, with the detector version inside every key through the
analysis the block descriptor names. `flow/address`'s `block-knobs` is the
invalidation table, and `address-test` does not trust it: it re-freezes the
take once per knob and asserts the biconditional, that a block's bytes
changed if and only if its key changed. That found `brow-pos` not depending
on `contour-avg` — the brow ring is smoothed, the raise is not.
Tier 3 — frames and audio — is served by the hash of its bytes out of the
same store. A manifest now names frames and carries a URL for each, so the
frame layout stopped being a shared secret between a shell script and a
ClojureScript namespace, and the `?v=` cache-buster went with it: a blob's
name is the hash of its contents, so a stale copy is not a thing that can
happen. The synthetic take's `audio.wav` moved to `static/arthur/` — an
asset the project owns, not an extraction that churns.
The server verifies rather than trusting a name it was handed: it
recomputes every key from the descriptor stored beside it, refuses an
analysis that declares no detector version, and refuses a document naming
blocks it does not hold. It hashes the descriptor TEXT, because JS prints
an integral double as `1` and Python as `1.0`, and a scheme where both ends
re-render the numbers disagrees on the first parameter that happens to be
whole.
Two loose ends from step 8 closed on the way. `pack` no longer takes a
`(track, frame)` predicate whose call sites each re-derived a feature from
an index — every track names the feature it follows, which deleted five
hand-maintained mappings. And `:dev-http` is gone: Django serves the page,
shadow-cljs only builds into the staticfiles tree.
227 CLJS tests, 31 Django tests, green.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The absence mapping was asserted for two of the ten dense tracks. Each block
hands `pack` an :absent predicate that derives a feature from a track INDEX, so
the predicate and the vector literal beside it agree only by hand, in four
places — and three of the four blocks had nothing checking them.
Give each feature a DIFFERENT gap window, so a track wired to the wrong feature
shows up as absence inside somebody else's window. A single shared window passes
under any permutation, which is the failure port-plan mechanical fact #2 warns
about: swap left for right and every part is still roughly where it belongs, so
it survives inspection.
Verified by mutation, since a test that cannot fail is worth nothing. Swapping
the brow block's two tracks, and swapping eye-block tracks 1 and 3 while leaving
0 and 2 correct, both now fail loudly; neither was caught before.
Also pin the teeth. They are their own feature so they can carry their own
:area :teeth parameters, which means an occluded mouth sets no bit on them; they
are dropped regardless because they stencil on :mouth-in and scene/finish drops a
node whose stencil drew nothing. Both frames come from an unannotated reference
clip, because the interior only draws on an open mouth and a frame the mouth was
shut on would have passed for the wrong reason.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B87NVmiU36qQmN9gmFYnJ9
Step 8's data model, ahead of its controls. Nothing here is a UI.
domain/params holds every knob's definition once — default, applicable area,
value constraints and the areas a change would force to regenerate. flow/take's
literal knob map becomes a view of it, so the take's defaults and the future
parameter panel cannot drift apart.
domain/feature adds subjects, features and groups as document data the renderer
never reads. A feature ID is stable for the whole clip, across occlusion: a run
of visible frames is not a new identity. An eye pair is an explicit group of one
or two eyes of the same subject, so a profile view with one identified eye needs
no invented partner. Settings resolve area -> subject -> group -> feature, and
dropping an eye from a pair materialises its effective values first so playback
does not jump. scene/problems now validates all of it.
Presence becomes per-feature rather than per-subject. freeze's :absent predicate
takes a track as well as a frame, so one occluded eye can be absent while its
partner still has a value; a full-face miss still marks everything absent. A
manifest may annotate known gaps as one-based inclusive intervals, which ingest
expands into observation tracks before measurement. An unobserved eye then gets
no vote in the iris pairing and cannot steer the shared gaze — gaze falls back to
whichever eye is visible. Temporal filters still see a sample on every frame,
held from the last observed one, because the numbers are a rectangular buffer;
the state mask, not the buffer, is what says the frame has no value.
js/app.js gets the same occlusion lesson: leading nulls from a face that starts
occluded used to throw away the whole take, and the neutral frame could be chosen
from a held duplicate pose.
Parameter editing, scoped regeneration and a feature-level detector remain. Until
one exists, footage without annotations falls back to the full-face mask rather
than claiming occlusions it cannot see.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B87NVmiU36qQmN9gmFYnJ9
Two bugs in the browser suite, both dating from step 7 adding eyes, brows and
teeth to the stage.
The shut-mouth check asked for exactly one tone on the whole canvas. That was
true when the stage held nothing but a mouth; since step 7 every frame carries
five or six tones whatever the mouth is doing, so the check failed on a correct
picture. Ask instead whether :mouth-dark is on the stage at all: nothing else
carries that tone, so it answers "is the interior drawn" without needing to know
where the mouth is. freeze_test already got this same rescoping; its browser twin
did not.
The failure went unnoticed because close() raced Chrome's own profile writes and
threw ENOTEMPTY out of main's `finally`, past the summary line and the
process.exit that reports the failure count. The suite therefore exited 2 and
printed no verdict whether it passed or failed. Teardown is now allowed to fail
out loud without taking the exit code with it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B87NVmiU36qQmN9gmFYnJ9
`stabilize` is three things wearing one name, and it is now three functions in two
stages: `flow/measure/anchor` fits the rigid transform, `flow/condition` smooths
its parameters, `flow/measure/mouth` measures the lip rings through the result.
Parity is on the COMPOSITION and not on the pieces -- a split that agreed
function by function and not end to end would be a split rather than a port.
The oracle now drives `stabilize` at three configurations and the port agrees to
1e-9 on ref, rigid, transforms, outer, inner and aperture, plus `smoothContours`
at three radii. Two of the three configurations are at aspect 0.5625, a 1080x1920
phone clip, because at aspect 1 `pick` is the identity: a port that dropped the
anisotropy correction outright would pass every other assertion in the suite.
148 tests, up from 134.
Three decisions worth the reading time.
`makeXform` is not ported, and its absence takes the face oval with it. It
centres on the oval's bounding box and zooms until the face is 80% of the raster
height, so every vertex it touched carried a cropping decision made once, at
analysis time, from one frame's landmarks. Geometry belongs in the node's own
local space with the framing as a transform on a node, so this is a deletion. The
oval's only other consumer was the placeholder plate outline, which is painting.
The residual is taken against the RAW fit, and the prototype took it against the
smoothed one. That is the only deliberate numeric divergence here, and parity is
kept by asserting `anchor/residuals` on exactly what the prototype handed it. The
number's job is to say whether a section is stabilisable at all; folding the
smoothing error into it makes a slider look like a property of the footage, and
docs/architecture.md lists the residual under stage 3, which requires it to be
knob-free. `condition/anchor` therefore replaces `:transforms` and leaves
`:residual` alone.
The stage order is not the strict chain the table in docs/architecture.md looks
like, and that document now says so. The fit is knob-free, conditioning smooths
it, and the rings are measured *through* the conditioned transform -- so
`anchor avg` does re-run the ring mapping, which is a few hundred frames of twenty
points. The guarantee was only ever about the part that reads a source pixel, and
that part never sees a transform.
Two things fall out and are asserted rather than assumed. Smoothing and
subsampling commute, because both are per-slot, which is what lets `vertices`
stay a stage-5 knob downstream of a stage-4 one -- and it is also why the port can
smooth the full twenty slots where the prototype smooths eight and still match.
And `condition/contours` is `geom/moving-average` per vertex per axis rather than
its own clamped window, so "radius 2" cannot come to mean two different things at
the two knobs.
One dead end recorded so nobody walks it twice: the synth's head is perfectly
rigid -- its jitter is a whole-head translation, which a similarity absorbs
exactly -- so every frame's rigid configuration is congruent with frame zero's and
the Procrustes mean IS frame zero to 1e-15, jitter or none. "The reference is the
mean and not frame zero" cannot be asserted on this track and is asserted in
geom-test, where the two can differ.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The frame went 5.86ms to 3.17ms -- a 170fps ceiling to 315 -- and `loop`/`recur`
is gone from src/ entirely.
Two real wins, both from measuring rather than guessing:
- `->rgba` was 3.16ms of that frame and was scene-independent: a `nth` into a
vector of vectors is four protocol dispatches per pixel, 64,000 pixels a frame.
The palette is now flattened once and cached by identity of the source vector
-- palettes are values, so identity is exactly the right test and there is no
invalidation to get wrong. At zoom 1 on a little-endian machine the inner loop
is one 32-bit write per pixel through a Uint32Array view of the same buffer:
0.11ms, 28x. Every other case walks bytes off the same flat palette.
raster-test pins both against a naive per-pixel reference at three zooms,
because a fast path that is subtly wrong about colour would look like a palette
bug rather than like an optimisation.
- The per-frame z sort was re-deriving a constant. Draw order is a function of
the z paths, which change when the scene changes and never because the playhead
moved, so `draw-rank` computes it once and a frame sorts small integers. Every
op drops its `:i` and `:z-path` fields as a result.
The loop pass, and an honest note on it: it came out NET POSITIVE on lines, which
is the wrong direction for a cleanup. geom is -3 (transduce for the accumulators,
`(-> (iterate refine ref) (nth iters))` for Procrustes, which is what the
algorithm says rather than a counter that happens to stop), channel -3,
fill-poly!'s copy loop 7 lines to 1. Against that, eval-into went from one
four-deep pyramid with seven positional parameters to `place` / `emit` / a fold
over a ctx map -- less nesting, more lines, and a different change from "fix the
loops" that should not have been bundled with it.
Two idioms were reverted for being worse here than what they replaced, both the
same mistake -- reaching for a form that allocates inside a hot loop:
- `partition 2` over an `array-seq` per scanline is some five thousand throwaway
objects a frame and took draw from 0.88ms to 1.48ms. Now a pairwise `dotimes`
over the array.
- `z-lex` via `(map compare a b)` allocated three lazy seqs per call, ~700 calls
a frame. Made moot by `draw-rank`.
And one DRY move reverted for coupling things that only coincide: a `geom-path`
table had `node/valid-paths` and `scene/emit` deriving from one source, which
ties what a kind may CARRY to what the renderer READS off it. Those are the same
today and are not the same question, and the table put a spec change in charge of
what gets drawn, across a namespace boundary. `emit`'s three branches are three
different marks and stay three branches.
Kept, because it is one operation with two callers rather than two concerns that
rhyme: `lineage`, which `depth` and `z-path` were both walking separately. Its
cycle check is now a length bound -- a chain that does not repeat cannot be
longer than the node count -- instead of a `seen` set.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PDfHGdV39zu6rvgbBTfDaT
Steps 2 and 3 land together because the model revisions in the middle changed
code from both, and splitting them now would invent intermediate states that
never built.
domain/channel value-at across framed/keyed/dense, plus a cursor
domain/node decomposed transform, composition order, time maps
domain/scene topological order, z paths, eval-frame and resolver
clock audio-clocked frame derivation, outside app-db
db/events/subs re-frame arrives; the playhead is document state
ui/player the rAF loop; reads, blits, dispatches (almost) nothing
ui/shell transport
133 tests, 1158 assertions. The scene plays at 30fps against audio, scrubs, and
runs at 1/4x through 4x; verified by driving a real browser over CDP rather than
by assertion.
Two evaluators, on purpose. `eval-frame` is the specification -- allocating,
order-free, obviously correct. `resolver` is what playback uses: cached topo
order and z paths, a cursor per channel, a preallocated point buffer per node.
Both run the same walk, parameterised only by how a channel is read and where
points are written, because two independent implementations of frame evaluation
would drift and the drift would read as a rendering bug rather than as two
functions disagreeing. scene-test asserts they agree frame for frame in forward,
backward and random order.
Deviations and decisions, each with a reason:
- raster/fill-poly! is now a thin wrapper over fill-poly-buf!, which takes a flat
preallocated buffer. ONE scanline fill serves the analysis stages, which speak
{:x :y}, and frame evaluation, which hands over a buffer it owns. The parity
suite still passes pixel-for-pixel, which is what makes the rewrite safe.
- The state mask carries ABSENCE ONLY. An earlier draft gave it a hidden bit too,
per architecture.md's "hidden flag + palette index", and that bit was a dense
[:vis] wearing a different hat -- two mechanisms for one question, which is how
a part ends up hidden by one and shown by the other.
- The palette is a parameter of evaluation, not a global. A node names a TONE;
which ramp that tone is read in belongs to the timeline it sits in.
- :over layers and a symbol :rate THROW rather than being ignored. Neither is
built and nothing can produce one, so this can only fire on data that has run
ahead of the code. A silently dropped override is a hand correction the user
made once, watched fail, and has no reason to trust again.
Three findings the model produced rather than received:
- Presence propagates asymmetrically. An absent transform drops the subtree; an
absent [:geom :pts] drops only that node, because an absent mouth outline has
nothing to draw but the head it hangs off has not moved. That asymmetry is the
reason presence is tracked per channel and not per node.
- Z paths need lexicographic compare, not `compare`, which orders vectors by
count first -- so a cel three levels under "a1" would jump in front of a bare
"a2" and the layer order would mostly work.
- A node stencilled by something that drew nothing is dropped, not drawn
unclipped: an iris floating over the cheek is worse than a missing iris.
docs/ revised alongside, and those revisions are the load-bearing part:
- A scene, a timeline and a symbol are one type. The doc had two structures with
the same fields and never said so. Two axes of nesting are now separated --
parent/child within a timeline is flat with parent pointers, instance nesting
is by reference -- which is why "nestable" and "flat" only sounded
contradictory.
- Palettes are named, live on the project, and are ENABLED on a timeline as a
channel. Absent inherits; present travels with the timeline, so a symbol
authored against :night stays night wherever it is placed. The output index
space is the concatenation of the named ramps, which keeps one buffer and one
flat table and incidentally stops two nodes in different palettes colliding on
a stencil.
- Stabilisation is a channel, not a mode: {s, theta, tx, ty} IS [:xform :*], so
the normalise on/off/per-plate toggle is which of the three channel shapes the
:head node carries. Always measure and always store factored -- smoothing and
velocity-minimum key selection both need the split to exist in storage.
- There is no camera node and none is needed. Placement is a node transform, the
stage clips what hangs off it, and project dimensions are independent of the
footage. `makeXform` is therefore not to be ported: it bakes a cropping
decision into every stored vertex.
- Export is removed. The .take writer was for an Animator Pro render script; the
target is encoding video in the browser, and step 9 now says not to port the
old one.
demo/swarm is 120 shapes on six orbits, entirely dense blocks behind store
handles -- the shape freeze produces at step 5, and the first thing to exercise
that path under load. It plays at 30fps, and bench-test keeps a deliberately
loose floor under it because a performance regression here does not announce
itself: the picture stays correct and merely arrives late.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PDfHGdV39zu6rvgbBTfDaT
Scaffolds frontend/ (shadow-cljs, reagent 1.2.0, re-frame 1.4.3) and ports
everything below the data model, with the JS kept as a numeric oracle.
domain/landmarks index tables, verbatim
domain/ring subsample, offset, simplicity
domain/geom similarity fit, procrustes, moving average
domain/raster indexed scanline fill, stencil, disc, rect
domain/palette the ramp, and the no-sampled-RGB rule
58 tests, 166 assertions. Parity with js/ on the identical 72-frame synthetic
track: fit-similarity, procrustes-mean, fit-residual, moving-average,
smooth-transforms, offset-ring and subsample-slots to 1e-9; the raster
pixel-for-pixel over the whole buffer.
Three deviations from the JS, each for a reason:
- synth.cljs jitters from a SEEDED generator, not Math.random. Parity is only
checkable if both sides can be handed the same track, and a failing assertion
has to be reproducible. `:rand-fn` takes the generator over, so oracle.mjs
stubs js/Math.random and js/ itself stays untouched.
- raster/->rgba replaces toImageData. ImageData is a DOM type and domain/ may
not touch the DOM; returning plain bytes also lets the
no-intermediate-colours assertion run in node. ui/canvas wraps it later.
- offset-ring lives in domain/ring, not domain/geom, per architecture.md: it is
an operation on an ordered traversal, not on a transform.
Step 1's "done" also names the swapped-iris vote, but pairIrises is in
pipeline.js and belongs to step 7. The precondition is asserted instead --
`:swap-iris` really does move both blocks -- so the vote will have a track that
disagrees with it when it arrives.
One finding, recorded in full in the test that measures it: smooth-transforms
buys nothing on the synthetic track. Against jitter-free ground truth, radius 1
helps by 17% on one noise realisation and hurts by 0.5% on another, so its
benefit is within noise; from radius 2 up the cost is unambiguous, and by radius
5 the filter is below the true motion's own high-frequency energy, i.e.
smoothing away performance. The test pins the shape of the knob rather than a
preferred value. This may say more about the synth's jitter being unrealistically
small (+/-0.001 normalised) than about the knob; step 6 settles it on real
footage.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Copy previous looked for the nearest earlier ENABLED frame, and every frame is
enabled until you thin the strip out - so it resolved to f-1, which has nothing
on it, and the button appeared to only ever copy the immediate sibling while
reporting "nothing to copy".
It now takes the nearest earlier enabled frame that actually carries a drawing.
That behaves identically before and after you curate the strip, which is the
point: the rhythm of the drawings should not depend on whether you have got
round to deleting frames yet.
Cels stay tied to the keep-set. They are plate drawings and they hold until the
next enabled frame, as originally specified - an earlier version of this commit
gave them their own independent set, which is wrong for what they are.
Frames carrying a drawing are now marked in the strip, because "copy previous"
reaching back to a frame you cannot see is not much better than it reaching to
the wrong one.
Also fixes stale paint labels: onChange refreshed the strip and the panes but
not the paint header, so the drawn-frames readout lagged a copy behind. Split
the text update out of drawPaint so it can run from inside commit() without
re-rendering the canvas underneath itself.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A sketch, and labelled as one. It exists to test whether the aesthetic holds
when a human draws the background rather than the tracker deriving it, and it
is meant to be replaced by a real paint surface with onion skin and undo. Kept
to one dependency-free module so throwing it away is a delete, not surgery.
Cels are drawn on the frames that get their own drawing and hold until the
next one - the same rule the plate follows, and literally the same lookup, so
the two cannot disagree about what is on screen. Editing always targets the
cel you can see, so you can scrub anywhere and keep drawing.
Pen places vertices and closes on the first one. Edit drags vertices or whole
shapes, shift-click inserts, alt-click removes. Layers stack front-at-top with
per-layer colour, show/hide and reorder. Drag a strip thumbnail onto the canvas
to seed this cel from that one, every layer, as a deep copy - sharing the point
arrays would make two cels silently edit each other.
Two rules are enforced rather than left to discipline:
Colours are PALETTE INDICES, never RGB. Sampling colour from the source is the
one move docs/design.md calls irrecoverable, and a paint tool is exactly where
that discipline would leak, so the picker cannot express a colour outside the
ramp.
Vertices snap to the 320x200 grid. On a hard-edged indexed rasteriser a shape
nudged by 0.4px moves an edge by a whole pixel or not at all depending on where
it happens to land, so sub-pixel vertices shimmer instead of holding still.
Drawings autosave to localStorage per take name. They are the only thing here a
person made by hand; everything else regenerates. Not in the .take export yet.
Also adds serve.py, a no-store dev server. python3 -m http.server sends
Last-Modified and browsers cache ES modules on it hard enough that a reload
serves a stale app.js against a fresh index.html: the new knobs appear, nothing
wires them, no error fires, and it reads as "your feature does not work". That
cost real time this session.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A brow at 320x200 is fourteen pixels wide and three tall. Its shape carries
almost nothing at that size; its height above the eye carries the expression,
and a brow raise is the most legible beat on a face. So the ring is traced and
the height is quantised - the split the eyes already got, where the lid is a
traced feature and the iris a quantised primitive.
The decomposition is the point. The traced ring already contains the real
height, so adding a quantised raise on top would move the brow twice. The
height is measured OUT of the ring, quantised, and put back, so the shape that
renders is his at a height that snaps between a few levels and holds.
Measured at both ends rather than as one number, because raise and tilt are
different expressions out of one mechanism: both ends up is surprise, inner up
alone is worry, inner down is anger. They share a dwell - the gaze quantiser,
renamed quantizeSnap now that it has two callers - so the brow hits its pose in
one frame instead of crawling into it with one end arriving before the other.
Measured against the eye's corner midpoint, never its lid. Same trap the gaze
origin has and worth avoiding twice: brows and lids move together constantly,
so a brow that jumped on every blink would read as a tic. Rest pose from the
take median rather than the neutral frame, for the reason gaze learned the hard
way - that frame is picked by minimum mouth aperture and says nothing about the
brows.
Two correspondences resolved from geometry, not declared: which ring is which
brow, and which end is the outer one. The second matters more - backwards, the
tilt mirrors and worry renders as its own opposite, which reads as a directed
performance choice rather than a bug and would never be questioned. Which EDGE
is upper is deliberately left unresolved: it traverses the same ring the other
way, an even-odd fill has no winding, and both ends still land on fixed slots.
Also fixes a bug from the exposure work: the live render applied exposure to
the plate and the mouth but not to the eyes, so on 2s the preview and the
export disagreed. A preview that disagrees with the export is the one bug this
tool cannot afford. perfIndex now exists as a named thing so the two paths
cannot drift apart again.
91 -> 105 assertions. Ground truth on all four synthetic brow poses, tilt
separating worry from anger by sign, a blink not faking a raise, and a shared
dwell never emitting a half-raised brow.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
105 frames at 24fps from a screen recording, replacing IMG_8486. Ripped dense
on purpose now that exposure chooses the timing at render time.
Near-frontal this is not - it is a 3/4 view, so residual will run high and it
is the case the design doc says wants hand-drawn plates. That makes it the
better test clip for the eyes: there is real gaze movement in it, and the
out-of-plane bias on the projected iris offset is visible rather than
hypothetical.
frames/ is gitignored and regenerable: ./extract.sh <clip> 24
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Three parts per eye, stacked the way the mouth is - dark lash ring, sclera
inside it, iris inside that, square pupil in the iris. A blink then costs
nothing: when the lid shuts the traced ring goes flat and the lash line
collapses to a lens, which is a closed eye, drawn correctly, for free.
Lids are a FEATURE, rotoscoped like the mouth: head-local, a key on every
frame, the same contour avg knob. The iris is a PRIMITIVE - a disc at a
quantised position - and that is where the stylisation lives.
Line of sight. Gaze is the iris centre relative to the midpoint of the eye's
two corners, in units of corner distance. Both corners are in RIGID, so the
origin and the scale are immune to the performance being measured; against the
lid ring's centroid instead, every blink would drag the origin down and fake a
glance at the floor on exactly the frames where the eye is most visible. Both
eyes share one gaze - at this size the difference between the two measurements
is noise, not vergence, and independent per-eye noise reads as wall-eyed
immediately. Openness stays per-eye so a wink survives.
Gaze is then quantised to a pixel grid with a dwell, which is not a
stylisation imposed on the truth: real eyes move in saccades, and the smooth
drift left in the measurement is tracker noise plus head-compensation error.
Snapping to a grid removes the noise and recovers the saccade in one operation.
The iris is placed in the frame of the already-smoothed, already-subsampled lid
ring - slots 0 and 8 of a 16-slot ring are the corners, and subsampling to any
even budget keeps them at 0 and n/2 - so it cannot drift relative to its own
eye. Size is authored from the take mean, never remeasured per frame: a radius
that breathes by a fraction of a pixel flickers a pixel on and off around the
whole silhouette. iris anchor toggles steady/free/locked, because how much the
eye wanders turns out to be an aesthetic choice and not only a correctness one.
Blinking gets hysteresis and a dwell like the teeth, plus one knob they do not
have: blink hold. A blink is one frame at 12fps and a single frame of closed
eye reads as a dropped frame, so once the eye shuts it stays shut long enough
to be legible. Detection accuracy is not the problem; legibility is.
The pupil is a square because at three pixels a circle is a plus sign with the
corners gnawed off, and it changes shape as it moves. Drawn from a rounded
centre shared with the iris so it is exactly its nominal size on every frame.
Iris/pupil clip by colour key against the indexed buffer, the way Animator Pro
would: the lid crops the iris at extreme gaze for free, so nothing has to clamp
the gaze, which would flatten the performance at the extremes that carry it.
Which iris block belongs to which eye is RESOLVED from geometry, not declared.
A swap looks almost right - each eye still has a disc roughly where it belongs
- so it survives an eyeball and then reads as a subtly wall-eyed character
forever. Voted across every frame; the test feeds a deliberately swapped track.
Also: exposure. Aesthetic sparseness was set by the extraction rate, which made
the timing a property of a directory of PNGs - auditioning 12 against 24 meant
re-ripping and re-detecting the whole clip. It is now a render-time grid, on
1s/2s/3s/4s, so the dense track keeps everything and the audio clock is
untouched. The take format already carried an exposure field; it was never
driven. Everything rides the same grid, because a head cutting on the odd
frames while the mouth cuts on the even ones reads as two performances laid
over each other.
41 -> 91 assertions. The load-bearing new ones: the iris pairing follows a
swapped track, a blink does not fake a change of gaze, a stencilled disc cannot
spill past its clip, a 3px pupil is 3x3 at every sub-pixel centre, and exposure
never reads a pose from the future.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The test renderer turned out to be the product. Everything that decides how the
work looks - stabilisation, reduction, timing, frame removal, palette - already
happens here, and the flat indexed output already reads the way it should.
The reason to leave is in the original design's own rule: never make a timing
decision that requires a full render to evaluate. Honouring that moved every
judgement out of Animator Pro, which left the host doing nothing but writing a
file, in exchange for modal UI, minutes-long renders, one-level undo, FLX delta
invariants, a single tween state and a cel singleton.
What does NOT change is the constraint. 320x200, indexed palette, flat fills,
no antialiasing - inherited, but load-bearing rather than accidental. The
rasteriser writes palette indices and expands to RGBA only at the end precisely
so nothing can soften an edge. Modern conveniences belong in the workflow.
Adds docs/design.md: the principles, carried over without the Poco/FLX/cel
machinery, plus architecture and an honest list of what is missing - the
largest gap being that plates still have nowhere to be drawn.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
"It feels like it does nothing" is indistinguishable from "it does nothing",
and at 24fps a lead of 1 is 42ms - small enough to reasonably doubt. So the
shift is now provable and visible rather than taken on trust:
- shiftIndex is a pure exported function with assertions covering identity,
both directions, and clamping at each end
- the frame label always shows the mouth frame, not only when shifted, so the
number can be watched diverging from f; non-zero leads also report in ms
- the slider readout carries an explicit + sign
- the stabilised pane draws the unshifted contour as a dark-green ghost when a
lead is set, so the offset is something you can see
Also: leadIndex called opts() on every invocation - ~14 DOM reads, once per
strip thumbnail, so ~1000 per redraw at 74 frames. It reads a cached scalar now.
The "different pose" assertion checks the whole track rather than one pair:
synthetic poses hold for nine-frame beats, so a single pair can legitimately be
identical while the shift works correctly.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
opts() declared `verts` twice - once from the lip slider and again from the
teeth slider. Duplicate keys in an object literal are silent in JS and the last
one wins, so the teeth vertex control was quietly setting the lip vertex budget
while the lip control did nothing at all.
Renamed to teethVerts, with interior.js reading it under that name.
selftest now parses the opts() literal and fails on duplicate keys, verified to
catch this exact case.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Smoothing does not literally delay anything - a centred moving average has zero
phase lag - but it blurs onsets, so the visually salient moment of a mouth
opening moves later even though the mean does not. Animators also draw mouth
shapes a frame or two ahead of the sound as standard practice, so this is the
normal control rather than a workaround.
Only the performance tracks shift; the head and audio stay put, since it is the
mouth that should anticipate. In photo-underlay modes the vector mouth will
therefore no longer match the frame behind it, which is expected.
The lead is baked into the exported take - key f carries the pose from source
frame f+lead - so the Animator Pro renderer never needs to know about it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A knob wired in app.js but missing from the markup throws during wiring, aborts
the module and leaves a blank page - a symptom pointing nowhere near its cause.
It has now happened twice, so it gets a check rather than vigilance: the
selftest fetches both files and compares the id sets in each direction.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The band filled the mouth because a band is the wrong reduction: the bright
region is a blob, and reading it as "everything above a line" throws the shape
away.
Extracting a contour reintroduces the vertex-correspondence problem that made
me avoid it, but for a blob there is a way out. Radial sampling from the
centroid along N fixed directions makes vertex k always mean "the extent in
direction k": correspondence holds by construction, the count is fixed, and
temporal smoothing cannot reorder anything. It also yields a star-shaped
reduction, which suits flat colour.
Tongue rejection, which the band had no way to express:
- pixels red relative to their own brightness are dropped (teeth are neutral)
- component choice is biased toward the top of the cavity, since area alone
picks the tongue when the mouth is wide
- separate inner and outer controls: cavity erode pulls the sampled region off
the lip edge, blob grow/erode resizes the found blob
Also: a knob wired in app.js but missing from index.html threw during wiring
and left a blank page with nothing useful in the console - which is exactly
what happened to teethDwell in the previous commit. el() now names the missing
id, and window.onerror surfaces it in the status line.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Three causes, all of them mine:
Otsu always returns a split, including on a homogeneous region - given a dark
cavity with no teeth it invents a threshold and calls half the pixels bright.
The gate is now the separation between the two class means, which is the only
thing that says whether the split means anything. Coverage was the wrong
signal: it is high both when the mouth is full of teeth and when the region is
uniformly dark and Otsu has split noise.
The row scan tracked the last qualifying row anywhere rather than where the run
from the top stops, so one bright row near the bottom - a lit lower lip inside
the ring - pushed the line to full height. It now breaks at the first failing
row once the run has started.
MediaPipe's inner lip landmarks sit slightly outside the real opening, so the
sampled region included lip pixels, which are bright and sit exactly at the
boundary where they do most damage. The ring is now eroded toward its centroid
before sampling, with the amount exposed as a knob.
Adds a diagnostic panel showing the sampled crop, pixels above threshold, and
the resolved line, because tuning this from numbers alone does not tell you
whether the region being measured is even the right region.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
MediaPipe has no landmarks inside the lips, so teeth have to come from pixels.
Tracing the bright blob would give a new contour every frame with no vertex
correspondence - the exact boil docs/roto-puppet.md warns about.
So the measurement yields a scalar, not a shape: Otsu within the cavity, scanned
from the top for where the bright run stops, giving one line height per frame.
The teeth polygon is the inner lip ring clipped to that line, so the silhouette
is always the mouth's own shape and cannot disagree with the lips around it,
and the only per-frame variable is a single number that smooths trivially.
Presence gets hysteresis and minimum dwell, as plate selection does: a teeth
block blinking on and off for single frames is worse than one simply absent.
Tongue is not implemented. The same scalar approach would apply, gated on
redness rather than brightness, but it is not visible in the test footage - the
cavity reads dark with a bright upper-teeth band and nothing else.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
