noe/arthur
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Compare commits

base: noe:master
Branches Tags
noe:master
noe:port/cljs-scaffold
noe:eyes
..
compare: noe:eyes
Branches Tags
noe:master
noe:port/cljs-scaffold
noe:eyes

No commits in common. "master" and "eyes" have entirely different histories.
master ... eyes

252 changed files with 53 additions and 70784 deletions
Download patch file Download diff file Expand all files Collapse all files

1
.claude/worktrees/tracing-layers

@ -1 +0,0 @@
Subproject commit f4dd04764204506fc275180364d0366d693036f4

18
.dockerignore
View file

@ -1,18 +0,0 @@
.git
.venv
**/node_modules
frontend/.shadow-cljs
frontend/out
frontend/.cpcache
frontend/test/browser/out
static/arthur/js
db.sqlite3
var
frames
scratch
audio.wav
manifest.json
*.take
*.tflite
.claude
.venv*

35
.gitignore vendored
View file

@ -1,40 +1,5 @@
# extract.sh's output. It is TIER 3 — immutable, large, and the backend's to serve
# once `manage.py ingest_bundle` has hashed it into var/blobs — so none of it
# belongs in the repo. `audio.wav` was tracked before step 9 because the synthetic
# take borrowed it for a clock; that copy now lives at static/arthur/audio.wav,
# which is an asset the project owns rather than an extraction that churns.
frames/
/audio.wav
/manifest.json
# local extracted takes for comparing source cadences
/scratch/
*.task
*.take
*.tflite
# CLJS build
frontend/node_modules/
# screenshots from the browser suite; regenerated by `npm run browser`
frontend/test/browser/out/
frontend/.shadow-cljs/
frontend/out/
frontend/.cpcache/
static/arthur/js/
# mise-managed venv for the Django half
.venv/
# the Django half's own state: the document database, the content-addressed blob
# store (tiers 2 and 3), and collectstatic's output
db.sqlite3
db.sqlite3-shm
db.sqlite3-wal
/var/
# vim swap files
*.swp
# Python bytecode
__pycache__/
*.py[cod]

43
Dockerfile
View file

@ -1,43 +0,0 @@
FROM node:20-bookworm AS frontend
RUN apt-get update \
&& apt-get install -y --no-install-recommends ca-certificates curl tar \
&& rm -rf /var/lib/apt/lists/* \
&& curl -fsSL 'https://api.adoptium.net/v3/binary/latest/21/ga/linux/x64/jdk/hotspot/normal/eclipse' -o /tmp/jdk.tar.gz \
&& mkdir -p /opt/java \
&& tar -xzf /tmp/jdk.tar.gz -C /opt/java --strip-components=1 \
&& rm /tmp/jdk.tar.gz
ENV JAVA_HOME=/opt/java
ENV PATH="/opt/java/bin:${PATH}"
WORKDIR /app/frontend
COPY frontend/package.json frontend/package-lock.json ./
RUN npm ci
COPY frontend/ ./
RUN npx shadow-cljs release app
FROM python:3.12-slim-bookworm
ENV PYTHONDONTWRITEBYTECODE=1 \
PYTHONUNBUFFERED=1 \
DJANGO_DEBUG=0 \
DJANGO_DB_PATH=/data/db.sqlite3 \
DJANGO_BLOB_ROOT=/data/blobs
RUN apt-get update \
&& apt-get install -y --no-install-recommends ffmpeg \
&& rm -rf /var/lib/apt/lists/* \
&& useradd --create-home --shell /usr/sbin/nologin app
WORKDIR /app
COPY requirements.txt ./
RUN pip install --no-cache-dir -r requirements.txt
COPY . ./
COPY --from=frontend /app/static/arthur/js/ ./static/arthur/js/
RUN python manage.py collectstatic --noinput \
&& mkdir -p /data \
&& chown -R app:app /app /data
USER app
EXPOSE 8000
CMD ["sh", "-c", "python manage.py migrate --noinput && exec daphne --bind 0.0.0.0 --port 8000 server.asgi:application"]

123
README.md
View file

@ -14,62 +14,6 @@ Animator Pro, where this started, but they are why the output looks right —
modern conveniences belong in the workflow, not the output. See
[docs/design.md](docs/design.md).
## ClojureScript port
The active port plays the synthetic take, accepts video uploads, transcodes them
to an H.264 proxy and decodable stream plus audio and tracing stills, analyzes real footage
for mouth, eyes, brows and pixel-derived teeth, and saves the project with
reusable analysis data. The step 8 data model
represents persistent feature IDs, eye pairs and feature-level observation gaps;
its controls are still pending. See the [port plan](docs/port-plan.md).
```sh
mise install # both halves
pip install -r requirements.txt
mise exec -- python manage.py migrate
./do start # Django + frontend watcher
```
In the app, drop a video on the media pool — it uploads, extracts and runs
detection — then **save**. Opening that project on another client reuses its saved landmarks and
mouth crops without detecting source frames again. The upload path derives its
footage response from database records; it does not create or consume a
`manifest.json` file. See [frontend/README.md](frontend/README.md) for details.
Anything under "## Run" and below describes the older JS prototype, which still
runs separately on port 8777.
### Deploy to Fly.io
The app uses a persistent Fly volume for its SQLite document database and
content-addressed footage blobs. Create the app once, set its Django secret, and
deploy from the repository root:
```sh
fly apps create arthur --org personal
fly volumes create data --region iad --size 1
fly secrets set DJANGO_SECRET_KEY="$(openssl rand -hex 32)" --app arthur
fly deploy --app arthur
```
The deployed app is at <https://arthur.fly.dev>. The container builds the
ClojureScript frontend, collects static assets, and runs database migrations on
startup.
### How it is stored
Three tiers, cut by mutability and size — the full argument is in
[docs/architecture.md](docs/architecture.md):
| Tier | What | Where |
| --- | --- | --- |
| 1 **authored** | the scene: nodes, channels, features, time maps | the database, as independently addressed leaves. Kilobytes |
| 2 **derived** | detected landmarks, raw mouth crops, and dense channel blocks | `var/blobs`, addressed by analysis and block inputs, including the detector version |
| 3 **source** | the uploaded video, H.264 proxy and elementary stream, tracing stills, and audio | the same blob store, by the hash of their bytes |
Only tier 1 is the document. Tier 2 is a pure function of tiers 1 and 3, so a
saved project names its blocks rather than carrying them, and a knob change gives
a block a new name rather than overwriting an old one.
## Run
```sh
@ -89,35 +33,27 @@ wasm, which is fetched from a CDN on first use.
For real footage:
Upload it in the app. `./extract.sh` still writes the old PNG-sequence bundle and
`ingest_bundle` still registers it, but footage ingested that way has no decodable
stream and the loader will say so — the measured pixels come out of the video now.
```sh
./extract.sh /path/to/clip.mov 12 # -> frames/*.png, audio.wav, manifest.json
```
MediaPipe's wasm and `face_landmarker.task` are both local; nothing in detection
touches the network.
then **Load frames**. MediaPipe's wasm is fetched from jsdelivr on first use;
`face_landmarker.task` is local.
Detection reads the H.264 elementary stream with WebCodecs, one coded frame at a
time. The proxy has no B-frames, so decode order is frame order. Each decoded
frame reaches MediaPipe in VIDEO running mode at its footage timestamp. The
decoder and detector advance together, with a pause between frames so progress
can paint. Saved analyses reuse their stored crop pixels and measure them with
the same pauses.
Frames are pre-extracted rather than decoded in the page because browser video
seeking is approximate and `requestVideoFrameCallback` only delivers frames at
playback speed — neither gives a deterministic per-frame pass.
This is what replaced the PNG sequence, which was 112MB for 7.6 seconds and would
be 1.1GB at the 900-frame limit. The proxy is 6MB, and the landmarks barely
notice: detected off decoded H.264 rather than off the PNGs, they moved at most
0.0033 of frame width.
`manifest.json` records the true extraction rate. The page reads it rather than
assuming, because a guessed fps desynchronises audio from picture — and sync is
the one thing this view exists to show.
The server's footage manifest records the proxy's frame rate and frame count;
the page reads that rate because a guessed fps desynchronises audio from picture.
Choosing a lower picture rate happens after analysis.
**Picture fps** decides how often the finished roto gets a new pose. Analyze all
source frames, then sample those frozen poses at 12, 24 or the source rate while
keeping the original duration and audio. **Exposure** can hold a drawing across
more than one picture slot. The tracing editor chooses source frames for cel
references separately. Shared timing is the useful default for mouth, eyes,
teeth and plate so their changes read as one performance.
**Exposure** decides how often the picture gets a new drawing: rip at 24 and
render `on 2s` for 12, `on 3s` for 8. The dense track and the audio are
untouched, so it is a dropdown rather than a re-rip, and the export emits keys
only on the grid instead of the same pose twice. Everything rides the same grid
— mouth, eyes, teeth, plate — because a head cutting on the odd frames while the
mouth cuts on the even ones reads as two performances laid over each other.
**Audio is the playback clock**: `frame = floor(audio.currentTime * fps)`. A slow
render loop therefore drops frames instead of drifting, and ½x / ¼x work by
@ -378,13 +314,13 @@ the tool a person made by hand; everything else regenerates. They are not in the
## Two kinds of sparseness
Sparseness has two unrelated causes, and conflating them was the original design
error here. **Aesthetic** sparseness is chosen from the full analyzed source
track at rendering time. **Labour** sparseness is a human drawing each cel and
selecting which source frames to use as tracing references.
error here. **Aesthetic** sparseness is set by the extraction rate — pick 12fps and
you have already chosen your timing. **Labour** sparseness is a human drawing
each one, and it binds only on the plate.
Aesthetic sparseness is the **picture fps** control, with exposure available for
longer holds. Both happen after analysis, so auditioning 12 against 24 needs no
re-extraction or re-detection.
Aesthetic sparseness is the **exposure** control, not the extraction rate —
making it a render-time grid means auditioning 12 against 24 costs a dropdown
instead of a re-rip and a full re-detection.
So the mouth keeps **every** frame: it is traced, and therefore free. In limited
animation lip sync is routinely the densest element, on 1s, while heads hold on
@ -437,16 +373,3 @@ performer→character calibration (currently identity, fitting the face oval to
canvas); the override layer; anything on the Animator Pro side. The plate is a
face-oval polygon per kept frame — it exists so the mouth has a face to read
against, not to look good.
In the port specifically: the parameter UI and scoped regeneration (the model is
built, the controls are not); automatic per-feature detection, so presence still
comes from the full-face mask plus a manifest annotation; multiplayer, for which
step 9 built the addressing and none of the socket; and in-browser extraction, so
`extract.sh` plus `manage.py ingest_bundle` is still how footage arrives.
Two smaller things that are known and undecided. `measure/brows` takes no
`presence` where `measure/eyes` does, so an occluded brow affects the freeze mask
but not brow measurement, and occluded landmarks still enter contour smoothing —
asymmetric with the eyes, and it is not settled which way is right. And `open`
takes the most recently updated project and shows its first clip: there is no
project browser, and the runtime store holds one clip at a time.

BIN
audio.wav Normal file
View file

Binary file not shown.

0
clips/__init__.py
View file

63
clips/admin.py
View file

@ -1,63 +0,0 @@
"""The admin, which is here for one reason: tier 1 is readable.
docs/architecture.md's argument against a CRDT is partly this — "the canonical
document moves into an opaque blob, and every server-side thing that reads the
document needs it materialised back out". A leaf is transit-as-JSON in a
JSONField, so it is legible here, and that is a property worth being able to see.
"""
from django.contrib import admin
from .models import Analysis, Block, Blob, Clip, Footage, FootageFrame, Leaf, Project, Revision
@admin.register(Project)
class ProjectAdmin(admin.ModelAdmin):
list_display = ("name", "id", "seq", "updated")
search_fields = ("name", "id")
@admin.register(Clip)
class ClipAdmin(admin.ModelAdmin):
list_display = ("cid", "project", "name")
list_filter = ("project",)
@admin.register(Leaf)
class LeafAdmin(admin.ModelAdmin):
list_display = ("path", "project", "version", "updated")
list_filter = ("project",)
search_fields = ("path",)
@admin.register(Revision)
class RevisionAdmin(admin.ModelAdmin):
list_display = ("project", "seq", "summary", "author", "created")
@admin.register(Footage)
class FootageAdmin(admin.ModelAdmin):
list_display = ("label", "source", "fps", "frames", "width", "height", "created")
@admin.register(FootageFrame)
class FootageFrameAdmin(admin.ModelAdmin):
list_display = ("footage", "index", "blob")
list_filter = ("footage",)
@admin.register(Analysis)
class AnalysisAdmin(admin.ModelAdmin):
list_display = ("key", "detector", "version", "footage", "created")
search_fields = ("key", "detector", "version")
@admin.register(Block)
class BlockAdmin(admin.ModelAdmin):
list_display = ("key", "role", "analysis", "data", "state", "created")
list_filter = ("role",)
search_fields = ("key",)
@admin.register(Blob)
class BlobAdmin(admin.ModelAdmin):
list_display = ("digest", "media_type", "size", "created")

14
clips/apps.py
View file

@ -1,14 +0,0 @@
from django.apps import AppConfig
class ClipsConfig(AppConfig):
"""The one app.
`clips` because the CLIP is the entity the whole tool is about and the one the
prototype had exactly one of and never named — `state` in `js/app.js` is a clip
with its analysis inlined and its palette global. Project, Footage, Analysis,
Block, Leaf and Revision all hang off it.
"""
default_auto_field = "django.db.models.BigAutoField"
name = "clips"

146
clips/blobs.py
View file

@ -1,146 +0,0 @@
"""The content-addressed blob store: tiers 2 and 3 on disk.
One store for both, and docs/architecture.md says why in a sentence: once tier 3
is decoded by the app rather than by a shell script, frames and audio become "the
same kind of thing as tier 2 — a cache with a hash". So there is one place that
writes bytes, one that reads them, and one URL shape for both.
TWO KINDS OF HASH, AND THEY ARE NOT THE SAME HASH. A blob is named by the sha256
of its BYTES: that is what makes identical frames in two extractions one file. A
derived thing — an analysis artifact, a dense block — is named by a sha256 over
its INPUTS, which is what lets the client ask for the block the current settings
want before anything has computed it. So `Block.key` is an input hash and
`Block.data.digest` is a byte hash, and conflating them would break the half of
addressing that answers questions about work not yet done.
"""
import hashlib
import os
import tempfile
import zlib
from pathlib import Path
from django.conf import settings
CHUNK = 1 << 20
CROP_MEDIA_TYPE = "application/zlib"
def digest_bytes(data: bytes) -> str:
return hashlib.sha256(data).hexdigest()
def digest_file(path: Path) -> str:
h = hashlib.sha256()
with open(path, "rb") as fh:
while chunk := fh.read(CHUNK):
h.update(chunk)
return h.hexdigest()
def path_for(digest: str) -> Path:
"""Where a blob lives.
Fanned out two levels, so that a take's worth of frames does not put a hundred
thousand entries in one directory — which is slow on every filesystem and
unusable on some.
"""
if len(digest) != 64 or any(c not in "0123456789abcdef" for c in digest):
raise ValueError(f"not a sha256: {digest!r}")
return Path(settings.BLOB_ROOT) / digest[:2] / digest[2:4] / digest
def write(data: bytes) -> tuple[str, int]:
"""Store bytes, return (digest, size). Writing the same bytes twice is a
no-op, which is what content addressing is for."""
digest = digest_bytes(data)
dest = path_for(digest)
if not dest.exists():
dest.parent.mkdir(parents=True, exist_ok=True)
tmp = dest.with_suffix(".part")
with open(tmp, "wb") as fh:
fh.write(data)
os.replace(tmp, dest)
return digest, len(data)
def write_stream(chunks) -> tuple[str, int]:
"""Store an uploaded file without reading the whole video into memory."""
root = Path(settings.BLOB_ROOT)
root.mkdir(parents=True, exist_ok=True)
digest = hashlib.sha256()
size = 0
with tempfile.NamedTemporaryFile(dir=root, prefix="upload-", delete=False) as out:
temporary = Path(out.name)
try:
for chunk in chunks:
digest.update(chunk)
size += len(chunk)
out.write(chunk)
except BaseException:
temporary.unlink(missing_ok=True)
raise
dest = path_for(digest.hexdigest())
dest.parent.mkdir(parents=True, exist_ok=True)
if dest.exists():
temporary.unlink()
else:
os.replace(temporary, dest)
return digest.hexdigest(), size
def write_compressed_stream(chunks) -> tuple[str, int]:
"""Store a losslessly compressed stream; the digest names stored bytes."""
compressor = zlib.compressobj()
def compressed():
for chunk in chunks:
if part := compressor.compress(chunk):
yield part
if part := compressor.flush():
yield part
return write_stream(compressed())
def adopt(source: Path) -> tuple[str, int]:
"""Store a file already on disk, by hard link where the filesystem allows it.
112MB of PNGs is a normal extraction and copying them into a second place in
the tree for no reason is not. A hard link is exact — the blob is immutable, so
two names for one inode is the whole of what is wanted — and a copy is the
fallback when `extract.sh` wrote to another volume.
"""
digest = digest_file(source)
dest = path_for(digest)
size = source.stat().st_size
if not dest.exists():
dest.parent.mkdir(parents=True, exist_ok=True)
try:
os.link(source, dest)
except OSError:
tmp = dest.with_suffix(".part")
with open(source, "rb") as src, open(tmp, "wb") as out:
while chunk := src.read(CHUNK):
out.write(chunk)
os.replace(tmp, dest)
return digest, size
def read(digest: str) -> bytes:
with open(path_for(digest), "rb") as fh:
return fh.read()
def png_size(path: Path) -> tuple[int, int]:
"""A PNG's dimensions, out of its IHDR.
Twenty-four bytes rather than a dependency. The footage's width and height are
manifest data — docs/architecture.md's entity model puts them there — and
Pillow to read two integers out of a header that has held them in the same
place since 1996 is not a trade worth making.
"""
with open(path, "rb") as fh:
head = fh.read(24)
if head[:8] != b"\x89PNG\r\n\x1a\n" or head[12:16] != b"IHDR":
raise ValueError(f"{path} is not a PNG")
return int.from_bytes(head[16:20], "big"), int.from_bytes(head[20:24], "big")

87
clips/consumers.py
View file

@ -1,87 +0,0 @@
"""One socket per open project, and it is tl's, nearly line for line.
Two things ride it. DELTAS, which the server sends after a write commits — the
socket is read-only for the document, and a dropped socket cannot lose a write.
PRESENCE, which peers gossip between themselves: all the server does is hand out
a connection id and stamp the sender's identity onto every message, so nobody can
post as somebody else.
"""
import json
import uuid
from asgiref.sync import async_to_sync
from channels.generic.websocket import AsyncWebsocketConsumer
from channels.layers import get_channel_layer
# Who is connected, per project: {group: {cid: presence}}. A cache of what has
# already been relayed, so a joiner gets the room in one message. Process-local,
# like the in-memory channel layer this runs on.
ROOMS = {}
def group(project_id):
return f"project_{project_id}"
def broadcast(project_id, delta, kind="delta"):
"""Send a committed write to everyone in the project's room. `access` says
only that who may write has changed, and each client asks for itself."""
async_to_sync(get_channel_layer().group_send)(
group(project_id), {"type": "project.delta", "delta": {"kind": kind, **delta}},
)
class ProjectConsumer(AsyncWebsocketConsumer):
RELAYED = ("state",)
@property
def room(self):
return ROOMS.setdefault(self.group, {})
async def connect(self):
self.group = group(self.scope["url_route"]["kwargs"]["project_id"])
self.cid = uuid.uuid4().hex[:12]
user = self.scope.get("user")
self.username = user.get_username() if user and user.is_authenticated else None
await self.channel_layer.group_add(self.group, self.channel_name)
await self.accept()
me = {"cid": self.cid, "user": self.username}
others = list(self.room.values())
self.room[self.cid] = me
await self.send(text_data=json.dumps({"kind": "welcome", **me}))
await self.send(text_data=json.dumps({"kind": "roster", "peers": others}))
await self._relay({"kind": "join"})
async def disconnect(self, code):
if hasattr(self, "cid"):
self.room.pop(self.cid, None)
if not self.room:
ROOMS.pop(self.group, None)
await self._relay({"kind": "leave"})
await self.channel_layer.group_discard(self.group, self.channel_name)
async def receive(self, text_data=None, bytes_data=None):
try:
msg = json.loads(text_data or "{}")
except ValueError:
return
if not isinstance(msg, dict) or msg.get("kind") not in self.RELAYED:
return
if self.cid in self.room:
self.room[self.cid].update(
{k: v for k, v in msg.items() if k not in ("kind", "cid", "user")}
)
await self._relay(msg)
async def _relay(self, msg):
await self.channel_layer.group_send(
self.group,
{"type": "peer.msg", "msg": {**msg, "cid": self.cid, "user": self.username}},
)
async def peer_msg(self, event):
await self.send(text_data=json.dumps(event["msg"]))
async def project_delta(self, event):
await self.send(text_data=json.dumps(event["delta"]))

501
clips/extraction.py
View file

@ -1,501 +0,0 @@
"""Upload a video once, then turn it into the two things the app actually reads.
WHAT CHANGED AND WHY. This used to decode one PNG per source frame and store every
one of them. A 7.6-second 1440x1920 take is 112MB that way, and the 900-frame limit
is 1.1GB — for pixels whose only consumer was a canvas that MediaPipe then read
once. The page now detects from the video itself (see `frontend/src/arthur/flow/
ingest.cljs`), so this produces:
THE PROXY. One browser-safe H.264/yuv420p MP4, CFR, `+faststart`. The same take
is 6MB. This is the analysis source, and it is re-encoded RATHER THAN KEPT AS
UPLOADED even when the upload is already H.264, for two reasons that are both
about not guessing: an iPhone's HEVC is not decodable in every browser, and the
footage's identity is the digest of this file — one produced by one ffmpeg
invocation, not one that depends on which branch the source happened to take.
THE TRACING STILLS. One JPEG per frame, long edge capped, for the tracing editor
to draw over. Reference images; nothing measures them. They are not in the
footage digest — see `models.Footage`.
The proxy is probed after it is written rather than before. `width`, `height` and
`frames` are properties of the file the browser will decode, and taking them from
the source instead is how a scaler or a dropped frame becomes a silent one-frame
offset between the landmarks and the audio.
"""
import hashlib
import json
import subprocess
import tempfile
import threading
import time
from fractions import Fraction
from pathlib import Path
from django.db import close_old_connections, transaction
from . import blobs
from .models import Blob, Extraction, Footage, FootageFrame
_active = set()
_lock = threading.Lock()
TIMEOUT = 3600
# Visually lossless enough that landmarks do not move: measured against the same
# frames as PNGs, IMAGE-mode landmarks shifted at most 0.0033 of frame width.
PROXY_CRF = "18"
# The long edge of a tracing still. The proxy keeps full resolution because the
# detector reads it; a still only has to be good enough to draw a cel over.
TRACING_EDGE = 1280
TRACING_QUALITY = "4"
def _command(args):
result = subprocess.run(args, capture_output=True, text=True, timeout=TIMEOUT)
if result.returncode:
raise ValueError((result.stderr or result.stdout or "media tool failed")[-1200:])
return result.stdout
def _run_with_progress(job, args, root, name, total, span):
"""Run one ffmpeg and publish its live frame count as `span` of the job.
ffmpeg's `-progress` file is the only honest source for this: parsing its
stderr means parsing a format that is explicitly not an interface, and a
spinner that is not attached to frames is a spinner that lies on a long take.
"""
progress_path = root / f"{name}.progress"
log_path = root / f"{name}.log"
first, last = span
args = ["ffmpeg", "-hide_banner", "-loglevel", "error", "-y",
"-stats_period", "0.25", "-progress", str(progress_path)] + args
with open(log_path, "wb") as log:
proc = subprocess.Popen(args, stdout=log, stderr=subprocess.STDOUT)
deadline = time.monotonic() + TIMEOUT
try:
while proc.poll() is None:
if time.monotonic() >= deadline:
raise TimeoutError(f"{name} timed out")
if progress_path.exists():
lines = progress_path.read_text(errors="replace").splitlines()
count = next((int(line[6:].strip()) for line in reversed(lines)
if line.startswith("frame=") and
line[6:].strip().isdigit()), 0)
if count and total:
reached = first + int((last - first) * min(1.0, count / total))
if reached > job.progress:
job.progress = reached
job.save(update_fields=["progress", "updated"])
time.sleep(0.2)
finally:
if proc.poll() is None:
proc.kill()
proc.wait()
if proc.returncode:
raise ValueError(log_path.read_text(errors="replace")[-1200:] or f"{name} failed")
def _encode_proxy(job, source_path, proxy_path, facts, root):
"""The uploaded video -> one H.264 file every browser can decode and seek."""
total = facts.get("reported_frames") or round(facts["duration"] * facts["fps"])
_run_with_progress(
job,
["-i", str(source_path), "-an",
# Constant frame rate at the rate `probe` chose. This RESAMPLES rather
# than asserts: the upload is allowed to be variable, and this is the
# step that makes the thing the page measures not be.
"-fps_mode", "cfr", "-r", facts.get("rate") or str(facts["fps"]),
"-c:v", "libx264", "-preset", "veryfast", "-crf", PROXY_CRF,
# NO B-FRAMES, AND THIS IS THE LOAD-BEARING FLAG. It is what makes
# decode order presentation order, so the page can treat access unit k
# of the elementary stream as frame k without demuxing a container or
# consulting a timestamp. With them x264 has a
# two-frame reordering delay, ffmpeg compensates by writing an edit list
# (`elst` media_time 1024 at timebase 1/15360 — exactly two frames), and
# the browser then lives on two timelines at once: `currentTime` obeys the
# edit list and the `mediaTime` reported by requestVideoFrameCallback does
# not. Seek to frame 0 and the browser correctly hands back a frame whose
# mediaTime says 2. Software decoding hides it; hardware decoding does
# not, which is the worst possible way for it to be wrong. Without
# B-frames DTS equals PTS, no edit list is written, and the two timelines
# are the same one. It also makes decode order presentation order, should
# this ever be fed to a WebCodecs VideoDecoder.
"-bf", "0",
# yuv420p and an even frame size are what makes this playable everywhere
# rather than only in the browser that happened to be tested.
"-pix_fmt", "yuv420p", "-vf", "scale=trunc(iw/2)*2:trunc(ih/2)*2",
"-movflags", "+faststart", str(proxy_path)],
root, "proxy", total, (0, 55))
def _elementary_stream(proxy_path, out_path):
"""The proxy's video, unwrapped into a raw Annex-B H.264 stream.
A STREAM COPY, not a second encode: the same coded frames as the MP4, with
the container's length-prefixed NAL units rewritten as start-code-delimited
ones. It costs a file read and nothing else.
This exists because the page decodes with WebCodecs, and `VideoDecoder` takes
demuxed chunks rather than a container. Handing it Annex-B means the client
needs no demuxer: NAL start codes are findable in a loop, and because the
proxy is encoded with no B-frames, decode order is presentation order — so
access unit k IS frame k, with no container timing to consult and no clock to
reconcile. That is the whole reason this file is worth the bytes it costs.
"""
_command(["ffmpeg", "-hide_banner", "-loglevel", "error", "-y",
"-i", str(proxy_path), "-an", "-c:v", "copy",
"-bsf:v", "h264_mp4toannexb", "-f", "h264", str(out_path)])
def _extract_stills(job, proxy_path, frames_dir, frames, root):
"""The proxy -> one tracing JPEG per frame, long edge capped."""
_run_with_progress(
job,
["-i", str(proxy_path), "-fps_mode", "passthrough",
"-vf", f"scale='if(gt(iw,ih),min({TRACING_EDGE},iw),-2)':"
f"'if(gt(iw,ih),-2,min({TRACING_EDGE},ih))'",
"-q:v", TRACING_QUALITY, str(frames_dir / "%04d.jpg")],
root, "stills", frames, (55, 85))
MAX_RATE = 120 # a capture rate; past this the container is describing something else
# How many packet timestamps `_measured_rate` reads, and the fewest intervals it
# will draw a conclusion from. 300 is a flat cost on a long take and still a
# wide enough sample for a median; below 8 intervals there is not enough of a
# stream to outvote one odd timestamp, so the metadata is left to speak.
RATE_SAMPLE = 300
RATE_MINIMUM = 8
# How far a declared rate may sit from the measured one and still be taken as
# what the stream is: 2% covers 30 against 30000/1001 and nothing like 120
# against 30.
RATE_TOLERANCE = 0.02
def probe_image(path):
"""An uploaded still's pixel size as (width, height), refusing anything that
is not one picture."""
data = json.loads(_command(["ffprobe", "-v", "error", "-show_streams",
"-of", "json", str(path)]))
video = [s for s in data.get("streams", []) if s.get("codec_type") == "video"]
if len(video) != 1 or not (video[0].get("width") and video[0].get("height")):
raise ValueError("the uploaded file is not an image")
return int(video[0]["width"]), int(video[0]["height"])
def probe_audio(path):
"""The length of an uploaded sound in seconds, refusing a file with no audio."""
data = json.loads(_command(["ffprobe", "-v", "error", "-show_streams",
"-show_format", "-of", "json", str(path)]))
if not any(s.get("codec_type") == "audio" for s in data.get("streams", [])):
raise ValueError("the uploaded file has no audio stream")
duration = float(data.get("format", {}).get("duration") or 0)
if duration <= 0:
raise ValueError("the sound's length is unknown")
return duration
def _measured_rate(path):
"""The rate the stream's own packet timestamps imply, or None.
THE CONTAINER'S SUMMARY OF ITSELF IS NOT EVIDENCE, and this is the function
that goes and looks. An iPhone's `r_frame_rate` is 120 on footage whose
timestamps are 1/30s apart, which is the difference between 323 frames and
1293 — four times the encode, four times the tracing stills, four times the
blobs, for 970 frames that are copies of their neighbours.
It reads TIMESTAMPS, not frames: `-show_entries packet=pts_time` demuxes
without decoding, so this costs a file read and no pixels. The times are
SORTED before differencing because a stream with B-frames arrives in decode
order — an HEVC clip's first packets come out 0, 0.133, 0.067, 0.033 — and
differencing that order measures the reordering rather than the rate.
THE MEDIAN INTERVAL, which is what makes this safe on genuinely variable
input. It answers "how far apart are two frames normally", so a take held on
one frame for a second still reports the rate of the parts that move, and
choosing it keeps every distinct frame — the property `probe` used to reach
for by taking the nominal rate. Only the last few intervals of the sample are
unreliable (a frame whose turn comes after the window is missing from it), and
a median does not care.
Returning None is the honest answer for a clip too short to sample, and this
also swallows a probe that fails outright: the rate the metadata declares is
the documented fallback, so an optimisation must not be able to refuse an
upload that would otherwise have been accepted.
"""
try:
text = _command(["ffprobe", "-v", "error", "-select_streams", "v:0",
"-show_entries", "packet=pts_time", "-of", "json",
"-read_intervals", f"%+#{RATE_SAMPLE}", str(path)])
packets = json.loads(text).get("packets") or []
times = sorted(float(packet["pts_time"]) for packet in packets
if (packet.get("pts_time") or "N/A") != "N/A")
except (ValueError, OSError):
return None
intervals = sorted(b - a for a, b in zip(times, times[1:]) if b > a)
if len(intervals) < RATE_MINIMUM:
return None
median = intervals[len(intervals) // 2]
return 1.0 / median if median > 0 else None
def _choose_rate(nominal, average, measured):
"""The rate to resample onto, as an exact Fraction.
A DECLARED RATE IS PREFERRED WHEN IT AGREES WITH THE TIMESTAMPS, because it is
the exact rational the stream was authored at — 30000/1001 is not a float, and
`limit_denominator` on a measured 29.97 is a guess at a number the container
already states. So the measured rate is used to CHOOSE between what the
container declares, and only stands in itself when neither declaration
describes the stream.
"""
candidates = [rate for rate in (nominal, average) if 0 < rate <= MAX_RATE]
if measured:
agreeing = [rate for rate in candidates
if abs(float(rate) - measured) <= RATE_TOLERANCE * measured]
if agreeing:
return min(agreeing, key=lambda rate: abs(float(rate) - measured))
from_timestamps = Fraction(measured).limit_denominator(1001)
if 0 < from_timestamps <= MAX_RATE:
return from_timestamps
# Nothing to go on but the metadata, and nominal first keeps the rate that
# drops no distinct frame. An unusable pair falls through to the refusal
# below, which names the rate the file claimed rather than one of these.
return nominal if 0 < nominal <= MAX_RATE else average
def probe(path):
"""What the upload is, as far as choosing a proxy rate goes.
IT NO LONGER REFUSES VARIABLE-FRAME-RATE INPUT, and the reason is the proxy.
That refusal was written when the page measured the source's own frames, where
a wandering frame duration really does break `frame = floor(t * fps)`. Nothing
measures the source now: ffmpeg resamples it onto a constant rate, and the
proxy — constant by construction, and re-probed after it is written — is the
only timeline anything downstream sees.
Keeping the check would have been worse than useless, because the thing it
tested is not reliable. Ordinary iPhone footage, shot straight from the camera
app, reports `avg_frame_rate` 8670/299 and `nb_frames` 289 on a stream whose
decoded timestamps are 280 frames exactly 1/30s apart. The container's summary
of itself disagreed with the container's own contents, so the guard rejected
CFR video for being variable.
THE RATE IS MEASURED AND THE DECLARATIONS ARE VOTED ON, which is the same
distrust applied to the one number that still comes from here. This used to
take `r_frame_rate` outright — the rate every timestamp in the stream can be
expressed at, and so the rate that keeps every distinct source frame. The
trouble is that it is not a claim about frames at all: the file above declares
120 and holds 30, and resampling it up cost four times the encode, four times
the tracing stills and four times the blobs for 970 duplicated frames. So
`_measured_rate` reads the timestamps, `_choose_rate` keeps whichever declared
rate they bear out, and the nominal rate is believed when it is true rather
than because it is nominal.
Duration is preserved either way — ffmpeg's CFR conversion is driven by
timestamps, so the audio stays in sync at any rate — and the median interval
keeps the no-distinct-frame-dropped property that taking the nominal rate was
reaching for. See `_measured_rate`.
"""
data = json.loads(_command(["ffprobe", "-v", "error", "-show_streams",
"-show_format", "-of", "json", str(path)]))
video = next((s for s in data.get("streams", []) if s.get("codec_type") == "video"), None)
if not video:
raise ValueError("the uploaded file has no video stream")
nominal = Fraction(video.get("r_frame_rate") or "0")
average = Fraction(video.get("avg_frame_rate") or "0")
if nominal <= 0 and average <= 0:
raise ValueError("the video's frame rate is unknown")
measured = _measured_rate(path)
rate = _choose_rate(nominal, average, measured)
if not 0 < rate <= MAX_RATE:
raise ValueError(f"the video reports a frame rate of {float(rate):g}, which is "
"not a rate footage can be measured at")
duration = float(data.get("format", {}).get("duration") or 0)
# if duration > 0 and duration * float(rate) > 901:
# raise ValueError("video is longer than the 900-frame footage limit")
frames = video.get("nb_frames")
return {"fps": float(rate),
# The exact rate, for ffmpeg. 30000/1001 is not a float, and handing
# `-r` a rounded one is how a long take drifts out of sync.
"rate": f"{rate.numerator}/{rate.denominator}",
"nominal_fps": float(nominal), "average_fps": float(average),
# What the timestamps said, and null when there were too few to ask.
# Recorded because it is the input to a decision this file used not to
# make, and the one number that explains a chosen rate matching
# neither declaration.
"measured_fps": measured,
"width": int(video["width"]), "height": int(video["height"]),
"duration": duration,
# KEPT, AND NO LONGER TRUSTED AS A COUNT. See the docstring: this is
# the container's claim about itself, it is wrong on ordinary phone
# footage, and `run` checks the proxy's DURATION instead.
"reported_frames": int(frames) if frames and frames.isdigit() else None,
"has_audio": any(s.get("codec_type") == "audio" for s in data.get("streams", [])),
"vfr": nominal != average}
def _refuse_a_shifted_timeline(path):
"""The proxy must put frame `i` at `i / fps` on BOTH of the browser's clocks.
Asserted rather than assumed, because the failure is silent and the symptom is
unrecognisable. An encoder delay makes ffmpeg write an edit list, `currentTime`
then obeys it while `requestVideoFrameCallback`'s `mediaTime` does not, and the
page's frame walk is uniformly off by the delay — on hardware decoding only. It
cost two wrong diagnoses to find, so it does not get to come back silently if
somebody changes an encoder flag.
"""
data = json.loads(_command(["ffprobe", "-v", "error", "-select_streams", "v:0",
"-show_streams", "-of", "json", str(path)]))
stream = data["streams"][0]
if int(stream.get("has_b_frames") or 0):
raise ValueError(
"the proxy was encoded with B-frames, whose reordering delay makes the "
"browser's seek clock and its frame-timestamp clock disagree")
if float(stream.get("start_time") or 0) != 0:
raise ValueError(f"the proxy starts at {stream['start_time']}s rather than 0")
def count_frames(path):
"""How many frames a file really holds, counted rather than reported.
`nb_frames` is a container's claim. This is the decoder's answer, and it is
what the page will get when it walks the proxy — so a disagreement between the
two has to be settled before the count reaches a manifest, not after it has
become a one-frame audio offset nobody can find.
"""
text = _command(["ffprobe", "-v", "error", "-select_streams", "v:0",
"-count_frames", "-show_entries", "stream=nb_read_frames",
"-of", "default=nokey=1:noprint_wrappers=1", str(path)])
counted = text.strip()
if not counted.isdigit():
raise ValueError("could not count the proxy's frames")
return int(counted)
def extraction_key(source, settings):
# Scheme 3: the extraction now also produces the elementary stream the page
# decodes, so a job run under scheme 2 did not make everything this one does.
text = json.dumps({"scheme": 3, "source": source.blob_id, "settings": settings},
sort_keys=True, separators=(",", ":"))
return "sha256:" + hashlib.sha256(text.encode()).hexdigest()
def _register(job, proxy_path, stream_path, stills, audio_path, facts):
proxy_digest, proxy_size = blobs.adopt(proxy_path)
stream_digest, stream_size = blobs.adopt(stream_path)
audio_digest, audio_size = blobs.adopt(audio_path)
still_blobs = [(index, *blobs.adopt(path)) for index, path in enumerate(stills)]
width, height, fps, frames = facts["width"], facts["height"], facts["fps"], facts["frames"]
# The footage's own identity: the bytes the page will measure, the audio it
# will clock against, and the rate that ties them together. Scheme 2 — scheme
# 1 hashed a PNG per frame, and those footages name pixels this no longer has.
h = hashlib.sha256()
h.update(f"arthur-footage-2/{fps}/{frames}/{width}x{height}\n".encode())
h.update(proxy_digest.encode())
h.update(audio_digest.encode())
with transaction.atomic():
proxy_blob, _ = Blob.objects.get_or_create(
digest=proxy_digest, defaults={"size": proxy_size, "media_type": "video/mp4"})
stream_blob, _ = Blob.objects.get_or_create(
digest=stream_digest, defaults={"size": stream_size, "media_type": "video/h264"})
audio_blob, _ = Blob.objects.get_or_create(
digest=audio_digest, defaults={"size": audio_size, "media_type": "audio/wav"})
footage, created = Footage.objects.get_or_create(
digest=h.hexdigest(),
defaults={"label": job.source.filename[:200], "source": job.source.filename[:200],
"fps": fps, "frames": frames, "width": width, "height": height,
"audio": audio_blob, "video": proxy_blob, "stream": stream_blob})
if not created and not footage.stream_id:
# The same footage by identity, extracted before the elementary
# stream existed. Its digest is over the proxy and the audio, which
# have not changed — so this is the same footage gaining a file it
# was always entitled to, not a different one.
footage.stream = stream_blob
if not footage.video_id:
footage.video = proxy_blob
footage.save(update_fields=["stream", "video"])
if created:
rows = []
for index, digest, size in still_blobs:
blob, _ = Blob.objects.get_or_create(
digest=digest, defaults={"size": size, "media_type": "image/jpeg"})
rows.append(FootageFrame(footage=footage, index=index, blob=blob))
FootageFrame.objects.bulk_create(rows)
return footage
def run(key):
close_old_connections()
try:
job = Extraction.objects.select_related("source", "source__blob").get(key=key)
job.state, job.progress, job.error = "running", 0, ""
job.save(update_fields=["state", "progress", "error", "updated"])
facts = job.source.probe
source_path = blobs.path_for(job.source.blob_id)
with tempfile.TemporaryDirectory(prefix="arthur-extract-") as directory:
root = Path(directory)
proxy_path = root / "proxy.mp4"
_encode_proxy(job, source_path, proxy_path, facts, root)
# Everything downstream describes the PROXY, not the upload.
proxy_facts = probe(proxy_path)
_refuse_a_shifted_timeline(proxy_path)
frames = count_frames(proxy_path)
#if not 1 <= frames <= 900:
# raise ValueError(f"the proxy holds {frames} frames; the limit is 1–900")
# CHECKED AS A DURATION, not as a frame count. The page's clock is
# `frame = floor(audio.currentTime * fps)`, so what must not drift is
# how long the picture lasts against how long the audio lasts — and
# the source's own frame count is a number this has already caught
# lying. A resample to a constant rate legitimately changes the count
# and must not change the duration.
drift = abs(frames / proxy_facts["fps"] - facts["duration"])
if facts["duration"] > 0 and drift > 0.5:
raise ValueError(
f"the proxy runs {frames / proxy_facts['fps']:.2f}s and the upload "
f"runs {facts['duration']:.2f}s; refusing footage whose picture and "
"audio would drift")
proxy_facts["frames"] = frames
stream_path = root / "proxy.h264"
_elementary_stream(proxy_path, stream_path)
frames_dir = root / "stills"
frames_dir.mkdir()
_extract_stills(job, proxy_path, frames_dir, frames, root)
stills = sorted(frames_dir.glob("*.jpg"))
if len(stills) != frames:
raise ValueError(f"wrote {len(stills)} tracing stills for {frames} frames")
job.progress = 85
job.save(update_fields=["progress", "updated"])
audio_path = root / "audio.wav"
if facts["has_audio"]:
_command(["ffmpeg", "-hide_banner", "-loglevel", "error", "-y",
"-i", str(source_path), "-vn", "-ac", "1", "-ar", "44100",
str(audio_path)])
else:
_command(["ffmpeg", "-hide_banner", "-loglevel", "error", "-y",
"-f", "lavfi", "-i", "anullsrc=r=44100:cl=mono",
"-t", str(frames / proxy_facts["fps"]), "-c:a", "pcm_s16le",
str(audio_path)])
footage = _register(job, proxy_path, stream_path, stills, audio_path, proxy_facts)
job.footage, job.state, job.progress = footage, "done", 100
job.save(update_fields=["footage", "state", "progress", "updated"])
except Exception as exc:
Extraction.objects.filter(key=key).update(state="failed", error=str(exc)[:2000])
finally:
with _lock:
_active.discard(key)
close_old_connections()
def enqueue(key):
with _lock:
if key in _active:
return
_active.add(key)
threading.Thread(target=run, args=(key,), daemon=True,
name=f"arthur-extract-{key[7:15]}").start()

0
clips/management/__init__.py
View file

0
clips/management/commands/__init__.py
View file

57
clips/management/commands/compress_crop_blocks.py
View file

@ -1,57 +0,0 @@
"""Compress existing raw mouth crop blocks without changing their public bytes."""
import hashlib
import zlib
from django.core.management.base import BaseCommand, CommandError
from django.db import transaction
from django.db.models.deletion import ProtectedError
from clips import blobs
from clips.models import Blob, Block
class Command(BaseCommand):
help = "Compress existing source/crops blobs and remove unreferenced raw copies"
def handle(self, *args, **options):
converted = 0
before = after = 0
for block in Block.objects.filter(role="source/crops").select_related("data"):
old = block.data
if old.media_type == blobs.CROP_MEDIA_TYPE:
continue
old_digest = old.digest
with open(blobs.path_for(old_digest), "rb") as source:
digest, size = blobs.write_compressed_stream(
iter(lambda: source.read(blobs.CHUNK), b"")
)
check = hashlib.sha256()
decompressor = zlib.decompressobj()
with open(blobs.path_for(digest), "rb") as compressed:
while chunk := compressed.read(blobs.CHUNK):
check.update(decompressor.decompress(chunk))
check.update(decompressor.flush())
if not decompressor.eof or check.hexdigest() != old_digest:
raise CommandError(f"crop compression failed verification: {block.key}")
with transaction.atomic():
new, _ = Blob.objects.get_or_create(
digest=digest,
defaults={"size": size, "media_type": blobs.CROP_MEDIA_TYPE},
)
changed = Block.objects.filter(key=block.key, data=old).update(data=new)
if not changed:
continue
converted += 1
before += old.size
after += size
if old_digest != new.digest:
try:
old.delete()
except ProtectedError:
pass
else:
blobs.path_for(old_digest).unlink(missing_ok=True)
self.stdout.write(
f"Compressed {converted} crop blocks: {before:,} -> {after:,} bytes"
)

120
clips/management/commands/ingest_bundle.py
View file

@ -1,120 +0,0 @@
"""Register an extracted bundle as tier 3.
python manage.py ingest_bundle # ./manifest.json
python manage.py ingest_bundle scratch/my-take # that bundle
WHAT THIS REPLACES. Until step 9 the page fetched `/manifest.json` and then built
`frames/0001.png` itself, with shadow-cljs's `:dev-http` serving the repo root. So
the frame layout was a shared secret between a shell script and a ClojureScript
namespace, and "where the frames are" was answered by a directory listing.
Now the server names every frame, and the client asks it. The frames go into the
content-addressed blob store — by hard link, so 112MB of PNGs is not copied — and
the manifest the client receives carries a URL per frame. That is the whole of what
makes the frames the backend's to serve, and it is what the in-browser wasm-ffmpeg
extraction docs/architecture.md describes will upload INTO, without the client
learning anything new when it arrives: the same blobs, the same manifest, a
different producer.
`extract.sh` still does the decoding. It is out of step 9's scope, it works, and it
is the only part of this that needs a terminal.
"""
import json
from pathlib import Path
from django.core.management.base import BaseCommand, CommandError
from django.db import transaction
from clips import blobs
from clips.models import Blob, Footage, FootageFrame
class Command(BaseCommand):
help = "Register an extracted frames+audio+manifest bundle as footage."
def add_arguments(self, parser):
parser.add_argument(
"bundle", nargs="?", default=".",
help="a directory holding manifest.json, or the manifest itself",
)
parser.add_argument("--label", default="", help="what to call it in the UI")
def handle(self, *args, **options):
manifest_path = Path(options["bundle"])
if manifest_path.is_dir():
manifest_path = manifest_path / "manifest.json"
if not manifest_path.exists():
raise CommandError(f"{manifest_path} does not exist — run ./extract.sh first")
manifest = json.loads(manifest_path.read_text())
root = manifest_path.parent
frames_dir = root / manifest["dir"]
audio_path = root / manifest["audio"]
count = int(manifest["frames"])
pngs = sorted(frames_dir.glob("*.png"))
if len(pngs) != count:
raise CommandError(
f"the manifest says {count} frames and {frames_dir} holds {len(pngs)}; "
"refusing an inaccurate footage"
)
if not audio_path.exists():
raise CommandError(f"{audio_path} does not exist")
width, height = blobs.png_size(pngs[0])
self.stdout.write(f"hashing {len(pngs)} frames…")
frame_blobs = []
for i, png in enumerate(pngs):
digest, size = blobs.adopt(png)
frame_blobs.append((i, digest, size))
if (i + 1) % 25 == 0 or i + 1 == len(pngs):
self.stdout.write(f" {i + 1}/{len(pngs)}")
audio_digest, audio_size = blobs.adopt(audio_path)
# The footage's own identity: every frame in order, plus the audio and the
# rate. Two extractions of one clip at one rate are one footage, so an
# analysis over it is reusable across both.
import hashlib
h = hashlib.sha256()
h.update(f"arthur-footage-1/{manifest['fps']}/{count}/{width}x{height}\n".encode())
for _, digest, _ in frame_blobs:
h.update(digest.encode())
h.update(audio_digest.encode())
footage_digest = h.hexdigest()
if existing := Footage.objects.filter(digest=footage_digest).first():
self.stdout.write(self.style.SUCCESS(f"already ingested: {existing.id}"))
return
with transaction.atomic():
audio_blob, _ = Blob.objects.get_or_create(
digest=audio_digest,
defaults={"size": audio_size, "media_type": "audio/wav"},
)
footage = Footage.objects.create(
digest=footage_digest,
label=options["label"] or manifest.get("source") or frames_dir.name,
source=manifest.get("source") or "",
fps=float(manifest["fps"]),
frames=count,
width=width,
height=height,
audio=audio_blob,
feature_absence=manifest.get("feature-absence") or {},
)
rows = []
for index, digest, size in frame_blobs:
blob, _ = Blob.objects.get_or_create(
digest=digest, defaults={"size": size, "media_type": "image/png"}
)
rows.append(FootageFrame(footage=footage, index=index, blob=blob))
FootageFrame.objects.bulk_create(rows)
self.stdout.write(
self.style.SUCCESS(
f"{count} frames at {manifest['fps']}fps, {width}x{height} -> footage {footage.id}"
)
)

149
clips/migrations/0001_initial.py
View file

@ -1,149 +0,0 @@
# Generated by Django 5.2.17 on 2026-09-28 04:44
import django.db.models.deletion
import uuid
from django.db import migrations, models
class Migration(migrations.Migration):
initial = True
dependencies = [
]
operations = [
migrations.CreateModel(
name='Blob',
fields=[
('digest', models.CharField(max_length=64, primary_key=True, serialize=False)),
('media_type', models.CharField(default='application/octet-stream', max_length=100)),
('size', models.BigIntegerField()),
('created', models.DateTimeField(auto_now_add=True)),
],
),
migrations.CreateModel(
name='Project',
fields=[
('id', models.UUIDField(default=uuid.uuid4, editable=False, primary_key=True, serialize=False)),
('name', models.CharField(default='untitled', max_length=200)),
('seq', models.PositiveBigIntegerField(default=0)),
('palette', models.CharField(default='arthur/default', max_length=64)),
('created', models.DateTimeField(auto_now_add=True)),
('updated', models.DateTimeField(auto_now=True)),
],
options={
'ordering': ['-updated'],
},
),
migrations.CreateModel(
name='Analysis',
fields=[
('key', models.CharField(max_length=71, primary_key=True, serialize=False)),
('descriptor', models.TextField()),
('detector', models.CharField(max_length=64)),
('version', models.CharField(max_length=64)),
('created', models.DateTimeField(auto_now_add=True)),
('artifact', models.ForeignKey(blank=True, help_text='the dense landmark track, once bake A is uploaded', null=True, on_delete=django.db.models.deletion.SET_NULL, related_name='analysis_for', to='clips.blob')),
],
options={
'verbose_name_plural': 'analyses',
},
),
migrations.CreateModel(
name='Block',
fields=[
('key', models.CharField(max_length=71, primary_key=True, serialize=False)),
('descriptor', models.TextField()),
('role', models.CharField(max_length=32)),
('created', models.DateTimeField(auto_now_add=True)),
('analysis', models.ForeignKey(blank=True, null=True, on_delete=django.db.models.deletion.SET_NULL, related_name='blocks', to='clips.analysis')),
('data', models.ForeignKey(on_delete=django.db.models.deletion.PROTECT, related_name='block_data_for', to='clips.blob')),
('state', models.ForeignKey(blank=True, help_text='the per-track absence mask, when the take has one', null=True, on_delete=django.db.models.deletion.PROTECT, related_name='block_state_for', to='clips.blob')),
],
),
migrations.CreateModel(
name='Footage',
fields=[
('id', models.UUIDField(default=uuid.uuid4, editable=False, primary_key=True, serialize=False)),
('digest', models.CharField(max_length=64, unique=True)),
('label', models.CharField(blank=True, max_length=200)),
('source', models.CharField(blank=True, max_length=200)),
('fps', models.FloatField()),
('frames', models.PositiveIntegerField()),
('width', models.PositiveIntegerField()),
('height', models.PositiveIntegerField()),
('feature_absence', models.JSONField(blank=True, default=dict)),
('created', models.DateTimeField(auto_now_add=True)),
('audio', models.ForeignKey(on_delete=django.db.models.deletion.PROTECT, related_name='audio_for', to='clips.blob')),
],
options={
'ordering': ['-created'],
},
),
migrations.AddField(
model_name='analysis',
name='footage',
field=models.ForeignKey(blank=True, null=True, on_delete=django.db.models.deletion.SET_NULL, related_name='analyses', to='clips.footage'),
),
migrations.CreateModel(
name='Revision',
fields=[
('id', models.BigAutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')),
('seq', models.PositiveBigIntegerField()),
('author', models.CharField(blank=True, max_length=200)),
('summary', models.CharField(blank=True, max_length=500)),
('document', models.JSONField(help_text='every leaf of the project, by path')),
('created', models.DateTimeField(auto_now_add=True)),
('project', models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, related_name='revisions', to='clips.project')),
],
options={
'ordering': ['-seq'],
},
),
migrations.CreateModel(
name='FootageFrame',
fields=[
('id', models.BigAutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')),
('index', models.PositiveIntegerField(help_text="0-based; source frame index + 1 is the PNG's name")),
('blob', models.ForeignKey(on_delete=django.db.models.deletion.PROTECT, related_name='frame_for', to='clips.blob')),
('footage', models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, related_name='frame_set', to='clips.footage')),
],
options={
'ordering': ['index'],
'constraints': [models.UniqueConstraint(fields=('footage', 'index'), name='one_blob_per_frame')],
},
),
migrations.CreateModel(
name='Leaf',
fields=[
('id', models.BigAutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')),
('path', models.CharField(max_length=300)),
('value', models.JSONField()),
('version', models.PositiveBigIntegerField(default=1)),
('updated', models.DateTimeField(auto_now=True)),
('project', models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, related_name='leaves', to='clips.project')),
],
options={
'ordering': ['path'],
'constraints': [models.UniqueConstraint(fields=('project', 'path'), name='one_leaf_per_path')],
},
),
migrations.CreateModel(
name='Clip',
fields=[
('id', models.BigAutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')),
('cid', models.SlugField(max_length=64)),
('name', models.CharField(blank=True, max_length=200)),
('order', models.IntegerField(default=0)),
('analysis', models.ForeignKey(blank=True, null=True, on_delete=django.db.models.deletion.SET_NULL, related_name='clips', to='clips.analysis')),
('blocks', models.ManyToManyField(blank=True, help_text="the tier-2 blocks this clip's channels name", related_name='clips', to='clips.block')),
('footage', models.ForeignKey(blank=True, null=True, on_delete=django.db.models.deletion.SET_NULL, related_name='clips', to='clips.footage')),
('project', models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, related_name='clips', to='clips.project')),
],
options={
'ordering': ['order', 'cid'],
'constraints': [models.UniqueConstraint(fields=('project', 'cid'), name='one_cid_per_project')],
},
),
]

22
clips/migrations/0002_remove_analysis_artifact_analysis_source_blocks.py
View file

@ -1,22 +0,0 @@
# Generated by Django 5.2.17 on 2026-09-28 13:10
from django.db import migrations, models
class Migration(migrations.Migration):
dependencies = [
('clips', '0001_initial'),
]
operations = [
migrations.RemoveField(
model_name='analysis',
name='artifact',
),
migrations.AddField(
model_name='analysis',
name='source_blocks',
field=models.ManyToManyField(blank=True, help_text='pixel-dependent landmarks, detection mask and mouth crops', related_name='source_for', to='clips.block'),
),
]

39
clips/migrations/0003_source_extraction.py
View file

@ -1,39 +0,0 @@
# Generated by Django 5.2.17 on 2026-09-28 13:23
import django.db.models.deletion
import uuid
from django.db import migrations, models
class Migration(migrations.Migration):
dependencies = [
('clips', '0002_remove_analysis_artifact_analysis_source_blocks'),
]
operations = [
migrations.CreateModel(
name='Source',
fields=[
('id', models.UUIDField(default=uuid.uuid4, editable=False, primary_key=True, serialize=False)),
('filename', models.CharField(max_length=255)),
('probe', models.JSONField(default=dict)),
('created', models.DateTimeField(auto_now_add=True)),
('blob', models.OneToOneField(on_delete=django.db.models.deletion.PROTECT, related_name='video_source', to='clips.blob')),
],
),
migrations.CreateModel(
name='Extraction',
fields=[
('key', models.CharField(max_length=71, primary_key=True, serialize=False)),
('settings', models.JSONField(default=dict)),
('state', models.CharField(default='queued', max_length=16)),
('progress', models.PositiveIntegerField(default=0)),
('error', models.TextField(blank=True)),
('created', models.DateTimeField(auto_now_add=True)),
('updated', models.DateTimeField(auto_now=True)),
('footage', models.ForeignKey(blank=True, null=True, on_delete=django.db.models.deletion.SET_NULL, related_name='extractions', to='clips.footage')),
('source', models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, related_name='extractions', to='clips.source')),
],
),
]

24
clips/migrations/0004_footage_video_alter_footageframe_index.py
View file

@ -1,24 +0,0 @@
# Generated by Django 5.2.17 on 2026-09-28 15:10
import django.db.models.deletion
from django.db import migrations, models
class Migration(migrations.Migration):
dependencies = [
('clips', '0003_source_extraction'),
]
operations = [
migrations.AddField(
model_name='footage',
name='video',
field=models.ForeignKey(blank=True, help_text='the browser-safe proxy the page detects from; null on pre-proxy footage', null=True, on_delete=django.db.models.deletion.PROTECT, related_name='video_for', to='clips.blob'),
),
migrations.AlterField(
model_name='footageframe',
name='index',
field=models.PositiveIntegerField(help_text="0-based; source frame index + 1 is the JPEG's name"),
),
]

24
clips/migrations/0005_footage_stream_alter_footage_video.py
View file

@ -1,24 +0,0 @@
# Generated by Django 5.2.17 on 2026-09-28 17:11
import django.db.models.deletion
from django.db import migrations, models
class Migration(migrations.Migration):
dependencies = [
('clips', '0004_footage_video_alter_footageframe_index'),
]
operations = [
migrations.AddField(
model_name='footage',
name='stream',
field=models.ForeignKey(blank=True, help_text="the proxy's video as raw Annex-B H.264: what the page DECODES, one access unit per frame; null on footage extracted before it", null=True, on_delete=django.db.models.deletion.PROTECT, related_name='stream_for', to='clips.blob'),
),
migrations.AlterField(
model_name='footage',
name='video',
field=models.ForeignKey(blank=True, help_text='the browser-safe proxy, playable and seekable', null=True, on_delete=django.db.models.deletion.PROTECT, related_name='video_for', to='clips.blob'),
),
]

15
clips/migrations/0006_project_schema_version.py
View file

@ -1,15 +0,0 @@
from django.db import migrations, models
class Migration(migrations.Migration):
dependencies = [
("clips", "0005_footage_stream_alter_footage_video"),
]
operations = [
migrations.AddField(
model_name="project",
name="schema_version",
field=models.PositiveIntegerField(default=1),
),
]

75
clips/migrations/0007_symbols_not_timelines.py
View file

@ -1,75 +0,0 @@
"""Schema 2: a document holds symbols, not timelines, and no symbol is reserved.
Three renames, each in the stored transit and nowhere else:
clip/<cid>/timeline/... -> clip/<cid>/symbol/...
a node leaf's :kind :symbol -> :kind :instance
a feature leaf's :timeline key -> :symbol
A leaf value is transit's map form, ["^ ", k1, v1, k2, v2, ...]. Only TOP-LEVEL
pairs are rewritten, and only literal ones: transit caches a repeated keyword as
"^N", and a rename that met a cache reference where it expected the keyword would
be guessing. Every saved leaf at the time of writing had these as literals; if one
does not, the migration stops rather than writing a document that decodes to
something else.
Renaming a cached keyword in place is safe because the cache is positional: the
literal keeps its slot, so any later "^N" that referred to it now refers to the
new name, which is what it meant.
"""
import re
from django.db import migrations, models
PATH = re.compile(r"^(clip/[^/]+/)timeline(/|$)")
def _rename_pair(value, key, old, new, path):
if not (isinstance(value, list) and value[:1] == ["^ "]):
return value
out = list(value)
for i in range(1, len(out) - 1, 2):
if out[i] != key:
continue
if old is None:
out[i] = new
elif out[i + 1] == old:
out[i + 1] = new
elif isinstance(out[i + 1], str) and out[i + 1].startswith("^") and out[i + 1] != "^ ":
raise RuntimeError(f"leaf {path!r} has a cached {key} value; migrate it by hand")
return out
def forwards(apps, schema_editor):
Leaf = apps.get_model("clips", "Leaf")
Project = apps.get_model("clips", "Project")
for leaf in Leaf.objects.all():
path = PATH.sub(r"\1symbol\2", leaf.path)
value = leaf.value
parts = path.split("/")
if len(parts) == 6 and parts[2] == "symbol" and parts[4] == "node":
value = _rename_pair(value, "~:kind", "~:symbol", "~:instance", leaf.path)
if len(parts) == 4 and parts[2] == "feature":
value = _rename_pair(value, "~:timeline", None, "~:symbol", leaf.path)
if path != leaf.path or value != leaf.value:
leaf.path = path
leaf.value = value
leaf.version += 1
leaf.save(update_fields=["path", "value", "version"])
Project.objects.update(schema_version=2)
class Migration(migrations.Migration):
dependencies = [
("clips", "0006_project_schema_version"),
]
operations = [
migrations.AlterField(
model_name="project",
name="schema_version",
field=models.PositiveIntegerField(default=2),
),
migrations.RunPython(forwards, migrations.RunPython.noop),
]

39
clips/migrations/0008_owners_editors_leaf_seq.py
View file

@ -1,39 +0,0 @@
from django.conf import settings
from django.db import migrations, models
import django.db.models.deletion
def orphans(apps, schema_editor):
# Every project has an owner, and none of the ones saved before owners did.
apps.get_model("clips", "Project").objects.all().delete()
class Migration(migrations.Migration):
dependencies = [
("clips", "0007_symbols_not_timelines"),
migrations.swappable_dependency(settings.AUTH_USER_MODEL),
]
operations = [
migrations.RunPython(orphans, migrations.RunPython.noop),
migrations.AddField(
model_name="leaf",
name="seq",
field=models.PositiveBigIntegerField(
default=0, help_text="the project seq of the write that last changed it"),
),
migrations.AddField(
model_name="project",
name="editors",
field=models.ManyToManyField(blank=True, related_name="shared_projects",
to=settings.AUTH_USER_MODEL),
),
migrations.AddField(
model_name="project",
name="owner",
field=models.ForeignKey(on_delete=django.db.models.deletion.CASCADE,
related_name="projects", to=settings.AUTH_USER_MODEL),
preserve_default=False,
),
]

18
clips/migrations/0009_revision_blocks.py
View file

@ -1,18 +0,0 @@
# Generated by Django 5.2.17 on 2026-09-30 01:54
from django.db import migrations, models
class Migration(migrations.Migration):
dependencies = [
('clips', '0008_owners_editors_leaf_seq'),
]
operations = [
migrations.AddField(
model_name='revision',
name='blocks',
field=models.JSONField(default=dict, help_text="each clip's tier-2 block keys, by cid, so a restore can name them"),
),
]

25
clips/migrations/0010_sounds.py
View file

@ -1,25 +0,0 @@
# Generated by Django 5.2.17 on 2026-09-30 07:14
import django.db.models.deletion
import uuid
from django.db import migrations, models
class Migration(migrations.Migration):
dependencies = [
('clips', '0009_revision_blocks'),
]
operations = [
migrations.CreateModel(
name='Sound',
fields=[
('id', models.UUIDField(default=uuid.uuid4, editable=False, primary_key=True, serialize=False)),
('filename', models.CharField(max_length=255)),
('duration', models.FloatField(help_text='seconds, as ffprobe reports it')),
('created', models.DateTimeField(auto_now_add=True)),
('blob', models.ForeignKey(on_delete=django.db.models.deletion.PROTECT, related_name='sound_for', to='clips.blob')),
],
),
]

13
clips/migrations/0011_occurrence_schema.py
View file

@ -1,13 +0,0 @@
from django.db import migrations, models
class Migration(migrations.Migration):
dependencies = [("clips", "0010_sounds")]
operations = [
migrations.AlterField(
model_name="project",
name="schema_version",
field=models.PositiveIntegerField(default=3),
),
]

18
clips/migrations/0012_sound_label.py
View file

@ -1,18 +0,0 @@
# Generated by Django 5.2.17 on 2026-10-01 04:38
from django.db import migrations, models
class Migration(migrations.Migration):
dependencies = [
('clips', '0011_occurrence_schema'),
]
operations = [
migrations.AddField(
model_name='sound',
name='label',
field=models.CharField(blank=True, help_text='what a person called it; the filename when empty. Separate from `filename` because the name on disk is a fact about the upload and renaming must not rewrite it', max_length=200),
),
]

45
clips/migrations/0013_palette_track.py
View file

@ -1,45 +0,0 @@
"""Schema 4: split animated symbol palettes from the authoring palette.
Before schema 4 a symbol's ``:palette`` leaf value was always a channel. It now
names the static palette used when that symbol is the viewed root, while the
old channel is retained as ``:palette-channel`` compatibility data. New edits
use a real lane symbol referenced by ``:palette-track``.
"""
from django.db import migrations, models
def forwards(apps, schema_editor):
Leaf = apps.get_model("clips", "Leaf")
Project = apps.get_model("clips", "Project")
for leaf in Leaf.objects.filter(path__contains="/symbol/"):
parts = leaf.path.split("/")
if len(parts) != 4 or parts[2] != "symbol":
continue
value = leaf.value
if not (isinstance(value, list) and value[:1] == ["^ "]):
continue
out = list(value)
changed = False
for i in range(1, len(out) - 1, 2):
if out[i] == "~:palette" and isinstance(out[i + 1], list):
out[i] = "~:palette-channel"
changed = True
if changed:
leaf.value = out
leaf.version += 1
leaf.save(update_fields=["value", "version"])
Project.objects.update(schema_version=4)
class Migration(migrations.Migration):
dependencies = [("clips", "0012_sound_label")]
operations = [
migrations.AlterField(
model_name="project",
name="schema_version",
field=models.PositiveIntegerField(default=4),
),
migrations.RunPython(forwards, migrations.RunPython.noop),
]

15
clips/migrations/0014_multiple_analyses.py
View file

@ -1,15 +0,0 @@
from django.db import migrations, models
class Migration(migrations.Migration):
dependencies = [("clips", "0013_palette_track")]
operations = [
migrations.RemoveField(model_name="clip", name="analysis"),
migrations.RemoveField(model_name="clip", name="footage"),
migrations.AlterField(
model_name="project",
name="schema_version",
field=models.PositiveIntegerField(default=5),
),
]

48
clips/migrations/0015_tracing_images.py
View file

@ -1,48 +0,0 @@
"""Schema 6: tracing is a symbol.
A face's `:head :trace` is gone: its footage is a placement of a `:type :trace`
symbol under the head, the trace keys are that placement's `:time :holds`, and the
head follows them with `:reads`. Nothing is converted. Every project is marked 6,
and one that still carries a `:trace` is refused when it is opened, by name and
with what to do about it; the rest open as they did.
Images are stills to trace over, stored like sounds.
"""
import uuid
from django.db import migrations, models
import django.db.models.deletion
def forwards(apps, schema_editor):
apps.get_model("clips", "Project").objects.update(schema_version=6)
class Migration(migrations.Migration):
dependencies = [("clips", "0014_multiple_analyses")]
operations = [
migrations.CreateModel(
name="Image",
fields=[
("id", models.UUIDField(default=uuid.uuid4, editable=False,
primary_key=True, serialize=False)),
("filename", models.CharField(max_length=255)),
("label", models.CharField(
blank=True, max_length=200,
help_text="what a person called it; the filename when empty")),
("width", models.PositiveIntegerField(help_text="pixels, as ffprobe reports them")),
("height", models.PositiveIntegerField()),
("created", models.DateTimeField(auto_now_add=True)),
("blob", models.ForeignKey(on_delete=django.db.models.deletion.PROTECT,
related_name="image_for", to="clips.blob")),
],
),
migrations.AlterField(
model_name="project",
name="schema_version",
field=models.PositiveIntegerField(default=6),
),
migrations.RunPython(forwards, migrations.RunPython.noop),
]

42
clips/migrations/0016_an_anchor_is_a_peg.py
View file

@ -1,42 +0,0 @@
"""Schema 7: an anchor is a peg.
`[:xform :anchor]` is gone from the transform. `T(a)·M·T(-a)` is a transform
conjugated by a translation — "do M in a frame shifted by a" — and a parent
already is a shifted frame, so an anchor was a peg written inline: one that could
not be selected, keyed, shared between nodes, or placed above a measured channel.
A pivot nobody chose is now derived from what the node draws, per drag, and stored
nowhere; a pivot to keep is a peg, an ordinary `:group` parent.
Nothing is converted, as in schema 6. Every project is marked 7, and one that
still carries an anchor is refused when it is opened, by name and with what to do
about it — `node/problems` in the frontend.
Not converted rather than not worth converting. Dropping an anchor is in fact
pixel-exact wherever rotation and scale are the identity, since the anchor
cancels out of the composition there — and that is everywhere a freeze, a drop or
a new drawing wrote one. It is NOT exact on anything a hand has since turned or
scaled, where the composed translation is `a + p - M·a`, and it cannot be made
exact at all where `pos` is dense, because tier 2 is content-addressed and not
rewritable here. A conversion would therefore be silent and right for most nodes
and silent and wrong for exactly the ones somebody had hand-placed, which is the
worse failure: a refusal names the document and says what to do.
"""
from django.db import migrations, models
def forwards(apps, schema_editor):
apps.get_model("clips", "Project").objects.update(schema_version=7)
class Migration(migrations.Migration):
dependencies = [("clips", "0015_tracing_images")]
operations = [
migrations.AlterField(
model_name="project",
name="schema_version",
field=models.PositiveIntegerField(default=7),
),
migrations.RunPython(forwards, migrations.RunPython.noop),
]

45
clips/migrations/0017_a_node_has_a_pivot.py
View file

@ -1,45 +0,0 @@
"""Schema 8: a node has a pivot.
`[:xform :pivot]` is back in the transform, as the point rotation and scale are
composed about: `local = T(pos)·T(piv)·R·K·S·T(-piv)`. Schema 7 deleted it, on
the argument that an anchor is a peg — true as algebra, and not true as a feature.
A peg is a node, and a turn about a point that is not the turning node's own
origin still has to solve for a position to hold that point still; that solution
is an arc in the angle while a position channel tweens along the chord, so it is
right on the frame it is written and wrong on every frame between two keys. A
drawing escaped it, since its origin is the middle of what it draws. A symbol
instance could not: its origin is its symbol's, which is the top-left corner of
the stage, so one keyed turn of an instance swung its drawing round that corner
on an orbit the size of the stage.
CONVERTED, unlike 6 and 7, because adding this one is exact. A schema-7 node has
no pivot; an absent pivot reads as [0 0]; and T(pos)·T(0)·M·T(-0) is T(pos)·M to
the last bit of the mantissa. Every stored document therefore composes to exactly
the matrices it composed to before, dense tier-2 transforms included, so there is
nothing to guess at and no node a conversion could silently move. The version is
restamped and nothing else is touched.
What a converted document does NOT get is a pivot somebody chose: nodes placed
before this carry none, so they still turn about their own origin until the first
turn or scale writes one — `gesture/with-pivot`, from the middle of what the node
draws at that moment — or until the cross is dragged (ctrl/cmd-drag on the stage).
"""
from django.db import migrations, models
def forwards(apps, schema_editor):
apps.get_model("clips", "Project").objects.update(schema_version=8)
class Migration(migrations.Migration):
dependencies = [("clips", "0016_an_anchor_is_a_peg")]
operations = [
migrations.AlterField(
model_name="project",
name="schema_version",
field=models.PositiveIntegerField(default=8),
),
migrations.RunPython(forwards, migrations.RunPython.noop),
]

0
clips/migrations/__init__.py
View file

366
clips/models.py
View file

@ -1,366 +0,0 @@
"""The entity model, as tables.
It follows docs/architecture.md's model exactly, and the one thing worth reading
it for is which tier each table is in, because that is what decides whether a row
is a document, a cache entry or a source.
TIER 1, the document. Project, Clip, Leaf, Revision. Kilobytes, authored,
versioned, and the only tier anything will ever sync.
TIER 2, derived. Analysis, Block. Content-addressed by a hash over every input
that produced them — including the detector version — so a stale bake is
unreachable rather than wrong, and a collaborator's bake is fetchable by the
same key.
TIER 3, source. Footage, FootageFrame. Immutable, by hash.
Blob is under all three of them: bytes, named by the sha256 of themselves.
WHAT IS DELIBERATELY NOT HERE. `Clip` does not store fps, frames, width or height.
They are in the document — the `timing` and `stage` leaves — and a copy of them in
a column is a copy that comes to disagree with the scene it describes. The columns
`Clip` does have are the ones the SERVER needs to answer a question about a clip
without parsing its leaves: which footage, which analysis, which blocks.
"""
import uuid
from django.conf import settings
from django.db import models
from django.utils import timezone
class Blob(models.Model):
"""Bytes, named by the sha256 of themselves. The file is on disk under
`BLOB_ROOT`; this row is the index and the size."""
digest = models.CharField(primary_key=True, max_length=64)
media_type = models.CharField(max_length=100, default="application/octet-stream")
size = models.BigIntegerField()
created = models.DateTimeField(auto_now_add=True)
def __str__(self):
return f"{self.digest[:12]}… {self.size}B {self.media_type}"
class Source(models.Model):
"""An uploaded video, identified by its byte digest."""
id = models.UUIDField(primary_key=True, default=uuid.uuid4, editable=False)
blob = models.OneToOneField(Blob, on_delete=models.PROTECT, related_name="video_source")
filename = models.CharField(max_length=255)
probe = models.JSONField(default=dict)
created = models.DateTimeField(auto_now_add=True)
class Sound(models.Model):
"""An uploaded sound file — mp3, wav, whatever the browser can decode — kept
as uploaded. Not footage: it has no frames and nothing measures it, so it
skips extraction and an audio node plays its bytes directly."""
id = models.UUIDField(primary_key=True, default=uuid.uuid4, editable=False)
blob = models.ForeignKey(Blob, on_delete=models.PROTECT, related_name="sound_for")
filename = models.CharField(max_length=255)
label = models.CharField(
max_length=200, blank=True,
help_text="what a person called it; the filename when empty. Separate "
"from `filename` because the name on disk is a fact about the "
"upload and renaming must not rewrite it",
)
duration = models.FloatField(help_text="seconds, as ffprobe reports it")
created = models.DateTimeField(auto_now_add=True)
class Image(models.Model):
"""An uploaded still — a drawing, a photo, a model sheet — kept as uploaded, to
be traced over. Never part of the picture: a document names its blob as a
tracing symbol's `:media`, and the page draws it over the stage."""
id = models.UUIDField(primary_key=True, default=uuid.uuid4, editable=False)
blob = models.ForeignKey(Blob, on_delete=models.PROTECT, related_name="image_for")
filename = models.CharField(max_length=255)
label = models.CharField(max_length=200, blank=True,
help_text="what a person called it; the filename when empty")
width = models.PositiveIntegerField(help_text="pixels, as ffprobe reports them")
height = models.PositiveIntegerField()
created = models.DateTimeField(auto_now_add=True)
class Extraction(models.Model):
"""One requested decode of a source into immutable footage."""
key = models.CharField(primary_key=True, max_length=71)
source = models.ForeignKey(Source, on_delete=models.CASCADE, related_name="extractions")
settings = models.JSONField(default=dict)
state = models.CharField(max_length=16, default="queued")
progress = models.PositiveIntegerField(default=0)
error = models.TextField(blank=True)
footage = models.ForeignKey(
"Footage", null=True, blank=True, on_delete=models.SET_NULL,
related_name="extractions",
)
created = models.DateTimeField(auto_now_add=True)
updated = models.DateTimeField(auto_now=True)
class Footage(models.Model):
"""Tier 3: the frames and audio of one extraction, immutable.
`digest` is over the PROXY VIDEO's digest plus the audio's and the rate, so
two extractions of the same clip at the same settings are one footage and the
same analysis can be reused across both.
THE PROXY IS THE ANALYSIS SOURCE AND THE FRAMES ARE NOT. `video` is one
browser-safe H.264 file, and it is what the page seeks through to detect
landmarks. `frame_set` is a JPEG per frame at tracing size: reference stills
for the tracing editor, never the thing measured. The two are not
interchangeable, and which one carries the pixels an analysis was computed
from is the difference between a 6MB take and a 1.1GB one.
So the frame JPEGs are deliberately NOT in `digest`. They are a rendering of
this footage for a human to trace over; re-rendering them at another size does
not make it different footage, and putting them in the identity would throw
away every analysis when the tracing size changed.
`feature_absence` is the manifest annotation step 8 introduced: known
occlusion intervals, one-based and inclusive, expanded into presence tracks by
the loader. An input format, not a control UI.
"""
id = models.UUIDField(primary_key=True, default=uuid.uuid4, editable=False)
digest = models.CharField(max_length=64, unique=True)
label = models.CharField(max_length=200, blank=True)
source = models.CharField(max_length=200, blank=True)
fps = models.FloatField()
frames = models.PositiveIntegerField()
width = models.PositiveIntegerField()
height = models.PositiveIntegerField()
audio = models.ForeignKey(Blob, on_delete=models.PROTECT, related_name="audio_for")
video = models.ForeignKey(
Blob, null=True, blank=True, on_delete=models.PROTECT, related_name="video_for",
help_text="the browser-safe proxy, playable and seekable",
)
stream = models.ForeignKey(
Blob, null=True, blank=True, on_delete=models.PROTECT, related_name="stream_for",
help_text="the proxy's video as raw Annex-B H.264: what the page DECODES, "
"one access unit per frame; null on footage extracted before it",
)
feature_absence = models.JSONField(default=dict, blank=True)
created = models.DateTimeField(auto_now_add=True)
class Meta:
ordering = ["-created"]
def __str__(self):
return f"{self.label or self.source or self.id} ({self.frames}f @{self.fps})"
class FootageFrame(models.Model):
"""One tracing still. A row rather than an entry in a JSON list, because a
frame is a thing the server serves, and because a blob's references have to be
countable before anything can be collected.
A REFERENCE IMAGE, NOT A MEASUREMENT INPUT. See `Footage.video`."""
footage = models.ForeignKey(Footage, on_delete=models.CASCADE, related_name="frame_set")
index = models.PositiveIntegerField(help_text="0-based; source frame index + 1 is the JPEG's name")
blob = models.ForeignKey(Blob, on_delete=models.PROTECT, related_name="frame_for")
class Meta:
ordering = ["index"]
constraints = [
models.UniqueConstraint(fields=["footage", "index"], name="one_blob_per_frame"),
]
class Analysis(models.Model):
"""Tier 2: one detector, at one version, over one footage.
`key` is a content address over every input, and `descriptor` is the exact
canonical text that key is the sha256 of — sent by the client and stored, not
recomputed here. `clips/views.py` says why that is the honest arrangement: JS
prints an integral double as `1` and Python as `1.0`, so a scheme where both
sides re-render the numbers breaks on the first one of them.
`detector` and `version` are columns as well as descriptor fields so that the
question "which model produced this take" is answerable in the admin and in a
query, rather than only by parsing a hash's preimage.
"""
key = models.CharField(primary_key=True, max_length=71)
descriptor = models.TextField()
detector = models.CharField(max_length=64)
version = models.CharField(max_length=64)
footage = models.ForeignKey(
Footage, null=True, blank=True, on_delete=models.SET_NULL, related_name="analyses"
)
source_blocks = models.ManyToManyField(
"Block", blank=True, related_name="source_for",
help_text="pixel-dependent landmarks, detection mask and mouth crops",
)
created = models.DateTimeField(auto_now_add=True)
class Meta:
verbose_name_plural = "analyses"
def __str__(self):
return f"{self.detector} {self.version} → {self.key[7:19]}…"
class Block(models.Model):
"""Tier 2: one dense channel block.
Two hashes, and they are not the same hash. `key` is over the block's INPUTS,
which is what lets a client ask for the block its current settings want before
anything has computed it. `data.digest` is over the bytes. See clips/blobs.py.
"""
key = models.CharField(primary_key=True, max_length=71)
descriptor = models.TextField()
role = models.CharField(max_length=32)
analysis = models.ForeignKey(
Analysis, null=True, blank=True, on_delete=models.SET_NULL, related_name="blocks"
)
data = models.ForeignKey(Blob, on_delete=models.PROTECT, related_name="block_data_for")
state = models.ForeignKey(
Blob, null=True, blank=True, on_delete=models.PROTECT, related_name="block_state_for",
help_text="the per-track absence mask, when the take has one",
)
created = models.DateTimeField(auto_now_add=True)
def __str__(self):
return f"{self.role} {self.key[7:19]}…"
class Project(models.Model):
"""Tier 1: the document's root.
`schema_version` identifies the stored document format. `seq` counts writes
to this particular project; it is not a format version. Every write bumps
`seq`, and a client that sees `seq > local + 1` refetches.
ANYONE WITH THE LINK CAN VIEW; the owner and the editors can write. Every
project has an owner.
"""
id = models.UUIDField(primary_key=True, default=uuid.uuid4, editable=False)
owner = models.ForeignKey(
settings.AUTH_USER_MODEL, on_delete=models.CASCADE, related_name="projects",
)
editors = models.ManyToManyField(
settings.AUTH_USER_MODEL, blank=True, related_name="shared_projects",
)
name = models.CharField(max_length=200, default="untitled")
schema_version = models.PositiveIntegerField(default=8)
seq = models.PositiveBigIntegerField(default=0)
palette = models.CharField(max_length=64, default="arthur/default")
created = models.DateTimeField(auto_now_add=True)
updated = models.DateTimeField(auto_now=True)
class Meta:
ordering = ["-updated"]
def __str__(self):
return f"{self.name} ({self.id})"
def bump(self):
"""The next seq, taken with an UPDATE so that inside a transaction it is
also the write lock: two concurrent saves cannot both get the same one."""
Project.objects.filter(id=self.id).update(
seq=models.F("seq") + 1, updated=timezone.now()
)
self.refresh_from_db(fields=["seq", "updated"])
return self.seq
def can_edit(self, user):
return user.is_authenticated and (
user.id == self.owner_id or self.editors.filter(id=user.id).exists()
)
class Clip(models.Model):
"""Tier 1: the unit of work, and the thing leaf paths are scoped by.
`cid` is what appears in `clip/<cid>/...`, so it is the clip's identity as far
as addressing is concerned and it does not change.
"""
project = models.ForeignKey(Project, on_delete=models.CASCADE, related_name="clips")
cid = models.SlugField(max_length=64)
name = models.CharField(max_length=200, blank=True)
order = models.IntegerField(default=0)
blocks = models.ManyToManyField(
Block, blank=True, related_name="clips",
help_text="the tier-2 blocks this clip's channels name",
)
class Meta:
ordering = ["order", "cid"]
constraints = [
models.UniqueConstraint(fields=["project", "cid"], name="one_cid_per_project"),
]
def __str__(self):
return f"{self.cid} of {self.project.name}"
class Leaf(models.Model):
"""Tier 1: one independently addressed, independently versioned piece of the
document.
The value is transit-as-JSON in a JSONField, so the column holds JSON rather
than a string containing JSON: the admin can read a leaf, and the field-wise
merge of a channel leaf that docs/architecture.md describes as fifteen lines of
Python is possible over it. `version` is the entity tag a conditional write
compares — RFC 7232, not a bespoke invention.
"""
project = models.ForeignKey(Project, on_delete=models.CASCADE, related_name="leaves")
path = models.CharField(max_length=300)
value = models.JSONField()
version = models.PositiveBigIntegerField(default=1)
seq = models.PositiveBigIntegerField(
default=0, help_text="the project seq of the write that last changed it",
)
updated = models.DateTimeField(auto_now=True)
class Meta:
ordering = ["path"]
constraints = [
models.UniqueConstraint(fields=["project", "path"], name="one_leaf_per_path"),
]
@property
def etag(self):
return f'"{self.version}"'
def __str__(self):
return f"{self.path}@{self.version}"
class Revision(models.Model):
"""Tier 1: a snapshot of the authored layer, with a user and a summary — a
named snapshot, which is how a person marks a version now that every edit
saves itself.
ON AN EXPLICIT TRIGGER, not on every save. tl snapshots a small annotation
layer; arthur's tier 1 will contain cel polygons, so a snapshot per save bloats
the table — docs/architecture.md's "revisions need a coarser trigger". So this
is written by `POST /api/projects/<id>/revisions`, which is a "mark version"
button, and never by a save.
"""
project = models.ForeignKey(Project, on_delete=models.CASCADE, related_name="revisions")
seq = models.PositiveBigIntegerField()
author = models.CharField(max_length=200, blank=True)
summary = models.CharField(max_length=500, blank=True)
document = models.JSONField(help_text="every leaf of the project, by path")
blocks = models.JSONField(
default=dict, help_text="each clip's tier-2 block keys, by cid, so a restore can name them",
)
created = models.DateTimeField(auto_now_add=True)
class Meta:
ordering = ["-seq"]
def __str__(self):
return f"{self.project.name} r{self.seq}: {self.summary}"

7
clips/routing.py
View file

@ -1,7 +0,0 @@
from django.urls import path
from .consumers import ProjectConsumer
websocket_urlpatterns = [
path("ws/projects/<uuid:project_id>", ProjectConsumer.as_asgi()),
]

31
clips/templates/clips/index.html
View file

@ -1,31 +0,0 @@
{% load static %}<!doctype html>
{% comment %}
The host page, served by Django since port-plan step 9.
It carries no styles of its own any more. They are `static/arthur/app.css`, which
staticfiles serves from the same tree as the bundle — the page grew a five-pane
application chrome and "the styles" stopped being a thing you read in passing on
the way to the markup.
The bundle is unchanged: shadow-cljs writes it into `static/arthur/js` and
staticfiles serves it from there, so `manage.py runserver` and `shadow-cljs watch
app` are the whole dev loop with nothing copying files between them.
The CSRF token is rendered so that Django sets its cookie, which is what
`arthur.fx.http` reads to write the `X-CSRFToken` header. Saves are ordinary POSTs
and PUTs with ordinary CSRF protection — no endpoint in this app is exempt.
{% endcomment %}
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>arthur</title>
<link rel="stylesheet" href="{% static 'arthur/app.css' %}?v={{ css_version }}">
</head>
<body>
{% csrf_token %}
<div id="app"></div>
<script src="{% static 'mediapipe/vision_bundle.js' %}"></script>
<script src="{% static 'arthur/js/main.js' %}?v={{ js_version }}"></script>
</body>
</html>

0
clips/tests/__init__.py
View file

1218
clips/tests/test_api.py
View file

File diff suppressed because it is too large Load diff

46
clips/urls.py
View file

@ -1,46 +0,0 @@
"""The API, which is nine endpoints and no framework.
The shape is RFC 7232 over addressed resources: a leaf is a resource, its version
is an entity tag, and a conditional write answers 409. docs/architecture.md is
explicit that this part is not a bespoke invention — "optimistic concurrency
control over addressed resources with an entity tag" is what HTTP has done for
thirty years — so the plumbing here is deliberately boring.
WRITES ARE ON HTTP AND STAY THERE. When the websocket arrives it carries presence
and broadcasts, and not writes: auth, idempotency, status codes, retries and
conditional requests all come for free here, and a dropped socket cannot lose a
write.
"""
from django.urls import path
from . import views
urlpatterns = [
path("me", views.me),
path("login", views.login),
path("signup", views.signup),
path("logout", views.logout),
path("detector", views.detector),
path("sources", views.sources),
path("sounds", views.sounds),
path("sounds/<uuid:sound_id>", views.sound_detail),
path("images", views.images),
path("images/<uuid:image_id>", views.image_detail),
path("extractions", views.extractions),
path("extractions/<str:key>", views.extraction_detail),
path("footage", views.footage_list),
path("footage/<uuid:footage_id>", views.footage_detail),
path("projects", views.projects),
path("symbols", views.symbols),
path("projects/<uuid:project_id>", views.project_detail),
path("projects/<uuid:project_id>/leaves/<path:leaf_path>", views.leaf_detail),
path("projects/<uuid:project_id>/revisions", views.revisions),
path("projects/<uuid:project_id>/revisions/<int:revision_id>/restore", views.restore),
path("projects/<uuid:project_id>/editors", views.editors),
path("projects/<uuid:project_id>/editors/<str:username>", views.editors),
path("analyses", views.analyses),
path("analyses/<str:key>", views.analysis_detail),
path("blocks", views.blocks),
path("blocks/missing", views.blocks_missing),
path("blocks/<str:key>", views.block_detail),
]

1202
clips/views.py
View file

File diff suppressed because it is too large Load diff

59
do
View file

@ -1,59 +0,0 @@
#!/usr/bin/env python3
"""Run the local app and its frontend watcher with one command."""
import os
import signal
import socket
import subprocess
import sys
import time
from pathlib import Path
ROOT = Path(__file__).resolve().parent
def start():
with socket.socket() as probe:
if probe.connect_ex(("127.0.0.1", 8778)) == 0:
raise SystemExit("port 8778 is already in use")
commands = [
(["mise", "exec", "--", "python", "manage.py", "runserver", "8778"], ROOT),
(["mise", "exec", "--", "npx", "shadow-cljs", "watch", "app"], ROOT / "frontend"),
]
children = []
stopping = False
def stop(_signal, _frame):
nonlocal stopping
stopping = True
signal.signal(signal.SIGINT, stop)
signal.signal(signal.SIGTERM, stop)
try:
for command, directory in commands:
children.append(subprocess.Popen(command, cwd=directory, start_new_session=True))
print("arthur: http://localhost:8778 (Ctrl-C stops both processes)", flush=True)
while not stopping:
for child in children:
if child.poll() is not None:
raise SystemExit(f"app process exited with status {child.returncode}")
time.sleep(0.25)
finally:
for child in children:
if child.poll() is None:
os.killpg(child.pid, signal.SIGTERM)
for child in children:
try:
child.wait(timeout=5)
except subprocess.TimeoutExpired:
os.killpg(child.pid, signal.SIGKILL)
child.wait()
if __name__ == "__main__":
if sys.argv[1:] == ["start"]:
start()
else:
raise SystemExit("usage: ./do start")

937
docs/animation-model.md
View file

@ -1,937 +0,0 @@
# arthur — the animation model
The revised target for lanes, occurrences, source playback, shared editing, and
multi-view UX is [The Lane Model](lane-model.md). It supersedes conflicting
proposals below. Backward compatibility is not required; this document still
contains descriptions of earlier shapes and planned features.
The data that describes a moving picture: what the primitives are, how they
nest, how they change over time, and how rotoscoped and hand-authored work end
up being the same thing with one flag between them.
`docs/design.md` is the aesthetic argument. `docs/architecture.md` is where the
code goes. This is the type that both of them are about.
## What this replaces
`docs/design.md` has a table of five kinds of part — plate, feature, interior,
primitive, scalar — each with its own source, vocabulary and interpolation. That
table is a good description of **where data comes from** and a bad description of
**what data is**, and the current code follows it too literally: eyes, brows,
teeth and mouth each get their own build function, their own key shape and their
own path through the prototype's writer.
They are all one thing. A part is a **node** with **channels**, and the five
kinds collapse into differences of which channels exist and who filled them in.
## Prior art, and what each one gets right
| System | The idea worth taking |
| --- | --- |
| **Flash / SWF** | A **library of symbols** and a timeline of **instances** at depths. "Framed" content that simply exists on a frame, versus tweened content. `DefineMorphShape` requires matching vertex counts — the fixed-topology rule, arrived at from the other direction. |
| **Blender** | Animation is **addressed by path into the data** (`location[0]`), not stored as fields on the object. An Action is a bag of F-Curves. That decoupling is what makes the dope sheet, the graph editor and the NLA three views of one dataset. Also: parenting captures a `parent_inverse` so the child does not jump. |
| **After Effects** | Every leaf property is animatable, uniformly. Property groups form a tree. Pre-comps nest arbitrarily and a pre-comp is just a layer. |
| **Lottie** | The uniform property shape: `{a: 0, k: <value>}` or `{a: 1, k: [<keys>]}`. One representation for static and animated, which is exactly "framed or keyframed". |
| **Grease Pencil** | A 2D layer holds frames at frame numbers, and a frame **holds until the next one**. Hold is the default, not a special case. |
What none of them get right for this project: colour. All four store RGB on the
shape. `docs/design.md` forbids that, so colour is a palette index here and it is
a channel like any other.
## The one idea
**Analysis is a channel generator.** It does not produce a different kind of
data; it produces keys, densely, on the same channels a hand would fill in
sparsely. So:
```
footage ──▶ analysis ──▶ FREEZE ──▶ channels on nodes ──▶ evaluate ──▶ raster
▲
hand authoring ──┘
```
Freezing is not a conversion into a second format. There is one format, and
freezing fills it in. That is what makes "the only difference is a special flag"
literally true: the flag is provenance on a channel, and nothing in the renderer
reads it.
## Node
A node is an instance in the scene. The tree is stored **flat, with parent
pointers** — never as nested maps.
```clojure
{:id :mouth
:name "mouth"
:kind :poly ; :poly :disc :rect :group :bitmap :symbol
:parent :head ; nil at the root
:z "a3" ; fractional index, ordered among all siblings
:symbol nil ; or :sym/blink — see Symbols
:stencil :mouth-in ; colour-key clip; structural, not a channel
:span [0 240] ; in/out in the parent's frame space
:pinv [1 0 0 1 0 0] ; parent-inverse, captured when parented
:channels {...}}
```
Flat with pointers, for four reasons that all point the same way: any node is
addressable without a walk; reparenting is a one-field write rather than a
subtree move; an edit to a leaf does not change the identity of its ancestors, so
re-frame's structural sharing keeps ancestor subs from invalidating; and it is
what lets every node be its own sync leaf. Flash, Blender and AE all store it
this way.
`:span` is Lottie's `ip`/`op` and Flash's `PlaceObject`/`RemoveObject`: the range
over which the node exists at all. Distinct from a `[:vis]` channel, which
blinks an existing node on and off.
**Every node has the same two maps into its parent**, whatever kind it is:
- **space** — the matrix its transform channels compose to, times a `:pinv` if
it has been moved in from elsewhere;
- **time** — `local = rate · (parent − at)`, from `:time :at` and `:rate`,
identity when absent. `:span` and every key are in the node's **own** frames.
A move keeps a node's world maps and re-expresses them under its new parent:
the matrix becomes a `:pinv`, the time becomes a new `:at` and `:rate`, and its
channels, keys and span are not touched. Both maps are affine, so any depth of
nesting is one map and every move is one inverse. `node/time-of`,
`node/then-time` and `node/placed-span` are the time half; `clip/move-node` and
`clip/group` are the move.
### Subjects and tracked features
Scene nodes describe drawings, not tracking identity. A scene may also carry a
flat `:features` map. A feature ID stays stable for the whole clip, including
frames where that feature is occluded and later reappears:
```clojure
:subjects {:face-1 {:id :face-1}}
:features
{:eye-r {:id :eye-r :subject :face-1 :area :eye
:nodes [:eye-r :eye-r-in :iris-r :pupil-r] :params {}}
:eye-l {:id :eye-l :subject :face-1 :area :eye
:nodes [:eye-l :eye-l-in :iris-l :pupil-l] :params {}}
:mouth {:id :mouth :subject :face-1 :area :mouth
:nodes [:mouth :mouth-in] :params {}}
;; The teeth are their OWN feature and not three nodes of the mouth. A feature
;; carries the params of exactly one area, and the teeth have an `:area :teeth`
;; of their own — the otsu threshold, the tongue rejection, the radial contour's
;; vertex budget — which could not be reached if they were part of `:mouth`.
;; The coupling that made them look like the mouth's is real and is enforced
;; elsewhere: `:teeth` is STENCILLED by `:mouth-in`, and a node whose stencil drew
;; nothing is dropped, so an absent mouth takes the teeth with it without either
;; of them sharing an absence mask. An earlier draft of this block listed them
;; together; the code is right and this document was wrong.
:teeth {:id :teeth :subject :face-1 :area :teeth
:nodes [:teeth] :params {}}}
:groups
{:eyes-1 {:id :eyes-1 :kind :eye-pair :subject :face-1
:members [:eye-r :eye-l] :params {}}}
```
An eye pair is an explicit relationship between one or two eyes of the **same
subject**. It may have one member when only one eye has been identified; it does
not invent a second eye. Five subjects with nine identified eyes can have four
two-member pairs and one one-member pair. Each eye still has its own feature ID
and presence track. A group is a settings association, not a scene parent or a
tracking ID. Membership lives only on the group, avoiding a second pointer on
the feature that could disagree with it.
Each feature resolves settings from its area's definitions, then its group,
then its own `:params`. An eye can therefore inherit a pair setting or override
it without changing its partner. Removing it from a pair copies its effective
values into the feature first, so the result does not jump. Feature identity
and pair membership are clip-wide; a future parameter track can vary values
over time without splitting a feature at an observation gap.
Parameter definitions live in one registry: key, default, applicable area,
value constraints and affected areas. The registry supplies the take's defaults
today. The parameter UI and regeneration from edited values are later work.
Dense channel state records whether a measurement exists **for that feature on
that frame**. Occlusion means absent data on that frame, not a false `[:vis]`
value and not the end of the feature's identity. A full-face detection failure
makes all its features absent. A single occluded eye need only make that eye's
channels absent. Footage can carry explicit feature absence intervals in its
manifest, with one-based inclusive source frame numbers, for example
`"feature-absence": {"eye-r": [[10, 14]]}`. The loader expands these into
per-frame observation tracks before measurement. Unobserved landmarks may fill
rectangular numeric buffers, but they cannot contribute to an eye's contour,
blink or shared gaze. When one eye is absent, gaze uses the observed eye.
Until a detector supplies feature-level confidence, footage without annotations
uses the full-face detection mask as the fallback; it must not claim to detect
individual occlusions that it cannot see.
## Channel
Every animatable property is a channel, and channels are addressed **by path**:
```clojure
:channels
{[:xform :pos] {:animated? false :value [0.0 0.0]}
[:xform :rot] {:animated? false :value 0.0}
[:xform :scale] {:animated? false :value [1.0 1.0]}
[:xform :skew] {:animated? false :value [0.0 0.0]}
[:geom :pts] {:animated? true :interp :hold :dense {...} :generated {...}}
[:style :color] {:animated? false :value :skin-dark}
[:vis] {:animated? true :interp :hold :keys {0 true, 37 false}}}
```
A path is a **vector**, not a string — CLJS maps take vectors as keys natively,
so Blender's `data_path` idea arrives with no parsing. The set of valid paths for
a node follows from its `:kind`, and that is a spec, not a schema migration.
Three channel shapes, and the uniformity across them is the point:
```clojure
;; FRAMED — one static thing. No animation, no vertex correspondence to worry
;; about. A painted background cel is this.
{:animated? false :value v}
;; KEYED — sparse, authored, in the document. Undoable and syncable.
{:animated? true :interp :hold :keys {0 v, 4 v, 12 v}}
;; DENSE — generated, one value per frame, held in tier 2 as a typed array.
{:animated? true :interp :hold
:dense {:store "sha256:…" :offset 0 :stride 40 :frames 600}
:generated {...}}
```
`:interp` defaults to `:hold`, which `docs/design.md` requires of every cut part.
An authored keyed channel may also carry `:segments {8 :linear}`: the key at 8
tweens toward the next key, while other gaps use the channel default. The
transition belongs to the gap starting at a key, so a shape can cut into one
drawing and tween out of it. Per-key easing beyond hold and linear is deferred.
### Keys are a map by frame, not a list
Already argued in `docs/architecture.md` for merge reasons; here it also gives
"the most recent key at or before `f`" as a `rsubseq` on a sorted map instead of
a scan. **Store a plain map** in the document — transit and JSON both lose
sortedness — and build the sorted index in the resolver.
### The flag lives on the channel, not the node
```clojure
:generated {:by :roto/lips-outer
:analysis "sha256:…" ; which analysis artifact
:params {:verts 8 :contour-avg 1 :aperture-cut 0.004}}
```
Present means the UI offers a parameter panel and a re-freeze button. Absent
means the UI offers the keys directly. **The renderer never reads it.**
It belongs on the channel rather than the node because a node routinely wants
both at once: a mouth whose `[:geom :pts]` is rotoscoped and whose `[:xform :pos]`
is hand-animated to sit on a plate. Putting the flag on the node would forbid the
most useful thing in the model.
### Channels are layered
A channel is a base plus optional override layers, and a layer declares how it
combines:
```clojure
{:animated? true :interp :hold
:dense {...} :generated {...}
:over [{:id :nudge :support [88 98] :op :offset
:values {:animated? true :interp :linear :keys {88 [2 0], 96 [0 0]}}}
{:id :redraw :support [104 105] :op :replace
:values {:animated? false :value [[3 7] [4 7] …]}}]}
```
A LAYER'S VALUES ARE A CHANNEL, which is what keeps a constant adjustment, a
ramp and a return motion from being three mechanisms: a framed one says the same
thing on every frame it covers, a keyed one moves. They read through `value-at`
and `cursor` like any channel, one reading head each, so the specification and
the playback path share their blending and differ only in how they read — and a
layer's values may not carry layers of their own, which the stack already
orders.
`:support` is half-open and explicit, `[in out)`. Outside it a layer is inactive
and the base evaluates exactly as it did before, which is the difference between
a bounded correction and inserting boundary keys — the latter alters the
neighbouring segments. And a layer has NO TIME SPACE of its own: its support and
its values' keys are in the frames the base channel's keys are in, the node's
own. A correction on a lane is therefore in lane frames and reaches across the
drawings exposed under it; one on a single occurrence is in that occurrence's
frames and travels with it when the exposure moves. Ownership had already
answered the question, so there is no field to disagree with.
- **`:offset`** adds a delta to the base. "Nudge the mouth two pixels right for
ten frames" survives a re-freeze at different parameters, because it was never
a position — it was a correction.
- **`:replace`** wins outright. For the frame where detection simply failed.
This is what `docs/design.md` means by an override layer, and it is why
re-freezing is safe: the base is regenerated, the layers are untouched. It is
Blender's NLA blending and AE's effect stack at one property.
WHEN THE BASE OUTGROWS A CORRECTION it is a CONFLICT, which is neither a dropped
layer nor an applied one. Turning `:verts` gives the mouth a different number of
points, and an `:offset` is a row of components that has to match: so the
regeneration records `:conflict` on the layer, the layer stays in the document,
the picture is the base meanwhile, and `clip/conflicts` is the list a view
offers to resolve. Deliberately not `problems` — the document loads and saves
fine, it just contains a decision nobody has made yet. A later regeneration
that restores the shape clears the mark. Only `:offset` can conflict; `:replace`
states a whole value and has nothing to agree with.
A correction is NOT a hand placement. `regenerate-head` leaves the head's
authored channels alone once somebody has placed it by hand, and it compares the
channels WITHOUT their layers to decide: otherwise the first correction anyone
made would stop the head following re-measurement forever, which is the opposite
of what a layer is for.
Layers are what "set it by hand" means for anything measured, and the measured
channel does not need to know. A hand-set gaze is an `:over` on
`[:xform :pos]` of the iris; a hand-set mouth shape is an `:over` on
`[:geom :pts]`. Turning the gaze-step or gaze-dwell knob regenerates the base and
leaves the correction alone, which is the entire reason a correction is stored as
a layer rather than written into the track.
**A `:replace` layer overrides absence, an `:offset` layer does not.** Sampling a
channel is: read the base, then apply the layers — and the base coming back
`absent` does not short-circuit that. `:replace` is explicitly for the frame
where detection failed, so it has to be able to supply a value where there is
none; `:offset` is a delta, and there is nothing to nudge, so an offset over an
absent base stays absent. Implemented the obvious way — bail out on absence
before reaching the layers — the one case the feature exists for is the one case
it would not cover.
### One signal, two nodes
Gaze is deliberately **one measurement shared by both eyes**: at this size the
per-eye difference is noise, and independent noise reads as wall-eyed
immediately, which is the most expensive artefact on a face. But it is stored as
`[:xform :pos]` on `:iris-r` and on `:iris-l`, which are two channels on two
nodes with two different parents — so the invariant lives in `measure` and
nothing in the document enforces it.
That matters as soon as either one can be overridden by hand, because an `:over`
on one iris alone reproduces exactly the artefact the shared measurement exists
to prevent. Until drivers exist, **the override is on both or on neither**, and
that is a rule the UI has to keep rather than one the data can.
This is the case that will eventually justify **drivers** — one value, evaluated
once, feeding several channels — which is why gaze is named in Deferred as the
obvious first one. Nothing here forecloses it: a driver needs a place in the
document and a `:driven-by` on a channel, both of which are additive, and an
absent key means "not driven". So it stays deferred, and the shape does not have
to change to allow it.
## Transform: decomposed, never a matrix
```clojure
{:pos [x y] :rot θ :scale [sx sy] :skew [kx ky]}
```
Stored decomposed for two reasons. Each component has to be independently
keyframable, which is the entire point of channels. And interpolating matrix
entries is meaningless — a rotation tweened through its matrix shears on the way.
Composition, per node:
```
local = T(pos) · T(piv) · R(rot) · K(skew) · S(scale) · T(-piv)
world = world(parent) · pinv · local
```
`:pinv` is Blender's `parent_inverse`, captured at the moment of parenting so the
child does not jump when it acquires a parent. Small, and its absence is the kind
of thing that makes a parenting feature feel broken.
### A node has a `:pivot`, and a peg is still a peg
Rotation and scale happen about the node's **pivot**, `[:xform :pivot]`, a point
in its own coordinates:
```
local = T(pos) · T(piv) · R(rot) · K(skew) · S(scale) · T(-piv)
= T(pos + piv - M·piv) · M
```
Toon Boom gives every layer and every peg a pivot, Flash gives every instance a
transformation point, After Effects calls it the anchor point. All three store
it, and the reason is one sentence: **a turn has to be a turn on every frame**,
and the only way to keep a point still through an interpolated angle is for the
angle to be composed about that point.
This was deleted in schema 7 and restored in schema 8, and the argument for
deleting it was *not wrong*, which is why it is worth writing down. It was:
```
T(pos) · T(a) · R·K·S · T(-a) ≡ peg at pos+a carrying R·K·S, child at -a
```
to the last bit of the mantissa — `node-test` asserts it, still. `T(a)·M·T(-a)`
is `M` conjugated by a translation, which is "do `M` in a frame shifted by `a`",
and a **parent already is a shifted frame**. So an anchor was a peg written
inline, and a peg can be selected, keyed, shared between nodes and put above a
measured channel. Same expressive content, strictly more reach.
**What that identity does not say is what a node turns about when nobody has
made a peg.** It is an equivalence between a pivot and a peg *that already
exists*; it is silent on the default, and the default is what a person meets.
With no pivot in the composition, a turn about any point that is not the node's
own origin has to be paid for by writing `pos` as well — `gesture/about` solves
for it:
```
q = M⁻¹(c − t) the material point under c
p' = c − M'·q
```
and that solution is an **arc** in the angle while `pos` interpolates along the
**chord**:
| | pivot = origin | pivot ≠ origin |
| --- | --- | --- |
| one drag | right | right |
| between two keys | right | **wrong**, by the sagitta of the arc |
A 360° turn is where that is unmissable: 0° and 360° are the only two frames
where a wrong pivot cannot be seen at all, so the keys look right and every
frame between them is wrong.
**A drawing escaped it. A symbol instance could not.** `paint/centred` puts a
shape's origin on the middle of what it draws the moment it is drawn, so for a
drawing the pivot *is* the origin, `about` has nothing to do, and a keyed turn is
right between its keys. An instance's origin is its **symbol's**, and a symbol is
drawn on the stage, so its origin is the stage's top-left corner. Measured from
the document this was reported on: a symbol holding six drawn shapes had its
content centred at (99, 127), 161 px from its own origin, on a 320×200 stage. One
instance of it, keyed `rot` 0 → 60 and dragged round by hand, put the drawing at
(115, 116) on frame 0 and (241, 104) on frame 60 — both where they were put — and
at (−88, 121) on frame 30, a stage and a half from either. The answer on offer
was "make a peg first", for wanting to spin a drawing.
So the pivot is back, with the default and the escape hatch spelled out, because
a stored pivot without either is the field that was deleted:
| | what | where |
| --- | --- | --- |
| **the default, for a node nobody has pivoted** | the middle of what it draws — `pick/bounds-of`, the same call the selection box comes from, so the cross starts out on the middle of the box | `gesture/pivot` |
| **choosing it, invisibly** | the first turn or scale writes that middle down, in the same edit, with the `pos` that holds the picture still | `gesture/with-pivot` |
| **choosing it, by hand** | ⌃/⌘-drag the cross on the stage: the pivot goes under the pointer and nothing moves | `gesture/repivot`, `::ui/repivot` |
| **a placement** | `clip/place-symbol` stores the middle of what the symbol draws as the instance's pivot, so an instance turns about its drawing from the moment it is dropped | `clip/place-symbol` |
| **putting it back** | ⌖ beside the pivot row in the inspector: back to the middle of what the node draws *now*, moving nothing | `gesture/centred`, `::ui/centre-pivot` |
**A pivot is a choice, and does not follow the drawing.** Once it is the node's
own, the derived middle is never consulted for it again. This is the half the
old stored anchor got right and the derived pivot got wrong: adding a shape
inside a symbol must not re-aim every keyed spin of every instance of it, and a
pivot that tracked the content did exactly that, silently, with nothing changing
on screen at the moment it happened. The cross is visible and draggable and ⌖
puts it back, which is what the anchor was missing — it was never the storing
that was wrong.
**A peg is an ordinary `:group` parent, `nest/peg`, with `:pinv` captured so
nothing moves when it appears.** It is no longer the answer to "this turns about
the wrong point", and it is still the answer to three things a node's own pivot
is not:
| want | why the node's own pivot is not it | what the peg does |
| --- | --- | --- |
| a pivot **shared** between nodes — an arm and a forearm about one shoulder | two pivots that have to agree frame for frame are not one pivot | one transform, two children hanging off it |
| a **second** transform on one node — a drawing spinning about its middle while the limb swings about the shoulder | a node has one `rot` | stack them, as Harmony does |
| a hand transform over a **measured** one | `gesture/refusal` turns a drag on a measured channel away, because the next regenerate would discard it | the peg's channels are its own, so the hand transform composes outside the measurement, which stays regenerable |
The pivot of a measured node is *not* in that table: `[:xform :pivot]` is
authored on every node alike, never dense and never regenerated, so a traced
mouth can be told to turn about its own middle without a peg and with nothing a
regenerate will throw away. That is the row that used to be impossible — writing
an anchor under a measured `M` moved the thing it was meant to leave alone,
because the old composition was `T(pos)·M·T(-a)` and `pos` was the measurement's.
The conjugated form has no such problem: `T(a)·M·T(-a)` is the identity at `a`
whatever `M` is.
`demo/stage` places its seven faces on pegs, and that is now one way of writing
something a pivot says directly: the faces' `:scale` is **keyed** — they pulse —
and the source's middle has to stay on its authored centre throughout, which a
static `pos` cannot do since `T(pos)·S(k(f))` moves that point whenever `k`
changes. `T(center)·S(k(f))·T(-origin)` does, for every `k`, and so does one
instance with its pivot on the middle. The demo is left as it is, pegs and all:
it is a hand-authored scene that renders correctly and `instance-test` asserts
its structure, and a peg carrying a keyed scale is a perfectly good thing to
have written.
`gesture/about` survives for the one gesture whose pivot belongs to no node: a
**multi-selection** scaling about the middle of its shared box, where every
member has to move to keep the arrangement. Nobody keys that.
Schema 8 is the first version that **converts** rather than refusing. A schema-7
node has no pivot, an absent pivot reads as `[0 0]`, and `T(pos)·T(0)·M·T(-0)` is
`T(pos)·M` to the bit — so every stored document composes to exactly the matrices
it did, dense tier-2 transforms included, and the migration only restamps the
version. What a converted document does not get is a pivot anybody chose; its
nodes still turn about their origins until the first turn writes one or the cross
is dragged.
**The similarity fit already produces a decomposition.** `fitSimilarity` returns
`{s θ tx ty}`, which drops straight into `[:xform :scale]`, `[:xform :rot]` and
`[:xform :pos]` with no conversion. The analysis output and the animation model
meet without an adapter, which is a sign the decomposition is the right one.
## What space geometry is in
**`[:geom :pts]` is always in the node's own local space, and the transform
chain says what that means.** There is no global geometry space and no decision
to make about one.
| Node | Its local space | Why that one |
| --- | --- | --- |
| a rotoscoped feature | head-local, isotropic, unit = one image height | what the anchor fit already produces; the `xform` to raster is not applied and not stored |
| a painted cel | the stage, in pixels, grid-snapped | the artist is placing pixels, so the pixel grid is the thing being authored |
| a primitive under a feature | its parent's | the iris is positioned on the lid ring, not on the stage |
This looks like a small clarification and it removes a whole class of argument.
The prototype bakes the framing into the numbers: `toRasterRing` applies
`makeXform`, which centres on the face oval's bounding box and zooms until the
face is 80% of the raster height, so **every stored vertex carries a cropping
decision** that was made once, at analysis time, from one frame's landmarks.
Dropping that step is a deletion, not a feature, and after it the framing is
simply a transform on a node.
Grid snapping belongs to the cel and not to the roto, for the same reason: a cel
is authored on the grid and a traced contour is not. So it is a property of a
node's space rather than a rule about all geometry, and the tension between
"integer polygons" and "arbitrary placement" was never real.
Each dense block therefore carries its own **fixed-point scale** in its header,
because a block in image-height units and a block in stage pixels need different
ones to fill an `Int16` usefully.
### There is no camera node
A camera is a global transform over everything, and nothing here wants one.
Placing the face on the stage is a transform on a node, which already exists;
what is not on the stage hangs off the edges and the canvas clips it. Every fill
in `domain/raster` already clamps rather than assuming it is inside, so drawing
past the edge is not a feature to add.
Project dimensions are therefore **independent of the footage**. A 1440x1920
portrait clip composited onto a 320x200 stage is not a problem to solve — the
head is placed and scaled where it belongs and the rest of the frame is simply
not on stage. The full frame stays *available* for tracing without being
*visible*, and those are different requirements.
## Head motion: free or anchored to measured frames
`stabilize` produces `{s, θ, tx, ty}` per source frame. Its inverse is stored
densely on `:head`'s position, rotation and scale channels. The same measured
track serves every placement choice:
```clojure
;; no :anchors — free: read the measured transform at the current frame
;; one key — lock to a chosen measured frame throughout
:anchors {0 12}
;; several keys — cut to another measured head transform at frame 40
:anchors {0 12, 40 42}
```
The map is `local change frame -> measured source frame`. A single lock is a
one-key map. Position, rotation and scale read the same held source frame. The
frame set belongs to head placement, independently of plate drawings and stage
pose cuts. No measured block is copied into authored transform keys.
**Always measure, always store factored.** The fit is computed and the geometry
is stored head-local in every mode. Only the frame address used to read the
head's measured transform changes. Two things downstream require that split:
- *Smoothing.* "Smooth the transform, never the contour" only means anything
while the two are separate.
- *Key selection.* A velocity minimum is "articulation paused" in head-local
space and "the head happened to be still" in image space.
This is a document edit, not a reason to re-analyse. A registered tracing photo
will use its own source frame's stabilising transform followed by the same
selected head placement, so it aligns with the vectors drawn over it.
### Two nodes, because two different things want that transform
```
:face group — AUTHORED. where the face sits on the stage, and how big.
:head group — MEASURED. dense head motion read at the selected frame.
:mouth :mouth-in :teeth :lid-r :lid-l :brow-r :brow-l …
```
Changing anchor keys edits `:head` and never touches `:place`, so it cannot move
something that was placed by hand. A group node is free, and keeping the authored
and the measured transform apart is the whole reason the transform is decomposed
in the first place.
## Time maps — exposure, lead and symbol timing are one thing
Every node may map the frame it is evaluated at:
```clojure
:time {:mode :inherit} ; the default, and almost always right
:time {:mode :map :expose 2 :offset -1 :rate 1.0 :loop? false}
:time {:mode :map :source-fps 30 :sample-fps 12} ; root: lower picture cadence
```
Three features that look unrelated are this one mechanism:
- **exposure** is `⌊f/n⌋·n`,
- **picture fps** quantises source time to a chosen picture grid, then reads the
latest source pose at or before that time; source analysis and audio keep their
original cadence,
- **mouth lead** is `f + k`,
- **a symbol instance's timing** is `(f - at)·rate + in`, with optional looping.
Composed along the nesting chain, outermost first. Two rules follow, and they are
different rules:
- **Exposure inherits strictly.** `docs/design.md` is emphatic that everything
rides one grid, because a head cutting on odd frames against a mouth cutting on
even ones reads as two performances. The model permits a per-node grid; the
default must be `:inherit`, and setting it lower is a deliberate act the UI
should make feel like one.
- **Offset is per-node by design.** Mouth lead applies to performance nodes and
*not* to the plate, which is the whole point of it — so the offset genuinely
belongs at the node, not the clip.
## Symbols, and why a scene is one
A **symbol** is an ordered bag of nodes in its own frame space. (Earlier drafts
and code called this a *timeline*; that word now means only the UI pane that
shows one.)
```clojure
{:frames 91
:palette {...} ; see Palettes
:nodes {id -> node}}
```
That is the whole type, and **everything that holds nodes is one of these**:
- what a document opens on is a symbol, and **no symbol is reserved** — a new
document's is called `main` only because it has to be called something,
- anything placed inside another symbol is a symbol,
- a node with `:kind :instance` is an **instance** of one.
An earlier draft of this document had a scene and a `:kind :timeline` symbol as
two structures with the same fields and never said they were the same thing.
They are. Flash's `_root` is a MovieClip; After Effects' "a pre-comp is just a
layer" is already in the prior-art table above. Collapsing them is what makes
nesting arbitrary and free, rather than a feature to be added.
### Two axes of nesting, and they are different
This is the distinction the flat-storage rule is about, and conflating the two is
why "nested" and "flat with parent pointers" sound contradictory when they are
not:
| Axis | What nests | How it is stored |
| --- | --- | --- |
| **parent / child** | transform composition within one timeline | **flat, with parent pointers** — never nested maps |
| **instance** | a timeline inside another timeline | by reference into the library |
Each timeline is flat. Timelines nest. Every argument for flat storage —
addressability, one-field reparenting, structural sharing, per-node sync leaves —
is about the first axis and is untouched by the second.
The instance boundary is also **the only place the frame space changes.** Within
a timeline, `:time` is exposure and lead: a shift inside one space. At an
instance it is `(f - at)·rate + in`, into a different one. That is why `:rate` is
meaningless on an ordinary node and why sampling one must fail loudly rather than
be ignored.
### What is scoped to a timeline
Three fields on a node only have meaning relative to a timeline, and the answer
for all three is the same — **their own**:
- **`:z`** orders among siblings; a node cannot interleave with nodes inside a
nested instance. The instance occupies one position in its parent's order and
its contents sort beneath it, which the z path gives for free by being a
vector.
- **`:stencil`** names a node in the same timeline. A colour key does not
naturally respect a boundary — it is just pixels — so this is a rule rather
than a consequence, and it is Flash's rule for masks.
- **`:span`** is in the parent node's frame space.
### Instances
A node with `:kind :instance` and `:source {:symbol :sym/blink}` places one, and
its `:playback` says how time runs inside it — which drawing is used and how it
is played are separate facts, per [the lane model](lane-model.md). Its own channels
compose *over* the symbol's, so one definition is placed many times and tinted,
offset or retimed at each placement — that is how a three-frame blink is reused
at frames 40, 88 and 200 without copying it.
This is also where `docs/design.md`'s "closed vocabulary is right for the head"
lands: a plate library is a set of `:sym/head-*` timelines, and the strip chooses
which is instanced on which frame.
**Cursors and point buffers are per-instance, not per-node.** Two instances of
one symbol sit at different frames in their own space, so they cannot share a
reading head over the same channel. The resolver keys its caches by the instance
path, not by node id — which is a detail of `Making it fast` below, and the one
place symbol nesting is not free.
### Audio placements and controls
Sound is placed on a timeline as a separate `:audio` node. It uses the same
`:span`, `:time`, and channel representation as a drawn node. A `:linked-to` id
records which picture instance it was placed with; it does not force the two
spans or source in-points to match.
```clojure
{:id :voice-right :kind :audio :parent :root :z "a4"
:linked-to :right
:source {:footage "f8cace9e-..."}
:span [48 260]
:time {:mode :map :at 48 :in 0 :rate 1}
:channels {[:audio :gain]
{:animated? true :interp :linear
:keys {48 0.0, 60 1.0, 245 1.0, 259 0.0} :over []}}}
```
`[:audio :gain]`, `[:audio :pan]`, and `[:audio :rate]` are ordinary scalar
channels. They may be framed, keyed, or dense; numeric keyed channels can ramp
linearly. The time map sets the placement's base source rate, and
`[:audio :rate]` multiplies it. Audio is mixed from the referenced immutable
footage when the clip opens. The mix is derived output; the saved document holds
the nodes and channel keys, not another audio file. One audio element plays that
mix and remains the clock for both sound and picture.
This is also the boundary for a future control surface. A control has a stable
target, such as a feature's `:verts` setting or an audio node's
`[:audio :gain]` channel. The UI and a MIDI binding can address both through the
same control interface. Their update costs differ: gain can be keyed over time;
changing the number of lip vertices changes topology and must regenerate its
dense geometry. A topology setting cannot be treated as a per-frame gain curve.
## Evaluating a frame
```clojure
(defn eval-frame
"Scene at clip frame f -> draw ops in z order. Pure."
[scene f] ...)
```
1. Walk nodes in **topological order** by parent depth (cached; recompute only
when parentage changes).
2. Skip nodes outside `:span`.
3. Apply the node's time map to get its own local frame `fn`.
4. **Sample** each channel at `fn`: a map lookup for framed, a sorted-index
lookup for keyed, an array read for dense. Then apply `:over` layers.
5. Compose `world` from the parent's.
6. Transform geometry into raster space, writing into a **preallocated buffer**
owned by the node.
7. Emit `{:kind :poly :pts buf :n 20 :color idx :stencil id}`.
8. Sort by resolved `z`.
The op list is the boundary with stage 7 in `docs/architecture.md`: the
rasteriser takes ops and knows nothing about nodes, channels or time.
**A tracing layer is an op that never reaches the raster.** Footage or a still
to draw over is a symbol with `:type :trace` and a `:media`, placed by an ordinary
instance — so it is moved, scaled, trimmed, held and put in a lane like anything
else — and it resolves to one `:trace` op: `{:kind :trace :node :layer :media
:frame :size :m}`. The raster refuses that kind, the player hands it to a
`drawImage` on a separate canvas over the picture, and `clip/resolver` makes one
only when asked with `:tracing?`, which only the stage does. An export, a
symbol's centre and a thumbnail never ask, so a reference cannot reach the
picture by any path that forgets to filter it. See `docs/tracing-symbol-plan.md`.
A face's footage is one of these, placed as `:plate` under `:head` with the
anchor fit itself as its measured transform — the inverse of the head's, over
image height. Its world is `head · fit · 1/H`, so on a frame where the head and
the plate read the same measured frame the two cancel and the photo sits where
the face was filmed; on any other frame it rides the head. Registration is the
ordinary walk, not a matrix built beside it.
Which frame it shows is the placement's: `:time {:holds [...]}` holds it on
chosen frames, and a head with `:reads {:holds-of :plate}` jumps to the same
ones. Whether it is showing at all is the editor's, `[:ui :tracing]`.
### Making it fast in CLJS
Three things, and only these three matter:
- **Decomposed and persistent for storage; flat and mutable for evaluation.**
Composed transforms are 6-element `Float64Array`s, not maps. Every renderer
does this; the storage form and the evaluation form are allowed to differ.
- **A cursor per channel.** Playback is sequential, so "most recent key at or
before `f`" is an advance of a saved index, O(1) amortised. Binary search only
on a seek. This is the difference between a `rsubseq` allocation per channel per
frame and none.
- **Preallocated point buffers per node.** Fixed topology means the size is known
at freeze time, so the vertices — the overwhelming majority of the per-frame
bytes — are written into a buffer the node already owns. A frame still
allocates its op maps and the sorted op vector; that is a dozen small objects
against hundreds of points, and pooling them would buy nothing and cost the
ability to pass an op list around as plain data. At 30fps, per-vertex
allocation is the thing that will make this stutter.
Because the buffers are reused, **ops must be consumed before the next frame is
asked for.** That is the contract the rAF loop wants anyway: it reads, blits,
and dispatches nothing.
### What is in app-db, and what is not
| In app-db (tier 1) | In tier 2, behind a handle |
| --- | --- |
| nodes, parentage, z, spans, stencils | dense channel blocks |
| channel definitions, `:interp`, `:generated` | analysis artifacts |
| **framed** values, **keyed** keys, `:over` layers | preallocated eval buffers |
| library / symbol definitions | composed transform scratch |
The rule: **anything a human placed is in the document; anything a generator
produced is a handle.** Which is the same line `docs/architecture.md` draws for
sync and baking, arrived at again from the renderer's side.
## The current parts, in this model
Proof that it covers what exists, not just what is wanted:
| Now | Becomes |
| --- | --- |
| `mouth` outer ring, every frame | node `:mouth`, `[:geom :pts]` dense, `:generated {:by :roto/lips-outer}` |
| `mouth_in`, hidden below aperture | node `:mouth-in`, parent `:mouth`, `[:geom :pts]` dense + `[:vis]` dense |
| `teeth` from image content | node `:teeth`, stencil `:mouth-in`, `[:geom :pts]` dense, `:generated {:by :interior/teeth}` |
| lid rings | nodes `:lid-r/-l`, `[:geom :pts]` dense |
| lash line (`offsetRing`) | not data — a stage-6 parameter on the node, `{:grow px}` |
| iris disc | node `:iris-r`, `:kind :disc`, parent `:lid-r`, stencil `:sclera-r`, `[:xform :pos]` dense (quantised at freeze), radius framed |
| square pupil | node `:pupil-r`, `:kind :rect`, parent `:iris-r`, stencil `:iris-r` |
| brow ring + quantised raise | node `:brow-r`, `[:geom :pts]` dense (the traced ring with height removed), `[:xform :pos]` dense (the quantised raise). **The decomposition design.md insists on is two channels.** |
| head plate, kept frames | node `:head`, `:symbol` per instance, keys on `[:symbol]` at kept frames |
| `makeXform` face-oval crop | **gone.** Placement is `[:xform :*]` on the face's own `:place`; the stage clips |
| `stabilize` transforms | dense `[:xform :*]` on `:head` (the inverse fit) and on its `:plate` (the fit), read where `:reads` and `:time :holds` say |
| registered underlay | the face's `:plate`, an instance of the footage's tracing symbol under `:head`; a `:trace` op the raster never sees |
| painted background cel | node per layer, `[:geom :pts]` **framed**, `[:style :color]` framed |
| `mouth lead` | `:time {:offset k}` on performance nodes only |
| `exposure` | `:time {:expose n}` on the clip root, inherited |
| picture fps | resolver samples marked generated channels at the picture rate; authored keys keep their own time |
| hand correction | an `:over` layer, `:offset` or `:replace` |
The brow row is the one worth looking at twice. `docs/design.md` argues at length
that the traced ring already contains the height, so the quantised raise must be
measured *out* and put *back* or the brow moves twice. In this model that is not
an argument to remember — it is two channels on one node, and getting it wrong
would mean writing the height into both.
## Palettes
Three levels, and keeping them apart is what makes a palette swap a
**reinterpretation** rather than an edit:
| Level | Holds | Lives on |
| --- | --- | --- |
| **tone** | which mark this is — `:skin-dark` | `[:style :color]`, a channel on the node |
| **ramp** | what that tone looks like *here* | `:palette`, a channel on the timeline |
| **the ramps** | every named palette | the project |
A node names a **tone**, never a colour and never a ramp. Which ramp the tone is
read in is decided by the timeline the node is in. So the same drawing reads day
or night without one stored value changing — which is the entire payoff of
indexed colour, and is why `docs/design.md` forbids sampled RGB: once a shape
holds a measured colour there is nothing left to reinterpret.
Named palettes are **variants over one tone vocabulary**, not arbitrary colour
lists. `:day` and `:night` both define `:skin-dark`; that is what keeps a swap
total and keeps `docs/design.md`'s closed vocabulary closed. A tone the ramp in
scope does not define resolves to the loud magenta, like any other missing index.
### The scope rule
`:palette-track` points to an ordinary lane symbol. Its clips are instances of
restricted palette symbols: a palette symbol owns no nodes and points at exactly
one project palette.
```clojure
{:id :shot :frames 91 :palette :day :palette-track :shot-palettes :nodes {...}}
{:id :shot-palettes :type :palette-track :display :lane :frames 91
:nodes {:day-clip {:kind :instance :source {:symbol :day-palette} ...}
:dusk-clip {:kind :instance :source {:symbol :dusk-palette} ...}}}
{:id :day-palette :type :palette :palette-ref :day :frames 1 :nodes {}}
```
`:palette` is the symbol's authoring/preview palette. It seeds evaluation only
when that symbol is the viewed root; nested symbols do not replace the root's
choice merely because they were authored under another ramp. When absent, the
project default seeds evaluation.
Covered clips of the viewed root's palette track override that seed. An
uncovered lane interval is a genuine gap, restoring the authoring palette or
project default. Palette clips use the same trim, roll, slide, claim-time and
undo commands as visual clips; palette code does not duplicate those edits.
Thus palette-track coverage, authoring preview, and project fallback are
separate facts rather than three accidental meanings of one field. There is no
second keyed palette control on symbols or instances: time-varying palette
changes are authored only as clips in the palette lane.
### One index space, partitioned by palette
A raster is one `Uint8Array` and an index means one colour in it, so two ramps in
one frame cannot both own index 2. The resolution: **the output index space is
the concatenation of the named palettes**, and a tone resolves to
`palette-base + tone-index`.
Everything downstream is then unchanged — one buffer, one flat table for
`->rgba`, no per-frame palette construction, and an index does not change meaning
between frames, so bakes and thumbnails stay valid.
Two consequences worth stating rather than discovering:
- **The limit is real and reachable.** 256 indices over a nine-tone vocabulary is
twenty-eight palettes. Detect it and say so; do not let it arrive as wrapped
colour.
- **It makes the stencil sharper.** A stencil is a colour key, so two nodes
sharing a tone share a stencil — a genuine weakness of the technique.
Partitioning the index space by palette means two nodes in *different* palettes
no longer collide at all, and the resolved stencil picks up whichever index the
stencil node actually drew in.
### Where it is resolved
At the op boundary, and nowhere else. `[:style :color]` holds a keyword all the
way through evaluation; the walk carries the palette in scope the same way it
carries the parent transform and the local frame; the op carries a resolved
index. The rasteriser never sees a tone name and the node never sees an index.
This also means the palette is a **parameter of evaluation**, not a global. The
resolver takes it alongside the store.
## Format on disk and on the wire
Tier 1 is EDN/transit: the node tree, channel definitions, framed values, keys,
layers, library. Kilobytes, human-readable, diffable, and leaf-addressable for
sync.
Dense blocks are separate content-addressed binaries — `Int16Array` for
geometry, `Float32Array` for transforms — with a small header naming the channel
path, frame count, stride, and the **fixed-point scale** of the node-local space
the block is in. Geometry is stored in the node's own space, not in raster space;
see "What space geometry is in".
**Not Lottie internally**, despite the property shape being borrowed from it.
Lottie has no palette-indexed colour, its shapes are bezier with in/out tangents
where these are integer polygons, and its interpolation defaults are the opposite
of what is wanted. It is a fine thing to write out one day and a bad thing to
store.
Output is deliberately not specified here. The target is encoding video in the
browser, which touches the op list and nothing above it — a writer consumes
frames, and frames are what stage 7 already produces.
## Deferred
- **Per-key easing.** The structure allows it; nothing should use it until a
parented transform on a painted cel asks for it.
- **More than two channel layers.** The `:over` vector is already a list; a real
blend stack with weights is the NLA, and it is not needed to fix a bad frame.
- **Skew beyond the field.** `[:xform :skew]` is in the transform and in the
composition order from the start, because adding a component to a decomposition
later means migrating every stored transform.
- **Instance channel overrides on symbols.** Compose-over is specified; only
colour and transform need it at first.
- **Constraints and drivers.** Blender's other half. A gaze that aims at a null
object is the obvious first one, and it is a long way off. Until then the one
gaze shared by two iris nodes is a UI rule, not a stored relationship — see
"One signal, two nodes".

1059
docs/architecture.md
View file

File diff suppressed because it is too large Load diff

135
docs/clipboard-plan.md
View file

@ -1,135 +0,0 @@
# Selection and clipboard
This is the implementation contract for multi-selection, copy, cut, paste,
duplicate, and duplicate unique. It deliberately replaces any incidental older
behavior. The document model is the authority; the stage and timeline are two
views of the same editor state.
## One selection
`[:ui :selections]` is the ordered selection set. Its last member is the primary
selection in `[:ui :selection]`, used by the inspector and single-subject tools.
Every member is an occurrence address:
```clojure
[:node owner-symbol-id node-id row-path]
```
The row path distinguishes two occurrences of shared content. Commands that
write the document canonicalize those addresses before acting:
- invalid and non-node addresses are ignored;
- the same owned node, `[owner-symbol-id node-id]`, is acted on once;
- when one selected row path is below another selected row path, only the
ancestor is a clipboard root. Its ordinary parent-pointer subtree comes with
it, and an instance already displays the symbol it references, so also
materializing the visibly nested selection would duplicate it twice.
Plain click replaces the selection. Shift-click toggles membership, on both the
stage and timeline. A stage marquee replaces, or with Shift adds to, the same
set. Timeline rows, bars, and cel blocks render membership from that same set;
the primary member gets the inspector/focus treatment.
Creation targeting is derived rather than stored. The primary (last-selected)
occurrence is the preferred row; the playhead validates it and, when necessary,
walks outward to the nearest valid containing occurrence. A target is singular
even when selection is plural. See `docs/creating-in.md` for the resolver
contract.
## Clipboard value
The clipboard is editor state, not document state and not history. Copy records
a detached snapshot of each canonical root and its complete parent-pointer
subtree. It records source occurrence paths and authored node data, but normal
copy deliberately keeps referenced symbol identities. Therefore a pasted
instance is another use of the same symbol. The clipboard survives cutting its
nodes because it contains the node snapshot, not merely their addresses.
This first implementation is the application's clipboard, not the operating
system clipboard. It is consequently project-local and has no serialization or
cross-project identity collision policy hidden inside it.
## Paste
Paste resolves one destination from the primary selection and playhead. An
ordinary occurrence is usable only while the playhead maps through every
enclosing occurrence and lies within its extent. Otherwise resolution walks
outward, with the open symbol as the total fallback. A selected lane is an
insertion surface only while all occurrences enclosing its parent symbol are
valid. Multi-selection supplies one ordered payload, not several destinations;
only its primary member anchors this resolution.
The earliest finite start among the copied roots is aligned with the playhead
in the destination. All other root starts retain their offset from it. Roots
without a finite span remain timeless; paste does not invent a span for a shape
that was authored for the whole symbol. Parent/child timing, transforms,
channels, corrections, playback, stencil links, and relative root order are
otherwise copied exactly. Every node receives a new identity, and all internal
parent and stencil references are remapped.
In an ordinary composition overlap is valid. In a lane the pasted finite spans
claim their intervals using the lane's existing overwrite rule: covered cels
are removed, crossing cels are trimmed or split, and the pasted roots do not
overlap one another. Pasting a timeless root into a lane is refused. Validation
is all-or-nothing.
After paste, the new roots are the selection, in clipboard order, and the last
one is primary.
## Cut
Cut first takes exactly the same snapshot as copy, then deletes every canonical
root and its parent-pointer subtree. The clipboard write is editor state; the
whole document deletion is one history transaction. The selection is cleared.
Undo restores the deleted document nodes. It does not roll back the clipboard,
which matches ordinary editor behavior.
## Duplicate and Duplicate Unique
Duplicate does not read the insertion target or playhead. It is a local
operation beside the selected material:
- in an ordinary composition, copies keep the originals' parent, transform,
timing, and span, and are stacked immediately in front;
- for direct children of a lane, the selected temporal envelope is repeated
immediately after itself. Relative timing and gaps inside the selected set are
preserved, and later cels ripple forward by the envelope duration. This is
the lane's useful "duplicate forward" behavior; it is not a second command.
Selections in several owners are handled per owner in one command. Thus two
lane selections repeat in their respective lanes, while a selected composition
node duplicates in place, all as one history step.
Normal Duplicate preserves symbol references, just like normal copy/paste.
Duplicate Unique performs the same placement but deep-copies the complete graph
of every referenced symbol. One shared remap table is used for the whole batch,
so two duplicated instances that shared a nested part still share one new copy
with each other, while sharing nothing mutable with the originals. Immutable
media/store blocks may remain shared.
After either duplicate command, the new roots replace the selection.
## History, refusal, and stale state
Cut, paste, duplicate, and duplicate unique each call one domain command and
commit through one `edit/transaction`; each is exactly one undo/redo step no
matter how many nodes or symbols it touches. Copy and selection do not
touch history. A refusal changes no document leaves and creates no history step.
Undo/redo filters the complete selection set against the restored document and
repairs the primary selection. Clipboard payloads remain snapshots. Paste
validates the resolved destination and all remapped references at commit time,
so a stale selection or a newly impossible symbol cycle refuses rather than
partially editing.
## Required tests
Domain tests cover canonical ancestor/descendant selection, subtree ID remaps,
normal shared references, deep unique graph remaps, multi-root relative timing,
composition overlap, lane overwrite, lane forward duplication and ripple,
mixed-owner duplication, cycle refusal, stale targets, and all-or-nothing
failure. Event tests cover copy without history, atomic cut/paste/duplicates,
resulting multi-selection, target fallback at the playhead,
and one undo plus redo of each mutation. Browser tests cover mirrored stage and
timeline selection, Shift-toggle on labels/bars/cels, singular derived creation
targeting, and the keyboard commands.

234
docs/correction-authoring-plan.md
View file

@ -1,234 +0,0 @@
# Correction authoring implementation plan
Written against `2f1c9b9` (2026-09-30), following the lane handoff in
`7a54bfc`. Implemented on `codex/correction-authoring`; this now records the
scope and acceptance criteria of that implementation.
The cel-sheet targeting work described below was removed with the cel-sheet UI
on 2026-10-01; it remains here only as history of that implementation.
Read [lane-handoff.md](lane-handoff.md) and the correction section of
[lane-model.md](lane-model.md) first. Their ownership and document rules remain
the foundation. The choices below settle the first implementation's scope.
## Outcome
A person can select a lane or cel, specify a range, and apply Constant
adjustment, Ramp, or Return motion to rotation or position. The result is one
correction layer and one undo step. It works from either timing view. A lane
correction crosses drawing boundaries; a cel correction travels with its cel.
Regeneration preserves the hand work and presents incompatible layers for an
explicit decision. Frames outside the support evaluate exactly as before.
Finish this vertical slice before adding more property types or gestures.
Numeric range fields and an Apply button are sufficient for this pass. Dragging
a range or manipulating a peak on the stage can later issue the same command.
## 1. Fix sheet targeting first
In `ui/timeline.cljs`, `cel-sheet` currently drops each lane row's `:select`.
An occupied cell selects its cel; a gap only seeks, leaving the previous target
selected. Thus clicking lane B's gap after selecting lane A can send an insert
or overwrite to A.
Carry the row's complete selection address into its column and gap cells.
An occupied cell selects its cel; a gap selects its lane. Make the column header
select the lane too: this is the explicit way to author across drawings.
Preserve full paths, not just `(peek path)`, as view identity. Keep seek and
selection dispatch order deterministic.
Add a two-lane browser case: select A, click a gap in B, overwrite, assert that
only B changes, and undo once. Add a header-selection assertion. Retain the
existing occupied-cell hold test. Do not redesign sheet rendering in this step.
## 2. Make stack compatibility consistent
There is a concrete discrepancy at this HEAD:
- `channel/problems` uses `stack-conflict`, accounting for prior replacements.
- `channel/conflicts` and `flow/regenerate.cljs`'s `rebased` use
`conflict-with`, comparing an offset directly with the base.
A two-component base, a covering three-component replacement, then a
three-component offset is valid and evaluates correctly, but the latter paths
can report or mark that offset incompatible. Conversely a replacement can make
an offset incompatible even when it fits the original base.
Extract one ordered-stack compatibility operation and use it for validation,
conflict discovery, regeneration, and resolution. Keep `conflict-with` if useful
for the narrower question its name/docstring describe. Do not use it alone to
decide whether a stacked layer is applicable.
Compute compatibility against the values that can actually reach a layer over
its support. Partition at overlapping support boundaries if needed: two adjacent
replacements can jointly cover an offset even though neither covers it alone.
An empty replacement channel does not supply a value and must not erase the
possible input shape. Preserve the evaluator's absence behavior. Explicitly
marked conflicts are skipped, so later layers must be checked against the stack
that actually runs. During regeneration, recompute compatibility in order, using
each preceding layer's resulting active/conflicted state. Preserve IDs, values,
support, and order; update compatibility reasons without dropping hand work.
Use structural shape reasoning for dense data rather than requiring every block
to be sampled. If unknown shape or missing samples limit what can be proven,
retain the current absence contract and document that limit; do not claim an
unconditional proof of runtime safety from incomplete metadata.
Tests: covering replacement of a different shape; partial coverage; adjacent
covering replacements; empty replacement; inactive conflicted replacement;
regeneration changing base shape; and the same cases through cursor evaluation.
Assert that an accepted compatible stack is not listed as a conflict, and that
an incompatible regenerated layer remains persisted but is skipped.
## 3. Pure correction commands
Add `frontend/src/arthur/domain/correction.cljs`. It owns authoring and resolving
corrections; `channel.cljs` continues to own evaluation and compatibility.
Suggested API (names may follow repository conventions):
```clojure
(add clip sid node-id channel-path
{:id layer-id :support [a b] :motion :return
:start 0 :peak angle :peak-frame p})
(remove-layer clip sid node-id channel-path layer-id)
(retry-layer clip sid node-id channel-path layer-id)
```
Return `{:clip updated :selection node-id}` or `{:refused reason}`. IDs come
from the event caller (`random-uuid`), never from the pure command. Reject a nil
ID or one already used within that channel stack. Address layers by the full
symbol/node/channel/layer tuple; no global layer registry is needed.
Resolve the base via `node/channels`, which supplies defaults. A lane with no
explicit rotation channel already has a zero rotation; materialize that channel
with its new `:over`. Preserve every existing base field, generated provenance,
and previous layer. Never route this through a setter that bakes the correction
into base keys. Append to the ordered stack and validate the resulting document.
Do not run correction edits through `lane/finish`, whose extent policy belongs
to cel arrangement. Use `clip/problems` for the candidate document instead.
First authoring properties: `[:xform :rot]` (scalar radians) and `[:xform :pos]`
(two numeric components). First blend operation: `:offset`. UI labels must say
offset/delta, since a target offset of 20 degrees does not mean an absolute
rotation of 20 degrees. Keep existing `:replace` evaluation and loaded stacks;
there is no new replacement-authoring UI in this slice.
All command support endpoints are finite integer OWNER frames, `[a b)`, with
`a < b`. Do not ban negative owner frames merely because displayed shot frames
start at zero. Validate all supplied values for finite numbers and exact shape.
Refuse unknown targets, unsupported properties/motions, malformed ranges, and
incompatible stacks with useful messages. Refusal must not mutate store/history.
Motion construction uses existing channels only:
| Command | Values | Minimum samples |
| --- | --- | --- |
| Constant adjustment | `(ch/framed delta)` | 1 |
| Ramp | `(ch/keyed {a start, (dec b) end} :linear)` | 2 |
| Return motion | `(ch/keyed {a start, p peak, (dec b) start} :linear)` | 3 |
For Return, require integer `a < p < b-1`. Default the UI peak to
`a + floor((b-a-1)/2)`; on an even-length range the earlier middle sample wins.
Expose the peak frame so this is visible and adjustable. Never place an endpoint
at `b`: it is outside the selected samples. `[10 13)` with start 0 and peak 0.5
must yield offsets `0, 0.5, 0` at 10, 11, 12. Support controls the boundary;
there is no need to insert zero keys into the base before/after it.
## 4. Owner and range UI
Add a Corrections section to the right pane (`ui/params.cljs`), extracting a
`ui/corrections.cljs` component if that keeps the pane readable. Provide an
explicit target readout (symbol, lane or cel), Rotation/Position, motion choice,
From/Through fields, relevant value fields, peak frame for Return, and Apply.
Offer a selected cel's owning lane as an explicit target choice. Do not silently
promote a cel edit to a lane edit. Shared drawing content is outside this first
UI; it has different sharing consequences.
For this first pass the range fields explicitly read **owner frames**, with
inclusive From/Through converted to `[from, through+1)`. This is a deliberate
UI scope choice, not a claim that a displayed shot range and owner range are
interchangeable. It lets nested and retimed owners be addressed without an
unproven range conversion. Match the existing zero-based numbering and show the
owner beside the range. The handoff must record that displayed-range dragging
is still outstanding.
Use an explicit three-sample initial draft in owner coordinates; for a cel,
prefer its span start where it is integral. Show the range, allow adjustment,
and do not extend a cel or shot to make the correction visible. Reset the draft
when the target changes. Rotation is shown in degrees and converted to radians
at the event boundary, following the existing inspector convention. Position
uses x/y inputs in the owner's transform coordinates.
Draft inputs must not write document state, start history groups, or invoke the
existing inspector `number-input`'s hold/settle behavior. Apply dispatches one
event; success uses `edit/transaction` once and preserves the selected node's
full address. Do not use a layer ID as node selection. Validate again at Apply,
since the target/document may have changed since the draft was opened.
If adding a “use playhead” convenience, prove its mapping separately. The clock
of a cel's transform is its own node clock, not the drawing source clock selected
by `:playback`. `nest/inside` on the complete cel path enters the source and is
therefore the wrong shortcut. The existing `selection-frame` resolves only the
owning symbol; node/ancestor time conversion remains necessary. Floors, loops,
and nonintegral mappings must never silently snap an authored range. Omit this
convenience rather than expanding the first pass into a new timing system.
## 5. Conflict actions and regeneration proof
Show a document-wide list from `clip/conflicts` in the pane, including symbol,
node, property, layer ID, and reason. Keep it accessible even when a different
node is selected. Also list the selected target's layers in stack order with
support, motion values, and status; do not require a new persisted motion label.
Provide Remove correction and Retry compatibility. Remove is explicit and
undoable; Retry rechecks the complete candidate stack and clears a conflict only
when it is valid. If retry would invalidate a downstream offset, refuse and say
why. Similarly, removing a replacement that makes a later offset invalid must
refuse, rather than commit an invalid document or silently remove more layers.
A retry that changes nothing must not manufacture an undo step.
These are minimal resolution actions, not topology remapping. Automatic geometry
remapping, reordering layers, editing arbitrary stored vector values, and resolving
removed targets are separate work. Preserve all existing generated-data behavior.
Exercise the actual `flow/regenerate.cljs` entry points in integration tests:
author a correction, regenerate compatible base data, and confirm the correction
survives with its ID/support/values intact and affects the new base. Then change
topology on a geometry fixture and assert persisted actionable conflicts. The
geometry fixture can use an existing layer directly; geometry authoring is not
required to expose and resolve a conflict already present in a document.
## 6. Verification and completion
Use focused tests that establish observable promises:
- Domain: each motion's sample values; refusal on short ranges/nonfinite values;
default channel materialization; unchanged base and prior layers; duplicate ID.
- Evaluation: compare before/after on every frame outside support, including
neighboring interpolated frames. Cursor/spec agreement in nonmonotonic order.
- Ownership: a lane Return crosses a drawing boundary; a cel correction moves
with the cel and survives split/trim. Neither alters another use of its drawing.
- Sampling: picture-rate/pose selection changes the generated base frame while
the authored correction still reads owner time. Keep HEAD's regression tests.
- Events: one Apply is one undo step; undo/redo restores complete layer data;
refusal leaves clip/history unchanged; stale target refuses; selection survives.
- Persistence: leaf and Transit round trips retain layers, order, IDs, conflicts.
- Browser: both views can select a target and apply the same correction using
actual controls. Assert evaluated results and history, not only a layer count.
Include the two-lane gap-targeting case from step 1.
- Regeneration and conflict actions: use the real flow and test undoable removal,
valid retry, invalid retry, and removal that would break a downstream layer.
Run the suites documented in `lane-handoff.md`: CLJS tests, lane and take browser
flows, Django tests, and optimized frontend build. Restore the dev app bundle
after the release build. Note that `take.mjs` writes a local project. Report
actual results and any unrun checks; do not copy previous test counts as evidence.
Suggested commit sequence: sheet targeting; consistent stack compatibility;
pure correction commands; pane/events plus browser proof; updated handoff.
Keep each commit coherent and tested. No schema version bump should be needed:
the layers already have a persisted representation.
Update `lane-handoff.md` and `lane-model.md` with what shipped, the owner-frame
range UI limitation, conflict actions available, and verified test counts. Done
means a person can author and undo the correction, regenerate its base, and see
either their preserved edit or a useful conflict. A constructor without reachable
controls, or controls without that regeneration proof, does not finish this work.

155
docs/creating-in.md
View file

@ -1,155 +0,0 @@
# Creating in
Revised 2026-10-02.
Arthur's creation model is a layer list plus a playhead. The primary active row
is the preferred place for a new thing. The playhead decides whether that row is
present in the current occurrence and supplies the frame at which the thing is
created.
This is editor behavior, not document structure. A saved clip has symbols,
instances and spans; it does not have timeline rows or a creation target.
## Selection and the active row
The selection set names the objects affected by copy, delete and transform. Its
primary member is also the active row, analogous to the active layer in a paint
program. There is no second targeting gesture for a user to maintain.
- Clicking a row label, its timeline body, or the same occurrence on the stage
makes that occurrence primary.
- Shift/marquee selection may retain several objects, but only the primary row
anchors creation.
- Clicking blank stage or timeline space returns the active row to the symbol in
the current tab.
- Row disclosure is independent. Selecting or creating never implicitly expands
a row.
The selection does not change merely because the playhead moves. An off-frame
object remains available for copying, deletion and inspection.
## Resolving the active row at the playhead
Resolution starts with the structural destination implied by the primary row:
- A lane row means the lane symbol itself.
- An ordinary symbol-instance row means the symbol placed by that occurrence.
- A non-instance row means the symbol containing that node.
- No row means the symbol in the current tab.
For an ordinary symbol occurrence, the playhead must be within that occurrence's
extent in the current nested context. Extents are tested after walking all parent
time maps and use the half-open interval `[in, out)`. If the preferred occurrence
is not present, resolution walks outward to the nearest parent whose occurrence
is present. A containing lane is therefore the natural fallback from an inactive
cel. If no nested occurrence is present, the current tab is the destination.
The preferred row is retained during fallback. Scrubbing back into its extent
makes it the effective destination again.
A lane differs only in what its row means. It is an insertion surface across its
containing timeline and does not require an existing cel under the playhead. A
cel within the lane is still an ordinary symbol occurrence with an extent.
For `A -> B -> lane C -> cel D`:
- over D, with D primary, creation happens inside D;
- past D but while C is available, creation inserts a new cel in C;
- outside C's containing occurrence but inside B, creation happens inside B;
- outside every nested occurrence, creation happens in A, the current tab.
## What creation does
Once resolved, every creation command follows the destination kind.
### Lane destination
- A new or dropped symbol is instantiated directly in the lane at the playhead.
- Its interval claims that time. Existing cels under the interval are removed or
trimmed by the lane's ordinary claim-time command.
- Beginning a drawing creates a new one-frame drawing symbol in the lane and the
finished shape is a child of that drawing.
- Selecting an existing cel changes the destination from the lane to the symbol
placed by that cel; subsequent symbols and shapes become children there.
Double-clicking a lane creates an empty cel at the playhead; beginning a drawing
creates a drawing cel there. These are the same lane-creation operation with
different payloads. The pointer chooses the lane, never a second creation time.
The resulting cel is selected, so it immediately becomes the preferred target:
drawing again enters that cel's symbol instead of replacing it.
Thus no separate "new cel" versus "add inside" mode is needed. Selecting the
lane header says new cel; selecting a cel says add inside.
### Ordinary symbol destination
- A new symbol is instantiated as a child at the mapped playhead frame.
- A new shape is authored directly in the symbol at that frame.
- Stage coordinates are transformed through the occurrence into the destination
symbol's local coordinates.
### Explicit timeline drop
A timeline drop uses the row and frame under the pointer, not the stored active
row and playhead. Dropping onto a lane therefore always instantiates in that lane
and claims the pointer's interval. A stage drop uses the resolved active row and
the playhead.
Paste, imported symbols and converted footage obey the same resolver as direct
creation. They must not each reconstruct nesting or extent fallback separately.
## Multi-selection and paste
Multi-selection does not create multiple insertion targets. Copy and cut take
the canonical forest of selected roots as one ordered payload; the primary
(last-selected) occurrence alone supplies the preferred row for a later paste.
At paste time that row and the current playhead resolve one effective
destination, and every root in the payload is inserted there in one transaction.
The earliest finite root start is aligned to the destination frame. Other roots
keep their timing offsets, hierarchy, and clipboard order. In an ordinary symbol
the roots may overlap. In a lane every root must have a finite span and the
payload's root spans must not overlap one another; valid spans claim their times
and trim or remove existing cels as a batch. An invalid member refuses the whole
paste rather than inserting a partial payload.
After paste, all new roots form the selection and the final root is primary, so
it becomes the preferred row for the next creation. Pasting one payload into
several selected destinations is intentionally not implicit: that would be a
separate distribute command. Duplicate is also distinct from paste—it stays
beside each source in its original owner and does not consult the playhead or
creation target.
## Palette lanes
Palette lanes use the same row-and-playhead resolution. Their content filter and
transition command remain palette-specific: only palettes and palette
transitions can be inserted there. This is a type restriction, not a second
targeting model.
## Implementation boundary
One pure resolver returns the effective destination:
```clojure
{:kind :lane | :symbol
:sid destination-symbol
:path effective-occurrence-path
:frame destination-local-frame
:matrix destination-to-current-tab-transform}
```
Callers may add the unchanged document as `:clip` or rename `:frame` to `:at`,
but they must not reinterpret the active row. Polygon creation, symbol creation,
stage drops, paste, import and footage conversion all consume this answer.
Explicit timeline drops use the same structural row rule with the row and frame
under the pointer; unlike playhead resolution, an invalid pointer destination is
refused rather than allowed to fall outward.
The invariants are:
1. The primary row is the preferred structural destination.
2. The playhead validates occurrences and supplies creation time.
3. Inactive targets fall outward; selection does not follow them.
4. A lane row inserts a cel, while a cel row enters its symbol.
5. Stage creation uses the playhead; timeline drops use pointer time.

510
docs/frame-selection.md
View file

@ -1,510 +0,0 @@
# Frame selection
> Since `docs/tracing-symbol-plan.md`: `domain/trace` is gone. Trace keys are
> the face's `:plate` placement's `:time :holds`, the origin is the head's
> `:reads`, and the photo's registration is the plate's own measured channels.
> Where this document names `trace/measured-local` or `trace/prepare`, read the
> plate's or the head's measured channels and `node/hold`.
Two mechanisms. One vocabulary. An earlier draft of this document claimed they
were one component used twice — because `suggestPlateFrames` in the old
`js/pipeline.js` and the never-built "performance poses" of
[timing-handoff](timing-handoff.md) looked like the same function — and that claim
is wrong. They share how a selection is *read* and how the hand overrides one.
They do not share how frames get chosen, because the two are answering questions
of different shapes.
[Time selection](time.md) is the floor both stand on: an output frame reads the
latest native frame at or before its time, and nothing rewrites the dense
measurements. That is a *cadence*: an answer with no opinion about content. It
cannot know that the one frame where the eye is fully closed is worth more than
its neighbours, so at 12fps out of 30 it drops that frame two times in three.
This document is how the picture gets an opinion.
## The one idea
**A selection is a set of frames chosen out of a dense measurement, and read by
holding the latest one at or before now.**
The holding half already exists and is already shared: `pose/held-frame` is called
by `node/hold` (a placement's `:time :holds`) and by `pose/source-frame`, which is the two sites agreeing
about reading. The hand half is shared too — see *Three layers* below. Choosing is
what differs.
| | Plate drawings (tracing) | Performance poses |
| --- | --- | --- |
| the question | which frames does an artist have to draw a head on? | which frames does the picture change a shape on? |
| the cost being managed | a person drawing | a pose looking wrong |
| signal | the measured head's motion | — none; a stored cut |
| the baseline it improves on | drawing on 2s | the cadence, or the exposure grid |
| the shape of the answer | a non-uniform set out of dense | the same grid, nudged |
| lives on | the face's `:plate` `:time :holds` | the instance's `:playback :tracks` |
| hand edit today | `::project/toggle-hold` | `pose/put-cut` / `pose/remove-cut` |
| UI today | `params/layer-section` | **none** |
| proposes today | **nothing** | **nothing** |
## Why they are not one function
A plate selection has to be **non-uniform**, and that is the whole reason it
exists. A head still for sixty frames and then whipping across in ten wants two
drawings for the first stretch and eight for the second. Drawing on 2s gives
thirty-five drawings, most of them identical, and no amount of nudging a uniform
grid will produce the distribution that is wanted — the spacing itself is the
answer. That is what the prototype's walk was for, and it is why a cost knob
(`:tolerance`) belongs on this side: the artist is buying drawings.
A performance selection is **not choosing sparseness at all**. The output rate or
the exposure setting has already chosen it. The question left over is only *which*
native frame each already-decided slot reads, and the failure it fixes is narrow:
a slot landing one or two frames off the closure. Nudging the grid is the right
size of answer, and there is nothing for a tolerance to mean.
There is a second, harder reason, and it is the one that settles it:
**A selection cannot put a frame on screen that the output grid never samples.**
At 12fps out of 30, output frame 5 reads native 12 and output frame 6 reads native
15. A closure at native 13 is *between* them. Protecting frame 13 in a set of
kept frames makes it available and makes it the frame held across 13 and 14 in
native space — and at a 12fps output it still never appears, exactly as
[time.md](time.md) says: an event between output frames cannot create an extra
frame in a 12fps output. Only moving what output frame 6 reads can show it. So
the performance side has to act on the grid, not on a set beside it.
## Three layers, and the middle one is derived
The trap this is designed around is stated in
[timing-handoff](timing-handoff.md) and is worth repeating because it is the
only hard rule here:
> Store manual edits separately from generated proposals so changing the rate or
> tolerance retains hand decisions.
So a selection is:
```clojure
{:policy {:tolerance 0.02} ; what the proposer was asked for
:keep #{47} ; frames the hand insists on
:drop #{30}} ; frames the hand refuses
```
and the effective set is `(proposed ∪ keep) \ drop`, always containing frame 0.
`:keep` and `:drop` are the document. The proposal is not: it is recomputed from
`:policy` and the dense signal whenever either changes. **Re-suggesting at a new
tolerance must never cost somebody their pinned blink**, and that is the entire
reason the hand decisions are stored as their own two sets rather than as the
resulting frame list.
This layering is the part that really is shared. On the performance side there is
no `:policy` worth storing — the grid is the policy — but `:keep` and `:drop` mean
exactly what they mean on the plate side, and `select/effective` is the one
implementation for both. A preserve mark *is* a keep.
### Materialise the result, do not derive it on the render path
The effective set is written back to where each site already reads it —
the plate's `:time :holds`, or the pose track — so that every existing reader is untouched
and nothing on the per-frame path has to open a dense block. Proposing is a
command, not a subscription. `ch/value-at` allocates per call and says so; that
is fine for a button press over a few hundred frames and would not be fine at
30fps.
This means the stored frame list is redundant with `policy + keep + drop`. That
is deliberate and it is the cheap direction of the trade: a stale list is
recoverable by pressing Suggest again, and a dense read per node per frame is
not recoverable at all.
## The plate selection: a non-uniform chooser
`arthur.domain.select`, built — see *What is revertible* for the one part of it
that is not yet wanted. Pure, no store access, no clip access: the site hands it a
signal it has already read.
```clojure
(defn propose
"Frames worth keeping out of `n`, given `signal`."
[n signal {:keys [tolerance protect]}])
(defn effective
"`proposed` with the hand's decisions applied. Always contains 0."
[proposed keep drop])
```
`propose` is the prototype's walk, generalised off landmarks:
1. keep frame 0, make it the anchor;
2. settle the protected frames from `:protect` *before* walking;
3. for each later frame, keep it when it is protected, or when
`distance(anchor, f) > tolerance`. Either way it becomes the anchor.
`distance` is the max absolute difference over components, so a signal of
landmark pairs and a signal of one number both work without the caller saying
which it handed over. Every sample in a signal is the same width, and a signal of
two widths is refused: comparing the prefix two samples happen to share would let
a reader that drops a component read as no movement at all.
**A protected frame anchors the walk like any other kept frame**, which is why
protection is settled first and is not unioned onto the walk's result. The anchor
is what is on screen; once a protected frame is kept the viewer is looking at it,
so measuring the next frame's drift from a frame no longer displayed is wrong.
**An absent measurement is `nil`, and converting to that is the reader's job.** A
frame where the face was not found says nothing about the signal: it cannot move
the anchor and it is not a frame worth keeping. `trace/measured-local` already
returns nil there. A protected frame with no measurement is still kept — a plate
frame is a frame somebody draws on whether or not the detector found a face.
**A non-finite tolerance falls back to nought.** A cleared slider reads as NaN and
every comparison against NaN is false, which taken literally proposes frame 0
alone and collapses the whole take to one drawing. Nought proposes every frame
that changes, which is merely the baseline back again: wrong in a way somebody can
see and undo.
### The signal
The head's measured transform, applied to a fixed reference quad, giving
displacement in stage units — so a tolerance means "the head has moved this far"
and is a number a person can reason about. `trace/measured-local` already builds
that matrix per frame and is private; make it public rather than writing a second
one. Map it over the frames and transform four corners through `node/apply-pt!`.
The quad's size is a real parameter hiding in the word "fixed": it sets how much
rotation registers against translation. Give it a name and a comment rather than
an inline literal.
Do not reach for raw landmarks. The prototype used them because it had them lying
around; the transform is what the drawing actually follows, it is already on the
node, and it is three channels instead of a block.
### The upgrade path, and do not start here
The greedy walk is order-dependent and slightly suboptimal. The optimal version
is a dynamic program — choose `k` frames minimising held-reconstruction error,
which is textbook segmented least squares and is O(n²k), nothing at n≈300 — and
it keeps extrema *for free*, because an extremum is exactly where a zero-order
hold is most wrong.
Build the greedy one first anyway. It is proven, it shipped in the prototype, and
having two implementations to compare is how the DP gets tested. Swap it behind
`propose` afterwards, where the signature already permits it.
Most of `select_test` pins the greedy walk's exact output, deliberately, for that
comparison — so expect to rewrite those expectations when the DP lands, and keep
them as greedy-specific tests rather than deleting them. The assertion that is a
*spec* rather than a pinned vector, and should be written on this side before the
swap, is:
> reading the signal through the selection, held, never differs from the dense
> measurement by more than `tolerance`
That is what makes the tolerance number mean something to a person. It is true of
the greedy walk by construction and it is what the DP optimises, so it survives
the swap untouched.
## The performance selection: a preserve-snap on the grid
Not built. The rule is ten lines; getting the two halves of it into the same place
is the work. An earlier draft of this section said "about thirty lines" and that
was understated — see *Where it goes* below.
A grid slot already picks a native frame — `cadence/frame` for the output rate, or
the exposure fold for a deliberate hold at full rate. Write `d(k)` for the native
frame slot `k` defaults to. Slot `k` is the first slot to cover everything in
`(d(k-1), d(k)]`, and the frames strictly inside that interval are the ones the
grid shows to nobody. So:
> **Slot `k` reads the latest preserved frame in `(d(k-1), d(k)]`, and `d(k)` when
> there is none.**
That is the whole mechanism. It recovers a dropped frame out of the slot's own gap,
it can never read a frame another slot already showed, and it cannot reach past
`d(k)`.
**Snap backward only, never forward**, and note which direction that actually is,
because it is easy to get backwards. The closure at native 13 in a 12-from-30
output is recovered by slot **6** — whose default is 15 — reading 13. It is *not*
recovered by slot 5, whose default is 12, reaching forward to 13: slot 5's instant
is 5/12s = 0.4167s and native 13's is 13/30s = 0.4333s, so that would show the
closure 17ms before the mouth shut. `cadence/frame`'s contract is the latest native
frame at or before the slot's time and `cadence_test` asserts
`selected <= f*native/grid` over every grid and native pair, so reaching forward
breaks a tested invariant as well as the no-lead rule. Reading 13 at slot 6 shows
the closure two native frames late, which is the same lateness every hold already
has.
**The marks come from a cut that is already stored.** `flow/freeze` computes both
closures and keys them as `[:vis]`, with the thresholding and hysteresis already
decided:
- the mouth, from the aperture relative to the take's peak — `[:vis]` on
`:mouth-in`, provenance `:roto/mouth-aperture` (`freeze.cljs:473`);
- the eyes, from `condition/resolve-blink` with its cut, dwell and hold — `[:vis]`
keyed per eye part, provenance `:roto/blink` (`freeze.cljs:533`).
So "preserve the frames the cut says shut" reads what the document already holds.
No new signal, no new dense track, no threshold decided twice. Brows have no
closure and get no marks, which is correct: there is no extreme brow position
worth protecting.
**Precompute the mark set when the resolver is built**, the way `pose/prepare` and
`trace/prepare` already do (`symbol.cljs:420`, `:470`, `:536`). A `[:vis]` channel
is keys, not dense, so reading it per node per frame would be cheap — but the snap
also needs the marks sorted for a backward lookup, and building that per frame is
the one thing [animation-model.md](animation-model.md) and the render-path note
above both forbid.
### Where it goes, and why it is not a one-liner
The rule needs two things that currently live at opposite ends of the resolver:
- **the slot interval** `(d(k-1), d(k)]`, which needs the output slot index and the
fps ratio. Both exist at `clip.cljs:261`, the single place the grid becomes a
native frame: `(cadence/frame f (or (:grid-fps opts) (:fps clip)) (fps clip sid))`.
- **the marks**, which are per pose group, and so belong where group identity
exists: `symbol/base-channel-frame` (`symbol.cljs:399`), whose `:pose-sampled?`
branch already calls `pose/source-frame` with `(js/Math.floor lf)` as the default
pose. **That default is the snap's seat.** Replacing it leaves an explicit hand
cut winning over a snap, which is correct — manual precedence is absolute — and
costs no new plumbing on the pose side, because per-group choices are already
threaded and already prepared.
By the time control reaches `base-channel-frame` there is only `lf`, a native local
frame that placement and retime have already been through, so the slot interval
cannot be recovered there. It has to be threaded down from `clip.cljs:261`
alongside the frame. That is the actual work of this step: two namespaces' internal
signatures, not a drop-in.
**Do not take the shortcut of snapping at `clip.cljs:261` itself.** It is right
there, it needs no threading, and it is wrong: one native frame per output frame
means the *whole picture* reads 13 instead of 15, so the head goes two frames stale
for one output frame to fix the mouth. At 12fps that is a 67ms hitch on a moving
head, and it fights the trace selection, which has its own opinion about which head
frame to show. The snap is per group because the thing being recovered is one
group's closure.
### Why not an aperture signal
An earlier draft had this side read the group's aperture as a single-component
signal and hand it to `propose` with `:protect :extrema`. It cannot. The aperture
is the separation of landmarks 13 and 14, at positions 5 and 15 of the 20-slot
`LIPS-INNER` ring, and `freeze/rings->flat` subsamples the ring to the `verts`
budget: those two positions survive only when `verts` is a multiple of four. At
`verts` 6, 10, 14 and 18 — all legal, all even — they are not in the stored data
at all. `ring/subsample-slots` used to claim otherwise and has been corrected.
`flow/measure/mouth` does compute the exact scalar and `freeze` does throw it away
after thresholding, so storing it was an option. Reading the cut is strictly less
work and decides nothing twice.
## How the two interact
They are keyed in **the same frame space**: `clip/resolver` hands an instance's
`:playback :tracks` down into the child it places, and `symbol/base-channel-frame`
reads both the trace and the pose choices at `lf`, the node's local frame inside
that face. A trace frame and a pose cut are the same kind of number.
What differs is the owner, and that asymmetry is load-bearing:
- the **trace** is the face's, on its `:head` — every instance of that face shares
it, because it says how the drawings were made;
- the **pose tracks** are the instance's — two placements of one face can be
timed differently.
### The hazard, which is already written down
[animation-model.md](animation-model.md) states it for exposure and it is the
same hazard here:
> a head cutting on odd frames against a mouth cutting on even ones reads as two
> performances
**It is not a correctness problem.** The mouth is a child of `:head` and its
geometry is stored head-local, so a mouth from frame 17 composed onto a head held
at frame 12 is exactly lip-sync on a held drawing — the decomposition already
decoupled them and nothing is geometrically wrong. The problem is perceptual, and
perceptual problems want a constraint rather than a repair.
### Nest them, do not couple them
The two are not peers. One is coarse and expensive — the prototype's own comment
says it: *"The cost being managed is an artist drawing a head, which is why the
signal is head pose and not the mouth — the mouth is traced and free."* The other
is fine and cheap.
So the rule is a subset, in one direction only:
**Every kept plate frame is a preserved frame of the performance selection.**
When the drawing changes, the performance changes with it, so the two can never
cut against each other on neighbouring frames. The mouth stays free to change on
frames where the head does not, which is what shooting a held drawing with a live
mouth *is*.
This is why the preserve set is a set and not a closure predicate: plate frames
and shut-mouth frames go into the same pile, and the snap does not care which is
which. It needs no new mechanism on either side.
The reverse is a suggestion and never automatic. A mouth closure is a reasonable
place to want a new drawing, but proposing one spends somebody's afternoon. Offer
it; do not take it.
It also composes with `:origin` for free. A head on `:continuous` has opted out
of its own selection, so there are no plate frames to preserve and the constraint
is vacuous — which is correct, because a continuously moving head cannot cut
against anything.
### Between the kept frames is a third shared field
The head's `:reads` (once `:trace :origin`) is not a tracing setting. It is the answer to *what happens
between kept frames*, and the plate selection has to answer it:
- `:continuous` — ignore the selection for this purpose and read the frame you
are on,
- `:keys` — jump to each kept frame and hold it, a hold and not a tween,
- `:start` — hold the first forever.
The performance side does not need the field: a snap picks which frame a slot
reads and the grid does the holding, so `:keys` is the only behaviour there is.
Do not rename `:origin` to match anything: for a head it genuinely means where the
face's origin goes, and a saved field in a shipped UI is not worth churning for a
vocabulary tidy.
## The modes
One setting, two states, and **"manual" is not a third state.**
- **off** — the cadence alone, which is what ships today. The output frame
reads the latest native frame at or before it, and nothing has an opinion.
- **smart frame picking** — on the plate side the proposal is live, and `:policy`
holds the tolerance; on the performance side the snap is active.
Hand keeps and drops apply in **both** states, which is why they are not a mode:
turning smart picking off must not throw away the frames somebody pinned, and
pinning a frame with smart picking off is a perfectly reasonable thing to want.
Manual precedence is absolute — a drop beats a proposal, always.
The name on the toggle should be the same word in both sections. "Smart frame
picking" is fine. What it must not be is two different names for the one idea,
which is how these became two features the first time. That the two sections are
now backed by different code is an implementation fact and must not reach the UI.
## The UI
One component rendered twice, in `ui/params.cljs`:
```
smart frame picking [ off | on ]
tolerance [ ----•------- ] 0.02 (plate section only)
[ Suggest ]
frames 0 12 30 47* 61 (* = kept by hand, strikethrough = dropped)
```
The frame strip already exists in miniature — `params/trace-keys` draws the trace
keys as seek buttons (`params.cljs:386`). Lift it into a shared component and
give it three affordances: click to seek, a modifier to pin, a modifier to drop.
A pinned frame and a proposed frame must be visually distinct, because "will this
survive me moving the slider?" is the question the strip exists to answer.
Render it in two sections:
- **`tracing · <face>`** (`params.cljs:426`), beside the existing origin row,
with the tolerance slider and Suggest.
- **`performance · <group>`** — new, on an instance's inspector, one per pose
group the placed symbol has. No tolerance: the strip shows the preserved frames
and the grid slots that snapped to them.
## Order to build it
1. **`domain/select` with the greedy walk, and its tests.** **Done**, in commit
`02069e8`. Pure, no store, no clip. Note that the extrema part of it is not
wanted by anything below — see *What is revertible*.
2. **The preserve-snap**, pulled forward ahead of the plate side because it is the
half with no UI and nothing proposing today, and because it needs nothing from
the plate side except a fold that can land last. Three pieces: the mark set from
the stored `[:vis]` cuts, built in a prepare step beside `pose/prepare`; the
slot interval threaded from `clip.cljs:261`; the rule itself, seated in
`base-channel-frame`'s default pose. The test that matters is the same synthetic
case `select_test` uses — a one-frame closure at native 13 that a 12-from-30
grid drops — asserted end to end this time: the resolver shows a shut mouth on
exactly one output frame, and the head's frame does not move while it happens.
Write it first.
3. **The plate signal reader.** `trace/head-signal`: make `trace/measured-local`
public, map it over the frames, four corners through `node/apply-pt!`. Test
that it returns a vector of the right length, that a motionless take proposes
`[0]`, and that a stretch where the face was not found becomes `nil` rather
than a pose, a zero or a gap in the vector. Add the held-reconstruction
invariant from *The upgrade path* here, since this is the first place a real
signal exists to assert it over.
4. **Storage for the plate selection.** `:policy`/`:keep`/`:drop` beside
the plate's `:time :holds`. Extend `leaf/leaves` and the key whitelists in the same
commit — a field without a leaf saves silently and comes back missing, which
is the one bug persistence must not be able to have. Round-trip test.
5. **Re-suggest preserves hand decisions.** Propose at one tolerance, pin a frame,
drop a frame, propose at another, assert both survive. Settle here whether the
hand's `:keep` is also fed to `propose` as `:protect`: under the layering as
written it is not, so a pinned frame does not re-anchor the walk even though it
is on screen, which contradicts the anchor rule above. It is the one live
caller for `propose`'s `:protect` frames.
6. **Fold the plate frames into the mark set**, which is the nesting rule and is
one line once both sides exist.
7. **The shared UI component**, then its two mountings.
## What is revertible, and where it is
Commit `02069e8` contains one part that **nothing below asks for**: extrema
detection. Specifically `segments`, `turns`, the `:extrema` branch of `propose`,
the multi-component refusal that exists only to guard it, and three tests —
`a-one-frame-closure-survives-only-because-it-is-protected`,
`extrema-are-turns-worth-more-than-the-tolerance` and
`extrema-are-refused-on-a-multi-component-signal`.
It has no caller because plate selections take no extrema by design — displacement
is the whole story for a head — and the performance side reads a stored cut
instead of finding extrema in a signal. It is correct, tested and speculative.
Revert it if the shape above holds. Keep it if either of these turns out to be
wanted: a group whose extreme is not a closure and therefore has no `[:vis]` cut
to read (a mouth at its widest, a head at the top of a nod), or a take whose mouth
never shuts far enough to cross `aperture-cut`, where a peak-relative threshold
marks nothing and an extremum would still find the most closed frame. Neither is
asked for today. The DP in *The upgrade path* gets extrema for free regardless, so
reverting costs nothing that cannot be had again more cheaply.
## Traps
- **Do not let Suggest write `:keep`.** The proposal and the hand are different
layers; collapsing them is the bug this whole shape exists to avoid, and it
will look like it works right up until somebody moves the tolerance slider.
- **Do not snap a grid slot forward.** Every hold and every pick is "at or
before". Showing a closure before the mouth shut is a lead, which is a different
control for a different reason.
- **Do not quantise a plate selection onto the output grid.** A kept frame is a
native frame and lands where it lands. [time.md](time.md) is explicit that an
event between output frames appears on the next one; forcing kept frames onto
the grid would re-create the problem the selection exists to solve. The snap is
the opposite operation and is on the other side of the fence: it moves the grid's
pick, never the kept frame.
- **Do not read the aperture pair off a subsampled ring.** Positions 5 and 15 of
`LIPS-INNER` survive only at `verts` divisible by four.
- **Do not thin the plate selection with the mouth's.** Two scopes, two owners,
and the nesting runs one way only: plate frames preserve performance frames,
never the reverse.
- **Do not give the performance side a tolerance.** The grid has already chosen
the sparseness. A second knob there would be a control with nothing to control.
- **A skipped frame, a hidden feature and an absent measurement remain three
different facts.** A selection says nothing about visibility and nothing about
whether a face was found. Note that the preserve marks are *derived from* a
visibility cut, which makes this easy to blur: the cut says the interior is
hidden, the mark says the frame is worth landing on, and one is not the other.
## Not in scope
- Automatic *grouping* of related parts beyond the existing `:pose-group`.
Related parts must share one selection — a mouth outline, its interior, the
teeth and the generated visibility reading different frames is the bug that
grouping prevents — and `:pose-group` is where the grouping already lives.
- A per-instance request for a different tolerance than the symbol's. The
resolver threads no such option today and should not grow one until something
needs it.
- Variable frame rate. [time.md](time.md) assumes constant fps and so does this;
a VFR source needs presentation timestamps before any of this means anything.

247
docs/lane-handoff.md
View file

@ -1,247 +0,0 @@
# Lane and symbol-clip handoff
Status (2026-10-01): the timeline is the one timing interface. A lane is a
generic non-overlapping row of symbol clips; it is not a special drawing type.
Dropping a library symbol makes a naturally playing clip, while creating a new
empty symbol makes a one-frame held clip at the playhead. With no destination
lane, either operation creates one. Existing legacy root symbol rows can be
dragged into a lane. Blocks move by mouse; edge drags claim time by trimming
neighbors; Shift-edge drags ripple every later clip; and the center of a shared
cut composes the two edge edits into a rolling edit. Linked audio follows picture
moves while its edges remain independently trimmable.
The commits beginning at `3d3c1bb` are the argument for the model and are worth
reading before touching what they did — they are the design record, more than
this file is.
3d3c1bb An occurrence is a node, with a clock of its own
9446829 Reuse, duplicate and make unique: deciding what is shared
26517af A position is an argument, not another command
94c0a21 A correction is a layer, and a layer's values are a channel
72b57e3 Regenerate the base, keep the hand work, and say when you cannot
76106d3 The shot is as long as somebody said it was
598c186 One word for one thing: it is a cel
## Read first, in this order
1. [The Lane Model](lane-model.md) — the design, and the status note under
*Proof obligations* says what is built. It supersedes `animation-model.md`,
`timing-model.md` and `architecture.md` wherever they overlap.
2. `frontend/src/arthur/domain/lane.cljs` — every command, and the reasoning in
its docstrings.
3. `frontend/test/arthur/domain/lane_test.cljs` — what the model is asserted to
do. It is the fastest way to see the shapes.
4. `frontend/src/arthur/domain/channel.cljs`, the correction-layer section.
## Vocabulary — one word for one thing
Renamed in `598c186`, after four words had accumulated for one object. Use these
and do not reintroduce the others.
| word | means |
| --- | --- |
| instance | the `:kind`. The general thing, anywhere in a document |
| clip | an instance in a lane. It can hold one source frame or play a symbol naturally |
| cel | specifically a one-frame source held over a clip's duration; the empty-symbol/drawing creation policy |
| lane | a group with `:layout :sequence` |
| drawing | content authored into a symbol; not a different timeline node type |
| placement | ONLY where a node sits: `nest/placement`, and the transform that puts a face on the stage. Never the node itself |
`occurrence` and `exposure` are not words for a cel. **`exposure` means something
else and still does**: `:time :expose` is how many frames each step of a subtree
lasts, which is what shooting on twos is — `node/expose`, `clock/exposed-frame`,
`subs/render ::exposure`. Keeping these apart is why the block is called a cel.
`:layout :sequence` stays as the field, and is the one place two words are kept
on purpose: the layout names the RULE — children follow one another and may not
overlap — and a group carrying it is called a lane. `node/lane?` is where they
meet.
## Decisions already made — do not re-litigate
These were each argued out and are load-bearing. Changing one is a design
decision, not a cleanup.
- **The shot length is authored.** `:frames` is the symbol's window; the
occupied extent of its lanes is a different fact derived from the cels. A
command grows the window only when the caller passes `:extent :grow-symbol`,
and never shrinks it. Blanking the end of a shot leaves empty frames at the
end, because deriving the window from the extent would make deleting the last
drawing silently shorten the film. `lane/finish`.
- **Placing ripples; overwrite is `blank` then non-rippling placement.**
`lane/overwrite-drawing` composes those pieces as one transaction. Insertion
retains its ripple rule; overwrite does not move any surviving cel.
- **A position inside a cel refuses and names `split`.** One command must not
quietly perform two. The UI offers the retry.
- **A correction has no time space of its own.** Its `:support` and its values'
keys are in the frames the base channel's keys are in — the node's. A
correction on a lane is in lane frames and reaches across the drawings under
it; one on a cel travels with that cel. Ownership already answered it.
- **A layer's values are a channel.** Constant, ramp and return motion are one
mechanism. Do not add a second way to say what a value is over time.
- **A conflict is not a `problem`.** A document whose topology outgrew a
correction loads, evaluates and saves; `clip/conflicts` lists the decisions
waiting for a person. `problems` means the document will not load.
- **Refuse rather than guess.** Every command returns `{:clip :selection}` or
`{:refused why}`, never a half-applied edit. Where the model needs a choice
nobody has made, refusing and saying why is the behaviour, not a placeholder.
- **A clip is not a row.** Rows, expansion and selection are editor state. The
document has never known about rows and must not learn — which is what let
the row model change three times in one sitting (blocks, then a
selected-clip portal, then sound lanes under the audio heading) without
touching a single document.
- **A lane is generic.** Drawing creation, library placement, and adopting an
existing root instance all produce the same child instance shape. The only
difference is playback policy: a new empty drawing holds source frame zero;
a dropped library symbol plays at speed one.
- **Lanes are explicit.** A symbol may contain ordinary overlapping children
without a lane. Selecting a lane row opts creation into its claim-time
behavior; selecting a cel enters that cel's source symbol instead.
- **Placement claims time.** Lanes never store overlaps. A new or extended clip
trims, removes, or splits whatever previously owned the claimed interval.
Real compositing overlap uses another lane, where ordering remains explicit.
## The timeline opens the whole document
Expanding a lane opens exactly one clip — the selected one — and that portal
opens the lanes and nodes of the symbol it places, recursively, mapped into
the open symbol's ruler. The portal follows the LINEAGE of the selection, so
working on something nested keeps the rows that revealed it open. A held clip
opens too, with its rows marked `:unmapped?`: shown across the hold, with no
keys and no draggable edges, because a frozen clock gives its frames no place
on this ruler. `docs/lane-nesting-notes.md` has the reasoning and what is
still missing.
Double-clicking a clip opens the symbol it places as a tab, the same as
double-clicking that symbol in the pool. Shift while dragging a clip body
turns the temporal move into a structural one — see the nesting notes for why
that is mostly refused today.
## Current timeline interaction
- Creation follows the primary active row and the playhead; the complete rule is
in `docs/creating-in.md`. A lane row creates a new cel and claims its interval,
while selecting a cel creates inside the symbol that cel places.
- Drawing with a lane row active creates a new one-frame drawing cel. Dropping a
library symbol there creates a natural-duration playing cel. Explicit timeline
drops use the row and frame under the pointer.
- Dragging a clip body moves it. A linked audio node follows a picture move;
moving or trimming the audio itself remains independent.
- Dragging a right edge changes its endpoint. Growth consumes adjacent spans
instead of overlapping them. Shift-drag inserts or removes lane time by moving
every later clip by the same delta.
- At a shared boundary, the left and right hit zones trim one side. The center
is a rolling edit: right-edge resize followed by left-edge resize at one frame.
- Split, trim-in, and trim-out are direct buttons and are disabled without an
editable selected span. Movement is a mouse gesture, not a toolbar command.
## Next steps, in order
The implemented correction slice and its remaining UI limits are recorded in
[Correction authoring](correction-authoring-plan.md).
1. **Slip source and retime.** Both have real design questions open and the doc
says to refuse rather than approximate: retime needs a defined warp and
interpolation behaviour, and is not moving keys whose numbers happen to fall
inside a selection.
2. **Deleting reused content.** Reference discovery exists (`node/sources`,
`clip/places`, `clip/contains-symbol?`); the policy does not.
3. **Displayed-range correction gestures.** The first correction panel asks for
explicit owner frames. Dragging a range in a retimed/nested view still needs
a proved mapping; do not make it snap through floors or loops.
4. **Collaboration.** `lane-model.md` is explicit that one leaf per channel does
NOT solve two people editing different keys of the same channel. No conflict
policy exists for that.
## Mechanisms to reuse — these keep paying out
- **`:span` is in the node's OWN frames** and `:time` says where they land in
the lane. Moving an edge of a cel is therefore one write to `:span`, with
`:time` and `:playback` untouched. This is why split costs nothing, why the
two halves of a split go on meaning what the one cel meant, why trimming the
front of a playing insert starts it later into its animation instead of
restarting it, and why extending a hold leaves lane keys alone. `lane/local`
and `lane/edged` are the whole geometry; trim, split and blank are all it.
- **`lane/finish`** is the one commit path: it validates, applies the shot-length
policy, and returns the refusal. New commands go through it.
- **`:required-frames` plus the retry event** is the pattern for "this needs a
decision you have not made": the domain reports what it would need, the UI
offers one button. `events/ui/lane-retry`.
- **`lane/lane-frame`** converts a symbol frame to a lane frame, or returns nil
through a stepped or looping lane where there is no single answer. Nil refuses;
it never snaps.
- **`channel/conflict-with`** is the rule for whether one offset fits a base,
used by `conflicts` and regeneration. Validation additionally follows prior
replacement layers, so it cannot approve a stack that throws when read.
- **Generated sampling applies to the base, not the hand correction.** Picture
rate and pose selection may choose an earlier generated frame; correction
support and values still read the node's current authored frame.
- **Correction commands live in `domain/correction.cljs`.** IDs come from the
event caller; the pure command materializes default transform channels,
appends one layer, and validates the complete document. The inspector authors
rotation and position offsets in explicit owner frames. One Apply is one undo
step. `channel/reconcile` is the shared ordered-stack compatibility rule used
by validation, conflict reporting, and regeneration.
- **Two test patterns worth copying.** `the-cursor-agrees-with-the-specification-in-any-frame-order`
holds the optimized cursor to `value-at` in forward, backward and random order
— add a case to it for any new channel shape. And `drawn` in `lane_test`
samples every frame before and after an edit, which is how split and trim are
proved to change nothing: state a claim as "the same picture" rather than as
numbers computed by hand.
## Known gaps and traps
- **Audio is a clip in a lane too, and a lane holds one kind.** A sound placed
from the pool lands in a lane and is moved and trimmed by the same commands
as picture. The capability the earlier note asked for is the homogeneity
rule rather than a field: `symbol/lane-problems` refuses a lane holding both
kinds, and `lane/place-symbol` and `lane/adopt` refuse BEFORE claiming time,
because placement claims time and would otherwise have deleted the sound to
make room for the picture and left a valid document behind. Audio nested
inside a placed symbol — a take's own sound — is still shown flattened by
`nest/audio-tracks`; what is in a lane of the open symbol is drawn as a lane
and not flattened twice.
- **`:z` is required on cels and means nothing there.** A lane never has two
cels on one frame, so draw order between them cannot matter. `node/problems`
requires `:z` on every node uniformly, which is its own kind of simplicity —
but the field is noise on a cel.
- **`channel/offset-onto` throws** on a shape mismatch that no regeneration has
recorded as a conflict. That is deliberate — a correction that silently does
not take is the failure the design exists to prevent, and `channel/problems`
catches the authored case — but it is a throw in the read path, so any new
producer of layers must not create a mismatched one.
- **`docs/timing-handoff.md` is a separate, unreconciled thread.** Performance-
pose selection and plate drawings/tracing, instance-specific picture-rate
requests, `pose/put-cut` addressing only `:main`. It predates the lane model
and nobody has squared the two.
- **Slip and retime are still absent.** The timeline action strip now applies
split/trim uniformly to a selected root or lane clip, but source-time slip and
retime still need their own proved semantics before they become controls.
- **`shadow-cljs release app` clobbers the dev bundle.** Both builds write
`../static/arthur/js`, which Django serves, and the optimized build does not
export the `arthur` global — so after a release the browser tests fail with
`ReferenceError: arthur is not defined`. Run `npx shadow-cljs compile app` to
restore it. A running `watch app` does not notice; it rebuilds on the next
source change.
## Running it
From `frontend/`:
npx shadow-cljs compile test && node out/node-tests.js # 469 tests, 9,592 assertions
npx shadow-cljs compile app # the bundle Django serves
npx shadow-cljs release app # then `compile app` again — see above
The browser tests need the Django dev server up (`mise exec -- python manage.py
runserver 8778` from the repo root) and a compiled dev bundle:
node --experimental-websocket test/browser/lane.mjs # generic symbol-lane flow
CHROME=/usr/bin/chromium node --experimental-websocket test/browser/take.mjs
`take.mjs` defaults to a macOS Chrome path, hence `CHROME=`. It writes a real
project to the local server by design; `lane.mjs` never writes to the server.
From the repo root: `mise exec -- python manage.py test clips` — 56 tests.
Documents are schema 3. A version 2 document is not read and nothing converts
one; there is no backward compatibility to preserve anywhere in this work.

71
docs/lane-is-a-view-notes.md
View file

@ -1,71 +0,0 @@
# Implementation notes — "A lane is a view"
Running log for `docs/lane-is-a-view-plan.md`. `[ ]` not started, `[~]` in
progress, `[x]` done with `npm test` green.
Baseline at `fb38990`: 475 tests, 9621 assertions, 0 failures.
## Order of work
The plan's seven steps, re-grouped — see *Deviation from the plan's order* below.
- [x] A. Domain: `symbol/children`, `symbol/lane?`, `symbol/overlaps`,
`lane.cljs` → `span.cljs`, the overlap check in `span/finish`
(plan steps 3, 4, and the domain half of 5)
- [x] B. Events: re-base callers and remove lane-node-specific commands; keep
explicit `::new-lane` and cross-lane adoption (plan step 5)
- [x] C. UI: row per symbol, explicit lane creation, and drag handling
(plan steps 1, 2, and the UI half of 5)
- [x] D. Audio: delete `holds-other?`, the mixed-lane refusal, the `in-lane`
filter in `sound-rows` (plan step 6)
- [x] E. Tests: `domain/sequence_test`, `events/lane_test`, `browser/lane.mjs`
- [x] F. Shift-to-reparent still works, untouched (plan step 7)
Not in this pass — see *Left for a second pass*: tearing out the held cel, the
instance-playback control, drawing a loop's repeats, the audio period guard.
## Deviation from the plan's order
The plan's steps 1 and 2 are display work that keys off "a symbol's children",
and step 3 is what MAKES the cels a symbol's children. Until then a cel's
`:parent` is the lane node, so there is nothing for the display to read: step 2
cannot draw "a symbol's children as blocks" while the children belong to a
group. So the data model moves first (A) and the display follows (C). The
content of each step is unchanged; only the order is.
The one thing this gives up is the plan's promise that every step leaves the
editor usable — between A and C the timeline draws the new shape with the old
code. `npm test` is green at each step either way.
## Decisions taken
1. **Lane mode is `:display :lane` on the SYMBOL** — plan's recommendation 2,
and open question 1 answered "the symbol, not the instance". A symbol placed
twice is drawn as a lane in both places. Added to `symbol/symbol-keys` and to
`leaf/leaves`' `select-keys` so it saves like `:frames`.
2. **A symbol's children are its parent-less nodes** that have a placed span.
The plan's step 6 settles it: "an audio node is already a parent-less child
of a symbol, which is exactly the new shape". Span-less nodes — a shape on
screen for the whole shot — are not in the sequence and are skipped, which is
also what stops the commands destructuring a nil span.
3. **A symbol holds at most one sequence.** It follows from 1 and 2: the
container is the symbol. Two lanes of picture is now two symbols placed in a
third, which is what compositing already was.
4. **The open symbol gets a row of its own in lane mode**, and only then. The
blocks have to sit on a row and the open symbol had none; expanding it turns
its children into ordinary rows. Not a row always, which would shift every
row in the pane for no gain.
5. **Open question 2** — a lane row's edge drag trims the PLACING INSTANCE's
span, via `span/resize-out`, like the handle on every other row. Rippling
the children is what the cel blocks' own edges already do, and giving one
handle two meanings is what the plan refuses elsewhere.
6. **Open question 3** — the lane work first, the held cel after. The plan says
they are independent, and the held cel is joined to a loop control that does
not exist yet; doing it second costs one more pass over `lane_test`'s
fixtures and risks nothing.
7. **Lane creation stays explicit.** A blank document and `new symbol` create
ordinary symbols. The separate `new → lane` command creates and places a
symbol with `:display :lane` in the effective creation target derived from
selection and playhead; the new lane then becomes the primary selection.
## Notes

301
docs/lane-is-a-view-plan.md
View file

@ -1,301 +0,0 @@
# A lane is a view
Plan, 2026-10-01, written at `2dc5735`. It undoes the lane model as a thing in
the document and keeps what it was for. Build on what is there and tear out
half of it.
## The decision
> A lane is a view over a symbol with sequential, non-overlapping children.
Nothing in the document is a lane. There is no lane type, no lane group, no
`:layout :sequence`, no lane commands and no lane validation. The word
survives in exactly two places: the UI, where a symbol can be DRAWN as a lane,
and the drag handling that re-spans a symbol's children while it is being
drawn that way.
The display model goes back to a row per symbol. A symbol in lane mode draws
its children as blocks on its own single row; expanded, they are rows like
anything else. Everything else is an ordinary row that expands into what it
places.
## What a lane was, and what each part becomes
| was | becomes |
| --- | --- |
| a group node with `:layout :sequence` | nothing — the symbol is the container |
| `node/lane?` | a view question: is this symbol drawn in lane mode |
| `symbol/lane-clips nodes lane-id` | the children of a symbol, sorted by `node/placed-span` |
| `symbol/lane-problems` | `symbol/overlaps`, a diagnostic the write path calls |
| `lane/lane-frame` | `clip/source-time` — one clock instead of two |
| `domain/lane.cljs` | re-based onto `domain/span.cljs`: re-spanning a symbol's children |
| `clip/lane-node`, the born-with lane | gone; a symbol is born empty again |
| `::ui/new-lane`, `::ui/adopt-in-lane`, lane renaming | gone, gone, and ordinary node renaming |
`lane.cljs`'s fourteen commands are not deleted — they are what "endpoint drag
overlap handling" means, and they already do the right arithmetic. What
changes is their subject: every one of them currently takes a host symbol AND
a lane id and asks `lane-clips nodes lane-id`; each takes a symbol and asks
for its children. `extend-hold`, `resize-out`, `resize-in`, `roll`, `blank`,
`place-symbol`, `adopt`, `append-drawing`, `reuse-drawing`,
`duplicate-drawing`, `overwrite-drawing`, `make-unique`. `span/finish` is
already the one commit path and stays exactly as it is.
Put them in `span.cljs`, which already owns "one write to one node's span" and
`finish`. The sequence operations are the same subject — re-spanning children
— and keeping them apart was a consequence of lanes existing.
## Children in lane mode never overlap
This is an invariant, not a condition to check for and report. Placement
claims time: anything placed, moved or grown over occupied time TRIMS the
extents it lands on — trimming the incumbent, removing one wholly covered, or
splitting one it lands inside — so the result has no overlap because the
operation that could have made one did not. That is `blank` followed by a
non-rippling placement, which is what `overwrite-drawing` already composes.
Enforced at the boundary, which already exists: `span/finish` is the single
commit path for every one of these commands, it validates before it returns,
and it refuses rather than half-applying. So `finish` gains the overlap check
for a symbol in lane mode, and no command can commit one. An overlap that
appears anyway is a bug in a command, not a state to design around.
Keep the check as a named diagnostic — `symbol/overlaps`, taking a symbol and
returning the pairs — used three ways:
1. `span/finish` refuses when it would commit one.
2. The test suite asserts no command can produce one: a property over the
commands in the style of `drawn` in `lane_test`, which samples rather than
computing expected numbers by hand.
3. A document that somehow arrives holding one still LOADS — a display hint
must never be able to stop a document loading — and the timeline draws it
visibly wrong with the status line saying so. Not `clip/problems`, which
means the document will not load, and not `clip/conflicts`, which means a
person has a decision to make. This is neither: it is a bug report.
Toggling lane mode ON for a symbol whose children already overlap is the one
place a person can ask for the impossible. Refuse it and say why, with the
`:required-frames` retry pattern offering to trim them into a sequence — the
domain reports what it would need, the UI offers one button.
Outside lane mode nothing is enforced, because overlapping children are what
compositing IS. An endpoint drag there is an ordinary span edit that may
overlap; the claim-time rule follows the mode.
## The two drag intentions
Unchanged from `docs/lane-nesting-notes.md`, and both kept:
- **Plain drag** of a clip body is temporal: it moves in time, within its
symbol or into another symbol drawn as a lane, and it REPLACES — trimming,
removing and splitting extents as needed so nothing overlaps.
- **Shift-drag** is structural: the dragged node goes INSIDE the symbol the
clip under the pointer places, through `nest/move-node`, which preserves the
world transform and the root timing. This must keep working for symbols
contained in a lane, which is the case it exists for.
Overlap cannot distinguish them — dropping on occupied time already means
claiming it — so the modifier says which, and the label by the pointer says it
back. `nest/move-refusal` already answers before the drop.
## Where lane mode lives
A symbol is drawn as a lane because somebody said so, not because of what its
children happen to look like at this moment. Deriving it from "the children do
not currently overlap" means a symbol stops being a lane the moment anything
overlaps, and the rules that maintain non-overlap switch off exactly when they
are needed.
Two options:
1. **Editor state**, `[:ui :lane-mode #{sid}]`. Purest reading of "a lane is a
view". But the drag rules follow the mode, so an unsaved, per-person toggle
would decide whether dropping a symbol trims its neighbour or composites
over it — the same gesture doing two different things to the document
depending on something the document does not record.
2. **A display hint on the symbol**, e.g. `:display :lane`, saved like any
other field (`clip-keys`, `leaf/leaves`, `leaf/clip` in the same commit).
Still not a type: nothing in evaluation reads it, `symbol/problems` does
not check it, and a symbol with it set behaves identically on the stage.
Recommended: 2. It is one field, it keeps editing rules reproducible between
people, and it does not make the symbol a different kind of thing. The thing
to hold the line on is that nothing outside the timeline, and the commit
path's overlap check, is allowed to read it.
## Two things are called loop
Before any of this, name them apart, in the way the vocabulary table in
`lane-handoff.md` names a cel apart from an exposure.
- **Loop playback** is the transport repeating the open symbol while it plays.
It is `[:playback :loop?]` in app-db, the ⟳ button in the strip, and it is
EDITOR STATE. The document does not know about it.
- **A looping instance** is a node repeating the symbol it places: a four-frame
tire turning for the hundred and twenty frames the instance is on screen.
It is `:playback {:end :loop}` on the node, and it is in the DOCUMENT.
The car tire is the second one, and here is its actual status: it already
works in the evaluator and cannot be asked for. `node/placed-frame` does the
modulo, `node/problems` already admits `:end` of `:stop`, `:hold` or `:loop`,
`nest/audio-tracks` already expands a loop into its periods — and nothing in
the UI sets it. It is implemented and unreachable.
So three things are missing, and they are the work:
1. **A control.** Where an instance's playback is edited: `:in`, `:speed`, and
what happens at the end. One place, three fields, rather than a loop
checkbox somewhere else.
2. **Drawing the repeats.** `ui/timeline`'s docstring already admits that only
the first pass of a looping instance is drawn, so a tire turning thirty
times shows one turn's keys and then nothing. A looping block should show
its passes — at minimum the period boundaries, so the row says how many
times round it goes.
3. **The audio period guard** below, which a loop needs whether or not holds
become loops.
## Tear out the held cel
A held cel is a 1-frame symbol shown for many frames. A 1-frame symbol with
`:end :loop`, lengthened, is the same picture by a different route — and the
second route is a case the model already has, so keeping the first one is
keeping a special case for free.
**What is already true**, so that nothing has to move: `:span` is on the node,
in its own frames; `:time` (`:at`, `:rate`) is on the node; looping is on the
node, as `:playback :end`. The SYMBOL owns only `:frames`, the authored
window. So looping and span are instance properties already, and this change
is about deleting a mode, not relocating a field.
It does mean the two changes are joined at one point: making every drawing a
looping instance is not safe until a looping instance can be seen and edited,
or every drawing in the document acquires a property with no control on it.
**What has to be decided and collapsed:**
- There are two spellings of looping — `:time :loop?` and `:playback :end
:loop` — and both are read, in `node/placed-frame` and in
`nest/audio-tracks`. Keep one. `:playback {:in :speed :end}` already says
what happens at the ends, so `:end :loop` is the one to keep and
`:time :loop?` is the one to delete.
- `:playback :speed 0` stops being produced. Make it illegal in
`node/problems` rather than legal-but-unused, so a frozen clock has exactly
one spelling: a 1-frame loop.
- `lane/extend-hold` exists only because holds were special — it refuses
anything whose speed is not 0 and then edits a span. Once a hold is a loop,
lengthening one IS `resize-out`, and the command collapses into it.
- `cel` survives as the word for the creation policy — a new empty symbol is
one frame — but stops naming a playback mode.
**What it buys, and this is the point:** one rule for nesting, which settles
the refusal that blocks shift-to-reparent today. `nest/inside` currently has
no `:time` for a hold, for `:end :hold`, or for a loop, and so refuses all
three. The general rule that covers all of them: **resolve the move with the
destination's map at the CURRENT frame — the affine piece the current frame
falls in.**
- A loop of length L is affine within the period the current frame is in.
Timing is preserved inside that period and repeats after it, which is what
looping means.
- A 1-frame loop — the ex-hold — has a period of length 1, so the map within
it is trivially invertible and lands the moved node on frame 0, aligned to
the current frame. That is exactly the answer `docs/lane-nesting-notes.md`
argues for from first principles, arrived at here as an instance of the
general rule instead of a special case.
- `:end :hold` is affine in the played part and frozen in the tail, which the
same sentence covers.
**The trap, which must be handled in the same change.** `nest/audio-tracks`
expands a loop into one walk PER PERIOD:
periods (range (floor (/ (to-local source lo) length))
(ceil (/ (to-local source hi) length)))
Today a held cel is skipped entirely — `(pos? speed)` is the guard, and the
comment says a visual freeze does not emit a sustained audio sample. Turn
every drawing into a 1-frame loop and that guard stops firing: a drawing held
for 120 frames becomes 120 recursive walks, and any sound inside it is emitted
120 times. That is both a wrong mix and a performance cliff on the most common
node in the document. Required with this change: a cheap `symbol/audible?`
precheck so a source with no audio anywhere inside it is never period-expanded,
and a cap or a different formulation for the ones that are.
Also worth knowing before the change: `ui/timeline`'s own docstring already
says a looping instance draws only its first pass. With every drawing a loop,
that sentence now describes every drawing — harmless, since one pass of a
1-frame symbol is the whole of it, but the docstring should stop sounding like
a limitation.
**Dropping into a 1-frame symbol.** The window is authored and crops what it
holds, so a 10-frame symbol dropped into a 1-frame drawing shows its frame 0
and nothing else. That is consistent — `:frames` is the shot length and
`:extent :grow-symbol` is the opt-in — but it is probably not what somebody
dragging means. Offer the growth through the `:required-frames` retry the
model already uses: the command reports what it would need, the UI offers one
button.
## Order of work
Each step compiles, passes `npm test`, and leaves the editor usable.
1. **Row per symbol.** In `ui/timeline.cljs`, delete `portal`, the `::portal`
hint row, the `under?` lineage predicate and the `chosen` argument; emit a
row for a clip whose parent is a lane instead of skipping it. Keep the
`:cels` blocks for the collapsed row, and keep `inside-rows` — including
its `:unmapped?` branch, which is what makes a held drawing's contents
reachable at all.
2. **Lane mode as a hint.** Add the field and the toggle, draw a symbol's
children as blocks when it is set and as rows when it is not, and move the
lane-row drag handling onto it. Both display paths now exist and nothing in
the domain has changed.
3. **Re-base the commands.** Move `lane.cljs` into `span.cljs`, replacing
`(lane-clips nodes lane-id)` with the symbol's children and dropping the
`lane-id` argument. `frontend/test/arthur/domain/lane_test.cljs` is the
proof: its fixtures should change and its assertions should not, and any
assertion that has to change is a behaviour change worth noticing.
4. **Move the invariant.** `symbol/lane-problems` becomes `symbol/overlaps`,
called by `span/finish` for a symbol in lane mode, plus the property test
that no command can produce an overlap.
5. **Delete the rest.** `node/lane?`, `symbol/lane-clips`, `clip/lane-node`,
`::ui/new-lane`, `::ui/adopt-in-lane`, the lane branch of
`::ui/new-symbol`, lane renaming, `aimed-lane`, and the
`:lane?`/`sound-lane?` row flags. Rename what is left so the word does not
appear outside the timeline.
6. **Audio falls out.** An audio node is already a parent-less child of a
symbol, which is exactly the new shape — so the audio-in-lane rules added
in `2dc5735` (`holds-other?`, the mixed-lane refusal, the `in-lane` filter
in `sound-rows`) delete rather than migrate. A symbol drawn as a lane whose
children are sounds is an audio lane, and that is the whole of it.
7. **Shift-to-reparent stays** as it is: `nest/move-node` and
`nest/move-refusal` never knew about lanes.
## What must not be lost
All of this was broken at some point today and is now proved; each has a test
to keep.
- A held clip's contents are reachable from the root timeline, with no keys
and no draggable edges — `source-time` is nil for a hold, and the walk used
to stop there.
- Double-clicking a clip opens its symbol as a tab, and the editor survives
it: `symbol/lineage` must not report a cycle for an id the symbol does not
hold, and opening a symbol must drop a selection pointing into the one being
left.
- Selection waits for pointer-up, so a press does not re-draw the timeline out
from under the gesture it is starting.
- A drop never silently deletes what it lands on.
- A sound is drawn once, not twice.
## Open questions
1. **Is lane mode a property of the symbol or of the instance placing it?** A
symbol placed twice would be drawn the same way in both places under the
first reading. That is probably right, and worth saying out loud.
2. **Does a lane row's edge drag trim the placing instance's span, or ripple
the children?** Same handle, two commands; the row is now an instance, so
it has a span of its own for the first time.
3. **Does tearing out the held cel come before or after the lane work?** It
is independent of it — `nest` never knew about lanes — and it is what makes
shift-to-reparent work on the thing people would actually drag onto. Doing
it first means the lane work lands on a model with one playback mode fewer;
doing it after means two changes to `lane_test`'s fixtures instead of one.

591
docs/lane-model.md
View file

@ -1,591 +0,0 @@
# The Lane Model
Revised 2026-10-01. Clip ownership, source playback, one-row generic lanes,
direct clip movement and edge editing, correction evaluation, and correction
authoring for rotation and position are implemented. The former cel-sheet
projection was removed: the timeline is the single timing interface. Sections
below that describe a cel sheet are retained as design history and are superseded
by this revision. Retiming commands are not.
See the status note under
[Proof obligations](#proof-obligations-and-implementation-order).
[Lane and cel handoff](lane-handoff.md) records what is built, the decisions
that are settled, and what to do next.
This revises the Claude artifact [The Lane Model](https://claude.ai/code/artifact/cd42981d-ed08-493f-94df-b7dd6657f0e6).
Its prose and diagram source were recovered from session
`1c603f71-84eb-498e-aeaf-4c0346f1f513`; the live artifact was not accessible for
reading or editing here. This repository document is the revised design. The
original artifact has not been updated, and edits made there outside the recorded
session may not be represented here.
For the subjects covered here, this document supersedes the original artifact
and conflicting proposals in `animation-model.md`, `timing-model.md`, and
`architecture.md`. Those documents retain useful detail about the existing system.
## Goal and compatibility policy
Arthur is one animation document with several ways to see and edit it: drawing
on the stage, arranging clips, timing cels, editing curves, and generating
motion from footage. Each view exposes relevant facts and invokes shared editing
operations. Switching views must preserve the meaning of the work.
The user explicitly requires no backward compatibility. Replace obsolete shapes,
APIs, and tests when a better model requires it. Do not retain compatibility
branches, adapters, or migrations solely to preserve the current document format.
A format marker can reject unsupported files clearly; it does not promise to
convert them. This policy does not authorize deleting existing user assets.
Simplicity means predictable composition, clear ownership, and few independent
rules. Minimizing field count is secondary to representing independent choices.
## What stays
- A symbol is the one container for authored scene nodes. A drawing can be a
one-frame symbol; an animation uses the same container over more frames.
- Nodes have stable identities and flat parent references. Shared content is
referenced rather than copied implicitly.
- Animatable properties are addressed by channel paths. Generated and authored
values participate in the same evaluation machinery.
- Authored data, generated blocks, and source media remain separate. Documents
reference immutable blocks; caches and resolver indexes remain derived.
- A pure reference evaluator specifies the result. Playback, seeking, preview,
export, and optimized cursors must agree with it.
- Existence, visibility, and missing measured data remain distinct facts.
## Content, cels, lanes, and rows
These have different identities and responsibilities:
| Concept | Owns | Example |
| --- | --- | --- |
| Content | Reusable nodes and their animation | Drawing `a2`, an animated head, or a sound asset |
| Cel | One use of content, its interval, source playback, and local treatment | `a2` exposed on frames 12–16 |
| Lane | A sequence of cels and shared properties | The girl's drawings and the girl's overall transform |
| View row or column | Presentation and editor state | Timeline row, cel-sheet column, or property curve |
Use the existing instance/node identity mechanism for cels. A cel
should not acquire a second identity system just because it is shown as a cel.
Lanes group cels; they do not introduce another node-holding content type.
The concrete candidate below uses existing group and instance nodes; its ownership
boundaries are part of the design. It is now the implemented shape, and the field
spellings below are the ones the runtime reads.
Each cel has a stable ID. Moving it, changing its hold, swapping its source,
or trimming it preserves that ID. Repeating it creates a new cel that may
reference the same content. A split retains the original ID on the left and gives
the right piece a new ID; commands return the resulting selection explicitly.
Cels are the canonical authored arrangement. A source-at-time channel or
interval index may be compiled from them for evaluation, but is not a second
editable copy of the schedule. This replaces the earlier proposal that every cel
must be represented solely as a source key. Ordinary property animation still
uses lightweight keys; it does not need cel objects.
A lane has non-overlapping half-open cel intervals `[start,end)`
in its own time space. Uncovered intervals are gaps. Empty lanes are valid.
Compositing and simultaneous sounds are represented by multiple lanes or ordinary
scene composition; an accidental overlap never silently selects a winner.
Transitions, if added, need explicit overlap and mixing semantics.
Properties can belong to content, one cel, or the lane. For example:
- Rotate the reusable drawing: all its uses change.
- Rotate one cel: only that cel changes.
- Animate the lane's rotation: whichever drawing is showing follows it.
A cel can have its own transform, gain, corrections, and source timing
while remaining a block in the same timeline row. Independent treatment never
requires a new row or an otherwise unnecessary wrapper symbol.
### Concrete candidate: a lane and ordinary instances
A lane is a group node with `:layout :sequence`. Its cels are ordinary
instance nodes whose `:parent` points to the group. All remain in their symbol's
flat node map. The sequence constraint is document semantics; which rows the UI
expands remains editor state. Ordinary groups retain unconstrained composition.
This example is a document the runtime accepts, built and evaluated by
`frontend/test/arthur/domain/lane_test.cljs`. Times here are zero-based. Channels
use the existing representation; cel source references and playback have
replaced the `[:source]` channel, which no longer exists.
```clojure
;; Within :main's :nodes; referenced drawings/animations live in :symbols.
{:girl
{:id :girl :kind :group :layout :sequence :z "b"
:channels {[:xform :rot]
{:animated? true :interp :linear :keys {0 0, 6 30, 12 0}}}}
:cel-a
{:id :cel-a :kind :instance :parent :girl :z "a"
:time {:at 0 :rate 1} :span [0 4]
:source {:symbol :drawing-a}
:playback {:in 0 :speed 0 :end :stop}}
:cel-b
{:id :cel-b :kind :instance :parent :girl :z "b"
:time {:at 4 :rate 1} :span [0 4]
:source {:symbol :drawing-b}
:playback {:in 0 :speed 0 :end :stop}
:channels {[:xform :pos]
{:animated? true :interp :hold :keys {0 [0 0], 1 [2 0]}}}}
:animated-insert
{:id :animated-insert :kind :instance :parent :girl :z "c"
:time {:at 8 :rate 1} :span [0 4]
:source {:symbol :wave}
:playback {:in 3 :speed 1 :end :stop}}}
```
The lane's rotation reads lane time. Each cel's channels read cel
time. Its content reads source time. The stills sample frame 0, while the insert
samples source frames 3, 4, 5, and 6. The group transform composes with the
cel transform and then the content's own transform.
`:span` remains in the node's own coordinates, consistent with ordinary nodes.
The interval in lane time is derived through `:time`; do not also store parent
start/end values. Sequence children require finite intervals and positive
placement rates. Ordering and overlap checks use the mapped intervals, not `:z`.
The current sequence group contains visual symbol clips. Audio remains an
independent root node (and can be linked to picture); if audio lanes are added,
their capability must be explicit rather than inferred per frame.
A source reference is fixed within a cel. The lane changes content when
another cel becomes active. This is a deliberate revision of the original
diagnosis that making `:of` a channel was necessary to avoid vertical growth:
multiple instances can occupy one row when the view presents their containing
sequence. A lane-level source schedule is therefore derived, not authored twice.
## Source selection and source playback are independent
The current implementation makes framed sources play and keyed sources hold.
Retire that rule. Channel storage shape must not determine playback behavior.
Adding or removing a key must not turn a still into an animation or vice versa.
A cel names content and describes how its source time is sampled. In the
basic case, after mapping lane time into cel time:
```text
source_time = in_point + speed × cel_time
```
A newly created cel starts at local time zero. Moving it preserves this
origin relative to its content. Trimming can narrow its local support without
resetting that origin; split pieces likewise preserve the source and property
values at the cut. Trimming, slipping, and retiming are distinct operations with
explicitly different effects on the interval and the source map.
| Intent | Source playback |
| --- | --- |
| Hold a drawing | Constant source frame, equivalently speed 0 |
| Play an animated symbol | Advancing source time, normally speed 1 |
| Cut between animations | Several cels, each with its own in-point and speed |
| Mix stills and animation in a lane | Constant and advancing maps in the same sequence |
The source reference itself is discrete and never numerically interpolated.
Interpolation belongs to properties that support it; a property registry should
declare value types, defaults, and permitted interpolation and correction modes.
Generic key toggles must consult those capabilities rather than assume every
non-boolean value can be tweened.
Define source bounds and end behavior explicitly: stop contributing outside the
source, hold an endpoint, or loop an explicit range. A still uses a valid constant
frame. A loop uses a nonempty half-open range and a defined modulo rule. Playback
never guesses these policies from whether a channel happens to have keys.
Audio shares cel arrangement, trimming, gain ownership, and clock mapping.
It does not inherit visual frame-hold semantics: holding one audio sample is not
an audio freeze effect. Validate supported playback policies by media capability.
Actual audio scheduling must follow active cels, including gaps and cuts,
rather than playing every sound reachable through a structural reference.
## Time spaces and sampling
Name the relevant space whenever an API accepts a time or range: project,
symbol/lane, cel, or source. Store authored frame coordinates exactly;
avoid cumulative rounding when moving through nested mappings. Quantize at a
declared sampling boundary, not at every traversal step. Audio also needs its
continuous clock/sample space rather than visual frame quantization.
A hold is an evaluable time map with no unique inverse. A loop can map many
displayed cels to one source time. APIs must distinguish forward sampling
from inverse editing, and expose enough context to resolve a cel or
explicitly refuse an ambiguous operation. Do not report a missing time map merely
because inversion is unavailable.
Separate invertible placement timing from source sampling. A zero source speed
can mean hold without making the cel's own edit clock non-invertible.
Reparenting through changing transforms or non-invertible timing must either
preserve the full result by an explicit bake or return a reason it cannot; a
matrix captured at one frame does not prove preservation across the animation.
The source's frame step, generated-pose sampling, and the lane's transform clock
are independent scopes. Drawing on twos must not accidentally step a smooth lane
transform. An explicit whole-subtree stepping operation can exist separately.
Share the quantization primitive where possible, but retain its units, phase,
rounding policy, and order relative to retiming and lead. The original suggestion
that cel and picture-rate sampling are simply one floor is insufficient:
noninteger grids and source-frame quantization require specified behavior.
Identity timing can be implicit; remove `:time :mode` if it only duplicates that.
A cel interval is authored. Lane content extent is derived from its
cels, including the explicit end of the last one. A separately authored
container trim/window is legitimate when it intentionally gates children. Do not
conflate that window with occupied extent or infer a final hold from the next key
when no next key exists. A range of frame numbers alone cannot encode visibility
or a missing measurement.
## Shared editing operations
Every view issues the same domain commands. A command accepts an explicit target
and edit policy, computes a valid change, and returns the change, resulting
selection, and any refusal reason. A button and a drag must not implement two
versions of cel extension.
An edit target identifies the symbol, cel path, selected entities or
properties, and the time range with its space. Navigation also distinguishes
editing shared content directly from editing it through a particular cel.
Crossing a source cut must not silently redirect an active drawing edit to a
different symbol: retain the explicit content target until navigation changes it.
Core commands include new drawing, reuse drawing, duplicate drawing, make unique,
blank range, split, trim, move, extend cel, slip source, retime, and apply a
bounded property edit. Ripple/overwrite policy and the set of affected lanes are
explicit command arguments. Preview consequences before committing a gesture.
New drawing creates fresh empty content and a cel. Blank range removes
content coverage without inventing a hidden drawing. These are different actions.
Reuse creates another cel pointing at existing content. Duplicate creates
a new content identity. Make unique rebinds the selected cel only.
Copy semantics must specify nested sharing. A normal content copy duplicates its
owned nodes and channels while preserving references to other reusable symbols.
For a fully independent drawing assembled from nested symbols, provide an
explicit deep-copy operation with ID remapping. Never promise decoupling while
leaving the relevant edited object shared. Immutable media blocks may remain shared.
Commands are atomic undo transactions, even when they touch several leaves.
Pointer movement and keyboard invocation use explicit begin/preview/commit or
cancel boundaries; a timing heuristic alone must not decide user intent.
Collaboration applies a transaction consistently, validates affected references,
and detects conflicts at the owned data being changed. One leaf per channel does
not solve simultaneous edits to different keys of that same channel; define a
conflict policy rather than claiming that granularity solves all collaboration.
### Default timing behavior: cel edits preserve lane keys
Working default from the follow-up discussion: extending a drawing's hold changes
cel timing, leaving lane animation at its authored times. The user raised
keeping keyframes in place as a possibility; this is the proposed predictable
default, not a claim that they selected every timing policy below.
Ownership supplies the remaining rule: properties attached to a cel
travel with it. Extending its end does not stretch those properties; moving it
changes where their existing local times land. No per-key attachment flag is
needed to recover ownership that the document already expresses.
For the concrete example, extend `:cel-a` by two lane frames with ripple:
| Fact | Before | After |
| --- | --- | --- |
| Cel A's lane interval | `[0,4)` | `[0,6)` |
| Cel B's lane interval | `[4,8)` | `[6,10)` |
| Animated insert's lane interval | `[8,12)` | `[10,14)` |
| Girl's rotation peak | Lane frame 6 | Lane frame 6 |
| B's position change | B frame 1, lane frame 5 | B frame 1, lane frame 7 |
| Insert's first source frame | Source frame 3 | Source frame 3 |
The rotation peak now coincides with a different point in the drawing sequence.
That is the intended consequence of changing cels underneath timed motion.
The position correction stays attached to drawing B's cel. Neither the
background's keys nor audio on another lane moves.
The command contract for this edit names the symbol and cel, a delta in
lane frames, `:ripple` behavior, and an explicit scope of cel timing. It
extends A's local support by the delta converted through A's placement rate,
and shifts subsequent cel placements by that delta in lane time. It does
not modify any channel's key map, source in-point, or playback speed. Reject a
nonpositive resulting duration. Validate and commit the entire change together.
The symbol's authored end is another explicit boundary: preview an overflow and
offer to extend the symbol or cancel. A command can request that extension as
part of its transaction; it must not silently truncate later cels or grow
other uses of a shared symbol. In the example, a 12-frame symbol needs an explicit
extension to 14 frames or the edit must be refused without partial changes.
Retime performance is a separate operation over explicitly selected cels
and channels. It applies the same time transformation to their relevant clocks,
keys, and correction supports. Stretching an interval requires a defined warp
and interpolation behavior; it is not merely moving keys whose frame numbers
happen to lie inside the selection. Until supported, refuse this operation
rather than approximating it with a cel ripple.
The initial UI should default stage transforms to the lane when drawing in a cel
workflow, so movement usually remains independent of cel timing. The
inspector names the target: lane motion, this cel, or shared drawing.
Changing that scope is explicit. It changes what the edit means, not just which
panel happens to be open.
## Three-frame rotation and correction layers
A range says where an edit applies; it does not specify the motion. Offer distinct
commands for a constant adjustment, a ramp, and a return-to-start motion. For UI
frames 10–12, the internal range contains exactly three frame samples after
conversion from the displayed numbering convention.
- Constant adjustment: the same offset throughout those three samples.
- Ramp: interpolate from the specified start value to the target over the range.
- Return motion: interpolate from the starting value to a peak and back.
For a return motion sampled on three frames, the values can be `0, angle, 0`.
Outside the selected range, the underlying animation must evaluate exactly as it
did before. A range-scoped correction layer expresses this directly; blindly
inserting boundary keys can alter neighboring segments or destroy existing motion.
Implement corrections as an ordered stack over the base channel. Each correction
has stable identity, explicit support interval, blend operation, and values in a
named time space. Outside its support it is inactive. `replace` can supply a value
over an absent base; `offset` cannot offset a nonexistent value. Blend capability
depends on property type, and geometry corrections require compatible topology.
Regeneration replaces the generated base and preserves corrections. If changed
topology or removed targets make a correction incompatible, report a resolvable
conflict instead of silently dropping or misapplying it. Provenance explains
where the base came from; explicit sampling policy determines its playback.
This is core to the workflow: generate motion, correct it by hand, adjust the
generator, and keep the corrections. It should be proven before adding many views.
## Other unifications worth keeping
Pose choices, tracing-frame choices, and ordinary held values should share the
channel evaluator and cursor infrastructure. Preserve their different ownership,
fallback behavior, and sampling scope. A pose choice must address the relevant
content/feature explicitly; switching to another symbol must not accidentally
reuse a track just because both symbols contain a node with the same local name.
Keep the two animation idioms distinct: keyed geometry modifies one mark over
time; drawing substitution selects content that may have different structure.
Linear geometry interpolation requires compatible vertex correspondence, not
merely two drawings that happen to look related.
Derived library grouping may collect drawings used by a single lane. This is a
convenience, not ownership or deletion authority. Reference discovery for cycle
validation, copying, and deletion examines all structural references, including
currently inactive cels. Authored folders, favorites, and labels remain
legitimate user data even when the UI could have suggested defaults.
## UX: location, selection, and controls
The breadcrumb sits above the timeline and states the editing location, shared
content identity, and cel context when applicable. Show local time and
its project context where a useful mapping exists. Holds and loops need an honest
description instead of a fictitious unique global frame.
Creation follows the primary active row, resolved at the playhead as specified in
[`creating-in.md`](creating-in.md). The wider selection set still names what copy,
delete and transform affect; it is not a second list of creation destinations. A
shared drawing indicates its reuse and offers Make this cel unique. Names help identify content;
linked-use indicators must rely on IDs, because different drawings can share names.
| Surface | Primary scope and controls |
| --- | --- |
| Topbar | Project name, save/open/export, project rate and stage size |
| Location bar | Breadcrumb, add lane/content, shared-content context |
| Cel action strip | New drawing, duplicate drawing, hold longer/shorter, blank range |
| Lane header | Lane selection, lock, mute/solo where applicable, onion settings, expansion |
| Stage tools | Drawing and transform modes, active target and scope |
| Inspector | Selected content/cel/lane properties and valid key controls |
Cel actions have visible contextual buttons, shortcuts, a context menu, and
command-palette entries. These are different entrances to the same commands.
Shortcut names from the original sketch (`N`, `D`, `H`, `B`, `K`) are provisional;
their meanings must match the visible labels and avoid tool conflicts.
The inspector normally edits values and the timeline normally edits timing, but
this is an organizational default. Numeric duration and in-point controls are
useful inspector edits to the same domain facts. Do not ban a convenient control
just to preserve a visual division.
Default nesting navigation enters content; expanding a lane reveals properties.
Other views may show hierarchies differently without changing the document.
Tabs can pin explicit locations. Zoom, expansion, onion preferences, and current
selection are editor state rather than animation content. Persistent workspace
preferences can be saved separately.
## A session, revised
1. In `main`, create a girl lane and a new drawing. Draw; use New drawing (`N`)
to create the next one with the previous cel ghosted behind it.
2. Use Duplicate drawing (`D`) when the current shapes are the starting point.
Use Reuse drawing for a deliberately linked cel. The UI shows the
difference before an edit can change other uses.
3. Time the performance. Hold longer (`H`) extends the selected cel and
ripples later cels in the explicitly targeted lane. A trim gesture
can use overwrite instead. The preview shows which boundaries will move.
4. Choose a two-frame default cel for newly created drawings, or run a
separate Retime cels command on a selected range. This does not quantize
lane transforms or silently retime already authored cels.
5. Place the background in a lane below. Its source holds one frame throughout
its cel. Key the lane's X position at the beginning and end and choose
linear interpolation. The background slides while the girl's drawings cut.
6. Select three frames on the girl's lane, choose Return motion, and rotate to
the desired peak. A bounded rotation correction affects the girl across any
drawing boundaries in that range. Existing motion survives outside it.
7. Insert a playing animated symbol among the girl's held drawings. Set that
cel's source playback to advance. No lane conversion is required.
The timeline shows named cel blocks with property marks and optional curve
subrows. The cel sheet shows the same cels by frame and lane. The
graph editor edits the same properties; the stage resolves the same document.
Onion skin is configurable and counts neighboring cel events, skipping gaps
by default; a long hold does not consume the budget. Repeated uses of the same
drawing remain distinct events. Deduplicating identical ghosts is a display option.
## Proof obligations and implementation order
The source-channel prototype has been removed: a cel names one symbol
and carries its own playback clock, and `node/problems` rejects the old
`[:source]` channel. What a lane IS lives in `arthur.domain.symbol` beside the
other rules about a node map; `arthur.domain.lane` holds the commands over
one — add lane, place a drawing (new, reused or duplicated), make unique, split,
trim, move, blank and extend hold. Each is one history step, and each refuses rather than
half-applying. The timeline draws a lane's cels as cel blocks on the
lane's own row, and offers Make unique only where the selected cel actually
shares its drawing.
There is ONE placement function and a position argument, so appending is not a
different operation from inserting: `:end` is a position like any other, the one
where nothing has to move. Placing ripples — cels at or after the
position move later by the new cel's duration — and `:keep` versus
`:grow-symbol` still decides what happens at the shot's end. OVERWRITE is not a
policy argument yet, deliberately: taking frames away from the cel
already there is trimming, and until `trim` exists, placement that would need it
refuses instead of approximating it. A position inside an existing cel
refuses too, and names `split` — one command does not quietly perform two.
Splitting turned out to cost almost nothing, which is evidence for the
representation rather than for the command. The two pieces keep ONE `:time` and
differ only in `:span`, so the right piece's own frames carry on where the
left's stopped and its source clock, keys and corrections go on meaning what
they meant — a held drawing holds the same frame either side, a playing insert
plays through the cut without a seam, and the test for it samples every frame
before and after and asserts the picture is identical. That falls out of `:span`
being in the node's own coordinates; it is not something split arranges.
Content copies are shallow by default and keep their references to other
symbols; `:deep? true` is the explicit copy that shares nothing, so the promise
of independence is only made where it is kept.
THE SHOT LENGTH IS AUTHORED, which is the decision the range commands forced.
`:frames` is the symbol's window — how long the shot IS — and the occupied
extent of its lanes is a different fact derived from the cels. A command
grows the window only when the caller says `:grow-symbol`, and never shrinks it:
blanking the end of a shot leaves a shot with empty frames at the end, because
that is a true statement about what somebody authored, and deriving the window
from the extent would make deleting the last drawing quietly shorten the film.
`finish` keeps the two numbers apart by name now rather than by a `max` that
read like an accident.
Trim NARROWS one edge and moves nothing else; lengthening is `extend-hold`,
which carries the ripple and shot-length policies because it needs them.
Move is one write to `:time :at` and REFUSES a destination that would overlap,
because moving a drawing and re-timing the ones around it are different
intentions — clear the room with `blank` or `trim` first, which is the
composition. Blank leaves a gap and does not close it; a cel wholly inside
the range goes, one overlapping an end is trimmed to it, and the one spanning
the range is split. Their drawings stay in the library, since a lane does not
own its content.
All three are the same geometry as `split`: a `:span` is in the cel's own
frames, so moving an edge is one write and `:time` and `:playback` are never
touched. That is why trimming the front of a playing insert starts it later into
its animation instead of restarting it — the difference between trimming and
slipping, and the reason they stay separate commands.
Correction layers EVALUATE. `channel/problems` used to refuse an `:over` stack
and `value-at`/`cursor` used to throw on one; both now read it, and the
agreement test that holds the optimized cursor to the specification covers
stacked channels in forward, backward and random frame order. A layer's values
are themselves a channel, so a constant adjustment, a ramp and a return motion
are one mechanism; `:support` is half-open and a layer is inactive outside it;
and a layer has no time space of its own, because the node its channel is on
already has one. Nothing had to change in the codec — a channel is one leaf, so
a correction persists inside it — and nothing had to change in validation
plumbing, since `node/problems` already reports every channel's problems.
Both halves of ownership are under test at lane level: a three-frame correction
on the girl's lane reaches across the drawing boundary beneath it and leaves
every frame outside its support identical, and a correction owned by one
cel travels with that cel when a hold before it grows.
Regeneration keeps them, which is the obligation the layer design exists to
meet: `rebased` replaces a base and carries its corrections across, and a
correction the new base no longer fits is MARKED rather than dropped or
misapplied — `clip/conflicts` lists those for a view to offer, separately from
`problems`, because a conflict is a decision nobody has made yet and not a
document that will not load. Turning the mouth's `:verts` knob is a real
topology change and is what the test uses. Two latent faults turned up there and
are fixed: `regenerate-head` compared authored channels to measured ones
directly, so the first correction on the head would have stopped it following
re-measurement for good; and an incompatible offset threw in the read path,
which would have taken the stage down on exactly the case the model says to
report.
Overwrite is `blank` followed by non-rippling placement, composed inside one
transaction; insertion keeps its ripple rule. Still unbuilt: slip source,
retime, and deleting reused content. A lane cannot hold AUDIO cels — `lane-problems`
requires visual ones, though this document says a lane may hold either and
should reject only a mixture.
`domain/correction.cljs` now produces Constant adjustment, Ramp, and Return
motion layers for rotation and position. The inspector exposes them on a selected
lane or cel using an explicit range in that owner's frames; this deliberately
leaves displayed-range dragging through nested or retimed owners for later. One
Apply is one undo step. Conflicted layers are listed, can be removed, and can be
retried when the complete ordered stack is compatible again. Validation,
conflict reporting, and regeneration share that ordered-stack rule, including
coverage by adjacent replacement layers. Slip source and retime are still not
implemented; a refusal is the current behavior where the model demands an
explicit choice nobody has made yet.
The cel sheet is the same projected cels and selection addresses with its axes
turned: frames down and lanes across, so commands selected there and in the
timeline have identical targets; a gap selects its column's lane rather than
retaining a stale selection from another column. The suite stands at 437 tests and 5,804
assertions, with `frontend/test/browser/lane.mjs` driving the editor through
create, hold, overflow, undo, reuse, make unique, duplicate, split, insert,
trim, move, blank, correction authoring, and two-lane sheet targeting. Rewrite tests that encode superseded
behavior rather than preserving behavior to keep them green.
Build small adversarial documents and test their domain operations before
expanding the interface:
| Scenario | Required invariant |
| --- | --- |
| Same drawing exposed twice, then one made unique | Linked edits affect both before copying and only the selected content after |
| Holds, playing inserts, nonzero in-points, and gaps on one lane | Source behavior is independent of property key count and channel encoding |
| Adjacent cels, final hold, split, trim, ripple, and overwrite | Exact boundaries, stable IDs, deterministic collision handling |
| Extend a hold under lane keys and cel-local corrections | Lane key times remain fixed; later cel corrections travel with their owners; source playback origins survive |
| Ripple beyond the symbol end | Explicit extent policy; refusal leaves the document unchanged; resizing and retiming undo together |
| Girl on twos over a moving background | Drawing cadence does not quantize either lane's continuous properties |
| Three-frame correction crossing a drawing boundary | Exact support, same result outside it, one undo step |
| Nested retiming, holds, loops, and fractional sampling | Explicit time spaces; ambiguous inverse edits cannot silently choose a target |
| Audio inside changing source cels | Only active intervals sound, with correct trim and source timing |
| Regenerate with corrections and a topology change | Compatible edits survive; incompatible ones produce actionable conflicts |
| Reference cycles and deletion of reused content | Inactive references are validated too; no dangling references |
| Save/load and command undo/redo | Identity, source maps, corrections, and evaluation round-trip |
| Timeline and cel-sheet invocation of one command | Identical document changes and selection targets |
| Random forward/backward seeks and export | Reference and optimized evaluation agree, including defaults and absence |
| Concurrent commands on overlapping and disjoint targets | Transactions remain valid; conflicts are explicit and undo preserves others' work |
Implementation order: cel ownership and playback semantics; shared
commands and validation; correction layers and time-addressing contracts; then
breadcrumb, cel strip, and a cel-sheet projection. Use those two temporal
views plus direct stage editing to prove the model before broadening the UI.
A new presentation should not require duplicate animation state. A genuinely new
authoring capability may require new domain data. The model is successful when
such additions have a clear owner and compose with existing operations, not when
it can claim that no future feature will ever need another field.

161
docs/lane-nesting-notes.md
View file

@ -1,161 +0,0 @@
# Lane nesting interaction notes
Status: design note, 2026-10-01. This records the interaction before more lane
UI is implemented.
## The capability that must not be lost
A lane owns temporal placement, but a symbol instance is still a doorway into
another symbol. A drawing accidentally authored at the root must be movable into
an instance in any lane, including another lane, without changing its visible
position or timing.
That operation already exists as `nest/move-node`. It resolves the source and
destination at the current root frame, transplants the node, and re-expresses
its transform and time under the new parent. The lane UI must expose a target
path for it; it must not replace it with a weaker `:parent` assignment.
There are therefore two different drag intentions:
1. **Temporal move:** drag a clip body onto lane space. It remains a clip in a
lane, moves in time, and claims the destination interval by trimming/removing
incumbents.
2. **Structural move:** drag from the clip's grab affordance onto another symbol
instance. The dragged node is transplanted into the target instance's source
symbol with `nest/move-node`, preserving its world transform and root timing.
These cannot be inferred from overlap alone. Dropping clip A onto time occupied
by clip B already means “A claims that time and trims B.” Structural nesting
therefore needs an explicit grab affordance/mode. Its cursor is `grab` and
`grabbing`; trim edges keep their resize cursors and the ordinary body keeps its
timeline-move behavior.
Both visible clip blocks and an expanded symbol header are structural drop
targets. This permits moving a root drawing directly into `symbol-3` even when
its lane is collapsed.
## Compact expansion: one selected-clip portal
Expanding a lane must not restore row-per-clip vertical growth. Instead, an
expanded lane reveals exactly one clip portal: the currently selected clip in
that lane.
```text
▾ foreground lane [symbol-1][symbol-2][symbol-3]
▾ symbol-3 instance/source header and drop target
▸ body lane nested rows, mapped to the root ruler
▸ face lane
position nested keyframes mapped to root time
```
- Selecting another block in the same lane swaps the portal in place.
- With no selected clip in that lane, expansion shows a compact “select a clip
to inspect” row. It must not follow the playhead during playback; that would
make the timeline restructure itself while playing.
- The portal header represents the selected instance and is the structural drop
target for moving root or sibling content into its source symbol.
- Sub-expanding the portal uses the existing recursive symbol-row walk. Nested
lanes and channels are mapped through the instance clock into the open/root
ruler, as ordinary expanded instances already are.
- The lane's own transform/channel rows remain available separately. They affect
every clip in the lane and are not properties of the selected portal.
This keeps the cost of inspection constant: an expanded lane adds one selected
symbol branch, not one branch for every temporal clip it contains.
## Keyframe visibility
Two levels should be visible without changing editors:
- The selected clip's instance-level keys (transform, visibility, corrections)
appear as ticks inside that clip block on the lane row.
- Expanding the lane opens the selected clip portal, where source-symbol and
recursively nested keys appear on their own rows, mapped to root time.
Thus the collapsed lane answers “where does this clip change?” and the expanded
portal answers “which property inside this symbol changes?” The second view is
still the root timeline; entering the symbol is not required merely to see or
edit its keys.
## Drag targets and feedback
- Grab onto lane background: move/adopt the instance into that lane.
- Grab onto a symbol clip: structurally transplant into that clip's source
symbol.
- Grab onto the expanded portal header: the same structural transplant, with a
larger and less ambiguous target.
- Grab onto itself or one of its descendants: refuse before drop to prevent a
symbol cycle.
- A structural target receives an inset highlight and the preview stays in that
target. A lane-time target receives the dashed temporal clip preview.
- Successful structural drops expand the target lane and select the moved node
beneath the target portal, so the result is immediately visible.
## Data model consequence
No lane-as-symbol type is required. The hierarchy remains:
```text
symbol -> sequence lane -> instance clip -> source symbol -> its lanes/nodes
```
Lane membership owns time partitioning. Symbol instances own composition
nesting. The UI may present the selected instance below its lane, but that is a
derived portal, not another ownership edge and not a duplicated node.
## Implementation order
1. ~~Render instance-level key ticks within lane clips.~~ Done: a clip's keys
are on its block, drawn after the blocks so they land on the one they
belong to.
2. ~~Add selected-clip portal expansion to `timeline/rows`.~~ Done, with two
additions the note did not anticipate:
- The portal is chosen by the whole LINEAGE of the selection, not the
selected id. Selecting a shape inside the clip, or the end of its span,
is still working inside that clip, and matching the id alone closed the
portal the moment anything under it was touched.
- A HELD clip opens too. `clip/source-time` is nil for a hold, so the walk
used to stop there and the inside of every drawing was unreachable from
the root timeline. Its rows are now shown across the hold and marked
`:unmapped?`: no keys, and no draggable edges, because no frame inside it
has a place on this ruler.
3. ~~Add the explicit structural affordance.~~ Done as SHIFT on a clip-body
drag rather than a separate grab handle: shift turns a temporal move into a
structural one, the target clip takes an inset highlight, and a label by the
pointer says which of the two is about to happen.
4. Route structural drops through `nest/move-node`. **Wired, and blocked in
the domain.** The gesture asks `nest/move-refusal` on the way past, so the
label says before the drop what the command would say after it. Two
refusals stand in the way of ordinary use:
- *both have to be on screen at this frame.* Inherent, and worth keeping:
the move preserves the world transform and there is no common frame to
preserve it at otherwise. It does mean nesting one clip into another in
the SAME lane can never work — a lane never overlaps itself — so this is
a between-lanes gesture with the playhead somewhere both are showing.
- *a held or looping clip has no clock to move through.* `nest/inside`
returns no `:time` for a hold, and a held one-frame drawing is the most
common thing in a document, so today nesting into one is refused — which
is most of what anybody would try.
5. After the transplant, expand the destination portal and reveal/select the
moved row. `::ui/move-node` already selects the moved node and opens the
rows down to it; the portal follows from the lineage rule in 2.
## The held destination, unresolved
A held cel shows ONE source frame for its whole span, so there is no
invertible map from the lane's frames to the drawing's and `move-node`
refuses. But the refusal is stronger than the facts require. Inside a frozen
destination only one frame is ever observed, so:
- the RATE of any map into it is unobservable — every rate shows frame `in`;
- what IS observable is that the moved node should show, at that one frame,
what it shows now at the current root frame.
That pins a unique sensible answer — rate 1, aligned so the current frame maps
to the shown frame — and nothing else about the mapping can be seen. If that
argument holds, it is a rule rather than a guess, and it is the difference
between structural nesting working for drawings and not working at all. It
needs its own proof: a drawing authored at the root, nested into a held cel in
another lane, sampled before and after to show the same picture, in the style
of `drawn` in `lane_test`.

103
docs/multi-face-representation.md
View file

@ -1,103 +0,0 @@
# Multi-face representation
Status (2026-09-29): representation work complete. Reopen it for a concrete
requirement, rather than another round of abstract alternatives.
Implemented: each tracked face has a drawing timeline, placed by an ordinary
symbol instance. Timelines already provide local node names, independent playback,
and persistence. No new kind of scene container is needed.
```clojure
:symbols
{:main {:nodes {:root {:time {:mode :map :expose 2}}
:face {:parent :root :channels <source-to-stage placement>}
:face-1 {:kind :instance :source {:symbol :face-1}
:parent :face :z "a0"}
:face-2 {:kind :instance :source {:symbol :face-2}
:parent :face :z "a1"}}}
:face-1 {:nodes {:head {...} :mouth {:parent :head ...} ...}}
:face-2 {:nodes {:head {...} :mouth {:parent :head ...} ...}}}
:features
{:face-1/mouth {:subject :face-1 :timeline :face-1 :area :mouth
:nodes [:mouth :mouth-in]}
:face-2/mouth {:subject :face-2 :timeline :face-2 :area :mouth
:nodes [:mouth :mouth-in]}}
```
The example omits ordinary ids, frame counts and channel details.
## What belongs where
- A **subject** identifies a source track and supplies shared measurement settings.
Its timeline has the same id and contains its measured `:head`.
- A **feature** owns nodes in an explicitly named timeline. Its clip-level id is
qualified when generated; ownership is read from fields, never parsed from ids.
- A **node** has a local name. Parents, stencils and pose groups use local names too.
- An **instance** places and retimes a drawing. Its pose tracks can hold one face's
mouth while the other face continues moving.
Subject metadata and drawing timelines remain separate facts. Hand-drawn timelines
need no subject. Features retain explicit timeline references, so their locations
are not inferred from their labels.
Both filmed faces share one source-to-stage transform. Fitting them independently
would stack them at the center. Additional placement uses each instance's ordinary
channels. Cross-face draw order is the instances' `:z` order.
## Consequences
Regeneration updates the addressed timeline directly. There is no temporary swap
into `:main`, no special stage regeneration path, and no renaming of parents or
stencils. Composing a stage moves the take's root into the library and preserves
its child timelines. Settings appear once per tracked object, since all placements
read that same drawing.
Presence masks inside a subject use local feature names, matching measurement.
Freeze qualifies them when building block descriptors. Head and retained-source
blocks explicitly name their subject; otherwise two faces with identical detection
masks could produce different bytes under the same key. Analysis addresses also
include detection capacity and assignment settings, so old single-face detection
results cannot satisfy a new multi-face request.
Validation counts node ownership by `[timeline node]`. The server requires one
complete set of retained source roles **per subject**, rather than exactly three
blocks for the entire analysis.
Nested rectangles retain fractional sizes until rasterization. Rounding inside a
face timeline discarded small head-local pupils before the source-to-stage scale
was applied. This was a real rendering error missed by the earlier proposal's
coordinate-only benchmark; the regression now compares all mark extents as well.
## Verification and limits
`frontend/test/arthur/flow/multi_face_test.cljs` exercises two distinct subjects,
source block separation, detection and feature gaps, independent pose cuts and
regeneration, nested stage save/load, and equivalence to a flat single-face scene.
Existing geometry, raster, source, and regeneration tests cover the same paths.
`clips/tests/test_api.py` checks complete source roles per subject and immutability.
The browser suite checks rendering, pupils, playback, save/open, drawing and upload.
Assignment remains a nearest-centroid heuristic, with a version and distance gate
recorded in the analysis. Reordered detections, late arrivals and gaps are tested;
identity through crossings or long disappearances is not guaranteed. Assignment
happens before measurement, so correcting it requires measuring again.
This changes the freeze and retained-source contracts. It does not migrate older
flat captures; reanalyze their footage to use the new regeneration path. Existing
rendering and leaf codecs still understand their node/channel representation.
## Next steps
1. Commit the verified checkpoint: 319 frontend tests, 44 API tests, browser
checks, and app/test builds passed. Builds reported no warnings.
2. Exercise real two-person footage, especially crossings, late arrivals and
disappearances. Check assignment before treating the resulting geometry as
evidence about the representation.
3. Build performance-pose selection and instance-scoped picture rates, reusing
the existing held-frame lookup and pose groups.
4. Then build plate-drawing selection and independent tracing references.
The [timing handoff](timing-handoff.md) owns the detailed next implementation
sequence. Older flat captures need reanalysis unless a migration is separately
undertaken to preserve their authored edits.

66
docs/one-grid-plan.md
View file

@ -1,66 +0,0 @@
# The editing grid is the symbol's own frames
## What was wrong
Every number a person authors — a span, a `:time :at`, a key, a cut — is in the
frame space of the symbol it lives in (`docs/time.md`, and that part is right).
The timeline, though, drew its ruler in OUTPUT frames: `clip/output-frames`, the
transport's length. In a 12fps project holding 30fps symbols those two spaces sit
at a ratio of 2.5, so:
* a clip at symbol frame 31 was drawn at ruler frame 12.4 — **no clip edge landed
on a frame mark**, because almost none of them can;
* a gesture measured in ruler frames had to be multiplied into the symbol's
frames and rounded (`nest/dragged`), so dragging one ruler frame moved the clip
2 or 3 symbol frames — **0.8 or 1.2 ruler frames, never the 1 the pointer
said**. That is the jumpiness;
* the preview drew the gesture's own number of ruler frames while the commit
wrote the rounded one, so **the ghost sat somewhere the settled clip did not**;
* before the rounding was added, the fraction went into the document and every
later edge edit on that clip was refused for ever (`span/*` refuses a
fractional edge, as it should).
One cause, four symptoms. None of them is an edge case to patch.
## The model
1. **A symbol's own frames are the only coordinate anything authored lives in.**
Unchanged.
2. **The editor edits in the open symbol's frames.** The ruler, the marks, the
playhead's position on it, every pointer→frame answer, every drop frame and
every drag delta are the open symbol's frames. No multiplication anywhere in
the gesture path, so no rounding and nothing fractional to refuse.
3. **The output grid is playback's alone** — the clock, the audio mix, export,
and the frame the stage draws. Exactly two pure functions cross between them
and nothing else does:
* `clip/shown-frame clip sid f` — which of `sid`'s frames output frame `f`
shows (`cadence/frame`: the latest at or before it).
* `clip/first-output-frame clip sid n` — the output frame that first shows
symbol frame `n`; the inverse, for seeking from the ruler.
4. `nest/inside`, `nest/placement`, `nest/spans` and the gestures take the
subject symbol's OWN frame. They used to take an output frame and multiply it
secretly, which is what made every caller's units a guess. A caller holding
the playhead converts with `clip/shown-frame`, at its own edge, visibly.
## The gestures
5. **One pointer→frame function** for the whole timeline, `frame-under`. A drag's
delta is the difference of two of its answers, never a pixel ratio rounded
separately — so the preview and the commit are the same number by
construction.
6. **The junction between two clips is one handle with one meaning**: roll. It
moves the end of the left clip and the start of the right one together, which
is `span/roll`, which is already nothing but `resize-out` then `resize-in`.
The three 4px-wide zones it used to pick between — trim-left, roll,
trim-right, inside twelve pixels — were the "it just picks one" the handle
was accused of. An edge that is not shared still has its own in/out handles.
## Audio goes with the picture it belongs to
A take's sound lived inside the take symbol, so placing the take brought it and
placing the FACE the take is made of brought nothing. A symbol now says what it
sounds like — `:audio`, a sound source — and placing one places a linked audio
node beside the clip. Detection sets it on the face it extracts, which is the
automatic link; `::ui/link-audio` sets or clears it by hand, which is the manual
one. `:linked-to` on the audio node already existed and already follows a moved
picture.

470
docs/port-plan.md
View file

@ -1,470 +0,0 @@
# arthur — port plan and handoff
Self-contained. You should not need any prior conversation to execute this.
**Implementation status (2026-09-29):** steps 0–9 are in. Step 6 reads extracted
footage, detects landmarks with local MediaPipe assets at full source cadence, and
runs the same freeze path as the synthetic take. The scene time map can sample the
frozen roto at a lower picture fps without changing source analysis, duration or
audio. Step 7 adds dense eyelids, shared gaze, brows and pixel-derived teeth.
Step 8's data model has stable feature identity, feature-level presence, explicit
eye pairs and shared parameter definitions; a manifest can supply known feature
absence intervals through measurement and freeze. Step 9 adds the Django backend,
the three-tier split, content-addressed tier 2 with the detector version inside
every key, leaf addressing for tier 1, and project load/save that round-trips.
Step 8 now has parameter controls and scoped regeneration from retained source.
Multi-face representation is complete: each tracked subject has a drawing
timeline, placed by an ordinary symbol instance. See
[multi-face representation](multi-face-representation.md) for the implemented
model, verification and compatibility limits.
**Next, in order:** commit the verified checkpoint; exercise real two-person
footage, including crossings and disappearances; build performance-pose
Suggest/Keep/Drop and instance-scoped picture rates; then add plate-drawing
selection and independent tracing references. The
[timing handoff](timing-handoff.md) records current code and implementation order.
Reopen the representation only for a concrete requirement it cannot express.
**Still open:** real-footage identity validation, automatic per-feature detection,
the timing and tracing work above, and time-varying parameter settings. Older flat
captures need reanalysis for the new regeneration path; no migration is included.
The step descriptions below retain the original port scope; this status and the
linked handoffs describe subsequent work.
## What arthur is
A tool that turns live-action video into 2D animation that reads as
hand-authored: flat polygons, a tiny indexed palette, hard edges, 320×200, no
antialiasing, motion carried by silhouette. It tracks a face out of a clip,
reduces the lip contour to a handful of vertices, derives teeth from image
content, and renders flat indexed fills.
It currently works, as vanilla JS ES modules with no build step. `python3
serve.py`, open `127.0.0.1:8777`. **Synthetic take** exercises everything below
detection with no video needed.
This plan converts it to ClojureScript + re-frame, restructured around one
uniform animation data model, and adds a Django backend for persistence and
(later) collaboration.
## Status of the existing documents
| File | What it is | Authority |
| --- | --- | --- |
| `js/**` | the working tool, ~4,800 lines | **authoritative.** The comments encode bugs that actually happened. |
| `docs/animation-model.md` | the target data model: nodes, channels, symbols, time maps | build to this |
| `docs/architecture.md` | module layout, stages, sync and baking design | build to this; much of it is future scope |
| `docs/design.md`, `README.md` | prior synthesis by an earlier agent | useful, **not authoritative**. Revise freely. Do not treat its aesthetic claims as settled requirements. |
Where a document and the code disagree, the code wins, and the invariant list
below is lifted from the code for exactly that reason.
## Target repo layout
Both halves live here. Django at the root, because `manage.py` at the root is the
convention and keeps every `python manage.py` invocation working with no `cd`.
```
arthur/
mise.toml toolchain for both halves
manage.py
requirements.txt
server/ Django project: settings, urls, asgi, wsgi
clips/ Django app: models, views, consumers, routing, migrations
frontend/ the CLJS app
shadow-cljs.edn
package.json
src/arthur/** namespace root stays arthur.* whatever the dir is called
test/arthur/**
static/arthur/js/ shadow-cljs output, collected by Django staticfiles
static/arthur/audio.wav the synthetic take's clock. NOT extract.sh's output —
that is tier 3 and lives in the blob store
var/blobs/ the content-addressed blob store: tiers 2 and 3. Gitignored
docs/
js/ index.html serve.py extract.sh the old tool — see "the oracle"
```
The namespaces step 9 added, since the list under **Namespaces** in
`docs/architecture.md` predates them:
```
domain/sha256.cljs SHA-256, synchronous and pure, byte-compatible with hashlib
domain/canon.cljs the one canonical text for a descriptor, so hashing it means
something
domain/leaf.cljs leaf addressing: the document as path -> value
domain/wire.cljs transit for tier 1, base64 for tier 2
domain/project.cljs clip <-> the document and blocks that travel
flow/address.cljs tier-2 keys, and the invalidation table they are built from
fx/http.cljs the only namespace that talks to the server
events/project.cljs save and open
```
`clips` is a naming call, not a constraint — it is the Django app holding
Project, Clip, Footage, Analysis, Leaf and Revision. Rename in one line if
something fits better.
Dev runs two processes and they do not talk to each other: Django serves the page,
`shadow-cljs watch app` rebuilds into `static/arthur/js`, which is already
`:output-dir` in `shadow-cljs.edn`. `:dev-http` is gone.
```sh
mise exec -- python manage.py runserver 8778 # from the repo root
cd frontend && mise exec -- npx shadow-cljs watch app
```
## Toolchain
`mise install` from the repo root. `mise.toml` pins java 21+, node 20, clojure,
python 3.12, and creates `.venv`.
Verified to resolve cleanly: `reagent 1.2.0`, `re-frame 1.4.3`, current
shadow-cljs.
## Scope
**In:** analysis → keyframes → playback. The pure numeric core, the animation
data model, a player, the measurement stages, and freezing measurements into
channels.
**Out, and do not build it:** paint and cels; `suggest` (it only decides which
frames get a hand-drawn cel, so it has no job until drawing exists); the timeline
and sequences; symbols and the plate library; multiplayer; the override layer.
Each is designed for in `docs/architecture.md` and `docs/animation-model.md`.
Leave the `:over` field present and empty; leave `:symbol` out entirely.
## The data model
Full specification in `docs/animation-model.md`. The subset to build:
```clojure
;; The scene is a flat map of id -> node. Parent pointers, never nested maps.
{:id :mouth :kind :poly :parent :head :z "a3" :stencil nil :span [0 240]
:time {:mode :inherit} ; or {:mode :map :expose 2 :offset -1 :rate 1.0}
:channels
{[:xform :pos] {:animated? false :value [0.0 0.0]}
[:xform :rot] {:animated? false :value 0.0}
[:xform :scale] {:animated? false :value [1.0 1.0]}
[:xform :skew] {:animated? false :value [0.0 0.0]}
[:geom :pts] {:animated? true :interp :hold
:dense {:store "sha256:…" :offset 0 :stride 40 :frames 600}
:generated {:by :roto/lips-outer :analysis "sha256:…"
:params {:verts 8 :contour-avg 1}}
:over []}
[:style :color] {:animated? false :value :skin-dark}
[:vis] {:animated? false :value true}}}
```
Three channel shapes, one accessor `(value-at channel f)`:
- `{:animated? false :value v}` — static. A thing that simply exists.
- `{:animated? true :interp :hold :keys {0 v, 4 v}}` — sparse, authored, in the
document. **Keys are a map by frame, never a vector.** Store a plain map
(transit loses sortedness) and build the sorted index in the resolver.
- `{:animated? true :interp :hold :dense {...}}` — generated, one value per
frame, in a typed array outside app-db.
`:generated` is provenance and **the renderer never reads it.** It is what the UI
uses to offer a parameter panel instead of raw keys. It lives on the *channel*,
not the node, because a node wants a rotoscoped `[:geom :pts]` and a
hand-animated `[:xform :pos]` at the same time.
`:skew`, `:span` and `:over` stay in the shape even though nothing drives them
yet: each is a component of a decomposition or of a composition order, and adding
one later migrates every stored transform.
`:anchor` was in this list and has since been **deleted**, which is the one place
the reasoning above came out wrong. It is not a component of the decomposition:
`T(a)·M·T(-a)` is `M` conjugated by a translation, and a parent already is a
translated frame, so an anchor is a peg written inline — one that cannot be
selected, keyed, shared, or put above a measured channel. Rotation and scale
happen about the node's own origin; a pivot nobody chose is derived per drag by
`domain/gesture` and a pivot somebody chose is a peg. See
docs/animation-model.md, "There is no `:anchor`, because an anchor is a peg".
Transform composition, per node:
```
local = T(pos) · R(rot) · K(skew) · S(scale)
world = world(parent) · pinv · local
```
## What the prototype knows that you would otherwise rediscover
**The JS is a prototype.** Its conclusions about what looks right are provisional
and you may revisit any of them; several contradict each other already. But a few
things in it are not taste — they are facts about MediaPipe, about the maths, or
about what an operation means — and those cost real time to rediscover.
### Mechanical. Getting these wrong produces wrong output, not a different look.
1. **MediaPipe's normalised space is anisotropic.** It divides x by image *width*
and y by *height*, so equal numbers do not mean equal pixels. Multiply x by
`aspect = W/H` before any fit, or a "similarity" fitted in that space is not
one and head roll comes out subtly wrong. When mapping pixels for an underlay,
**both** axes divide by `imgH`.
2. **MediaPipe's left/right naming is viewer-relative in some places and
subject-relative in others.** Any left/right pairing read off a table is a coin
flip, and a swap looks *almost* right — each eye still has an iris roughly
where it belongs — so it survives inspection. Resolve it from geometry.
3. **Ring tables are ordered traversals**, and slot position is the vertex's
identity. That is what makes temporal correspondence possible at all, whatever
you decide the shapes should look like. `subsampleSlots` returns ring
*positions*, not landmark ids.
4. **A wrongly-ordered ring self-intersects, and it is invisible at odd vertex
budgets and obvious at even ones.** If you keep ordered rings, assert
simplicity in a test; no amount of looking will catch it reliably.
5. **Scaling a ring to thicken it collapses when the ring is degenerate** — a shut
eyelid scaled by 1.1 is still shut, so the lash line vanishes on exactly the
frames where it is the whole drawing. A fixed radial offset does not. Maths,
not taste.
6. **A fractional centre for a small integer-sized shape changes its size.**
Round the origin, not the extents, or a 3px mark is 3px on one frame and 4px on
the next.
7. **Order of operations on time:** flooring onto a grid and shifting against the
clock do not commute. Shift first and the floor discards it on most frames.
### Choices the prototype made. Revisit freely; here is what each was for.
| Choice | Its stated reason | How you would learn it was wrong |
| --- | --- | --- |
| similarity (4 DOF), not affine | extra DOF absorbs out-of-plane head rotation as shear and smears it into the mouth | the residual readout stops responding to head turn |
| reference is the Procrustes mean over the shot, not frame 0 | no single frame's idiosyncrasies get baked into every other | one frame's detection error biases the whole take |
| smooth the transform, not the contour | sparse keys at velocity minima rejected detector noise for free | it was already broken by a "bounded exception" once keys went dense, so it was never a law |
| gaze measured against the eye's corner midpoint | measured against the lid, every blink drags the origin down and fakes a glance at the floor | gaze correlates with blinks |
| one gaze shared by both eyes | at this size the per-eye difference is noise, and independent noise reads as wall-eyed | a wink or a real vergence is lost |
| hold, never interpolate | a tweened mouth reads as puppet software | motion looks stepped rather than snappy |
| palette indices, never sampled RGB | sampling colour produces a pixel-art filter irrecoverably | — |
These are where to look first if the output is wrong. They are also where to look
first if you want to change the look.
## Conventions
- `domain/*` may not require `flow/*`; neither may require `re-frame`.
- Every flow function is `(f params inputs) -> output`. No state, no db, no atoms.
- Nothing below `subs/` calls `subscribe`.
- Every analysis function that reads pixels takes a `debug?` flag and returns its
intermediate masks alongside its result, the way
`interior.js/extractTeeth(..., wantDebug)` already does.
- Port the invariant comments across verbatim. They are the most valuable text in
the repo.
## The oracle
**Keep `js/`, `index.html` and `serve.py` in the tree through step 5.** They cost
nothing, `serve.py` still runs the old tool, and they are the numeric oracle:
run both implementations on the same synthetic track and diff.
`fit-similarity` and `procrustes-mean` should agree to **1e-9**; a larger gap is a
port bug, not float noise.
**Parity proves the port is faithful, not that the answer is right.** The JS is a
prototype, so keep the two kinds of test apart: a *parity* test pins behaviour
while you move it, and is deleted once the move is done; a *correctness* test
asserts something you have decided you want, and stays. Conflating them bakes the
prototype's mistakes into the rewrite and makes them permanent. Delete them in one commit once the CLJS player renders
the synthetic take correctly.
**Do not port the debug views** (`drawPanes`, `drawInteriorDebug`,
`drawEyeOverlay`, `drawGazeDebug` in `js/app.js`). The knowledge in them is not
the canvas calls — it is *which things you must see to tune teeth*: the source
crop, the in-region mask, the surviving mask, and the local contour. That contract
already exists as `extractTeeth(..., wantDebug)` returning
`debugCanvas(src, inReg, mask, pw, ph, local)`. **Port the payload, skip the
drawing.** Redrawing it is ten lines whenever it is wanted.
## Steps
Each step ends somewhere runnable. Do not proceed past a step whose "done" does
not hold.
### 0 — scaffold and the oracle
`mise install`. Create `frontend/` with shadow-cljs, reagent, re-frame. Port
`synth.js` (the synthetic landmark generator, including its `swapIris` flag) and
the numeric assertions from `selftest.js` to `cljs.test`.
**Done:** the suite runs and fails informatively.
### 1 — the pure bottom
Port verbatim: `landmarks.js` → `domain/landmarks`, `mathutil.js` → `domain/geom`,
ring helpers → `domain/ring`, `raster.js` → `domain/raster`, the palette →
`domain/palette`.
**Done:** tests pass, including ring simplicity and the swapped-iris vote. Numeric
agreement with the JS to 1e-9. Nothing renders.
### 2 — the data model, with no analysis in it
`domain/channel` (`value-at` across all three shapes, plus a per-channel cursor),
`domain/node` (transform composition), `domain/scene` (topological order by parent
depth, `eval-frame` → draw ops in z order).
Hand-write a scene in EDN — a rectangle parented to a group whose
`[:xform :pos]` is keyed on four frames — and render it through `domain/raster`
into a canvas.
This is deliberately before any analysis. **The data model has never been
validated; find out here**, with fifty lines to throw away, rather than after
porting nine hundred lines of measurement into a shape that does not work.
**Done:** something moves on screen.
### 3 — the player
`clock` (audio-clocked: `frame = ⌊currentTime · fps⌋`, so a slow loop drops frames
instead of drifting; ½× and ¼× come free from `playbackRate`), the rAF loop, a
`::resolver` sub, and transport UI.
The loop reads and blits and **dispatches nothing**. The sub yields a resolver
closure; the loop applies it at the playhead. The playhead itself lives in app-db
like everything else — with layer-2 extractors and layer-3 computations, a
playhead tick does not invalidate the expensive stages.
**Done:** the hand-written scene plays at 30fps against audio, scrubs, and runs at
½× and ¼×.
### 4 — measure: anchor and mouth
Port `stabilize` and the lip rings out of `pipeline.js`. Split **condition**
(`smoothTransforms`, `smoothContours`) into its own stage so the two smoothing
knobs do not re-run measurement.
**Done:** measured numbers match the JS on the synthetic track. Note that
parity here is on `stabilize`'s output, not on `toRasterRing`'s — the framing
step is being deleted, not ported.
### 5 — freeze
The new module, and the heart of this work: measurements → channels. A dense
`[:geom :pts]` block per node — `Int16Array[frames × verts × 2]` in the node's
own local space, with the block's fixed-point scale in its header — plus
`:generated`. Fixed topology is what makes this a rectangular array with no
per-frame header.
**Do not port `makeXform`.** The prototype bakes the framing into the stored
numbers: it centres on the face oval's bbox and zooms until the face is 80% of
the raster height, so every vertex carries a cropping decision made once from one
frame's landmarks. Dropping it is a deletion. Placement becomes `[:xform :*]` on
an authored `:place` node inside the face, the stage clips whatever hangs off,
and project
dimensions stop being tied to the footage. See "What space geometry is in" in
`docs/animation-model.md`.
The anchor transform freezes onto `:head`, one level under `:place`, and the
normalise on/off/per-plate toggle is which of the three channel shapes that node
carries. Always measure and always store factored, whatever the toggle says:
smoothing and velocity-minimum key selection both require the split to exist in
storage.
**Done:** the synthetic take plays back as a moving mouth. Full vertical slice.
### 6 — detect
MediaPipe interop behind one namespace; real frames, real audio, real fps from
the manifest. **Vendor the wasm** rather than fetching from jsdelivr — it is
currently the only thing in the tool that silently requires a network.
Decode every source frame for analysis. A lower output picture fps is a time map
over frozen channels, not a reduced detection track. Selecting source frames to
trace into cels is independent again and remains outside this port's paint scope.
**Done:** real footage plays back as a rotoscoped mouth.
### 7 — the rest of measure
Eyes (openness, gaze, iris pairing vote, blink resolution with its `hold`), brows
(raise and tilt at both ends, both correspondence votes), interior (otsu,
morphology, components, radial contour). Each keeps its `debug?` payload.
Two things fall out of the model instead of being written: the brow's
measure-the-height-out-and-put-it-back is `[:geom :pts]` plus `[:xform :pos]`, two
channels on one node; and the iris is a `:disc` node parented to the lid ring and
stencilled by the sclera.
**Done:** the same face parts are measured and rendered through the CLJS scene,
minus paint. The fixed pixel thresholds remain provisional; step 8 exposes their
parameters for tuning without changing the source track or picture timing.
### 8 — knobs — DONE for static settings and scoped regeneration
Build the parameter model before its UI. Define each parameter once with its
default, validation, applicable area and regeneration dependencies. Store values
by stable subject and feature ID. Represent an eye pair as one group with one or
two eye member IDs from the same subject; a profile view with one identified eye
needs no invented partner. Each eye may override a pair value. Removing an eye
from a pair materialises its effective values so playback does not change. Keep the
existing frozen channels as renderer input; settings and provenance do not enter
the render path.
Carry feature-level presence through freeze and dense channel state. The same
feature ID covers every observed run across occlusion; a missing measurement
has no channel value on that frame. Full-face detection is the fallback mask
until there is a feature-level detector or authored presence data. A shared gaze
measurement may still feed two independently identified eyes. A small manifest
annotation can supply feature absence intervals now: the loader expands them
before measurement, so invalid eye landmarks are ignored and gaze uses the
visible eye. This is an input format, not a control UI or an automatic detector.
Use leaf-addressable settings under the clip, subject, feature and optional
group. Retain source measurements so a setting change can regenerate affected
channels without re-detecting footage. Static parameter controls and scoped
regeneration are implemented for takes and composed stages. Time-varying
parameter values remain deferred.
### 9 — backend — DONE
Django project, the `clips` app, models for
Project/Clip/Footage/FootageFrame/Analysis/Block/Leaf/Revision/Blob, and project
load/save. Round-tripping a project through the server is the proof the model
serialises.
Django was the easy half; the tier split was the work. What it came to:
**Tier 2 keys are content addresses over every input**, and the detector version
is in every one of them, through the analysis id that each block descriptor names.
`flow/address`'s `block-knobs` is the invalidation table, and it is not trusted:
`address-test` re-freezes the take once per knob and asserts the biconditional —
a block's bytes changed if and only if its key changed. That test found two
things reading the code would not have. `brow-pos` does not depend on
`contour-avg`, because the brow RING is smoothed and the raise is not. And the
first version of the test was itself wrong: a 3% perturbation of `gaze-gain`
moves every sample inside the grid cell `quantize-snap` had already rounded it
into, so the bytes came out identical and the knob looked like an input the block
did not have.
**The server verifies what it is handed.** It recomputes every key from the
descriptor stored beside it and refuses a mismatch, refuses an analysis that does
not declare a detector version, and refuses a document naming blocks it does not
hold. It hashes the descriptor TEXT rather than re-rendering it from parsed
values, because JS prints an integral double as `1` and Python as `1.0` — a
scheme where both sides re-render breaks on the first parameter whose value
happens to be whole.
**Tier 3 is served by hash.** `extract.sh` still decodes; `manage.py
ingest_bundle` hashes the result into the blob store, by hard link. The manifest
the client receives now carries a URL per frame, so the frame layout stopped being
a shared secret between a shell script and a ClojureScript namespace. The
cache-busting `?v=` on every frame URL went with it: a blob's name is the hash of
its bytes, so a stale copy is not a thing that can happen.
**Leaf addressing exists**, with conditional writes and a monotonic project
version, so the sync design has nothing to retrofit. The socket, presence and
broadcasts are still out of scope.
Two loose ends from step 8 closed on the way. `pack` no longer takes a
`(track, frame)` predicate whose call sites each derived a feature from an index —
every track names the feature it follows, which deleted five hand-maintained
mappings and handed `flow/address` the same list for its observation digest. And
the `presence-check` binding in a `let` nobody read is now an ordinary `doseq`.
**Output is not in this plan.** The `.take` writer in `js/take.js` was for
driving an Animator Pro render script and it is not where this is going: the
target is encoding video in the browser, and that is a separate piece of design
nobody should pre-empt by porting the old one.
## Two things to not foreclose
Feature controls now handle more than one face; editing presence remains future work.
The underlying identity, occlusion and group association model begins in step 8:
- **Presence is not visibility.** An occluded feature has *no value* on a frame,
which is different from a part being hidden. Dense blocks carry a per-track,
per-frame absence mask; `[:vis]` remains the sole hiding mechanism.
- **Params carry stable identity.** A subject and its features keep their IDs
across observation gaps. A run of visible frames is not a new identity.
The current identity tracker uses nearest-centroid assignment. Validate it on
real crossings and disappearances before choosing a more elaborate policy; the
iris and brow correspondence code offers whole-take voting as one option.

41
docs/time.md
View file

@ -1,41 +0,0 @@
# Time selection
Project `:fps` is the playback and export grid. Each symbol has its own native
`:fps` and `:frames`; keys, spans, trace choices and corrections stay in that
native space. A symbol without an explicit rate inherits the document rate;
changing project fps first records that rate so its existing timing stays put.
The untouched symbol in a new document is deliberately different: it has no
authored timing to preserve, so it stays on the project grid and its empty frame
extent is rescaled to keep the same duration. This makes changing fps before
authoring establish the editor's grid instead of preserving the 30fps default.
An output frame selects the latest native frame at or before its time:
`floor(output-frame * native-fps / output-fps)`. Thus 30fps content in a 12fps
project reads source frames 0, 2, 5, 7, 10… and retains its duration. A partial
last output frame is included. Changing back to 30 restores the original grid.
Nothing rewrites or discards the dense measurements.
The same boundary selection runs when entering a placed symbol. Placement and
artistic speed are applied before selection; the stored `:time :rate` and
`:playback :speed` never contain a frame-rate conversion. The derived maps used
by timeline rows, picking and editing account for the units of each symbol.
`clip/frames` is a native length; `clip/output-frames` is a transport/export
length. Resolver frame queries return native frames for edits.
There is one fps control. The old transient picture-fps control and node
sample-fps fields are gone. Existing exposure, trace choices and per-instance
pose tracks remain available: a pose track can hold a chosen closed-mouth frame
without deleting its neighboring measurements. Those choices stay in native
frames when output fps changes. Automatic content-aware frame selection is not
implemented; [frame-selection.md](frame-selection.md) is how it should be. An
event between output frames appears on the next output frame; it cannot create
an extra frame in a 12fps output.
Audio uses continuous time through the same derived placement maps, without
picture floors or holds. Frame-rate units cancel before Web Audio playbackRate
is set, so only deliberate speed changes affect pitch and duration. Export and
playback use the same output count and resolver.
Earlier imports with frame-rate conversion baked into stored retimes must be
re-imported. There is no second reader for that representation. Source video
presentation timestamps are still future work; this model assumes constant fps.

128
docs/timing-handoff.md
View file

@ -1,128 +0,0 @@
# Timing and frame-selection handoff
Status (2026-09-29): the multi-face representation is complete. Each face has a
local drawing timeline and an ordinary symbol instance. Keep that model; the next
feature is performance-pose selection, followed by plate drawings and tracing.
See [multi-face representation](multi-face-representation.md) for verification
and compatibility limits.
## Next steps, in order
1. **Commit the verified checkpoint.** Representation, scoped regeneration,
nested stage composition and source persistence are implemented and tested.
2. **Exercise real two-person footage.** Include crossings, late arrivals and
disappearances. Assignment is still a nearest-centroid heuristic; inspect
whether identities, landmarks and mouth crops stay together. Correcting an
assignment requires measuring again. Do not redesign the representation to
compensate for an assignment failure.
3. **Build performance-pose selection.** Propose frames from a target picture
rate, allow explicit Keep/Drop edits, and apply requests per instance. Reuse
the existing held-frame lookup and generated pose groups. Keep authored keys
and audio timing intact.
4. **Then build plate drawings and tracing.** Suggest drawing frames from head
displacement, allow manual choices, and give each cel an independently
selectable tracing reference.
Older flat captures need reanalysis for the new regeneration path. Migrating their
existing authored edits is separate work; it is not implemented by this checkpoint.
## Timing decisions
Keep the dense analyzed frames. Generated motion holds the most recent selected
source pose; removing a selected pose never deletes source data or shortens the
clip. Store edits in the animation's local frame space, so moving an instance
does not move its edits. Authored keys follow intentional instance retiming but
must not be quantized by a picture-rate request. Clip FPS and audio duration stay
fixed.
There are two selections with different owners, sharing held-frame lookup:
- **Performance poses:** propose a kept-frame list from the target picture rate,
then apply explicit keep/drop edits. A parent instance may request a lower
rate. Mouth outline, interior, teeth and visibility read the same selected
source frame; likewise each eye's coupled parts. Use group overrides when
needed, rather than a setting on every channel. Head motion currently has its
own anchor selection; do not silently put it under mouth timing.
- **Plate drawings:** start with frame 0, walk measured rigid head poses, and
suggest a frame when maximum landmark displacement from the last kept pose
exceeds tolerance. Let the artist add/remove frames. A removed drawing stays
stored so it can reappear if restored. This selection does not thin the mouth.
A target rate is approximate. Pin a useful closed-mouth pose at its actual frame,
even if that produces more changes than the target. Do not show a future pose
early to fit a grid. Manual drop wins over an automatic suggestion; make removal
of the only closed pose in a beat visible in the UI. Skip missing detections when
suggesting a replacement. Keep a frame-zero selection and hold the last selection
through the end. A skipped pose (hold), `[:vis] false` (hidden), and an absent
measurement remain different facts.
Store manual edits separately from generated proposals so changing the rate or
tolerance retains hand decisions. Selection edits change the document, not dense
blocks or analysis addresses. Verify save/open for every new field; extend leaf
handling and the relevant key whitelist if its storage location requires it.
## Current code: reuse these mechanisms
- `domain/pose.cljs` already has `prepare`, `held-frame` and `source-frame`.
Instance `:playback :tracks` map local change frames to held source frames,
keyed by pose group. Reuse this lookup; frame suggestion and Keep/Drop policy
are the missing layer. An explicit cut is not itself a complete selection UI.
- `freeze/performance-nodes` marks generated animated channels with
`:pose-sampled?` and local `:pose-group` names. This includes keyed visibility
as well as dense geometry. `:generated` remains provenance for regeneration.
- `symbol/channel-frame` already applies explicit pose choices and default
picture sampling to marked channels. Playback and export both use
`clip/resolver` with `:picture-fps`; there is no need for a second sampling
implementation. Export's pose count is still a rate-based estimate.
- The picture-rate option is currently passed through the resolver tree
unchanged. Instance-specific parent requests are still to be implemented.
Instance offset/rate must apply before selecting the local source pose.
- `pose/put-cut` and `remove-cut` currently address instances in `:main`.
A take's face instances are there, but a composed stage nests them inside a
shared source timeline. Make the editing scope explicit when adding nested
controls. A request on one outer placement must not rewrite the shared
drawing's playback settings for every placement.
- Generic root `:time :expose` still retimes descendants, and frozen takes still
store it. Paint nodes are rootless to escape it. When the selection path
replaces take picture cadence, remove that redundant quantization from the
take default; preserve intentional generic time maps. Moving exposure to
`:head` would still retime authored children.
- `freeze/head-mode` supports `:free` and `:anchored`. It keeps measured channels
dense and writes optional per-subject `:anchors` maps; it does **not** implement
`:per-plate` mode or materialize transform keys from `:kept`. Plate selection
should reuse held measured-frame addresses where appropriate, without
rerunning analysis or copying the measurements.
- `:over` hand corrections are currently refused by the channel reader. Their
future application belongs after generated pose selection.
## Performance-pose implementation sequence
1. Add pure proposal and Keep/Drop policy around the existing held-frame lookup.
Cover frame zero, nondivisible rates, manual precedence, missing poses and a
protected mouth closure. Preserve all source frames.
2. Feed instance requests and group selections into the existing channel read
path. Cover two faces, two differently timed placements of one source, nested
instances, coupled visibility/geometry, and authored keys at their normal time.
Use this same path for preview and export; keep audio duration unchanged.
3. Wire the performance strip's Suggest/Keep/Drop controls and persistence.
Replace the export pose estimate with the actual selection count. Retire the
take's redundant root exposure only when this path replaces its behavior.
## Plate drawings and tracing, afterward
The old suggestion algorithm is `js/pipeline.js:suggestPlateFrames`; the strip,
worksheet and tracing photo are in `js/app.js`. Port the useful policy over the
measured head poses and reuse held-frame lookup for the resulting drawing set.
Give a cel an editor-only source-frame reference, defaulting to its plate frame
but independently changeable. It may point to a frame omitted from either rendered
selection. Register the photo using that source frame's measured transform. The
old prototype coupled photo and cel addresses; independent tracing is new work.
The old iris socket lock, gaze origin, CLJS head anchors and registration pivot
are separate settings. Clarify what an "origin-lock" request means before adding
that control.
Keep the UI to two scopes: **performance poses** and **plate drawings**, each with
Suggest/Keep/Drop. Tracing reference and lock controls live with the cel or feature
they affect. No general keyframe framework is needed for this work.

93
docs/timing-model.md
View file

@ -1,93 +0,0 @@
# Timing model
[Time selection](time.md) defines the current frame-rate representation.
[The Lane Model](lane-model.md) defines the revised target for occurrence timing,
source playback, sampling scope, and inverse editing. It supersedes conflicting
proposals here; the sections below describe earlier implementation decisions.
The source footage, authored drawings, generated face motion, and stage placement
have different frame decisions. They share a clock but do not share one kept-frame
list. `timing-handoff.md` records earlier implementation notes.
## Frame spaces
- A source frame addresses a decoded image and its measured face data. Keep the
source cadence and, for variable-rate video, its presentation timestamp.
- A timeline frame addresses authored keys in the clip or symbol's local space.
- A stage frame is mapped through the symbol instance's offset and rate before
local frame decisions are read. Moving a placement does not rewrite its keys.
The analyzed source poses remain dense. A lower picture rate or a skipped pose
never removes source data or shortens audio.
## Head placement
Analysis fits each source frame's rigid landmarks into one common head-local
space. Its inverse is the measured head transform, stored densely on `:head`.
The head node has one optional anchor map:
```clojure
;; no :anchors free movement: read measured frame f at f
:anchors {0 12} ; one lock: use frame 12's transform throughout
:anchors {0 12, 40 42} ; keyed locks: switch to frame 42 at local frame 40
```
A key is `(local change frame -> measured source frame)`. Its value holds to the
next key. The map chooses position, rotation and scale together. Frame zero must
have a key when the map exists. The dense transform blocks remain intact, so
editing anchors is a small document change and re-freezing can replace the
measurements without losing the anchor choices.
A source image used for tracing should be registered with that image's measured
stabilizing transform, then the selected head transform, then the authored
`:place` placement the face carries. This makes the photo and head-local vectors share the same
orientation and position. Tracing-photo selection is a separate editor address;
it does not choose the head anchor.
The prototype stabilizes into the shot's mean rigid pose and uses an early
closed-mouth frame for raster framing. Those are internal analysis and framing
choices. The authored head-anchor map above controls which measured head pose is
shown over each range. It is independent of plate drawing starts.
## Performance poses
Generated mouth, eye, and brow channels can be sampled at a lower picture rate
without retiming authored keys. The normal rule picks the latest available pose
at or before a picture-grid time. A future performance policy may add important
closed-mouth poses and store manual keeps/drops separately from the rate's
proposal. Related parts should share a selected pose by default: a mouth outline,
interior, teeth and generated visibility must not disagree about its frame.
## Stage placement
A symbol placement has optional pose-cut tracks, separate from head anchors:
```clojure
:playback {:tracks {:mouth {0 12, 8 27}
:eye-r {0 0, 4 6}}}
```
These maps are also `(local change frame -> source pose frame)`. They select which
baked/generated shape pose appears on that placement. Before the first explicit
cut, normal generated motion continues. Cuts hold, without interpolation, until
the next cut. A `[:node id]` track can override one shape in a shared group.
Authored cels, transforms, and audio remain on their normal local time.
The current implementation reads retained frozen channels. A separate resolved
geometry bake is not implemented; when added, it must preserve addressable
candidate poses so stage cuts can still select any of them.
## Ownership
| Choice | Owner | Current state |
| --- | --- | --- |
| Source frames and timestamps | Footage/analysis | Constant-rate frame indexing exists; variable timestamps remain future work |
| Head anchor map | `:head` node | Implemented, stored with the node |
| Trace frames (photo address) and origin | The face's `:plate` `:time :holds`, and its `:head` `:reads` | Implemented, see `docs/tracing-symbol-plan.md` |
| Showing a tracing layer, and its opacity | Editor state, `[:ui :tracing]` | Implemented; a drawing aid, never saved or keyed |
| Generated picture-rate proposal and closure protection | Roto clip/symbol | Generated-only picture sampling exists; closure protection remains future work |
| Stage pose cuts | Symbol instance | Implemented, stored with the instance |
Preview and export use the same resolver for generated picture sampling and stage
cuts. Export still emits every timeline frame at the clip's audio rate.

411
docs/tracing-symbol-plan.md
View file

@ -1,411 +0,0 @@
# Plan: tracing is a symbol
Status: built, 2026-10-03, on branch `worktree-tracing-layers`. Where the build
differs from the plan below, the build wins:
- The head's field is `:reads` (`{:holds [...]}` or `{:holds-of :plate}`), not
`:follow`.
- A still is not one frame. It gets as many frames as remain in the symbol it is
dropped into, at that symbol's rate, and is trimmed like any clip.
- An image's `:media` is `{:image <blob sha256>}`, served at `/blob/<sha>`. The
server's `Image` row exists for the pool's list and labels, not for identity.
- Dropping media that a tracing symbol in the document already shows reuses that
symbol.
- Nothing can be created or dropped inside a tracing symbol (`creation/target`,
`drop-destination-at`, `nest/move-refusal`), and one cannot be opened in a tab.
- Schema 6. Every project is marked 6. One that still carries `:trace` on a
node is refused when opened, with what to do about it, rather than all old
projects being refused.
No backward compatibility (see `lane-model.md`, "Goal and compatibility policy").
## What is wrong today
Tracing is not a thing in the document. It is three mechanisms that each know
about faces:
- **`:trace {:frames :origin}` on a face's `:head`.** It decides which measured
frame the head reads (`symbol/base-channel-frame` → `trace/held-frame`), and
the underlay also reads it to decide which photo to show (`trace/photo-frame`).
- **`ui/underlay`**, a painter that walks `trace/shown` → `trace/faces` (a
separate instance walk), looks up each face's subject → analysis → footage,
asks the resolver where `[...path :head]` went, and builds the photo's matrix
by hand: `world(head) · M(p)⁻¹ · 1/imageH` (`trace/photo-matrix`).
- **`[:ui :trace {:faces #{} :opacity}]`**, a per-face switch in editor state,
plus the special "opening a face shows its footage, opening a take does not"
rule (`trace/showing-for`).
What you cannot do: place footage or a still as a reference where you like, move
it, scale it, turn it, trim it, hold it, put it in a lane, or trace something
that is not a tracked face. The photo is not selectable and has no row. The one
case that works (a face over its own footage) runs on code that nothing else
uses.
## The model in one paragraph
A **tracing symbol** is a symbol with `:type :trace`. It has no nodes. It names
media (a footage range or a still image) and has that media's frame count, fps and
pixel size. It is **placed by an ordinary instance**, so lanes, spans, trim,
split, move, playback (`:in :speed :end`), transform, nesting, selection, picking
and gestures all work on it with no new code. It is evaluated to a single
**`:trace` op**, which never reaches the raster or an export. A face's footage is
not a special case. It is one of these instances, placed inside the face as a
child of `:head` and carrying the measured registration transform. The trace keys
become **holds on that instance's time**. The head's "origin" becomes the head
saying **which node's frames it follows**.
This follows the precedent of `:type :palette` symbols (no nodes, they name an
asset, and they are placed by instances in a lane), so the uniformity rule holds
(see the `model-uniformity` memory). Nothing new holds nodes, and no special
instance kind is added.
## Data model
### The tracing symbol
```clojure
:footage-8625 ; a symbol id like any other
{:id :footage-8625 :name "8625.mov"
:type :trace
:media {:footage #uuid "f8ca…" :range [12 241]} ; or {:image "sha256:…"}
:frames 229 ; the range's length; 1 for a still
:fps 30 ; the footage's own rate, so cadence handles 30→12 for free
:width 1440 :height 1920 ; pixel size = the symbol's own stage
:audio {:footage #uuid "f8ca…"} ; optional, the existing "a symbol says what it sounds like"
:nodes {}}
```
- `:media` is the one new symbol key. Add it to `symbol/symbol-keys` and to the
symbol leaf's `select-keys` in `leaf/leaves`.
- The symbol's local space is **pixels of the media**: `[0 w) × [0 h)`.
`clip/center` of a node-less symbol already returns its stage middle, which here
is `[w/2 h/2]`. So `place-symbol` puts the anchor at the image centre with no
new code.
- `symbol/problems`: `:type :trace` needs `:media` with exactly one of
`:footage`/`:image`, needs `(empty? nodes)`, and for footage needs
`(= frames (- end start))`.
- **One tracing symbol per footage range, shared.** Two faces from one take
place the same symbol, and so does a hand-placed reference. Like any symbol it
is reused by reference, and `bring/symbols` copies it like any other.
### Placing it
It is an ordinary `:kind :instance` with `:source {:symbol :footage-8625}`:
- **Start and end** are `:span` (in its own frames) and `:time :at`. You get
these from the existing trim, split, move and roll.
- **Which frame shows** comes from `:playback {:in :speed :end}`. Footage plays
with speed 1, a frozen frame has speed 0, and a still is 1 frame with `:end :hold`.
- **Placement in space** is `[:xform …]`, the same as every node: gestures,
inspector and keys.
- **In a lane, or on its own**: it is a parent-less spanned node, so a `:display
:lane` symbol draws it as a block like any cel. It can also sit as a free node
or under a group. To have a reference with its own tab, wrap it in an ordinary
symbol.
**Default on creation** (in the drop event, not in `place-symbol`): scale so the
image's height fits the stage height, centred on the drop point. This is a
creation default like "use center anchor when dropping", and nothing updates it
afterwards.
### Holds: the one new time feature
`:time {:holds [0 12 30]}` floors a node's local frame to the last hold at or
before it. Before the first hold, the first hold applies. This is `:expose`
generalized from a regular grid to authored frames. It is applied in
`node/local-frame` at the same point as `:expose`, and is inherited in the same
way. It is in the node's **own** frames. On a tracing instance with
`:in 0 :speed 1`, own frames are source frames, so the hold list is the set of
traced frames.
It is general on purpose. Holding a playing symbol on chosen drawings is the
same feature. It costs about three lines in `local-frame`, one `problems` clause
(sorted, distinct, finite), and the timeline drawing hold frames as marks on the
row.
`symbol/frame-map` refuses floors (`:expose > 1`). It must also refuse a
non-empty `:holds`, for the same reason.
## The face: "a child symbol that represents the trace"
The face symbol after a freeze:
```
:place group authored source→stage mapping (image heights → stage px)
:head group measured M(p) :reads — see below
:mouth … parts generated, every frame
:plate instance of :footage-8625 ← the trace
measured channels: fit(p) = M(p)⁻¹ · S(1/imageH)
:time {:holds [0 12 30]} ← the trace keys
```
### Registration comes from the parent
The plate is a child of `:head`. Its own measured channels are the
**stabilizing fit** at frame p: the inverse of the head's measured transform,
with pixels → image heights folded into the scale (it stays a similarity, so it
decomposes into `pos`/`rot`/`scale`). Freeze already computes the fit; the
head's measured channels are its inverse (`freeze/invert`). The plate gets a
second dense block, which is three small channels.
Because holds are a **time** floor, the plate's channels and the frame its
content shows are read at the **same** held frame q. Its world transform is:
```
world(plate) = place · M(p_head) · M(q)⁻¹ · S(1/H)
```
That is exactly `trace/photo-matrix`, but now it falls out of the ordinary walk.
It is registered to whatever the head is doing, by construction:
| head reads (p_head) | plate shows (q) | result |
| --- | --- | --- |
| f (continuous) | f (no holds) | footage where filmed |
| f (continuous) | held key | held photo rides the moving head (today's behaviour) |
| held key (same as plate) | held key | `M(q)·M(q)⁻¹ = I`: photo sits where filmed |
| 0 (start) | f or held | stabilized footage under a still head |
If a frame has no measurement, the plate's dense channel has nothing there, so
`xform-at` gives nil, so the plate is not placed and no photo shows. That is what
happens today too.
### Trace keys versus origin: who owns what
The two decisions are separate and stay separate:
- **Trace keys** are which footage frames get drawn over (the "plate drawings"
in `frame-selection.md`). They belong to the **plate**, as `:time :holds`. A
hand-placed tracing layer with holds is the *same thing*: the face's plate is
an ordinary tracing placement and nothing more.
- **Origin** is how the head moves between kept frames. `frame-selection.md`
already says `:origin` "is not a tracing setting" but a performance one, so it
belongs to the **head**:
```clojure
:head {…} ; continuous — reads its own frame
:head {… :reads {:holds [0]}} ; start
:head {… :reads {:holds-of :plate}} ; at keys — reads its measured channels at
; the frames :plate's holds select
```
`{:holds [...]}` is also what a face with no footage uses for "at keys", since it
has no plate to follow.
`:reads` changes only the **head's own channel reads**, not its children's
frames. The parts must keep running every frame, which is why this cannot be a
`:time` hold on the head. It is today's `traces` branch of
`base-channel-frame` with the hold list read from the named node. A node
reference has precedent (`:stencil`, `:pose-group`). It points from the follower
to the thing followed, so there is still one stored list of frames and nothing to
keep in sync. Validate it in `symbol/problems` the way `:stencil` is validated:
the target exists in the symbol, and it is not the head itself or an ancestor of
the head.
Rejected alternatives, so they are not re-proposed:
- **Keys on the head, and the plate reads them.** This is today's direction. It
makes the plate special: a free tracing layer could not have keys that a face's
plate also understands.
- **Hold `:time` on `:head`.** Exposure inherits strictly, so the mouth and eyes
would freeze along with the head.
- **A wrapper "registered footage" symbol holding the fit.** It is correct but
adds a symbol per face. The time-floor holds already put the fit read and the
content read on the same frame, so the wrapper buys nothing.
- **Plate as a sibling of `:head` at identity.** It is only registered when the
head and the photo read the same frame, so it breaks the continuous-plus-holds
and start rows above.
## Evaluation: an op that is never rendered
- **`clip/resolver`**, in the instance branch: when the source symbol has
`:type :trace`, it does not recurse. If `(:tracing? opts)` is set, it emits one op:
```clojure
{:kind :trace :node [id] :m <copy of world> :media … :frame shown-frame :size [w h]}
```
The frame comes from the same `placed-frame` path as any instance, so playback,
holds and the fps cadence apply. Do not build a child resolver for a trace
symbol.
- **`transform-op`** gets a `:trace` case that composes the matrix. Row paths,
solo filtering and nesting at any depth then work for free.
- **The output guarantee is structural.** `:tracing?` defaults to false. Only the
stage's `::render/resolver` passes true. Export, `clip/center` (a big photo
must not pull a symbol's pivot), thumbnails and the bench never ask for trace
ops. `raster/draw-ops!` keeps throwing on unknown kinds, so a leak fails loudly.
- **`ui/player`** sends picture ops to the raster and `:trace` ops to the
painter.
- **`ui/underlay` becomes `ui/tracing`.** It paints `:trace` ops in draw order
with `drawImage` at `op.m` and the global opacity. The media URL is the footage
manifest's `urls[range-start + frame]`, or the image blob URL. The three
steadiness fixes stay: LRU cache, hold the last still per op `:node`, and read
ahead while playing. Everything that walked faces is deleted.
- **`pick`**: a `:trace` op is hit when the point, mapped through `m⁻¹`, falls in
`[0 w) × [0 h)`. Picture ops are tested first and trace ops only if nothing
drawn is under the pointer, so a full-frame photo does not steal every click.
When the global switch is off, traces are neither painted nor picked.
- **Gestures**: no change. The face's plate is measured, so `gesture/refusal`
already says "place the instance it is in". A hand-placed layer is authored and
moves, turns and scales like anything else.
## On and off
All of it is EDITOR STATE, as ed88c5e decided for the per-face switch: showing
a reference is a way of looking at the stage, so it is not an undo step, does not
travel to collaborators, and cannot reach an export.
```clojure
[:ui :tracing {:on? true :opacity 0.5 :hidden #{[sid node-id] …}}]
```
- **One layer** is an entry in `:hidden`, keyed by the symbol the tracing
instance is in and its node id. The symbol id is needed because every face's
plate is called `:plate`. Keyed this way, hiding a face's plate hides it in
every placement of that face, which is what the per-face switch did. Shown is
the default, so opening a face shows its footage with no setup.
- **How it's applied**: `clip/resolver` already knows which symbol and node a
trace op comes from, so the op carries `:layer [sid id]`. The painter and
`pick` skip ops whose layer is hidden. The resolver does not change when you
toggle a layer, so nothing is rebuilt.
- **Global**: `:on?` and `:opacity`, a toggle plus an opacity slider in
`ui/palette/bar` next to the palette controls. When it is off, traces are
neither painted nor picked.
- **Switching one layer on also switches the global setting on.** This applies
from the tracing instance's inspector, from its timeline row, and from the
face/roto section. A layer you just enabled must not stay invisible behind a
switch you forgot. Turning one layer off never touches the global switch. It is
one event (`::ui/show-trace layer on?`) that updates `:hidden` and, when
turning on, sets `:on? true` in the same handler, so the three places cannot
behave differently.
- **`[:vis]`** still works on a tracing instance like on any node: key it to
show a reference only over part of the shot. It is not the on/off switch.
- **Deleted**: `[:ui :trace :faces]`, `::trace-face`, `::trace-faces`,
`trace/showing-for`, `traceable-faces`, `shown`, `faces`, and the "a take shows
nothing by default" rule.
## UI
- **Timeline.** A tracing instance is an ordinary row or clip block, styled to
read as reference-only (hatched block, an eye icon in place of the colour
chip). Hold frames are marks on the row. The row's eye button dispatches
`::ui/show-trace`, which replaces the face row's `tl-trace` button.
- **Inspector, tracing instance.** Show the media (footage name and range, or
image), an on/off control (which goes through `::ui/show-trace`), and the hold
list ("hold here" / "remove hold" plus seek buttons, which is `trace-keys`
re-aimed at `:time :holds`). Playback, transform and span use the existing
sections.
- **Inspector, face/roto section.** The same hold controls, aimed at the face's
`:plate`. Origin buttons write `:head :reads`. The section is found by "this
face has a `:plate`", not by `trace/traceable?`.
- **Palette bar.** The global toggle and opacity slider.
- **Making one.**
- Footage: the convert dialog gets a choice between "animate faces" (today's
flow) and "tracing layer" (no detection). The tracing-layer choice makes the
`:type :trace` symbol for the chosen range and places it where the video was
dropped.
- Footage can also be dragged from the pool with a modifier or as a second
drag kind.
- A still image: drop it on the pool and it uploads; drop it on the stage or
timeline and it is placed with `:end :hold` and a span of the host's
remaining frames.
## Freeze, bring, regenerate
- **`freeze/subject-part`** emits the `:plate` instance under `:head`, with fit
measured channels and `:time {:holds []}`, and emits one `:type :trace` symbol
for the analysed footage range. `bring/take` already copies every symbol the
take reaches and rewrites `:source :symbol` references, so the tracing symbol
comes along. It gets the footage's `:audio` too, so a dropped tracing layer
can bring its sound through the existing link.
- **`head-mode`** writes `:reads` on the head and `:holds` on the plate, in
place of `:trace`. The `frame-selection.md` plate proposal materializes into the
plate's `:time :holds` in place of `:trace :frames`.
- **Regenerate** replaces both measured blocks (head and plate) and keeps
`:holds` and `:reads`. Check that `regenerate-head`'s measured/authored
comparison handles a second measured node.
## Server
- `Image` endpoints mirroring `Sound`: `POST /api/images` stores a `Blob` and
returns `{id, width, height, url}`, and `GET /api/images` lists them. The
migration is a model only. Bump `schema_version` and refuse older documents
clearly. Do not convert them.
## Deleted outright
`trace/of`, `prepare`, `held-frame`, `problems`, `toggle-frame`, `photo-frame`,
`measured-local`, `photo-matrix`, `traceable?`, `faces`, `traceable-faces`,
`showing-for`, `shown`, `opacity-default`. `domain/trace.cljs` probably goes
away entirely; if anything is left, it is a hold helper that belongs in `node`.
Also deleted: `symbol/prepared-traces` and the `traces` arm of
`base-channel-frame`, which becomes the `:reads` lookup. `:trace` on nodes (and
`symbol/problems` reports it as removed). `::project/set-trace`.
`::render/underlay` and `::render/tracing` become one `::render/tracing` that
returns `[:ui :tracing]`. The face lookup in `params/view`.
Expected effect on code size: net negative. One time-floor clause, one op kind,
one resolver branch, one pick case and a `:reads` lookup replace the face walk,
the hand-built photo matrix, per-face showing state and the underlay's face
bookkeeping. Measure the change and report the number honestly (see the
`cljs-style` memory).
## Tests
**Domain (node).**
- `:holds` in `local-frame`, with eval-frame and resolver agreeing forward,
backward and in random order.
- A trace op appears only with `:tracing?`, and `export/run!` output contains no
`:trace` op.
- `transform-op` composes the matrix through two nesting levels.
- `clip/center` ignores traces, and is `[w/2 h/2]` for a tracing symbol.
- `pick` returns picture ops before traces and inverse-maps through a rotated
layer.
- `symbol/problems` covers `:type :trace`, `:reads` targets and cycles, and
`:holds` shape.
- Registration: port `trace_test`'s photo-matrix assertions onto the walk.
For every row of the registration table, the plate's world transform equals
the expected matrix. At a held key with `:reads {:holds-of :plate}`, it equals
`world(:place) · S(1/H)`.
- Freeze emits the plate and the tracing symbol. Regeneration keeps the holds.
- The `::ui/show-trace` event sets the global switch on when turning a layer on
and leaves it alone when turning one off.
**Browser** (CDP, see the `arthur-verify-dont-guess` memory):
- Drop a still, then drag, turn and scale it.
- Toggle a layer on with the global switch off, and confirm both are on and the
photo paints.
- Export a frame and check it has no photo pixels.
- Open a face: the plate is registered at a hold with origin "at keys", and
stabilized with origin "start".
## Order of work, each step green
1. `:time :holds` in `node/local-frame`, `problems`, `frame-map`, and timeline
marks.
2. `:type :trace` symbol, `:media`, the `:trace` op behind `:tracing?`,
`transform-op`, the player split, `ui/tracing` painter and `pick`. Footage
media only, placed by hand from a REPL or test document.
3. The face: freeze and bring emit the plate and tracing symbol, `:reads`
replaces `:trace`, delete the old trace and underlay paths, and re-aim the
inspector's face section.
4. On/off: the `:hidden` set and `:layer` on trace ops, the row eye, the
`::ui/show-trace` rule, the global toggle and opacity in the palette bar, and
delete `[:ui :trace :faces]`.
5. Creation: "tracing layer" in the convert dialog, pool drag, then the image
endpoint, pool images and image drops.
6. Docs: `animation-model.md` "A photographic underlay is not an op" becomes "a
trace is an op that never reaches the raster". Update `frame-selection.md`
(where plate keys live) and `lane-model.md` (tracing clips in lanes).
## Open questions
1. **"A take shows its faces' footage."** Dropping the old "only in a face's own
tab" default means opening a take shows every face's plate if the global
switch is on. The recommendation is to accept that, since the switch is one
click.
2. **Lane-model audio rule.** `lane-problems` requires visual cels. A tracing cel
is visual for this purpose, but say so explicitly when a lane may hold both
tracing and drawing cels.

83
extract.sh
View file

@ -7,77 +7,32 @@
# pass. A PNG sequence is exact, instantly seekable, and reproducible.
#
# Audio comes out alongside because the page uses it as the PLAYBACK CLOCK -
# frame = floor(audio.currentTime * fps). Detection sees every decoded source
# frame; a lower drawing rate is a later playback choice, never an extraction
# choice. This script accepts CFR footage because frame-index timing needs a
# single rate. VFR needs per-frame timestamps in the manifest first.
# frame = floor(audio.currentTime * fps) - so picture and sound cannot drift
# apart no matter how long the shot is or how slow the render loop runs.
set -euo pipefail
src="${1:?usage: ./extract.sh CLIP [BUNDLE_DIR]}"
bundle="${2:-.}"
if [[ "$bundle" =~ ^[0-9]+([.][0-9]+)?$ ]]; then
echo "The FPS argument was removed: extraction always keeps the source rate. Use a directory as argument 2." >&2
exit 2
fi
if [[ "$bundle" = /* || "$bundle" = *..* ]]; then
echo "BUNDLE_DIR must be a relative directory inside this repo" >&2
exit 2
fi
src="${1:?usage: ./extract.sh CLIP [FPS] [OUTDIR]}"
fps="${2:-12}"
out="${3:-frames}"
probe=$(ffprobe -v error -select_streams v:0 \
-show_entries stream=r_frame_rate,avg_frame_rate,nb_frames \
-of json "$src")
fps=$(python3 -c '
import json, sys
from fractions import Fraction
streams = json.load(sys.stdin).get("streams", [])
if not streams:
raise SystemExit("no video stream in source")
s = streams[0]
nominal = Fraction(s["r_frame_rate"])
average = Fraction(s["avg_frame_rate"])
if nominal <= 0 or average <= 0 or abs(float(nominal / average) - 1) > 0.001:
raise SystemExit("variable-frame-rate source needs timestamp-aware playback; refusing to guess its fps")
print(float(average))
' <<< "$probe")
rm -rf "$out"; mkdir -p "$out"
ffmpeg -hide_banner -loglevel warning -i "$src" -vf "fps=$fps" "$out/%04d.png"
count=$(ls -1 "$out" | wc -l)
if [[ "$bundle" = "." ]]; then
dir="frames"; audio_path="audio.wav"; manifest_path="manifest.json"
else
dir="${bundle%/}/frames"
audio_path="${bundle%/}/audio.wav"
manifest_path="${bundle%/}/manifest.json"
fi
rm -rf "$dir"; mkdir -p "$dir"
ffmpeg -hide_banner -loglevel warning -i "$src" -fps_mode passthrough "$dir/%04d.png"
count=$(find "$dir" -maxdepth 1 -name '*.png' -type f | wc -l | tr -d ' ')
expected=$(python3 -c 'import json,sys; print(json.load(sys.stdin)["streams"][0].get("nb_frames", ""))' <<< "$probe")
if [[ "$expected" =~ ^[0-9]+$ && "$count" != "$expected" ]]; then
echo "decoded $count frames but source reports $expected; refusing an inaccurate manifest" >&2
exit 1
fi
# Mono is enough for judging sync and halves the file. A silent clock lets a
# mute source use the same audio-driven transport.
# Mono is enough for judging sync and halves the file. Absent audio is not fatal.
if ffprobe -v error -select_streams a:0 -show_entries stream=codec_type \
-of csv=p=0 "$src" 2>/dev/null | grep -q audio; then
ffmpeg -hide_banner -loglevel warning -y -i "$src" -vn -ac 1 -ar 44100 "$audio_path"
ffmpeg -hide_banner -loglevel warning -y -i "$src" -vn -ac 1 -ar 44100 audio.wav
audio='"audio.wav"'
echo "audio -> audio.wav"
else
duration=$(python3 -c 'import sys; print(int(sys.argv[1]) / float(sys.argv[2]))' "$count" "$fps")
ffmpeg -hide_banner -loglevel warning -y -f lavfi -i anullsrc=r=44100:cl=mono \
-t "$duration" -c:a pcm_s16le "$audio_path"
audio='null'
echo "no audio stream"
fi
# JSON escaping belongs to a JSON writer, especially for source filenames.
python3 - "$fps" "$count" "$dir" "$audio_path" "$src" "$manifest_path" <<'PY'
import json, os, sys
fps, count, frames, audio, source, path = sys.argv[1:]
with open(path, "w") as out:
json.dump({"fps": float(fps), "frames": int(count), "dir": frames,
"audio": audio, "source": os.path.basename(source)}, out)
out.write("\n")
PY
# The page must know the true extraction rate: if it guessed, audio and picture
# would drift. Source of truth lives here, next to the frames it describes.
printf '{"fps":%s,"frames":%s,"dir":"%s","audio":%s,"source":"%s"}\n' \
"$fps" "$count" "$out" "$audio" "$(basename "$src")" > manifest.json
echo "$count source frames at ${fps}fps -> $dir/ ($manifest_path written)"
echo "$count frames at ${fps}fps -> $out/ (manifest.json written)"

38
fly.toml
View file

@ -1,38 +0,0 @@
app = "arthur"
primary_region = "iad"
[build]
dockerfile = "Dockerfile"
[env]
DJANGO_DEBUG = "0"
DJANGO_ALLOWED_HOSTS = ".fly.dev"
DJANGO_CSRF_TRUSTED = "https://arthur.fly.dev"
DJANGO_DB_PATH = "/data/db.sqlite3"
DJANGO_BLOB_ROOT = "/data/blobs"
[[mounts]]
source = "data"
destination = "/data"
initial_size = "1gb"
[http_service]
internal_port = 8000
force_https = true
auto_stop_machines = "stop"
auto_start_machines = true
min_machines_running = 1
processes = ["app"]
[[http_service.checks]]
interval = "30s"
timeout = "5s"
grace_period = "60s"
method = "GET"
path = "/"
headers = { Host = "arthur.fly.dev" }
[[vm]]
cpu_kind = "shared"
cpus = 1
memory = "1gb"

384
frontend/README.md
View file

@ -1,384 +0,0 @@
# frontend
The ClojureScript half. See `docs/port-plan.md` for what is being built and in
what order; this file is only how to run it.
## Once
```sh
mise install # from the REPO ROOT
pip install -r requirements.txt # the Django half; one dependency
mise exec -- python manage.py migrate # the document database
cd frontend && npm install
```
`java` must be 21+. On an older JDK shadow-cljs fails with "CompilerOptions has
been compiled by a more recent version of the Java Runtime", which reads like a
shadow-cljs bug and is not one. `mise install` is what prevents it.
## The tests
```sh
cd frontend && mise exec -- npm test
```
Two things: compile the `:test` build, run it under node.
```
shadow-cljs compile test
node out/node-tests.js
```
Run them separately if a compile error is in the way.
**Run them through `mise`**, or make sure `mise`'s node is first on PATH. `java`
must be 21+ and node 20.19+. On an nvm node 20.11 shadowing the pinned one,
things fail in ways that read like the code being broken and are not.
### And the Django one
```sh
mise exec -- python manage.py test clips # from the REPO ROOT
```
Tests cover the API: the blob store, key verification, the load/save round trip,
the conditional write, source analysis blocks, and video upload and extraction.
The two groups worth
reading are the ones that make the tier split a property of the system rather than
a convention in ClojureScript — the server recomputes every tier-2 key it is
handed, and refuses a block whose analysis does not declare a detector version.
### And the browser one
Step 5's done-criterion is a PICTURE, and no assertion in `cljs.test` can check
one: a take that resolves to the right numbers and draws nothing would pass every
test in `arthur.flow.freeze-test`. A blank canvas under a perfectly correct
transport is the bug class unit tests miss, and it has happened here once.
Step 9's is a picture too, for a different reason: the ways a document survives a
round trip LOOKING correct are the interesting ones. So the suite now also saves
the take, reopens it, and checks the frames are the same pixels.
It drives a real Chrome over CDP, and needs both processes up:
```sh
mise exec -- python manage.py runserver 8778 # from the REPO ROOT
cd frontend && mise exec -- npx shadow-cljs watch app # in one shell
cd frontend && mise exec -- npm run browser # in another
```
`ARTHUR_URL` overrides the page it drives; it defaults to
`http://localhost:8778/index.html`, which since step 9 is Django's.
No dependencies. Playwright is not installed and CDP needs none —
`node --experimental-websocket` has a global `WebSocket` and
`--headless=new --remote-debugging-port=N` is the whole of the other side. It
reads the canvas's own pixels rather than a screenshot, because the CSS scales
the stage up by 2 and a screenshot is four pixels per raster pixel; it writes
PNGs into `test/browser/out/` anyway, so "it drew something" can be checked by
eye as well as by count.
## The app
Two processes, which do not talk to each other:
```sh
mise exec -- python manage.py runserver 8778 # from the REPO ROOT
cd frontend && mise exec -- npx shadow-cljs watch app
```
Then open **<http://localhost:8778/>**. Django serves the page from
`clips/templates/clips/index.html`, its styles from `static/arthur/app.css`, and
the bundle out of `static/arthur/js`, where `shadow-cljs` already writes it — so
nothing copies files between the two.
### The window
One screen, five panes, no scrolling page. `src/arthur/ui/shell.cljs` is the grid
and nothing else; each pane owns its own subscriptions.
```
top the document: its name and last status, export, new / open / save
left media pool — the open document's symbols, and footage on the server
centre the palette strip (16 slots) above the stage
right inspector — the clip, the selected node, the tracked objects
bottom timeline — transport, ruler, a row per node
```
**It opens on a blank document**, and **new** makes another one. Nothing is
loaded until it is asked for.
**Whole documents live under `open ▾`, not in the media pool**, and the split is
load-bearing rather than tidy. Opening a project REPLACES the stage; everything
in the pool is a thing to put ON it. Listing documents beside the symbols inside
one of them makes them read as two kinds of the same thing. The menu lists the
projects the server holds; the built-in scenes are under their own heading,
italic, and are not projects — they are compiled into the bundle and the server
has never heard of them.
Everything that holds nodes is a **symbol**, and none is special: a new document
has one called `main` because it has to be called something. Which symbol is on
screen is editor state, `[:ui :open]`, not a fact about the document — the stage
draws it, the timeline lists it, the transport plays it and a new shape goes into
it. A document opens on the longest symbol nothing else places.
Selection lives in app-db under `:ui`, as `[:node <symbol> <node>]`,
`[:symbol <id>]` or `[:subject|:feature|:group <id>]` — four panes ask what is
selected, and a ratom private to one of them can only be shared by making the
other three require it.
**Drop a video on the media pool** and it uploads, extracts and goes straight on
into detection. Dragging a symbol out of the pool onto the stage places an
instance of it at the playhead.
The timeline's rows are the open symbol's nodes, front-most first, with a dot per
keyframe and a bar over the frames the node exists on; a dense channel is hatched
rather than ticked, because one value per frame is a solid block that says less
than the bar does. Opening a row shows its channels; opening an **instance** row
shows the symbol it places, with every frame number mapped back into the open
symbol's frame space — see the namespace docstring in `ui/timeline.cljs`, which
is where that mapping is argued.
### Paint sketch
Pick a tone from the palette strip, click **polygon**, place at least three
vertices on the stage, then click **finish**. Select a shape — on the stage, or by
its timeline row — to drag its vertices. Scrub to another frame and click
**drawing key here** in the inspector to copy the visible outline there; the
previous drawing holds until that key. The numbered key buttons jump to editable
keys. The transition control between two drawing keys can switch that gap between
a hold and linear vertex tweening. Other gaps keep their own timing. Tweening works
best when the same vertex
keeps the same meaning in every drawing. Paint shapes use the timeline clock
directly, so the roto exposure grid does not delay a drawing key or step its
tween. Use the project **save** button to persist the drawings.
`/index.html` still works, and that is deliberate: it is the URL the browser suite
has used since step 5, when shadow-cljs's `:dev-http` did no directory-index
resolution and the suite learned to ask for the file.
Four built-in clips, under **built-in examples** in the open menu:
| | |
| --- | --- |
| `take` | the synthetic take, head **as filmed**. Step 5's deliverable: a moving mouth, frozen into dense channels, with no video file anywhere. |
| `locked` | the same freeze, head **locked**. The same blocks — `:head`'s channels are written as framed identity instead of as a dense track, and nothing in tier 2 differs. |
| `demo` | the hand-written scene from step 2. Not a face: the smallest scene that exercises every mechanism the model claims to have, so that each one is visible when it breaks. |
| `swarm` | a hundred and twenty dense nodes. Not useful; it is the load test. |
`take` and `locked` are the pair worth looking at together, because switching
between them is the whole of what "stabilisation is a channel, not a mode" means.
The demo scene itself is `src/arthur/demo/scene.edn`. Both the synthetic take
and real footage use `src/arthur/flow/take.cljs` for the measurement order and
`src/arthur/flow/freeze.cljs` for the landmark-to-channel conversion.
**8625 stage study**, in the open menu, loads the locally saved `IMG_8625.MOV` project and places its
post-processed face symbol twice. The stage layout is
`src/arthur/demo/stage_8625.edn`: the right picture and sound start at frame 48,
and the two pictures overlap slightly in stage space. Audio has its own
nodes, linked to the picture instances but with independent spans and gain
channels. The right sound swells and pans across the stage, then fades out at
frame 260 while its picture continues to
frame 280. The row needs that saved 8625 project in the local server database.
### Projects and the EDN fixtures
The EDN files under `demo/` are authored examples compiled into the frontend.
They seed a clip in memory; the server does not read EDN. Clicking **save** on a
clip without a project id creates a project through `POST /api/projects`, uploads
any missing content-addressed blocks, then writes the clip's addressed leaves
through `PUT /api/projects/<id>`. Each leaf value is Transit JSON inside the
request's ordinary JSON envelope. Python stores those values in JSON columns and
does not need an EDN parser. **open** reads the leaves and blocks and rebuilds
the same ClojureScript clip.
The intended editor creates and changes that in-memory clip directly: a project
browser and **new stage** action, timeline instance placement, node and channel
editors, then the existing save path. EDN remains useful for checked-in examples
and reproducible studies. The UI now has the blank-stage action (**new**), a
project browser (`open ▾`), and placement by dragging a symbol out of the media
pool; node and channel editors are still to come — the inspector reports a
channel's shape but has nowhere to change its values.
### Real footage
Drop a video on the media pool, or use its **+** button. The server probes it, re-encodes it
to an H.264 proxy and a raw stream of the same coded frames, pulls WAV audio and one tracing JPEG per frame,
then makes the resulting footage selectable and runs detection on it. **roto**, in
the pool's header, does the same for footage that is already there. Extraction progress is currently read from `/api/extractions/<key>`; a
future WebSocket can push the same job state. The uploaded bytes, extraction job,
and decoded footage have separate records, so the same uploaded video can be
reopened without decoding it again.
**The proxy is what gets measured, and the stills are not.** `flow/ingest` cuts
its raw H.264 stream into coded frames and decodes them in order with WebCodecs.
The proxy has no B-frames, so decode order matches frame order. `flow/detect`
hands each decoded frame to MediaPipe in **VIDEO** running mode at
`i * 1000 / fps` milliseconds. That timestamp has to increase
strictly and has to be real footage time: video mode is a tracker, it reads the
gap between timestamps as motion, and a repeat leaves the graph in an error state
that every later call re-throws. The JPEGs beside the proxy are reference images
for the tracing editor and nothing measures them, so they are not in the footage
digest.
It is re-encoded even when the upload is already H.264, for two reasons: an
iPhone's HEVC is not decodable in every browser, and the footage's identity is the
proxy's digest — one produced by one ffmpeg invocation, not one that depends on
which branch the source happened to take. Its raw stream is copied from that
proxy without another encode.
The command-line route is also available for an existing extracted bundle:
```sh
./extract.sh /path/to/clip.mov # decode to frames + audio + manifest
mise exec -- python manage.py ingest_bundle
```
`extract.sh` keeps every source frame and writes `frames/0001.png` onward,
`audio.wav` and `manifest.json`. Variable frame rate sources are rejected until the
manifest and clock carry per-frame timestamps.
`ingest_bundle` then hashes all of it into the content-addressed blob store under
`var/blobs` — by hard link, so 112MB of PNGs is not copied — and registers one
`Footage` row. From then on the frames are the backend's: `GET /api/footage/<id>`
answers with a manifest carrying **a URL per frame**, and the app fetches those.
That replaced a shared secret. Until step 9 the page fetched `/manifest.json` off
the filesystem and built `frames/0001.png` itself, with shadow-cljs serving the
repo root — so the frame layout was agreed between a shell script and a
ClojureScript namespace, and "where are the frames" was answered by a directory
listing. The cache-busting `?v=` that used to hang off every frame URL went with
it: a blob's name is the hash of its bytes, so re-extracting gives a frame a
different URL rather than overwriting one.
To keep several takes, pass a bundle directory; each ingests separately and both
stay selectable in the app.
```sh
./extract.sh /path/to/clip.mov scratch/my-take
mise exec -- python manage.py ingest_bundle scratch/my-take
```
`scratch/` is ignored by Git, as are `frames/`, `audio.wav` and `manifest.json` at
the root — all of it is extraction output, and tier 3 does not belong in the repo.
Loading detects one face per frame, measures the mouth, eyes and brows from
landmarks and the teeth from source pixels, then freezes them into channels,
and opens the footage clip. Detection happens once when you load;
playback only resolves channels and paints. Frames without a detection remain
marked absent even though their neighbouring poses are used to condition the
track. The scene now records stable subject and feature IDs and explicit eye
pairs; dense channels can mark one feature absent while another is observed.
Current MediaPipe loading supplies only the full-face detection mask. The stage
stays 320×200 regardless of the footage dimensions. Real
footage starts at the source picture rate. The **picture** buttons in the inspector sample the
frozen roto at lower rates while the source track, duration and audio clock stay
unchanged. Picking frames to trace into cels is a separate future editing step.
**save** also stores the detection mask, dense landmarks and raw RGBA mouth crops
as three analysis blocks. **open** restores these without running MediaPipe or
loading source PNGs. The frozen shapes remain separate channel blocks.
For known occlusion intervals, an extracted manifest may add
`"feature-absence": {"eye-r": [[10, 14]]}`. Frame numbers are one-based and
inclusive, matching PNG filenames. The eye remains the same feature when it
reappears; the other eye and the mouth continue through the gap. This is an
input annotation, with no UI for editing it yet.
MediaPipe's JS, wasm and model are under `public/mediapipe/`, served by Django's
staticfiles under `/static/mediapipe/`. No CDN is used by this app. See that
directory's README for provenance.
The server reports what it serves at `GET /api/detector`: the package version plus
the **sha256 of the model asset**, and that string goes inside the content address
of every block a detection produces. Asked rather than assumed, because a version
constant in the client is one somebody has to remember to bump — and
`docs/architecture.md` is explicit that a model upgrade silently reusing old
landmarks presents as "the tool got worse", with no event to attach it to.
Port 8778 is deliberately not 8777. `python3 serve.py` from the repo root still
runs the old JS tool on 8777, and the two are meant to run side by side.
## Saving
**new**, **open** and **save** in the top bar. A save has three ordered stages:
is the tier split:
1. the **analysis** record, so every block stored afterwards can name the detector
version that produced it. The server refuses a block whose analysis it does not
know.
2. ask which **blocks** are missing, upload the source analysis blocks and frozen
channel blocks, then link the source blocks to the analysis.
3. the **document** — tier 1, as leaves. The server refuses a clip that names
blocks it does not hold, so a saved document cannot load into a blank stage
somewhere else.
The status line says what happened: `saved r3 · 64 leaves · 8 blocks`. Saving an
unchanged document says `0 leaves · 0 blocks`, which is both halves of the
addressing working at once — an unchanged leaf keeps its version, and a
content-addressed block is already there.
Saving `swarm` is deliberately visible as a failure: its blocks have
hand-written names and a document may only name content addresses.
`open ▾` lists every project the server holds, newest first, and shows the first
clip of whichever one is picked — the store holds one clip at a time.
## The oracle, which is finished
`js/` was the numeric oracle through step 4: `test/parity/` ran both
implementations on the same synthetic track and diffed `fit-similarity`,
`procrustes-mean`, the raster and `stabilize` to 1e-9.
**It was deleted at step 5, on purpose.** Parity proves the port is FAITHFUL, not
that the answer is RIGHT. The JS is a prototype and several of its conclusions
contradict each other; a parity test pins behaviour while code moves, and keeping
it afterwards would bake the prototype's mistakes into the rewrite and make them
permanent. `docs/port-plan.md` says to delete it in one commit once the CLJS
player renders the synthetic take, and that is what happened.
`js/` itself stays as the reference for the MediaPipe setup, face measurements
and pixel extraction. Its comments encode bugs that actually happened.
## Layout
```
src/arthur/domain/ pure. No re-frame, no DOM, no flow/.
src/arthur/fx/ the only namespaces that talk to the network
src/arthur/flow/ the stages. `(f params inputs) -> output`, no state.
src/arthur/synth.cljs the synthetic track. In src/ because the take PLAYS it —
it stands in for flow/detect, and a tool that needs a
video file before it shows you anything is one you
cannot debug.
src/arthur/demo.cljs the hand-written scene, read from demo/scene.edn
src/arthur/demo/take.cljs the synthetic source for the shared flow/take path
src/arthur/ui/canvas.cljs indexed raster blit to the display canvas
test/arthur/support/ machinery shared between suites; not tests itself
test/browser/ drives a real Chrome over CDP. Not run by `npm test`.
public/mediapipe/ vendored wasm and model, served under /static/mediapipe/
```
`public/` holds nothing but those assets now. The host page that used to sit beside
them is `clips/templates/clips/index.html`.
## Two evaluators, on purpose
`domain/symbol` has both `eval-frame` and `resolver`, and they are not
alternatives:
- **`(eval-frame symbol f store)`** is the specification. Allocating, order-free,
obviously correct. Tests and one-off renders use it.
- **`(resolver symbol store)` -> `(fn [f] ops)`** is what playback uses. It caches
the topological order and the z paths, holds a cursor per channel and reuses
one point buffer per node, so a frame allocates the op maps and nothing else.
Both run the same walk, parameterised by how a channel is read and where its
points are written — two independent implementations of frame evaluation would
drift, and the drift would look like a rendering bug rather than like two
functions disagreeing. What differs between them is exactly the part that can be
wrong, and `scene-test` asserts they agree frame for frame in forward, backward
and random order.
Because the resolver reuses its buffers, **ops must be rasterised before the
next frame is asked for.** That is the contract the rAF loop wants anyway: it
reads, blits, and dispatches nothing.

1657
frontend/package-lock.json generated
View file

File diff suppressed because it is too large Load diff

20
frontend/package.json
View file

@ -1,20 +0,0 @@
{
"name": "arthur-frontend",
"private": true,
"version": "0.0.1",
"scripts": {
"watch": "shadow-cljs watch app",
"release": "shadow-cljs release app",
"test": "shadow-cljs compile test && node out/node-tests.js",
"browser": "node --experimental-websocket test/browser/take.mjs"
},
"dependencies": {
"@mediapipe/tasks-vision": "1.0.1",
"polygon-clipping": "^0.15.7",
"react": "^18.3.1",
"react-dom": "^18.3.1"
},
"devDependencies": {
"shadow-cljs": "^2.28.21"
}
}

218
frontend/public/mediapipe/LICENSE
View file

@ -1,218 +0,0 @@
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
1. Definitions.
"License" shall mean the terms and conditions for use, reproduction,
and distribution as defined by Sections 1 through 9 of this document.
"Licensor" shall mean the copyright owner or entity authorized by
the copyright owner that is granting the License.
"Legal Entity" shall mean the union of the acting entity and all
other entities that control, are controlled by, or are under common
control with that entity. For the purposes of this definition,
"control" means (i) the power, direct or indirect, to cause the
direction or management of such entity, whether by contract or
otherwise, or (ii) ownership of fifty percent (50%) or more of the
outstanding shares, or (iii) beneficial ownership of such entity.
"You" (or "Your") shall mean an individual or Legal Entity
exercising permissions granted by this License.
"Source" form shall mean the preferred form for making modifications,
including but not limited to software source code, documentation
source, and configuration files.
"Object" form shall mean any form resulting from mechanical
transformation or translation of a Source form, including but
not limited to compiled object code, generated documentation,
and conversions to other media types.
"Work" shall mean the work of authorship, whether in Source or
Object form, made available under the License, as indicated by a
copyright notice that is included in or attached to the work
(an example is provided in the Appendix below).
"Derivative Works" shall mean any work, whether in Source or Object
form, that is based on (or derived from) the Work and for which the
editorial revisions, annotations, elaborations, or other modifications
represent, as a whole, an original work of authorship. For the purposes
of this License, Derivative Works shall not include works that remain
separable from, or merely link (or bind by name) to the interfaces of,
the Work and Derivative Works thereof.
"Contribution" shall mean any work of authorship, including
the original version of the Work and any modifications or additions
to that Work or Derivative Works thereof, that is intentionally
submitted to Licensor for inclusion in the Work by the copyright owner
or by an individual or Legal Entity authorized to submit on behalf of
the copyright owner. For the purposes of this definition, "submitted"
means any form of electronic, verbal, or written communication sent
to the Licensor or its representatives, including but not limited to
communication on electronic mailing lists, source code control systems,
and issue tracking systems that are managed by, or on behalf of, the
Licensor for the purpose of discussing and improving the Work, but
excluding communication that is conspicuously marked or otherwise
designated in writing by the copyright owner as "Not a Contribution."
"Contributor" shall mean Licensor and any individual or Legal Entity
on behalf of whom a Contribution has been received by Licensor and
subsequently incorporated within the Work.
2. Grant of Copyright License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
copyright license to reproduce, prepare Derivative Works of,
publicly display, publicly perform, sublicense, and distribute the
Work and such Derivative Works in Source or Object form.
3. Grant of Patent License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
(except as stated in this section) patent license to make, have made,
use, offer to sell, sell, import, and otherwise transfer the Work,
where such license applies only to those patent claims licensable
by such Contributor that are necessarily infringed by their
Contribution(s) alone or by combination of their Contribution(s)
with the Work to which such Contribution(s) was submitted. If You
institute patent litigation against any entity (including a
cross-claim or counterclaim in a lawsuit) alleging that the Work
or a Contribution incorporated within the Work constitutes direct
or contributory patent infringement, then any patent licenses
granted to You under this License for that Work shall terminate
as of the date such litigation is filed.
4. Redistribution. You may reproduce and distribute copies of the
Work or Derivative Works thereof in any medium, with or without
modifications, and in Source or Object form, provided that You
meet the following conditions:
(a) You must give any other recipients of the Work or
Derivative Works a copy of this License; and
(b) You must cause any modified files to carry prominent notices
stating that You changed the files; and
(c) You must retain, in the Source form of any Derivative Works
that You distribute, all copyright, patent, trademark, and
attribution notices from the Source form of the Work,
excluding those notices that do not pertain to any part of
the Derivative Works; and
(d) If the Work includes a "NOTICE" text file as part of its
distribution, then any Derivative Works that You distribute must
include a readable copy of the attribution notices contained
within such NOTICE file, excluding those notices that do not
pertain to any part of the Derivative Works, in at least one
of the following places: within a NOTICE text file distributed
as part of the Derivative Works; within the Source form or
documentation, if provided along with the Derivative Works; or,
within a display generated by the Derivative Works, if and
wherever such third-party notices normally appear. The contents
of the NOTICE file are for informational purposes only and
do not modify the License. You may add Your own attribution
notices within Derivative Works that You distribute, alongside
or as an addendum to the NOTICE text from the Work, provided
that such additional attribution notices cannot be construed
as modifying the License.
You may add Your own copyright statement to Your modifications and
may provide additional or different license terms and conditions
for use, reproduction, or distribution of Your modifications, or
for any such Derivative Works as a whole, provided Your use,
reproduction, and distribution of the Work otherwise complies with
the conditions stated in this License.
5. Submission of Contributions. Unless You explicitly state otherwise,
any Contribution intentionally submitted for inclusion in the Work
by You to the Licensor shall be under the terms and conditions of
this License, without any additional terms or conditions.
Notwithstanding the above, nothing herein shall supersede or modify
the terms of any separate license agreement you may have executed
with Licensor regarding such Contributions.
6. Trademarks. This License does not grant permission to use the trade
names, trademarks, service marks, or product names of the Licensor,
except as required for reasonable and customary use in describing the
origin of the Work and reproducing the content of the NOTICE file.
7. Disclaimer of Warranty. Unless required by applicable law or
agreed to in writing, Licensor provides the Work (and each
Contributor provides its Contributions) on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied, including, without limitation, any warranties or conditions
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
PARTICULAR PURPOSE. You are solely responsible for determining the
appropriateness of using or redistributing the Work and assume any
risks associated with Your exercise of permissions under this License.
8. Limitation of Liability. In no event and under no legal theory,
whether in tort (including negligence), contract, or otherwise,
unless required by applicable law (such as deliberate and grossly
negligent acts) or agreed to in writing, shall any Contributor be
liable to You for damages, including any direct, indirect, special,
incidental, or consequential damages of any character arising as a
result of this License or out of the use or inability to use the
Work (including but not limited to damages for loss of goodwill,
work stoppage, computer failure or malfunction, or any and all
other commercial damages or losses), even if such Contributor
has been advised of the possibility of such damages.
9. Accepting Warranty or Additional Liability. While redistributing
the Work or Derivative Works thereof, You may choose to offer,
and charge a fee for, acceptance of support, warranty, indemnity,
or other liability obligations and/or rights consistent with this
License. However, in accepting such obligations, You may act only
on Your own behalf and on Your sole responsibility, not on behalf
of any other Contributor, and only if You agree to indemnify,
defend, and hold each Contributor harmless for any liability
incurred by, or claims asserted against, such Contributor by reason
of your accepting any such warranty or additional liability.
END OF TERMS AND CONDITIONS
APPENDIX: How to apply the Apache License to your work.
To apply the Apache License to your work, attach the following
boilerplate notice, with the fields enclosed by brackets "[]"
replaced with your own identifying information. (Don't include
the brackets!) The text should be enclosed in the appropriate
comment syntax for the file format. We also recommend that a
file or class name and description of purpose be included on the
same "printed page" as the copyright notice for easier
identification within third-party archives.
Copyright [yyyy] [name of copyright owner]
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
===========================================================================
For files under tasks/cc/text/language_detector/custom_ops/utils/utf/
===========================================================================
/*
* The authors of this software are Rob Pike and Ken Thompson.
* Copyright (c) 2002 by Lucent Technologies.
* Permission to use, copy, modify, and distribute this software for any
* purpose without fee is hereby granted, provided that this entire notice
* is included in all copies of any software which is or includes a copy
* or modification of this software and in all copies of the supporting
* documentation for such software.
* THIS SOFTWARE IS BEING PROVIDED "AS IS", WITHOUT ANY EXPRESS OR IMPLIED
* WARRANTY. IN PARTICULAR, NEITHER THE AUTHORS NOR LUCENT TECHNOLOGIES MAKE ANY
* REPRESENTATION OR WARRANTY OF ANY KIND CONCERNING THE MERCHANTABILITY
* OF THIS SOFTWARE OR ITS FITNESS FOR ANY PARTICULAR PURPOSE.
*/

17
frontend/public/mediapipe/README.md
View file

@ -1,17 +0,0 @@
# Local MediaPipe assets
`vision_bundle.js` and the four wasm loader/binary files under `wasm/` come
from `@mediapipe/tasks-vision` **1.0.1**, pinned in `frontend/package.json`.
`FilesetResolver.forVisionTasks` uses the SIMD pair where supported and the
no-SIMD pair elsewhere. The package and these files are Apache-2.0; see
[LICENSE](LICENSE).
`face_landmarker.task` is the official [Face Landmarker model](https://storage.googleapis.com/mediapipe-models/face_landmarker/face_landmarker/float16/1/face_landmarker.task).
Its SHA-256 is
`64184e229b263107bc2b804c6625db1341ff2bb731874b0bcc2fe6544e0bc9ff`.
These are served from `/mediapipe/` so detection needs no CDN at runtime. The
browser bundle is loaded as a script before the CLJS app because Shadow CLJS
cannot parse the package's CommonJS bundle (its dynamic `import()` is unsupported
by the current compiler). `flow/detect.cljs` is the sole call site for its
`Vision` global.

BIN
frontend/public/mediapipe/face_landmarker.task
View file

Binary file not shown.

2
frontend/public/mediapipe/vision_bundle.js
View file

File diff suppressed because one or more lines are too long

8840
frontend/public/mediapipe/wasm/vision_wasm_internal.js
View file

File diff suppressed because it is too large Load diff

BIN
frontend/public/mediapipe/wasm/vision_wasm_internal.wasm
View file

Binary file not shown.

8831
frontend/public/mediapipe/wasm/vision_wasm_nosimd_internal.js
View file

File diff suppressed because it is too large Load diff

BIN
frontend/public/mediapipe/wasm/vision_wasm_nosimd_internal.wasm
View file

Binary file not shown.

49
frontend/shadow-cljs.edn
View file

@ -1,49 +0,0 @@
;; Two builds and no more:
;;
;; app the tool. Output goes straight into the Django staticfiles tree, so
;; `python manage.py runserver` and `shadow-cljs watch app` are the whole
;; dev loop with nothing copying files between them.
;; test :node-test, because everything below `ui/` and `fx/` is pure and has
;; no business needing a browser to be asserted about. The canvas-facing
;; parts get asserted through domain/raster's byte buffer instead, which
;; is what the JS selftest already did.
{:source-paths ["src" "test"]
:dependencies [[reagent "1.2.0"]
[re-frame "1.4.3"]
;; The document's wire format. JSON would do for the shape of tier
;; 1 but not for its VALUES: channel keys are a map by FRAME
;; NUMBER and every id is a keyword, and JSON has neither, so a
;; save would quietly turn `{0 v}` into `{"0" v}` and `:mouth`
;; into "mouth". Transit is JSON on the wire, so Django stores a
;; leaf in a JSONField and the admin can still read it.
[com.cognitect/transit-cljs "0.8.280"]]
;; THERE IS NO :dev-http, since step 9. Django serves the page — one template,
;; out of `clips/templates/` — and shadow-cljs only builds into the staticfiles
;; tree, which is what `:output-dir` below already did. So the dev loop is two
;; processes that do not talk to each other:
;;
;; mise exec -- python manage.py runserver 8778 (from the repo root)
;; cd frontend && mise exec -- npx shadow-cljs watch app
;;
;; What went away with the key was a set of problems rather than a feature. The
;; two roots it needed — `public` for the host page and `..` for the repo root, IN
;; THAT ORDER, because the root has the old tool's index.html and serving that one
;; instead would look like the port having regressed to a suspiciously complete
;; tool — were a way of reaching frames/, audio.wav and manifest.json off the
;; filesystem. Those are tier 3, and tier 3 is now the backend's, by hash.
;;
;; 8778 is still deliberately not 8777, which is still the old JS tool's under
;; `python3 serve.py`. The two are meant to run side by side.
:builds
{:app {:target :browser
:output-dir "../static/arthur/js"
:asset-path "/static/arthur/js"
:compiler-options {:source-map true}
:modules {:main {:init-fn arthur.core/init}}}
:test {:target :node-test
:output-to "out/node-tests.js"
:ns-regexp "-test$"}}}

262
frontend/src/arthur/audio/mix.cljs
View file

@ -1,262 +0,0 @@
(ns arthur.audio.mix
"Render independently placed audio tracks into one stage audio clock.
The mix is derived from saved audio track leaves and immutable footage blobs.
The transport still has ONE clock, so seeking, rate changes and looping stay
tied to the same position the picture is drawn from.
THE AUDIO BUFFER IS THE PRODUCT AND THE WAV IS ONE PACKAGING OF IT. `buffer!`
renders and the wrappers below it package, rather than the render being spelled
once per consumer.
PLAYBACK NO LONGER PACKAGES AT ALL. It takes the buffer as it is — see
`clock-source!` and `arthur.clock.graph` — because encoding a WAV so an
`<audio>` element had a URL to hold cost O(the clip's length) on the main
thread, on open, on every tab switch and on every edit to a track. The WAV is
now what an EXPORT wants: bytes for an archive, or the buffer itself for a
muxer. `clock!` below is the element backend's packaging and goes when it does."
(:require [arthur.domain.channel :as ch]
[arthur.domain.clip :as clip]
[arthur.domain.nest :as nest]
[arthur.domain.node :as node]))
(defn wav-bytes
"An `AudioBuffer` -> the bytes of a 16-bit PCM WAV.
PEAK-NORMALISED ONLY IF IT WOULD CLIP. A mix of several tracks can sum past
1.0, and 16-bit PCM has nowhere to put that, so the alternative to scaling is
audible clipping on exactly the loudest moment. Below the threshold nothing is
touched, so a single-track mix is the footage's own audio sample for sample."
[^js buffer]
(let [channels (.-numberOfChannels buffer)
frames (.-length buffer)
rate (.-sampleRate buffer)
bytes (js/ArrayBuffer. (+ 44 (* frames channels 2)))
view (js/DataView. bytes)
samples (mapv #(.getChannelData buffer %) (range channels))
;; A HAND-WRITTEN LOOP over the typed arrays, not `(reduce max (for ...))`.
;; The lazy sequence that read beautifully allocated one boxed double per
;; SAMPLE — ten million of them for a three-minute mix — and spent the
;; whole of a sixteen-second project open walking them and collecting
;; them. Same arithmetic, no allocation.
peak (loop [c 0 p 0]
(if (< c channels)
(recur (inc c)
(let [^js data (nth samples c)]
(loop [i 0 p p]
(if (< i frames)
(recur (inc i)
(let [a (js/Math.abs (aget data i))]
(if (> a p) a p)))
p))))
p))
level (if (> peak 0.98) (/ 0.98 peak) 1)]
(doseq [[offset word] [[0 "RIFF"] [8 "WAVE"] [12 "fmt "] [36 "data"]]]
(dotimes [i 4] (.setUint8 view (+ offset i) (.charCodeAt word i))))
(.setUint32 view 4 (- (.-byteLength bytes) 8) true)
(.setUint32 view 16 16 true)
(.setUint16 view 20 1 true)
(.setUint16 view 22 channels true)
(.setUint32 view 24 rate true)
(.setUint32 view 28 (* rate channels 2) true)
(.setUint16 view 32 (* channels 2) true)
(.setUint16 view 34 16 true)
(.setUint32 view 40 (* frames channels 2) true)
;; Channel-outer so the channel's array is looked up once rather than once
;; per frame. The byte offsets are unchanged, so the interleaving is too.
(dotimes [c channels]
(let [^js data (nth samples c)]
(dotimes [i frames]
(let [sample (* level (aget data i))]
(.setInt16 view (+ 44 (* (+ (* i channels) c) 2))
(js/Math.round (* 32767 (max -1 (min 1 sample)))) true)))))
(js/Uint8Array. bytes)))
(defn- wav-url [^js buffer]
(js/URL.createObjectURL
(js/Blob. #js [(wav-bytes buffer)] #js {:type "audio/wav"})))
(defn- fetch-ok! [url what]
(-> (js/fetch url)
(.then (fn [response]
(when-not (.-ok response)
(throw (ex-info (str "audio track's " what " is missing")
{:url url :status (.-status response)})))
response))))
(defn- decode-bytes! [bytes]
(.decodeAudioData (js/OfflineAudioContext. 1 1 44100) bytes))
(defn- source!
"Promise of `[source {:buffer :fps}]` for an audio node's `:source`. Footage
counts its frames at its own rate; a sound file has no frames of its own, so
its `:fps` is nil and it counts in the document's."
[{:keys [footage sound] :as source}]
(if sound
(-> (fetch-ok! (str "/api/sounds/" sound) "sound")
(.then #(.json %))
(.then #(fetch-ok! (.-audio %) "blob"))
(.then #(.arrayBuffer %))
(.then decode-bytes!)
(.then (fn [buffer] [source {:buffer buffer}])))
(-> (fetch-ok! (str "/api/footage/" footage) "footage")
(.then #(.json %))
(.then (fn [^js manifest]
(-> (fetch-ok! (.-audio manifest) "blob")
(.then #(.arrayBuffer %))
(.then decode-bytes!)
(.then (fn [buffer] [source {:buffer buffer :fps (.-fps manifest)}]))))))))
(defn- automate! [^js param channel start end fps factor default store]
(let [channel (or channel (ch/framed default))
sample (fn [f] (ch/value-at channel
(if-let [{:keys [at rate]} (:sample-time channel)]
(js/Math.floor (* rate (- f at))) f)
store))]
(.setValueAtTime param (* factor (sample start)) (/ start fps))
(cond
(:dense channel)
(doseq [f (range (inc start) end)]
(.setValueAtTime param (* factor (sample f)) (/ f fps)))
(:animated? channel)
(doseq [[f v] (sort-by key (:keys channel))
:when (and (> f start) (< f end))]
(if (= :linear (:interp channel))
(.linearRampToValueAtTime param (* factor v) (/ f fps))
(.setValueAtTime param (* factor v) (/ f fps)))))))
(defn tracks-of
"The sounds symbol `sid` plays, including those inside what it places — see
`nest/audio-tracks`. Playback mixes the open symbol's."
[document sid]
(nest/audio-tracks document sid))
(defn- render! [document sid sources store]
(let [fps (:fps document)
frames (clip/output-frames document sid)
tracks (tracks-of document sid)
output (js/OfflineAudioContext.
2 (js/Math.ceil (* (/ frames fps) 44100)) 44100)]
(doseq [track tracks]
(let [[start end] (or (node/placed-span track) [0 frames])
start (max 0 start)
end (min frames end)
{:keys [buffer fps] :or {fps (:fps track)}} (get sources (:source track))
sound (.createBufferSource output)
gain (.createGain output)
pan (.createStereoPanner output)]
(when (< start end)
(set! (.-buffer sound) buffer)
(set! (.-loop sound) (boolean (get-in track [:time :loop?])))
(automate! (.-playbackRate sound)
(get-in track [:channels [:audio :rate]])
start end (:fps document) (* (or (get-in track [:time :rate]) 1) (/ (:fps document) fps)) 1 store)
(automate! (.-gain gain)
(get-in track [:channels [:audio :gain]])
start end (:fps document) 1 1 store)
(automate! (.-pan pan)
(get-in track [:channels [:audio :pan]])
start end (:fps document) 1 0 store)
(.connect sound gain)
(.connect gain pan)
(.connect pan (.-destination output))
(.start sound (/ start (:fps document)) (/ (node/local-frame track start) fps))
(.stop sound (/ end (:fps document))))))
(.startRendering output)))
(defn buffer!
"Promise of the `AudioBuffer` one symbol's audio tracks mix down to, or nil
when it has none.
The raw product. `mix!` packages it as a WAV URL for the transport and
`export/frames` packages it as WAV bytes in an archive; a muxer would take it as
it is, which is why this is the function the others are written in terms of."
[document sid store]
(let [tracks (tracks-of document sid)]
(if (empty? tracks)
(js/Promise.resolve nil)
(-> (js/Promise.all
(into-array (map source! (distinct (map :source tracks)))))
(.then (fn [pairs] (render! document sid (into {} (array-seq pairs)) store)))))))
(defn decode!
"Promise of the `AudioBuffer` behind a URL. What a clip whose audio is a plain
file rather than placed tracks exports."
[url]
(-> (js/fetch url)
(.then (fn [^js response]
(when-not (.-ok response)
(throw (ex-info "the clip's audio did not load"
{:url url :status (.-status response)})))
(.arrayBuffer response)))
(.then decode-bytes!)))
(defn fit-buffer
"Fit fallback audio to the open timeline, padding with silence or trimming.
The buffer and clock must share a duration so looping wraps at the timeline's
end rather than repeating a short soundtrack underneath a longer animation."
[^js buffer seconds]
(let [rate (.-sampleRate buffer)
frames (max 1 (js/Math.ceil (* seconds rate)))]
(if (= frames (.-length buffer))
buffer
(let [channels (.-numberOfChannels buffer)
fitted (.createBuffer (js/OfflineAudioContext. channels 1 rate)
channels frames rate)]
(dotimes [c channels]
(.set (.getChannelData fitted c)
(.subarray (.getChannelData buffer c) 0 (min frames (.-length buffer)))))
fitted))))
(defn clock-source!
"Promise of `{:buffer :seconds}` — the audio the transport runs its clock on
while symbol `sid` is open, and how long that clock is.
Same order of preference as the WAV packaging in `clock!` below: the symbol's
own placed tracks, mixed; the document's audio file, for the symbol the
document opens on and only that one; and otherwise NO BUFFER AT ALL and the
symbol's own length.
THE SILENT CASE IS WHY THE DURATION IS RETURNED BESIDE THE BUFFER rather than
read off it. A symbol with no sound still needs a clock exactly as long as it
is, and `clock!` had to synthesize that silence and then ENCODE it, full
length, so an element had a duration to report. There is nothing to decode for
a symbol with no sound: saying how long it is answers the only question the
silence was ever asked."
[document sid fallback-url store]
(-> (buffer! document sid store)
(.then (fn [^js buffer]
(cond
buffer
{:buffer buffer :seconds (.-duration buffer)}
(and fallback-url (= sid (clip/opens-on document)))
(-> (decode! fallback-url)
(.then (fn [b]
(let [seconds (/ (clip/output-frames document sid) (:fps document))]
{:buffer (fit-buffer b seconds) :seconds seconds}))))
:else
{:buffer nil
:seconds (/ (clip/output-frames document sid) (:fps document))})))))
(defn clock!
"Promise of the URL the transport should play while symbol `sid` is open.
THE ELEMENT BACKEND'S PACKAGING of `clock-source!`, kept while that backend is
— see `arthur.clock`. Every cost this namespace had on open is in the two
`wav-url` calls below.
The frame is derived from the audio element and from nothing else, so every
open symbol needs a sound exactly as long as it is. In order: its own placed
tracks, mixed; the document's audio file, for the symbol the document opens on
and only that one; and otherwise SILENCE of the symbol's length — a ten-frame
symbol played against the whole take's soundtrack would run ten frames and then
keep the clock going for minutes."
[document sid fallback-url store]
(-> (clock-source! document sid fallback-url store)
(.then (fn [{:keys [buffer seconds]}]
(wav-url (or buffer
(.createBuffer (js/OfflineAudioContext. 1 1 44100)
1 (max 1 (js/Math.ceil (* 44100 seconds))) 44100)))))))

148
frontend/src/arthur/clock.cljs
View file

@ -1,148 +0,0 @@
(ns arthur.clock
"The audio clock. Lives OUTSIDE app-db, deliberately.
THE FRAME IS DERIVED FROM THE AUDIO, never counted:
frame = ⌊position · fps⌋
A loop that counted frames and hoped to keep up would drift, and drift against
a voice is the one artefact that cannot be fixed downstream — a lip-sync tool
whose sync wanders is not a lip-sync tool. Deriving instead means a slow frame
DROPS the frames it missed and the next one lands where the audio already is.
The failure mode becomes a visible stutter rather than an invisible slide, and
those are very different bugs to own.
½× and ¼× are the backend's playback rate and nothing else. The audio slows,
the position advances proportionally, and the derived frame follows — so slow
motion cannot desync by construction. Implementing rate as a multiplier on a
counted frame would give the picture a rate and the sound another.
It is outside app-db because the audio is the source of truth and copying it
into the db every frame would make the db a lagging mirror of something
authoritative elsewhere. What DOES belong in the db is the playhead as a piece
of document state — see events/playback — and that is written from here, not
read by here.
TWO BACKENDS, ONE ARITHMETIC. `position` comes from either a Web Audio graph
(`clock.graph`, the default) or an `<audio>` element (`clock.element`, kept
switchable while the first earns trust). Everything below the position — the
derivation, the clamp, the exposure grid — is here and is the same either way,
so the two can be compared on the same take rather than swapped on faith.
Build with `:closure-defines {arthur.clock/BACKEND \"element\"}`, or call
`use-backend!` from the console, to put the element back."
(:require [arthur.clock.element :as element]
[arthur.clock.graph :as graph]
[arthur.clock.transport :as t]
[arthur.domain.node :as node]))
(goog-define ^String BACKEND "graph")
(defonce ^:private mode (atom (keyword BACKEND)))
;; `{:source x :backend b}`. The source is kept so that re-attaching the same
;; thing can be recognised as the no-op it is — see `install!`.
(defonce ^:private current (atom nil))
(defn graph?
"Whether the app should wire up the graph backend. Read by `ui/shell`, which
renders the audio element only when this is false, and by `events/playback`,
which fetches a buffer rather than a WAV URL when it is true."
[]
(and (= :graph @mode) (graph/available?)))
(defn use-backend!
"Switch backends. Takes effect on the next thing that attaches one, which is
the next tab switch or document open — nothing is torn down under a take
that is already playing."
[m]
(reset! mode m))
(defn- install!
"Put a backend on `source`, unless `source` is already the clock's.
IDEMPOTENCE IS LOAD-BEARING HERE. The element's `:ref` is an inline closure,
so React hands it the same node again on every re-render of the shell —
rebuilding the clock there would mean a pane being dragged released whatever
was playing. The source is the identity: the element, or the buffer and its
length."
[source make]
(let [{:keys [backend] prev :source} @current]
(when-not (and backend (= prev source))
(when backend (t/-release! backend))
(reset! current {:source source
:backend (when (some? source) (make))}))))
(defn attach!
"Hand the clock an audio element. Idempotent. Element backend only — the
`:ref` that calls this is on a node `ui/shell` renders only in that mode."
[audio-el]
(install! audio-el #(element/backend audio-el)))
(defn attach-buffer!
"Hand the clock `seconds` of audio to run on, as an `AudioBuffer` or as nil
for silence of that length. Graph backend only."
[buffer seconds]
;; Keyed on the length as well as the buffer, because two silent clocks of
;; different lengths are two different clocks and both have a nil buffer.
(install! [buffer seconds] #(graph/backend buffer seconds)))
(defn- clamp [f frames]
(-> f (max 0) (min (dec frames))))
(defn frame
"The clip frame the audio is currently on."
[fps frames]
(if-let [b (:backend @current)]
(clamp (js/Math.floor (* (t/-position b) fps)) frames)
0))
(defn playing? []
(boolean (when-let [b (:backend @current)] (t/-playing? b))))
(defn rate []
(if-let [b (:backend @current)] (t/-rate b) 1.0))
(defn set-rate! [r]
(when-let [b (:backend @current)] (t/-set-rate! b r)))
(defn play! []
(when-let [b (:backend @current)] (t/-play! b)))
(defn pause! []
(when-let [b (:backend @current)] (t/-pause! b)))
(defn seek!
"Put the audio at the start of frame f. Seeking to the frame's start rather
than its middle keeps `frame` idempotent: seek to f, read back f."
[fps frames f]
(when-let [b (:backend @current)]
(t/-seek! b (/ (clamp f frames) fps))))
(defn set-loop!
"Wrap at the end instead of stopping. The frame stays derived — the position
simply returns to zero — so nothing about the sync changes, which is the point
of not counting frames.
It earns its place at 2x and 4x, where the whole clip is gone in under four
seconds and a profile wants more than that to look at."
[on?]
(when-let [b (:backend @current)] (t/-set-loop! b on?)))
(defn set-muted! [on?]
(when-let [b (:backend @current)] (t/-set-muted! b on?)))
(defn duration-frames
"How many frames the audio actually covers, which need not be the clip's
length. Reported rather than assumed: a clip longer than its audio is a
legitimate thing to be told about, not a thing to silently truncate."
[fps]
(when-let [b (:backend @current)]
(let [d (t/-duration b)]
(when (and d (js/isFinite d)) (js/Math.ceil (* d fps))))))
(defn exposed-frame
"The frame a clip-level exposure grid holds `f` back onto. The player shows it
as a readout so that `exposure 2` is visibly doing something at the transport
rather than only inside the scene."
[f expose]
(node/expose f expose))

45
frontend/src/arthur/clock/element.cljs
View file

@ -1,45 +0,0 @@
(ns arthur.clock.element
"The clock on an `<audio>` element. The original backend, kept switchable.
Reads `currentTime` and takes it as the position. That is the whole of it, and
it is why this backend needs a WAV: an element holds a URL, so a mixdown has
to be encoded and blobbed before it can be played — see `audio/mix`'s
`clock!`. The encode is O(the clip's length) and runs on the main thread, so a
long take costs seconds of frozen UI on open, on every tab switch and on every
edit to a track. `arthur.clock.graph` exists to not do that.
What this backend still has that the graph one has to be told: the OS media
keys and the media session, which the element gets from the browser for free.
KEPT so the two can be compared on the same document rather than swapped on
faith. When the graph backend has been trusted for a while, this namespace and
`mix/clock!`'s WAV packaging go together."
(:require [arthur.clock.transport :as t]))
(deftype Element [^js el]
t/Transport
(-position [_] (.-currentTime el))
(-duration [_] (.-duration el))
(-playing? [_] (and (not (.-paused el)) (not (.-ended el))))
(-rate [_] (.-playbackRate el))
(-set-rate! [_ r] (set! (.-playbackRate el) r))
(-play! [_]
;; Returns a promise that rejects if the browser has not seen a gesture yet.
;; Swallowed: the transport button IS the gesture, so this can only fire on a
;; programmatic play, where a console error is noise rather than news.
(some-> (.play el) (.catch (fn [_]))))
(-pause! [_] (.pause el))
(-seek! [_ seconds] (set! (.-currentTime el) seconds))
(-set-loop! [_ on?] (set! (.-loop el) (boolean on?)))
(-set-muted! [_ on?] (set! (.-muted el) (boolean on?)))
(-release! [_]
;; Nothing. React owns the element and unmounting it is what stops it; this
;; backend is a few property accesses wrapped in a type and holds no more
;; than that.
nil))
(defn backend
"Wrap an audio element — or anything that answers the same five properties,
which is what `clock-test`'s fake is."
[el]
(->Element el))

266
frontend/src/arthur/clock/graph.cljs
View file

@ -1,266 +0,0 @@
(ns arthur.clock.graph
"The clock on a Web Audio graph. No encode, no blob, no element.
THE BUFFER IS PLAYED, NOT PACKAGED. `audio/mix`'s `buffer!` already renders
the mixdown; the element backend then spent O(the clip's length) encoding that
buffer to a WAV purely so a `src` attribute had something to point at. An
`AudioBufferSourceNode` takes the buffer as it is, so opening a document costs
one node instead of a hundred megabytes of 16-bit PCM.
THE FRAME IS STILL DERIVED, AND FROM A BETTER CLOCK. `AudioContext.currentTime`
is the audio device's own position in double precision, advancing every 128
samples; an element's `currentTime` is whatever the media pipeline last
published and is permitted to lag. Nothing is counted here either — position is
an anchor plus elapsed context time, and the anchor is re-set on every seek and
every rate change, so no arithmetic accumulates across either.
AND IT IS LATENCY-COMPENSATED, which is the one thing this backend must get
right. `currentTime` is the time of the quantum being RENDERED, which the
speaker is `outputLatency` behind — tens of milliseconds, far more over
Bluetooth. Report the renderer's position and the picture leads the sound by
exactly that much, which in a lip-sync tool is the only artefact that matters.
So `scheduled` is the bookkeeping and `-position` is `scheduled` read one
latency in the past. The element has the same lag underneath; the difference is
that this one is a number we can subtract rather than an error we inherit.
A SYMBOL WITH NO SOUND GETS A CLOCK ANYWAY, with no buffer at all: `seconds`
is its length and the context's clock does the rest. The element backend had to
synthesize silence and encode THAT to a WAV, full length, so the element had a
duration to report — the clearest sign that the element was driving the design
rather than serving it."
(:require [arthur.clock.transport :as t]))
(defonce ^:private shared (atom nil))
(defn available?
"Whether this backend can run at all. False under node, where the tests live."
[]
(exists? js/AudioContext))
(defn context
"The app's one `AudioContext`, made on first use.
Lazy because constructing one before anything wants to play is how a browser
decides the page is trying to autoplay, and because `available?` is false in
the test runner."
[]
(or (:ctx @shared)
(let [ctx (js/AudioContext.)
;; ONE output gain for the app, not one per backend: a tab switch
;; replaces the backend, and a gain node per backend would leave the
;; old one wired to the destination. Mute lives here for the same
;; reason — it is a property of the transport, not of whichever
;; buffer happens to be loaded.
out (.createGain ctx)]
(.connect out (.-destination ctx))
(:ctx (reset! shared {:ctx ctx :out out})))))
(defn output [] (do (context) (:out @shared)))
(defn- latency
"How far ahead of the speaker `currentTime` is, in seconds.
`outputLatency` is the whole path and the number we want. Firefox reports 0
until the graph has actually run, so `baseLatency` stands in — it is only the
graph's own buffering and therefore an underestimate, which errs towards the
picture leading slightly rather than the correction overshooting. Neither
exists everywhere; 0 is then no worse than the element."
[^js ctx]
(let [out (.-outputLatency ctx)
base (.-baseLatency ctx)]
(cond
(and (number? out) (js/isFinite out) (pos? out)) out
(and (number? base) (js/isFinite base) (pos? base)) base
:else 0)))
(defn- seg-at
"The segment governing context time `t`."
[segs t]
(or (last (filter #(<= (:from %) t) segs)) (first segs)))
(defn- raw
"Where the audio is at context time `t`, in seconds into the buffer.
PIECEWISE, and that is the whole subtlety of this namespace. Audio already
rendered cannot be re-rated: when the transport goes 1x -> 4x, the samples
still travelling to the speaker were rendered at 1x, so reading them back at
4x jumps the playhead BACKWARDS by three output latencies — about fourteen
frames on a laptop, which is a visible lurch on every rate change and was
exactly what the first version of this did. So each `play`, `pause`, `seek`
and rate change records a segment, and a position is read against whichever
segment was in force when that audio was rendered.
Evaluated before the oldest segment we kept, it clamps. That is what makes a
seek read back exactly what was seeked to: a seek has nothing in flight worth
honouring — the user has jumped — so its segment starts the timeline over."
[segs t]
(let [t (max t (:from (first segs)))
s (seg-at segs t)]
(+ (:anchor s) (* (:rate s) (- t (:from s))))))
(defn- at-renderer
"Where the graph has rendered up to: `raw` at the context's own clock."
[{:keys [^js ctx segs]}]
(raw segs (.-currentTime ctx)))
(defn- at-speaker
"Where the sound being heard is: `raw` one output latency in the past. See the
namespace docstring — this is the whole of the compensation."
[{:keys [^js ctx segs]}]
(raw segs (- (.-currentTime ctx) (latency ctx))))
(defn- running?
"Whether the renderer is advancing. Read off the segments rather than kept
beside them, so there is one answer and not two that can disagree."
[{:keys [segs]}]
(pos? (:rate (last segs))))
(defn- prune
"Drop segments no position can still need: everything before the last one that
began at or before the in-flight window. Unbounded history would otherwise
grow by one entry per rate change for the life of the document."
[segs cutoff]
(let [n (count (take-while #(<= (:from %) cutoff) segs))]
(if (<= n 1) segs (subvec segs (dec n)))))
(defn- position
"`at-speaker`, brought inside the audio. Clamped before the wrap, so the few
milliseconds of negative position right after a looped play — the first
samples are still in the output buffer — read as 0 rather than as the end."
[{:keys [seconds loop?] :as st}]
(let [p (at-speaker st)]
(cond
(not (and seconds (pos? seconds))) 0
loop? (mod (max 0 p) seconds)
:else (-> p (max 0) (min seconds)))))
(defn- done?
"Run off the end. Judged at the SPEAKER, so playback is still reported as
running while the last scheduled samples are on their way out."
[{:keys [seconds loop?] :as st}]
(and (not loop?) seconds (pos? seconds) (>= (at-speaker st) seconds)))
(defn- spin-up!
"A fresh source node playing from where the renderer now is.
`AudioBufferSourceNode`s are single-use, so a play, a seek while playing and a
resumed pause each make a new one; they are cheap, which is the point of this
backend.
Nil when there is no buffer — the silent clock runs on the context alone."
[{:keys [^js ctx ^js buffer ^js out rate loop? seconds] :as st}]
(when buffer
(let [node (.createBufferSource ctx)]
(set! (.-buffer node) buffer)
(set! (.-loop node) (boolean loop?))
(set! (.-value (.-playbackRate node)) rate)
(.connect node out)
;; `when` 0 is "as soon as the graph can"; the offset is where in the
;; buffer to begin. Clamped because starting past the end is a range
;; error rather than a no-op in some engines.
(.start node 0 (-> (at-renderer st) (max 0) (min (or seconds 0))))
node)))
(defn- spin-down! [{:keys [^js node]}]
(when node
(try (.stop node) (catch :default _ nil))
(try (.disconnect node) (catch :default _ nil))))
(defn- restart!
"Begin a new timeline at `anchor`, running at `rate` or held when it is 0.
A RESET rather than a segment, for the three cases that have nothing in flight
worth honouring: a play starts fresh, a seek means the user has jumped, and a
pause wants one stable number for the readout rather than a position that
creeps forward as the output buffer drains."
[state anchor rate]
(let [st @state]
(spin-down! st)
(swap! state assoc
:segs [{:from (.-currentTime ^js (:ctx st)) :anchor anchor :rate rate}]
:node nil)
(when (pos? rate)
(swap! state assoc :node (spin-up! @state)))))
(deftype Graph [state]
t/Transport
(-position [_] (position @state))
(-duration [_] (:seconds @state))
(-playing? [_] (let [st @state] (boolean (and (running? st) (not (done? st))))))
(-rate [_] (:rate @state))
(-set-rate! [_ r]
;; A SEGMENT, not a reset — the one case where what is already in flight has
;; to keep its old rate or the playhead lurches. See `raw`.
(let [st @state
now (.-currentTime ^js (:ctx st))]
(if (running? st)
(let [anchor (at-renderer st)]
(swap! state #(-> %
(assoc :rate r)
(update :segs (fn [segs]
(prune (conj segs {:from now :anchor anchor :rate r})
(- now (latency ^js (:ctx st))))))))
(when-let [^js node (:node st)]
(set! (.-value (.-playbackRate node)) r)))
;; Stopped: nothing is in flight and nothing is rendering, so the rate
;; is simply what the next play will run at.
(swap! state assoc :rate r))))
(-play! [_]
(let [st @state]
;; `done?` counts as not playing. It is DERIVED — no flag is cleared when
;; the take runs off its end — so testing "running" alone would make play
;; a dead button from the moment the audio finished.
(when (or (not (running? st)) (done? st))
;; The context starts suspended and only a gesture may resume it. The
;; transport button IS the gesture, so a rejection here means a
;; programmatic play and is noise rather than news — as on the element.
(some-> (.resume ^js (:ctx st)) (.catch (fn [_])))
;; Play at the end starts over, which is what the element does.
(restart! state (if (done? st) 0 (position st)) (:rate st)))))
(-pause! [_]
(let [st @state]
(when (running? st)
;; Anchored at what was HEARD, not at what was scheduled, so play after
;; pause resumes from where the sound stopped. It re-plays the last few
;; milliseconds rather than skipping them, which is the kinder of the
;; two roundings.
(restart! state (position st) 0))))
(-seek! [_ seconds]
(restart! state seconds (if (running? @state) (:rate @state) 0)))
(-set-loop! [_ on?]
(swap! state assoc :loop? (boolean on?))
;; The audio thread reads `loop` every quantum, so a live node picks this up
;; without being restarted — the same as setting `loop` on a playing element.
(when-let [^js node (:node @state)]
(set! (.-loop node) (boolean on?))))
(-set-muted! [_ on?]
(set! (.-value (.-gain ^js (:out @state))) (if on? 0 1)))
(-release! [_]
;; The source node is wired to the output the whole app shares, so a backend
;; dropped while playing would go on being heard under the one that replaced
;; it. Nothing else here needs releasing: the context and its gain outlive
;; every backend by design.
(let [st @state]
(spin-down! st)
(swap! state assoc
:segs [{:from (.-currentTime ^js (:ctx st))
:anchor (position st) :rate 0}]
:node nil))))
(defn backend
"A clock on `buffer`, `seconds` long. A nil buffer is a silent clock of that
length — see the namespace docstring.
The context and output are injected by the four-argument form so the whole of
this is assertable against a fake in node, where there is no Web Audio."
([buffer seconds] (backend (context) (output) buffer seconds))
([^js ctx ^js out buffer seconds]
(->Graph (atom {:ctx ctx :out out :buffer buffer :seconds seconds
:rate 1.0 :loop? false :node nil
:segs [{:from (.-currentTime ctx) :anchor 0 :rate 0}]}))))

38
frontend/src/arthur/clock/transport.cljs
View file

@ -1,38 +0,0 @@
(ns arthur.clock.transport
"What the clock needs of a thing that plays sound, and nothing more.
`arthur.clock` does the arithmetic — the frame derivation, the clamping, the
exposure grid — and a backend only has to answer where the sound has got to
and do as it is told. Keeping the protocol this thin is what makes the two
implementations comparable: if the graph backend and the element backend
disagree about a take's sync, the difference is in these nine methods and not
in anything derived from them.
POSITION IS WHERE THE SPEAKER IS, not where the renderer is. A backend that
schedules audio ahead of the output owes the difference back here — see
`arthur.clock.graph` — because the picture is drawn against this number and a
lip-sync tool that draws against the scheduler leads the sound it is matching.")
(defprotocol Transport
(-position [this]
"Seconds into the audio, as heard. Never negative, never past `-duration`,
and wrapped rather than clamped while looping.")
(-duration [this]
"Seconds of audio, or nil when the backend has not been told yet.")
(-playing? [this]
"Running AND not finished. A backend that has reached its end reports false
even if nothing told it to stop, because the end of the sound is the
authority on playback having stopped — see `ui/player`'s loop.")
(-rate [this])
(-set-rate! [this r])
(-play! [this])
(-pause! [this])
(-seek! [this seconds]
"Put the audio at `seconds`. Reading `-position` back must give the same
number, which is what makes a scrub idempotent.")
(-set-loop! [this on?])
(-set-muted! [this on?])
(-release! [this]
"Give up whatever this backend holds, because something else is about to be
the clock. NOT a pause: a backend that owns nothing has nothing to do here,
and the position it was last at is no longer anybody's business."))

53
frontend/src/arthur/core.cljs
View file

@ -1,53 +0,0 @@
(ns arthur.core
"The app's entry point.
port-plan step 3: the hand-written scene plays at 30fps against audio, scrubs,
and runs at ½× and ¼×."
(:require [arthur.db :as db]
[arthur.events.collab :as collab]
[arthur.events.footage :as footage]
[arthur.events.history :as history]
[arthur.events.playback]
[arthur.events.paint]
[arthur.events.project :as project]
[arthur.events.ui]
[arthur.subs.playback]
[arthur.subs.render]
[arthur.subs.ui]
[arthur.ui.index :as index]
[arthur.ui.player :as player]
[arthur.ui.shell :as shell]
[arthur.ui.tools :as tools]
[re-frame.core :as rf]
[reagent.dom.client :as rdc]))
(defonce root (atom nil))
(rf/reg-event-db ::init (fn [_ _] db/default))
(defn ^:dev/after-load mount []
;; A hot reload changes the scene or the rasteriser and not the playhead, so
;; the loop would otherwise sit on an unchanged frame number and never redraw.
(rf/clear-subscription-cache!)
(player/refresh-subs!)
(rdc/render @root [:<> [shell/view] [index/view]]))
(defn init []
(rf/dispatch-sync [::init])
;; A blank document, before the first render. Synchronous for the same reason
;; `::init` is: the shell reads the clip's dimensions, and mounting against a
;; db that has no clip in it yet is a frame of nothing for no reason.
(rf/dispatch-sync [::project/new])
;; What the server already holds, asked for once. The list is small — a row per
;; ingested take — and having it before the first click is what lets the footage
;; picker be a picker rather than a path to type.
(rf/dispatch [::footage/refresh])
(rf/dispatch [::project/list-symbols])
;; After the blank document, so an address that names a project opens it over
;; the blank one, and the blank one is what a bad address leaves on screen.
(collab/start!)
(history/install-keys!)
(tools/install-keys!)
(reset! root (rdc/create-root (js/document.getElementById "app")))
(mount)
(player/start!))

195
frontend/src/arthur/db.cljs
View file

@ -1,195 +0,0 @@
(ns arthur.db
"app-db: authored data and ids. Nothing derived, and nothing large.
That sounds like hygiene and it is the precondition for two things that are
otherwise unbuildable — spec validation on every event, which is only
affordable over authored data, and cheap writes, since every edit `assoc`es
into this map and every mounted layer-2 sub compares the result.
So the clip is here (it is a document — a human placed every node) and dense
channel blocks are not; they live behind a handle in `store`. The hand-written
demo clip has no dense blocks and its store is empty; the swarm and the take
are entirely dense."
(:require [arthur.demo :as demo]
[arthur.domain.clip :as domain-clip]
[arthur.demo.swarm :as swarm]
[arthur.demo.take :as take]))
(defn- entry
"A clip plus what the transport and the stage read off it.
Read OFF the clip rather than written again beside it: copying a number by hand
into this table is how it comes to disagree with the document it describes.
There is no `:frames` here, because a length belongs to a symbol and which
symbol is open is the editor's state — see `events/playback/frames`."
[label-key label clip store]
(merge {:label label :clip clip :store store
;; A static asset since step 9, and not the repo root's `audio.wav`.
;; That file is `extract.sh`'s output — tier 3, which the backend now
;; serves by hash — and the synthetic take needs a sound of its own so
;; that the clock has something to run against with no footage ingested.
:audio "/static/arthur/audio.wav"
:cid (name label-key)}
(select-keys clip [:fps :width :height])))
(def clips
"The hand-made clips, selectable from the transport.
`:swarm` is the load test: a hundred and twenty nodes, entirely dense. The two
takes are step 5's deliverable and they are ONE freeze — the same blocks, with
`:head` written as a dense track in one and as framed identity in the other, so
the button that switches between them switches a document field and nothing
else."
{:demo {:label "demo" :entry (delay (entry :demo "demo" demo/clip nil))}
:swarm {:label "swarm" :entry (delay (entry :swarm "swarm" @swarm/clip @swarm/store))}
:take {:label "take" :entry (delay (entry :take "take" @take/clip @take/store))}
:take-locked {:label "locked" :entry (delay (entry :take-locked "locked" @take/locked @take/store))}})
(defn clip-entry [id]
(some-> (get-in clips [id :entry]) deref))
(def tracing
"How tracing layers show on the stage: all of them or none, how strongly, and
which ones are switched off, as `[symbol-id node-id]` — the symbol a tracing
placement is in and its id, so a face's footage is one layer wherever the face
is placed.
THE EDITOR'S, NOT THE DOCUMENT'S. Showing a reference is a way of looking at
the stage, like solo and zoom: not an undo step, not sent to collaborators,
and it cannot reach an export. A layer is shown unless it is in `:hidden`, so
footage brought in shows without being found and switched on first — once
tracing itself is on, which it is not until asked for."
{:on? false :opacity 0.5 :hidden #{}})
(def default
{;; --- the document ---
;;
;; NOTHING IS LOADED. `core/init` dispatches `::project/new` before the first
;; render, so the app opens on a blank stage rather than on whichever built-in
;; scene happened to be convenient — the demos, the swarm and the two takes are
;; rows in the media pool like anything else, and reference material is not a
;; default. The values below are what a blank document is; they are replaced by
;; that dispatch and exist so this map is a valid db on its own.
:clip/current nil
:paint/revision 0
:palette :arthur/default ; a NAME; the ramp itself is project data
;; --- the clip ---
;;
;; Including the STAGE DIMENSIONS, which are the project's and not the
;; footage's. That is what deleting `makeXform` buys — the framing became a
;; transform on a node, so nothing downstream of the freeze knows the frame
;; size — and it is why ui/player no longer hardcodes 320x200.
:clip (let [c (domain-clip/blank)]
{:fps (:fps c)
:width (:width c) :height (:height c)
:audio nil})
;; Which ingested footage to detect, and what the last load said. The list
;; comes from the server — tier 3 is the backend's since step 9 — so there is
;; no path to type any more.
:footage {:id nil :label nil :loading? false :status nil
:available [] :chosen nil :uploaded #{}}
;; Every symbol in every saved project, for the pool's all-assets folder. Rows
;; from `/api/symbols`, nothing loaded: a symbol from elsewhere is fetched when
;; it is dropped.
:assets {:symbols [] :palettes [] :loading? false}
;; The document's own identity on the server. `:seq` is the monotonic project
;; version: a client that sees a delta with `seq > local + 1` refetches, which
;; is what will make staleness self-healing once there is a broadcast to miss.
:project {:id nil :cid nil :name nil :seq nil :busy? false :status nil}
;; What the server holds, for the open menu. A list of rows and nothing more —
;; opening one fetches the document itself.
:projects {:items [] :loading? false}
;; --- transport ---
;;
;; The playhead is in app-db like everything else. An earlier draft of
;; docs/architecture.md put it in a standalone ratom to dodge an invalidation
;; storm that does not happen: with layer-2 extractors and layer-3
;; computations, a tick re-runs one cheap extractor per mounted sub, each
;; returning the same value for every subtree the tick did not touch, and
;; therefore notifying nobody. ::resolver does not re-run.
;;
;; Two reasons it belongs here rather than outside: a seek in the event log is
;; how scrubbing becomes inspectable in re-frame-10x, and a collaborator's
;; playhead is a feature — putting it outside app-db puts it outside the
;; machinery that would share it.
;; --- export ---
;;
;; The REQUEST and its progress, never the frames. Which symbol to write and
;; at what integer zoom is authored state like anything else; the megabytes the
;; render produces are handed straight to a download and never enter the db.
;; `:isolate` is the placement to render alone, or nil for the whole symbol;
;; `:symbol` nil means whichever symbol is open.
:export {:symbol nil :isolate nil :zoom 4 :busy? false :done 0 :total 0
:status nil}
:playback {:frame 0
:playing? false
:rate 1.0
;; Both for profiling: loop so a run at 4x lasts longer than the
;; clip, mute so sitting in one does not require enduring it.
:loop? false
:muted? false}
;; --- the editor's own state ---
;;
;; IN app-db, not in ratoms beside the components that read it. What is
;; selected is asked by four panes at once — the params pane renders it, the
;; timeline highlights its row, the stage draws its handles, the palette says
;; which tone a new shape gets — and a `defonce` atom private to one namespace
;; can only be shared by making the other three require that namespace for its
;; state. It is also small and authored, which is the bar `arthur.db` sets.
;;
;; `:selection` is a vector whose first element says what kind of thing it
;; names, so a pane dispatches on it rather than on which of several
;; "selected-x" keys happens to be non-nil:
;;
;; [:node <symbol> <node>] a shape or an instance
;; [:symbol <id>] a symbol
;; [:subject <id>] [:feature <id>] [:group <id>] a tracked object
;;
;; `:draft` is the polygon being clicked out, flat [x y x y …] as geometry is
;; stored everywhere. `:expanded` holds timeline row PATHS — a path and not a
;; node id, because one symbol placed twice is two rows that open separately.
;;
;; `:open` is the symbol on screen — the one the stage draws, the timeline
;; lists, the transport plays and a new shape goes into — and `:tabs` the
;; symbols open beside it. Editor state and not the document's, because no
;; symbol is special to the document: which one you are looking at is a fact
;; about you.
;;
;; `:knobs` holds a generated setting's value WHILE THE REGENERATION IS IN
;; FLIGHT, keyed by [scope id knob]. Moving a slider dispatches a preview that
;; re-freezes blocks asynchronously, so until it lands the clip still reports
;; the old value — and a slider reading from the clip would spring back under
;; the user's finger on every frame of the drag.
:ui {:open nil
:tabs []
:selection nil
:selections []
:tone 1
:tool :select
:brush 6
:auto-key? false
:draft []
:knobs {}
:tracing tracing
:expanded #{}}})
(def rates
"The transport's rates — all of them `playbackRate` on the audio element, so
the picture cannot drift from the sound at any of them.
2x and 4x are there to be profiled at rather than watched. A 30fps clip at 2x
wants sixty clip frames a second against a 60Hz display, so every animation
frame has to paint a new one: it is the point where the loop stops having
slack. Past that the clock starts dropping frames rather than falling behind,
which is the whole reason the frame is derived from the audio instead of
counted — and the transport reports the drop rate so that it is visible rather
than merely survivable."
[0.25 0.5 1.0 2.0 4.0])

29
frontend/src/arthur/demo.cljs
View file

@ -1,29 +0,0 @@
(ns arthur.demo
"The hand-written clip from port-plan step 2, and nothing else.
The EDN is a resource rather than a literal in this file so that the test and
the page read the SAME bytes. If the scene were written twice, the one the test
validates would not be the one that renders, and the model would be validated
against a scene nobody ever looked at."
(:require [arthur.domain.clip :as domain-clip]
[arthur.domain.palette :as pal]
[arthur.domain.symbol :as symbol]
[cljs.reader :as reader]
[shadow.resource :as rc]))
(def source (rc/inline "arthur/demo/scene.edn"))
(def clip (reader/read-string source))
(def main
"The scene's one symbol: what an evaluator takes. `clip` is the document."
(domain-clip/symbol clip :main))
(def fps (:fps clip))
(def frames (domain-clip/frames clip :main))
(defn ops-at
"Draw ops for one frame, via the specification path. The page uses
`symbol/resolver` instead; this is here for the REPL."
[f]
(symbol/eval-frame main f nil pal/index-of nil))

102
frontend/src/arthur/demo/scene.edn
View file

@ -1,102 +0,0 @@
;; A scene written by hand, before any analysis exists.
;;
;; port-plan step 2 is deliberately ahead of measurement: the data model has
;; never been validated, and it is worth finding out here, with fifty lines to
;; throw away, rather than after nine hundred lines of measurement have been
;; ported into a shape that does not work.
;;
;; So this is not a demo of a face. It is the smallest scene that exercises every
;; mechanism the model claims to have, chosen so that each one is visible when it
;; breaks:
;;
;; exposure inherited from the clip root the motion steps on 2s
;; a keyed [:xform :pos], sparse, held the card jumps between 4 poses
;; transform composition through a group the eye rides the card
;; rotation about an anchor the card turns, it does not swing
;; a stencil as a colour key the iris cannot leave the card
;; a stencil chain nor can the pupil
;; a keyed [:vis] the bar blinks off and back
;; a :span the bar does not exist at either end
;; fractional z among siblings the bar is behind, the pupil in front
;;
;; Everything is in 320x200 raster space, which is what [:geom :pts] holds.
{:name "step-2 demo"
;; 229 frames at 30fps is 7.63s, which covers audio.wav (7.601s) with a frame to
;; spare. fps belongs to the CLIP rather than to the timeline — a timeline has a
;; frame space, not a rate — and it is here only because there is one clip.
:fps 30
;; The STAGE, in pixels. The project's dimensions, not the footage's — which is
;; what makes `makeXform` deletable: placement is a transform on a node and the
;; stage clips whatever hangs off. Here everything is authored in stage pixels
;; already, because a hand-written scene is a painted one.
:width 320
:height 200
:symbols
{:main
{:id :main
:frames 229
:nodes
{;; The clip root. EXPOSURE LIVES HERE and is inherited, because
;; docs/design.md is emphatic that everything rides one grid: a head cutting on
;; odd frames against a mouth cutting on even ones reads as two performances.
;; Setting it lower on a child is possible and is meant to feel deliberate.
:root
{:id :root :name "clip" :kind :group :parent nil :z "a1"
:time {:mode :map :expose 2}}
;; A bar, behind everything, purely to assert that :vis and :span are different
;; questions. It stops existing outside [6 66) — nothing to hide, nothing to
;; hold — and inside that range it is switched off between frames 76 and 153.
:bar
{:id :bar :name "bar" :kind :poly :parent :root :z "a0"
:span [19 210]
:channels
{[:vis] {:animated? true :interp :hold :keys {0 true, 76 false, 153 true} :over []}
[:geom :pts] {:animated? false :value [20 168 300 168 300 176 20 176]}
[:style :color] {:animated? false :value :brow}}}
;; The group the plan asks for: four sparse keys on [:xform :pos], held. At
;; exposure 2 the card reads its pose from an even frame, so a key landing on
;; an odd frame would be seen on the even frame after it — which is the whole
;; reason exposure is applied before anything else and not folded into keys.
:swing
{:id :swing :name "swing" :kind :group :parent :root :z "a1"
:channels
{[:xform :pos] {:animated? true :interp :hold
:keys {0 [90.0 100.0], 57 [200.0 70.0], 114 [230.0 140.0], 171 [110.0 150.0]}
:over []}
[:xform :rot] {:animated? true :interp :hold
:keys {0 0.0, 57 0.35, 114 0.0, 171 -0.35}
:over []}}}
;; The rectangle. Its points are centred on the origin and its :anchor is the
;; origin too, so :swing's rotation TURNS it rather than swinging it round a
;; corner — which is the failure mode :anchor exists to prevent.
:card
{:id :card :name "card" :kind :poly :parent :swing :z "a1"
:channels
{[:geom :pts] {:animated? false :value [-44 -30 44 -30 44 30 -44 30]}
[:style :color] {:animated? false :value :skin-base}}}
;; A disc stencilled by the card. The stencil is a COLOUR KEY, not a node
;; reference — it is the take format's clip= — so the iris is written only over
;; pixels that currently hold the card's index. Push the radius up and it is
;; cropped by the card's edge rather than spilling, at any position, with no
;; clamp anywhere.
:iris
{:id :iris :name "iris" :kind :disc :parent :card :stencil :card :z "a2"
:channels
{[:xform :pos] {:animated? false :value [14.0 -6.0]}
[:geom :radius] {:animated? false :value 13.0}
[:style :color] {:animated? false :value :iris}}}
;; The pupil is a SQUARE, and three pixels of it. A circle of radius 1.5 is not
;; a circle, it is a plus sign with the corners gnawed off, and it changes shape
;; as it moves. Stencilled by the iris, which is itself already cropped by the
;; card, so the clip composes without the chain being expressed anywhere.
:pupil
{:id :pupil :name "pupil" :kind :rect :parent :iris :stencil :iris :z "a3"
:channels
{[:geom :size] {:animated? false :value 5.0}
[:style :color] {:animated? false :value :pupil}}}}}}}

120
frontend/src/arthur/demo/stage.cljs
View file

@ -1,120 +0,0 @@
(ns arthur.demo.stage
"A saved take placed seven times on a stage. The EDN is the authored layout."
(:require [arthur.domain.channel :as ch]
[cljs.reader :as reader]
[shadow.resource :as rc]))
(def layout (reader/read-string (rc/inline "arthur/demo/stage_8625.edn")))
(defn- position-track
"Where the peg sits over time: its `center` plus a slow two-axis drift. The
peg's own position, so the stage point the face is pinned to is what moves —
not an offset that has to be kept in step with a changing scale."
[center drift phase frames]
(let [base center
[dx dy] drift
wave (fn [f period] (js/Math.sin (* 2 js/Math.PI (/ (+ f phase) period))))
x0 (wave 0 96)
y0 (wave 0 132)]
(ch/keyed
(into {}
(for [f (conj (vec (range 0 frames 20)) (dec frames))]
[f [(+ (first base) (* dx (- (wave f 96) x0)))
(+ (second base) (* dy (- (wave f 132) y0)))]]))
:linear)))
(defn compose
"The authored layout plus a source clip -> the composed stage document.
A PEG'S IDENTITY IS AUTHORED TOO, beside its instance's, for the reason the
instance's is: `compose` is a pure function of the layout, so a generated one
would make the same stage a different document on every call.
AN INSTANCE IS KEYED BY ITS :uuid, not by the authored id. The authored id
(`:left`, `:voice-right`) is a handle for reading the EDN and for the
`:linked-to` written there; it does not appear in the document this returns.
What replaces it is an identity that means one instance and nothing else: seven
instances of one symbol are seven different things to name — to export on their
own, to link a voice to, to point at later — and an id like `:left` is a
description of where a thing sits, which is exactly what changes when the stage
is re-arranged. `:name` carries the label for a human and `:source :symbol`
carries the symbol, so the node still says what it is and which drawing it
plays — and `:playback` says how time runs inside it, which is a separate
question from which drawing that is."
[source]
(let [{:keys [name width height frames symbol instances audio scale]} layout
;; Which point of the SOURCE is pinned to the stage `:center`. Its
;; middle, unless the layout says otherwise for an off-centre drawing.
default-origin (or (:origin layout)
[(/ (:width source) 2) (/ (:height source) 2)])
original (get-in source [:symbols :main])
;; Authored id -> uuid, so the `:linked-to` in the EDN resolves to the
;; identity the document uses. Built before either pass because the audio
;; nodes refer to the instances.
by-id (into {} (map (juxt :id :uuid)) (concat instances audio))
uuid-of (fn [what id]
(or (get by-id id)
(throw (ex-info "the stage layout names an instance that is not there"
{:in what :id id
:known (vec (sort-by str (keys by-id)))}))))
;; EVERY PLACEMENT IS A PEG AND AN INSTANCE, and this layout is what
;; makes the pair necessary rather than tidy. `:scale` is KEYED — the
;; faces pulse — and it has to happen about the point the face is pinned
;; to. A stored `[:xform :anchor]` used to buy that: a static position
;; and a moving scale, turning about a fixed point. No static position
;; can do it alone, because `T(pos)·S(k(f))` moves the source's middle
;; whenever `k` changes, so holding it still would mean keying `pos` in
;; lockstep with `scale` — two channels that have to agree frame for
;; frame, which is the thing channels exist to avoid.
;;
;; A peg does it with neither:
;;
;; peg pos = center (+ drift), scale = k(f)
;; └ face pos = -origin
;;
;; world = T(center)·S(k(f))·T(-origin)
;;
;; which takes `origin` to `center` for EVERY k, with nothing keyed that
;; was not keyed before. That is the same matrix the anchor produced —
;; `node-test` asserts the identity — and it is reachable, keyable and
;; selectable, which the anchor was not.
;;
;; THE PEG IS THE PLACEMENT: it carries WHERE (pos, scale) and WHEN
;; (`:at`, `:span`), and the face under it carries only which drawing and
;; the offset to its origin. The time map has to be the peg's, because
;; `:scale` is read in the placement's own frames — that is what staggers
;; the entrances' growth — and `:span` goes with it so that
;; `node/placed-span` still answers where the placement sits on the stage.
;; The face then reads its peg's frames as its own and shows whenever the
;; peg does.
nodes (into
{:root {:id :root :name "stage" :kind :group :z "a1"}}
(mapcat (fn [{:keys [uuid peg name z span at center origin drift phase]}]
(let [origin (or origin default-origin)]
[[peg {:id peg :name name :kind :group
:parent :root :z z :span span
:time {:mode :map :at at :rate 1}
:channels {[:xform :pos]
(if drift
(position-track center drift phase frames)
(ch/framed center))
[:xform :scale] scale}}]
[uuid {:id uuid :name (str name " face") :kind :instance
:parent peg :z "a1"
:source {:symbol symbol}
:channels {[:xform :pos]
(ch/framed (mapv - origin))}}]]))
instances))
nodes (into nodes
(map (fn [{:keys [uuid linked-to z source span at gain pan]}]
[uuid {:id uuid :kind :audio :parent :root :z z
:linked-to (uuid-of uuid linked-to)
:source source :span span
:time {:mode :map :at at :rate 1}
:channels (cond-> {[:audio :gain] gain}
pan (assoc [:audio :pan] pan))}])
audio))]
(assoc source :name name :width width :height height
:symbols (assoc (:symbols source)
:main {:id :main :frames frames :nodes nodes}
symbol (assoc original :id symbol)))))

89
frontend/src/arthur/demo/stage_8625.edn
View file

@ -1,89 +0,0 @@
;; A local stage sketch. The source is the saved, post-processed IMG_8625.MOV
;; clip in the project store; its dense channel blocks are shared by all seven
;; instances. Centers, drift and timing are authored in stage pixels and frames.
{:source-project "4379f900-bdd2-409b-acf6-32081f8ce01f"
:source-cid "f8cace9e-4ad3-4796-973c-c62eeebe3d01"
:symbol :sym/face-8625
:name "8625 stage study"
:width 320 :height 200 :frames 280
;; Each :center below places the source clip's center on the stage; :origin
;; overrides which point of the source that is, for an off-centre drawing.
;; :peg is the identity of the transform node the placement hangs off — it
;; carries :center and :scale, the face below it carries -:origin, and that pair
;; is what makes a KEYED scale happen about the pinned point. See demo/stage.
;; Each placement reads this pulse in its own local time, so the staggered
;; entrances start their growth at different moments on the master timeline.
:scale {:animated? true :interp :linear
:keys {0 [0.4 0.4], 12 [0.56 0.56], 24 [0.48 0.48],
48 [0.52 0.52], 72 [0.48 0.48], 96 [0.52 0.52],
120 [0.48 0.48], 144 [0.52 0.52], 168 [0.48 0.48],
192 [0.52 0.52], 216 [0.48 0.48], 240 [0.52 0.52],
279 [0.48 0.48]}
:over []}
;; Audio placements are ordinary timeline nodes with channel parameters.
;; :linked-to is an editorial link; their spans and time maps are independent.
;; A span is in the placement's OWN frames and :at is where its frame 0 lands on
;; the stage, so every entrance below plays from its own start.
:audio
[{:id :voice-left :uuid #uuid "eeaa49c3-1238-469f-bf54-44929e379f6b"
:linked-to :left :z "a3"
:source {:footage "f8cace9e-4ad3-4796-973c-c62eeebe3d01"}
:at 0 :span [0 280]
:gain {:animated? false :value 1.0}}
{:id :voice-right :uuid #uuid "468239dd-0e3a-4e5c-ac6f-1858430a0355"
:linked-to :right :z "a4"
:source {:footage "f8cace9e-4ad3-4796-973c-c62eeebe3d01"}
:at 48 :span [0 212]
:gain {:animated? true :interp :linear
:keys {48 0.0, 60 1.0, 90 0.35, 115 0.9, 145 0.45,
170 1.0, 195 0.4, 220 0.85, 245 1.0, 259 0.0}
:over []}
:pan {:animated? true :interp :linear
:keys {48 -0.8, 90 -0.8, 130 0.7, 175 0.7, 220 -0.65, 259 0.65}
:over []}}]
;;
;; EVERY PLACEMENT CARRIES A :uuid, and it is authored here rather than generated
;; in `compose`. The uuid is the node's identity in the composed document — it is
;; the key in the timeline's node map — so generating one per load would give the
;; same stage a different document on every load, and nothing that refers to a
;; placement (`:linked-to` above, an export target in the UI, a comment in a
;; review) could survive a reload. The `:id` beside it stays as the AUTHORING
;; handle: it is what the reader of this file uses to see which placement is
;; which, and what the `:linked-to` above names, and `compose` resolves it to the
;; uuid. Nothing downstream of `compose` sees the authored id.
:instances
[{:id :left :uuid #uuid "ee7321c8-faf1-46d7-8029-37771898accb"
:peg #uuid "b1e0a7c4-5f3d-4a8e-9c21-70e4d1a90f01"
:name "8625 left" :z "a1"
:at 0 :span [0 280]
:center [40 40] :drift [3 2] :phase 0}
{:id :right :uuid #uuid "1aa0da78-b4ed-4bb6-8d70-b09a3ec5e2c3"
:peg #uuid "b1e0a7c4-5f3d-4a8e-9c21-70e4d1a90f02"
:name "8625 right" :z "a2"
:at 48 :span [0 232]
:center [120 40] :drift [-3 2] :phase 17}
{:id :top-third :uuid #uuid "23bb697d-eba7-4af6-a86c-606c50107088"
:peg #uuid "b1e0a7c4-5f3d-4a8e-9c21-70e4d1a90f03"
:name "8625 top third" :z "a5"
:at 24 :span [0 256]
:center [200 40] :drift [2 -3] :phase 31}
{:id :top-fourth :uuid #uuid "f4f0241d-026e-4e50-9bea-a4ccde896d8a"
:peg #uuid "b1e0a7c4-5f3d-4a8e-9c21-70e4d1a90f04"
:name "8625 top fourth" :z "a6"
:at 72 :span [0 208]
:center [280 40] :drift [-2 -2] :phase 49}
{:id :bottom-left :uuid #uuid "8f594d72-a97f-4a32-82fd-08d1670a2218"
:peg #uuid "b1e0a7c4-5f3d-4a8e-9c21-70e4d1a90f05"
:name "8625 bottom left" :z "a7"
:at 96 :span [0 184]
:center [70 135] :drift [3 -2] :phase 63}
{:id :bottom-middle :uuid #uuid "63f3fb32-9e94-4d68-a1c2-12e6de2d04b5"
:peg #uuid "b1e0a7c4-5f3d-4a8e-9c21-70e4d1a90f06"
:name "8625 bottom middle" :z "a8"
:at 120 :span [0 160]
:center [160 135] :drift [-2 3] :phase 81}
{:id :bottom-right :uuid #uuid "fa338701-cb21-4d45-89f1-a5e706f045ec"
:peg #uuid "b1e0a7c4-5f3d-4a8e-9c21-70e4d1a90f07"
:name "8625 bottom right" :z "a9"
:at 144 :span [0 136]
:center [250 135] :drift [2 2] :phase 107}]}

168
frontend/src/arthur/demo/swarm.cljs
View file

@ -1,168 +0,0 @@
(ns arthur.demo.swarm
"A hundred and twenty shapes, orbiting, spinning, pulsing and blinking.
Not useful. It is here because it is the first thing to exercise the DENSE
channel path end to end — typed-array blocks behind a store handle, one value
per frame, read through a cursor — which until now had tests and no traffic.
Step 5 writes exactly this shape out of the freeze module, so it is worth
knowing the resolver can carry it at rate before anything depends on that.
Everything is generated from deterministic trigonometry rather than from a
random seed: the same scene every load, so a stutter or a wrong pose is
reproducible instead of being a thing that happened once.
Layout of each block is the rectangular one freeze produces — node-major,
frame-minor, no per-frame header and no indirection:
offset(node i) = i · frames · stride
value(i, f) = data[offset(i) + f · stride]"
(:require [arthur.domain.channel :as ch]
[arthur.domain.palette :as pal]))
(def frames 229)
(def fps 30)
(def n-orbits 6)
(def n-shapes 120)
(def ^:private TAU (* 2 js/Math.PI))
;; Every tone except the background, so the swarm uses the whole ramp.
(def ^:private tones
(vec (remove #{:bg} (map :name pal/entries))))
(defn- regular-poly
"A closed n-gon about the origin, flat in [x0 y0 x1 y1 …] — the same layout a
dense block holds, which is the point of geometry being flat everywhere."
[n radius phase]
(vec (mapcat (fn [k]
(let [a (+ phase (/ (* TAU k) n))]
[(* radius (js/Math.cos a))
(* radius (js/Math.sin a))]))
(range n))))
;; ---------------------------------------------------------------------------
;; the dense blocks
(defn- fill-block!
"Write one node's frames into a node-major block."
[^js data i stride f->vals]
(let [base (* i frames stride)]
(dotimes [f frames]
(let [vs (f->vals f)
o (+ base (* f stride))]
(dotimes [k stride]
(aset data (+ o k) (nth vs k)))))))
(defn- orbit-blocks []
(let [pos (js/Float32Array. (* n-orbits frames 2))
rot (js/Float32Array. (* n-orbits frames 1))]
(dotimes [i n-orbits]
(let [ph (/ (* TAU i) n-orbits)
;; Lissajous, so the six orbits drift in and out of phase with each
;; other instead of marching in step.
wx (+ 0.011 (* 0.004 (mod i 3)))
wy (+ 0.017 (* 0.003 (mod i 4)))
spin (* 0.008 (if (even? i) 1 -1) (inc (mod i 3)))]
(fill-block! pos i 2
(fn [f] [(+ 160 (* 104 (js/Math.sin (+ (* f wx) ph))))
(+ 100 (* 64 (js/Math.sin (+ (* f wy) (* 1.7 ph)))))]))
(fill-block! rot i 1 (fn [f] [(* f spin)]))))
{"swarm/orbit-pos" {:data pos :state nil}
"swarm/orbit-rot" {:data rot :state nil}}))
(defn- shape-blocks []
(let [pos (js/Float32Array. (* n-shapes frames 2))
rot (js/Float32Array. (* n-shapes frames 1))
scale (js/Float32Array. (* n-shapes frames 2))
;; The state mask: a handful of shapes wink out entirely for a stretch.
;; ABSENT, not hidden — this is the mask meaning "there is no value on
;; this frame", which is what an occluded subject will mean at step 6.
state (js/Uint8Array. (* n-shapes frames))]
(dotimes [i n-shapes]
(let [ph (/ (* TAU i) n-shapes)
ring (+ 18 (* 26 (js/Math.abs (js/Math.sin (* 2.3 ph)))))
wob (+ 0.03 (* 0.02 (mod i 5)))
spin (* (if (zero? (mod i 3)) -1 1) (+ 0.02 (* 0.011 (mod i 7))))
pulse (+ 0.05 (* 0.013 (mod i 6)))]
(fill-block! pos i 2
(fn [f]
;; Orbit position plus a small independent wobble, so no
;; two neighbours trace the same path.
(let [a (+ ph (* f 0.014 (if (even? i) 1 -1)))]
[(+ (* ring (js/Math.cos a)) (* 5 (js/Math.sin (* f wob))))
(+ (* ring (js/Math.sin a)) (* 5 (js/Math.cos (+ 1.1 (* f wob)))))])))
(fill-block! rot i 1 (fn [f] [(+ ph (* f spin))]))
(fill-block! scale i 2
(fn [f]
(let [s (+ 1.0 (* 0.45 (js/Math.sin (+ ph (* f pulse)))))]
[s s])))
;; Every eleventh shape is absent for a window that moves with i.
(when (zero? (mod i 11))
(let [from (mod (* i 9) frames)
to (min frames (+ from 34))]
(doseq [f (range from to)]
(aset state (+ (* i frames) f) ch/absent-bit))))))
{"swarm/pos" {:data pos :state state}
"swarm/rot" {:data rot :state nil}
"swarm/scale" {:data scale :state nil}}))
;; ---------------------------------------------------------------------------
;; the nodes
(defn- dense [store i stride]
{:animated? true :interp :hold
:dense {:store store :offset (* i frames stride) :stride stride :frames frames}
;; Provenance, which nothing in the renderer reads. Here it is honest about
;; where these numbers came from, the same way :roto/lips-outer will be.
:generated {:by :demo/swarm}
:over []})
(defn- orbit-node [i]
{:id (keyword (str "orbit-" i)) :kind :group :parent :root
:z (str "b" i)
:channels {[:xform :pos] (dense "swarm/orbit-pos" i 2)
[:xform :rot] (dense "swarm/orbit-rot" i 1)}})
(defn- shape-node [i]
(let [orbit (keyword (str "orbit-" (mod i n-orbits)))
tone (nth tones (mod i (count tones)))
kind (case (mod i 7) 5 :disc 6 :rect :poly)
verts (+ 3 (mod i 10))
size (+ 3.5 (* 0.9 (mod i 8)))
base {:id (keyword (str "s-" i)) :kind kind :parent orbit
;; Fractional index among siblings. Zero-padded so the strings
;; sort the way the numbers do — "c9" would otherwise land after
;; "c10", which is the classic way a z order goes subtly wrong.
:z (str "c" (.padStart (str i) 4 "0"))
:channels {[:xform :pos] (dense "swarm/pos" i 2)
[:xform :rot] (dense "swarm/rot" i 1)
[:xform :scale] (dense "swarm/scale" i 2)
[:style :color] (ch/framed tone)}}]
(update base :channels merge
(case kind
:poly {[:geom :pts] (ch/framed (regular-poly verts size (* 0.3 i)))}
:disc {[:geom :radius] (ch/framed (* 0.75 size))}
:rect {[:geom :size] (ch/framed (js/Math.round size))}))))
(def store
(delay (merge (orbit-blocks) (shape-blocks))))
(def clip
(delay
{:name "swarm"
:fps fps
:width 320
:height 200
:symbols
{:main
{:id :main
:frames frames
:nodes
(into {:root {:id :root :kind :group :parent nil :z "a1"
;; On 2s, like everything else. A hundred and twenty shapes
;; cutting on one grid reads as animation; the same shapes on
;; their own grids read as a screensaver, which is the whole
;; argument for exposure inheriting strictly.
:time {:mode :map :expose 2}}}
(concat (map (juxt :id identity) (map orbit-node (range n-orbits)))
(map (juxt :id identity) (map shape-node (range n-shapes)))))}}}))

98
frontend/src/arthur/demo/take.cljs
View file

@ -1,98 +0,0 @@
(ns arthur.demo.take
"The synthetic take: the whole vertical slice, with no video file in it.
This is port-plan step 5's deliverable. `flow/take` now composes the shared
measurement path for both this generator and real footage —
synth ──▶ measure/anchor ──▶ condition/anchor
│ │
└──▶ mouth, eyes, brows ◀─┘
│
condition/parts
│
FREEZE ──▶ channels on nodes
│
symbol/resolver ──▶ raster
— and the order of that diagram is the whole argument for the stage split. The
anchor fit is knob-free. Conditioning smooths its four parameters. The rings are
then measured THROUGH the conditioned transform, so `anchor avg` does re-run the
ring mapping — a few hundred frames of twenty points, free — and does not re-run
anything that reads a source pixel, because that part takes the landmarks and
the frames and never the transform.
TWO CLIPS, ONE STORE. `:take` reads each measured head transform in time;
`:take-locked` holds the measured transform from frame zero. They share the
same dense blocks; only the head node's anchor map differs."
(:require [arthur.flow.address :as address]
[arthur.flow.freeze :as freeze]
[arthur.flow.take :as take]
[arthur.synth :as synth]))
(def frames 229)
(def fps 30)
(def ^:private stage
;; The project's dimensions, and NOT the footage's. This is what deleting
;; `makeXform` buys: the head is placed and scaled on the stage by a transform
;; on a node, so a 1440x1920 portrait clip and a 320x200 stage are not a
;; conflict to resolve. Whatever hangs off the edge is clipped.
[320 200])
(def ^:private aspect
;; The synth writes x and y in the SAME unit, so its normalised space is already
;; isotropic and the anisotropy correction is the identity here. Real footage
;; passes W/H from the manifest at step 6. Worth knowing while reading anything
;; here: aspect 1 is the one setting at which a port that dropped the
;; anisotropy correction entirely would still look right, which is why
;; anchor-test exercises 0.5625 and this does not.
1)
(def analysis
"Stage 2's output, synthesised. A seeded generator, so a wrong pose is
reproducible rather than something that happened once."
(delay (synth/synth-dense frames {:seed 1})))
(def measured
"Stages 3 and 4, in the order the stage split requires."
(delay (take/measure (assoc take/knobs :aspect aspect :fps fps)
{:dense @analysis})))
(def params
"What the freeze was handed. Public because it is the honest way to re-freeze at
other settings — a test that built its own copy would be asserting about a clip
nobody looks at."
(merge take/knobs
{:name "take"
:fps fps
:stage stage
;; On 2s. docs/design.md is emphatic that everything rides ONE grid: a
;; head cutting on odd frames against a mouth cutting on even ones reads
;; as two performances, so exposure lives on the clip root and inherits.
:expose 2
;; The head as filmed. `:take-locked` is the same freeze with this one
;; field changed, which is the point.
:head :free
;; Provenance, and now a content address. There is no detector here, so
;; the generator IS the detector and its seed is the source: two synth
;; takes at different seeds are different analyses, which is the same
;; statement content addressing makes about two model versions.
:analysis (address/analysis {:detector "synth"
:version "mulberry32"
:seed 1
:frames frames
:fps fps
:aspect aspect})}))
(def subject :face-1)
(def frozen
(delay (freeze/clip params {subject @measured})))
(def store (delay (:store @frozen)))
(def clip (delay (:clip @frozen)))
(def locked
"The same blocks, with `:head` held at measured frame zero."
(delay (freeze/head-mode {:trace {:origin :start}} @frozen)))

151
frontend/src/arthur/domain/bring.cljs
View file

@ -1,151 +0,0 @@
(ns arthur.domain.bring
"Bringing symbols into a clip from another: out of a saved project, or out of
a freeze of new footage.
Copied, never linked. What comes in gets ids of its own where they are taken,
and editing it here does not touch where it came from. Tracking identities and
the analysis they were measured by come along only when the receiving clip can
hold them; otherwise what comes in is drawing, which plays but does not re-tune.
Plain data in and out — documents, and in `placed` a document with its store —
so the events that fetch them are only fetching."
(:refer-clojure :exclude [take])
(:require [arthur.domain.clip :as clip]
[arthur.domain.feature :as feature]
[arthur.domain.node :as node]
[clojure.string :as string]))
(defn symbols
"Copy symbols `roots` of clip `other`, and every symbol they place, into
`clip`. Returns `{:clip :ids}`, where `:ids` maps each copied symbol's id in
`other` to its id here.
AN ID THAT IS TAKEN IS RENAMED, never merged: two symbols that happen to share
an id are two drawings, and a cel's `:source :symbol` inside the copy is
rewritten to follow. `wanted` maps a root's id in `other` to the id it should preferably get,
which is how a symbol made from footage is called what the person typed rather
than `:main`.
Only symbols travel. What else `other` holds — tracking identities, an analysis
— is the caller's decision, because whether it can come too depends on what
`clip` already has."
[clip other roots wanted]
(let [;; A tree walk is safe because placing cannot make a cycle.
reach (into #{} (mapcat #(tree-seq any? (partial clip/places other) %)) roots)
ids (reduce (fn [ids sid]
(let [taken? #(or (contains? (:symbols clip) %)
(some #{%} (vals ids)))]
(assoc ids sid (clip/free-id taken? (get wanted sid sid)))))
{} (sort-by str reach))
;; Cel identity and timing stay put; content references follow
;; the symbol IDs assigned in the destination document.
repoint (fn [n ids]
(if (node/source n)
(update-in n [:source :symbol] ids)
n))
copy (fn [sid]
(-> (clip/symbol other sid)
(assoc :id (ids sid) :fps (or (:fps (clip/symbol other sid)) (:fps other)))
(update :nodes #(into {} (map (fn [[id n]] [id (repoint n ids)])) %))))]
{:clip (reduce (fn [c sid] (assoc-in c [:symbols (ids sid)] (copy sid))) clip reach)
:ids ids}))
(defn symbol-id
"An id for a symbol a person has named: the name, lower-cased and hyphenated,
or `:symbol` when nothing of it survives."
[label]
(let [slug (-> (str label) string/lower-case
(string/replace #"[^a-z0-9]+" "-")
(string/replace #"^-+|-+$" ""))]
(keyword (if (seq slug) slug "symbol"))))
(defn take
"Put `frozen`, a take, into `clip` as ONE symbol called `label`. Returns the
changed clip, the imported symbol id and the old-to-scoped subject ids.
`frozen` is what `flow/freeze/clip` makes: a `:main` that places one symbol per
tracked face. `:main` becomes the named symbol — it is what holds the faces in
stage pixels, so it is the thing worth placing. Each generated face symbol gets
an automatic association with the take's SOUND, source frames `range` of
footage `footage-id`. Thus the take is heard through the faces it places, and
a face subsequently placed by itself still brings its sound. This is the same
shape a future manual symbol/sound association can write; dropping a face is
not a special operation.
A multi-face take consequently reaches the same recording through several
symbols. `nest/audio-tracks` collapses simultaneous copies carrying the same
`:media-link`; placing those faces at different times still schedules each one.
The unique imported symbol id scopes every subject id. Thus two takes may both
arrive with detector subject `:face-1` without colliding in the document."
[clip frozen label footage-id range]
(let [scope (fn [root subject] (keyword (str (name root) "." (name subject))))
root (clip/free-id
(fn [candidate]
(or (contains? (:symbols clip) candidate)
(some #(contains? (:symbols clip) (scope candidate %))
(keys (:subjects frozen)))))
(symbol-id label))
scoped (partial scope root)
wanted (into {:main root :footage (scoped :footage)}
(map (fn [subject] [subject (scoped subject)]))
(keys (:subjects frozen)))
{c :clip ids :ids} (symbols clip frozen [:main] wanted)
sid (ids :main)
faces (mapv ids (sort-by str (keys (:subjects frozen))))
sound {:id :sound :name "sound" :kind :audio :parent nil
:z "z-sound" :source {:footage footage-id}
;; All automatic copies name one recording. The audio walk
;; uses this identity only to avoid mixing that recording once
;; per detected face when the complete take is played.
:media-link [:footage footage-id range]
;; Source frame `start` plays on the symbol's 0.
:span range
:time {:mode :map :at (- (first range)) :rate 1}}
feature-ids (into {}
(map (fn [[id f]]
[id (feature/owned (ids (:subject f)) (keyword (name id)))]))
(:features frozen))
subjects (into {}
(map (fn [[id subject]]
(let [new-id (ids id)]
[new-id (assoc subject :id new-id
:footage footage-id)])))
(:subjects frozen))
features (into {}
(map (fn [[id f]]
(let [new-id (feature-ids id)]
[new-id (-> f
(assoc :id new-id
:subject (ids (:subject f))
:symbol (ids (:symbol f))))])))
(:features frozen))
groups (into {}
(map (fn [[id g]]
(let [new-subject (ids (:subject g))
new-id (feature/owned new-subject (keyword (name id)))]
[new-id (-> g
(assoc :id new-id :subject new-subject)
(update :members #(mapv feature-ids %)))])))
(:groups frozen))]
{:sid sid
:subject-ids (select-keys ids (keys (:subjects frozen)))
:clip (-> (reduce (fn [document face]
(assoc-in document [:symbols face :nodes :sound] sound))
c faces)
(assoc-in [:symbols sid :name] (str label))
(update :analyses merge (:analyses frozen))
(update :subjects merge subjects)
(update :features merge features)
(update :groups merge groups))}))
(defn placed
"`entry` — a document and its store — once `brought` holds the symbols brought
in and `store` their blocks: the stores merged and an instance of `sid` placed
in `host` at `frame`, its middle on stage pixel `point` or where it was drawn
when there is none. See `clip/place-symbol`."
[entry brought store sid host frame uuid point]
(let [st (merge (:store entry) store)]
(assoc entry :store st :clip (clip/place-symbol brought st host sid
(* frame (:rate (clip/grid-time brought host))) uuid point))))

25
frontend/src/arthur/domain/cadence.cljs
View file

@ -1,25 +0,0 @@
(ns arthur.domain.cadence
"Select integer content frames across frame grids. Stored frames never change.")
(defn ratio [grid native]
(if (and grid native (pos? grid) (pos? native)) (/ native grid) 1))
(defn frame
"Latest native frame at or before reader frame f. Never sample the future."
[f grid native]
(js/Math.floor (if (and grid native) (/ (* f native) grid) f)))
(defn frames
"Reader frames covering a native length, including a partial final frame."
[n grid native]
(when n (max 1 (js/Math.ceil (/ n (ratio grid native))))))
(defn reader-frame
"The earliest reader frame whose `frame` is at or after native frame n — the
inverse of `frame`, as far as it has one.
`frame` is a floor, so several reader frames can show one native frame and a
native frame between two of them is shown by none: this answers with the
reader frame that first reaches it, which is what seeking to a mark means."
[n grid native]
(js/Math.ceil (if (and grid native) (/ (* n grid) native) n)))

77
frontend/src/arthur/domain/canon.cljs
View file

@ -1,77 +0,0 @@
(ns arthur.domain.canon
"One canonical text for a map, so that hashing it means something.
A content address is a hash of a DESCRIPTION of every input, and a description
only addresses anything if the same inputs always write the same bytes. A CLJS
map has no key order, `pr-str` will happily print `{:a 1 :b 2}` in either order
between runs, and JSON has no canonical form of its own. So this is the one
place that decides.
The text is VALID JSON, deliberately. The server stores it beside the key and
verifies `sha256(descriptor) == key` (clips/views.py), and it also has to read
two fields out of it to enforce that a detector version was declared at all.
Hashing the text the client sent, rather than recomputing it from parsed
values, is what keeps that check free of a cross-language float-formatting
agreement nobody could hold: Python writes `1.0` where JS writes `1`, and a
scheme where both sides re-render the numbers would break on the first integral
double. The bytes are the contract; the schema on top of them is a convention.
It is also meant to be READ. A stale bake presents as a picture that will not
update, and the descriptor is the only thing that can say which input moved, so
it is short, flat where it can be, and never has a 229-frame mask inlined —
see `arthur.flow.address`, which digests masks before they reach here.
Three refusals, all of them cases where a canonical text is not possible or
the key would be ambiguous:
A KEYWORD VALUE. Keys are keywords and become their names, because a key is
a name and nothing else. A keyword VALUE is refused instead of being named,
because then `:mouth` and \"mouth\" would hash alike, and the server would be
reading a field whose type depended on the caller's mood. Callers convert at
the boundary, which is also what makes the stored JSON clean.
A SET. Unordered, so there is no one text for it. Sort it into a vector at
the call site, where it is obvious which order was meant.
NaN OR INFINITY. Neither is JSON, and both mean a measurement went wrong
upstream of here — silently addressing it would cache the mistake."
(:require [clojure.string :as str]))
(defn- number->text [x]
(when-not (js/Number.isFinite x)
(throw (ex-info "a descriptor cannot hold NaN or infinity" {:value x})))
;; `(str 1.0)` is "1" and `(str 0.12)` is "0.12": JS prints the shortest decimal
;; that round-trips, so this is stable without a format string.
(str x))
(defn- key->text [k]
(cond
(keyword? k) (subs (str k) 1) ; :a -> "a", :roto/b -> "roto/b"
(string? k) k
:else (throw (ex-info "a descriptor key is a keyword or a string"
{:key k :type (type k)}))))
(declare write)
(defn- write-map [m]
(str "{"
(str/join "," (map (fn [[k v]] (str (js/JSON.stringify (key->text k)) ":" (write v)))
(sort-by (comp key->text key) (seq m))))
"}"))
(defn write
"The canonical JSON text of a descriptor value."
[v]
(cond
(nil? v) "null"
(true? v) "true"
(false? v) "false"
(number? v) (number->text v)
(string? v) (js/JSON.stringify v)
(map? v) (write-map v)
(set? v) (throw (ex-info "a descriptor cannot hold a set: sort it into a vector where the order is visible"
{:value v}))
(keyword? v) (throw (ex-info "a descriptor cannot hold a keyword VALUE: name it at the call site, so \"mouth\" and :mouth cannot address the same block"
{:value v}))
(sequential? v) (str "[" (str/join "," (map write v)) "]")
:else (throw (ex-info "not a descriptor value" {:value v :type (type v)}))))

643
frontend/src/arthur/domain/channel.cljs
View file

@ -1,643 +0,0 @@
(ns arthur.domain.channel
"A channel is one animatable property, sampled at a frame.
Three shapes, and the uniformity across them is the entire point of the model
— analysis does not produce a different kind of data, it produces keys densely
on the same channels a hand fills in sparsely:
FRAMED {:animated? false :value v}
A thing that simply exists. A painted background cel is this.
KEYED {:animated? true :interp :hold :keys {0 v, 4 v, 12 v}}
Sparse, authored, in the document. Undoable and syncable.
DENSE {:animated? true :interp :hold
:dense {:store \"sha256:…\" :offset 0 :stride 40 :frames 600}
:generated {...}}
Generated, one value per frame, in a typed array outside app-db.
`value-at` reads all three and is the specification. `cursor`/`sample!` is the
fast path for playback and must agree with it exactly; scene-test asserts that
across forward, backward and random frame order, because a cursor that drifts
is a bug you would see as the wrong pose rather than as an error.
KEYS ARE A MAP BY FRAME, NEVER A VECTOR, and the map stored in the document is
a PLAIN map — transit and JSON both lose sortedness, so the sorted index is
built here at read time and never persisted.
:generated is provenance. NOTHING IN HERE READS IT, and nothing downstream may:
it exists so the UI can offer a parameter panel instead of raw keys. It lives
on the channel rather than the node because a mouth wants a rotoscoped
[:geom :pts] and a hand-animated [:xform :pos] at the same time, and putting
the flag on the node would forbid the most useful thing in the model.")
;; ---------------------------------------------------------------------------
;; the state mask
;;
;; PRESENCE IS NOT VISIBILITY, and the distinction is free now and expensive to
;; retrofit. A part that is hidden EXISTS and is not drawn, which is `[:vis]`, a
;; channel like any other. A subject that is occluded has NO VALUE on that frame
;; — there is nothing to hide and nothing to fall back on — and that is this.
;;
;; The mask carries ABSENCE ONLY. An earlier draft gave it a hidden bit as well,
;; per docs/architecture.md's "hidden flag + palette index", and that bit was
;; simply a dense `[:vis]` wearing a different hat: two mechanisms for one
;; question, which is how you end up with a part that is hidden by one and shown
;; by the other. Hiding is a channel; absence is a state. Remaining bits are
;; reserved.
(def ^:const present 0)
(def ^:const absent-bit 1)
(def absent
"Sampled value for a frame the subject was not on.
Distinct from a part being switched off, which is `[:vis]` being false, and
distinct from a part having no keys. The identity tracker, when it arrives,
needs somewhere to say \"not on screen\" without inventing a pose."
::absent)
(defn nothing?
"True when there is no value to draw with."
[v]
(identical? v absent))
;; ---------------------------------------------------------------------------
;; constructors, for hand-written scenes and tests
(defn framed [v] {:animated? false :value v})
(defn keyed
"A channel of keys, and how each one leads to the next. `interp` is an
argument, never a default: `:hold` and `:linear` are the difference between a
cut and a tween, which is the whole content of the channel."
[ks interp]
{:animated? true :interp interp :keys ks :over []})
;; ---------------------------------------------------------------------------
(defn component
"Component i of a multi-component channel value.
An authored value is a CLJS vector; a value read out of a dense block is a
typed-array view over the block, because copying it would allocate per node
per frame. Both have to read the same way here or every consumer downstream
grows the same two-way branch."
[v i]
(if (vector? v) (-nth v i) (aget v i)))
(defn frames
"Sorted vector of the frames a keyed channel has keys on, or nil. Built here
and cached by `cursor`; `value-at` rebuilds it, which is why `value-at` is the
specification and not the playback path."
[ch]
(when-let [ks (:keys ch)]
(vec (sort (keys ks)))))
;; ---------------------------------------------------------------------------
;; correction layers
;;
;; `:over` is an ORDERED STACK on top of whatever the channel already says.
;; Generated motion stays the base; a hand correction is a layer above it, so
;; regenerating replaces the base and the corrections survive. That is the whole
;; reason the stack exists rather than the hand edit being written into the keys.
;;
;; A LAYER'S VALUES ARE A CHANNEL. A constant adjustment is a framed one, a ramp
;; or a return motion is a keyed one, and neither needs a second way of saying
;; what a value is over time: layers read through `value-at` and `cursor` like
;; anything else, which is also what stops the fast path and the specification
;; from being two implementations of blending.
;;
;; A LAYER HAS NO TIME SPACE OF ITS OWN. Its `:support` and its values' keys are
;; in the frames the base channel's keys are in — the node's own. A correction on
;; a lane is therefore in lane frames and crosses the drawing boundaries under
;; it; a correction on one cel is in that cel's frames and travels
;; with it when it moves. Ownership already answered the question, so there is no
;; field to get wrong.
(defn layer
"One correction: `values` applied to the base wherever `support` covers the
frame. `op` is `:offset` or `:replace`."
[id support op values]
{:id id :support support :op op :values values})
(defn- covers?
"Half-open, as a span is: a correction over frames 10 to 12 is `[10 13)`."
[[in out] f]
(and (<= in f) (< f out)))
(defn- width
"Components in a value, or nil for a number. A dense value is a typed-array
view, an authored one a vector, and a correction has to add to either."
[v]
(cond (number? v) nil (vector? v) (count v) :else (.-length v)))
(defn- shape
"What kind of value this is, for asking whether one can be added to another:
`:scalar`, a component count, or `:opaque` for a value that is neither — a
`[:vis]` boolean is opaque, and can be replaced but not offset."
[v]
(cond
(number? v) :scalar
(vector? v) (count v)
(and (some? v) (number? (.-length v))) (.-length v)
:else :opaque))
(defn value-shape
"The shape of the values a channel yields, without sampling it, or nil where
there is nothing to read it off — an empty key map says nothing about what its
values would have been, and nil must not be taken for a scalar."
[ch]
(cond
(not (:animated? ch)) (when (some? (:value ch)) (shape (:value ch)))
(:dense ch) (if (= 1 (:stride (:dense ch))) :scalar (:stride (:dense ch)))
(seq (:keys ch)) (shape (val (first (:keys ch))))
:else nil))
(defn- shape-conflict [base-shape correction-shape]
(cond
(or (nil? base-shape) (nil? correction-shape)) nil
(= :opaque base-shape) "the base is not a number or a row of components"
(= :opaque correction-shape) "the correction is not a number or a row of components"
(not= base-shape correction-shape)
(str "the base has " (pr-str base-shape) " and the correction "
(pr-str correction-shape) " — a correction cannot offset a value of"
" a different shape")))
(defn conflict-with
"Why correction `l` cannot apply to base channel `base`, or nil.
ONLY `:offset` can conflict. It adds component by component, so it needs the
base to have the components it has — which is what a topology change takes
away when a re-freeze gives a mouth a different number of points. `:replace`
states a whole value and so has nothing to agree with.
Shapes that cannot be read yet do not conflict: an empty key map is not a
disagreement, it is a channel with nothing in it."
[base l]
(when (= :offset (:op l))
(shape-conflict (value-shape base) (value-shape (:values l)))))
(defn- support-of [l]
(let [s (:support l)]
(when (and (vector? s) (= 2 (count s))
(every? number? s) (< (first s) (second s)))
s)))
(defn stack-conflict
"Why layer `i` can encounter a value of the wrong shape after the active
layers before it, or nil.
Replacement coverage is considered at every interval boundary. This matters
when adjacent replacements jointly cover an offset: neither covers its whole
support, but the base can never reach it. A conflicted replacement is skipped,
exactly as the evaluator skips it."
[ch i]
(let [l (nth (:over ch) i nil)]
(when (and (= :offset (:op l)) (support-of l))
(let [[a b] (support-of l)
prior (take i (:over ch))
cuts (->> prior
(keep support-of)
(mapcat identity)
(filter #(< a % b))
(into [a b])
distinct sort)
;; Shape at a point is the last active, nonempty replacement's
;; shape, or the base shape when no replacement supplies a value.
at (fn [f]
(or (last (keep (fn [p]
(let [s (support-of p)
v (value-shape (:values p))]
(when (and (= :replace (:op p))
(not (:conflict p)) v s
(covers? s f))
v)))
prior))
(value-shape ch)))
shapes (into #{} (map (fn [[x y]] (at (/ (+ x y) 2))))
(partition 2 1 cuts))
v (value-shape (:values l))]
(some #(shape-conflict % v) shapes)))))
(defn reconcile
"Recheck an ordered layer stack against this channel's base.
Old conflict marks are findings from an earlier base, so they are cleared and
recomputed in order. A newly conflicted replacement is then invisible to the
layers after it, matching evaluation. Nothing is dropped or reordered."
[ch]
(let [layers (mapv #(dissoc % :conflict) (:over ch))]
(reduce (fn [out l]
(let [candidate (assoc ch :over (conj out l))
why (stack-conflict candidate (count out))]
(conj out (cond-> l why (assoc :conflict why)))))
[] layers)))
(defn conflicts
"Corrections on `ch` that cannot apply to its base, as `[{:id :why}]`.
NOT `problems`. A conflict is a legitimate state for a document to be in: a
regeneration changed the topology under a correction that was right when it was
made, and resolving it is a person's decision, not a reason the document will
not load. `flow/regenerate` records one on the layer, a conflicted layer is not
applied, and this is how a view finds them to offer."
[ch]
(vec (for [[i l] (map-indexed vector (:over ch))
:let [why (or (:conflict l) (stack-conflict ch i))]
:when why]
{:id (:id l) :why why})))
(defn- offset-onto
"`base` plus `v`, component-wise. A vector, never a write into `base`, which
for a dense channel is a view onto the block itself."
[base v ch]
(let [wb (width base) wv (width v)]
(cond
(and (nil? wb) (nil? wv)) (+ base v)
(and wb wv (= wb wv))
(mapv (fn [i] (+ (component base i) (component v i))) (range wb))
:else
(throw (ex-info "a correction cannot offset a value of a different shape"
{:base wb :correction wv :channel (dissoc ch :dense)})))))
(defn- eye-opening-onto [points amount]
(let [n (width points)
ys (map #(component points %) (range 1 n 2))
center (/ (+ (reduce min ys) (reduce max ys)) 2)]
(mapv (fn [i] (let [v (component points i)]
(if (odd? i) (+ center (* amount (- v center))) v)))
(range n))))
(defn- over-at
"Fold `ch`'s layers onto `base` at frame f. `read` samples one layer's values
and is the only thing that differs between the specification and the cursor."
[ch f base read]
(reduce-kv
(fn [v i {:keys [support op values conflict]}]
;; A conflicted correction is neither applied nor forgotten: it stays in
;; the document, `conflicts` reports it, and a person decides. Applying it
;; would misapply it; removing it would throw away hand work.
(if (or conflict (not (covers? support f)))
v
(let [x (read i values f)]
(cond
(nothing? x) v
(= :replace op) x
;; `replace` can supply a value over an absent base; `offset` has
;; nothing to add to and says so rather than inventing a pose.
(nothing? v) absent
(= :eye-opening op) (eye-opening-onto v x)
:else (offset-onto v x ch)))))
base
(vec (:over ch))))
;; ---------------------------------------------------------------------------
;; dense blocks
(defn- absent-at?
"Is the subject absent on frame f of this block's slice?
INDEXED THE WAY THE DATA IS. A block is node-major — offset(node i) =
i·frames·stride — so a block holding several nodes holds several mask regions,
and the one belonging to this channel starts at offset/stride. Indexing the
mask by f alone reads the FIRST node's absence for every node in the block,
which is not a subtly wrong pose: it is every part in the block vanishing on
the frames where one of them was occluded."
[state offset stride f]
(and (some? state)
(pos? (bit-and (aget state (+ (quot offset stride) f)) absent-bit))))
(defn dense-at
"Read frame f out of a dense block.
`store` is {store-key -> {:data <typed array> :state <Uint8Array or nil>}},
tier 2, behind a handle and never in app-db.
The frame is CLAMPED into the block. A time map with an offset deliberately
reads the future — mouth lead is the whole reason `:offset` exists — so the
last frame of a leading track is asked for a frame past the end on every one
of the last `lead` frames. Clamping there is what the JS `shiftIndex` does and
it is the right answer: the track holds its final pose. Returning nothing
instead would blank the mouth at the end of every take.
stride 1 yields a number; anything wider yields a SUBARRAY VIEW over the
block, not a copy. Fixed topology is what makes that possible — the frame's
data is a rectangular slice at a known offset with no per-frame header.
FIXED POINT. `:scale` in the block header means the stored integers are the
value times that scale, so a block of geometry in image-height units fills an
Int16 usefully and a block of stage pixels — which wants a different scale
entirely — fills one too. It is in the header rather than agreed by convention
for exactly that reason, and it is why the block in memory is byte for byte the
block on the wire: a handle that names a sha256 has to name the bytes you
actually hold.
Decoding costs the view. `out` is a stride-sized destination the caller owns —
`cursor` allocates one per channel — because a copy per node per frame is the
allocation this whole model is arranged to avoid; passing nil allocates, which
is what `value-at`, the specification, does — and it says so by passing nil,
because there is no arity here that decides it for a caller."
[{:keys [store offset stride scale] nf :frames} f st out]
(let [{:keys [data state]} (get st store)]
(when (nil? data)
(throw (ex-info "dense channel's store key is not in the store"
{:store store :have (vec (sort (map str (keys st))))})))
(let [f (-> f (max 0) (min (dec nf)))]
(if (absent-at? state offset stride f)
absent
(let [o (+ offset (* f stride))]
(cond
(= 1 stride) (let [v (aget data o)] (if scale (/ v scale) v))
(nil? scale) (.subarray data o (+ o stride))
:else (let [dst (or out (js/Float64Array. stride))]
(dotimes [k stride]
(aset dst k (/ (aget data (+ o k)) scale)))
dst)))))))
;; ---------------------------------------------------------------------------
;; the specification
(defn segment-interp
"How the key at `left` leads to the next key. A channel default remains useful
for uniform tracks; :segments overrides only the gaps an artist chose."
[ch left]
(get (:segments ch) left (:interp ch)))
(defn- interpolate [ch f left right]
(let [a (get (:keys ch) left)]
(if (and (= :linear (segment-interp ch left)) right (>= f left) (> right left))
(let [b (get (:keys ch) right)
t (/ (- f left) (- right left))]
(cond
;; Palette-choice channels interpolate identities into a blend
;; descriptor. The renderer keeps indexed geometry in the left
;; palette's bank and blends that bank's ramp toward the right one.
;; Palette ids are opaque identities. Built-ins happen to use
;; keywords, while palettes made in the editor use UUIDs; treating
;; the latter as numbers produces NaN and therefore the renderer's
;; pink bad-data sentinel.
(and (= :palette (:semantic ch)) (some? a) (some? b))
{:from a :to b :t t}
(vector? a)
(mapv (fn [x y] (+ x (* t (- y x)))) a b)
:else (+ a (* t (- b a)))))
a)))
(defn- keyed-at
"The most recent key at or before f, CLAMPED to the first key below it.
Hold is the default and clamping at the low end is the JS `activeKey`'s
behaviour, kept: a channel's first key is the pose the part starts in, so a
frame before it reads that pose rather than having no value. This is not the
same question as presence — a part with no value at all is `absent`, which is
a state bit, not an empty key map."
[ch f]
(let [fr (sort (keys (:keys ch)))
left (or (last (take-while #(<= % f) fr)) (first fr))
right (first (drop-while #(<= % f) fr))]
(interpolate ch f left right)))
(defn repair-frame
"Map a damaged frame to a donor in the current base. Latest interval wins;
donors are sampled directly, never recursively through other repairs."
[ch f]
(reduce (fn [frame {:keys [from through donor]}]
(if (<= from f through) donor frame))
f (:repairs ch)))
(defn value-at
"Sample a channel at frame f. THE SPECIFICATION — correct, allocating, and
O(n) in the keys. `cursor`/`sample!` is what playback uses.
`store` IS AN ARGUMENT, NEVER A DEFAULT. A dense channel cannot be read
without the tier-2 store it names, and an arity that filled in nil let a
caller omit it, read correctly for every channel that happened not to be
dense, and throw the first time a selection landed on one that was. That is
how `gesture/values` took the stage down on an iris. A caller with no store
says `nil` and means it."
([ch f store] (value-at ch f f store))
([ch base-f correction-f store]
(let [base-f (repair-frame ch base-f)
base (cond
(not (:animated? ch)) (:value ch)
(:dense ch) (dense-at (:dense ch) base-f store nil)
(:keys ch) (let [ks (:keys ch)]
(if (empty? ks) absent (keyed-at ch base-f)))
:else
(throw (ex-info "animated channel has neither :keys nor :dense"
{:channel ch})))]
(if (seq (:over ch))
(over-at ch correction-f base
(fn [_ values f] (value-at values f store)))
base))))
;; ---------------------------------------------------------------------------
;; the playback path
;;
;; Playback is SEQUENTIAL, so "the most recent key at or before f" is an advance
;; of a saved index rather than a search. The difference at 30fps is a `sort` and
;; a `take-while` allocation per channel per frame against none, which is the
;; difference between the model being usable and being a demo.
(defn- bsearch
"Largest index i with ks[i] <= f, or 0 when f precedes every key (hold clamps
low, see keyed-at)."
[ks f]
(loop [lo 0, hi (dec (count ks)), best 0]
(if (> lo hi)
best
(let [mid (bit-shift-right (+ lo hi) 1)]
(if (<= (nth ks mid) f)
(recur (inc mid) hi mid)
(recur lo (dec mid) best))))))
(deftype Cursor [ch ks store buf overs ^:mutable i]
Object
(toString [_] (str "#Cursor{" (pr-str (if ks :keyed (if (:dense ch) :dense :framed))) " i=" i "}")))
(defn cursor
"A reading head on one channel. Build once per channel per resolver, then
`sample!` it per frame. Holds the sorted key index, which is why the index is
built here and not in the document — and the decode buffer a fixed-point block
needs, for the same reason the resolver owns one point buffer per node.
Only a wide fixed-point block gets a buffer: a stride-1 block decodes to a
number and a block with no `:scale` is handed back as a view.
A correction layer gets a reading head of its own, because its values are a
channel and this is how a channel is read fast. One level deep: a layer's
values may not themselves carry layers, which `problems` refuses.
`store` is an argument for the reason it is one on `value-at`."
[ch store]
(let [d (:dense ch)]
(->Cursor ch
(when (and (:animated? ch) (not d) (seq (:keys ch))) (frames ch))
store
(when (and d (:scale d) (> (:stride d) 1))
(js/Float64Array. (:stride d)))
(mapv #(cursor (:values %) store) (:over ch))
0)))
(defn- base-sample!
"What the cursor's channel says at f BEFORE its corrections. Advancing the
reading head is this function's whole job, and it is separate from blending so
that a layer cannot accidentally be read through the base's index."
[^Cursor cur ch ks f]
(cond
(not (:animated? ch)) (:value ch)
(:dense ch) (dense-at (:dense ch) f (.-store cur) (.-buf cur))
(nil? ks) absent ; animated with an empty key map
:else
(let [n (count ks)
i (.-i cur)
last (dec n)
i' (cond
;; still inside the key the cursor sits on
(and (<= (nth ks i) f)
(or (= i last) (> (nth ks (inc i)) f)))
i
;; the next one — one frame of playback crossed one key
(and (< i last)
(<= (nth ks (inc i)) f)
(or (= (inc i) last) (> (nth ks (+ i 2)) f)))
(inc i)
:else (bsearch ks f))]
(set! (.-i cur) i')
(interpolate ch f (nth ks i') (when (< i' last) (nth ks (inc i')))))))
(defn sample!
"Value of the cursor's channel at f. O(1) when f is at or one key past where
the cursor already sits — the playback case — and O(log n) otherwise, which is
a seek. Advancing and seeking are deliberately different costs: a scrub can
afford a binary search and a frame cannot.
A correction layer is sampled through its OWN cursor, so a stacked channel is
still one reading head per key map and `value-at` stays the specification for
the blending as well as for the base."
([cur f] (sample! cur f f))
([^Cursor cur base-f correction-f]
(let [ch (.-ch cur)
base (base-sample! cur ch (.-ks cur) (repair-frame ch base-f))]
(if (seq (:over ch))
(over-at ch correction-f base
(fn [i _ f] (sample! (nth (.-overs cur) i) f)))
base))))
;; ---------------------------------------------------------------------------
(defn describe
"Which of the three shapes, for error messages and the parameter panel."
[ch]
(cond
(not (:animated? ch)) :framed
(:dense ch) :dense
:else :keyed))
(defn problems
"Human-readable reasons this map is not a channel. Empty means it is one."
[ch]
(let [values (when (map? (:keys ch)) (vals (:keys ch)))
first-value (first values)
linear-values? (or (every? number? values)
(and (= :palette (:semantic ch)) (every? some? values))
(and (vector? first-value)
(pos? (count first-value))
(every? (fn [v] (and (vector? v)
(= (count v) (count first-value))
(every? number? v)))
values)))
linear? (or (= :linear (:interp ch))
(some #{:linear} (vals (:segments ch))))]
(cond-> []
(and (contains? ch :repairs)
(not (and (vector? (:repairs ch))
(every? (fn [{:keys [id from through donor]}]
(and id (every? integer? [from through donor])
(<= 0 from through) (<= 0 donor)))
(:repairs ch)))))
(conj "repairs require an ID and nonnegative whole donor and interval frames")
(not (map? ch))
(conj "not a map")
(and (map? ch) (not (contains? ch :animated?)))
(conj ":animated? is required — the flag is what makes framed and keyed one type")
(and (map? ch) (:animated? ch) (not (or (:keys ch) (:dense ch))))
(conj "animated but has neither :keys nor :dense")
(and (map? ch) (:animated? ch) (:keys ch) (:dense ch))
(conj "has both :keys and :dense; a channel is one shape at a time")
(and (map? ch) (:keys ch) (not (map? (:keys ch))))
(conj (str ":keys is a " (if (vector? (:keys ch)) "vector" "non-map")
" — keys are a MAP by frame, so a merge can be per-key"))
(and (map? ch) (:keys ch) (map? (:keys ch)) (not (every? number? (keys (:keys ch)))))
(conj ":keys has a non-numeric frame")
(and (map? ch) (:animated? ch) (not (#{:hold :linear nil} (:interp ch))))
(conj (str ":interp " (:interp ch) " is not implemented"))
(and (map? ch) (contains? ch :segments)
(or (not (map? (:segments ch)))
(not (:keys ch))
(not (every? (set (keys (:keys ch))) (keys (:segments ch))))
(not (every? #{:hold :linear} (vals (:segments ch))))))
(conj ":segments must map existing key frames to :hold or :linear")
(and (map? ch) linear?
(or (:dense ch) (not linear-values?)))
(conj ":linear interpolation needs numeric keys of one shape")
(and (map? ch) (contains? ch :over) (not (vector? (:over ch))))
(conj ":over is an ORDERED stack, so it is a vector")
(and (map? ch) (vector? (:over ch)))
(into (for [{:keys [id support op values]} (:over ch)
p (cond-> []
(nil? id)
(conj "needs an :id — a correction has an identity a regeneration can keep")
(not (and (vector? support) (= 2 (count support))
(every? #(and (number? %) (js/Number.isFinite %)) support)
(< (first support) (second support))))
(conj (str ":support " (pr-str support)
" must be a finite, increasing [in out)"))
(not (#{:offset :replace :eye-opening} op))
(conj (str ":op " (pr-str op) " is not :offset, :replace or :eye-opening"))
;; One level. A layer over a layer is an ordering mechanism
;; the stack already is, and it would make the read
;; unbounded in depth for nothing.
(seq (:over values))
(conj "a layer's values cannot carry layers of their own")
(seq (problems (dissoc values :over)))
(conj (str "values are not a channel: "
(first (problems (dissoc values :over))))))]
(str "correction " (pr-str id) " " p)))
;; A shape mismatch NOBODY HAS RECORDED is an authoring bug; one a
;; regeneration recorded is a conflict awaiting a person, and `conflicts`
;; reports those. The distinction is what keeps a topology change from
;; making a document that will not load.
(and (map? ch) (vector? (:over ch)))
(into (for [[i l] (map-indexed vector (:over ch))
:when (not (:conflict l))
:let [why (stack-conflict ch i)]
:when why]
(str "correction " (pr-str (:id l)) " " why)))
;; A scale of zero divides every value in the block by zero, and a negative
;; one mirrors the geometry. Both are authored-data bugs that present as a
;; part drawn nowhere or inside out, not as an error.
(and (map? ch) (:dense ch) (contains? (:dense ch) :scale)
(not (and (number? (:scale (:dense ch))) (pos? (:scale (:dense ch))))))
(conj (str ":dense :scale is " (pr-str (:scale (:dense ch)))
" — a fixed-point scale is a positive number the stored integers"
" were multiplied by")))))

820
frontend/src/arthur/domain/clip.cljs
View file

@ -1,820 +0,0 @@
(ns arthur.domain.clip
"A CLIP: the unit of work, and a library of symbols.
{:name \"take\"
:fps 30
:width 320 :height 200
:analyses {analysis-id {...}}
:subjects {...} :features {...} :groups {...}
:symbols {:main {:id :main :frames 229 :nodes {...}}}}
Every field here is a fact about the clip and NOT about a bag of nodes, which is
the cut this namespace exists to make. Before it, one map carried both: `:fps`,
the stage dimensions, the analysis record and the tracking identities sat beside
`:nodes`. The cost of leaving them together was not untidiness. It was that a
SYMBOL had nowhere to live: a symbol is a bag of nodes with a frame space and
nothing else, so under the old shape it would have had to be a clip with seven
meaningless fields.
Now there is one node-holding type — `arthur.domain.symbol` — and a clip holds
a MAP of them. A `:kind :instance` node places one symbol inside another, and
the clip resolver gives each instance its own reading heads.
WHAT IS NOT HERE: how nested symbols' frames and coordinates relate, and
moving nodes between them, are `arthur.domain.nest`; bringing symbols in from
another clip is `arthur.domain.bring`. This namespace is the document and the
operations that only need the document.
NO SYMBOL IS SPECIAL. There is no reserved root id: which symbol is on screen
is the editor's state, not the document's, and every function here that needs
a symbol is told which. A new document has one symbol called `:main` because
it has to be called something, and that is all the name means — it can be
renamed or placed inside another symbol like any of them. What the document
does say is `:root`, which symbol it opens on: a pointer, not a kind of
symbol, the way a Flash file names its scene. See `opens-on` for why that
cannot be worked out instead.
:fps is the output grid. A symbol's optional :fps names the native grid its
frames were authored or measured on; absent means the document's grid."
(:refer-clojure :exclude [symbol])
(:require [arthur.domain.cadence :as cadence]
[arthur.domain.channel :as ch]
[arthur.domain.feature :as feature]
[arthur.domain.node :as node]
[arthur.domain.palette :as pal]
[arthur.domain.pose :as pose]
[arthur.domain.symbol :as symbol]))
(def clip-keys
"Every top-level field of a clip, and the reason `arthur.domain.leaf` refuses
one it does not know: a field added to the clip without a leaf to save it in is
a field that saves silently and comes back missing. The failure is a document
that loses something on every round trip, which is the one bug a persistence
layer must not be able to have. Add the field here and to `leaf/leaves` and
`leaf/clip` in the same commit."
#{:name :fps :analyses :subjects :features :groups :width :height :symbols
:palettes :default-palette :root})
(defn symbol
"One of the clip's symbols, by id."
[clip sid]
(get-in clip [:symbols sid]))
(defn trace?
"Is `sym` a tracing symbol: footage or a still to draw over, which is placed and
moved like any symbol and never drawn into the picture? See `trace-op`."
[sym]
(= :trace (:type sym)))
(defn symbol-name
"What to call a symbol: its `:name`, or its id when it has none."
[clip sid]
(or (:name (symbol clip sid)) (name sid)))
(defn node-label
"What to call node `n` on screen.
A NAME A PERSON TYPED WINS, and for an instance that is the ONLY thing `:name`
now means: `place-symbol` deliberately does not copy the symbol's name onto the
node it makes. Two instances of one symbol are told apart by what somebody
called them — `8625 left` and `8625 right` of one `face` — and reading through
in front of that would collapse them to the same word.
OTHERWISE AN INSTANCE IS LABELLED BY WHAT IT PLACES, read through on every
render. A name copied at creation goes stale the moment the symbol is renamed,
and then the document shows one thing under two names: the symbol reads `bg` in
its tab while an instance of it still reads `symbol-18`, which is how a person
comes to paste a symbol into itself without being able to see that is what they
are doing. `problems` refuses that cycle; this is why it stops looking like a
reasonable thing to try.
An id is a uuid for a placement and a keyword for an authored node, and neither
reads as a name, so the last resort is a legible stand-in rather than `(str
id)` — `:face-1` keeps its colon and a uuid pushes a column open."
[clip id n]
(or (:name n)
(some->> (node/source n) (symbol-name clip))
(if (keyword? id) (subs (str id) 1) (subs (str id) 0 8))))
(defn frames
"A symbol's length. Read off the symbol, never copied beside it."
[clip sid]
(:frames (symbol clip sid)))
(defn fps [clip sid] (or (:fps (symbol clip sid)) (:fps clip)))
(defn set-fps
"Change the output grid without rewriting authored content's frames.
A new document's empty symbol is the one exception: it has no native rate yet,
so it follows the project grid and its empty extent is rescaled to preserve its
duration. Once a symbol contains anything, changing the project rate records
the old effective rate on it before changing the output grid."
[clip rate]
(let [old (:fps clip)]
(-> clip
(update :symbols
#(into {}
(map (fn [[sid sym]]
[sid (cond
(:fps sym) sym
(empty? (:nodes sym))
(update sym :frames cadence/frames rate old)
:else (assoc sym :fps old))]))
%))
(assoc :fps rate))))
(defn output-frames [clip sid]
(cadence/frames (frames clip sid) (:fps clip) (fps clip sid)))
(defn grid-time [clip sid]
{:at 0 :rate (cadence/ratio (:fps clip) (fps clip sid))})
;; ---------------------------------------------------------------------------
;; the only crossing between the output grid and a symbol's own frames
;;
;; Everything authored is in a symbol's own frames and every editing gesture is
;; too — see `docs/one-grid-plan.md`. The output grid belongs to playback: the
;; clock, the audio mix, export, and the frame number the stage is drawing. These
;; two functions are the whole of the way between, so a caller that has a
;; playhead and needs a document coordinate says so in one visible call instead
;; of multiplying by a rate it had to know about.
(defn shown-frame
"Which of symbol `sid`'s own frames the output frame `f` shows."
[clip sid f]
(cadence/frame f (:fps clip) (fps clip sid)))
(defn first-output-frame
"The output frame that first shows symbol `sid`'s own frame `n`: what to seek
to to put the playhead on a mark of `sid`'s ruler."
[clip sid n]
(cadence/reader-frame n (:fps clip) (fps clip sid)))
(defn source-time
"Derived cel-to-content map. Frame-rate units never enter stored retimes."
[clip host n]
(when-let [t (node/source-time n)]
(let [r (cadence/ratio (fps clip host) (fps clip (node/source n)))]
(-> t (update :rate * r) (update :at / r)))))
(defn placed-frame [clip host n f]
(let [child (node/source n)
r (cadence/ratio (fps clip host) (fps clip child))]
(some-> (node/placed-frame (update-in n [:playback :speed] #(* (or % 1) r))
f (frames clip child))
(update :frame js/Math.floor))))
(defn stage
"A symbol's stage as `[width height]`: its own, or the clip's where it has none.
Absent rather than copied in at creation, so a symbol nobody has sized follows
the project's size when that changes."
[clip sid]
(let [sym (symbol clip sid)]
[(or (:width sym) (:width clip)) (or (:height sym) (:height clip))]))
(defn update-symbol
"Apply f to one symbol in place."
[clip sid f & args]
(apply update-in clip [:symbols sid] f args))
(defn places
"The ids of the symbols `sid` places, directly."
[clip sid]
(into #{} (mapcat node/sources) (vals (:nodes (symbol clip sid)))))
(defn contains-symbol?
"Whether `inner` is `outer` or is placed anywhere inside it. Placing `outer`
into `inner` when this is true is a cycle."
[clip outer inner]
(let [seen (volatile! #{})]
(letfn [(walk [sid]
(or (= sid inner)
(when-not (@seen sid)
(vswap! seen conj sid)
(some walk (places clip sid)))))]
(boolean (walk outer)))))
(defn unplaced
"The symbols no other symbol places, sorted by id. What to open when a
document is opened."
[clip]
(let [placed (into #{} (mapcat #(places clip %)) (keys (:symbols clip)))]
(vec (sort-by str (remove placed (keys (:symbols clip)))))))
(defn- longest-unplaced
"The longest symbol nothing else places, ties broken by id, and never a
tracing symbol: that is footage nobody has placed yet, as long as its take."
[clip]
(first (sort-by (fn [sid] [(- (or (frames clip sid) 0)) (str sid)])
(remove #(trace? (symbol clip %)) (unplaced clip)))))
(defn opens-on
"The symbol a document opens on: its `:root`.
IT IS STORED, NOT WORKED OUT. It used to be the longest unplaced symbol, on
the theory that the one containing everything else is always that. A take
disproves it: imported footage is a symbol a thousand frames long, and the
moment its instance is deleted, or the drop lands somewhere other than the
root, nothing places it and it outranks a 120-frame `:main`. The document then
opened on the take, and `set-root-fps` rewrote the take's rate to the
project's on the way in.
A document with no `:root` — a demo, a fixture — still gets the old answer."
[clip]
(let [root (:root clip)]
(if (contains? (:symbols clip) root) root (longest-unplaced clip))))
(defn pin-root
"Give a document saved before `:root` existed the root it was made with.
Every such document started as `blank`, whose root is `:main`, and ids never
change, so `:main` is the answer wherever it survives; the old rule is only
for documents that never had one."
[clip]
(cond-> clip
(not (:root clip))
(assoc :root (if (contains? (:symbols clip) :main) :main (longest-unplaced clip)))))
(defn set-root-fps
"Set the document/output rate and the root symbol's editing rate together.
Project FPS is the root timeline's clock. Nested symbols keep their own native
rates and are sampled when placed across that boundary; only the root changes
here. Frame numbers are authored positions, so changing the rate does not
rewrite them or silently move cuts and keys."
[clip rate]
(let [root (opens-on clip)]
(cond-> (assoc clip :fps rate)
root (assoc-in [:symbols root :fps] rate))))
(def ^:const blank-frames
"How long a new document is before anything says otherwise. Four seconds at 30,
which is long enough to key something into and short enough to scrub by hand."
120)
(defn blank
"A new, empty document: one symbol, and nothing in it.
A SYMBOL IS BORN EMPTY. It used to be born holding a lane, because a lane was
the only place temporal content could go; now the symbol itself is the
container — see `symbol/children` — so there is nothing to invent and the
first drop into a symbol is the same operation as the second.
The tracking maps are ABSENT rather than empty, because `leaf/leaves` writes no
leaf for an empty one and so cannot bring it back: a blank document that opened
as a different map than it saved from is exactly the round trip that namespace
promises not to have. Nothing drawn by hand has them either — the demo scene
and the swarm carry no `:subjects` — so every reader already reads absence as
none, and `clip-keys` says which fields MAY be here, not which must."
[]
{:name "untitled"
:fps 30
:width 320 :height 200
:palettes {pal/default-id pal/default-palette}
:default-palette pal/default-id
:root :main
;; No native fps yet: an untouched canvas follows the project grid. Imported
;; and generated symbols carry their own rate explicitly.
:symbols {:main {:id :main :frames blank-frames :nodes {}}}})
(defn- op-path [op] (let [n (:node op)] (if (vector? n) n [n])))
(defn- own-span
"Where in `ops` the symbol at row path `path` is drawn: the end of its layer
when it has one, and the first and last of its own ops."
[ops path]
(let [own? #(and (< (count path) (count (op-path %)))
(= path (subvec (op-path %) 0 (count path))))
begin (first (keep-indexed #(when (and (= :begin (:kind %2)) (= path (op-path %2))) %1) ops))
end (when begin
(reduce (fn [depth i]
(case (:kind (ops i))
:begin (inc depth)
:end (if (= 1 depth) (reduced i) (dec depth))
depth))
0 (range begin (count ops))))
mine (keep-indexed #(when (own? %2) %1) ops)]
{:end (when (integer? end) end) :from (first mine) :to (last mine)}))
(defn- insert-at [ops i & more] (into (into (subvec ops 0 i) more) (subvec ops i)))
(defn atop
"`ops` with `op` drawn on top of what the symbol at row path `path` draws —
in ITS stacking context, where a shape added to it would land, so what is
above that symbol stays above. At the end when it drew nothing."
[ops path op]
(let [ops (vec ops)
{:keys [end to]} (own-span ops path)]
(cond end (insert-at ops end op)
to (insert-at ops (inc to) op)
:else (conj ops op))))
(defn in-layer
"`ops` with `op` drawn last INSIDE the symbol at row path `path` — in its
layer, so a knockout clears only what that symbol drew, as one saved there
would. A symbol that has no layer yet is given one around its ops. `ops`
unchanged when that symbol drew nothing."
[ops path op]
(let [ops (vec ops)
{:keys [end from to]} (own-span ops path)]
(cond
end (insert-at ops end op)
from (-> ops
(insert-at (inc to) op {:kind :end :node path})
(insert-at from {:kind :begin :node path}))
:else ops)))
(defn- transform-op
"Put a symbol's already resolved mark into its instance's parent space. Its
name becomes its path of instances down to it, the path its timeline row has."
[op m path]
(let [at (fn [x y] [(+ (* (aget m 0) x) (* (aget m 2) y) (aget m 4))
(+ (* (aget m 1) x) (* (aget m 3) y) (aget m 5))])
scale (node/mean-scale m)
n (:node op)
op (assoc op :node (if (vector? n) (into path n) (conj path n)))]
(case (:kind op)
:poly (let [out (js/Float64Array. (.-length (:pts op)))]
(dotimes [i (:n op)]
(let [[x y] (at (aget (:pts op) (* 2 i))
(aget (:pts op) (inc (* 2 i))))]
(aset out (* 2 i) x)
(aset out (inc (* 2 i)) y)))
(assoc op :pts out))
:disc (let [[x y] (at (:cx op) (:cy op))]
(assoc op :cx x :cy y :r (* scale (:r op))))
:rect (let [[x y] (at (:cx op) (:cy op))]
(assoc op :cx x :cy y :size (* scale (:size op))))
:trace (assoc op :m (node/mul! (node/mat) m (:m op)))
op)))
(defn- trace-op
"The one op an instance of tracing symbol `sym` makes, showing its frame `frame`
at world `m`, from node `id` of symbol `sid`.
NOT A PICTURE OP. Nothing indexed can show a photo, so the raster refuses this
kind and the player hands it to `ui/tracing` instead; and the resolver makes one
only when asked with `:tracing?`, which only the stage does. An export, a
symbol's centre and a thumbnail never ask, so a reference cannot reach the
picture by any path that forgets to filter it.
`:layer` is what the on/off switch for one layer is keyed by: the symbol the
placement is in and its id, so a face's plate is one layer wherever the face
is placed."
[sym frame m sid id]
{:kind :trace
:node id
:layer [sid id]
:media (:media sym)
:frame frame
:size [(:width sym) (:height sym)]
:m (js/Float64Array.from m)})
(defprotocol IActivePalette
(active-palette [this]
"The palette selected by this resolver's most recently resolved frame."))
(defn- channel-value [store selection frame]
(cond
(nil? selection) nil
(and (map? selection) (contains? selection :animated?))
(ch/value-at selection frame store)
:else selection))
(defn palette-at
"The palette symbol `owner` draws in at `frame`, as the stage's root does: a
palette id, or `{:from :to :t}` while its palette lane blends from one to the
next. `inherited` is what an unset palette falls back to, then `default`."
[clip store default owner frame inherited]
;; A palette track has meaningful uncovered time. Ordinary held
;; channels clamp to their first key before it, but doing that
;; here would erase the gap before the first palette segment.
(let [channel-value (partial channel-value store)
fallback (or (channel-value (:palette owner) frame)
inherited default)
materialize (fn [choice]
(cond
(= pal/inherit choice) fallback
(map? choice) (-> choice
(update :from #(if (= pal/inherit %) fallback %))
(update :to #(if (= pal/inherit %) fallback %)))
:else choice))
track (:palette-channel owner)
track-value (if-let [ks (:keys track)]
(some->> (keys ks)
(filter #(<= % frame))
sort last
(get ks))
(channel-value track frame))]
(or (when-let [track-sid (and (keyword? (:palette-track owner))
(:palette-track owner))]
(let [track-symbol (symbol clip track-sid)
palette-clip (first
(filter (fn [n]
(let [[a b] (node/placed-span n)]
(and a (<= a frame) (< frame b))))
(symbol/children (:nodes track-symbol))))
palette-symbol (some-> palette-clip node/source
(#(symbol clip %)))
fallback (when (= :palette (:type palette-symbol))
(:palette-ref palette-symbol))
choice (get-in palette-clip [:channels [:palette]])
start (some-> palette-clip node/placed-span first)]
(or (when (and choice start)
(materialize (channel-value choice (- frame start))))
fallback)))
track-value
fallback)))
(defn resolver
"Resolve an output frame, selecting native content at each symbol boundary.
Every instance owns its cursors and buffers. The IResolver queries return
native node frames and world matrices for the last rendered output frame."
[clip sid store palette opts]
(let [context? (and (map? palette) (:palettes palette) (:offsets palette))
active-palette-state (atom (:default palette))]
(letfn [(root-selection-at [owner frame inherited]
(palette-at clip store (:default palette) owner frame inherited))
(selection-at [owner frame inherited]
(let [selection (:palette owner)
chosen (cond
(nil? selection) nil
(and (map? selection) (contains? selection :animated?))
(ch/value-at selection frame store)
:else selection)]
(or chosen inherited (:default palette))))
(build [sid chain pose-tracks root?]
(when (some #{sid} chain)
(throw (ex-info "symbol cycle" {:chain (conj chain sid)})))
(let [sym (or (symbol clip sid)
(throw (ex-info "an instance names a missing symbol" {:symbol sid})))
active (volatile! (when context? (:default palette)))
nodes (:nodes sym)
rank (symbol/draw-rank nodes (symbol/order nodes))
ids (sort-by rank (keys nodes))
own (symbol/resolver sym store (if context?
#(pal/render-index palette @active %)
palette)
(assoc opts :pose-tracks pose-tracks))
;; Each cel owns its source resolver and mutable buffers. A
;; tracing symbol has nothing to resolve: it is one op.
children (into {}
(for [[id n] nodes
:when (= :instance (:kind n))
child (sort-by str (node/sources n))
:when (not (trace? (symbol clip child)))]
[[id child] (build child (conj chain sid)
(get-in n [:playback :tracks]) false)]))
;; The instances that were on the last frame, and WHICH
;; drawing each was showing — a row path is read back through
;; the child that was actually resolved, not the only one
;; there used to be. Their resolvers still hold the frame
;; before whenever they were not on.
entered (volatile! {})
;; A symbol that knocks out is drawn into a layer of its own,
;; so what it clears is only ever its own.
layered? (symbol/knocks? nodes)
step (fn [f pre inherited forced]
(when context?
(vreset! active (or forced
(if root?
(root-selection-at sym (js/Math.floor f) inherited)
inherited)
(:default palette))))
;; The same decision that maps local drawing slots to
;; the active bank also names the clear colour.
(when root? (reset! active-palette-state @active))
(vreset! entered {})
(let [by-id (into {} (map (juxt :node identity))
(own (js/Math.floor f) (js/Math.floor pre)))]
(cond-> (into (if layered? [{:kind :begin :node []}] [])
(mapcat
(fn [id]
(let [n (get nodes id)]
(if (= :instance (:kind n))
(let [m (symbol/world-of own id)
local (symbol/frame-of own id)
prior (symbol/pre-frame-of own id)
length (frames clip (node/source n))
shown (when (and m (number? local))
(placed-frame clip sid n local))
frame (:frame shown)]
(cond
(not (and frame (<= 0 frame) (< frame length)))
[]
(trace? (symbol clip (:symbol shown)))
(when (:tracing? opts)
[(trace-op (symbol clip (:symbol shown)) frame m sid id)])
:else
(do (vswap! entered assoc id (:symbol shown))
(map #(transform-op % m [id])
((get children [id (:symbol shown)])
frame
;; The slot's interval crosses the
;; boundary by being MAPPED, not
;; carried: the previous slot's
;; frame goes through the same
;; placement and retime as this
;; one, so the gap comes out in the
;; child's frames and at the child's
;; rate. An instance appearing for
;; the first time on this slot has no
;; previous frame, and so no gap.
(or (:frame (when (number? prior)
(placed-frame clip sid n prior)))
(dec frame))
@active
(when (and context? (:palette n))
(selection-at n frame nil)))))))
(when-let [op (get by-id id)] [op]))))
ids))
layered? (conj {:kind :end :node []}))))]
(reify
IFn
(-invoke [_ f] (step f (dec f) nil nil))
(-invoke [_ f pre] (step f pre nil nil))
(-invoke [_ f pre inherited forced] (step f pre inherited forced))
symbol/IResolver
(world-of [_ [id & more]]
(if more
(when-let [w (and (contains? @entered id)
(symbol/world-of (get children [id (get @entered id)])
(vec more)))]
(node/mul! (node/mat) (symbol/world-of own id) w))
(symbol/world-of own id)))
(frame-of [_ [id & more]]
(if more
(when (contains? @entered id)
(symbol/frame-of (get children [id (get @entered id)]) (vec more)))
(symbol/frame-of own id)))
(pre-frame-of [_ [id & more]]
(if more
(when (contains? @entered id)
(symbol/pre-frame-of (get children [id (get @entered id)]) (vec more)))
(symbol/pre-frame-of own id))))))]
(let [r (build sid [] nil true)
grid (or (:grid-fps opts) (:fps clip))
native (fps clip sid)]
(reify
IFn
(-invoke [_ f]
(r (cadence/frame f grid native)
;; `d(k-1)`: the native frame the slot BEFORE this one selected, which
;; with `d(k)` is the interval `(d(k-1), d(k)]` the preserve-snap may
;; reach back into — the frames this slot is the first to cover, and so
;; the ones the grid would otherwise show to nobody. Slot 0 has no slot
;; before it, so its interval is its own frame alone.
;;
;; THE SNAP DOES NOT HAPPEN HERE, although the interval is born here and
;; nothing would need threading. One native frame per output frame means
;; the WHOLE PICTURE reading 13 instead of 15 — a head going two frames
;; stale, a 67ms hitch at 12fps, to fix one group's mouth. It is per
;; group, so it is seated where groups exist.
(if (pos? f) (cadence/frame (dec f) grid native) -1)
(when context? (:default palette))
nil))
symbol/IResolver
(world-of [_ path] (symbol/world-of r path))
(frame-of [_ path] (symbol/frame-of r path))
(pre-frame-of [_ path] (symbol/pre-frame-of r path))
IActivePalette
(active-palette [_] @active-palette-state))))))
(defn center
"The middle of everything symbol `sid` draws, over all its frames, in its own
coordinates. ALL frames rather than the first, so a symbol whose drawing
enters late, or travels, still has its middle where the drawing is. A symbol
that draws nothing gets the STAGE's middle, which is where a drawing made into
it will be, because drawings are made on the stage.
WHERE A DROP LANDS, AND WHAT THE INSTANCE TURNS ABOUT. `place-symbol` puts this
point under the pointer and stores it as the instance's `[:xform :pivot]`, and
`ui/drag`'s ghost draws the cross there so a drop lands where it was aimed —
one point, one meaning, three uses.
IT IS A DEFAULT AND NOT A CACHE, which is the distinction the stored anchor got
wrong. A pivot written here is a CHOICE made on the instance's behalf at the
moment it is placed, the same way `paint/centred` chooses a drawing's origin
when it is drawn; editing the symbol afterwards does not revise either, and the
cross is draggable so neither is a trap. What the anchor got wrong was being
invisible and unmovable, not being stored."
[clip store sid]
(let [resolve (resolver clip sid store pal/index-of {:grid-fps (fps clip sid)})
bounds (fn [[x0 y0 x1 y1 :as b] x y]
(if b [(min x0 x) (min y0 y) (max x1 x) (max y1 y)] [x y x y]))
[x0 y0 x1 y1]
(reduce
(fn [b {:keys [kind pts n cx cy r size]}]
(case kind
:poly (reduce (fn [b i] (bounds b (aget pts (* 2 i)) (aget pts (inc (* 2 i)))))
b (range n))
:disc (-> b (bounds (- cx r) (- cy r)) (bounds (+ cx r) (+ cy r)))
:rect (let [h (/ size 2)] (-> b (bounds (- cx h) (- cy h)) (bounds (+ cx h) (+ cy h))))
b))
nil
(mapcat resolve (range (frames clip sid))))]
(if x0
[(/ (+ x0 x1) 2) (/ (+ y0 y1) 2)]
(mapv #(/ % 2) (stage clip sid)))))
(defn place-symbol
"An instance of symbol `sid`, inside symbol `host`, at `frame` of `host`.
THE MIDDLE GOES UNDER THE POINTER, AND IS WHAT THE INSTANCE TURNS ABOUT.
`center` says where the symbol's drawing sits in its own coordinates; `pos` is
set so that point lands on `point`, a stage pixel — without one, a drop on the
timeline, the drawing stays where it was drawn — and the same point is stored as
the instance's `[:xform :pivot]`, so a turn or a scale happens about the middle
of the drawing rather than about the symbol's origin.
WITHOUT THAT PIVOT AN INSTANCE TURNS ABOUT THE CORNER OF THE STAGE. A symbol's
origin is the stage's, because that is where its contents were drawn, so the
middle of a drawing inside one is typically a hundred-odd pixels away from it on
a 320x200 stage. `node/local!` composes about the pivot, so this one stored
point is the difference between spinning in place and orbiting the top-left
corner. See `domain/gesture`.
THE UUID IS AN ARGUMENT. An instance's identity is the key it has in the node
map — it is what `:linked-to`, an export target and a saved leaf all name — so
generating one in here would make this function's result depend on when it was
called, and this namespace is the pure one.
The instance's own time starts where it was dropped: `:at frame` means frame 0
of the symbol plays on `frame` of `host`, which is what dragging something onto
a playhead is asking for. Its `:span` is in its OWN frames — the whole symbol,
0 to its length — wherever it was dropped; see `node/placed-span`.
Refused, returning the clip unchanged, when it would make a cycle: a symbol
cannot be placed inside itself or inside anything it places."
[clip store host sid frame uuid point]
(let [target (symbol clip sid)
end (frames clip host)]
(if (or (nil? target) (nil? end) (nil? frame) (neg? frame) (>= frame end)
(contains-symbol? clip sid host))
clip
(let [middle (center clip store sid)]
(update-symbol
clip host assoc-in [:nodes uuid]
{:id uuid
:kind :instance
:parent nil
;; Lexicographic draw order, as `domain/paint` does it: an instance made
;; later sits above one made earlier, and neither has to renumber.
:z (str "z" (js/Date.now) "-" (name sid))
:span [0 (cadence/frames (:frames target) (fps clip host) (fps clip sid))]
:time {:mode :map :at frame :rate 1}
:source {:symbol sid}
:playback {:in 0 :speed 1 :end :stop}
:channels {[:xform :pos] {:animated? false
:value (if point (mapv - point middle) [0 0])}
[:xform :pivot] {:animated? false :value middle}}})))))
(defn place-sound
"Place a sound at host frame `frame`. Its span uses `source`'s fps when
supplied, otherwise the host's. `rate` is a deliberate playback speed."
[clip host source label length rate frame uuid]
(let [end (frames clip host)]
(if (or (nil? end) (nil? frame) (neg? frame) (>= frame end))
clip
(update-symbol
clip host assoc-in [:nodes uuid]
{:id uuid
:name label
:kind :audio
:parent nil
:z (str "z" (js/Date.now) "-sound")
:source source
:span [0 (max 1 length)]
:time {:mode :map :at frame :rate rate}}))))
(defn fresh-id
"The first `:symbol-N` the clip does not already hold. Readable because an id
shows up in saved leaf paths, and deterministic because this namespace is pure."
[clip]
(first (remove (:symbols clip) (map #(keyword (str "symbol-" %)) (iterate inc 1)))))
(defn new-symbol
"A new, empty symbol `sid`, placed inside `host` at `frame` and running to the
end of it. Placed at the origin, so whatever is drawn into it lands where it was
drawn until the instance is moved."
[clip host sid frame uuid]
(let [end (frames clip host)]
(if (or (nil? end) (symbol clip sid) (nil? frame) (neg? frame) (>= frame end))
clip
(-> clip
(assoc-in [:symbols sid] {:id sid :name (name sid) :fps (fps clip host)
:frames (- end frame) :nodes {}})
(place-symbol nil host sid frame uuid nil)))))
(defn free-id
"`wanted`, or the first `wanted-2`, `wanted-3`… `taken?` does not claim.
Keeps the namespace, so `:sym/face` becomes `:sym/face-2`."
[taken? wanted]
(first (remove taken?
(cons wanted
(map #(keyword (namespace wanted) (str (name wanted) "-" %))
(iterate inc 2))))))
(defn conflicts
"Every hand correction in the document that its base has outgrown, as
`[{:symbol :node :channel :id :why}]`.
SEPARATE FROM `problems` on purpose. A conflict is a document a person still
has to make a decision about — a regeneration changed the topology under a
correction that was right when it was made — and not a reason the document
will not load. Nothing is dropped and nothing is misapplied meanwhile: the
layer stays where it is, the picture is the base, and this is the list a view
offers to resolve."
[clip]
(vec (for [[sid sym] (:symbols clip)
[id n] (:nodes sym)
[prop c] (:channels n)
{:keys [why] :as x} (ch/conflicts c)]
(assoc (select-keys x [:id]) :symbol sid :node id :channel prop :why why))))
(defn problems
"Human-readable reasons this clip will not evaluate or save."
[clip]
(vec
(concat
(for [k (remove clip-keys (keys clip))]
(str "clip has a field with no leaf to save it in: " (pr-str k)))
(when-not (map? (:symbols clip))
[":symbols must be a map of id -> symbol"])
(when (and (contains? clip :analyses) (not (map? (:analyses clip))))
[":analyses must be a map of analysis id -> analysis"])
(for [[id analysis] (:analyses clip)
:when (not= id (:id analysis))]
(str "analysis under key " (pr-str id) " has :id " (pr-str (:id analysis))))
(for [[id subject] (:subjects clip)
:when (not (contains? (:analyses clip) (:analysis subject)))]
(str "subject " (pr-str id) " names missing analysis "
(pr-str (:analysis subject))))
(for [[id subject] (:subjects clip)
:when (not (keyword? (:source-subject subject)))]
(str "subject " (pr-str id) " has no source subject"))
(when (and (contains? clip :palettes) (not (map? (:palettes clip))))
[":palettes must be a map of id -> palette"])
(when (and (contains? clip :default-palette) (map? (:palettes clip))
(not (contains? (:palettes clip) (:default-palette clip))))
[":default-palette must name a project palette"])
(when (and (contains? clip :root) (map? (:symbols clip))
(not (contains? (:symbols clip) (:root clip))))
[(str ":root names missing symbol " (pr-str (:root clip)))])
(for [[id p] (:palettes clip)
:when (or (not= id (:id p)) (not (pal/valid-palette? p)))]
(str "palette " (pr-str id) " is invalid or has a different :id"))
(when-not (or (nil? (:fps clip)) (and (number? (:fps clip)) (pos? (:fps clip))))
[(str ":fps is " (pr-str (:fps clip)) " — a rate is a positive number")])
(for [[id sym] (:symbols clip)
:when (not= id (:id sym))]
(str "symbol under key " (pr-str id) " has :id " (pr-str (:id sym))))
(for [[id sym] (:symbols clip)
p (symbol/problems sym)]
(str "symbol " (pr-str id) ": " p))
(for [[sid sym] (:symbols clip)
[id n] (:nodes sym)
:when (= :instance (:kind n))
missing (remove (:symbols clip) (node/sources n))]
(str "symbol " (pr-str sid) " instance " (pr-str id)
" names missing symbol " (pr-str missing)))
;; THE INVARIANT `place-symbol` AND `ui/drag` ALREADY ENFORCE, stated here so
;; that every command is checked against it rather than the two that remember
;; to ask. A symbol placed inside itself, or inside anything it places, has no
;; finite expansion: `build` above and `nest/audio-tracks` both walk instances
;; and both throw on the way round. Paste reached this function without it and
;; wrote a document that saved, loaded, and only then threw — which is the one
;; outcome `problems` exists to make impossible.
(for [[sid sym] (:symbols clip)
[id n] (:nodes sym)
:when (= :instance (:kind n))
src (node/sources n)
;; A source that does not exist is the rule above's to report, not this
;; one's, so it does not get named twice.
:when (and (contains? (:symbols clip) src)
(contains-symbol? clip src sid))]
(str "symbol " (pr-str sid) " instance " (pr-str id) " places "
(pr-str src) (if (= src sid) ", which is itself" ", which contains it")))
;; Pose tracks belong to this cel's single source symbol.
(for [[sid sym] (:symbols clip)
[id n] (:nodes sym)
:when (= :instance (:kind n))
:let [targets (keep #(get-in clip [:symbols %]) (node/sources n))
active (filter (fn [node]
(some :pose-sampled? (vals (:channels node))))
(mapcat #(vals (:nodes %)) targets))
groups (set (concat
(map #(or (:pose-group %) (:id %)) active)
(map #(vector :node (:id %)) active)))]
p (pose/problems (get-in n [:playback :tracks])
(apply max 0 (keep :frames targets)) groups)]
(str "symbol " (pr-str sid) " instance " (pr-str id) ": " p))
(for [[sid sym] (:symbols clip)
[id n] (:nodes sym)
:when (and (= :audio (:kind n)) (:linked-to n)
(not (contains? (:nodes sym) (:linked-to n))))]
(str "symbol " (pr-str sid) " audio " (pr-str id)
" links to missing node " (pr-str (:linked-to n))))
(feature/problems clip))))

321
frontend/src/arthur/domain/clipboard.cljs
View file

@ -1,321 +0,0 @@
(ns arthur.domain.clipboard
"Pure multi-node clipboard commands.
A clipboard value is a detached forest of node maps. Normal copies keep symbol
references; `duplicate` with `:unique?` copies the complete referenced symbol
graph once for the whole forest. UI state, playhead conversion, and history
stay in events.ui. See docs/clipboard-plan.md."
(:require [arthur.domain.bring :as bring]
[arthur.domain.clip :as clip]
[arthur.domain.nest :as nest]
[arthur.domain.node :as node]
[arthur.domain.span :as span]
[arthur.domain.symbol :as symbol]))
(defn- prefix? [a b]
(and (<= (count a) (count b)) (= a (subvec b 0 (count a)))))
(defn- node-selections [clip selections]
(->> selections
(keep (fn [[kind sid id path :as address]]
(when (and (= :node kind) id (get-in clip [:symbols sid :nodes id]))
{:address address :sid sid :id id :path (vec (or path [id]))})))
;; One owned node reached through two shared occurrences is still one edit.
(reduce (fn [{:keys [seen out] :as acc} {:keys [sid id] :as x}]
(if (contains? seen [sid id]) acc
{:seen (conj seen [sid id]) :out (conj out x)}))
{:seen #{} :out []})
:out))
(defn canonical
"Valid selected node occurrences, with anything visibly below another selected
occurrence omitted. Order is selection order and therefore keeps the primary
member last in the ordinary case."
[clip selections]
(let [xs (node-selections clip selections)]
(filterv (fn [{p :path}]
(not-any? (fn [{q :path}]
(and (< (count q) (count p)) (prefix? q p)))
xs))
xs)))
(defn- subtree [nodes root]
(into {} (filter (fn [[id _]] (some #{root} (symbol/lineage nodes id)))) nodes))
(defn snapshot
"Snapshot the canonical selected forest, or `{:refused why}`."
[clip selections]
(let [roots (canonical clip selections)]
(if (empty? roots)
{:refused "select something to copy"}
{:clipboard
{:items
(mapv (fn [{:keys [sid id path]}]
(let [nodes (get-in clip [:symbols sid :nodes])]
{:sid sid :root id :path path :parent (:parent (get nodes id))
:nodes (subtree nodes id)}))
roots)}})))
(defn cut
"Delete a previously snapshotted forest. Snapshotting first is what makes cut
retain data even though its source nodes are gone."
[clip {:keys [items]}]
(let [after (reduce (fn [c {:keys [sid root]}] (nest/delete-node c sid root)) clip items)]
(if-let [why (first (clip/problems after))]
{:refused why}
{:clip after :selections []})))
(defn- fresh
[taken fresh-id]
(loop [id (fresh-id)]
(if (contains? taken id) (recur (fresh-id)) id)))
(defn- allocate
[clip items fresh-id]
(loop [pending (vec (mapcat (fn [[i item]] (map #(vector i %) (keys (:nodes item))))
(map-indexed vector items)))
taken (into #{} (mapcat (comp keys :nodes val) (:symbols clip)))
ids {}]
(if-let [k (first pending)]
(let [id (fresh taken fresh-id)]
(recur (subvec pending 1) (conj taken id) (assoc ids k id)))
ids)))
(defn- unique-content
[clip items unique?]
(let [roots (into #{} (comp (mapcat #(vals (:nodes %))) (keep node/source)) items)]
(if (and unique? (seq roots))
(let [{c :clip ids :ids} (bring/symbols clip clip roots {})]
[c ids])
[clip {}])))
(defn- materialize
[clip items fresh-id unique? paste?]
(let [[clip source-ids] (unique-content clip items unique?)
ids (allocate clip items fresh-id)
made
(mapv
(fn [[i {:keys [sid root path parent nodes]}]]
(let [id-of #(get ids [i %])
copied (into {}
(map (fn [[old n]]
(let [id (id-of old)]
[id (cond-> (assoc n :id id)
(:parent n) (assoc :parent (id-of (:parent n)))
(:stencil n) (assoc :stencil (id-of (:stencil n)))
(node/source n)
(assoc-in [:source :symbol]
(get source-ids (node/source n)
(node/source n))))])))
nodes)
new-root (id-of root)
;; Paste reparents roots into one explicit destination.
;; Duplicate leaves them beside their originals.
copied (assoc-in copied [new-root :parent]
(when-not paste? parent))]
{:source-sid sid :old-root root :path path :root new-root
:nodes copied}))
(map-indexed vector items))]
{:clip clip :items made}))
(defn- shifted [n delta]
(if (node/placed-span n)
(update-in n [:time :at] (fnil + 0) delta)
n))
(defn- top-zs [nodes n]
(let [base (or (last (sort (map #(or (:z %) "") (vals nodes)))) "")]
(map #(str base (apply str (repeat % "m"))) (range 1 (inc n)))))
(defn- add-composition
[clip sid items at]
(let [starts (keep #(some-> (get-in % [:nodes (:root %)]) node/placed-span first) items)
anchor (when (seq starts) (apply min starts))
delta (if anchor (- at anchor) 0)
existing (get-in clip [:symbols sid :nodes])
zs (top-zs existing (count items))
nodes (reduce (fn [nodes [{:keys [root] copied :nodes} z]]
(into nodes (assoc-in copied [root]
(-> (get copied root)
(shifted delta)
(assoc :z z)))))
existing (map vector items zs))]
(assoc-in clip [:symbols sid :nodes] nodes)))
(defn- add-lane
[clip sid items at fresh-id]
(let [roots (map #(get-in % [:nodes (:root %)]) items)
starts (map #(some-> % node/placed-span first) roots)
anchor (when (every? some? starts) (apply min starts))
intervals (when anchor
(sort-by first
(map (fn [n]
(let [[lo hi] (node/placed-span n)]
[(+ at (- lo anchor)) (+ at (- hi anchor))]))
roots)))
overlaps? (some (fn [[[a b] [c d]]] (and (< a d) (< c b)))
(partition 2 1 intervals))]
(if (some nil? starts)
{:refused "a lane accepts copied things only when they have a finite span"}
(if overlaps?
{:refused "overlapping copied things cannot be pasted into one lane"}
(reduce
(fn [result item]
(if (:refused result)
(reduced result)
(let [c (:clip result)
root (:root item)
n (get-in item [:nodes root])
desired (+ at (- (first (node/placed-span n)) anchor))
r (span/place-node c sid n desired
{:extent :grow-symbol
:remainder-id (fresh (into #{} (keys (get-in c [:symbols sid :nodes])))
fresh-id)})]
(if-let [made (:clip r)]
{:clip (update-in made [:symbols sid :nodes]
into (dissoc (:nodes item) root))}
r))))
{:clip clip} items)))))
(defn paste
"Paste `clipboard` into `sid`, anchoring its first finite start at `at`.
`fresh-id` is supplied by the event so this domain command remains testable."
[clip clipboard sid at {:keys [fresh-id] :or {fresh-id random-uuid}}]
(cond
(nil? (clip/symbol clip sid)) {:refused "the paste target no longer exists"}
(not (and (integer? at) (not (neg? at))))
{:refused "the playhead is not on one frame of the paste target"}
(empty? (:items clipboard)) {:refused "there is nothing to paste"}
:else
(let [{base :clip items :items} (materialize clip (:items clipboard) fresh-id false true)
r (if (symbol/lane? (clip/symbol base sid))
(add-lane base sid items at fresh-id)
{:clip (add-composition base sid items at)})
made (:clip r)
why (when made (first (clip/problems made)))]
(cond
(:refused r) r
why {:refused why}
:else {:clip made :roots (mapv :root items)}))))
(defn- add-duplicate-composition [clip sid items]
(let [existing (get-in clip [:symbols sid :nodes])
zs (top-zs existing (count items))]
(assoc-in clip [:symbols sid :nodes]
(reduce (fn [nodes [{:keys [root] copied :nodes} z]]
(into nodes (assoc-in copied [root :z] z)))
existing (map vector items zs)))))
(defn- add-duplicate-lane
[clip sid items fresh-id]
(let [old-roots (map #(get-in clip [:symbols sid :nodes (:old-root %)]) items)
spans (map node/placed-span old-roots)
start (apply min (map first spans))
end (apply max (map second spans))
duration (- end start)
selected (set (map :old-root items))
shifted-clip
(update-in clip [:symbols sid :nodes]
(fn [nodes]
(reduce (fn [ns n]
(let [lo (some-> (node/placed-span n) first)]
(if (and (nil? (:parent n))
(not (contains? selected (:id n)))
lo (>= lo end))
(update-in ns [(:id n) :time :at] (fnil + 0) duration)
ns)))
nodes (vals nodes))))]
(reduce
(fn [result item]
(if (:refused result)
(reduced result)
(let [c (:clip result)
root (:root item)
n (get-in item [:nodes root])
old (get-in clip [:symbols sid :nodes (:old-root item)])
desired (+ (first (node/placed-span old)) duration)
r (span/place-node c sid n desired
{:extent :grow-symbol
:remainder-id (fresh (into #{} (keys (get-in c [:symbols sid :nodes])))
fresh-id)})]
(if-let [made (:clip r)]
{:clip (update-in made [:symbols sid :nodes]
into (dissoc (:nodes item) root))}
r))))
{:clip shifted-clip}
(sort-by #(first (node/placed-span
(get-in clip [:symbols sid :nodes (:old-root %)]))) items))))
(defn duplicate
"Duplicate a snapshot beside its sources. Direct finite children of lane
symbols repeat forward and ripple later cels; everything else copies in place.
With `:unique?`, referenced symbol graphs are deep-copied once for the batch."
[clip clipboard {:keys [fresh-id unique?] :or {fresh-id random-uuid}}]
(if (empty? (:items clipboard))
{:refused "select something to duplicate"}
(let [{base :clip items :items}
(materialize clip (:items clipboard) fresh-id unique? false)
groups (vals (group-by :source-sid items))
result
(reduce
(fn [result group]
(if (:refused result)
(reduced result)
(let [c (:clip result)
sid (:source-sid (first group))
lane? (symbol/lane? (clip/symbol c sid))
[lane-items other]
((juxt filter remove)
#(let [old (get-in clip [:symbols sid :nodes (:old-root %)])]
(and lane? (nil? (:parent old)) (node/placed-span old)))
group)
c (if (seq other) (add-duplicate-composition c sid other) c)
r (if (seq lane-items)
(add-duplicate-lane c sid lane-items fresh-id)
{:clip c})]
r)))
{:clip base} groups)
made (:clip result)
why (when made (first (clip/problems made)))]
(cond
(:refused result) result
why {:refused why}
:else {:clip made
:roots (mapv (fn [{:keys [source-sid root path]}]
{:sid source-sid :id root
:path (conj (vec (butlast path)) root)})
items)}))))
(defn move-many
"Reparent the selected roots atomically, preserving their world transforms and
clocks. Optional delta places the forest later on the open ruler."
[document store open selections to frame delta]
(let [roots (canonical document selections)
under? (fn [path] (prefix? path (vec to)))]
(cond
(empty? roots) {:refused "select something to move"}
(some #(under? (:path %)) roots) {:refused "a selection cannot go inside itself"}
:else
(let [moved
(reduce (fn [result {:keys [path]}]
(if (:refused result) (reduced result)
(let [r (nest/move-node (:clip result) store open path to frame)]
(if (:refused r) (reduced r)
{:clip (:clip r)
:selections (conj (:selections result)
[:node (:sid r) (:id r) (conj (vec to) (:id r))])}))))
{:clip document :selections []} roots)]
(if (:refused moved) moved
(let [shifted (nest/slide-many (:clip moved) open
(mapv #(nth % 3) (:selections moved)) (or delta 0))
checked (if (:refused shifted) shifted
(reduce (fn [result sid]
(if (:refused result) (reduced result)
(span/finish (:clip result) sid
(get-in (:clip result) [:symbols sid :nodes])
nil :grow-symbol)))
shifted (distinct (map :sid roots))))]
(if (:refused checked) checked
(if-let [why (first (clip/problems (:clip checked)))]
{:refused why}
(assoc checked :selections (:selections moved))))))))))

193
frontend/src/arthur/domain/correction.cljs
View file

@ -1,193 +0,0 @@
(ns arthur.domain.correction
"Pure commands that author and resolve correction layers.
Evaluation belongs to `channel`; this namespace only constructs a layer,
places it on its owning node, and refuses a document that would not be valid."
(:require [arthur.domain.channel :as ch]
[arthur.domain.clip :as clip]
[arthur.domain.node :as node]))
(def ^:private supported-paths #{[:xform :rot] [:xform :pos]})
(def ^:private motions #{:constant :ramp :return})
(defn- finite? [x] (and (number? x) (js/Number.isFinite x)))
(defn- numeric-value? [v]
(or (finite? v)
(and (vector? v) (pos? (count v)) (every? finite? v))))
(defn- same-shape? [a b]
(or (and (number? a) (number? b))
(and (vector? a) (vector? b) (= (count a) (count b)))))
(defn- expected-value? [path v]
(case path
[:xform :rot] (finite? v)
[:xform :pos] (and (vector? v) (= 2 (count v)) (every? finite? v))
false))
(defn- values-channel
[{:keys [motion support delta start end peak peak-frame]}]
(let [[a b] support]
(case motion
:constant (ch/framed delta)
:ramp (ch/keyed {a start, (dec b) end} :linear)
:return (ch/keyed {a start, peak-frame peak, (dec b) start} :linear)
nil)))
(defn- invalid
[path {:keys [id support motion delta start end peak peak-frame]} existing]
(let [[a b] (when (and (vector? support) (= 2 (count support))) support)
samples (case motion :constant [delta] :ramp [start end]
:return [start peak] [])]
(cond
(nil? id) "a correction needs an ID"
(some #(= id (:id %)) existing) "the correction ID is already used on this channel"
(not (contains? supported-paths path)) "that property does not support correction authoring"
(not (contains? motions motion)) "choose constant, ramp, or return motion"
(not (and (integer? a) (integer? b) (< a b)))
"support must be an increasing [in out) of whole owner frames"
(not-every? numeric-value? samples) "correction values must be finite numbers"
(not-every? #(expected-value? path %) samples)
"correction values do not have the property's shape"
(and (= :ramp motion) (< (- b a) 2)) "a ramp needs at least two samples"
(and (= :return motion) (< (- b a) 3)) "return motion needs at least three samples"
(and (= :return motion)
(not (and (integer? peak-frame) (< a peak-frame (dec b)))))
"the return peak must be a whole owner frame inside both endpoints"
(and (#{:ramp :return} motion) (not (same-shape? start (if (= :ramp motion) end peak))))
"motion endpoints must have the same shape")))
(defn- finish [candidate selection]
(if-let [why (first (clip/problems candidate))]
{:refused why}
{:clip candidate :selection selection}))
(defn add
"Append one offset correction to a node channel.
Support and value keys are in the selected node's own frames. Defaults are
materialized through `node/channels`, so correcting an unkeyed transform does
not need a special representation."
[document sid node-id path spec]
(let [n (get-in document [:symbols sid :nodes node-id])
base (when n (get (node/channels n) path))
existing (:over base)
why (cond
(nil? (clip/symbol document sid)) "the owning symbol does not exist"
(nil? n) "the correction target does not exist"
(nil? base) "the correction target has no such channel"
:else (invalid path spec existing))]
(if why
{:refused why}
(let [layer (ch/layer (:id spec) (:support spec) :offset (values-channel spec))
corrected (update base :over (fnil conj []) layer)
candidate (assoc-in document [:symbols sid :nodes node-id :channels path] corrected)]
(finish candidate node-id)))))
(defn remove-layer
"Remove one named layer, refusing when a later layer depended on its shape."
[document sid node-id path layer-id]
(let [at [:symbols sid :nodes node-id :channels path]
c (get-in document at)
layers (:over c)]
(cond
(nil? c) {:refused "the correction channel does not exist"}
(not-any? #(= layer-id (:id %)) layers) {:refused "the correction does not exist"}
:else (finish (assoc-in document at
(assoc c :over (vec (remove #(= layer-id (:id %)) layers))))
node-id))))
(defn retry-layer
"Clear one recorded conflict when the complete resulting stack is valid."
[document sid node-id path layer-id]
(let [at [:symbols sid :nodes node-id :channels path]
c (get-in document at)
found (some #(when (= layer-id (:id %)) %) (:over c))]
(cond
(nil? found) {:refused "the correction does not exist"}
(nil? (:conflict found)) {:refused "the correction has no recorded conflict"}
:else
(finish (update-in document (conj at :over)
(fn [layers]
(mapv #(if (= layer-id (:id %)) (dissoc % :conflict) %) layers)))
node-id))))
(defn borrow-pose [document sid {:keys [from through donor head?] :as spec} store]
(let [frames (get-in document [:symbols sid :frames])
ids (into #{} (mapcat :nodes)
(filter #(= sid (:symbol %)) (vals (:features document))))
ids (cond-> ids head? (conj :head))
paths (for [id ids [path c] (get-in document [:symbols sid :nodes id :channels])]
[id path c])]
(cond
(not (and (integer? frames) (every? integer? [from through donor])
(<= 0 from through (dec frames)) (<= 0 donor (dec frames))))
{:refused "choose whole face frames within this symbol"}
(<= from donor through) {:refused "choose a clean donor outside the repair interval"}
(empty? paths) {:refused "this symbol has no tracked face features"}
(not-any? (fn [[_ path c]]
(and (= path [:geom :pts])
(not (ch/nothing? (ch/value-at (dissoc c :repairs :over) donor store)))))
paths)
{:refused "the donor has no face pose; choose another frame"}
:else
(finish (reduce (fn [doc [node path _]]
(update-in doc [:symbols sid :nodes node :channels path :repairs]
(fnil conj []) (select-keys spec [:id :from :through :donor])))
document paths) nil))))
(declare remove-eye-keys)
(defn remove-repair [document sid repair-id]
{:clip (reduce (fn [doc [id path]]
(update-in doc [:symbols sid :nodes id :channels path :repairs]
#(vec (remove (fn [r] (= repair-id (:id r))) %))))
(-> document
(remove-eye-keys sid repair-id :l) :clip
(remove-eye-keys sid repair-id :r) :clip)
(for [[id n] (get-in document [:symbols sid :nodes])
[path c] (:channels n) :when (:repairs c)] [id path]))})
(defn eye-key
"Key a procedural lid adjustment and gaze offset over one repair interval."
[document sid repair-id side frame {:keys [opening gaze-x gaze-y]}]
(let [outer (keyword (str "eye-" (name side)))
inner (keyword (str "eye-" (name side) "-in"))
iris (keyword (str "iris-" (name side)))
repair (some #(when (= repair-id (:id %)) %)
(get-in document [:symbols sid :nodes outer :channels [:geom :pts] :repairs]))
{:keys [from through]} repair
layer-id (str repair-id "/eye/" (name side))
edits [[outer [:geom :pts] :eye-opening opening 1]
[inner [:geom :pts] :eye-opening opening 1]
[iris [:xform :pos] :offset [gaze-x gaze-y] [0 0]]]]
(cond
(nil? repair) {:refused "this eye has no such repair interval"}
(not (and (integer? frame) (<= from frame through)))
{:refused "move the playhead inside this repair interval"}
(not (and (every? finite? [opening gaze-x gaze-y]) (<= 0 opening 3)))
{:refused "eye opening must be between 0 and 3; gaze offsets must be finite"}
:else
(finish
(reduce
(fn [doc [id path op value neutral]]
(update-in doc [:symbols sid :nodes id :channels path :over]
(fn [layers]
(let [existing (some #(when (= layer-id (:id %)) %) layers)
values (or (:values existing)
(ch/keyed {from neutral through neutral} :linear))
layer (ch/layer layer-id [from (inc through)] op
(assoc-in values [:keys frame] value))]
(conj (vec (remove #(= layer-id (:id %)) layers)) layer)))))
document edits)
nil))))
(defn remove-eye-keys [document sid repair-id side]
(let [layer-id (str repair-id "/eye/" (name side))]
{:clip (reduce (fn [doc [id path]]
(update-in doc [:symbols sid :nodes id :channels path :over]
#(vec (remove (fn [l] (= layer-id (:id l))) %))))
document
(for [[id n] (get-in document [:symbols sid :nodes])
[path c] (:channels n) :when (:over c)] [id path]))}))

47
frontend/src/arthur/domain/crc32.cljs
View file

@ -1,47 +0,0 @@
(ns arthur.domain.crc32
"CRC-32, as PNG chunks and ZIP entries both define it.
ONE implementation for both, and that is not premature sharing: a PNG chunk's
trailing checksum and a ZIP local header's `crc-32` field are the same function
of the same bytes — IEEE 802.3, reflected, with an initial and final complement
— down to the polynomial. Two copies would be two chances to get the table
wrong in a way that reads as \"the file is corrupt\" rather than as \"these two
functions disagree\".
It lives beside `domain/sha256` for the same reason that one does: a digest is
a pure function of bytes with no DOM in it, so every assertion about it runs
under node.")
(def ^:private table
;; The standard 256-entry table, built once. The bit-twiddling loop IS the
;; definition of the polynomial and there is no collection idiom hiding in it:
;; each entry is eight dependent shifts of one accumulator.
(let [t (js/Uint32Array. 256)]
(dotimes [n 256]
(aset t n (loop [c n k 0]
(if (= k 8)
c
(recur (if (odd? c)
(bit-xor 0xedb88320 (unsigned-bit-shift-right c 1))
(unsigned-bit-shift-right c 1))
(inc k))))))
t))
(defn of
"CRC-32 of a byte array, or of the half-open range [from to) of one, as an
unsigned 32-bit number.
`loop` over the bytes rather than a reduce over a `range`: this walks the whole
of every PNG written, which at 1920x1200 is seven megabytes a frame, and a seq
cell per byte is the allocation the rest of this codebase is arranged to
avoid."
([bytes] (of bytes 0 (.-length bytes)))
([bytes from to]
(-> (loop [c 0xffffffff i from]
(if (>= i to)
c
(recur (bit-xor (aget table (bit-and (bit-xor c (aget bytes i)) 0xff))
(unsigned-bit-shift-right c 8))
(inc i))))
(bit-xor 0xffffffff)
(unsigned-bit-shift-right 0))))

67
frontend/src/arthur/domain/creation.cljs
View file

@ -1,67 +0,0 @@
(ns arthur.domain.creation
"Resolve where a new thing goes from the primary selection and playhead.
This namespace owns no editor state. A row address is a preference; walking
the occurrence path at one frame answers which preferred or enclosing symbol
is actually available. See `docs/creating-in.md`."
(:require [arthur.domain.clip :as clip]
[arthur.domain.nest :as nest]
[arthur.domain.node :as node]
[arthur.domain.symbol :as symbol]))
(defn- path-node
"The node at the end of an occurrence `path`, walked from `open`.
Selection also carries an owner sid and node id for commands that edit the
node directly. Those are deliberately not used here: the path is the address
of the row as seen from the open symbol, and is the only part that describes
every enclosing occurrence at arbitrary depth."
[document open path]
(loop [sid open [id & more] (seq path) found nil]
(if-not id
found
(when-let [n (get-in document [:symbols sid :nodes id])]
(if (seq more)
(when-let [inner (and (= :instance (:kind n)) (node/source n))]
(recur inner more n))
n)))))
(defn preferred-path
"The container path structurally implied by `selection`.
Selecting an instance means inside it. Selecting any other node means its
containing symbol. An empty or non-node selection means the open symbol."
[document open selection]
(let [[kind _sid id selected-path] selection
path (when (= :node kind)
(vec (or (seq selected-path) (when id [id]))))
selected (path-node document open path)]
(cond
(empty? path) []
(= :instance (:kind selected)) path
:else (vec (butlast path)))))
(defn target
"The nearest creation context available at `frame` of `open`.
Every non-empty candidate is validated by `nest/inside`, so every enclosing
symbol occurrence must be under the playhead in the current context. Walking
outward stops at the first valid symbol. A lane needs no cel at that frame,
but the occurrence chain that reaches the lane must still be valid. The empty
path always resolves to the open symbol.
A tracing symbol is never one: it holds a picture to draw over and no nodes,
so selecting a tracing layer creates beside it, as selecting a shape does.
Returns `{:kind :lane|:symbol :sid :path :frame :matrix :time}`."
[document store open selection frame]
(let [preferred (preferred-path document open selection)]
(some (fn [path]
(when-let [inside (nest/inside document store open path frame)]
(when-let [sid (and (:sid inside)
(not (clip/trace? (clip/symbol document (:sid inside))))
(:sid inside))]
(assoc inside :path path
:kind (if (symbol/lane? (clip/symbol document sid))
:lane :symbol)))))
(take (inc (count preferred)) (iterate pop preferred)))))

88
frontend/src/arthur/domain/cut.cljs
View file

@ -1,88 +0,0 @@
(ns arthur.domain.cut
"The eraser's cut: a shape's ring with a stroke taken out of it.
Illustrator's and Flash's eraser, not a raster one: what is left is the
SHAPE, with the cut edge made of its own points — points the pen moves, keys
and tweens like any others. A cut through the middle leaves two pieces, and
a cut inside it leaves a hole, which is bridged into the one ring as a brush
stroke's is (see `outline`).
In stage pixels, on both sides: the preview cuts the rings it is about to
draw, and the saved cut cuts the same rings and takes the answer back into
the shape's own coordinates — one function, so the preview is the result."
(:require ["polygon-clipping" :as clipping]
[arthur.domain.channel :as channel]
[arthur.domain.nest :as nest]
[arthur.domain.node :as node]
[arthur.domain.outline :as outline]
[arthur.domain.paint :as paint]))
(defn- area [ring]
(let [ps (vec (partition 2 ring)) n (count ps)]
(js/Math.abs (/ (reduce + (map (fn [i] (let [[ax ay] (ps i) [bx by] (ps (mod (inc i) n))]
(- (* ax by) (* bx ay))))
(range n)))
2))))
(defn cut
"Ring `ring` with `cutters` — each `[outer & holes]` — taken out of it: one
ring per piece left, biggest first, an empty vector when nothing is left, or
nil when the cut does not touch it."
[ring cutters]
(let [shape #js [(outline/->js ring)]
knife (into-array (map #(into-array (map outline/->js %)) cutters))]
(when (seq (array-seq (clipping/intersection shape knife)))
(->> (array-seq (clipping/difference shape knife))
(map (fn [^js poly] (mapv outline/->ring (array-seq poly))))
(sort-by (comp - area first))
(mapv outline/join)))))
(defn- through [m pts]
(let [out (js/Float64Array. 2)]
(into [] (mapcat (fn [[x y]] (node/apply-pt! out 0 m x y) [(aget out 0) (aget out 1)]))
(partition 2 pts))))
(defn erase
"`clip` with `cutters`, in the stage pixels of symbol `open` at frame `f`,
cut out of each shape at row path in `paths`, on the frame each is showing.
The shape keeps the biggest piece, on a key at that frame — made there if
the frame had none, so the keys either side keep their points. Every other
piece is a new shape of the same colour, its id the next of `ids`. A shape
cut away entirely is deleted.
TWO SPACES COME BACK OUT, and they are not the same one. The piece the shape
KEEPS is written into that shape's own geometry, so it comes back through
`world⁻¹`, the node's own coordinates. Every other piece becomes a NEW node
beside it, whose points are read against its own fresh transform, so those come
back through `parent⁻¹` — the space a node's `pos` lives in, which is what
`paint/new-shape` takes and centres.
Both were `world⁻¹` before, and that was wrong for the new pieces by exactly
the cut shape's own transform. It could not be seen while every drawing had an
identity transform, which was true of all of them for as long as a stroke was
stored exactly as drawn: the two spaces coincided, so erasing a shape nobody had
moved worked, and erasing one somebody had moved scattered the offcuts."
[clip store open f paths cutters ids]
(first
(reduce
(fn [[clip ids] path]
(let [{:keys [sid id frame world parent]} (nest/placement clip store open path f)
n (get-in clip [:symbols sid :nodes id])
geom (get-in n [:channels paint/geometry])
inv (when world (node/invert world))
up (when parent (node/invert parent))
left (when (and inv up geom)
(cut (through world (channel/value-at geom frame store)) cutters))]
(cond
(nil? left) [clip ids]
(empty? left) [(nest/delete-node clip sid id) ids]
:else
(let [[keep & more] left
colour (channel/value-at (get-in n [:channels [:style :color]]) frame store)
kept (-> (if (contains? (:keys geom) frame) clip (paint/add-key clip sid id frame))
(paint/set-points sid id frame (through inv keep)))]
[(reduce (fn [c [nid pts]] (paint/new-shape c sid nid frame (through up pts) colour))
kept (map vector ids more))
(drop (count more) ids)]))))
[clip ids] paths)))

98
frontend/src/arthur/domain/feature.cljs
View file

@ -1,98 +0,0 @@
(ns arthur.domain.feature
"Tracked subjects, feature ownership, and eye-pair settings.
Features name their symbol explicitly; node ids are local to that symbol."
(:require [arthur.domain.params :as params]))
(defn owned
"A deterministic clip-level feature or group id. Nodes keep local names."
[subject role]
(keyword (subs (str subject) 1) (name role)))
(defn group-for [clip feature-id]
(first (filter (fn [[_ group]] (some #{feature-id} (:members group)))
(:groups clip))))
(defn effective-params
"Resolve static settings for one feature. A future parameter channel can
replace a scalar at this boundary without changing feature or pair identity."
[clip feature-id]
(let [{:keys [subject area params] :as feature} (get-in clip [:features feature-id])
[_ group] (group-for clip feature-id)]
(when-not feature
(throw (ex-info "unknown feature" {:feature feature-id})))
(merge (params/for-area :subject)
(get-in clip [:subjects subject :params])
(params/for-area area)
(:params group)
params)))
(defn remove-from-pair
"Keep the eye's current settings when its association is removed. Empty pairs
are removed; a one-eye pair remains valid and can acquire a partner later."
[clip feature-id]
(if-let [[group-id group] (group-for clip feature-id)]
(let [area (get-in clip [:features feature-id :area])
values (select-keys (effective-params clip feature-id)
(keys (params/for-area area)))
members (vec (remove #{feature-id} (:members group)))]
(-> clip
(assoc-in [:features feature-id :params] values)
(update :groups (fn [groups]
(if (seq members)
(assoc-in groups [group-id :members] members)
(dissoc groups group-id))))))
clip))
(defn problems
"Check tracked identities and symbol-local node ownership."
[clip]
(let [subjects (:subjects clip)
features (:features clip)
groups (:groups clip)
memberships (mapcat (comp :members val) groups)
node-owners (for [[_ f] features n (:nodes f)]
[(:symbol f) n])]
(vec
(concat
(for [[id s] subjects :when (not= id (:id s))]
(str "subject " (pr-str id) " has a different :id"))
(for [[id s] subjects
:when (not (params/valid-settings? :subject (or (:params s) {})))]
(str "subject " (pr-str id) " has invalid settings"))
(for [[id _] subjects
:when (not (seq (get-in clip [:symbols id :nodes :head :measured])))]
(str "subject " (pr-str id) " has no measured head in its symbol"))
(for [[id f] features :when (not= id (:id f))]
(str "feature " (pr-str id) " has a different :id"))
(for [[id f] features :when (not (contains? subjects (:subject f)))]
(str "feature " (pr-str id) " has no subject"))
(for [[id f] features :when (not (contains? (disj params/areas :subject) (:area f)))]
(str "feature " (pr-str id) " has an unknown area"))
(for [[id f] features
:when (not (params/valid-settings? (:area f) (or (:params f) {})))]
(str "feature " (pr-str id) " has invalid settings for " (pr-str (:area f))))
(for [[id f] features
:when (not (contains? (:symbols clip) (:symbol f)))]
(str "feature " (pr-str id) " names a missing symbol"))
(for [[id f] features node-id (:nodes f)
:let [owned-nodes (get-in clip [:symbols (:symbol f) :nodes])]
:when (not (contains? owned-nodes node-id))]
(str "feature " (pr-str id) " refers to missing node " (pr-str node-id)))
(for [[id n] (frequencies node-owners) :when (> n 1)]
(str "node " (pr-str id) " belongs to more than one feature"))
(for [[id g] groups :when (not= id (:id g))]
(str "group " (pr-str id) " has a different :id"))
(for [[id g] groups
:when (not (and (= :eye-pair (:kind g))
(<= 1 (count (:members g)) 2)
(= (count (:members g)) (count (distinct (:members g))))))]
(str "group " (pr-str id) " must be an eye pair of one or two distinct eyes"))
(for [[id g] groups
:when (not (params/valid-settings? :eye (or (:params g) {})))]
(str "group " (pr-str id) " has invalid eye settings"))
(for [[id g] groups member (:members g)
:let [f (get features member)]
:when (not (and f (= :eye (:area f)) (= (:subject g) (:subject f))))]
(str "group " (pr-str id) " has an eye from another subject or an unknown feature"))
(for [[id n] (frequencies memberships) :when (> n 1)]
(str "feature " (pr-str id) " belongs to more than one group"))))))

136
frontend/src/arthur/domain/geom.cljs
View file

@ -1,136 +0,0 @@
(ns arthur.domain.geom
"2D similarity transforms and temporal smoothing.
A transform is {:s :theta :tx :ty}; a point is {:x :y}. Both stay maps at this
layer: this is the numeric oracle the JS is diffed against, and a faithful
port is worth more here than a fast one. The dense typed-array
representations appear at the freeze boundary, not below it.")
(defn centroid
"Mean of a point set."
[pts]
(let [n (count pts)]
{:x (/ (transduce (map :x) + 0.0 pts) n)
:y (/ (transduce (map :y) + 0.0 pts) n)}))
(defn fit-similarity
"Least-squares similarity (translation + rotation + uniform scale, 4 DOF)
mapping P onto Q. Closed form; no iteration.
Deliberately NOT affine or homography: the extra degrees of freedom absorb
out-of-plane head rotation as shear/perspective and smear it into the mouth.
Four DOF removes exactly translation, roll and depth-scale, and leaves yaw and
pitch as a measurable residual."
[P Q]
(let [n (count P)
cp (centroid P)
cq (centroid Q)
;; Dot, cross and squared norm of the centred configurations, in one
;; pass. Reduced in input order, so the floating-point result is bit for
;; bit what an index loop would give and the 1e-9 parity against the JS
;; holds.
[a b norm]
(reduce (fn [[a b norm] [p q]]
(let [px (- (:x p) (:x cp)) py (- (:y p) (:y cp))
qx (- (:x q) (:x cq)) qy (- (:y q) (:y cq))]
[(+ a (+ (* px qx) (* py qy))) ; dot
(+ b (- (* px qy) (* py qx))) ; cross
(+ norm (+ (* px px) (* py py)))]))
[0.0 0.0 0.0]
(map vector P Q))
pcx (:x cp) pcy (:y cp)
qcx (:x cq) qcy (:y cq)
theta (js/Math.atan2 b a)
;; A degenerate configuration has nothing to recover a scale from. Fall
;; back to 1 rather than dividing by zero: one bad detection frame would
;; otherwise poison the Procrustes mean and therefore every frame.
s (if (> norm 1e-12) (/ (js/Math.hypot a b) norm) 1)
c (js/Math.cos theta)
sn (js/Math.sin theta)]
{:s s
:theta theta
:tx (- qcx (* s (- (* c pcx) (* sn pcy))))
:ty (- qcy (* s (+ (* sn pcx) (* c pcy))))}))
(defn apply-sim [tf p]
(let [c (js/Math.cos (:theta tf))
sn (js/Math.sin (:theta tf))]
{:x (+ (* (:s tf) (- (* c (:x p)) (* sn (:y p)))) (:tx tf))
:y (+ (* (:s tf) (+ (* sn (:x p)) (* c (:y p)))) (:ty tf))}))
(defn apply-sim-all [tf pts]
(mapv #(apply-sim tf %) pts))
(defn fit-residual
"Residual RMS after the fit, in the units of Q. Rises with out-of-plane
rotation, so it is the signal for \"this section is not stabilisable\"."
[tf P Q]
(let [sq (fn [d] (* d d))]
(js/Math.sqrt
(/ (transduce (map (fn [[p q]]
(let [m (apply-sim tf p)]
(+ (sq (- (:x m) (:x q)))
(sq (- (:y m) (:y q)))))))
+ 0.0 (map vector P Q))
(count P)))))
(defn procrustes-mean
"Generalised Procrustes: the reference is the MEAN rigid configuration over the
shot, not frame zero, so no single frame's idiosyncrasies get baked into every
other frame. Three passes is plenty."
([frames-rigid] (procrustes-mean frames-rigid 3))
([frames-rigid iters]
(let [n (count frames-rigid)
;; One pass: fit every frame onto the current reference, sum the
;; aligned configurations, divide. Iterative refinement, so the whole
;; thing is `iterate` taken `iters` deep — which is what the algorithm
;; actually says, rather than a counter that happens to stop.
refine (fn [ref]
(->> frames-rigid
(reduce (fn [acc rig]
(let [moved (apply-sim-all (fit-similarity rig ref) rig)]
(mapv (fn [a m] {:x (+ (:x a) (:x m))
:y (+ (:y a) (:y m))})
acc moved)))
(mapv (constantly {:x 0.0 :y 0.0}) ref))
(mapv (fn [p] {:x (/ (:x p) n) :y (/ (:y p) n)}))))]
(-> (iterate refine (mapv (fn [p] {:x (:x p) :y (:y p)}) (first frames-rigid)))
(nth iters)))))
(defn moving-average
"`radius` is in frames either side: 0 is off, 1 averages over 3 frames, 2 over
5. Expressed as a radius rather than a window so that \"off\" is 0 and every
value is symmetric - an even window would be lopsided in time."
[vals radius]
(if (<= radius 0)
(vec vals)
(let [v (vec vals)
n (count v)
half (js/Math.floor radius)]
(mapv (fn [i]
;; Clamped at the ends rather than shortened, so every output is an
;; average of the same COUNT of samples and the first frame is not
;; noisier than the rest.
(let [lo (- i half) hi (+ i half)]
(/ (reduce + (map (fn [j] (nth v (min (dec n) (max 0 j))))
(range lo (inc hi))))
(inc (- hi lo)))))
(range n)))))
(defn smooth-transforms
"Smooth the four transform parameters, NEVER the contour. Landmark jitter of a
pixel is smeared into the mouth by the inverse transform, so the transform is
where the low-pass belongs; smoothing the contour would destroy the
performance, which is the entire asset.
Angles are smoothed as (cos, sin) so wrapping cannot produce a spike."
[tfs radius]
(let [c (moving-average (map #(js/Math.cos (:theta %)) tfs) radius)
sn (moving-average (map #(js/Math.sin (:theta %)) tfs) radius)
s (moving-average (map :s tfs) radius)
tx (moving-average (map :tx tfs) radius)
ty (moving-average (map :ty tfs) radius)]
(mapv (fn [i] {:theta (js/Math.atan2 (nth sn i) (nth c i))
:s (nth s i)
:tx (nth tx i)
:ty (nth ty i)})
(range (count tfs)))))

384
frontend/src/arthur/domain/gesture.cljs
View file

@ -1,384 +0,0 @@
(ns arthur.domain.gesture
"Moving, turning and scaling a node by hand on the stage, as channel values.
A drag says where the pointer went in stage pixels; this says what that makes
the node's `[:xform :pos]`, `[:xform :rot]`, `[:xform :scale]` or
`[:xform :pivot]`, given its `nest/placement`. Whatever is above the node —
instances, parents, a `:pinv` — is in the placement's matrices, so a shape five
symbols down moves under the pointer like one on top.
A GESTURE TURNS AND SCALES ABOUT THE NODE'S OWN PIVOT, and nothing here solves
for a position to fake one with. `node/local!` composes the rotation and the
scale about `[:xform :pivot]`, so a turn is `rot` alone, ALWAYS — one channel,
one key, and the pivot held exactly on every frame between two keys because the
matrix is built about it on every frame. This is Toon Boom's layer pivot and
Flash's transformation point, and the reason both store one.
A PIVOT NOBODY HAS CHOSEN IS THE MIDDLE OF WHAT THE NODE DRAWS, and the first
turn or scale WRITES IT DOWN — `pivot` derives it from `pick/bounds-of` on the
frame the drag starts, and `with-pivot` turns that into a `[:xform :pivot]` and
the `[:xform :pos]` that leaves the picture exactly where it is. After that it
is an ordinary stored, keyable, draggable channel, and the derived value is
never consulted again: a pivot is a CHOICE, and re-deriving it per drag means
editing a symbol silently moves what its instances turn about.
WHAT THIS REPLACES, because it was a deletion that cost a feature. There used
to be no pivot in the decomposition at all: a drag derived the middle of the
drawing, and `about` solved for the `pos` that holds that point still under the
new angle. That solution is an ARC in the angle while `pos` interpolates along
the CHORD, so it is exact on the frame it is written and WRONG EVERYWHERE
BETWEEN TWO KEYS. A drawing escaped it, because `paint/centred` puts a shape's
origin on the middle of what it draws and the correction is then nil — but a
SYMBOL INSTANCE cannot: its origin is its symbol's, and a symbol is drawn on the
stage, so its origin is the stage's top-left corner. One keyed turn of an
instance therefore swung its drawing round the corner of the stage on an orbit
the size of the stage. The answer at the time was a peg, and a peg is a real
thing — it is how a pivot is SHARED, or put over a measured transform — but it
is not something anybody should have to make in order to spin a drawing.
`about` is still here and is still that equation, for the one gesture that
genuinely has a pivot belonging to no node: a MULTI-SELECTION scaling about the
middle of its shared box. Nobody keys that.
The normal keying rule is `node/set-channel`, the inspector's: a channel with
keys gets one on the node's own frame, and one without has its one value
changed. Auto-key deliberately replaces that rule with `set-keyed-channel`,
so touching an otherwise static transform starts its animation at this frame."
(:require [arthur.domain.channel :as ch]
[arthur.domain.node :as node]))
(defn values
"Node `n`'s transform on its own frame `f`, as vectors.
`store` IS NOT OPTIONAL, though `ch/value-at` would let it be. A measured
transform is a dense channel, and a dense channel read without the tier-2
store it names throws — so leaving it off read correctly for every hand-placed
node and crashed the stage the moment a selection landed on an iris, a brow or
a head. Those are not hard to land on: `pick/choose` keeps a selection at the
depth it already has, so once anything inside a face is selected, an ordinary
click beside it selects its neighbour — which near the eyes is an iris."
[n f store]
(let [at #(ch/value-at (get (node/channels n) [:xform %]) f store)
xy #(let [v (at %)] [(ch/component v 0) (ch/component v 1)])]
{:pos (xy :pos) :pivot (xy :pivot) :rot (at :rot)
:scale (xy :scale) :skew (xy :skew)
;; WHETHER THE NODE HAS A PIVOT OF ITS OWN, and not what it is: a node
;; with no `[:xform :pivot]` channel reads `[0 0]` off `node/defaults`,
;; which is a real pivot — a drawing's own middle — and also what a node
;; nobody has pivoted yet looks like. The two have to be told apart
;; exactly once, when a drag decides whether to write the derived middle
;; down; see `pivot` and `with-pivot`.
:chosen? (contains? (:channels n) [:xform :pivot])}))
(defn- measured-channel? [n path]
(let [c (get-in n [:channels path])]
(boolean (or (:dense c) (:generated c)))))
(defn refusal
"Why `kind` cannot transform node `n`, or nil. Measured position and rotation
accept authored correction layers; measured scale cannot yet be decomposed
safely. The one-argument form asks whether any stage gesture is possible."
([n] (when (node/measured? n)
"its transform is measured — use a correction, or put a peg over it"))
([n kind]
(when (and (= :scale kind) (measured-channel? n [:xform :scale]))
"its scale is measured — put a peg over it and scale that")))
(defn- through [m [x y]]
(let [out (js/Float64Array. 2)]
(node/apply-pt! out 0 m x y)
[(aget out 0) (aget out 1)]))
(defn linear
"The node's own linear part, `R(rot)·K(skew)·S(scale)`, as a 2x3 whose
translation is zero — so putting a point through it applies the rotation, skew
and scale and nothing else. `node/local!` with a zero `pos` rather than a
second closed form, so there is one place the decomposition is written out."
[{:keys [rot scale skew]}]
(node/local! (node/mat) [0 0] [0 0] rot scale skew))
(defn local-of
"The node's whole local transform, `T(pos)·T(piv)·R·K·S·T(-piv)` — what takes a
point in the node's own coordinates to its parent's."
[{:keys [pos pivot rot scale skew]}]
(node/local! (node/mat) pos pivot rot scale skew))
(defn about
"The `pos` that keeps parent-space point `c` still while the node's linear part
changes from `v`'s to `v'`'s. Nil when `v`'s is singular — a node scaled to
nothing has no point under `c` to hold.
FOR A PIVOT THAT IS NO NODE'S, which since the pivot went into the
decomposition is one gesture and only one: a multi-selection scaling about the
middle of its shared box, where every member has to move to keep the
arrangement and none of them owns the point. `turn` and `scale` do not call
this, and the namespace docstring says why — the position it solves for is an
ARC in the angle and `pos` tweens along the CHORD, so it is exact on the frame
it is written and wrong between two keys.
Local is `T(t)·M` with `t = pos + a − M·a`, the composed translation. The
material point sitting under `c` is `q = M⁻¹(c − t)`, holding it there under the
new `M'` wants `t' = c − M'·q`, and the `pos` that composes to that `t'` is
p' = t' − a + M'·a = c − M'·(q − a) − a
Checkable at both ends it has to be right at: with `c` the node's own pivot
point, `c = pos + a`, so `q = a` and `p' = c − a = pos` — turning about your own
pivot never moves you. And with no pivot at all, `a = 0`, it is the familiar
`p' = c − M'·M⁻¹(c − p)`."
[v v' c]
(let [m (linear v)
l (local-of v)
t [(aget l 4) (aget l 5)]
a (:pivot v)]
(when-let [inv (node/invert m)]
(let [q (through inv (mapv - c t))]
(mapv - (mapv - c (through (linear v') (mapv - q a))) a)))))
(defn middle
"The middle of `bounds` — what the node draws, in its own coordinates — through
its own transform, so a parent-space point; or its own origin when it draws
nothing. What a node nobody has pivoted yet turns about."
[v bounds]
(let [[x0 y0 x1 y1] bounds]
(through (local-of v)
(if bounds [(/ (+ x0 x1) 2) (/ (+ y0 y1) 2)] [0 0]))))
(defn pivot
"The parent-space point a drag on this node turns and scales about: ITS OWN
PIVOT, `pos + piv` — or, for a node nobody has pivoted yet, the middle of
`bounds`, what it draws in its own coordinates, through its own transform.
THE CHOSEN ONE WINS, and that is the whole of the rule. A pivot is where
somebody put it: it does not follow the drawing afterwards, any more than
Flash's transformation point or a Harmony layer's pivot does, because a turn
that quietly changes its centre when a symbol is edited is worse than one
sitting somewhere a hand can see and move it. `ui/stage` draws the cross here
and `::ui/repivot` drags it.
THE DERIVED ONE IS A DEFAULT AND NOT A BEHAVIOUR. `pick/bounds-of` on the same
frame is where the bounds come from, so the cross and the selection box start
out as one computation — and the first turn or scale WRITES IT DOWN, which is
`with-pivot`. A `:group` draws nothing, so `bounds` is nil and this is its own
origin, which is where a peg was put.
`middle` is the derived half on its own, for `centred` — which is the way back
to it once a pivot HAS been chosen."
[v bounds]
(if (:chosen? v) (mapv + (:pos v) (:pivot v)) (middle v bounds)))
(defn at-pivot?
"Is parent-space point `c` where the node already pivots?
WITHIN A MILLIONTH OF A PIXEL, because this asks a question about intent and
gets an answer in floating point: `paint/centred` subtracts the middle of a
ring from its own points, so re-deriving that middle from the result lands on
zero to within the rounding of the subtraction, a part in 1e14 of the
coordinates. A point that close to the pivot IS the pivot — there is no gesture
in which a millionth of a pixel is a pivot somewhere else — and the whole point
of asking is to leave a drawing's channels alone: a pivot of `[0 0]` written
onto a shape that already turns about its own middle is a key nobody asked for
on a value that was already right."
[v c]
(let [[dx dy] (mapv - c (mapv + (:pos v) (:pivot v)))]
(< (js/Math.hypot dx dy) 1e-6)))
(defn repivot
"The channel values that put the node's pivot on parent-space point `c` WITHOUT
MOVING THE PICTURE, or nil when its linear part is singular.
Local is `T(pos)·T(a)·M·T(-a)`, so the new pivot has to be the material point
that is under `c` now, and the new position has to put it there:
a' = a + M⁻¹(c − (pos + a)) the point under c, in the node's own space
p' = c − a' so that p' + a' = c
and then `p' + a' + M(x − a')` is `pos + a + M(x − a)` for EVERY x, which is
the \"moving nothing\" in the first line, exact and not to a tolerance.
WITH NO ROTATION OR SCALE IT DOES NOT TOUCH `pos` AT ALL — `M = I` gives
`a' = c − pos` and `p' = pos` — which is the ordinary case of setting a pivot up
before animating, and is why this can be done quietly inside a first turn
without starting a position channel nobody asked for.
EXACT ON THE FRAME IT IS WRITTEN. `M` is this frame's, so on a node whose
rotation or scale is already keyed, the compensation that holds the picture
still here is not the one that would hold it still three frames later. That is
not an artefact of the arithmetic: moving a pivot genuinely changes what the
keyed angles mean. Flash and Harmony both let you do it and both move the
in-betweens; the alternative is refusing to repivot anything already animated,
which is the node a pivot is most often wrong on."
[v c]
(when-let [inv (node/invert (linear v))]
(let [a' (mapv + (:pivot v) (through inv (mapv - c (mapv + (:pos v) (:pivot v)))))]
{[:xform :pivot] a'
[:xform :pos] (mapv - c a')})))
(defn centred
"The channel values that put the node's pivot back on the middle of what it
draws NOW, moving nothing. Nil when its linear part is singular.
THE WAY BACK, and the thing a stored pivot needs in order to be safe to store.
A pivot does not follow the drawing — that is the point of storing it, since a
keyed spin must not be re-aimed by someone drawing one more shape inside the
symbol — but a drawing does grow, and \"put it back in the middle of what is
there now\" is then an obvious thing to want and an unobvious thing to do by
hand. It is `repivot` at the point `pivot` would have derived, so the button
and the default cannot disagree: this is exactly where an untouched node's
cross already is."
[v bounds]
(repivot v (middle v bounds)))
(defn- with-pivot
"`[v vs]`: the node's transform with its pivot on parent-space `c`, and the
channel values that put it there — or `v` untouched and nil, when that is where
it pivots already or when it has a pivot of its own.
THE ONE PLACE A DERIVED PIVOT BECOMES A STORED ONE. `turn` and `scale` both
start here, so the first drag on a node nobody has pivoted writes the middle of
what it draws down with the gesture, in the same edit, and every drag after it
turns about the stored one. The returned `v` carries the new pivot, because the
gesture itself is measured about it: scaling about the pivot it is in the act of
choosing is one answer, not two."
[v c]
(if (and c (not (:chosen? v)) (not (at-pivot? v c)))
(if-let [vs (repivot v c)]
[(assoc v :pivot (get vs [:xform :pivot]) :pos (get vs [:xform :pos])) vs]
[v nil])
[v nil]))
(defn move
"The node's position with the drag carried from stage point `p0` to `p1`.
ONE CHANNEL, AND IT NEVER DISTURBS THE PIVOT: `[:xform :pivot]` is in the
node's own coordinates, so it travels with the node and a move is `pos` alone,
exactly as it was before there was a pivot at all."
[{:keys [parent]} {:keys [pos]} p0 p1]
(when-let [inv (node/invert parent)]
{[:xform :pos] (mapv + pos (mapv - (through inv p1) (through inv p0)))}))
(defn angle
"The angle of stage point `p` about parent-space pivot `c`, in the space the
node's rotation is in."
[{:keys [parent]} c p]
(when-let [inv (node/invert parent)]
(let [[x y] (mapv - (through inv p) c)]
(js/Math.atan2 y x))))
(defn turn
"The node turned by `da` radians about parent-space point `c`: the rotation, and
— only on the first turn of a node nobody has pivoted — the pivot and position
that put its pivot on `c` without moving it.
ONE CHANNEL, ONCE THE PIVOT IS ITS OWN, and that is the whole reason the pivot
is in `node/local!` rather than solved for here. `rot` is the only thing a turn
changes, so a keyed turn interpolates ONE number and the matrix is composed
about the pivot on every frame of it: the pivot is held exactly between two keys
and not merely at them. A `pos` written beside the rotation — which is what
`about` solves for, and what this used to do for anything whose origin was not
its middle — tweens along the chord of an arc it has no way to know about, and a
360° turn keyed that way leaves the stage in the middle and comes back.
`c` IS A DEFAULT, NOT A TARGET. A node with its own pivot ignores it and turns
about what it has, which is what makes a pivot something a hand can place and
rely on; `pivot` is where `c` comes from either way."
[v c da]
(let [[v vs] (with-pivot v c)]
(merge vs {[:xform :rot] (+ (:rot v) da)})))
(defn scale
"The node's scale with the point under stage `p0` taken to `p1`, about its own
pivot, along the node's own axes — or by the same factor on both when
`uniform?` — and, on the first scale of a node nobody has pivoted, the pivot and
position that put its pivot on `c` without moving it.
The factors are measured in the node's OWN coordinates, which is what makes a
corner drag track the pointer on a node that has been turned, and they are
measured FROM THE PIVOT `q`: scaling by `k` takes `q + d` to `q + k·d`, so the
factor a corner wants is the ratio of its offsets from the pivot before and
after. `with-pivot` runs first because a pivot being chosen by this very drag is
the pivot the drag has to be measured about."
[{:keys [world]} v c p0 p1 uniform?]
(let [[v vs] (with-pivot v c)]
(when-let [winv (node/invert world)]
(let [q (:pivot v)
a (mapv - (through winv p0) q)
b (mapv - (through winv p1) q)
k (fn [a b] (if (< (js/Math.abs a) 1e-6) 1 (/ b a)))
r (when uniform?
(let [aa (reduce + (map * a a))]
(if (< aa 1e-9) 1 (/ (reduce + (map * a b)) aa))))
s (:scale v)
s' (if r (mapv #(* r %) s) (mapv * s (map k a b)))]
(merge vs {[:xform :scale] s'})))))
(defn scale-by
"The node scaled by factor `k` on both axes about parent-space point `c`, and
the position that holds `c` still.
What a MULTI-SELECTION scales by: one factor for everything about the shared
box, so a group of shapes keeps its arrangement instead of each member solving
for its own factors. The point belongs to the box and not to any node in it, so
this is the one gesture that still writes a position to hold a pivot still —
`about`, with everything its docstring says that costs. `scale` is the
single-node form, where the factors come out of the pointer in the node's own
axes and the pivot is the node's own."
[v c k]
(let [v' (update v :scale #(mapv (partial * k) %))]
(cond-> {[:xform :scale] (:scale v')}
c (into (when-let [p (about v v' c)] {[:xform :pos] p})))))
(def ^:private manual-layer ::manual-transform)
(defn- plus [a b]
(if (number? a) (+ a b) (mapv + (vec a) (vec b))))
(defn- minus [a b]
(if (number? a) (- a b) (mapv - (vec a) (vec b))))
(defn- corrected-channel
"Set the effective value of a measured channel without replacing its base."
[channel f target store extent auto-key?]
(let [current (ch/value-at channel f store)
layers (vec (:over channel))
i (first (keep-indexed #(when (= manual-layer (:id %2)) %1) layers))
old (when i (ch/value-at (get-in layers [i :values]) f store))
zero (if (number? target) 0 (vec (repeat (count target) 0)))
value (plus (or old zero) (minus target current))
values (or (when i (get-in layers [i :values])) (ch/framed zero))
values ((if auto-key? node/set-keyed-channel node/set-channel)
{:kind :group :channels {[:xform :pos] values}}
[:xform :pos] f value)
values (get-in values [:channels [:xform :pos]])
layer (ch/layer manual-layer [0 extent] :offset values)]
(assoc channel :over (if i (assoc layers i layer) (conj layers layer)))))
(defn apply-values
"Clip with channel values `vs`, `{path value}`, written into node `id` of
symbol `sid` on the node's own frame `f`, by the keying rule above. The sixth
argument arms auto-key; the five-argument form retains the normal rule."
([clip sid id f vs] (apply-values clip sid id f vs false nil))
([clip sid id f vs auto-key?] (apply-values clip sid id f vs auto-key? nil))
([clip sid id f vs auto-key? store]
(let [put (if auto-key? node/set-keyed-channel node/set-channel)]
(update-in clip [:symbols sid :nodes id]
#(reduce-kv
(fn [n path v]
(if (measured-channel? n path)
(update-in n [:channels path] corrected-channel f v store
(get-in clip [:symbols sid :frames]) auto-key?)
(put n path f v)))
% vs)))))
(defn apply-take
"Apply a buffered performance take. `take` is keyed by `[symbol node]`, then
local frame, then channel path. It becomes ordinary authored keys in one
document edit rather than making the edit pipeline run for every sample."
([clip take] (apply-take clip take nil))
([clip take store]
(reduce-kv
(fn [c [sid id] frames]
(reduce-kv (fn [c f values]
(apply-values c sid id f values true store))
c frames))
clip take)))

Some files were not shown because too many files have changed in this diff Show more