arthur/docs/tracing-symbol-plan.md

Plan: tracing is a symbol

Status: built, 2026-10-03, on branch worktree-tracing-layers. Where the build differs from the plan below, the build wins:

• The head's field is :reads ({:holds [...]} or {:holds-of :plate}), not :follow.
• A still is not one frame. It gets as many frames as remain in the symbol it is dropped into, at that symbol's rate, and is trimmed like any clip.
• An image's :media is {:image <blob sha256>}, served at /blob/<sha>. The server's Image row exists for the pool's list and labels, not for identity.
• Dropping media that a tracing symbol in the document already shows reuses that symbol.
• Nothing can be created or dropped inside a tracing symbol (creation/target, drop-destination-at, nest/move-refusal), and one cannot be opened in a tab.
• Schema 6. Every project is marked 6. One that still carries :trace on a node is refused when opened, with what to do about it, rather than all old projects being refused.

No backward compatibility (see lane-model.md, "Goal and compatibility policy").

What is wrong today

Tracing is not a thing in the document. It is three mechanisms that each know about faces:

• :trace {:frames :origin} on a face's :head. It decides which measured frame the head reads (symbol/base-channel-frame → trace/held-frame), and the underlay also reads it to decide which photo to show (trace/photo-frame).
• ui/underlay, a painter that walks trace/shown → trace/faces (a separate instance walk), looks up each face's subject → analysis → footage, asks the resolver where [...path :head] went, and builds the photo's matrix by hand: world(head) · M(p)⁻¹ · 1/imageH (trace/photo-matrix).
• [:ui :trace {:faces #{} :opacity}], a per-face switch in editor state, plus the special "opening a face shows its footage, opening a take does not" rule (trace/showing-for).

What you cannot do: place footage or a still as a reference where you like, move it, scale it, turn it, trim it, hold it, put it in a lane, or trace something that is not a tracked face. The photo is not selectable and has no row. The one case that works (a face over its own footage) runs on code that nothing else uses.

The model in one paragraph

A tracing symbol is a symbol with :type :trace. It has no nodes. It names media (a footage range or a still image) and has that media's frame count, fps and pixel size. It is placed by an ordinary instance, so lanes, spans, trim, split, move, playback (:in :speed :end), transform, nesting, selection, picking and gestures all work on it with no new code. It is evaluated to a single :trace op, which never reaches the raster or an export. A face's footage is not a special case. It is one of these instances, placed inside the face as a child of :head and carrying the measured registration transform. The trace keys become holds on that instance's time. The head's "origin" becomes the head saying which node's frames it follows.

This follows the precedent of :type :palette symbols (no nodes, they name an asset, and they are placed by instances in a lane), so the uniformity rule holds (see the model-uniformity memory). Nothing new holds nodes, and no special instance kind is added.

Data model

The tracing symbol

:footage-8625                                 ; a symbol id like any other
{:id :footage-8625 :name "8625.mov"
 :type :trace
 :media {:footage #uuid "f8ca…" :range [12 241]}   ; or {:image "sha256:…"}
 :frames 229          ; the range's length; 1 for a still
 :fps 30              ; the footage's own rate, so cadence handles 30→12 for free
 :width 1440 :height 1920   ; pixel size = the symbol's own stage
 :audio {:footage #uuid "f8ca…"}   ; optional, the existing "a symbol says what it sounds like"
 :nodes {}}

• :media is the one new symbol key. Add it to symbol/symbol-keys and to the symbol leaf's select-keys in leaf/leaves.
• The symbol's local space is pixels of the media: [0 w) × [0 h). clip/center of a node-less symbol already returns its stage middle, which here is [w/2 h/2]. So place-symbol puts the anchor at the image centre with no new code.
• symbol/problems: :type :trace needs :media with exactly one of :footage/:image, needs (empty? nodes), and for footage needs (= frames (- end start)).
• One tracing symbol per footage range, shared. Two faces from one take place the same symbol, and so does a hand-placed reference. Like any symbol it is reused by reference, and bring/symbols copies it like any other.

Placing it

It is an ordinary :kind :instance with :source {:symbol :footage-8625}:

• Start and end are :span (in its own frames) and :time :at. You get these from the existing trim, split, move and roll.
• Which frame shows comes from :playback {:in :speed :end}. Footage plays with speed 1, a frozen frame has speed 0, and a still is 1 frame with :end :hold.
• Placement in space is [:xform …], the same as every node: gestures, inspector and keys.
• In a lane, or on its own: it is a parent-less spanned node, so a :display :lane symbol draws it as a block like any cel. It can also sit as a free node or under a group. To have a reference with its own tab, wrap it in an ordinary symbol.

Default on creation (in the drop event, not in place-symbol): scale so the image's height fits the stage height, centred on the drop point. This is a creation default like "use center anchor when dropping", and nothing updates it afterwards.

Holds: the one new time feature

:time {:holds [0 12 30]} floors a node's local frame to the last hold at or before it. Before the first hold, the first hold applies. This is :expose generalized from a regular grid to authored frames. It is applied in node/local-frame at the same point as :expose, and is inherited in the same way. It is in the node's own frames. On a tracing instance with :in 0 :speed 1, own frames are source frames, so the hold list is the set of traced frames.

It is general on purpose. Holding a playing symbol on chosen drawings is the same feature. It costs about three lines in local-frame, one problems clause (sorted, distinct, finite), and the timeline drawing hold frames as marks on the row.

symbol/frame-map refuses floors (:expose > 1). It must also refuse a non-empty :holds, for the same reason.

The face: "a child symbol that represents the trace"

The face symbol after a freeze:

:place    group     authored source→stage mapping (image heights → stage px)
  :head   group     measured M(p)                      :reads   — see below
    :mouth … parts  generated, every frame
    :plate instance  of :footage-8625                   ← the trace
                     measured channels: fit(p) = M(p)⁻¹ · S(1/imageH)
                     :time {:holds [0 12 30]}           ← the trace keys

Registration comes from the parent

The plate is a child of :head. Its own measured channels are the stabilizing fit at frame p: the inverse of the head's measured transform, with pixels → image heights folded into the scale (it stays a similarity, so it decomposes into pos/rot/scale). Freeze already computes the fit; the head's measured channels are its inverse (freeze/invert). The plate gets a second dense block, which is three small channels.

Because holds are a time floor, the plate's channels and the frame its content shows are read at the same held frame q. Its world transform is:

world(plate) = place · M(p_head) · M(q)⁻¹ · S(1/H)

That is exactly trace/photo-matrix, but now it falls out of the ordinary walk. It is registered to whatever the head is doing, by construction:

head reads (p_head)	plate shows (q)	result
f (continuous)	f (no holds)	footage where filmed
f (continuous)	held key	held photo rides the moving head (today's behaviour)
held key (same as plate)	held key	M(q)·M(q)⁻¹ = I: photo sits where filmed
0 (start)	f or held	stabilized footage under a still head

If a frame has no measurement, the plate's dense channel has nothing there, so xform-at gives nil, so the plate is not placed and no photo shows. That is what happens today too.

Trace keys versus origin: who owns what

The two decisions are separate and stay separate:

• Trace keys are which footage frames get drawn over (the "plate drawings" in frame-selection.md). They belong to the plate, as :time :holds. A hand-placed tracing layer with holds is the same thing: the face's plate is an ordinary tracing placement and nothing more.
• Origin is how the head moves between kept frames. frame-selection.md already says :origin "is not a tracing setting" but a performance one, so it belongs to the head:

:head {…}                            ; continuous — reads its own frame
:head {… :reads {:holds [0]}}        ; start
:head {… :reads {:holds-of :plate}}  ; at keys — reads its measured channels at
                                     ; the frames :plate's holds select

{:holds [...]} is also what a face with no footage uses for "at keys", since it has no plate to follow.

:reads changes only the head's own channel reads, not its children's frames. The parts must keep running every frame, which is why this cannot be a :time hold on the head. It is today's traces branch of base-channel-frame with the hold list read from the named node. A node reference has precedent (:stencil, :pose-group). It points from the follower to the thing followed, so there is still one stored list of frames and nothing to keep in sync. Validate it in symbol/problems the way :stencil is validated: the target exists in the symbol, and it is not the head itself or an ancestor of the head.

Rejected alternatives, so they are not re-proposed:

• Keys on the head, and the plate reads them. This is today's direction. It makes the plate special: a free tracing layer could not have keys that a face's plate also understands.
• Hold :time on :head. Exposure inherits strictly, so the mouth and eyes would freeze along with the head.
• A wrapper "registered footage" symbol holding the fit. It is correct but adds a symbol per face. The time-floor holds already put the fit read and the content read on the same frame, so the wrapper buys nothing.
• Plate as a sibling of :head at identity. It is only registered when the head and the photo read the same frame, so it breaks the continuous-plus-holds and start rows above.

Evaluation: an op that is never rendered

• clip/resolver, in the instance branch: when the source symbol has :type :trace, it does not recurse. If (:tracing? opts) is set, it emits one op:

  {:kind :trace :node [id] :m <copy of world> :media … :frame shown-frame :size [w h]}
  

  The frame comes from the same placed-frame path as any instance, so playback, holds and the fps cadence apply. Do not build a child resolver for a trace symbol.

• transform-op gets a :trace case that composes the matrix. Row paths, solo filtering and nesting at any depth then work for free.

• The output guarantee is structural. :tracing? defaults to false. Only the stage's ::render/resolver passes true. Export, clip/center (a big photo must not pull a symbol's pivot), thumbnails and the bench never ask for trace ops. raster/draw-ops! keeps throwing on unknown kinds, so a leak fails loudly.

• ui/player sends picture ops to the raster and :trace ops to the painter.

• ui/underlay becomes ui/tracing. It paints :trace ops in draw order with drawImage at op.m and the global opacity. The media URL is the footage manifest's urls[range-start + frame], or the image blob URL. The three steadiness fixes stay: LRU cache, hold the last still per op :node, and read ahead while playing. Everything that walked faces is deleted.

• pick: a :trace op is hit when the point, mapped through m⁻¹, falls in [0 w) × [0 h). Picture ops are tested first and trace ops only if nothing drawn is under the pointer, so a full-frame photo does not steal every click. When the global switch is off, traces are neither painted nor picked.

• Gestures: no change. The face's plate is measured, so gesture/refusal already says "place the instance it is in". A hand-placed layer is authored and moves, turns and scales like anything else.

On and off

All of it is EDITOR STATE, as ed88c5e decided for the per-face switch: showing a reference is a way of looking at the stage, so it is not an undo step, does not travel to collaborators, and cannot reach an export.

[:ui :tracing {:on? true :opacity 0.5 :hidden #{[sid node-id] …}}]

• One layer is an entry in :hidden, keyed by the symbol the tracing instance is in and its node id. The symbol id is needed because every face's plate is called :plate. Keyed this way, hiding a face's plate hides it in every placement of that face, which is what the per-face switch did. Shown is the default, so opening a face shows its footage with no setup.
• How it's applied: clip/resolver already knows which symbol and node a trace op comes from, so the op carries :layer [sid id]. The painter and pick skip ops whose layer is hidden. The resolver does not change when you toggle a layer, so nothing is rebuilt.
• Global: :on? and :opacity, a toggle plus an opacity slider in ui/palette/bar next to the palette controls. When it is off, traces are neither painted nor picked.
• Switching one layer on also switches the global setting on. This applies from the tracing instance's inspector, from its timeline row, and from the face/roto section. A layer you just enabled must not stay invisible behind a switch you forgot. Turning one layer off never touches the global switch. It is one event (::ui/show-trace layer on?) that updates :hidden and, when turning on, sets :on? true in the same handler, so the three places cannot behave differently.
• [:vis] still works on a tracing instance like on any node: key it to show a reference only over part of the shot. It is not the on/off switch.
• Deleted: [:ui :trace :faces], ::trace-face, ::trace-faces, trace/showing-for, traceable-faces, shown, faces, and the "a take shows nothing by default" rule.

UI

• Timeline. A tracing instance is an ordinary row or clip block, styled to read as reference-only (hatched block, an eye icon in place of the colour chip). Hold frames are marks on the row. The row's eye button dispatches ::ui/show-trace, which replaces the face row's tl-trace button.
• Inspector, tracing instance. Show the media (footage name and range, or image), an on/off control (which goes through ::ui/show-trace), and the hold list ("hold here" / "remove hold" plus seek buttons, which is trace-keys re-aimed at :time :holds). Playback, transform and span use the existing sections.
• Inspector, face/roto section. The same hold controls, aimed at the face's :plate. Origin buttons write :head :reads. The section is found by "this face has a :plate", not by trace/traceable?.
• Palette bar. The global toggle and opacity slider.
• Making one.
  • Footage: the convert dialog gets a choice between "animate faces" (today's flow) and "tracing layer" (no detection). The tracing-layer choice makes the :type :trace symbol for the chosen range and places it where the video was dropped.
  • Footage can also be dragged from the pool with a modifier or as a second drag kind.
  • A still image: drop it on the pool and it uploads; drop it on the stage or timeline and it is placed with :end :hold and a span of the host's remaining frames.

Freeze, bring, regenerate

• freeze/subject-part emits the :plate instance under :head, with fit measured channels and :time {:holds []}, and emits one :type :trace symbol for the analysed footage range. bring/take already copies every symbol the take reaches and rewrites :source :symbol references, so the tracing symbol comes along. It gets the footage's :audio too, so a dropped tracing layer can bring its sound through the existing link.
• head-mode writes :reads on the head and :holds on the plate, in place of :trace. The frame-selection.md plate proposal materializes into the plate's :time :holds in place of :trace :frames.
• Regenerate replaces both measured blocks (head and plate) and keeps :holds and :reads. Check that regenerate-head's measured/authored comparison handles a second measured node.

Server

• Image endpoints mirroring Sound: POST /api/images stores a Blob and returns {id, width, height, url}, and GET /api/images lists them. The migration is a model only. Bump schema_version and refuse older documents clearly. Do not convert them.

Deleted outright

trace/of, prepare, held-frame, problems, toggle-frame, photo-frame, measured-local, photo-matrix, traceable?, faces, traceable-faces, showing-for, shown, opacity-default. domain/trace.cljs probably goes away entirely; if anything is left, it is a hold helper that belongs in node.

Also deleted: symbol/prepared-traces and the traces arm of base-channel-frame, which becomes the :reads lookup. :trace on nodes (and symbol/problems reports it as removed). ::project/set-trace. ::render/underlay and ::render/tracing become one ::render/tracing that returns [:ui :tracing]. The face lookup in params/view.

Expected effect on code size: net negative. One time-floor clause, one op kind, one resolver branch, one pick case and a :reads lookup replace the face walk, the hand-built photo matrix, per-face showing state and the underlay's face bookkeeping. Measure the change and report the number honestly (see the cljs-style memory).

Tests

Domain (node).

• :holds in local-frame, with eval-frame and resolver agreeing forward, backward and in random order.
• A trace op appears only with :tracing?, and export/run! output contains no :trace op.
• transform-op composes the matrix through two nesting levels.
• clip/center ignores traces, and is [w/2 h/2] for a tracing symbol.
• pick returns picture ops before traces and inverse-maps through a rotated layer.
• symbol/problems covers :type :trace, :reads targets and cycles, and :holds shape.
• Registration: port trace_test's photo-matrix assertions onto the walk. For every row of the registration table, the plate's world transform equals the expected matrix. At a held key with :reads {:holds-of :plate}, it equals world(:place) · S(1/H).
• Freeze emits the plate and the tracing symbol. Regeneration keeps the holds.
• The ::ui/show-trace event sets the global switch on when turning a layer on and leaves it alone when turning one off.

Browser (CDP, see the arthur-verify-dont-guess memory):

• Drop a still, then drag, turn and scale it.
• Toggle a layer on with the global switch off, and confirm both are on and the photo paints.
• Export a frame and check it has no photo pixels.
• Open a face: the plate is registered at a hold with origin "at keys", and stabilized with origin "start".

Order of work, each step green

1. :time :holds in node/local-frame, problems, frame-map, and timeline marks.
2. :type :trace symbol, :media, the :trace op behind :tracing?, transform-op, the player split, ui/tracing painter and pick. Footage media only, placed by hand from a REPL or test document.
3. The face: freeze and bring emit the plate and tracing symbol, :reads replaces :trace, delete the old trace and underlay paths, and re-aim the inspector's face section.
4. On/off: the :hidden set and :layer on trace ops, the row eye, the ::ui/show-trace rule, the global toggle and opacity in the palette bar, and delete [:ui :trace :faces].
5. Creation: "tracing layer" in the convert dialog, pool drag, then the image endpoint, pool images and image drops.
6. Docs: animation-model.md "A photographic underlay is not an op" becomes "a trace is an op that never reaches the raster". Update frame-selection.md (where plate keys live) and lane-model.md (tracing clips in lanes).

Open questions

1. "A take shows its faces' footage." Dropping the old "only in a face's own tab" default means opening a take shows every face's plate if the global switch is on. The recommendation is to accept that, since the switch is one click.
2. Lane-model audio rule. lane-problems requires visual cels. A tracing cel is visual for this purpose, but say so explicitly when a lane may hold both tracing and drawing cels.