arthur/README.md

roto

Video → take file builder for the Animator Pro rotoscope/puppet pipeline described in ../docs/roto-puppet.md.

This is the analysis and tuning half. It stabilises a face out of a clip, reduces the lip contour to a handful of vertices, selects sparse keys on motion extremes, and previews the result as flat indexed fills — so the look and timing can be judged in seconds rather than through a minutes-long Animator Pro render. It emits a .take file; nothing here touches Animator Pro.

All policy lives here. The take arrives at the renderer with its keys already chosen.

Run

python3 -m http.server 8777        # from this directory
# open http://127.0.0.1:8777

Synthetic take needs no video and exercises everything below detection.

For real footage:

./extract.sh /path/to/clip.mov 12   # -> frames/*.png, audio.wav, manifest.json

then Load frames. MediaPipe's wasm is fetched from jsdelivr on first use; face_landmarker.task is local.

Frames are pre-extracted rather than decoded in the page because browser video seeking is approximate and requestVideoFrameCallback only delivers frames at playback speed — neither gives a deterministic per-frame pass.

manifest.json records the true extraction rate. The page reads it rather than assuming, because a guessed fps desynchronises audio from picture — and sync is the one thing this view exists to show.

Audio is the playback clock: frame = floor(audio.currentTime * fps). A slow render loop therefore drops frames instead of drifting, and ½x / ¼x work by setting playbackRate with the picture following for free.

Shooting for it

Near-frontal, good light, consistent scale, head reasonably still. Hold a neutral closed mouth for a second at the top of the take: that frame is picked automatically as the neutral and drives calibration and the placeholder plate.

Out-of-plane head rotation cannot be stabilised away by a 2D similarity transform — the residual readout rises when it happens. The pipeline's answer is hand-drawn head plates, which this tool does not yet do.

Knobs

Knob	What it does
vertices	Lip vertex budget. The reduction past what the footage supports is the style.
contour avg ±f	Radius in frames. 0 off, 1 = ±1. Removes per-frame landmark jitter.
anchor avg ±f	Radius on the four similarity parameters. Smooths the transform.
closed-mouth cut	Aperture below which the mouth interior is emitted as hidden.
suggest tolerance	Max head movement before a new plate drawing is required. Affects Suggest only.

Teeth

MediaPipe has no landmarks inside the lips: the inner ring bounds the cavity and everything within it is just pixels. So teeth come from the image — but tracing the bright blob would produce a new contour every frame with no vertex correspondence, which is precisely the boil the design exists to avoid.

So the extraction yields a scalar, not a shape. The teeth polygon is the inner lip ring clipped to a horizontal line, and only that line's height is measured (Otsu threshold within the cavity, scanned from the top). The silhouette is therefore always the mouth's own shape — stable by construction — and the only per-frame variable is one number, which smooths trivially. It is also how the shape gets drawn by hand: a band bounded by the lip.

Presence uses hysteresis plus a minimum dwell, the same treatment plate selection gets, because a teeth block that blinks on and off for single frames is worse than one that is simply absent.

Tongue would work the same way — a shape filling the lower cavity, gated on a redness rather than a brightness statistic. Not implemented, because it is not visible in the test footage: the cavity reads as dark with a bright upper-teeth band and nothing else.

The plate is reference, not art

The plate layer has several representations because its job changes. Cycle with B:

Mode	For
photo dim / photo	Drawing over. The source frame, stabilised.
posterized	The footage quantised into the ramp — a look test asked of the source rather than of a drawing.
oval	A flat stand-in, to judge the mouth against something.
oval + photo	Checking the stand-in against the real head.
none	Mouth alone.

Photo modes are registered: the frame is mapped into raster space through the same transform chain the contours go through, so the head sits still and a drawing traced from it is already aligned to the mouth. An unregistered underlay would be decorative.

MediaPipe's face oval is the face boundary — it cuts at the hairline and excludes hair, ears, jaw underside and neck — so as a head silhouette it is an egg by construction, and no landmark precision fixes that. Hence the photo.

Save frame 4x writes the current registered composite as a 1280×800 PNG to draw on.

Two kinds of sparseness

Sparseness has two unrelated causes, and conflating them was the original design error here. Aesthetic sparseness is set by the extraction rate — pick 12fps and you have already chosen your timing. Labour sparseness is a human drawing each one, and it binds only on the plate.

So the mouth keeps every frame: it is traced, and therefore free. In limited animation lip sync is routinely the densest element, on 1s, while heads hold on 2s and 3s.

The frame strip is the editing surface for the other half: which frames need their own plate drawing. Everything starts kept; delete what you don't want. Suggest runs error-tolerance decimation over head pose as a starting point, then you hand-correct.

contour avg is a deliberate, bounded exception to "never smooth the contour" in ../docs/roto-puppet.md. That rule held while keys were sparse, because sampling at velocity minima rejected detector noise for free. With a key on every frame it does not, so a radius shorter than the shortest articulation worth keeping is justified — at 12fps, articulation spans 3–6 frames and detector noise is per-frame, so ±1 separates them and ±3 starts eating speech.

Tests

chromium --headless --virtual-time-budget=8000 --dump-dom \
  http://127.0.0.1:8777/selftest.html | grep -oE '(PASS|FAIL) [0-9/]+'

Or open selftest.html. 29 assertions over the stages below detection.

The ring-simplicity check is the load-bearing one. Because hold parts cut between poses instead of interpolating, a ring whose vertex order is wrong self-intersects and renders as blocks meeting at corners — and it is invisible at odd verts/2 and obvious at even, so it needs an assertion rather than an eyeball.

Not done yet

Eyes and irises; hand-drawn head plates and per-plate mouth slots (the strip decides which frames need one, but you cannot yet supply the drawing); real performer→character calibration (currently identity, fitting the face oval to the canvas); the override layer; anything on the Animator Pro side. The plate is a face-oval polygon per kept frame — it exists so the mouth has a face to read against, not to look good.