roto

Video → take file builder for the Animator Pro rotoscope/puppet pipeline described in ../docs/roto-puppet.md.

This is the analysis and tuning half. It stabilises a face out of a clip, reduces the lip contour to a handful of vertices, selects sparse keys on motion extremes, and previews the result as flat indexed fills — so the look and timing can be judged in seconds rather than through a minutes-long Animator Pro render. It emits a .take file; nothing here touches Animator Pro.

All policy lives here. The take arrives at the renderer with its keys already chosen.

Run

python3 -m http.server 8777        # from this directory
# open http://127.0.0.1:8777

Synthetic take needs no video and exercises everything below detection.

For real footage:

./extract.sh /path/to/clip.mp4 24  # -> frames/0001.png …

then Load frames. MediaPipe's wasm is fetched from jsdelivr on first use; face_landmarker.task is local.

Frames are pre-extracted rather than decoded in the page because browser video seeking is approximate and requestVideoFrameCallback only delivers frames at playback speed — neither gives a deterministic per-frame pass.

Shooting for it

Near-frontal, good light, consistent scale, head reasonably still. Hold a neutral closed mouth for a second at the top of the take: that frame is picked automatically as the neutral and drives calibration and the placeholder plate.

Out-of-plane head rotation cannot be stabilised away by a 2D similarity transform — the residual readout rises when it happens. The pipeline's answer is hand-drawn head plates, which this tool does not yet do.

Knobs

Knob	What it does
vertices	Lip vertex budget. The reduction past what the footage supports is the style.
min hold	Minimum frames between keys.
change gate	Mean vertex movement required before a new key is accepted.
vel smoothing	Window on the velocity signal used to find extremes.
anchor smoothing	Window on the four similarity parameters. Smooths the transform, never the contour.
exposure	Grid that key frames snap onto.
closed-mouth cut	Aperture below which the mouth interior is emitted as hidden.

Keys go on velocity minima, not distance thresholds: a threshold fires at the frame it was crossed — partway through a transition — so poses land mushy and late. The timeline shows the velocity curve, candidate minima, accepted keys (f) and their pre-snap extremes (src); a large f/src gap means min-hold and exposure are fighting.

Tests

chromium --headless --virtual-time-budget=8000 --dump-dom \
  http://127.0.0.1:8777/selftest.html | grep -oE '(PASS|FAIL) [0-9/]+'

Or open selftest.html. 29 assertions over the stages below detection.

The ring-simplicity check is the load-bearing one. Because hold parts cut between poses instead of interpolating, a ring whose vertex order is wrong self-intersects and renders as blocks meeting at corners — and it is invisible at odd verts/2 and obvious at even, so it needs an assertion rather than an eyeball.

Not done yet

Eyes and irises; hand-drawn head plates and per-plate mouth slots; real performer→character calibration (currently identity, fitting the face oval to the canvas); the override layer; anything on the Animator Pro side. The placeholder plate is a frozen face-oval polygon — it exists so the mouth has a face to read against, not to look good.