Your Name
7a6bdea157
Analysis half of the pipeline in docs/roto-puppet.md. Stabilises a face out of a clip via a similarity fit on rigid landmarks, reduces the lip contour to a fixed vertex budget, selects sparse keys on velocity minima, and previews the result as flat indexed fills so timing can be judged without an Animator Pro render. - landmarks.js ordered lip/oval rings; slot position is vertex identity - mathutil.js closed-form 2D similarity, Procrustes mean, transform smoothing - pipeline.js stabilise -> subsample -> key-select - raster.js indexed scanline fill, no antialiasing - take.js take-file writer - selftest.js 29 assertions, incl. ring simplicity at every vertex budget Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
86 lines
3.4 KiB
Markdown
86 lines
3.4 KiB
Markdown
# roto
|
|
|
|
Video → **take file** builder for the Animator Pro rotoscope/puppet pipeline
|
|
described in `../docs/roto-puppet.md`.
|
|
|
|
This is the analysis and tuning half. It stabilises a face out of a clip, reduces
|
|
the lip contour to a handful of vertices, selects sparse keys on motion extremes,
|
|
and previews the result as flat indexed fills — so the *look and timing* can be
|
|
judged in seconds rather than through a minutes-long Animator Pro render. It
|
|
emits a `.take` file; nothing here touches Animator Pro.
|
|
|
|
All policy lives here. The take arrives at the renderer with its keys already
|
|
chosen.
|
|
|
|
## Run
|
|
|
|
```sh
|
|
python3 -m http.server 8777 # from this directory
|
|
# open http://127.0.0.1:8777
|
|
```
|
|
|
|
**Synthetic take** needs no video and exercises everything below detection.
|
|
|
|
For real footage:
|
|
|
|
```sh
|
|
./extract.sh /path/to/clip.mp4 24 # -> frames/0001.png …
|
|
```
|
|
|
|
then **Load frames**. MediaPipe's wasm is fetched from jsdelivr on first use;
|
|
`face_landmarker.task` is local.
|
|
|
|
Frames are pre-extracted rather than decoded in the page because browser video
|
|
seeking is approximate and `requestVideoFrameCallback` only delivers frames at
|
|
playback speed — neither gives a deterministic per-frame pass.
|
|
|
|
## Shooting for it
|
|
|
|
Near-frontal, good light, consistent scale, head reasonably still. Hold a neutral
|
|
closed mouth for a second at the top of the take: that frame is picked
|
|
automatically as the neutral and drives calibration and the placeholder plate.
|
|
|
|
Out-of-plane head rotation cannot be stabilised away by a 2D similarity
|
|
transform — the `residual` readout rises when it happens. The pipeline's answer
|
|
is hand-drawn head plates, which this tool does not yet do.
|
|
|
|
## Knobs
|
|
|
|
| Knob | What it does |
|
|
| --- | --- |
|
|
| vertices | Lip vertex budget. The reduction past what the footage supports *is* the style. |
|
|
| min hold | Minimum frames between keys. |
|
|
| change gate | Mean vertex movement required before a new key is accepted. |
|
|
| vel smoothing | Window on the velocity signal used to find extremes. |
|
|
| anchor smoothing | Window on the four similarity parameters. Smooths the *transform*, never the contour. |
|
|
| exposure | Grid that key frames snap onto. |
|
|
| closed-mouth cut | Aperture below which the mouth interior is emitted as `hidden`. |
|
|
|
|
Keys go on **velocity minima**, not distance thresholds: a threshold fires at the
|
|
frame it was crossed — partway through a transition — so poses land mushy and
|
|
late. The timeline shows the velocity curve, candidate minima, accepted keys
|
|
(`f`) and their pre-snap extremes (`src`); a large `f`/`src` gap means min-hold
|
|
and exposure are fighting.
|
|
|
|
## Tests
|
|
|
|
```sh
|
|
chromium --headless --virtual-time-budget=8000 --dump-dom \
|
|
http://127.0.0.1:8777/selftest.html | grep -oE '(PASS|FAIL) [0-9/]+'
|
|
```
|
|
|
|
Or open `selftest.html`. 29 assertions over the stages below detection.
|
|
|
|
The ring-simplicity check is the load-bearing one. Because `hold` parts *cut*
|
|
between poses instead of interpolating, a ring whose vertex order is wrong
|
|
self-intersects and renders as blocks meeting at corners — and it is invisible at
|
|
odd `verts/2` and obvious at even, so it needs an assertion rather than an
|
|
eyeball.
|
|
|
|
## Not done yet
|
|
|
|
Eyes and irises; hand-drawn head plates and per-plate mouth slots; real
|
|
performer→character calibration (currently identity, fitting the face oval to the
|
|
canvas); the override layer; anything on the Animator Pro side. The placeholder
|
|
plate is a frozen face-oval polygon — it exists so the mouth has a face to read
|
|
against, not to look good.
|