Registered photo underlay as the plate reference

The generated face oval was never going to be good enough to draw from:
MediaPipe's face oval is the FACE boundary, cut at the hairline and excluding
hair, ears, jaw and neck, so it is an egg by construction. Segmentation would
give a real head outline but costs a 16MB model and per-frame inference for a
shape that gets replaced by a drawing anyway.

So the plate layer becomes switchable, and the useful modes are photographic:
the source frame mapped into raster space through the same transform chain the
contours go through. Registration is the whole point - the head sits still and
a drawing traced from the underlay is already aligned to the mouth. An
unregistered underlay would be decoration.

- underlay.js: pixel->raster affine (a general affine, since MediaPipe
  normalises x by width and y by height), registered draw, palette posterise
- plate modes: photo / photo dim / posterized / oval / oval+photo / none, B cycles
- worksheet cells are registered composites rather than raw crops
- Save frame 4x writes a 1280x800 PNG to draw on
- selftest: FACE_OVAL simplicity, which was never asserted; a wrong ordering
  there reads as a lumpy plate rather than an obvious bowtie

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Your Name 2026-09-24 14:57:59 -04:00
parent 5ac919b9d6
commit de90bfd907
6 changed files with 191 additions and 20 deletions

63
js/underlay.js Normal file
View file

@ -0,0 +1,63 @@
// Registered photo underlay: the source frame mapped into raster space through
// the same transform chain the vector shapes go through, so a drawing made over
// it lands on the shapes.
//
// Without registration an underlay is decorative. With it, the photo is
// stabilised exactly as the contours are - the head sits still - and tracing over
// it produces plate art already aligned to the mouth.
import { applySim } from './mathutil.js';
// Compose pixel-space -> raster-space into one affine.
//
// MediaPipe normalises x by width and y by height, so normalised space is a
// stretched pixel space and the composition is a general affine rather than a
// similarity. Three mapped points determine it exactly.
export function frameAffine(tf, xform, imgW, imgH) {
const map = (px, py) => xform(applySim(tf, { x: px / imgW, y: py / imgH }));
const P0 = map(0, 0), P1 = map(imgW, 0), P2 = map(0, imgH);
return {
a: (P1.x - P0.x) / imgW, b: (P1.y - P0.y) / imgW,
c: (P2.x - P0.x) / imgH, d: (P2.y - P0.y) / imgH,
e: P0.x, f: P0.y,
};
}
// Draw the registered source frame into a raster-sized 2D context.
export function drawRegistered(ctx, img, tf, xform, zoom, alpha = 1) {
const m = frameAffine(tf, xform, img.naturalWidth, img.naturalHeight);
ctx.save();
ctx.globalAlpha = alpha;
ctx.setTransform(m.a * zoom, m.b * zoom, m.c * zoom, m.d * zoom, m.e * zoom, m.f * zoom);
ctx.imageSmoothingEnabled = true;
ctx.drawImage(img, 0, 0);
ctx.restore();
ctx.setTransform(1, 0, 0, 1, 0, 0);
}
// Quantise a registered frame straight into palette indices.
//
// Doubles as a look test: it shows what the footage becomes in the chosen ramp,
// with no dithering and no antialiasing, which is the question "will these tones
// read" asked directly of the source rather than of a drawing.
export function posterizeInto(raster, img, tf, xform, paletteHex) {
const tmp = document.createElement('canvas');
tmp.width = raster.w; tmp.height = raster.h;
const g = tmp.getContext('2d', { willReadFrequently: true });
g.fillStyle = '#000'; g.fillRect(0, 0, raster.w, raster.h);
drawRegistered(g, img, tf, xform, 1, 1);
const px = g.getImageData(0, 0, raster.w, raster.h).data;
const pal = paletteHex.map((h) => {
const s = h.replace('#', '');
return [parseInt(s.slice(0, 2), 16), parseInt(s.slice(2, 4), 16), parseInt(s.slice(4, 6), 16)];
});
for (let i = 0, n = raster.w * raster.h; i < n; i++) {
const r = px[i * 4], g2 = px[i * 4 + 1], b = px[i * 4 + 2];
let best = 0, bd = Infinity;
for (let k = 0; k < pal.length; k++) {
const d = (pal[k][0] - r) ** 2 + (pal[k][1] - g2) ** 2 + (pal[k][2] - b) ** 2;
if (d < bd) { bd = d; best = k; }
}
raster.buf[i] = best;
}
}