Eyes: lids, blinking, line of sight
Three parts per eye, stacked the way the mouth is - dark lash ring, sclera inside it, iris inside that, square pupil in the iris. A blink then costs nothing: when the lid shuts the traced ring goes flat and the lash line collapses to a lens, which is a closed eye, drawn correctly, for free. Lids are a FEATURE, rotoscoped like the mouth: head-local, a key on every frame, the same contour avg knob. The iris is a PRIMITIVE - a disc at a quantised position - and that is where the stylisation lives. Line of sight. Gaze is the iris centre relative to the midpoint of the eye's two corners, in units of corner distance. Both corners are in RIGID, so the origin and the scale are immune to the performance being measured; against the lid ring's centroid instead, every blink would drag the origin down and fake a glance at the floor on exactly the frames where the eye is most visible. Both eyes share one gaze - at this size the difference between the two measurements is noise, not vergence, and independent per-eye noise reads as wall-eyed immediately. Openness stays per-eye so a wink survives. Gaze is then quantised to a pixel grid with a dwell, which is not a stylisation imposed on the truth: real eyes move in saccades, and the smooth drift left in the measurement is tracker noise plus head-compensation error. Snapping to a grid removes the noise and recovers the saccade in one operation. The iris is placed in the frame of the already-smoothed, already-subsampled lid ring - slots 0 and 8 of a 16-slot ring are the corners, and subsampling to any even budget keeps them at 0 and n/2 - so it cannot drift relative to its own eye. Size is authored from the take mean, never remeasured per frame: a radius that breathes by a fraction of a pixel flickers a pixel on and off around the whole silhouette. iris anchor toggles steady/free/locked, because how much the eye wanders turns out to be an aesthetic choice and not only a correctness one. Blinking gets hysteresis and a dwell like the teeth, plus one knob they do not have: blink hold. A blink is one frame at 12fps and a single frame of closed eye reads as a dropped frame, so once the eye shuts it stays shut long enough to be legible. Detection accuracy is not the problem; legibility is. The pupil is a square because at three pixels a circle is a plus sign with the corners gnawed off, and it changes shape as it moves. Drawn from a rounded centre shared with the iris so it is exactly its nominal size on every frame. Iris/pupil clip by colour key against the indexed buffer, the way Animator Pro would: the lid crops the iris at extreme gaze for free, so nothing has to clamp the gaze, which would flatten the performance at the extremes that carry it. Which iris block belongs to which eye is RESOLVED from geometry, not declared. A swap looks almost right - each eye still has a disc roughly where it belongs - so it survives an eyeball and then reads as a subtly wall-eyed character forever. Voted across every frame; the test feeds a deliberately swapped track. Also: exposure. Aesthetic sparseness was set by the extraction rate, which made the timing a property of a directory of PNGs - auditioning 12 against 24 meant re-ripping and re-detecting the whole clip. It is now a render-time grid, on 1s/2s/3s/4s, so the dense track keeps everything and the audio clock is untouched. The take format already carried an exposure field; it was never driven. Everything rides the same grid, because a head cutting on the odd frames while the mouth cuts on the even ones reads as two performances laid over each other. 41 -> 91 assertions. The load-bearing new ones: the iris pairing follows a swapped track, a blink does not fake a change of gaze, a stencilled disc cannot spill past its clip, a 3px pupil is 3x3 at every sub-pixel centre, and exposure never reads a pose from the future. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Your Name
parent
c2603102da
commit
e497e9ac4d
11 changed files with 1342 additions and 42 deletions
428
js/app.js
428
js/app.js
|
|
@ -1,10 +1,12 @@
|
|||
import { FaceLandmarker, FilesetResolver } from 'https://cdn.jsdelivr.net/npm/@mediapipe/tasks-vision@1.0.1/vision_bundle.mjs';
|
||||
import { LIPS_OUTER, LIPS_INNER, FACE_OVAL } from './landmarks.js';
|
||||
import { stabilize, toRasterRing, smoothContours, suggestPlateFrames, heldFrame, shiftIndex } from './pipeline.js';
|
||||
import { LIPS_OUTER, LIPS_INNER, FACE_OVAL,
|
||||
EYE_R_RING, EYE_L_RING, IRIS_A, IRIS_B } from './landmarks.js';
|
||||
import { stabilize, toRasterRing, smoothContours, suggestPlateFrames, heldFrame, shiftIndex,
|
||||
exposeIndex, eyeSignals, gazeOrigin, quantizeGaze, resolveBlink } from './pipeline.js';
|
||||
import { IndexedRaster } from './raster.js';
|
||||
import { drawRegistered, posterizeInto } from './underlay.js';
|
||||
import { extractTeeth } from './interior.js';
|
||||
import { applySim } from './mathutil.js';
|
||||
import { applySim, offsetRing } from './mathutil.js';
|
||||
import { writeTake } from './take.js';
|
||||
import { synthDense } from './synth.js';
|
||||
|
||||
|
|
@ -16,8 +18,21 @@ const PALETTE = [
|
|||
{ name: 'skin_dark', hex: '#7a4f3a' },
|
||||
{ name: 'mouth_dark', hex: '#24161a' },
|
||||
{ name: 'teeth', hex: '#d9cfc2' },
|
||||
// Sclera is not white, and that is authored, not measured. A true white at
|
||||
// 320x200 next to a warm skin ramp reads as a hole punched in the face; the
|
||||
// eye sits in a socket, in shadow, so it is a dimmer and cooler tone than the
|
||||
// teeth, which catch the light. The iris is one dark tone: at this size an
|
||||
// iris is about five pixels across and a pupil inside it would be one, so the
|
||||
// iris IS the pupil. Resolving it further would be drawing detail the format
|
||||
// cannot hold.
|
||||
{ name: 'eye_white', hex: '#c9c3b4' },
|
||||
// Three tones for the eye - sclera, iris, pupil - which is the "two or three
|
||||
// tones per part" budget, spent where it buys the most: an eye with no tonal
|
||||
// step inside it reads as a hole.
|
||||
{ name: 'iris', hex: '#4a5468' },
|
||||
{ name: 'pupil', hex: '#171a22' },
|
||||
];
|
||||
const IDX = { bg: 0, base: 1, dark: 2, mouth: 3, teeth: 4 };
|
||||
const IDX = { bg: 0, base: 1, dark: 2, mouth: 3, teeth: 4, white: 5, iris: 6, pupil: 7 };
|
||||
|
||||
const state = {
|
||||
dense: null, images: [], stab: null, xform: null,
|
||||
|
|
@ -27,8 +42,11 @@ const state = {
|
|||
fps: 12, audio: null, // fps comes from manifest.json, never guessed
|
||||
aspect: 1, // imgW/imgH; converts MediaPipe's anisotropic space
|
||||
lead: 0, // performance-track offset in frames
|
||||
exposure: 1, // 1 = on 1s, 2 = on 2s. Picture holds; audio does not.
|
||||
interior: null, // per-frame teeth measurement from image content
|
||||
teeth: null, // resolved per-frame {show, t} after knobs
|
||||
eyes: null, // resolved per-frame lid rings, shut flags, iris discs
|
||||
eyeSig: null, // raw eye measurement, kept for the gaze readout
|
||||
};
|
||||
|
||||
const el = (id) => {
|
||||
|
|
@ -53,6 +71,19 @@ const opts = () => ({
|
|||
contourSmooth: +el('contourSmooth').value,
|
||||
apertureThresh: +el('apertureThresh').value / 1000,
|
||||
tol: +el('tol').value / 1000,
|
||||
exposure: +el('exposure').value,
|
||||
irisAnchor: el('irisAnchor').value,
|
||||
gazeOrigin: el('gazeOrigin').value,
|
||||
eyeVerts: +el('eyeVerts').value,
|
||||
lashPx: +el('lashPx').value,
|
||||
irisSize: +el('irisSize').value / 100,
|
||||
gazeGain: +el('gazeGain').value / 100,
|
||||
gazeStep: +el('gazeStep').value, // whole raster pixels
|
||||
pupilPx: +el('pupilPx').value,
|
||||
gazeDwell: +el('gazeDwell').value,
|
||||
blinkCut: +el('blinkCut').value / 1000,
|
||||
blinkHold: +el('blinkHold').value,
|
||||
blinkDwell: +el('blinkDwell').value,
|
||||
});
|
||||
|
||||
function status(msg, kind = '') {
|
||||
|
|
@ -167,6 +198,7 @@ function rebuild(resetKeep) {
|
|||
if (!state.dense) return;
|
||||
const o = opts();
|
||||
state.lead = o.lead;
|
||||
state.exposure = o.exposure;
|
||||
const N = state.dense.length;
|
||||
|
||||
state.stab = stabilize(state.dense, o.smoothWin, state.aspect);
|
||||
|
|
@ -193,6 +225,7 @@ function rebuild(resetKeep) {
|
|||
state.extractKey = extractKey(o);
|
||||
}
|
||||
state.teeth = resolveTeeth(o);
|
||||
state.eyes = buildEyes(o);
|
||||
|
||||
// Plate outline per frame, so a kept frame shows its own head shape.
|
||||
state.plates = state.stab.oval.map((r) => r.map(state.xform));
|
||||
|
|
@ -220,6 +253,124 @@ function faceBoxes() {
|
|||
});
|
||||
}
|
||||
|
||||
// Eyes: lid rings traced per frame, blinks resolved per eye, one gaze shared.
|
||||
//
|
||||
// Lids are a FEATURE in the part table - rotoscoped, open vocabulary, a key on
|
||||
// every frame - so they get exactly the mouth's treatment, including the same
|
||||
// bounded contour average. The iris is a PRIMITIVE: a disc whose position is
|
||||
// quantised, which is where the stylisation lives.
|
||||
function buildEyes(o) {
|
||||
const st = state.stab, N = state.dense.length;
|
||||
const sig = eyeSignals(st);
|
||||
state.eyeSig = sig;
|
||||
|
||||
const blink = { cut: o.blinkCut, dwell: o.blinkDwell, hold: o.blinkHold };
|
||||
const shutR = resolveBlink(sig.openR, blink);
|
||||
const shutL = resolveBlink(sig.openL, blink);
|
||||
|
||||
// Head-local, subsampled, contour-averaged - the identical chain the mouth
|
||||
// takes, with the identical knob. The eye tracks the face, because the face
|
||||
// is what it is attached to; what gets removed is per-frame detector jitter,
|
||||
// not the motion.
|
||||
const lidR = smoothContours(
|
||||
st.lidR.map((r) => toRasterRing(r, EYE_R_RING, o.eyeVerts, state.xform)), o.contourSmooth);
|
||||
const lidL = smoothContours(
|
||||
st.lidL.map((r) => toRasterRing(r, EYE_L_RING, o.eyeVerts, state.xform)), o.contourSmooth);
|
||||
|
||||
// Where the iris hangs. Three behaviours, because this turns out to be an
|
||||
// aesthetic choice and not only a correctness one.
|
||||
//
|
||||
// STEADY (default) reads the socket back off the DRAWN ring. Slots 0 and 8 of
|
||||
// a 16-slot lid ring are the two corners, and subsampling to any even budget n
|
||||
// keeps them at output indices 0 and n/2 - so the ring that gets rendered
|
||||
// carries its own corners with it. The iris is then placed in the frame of the
|
||||
// exact polygon it sits inside, after smoothing, after subsampling: it cannot
|
||||
// drift relative to its own eye, and it inherits the contour average for free.
|
||||
//
|
||||
// FREE reads the raw per-frame corners instead, jitter and all. It is what the
|
||||
// eyes did before any of this, and it is not simply worse - the detector noise
|
||||
// reads as liveliness, the eye never sits perfectly still, and against flat
|
||||
// hand-drawn plates that restlessness can be the thing that sells it. It is
|
||||
// also the honest baseline to compare the other two against.
|
||||
//
|
||||
// LOCKED pins the socket to the take's mean, so the eye never moves in the
|
||||
// head at all. Watch it against a photo underlay and the drawn eyes hang still
|
||||
// over a face whose eyes are moving - that is the registration cost, and it is
|
||||
// real - but once the plate is a drawing rather than a photograph, nothing is
|
||||
// being registered against and it reads as a deliberately locked-off stare.
|
||||
const ringSocket = (ring) => {
|
||||
const a = ring[0], b = ring[ring.length / 2];
|
||||
return { cx: (a.x + b.x) / 2, cy: (a.y + b.y) / 2, w: Math.hypot(a.x - b.x, a.y - b.y) };
|
||||
};
|
||||
const rawSocket = (corners, f) => {
|
||||
const a = state.xform(corners[f][0]), b = state.xform(corners[f][1]);
|
||||
return { cx: (a.x + b.x) / 2, cy: (a.y + b.y) / 2, w: Math.hypot(a.x - b.x, a.y - b.y) };
|
||||
};
|
||||
const meanSocket = (rings) => {
|
||||
const acc = rings.reduce((a, r) => {
|
||||
const k = ringSocket(r);
|
||||
return { cx: a.cx + k.cx, cy: a.cy + k.cy, w: a.w + k.w };
|
||||
}, { cx: 0, cy: 0, w: 0 });
|
||||
const n = rings.length;
|
||||
return { cx: acc.cx / n, cy: acc.cy / n, w: acc.w / n };
|
||||
};
|
||||
const socketFor = (rings, corners) => {
|
||||
if (o.irisAnchor === 'locked') { const k = meanSocket(rings); return () => k; }
|
||||
if (o.irisAnchor === 'free') return (f) => rawSocket(corners, f);
|
||||
return (f) => ringSocket(rings[f]);
|
||||
};
|
||||
const skR = socketFor(lidR, st.cornersR), skL = socketFor(lidL, st.cornersL);
|
||||
const socket = (ring) => ringSocket(ring);
|
||||
|
||||
// Iris radius comes from the take's MEAN eye width, not the current frame's.
|
||||
// Size is authored; only position is tracked. A radius recomputed per frame
|
||||
// would breathe by a fraction of a pixel as the fit's depth-scale wanders,
|
||||
// and at this resolution a fraction of a pixel is a pixel flicking on and off
|
||||
// around the whole silhouette.
|
||||
const meanW = (rings) => rings.reduce((a, r) => a + ringSocket(r).w, 0) / rings.length;
|
||||
const wR = meanW(lidR), wL = meanW(lidL), w = (wR + wL) / 2;
|
||||
|
||||
// Calibrate against the neutral, apply the artist's gain, and only then
|
||||
// quantise - the grid should be a grid of DRAWN positions, because that is
|
||||
// what a viewer reads. Gain is an authored parameter: measured gaze excursion
|
||||
// is small and a character's eye usually wants more throw than a performer's,
|
||||
// which is a decision for a person and not for the detector.
|
||||
const origin = gazeOrigin(sig.gazeRaw, o.gazeOrigin, state.neutral);
|
||||
state.gazeOriginValue = origin;
|
||||
const px = sig.gazeRaw.map((g) => ({
|
||||
x: (g.x - origin.x) * o.gazeGain * w,
|
||||
y: (g.y - origin.y) * o.gazeGain * w,
|
||||
}));
|
||||
const gaze = quantizeGaze(px, o.gazeStep, o.gazeDwell);
|
||||
|
||||
const eye = (sk, lids, shut, rad, f) => {
|
||||
const e = sk(f);
|
||||
return {
|
||||
// The lash line is the lid ring pushed outward by a fixed number of
|
||||
// pixels, exactly as the mouth's outer ring sits outside its inner one.
|
||||
// When the eye shuts, the traced ring goes near-degenerate and this
|
||||
// collapses to a lens - which is a closed eye, drawn correctly, for free.
|
||||
lash: offsetRing(lids[f], o.lashPx),
|
||||
lid: lids[f],
|
||||
shut: shut[f],
|
||||
// Rounded to whole pixels. The rasteriser quantises everything anyway, so
|
||||
// this costs nothing - but it means the iris and the square pupil share
|
||||
// one integer centre, so the pupil is exactly its nominal size on every
|
||||
// frame instead of spilling to the next pixel on some and not others.
|
||||
iris: { x: Math.round(e.cx + gaze[f].x), y: Math.round(e.cy + gaze[f].y), r: rad },
|
||||
pupil: o.pupilPx,
|
||||
};
|
||||
};
|
||||
|
||||
return {
|
||||
gazePx: px, gaze, shutR, shutL, hasIris: sig.hasIris,
|
||||
frames: Array.from({ length: N }, (_, f) => ({
|
||||
r: eye(skR, lidR, shutR, (wR * o.irisSize) / 2, f),
|
||||
l: eye(skL, lidL, shutL, (wL * o.irisSize) / 2, f),
|
||||
})),
|
||||
};
|
||||
}
|
||||
|
||||
// Presence gets hysteresis and a minimum dwell, the same treatment plate
|
||||
// selection gets: a teeth block that blinks on and off for single frames is
|
||||
// worse than one that is simply absent. Appearing needs a clear signal, staying
|
||||
|
|
@ -275,7 +426,7 @@ function resolveTeeth(o) {
|
|||
// stand-in until a drawing exists.
|
||||
function renderFrame(f, mode = plateMode()) {
|
||||
const r = new IndexedRaster(RW, RH);
|
||||
const pf = heldFrame(keptSorted(), f); // the plate frame on screen
|
||||
const pf = plateIndex(f); // the plate frame on screen
|
||||
|
||||
if (mode === 'posterize' && state.images[pf]) {
|
||||
posterizeInto(r, state.images[pf], state.stab.transforms[pf], state.xform,
|
||||
|
|
@ -285,6 +436,12 @@ function renderFrame(f, mode = plateMode()) {
|
|||
if (mode === 'oval' || mode === 'oval+photo') r.fillPoly(state.plates[pf], IDX.base);
|
||||
}
|
||||
|
||||
// Eyes run on the CLOCK, not on the mouth lead. The lead is a lip-sync
|
||||
// device: it exists because a mouth shape anticipates the sound it makes.
|
||||
// Nothing about a blink or a glance is tied to the audio, so shifting the
|
||||
// eyes would only slide them off the head that carries them.
|
||||
if (state.eyes) drawEyes(r, state.eyes.frames[f]);
|
||||
|
||||
const mf = leadIndex(f); // performance frame, possibly ahead
|
||||
r.fillPoly(state.outer[mf], IDX.dark); // mouth keeps every frame
|
||||
if (!state.hidden[mf]) {
|
||||
|
|
@ -295,13 +452,34 @@ function renderFrame(f, mode = plateMode()) {
|
|||
return r;
|
||||
}
|
||||
|
||||
// Lash ring, then sclera, then iris - the same three-layer structure the mouth
|
||||
// has, for the same reason: the dark ring outside the pale interior is what
|
||||
// makes a flat shape read as an opening rather than a blob.
|
||||
//
|
||||
// The iris is stencilled to the sclera it was just drawn over, so the lid crops
|
||||
// it automatically. Nothing needs to clamp the gaze to keep the iris inside the
|
||||
// eye, which matters because a clamp would flatten the performance at exactly
|
||||
// the extremes that carry it.
|
||||
function drawEyes(r, e) {
|
||||
for (const s of [e.r, e.l]) {
|
||||
r.fillPoly(s.lash, IDX.dark);
|
||||
if (s.shut) continue; // a shut eye IS the lash line, alone
|
||||
r.fillPoly(s.lid, IDX.white);
|
||||
r.fillDisc(s.iris.x, s.iris.y, s.iris.r, IDX.iris, IDX.white);
|
||||
// Stencilled to the iris, which is itself stencilled to the sclera - so the
|
||||
// pupil is cropped by the lid transitively, and a blink or an extreme gaze
|
||||
// takes the right bite out of it without anything having to compute where.
|
||||
if (s.pupil) r.fillRect(s.iris.x, s.iris.y, s.pupil, IDX.pupil, IDX.iris);
|
||||
}
|
||||
}
|
||||
|
||||
const plateMode = () => el('plateMode').value;
|
||||
|
||||
// Photo modes composite under the indexed layer, so the flat shapes stay exactly
|
||||
// as they render while the reference sits behind them.
|
||||
function compositeRender(canvas, f, zoom) {
|
||||
const mode = plateMode();
|
||||
const pf = heldFrame(keptSorted(), f);
|
||||
const pf = plateIndex(f);
|
||||
const img = state.images[pf];
|
||||
const showPhoto = img && (mode === 'photo' || mode === 'photo-dim' || mode === 'oval+photo');
|
||||
|
||||
|
|
@ -347,11 +525,22 @@ const keptSorted = () => [...state.keep].sort((a, b) => a - b);
|
|||
// Positive lead = the mouth arrives earlier. Only performance parts shift; the
|
||||
// head stays with the audio, because it is the mouth that should anticipate.
|
||||
function leadIndex(f) {
|
||||
// Reads a cached scalar, not opts(): this runs once per strip thumbnail, and
|
||||
// Reads cached scalars, not opts(): this runs once per strip thumbnail, and
|
||||
// calling opts() here meant ~14 DOM reads x 74 frames on every redraw.
|
||||
return shiftIndex(f, state.lead, state.dense.length);
|
||||
//
|
||||
// Exposure first, then lead. The grid decides WHICH frames get a new drawing;
|
||||
// the lead then shifts which pose that drawing carries, by whole frames of the
|
||||
// original track. Applying them the other way round would put the changes on
|
||||
// the wrong beats - the picture would update on the odd frames instead of
|
||||
// holding on the twos.
|
||||
return shiftIndex(exposeIndex(f, state.exposure), state.lead, state.dense.length);
|
||||
}
|
||||
|
||||
// The plate rides the same grid, so the whole picture updates together. On 2s
|
||||
// means on 2s - a head that cut on the odd frames while the mouth cut on the
|
||||
// even ones would read as two performances laid over each other.
|
||||
const plateIndex = (f) => heldFrame(keptSorted(), exposeIndex(f, state.exposure));
|
||||
|
||||
function blit(canvas, raster, zoom) {
|
||||
canvas.width = RW * zoom; canvas.height = RH * zoom;
|
||||
canvas.getContext('2d').putImageData(raster.toImageData(PALETTE.map((p) => p.hex), zoom), 0, 0);
|
||||
|
|
@ -372,21 +561,42 @@ function drawReadout() {
|
|||
el('readout').textContent =
|
||||
`${state.dense.length} frames → ${kept.length} drawings · ` +
|
||||
`teeth on ${teethFrames}f · ` +
|
||||
`${blinkRuns(state.eyes.shutR).length}/${blinkRuns(state.eyes.shutL).length} blinks R/L · ` +
|
||||
`${gazeCells(state.eyes.gaze)} gaze cells · ` +
|
||||
(state.exposure > 1
|
||||
? `on ${state.exposure}s = ${(state.fps / state.exposure).toFixed(4).replace(/\.?0+$/, '')}fps · `
|
||||
: '') +
|
||||
(lead ? `mouth leads ${lead}f (${(lead / state.fps * 1000).toFixed(0)}ms) · ` : '') +
|
||||
`holds ${Math.min(...runs)}–${Math.max(...runs)} frames · ` +
|
||||
`neutral f${state.neutral} · residual ` +
|
||||
`${(state.stab.residual.reduce((a, b) => a + b, 0) / state.dense.length).toFixed(4)}`;
|
||||
}
|
||||
|
||||
// Blinks as RUNS, not as shut frames: a three-frame blink is one blink, and the
|
||||
// count is only useful as "did the performer blink six times or sixty".
|
||||
function blinkRuns(shut) {
|
||||
const runs = [];
|
||||
for (let f = 0; f < shut.length; f++) {
|
||||
if (shut[f] && !shut[f - 1]) runs.push(f);
|
||||
}
|
||||
return runs;
|
||||
}
|
||||
|
||||
// How many distinct positions the iris ever occupies. This is the number the
|
||||
// gaze knobs exist to control: two or three is a character who looks at things,
|
||||
// forty is an unquantised iris sliding around, which is what the grid is for.
|
||||
const gazeCells = (gaze) => new Set(gaze.map((g) => `${g.x},${g.y}`)).size;
|
||||
|
||||
function drawPanes() {
|
||||
const f = state.frame, kept = keptSorted();
|
||||
const pf = heldFrame(kept, f);
|
||||
const pf = plateIndex(f);
|
||||
// The mouth frame is always shown, not only when shifted, so the number can be
|
||||
// watched diverging from f rather than taken on trust.
|
||||
const lead = state.lead;
|
||||
el('framelabel').textContent =
|
||||
`f ${f} / ${state.dense.length - 1} · ${(f / state.fps).toFixed(2)}s · ` +
|
||||
`plate f${pf} · mouth f${leadIndex(f)}` +
|
||||
(state.exposure > 1 && f % state.exposure ? ' (held)' : '') +
|
||||
(lead ? ` (${lead > 0 ? '+' : ''}${lead} = ${(lead / state.fps * 1000).toFixed(0)}ms)` : '') +
|
||||
(state.keep.has(f) ? ' · KEPT' : ' · held');
|
||||
|
||||
|
|
@ -404,12 +614,14 @@ function drawPanes() {
|
|||
const map = (p) => ({ x: dx + (p.x * im.naturalWidth - sx) * s, y: dy + (p.y * im.naturalHeight - sy) * s });
|
||||
strokePts(g1, LIPS_OUTER.map((i) => map(state.dense[f][i])), '#4ade80');
|
||||
strokePts(g1, LIPS_INNER.map((i) => map(state.dense[f][i])), '#f87171');
|
||||
drawEyeOverlay(g1, map, f);
|
||||
} else {
|
||||
g1.fillStyle = '#555'; g1.font = '13px system-ui';
|
||||
g1.fillText('synthetic — no source frames', 14, 24);
|
||||
const sc = (p) => ({ x: p.x * c1.width, y: p.y * c1.height });
|
||||
strokePts(g1, LIPS_OUTER.map((i) => sc(state.dense[f][i])), '#4ade80');
|
||||
strokePts(g1, LIPS_INNER.map((i) => sc(state.dense[f][i])), '#f87171');
|
||||
drawEyeOverlay(g1, sc, f);
|
||||
}
|
||||
|
||||
const c2 = el('cv-stab'), g2 = c2.getContext('2d');
|
||||
|
|
@ -426,9 +638,18 @@ function drawPanes() {
|
|||
if (mf !== f) strokePts(g2, z(state.outer[f]), '#2f6b46');
|
||||
strokePts(g2, z(state.outer[mf]), '#4ade80');
|
||||
if (!state.hidden[mf]) strokePts(g2, z(state.inner[mf]), '#f87171');
|
||||
for (const e of [state.eyes.frames[f].r, state.eyes.frames[f].l]) {
|
||||
strokePts(g2, z(e.lid), e.shut ? '#f87171' : '#60a5fa');
|
||||
if (e.shut) continue;
|
||||
g2.strokeStyle = '#fbbf24';
|
||||
g2.beginPath();
|
||||
g2.arc(e.iris.x * ZOOM, e.iris.y * ZOOM, e.iris.r * ZOOM, 0, Math.PI * 2);
|
||||
g2.stroke();
|
||||
}
|
||||
|
||||
compositeRender(el('cv-render'), f, ZOOM);
|
||||
drawInteriorDebug(f);
|
||||
drawGazeDebug(f);
|
||||
}
|
||||
|
||||
// What the teeth measurement actually saw: sampled region, pixels above
|
||||
|
|
@ -458,6 +679,78 @@ function drawInteriorDebug(fRaw) {
|
|||
`area ${m.area}px · ${te.show ? 'SHOWN' : 'hidden'}`;
|
||||
}
|
||||
|
||||
// Lid rings and the iris, on the raw frame. Landmark overlays are how you tell
|
||||
// a tracking failure from a knob set wrong, and the eyes need it more than the
|
||||
// mouth does: an iris that has latched onto an eyebrow looks, in the flat
|
||||
// render alone, exactly like a gaze gain that is too high.
|
||||
function drawEyeOverlay(g, map, f) {
|
||||
const lm = state.dense[f];
|
||||
for (const ring of [EYE_R_RING, EYE_L_RING]) {
|
||||
strokePts(g, ring.map((i) => map(lm[i])), '#60a5fa');
|
||||
}
|
||||
if (!state.eyes.hasIris) return;
|
||||
for (const iris of [IRIS_A, IRIS_B]) {
|
||||
strokePts(g, iris.slice(1).map((i) => map(lm[i])), '#fbbf24');
|
||||
}
|
||||
}
|
||||
|
||||
// The gaze field: every position the iris takes over the whole take, plus where
|
||||
// it is now. Tune against this, not against the numbers - "4 cells" tells you
|
||||
// the quantisation is working, but only the picture tells you whether the four
|
||||
// are the four looks the performance actually has.
|
||||
function drawGazeDebug(f) {
|
||||
const cv = el('cv-gaze'), S = 150;
|
||||
cv.width = S; cv.height = S;
|
||||
const g = cv.getContext('2d');
|
||||
g.fillStyle = '#0d0f16'; g.fillRect(0, 0, S, S);
|
||||
|
||||
const ex = state.eyes;
|
||||
// Scale so the widest excursion in the take fills the box, with a floor so a
|
||||
// nearly-still gaze does not get magnified into a light show.
|
||||
let m = 2;
|
||||
for (const p of ex.gazePx) m = Math.max(m, Math.abs(p.x), Math.abs(p.y));
|
||||
const k = (S / 2 - 8) / m;
|
||||
const X = (v) => S / 2 + v * k, Y = (v) => S / 2 + v * k;
|
||||
|
||||
const o = opts();
|
||||
if (o.gazeStep > 0) {
|
||||
g.strokeStyle = '#1b2030'; g.lineWidth = 1;
|
||||
for (let i = -20; i <= 20; i++) {
|
||||
const v = i * o.gazeStep;
|
||||
if (Math.abs(v) > m) continue;
|
||||
g.beginPath(); g.moveTo(X(v), 0); g.lineTo(X(v), S); g.stroke();
|
||||
g.beginPath(); g.moveTo(0, Y(v)); g.lineTo(S, Y(v)); g.stroke();
|
||||
}
|
||||
}
|
||||
g.strokeStyle = '#2a2f3e';
|
||||
g.beginPath(); g.moveTo(S / 2, 0); g.lineTo(S / 2, S);
|
||||
g.moveTo(0, S / 2); g.lineTo(S, S / 2); g.stroke();
|
||||
|
||||
g.fillStyle = '#2f6b46';
|
||||
for (const p of ex.gaze) g.fillRect(X(p.x) - 1.5, Y(p.y) - 1.5, 3, 3);
|
||||
|
||||
const raw = ex.gazePx[f], q = ex.gaze[f];
|
||||
g.fillStyle = '#8891a5';
|
||||
g.fillRect(X(raw.x) - 1, Y(raw.y) - 1, 2, 2);
|
||||
g.fillStyle = '#fbbf24';
|
||||
g.beginPath(); g.arc(X(q.x), Y(q.y), 4, 0, Math.PI * 2); g.fill();
|
||||
|
||||
const sig = state.eyeSig, fr = state.eyes.frames[f];
|
||||
const og = state.gazeOriginValue;
|
||||
// Per-eye raw gaze is the diagnostic for a wrong-looking eyeline. If the two
|
||||
// agree and both point the wrong way, the ORIGIN is wrong. If they disagree in
|
||||
// a sustained way, it is out-of-plane head rotation biasing the projection,
|
||||
// which no 2D measurement can undo.
|
||||
const sgn = (v) => `${v >= 0 ? '+' : ''}${v.toFixed(3)}`;
|
||||
el('eyeinfo').textContent =
|
||||
`open R ${sig.openR[f].toFixed(3)} L ${sig.openL[f].toFixed(3)} / cut ${o.blinkCut.toFixed(3)}\n` +
|
||||
`${fr.r.shut ? 'R SHUT ' : ''}${fr.l.shut ? 'L SHUT' : ''}${!fr.r.shut && !fr.l.shut ? 'both open' : ''}\n` +
|
||||
`gaze ${q.x >= 0 ? '+' : ''}${q.x.toFixed(1)}, ${q.y >= 0 ? '+' : ''}${q.y.toFixed(1)} px\n` +
|
||||
`raw R ${sgn(sig.gazeR[f].x)} L ${sgn(sig.gazeL[f].x)} (x, eye widths)\n` +
|
||||
`origin ${o.gazeOrigin} ${sgn(og.x)}, ${sgn(og.y)}` +
|
||||
(ex.hasIris ? '' : ' — no iris landmarks');
|
||||
}
|
||||
|
||||
function strokePts(g, pts, color, lw = 1) {
|
||||
g.strokeStyle = color; g.lineWidth = lw;
|
||||
g.beginPath();
|
||||
|
|
@ -538,12 +831,57 @@ function drawWorksheet() {
|
|||
|
||||
/* ---------- export ---------- */
|
||||
|
||||
// Six parts, three per eye, mirroring the mouth's lash/interior/content stack.
|
||||
// `clip` is what tells the renderer the iris is stencilled by the sclera rather
|
||||
// than merely drawn after it - without it an extreme gaze would put the iris on
|
||||
// the cheek.
|
||||
function eyeParts(grid) {
|
||||
const out = [];
|
||||
// Eyes ride the exposure grid but NOT the mouth lead: the lead is a lip-sync
|
||||
// device and nothing about a blink is tied to the audio.
|
||||
const src = (f) => exposeIndex(f, state.exposure);
|
||||
[['r', 20], ['l', 23]].forEach(([side, z]) => {
|
||||
const at = (f) => state.eyes.frames[src(f)][side];
|
||||
out.push(
|
||||
{ name: `eye_${side}`, kind: 'poly', z, color: 'skin_dark', interp: 'hold',
|
||||
keys: grid.map((f) => ({ f, src: src(f), pts: at(f).lash })) },
|
||||
{ name: `eye_${side}_in`, kind: 'poly', z: z + 1, color: 'eye_white', interp: 'hold',
|
||||
parent: `eye_${side}`,
|
||||
keys: grid.map((f) =>
|
||||
(at(f).shut ? { f, hidden: true } : { f, src: src(f), pts: at(f).lid })) },
|
||||
{ name: `iris_${side}`, kind: 'disc', z: z + 2, color: 'iris', interp: 'hold',
|
||||
parent: `eye_${side}_in`, clip: `eye_${side}_in`,
|
||||
keys: grid.map((f) => {
|
||||
const e = at(f);
|
||||
return e.shut ? { f, hidden: true }
|
||||
: { f, src: src(f), c: { x: e.iris.x, y: e.iris.y }, r: e.iris.r };
|
||||
}) },
|
||||
);
|
||||
if (!state.eyes.frames[0].r.pupil) return;
|
||||
out.push(
|
||||
{ name: `pupil_${side}`, kind: 'rect', z: z + 3, color: 'pupil', interp: 'hold',
|
||||
parent: `iris_${side}`, clip: `iris_${side}`,
|
||||
keys: grid.map((f) => {
|
||||
const e = at(f);
|
||||
return e.shut ? { f, hidden: true }
|
||||
: { f, src: src(f), c: { x: e.iris.x, y: e.iris.y }, size: e.pupil };
|
||||
}) },
|
||||
);
|
||||
});
|
||||
return out;
|
||||
}
|
||||
|
||||
function exportTake() {
|
||||
const kept = keptSorted();
|
||||
const N = state.dense.length;
|
||||
// Output frames that actually carry a key. Everything between them is a hold,
|
||||
// which the take format already expresses, so on 2s emits half the keys rather
|
||||
// than emitting each pose twice.
|
||||
const grid = [];
|
||||
for (let f = 0; f < N; f += state.exposure) grid.push(f);
|
||||
const take = {
|
||||
name: el('takename').value || 'line_01',
|
||||
frames: N, width: RW, height: RH, exposure: 1, fps: state.fps,
|
||||
frames: N, width: RW, height: RH, exposure: state.exposure, fps: state.fps,
|
||||
palette: PALETTE,
|
||||
slot: { x: RW / 2, y: RH / 2 },
|
||||
parts: [
|
||||
|
|
@ -554,14 +892,19 @@ function exportTake() {
|
|||
// key f carries the pose from source frame f+lead - so the renderer never
|
||||
// needs to know about it.
|
||||
{ name: 'mouth', kind: 'poly', z: 30, color: 'skin_dark', interp: 'hold',
|
||||
keys: state.outer.map((_, f) => ({ f, src: leadIndex(f), pts: state.outer[leadIndex(f)] })) },
|
||||
keys: grid.map((f) => ({ f, src: leadIndex(f), pts: state.outer[leadIndex(f)] })) },
|
||||
{ name: 'mouth_in', kind: 'poly', z: 31, color: 'mouth_dark', interp: 'hold', parent: 'mouth',
|
||||
keys: state.inner.map((_, f) => {
|
||||
keys: grid.map((f) => {
|
||||
const m = leadIndex(f);
|
||||
return state.hidden[m] ? { f, hidden: true } : { f, src: m, pts: state.inner[m] };
|
||||
}) },
|
||||
// Eyes. The lids are traced, so like the mouth they cost nothing and keep
|
||||
// every frame. The iris is a primitive: its quantised position means the
|
||||
// key stream is dense but the VALUES change only on saccades, so a
|
||||
// hold-interpolating renderer cuts between fixations by itself.
|
||||
...eyeParts(grid),
|
||||
{ name: 'teeth', kind: 'poly', z: 32, color: 'teeth', interp: 'hold', parent: 'mouth_in',
|
||||
keys: state.teeth.map((_, f) => {
|
||||
keys: grid.map((f) => {
|
||||
const m = leadIndex(f), te = state.teeth[m];
|
||||
return te.show && te.pts ? { f, src: m, pts: te.pts } : { f, hidden: true };
|
||||
}) },
|
||||
|
|
@ -577,7 +920,9 @@ function exportTake() {
|
|||
a.href = URL.createObjectURL(new Blob([text], { type: 'text/plain' }));
|
||||
a.download = `${take.name}.take`;
|
||||
a.click();
|
||||
status(`exported — ${kept.length} plate drawings, ${N} mouth frames`, 'ok');
|
||||
status(`exported — ${kept.length} plate drawings, ${grid.length} mouth keys` +
|
||||
(state.exposure > 1 ? ` on ${state.exposure}s` : '') + ', ' +
|
||||
`${blinkRuns(state.eyes.shutR).length + blinkRuns(state.eyes.shutL).length} blinks`, 'ok');
|
||||
}
|
||||
|
||||
/* ---------- wiring ---------- */
|
||||
|
|
@ -605,6 +950,7 @@ async function runFrames() {
|
|||
state.interior = measureAll(images, dense, opts());
|
||||
el('scrub').max = dense.length - 1;
|
||||
state.frame = 0;
|
||||
labelExposure();
|
||||
rebuild(true);
|
||||
const dur = (dense.length / state.fps).toFixed(2);
|
||||
status(`${images.length} frames · ${images[0].naturalWidth}x${images[0].naturalHeight} · ` +
|
||||
|
|
@ -626,24 +972,44 @@ function runSynthetic() {
|
|||
state.dense = synthDense(72);
|
||||
el('scrub').max = 71;
|
||||
state.frame = 0;
|
||||
labelExposure();
|
||||
rebuild(true);
|
||||
status('synthetic — exercises everything below detection', 'ok');
|
||||
}
|
||||
|
||||
// How each slider's raw value reads out. A table rather than the conditional
|
||||
// chain this used to be: that chain grew a branch per knob and was one ternary
|
||||
// away from being unreadable.
|
||||
const FMT = {
|
||||
apertureThresh: (v) => (v / 1000).toFixed(3),
|
||||
tol: (v) => (v / 1000).toFixed(3),
|
||||
blinkCut: (v) => (v / 1000).toFixed(3),
|
||||
teethOn: (v) => (v / 100).toFixed(2),
|
||||
teethErode: (v) => (v / 100).toFixed(2),
|
||||
tongueReject: (v) => (v / 100).toFixed(2),
|
||||
topBias: (v) => (v / 100).toFixed(2),
|
||||
irisSize: (v) => `${v}%`,
|
||||
gazeGain: (v) => (v / 100).toFixed(2),
|
||||
gazeStep: (v) => (v ? `${v}px` : 'off'),
|
||||
pupilPx: (v) => (v ? `${v}px` : 'off'),
|
||||
lashPx: (v) => `${v}px`,
|
||||
lead: (v) => (v > 0 ? `+${v}` : String(v)),
|
||||
};
|
||||
|
||||
for (const id of ['verts', 'smoothWin', 'contourSmooth', 'apertureThresh', 'tol',
|
||||
'teethOn', 'teethDwell', 'teethErode', 'tongueReject', 'blobGrow',
|
||||
'topBias', 'teethVerts', 'teethSmooth', 'lead']) {
|
||||
'topBias', 'teethVerts', 'teethSmooth', 'lead',
|
||||
'eyeVerts', 'lashPx', 'irisSize', 'pupilPx', 'gazeGain',
|
||||
'gazeStep', 'gazeDwell', 'blinkCut', 'blinkHold', 'blinkDwell']) {
|
||||
const show = () => {
|
||||
el(id + 'v').textContent = FMT[id] ? FMT[id](+el(id).value) : el(id).value;
|
||||
};
|
||||
el(id).addEventListener('input', () => {
|
||||
el(id + 'v').textContent = id === 'apertureThresh' || id === 'tol'
|
||||
? (+el(id).value / 1000).toFixed(3)
|
||||
: ['teethOn', 'teethErode', 'tongueReject', 'topBias'].includes(id)
|
||||
? (+el(id).value / 100).toFixed(2)
|
||||
: id === 'lead' && +el(id).value > 0 ? `+${el(id).value}`
|
||||
: el(id).value;
|
||||
show();
|
||||
if (id === 'tol') return; // tol only matters when you ask for a suggestion
|
||||
rebuild(false);
|
||||
});
|
||||
el(id + 'v').textContent = el(id).value;
|
||||
show();
|
||||
}
|
||||
|
||||
function seekTo(f) {
|
||||
|
|
@ -657,6 +1023,22 @@ el('btn-frames').onclick = runFrames;
|
|||
el('btn-synth').onclick = runSynthetic;
|
||||
el('btn-export').onclick = exportTake;
|
||||
el('plateMode').addEventListener('change', () => { if (state.dense) drawAll(); });
|
||||
el('exposure').addEventListener('change', () => { if (state.dense) rebuild(false); });
|
||||
for (const id of ['irisAnchor', 'gazeOrigin']) {
|
||||
el(id).addEventListener('change', () => { if (state.dense) rebuild(false); });
|
||||
}
|
||||
|
||||
// Label the exposure options in the only units that mean anything here: the
|
||||
// rate the picture actually changes at, which depends on the clip's own rate.
|
||||
// "on 2s" is the animator's name for it and the number is what you hear against
|
||||
// the audio, so the menu says both.
|
||||
function labelExposure() {
|
||||
for (const opt of el('exposure').options) {
|
||||
const n = +opt.value;
|
||||
const rate = (state.fps / n).toFixed(4).replace(/\.?0+$/, '');
|
||||
opt.textContent = `${rate} fps · on ${n}s`;
|
||||
}
|
||||
}
|
||||
el('btn-saveframe').onclick = () => {
|
||||
if (!state.dense) return;
|
||||
const cv = document.createElement('canvas');
|
||||
|
|
@ -775,3 +1157,5 @@ else if (location.hash === '#frames') runFrames();
|
|||
else status('ready — Load frames, then step with \u2190 \u2192 and delete with X');
|
||||
|
||||
window.__roto = state; // headless smoke test reads this
|
||||
window.__render = compositeRender; // ...and renders arbitrary frames off-screen
|
||||
window.__lead = leadIndex; // ...and resolves the performance frame
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue