noe/arthur
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Fix horizontal stretch from MediaPipe's anisotropic normalised space

Browse source 
MediaPipe normalises x by image WIDTH and y by image HEIGHT, so for a 1080x1920
clip one unit of x is 1080px and one unit of y is 1920px. makeXform applied a
single scale to both, stretching everything horizontally by H/W - 1.78x on this
footage. The photo underlay looked equally squashed because frameAffine divided
x by imgW, matching the equally wrong vector shapes rather than disagreeing
with them.

Fixed at ingest: landmarks convert to an isotropic space whose unit is one image
height (x *= W/H), so equal numbers mean equal pixels everywhere downstream.
Pixel mapping follows - both axes divide by imgH.

This also silently fixes head roll. fitSimilarity was fitting a rotation in a
sheared space, so the "similarity" it recovered was not one, and stabilisation
of rolled heads was subtly wrong.

selftest: a shape circular in pixel space must stay circular in raster space,
checked at 1080x1920, 1920x1080 and 640x640. Fails at ratio 1.78 without the
conversion.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Your Name 2026-09-24 15:03:12 -04:00
parent de90bfd907
commit 29b0bd6690
4 changed files with 64 additions and 14 deletions
Download patch file Download diff file Expand all files Collapse all files

34
js/selftest.js
View file

@ -113,6 +113,40 @@ export function run() {
const apRange = Math.max(...stab.aperture) - Math.min(...stab.aperture);
ok('stabilisation preserves mouth motion', apRange > 0.05, `aperture range ${apRange.toFixed(4)}`);
// ASPECT: a shape that is circular in PIXEL space must stay circular in raster
// space. MediaPipe normalises x by width and y by height, so for a portrait
// frame equal normalised numbers are unequal pixel distances; feeding those
// straight through stretches everything horizontally by H/W. This asserts the
// isotropic conversion, and fails at ~1.78 for a 1080x1920 clip without it.
for (const [W, H] of [[1080, 1920], [1920, 1080], [640, 640]]) {
const aspect = W / H;
const N = 24, cx = 0.5, cy = 0.5, rPx = 200;
// a true circle of radius rPx, expressed in MediaPipe normalised coords
const circleFrames = [];
for (let t = 0; t < 4; t++) {
const pts = new Array(478).fill(null).map(() => ({ x: 0.5, y: 0.5, z: 0 }));
RIGID.forEach((id, k) => {
const a = (k / RIGID.length) * Math.PI * 2;
pts[id] = { x: cx + (120 * Math.cos(a)) / W, y: cy + (120 * Math.sin(a)) / H, z: 0 };
});
LIPS_OUTER.forEach((id, k) => {
const a = -(k / LIPS_OUTER.length) * Math.PI * 2;
pts[id] = { x: cx + (rPx * Math.cos(a)) / W, y: cy + (rPx * Math.sin(a)) / H, z: 0 };
});
FACE_OVAL.forEach((id, k) => {
const a = -(k / FACE_OVAL.length) * Math.PI * 2;
pts[id] = { x: cx + (420 * Math.cos(a)) / W, y: cy + (420 * Math.sin(a)) / H, z: 0 };
});
circleFrames.push(pts);
}
const st2 = stabilize(circleFrames, 0, aspect);
const ring = toRasterRing(st2.outer[0], LIPS_OUTER, 16, (p) => p);
const xs = ring.map((p) => p.x), ys = ring.map((p) => p.y);
const ratio = (Math.max(...xs) - Math.min(...xs)) / (Math.max(...ys) - Math.min(...ys));
ok(`circle stays circular at ${W}x${H}`, Math.abs(ratio - 1) < 0.02,
`w/h ratio ${ratio.toFixed(4)}`);
}
// key selection
const xf = (p) => ({ x: p.x * 320, y: p.y * 200 });
const shapes = stab.outer.map((r) => toRasterRing(r, LIPS_OUTER, 8, xf));