noe/arthur
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Fix horizontal stretch from MediaPipe's anisotropic normalised space

Browse source 
MediaPipe normalises x by image WIDTH and y by image HEIGHT, so for a 1080x1920
clip one unit of x is 1080px and one unit of y is 1920px. makeXform applied a
single scale to both, stretching everything horizontally by H/W - 1.78x on this
footage. The photo underlay looked equally squashed because frameAffine divided
x by imgW, matching the equally wrong vector shapes rather than disagreeing
with them.

Fixed at ingest: landmarks convert to an isotropic space whose unit is one image
height (x *= W/H), so equal numbers mean equal pixels everywhere downstream.
Pixel mapping follows - both axes divide by imgH.

This also silently fixes head roll. fitSimilarity was fitting a rotation in a
sheared space, so the "similarity" it recovered was not one, and stabilisation
of rolled heads was subtly wrong.

selftest: a shape circular in pixel space must stay circular in raster space,
checked at 1080x1920, 1920x1080 and 640x640. Fails at ratio 1.78 without the
conversion.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Your Name 2026-09-24 15:03:12 -04:00
parent de90bfd907
commit 29b0bd6690
4 changed files with 64 additions and 14 deletions
Download patch file Download diff file Expand all files Collapse all files

10
js/underlay.js
View file

@ -10,11 +10,13 @@ import { applySim } from './mathutil.js';
// Compose pixel-space -> raster-space into one affine.
//
// MediaPipe normalises x by width and y by height, so normalised space is a
// stretched pixel space and the composition is a general affine rather than a
// similarity. Three mapped points determine it exactly.
// Landmarks are converted to an isotropic space (unit = one image height) before
// fitting, so pixels map in the same way: BOTH axes divide by imgH, not by their
// own dimension. Dividing x by imgW here instead is what stretched the underlay
// horizontally by H/W and made it disagree with nothing - it matched the equally
// wrong vector shapes.
export function frameAffine(tf, xform, imgW, imgH) {
const map = (px, py) => xform(applySim(tf, { x: px / imgW, y: py / imgH }));
const map = (px, py) => xform(applySim(tf, { x: px / imgH, y: py / imgH }));
const P0 = map(0, 0), P1 = map(imgW, 0), P2 = map(0, imgH);
return {
a: (P1.x - P0.x) / imgW, b: (P1.y - P0.y) / imgW,