noe/arthur
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Fix horizontal stretch from MediaPipe's anisotropic normalised space

Browse source 
MediaPipe normalises x by image WIDTH and y by image HEIGHT, so for a 1080x1920
clip one unit of x is 1080px and one unit of y is 1920px. makeXform applied a
single scale to both, stretching everything horizontally by H/W - 1.78x on this
footage. The photo underlay looked equally squashed because frameAffine divided
x by imgW, matching the equally wrong vector shapes rather than disagreeing
with them.

Fixed at ingest: landmarks convert to an isotropic space whose unit is one image
height (x *= W/H), so equal numbers mean equal pixels everywhere downstream.
Pixel mapping follows - both axes divide by imgH.

This also silently fixes head roll. fitSimilarity was fitting a rotation in a
sheared space, so the "similarity" it recovered was not one, and stabilisation
of rolled heads was subtly wrong.

selftest: a shape circular in pixel space must stay circular in raster space,
checked at 1080x1920, 1920x1080 and 640x640. Fails at ratio 1.78 without the
conversion.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Your Name 2026-09-24 15:03:12 -04:00
parent de90bfd907
commit 29b0bd6690
4 changed files with 64 additions and 14 deletions
Download patch file Download diff file Expand all files Collapse all files

9
js/app.js
View file

@ -23,6 +23,7 @@ const state = {
keep: new Set(), // frames that get their own plate drawing
frame: 0, playing: false, faceBox: null,
fps: 12, audio: null, // fps comes from manifest.json, never guessed
aspect: 1, // imgW/imgH; converts MediaPipe's anisotropic space
};
const el = (id) => document.getElementById(id);
@ -134,7 +135,7 @@ function rebuild(resetKeep) {
const o = opts();
const N = state.dense.length;
state.stab = stabilize(state.dense, o.smoothWin);
state.stab = stabilize(state.dense, o.smoothWin, state.aspect);
const ap = state.stab.aperture;
const head = Math.max(1, Math.floor(N / 4));
@ -440,11 +441,13 @@ async function runFrames() {
}
const { dense, missing } = await detectAll(images);
state.images = images; state.dense = dense;
state.aspect = images[0].naturalWidth / images[0].naturalHeight;
el('scrub').max = dense.length - 1;
state.frame = 0;
rebuild(true);
const dur = (dense.length / state.fps).toFixed(2);
status(`${images.length} frames · ${state.fps}fps · ${dur}s` +
status(`${images.length} frames · ${images[0].naturalWidth}x${images[0].naturalHeight} · ` +
`${state.fps}fps · ${dur}s` +
(state.audio ? ' · audio loaded' : ' · no audio') +
(missing.length ? ` · no face on ${missing.length} (held previous)` : ''),
missing.length ? 'warn' : 'ok');
@ -455,6 +458,8 @@ function runSynthetic() {
state.images = [];
attachAudio(null);
state.fps = 12;
state.aspect = 1; // synthetic landmarks are generated square
state.dense = synthDense(72);
el('scrub').max = 71;
state.frame = 0;