Flat-ground distance carries a ~11% range-proportional scale error, not just scatter — multi-view self-consistency measures it and DA3 can anchor it

[jonfroehlich]

The measurement

Multi-view association in the auto-labeler (sidewalk-auto-labeler#27) fuses detections of the same physical ramp seen from several panos. That makes a new check possible: the same ramp, raycast from cameras at different distances, should land in the same place. It does not.

For every fused site, take each member's signed along-ray residual — how far that view's ground point sits beyond (or short of) the fused position, measured along its own ray — and regress it on that member's range. Demeaned within each site (fixed effects, so per-site offsets cancel), pooled across sites with ≥3 refit members spanning ≥4 m of range:

city	source	sites	members	slope (m per m)
paterson	GSV	4,433	19,588	+0.1159 ± 0.0021
gainesville	GSV	1,774	7,033	+0.1179 ± 0.0038
bend	GSV	9,388	42,240	+0.1082 ± 0.0013
richmond	Mapillary	890	7,040	+0.1275 ± 0.0058
sao_paulo	GSV	1,877	8,156	+0.0572 ± 0.0034

All 50–90σ. The discrepancy is not scatter — it is proportional to range, the signature of a scale error in d = camera_height / tan(depression), at roughly 11% almost everywhere.

At 20 m that is a ~2.3 m radial displacement, and because a street's cameras mostly sit along one line, it does not average out — it pushes detections consistently away from the roadway.

This is the same effect §0 of detection_recall_analysis.md already saw

That doc reports flat-ground geometry and DA3 metric depth agreeing to within 6.5–8.5%, and notes geometry placing four Richmond ramps above the horizon. Those were read as agreement plus rig noise. The multi-view result says the residual has structure: it is a proportional scale error, not a spread. A scale error is a one-constant fix; noise is not.

Two independent methods — an external monocular depth model, and pure self-consistency across views — now point the same way, and neither shares the other's failure modes.

What it is not

The obvious explanation is that the assumed camera height is too high (+0.116 implies ~2.33 m against the 2.5 m this repo assumes and the 2.6 m the labeler's geo.py assumes — worth reconciling in its own right). That hypothesis fails its own test. Re-fusing the labeler's runs over 2.10–2.60 m does lower raw within-site scatter at ~2.33 m, but lowering the assumed height shrinks every range proportionally, so raw metres shrink for free. Normalize scatter by the assumed height and 2.60 m is monotonically the best value tried, on both a GSV and a Mapillary city. So this is not simply the wrong constant.

The decisive experiment, using tooling that already exists here

The multi-view regression proves the disagreement is range-proportional but cannot say which end is right — it is anchored to a weighted mean of the same biased rays. scripts/analysis/depth_extract_da3.py can anchor it:

Regress DA3 metric depth on flat-ground distance for benchmark detections and look at the slope, not the correlation.

• Slope ≠ 1 → the geometry is biased. Fixable as a calibration; the distance buckets in detection_recall_analysis.md §1 and precision_by_distance.py are stretched, and the apparent-size relation W/d × (4096/2π) inherits it directly.
• Slope ≈ 1, multi-view trend persists → the geometry is fine and the peak placement drifts: the heatmap's argmax moves relative to the true ground-contact point as a ramp shrinks with distance. That is a model property, and it would sit with Extend RampNet output from points to boxes/polygons (point-prompted SAM2; CenterNet-style size head; segmentation) #83 (points → extent) — a size head or mask gives a bottom edge to raycast instead of a blob centre.

Splitting those two is cheap and needs no new data.

Why it is worth doing

• Distance-conditioned results. Recall by distance (0.842 / 0.879 / 0.812 / 0.564 across the buckets) is reported against an axis that may be stretched ~11%; bucket edges move.
• Position accuracy downstream. Auto-labeler world positions inherit it, and it is the largest single systematic we can currently name — median agreement with GT is ~0.8–1.6 m, so a 2 m bias at 20 m is not a rounding error.
• Mensuration (RampNet 2.0: measurement, condition inference, and tagging — we already collect the supervision and discard it at ingest #86, Extend RampNet output from points to boxes/polygons (point-prompted SAM2; CenterNet-style size head; segmentation) #83). Any metric quantity read off the ground plane — ramp width, dual-ramp separation, running slope — is multiplied by the same scale error. Calibrating it is a prerequisite, not a refinement.

Note the Mapillary/GSV split does not separate the hypotheses: richmond (Mapillary, varied rigs, SfM poses) shows the largest slope while sharing no camera geometry with GSV, which argues against a single rig constant and toward something shared — the model, or the flat-ground assumption itself. São Paulo, the hilliest split, is the outlier at half the slope, which is the opposite of what pure terrain noise would predict and deserves its own look.

Measurement scripts are in the auto-labeler scratchpad (heightbias.py, heightsweep.py); the method above is complete enough to reproduce from sites.jsonl alone.

🤖 Generated with Claude Code (claude-opus-5[1m])

[jonfroehlich]

The "What it is not" section here says the wrong-camera-height hypothesis fails its own test,
because sweeping the assumed height over 2.10–2.60 m never beats 2.60 once scatter is normalized.
That test is sound, but the conclusion drawn from it isn't quite right. There is no constant to
find — GSV's camera height is per-panorama, and it varies by more than a factor of two.

GSV serves its own depth, and it says so directly

streetlevel.streetview.find_panorama_by_id(pid, download_depth=True) still returns a metric depth
map — 160 of 160 panoramas I sampled across paterson / bend / gainesville / sao_paulo, capture
dates 2011 through 2026. 512×256, -1.0 as the sky sentinel.

It is internally consistent in the way that matters here. Since a ground point at depression θ sits
at euclidean range h / sin θ, the quantity depth · sin θ recovers the camera height — and on any
single panorama it is constant to 3–4 decimal places from 90° down to ~10° depression. Google's
maps are plane-fitted, so the road is genuinely one plane with a self-consistent metric scale. That
makes the per-panorama height directly readable rather than inferred.

Taking it as the median of depth · sin θ over the 20–60° band (road-dominated, clear of the nadir
seam and of the far field), 40 panoramas per city:

city	n	min	p25	median	p75	max	exactly 2.500
paterson	40	1.622	1.859	1.960	2.390	2.500	12%
bend	40	2.173	2.375	2.405	2.465	2.500	18%
gainesville	40	1.114	1.675	1.836	2.320	2.500	12%
sao_paulo	40	1.422	2.121	2.251	2.403	2.500	15%

14% return exactly 2.500 — a Google default rather than a measurement, and it needs filtering.
Excluding those: n=137, min 1.114, median 2.206, max 2.497.

The variation tracks capture vintage, non-monotonically, which is what a changing rig fleet looks
like:

capture year	n	median implied height
2016–2018	6	1.90
2020–2021	8	2.36
2023–2024	64	2.39
2025	32	1.891
2026	23	1.703

Why the sweep failed

Two reasons, and neither is "the geometry is fine":

1. A single constant cannot track a per-panorama quantity. The best global height for a city is
   just a summary of that city's vintage mix, so it will look mediocre against every alternative and
   never look right. That is precisely the signature the sweep produced.
2. Normalizing scatter by the assumed height, as the sweep does, cannot rescue it either — that
   correctly removes the free-shrinkage artefact the issue identifies, but a per-panorama error
   remains after any global rescaling. The sweep was measuring the wrong family of hypotheses.

How it compares to the +11% slope measured here

Directionally consistent, and the same order of magnitude: the labeler's assumed 2.6 m against a
pooled median of 2.206 m implies a ~18% range overestimate, versus the ~11% range-proportional
slope this issue measures. Per city it is mixed — bend lines up well (2.405 m implies 8%, measured
+10.8%), paterson does not (1.960 m implies 33%, measured +11.6%).

I would not read the per-city numbers as either confirmation or refutation, because the two
populations aren't matched: this survey strides through results.jsonl, so it samples processed
panoramas, while the slope regression is over sites with ≥3 refit members spanning ≥4 m of range —
which over-weights well-covered streets and therefore a different vintage mix. Matching the sampling
is a prerequisite for a real comparison, and now an easy one, since the height is readable per
panorama.

What this changes about the decisive experiment

The DA3 regression proposed here is still worth running, but its role should change. On GSV you no
longer need DA3 as the anchor — Google's own depth is the anchor, at 100% availability and
without the concern that DA3's metric scale is itself learned from training statistics and
unvalidated on this imagery. The higher-value use of DA3 is the reverse direction: calibrate DA3
against GSV depth on GSV, then carry the calibrated model to Mapillary, which serves no depth of
its own and is where the flat-ground assumption is worst (varied consumer rigs, the ρ = 0.81 vs 0.95
gap in detection_recall_analysis.md §0). That converts DA3 from a reference with an assumed bias
into one with a measured bias.

It also explains the four Richmond ramps that geometry placed above the horizon, structurally rather
than as rig noise: a per-panorama height and a fitted plane tilt move the horizon, and a plane
fitted from the depth map gives both.

Caveats worth carrying: 512×256 is ~0.7°/px and the map is a coarse planar approximation, so it is
reliable for a ramp on a road plane and least reliable near plane boundaries — which is where curbs
live. And the 1.114 m minimum in gainesville is either a genuinely low rig or a panorama whose
20–60° band isn't road, so the plane-fit residual should travel with the height and be used to
reject, not be assumed clean.

Related: label-latlng-estimation#4
(depth extraction and the ground-truth extension), and the geo.py constant filed against the
auto-labeler.

🤖 Generated with Claude Code (claude-opus-5[1m])