GSV serves a native per-pano depth map (streetlevel already parses it): an independent anchor for #101 and the visibility test #102 needs — archive it before the endpoint changes

[jonfroehlich]

GSV serves a native depth map alongside every official pano, and streetlevel — already pinned in environment.lock.yml (0.12.3) — parses it: find_panorama_by_id(pano_id, download_depth=True) returns pano.depth, a DepthMap with a numpy array of meters (−1 where there is no surface, i.e. horizon/sky), and the same object carries the full pose the depth is registered to (heading/pitch/roll in radians, elevation in meters, lat/lon).

This is not a new capability we lack — we already built a metric-depth instrument (scripts/analysis/depth_extract_da3.py, results in docs/detection_recall_analysis.md). What the native map adds is specific: it is free at any scale (a KB-scale metadata blob per pano, no GPU, no rendering), it is registered to the pano's own pose, and its failure modes are disjoint from both instruments we have. Two open issues are each missing exactly one of those properties.

What the data actually is

It is coarse and surface-model-shaped, not photogrammetric. The library author's read: "Appears to be a synthetic depth map created from elevation data and building footprints." Taking that at face value (worth verifying on a hilly city as part of the first probe):

• Ground range at a pixel: yes. Depth at a ramp's pixel is the ray-to-terrain intersection — which is precisely the flat-ground formula d = camera_height / tan(depression) with our two assumptions (flat plane, 2.5 m) replaced by Google's terrain model and Google's camera model.
• Structures: yes. Small occluders: no. Buildings and terrain are in the model; parked vans, hedges, and poles almost certainly are not.
• Ramp-scale geometry: no. It locates and scales; it cannot measure running slope or surface shape.
• Official coverage only. Third-party panos and Mapillary are out; richmond stays DA3-only.

An independent anchor for #101

#101 measured a ~11% range-proportional scale error in flat-ground distance via multi-view self-consistency, and proposed DA3 as the anchor to decide between its two hypotheses (geometry bias vs. heatmap peak drift). The native map is a second anchor with different failure modes — flat-ground bakes in our camera-height and flat-plane assumptions, DA3 brings monocular network priors, the native map brings Google's pose and terrain model. And because it shares the ray with the flat-ground estimate while replacing the ground, the regression of native depth on flat-ground distance is unusually clean:

• Slope ≈ 1.11 → the geometry is biased, confirmed by an instrument that shares no assumptions with DA3; the calibration constant falls out directly, and the distance axes in detection_recall_analysis.md §1 get corrected.
• Slope ≈ 1 while the multi-view trend persists → the rays are fine and the peak placement drifts with apparent size — a model property, pointing at Extend RampNet output from points to boxes/polygons (point-prompted SAM2; CenterNet-style size head; segmentation) #83 (points → extent).

It covers four of #101's five cities (bend, paterson, gainesville, sao_paulo — all but richmond). São Paulo is also #101's flagged outlier (half the slope, hilliest split): a terrain-aware instrument tests the terrain explanation directly.

The visibility test #102 calls non-optional

#102's mined-yield table uses proximity as a stand-in for visibility, and names occlusion as the accuracy ceiling: labelling an occluded non-detection as a miss teaches the model to hallucinate ramps behind obstructions. The native map gives the obvious test — march the ray from a candidate pano toward the fused site; if the depth surface intersects well short of the expected range, the site was not visible, don't mine.

Honest limit, per the synthetic-from-footprints read: this catches buildings, walls, and terrain, not vehicles or vegetation. It is a necessary filter, not a sufficient one — but it tightens #102's upper bound for the cost of a metadata fetch, and it composes with #102's proposed mined-precision check against the judged panos (which measures whatever occlusion slips through).

Replacing the flat-ground lift at the source (sidewalk-auto-labeler#27)

Fusion currently lifts each member detection with the flat-ground formula — that is where #101's bias enters. On GSV panos the lift can take measured range instead. Two properties matter beyond accuracy: the scale is consistent across panos (one reconstruction), where per-image monocular depth wobbles exactly in the way that hurts cross-view clustering; and it removes the systematic error rather than calibrating it post hoc. This is the labeler's change, not this repo's, but the instrument validation above is the prerequisite for it.

Smaller uses, one cheap probe each

• Ground-vs-vertical FP gate. benchmark/model_detections/ is committed; check each RampNet FP at the 0.30 operating point against the surface under it. The Lower the deployment operating point (peak threshold 0.55 → ~0.25–0.35): a free, recall-first lever #54 operating-point change bought +7.7 R for −4.8 P — if a meaningful share of the new FPs sit off-ground, part of that precision comes back for free. (Expectation management: detection_recall_analysis.md §2 found precision flat with distance, so this may well return "nothing off-ground" — that is a result too.)
• Above-horizon rescue. Geometry returns infinity for uphill ramps; DA3 rescued 4 in richmond at GPU cost. On GSV splits the native map does this for free.
• Distance for every training label. Distance-conditioned target σ (the far-field finding in Qualitative failure taxonomy: bucket each model's FP/FN from cached detections #46 was σ+calibration, not vocabulary) needs per-label range at Stage 1 scale — a real GPU job under DA3, a metadata fetch here.
• Mensuration scale (RampNet 2.0: measurement, condition inference, and tagging — we already collect the supervision and discard it at ingest #86). Any width/separation read off the ground plane inherits Flat-ground distance carries a ~11% range-proportional scale error, not just scatter — multi-view self-consistency measures it and DA3 can anchor it #101's scale error; this is the cheap anchor for it. The depth itself cannot measure the ramp — see above.

The endpoint can change; archive first

This rides the same unofficial metadata endpoint family as the tile fetches, which have already changed behavior once (the paper-era fetch path died when the tile server started rejecting default user agents). Every experiment above survives that only if the payloads are archived now:

1. Tier 1 — judged benchmark panos (~1.3k across the GSV splits + manual gold): an afternoon, KB-scale blobs. Archive the raw payloads verbatim (primary source), content-hashed per the replication rule, with a committed decoder — not just decoded arrays. Can ride along with the Publish deployment validation ground-truth as a HuggingFace dataset (Bend GSV + Richmond Mapillary) #21 HF upload.
2. Tier 2 — the auto-labeler run panos for the four GSV fusion cities (tens of thousands; needed for the Flat-ground distance carries a ~11% range-proportional scale error, not just scatter — multi-view self-consistency measures it and DA3 can anchor it #101 regression at the scale the multi-view measurement ran, and for the Mine hard positives from multi-view consensus: a training-label source that does not depend on city inventories (#96, #59) #102 visibility test).
3. Tier 3 — the Stage 1 training set, if the σ experiment goes ahead.

The harvest is itself an experiment: depth availability rate by city and capture era is unknown (official-coverage-only, and older panos may lack it). Whatever the rate is, it is a result to record — a split with poor availability caps everything above, and that gap should be stated next to any number derived from the rest.

Proposed order

1. Tier-1 harvest + availability numbers.
2. Native-vs-flat-ground slope regression on benchmark GT and detections; three-way with DA3 where it exists (bend). This is the Flat-ground distance carries a ~11% range-proportional scale error, not just scatter — multi-view self-consistency measures it and DA3 can anchor it #101 anchor.
3. Re-run Mine hard positives from multi-view consensus: a training-label source that does not depend on city inventories (#96, #59) #102's mined-precision experiment on one GSV city with the visibility test in the loop.
4. FP ground-gate probe.
5. Fusion lift replacement, in the labeler (blocked on 2).

🤖 Generated with Claude Code (claude-fable-5)