You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
ProjectSidewalk/sidewalk-panorama-tools#54 documents a defect in the coordinate every Project Sidewalk pipeline crops around: stored pano_x/pano_y carry an uncorrected rig-tilt term. The root-cause diagnosis is ProjectSidewalk/SidewalkWebpage#4784 — povToPanoCoord corrects x by cameraHeading but maps y as pure linear pitch, so per-pano rig pitch/roll never enters, and the residual lands in the stored pano_y.
That issue names RampNet as one of three live consumers, but deliberately scopes itself to the canonical cropper. This is the RampNet half: where it enters here, how large it is in our units, and what it costs. Filing it now because 2.0 regenerates the crop-training path anyway — the fix should ride that regeneration instead of becoming a migration afterwards.
Where it enters
One place, and not the one #54 guessed. stage_one/dataset_generation/ never touches Project Sidewalk coordinates — it works from government GPS, and its pano_x/pano_y are projected UTM metres, not label pixels. (#54 says otherwise; that detail is wrong and I will correct it there.) The contamination enters one step upstream, in the crop model's training data:
stage_one/crop_model/ps_model/data/download_data.py:259-274 reads Panorama X / Panorama Y from /v2/rawLabels across 12 Project Sidewalk deployments, then:
stitches the pano to 8192×4096 (fetch_panorama);
picks a yaw snapped to 30° (nearest_theta = round(theta / 30) * 30), derived from Panorama X;
renders a 90°-FOV 2048×2048 perspective view at a fixed −30° pitch;
projects the label into that view and keeps the middle third (683×2048);
writes the projected point into the filename.
ps_model/model/train.py turns that point into a Gaussian heatmap target (σ = 12 on a 256×88 grid, from a 1024×352 input). From there it propagates: round-1 crop model → round-2 fine-tune → dataset_generation/inference_isolator.py, which is what places every point in the 214,376-pano / 849,895-label Stage 1 dataset → Stage 2's heatmap targets.
Our failure mode differs from CropRunner's, and is worse in one respect. CropRunner centers a shrinking box on the label, so #54's headline symptom is the object walking toward the crop edge. RampNet never centers on the label — it renders a fixed strip and records where the label lands inside it. The object therefore always stays in frame; what moves instead is the supervision target, off the object it is supposed to mark. Nothing about the crop looks wrong.
How big it is, in our units
Per degree of rig tilt:
stage
px per degree
stitched 8192×4096 pano (pano_height / 180)
~22.8
2048×2048 perspective render (f = 1024 px/rad on axis, ×sec²ψ off it; ψ ≈ 23° for a ramp ~20 m out)
~18–21
1024×352 model input
~9–10
256×88 heatmap target, against σ = 12
~2.2–2.6, i.e. ≈0.2 σ
At typical rig tilts of 1–3°, the target sits 0.2–0.6 σ off the object. Sub-σ — which degrades supervision rather than destroying it, and is exactly why this has never announced itself.
Two properties make it hard to catch:
The error is tilt · cos(heading − tilt_axis), so it is zero-mean over headings. No aggregate check finds it and no constant offset fixes it; only a per-pano, heading-resolved test sees it. It presents as variance in the supervision — a broadened effective target — not as a bias.
The 30° yaw snap does not help. Labels occur at all headings, so the sinusoid is sampled in full.
For downstream calibration: the pano matching radius is PANO_RADIUS_NORMALIZED = 0.022 against scale_y = 512, i.e. 22.5 units ≈ 7.9° of latitude. So 2° of tilt is about a quarter of the vertical match tolerance.
Two things partly absorb it, both unquantified. Round 2 fine-tunes on manually placed crop labels, which carry no Project Sidewalk convention — how much that re-centers the round-1 bias is unknown. And Stage 2 never sees PS coordinates directly; it inherits whatever placement the crop model learned. Neither is a reason to skip the measurement. Both are reasons it has to run end-to-end rather than stop at the crop.
What we already know about the other half, and why it settles less than it looks
#54 flags that which frame the stitched panos are in cannot be assumed. This repo holds partial evidence, and it is reassuring for GSV: docs/curb_ramp_data_sourcing.md reports ground-truth ramps placed at or above the horizon — geometrically impossible for a ground ramp, hence a direct tell of an unleveled rig — at 0 of 994 GSV ramps, 5 of 1,066 Mapillary (0.5%), and 12 of 300 on Budapest (4%, consumer rig). That is consistent with GSV equirectangulars being close to gravity-rectified, matching ProjectSidewalk/sidewalk-auto-labeler#42.
But that test is far weaker than it looks at the tilts that matter. It only fires when tilt exceeds the ramp's own depression angle, atan(camera_height / d) — roughly 1.4° at 100 m, 2.9° at 50 m, and 7° at 20 m. A uniform 2° tilt would leave 0 of 994 untouched. "0 of 994" bounds gross rig error; it says nothing about the 1–3° range this issue is about.
And if the stitched GSV pano is gravity-rectified while the stored pano_y is not tilt-corrected, then the two are in different frames and the crop-supervision error is real at full magnitude. That is precisely the frame bookkeeping #54 says must be settled rather than assumed — and RampNet is the pipeline where settling it has the largest downstream footprint.
Every distance number in this repo comes from d = camera_height / tan(depression), a direct function of pixel row. Residual tilt is therefore a first-order distance error that grows with range: for small angles d' ≈ d · (1 + ε · d / h), so 1° of tilt is ~8% at 10 m and ~17% at 20 m.
Worth stating precisely: that is quadratic in d, not the range-proportional scale error #101 measures at ~11%, and not obviously #112's 25–30% stretch either. So tilt is a candidate contributor with a distinguishable functional form — useful to those issues as a discriminator rather than as a competing explanation. above_horizon() in scripts/analysis/miss_decomposition.py already names the unleveled rig as the bias it guards against.
Scope
Measurement — shares an instrument with #54, so run it once and report both:
Fetch per-pano pitch/roll via streetlevel for a sample of CurbRamp labels across the 12 deployments, stratified by tilt magnitude
Settle the frame question for our stitch specifically: fetch_panorama assembles raw tiles from the GSV tile endpoint directly, not through streetlevel, so the auto-labeler#42 rectification finding does not transfer for free
Measure crop-model target displacement as a fraction of σ, resolved by heading — the prediction is a zero-mean sinusoid with amplitude ∝ tilt
Report whether the round-2 manual fine-tune measurably re-centers it
The architectural half — one cropper, not four
RampNet carries a private fetch-and-project implementation (fetch_panorama, equirectangular_to_perspective, equirectangular_point_to_perspective) that duplicates what sidewalk-panorama-tools exists to provide. It has already forked once inside this repo: the paper-era fetch path stopped working (the tile endpoint 403s requests' default user agent), and the production copy had to be lifted into rampnet/gsv.py on the #103 branch (f5d8ca9) just to get one importable definition. A second cropper in a second repo is the same drift with a longer feedback loop.
For 2.0, consume the canonical cropper from sidewalk-panorama-tools rather than maintaining ours — the tilt correction then arrives as a dependency update instead of a port
Keep what is genuinely RampNet-specific (the fixed-pitch strip convention) as a thin layer over that cropper's projection, not a parallel implementation of it
A null is a real result. If our stitch and the stored pano_y are in the same frame for the eras we use, the crop supervision is clean and this reduces to the consolidation task alone.
Correcting the supervision means regenerating the crop-model training set and re-running the round-1 → round-2 chain. That is affordable exactly once — while 2.0 is regenerating it anyway — which is the reason to file this now rather than after.
Why
ProjectSidewalk/sidewalk-panorama-tools#54 documents a defect in the coordinate every Project Sidewalk pipeline crops around: stored
pano_x/pano_ycarry an uncorrected rig-tilt term. The root-cause diagnosis is ProjectSidewalk/SidewalkWebpage#4784 —povToPanoCoordcorrects x bycameraHeadingbut maps y as pure linear pitch, so per-pano rig pitch/roll never enters, and the residual lands in the storedpano_y.That issue names RampNet as one of three live consumers, but deliberately scopes itself to the canonical cropper. This is the RampNet half: where it enters here, how large it is in our units, and what it costs. Filing it now because 2.0 regenerates the crop-training path anyway — the fix should ride that regeneration instead of becoming a migration afterwards.
Where it enters
One place, and not the one #54 guessed.
stage_one/dataset_generation/never touches Project Sidewalk coordinates — it works from government GPS, and itspano_x/pano_yare projected UTM metres, not label pixels. (#54 says otherwise; that detail is wrong and I will correct it there.) The contamination enters one step upstream, in the crop model's training data:stage_one/crop_model/ps_model/data/download_data.py:259-274readsPanorama X/Panorama Yfrom/v2/rawLabelsacross 12 Project Sidewalk deployments, then:fetch_panorama);nearest_theta = round(theta / 30) * 30), derived fromPanorama X;ps_model/model/train.pyturns that point into a Gaussian heatmap target (σ = 12 on a 256×88 grid, from a 1024×352 input). From there it propagates: round-1 crop model → round-2 fine-tune →dataset_generation/inference_isolator.py, which is what places every point in the 214,376-pano / 849,895-label Stage 1 dataset → Stage 2's heatmap targets.Our failure mode differs from CropRunner's, and is worse in one respect. CropRunner centers a shrinking box on the label, so #54's headline symptom is the object walking toward the crop edge. RampNet never centers on the label — it renders a fixed strip and records where the label lands inside it. The object therefore always stays in frame; what moves instead is the supervision target, off the object it is supposed to mark. Nothing about the crop looks wrong.
How big it is, in our units
Per degree of rig tilt:
pano_height / 180)At typical rig tilts of 1–3°, the target sits 0.2–0.6 σ off the object. Sub-σ — which degrades supervision rather than destroying it, and is exactly why this has never announced itself.
Two properties make it hard to catch:
tilt · cos(heading − tilt_axis), so it is zero-mean over headings. No aggregate check finds it and no constant offset fixes it; only a per-pano, heading-resolved test sees it. It presents as variance in the supervision — a broadened effective target — not as a bias.For downstream calibration: the pano matching radius is
PANO_RADIUS_NORMALIZED = 0.022againstscale_y = 512, i.e. 22.5 units ≈ 7.9° of latitude. So 2° of tilt is about a quarter of the vertical match tolerance.Two things partly absorb it, both unquantified. Round 2 fine-tunes on manually placed crop labels, which carry no Project Sidewalk convention — how much that re-centers the round-1 bias is unknown. And Stage 2 never sees PS coordinates directly; it inherits whatever placement the crop model learned. Neither is a reason to skip the measurement. Both are reasons it has to run end-to-end rather than stop at the crop.
What we already know about the other half, and why it settles less than it looks
#54 flags that which frame the stitched panos are in cannot be assumed. This repo holds partial evidence, and it is reassuring for GSV:
docs/curb_ramp_data_sourcing.mdreports ground-truth ramps placed at or above the horizon — geometrically impossible for a ground ramp, hence a direct tell of an unleveled rig — at 0 of 994 GSV ramps, 5 of 1,066 Mapillary (0.5%), and 12 of 300 on Budapest (4%, consumer rig). That is consistent with GSV equirectangulars being close to gravity-rectified, matching ProjectSidewalk/sidewalk-auto-labeler#42.But that test is far weaker than it looks at the tilts that matter. It only fires when tilt exceeds the ramp's own depression angle,
atan(camera_height / d)— roughly 1.4° at 100 m, 2.9° at 50 m, and 7° at 20 m. A uniform 2° tilt would leave 0 of 994 untouched. "0 of 994" bounds gross rig error; it says nothing about the 1–3° range this issue is about.And if the stitched GSV pano is gravity-rectified while the stored
pano_yis not tilt-corrected, then the two are in different frames and the crop-supervision error is real at full magnitude. That is precisely the frame bookkeeping #54 says must be settled rather than assumed — and RampNet is the pipeline where settling it has the largest downstream footprint.Adjacent: the distance axis (#101, #112)
Every distance number in this repo comes from
d = camera_height / tan(depression), a direct function of pixel row. Residual tilt is therefore a first-order distance error that grows with range: for small anglesd' ≈ d · (1 + ε · d / h), so 1° of tilt is ~8% at 10 m and ~17% at 20 m.Worth stating precisely: that is quadratic in
d, not the range-proportional scale error #101 measures at ~11%, and not obviously #112's 25–30% stretch either. So tilt is a candidate contributor with a distinguishable functional form — useful to those issues as a discriminator rather than as a competing explanation.above_horizon()inscripts/analysis/miss_decomposition.pyalready names the unleveled rig as the bias it guards against.Scope
Measurement — shares an instrument with #54, so run it once and report both:
streetlevelfor a sample ofCurbRamplabels across the 12 deployments, stratified by tilt magnitudefetch_panoramaassembles raw tiles from the GSV tile endpoint directly, not throughstreetlevel, so the auto-labeler#42 rectification finding does not transfer for freeThe architectural half — one cropper, not four
RampNet carries a private fetch-and-project implementation (
fetch_panorama,equirectangular_to_perspective,equirectangular_point_to_perspective) that duplicates whatsidewalk-panorama-toolsexists to provide. It has already forked once inside this repo: the paper-era fetch path stopped working (the tile endpoint 403srequests' default user agent), and the production copy had to be lifted intorampnet/gsv.pyon the #103 branch (f5d8ca9) just to get one importable definition. A second cropper in a second repo is the same drift with a longer feedback loop.sidewalk-panorama-toolsrather than maintaining ours — the tilt correction then arrives as a dependency update instead of a portHonest limits
pano_yare in the same frame for the eras we use, the crop supervision is clean and this reduces to the consolidation task alone.Related
🤖 Generated with Claude Code (claude-opus-5[1m])