Port StarPilot vision speed limits to FrogPilot

cereal/custom.capnp

@@ -79,6 +79,7 @@ struct FrogPilotCarState @0xf35cc4560bbf6ec2 {
   pauseLongitudinal @11 :Bool;
   sportGear @12 :Bool;
   trafficModeEnabled @13 :Bool;
+  drivingGear @14 :Bool;  # Drive or low; defaults false for older publishers.
 }
 
 struct FrogPilotDeviceState @0xda96579883444c35 {
@@ -185,6 +186,7 @@ struct FrogPilotPlan @0xf98d843bfd7004a3 {
   vCruise @36 :Float32;
   weatherDaytime @37 :Bool;
   weatherId @38 :Int16;
+  slcVisionSpeedLimit @39 :Float32;
 }
 
 struct FrogPilotRadarState @0xb86e6369214c01c8 {

common/params_keys.h

@@ -410,6 +410,7 @@ inline static std::unordered_map<std::string, ParamKeyAttributes> keys = {
     {"SLCPriority", {PERSISTENT, STRING, "", "", 2}},
     {"SLCPriority1", {PERSISTENT, STRING, "Map Data", "Map Data", 2}},
     {"SLCPriority2", {PERSISTENT, STRING, "Dashboard", "Dashboard", 2}},
+    {"SLCPriority3", {PERSISTENT, STRING, "Vision", "Vision", 2}},
     {"SNGHack", {PERSISTENT, BOOL, "1", "0", 2}},
     {"SoundPack", {PERSISTENT, STRING, "frog", "stock", 0}},
     {"SoundToDownload", {CLEAR_ON_MANAGER_START, STRING, "", ""}},
@@ -476,6 +477,9 @@ inline static std::unordered_map<std::string, ParamKeyAttributes> keys = {
     {"VEgoStopping", {PERSISTENT, FLOAT, "0.0", "0.0", 3}},
     {"VEgoStoppingStock", {PERSISTENT, FLOAT, "0.0", "0.0", 3}},
     {"VeryLongDistanceButtonControl", {PERSISTENT, INT, "6", "0", 2}},
+    {"VisionSpeedLimit", {CLEAR_ON_MANAGER_START | CLEAR_ON_ONROAD_TRANSITION | CLEAR_ON_OFFROAD_TRANSITION, JSON, "{}", "{}"}},
+    {"VisionSpeedLimitDetection", {PERSISTENT, BOOL, "0", "0", 2}},
+    {"VisionSpeedLimitStatus", {CLEAR_ON_MANAGER_START | CLEAR_ON_ONROAD_TRANSITION | CLEAR_ON_OFFROAD_TRANSITION, STRING, "", ""}},
     {"VoltSNG", {PERSISTENT, BOOL, "0", "0", 2}},
     {"WarningImmediateVolume", {PERSISTENT, INT, "101", "101", 2}},
     {"WarningSoftVolume", {PERSISTENT, INT, "101", "101", 2}},

docs/vision_speed_limits.md

@@ -0,0 +1,127 @@
+# Vision speed limits
+
+Adds StarPilot's sign detector and number classifier, with independent PaddleOCR text
+verification, as an optional **Vision** source in FrogPilot's Speed Limit Controller.
+This port targets `MAKE-PRS-HERE` at `f8fb0668ed7fd13caf5a21e1ac27f6378df1b37f`;
+the default prebuilt distribution is not the source tree used for this change.
+
+## Use
+
+In Qt settings, select the Advanced tuning level and enable **Vision Speed Limits
+(U.S.)** under Speed Limit Controller. Set **Speed Limit Source Priority** to
+choose Vision's position relative to Map Data and Dashboard. Three slots are
+supported; existing first/second choices are preserved and the third defaults to
+Vision. Detection is off by default. Highest/Lowest modes use available sources.
+
+For display without longitudinal control, enable the same toggle under Visuals /
+Navigation Widgets and enable **Show Speed Limits**. Detection alone does not
+enable longitudinal control. Existing offsets, driver confirmations, gas overrides,
+cruise-speed caps and disengagement behavior remain in the controller. The optional
+source panel includes Vision.
+
+Supported readings are U.S. 5-80 mph signs in increments of 5. Metric displays
+convert those mph readings; display units never determine sign units.
+
+## Architecture
+
+- **Worker:** `speed_limit_vision` runs onroad when enabled, using OpenCV with two
+  CPU threads on device helper cores 0-2. Scheduling targets 6 Hz while scanning
+  and 10 Hz during two-second follow-ups, limited by measured inference cost and
+  CPU/memory pressure. These are scheduling targets, not measured device rates.
+  Processing backoff is 1.5 times inference duration normally and 1.0 during
+  follow-up. High thermal state or critical memory pressure clears the source.
+- **Inputs:** vehicle validity and drive/low gear come from `frogpilotCarState`,
+  preserving the full `carState` channel's 15 existing readers. The worker checks
+  payload validity and source age (0.1 seconds for vehicle state, 5 for device
+  state); its inference-limited receive frequency does not measure publisher health.
+  It prefers road camera and falls back to wide road. NV12 decoding respects stride
+  and UV-plane offset. Buffer layout and timestamps establish frame acceptance
+  because this camera producer does not populate VisionIPC's `valid` flag. Frames
+  older than 0.5 seconds, duplicate timestamps, invalid/offroad state and non-driving
+  gears cannot confirm a sign.
+- **Recognition:** model hashes and output contracts are checked. Up to four
+  non-overlapping proposals receive bounded crop reads; conflicting numbers reject
+  the result. Agreeing crops use the lowest classifier score without a crop-count
+  bonus or a weighted detector score; this is not calibrated recognition accuracy.
+  A color filter rejects advisory signage and adjusts brightness with
+  bounded gain for shadows. Tinted panels require stronger detector/classifier
+  scores. A distinct yellow header above a white panel rejects the observed school/
+  conditional layout before crop expansion; uniformly tinted signs remain eligible.
+  This does not detect every condition or its activation status.
+  Glare-rejected panels additionally require an exact `SPEED LIMIT` heading,
+  with at most two OCR alignments per proposal. Every accepted number, including on
+  neutral white panels, must also match a separate OCR digit read. This uses at most
+  two aspect-preserving crops and requires confidence for each digit. A confident
+  disagreement rejects the proposal. OCR verifies the classifier's number rather
+  than supplying a replacement. Automatic increases and driver settings are preserved.
+  See [model provenance and preparation](../frogpilot/assets/vision_models/README.md).
+- **Confirmation:** two matching, separately captured frames within two seconds are required,
+  using camera capture times. Crop variants do not count as separate observations.
+  Changes below 30 mph from 30 mph or above require at least 0.90 confidence. A
+  confirmed match ends follow-up; weak reads cannot start it. Strong numeric
+  candidates with unreadable headings can request a two-second retry followed by a
+  two-second cooldown, without publishing a limit. Confirmed speed changes are logged.
+- **Freshness:** one JSON value in shared `Params(memory=True)` carries m/s,
+  confidence, detection time and last processed frame time. The SLC independently
+  expires results after 300 seconds without matching observations or three seconds
+  without a new processed frame. Republishing extends neither deadline. Live frames
+  without a sign preserve the held limit until expiry. This five-minute ceiling is
+  a bounded retention policy, not proof that the sign still applies. Fresh matched
+  map segment/direction changes and camera switches clear confirmation; road names
+  do not identify a segment. A match must have a loaded tile, positive way ID and a
+  location timestamp no more than two seconds old. Without a usable map match, the
+  worker cannot detect road transitions; time and camera expiry still apply.
+- **Controller:** expired/disabled Vision cannot survive through Previous Limit
+  fallback, and Vision readings are never persisted as `PreviousSpeedLimit`.
+  Source handovers compare against the last accepted target, so losing Vision
+  cannot bypass higher/lower confirmation. Pending decisions and queued taps are
+  tied to their source. Valid replacements still increase automatically when
+  confirmation is disabled. With no accepted replacement, an engaged controller
+  keeps the last Vision cruise ceiling while the displayed reading becomes
+  unavailable. A new accepted limit, disengagement, disabling SLC or an explicit
+  cruise increase releases that ceiling; gas overrides retain their selected mode.
+  Curve speed reductions do not become permanent Vision ceilings.
+
+## Validation
+
+With this repository's Linux development dependencies, build the native targets
+and run the focused checks:
+
+```sh
+scons --minimal -j4 common/params_pyx.so \
+  msgq_repo/msgq/ipc_pyx.so msgq_repo/msgq/visionipc/visionipc_pyx.so \
+  selfdrive/pandad/pandad_api_impl.so \
+  selfdrive/controls/lib/longitudinal_mpc_lib/c_generated_code/acados_ocp_solver_pyx.so \
+  selfdrive/ui/ui
+pytest -n0 frogpilot/system/tests/test_speed_limit_vision.py \
+  frogpilot/controls/tests/test_vision_speed_limit_controller.py common/tests/test_params.py
+ruff check frogpilot/common/vision_speed_limit.py \
+  frogpilot/system/speed_limit_vision.py frogpilot/system/vision_speed_limit_model.py \
+  frogpilot/system/vision_speed_limit_text.py \
+  frogpilot/system/tests/test_speed_limit_vision.py \
+  frogpilot/controls/tests/test_vision_speed_limit_controller.py
+uv lock --check
+```
+
+Recorded results are kept in [the validation record](vision_speed_limits_validation.md).
+
+## Limitations and review items
+
+- Model [licensing and provenance](../frogpilot/assets/vision_models/README.md) remain
+  an upstream inclusion question. The StarPilot weights declare AGPL-3.0; retaining
+  notices does not resolve corresponding-source obligations or license acceptance.
+- The models cannot establish lane applicability or whether a conditional school
+  or construction limit is active. Matching numbers in two frames do not establish
+  physical sign identity or make their errors statistically independent. Recognition
+  errors and missed signs are possible.
+  This is not a European/Canadian sign model. Python/Mici settings and source display
+  are not implemented or validated by this Qt port.
+- Broader labeled-route accuracy, sustained control/model deadlines, thermal/memory
+  headroom, camera restarts and drive transitions remain unestablished. A timing
+  simulation missed the brief 40 mph sign in 10/40 starting phases at 1.5 times the
+  estimated inference cost. Short hardware checks and road feedback do not establish
+  general reliability or sustained inference rates.
+
+Detailed investigation history is retained in the
+[PR discussion](https://github.com/FrogAi/FrogPilot/pull/321#issuecomment-5754480633). Private route recordings
+and device diagnostics are not distributed with the port.

docs/vision_speed_limits_validation.md

@@ -0,0 +1,75 @@
+# Vision speed limit validation record
+
+Results below cover the runtime at `ba5d6469dfa4da0598f3365063e3ac8fcf4a6a94`,
+tested on 2026-09-21. These are development checks, not a general accuracy or
+safety claim. Private recordings are not distributed with the port.
+
+## Automated and recorded checks
+
+- 284 checks passed on the Linux host and native ARM comma 3. They cover final
+  cruise behavior across source expiry, higher/lower/equal replacements,
+  confirmation, denial, Mapbox handover, driver overrides, curve interaction and
+  disengagement, as well as model contracts, worker freshness and native Params.
+  Changed Python passes Ruff; the dependency lock and diff checks pass.
+- A paired comparison covered 963 development frames: 858 samples from ten drive
+  clips, including complete known sign approaches; 101 incident frames; two public
+  school-sign frames; and two later nighttime school-sign frames. The final model
+  retained all 56 ordinary accepted readings and rejected the four conditional
+  readings. No other acceptance changed. These are repeated views of a few signs,
+  not 56 independently validated signs.
+- Component-removal comparisons showed why the remaining bounded fallbacks stay:
+  removing crop expansions lost three accepted frames, tint support lost 12, and
+  the glare fallback lost 22. One heading alignment lost one, and one digit
+  alignment lost two. Removing the weighted confidence/crop bonus in favor of the
+  lowest agreeing classifier score preserved the sampled readings. These
+  development comparisons do not calibrate the scores or establish unseen-route
+  accuracy.
+- Full two-minute host replay supplied all 2,400 camera frames through native
+  padded-NV12 VisionIPC, the worker, Params and FrogPilot's cruise controller, with
+  recorded message timestamps rebased consistently. No frame was over 100 ms late.
+  Accepted limits were only 30 mph or unavailable. The maximum engaged cruise
+  target during the incident window was 34.461 mph. After camera expiry the
+  reading was zero and the engaged target remained 34.797 mph, without jumping to
+  the higher stored cruise setting. Disengaged targets are not acceleration commands.
+
+## Device preparation
+
+The staged native `scons --minimal -j2` build passed in 346 seconds, followed by
+model startup and the ARM checks above. The installed checkout remained untouched
+during preparation. No new schema, C++, UI or dependency changes were needed for
+this cleanup; staging still rebuilt native model artifacts for its new path.
+
+The C3 processed 26 hash-checked recorded NV12 fixtures through its actual worker
+and controller. Ordinary 30→40→30 changes passed; two route shields, all 16
+previously false-70 frames and two conditional-header frames retained the held
+30 mph limit. Camera-stale expiry passed. Inference took 0.19–0.73 seconds.
+
+Physical-camera/driving-model/calibration coexistence also passed: 60 seconds
+with Vision off and 180 seconds on, excluding the first ten seconds of each phase
+from message counts. The 50/170-second measurement windows contained 1,001/3,401
+valid model and odometry messages and 200/680 valid calibration messages, with no
+invalid messages. Enabled inference samples took 0.64–0.75 seconds. The phase-end
+device reports were thermally green, with 58%/60% memory use. This bounded desk
+exercise does not establish sustained road-control deadlines or thermal limits.
+
+Full-rate HEVC replay on the C3 is not a passing result: the earlier software
+decoder could not provide 20 FPS. Temporal replay runs on the host; device checks
+use decoded recorded frames and physical cameras.
+
+## Remaining limits
+
+The five-minute observation ceiling is a retention policy. Fresh matched map
+segment/direction changes now clear it, including unnamed roads. Without a usable
+map match, road transitions remain unobserved. Physical sign tracking is not
+implemented: overlap alone would reject the known moving 40/30 mph approaches.
+
+Both the old and simplified score could read 20 mph on a school sign without
+knowing whether its condition was active. Visible yellow-header layouts are now
+rejected, including the public fixtures that previously tested only the printed
+number. This does not solve other conditional layouts or lane applicability.
+
+Broader independently labeled routes, missed-sign rates, sustained control
+deadlines, thermal limits and road validation remain open. Model licensing and
+maintainer acceptance are separate upstream prerequisites. Earlier results and
+the 30→70 incident investigation remain in the Git/PR history; an earlier positive
+drive report is not evidence that these limits have been resolved.