Port StarPilot vision speed limits to FrogPilot

[AppleExpl01t]

Adds camera sign recognition as an optional Vision source in FrogPilot's existing Speed Limit Controller. The controller continues to own source priorities, offsets, confirmations, driver overrides and cruise caps. Detection is off by default; Qt supports control and display-only use. Recognized U.S. values are mph regardless of display units.

Draft. A nighttime drive exposed a repeated 30→70 mph misread that raised the cruise target. Every accepted classifier number now requires agreement from the separate OCR model, followed by matching observations in two captured frames. Automatic increases remain available when confirmation is disabled.

The revised integration also fixes two source-loss behaviors: an expired Vision reading cannot bypass confirmation when handing over to a higher map limit, and losing the reading alone cannot accelerate toward a higher stored cruise setting. The displayed reading becomes unavailable while the engaged controller retains its last Vision cruise ceiling until an accepted replacement or the documented driver actions release it.

Recognition rejects the observed yellow-header school-sign layout rather than assuming its condition is active. It uses the lowest agreeing classifier score, without a weighted detector score or a crop-count bonus. This score is not calibrated accuracy. Measured crop/tint/glare/OCR fallbacks remain bounded. The worker has explicit input preparation, frame processing and clock ownership; shared test helpers and separate setup/validation documents reduce duplication. Fresh map segment/direction changes replace road-name-only clearing.

Validation

• 284 checks passed on both the Linux host and native ARM C3. Changed-Python Ruff, dependency-lock and diff checks passed.
• A paired 963-frame development comparison preserved all 56 ordinary accepted readings and rejected four school-header readings; no other acceptance changed. These are repeated views of a few signs, not an independent accuracy benchmark.
• Full two-minute host replay supplied 2,400 frames through native VisionIPC, Params and FrogPilot cruise control. It selected only 30 mph or unavailable, kept the engaged incident target at or below 34.461 mph, and retained a 34.797 mph cruise ceiling after camera expiry.
• The C3 passed all 26 recorded NV12 fixtures: 30→40→30 recognition, route-shield rejection, all 16 false-70 frames, two school-header frames and stale-camera expiry. Inference took 0.19–0.73 seconds.
• Native builds and physical-camera/model/calibration coexistence passed. After warmup, 50/170-second Vision-off/on windows produced 1,001/3,401 valid model and odometry messages and 200/680 valid calibration messages, with zero invalid messages. Enabled inference samples took 0.64–0.75 seconds; phase-end thermal status was green and memory use was 60% with Vision enabled.

Installed and reboot-verified on the C3 as VSL-Port at b7827177d. Native artifact hashes and the running Panda firmware signature matched, Qt and expected services were running, and all 21 checked tuning/SLC settings were preserved. The post-reboot offroad check recorded zero invalid device, manager or Panda messages. This is desk verification; no new road test is claimed.

The tested runtime is ba5d6469dfa4da0598f3365063e3ac8fcf4a6a94; b7827177d42637cc43942f817c4b08deed2d1cd1 changes only its documentation and also passed the staged native build. Setup and behavior and validation details describe the evidence and limits.

Broader labeled-route accuracy, missed signs, physical sign identity, lane/conditional applicability, sustained control deadlines and road validation remain open. The header guard handles an observed layout; it does not determine conditional activation. Full-rate HEVC replay on C3 was not validated because software decoding could not provide 20 FPS. Python/Mici UI integration is outside this Qt port. Private recordings are not included.

Credit and licensing

Credit to firestar5683 and the StarPilot contributors for the detector/classifier and original preprocessing, and PaddlePaddle/PaddleOCR for text recognition. Models are pinned and license notices are retained. The StarPilot weights declare AGPL-3.0; licensing/provenance and corresponding-source acceptance remain upstream prerequisites. See model provenance and licenses. Cleanup and test results do not imply maintainer approval.

[FrogAi]

Thank you for your PR! If you're not already in the FrogPilot Discord, feel free to join and let me know you've opened a PR!

[AppleExpl01t]

Detailed VSL investigation and validation history

The feature document is being condensed to usage, architecture, reproduction commands, current validation and limitations. This preserves the detailed investigation history from the previous documentation revision for reviewer reference. These are the existing September 19-20 results; the documentation cleanup does not change runtime code or rerun those experiments.

Host, replay and C3 investigation records (2026-09-19 to 2026-09-20)

Host validation record (2026-09-19)

Linux x86-64 under WSL, Python 3.12.3, OpenCV 4.11.0:

• 142 tests passed: 129 feature tests and 13 existing Params tests, using the
  repository pytest configuration and native parameter bindings.
• The parameter/messaging/VisionIPC bindings compiled, and the complete Qt UI
  compiled and linked with the repository SCons configuration.
• An isolated 2160x1080 Qt preview confirmed the Vision toggle and source selection
  dialogs render, all three priorities save correctly, and cancelling the second
  dialog preserves the existing order. This is a desktop settings check, not a
  complete device image or on-device UI validation.
• New modules/tests and the edited controller/process configuration pass Ruff.
  The variables module has 56 pre-existing diagnostics; comparison against the
  base revision found no new diagnostics in changed existing Python files.
• uv lock --check --offline and git diff --check passed. Model checksums match
  the pinned StarPilot files; positive frame inference and blank-image rejection
  ran using the actual ONNX weights.

These host results do not establish comma hardware performance, labeled route
accuracy, native Mici UI support, or license clearance.

C3 investigation and follow-up validation (2026-09-19)

Initial vehicle testing of 4ff97619 on a comma 3 running AGNOS 12.8 exposed
communication errors. The full process stack exceeded carState's 15-reader
capacity when the vision worker subscribed, repeatedly evicting readers and
invalidating calibration and downstream outputs. Temporarily pausing only the
vision worker restored their validity. Isolated input replay did not reproduce
the failure because it did not have the full stack's reader count.

The worker now consumes frogpilotCarState, which carries the same CAN-valid flag
and a drive/low gear flag. No checks in calibration or the vehicle control
processes have been relaxed. At this stage, road recognition had not been validated.

A separate worker bug was reproduced using the C3's physical cameras and driving
model, with isolated messaging and recorded/synthetic vehicle-state inputs.
Inference took approximately 0.33–0.37 seconds, reducing the worker's receive rate
and incorrectly triggering its own frequency check. The worker now checks message
validity and source age instead. The same bench test then stayed in Scanning;
calibration produced 60 valid and zero invalid messages in each 15-second measured
window with vision disabled and enabled. This short, isolated test does not
validate the complete vehicle process stack, recognition accuracy, or sustained
thermal performance.

The reader-capacity fix passes 100 host tests (87 feature tests and 13 Params
tests), including preservation of all 15 existing car-state readers and the
auxiliary publisher's gear/validity behavior, the real SubMaster frequency tracker,
and rejection of stale, future-dated, and invalid inputs. The native Qt UI and the
CAN/Panda binding compile with the updated schema. The feature modules and tests
pass Ruff; the existing long line in card.py is unchanged.

Revision 77fc9ba3 was then built and started on the C3 in the parked vehicle,
with ignition on and FrogPilot disengaged. After a 12-second settling period,
a 20-second observation of the complete running stack found:

• All 12 monitored services were alive, valid, and within their receive-frequency
  checks. None of their sampled messages was invalid, including 80 calibration
  messages and 401 messages each from live pose, longitudinal planning, radar,
  live parameters, and driver assistance.
• No selfdrive alert was present in 1,999 observed state messages, and no managed
  process that should have been running was stopped.
• carState held a stable set of 15 reader IDs throughout the window, compared
  with repeated eviction before the fix. These counters were read directly from
  shared memory; adding a diagnostic subscriber to this full channel would itself
  exceed the limit.
• The vision worker was running. Its Idle status was expected in Park; this
  parked check did not exercise inference while driving.

The temporary calibration instrumentation was removed from the device's source
after collection. These results verify recovery from the reproduced parked
communication failure, not road sign accuracy or driving behavior.

Drive investigation and shadow-filter regression (2026-09-19)

The subsequent C3 road trial reported Vision N/A. Recorded messages during the
driving segments were valid, the vision worker remained running with no logged
inference exceptions, and the auxiliary gear flag correctly indicated Drive.
The device stayed below the worker's thermal and memory pause thresholds, but
slcVisionSpeedLimit remained zero. Existing logs do not record each vision
frame's decisions, so they cannot establish the exact live inference cadence or
explain every missed sign.

Offline replay reproduced a specific rejection on a clearly visible 30 mph sign:
the proposal model located it, but the color filter rejected its shadowed white
panel before classification. Direct classification of that crop correctly read
30 mph. The filter now adjusts its brightness reference using the crop's 90th
percentile, with gain limited to 3x. It preserves hue/saturation checks, the
original model inputs, confidence thresholds, and temporal confirmation.

At 6.67 sampled frames per second over a 60-second full-resolution recording,
the original pipeline produced no readings; the correction produced six 30 mph
readings on successive sampled frames and confirmed the limit. No other speed
was produced in that sample. This replay rate is not a measurement of C3 runtime
performance, and this single sign does not establish general recognition accuracy.

All 142 host tests pass, including shadowed neutral/cool white signs, dim colored
sign rejection, featureless crops, and bounded handling of near-black inputs.
The four new dim-white regression cases fail before the fix. The changed Python
files pass Ruff. These were the results before the subsequent C3 bench checks
and glare investigation below. No native UI/schema rebuild is introduced by
the recognition and scheduling changes.
Private route recordings are not included in the repository.

Glare recognition and scheduling follow-up (2026-09-20)

Further review found two independent limitations: sunset-tinted and glare-covered
white signs still failed the color filter, and the original processing backoff
could leave too few inference opportunities before a sign passed. Simply copying
StarPilot's color thresholds also admitted Route 50 shields from the same footage.
The updated recognition gates and bounded follow-up scheduling address these
reproduced cases without counting crop variants as independent observations.

Current host checks pass 219 tests (206 feature tests and 13 Params tests),
including the shipped heading model's preprocessing/dictionary, rejected route,
speed-bump and weight-limit headings, strong-score requirements, finite output
checks, capture-time confirmation and follow-up bounds. The synthetic heading
tests verify the model interface, not road accuracy. Changed Python files pass
Ruff. The converted heading graph passes ONNX validation and comparison against
the pinned original graph; its preparation script reproduces the bundled hash.

Dense inference over ten 60-second full-resolution clips (12,000 actual frames)
recognizes the inspected shadowed 30, tinted 40 and glare-covered 30 signs.
Neither the inspected recreation sign nor the Route 6/50 shields yields an
accepted speed. This is a small, geographically limited regression set used
during development, not an independent accuracy benchmark. Individual-frame
results do not establish that an on-device worker confirms signs in time.

The same ten minutes were decoded to NV12 and sent at their recorded camera
timestamps through native VisionIPC, the actual vision daemon, shared Params,
and FrogPilot's FrogPilotVCruise/SLC in a separate WSL checkout. Messaging and
parameters used unique test prefixes. Recorded qlog vehicle/device/map payloads
were held between samples and republished at service rates; Vision-only selection
was enabled. This exercised the feature and cruise-target integration, not the
driving model, the complete manager stack, vehicle actuation or the onroad UI.

• The first drive delivered 8,400 frames. With a processing delay floor of 1.25
  times the prior C3 per-network cost estimate, the selected limit changed to
  30 mph at 58.19 seconds, 40 at 110.08, and 30 at 317.17. Recorded road-name
  changes cleared it at 157.82 and 395.81 seconds. Configured offsets and cruise
  caps remained in the existing controller path.
• The second drive delivered 3,600 frames at native host inference speed. It
  selected 30 mph at 66.03 seconds and cleared it when the recorded gear changed
  to Park. Neither replay selected another speed or logged an inference failure.
• Maximum camera delivery lateness was 141 ms in the first run and 111 ms in
  the second, below the worker's 500 ms input-age limit. These are host transport
  measurements, not C3 performance measurements.
• A focused native-camera glare replay confirmed 30 mph with a 1.5-times cost
  floor and removed it three seconds after camera input ended. Trying the tighter
  heading alignment first reduced inference work without changing the accepted
  heading set. Rejected headings can request the bounded retry described above.
• A simulation using the actual daemon/Ratekeeper and the dense NV12 inference
  cache recovered the full sequence at all 40 starting phases at both the cost
  estimate and 1.25 times that estimate. At 1.5 times the estimate, 30/40 phases
  passed; ten missed the brief 40 mph observation window. Slowdown margin is
  limited. The delay model uses earlier C3 measurements (212 ms base, 29 ms per
  numeric classification and 150 ms per heading inference); it does not model
  final-device contention and is not a hardware deadline guarantee.

Earlier RGB-only replay and timing simulations missed failures exposed by this
NV12 path. These regressions therefore cannot be replaced by a few still-image
successes. Raw recordings and private diagnostic outputs are not distributed.

Before adding the heading fallback, a native C3 build completed and an isolated
test passed actual padded NV12 VisionIPC frames through the ONNX models, shared
parameters and Vision-only SLC: 30 then 40 mph, with no change on route shields
and removal after camera input expired. Two-thread inference on those examples
took about 0.25–0.34 seconds. Physical camera/model/calibration coexistence tests
measured 15 seconds per phase after settling, with 60 valid calibration messages
and 301 valid messages each from modelV2 and cameraOdometry, and zero invalid
messages, both with vision disabled and with it enabled.

Those earlier C3 results precede the heading fallback. The final three-model
candidate was subsequently built, installed and checked on the C3 as recorded
below. Host replay alone cannot establish ARM scheduling, sustained thermal
headroom or driving safety.

Desktop process and UI replay

A separate WSL checkout now runs the actual modeld, controlsd, plannerd,
frogpilot_process, and speed_limit_vision entry points together with the
native Qt UI. The saved road video enters through padded NV12 VisionIPC at
20 Hz. Recorded vehicle, engagement, calibration and radar messages supply
the inputs that normally come from hardware. The driving model, plans,
controls and Vision limits are computed live; their output messages are not
replayed from the recording.

The desktop adapter uses its own messaging/Params prefix, the recorded
configuration with Vision as the only SLC priority, and locally compiled CUDA
driving models. Cloud/asset maintenance is disabled for this private replay.
Only the narrow camera recording is available, so modeld uses its existing
single-camera path. This is not a closed-loop vehicle simulation or proof of
the original two-camera driving model's accuracy.

Running the real vision entry point exposed a confirmation-time crash: the
worker passed structured fields to cloudlog.info, which does not accept
them. It now uses FrogPilot's cloudlog.event API. A regression test exercises
confirmation, snapshot publication and continued processing with the real
SwagLogger; the earlier observer logger had hidden this error.

The visible desktop replay also exposed clipping of the fifth source row at
the smaller window size. Source rows, icons and fonts now fit the available
height below the speed-limit sign. The native Qt target builds successfully,
and the 1620 by 810 window visibly displays the active Vision source.

The combined desktop run selected 30 mph at recording time 58.20 seconds,
40 mph at 110.45 seconds, cleared the source after the road change at
158.65 seconds, and selected the glare-obscured 30 mph sign at 316.30 seconds.
The real frogpilotPlan carried those values. At the final check, the model,
FrogPilot plan, longitudinal plan and control channels were alive, valid and
within their message-frequency checks. Screenshots verified the visible
Vision row at both 30 and 40 mph. These are host timings, with CUDA driving
inference and CPU VSL inference; they are not C3 timing measurements.

Final candidate C3 validation and tester feedback (2026-09-20)

Runtime revision 42f682c7bfaeed4bc7d4a2b05ec62e510112503e, including the
heading fallback, confirmation logging fix and Qt source-row fix, was built in
a separate staged checkout on the C3 before activation. Native
scons --minimal -j2 completed successfully in 325.6 seconds. The subsequent
build at the canonical installation path passed in 15.5 seconds, and the device
rebooted into that revision with its native Qt UI running.

• The native padded-NV12 VisionIPC/ONNX/shared-Params/Vision-only SLC test
  confirmed 30, then 40, then 30 mph from the recorded fixtures. The inspected
  hard negatives did not change the held limit, and camera expiry removed it.
  Individual fixture processing took 0.20–0.58 seconds; these are short test
  measurements, not a sustained onroad inference-rate guarantee.
• An isolated physical-camera/model/calibration coexistence test measured
  15 seconds per phase after settling. With vision off/on respectively,
  modelV2 and cameraOdometry each produced 300/301 valid messages, and
  calibration produced 60/60. None was invalid. The enabled vision worker
  remained in Scanning under a recorded recognition workload; sampled
  inference times were approximately 0.58–0.61 seconds.
• After reboot, a 20-second observation of the full stack in Park found all
  14 monitored channels alive, valid and within their frequency checks, with
  zero invalid messages, no selfdrive alerts and no unexpectedly stopped
  managed processes. There were 400 messages each from modelV2 and
  cameraOdometry, and 80 calibration messages. The same 15 carState readers
  remained present. Vision was correctly Idle in Park, so this check verified
  startup and communication health rather than sign recognition.
• The installed candidate's 12 manifest hashes matched, and the eight backed-up
  steering/tuning settings remained unchanged through installation and reboot.

After installation, the tester reported that the feature worked very well on
a subsequent drive and requested maintainer review. This is user-reported road
experience, not an independently scored route evaluation; no distance, sign
count, lighting coverage or miss rate was recorded with that report.

The native checks and road feedback supersede the earlier uninstalled/untested
status. General recognition accuracy, sustained load/thermal margin and the
other limitations above remain open for review. The timing simulation's missed
40 mph signs at 1.5 times the cost estimate also remain relevant. Private route
recordings and device diagnostics are not included in this repository.