Heartbeat keeps reporting cameras as alive when they can be actually down (stale cached frame)

[fe51]

Pitch

The engine sends a device-liveness heartbeat to the PyroNear API for a camera as long as the camera API returns any frame for it BUT that frame comes from a cache that is never invalidated. Once a camera has produced one image, it will keep heart-beating "alive" to the cloud indefinitely even after the physical camera goes offline. Operators watching camera health on the platform can't trust the heartbeat to tell them a camera is actually reachable.

Why it's a problem

The heartbeat is gated on frame is not None, and the frame is a cached image, not a live capture:

• SystemController.inference_loop fetches a frame, and only calls engine.predict(...) if it's non-None — core.py:201-205.
• predict is the only place the heartbeat fires — engine.py:268-269 → engine.py:222-225 (api_client[ip].heartbeat()).
• The frame is read from cam.last_images — routes_cameras.py:254-256 — which is only written on a successful capture and never cleared on failure: the patrol loop logs and continues, leaving the old frame in place — patrol.py:89-96.
• There is no TTL and no clear-on-failure. The only place the cache is cleared is the unrelated stuck-PTZ reboot path — stuck_detector.py:166-168.

Result: after a camera delivers at least one frame, /latest_image keeps returning that stale-but-valid frame, so predict() keeps running and the heartbeat keeps flowing — regardless of whether the camera is reachable now. The heartbeat effectively means "the camera API is up and once had an image for this pose," not "the camera is reachable."

Notes on the edges:

• A camera that never delivered a frame is reported correctly (empty cache → HTTP 204 → None → predict skipped → no heartbeat).
• Heartbeats are also suppressed while an RTSP/SRT stream is active, since the whole inference pass is skipped — core.py:187-198 — so an active stream masks camera health too.

Suggestions to be discussed

• OPTION A : Timestamp the cache (preferred). Store (image, captured_at) in last_images and have /latest_image return 204 (or a stale flag) when the newest capture is older than N patrol cycles. This reuses the existing frame is None gate, so dead cameras naturally stop heart-beating with no new call path.
• OPTION B Decouple heartbeat from the cached frame. Do a real liveness probe of the camera (or track last-successful-capture time in the camera API and expose it) before sending the heartbeat, instead of inferring liveness from a possibly-stale image.

[fe51]

Option A seems smaller, lower-risk change might be relevant as a starting point. @MateoLostanlen what do you think ?