[TeleViz] XR controller controls, in-headset HUD, and a live status panel

docs/source/getting_started/televiz.rst

@@ -660,6 +660,14 @@ QuadLayer / CylinderLayer / EquirectLayer
 
 - ``submit(left, right=None, stream=0)`` — submit a frame (mono: ``left`` only; stereo: both).
 - ``set_placement(placement)`` / ``placement()`` — placement swap, thread-safe vs the frame loop.
+- ``set_stereo_baseline_mm(mm)`` / ``stereo_baseline_mm`` — live per-eye offset, thread-safe vs
+  the frame loop; applies on the next frame. Inert while the layer is mono, since ``stereo`` is
+  fixed at construction. Raises ``ValueError`` on a non-finite value.
+- ``set_stereo_convergence_deg(deg)`` / ``stereo_convergence_deg`` (``CylinderLayer`` /
+  ``EquirectLayer``) — the same idea as a per-eye yaw instead of a translation. Prefer it on a
+  curved surface: translating one gives full disparity dead ahead and less toward the edges, and
+  none at all at infinite radius, while a rotation is uniform across the arc and works at any
+  radius.
   ``QuadLayer`` accepts ``None`` (fullscreen, window mode); the shaped layers validate and raise
   ``ValueError`` on bad shape parameters.
 - ``set_visible(visible)`` / ``is_visible()``.

docs/source/references/camera_streaming.rst

@@ -230,6 +230,56 @@ In XR, how a plane follows the operator's head is the per-camera ``lock_mode`` u
 Lazy-mode knobs live under ``placements.<name>``: ``look_away_angle_deg``,
 ``reposition_distance``, ``reposition_delay_s``, ``transition_duration_s``.
 
+Controller bindings
+^^^^^^^^^^^^^^^^^^^
+
+In XR the controllers retune the view live, without editing the YAML and restarting. The right
+hand changes how the feed looks; the left, what surface it is mapped onto:
+
+.. figure:: ../_static/camera-viz-controls.svg
+   :alt: camera_viz controller bindings, left and right
+   :width: 100%
+
+   Bindings at a glance. The table below is the same thing in words.
+
+.. list-table::
+   :header-rows: 1
+   :widths: 22 78
+
+   * - Input
+     - Effect
+   * - Right stick ←/→
+     - Stereo plane gap — how far apart the two eyes' planes sit
+       (``placements.<cam>.stereo_plane_distance_cm``). Widening it pushes the scene back
+       instead of packing it into the space in front of the planes. Stereo cameras only; on
+       ``equirect``, which has no gap to set, the same stick pans the panorama.
+   * - Right stick click
+     - Recenter on your view: an ``equirect`` yaws so the middle of the panorama lands dead
+       ahead, and a placed surface re-snaps its anchor.
+   * - ``A``
+     - Cycle the lock mode: ``world`` → ``head`` → ``gimbal`` → ``lazy``.
+   * - ``B``
+     - Toggle mono / stereo. Stereo cameras only.
+   * - ``X``
+     - Cycle the shape: ``quad`` → ``cylinder`` → ``equirect``.
+   * - ``Y``
+     - Reset everything to the YAML values.
+   * - Left stick
+     - Retunes the active shape: ``quad`` size / vertical position, ``cylinder`` arc width /
+       vertical position, ``equirect`` horizontal / vertical span.
+
+Changes apply to every camera at once and appear both on the terminal status panel and on a
+head-locked panel in the headset that auto-hides shortly after. Neither toggle reallocates:
+``B`` sends the left frame to both eyes, and ``X`` flips visibility between shapes built at
+startup, so the extra shapes cost VRAM (reported at startup) rather than a stall on the press.
+
+The stereo gap is bounded below **divergent parallax** — a gap wider than your IPD would need
+the eyes to splay outward — and the headset's measured IPD sets that ceiling, not the config.
+The HUD suggests a gap derived from the plane distance and that IPD.
+
+Bindings, rates and limits live under ``display.controls``; see the
+:code-file:`README <examples/camera_viz/README.md>` for the full set.
+
 Display surfaces
 ----------------
 
@@ -252,7 +302,10 @@ panoramic sources look better on a curved surface: set ``shape`` per camera unde
        visor).
    * - ``equirect``
      - Full 360°×180° sphere around the operator, for equirectangular panorama / VR-video
-       sources. Lock modes don't apply.
+       sources. The middle of the texture points along ``equirect_yaw_deg`` (0 = the
+       direction the headset last recentered on, positive to the left), which is how a camera
+       mounted facing another way gets aimed. Lock modes don't apply — the sphere is centred
+       on the operator and follows them everywhere.
 
 Curved shapes exist only in XR mode — the viewer exits with an error in window mode. Stereo
 sources render per-eye textures on the same surface, and ``stereo_baseline_mm`` adds a per-eye
@@ -276,6 +329,31 @@ serving. Useful flags:
 
 Run ``camera_viz.py --help`` for the rest (install dir, env-config file, WSS proxy toggle).
 
+Runtime settings themselves are declared in the config rather than exported. The viewer ships
+defaults (pose wait off, runtime foveation on), and ``display.cloudxr`` overrides them per
+deployment — list only what you want to change:
+
+.. code-block:: yaml
+
+   display:
+     cloudxr:
+       NV_DEVICE_PROFILE: apple-vision-pro
+       NV_ENABLE_POSE_WAIT: null   # null drops a viewer default
+
+The viewer writes these to a generated ``--cloudxr-env-config`` file, which outranks the process
+environment — so a value left in your shell (for example after sourcing
+``~/.cloudxr/run/cloudxr.env``, as ``--no-launch-cloudxr-runtime`` suggests) cannot silently
+override the config, and neither can it override ``--cloudxr-device-profile``. Booleans are
+written in the lowercase spelling the runtime's parser recognises; the launcher-computed keys
+(``XR_RUNTIME_JSON``, ``NV_CXR_RUNTIME_DIR``, ``NV_CXR_OUTPUT_DIR``, ``XRT_NO_STDIN``) are
+rejected by name. Passing ``--cloudxr-env-config`` yourself takes precedence over the generated
+file.
+
+An unknown variable name warns with a suggestion before the runtime launches, and is passed
+through regardless — the runtime has more settings than the viewer lists. To confirm what the
+runtime resolved, read the settings dump at the top of the newest
+``~/.cloudxr/logs/cxr_server.*.log``.
+
 .. _split-mode:
 
 Split mode — robot → workstation over RTP
@@ -363,8 +441,9 @@ its own plane (and, in split mode, its own RTP port). Abbreviated:
          lock_mode: lazy         # world | head | lazy | gimbal
          distance: 1.5
          # size: [w_m, h_m]
-         # stereo_baseline_mm: 0
+         # stereo_plane_distance_cm: 0   # gap between the eyes' planes
          # shape: quad           # quad | cylinder | equirect (cylinder/equirect are XR-only)
+         # equirect_yaw_deg: 0.0  # equirect: heading the middle of the feed points at
          # compositor: openxr    # openxr (default) | televiz — quads only
          # cylinder_radius_m: 2.0
          # cylinder_angle_deg: 90

examples/camera_viz/README.md

@@ -129,8 +129,10 @@ display:                      # camera_viz only
       offset_x: 0.0
       offset_y: 0.0
       # size: [w_m, h_m]
-      # stereo_baseline_mm: 0  # stereo cams: 0 = both eyes share the world quad
-                               # (parallax from the frames); ~65 = virtual IPD push
+      # stereo_plane_distance_cm: 0   # stereo cams: gap between the left- and
+                                      # right-eye planes; 0 = both eyes share
+                                      # one plane. See "Stereo plane distance"
+      # equirect_yaw_deg: 0.0  # equirect: heading the middle of the feed points at
       # shape: quad            # quad (default) | cylinder | equirect — XR only for
                                # the curved shapes
       # compositor: openxr     # openxr (default) | televiz — quads only
@@ -169,15 +171,147 @@ display:
 
 Lazy knobs under `placements.<name>`: `look_away_angle_deg`, `reposition_distance`, `reposition_delay_s`, `transition_duration_s`.
 
+## Controller bindings (XR)
+
+Retune the view without editing YAML and restarting. The right hand changes how the feed looks, the left what surface it's on. Quest and Pico report these identically.
+
+| Input | Effect |
+|---|---|
+| **Right stick** ←/→ | Stereo plane gap. Stereo cameras only — on `equirect`, pans instead |
+| **Right stick click** | Recenter on your view |
+| **A** | Lock mode: `world` → `head` → `gimbal` → `lazy` |
+| **B** | Mono / stereo. Stereo cameras only |
+| **X** | Shape: `quad` → `cylinder` → `equirect` |
+| **Y** | Reset everything to the YAML values |
+| **Left stick** | Per shape, below |
+
+| Shape | ←/→ | ↑/↓ |
+|---|---|---|
+| `quad` | size, aspect preserved | slide up / down |
+| `cylinder` | arc width | slide up / down |
+| `equirect` | horizontal span | vertical span |
+
+Recentering reads per shape: an `equirect` yaws so the middle of the panorama lands dead ahead, and a placed surface re-snaps its anchor — the way back from a `world`-locked plane left in another part of the room.
+
+Changes apply to every camera at once and appear on the status panel and an in-headset HUD that auto-hides ~2.5 s later (`hud: false` to disable). Nothing is written back to the YAML.
+
+Neither toggle reallocates: **B** sends the left frame to both eyes, **X** flips visibility between shapes all built at startup. The cost is VRAM — two extra layers per camera, reported at startup; `shape_switching: false` keeps only the configured shape.
+
+A quad's `distance` and a cylinder's `cylinder_radius_m` stay YAML-only. Apparent size is `2·atan((w/2)/d)` and arc width is `radius × angle`, so moving either surface further away enlarges it by the same factor and looks identical — they'd duplicate the size axis. What they *do* change is the real distance to the surface, which the stereo gap works from.
+
+```yaml
+display:
+  controls:
+    enabled: true                      # false disables the bindings
+    hud: true                          # in-headset readout
+    shape_switching: true              # keep all 3 shapes resident for X
+    deadzone: 0.2                      # stick rest-position tolerance
+    plane_distance_rate_cm_per_s: 2.0  # held-stick ramp rates
+    size_rate_m_per_s: 0.5
+    offset_rate_m_per_s: 0.5
+    angle_rate_deg_per_s: 40.0
+```
+
+Limits — `plane_distance_min_cm` / `plane_distance_max_cm`, `size_range_m`, `offset_y_range_m`, `cylinder_angle_range_deg`, `equirect_h_range_deg`, `equirect_v_half_range_deg` — all default inside what the layers accept. See `configs/zed.yaml`.
+
+### Stereo plane distance
+
+A stereo layer draws each eye's image on its own plane; `stereo_plane_distance_cm` is the gap between them.
+
+```yaml
+display:
+  placements:
+    zed:
+      distance: 1.0                  # how far away the planes are
+      stereo_plane_distance_cm: 5.0  # how far apart they are
+```
+
+At `0` both eyes share one plane, so the whole scene — near objects and far background alike — is packed into the space between you and `distance`. Widening the gap pushes it back and lets it spread out. Applied exactly as given (×10 into the layer's `stereo_baseline_mm`); the stick moves it in 0.1 cm steps.
+
+The HUD suggests a value from the plane distance and the headset's measured IPD, shown beside the one in use (`5.0/5.2` on the panel). Advice only — the stick sets the value, which is also how you correct a headset whose IPD setting doesn't match your eyes.
+
+The stick can't reach **divergent parallax**: at a gap equal to your IPD the eyes' rays are parallel, and beyond it they would have to splay outward — the classic cause of stereo eye strain. The ceiling comes from the measured IPD, not the config.
+
+**B** parks the gap at zero while mono, since one image on two separated planes would only shift its depth, and restores it on the way back.
+
+Not applicable to `equirect`: the gap shifts each eye's surface, and the sphere sits at infinite radius where translating it does nothing, so the stick skips it there and pans instead. Curved layers have a rotation-based equivalent in Televiz (`stereo_convergence_deg`), which is uniform across the arc and works at any radius.
+
+> **Not the camera's baseline.** That is the physical gap between the camera's two lenses — fixed in hardware, baked into the pixels, and what sets the scene's depth *scale*. This only moves where that scene sits.
+
+> Live adjustment needs `Layer.set_stereo_baseline_mm`, newer than the released `isaacteleop` wheel. On an older wheel that one binding disables itself with a notice; A, B, X and Y still work.
+
+### Aiming an equirect
+
+The middle of the texture maps to the sphere pose's **−z**, and the middle row to its horizon. With no rotation that is the reference space's forward — wherever the headset faced when it last recentered — so a camera mounted facing another way puts the interesting part of the feed off to one side.
+
+```yaml
+display:
+  placements:
+    sky:
+      shape: equirect
+      equirect_yaw_deg: -90.0  # heading the middle of the feed points at
+```
+
+Positive is a turn to the left. The right stick pans it live and the stick click snaps it to your view; **Y** puts it back to the YAML value. The heading in use rides in the panel's `shape` column (`equirect +35°`).
+
+The sphere has no lock mode: it is centred on you and follows your head everywhere, so `world` / `head` / `lazy` and the `distance` / `size` knobs are all blank for it. This heading is its whole placement.
+
 ---
 
+## CloudXR runtime settings
+
+The runtime is configured by environment variables, and there are three ways to set one and be silently ignored: a name nothing reads, a stale `source ~/.cloudxr/run/cloudxr.env` in your shell (which outranks both `os.environ` and `--cloudxr-device-profile`), and a boolean spelled `False` — which the runtime's exact-match parser doesn't recognise, so it reads as **true**.
+
+camera_viz ships defaults in `cloudxr_env.DEFAULT_ENV` — pose wait off, runtime foveation on — so a config that says nothing gets both. `display.cloudxr` is a per-deployment override; list only what you want to change:
+
+```yaml
+display:
+  cloudxr:
+    NV_DEVICE_PROFILE: apple-vision-pro
+    NV_ENABLE_POSE_WAIT: null   # null drops a camera_viz default
+```
+
+camera_viz writes these to a generated `--cloudxr-env-config` file, the one tier that outranks the process environment. Booleans are lowercased for you; the launcher-computed keys (`XR_RUNTIME_JSON`, `NV_CXR_RUNTIME_DIR`, `NV_CXR_OUTPUT_DIR`, `XRT_NO_STDIN`) are refused by name. Passing `--cloudxr-env-config` yourself wins — camera_viz won't overwrite it.
+
+An unknown variable name warns with a suggestion before anything launches — `NV_CXR_DEVICE_PROFILE` is not a name the runtime reads, and it's one keystroke from one that is — but is still passed through, since the runtime has more knobs than camera_viz lists.
+
+To confirm what the runtime actually resolved, read the settings dump at the top of the newest `~/.cloudxr/logs/cxr_server.*.log`. Monado's `DEBUG_GET_ONCE` options (`NV_ENABLE_POSE_WAIT` among them) aren't in that dump; run with `XRT_PRINT_OPTIONS=true` and each is logged as `NAME=value (raw)`, where `(nil)` means it never arrived.
+
+> `NV_DEVICE_PROFILE` is device matching, not tuning. Anything other than `quest3` selects separate-frames packing, and `isFrameClientReconstructed()` bails unless packing is packed-frame — so a non-Quest profile gives up client-reconstructed streaming.
+
+---
+
+## Status panel
+
+On a terminal, camera_viz redraws a snapshot in place instead of scrolling a log:
+
+```
+camera_viz  xr · local · 1 camera
+────────────────────────────────────────────────────────────────────
+  render 58.0 fps (target 72)   missed 0   gpu 2.1 ms
+
+  camera      shape          lock    eyes   size m  height m  planes cm   submit/s
+  zed         cylinder       lazy    stereo 1.00    +0.00     5.0/5.2     64.0
+
+  headset IPD 63 mm
+
+  stereo planes: 5.0 cm  ·  suggested: 5.2 cm · IPD: 63 mm
+```
+
+`planes cm` is the value in use and the suggestion. `-` means the field doesn't apply — equirect has no gap, window mode has no controls. While the panel is live it owns stderr: control messages and source lifecycle lines go through it rather than printing underneath and pushing it out of alignment. When stderr isn't a terminal (piped, or a `deploy`ed systemd unit) it falls back to one line every 5 s with the same numbers.
+
 ## Layout
 
 ```
 camera_viz/
 ├── camera_viz.sh        — CLI: setup / loopback / run / deploy / service-*
-├── camera_viz.py        — receiver / viewer
+├── camera_viz.py        — receiver / viewer (entrypoint + wiring)
 ├── camera_streamer.py   — robot-side RTP sender (per-camera supervisor)
+├── config.py            — YAML → SourceEntry: parse + validate, no allocation
+├── display.py           — VizSession + one layer per surface
+├── cloudxr_env.py       — display.cloudxr → the runtime's env file
+├── dashboard.py         — terminal status panel
+├── controls/            — XR controller bindings, shapes, stereo geometry, HUD
 ├── pipeline/            — source ABC + threaded runner
 ├── placements/          — XR lock-mode strategies
 ├── sources/             — V4L2 / OAK-D / ZED / synthetic / video replay / rtp_h264