connect: run the GUI on a cluster node, browse it locally (magnetics-connect) - #89
Merged
Conversation
Run the server on a cluster work node, browse it locally — from a laptop off-site or a NoMachine desktop on the site gateway. One ssh connection carries the -L tunnel and the server (loopback-only, free port asked from the remote, real node name reported since submit nodes are load-balanced); the forced pty SIGHUPs the server on disconnect so nothing is orphaned on a shared node. Stdlib-only and standalone so it also runs as a bare python3 connect.py on gateways with no magnetics install. Verified live laptop -> cybele -> omega-b: tunnel, readiness poll, streamed remote logs, and clean teardown. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Running the GUI on an omega work node (the NoMachine -> cybele -> omega flow) sent the remote backend's ssh to omega.gat.com *from* omega. That node's known_hosts has no entry for it, so ssh stopped at the host-key confirmation prompt — which a server process has no tty to answer. The pull sat at 0% forever instead of failing. on_cluster_host() compares our short hostname against the device's configured cluster host, gated on already being inside the site domain, so load-balanced submit nodes (omega-a/-b) and workers (omega25) all match the configured omega.gat.com. The site gateway does NOT match: from cybele, ssh to omega is still the correct path. Where it matches, the pull runs in-process — toksearch when importable, else mdsthin straight to mdsip, which needs no ssh on-site. MAGNETICS_ON_CLUSTER forces either answer. Verified on omega-a: the same --backend remote pull that hung now reports 'already on the cluster; remote->mdsthin' and fetches. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The on-cluster fix picked its backend with `import toksearch` inside the GUI's own venv -- which never has toksearch; it lives in the site conda env. So every on-cluster pull silently fell through to mdsthin/mdsip, ~17x slower than the native PTDATA read (164s vs ~10s for the same shot), making a pull ON omega slower than the laptop's remote pull, which had been using toksearch all along. On the cluster now try, in order: toksearch importable here -> in-process; else the device's cluster interpreter has it -> run_on_cluster subprocess (the exact command run_remote sends over SSH, minus the SSH); else mdsthin, and say so. run_on_cluster exposes ONLY the magnetics package (a temp-dir symlink) to that interpreter -- putting our whole 3.14 venv site-packages on a 3.11 conda PYTHONPATH shadowed its numpy and broke `import toksearch` (the first attempt's crash). Deps resolve from the conda env, as the SSH path already relies on. Verified on omega-a: 184927 rotating, 200ms window -> 53/67 channels in 9.6s via toksearch (was 164s via mdsthin). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Bare `uvx ruff check` resolves the latest release. ruff 0.16.0 broadened the default rule set (E501, N802, S-codes, …), so CI's lint step started failing on unchanged, pre-existing code — every open PR, not one diff. Pin the CI `uvx ruff@0.15.22` invocations and cap the dev-group ruff at <0.16 so local `uv run ruff` matches. Adopt a newer ruff deliberately, in a PR that also fixes its findings, rather than by surprise on release day. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
magnetics-connect assumed the server was already installed on the remote. For the GA hand-off it now bootstraps: run an installed `magnetics` if present (fast path); else install uv with the official root-free installer (~/.local/bin, no sudo); else launch via `uvx magnetics`, which fetches the package and provisions its own Python (cluster system Pythons are far too old). Zero-touch for a fresh node with only connect.py. Targets PyPI (`uvx magnetics`); until it is published, --install-from points uvx at a wheel URL/path, and the fast path uses any existing install (the shim in the meantime). --remote-cmd still fully overrides for module-load setups. Bumped the readiness timeout to 300s for first-run provisioning. Verified on omega-a: the fast path reaches the existing install and serves through the tunnel; no orphaned processes on teardown. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
3 tasks
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Findings from the branch review, plus one leak they surfaced: - teardown: the remote server lives as long as the ssh carrying it, but only the Ctrl-C path terminated that child. A bare `kill <pid>` (SIGTERM) skipped cleanup entirely and orphaned a server on the shared cluster node. Move the terminate into the finally so every exit covers it, and route SIGTERM into the same path. Verified on omega-a: 1 server before SIGTERM, 0 after. - Ctrl-C: child.wait(timeout=10) could raise TimeoutExpired (not a subclass of KeyboardInterrupt) out of the handler, turning a clean stop into a traceback when the remote was wedged. Escalate to kill() instead. - cluster_python_has_toksearch: Path(python).exists() pre-rejected bare command names that subprocess would resolve via PATH, silently downgrading an on-cluster pull to mdsthin (~5-7x slower). Resolve with shutil.which. - drop the write-only _pump tail deque (lines already stream live). - CLAUDE.md: the CI checklist still said bare `uvx ruff check .`, which resolves the latest release — the drift the pin was added to prevent. Still compiles + runs on Python 3.9 (the gateway constraint). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
4 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Users at GA reach the analysis stack by remote-desktopping (NoMachine) into a
virtual workstation on the cybele login cluster, then SSHing to a omega
work node. That breaks the webapp: the server's
127.0.0.1:PORTis on omega,but the browser is on cybele.
The fix is not an X-forwarded browser — it's the standard SSH tunnel, which
rides the connection you're already making. This adds a launcher that does the
whole dance in one command, abstracted over host/gateway so it works from a
laptop off-site, from a site gateway desktop, or toward any cluster with the
same login-node pattern (flux, feyn, …).
What it does
magnetics-connect <host>opens one authenticated SSH connection that:are load-balanced — "omega" resolves to omega-a or omega-b run to run);
bash -lcsomodule/profile PATHs apply;
-Lport-forward on that same connection;Only HTTP crosses the wire: fetches and fits run next to the data, plots stream
back. Ctrl-C stops both ends — the forced remote pty SIGHUPs the server, so
nothing is orphaned on a shared node.
Why a tunnel rather than binding 0.0.0.0
I measured the firewall: cybele → omega on a high TCP port is not blocked, so
a direct bind would "work". It's still the wrong default — it exposes an
unauthenticated API that triggers data fetches and serves shot data to the whole
GA internal network, and it collides when two users pick the same port. A
direct-bind mode should come with a Jupyter-style access token; that's a
follow-up, not this PR.
Standalone by design
src/magnetics/connect.pyis stdlib-only and self-contained so it runs on agateway node with only system python3 and no magnetics install:
Verified under Python 3.9 (cybele ships 3.9.25). CLAUDE.md notes the constraint
so it isn't refactored away.
Verification
logs streamed back with an
[omega]prefix, clean teardown with zero orphanedprocesses on the node. Repeated against a real install of this wheel on omega:
the bundled GUI and
/api/*both serve through the tunnel.tests/test_connect_launcher.py) with ssh/subprocessfaked, covering probe → tunnel → serve command construction, the jump flag,
{port}substitution, data-dir quoting, probe failure, and thenever-became-ready path.
ty check.Follow-ups (not in this PR)
module load magneticsreplaces themanual install.
--host 0.0.0.0mode for users who'd rather skipthe tunnel.
🤖 Generated with Claude Code