From 137d1aa8ea1af227038a32c806c2adeb29c18f35 Mon Sep 17 00:00:00 2001 From: Anton Nekipelov <226657+anton-107@users.noreply.github.com> Date: Tue, 1 Sep 2026 08:37:33 +0000 Subject: [PATCH] Correct stale architecture docs in experimental/ssh/README.md The compute requirements claimed terminal SSH works on standard access mode. It does not: ValidateClusterAccess requires Dedicated access mode with single_user_name set, and both entry points call it (client.go on a direct `connect --cluster`, setup.go on `ssh setup`), so a standard, dedicated-to-a-group, or no-isolation cluster is rejected before any work happens. The two design diagrams had drifted from the code as well: - Keys travel through a per-session secret scope, not a pub-key baked into the bootstrap notebook. The notebook is `ssh-server-bootstrap`, not `start-server-with-pub-key.ipynb`. - Workspace paths are /Workspace/Users/$me/.databricks/ssh-tunnel/$v/..., not ~/.ssh/$v/..., and metadata.json also carries cluster_id and usage_policy_id. - The server serves /logs alongside /ssh and /metadata, publishes metadata before it starts accepting connections, and never deletes metadata.json - it just exits, which is why the client must re-probe /metadata through the driver proxy rather than trusting the file. - The ProxyCommand takes --cluster/--metadata flags, and the websocket URL includes /o/$workspaceId and ?id=$connId. - The high-level diagram showed per-Spark-user servers, an arrangement that only made sense while standard access mode was in scope. Co-authored-by: Isaac --- experimental/ssh/README.md | 93 +++++++++++++++++++++----------------- 1 file changed, 52 insertions(+), 41 deletions(-) diff --git a/experimental/ssh/README.md b/experimental/ssh/README.md index 769006ef2df..5f9194db7cc 100644 --- a/experimental/ssh/README.md +++ b/experimental/ssh/README.md @@ -3,8 +3,13 @@ The SSH tunnel lets customers connect any IDE to Databricks compute to run and debug all code - including non-Spark/ML - with environment parity, and simple setup. ## Compute Requirements -- Dedicated (single user) access mode if you want to use Remote Development tools in IDEs -- Dedicated or standard access mode for terminal SSH connections +- Serverless compute, which is the default when `--cluster` is omitted, or +- A cluster in Dedicated access mode assigned to a single user, not to a group. + +`ValidateClusterAccess` (`internal/client/client.go`) rejects every other access mode up front, +for terminal SSH sessions as well as for IDE Remote Development: the tunnel runs as a job that +attaches as a single user. Standard access mode (`USER_ISOLATION`), Dedicated-to-a-group, and +no-isolation clusters all fail with `cluster '' must be a dedicated single-user cluster`. ## Usage A. With local ssh config setup: @@ -36,36 +41,35 @@ config: layout: dagre --- flowchart TD - n1(["Client A"]) + n1(["Client"]) subgraph s1["Control Plane"] n3["Jobs API"] n2["Driver Proxy API"] n11["Workspace API"] + n13["Secrets API"] end - subgraph s3["Spark User A or root"] - n4["SSH Server A"] - end - subgraph s4["Spark User B or root"] - n6["SSH Server B"] - end - subgraph s2["Cluster"] - s3 - s4 - n12["Workspace Filesystem"] + subgraph s2["Compute - dedicated single user, or root"] + n4["SSH Server"] + n7["sshd, one process per connection"] end - n1 -. "1 - start an ssh server job" .-> n3 - n3 -. "2 - start ssh server" .-> n4 - n4 <-. "3 - save the ssh server port number" .-> n12 - n1 <-. "4 - get ssh server port number" .-> n11 - n1 <-. "6 - websocket connection" .-> n2 - n2 <-. "7 - websocket connection" .-> n4 - n6 <-.-> n12 + n1 -. "1 - store the client key pair" .-> n13 + n1 -. "2 - upload the CLI binary" .-> n11 + n1 -. "3 - start an ssh server job" .-> n3 + n3 -. "4 - start ssh server" .-> n4 + n4 -. "5 - read the client public key and store the host key" .-> n13 + n4 -. "6 - publish the ssh server port number" .-> n11 + n1 -. "7 - get the ssh server port number" .-> n11 + n1 <-. "8 - websocket connection" .-> n2 + n2 <-. "9 - websocket connection" .-> n4 + n4 <-. "10 - stdio" .-> n7 style s2 stroke:#757575 style s1 stroke:#757575 - style s4 stroke-dasharray: 5 5 - style n6 stroke-dasharray: 5 5 ``` +The client public key reaches the server through a secret scope rather than through the +bootstrap notebook, and the server host key is persisted in the same scope so its fingerprint +survives a restart - which is what makes `StrictHostKeyChecking accept-new` safe here. + Connection flow: ```mermaid --- @@ -77,36 +81,39 @@ sequenceDiagram participant P1 as databricks ssh connect participant P2 as ssh client participant P3 as databricks ssh connect --proxy - participant P4 as wsfs + participant P4 as workspace api + participant P5 as secrets api participant P6 as databricks ssh server participant P7 as sshd - Note over P1,P6: Try to get a port and a remote user name of an existing server
($v is databricks CLI version, $cluster is supplied by the user) + Note over P1,P6: Try to get a port and a remote user name of an existing server.
$v is the databricks CLI version. $s is the session id:
the cluster id, or the --name value for serverless. activate P1 - P1 ->> P4: GET ~/.ssh/$v/$cluster/metadata.json - P4 -->> P1: {port: xxxx} or error - P1 ->> P6: GET /driver-proxy-api/$cluster/$port/metadata - P6 -->> P1: {user: spark-xxxx} or {user: root} or error - Note over P1,P6: Start the new server in the case of an error + P1 ->> P4: GET /Workspace/Users/$me/.databricks/ssh-tunnel/$v/$s/metadata.json + P4 -->> P1: {port, cluster_id, usage_policy_id} or error + P1 ->> P6: GET /driver-proxy-api/o/$workspaceId/$cluster/$port/metadata + P6 -->> P1: the OS user the server runs as - root - or error + Note over P1,P6: Start a new server if either step failed, or if the
running one was started under a different usage policy. opt - P1 -->> P1: generate
key pair - P1 -->> P4: PUT ~/.ssh/$v/bin/databricks, unless it's already there - P1 ->> P4: PUT ~/.ssh/$v/$cluster/start-server-with-pub-key.ipynb - P1 ->> P6: jobs/runs/submit start-server-with-pub-key.ipynb $cluster + P1 ->> P5: create scope $me-$s-ssh-tunnel-keys,
generate and store the client key pair unless already there + P1 -->> P4: PUT ssh-tunnel/$v/$cliName/databricks, unless it's already there + P1 ->> P4: PUT ssh-tunnel/$v/$s/ssh-server-bootstrap + P1 ->> P6: jobs/runs/submit ssh-server-bootstrap $cluster activate P6 - P6 ->> P6: start self-kill-timeout
generate server key pair
create custom sshd config
listen for /ssh and /metadata on a free port - P6 ->> P4: PUT ~/.ssh/$v/$cluster/metadata.json
{port: xxxx} - loop unil successful or timed out + P6 ->> P6: pick a free port, starting at 7772 + P6 ->> P4: PUT ssh-tunnel/$v/$s/metadata.json
{port, cluster_id, usage_policy_id} + P6 ->> P5: read the client public key,
generate and store the server host key unless already there + P6 ->> P6: write authorized_keys and a custom sshd config
start self-kill-timeout
listen for /ssh, /metadata and /logs + loop until successful or timed out - 30 attempts, 2s apart P1 -> P6: Get port and remote user name of the server (sequence 1 - 4 above) end end Note over P1,P7: We know the port and the user, spawn "ssh" - P1 ->> P2: ssh -l $user -i $key
-o ProxyCommand="databricks ssh connect --proxy $cluster $user $port" + P1 ->> P2: ssh -l $user -i $key -o ServerAliveInterval=30
-o ProxyCommand="databricks ssh connect --proxy
--cluster=$cluster --metadata=$user,$port,$cluster" activate P2 P2 ->> P3: exec ProxyCommand activate P3 - P3 ->> P6: wss:/dirver-proxy-api/$cluster/$port/ssh + P3 ->> P6: wss:/driver-proxy-api/o/$workspaceId/$cluster/$port/ssh?id=$connId P6 ->> P6: stop self-kill-timeout - P6 ->> P7: /usr/sbin/sshd -i -f config + P6 ->> P7: /usr/sbin/sshd -f config -i activate P7 P2 -> P7: pubkey auth loop until the connection is closed
by ssh client, sshd, or driver-proxy @@ -118,8 +125,12 @@ sequenceDiagram deactivate P1 end break when the last ws connection drops - P6 ->> P6: start self-kill-timeout - P6 ->> P4: DELETE ~/.ssh/$v/$cluster/metadata.json + P6 ->> P6: start self-kill-timeout, then exit when it fires deactivate P6 end ``` + +Note that `metadata.json` is published before the server starts accepting connections, and is +left behind when the server exits. Neither its presence nor its contents prove that a server is +running, which is why the client always re-checks `/metadata` through the driver proxy before +reusing a port.