scsd: send a sequenced recv_ack after a peer's DISCONNECT_RSP (vms-2f3) - #1030
scsd: send a sequenced recv_ack after a peer's DISCONNECT_RSP (vms-2f3)#1030baron-3dl wants to merge 1 commit into
Conversation
…teardown acks (vms-2f3) The booted-node cluster join reaches PROVISIONAL membership on a real VAX (SDA CLUB Nodes=3, OVMXJ0 member flags) but never COMMITS: the barrier stalls, VAX1 times out ~90s later and resets the VC, and CLUSTER_NODES never ticks 2->3. Root cause (wire + VAX-internal SDA, both-ways confirmed): when VAX1 tears down a transient SCS$DIRECTORY connection, its final DISCONNECT_RSP is a sequenced frame (send_seq=N). OVMX advances its VC recv_seq and answers with a standalone 0x48 credit-return (send_seq==0) -- but VMS does NOT count a credit-return as the acknowledgment for a sequenced control frame; it clears "unacked" only on a recv_ack PIGGYBACKED on a sequenced frame (send_seq!=0). After the teardown OVMX goes sequenced-silent to VAX1 (0 sequenced frames +20->110s in the 1029c run; only 0x48 credits), so its piggybacked recv_ack never reaches N. VAX1 therefore retransmits its DISCONNECT_RSP every ~3s for the whole run (37x; a real joiner leaves ~2, because its ongoing DLM-rebuild traffic keeps recv_ack advancing) -- the exact VAX1 SDA "Unacked messages: 1" (VAX2=0) that keeps VAX1's SCS channel to OVMX from ever going clean, blocking the transition commit. Fix: on a received DISCONNECT_RSP, arm a one-shot obligation (peer_state vc_owe_seq_ack) to emit a SEQUENCED cat-0x04 ack on the still-open CM VC (same per-peer VC), carrying recv_ack=vc.seq.recv_seq (>= the DISCONNECT_RSP send_seq). scsd_ack_flush_tick fires it promptly (no flush timer) so the retransmit loop never starts; any cat-0x04 (incl. an organic op-0x06 ack) disarms it, so at most one extra frame per teardown and none when other sequenced traffic exists. NARROW BY CONSTRUCTION: armed ONLY by a DISCONNECT_RSP, keyed on the VC recv_seq -- a different axis from ps->sysap_recv (the SYSAP high-water the op-0x06/join ack cadence uses), so it cannot perturb the currently-working provisional-member path (the STRAYACK census warns against widening the sysap-keyed trigger; this does not touch it). Reuses the existing cm_send_ack / scs_member_build_ack cat-0x04 emitter -- no new frame builder, FSM row, or credit-path change. Test added (test_disconnect_rsp_arms_and_a_cat04_clears_the_sequenced_ack): a real captured teardown arms the obligation on the VC recv_seq and a cat-0x04 clears it. All 26 vmsscs unit tests pass. Lab-adjudicated (necessary prerequisite; may not alone fire the commit if that also needs the full DLM rebuild): (1) 0x7b DISCONNECT_RSP retransmits 37->~2, (2) SDA Unacked->0, (3) barrier commit -> CLUSTER_NODES 2->3. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PhM3QcmFEY3p8YNnHGaYwP
|
Held as draft — the fix fired cleanly but is inert against VAX1's retransmit accounting. Lab re-fire (run 10301): SCSD-I-DISCSEQACK fired ×2 at +17s emitting the sequenced cat-0x04 ack (recv_seq=30, on the CM VC), but VAX1 retransmitted its DISCONNECT_RSP 30× anyway until the ~90s VC-reset timeout ended the loop — not the fix. All three signals negative (retransmit stayed ~30, Unacked=0 only post-reset, barrier 5/12, CN=2). Key finding: a cat-0x04 ack is already an env-0x4b sequenced frame carrying recv_ack on the shared per-peer VC — so 'wrong frame type' is an incomplete explanation. VMS's SCS-transport retransmit-clear accounting counts something our model doesn't capture (candidate: per-connection sequence, so a frame on the CM conid doesn't clear a retransmit on the closed directory conid). Holding until the exact field-level diff between the reference's DISC_RSP-stopping frame and this cat-0x04 is grounded — not another blind frame-swap. The arm-on-DISCONNECT_RSP / emit-on-VC scaffolding is sound and reusable; only the ack vehicle is unresolved. |
The stall
The booted-node cluster join reaches provisional membership on a real VAX (SDA CLUB Nodes=3, OVMXJ0 with member flags) but never commits — the barrier stalls, VAX1 times out ~90s later and resets the VC, and
CLUSTER_NODESnever ticks 2→3.Root cause (wire + VAX-internal SDA, both-ways confirmed)
When VAX1 tears down a transient SCS$DIRECTORY connection, its final DISCONNECT_RSP is a sequenced frame (send_seq=N). OVMX advances its VC recv_seq and answers with a standalone 0x48 credit-return (send_seq==0) — but VMS does not count a credit-return as the acknowledgment for a sequenced control frame; it clears "unacked" only on a recv_ack piggybacked on a sequenced frame (send_seq≠0). After the teardown OVMX goes sequenced-silent to VAX1 (0 sequenced frames +20→110s in the 1029c run; only credits), so its piggybacked recv_ack never reaches N. VAX1 retransmits its DISCONNECT_RSP every ~3s the whole run (37×; a real joiner leaves ~2, because its ongoing DLM-rebuild traffic keeps recv_ack advancing) — the exact VAX1 SDA "Unacked messages: 1" (VAX2=0) that keeps VAX1's SCS channel to OVMX from ever going clean, blocking the transition commit.
Fix
On a received DISCONNECT_RSP, arm a one-shot obligation (
peer_state.vc_owe_seq_ack) to emit a sequenced cat-0x04 ack on the still-open CM VC (same per-peer VC), carryingrecv_ack = vc.seq.recv_seq(≥ the DISCONNECT_RSP send_seq).scsd_ack_flush_tickfires it promptly (no flush timer) so the retransmit loop never starts; any cat-0x04 (incl. an organic op-0x06 ack) disarms it → at most one extra frame per teardown, none when other sequenced traffic exists.Regression safety
Narrow by construction: armed only by a DISCONNECT_RSP, keyed on the VC recv_seq — a different axis from
ps->sysap_recv(the SYSAP high-water the op-0x06/join ack cadence uses). The STRAYACK census warns against widening the sysap-keyed trigger; this doesn't touch it. Reuses the existingcm_send_ack/scs_member_build_ackemitter — no new frame builder, FSM row, or credit-path change.Tests
test_disconnect_rsp_arms_and_a_cat04_clears_the_sequenced_ack— a real captured teardown arms the obligation on the VC recv_seq, and a cat-0x04 clears it. All 26 vmsscs unit tests pass.⚠ Verification bar
H2 is OVMX↔OVMX and won't exercise the real-VAX teardown/retransmit — CI-green is necessary-not-sufficient; the lab is the proof. Lab-adjudicated three signals: (1) 0x7b DISCONNECT_RSP retransmits 37→~2, (2) SDA Unacked→0, (3) barrier commit → CLUSTER_NODES 2→3. Signals 1+2 prove the channel-clean fix landed; signal 3 tells us whether this prerequisite alone fires the commit or whether the full DLM rebuild with VAX1 is also needed. Milestone stays un-cut until sustained OVMXJ0=member at CLUSTER_NODES=3, cap-denied, both consoles.
🤖 Generated with Claude Code