From 567cc71b848fae8167dc5986944b897109ea2a15 Mon Sep 17 00:00:00 2001 From: alice Date: Tue, 1 Sep 2026 16:24:52 +0000 Subject: [PATCH] scsd: send a sequenced recv_ack after a peer's DISCONNECT_RSP so the teardown acks (vms-2f3) The booted-node cluster join reaches PROVISIONAL membership on a real VAX (SDA CLUB Nodes=3, OVMXJ0 member flags) but never COMMITS: the barrier stalls, VAX1 times out ~90s later and resets the VC, and CLUSTER_NODES never ticks 2->3. Root cause (wire + VAX-internal SDA, both-ways confirmed): when VAX1 tears down a transient SCS$DIRECTORY connection, its final DISCONNECT_RSP is a sequenced frame (send_seq=N). OVMX advances its VC recv_seq and answers with a standalone 0x48 credit-return (send_seq==0) -- but VMS does NOT count a credit-return as the acknowledgment for a sequenced control frame; it clears "unacked" only on a recv_ack PIGGYBACKED on a sequenced frame (send_seq!=0). After the teardown OVMX goes sequenced-silent to VAX1 (0 sequenced frames +20->110s in the 1029c run; only 0x48 credits), so its piggybacked recv_ack never reaches N. VAX1 therefore retransmits its DISCONNECT_RSP every ~3s for the whole run (37x; a real joiner leaves ~2, because its ongoing DLM-rebuild traffic keeps recv_ack advancing) -- the exact VAX1 SDA "Unacked messages: 1" (VAX2=0) that keeps VAX1's SCS channel to OVMX from ever going clean, blocking the transition commit. Fix: on a received DISCONNECT_RSP, arm a one-shot obligation (peer_state vc_owe_seq_ack) to emit a SEQUENCED cat-0x04 ack on the still-open CM VC (same per-peer VC), carrying recv_ack=vc.seq.recv_seq (>= the DISCONNECT_RSP send_seq). scsd_ack_flush_tick fires it promptly (no flush timer) so the retransmit loop never starts; any cat-0x04 (incl. an organic op-0x06 ack) disarms it, so at most one extra frame per teardown and none when other sequenced traffic exists. NARROW BY CONSTRUCTION: armed ONLY by a DISCONNECT_RSP, keyed on the VC recv_seq -- a different axis from ps->sysap_recv (the SYSAP high-water the op-0x06/join ack cadence uses), so it cannot perturb the currently-working provisional-member path (the STRAYACK census warns against widening the sysap-keyed trigger; this does not touch it). Reuses the existing cm_send_ack / scs_member_build_ack cat-0x04 emitter -- no new frame builder, FSM row, or credit-path change. Test added (test_disconnect_rsp_arms_and_a_cat04_clears_the_sequenced_ack): a real captured teardown arms the obligation on the VC recv_seq and a cat-0x04 clears it. All 26 vmsscs unit tests pass. Lab-adjudicated (necessary prerequisite; may not alone fire the commit if that also needs the full DLM rebuild): (1) 0x7b DISCONNECT_RSP retransmits 37->~2, (2) SDA Unacked->0, (3) barrier commit -> CLUSTER_NODES 2->3. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01PhM3QcmFEY3p8YNnHGaYwP --- src/vmsscs/scsd.c | 45 ++++++++++++++++++++++++ tests/vmsscs/test_scsd_wire.c | 65 +++++++++++++++++++++++++++++++++++ 2 files changed, 110 insertions(+) diff --git a/src/vmsscs/scsd.c b/src/vmsscs/scsd.c index 6d698683d..6cc3b41ed 100644 --- a/src/vmsscs/scsd.c +++ b/src/vmsscs/scsd.c @@ -1096,6 +1096,16 @@ struct peer_state { uint16_t sysap_acked; /* vms-760: highest peer send-msg# we have cat-0x04 acked */ long cm_acks; /* cat-0x04 acks emitted to this peer */ long cm_last_ack_ms; /* monotonic_ms() of our last cat-0x04 ack */ + /* vms-2f3: a peer's DISCONNECT_RSP advanced our VC recv_seq while OVMX is + * otherwise sequenced-silent to that peer. A standalone 0x48 credit-return + * (send_seq==0) does NOT clear VMS's "unacked" for a sequenced control frame; + * only a recv_ack piggybacked on a SEQUENCED frame (send_seq!=0) does. So OVMX + * owes the peer one sequenced cat-0x04 ack, else the peer retransmits its + * DISCONNECT_RSP every ~3s forever (SDA "Unacked messages 1") and the cluster + * transition never commits. Armed on the received DISCONNECT_RSP, fired by + * scsd_ack_flush_tick on the still-open CM VC, disarmed by any cat-0x04. */ + int vc_owe_seq_ack; + uint16_t vc_owe_seq_ack_seq; /* the VC recv_seq the owed ack must cover (for the log line) */ /* vms-760: cluster-wide state-transition barrier (spec 4p). */ uint32_t barrier_epoch; /* body[12:16] latched from the coordinator's op 0x09 */ uint8_t xition_class; /* vms-e4b: body[17] of the transition-open in @@ -5207,6 +5217,15 @@ static int scsd_disconnect_dialogue(int sock, int ifindex, struct peer_state *ps disc_req_recv++; } else { disc_rsp_recv++; + /* vms-2f3: a received DISCONNECT_RSP has already advanced ps->vc.seq.recv_seq + * (the credit block runs before this control dispatch). OVMX will otherwise + * answer only with a 0x48 credit-return, which VMS does NOT count as the ack + * for this sequenced frame -- so arm a SEQUENCED cat-0x04 recv_ack on the + * still-open CM VC (same per-peer VC) to cover it. Without this the peer + * retransmits the DISCONNECT_RSP every ~3s (Unacked=1) and blocks the + * transition commit; a real joiner's ongoing sequenced traffic covers it. */ + ps->vc_owe_seq_ack = 1; + ps->vc_owe_seq_ack_seq = ps->vc.seq.recv_seq; } /* Step 1. Answering a peer's DISCONNECT_REQ is an answer, not an @@ -6447,6 +6466,12 @@ static int cm_send_ack(int sock, int ifindex, struct peer_state *ps, ps->sysap_acked = ps->sysap_recv; ps->cm_last_ack_ms = monotonic_ms(); ps->cm_acks++; + /* vms-2f3: ANY cat-0x04 carries recv_ack=vc.seq.recv_seq, which satisfies a + * pending DISCONNECT_RSP sequenced-ack obligation -- so if an organic ack + * (op-0x06 burst etc.) already fired, the standalone disc-ack never emits. + * At most one extra cat-0x04 per teardown, and none when other sequenced + * traffic exists. */ + ps->vc_owe_seq_ack = 0; /* vms-584 STRAY-ACK INSTRUMENTATION. A 26-capture census of the * reference gives a rule we do NOT currently match: * @@ -6532,6 +6557,26 @@ static void scsd_ack_flush_tick(struct scsd_rx *rx, long now_ms) (now_ms - (long)ps->cm_last_ack_ms) >= (long)SCS_CM_ACK_FLUSH_MS) { cm_send_ack(rx->sock, (int)rx->ifindex, ps, rx->our_hw_mac, rx->our_src_logical); + } else if (ps->vc_owe_seq_ack) { + /* vms-2f3: OVMX owes this peer a SEQUENCED recv_ack for a received + * DISCONNECT_RSP (see peer_state.vc_owe_seq_ack). Fire PROMPTLY (no + * flush timer) so the peer's ~3s retransmit loop never even starts. + * cm_send_ack emits a cat-0x04 carrying recv_ack = vc.seq.recv_seq + * (>= the DISCONNECT_RSP send_seq) on the still-open CM VC -- the same + * per-peer VC the directory teardown rode -- and disarms the flag on + * success. NARROW BY CONSTRUCTION: armed ONLY by a DISCONNECT_RSP, + * keyed on the VC recv_seq, never on ps->sysap_recv -- so it is a + * different axis from the op-0x06/join ack cadence above and cannot + * perturb the currently-working path (STRAYACK census, cm_send_ack). */ + log_ts(stdout); + printf(" SCSD-I-DISCSEQACK, owe a sequenced recv_ack for a peer's" + " DISCONNECT_RSP (VC recv_seq=%u) -- emitting a cat-0x04 on the" + " CM VC so VMS counts it, else the peer retransmits +" + " Unacked=1 blocks the transition commit (vms-2f3)\n", + (unsigned)ps->vc_owe_seq_ack_seq); + fflush(stdout); + cm_send_ack(rx->sock, (int)rx->ifindex, ps, rx->our_hw_mac, + rx->our_src_logical); } } } diff --git a/tests/vmsscs/test_scsd_wire.c b/tests/vmsscs/test_scsd_wire.c index 89cf988dc..59c777819 100644 --- a/tests/vmsscs/test_scsd_wire.c +++ b/tests/vmsscs/test_scsd_wire.c @@ -9195,6 +9195,70 @@ static void test_matching_disconnect_rsp_closes_the_connection(void) "the full teardown scored %lu illegal events", conn_illegal_events); } +/* + * vms-2f3: a received DISCONNECT_RSP arms a SEQUENCED recv_ack obligation, and a + * cat-0x04 on the CM VC clears it. OVMX otherwise answers a peer's teardown only + * with a 0x48 credit-return (send_seq==0), which VMS does NOT count as the ack for + * the sequenced DISCONNECT_RSP -- so the peer retransmits it every ~3s (SDA + * "Unacked messages 1") and the cluster transition never commits. Confirmed on the + * lab-2 wire: OVMX went sequenced-silent to VAX1 after the teardown (0 sequenced + * frames +20->110s) while a real joiner's ongoing sequenced traffic covers it. + * This is that fix as a test: the DISC_RSP arms the obligation on the VC recv_seq, + * and any cat-0x04 (which carries recv_ack=vc.seq.recv_seq) satisfies + clears it. + */ +static void test_disconnect_rsp_arms_and_a_cat04_clears_the_sequenced_ack(void) +{ + struct rxworld r; + struct scs_cdt *cdt = disc_world_init(&r); + if (cdt == NULL) { + return; + } + rx_feed(&r, cap_disconnect_req_to_ovmx, sizeof(cap_disconnect_req_to_ovmx)); + if (scs_conn_state_of(cdt) != SCS_CONN_DISC_MATCH) { + return; + } + struct peer_state *rps = + peer_find_or_add(&r.w.cfg, &r.w.pdt, r.w.peers, ovmx760_member_mac); + CHECK(rps != NULL, "peer slot for the answering peer"); + if (rps == NULL) { + return; + } + /* No obligation yet, and no SYSAP ack owed -- so only the DISC_RSP path can + * arm the sequenced-ack obligation. */ + rps->vc_owe_seq_ack = 0; + rps->sysap_acked = rps->sysap_recv; + + rx_feed(&r, cap_disc_rsp_to_ovmx, sizeof(cap_disc_rsp_to_ovmx)); + + /* ARM (scsd_disconnect_dialogue): the received DISCONNECT_RSP advanced the VC + * recv_seq and armed the sequenced-ack obligation. A 0x48 credit-return alone + * does not clear VMS's unacked for a sequenced frame; a piggybacked recv_ack + * on a sequenced frame does. */ + CHECK(rps->vc_owe_seq_ack == 1, + "a received DISCONNECT_RSP did not arm the sequenced-ack obligation " + "(vc_owe_seq_ack=%d) -- OVMX would answer only with an uncounted 0x48 " + "credit and the peer would retransmit forever", rps->vc_owe_seq_ack); + CHECK(rps->vc_owe_seq_ack_seq == rps->vc.seq.recv_seq, + "owed-ack seq %u != advanced VC recv_seq %u", + rps->vc_owe_seq_ack_seq, rps->vc.seq.recv_seq); + + /* FIRE + DISARM: a cat-0x04 on the CM VC (what scsd_ack_flush_tick's new branch + * emits) carries recv_ack=vc.seq.recv_seq (>= the DISCONNECT_RSP send_seq) and + * clears the obligation, so it fires at most once per teardown. */ + rps->cm_local_conid = 0x08000002u; /* CM VC up, as when a teardown blocks the join */ + rps->cm_remote_conid = 0xD6000002u; + long acks_before = rps->cm_acks; + (void)cm_send_ack(r.rx.sock, r.rx.ifindex, rps, r.rx.our_hw_mac, + r.rx.our_src_logical); + CHECK(rps->cm_acks == acks_before + 1, + "the sequenced cat-0x04 ack did not go out (cm_acks %ld -> %ld)", + acks_before, rps->cm_acks); + CHECK(rps->vc_owe_seq_ack == 0, + "the cat-0x04 did not clear the sequenced-ack obligation " + "(vc_owe_seq_ack=%d) -- it would re-fire every tick", + rps->vc_owe_seq_ack); +} + /* * (3) THE p. 2-27 SIMULTANEOUS CASE. "When each node receives the * DISCONNECT_REQ from the other node, it replies with a DISCONNECT_RSP. It @@ -10999,6 +11063,7 @@ int main(void) * OLD gate would have refused outright. */ test_control_dispatch_survives_a_frame_the_legacy_marker_would_have_refused(); test_matching_disconnect_rsp_closes_the_connection(); + test_disconnect_rsp_arms_and_a_cat04_clears_the_sequenced_ack(); test_simultaneous_disconnect_sends_no_second_request(); test_shutdown_disconnects_every_open_connection(); test_clean_shutdown_kill_switch_through_the_daemon();