PBFT TCP integration tests time out before commit and checkpoint

[blacks1ne]

Summary

Two PBFT-over-TCP library tests consistently time out in the current checkout, including when each is run in isolation with loopback networking available:

• runtime::pbft_net::tests::pbft_coordinator_survives_primary_failure
• runtime::pbft_net::tests::lagging_replica_catches_up_via_checkpoint_state_transfer

This leaves the advertised primary-failover and checkpoint state-transfer paths without a green end-to-end test.

Environment

• Commit: da9f90f989cb358c489461afadb47885cac23280
• Linux
• Rust test profile, dependencies resolved from the checked-in Cargo.lock

Reproduction

cargo test -p balance-lang \
  runtime::pbft_net::tests::pbft_coordinator_survives_primary_failure \
  -- --exact --nocapture

cargo test -p balance-lang \
  runtime::pbft_net::tests::lagging_replica_catches_up_via_checkpoint_state_transfer \
  -- --exact --nocapture

The full workspace run also reproduces both failures:

cargo test --workspace

Actual behavior

The primary-failure test fails before the primary is stopped; its initial write under the healthy primary never commits:

first write should commit: Err("pbft: operation on 'a' did not reach 2 executions across 4 replicas (timeout)")

The isolated test fails after roughly 16 seconds at crates/balance-lang/src/runtime/pbft_net.rs:1094.

The checkpoint catch-up test fails before reviving the lagging replica:

the 3 live replicas should commit 4 ops and checkpoint

The isolated test fails after roughly 10 seconds at crates/balance-lang/src/runtime/pbft_net.rs:1326.

In the full workspace run, CLI unit tests passed (19/19), persistence integration tests passed (4/4), program-corpus tests passed (2/2), and the library suite reached 752 passed / 2 failed before Cargo exited non-zero.

Expected behavior

• A four-replica cluster should commit the initial write under its healthy primary.
• After stopping the primary, the remaining replicas should perform a view change and commit the next write.
• Three live replicas should commit four operations and form the checkpoint required for later state transfer.
• cargo test --workspace should complete without these timeouts.

Notes for investigation

• pbft_coordinator_survives_primary_failure failing on the first write suggests the problem precedes the fault-injection/view-change portion.
• lagging_replica_catches_up_via_checkpoint_state_transfer failing before replica revival suggests the initial three-node live quorum is not making normal-case progress.
• Both tests use LocalSet plus loopback TCP drivers. Since the failures reproduce individually, they are not solely caused by parallel test interference or port collisions.
• Other PBFT network tests passed in the same full run, including normal-case TCP, coordinator-driven cluster, and primary view-change tests, so compare setup/timing and client submission paths between those cases.

Acceptance criteria

1. Both isolated commands pass repeatedly.
2. The full workspace suite passes with normal parallel test execution.
3. Failure diagnostics expose replica/view/sequence progress if either bounded wait expires again.