test(oracle): add Binance-only live soak suite - #807
Conversation
|
Wave 7 live validation update:
Post-fix live smoke PASS:
The PR remains draft until a fresh 24-hour run passes. |
|
Wave 7 soak update (second formal-run diagnosis and harness fix):
Live regression proof using Binance Futures testnet plus finalized Polygon data:
A fresh formal 24-hour run is starting from this commit. The PR remains draft until that run completes successfully. |
|
Wave 7 burn-in update The previous formal soak stopped after 4,615 seconds when numeric historical `eth_call` requests returned different state snapshots even though every replica first reported the same block hash. The suite now binds every NativeOracle/resolver read to the exact canonical hash with EIP-1898 (`blockHash` plus `requireCanonical`) in commit `c079007a4e`. Validation after the fix:
The PR remains draft while the fresh 24-hour run is pending. |
|
Multi-feed update pushed in `d259517172`. The live suite now runs independent NVDAUSDT (1001), BTCUSDT (1002), and ETHUSDT (1003) index-kline tasks through the same feedId-namespaced resolver. The prior single-feed 24-hour attempt was intentionally stopped after 8,349.273 healthy seconds to add this requested coverage. Post-change gates are PASS: a 305.971s high-frequency run with node4 restart (all feeds 6 advances; BTC/ETH 6 observed price changes each; recovery 11.136s), followed by a 1,803.466s burn-in (all feeds 31 advances; BTC/ETH 31 changes each; 4/4 relayer quorum; callback counts 33/33/33 and Polymarket 1). The first smoke exposed a cross-contract RPC view transition after restart. The final implementation preserves the strict nonce-to-bucket invariant and uses a progress/resolver/progress seqlock around EIP-1898 hash-bound calls. The rerun and burn-in both passed. A fresh three-feed 24-hour run is starting; the PR remains draft. |
Three-feed 24h attempt: 12h epoch/restart collision foundThe first formal three-feed run stayed healthy for 43,185.741 seconds (about 12h):
The run then failed at the scheduled node4 restart because the restart was aligned to the two-hour epoch boundary. This was not an Oracle-value divergence. During the epoch-8 reconfiguration, node3 timed out and received execution responses out of order: it learned the suffix-block timestamp before the actual reconfiguration-block timestamp, then panicked in Commit |
Epoch-guard live regression: PASSCommit
The repeated formal 24-hour run can now start from this head. The PR remains draft pending that gate. |
|
Wave 7 soak update:
The PR remains draft while a clean 24-hour rerun is in progress. |
|
Wave 7 restart validation update The second formal soak reached the 12-hour restart point with all Oracle invariants healthy. During the restart, the harness hit its generic 30-second RPC startup timeout even though node4 was still alive; its persisted database recovery reached config load at 46 seconds. This was a test-harness timeout, not an Oracle consensus or chain liveness failure. Fix pushed in
Regression result:
The earlier 12-hour run also exercised the bounded canonical snapshot retry once: an execution-view transition converged on the second attempt instead of producing a false failure. The PR remains draft. A fresh 24-hour run with the built-in 12-hour restart is the final acceptance gate. |
Formal soak update: restart passed, independent Aptos liveness blocker reproducedThe formal 24-hour run reached and successfully completed its built-in coordinated node4 restart:
This is in addition to the earlier 20-minute burn-in and 8-minute targeted restart regression, where node4 recovered in 11.164 seconds and 12.153 seconds respectively. The run later failed near hour 15 for an independent, already tracked Aptos JWK issue, so it is intentionally not recorded as a 24-hour PASS:
This reproduces Galxe/gravity-audit#910. Galxe/gravity-aptos#78 has now been rebased onto the current aptos-node base, reduced to the two-file JWK manager fix plus tests, and marked ready for review. The full aptos-jwk-consensus package test passes 2/2. This PR remains Draft. After #78 merges, this branch will pin the merged Aptos revision, run a short multi-restart regression, and then restart the formal 24-hour soak from a clean environment. |
|
Wave 7 focused multi-restart live regression: PASS. Test topology and data:
Results:
Residual observation: after all assertions passed, the final whole-cluster teardown stopped node4 last and exposed an existing graceful-shutdown panic in consensus_mempool_handler (futures select with all inputs completed and no complete handler). This did not occur during any coordinated node4 restart and does not invalidate the restart recovery result, but should be fixed separately. Release ordering remains: merge gravity-aptos #78, update the gravity-reth Aptos pin, then replace the SDK test-only coordinated pins with merged commits before the formal 24-hour run. |
|
Formal Wave 7 candidate 24-hour soak started at 2026-08-07 10:48:13 Asia/Shanghai. Configuration:
Initial evidence at 305 seconds: all four nodes at height 1654; all three price feeds at nonce 8 with six observed price changes and 4/4 relayer quorum; Polymarket nonce 1; no fatal/panic matches. Expected completion is approximately 2026-08-08 10:48 Asia/Shanghai. The PR remains draft until the final 24-hour assertions pass. |
Repeated validator restart stress: PASSA focused live regression completed successfully against the current Wave 7 candidate:
Measured restart recovery times:
After every restart, the old PID was confirmed stopped, RPC returned, the node caught up to the target block, all three relayer checkpoints caught up, and the next canonical snapshot retained 4/4 relayer quorum. Final assertions:
The second full 24-hour run has now started with the default 12-hour guarded restart retained. Its per-feed stall window is 900 seconds because the first run demonstrated a Binance testnet NVDA bucket being published/backfilled more than six minutes late; the full-day minimum advance requirement remains 80% of expected one-minute buckets. |
24-hour live soak resultOperational target: PASS.
The final pytest process reported No additional 24-hour rerun is required for the accepted operational objective; the pagination change is isolated to final evidence collection. |
ByteYue
changed the title
c576307 to
3b6e17b
Compare
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 3b6e17b6fb
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if nonce > last_nonces[pair]: | ||
| max_price_gaps[pair] = max( | ||
| max_price_gaps[pair], now - last_price_advances[pair] | ||
| ) | ||
| last_price_advances[pair] = now |
There was a problem hiding this comment.
When a feed resumes after exceeding ORACLE_SOAK_STALL_TIMEOUT_SECONDS, _run_soak records the oversized gap but immediately resets last_price_advances[pair] before the assertion, so the asserted elapsed time is zero and the release soak can pass despite violating its stall budget. This is especially reproducible when _restart_validator blocks heartbeat sampling: pause deliveries longer than the configured budget, allow one delivery before the next replicated snapshot, and the nonce-advance path masks the outage; the node-height loop below has the same reset-before-check behavior. Validate the elapsed gap before updating either timestamp so a recovery cannot erase a prolonged stall.
AGENTS.md reference: AGENTS.md:L5-L6
Useful? React with 👍 / 👎.
Wave 7 manual live soak result: PASSTested PR head: A 4-validator Gravity devnet completed the Binance-only live oracle soak against real Binance testnet data. Result
Live price feeds
All three feeds repeatedly reached the required relayer quorum across all four validators and continued advancing after the restart. Coordinated restart coverage
No panic, chain divergence, relayer stall, nonce regression, or feed timeout was observed. The 8 emitted warnings were existing Python Verdict: PASS. The PR head completed the intended two-hour real-data, multi-validator live soak including the coordinated validator restart and post-restart recovery. |
Add one opt-in four-validator suite that discovers live Binance Futures testnet index-price buckets and a finalized Polygon Polymarket settlement before deployment, activates both tasks through governance at an epoch boundary, and verifies JWK quorum through execution-layer callbacks. The suite monitors replica convergence, monotonic source progress, exact Binance closed-bucket prices, relayer quorum, immutable Polymarket settlement, callback counts, and node4 restart recovery. Generated source mappings and metadata are ignored and removed on teardown, including deployed relayer configs. Validated with: - 3-minute live smoke: 5 bucket advances, 6 price callbacks, 1 settlement callback - 30-minute burn-in: 1,802.6s, 119 samples, 32 advances (24 required), 76.11s max price gap, 33 price callbacks, 1 settlement callback - 3-minute restart probe: node4 recovered in 11.17s, 4 bucket advances, 5 price callbacks, 1 settlement callback The documented 24-hour run restarts node4 halfway and requires at least 80% of expected minute advances.
Pin gravity-reth to the merged PR #423 commit and the Binance E2E genesis inputs to the merged contracts PR #116 commit. Regenerate Cargo.lock from the updated reth revision.\n\nValidated with:\n- CARGO_BUILD_JOBS=2 RUSTFLAGS='--cfg tokio_unstable' cargo check -p gravity_node --locked -j 2\n- oracle_live_soak helper tests (10 passed)
3b6e17b to
779085f
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 779085f323
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| (base_dir / node["id"] / "config" / "relayer_config.json").unlink( | ||
| missing_ok=True |
There was a problem hiding this comment.
When this suite fails under runner.py oracle_live_soak --no-cleanup, run_test_suite deliberately leaves the nodes running but still invokes post_stop; this unlink then removes each live node's relayer configuration (and lines 138-139 remove the metadata needed by --resume). A subsequent validator restart loads an empty relayer config and can no longer fetch any gravity://3/... task, defeating both post-failure debugging and resume validation. Reproduce by forcing a pytest assertion, running with --no-cleanup, inspecting the deleted config, and restarting one validator; cleanup should only remove these files after the cluster is actually stopped.
AGENTS.md reference: AGENTS.md:L8-L8
Useful? React with 👍 / 👎.
| try: | ||
| summary = await _run_soak( |
There was a problem hiding this comment.
The failure-reporting try starts only around _run_soak and ends before callback-event validation, so a callback-count mismatch at the final gate raises without writing oracle_live_soak_summary.json; setup and activation failures before this line have the same behavior. Because pre_deploy removes any previous summary, consumers expecting the documented machine-readable final report receive no artifact for these important failure modes. Expand the reporting scope so the setup and post-soak assertions also produce a failed summary.
Useful? React with 👍 / 👎.
2 participants
Summary
Add one opt-in, non-gating Wave 7 live soak suite for the initial Gravity Oracle launch profile:
NVDAUSDT(feed ID 1001),BTCUSDT(1002), andETHUSDT(1003), allsourceType=3;The suite is excluded from the runner default test set and must be named explicitly.
Packed Price Feed V1 compatibility
This branch is rebased onto current SDK
mainand pins the exact candidate revisions from:d90d81583b95adac73e86c8aafd27530414df03f;e63817884bd015b73a9d3568828e4f50a07cd83a.The E2E assertions now consume the packed resolver getter
(uint32 roundId, uint48 resolvedAtMs, uint96 price). A shared golden vector checks the exact 32-byte Price Feed V1 body and the exact 192-byte NativeOracle callback wrapper, down from the previous 320-byte wrapper.Binance-only release profile
This PR implements the release boundary in Galxe/gravity-audit#1093:
[1, 3], not source type 6;pre_deploywrites exactly threegravity://3/...Binance URI mappings;The merged deterministic Polymarket transport suite remains separate regression coverage. This PR does not revert generic source type 6 contracts or runtime code; it keeps that transport dormant through configuration absence.
Architecture exercised
pre_deployselects one common closed-minute anchor for all three Binance pairs.Generated mappings and metadata are removed on teardown, including copies deployed into node config directories.
Acceptance checks
gravity://3/...mappings are active.Validation
Build and focused tests
make MODE=quick-release gravity_node gravity_cli: PASS;git diff --check: PASS;Live Binance smoke
Two-hour packed V1 gate
Reproduce the two-hour gate
The Binance Futures testnet
indexPriceKlinesendpoint is public and does not require an API key. Binance values are testnet index data and must be labeled that way in demos. Runtime evidence stays in the ignored suite artifacts directory. The runner performs global localgravity_nodecleanup, so it must not run beside another local Gravity cluster.Tracks Galxe/gravity-audit#1038, Galxe/gravity-audit#1092, and Galxe/gravity-audit#1093.