fix(horizon): honour shutdown, resume stream cursor, skip unhashed records, persist asset_issuer - #584
Merged
Markadrian6399 merged 2 commits intoAug 27, 2026
Conversation
…cords, persist asset_issuer Closes StellarGateLabs#226 Closes StellarGateLabs#228 Closes StellarGateLabs#224 Closes StellarGateLabs#223 > [!WARNING] > This branch is implemented but untested. No tests were added and no test > suite was run — deliberately, per the request. cargo build is clean. > The tests each issue asks for are not in this branch. Do not merge without > running the full suite. poll_once looped over Horizon pages until caught up without ever checking the shutdown signal, so SIGTERM during a backlog drain was ignored until it finished or the 30s grace killed the task mid-page. The receiver is now threaded into poll_once and checked at every page boundary, immediately after the cursor checkpoint. POLL_MAX_PAGES_PER_CYCLE (default 50, 0 = unlimited) bounds one cycle even with no shutdown pending. run_stream_listener hard-coded cursor=now on every process start, throwing away the resume the poller persists. Payments that landed while the process was down were invisible to the ~1s path and recoverable only by the poller. The stream now seeds from its own persisted cursor (horizon_stream_cursor), falls back to the poller's (horizon_payment_cursor), and only then to the live edge. The two keys are separate so neither listener drags the other. reconcile_payment defaulted a missing transaction_hash to the empty string and used it as half of the processed_transactions primary key, so two different unhashed transactions looked like the same one and the second was silently discarded as already credited. Such records are now skipped and counted; the poller re-sees them on the next cycle. The schema enforces a non-empty tx_hash via CHECK. payments stored only the asset code, leaving the issuer in process config: editing ACCEPTED_ASSETS retroactively changed what historical rows meant. asset_issuer TEXT is added to payments (NULL = native), resolved at create time and persisted so the pair is a stable, complete asset identity. Exposed in GET /payments/:id (owner view) and every webhook payload. verify() matches against the stored issuer rather than today's config.
|
@Manuel1234477 Great news! 🎉 Based on an automated assessment of this PR, the linked Wave issue(s) no longer count against your application limits. You can now already apply to more issues while waiting for a review of this PR. Keep up the great work! 🚀 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Four defects in the Horizon settlement path. Three of them (#226, #228, #224) only
bite on a restart or an unexpected payload, which is why they survived; the fourth
(#223) is a missing column that makes historical rows unauditable.
Closes #226
Closes #228
Closes #224
Closes #223
Warning
This branch is implemented but untested. No tests were added and no test
suite was run — deliberately, per the request.
cargo build,cargo clippy --all-targetsandcargo fmt --checkare clean, and the existing suite wasupdated only where a struct literal needed the new field to keep compiling.
The tests each issue asks for (a multi-page catch-up against a mock Horizon, a
restart-gap stream test, a
transaction_hash: Noneunit test) are not inthis branch and are still owed. Do not merge without running the suite.
#226 —
poll_oncenow observes the shutdown signalpoll_oncepaged forward until caught up without ever checking shutdown. Agateway down for a while faces a catch-up measured in minutes — each page is up
to 200 records, each meaning DB writes and an outbound webhook with retries —
and none of it saw SIGTERM. The task was killed mid-page once the 30 s grace
expired, replaying the unfinished page on the next boot and making the next
shutdown worse.
poll_onceand checked at the top of every pageiteration — i.e. immediately after the previous page's cursor checkpoint, so
returning early never loses processed work.
POLL_MAX_PAGES_PER_CYCLE(default50,0= unlimited) bounds onecycle even with no shutdown pending, so a catch-up cannot monopolise the
poller task indefinitely. The next cycle resumes from the checkpoint.
#228 — the stream listener resumes from a persisted cursor
run_stream_listenerhard-codedcursor = "now"on every process start,discarding exactly the resume the poller works to persist. The advertised ~1 s
settlement latency therefore did not hold across a deploy: the gap was left to
the poller, whose catch-up is slower and which is disabled outright in
STELLAR_LISTENER_MODE=poll.horizon_stream_cursor),falls back to the poller's (
horizon_payment_cursor), and only then to thelive edge — so only a genuinely fresh database baselines at
now.kv_statekeys, so neither listener dragsthe other backwards. The stream persists its cursor as each event advances it;
a write failure is logged and tolerated (the poller remains the backstop).
idempotent through
processed_transactions.#224 — records with no
transaction_hashare skipped, never creditedreconcile_paymentdefaulted a missing hash to""and used it as half of theprocessed_transactionsprimary key. Two different unhashed transactionstherefore looked like the same transaction, and the second was discarded as
"already credited" when it never had been: money on chain, never credited to the
merchant. The empty string also landed in
payments.tx_hash, pointing themerchant-visible record at nothing.
warn!, and the poller re-sees it on thenext cycle — so skipping is safe and self-healing.
stellargate_horizon_records_skipped_total{reason="missing_tx_hash"}on
/metrics, so operators can see it happening.processed_transactions.tx_hashcarries aCHECK (tx_hash <> '')for newdatabases, and
record_processed_txrefuses an empty hash outright.amount they carry was really received, and dropping the row would silently
reduce an intent's paid total. They need reconciling by hand.
#223 —
payments.asset_issuerThe table stored only the asset code, so which USDC a row referred to lived in
process configuration: editing
ACCEPTED_ASSETSretroactively reinterpretedevery historical row,
"asset": "USDC"in a webhook did not say which USDC,and after an incident there was no way to prove what an intent was priced in.
asset_issuer TEXTadded topayments(nullable — NULL means native).payments::createresolves the wholeAcceptedAssetand persists both halves.GET /payments/:id(owner view) and in every webhook payload;documented in
openapi.yaml,README.mdandWEBHOOK_REFERENCE.md.verify()now matches against the issuer recorded on the intent, fallingback to configuration only for pre-migration rows — so editing
ACCEPTED_ASSETSno longer changes what a historical row means.kv_statemarker rather than re-running every boot (re-running would let alater config edit rewrite history a second time). Best-effort by nature, and
documented as such.
Notes for review
poll_once's signature changed to take&watch::Receiver<bool>;run_polleris the only caller.
db::Payment,db::NewPayment,AppStateandmetrics::rendereach gained afield/parameter, which is most of the diff's line count in the test files.
db::backfill_asset_issuersis called frommainrather thandb::migratebecause it needs the accepted-asset allow-list.