feat(0038): Prices Ledger Processor — SNS-fan-out doorbell + ingest CDK#34
Open
karczuRF wants to merge 12 commits into
Open
feat(0038): Prices Ledger Processor — SNS-fan-out doorbell + ingest CDK#34karczuRF wants to merge 12 commits into
karczuRF wants to merge 12 commits into
Conversation
Prep for adding a notes/ subdir. Rename README + shift relative links one level deeper to match the new path depth.
G-note describing the local-only Lambda binary scope (Part A), explicit non-goals (Part B), the cross-team contract agenda for the BE conversation (Part C), open questions (Part D), and the production-rewrite punch list once the engineering gates (BE 0227 mTLS endpoint, task 0047 throughput verification) clear (Part E).
Incorporate the production patterns from soroban-block-explorer's crates/indexer/ that the prices-api Lambda must mirror: doorbell- cursor reconcile loop with reservedConcurrentExecutions=1 for ordering correctness, S3->SQS trigger (body ignored), Galexie key derivation, [50,200,800]ms retry envelope, safe-error redaction, and mTLS via Parameters and Secrets Lambda Extension. Major revisions: - New section 1 — load-bearing vs chosen vs not-inherited patterns from BE, with explicit code-path citations. - Part A rewritten as doorbell-cursor loop with three trait seams (Cursor, ObjectFetcher, OhlcvSink) sized for production swap. - Part A.4 expands xdr-parser distribution model into four options with a recommendation (git Cargo dep pinned to sha for prototype, tag-pinned for production). - Part C.1 pivots — register our SQS queue as second notification target on BE bucket, not our Lambda directly. - Part C.2 corrects SSM-at-runtime to CDK-time SSM reads baked into Lambda env vars. - Parts C.5 and C.6 added — reuse of db-clickhouse::mtls, Caddyfile CN_USER_MAP for prices-api CNs. - Part D.1 added — cursor source design question (own table vs cross-DB read vs S3-derived) with a stake.
Add packages/prices-ledger-processor — the local-only prototype of the Prices Ledger Processor Lambda. Mirrors BE's production indexer shape (doorbell-cursor reconcile, deterministic Galexie key derivation, [50,200,800]ms retry envelope, safe-error redaction) but runs against fixture files on disk through trait-seam abstractions that swap to AWS in production. Three trait seams sized for production swap: - ObjectFetcher (LocalDiskFetcher → aws_sdk_s3::get_object) - Cursor (StubFileCursor → CH-backed cursor per G-note D.1) - OhlcvSink (StdoutJsonSink / SqlFileSink → clickhouse::Client) The reconcile loop reads the cursor, derives the next S3 key, fetches via the trait, decodes via an injected LedgerDecoder, calls ledger_processor::dispatch per (tx_id, contract_id) group, buckets trades into 1-min OHLCV per ADR 0004, writes to the sink, and advances the cursor LAST (the ordering barrier mirroring BE's "ledgers row written last" pattern). Phase 1 simplifications flagged for the BE meeting: - LedgerDecoder is a no-op in the binary; tests use a fake decoder that produces ledger-sequence-only DecodedLedger values. The real xdr-parser walk lands in Phase 2. - Canonical (base, quote) is lexicographic on (token_in, token_out) — placeholder, not a production policy. - Prices are f64; volumes are i128. 27 tests pass (23 unit + 4 e2e). Workspace builds clean; clippy clean on the new crate.
Wire the Phase 1 scaffolding to BE's actual xdr-parser crate
(workspace dep already pointing at soroban-block-explorer's develop
branch). The reconcile loop now decompresses .xdr.zst payloads,
deserializes LedgerCloseMetaBatch via stellar-xdr, walks V0/V1/V2
tx_processing, calls xdr_parser::extract_events per tx, adapts the
returned tagged JSON to extractors-core::TaggedValue, and groups
by (tx_id, contract_id) for dispatch.
The event filter keeps TxLevel + PerOp (Protocol 23+) sources
and drops Diagnostic — diagnostic events can include byte-identical
Contract-typed mirrors of consensus events (BE task 0182), which
would double-count if ingested.
Binary topology now matches the spec's two modes:
- src/main.rs is the Lambda entrypoint named `prices-ledger-processor`
— wires an SqsEvent handler that ignores the doorbell body, runs
reconcile, and returns SqsBatchResponse with per-message
batch-item-failures on hard errors (mirrors BE's pattern).
- src/bin/cli.rs is the CLI binary named `prices-cli` for local
iteration via `cargo run --bin prices-cli`.
JSON→TaggedValue adapter covers the AMM-relevant ScVal shapes
(sym, string, address, i128, u128, vec, map); unsupported types
collapse to Null. 7 new unit tests verify the adapter on direct
and nested shapes.
Manual smoke: copied 3 contiguous BE ledgers (62460540-62460542)
into the gitignored fixtures dir and ran:
prices-cli --cursor 62460539 --max-iterations 5 \
--fixtures-dir packages/prices-ledger-processor/fixtures/ledgers \
--cursor-file /tmp/prices-smoke-cursor.txt
Result: 3 ledgers processed end-to-end, cursor advanced
62460539 → 62460542, "reached gap on S3" at 62460543 as expected.
Zero trades emitted because VenueRegistry is empty (no Phoenix
pools registered) — kernel correctness, not decode behaviour;
populating the registry is configuration work tracked separately.
Workspace builds clean, clippy clean on this crate, 34 tests pass
(23 Phase 1 unit + 7 new decode + 4 e2e).
Local-prototype deliverables shipped on branch feat/0038_prices-ledger-processor-lambda (PR #34): the design spec, Phase 1 scaffolding, and Phase 2 real XDR decode. Task now blocks on the cross-team conversation that resolves Part C of notes/G-local-prototype-spec.md — SQS notification ownership, env-var injection contract, xdr-parser pinning + semver, db-clickhouse::mtls reuse, Caddyfile CN_USER_MAP, and cert issuance. Original engineering gates (BE 0227, task 0047) also remain open. Unblocks when the meeting answers Part C and either gating event clears.
The 2026-06-10 cross-team meeting resolved Part C.1 of the
local-prototype spec (SQS notification ownership) in favour of SNS
fan-out: BE repoints its bucket notification from S3->SQS to
S3->SNS->SQS, and prices-api owns its own prices-ingest queue + DLQ
subscribing to BE's topic. The doorbell-cursor reconcile loop is
unaffected (the Lambda ignores the message body), so this is a
documentation + narrative change only.
Move task 0038 blocked -> active, add the decision history entry,
resolve spec C.1 inline with the final topology, and add the new
/platform/{env}/ledger-events-topic-arn handshake key to C.2.
Wire the live Prices Ledger Processor Lambda and its doorbell source
into ComputeStack, mirroring BE's compute-stack.ts: a prices-owned
prices-ingest-{env} SQS queue + DLQ subscribed to BE's ledger-events
SNS topic (rawMessageDelivery), the ledger-processor Lambda
(ARM64/provided.al2023, reservedConcurrency=1, batchSize=1,
visibility=timeout+60s, maxReceiveCount=10), the event-source-mapping,
and IAM (S3 read on BE's bucket, CloudWatch lag metric, X-Ray). The
env-var contract is sourced from /platform/{env}/* SSM at deploy,
including the new ledger-events-topic-arn key from the SNS decision.
The wiring lives in ComputeStack rather than a separate stack: the
ESM and queue/bucket grants mutate the Lambda role's policy, so a
split creates a CloudFormation dependency cycle (BE keeps the same
single-stack shape). Uses lambda.Function + Code.fromAsset since the
infra does not carry cargo-lambda-cdk; RustFunction is a follow-up.
Adds a ledgerProcessor config block to EnvironmentConfig with
reservedConcurrency pinned to exactly 1 in validateConfig — serial
execution is the ordering guarantee, not a tunable.
Prepare-only: deploy is gated on BE 0227 + task 0047 + BE publishing
the platform SSM keys/topic. nx build + cdk synth both pass.
karczuRF
changed the title
The 2026-06-10 meeting resolved the SNS-vs-direct-notification
question (task 0038 §C.1) in favour of SNS fan-out. Add a BE-facing
note that specs item 1 of 0050 concretely: the stellar-ledger-data
compute-stack.ts change (topic + repoint to SnsDestination + own-queue
re-subscribe with rawMessageDelivery), the canonical
/platform/{env}/ledger-events-topic-arn SSM key, the same-account
"no cross-account policy / no prices queue ARN" notes, and a cutover
plan. Grounded in BE's compute-stack.ts and 0038's CDK (PR #34).
Convert 0050 to directory form, add a history entry, and flag that the
SNS item can ship independently of BE 0227 / task 0047.
karczuRF
added a commit
that referenced
this pull request
Jun 11, 2026
The 2026-06-10 meeting resolved the SNS-vs-direct-notification
question (task 0038 §C.1) in favour of SNS fan-out. Add a BE-facing
note that specs item 1 of 0050 concretely: the stellar-ledger-data
compute-stack.ts change (topic + repoint to SnsDestination + own-queue
re-subscribe with rawMessageDelivery), the canonical
/platform/{env}/ledger-events-topic-arn SSM key, the same-account
"no cross-account policy / no prices queue ARN" notes, and a cutover
plan. Grounded in BE's compute-stack.ts and 0038's CDK (PR #34).
Convert 0050 to directory form, add a history entry, and flag that the
SNS item can ship independently of BE 0227 / task 0047.
Self-contained, copy-pasteable runbook for the BE team to implement the
S3 -> SNS -> SQS fan-out on stellar-ledger-data. Grounded in the current
soroban-block-explorer compute-stack.ts: real line refs (notification at
L386-389), the three missing imports, and the /platform/{env}/* SSM keys
framed as net-new (BE publishes none today). Cross-links the existing
G-be-sns-fanout-ask rationale note rather than duplicating it.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Live ingestion for prices-api: the Prices Ledger Processor Lambda, mirroring BE's indexer (doorbell-cursor, in-order, S3→SNS→SQS).
Scope (in order added)
1. Local prototype (Phase 1–2)
packages/prices-ledger-processormirrors BE's indexer structure with three production swap-seams (ObjectFetcher,Cursor,OhlcvSink). The doorbell-cursor reconcile loop reads the cursor, derives the Galexie S3 key forcursor+1, fetches, decodes, dispatches via the 0037 kernel, buckets to 1-min OHLCV, and advances the cursor last — the ordering barrier. Runs against local fixtures.2. SNS fan-out decision (2026-06-10 cross-team meeting)
Resolves spec §C.1 (SQS notification ownership). BE repoints its bucket notification
S3→SQS→S3→SNS→SQS(rawMessageDelivery, indexer parser unchanged); prices-api owns its ownprices-ingest-{env}queue + DLQ subscribing to BE's topic — failure isolation. Topology + handshake recorded innotes/G-local-prototype-spec.md§C.1/§C.2; topic ownership tracked by task 0050. The reconcile loop is unaffected (doorbell body is ignored).3. Ingest CDK wiring (prepare-only)
Folded into
infra/.../compute-stack.ts(single stack avoids a CFN dependency cycle, matching BE): prices-owned SQS + DLQ, SNS subscription to BE's importedledger-eventstopic, the ledger-processor Lambda (ARM64/provided.al2023,reservedConcurrency=1,batchSize=1,visibility=timeout+60s,maxReceiveCount=10), event-source-mapping, and IAM (S3 read on BE's bucket, CloudWatch lag metric, X-Ray). Env-var contract from/platform/{env}/*SSM at deploy (incl. the newledger-events-topic-arnkey).ledgerProcessorconfig block added withreservedConcurrencypinned to 1 invalidateConfig.Verification
cargo test -p prices-ledger-processor✅ (4/4 — incl. contiguous-order + idempotency)nx build+cdk synth Prices-production-Compute✅ (2 SQS, 1 SNS subscription, 1 queue policy, 1 Function, 1 ESM; no cycle)Not in scope / gated
ObjectFetcher, CH-backed cursor (D.1), mTLS CHOhlcvSink, adoptcargo-lambda-cdkRustFunction, lag alarm.Task: 0038 (active)