feat(supply): add off-consensus IOTX total-supply conservation observer

[raullenchai]

Summary

This PR adds two tiers of IoTeX's total-supply conservation monitoring, all off-consensus and non-fatal:

• L2 — per-block exact tracker. A per-block running total (R1 + R2 + R3) maintained from the already-produced state-diff write queue. IOTX has no post-genesis mint path, so any single block whose supply increases is the ex-nihilo mint signature, caught to 1 rau — no loose cap slack.
• L3 — periodic reconciliation. An opt-in, read-only observer that periodically recomputes the total supply and checks it never exceeds the genesis endowment.

Motivation: the 2026-08 Harmony "empty block" incident let an attacker mint ~40B native tokens (≈26% of supply) out of thin air. IoTeX's supply can never exceed its genesis endowment, so recomputing the circulating supply against that cap is a cheap early-warning for the same class of bug.

Why a gated observer, not a per-block scan

This observer walks the entire Account namespace, which holds the state-factory read lock and would stall block commits (liveness) if run every minute on every validator. It is therefore opt-in via blockchain.SupplyCheckConfig (default disabled) and intended to run daily / per-epoch on a dedicated auditor node — not on ordinary validators.

It is deliberately off-consensus and non-fatal:

• never rejects a block or stops consensus → cannot brick the chain;
• only logs Error and exposes Prometheus gauges on violation;
• no state-transition change → no hardfork.

Invariant

R1 = sum(genuine primary account balances)   // Account namespace (incl. EVM contracts)
R2 = rewarding fund.totalBalance             // Rewarding namespace
R3 = staking bucketPool.total.amount         // Staking namespace
Total = R1 + R2 + R3 <= genesisTotal         // upper bound (supply only decreases)

genesisTotal is derived programmatically from genesis config (not hardcoded).

Review-driven rework (per @envestcc's review on this PR)

The reviewer flagged two merge-blockers in the first version; both are fixed:

1. P0 — crash / over-count. The Account namespace also holds legacy Poll/Rewarding states (pre-Greenland, never deleted). Feeding them to state.Account.Deserialize panics (unknown account type / invalid balance) and, where it doesn't, silently over-counts bogus balances via wire-tag collisions (Fund/Admin → inflated R1). sumPrimaryBalances now decodes entries with a strict, non-panicking local decoder (decodeAccount) that skips anything that isn't a genuine, canonically-encoded account — so the scan can neither crash the node nor false-positive on those legacy payloads.

2. Liveness stall. The full scan now runs only when enabled in config, at a configurable interval, on opt-in auditor nodes — not every minute on every validator.

Tests cover the strict decoder accepting every genuine account shape (EOA / funded / ZERO_NONCE / contract) while rejecting legacy Fund, Admin, Exempt, rewardAccount and staking TotalAmount, plus an end-to-end Observer.Check against a namespace poisoned with those legacy states proving it neither panics nor over-counts.

L2 (in this PR) / L1 (next phase), per @envestcc's review shape

• L2 — per-block exact delta (in this PR). supplychecker.SupplyTracker is wired into the state factory's StateDiffCallback (invoked after the factory mutex is released, so it can't stall commit) and maintains the running R1+R2+R3 total, asserting each block's net delta is <= 0. WriteQueueEntry now carries the prior value (gated behind a registered consumer so ordinary validators pay no extra disk lookups), which is what lets the exact per-key delta be computed with no extra state read.
• L1 — journal↔ledger reconcile (next phase). Reconcile per-address TransactionLog deltas against the actual state writes. Any mismatch = a balance change no action authorized. The prior-value capture it needs is now in place.

Acceptance gate for L1: a bolt-backed mainnet replay asserting the invariant at every height. This PR adds a real-factory.Factory bolt integration test (TestBoltBackedSupplyConservation) closing the earlier gap that the in-memory fakeStateReader only ever inserted state.Account.

Changes

• supplychecker/: new package —
  • L3 observer (Check, Run), strict non-panicking account decoder, genesis-cap derivation, per-reservoir readers.
  • L2 per-block tracker (tracker.go) wired to the factory's state-diff callback.
• blockchain/config.go: SupplyCheckConfig (opt-in L3 flag + interval, default off).
• chainservice/: wire the L2 tracker + L3 observer, gated by SupplyCheckConfig.
• state/factory/: WriteQueueEntry now captures the pre-block PriorValue (enabling exact per-address deltas); AddDiffCallback lets the tracker coexist with ioSwarm's callback.

Tests

go test ./supplychecker/ ./chainservice/ ./state/factory/ ./ioswarm/

All pass — including the bolt-backed TestBoltBackedSupplyConservation against a real factory.Factory (L2 tracker + L3 observer + v2 namespace keys), per-block delta conservation/mint-detection unit tests, and a factory-level PriorValue-gating test.

---

Follow-ups / self-review

• Missing reserve states are treated as zero (sound: under-counts, never a false violation).
• Codecov patch coverage kept high; duplication from a shared parseDecimal helper removed.

[codecov]

Codecov Report

❌ Patch coverage is 78.75817% with 65 lines in your changes missing coverage. Please review.
✅ Project coverage is 61.30%. Comparing base (436e4d1) to head (18170a0).
⚠️ Report is 321 commits behind head on master.

Files with missing lines	Patch %	Lines
supplychecker/tracker.go	76.99%	14 Missing and 12 partials ⚠️
supplychecker/supplychecker.go	82.87%	14 Missing and 11 partials ⚠️
state/factory/statedb.go	7.69%	12 Missing ⚠️
chainservice/builder.go	0.00%	1 Missing ⚠️
chainservice/chainservice.go	93.75%	1 Missing ⚠️

❌ Your project check has failed because the head coverage (61.30%) is below the target coverage (85.00%). You can increase the head coverage or adjust the target coverage.

Additional details and impacted files

@@             Coverage Diff             @@
##           master    #4971       +/-   ##
===========================================
- Coverage   74.83%   61.30%   -13.54%     
===========================================
  Files         378      469       +91     
  Lines       31624    45775    +14151     
===========================================
+ Hits        23666    28061     +4395     
- Misses       6747    14350     +7603     
- Partials     1211     3364     +2153     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[envestcc]

Nice motivation — supply conservation is absolutely worth monitoring. But I think the current approach has a couple of problems that make it unsafe to merge as-is, and I'd suggest a different shape for the check.

The full scan is too expensive to run every minute.

States(NamespaceOption(AccountKVNamespace)) walks the entire Account bucket. readStates calls Filter(ns, alwaysTrue, nil, nil), and BoltDB.Filter copies every key and value into memory inside a single bolt read transaction. On mainnet that's millions of entries, allocated fresh every 60 seconds.

And it does that while holding a lock that blocks block commit.

stateDB.States holds sdb.mutex.RLock() for the whole scan (statedb.go:588), while PutBlock takes sdb.mutex.Lock() across ws.Commit() (statedb.go:528). So a long scan stalls block processing. The PR says it can't affect consensus — it doesn't reject blocks, but it can definitely stall them, and for liveness that's the same thing.

The Account namespace isn't only accounts — and this one actually crashes the node.

state/tables.go:14-24 documents that the Account namespace also holds legacy Poll and Rewarding states. Those were written pre-Greenland via putStateV1 and are never deleted (stateCheckLegacy still falls back to reading them).

Decoding them as state.Account doesn't error — it silently produces a bogus balance, because rewardingpb.Fund field 2 (unclaimedBalance) and accountpb.Account field 2 (balance) have the same field number and wire type. I tested it:

Fund{unclaimedBalance:"1e26"}   -> err=<nil>  Account.Balance = 1e26
Admin{epochReward:"12500", ...} -> err=<nil>  Account.Balance = 12500

So R1 is over-counted, which inverts the "under-counting only, never a false positive" argument in the description.

Worse: the genesis rewarding admin state is written at height 0 (mainnet Greenland is 6544441, so UseV2Storage is false there) with productivityThreshold: 85. That's field 7, and accountpb.Account field 7 is the AccountType enum, which only allows {0,1}. Account.FromProto panics on anything else:

>>> PANIC: unknown account type

There's no recover() anywhere on that path and Run is launched as a bare goroutine, so this takes the whole process down. On mainnet the node would start, tick after 60s, hit that key, and die — in a loop.

If you keep any version of the scan, please don't hand raw bytes to state.Account.Deserialize — it's written for trusted input and panics by design. A local strict decoder works well; I tested one that checks len(pb.ProtoReflect().GetUnknown()) > 0, validates the enum and balance instead of panicking, and requires a canonical re-marshal round-trip. It accepts every real account shape (EOA, funded, contract with Root/CodeHash, ZERO_NONCE) and rejects Fund, Admin, rewardAccount, Exempt, CandidateList, and staking TotalAmount.

A suggestion on the check itself.

Beyond the implementation issues, total <= genesisCap has slack equal to the cumulative base-fee burn, which grows forever. An attacker minting less than that is invisible. It only catches Harmony-scale events.

IOTX has no post-genesis mint path at all — the rewarding fund is pre-seeded, fees are transfers, and base fee is the only burn (protocol.go:112). So you can check an exact equality per block instead of a loose bound, with no state scan:

L1 — journal ↔ ledger reconciliation (per block, O(block write set)). Every balance movement already emits a TransactionLog with a sender and a recipient: NATIVE_TRANSFER, IN_CONTRACT_TRANSFER, GAS_FEE, PRIORITY_FEE, BLOB_FEE, CREATE_BUCKET, DEPOSIT_TO_BUCKET, WITHDRAW_BUCKET, CANDIDATE_SELF_STAKE, CANDIDATE_REGISTRATION_FEE, DEPOSIT_TO_REWARDING_FUND, CLAIM_FROM_REWARDING_FUND. Two independent things to compare: the per-address deltas the logs say should happen, and the per-address deltas actually written to state. StateDiffCallback already hands you the latter as []WriteQueueEntry, and — importantly — it's invoked after sdb.mutex.Unlock() with its return value ignored, so it genuinely can't stall or brick anything. Any mismatch is a balance change no action authorized, which is exactly the ex-nihilo mint signature. Minimum detectable deviation: 1 rau.

One small thing needed: WriteQueueEntry only carries the new value, so CaptureWriteQueue would need to also capture the prior value for the delta. It's in the working set store cache already, and it's an observability-only change — still no hardfork.

L2 — running total (per block, O(1)). Maintain supply as a scalar updated by the L1-verified deltas, and assert supply(N) == supply(N-1) - Σ(baseFee × gasConsumed). Exact equality, monotonically non-increasing.

Then this PR's full scan becomes the right implementation for a third tier: a periodic reconciliation against L2's running total, run daily or per epoch on a dedicated auditor node rather than every minute on every validator.

Two things that would help either way:

• Replay mainnet from genesis and assert the invariant at every height, before shipping. That's the only way to know the accounting model is complete, and it would surface the namespace issue immediately.
• An integration test against a real factory.Factory. The fakeStateReader only ever puts state.Account into the Account namespace, which is why all of the above passes CI. Note it'd need bolt — memKVStore.Filter returns "in-memory KVStore does not support Filter()".

Happy to help with the L1/L2 piece if useful.

[raullenchai]

Thanks @envestcc — this is a thorough and correct review, and I've verified every point against the code. I'm going to rework the PR rather than merely patch around it. Concretely, confirming each:

1. Full scan stalls block commit (liveness). stateDB.States takes sdb.mutex.RLock() for the whole scan (state/factory/statedb.go, States), while PutBlock takes sdb.mutex.Lock() across ws.Commit(). My "off-consensus, can't affect consensus" claim was wrong — a long scan blocks commit. Agreed, this shape is not acceptable to run every minute on every validator.

2. P0: the scan can crash the node. The Account namespace also holds legacy Poll/Rewarding states written pre-Greenland and never deleted (state/tables.go documents admin, exempt, fund, reward histories, CandidatesList under it; UseV2Storage is false at mainnet Greenland=6544441 so they land there via putStateV1). Feeding those raw bytes to state.Account.Deserialize → Account.FromProto panic("unknown account type") / panic(invalid balance) (state/account.go), with no recover() on the path and Run launched as a bare goroutine → node dies in a loop. This alone makes the current code unmergeable.

3. Over-count, not under-count. Shared wire types let Fund/Admin decode as bogus Account balances (1e26), so R1 is over-counted — inverting the stated "never false-positive" property.

4. Loose bound. total <= cap has slack equal to cumulative base-fee burn, so sub-Harmony-scale mints are invisible.

Your proposed shape is the right one and the primitives already exist:

• L1 (per-block, off-path): reconcile per-address deltas from TransactionLog (sender/recipient/amount/type) against the actual state writes from StateDiffCallback/WriteQueueEntry. StateDiffCallback is invoked after sdb.mutex.Unlock() (state/factory/statedb.go, PutBlock) with its return value ignored, so it can't stall or brick the chain. Catching any delta an action didn't authorize = the ex-nihilo mint signature, down to 1 rau.
• L2 (per-block O(1)): maintain a scalar supply and assert supply(N) == supply(N-1) - Σ(baseFee×gas).
• L3 (periodic reconciliation): the full scan, but with a strict non-panicking decoder and run on a dedicated auditor node / daily or per-epoch — not every minute on every validator.

Two prerequisites I'll also take on:

• A mainnet replay from genesis asserting the invariant at every height, to prove the accounting model is complete and to surface the namespace issue up front.
• An integration test against a real bolt factory.Factory (not the fakeStateReader, which only ever writes Account and thus can't catch this).

I'll rework the PR along L1→L2→L3 and would welcome help with the WriteQueueEntry prior-value capture you mentioned. Thanks for catching this.

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
1 out of 2 committers have signed the CLA.

✅ raullenchai
❌ raullenspark1

---

raullenspark1 seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[raullenchai]

@envestcc — rework requested in your review is now pushed in 2859ec6a (branch feat/supply-conservation-observer). Concretely:

1. P0 crash + over-count → fixed. sumPrimaryBalances no longer feeds raw Account-namespace bytes to state.Account.Deserialize. It now reads each entry through a strict, non-panicking local decoder (decodeAccount) that rejects anything that isn't a genuine, canonically-encoded account:

• payloads with unknown proto fields (legacy Fund / Admin / exempt / rewardAccount / CandidatesList wire-tag collisions);
• an AccountType enum outside {DEFAULT, ZERO_NONCE} (e.g. Admin.productivityThreshold:85 → field 7, which previously panicked);
• a balance that isn't a decimal integer (e.g. Fund's 1e26, which previously over-counted R1);
• any non-canonical wire encoding (round-trip).

So the scan can neither crash the node nor over-count the legacy states into R1. Added TestDecodeAccountStrict, TestSumPrimaryBalancesRejectsLegacyStates, and an end-to-end TestObserverCheckToleratesLegacyStatesEndToEnd that poisons the namespace with the exact legacy payloads and asserts no panic + no over-count.

2. Liveness stall → fixed. The full-namespace scan was moved out of the ordinary-validator path entirely. It's now opt-in via blockchain.SupplyCheckConfig (default disabled) and runs at a configurable interval — intended for daily/per-epoch runs on a dedicated auditor node, not every minute on every validator.

As discussed, this PR is the L3 tier. L1 (per-block journal↔ledger reconcile via StateDiffCallback, which already runs outside the state-factory mutex) and L2 (per-block scalar running supply, O(1)) are the immediate follow-up, and a bolt-backed mainnet replay is the acceptance gate for them. I'd welcome your help on the WriteQueueEntry prior-value capture for L1 as you offered.

go test ./supplychecker/ ./chainservice/ ./blockchain/ all pass.

[sonarqubecloud]

Quality Gate failed

Failed conditions
3.2% Duplication on New Code (required ≤ 3%)

See analysis details on SonarQube Cloud

[raullenchai]

Thanks @envestcc — following your L1/L2 review shape, the loose-bound concern is now addressed and pushed (18170a09).

L2 (per-block exact delta) — now in this PR. supplychecker.SupplyTracker is wired into the factory's StateDiffCallback (invoked after the factory mutex is released, so it cannot stall commit) and maintains the running R1+R2+R3 total, asserting each block's net delta is never positive. IOTX has no post-genesis mint path, so a positive delta is the ex-nihilo mint signature — caught to 1 rau, no genesis-cap slack.

WriteQueueEntry prior-value capture — in, as you suggested. CaptureWriteQueue now records each key's pre-block PriorValue so an exact per-key delta can be computed with no extra state read at callback time. Because that base-store read would otherwise run on every validator's commit path, it is gated behind whether a state-diff consumer is registered — an ordinary auditor-less validator pays zero extra disk lookups per block. Added AddDiffCallback so the tracker coexists with ioSwarm's SetDiffCallback rather than clobbering it.

Real-factory integration gap — closed. TestBoltBackedSupplyConservation drives a real bolt factory.Factory and validates the L2 tracker + L3 observer against the actual v2 namespace keys.

L1 (journal↔ledger reconcile) remains the next phase; the prior-value plumbing it needs is now in place. Open to your help there.

go test ./supplychecker/ ./chainservice/ ./state/factory/ ./ioswarm/ all pass.