chore: Incremental account codes pruning - #2463
Open
sergerad wants to merge 1 commit into
Open
Conversation
Collaborator
|
This is perhaps a dumb question; but why do we even need to prune code? I imagine code changes are rare and even then code would largely be shared amongst most accounts? |
Comment on lines
+1290
to
+1291
| // Seed [2u8; 32] keeps the account ID distinct from the other test helpers. | ||
| build_account_with_code_seeded(push_value, [2u8; 32]) |
Collaborator
There was a problem hiding this comment.
Please remove the test_ prefix
Comment on lines
+1664
to
+1667
| /// Prune test 6: `prune_progress` records the cutoff of the last codes prune; re-pruning at the | ||
| /// same or a lower cutoff deletes nothing and never moves the marker backwards. | ||
| #[test] | ||
| fn test_prune_account_codes_marker_never_regresses() { |
Collaborator
There was a problem hiding this comment.
I don't quite understand this check tbh. When or how would this happen; and if it does, why does it matter?
Comment on lines
+1739
to
+1753
| /// The prune is churn-driven: a code pinned at the previous prune (some row with | ||
| /// `valid_until > prev_cutoff` referenced it) can only become collectable now if the row holding | ||
| /// its maximal `valid_until` expired inside `(prev_cutoff, cutoff_block]`. Candidate codes are | ||
| /// therefore collected from that window — an `idx_accounts_code_validity` range scan sized by | ||
| /// churn since the previous prune — and each is deleted only if the `idx_accounts_code_probe` | ||
| /// existence probe finds no row still referencing it past the cutoff. The previous cutoff is | ||
| /// persisted in `prune_progress` within the same transaction; when absent (first prune after | ||
| /// migration, or a fresh database) a full pass over all rows valid past the cutoff runs instead. | ||
| /// | ||
| /// Correctness of the windowed candidate set rests on two invariants: | ||
| /// - Rows are only ever closed to the `block_num` of the block currently being applied, which is | ||
| /// always above the cutoff, so every expiry crosses the window of some later prune. A write path | ||
| /// that back-dated `valid_until` below the current cutoff would leak the code forever. | ||
| /// - Every `account_codes` row is inserted alongside an `accounts` row referencing it (see | ||
| /// [`upsert_accounts`]); an orphan code with no referencing row would never become a candidate. |
Collaborator
There was a problem hiding this comment.
I think I've re-read this 5x now - I don't understand what this is saying. Is there a simpler way to explain this?
This is the first time I've encountered churn driven.
Comment on lines
+1774
to
+1776
| // Codes are already pruned through this cutoff and nothing can become collectable while the | ||
| // cutoff stands still, so skip without moving the marker backwards. | ||
| Some(prev_cutoff) if prev_cutoff >= cutoff_block => return Ok(0), |
Collaborator
There was a problem hiding this comment.
When would this happen? And if its caller error, should this value not be determined from within this function?
Collaborator
There was a problem hiding this comment.
I guess I don't understand why we aren't just pruning one block at a time as the chain moves?
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes #2444.
The vault and storage prunes are incremental: each block only visits rows that expired since the previous prune. The account-code prune instead re-scanned every account row still valid past the cutoff on every block.
This PR makes the codes prune churn-driven. A code pinned at one prune can only become collectable at a later one if its last referencing row expired between the two cutoffs, so the prune scans just that window for candidates and probes each for a surviving reference — O(churn) per block, like the other two prunes.
Implementation notes:
005addsidx_accounts_code_probe(backs the existence probe) and a single-rowprune_progresstable recording the cutoff through which code pruning has completed. The marker is updated in the same transaction as the prune, so it is crash-consistent and survives restarts.prune_tip), the prune now skips entirely instead of re-scanning the live set.Changelog