diff --git a/TODO.md b/TODO.md index 15c6e0c..cb38bdf 100644 --- a/TODO.md +++ b/TODO.md @@ -88,22 +88,64 @@ current build depends on them. still authenticates end-to-end, so the transport only provides reachability). -## Voting (coinholder polling) - -- [ ] **Automated poll source / discovery** — voting currently uses **manual - ballot entry**: the user pastes the poll's published ballot-definition JSON - and its reception address, answers, and casts (a shielded memo via the FROST - send path; weight is set by the poll's off-chain balance snapshot). Replace - the manual paste with a programmatic source once one is available: - - Fetch active polls from a configurable **poll-source URL** (Zodl exposes a - "custom poll sources" config; confirm the feed format), render the ballot - automatically, and show live/closed status + results. - - The definitive protocol design is ValarDragon/Valar's; get the poll-feed + - registration/snapshot spec from there (or the Zodl integration docs) to - match eligibility exactly. Keep manual entry as the fallback/offline path. - - The casting core (`voting.rs`: memo v1 encode/validate/poll-hash, - `prepare_vote`) is source-agnostic and already done — this is only about - *where the ballot comes from* and surfacing results. +## Voting (protocol / coinholder governance) + +- [ ] **Migrate to ValarGroup Shielded Vote (full rebuild; supersedes memo v1)** — + the real coinholder-vote protocol we should target is **ValarGroup Shielded + Vote**: https://valargroup.gitbook.io/shielded-vote-docs . It is a live, + cryptographically-private, on-chain voting system on a **dedicated vote + chain**, used infrequently to gauge protocol-upgrade sentiment *before* + committing engineering resources — exactly Cyze's governance use case. + + **This is not a change to the current memo format — it is a different + system.** Our shipped `core/src/voting.rs` implements the informal + **zec-coin-polling "Vote Cast Memo v1"** (a JSON memo cast as a shielded send + to a reception address, tallied off-chain from a *transparent*-balance + snapshot). Shielded Vote has no vote memo: a vote is a ZK-proven **Vote + Commitment** (VAN consumed → new VAN + `H(DOMAIN_VC, round_id, shares_hash, + proposal_id, vote_decision)`) plus 16 ElGamal share ciphertexts, submitted to + REST endpoints on the vote chain. **Expect to scrap `voting.rs` and its UI** + (`VoteTab`/`parseBallot` in `src/screens/Groups.tsx`, `wallet_prepare_vote`, + `VoteEntry`/`BallotDefinition`) and rebuild around the SDK below. Keep memo v1 + only if a lightweight, no-infra sentiment poll is still wanted; otherwise + remove it so the two are never confused. + + **Confirmed (2026-08-04, from the user):** + 1. **Infrastructure is live** — vote chain + election authority + PIR fleet + are running; there is a real network to build/test against. + 2. **FROST-compatible** — the delegation step (ZKP1) takes an externally + produced re-randomized spend-auth signature via a governance PCZT + (`(rk, sighash, spend_auth_sig)`), which maps onto Cyze's existing FROST + re-randomized Orchard signing ceremony. Confirm the exact PCZT hand-off + when building. + 3. **Ironwood supported going forward** — vote weight snapshots the group's + shielded note holdings, and Ironwood is covered, so a post-NU6.3 shielded + treasury can vote (this also fixes memo v1's flaw that only *transparent* + balances counted — a shielded FROST treasury effectively couldn't vote). + 4. **Crate versions: pin to whatever is in production at build time.** Both + SDK crates are published and moving fast — snapshot the then-current + production versions rather than an early rc: + - `zcash_voting` — client lib (ZKP1/2/3 via Halo2, ElGamal, governance + PCZT, Merkle witnesses, SQLite round state). Repo: + https://github.com/valargroup/zcash_voting + - `pir-client` — nullifier non-membership PIR queries. + (Swift SDK exists too, but Cyze is Rust — use the crates directly.) + + **Rough shape of the wallet-side flow** (see the Integration Guide): discover + + validate vote config → `GET /shielded-vote/v1/rounds/active` → PIR + nullifier proofs → build+prove **ZKP1 delegation** (governance PCZT, FROST + spend-auth) → `POST /delegate-vote` → sync the vote-commitment tree → per + proposal, build **ZKP2** and `POST /cast-vote` → split into 16 ElGamal shares + and `POST /shares` with staggered anti-censorship `submit_at` timing → read + `GET /tally-results/{round_id}` once the round is `FINALIZED`. Note + `vote_round_id` encoding is context-sensitive (hex in config/URLs/shares, + base64 in delegate/cast bodies). + + **Suggested first step — a scoping spike** before any UI: add the production + `zcash_voting` + `pir-client` crates, hit a live round's `/rounds/active`, and + prove the FROST-produced re-randomized spend-auth sig feeds ZKP1 end-to-end. + That de-risks the one genuinely novel part (threshold signing into their + prover) cheaply. Own branch, own PR; larger effort than any current item. ## Wallet (Zcash) diff --git a/docs/SYNC_OPTIMIZATION.md b/docs/SYNC_OPTIMIZATION.md new file mode 100644 index 0000000..7990239 --- /dev/null +++ b/docs/SYNC_OPTIMIZATION.md @@ -0,0 +1,148 @@ +# Sync optimization — design & roadmap + +Status: **driver implemented, pending testnet validation** (branch +`feat/sync-optimizations`). The pipelined driver is built and off by default; it +becomes the default only after the validation gate below passes. This is the next +major targeted update. Its goal is to cut the wall-clock latency of wallet +sync — especially the recovery / large-range case — without leaving the +Ironwood-capable `zcash_client_backend` (ECC) stack. + +## Why we stay on the ECC stack + +We surveyed the open-source Zcash wallets (YWallet/ZKool, Zingo, Cake, ZODL, +Vizor). All but ZODL are light wallets on lightwalletd compact blocks; the +differentiation is entirely in the scan engine. + +- **YWallet/ZKool** use **Warp Sync** — the fastest engine — but it is + Sapling/Orchard-oriented, keeps its own DB schema, and **has no Ironwood + support**. Adopting it wholesale would fork us off the maintained NU6.3 stack + for a rewrite: wrong trade for a funds wallet on Ironwood. +- **ZODL** offers a full/hybrid node mode — a privacy feature, not a general + latency win, and a large architectural add. + +So we **port Warp's ideas onto our existing stack** rather than switch engines. + +## What we already have (main) + +- **Tip-height birthday for new wallets** — a new group starts at the chain tip, + so it never scans pre-creation history. (The single biggest first-sync win; done.) +- **Subtree-root tree init** — `zcash_client_backend::sync::run` calls + `update_subtree_roots` (GetSubtreeRoots), so the note-commitment tree is + initialized without replaying all history. +- **Spend-before-sync ordering** — `run` scans `suggest_scan_ranges()` in + priority order (ChainTip/Verify first), so the balance surfaces before a full + catch-up finishes, and the UI polls it every ~5s. +- **Configurable batch size** — `sync_group(batch_size)`, clamped + `[MIN,MAX]_SYNC_BATCH_SIZE`, persisted via `Settings.sync_batch_size`. + +## What's missing (this update) + +The upstream `sync::run` is explicit that "block batches are not downloaded in +parallel with scanning." Two levers remain, and both require driving the sync +loop ourselves instead of calling `sync::run`: + +1. **Pipelining** — overlap network download with CPU trial-decryption. This is + Warp's core advantage and the single biggest safe win left. **Implemented and + now the standard (only) sync driver** (`run_pipelined` in `wallet.rs`). The + stock `sync::run` path and its on-disk block cache were removed once the + pipelined driver was validated. +2. **Adaptive batch size** — grow the batch over empty ranges (cheap to scan), + shrink over dense ranges (expensive), instead of one fixed size for the run. + **Deliberately deferred**: the pipelined driver keeps the *same* fixed batch + units as the stock driver so its output is byte-identical and the validation + gate below is a clean equality check. Adaptive sizing changes the scan units, + so it lands as a separate follow-up once pipelining is validated and default. + +A third lever is crate-gated: + +3. **Parallel trial decryption** — the pinned `zcash_note_encryption` 0.4.2 does + batch decryption single-threaded. Getting multi-core decryption needs either a + `zcash_note_encryption` with the `multicore` feature or wiring the + `zcash_client_backend` `sync-decryptor` (rayon) pipeline. Deferred to the next + crate-cohort bump; tracked here so it isn't forgotten. + +## Approach: a custom pipelined driver, alongside `sync::run` + +Pipelining needs control of the loop, so we add a **custom driver** +(`run_pipelined` / `running_pipelined` in `wallet.rs`) that faithfully reproduces +the upstream `run`/`running` control flow (subtree roots → chain tip → verify +pass → historic ranges), changing only how batches are fed: + +- A **producer** task downloads each batch's compact blocks **into memory** and + the chain-state anchor, then hands `(ScanRange, Vec, ChainState)` + over a **bounded channel** (capacity 2) so download runs up to two batches + ahead. A cloned tonic client shares the underlying HTTP/2 connection, so this + adds no new socket. +- The **consumer** (main task) receives a ready batch, wraps its blocks in an + in-memory `BlockSource` (`MemBlockSource`), and runs `scan_cached_blocks` on it + (CPU-bound). Scanning is transactional per batch via `put_blocks`, so an + interrupted or cancelled batch leaves the db consistent at a batch boundary — + the same guarantee the stock driver gives. +- On a **reorg / continuity error** or a newly-added higher-priority range, the + consumer breaks, the producer is aborted, and the pass restarts from + `suggest_scan_ranges` — exactly the upstream `return Ok(true)` → outer-loop + behavior. + +Because each batch is downloaded fresh into memory and never persisted, the +pipelined path **never touches the on-disk `FsCache`** — no file writes, no cache +mutex contended between producer and consumer, and nothing to truncate on a +reorg rewind (only the db is rewound). + +**Transparent UTXO refresh is intentionally omitted.** Upstream `running` performs +it only under the `transparent-inputs` feature, which our `zcash_client_backend` +build does not enable (group accounts are Orchard-only view keys). The stock +driver we run today therefore does not perform it either, so omitting it keeps +the two byte-identical. + +Correctness-critical logic (reorg rewind, verify ranges, subtree roots, +chain-tip update, batch splitting) is **ported faithfully** from the upstream +`sync.rs` we depend on; only the download/scan overlap is new. The batch splitter +(`split_scan_range`) has a unit test asserting it produces the exact same units as +the upstream step-7 splitter. + +### A note on overlap and the runtime + +`scan_cached_blocks` is synchronous and CPU-bound; the consumer calls it directly +on the async task. On the multi-threaded Tokio runtime the app uses, the producer +keeps downloading the next batches on other worker threads while the consumer +thread scans — which is where the latency win comes from. On a single-threaded +runtime the code is still correct (no overlap, identical result). Moving the scan +onto `spawn_blocking` to guarantee overlap regardless of runtime is a possible +future refinement; it is not needed for correctness. + +### Status: standard driver + +The pipelined driver was validated on testnet (byte-identical wallet state vs the +stock driver, faster on high-latency links) and is now the **standard, only** sync +path. `sync_group` always calls `run_pipelined`; the `experimental_pipelined_sync` +setting, the stock `zcash_client_backend::sync::run` call, and the on-disk +`FsBlockDb`/`FsCache` block cache it required have been removed. The pipelined +driver streams blocks straight from the network to the scanner in memory. + +### Validation gate before default + +Before flipping the default to the pipelined driver: + +1. A testnet recovery sync (large range) produces a **byte-identical** wallet + state to the stock driver (same balance, notes, witnesses, scanned height). +2. A reorg is exercised (or simulated) and recovers correctly. +3. A shielded send after a pipelined sync builds, signs, and broadcasts. + +## Sequencing + +1. **[done] Settings gate + custom pipelined driver** (prefetch download while + scanning), off by default. ← core of the work; `run_pipelined` in `wallet.rs`. +2. **[next] Testnet validation** against the stock driver; flip default if it + passes (see the validation gate above). +3. **[follow-up] Adaptive batch size** — grow/shrink the batch by range density, + once pipelining is the validated default. +4. **[next crate bump] Parallel trial decryption** via note-encryption + `multicore` / the `sync-decryptor` pipeline. +5. **[optional, infra] Zaino indexer** — evaluate a Rust indexer (Zingo's path) + in place of stock lightwalletd for richer per-request data. Composes with the + above; not a wallet rewrite. + +## Explicit non-goals + +- No wholesale swap to Warp/ZKool's engine (no Ironwood support; own DB). +- No full/hybrid-node mode (privacy feature, not latency; large surface). diff --git a/docs/SYNC_PIPELINE_UAT.md b/docs/SYNC_PIPELINE_UAT.md new file mode 100644 index 0000000..879f6d6 --- /dev/null +++ b/docs/SYNC_PIPELINE_UAT.md @@ -0,0 +1,93 @@ +# UAT — pipelined sync (now the standard driver) + +Acceptance checklist for the pipelined sync driver (`feat/sync-optimizations`). +Run on **testnet** first. + +> **Note:** the pipelined driver is now the **standard, only** sync path — there +> is no `experimental_pipelined_sync` toggle anymore, and the stock +> `sync::run` path was removed. Sections **A0a** and **A** (the on/off +> checkbox test and the stock-driver baseline) are therefore **historical** — you +> can no longer switch to the stock driver in-app to compare. If you still want an +> equality baseline, capture it from an older build; otherwise start at **B** and +> validate the single driver's correctness, speed, incremental behavior, +> cancellation, and post-sync send. + +See `docs/SYNC_OPTIMIZATION.md` for the design. + +## Setup + +- [ ] Build the current branch: `npm run tauri build` (or `cargo build` for a dev + backend), and launch the freshly built binary — not a previously installed + bundle. +- [ ] Use a **testnet** group with a known, non-trivial history (funded a few + times, at least one send), so scanning actually finds notes. + +## A0a. The toggle itself (checkbox wiring) — HISTORICAL + +- [ ] *(No longer applies — the toggle was removed and pipelined is the only path.)* + +## A. Baseline with the stock driver (control) — HISTORICAL + +- [ ] *(No longer runnable in-app — the stock driver was removed. Kept for + reference; use an older build if you need a stock baseline to compare.)* + +## B. Pipelined driver — clean-state correctness (the core test) + +- [ ] Delete the wallet db (force a full rescan from birthday) and sync to the tip. +- [ ] **Balance is correct** — total (Ironwood), plus any legacy Orchard, matches + the group's known funds and what block explorers show. +- [ ] Received-note count is correct. +- [ ] Transaction history is complete (txids, amounts, memos). +- [ ] Scanned-to height reaches the chain tip. +- [ ] Wall-clock sync time is reasonable (faster on a high-latency link is the + whole point). If you kept a stock baseline from an older build, it should be + **≤** that. + +## C. Incremental sync + +- [ ] With the pipelined wallet already at the tip, wait for / cause a new inbound + testnet payment, then "Sync Now". +- [ ] Only the new blocks are scanned (fast), the new note appears, and the balance + increases by the expected amount. +- [ ] Sync a second time with no new activity → completes quickly, balance + unchanged (no double-count, no drift). + +## D. Cancellation / resume + +- [ ] Start a full rescan (delete db) with the pipelined driver, then hit "Sync + Now" (or switch away) mid-sync to cancel it. +- [ ] App stays responsive; no panic; no error toast beyond an expected + "cancelled". +- [ ] Start sync again → it resumes and completes, ending at the same + balance/height as B (cancellation left the db consistent at a batch + boundary, not corrupted). + +## E. Reorg tolerance (best-effort) + +- [ ] If a testnet reorg happens to occur during a sync, confirm it recovers: the + log shows a "chain reorg detected … rewinding" line and the sync finishes at + the correct tip with the correct balance. (Hard to force on demand; watch for + it opportunistically during A–D.) + +## F. Send after a pipelined sync (funds path) + +- [ ] After a pipelined sync, build + FROST-sign + broadcast a small testnet send. +- [ ] Transaction is accepted by the node (no branch-id / MissingSpendAuthSig / + note-selection errors). +- [ ] After it confirms, a re-sync shows the spend and the reduced balance + correctly. + +## G. Regression — HISTORICAL + +- [ ] *(No longer applies — there is no flag to turn off; the pipelined driver is + the only path.)* + +## Sign-off + +- [ ] B balances/notes/history/height are correct (against known funds / an + explorer). +- [ ] C, D, F pass on testnet. +- [ ] No panics, no stuck syncs, UI responsive throughout. + +Only after this passes on testnet: repeat B/F once on **mainnet** with a small +balance before relying on it broadly. diff --git a/src-tauri/Cargo.lock b/src-tauri/Cargo.lock index 06d7108..40c3e29 100644 --- a/src-tauri/Cargo.lock +++ b/src-tauri/Cargo.lock @@ -3711,11 +3711,17 @@ dependencies = [ "subtle", ] +[[package]] +name = "pastey" +version = "0.2.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2ee67f1008b1ba2321834326597b8e186293b049a023cdef258527550b9935b4" + [[package]] name = "pczt" -version = "0.9.2" +version = "0.9.3" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "aead0b7ecb4363d8560ac76687c02ef896292fb836c1c68f3a4d99443f32e1a0" +checksum = "a5592f4f3eba7f9344cc423f45b6e65911e0630c9b89968d5a20792aadd5a0eb" dependencies = [ "blake2b_simd", "bls12_381", @@ -7478,9 +7484,9 @@ dependencies = [ [[package]] name = "zcash_client_backend" -version = "0.24.0-rc.6" +version = "0.24.0-rc.7" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "832ace878c125814bb9e82228d3661499124c6f2936e60fc1df3c6876eb54624" +checksum = "f49636f1d22b0511b2a117548cc9dcf569440a3589b06bf6df422c6b738f6231" dependencies = [ "async-trait", "base64 0.22.1", @@ -7533,9 +7539,9 @@ dependencies = [ [[package]] name = "zcash_client_sqlite" -version = "0.22.0-rc.6" +version = "0.22.0-rc.8" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c4a789c0038359968f93bfe3952d2b942e3bae5582f79c2f31137c6fd736a225" +checksum = "c44798ac3735e85c523899c4c8e06cf94e422a6ceedc9ee5b256a3cf53dd1c96" dependencies = [ "bip32", "bitflags 2.13.1", @@ -7549,6 +7555,8 @@ dependencies = [ "maybe-rayon", "nonempty", "orchard", + "pastey", + "pczt", "prost", "rand 0.8.7", "rand_core 0.6.4", @@ -7635,10 +7643,11 @@ dependencies = [ [[package]] name = "zcash_pool_migration" -version = "0.1.0-rc.5" +version = "0.1.0-rc.7" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9010904169a92805128b3a8eab0dd4ff79fd164dd416af68aaeac5c7ebbd2522" +checksum = "532ac156acab2144a88b3e55763ee9329704863a34839e55c2fa705addf3e57e" dependencies = [ + "blake2b_simd", "corez", "getset", "incrementalmerkletree", diff --git a/src-tauri/core/Cargo.toml b/src-tauri/core/Cargo.toml index 108f655..6a979a2 100644 --- a/src-tauri/core/Cargo.toml +++ b/src-tauri/core/Cargo.toml @@ -52,7 +52,7 @@ ff = { version = "0.13", optional = true } # Wallet/light-client stack (feature `wallet`). Heavy: pulls the zcash # light-client backend, lightwalletd gRPC, PCZT builder, and sqlite. -zcash_client_backend = { version = "=0.24.0-rc.6", optional = true, default-features = false, features = [ +zcash_client_backend = { version = "=0.24.0-rc.7", optional = true, default-features = false, features = [ "orchard", "sync", "lightwalletd-tonic", @@ -60,7 +60,7 @@ zcash_client_backend = { version = "=0.24.0-rc.6", optional = true, default-feat "lightwalletd-tonic-tls-webpki-roots", "pczt", ] } -zcash_client_sqlite = { version = "=0.22.0-rc.6", optional = true, features = ["orchard", "transparent-inputs", "unstable", "serde"] } +zcash_client_sqlite = { version = "=0.22.0-rc.8", optional = true, features = ["orchard", "transparent-inputs", "unstable", "serde"] } pczt = { version = "0.9.1", optional = true, features = ["signer"] } zcash_keys = { version = "0.16.1", optional = true, features = ["orchard"] } zcash_protocol = { version = "0.10.3", optional = true } diff --git a/src-tauri/core/src/wallet.rs b/src-tauri/core/src/wallet.rs index b14ea85..2995a85 100644 --- a/src-tauri/core/src/wallet.rs +++ b/src-tauri/core/src/wallet.rs @@ -246,26 +246,25 @@ pub async fn lightwalletd_info(url: &str) -> Result use std::path::{Path, PathBuf}; use rand::rngs::OsRng; -use async_trait::async_trait; -use prost::Message; use zcash_client_backend::data_api::chain::error::Error as ChainError; -use zcash_client_backend::data_api::chain::{BlockCache, BlockSource}; -use zcash_client_backend::data_api::scanning::ScanRange; +use zcash_client_backend::data_api::chain::{ + BlockSource, ChainState, CommitmentTreeRoot, +}; +use zcash_client_backend::data_api::scanning::{ScanPriority, ScanRange}; use zcash_client_backend::data_api::wallet::{ create_pczt_from_proposal, propose_standard_transfer_to_address, ConfirmationsPolicy, }; -use zcash_client_backend::data_api::{AccountBirthday, AccountPurpose, WalletRead, WalletWrite}; +use zcash_client_backend::data_api::{ + AccountBirthday, AccountPurpose, WalletCommitmentTrees, WalletRead, WalletWrite, +}; use zcash_client_backend::fees::StandardFeeRule; use zcash_client_backend::wallet::OvkPolicy; use zcash_client_backend::proto::compact_formats::CompactBlock; use zcash_client_backend::proto::service::{BlockId, ChainSpec}; -use zcash_client_sqlite::chain::init::init_blockmeta_db; -use zcash_client_sqlite::chain::BlockMeta; use zcash_client_sqlite::util::SystemClock; use zcash_client_sqlite::wallet::init::init_wallet_db; -use zcash_client_sqlite::{FsBlockDb, WalletDb}; +use zcash_client_sqlite::WalletDb; use zcash_keys::keys::UnifiedFullViewingKey; -use zcash_primitives::block::BlockHash; use zcash_protocol::consensus::BlockHeight; use zcash_protocol::memo::{Memo, MemoBytes}; @@ -773,6 +772,7 @@ pub async fn init_group_account( .map_err(|e| CoreError::Crypto(format!("wallet accounts: {e}")))? .is_empty() { + tracing::debug!(group = %group_id, "wallet setup: account already imported; nothing to do"); return Ok(0); // already imported } @@ -780,13 +780,25 @@ pub async fn init_group_account( let ufvk = UnifiedFullViewingKey::decode(¶ms, ufvk_str) .map_err(|e| CoreError::Crypto(format!("invalid UFVK: {e}")))?; + // These are single unary RPCs, not the long block stream, so bound them. A + // server that accepts the TCP connection but never answers (a misconfigured + // proxy, a stalled `get_tree_state` for a deep birthday) would otherwise leave + // wallet setup spinning forever with no error and no log line. + let rpc_timeout = std::time::Duration::from_secs(30); + + tracing::info!(group = %group_id, url = %lightwalletd_url, "wallet setup: connecting to lightwalletd"); let mut client = connect(lightwalletd_url).await?; - let tip = client - .get_latest_block(ChainSpec {}) + let tip = tokio::time::timeout(rpc_timeout, client.get_latest_block(ChainSpec {})) .await + .map_err(|_| { + CoreError::Connection( + "get_latest_block timed out — lightwalletd accepted the connection but did not respond".into(), + ) + })? .map_err(|e| CoreError::Connection(format!("get_latest_block: {e}")))? .into_inner() .height; + tracing::info!(group = %group_id, tip, "wallet setup: connected; got chain tip"); let nu5 = params .activation_height(NetworkUpgrade::Nu5) @@ -803,47 +815,145 @@ pub async fn init_group_account( // request the frontier as of the block *before* the first one to scan. // Fetching the treestate at `scan_from` itself would skip that block — and // with it the transaction that funded the group. - let treestate = client - .get_tree_state(BlockId { + tracing::info!(group = %group_id, scan_from, "wallet setup: fetching tree state for the account birthday"); + let treestate = tokio::time::timeout( + rpc_timeout, + client.get_tree_state(BlockId { height: scan_from.saturating_sub(1), hash: vec![], - }) - .await - .map_err(|e| CoreError::Connection(format!("get_tree_state: {e}")))? - .into_inner(); + }), + ) + .await + .map_err(|_| { + CoreError::Connection( + "get_tree_state timed out — lightwalletd did not return the birthday tree state".into(), + ) + })? + .map_err(|e| CoreError::Connection(format!("get_tree_state: {e}")))? + .into_inner(); let birthday = AccountBirthday::from_treestate(treestate, None) .map_err(|_| CoreError::Crypto("could not derive account birthday from treestate".into()))?; db.import_account_ufvk(group_id, &ufvk, &birthday, AccountPurpose::ViewOnly, None) .map_err(|e| CoreError::Crypto(format!("import account: {e}")))?; + tracing::info!(group = %group_id, scan_from, "wallet setup: view-only account imported; will scan from this height"); Ok(scan_from) } -/// A `BlockCache` over `FsBlockDb`. `FsBlockDb` ships only `BlockSource`, so we -/// wrap it and add the cache-management methods `sync::run` requires (cache -/// downloaded compact blocks as files on disk, read them back, prune them). +/// How many blocks each sync batch downloads and scans at once when the caller +/// gives no override. Larger batches amortize the per-batch gRPC round-trip and +/// database-transaction overhead across more blocks, which is the dominant cost +/// once trial decryption finds nothing (the common case for a wallet catching up +/// over empty history). Compact blocks are small, so a few thousand per batch is +/// comfortable in memory. Tunable per-install via settings; see [`sync_group`]. +pub const DEFAULT_SYNC_BATCH_SIZE: u32 = 5_000; + +/// Clamp bounds for a caller-supplied batch size. Below the floor the round-trip +/// overhead dominates; above the ceiling a batch's worth of compact blocks can +/// spike memory (they are all held in a `Vec` while the batch is scanned). +pub const MIN_SYNC_BATCH_SIZE: u32 = 500; +pub const MAX_SYNC_BATCH_SIZE: u32 = 25_000; + +/// Options controlling how a sync runs. +#[derive(Debug, Clone, Copy, Default)] +pub struct SyncOptions { + /// Blocks to download and scan per batch; `None` uses + /// [`DEFAULT_SYNC_BATCH_SIZE`], clamped into `[MIN, MAX]_SYNC_BATCH_SIZE`. + pub batch_size: Option, +} + +/// Sync the group's wallet: download and trial-decrypt compact blocks from +/// lightwalletd into the local db. Long-running; touches the network. /// -/// `FsBlockDb` holds a rusqlite `Connection` (not `Sync`), but `BlockCache` -/// requires `Sync`, so the inner db is behind a `Mutex`. The cache error type is -/// `io::Error` because `FsBlockDbError` does not implement `std::error::Error`, -/// which `sync::run` requires. -struct FsCache { - inner: std::sync::Mutex, - blocks_dir: PathBuf, +/// Driven by the pipelined [`run_pipelined`] driver, which overlaps block +/// download with CPU trial-decryption and streams blocks straight from the +/// network to the scanner (no on-disk block cache). +pub async fn sync_group( + data_dir: &Path, + group_id: &str, + network: WalletNetwork, + lightwalletd_url: &str, + db_key: &[u8], + opts: SyncOptions, + cancel: &tokio_util::sync::CancellationToken, +) -> Result<(), CoreError> { + let batch_size = opts + .batch_size + .unwrap_or(DEFAULT_SYNC_BATCH_SIZE) + .clamp(MIN_SYNC_BATCH_SIZE, MAX_SYNC_BATCH_SIZE); + let (db_path, _) = wallet_paths(data_dir, group_id, network); + let mut db = open_db(&db_path, network, db_key)?; + + let mut client = connect(lightwalletd_url).await?; + let params = network.params(); + // The driver scans in transactional batches, so dropping the future between + // batches leaves the db consistent (just short of the tip). That makes it + // safe to race against a cancellation token: "Sync Now" trips the token to + // abandon a stalled run, and a fresh sync resumes from where this one left + // off. Without this, a stuck stream would keep the sync pending forever. + let result = tokio::select! { + biased; + _ = cancel.cancelled() => Err(CoreError::Cancelled), + res = run_pipelined(&mut client, ¶ms, &mut db, batch_size) => res, + }; + // Turn known, actionable failures into a message that says what to do, while + // keeping the raw server error appended for diagnosis. + result.map_err(annotate_sync_error) } -fn io_err(e: impl std::fmt::Display) -> std::io::Error { - std::io::Error::other(e.to_string()) +/// Rewrite a raw sync failure into an actionable message when it matches a known +/// cause, preserving the original text after an em-dash for debugging. Applies to +/// both sync drivers (both surface a `CoreError::Connection` carrying the raw +/// lightwalletd/tonic error string). +/// True when a connection error is a lightwalletd rejecting a shielded pool it +/// doesn't recognize — Ironwood (NU6.3) on a server that predates it, which +/// answers `get_subtree_roots` for that pool with gRPC InvalidArgument +/// "invalid shielded protocol value". +fn is_invalid_shielded_protocol(e: &CoreError) -> bool { + let lower = match e { + CoreError::Connection(m) => m.to_lowercase(), + _ => return false, + }; + lower.contains("invalid shielded protocol") + || (lower.contains("shielded protocol") && lower.contains("invalid")) } -impl FsCache { - fn lock(&self) -> Result, std::io::Error> { - self.inner.lock().map_err(|_| io_err("block cache lock poisoned")) +fn annotate_sync_error(e: CoreError) -> CoreError { + // A lightwalletd that predates Ironwood (NU6.3) doesn't know the Ironwood + // shielded protocol, so the very first sync step — fetching subtree roots for + // all pools, including Ironwood — is rejected with "invalid shielded protocol + // value". The whole sync then aborts. This is a server-capability problem, not + // a wallet bug, and the fix is to point at an Ironwood-capable server. (The + // pipelined driver skips Ironwood roots on this error instead; this friendly + // message is for the stock driver, which can't.) + if is_invalid_shielded_protocol(&e) { + let raw = match &e { + CoreError::Connection(m) => m.clone(), + _ => return e, + }; + return CoreError::Connection(format!( + "This lightwalletd server doesn't support Ironwood (NU6.3). Syncing has \ + to fetch the Ironwood note-commitment tree, and the server rejected \ + that request (\"invalid shielded protocol value\"). Switch to an \ + Ironwood-capable lightwalletd in the wallet's network settings, then \ + sync again. — {raw}" + )); } + + e } -impl BlockSource for FsCache { - type Error = std::io::Error; +/// An in-memory [`BlockSource`] over one batch of already-downloaded compact +/// blocks. The pipelined driver hands each batch straight from the network to +/// the scanner through this, so a sync never touches an on-disk block cache (no +/// file writes, no cache mutex contention between the download-ahead producer and +/// the scanning consumer). Scanning is fully transactional via `put_blocks`, so an +/// interrupted batch leaves the db consistent. +struct MemBlockSource(Vec); + +impl BlockSource for MemBlockSource { + // Reading from an owned `Vec` can't fail. + type Error = std::convert::Infallible; fn with_blocks( &self, @@ -854,139 +964,442 @@ impl BlockSource for FsCache { where F: FnMut(CompactBlock) -> Result<(), ChainError>, { - let db = self.lock().map_err(ChainError::BlockSource)?; - let mut height = from_height.unwrap_or_else(|| BlockHeight::from_u32(0)); + let start = from_height.map(u32::from); let mut remaining = limit.unwrap_or(usize::MAX); - while remaining > 0 { - let meta = match db.find_block(height).map_err(|e| ChainError::BlockSource(io_err(e)))? { - Some(m) => m, - None => break, // contiguous run ended - }; - let bytes = std::fs::read(meta.block_file_path(&self.blocks_dir)) - .map_err(ChainError::BlockSource)?; - let block = - CompactBlock::decode(&bytes[..]).map_err(|e| ChainError::BlockSource(io_err(e)))?; - with_block(block)?; - height = height + 1; + for cb in &self.0 { + if remaining == 0 { + break; + } + // The producer downloads exactly the requested range, but honour + // `from_height`/`limit` defensively so this matches the disk cache's + // contract (ascending, contiguous from `from_height`). + if let Some(s) = start { + if (cb.height as u32) < s { + continue; + } + } + with_block(cb.clone())?; remaining -= 1; } Ok(()) } } -#[async_trait] -impl BlockCache for FsCache { - fn get_tip_height( - &self, - _range: Option<&ScanRange>, - ) -> Result, Self::Error> { - self.lock()?.get_max_cached_height().map_err(io_err) +/// One prefetched batch handed from the download producer to the scan consumer: +/// the range it covers, its compact blocks, and the chain-state anchor immediately +/// before the range (needed by `scan_cached_blocks`). +type PrefetchedBatch = (ScanRange, Vec, ChainState); + +/// Split a suggested scan range into `batch_size`-block sub-ranges, preserving +/// priority. Ported verbatim from the upstream `sync::running` step-7 splitter so +/// the pipelined driver scans in the exact same units as the stock driver. +fn split_scan_range(range: ScanRange, batch_size: u32) -> Vec { + let mut acc = range; + let mut out = Vec::new(); + loop { + if acc.is_empty() { + break; + } + match acc.split_at(acc.block_range().start + batch_size) { + Some((cur, next)) => { + out.push(cur); + acc = next; + } + None => { + out.push(acc); + break; + } + } } + out +} - async fn read(&self, range: &ScanRange) -> Result, Self::Error> { - let range = range.block_range().clone(); - let db = self.lock()?; - let mut blocks = Vec::new(); - let mut height = range.start; - while height < range.end { - match db.find_block(height).map_err(io_err)? { - Some(meta) => { - let bytes = std::fs::read(meta.block_file_path(&self.blocks_dir))?; - blocks.push(CompactBlock::decode(&bytes[..]).map_err(io_err)?); +/// Custom pipelined sync driver: same control flow as +/// `zcash_client_backend::sync::run`, but the historic-range scan overlaps block +/// download with trial-decryption. Correctness-critical logic (subtree roots, +/// chain-tip update, verify pass, reorg/continuity rewind, priority re-ordering) +/// is ported faithfully from the upstream `sync.rs`; only the download/scan +/// overlap in step 7 is new. Produces the same wallet state as the stock driver. +/// +/// Note: the transparent-UTXO refresh in upstream `running` is gated on the +/// `transparent-inputs` feature, which our `zcash_client_backend` build does not +/// enable (group accounts are Orchard-only view keys), so the stock driver we run +/// today does not perform it either. Omitting it here keeps the two byte-identical. +async fn run_pipelined( + client: &mut CompactTxStreamerClient, + params: &Network, + db: &mut GroupDb, + batch_size: u32, +) -> Result<(), CoreError> { + // 1) & 2) Download note-commitment subtree roots and hand them to the db, so + // the trees are initialized without replaying all history. One-time; no + // pipelining benefit, so it stays serial. + update_subtree_roots_pipelined(client, db).await?; + + // Re-run the per-session loop until the wallet's view of the chain tip is + // valid (mirrors `while running(..).await? {}` upstream). + while running_pipelined(client, params, db, batch_size).await? {} + + Ok(()) +} + +/// One pass of the pipelined sync loop. Returns `true` when the suggested scan +/// ranges changed underneath us (continuity error, or a newly higher-priority +/// range) and the caller should restart from a fresh `suggest_scan_ranges`. +async fn running_pipelined( + client: &mut CompactTxStreamerClient, + params: &Network, + db: &mut GroupDb, + batch_size: u32, +) -> Result { + // 3) & 4) Refresh the chain tip so `suggest_scan_ranges` reflects new blocks. + update_chain_tip_pipelined(client, db).await?; + + // 6) Verify pass. Any `Verify`-priority range is always first; it is small + // (a short reorg-check window), so we scan it serially — pipelining it buys + // nothing and the loop may re-request ranges after each one. + loop { + let scan_ranges = db + .suggest_scan_ranges() + .map_err(|e| CoreError::Crypto(format!("suggest_scan_ranges: {e}")))?; + match scan_ranges.first() { + Some(sr) if sr.priority() == ScanPriority::Verify => { + let sr = sr.clone(); + let blocks = download_blocks_pipelined(client, &sr).await?; + let chain_state = + download_chain_state_pipelined(client, sr.block_range().start - 1).await?; + let src = MemBlockSource(blocks); + if scan_batch(params, &src, db, &chain_state, &sr)? { + // Ranges changed; re-request and re-check for a Verify range. + continue; } - None => break, + // Cache and scanned data are locally consistent; done verifying. + break; } - height = height + 1; + _ => break, } - Ok(blocks) } - async fn insert(&self, compact_blocks: Vec) -> Result<(), Self::Error> { - let mut metas = Vec::with_capacity(compact_blocks.len()); - for cb in &compact_blocks { - let meta = BlockMeta { - height: BlockHeight::from_u32(cb.height as u32), - block_hash: BlockHash::from_slice(&cb.hash), - block_time: cb.time, - sapling_outputs_count: cb.vtx.iter().map(|tx| tx.outputs.len() as u32).sum(), - orchard_actions_count: cb.vtx.iter().map(|tx| tx.actions.len() as u32).sum(), + // 7) Historic ranges, pipelined. Snapshot the suggested ranges, split them + // into batches, and download-ahead while scanning. + let scan_ranges = db + .suggest_scan_ranges() + .map_err(|e| CoreError::Crypto(format!("suggest_scan_ranges: {e}")))?; + let batches: Vec = scan_ranges + .into_iter() + .flat_map(|r| split_scan_range(r, batch_size)) + .collect(); + if batches.is_empty() { + return Ok(false); + } + + // Producer: download each batch's blocks + chain-state anchor and hand them + // over a bounded channel (capacity 2) so download runs up to two batches + // ahead of scanning. A cloned tonic client shares the underlying HTTP/2 + // connection, so this adds no new socket. + let (tx, mut rx) = tokio::sync::mpsc::channel::>(2); + let mut producer_client = client.clone(); + let producer = tokio::spawn(async move { + for sr in batches { + let dl_start = std::time::Instant::now(); + let blocks = match download_blocks_pipelined(&mut producer_client, &sr).await { + Ok(b) => b, + Err(e) => { + let _ = tx.send(Err(e)).await; + return; + } + }; + tracing::debug!( + "pipelined download: {} blocks for {} in {} ms", + blocks.len(), + sr, + dl_start.elapsed().as_millis() + ); + let chain_state = match download_chain_state_pipelined( + &mut producer_client, + sr.block_range().start - 1, + ) + .await + { + Ok(cs) => cs, + Err(e) => { + let _ = tx.send(Err(e)).await; + return; + } }; - std::fs::write(meta.block_file_path(&self.blocks_dir), cb.encode_to_vec())?; - metas.push(meta); + // If the consumer has hung up (ranges changed, or an error broke the + // loop) stop downloading. + if tx.send(Ok((sr, blocks, chain_state))).await.is_err() { + return; + } + } + }); + + // Consumer: scan each prefetched batch in order. `scan_batch` is CPU-bound + // and synchronous; on a multi-threaded runtime the producer keeps downloading + // the next batches on other worker threads while this one scans, which is the + // whole point. Scanning is transactional per batch, so bailing out early (or + // being dropped on cancellation) leaves the db consistent at a batch boundary. + let mut result = Ok(false); + let scan_run_start = std::time::Instant::now(); + let mut scanned_blocks: u64 = 0; + while let Some(item) = rx.recv().await { + let (sr, blocks, chain_state) = match item { + Ok(v) => v, + Err(e) => { + result = Err(e); + break; + } + }; + let n = blocks.len() as u64; + let src = MemBlockSource(blocks); + let scan_start = std::time::Instant::now(); + let outcome = scan_batch(params, &src, db, &chain_state, &sr); + // Per-batch scan cost and cumulative throughput. This is the CPU-bound leg + // (trial decryption + note-commitment tree updates); logging it here makes + // the download-vs-scan split visible when diagnosing slow syncs. + scanned_blocks += n; + let secs = scan_run_start.elapsed().as_secs_f64(); + tracing::info!( + "pipelined scan: {} blocks for {} in {} ms ({:.0} blocks/s cumulative over {} blocks)", + n, + sr, + scan_start.elapsed().as_millis(), + if secs > 0.0 { scanned_blocks as f64 / secs } else { 0.0 }, + scanned_blocks + ); + match outcome { + Ok(true) => { + // Ranges changed (continuity error or a new higher-priority + // range); restart the whole pass from fresh suggestions. + result = Ok(true); + break; + } + Ok(false) => {} + Err(e) => { + result = Err(e); + break; + } } - self.lock()?.write_block_metadata(&metas).map_err(io_err) } - async fn delete(&self, range: ScanRange) -> Result<(), Self::Error> { - // Remove cached blocks at/above the range start (keep everything below). - let start = u32::from(range.block_range().start); - self.lock()? - .truncate_to_height(BlockHeight::from_u32(start.saturating_sub(1))) - .map_err(io_err) + // Stop the producer: either it already finished, or we broke early and it + // should abandon any in-flight download. + producer.abort(); + result +} + +/// Scan one batch and interpret the outcome, mirroring the upstream `scan_blocks` +/// helper: on a continuity error, rewind the db and signal a restart; otherwise +/// signal a restart if scanning surfaced a higher-priority range. The in-memory +/// source needs no cache truncation on rewind (each batch is downloaded fresh). +fn scan_batch( + params: &Network, + src: &MemBlockSource, + db: &mut GroupDb, + chain_state: &ChainState, + scan_range: &ScanRange, +) -> Result { + use zcash_client_backend::data_api::chain::scan_cached_blocks; + + let scan_result = scan_cached_blocks( + params, + src, + db, + scan_range.block_range().start, + chain_state, + scan_range.len(), + ); + + match scan_result { + Err(ChainError::Scan(err)) if err.is_continuity_error() => { + // Rewind to at least one block before the error height, matching the + // upstream heuristic (10 blocks of slack). + let rewind_height = err.at_height().saturating_sub(10); + tracing::info!( + "chain reorg detected at {}, rewinding to {}", + err.at_height(), + rewind_height + ); + db.truncate_to_height(rewind_height) + .map_err(|e| CoreError::Crypto(format!("truncate on reorg: {e}")))?; + Ok(true) + } + Ok(_) => { + // If scanning added a range of higher priority than the one we just + // scanned, invalidate the current ordering and restart. + let latest = db + .suggest_scan_ranges() + .map_err(|e| CoreError::Crypto(format!("suggest_scan_ranges: {e}")))?; + Ok(latest + .first() + .map(|r| r.priority() > scan_range.priority()) + .unwrap_or(false)) + } + Err(e) => Err(CoreError::Crypto(format!("scan: {e}"))), } } -/// How many blocks each sync batch downloads and scans at once when the caller -/// gives no override. Larger batches amortize the per-batch gRPC round-trip and -/// database-transaction overhead across more blocks, which is the dominant cost -/// once trial decryption finds nothing (the common case for a wallet catching up -/// over empty history). Compact blocks are small, so a few thousand per batch is -/// comfortable in memory. Tunable per-install via settings; see [`sync_group`]. -pub const DEFAULT_SYNC_BATCH_SIZE: u32 = 5_000; +/// Download the subtree roots for all three shielded pools and store them, so the +/// note-commitment trees are initialized without replaying history. Ported from +/// the upstream `update_subtree_roots` (Sapling + Orchard + Ironwood). +async fn update_subtree_roots_pipelined( + client: &mut CompactTxStreamerClient, + db: &mut GroupDb, +) -> Result<(), CoreError> { + use zcash_client_backend::proto::service::ShieldedProtocol; + + // The concrete root-hash types (`sapling::Node`, `MerkleHashOrchard`) are + // inferred from the `put_*` calls below, so this compiles without naming the + // Sapling crate (not a direct dependency of this crate). + let sapling_roots = download_subtree_roots(client, ShieldedProtocol::Sapling).await?; + db.put_sapling_subtree_roots(0, &sapling_roots) + .map_err(|e| CoreError::Crypto(format!("put sapling subtree roots: {e}")))?; + + let orchard_roots = download_subtree_roots(client, ShieldedProtocol::Orchard).await?; + db.put_orchard_subtree_roots(0, &orchard_roots) + .map_err(|e| CoreError::Crypto(format!("put orchard subtree roots: {e}")))?; + + // Ironwood (NU6.3) is newer than some lightwalletd deployments — notably on + // testnet. Such a server rejects the Ironwood subtree-roots request with + // "invalid shielded protocol value". A server that doesn't know the pool has + // no Ironwood notes to report, so treat that one rejection as "no Ironwood + // roots yet" and keep syncing the other pools, rather than aborting the whole + // sync. (Ironwood value won't be tracked until an Ironwood-capable server is + // used, but on such a server there is none to miss.) Any other error still + // fails the sync. + match download_subtree_roots(client, ShieldedProtocol::Ironwood).await { + Ok(ironwood_roots) => { + db.put_ironwood_subtree_roots(0, &ironwood_roots) + .map_err(|e| CoreError::Crypto(format!("put ironwood subtree roots: {e}")))?; + } + Err(e) if is_invalid_shielded_protocol(&e) => { + tracing::warn!( + "lightwalletd rejected the Ironwood subtree-roots request ({e}); \ + continuing without Ironwood tree state — this server predates \ + Ironwood (NU6.3). Ironwood funds won't be tracked until you use an \ + Ironwood-capable server." + ); + } + Err(e) => return Err(e), + } -/// Clamp bounds for a caller-supplied batch size. Below the floor the round-trip -/// overhead dominates; above the ceiling a batch's worth of compact blocks can -/// spike memory (they are all held in a `Vec` while the batch is scanned). -pub const MIN_SYNC_BATCH_SIZE: u32 = 500; -pub const MAX_SYNC_BATCH_SIZE: u32 = 25_000; + Ok(()) +} -/// Sync the group's wallet: download and trial-decrypt compact blocks from -/// lightwalletd into the local db. Long-running; touches the network. -/// -/// `batch_size` is how many blocks to download and scan per batch; `None` uses -/// [`DEFAULT_SYNC_BATCH_SIZE`]. Any value is clamped into -/// `[MIN_SYNC_BATCH_SIZE, MAX_SYNC_BATCH_SIZE]`. -pub async fn sync_group( - data_dir: &Path, - group_id: &str, - network: WalletNetwork, - lightwalletd_url: &str, - db_key: &[u8], - batch_size: Option, - cancel: &tokio_util::sync::CancellationToken, +/// Stream the subtree roots for one shielded pool from lightwalletd. Ported from +/// the upstream `download_subtree_roots`. +async fn download_subtree_roots( + client: &mut CompactTxStreamerClient, + protocol: zcash_client_backend::proto::service::ShieldedProtocol, +) -> Result>, CoreError> +where + H: zcash_primitives::merkle_tree::HashSer, +{ + use zcash_client_backend::proto::service::GetSubtreeRootsArg; + + let request = GetSubtreeRootsArg { + start_index: 0, + shielded_protocol: protocol as i32, + max_entries: 0, + }; + + let mut stream = client + .get_subtree_roots(request) + .await + .map_err(|e| CoreError::Connection(format!("get_subtree_roots: {e}")))? + .into_inner(); + + let mut roots = Vec::new(); + while let Some(root) = stream + .message() + .await + .map_err(|e| CoreError::Connection(format!("subtree root stream: {e}")))? + { + let root_hash = H::read(&root.root_hash[..]) + .map_err(|e| CoreError::Crypto(format!("subtree root hash: {e}")))?; + roots.push(CommitmentTreeRoot::from_parts( + BlockHeight::from_u32(root.completing_block_height as u32), + root_hash, + )); + } + Ok(roots) +} + +/// Fetch the current chain tip and record it, so `suggest_scan_ranges` accounts +/// for newly mined blocks. Ported from the upstream `update_chain_tip`. +async fn update_chain_tip_pipelined( + client: &mut CompactTxStreamerClient, + db: &mut GroupDb, ) -> Result<(), CoreError> { - let batch_size = batch_size - .unwrap_or(DEFAULT_SYNC_BATCH_SIZE) - .clamp(MIN_SYNC_BATCH_SIZE, MAX_SYNC_BATCH_SIZE); - let (db_path, blocks_dir) = wallet_paths(data_dir, group_id, network); - std::fs::create_dir_all(&blocks_dir)?; - let mut db = open_db(&db_path, network, db_key)?; + let tip_height: BlockHeight = client + .get_latest_block(ChainSpec::default()) + .await + .map_err(|e| CoreError::Connection(format!("get_latest_block: {e}")))? + .get_ref() + .height + .try_into() + .map_err(|_| CoreError::Crypto("lightwalletd returned an invalid tip height".into()))?; + db.update_chain_tip(tip_height) + .map_err(|e| CoreError::Crypto(format!("update chain tip: {e}")))?; + Ok(()) +} - let mut inner = FsBlockDb::for_path(&blocks_dir) - .map_err(|e| CoreError::Crypto(format!("block cache: {e}")))?; - init_blockmeta_db(&mut inner) - .map_err(|e| CoreError::Crypto(format!("init block cache: {e}")))?; - let cache = FsCache { - inner: std::sync::Mutex::new(inner), - // FsBlockDb stores its compact-block files in `/blocks`, so the - // cache must read/write there (not the root we passed to `for_path`). - blocks_dir: blocks_dir.join("blocks"), +/// Download the compact blocks in `scan_range` into memory. Ported from the +/// upstream `download_blocks`, but returns the blocks instead of writing them to +/// a disk cache, so the producer can hand them straight to the scanner. +async fn download_blocks_pipelined( + client: &mut CompactTxStreamerClient, + scan_range: &ScanRange, +) -> Result, CoreError> { + use zcash_client_backend::proto::service::BlockRange; + + let start = BlockId { + height: scan_range.block_range().start.into(), + hash: vec![], }; + let end = BlockId { + height: (scan_range.block_range().end - 1).into(), + hash: vec![], + }; + let range = BlockRange { + start: Some(start), + end: Some(end), + pool_types: vec![], + }; + let mut stream = client + .get_block_range(range) + .await + .map_err(|e| CoreError::Connection(format!("get_block_range: {e}")))? + .into_inner(); - let mut client = connect(lightwalletd_url).await?; - let params = network.params(); - // `sync::run` scans in transactional batches, so dropping its future between - // batches leaves the db consistent (just short of the tip). That makes it - // safe to race against a cancellation token: "Sync Now" trips the token to - // abandon a stalled run, and a fresh sync resumes from where this one left - // off. Without this, a stuck stream would keep the sync pending forever. - tokio::select! { - biased; - _ = cancel.cancelled() => Err(CoreError::Cancelled), - res = zcash_client_backend::sync::run( - &mut client, ¶ms, &cache, &mut db, batch_size, - ) => res.map_err(|e| CoreError::Connection(format!("sync: {e}"))), + let mut blocks = Vec::new(); + while let Some(cb) = stream + .message() + .await + .map_err(|e| CoreError::Connection(format!("block stream: {e}")))? + { + blocks.push(cb); } + Ok(blocks) +} + +/// Fetch the chain-state anchor at `block_height` (the tree state just before a +/// range's first block). Ported from the upstream `download_chain_state`. +async fn download_chain_state_pipelined( + client: &mut CompactTxStreamerClient, + block_height: BlockHeight, +) -> Result { + client + .get_tree_state(BlockId { + height: block_height.into(), + hash: vec![], + }) + .await + .map_err(|e| CoreError::Connection(format!("get_tree_state: {e}")))? + .into_inner() + .to_chain_state() + .map_err(|e| CoreError::Crypto(format!("chain state: {e}"))) } /// Which shielded pool an action belongs to. Post-NU6.3 a single transaction can @@ -1967,6 +2380,97 @@ mod tests { assert!(WalletNetwork::Main.default_lightwalletd().starts_with("https://")); } + #[test] + fn annotate_sync_error_flags_non_ironwood_server() { + // The exact string a pre-Ironwood lightwalletd returns, as wrapped by the + // stock driver. + let raw = "sync: Error while communicating with lightwalletd server: \ + status: InvalidArgument, message: \"Error: Invalid shielded \ + protocol value.\""; + let out = annotate_sync_error(CoreError::Connection(raw.to_string())); + match out { + CoreError::Connection(m) => { + assert!(m.contains("doesn't support Ironwood"), "friendly headline: {m}"); + assert!(m.contains("network settings"), "actionable guidance: {m}"); + // Raw detail is preserved after the em-dash separator. + assert!(m.contains(" — "), "keeps raw detail: {m}"); + assert!(m.contains("Invalid shielded protocol value"), "raw text: {m}"); + } + other => panic!("expected Connection error, got {other:?}"), + } + } + + #[test] + fn annotate_sync_error_passes_through_unrelated() { + // An unrelated connection error is returned unchanged (no false headline). + let raw = "sync: Error while communicating with lightwalletd server: \ + transport error"; + match annotate_sync_error(CoreError::Connection(raw.to_string())) { + CoreError::Connection(m) => { + assert_eq!(m, raw); + assert!(!m.contains("Ironwood")); + } + other => panic!("expected Connection error, got {other:?}"), + } + // Cancellation is untouched. + assert!(matches!( + annotate_sync_error(CoreError::Cancelled), + CoreError::Cancelled + )); + } + + #[test] + fn detects_unsupported_ironwood_pool_for_graceful_skip() { + // The rejection the pipelined driver skips Ironwood roots on. + let raw = "get_subtree_roots: status: InvalidArgument, message: \ + \"Error: Invalid shielded protocol value.\""; + assert!(is_invalid_shielded_protocol(&CoreError::Connection(raw.into()))); + // Unrelated errors and non-connection variants are not mistaken for it. + assert!(!is_invalid_shielded_protocol(&CoreError::Connection( + "get_subtree_roots: transport error".into() + ))); + assert!(!is_invalid_shielded_protocol(&CoreError::Cancelled)); + } + + /// The pipelined driver must scan in the exact same batch units as the stock + /// driver, or its result could diverge. This locks the splitter's behaviour to + /// the upstream `sync::running` step-7 semantics: contiguous, priority- + /// preserving, `batch_size`-block sub-ranges that exactly cover the input and + /// never produce an empty range. + #[test] + fn split_scan_range_matches_upstream_batching() { + let h = BlockHeight::from_u32; + let range = ScanRange::from_parts(h(100)..h(1050), ScanPriority::Historic); + + // An evenly-plus-remainder range → full batches then a short tail. + let batches = split_scan_range(range.clone(), 400); + assert_eq!(batches.len(), 3); + assert_eq!(*batches[0].block_range(), h(100)..h(500)); + assert_eq!(*batches[1].block_range(), h(500)..h(900)); + assert_eq!(*batches[2].block_range(), h(900)..h(1050)); + // Priority is preserved on every sub-range. + assert!(batches.iter().all(|b| b.priority() == ScanPriority::Historic)); + // Contiguous cover: no gaps, no overlaps, no empty ranges. + assert!(batches.iter().all(|b| !b.is_empty())); + for w in batches.windows(2) { + assert_eq!(w[0].block_range().end, w[1].block_range().start); + } + assert_eq!(batches.first().unwrap().block_range().start, h(100)); + assert_eq!(batches.last().unwrap().block_range().end, h(1050)); + + // A range smaller than one batch → a single batch equal to the input. + let small = ScanRange::from_parts(h(10)..h(30), ScanPriority::ChainTip); + let one = split_scan_range(small.clone(), 5000); + assert_eq!(one.len(), 1); + assert_eq!(*one[0].block_range(), h(10)..h(30)); + + // A range that is an exact multiple of the batch size → no empty tail. + let exact = ScanRange::from_parts(h(0)..h(1000), ScanPriority::Historic); + let even = split_scan_range(exact, 500); + assert_eq!(even.len(), 2); + assert_eq!(*even[1].block_range(), h(500)..h(1000)); + } + /// The receive address the wallet's key crate (`zcash_keys`) derives from /// our group UFVK must equal the address our derivation produced — proving /// our deterministically-derived keys are standard, wallet-usable Orchard diff --git a/src-tauri/src/commands/wallet.rs b/src-tauri/src/commands/wallet.rs index 9a10339..2275f93 100644 --- a/src-tauri/src/commands/wallet.rs +++ b/src-tauri/src/commands/wallet.rs @@ -230,11 +230,15 @@ pub async fn wallet_sync(state: State<'_, AppState>, group_id: String) -> AppRes // thrash). We register+cancel the prior same-group token above first, so a // restart of this group releases the gate before we wait on it; an // active-wallet switch cancels the other group's sync, so this rarely blocks. + // Holding it for the whole run also means a restarting sync can't race a + // cancelled one's still-open db connection — the "database is locked" case. let _gate = state.sync_gate.lock().await; - let batch_size = state.load_settings().sync_batch_size; + let opts = wallet::SyncOptions { + batch_size: state.load_settings().sync_batch_size, + }; let result = wallet::sync_group( - &state.data_dir, &group_id, network, &url, db_key.as_ref(), batch_size, &cancel, + &state.data_dir, &group_id, network, &url, db_key.as_ref(), opts, &cancel, ) .await; diff --git a/src-tauri/src/state.rs b/src-tauri/src/state.rs index 7f8cc3b..b41173d 100644 --- a/src-tauri/src/state.rs +++ b/src-tauri/src/state.rs @@ -134,6 +134,10 @@ pub struct AppState { /// `wallet_sync` holds this for its whole run; combined with cancelling the /// previous group's sync on an active-wallet switch, it guarantees the app's /// processing stays focused on a single wallet. + /// + /// This global gate also subsumes the earlier per-group sync lock: since only + /// one sync ever runs, a restarting sync can no longer race a cancelled one's + /// still-open db connection, so the "database is locked" fix comes for free. pub sync_gate: Mutex<()>, /// Epoch-millis of the last user activity, used to drive the idle auto-lock. pub last_activity: AtomicI64, diff --git a/src/screens/DkgWizard.tsx b/src/screens/DkgWizard.tsx index 8680d83..f258b3e 100644 --- a/src/screens/DkgWizard.tsx +++ b/src/screens/DkgWizard.tsx @@ -189,7 +189,7 @@ export default function DkgWizard() {

New DKG ceremony

-
+
-

- You are about to sign and broadcast a transaction on the Zcash - mainnet. This will move real funds. -

- - +
@@ -146,8 +141,7 @@ function MainnetConfirmModal({ onChange={(e) => setAck(e.target.checked)} /> @@ -250,13 +244,12 @@ function ReceiveShieldCard({ groupId, fallback }: { groupId: string; fallback: s

Receive / Shield into group

- Send Zcash to this unified address to fund the group. Funds arrive in the - group's shielded Ironwood pool and become spendable by - the threshold. To shield transparent funds, send them - here from a personal wallet — the receive itself is the shielding step. + Send Zcash to this address to fund the group. Funds become spendable by + the threshold; sending from a transparent wallet shields them in the same + step.

0.001 ${unit(isMainnet)} above fees)` + `Legacy balance too low to move (need > 0.001 ${unit(isMainnet)} above fees)` ); return walletPrepareSend(group.id, addr, orchard - CONSOLIDATE_FEE_BUFFER); }, @@ -1061,6 +1054,15 @@ function GroupWallet({ group, isMainnet }: { group: GroupSummary; isMainnet: boo if (!group.ciphersuite.includes("Pallas")) return null; const s = status.data; + // Surface a failed status read instead of masking it as a permanent "Setting + // up…". `init` only runs *after* status returns, so if the status read errors + // (keystore locked, key derivation, a db problem), the wallet otherwise sits on + // the spinner forever with nothing in the logs — exactly the "stuck setting up" + // symptom. This read is local (no network), so an error here is not the server. + const statusErr = + status.isError && !s + ? ((status.error as unknown as AppError)?.message ?? String(status.error)) + : null; // Prefer the live probe while a sync is running; the cached status is stale // until the whole catch-up returns. const live = sync.isPending ? progress.data : undefined; @@ -1075,14 +1077,26 @@ function GroupWallet({ group, isMainnet }: { group: GroupSummary; isMainnet: boo return (

-

Wallet (Zcash · Orchard + Ironwood)

- {!s || (!s.initialized && (init.isPending || !err)) ? ( +

Wallet

+ {statusErr ? ( + <> +

+ Couldn't read this wallet: {statusErr}. This is a local read (no + network), so it usually means the keystore is locked or the wallet + database can't be opened — not the lightwalletd server. +

+ + + ) : !s ? ( +

Loading wallet…

+ ) : !s.initialized && (init.isPending || !err) ? (

Setting up the group's view-only wallet…

) : !s.initialized ? ( <>

- Couldn't set up the wallet — check the lightwalletd endpoint on the{" "} - Wallet page, then retry. + Couldn't set up the wallet{err ? `: ${err}` : ""}. Check the + lightwalletd endpoint on the Wallet page, + then retry.

@@ -1297,24 +1304,7 @@ function GroupWallet({ group, isMainnet }: { group: GroupSummary; isMainnet: boo /> ) : ( <> - {isMainnet && ( -
- - ⚠ Mainnet — transactions move real ZEC and - are irreversible. Verify every address and amount carefully. - -
- )} - - {/* Mode toggle: shielded Orchard send vs. unshield to transparent. */} + {/* Mode toggle: shielded send vs. unshield to transparent. */}
)} @@ -1492,7 +1481,7 @@ function GroupWallet({ group, isMainnet }: { group: GroupSummary; isMainnet: boo {isVote ? "Cast vote" : isMigration - ? "Migrate Orchard → Ironwood" + ? "Move to Ironwood" : isConsolidation ? "Consolidation transaction" : draft.is_unshield @@ -1516,12 +1505,10 @@ function GroupWallet({ group, isMainnet }: { group: GroupSummary; isMainnet: boo
{isMigration ? ( - Sweeps the group's sealed Orchard balance back to - its own address; because every post-NU6.3 shielded output lands in - the Ironwood pool, this moves the funds across the - turnstile into Ironwood.{" "} + Moves the group's legacy funds into the active{" "} + Ironwood pool via a self-send.{" "} {draft.spends.length} note{draft.spends.length !== 1 ? "s" : ""}{" "} - will be signed (one round each). A small network fee applies. + will be signed. A small network fee applies. ) : ( @@ -1536,9 +1523,9 @@ function GroupWallet({ group, isMainnet }: { group: GroupSummary; isMainnet: boo {!isConsolidation && draft.is_unshield && (
- Unshield — moves {zec(draft.amount_zatoshis)} {unit(isMainnet)} from - the group's shielded Orchard pool to a transparent address. The amount - and recipient will be publicly visible on-chain. + Unshield — moves {zec(draft.amount_zatoshis)} {unit(isMainnet)} to + a transparent address, so the amount and recipient will be{" "} + public on-chain.
)} @@ -1679,7 +1666,28 @@ function GroupWallet({ group, isMainnet }: { group: GroupSummary; isMainnet: boo )} )} - {err &&
{err}
} + {err && } +
+ ); +} + +/** Render an error. When the backend has attached a raw server detail after + * " — " (as `annotate_sync_error` does for known, actionable failures), show the + * actionable headline prominently and keep the raw text below it, dimmed, so the + * user sees what to do first without losing the log detail for debugging. */ +function SyncErrorView({ text }: { text: string }) { + const sep = text.indexOf(" — "); + if (sep === -1) { + return
{text}
; + } + const headline = text.slice(0, sep); + const detail = text.slice(sep + 3); + return ( +
+ {headline} +
+ {detail} +
); } @@ -1829,7 +1837,7 @@ function SendSessionPanel({ )} {meta.isUnshield && (
- Unshield — moving funds from the group's shielded Orchard pool to a transparent address (publicly visible on-chain). + Unshield — moving funds to a transparent address (public on-chain).
)}
Sending
@@ -2027,7 +2035,7 @@ export function GroupKeys({ group, masked = false }: { group: GroupSummary; mask {orchard && keys.data && ( <> - +
- The viewing key (nk,{" "} - rivk) is derived - deterministically from the group's ak, - so every member computes this same address. Funds sent here are - spendable only by a threshold of the group. The UFVK grants{" "} - viewing access — share it only within the group. The - address is encoded for the network selected on the{" "} - Wallet page (mainnet by default). + Every member derives this same address from the group's key. Funds + sent here are spendable only by a threshold of the group. The UFVK + grants viewing access — share it only within the group.
)} {orchard && keys.isError && (
- Could not derive the Orchard address for this group. + Could not derive the address for this group.
)} @@ -2485,7 +2488,7 @@ export function GroupWalletPage() {

{group.description || "(unnamed group)"} — Wallet

- A Zcash wallet is only available for RedPallas (Orchard) groups.{" "} + A Zcash wallet is only available for RedPallas groups.{" "} Back to group details.

@@ -2495,21 +2498,22 @@ export function GroupWalletPage() { return (
-

+

{group.description || "(unnamed group)"} — Wallet - {isMainnet && ( - - ⚠ MAINNET - - )} + + {isMainnet ? "Mainnet" : "Testnet"} +

← Group details diff --git a/src/screens/SessionSetup.tsx b/src/screens/SessionSetup.tsx index 773c5a7..235e188 100644 --- a/src/screens/SessionSetup.tsx +++ b/src/screens/SessionSetup.tsx @@ -42,12 +42,9 @@ function TransportSecurityNote() {
Transport is not authentication. However you expose the - server — direct, tunnel, or reverse proxy — FROSTd messages stay - end-to-end authenticated and encrypted by the app's own Noise layer - (participants are verified by their communication public keys). A tunnel - or TLS proxy only makes the server reachable; it never replaces that - application-layer security, so keep participant key verification in place - regardless of the transport you choose. + server, FROSTd messages stay end-to-end encrypted by the app's own Noise + layer, and participants are verified by their keys. Keep that key + verification in place whatever transport you use.
); @@ -84,10 +81,10 @@ export default function SessionSetup() { {!configured ? (
- Welcome — let's configure your session. A FROST - ceremony coordinates through one frostd{" "} - server. Choose your role and how you'll connect below, then save it. - You can change this any time from Zcash → Session Configuration. + Welcome — configure your session. A ceremony runs + through one frostd server. Pick your role + below and save; change it anytime from{" "} + Zcash → Session Configuration.
) : ( diff --git a/src/screens/Wallet.tsx b/src/screens/Wallet.tsx index b247d9a..664b7ec 100644 --- a/src/screens/Wallet.tsx +++ b/src/screens/Wallet.tsx @@ -10,81 +10,6 @@ import { LightwalletdInfo, } from "../ipc/commands"; -/** Themed confirmation dialog for switching to mainnet — replaces the plain - * browser confirm() so it matches the application's design language. */ -function SwitchNetworkModal({ - onConfirm, - onCancel, -}: { - onConfirm: () => void; - onCancel: () => void; -}) { - return ( -
-
-
-
- CYZE · NETWORK SETTINGS -
- Switching to Mainnet -
- -
- - Mainnet transactions move real ZEC and are{" "} - irreversible once broadcast. Only switch if you are - ready to handle live funds. - -
- -

- You can switch back to testnet at any time from this page. -

- -
- - -
-
-
- ); -} - /** Known public lightwalletd endpoints per network (user can also type their own). */ const PRESETS: Record = { test: [ @@ -104,7 +29,6 @@ export default function Wallet() { const [info, setInfo] = useState(null); const [testErr, setTestErr] = useState(null); const [testing, setTesting] = useState(false); - const [showMainnetModal, setShowMainnetModal] = useState(false); // Mainnet is the default (matches the backend), so the app opens on the network // it is actually used on rather than quietly pointing at testnet. @@ -146,44 +70,21 @@ export default function Wallet() { } }; - const savedNet = config.data?.network ?? "main"; const isMainnet = net === "main"; return (

Wallet

- {/* Persistent mainnet danger banner — shown any time the active network - is main, both here and as a reminder before the user navigates away. */} - {savedNet === "main" && ( -
- - ⚠ You are on Mainnet. Transactions here move{" "} - real ZEC. Double-check every recipient address and - amount before signing. Signed transactions are irreversible once - broadcast. - -
- )} -

- Cyze syncs Zcash shielded funds as a light client: it scans compact - blocks locally with your group's viewing key and talks to a configurable{" "} - lightwalletd server (no full node - required). Start on testnet to try it with faucet funds; - switch to mainnet once you're ready. + Cyze syncs Zcash as a light client against a configurable{" "} + lightwalletd server — no full node + needed. Start on testnet with faucet funds; switch to mainnet when ready.

Network

-
+
- - {isMainnet - ? "Real ZEC — transactions are irreversible." - : "Safe for testing with faucet funds."} + + {isMainnet ? "Live network — real ZEC." : "Test network — faucet funds."}
@@ -282,18 +182,6 @@ export default function Wallet() {
- - {showMainnetModal && ( - { - setShowMainnetModal(false); - setNetwork("main"); - setUrl(""); - }} - onCancel={() => setShowMainnetModal(false)} - /> - )} -
); } diff --git a/src/screens/Wallets.tsx b/src/screens/Wallets.tsx index 83a5511..9df3952 100644 --- a/src/screens/Wallets.tsx +++ b/src/screens/Wallets.tsx @@ -84,8 +84,8 @@ export default function Wallets() { ) : wallets.length === 0 ? (

- No Zcash wallets yet. A wallet is created for each RedPallas (Orchard) - group — create or join one under 2 · Groups. + No Zcash wallets yet. A wallet is created for each RedPallas group — + create or join one under 2 · Groups.

) : ( diff --git a/src/styles.css b/src/styles.css index debed20..1b8ba87 100644 --- a/src/styles.css +++ b/src/styles.css @@ -106,6 +106,11 @@ h3 { font-size: 15px; color: var(--text-dim); } label { display: block; font-size: 13px; color: var(--text-dim); margin-bottom: 6px; } +/* DKG wizard: make each section's title read as a heading — brighter and bold — + so it stands out from the dim helper text below it. Excludes the participant + checkbox rows, which are option labels, not section titles. */ +.dkg-form label:not(.multiselect-option) { color: var(--text); font-weight: 600; } + input[type="text"], input[type="password"], input[type="number"], textarea, select { font-family: inherit; width: 100%;