zeta

A Zcash on-chain data stack: pull raw chain data from your own node, shape it into tidy tables with dbt on DuckDB, show it in a dashboard, and sell it per row over x402.

The architecture follows tiders-stack, with DuckDB in place of ClickHouse and a Zcash-native ingestion service in place of Tiders.

#	Component	Role	What zeta uses
1	Database	Stores the data	DuckDB (embedded) over a Parquet lake
2	Ingestion	Pulls raw data from the chain	zeta (ingestion/), reading Zebra/Zakura JSON-RPC
3	Transformation	Raw → useful tables	dbt-duckdb (dbt/)
4	Dashboard	Static site over the marts	Evidence (dashboards/zcash/)
5	Server	Serves the dashboard, sells the data	tiders-x402-server (x402-server/)

 Zakura / Zebra node ──RPC──▶ ingestion ──▶ data/raw/zcash/<table>/*.parquet      (append-only lake)
                                                   │
                                        dbt build  ▼  read_parquet()
                                     data/warehouse/zeta{_dev}.duckdb             (stg views, int + mart tables)
                                                   │
                                          publish  ▼  mart__* tables only
                                     data/serve/zeta.duckdb  ──▶ Evidence build ──▶ dashboards/zcash/build
                                                   │                                       │
                                                   └──────────▶ x402-server :4021 ◀────────┘
                                                               /api/query (paid)  /zcash/ (dashboard)

Quick start

Requirements: Docker with Compose, and a synced Zebra-compatible node with RPC enabled.

cp .env.example .env          # set ZCASH_RPC_URL and PAY_TO_ADDRESS
make up                       # build + start the containers

# Recent window first (a few minutes), full history later:
make zcash-ingest FROM_BLOCK=3200000
make zcash-dbt
make zcash-publish
make zcash-dashboard
# or all four: make zcash

Open http://localhost:4021/ for the landing page and http://localhost:4021/zcash/ for the dashboard.

Full history

make zcash-ingest FROM_BLOCK=0   # backfills below what the lake already holds
make zcash FULL_REFRESH=1        # incremental models must rebuild after a backfill

On Zakura the backfill runs at roughly 1,900 blocks/s on recent blocks, dropping to about 170 blocks/s through the 2022 sandblasting era (heights ~1.7M–2.0M), where blocks are up to 2 MB each. Full history gives exact fees for every transaction and exact transparent address balances. With a partial window, spends of outputs created before the window are left unresolved (fee_zat is NULL).

Node: Zakura

zeta reads from the zakura container (a Zebra fork with the same RPC surface). Its RPC is enabled through environment variables, since Zakura reads ZAKURA_<SECTION>__<KEY> settings:

docker run -d --name zakura --restart unless-stopped \
  -v zakurad-cache:/home/zebra/.cache/zakura \
  -p 8233:8233 -p 127.0.0.1:8242:8232 \
  -e ZAKURA_RPC__LISTEN_ADDR=0.0.0.0:8232 \
  -e ZAKURA_RPC__ENABLE_COOKIE_AUTH=false \
  zakuracore/zakura:1.4.0

RPC is published on loopback only, so cookie auth is off. ZCASH_RPC_URL=http://127.0.0.1:8242. Any Zebra node works too; point ZCASH_RPC_URL at it.

Benchmarking nodes

make bench-nodes    # Zebra :8232 vs Zakura :8242 -> benchmarks/<timestamp>-zebra-vs-zakura.md

The script checks that every node yields byte-identical extracted rows. It then measures raw node serving speed, zeta with one process, and zeta with parallel workers, across the 2019, 2022-spam and recent eras. Zakura's biggest lead is in the spam era, which dominates full-history backfill.

The pieces

1. Ingestion: ingestion/zeta

python -m zeta ingest calls getblock <h> 2 and flattens each block into five raw tables. It runs ZETA_WORKERS processes in parallel (default 4), one chunk each, each with ZETA_RPC_CONCURRENCY requests in flight (default 8). One Python process can't keep up with the node on large blocks, so the parallel workers roughly double spam-era throughput.

Table	Grain
blocks	block
block_value_pools	block × pool (transparent, sprout, sapling, orchard, ironwood, lockbox; new pools become new rows)
transactions	transaction, with per-pool component counts and value balances
transparent_inputs	transparent input
transparent_outputs	transparent output

Note ciphertexts, proofs and signatures are dropped: they are opaque without viewing keys and would dominate storage.

Raw data is written as Parquet, not into DuckDB. DuckDB allows one writer process per file, and the x402 server holds its file open for as long as it runs. Parquet files keep ingestion, dbt and the server from ever contending for a lock.

Each chunk of ZETA_CHUNK_SIZE blocks is one file per table, plus a _chunks/<range>.done marker written last. That makes runs resumable and crash-safe; chunks finish out of order under parallel workers, and any gaps an interrupted run leaves are filled on the next run. The partial chunk at the tip is re-fetched whole on the next run, so the lake keeps one file per range. If a chunk's last block is no longer on the node's best chain, that chunk is rewound. make zcash-status shows coverage and gaps.

2. Transformation: dbt/

• stg__zcash__*: views over the Parquet lake: renames only.
• int__zcash__inputs_valued: joins every transparent input to the output it spends. getblock doesn't return prevout values, so this join is what makes fees computable.
• mart__zcash__*: the product:

Mart	What it is
transactions	Every tx: tx_type (transparent / shielding / deshielding / fully_shielded / mixed / coinbase), pools used, fee_zat, net_shielded_flow_zat
blocks	Every block: payout address, fees, interval, per-pool deltas, note commitment tree sizes
value_pools_daily	End-of-day balance and net flow per pool
chain_daily	Activity, tx mix, fees, transparent vs shielded supply
miners_daily	Blocks and share per coinbase payout address
transparent_address_balances	Balance and lifetime activity per t-address

The large models are incremental on height. Each run rebuilds the last 100 blocks to absorb shallow reorgs. Tests include a consensus invariant: per block, the sum of each shielded pool's tx value balances must equal the pool delta the node reports.

3. Publish: scripts/publish.py

This step copies the mart__* tables from a warehouse into a fresh data/serve/zeta.duckdb, swaps it in atomically, and restarts the server so it reopens the file. dbt never writes to the file the server reads. The same file feeds the Evidence build, so the dashboard shows exactly what is for sale.

4. Dashboard: dashboards/zcash

The dashboard was scaffolded with tiders-x402-server dashboard zcash, which provides the wallet connect button and the x402 download components. You edit pages/*.md and sources/local_duckdb/*.sql. Pages: overview, value pools, mining, and a dataset page with a preview and paid download for each table.

5. Server: x402-server/zeta-x402-server.yaml

The config lists which tables are for sale and their prices. Row-level tables cost 0.0001 USDC per row, or 0.00005 per row at 10,000 rows and up. Daily aggregates cost 0.001 per row. Price and dashboard edits hot-reload. The default token is USDC on Base Sepolia (testnet). For real payments set X402_TOKEN=usdc/base and a mainnet facilitator (e.g. Coinbase CDP; see the commented cdp_auth block).

The server only accepts a safe SQL subset (no joins or subqueries), so every mart is a flat, self-contained table.

Dev vs prod

dbt writes to data/warehouse/zeta_dev.duckdb by default. TARGET=prod writes zeta.duckdb. make zcash-publish publishes whichever target you pass, dev by default.

make zcash TARGET=prod

Layout

.
├── docker-compose.yaml        # ingestion, dbt, x402-server, dashboard-builder (+ optional cloudflared)
├── Makefile                   # includes projects/*.mk
├── projects/zcash.mk          # zcash-ingest / -dbt / -publish / -dashboard / -status / -scaffold
├── .env.example
├── ingestion/zeta/            # (2) rpc.py, extract.py, lake.py, cli.py
├── dbt/models/{stg,int,mart}/ # (3)
├── scripts/publish.py         # warehouse -> serve
├── dashboards/zcash/          # (4) Evidence project
├── x402-server/               # (5) server config + Dockerfile
└── data/                      # git-ignored: raw/, warehouse/, serve/

Keeping it fresh

Run the pipeline on a schedule, for example with cron every 30 minutes:

*/30 * * * * cd /path/to/zeta && make zcash TARGET=prod >> data/pipeline.log 2>&1

Exposing it publicly

Uncomment the cloudflared service in docker-compose.yaml, put TUNNEL_TOKEN in .env, and point the tunnel's public hostname at http://x402-server:4021. Set X402_BASE_URL to that hostname.