feature(Files): Download one file per day for the daily parquet layout

[szemyd]

What

From 2026-08-01 the API returns one parquet per day rather than one per month, and FileInfo gains an optional day (dream-faster/unravel-router#804, now merged). The CLI discarded that field.

FileInfo had no Day, so processor.go named every output from year and month alone:

filename := fmt.Sprintf("%d-%02d.parquet", f.Year, f.Month)

Every daily file in a month therefore resolved to the same path:

2026-08-01 ─┐
2026-08-02 ─┼─> outputDir/2026-08.parquet
   ...     ─┘

Up to --max-concurrent goroutines then raced to os.Create (which truncates) and io.Copy into that one path. A 31-day August left a single file holding one arbitrary day's data, or an interleave of several — and the CLI exited 0 reporting Successfully downloaded 31 files, listing the same name 31 times.

Silent, so a user has no signal that 30 days are missing.

Changes

Naming. Daily files become 2026-08-01.parquet; monthly files keep 2026-08.parquet. Both parts are zero-padded so a directory listing sorts by date. That is a choice about local filenames and deliberately not a mirror of the R2 key, which writes the month unpadded — there's a comment saying so, because "aligning" them would be a natural-looking mistake.

Day is *int, not int. An absent field stays distinct from a literal 0. With a plain int, a stray 0 from the API would read as "monthly" and put a daily file back on the monthly file's path — the exact failure being fixed.

A collision guard. Names are resolved and checked for duplicates before anything is written; a colliding set fails the run with both indices and the shared name. This, rather than the naming, is what makes the bug class unreachable — any future response shape that maps two files to one path now errors instead of letting the last writer win.

Retry-loop handle leak (same function, same symptom). downloadToFile used defer inside its retry loop for both the response body and the destination file. A deferred call runs at function return, not at the end of an iteration, so every attempt's handles stayed open for the whole retry sequence — a failed attempt could still be flushing into a path a later attempt had already truncated. One attempt is now downloadOnce, which closes what it opens and reports whether the failure is worth retrying (transport and HTTP errors yes, a local filesystem error no, preserving the previous behaviour of not retrying os.Create failures).

README. The Output section described files as "named by year and month". It now covers both granularities, the changeover date, and shows the two filename shapes.

Tests

10 new cases in daily_files_test.go, covering naming, chronological sortability, collision rejection, optional-day JSON decoding, and end-to-end CLI runs against a stub API.

The load-bearing one is TestCLI_DailyFilesWriteOnePerDay: three days in, three files out, each asserted to hold its own day's bytes — distinct names alone would still pass if every file held the same content. Pre-fix it produces one file.

Reverting the naming fix fails four tests, including both end-to-end cases. gofmt and go vet clean.

The five integration tests that require APERIODIC_API_KEY fail identically on main without it (requireAPIKey calls t.Fatal, not t.Skip); CI supplies the secret.

Note

gofmt also realigned the pre-existing Exchange const block in types.go while formatting the new field — three lines of whitespace, unrelated to the behaviour change.

Related

• dream-faster/unravel-router#804 — the API change that adds day (merged)
• feature(Files): Order downloaded parquet files by day client#30 — the same fix for the Python SDK, where the symptom is misordered rows rather than lost files
• dream-faster/unravel-router#805 — docs and changelog for the changeover

---

Generated by Claude Code

[szemyd]

QA: daily parquet downloads — PASS

An August range produces one correctly-named, distinct file per day. 2026-08-01 … 2026-08-05, five files on disk, five distinct MD5s, chronological ls order, none empty. The same command on main writes one file while printing Successfully downloaded 5 files and exiting 0.

Tested against https://staging.aperiodic.io/api/v1, not production — see Deviations below.

Phase	Check	Result	Evidence
4	Old binary vs new, same command	PASS	main (7f83bbc, no parquetFilename) → 1 file 2026-08.parquet, stdout lists 2026-08.parquet ×5, exit=0. Branch (a45b8da) → 5 files, exit=0. Old file's MD5 258484c1b224a45f706a4265c0f85a08 is byte-identical to the branch's 2026-08-03.parquet — an arbitrary middle day survived; 4 of 5 days silently lost
0	API serving daily files	PASS	2026-08-01.parquet (5766 B), 2026-08-02.parquet (5760 B), 2026-08-03.parquet (5754 B)
1.1	Five files exist	PASS	ls "$OUT"/*.parquet | wc -l → 5
1.2	Named 2026-08-01 … -05, no bare 2026-08.parquet	PASS	exactly those 5 names; bare 2026-08.parquet absent
1.3	All non-empty	PASS	find -size 0 → no output
1.4	Five distinct digests	PASS	7046cefb…, 8b81d0cf…, 258484c1…, c0e518ae…, 7735b80a… → 5 unique
1.5	ls order chronological	PASS	2026-08-01 → -02 → -03 → -04 → -05
2	Boundary 2026-07-28…2026-08-03	PASS	2026-07.parquet (67933 B) + 3 dailies = 4 files, 4 distinct digests. No bare 2026-08.parquet — month is not double-served
3	Pre-cutover monthly unchanged	PASS	2025-01.parquet, 2025-02.parquet, 2025-03.parquet; zero day-suffixed names; 3 distinct digests
5	Files are real parquet holding the claimed day	PASS	2026-08-01: shape (24, 25), time ∈ 00:00:00–23:00:00 of 08-01. 2026-08-02: (24, 25), time ∈ 08-02. No stray hive columns
6	gofmt -l .	PASS	no output
6	go vet ./...	PASS	no output
6	10 new cases in daily_files_test.go	PASS	10 func Test*, all --- PASS
6	5 integration tests with API key set	PASS in CI (not reproducible locally)	CI green on this PR — all 3 Test jobs pass, using secrets.APERIODIC_API_KEY against production. They fail locally only because the key used for this QA is staging-scoped, while requireAPIKey pins tests to production (client_test.go:24). Pre-existing and byte-identical between main and this branch

Setup: branch a45b8da, Go 1.26.2, go build -o ./aperiodic-branch ./cmd/aperiodic. Every phase used a fresh mktemp -d; file counts come from ls/md5, never the CLI's own success line. Dataset: l2_imbalance / okx-perps / perpetual-BTC-USDT:USDT / 1h.

Blocking issues

None. Nothing found here should stop #17 from merging.

Deviations from the QA brief

1. Ran against staging, not production. The brief states the API change is "merged and live" and says to leave APERIODIC_API_URL unset. It isn't deployed to production — confirmed: a production August range returns no daily files, and a production API-key lookup returns PostgREST PGRST116 ("0 rows") surfaced as 401. All phases therefore ran with APERIODIC_API_URL=https://staging.aperiodic.io/api/v1 plus a Cloudflare Access service token. These results validate the CLI against the staging deployment only.
2. Phase 4 used a binary built from main, not a released one. No released aperiodic exists on this machine and the brief says not to install one. Building main (7f83bbc) from source is a tighter comparison anyway — same toolchain, same source tree, only the fix differing.

Not the CLI's fault

• The 5 integration tests failing on my machine were environmental only (staging-scoped key vs. production-pinned tests). CI is green on this PR. Nothing to fix.
• No dataset gaps hit. l2_imbalance on okx-perps returned complete August data and read cleanly — the in-flight hive-column repair did not affect any file opened here.

Couldn't determine

• Production behaviour. Everything above is staging. Needs the router deploy plus a production-provisioned key to re-verify.
• The collision guard on real data. resolveFilenames aborting a run is covered only by the mock test TestCLI_CollidingFilesFailTheRun; the live API can't be made to double-serve a month on demand.
• Breadth. One metric/exchange/symbol/interval end-to-end. Other datasets are unexercised, though naming is metric-independent in parquetFilename.
• Real-API coverage of the daily path. The 5 integration tests do exercise the live production API, but only the monthly layout (2024 data). The daily-file behaviour is covered by mocks alone until production serves daily objects.

---

Updated after checking CI: the 5 integration tests pass there. An earlier version of this comment listed them as FAIL, based on a local run with a staging-scoped key.