feature(Files): Order downloaded parquet files by day by szemyd · Pull Request #30 · aperiodic-io/client

szemyd · 2026-08-13T19:50:10Z

What

From 2026-08-01 the API serves one parquet per day rather than one per month (see dream-faster/unravel-router#804), so a single response can contain many files sharing the same year/month.

The bug this fixes

_get_files_from_bucket_async sorted downloaded files on (year, month):

results_sorted = sorted(results, key=lambda x: (x[0], x[1]))

With daily files every file in a month collides on that key. sorted is stable, so ordering fell through to the order results happened to be in — and on the default path (show_progress=True) results are collected via asyncio.as_completed, i.e. in download-completion order. Same-month rows were therefore concatenated nondeterministically.

There is no safety net downstream: the trailing filter/sort is guarded on has_column(combined, "timestamp"), but the parquets carry the interval start as time, not timestamp, so that branch never runs. Concat order is the final row order.

The fix

Each download now carries a (year, month, day) sort key. Monthly files have no day and fall back to 0, so they sort ahead of that month's daily files.

download_parquet_bytes is untouched in both transports; the key is attached by a thin _download_file wrapper in endpoints/utils.py.

Public API

Unchanged. Callers still pass start_date/end_date and get back a single DataFrame. The only type change is FileInfo.day, added as NotRequired[int], so responses without it keep validating.

Tests

tests/test_daily_files.py — 6 cases driving _get_files_from_bucket_async with mocked presigned-URL and download layers, where each file's payload is a one-row parquet tagged with a marker so the concatenated order is directly assertable. Downloads are forced to complete out of order via staggered delays.

Covers: daily files in day order, ordering with and without the progress bar, crossing a month boundary, monthly-only responses unchanged, the empty-file-list path, and a monthly file sorting ahead of the same month's daily files.

That last case no longer describes production — the cutover sits on a month boundary, so a month is served either monthly or daily and never both — but it is kept and labelled as such, because it pins what the missing-day fallback means and stops the client silently scrambling if a month is ever served both ways.

Verified the tests are discriminating: reverting the sort key to the old day-blind form fails the two same-month ordering cases.

Unit suite passes (26). tests/test_polars_backend.py::test_get_metrics_returns_polars_dataframe fails locally both with and without this change because it needs APERIODIC_API_KEY; it is an integration test and unrelated. ruff check reports the same 2 pre-existing PLR0917 findings before and after.

Why this is in scope

The unravel-router apps/data bump workflow dispatches this repo's tests against staging, so the daily-file behavior needs coverage here for that gate to be meaningful.

From 2026-08-11 the API serves one parquet per day rather than one per month, so a single response can contain many files sharing the same year/month. Downloads complete out of order and the default progress-bar path collects them via asyncio.as_completed, so sorting on (year, month) alone left same-month rows in a nondeterministic order before concat. Each download now carries a (year, month, day) sort key. Monthly files have no `day` and sort ahead of that month's daily files, matching their real chronology: a month is only served as a monthly file for the days before the cutover. Public signatures are unchanged — callers still pass start_date/end_date. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YW171yVjpD7c59PtLT4zKz

The cutover moved to the first of a month, so the API no longer serves a month from both layouts. The mixed monthly/daily ordering case stays covered — it is what the missing-`day` fallback means — but no longer claims to describe production. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YW171yVjpD7c59PtLT4zKz

szemyd

QA review — file ordering across the daily cutover

Tested this branch (commit d5e278f) against the #804 API preview
(dream-faster/unravel-router). Full QA record: dream-faster/unravel-router#807.

The ordering fix works

Three runs of get_metrics(..., show_progress=True) over 2026-08-11 … 2026-08-16
(6 daily files, the code path where downloads complete out of order):

run1: rows=144 sorted=True dups=0 min=2026-08-11 00:00 max=2026-08-16 23:00 order_hash=-0x30ce9b23759f6012
run2: rows=144 sorted=True dups=0 min=2026-08-11 00:00 max=2026-08-16 23:00 order_hash=-0x30ce9b23759f6012
run3: rows=144 sorted=True dups=0 min=2026-08-11 00:00 max=2026-08-16 23:00 order_hash=-0x30ce9b23759f6012

Identical across runs including row order, sorted, zero duplicates. The (year, month, day)
sort key with file_info.get("day", 0) correctly places a monthly file ahead of the same
month's daily files. NotRequired[int] on FileInfo.day matches what the API actually
returns — verified that monthly entries omit the key entirely rather than sending null.

Caveat on the evidence: I could not run this over the backfilled days (2026-08-01…09),
because they raise before ordering is reachable — see below. So the fix is verified on daily
files generally, not on the ones the August backfill produced.

What to fix

1 — TIMESTAMP_COL doubles as a query-param name and a data column name. In
src/aperiodic/config.py, TIMESTAMP_COL = "timestamp" is used both as the parameter naming
the timestamp type (timestamp=exchange) and as the column to sort by in
_get_files_from_bucket_async:

if backend.has_column(combined, TIMESTAMP_COL):      # "timestamp"
    combined = backend.from_epoch_ms(combined, TIMESTAMP_COL)
    ...
    combined = backend.sort_by(combined, TIMESTAMP_COL)

The parquets' time column is time, not timestamp. Against well-formed objects this guard is
always false, so the trailing sort never runs — meaning concatenation order is the only
thing establishing row order, which is exactly why this PR's sort key carries all the weight.
Worth making that explicit rather than incidental.

Against malformed objects it's worse. The August backfill (in
aperiodic-io/queries-historical) wrote daily files carrying a real timestamp column holding
the VARCHAR "exchange" — a hive partition value. The guard passes, and from_epoch_ms tries
to parse "exchange" as epoch ms:

InvalidOperationError: conversion from `str` to `datetime[ms]` failed in column 'timestamp' … ["exchange", "exchange", "exchange"]

Suggested: sort on time, or resolve the time column by presence rather than via a constant
shared with the query-param name, and separate the two uses so a partition value can never be
read as data.

2 — schema drift across files fails illegibly. Those same backfilled files have 17 columns
where monthly and natively-written daily files have 13, so any range spanning them dies in
concat:

ShapeError: unable to append to a DataFrame of width 17 with a DataFrame of width 13

Nothing in the message says which file, which columns, or that the objects are inconsistent.
Consider how="diagonal_relaxed", or an explicit pre-concat schema check naming the offending
file and the differing columns.

Both issues are pre-existing, not introduced here — so I'm commenting rather than requesting
changes, and this is fine to merge on its own merits. But (1) is close enough to this PR's
subject that fixing it here would be natural, and until the upstream data is repaired,
get_metrics raises for every August 2026 query regardless of this fix:

backfilled days only (08-01..08-09)   → InvalidOperationError
mixed range (08-01..08-16)            → ShapeError
native days only (08-11..08-16)       → OK, 144 rows
pre-cutover monthly (2025-03..04)     → OK, 1464 rows

Repro

import datetime as dt, aperiodic, os
K = os.environ["APERIODIC_API_KEY"]
common = dict(metric="l1_price", timestamp="exchange", interval="1h",
              exchange="binance-futures", symbol="perpetual-BTC-USDT:USDT")
aperiodic.get_metrics(K, **common, start_date=dt.date(2026,8,1), end_date=dt.date(2026,8,16))

Needs CF_ACCESS_CLIENT_ID / CF_ACCESS_CLIENT_SECRET for the preview — get_headers already
forwards them, which was a nice surprise.

Two problems the cutover QA surfaced, both pre-existing. `TIMESTAMP_COL = "timestamp"` served two unrelated roles: the `timestamp=` query parameter naming the timestamp *type*, and the column to filter and sort by. The parquets carry `time`, so the guard was always false and the trailing filter and sort never ran at all — concatenation order was the only thing establishing row order, and a range was never trimmed to its exact bounds. The same collision turned a schema defect into a crash. When an object did carry a `timestamp` column — as the August 2026 backfill's did, holding the partition value "exchange" — the client fed it to `from_epoch_ms` and raised trying to read a string as a datetime. Splits the constant into TIMESTAMP_PARAM and TIME_COLUMN, and filters and sorts on `time`, which is already a timestamp and needs no epoch conversion. Mismatched column sets across downloaded files now raise with the offending file and the differing columns named, instead of `ShapeError: unable to append to a DataFrame of width 17 with a DataFrame of width 13` — which is true but reads as a client bug rather than a bad object. `filter_datetime_range` takes an explicit column in both backends, defaulting to the previous "datetime", and both gain `column_names`. Found by the QA in dream-faster/unravel-router#807.

szemyd · 2026-08-18T13:35:50Z

QA summary — SDK verified against real R2 data

Re-ran the monthly→daily cutover QA end to end against the branch preview worker and live R2 buckets. The SDK passed every check, including the mixed backfilled/natively-written range that previously blew up with ShapeError.

Build under test: 74141dab22aec854c81dc4de11fd5300b5f39385 (force-reinstalled from this branch). Confirmed aperiodic.config.TIME_COLUMN == 'time' and that AperiodicDataError is exported — i.e. the trailing sort now targets the real column, and schema drift is hardened.

Dataset: l2_imbalance / l2_liquidity on okx-perps, 1h, perpetual-BTC-USDT:USDT. That group's repair backfill (queries-historical#283 + overwrite re-slice) had completed, and I verified at the object level first: all 17 August day objects are column-identical to their monthly source — same names, order and types — read with hive_partitioning = false and confirmed independently by pulling the bytes and opening them with polars.read_parquet(BytesIO(...)). No timestamp/exchange/symbol/year anywhere.

Results

Check	Result	Evidence
a · Repaired range reads	PASS	`2026-08-01…08-05` → `rows=120 width=25 sorted=True dups=0`, `tmin=2026-08-01 00:00 tmax=2026-08-05 23:00`. No `InvalidOperationError`
b · Mixed old/new range	PASS	`2026-08-08…08-13` spanning backfilled days 8–10 and natively-written 11–13 → `rows=96 width=25 sorted=True dups=0`. 6 files concatenated, uniform width, no `ShapeError`
c · Ordering deterministic	PASS	3 consecutive runs with `show_progress=True`: all three identical — `{rows:96, width:25, sorted:True, dups:0, tmin:2026-08-08 00:00, tmax:2026-08-11 23:00}`
d · Range trimmed	PASS	`2026-07-15…08-02` (July arrives as a whole-month file) → 456 rows; rows before `07-15` = 0, rows on/after `08-03` = 0
e · Pre-cutover unchanged	PASS	`2025-03-01…04-30` → 1464 rows (61×24), width 25, sorted
f · Schema drift reported well	PASS	See below
g · Both metrics in group	PASS	`l2_liquidity` repeat of (a) → `rows=120 width=17`; of (b) → `rows=96 width=17`, sorted, zero dups

On (f) — the negative control

binance-futures / l1 had already finished repairing by the time I reached this check (all day objects width 13, LastModified 2026-08-18 12:24:38Z), so it was no longer usable as a drift source. I substituted l1_imbalance / okx-perps, still mid-repair with days 01–09 carrying the four extra columns (width 15 vs monthly 11, LastModified 2026-08-14 22:24Z) and days 11–17 clean.

Requesting 2026-08-08…08-12 across that boundary raised the right thing — a typed error naming the file and the columns, not a bare ShapeError:

AperiodicDataError: Inconsistent columns across downloaded files: 2026-08-11 has 11 column(s),
expected 15. Unexpected: none; missing: ['exchange', 'symbol', 'timestamp', 'year'].
This is a defect in the stored object, not in the query.

One nit worth a follow-up (not blocking): _require_uniform_schema uses the first file chronologically as the baseline. When the leading days are the broken wide ones — which is exactly the shape a partial repair leaves behind — the message accuses the healthy file. Here it reports the clean 2026-08-11 as "missing" exchange, symbol, timestamp, year and asserts "this is a defect in the stored object", pointing at the only correct object in the set. Type, date and column list are all right; only the polarity is backwards. Picking the modal schema as the baseline, or naming both sides, would read better.

Not covered

One symbol per metric on a single interval (1h); no sweep across the 646 okx-perps symbols or other intervals.
pandas output path untested — all runs used the default polars backend.
Unrelated to this PR, but context for anyone reading the row counts: okx-perps / l2 day objects for 12–17 August are 0-row (schema correct, so they concat cleanly and nothing raises). Verified l2-group-specific — l1_price/l1_imbalance return a full 24 rows on those same days. That's a producer-side data gap, not an SDK issue, and it's why the (b) range reports 96 rows rather than 144.

#17) ## What From **2026-08-01** the API returns one parquet per day rather than one per month, and `FileInfo` gains an optional `day` (dream-faster/unravel-router#804, now merged). The CLI discarded that field. `FileInfo` had no `Day`, so `processor.go` named every output from year and month alone: ```go filename := fmt.Sprintf("%d-%02d.parquet", f.Year, f.Month) ``` Every daily file in a month therefore resolved to the same path: ``` 2026-08-01 ─┐ 2026-08-02 ─┼─> outputDir/2026-08.parquet ... ─┘ ``` Up to `--max-concurrent` goroutines then raced to `os.Create` (which truncates) and `io.Copy` into that one path. A 31-day August left **a single file** holding one arbitrary day's data, or an interleave of several — and the CLI exited 0 reporting `Successfully downloaded 31 files`, listing the same name 31 times. Silent, so a user has no signal that 30 days are missing. ## Changes **Naming.** Daily files become `2026-08-01.parquet`; monthly files keep `2026-08.parquet`. Both parts are zero-padded so a directory listing sorts by date. That is a choice about local filenames and deliberately *not* a mirror of the R2 key, which writes the month unpadded — there's a comment saying so, because "aligning" them would be a natural-looking mistake. **`Day` is `*int`, not `int`.** An absent field stays distinct from a literal `0`. With a plain int, a stray `0` from the API would read as "monthly" and put a daily file back on the monthly file's path — the exact failure being fixed. **A collision guard.** Names are resolved and checked for duplicates *before* anything is written; a colliding set fails the run with both indices and the shared name. This, rather than the naming, is what makes the bug class unreachable — any future response shape that maps two files to one path now errors instead of letting the last writer win. **Retry-loop handle leak (same function, same symptom).** `downloadToFile` used `defer` inside its retry loop for both the response body and the destination file. A deferred call runs at function return, not at the end of an iteration, so every attempt's handles stayed open for the whole retry sequence — a failed attempt could still be flushing into a path a later attempt had already truncated. One attempt is now `downloadOnce`, which closes what it opens and reports whether the failure is worth retrying (transport and HTTP errors yes, a local filesystem error no, preserving the previous behaviour of not retrying `os.Create` failures). **README.** The Output section described files as "named by year and month". It now covers both granularities, the changeover date, and shows the two filename shapes. ## Tests 10 new cases in `daily_files_test.go`, covering naming, chronological sortability, collision rejection, optional-`day` JSON decoding, and end-to-end CLI runs against a stub API. The load-bearing one is `TestCLI_DailyFilesWriteOnePerDay`: three days in, three files out, each asserted to hold **its own** day's bytes — distinct names alone would still pass if every file held the same content. Pre-fix it produces one file. Reverting the naming fix fails four tests, including both end-to-end cases. `gofmt` and `go vet` clean. The five integration tests that require `APERIODIC_API_KEY` fail identically on `main` without it (`requireAPIKey` calls `t.Fatal`, not `t.Skip`); CI supplies the secret. ## Note `gofmt` also realigned the pre-existing `Exchange` const block in `types.go` while formatting the new field — three lines of whitespace, unrelated to the behaviour change. ## Related - dream-faster/unravel-router#804 — the API change that adds `day` (merged) - aperiodic-io/client#30 — the same fix for the Python SDK, where the symptom is misordered rows rather than lost files - dream-faster/unravel-router#805 — docs and changelog for the changeover --- _Generated by [Claude Code](https://claude.ai/code/session_01YW171yVjpD7c59PtLT4zKz)_ Co-authored-by: Claude <noreply@anthropic.com>

claude added 2 commits August 13, 2026 19:49

szemyd commented Aug 17, 2026

View reviewed changes

szemyd merged commit 86fe0ba into main Aug 18, 2026
21 checks passed

szemyd deleted the claude/daily-parquet-files-api-pkgmad branch August 18, 2026 13:36

szemyd mentioned this pull request Aug 18, 2026

feature(Files): Download one file per day for the daily parquet layout aperiodic-io/cli#17

Merged

Navigation Menu

Uh oh!

feature(Files): Order downloaded parquet files by day - #30

feature(Files): Order downloaded parquet files by day#30
szemyd merged 3 commits into
mainfrom
claude/daily-parquet-files-api-pkgmad

szemyd commented Aug 13, 2026 •

edited

Loading

Uh oh!

szemyd left a comment

Uh oh!

szemyd commented Aug 18, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

szemyd commented Aug 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

The bug this fixes

The fix

Public API

Tests

Why this is in scope

Related

Uh oh!

szemyd left a comment

Choose a reason for hiding this comment

QA review — file ordering across the daily cutover

The ordering fix works

What to fix

Repro

Uh oh!

szemyd commented Aug 18, 2026

QA summary — SDK verified against real R2 data

Results

On (f) — the negative control

Not covered

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

szemyd commented Aug 13, 2026 •

edited

Loading