perf(query-engine): ClickStack-style skip indexes + token-aware log search

[Makisuo]

What & why

Applies the schema/query optimizations from ClickHouse's Making ClickStack 5x faster to Maple's telemetry read path. The blog's biggest wins map directly onto gaps Maple had:

ClickStack idea	Maple before	This PR
Text index on the log body	logs.Body had no index — search was a full-partition ILIKE scan behind a max_block_size=512 OOM guardrail	tokenbf_v1 on lower(Body) + a semantics-safe hasToken pre-filter
Attribute skip indexes	logs had zero attribute-map indexes (traces had 4)	four bloom_filter indexes on logs
"Items" indexes for map equality	attribute equality couldn't prune on the exact key=value pair	idx_*_attr_items bloom over arrayMap((k,v)->concat(k,'=',v),…) on traces
Benchmark-driven	no committed benchmark suite	bench suite compiled from the real DSL + A/B + regression gate

Changes

Indexes (packages/domain/src/tinybird/datasources.ts)

• logs: idx_body_tokens (tokenbf_v1(32768,3,0) on lower(Body)) + blooms on mapKeys/mapValues of LogAttributes and ResourceAttributes.
• traces: idx_span_attr_items + idx_resource_attr_items over the concatenated key=value "Items" expression.
• Index expressions are defined once in index-exprs.ts and imported by the datasource, the migration, and the query predicate so they can't drift (a mismatch silently disables the index).

Query engine

• body-search.ts: bodySearchConditions AND-s hasToken(lower(Body), t) for interior, separator-bounded, ASCII tokens only, keeping the ILIKE as the final predicate. A single word or edge token yields no pre-filter → today's behavior. Whole-token semantics ≠ substring, so this is the only sound derivation; result sets are unchanged.
• buildAttrFilterCondition(…, { itemsIndex }): equality filters emit has(items,'k=v') (OR-ed across HTTP semconv aliases) AND-ed with the exact map equality. Enabled only on the raw traces table (not trace_detail_spans or metrics, which lack the index).
• New builder primitives CH.hasToken / CH.has.

Migration & rollout

• 0005_body_and_attr_indexes.ts adds every index with ADD INDEX IF NOT EXISTS and no MATERIALIZE (a materialize over billion-row parts exceeds the Worker subrequest budget; indexes cover new parts and backfill via the 30-day TTL). SCHEMA_VERSION bumps 4→5.

Benchmark harness (apps/api/scripts)

• bench suite compiles 13 representative cases from the real DSL; --table-map from=to rewrites FROM/JOIN for A/B against shadow tables; bench compare --max-regression-pct N exits non-zero as a CI gate. See BENCH.md.

Testing

• Unit: builder 72, domain 174, query-engine 727, apps/api bench 11 — all green. Repo typecheck 24/24.
• Compiled-SQL assertions cover the token pre-filter (interior-only, single-word fallback, non-ASCII), the items pre-filter (equality-only, alias OR, negated/contains/empty exclusions), and the trace-detail vs raw-traces path split.

⚠️ Pre-merge gates (need a live warehouse — couldn't run locally)

1. Tinybird deploy check — tb build + branch tb --cloud deploy --check to confirm managed Tinybird accepts tokenbf_v1 and the arrayMap expression index as an in-place ADD INDEX (no FORWARD_QUERY/rebuild). Fallback if rejected: keep those indexes BYO-migration-only.
2. Live EXPLAIN indexes=1 granule-drop check on a dev ClickHouse (the local clickhouse binary is macOS-Gatekeeper-blocked in the dev box used here).
3. Rollout sequencing — the SCHEMA_VERSION bump un-readies BYO-ClickHouse ingest until applySchema stamps v5; deploy API + applySchema + Rust gateway together.

Follow-up (separate PR)

The time-bucketed sort-key / read_in_order spike (Phase 3) — the bench suite --table-map A/B harness in this PR is built to drive it against a dedicated ClickHouse.

🤖 Generated with Claude Code

---

[pullfrog]

Your LLM provider API key was rejected. Rotate the key in your provider dashboard, then update the matching GitHub Actions secret.

Update repo secret → · Model settings → · Setup docs → · Ask in Discord →

  ^{｜ ⚠️ this action is pinned to a commit SHA, which freezes the cleanup step — switch to @v0 or keep the SHA fresh with Dependabot ｜ Rerun failed job ➔ ｜ View workflow run ｜ via Pullfrog ｜ Using Claude Opus ｜ 𝕏}

[devin-ai-integration]

Devin Review found 1 potential issue.

[devin-ai-integration]

🚩 Log timeseries raw path does not use bodySearchConditions, but this is pre-existing

The bodySearchConditions integration was applied to logsCountQuery (packages/query-engine/src/ch/queries/logs.ts:360) and logsListQuery (line 455), but the timeseries query's raw path was not updated. This is NOT a regression: the timeseries raw path never had a search filter — canUseLogsAggregateInterior (line 81) returns false when opts.search is set, routing to the raw-only scan, but the raw timeseries scan itself doesn't filter by search term. The timeseries query counts all logs in the window for the volume chart, while search filtering is only applied to the list and count queries. If search-filtered timeseries is ever needed, bodySearchConditions should be added there too.

---

Was this helpful? React with 👍 or 👎 to provide feedback.

[github-actions]

Ingest Rust Test + Benchmark Results

Commit: 9cce7cd37143cde2916f561ee9152a2a3454c5d8

Load Benchmark — tinybird mode, median of 3 run(s) vs main

Metric	main (median)	PR (median)	Delta
Requests/sec	2238.69	2079.38	-7.1% worse
Rows/sec	22386.95	20793.82	-7.1% worse
p50 latency	27.72 ms	30.33 ms	+9.4% worse
p95 latency	32.65 ms	38.48 ms	+17.9% worse
p99 latency	39.37 ms	41.30 ms	+4.9% worse
Export catch-up	0.026 s	0.026 s	+0.4% worse
Max RSS	101.18 MiB	101.32 MiB	+0.1% worse
Failures	0	0	same

Same code path on both sides (same LOAD_TEST_INGEST_MODE), so the delta column is meaningful. Numbers come from ubuntu-latest, which is noisy — treat single-digit-percent deltas as noise.

PR load benchmark JSON (per-iteration)

[
  {
    "ingest_mode": "tinybird",
    "requests": 2000,
    "successes": 2000,
    "failures": 0,
    "rows_sent": 20000,
    "rows_exported": 20000,
    "imports": 29,
    "duration_seconds": 1.053399643,
    "export_catchup_seconds": 0.025876718,
    "request_rps": 1898.6146552168518,
    "row_rps": 18986.146552168517,
    "p50_ms": 31.411,
    "p95_ms": 45.195,
    "p99_ms": 48.864,
    "max_rss_mb": 102.3828125,
    "max_cpu_percent": 71.4,
    "avg_cpu_percent": 58.03333333333334
  },
  {
    "ingest_mode": "tinybird",
    "requests": 2000,
    "successes": 2000,
    "failures": 0,
    "rows_sent": 20000,
    "rows_exported": 20000,
    "imports": 24,
    "duration_seconds": 0.961824271,
    "export_catchup_seconds": 0.025820273,
    "request_rps": 2079.3819206918306,
    "row_rps": 20793.81920691831,
    "p50_ms": 30.327,
    "p95_ms": 33.965,
    "p99_ms": 35.539,
    "max_rss_mb": 101.3203125,
    "max_cpu_percent": 76.7,
    "avg_cpu_percent": 46.650000000000006
  },
  {
    "ingest_mode": "tinybird",
    "requests": 2000,
    "successes": 2000,
    "failures": 0,
    "rows_sent": 20000,
    "rows_exported": 20000,
    "imports": 25,
    "duration_seconds": 0.93503533,
    "export_catchup_seconds": 0.026451309,
    "request_rps": 2138.9566103347133,
    "row_rps": 21389.56610334713,
    "p50_ms": 28.583,
    "p95_ms": 38.484,
    "p99_ms": 41.3,
    "max_rss_mb": 96.8828125,
    "max_cpu_percent": 76.7,
    "avg_cpu_percent": 55.0
  }
]

main load benchmark JSON (per-iteration)

[
  {
    "ingest_mode": "tinybird",
    "requests": 2000,
    "successes": 2000,
    "failures": 0,
    "rows_sent": 20000,
    "rows_exported": 20000,
    "imports": 28,
    "duration_seconds": 1.021057731,
    "export_catchup_seconds": 0.025281856,
    "request_rps": 1958.7531040397166,
    "row_rps": 19587.531040397167,
    "p50_ms": 31.613,
    "p95_ms": 38.266,
    "p99_ms": 44.953,
    "max_rss_mb": 104.703125,
    "max_cpu_percent": 70.6,
    "avg_cpu_percent": 58.0
  },
  {
    "ingest_mode": "tinybird",
    "requests": 2000,
    "successes": 2000,
    "failures": 0,
    "rows_sent": 20000,
    "rows_exported": 20000,
    "imports": 25,
    "duration_seconds": 0.893377714,
    "export_catchup_seconds": 0.025861955,
    "request_rps": 2238.6947521281018,
    "row_rps": 22386.947521281018,
    "p50_ms": 27.718,
    "p95_ms": 32.652,
    "p99_ms": 39.367,
    "max_rss_mb": 99.7421875,
    "max_cpu_percent": 78.5,
    "avg_cpu_percent": 55.9
  },
  {
    "ingest_mode": "tinybird",
    "requests": 2000,
    "successes": 2000,
    "failures": 0,
    "rows_sent": 20000,
    "rows_exported": 20000,
    "imports": 23,
    "duration_seconds": 0.886745797,
    "export_catchup_seconds": 0.025777986,
    "request_rps": 2255.437811790384,
    "row_rps": 22554.378117903838,
    "p50_ms": 27.496,
    "p95_ms": 29.855,
    "p99_ms": 38.386,
    "max_rss_mb": 101.18359375,
    "max_cpu_percent": 82.1,
    "avg_cpu_percent": 49.349999999999994
  }
]

WAL-acked microbench (cargo bench --bench ingest_bench)

   Compiling maple-ingest v0.1.0 (/home/runner/work/maple/maple/apps/ingest)
    Finished `bench` profile [optimized] target(s) in 41.01s
     Running benches/ingest_bench.rs (target/release/deps/ingest_bench-581d2100de893627)
Gnuplot not found, using plotters backend
test ingest_accept/logs_10_rows_wal_ack ... bench:      505164 ns/iter (+/- 30271)
test ingest_accept/traces_10_spans_wal_ack ... bench:      502560 ns/iter (+/- 30715)

cargo test

test telemetry::tests::metrics_summary_data_points_are_dropped ... ok
test telemetry::tests::metrics_emit_exactly_the_jsonpaths_declared_in_datasources_ts ... ok
test telemetry::tests::migrate_legacy_shard_relocates_frames_into_lanes ... ok
test telemetry::tests::pipeline_can_start_for_clickhouse_only_without_tinybird_credentials ... ok
test telemetry::tests::clickhouse_export_drops_passworded_non_https_endpoint_without_sending ... ok
test telemetry::tests::pipeline_e2e_exports_gzip_ndjson_to_fake_tinybird ... ok
test telemetry::tests::pipeline_e2e_exports_metrics_to_fake_tinybird ... ok
test telemetry::tests::sampling_keeps_errors_even_when_ratio_low ... ok
test telemetry::tests::scraper_contract::scraper_otlp_json_decodes_with_gateway_serde_and_encodes_to_rows ... ok
test telemetry::tests::signal_tag_round_trips_all_variants ... ok
test telemetry::tests::pipeline_e2e_exports_traces_to_fake_tinybird ... ok
test telemetry::tests::telemetry_signal_as_str_is_canonical_lowercase ... ok
test telemetry::tests::timestamp_has_nano_precision ... ok
test telemetry::tests::timestamps_match_clickhouse_datetime64_nine_format ... ok
test telemetry::tests::trace_encoder_matches_tinybird_row_shape ... ok
test telemetry::tests::traces_emit_exactly_the_jsonpaths_declared_in_datasources_ts ... ok
test telemetry::tests::wal_partial_drain_advances_cursor_without_truncating ... ok
test telemetry::tests::wal_round_trips_frame ... ok
test telemetry::tests::wal_truncates_after_full_drain_allowing_further_appends ... ok
test telemetry::tests::pipeline_exports_ready_org_to_clickhouse_without_tinybird_calls ... ok
test telemetry::tests::slow_clickhouse_lane_does_not_block_cosharded_tinybird_org ... ok
test telemetry::tests::clickhouse_breaker_sheds_after_threshold_failures ... ok

test result: ok. 36 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.78s

     Running unittests src/bin/load_test.rs (target/debug/deps/load_test-661a0aa1eb3f6d6d)

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

     Running unittests src/main.rs (target/debug/deps/maple_ingest-c33bf80c577edb95)

running 38 tests
test autumn::tests::allowed_only_no_balance_field ... ok
test autumn::tests::flat_hardcap_with_remaining_allows ... ok
test autumn::tests::flat_hardcap_depleted_blocks ... ok
test autumn::tests::flat_overage_allows ... ok
test autumn::tests::flat_unlimited_allows ... ok
test autumn::tests::flat_sub_one_gb_remaining_still_allows ... ok
test autumn::tests::nested_balance_object_depleted_blocks ... ok
test autumn::tests::nested_balance_object_with_remaining_allows ... ok
test autumn::tests::nested_overage_allows ... ok
test autumn::tests::null_balance_no_subscription_blocks ... ok
test autumn::tests::unrecognized_shape_returns_none ... ok
test tests::api_error_from_pipeline_maps_variants_to_status ... ok
test tests::api_error_kind_maps_status_to_stable_label ... ok
test tests::clickhouse_destination_is_terminal_in_dual_mode ... ok
test tests::clickhouse_destination_uses_native_pipeline_even_in_forward_mode ... ok
test tests::clickhouse_target_resolver_decrypts_current_schema_password ... ok
test tests::clickhouse_target_resolver_requires_current_schema ... ok
test tests::cloudflare_ndjson_payload_parses_multiple_records ... ok
test tests::clickhouse_target_resolver_rejects_password_over_http ... ok
test tests::cloudflare_log_record_maps_body_severity_and_attributes ... ok
test tests::cloudflare_timestamps_support_rfc3339_unix_and_unix_nano ... ok
test tests::cloudflare_validation_payload_is_detected ... ok
test tests::decrypt_aes256_gcm_matches_node_crypto_fixture ... ok
test tests::enrichment_overwrites_tenant_fields ... ok
test tests::extract_ingest_key_returns_sentinel_literal_unchanged ... ok
test tests::rejection_span_status_is_error_only_for_5xx ... ok
test tests::hash_is_deterministic ... ok
test tests::resolve_ingest_key_keeps_stale_schema_on_managed_native_path ... ok
test tests::resolve_connector_refreshes_routing_before_auth_cache_expires ... ok
test tests::resolve_ingest_key_returns_none_when_hash_missing ... ok
test tests::resolve_ingest_key_refreshes_routing_before_auth_cache_expires ... ok
test tests::resolve_ingest_key_returns_self_managed_false_when_no_settings_row ... ok
test tests::resolve_ingest_key_returns_self_managed_true_when_active_settings_row ... ok
test tests::sentinel_token_matches_only_exact_literal ... ok
test tests::tinybird_destination_keeps_forward_mode_on_forward_path ... ok
test autumn::tests::fails_open_on_transport_error ... ok
test tests::resolve_ingest_key_serves_last_known_routing_when_refresh_fails ... ok
test tests::forward_mode_switches_ready_org_to_clickhouse_without_forwarding_again ... ok

test result: ok. 38 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.23s

   Doc-tests maple_ingest

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s