Support range index on ingestion-aggregated no-dictionary columns

[raghavyadav01]

Problem

Configuring a BitSliced (version 2) range index on an ingestion-aggregated metric column causes segments to fail to commit.

Ingestion-time metrics aggregation (aggregationConfigs, or the legacy aggregateMetrics flag) forces its aggregated metric columns to be no-dictionary and skips min/max tracking while the consuming segment is being built — the values mutate in place, so a running min/max would be meaningless (MutableSegmentImpl deliberately skips it). At segment commit time min/max are therefore null, and the BitSliced range index creator reads them for a single-value no-dictionary column:

// RangeIndexType.createIndexCreator
return new BitSlicedRangeIndexCreator(context.getIndexDir(), fieldSpec,
    context.getMinValue(), context.getMaxValue());   // min/max are null here

For INT/LONG columns this throws a NullPointerException at segment build time (((Number) minValue).longValue()), so the affected real-time segments can never commit.

Fix

Rather than reject the combination, recover the value domain at build time:

1. MutableNoDictColumnStatistics — when the mutable segment reports null min/max, compute them with a single scan of the sealed forward index. Scoped to single-value INT/LONG, the only stored types whose BitSliced range index reads min/max (FLOAT/DOUBLE use the full floating-point ordinal domain and never dereference min/max). The recovered values also flow into the committed segment metadata via the existing metadata path, so aggregated columns now get accurate segment-level min/max (a pruning improvement).

2. RangeIndexHandler — the index-reload path recomputes min/max from the forward index for segments that were committed before this change (whose metadata min/max are still null), via a new min/max override on IndexCreationContext.Builder. The recovered domain is written into the range index header (which the reader uses for the subtract-min); for such legacy segments the reader still reads a null metadata max and falls back to Long.MAX_VALUE — results stay correct (the RangeBitmap domain is self-contained), only segment-level max pruning is weaker until the segment is rebuilt. Segments sealed after this change carry proper metadata min/max and do not hit that path.

3. Reverts the interim table-config rejection in RangeIndexType.validate() and the loud-fail guard in BitSlicedRangeIndexCreator (both introduced on this branch, never released).

Notes / discussion

• The seal-path recovery lives in the statistics layer, which has no visibility into whether a range index is configured, so the one-time scan runs for every aggregated INT/LONG no-dictionary column at commit (comparable in cost to the isSorted() scan already performed there). This is the "overhead we pay to add the range index"; it doubles as a pruning benefit since these columns previously had no segment min/max. Happy to gate it further if preferred.

Testing

• MutableNoDictColumnStatisticsTest — min/max recovered for single-value INT/LONG when metadata is null; FLOAT/DOUBLE and multi-value columns intentionally stay null.
• RealtimeSegmentConverterTest — end-to-end: an ingestion-aggregated no-dictionary LONG metric column with a range index seals successfully, its column metadata carries the recovered min/max, and a BitSliced range index is present on the column.
• IndexCombinationValidationTest — the previously-rejected combinations (both aggregation paths, REALTIME) now validate; version-1, offline, and non-aggregated cases still validate.
• BitSlicedIndexCreatorTest — reverted to the pre-guard state.

Release Notes

A version-2 (BitSliced) range index on an ingestion-aggregated no-dictionary column is now supported: its min/max value domain is recovered from the sealed forward index at segment build time instead of failing to commit.

[codecov-commenter]

Codecov Report

❌ Patch coverage is 57.53425% with 31 lines in your changes missing coverage. Please review.
✅ Project coverage is 65.67%. Comparing base (504efc4) to head (542f75a).
⚠️ Report is 29 commits behind head on master.

Files with missing lines	Patch %	Lines
.../index/loader/invertedindex/RangeIndexHandler.java	22.58%	22 Missing and 2 partials ⚠️
...inot/segment/spi/creator/IndexCreationContext.java	25.00%	4 Missing and 2 partials ⚠️
...converter/stats/MutableNoDictColumnStatistics.java	97.05%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##             master   #19111      +/-   ##
============================================
- Coverage     65.70%   65.67%   -0.04%     
  Complexity     1423     1423              
============================================
  Files          3439     3439              
  Lines        218064   218133      +69     
  Branches      34679    34693      +14     
============================================
- Hits         143289   143249      -40     
- Misses        63226    63323      +97     
- Partials      11549    11561      +12     

Flag	Coverage Δ
custom-integration1	100.00% <ø> (ø)
integration	100.00% <ø> (ø)
integration1	100.00% <ø> (ø)
integration2	0.00% <ø> (ø)
java-25	65.67% <57.53%> (-0.04%)	⬇️
temurin	65.67% <57.53%> (-0.04%)	⬇️
unittests	65.66% <57.53%> (-0.04%)	⬇️
unittests1	56.99% <12.32%> (-0.05%)	⬇️
unittests2	38.00% <54.79%> (-0.02%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[Jackie-Jiang]

This is not the correct fix. We should allow building range index, and scan the column to figure out min/max value when they are unavailable.

[raghavyadav01]

Thanks @Jackie-Jiang .
Will the following approach be ok ?

I am thinking to compute min/max on demand in MutableNoDictColumnStatistics — when the mutable segment reports null (aggregated columns skip min/max tracking since the value mutates), scan the sealed forward index once (same pass isSorted() already does), before the BitSliced creator is constructed.

This adds one extra O(numDocs) scan per such column at seal time. Will this be OK?

[Jackie-Jiang]

I am thinking to compute min/max on demand in MutableNoDictColumnStatistics — when the mutable segment reports null (aggregated columns skip min/max tracking since the value mutates), scan the sealed forward index once (same pass isSorted() already does), before the BitSliced creator is constructed.

This adds one extra O(numDocs) scan per such column at seal time. Will this be OK?

Yes. That is the overhead we need to pay in order to add range index. We should also check how RangeIndexHandler handles null min/max value.