Expose common segment metadata as built-in virtual columns

[xiangfu0]

Summary

Adds five built-in virtual columns that expose common segment metadata to queries, alongside the existing $docId / $hostName / $segmentName / $partitionId.

Column	Type	Value
$creationTime	TIMESTAMP	segment creation time
$startTime	TIMESTAMP	segment time-range start, normalized from the time column's own unit
$endTime	TIMESTAMP	segment time-range end
$totalDocs	INT	documents stored in the segment (documents indexed so far while CONSUMING)
$crc	LONG	segment CRC

SELECT $segmentName, $crc, $totalDocs, $creationTime
FROM myTable GROUP BY 1, 2, 3, 4

This makes a number of operational questions answerable directly in SQL — spotting replicas of a segment whose CRC has diverged, finding segments created before a rollout, or looking at per-segment document skew — without walking the segment metadata REST API.

Like the existing built-ins, these are excluded from SELECT * and only materialize when named explicitly.

Behavior

Each column is a constant single-value column within a segment. The value is read from SegmentMetadata when the column is built rather than baked into the field spec, so a mutable segment — which rebuilds its virtual data sources on every access — picks up metadata that was not yet available when it was created. Within one build the value is resolved exactly once, so the dictionary, the column metadata and the null value vector can never disagree.

Metadata that genuinely does not exist yet reads as SQL NULL, not a sentinel. A CONSUMING segment has no time range and no CRC until it is committed, and a table without a time column never has a time range. This required teaching the virtual column path to carry a null value vector: VirtualColumnIndexContainer now serves StandardIndexes.nullValueVector(), and VirtualColumnProvider gained a buildNullValueVector hook that defaults to "no nulls", so existing providers are unaffected. Without it the placeholder in the forward index would be indistinguishable from a real value once null handling is enabled — e.g. MIN($startTime) returning the epoch on a consuming segment, and IS NOT NULL matching every row.

The time columns are TIMESTAMP rather than LONG so the unit is carried by the type rather than by the column name, and results render as readable timestamps.

$totalDocs counts the documents physically stored in the segment, so for an upsert table it also includes documents that have been replaced and are no longer returned by queries. On a hybrid table the broker's time boundary can likewise hide rows that are present in the segment.

Supporting changes

• New BuiltInVirtualColumnDefinitions in pinot-spi is the single source of each column's name, data type, and single-value/multi-value shape. The broker side (TableCache#addBuiltInVirtualColumns) and the server side (VirtualColumnProviderFactory#addBuiltInVirtualColumnsToSegmentSchema) both build their field specs from it, so the two can no longer disagree on a type. VirtualColumnProviderFactory resolves providers through a map whose key set is checked against the definitions in a static initializer, so a column added without a provider fails at class load rather than aborting every segment load on every server.

• The constant-value machinery moves out of DefaultNullValueVirtualColumnProvider into BaseConstantValueVirtualColumnProvider; the former becomes a thin subclass. Its fully-qualified name is stored in field specs and resolved reflectively, so it keeps its name and package. It also gained a type check that names the offending column and provider instead of throwing a bare ClassCastException from inside segment loading.

• Fixes a pre-existing bug in SchemaInfo — reviewable independently of the feature; happy to split it out if preferred, which computed getDimensionFieldSpecs().size() - 3 with a comment naming three virtual columns. That has been off by one since $partitionId was added, and these five columns would have made GET /schemas/info over-report user dimension counts by six in the controller UI. It now excludes built-in virtual columns by name, with a test pinning the invariant.

Note for reviewers: EXPLAIN assertions shift

Calcite plans reference fields positionally, and all $-prefixed columns sort first in a table's row type — so every added built-in virtual column shifts every table ordinal in every EXPLAIN assertion. Five new columns means +5 in NullHandlingIntegrationTest, MultiStageEngineExplainIntegrationTest and OfflineClusterIntegrationTest (and MultiNodesOfflineClusterIntegrationTest, which inherits). The $partitionId PR paid the same tax. Similarly the aggregate metadata API reports one entry per column of a loaded segment, so its expected count goes from 83 to 88.

SPI surface

pinot-spi gains one public class (BuiltInVirtualColumnDefinitions) and five public constants, and public BUILT_IN_VIRTUAL_COLUMNS grows from 4 to 9 entries. The addition is source- and binary-compatible, but that set is a filter predicate rather than an inert constant: its only non-test consumer outside this diff is SegmentMetadataImpl#addPhysicalColumns, which drops any name in the set from a segment's physical column list. Downstream/plugin code branching on the set inherits the same change.

Backward compatibility / rolling upgrade

The change is additive: no wire format, segment format, or ZK format change, and SELECT * is unaffected in both engines because $-prefixed columns are excluded from star expansion.

Broker-side and server-side schemas are updated in the same commit but roll out independently. Upgrade servers before brokers: an upgraded broker advertises the new columns, so a query naming one during the upgrade window returns partial results plus per-segment exceptions from servers still on the old build. No existing query is affected — only queries that opt into the new columns.

Two REST responses change content:

• GET /tables/{table}/metadata gains 5 entries per table in columnLengthMap / columnCardinalityMap (the expected count in OfflineClusterIntegrationTest moves from 83 to 88). Loaded segments carry virtual columns in their column metadata, as they already did for the existing four.
• GET /schemas/info numDimensionFields decreases by one for every existing schema. The old size() - 3 had been over-reporting by one ever since $partitionId was added; the new value is the true user-dimension count. Operators tracking that field will see a one-off shift.

SchemaUtils.validate now rejects a user column named after a built-in virtual column. Such a schema is already broken today (the column is excluded from SELECT * and, once the name is reserved, shadowed by the virtual column), but this is a new validation failure for anyone who has one.

Null semantics and segment pruning

A column whose metadata is unavailable stores a placeholder and reports every document as null. It deliberately publishes no ColumnMetadata min/max: segment pruners read min/max without consulting the null value vector, so publishing the placeholder would let an all-null column reorder and prune segments as if it held a real extreme value — e.g. a CONSUMING segment's epoch $creationTime sorting first in ORDER BY $creationTime ASC LIMIT n and pruning away the committed segments that hold the answer. Both SelectionQuerySegmentPruner and ColumnValueSegmentPruner keep a segment that reports no min/max.

Cost

Each column adds a constant dictionary, readers and column metadata per immutable segment — roughly 1–2 KB per segment across the five, so on the order of 35 MB on a server holding 20k segments. There is no config gate, matching the existing four built-ins.

Testing

• SegmentMetadataVirtualColumnProviderTest — values, column metadata, and the NULL path for both a missing SegmentMetadata and a real consuming-segment SegmentMetadataImpl, across the several representations of an unset creation time, plus a negative test for the value type check.
• BuiltInVirtualColumnDefinitionsTest — pins the definitions against the declared names, and that field specs are never shared between schemas.
• LoaderTest iterates the full built-in set and asserts both the value path and the NULL path on a really-loaded segment.
• MutableSegmentImplTest pins the CONSUMING-segment shape.
• OfflineClusterIntegrationTest covers both query engines: data schema, per-segment $totalDocs summing to the table row count, millisecond normalization per segment checked against DaysSinceEpoch * 86_400_000, filtering by $crc, and enableNullHandling=true seeing no nulls where all metadata is available.
• BaseClusterIntegrationTestSet (via HybridClusterIntegrationTest) asserts results on the CONSUMING path instead of only checking that nothing throws.
• Full clean test across pinot-spi → pinot-common → pinot-segment-local → pinot-core → pinot-query-planner, plus every integration test class touched above. All green locally.

Docs

The virtual-columns page in pinot-docs needs a matching update for these five columns, the TIMESTAMP types, and the NULL-on-unavailable behavior.

Labels

feature, release-notes

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 100.00%. Comparing base (3ffb614) to head (834a096).

Additional details and impacted files

@@              Coverage Diff               @@
##             master    #19179       +/-   ##
==============================================
+ Coverage     66.63%   100.00%   +33.36%     
+ Complexity     1423         6     -1417     
==============================================
  Files          3443         3     -3440     
  Lines        218663         6   -218657     
  Branches      34801         0    -34801     
==============================================
- Hits         145705         6   -145699     
+ Misses        61230         0    -61230     
+ Partials      11728         0    -11728     

Flag	Coverage Δ
integration	100.00% <ø> (ø)
integration1	100.00% <ø> (ø)
integration2	0.00% <ø> (ø)
java-25	100.00% <ø> (+33.36%)	⬆️
lane-a	100.00% <ø> (ø)
lane-b	0.00% <ø> (ø)
temurin	100.00% <ø> (+33.36%)	⬆️
unittests	?
unittests1	?
unittests2	?

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[Copilot]

Pull request overview

This PR expands Pinot’s built-in virtual column surface by adding five new $-prefixed, segment-scoped virtual columns that expose common SegmentMetadata via SQL, while ensuring correct NULL semantics when metadata is genuinely unavailable (e.g., CONSUMING segments).

Changes:

• Introduces $creationTime, $startTime, $endTime, $totalDocs, and $crc as built-in virtual columns (excluded from SELECT *, materialized only when explicitly selected).
• Centralizes built-in virtual column definitions (name + type + SV/MV shape) in pinot-spi and reuses them on both broker/controller and server paths to prevent type drift.
• Extends the virtual column implementation to support null value vectors, and updates unit/integration tests and explain-plan ordinals accordingly.

Reviewed changes

Copilot reviewed 27 out of 27 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
pinot-spi/src/test/java/org/apache/pinot/spi/data/SchemaInfoTest.java	Adds a regression test ensuring built-in virtual columns are excluded from user dimension counts.
pinot-spi/src/test/java/org/apache/pinot/spi/data/BuiltInVirtualColumnDefinitionsTest.java	New tests pinning definitions ↔ declared-name set equality, $ prefixing, idempotency, and non-overwrite behavior.
pinot-spi/src/main/java/org/apache/pinot/spi/utils/CommonConstants.java	Adds constants for the five new built-in virtual columns and expands BUILT_IN_VIRTUAL_COLUMNS.
pinot-spi/src/main/java/org/apache/pinot/spi/data/SchemaInfo.java	Fixes dimension counting to exclude built-in virtual columns by name (instead of -3).
pinot-spi/src/main/java/org/apache/pinot/spi/data/BuiltInVirtualColumnDefinitions.java	New SPI “single source of truth” for built-in virtual column field spec shape and schema injection helper.
pinot-segment-local/src/test/java/org/apache/pinot/segment/local/segment/virtualcolumn/SegmentMetadataVirtualColumnProviderTest.java	New unit tests covering value extraction, metadata consistency, and NULL vector behavior for segment-metadata columns.
pinot-segment-local/src/test/java/org/apache/pinot/segment/local/segment/index/loader/LoaderTest.java	Extends loader assertions to cover the full built-in set and validates NULL-vector behavior for time-range columns.
pinot-segment-local/src/test/java/org/apache/pinot/segment/local/indexsegment/mutable/MutableSegmentImplTest.java	Adds explicit assertions for CONSUMING-like mutable segment behavior for the new metadata columns and adjusts comparison exclusions.
pinot-segment-local/src/test/java/org/apache/pinot/segment/local/indexsegment/mutable/MutableSegmentImplRawMVTest.java	Mirrors mutable/immutable comparison exclusions for segment-level metadata columns in raw MV tests.
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/virtualcolumn/VirtualColumnProviderFactory.java	Refactors built-in virtual column schema injection to be definition-driven and adds provider coverage validation.
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/virtualcolumn/VirtualColumnProvider.java	Adds an overridable buildNullValueVector() hook and wires it into virtual column index container construction.
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/virtualcolumn/VirtualColumnIndexContainer.java	Extends the container to serve StandardIndexes.nullValueVector() and closes it when present.
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/virtualcolumn/SegmentTotalDocsVirtualColumnProvider.java	New provider implementing $totalDocs as a constant derived from context doc count.
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/virtualcolumn/SegmentStartTimeVirtualColumnProvider.java	New provider implementing $startTime from SegmentMetadata#getTimeInterval() with NULL when absent.
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/virtualcolumn/SegmentEndTimeVirtualColumnProvider.java	New provider implementing $endTime from SegmentMetadata#getTimeInterval() with NULL when absent.
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/virtualcolumn/SegmentCreationTimeVirtualColumnProvider.java	New provider implementing $creationTime with unset creation-time detection.
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/virtualcolumn/SegmentCrcVirtualColumnProvider.java	New provider implementing $crc with unset CRC detection.
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/virtualcolumn/BaseSegmentMetadataVirtualColumnProvider.java	New base provider adding “value or placeholder + all-null vector” behavior for unavailable segment metadata.
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/virtualcolumn/BaseConstantValueVirtualColumnProvider.java	New reusable base for constant-value virtual columns with consistent dictionary/forward/inverted/metadata building and type checks.
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/column/DefaultNullValueVirtualColumnProvider.java	Refactors existing provider to subclass the new constant-value base while preserving FQCN stability.
pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/OfflineClusterIntegrationTest.java	Adds end-to-end validation of the new columns (types, normalization, filtering, null-handling semantics) and updates explain/metadata counts.
pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/NullHandlingIntegrationTest.java	Updates explain-plan ordinal assertions (+5 shift) to account for the new $-prefixed columns.
pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/MultiStageEngineExplainIntegrationTest.java	Updates explain-plan ordinal assertions (+5 shift) for MSE plans.
pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/BaseClusterIntegrationTestSet.java	Adds hybrid-table assertions for $totalDocs, $crc, and $creationTime across potentially CONSUMING segments.
pinot-controller/src/test/java/org/apache/pinot/controller/helix/TableCacheTest.java	Updates expected schema and column-name maps to include the new built-ins.
pinot-controller/src/test/java/org/apache/pinot/controller/api/resources/PinotQueryResourceStaticValidationTest.java	Makes built-in virtual column count assertions size-driven rather than hard-coded.
pinot-common/src/main/java/org/apache/pinot/common/config/provider/TableCache.java	Switches broker/controller built-in injection to use BuiltInVirtualColumnDefinitions.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.