Add end-to-end Parquet VARIANT support

[xiangfu0]

Summary

Add an end-to-end VARIANT user journey for Apache Pinot:

• define a single-value VARIANT dimension retained in Pinot's raw forward index
• ingest top-level, non-repeated Apache Parquet VARIANT(1) values from unshredded and shredded files
• optionally materialize frequently queried scalar paths during ingestion while retaining the original value
• expose Spark-style parse, extraction, type, existence, null, and JSON-rendering functions
• carry Variant through Pinot schemas, segments, query plans, responses, DDL, Java/JDBC clients, and both query engines
• provide a packaged batch quickstart plus integration and mixed-version compatibility coverage

Raw Variant values are deliberately opaque to operations that require equality, hashing, or ordering. Queries must extract a typed scalar before comparing, grouping, joining, sorting, using set operations, or applying general aggregates. COUNT(raw_variant) remains supported.

The persisted format, null semantics, ownership boundaries, and rollout contract are documented in pinot-spi/VARIANT_DESIGN.md.

End-to-end flow

Define a table column with "dataType": "VARIANT", storage null handling enabled, dictionary disabled, and a raw forward index. The native Parquet reader reconstructs VARIANT(1) metadata/value pairs—including supported shredded layouts—into Pinot's versioned PVAR envelope.

Frequently queried fields can be materialized with ingestion transforms while the full Variant remains available:

SET enableNullHandling=true;

SELECT eventType, COUNT(*)
FROM variantEvents
GROUP BY eventType
ORDER BY eventType;

SELECT
  eventId,
  variant_get(payload, '$.user.id', 'STRING') AS userId,
  variant_get(payload, '$.amount', 'DOUBLE') AS amount
FROM variantEvents
WHERE eventType = 'checkout'
ORDER BY eventId;

SELECT eventId, variantToJson(payload)
FROM variantEvents
ORDER BY eventId;

Supported scope

• top-level, non-repeated Parquet VARIANT(1) columns
• single-value Pinot VARIANT dimensions without dictionaries
• raw forward-index retention and optional scalar materialization
• storage null handling enabled by schema or table configuration
• query-time Variant functions with SET enableNullHandling=true
• single-stage and multi-stage query execution

Nested/repeated Variant columns, streaming ingestion, and quoted path keys are outside the initial scope.

A JSON index is intentionally not accepted on a VARIANT column. The existing JSON index expects Pinot JSON text and its query rewrite semantics; treating the binary PVAR envelope as JSON would be incorrect. Indexing Variant paths should be introduced separately with an explicit physical/index contract. Materialized scalar columns are the supported indexed path for this PR.

The automatic Parquet reader selection preserves existing Avro-metadata precedence. Files containing Avro metadata must explicitly select the native Parquet reader to ingest Variant.

Production safety

• centralizes PVAR framing and validation in VariantEnvelope and freezes version-1 bytes with golden tests
• assigns protobuf wire value 24 to Variant while preserving UUID/UUID_ARRAY at 22/23
• rejects unsupported schemas, table/index configurations, query null-handling modes, and raw Variant operations
• validates Parquet decimal precision, scale, bounds, hostile exponents, malformed rows, and buffer ownership
• handles array-backed, direct, sliced, and read-only Parquet buffers without changing source positions or limits
• compiles constant paths/types once and reuses cursors in ingestion, single-stage, and multi-stage execution
• distinguishes SQL null, encoded Variant null, the string "null", missing paths, and conversion failures across query and client response formats
• keeps Parquet column/schema dependencies isolated to the input plugin

Rolling upgrades

Variant remains an explicit full-fleet activation feature: upgrade controllers, brokers, servers, clients, and external ingestion jobs before registering a Variant schema, and do not roll back while Variant tables are active.

Compatibility coverage verifies:

• all-old components reject the unknown wire type deterministically
• a new broker with old servers fails with upgrade guidance
• a mixed old/new server fleet rejects the whole query without returning partial rows
• an all-new fleet succeeds
• an old broker with upgraded servers remains queryable for the verified path
• capability checks, readiness retries, expected-error matching, and rollback are bounded and deterministic

Quickstart

./mvnw clean install -DskipTests -Pbin-dist -Pbuild-shaded-jar
build/bin/quick-start-variant-batch.sh

The quickstart registers variantEvents, ingests the committed five-row Parquet fixture, uploads the segment, and runs materialized-field, nested-extraction, JSON-rendering, and null-semantics queries. Its resources live under pinot-tools/src/main/resources/examples/batch/variantEvents.

Verification

• 21 VariantTypeTest scenarios across both query engines in the full integration-test reactor
• all-old, new/old, mixed-server, all-new, old-broker/new-server, and rollback compatibility flows
• post-rebase 63-module, 956-goal JDK 25 reactor covering the full Variant integration path
• spotless, checkstyle, license formatting, and license checks across all 16 affected modules
• clean four-commit history on current upstream/master
• git diff --check

Performance benchmark

A 2,000,000-row workstation benchmark compared identical logical events represented as JSON text and Parquet VARIANT(1), then ingested into raw Pinot columns with ZSTD compression. A third JSON scenario added Pinot's JSON index and is reported separately because it is an index-assisted comparison, not a format-only comparison. The reproducible harness is preserved in the benchmark snapshot.

Ingestion and storage

Representation	Median build	Throughput	Source Parquet	Pinot segment	Result
JSON raw	32.942 s	60,760 rows/s	285.52 MiB	294.16 MiB	Baseline
VARIANT raw	14.315 s	139,714 rows/s	279.18 MiB	298.89 MiB	2.29× faster ingestion
JSON + JSON index	194.873 s	10,263 rows/s	285.52 MiB	1,253.70 MiB	Index-assisted

Compared with raw JSON, the VARIANT source was 2.22% smaller and its Pinot segment was 1.61% larger. Adding the JSON index made the JSON segment 4.26× the size of the raw JSON segment.

Query latency

Each value is the median of four round-level p50 values. Every round used two warmups and 10 measured sequential executions.

Workload	JSON p50	VARIANT p50	Result
Materialized eventType count (control)	0.326 ms	0.349 ms	Effectively neutral
Nested numeric sum	2,674.153 ms	592.766 ms	VARIANT 4.49× faster
Nested tier filter	2,704.314 ms	681.794 ms	VARIANT 3.98× faster
Four-path numeric aggregate	9,037.254 ms	1,051.095 ms	VARIANT 8.60× faster
Selective country filter	JSON index: 6.553 ms	VARIANT raw scan: 645.687 ms	JSON index 98.69× faster; not format-only

The raw comparison used four Parquet files/Pinot segments and four ingestion rounds with alternating JSON/VARIANT build order. Source generation was excluded from ingestion timing, and every query result was checked against independently generated ground truth.

These are paired, sequential, warm-cache, single-stage measurements from an Apple M2 Max workstation with 32 GB RAM, macOS 26.5.2, and Temurin 25 on AC power. They are directional workstation evidence, not distributed production-cluster throughput: network transfer, multi-stage execution, concurrent clients, distributed scheduling, and production hardware were outside the run.

Review areas

This PR changes public SPI and query-wire behavior. The main ownership reviews are:

• SPI, schema, and query-wire compatibility
• Parquet and input-format ingestion
• single-stage and multi-stage query semantics
• Java, JDBC, Arrow, JSON, and gRPC response behavior
• rolling-upgrade activation and rollback

[codecov-commenter]

Codecov Report

❌ Patch coverage is 38.17583% with 1315 lines in your changes missing coverage. Please review.
✅ Project coverage is 39.03%. Comparing base (e51b4e4) to head (8b38248).
⚠️ Report is 2 commits behind head on master.

Files with missing lines	Patch %	Lines
...va/org/apache/pinot/common/utils/VariantUtils.java	29.34%	452 Missing and 56 partials ⚠️
...ransform/function/VariantGetTransformFunction.java	0.00%	106 Missing ⚠️
...uery/runtime/operator/operands/VariantOperand.java	2.27%	85 Missing and 1 partial ⚠️
.../function/ParseJsonToVariantTransformFunction.java	0.00%	64 Missing ⚠️
...anner/validation/VariantTypeValidationVisitor.java	32.89%	43 Missing and 8 partials ⚠️
...ansform/function/BaseVariantTransformFunction.java	0.00%	49 Missing ⚠️
...va/org/apache/pinot/spi/utils/VariantEnvelope.java	61.53%	25 Missing and 10 partials ⚠️
...n/inputformat/parquet/ParquetVariantConverter.java	81.75%	14 Missing and 13 partials ⚠️
...not/common/evaluator/InbuiltFunctionEvaluator.java	66.17%	11 Missing and 12 partials ⚠️
...ime/operator/operands/LiteralParseJsonOperand.java	4.16%	22 Missing and 1 partial ⚠️
... and 51 more

❗ There is a different number of reports uploaded between BASE (e51b4e4) and HEAD (8b38248). Click for more details.

HEAD has 16 uploads less than BASE

Flag	BASE (e51b4e4)	HEAD (8b38248)
unittests1	1	0
unittests	2	1
java-25	5	1
temurin	5	1
integration	3	0
integration1	1	0
integration2	1	0
custom-integration1	1	0

Additional details and impacted files

@@              Coverage Diff              @@
##             master   #19101       +/-   ##
=============================================
- Coverage     66.65%   39.03%   -27.63%     
- Complexity     1423     1494       +71     
=============================================
  Files          3443     3457       +14     
  Lines        218632   220691     +2059     
  Branches      34793    35163      +370     
=============================================
- Hits         145726    86136    -59590     
- Misses        61192   126615    +65423     
+ Partials      11714     7940     -3774     

Flag	Coverage Δ
custom-integration1	?
integration	?
integration1	?
integration2	?
java-25	39.03% <38.17%> (-27.63%)	⬇️
temurin	39.03% <38.17%> (-27.63%)	⬇️
unittests	39.03% <38.17%> (-27.63%)	⬇️
unittests1	?
unittests2	39.03% <38.17%> (+0.12%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[Copilot]

Pull request overview

This PR adds end-to-end support for the VARIANT logical type across Pinot ingestion (notably Parquet VARIANT(1)), query planning/execution (single-stage + multi-stage), response encoders, and Java/JDBC client surfaces, while explicitly rejecting unsupported raw-VARIANT operations (e.g., ordering/grouping/join keys) with actionable errors.

Changes:

• Introduces VARIANT as a first-class Pinot data type, including schema/DDL handling, wire/proto support, and null-sentinel semantics.
• Adds Parquet VARIANT(1) ingestion support via the native Parquet reader and includes a packaged “variant batch” quickstart with resources and CI coverage.
• Adds query-time VARIANT functions (parse/get/existence/type/json rendering) and enforces operator-level restrictions where raw VARIANT lacks equality/hash/ordering semantics.

Reviewed changes

Copilot reviewed 174 out of 175 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
pom.xml	Adds managed dependency for org.apache.parquet:parquet-variant.
pinot-tools/src/test/java/org/apache/pinot/tools/admin/command/QuickStartTest.java	Maps new quickstart command strings to VariantQuickStart.
pinot-tools/src/main/resources/examples/batch/variantEvents/*	Adds VARIANT batch quickstart schema/table config, ingestion spec, and documentation.
pinot-tools/pom.xml	Adds packaged quick-start-variant-batch program entry.
pinot-sql-ddl/src/main/, src/test/	Adds DDL compile/reverse/roundtrip support for VARIANT and validates default emission rules.
pinot-spi/src/main/, src/test/	Adds PinotDataType.VARIANT conversion/ingestion mapping and schema validation updates.
pinot-segment-spi/src/main/**	Handles VARIANT default-null sentinel persisted as empty string in segment metadata.
pinot-segment-local/src/main/, src/test/	Adds VARIANT validation/sanitization rules; disables min/max, ordering, dictionary creation where unsupported; adds targeted tests.
pinot-query-planner/src/main/, src/test/	Adds VARIANT type mapping (Calcite/plan nodes), plan validation, constant-folding guardrails, and serde coverage.
pinot-query-runtime/src/main/, src/test/	Enforces VARIANT restrictions at runtime operators (sort/window/set/group-by/etc.) and adds operator tests.
pinot-core/src/main/, src/test/	Enforces VARIANT restrictions in single-stage operators/aggregations/distinct/order-by and adds tests.
pinot-common/src/main/, src/test/	Adds VARIANT to encoders, datablock equality, proto enum, function canonicalization, and introduces scalar VariantFunctions.
pinot-clients/pinot-jdbc-client/src/main/, src/test/	Treats VARIANT as canonical JSON text (VARCHAR) while preserving encoded-variant-null vs SQL-null behavior.
pinot-clients/pinot-java-client/src/main/, src/test/	Preserves legacy getString() null behavior while distinguishing VARIANT nulls.
pinot-plugins/pinot-input-format/pinot-parquet/src/main/**	Enhances Parquet reader selection/metadata helpers and initializes native reader with schema-bound converters for VARIANT.
pinot-controller/src/main/, src/test/	Ensures stored-vs-compiled schema default comparisons handle VARIANT byte[] defaults by content.
pinot-compatibility-verifier/**	Adds file-contains operation and VARIANT mixed-version suite coverage including old-broker/new-servers phase.
.github/workflows/scripts/.pinot_quickstart.sh	Adds CI validation for the packaged VARIANT batch quickstart.

[Copilot]

Pull request overview

Copilot reviewed 174 out of 175 changed files in this pull request and generated no new comments.

Suppressed comments (1)

pinot-query-runtime/src/main/java/org/apache/pinot/query/runtime/operator/SortOperator.java:81

• The validation uses supportsOrdering() but the error text always mentions raw VARIANT. supportsOrdering() is also false for other non-orderable logical types (e.g. OBJECT / array types), so this can surface a misleading VARIANT-specific message when the real unsupported type is something else.

Consider tailoring the message based on the actual column type (keep the VARIANT guidance when type == VARIANT, otherwise emit a generic "ORDER BY does not support values of type X" message).

[xiangfu0]

Reviewed the full VARIANT PR (checked out the branch; ran domain-focused passes over SPI/envelope, Parquet ingestion, the VariantUtils codec, single-stage + multi-stage enforcement, schema/table validation, and wire/client/compat). Overall this is very carefully engineered — the PVAR envelope, the four null-state handling, and the defense-in-depth opacity guards all check out, and I found no data-corruption or crash bugs in the core paths.

Two items I'd suggest fixing before merge (inline: an OBJECT-comparison regression and a partial-upsert validation gap), plus a few low-severity polish notes.

Verified-correct highlights: proto appends VARIANT=24 with no renumbering and ColumnDataType is name-serialized (no ordinal()-indexed tables exist repo-wide, so the mid-enum insertion is safe); Parquet buffer ownership / shredded-vs-unshredded selection / reader precedence / reinit safety; codec bounds + per-row cursor reset + thread-confinement; and the full set of schema/table restrictions (dictionary / secondary indexes / PK / partition / sorted / star-tree (default and explicit) / metrics-agg / upsert-comparison all rejected, min/max never published, never sorted).

[xiangfu0]

Behavior regression + misleading message (Medium). This guard requires both operands to be UNKNOWN || supportsOrdering(), but ColumnDataType.supportsOrdering() is false for OBJECT, arrays, and MAP — not just VARIANT. The class Javadoc just above ("if either side is null or OBJECT, we best-effort cast data into the other side's data type") documents OBJECT as an intentionally-supported case, and OBJECT = OBJECT previously took the equal-types branch and ran. After this change such a comparison throws "Raw VARIANT values do not support comparison" — both a narrowing of the existing contract and a wrong error for non-VARIANT types.

Suggest gating specifically on the VARIANT case (or restoring the OBJECT/UNKNOWN best-effort path) and making the message name the actual offending type. Same pattern applies to the In guard at line 120 (supportsEquality(), also false for OBJECT/arrays).

[xiangfu0]

Fixed in xiangfu0#236. The comparison/IN/DISTINCT-FROM guards now reject only raw VARIANT (!= ColumnDataType.VARIANT) instead of any non-orderable/non-equatable type, so OBJECT/array/MAP comparisons keep their prior behavior. Applied to both engines: FilterOperand (Predicate, In), TransformOperandFactory (distinct-from), and the single-stage BinaryOperatorTransformFunction / InTransformFunction, which had the same guard. Added FilterOperandTest.testNonVariantOpaqueTypesAreNotRejectedAsVariant.

[xiangfu0]

Validation gap (Medium). This VARIANT OVERWRITE-only check only runs for columns explicitly present in partialUpsertStrategies. A VARIANT column not listed here falls back to upsertConfig.getDefaultPartialUpsertStrategy() at merge time — see PartialUpsertColumnarMerger#merge (_column2Mergers.getOrDefault(column, _defaultColumnValueMerger)), where the default merger is built from getDefaultPartialUpsertStrategy() and is user-settable to INCREMENT/APPEND/UNION/IGNORE.

Failing scenario: a partial-upsert table with an unlisted VARIANT dimension and defaultPartialUpsertStrategy: INCREMENT. Validation passes, then at runtime IncrementMerger is applied to the byte envelope → failure/corruption; IGNORE would silently retain stale variant values. This violates the documented "partial upsert only OVERWRITE" contract.

Suggest: when a VARIANT column exists under active partial upsert, also require the effective default strategy to be OVERWRITE. (Also note the custom partialUpsertMergerClass path applies no VARIANT guard — likely acceptable as user code, but worth a doc note.)

[xiangfu0]

Fixed in xiangfu0#236. validatePartialUpsertStrategies now validates the effective strategy for every VARIANT column up front: it uses the explicit entry if listed, otherwise defaultPartialUpsertStrategy, and requires OVERWRITE. It also rejects a custom partialUpsertMergerClass on VARIANT columns (which cannot be validated against the OVERWRITE-only contract). Added 3 tests covering unlisted-column default, valid default, and custom-merger rejection.

[xiangfu0]

Misleading message (Low). supportsOrdering() is false for VARIANT and arrays/OBJECT/MAP/STRUCT/LIST, so this rejects e.g. ORDER BY array_col with a VARIANT-specific message. The rejection is correct; only the text is wrong for non-VARIANT types. Same hard-coded-VARIANT-message-on-a-generic-supportsOrdering()-gate appears in SortedMailboxReceiveOperator, OrderByComparatorFactory, and VariantTypeValidationVisitor#visitSort/visitWindow. Consider naming the actual unsupported type.

[xiangfu0]

Fixed in xiangfu0#236. The ORDER BY / sort / window / set-op / aggregate rejections now name the actual unsupported type (e.g. "ORDER BY does not support OBJECT values") and keep the raw-VARIANT wording plus variantGet guidance only for VARIANT. Applied to SortOperator, SortedMailboxReceiveOperator, OrderByComparatorFactory, and VariantTypeValidationVisitor.

[xiangfu0]

Consistency (Low, not exploitable). validateAggregateInputs iterates only the agg-call operands; it never checks AggregateNode.getGroupKeys(), so the planner gate doesn't reject GROUP BY raw_variant. This is not a hole — MultistageGroupByExecutor and the single-stage GroupBy operators reject it at runtime via supportsEquality()/supportsHashing(). But since every other operation (sort, set-op, join, window partition) is validated in this visitor, adding the group-key check here would make the planner the consistent authoritative gate.

[xiangfu0]

Fixed in xiangfu0#236. validateAggregateInputs(AggregateNode, ...) now also validates AggregateNode.getGroupKeys() (equality+hashing), so the planner gate rejects GROUP BY raw_variant consistently with sort/join/set-op/window (runtime already rejected it). Added testRejectsRawVariantGroupByKey.

[xiangfu0]

Follow-up on my review: all four findings are now fixed and merged into this PR (head is now 8b38248f).

Severity	Finding	Fix
Medium	Comparison / IN / IS DISTINCT FROM guards rejected any non-orderable/non-equatable type (OBJECT, arrays, MAP) with a VARIANT-specific error, narrowing existing behavior	Guards now reject only raw VARIANT, in both engines: FilterOperand, TransformOperandFactory, BinaryOperatorTransformFunction, InTransformFunction
Medium	VARIANT OVERWRITE-only partial-upsert check only covered listed columns; unlisted columns use defaultPartialUpsertStrategy (and a custom partialUpsertMergerClass), which could silently apply INCREMENT/APPEND/UNION/IGNORE to a PVAR envelope	TableConfigUtils now validates the effective strategy for every VARIANT column and rejects a custom merger on VARIANT columns
Low	Opacity errors said "raw VARIANT" for any non-orderable type	Messages now name the actual type; raw-VARIANT wording + variantGet guidance kept only for VARIANT (VariantTypeValidationVisitor, SortOperator, SortedMailboxReceiveOperator, OrderByComparatorFactory)
Low	Planner did not validate GROUP BY keys (runtime did)	VariantTypeValidationVisitor.validateAggregateInputs now validates AggregateNode group keys

Regression tests were added for each. spotless/checkstyle/license are clean on all four affected modules, and every affected unit-test class passes locally on JDK 25 (VariantTableConfigValidationTest, FilterOperandTest, VariantTypeValidationVisitorTest, and the pinot-core comparison/IN/ORDER BY suites).

On the current CI run, the Linter and all VARIANT-exercising tests pass. The three red checks are pre-existing flakes unrelated to these changes and to VARIANT: FilteredAggregationsTest.testFilterVsCase (Unit Set 1, same as before this PR) and testGeneratedQueries/testConcurrentQueries random-query flakes in the two integration sets (no VariantType* failures). A re-run should clear them.