[UUID 8/8] UUID integration tests, benchmarks and docs - #18876
Open
xiangfu0 wants to merge 7 commits into
Open
Conversation
This was referenced Jun 29, 2026
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## master #18876 +/- ##
============================================
- Coverage 66.65% 66.61% -0.05%
Complexity 1423 1423
============================================
Files 3443 3447 +4
Lines 218632 218995 +363
Branches 34793 34888 +95
============================================
+ Hits 145726 145874 +148
- Misses 61192 61382 +190
- Partials 11714 11739 +25
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
xiangfu0
force-pushed
the
uuid-split/08-it-bench-docs
branch
from
June 29, 2026 12:05
93272e3 to
4856ac5
Compare
xiangfu0
commented
Jun 29, 2026
xiangfu0
left a comment
xiangfu0
left a comment
Contributor
Author
There was a problem hiding this comment.
Found critical mixed-version compatibility issues inherited in this stack head; see inline comments.
| // does not contain this enum constant. This is safe because DataSchema serialization uses enum names (not ordinals) | ||
| // via ColumnDataType.name() / ColumnDataType.valueOf(). If ordinal-based serialization is ever added for | ||
| // ColumnDataType, UUID must be moved to the end of the enum (as was done for FieldSpec.DataType.UUID). | ||
| UUID(BYTES, null) { |
Contributor
Author
There was a problem hiding this comment.
This makes UUID a new broker/server wire-visible ColumnDataType. DataSchema.toBytes() emits the enum name, and older peers still throw in parseColumnDataType(...) once they see UUID, so rolling upgrades and rollback are unsafe as soon as UUID-typed results are in flight. Please keep the wire representation on an existing type until all peers are upgraded, or add an explicit mixed-version compatibility path plus coverage.
| // does not know UUID = 22 / UUID_ARRAY = 23 will fail planning with UnknownEnumValueException when receiving a plan | ||
| // that includes a UUID literal. Avoid issuing UUID queries until all brokers and servers are upgraded. See the | ||
| // matching note on DataSchema.toBytes and ProtoExpressionToRexExpression#convertColumnDataType. | ||
| UUID = 22; |
Contributor
Author
There was a problem hiding this comment.
This introduces new UUID / UUID_ARRAY proto enum values with no compatibility path for older MSQ peers. Older brokers/servers decode them as UNRECOGNIZED and throw in convertColumnDataType(...), so a UUID literal can break mixed-version planning before execution even starts. Please encode UUID literals using an existing wire type until the cluster is homogeneous, or add version-gated dual-read/dual-write behavior with mixed-version tests.
xiangfu0
force-pushed
the
uuid-split/08-it-bench-docs
branch
7 times, most recently
from
July 5, 2026 19:24
1eb0c13 to
ce33dae
Compare
xiangfu0
force-pushed
the
uuid-split/08-it-bench-docs
branch
12 times, most recently
from
July 13, 2026 08:02
5284a05 to
e4a51de
Compare
xiangfu0
force-pushed
the
uuid-split/08-it-bench-docs
branch
10 times, most recently
from
July 21, 2026 08:10
b977430 to
1d67c2c
Compare
xiangfu0
force-pushed
the
uuid-split/08-it-bench-docs
branch
3 times, most recently
from
July 28, 2026 08:02
6dcfae2 to
2d38a93
Compare
28 tasks
xiangfu0
force-pushed
the
uuid-split/08-it-bench-docs
branch
7 times, most recently
from
August 8, 2026 09:36
51eb86e to
e848ec1
Compare
xiangfu0
force-pushed
the
uuid-split/08-it-bench-docs
branch
from
August 9, 2026 09:15
e848ec1 to
669aea4
Compare
Part 5/8 of splitting apache#18140 (logical UUID type). Rebased onto latest master; stacked on uuid-split/04-sse-predicates-cast. Downstream references use the UuidKey class merged in apache#18869.
… path getConvertedKey had `case UUID` falling through to BYTES, returning the raw byte[]. The other reduce path converts group keys via ColumnDataType#convert, and UUID is the one type whose converted form is not its stored bytes -- it yields a java.util.UUID. PredicateRowMatcher casts that directly (see apache#18872), so the byte[] made GROUP BY ... HAVING over a UUID column fail with "ClassCastException: class [B cannot be cast to class java.util.UUID". Delegating to columnDataType.convert(...) keeps the two paths identical by construction rather than by duplicated knowledge. Covered by a new UuidAggregationTest integration test rather than a unit test. The unit-level BaseQueriesTest harness cannot reach this code, which is why it was uncovered; a query-level test goes through the real broker reduce. Verified by reverting the fix: testGroupByUuidColumnWithHaving and testGroupByUuidColumnWithHavingReturningFinalResult both fail with the ClassCastException and both pass with it. The test also covers GROUP BY key rendering, DISTINCT (BytesDistinctTable no longer hard-codes hex) and DISTINCTCOUNT / DISTINCTCOUNTHLL / DISTINCTCOUNTBITMAP over a UUID column.
Both callers key on UuidKey already (NoDictionary{Single,Multi}ColumnGroupKey
Generator, via UuidKey.fromBytes), so UuidKey.fromObject accepted five input
types where exactly one is ever passed, and ran an instanceof chain per row in
the group-by loop.
Casting directly matches the sibling maps -- DoubleToIdMap casts to double --
and keeps the input type deterministic, which is what was asked for on the
equivalent PredicateRowMatcher branch in apache#18872.
…cal string The previous version rendered each UUID as its 36-char canonical string so that DISTINCTCOUNTHLL(uuidCol) would equal DISTINCTCOUNTHLL(CAST(uuidCol AS STRING)). No other logical type provides that guarantee: the scan path switches on the stored type, so TIMESTAMP offers its raw millis and BOOLEAN its int, and neither matches a CAST to STRING. The UUID rendering was inventing a cross-type equivalence at the cost of a String allocation per row in the aggregation loop. UUID now hashes its stored 16 bytes. Verified byte[] is content-hashed rather than identity-hashed by both HyperLogLog (clearspring MurmurHash) and UltraLogLogUtils.OBJECT_FUNNEL (putBytes). A minimal guard is still needed at each site, because unlike LONG or INT the stored BYTES type is not a scalar case in this family: the scan path has no `case BYTES`, and the dictionary path reads BYTES as serialized sketch state. - AggregationFunctionUtils: the three UUID blocks are gone; the BYTES guard now excludes UUID so it falls through to the scalar path, which offers dictionary.get(i) -- the stored byte[] -- exactly as the scan path does. - DistinctCountBitmap hashes Arrays.hashCode(bytes). - DistinctCountThetaSketch cannot take scalar bytes (BYTES there means "serialized sketch"), so it surfaces the stored hex rendering instead. Replaces testUuidDistinctCountHllMatchesStringDistinctCountHll, which asserted the invariant being dropped, with one pinning the new behaviour.
Part 6/8 of splitting apache#18140 (logical UUID type). Rebased onto latest master; stacked on uuid-split/05-agg-groupby-distinct. Downstream references use the UuidKey class merged in apache#18869.
Part 7/8 of splitting apache#18140 (logical UUID type). - UuidPartitionFunction: hashes the 16-byte UUID form via Murmur2, matching what an external producer keyed on raw UUID bytes computes - PartitionerFactory / TableConfigPartitioner: thread the column's logical DataType through so UUID columns render canonically instead of as bare hex - UUID_ARRAY entries for the array scalar functions The UUID scalar functions and multi-stage UDF wrappers that were previously part of this layer now live in their own PR (apache#19091) so they can be reviewed and merged in parallel.
Part 8/8 of splitting apache#18140 (logical UUID type). Rebased onto latest master; stacked on uuid-split/07-udfs-partitioning. Downstream references use the UuidKey class merged in apache#18869.
xiangfu0
force-pushed
the
uuid-split/08-it-bench-docs
branch
from
August 9, 2026 09:20
669aea4 to
4b1626f
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Parent tracking issue: #16619
What
End-to-end verification and documentation.
Changes
UuidTypeTest,UuidTypeRealtimeTest,UuidUpsertRealtimeTest(extendCustomDataQueryClusterIntegrationTest)BenchmarkUuidGroupingAndLookup,BenchmarkUuidQueryExecutionREADMEusage / limitations / migrationEnables
Offline + realtime + upsert integration coverage and performance benchmarks. The top of the stack — its cumulative diff equals the original #18140.
Depends on
#18874 (benchmarks import multi-stage runtime classes). #18869 and #18871 have already merged.
About this PR / how to review
This is part 8 of 8 splitting #18140 (first-class logical
UUIDtype) into layered PRs, as requested there.The split is enabled by the v1 design:
DataType.UUIDhas stored typeBYTES, so most paths handle it automatically; each PR adds explicit UUID semantics to one subsystem. Head branch lives onxiangfu0/pinot.This PR is stacked on #18875 (branch
uuid-split/07-udfs-partitioning). Because GitHub PRs to apache must base onmaster, the Files-changed tab is cumulative — it now includes layers 4–8 (layers 1–3 have merged) until the remaining PRs below it merge. Review the commit titled[UUID 8/8] UUID integration tests, benchmarks and docs— that is this layer's change. Each parent merge shrinks this diff after a rebase.Full stack (merge bottom → top)
[UUID 1/8] Add logical UUID type foundation (pinot-spi) #18869 — [UUID 1/8] logical UUID type foundation (pinot-spi)merged[UUID 2/8] UUID ingest and segment storage #18870 — [UUID 2/8] UUID ingest and segment storagemerged[UUID 3/8] UUID result rendering (DataSchema, Arrow/JSON encoders) #18871 — [UUID 3/8] UUID result rendering (DataSchema, Arrow/JSON encoders)mergedWith #18869 and #18871 landed, PRs 4/7 (#18872, #18875) and 5/6 (#18873, #18874) can be reviewed in parallel; PR 8 (#18876) needs #18874.
Full feature description, v1 design contract, scope exclusions, and benchmark numbers: #18140.