Add UUID scalar functions and multi-stage UDF wrappers - #19091
Conversation
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## master #19091 +/- ##
============================================
+ Coverage 65.53% 65.62% +0.09%
Complexity 1423 1423
============================================
Files 3432 3450 +18
Lines 218151 218378 +227
Branches 34664 34670 +6
============================================
+ Hits 142964 143318 +354
+ Misses 63627 63498 -129
- Partials 11560 11562 +2
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
There was a problem hiding this comment.
Pull request overview
Adds UUID-focused scalar functions to pinot-common and registers corresponding multi-stage UDF wrappers in pinot-query-runtime, enabling UUID validation, conversion, accessors, and v4/v7 generation through Pinot’s function/UDF surfaces.
Changes:
- Introduces polymorphic
IS_UUID/TO_UUIDscalar functions (STRING or BYTES) via a shared dispatch base class. - Adds UUID conversion/accessor/generator scalar functions (
UUID_TO_BYTES,BYTES_TO_UUID,UUID_TO_STRING,UUID_VERSION,UUID_TIMESTAMP,UUID_V4,UUID_V7). - Adds multi-stage UDF wrapper classes and unit tests covering conversions, null handling, and v4/v7 properties.
Reviewed changes
Copilot reviewed 15 out of 15 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| pinot-query-runtime/src/main/java/org/apache/pinot/query/runtime/function/BytesToUuidUdf.java | Multi-stage UDF wrapper for BYTES_TO_UUID. |
| pinot-query-runtime/src/main/java/org/apache/pinot/query/runtime/function/IsUuidUdf.java | Multi-stage UDF wrapper for IS_UUID, including example signatures. |
| pinot-query-runtime/src/main/java/org/apache/pinot/query/runtime/function/ToUuidUdf.java | Multi-stage UDF wrapper for TO_UUID. |
| pinot-query-runtime/src/main/java/org/apache/pinot/query/runtime/function/UuidTimestampUdf.java | Multi-stage UDF wrapper for UUID_TIMESTAMP. |
| pinot-query-runtime/src/main/java/org/apache/pinot/query/runtime/function/UuidToBytesUdf.java | Multi-stage UDF wrapper for UUID_TO_BYTES. |
| pinot-query-runtime/src/main/java/org/apache/pinot/query/runtime/function/UuidToStringUdf.java | Multi-stage UDF wrapper for UUID_TO_STRING. |
| pinot-query-runtime/src/main/java/org/apache/pinot/query/runtime/function/UuidV4Udf.java | Multi-stage UDF wrapper for UUID_V4. |
| pinot-query-runtime/src/main/java/org/apache/pinot/query/runtime/function/UuidV7Udf.java | Multi-stage UDF wrapper for UUID_V7. |
| pinot-query-runtime/src/main/java/org/apache/pinot/query/runtime/function/UuidVersionUdf.java | Multi-stage UDF wrapper for UUID_VERSION. |
| pinot-common/src/main/java/org/apache/pinot/common/function/scalar/uuid/AbstractStringOrBytesUuidFunction.java | Shared dispatch helper for STRING/BYTES UUID scalar functions. |
| pinot-common/src/main/java/org/apache/pinot/common/function/scalar/uuid/IsUuidScalarFunction.java | Implements polymorphic IS_UUID. |
| pinot-common/src/main/java/org/apache/pinot/common/function/scalar/uuid/ToUuidScalarFunction.java | Implements polymorphic TO_UUID with Calcite UUID return typing. |
| pinot-common/src/main/java/org/apache/pinot/common/function/scalar/uuid/UuidConversionFunctions.java | Adds UUID conversion/accessor/generator scalar functions (incl. non-deterministic v4/v7). |
| pinot-common/src/test/java/org/apache/pinot/common/function/scalar/StringFunctionsTest.java | Adds mixed-case UUID round-trip coverage via toUUIDBytes/fromUUIDBytes. |
| pinot-common/src/test/java/org/apache/pinot/common/function/scalar/uuid/UuidConversionFunctionsTest.java | New unit tests for conversions, accessors, generators, and null handling. |
Comments suppressed due to low confidence (1)
pinot-common/src/main/java/org/apache/pinot/common/function/scalar/uuid/IsUuidScalarFunction.java:50
- Same null-handling issue as the STRING overload: marking the BYTES overload as
nullableParameters=trueforces evaluation on null inputs and returnsfalseinstead of propagating null. This also diverges from theIsUuidUdfexamples which expect null when null handling is enabled.
BYTES_FUNCTION_INFO =
new FunctionInfo(IsUuidScalarFunction.class.getMethod("isUuid", byte[].class), IsUuidScalarFunction.class,
true);
5846a27 to
e45cb81
Compare
Part 7/8 of splitting apache#18140 (logical UUID type). - UuidPartitionFunction: hashes the 16-byte UUID form via Murmur2, matching what an external producer keyed on raw UUID bytes computes - PartitionerFactory / TableConfigPartitioner: thread the column's logical DataType through so UUID columns render canonically instead of as bare hex - UUID_ARRAY entries for the array scalar functions The UUID scalar functions and multi-stage UDF wrappers that were previously part of this layer now live in their own PR (apache#19091) so they can be reviewed and merged in parallel.
e45cb81 to
fe30a2e
Compare
Now disjoint from #18875#18875 has been updated to drop these files, so the two PRs no longer overlap and can be reviewed and merged in either order:
Doc comments here were also converted to Re-verified on the current |
fe30a2e to
337463a
Compare
Adds the UUID scalar function surface on top of the logical UUID type: - IS_UUID / TO_UUID: polymorphic over STRING and BYTES, sharing dispatch via AbstractStringOrBytesUuidFunction - UUID_TO_BYTES, BYTES_TO_UUID, UUID_TO_STRING, UUID_VERSION, UUID_TIMESTAMP - UUID_V4 / UUID_V7 generators, marked isDeterministic = false so the broker's CompileTimeFunctionsInvoker does not fold one value across every row - Matching multi-stage Udf wrappers so the functions are registered with the MSE UDF registry Depends only on UuidUtils (apache#18869) and ColumnDataType.UUID (apache#18871), both already on master. Split out of apache#18875 so it can be reviewed independently of the UUID partitioning work.
337463a to
102c23e
Compare
Part 7/8 of splitting apache#18140 (logical UUID type). - UuidPartitionFunction: hashes the 16-byte UUID form via Murmur2, matching what an external producer keyed on raw UUID bytes computes - PartitionerFactory / TableConfigPartitioner: thread the column's logical DataType through so UUID columns render canonically instead of as bare hex - UUID_ARRAY entries for the array scalar functions The UUID scalar functions and multi-stage UDF wrappers that were previously part of this layer now live in their own PR (apache#19091) so they can be reviewed and merged in parallel.
|
Documentation follow-up: pinot-contrib/pinot-docs#952 |
Part 7/8 of splitting apache#18140 (logical UUID type). - UuidPartitionFunction: hashes the 16-byte UUID form via Murmur2, matching what an external producer keyed on raw UUID bytes computes - PartitionerFactory / TableConfigPartitioner: thread the column's logical DataType through so UUID columns render canonically instead of as bare hex - UUID_ARRAY entries for the array scalar functions The UUID scalar functions and multi-stage UDF wrappers that were previously part of this layer now live in their own PR (apache#19091) so they can be reviewed and merged in parallel.
Part 7/8 of splitting apache#18140 (logical UUID type). - UuidPartitionFunction: hashes the 16-byte UUID form via Murmur2, matching what an external producer keyed on raw UUID bytes computes - PartitionerFactory / TableConfigPartitioner: thread the column's logical DataType through so UUID columns render canonically instead of as bare hex - UUID_ARRAY entries for the array scalar functions The UUID scalar functions and multi-stage UDF wrappers that were previously part of this layer now live in their own PR (apache#19091) so they can be reviewed and merged in parallel.
Part 7/8 of splitting apache#18140 (logical UUID type). - UuidPartitionFunction: hashes the 16-byte UUID form via Murmur2, matching what an external producer keyed on raw UUID bytes computes - PartitionerFactory / TableConfigPartitioner: thread the column's logical DataType through so UUID columns render canonically instead of as bare hex - UUID_ARRAY entries for the array scalar functions The UUID scalar functions and multi-stage UDF wrappers that were previously part of this layer now live in their own PR (apache#19091) so they can be reviewed and merged in parallel.
Part 7/8 of splitting apache#18140 (logical UUID type). - UuidPartitionFunction: hashes the 16-byte UUID form via Murmur2, matching what an external producer keyed on raw UUID bytes computes - PartitionerFactory / TableConfigPartitioner: thread the column's logical DataType through so UUID columns render canonically instead of as bare hex - UUID_ARRAY entries for the array scalar functions The UUID scalar functions and multi-stage UDF wrappers that were previously part of this layer now live in their own PR (apache#19091) so they can be reviewed and merged in parallel.
Part 7/8 of splitting apache#18140 (logical UUID type). - UuidPartitionFunction: hashes the 16-byte UUID form via Murmur2, matching what an external producer keyed on raw UUID bytes computes - PartitionerFactory / TableConfigPartitioner: thread the column's logical DataType through so UUID columns render canonically instead of as bare hex - UUID_ARRAY entries for the array scalar functions The UUID scalar functions and multi-stage UDF wrappers that were previously part of this layer now live in their own PR (apache#19091) so they can be reviewed and merged in parallel.
Part 7/8 of splitting apache#18140 (logical UUID type). - UuidPartitionFunction: hashes the 16-byte UUID form via Murmur2, matching what an external producer keyed on raw UUID bytes computes - PartitionerFactory / TableConfigPartitioner: thread the column's logical DataType through so UUID columns render canonically instead of as bare hex - UUID_ARRAY entries for the array scalar functions The UUID scalar functions and multi-stage UDF wrappers that were previously part of this layer now live in their own PR (apache#19091) so they can be reviewed and merged in parallel.
Part 7/8 of splitting apache#18140 (logical UUID type). - UuidPartitionFunction: hashes the 16-byte UUID form via Murmur2, matching what an external producer keyed on raw UUID bytes computes - PartitionerFactory / TableConfigPartitioner: thread the column's logical DataType through so UUID columns render canonically instead of as bare hex - UUID_ARRAY entries for the array scalar functions The UUID scalar functions and multi-stage UDF wrappers that were previously part of this layer now live in their own PR (apache#19091) so they can be reviewed and merged in parallel.
Part 7/8 of splitting apache#18140 (logical UUID type). - UuidPartitionFunction: hashes the 16-byte UUID form via Murmur2, matching what an external producer keyed on raw UUID bytes computes - PartitionerFactory / TableConfigPartitioner: thread the column's logical DataType through so UUID columns render canonically instead of as bare hex - UUID_ARRAY entries for the array scalar functions The UUID scalar functions and multi-stage UDF wrappers that were previously part of this layer now live in their own PR (apache#19091) so they can be reviewed and merged in parallel.
Part 7/8 of splitting apache#18140 (logical UUID type). - UuidPartitionFunction: hashes the 16-byte UUID form via Murmur2, matching what an external producer keyed on raw UUID bytes computes - PartitionerFactory / TableConfigPartitioner: thread the column's logical DataType through so UUID columns render canonically instead of as bare hex - UUID_ARRAY entries for the array scalar functions The UUID scalar functions and multi-stage UDF wrappers that were previously part of this layer now live in their own PR (apache#19091) so they can be reviewed and merged in parallel.
Part 7/8 of splitting apache#18140 (logical UUID type). - UuidPartitionFunction: hashes the 16-byte UUID form via Murmur2, matching what an external producer keyed on raw UUID bytes computes - PartitionerFactory / TableConfigPartitioner: thread the column's logical DataType through so UUID columns render canonically instead of as bare hex - UUID_ARRAY entries for the array scalar functions The UUID scalar functions and multi-stage UDF wrappers that were previously part of this layer now live in their own PR (apache#19091) so they can be reviewed and merged in parallel.
Part 7/8 of splitting apache#18140 (logical UUID type). - UuidPartitionFunction: hashes the 16-byte UUID form via Murmur2, matching what an external producer keyed on raw UUID bytes computes - PartitionerFactory / TableConfigPartitioner: thread the column's logical DataType through so UUID columns render canonically instead of as bare hex - UUID_ARRAY entries for the array scalar functions The UUID scalar functions and multi-stage UDF wrappers that were previously part of this layer now live in their own PR (apache#19091) so they can be reviewed and merged in parallel.
Parent tracking issue: #16619
What
Adds UUID scalar functions and their multi-stage UDF wrappers, including a shared polymorphic input contract for UUID semantic operations.
Changes
IS_UUID,TO_UUID,UUID_TO_STRING,UUID_TO_BYTES,UUID_VERSION, andUUID_TIMESTAMPacceptSTRING,BYTES, or logicalUUIDUUID_TO_STRINGalways returns canonical lowercase dashed UUID text for all three input representationsBYTES_TO_UUIDremains a strictBYTESconversion functionUUID_V4/UUID_V7generators remain zero-argument and non-deterministic, preventing compile-time foldingUUIDandUUID_ARRAYtypesUdfwrappers expose the polymorphic contract to the multi-stage registryIS_UUIDreturnsfalseonly for non-null invalid inputsIS_UUIDTO_UUIDUUID_TO_STRINGUUID_TO_BYTESUUID_VERSIONUUID_TIMESTAMPGeneric
BYTESrendering is unchanged and remains hexadecimal. Canonical UUID decoding occurs only inside UUID-aware functions.How to reproduce the fixed behavior
Before the UUID type bridge and wrapper dispatch were included, direct unit calls could pass while ingestion, single-stage, or multi-stage evaluation failed to resolve or render byte-backed logical UUID values correctly.
The following expressions now plan and execute through the polymorphic contract:
All three results are:
Why this is a standalone PR
Split out of #18875 ([UUID 7/8]) so the scalar function surface can be reviewed independently from UUID partitioning.
This branch is based directly on
master. It contains the UUID scalar plumbing required by these functions, rather than relying on unmerged UUID branches. The remaining #18875 scope isUuidPartitionFunction,PartitionerFactory/TableConfigPartitionerwiring, andUUID_ARRAYsupport in array scalar functions.Testing
UuidUtilsTest,UuidConversionFunctionsTest, andScalarTransformFunctionWrapperTest: canonical rendering, all input types, byte-backed dispatch, invalid inputs, timestamps/versions, and null propagationQueryCompilationTest, both plan-node converter tests, andRexExpressionSerDeTest: polymorphic type inference plus UUID/UUID_ARRAY planner and protobuf round tripsUuidUdfTestandPinotFunctionEnvGeneratorTest: wrapper metadata/examples and UUID schema generationtest-compilereactor passed with JDK 25 and-Xlint:allFull feature description and design contract: #18140.