Skip to content

Add UUID scalar functions and multi-stage UDF wrappers - #19091

Merged
xiangfu0 merged 4 commits into
apache:masterfrom
xiangfu0:uuid-split/07a-scalar-udfs
Jul 30, 2026
Merged

Add UUID scalar functions and multi-stage UDF wrappers#19091
xiangfu0 merged 4 commits into
apache:masterfrom
xiangfu0:uuid-split/07a-scalar-udfs

Conversation

@xiangfu0

@xiangfu0 xiangfu0 commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

Parent tracking issue: #16619

What

Adds UUID scalar functions and their multi-stage UDF wrappers, including a shared polymorphic input contract for UUID semantic operations.

Changes

  • IS_UUID, TO_UUID, UUID_TO_STRING, UUID_TO_BYTES, UUID_VERSION, and UUID_TIMESTAMP accept STRING, BYTES, or logical UUID
  • UUID_TO_STRING always returns canonical lowercase dashed UUID text for all three input representations
  • BYTES_TO_UUID remains a strict BYTES conversion function
  • UUID_V4 / UUID_V7 generators remain zero-argument and non-deterministic, preventing compile-time folding
  • UUID Java/Pinot/Calcite type bridges and byte-backed single-stage UUID dispatch are included so this PR is standalone
  • Multi-stage planner/protobuf bridges preserve logical UUID and UUID_ARRAY types
  • Matching custom Udf wrappers expose the polymorphic contract to the multi-stage registry
  • Nulls are propagated by the engine; IS_UUID returns false only for non-null invalid inputs
  • The UDF example harness now supports UUID schemas, canonical wire-value comparison, and generated snapshots for all UUID UDFs
Function STRING BYTES UUID
IS_UUID yes yes yes
TO_UUID yes yes yes
UUID_TO_STRING yes yes yes
UUID_TO_BYTES yes yes yes
UUID_VERSION yes yes yes
UUID_TIMESTAMP yes yes yes

Generic BYTES rendering is unchanged and remains hexadecimal. Canonical UUID decoding occurs only inside UUID-aware functions.

How to reproduce the fixed behavior

Before the UUID type bridge and wrapper dispatch were included, direct unit calls could pass while ingestion, single-stage, or multi-stage evaluation failed to resolve or render byte-backed logical UUID values correctly.

The following expressions now plan and execute through the polymorphic contract:

SELECT
  UUID_TO_STRING('550E8400-E29B-41D4-A716-446655440000'),
  UUID_TO_STRING(UUID_TO_BYTES('550e8400-e29b-41d4-a716-446655440000')),
  UUID_TO_STRING(TO_UUID('550e8400-e29b-41d4-a716-446655440000'));

All three results are:

550e8400-e29b-41d4-a716-446655440000

Why this is a standalone PR

Split out of #18875 ([UUID 7/8]) so the scalar function surface can be reviewed independently from UUID partitioning.

This branch is based directly on master. It contains the UUID scalar plumbing required by these functions, rather than relying on unmerged UUID branches. The remaining #18875 scope is UuidPartitionFunction, PartitionerFactory / TableConfigPartitioner wiring, and UUID_ARRAY support in array scalar functions.

Testing

  • UuidUtilsTest, UuidConversionFunctionsTest, and ScalarTransformFunctionWrapperTest: canonical rendering, all input types, byte-backed dispatch, invalid inputs, timestamps/versions, and null propagation
  • QueryCompilationTest, both plan-node converter tests, and RexExpressionSerDeTest: polymorphic type inference plus UUID/UUID_ARRAY planner and protobuf round trips
  • UuidUdfTest and PinotFunctionEnvGeneratorTest: wrapper metadata/examples and UUID schema generation
  • Generated UDF integration snapshots for all nine UUID functions; no unsupported UUID type or wire-value mismatches
  • 133 focused UUID/scalar/UDF tests passed
  • 259 focused planner/serde/query compilation tests passed
  • Full 63-module test-compile reactor passed with JDK 25 and -Xlint:all
  • Required Spotless, Checkstyle, license-format, and license-check gates passed for all affected modules

Full feature description and design contract: #18140.

@codecov-commenter

codecov-commenter commented Jul 27, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 96.49123% with 8 lines in your changes missing coverage. Please review.
✅ Project coverage is 65.62%. Comparing base (779210b) to head (8474f67).

Files with missing lines Patch % Lines
...unction/scalar/uuid/AbstractUuidInputFunction.java 78.78% 4 Missing and 3 partials ⚠️
...rg/apache/pinot/common/function/FunctionUtils.java 66.66% 1 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##             master   #19091      +/-   ##
============================================
+ Coverage     65.53%   65.62%   +0.09%     
  Complexity     1423     1423              
============================================
  Files          3432     3450      +18     
  Lines        218151   218378     +227     
  Branches      34664    34670       +6     
============================================
+ Hits         142964   143318     +354     
+ Misses        63627    63498     -129     
- Partials      11560    11562       +2     
Flag Coverage Δ
custom-integration1 100.00% <ø> (ø)
integration 100.00% <ø> (ø)
integration1 100.00% <ø> (ø)
integration2 ?
java-25 65.62% <96.49%> (+0.09%) ⬆️
temurin 65.62% <96.49%> (+0.09%) ⬆️
unittests 65.62% <96.49%> (+0.09%) ⬆️
unittests1 57.03% <96.49%> (+0.13%) ⬆️
unittests2 37.91% <18.85%> (+<0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds UUID-focused scalar functions to pinot-common and registers corresponding multi-stage UDF wrappers in pinot-query-runtime, enabling UUID validation, conversion, accessors, and v4/v7 generation through Pinot’s function/UDF surfaces.

Changes:

  • Introduces polymorphic IS_UUID / TO_UUID scalar functions (STRING or BYTES) via a shared dispatch base class.
  • Adds UUID conversion/accessor/generator scalar functions (UUID_TO_BYTES, BYTES_TO_UUID, UUID_TO_STRING, UUID_VERSION, UUID_TIMESTAMP, UUID_V4, UUID_V7).
  • Adds multi-stage UDF wrapper classes and unit tests covering conversions, null handling, and v4/v7 properties.

Reviewed changes

Copilot reviewed 15 out of 15 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
pinot-query-runtime/src/main/java/org/apache/pinot/query/runtime/function/BytesToUuidUdf.java Multi-stage UDF wrapper for BYTES_TO_UUID.
pinot-query-runtime/src/main/java/org/apache/pinot/query/runtime/function/IsUuidUdf.java Multi-stage UDF wrapper for IS_UUID, including example signatures.
pinot-query-runtime/src/main/java/org/apache/pinot/query/runtime/function/ToUuidUdf.java Multi-stage UDF wrapper for TO_UUID.
pinot-query-runtime/src/main/java/org/apache/pinot/query/runtime/function/UuidTimestampUdf.java Multi-stage UDF wrapper for UUID_TIMESTAMP.
pinot-query-runtime/src/main/java/org/apache/pinot/query/runtime/function/UuidToBytesUdf.java Multi-stage UDF wrapper for UUID_TO_BYTES.
pinot-query-runtime/src/main/java/org/apache/pinot/query/runtime/function/UuidToStringUdf.java Multi-stage UDF wrapper for UUID_TO_STRING.
pinot-query-runtime/src/main/java/org/apache/pinot/query/runtime/function/UuidV4Udf.java Multi-stage UDF wrapper for UUID_V4.
pinot-query-runtime/src/main/java/org/apache/pinot/query/runtime/function/UuidV7Udf.java Multi-stage UDF wrapper for UUID_V7.
pinot-query-runtime/src/main/java/org/apache/pinot/query/runtime/function/UuidVersionUdf.java Multi-stage UDF wrapper for UUID_VERSION.
pinot-common/src/main/java/org/apache/pinot/common/function/scalar/uuid/AbstractStringOrBytesUuidFunction.java Shared dispatch helper for STRING/BYTES UUID scalar functions.
pinot-common/src/main/java/org/apache/pinot/common/function/scalar/uuid/IsUuidScalarFunction.java Implements polymorphic IS_UUID.
pinot-common/src/main/java/org/apache/pinot/common/function/scalar/uuid/ToUuidScalarFunction.java Implements polymorphic TO_UUID with Calcite UUID return typing.
pinot-common/src/main/java/org/apache/pinot/common/function/scalar/uuid/UuidConversionFunctions.java Adds UUID conversion/accessor/generator scalar functions (incl. non-deterministic v4/v7).
pinot-common/src/test/java/org/apache/pinot/common/function/scalar/StringFunctionsTest.java Adds mixed-case UUID round-trip coverage via toUUIDBytes/fromUUIDBytes.
pinot-common/src/test/java/org/apache/pinot/common/function/scalar/uuid/UuidConversionFunctionsTest.java New unit tests for conversions, accessors, generators, and null handling.
Comments suppressed due to low confidence (1)

pinot-common/src/main/java/org/apache/pinot/common/function/scalar/uuid/IsUuidScalarFunction.java:50

  • Same null-handling issue as the STRING overload: marking the BYTES overload as nullableParameters=true forces evaluation on null inputs and returns false instead of propagating null. This also diverges from the IsUuidUdf examples which expect null when null handling is enabled.
      BYTES_FUNCTION_INFO =
          new FunctionInfo(IsUuidScalarFunction.class.getMethod("isUuid", byte[].class), IsUuidScalarFunction.class,
              true);

@xiangfu0
xiangfu0 force-pushed the uuid-split/07a-scalar-udfs branch from 5846a27 to e45cb81 Compare July 28, 2026 08:02
xiangfu0 added a commit to xiangfu0/pinot that referenced this pull request Jul 28, 2026
Part 7/8 of splitting apache#18140 (logical UUID type).

- UuidPartitionFunction: hashes the 16-byte UUID form via Murmur2, matching what
  an external producer keyed on raw UUID bytes computes
- PartitionerFactory / TableConfigPartitioner: thread the column's logical
  DataType through so UUID columns render canonically instead of as bare hex
- UUID_ARRAY entries for the array scalar functions

The UUID scalar functions and multi-stage UDF wrappers that were previously part
of this layer now live in their own PR (apache#19091) so they can be reviewed and
merged in parallel.
@xiangfu0
xiangfu0 force-pushed the uuid-split/07a-scalar-udfs branch from e45cb81 to fe30a2e Compare July 28, 2026 08:35
@xiangfu0

Copy link
Copy Markdown
Contributor Author

Now disjoint from #18875

#18875 has been updated to drop these files, so the two PRs no longer overlap and can be reviewed and merged in either order:

Doc comments here were also converted to /// markdown Javadoc to match the surrounding UUID code (UuidPartitionFunction, TableConfigPartitioner, DataSchema).

Re-verified on the current master base: pinot-query-runtime -am test-compile succeeds, UuidConversionFunctionsTest + StringFunctionsTest pass 191/191, and spotless / checkstyle / license are clean.

@xiangfu0
xiangfu0 force-pushed the uuid-split/07a-scalar-udfs branch from fe30a2e to 337463a Compare July 29, 2026 08:01
@Jackie-Jiang Jackie-Jiang added the feature New functionality label Jul 29, 2026
@Jackie-Jiang Jackie-Jiang added documentation Improvements or additions to documentation release-notes Referenced by PRs that need attention when compiling the next release notes labels Jul 29, 2026
xiangfu0 added 3 commits July 29, 2026 19:27
Adds the UUID scalar function surface on top of the logical UUID type:

- IS_UUID / TO_UUID: polymorphic over STRING and BYTES, sharing dispatch via
  AbstractStringOrBytesUuidFunction
- UUID_TO_BYTES, BYTES_TO_UUID, UUID_TO_STRING, UUID_VERSION, UUID_TIMESTAMP
- UUID_V4 / UUID_V7 generators, marked isDeterministic = false so the broker's
  CompileTimeFunctionsInvoker does not fold one value across every row
- Matching multi-stage Udf wrappers so the functions are registered with the
  MSE UDF registry

Depends only on UuidUtils (apache#18869) and ColumnDataType.UUID (apache#18871), both
already on master. Split out of apache#18875 so it can be reviewed independently of
the UUID partitioning work.
@xiangfu0
xiangfu0 force-pushed the uuid-split/07a-scalar-udfs branch from 337463a to 102c23e Compare July 30, 2026 02:47
@xiangfu0
xiangfu0 merged commit c500e4f into apache:master Jul 30, 2026
11 of 12 checks passed
@xiangfu0
xiangfu0 deleted the uuid-split/07a-scalar-udfs branch July 30, 2026 07:55
xiangfu0 added a commit to xiangfu0/pinot that referenced this pull request Jul 30, 2026
Part 7/8 of splitting apache#18140 (logical UUID type).

- UuidPartitionFunction: hashes the 16-byte UUID form via Murmur2, matching what
  an external producer keyed on raw UUID bytes computes
- PartitionerFactory / TableConfigPartitioner: thread the column's logical
  DataType through so UUID columns render canonically instead of as bare hex
- UUID_ARRAY entries for the array scalar functions

The UUID scalar functions and multi-stage UDF wrappers that were previously part
of this layer now live in their own PR (apache#19091) so they can be reviewed and
merged in parallel.
@xiangfu0

Copy link
Copy Markdown
Contributor Author

Documentation follow-up: pinot-contrib/pinot-docs#952

xiangfu0 added a commit to pinot-contrib/pinot-docs that referenced this pull request Jul 30, 2026
xiangfu0 added a commit to xiangfu0/pinot that referenced this pull request Jul 31, 2026
Part 7/8 of splitting apache#18140 (logical UUID type).

- UuidPartitionFunction: hashes the 16-byte UUID form via Murmur2, matching what
  an external producer keyed on raw UUID bytes computes
- PartitionerFactory / TableConfigPartitioner: thread the column's logical
  DataType through so UUID columns render canonically instead of as bare hex
- UUID_ARRAY entries for the array scalar functions

The UUID scalar functions and multi-stage UDF wrappers that were previously part
of this layer now live in their own PR (apache#19091) so they can be reviewed and
merged in parallel.
xiangfu0 added a commit to xiangfu0/pinot that referenced this pull request Aug 2, 2026
Part 7/8 of splitting apache#18140 (logical UUID type).

- UuidPartitionFunction: hashes the 16-byte UUID form via Murmur2, matching what
  an external producer keyed on raw UUID bytes computes
- PartitionerFactory / TableConfigPartitioner: thread the column's logical
  DataType through so UUID columns render canonically instead of as bare hex
- UUID_ARRAY entries for the array scalar functions

The UUID scalar functions and multi-stage UDF wrappers that were previously part
of this layer now live in their own PR (apache#19091) so they can be reviewed and
merged in parallel.
xiangfu0 added a commit to xiangfu0/pinot that referenced this pull request Aug 3, 2026
Part 7/8 of splitting apache#18140 (logical UUID type).

- UuidPartitionFunction: hashes the 16-byte UUID form via Murmur2, matching what
  an external producer keyed on raw UUID bytes computes
- PartitionerFactory / TableConfigPartitioner: thread the column's logical
  DataType through so UUID columns render canonically instead of as bare hex
- UUID_ARRAY entries for the array scalar functions

The UUID scalar functions and multi-stage UDF wrappers that were previously part
of this layer now live in their own PR (apache#19091) so they can be reviewed and
merged in parallel.
xiangfu0 added a commit to xiangfu0/pinot that referenced this pull request Aug 4, 2026
Part 7/8 of splitting apache#18140 (logical UUID type).

- UuidPartitionFunction: hashes the 16-byte UUID form via Murmur2, matching what
  an external producer keyed on raw UUID bytes computes
- PartitionerFactory / TableConfigPartitioner: thread the column's logical
  DataType through so UUID columns render canonically instead of as bare hex
- UUID_ARRAY entries for the array scalar functions

The UUID scalar functions and multi-stage UDF wrappers that were previously part
of this layer now live in their own PR (apache#19091) so they can be reviewed and
merged in parallel.
xiangfu0 added a commit to xiangfu0/pinot that referenced this pull request Aug 5, 2026
Part 7/8 of splitting apache#18140 (logical UUID type).

- UuidPartitionFunction: hashes the 16-byte UUID form via Murmur2, matching what
  an external producer keyed on raw UUID bytes computes
- PartitionerFactory / TableConfigPartitioner: thread the column's logical
  DataType through so UUID columns render canonically instead of as bare hex
- UUID_ARRAY entries for the array scalar functions

The UUID scalar functions and multi-stage UDF wrappers that were previously part
of this layer now live in their own PR (apache#19091) so they can be reviewed and
merged in parallel.
xiangfu0 added a commit to xiangfu0/pinot that referenced this pull request Aug 5, 2026
Part 7/8 of splitting apache#18140 (logical UUID type).

- UuidPartitionFunction: hashes the 16-byte UUID form via Murmur2, matching what
  an external producer keyed on raw UUID bytes computes
- PartitionerFactory / TableConfigPartitioner: thread the column's logical
  DataType through so UUID columns render canonically instead of as bare hex
- UUID_ARRAY entries for the array scalar functions

The UUID scalar functions and multi-stage UDF wrappers that were previously part
of this layer now live in their own PR (apache#19091) so they can be reviewed and
merged in parallel.
xiangfu0 added a commit to xiangfu0/pinot that referenced this pull request Aug 5, 2026
Part 7/8 of splitting apache#18140 (logical UUID type).

- UuidPartitionFunction: hashes the 16-byte UUID form via Murmur2, matching what
  an external producer keyed on raw UUID bytes computes
- PartitionerFactory / TableConfigPartitioner: thread the column's logical
  DataType through so UUID columns render canonically instead of as bare hex
- UUID_ARRAY entries for the array scalar functions

The UUID scalar functions and multi-stage UDF wrappers that were previously part
of this layer now live in their own PR (apache#19091) so they can be reviewed and
merged in parallel.
xiangfu0 added a commit to xiangfu0/pinot that referenced this pull request Aug 7, 2026
Part 7/8 of splitting apache#18140 (logical UUID type).

- UuidPartitionFunction: hashes the 16-byte UUID form via Murmur2, matching what
  an external producer keyed on raw UUID bytes computes
- PartitionerFactory / TableConfigPartitioner: thread the column's logical
  DataType through so UUID columns render canonically instead of as bare hex
- UUID_ARRAY entries for the array scalar functions

The UUID scalar functions and multi-stage UDF wrappers that were previously part
of this layer now live in their own PR (apache#19091) so they can be reviewed and
merged in parallel.
xiangfu0 added a commit to xiangfu0/pinot that referenced this pull request Aug 8, 2026
Part 7/8 of splitting apache#18140 (logical UUID type).

- UuidPartitionFunction: hashes the 16-byte UUID form via Murmur2, matching what
  an external producer keyed on raw UUID bytes computes
- PartitionerFactory / TableConfigPartitioner: thread the column's logical
  DataType through so UUID columns render canonically instead of as bare hex
- UUID_ARRAY entries for the array scalar functions

The UUID scalar functions and multi-stage UDF wrappers that were previously part
of this layer now live in their own PR (apache#19091) so they can be reviewed and
merged in parallel.
xiangfu0 added a commit to xiangfu0/pinot that referenced this pull request Aug 9, 2026
Part 7/8 of splitting apache#18140 (logical UUID type).

- UuidPartitionFunction: hashes the 16-byte UUID form via Murmur2, matching what
  an external producer keyed on raw UUID bytes computes
- PartitionerFactory / TableConfigPartitioner: thread the column's logical
  DataType through so UUID columns render canonically instead of as bare hex
- UUID_ARRAY entries for the array scalar functions

The UUID scalar functions and multi-stage UDF wrappers that were previously part
of this layer now live in their own PR (apache#19091) so they can be reviewed and
merged in parallel.
xiangfu0 added a commit to xiangfu0/pinot that referenced this pull request Aug 9, 2026
Part 7/8 of splitting apache#18140 (logical UUID type).

- UuidPartitionFunction: hashes the 16-byte UUID form via Murmur2, matching what
  an external producer keyed on raw UUID bytes computes
- PartitionerFactory / TableConfigPartitioner: thread the column's logical
  DataType through so UUID columns render canonically instead of as bare hex
- UUID_ARRAY entries for the array scalar functions

The UUID scalar functions and multi-stage UDF wrappers that were previously part
of this layer now live in their own PR (apache#19091) so they can be reviewed and
merged in parallel.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation feature New functionality release-notes Referenced by PRs that need attention when compiling the next release notes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants