[Umbrella] First-class UUID type support

[ankitsultana]

Creating an issue to gauge the community interest.

UUIDs are super-common in Pinot, and I am sure this is not limited to us at Uber. At present users can only use String columns to deal with UUIDs. This means that:

• Storage: They are stored as 36 bytes in uncompressed file formats.
• Scans: They are scanned as 36 bytes, and during conversion to String another 36 bytes are allocated for the String's internal byte buffer. While the allocation buffer is usually re-used, the String internal buffer is not. Moreover, it costs additional CPU cycles to copy over the larger byte count.
• In-Memory Representation: After scanning, UUIDs are passed around as 36 byte values. This adds a memory/cpu overhead to a lot of operations like data shuffles, open-addressed hash table comparisons, etc.

UUIDs in the end are 2 long values and can be represented using only 16 bytes. There are some usability benefits too, but regardless wanted to share this as something we are exploring.

Last year we had released the UUID Hash Function for Upsert Primary Keys and that has been quite useful at Uber in increasing the per-server primary key capacity: #12538

---

Implementation tracking

This is the umbrella tracker for first-class logical UUID support in Pinot. The original monolithic implementation, design contract, migration notes, and benchmarks are in #18140. That implementation was split into reviewable layers below.

Core implementation

Merge the numbered stack from bottom to top unless a PR is called out as independent.

• [UUID 1/8] Add logical UUID type foundation (pinot-spi) #18869 — logical UUID type foundation (pinot-spi) — merged
• [UUID 2/8] UUID ingest and segment storage #18870 — UUID ingest and segment storage — merged
• [UUID 3/8] UUID result rendering (DataSchema, Arrow/JSON encoders) #18871 — UUID result rendering (DataSchema, Arrow, and JSON) — merged
• [UUID 4/8] Server-side predicate evaluation for the logical UUID type #18872 — server-side predicates, CAST, and transforms
• [UUID 5/8] UUID aggregation, group-by and distinct #18873 — aggregation, group-by, and distinct
• [UUID 6/8] UUID multi-stage engine (planner + runtime) #18874 — multi-stage planner and runtime
• [UUID 7/8] UUID partitioning #18875 — UUID partitioning
• [UUID 8/8] UUID integration tests, benchmarks and docs #18876 — integration tests, benchmarks, and documentation
• Add UUID scalar functions and multi-stage UDF wrappers #19091 — scalar functions and multi-stage UDF wrappers — independent; split from [UUID 7/8] UUID partitioning #18875

Stack note: #18873 is currently stacked on #18872, #18874 on #18873, and #18875 on #18874. #18876 still carries the older layer-7 UDF commit, so it should be restacked on the current #18875 and reconciled with #19091 before merge.

Supporting merged work

• Make VARIANT/UUID non reserved keyword #16471 — keep UUID/VARIANT non-reserved after the Calcite upgrade
• Refactor Avro/Parquet RecordExtractor; finalize logical-type contract #18400 — Avro/Parquet record-extractor logical UUID contract
• Rework UUID type conversion to use java.util.UUID as the external form #18927 — align external java.util.UUID and internal byte[16] conversion
• Fix raw BYTES column min/max generation #18952 — raw BYTES min/max correctness split from [UUID 2/8] UUID ingest and segment storage #18870 review
• Add MV BYTES and BIG_DECIMAL support to GenericRow serializer/deserializer #18969 — MV BYTES/BIG_DECIMAL GenericRow SerDe split from [UUID 2/8] UUID ingest and segment storage #18870 review
• Map recommender Avro schema by original data type, not stored type #18970 — map recommender Avro schemas from logical data types
• Generate UUID sample data in the schema recommender #18973 — generate UUID sample data in the schema recommender
• Render non-String scalars in jsonPathString without JSON quotes #19029 — render UUID and other non-String scalars in jsonPathString without JSON quotes
• Generate Avro schemas from Pinot logical data types (#19071) #19073 — generate Avro schemas from Pinot logical data types

Documentation PRs

• docs: document recommender UUID support (apache/pinot#18973) pinot-contrib/pinot-docs#917 — recommender UUID support
• docs: Document UUID ingest and storage pinot-contrib/pinot-docs#936 — UUID ingest and storage
• docs: describe UUID query result rendering pinot-contrib/pinot-docs#940 — UUID query result rendering

Existing UUID capabilities

• Adding byte functions for UUIDs #11988 — UUID string/byte scalar helpers
• Hash Function for UUID Primary Keys #12538 — UUID hash function for upsert primary keys

Historical or superseded PRs

• Make UUID and VARIANT non-reserved keywords in the parser configs to preserve backward compatibility post Calcite upgrade #16470 — earlier parser-compatibility PR, superseded by Make VARIANT/UUID non reserved keyword #16471
• UUID support for Pinot #17083 — earlier broad UUID support attempt (closed)
• Fix UUID gRPC query path to emit string literals for UUID columns #18192 — isolated UUID gRPC query-path fix (closed; folded into the later implementation)
• Add first-class logical UUID data type #18239 — earlier monolithic logical UUID iteration (closed; superseded by Add first-class logical UUID type support #18140)
• Remove redundant UUID special-casing in FieldSpec.DataType #18926 — UUID FieldSpec.DataType cleanup proposal (closed; replaced by Rework UUID type conversion to use java.util.UUID as the external form #18927)

Completion criteria

• Merge the remaining core stack and Add UUID scalar functions and multi-stage UDF wrappers #19091.
• Cover offline, realtime, and upsert ingestion plus single-stage and multi-stage query behavior.
• Verify predicates, casts, rendering, aggregation/grouping, joins, partitioning, and UDF behavior.
• Land integration tests, benchmarks, user documentation, migration guidance, and rolling-upgrade notes.
• Close or supersede Add first-class logical UUID type support #18140 after the split implementation is complete.

[tarun11Mavani]

@ankitsultana this is an excellent suggestion. I ran a micro benchmarking for some of the common operations on UUIDs such as equality, parsing, storing, serialization, sorting etc and saw 2x-5x improvements when using long[], byte[] or Java's native UUID.
I also saw that Binary UUID format is 5x faster for parsing/serialization (critical for Pinot I/O) but Native UUID objects are 2-4x faster for in-memory operations (comparisons, equality, sorting).
I would like to work on this. I can spend more time on this and present a high level proposal on this. Please assign this issue to me.

[github-actions]

This issue has been automatically marked as stale because it has not had any activity in the last 150 days. It will be closed in 30 days if no further activity occurs. If this issue is still relevant, please leave a comment to keep it open. Thank you for your contributions!

[ankitsultana]

@xiangfu0 is adding this finally 🚀 #18140