Proposal: Stateful Stream Processing in Apache Pinot

[manikBS]

Currently, Pinot supports real-time ingestion with limited stateless transformations. More advanced use cases—such as stateful aggregations (GROUP BY), stream–stream joins, deduplication, and windowed computations—require external systems like Apache Spark or Apache Beam, adding operational complexity and latency.

This proposal is inspired by systems like Materialize, which integrate stateful stream processing with storage and querying, while adapting the design to fit Pinot's distributed architecture.

I have prepared a detailed architecture proposal and would appreciate the community's feedback before moving forward with implementation. If there is interest, I'd be happy to share the complete design in a follow-up email.

https://docs.google.com/document/d/1rFgYSyxmKgEJrihVbamhZgYQSRx1YW0cBYDA9C1vCpk/view

[manikBS]

Hi @Jackie-Jiang ,
Can you assign this to me?
And also I wanted to check whether it aligns with the project's vision and roadmap.

[Jackie-Jiang]

What is the main benefit of using this new component vs using Flink?

[manikBS]

• Operational cost: The Flink approach requires maintaining separate Pinot and Flink clusters and kafka cluster for transferring data from flink to pinot. With the native approach, a single Pinot cluster can handle both stateful streaming operations (e.g. joins) and queries.
• Latency: In the Flink approach, events typically go through Kafka -> Flink -> Kafka -> Pinot before becoming queryable. The native approach can directly update Pinot's realtime segments, eliminating the intermediate Kafka hop and reducing the latency between receiving an event from Kafka and making the resulting state queryable.
• Architecture: Pinot currently handles ingestion and query serving through its servers. The proposed design introduces dedicated Stream Servers for ingestion and stateful processing, while query serving continues through the existing servers
• Fault tolerance: Flink's checkpoint-based fault tolerance relies on checkpoin barriers and coordinated snapshots, which can be challenging when the upstream source is not fully under our control. In the proposed design state can be checkpointed at the realtime segment creation boundary as default checkpoint within Pinot.

[Jackie-Jiang]

cc @xiangfu0 @kishoreg for feedback