Several Pinot workflows rely on broker routing converging with server-side segment state changes before the old serving path becomes unavailable. A stale routing entry can send a query to a server after that server has dropped the segment, producing SERVER_SEGMENT_MISSING.
-
Server readiness versus broker readiness
Issue #16565 describes a server returning healthy before brokers had resumed routing to it. During a rolling restart, this allowed the next server to be restarted while neither replica was queryable.
-
Segment deletion / retention
Deletion has the inverse ordering requirement. The controller updates IdealState, servers asynchronously process the resulting transition and remove the segment, and brokers asynchronously rebuild routing. If a broker routes a query using the old assignment after the server has removed the segment, the query fails.
This is less likely for ordinary retention when time pruning excludes expired segments, but it remains possible for queries spanning the retention boundary and for explicit segment deletion.
- Rebalance transition
Even with add-before-remove rebalance, a query can be dispatched immediately before the source server’s state transition reaches broker routing. If the source has removed the segment by request handling time. We have observed the following errors at least 4 times in a month due to this:
Query processing exceptions:
{235=1 segments [<segment>] missing on server: <server>}
Several Pinot workflows rely on broker routing converging with server-side segment state changes before the old serving path becomes unavailable. A stale routing entry can send a query to a server after that server has dropped the segment, producing SERVER_SEGMENT_MISSING.
Server readiness versus broker readiness
Issue #16565 describes a server returning healthy before brokers had resumed routing to it. During a rolling restart, this allowed the next server to be restarted while neither replica was queryable.
Segment deletion / retention
Deletion has the inverse ordering requirement. The controller updates IdealState, servers asynchronously process the resulting transition and remove the segment, and brokers asynchronously rebuild routing. If a broker routes a query using the old assignment after the server has removed the segment, the query fails.
This is less likely for ordinary retention when time pruning excludes expired segments, but it remains possible for queries spanning the retention boundary and for explicit segment deletion.
Even with add-before-remove rebalance, a query can be dispatched immediately before the source server’s state transition reaches broker routing. If the source has removed the segment by request handling time. We have observed the following errors at least 4 times in a month due to this: